Sync CNV runbook VirtHandlerDown.md (Updated at 2026-06-14 13:35:42 +0000 UTC)

alerts/openshift-virtualization-operator/VirtHandlerDown.md

@@ -2,58 +2,97 @@
 
 ## Meaning
 
-No running `virt-handler` pod has been detected for 10 minutes..
-
-The `virt-handler` runs on every node that can schedule VMIs. It is
-responsible for domain lifecycle, network configuration, and other
-node-level operations for virtual machine instances.
+No running `virt-handler` pod has been detected for 10 minutes.
+
+The alert expression evaluates
+`cluster:kubevirt_virt_handler_pods_running:count == 0` with a `for`
+duration of 10 minutes. The recording rule counts pods in `Running`
+phase matching `virt-handler-.*`. Because the recording rule
+aggregates across all pods, the alert carries only static labels
+(`severity`, `operator_health_impact`) and does not include
+per-pod or per-node dimensions.
+
+The `virt-handler` runs as a DaemonSet on every node that can
+schedule VMIs. It is responsible for domain lifecycle, network
+configuration, and other node-level operations for virtual machine
+instances.
+
+In newer versions of OpenShift Virtualization, the alert expression is reworked
+to
+surface additional diagnostic labels (`pod`, `node`, `reason`) when
+a container waiting reason is available. If your alert includes
+these labels, see step 1 of the diagnosis below.
 
 ## Impact
 
-Virtual machine instances (VMIs) on affected nodes cannot be managed properly.
-New VMIs may not start on nodes without a running `virt-handler`, and
-existing VMIs may not receive updates or clean shutdowns.
+Virtual machine instances (VMIs) on affected nodes cannot be managed
+properly. New VMIs may not start on nodes without a running
+`virt-handler`, and existing VMIs may not receive updates or clean
+shutdowns.
 
 ## Diagnosis
 
-1. Set the `NAMESPACE` environment variable:
+1. **Check the alert labels**:
+
+   If the alert includes a `reason` label (for example,
+   `CrashLoopBackOff`, `ErrImagePull`, `ImagePullBackOff`), it
+   directly identifies why `virt-handler` is down. The `pod` and
+   `node` labels identify the affected pod and node. Skip to
+   [Mitigation](#mitigation) for the matching root cause. If these
+   labels are not present, continue with the steps below.
+
+2. Set the `NAMESPACE` environment variable:
 
    ```bash
-   $ export NAMESPACE="$(oc get kubevirt -A -o custom-columns="":.metadata.namespace)"
+   $ export NAMESPACE="$(oc get kubevirt -A \
+       -o custom-columns="":.metadata.namespace)"
    ```
 
-2. Check the status of the `virt-handler` DaemonSet and pods:
+3. Check the status of the `virt-handler` DaemonSet and pods:
 
    ```bash
    $ oc -n $NAMESPACE get daemonset virt-handler -o yaml
    $ oc -n $NAMESPACE get pods -l kubevirt.io=virt-handler
    ```
 
-3. Check DaemonSet events and pod status:
+4. Check DaemonSet events and pod status:
 
    ```bash
    $ oc -n $NAMESPACE describe daemonset virt-handler
    $ oc -n $NAMESPACE describe pod -l kubevirt.io=virt-handler
    ```
 
-4. Check for node issues (e.g. nodes not ready or taints):
+5. Check for node issues (for example, nodes not ready or taints):
 
    ```bash
    $ oc get nodes
    ```
 
-5. If any `virt-handler` pod exists, review its logs:
+6. If any `virt-handler` pod exists, review its logs:
 
    ```bash
-   $ oc -n $NAMESPACE logs <virt-handler-pod-name> --previous
-   $ oc -n $NAMESPACE logs <virt-handler-pod-name>
+   $ oc -n $NAMESPACE logs -l kubevirt.io=virt-handler \
+       --previous
+   $ oc -n $NAMESPACE logs -l kubevirt.io=virt-handler
    ```
 
 ## Mitigation
 
-Identify why `virt-handler` pods are down (e.g. DaemonSet not scheduling, pods
-crashing, node issues, image pull failures) and restore the DaemonSet so
-`virt-handler` runs on schedulable nodes.
+Try to identify the root cause and resolve the issue. Common
+causes include:
+
+- **CrashLoopBackOff**: The `virt-handler` container is crashing
+  repeatedly. Check the pod logs for the root cause (panic, OOM,
+  misconfiguration).
+- **ErrImagePull / ImagePullBackOff**: The container image cannot
+  be pulled. Verify the image reference, registry availability,
+  and pull secrets.
+- **Pods absent**: No `virt-handler` pods exist. Check whether the
+  DaemonSet has been deleted, is not scheduling due to node taints,
+  or is blocked by resource constraints.
+- **Node issues**: Nodes may be in `NotReady` state, under resource
+  pressure, or have taints that prevent the DaemonSet from
+  scheduling.
 
 If you cannot resolve the issue, log in to the
 [Customer Portal](https://access.redhat.com) and open a support case,