feat(metrics): count workloads the operator declines to scan

[JanWelker]

Refs #1668. Independent of #3072 — that one counts reports rejected on write; this counts scans that were never attempted. Together they cover both ways a workload ends up with no report.

The gap

SubmitScanJob returns nil in four places:

if errors.Is(err, kube.ErrReplicaSetNotFound) || errors.Is(err, kube.ErrNoRunningPods) ||
    errors.Is(err, kube.ErrUnSupportedKind) {
    log.V(1).Info("ignoring vulnerability scan", "reason", err)
    return nil
}
// ...
if len(scanJob.Spec.Template.Spec.Containers) == 0 {
    log.V(1).Info("ignoring vulnerability scan", "no containers in the pod spec for resource", ...)
    return nil
}

Each is a deliberate decision not to scan, and each leaves nothing behind — no scan job, no report, no event. Every existing metric is derived from report objects that exist, which by construction cannot cover a report that was never going to be created. So a skipped workload and a workload scanned clean are indistinguishable.

What this adds

trivy_workload_scan_skipped_total{kind="ReplicaSet",namespace="rook-ceph",reason="no_running_pods"} 2

reason is a closed set of four values, so cardinality stays bounded:

reason	meaning
replicaset_not_found	current-revision ReplicaSet could not be resolved
no_running_pods	no running pod to read a pod spec from
unsupported_kind	kind is not scannable
no_containers	generated scan job had no containers

ScanSkippedReason returns "" for anything else, so an unrelated error still propagates as a failure instead of being silently counted as a skip.

Deliberately not changing the log level

Promoting those V(1) lines would be noisy — no_running_pods fires on every reconcile of a scaled-to-zero Deployment. A counter you can aggregate is the right shape; a line per event is not:

sum by (namespace, kind, reason) (increase(trivy_workload_scan_skipped_total[24h]))

Why I went looking

In one cluster, 79 containers have a ConfigAuditReport and no VulnerabilityReport. Config audits need no scan job; vulnerability and exposed-secret reports do — and neither exists for them. Forcing a reconcile produced no scan job at all, while an unrelated new workload was scanned within ten seconds, so the operator is responsive. Every Deployment's deployment.kubernetes.io/revision matches its ReplicaSet, which rules out ErrReplicaSetNotFound via a revision mismatch.

Which of the four applies, I still don't know — and that is exactly the point. There is currently no way to find out short of enabling dev-mode logging or rebuilding the operator. I'd rather ship the instrument than guess at the cause.

Testing

• Unit tests for the classifier (including a wrapped sentinel via %w, and that unrelated errors are not classified as skips) and for the counter via testutil.CollectAndCompare.
• go build ./..., go vet, and the existing pkg/metrics, pkg/vulnerabilityreport, pkg/vulnerabilityreport/controller and pkg/operator suites all pass on go1.26.7 with GOEXPERIMENT=jsonv2.
• Docs section added to docs/tutorials/integrations/metrics.md.

No chart values changed, so no mage generate:docs run was needed.

🤖 Generated with Claude Code