fix: zero-fill missing status codes to stop missing 5xx alert swings

[shreekarshetty]

Summary

Rare status codes (507, 555, etc) go stale and drop out of zone sums when quiet for a few minutes, causing phantom drop/recovery swings that trip CloudflareZone5xxZscoreWarn.

This fixes this by zero-filling known but absent statuses per minute instead of leaving them silent. It covers both v2 (Adaptive Groups) and non-v2 (1m Groups) status metrics

See https://supabase.slack.com/archives/C0B4U36LJEB/p1786699723052219

Claude details

Root cause

Cloudflare's Adaptive Groups API only returns a (zone, status) row in a minute when that status had at least one request. Engine.Ingest only re-emits a counter when an Observation for that exact (key, bucket) arrives, so a rare 5xx code (501, 507, 520, 523, 525, 555 on supabase.co; 504, 530 on snapcloud.dev) that goes quiet for a few minutes just stops getting pushed. Once the gap outlives VictoriaMetrics' staleness window, the series drops out of sum by (zone) (...) entirely, making the zone's total 5xx count appear to fall - then jump back the instant the code reappears. That swing is what trips cloudflare:zone_5xx_zscore.

Verified against the three real alerts (08:45, 09:15, 09:25 UTC, 2026-08-14):

• supabase.co: status 555 gap 09:14:30-09:32 UTC (17.5min), 507 gap 09:24-09:36 UTC (12min), plus shorter gaps on 501/520/525
• snapcloud.dev: status 504 gaps of 12-14min, 530 appears for a single 30s sample then vanishes

httpRequests1mGroups (non-v2) has the identical shape: it also only lists a status in its response when that status had at least one request that minute. Confirmed directly against 3 days of supabase.co data: status 507 had 174 gaps (max 51min), 523 had 114 gaps (max 119min), 526's max gap ran 1093min, 599's max gap ran over 50 hours.

Fix

A superseded earlier version of this fix (#47, closed) used a timer: re-push every known key's last value every tick unless it's been idle past a cap, to bound (but not eliminate) the risk of masking a genuine sustained outage. That was solving with a guess something Cloudflare already answers directly: a minute row that comes back at all is a complete, authoritative list of every status that occurred - a known status missing from it is a confirmed zero, not an ambiguous gap.

Both flattenHTTPAdaptiveGroups (v2) and flattenHTTP1mGroups (non-v2) now track, per zone, every status code ever seen on their respective path and zero-fill any that a given minute's response doesn't mention, feeding a real Value: 0 Observation through the existing (already-tested) counter-chain/tracker pipeline - no new Engine API surface. The zero-fill logic itself (zoneStatusSet, zeroFillMissingStatuses) is shared between both paths; each keeps its own known-status registry (knownStatusesV2 / knownStatuses1m) since the two datasets are fetched and timed independently.

Because zero-fill only ever fires for a (zone, minute) a fetch actually, successfully returned data for, it can't mask a real outage the way the timer could: if a chunk fetch fails outright, this never runs for that zone, and its keys fall through to real staleness exactly as today - CloudflareZoneRequestsMetricMissing still catches a genuine sustained failure at its normal ~10 minute speed, no grace period needed.

Testing

cfetch/zerofill_test.go (v2) and cfetch/zerofill_1m_test.go (non-v2) - TDD throughout: tests written first (confirmed failing to compile without the relevant field/method), then implemented. Covers: zero-fill of a previously-seen-now-absent status, no zero-fill before a status has ever been seen, zero-fill knowledge isolated per zone, the existing enabled metric filter still works, and (non-v2 only) every other metric flattenHTTP1mGroups emits is unaffected by the refactor. Full suite (go test ./...), go vet ./..., and gofmt all clean.

Scope

Left cfgql.FetchZones's silent per-chunk error swallowing alone (real defect - no error propagation, no metric - but this fix's safety doesn't depend on it, since zero-fill inherently can't fire off a failed fetch).

⚡ Built with Claude Code

[davidbirdsong]

Note for history: what Engine.seed does, and why it's not the same thing as the new known-status maps

This PR adds knownStatusesV2/knownStatuses1m on cfetch.Fetcher. They're populated only from statuses this process has itself observed via Fetch(), live or backfill. On restart, both maps reset to empty and rebuild from scratch as new responses arrive.

It's worth being explicit about how this relates to Engine.seed() (converge/engine.go), since the two are easy to conflate: seed() queries VictoriaMetrics at startup with selector {__name__=~"cloudflare_zone_.*"}, which happens to match both status metric names (cloudflare_zone_requests_status, cloudflare_zone_requests_status_v2). So the query VM seed already runs does return, as a side effect, every (zone, status) pair VM has recently recorded. It would be reasonable to look at that and assume it already solves cold-start for the known-status maps too. It doesn't: seed() only reads sample.Value from each result to call e.SeedChainBase(sample.Key, sample.Value), restoring counter continuity so a restarted process doesn't reset Prometheus counters to zero. It never inspects the zone/status labels on those same samples, and nothing about that result reaches cfetch.Fetcher.

The two mechanisms solve different problems:

• Engine.seed / ChainSeeder: counter continuity across restarts. Lives in converge, generic across any Fetcher/Sink pair.
• knownStatusesV2/knownStatuses1m: preventing a quiet status code from aging out of VM's staleness window. Lives in cfetch, specific to this one Fetcher implementation.

Because converge.Run only knows Fetcher by its single method interface, there's currently no path for VM seeded data to reach cfetch's internal maps, and reusing seed()'s query for that purpose would mean coupling the generic converge package to cfetch specific state, plus trusting that VM's pre-existing history is itself free of the exact gap pattern this PR fixes. That's a separate, smaller piece of work if we ever want to close the narrow post-restart window before a status is first re-observed. It's out of scope here: this PR closes the open-ended, never self-healing version of the staleness bug; the cold-start window it leaves behind is bounded and resolves itself the next time each status reappears.