Skip to content

πŸ› fix: drive Activity Monitor CI counts from check webhooks - #16528

Merged
clubanderson merged 2 commits into
v5from
fix/webhook-ci-counts
Nov 4, 2026
Merged

clubanderson merged 2 commits into
v5from
fix/webhook-ci-counts

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #16521

Why rows read CI ?/? Β· no checks reported yet

checks_passed/checks_total (PR #16494) are computed from pr.CIActivity, whose only producer was enrichPRCI inside EnrichCIStatus, which the eval cycle runs for actionable/held PRs only, every eval_interval_webhook_s (900 s). Three gaps:

  1. Head moved, evidence didn't. After a pull_request synchronize the webhook re-read patches HeadSHA, but CIActivity still describes the old head, so stage_tracker_build.go discards it (!EqualFold(CIActivity.Head, Head)) β†’ ?/? until the next 15-minute cycle.
  2. Sidelined PRs (RepoWorkPR: draft / filtered / conflicts) carried no HeadSHA and were never enriched β†’ always ?/? (e.g. πŸ› fix: bound cluster health collection within heartbeat budgetΒ #16506).
  3. Check webhooks carried no check data forward. check_run/check_suite/status deliveries only invalidated caches and queued a GET /pulls/{n} (1 544 req/h on hive's spoke, the top consumer in /api/gh-rate-limits) β€” a PR object has no check info, so the chevron never moved from a delivery.

Design (operator direction: "use webhooks to update the CI counts")

  • New pkg/github/webhook_ci_evidence.go: a process-wide per-(repo, head) CI evidence store. check_run, workflow_job and status payloads are parsed into PRActivityCheck (name, status, conclusion, node id, App, since) and upserted in place β€” same run id updates, newer run of the same check replaces, older observation is dropped. No GitHub request.
  • EnrichCIStatus records its listing into the store; the tracker reads github.HeadCIActivity(repo, head, base) which overlays newer webhook events on the newest listing (source: github-check-runs+github-webhook).
  • When a check delivery resolves to PR(s) and no listing exists for the head yet (first delivery after a push, sidelined PR), one coalesced check-run listing of that head is scheduled (debounce webhook_refresh_debounce_ms, hive:webhook_ci_refetch, budget/caution guarded), attaching the required-check set so pass/total is right from the first row.
  • Payload-served deliveries (ci_evidence.applied > 0) skip the PR re-read; the tracker cache is dropped and the stage-tracker SSE nudge is sent via a new RefreshCIEvidence hook, so the chevron updates within seconds.
  • Headβ†’PR memory in the store, since a check delivery invalidates the PR detail cache and later status/workflow_job payloads name no PR.
  • Sidelined RepoWorkPR rows now carry HeadSHA.
  • Poll path (EnrichCIStatus each cycle) is unchanged as the backstop for spokes without webhooks.
  • /api/status.webhooks gains last_ci_event_at, ci_events_1h, ci_events_applied_1h, ci_refetches_1h.
  • Hub relay allowlist adds workflow_run, workflow_job; docs updated.

Webhook delivery today (hive's own spoke)

  • Hub relays check_suite, check_run, status (pkg/hub/webhook.go); the spoke received ~5 100 events/h, consistent with check fan-out. Until this PR there was no per-type counter β€” after deploy ci_events_1h answers directly.
  • Operator action (optional): workflow_run/workflow_job were not relayed and are not in the documented App subscription. If job-level payloads are wanted, add the Workflow jobs (and Workflow runs) events to the GitHub App (requires Actions: read). check_run/check_suite (Checks: read) and status are already subscribed and sufficient for the counts.

Verification

  • cd src && go test ./pkg/dashboard/ ./pkg/github/ ./pkg/hub/, go vet, gofmt -l, go build ./... β€” green.
  • New tests (fixtures, no sleeps): TestWebhookCI* / TestHeadCIActivity* (payload β†’ counts, upsert, one coalesced re-read), TestStageTrackerCICountsFromWebhookEvidence (stale-head and sidelined PR rows count), TestGitHubWebhookCheckRunAppliesCIEvidenceWithoutGitHubCall (signed check_run β†’ ci_evidence.applied=1, no PR re-read, no GitHub call, status health fields).
  • After deploy: /api/status?fields=webhooks β†’ ci_events_1h climbing, ci_refetches_1h β‰ͺ ci_events_1h; /api/stage-tracker rows with source containing github-webhook; dashboard:webhook-tracker-refresh GET /pulls/{n} falls off the top of /api/gh-rate-limits.

check_run / workflow_job / status deliveries are applied straight into a
per-(repo, head) check set the tracker's CI chevron counts from; a single
coalesced check-run listing runs only when the head has no listing yet.
Payload-served check deliveries no longer queue a PR re-read. Sidelined
PRs carry HeadSHA so their rows count too. /api/status.webhooks reports
last_ci_event_at, ci_events_1h, ci_events_applied_1h, ci_refetches_1h.
The hub relays workflow_run and workflow_job.

Fixes #16521

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 4, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16514 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/cmd/hive/main.go
#16513 ✨ feat(dashboard): feed published checkpoint diff regions to Code activity same files src/cmd/hive/main.go
#16500 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/cmd/hive/main.go
#16456 ✨ feat(github): enforce pin-only exception in merge-request relay same files src/pkg/github/client.go

@hivecommons-hive

hivecommons-hive Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • relative links and anchors resolve

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the new store, the webhook path end-to-end (HandleWebhookEvent β†’ applyWebhookCIEvidence β†’ ciRefetcher β†’ refetchHeadCI) and the tracker consumer at stage_tracker_activity.go. The evidence store, upsert semantics, listing/event precedence and the workflow_job relay all look correct to me. Two verified gaps between what the body/docs say and what the code does β€” one of them defeats the stated purpose of the coalesced re-read.

1. The re-read never learns the base branch, so the required-check set is never attached (medium).

HandleWebhookEvent drops the PR detail-cache entries for every alias before it calls the CI step:

  • src/pkg/github/webhook_cache.go:662 β€” c.invalidateWebhookPR(aliases, n) β†’ sharedPRDetailCache.invalidate(repo, number) for each alias (webhook_cache.go:890-896)
  • src/pkg/github/webhook_cache.go:666 β€” res.CIEvidence = c.applyWebhookCIEvidence(event, p, configured, numbers)

Then applyWebhookCIEvidence tries to read the base ref back out of that same cache, on the same aliases (webhookRepoAliases(configured, repo) is the same call that produced aliases at webhook_cache.go:591):

  • src/pkg/github/webhook_ci_evidence.go:495-496 β€” sharedPRDetailCache.peek(alias, numbers[0]) … target.baseRef = pr.GetBase().GetRef()

The entry was just deleted, so peek misses on every delivery and target.baseRef is always "". refetchHeadCI then skips the required-check resolution entirely:

  • src/pkg/github/webhook_ci_evidence.go:689 β€” if t.baseRef != "" { … attachRequiredChecks(ctx, pr) }

Consequence: the listing recorded by the re-read has no Required/RequiredChecks, so stageTrackerCIActivity falls through to stageTrackerCountHeadChecks (src/pkg/dashboard/stage_tracker_activity.go:70-71) and the row counts every check run on the head rather than the required set, until the next full EnrichCIStatus cycle. For a sidelined RepoWorkPR that cycle never comes, so those rows count all checks permanently. The body's "attaching the required-check set so pass/total is right from the first row" is not what ships, and the re-read still spends its REST listing. The existing webhookCIFailureResult already handles this ordering correctly by peeking before the invalidation loop (webhook_cache.go:898-900); the simplest fix is to capture the base ref there, or to carry it in ciRefetchTarget from the payload (check_run.pull_requests[].base.ref is present on check_run deliveries). None of the tests seed the detail cache, so this path is currently unexercised β€” TestWebhookCIRefetchListsHeadOnceAndRecordsListing enqueues with no baseRef (webhook_ci_evidence_test.go:293).

2. Docs and body say the re-read is debounced by webhook_refresh_debounce_ms; it is a fixed 2 s (low).

  • src/docs/operator-reference/runtime-and-quota.md:333 β€” "debounced by the same webhook_refresh_debounce_ms"
  • src/pkg/github/webhook_ci_evidence.go:1219 area: const defaultCIRefetchDebounce = 2 * time.Second, applied at ciRefetcher() when r.debounce <= 0; nothing in the diff (or in cmd/hive/main.go) sets ciRefetch.debounce from config.go:5039. An operator raising webhook_refresh_debounce_ms to tame a fan-out will see no effect on this listing. Either wire it or change the doc sentence.

Neither is a reason not to land the webhook-applied path, which is the bulk of the value here and reads correctly. Flagging both so #1 doesn't get discovered as "sidelined rows show 38/40" later.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read webhook_ci_evidence.go and the callers at head 24b8c71; this looks correct to me. What I checked:

  • Store is bounded and locked: ciEvidenceStore.mu guards every map access; pruneLocked (:144-160) drops heads idle β‰₯48h and evicts oldest past ciEvidenceMaxHeads on every write. merged (:260) copies events out under the lock before building the result, so the tracker's read can't race a delivery.
  • No double counting: sameCheckIdentity (:358) keys on (Kind, Name, AppID). Webhook-built checks use Kind:"" for check_run/workflow_job and "commit_status" for status (:388-440), exactly what the REST listing assigns in pr_ci_activity.go:129, so a delivered check replaces its listed counterpart rather than appending. workflow_job is stamped with the Actions App ID (:410), matching the check_run App for the same job.
  • Ordering: checkBefore (:352) uses completed_at for completed and started_at otherwise, so a late-arriving created delivery for a run already seen completed is dropped (completed β‰₯ started). recordListed (:165) clears events not newer than the listing, and merged returns the plain listing when eventsAt ≀ ObservedAt β€” a listing after a burst of deliveries wins.
  • "Refetch only when the head has none yet" holds: the eval cycle's own listing feeds recordListed (client.go:2570) before merged, so hasListed is true for every head the cycle covered; the refetcher coalesces per head with a 2s debounce and bails on APIBudgetCritical / rate-limit caution (:659-665). A failed refetch just drops the target β€” the next delivery re-enqueues it β€” so the row degrades to event-only counts instead of ?/?.
  • go build of pkg/github, pkg/dashboard, pkg/hub, cmd/hive passes; gofmt clean. Body and changelog match the diff (new /api/status.webhooks fields and hub workflow_run/workflow_job relay are both present).

One operator-facing note, not a defect: the store is in-memory, so after a hub restart every active head's first delivery schedules one coalesced listing until the next eval cycle backfills recordListed. That's bounded by the guards above; worth a line in runtime-and-quota.md if restarts on busy repos are common.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 718d719 into v5 Nov 4, 2026
78 of 80 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/webhook-ci-counts branch November 4, 2026 03:28
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› Activity Monitor CI chevron stays at ?/? because check webhooks never update CI evidence

1 participant