Repository navigation
π fix: drive Activity Monitor CI counts from check webhooks - #16528
Conversation
check_run / workflow_job / status deliveries are applied straight into a per-(repo, head) check set the tracker's CI chevron counts from; a single coalesced check-run listing runs only when the head has no listing yet. Payload-served check deliveries no longer queue a PR re-read. Sidelined PRs carry HeadSHA so their rows count too. /api/status.webhooks reports last_ci_event_at, ci_events_1h, ci_events_applied_1h, ci_refetches_1h. The hub relays workflow_run and workflow_job. Fixes #16521 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. Review findings from Read the new store, the webhook path end-to-end ( 1. The re-read never learns the base branch, so the required-check set is never attached (medium).
Then
The entry was just deleted, so
Consequence: the listing recorded by the re-read has no 2. Docs and body say the re-read is debounced by
Neither is a reason not to land the webhook-applied path, which is the bulk of the value here and reads correctly. Flagging both so #1 doesn't get discovered as "sidelined rows show 38/40" later. |
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Read webhook_ci_evidence.go and the callers at head 24b8c71; this looks correct to me. What I checked:
- Store is bounded and locked:
ciEvidenceStore.muguards every map access;pruneLocked(:144-160) drops heads idle β₯48h and evicts oldest pastciEvidenceMaxHeadson every write.merged(:260) copieseventsout under the lock before building the result, so the tracker's read can't race a delivery. - No double counting:
sameCheckIdentity(:358) keys on(Kind, Name, AppID). Webhook-built checks useKind:""forcheck_run/workflow_joband"commit_status"forstatus(:388-440), exactly what the REST listing assigns inpr_ci_activity.go:129, so a delivered check replaces its listed counterpart rather than appending.workflow_jobis stamped with the Actions App ID (:410), matching thecheck_runApp for the same job. - Ordering:
checkBefore(:352) usescompleted_atfor completed andstarted_atotherwise, so a late-arrivingcreateddelivery for a run already seencompletedis dropped (completed β₯ started).recordListed(:165) clears events not newer than the listing, andmergedreturns the plain listing wheneventsAt β€ ObservedAtβ a listing after a burst of deliveries wins. - "Refetch only when the head has none yet" holds: the eval cycle's own listing feeds
recordListed(client.go:2570) beforemerged, sohasListedis true for every head the cycle covered; the refetcher coalesces per head with a 2s debounce and bails onAPIBudgetCritical/ rate-limit caution (:659-665). A failed refetch just drops the target β the next delivery re-enqueues it β so the row degrades to event-only counts instead of?/?. go buildofpkg/github,pkg/dashboard,pkg/hub,cmd/hivepasses; gofmt clean. Body and changelog match the diff (new/api/status.webhooksfields and hubworkflow_run/workflow_jobrelay are both present).
One operator-facing note, not a defect: the store is in-memory, so after a hub restart every active head's first delivery schedules one coalesced listing until the next eval cycle backfills recordListed. That's bounded by the guards above; worth a line in runtime-and-quota.md if restarts on busy repos are common.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #16521
Why rows read
CI ?/? Β· no checks reported yetchecks_passed/checks_total(PR #16494) are computed frompr.CIActivity, whose only producer wasenrichPRCIinsideEnrichCIStatus, which the eval cycle runs for actionable/held PRs only, everyeval_interval_webhook_s(900 s). Three gaps:pull_request synchronizethe webhook re-read patchesHeadSHA, butCIActivitystill describes the old head, sostage_tracker_build.godiscards it (!EqualFold(CIActivity.Head, Head)) β?/?until the next 15-minute cycle.RepoWorkPR: draft / filtered / conflicts) carried noHeadSHAand were never enriched β always?/?(e.g. π fix: bound cluster health collection within heartbeat budgetΒ #16506).check_run/check_suite/statusdeliveries only invalidated caches and queued aGET /pulls/{n}(1 544 req/h on hive's spoke, the top consumer in/api/gh-rate-limits) β a PR object has no check info, so the chevron never moved from a delivery.Design (operator direction: "use webhooks to update the CI counts")
pkg/github/webhook_ci_evidence.go: a process-wide per-(repo, head)CI evidence store.check_run,workflow_jobandstatuspayloads are parsed intoPRActivityCheck(name, status, conclusion, node id, App, since) and upserted in place β same run id updates, newer run of the same check replaces, older observation is dropped. No GitHub request.EnrichCIStatusrecords its listing into the store; the tracker readsgithub.HeadCIActivity(repo, head, base)which overlays newer webhook events on the newest listing (source: github-check-runs+github-webhook).webhook_refresh_debounce_ms,hive:webhook_ci_refetch, budget/caution guarded), attaching the required-check set sopass/totalis right from the first row.ci_evidence.applied > 0) skip the PR re-read; the tracker cache is dropped and thestage-trackerSSE nudge is sent via a newRefreshCIEvidencehook, so the chevron updates within seconds.status/workflow_jobpayloads name no PR.RepoWorkPRrows now carryHeadSHA.EnrichCIStatuseach cycle) is unchanged as the backstop for spokes without webhooks./api/status.webhooksgainslast_ci_event_at,ci_events_1h,ci_events_applied_1h,ci_refetches_1h.workflow_run,workflow_job; docs updated.Webhook delivery today (hive's own spoke)
check_suite,check_run,status(pkg/hub/webhook.go); the spoke received ~5 100 events/h, consistent with check fan-out. Until this PR there was no per-type counter β after deployci_events_1hanswers directly.workflow_run/workflow_jobwere not relayed and are not in the documented App subscription. If job-level payloads are wanted, add the Workflow jobs (and Workflow runs) events to the GitHub App (requires Actions: read).check_run/check_suite(Checks: read) andstatusare already subscribed and sufficient for the counts.Verification
cd src && go test ./pkg/dashboard/ ./pkg/github/ ./pkg/hub/,go vet,gofmt -l,go build ./...β green.TestWebhookCI*/TestHeadCIActivity*(payload β counts, upsert, one coalesced re-read),TestStageTrackerCICountsFromWebhookEvidence(stale-head and sidelined PR rows count),TestGitHubWebhookCheckRunAppliesCIEvidenceWithoutGitHubCall(signedcheck_runβci_evidence.applied=1, no PR re-read, no GitHub call, status health fields)./api/status?fields=webhooksβci_events_1hclimbing,ci_refetches_1hβͺci_events_1h;/api/stage-trackerrows withsourcecontaininggithub-webhook;dashboard:webhook-tracker-refresh GET /pulls/{n}falls off the top of/api/gh-rate-limits.