Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-webhook-ci-counts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Activity Monitor CI chevron counts (`CI 7/11`) are now driven by `check_run`/`workflow_job`/`status` webhook deliveries applied straight into a per-head check set — no PR re-read per check fan-out — with one coalesced check-run listing only when the head has none yet, so rows no longer sit at `CI ?/?` after a push or for sidelined PRs until the next eval cycle; `/api/status.webhooks` gains `last_ci_event_at`, `ci_events_1h`, `ci_events_applied_1h`, `ci_refetches_1h` and the hub relays `workflow_run`/`workflow_job` (hivecommons/hive#16521)
4 changes: 4 additions & 0 deletions src/cmd/hive/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -3440,6 +3440,10 @@ func (b *boot) bootDashboardAPIWith(deps bootDashboardAPIDeps) {
// re-reads the issue into the cached snapshot and the Activity Monitor
// the way an inbound label webhook does (hivecommons/hive#14952).
github.SetItemRefreshHook(b.dashSrv.RefreshSelfEditedItem)
// A check/status delivery applied to a head's check set, or the
// coalesced re-read it scheduled, re-derives the Activity Monitor row
// at once (hivecommons/hive#16521).
github.SetCIEvidenceHook(b.dashSrv.RefreshCIEvidence)
wireSpektacularRunner(b.cfg, b.dashSrv, b.logger)
// #8380: chain GitHub comments/labels behind the relay yank on takeover.
ghClientFn := func() *github.Client { return b.ghClient }
Expand Down
29 changes: 24 additions & 5 deletions src/docs/operator-reference/runtime-and-quota.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,9 @@ manage are ignored.
webhook with the same secret works too). Subscribe the App to these events:
`pull_request`, `pull_request_review`, `pull_request_review_comment`,
`issue_comment`, `issues`, `check_suite`, `check_run`, `status`, `push`,
and, with `governor.work_source.type: github_projects`, `projects_v2_item`
optionally `workflow_job` (Actions: read; lets the CI counts on the
activity monitor move from the job payload for Apps that do not surface
check runs), and, with `governor.work_source.type: github_projects`, `projects_v2_item`
(organization-scoped; see [Work sources](../work-sources.md#type-github_projects--github-projects-projects-v2)).
3. **Hub relay.** The hub verifies the HMAC and relays the body byte-for-byte,
with the original signature and `X-GitHub-Delivery`, to the spoke whose
Expand All @@ -171,8 +173,8 @@ v5 and v6 spokes) tracking one repo all stay on webhook mode.
#### What a delivery does

- `pull_request`, `pull_request_review`, `pull_request_review_comment`,
`check_suite`, `check_run`, `status`, `issue_comment` (on PRs only) and
`push` invalidate the cached PR detail and the GraphQL batch
`check_suite`, `check_run`, `workflow_job`, `status`, `issue_comment` (on
PRs only) and `push` invalidate the cached PR detail and the GraphQL batch
check-run/review entries for the affected PR and mark it dirty. Check and
status events are matched by PR number and head SHA (fork PRs arrive
without a PR list); `push` invalidates every open PR whose base or head
Expand Down Expand Up @@ -288,8 +290,7 @@ v5 and v6 spokes) tracking one repo all stay on webhook mode.
`issues` — any action) also drops the `/api/stage-tracker` cache at once
and queues a **targeted re-read of just that item**: one `GET` of the PR
or issue, debounced by `dashboard.stage_tracker.webhook_refresh_debounce_ms`
(default `2000`) and coalesced per item, so a check fan-out of twenty
`check_run` deliveries costs one request. The re-read patches that row of
(default `2000`) and coalesced per item. The re-read patches that row of
the status snapshot in place — labels, title, draft flag, last activity;
a closed or merged item leaves — then drops the tracker cache again and
sends the `stage-tracker` SSE nudge. The same re-read is handed to the
Expand Down Expand Up @@ -318,6 +319,24 @@ v5 and v6 spokes) tracking one repo all stay on webhook mode.
also lists each repo's items closed since its last look every
`dashboard.stage_tracker.closed_sweep_interval_s` (default `300`; one
list call per repo; negative disables) and evicts any still shown.
- **CI counts on the activity monitor move from the check payload.** A
`check_run`, `workflow_job` or `status` delivery is applied directly into
the per-head check set the tracker's CI chevron counts (`CI 7/11`): the
named check is upserted by run id (or check name and producing App), a
newer run replaces an older one, and the row is pushed over the
`stage-tracker` SSE nudge without any GitHub request — a fan-out of
twenty `check_run` deliveries costs zero requests. The payload-only
`check_run`/`status` delivery no longer queues the PR re-read above.
When no check-run listing exists yet for that head (first delivery after
a push, or a sidelined PR the eval cycle never enriched) one coalesced
listing of the head's check runs is scheduled instead (debounced by the
same `webhook_refresh_debounce_ms`), so the required-check set and the
`pass/total` denominator are right from the first row. The polling
path's `EnrichCIStatus` still refreshes every actionable PR each cycle
and is the backstop for spokes without webhooks.
`/api/status.webhooks.last_ci_event_at`, `ci_events_1h`,
`ci_events_applied_1h` and `ci_refetches_1h` show whether check
deliveries actually arrive and how many were served from the payload.

#### Polling cadence vs webhook patch vs reconcile

Expand Down
2 changes: 1 addition & 1 deletion src/docs/scale-envelope.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ones to:
2. **Truncation must preserve the prioritization signal, not cut an arbitrary slice.**
This holds today and is easy to assume backwards:
- Issues are sorted by actionable rank tier, then **oldest first** within each tier before any cap is applied
(`RankActionableIssues`, `src/pkg/github/client.go:5110-5120`, descending `AgeMinutes` within each tier).
(`RankActionableIssues`, `src/pkg/github/client.go:5122-5134`, descending `AgeMinutes` within each tier).
- PRs are sorted by **review class then oldest-first within class**
(`SortPullRequestsForReview`, `src/pkg/github/review_priority.go:334`;
fixes → refactors/docs → tests).
Expand Down
3 changes: 3 additions & 0 deletions src/pkg/dashboard/api_github_webhook.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,9 @@ func (s *Server) handleGitHubWebhook(w http.ResponseWriter, r *http.Request) {
if reactions := s.reactToCIFailure(res); len(reactions) > 0 {
resp["ci_failure"] = reactions
}
if res.CIEvidence != nil {
resp["ci_evidence"] = res.CIEvidence
}
if n, ok := sentinelPREventTarget(event, body); ok && s.deps.SentinelPREvent != nil {
s.deps.SentinelPREvent(res.ConfiguredRepo, n)
resp["sentinel"] = n
Expand Down
6 changes: 5 additions & 1 deletion src/pkg/dashboard/stage_tracker_build.go
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,10 @@ func (s *Server) buildStageTracker(status *StatusPayload, store *stagelabels.Sto
stageTrackerApplyFixerStatus(&it, review.FixDispatchStatusFor(dispatchState, qualifyRepo(repoName), pr.Number, pr.HeadSHA), workers)
activityPR := pr
activityPR.Repo = qualifyRepo(repoName)
// Checks delivered by webhook since the last listing, laid over it;
// a head the cycle never listed counts from the deliveries alone
// (hivecommons/hive#16521).
activityPR.CIActivity = github.HeadCIActivity(activityPR.Repo, pr.HeadSHA, pr.CIActivity)
it.CIActivity = stageTrackerCIActivity(activityPR, now)
var bundle *evidence.Bundle
if evidenceVerifier != nil && pr.HeadSHA != "" {
Expand Down Expand Up @@ -501,7 +505,7 @@ func (s *Server) buildStageTracker(status *StatusPayload, store *stagelabels.Sto
default:
w := it.WorkPR
addPR(repoName, github.PullRequest{Repo: w.Repo, Number: w.Number, Title: w.Title, Author: w.Author, Labels: w.Labels, Draft: w.Draft,
AppAuthored: w.AppAuthored, HiveAttributed: w.HiveAttributed, CreatedAt: w.CreatedAt, UpdatedAt: w.UpdatedAt, URL: w.URL, BaseRef: w.BaseRef, LinkedIssues: w.LinkedIssues, ReferencedIssues: w.ReferencedIssues}, nil, w.Bucket, w.Reason)
AppAuthored: w.AppAuthored, HiveAttributed: w.HiveAttributed, CreatedAt: w.CreatedAt, UpdatedAt: w.UpdatedAt, URL: w.URL, BaseRef: w.BaseRef, HeadSHA: w.HeadSHA, LinkedIssues: w.LinkedIssues, ReferencedIssues: w.ReferencedIssues}, nil, w.Bucket, w.Reason)
}
}
}
Expand Down
3 changes: 3 additions & 0 deletions src/pkg/dashboard/stage_tracker_status_collection_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ import (
func TestStageTrackerRESTCheckPaginationMustCompleteBeforeProjection(t *testing.T) {
for _, mode := range []string{"complete", "partial", "malformed"} {
t.Run(mode, func(t *testing.T) {
// Each mode observes the same head; the shared per-head CI
// evidence store must not carry one mode's listing into the next.
t.Cleanup(github.ResetCIEvidenceForTest(nil))
reads := 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
Expand Down
145 changes: 145 additions & 0 deletions src/pkg/dashboard/stage_tracker_webhook_ci_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
package dashboard

import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"

"github.com/hivecommons/hive/pkg/github"
)

// TestStageTrackerCICountsFromWebhookEvidence: check deliveries applied to
// the per-head evidence store drive the CI chevron counts for a PR whose
// status snapshot carries no (or stale) CIActivity — including sidelined
// PRs the eval cycle never enriches (hivecommons/hive#16521).
func TestStageTrackerCICountsFromWebhookEvidence(t *testing.T) {
now := time.Date(2026, 10, 17, 9, 0, 0, 0, time.UTC)
t.Cleanup(github.ResetCIEvidenceForTest(func() time.Time { return now }))
s, store := stageTrackerFixture(t, now)

appID := int64(15368)
done := now.Add(-time.Minute)
checks := []github.PRActivityCheck{
{Name: "build", State: "completed", Conclusion: "success", NodeID: "CR_build", AppID: &appID, Since: &done},
{Name: "test", State: "in_progress", NodeID: "CR_test", AppID: &appID, Since: &done},
}
if n := github.RecordCIEvidenceForTest("acme/console", "sha-103", checks); n != 2 {
t.Fatalf("recorded %d checks, want 2", n)
}
if n := github.RecordCIEvidenceForTest("acme/console", "sha-106", checks[:1]); n != 1 {
t.Fatalf("recorded %d checks, want 1", n)
}

s.statusMu.Lock()
repo := &s.status.Repos[0]
for i, raw := range repo.OpenPrs {
pr, ok := raw.(FrontendPR)
if !ok || pr.Number != 103 {
continue
}
pr.HeadSHA = "sha-103"
// Evidence from a previous head must not win over the new head.
stale := now.Add(-time.Hour)
pr.CIActivity = &github.PRCIActivity{Head: "old-103", Source: "github-check-runs", ObservedAt: &stale,
Checks: []github.PRActivityCheck{{Name: "build", State: "completed", Conclusion: "failure"}}}
repo.OpenPrs[i] = pr
}
repo.NonActionablePrs = []any{
github.RepoWorkPR{Repo: "acme/console", Number: 106, Title: "WIP draft", Author: "zoe", Draft: true, Bucket: "draft", Reason: "Draft", HeadSHA: "sha-106", CreatedAt: now.Add(-24 * time.Hour)},
}
s.statusMu.Unlock()

resp := s.buildStageTracker(s.status, store, now)
pr103 := trackerItem(t, resp, "pr", 103)
if pr103.CIActivity == nil || pr103.CIActivity.Head != "sha-103" || pr103.CIActivity.ChecksTotal != 2 || pr103.CIActivity.ChecksPassed != 1 || pr103.CIActivity.ChecksFailed != 0 {
t.Fatalf("pr103 CI = %+v, want 1/2 on sha-103 from webhook evidence", pr103.CIActivity)
}
pr106 := trackerItem(t, resp, "pr", 106)
if pr106.CIActivity == nil || pr106.CIActivity.ChecksTotal != 1 || pr106.CIActivity.ChecksPassed != 1 {
t.Fatalf("sidelined pr106 CI = %+v, want 1/1 from webhook evidence", pr106.CIActivity)
}
}

// TestGitHubWebhookCheckRunAppliesCIEvidenceWithoutGitHubCall: a
// `check_run` delivery that names its PR updates the CI evidence from the
// payload alone — no PR re-read is queued, the GitHub test server would
// fail the test on any call — and /api/status.webhooks reports CI event
// health.
func TestGitHubWebhookCheckRunAppliesCIEvidenceWithoutGitHubCall(t *testing.T) {
const secret = "s3cr3t"
t.Setenv(githubWebhookSecretEnvVar, secret)
now := time.Date(2026, 10, 17, 9, 0, 0, 0, time.UTC)
t.Cleanup(github.ResetWebhookTrackerForTestWithClock(func() time.Time { return now }))
t.Cleanup(github.ResetCIEvidenceForTest(func() time.Time { return now }))
s := newGitHubWebhookTestServer(t)

// A prior listing exists for the head, so the delivery needs no re-read.
appID := int64(15368)
started := now.Add(-2 * time.Minute)
listedAt := now.Add(-time.Minute)
github.RecordCIListingForTest("webhookorg/webhookrepo", "abc123", &github.PRCIActivity{
Head: "abc123", Source: "github-check-runs", ObservedAt: &listedAt,
Checks: []github.PRActivityCheck{{Name: "unit", State: "in_progress", NodeID: "CR_unit", AppID: &appID, Since: &started}},
})

body := []byte(`{"action":"completed","repository":{"full_name":"webhookorg/webhookrepo"},
"check_run":{"id":1,"node_id":"CR_unit","name":"unit","status":"completed","conclusion":"success","head_sha":"abc123",
"started_at":"2026-10-17T08:58:00Z","completed_at":"2026-10-17T08:59:30Z","app":{"id":15368,"name":"GitHub Actions"},
"pull_requests":[{"number":5}]}}`)
rec := postGitHubWebhook(s, "check_run", signGitHubWebhookForTest(secret, body), body)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d (%s)", rec.Code, rec.Body.String())
}
var got struct {
Status string `json:"status"`
TrackerRefresh *int `json:"tracker_refresh"`
CIEvidence *struct {
HeadSHA string `json:"head_sha"`
Applied int `json:"applied"`
Refetch bool `json:"refetch"`
PRs []int `json:"prs"`
} `json:"ci_evidence"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
if got.Status != "ok" || got.CIEvidence == nil || got.CIEvidence.Applied != 1 || got.CIEvidence.Refetch || got.CIEvidence.HeadSHA != "abc123" || len(got.CIEvidence.PRs) != 1 || got.CIEvidence.PRs[0] != 5 {
t.Fatalf("response = %s, want ci_evidence applied 1 on abc123 for PR 5 with no refetch", rec.Body.String())
}
if got.TrackerRefresh != nil {
t.Fatalf("payload-served check delivery must not queue a PR re-read: %s", rec.Body.String())
}
completed := now.Add(-30 * time.Second)
merged := github.HeadCIActivity("webhookorg/webhookrepo", "abc123", nil)
if merged == nil || len(merged.Checks) != 1 || merged.Checks[0].Conclusion != "success" || merged.Checks[0].Since == nil || !merged.Checks[0].Since.Equal(completed) {
t.Fatalf("head evidence = %+v, want the completed unit run", merged)
}
pr := github.PullRequest{Number: 5, HeadSHA: "abc123"}
if ci := stageTrackerCIActivity(github.PullRequest{Number: 5, HeadSHA: "abc123", CIActivity: merged}, now); ci.ChecksTotal != 1 || ci.ChecksPassed != 1 {
t.Fatalf("tracker counts for %d = %+v, want 1/1", pr.Number, ci)
}

s.statusMu.Lock()
s.status = &StatusPayload{HiveID: "webhook-test"}
s.statusMu.Unlock()
statusRec := httptest.NewRecorder()
s.handleStatus(statusRec, httptest.NewRequest(http.MethodGet, "/api/status?fields=webhooks", bytes.NewReader(nil)))
var status struct {
Webhooks struct {
LastCIEventAt *time.Time `json:"last_ci_event_at"`
CIEvents1h int `json:"ci_events_1h"`
CIEventsApplied1h int `json:"ci_events_applied_1h"`
CIRefetches1h int `json:"ci_refetches_1h"`
} `json:"webhooks"`
}
if err := json.Unmarshal(statusRec.Body.Bytes(), &status); err != nil {
t.Fatal(err)
}
w := status.Webhooks
if w.LastCIEventAt == nil || !w.LastCIEventAt.Equal(now) || w.CIEvents1h != 1 || w.CIEventsApplied1h != 1 || w.CIRefetches1h != 0 {
t.Fatalf("/api/status webhooks CI health = %+v, want one event applied at %s", w, now)
}
}
21 changes: 21 additions & 0 deletions src/pkg/dashboard/webhook_tracker_refresh.go
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,13 @@ func (s *Server) scheduleWebhookTrackerRefresh(res github.WebhookEventResult, de
if !s.webhookTrackerRefreshEnabled() {
return 0
}
if res.CIEvidence != nil && res.CIEvidence.Applied > 0 {
// The payload itself moved the row's CI evidence
// (hivecommons/hive#16521); the PR object did not change, so the
// re-read would only spend a request to confirm labels and title.
s.invalidateStageTrackerForWebhook()
return 0
}
keys := webhookTrackerKeys(res)
if len(keys) == 0 {
return 0
Expand All @@ -261,6 +268,17 @@ func (s *Server) scheduleWebhookTrackerRefresh(res github.WebhookEventResult, de
return len(keys)
}

// RefreshCIEvidence is the github.CIEvidenceHook: a check delivery or a
// coalesced check-run re-read changed the CI evidence for repo's head, so
// the tracker payload is dropped and the page nudged over SSE; the next GET
// re-derives the row's counts from the merged evidence.
func (s *Server) RefreshCIEvidence(repo, head string) {
if s == nil {
return
}
s.invalidateStageTrackerForWebhook()
}

// selfEditDeliveryID tags the re-reads the hive's own GitHub edits ask for,
// in place of a webhook delivery id, in ItemPatches and the logs.
const selfEditDeliveryID = "self:edit"
Expand Down Expand Up @@ -804,6 +822,9 @@ func patchWorkPRAny(raw any, snap github.WebhookItemSnapshot) (any, bool) {
w.Title = snap.Title
w.Labels = append([]string(nil), snap.Labels...)
w.Draft = snap.Draft
if snap.HeadSHA != "" {
w.HeadSHA = snap.HeadSHA
}
if !snap.UpdatedAt.IsZero() {
w.UpdatedAt = snap.UpdatedAt
}
Expand Down
9 changes: 1 addition & 8 deletions src/pkg/github/ci_failure_signal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,7 @@ func TestWebhookCIFailure_EventsAndConclusions(t *testing.T) {
}
}

type checkRunPayload = struct {
HeadSHA string `json:"head_sha"`
Name string `json:"name"`
Conclusion string `json:"conclusion"`
DetailsURL string `json:"details_url"`
HTMLURL string `json:"html_url"`
PullRequests []webhookPRRef `json:"pull_requests"`
}
type checkRunPayload = webhookCheckRunPayload

type checkSuitePayload = struct {
HeadSHA string `json:"head_sha"`
Expand Down
14 changes: 14 additions & 0 deletions src/pkg/github/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ type Client struct {
graphQLPRBatchEnabled func() bool
graphQLPRBatchPageSize func() int
prBatchMu sync.Mutex
// ciRefetch coalesces the check-run re-reads CI webhook deliveries ask
// for on heads no listing has covered (webhook_ci_evidence.go).
ciRefetchOnce sync.Once
ciRefetch *ciRefetcher
prBatchCheckRuns map[prBatchCheckRunKey][]*gh.CheckRun
prBatchCheckObservedAt map[prBatchCheckRunKey]time.Time
prBatchReviews map[string]map[int]prReviewState
Expand Down Expand Up @@ -2560,6 +2564,11 @@ func (c *Client) EnrichCIStatus(ctx context.Context, prs []PullRequest) {
prs[i].SharedCIIncident = c.openSharedCIIncident(ctx, &prs[i], incidents)
}
facts.attach(ctx, &prs[i], reported)
// The listing is the head's complete observation; checks delivered
// by webhook since then are laid over it (hivecommons/hive#16521).
owner, name := c.splitRepo(prs[i].Repo)
sharedCIEvidence().recordListed(owner+"/"+name, prs[i].HeadSHA, prs[i].CIActivity)
prs[i].CIActivity = sharedCIEvidence().merged(owner+"/"+name, prs[i].HeadSHA, prs[i].CIActivity)
}
c.enrichBaseBranchRed(ctx, prs)
}
Expand Down Expand Up @@ -4843,6 +4852,10 @@ type RepoWorkPR struct {
Reason string `json:"reason,omitempty"`
// BaseRef is the target branch, retained for the covering-PR chip.
BaseRef string `json:"base_ref,omitempty"`
// HeadSHA is the PR's head commit from the list payload, so the activity
// monitor can match webhook-delivered CI evidence to a sidelined row
// (hivecommons/hive#16521).
HeadSHA string `json:"head_sha,omitempty"`
// LinkedIssues mirrors PullRequest.LinkedIssues so a sidelined PR can
// still stand under the issue its `Fixes #N` names in the activity
// monitor. Stamped by EnrichReviewSignalsAndWorkPRs, not by the
Expand Down Expand Up @@ -4874,6 +4887,7 @@ func (c *Client) repoWorkPR(repo string, pr *gh.PullRequest, labels []string, hi
UpdatedAt: pr.GetUpdatedAt().Time,
URL: pr.GetHTMLURL(),
BaseRef: prBaseRef(pr),
HeadSHA: prHeadSHA(pr),
Bucket: bucket,
Reason: reason,
// The list payload carries the body, so the soft issue links cost
Expand Down
Loading
Loading