From 7d38cf19ff35f68678e9ebdee2180c7ddf5b595a Mon Sep 17 00:00:00 2001 From: Andrew Anderson Date: Mon, 9 Nov 2026 20:23:08 -0700 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20fix(dashboard):=20CI=20chevron?= =?UTF-8?q?=20no=20longer=20stays=20lit=20after=20every=20check=20has=20co?= =?UTF-8?q?ncluded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A required check with no run on the head is a settled word, never inferred motion from a part-filled tally; at 060 reviewed with every reported check concluded, Review is the lit chevron and CI is its blocker only when red. A CI snapshot that still claims a check running or queued after dashboard.stage_tracker.ci_activity_stale_m (default 10) is re-listed from GitHub on the next tracker build, bypassing the batch cache, ETag and stage, coalesced per head. Fixes hivecommons/hive#17508 Signed-off-by: Andrew Anderson --- ...fixed-ci-chevron-stale-after-completion.md | 1 + .../dashboard-sections/activity-monitor.md | 1 + .../operator-reference/runtime-and-quota.md | 1 + src/pkg/config/config.go | 23 +++++ src/pkg/dashboard/stage_tracker_activity.go | 12 +++ .../dashboard/stage_tracker_activity_test.go | 95 +++++++++++++++++++ src/pkg/dashboard/stage_tracker_build.go | 21 ++++ .../stage_tracker_remediation_chevron.test.js | 10 +- .../stage_tracker_stale_ci_lead_test.go | 49 ++++++++++ .../static-src/js/tracker-row-strip.js | 9 +- src/pkg/dashboard/static/index.html | 9 +- src/pkg/github/ci_activity_stale.go | 81 ++++++++++++++++ src/pkg/github/webhook_ci_evidence.go | 3 + src/pkg/github/webhook_ci_evidence_test.go | 36 +++++++ 14 files changed, 348 insertions(+), 3 deletions(-) create mode 100644 changelog.d/fixed-ci-chevron-stale-after-completion.md create mode 100644 src/pkg/dashboard/stage_tracker_stale_ci_lead_test.go create mode 100644 src/pkg/github/ci_activity_stale.go diff --git a/changelog.d/fixed-ci-chevron-stale-after-completion.md b/changelog.d/fixed-ci-chevron-stale-after-completion.md new file mode 100644 index 0000000000..2b38634152 --- /dev/null +++ b/changelog.d/fixed-ci-chevron-stale-after-completion.md @@ -0,0 +1 @@ +- Dashboard: the Activity Monitor's CI chevron no longer stays lit as running after every check on the head has concluded. A required check that never produced a run is a settled word (*hasn't reported*), never inferred motion from a part-filled tally; at `060 reviewed` with all reported checks concluded, Review is the lit chevron and CI is its blocker only when red. A CI snapshot that still claims a check running or queued after `dashboard.stage_tracker.ci_activity_stale_m` (default 10 minutes) is re-listed from GitHub on the next tracker build regardless of the batch cache, ETag or stage. ([#17508](https://github.com/hivecommons/hive/issues/17508)) diff --git a/src/docs/dashboard-sections/activity-monitor.md b/src/docs/dashboard-sections/activity-monitor.md index 210528fb84..e01feef7a1 100644 --- a/src/docs/dashboard-sections/activity-monitor.md +++ b/src/docs/dashboard-sections/activity-monitor.md @@ -454,6 +454,7 @@ An *active* chevron is filled amber and carries words; an inactive one is its ic 1. **Exactly one lit stage per row.** One chevron is active — the column where the item is or where it is stuck; later stages are shown dim, with a muted queued note when they are next ([#17335](https://github.com/hivecommons/hive/issues/17335): a push during review lit *CI 7/11 · 4 running* and *Review 5/5 · adjudicating* amber together). 2. **CI and Review resolve first.** A track is active only while it genuinely runs on the current head (a check-run in progress or queued, a ring not yet full; a reviewer running or queued). The CI count, its pending names and its motion cover tests only: Prow's `tide` merge-gate status (pending on every PR until it carries lgtm/approved and all presubmits are green), `enforce-guardrails` and the DCO sign-off context (`dco` / `DCO`) are merge gates, not checks, so a PR whose only pending context is `tide` reads *CI 18/18* and lights Review or the Merge queue — never *CI 18/19 · 'tide' running* (`github.IsMergeGateContext`; merge eligibility still enforces the gates). If either runs, the PR is waiting on it: the stage chevron (PR in progress, Merge queue …) renders settled-current — filled but still, icon only, its words in the tooltip — and nothing else is active. Settled-current is always the neutral fill: the Open column's draft, blocked, New and stale tints and the held-gate amber never repaint it, so the running track is the row's only coloured chevron ([#17461](https://github.com/hivecommons/hive/issues/17461): a draft PR with no stage yet read as a slate Open beside amber CI). + A CI word is motion only when GitHub reports it: a check run whose status is not `completed`, or a non-merge-gate commit status still `pending`. A required check with no run on the head is settled (*hasn't reported*), never *running*, however many of its neighbours are green; at `060 reviewed` with every reported check concluded, **Review** is the lit chevron and CI is its blocker only when red. A CI snapshot that still claims a check running or queued after `dashboard.stage_tracker.ci_activity_stale_m` (default 10 minutes) is re-listed from GitHub on the next tracker build regardless of the batch cache, ETag or stage, so a listing taken while checks were starting cannot stay lit for hours ([#17508](https://github.com/hivecommons/hive/issues/17508): *CI 2/3 · running* eight hours after every check had passed). 3. **CI leads Review.** When both tracks would light at once — CI running while the review is queued, in progress or being re-adjudicated — CI is the lit chevron and Review is dim with a small note (*re-review queued* after a push, *review in progress*, *review queued*); its words stay in the tooltip, ending *… behind CI*. The stage chevron is the fallback: active only when no track is in motion. 4. **The PR in progress chevron speaks the PR's lifecycle** — draft, pushed, rebasing, *Unassigned · next: scanner*, *Deferred · waiting on #N*, *Waiting for @author · fix owed* — never a CI or review verdict. *CI failing*, *CI 4/11 · 1 failing* and *changes requested* are the CI and Review chevrons' words; the PR in progress tooltip keeps the red reason as detail. 5. **A human gate is a pill, not a stage.** A PR parked at 085 for a person — a reviewer's question, a permission only a maintainer holds, a rerun to approve — with checks running, red or unreported, or its review verdict pending, is drawn at its real stage: CI (with motion while checks run) or Review is the one lit chevron, wearing an amber marker (`ham-human-gated`), and the ask is the **needs human decision** pill in the row's pill strip (human icon, the question in its hover, the decision popover on click); the lit track's tooltip ends *· needs a person: *. It never lights a *Needs human · decision* chevron in the PR in progress column while CI and Review spin dim ([#17453](https://github.com/hivecommons/hive/issues/17453): #17442 did, with CI 3/9 running; #17416 read *Needs human · review* the same way). Only a merge-ready, queued or merging 085 is a Merge queue matter. diff --git a/src/docs/operator-reference/runtime-and-quota.md b/src/docs/operator-reference/runtime-and-quota.md index 3c67b48d2b..dd3d67c23f 100644 --- a/src/docs/operator-reference/runtime-and-quota.md +++ b/src/docs/operator-reference/runtime-and-quota.md @@ -645,6 +645,7 @@ dashboard: webhook_refresh_replay_window_s: 1800 # re-apply re-reads to snapshots published meanwhile closed_sweep_interval_s: 300 # list items closed since the last look and evict rows still shown; -1 disables waiting_push_stale_h: 4 # hours without a push on a CI / Review row before the pill's age turns muted-amber + ci_activity_stale_m: 10 # minutes a CI snapshot may still claim a check running before the head's check runs are re-listed; -1 disables toast_info_s: 15 # how long the tracker's result / waiting-on-you toasts stay (3-120 s); the merge banner stays twice as long toast_error_s: 30 # how long a failed row action's toast stays (3-120 s); editable from the Activity Monitor's ⚙ → Toasts & timing merged_hold_s: 8 # ceiling on the "Merging · merge landed" bridge; celebration fires on the merged fact (min 3) diff --git a/src/pkg/config/config.go b/src/pkg/config/config.go index 894772dd14..96b71636ef 100644 --- a/src/pkg/config/config.go +++ b/src/pkg/config/config.go @@ -5211,6 +5211,12 @@ type DashboardStageTrackerConfig struct { // age takes the muted-stale tone (hivecommons/hive#15784). 0 means the // default (4); a negative value never marks the wait stale. WaitingPushStaleH int `yaml:"waiting_push_stale_h,omitempty" json:"waiting_push_stale_h,omitempty"` + // CIActivityStaleM is how many minutes a head's CI snapshot may claim a + // check still running or queued before the tracker re-lists the head's + // check runs on its next build, whatever the batch cache or ETag says + // (hivecommons/hive#17508). 0 means the default (10); a negative value + // disables the re-list. + CIActivityStaleM int `yaml:"ci_activity_stale_m,omitempty" json:"ci_activity_stale_m,omitempty"` // ToastInfoS is how many seconds the activity monitor's informational // toasts (action results, waiting-on-human nudges, the merge banner) // stay up before dismissing themselves (hivecommons/hive#15813). @@ -5356,6 +5362,23 @@ func (c *Config) StageTrackerWebhookRefreshDebounce() time.Duration { return defaultStageTrackerWebhookRefreshDebounce } +// defaultStageTrackerCIActivityStale is dashboard.stage_tracker.ci_activity_stale_m's default. +const defaultStageTrackerCIActivityStale = 10 * time.Minute + +// StageTrackerCIActivityStale is the effective +// dashboard.stage_tracker.ci_activity_stale_m as a duration: how old a CI +// snapshot that still claims a check running or queued may be before the +// tracker re-lists the head's check runs. 0 disables the re-list. +func (c *Config) StageTrackerCIActivityStale() time.Duration { + if c == nil || c.Dashboard.StageTracker.CIActivityStaleM == 0 { + return defaultStageTrackerCIActivityStale + } + if c.Dashboard.StageTracker.CIActivityStaleM < 0 { + return 0 + } + return time.Duration(c.Dashboard.StageTracker.CIActivityStaleM) * time.Minute +} + // StageTrackerCacheTTL is the effective dashboard.stage_tracker.cache_ttl_s: // the configured value, else the governor eval interval, else its default. func (c *Config) StageTrackerCacheTTL() time.Duration { diff --git a/src/pkg/dashboard/stage_tracker_activity.go b/src/pkg/dashboard/stage_tracker_activity.go index e392530d55..5bb980b41a 100644 --- a/src/pkg/dashboard/stage_tracker_activity.go +++ b/src/pkg/dashboard/stage_tracker_activity.go @@ -142,6 +142,9 @@ func stageTrackerCIActivity(pr github.PullRequest, now time.Time) *StageTrackerA // Required checks with no run at all, or only a pending commit status: // on a Prow repository, a presubmit that never ran or has not finished. presubmitWaiting := map[string]bool{} + // Required checks with no run on the head while commit statuses were + // not read: unknown rather than missing, with the no-run caption. + noRunUnread := map[string]bool{} for _, observation := range observations { name := observation.name check := github.PRActivityCheck{} @@ -154,8 +157,14 @@ func stageTrackerCIActivity(pr github.PullRequest, now time.Time) *StageTrackerA case len(observation.checks) > 1: state = "unknown" case !ok: + // No run of that name on the head. With commit statuses read it + // is missing; without them a status could still provide it, so + // the word stays unknown — but a settled unknown, never motion + // (hivecommons/hive#17508), with the no-run caption below. if a.StatusesKnown { state = "missing" + } else { + noRunUnread[name] = true } case check.State == "queued" || check.State == "waiting" || check.State == "pending" || check.State == "requested": state = "queued" @@ -304,6 +313,9 @@ func stageTrackerCIActivity(pr github.PullRequest, now time.Time) *StageTrackerA if statusesMissing { out.Caption = "CI unknown · commit statuses not observed for this head" out.Detail = "A required check is reported as a commit status rather than a check run, and hive has not yet fetched commit statuses for the PR's current commit." + } else if noRunUnread[out.Blocker] { + out.Caption = "CI waiting · required check " + check + " hasn't reported for this head yet" + out.Detail = "Branch protection requires a check named " + check + ", but no check run with that name exists for the PR's current commit, and hive has not yet read its commit statuses. Usually the workflow that produces it has not been triggered; a push, re-run, or label can start it." } else { out.Caption = "CI unknown · " + check + " reported but with no conclusion" out.Detail = "Required check " + check + " exists for the PR's current commit but hive cannot read a clear result from it (several runs share the name, or the run ended with no conclusion)." diff --git a/src/pkg/dashboard/stage_tracker_activity_test.go b/src/pkg/dashboard/stage_tracker_activity_test.go index 9e7b132347..201ba9d732 100644 --- a/src/pkg/dashboard/stage_tracker_activity_test.go +++ b/src/pkg/dashboard/stage_tracker_activity_test.go @@ -792,3 +792,98 @@ func TestStageTrackerActivityCIExcludesMergeGateContexts(t *testing.T) { } }) } + +// hivecommons/promptargs#467 at stage/060-reviewed: branch protection +// requires build-and-test (22), (24) and (26); only (24) and (26) exist on +// the head (both green, with DCO, FOSSA, mutation and friends), Tide is +// pending. The CI chevron read `CI 2/3 · running` for eight hours +// (hivecommons/hive#17508). A required name with no run on the head is a +// settled "missing", never motion, whether or not commit statuses were read; +// the tally counts what GitHub reports. +func TestStageTrackerCIActivityNeverReportedRequiredIsSettled(t *testing.T) { + observed := time.Date(2026, 11, 9, 18, 24, 40, 0, time.UTC) + now := observed.Add(8 * time.Hour) + actions := int64(15368) + done := func(name string) github.PRActivityCheck { + return github.PRActivityCheck{Name: name, State: "completed", Conclusion: "success", Head: "045b8e1", AppID: &actions, Since: &observed} + } + for _, statusesKnown := range []bool{true, false} { + pr := github.PullRequest{HeadSHA: "045b8e1", BaseRef: "main", CIActivity: &github.PRCIActivity{ + Head: "045b8e1", Source: "github-check-runs", ObservedAt: &observed, StatusesKnown: statusesKnown, + RequiredKnown: true, RequiredSource: "protection", + Required: []string{"build-and-test (22)", "build-and-test (24)", "build-and-test (26)"}, + RequiredChecks: []github.PRRequiredCheck{ + {Name: "build-and-test (22)", AppKnown: true, AppID: &actions}, + {Name: "build-and-test (24)", AppKnown: true, AppID: &actions}, + {Name: "build-and-test (26)", AppKnown: true, AppID: &actions}, + }, + Checks: []github.PRActivityCheck{ + done("build-and-test (24)"), done("build-and-test (26)"), done("DCO"), done("mutation"), done("verify"), done("FOSSA"), done("copilot-automation"), + {Name: "tide", State: "pending", Kind: "commit_status", Head: "045b8e1", Description: "Not mergeable. Needs lgtm, approved labels."}, + }, + }} + got := stageTrackerCIActivity(pr, now) + // Statuses read: missing. Not read: unknown (a status could still + // provide it) — either way settled, never active. + want := "missing" + if !statusesKnown { + want = "unknown" + } + if got.Active || got.State != want || got.Blocker != "build-and-test (22)" { + t.Fatalf("statusesKnown=%v: never-reported required check = %+v, want settled %s", statusesKnown, got, want) + } + if !strings.HasPrefix(got.Caption, "CI waiting · required check 'build-and-test (22)' hasn't reported") { + t.Fatalf("statusesKnown=%v: caption = %q", statusesKnown, got.Caption) + } + if got.ChecksTotal != 3 || got.ChecksPassed != 2 || got.ChecksFailed != 0 || len(got.ChecksPending) != 1 { + t.Fatalf("statusesKnown=%v: tally = %d/%d failed=%d pending=%v", statusesKnown, got.ChecksPassed, got.ChecksTotal, got.ChecksFailed, got.ChecksPending) + } + if strings.Contains(got.Caption, "running") || strings.Contains(got.Caption, "no conclusion") { + t.Fatalf("statusesKnown=%v: caption claims motion or ambiguity: %q", statusesKnown, got.Caption) + } + // Every check GitHub reports is concluded: nothing claims motion, + // so no stale re-list is owed however old the snapshot. + if github.CIActivityClaimsMotion(pr.CIActivity) || github.CIActivityStale(pr.CIActivity, now, 10*time.Minute) { + t.Fatalf("statusesKnown=%v: a concluded head claims motion", statusesKnown) + } + } +} + +// A snapshot that still says a check runs is re-listed once it is older +// than dashboard.stage_tracker.ci_activity_stale_m; a fresh one, a +// concluded one, or one whose only motion is Tide's pending status is not. +func TestStageTrackerRelistsStaleRunningCI(t *testing.T) { + now := time.Date(2026, 11, 10, 2, 50, 0, 0, time.UTC) + running := func(age time.Duration, checks ...github.PRActivityCheck) *github.PRCIActivity { + at := now.Add(-age) + return &github.PRCIActivity{Head: "head", Source: "github-check-runs", ObservedAt: &at, Checks: checks} + } + inProgress := github.PRActivityCheck{Name: "build-and-test (22)", State: "in_progress"} + green := github.PRActivityCheck{Name: "build-and-test (24)", State: "completed", Conclusion: "success"} + tide := github.PRActivityCheck{Name: "tide", State: "pending", Kind: "commit_status"} + for _, tc := range []struct { + name string + a *github.PRCIActivity + want bool + }{ + {"running for 17m", running(17*time.Minute, inProgress, green), true}, + {"queued for 17m", running(17*time.Minute, github.PRActivityCheck{Name: "lint", State: "queued"}), true}, + {"running for 3m", running(3*time.Minute, inProgress), false}, + {"all concluded for 8h", running(8*time.Hour, green), false}, + {"only tide pending for 8h", running(8*time.Hour, green, tide), false}, + {"no observation time", &github.PRCIActivity{Head: "head", Checks: []github.PRActivityCheck{inProgress}}, false}, + } { + if got := github.CIActivityStale(tc.a, now, 10*time.Minute); got != tc.want { + t.Errorf("%s: stale = %v, want %v", tc.name, got, tc.want) + } + } + if github.CIActivityStale(running(time.Hour, inProgress), now, 0) { + t.Errorf("a disabled guard (0) relisted") + } + + s := &Server{deps: &Dependencies{}} + pr := github.PullRequest{Repo: "hivecommons/promptargs", Number: 467, HeadSHA: "head", CIActivity: running(17*time.Minute, inProgress)} + if s.stageTrackerRelistStaleCI(pr, now) { + t.Fatalf("relisted with no GitHub client") + } +} diff --git a/src/pkg/dashboard/stage_tracker_build.go b/src/pkg/dashboard/stage_tracker_build.go index e56e7682ba..ddde26791b 100644 --- a/src/pkg/dashboard/stage_tracker_build.go +++ b/src/pkg/dashboard/stage_tracker_build.go @@ -467,6 +467,7 @@ func (s *Server) buildStageTracker(status *StatusPayload, store *stagelabels.Sto // a head the cycle never listed counts from the deliveries alone // (hivecommons/hive#16521). activityPR.CIActivity = github.HeadCIActivity(activityPR.Repo, pr.HeadSHA, pr.CIActivity) + s.stageTrackerRelistStaleCI(activityPR, now) it.CIActivity = stageTrackerCIActivity(activityPR, now) var bundle *evidence.Bundle if evidenceVerifier != nil && pr.HeadSHA != "" { @@ -1079,3 +1080,23 @@ func stageTrackerSinceUnblock(it *StageTrackerItem, rec stagelabels.Record, hasR t := rec.UnblockedAt it.StageSince = &t } + +// stageTrackerRelistStaleCI asks for one fresh check-run listing of a head +// whose CI snapshot still claims a check running or queued after +// dashboard.stage_tracker.ci_activity_stale_m (default 10). The cycle's +// listing can outlive the checks it describes — the GraphQL batch skips a +// clean repository, an ETag replays — and nothing else re-reads a head +// whose PR has not changed; promptargs#467 rendered `CI 2/3 · running` for +// eight hours after every check had passed (hivecommons/hive#17508). The +// listing is coalesced and budget-aware; it lands in the webhook CI +// evidence store for the next build. +func (s *Server) stageTrackerRelistStaleCI(pr github.PullRequest, now time.Time) bool { + if s == nil || s.deps == nil || s.deps.GHClient == nil || pr.HeadSHA == "" { + return false + } + maxAge := s.deps.Config.StageTrackerCIActivityStale() + if !github.CIActivityStale(pr.CIActivity, now, maxAge) { + return false + } + return s.deps.GHClient.RequestHeadCIRelist(pr.Repo, pr.Number, pr.HeadSHA, pr.BaseRef, maxAge) +} diff --git a/src/pkg/dashboard/stage_tracker_remediation_chevron.test.js b/src/pkg/dashboard/stage_tracker_remediation_chevron.test.js index 2d13499f23..392ef02d45 100644 --- a/src/pkg/dashboard/stage_tracker_remediation_chevron.test.js +++ b/src/pkg/dashboard/stage_tracker_remediation_chevron.test.js @@ -305,9 +305,17 @@ test('#16539 shape: a queued swarm reads 0/5 and, as the stage\'s own track, is // hivecommons/hive#16631: a CI ring that is not full is a running ring. // 13/19 with checks still pending is active with its count beside the // running Review; only a complete 19/19 ring is the silent green glyph. +// The backend says running/queued whenever a check is in motion; its own +// `unknown` beside a part-filled ring (a required name with no run, statuses +// unread) is settled, not inferred motion (hivecommons/hive#17508). test('#16628 shape: CI 13/19 with pending checks + review in progress → both tracks active with words', () => { const review = { head: 'abc1234def', state: 'reviewing', active: true, caption: 'Review 0/5 · in progress', perspectives_done: 0, perspectives_total: 5, source: 'dispatch', observed_at: seen }; - for (const state of ['running', 'unknown', 'queued']) { + const unknown = cells(redPR({ display_column: 50, stage: 50, + ci_activity: { head: 'abc1234def', state: 'unknown', caption: 'CI waiting', checks_passed: 13, checks_total: 19, source: 'checks', observed_at: seen }, + review_activity: review })); + assert.doesNotMatch(unknown[4], /class="ham-seg ham-activity-track active ham-current"/, 'unknown: a part-filled ring with no motion reported is not lit'); + assert.equal(litCount(unknown.join('')), 1, 'unknown: only Review is in motion'); + for (const state of ['running', 'queued']) { const c = cells(redPR({ display_column: 50, stage: 50, ci_activity: { head: 'abc1234def', state, caption: 'CI ' + state, checks_passed: 13, checks_total: 19, source: 'checks', observed_at: seen }, review_activity: review })); diff --git a/src/pkg/dashboard/stage_tracker_stale_ci_lead_test.go b/src/pkg/dashboard/stage_tracker_stale_ci_lead_test.go new file mode 100644 index 0000000000..6a8a07ff82 --- /dev/null +++ b/src/pkg/dashboard/stage_tracker_stale_ci_lead_test.go @@ -0,0 +1,49 @@ +package dashboard + +import ( + "os/exec" + "strings" + "testing" +) + +// hivecommons/promptargs#467 at stage/060-reviewed with every check on the +// head concluded (one required name never reported, Tide pending): the row +// lights Review, not CI, and a backend `unknown` word beside a 2/3 ring is +// not inferred motion (hivecommons/hive#17508). A red check at 060 still +// makes CI the blocker. +func TestStageTrackerReviewedRowWithConcludedChecksLightsReview(t *testing.T) { + node, err := exec.LookPath("node") + if err != nil { + t.Skip("node not available") + } + html := indexHTML(t) + script := strings.Join([]string{ + "const assert = require('assert');", + jsFunc(t, html, "stageTrackerWaitingOnPerson"), + jsFunc(t, html, "stageTrackerTrackRunning"), + jsFunc(t, html, "stageTrackerStageTrack"), + jsFunc(t, html, "stageTrackerStalledTrack"), + jsFunc(t, html, "stageTrackerTrackWouldLight"), + jsFunc(t, html, "stageTrackerLeadTrack"), + }, "\n") + ` +const ci = state => ({ head: '045b8e1', state, active: false, checks_total: 3, checks_passed: 2, checks_failed: 0, checks_pending: ['build-and-test (22)'], blocker: 'build-and-test (22)' }); +const row = (stage, ciState, reviewState) => ({ kind: 'pr', repo: 'hivecommons/promptargs', number: 467, head: '045b8e1', stage, ci_activity: ci(ciState), review_activity: { head: '045b8e1', state: reviewState } }); + +for (const st of ['missing', 'unknown', 'passing']) { + const item = row(60, st, 'approved'); + assert.strictEqual(stageTrackerTrackRunning(item, 'ci'), false, 'CI ' + st + ' with a 2/3 ring is not motion at 060'); + assert.strictEqual(stageTrackerStalledTrack(item), 'review', 'at 060 with CI ' + st + ' the stalled track is Review'); + assert.strictEqual(stageTrackerLeadTrack(item, true, stageTrackerStalledTrack(item)), 'review', 'Review is the one lit chevron at 060 with CI ' + st); +} +assert.strictEqual(stageTrackerStalledTrack(row(60, 'failing', 'approved')), 'ci', 'a red check at 060 keeps CI the blocker'); +// 050 keeps its rule: CI not green is the blocker when the review has settled. +assert.strictEqual(stageTrackerStalledTrack(row(50, 'missing', 'approved')), 'ci'); +// Genuine motion still lights CI at 060. +const live = row(60, 'running', 'approved'); live.ci_activity.active = true; +assert.strictEqual(stageTrackerTrackRunning(live, 'ci'), true); +assert.strictEqual(stageTrackerLeadTrack(live, true, ''), 'ci'); +` + if out, err := runNodeStdin(node, script); err != nil { + t.Fatalf("node fixture failed: %v\n%s", err, out) + } +} diff --git a/src/pkg/dashboard/static-src/js/tracker-row-strip.js b/src/pkg/dashboard/static-src/js/tracker-row-strip.js index 86c75c68be..549bf9083e 100644 --- a/src/pkg/dashboard/static-src/js/tracker-row-strip.js +++ b/src/pkg/dashboard/static-src/js/tracker-row-strip.js @@ -209,6 +209,10 @@ // person (human_decision) is at Review: the ask came from the // review, whatever CI has or has not reported (hivecommons/hive#17453). if (on('review') === 'human_decision' && typeof stageTrackerPersonActsOnPR === 'function' && stageTrackerPersonActsOnPR(item)) return 'review'; + // 060 reviewed with every check on the head concluded: the review is + // the row's own stage, so Review is the lit chevron; CI is its + // blocker only when a check is red (hivecommons/hive#17508). + if (Number(item.stage) === 60 && on('ci') !== 'failing') return 'review'; if (on('ci') !== 'passing') return 'ci'; return 'review'; } @@ -278,7 +282,10 @@ // waiting on checks, whatever its cached counts say. if ((item.decision_needed && item.decision_needed.question) || item.human_attention) return false; const total = Number(a.checks_total) || 0, done = (Number(a.checks_passed) || 0) + (Number(a.checks_failed) || 0); - const settledWords = ['passing', 'failing', 'skipped', 'missing', 'human_decision']; + // The backend's own `unknown` (an ambiguous required run, statuses + // not read) is not motion either: a 2/3 ring with a never-reported + // check lit CI for eight hours on a green head (hivecommons/hive#17508). + const settledWords = ['passing', 'failing', 'skipped', 'missing', 'human_decision', 'unknown']; return total > 0 && done < total && !settledWords.includes(a.state); } diff --git a/src/pkg/dashboard/static/index.html b/src/pkg/dashboard/static/index.html index a775fe6a31..f393a0937c 100644 --- a/src/pkg/dashboard/static/index.html +++ b/src/pkg/dashboard/static/index.html @@ -18673,6 +18673,10 @@

Where to get help

// person (human_decision) is at Review: the ask came from the // review, whatever CI has or has not reported (hivecommons/hive#17453). if (on('review') === 'human_decision' && typeof stageTrackerPersonActsOnPR === 'function' && stageTrackerPersonActsOnPR(item)) return 'review'; + // 060 reviewed with every check on the head concluded: the review is + // the row's own stage, so Review is the lit chevron; CI is its + // blocker only when a check is red (hivecommons/hive#17508). + if (Number(item.stage) === 60 && on('ci') !== 'failing') return 'review'; if (on('ci') !== 'passing') return 'ci'; return 'review'; } @@ -18742,7 +18746,10 @@

Where to get help

// waiting on checks, whatever its cached counts say. if ((item.decision_needed && item.decision_needed.question) || item.human_attention) return false; const total = Number(a.checks_total) || 0, done = (Number(a.checks_passed) || 0) + (Number(a.checks_failed) || 0); - const settledWords = ['passing', 'failing', 'skipped', 'missing', 'human_decision']; + // The backend's own `unknown` (an ambiguous required run, statuses + // not read) is not motion either: a 2/3 ring with a never-reported + // check lit CI for eight hours on a green head (hivecommons/hive#17508). + const settledWords = ['passing', 'failing', 'skipped', 'missing', 'human_decision', 'unknown']; return total > 0 && done < total && !settledWords.includes(a.state); } diff --git a/src/pkg/github/ci_activity_stale.go b/src/pkg/github/ci_activity_stale.go new file mode 100644 index 0000000000..93205673f0 --- /dev/null +++ b/src/pkg/github/ci_activity_stale.go @@ -0,0 +1,81 @@ +package github + +import ( + "strings" + "time" +) + +// CIActivityClaimsMotion reports whether the snapshot still shows a check on +// its head executing or waiting: a check run whose status is not completed, +// or a commit status still pending — merge-gate contexts (Tide, DCO) aside, +// since those wait on review, not on CI. +func CIActivityClaimsMotion(a *PRCIActivity) bool { + if a == nil { + return false + } + for _, check := range a.Checks { + if check.Head != "" && !strings.EqualFold(check.Head, a.Head) { + continue + } + if IsMergeGateContext(check.Name) { + continue + } + switch check.State { + case "completed", "unknown", "": + continue + } + return true + } + return false +} + +// CIActivityStale reports whether a snapshot that still claims a check in +// motion was observed more than maxAge ago. GitHub's check runs finish in +// minutes; a snapshot that has said "running" for longer than that was not +// refreshed — the batch cache skipped a clean repository, the ETag replayed, +// or the listing was taken while the checks were starting +// (hivecommons/hive#17508: `CI 2/3 · running` eight hours after every check +// had passed). maxAge <= 0 disables the guard. +func CIActivityStale(a *PRCIActivity, now time.Time, maxAge time.Duration) bool { + if a == nil || maxAge <= 0 || a.ObservedAt == nil || a.ObservedAt.IsZero() { + return false + } + if !CIActivityClaimsMotion(a) { + return false + } + return now.Sub(*a.ObservedAt) > maxAge +} + +// RequestHeadCIRelist queues one coalesced check-run listing for the head, +// bypassing the GraphQL batch cache and any ETag: the result lands in the +// webhook CI evidence store, which HeadCIActivity lays over the cycle's +// listing on the next tracker build. A head relisted within minGap is left +// alone so a genuinely long-running check costs one listing per gap, not +// one per build. It reports whether a listing is now pending. +func (c *Client) RequestHeadCIRelist(repo string, number int, head, baseRef string, minGap time.Duration) bool { + if c == nil || strings.TrimSpace(head) == "" || number <= 0 { + return false + } + r := c.ciRefetcher() + key := ciEvidenceKey(repo, head) + r.mu.Lock() + if r.relisted == nil { + r.relisted = map[ciHeadKey]time.Time{} + } + now := sharedWebhookTracker().now() + if last, ok := r.relisted[key]; ok && minGap > 0 && now.Sub(last) < minGap { + r.mu.Unlock() + return false + } + r.relisted[key] = now + for k, at := range r.relisted { + if now.Sub(at) > ciRelistMemory { + delete(r.relisted, k) + } + } + r.mu.Unlock() + return r.enqueue(ciRefetchTarget{repo: repo, fullName: repo, head: head, number: number, baseRef: baseRef}) +} + +// ciRelistMemory bounds how long a head's last stale-relist time is kept. +const ciRelistMemory = 24 * time.Hour diff --git a/src/pkg/github/webhook_ci_evidence.go b/src/pkg/github/webhook_ci_evidence.go index da5f1a5893..8b7a69c8e2 100644 --- a/src/pkg/github/webhook_ci_evidence.go +++ b/src/pkg/github/webhook_ci_evidence.go @@ -559,6 +559,9 @@ type ciRefetcher struct { newTimer func(d time.Duration, f func()) ciRefetchTimer fetch func(ctx context.Context, t ciRefetchTarget) error ctx func() context.Context + // relisted is when each head was last relisted for a stale snapshot + // (RequestHeadCIRelist). + relisted map[ciHeadKey]time.Time // flushes counts debounce windows that fired (tests). flushes int } diff --git a/src/pkg/github/webhook_ci_evidence_test.go b/src/pkg/github/webhook_ci_evidence_test.go index b57cb4917f..20bc3d3d83 100644 --- a/src/pkg/github/webhook_ci_evidence_test.go +++ b/src/pkg/github/webhook_ci_evidence_test.go @@ -302,3 +302,39 @@ func TestWebhookCIRefetchListsHeadOnceAndRecordsListing(t *testing.T) { t.Fatalf("cancelled context still listed: %d", listings) } } + +// A tracker build that finds a head's CI snapshot still claiming motion past +// the stale window asks for one coalesced listing of that head, bypassing +// the batch cache; the same head is not relisted again within the window, +// so a genuinely long-running check costs one listing per window, not one +// per build (hivecommons/hive#17508). +func TestRequestHeadCIRelistCoalescesPerHead(t *testing.T) { + c, clock := setupWebhookCacheTest(t) + t.Cleanup(ResetCIEvidenceForTest(clock.Now)) + var fired func() + c.ciRefetch = &ciRefetcher{newTimer: func(_ time.Duration, f func()) ciRefetchTimer { fired = f; return &ciTestTimer{} }} + var fetched []ciRefetchTarget + c.ciRefetch.fetch = func(_ context.Context, t ciRefetchTarget) error { fetched = append(fetched, t); return nil } + + if !c.RequestHeadCIRelist("hivecommons/promptargs", 467, "045b8e1", "main", 10*time.Minute) { + t.Fatalf("first stale request not queued") + } + if c.RequestHeadCIRelist("hivecommons/promptargs", 467, "045b8e1", "main", 10*time.Minute) { + t.Fatalf("a head relisted seconds ago was queued again") + } + if c.ciRefetch.pendingCount() != 1 || fired == nil { + t.Fatalf("pending=%d timer armed=%v, want one coalesced listing", c.ciRefetch.pendingCount(), fired != nil) + } + fired() + if len(fetched) != 1 || fetched[0].head != "045b8e1" || fetched[0].number != 467 || fetched[0].baseRef != "main" { + t.Fatalf("fetched = %+v, want one listing of 045b8e1 for #467 on main", fetched) + } + + clock.now = clock.now.Add(11 * time.Minute) + if !c.RequestHeadCIRelist("hivecommons/promptargs", 467, "045b8e1", "main", 10*time.Minute) { + t.Fatalf("a head still claiming motion past the window was not relisted again") + } + if c.RequestHeadCIRelist("hivecommons/promptargs", 0, "045b8e1", "main", 10*time.Minute) || c.RequestHeadCIRelist("hivecommons/promptargs", 467, "", "main", 10*time.Minute) { + t.Fatalf("a request without a PR number or head was queued") + } +}