Skip to content

🐛 fix(dashboard): CI chevron no longer stays lit after every check has concluded - #17509

Merged
clubanderson merged 1 commit into
v5from
fix/ci-chevron-stale
Nov 10, 2026
Merged

clubanderson merged 1 commit into
v5from
fix/ci-chevron-stale

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #17508

Problem

The Activity Monitor on hive's own spoke showed hivecommons/promptargs#467 (head 045b8e1, stage/060-reviewed) with the CI chevron lit — CI 2/3 · 'build-and-te…' running — for ~8 hours after every check on the head had SUCCEEDED (18:24Z); only Prow's tide status was pending.

Root cause

Branch protection on promptargs/main requires build-and-test (22), (24), (26), but the workflow has no 22 leg: no run named build-and-test (22) ever existed on the head (governor logs: required-check-missing:build-and-test (22) throughout; the "CI" workflow run has exactly two jobs). The tracker then:

  1. stage_tracker_activity.go left the never-reported required name as unknown (statuses unread) with a 2/3 tally and the name in checks_pending;
  2. tracker-row-strip.js stageTrackerTrackRunning inferred motion from any part-filled ring whose state was not a settled word (unknown was not), lighting CI and spelling the pending name as running;
  3. stageTrackerStalledTrack picks CI at 060 whenever CI is not passing, so CI, not Review, held the lit slot;
  4. nothing re-lists a head whose PR has not changed (GraphQL batch skips clean repos and reuses the cached check observation; no staleness guard), so the snapshot could never correct itself. The "since 17m" was the 060↔050 stage-label flap at 02:35–02:37Z, not CI.

Fix

  • Go stageTrackerCIActivity: a required name with no run is settled — missing when statuses were read, unknown (a status could still provide it) when not — with the hasn't reported caption, never "reported but with no conclusion"; Active stays false.
  • JS: the backend's unknown is a settled word for inferred motion; at 060 reviewed the stalled track is Review unless CI is red.
  • Staleness guard: github.CIActivityStale (check-run status != completed or non-merge-gate pending status, observed > dashboard.stage_tracker.ci_activity_stale_m, default 10, -1 disables) → Client.RequestHeadCIRelist queues one coalesced REST listing of the head through the existing CI refetcher (budget-aware, lands in the webhook CI evidence store, which HeadCIActivity lays over the cycle's listing), at most once per window per head.
  • Doc line in activity-monitor.md, operator-reference key, changelog fragment.

Tests

  • TestStageTrackerCIActivityNeverReportedRequiredIsSettled — fix: cadence 'pause' means off (no kicks), not force-paused #467's data (required 22/24/26, only 24/26 exist, green neighbours, tide pending): not active, tally 2/3, hasn't reported caption, no motion claimed.
  • TestStageTrackerRelistsStaleRunningCI — stale detection table (running 17m → relist; fresh / concluded / tide-only → not; 0 disables).
  • TestRequestHeadCIRelistCoalescesPerHead — one listing per head per window through the refetcher.
  • TestStageTrackerReviewedRowWithConcludedChecksLightsReview — JS: at 060 with CI missing/unknown/passing, Review is the one lit chevron; red CI stays the blocker; genuine motion still lights CI.
  • #16628 shape JS test updated: unknown beside a part-filled ring is no longer inferred motion.

Evidence

Tracker payload (/api/stage-tracker, post-merge), pod logs (--since=10h), GraphQL statusCheckRollup probes (no EXPECTED contexts for missing required checks), branch-protection required_status_checks, and the workflow run's jobs were captured on the operator's machine under ~/hive-work/stale-ci-*.json.

…s concluded

A required check with no run on the head is a settled word, never
inferred motion from a part-filled tally; at 060 reviewed with every
reported check concluded, Review is the lit chevron and CI is its
blocker only when red. A CI snapshot that still claims a check running
or queued after dashboard.stage_tracker.ci_activity_stale_m (default
10) is re-listed from GitHub on the next tracker build, bypassing the
batch cache, ETag and stage, coalesced per head.

Fixes #17508

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 10, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

/lgtm
/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@clubanderson: you cannot LGTM your own PR.

Details

In response to this:

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 10, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17505 🐛 dashboard: one human-gate pill per gated row — fixed verbs per reason, no needs-human chevron words, Confirm safe for sentinel alerts same files src/docs/dashboard-sections/activity-monitor.md
src/pkg/dashboard/static-src/js/tracker-row-strip.js
src/pkg/dashboard/static/index.html

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read against the v5 tree; this looks correct to me.

What I checked:

  • CIActivityClaimsMotion (src/pkg/github/ci_activity_stale.go) treats only completed / unknown / "" as settled. That matches how State is populated: check runs store cr.GetStatus() (src/pkg/github/pr_ci_activity.go:58, so finished runs are completed), and commit statuses are written as completed with the result in Conclusion, or pending / unknown (pr_ci_activity.go:133-146, webhook_ci_evidence.go:426-439). So a fully green head is not relisted every window — the stale re-list only fires while a run genuinely still says queued/in_progress/pending.
  • The new noRunUnread caption branch in stage_tracker_activity.go is reachable: statusesMissing requires out.State == "passing" (stage_tracker_activity.go:199), and a never-reported required name leaves the state unknown, so it does not shadow the new else if.
  • RequestHeadCIRelist uses the existing ciRefetcher / enqueue / ciEvidenceKey / ciRefetchTarget{…baseRef} plumbing from webhook_ci_evidence.go:541-624; the per-head minGap gate bounds it to one listing per window, and the test covers the 11-minute re-arm.
  • JS: the new 060 rule in stageTrackerStalledTrack sits after the human_decision rule and before the ci !== 'passing' fallback, and stageTrackerStageTrack only claims 050, so 060 rows now light Review unless a check is red — the node fixture asserts exactly that, including the running+active case still lighting CI.

One minor observation, not a defect: RequestHeadCIRelist sweeps the whole relisted map under the lock on every call. With ciRelistMemory at 24h and one entry per stale head that is small, so fine as-is.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

docs-currency — read head 7d38cf1. Docs track the behavior change well (activity-monitor.md:457, runtime-and-quota.md:648, config.go:5214-5219, changelog fragment). Two small currency items, neither blocking:

  1. src/docs/webhooks.md "Configuration and defaults" table omits the new key. That table (webhooks.md:107-127) lists every other dashboard.stage_tracker.* freshness knob — webhook_refresh_debounce_ms, webhook_refresh_replay_window_s, closed_sweep_interval_s, waiting_push_stale_h, merged_hold_s — and ci_activity_stale_m is precisely a guard against the batch cache / ETag replay the table describes. Suggested row after webhooks.md:126:

    | `dashboard.stage_tracker.ci_activity_stale_m` | `10` | minutes a head's CI snapshot may still claim a check running or queued (check run not `completed`, non-merge-gate status `pending`) before one coalesced check-run listing of the head is queued, bypassing the batch cache and ETag; at most one listing per head per window; negative disables |
    
  2. "re-listed … on the next tracker build" is one build early. stageTrackerRelistStaleCI (stage_tracker_build.go:1084-1102) only queues the listing via RequestHeadCIRelist (ci_activity_stale.go:52-80); its result lands in the webhook CI evidence store and is laid over by HeadCIActivity on the build after that (stage_tracker_build.go:469-470, and the Go comment itself says "lands … for the next build"). The sentence in activity-monitor.md:457 and the changelog fragment reads as if the same build corrects the row. Suggest "is re-listed from GitHub (queued on the build that notices, applied on the following one)". Info only.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Intent alignment (#17508) — read at head 7d38cf1. The three fixes the issue asks for are implemented: a never-reported required name is settled with the hasn't reported caption and Active stays false (stage_tracker_activity.go:159-167, :315-318); unknown is a settled word for inferred motion and 060 lights Review unless CI is red (tracker-row-strip.js:212-216, :285-290); a stale snapshot queues one coalesced relist via the existing refetcher (stage_tracker_build.go:470, ci_activity_stale.go:52-78), and merged() prefers the newer store listing over the cycle's base (webhook_ci_evidence.go:278-280), so the relist does correct the snapshot. Tests cover #467's exact data.

One in-scope gap — the staleness guard on commit statuses is self-defeating and drops status evidence

CIActivityClaimsMotion counts a non-merge-gate commit status still pending as motion (ci_activity_stale.go:12-13, :22-27), and the doc/changelog promise a relist for it. But the relist it queues is refetchHeadCI, which calls only Checks.ListCheckRunsForRef and builds the activity with prCIActivity from check runs alone (webhook_ci_evidence.go:684-701, pr_ci_activity.go:52-75) — no commit statuses, StatusesKnown false. recordListed then replaces the stored listing wholesale (:173), and merged() returns that store listing instead of the cycle's status-bearing base whenever it is newer (:278-280).

Consequence on a Prow repo whose presubmit is a commit status and genuinely runs > 10 min (common): the relist cannot resolve the pending status that triggered it; instead the status disappears from the merged view, the required name falls into the !ok && !StatusesKnown branch (stage_tracker_activity.go:159-167) and reads hasn't reported / unknown — which this same PR now makes a settled, unlit word — while the check is actually running. That inverts the issue's "a CI word is motion only when GitHub reports it" rule in the other direction, and also drops tide/DCO status evidence from the row.

Smallest fix: only check runs can make a snapshot stale, since only check runs are relisted:

	for _, check := range a.Checks {
		if check.Head != "" && !strings.EqualFold(check.Head, a.Head) {
			continue
		}
		// Only check runs are relisted (refetchHeadCI lists check runs, not
		// commit statuses), so only a check run can make the snapshot stale.
		if check.Kind == "commit_status" || IsMergeGateContext(check.Name) {
			continue
		}

and drop "or a non-merge-gate commit status still pending" from the CIActivityClaimsMotion doc comment, activity-monitor.md:457 and the changelog fragment. (Alternatively make refetchHeadCI also list combined statuses and set StatusesKnown, but that is a larger change and touches the budget-gated REST path.) A table case in TestStageTrackerRelistsStaleRunningCI with {Name: "pull-foo-unit", State: "pending", Kind: "commit_status"} aged 17m expecting false would pin it.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review (head 7d38cf1, read with the surrounding tree: webhook_ci_evidence.go merged/recordListed/refetchHeadCI, protection_facts.go attachRequiredChecks, tracker-row-strip.js callers).

The fix lines up with #17508: the Go unknown path was already Active=false (stage_tracker_activity.go:330), so the lit chevron came from the JS inferred-motion rule — adding unknown to settledWords (tracker-row-strip.js:288) and the 060 stalled-track rule (:215) close it, and the new tests cover the #467 data in both the statuses-read and unread shapes. The relist path is sound: refetchHeadCI records a listing with ObservedAt = now and attachRequiredChecks re-reads commit statuses when baseRef is set, so merged() (webhook_ci_evidence.go:278) prefers the fresh listing without dropping Prow status evidence.

One low-severity edge case, advisory only:

  • CIActivityClaimsMotion (ci_activity_stale.go:12-36) treats any non-merge-gate commit status still pending as motion. A context that stays pending by design and is not in IsMergeGateContext (a CLA/licence bot waiting on a signer, a WIP context, a preview-deploy status that never resolves) makes CIActivityStale true forever, so stageTrackerRelistStaleCI (stage_tracker_build.go:470) queues one REST listing per head per ci_activity_stale_m window indefinitely — the relist can never clear a status that is genuinely still pending. It is bounded (one per window, refused at critical budget), but on a spoke with many such PRs this is a steady ~6 listings/hour/head of pure cost. A cheap guard: skip the relist when the only motion is a pending commit status whose Since is older than, say, 6× maxAge, or cap consecutive relists per head that returned the same pending set.

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing (advisory)

No surviving mutants in changed files (or the run hit its time budget before reporting any).

Advisory only; this job never blocks the PR (#17349).

@clubanderson
clubanderson merged commit 0994434 into v5 Nov 10, 2026
87 of 94 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/ci-chevron-stale branch November 10, 2026 03:39
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Activity Monitor: CI chevron stayed lit as running for 8h after every check had concluded (promptargs#467)

1 participant