Skip to content

🐛 Reconcile stalled GitHub Actions check reporting - #15941

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
fix/ci-reporting-reconciliation
Oct 31, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
fix/ci-reporting-reconciliation

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #15938. Owner-authorized detection/escalation implementation, independent of #15905 and experimental integration. Automatic recovery gap is explicit.

Actual inconsistency / prior art

Owner observed #15841 exact head 1731402a81418f7207f6337eafa68fe971b5b279: required checks remained queued/in progress while their owning Actions run 41108783466 was REST completed/failure. This proves reporting inconsistency, not infrastructure cause. Parent already performed ONE justified same-head rerun; this PR performs no rerun and newer active/success executions must supersede old terminal evidence. #15849 is another session's responsibility and untouched.

Shared EvaluateCommitCI judges head checks; existing opaqueWorkflowRuns deliberately excludes runs with jobs. Thus unfinished check reporting on terminal runs remains ordinary CI waiting. Reconcile observationally through that shared evaluator, used by automerge lanes and merge-request watcher, without rewriting CI conclusions.

Delivered semantics

  • Exact PR head + known required set + unfinished checks; commit-only/base observations unchanged.
  • Verify Actions App/check head and suite, owning run's repository/workflow/event/number/latest attempt, and latest-attempt job's run/head/attempt/check API URL. Re-read owning run and PR head/base before accepting a contradiction.
  • Require two independently fetched consistent samples across a five-minute grace. Run terminal age alone is insufficient. New execution/attempt/head, metadata changes, API gaps and restart cannot inherit old confirmation.
  • Ordinary active/queued workflows are not stale, regardless of duration. Older canceled evidence cannot override newer execution/success. Unknown provider/provenance/status/API/permission/pagination is not green or a fabricated workflow verdict.
  • Read-only CIReportingObservation: repo, PR/head, check ID/name/state, run/latest attempt, job/state/result, workflow conclusion/update time/URL, observation time and distinct state/recovery guidance. ci-reporting-stalled identifies exact actionable provenance; API refusal is ci-reporting-unverified.
  • Existing reason tooltip only; no tracker rows/layout/sorts/filters/stages/preparation rail/raw DTO/native integration/settings changes. Reporting reasons retain CI-pending stage classification and never satisfy required checks or bypass any hold/review/authorization/current-base policy.
  • Bounded identity/PR/head/check-set cache (1,024 entries, check/run/job-page bounds), single-flight reads outside locks. Five-minute recheck/grace (HIVE_CI_REPORTING_RECHECK / HIVE_CI_REPORTING_GRACE defaults); verified check/status/workflow events dirty only their head, refresh at most once/minute. No render/API polling or duplicate-event grace shortcut.

Explicit automatic recovery gap

No rerun RPC, recovery claim/executor, live permission/config change or production rollout. Confirmed stalls ask an owner to inspect the current exact workflow attempt/logs before one affected-workflow rerun. No rerun-all, cancellation, rebase or ref mutation.

Existing shared-CI comment dedupe happens after mutation; /data/convergence/mutation ledger/flock only serializes actors sharing that filesystem and OFF mode bypasses it. Neither proves cross-hive atomic pre-effect authority. Integration author is coordinating a future narrow adapter from this read-only episode signal, not a competing reconciler. Future recovery requires enforced designated executor/observer-only other hives plus durable exact-action pre-effect intent. Actions RPC lacks fence/idempotency input: designated routing/forge CAS alone is not automatic failover or exactly-once; ambiguous dispatch must never be stolen/retried merely after TTL. Detection does not depend on enabling integration.

CI-only regression verdict

Added exact-run lifecycle/grace, active-old workflow, failed versus successful workflow with unfinished reporting, latest attempt/older canceled execution, foreign run/head/repo/suite/job/check URL, API refusal/rate limit/unavailable, unknown states, head/base change during reads, duplicate event/single-flight/cache bounds, restart/clock/API-gap reset, actual successful newer checks, unchanged commit/base behavior and existing tooltip coverage. HTTP fixture refuses every non-GET method.

gofmt and git diff --check completed. No local test/build/vet/lint/docs guards under AGENTS. Will own full immutable-head CI/build/lint including ARM before owner-authorized guarded DCO admin squash, without extra review wait or labels/rules manipulation.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 31, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive
hivecommons-hive Bot merged commit 85e9ec7 into v5 Oct 31, 2026
79 of 82 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/ci-reporting-reconciliation branch October 31, 2026 02:47
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#15941cc573cc` sha256:28b887c8528510fe4107558b2715334d41571cb695e04176719fd8abd018a642 — download from the dashboard or `hivectl review evidence #15941`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at cc573cc.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the tree at cc573cc plus the callers of EvaluateCommitCI. The observation module itself is bounded and read-only as the body says; the problem is where its result is written.

Finding (high): rewriting st.Reason silences the workflow-approval detector on trusted lanes.

  • commit_ci.go:662-671 — when the walk ends in required-check-pending:<name> and PRNumber > 0, ciReportingReason can replace st.Reason with ci-reporting-stalled:… or ci-reporting-unverified:…. ciReportingReason (ci_reporting.go:404-411) returns the unverified form for any observation whose Detail != "missing-actions-provenance" — i.e. owning-workflow-not-verified, owning-workflow-status-unknown, check-not-in-latest-attempt, every API-error branch, etc.
  • automerge_sweep_trusted.go:148-156 — the trusted-lane sweep passes that reason straight into detectActionRequiredWorkflowRuns, and automerge_held_runs.go:179 only proceeds when the reason has prefix required-check-pending: or equals check-pending / ci-unverified. The new prefixes are not in that list.
  • The exact case that detector exists for (its doc at automerge_held_runs.go:163-166: required context exists, run is held for approval) now routes through ci_reporting.go:281-290: a held run has status/conclusion action_required, which is not in the queued|in_progress|waiting|requested|pending list, so it lands in default: → owning-workflow-status-unknown → state stays unverified → ciReportingReason emits ci-reporting-unverified:… → the HasPrefix(required-check-pending:) check at :179 fails → workflowApprovalRequiredReason and the /ok-to-test posting path never run. The PR stays parked under a reason that says "owner-action-required" instead of the hive performing the approval it already knows how to do.
  • Same shape at stage_tracker_merge_gate.go:243 (stageTrackerCIPendingReason), which the PR did not extend, so stageTrackerMergeAttemptStale (:232) stops treating these rows as waiting on CI. Lower impact (a stale merge-failed badge persists), but it is the same missed consumer.

Two ways out, either is small: keep st.Reason as required-check-pending:<name> and let callers read st.Reporting (the field already exists and both callers already log it), or add the two new prefixes to automerge_held_runs.go:179 and stage_tracker_merge_gate.go:243 with a test that drives an action_required run through commitGreenForPR → detectActionRequiredWorkflowRuns. cistatus.ReasonIsPending and humanizeStageReason were updated, so the pattern was understood — these two were just missed. The test files assert only the prefix shape, which is why they stay green.

Everything else I checked holds: cache key includes check IDs+status so status changes bypass the 5m recheck; busy guards concurrent observation; webhook_cache.go workflow_run routing is additive; no writes to GitHub.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson

Copy link
Copy Markdown
Member Author

Verified source merge / post-merge CI ownership

Source actually merged by the hivecommons-hive App at 2026-10-31T02:47:15Z, squash 85e9ec78ed968e679beb6c33d6aab80e32836322, immutable source/check head cc573cc468ea65d56cfa8dae874d6988cbd0ba1c. I did NOT invoke a merge and will not issue another. Verified squash DCO; issue #15938 CLOSED. Owned clean worktree and local/remote feature branch removed safely; fresh origin/v5 includes the squash, shared working files untouched.

All required Go build/test/coverage/lint/security/DCO checks passed; no correction commits, manual cancellations or CI reruns were needed. Historical policy/DCO cancellations have newer successful same-head replacements. Full image CI is NOT yet claimed terminal: actual REST workflow 41160707817, attempt 1, and main ARM64 job 124417447794 are still in_progress on this exact head, at Build and push by digest. This is an ordinary active workflow, not a reporting-stall claim. I retain responsibility for the final terminal exact-head CI receipt.

Source delivered: src/pkg/github/ci_reporting.go exported CIReportingObservation; CommitCIState.Reporting populated through shared EvaluateCommitCI, with exact run/suite/latest-attempt/job/check provenance, independent propagation-grace observations, bounded event-aware single-flight cache, explicit unknown states and CI-pending-compatible existing tooltip reasons. No tracker rows/layout/rail or gate weakening.

Recovery gap unchanged: no new rerun RPC/executor/claim authority, no automatic reporting-stall recovery or credential expansion. Confirmed cases expose owner inspection/action. Future recovery must separately prove atomic pre-effect exact-episode authority and enforced designated executor; local flock/post-effect comments/destination CAS alone cannot provide cross-hive Actions fencing. Ambiguous RPC intent cannot be automatically stolen/retried after TTL. Integration author notified of landed DTO; detection does not depend on enabling integration.

No production rollout, runtime configuration, protection change or workflow rerun was performed by this task. Parent's separate single affected-workflow rerun 41108783466 attempt2 succeeded at 02:24:25Z, verified read-only; no broader PR-green/merge-readiness or infrastructure-cause claim is inferred.

@clubanderson

Copy link
Copy Markdown
Member Author

FINAL terminal exact-head CI receipt

The outstanding main ARM64 image job 124417447794 completed SUCCESS at 2026-10-31T02:58:12Z; owning image workflow 41160707817 attempt1 completed SUCCESS at 02:58:13Z, both explicitly bound to tested head cc573cc468ea65d56cfa8dae874d6988cbd0ba1c. All applicable CI is now terminal green: 67 success / 15 scope-skipped / 0 actual failures / 0 unfinished checks (includes two successful post-merge event checks). All required checks reread SUCCESS. Historical native-policy/DCO cancellations have proven newer successful same-head replacements; remaining Tide Needs approved, lgtm is controller metadata, not unfinished CI.

CI sources:

Actual source merge remains 2026-10-31T02:47:15Z by the hivecommons-hive App, squash 85e9ec78ed968e679beb6c33d6aab80e32836322; source merged before optional ARM completion, not by my merge invocation. Verified DCO/closed #15938; own worktree/local+remote feature branch safely cleaned, fresh origin/v5 includes squash; integration author received landed DTO contract. No CI correction commits or manual reruns/cancellations were needed.

Recovery remains detection/escalation ONLY. No new Actions rerun executor or proven cross-hive pre-effect claim authority is delivered by this PR. No unsafe TTL-based retry/steal, failover/exactly-once guarantee, configuration/permission/protection change, production rollout or native activation. Runtime adoption/publication remain separate from this verified source/CI receipt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Detect terminal-workflow CI reporting stalls without unsafe reruns

1 participant