Repository navigation
🐛 Reconcile stalled GitHub Actions check reporting - #15941
Conversation
Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at cc573cc. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
There was a problem hiding this comment.
Read the tree at cc573cc plus the callers of EvaluateCommitCI. The observation module itself is bounded and read-only as the body says; the problem is where its result is written.
Finding (high): rewriting st.Reason silences the workflow-approval detector on trusted lanes.
commit_ci.go:662-671— when the walk ends inrequired-check-pending:<name>andPRNumber > 0,ciReportingReasoncan replacest.Reasonwithci-reporting-stalled:…orci-reporting-unverified:….ciReportingReason(ci_reporting.go:404-411) returns the unverified form for any observation whoseDetail != "missing-actions-provenance"— i.e.owning-workflow-not-verified,owning-workflow-status-unknown,check-not-in-latest-attempt, every API-error branch, etc.automerge_sweep_trusted.go:148-156— the trusted-lane sweep passes that reason straight intodetectActionRequiredWorkflowRuns, andautomerge_held_runs.go:179only proceeds when the reason has prefixrequired-check-pending:or equalscheck-pending/ci-unverified. The new prefixes are not in that list.- The exact case that detector exists for (its doc at
automerge_held_runs.go:163-166: required context exists, run is held for approval) now routes throughci_reporting.go:281-290: a held run hasstatus/conclusionaction_required, which is not in thequeued|in_progress|waiting|requested|pendinglist, so it lands indefault:→owning-workflow-status-unknown→ state staysunverified→ciReportingReasonemitsci-reporting-unverified:…→ theHasPrefix(required-check-pending:)check at:179fails →workflowApprovalRequiredReasonand the/ok-to-testposting path never run. The PR stays parked under a reason that says "owner-action-required" instead of the hive performing the approval it already knows how to do. - Same shape at
stage_tracker_merge_gate.go:243(stageTrackerCIPendingReason), which the PR did not extend, sostageTrackerMergeAttemptStale(:232) stops treating these rows as waiting on CI. Lower impact (a stale merge-failed badge persists), but it is the same missed consumer.
Two ways out, either is small: keep st.Reason as required-check-pending:<name> and let callers read st.Reporting (the field already exists and both callers already log it), or add the two new prefixes to automerge_held_runs.go:179 and stage_tracker_merge_gate.go:243 with a test that drives an action_required run through commitGreenForPR → detectActionRequiredWorkflowRuns. cistatus.ReasonIsPending and humanizeStageReason were updated, so the pattern was understood — these two were just missed. The test files assert only the prefix shape, which is why they stay green.
Everything else I checked holds: cache key includes check IDs+status so status changes bypass the 5m recheck; busy guards concurrent observation; webhook_cache.go workflow_run routing is additive; no writes to GitHub.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Verified source merge / post-merge CI ownershipSource actually merged by the hivecommons-hive App at 2026-10-31T02:47:15Z, squash All required Go build/test/coverage/lint/security/DCO checks passed; no correction commits, manual cancellations or CI reruns were needed. Historical policy/DCO cancellations have newer successful same-head replacements. Full image CI is NOT yet claimed terminal: actual REST workflow 41160707817, attempt 1, and main ARM64 job 124417447794 are still Source delivered: Recovery gap unchanged: no new rerun RPC/executor/claim authority, no automatic reporting-stall recovery or credential expansion. Confirmed cases expose owner inspection/action. Future recovery must separately prove atomic pre-effect exact-episode authority and enforced designated executor; local flock/post-effect comments/destination CAS alone cannot provide cross-hive Actions fencing. Ambiguous RPC intent cannot be automatically stolen/retried after TTL. Integration author notified of landed DTO; detection does not depend on enabling integration. No production rollout, runtime configuration, protection change or workflow rerun was performed by this task. Parent's separate single affected-workflow rerun 41108783466 attempt2 succeeded at 02:24:25Z, verified read-only; no broader PR-green/merge-readiness or infrastructure-cause claim is inferred. |
FINAL terminal exact-head CI receiptThe outstanding main ARM64 image job 124417447794 completed SUCCESS at 2026-10-31T02:58:12Z; owning image workflow 41160707817 attempt1 completed SUCCESS at 02:58:13Z, both explicitly bound to tested head CI sources:
Actual source merge remains 2026-10-31T02:47:15Z by the hivecommons-hive App, squash Recovery remains detection/escalation ONLY. No new Actions rerun executor or proven cross-hive pre-effect claim authority is delivered by this PR. No unsafe TTL-based retry/steal, failover/exactly-once guarantee, configuration/permission/protection change, production rollout or native activation. Runtime adoption/publication remain separate from this verified source/CI receipt. |
Fixes #15938. Owner-authorized detection/escalation implementation, independent of #15905 and experimental integration. Automatic recovery gap is explicit.
Actual inconsistency / prior art
Owner observed #15841 exact head
1731402a81418f7207f6337eafa68fe971b5b279: required checks remained queued/in progress while their owning Actions run41108783466was RESTcompleted/failure. This proves reporting inconsistency, not infrastructure cause. Parent already performed ONE justified same-head rerun; this PR performs no rerun and newer active/success executions must supersede old terminal evidence. #15849 is another session's responsibility and untouched.Shared
EvaluateCommitCIjudges head checks; existingopaqueWorkflowRunsdeliberately excludes runs with jobs. Thus unfinished check reporting on terminal runs remains ordinary CI waiting. Reconcile observationally through that shared evaluator, used by automerge lanes and merge-request watcher, without rewriting CI conclusions.Delivered semantics
CIReportingObservation: repo, PR/head, check ID/name/state, run/latest attempt, job/state/result, workflow conclusion/update time/URL, observation time and distinct state/recovery guidance.ci-reporting-stalledidentifies exact actionable provenance; API refusal isci-reporting-unverified.HIVE_CI_REPORTING_RECHECK/HIVE_CI_REPORTING_GRACEdefaults); verified check/status/workflow events dirty only their head, refresh at most once/minute. No render/API polling or duplicate-event grace shortcut.Explicit automatic recovery gap
No rerun RPC, recovery claim/executor, live permission/config change or production rollout. Confirmed stalls ask an owner to inspect the current exact workflow attempt/logs before one affected-workflow rerun. No rerun-all, cancellation, rebase or ref mutation.
Existing shared-CI comment dedupe happens after mutation;
/data/convergence/mutationledger/flock only serializes actors sharing that filesystem and OFF mode bypasses it. Neither proves cross-hive atomic pre-effect authority. Integration author is coordinating a future narrow adapter from this read-only episode signal, not a competing reconciler. Future recovery requires enforced designated executor/observer-only other hives plus durable exact-action pre-effect intent. Actions RPC lacks fence/idempotency input: designated routing/forge CAS alone is not automatic failover or exactly-once; ambiguous dispatch must never be stolen/retried merely after TTL. Detection does not depend on enabling integration.CI-only regression verdict
Added exact-run lifecycle/grace, active-old workflow, failed versus successful workflow with unfinished reporting, latest attempt/older canceled execution, foreign run/head/repo/suite/job/check URL, API refusal/rate limit/unavailable, unknown states, head/base change during reads, duplicate event/single-flight/cache bounds, restart/clock/API-gap reset, actual successful newer checks, unchanged commit/base behavior and existing tooltip coverage. HTTP fixture refuses every non-GET method.
gofmtandgit diff --checkcompleted. No local test/build/vet/lint/docs guards under AGENTS. Will own full immutable-head CI/build/lint including ARM before owner-authorized guarded DCO admin squash, without extra review wait or labels/rules manipulation.