Skip to content

🐛 fix(github): hold skewed heads from first observation without expect_sha and log head-time skew - #16155

Merged
hivecommons-hive[bot] merged 3 commits into
v5from
scanner/fix-16147
Nov 1, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
v5from
scanner/fix-16147

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

When GitHub's head time ran ahead of the host clock, hive-merge requests without expect_sha had no persisted wait state, so the min_head_age hold read "head first seen 0s ago" on every poll and lasted the whole skew. The CI gate now keeps its own in-memory first observation per repo/PR/head for those requests (a new head starts a new hold, so it stays fail-closed). CommitCIState also gains HeadTimeAheadBy, and both the automerge sweep and the merge-request gate log it, so a skew-handled head can be told apart from a normally aged one.

Closes #16147
Closes #16152

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…t_sha and log head-time skew

Closes #16147
Closes #16152

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 1, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16052 🐛 fix: prepare independent ready PRs without completed-refresh FIFO stalls same files src/pkg/github/automerge/automerge_sweep_queued.go

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 1, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read against src/pkg/github/merge_request_watcher.go, merge_ci_gate.go, merge_request_state.go and commit_ci.go at the PR head. Looks correct to me.

  • The mechanism in the body is real: the watcher calls putMergeRequestState with SHA: req.ExpectSHA (merge_request_watcher.go:483-487) and putMergeRequestState returns early on an empty SHA (merge_request_state.go:114-117), so prState.PendingSince is zero on every poll for a no-expect_sha request and EvaluateCommitCI's skew branch (commit_ci.go:674-681) clamps age to 0 indefinitely.
  • mergeRequestHeadFirstSeen (merge_request_state.go:137-158) is only consulted when minHeadAge > 0 && firstSeen.IsZero() && expectSHA == "" (merge_ci_gate.go:203-205), takes mergeStateMu itself and is not called under it, and a different head SHA overwrites the entry, so a new push starts a fresh hold. Retention sweep reuses mergeRequestStateRetention.
  • HeadTimeAheadBy is set only in the skewed branch, so the new log lines in both gates fire only on actual skew.
  • TestMergeCIGate_SkewedHeadWithoutExpectSHAHoldsFromFirstObservation covers the stable first observation, release after one window, and re-hold on a new head.

No findings.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against src/pkg/github/ at the merge-base plus the watcher caller and the merge-state ledger. Looks correct to me.

  • The premise holds: merge_request_watcher.go:465 passes prState.PendingSince, and putMergeRequestState (merge_request_state.go:115) returns before writing when st.SHA == "", so a request without expect_sha genuinely never persists a wait state. Before this change the skew branch in EvaluateCommitCI (commit_ci.go:674) therefore saw a zero HeadFirstSeenAt and clamped to 0s on every poll.
  • mergeRequestHeadFirstSeen (merge_request_state.go:136) is keyed by repo#PR and replaced when the SHA differs, so a new head starts a fresh hold; it takes mergeStateMu like the rest of the ledger and reuses mergeRequestStateKey (already called under that lock at :107), so no new lock ordering. The gate only consults it when minHeadAge > 0 && firstSeen.IsZero() && expectSHA == "" (merge_ci_gate.go:203), so the persisted path for pinned requests is unchanged.
  • HeadTimeAheadBy is set only inside the skewed branch, so a normally aged head reports 0; both log sites gate on > 0.
  • TestMergeCIGate_SkewedHeadWithoutExpectSHAHoldsFromFirstObservation reproduces the 0s-every-poll bug (second poll with zero firstSeen must still read 5m) and clears after one window; the wantAheadBy additions cover [reviewer] Review backlog (style): Skew skip is silent; issue proposed a distinct reason/log #16147.
  • scale-envelope.md line-number bump matches the five lines added to client.go ahead of RankActionableIssues.

No findings. Body matches the diff.

Evidence bundle `#1615504038df` sha256:becf599fa707bfb2a59ae321c362f7f580b3536073b3373fbec87ca62890b6cd — download from the dashboard or `hivectl review evidence #16155`.

…lient.go line shift

check-docs-citations flagged src/docs/scale-envelope.md:37 because the
AgeMinutes sort in RankActionableIssues moved from client.go:5012-5022 to
5017-5027 on this branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 1, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review @ 9820c7f

The merge-request fix itself checks out: mergeRequestHeadFirstSeen (merge_request_state.go:138-158) is keyed per repo/PR, SHA-matched with EqualFold, guarded by mergeStateMu, and only consulted when firstSeen is zero and expectSHA is empty (merge_ci_gate.go:203-205), so the persisted-state path for pinned requests is unchanged. The new test (merge_ci_gate_test.go:1181-1215) exercises hold → clear → new-head re-hold.

1. The automerge skew log is unreachable, so the "both paths log it" claim is not delivered (medium, in-scope)
commitGreenForPRSkipping logs only when st.HeadTimeAheadBy > 0 (automerge_sweep_queued.go:368-371). But EvaluateCommitCI sets HeadTimeAheadBy only when now().Sub(opts.HeadPushedAt) < 0 (commit_ci.go:674-676), and the automerge path has already bounded HeadPushedAt to the host clock before calling: headPushedAtWith returns via hostClockBound (automerge_head_age.go:205,211,213), which returns headTime only if !headTime.After(now) and otherwise hostFirst/now, both ≤ now (automerge_head_age.go:219-227). The only unbounded return (automerge_head_age.go:198) requires minHeadAge <= 0, pinned.IsZero(), or a non-negative age ≥ min — none of which produce a skewed evaluation. So on the automerge sweep the skew is still absorbed silently inside headPushedAtWith, exactly the gap #16147 described, and the new log line is dead code (no test covers it, which is consistent). Fix: emit the skew log from headPushedAtWith when hostClockBound substitutes (e.g. compute aheadBy := headTime.Sub(at) there and c.info(...)), and drop or keep-as-defensive the branch at automerge_sweep_queued.go:368.

2. Advisory — mergeHeadFirstSeen entries set At only on insert and never refresh it on hits (merge_request_state.go:152-157), so a head polled continuously for > mergeRequestStateRetention (7d) is evicted on the next unrelated insert and re-held for one more window. Negligible in practice; noting only because the no-expect_sha path is also the one not bounded by mergeRequestMaxCIWait (watcher never tracks it).

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 1, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit f8d1c97 into v5 Nov 1, 2026
74 of 75 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-16147 branch November 1, 2026 10:57
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

0 participants