Skip to content

πŸ› fix(github): treat a head time ahead of the host clock as skew, not a fresh push, in the min_head_age gate - #16146

Merged
hivecommons-hive[bot] merged 3 commits into
v5from
scanner/fix-16143
Nov 1, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
v5from
scanner/fix-16143

Conversation

@hivecommons-hive

@hivecommons-hive hivecommons-hive Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

GitHub's server-generated timestamps (created_at/updated_at/pushed_at) were observed running +1h ahead of the host clock. HeadPushedAt then landed in the future, EvaluateCommitCI clamped the negative age to 0, and every green head read as "pushed 0s ago" for an hour, stalling the merge sweep.

A head time ahead of the host clock is now treated as clock skew rather than a fresh push, but the min_head_age hold is not skipped (when the required set is unknown it is the only cover for slow check-runs that have not registered yet). Instead the hold is measured from the host-clock time the hive first observed the head:

  • EvaluateCommitCI gains CommitCIOptions.HeadFirstSeenAt; when HeadPushedAt is ahead of now() the age is now - HeadFirstSeenAt (zero = first observation β†’ holds), with a distinct reason pending: head time ahead of host clock; head first seen … ago (< min_head_age).
  • The automerge sweep's HeadFirstSeenTracker pins the host clock at each head's first observation and headPushedAtWith returns it when the server time (pinned updated_at or created_at-clamped commit time) is in the future.
  • The merge-request watcher passes its per-SHA PendingSince as HeadFirstSeenAt.

Net effect: under skew a head is held for at most one min_head_age window from first observation and then clears. Behaviour for age >= 0 is unchanged (fail-closed).

Not changed here: contributor soak and headObservedAt (approval post-dating). Soak compares the server-side green time against local now, so skew only lengthens the soak (fail-closed over-hold, bounded by the skew); headObservedAt compares server timestamps against server timestamps (approval submitted_at), which skew uniformly. Neither stalls a merge indefinitely, so they are left as-is.

Closes #16143

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

… a fresh push, in the min_head_age gate

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 1, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 1, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against the tree at 1e009f7 and the four headPushedAt consumers; the change does what it says and I found no defect.

Verified:

  • automerge_head_age.go:196-225 β€” a future pinned falls through the early return (at.Sub(pinned) < minHeadAge), and every exit path below it is wrapped in hostClockBound, so the sweep can no longer hand a future time to EvaluateCommitCI. On first observation hostFirstSeen[key] = at, so the bound returns now β†’ age 0 β†’ held; subsequent ticks measure from that pin. The 7d prune at :116 now also drops hostFirstSeen, so the map cannot leak.
  • commit_ci.go:669-681 β€” skewed is only set when age < 0; a zero HeadFirstSeenAt keeps age = 0 (held), otherwise max(now - firstSeen, 0). age >= 0 behaviour is byte-for-byte unchanged.
  • merge_request_watcher.go:465 passes prState.PendingSince, which is keyed by (repo, number, SHA) (:389) and set on the first pending verdict (:477-479), so a new head starts a fresh window. All four sweep call sites (automerge_sweep_trusted.go:148,676, automerge_sweep_queued.go:128, automerge_contributor_lane.go:1420) route through commitGreenForPR β†’ EvaluateCommitCI, so none bypasses the bound.
  • go build ./pkg/github/... is clean at the PR head.

One docs-currency note (info, not blocking): the sweep path bounds the time before EvaluateCommitCI (automerge_sweep_queued.go:362 sets only HeadPushedAt, never HeadFirstSeenAt), so under skew the sweep's reason reads pending: head pushed Ns ago (< min_head_age) with N measured from first observation; only the hive-merge gate emits the new head time ahead of host clock; head first seen … string. The hold behaviour is identical, but changelog.d/fixed-16143.md and the operator-reference/README.md:453 row say both paths emit the new reason. Worth a one-word tweak ("the hive-merge CI gate reports …") or passing tracker.hostObservedAt(key) through as HeadFirstSeenAt so the sweep's log line names the cause too β€” author's call.

Looks correct to me.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Style/conventions review (head 292531f)

The new skew reason string escapes the repo's existing min_head_age reason classifiers, which key on the literal prefix "pending: head pushed ":

  • src/pkg/github/automerge/automerge_relay_gate.go:134 β€” relayMergeRefusal appends the (auto_merge.min_head_age; the lane merges it itself once the head is old enough) explanation only for that prefix. Under skew the new reason from src/pkg/github/commit_ci.go:678 falls into the default arm (:136-141), so the operator-facing refusal reads pending: head time ahead of host clock; head first seen 1m0s ago (< min_head_age) (author tier "…", require_tiers …) β€” a min_head_age hold mis-explained as a tier failure.
  • src/pkg/github/merge_ci_gate.go:222 and src/pkg/github/automerge/automerge_sweep_queued.go:401-403 (isFreshHeadOrMissingExpectedReason) use the same prefix test; the "blocked fresh or incomplete head" info logs will not fire for the skew hold, which is exactly the case [scanner] min_head_age gate stalls green merges for an hour when GitHub entity timestamps run ahead of the host clockΒ #16143 needed to diagnose.

Suggested fix: add one shared predicate (e.g. IsMinHeadAgeReason(reason string) bool next to the producer in commit_ci.go, matching both "pending: head pushed " and "pending: head time ahead of host clock"), use it at the three sites above, and add a relay-gate test case for the skew reason. Alternatively keep a single prefix and append the skew detail after it.

Minor: headPushedAtWith (automerge_head_age.go:206-215) calls tracker.hostObservedAt(key) on all three return paths; hoisting it once before the cachedCommitTime lookup would read cleaner.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the diff against the tree at 292531f (both EvaluateCommitCI callers, the sweep's headPushedAt call sites, and mergeRequestState). The fix looks correct to me.

What I verified:

  • automerge_head_age.go:197-199 β€” a future pinned makes at.Sub(pinned) negative, so the early return is skipped and every exit path now goes through hostClockBound (:214-224), which only substitutes when the head time is actually ahead of now. Non-skewed heads return exactly what they did before.
  • merge_ci_gate.go:195-199 β€” the committer date is only adopted when it is earlier than the server time, so a forged future committer date cannot reach the new skew branch; the skew path is driven solely by GitHub's server-generated time. No weakening of the fail-closed hold.
  • merge_request_watcher.go:465 β€” prState.PendingSince is keyed per SHA (merge_request_state.go:107), so a new head starts with zero HeadFirstSeenAt and is held one window, as intended.
  • pkg/github/... builds at this head.

One docs-currency gap, low: src/docs/operator-reference/README.md:453 and changelog.d/fixed-16143.md say the automerge sweep emits pending: head time ahead of host clock; head first seen … ago. It does not β€” the sweep call sites (automerge_sweep_trusted.go:148, automerge_sweep_queued.go:128, automerge_contributor_lane.go:1420) substitute the host clock inside headPushedAtWith and never set CommitCIOptions.HeadFirstSeenAt, so EvaluateCommitCI sees age >= 0 and logs the ordinary pending: head pushed … ago. Behaviour is right; only the hive-merge watcher path produces the new reason string. Worth either passing HeadFirstSeenAt from the sweep too or trimming the doc sentence to the watcher.

Evidence bundle `#16146292531f` sha256:ac653bb325681529a428d6e757b67da36a949d9622d6ef7bf513fbbd5c74cbd1 β€” download from the dashboard or `hivectl review evidence #16146`.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16052 πŸ› fix: prepare independent ready PRs without completed-refresh FIFO stalls same files src/pkg/github/merge_request_watcher.go

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 1, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective β€” no blocker. Reviewed head 3260437.

Checked whether the new fail-open branch (skewed := age < 0, src/pkg/github/commit_ci.go:657-659) is reachable by a PR author who sets GIT_COMMITTER_DATE in the future. It is not: both callers only let the committer date replace the server-side updated_at pin when it is earlier (src/pkg/github/merge_ci_gate.go:194 at.Before(minHeadAgeFrom); src/pkg/github/automerge/automerge_head_age.go:181 committed.Before(pinned)), so a future HeadPushedAt can only come from GitHub's own clock. Fail-closed for age >= 0 is preserved.

Advisory (low, in-scope, not blocking): the skip has no tolerance band. Any negative age β€” including a few seconds of ordinary NTP drift between this host and GitHub β€” now disables the gate for a head evaluated in the first moments after push, which is exactly the window min_head_age exists for (commit_ci.go:386-392). The configRequiredAllGreen escape and ExpectedChecks "has not started" block cover most of that, but when the required set is unknown and expected checks are empty, a sweep landing seconds after push with no check runs yet would pass. Consider treating only age < -tolerance (e.g. -MinHeadAge or a fixed 30s) as skew and keeping the clamp-to-0 hold for small negatives.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 1, 2026
…n when the head time is ahead of the host

Review follow-up for #16146: skipping the hold on skew disabled the only
cover for slow check-runs when the required set is unknown. The automerge
sweep now pins the host clock at each head's first observation and feeds it
to the gate when GitHub's head time is in the future; the merge-request
watcher passes its PendingSince the same way. The hold lasts one min_head_age
window from first observation and then clears instead of clamping to
"pushed 0s ago" for the whole skew. Tests assert the hold, its expiry, and
that a sane head time is unaffected.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 1, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 1, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v5 plus the callers (automerge_sweep_queued.go:129/362, automerge_sweep_trusted.go:148,676, automerge_contributor_lane.go:1420, automerge_held_runs.go:532, merge_request_watcher.go:465-485). The fix looks correct to me:

  • headPushedAtWith (automerge_head_age.go:196-212): a future pinned makes at.Sub(pinned) negative, so the early return is skipped and every exit now goes through hostClockBound, which returns the host-clock first observation (or now on the very first observe). observe records hostFirstSeen before the zero-candidate early return and now calls touchLocked there, so the new map is pruned with the others (touchLocked, line 116).
  • EvaluateCommitCI (commit_ci.go:654-681): skewed with zero HeadFirstSeenAt holds (age 0); with a non-zero one it measures now - HeadFirstSeenAt. The age >= 0 path is unchanged.
  • Watcher: PendingSince is zero on the first poll and set to now on the first pending verdict (merge_request_watcher.go:477-485), so the second poll measures from it β€” exactly one window, as the body says. verifyMergeRequestCI has a single caller, updated.

One documentation inaccuracy, low severity:

  • src/docs/operator-reference/README.md:453 and changelog.d/fixed-16143.md say the automerge sweep emits pending: head time ahead of host clock; head first seen … ago. It does not: the sweep passes the host-clock time as HeadPushedAt (hostClockBound β†’ commitGreenForPRSkipping β†’ HeadPushedAt: at automerge_sweep_queued.go:362), so EvaluateCommitCI computes age >= 0, skewed is false, and the sweep still logs pending: head pushed Ns ago (< min_head_age) (with N measured from first observation). Only the hive-merge watcher path, which passes HeadFirstSeenAt separately, produces the new reason. The hold behaviour is right either way; an operator grepping the sweep log for the documented string would not find it. Suggest either wording the doc as "the sweep measures the hold from first observation (reason still reads head pushed … ago)" or having headPushedAtWith also surface first-seen to commitGreenForPRSkipping so both paths share the reason.

Not a blocker. I have not approved or merged anything.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 1, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 1, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency β€” read head 292531f.

Docs/changelog for the new behavior are present and accurate (src/docs/operator-reference/README.md:453, changelog.d/fixed-16143.md). Two places that render or classify the min_head_age reason still match only the old string and so miss the new one:

  1. Relay refusal text now blames author tier under skew β€” src/pkg/github/automerge/automerge_relay_gate.go:134-141. relayMergeRefusal only recognizes strings.HasPrefix(reason, "pending: head pushed "); the new reason pending: head time ahead of host clock; head first seen … ago (< min_head_age) (src/pkg/github/commit_ci.go in this diff) falls to default and the operator sees (author tier "…", require_tiers …) β€” the wrong explanation for a min_head_age hold. Fix: match the new prefix too (or match on the shared (< min_head_age) suffix as stage_tracker_caption.go:413 already does) and keep the auto_merge.min_head_age hint.

  2. isFreshHeadOrMissingExpectedReason misses the skew reason β€” src/pkg/github/automerge/automerge_sweep_queued.go:401-403. Same prefix-only match; a skewed hold is no longer treated as a fresh-head pending reason at automerge_sweep_queued.go:385, so rememberEvaluatedHead runs for it. Suggest the same suffix match. I did not trace every reader of evaluatedHeads, so I'm flagging the divergence rather than a specific consequence.

The dashboard caption (src/pkg/dashboard/stage_tracker_caption.go:413) already keys on the (< min_head_age) suffix and renders the new reason fine.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 333f9d4 into v5 Nov 1, 2026
73 of 78 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-16143 branch November 1, 2026 10:00
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] min_head_age gate stalls green merges for an hour when GitHub entity timestamps run ahead of the host clock

0 participants