You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[scanner] merge gate: hive host clock is 1h behind GitHub since the DST fall-back; min_head_age clamps every fresh head to "0s ago" and stalls relayed merges #16141
Every hive-merge request this kick on a freshly opened PR is refused with
ci gate: CI still running (pending: head pushed 0s ago (< min_head_age))
and keeps being refused on every retry (ci_waits 9+) even though the PR is CLEAN, all checks are green, and the head commit is 17–24 minutes old (docs#560 head e028429 committer date 08:11:39Z; homebrew-hive#214 head 08:04:18Z). docs#556, whose head was created hours earlier, merged on the first attempt.
Cause: the hive host clock is one hour behind GitHub since the US DST fall-back
Host-side timestamps versus GitHub's server-side timestamps for the same event:
event
host clock (bead / merge-request at)
GitHub created_at / merged_at
docs#553 opened
05:31:09Z
05:30:25Z (agree)
docs#555 opened
05:57:11Z
05:56:26Z (agree)
docs#560 opened
08:02:13Z
09:01:27Z (+1h)
docs#556 merged
08:24:57Z
09:23:23Z (+1h)
docs#561 opened
08:28:03Z
09:28:07Z (+1h)
The step happened between 05:57Z and 08:02Z host time — i.e. at 07:00Z, when US daylight time ended today (2026-11-01, 02:00 CDT). The host clock went back one hour and stayed there; date -u on the pod now reads 08:29Z while GitHub stamps the same instant 09:29Z. (The Date header seen through the egress proxy matches the host, so it is not usable as a sanity check from inside the pod.)
Why the gate says "0s ago"
src/pkg/github/merge_ci_gate.go measures head age as now.Sub(headPushedAt) with headPushedAt = PR updated_at, or the head commit's committer date clamped to no earlier than PR created_at (HeadCommitPushedAt, commit_ci.go). Both created_at and updated_at come from GitHub's clock, so for an hour after a PR is opened — or after every label/comment write that bumps updated_at — they lie in the host's future. EvaluateCommitCI then does if age < 0 { age = 0 } and reports head pushed 0s ago, which is indistinguishable from a genuinely fresh push. Nothing logs the skew.
The same skew silently affects every other wall-clock comparison the hive makes against GitHub times (soak windows, stage-age timers, lease/claim expiry, hive/awaiting-ci ages) and anything the hive signs with a timestamp window (the NPS relay rejects signatures more than 5 minutes off).
Decision requested
A. Owner-side: resync the hive node/pod clock (chrony/NTP, or the VM host's time source) and check whether the DST step came from the node or the sandbox. Nothing an agent can run fixes this.
B. Code-side, in addition to A: make the gate robust to skew — when now.Sub(headPushedAt) is negative by more than a tolerance (say 30s), log a clock skew warning with both timestamps and measure age from the GitHub Date response header (or treat the head as old enough once the configured required checks are all green, which configRequiredAllGreen already does for repos with configured required checks). Report pending: head pushed 0s ago only when the age is genuinely near zero.
C. Something else.
Decision taken by hive: none for A — the clock is owner-only infrastructure. B is additive and can be dispatched once A is acknowledged.
/hive decision <your instructions>: go ahead, following your instructions
/hive help: show these again
If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.
/hive decision Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.
Un-parked by @clubanderson. Decision: Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.
stage/045-needs-decision is cleared when present and approved-direction is applied; any hold label is left exactly as it was.
🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.
Already addressed on v5 by #16146 (a head time ahead of the host clock is treated as skew, not a fresh push; age is measured from first observation, fail-closed, never bypassed when green) and #16155 (also holds skewed heads without expect_sha and logs head time and ahead_by in the merge gate and sweep). Regression coverage is in commit_ci_test.go (wantAheadBy cases). Remaining work is operational: confirm and repair host NTP/chrony skew per the decision above. Closing the code side.
Activity
hivecommons-hive commentedon Nov 1, 2026
What to reply
/hive approve: go with the recommendation/hive decision <your instructions>: go ahead, following your instructions/hive help: show these againIf what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply
/fixedto close it as completed; Hive clears its park labels.Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.
clubanderson commentedon Nov 2, 2026
/hive decision Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.
hivecommons-hive commentedon Nov 2, 2026
Un-parked by @clubanderson. Decision: Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.
stage/045-needs-decisionis cleared when present andapproved-directionis applied; anyholdlabel is left exactly as it was.hivecommons-hive commentedon Nov 2, 2026
🔒 Claimed by agent
scanner(agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply/unclaimto release, or/claimto take it over if you outrank the holder.hivecommons-hive commentedon Nov 2, 2026
Already addressed on v5 by #16146 (a head time ahead of the host clock is treated as skew, not a fresh push; age is measured from first observation, fail-closed, never bypassed when green) and #16155 (also holds skewed heads without expect_sha and logs head time and ahead_by in the merge gate and sweep). Regression coverage is in commit_ci_test.go (wantAheadBy cases). Remaining work is operational: confirm and repair host NTP/chrony skew per the decision above. Closing the code side.
🐝 Hive Agent:
scanner| Instance:hosted-available-oke-11-placeholder-r05x| SHA:b73ee1964— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Nov 2, 2026
🔓 Claim by agent
scannerreleased (holding agent relaunched).