Skip to content

[scanner] merge gate: hive host clock is 1h behind GitHub since the DST fall-back; min_head_age clamps every fresh head to "0s ago" and stalls relayed merges #16141

Closed

Description

@hivecommons-hive

Symptom

Every hive-merge request this kick on a freshly opened PR is refused with

ci gate: CI still running (pending: head pushed 0s ago (< min_head_age))

and keeps being refused on every retry (ci_waits 9+) even though the PR is CLEAN, all checks are green, and the head commit is 17–24 minutes old (docs#560 head e028429 committer date 08:11:39Z; homebrew-hive#214 head 08:04:18Z). docs#556, whose head was created hours earlier, merged on the first attempt.

Cause: the hive host clock is one hour behind GitHub since the US DST fall-back

Host-side timestamps versus GitHub's server-side timestamps for the same event:

event host clock (bead / merge-request at) GitHub created_at / merged_at
docs#553 opened 05:31:09Z 05:30:25Z (agree)
docs#555 opened 05:57:11Z 05:56:26Z (agree)
docs#560 opened 08:02:13Z 09:01:27Z (+1h)
docs#556 merged 08:24:57Z 09:23:23Z (+1h)
docs#561 opened 08:28:03Z 09:28:07Z (+1h)

The step happened between 05:57Z and 08:02Z host time — i.e. at 07:00Z, when US daylight time ended today (2026-11-01, 02:00 CDT). The host clock went back one hour and stayed there; date -u on the pod now reads 08:29Z while GitHub stamps the same instant 09:29Z. (The Date header seen through the egress proxy matches the host, so it is not usable as a sanity check from inside the pod.)

Why the gate says "0s ago"

src/pkg/github/merge_ci_gate.go measures head age as now.Sub(headPushedAt) with headPushedAt = PR updated_at, or the head commit's committer date clamped to no earlier than PR created_at (HeadCommitPushedAt, commit_ci.go). Both created_at and updated_at come from GitHub's clock, so for an hour after a PR is opened — or after every label/comment write that bumps updated_at — they lie in the host's future. EvaluateCommitCI then does if age < 0 { age = 0 } and reports head pushed 0s ago, which is indistinguishable from a genuinely fresh push. Nothing logs the skew.

The same skew silently affects every other wall-clock comparison the hive makes against GitHub times (soak windows, stage-age timers, lease/claim expiry, hive/awaiting-ci ages) and anything the hive signs with a timestamp window (the NPS relay rejects signatures more than 5 minutes off).

Decision requested

  • A. Owner-side: resync the hive node/pod clock (chrony/NTP, or the VM host's time source) and check whether the DST step came from the node or the sandbox. Nothing an agent can run fixes this.
  • B. Code-side, in addition to A: make the gate robust to skew — when now.Sub(headPushedAt) is negative by more than a tolerance (say 30s), log a clock skew warning with both timestamps and measure age from the GitHub Date response header (or treat the head as old enough once the configured required checks are all green, which configRequiredAllGreen already does for repos with configured required checks). Report pending: head pushed 0s ago only when the age is genuinely near zero.
  • C. Something else.

Decision taken by hive: none for A — the clock is owner-only infrastructure. B is additive and can be dispatched once A is acknowledged.

Risk: owner-only infrastructure (host clock / NTP).

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    agent/scannerCreated or modified by the scanner agent
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Nov 1, 2026
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 1, 2026
  3. hivecommons-hive commented on Nov 1, 2026

    @hivecommons-hive
    ContributorAuthor

    What to reply

    • /hive approve: go with the recommendation
    • /hive decision <your instructions>: go ahead, following your instructions
    • /hive help: show these again

    If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

    Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

  4. clubanderson commented on Nov 2, 2026

    @clubanderson
    Member

    /hive decision Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.

  5. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Nov 2, 2026
  6. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    Un-parked by @clubanderson. Decision: Proceed with both operational diagnosis and additive code hardening, without weakening min_head_age or merge gates. First measure current UTC skew at the actual Hive pod/node against independently trusted time; DST does not normally change UTC, so do not treat the reported cause as proven. Have the infrastructure operator repair NTP/chrony or the VM time source using the normal platform procedure if skew is confirmed; do not set a pod clock or blindly step shared-host time. In code, explicitly report excessive future timestamps with both instants and add negative-age regression coverage. Keep genuinely unknown/future head age fail-closed; do not bypass the age requirement merely because checks are green, and do not trust a proxy Date header without validation. Code work need not wait for the operational diagnosis. Owner-delegated decision from the November 2 request; infrastructure execution and recovery still require actual evidence.

    stage/045-needs-decision is cleared when present and approved-direction is applied; any hold label is left exactly as it was.

  7. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    🔒 Claimed by agent scanner (agent on hive hosted-available-oke-11-placeholder-r05x). The claim lasts 2h0m0s without activity and renews while the work continues. Reply /unclaim to release, or /claim to take it over if you outrank the holder.

  8. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    Already addressed on v5 by #16146 (a head time ahead of the host clock is treated as skew, not a fresh push; age is measured from first observation, fail-closed, never bypassed when green) and #16155 (also holds skewed heads without expect_sha and logs head time and ahead_by in the merge gate and sweep). Regression coverage is in commit_ci_test.go (wantAheadBy cases). Remaining work is operational: confirm and repair host NTP/chrony skew per the decision above. Closing the code side.


    🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: b73ee1964

    — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

  9. hivecommons-hive commented on Nov 2, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by agent scanner released (holding agent relaunched).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions