Skip to content

🐛 fix(automerge): a refused update-branch, merge or merge-forward is sent once per PR head, not again on every sweep - #17178

Merged
hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/automerge-once-per-head-ledgers
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/automerge-once-per-head-ledgers

Conversation

@Danathar

@Danathar Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

The queued, trusted-author and contributor merge lanes run every minute on a brand-new automerge engine. Several "do this once per PR head" marks lived on that engine, so they were empty again on every tick. On projectbluefin this re-sent a refused update-branch for fsdk-containers#508 about every 75 seconds. It also logged "once per head" holds on every sweep, about 15 PRs every 80 seconds. This PR moves those marks onto the long-lived GitHub client, the same idea as #17169, so every lane's engine shares them for the life of the process. It is the follow-up to #17169's finding and merges cleanly with it in either order. Refs #17169

What went wrong

SweepTrustedAuthorAutoMerges, SweepContributorLaneAutoMerges and SweepQueuedAutoMerges each call New(transport, opts). cmd/hive's runAutoMergeSweepIfDue calls them every minute. Only the self-authored loop keeps one engine alive, and that loop also runs the trusted and contributor lanes. So any per-head memo on the engine works for the long-lived loop but resets for the one-minute sweep.

Live evidence from the projectbluefin hive since the dd6b372 deploy (read with log_read, 2026-11-07 22:14Z to 2026-11-08 02:14Z):

  • Refused update-branch, repeated. automerge update-branch refused: the base changed workflow files and the head is in a fork the app cannot push them to for fsdk-containers#508: 11 times between 01:45Z and 02:11Z, each one preceded by head is behind the current base; updating before merge. Head and base were unchanged, so each line is one more refused PUT /pulls/508/update-branch (about 45 an hour). The memo that should stop it (branchUpdateWorkflowRefused, added for fix(ci): isolate invalid GNOME extension heads projectbluefin/utah#1001) was on the engine.
  • "Logged once per head", logged every sweep. merge train: PR left out of every train: 400 lines in 53 minutes, about 15 PRs every ~80 s (for example utah-packages#924, utah#1141, testsuite#1116). This is log volume only, with no API cost, but it buries real lines.
  • Latent, same mechanism, no hits since the deploy:
    • a merge GitHub refused with 405 (review required, method disabled, checks expected) is re-PUT every minute instead of every 30 minutes;
    • the trusted-author merge-forward (POST /merges, a push to a person's branch) is tried every sweep instead of once per head.

What changes

  • New Client.SharedState(name, build) in pkg/github: a process-wide slot other packages own and type.
  • New sweepMemory in automerge, fetched from that slot by New(). It holds:
    • the merge-forward attempt and its workflow refusal;
    • the update-branch workflow refusal;
    • the merge-refusal memory (mergeRefusals);
    • the logged-hold set (terminalHoldsLogged).
  • Idle once-marks are pruned after 24 h once the set is large. mergeRefusals keeps its own 30-minute recheck and prune.
  • The read-only IntegrationEligibility probe gets a private memory. A probe cannot write, so it must never mark a head as tried for the real sweep. A transport that keeps no shared state, such as the test fakes, keeps per-engine memory as before.
  • Not touched: the contributor held-run ledger, which is 🐛 fix(automerge): a refused held-run approval is tried once per PR head, not again on every sweep #17169's. contributor_held_runs.go and the heldRunApprovals field are left as they are, so the two PRs merge in either order. A trial merge of both branches builds and passes the automerge package.

Every cross-sweep field on the engine

Field What it gates Reset every tick? Live cost since 22:14Z Outcome
heldRunApprovals: update-branch workflow refusal PUT update-branch (write) Yes (trusted and contributor one-shot lanes) fsdk-containers#508: 11 refused PUTs in 26 min Fixed
heldRunApprovals: merge-forward attempt and workflow refusal POST /merges merge commit and conflict note (writes) Yes (trusted lane) No lines; latent Fixed
mergeRefusals PUT /pulls/N/merge after a 405, and the native-queue method refusal Yes (queued lane runs only one-shot; trusted and contributor both) No merge-refused lines; latent (#15419 pattern) Fixed
terminalHoldsLogged "Once per head" hold logs; OnTerminalHold hand-off (self lane only) Yes, for one-shot lanes About 450 "left out of every train" lines an hour Fixed
heldRunApprovals: @dependabot rebase nudge Comment No: only in trySweepSelfAuthoredPR, which runs on the long-lived loop. The comment is also deduped by the effects journal 0 lines Already harmless
heldRunApprovals: bot takeover Branch push and App PR No: self lane only 0 lines Already harmless
heldRunApprovals: contributor held runs POST actions/runs/{id}/approve Yes About 270 calls an hour #17169
unauthorized, unauthorizedLabelReady hive:unauthorized label and note Yes 188 lane unauthorized lines, all the #17169 case. Writes are durable: the label is read from the PR, and the note is upserted by its marker and the heads it covers Harmless here
workflowApprovalCommands, Scans, Untrusted, WorkflowChecks /ok-to-test comment and comment scans No: only reached from the self-authored sweep n/a Already harmless
humanReturnChecked GET label events once per head (stage labels) Yes Reads only; narrow case Left (read)
maintainerPerm GET collaborator permission (10 min TTL) Yes Reads only Left (read)
contributorMergedReposCache search/issues (1 h TTL) Yes Only with hive-wide auto-promotion; not in top consumers Left (read), follow-up candidate
closingNonPRTargets, closingCommitRefs, approvalReach, baseRefresh Read memos Yes Reads only Left (read)
nativeQueueProbes, inactiveQueueLast Native-queue probe and listing throttles Yes Reads; chairlift pilot only Left (read)
mergeTrain* (files, attempts, merged, logged, verdicts, seats) File lists, rebase loop guard, train order Yes The per-minute options carry no RebaseResolver, so one-shot engines never request rebases. Files are re-read Left (no writes)
selfSkipWarned, currentBaseFallbackWarned, requiredChecksFallbackWarned Warn-once logs Yes Logs only Left
evaluatedHeads Nothing (written, never read) n/a n/a Dead field, left
claimants Per-pass by design n/a n/a n/a

These are already process-wide through Options or package defaults: mergeQueueBases, contributorSoak, headFirstSeen, headSightings, sweepClock, the stage store and tracker, and base preparation.

Files

  • src/pkg/github/shared_state.go (new), src/pkg/github/client.go: the SharedState slot.
  • src/pkg/github/automerge/automerge_sweep_memory.go (new): sweepMemory and its once and logged sets.
  • automerge_sweep.go: engine fields, New() wiring, the private-memory option.
  • automerge_merge_forward.go, automerge_v6_lane.go, automerge_held_runs.go, automerge_merge_refusal.go: use the shared memory.
  • automerge_integration_eligibility.go: the probe keeps a private memory.
  • Tests: automerge_sweep_memory_test.go (new); automerge_contributor_lane_test.go fixture gains an update-branch status.
  • changelog.d/fixed-automerge-once-per-head-memory.md.

Tests

Each test builds one engine per sweep on one shared client, the way production does. The contributor-lane tests call the package-level SweepContributorLaneAutoMerges again and again.

Test On v6 tip With this PR
TestContributorLaneRefusedUpdateBranchNotRetriedAcrossSweeps Fails: 3 PUTs in 3 sweeps Passes: 1
TestContributorLaneRefusedMergeNotRetriedAcrossSweeps Fails: 3 merge PUTs Passes: 1, and 2 after the 30-minute recheck
TestMergeForwardTriedOncePerHeadAcrossSweeps Fails: 3 POSTs and 3 notes; 3 POSTs on the workflow refusal Passes: 1
TestCurrentBaseGateRefusedUpdateNotRetriedAcrossSweeps Fails: 2 PUTs by the second sweep Passes: 1, and 2 after the base moves
TestMergeTrainSkipLoggedOnceAcrossSweeps Fails: logged 3 times Passes: 1, and 2 on a new head
TestSweepMemorySharedPerClient Uses the new API, so it was not run on tip Passes: shared per client; the probe and other clients are separate

Full packages run: ./pkg/github/automerge/, ./pkg/github/, ./pkg/writeback/ and ./cmd/hive/ all pass. go vet is clean, check-docs-citations.py passes, and the changelog-fragment and DCO checks pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

The queued, trusted-author and contributor lanes run every minute on a
fresh engine (SweepTrustedAuthorAutoMerges, SweepContributorLaneAutoMerges,
cmd/hive runAutoMergeSweepIfDue). The marks meant to make a refused write
happen once per head lived on that engine, so they were empty on every
tick:

- a refused update-branch (fork head, base changed workflow files) was
  re-PUT every sweep (projectbluefin fsdk-containers#508, ~every 75 s);
- a merge refused with 405 was re-PUT every minute, not every 30;
- the trusted-author merge-forward was re-POSTed every sweep;
- "logged once per head" holds were logged on every sweep.

The marks now live in a sweepMemory kept on the long-lived GitHub client
(Client.SharedState) and shared by every engine built over it. A read-only
integration-eligibility probe gets a private memory so it never marks a
head as tried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 8, 2026
Danathar added a commit to Danathar/hive that referenced this pull request Nov 8, 2026
…run ledger

The integration eligibility probe builds its own engine with the live
options. With HeldRunApprovals unset, that engine picked up the process-wide
ledger. approveHeldRunsForRegistryTrusted claims a head before it calls the
approve API, and the probe's read-only transport refuses that call. So a
probe could mark a head as tried, and the real contributor-lane sweep would
never approve it. integrationReadOnlyOptions now gives the probe a private
ledger.

Found by the agent auditing hivecommons#17178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17130 ✨ feat(github): the hive files grouped issues from GitHub code scanning alerts (opt-in per repo) same files src/pkg/github/client.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v6 tree and the callers. The mechanism is sound and I found no defect; one low-priority consolidation note.

What I verified

  • cmd/hive/main_sweeps.go:937-954 passes the *github.Client itself as the Transport to SweepQueuedAutoMerges/SweepTrustedAuthorAutoMerges/SweepContributorLaneAutoMerges, so the transport.(interface{ SharedState(...) }) assertion in sharedSweepMemory (automerge_sweep_memory.go:56) succeeds in production and the one-shot engines really do share one sweepMemory. It is the same assertion pattern New already uses for MutationBoundary().
  • Locking is preserved: every former c.mergeRefusalsMu section now takes m.mergeRefusalsMu on the shared struct, and checksExpectedKnown is still only called under that lock from knownMergeRefusal (automerge_merge_refusal.go:188-200).
  • Pruning cannot drop a live mark: hasOnce/markOnce refresh the timestamp on every read, so a head that is still being swept is never idle; only marks untouched for 24h go, and only once the set reaches 1024 (automerge_sweep_memory.go:112-121).
  • integrationReadOnlyOptions sets isolatedSweepMemory, so New skips the shared memory and memory() lazily builds a private one — the probe cannot mark a head as tried.
  • The body's "merges cleanly with #17169 in either order" claim holds: git merge-tree pr17169 pr17178 produces a tree with no conflicts.

Note (low, not blocking)

After this and #17169 both land, github.Client will carry two sharing mechanisms for the same job: #17169 adds dedicated heldRunApprovals/installationPerms fields with a HeldRunApprovals() accessor (client.go:182-183 on that branch), while this PR adds the generic SharedState(name, build) slot (shared_state.go:19). Either is fine; having both is one more thing the next reader has to learn. Since both are yours, consider moving #17169's ledger onto SharedState in whichever PR lands second, or say in a comment why the held-run ledger deserves a typed field.

Open question, not a finding: a refused update-branch / merge-forward is now remembered for the process lifetime at that head (until a new head or restart), with no re-check if the workflows permission is later granted. That matches how the long-lived self loop already behaved and the "once per PR head" intent, so I'm only flagging that #17169 chose a permission-aware retry rule for approvals and this PR does not for pushes.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read against the tree at fa425ee. Looks correct to me.

  • The fix depends on sharedSweepMemory (automerge_sweep_memory.go:54-64) type-asserting SharedState on the transport. I checked the production call sites: runAutoMergeSweepIfDue hands *github.Client straight to SweepTrustedAuthorAutoMerges/SweepContributorLaneAutoMerges (src/cmd/hive/main_sweeps.go:851,954,972), and main.go:2481, merge_eligibility.go:85, review_preparation.go:32 likewise, so the shared slot is reached; base_preparation_visibility.go:6 passes nil, which falls back to a private memory via memory().
  • automerge_merge_refusal.go — every former c.mergeRefusals access is now m.mergeRefusals under m.mergeRefusalsMu; checksExpectedKnown is still only called from knownMergeRefusal with that lock held, as before.
  • automerge_integration_eligibility.go:47-49 isolates the read-only probe, matching what 🐛 fix(automerge): a refused held-run approval is tried once per PR head, not again on every sweep #17169 does for its ledger.

This is not a duplicate of #17169: that one owns the held-run approval ledger, this one owns update-branch/merge-forward/merge-refusal/hold-logged marks. They do touch the same hunk of New(); the author says both merge in either order.

Evidence bundle `#17178998a828` sha256:b79e8750fb41ffd23840f320799aae0fc2446b3be13edadc2a67a61c56e63ca4 — download from the dashboard or `hivectl review evidence #17178`.

hivecommons-hive Bot added a commit that referenced this pull request Nov 8, 2026
…ad, not again on every sweep (#17169)

* fix(automerge): approve a contributor's held runs once per head, not once per sweep

The contributor lane's once-per-head mark for held-run approvals lived on
the automerge engine, which SweepContributorLaneAutoMerges rebuilds every
tick. A head whose approvals GitHub refused with 403 (App without
actions: write) was therefore re-POSTed on every sweep: on projectbluefin
dakota-iso#307 and fsdk-containers#508 cost ~270 charged
POST actions/runs/{id}/approve calls an hour, all 403.

Move the pass and the refusal into a process-wide ledger on the GitHub
client (HeldRunApprovals). A refused head keeps reporting
unauthorized:approve-workflow-runs without another call until the head
changes or the installation gains actions: write (read via
InstallationPermission, cached 5 min). Only send runs GitHub's approve
endpoint accepts: fork pull_request runs and Actions-bot runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(automerge): drop an unused constant; docs name the contributor lane's grant retry

golangci-lint flagged contributorHeldRunsApprovalKeyPrefix as unused after
the ledger moved to the long-lived client. The hive reviewer pointed out
that github-app-setup.md still said a refused head needs Approve and run
after the grant. The contributor lane now retries a refused head once the
installation reads actions: write, so the doc says so. hive-merge.md also
notes that a ledger entry untouched for 24 hours is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(automerge): the read-only eligibility probe keeps a private held-run ledger

The integration eligibility probe builds its own engine with the live
options. With HeldRunApprovals unset, that engine picked up the process-wide
ledger. approveHeldRunsForRegistryTrusted claims a head before it calls the
approve API, and the probe's read-only transport refuses that call. So a
probe could mark a head as tried, and the real contributor-lane sweep would
never approve it. integrationReadOnlyOptions now gives the probe a private
ledger.

Found by the agent auditing #17178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* docs(scale-envelope): re-point the RankActionableIssues citation after client.go grew

The new held-run ledger fields on Client shifted RankActionableIssues by five
lines, so the docs citation check reported drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive
hivecommons-hive Bot merged commit 78cf72c into hivecommons:v6 Nov 8, 2026
67 of 68 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 99b5502.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant