Repository navigation
🐛 fix(automerge): a refused held-run approval is tried once per PR head, not again on every sweep - #17169
Conversation
…once per sweep
The contributor lane's once-per-head mark for held-run approvals lived on
the automerge engine, which SweepContributorLaneAutoMerges rebuilds every
tick. A head whose approvals GitHub refused with 403 (App without
actions: write) was therefore re-POSTed on every sweep: on projectbluefin
dakota-iso#307 and fsdk-containers#508 cost ~270 charged
POST actions/runs/{id}/approve calls an hour, all 403.
Move the pass and the refusal into a process-wide ledger on the GitHub
client (HeldRunApprovals). A refused head keeps reporting
unauthorized:approve-workflow-runs without another call until the head
changes or the installation gains actions: write (read via
InstallationPermission, cached 5 min). Only send runs GitHub's approve
endpoint accepts: fork pull_request runs and Actions-bot runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Docs-currency review of head 1681836. The touched docs (hive-merge.md:884-899, operator-reference/README.md:548, the changelog fragment) match the code: approvableHeldRun (automerge_contributor_held_runs.go) sends only fork pull_request / Actions-bot runs, and the "at most every five minutes" re-read is workflowsGrantTTL = 5 * time.Minute (src/pkg/github/issue_workflows_handoff.go:166). Advisory only; nothing blocking.
src/docs/github-app-setup.md:46 does not carry the new retry rule. That paragraph is where an operator goes after granting Actions: Read and write. For the gated-run sweep it says a refused head is not retried after the grant ("takes effect for new heads (or after a hive restart), while an already-refused head needs Approve and run"), then describes the contributor-lane path's 403 outcome without saying it now behaves differently: HeldRunApprovals.ReleaseOnGrant (held_run_approvals.go) re-arms a refused head once the installation reads actions: write, so no click or restart is needed for that lane. An operator reading only this file will click Approve and run unnecessarily, or expect a retry that the sweep path does not do.
… logs `lane unauthorized` once per head, parks the PR at 085 (`Awaiting human: approve workflow runs` in the activity monitor), does not POST that head again until it changes or the installation gains **Actions: Read and write** (unlike the gated-run sweep above, the contributor lane re-reads the grant at most every five minutes and tries a refused head once more when it appears; a restart also tries each head once more), and clears all of it the tick the required checks report ([#14589](https://github.com/hivecommons/hive/issues/14589)).
Minor (info): hive-merge.md:896-897 says the pass and refusal are "kept for the life of the hive process", while HeldRunApprovals.lookupLocked evicts an entry untouched for heldRunApprovalTTL (24 h). A swept PR refreshes seen every tick so this only matters for a PR not visited for a day (e.g. parked behind an exclude_labels label and then unlabelled), which then gets one more pass. Worth a parenthetical ("an entry no sweep has touched for 24 h is dropped") if you touch the paragraph again.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Correctness review at head Low — the one "retry on grant" pass can be burned by a stale installation token, then never retried for that head. type heldRunApprovalEntry struct {
head string
refused bool
+ refusedAt time.Time
refusedRuns []int64
grant string
grantKnown bool
seen time.Time
}
...
entry.refused = true
+ entry.refusedAt = time.Now()
...
- if !ok || !entry.refused || (entry.grantKnown && permissionAllowsWrite(entry.grant)) {
+ // A refusal recorded while the grant already read write may have gone
+ // out on an installation token minted before the grant (tokens live ~1h);
+ // allow one more pass per token lifetime instead of never.
+ if !ok || !entry.refused || (entry.grantKnown && permissionAllowsWrite(entry.grant) && time.Since(entry.refusedAt) < time.Hour) {
return false
}( Review findings from Read the diff against v6 One thing that will likely stop CI:
Open question, not a defect: Review findings from Read the ledger and the lane against the tree at
One note for whoever merges: #17178 (same author) edits the same region of |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
…ne's grant retry golangci-lint flagged contributorHeldRunsApprovalKeyPrefix as unused after the ledger moved to the long-lived client. The hive reviewer pointed out that github-app-setup.md still said a refused head needs Approve and run after the grant. The contributor lane now retries a refused head once the installation reads actions: write, so the doc says so. hive-merge.md also notes that a ledger entry untouched for 24 hours is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Both points are fixed in 3d6cccb:
The docs citation and link checks pass, and so do the automerge held-run tests. |
…run ledger The integration eligibility probe builds its own engine with the live options. With HeldRunApprovals unset, that engine picked up the process-wide ledger. approveHeldRunsForRegistryTrusted claims a head before it calls the approve API, and the probe's read-only transport refuses that call. So a probe could mark a head as tried, and the real contributor-lane sweep would never approve it. integrationReadOnlyOptions now gives the probe a private ledger. Found by the agent auditing hivecommons#17178. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
One more fix in c3985f0, found while auditing #17178. The read-only integration eligibility probe was using the process-wide held-run ledger. |
…r client.go grew The new held-run ledger fields on Client shifted RankActionableIssues by five lines, so the docs citation check reported drift. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
There was a problem hiding this comment.
Read the diff against the v6 tree alongside #17178 (same files, same one-shot-engine root cause). The ledger logic is sound; I found no defect. One red check and two notes.
What I verified
HeldRunApprovals.Claim(held_run_approvals.go:94-107) replaces the PR's entry when the head differs, so a push re-arms the pass;lookupLockednever evicts an entry on a stale-head read, only onClaim, so a late sweep for an old head cannot wipe the current head's record.ReleaseOnGrant(held_run_approvals.go:139-152) cannot loop: it only releases when the grant is nowwrite/adminand the refusal was recorded with the grant unknown or below write. The retry's second refusal is recorded viaRecordRefusalwithgrantKnown=true, write, after which theentry.grantKnown && permissionAllowsWrite(entry.grant)guard holds.- In
approveHeldRunsForRegistryTrusted(automerge_contributor_held_runs.go:80-92) both theRefusedandAttemptedchecks sit before the workflow-run listing, so a remembered head costs no API call per sweep, matching the pre-change ordering.actionsPermissiongoes throughInstallationPermission's cache, so the per-sweep permission read on a refused head is also free inside the TTL. - Every ledger method is nil-receiver safe, and
Newfalls back to a private&HeldRunApprovals{}when the transport has none, so engines in existing tests keep their old per-engine behaviour. integrationReadOnlyOptionshands the probe a private ledger — a read-only evaluation cannot consume the real sweep's one pass.git merge-treeof this head against #17178's shows no conflicts, as #17178's body claims.
Red check: relative links and anchors resolve fails on c3985f0. I could not pin the link from here; the three touched docs (github-app-setup.md, hive-merge.md, operator-reference/README.md) are where to look.
Note (low): with #17178 this PR gives github.Client a typed heldRunApprovals field + HeldRunApprovals() accessor (client.go:182-183) while #17178 adds a generic SharedState(name, build). Both solve the same problem; worth putting the ledger on SharedState in whichever lands second (I left the mirror note on #17178).
Open question, not a finding: approvableHeldRun (automerge_contributor_held_runs.go:235-247) returns false when the run payload has no head_repository.full_name. Before this change such a run was POSTed; now it is silently left for a person. If GitHub ever omits head_repository on a fork pull_request run that is the behaviour you want? If so a DEBUG line naming the skipped run would help the next operator who asks why a head was never approved.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thanks.
|
Summary
The contributor lane approves a registry-trusted contributor's held fork runs "once per head", but the mark that enforced that lived on the automerge engine, and the sweep builds a new engine every tick. So when GitHub refused the approve with 403 (the App has no
actions: write), the next sweep tried again, and the one after that. On projectbluefin two PRs spent about 270 charged calls an hour onPOST actions/runs/{id}/approve, every one a 403. This PR moves the pass and the refusal into a process-wide ledger on the GitHub client, so a refused head is tried once and then reported as the PR's blocker until the head changes or the App gainsactions: write. It also stops sending runs GitHub's approve endpoint can never accept.What went wrong
Live hive
hosted-projectbluefin-knuckle(build dd6b372), 2026-11-08:14fdfdd) had three held forkpull_requestruns fromkylerankin/dakota-iso(Lint, Test, Smoke Test). projectbluefin/fsdk-containers#508 (headc00801f) had three fromKaribhou/fsdk-containers(skill-catalog, image-catalog, Build images).lane unauthorized … step=approve-workflow-runsfor the same head on every sweep: 61 times between 00:40:50Z and 01:44:16Z, roughly every 75–90 s per PR. Each one is a pass that POSTed all three runs and got403 Resource not accessible by integration. That is about 183 POSTs in the hour, or 6 per sweep at ~45 sweeps an hour, about 270 an hour. Another session measured 174 charged approve calls withlast_status403 underhive:contributor_lane_automerge_sweepintop_consumersfor the same period.lane unauthorized cleared). dakota-iso#307 then merged.Cause:
SweepContributorLaneAutoMergesrunsNew(transport, opts)every tick. The once-per-head map (heldRunApprovals) and the unauthorized state were fields on that throwaway engine, so each tick started empty. The existing tests ran two ticks on one engine, which hid this.The first report blamed Prow
pull_request_reviewruns (dakota-iso run 42341060495). That part was wrong. This path lists runs withevent=pull_requestand keeps only runs from the PR's head repository, so that run (head repositoryprojectbluefin/dakota-iso) was never sent. The refused runs were real fork runs. The App simply lacks the permission.What changes
hgithub.HeldRunApprovalslives on the long-livedClientand keeps one entry per PR head: whether a pass ran, and, if GitHub refused, which runs it refused and what the App'sactionsgrant read at the time. The engine gets it from the transport.Options.HeldRunApprovalscan override it.unauthorized:approve-workflow-runs, so it stays at 085 with its label and its one existing note. There is no new comment, no listing and no POST on later sweeps. The lane tries again only on a new head, or when the installation now grantsactions: writeand did not at refusal.Client.InstallationPermissionreads that grant at most every 5 minutes, through the same probe the workflows-grant check uses. A restart tries each head once more.approvableHeldRunmatches GitHub's rule: apull_requestrun whose head repository is not the base repository, or a run the Actions bot queued. Same-repository runs andpull_request_review/pull_request_targetruns are left for a person, because GitHub answers them with 403 forever. The fork test compares repository names rather than readinghead_repository.fork, which istruefor any repository that is itself a fork (projectbluefin/dakota-isois).Files
src/pkg/github/held_run_approvals.go(new): the ledger andInstallationPermission.src/pkg/github/client.go: two fields.src/pkg/github/automerge/automerge_sweep.go: options and wiring.src/pkg/github/automerge/automerge_contributor_held_runs.go: uses the ledger, adds the run filter, removes the engine-local once-per-head helpers.hive-merge.mdandoperator-reference/README.mddescribe the retry rule and the fork-only filter.changelog.d/fixed-held-run-approve-once.md.Tests
New. Each failed on v6 tip (9975c5d) and passes here:
TestContributorLaneRefusedHeldRunsNotRetriedAcrossSweepsbuilds one engine per sweep on one transport, as production does. Tip:approve POSTs = [11 12 11 12 11 12 11 12]over 4 sweeps. Now[11 12], with one note and one label.TestContributorLaneRefusedHeldRunsRetryOnGrantOrNewHead: same head, same grant, no retry (tip: 4 POSTs after the second pass). Gainingactions: writeretries once. A new head retries once. A fork run GitHub accepts is approved.TestContributorLaneApprovesOnlyForkPullRequestHeldRuns: apull_request_revieworpull_request_targetrun in the listing is never approved, while the forkpull_requestrun and an Actions-bot run are (tip:[11 21 22 24 11 21 22 24]).TestApprovableHeldRun,TestHeldRunApprovalsOncePerHeadAndRefusal,TestHeldRunApprovalsEvictsStaleEntriesandTestInstallationPermissionCachesTheGrantare unit tests of the new code. They do not compile on tip.TestContributorLaneApprovesHeldRunsForRegistryTrustednow expects no approval (tip approves it). The other held-run fixtures use a fork head and carryevent: pull_request, as GitHub's payload does.Full runs:
go test ./pkg/github/automerge/ ./pkg/github/ ./pkg/writeback/,go veton both packages,check-docs-citations.py,check-docs-links.py.How to check after deploy
In
fleet_status→ghRateLimits.top_consumers,POST /repos/{owner}/{repo}/actions/runs/{run_id}/approveunderhive:contributor_lane_automerge_sweepshould drop out. A PR whose held runs the App cannot approve should show onelane unauthorizedlog line per head, not one per sweep:log_read include="\"msg\":\"lane unauthorized\",\"start\"".Not changed: the separate gated-run approval sweep (
SweepWorkflowApprovals,hive:workflow_auto_approval) already keeps its once-per-head ledger on the long-lived client and was not in this loop.🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1