Skip to content

🐛 fix(automerge): a refused held-run approval is tried once per PR head, not again on every sweep - #17169

Merged
hivecommons-hive[bot] merged 5 commits into
hivecommons:v6from
Danathar:fix/held-run-approve-once
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
hivecommons:v6from
Danathar:fix/held-run-approve-once

Conversation

@Danathar

@Danathar Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

The contributor lane approves a registry-trusted contributor's held fork runs "once per head", but the mark that enforced that lived on the automerge engine, and the sweep builds a new engine every tick. So when GitHub refused the approve with 403 (the App has no actions: write), the next sweep tried again, and the one after that. On projectbluefin two PRs spent about 270 charged calls an hour on POST actions/runs/{id}/approve, every one a 403. This PR moves the pass and the refusal into a process-wide ledger on the GitHub client, so a refused head is tried once and then reported as the PR's blocker until the head changes or the App gains actions: write. It also stops sending runs GitHub's approve endpoint can never accept.

What went wrong

Live hive hosted-projectbluefin-knuckle (build dd6b372), 2026-11-08:

  • projectbluefin/dakota-iso#307 (head 14fdfdd) had three held fork pull_request runs from kylerankin/dakota-iso (Lint, Test, Smoke Test). projectbluefin/fsdk-containers#508 (head c00801f) had three from Karibhou/fsdk-containers (skill-catalog, image-catalog, Build images).
  • The hive log shows lane unauthorized … step=approve-workflow-runs for the same head on every sweep: 61 times between 00:40:50Z and 01:44:16Z, roughly every 75–90 s per PR. Each one is a pass that POSTed all three runs and got 403 Resource not accessible by integration. That is about 183 POSTs in the hour, or 6 per sweep at ~45 sweeps an hour, about 270 an hour. Another session measured 174 charged approve calls with last_status 403 under hive:contributor_lane_automerge_sweep in top_consumers for the same period.
  • It stopped only when Danathar clicked Approve and run (01:38Z and 01:45Z, lane unauthorized cleared). dakota-iso#307 then merged.

Cause: SweepContributorLaneAutoMerges runs New(transport, opts) every tick. The once-per-head map (heldRunApprovals) and the unauthorized state were fields on that throwaway engine, so each tick started empty. The existing tests ran two ticks on one engine, which hid this.

The first report blamed Prow pull_request_review runs (dakota-iso run 42341060495). That part was wrong. This path lists runs with event=pull_request and keeps only runs from the PR's head repository, so that run (head repository projectbluefin/dakota-iso) was never sent. The refused runs were real fork runs. The App simply lacks the permission.

What changes

  • Process-wide ledger. hgithub.HeldRunApprovals lives on the long-lived Client and keeps one entry per PR head: whether a pass ran, and, if GitHub refused, which runs it refused and what the App's actions grant read at the time. The engine gets it from the transport. Options.HeldRunApprovals can override it.
  • No retry on a refused head. The PR keeps reporting unauthorized:approve-workflow-runs, so it stays at 085 with its label and its one existing note. There is no new comment, no listing and no POST on later sweeps. The lane tries again only on a new head, or when the installation now grants actions: write and did not at refusal. Client.InstallationPermission reads that grant at most every 5 minutes, through the same probe the workflows-grant check uses. A restart tries each head once more.
  • Only approvable runs are sent. approvableHeldRun matches GitHub's rule: a pull_request run whose head repository is not the base repository, or a run the Actions bot queued. Same-repository runs and pull_request_review/pull_request_target runs are left for a person, because GitHub answers them with 403 forever. The fork test compares repository names rather than reading head_repository.fork, which is true for any repository that is itself a fork (projectbluefin/dakota-iso is).

Files

  • src/pkg/github/held_run_approvals.go (new): the ledger and InstallationPermission.
  • src/pkg/github/client.go: two fields.
  • src/pkg/github/automerge/automerge_sweep.go: options and wiring.
  • src/pkg/github/automerge/automerge_contributor_held_runs.go: uses the ledger, adds the run filter, removes the engine-local once-per-head helpers.
  • Docs: hive-merge.md and operator-reference/README.md describe the retry rule and the fork-only filter.
  • changelog.d/fixed-held-run-approve-once.md.

Tests

New. Each failed on v6 tip (9975c5d) and passes here:

  • TestContributorLaneRefusedHeldRunsNotRetriedAcrossSweeps builds one engine per sweep on one transport, as production does. Tip: approve POSTs = [11 12 11 12 11 12 11 12] over 4 sweeps. Now [11 12], with one note and one label.
  • TestContributorLaneRefusedHeldRunsRetryOnGrantOrNewHead: same head, same grant, no retry (tip: 4 POSTs after the second pass). Gaining actions: write retries once. A new head retries once. A fork run GitHub accepts is approved.
  • TestContributorLaneApprovesOnlyForkPullRequestHeldRuns: a pull_request_review or pull_request_target run in the listing is never approved, while the fork pull_request run and an Actions-bot run are (tip: [11 21 22 24 11 21 22 24]).
  • TestApprovableHeldRun, TestHeldRunApprovalsOncePerHeadAndRefusal, TestHeldRunApprovalsEvictsStaleEntries and TestInstallationPermissionCachesTheGrant are unit tests of the new code. They do not compile on tip.
  • Updated: the same-repo case in TestContributorLaneApprovesHeldRunsForRegistryTrusted now expects no approval (tip approves it). The other held-run fixtures use a fork head and carry event: pull_request, as GitHub's payload does.

Full runs: go test ./pkg/github/automerge/ ./pkg/github/ ./pkg/writeback/, go vet on both packages, check-docs-citations.py, check-docs-links.py.

How to check after deploy

In fleet_status → ghRateLimits.top_consumers, POST /repos/{owner}/{repo}/actions/runs/{run_id}/approve under hive:contributor_lane_automerge_sweep should drop out. A PR whose held runs the App cannot approve should show one lane unauthorized log line per head, not one per sweep: log_read include="\"msg\":\"lane unauthorized\",\"start\"".

Not changed: the separate gated-run approval sweep (SweepWorkflowApprovals, hive:workflow_auto_approval) already keeps its once-per-head ledger on the long-lived client and was not in this loop.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…once per sweep

The contributor lane's once-per-head mark for held-run approvals lived on
the automerge engine, which SweepContributorLaneAutoMerges rebuilds every
tick. A head whose approvals GitHub refused with 403 (App without
actions: write) was therefore re-POSTed on every sweep: on projectbluefin
dakota-iso#307 and fsdk-containers#508 cost ~270 charged
POST actions/runs/{id}/approve calls an hour, all 403.

Move the pass and the refusal into a process-wide ledger on the GitHub
client (HeldRunApprovals). A refused head keeps reporting
unauthorized:approve-workflow-runs without another call until the head
changes or the installation gains actions: write (read via
InstallationPermission, cached 5 min). Only send runs GitHub's approve
endpoint accepts: fork pull_request runs and Actions-bot runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 8, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs-currency review of head 1681836. The touched docs (hive-merge.md:884-899, operator-reference/README.md:548, the changelog fragment) match the code: approvableHeldRun (automerge_contributor_held_runs.go) sends only fork pull_request / Actions-bot runs, and the "at most every five minutes" re-read is workflowsGrantTTL = 5 * time.Minute (src/pkg/github/issue_workflows_handoff.go:166). Advisory only; nothing blocking.

src/docs/github-app-setup.md:46 does not carry the new retry rule. That paragraph is where an operator goes after granting Actions: Read and write. For the gated-run sweep it says a refused head is not retried after the grant ("takes effect for new heads (or after a hive restart), while an already-refused head needs Approve and run"), then describes the contributor-lane path's 403 outcome without saying it now behaves differently: HeldRunApprovals.ReleaseOnGrant (held_run_approvals.go) re-arms a refused head once the installation reads actions: write, so no click or restart is needed for that lane. An operator reading only this file will click Approve and run unnecessarily, or expect a retry that the sweep path does not do.

… logs `lane unauthorized` once per head, parks the PR at 085 (`Awaiting human: approve workflow runs` in the activity monitor), does not POST that head again until it changes or the installation gains **Actions: Read and write** (unlike the gated-run sweep above, the contributor lane re-reads the grant at most every five minutes and tries a refused head once more when it appears; a restart also tries each head once more), and clears all of it the tick the required checks report ([#14589](https://github.com/hivecommons/hive/issues/14589)).

Minor (info): hive-merge.md:896-897 says the pass and refusal are "kept for the life of the hive process", while HeldRunApprovals.lookupLocked evicts an entry untouched for heldRunApprovalTTL (24 h). A swept PR refreshes seen every tick so this only matters for a PR not visited for a day (e.g. parked behind an exclude_labels label and then unlabelled), which then gets one more pass. Worth a parenthetical ("an entry no sweep has touched for 24 h is dropped") if you touch the paragraph again.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • check-failure

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review at head 1681836. The core fix is sound: the ledger is read from the long-lived *github.Client (automerge_sweep.go:576-583, and main_sweeps.go:972 passes ghClient *github.Client), contributorHeldRunsReason is checked before the refused path (automerge_contributor_held_runs.go:73-75) so a head whose checks later appear is not stuck on unauthorized:, and TestContributorLaneRefusedHeldRunsNotRetriedAcrossSweeps exercises one engine per tick. One advisory edge case:

Low — the one "retry on grant" pass can be burned by a stale installation token, then never retried for that head. AppAuth.Token serves the cached installation token until tokenExpiry - tokenRefreshBuffer (src/pkg/github/app.go:173-186), while InstallationPermission probes the installation with the App JWT and sees the new grant within 5 min (held_run_approvals.go:174-196). In that window ReleaseOnGrant releases (held_run_approvals.go:133-146), the POST goes out with a pre-grant token and gets 403, and RecordRefusal stores grant=write, grantKnown=true (automerge_contributor_held_runs.go:148-150) — after which ReleaseOnGrant returns false forever for this head (entry.grantKnown && permissionAllowsWrite(entry.grant)). The PR body/docs say the lane retries "when the installation now grants actions: write"; in this window it will not, until a push or restart. Not a regression vs. v6 (which spammed), but worth a bounded re-release, e.g.:

 type heldRunApprovalEntry struct {
 	head        string
 	refused     bool
+	refusedAt   time.Time
 	refusedRuns []int64
 	grant       string
 	grantKnown  bool
 	seen        time.Time
 }
...
 	entry.refused = true
+	entry.refusedAt = time.Now()
...
-	if !ok || !entry.refused || (entry.grantKnown && permissionAllowsWrite(entry.grant)) {
+	// A refusal recorded while the grant already read write may have gone
+	// out on an installation token minted before the grant (tokens live ~1h);
+	// allow one more pass per token lifetime instead of never.
+	if !ok || !entry.refused || (entry.grantKnown && permissionAllowsWrite(entry.grant) && time.Since(entry.refusedAt) < time.Hour) {
 		return false
 	}

(TestHeldRunApprovalsOncePerHeadAndRefusal's "refused again holding write → no release" assertion would still hold within the hour.)

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against v6 2ab4202 plus the callers. The fix does what it says: SweepContributorLaneAutoMerges (automerge_sweep.go:638) calls New(transport, opts) every tick, and main_sweeps.go:972 passes the long-lived *github.Client as the transport, so the HeldRunApprovals() type assertion in New picks up the client-owned ledger and the once-per-head mark now survives the one-shot engine. The 403 path records the refusal before markUnauthorized, and later sweeps return unauthorizedReason(...) without listing or POSTing. ReleaseOnGrant only releases when the grant is known write and was not known-write at refusal, so a token-authenticated client (known=false) never flips into a retry loop. Looks correct to me.

One thing that will likely stop CI:

  • src/pkg/github/automerge/automerge_contributor_held_runs.go:39 — contributorHeldRunsApprovalKeyPrefix is still declared, but its only reference (contributorHeldRunsApprovalKey, removed in this diff) is gone. src/.golangci.yml enables unused, so the golangci-lint job (in progress as I write) should report it. Deleting the const block entry (and its two-line comment) fixes it. Low severity, build-gate only.

Open question, not a defect: approvableHeldRun drops every same-repository action_required run. The old path approved those too, and TestContributorLaneApprovesHeldRunsForRegistryTrusted was flipped to expect no approval. If any served repo relies on the lane for same-repo held runs (e.g. first-time-contributor approval policy on a branch PR), that behaviour is gone; the PR body argues GitHub 403s them anyway, which matches the endpoint's documented rule, so I have not counted it as a finding.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the ledger and the lane against the tree at cae2e25. Looks correct to me.

  • src/pkg/github/held_run_approvals.go — Claim/RecordRefusal/Refused/ReleaseOnGrant are all under one mutex and keyed repo#number with the head compared in lookupLocked, so a new head naturally falls through to a fresh attempt. ReleaseOnGrant only fires when the grant is now write/admin and was not at refusal time.
  • automerge_contributor_held_runs.go:63-71 — the ledger path is only entered when contributorHeldRunsReason(reason) holds (a CI-evidence gap), so once a person clicks Approve and run and checks exist the refused entry is never consulted; the PR is not left reporting unauthorized:approve-workflow-runs. That was the one thing I wanted to rule out.
  • automerge_integration_eligibility.go:49-53 gives the read-only probe a private ledger, so a probe's refused transport cannot mark a head as tried for the real sweep.
  • automerge_sweep.go:576-593 falls back to the transport's HeldRunApprovals(); runAutoMergeSweepIfDue passes *github.Client directly (src/cmd/hive/main_sweeps.go:954,972), so the process-wide ledger is actually reached in production.

One note for whoever merges: #17178 (same author) edits the same region of New() in automerge_sweep.go; the author reports a trial merge of both passes, I did not repeat it.

Evidence bundle `#17169cae2e25` sha256:92f17561c38c989e162b61912a979c7e67354dc0dc7606f1f4172d159eb298ce — download from the dashboard or `hivectl review evidence #17169`.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17025 🌱 Forward-merge v5 into v6 (1 commits) same files src/docs/operator-reference/README.md
#17130 ✨ feat(github): the hive files grouped issues from GitHub code scanning alerts (opt-in per repo) same files src/docs/github-app-setup.md
src/docs/operator-reference/README.md
src/pkg/github/client.go
#17178 🐛 fix(automerge): a refused update-branch, merge or merge-forward is sent once per PR head, not again on every sweep same files src/pkg/github/automerge/automerge_integration_eligibility.go
src/pkg/github/automerge/automerge_sweep.go
src/pkg/github/client.go

…ne's grant retry

golangci-lint flagged contributorHeldRunsApprovalKeyPrefix as unused after
the ledger moved to the long-lived client. The hive reviewer pointed out
that github-app-setup.md still said a refused head needs Approve and run
after the grant. The contributor lane now retries a refused head once the
installation reads actions: write, so the doc says so. hive-merge.md also
notes that a ledger entry untouched for 24 hours is dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Both points are fixed in 3d6cccb:

  • golangci-lint: removed the unused contributorHeldRunsApprovalKeyPrefix. Local golangci-lint now reports 0 issues on the package.
  • github-app-setup.md: took your suggested wording. The contributor lane now retries a refused head once the installation reads Actions: Read and write; the gated-run sweep doesn't.
  • hive-merge.md: added the 24-hour parenthetical for ledger entries no sweep has touched.

The docs citation and link checks pass, and so do the automerge held-run tests.

…run ledger

The integration eligibility probe builds its own engine with the live
options. With HeldRunApprovals unset, that engine picked up the process-wide
ledger. approveHeldRunsForRegistryTrusted claims a head before it calls the
approve API, and the probe's read-only transport refuses that call. So a
probe could mark a head as tried, and the real contributor-lane sweep would
never approve it. integrationReadOnlyOptions now gives the probe a private
ledger.

Found by the agent auditing hivecommons#17178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

One more fix in c3985f0, found while auditing #17178. The read-only integration eligibility probe was using the process-wide held-run ledger. approveHeldRunsForRegistryTrusted claims a head before the approve call, and the probe's read-only transport refuses that call, so a probe could leave a head marked as tried and the real sweep would never approve it. integrationReadOnlyOptions now gives the probe a private ledger. TestIntegrationEligibilityHeldRunLedgerIsPrivate fails without the change. The automerge package passes, and vet and golangci-lint are clean.

hivecommons-hive Bot and others added 2 commits November 8, 2026 02:38
…r client.go grew

The new held-run ledger fields on Client shifted RankActionableIssues by five
lines, so the docs citation check reported drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v6 tree alongside #17178 (same files, same one-shot-engine root cause). The ledger logic is sound; I found no defect. One red check and two notes.

What I verified

  • HeldRunApprovals.Claim (held_run_approvals.go:94-107) replaces the PR's entry when the head differs, so a push re-arms the pass; lookupLocked never evicts an entry on a stale-head read, only on Claim, so a late sweep for an old head cannot wipe the current head's record.
  • ReleaseOnGrant (held_run_approvals.go:139-152) cannot loop: it only releases when the grant is now write/admin and the refusal was recorded with the grant unknown or below write. The retry's second refusal is recorded via RecordRefusal with grantKnown=true, write, after which the entry.grantKnown && permissionAllowsWrite(entry.grant) guard holds.
  • In approveHeldRunsForRegistryTrusted (automerge_contributor_held_runs.go:80-92) both the Refused and Attempted checks sit before the workflow-run listing, so a remembered head costs no API call per sweep, matching the pre-change ordering. actionsPermission goes through InstallationPermission's cache, so the per-sweep permission read on a refused head is also free inside the TTL.
  • Every ledger method is nil-receiver safe, and New falls back to a private &HeldRunApprovals{} when the transport has none, so engines in existing tests keep their old per-engine behaviour.
  • integrationReadOnlyOptions hands the probe a private ledger — a read-only evaluation cannot consume the real sweep's one pass.
  • git merge-tree of this head against #17178's shows no conflicts, as #17178's body claims.

Red check: relative links and anchors resolve fails on c3985f0. I could not pin the link from here; the three touched docs (github-app-setup.md, hive-merge.md, operator-reference/README.md) are where to look.

Note (low): with #17178 this PR gives github.Client a typed heldRunApprovals field + HeldRunApprovals() accessor (client.go:182-183) while #17178 adds a generic SharedState(name, build). Both solve the same problem; worth putting the ledger on SharedState in whichever lands second (I left the mirror note on #17178).

Open question, not a finding: approvableHeldRun (automerge_contributor_held_runs.go:235-247) returns false when the run payload has no head_repository.full_name. Before this change such a run was POSTed; now it is silently left for a person. If GitHub ever omits head_repository on a fork pull_request run that is the behaviour you want? If so a DEBUG line naming the skipped run would help the next operator who asks why a head was never approved.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@Danathar

Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks.

  • Red check: it was a docs citation drift, not a broken link. scale-envelope.md cited RankActionableIssues at its old line range. That's fixed in cae2e25, and check-docs-citations.py and check-docs-links.py are both clean on this head.
  • approvableHeldRun without head_repository: that's intended. It fails closed. GitHub includes head_repository on pull_request workflow runs, and without it we can't tell a fork run from a same-repo run. Same-repo runs are exactly the ones runs/{id}/approve refuses, so POSTing blind would bring back the wasted 403s. Such a head still shows as waiting on a person through the existing path. A DEBUG line is a fair idea, and I'll add one if this ever turns up in practice.
  • Ledger vs SharedState: agreed. I'll fold the held-run ledger into 🐛 fix(automerge): a refused update-branch, merge or merge-forward is sent once per PR head, not again on every sweep #17178's SharedState in a follow-up once both land, so neither PR depends on the other's merge order.

@hivecommons-hive
hivecommons-hive Bot merged commit 9a987d4 into hivecommons:v6 Nov 8, 2026
67 of 68 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at cae2e25.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant