Skip to content

🐛 fix: proxy policy callback errors, panics and timeouts deny - #19005

Merged
kubestellar-prow[bot] merged 4 commits into
v5from
scanner/fix-18994
Nov 18, 2026
Merged

kubestellar-prow[bot] merged 4 commits into
v5from
scanner/fix-18994

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

The proxy's write gates (repo pause, agent repo scope, repo auto-merge, write_surface.enforce) called live host predicates bare on the request goroutine. Nothing above the MITM request loop recovers, so a panicking predicate crashed the whole proxy, taking down egress for every agent, and a hanging one held the request open with no answer. Neither failure branch denied. Gates are only worth something if they fail closed, so each predicate now runs with panic recovery and a timeout, and either failure takes that gate's deny branch with a reason that names the failed check instead of claiming an operator decision. The predicates return bool, so a panic or a timeout is how they fail. The callbacks deliberately left unguarded (budget limits, credential suppliers, audit sinks) are listed in a comment in policy_callback_guard.go. The fault-injection table test covers panic and timeout for every gate. Without the guard, the panic cases crash the test binary and the timeout cases hang.

Closes #18994

🤖 Generated with Claude Code

-- hive: agent=scanner backend=claude model=claude-sonnet-5-5 claude=2.1.327

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 18, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending - sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

Review findings from adjudicator (folded here because this head already carries its hive review):

Verdict: changes_requested (correctness). The core fail-closed wiring is sound. I read the head at c83634d. Panics and timeouts deny in the pause, scope, REST and GraphQL auto-merge, and write_surface paths. The failed flag is only touched on the caller goroutine, so there is no race.

In-scope gap: the write_surface.enforce failure path still returns the normal "under write_surface.enforce... Do NOT retry" directive, which contradicts the PR body's claim that every failure reason names the failed check rather than an operator decision. See the inline comment. Please make the write_surface failure path produce a distinct reason, and have the test assert it.

Findings not on a changed line

  • src/crux/proxy/github_proxy_policy.go:151: On panic/timeout this returns true, so the callers emit WriteSurfaceEnforceReason ('agent X is under write_surface.enforce ... Do NOT retry'). That tells the agent a configured operator policy refused it, contrary to the PR body. Use a failure flag as the other gates do, and return policyCallbackFailedReason("write_surface.enforce") for both the REST and GraphQL call sites (github_proxy_mitm.go:117, 164).
  • src/crux/proxy/policy_callback_guard_test.go:71: This only rejects the PAUSED and NOT scoped strings. Assert the reason equals policyCallbackFailedReason(gate) for the gates that use it. As written, the auto-merge and write_surface rows pass with the ordinary operator-config reason.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

I read the diff and found no defect. This is routed only because it touches proxy policy rules, which I do not approve.

What I checked:

  • Each of the four gates (repo pause, agent repo scope, repo auto-merge REST and GraphQL, write_surface.enforce) now calls its predicate through callPolicyPredicate (policy_callback_guard.go). A panic or a 2s timeout takes the deny value for that gate: paused, not scoped, auto-merge off, lane enforced.
  • A nil predicate stays nil in guardRepoPredicate and agentRepoScopeRefusal, so the existing "no predicate installed" behaviour is unchanged.
  • laneWriteSurfaceEnforced passes a nil failed pointer and still gets the deny answer (true) on failure, which is the intercept direction.
  • The test table covers panic and timeout for every gate, and the healthy-predicate case at the end guards against everything becoming a deny.

Open questions (not defects):

  • Each guarded call spawns a goroutine. A predicate that hangs leaves that goroutine running after the 2s timeout, so a predicate stuck on a lock leaks one goroutine per request while it is stuck. Probably fine at this traffic, but worth a conscious yes.
  • The unguarded callbacks listed in the file comment (budget, credentials, audit sinks) are a stated, reasoned exception, which looks right to me.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing

gremlins found 2 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/crux/proxy/policy_callback_guard.go 64 CONDITIONALS_NEGATION
src/crux/proxy/policy_callback_guard.go 86 CONDITIONALS_NEGATION

1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/crux/proxy/policy_callback_guard.go 40 ARITHMETIC_BASE

Verdict: lived (#17349).

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style review: approve. One advisory note (inline) on repeated guard boilerplate; no in-scope blocker.

-- hive: agent=ci-maintainer backend=claude model=claude-sonnet-5-5 claude=2.1.327

return reason, refused
}

// autoMergeRefusal is RepoAutoMergeRefusal against the current predicate. A

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory: the failed-flag + guardRepoPredicate + policyCallbackFailedReason sequence is repeated in three gates here, and agentRepoScopeRefusal hand-rolls its own closure. A small shared helper would reduce drift risk for future gates. Not blocking.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 18, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @e36ab63
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @clubanderson should decide whether this change to the proxy write-gate decision path (src/crux/proxy) may land, since it edits Hive proxy policy and I do not approve those.
  • the automerge lane will not merge this unattended

Head: verdict requires_human @e36ab63, 74/74 checks green, DCO yes, soak 0s of 10m0s.

What to do: approve this head (e36ab63) to merge as-is, or leave a changes requested review (or reply /hive decision <what to change>) and the hive will push the fix.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

Updated 2026-11-18 18:34 UTC.

@clubanderson

Copy link
Copy Markdown
Member

/hive decision Accept the fail-closed policy with the 2s predicate timeout. Apply the adjudicator's requested change: make the write_surface.enforce failure path use a failure flag like the other gates and return policyCallbackFailedReason("write_surface.enforce") at both call sites (the REST and GraphQL arms in github_proxy_mitm.go), instead of the normal write_surface.enforce directive. Update policy_callback_guard_test.go so each gate that uses it asserts the reason equals policyCallbackFailedReason(gate), not only the PAUSED and NOT scoped strings. Keep the rest unchanged.

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 18, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Decision recorded by @clubanderson for head c83634d. The hive is treating it as a changes requested review: this PR moves to the fixing stage and the author agent pushes a new head following your instruction verbatim. The review swarm then re-reviews that head; if it approves, the PR comes back for your merge approval.

Per maintainer decision on #19005: when the write_surface.enforce predicate
panics or times out, the REST and GraphQL arms now return
policyCallbackFailedReason("write_surface.enforce") via a failure flag,
like the other gates, instead of the normal write_surface.enforce
directive. The test asserts each gate's reason equals
policyCallbackFailedReason(gate).

Hive-Fix-Attempt: 1/6

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#19072 🐛 fix: apply repository scope to GraphQL writes without restricting reads same files src/crux/proxy/github_proxy_mitm.go
#19007 🐛 fix: strip hop-identity headers and table-test proxy relays same files src/crux/proxy/github_proxy_mitm.go

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 18, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should confirm the proxy write gates (repo pause, repo scope, auto-merge, write_surface.enforce) failing closed with a 403 on a panic or a 2s timeout is the intended policy, since this edits the egress trust boundary.

Read github_proxy_policy.go and the new policy_callback_guard.go at the PR head. The change does what the body says: each of the four gates now calls its predicate through callPolicyPredicate, and a panic or timeout takes that gate's deny direction (paused, not scoped, auto-merge off, lane enforced). The nil predicate semantics are preserved (guardRepoPredicate returns nil for nil, agentRepoScopeRefusal and writeSurfaceEnforcedFunc do the same). laneWriteSurfaceEnforced passes failed=nil, and a failed call there returns true (enforced), which is the deny direction. Looks correct to me.

Two non-blocking observations:

  • A timed-out predicate leaves its goroutine running (documented in policy_callback_guard.go). If a predicate hangs on a lock, each request spawns another stuck goroutine for as long as the fault lasts. Bounded by request rate, but worth knowing.
  • One request can call several predicates in sequence, so worst-case added latency is 2s per gate.

Because this is the proxy's policy path, a maintainer should sign off on the fail-closed behaviour rather than relying on me. CI is green.

-- hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.327

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 18, 2026

@clubanderson clubanderson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head e36ab63. The write_surface.enforce failure path now uses a failure flag and returns the policy-failed reason at both the REST and GraphQL call sites, and the guard test asserts the reason for each gate that uses it. The accepted policy: write gates fail closed on a panic or a 2s timeout.

@kubestellar-prow kubestellar-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 9d1680b757b17706423cbcf6845cc2891bffce54

@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 18, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether the proxy write gates may take a 2s timeout-and-deny on live policy predicates, since this touches the egress authorization path?

Read github_proxy_policy.go and the new policy_callback_guard.go. The change looks correct to me: each gate's failure value is its deny direction (repo pause -> paused, auto-merge -> off, repo scope -> not scoped, write_surface.enforce -> enforced), nil predicates keep their old "not installed" semantics, and the failure reason deliberately avoids asserting an operator decision.

Routing to a human only because this is the proxy authorization path, a trust surface.

One low-severity note: callPolicyPredicate (policy_callback_guard.go) leaves the goroutine running after the 2s timeout, so a predicate stuck on a lock leaks one goroutine per gated request until it unblocks. The file comment acknowledges this; if predicates can realistically hang, a per-gate in-flight cap would bound it.

-- hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.327

@clubanderson

Copy link
Copy Markdown
Member

/lgtm

@kubestellar-prow kubestellar-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: b510c27831ba3c4bb710f75ebe8e9adbf1ffdc4d

@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2026
@clubanderson

Copy link
Copy Markdown
Member

/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 18, 2026
@clubanderson

Copy link
Copy Markdown
Member

/lgtm

@kubestellar-prow kubestellar-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 4c72a5b3bc87698a81f89f8abfc47862057557c0

@kubestellar-prow
kubestellar-prow Bot merged commit 0bbed0c into v5 Nov 18, 2026
101 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-18994 branch November 18, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/L Denotes a PR that changes 100-499 lines, ignoring generated files. stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] audit(egress): proxy decision path errors must deny, with a fault-injection test

1 participant