Repository navigation
🐛 fix: proxy policy callback errors, panics and timeouts deny #19005
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
c83634d
fix: proxy decision path denies on callback failure
hivecommons-hive[bot] e36ab63
fix: write_surface.enforce callback failure returns policy-failed reason
hivecommons-hive[bot] 5902612
Merge branch 'v5' into scanner/fix-18994
hivecommons-hive[bot] 876db30
Merge branch 'v5' into scanner/fix-18994
hivecommons-hive[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| - **Proxy write gates fail closed when a policy check fails.** If the repo pause, agent repo scope, auto-merge or write_surface.enforce check panics or hangs, the proxy now refuses that one request with a 403 naming the failed check, instead of crashing the whole proxy or holding the request open. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| package proxy | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "time" | ||
| ) | ||
|
|
||
| // The per-request write gates (repo pause, agent repo scope, repo auto-merge | ||
| // policy, write_surface.enforce) each consult a live predicate the hive | ||
| // installs through a Set*Func setter. Those predicates are host code reading | ||
| // live config, and the proxy used to call them bare on the request goroutine. | ||
| // That left two failure branches on the decision path that did not deny: | ||
| // | ||
| // - A panic. No goroutine above the MITM request loop recovers, so a panic | ||
| // in a predicate took down the whole proxy process (every agent's egress) | ||
| // instead of refusing the one request it was evaluating. | ||
| // - A hang. A predicate blocked on a lock or a slow store held the agent's | ||
| // request open indefinitely with no answer at all. | ||
| // | ||
| // The predicates return a plain bool, so there is no error value to | ||
| // mishandle; a panic or a timeout is how such a callback fails. Every gate | ||
| // now calls its predicate through callPolicyPredicate and treats either | ||
| // failure as the deny answer for that gate: paused, not scoped, auto-merge | ||
| // off, lane enforced. Fail closed, same direction as the dial-time | ||
| // classifier in plain_forward_guard.go refusing an address it cannot parse. | ||
| // | ||
| // Intentional exceptions, not routed through this guard: | ||
| // | ||
| // - githubBudgetCap / githubReserveFloor / githubReserveSnapshot are rate | ||
| // limits, not authorization; they default to config values when unset | ||
| // and a wrong answer costs quota, not a policy bypass. | ||
| // - agentTokenSource / linearCredential supply credentials rather than | ||
| // decide; a missing credential already makes the upstream refuse. | ||
| // - writeRefusedAudit and canaryLeakFunc are sinks called after a decision | ||
| // has been made; they cannot turn a deny into an allow. | ||
|
|
||
| // policyCallbackTimeout bounds one policy predicate call. The predicates read | ||
| // in-memory config, so anything near this is already a fault. A var only so | ||
| // the fault-injection test can shorten it. | ||
| var policyCallbackTimeout = 2 * time.Second | ||
|
|
||
| // policyCallbackFailedReason is the agent-facing 403 body when a gate could | ||
| // not be evaluated. It deliberately does not reuse the gate's own directive | ||
| // ("repository X is PAUSED by the operator"), which would assert an operator | ||
| // decision nobody made. | ||
| func policyCallbackFailedReason(gate string) string { | ||
| return "the proxy could not evaluate the " + gate + " policy for this request (the policy check failed or timed out), so the write is refused: policy checks fail closed. Retry shortly; if it keeps happening, report it to the operator as a hive fault." | ||
| } | ||
|
|
||
| // callPolicyPredicate runs fn with panic recovery and policyCallbackTimeout. | ||
| // ok is false when fn panicked or did not return in time; the caller must | ||
| // then take its deny branch. On timeout the goroutine running fn is left to | ||
| // finish on its own (Go cannot cancel it); the buffered channel lets it exit | ||
| // without a reader. | ||
| func (p *GitHubProxy) callPolicyPredicate(gate string, fn func() bool) (result, ok bool) { | ||
| type outcome struct { | ||
| value bool | ||
| panicked bool | ||
| cause any | ||
| } | ||
| done := make(chan outcome, 1) | ||
| go func() { | ||
| defer func() { | ||
| if r := recover(); r != nil { | ||
| done <- outcome{panicked: true, cause: r} | ||
| } | ||
| }() | ||
| done <- outcome{value: fn()} | ||
| }() | ||
| timer := time.NewTimer(policyCallbackTimeout) | ||
| defer timer.Stop() | ||
| select { | ||
| case o := <-done: | ||
| if o.panicked { | ||
| p.logPolicyCallbackFailure(gate, fmt.Sprintf("panic: %v", o.cause)) | ||
| return false, false | ||
| } | ||
| return o.value, true | ||
| case <-timer.C: | ||
| p.logPolicyCallbackFailure(gate, "timed out after "+policyCallbackTimeout.String()) | ||
| return false, false | ||
| } | ||
| } | ||
|
|
||
| func (p *GitHubProxy) logPolicyCallbackFailure(gate, cause string) { | ||
| if p == nil || p.logger == nil { | ||
| return | ||
| } | ||
| p.logger.Error("proxy: policy callback failed; denying the request", "gate", gate, "cause", cause) | ||
| } | ||
|
|
||
| // guardRepoPredicate wraps a func(repo) bool gate predicate so a failed call | ||
| // answers onFail (the gate's deny value) and sets *failed. A nil fn stays nil | ||
| // so the Refusal functions keep their "no predicate installed" semantics. | ||
| func (p *GitHubProxy) guardRepoPredicate(gate string, fn func(string) bool, onFail bool, failed *bool) func(string) bool { | ||
| if fn == nil { | ||
| return nil | ||
| } | ||
| return func(repo string) bool { | ||
| v, ok := p.callPolicyPredicate(gate, func() bool { return fn(repo) }) | ||
| if !ok { | ||
| *failed = true | ||
| return onFail | ||
| } | ||
| return v | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| package proxy | ||
|
|
||
| import ( | ||
| "strings" | ||
| "testing" | ||
| "time" | ||
| ) | ||
|
|
||
| // TestPolicyCallbackFailureDenies injects faults into every live policy | ||
| // predicate on the write-gate decision path and asserts the gate refuses. | ||
| // Before the guard, the panic rows crashed the test binary (no recover on the | ||
| // request path) and the hang rows blocked until the test timeout. | ||
| func TestPolicyCallbackFailureDenies(t *testing.T) { | ||
| prev := policyCallbackTimeout | ||
| policyCallbackTimeout = 50 * time.Millisecond | ||
| release := make(chan struct{}) | ||
| t.Cleanup(func() { | ||
| close(release) | ||
| policyCallbackTimeout = prev | ||
| }) | ||
|
|
||
| type fault struct { | ||
| name string | ||
| hit func() | ||
| } | ||
| faults := []fault{ | ||
| {"panic", func() { panic("injected policy fault") }}, | ||
| {"timeout", func() { <-release }}, | ||
| } | ||
|
|
||
| const mergeGraphQL = `{"query":"mutation { mergePullRequest(input:{pullRequestId:\"PR_kw\"}) { clientMutationId } }","variables":{"repo":"acme/app"}}` | ||
|
|
||
| gates := []struct { | ||
| name string | ||
| gate string // policy name in policyCallbackFailedReason; "" = bool-only check, no reason | ||
| check func(p *GitHubProxy, hit func()) (string, bool) | ||
| }{ | ||
| {"repo pause", "repo pause", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetRepoPausedFunc(func(string) bool { hit(); return false }) | ||
| return p.repoPauseRefusal("POST", "/repos/acme/app/issues") | ||
| }}, | ||
| {"agent repo scope", "agent repo scope", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetAgentRepoScopeFunc(func(string, string) bool { hit(); return true }) | ||
| return p.agentRepoScopeRefusal("scanner", "POST", "/repos/acme/app/issues") | ||
| }}, | ||
| {"repo auto-merge REST", "repo auto-merge", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetRepoAutoMergeEnabledFunc(func(string) bool { hit(); return true }) | ||
| return p.autoMergeRefusal("PUT", "/repos/acme/app/pulls/7/merge") | ||
| }}, | ||
| {"repo auto-merge GraphQL", "repo auto-merge", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetRepoAutoMergeEnabledFunc(func(string) bool { hit(); return true }) | ||
| return p.graphQLAutoMergeRefusal([]byte(mergeGraphQL)) | ||
| }}, | ||
| {"write_surface.enforce", "write_surface.enforce", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetWriteSurfaceEnforceFunc(func(string) bool { hit(); return false }) | ||
| reason, _, refused := p.writeSurfaceEnforceRefusal("scanner", "POST", "/repos/acme/app/issues", "") | ||
| return reason, refused | ||
| }}, | ||
| {"write_surface.enforce GraphQL", "write_surface.enforce", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetWriteSurfaceEnforceFunc(func(string) bool { hit(); return false }) | ||
| return p.writeSurfaceGraphQLRefusal("scanner", []byte(mergeGraphQL)) | ||
| }}, | ||
| {"write_surface.enforce MITM", "", func(p *GitHubProxy, hit func()) (string, bool) { | ||
| p.SetWriteSurfaceEnforceFunc(func(string) bool { hit(); return false }) | ||
| return "", p.laneWriteSurfaceEnforced("scanner") | ||
| }}, | ||
| } | ||
|
|
||
| for _, g := range gates { | ||
| for _, f := range faults { | ||
| t.Run(g.name+"/"+f.name, func(t *testing.T) { | ||
| p := &GitHubProxy{} | ||
| reason, denied := g.check(p, f.hit) | ||
| if !denied { | ||
| t.Fatalf("%s callback %s: gate allowed the request; want deny", g.name, f.name) | ||
| } | ||
| if g.gate != "" && reason != policyCallbackFailedReason(g.gate) { | ||
| t.Fatalf("%s callback %s: reason = %q; want policyCallbackFailedReason(%q)", g.name, f.name, reason, g.gate) | ||
| } | ||
| if strings.Contains(reason, "PAUSED by the operator") || strings.Contains(reason, "is NOT scoped") { | ||
| t.Fatalf("failure reason asserts an operator decision nobody made: %q", reason) | ||
| } | ||
| }) | ||
| } | ||
| } | ||
|
|
||
| // A healthy predicate still decides: the guard must not turn every | ||
| // answer into a deny. | ||
| p := &GitHubProxy{} | ||
| p.SetRepoPausedFunc(func(string) bool { return false }) | ||
| if reason, denied := p.repoPauseRefusal("POST", "/repos/acme/app/issues"); denied { | ||
| t.Fatalf("healthy pause predicate refused: %q", reason) | ||
| } | ||
| p.SetAgentRepoScopeFunc(func(string, string) bool { return true }) | ||
| if reason, denied := p.agentRepoScopeRefusal("scanner", "POST", "/repos/acme/app/issues"); denied { | ||
| t.Fatalf("healthy scope predicate refused: %q", reason) | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Advisory: the failed-flag + guardRepoPredicate + policyCallbackFailedReason sequence is repeated in three gates here, and agentRepoScopeRefusal hand-rolls its own closure. A small shared helper would reduce drift risk for future gates. Not blocking.