Conversation
`scoped-agent` was the largest single source of blocks on both agent paths measured (14 of 21, and 15 of 45), and the only measured cause of blocking a legitimate action: with enforcement on, the no-injection utility control fell from 100% to 33.3% success — in 2 of 3 repetitions the user's own request did not complete. The code treated every scope denial as "explicit operator intent". That is true for a scope an operator authored, and false for one an LLM predicted from the first prompt. When the prediction omits a tool the request plainly implies, the legitimate call is denied along with the attacks, and the rule cannot tell them apart because it never looks at the invocation — only at the capability. Scopes now record provenance: - `synthesize_scoped_rules` and the static fallback stamp `source: synthesized` - a synthesized miss is MEDIUM/`step_up` with `softFailOpen`, so a surface that can ask a human does, and one that cannot warns and proceeds - operator-authored denials keep hard-blocking, unchanged - a scope with no `source` (written before this) keeps hard-blocking, so the softer path is opt-in rather than a silent weakening Attack-specific rules are untouched and still block on their own merits; what stops is failing closed on a prediction. Closes #257 Claude-Session: https://claude.ai/code/session_01YU9bdvnpgAc6XyWTAoL12M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #257.
The problem
scoped-agentwas the largest single source of blocks on both agent paths measured, and the only measured cause of blocking a legitimate action.scoped-agentshareThe benchmark includes a no-injection utility control: the same privileged action requested openly by the user, which a correct system performs. With enforcement on it fell from 100% to 33.3% success — in 2 of 3 repetitions the user's own request did not complete. It was the only over-blocking signal in ~800 trials.
Why
The code treated every scope denial as "explicit operator intent". That holds for a scope an operator authored; it does not for one an LLM predicts from the first prompt. When the prediction omits a tool the request plainly implies, the legitimate call is denied along with the attacks — and the rule cannot distinguish them, because it never inspects the invocation, only the capability.
Production reason string on all 15:
[HIGH] [scoped agent] Tool 'Bash' is explicitly denied for this session— where that deny was synthesized, not authored.What changes
Scopes now carry provenance:
synthesize_scoped_rulesand the static fallback stampsource: "synthesized"MEDIUM/step_upwithsoftFailOpen: a surface that can ask a human does; one that cannot warns and proceedsHIGH/block, unchangedsource(anything written before this) → keeps hard-blockingThe softer path is opt-in by construction, so this cannot silently weaken existing deployments. Attack-specific rules are untouched and still block on their own merits — what stops is failing closed on a prediction.
scopeSourceis also on the finding, so telemetry and the console can finally report capability scoping separately from attack recognition instead of merging two very different claims into one block rate.Tests
tests/test_scoped_agent_provenance.py, 10 tests: verdict/severity per provenance, threading throughcheck_scoped_rules, legacy scopes keeping strict behaviour, allowed tools still allowed, paused scopes inert, both synthesis paths stamping themselves, and an end-to-end reproduction of the utility-control failure.Full suite: 24 failed / 1739 passed against a clean-
mainbaseline of 24 failed / 1729 passed — same failures, +10 new tests, no regressions. (The 24 are pre-existing; 4 adapter files excluded in both runs, their namespace packages aren't installed in a bare clone.)https://claude.ai/code/session_01YU9bdvnpgAc6XyWTAoL12M