fix(rules): MULTI-001 read "without revealing secrets" as concealment - #61
Merged
Merged
Conversation
The pattern matched "without|don't" followed by a hiding verb, with the object optional. So "without revealing secrets" fired, and a skill whose only instruction was not to print an API key was convicted of deferred hidden execution. This is the same defect as the consent inversion, mirrored. There the rule read a safety property (gate on user approval) as evasion; here it reads a second safety property (do not leak a credential) the same way. Both come from matching a phrase without asking what it is about. What is being hidden decides whether hiding it is a problem. Withholding the output, the command, the steps or the activity keeps the skill's behaviour from the person running it, which is what this rule is for. Withholding a secret, credential, key, token or PII is the instruction we want skills to carry. The object is now required rather than optional and enumerates only the first kind. A bare "without revealing" no longer fires: the phrase is ambiguous, and MULTI-001 is already on the high-false-positive list. Tests pin it in both directions and were confirmed to FAIL against the old pattern before the fix went in -- the two positive cases pass either way, so a one-sided test would have looked green against the bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
The object is optional, so
without revealing secretsmatches on the first half alone. A skill whose only instruction was "don't print the API key" was convicted of deferred hidden execution.This is the consent inversion again, mirrored. There the rule read a safety property — gate the action on user approval — as evasion. Here it reads a second safety property — do not leak a credential — the same way. Both come from matching a phrase without asking what it is about.
The fix
What is being hidden decides whether hiding it is a problem.
The object is now required and enumerates only the first list. A bare
without revealingwith no object no longer fires either — the phrase is ambiguous and MULTI-001 is already inHIGH_FP_RULES; guessing on ambiguity is how it got there.Verification
Tests pin both directions. Before committing the fix I reverted the pattern and re-ran:
The two positive tests (
test_detects_without_displaying,test_detects_withholding_the_skills_own_behaviour) pass under both patterns — so a one-sided test would have looked green against the bug. The narrowing removes false positives without costing detection.Full suite: 1,764 passed, 16 skipped. Ruff clean.
Impact on published numbers
This affects a ~72-skill group flagged on this phrasing. Those convictions should be treated as withdrawn pending a re-scan; MULTI-001 hits are capped at risk 74 when uncorroborated, so none of them were standalone MALICIOUS, but they were contributing to corroboration counts for skills that were.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
Generated by Claude Code