Skip to content

fix(rules): MULTI-001 read "without revealing secrets" as concealment - #61

Merged
Ap6pack merged 1 commit into
mainfrom
claude/fix-multi001-secrets-inversion
Sep 11, 2026
Merged

Ap6pack merged 1 commit into
mainfrom
claude/fix-multi001-secrets-inversion

Conversation

@Ap6pack

@Ap6pack Ap6pack commented Sep 11, 2026

Copy link
Copy Markdown
Owner

The bug

r"(?:without|don'?t)\s+(?:displaying?|showing?|printing?|logging?|outputting?|revealing?)"
r"(?:\s+(?:the\s+)?(?:output|result|command|anything))?",   # <- optional

The object is optional, so without revealing secrets matches on the first half alone. A skill whose only instruction was "don't print the API key" was convicted of deferred hidden execution.

This is the consent inversion again, mirrored. There the rule read a safety property — gate the action on user approval — as evasion. Here it reads a second safety property — do not leak a credential — the same way. Both come from matching a phrase without asking what it is about.

The fix

What is being hidden decides whether hiding it is a problem.

  • Concealment: the output, the command, the steps, the errors, the activity, what you are doing. Withholding these keeps the skill's behaviour from the person running it. Still fires.
  • Safety: secrets, credentials, passwords, API keys, tokens, private keys, PII, sensitive values. Withholding these is the instruction we want skills to carry. No longer fires.

The object is now required and enumerates only the first list. A bare without revealing with no object no longer fires either — the phrase is ambiguous and MULTI-001 is already in HIGH_FP_RULES; guessing on ambiguity is how it got there.

Verification

Tests pin both directions. Before committing the fix I reverted the pattern and re-ran:

FAILED test_withholding_secrets_is_a_safety_instruction
FAILED test_bare_without_revealing_is_too_ambiguous_to_convict
2 failed, 2 passed

The two positive tests (test_detects_without_displaying, test_detects_withholding_the_skills_own_behaviour) pass under both patterns — so a one-sided test would have looked green against the bug. The narrowing removes false positives without costing detection.

Full suite: 1,764 passed, 16 skipped. Ruff clean.

Impact on published numbers

This affects a ~72-skill group flagged on this phrasing. Those convictions should be treated as withdrawn pending a re-scan; MULTI-001 hits are capped at risk 74 when uncorroborated, so none of them were standalone MALICIOUS, but they were contributing to corroboration counts for skills that were.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL


Generated by Claude Code

The pattern matched "without|don't" followed by a hiding verb, with the
object optional. So "without revealing secrets" fired, and a skill whose
only instruction was not to print an API key was convicted of deferred
hidden execution.

This is the same defect as the consent inversion, mirrored. There the rule
read a safety property (gate on user approval) as evasion; here it reads a
second safety property (do not leak a credential) the same way. Both come
from matching a phrase without asking what it is about.

What is being hidden decides whether hiding it is a problem. Withholding
the output, the command, the steps or the activity keeps the skill's
behaviour from the person running it, which is what this rule is for.
Withholding a secret, credential, key, token or PII is the instruction we
want skills to carry.

The object is now required rather than optional and enumerates only the
first kind. A bare "without revealing" no longer fires: the phrase is
ambiguous, and MULTI-001 is already on the high-false-positive list.

Tests pin it in both directions and were confirmed to FAIL against the old
pattern before the fix went in -- the two positive cases pass either way,
so a one-sided test would have looked green against the bug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
@Ap6pack
Ap6pack merged commit 2f19ffb into main Sep 11, 2026
5 checks passed
@Ap6pack
Ap6pack deleted the claude/fix-multi001-secrets-inversion branch September 11, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants