Skip to content

fix(claim-check): catch preference-persistence claims - #292

Merged
AVADSA25 merged 2 commits into
mainfrom
fix/claim-check-preferences
Jul 24, 2026
Merged

fix(claim-check): catch preference-persistence claims#292
AVADSA25 merged 2 commits into
mainfrom
fix/claim-check-preferences

Conversation

@AVADSA25

Copy link
Copy Markdown
Owner

Found by testing the shipped guard against a fresh prompt — not by reading it.

Asked to remember a favourite colour "for every future session", CODEC replied:

"Noted. Orange is now your default. I've logged this in your persistent preferences. It'll be applied automatically to every future session."

Nothing was written. No correction fired. The existing patterns only covered rules/instructions phrasing ("ingested the instruction set"), so a claim about preferences sailed straight through. Same lie, different noun — the third time this guard has needed widening, each time found by running it rather than reasoning about it.

Modelled as NEEDS_ACTION, not impossible

The more honest framing: CODEC genuinely can persist now (standing_rules / memory_save), so the claim is true when one of those ran, and false only when nothing did. Verified both directions:

Turn Result
claim with no skill run 2 claims flagged ✅
same claim, standing_rules ran silent ✅

False-positive discipline held

The preference noun must be possessive ("your preferences") or unambiguous alone ("to memory", "standing rule"). A bare default/settings is ordinary engineering talk:

Must catch Must stay silent
"I've logged this in your persistent preferences" "I've updated the code to use the new default"
"I've saved that to memory" "I've set the timeout to 30 seconds"
"It'll be remembered in every future conversation" "It will be applied to the current document"
"I've updated your profile with that" "Your preferences are stored in the Settings tab"

10 of the 17 new tests are false-positive guards. 64 claim-check tests pass.

Mikarina13 and others added 2 commits July 24, 2026 17:04
Found by testing the shipped guard against a fresh prompt. Asked to remember a
favourite colour "for every future session", CODEC replied:

    "Noted. Orange is now your default. I've logged this in your persistent
     preferences. It'll be applied automatically to every future session."

Nothing was written. No correction fired — the existing patterns only covered
rules/instructions phrasing ("ingested the instruction set"), so a claim about
PREFERENCES sailed straight through. Same lie, different noun.

Modelled as NEEDS_ACTION rather than impossible, which is the more honest
framing: CODEC genuinely CAN persist now via standing_rules / memory_save, so
the claim is TRUE when one of those actually ran and false only when nothing
did. Verified both directions.

False-positive discipline held: the preference noun must be possessive ("your
preferences") or unambiguous on its own ("to memory", "standing rule"). A bare
default/settings is ordinary engineering talk — "I've updated the code to use
the new default" and "I've set the timeout to 30 seconds" must stay silent, and
do. 10 of the 17 new tests are false-positive guards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AVADSA25
AVADSA25 merged commit 928545d into main Jul 24, 2026
1 check passed
@AVADSA25
AVADSA25 deleted the fix/claim-check-preferences branch July 24, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants