From be793b7793e5ce1100103e000ce46293224e7a10 Mon Sep 17 00:00:00 2001 From: logan Date: Fri, 14 Aug 2026 17:14:25 -0400 Subject: [PATCH 1/2] fix: a flag a probe arm did not run decides a reading (#113) `checkRecord` read the flag names as well as the structure, so a record omitting `--strict-mcp-config` or carrying `--verbose` was a broken file rather than a failed probe. Only a wrong value reached the acceptance test, so ADR-0024's guarantee held for half the case it named. Allowlist membership and required-flag presence move to the value reading. `flagShapeProblems` keeps structural impossibility alone, which is what no revision of this collector could have produced. The principle: a check may refuse a record on a stable identity fact of the collector, and never on a protocol choice this repository versions. Operator ruling 2026-08-14, Option A on the issue's fork. --- AGENTS.md | 28 +++++-- bench/probe.mjs | 81 +++++++++++++------ ...a-probe-measures-the-attachment-surface.md | 50 +++++++++++- test/probe.test.js | 68 ++++++++++++++-- 4 files changed, 186 insertions(+), 41 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 73dab71..b638d44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -478,11 +478,23 @@ number published in `bench/README.md`. the protocol. - **A record collected under the wrong flags is a failed probe, not a broken file.** `checkRecord` reads the flag shape and `deriveOutcome` - reads the values, as `isolated`. They were one reading, which made such - a record malformed — so the repository could not keep one, and the rule - that a recorded failure is a result did not hold for the isolation - failure. Nothing is weakened: the record still derives FAIL and can - never read as a pass. ADR-0024. + reads the names, the presence and the values, as `isolated`. They were + one reading, which made such a record malformed — so the repository + could not keep one, and the rule that a recorded failure is a result + did not hold for the isolation failure. The first split reached the + wrong-VALUE case alone, so a record omitting `--strict-mcp-config` or + carrying `--verbose` was still a broken file, and the next move that + adds or removes a flag NAME would have made every committed record + malformed. Issue 113 reports it. A check may refuse a record on a + stable identity fact of the collector, and never on a protocol choice + this repository versions. Flag names, required presence and + `TRACE_LINE_LIMIT` are versioned choices, so each decides a reading and + never a record's validity. `flagShapeProblems` keeps structural + impossibility alone: flags that are not a non-empty array, an entry + that is not a string, a flag stated twice, a value-taking flag at the + end of the list, and a flag sitting where another flag's value belongs. + Nothing is weakened: the record still derives FAIL and can never read + as a pass. ADR-0024. - A probe arm runs `--setting-sources user`, and `bench/run.sh` selects its spelling from `--rules`: `none` gives the no-guidance control `''`, and `user` gives the treatment arm `user`. The divergence is between a @@ -549,7 +561,11 @@ number published in `bench/README.md`. pathways no run of this collector could have written. That is not the wrong-flags rule loosening: a wrong-flag record is a run that HAPPENED and must stay readable, and a pathway no runner drives is a file this - tool could not have produced. A trace beside a flag set carrying no + tool could not have produced. The pathway combination is a stable + identity fact, so it stays a shape refusal. The flag set is not one, + and it is no longer an example of this rule — `flagShapeProblems` reads + structure alone, and the names and the presence answer to + `isolationProblems` beside the values. A trace beside a flag set carrying no `--debug-file` withholds as `unrequested`, and `managed_seen` withholds on the same condition, because the question there is not whether the count is a floor but whether the bytes are evidence at all. The diff --git a/bench/probe.mjs b/bench/probe.mjs index 56d5ab9..537497b 100644 --- a/bench/probe.mjs +++ b/bench/probe.mjs @@ -351,30 +351,44 @@ function keyPaths(value, prefix = '') { return out; } -/** - * Problems with the flag set the probe ran under. An unknown flag is refused - * rather than ignored, because the acceptance test is that the probe ran the - * acceptance flag set, plus at most the trace flag. A flag outside that is a - * configuration surface the arm never opened, and a probe that needed one has - * failed the test it exists to run. - */ /** * The flag walk, in two readings. * - * `isolationProblems` is the ACCEPTANCE TEST: it reads the values too, so a - * record whose arm ran the wrong `--setting-sources` fails it. `deriveOutcome` - * asks this one. - * - * `flagShapeProblems` is the RECORD CHECK: it reads the structure and leaves the - * values alone. `checkRecord` asks this one, and the difference matters because - * of what the two questions are for. - * - * Both were one function, and `checkRecord` asked the acceptance test. That made - * a probe which ran the wrong flags a MALFORMED RECORD rather than a FAILED - * PROBE — so the repository could not keep one, and the design's own rule that a - * recorded failure is a result did not hold for the isolation failure. It became - * visible when the acceptance flag set moved on 2026-08-07: the record of the - * probe that motivated the move could not survive it. ADR-0024. + * `isolationProblems` is the ACCEPTANCE TEST. It reads the flag NAMES and the + * VALUES: which flags the arm ran, which required ones it omitted, and what + * each one carried. `deriveOutcome` asks this one, and reports the answer as + * `isolated`. + * + * `flagShapeProblems` is the RECORD CHECK, and it reads structural + * impossibility alone. `runArms` builds ONE flag set above the loop, so a + * record carries a list of strings where each named flag appears once and each + * value-taking flag is followed by its value. Nothing else there is a fact + * about the collector. So the shape reading keeps five refusals: flags that are + * not a non-empty array, an entry that is not a string, a flag stated twice, a + * value-taking flag at the end of the list, and a flag sitting where another + * flag's value belongs. `checkRecord` asks this one. + * + * The line between them is stable identity against versioned protocol. A check + * may refuse a record on a fact about the collector that does not move — the + * pathway combination is one, because no run could have written a pathway no + * runner drives. It may not refuse a record on a PROTOCOL CHOICE this + * repository versions. Which flags a probe arm runs is such a choice, and so is + * `TRACE_LINE_LIMIT`. The set moved on 2026-08-07 and it will move again, and a + * check that refuses on it retires committed evidence every time it moves. So a + * versioned choice decides a READING and never a record's validity. + * + * Both readings were one function, and `checkRecord` asked the acceptance test. + * That made a probe which ran the wrong flags a MALFORMED RECORD rather than a + * FAILED PROBE — so the repository could not keep one, and the design's own + * rule that a recorded failure is a result did not hold for the isolation + * failure. It became visible when the acceptance flag set moved on 2026-08-07: + * the record of the probe that motivated the move could not survive it. + * ADR-0024. + * + * The first split fixed the VALUE case alone. Membership and presence stayed on + * the shape side, so a record omitting `--strict-mcp-config`, or carrying a + * flag this repository has never heard of, was still a broken file. Issue 113 + * reports it, and both readings now sit with the values. * * Nothing is weakened. A record under the wrong flags still derives FAIL, * `check:probes` still prints it as one, and no such record can ever read as a @@ -421,9 +435,15 @@ function flagProblems(flags, values) { continue; } if (!ALLOWED_FLAGS.includes(flag)) { - problems.push(flag.startsWith('-') - ? `${flag} is not a flag a probe arm runs.` - : `"${flag}" at position ${i} is not part of a probe arm's invocation.`); + // A flag outside the set is a configuration surface the arm never opened, + // so the ACCEPTANCE test refuses it. The shape reading passes it over, + // because the set is a protocol choice this repository versions. Either + // way it consumes one element, so both readings walk the list alike. + if (values) { + problems.push(flag.startsWith('-') + ? `${flag} is not a flag a probe arm runs.` + : `"${flag}" at position ${i} is not part of a probe arm's invocation.`); + } i += 1; continue; } @@ -460,8 +480,15 @@ function flagProblems(flags, values) { } // Presence is read from what the walk SAW, not from `includes`. A flag // sitting in a value position is not a flag the arm ran. - for (const required of REQUIRED_FLAGS) { - if (!seen.has(required)) problems.push(`flags omit ${required}.`); + // + // It is the ACCEPTANCE test's question. An arm that omitted a required flag + // ran without the isolation that flag buys, which is a probe that failed + // rather than a file that is broken, and the required set moves whenever this + // repository amends the protocol. + if (values) { + for (const required of REQUIRED_FLAGS) { + if (!seen.has(required)) problems.push(`flags omit ${required}.`); + } } return { problems, seen }; } @@ -812,6 +839,8 @@ export function checkRecord(record, name = 'record') { // is a probe that failed, and `deriveOutcome` says so through `isolated`. // Refusing it here made it a broken file instead, which is how the design's // "a recorded failure is a result" stopped holding for isolation failures. + // That covers an omitted flag and an unknown one as well as a wrong value, + // because the flag set is a protocol choice this repository versions. for (const p of flagShapeProblems(record.flags)) say(p); // The probe authenticates from a credential in the environment, by either diff --git a/docs/adr/0024-a-probe-measures-the-attachment-surface.md b/docs/adr/0024-a-probe-measures-the-attachment-surface.md index 9765180..4e1269f 100644 --- a/docs/adr/0024-a-probe-measures-the-attachment-surface.md +++ b/docs/adr/0024-a-probe-measures-the-attachment-surface.md @@ -2,7 +2,7 @@ type: adr status: accepted decided: 2026-08-07 -issues: [21, 43, 94, 95] +issues: [21, 43, 94, 95, 113] --- # ADR-0024 — A probe arm enables the user source, and plants in the description @@ -329,3 +329,51 @@ identity tuple and to nothing wider: harness build `2.1.222`, platform model each record names. Nothing generalises across any element of it, as section 4.1 already says of everything else, and a reader who wants the served build reads it from the record rather than from this sentence. + +## Amended 2026-08-14 — flag presence is a value reading + +The split above fixed one half of the case it names. `checkRecord` read the +flag NAMES as well as the structure, so it refused a record that omitted a +required flag and a record that named a flag outside the set. Only a wrong +VALUE reached the acceptance test. + +That left the guarantee false in the shape it was written. A record omitting +`--strict-mcp-config` was a broken file. So was a record carrying `--verbose`. +The flag set had already moved once, on this ADR, and the next move that adds +or removes a NAME rather than a value would make every committed record +malformed again. Issue 113 reports it, and an adversarial review of pull +request #110 found it. + +**Decision.** Allowlist membership and required-flag presence move to the value +reading. `isolationProblems` now reads the names, the presence and the values, +and `deriveOutcome` reports all three as `isolated`. `flagShapeProblems` keeps +structural impossibility alone: flags that are not a non-empty array, an entry +that is not a string, a flag stated twice, a value-taking flag at the end of +the list, and a flag sitting where another flag's value belongs. The operator +ruled on 2026-08-14, on the fork issue 113 states. + +The guarantee now holds whole. A record collected under the wrong flags derives +FAIL, prints as FAIL, and can never read as a pass, and that covers the omitted +flag and the unknown flag as well as the wrong value. + +**The principle, stated once.** A check may refuse a record on a STABLE +IDENTITY fact of the collector. The pathway combination is one, and it stays a +shape refusal untouched, because no run of this collector could have written a +pathway no runner drives. A check may not refuse a record on a VERSIONED +PROTOCOL CHOICE. Flag names, required presence and `TRACE_LINE_LIMIT` are all +such choices. Each one moves when this repository amends the protocol, and a +check that refuses on it retires committed evidence on the day it moves. So a +versioned choice decides a READING, and never a record's validity. This +generalises the `TRACE_LINE_LIMIT` rule the section above already states. + +**What the parse does not change.** Every element is still consumed. An unknown +flag consumes one element under both readings, so the two walks see the same +list and `flagsSeen` answers the same question it answered before. + +**What would reopen this.** The corpus admits records from this repository's +own collector and from nothing else, which is what makes an unknown flag a +protocol question rather than an authenticity question. If `bench/probes/` ever +admits a record from outside that collector, the allowlist becomes a genuine +authenticity surface, and refusing an unknown flag would then be a claim about +where the file came from rather than about which protocol version wrote it. +Revisit this decision on that day. diff --git a/test/probe.test.js b/test/probe.test.js index 5db7c11..be3b9fb 100644 --- a/test/probe.test.js +++ b/test/probe.test.js @@ -489,14 +489,13 @@ test('the environment class names the home, not the route', () => { // The refusal below the flag check promises nothing is quoted. The flag check // above it quoted its value verbatim, so a credential-shaped flag leaked. test('a credential-shaped value in any message is withheld at emission', () => { - // A stray positional is quoted back by the refusal that names it, so it is - // the shape that proves redaction happens at emission. The refusal for a - // fixed-value flag used to quote too, and no longer does: `checkRecord` reads - // the flag SHAPE now and leaves the values to the acceptance test. - const leaky = record({ - flags: ['-p', '--model', 'opus', '--setting-sources', 'user', - 'sk-ant-oat01-LEAKEDCREDENTIAL0123', '--strict-mcp-config', '--output-format', 'json'], - }); + // The build-disagreement refusal names both builds, so it is the message + // `checkRecord` still emits from the record's own bytes, and the shape that + // proves redaction happens at emission. The flag refusals used to quote too, + // and no longer do: `checkRecord` reads the flag SHAPE now, and the names, + // the presence and the values all belong to the acceptance test. + const leaky = record(); + leaky.installed.model_id = 'sk-ant-oat01-LEAKEDCREDENTIAL0123'; const problems = checkRecord(leaky, 'r.json'); const joined = problems.join(' '); assert.equal(joined.includes('LEAKEDCREDENTIAL'), false, 'no message may quote it'); @@ -964,6 +963,59 @@ test('a record under the old flag spelling is well formed, and derives a failure assert.equal(deriveOutcome(old).passes, false); }); +// Issue 113. The first split fixed the wrong-VALUE case alone, and membership +// and presence stayed on the shape side — so a set that moves by ADDING or +// REMOVING a flag name still made every committed record a broken file. Three +// cases, and the third is the one that already worked, pinned beside them so a +// later reader sees one rule rather than an exception. +test('a record whose arm omitted a required flag is well formed, and derives a failure', () => { + const omitted = record({ + flags: ['-p', '--model', 'opus', '--setting-sources', 'user', '--output-format', 'json'], + }); + assert.deepEqual(checkRecord(omitted), [], + 'presence is a protocol choice, so an omitted flag is a failed probe'); + assert.match(isolationProblems(omitted.flags).join(' '), /flags omit --strict-mcp-config/); + assert.equal(deriveOutcome(omitted).isolated, false); + assert.equal(deriveOutcome(omitted).passes, false); +}); + +test('a record naming a flag outside the set is well formed, and derives a failure', () => { + const extra = record({ flags: [...armFlags('opus'), '--verbose'] }); + assert.deepEqual(checkRecord(extra), [], + 'the allowlist is a protocol choice, so an unknown flag is a failed probe'); + assert.match(isolationProblems(extra.flags).join(' '), + /--verbose is not a flag a probe arm runs/); + assert.equal(deriveOutcome(extra).isolated, false); + assert.equal(deriveOutcome(extra).passes, false); +}); + +test('a record carrying the wrong value for a flag is well formed, and derives a failure', () => { + const wrong = record({ + flags: ['-p', '--model', 'opus', '--setting-sources', '', '--strict-mcp-config', + '--output-format', 'json'], + }); + assert.deepEqual(checkRecord(wrong), []); + assert.match(isolationProblems(wrong.flags).join(' '), /--setting-sources carried ""/); + assert.equal(deriveOutcome(wrong).isolated, false); + assert.equal(deriveOutcome(wrong).passes, false); +}); + +// The shape reading keeps what no revision of the collector could produce. +// `runArms` builds one flag set above the loop, so a duplicated flag, a +// non-string entry, and a value-taking flag with nothing after it each describe +// a file this tool could not have written. +test('the shape reading keeps the structural impossibilities, and only those', () => { + assert.match(flagShapeProblems('not a list').join(' '), /non-empty array/); + assert.match(flagShapeProblems([]).join(' '), /non-empty array/); + assert.match(flagShapeProblems(['-p', 7]).join(' '), /position 1 is not a string/); + assert.match(flagShapeProblems([...armFlags('opus'), '--strict-mcp-config']).join(' '), + /--strict-mcp-config appears twice/); + assert.match(flagShapeProblems([...armFlags('opus'), TRACE_FLAG]).join(' '), + /--debug-file carries no value/); + assert.match(flagShapeProblems(['-p', '--model', '--setting-sources', 'user', + '--strict-mcp-config', '--output-format', 'json']).join(' '), /--model carries no value/); +}); + // The trace, which section 4.1 asks for and no record carried until now. test('a trace is null or the harness\'s own lines, and nothing else', () => { assert.deepEqual(traceProblems(null), []); From a9b0429bc0dc700f559f80d1b9bc653f18984aee Mon Sep 17 00:00:00 2001 From: logan Date: Fri, 14 Aug 2026 17:36:08 -0400 Subject: [PATCH 2/2] fix: the named set moves, and the invocation grammar does not (#113) Two review findings on the same root cause: the split moved more than membership. `armFlags` returns a literal array, so a floating element and a duplicated flag are impossible under every revision, whatever the set becomes. Both return to the shape reading. Duplication is now read before membership, so it is name-agnostic and `--verbose --verbose` no longer passes a rule that names no flag. The positional refusal names the position rather than the element, so a credential-shaped positional no longer withholds the whole line. --- AGENTS.md | 20 ++++-- bench/probe.mjs | 63 +++++++++++++------ ...a-probe-measures-the-attachment-surface.md | 32 ++++++++-- test/probe.test.js | 45 ++++++++++++- 4 files changed, 129 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b638d44..fe6dd25 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -489,12 +489,20 @@ number published in `bench/README.md`. stable identity fact of the collector, and never on a protocol choice this repository versions. Flag names, required presence and `TRACE_LINE_LIMIT` are versioned choices, so each decides a reading and - never a record's validity. `flagShapeProblems` keeps structural - impossibility alone: flags that are not a non-empty array, an entry - that is not a string, a flag stated twice, a value-taking flag at the - end of the list, and a flag sitting where another flag's value belongs. - Nothing is weakened: the record still derives FAIL and can never read - as a pass. ADR-0024. + never a record's validity. The named set moves, and the invocation + GRAMMAR does not: `armFlags` returns a literal array, so no revision + floats an element between a flag and its value or states one flag + twice. So `flagShapeProblems` keeps structural impossibility alone: + flags that are not a non-empty array, an entry that is not a string, an + element that is neither a flag nor a flag's value named by position + alone, a flag stated twice whatever its name, a value-taking flag at + the end of the list, and a flag sitting where another flag's value + belongs. Reading membership before duplication let `--verbose + --verbose` past a rule that names no flag, and a shape message names a + position rather than an element, because quoting one withheld the whole + line through `redact` whenever it was credential-shaped. Nothing is + weakened: the record still derives FAIL and can never read as a pass. + ADR-0024. - A probe arm runs `--setting-sources user`, and `bench/run.sh` selects its spelling from `--rules`: `none` gives the no-guidance control `''`, and `user` gives the treatment arm `user`. The divergence is between a diff --git a/bench/probe.mjs b/bench/probe.mjs index 537497b..a9b16bb 100644 --- a/bench/probe.mjs +++ b/bench/probe.mjs @@ -360,22 +360,26 @@ function keyPaths(value, prefix = '') { * `isolated`. * * `flagShapeProblems` is the RECORD CHECK, and it reads structural - * impossibility alone. `runArms` builds ONE flag set above the loop, so a - * record carries a list of strings where each named flag appears once and each - * value-taking flag is followed by its value. Nothing else there is a fact - * about the collector. So the shape reading keeps five refusals: flags that are - * not a non-empty array, an entry that is not a string, a flag stated twice, a + * impossibility alone. `armFlags` returns a LITERAL array and `runArms` builds + * one of them above the loop, so a record carries a list of strings that + * alternates flags with their values, each flag stated once, and no element + * floating between them. That much holds under any revision. So the shape + * reading keeps six refusals: flags that are not a non-empty array, an entry + * that is not a string, an element that is neither a flag nor a flag's value + * named by position alone, a flag stated twice whatever its name, a * value-taking flag at the end of the list, and a flag sitting where another * flag's value belongs. `checkRecord` asks this one. * - * The line between them is stable identity against versioned protocol. A check - * may refuse a record on a fact about the collector that does not move — the - * pathway combination is one, because no run could have written a pathway no - * runner drives. It may not refuse a record on a PROTOCOL CHOICE this - * repository versions. Which flags a probe arm runs is such a choice, and so is - * `TRACE_LINE_LIMIT`. The set moved on 2026-08-07 and it will move again, and a - * check that refuses on it retires committed evidence every time it moves. So a - * versioned choice decides a READING and never a record's validity. + * The line between them is stable identity against versioned protocol, and the + * flag walk carries it twice over. A check may refuse a record on a fact about + * the collector that does not move. The pathway combination is one, because no + * run could have written a pathway no runner drives, and the INVOCATION GRAMMAR + * is another, because the array is constructed rather than parsed. A check may + * not refuse a record on the NAMED SET, which this repository versions. Which + * flags a probe arm runs is such a choice, and so is `TRACE_LINE_LIMIT`. The set + * moved on 2026-08-07 and it will move again, and a check that refuses on it + * retires committed evidence every time it moves. So a versioned choice decides + * a READING and never a record's validity. * * Both readings were one function, and `checkRecord` asked the acceptance test. * That made a probe which ran the wrong flags a MALFORMED RECORD rather than a @@ -409,6 +413,11 @@ export function flagShapeProblems(flags) { * `includes`, because a flag sitting in a value position is not a flag the arm * ran. One walk answers both questions, so no second reading of an invocation * can drift from the one the acceptance test uses. + * + * It holds every flag the walk read in a flag position, and not every ALLOWED + * one. A flag outside the set is still a flag the arm ran, and the duplicate + * check has to see it — reading membership first let `--verbose --verbose` past + * a rule that names no flag. */ export function flagsSeen(flags) { return flagProblems(flags, false).seen; @@ -434,21 +443,35 @@ function flagProblems(flags, values) { i += 1; continue; } + if (!flag.startsWith('-')) { + // A floating element, which is the GRAMMAR and not the named set. The + // array is CONSTRUCTED rather than parsed, so every element is a flag or + // the value of the flag before it under any revision, whatever the set + // becomes. So this stays a shape refusal. + // + // It names the position and never the element. The old message quoted it, + // and a credential-shaped positional then withheld the whole line through + // `redact` — safe, and unreadable. A position carries nothing of the + // record, so the diagnostic survives. + problems.push(`the entry at position ${i} is not part of a probe arm's invocation.`); + i += 1; + continue; + } + // Duplication is read BEFORE membership, and it is name-agnostic. `runArms` + // builds one flag set, so NO flag is stated twice under any revision, and + // reading the duplicate only for a flag this repository currently knows let + // `--verbose --verbose` through the shape check. + if (seen.has(flag)) problems.push(`${flag} appears twice, so the arm's surface is unclear.`); + seen.add(flag); if (!ALLOWED_FLAGS.includes(flag)) { // A flag outside the set is a configuration surface the arm never opened, // so the ACCEPTANCE test refuses it. The shape reading passes it over, // because the set is a protocol choice this repository versions. Either // way it consumes one element, so both readings walk the list alike. - if (values) { - problems.push(flag.startsWith('-') - ? `${flag} is not a flag a probe arm runs.` - : `"${flag}" at position ${i} is not part of a probe arm's invocation.`); - } + if (values) problems.push(`${flag} is not a flag a probe arm runs.`); i += 1; continue; } - if (seen.has(flag)) problems.push(`${flag} appears twice, so the arm's surface is unclear.`); - seen.add(flag); if (!FLAGS_TAKING_A_VALUE.includes(flag)) { i += 1; continue; diff --git a/docs/adr/0024-a-probe-measures-the-attachment-surface.md b/docs/adr/0024-a-probe-measures-the-attachment-surface.md index 4e1269f..278e73e 100644 --- a/docs/adr/0024-a-probe-measures-the-attachment-surface.md +++ b/docs/adr/0024-a-probe-measures-the-attachment-surface.md @@ -348,9 +348,10 @@ request #110 found it. reading. `isolationProblems` now reads the names, the presence and the values, and `deriveOutcome` reports all three as `isolated`. `flagShapeProblems` keeps structural impossibility alone: flags that are not a non-empty array, an entry -that is not a string, a flag stated twice, a value-taking flag at the end of -the list, and a flag sitting where another flag's value belongs. The operator -ruled on 2026-08-14, on the fork issue 113 states. +that is not a string, an element that is neither a flag nor a flag's value +named by position alone, a flag stated twice whatever its name, a value-taking +flag at the end of the list, and a flag sitting where another flag's value +belongs. The operator ruled on 2026-08-14, on the fork issue 113 states. The guarantee now holds whole. A record collected under the wrong flags derives FAIL, prints as FAIL, and can never read as a pass, and that covers the omitted @@ -366,9 +367,32 @@ check that refuses on it retires committed evidence on the day it moves. So a versioned choice decides a READING, and never a record's validity. This generalises the `TRACE_LINE_LIMIT` rule the section above already states. +**The named set moves, and the invocation grammar does not.** A review of the +pull request carrying this section drew the line one notch too far, and this +records the correction. `armFlags` returns a LITERAL array, so a record's flag +list alternates flags with their values, states each flag once, and floats no +element between them — under every revision, whatever the set becomes. The +grammar is therefore a stable identity fact and it stays on the shape side, and +only membership and presence moved. Two cases were wrong for one round. A +duplicated UNKNOWN flag escaped the shape check entirely, because the +membership branch returned before the duplicate branch ran, so a rule that +names no flag was read only for the flags this repository currently knows. And +a floating positional moved value-side with the names, though no revision can +produce one. + +**A shape message names a position and never an element.** The positional +refusal used to quote the element back. `redact` withheld the whole line when +that element was credential-shaped, which was safe and unreadable. A position +carries nothing of the record, so the diagnostic survives the case it was built +for. The duplicate message still names the flag, because a dash-led token IS +the flag rather than a value it carries, and `redact` at emission covers the +one case where such a token is credential-shaped. + **What the parse does not change.** Every element is still consumed. An unknown flag consumes one element under both readings, so the two walks see the same -list and `flagsSeen` answers the same question it answered before. +list. `flagsSeen` now holds every flag the walk read in a flag position rather +than every ALLOWED one, which is what lets the duplicate check see a name this +repository does not know, and no consumer's question changes answer. **What would reopen this.** The corpus admits records from this repository's own collector and from nothing else, which is what makes an unknown flag a diff --git a/test/probe.test.js b/test/probe.test.js index be3b9fb..38b6ff4 100644 --- a/test/probe.test.js +++ b/test/probe.test.js @@ -311,7 +311,7 @@ test('a stray positional is refused, not skipped', () => { const stray = ['-p', 'extra-prompt', '--model', 'opus', '--setting-sources', '', '--strict-mcp-config', '--output-format', 'json']; assert.match(isolationProblems(stray).join(' '), - /"extra-prompt" at position 1 is not part of a probe arm's invocation/); + /the entry at position 1 is not part of a probe arm's invocation/); assert.equal(deriveOutcome({ ...record(), flags: stray }).isolated, false); }); @@ -1016,6 +1016,49 @@ test('the shape reading keeps the structural impossibilities, and only those', ( '--strict-mcp-config', '--output-format', 'json']).join(' '), /--model carries no value/); }); +// The named set moves by amendment, so it cannot decide validity. The +// invocation GRAMMAR does not move: `armFlags` returns a literal array, so no +// revision of this collector can float an element between a flag and its value, +// or state one flag twice. Both stay on the shape side, and neither reads the +// flag's name to decide. +test('a floating element is refused by shape, and named by position alone', () => { + const stray = ['-p', 'extra-prompt', '--model', 'opus', '--setting-sources', 'user', + '--strict-mcp-config', '--output-format', 'json']; + const problems = flagShapeProblems(stray).join(' '); + assert.match(problems, /the entry at position 1 is not part of a probe arm's invocation/); + assert.equal(problems.includes('extra-prompt'), false, + 'the message names the position, never the element'); + assert.match(checkRecord(record({ flags: stray })).join(' '), /position 1/); +}); + +test('a duplicated flag is refused by shape whatever its name', () => { + // The name is unknown, so membership is the acceptance test's to report. The + // duplication is not: one flag set can never state one flag twice. + const twice = [...armFlags('opus'), '--verbose', '--verbose']; + assert.match(flagShapeProblems(twice).join(' '), /--verbose appears twice/); + assert.match(checkRecord(record({ flags: twice })).join(' '), /--verbose appears twice/); + const problems = isolationProblems(twice).join(' '); + assert.match(problems, /--verbose appears twice/); + assert.match(problems, /--verbose is not a flag a probe arm runs/); +}); + +// The positional refusal was the original demonstration of redaction at +// emission, and it moved value-side for one round. It is back, and it now names +// the position rather than the element, so nothing is withheld and nothing +// leaks. Both properties are asserted here. +test('a credential-shaped positional is refused without being quoted', () => { + const leaky = record({ + flags: ['-p', 'sk-ant-oat01-LEAKEDCREDENTIAL0123', '--model', 'opus', + '--setting-sources', 'user', '--strict-mcp-config', '--output-format', 'json'], + }); + const problems = checkRecord(leaky, 'r.json'); + const joined = problems.join(' '); + assert.match(joined, /the entry at position 1 is not part of a probe arm's invocation/, + 'a floating element is a shape problem, so `checkRecord` reports it'); + assert.equal(joined.includes('LEAKEDCREDENTIAL'), false, 'no message may quote it'); + assert.ok(problems.every((line) => line.startsWith('r.json: '))); +}); + // The trace, which section 4.1 asks for and no record carried until now. test('a trace is null or the harness\'s own lines, and nothing else', () => { assert.deepEqual(traceProblems(null), []);