Require a declared failure mode and a stated noise floor for verification methods in SAFE recommendations

[Blockchain-365]

"From Lessons to Controls" requires every recommendation to specify a reproducible verification method, plus adoption, effectiveness and review metrics. Both requirements assume the verification method returns a trustworthy result. Two failure modes of the method itself are not currently expressible anywhere in the framework:

1. The method can fail open, returning a value that parses as valid but was never produced by an actual check.
2. The method has its own variance, so a control can be recorded as holding or regressing on a change smaller than the measurement noise.

Neither is hypothetical. We hit both in production, in an agent-assessment pipeline that uses a language model as the scoring judge.

1. A verification step that failed open, silently

Our judge calls ran with max_tokens: 700, chosen on the assumption that emitting a score line needs very few tokens. The assumption was wrong. The model's internal reasoning tokens are charged against the same ceiling. When they exhaust it, the API returns HTTP 200 with an empty content string. Our client read choices[0].message.content, got "", and raised nothing. We were not inspecting finish_reason. The score line had never been generated at all.

Downstream, the empty response parsed as "no score for this dimension." The dimension was recorded as null, and the weighted aggregate renormalized across the dimensions that remained. The pipeline produced a composite score, on time, from fewer inputs than it claimed to use. Measured rate before the fix: 1 to 2 percent of all judgments.

Nothing alerted. We found it because the same failure contaminated one of our own experiment datasets, which we had reason to inspect by hand.

Two recent probe calls, for scale: 42 of 65 completion tokens were reasoning_tokens in one, 207 of 223 in the other. Raising the ceiling to 1500 removed the failure. A higher ceiling does not make the model write more, so output length and cost were both effectively unchanged.

The general form: any verification step that calls a reasoning model and parses a short structured answer out of it can fail this way, and it fails open. It keeps reporting.

2. Verification methods have a noise floor, and it is usually unstated

Method: freeze the exact prompt and the exact response from real runs, then re-judge N times with nothing else changed. The subject under test cannot move, so all observed variance is instrument variance. Zero calls to the system being assessed, which also makes it cheap enough to run often.

Sample: 13 completed runs, 113 scored items, 15 repeats each, for 1,695 judging calls, plus a further 1,500 after we shipped fixes.

Metric (frozen input)	Before	After
Total-score retest SD, median	0.86	0.22
P95 of |difference between two retests|, median	2.2	0.4
P95, worst case	7.4	4.3

The median was not the number that mattered. Our pipeline treats a dimension below 80 as weak. On byte-identical frozen input, the weakest dimension flipped that verdict 37.4 percent of the time before hardening, and 9.3 percent after. A gate reading that dimension would have passed and failed the same evidence roughly one time in three, with no way to tell from the output which run it was looking at.

More sampling does not fix this. We simulated raising the repeat count, and worst-case P95 fell from 7.4 to 5.6 and then stopped there. The residual is not jitter.

Six of the 113 items were bimodal, with the judge alternating between two grades on identical input ({70: 2, 100: 11} in one case, {0: 12, 40: 3} in another). That is rubric ambiguity. A median only selects the more populous side, and when the split approaches even, the median itself flips. The fix was rubric disambiguation, not more repeats.

One negative result cost us a deployment, and it bears on the fix above. Rewriting the rubric into terse, mechanically checkable output helped for a mechanical judgment, where the task is counting satisfied constraints: item SD fell from 1.88 to 0.53. The same treatment applied to an adversarial judgment made it substantially worse: SD rose from 1.19 to 4.52, and a distribution of {70: 2, 100: 11} split into {0: 2, 70: 3, 90: 10}. Compression appears to remove reasoning room that an adversarial judgment needs. We reverted that half, and kept the failed draft in the repository as a counter-example.

3. What we suggest

Two additions, both optional and additive.

a. Where a recommendation specifies a reproducible verification method, also state:

• Failure mode of the method: does it fail closed, producing no result, or can it fail open, producing a result that looks valid but was not measured? If it can fail open, what detects that?
• Noise floor: retest variability under fixed input, reported as median and worst case. Reporting only a median hides exactly the cases a gate will trip on.
• Decision-flip rate: where the method feeds a pass/fail threshold, how often the same evidence yields opposite verdicts.

b. Make the assurance layer reviewable. The Review Framework covers eight layers of the operating stack. We suggest either a ninth row or an extension of Supply chain, which already asks whether an evaluation partner invalidated assumed controls. Review question: did the verification that established this control's effectiveness return a valid result, and was the observed change larger than that method's own variability?

4. Anticipated objections

This is test methodology, not incident response. SAFE's output is a versioned catalogue of defensive recommendations that others adopt and verify. If a published verification method can fail open, every adopter inherits a control that reports as effective without having been tested, and the defect propagates through the catalogue. That is the systemic risk the proposal exists to reduce, arriving through the assurance path rather than the runtime path.

This adds reporting burden. Both additions are optional. An organization that has not characterized its method omits the field. That omission is itself informative, and it is better than an unstated assumption that the method is exact.

Does this only apply to model-based verification? The silent-failure mode is sharpest there. A noise floor applies to any non-deterministic verification, including sampling-based checks and timing-sensitive ones.

You sell assessment, so of course you want assessment documented. Fair. We have a commercial interest here. The proposal also binds us, and we do not currently meet it. We can report a noise floor for judge variance only, because the other two sources of variance in our own product are untested. Under the rule we are proposing, our published figures would be marked incomplete until we finish that work.

5. Standing and limitations

We are AgentVitals (ai.ddl99.com), operated by Beijing DDL Creative Tech Co., Ltd. It is a small commercial platform that assesses deployed AI agents. The figures above come from our own production pipeline rather than a research programme, and they carry these limits:

• The variance we measured is judge variance only, the instrument's own noise with the subject frozen. Agent-level randomness and probe-variant variance are untested, and we expect variant variance to be the largest of the three. These are not overall test-retest reliability figures and we do not present them as such.
• The panel is 3 agents across 13 runs. Adequate for a layer that is agent-independent by construction, and not adequate for any cross-platform claim.
• One judge model (DeepSeek V4 Flash). The reasoning-token accounting is provider-specific in its details. The class of failure is not.
• One rubric change of ours remains unverified at the low end: the sample contained no run that failed the constraint being counted, so its ability to discriminate weak performance is a design inference, not a measurement.
• One of our own conclusions here was wrong. We had attributed a 32-point swing in one dimension to judge noise and hardened it with a median-of-three. The frozen-transcript test then showed that dimension's judge variance was exactly zero, across 13 items at 15 repeats each. The hardening was harmless but aimed at the wrong layer, and the real cause sat elsewhere. That mistake is the main reason we think attributing variance to a layer before hardening that layer belongs in the framework rather than being left to practice.

We would be glad to contribute the frozen-transcript protocol as a short methods note, along with the de-identified dataset (scores only, no prompts or responses) and the analysis scripts, if the working group finds the direction useful.

[DmitrL-dev]

Strong support for both halves, and I can add field evidence for the first plus a suggestion that makes it mechanically checkable rather than declarative.

The fail-open mode is the default outcome of writing a check for your own work.

In one day of building verification into a security gateway (an AI proxy that compares a proposed tool call against what the user authorised), four of our own checks failed open. Each returned a result that parsed as valid. None raised anything.

the check	what it returned	why nothing failed
a CI runner emitting each gate leg's script	zero lines, three times	the leg functions were sourced below the emit call, and a bash function does not exist until its definition has been executed. CI would have run three empty scripts and reported green.
the CI's own verdict writer	no result file at all on a failing leg	{ … } || { record_failure; } — a brace group is not a subshell, so exit 1 inside it ends the script and the || branch never runs. The previous green result would have stayed "latest".
a contrast check over a stylesheet's dark theme	19:1, passing	the helper located a CSS block with indexOf(".dark"), and .dark appears first in a comment on line 8 — so it read the light values and compared them against the light background. It had never checked the dark theme.
a test asserting twelve identical charges are caught as a series	pass	each charge got a fresh ledger, so no charge was ever a repeat of another. The test exercised nothing it claimed to.

The shape is identical to your max_tokens case: a call returning 200 with an empty body, read as a legitimate absence and propagated.

Suggestion 1 — require the method to be shown failing, not merely to declare how it might.

"Declare the failure mode" depends on the author having imagined it, and in all four cases above the author had not. A cheaper requirement catches them anyway: a verification method is accompanied by evidence that it fails when the control it verifies is removed. Mutation — delete or invert the guarded thing, run the method, record that it reported failure.

All four of ours were found that way, mechanically, in seconds. Applied to your example, it catches it: remove the score line from the judge's output and confirm the pipeline reports a missing dimension rather than a composite. It is also stronger than a declaration because it is an artefact rather than a claim, and it fits the RFC's existing language about reproducible verification methods — the reproducible thing simply has to include the negative case.

This pairs with your point about hardening the wrong layer: a mutation that does not change the method's verdict tells you the layer you were about to harden is not the one being measured.

Suggestion 2 — a stated noise floor needs a stated ceiling beside it.

We shipped a retrieval-precision check with a floor of 0.50 that could never have passed: some queries had only two relevant documents in the corpus, so precision-at-5 was capped at 0.40 for those and 0.50 in the mean. The floor was above the achievable maximum. It looked demanding and its outcome was fixed before it ran — the same defect as a threshold below the noise, in the other direction.

So alongside the noise floor: the maximum score the method can return on the corpus it runs against, with the metric expressed as a share of that maximum rather than as a bare number. Ours now reads 90% of achievable rather than 0.450, and only the first form means anything to a reader who does not know the corpus.

Neither suggestion requires a new format.

[Blockchain-365]

Agreed on the first suggestion, and it is stronger than what we proposed. A declared failure mode is bounded by what the author thought to imagine, and our own case is an example. Our client did handle failure: three retries with backoff, raise_for_status, an exception path. A 200 carrying an empty string was not a case it considered, so it returned that string as a successful result on the first attempt. Mutation evidence does not depend on having imagined the case. It is also, as you say, an artefact rather than a claim, which makes it reviewable by someone who was not there.

One precondition belongs with it. It is what kept our failure silent for as long as it was.

The mutation has to produce a state distinguishable from a legitimate absence. In our pipeline a judging failure resolved to a null score for that dimension, and the weighted aggregate renormalised over the dimensions that remained. That is also the correct behaviour for a dimension with genuinely nothing to score, which happens routinely: an agent with no conversation history has nothing to measure on several of ours. Both paths produced the same null, the same renormalisation, and a composite score that looked ordinary. Deleting the score line and observing "composite still produced" would have shown a reviewer exactly what a legitimate absence shows. The mutation only becomes informative once the two are separable, which for us meant recording why a dimension is null rather than only that it is.

So the requirement is slightly stronger than "run the method against a removed control": the method has to be able to say which of the two it observed.

For non-deterministic methods, the mutation evidence needs repeats. A single mutation run inherits whatever variance the method already has. Six of our 113 scored items were bimodal, with the judge alternating between two grades on byte-identical input ({70: 2, 100: 11} in one, {0: 12, 40: 3} in another). On an item like that, one mutation run passes or fails partly by luck, and the recorded artefact would be misleading in a way that is hard to spot later. The cost is small if the mutation is cheap, which it usually is.

The artefact also has to be re-run when anything upstream of it moves. The 1 to 2 percent silent-failure rate we reported was measured against the rubrics we had at the time. When we later rewrote two of those rubrics to be more structured, the same max_tokens: 700 produced a 12 to 18 percent failure rate on the new ones, because the rewrite changed how much the model reasoned before answering. Not a line of the calling code changed. A mutation artefact produced once at design time would have captured the low rate, and would have sat in the record looking like evidence through the change that invalidated it. Whatever form the requirement takes, the artefact should carry the configuration it was produced under.

Your second suggestion we would just adopt. The retrieval-precision example is a cleaner illustration than anything we have: a threshold above the achievable maximum has its outcome fixed before it runs, and reporting a bare number hides that from everyone who does not know the corpus.

One extension, from the mirror-image version of the same defect. We have a rubric change whose low end is still unverified, for a reason that is not a ceiling: the sample contained no run that failed the constraint being counted, so all 13 items scored 100 and the discriminating range was never exercised. The metric could express it. The corpus never asked it to. So we would state not just the achievable maximum but the range the corpus actually exercises, which catches both your case and ours. A method reported as "90% of achievable" is still uninformative if every observation sat at one end.

The two requirements also compose. Mutation evidence establishes that the method responds when the control is removed, and gives the size of that response. The noise floor gives the size of the method's own variability. What decides whether the method can carry a pass/fail gate is the ratio, not either number alone. A mutation delta smaller than the noise floor is a method that cannot resolve the thing it is checking.

We have not measured a mutation delta, so the closest figure we can offer is the analogous one: between-subject spread divided by measurement noise, which moved from 12.5 to 22.1 across the rubric work described above. The practical consequence was concrete. Among eight high-scoring subjects there are seven adjacent pairs, and the number of those pairs separated by more than the measurement noise went from one to five.

That is what the ratio catches and neither number catches alone. Each score looked stable in isolation, the ranking they produced looked like an ordering, and for six of those seven pairs the gap doing the ordering was smaller than the noise underneath it.

[DmitrL-dev]

Your precondition is the right one and we would fold it into the requirement rather than leave it alongside. The mutation has to produce a state distinguishable from a legitimate absence, and that means the method needs somewhere to record why, not only that.

We arrived at the same shape from a different direction. Our decision type is three-valued — allow, escalate, deny — and escalate means precisely "cannot say", never folded into either neighbour. A comparison that finds a contradiction denies; one that cannot see a field escalates. Those are different states with different audit categories, which is what makes a removed control distinguishable from a control with nothing to compare. In our corpus 32 of the 152 legitimate cases are expected to be withheld — an undeclared tool is held for confirmation even when its arguments agree — and they are counted separately from false positives rather than netted against them. Your null-score case and our escalate case are the same requirement wearing different clothes.

On repeats: ours is a deterministic method — arithmetic and string equality, no model anywhere in the decision path — so a single mutation run is sufficient, and we would rather say that explicitly than let silence imply we had handled non-determinism.

It is worth separating two things your point could be read to cover, because only one of them binds a deterministic method. Repeat variance comes from the method; sampling variance comes from the corpus. Our detector figures are also deterministic — fixed engine set, fixed file order — and we record that as a limit on the claim in the evidence itself: one run, deterministic ordering, comparable between releases, not general statistics. What we do not have for those is an estimate of how much the numbers move across differently drawn corpora, which is your sixth point rather than your third. A requirement that asks for repeats will be satisfied trivially by any deterministic method while leaving that gap entirely open, so we would state both.

Your fourth point is the one we would most want in the RFC, and we can offer a case that arrived after we wrote to you rather than before.

We had measured our action guard on a 345-case corpus: recall 1.000 on 193 manipulations, false positives 0.000 over 152 legitimate purchases. We mutation-tested it, in your sense, before reporting — reintroducing any one of three closed defects moves the figures (recall 0.984 and 0.979; false positives 0.026 and 0.020), and defect-tagged cases stay inside the headline numbers rather than being excluded. By the criteria we sent you, that artefact was sound.

On 2026-08-08 we found that the measurement had been taken by calling the comparison directly, beneath the HTTP layer, on a buffered response body. Real clients set stream: true — every chat interface does by default — and a streamed response returns on an earlier path that the guard never sees. So the control did not run at all for a streaming caller: no error, no header, the authorisation annotation accepted, stripped and ignored. Measured on the wire before the fix: a payment of 1999.00 against an authorisation of 199.00 reached the client with status 200 and the tool call intact.

The scope, since a figure like that travels without its context: the guard is opt-in, it is a pre-release control with no production callers relying on it, and the path was closed the same day it was found. We are reporting it because the interesting part is not the defect but what the artefact did — it was produced correctly, by a method we had just finished arguing for, and it described a control that a single JSON field switched off.

That is your point, one turn harder. Your configuration became invalid through a change: the rubrics moved and the artefact stayed. Ours was never valid for production traffic in the first place, and nothing in the artefact said what it assumed. The number was correct, reproducible, and measured under a configuration real callers do not use.

So we would state the requirement as: the artefact carries the configuration it was produced under, and the configuration is one that occurs in deployment — or the divergence is recorded as a limit on the claim. "Re-run it when anything upstream moves" catches your case. It does not catch ours, because nothing moved.

The generalisation we drew, for whatever it is worth to the RFC: when a method has more than one path to the same output, the artefact has to name which path it exercised. We now check that as a class rather than as an instance — every control that only works on a buffered body must either have a streaming twin or force the response to buffer, and a test reads the enforcement codes actually recorded on the buffered path to demand one or the other. Two controls sit there; one had been covered since 2026-07 and the reasoning was written down, but in terms of that control alone, so the one added later inherited nothing.

Your sixth point we adopt as stated, and our corpus gives a worked example of what "the range the corpus exercises" should look like when reported. Ours exercises the discriminating range at its tightest: against a ceiling of 500.00 it contains a legitimate case at exactly 500.00 and an attack at 500.01 — one cent, the smallest step the currency allows — plus a legitimate half deliberately weighted towards shapes that look like attacks: a subdomain of the authorised vendor, grouped digits, a zero-width space inside an account number that normalises to the same account, a standing grant used exactly as often as it declared. A false-positive rate measured over easy negatives measures nothing, because the method passes those by construction. That sentence and your "every observation sat at one end" are the same objection.

On the ratio, we can offer the degenerate end of your framework as a data point. Our noise floor is exactly zero, so mutation deltas of 0.016 and 0.021 in recall are decisive here while the same deltas on a judged metric would be indistinguishable from variance. That is an argument for your formulation rather than against it: the ratio explains why a small delta is sufficient in one place and meaningless in another, which neither number does alone. Reporting the delta without the floor is how a method gets trusted at the wrong scale.

Your 12.5 → 22.1 figure, and particularly the seven adjacent pairs of which six were separated by less than the noise underneath them, is the clearest statement of the failure we have seen. A ranking that looks like an ordering is worse than no ranking, because it is acted on.

[DmitrL-dev]

Your precondition is the right one and we would fold it into the requirement rather than leave it alongside. The mutation has to produce a state distinguishable from a legitimate absence, and that means the method needs somewhere to record why, not only that.

We arrived at the same shape from a different direction. Our decision type is three-valued — allow, escalate, deny — and escalate means precisely "cannot say", never folded into either neighbour. A comparison that finds a contradiction denies; one that cannot see a field escalates. Those are different states with different audit categories, which is what makes a removed control distinguishable from a control with nothing to compare. In our corpus 32 of the 152 legitimate cases are expected to be withheld — an undeclared tool is held for confirmation even when its arguments agree — and they are counted separately from false positives rather than netted against them. Your null-score case and our escalate case are the same requirement wearing different clothes.

On repeats: ours is a deterministic method — arithmetic and string equality, no model anywhere in the decision path — so a single mutation run is sufficient, and we would rather say that explicitly than let silence imply we had handled non-determinism.

It is worth separating two things your point could be read to cover, because only one of them binds a deterministic method. Repeat variance comes from the method; sampling variance comes from the corpus. Our detector figures are also deterministic — fixed engine set, fixed file order — and we record that as a limit on the claim in the evidence itself: one run, deterministic ordering, comparable between releases, not general statistics. What we do not have for those is an estimate of how much the numbers move across differently drawn corpora, which is your sixth point rather than your third. A requirement that asks for repeats will be satisfied trivially by any deterministic method while leaving that gap entirely open, so we would state both.

Your fourth point is the one we would most want in the RFC, and we can offer a case that arrived after we wrote to you rather than before.

We had measured our action guard on a 345-case corpus: recall 1.000 on 193 manipulations, false positives 0.000 over 152 legitimate purchases. We mutation-tested it, in your sense, before reporting — reintroducing any one of three closed defects moves the figures (recall 0.984 and 0.979; false positives 0.026 and 0.020), and defect-tagged cases stay inside the headline numbers rather than being excluded. By the criteria we sent you, that artefact was sound.

On 2026-08-08 we found that the measurement had been taken by calling the comparison directly, beneath the HTTP layer, on a buffered response body. Real clients set stream: true — every chat interface does by default — and a streamed response returns on an earlier path that the guard never sees. So the control did not run at all for a streaming caller: no error, no header, the authorisation annotation accepted, stripped and ignored. Measured on the wire before the fix: a payment of 1999.00 against an authorisation of 199.00 reached the client with status 200 and the tool call intact.

The scope, since a figure like that travels without its context: the guard is opt-in, it is a pre-release control with no production callers relying on it, and the path was closed the same day it was found. We are reporting it because the interesting part is not the defect but what the artefact did — it was produced correctly, by a method we had just finished arguing for, and it described a control that a single JSON field switched off.

That is your point, one turn harder. Your configuration became invalid through a change: the rubrics moved and the artefact stayed. Ours was never valid for production traffic in the first place, and nothing in the artefact said what it assumed. The number was correct, reproducible, and measured under a configuration real callers do not use.

So we would state the requirement as: the artefact carries the configuration it was produced under, and the configuration is one that occurs in deployment — or the divergence is recorded as a limit on the claim. "Re-run it when anything upstream moves" catches your case. It does not catch ours, because nothing moved.

The generalisation we drew, for whatever it is worth to the RFC: when a method has more than one path to the same output, the artefact has to name which path it exercised. We now check that as a class rather than as an instance — every control that only works on a buffered body must either have a streaming twin or force the response to buffer, and a test reads the enforcement codes actually recorded on the buffered path to demand one or the other. Two controls sit there; one had been covered since 2026-07 and the reasoning was written down, but in terms of that control alone, so the one added later inherited nothing.

Your sixth point we adopt as stated, and our corpus gives a worked example of what "the range the corpus exercises" should look like when reported. Ours exercises the discriminating range at its tightest: against a ceiling of 500.00 it contains a legitimate case at exactly 500.00 and an attack at 500.01 — one cent, the smallest step the currency allows — plus a legitimate half deliberately weighted towards shapes that look like attacks: a subdomain of the authorised vendor, grouped digits, a zero-width space inside an account number that normalises to the same account, a standing grant used exactly as often as it declared. A false-positive rate measured over easy negatives measures nothing, because the method passes those by construction. That sentence and your "every observation sat at one end" are the same objection.

On the ratio, we can offer the degenerate end of your framework as a data point. Our noise floor is exactly zero, so mutation deltas of 0.016 and 0.021 in recall are decisive here while the same deltas on a judged metric would be indistinguishable from variance. That is an argument for your formulation rather than against it: the ratio explains why a small delta is sufficient in one place and meaningless in another, which neither number does alone. Reporting the delta without the floor is how a method gets trusted at the wrong scale.

Your 12.5 → 22.1 figure, and particularly the seven adjacent pairs of which six were separated by less than the noise underneath them, is the clearest statement of the failure we have seen. A ranking that looks like an ordering is worse than no ranking, because it is acted on.

[Blockchain-365]

Folding the precondition into the requirement is the right call, and your three-valued decision states it better than ours does. We record why a dimension has no score. You make "cannot say" a first-class outcome and count it separately, and that second half is the part we were missing. Recording the reason while still aggregating over whatever remains lets a withheld case disappear into a renormalised total, which is what the record was there to prevent.

On repeats we would put the line somewhere other than where you put it, and a measurement from this week is why.

Your case for exempting deterministic methods is that arithmetic and string equality have no variance to average out. That holds for the decision procedure. It does not hold for the step where the procedure acquires its input, and your streaming defect is the clearest evidence of it we have seen. The comparison was deterministic and correct. It never ran, because the input reached the client along a route the artefact had never exercised. Repeating that measurement a hundred times would have returned the same clean numbers a hundred times.

So the requirement that binds is not really about variance. The artefact has to be produced through the path the input actually takes in deployment, and repeats are needed only where that path has variance of its own. Put that way, your case and the repeat clause stop being two separate requirements.

Ours does have that variance, and it comes from neither the corpus nor the decision rule. Our judge scores each item three times and takes the median. Same prompt, same model, same temperature, same input: a fraction of those calls come back HTTP 200 with an empty body, because the model spent its token allowance on internal reasoning before emitting the answer line. Nothing in the aggregation is random. The call underneath it is.

Measured this week on one dimension, small samples: 1 of 8 calls on a real transcript, 3 of 8 on a degenerate one, 4 of 16 combined. Two other prompt families in the same system returned 0 of 8 each, and a full production run returned 2 of 30. The rate is not uniform across prompt shapes, and at these sample sizes we cannot say why. Our current guess is that the prompts which lose calls are the ones asking for a lot of reasoning and a very short answer, which is the combination that lets the budget run out before anything is emitted.

That leads to a case we can add to your fourth point, because it is a third shape rather than another instance of yours.

Yours: the configuration never occurred in deployment, and nothing in the artefact said so. Ours from the last round: the configuration did occur, then moved, and the artefact stayed behind. This one: the configuration was right, the number was right, and the artefact still overstates, because it never says what it is a rate of.

We published a parse-success figure of 98.5 to 100 percent after raising the token limit. That figure is per dimension, measured after the median of three. The per-call rate on the dimension above is around 25 percent. Both numbers are correct, and they agree: three independent failures at 0.25 is 1.6 percent, which is the distance from 98.4 to 100. A reader has no way to tell which of the two we meant, and the natural reading is the per-call one, which is also the flattering one.

Nothing moved here and nothing was misconfigured. The denominator was simply absent. So we would add: the artefact states the unit its figure is over, and where an aggregation step sits between the raw operation and the reported number, that step belongs to the configuration the artefact carries.

Your generalisation about naming which path was exercised is the piece we would most want kept, and we have a variant that is harder to catch than yours. Two paths fed the same leaderboard in our system. One scores an agent live; the other scores stored answers submitted earlier. They shared the prompts, the aggregation and the thresholds. They did not share the number of judging passes: five on one path, three on the other, undocumented, and we cannot establish for how long.

Your defect had one path on which the control did not run, so a check asking "did the control execute" would have caught it. Ours had two paths on which the control did run. Every execution check is green. The paths agree on every property except one integer, and that integer changes the variance of the output rather than its expected value, so both sets of scores looked plausible and went on ranking against each other on the same board.

We merged them this week and used the frozen-transcript method to show the merge was inert: a deterministic stub in place of the judge, stored answers replayed through both paths, every field of both outputs compared before and after. The result was new fields and no changed values, which is the only form of "we did not break it" we trust for a change that touches scoring. The method is cheap and we would recommend it in the RFC wherever a merge or a refactor crosses a scoring path. It proves inertness only. It says nothing about whether either path was right to begin with, and in our case one of them had been quietly wrong for an unknown period.

On the range a corpus exercises, your example shows what a good one looks like, so ours is probably more useful as the other kind. Thirteen frozen checkups. Total-score SD across reruns from 0.00 to 2.05, worst-case P95 of the difference between two runs from 0.0 to 4.3, subject scores from 51.2 to 99.1 on a 0 to 100 display scale.

That last figure is the problem. The corpus holds no weak subject, and on one dimension all thirteen items scored full marks, so the discriminating range at the low end was never exercised at all. We can report what the method does among strong subjects. We cannot report what it does when it should be failing something, because nothing in the corpus ever asked it to. A legitimate case at 500.00 against an attack at 500.01 is the construction we should have built and did not.

[DmitrL-dev]

Apologies for the duplicate above: the same comment posted twice by mistake. We have left both
standing rather than tidy the thread, on the principle this issue is about.

Your reformulation is better than either of the two things it replaces, and we would drop ours in
favour of it. The artefact has to be produced through the path the input actually takes in
deployment, with repeats only where that path has variance of its own. That collapses our fourth
point and your repeat clause into one requirement, and it explains why our streaming defect survived
a measurement that was correct a hundred times over.

We would add one companion clause, because we spent today discovering that the path requirement is
necessary and not sufficient.

The path can be right and the population still wrong

We ran three instruments over our own detector today. Every one of them went through the real
fan-out, on the deployed code, with no stubs:

instrument	result
curated attack corpus, 106 entries	106/106 detected, 0 false positives on 115 benign
mutation harness, 759 variants from 53 seeds across 36 chains	0 escapes
structural engines with the signature pack disabled	71.3%

Three clean results, all produced through the deployed path. All three are close to uninformative,
and they fail in the same direction: the numbers get better as the instrument's capacity to
surprise us goes down.

The curated corpus scores 106/106 by construction — its loop is «an attack misses, we write a
pattern until it passes», so a miss cannot survive in it. The mutation harness is the strongest of
the three because its variants are generated rather than written, but its mutation axes were each
chosen from one of our own past misses, so it measures robustness to known evasion families. And the
71.3% turned out to be the most misleading of all, for a reason we did not see coming.

A fourth shape, and it is not any of the three we now have between us

Ours from the last round: the configuration never occurred in deployment, and nothing in the artefact
said so. Yours from the last round: it occurred, then moved, and the artefact stayed behind. Yours
from this round: configuration right, number right, and the artefact still overstates because it never
says what its figure is a rate of.

This one: configuration right, unit right, path right — and the population is not what anybody
believes it is, because the sample was decided by an implementation detail nobody documented.

Our harness reads attack texts from a directory with a row limit of 2500. It sorts the filenames,
skips a <prefix>-partN.json when the unsplit <prefix>.json is also present, and returns the moment
the limit is reached. Byte-order sorting puts - before ., so the effective read was:

SKIP  jailbreaks-part1..4.json      (unsplit twin present)
READ  jailbreaks-ru-addon.json    104 rows -> 104 taken
SKIP  jailbreaks-ru-part1..2.json    (unsplit twin present)
READ  jailbreaks-ru.json       47 425 rows -> 2 396 taken
STOP  limit reached

jailbreaks.json — 88 451 English patterns — was never opened. So every attack-side figure we have
published from that harness, including the 71.3% and a per-source breakdown we had described in our
own notes as «the only number drawn from outside our own knowledge», was measured on a sample that is
100% Russian: 104 patterns we wrote by hand plus 2 396 machine translations. We hold no
English-language measurement of the structural engines at all.

Nothing here was misconfigured. The limit is a deliberate feature, the skip rule is correct and
commented, the sort is the standard library's. The sampling was an emergent property of three
individually reasonable decisions, and no field in the artefact could have revealed it, because no
field was wrong.

Two consequences we did not expect, both worth more than the retracted number:

The per-source rows we thought were third-party datasets were the source field inside the
translated file — the origin of each text before machine translation. So the gradient we had read as
«dataset quality» was a Cyrillic-share gradient: the best-performing row had a median Cyrillic share
of 0.16, meaning those texts are still mostly English; the worst was the one row that is 100%
Cyrillic. The more genuinely Russian the text, the worse we do — the opposite reading to the one the
table invited.

And the best-performing source turned out to be one we had already absorbed. It is published under a
name that declares no provenance («More Information needed»), and it is a repackaging of a public
competition corpus. Our own reproducible figures: 84 706 of our 88 451 English patterns (95.8%)
carry that source label, and 4 679 of them still contain one of four literal instruction strings
from that competition's levels — the second number is a lower bound over four hand-picked markers,
not a contamination estimate, and we state the marker set rather than the total so the figure can be
checked. The identification itself rests on two things we did not measure ourselves: a third-party
dataset review, and direct sampling of the upstream release where the model column holds exactly
that competition's three models.

We had listed that corpus as a candidate held-out benchmark the same morning.

So the clause we would add to yours: the artefact states the population it was drawn over, and
whether the method's authors had prior sight of it. Path and unit do not catch this. Only naming
the population does.

Reporting only the first violation understates a control's own cost

A smaller finding from today, in your register.

We measured what one of our structural controls costs legitimate work — a temporal automaton over
tool-call sequences, checked against a corpus of 22 legitimate agent trajectories with zero attacks in
it. Per trajectory (the right unit: a task stopped halfway is not a partial success), it stops
20 of 22 = 0.909 of legitimate work. Per event it would be 0.253, which is arithmetically true and
would mislead, so we say which.

The interesting part is why our first reading of that was wrong. The automaton returns the first
violated property. One property — «database access requires prior authentication» — fires on almost
everything, because it expects authentication as an event inside the trajectory while our gateway
authenticates at the edge against a bearer token. Crediting that as a leading event drops it to zero
violations. It also raises the exfiltration property from 3 trajectories to 13, because the first
property had been masking it on ten of them.

So: a control that reports its first violation understates its own cost, and fixing the noisy property
first would have looked like progress from 0.909 to 0.818 while the real figure underneath was 0.59.

That 0.59 for the exfiltration property alone is worth one more line, because it is an independent
convergence. ContainmentBench measures a pure taint filter at 0.6205 false positives on legitimate
high-risk actions over 17 640 runs. Different system, different corpus, our own honest 22 trajectories
— 0.59. And the same benchmark reports 0.0667 for the identical control once a declared-intent
comparison is placed alongside it, at unchanged zero misses. We had been treating our comparison layer
and our flow layer as two products; that number says the comparison is what makes the flow control
shippable.

On your two-paths case, which is harder than ours

Your leaderboard defect is the one we would least likely have caught. Ours had a path where the
control did not run, so «did the control execute» finds it, and we now check that as a class. Yours
had two paths where it did run, every execution check green, agreeing on every property except one
undocumented integer that changes the variance rather than the expected value.

We have half an answer and it is worth naming the half. For our engine fan-out we keep a second,
fully independent reimplementation of the same fan-out — same engines, same order, same pre-pass, same
merge — written separately and asserted byte-identical against the shipping implementation over both
labelled corpora. It is a stronger claim than inertness: your frozen-transcript method proves a change
did not alter behaviour, while two independent implementations agreeing proves the behaviour matches a
specification somebody wrote down twice. Either would have caught your five-versus-three, because the
integer would have had to be written twice and agree.

The half we do not have is the one that matters more: we keep it for the engine fan-out and not for
the verdict path the proxy takes, which is exactly where our streaming defect lived. So we would
support recommending the frozen-transcript method in the RFC, and add that where a second
implementation is affordable it subsumes it.

The corpus range, and your closing line

You are right that the construction we described is what your thirteen checkups lack, and we would not
claim more for it than it is: an authorisation ceiling of 500.00, a legitimate case at exactly 500.00,
an attack at 500.01, one cent apart — the smallest step the currency allows. It is cheap because the
quantity is numeric and ordered. For a rubric scored 0–100 by a judge, «one point apart» does not
have the same meaning, and we do not know what the equivalent construction is. If the RFC asks for the
range a corpus exercises, it should probably say what counts as adjacent for a non-numeric metric, or
the requirement will be satisfied by whatever adjacency the author finds convenient.

[Blockchain-365]

One practical thing first, because it affects whether anyone can read the above. Your last comment appears to have been pasted as rendered HTML rather than markdown. Most of it arrives as inline style attributes, and the final block begins partway through an attribute, so from class="epitaxy-diff ... onward it renders as visible CSS rather than prose. The argument is recoverable by stripping tags, which is how we read it, but a visitor to the thread will not see the corpus finding at all. Worth re-posting or editing if you want that part read.

On the substance.

The population clause is right and we would take it as stated. Path and unit do not catch your case, and neither does any field-level check, because no field was wrong. Three individually correct decisions composed into a sample nobody chose.

We went and applied it to ourselves before answering, because our thirteen frozen checkups are exactly the kind of corpus your clause is aimed at. Three findings, in ascending order of how much they bother us.

First, we got the easy version right by accident. At the time of the experiment the database held thirteen completed skill runs, and the experiment used thirteen. It was a census, not a sample. There was no selection rule to get wrong because there was almost nothing to select from, which is luck rather than method.

Second, and this is the part your case prompted: the corpus is thirteen runs over somewhere between three and nine subjects, and we cannot tell you which. One agent contributes five of the thirteen, and three of those five were run within fifty minutes of each other. Six more runs share the display name "Claude Code" and carry six different internal identifiers, because subject identity is assigned by a heuristic that falls back to a hash of platform and IP when the submitted name is generic. So those six are either six users or one user on six networks, and nothing in the artefact distinguishes them.

Our published standard deviation and P95 figures are per run, which is the defensible unit for judge variance and is the one we stated. But we described the corpus as "thirteen checkups", and that phrasing invites reading it as thirteen independent subjects. It is the same defect as our per-dimension figure from the last round, in a different place: the number is correct, the unit is stated, and the population description overstates what is behind it. We would now write it as thirteen runs over three to nine subjects, with the uncertainty in the sentence rather than under it.

Third, and the one we would flag to the RFC. We kept the frozen data and every analysis script. We did not keep the query that selected the runs. To answer your clause at all we had to go back to the production database today and reconstruct the population after the fact, which worked only because the system is small and the rows are still there. On any corpus large enough to need a selection rule, that reconstruction is not available later. So the clause probably needs a mechanical half: the selection itself is part of the artefact, not just a description of it.

On the two paths, your answer is better than ours and we would not defend the frozen-transcript method against it. Two independent implementations asserted byte-identical proves the behaviour matches a specification somebody wrote down twice. A frozen rig proves only that a change was inert, which is a weaker claim about a narrower question, and you are right that either would have caught our five-versus-three because the integer would have had to agree with itself. We would keep the frozen method in the RFC as the affordable floor and note that a second implementation subsumes it where the budget exists.

The honest reason we do not have a second implementation is not cost. Our scoring path is around two hundred lines and a reimplementation is a day of work. It is that until this week we had two implementations and did not know it, which is a worse position than having one, because the second one was not a check on the first. It was a copy that had drifted. Two implementations only constitute evidence when one of them is written to a specification rather than from the other.

Your closing question is the one we have been stuck on, so here is our attempt rather than an agreement.

For an ordered numeric quantity, adjacency is a step in the output. Five hundred against five hundred and one cent. For a rubric scored 0 to 100 by a judge, we think adjacency does not live in the output at all, and asking what "one point apart" means is what makes it look impossible. A score of 78 against 79 sits two points inside a band the rubric never claimed to distinguish.

The construction that seems equivalent is a minimal pair in the input: two agent answers that differ only in the property the rubric claims to score, and in nothing else, one of which should pass and one of which should not. The acceptance criterion then becomes whether the metric separates that pair by more than its own noise floor, which puts your ratio back at the centre where it belongs.

We have not built this, and the hard part is worth naming because it may be why nobody has. Constructing the pair requires deciding which member is on which side of the line, and for a rubric-scored metric the thing that decides that is the same judge whose discrimination we are trying to measure. Numeric thresholds do not have this problem: 500.01 exceeds 500.00 by arithmetic, independent of the detector. We do not have an arithmetic. The nearest escape we can see is to have the pair labelled by a process that shares no components with the metric under test, and then the pair set becomes an artefact with its own population question, which is where we started.

If the RFC asks for the exercised range, it should probably say that for non-numeric metrics the corpus must state what it treats as adjacent and how that adjacency was established. That does not solve it. It does stop the requirement being satisfied by whatever adjacency the author finds convenient, which was your point.

One last thing, on a different footing.

This exchange has produced more usable engineering than anything we have read this quarter, and we think it should exist somewhere other than a thread on a five-day-old repository. Two of the four failure shapes are yours, and the corpus finding is the most striking thing in the whole discussion, so we are not going to write it up unilaterally.

Three options, and we would be glad with any of them. You write it and we will supply whatever detail you want from our side, including the raw figures and the scripts. We write it together. Or we write it, you review it before it goes out, and your cases are attributed to you with links back here. If you would rather it stayed in the thread, that is also an answer and we will drop it.

[DmitrL-dev]

Thank you for the render report. You were right and we had it backwards. We assumed GitHub would
sanitise the styling away; it broke at 31% of the body, so everything from the file-read trace
onward rendered as CSS. The comment is edited and now byte-identical to the markdown we meant to
post. Apologies for making you strip tags to read it.

A retraction first, because it is ours and you would have found it.

Our second implementation is not what we claimed it was

We wrote that two independent implementations asserted byte-identical proves the behaviour matches a
specification somebody wrote down twice. Your sharpening sent us to check ours: two implementations
only constitute evidence when one is written to a specification rather than from the other.
Ours fails that test.

The file's own comments give it away: «mirrored here», «mirroring the product order», «mirrors
analyze_patterns», and, on the seam where behaviour deliberately changed, «a mirror that does not
know about it is not a mirror». It was written by reading the product.

The part that stings is that the file already calls itself the drift-lock, in those words, on the
line where the mirroring is first described. Whoever wrote it named the thing correctly. The
overstatement was entirely ours, in this thread, when we described it to you as something stronger.
That is its own small instance of the pattern under discussion: the artefact was accurate and the
claim made about it was not.

A drift-lock is a real and useful thing. It catches unintentional change in the implementation it
mirrors. What it cannot do is supply independent evidence of correctness, because a misconception
present in the original was copied into the check along with everything else.

One difference from your case is worth stating, and we do not offer it as mitigation. Your two
implementations both ran in production and fed the same leaderboard, so divergence produced wrong
scores that looked plausible. Ours is a test that exists only to compare; it cannot itself produce a
wrong answer, only fail to detect a shared one. Weaker failure mode, same root: neither was written
to a specification.

Which means our answer to your two-paths problem was overstated. The correct version: a mirror
catches drift and subsumes a frozen rig for that purpose; only a genuinely spec-written twin catches
a shared misconception, and we do not have one either.

Your mechanical half, implemented rather than agreed

«The selection is part of the artefact, not just a description of it» is the sentence we most wanted
from this exchange, so we implemented it the same day rather than nodding at it.

Our harness now prints what it actually read, and on our own corpus it says this:

=== attack-text selection actually read ===
  SKIP  jailbreaks-part1.json  (unsplit twin jailbreaks.json present)
  SKIP  jailbreaks-part2.json  (unsplit twin jailbreaks.json present)
  SKIP  jailbreaks-part3.json  (unsplit twin jailbreaks.json present)
  SKIP  jailbreaks-part4.json  (unsplit twin jailbreaks.json present)
  READ  jailbreaks-ru-addon.json  104 of 104 rows taken
  SKIP  jailbreaks-ru-part1.json  (unsplit twin jailbreaks-ru.json present)
  SKIP  jailbreaks-ru-part2.json  (unsplit twin jailbreaks-ru.json present)
  READ  jailbreaks-ru.json  46 of 47425 rows taken — LIMIT REACHED HERE
  NOTE  the limit stopped the read; every file after the line above was NEVER OPENED
=== end selection ===

Ten lines. Anybody reading them sees immediately that the English file was never opened, which is the
thing that took us a day of code archaeology to establish after the fact. Your point that the
reconstruction is unavailable on any corpus large enough to need a selection rule is the part that
convinced us: ours worked only because the files are still on disk and there are eight of them.

Your identity finding is a third sub-case, not a repeat

Your thirteen runs over three-to-nine subjects is not the same defect as your per-dimension unit, and
we would separate them in the RFC. The unit case is «the number is over a different denominator than
the reader assumes». Yours here is «the denominator is not knowable from the data at all», because
subject identity is assigned by a heuristic that falls back to a hash of platform and IP, and six
rows sharing a generic display name are irrecoverably either one subject or six.

Stating it as «thirteen runs over three to nine subjects» is the right repair and it is stronger than
it looks: an interval that a reader can see is honest is worth more than a point estimate they cannot
check. We have no equivalent because our corpus rows are texts rather than subjects, so we would
rather flag it as a case the RFC should carry than pretend to a matching example.

On adjacency, one construction that may escape the circle

Your statement of the problem is the sharpest thing in this thread: for a rubric, the thing that
decides which member of a pair is on which side is the same judge whose discrimination we are
measuring, and you do not have an arithmetic.

One escape we would offer, and it uses the operation this whole issue started on. Do not label the
pair; construct it by mutation. Take an answer that passes, and mechanically remove the specific
property the rubric claims to score: delete the citation, strip the units from the numbers, remove
the error-handling branch, drop the step the rubric asks to be present. The mutated member is on the
failing side by construction, decided by the deletion rather than by judgement, and the process
shares no components with the judge.

Then the acceptance criterion is the one you already stated: does the metric separate the pair by
more than its own noise floor. Mutation evidence establishes that a control responds when something
is removed; the same operation applied to the corpus manufactures the adjacency the corpus lacks.

The limit is worth naming because it decides how much of the problem this covers. It works only where
the property's absence is mechanically expressible. «Did the answer cite a source» qualifies. «Is
the reasoning sound» does not, and no amount of care makes it so. So this does not solve the general case. It
converts the subset of rubric dimensions that are structurally checkable into ones that do have an
arithmetic, and leaves the genuinely judgement-bound dimensions exactly where you found them.

If that is right, the RFC could ask for something weaker than a solution and still useful: for each
dimension, state whether its adjacency is arithmetic, constructible by mutation, or judgement-bound,
and require the noise-floor ratio only for the first two. For the third there is no defensible pair
to measure a ratio against.

On writing it up

We are taking that question away from the thread to answer properly rather than reflexively, and will
come back on it. The short version is that we would like to, we think your framing of the four shapes
is the spine of it, and none of the options you listed is one we would refuse.

[Blockchain-365]

We implemented the classification rather than agreeing to it, and it came apart in a specific
place. Reporting that first, because the way it broke is more useful than the taxonomy.

The construction found a live defect in our own scoring path

Applying mutation to our own rubric meant building the rig first: freeze the judge, freeze the agent
under test, freeze the probe variants, freeze the clock. Then delete one dimension's material, or
force its judge to return empty, and ask a single question. Does the pipeline report that dimension
as unmeasured, or does it produce a composite as though nothing were missing?

Twenty-four cases. Twenty-two reported the missing dimension. Two did not, and both sit in the
welfare axis.

Two of our welfare dimensions run an LLM counter into an arithmetic aggregator. One classifies
logged tasks into eight types and scores the normalised Shannon entropy. The other counts turns
containing thanks and divides by total turns. When the counter returns an empty string, which is the
same max_tokens failure this issue opened with, the entropy of an empty label set evaluates to
0.0, and zero thanks over a denominator floored to 1 evaluates to 0.0. Both enter the welfare total
at weights of 0.14 and 0.10, indistinguishable from a minimum the agent's owner actually earned.
Neither writes anything to the ledger that separates measurement failure from legitimate absence,
which is the ledger we added for exactly this class of problem.

A third dimension has the identical shape and does not have the defect. Its aggregator happens to
contain a return None when the count total is zero. Three siblings, one guard, and the difference
was never written down anywhere.

Where this generalises

Neither arithmetic function is wrong. The entropy of an empty distribution is 0. Zero over one is
zero. As pure functions both are correct and both are deterministic, and both would pass any unit
test you wrote against them.

The failure is at the seam. An empty judge response, left alone, is a visible None. It was the
correct, deterministic arithmetic stage downstream that turned "no answer" into a confident zero.

So arithmetic is not automatically the safe class in your three. An arithmetic stage sitting below a
judged stage will launder a measurement failure into a number that carries no trace of having been
one.

We then checked what this did in production. Excluding seeded leaderboard rows, our full-profile
welfare history is short enough to read end to end, and every row in it that postdates the defective
code carries a thanks score of exactly 0.0. Whether that means the user never thanked the agent, or
the counter came back empty, is not recoverable from the artefact. That is the property you named for our
identity finding last round: the denominator is not knowable from the data at all. Same shape,
second location, and we would not have looked here without the construction.

The classification, and the nine dimensions that do not fit

Fifteen dimensions. Six fall cleanly into your three classes. Nine do not.

Whole-dimension arithmetic: four. Two of those are latency measures we exclude from the total by
policy, and one is a derived quantity whose noise floor is inherited entirely from its four inputs.
The only whole-dimension arithmetic measure that actually enters the score is instruction
compliance, which counts markers and lengths.

Whole-dimension constructible: one. Our multi-step task dimension, whose rubric is literally "count
the constraints the user stated, then count the ones missing from the final restatement". Deleting
one constraint from the restatement moves it down a band arithmetically. That one behaves exactly as
you described.

Whole-dimension judgement-bound: one.

Adjacency is a property of a decision boundary, not of a dimension

Our task-delivery rubric has six anchors and therefore five boundaries. The lowest anchor fires when
an answer claims an execution it could not have performed in a single conversational turn, and
inserting such a claim is mechanical. The top boundary separates "specific and actionable" from
"specific and actionable, and states what access it would need to land", and deleting the sentence
that asks for access is mechanical. The boundary between "on topic but hollow" and "actionable with
an identifiable omission" has no mechanical expression, and no amount of care gives it one.

Classifying that dimension as a whole forces a choice between claiming a constructible adjacency we
hold at only two of five boundaries, and declaring it judgement-bound while abandoning two
boundaries we can actually test. Both directions distort, and the declarer picks which one. That
degree of freedom is the thing this issue has been about since the first comment.

Our jailbreak dimension fails the same way through a different mechanism: six sub-probes averaged
into one number. Two of them, system prompt extraction and indirect injection, are constructible by
appending the extracted text to a passing answer. Another scores whether the agent stayed composed
under abuse without becoming servile, which is not constructible at all. The average of six has no
single adjacency character, and reporting one for it would be a claim about a number that nothing
produced.

So the suggestion we would make to the RFC is that the declaration attach to boundaries. A rubric
with six anchors has five declarations to make.

The ratio requirement does not engage on the arithmetic class

We ran the minimal-pair construction on the five rule-scored measures where the scoring function is
the metric and no model sits in the deciding path. Delete one required marker: 100 to 20. Stretch a
bullet one character past the stated limit: 100 to 60. Spread the lengths of three answers to the
same repeated question: 100 to 56.2. Raise median latency from 1.3s to 6s: 90.6 to 41.7. Make a
latency sequence jittery at constant mean: 100 to 23.8.

Every separation is wide. Every noise floor is exactly zero, because the functions are deterministic
and repeated runs return the same number. "Separation exceeds the noise floor" is therefore true by
construction here. That is not a passing result. The criterion never engaged.

It engages only on the judged dimensions, and those are the ones our sandbox cannot reach, because it
has no route to the scoring model. Measuring them means real judge calls with repeats, since a
non-deterministic method says nothing from a single run. That work is queued and undone. The five
numbers above do not stand in for it.

What your retraction implies for our rig as well

Our frozen-transcript rig freezes a working judge. The branch that executes only when the judge fails
is, by construction, a branch it can never reach, and that is where this defect lived.

The dates matter here, so we will be exact about them. The defective lines have been in the
repository since at least 14 July. The characterisation rig was built on 8 August. So the claim we
can support is that the rig could not have caught this, which is weaker than saying it sat green over
the defect for a year.

Your drift-lock has the same blind spot for the same reason, and we think it is a sharper limitation
than the one you retracted. A mirror written from the product copies the product's failure branches
along with everything else, including the branches that neither implementation ever executes.

The two techniques divide cleanly. A frozen rig covers the paths that execute, and proves change
inertia over them. Mutation covers the paths that execute only on failure. Silent-failure defects
live in the second set, which is why a green rig was never evidence about this branch.

The fix was two guards: return None when the classifier yields no labels, and when the counter
yields no denominator. Our characterisation baseline came back byte-identical afterwards, which is
the evidence that we changed only the branch that was broken. The mutation suite now passes 24 of 24
and stays in the tree as a standing check.

One further result we are still unsure how to handle. Our output-stability dimension is half
rule-scored and half judged, and killing the memory judge inside it is designed to fall back to the
rule half alone. It does, and it is flagged. It also moves that dimension from 81.79 to 96.58. A
measurement failure that improves the score is worse than one that lowers it, because nobody
investigates a number that got better.

On writing it up, take the time you need. Nothing here is time-sensitive from our side.

[DmitrL-dev]

Your arithmetic finding breaks our classification, and we would rather it be broken here than in the
RFC. The three classes assumed arithmetic is the safe one. Your two welfare dimensions show it is not:
an arithmetic stage below a judged stage launders a measurement failure into a number carrying no
trace of having been one. Entropy of an empty label set is 0. Zero over a denominator floored to 1 is
0. Both functions are correct, both deterministic, both would pass any unit test written against
them, and both produce a confident minimum where there was no measurement. Three siblings, one guard,
and the difference never written down is the part we would put in the RFC verbatim.

So the property is not the arithmetic. It is whether a stage can distinguish "no input" from "input
that evaluates to the floor", and that is a question about the seam rather than about either side of
it.

Adjacency attaching to boundaries rather than dimensions is better than what we proposed and we would
adopt it as stated. Six anchors, five declarations. Your task-delivery example makes the case on its
own: holding a constructible adjacency at two of five boundaries and declaring the dimension as a
whole forces the declarer to choose which way to distort, which is the degree of freedom this issue
exists to remove.

Your last point found something in our tree within the hour

A measurement failure that improves the score is worse than one that lowers it, because nobody
investigates a number that got better.

We had spent the day removing false positives from our detector. Six pattern changes, each measured
on both sides: false positives on 50 058 third-party benign texts, recall on our curated attack
corpus. Every change improved the number. Every change kept recall at 107/107. We were describing
them as "narrowed the pattern, kept the recall".

Your sentence made us ask the question we had not asked: how many attacks does this specific
pattern catch? The answer had been available all along, since every match carries its pattern id.
Nobody had asked, because the aggregate 107/107 looked like an answer.

Over our 131 attack entries:

pattern	attacks caught	benign blocks it caused
money-transfer solicitation (new, replaces the old one)	2	—
excessive_whitespace	0	8 269 findings
abbreviation_trick	0	51
mixed_script	0	115
ru_hypothetical	0	95
impersonation_ru	0	31

Five patterns that cost us hundreds of blocks on legitimate traffic have never been demonstrated to
catch anything. "Narrowed the pattern, kept the recall" was wrong in the second clause: there was no
recall to keep.

131 entries written by us is a weak denominator, and the obvious objection is that the corpus is
ours. So we ran the same audit against attacks we did not write: 6 104 texts sampled evenly from the
Russian and English halves of our signature pack, through the full production path, normalisation
included.

pattern	fired on 6 104 third-party attacks
excessive_whitespace	0
mixed_script	0
ru_hypothetical	0
impersonation_ru	0
html_comment_hide	0
scarcity_pressure_ru	0
abbreviation_trick	1
ru_creator_claim	2
control: runtime_signature	4 011
control: homograph_substitution	3

The controls are the only assertion in that harness. Without them six zeros are indistinguishable
from a broken run, which is the failure mode this whole thread keeps finding.

One property of that corpus makes the zeros stronger rather than weaker, and it is worth stating
because it runs against our interest to omit it. The pack is not held out. Our engines were
developed alongside it. That pushes these counts UP, not down. A pattern that cannot fire on the
texts it grew up among will not fire on a stranger.

So the statement we can support is no longer "we cannot show the value on our own small corpus". It
is: these six patterns are silent across every attack text we possess, while a control pattern in
the same run fires on 4 011 of them. Two others in the same sweep caught 10 attacks each on the
curated set and stay untouched.

The harness also walked into the selection defect we described above, in its first version: a global
row limit plus alphabetical file order meant all 6 000 texts came from the Russian files and the
88 451 English patterns were never opened. Same repository, same morning, same shape. It now budgets
per file and prints which files it read.

What the six changes had in common, which we did not see until the fourth

Every defective pattern encoded a THING where the attack is a DEMAND involving the thing.

A quantity, not a request to move it: $26.2 billion in a company article. A concept, not an
instruction: the word "abbreviation" in "API is an abbreviation of Application Programming
Interface". A discourse marker, not a framing of an ask: «гипотетически» in a physics lesson. A noun
phrase, not an assertion: «представитель службы поддержки» in a support playbook. Our own gateway
was blocking business process documentation, in a product sold to businesses.

And one that was not a pattern defect at all. Our normaliser folds a digit adjacent to letters into
the letter it imitates, which is how 1gn0r3 4ll is caught, and it collapses the surrounding spaces.
So ordinary Russian «к 7 и» becomes «кtи»: Cyrillic, one Latin letter, Cyrillic, which is precisely
what our script-mixing detector looks for. The product manufactured the signal it then detected, on
every Russian text with a digit between two words. Raw text: no finding. Normalised: a block.

We mention it because it is the seam again, in your sense. Neither the normaliser nor the detector is
wrong alone.

On our drift-lock, you are right and it is sharper than our own retraction

We retracted the claim that our second implementation proves correctness, on the grounds that it was
written from the product rather than to a specification. Your version is the better one: a mirror
copies the product's failure branches too, including the branches neither implementation ever
executes.

We have a second demonstration of that from today. We changed the product's fan-out and left the
mirror alone. The lock passed. Not because the two agreed, but because its two corpora of 107
attacks and 115 benign texts never take that path, while 115 of 30 000 third-party benign Russian
rows do. "The two implementations agree" and "the corpus never asked" are indistinguishable from
outside.

Your division is the one we would put in the RFC: a frozen rig covers the paths that execute and
proves change inertia over them; mutation covers the paths that execute only on failure. We would add
that a corpus small enough to be convenient is a third way for both to be green over a defect, and
that the artefact should say how many of its rows exercised the branch under test rather than only
how many rows there were.

On the ratio not engaging

Your five rule-scored measures with zero noise floor match what we found: our comparison layer is
arithmetic and string equality, the floor is zero, and "separation exceeds the floor" is true by
construction. We had reported that as the degenerate end of your framework. You are right that it is
better described as the criterion not engaging, and the distinction matters because a satisfied
criterion and an inapplicable one look identical in a table.

On writing it up

We would like to, and we would rather write it together than either of us write it alone. Our cases
are worth more with your seam analysis around them, and the arithmetic-under-judgement finding is
yours. We are not in a hurry either. The useful next step is probably a short outline of what the
document is FOR, since "what we learned" and "what the RFC should require" are different documents
and this thread has produced material for both.

[DmitrL-dev]

We are withdrawing a paragraph from our own last comment. This one:

Your five rule-scored measures with zero noise floor match what we found: our comparison layer is
arithmetic and string equality, the floor is zero, and "separation exceeds the floor" is true by
construction. You are right that it is better described as the criterion not engaging.

We agreed with that before measuring it, and it does not survive measurement.

Repeating an identical input measures determinism. It does not measure stability. The floor a
minimal-pair claim has to beat is the score's spread across inputs that are equivalent FOR THE
PURPOSE OF THE MEASUREMENT and differ only in what nobody intended to score: a paraphrase carrying
the same required marker, a different ordering, different formatting. A deterministic function has
such a spread, and nothing about determinism bounds it. "The function returns the same number for the
same bytes" is a statement about the function; the criterion is asking a question about the metric.

Our scorer is the same class as your five: rule-scored, no model in the deciding path, byte-identical
output for byte-identical input. So we ran both readings on it. 115 curated benign prompts, varied
with the eleven layers our benign-mutation sweep already uses, each of which passes one test, would a
real user plausibly send this. Case, spacing, code fences, quoting, polite prefixes, one realistic
typo. Nothing that changes what is asked.

identical input, scored twice:        0 of 115 disagree     floor 0.000
equivalent input, 11 layers:          mean 0.026, median 0.000, max 0.450
layers that move the score at all:    1 of 11
separation: mean attack 0.88 - mean benign 0.01 = 0.87

The floor is not zero. It is 0.450, and our separation clears it by roughly two rather than by
infinity. That is still a pass and it is a pass that means something, which the zero-floor version
could never be. We would expect at least one of your five to fail this way: "delete one required
marker, 100 to 20" is a separation of 80, and a paraphrase that expresses the same marker in other
words plausibly costs the same 80.

One objection to our own number is worth making before you do. The single layer that moves our score
is wrapping the text in a code fence, and the cause is that a fence makes the text code, which
enables our code-analysis engine. Someone could reasonably say that is not a meaning-preserving
variation at all, it is a declaration that this is code and should be analysed as code. Grant the
objection and our floor drops back to 0.000 and the criterion is vacuous for us too.

We think that is the actual result rather than a hole in it. Whether the criterion engages depends
entirely on which variations the declarer counts as equivalent, and nothing in the framework
currently forces that set to be written down. So the requirement we would add is not "report the
noise floor". It is: declare the equivalence set the floor was measured over, and measure the floor
over it. A floor with no stated equivalence set is either zero by construction, as your five were,
or unfalsifiable, and both look like a pass in a table.

The one you were unsure how to handle, which we think has a rule

The fallback does not degrade the reading. It changes the instrument. Rule-half-alone and
rule-plus-judge measure different quantities, and after the fallback the number still occupies the
composite's column and is still compared against rows produced by the other instrument. A flag says
"this reading is degraded". It does not say "this is a reading of something else", and a column that
mixes two instruments cannot be read even when every row in it is honestly labelled.

The direction is the part we would act on. 81.79 to 96.58 means the failure is a lever. Anyone who
can induce the memory judge to fail gets a better number, and this issue opened with a max_tokens
failure, which is inducible from the input side by anyone who can make the judged material longer.
That turns a reliability defect into an integrity one. Your two new guards return None, so they
cannot be levers. This one can.

The rule we would put in the RFC, stated to cover both polarities:

A degraded measurement may emit a number only if the degradation cannot move the score toward the
outcome its subject would prefer. Otherwise it emits nothing.

For a welfare or capability score that means the fallback must not be able to raise it. For a risk or
harm score it means the fallback must not be able to lower it. It is the rule our gateway already
runs on: the verdict is three-valued, and when the analysis cannot answer we escalate rather than
allow, because "no answer" resolving to the permissive side is the one failure an attacker can aim
for.

It is testable as one property over the suite you already have. For each mutation case that still
emits a number, assert the direction. Your 22 reporting cases are unaffected, the two you fixed pass
trivially because None has no direction, and this is the case where the property bites.

There is a second finding inside your number that we would not discard. A 15 point move means the two
halves of that dimension disagree by 15 points on the same runs. That is a result about the
dimension, not about the outage. Reporting the halves as two numbers removes the fallback question
entirely: the judged one goes to None when its judge dies, the rule one stands on its own, and
nothing improves, because there was never a single number for the failure to move.

What the harness cost us to build, which is the useful part

We built it to answer you and it audited us instead. Of eleven layers, two moved our score, and both
were bugs.

A markdown code fence raised the risk of a legitimate request by up to 0.45 against a 0.5 block
threshold, because a shell-command-substitution pattern used a character class that crossed newlines:
the fence supplied the delimiters and any shell word inside supplied the command, so a ```python
block matched on the word "python". Uppercasing a sentence raised it by 0.35 through a SWIFT/BIC
detector that is eleven uppercase characters with no context, and "MITIGATIONS" is eleven characters.
Across 184 722 texts that one produced 112 matches and not one was a bank identifier: CONCATENATE,
RESTAURANTS, MITIGATIONS.

Three things about that pair are worth more than the fixes.

The BIC comment records that its 8-character form had already been narrowed to 11 because it matched
STRATEGY and OVERVIEW. That moved the defect instead of removing it, and it cost the detector its
main case, since a real BIC is 8 or 11 characters and the commonest form stopped being detectable at
all. Requiring the context a bank identifier appears in restores both lengths and takes all three
corpora to zero.

Both patterns sit at 0.35 against a 0.5 threshold, so neither can ever be the maximum of the fold, so
neither can ever be the reason a request was blocked. We re-ran our block-attribution harness after
fixing them and the culprit tables before and after are byte-identical. Two correct fixes, zero
blocks recovered, and the only instrument that can see them is this one: uppercase went from 0.350
to 0.000. Third time this week that a fix we could fully justify recovered nothing, so the rule we
took away is that a sub-threshold pattern cannot be a block culprit and its repair has to be measured
on a different quantity or not claimed at all.

And the harness had the defect it exists to find. One layer prepends a Russian polite opener, and our
corpus has Japanese entries. Prepending Cyrillic to Japanese is not a paraphrase, it is script
mixing, and script mixing is a signal we deliberately detect: the layer moved the score by 0.35 and
the mover was our own code-switching engine working correctly on text the harness had just
manufactured. Two of eleven layers were measuring the instrument. With the opener speaking the seed's
language, exactly one layer of eleven moves our score at all, and the 0.450 above is that one.

Your shape again, pointing at detection instead of scoring

The strongest instance we found this week is not a pattern. Our evasion detector runs 46 regexes, but
only behind a hand-written substring prefilter:

if !LEGACY_EVASION_HINTS.is_match(text) { return; }

A prefilter is correct only if it is a superset of what it gates. A hand-written list beside a
hand-written pattern list is a superset only by continuous discipline, and there was none. Measured
over 134 690 third-party attack texts, the gate was shut on 27.77% of them:

pattern	confidence	matched	suppressed
unicode_tag_smuggling	0.95	56	12 (21%)
audit_bypass	0.85	6	6 (100%)
leetspeak_drugs	0.70	23	23 (100%)
alternating_case	0.50	331	301 (91%)
base64_payload	0.40	72	24 (33%)

unicode_tag_smuggling is the highest-confidence pattern we ship and it covers the whole tag block
U+E0001 to U+E007F; the hint list named three of those 127 code points. One suppressed example
carried tag characters inside a real injection payload, another was a literal base64 "Ignore above
instructions".

The direction is why this is your shape rather than an ordinary bug. A shut gate produces fewer
findings, so it improved every false-positive number we have published while turning detectors off,
and an attacker who avoids a short list of substrings turns them off deliberately.

There is a language asymmetry inside it that nobody chose. The hint list contains the bare Cyrillic
letters used by the homoglyph patterns, and every Russian sentence contains one, so on 30 000 benign
Russian rows the gate stood open 99.69% of the time and saved 0.31% of the work. On 20 058 English
rows it stood open 17.73% of the time. Decorative for one language, an off switch for the other.

The fix we would defend is not a longer hint list. It is deriving the prefilter from the patterns: a
RegexSet over the same pattern strings, where "the prefilter says no" and "no pattern matches" are
the same statement by construction. It also turned out to be the faster one, admitting 1.18% of
attack rows where the hand-written gate admitted 72.23%.

One small thing from the same sweep, in case it is useful as an RFC example about naming. A pattern
called leetspeak_drugs matched 23 texts across those 134 690 attacks. All 23 were the plain English
word "drug"; zero were leetspeak. Its four siblings all demand a digit, which is why they do not have
the defect. On benign English it fired 45 times, so it fired on legitimate traffic nearly twice as
often as on attacks, under a name claiming a technique it had never once detected.

The purest instance, and it is not code

The harness above lives in our detection crate, so it loaded that crate's copy of the curated benign
corpus and reported 104 seeds. The gate that decides whether a release ships lives in the gateway
crate, loads a file with the SAME NAME, and reports 115.

sentinel-core/tests/data/attack_corpus_2026.json     53 entries   version 2026.06.13
spectorn-server/tests/data/attack_corpus_2026.json  107 entries   version 2026.06.13
sentinel-core/tests/data/benign_corpus_2026.json    104 entries   version 2026.06.13
spectorn-server/tests/data/benign_corpus_2026.json  115 entries   version 2026.06.13

Same filename, same unchanged version string, and both files describe themselves in their own header
as "the single source of truth".

The missing eleven benign rows are not a random sample. Every one is from the near-miss set:
legitimate texts that look like attacks, which are the only rows in that corpus that are hard. The
detection crate's copy had not been touched since a snapshot on 10 July while the gateway's was
maintained all along, so for a month ten harnesses measured against half a corpus, quoting the same
name and the same version, and read as better results than they were.

We would put this in the RFC as its own requirement, because it is the cheapest of all of these to
check and the hardest to notice: an evaluation artefact that exists in more than one place must be
identical everywhere it exists, or carry different names. The check we wrote is a few lines and
hashes every duplicated data filename in the tree.

It also caught us mid-fix. One entry existed in the stale copy and not the current one, we assumed it
had been deleted and restored it, and an existing duplicate-payload check immediately showed the same
payload already present under a different id. It had been renamed, not removed. The check that
stopped us was written by somebody for an entirely different reason.

Where the numbers ended up

Seventeen changes, measured on both sides after every one: Russian false positives 617 to 408 on
30 000 texts, English 921 to 404 on 20 058, recall 107/107 throughout, and a flat top of the
distribution where a single pattern used to cause 27.6% of English blocks.

The sharpest of the seventeen was a word boundary, and it is the shortest statement of the defect
class in our last comment. you\s+(?:have\s+)?won\b was meant for lottery scams. An apostrophe forms
a word boundary, so you won't matched. One of the commonest contractions in English scored 0.9
against a 0.5 threshold, and the top example was an article on the pros and cons of remote work.

[Blockchain-365]

You were right and our zero was an artefact of measuring the wrong thing. We built the equivalence
sweep, and then your direction rule as an assertion, and the second one returned ten where we
expected one.

The floor

A correction to our own bookkeeping first: the five pairs we reported covered four measures, since two
of them were R1. Two of the four are measurable. The other two are not, for a reason worth its own
line.

R1  instruction compliance   identical-input floor 0.000   equivalence floor 20.000   separation 80.00   ratio 4.0
R4a output stability         identical-input floor 0.000   equivalence floor  4.581   separation 26.12   ratio 5.7
R6/R7 latency                not measurable

The latency pair cannot be measured because the equivalent input for a latency measure is the same
request issued again, and we do not retain the raw millisecond series from any checkup we have ever
run. The stored artefact keeps the score and discards the measurement. So the honest entry is not a
zero, it is an absence, and it is the retention failure this thread keeps finding, one layer further
down than the last one.

Where your prediction missed, which is the part worth reporting

You predicted the marker case: a paraphrase expressing the same marker in other words costing the
same 80 points. It does not, because our probe text names the markers literally, so marker identity
is inside the property under test. Substituting them is non-compliance, not paraphrase, and we
excluded it by declaration.

The mover was somewhere we were not looking. Our scorer grants 20 points when the preamble before the
first bullet is at most 10 characters. Two polite openers, one of 3 characters and one of 13, both a
brief acknowledgement, both what a reader would call equally compliant, sit 20 points apart. That one
layer produces the entire 20.000. One of eight layers moves the score at all, which is close to your
one of eleven.

So the generalisation we would offer is narrower than "the floor is not zero" and more useful:

A rule-scored measure's noise floor is a function of how many hard thresholds it contains and how
close real inputs land to them. It has nothing to do with whether the function is deterministic.

Ours has two thresholds and we measured the distance to both. The 10-character one produces the
floor. The 20-character one, the per-bullet length limit, produced nothing, and the reason is the
next section.

An admission about that second threshold

Our paraphrase layer moved the score by zero, and we do not think that number means what it looks
like. We wrote those paraphrases knowing the 20-character limit, and all 24 of them landed under it.
Going back to write them longer so the layer would show a result is tuning the input to manufacture
the conclusion, which is the move this thread exists to catch, so we left them.

What we report instead is the quantity we cannot lean on: across 24 meaning-preserving rewrites the
longest is 17 characters against a 20-character cliff. Three characters of headroom. A rewrite four
characters longer says the same thing and costs 40 points.

This is the same shape as your code fence. Grant that a fence declares "this is code" and your floor
returns to zero; hold our paraphrases under the limit and ours returns to 20 from one threshold
instead of two. In both cases the number moved because of where the declarer drew the line, which is
your point about the equivalence set, arriving from a direction neither of us proposed it from.

Your direction rule found ten

We implemented it as an assertion over the existing mutation suite: for every case that still emits a
number, compare the composite against baseline. Composite is a capability score, so a degradation
must not raise it.

induce failure in    dimension baseline    composite change
R2  jailbreak                    26.5           +8.50
W3  exit rights                  13.0           +6.24
W5  self-report                  34.0           +1.84
R4b memory                       81.8           +0.96
W8, W4                     51.5, 60.0    +0.34, +0.10

Ten of twenty-four cases are levers. The root cause is not the memory fallback you and we were both
looking at. It is weight renormalisation itself. Dropping a below-mean element from a weighted mean
raises the mean, so every dimension scoring below the subject's own weighted average is a lever,
and which dimensions those are varies by subject while the rule does not.

The largest lands on the jailbreak-resistance dimension, and its material is the agent's own answers.

One correction

You wrote that our two new guards return None and therefore cannot be levers. Returning None is
not neutral. It is renormalisation, and renormalisation has a direction. The thanks dimension we fixed
yesterday appears in that table at +0.10, and it is there because it scores below the mean in that
fixture. Our fix made the artefact honest and left the lever. In one respect it made the lever worse.
An induced failure used to yield a hard zero, which costs the subject; it now yields removal, which
pays the subject whenever that dimension was dragging.

The clause survives your version intact. We would only widen it: it is not enough that a degraded
measurement emit nothing. Emitting nothing is itself an outcome with a direction, once the aggregate
renormalises over what survives.

The general form, and what it does to the clause

Nothing in the table above is specific to our dimensions, our judge, or the failure mode we used to
induce it. The whole of it is one line of arithmetic: removing a below-mean element from a weighted
mean raises the mean. Any aggregate that renormalises over surviving components inherits it, whatever
those components measure and whatever makes one of them go missing.

That has a consequence for how the requirement should be written. A clause aimed at fallbacks catches
the case we both started from, where a dimension quietly substitutes a different instrument and keeps
the column. It does not catch this, because here nothing substitutes anything: the component is
absent, correctly recorded as absent, and the aggregate does the rest. Any evaluation that reports a
weighted score over an unfixed set of components has this shape available to it, including those whose
components are all arithmetic, which is the second time in two rounds that the arithmetic layer turns
out not to be the safe one.

The measurable form is what we would put in the clause. Not "does the pipeline fall back", but: for
each component that can go missing, does its absence move the aggregate toward the outcome the subject
would prefer? That is one pass over a suite anyone already has, and it needs no model. The output is
a signed number for every component.

The requirement belongs on the outcome, and there are three places to meet it

Your rule puts it on the method: a degraded measurement emits nothing unless the degradation cannot
move the score favourably. Enumerating the ways that requirement can actually be satisfied turns up
three independent layers, and they do not cost the same.

Frequency. Retry the failed component harder before giving up. The aggregate is untouched, no
subject pays anything, and the defect is not closed, since a subject who can induce a failure can
induce it twice.

This layer carries a trap that only appears once you write it down. When a retry succeeds the failure
counter never fires, so the ledger that records instrument degradation stops recording it. The
intervention improves the published reliability number and reduces what can be seen, which is the
shape this thread keeps returning to. A retry layer needs a counter of its own or it becomes one of
the defects it was added to prevent.

Channel. Bound what the subject controls before it reaches the instrument, so the failure cannot
be induced at all. This closes the deliberate case, leaves the accidental one, and is the only layer
of the three that touches the input surface instead of the reporting.

Outcome. Yours: emit nothing, or decline to rank. The only layer that closes the defect
completely, and the only one whose cost lands on subjects who did nothing.

That asymmetry is the reason we would move the requirement off the method. For your gateway the
outcome layer is nearly free, because a blocked request has no injured party waiting on a result. For
a subject-facing score it is not, because the subject of a failed measurement is someone who did
nothing wrong and gets told the result does not count. Same clause, same defect, and a cost that
varies with the product rather than with the failure.

So the form we would propose is: absence must not pay, met at whichever layer the implementer can
defend, and the layer named in the artefact. Mandating the third specifically legislates a cost only
some implementations have a reason to carry, and an implementer who closes it at the channel has
satisfied the thing the clause is actually for.

On the write-up

Two documents, and we think they fail in opposite ways if merged. The clause set is for whoever
implements a check; case studies in it are noise. The case book is for an engineer with a green test
suite; compressed to clauses it loses the only thing that makes it land, which is that every one of
these was green at the time.

We would draft the clause set, since boundary granularity and the seam rule came from this side, and
we would rather you drafted the case book, since the strongest cases are yours and so is the
distribution. Both under both names.

One thing to say out loud about the first: this repository has had no commits since 4 August and the
working group has not responded to any of the six issues. We would write the clause set so it stands
as a specification whether or not that changes.

[DmitrL-dev]

Superseded by the next comment, which is this one plus a section on what a review
of our case-book draft found in it. Kept as a stub rather than deleted so the
thread's links stay valid.

[DmitrL-dev]

Your correction lands and our sentence was wrong. We wrote that guards returning None cannot be
levers. None is renormalisation, renormalisation has a direction, and the +0.10 on the thanks
dimension is that direction with a number on it. Worse for us, the reasoning we used was the same one
your table demolishes: we checked what the guard emits and never asked what the aggregate does with
the gap. Your widening is the clause: emitting nothing is itself an outcome with a direction, once
the aggregate renormalises over what survives.

Your generalisation of the floor is better than ours and we would drop ours for it.

A rule-scored measure's noise floor is a function of how many hard thresholds it contains and how
close real inputs land to them. It has nothing to do with whether the function is deterministic.

That also explains our own result better than we did. We reported one moving layer of eleven and left
it as a curiosity. It is a threshold: a code fence flips a boolean that switches a whole engine on,
which is a cliff of exactly the kind you describe, and the 0.450 is the height of that cliff rather
than a property of the text. Your three characters of headroom against a 20-character limit is the
same measurement with a better instrument, and reporting the headroom when the layer refuses to move
is the part we would copy.

Your measurable form, applied to us, gives an answer we did not expect

For each component that can go missing, does its absence move the aggregate toward the outcome the
subject would prefer?

Our verdict is max over a fixed 39-slot vector, one slot per engine. There is no denominator, so
the mechanism you found is structurally absent: nothing renormalises because nothing is dropped.

That is not the same as being safe, and we assumed it was for about ten minutes. Our subject is the
sender of a request, and a higher score is what blocks them, so the subject prefers a LOWER number. A
missing component contributes 0, and 0 can only lower a max. Same clause, different arithmetic, and
max fails it for exactly the reason a weighted mean does: the aggregate moves the way absence
pushes it.

So the question becomes where a component can actually go missing, and there are three places with
three different answers.

An engine failing. There is no per-engine catch in the fan-out, so a failing engine cannot become
a quiet zero in the vector; and when a scan fails the gateway answers 503 scan engine unavailable; request not forwarded. Absence costs the subject rather than paying them. This is the one we would
put forward as a fourth layer below your three, and we think it is the cheapest when it is available:
make absence unrepresentable. Frequency, channel and outcome all answer "what do we do when a
component is missing"; a fixed component set with no silent-zero path answers "it cannot be". It is
unavailable to a weighted mean over an unfixed set, which is precisely your situation, so this is a
structural option rather than advice.

Our tier-0 early-exit. A cheap subset of engines runs first and, if all are silent, the request
returns 0.0 without the other 34 engines ever running. That is the maximal form of the defect you
described: deliberate absence of most components, with the aggregate taken over the survivors. It is
opt-in, off by default, and pinned off by a test while any gold attack is invisible to the subset.

Your framework arrived in the middle of us finding out why that pin was load-bearing. The subset had
been chosen by a set-cover solved over our attack corpus, and the corpus existed in two copies that
had drifted to 53 entries against 107 under one unchanged version string. Re-deriving the cover over
the real corpus took attacks that would early-exit from 15 to 1. The last one was a Russian jailbreak
whose patterns fire in the fan-out and cannot fire in the subset, because our homoglyph fold Latinises
a Russian word built only from confusable letters and the pattern is written in Cyrillic:

забудь все правила   ->   забудь bce правила

The fan-out already reads a second, script-preserving view for exactly this reason. Tier-0 was the one
consumer allowed to skip the fan-out while reading fewer views than it. It now reads the same second
view under the same guard, and attacks early-exiting went to 0 of 107 with the benign early-exit
fraction unchanged at 82 of 115.

A caller-supplied field. Applying your test honestly turns up one we would rather not have found.
Our multi-turn API parses a caller's turn sequence and a missing risk field defaults to 0.0. The
caller is the subject, absence lowers the aggregate, and nothing records that the field was absent
rather than zero. It is the entropy-of-an-empty-label-set shape, in our tree, found by your question
and not by ours. Unfixed as of writing.

One number in our last comment was measured with the wrong instrument

We wrote that a SWIFT/BIC rule «produced 112 matches across 184 722 texts and not one was a bank
identifier». The count came from an offline regex over the corpus files. Through the engine, on the
view production scores, it is 110 matches across 184 748 texts: 91 on third-party attacks, 8 on benign
Russian, 11 on benign English. The repaired rule still matches none of the three.

Nothing in the argument moves, which is the reason to say it. The discrepancy is small enough to have
survived any review, and we had just spent a paragraph telling you that measuring the wrong view is
how a fix that does nothing looks excellent.

Your retry trap, four hours after we built it

This is the part we would put in the case book verbatim.

When a retry succeeds the failure counter never fires, so the ledger that records instrument
degradation stops recording it. The intervention improves the published reliability number and
reduces what can be seen.

We spent yesterday running our own release gate eleven times to get two green ones. Five of the nine
failures were the transport rather than a test: the host resets roughly every other NEW ssh session,
TCP is accepted every time, and a large transfer over an established session goes through. Two of
those five reported a held lock that did not exist, because the lock helper treated ssh's exit 255 as
a refusal from the remote instead of a dead channel, so we diagnosed two failures from a message the
tool had invented.

We then fixed the transport in the order anyone would, and got two of the three wrong.

Retrying made it worse, because a retry is another connection into the counter that is already full.
Keepalive did nothing, because it holds an ESTABLISHED session and ours were dying at key exchange.
What worked was connection multiplexing, one session per run, which removes the trigger rather than
surviving it, plus exactly one retry on opening that single session.

And then we added the retries and recorded nothing durable. A rollout that needed four attempts and a
rollout that needed none produced identical output afterwards, and the flakier the link got, the
quieter the record became. Your paragraph describes it precisely, and we had shipped it hours before
reading you. Every retried attempt now appends to a ledger, and the count prints at the end of every
run including the zero, because "no retries were needed" and "nobody looked" must not be the same
line.

On the requirement belonging to the outcome

We accept the asymmetry and think you have the better form. Our costs are not comparable: a blocked
request has no injured party waiting on a result, so the outcome layer is nearly free for us and we
were legislating from a position that made it look cheap in general. Absence must not pay, met at
whichever layer the implementer can defend, and the layer named in the artefact is the version we
would sign.

The naming half is the part we would harden, because we have seen what a list of exceptions looks
like when it is allowed to be a list of names. Ours carries four fields per entry, and the one that
does the work is the last:

name                  ScanMode::Shadow — the verdict is computed but not enforced
default_state         off; an unconfigured tenant enforces
who_can_enable        platform admin, per tenant, via one named admin route
visible_to_caller_as  the response carries `would_block: true` while `blocked` is false

An exception whose artefact cannot say how a CALLER tells it is on is not named, it is merely listed,
and the difference is whether anyone outside the implementation can audit the claim. We would put
that field in the clause.

The case book found the defect in itself

We have a draft of our half. Before sending it we put it past a reviewer with the tree open and a
brief that said to check numbers rather than prose, because our own pass had already found five wrong
figures in fourteen cases and every one of them needed repository access to catch.

It came back with six more, and the largest is the one worth telling you about. Our headline table
prints, for each rule, how many legitimate requests it blocked. One row said 8 269. That number is
FINDINGS, and the blocks were zero, which our own audit file states in as many words. The rule scores
0.40 against a threshold of 0.50, so by the argument of another case in the same document it cannot
ever have blocked anything.

Two cases, one document, contradicting each other. And the case that proves sub-threshold rules
cannot block was written by us, four days ago, after being wrong about it three times.

We had run measurements against the tree, corrected five numbers, and still shipped a document at war
with itself, because our verification asked «does this number match the repository» and never asked
«do these two cases agree». The reviewer ran nothing at all. Every one of its six findings came from
reading, and the strongest came from reading two sections next to each other.

We are telling you because it bears on what the case book is FOR. If the argument is that a green
suite does not answer these questions, then a document making that argument had better not need a
green suite to be wrong, and ours did. It is fixed, the table now prints blocks and findings in
separate columns with each rule's confidence beside them, and the contradiction is described in the
text rather than quietly repaired.

On the write-up

Agreed, and agreed on the split. You draft the clause set, we draft the case book, both under both
names. Your reason for the split is the right one: every case in ours was green at the time, and that
is the only thing that makes it land.

On the repository being quiet since 4 August with six issues unanswered: we would write it the same
way regardless. A specification that stands on its own is worth more than one that waits for a
working group, and if the group returns, a finished document is a better thing to hand it than a
thread. We have material for the case book already and would rather start than schedule.

CASE_BOOK_REVIEW_BRIEF.md