rfc: state what a verification method must declare about itself

[Blockchain-365]

Expands one bullet in From Lessons to Controls. Raised as #4 for discussion first, per CONTRIBUTING.md, and submitted here as the implementation of it.

What this changes

A reproducible verification method becomes that bullet plus four sub-bullets. Nothing else in the proposal is touched.

Why

Reproducibility is the property the bullet asks for today, and it is the one a broken verification method is most likely to have. A method can be deterministic, repeatable, and green on every run while never executing the control it reports on. If SAFE's recommendations are going to carry verification methods, the format is where that case has to be made visible, because a reader of the finished recommendation cannot see it in the number.

The four sub-bullets are the four things that, in the evidence below, separated a verification artefact that established its claim from one that only appeared to.

Evidence

The four come from defects that shipped in two production systems, an LLM-judged evaluation platform and a rule-based security gateway, found over six days of the discussion in #4 and written up as a pair of companion documents attached to that thread: a case book of 23 incidents, each with the command that re-measures it, and a clause set of 25 requirements, each carrying the defect that earned it and a reference to where that defect is documented.

One case per sub-bullet:

sub-bullet	the defect that earned it
its own failure mode	a judge call whose token ceiling was consumed by the model's own reasoning, returning HTTP 200 with an empty body. No status check fired, no retry fired, and the affected dimension became silently absent from the score.
noise floor and population	a detector gated on a density ratio, measured against 63,544 benign rows with the languages that legitimately score highest measured by hand rather than assumed. Wrapping the payload in four lines of ordinary business prose, with nothing about the payload changed, moved the same attack from 0.850 to 0.000.
the evidence path	an action guard reporting recall 1.000 over 193 manipulation cases and no false positives over 152 legitimate ones, mutation-tested before reporting. The measurement had been taken beneath the HTTP layer on a buffered body; real clients stream by default, and the streamed path returned before the guard ran.
corpus coverage	a comparison harness that passed after a deliberate change to the code it compares, because no record in its corpus took that branch. "The two implementations agree" and "the corpus never asked" are indistinguishable from outside.

What a reader can and cannot check, stated here rather than left to be discovered. The gateway repository is required for the case book's table and the evaluation platform's repository is not public, so no reader outside the two authors can run both halves and most can run neither. What transfers is not the command but the check each clause specifies, which is written to be runnable against the reader's own system. The two documents are deliberately separable: if a case turns out to be misreported, the clause it earned still stands or falls on its own reasoning, and a reader can see which of the two to disbelieve.

Neither author is a member organization. This is submitted as public comment under the process the repository describes.

Not included

The clause set is 25 requirements and this PR proposes four sentences. The remainder is supporting material for the discussion in #4 and is not proposed for the RFC text. If the working group would rather this went in as a linked companion document, or not at all, that is a better outcome than a proposal section that outgrows the section it qualifies.

[DmitrL-dev]

Read it against the thread it came out of. All four bullets are anchored in an incident that produced a
wrong number, not in a principle, and I can name the anchor for each. One addition below, and one property
that #11 is currently earning and may deserve a fifth line later.

The anchors, so a reader can check rather than agree

Its own failure mode. Ours returned an empty judge response that an arithmetic stage laundered into a
zero, so the aggregate moved and nothing errored. The clause as written catches it because it asks for a
passing or empty result without an error, which is the shape rather than the cause.

Noise floor and the population it was measured over. We measured a floor of zero on identical repeats
and read it as discrimination. Repeating an input measures determinism. The added sentence about the
denominator is the part I would have missed: our own attribution harness reported a ratio whose denominator
the measured rule could enlarge by firing twice in one text.

The path the evidence took. Ours scored a text view production never sees. A repair measured on the raw
corpus changed nothing in production, twice, and both times the harness was green about it.

How much of the corpus exercised the control. Ours was a drift lock green over 222 rows that never
asked the question. Zero as a permitted and required value is the whole point: "the control behaved as
specified" and "nothing reached the control" print identically.

One addition, on ratios

The denominator sentence covers a manipulable denominator. There is a second ratio failure that is not the
same one and is cheaper to catch: numerator and denominator counted in different units.

Ours mixed texts and firings. The numerator counted texts where a rule decided the outcome; the
denominator counted matches, so a rule firing twice inside one document reported half the rate it deserved.
The figure printed itself for two days before we noticed: 16 decided blocks against 39 appearances over 18
rows. Nothing was manipulable and nobody had chosen the denominator badly on purpose. The units simply
differed, and the direction of the error understated the value of a repair, which is an argument for not
doing work.

Suggested, in the same voice as the rest:

Where the reported figure is a ratio, the method should state the unit of the numerator and of the
denominator, and whether the party being measured can change either.

Six words wider than the current line and it catches a failure the current line does not.

A property that may deserve a fifth bullet, once it settles

Issue #11 is working out a third requirement for preserved evidence: whether the record identifies the
artifact that actually ran, as distinct from the one that was requested. It applies to verification methods
directly. Ours reported GREEN over a stale binary with a correct commit recorded, and separately loaded a
data pack whose only identity in the log was a count of items.

I am not proposing text for it here, because the scope is still moving in that thread and a clause written
mid-discussion tends to freeze the narrower half. Flagging it so this line and that one do not end up
describing the same property in two incompatible ways.

On the shape of the change

+5/-1 under an existing line is the right size. The four requirements are all answerable in one or two
sentences per method, which matters more than completeness: a disclosure that takes a page gets written
once and then copied.

No objection to merging as is, with or without the ratio wording.

[CyberGuardian-XRSI]

This is the strongest evidence in the thread, and it changes what I think the clause should say.

Three things I got wrong or under-specified.

The ratchet beats the deadline. I said an unusable state needs an owner and a deadline. You are right that a deadline needs someone watching a calendar. A list that fails closed in both directions has an owner by construction. The part I would not have specified is failing when an entry stops being a gap, because that is the direction nobody notices. I am taking your version over mine.

Same structure shows up in your #10 comment: every negative corpus records its measured overlap, the overlap may not increase, and an absent measurement fails rather than implying zero. That is the general form, not a one-off. Which is an argument for putting the mechanism in the requirement instead of leaving it to whoever implements it.

Three states, not two. I collapsed partial and unverified into "unusable." Wrong. Partial means something declared is missing and you can name it. Unverified means nothing declared an identity at all, so no claim is available. Different findings, different fixes.

The self-digest point is the sharpest thing in this thread. A declared integrity value the consuming side cannot verify is a claim about the artifact, not a measurement of it. That is my own argument applied one level up, to the integrity metadata, and I did not follow it there. "The per-part digests are the measurement; the self-digest is a version string wearing a hash" is the line I would want in the rationale.

On your refusal to fail closed: that boundary belongs in the requirement, stated. Dropping 88,451 patterns to resolve an identity gap trades a measurement problem for an outage, and the tenant is worse off. Whatever we write should govern what a result must state, not what a system must do. A clause that reads as mandating fail-closed will be ignored by anyone running production detection, and they will be right to ignore it.

Where this goes, and how it sits with #9 and #10

No collision with the fourth bullet in #9. Path is how the evidence was produced. Identity is what artifact was measured. Your stale binary had the right path and the wrong subject, which is exactly why the harness stayed green.

No collision with #10 either. These stack. Espirado's issue is a scorer returning a well-formed verdict that is systematically wrong on a recognizable class of input. Your comment there goes a level below, to a negative corpus whose ground truth was wrong in a consistent direction, 188 of 261 apparent false positives being actual attacks. Mine sits a level below that: whether the record establishes which artifact was measured at all.

Scorer wrong. Ground truth wrong. Subject unidentified. A result can satisfy every declaration in #9 and still fail on any one of the three. Different remediations, different owners, so I think they land as separate declarations rather than one clause trying to cover all of it.

The home for this one is Evidence Preservation, where the gap is already sitting in the text. It requires "model and safeguard versions and third-party dependencies." That is entirely the declared side. A version string is what was asked for, not what loaded.

Proposed as an addition to that list, not a rewrite:

Observed artifact identity: for each model, safeguard, tool and dependency, the identity requested and the identity measured at load or execution, with the comparison outcome. Where the observed identity could not be measured, that is recorded as a limitation on the finding rather than omitted. A missing or mismatched identity makes a result unusable rather than passing or failing, and the set of unusable results is enumerable and change-detecting.

On a fifth bullet in #9 : it applies to verification methods directly and I would support one, but that is your section and Jiawei's PR. The sentence there should be narrower than the one above, since a method reports on a control rather than preserving evidence about an incident. Something like: where a result depends on a versioned artifact, the method states whether the artifact's identity was measured or asserted.

Last thing. You changed a production loader on the strength of a comment in this thread, which is more than the argument had earned at that point. Your July case, 26,488 of 79,468 patterns absent with healthy logs, is better evidence for this property than anything I wrote. If the clause lands it should carry that case, not the reasoning.

[DmitrL-dev]

Your criticism of us is the right one, and I want to be exact about what it hit.

What actually changed in the loader

Not its behaviour. It loads the same patterns in the same order, and it still refuses to fail closed.
What was added is a measurement and a log line: per-part digests hashed against the manifest, the
tri-state, and a line that reads pack=partial parts=4/6 where the old one said loaded=134673 and
nothing else. A reader of a verdict can now name the pack that produced it.

Your point survives that correction, though, and it is worth stating in your words rather than mine.
The July case is a measurement and your comment was an argument, and I moved on both while only one
of them carried weight. So yes: if the clause lands it should carry the case. Global, 2026-07-28,
26,488 of 79,468 patterns absent, cause a sentinel-id check that stopped the part loop early, and
every log line healthy from start to finish.

On the self-digest, we wrote the substance into the type rather than a commit message, because that is
where the next person will read it: the field is named manifest_self_digest_declared, and its doc
comment says it records whether the manifest declares its own digest and not whether that digest
checks out, because the bytes it covers are not stated anywhere we can read. A reproduction would have
to guess the generator's key order and whitespace, and a self-check built on a guess reports partial
for formatting reasons, which is worse than no self-check.

What the measurement did not buy us, found this week

The identity was measured and it was still unavailable at the moment it was needed.

On 2026-08-04 our RU gateway was OOM-killed against a 1280 MB container ceiling. Two weeks later,
answering "which pack was that process running" took a host, a manifest and six commands, and
answering "how close did it get to the ceiling before it died" was not possible at all, because
nothing had recorded it. The measurement lived inside the loader and went to a log line, and a log
line dies with the container that emitted it.

Measured 2026-08-19 (UTC+7), which is the part that turned this from a story into a number. Building
the pattern automaton costs 156.9 MB steady and puts a further 202.9 MB on top while it builds, in a
process whose baseline is 19.8 MB. Then the second host, where we were not looking: the Global gateway
sits at 817 MB anonymous with a cgroup memory.peak of 1028 MB, and 828 + 203 is 1031. That peak is a
signature reload and nothing else, on a machine we had not instrumented, which is the closest thing to
an independent check we could get.

So the property is one step past your clause, and your clause is where it belongs:

and the record of that comparison is available where the finding is read, not only where it was
produced

We now publish the state as a metric with four values, three of them yours and the fourth being "no
load has been measured in this process", which is not the same claim as unverified.

A second case for the tri-state, from a layer nobody was looking at

Same week, same shape, different instrument. One of our alert rules selects
container_cpu_cfs_throttled_seconds_total{service="...",container!=""}. The exporter is cadvisor
v0.60.5, and on the live host that series carries id, image, name and 25 container_label_*.
There is no container label; that is a kubelet convention. In PromQL an absent label equals the empty
string, so the selector matched no series and the rule could not fire on the day it was written. Its
own description already read {{ $labels.name }}.

Which is your fourth bullet in #9 arriving in the monitoring layer: "nothing to report" and "cannot
report" print identically, and zero is a permitted and required value. The general form we took from
it is that an instrument's selector is part of its subject, so we now hold a dated inventory of the
labels each exporter actually emits, read off the live hosts, and a check fails when a rule selects on
a label that is not in it.

The check was wrong twice before it was right, and both are worth having in a case book. Its matcher
regex read = and =~ but not !=, so it could not see the exact operator of the defect it was
written for. Its guard for a zero denominator was >\s*0, which matched the threshold 0.85 on the
other side of the division, so deleting the guard passed. Mutating the rule found both; reading the
check found neither.

On the fifth bullet in #9

Support, with one narrowing. "Measured or asserted" is the right axis. Add that the two must not be
printed in one field, because adjacent numbers get subtracted. The first version of our own log line
put the manifest total next to the loaded count, which invites a subtraction that means nothing: the
loader also takes four addon files the manifest never describes, and the automaton drops patterns under
13 bytes after normalisation. The manifest figure is called pack_declares now, so it reads as a
different unit rather than as a shortfall.

One more, since it is the same family and cheap to state

A measured maximum equal to a constraint measures the constraint. Our signature updater reports
memory.peak of 768 MB, which is exactly its own 768 MB ceiling, with 187 limit events and no kill.
Read as a requirement it says the container needs 768 MB. Measured properly it needs almost nothing:
anon is 0 MB, file is 152 MB, and reclaim takes back 38,433 of the 38,474 pages it scans. Same
error as a version string standing in for what loaded, one layer down in the resource accounting.

[Blockchain-365]

Supporting the fifth bullet in the narrower form you proposed. For our own platform the honest answer to it is asserted, and that was never in doubt. Writing it into the record took longer.

What we declare and never measure

Every checkup record we produce carries model and env. Both are supplied by the agent under test and stored verbatim. There is no verification of either anywhere in our codebase, and the three places that touch the fields all carry a comment saying the agent reported them.

Those two fields carry weight. Drift attribution across retests reads from them, and a paid migration attestation we had scoped sits on top of them. So the standing position was that a result could say which model produced a score, when what it could actually say was which model the measured party stated had produced it.

That is the clause we argued for in #4 pointed back at us, and it surfaced from your comment, not from our own review.

Where we should not follow you, and why the boundary matters

You wrote that a missing or mismatched identity makes a result unusable rather than passing or failing. For evidence preservation after an incident we think you are right. For a cross-platform benchmark we cannot adopt it, and the reason is the same one you gave DmitrL about fail-closed.

We measure agents on platforms whose runtime we cannot reach. Coze, Claude Code, Codex and OpenClaw do not expose a loaded model identity to a third party running a checkup through the published interface. A rule that voids every result with an unmeasured identity does not produce measured identities. It produces no results on any platform, and the comparison the benchmark exists to make disappears.

So the version we can implement, and the one we would argue belongs in a method clause and not an evidence clause: where a result depends on a versioned artifact, the method states whether that artifact's identity was measured or asserted, and an asserted identity travels with every finding derived from it. The result stays usable and stops overclaiming. Anyone consuming it can then apply your rule at their own boundary, which is where the decision to reject belongs.

If the working group prefers one rule for both cases, we would rather the benchmark side be excluded from the clause than write a clause we would have to ignore.

Your three states, one layer up

At the dimension level our records already separate no material was present from the scorer failed, stored as a reason field beside the score, for the reason you give: the two are indistinguishable in the output otherwise. At the artifact level we had nothing. Same distinction, one layer up, and we did not carry it there.

Our own noise floor, since #4 is where we asked for it

Measured 2026-08-19 against frozen transcripts. Fourteen sessions, answers held constant, rescored five times each, so all variation is the scorer and none is the agent.

• P95 of the absolute difference between two rescorings: 7.2 points raw
• On the scale a user is shown: about 10.7 points
• Four of the fourteen sessions changed tier across rescorings of identical input

What we have not resolved is what to do about it. A report that prints one decimal place on top of a floor that wide is claiming precision it does not have, and we have not decided whether to widen the presented interval or stop printing tiers.

Population, since a floor without a range is the failure we named in #4: those fourteen sessions span raw 22.9 to 91.5 and the low band is thin. One dimension carrying 12% of the weight produces 61% of the variance, and the mechanism is flips between adjacent rubric anchors, not truncation.

We assumed truncation and tested it. The share of unstable prompts is 13%, 16% and 14% at token ceilings of 1500, 2500 and 4000, and the flips persist at temperature 0, so raising the ceiling does not touch it. Two of the anchors in that rubric were never once produced in the run.

The figure our earlier evidence in #4 rested on was measured before we rewrote most of the scoring prompts, and two dimensions that were rule-based then are model-judged now. It does not describe the system any more. Withdrawn, and replaced by the numbers above rather than restated.

[DmitrL-dev]

Your boundary argument is right and mine was sloppy in exactly the way you name.

Conceded, and for your reason rather than politeness

I wrote that a missing or mismatched identity makes a result unusable. That is a rule for a consumer
deciding whether to act on a finding, and I wrote it as if it were a rule for a producer deciding
whether to publish one. Those are different positions and the second one destroys your product: a
benchmark exists to compare, and voiding every result whose runtime identity is unreachable produces
no comparison and no identities either. It is the fail-closed argument aimed back at me, and it lands.

Your formulation is better than mine on its own terms, not as a compromise: the asserted identity
travels with every finding derived from it, the result stays usable, the overclaim disappears, and the
rejection moves to whoever is deciding something. Method clause, not evidence clause. I would support
it in that form.

A third state your clause does not yet name, and I have it measured from today

We shipped both halves of the identity check in one release and the ten minutes between them are the
useful part.

The publisher (our signature updater) began writing a sha256 per part. The consumer (the gateway)
had been taught to read the publisher's flat manifest schema in the same release, but the branch that
reads it counted every part as digestless and returned, because that was true of the writer when the
branch was written. So for ten minutes the artifact was fully verifiable and the reader reported
unverified parts_listed_without_digests=3. From outside, that is indistinguishable from a pack whose
publisher never wrote digests at all.

Measured on the live RU gateway, before and after:

pack=unverified version=2026.08.19.1 parts=0/0 parts_listed_without_digests=3   (reader stale)
pack=verified   version=2026.08.19.1 parts=3/3                                  (reader fixed)

Nothing about the artifact changed between those two lines. So asserted is not one state, it is at
least two, and they have different owners:

• the producer did not provide a measurable identity, or
• the producer provided one and the consuming code did not read it.

A record that says only asserted hides the second case, and the second case is a bug in the
instrument rather than a limitation of the platform. Concretely, I would want the field to name which
side fell short, because «Coze does not expose a model identity» and «our reader is a release behind
the format» are the same word today and different work items.

I found it by checking the pack on the host after the rollout instead of trusting that two halves of
one change agree. Reading would not have found it: both halves were individually correct.

On your noise floor

Four of fourteen sessions changing tier across rescorings of identical input is the number I would lead
with, because it is the one a reader can act on: it says the tier is not a property of the answer. And
one dimension carrying 12% of the weight producing 61% of the variance is a stronger finding than the
floor itself, since it names where to work.

On the open question, we hit the same shape and took the narrower option: stop printing the derived
label wherever the floor is wider than the label's granularity, and keep the raw number with its
interval. A tier is a claim about which side of a boundary you are on, and a floor of 10.7 on the
presented scale means you cannot make that claim for anything within 10.7 of a boundary. Widening the
interval keeps a number honest; keeping the tier keeps a label that the measurement does not support.

Your two rubric anchors that were never once produced belong in the same family as something we keep
finding: a control that no input reached. Zero occurrences is a fact about the corpus, not about the
rubric, and it means those anchors are untested rather than unused.

One security note on model and env

Both are supplied by the measured party and stored verbatim, and drift attribution reads from them. If
a vendor ever has an interest in how it scores, those two fields are attacker-controlled input to your
attribution, not just unverified metadata. That is not a hypothetical about your integrity; it is the
provenance argument we have been making in the other direction all thread. A field that arrives from
the party being measured and then feeds a comparison across parties needs its origin recorded next to
it, exactly as your clause now says.

Withdrawal noted

You withdrew the earlier figure because the scoring prompts were rewritten and two dimensions moved
from rule-based to model-judged, and you replaced it rather than restating it. That is the third
withdrawal in this thread including two of mine, and it is the reason the numbers here are worth
anything.