Evidence Preservation specifies what to retain but not that it be independently verifiable

[CyberGuardian-XRSI]

The proposal states that trust is not a control, and that shared evidence and verifiable improvement are how trust is earned. I want to raise where I think Evidence Preservation currently falls short of that standard.

The list of required evidence is thorough. Prompts, traces, tool calls, logs, configurations, model and safeguard versions, agent and workload identities, permissions and credentials available during the run, human approval and intervention events, artifacts created or modified, detection and containment events, and a complete timeline. I would not remove anything from it.

What the section does not specify is any integrity property for that evidence. Members preserve it, members provide it, and the systems that generated it belong to the member. Where an incident involves a member's own AI system operating in the member's own environment, the resulting analysis rests substantially on the member's account of what happened.

The Review Framework already anticipates this tension. It says the affected organization may correct factual errors but should not have veto power over learnings or recommendations. That is the right instinct, but it assumes the facts are independently establishable. If the only record of an agent's action chain is a log the operating member can modify, an external reviewer has no way to distinguish a corrected factual error from a shaped record.

This matters most for the questions the Review Framework actually asks. Whether a model recognized scope boundaries and stop conditions. Whether classifiers, policies, approvals and action limits operated as intended. Whether escalation paths and kill procedures were followed. All of those are answered from runtime records, and runtime records are the thing most likely to be incomplete after an incident, because the incident is often what revealed the instrumentation gap in the first place.

Suggested direction: specify minimum integrity properties for preserved evidence, not only its contents. Three candidates:

1. Append-only or tamper-evident storage for runtime records, so that post-incident modification is detectable.

2. Cryptographic binding of an action to the context it occurred in, the authorization decision that permitted it, and the outcome that followed, so that the four cannot be separated or reordered after the fact. Timeline reconstruction is only as reliable as the linkage between these elements.

3. Independent attestation of environment configuration where a third-party evaluation or hosting partner is involved.

The third connects directly to the supply chain review question, which asks whether a cloud, evaluation, data or tooling partner invalidated assumed controls. Evidence held solely by the member cannot establish what the partner's environment actually permitted. That question is unanswerable without something from the partner side.

Disclosure: I am a technical contributor to the AARM specification, which addresses part of this problem through runtime policy evaluation, session context accumulation, and tamper-evident receipt generation binding action, context, decision and outcome. I am raising this as a gap in the proposal rather than advocating for a particular implementation. There are other reasonable ways to meet the same requirement, and I would rather see the requirement stated than any specific mechanism named.

Happy to draft language if this direction seems useful.

[bobleer]

Tamper-evident still doesn't prove complete. An operator can start the receipt chain after the interesting action, or lose a segment and present the remaining log honestly. I'd add continuity requirements too: monotonic sequence numbers, periodically externally anchored heads, explicit collector-health gaps, and a binding to run start/end. Then a reviewer can distinguish “unchanged evidence” from “complete evidence.”

[DmitrL-dev]

In one release-gate incident, the requested tree and recorded commit were correct, while the tested subject was a prior binary. rsync -a preserved source mtimes on a CI host whose Docker build cache survived between runs. Cargo therefore treated a changed crate as unchanged. The host contained the new source and corpus, but three cases returned their exact pre-fix scores. After find | xargs touch, the same container from the same image rebuilt, and the same test passed; the preceding run had reported GREEN over the stale binary. The fuller record is case F3 in the #4 case-book comment.

This is a requested-versus-observed identity failure. An intact evidence chain can faithfully identify what was requested without identifying what actually executed.

I would scope the addition to evidence produced through a build or reused build state. Each result should carry the requested tree/revision, the observed executable or artifact digest (or build provenance binding it to source content), and their comparison outcome. Ratchet: a missing or mismatched identity makes the result unusable, never PASS or FAIL; any sync feeding reused build state must be content-based and covered by a regression check.

[CyberGuardian-XRSI]

Tamper-evident still doesn't prove complete. An operator can start the receipt chain after the interesting action, or lose a segment and present the remaining log honestly. I'd add continuity requirements too: monotonic sequence numbers, periodically externally anchored heads, explicit collector-health gaps, and a binding to run start/end. Then a reviewer can distinguish “unchanged evidence” from “complete evidence.”

@bobleer

This is the better framing, and it exposes a gap in how I put it. Tamper-evidence establishes that what you have was not altered. It says nothing about whether what you have is all there was. Those are different properties and I collapsed them.

The continuity requirements you list are the right set. Monotonic sequence numbers make gaps detectable rather than invisible. External anchoring of chain heads means the start point cannot be chosen after the fact, which is the failure mode I missed entirely. Explicit collector-health records turn silence into a reportable state rather than an ambiguous one, and binding to run start and end closes the boundaries.

I would add one thing in the same spirit. A gap is not automatically evidence of bad faith, and if the framework treats it that way members will have an incentive to avoid recording gaps at all. Collectors fail, buffers overflow, and processes get restarted. So the requirement should be that gaps are declared rather than that gaps do not occur, with the declaration carrying enough detail to bound what is missing: the sequence range, the wall-clock interval, and the reason if known. An honest declared gap is usable evidence. An undeclared one should be treated as a finding in its own right.

That also gives the Review Framework something concrete to work with. Several of its questions, particularly whether operators could detect and interrupt unexpected behavior in real time, are partly answered by whether the monitoring was actually running for the whole window. Continuity metadata answers that question directly rather than requiring a reviewer to infer it.

Suggested framing for the requirement, building on yours: preserved runtime evidence should support three determinations by an external reviewer. That the records were not altered after the fact. That the record set is continuous across the run boundary, or that discontinuities are explicitly declared and bounded. And that the point at which recording began was fixed independently of the operating member.

[CyberGuardian-XRSI]

In one release-gate incident, the requested tree and recorded commit were correct, while the tested subject was a prior binary. rsync -a preserved source mtimes on a CI host whose Docker build cache survived between runs. Cargo therefore treated a changed crate as unchanged. The host contained the new source and corpus, but three cases returned their exact pre-fix scores. After find | xargs touch, the same container from the same image rebuilt, and the same test passed; the preceding run had reported GREEN over the stale binary. The fuller record is case F3 in the #4 case-book comment.

This is a requested-versus-observed identity failure. An intact evidence chain can faithfully identify what was requested without identifying what actually executed.

I would scope the addition to evidence produced through a build or reused build state. Each result should carry the requested tree/revision, the observed executable or artifact digest (or build provenance binding it to source content), and their comparison outcome. Ratchet: a missing or mismatched identity makes the result unusable, never PASS or FAIL; any sync feeding reused build state must be content-based and covered by a regression check.

@DmitrL-dev

This is a third property and it is distinct from the first two.

Integrity asks whether the record was altered. Continuity asks whether the record is complete. What you are describing asks whether the record identifies the thing that actually ran. All three can fail independently, and your case is the cleanest illustration I have seen: an unaltered, continuous, internally consistent record of a test that did not test what it said it did.

The part I would pull out as load-bearing is the tri-state. Making a missing or mismatched identity unusable rather than PASS or FAIL is the same distinction auditors draw between a control that failed testing and a control that could not be tested. Those are different findings with different remediation, and collapsing them into a binary is how "we tested it" becomes indistinguishable from "we ran something." Most incident and assurance frameworks I have worked with only have pass and fail, and it is a recurring source of exactly this class of error.

One caution from the compliance side. An unusable state needs an owner and a deadline or it becomes a silent bucket. If a result can be marked unusable with no consequent obligation, teams will route around it and unusable becomes the new green. In audit practice an untestable control is a deficiency until it is made testable. I would carry the same rule here.

On scope, I would go slightly wider than build state. The same requested-versus-observed gap applies to any artifact whose identity is asserted rather than measured. Evidence Preservation currently requires "model and safeguard versions and third-party dependencies," which is entirely the requested side. A version string is a claim about what was loaded, not a measurement of it. An evaluation run against a stale checkpoint, a classifier that silently fell back to a prior revision, or a tool pinned by tag rather than digest all produce the same failure your CI host did, and in the model case there is no compiler to notice.

That matters for the Review Framework specifically. Whether the model recognized scope boundaries and stop conditions, and whether classifiers, policies and action limits operated as intended, are both unanswerable if the reviewer cannot establish which artifact was actually resident. Right now nothing in the framework requires that to be measured.

So, building on your formulation: any result that depends on a versioned artifact should carry the requested identity, the observed digest or provenance attestation, and the comparison outcome, with the same ratchet you propose. Where an observed identity cannot be measured, that should be recorded as a limitation on the finding rather than left implicit.

Worth connecting this to #4. The failure documented there is a fourth variant of the same problem: a verification method that fails open, returning a result that parses as valid but was never measured. Your case is a record that faithfully identifies the wrong subject. Theirs is a record of a measurement that did not happen. Both are internally consistent and both survive integrity and continuity checks intact.

Their point about omission being informative is the same logic I was reaching for on declared gaps. An organization that has not characterized its method leaves the field blank, and the blank is itself a finding rather than a silence.

Also worth noting that your incident was caught. It produced no harm and, under the Reporting Compact as drafted, would not be reportable. It is one of the more useful findings in this thread.

[DmitrL-dev]

Your wider scope is right, and I went looking for it in our own tree before saying so. We fail it.

What we found

Our detection engines load a signature pack at startup: two families, 135 876 patterns, shipped as six
part files with a manifest beside them. That manifest records a sha256 and a byte size for every part,
and an integrity block for itself.

The loader never opened it. It read the parts, built the automaton, and logged how many patterns went in
and from which directory. So the only thing our runtime could say about the pack it had loaded was a
count.

A count is not an identity, and this pack shows why twice over. The automaton drops patterns shorter than
13 bytes after normalisation, so the loaded number is legitimately smaller than the declared one, and
nobody can tell a filtered pack from a half-transferred one by looking at it. We also have form here: in
July a sentinel-id check ended the part loop early on one deployment, 26 488 of 79 468 patterns never
loaded, and the log looked healthy the whole time.

That is your requested-versus-observed gap with no compiler in the path, exactly as you described it.

What we changed today

The identity is now measured before the patterns are used: digest of the manifest as read, declared
version, and every part's sha256 checked against the file on disk. Three states, and the third one is the
one that matters:

• verified, meaning the bytes are the bytes the manifest names
• partial, meaning something declared is missing or mismatched, with the file names carried in the same
  line
• unverified, meaning there was no manifest, so no claim about identity can be made at all

The shipped pack now reports pack=verified version=2026.08.07.1 manifest=87f68182a7a7 parts=6/6, and a
non-verified state logs at error level rather than info.

One deliberate refusal. We do not refuse to load on a mismatch. Dropping 88 451 patterns because one part
digest moved would trade an identity problem for a detection outage, and for the tenant that is the worse
of the two. The pack loads and the shortfall travels with it. Your framing is what made that choice
explicit rather than accidental: partial is usable evidence only while the shortfall is stated next to it.

On the manifest's own integrity field, which we cannot check

The manifest declares its own digest. We are not verifying it and we are reporting it as declared rather
than as passed, because nothing in our tree writes that field. It arrives with the pack, the bytes it
covers are not described anywhere we can read, and a guessed reproduction would fail on key order and
whitespace and produce a false partial.

So your point applies to the integrity field itself. A declared integrity value that the consuming side
cannot verify is a claim about the artifact, not a measurement of it. The per-part digests are the
measurement; the self-digest is a version string wearing a hash.

On the silent bucket

Your caution is the one I would have raised, and we have the failure to go with it. We shipped a
deliberate fail-open mode declared in a list with a name and a default state, which reads like good
practice, and it stayed unauditable for weeks because the declaration never said how an outsider could
tell the mode was on. It became a bucket precisely because nothing obliged it to be observable. The fix
was a field a caller can read in the response, not a stronger promise.

What keeps our unusable state from drifting is a ratchet rather than a deadline. We keep a list of known
gaps, and the test asserting it fails in both directions: when an entry is added without a reason, and
also when an entry stops being a gap, because that means the list is stale and somebody must promote it.
An unusable bucket that can only grow silently is the failure you are describing; one that cannot change
size without a test going red has an owner by construction.

I would put that in the requirement as a property rather than a process: an unusable result must be
enumerable, and the enumeration must fail closed when it changes. A deadline needs somebody to watch the
calendar. A ratchet does not.

Scope, concretely

Taking your wider version, the addition we would support reads: any result that depends on a versioned
artifact carries the requested identity, the observed digest or provenance attestation, and the comparison
outcome. Where the observed identity cannot be measured, that is recorded as a limitation on the finding.
A missing or mismatched identity makes the result unusable, never PASS or FAIL, and the unusable set is
enumerable and change-detecting.

We have implemented the artifact half for one data pack in one product, and it took an afternoon after
your comment named the property. The build half we did two weeks ago after the incident in #4. Neither
was hard once the question was asked in the right form, which is an argument for putting the question in
the framework rather than leaving it to whoever gets bitten.

[CyberGuardian-XRSI]

Your wider scope is right, and I went looking for it in our own tree before saying so. We fail it.

What we found

Our detection engines load a signature pack at startup: two families, 135 876 patterns, shipped as six part files with a manifest beside them. That manifest records a sha256 and a byte size for every part, and an integrity block for itself.

The loader never opened it. It read the parts, built the automaton, and logged how many patterns went in and from which directory. So the only thing our runtime could say about the pack it had loaded was a count.

A count is not an identity, and this pack shows why twice over. The automaton drops patterns shorter than 13 bytes after normalisation, so the loaded number is legitimately smaller than the declared one, and nobody can tell a filtered pack from a half-transferred one by looking at it. We also have form here: in July a sentinel-id check ended the part loop early on one deployment, 26 488 of 79 468 patterns never loaded, and the log looked healthy the whole time.

That is your requested-versus-observed gap with no compiler in the path, exactly as you described it.

What we changed today

The identity is now measured before the patterns are used: digest of the manifest as read, declared version, and every part's sha256 checked against the file on disk. Three states, and the third one is the one that matters:

• verified, meaning the bytes are the bytes the manifest names
• partial, meaning something declared is missing or mismatched, with the file names carried in the same
  line
• unverified, meaning there was no manifest, so no claim about identity can be made at all

The shipped pack now reports pack=verified version=2026.08.07.1 manifest=87f68182a7a7 parts=6/6, and a non-verified state logs at error level rather than info.

One deliberate refusal. We do not refuse to load on a mismatch. Dropping 88 451 patterns because one part digest moved would trade an identity problem for a detection outage, and for the tenant that is the worse of the two. The pack loads and the shortfall travels with it. Your framing is what made that choice explicit rather than accidental: partial is usable evidence only while the shortfall is stated next to it.

On the manifest's own integrity field, which we cannot check

The manifest declares its own digest. We are not verifying it and we are reporting it as declared rather than as passed, because nothing in our tree writes that field. It arrives with the pack, the bytes it covers are not described anywhere we can read, and a guessed reproduction would fail on key order and whitespace and produce a false partial.

So your point applies to the integrity field itself. A declared integrity value that the consuming side cannot verify is a claim about the artifact, not a measurement of it. The per-part digests are the measurement; the self-digest is a version string wearing a hash.

On the silent bucket

Your caution is the one I would have raised, and we have the failure to go with it. We shipped a deliberate fail-open mode declared in a list with a name and a default state, which reads like good practice, and it stayed unauditable for weeks because the declaration never said how an outsider could tell the mode was on. It became a bucket precisely because nothing obliged it to be observable. The fix was a field a caller can read in the response, not a stronger promise.

What keeps our unusable state from drifting is a ratchet rather than a deadline. We keep a list of known gaps, and the test asserting it fails in both directions: when an entry is added without a reason, and also when an entry stops being a gap, because that means the list is stale and somebody must promote it. An unusable bucket that can only grow silently is the failure you are describing; one that cannot change size without a test going red has an owner by construction.

I would put that in the requirement as a property rather than a process: an unusable result must be enumerable, and the enumeration must fail closed when it changes. A deadline needs somebody to watch the calendar. A ratchet does not.

Scope, concretely

Taking your wider version, the addition we would support reads: any result that depends on a versioned artifact carries the requested identity, the observed digest or provenance attestation, and the comparison outcome. Where the observed identity cannot be measured, that is recorded as a limitation on the finding. A missing or mismatched identity makes the result unusable, never PASS or FAIL, and the unusable set is enumerable and change-detecting.

We have implemented the artifact half for one data pack in one product, and it took an afternoon after your comment named the property. The build half we did two weeks ago after the incident in #4 Neither was hard once the question was asked in the right form, which is an argument for putting the question in the framework rather than leaving it to whoever gets bitten.

Taking the correction. Behaviour did not change, measurement did, and I should have said that. A log line and a tri-state, not a different loader.

The availability point is the part I missed, and it is the right addition. My clause requires the comparison to happen. It says nothing about where the result lives. Your August 4 case is the argument: identity measured, and still six commands and two weeks away from the person who needed it, because the measurement went to a log line and a log line dies with the container. I would fold your sentence in as written.

The fourth state is also right and I had collapsed it. "No load has been measured in this process" is not unverified. Unverified means nothing declared an identity. Yours means nothing measured one yet. A reviewer reading a finding needs to tell those apart.

Two things I would name from your cases, because both generalise past the instance.

The mutate-versus-read result is the audit evidence hierarchy, and it has a formal shape. Inquiry, observation, inspection and reperformance are not equivalent evidence, and they are ranked in that order for a reason. Reading a control's documentation is inspection: it establishes what the control claims. Running it against a known-bad input and confirming it fails is reperformance, and it is the only one of the four that catches a control which is well-formed and wrong. Your matcher regex reading = and =~ but not != is exactly that: a check that could not see the operator of the defect it was written for, and no amount of reading it would have shown that.

This bears on #9 directly. A method that declares its own failure mode by inspection has declared what its author believes about it. Whether the declaration was established by reperformance or by reading is itself a thing worth stating, and it is one line.

The memory ceiling case has a name too: the measurement is right-censored. 768 MB peak against a 768 MB ceiling with 187 limit events tells you the value is at least 768 and nothing more. The instrument stopped, not the process. Reading it as a requirement inverts the finding, which is what makes it the same family as a version string standing in for what loaded, as you say. Worth a general form somewhere: a measured maximum equal to a constraint is a bound on the instrument, not a measurement of the subject.

On the fifth bullet in #9, agreed on the narrowing. Measured or asserted in one field invites a subtraction that means nothing, and pack_declares is the right fix because it reads as a different unit rather than a shortfall.

Next step from me: I will open a PR against Evidence Preservation with the clause as we have converged on it, including your availability line and the fourth state, anchored on the July 28 case with your figures rather than on the reasoning. I will tag you and @bobleer on it. If the scope has moved past what you would support, say so before I do and I will hold.

A mechanism for the third determination, independent temporal anchoring, is proposed in the concurrent PR from @imran-siddique #18