diff --git a/docs/adrs/ADR-0010-evidence-freshness-before-promotion.md b/docs/adrs/ADR-0010-evidence-freshness-before-promotion.md new file mode 100644 index 0000000..2e78b1f --- /dev/null +++ b/docs/adrs/ADR-0010-evidence-freshness-before-promotion.md @@ -0,0 +1,134 @@ +# ADR 0010: Evidence freshness — re-verify the read set before promotion + +Status: Proposed + +Date: 2026-09-07 + +Related: ADR-0007 (claim-relative evidence receipts), ADR-0009 (discovery evidence before promotion), ADR-0001 (the nightly loop) + +## Context + +Every evidence gate in this repository freezes a policy *before* outcomes are +visible and then treats the resulting receipt as timeless. `ClaimReceipt` binds +experiment coverage. `DiscoveryEvidenceReceipt` binds scientific criteria. +Neither asks a question that turned out to matter more than either: **is this +evidence still about the tree we are promoting into?** + +That assumption is safe when a candidate lands the same night. It is not safe in +the regime this repository actually operates in. Between 2026-08-13 and +2026-09-07, 30+ evaluated candidates accumulated unmerged; the backlog was +landed in a single session on 2026-09-07. + +PR #11 made the failure concrete and measurable. It was evaluated on 2026-08-15 +against `dream.config.json` as it stood that day, asserting +`withDefaults(selfConfig).autoMerge === true` and golden-snapshotting the +compiled prompt. Its evaluation was honest and passed. Three weeks later the +repository deliberately set `autoMerge: false` and the compiled prompt evolved. +On landing, `main` went red on two tests. + +The load-bearing detail is that **git merged #11 cleanly**. There was no +conflict, because `dream.config.json` was never in #11's diff. It was in #11's +*read set*: a file the evaluation depended on but did not modify. + +> git protects the WRITE set. Nothing here protected the READ set. + +This is not "receipts get old". Age was never the mechanism — a three-week-old +candidate whose dependencies never moved is still valid. The mechanism is an +**undeclared environmental dependency set that nothing re-verifies at promotion +time**. + +## Decision + +Add a deterministic `EvidenceFreshnessPolicy` / `EvidenceFreshnessReceipt` pair +to `@dream-machine/witness`, and a `dream-machine freshness` CLI that performs +the I/O. + +The policy is frozen at evaluation time and contains only bounded metadata: + +1. stable policy identity +2. the base commit the evaluation ran against +3. canonical evaluation timestamp +4. the **declared read set** — every path whose content the evaluation's result + depends on, each with a SHA-256 digest of its content at evaluation time +5. whether an empty read set is rejected (default yes) +6. an optional secondary `maxAgeDays` bound + +At promotion time the same paths are re-digested against the target tree and +compared. Verdicts: + +- `FRESH` — every declared dependency still has its evaluated content +- `STALE` — one or more drifted, or an explicit age budget was exceeded +- `INDETERMINATE` — a declared path was not observed, or the observation carries + paths the policy never declared; unverifiable is never treated as fresh +- `INVALID` — malformed input, or the policy no longer matches its anchored digest + +Every receipt carries `authority: 'none'`, consistent with ADR-0007/0009. A +FRESH verdict is evidence that an evaluation still applies. It is never +permission to merge. + +Three deliberate non-goals, because getting them wrong makes the gate useless: + +- **A moved base commit is not staleness.** `main` advances constantly; a gate + that fires whenever `HEAD != baseCommit` fires always, gets ignored, and + protects nothing. `baseCommitMoved` is reported as context only. +- **Age is not the primary signal.** It is an optional secondary bound for + callers who want one, never the mechanism. +- **The witness primitive performs no I/O.** The caller supplies observed + digests, so the same pure function serves CI, a promotion gate, and tests. + +## Consequences + +A candidate can now carry, alongside its evaluation receipt, a machine-checkable +claim about the environment that evaluation assumed — and that claim is +re-checked against the tree it would land in. + +The cost is that the read set must be declared. An undeclared read set is not +evidence that a candidate has no environmental dependencies; it is evidence that +nobody looked, which is why `requireDeclaredDependencies` defaults to true. Read +sets that are too narrow will still miss drift; this gate raises the floor, it +does not prove independence. + +This does not close the promotion loop by itself. It supplies the missing +freshness input that a bounded auto-landing path would need before it could +safely act on a three-week-old ACCEPT verdict. + +## Alternatives Considered + +- **Re-run the full evaluation at promotion time.** Strictly stronger and + strictly more expensive; for a 30-candidate backlog it is the thing that does + not happen, which is how the backlog formed. Freshness is the cheap check that + says whether the expensive one is needed. +- **Gate on base-commit equality.** Trivially implementable, fires on every + candidate in a live repository, and would have been switched off within a day. +- **Gate on age alone.** Would have caught #11 (three weeks old) but also blocks + correct old candidates and passes a one-day-old candidate whose dependencies + moved. Wrong axis. +- **Rely on git conflict detection.** This is the status quo, and it is exactly + what missed #11: the diff merged cleanly. + +## Test Contract + +- `FRESH` when the declared read set is unchanged; unaffected by a moved HEAD or + by arbitrary age with no drift +- `STALE` reproducing PR #11's exact shape — a clean merge whose read set drifted +- `STALE` when an explicit age budget is exceeded; drift reported alongside age +- `INDETERMINATE` for unobserved declared paths and for undeclared observed paths +- `INVALID` for a policy rewritten after its digest was anchored (the evasion + path: silently dropping the dependency that drifted) +- rejection of absolute paths, `..` traversal, duplicate paths, malformed + digests/commits/timestamps, non-boolean and out-of-range runtime values, and + read sets beyond `MAX_EVIDENCE_DEPENDENCIES` +- receipts contain digests and paths only, never file content +- the CLI validates and anchors the policy **before** reading any dependency, so + an untrusted policy file cannot be used as a read/existence oracle +- cost bound: sub-quadratic scaling from 512 to 4096 dependencies + +Validated end-to-end against real history: stamping `dream.config.json` from +`8ce3857` (PR #11's evaluation base) and verifying against current `main` yields +`STALE`, `driftedPaths: ["dream.config.json"]`, exit 1. + +## References + +- PR #11, PR #95 (the repair), PR #96 (union-merge damage from the same landing) +- `packages/witness/src/evidence-freshness.ts` +- ADR-0007, ADR-0009 diff --git a/docs/adrs/INDEX.md b/docs/adrs/INDEX.md index e7f235e..6a9877e 100644 --- a/docs/adrs/INDEX.md +++ b/docs/adrs/INDEX.md @@ -13,6 +13,7 @@ Alternatives Considered → Test Contract → References. | [ADR-0006](./ADR-0006-root-cause-security-patch-evaluation.md) | Root cause security patch evaluation | Proposed | | [ADR-0007](./ADR-0007-claim-relative-evidence-receipts.md) | Claim-relative evidence receipts bind sufficiency and committed experiment coverage without granting authority | Proposed | | [ADR-0008](./ADR-0008-provenance-bound-environment-reconstruction.md) | Provenance bound environment reconstruction | Proposed | +| [ADR-0010](./ADR-0010-evidence-freshness-before-promotion.md) | Evidence freshness — re-verify the read set before promotion | Proposed | | [ADR-0100](./ADR-0100-edge-runtime-trust-boundaries.md) | Separate the Dream Machine control plane from the bedside runtime and actuator safety authority | Proposed | | [ADR-0101](./ADR-0101-uno-q-ruview-home-core-runtime.md) | Governed edge runtime on Arduino UNO Q with RuView HOMECORE | Proposed | | [ADR-0102](./ADR-0102-apple-watch-healthkit-local-bridge.md) | Apple Watch HealthKit local bridge with retrospective default and research-only live sensing | Proposed | diff --git a/packages/cli/src/index.test.ts b/packages/cli/src/index.test.ts index 6ba4f8f..2e7cefb 100644 --- a/packages/cli/src/index.test.ts +++ b/packages/cli/src/index.test.ts @@ -248,6 +248,133 @@ describe('ledger', () => { }); }); +describe('freshness', () => { + const cfgV1 = '{"autoMerge":true}'; + const cfgV2 = '{"autoMerge":false}'; + const srcV1 = 'export const compile = 1;'; + + async function stampPolicy(files: Record, extra: string[] = []) { + const io = mockIO(files); + const r = await run( + ['freshness', 'stamp', '--base', 'abc1234', '--paths', Object.keys(files).join(','), '--out', 'p.json', ...extra], + io, + ); + return { io, r }; + } + + it('stamp freezes the declared read set and reports a policy digest', async () => { + const { io, r } = await stampPolicy({ 'dream.config.json': cfgV1, 'src.ts': srcV1 }); + expect(r.code).toBe(0); + const doc = JSON.parse(io.files['p.json']); + expect(doc.policyDigest).toMatch(/^[0-9a-f]{64}$/); + expect(doc.policy.dependencies.map((d: { path: string }) => d.path).sort()).toEqual([ + 'dream.config.json', + 'src.ts', + ]); + }); + + it('verify is FRESH (exit 0) when the read set has not moved', async () => { + const { io } = await stampPolicy({ 'dream.config.json': cfgV1, 'src.ts': srcV1 }); + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'def5678'], io); + expect(r.code).toBe(0); + expect(JSON.parse(r.out).status).toBe('FRESH'); + }); + + // The PR #11 regression, end to end through the CLI. + it('verify is STALE (exit 1) when a read-set file drifted, even with no diff conflict', async () => { + const { io } = await stampPolicy({ 'dream.config.json': cfgV1, 'src.ts': srcV1 }); + io.files['dream.config.json'] = cfgV2; // changed underneath, never in the candidate's diff + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'def5678'], io); + expect(r.code).toBe(1); + const receipt = JSON.parse(r.out); + expect(receipt.status).toBe('STALE'); + expect(receipt.driftedPaths).toEqual(['dream.config.json']); + expect(r.err).toContain('Re-evaluate before promoting'); + }); + + it('verify is indeterminate (exit 2) when a declared dependency vanished', async () => { + const { io } = await stampPolicy({ 'dream.config.json': cfgV1, 'src.ts': srcV1 }); + delete io.files['src.ts']; + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'def5678'], io); + expect(r.code).toBe(2); + expect(JSON.parse(r.out).status).toBe('INDETERMINATE'); + expect(r.err).toContain('no longer present'); + }); + + it('verify rejects a policy whose read set was rewritten after stamping', async () => { + const { io } = await stampPolicy({ 'dream.config.json': cfgV1, 'src.ts': srcV1 }); + const doc = JSON.parse(io.files['p.json']); + doc.policy.dependencies = doc.policy.dependencies.filter( + (d: { path: string }) => d.path !== 'dream.config.json', + ); + io.files['p.json'] = JSON.stringify(doc); + io.files['dream.config.json'] = cfgV2; + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'def5678'], io); + // Rejected at parse time, before any dependency is read -- so there is no + // receipt on stdout at all, which is strictly stronger than emitting an + // INVALID one after having already touched the filesystem. + expect(r.code).toBe(2); + expect(r.out).toBe(''); + expect(r.err).toMatch(/digest mismatch/); + }); + + it('stamp fails clearly when a declared dependency cannot be read', async () => { + const io = mockIO({ 'a.ts': 'x' }); + const r = await run(['freshness', 'stamp', '--base', 'abc1234', '--paths', 'a.ts,missing.ts'], io); + expect(r.code).toBe(2); + expect(r.err).toContain('cannot read declared dependency missing.ts'); + }); + + it('stamp rejects an absolute or traversing dependency path', async () => { + const io = mockIO({ '/etc/shadow': 'root' }); + const r = await run(['freshness', 'stamp', '--base', 'abc1234', '--paths', '/etc/shadow'], io); + expect(r.code).toBe(2); + expect(r.err).toMatch(/repository-relative/); + }); + + it('never reads a path outside the repo, even before rejecting the policy', async () => { + // Regression: the CLI used to digest every declared path first and only + // then let the library reject traversal, which made an attacker-supplied + // policy file a read/existence oracle for paths outside the repository. + const reads: string[] = []; + const io = mockIO({ 'p.json': JSON.stringify({ + policy: { + policyId: 'evil', + baseCommit: 'abc1234', + evaluatedAt: '2026-09-01T00:00:00.000Z', + dependencies: [{ path: '../../../etc/hosts', digest: '0'.repeat(64) }], + requireDeclaredDependencies: true, + }, + policyDigest: '0'.repeat(64), + }) }); + const inner = io.readFile.bind(io); + io.readFile = async (p: string) => { reads.push(p); return inner(p); }; + + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'abc1234'], io); + expect(r.code).toBe(2); + expect(reads).toEqual(['p.json']); // the policy itself, and nothing else + expect(r.err).toMatch(/malformed policy|traverse/); + }); + + it('rejects a policy whose digest does not anchor its read set', async () => { + const io = mockIO({ 'a.ts': 'x' }); + await run(['freshness', 'stamp', '--base', 'abc1234', '--paths', 'a.ts', '--out', 'p.json'], io); + const doc = JSON.parse(io.files['p.json']); + doc.policyDigest = 'f'.repeat(64); // anchor no longer matches the policy + io.files['p.json'] = JSON.stringify(doc); + const r = await run(['freshness', 'verify', '--policy', 'p.json', '--head', 'def5678'], io); + expect(r.code).toBe(2); + expect(r.err).toMatch(/digest mismatch/); + }); + + it('errors with usage on a missing subcommand or flags', async () => { + const io = mockIO({}); + expect((await run(['freshness'], io)).code).toBe(2); + expect((await run(['freshness', 'stamp'], io)).code).toBe(2); + expect((await run(['freshness', 'verify'], io)).code).toBe(2); + }); +}); + describe('witness', () => { it('stamp prints the triple', async () => { const io = mockIO({ 'r.md': 'report body' }); diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 2c1f18c..3bb0dd9 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -17,7 +17,15 @@ import { EVALS, type LedgerRow, } from '@dream-machine/ledger'; -import { stamp, verify, verifySteps } from '@dream-machine/witness'; +import { + stamp, + verify, + verifySteps, + evaluateEvidenceFreshness, + evidenceFreshnessPolicyDigest, + type EvidenceDependency, + type EvidenceFreshnessPolicy, +} from '@dream-machine/witness'; import { serializeRoutine, scheduleInstructions } from '@dream-machine/schedule'; import { renderDashboard } from './tui.js'; import { classifyEntrypointResult, type ExecResult } from './entrypoint.js'; @@ -124,7 +132,9 @@ Commands: verify-entrypoint