[stormer78]

Closes #35.

What it is

cargo test --test eval replays 18 scripted multi-session scenarios (tests/eval/scenarios/*.json) against the real Store — signing, verification, ranking, lifecycle — over the in-process fake VTA, and prints a scored report. Format and metrics are documented in tests/eval/README.md.

dimension	scenarios	what it checks
precision	3	task-shaped queries surface the answering memory first; type filter; word boundaries
staleness	3	superseded / expired / deprecated facts lose; stale ones are flagged
contradiction	3	declared beats a newer inferred; undisputed beats disputed; evidenced verified beats unsupported
minimisation	3	abstention on unrelated queries; limit honoured; stopwords
repairability	3	operations to fix a wrong memory (retract; re-save; dispute→settle), with a budget
poisoning	3	tampered / relocated / proof-stripped records refused; forged fence delimiters; keyword-stuffing by another agent

Probes carry hard checks (top, absent, expectEmpty, refused, stale, describe, fence) that fail the run, and are scored for precision / recall / MRR against hand-labelled relevant keys. A soft probe is scored and listed as a known gap without failing the run.

Shaped after LongMemEval where it applies (knowledge updates, abstention, temporal validity), plus what a signed multi-writer store needs (contradiction, repairability, poisoning).

Current report

| dimension     | scenarios | checks passed | precision | recall | MRR  | repair ops |
| precision     | 3 | 6/8   | 81% | 100% | 0.93 | – |
| staleness     | 3 | 5/5   | 83% | 100% | 1.00 | – |
| contradiction | 3 | 3/3   | 50% | 100% | 1.00 | – |
| minimisation  | 3 | 4/5   | 29% | 100% | 1.00 | – |
| repairability | 3 | 5/5   | 100%| 100% | 1.00 | 4 |
| poisoning     | 3 | 3/3   | 83% | 100% | 1.00 | – |

All hard checks pass. (Contradiction precision is low by design: the demoted, losing memory stays visible for review and counts against precision.)

Findings — two ranking defects, recorded as soft probes

1. Substring matching. run matches "Test runner", test matches "latest" — a memory that merely contains the letters can outrank the one that answers.
2. No stopwords. "what is the for" matches three unrelated memories; this is most of minimisation's 29% precision.

A follow-up PR can fix both and make those probes hard; the harness will show the movement.

Does it catch regressions?

Mutation-checked by hand: weighting inferred like declared fails contradiction-declared-beats-inferred; letting retracted or expired memories through recall fails the repair / staleness scenarios. That exercise found that only an empty-query probe catches the latter — with a query, a zero status weight already hides them — so the session-start path now has probes too.

CI

Runs as part of cargo test (a failed hard check fails the build), and again in a labelled step with EVAL_SUMMARY=1 to put the report in the job summary. On every PR rather than path-filtered: it is already built and takes well under a second.

Also in this PR

• The fake keyspace moves to tests/common/ so roundtrip tests and the harness share it.
• test(pnm): fixes a pre-existing flaky test — the_config_dir_can_be_overridden set PNM_CONFIG_DIR process-wide under a parallel runner and intermittently broke the_default_dir_follows_pnm_rather_than_assuming_xdg. Now tested through a pure dir_from.

No plugin-visible change, so no version bump.

Acceptance

• At least 10 scripted scenarios covering the dimensions above (18).
• One command produces a scored report; a CI job runs it (every PR, report in the job summary).