Skip to content

test(eval): score recall across personal memory and Data Rooms - #50

Merged
stormer78 merged 1 commit into
mainfrom
test/eval-rooms
Nov 11, 2026
Merged

stormer78 merged 1 commit into
mainfrom
test/eval-rooms

Conversation

@stormer78

Copy link
Copy Markdown
Contributor

Follow-up to #34 / #49: the evaluation now scores room recall as well as personal memory.

Harness

  • Probes now use the same path as memory_recall: rooms::recall over personal memory and the scenario's rooms. A new optional sources field on a probe narrows where it looks. Room results are named room:<label>/<key> in every probe list.
  • New room step: writes a sealed room record signed by a named member. It can also write:
    • an open (unsigned) record;
    • a deprecated, retracted or expired record;
    • a deliberately broken record: alter (changed after signing), servedAs (moved to another key), or hostAuthor (the host records a different author).
  • New sources dimension. tests/eval/README.md documents the step and the field.

Scenarios (19–22)

scenario checks
sources-room-and-personal One ranking across both: an observed room record outranks an inferred memory. sources narrows to personal only or to one room. A memory type leaves rooms out.
sources-room-poisoning Altered, relocated and wrong-author room records are absent and reported refused, with a query and with none, and the fence holds.
sources-room-standing Deprecated records rank lower; retracted and expired records never appear, with a query and with none.
sources-room-abstention An unrelated query returns nothing even with rooms configured. An open room's record is recalled.

Numbers

before after
existing six dimensions unchanged unchanged
sources — 10/10 checks, precision 100%, MRR 1.00
all 29/29, precision 82% 39/39, precision 86%

Switching probes to the unified recall path moved no existing metric.

Mutation check: disabling room signature verification fails sources-room-poisoning (8/10, five hard failures), so the scenario guards the check it claims to.

Tests only, so no version bump (as with #35).

Probes now take memory_recall's path (rooms::recall), with optional `sources`. A `room` step writes a signed (or open, or deliberately broken) room record. Four scenarios in a new `sources` dimension cover one ranking across both, narrowing, room curation, abstention, and refusing room records whose signature does not hold. Existing scenarios' numbers are unchanged.

Signed-off-by: Glenn Gore <glenn.g@affinidi.com>
@stormer78
stormer78 merged commit dd37d49 into main Nov 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant