Skip to content

Verify: Recall scoring defects (prefix inflation, types, graph order) #467

Closed

Description

@possiblyneal

Part of #448

Question

Does BM25 prefix scoring inflate scores here, measured on the evals? Do search responses lose original Memory types? Do graph scores depend on start-node order (graph is off by default)? Does the procedural tier ever produce a procedure, and does consolidation lose provenance? Upstream: rohitg00#1514, rohitg00#1646, rohitg00#1558, rohitg00#1685, rohitg00#1545.

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    #467 verdict (worktree tmp/wt/verify-467, branch verify/467-recall-scoring, base fdc3d45)

    Operator reach figures are from tmp/verify/live-copy.sqlite (read-only): 42 Memories (40 latest: bug 19, fact 8, workflow 7, preference 2, pattern 2, architecture 1, rule 1), 0 with sessionIds, 0 procedural rows, 0 relations.

    rohitg00#1514 prefix inflation

    • Partly present. The unbounded SUM across prefix siblings is already fixed here (search-index.ts scoreTerms takes the best sibling; test/search-index-prefix.test.ts). Remaining half: a document that already matches the exact term still gets a prefix bonus on top.
    • Repro: test/search-index-prefix.test.ts "does not credit a prefix sibling to a document that already matches the exact term": fails 1/1 on base (8.39 vs 5.39, equal length, both exact tf 1).
    • Prototype: if (matchingDocs?.has(obsId)) continue; in the prefix loop (src/state/search-index.ts, 1 line). Targeted search-index tests pass.
    • Measured, NOT APPLIED (inconclusive). With the line in, eval:suite fast vs base-fdc3d459: pmb.pass 0.447 → 0.468 (Δ 0.021, CI [0.000, 0.106]), pmb.activePass 0.391 → 0.435 (CI [0.000, 0.217]), pmb.precision +0.014; coding-life dev and LongMemEval Δ 0.000. But eval:gate FAILED: prompt-submit precision 0.889 → 0.856 (−0.033, tolerance 0.02). A gain that never clears 0 against a gate loss does not ship. Line reverted; its repro test is kept aside in tmp/verify/467/prefix-lever-test.patch.
    • Retry if: a fuller pmb or LongMemEval run (holdout or full tier) shows a gain whose interval clears 0, and the prompt-submit precision loss is traced to a gate item it can be fixed for. Needs a docs/experiments.md row (inconclusive) at apply time.
    • Retry if: eval:suite --tier fast shows a gain on LongMemEval or coding-life dev with no loss on PrecisionMemBench.

    rohitg00#1646 original Memory type lost in search

    • Holds. memoryToObservation hard-codes type "decision"; compact smart-search and mem::search compact drop the saved type (fact, bug, rule...).
    • Repro: test/smart-search.test.ts "compact results keep a saved Memory's own type..." fails 1/1 on base (memoryType undefined).
    • Fix (applied in worktree): additive optional memoryType on CompressedObservation and CompactSearchResult, set in memoryToObservation, copied into the two compact builders (smart-search.ts, search.ts). Full/narrative/expanded shapes carry the whole observation so they get it from memoryToObservation. Index unchanged, no migration. Typecheck clean.
    • Reach: all 40 latest Memories are shown as "decision" today; 8 facts, 19 bugs etc. indistinguishable.
    • Class 2: no ranking or injected-text change; adds a field to the REST/MCP response (additive public shape, so mention in PR). No eval:suite needed. Does not touch recall rendering (the memory_recall full/narrative render code was not changed; upstream's budget-clipped preview handling not reproduced).

    rohitg00#1558 graph first-come visitedObs

    • Holds in code (graph-retrieval.ts scoreEntityMatches and scoreExpansion). Fork already sorts start nodes by id, so it is deterministic but still first-wins: an own observation (1.0) is buried at a 2-hop score if an earlier-id start node reaches it.
    • Repro: test/graph-retrieval.test.ts "scores an observation at its best path..." fails 1/1 on base (0.267 vs 1).
    • Fix (applied): keep best score per obsId (Map) in both functions; given obsIds still excluded in scoreExpansion.
    • Reach: latent. Graph extraction is off by default and the graph rows are deleted at startup (Measure the entity graph's Recall value; turn off graph extraction if it adds none #420). No eval:suite (graph leg is off in the evals). Low bar: fine to land or defer.
    • Retry if (if deferred): graph extraction is turned on by default.

    rohitg00#1685 procedural tier never yields a procedure

    • Defect confirmed on Operator data, fix NOT APPLIED. consolidation-pipeline.ts procedural tier uses m.sessionIds.length || 1 >= 2; Memories saved by memory_save/remember carry no sessionIds (Operator: 0 of 42), so frequency is always 1. Operator has 2 latest pattern Memories, 0 procedural rows ever.
    • Why not fixed: no stored field carries a recurrence count (sourceObservationIds also empty), so any fix invents a signal. And nothing reads KV.procedural for Recall (only skill-extract, diagnostics, mesh, export); a working tier would spend LLM calls and disk on rows the Agent never sees, against "least record that restates". Open question for the Operator: delete the procedural tier (and its diagnostics line) rather than fix it. Deleting is a behavior/API removal so not done here.
    • Repro: not written as failing (the right fix is undecided). Observable now: POST consolidate-pipeline tier procedural returns skipped "fewer than 2 recurring patterns" with 2 patterns.
    • Retry if: a Recall or Injection surface starts reading procedural memories, or Memories gain a recurrence counter.

    rohitg00#1545 consolidation provenance / version heads (mem::consolidate)

    • Three sub-claims hold in consolidate.ts, applied in worktree, all failing on base:
      1. Concurrent runs publish duplicate Memories: test "keeps one latest Memory when two runs overlap" fails 1/1 (2 latest). Fix: serialize mem::consolidate with withKeyedLock("mem::consolidate").
      2. Predecessor demoted before successor stored, so a failed write hides the Memory: test "keeps the previous version latest when writing its successor fails" fails 1/1 (no latest). Fix: store successor first, then demote. Residual: a failed demote leaves two latest briefly (upstream's rollback/repair not reproduced; mem::remember supersession handles heads).
      3. Failed model attempts do not count toward the 10-call budget: test "counts a failed model call..." fails 1/1 (12 calls). Fix: increment before the call.
    • Not applied: source-mutation rechecks, lost-acknowledgement confirmation and version-chain repair (distributed-write machinery; single Operator, one process; no evidence of occurrence).
    • Reach: low. Operator store has no consolidation-written Memories (all 42 have empty sessionIds; consolidate writes them). Class 2: no for 1 and 3 on Recall ranking; 2 changes storage write order only. No eval:suite.
    • Retry if (unapplied bits): an audit shows a Memory chain with 2 latest or an orphaned successor.

    Bar (tmp/verify/467-verify.log, run with the prefix line): typecheck 0 errors; npm test 3,122 passed, 1 skipped; eval:gate FAILED on prompt-submit precision only; eval:gate:rules pass; eval:suite exit 0, no metric worse. After reverting the prefix line: eval:gate and eval:gate:rules pass with every metric at baseline; the touched test files 67/67 pass; typecheck 0 errors. The applied set (memoryType field, graph best-path, consolidate lock/order/budget) does not change ranking, so the suite run above bounds it. Branch verify/467-recall-scoring (uncommitted, worktree tmp/wt/verify-467).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions