You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Does BM25 prefix scoring inflate scores here, measured on the evals? Do search responses lose original Memory types? Do graph scores depend on start-node order (graph is off by default)? Does the procedural tier ever produce a procedure, and does consolidation lose provenance? Upstream: rohitg00#1514, rohitg00#1646, rohitg00#1558, rohitg00#1685, rohitg00#1545.
Partly present. The unbounded SUM across prefix siblings is already fixed here (search-index.ts scoreTerms takes the best sibling; test/search-index-prefix.test.ts). Remaining half: a document that already matches the exact term still gets a prefix bonus on top.
Repro: test/search-index-prefix.test.ts "does not credit a prefix sibling to a document that already matches the exact term": fails 1/1 on base (8.39 vs 5.39, equal length, both exact tf 1).
Prototype: if (matchingDocs?.has(obsId)) continue; in the prefix loop (src/state/search-index.ts, 1 line). Targeted search-index tests pass.
Measured, NOT APPLIED (inconclusive). With the line in, eval:suite fast vs base-fdc3d459: pmb.pass 0.447 → 0.468 (Δ 0.021, CI [0.000, 0.106]), pmb.activePass 0.391 → 0.435 (CI [0.000, 0.217]), pmb.precision +0.014; coding-life dev and LongMemEval Δ 0.000. But eval:gate FAILED: prompt-submit precision 0.889 → 0.856 (−0.033, tolerance 0.02). A gain that never clears 0 against a gate loss does not ship. Line reverted; its repro test is kept aside in tmp/verify/467/prefix-lever-test.patch.
Retry if: a fuller pmb or LongMemEval run (holdout or full tier) shows a gain whose interval clears 0, and the prompt-submit precision loss is traced to a gate item it can be fixed for. Needs a docs/experiments.md row (inconclusive) at apply time.
Retry if: eval:suite --tier fast shows a gain on LongMemEval or coding-life dev with no loss on PrecisionMemBench.
Holds. memoryToObservation hard-codes type "decision"; compact smart-search and mem::search compact drop the saved type (fact, bug, rule...).
Repro: test/smart-search.test.ts "compact results keep a saved Memory's own type..." fails 1/1 on base (memoryType undefined).
Fix (applied in worktree): additive optional memoryType on CompressedObservation and CompactSearchResult, set in memoryToObservation, copied into the two compact builders (smart-search.ts, search.ts). Full/narrative/expanded shapes carry the whole observation so they get it from memoryToObservation. Index unchanged, no migration. Typecheck clean.
Reach: all 40 latest Memories are shown as "decision" today; 8 facts, 19 bugs etc. indistinguishable.
Class 2: no ranking or injected-text change; adds a field to the REST/MCP response (additive public shape, so mention in PR). No eval:suite needed. Does not touch recall rendering (the memory_recall full/narrative render code was not changed; upstream's budget-clipped preview handling not reproduced).
Holds in code (graph-retrieval.ts scoreEntityMatches and scoreExpansion). Fork already sorts start nodes by id, so it is deterministic but still first-wins: an own observation (1.0) is buried at a 2-hop score if an earlier-id start node reaches it.
Repro: test/graph-retrieval.test.ts "scores an observation at its best path..." fails 1/1 on base (0.267 vs 1).
Fix (applied): keep best score per obsId (Map) in both functions; given obsIds still excluded in scoreExpansion.
Retry if (if deferred): graph extraction is turned on by default.
rohitg00#1685 procedural tier never yields a procedure
Defect confirmed on Operator data, fix NOT APPLIED. consolidation-pipeline.ts procedural tier uses m.sessionIds.length || 1 >= 2; Memories saved by memory_save/remember carry no sessionIds (Operator: 0 of 42), so frequency is always 1. Operator has 2 latest pattern Memories, 0 procedural rows ever.
Why not fixed: no stored field carries a recurrence count (sourceObservationIds also empty), so any fix invents a signal. And nothing reads KV.procedural for Recall (only skill-extract, diagnostics, mesh, export); a working tier would spend LLM calls and disk on rows the Agent never sees, against "least record that restates". Open question for the Operator: delete the procedural tier (and its diagnostics line) rather than fix it. Deleting is a behavior/API removal so not done here.
Repro: not written as failing (the right fix is undecided). Observable now: POST consolidate-pipeline tier procedural returns skipped "fewer than 2 recurring patterns" with 2 patterns.
Retry if: a Recall or Injection surface starts reading procedural memories, or Memories gain a recurrence counter.
rohitg00#1545 consolidation provenance / version heads (mem::consolidate)
Three sub-claims hold in consolidate.ts, applied in worktree, all failing on base:
Concurrent runs publish duplicate Memories: test "keeps one latest Memory when two runs overlap" fails 1/1 (2 latest). Fix: serialize mem::consolidate with withKeyedLock("mem::consolidate").
Predecessor demoted before successor stored, so a failed write hides the Memory: test "keeps the previous version latest when writing its successor fails" fails 1/1 (no latest). Fix: store successor first, then demote. Residual: a failed demote leaves two latest briefly (upstream's rollback/repair not reproduced; mem::remember supersession handles heads).
Failed model attempts do not count toward the 10-call budget: test "counts a failed model call..." fails 1/1 (12 calls). Fix: increment before the call.
Not applied: source-mutation rechecks, lost-acknowledgement confirmation and version-chain repair (distributed-write machinery; single Operator, one process; no evidence of occurrence).
Reach: low. Operator store has no consolidation-written Memories (all 42 have empty sessionIds; consolidate writes them). Class 2: no for 1 and 3 on Recall ranking; 2 changes storage write order only. No eval:suite.
Retry if (unapplied bits): an audit shows a Memory chain with 2 latest or an orphaned successor.
Bar (tmp/verify/467-verify.log, run with the prefix line): typecheck 0 errors; npm test 3,122 passed, 1 skipped; eval:gate FAILED on prompt-submit precision only; eval:gate:rules pass; eval:suite exit 0, no metric worse. After reverting the prefix line: eval:gate and eval:gate:rules pass with every metric at baseline; the touched test files 67/67 pass; typecheck 0 errors. The applied set (memoryType field, graph best-path, consolidate lock/order/budget) does not change ranking, so the suite run above bounds it. Branch verify/467-recall-scoring (uncommitted, worktree tmp/wt/verify-467).
Activity
possiblyneal commentedon Nov 18, 2026
#467 verdict (worktree tmp/wt/verify-467, branch verify/467-recall-scoring, base fdc3d45)
Operator reach figures are from tmp/verify/live-copy.sqlite (read-only): 42 Memories (40 latest: bug 19, fact 8, workflow 7, preference 2, pattern 2, architecture 1, rule 1), 0 with sessionIds, 0 procedural rows, 0 relations.
rohitg00#1514 prefix inflation
if (matchingDocs?.has(obsId)) continue;in the prefix loop (src/state/search-index.ts, 1 line). Targeted search-index tests pass.rohitg00#1646 original Memory type lost in search
memoryTypeon CompressedObservation and CompactSearchResult, set in memoryToObservation, copied into the two compact builders (smart-search.ts, search.ts). Full/narrative/expanded shapes carry the whole observation so they get it from memoryToObservation. Index unchanged, no migration. Typecheck clean.rohitg00#1558 graph first-come visitedObs
rohitg00#1685 procedural tier never yields a procedure
m.sessionIds.length || 1 >= 2; Memories saved by memory_save/remember carry no sessionIds (Operator: 0 of 42), so frequency is always 1. Operator has 2 latest pattern Memories, 0 procedural rows ever.rohitg00#1545 consolidation provenance / version heads (mem::consolidate)
Bar (tmp/verify/467-verify.log, run with the prefix line): typecheck 0 errors; npm test 3,122 passed, 1 skipped; eval:gate FAILED on prompt-submit precision only; eval:gate:rules pass; eval:suite exit 0, no metric worse. After reverting the prefix line: eval:gate and eval:gate:rules pass with every metric at baseline; the touched test files 67/67 pass; typecheck 0 errors. The applied set (memoryType field, graph best-path, consolidate lock/order/budget) does not change ranking, so the suite run above bounds it. Branch
verify/467-recall-scoring(uncommitted, worktree tmp/wt/verify-467).