diff --git a/CLAUDE.md b/CLAUDE.md index da792d348c..21e53a0d1c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -364,6 +364,12 @@ closest one instead of writing a row, as an exact re-save does: it takes the new context and the union of tags, keeps its id, and is audited `lesson_strengthen` with `reason: "re-discovery"` and `match: "restated" | "exact"` (#378). +### Lesson recall candidates +`searchLessons` and `mem::lesson-recall` search the whole Lesson index before filtering by project, +live status and (for recall) minimum confidence. Global Lessons remain eligible for a project; +without a project all projects remain eligible. Recall then applies its existing candidate cap and +confidence/recency ranking; both paths retain their near-twin cap. See #438. + ### Lesson retention `mem::lesson-decay-sweep` lowers a Lesson's confidence by its `decayRate` per active week of Project Time down to a floor of 0.05, reinforced or not, and never deletes a live Lesson for diff --git a/docs/benchmarks/2026-11-20-scoped-lesson-candidates.md b/docs/benchmarks/2026-11-20-scoped-lesson-candidates.md new file mode 100644 index 0000000000..fae28d14d6 --- /dev/null +++ b/docs/benchmarks/2026-11-20-scoped-lesson-candidates.md @@ -0,0 +1,31 @@ +# 2026-11-20 — scoped Lesson candidates + +## Hypothesis and change + +Other projects' higher BM25 hits can exhaust the candidate cut before an eligible Lesson is considered. Search the full Lesson index, filter for project, live status and minimum confidence, then apply the existing candidate cut and near-twin ranking. Global Lessons remain visible to scoped searches; unscoped searches span projects. The full-index ranking can cost more on large stores. + +## Checks on pinned base `d309ec74` + +- The focused Lesson tests pass: 57 tests in two files, including 250 higher-scoring other-project hits on both paths, global and unscoped visibility, and tombstone and confidence filtering. Typecheck passes. +- The BM25 coding-life gate passes on instance 91, all quality metrics equal to baseline. Prompt-submit recall is 0.907, precision 0.889, no-answer clean 0.833, p95 7 ms. The pinned-rules gate also passes, session-start p95 3 ms and pre-compact p95 2 ms. +- The BM25 gate runs without the prompt Lesson reranker and cannot measure Lesson Injection quality. Full suite, broker-backed and holdout evals were not run here. + +## 2026-11-21 — comparison against `main` + +Both arms used the same read-only store copy (415 Lessons) and 5,591 paired project/prompt inputs with the same project resolver. The control was `main` at `929ccc12`; the candidate was the combined local tree for #479, #439, #436 and #438 (`0e1cbcd0`), not #438 in isolation. + +| Metric | Main | Candidate | +| --- | ---: | ---: | +| Lesson Recall results returned | 17,253 | 17,790 | +| Mean Lesson Recall call | 0.602 ms | 0.843 ms | +| Mean prompt-submit BM25 Lesson selection | ≈0.694 ms | ≈0.694 ms | + +Prompt-submit BM25 Lesson top-1 was unchanged on all 5,591 prompts. Lesson Recall added 866 same-project results across 388 prompts and displaced 329 live same-project results across 225 prompts. All displaced results were previously at ranks 4–10, with 271 at ranks 9–10. No Global or other-project result entered or left a list. The combined build passed 3,183 tests (1 skipped), typecheck and both BM25 Recall gates. + +This measures which Lessons were returned, not their usefulness or end-to-end reranked Injection quality. The change remains local. + +## Earlier verdict on `fdc3d459` + +The [#438 verdict](https://github.com/possiblyneal/agentmemory-sqlite/issues/438#issuecomment-6984901342), on `fdc3d459`, reported the original three regression tests failing on base and passing with the proposed fix, typecheck, 3,119 tests and both gates passing, and all 15 fast-suite metrics unchanged. On a read-only copy of the Operator's store (415 Lessons, 5,588 replayed prompts), the prompt Lesson top-1 was unchanged; recall gained 902 same-project Lessons on 211 prompts and displaced 335 on 229 prompts, all from ranks 4–10. No Global or other-project Lesson entered or left a list. This earlier measurement did not include the additional tombstone filter in this implementation. + +The earlier [prompt-submit Lesson measurement](./2026-11-11-prompt-submit-lessons.md) explains why BM25-only gates do not measure injected Lessons. No live store was accessed for this implementation. diff --git a/docs/experiments.md b/docs/experiments.md index d581b09e01..e59d17fcfe 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -34,6 +34,7 @@ A hypothesis comes from one of these, named in the Source column: | Date | Hypothesis | Source | Eval | Result | Verdict | Retry if | Evidence | |---|---|---|---|---|---|---|---| | 2026-11-21 | Counting before listing reduces Observation write latency below the Session cap (#479) | incident | Five paired capture rounds; `main` vs four-patch tree | `observe` p50 42 → 3 ms, p95 92 → 11 ms; median round `/livez` max 62 → 28 ms; count-only effect on this base not isolated | inconclusive | attribution on current `main` is needed: compare the count-only patch against the same base and seed | [2026-11-21 Session-cap count](./benchmarks/2026-11-21-session-cap-count.md) | +| 2026-11-20 | Filtering Lesson hits before the candidate cut recovers project Lessons starved by other projects (#438) | incident | 250-higher-hit regression, BM25 gates; `main` vs four-patch tree on 5,591 paired prompts; earlier #453 replay and fast suite | Regression and gates pass; combined tree adds 866 and displaces 329 same-project results (displaced ranks 4–10); BM25 Lesson top-1 unchanged | finding | — | [2026-11-20 scoped Lesson candidates](./benchmarks/2026-11-20-scoped-lesson-candidates.md) | | 2026-10-01 | Baseline: Recall loses content on the Injection paths, not on search | — | coding-agent-life-v2, all questions: search 20, pre-tool-use 15, session-start 3 | search recall 1.000 but 0/4 no-answer clean; pre-tool-use recall 0.667, leaking across projects on shared filenames; every path beats `random` | finding | — | [2026-10-01 coding-life](./benchmarks/2026-10-01-coding-agent-life-v2.md) | | 2026-10-03 | Filtering the prompt-submit Injection by a BM25 floor (≥ 5 and ≥ 0.5× best) raises precision over the unfiltered top K | — | coding-agent-life-v2 prompt-submit, n=15 | precision 0.303 → 0.822, no-answer clean 0.333 → 0.833, recall 0.963 → 0.907; mean Injection 184 chars | kept | — | [2026-10-03 coding-life](./benchmarks/2026-10-03-coding-agent-life-v2.md), #106 | | 2026-10-05 | A smart-search relevance floor (BM25 ≥ 3.5 and ≥ 0.5× best, or cosine ≥ 0.3 with a BM25 match) cuts noise without losing recall | — | coding-agent-life-v2 search, n=20 | bm25 precision 0.195 → 0.628, no-answer clean 0 → 0.250, recall 0.969 unchanged; embedding adapter precision 0.190 → 0.601, recall 1.000 | kept | — | [2026-10-05 coding-life](./benchmarks/2026-10-05-coding-agent-life-v2.md) | diff --git a/src/functions/lessons.ts b/src/functions/lessons.ts index 579d1316ba..3c389c6a44 100644 --- a/src/functions/lessons.ts +++ b/src/functions/lessons.ts @@ -58,7 +58,7 @@ async function ensureLessonIndex(kv: StateKV): Promise { export async function searchLessons(kv: StateKV, query: string, project: string | undefined, limit: number): Promise { const idx = await ensureLessonIndex(kv); const scored = idx - .search(query, Math.max(limit * 10, 100)) + .search(query, idx.size) .map((h) => ({ lesson: lessonRecords.get(h.obsId), score: h.score })) .filter((s): s is { lesson: Lesson; score: number } => !!s.lesson && !s.lesson.deleted && (!project || !s.lesson.project || s.lesson.project === project)) .slice(0, limit); @@ -237,17 +237,17 @@ export function registerLessonsFunctions(sdk: ISdk, kv: StateKV): void { const limit = data.limit ?? 10; const idx = await ensureLessonIndex(kv); + const hits = idx.search(data.query, idx.size); + const maxHit = hits.length > 0 ? hits[0].score : 0; const filtering = !!data.project || minConfidence > 0; - const fetchLimit = filtering + const candidateLimit = filtering ? Math.max(limit * 10, 100) : Math.max(limit * 5, 50); - const hits = idx.search(data.query, fetchLimit); - const maxHit = hits.length > 0 ? hits[0].score : 0; const scored: Array<{ lesson: Lesson; score: number }> = []; - for (let i = 0; i < hits.length; i++) { + for (let i = 0; i < hits.length && scored.length < candidateLimit; i++) { const l = lessonRecords.get(hits[i].obsId); - if (!l || l.confidence < minConfidence) continue; + if (!l || l.deleted || l.confidence < minConfidence) continue; if (data.project && l.project && l.project !== data.project) continue; const relevance = maxHit > 0 ? hits[i].score / maxHit : 0; diff --git a/test/lesson-recall-starvation.test.ts b/test/lesson-recall-starvation.test.ts new file mode 100644 index 0000000000..6ca44689a4 --- /dev/null +++ b/test/lesson-recall-starvation.test.ts @@ -0,0 +1,107 @@ +import { describe, it, expect, beforeEach, vi } from "vitest"; + +vi.mock("../src/logger.js", () => ({ + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +import { registerLessonsFunctions, resetLessonIndex, searchLessons } from "../src/functions/lessons.js"; +import type { Lesson } from "../src/types.js"; + +function mockKV() { + const store = new Map>(); + return { + get: async (scope: string, key: string): Promise => (store.get(scope)?.get(key) as T) ?? null, + set: async (scope: string, key: string, data: T): Promise => { + if (!store.has(scope)) store.set(scope, new Map()); + store.get(scope)!.set(key, data); + return data; + }, + delete: async (scope: string, key: string): Promise => { + store.get(scope)?.delete(key); + }, + list: async (scope: string): Promise => Array.from(store.get(scope)?.values() ?? []) as T[], + }; +} + +function mockSdk() { + const functions = new Map(); + return { + registerFunction: (id: string, handler: Function) => functions.set(id, handler), + registerTrigger: () => {}, + trigger: async (id: string, payload: unknown) => functions.get(id)!(payload), + }; +} + +function lesson(id: string, content: string, project: string): Lesson { + const now = new Date().toISOString(); + return { + id, content, context: "", confidence: 0.8, reinforcements: 0, source: "manual", + sourceIds: [], project, tags: [], createdAt: now, updatedAt: now, decayRate: 0.05, + }; +} + +const TARGET = "lsn_target"; +const QUERY = "deploy pipeline"; + +describe("Lesson recall filters by project before the candidate cut (#438)", () => { + let sdk: ReturnType; + let kv: ReturnType; + + beforeEach(async () => { + sdk = mockSdk(); + kv = mockKV(); + resetLessonIndex(); + registerLessonsFunctions(sdk as never, kv as never); + for (let i = 0; i < 250; i++) { + await kv.set("mem:lessons", `lsn_other_${i}`, lesson(`lsn_other_${i}`, `deploy pipeline deploy pipeline marker${i}`, "/other")); + } + await kv.set( + "mem:lessons", + TARGET, + lesson(TARGET, "When the deploy pipeline stalls on the staging host, check the runner's disk quota, the lockfile age and the pending migration queue before restarting anything", "/app"), + ); + }); + + it("searchLessons returns this project's matching Lesson", async () => { + const found = await searchLessons(kv as never, QUERY, "/app", 20); + expect(found.map((l) => l.id)).toContain(TARGET); + }); + + it("mem::lesson-recall returns this project's matching Lesson", async () => { + const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app" })) as { + lessons: Lesson[]; + }; + expect(result.lessons.map((l) => l.id)).toContain(TARGET); + }); + + it("mem::lesson-recall with a confidence floor still reaches past other projects' hits", async () => { + const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app", minConfidence: 0.5 })) as { + lessons: Lesson[]; + }; + expect(result.lessons.map((l) => l.id)).toContain(TARGET); + }); + it("keeps global Lessons visible to scoped and unscoped searches", async () => { + const global = { ...lesson("lsn_global", "deploy pipeline global guidance", "/app"), project: undefined }; + await kv.set("mem:lessons", global.id, global); + for (const project of ["/app", undefined]) { + const found = await searchLessons(kv as never, "deploy pipeline global guidance", project, 10); + expect(found.map((l) => l.id)).toContain(global.id); + const result = (await sdk.trigger("mem::lesson-recall", { query: "deploy pipeline global guidance", project })) as { lessons: Lesson[] }; + expect(result.lessons.map((l) => l.id)).toContain(global.id); + } + }); + + it("skips tombstones and low-confidence hits before the candidate cut", async () => { + for (let i = 0; i < 150; i++) { + const id = `lsn_ineligible_${i}`; + await kv.set("mem:lessons", id, { + ...lesson(id, `deploy pipeline deploy pipeline ineligible${i}`, "/app"), + ...(i < 75 ? { deleted: true } : { confidence: 0.1 }), + }); + } + const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app", minConfidence: 0.5 })) as { lessons: Lesson[] }; + expect(result.lessons.map((l) => l.id)).toContain(TARGET); + expect(result.lessons.every((l) => !l.deleted && l.confidence >= 0.5)).toBe(true); + }); + +});