Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,6 +364,12 @@ closest one instead of writing a row, as an exact re-save does: it takes the new
context and the union of tags, keeps its id, and is audited `lesson_strengthen` with
`reason: "re-discovery"` and `match: "restated" | "exact"` (#378).

### Lesson recall candidates
`searchLessons` and `mem::lesson-recall` search the whole Lesson index before filtering by project,
live status and (for recall) minimum confidence. Global Lessons remain eligible for a project;
without a project all projects remain eligible. Recall then applies its existing candidate cap and
confidence/recency ranking; both paths retain their near-twin cap. See #438.

### Lesson retention
`mem::lesson-decay-sweep` lowers a Lesson's confidence by its `decayRate` per active week of
Project Time down to a floor of 0.05, reinforced or not, and never deletes a live Lesson for
Expand Down
31 changes: 31 additions & 0 deletions docs/benchmarks/2026-11-20-scoped-lesson-candidates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# 2026-11-20 — scoped Lesson candidates

## Hypothesis and change

Other projects' higher BM25 hits can exhaust the candidate cut before an eligible Lesson is considered. Search the full Lesson index, filter for project, live status and minimum confidence, then apply the existing candidate cut and near-twin ranking. Global Lessons remain visible to scoped searches; unscoped searches span projects. The full-index ranking can cost more on large stores.

## Checks on pinned base `d309ec74`

- The focused Lesson tests pass: 57 tests in two files, including 250 higher-scoring other-project hits on both paths, global and unscoped visibility, and tombstone and confidence filtering. Typecheck passes.
- The BM25 coding-life gate passes on instance 91, all quality metrics equal to baseline. Prompt-submit recall is 0.907, precision 0.889, no-answer clean 0.833, p95 7 ms. The pinned-rules gate also passes, session-start p95 3 ms and pre-compact p95 2 ms.
- The BM25 gate runs without the prompt Lesson reranker and cannot measure Lesson Injection quality. Full suite, broker-backed and holdout evals were not run here.

## 2026-11-21 — comparison against `main`

Both arms used the same read-only store copy (415 Lessons) and 5,591 paired project/prompt inputs with the same project resolver. The control was `main` at `929ccc12`; the candidate was the combined local tree for #479, #439, #436 and #438 (`0e1cbcd0`), not #438 in isolation.

| Metric | Main | Candidate |
| --- | ---: | ---: |
| Lesson Recall results returned | 17,253 | 17,790 |
| Mean Lesson Recall call | 0.602 ms | 0.843 ms |
| Mean prompt-submit BM25 Lesson selection | ≈0.694 ms | ≈0.694 ms |

Prompt-submit BM25 Lesson top-1 was unchanged on all 5,591 prompts. Lesson Recall added 866 same-project results across 388 prompts and displaced 329 live same-project results across 225 prompts. All displaced results were previously at ranks 4–10, with 271 at ranks 9–10. No Global or other-project result entered or left a list. The combined build passed 3,183 tests (1 skipped), typecheck and both BM25 Recall gates.

This measures which Lessons were returned, not their usefulness or end-to-end reranked Injection quality. The change remains local.

## Earlier verdict on `fdc3d459`

The [#438 verdict](https://github.com/possiblyneal/agentmemory-sqlite/issues/438#issuecomment-6984901342), on `fdc3d459`, reported the original three regression tests failing on base and passing with the proposed fix, typecheck, 3,119 tests and both gates passing, and all 15 fast-suite metrics unchanged. On a read-only copy of the Operator's store (415 Lessons, 5,588 replayed prompts), the prompt Lesson top-1 was unchanged; recall gained 902 same-project Lessons on 211 prompts and displaced 335 on 229 prompts, all from ranks 4–10. No Global or other-project Lesson entered or left a list. This earlier measurement did not include the additional tombstone filter in this implementation.

The earlier [prompt-submit Lesson measurement](./2026-11-11-prompt-submit-lessons.md) explains why BM25-only gates do not measure injected Lessons. No live store was accessed for this implementation.
1 change: 1 addition & 0 deletions docs/experiments.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ A hypothesis comes from one of these, named in the Source column:
| Date | Hypothesis | Source | Eval | Result | Verdict | Retry if | Evidence |
|---|---|---|---|---|---|---|---|
| 2026-11-21 | Counting before listing reduces Observation write latency below the Session cap (#479) | incident | Five paired capture rounds; `main` vs four-patch tree | `observe` p50 42 → 3 ms, p95 92 → 11 ms; median round `/livez` max 62 → 28 ms; count-only effect on this base not isolated | inconclusive | attribution on current `main` is needed: compare the count-only patch against the same base and seed | [2026-11-21 Session-cap count](./benchmarks/2026-11-21-session-cap-count.md) |
| 2026-11-20 | Filtering Lesson hits before the candidate cut recovers project Lessons starved by other projects (#438) | incident | 250-higher-hit regression, BM25 gates; `main` vs four-patch tree on 5,591 paired prompts; earlier #453 replay and fast suite | Regression and gates pass; combined tree adds 866 and displaces 329 same-project results (displaced ranks 4–10); BM25 Lesson top-1 unchanged | finding | — | [2026-11-20 scoped Lesson candidates](./benchmarks/2026-11-20-scoped-lesson-candidates.md) |
| 2026-10-01 | Baseline: Recall loses content on the Injection paths, not on search | — | coding-agent-life-v2, all questions: search 20, pre-tool-use 15, session-start 3 | search recall 1.000 but 0/4 no-answer clean; pre-tool-use recall 0.667, leaking across projects on shared filenames; every path beats `random` | finding | — | [2026-10-01 coding-life](./benchmarks/2026-10-01-coding-agent-life-v2.md) |
| 2026-10-03 | Filtering the prompt-submit Injection by a BM25 floor (≥ 5 and ≥ 0.5× best) raises precision over the unfiltered top K | — | coding-agent-life-v2 prompt-submit, n=15 | precision 0.303 → 0.822, no-answer clean 0.333 → 0.833, recall 0.963 → 0.907; mean Injection 184 chars | kept | — | [2026-10-03 coding-life](./benchmarks/2026-10-03-coding-agent-life-v2.md), #106 |
| 2026-10-05 | A smart-search relevance floor (BM25 ≥ 3.5 and ≥ 0.5× best, or cosine ≥ 0.3 with a BM25 match) cuts noise without losing recall | — | coding-agent-life-v2 search, n=20 | bm25 precision 0.195 → 0.628, no-answer clean 0 → 0.250, recall 0.969 unchanged; embedding adapter precision 0.190 → 0.601, recall 1.000 | kept | — | [2026-10-05 coding-life](./benchmarks/2026-10-05-coding-agent-life-v2.md) |
Expand Down
12 changes: 6 additions & 6 deletions src/functions/lessons.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ async function ensureLessonIndex(kv: StateKV): Promise<SearchIndex> {
export async function searchLessons(kv: StateKV, query: string, project: string | undefined, limit: number): Promise<Lesson[]> {
const idx = await ensureLessonIndex(kv);
const scored = idx
.search(query, Math.max(limit * 10, 100))
.search(query, idx.size)
.map((h) => ({ lesson: lessonRecords.get(h.obsId), score: h.score }))
.filter((s): s is { lesson: Lesson; score: number } => !!s.lesson && !s.lesson.deleted && (!project || !s.lesson.project || s.lesson.project === project))
.slice(0, limit);
Expand Down Expand Up @@ -237,17 +237,17 @@ export function registerLessonsFunctions(sdk: ISdk, kv: StateKV): void {
const limit = data.limit ?? 10;

const idx = await ensureLessonIndex(kv);
const hits = idx.search(data.query, idx.size);
const maxHit = hits.length > 0 ? hits[0].score : 0;
const filtering = !!data.project || minConfidence > 0;
const fetchLimit = filtering
const candidateLimit = filtering
? Math.max(limit * 10, 100)
: Math.max(limit * 5, 50);
const hits = idx.search(data.query, fetchLimit);
const maxHit = hits.length > 0 ? hits[0].score : 0;

const scored: Array<{ lesson: Lesson; score: number }> = [];
for (let i = 0; i < hits.length; i++) {
for (let i = 0; i < hits.length && scored.length < candidateLimit; i++) {
const l = lessonRecords.get(hits[i].obsId);
if (!l || l.confidence < minConfidence) continue;
if (!l || l.deleted || l.confidence < minConfidence) continue;
if (data.project && l.project && l.project !== data.project) continue;

const relevance = maxHit > 0 ? hits[i].score / maxHit : 0;
Expand Down
107 changes: 107 additions & 0 deletions test/lesson-recall-starvation.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
import { describe, it, expect, beforeEach, vi } from "vitest";

vi.mock("../src/logger.js", () => ({
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },
}));

import { registerLessonsFunctions, resetLessonIndex, searchLessons } from "../src/functions/lessons.js";
import type { Lesson } from "../src/types.js";

function mockKV() {
const store = new Map<string, Map<string, unknown>>();
return {
get: async <T>(scope: string, key: string): Promise<T | null> => (store.get(scope)?.get(key) as T) ?? null,
set: async <T>(scope: string, key: string, data: T): Promise<T> => {
if (!store.has(scope)) store.set(scope, new Map());
store.get(scope)!.set(key, data);
return data;
},
delete: async (scope: string, key: string): Promise<void> => {
store.get(scope)?.delete(key);
},
list: async <T>(scope: string): Promise<T[]> => Array.from(store.get(scope)?.values() ?? []) as T[],
};
}

function mockSdk() {
const functions = new Map<string, Function>();
return {
registerFunction: (id: string, handler: Function) => functions.set(id, handler),
registerTrigger: () => {},
trigger: async (id: string, payload: unknown) => functions.get(id)!(payload),
};
}

function lesson(id: string, content: string, project: string): Lesson {
const now = new Date().toISOString();
return {
id, content, context: "", confidence: 0.8, reinforcements: 0, source: "manual",
sourceIds: [], project, tags: [], createdAt: now, updatedAt: now, decayRate: 0.05,
};
}

const TARGET = "lsn_target";
const QUERY = "deploy pipeline";

describe("Lesson recall filters by project before the candidate cut (#438)", () => {
let sdk: ReturnType<typeof mockSdk>;
let kv: ReturnType<typeof mockKV>;

beforeEach(async () => {
sdk = mockSdk();
kv = mockKV();
resetLessonIndex();
registerLessonsFunctions(sdk as never, kv as never);
for (let i = 0; i < 250; i++) {
await kv.set("mem:lessons", `lsn_other_${i}`, lesson(`lsn_other_${i}`, `deploy pipeline deploy pipeline marker${i}`, "/other"));
}
await kv.set(
"mem:lessons",
TARGET,
lesson(TARGET, "When the deploy pipeline stalls on the staging host, check the runner's disk quota, the lockfile age and the pending migration queue before restarting anything", "/app"),
);
});

it("searchLessons returns this project's matching Lesson", async () => {
const found = await searchLessons(kv as never, QUERY, "/app", 20);
expect(found.map((l) => l.id)).toContain(TARGET);
});

it("mem::lesson-recall returns this project's matching Lesson", async () => {
const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app" })) as {
lessons: Lesson[];
};
expect(result.lessons.map((l) => l.id)).toContain(TARGET);
});

it("mem::lesson-recall with a confidence floor still reaches past other projects' hits", async () => {
const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app", minConfidence: 0.5 })) as {
lessons: Lesson[];
};
expect(result.lessons.map((l) => l.id)).toContain(TARGET);
});
it("keeps global Lessons visible to scoped and unscoped searches", async () => {
const global = { ...lesson("lsn_global", "deploy pipeline global guidance", "/app"), project: undefined };
await kv.set("mem:lessons", global.id, global);
for (const project of ["/app", undefined]) {
const found = await searchLessons(kv as never, "deploy pipeline global guidance", project, 10);
expect(found.map((l) => l.id)).toContain(global.id);
const result = (await sdk.trigger("mem::lesson-recall", { query: "deploy pipeline global guidance", project })) as { lessons: Lesson[] };
expect(result.lessons.map((l) => l.id)).toContain(global.id);
}
});

it("skips tombstones and low-confidence hits before the candidate cut", async () => {
for (let i = 0; i < 150; i++) {
const id = `lsn_ineligible_${i}`;
await kv.set("mem:lessons", id, {
...lesson(id, `deploy pipeline deploy pipeline ineligible${i}`, "/app"),
...(i < 75 ? { deleted: true } : { confidence: 0.1 }),
});
}
const result = (await sdk.trigger("mem::lesson-recall", { query: QUERY, project: "/app", minConfidence: 0.5 })) as { lessons: Lesson[] };
expect(result.lessons.map((l) => l.id)).toContain(TARGET);
expect(result.lessons.every((l) => !l.deleted && l.confidence >= 0.5)).toBe(true);
});

});