Skip to content

Lesson recall cuts candidates before the project filter, starving this project's Lessons #438

Open

Description

@possiblyneal

Problem

Lesson recall takes the top max(limit*10, 100) BM25 hits across all projects, then filters by project, confidence and deleted. In a store where other projects' Lessons dominate a query, this project's matching Lesson ranks below the cut and is never returned. This hits both memory_lesson_recall and searchLessons(), which picks prompt submit's one Lesson candidate — so a relevant Lesson can be starved out of Injection.

Evidence

  • src/functions/lessons.ts:58-64 (searchLessons), :240-251 (mem::lesson-recall)

Upstream diagnosis: rohitg00#1721 (merged).

Acceptance criteria

  • The project / confidence / deleted filters apply before the candidate limit (or the search widens until limit eligible Lessons are found or hits run out)
  • Test: with 200 higher-scoring Lessons in another project, this project's matching Lesson is returned by both paths
  • npm run eval:gate passes; rerun eval:coding-life if prompt-submit numbers move

Activity

  1. added
    bugSomething isn't working
    ready-for-agentFully specified, ready for an AFK agent
    needs-triageMaintainer needs to evaluate this issue
    and removed
    ready-for-agentFully specified, ready for an AFK agent
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    Not verified on this tree yet. Held at needs-triage until Verify: lesson recall cuts candidates before the project filter gives a tested verdict: a failing reproduction and a fix that works here, or a reject.

  3. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    Verdict: apply

    Tested on verify/438-lesson-recall-starvation (from fdc3d459), per the bar in #449.

    Repro. test/lesson-recall-starvation.test.ts (3 tests) fails 3/3 on base. Lessons are cut to the top BM25 hits before the project and confidence filters, so a project whose Lessons rank below another project's get none back from searchLessons or mem::lesson-recall.

    Fix. Both paths search the whole Lesson index (idx.search(query, idx.size)) and filter before the cut. mem::lesson-recall keeps a candidate cap equal to the base fetch size (max(limit*10, 100) when filtering by project or confidence, else max(limit*5, 50)), so unfiltered calls score the same set as before.

    Checks. npm run typecheck 0 errors; npm test 3119 passed, 1 skipped; eval:gate and eval:gate:rules pass. eval:suite --tier fast vs base-fdc3d459: all 15 metrics Δ 0.000 (no eval exercises a starved project).

    Operator's data (read-only copy of the live store, 415 Lessons, 5,588 replayed prompts): the prompt-submit Lesson pick (top-1) is unchanged on every prompt. The mem::lesson-recall list returns 902 same-project Lessons the base cut never scored, on 211 prompts, and drops 335 same-project Lessons, on 229 prompts, all from ranks 4–10 (276 of them at ranks 9–10), displaced by better-scoring Lessons from the newly scored set. No Global or other-project Lesson enters or leaves a list. Total list length rises from 17,844 to 18,411.

  4. added
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions