Skip to content

refactor(semantic): extract the retrieval pipeline from search_notes - #125

Open
AlexMost wants to merge 7 commits into
mainfrom
extract-retrieval-pipeline-1
Open

AlexMost wants to merge 7 commits into
mainfrom
extract-retrieval-pipeline-1

Conversation

@AlexMost

Copy link
Copy Markdown
Owner

Extracts the leg-assembly orchestration out of search_notes into src/modules/semantic/retrieval-pipeline.ts, with the quick/deep effort profile single-sourced in effort-profiles.ts and the advertised description numbers derived from it (byte-identical text).

Behavior-preserving. No retrieval number changes value, no search_notes contract change, and zero edits to any pre-existing test file — that last one is the regression proof: the existing gate-routed search_notes suites pass unchanged against the rewired handler.

What moved:

  • effort-profiles.ts — the quick/deep profile (semantic pool + threshold, lexical note cap, expansion switch/limit, merged cap) plus LEXICAL_PER_NOTE_CAP, FALLBACK_THRESHOLD, DEFAULT_EXPANSION_FLOOR. retrieval-policy.ts's MODE_DEFAULTS now derives from it.
  • retrieval-pipeline.ts — source narrowing, both legs, the semantic degrade, existence filtering (seeds and their expansion targets), fusion and the cap slice. executeRetrieval's four caller obligations are now module-internal, and the pipeline is its only src/ caller.
  • search_notes keeps only MCP shaping: it decides whether the semantic leg runs and how to word the outcome (semantic_status, DEGRADED_NOTES, query_stats, found_in). runSearchForEntry loses 149 lines.
  • The description's effort line and the threshold/expansion_floor prose interpolate the profile constants, pinned by a new test against both the source (interpolated, not re-literalized) and the advertised schema text (the right numbers in the right slots).

New direct tests cover fusion ordering, cap-slice truncation, leg-pool overflow, existence filtering, semantic degrade and leg skipping — with no temp vault, registry stub, or embedder mock.

Part 1 of 3 for the extract-retrieval-pipeline change (next: eval rewired onto the module + drift-pin removal; then the docs sweep). The intermediate state is safe: eval/pipelines.ts and test/eval/production-fusion-pin.test.ts still hold their duplicate orchestration, and since production behavior is unchanged the pin keeps passing — acting as one more independent check on this refactor until PR 2 retires it.

Gates: npm test (1366 passing) · npm run lint · npm run typecheck · npm run build — all green.

Refs #109

🤖 Generated with Claude Code

AlexMost and others added 7 commits August 31, 2026 23:26
Refs #109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Moved the full empirical justification from retrieval-policy.ts (lines 16-19,
orphaned after the constant extraction) to effort-profiles.ts as a doc comment
above DEFAULT_EXPANSION_FLOOR, and deleted the stray remnant. The rationale
explains the empirical basis (0.89–0.985 scale) and behavior-preserving value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refs #109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eval pipeline

Refs #109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… constants

Refs #109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin the advertisement-derivation test to registerTool(...).spec.description
(the surface an MCP client actually sees) instead of the builder's raw
.description, and extend the source-token assertions to cover every AXES
interpolation so a revert to hardcoded literals fails the test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…schema

Fixes final-review findings on PR 1 of extract-retrieval-pipeline: the
threshold/expansion_floor description numbers were pinned only against
search-notes.ts's source text, so a swapped interpolation slot stayed
green. Assert against the tool's JSON-Schema-converted inputSchema
instead, and drop the vacuous toBeTruthy()/duplicate-constant
assertions it was riding alongside. Also restores the "limit is not
forwarded into executeRetrieval" invariant as a comment (lost when the
old MERGED_CAP block was deleted) and corrects effort-profiles.ts's
header, which claimed eval/ already reads these constants when it
still restates them as literals.

Refs #109

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant