fix(hook): claim no similarity when raw_score is null - #174
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The change is narrowly scoped, matches existing describe_match semantics, and is backed by targeted regression tests.
Pull request overview
This PR fixes score presentation in the Claude Code UserPromptSubmit recall hook (and the embedded copy written by palinode init) so that a fused rank (score) is no longer rendered as similarity when raw_score is explicitly present but null (BM25-only hit).
Changes:
- Update the hook jq to distinguish three cases:
raw_scoreabsent → “rank …”,raw_score: null→ “keyword match, rank …”,raw_score: number→ “N% match”. - Apply the same jq change to both
examples/hooks/palinode-user-prompt-submit.shand the embeddedUSER_PROMPT_SUBMIT_HOOK_SCRIPTinpalinode/cli/init.py. - Add/extend tests to cover null-vs-absent behavior and to pin hook rendering to
palinode.core.scoring.describe_match.
File summaries
| File | Description |
|---|---|
| tests/test_user_prompt_submit_hook.py | Adds regression tests for raw_score null vs absent and enforces hook output parity with describe_match. |
| palinode/cli/init.py | Updates the embedded recall hook jq to avoid treating fused rank as similarity in the BM25-only case. |
| examples/hooks/palinode-user-prompt-submit.sh | Updates the canonical recall hook jq with the same three-case score description logic. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
This is the one I'd point at if someone asked what a good contribution looks like here. The equivalence test is the reason — importing
One thing worth knowing, not a change request. The two implementations round differently at exact boundaries: jq's Copilot left a note on the keyword-only test — that Merging. That's six, and every one of them has left a guard behind. |
The Claude Code hook and the copy
palinode initwrites out, the last two surfaces from #150.(.raw_score // .score // 0)falls through on null, so a BM25-only hit rendered the fused rank as a percentage.ranker.py:301setsraw_score=Nonefor those explicitly, so the top hit of any query, including an irrelevant one, rendered as a near-certain match.The jq now runs the same three cases as
describe_match:(42%)(42% match)(100%)(keyword match, rank 1.00)(100%)(rank 1.00)Absent and null stay separate. A pre-0.12 server never sent the field, so the arm is unknown and the rank is all there is to say. A present null is a claim the ranker made and has no similarity behind it.
Both files carry the same jq, since
palinode initwrites the hook out andtest_embedded_copy_matches_canonical_exampleholds them byte-identical.Three tests, all failing on the unfixed hook. The last one renders five hits through the real script and asserts each line matches what
describe_matchreturns for the same dict, so the jq and the Python cannot drift apart quietly.Not touching
plugin/index.tsorplugins/core/src/index.ts.