Skip to content

fix(indexer): run the #2502 relevance guards on the search paths that grab - #2812

Merged
vavallee merged 3 commits into
mainfrom
fix/relevance-filter-single-implementation
Oct 2, 2026
Merged

vavallee merged 3 commits into
mainfrom
fix/relevance-filter-single-implementation

Conversation

@vavallee

@vavallee vavallee commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Summary

The relevance filter existed as two copies, filterRelevant and filterRelevantDebug. #2502 added its two guards (conflictingTitleAuthor and the title identity phrase gate) to filterRelevant only, and pinned them with tests that call filterRelevant directly.

Production never runs filterRelevant. The scheduler calls SearchBookWithOutcomes, which projects SearchBookWithDebug, and the interactive search calls SearchBookWithDebug. Both run filterRelevantDebug. filterRelevant is reached only by the scheduler's fallback for searchers that cannot report outcomes, which in practice is test stubs.

Since this was drafted, #2863 copied the author guard across by hand (shipped in v1.39.0), and #2921 added a volume number guard to both copies. The title identity gate was never copied, so on current main the production path still keeps:

Release For filterRelevant production path on main
The Power of Writing It Down by Ben Coes Power Down dropped kept
Beyond Order: 12 More Rules for Life 12 Rules for Life dropped kept
Rising of the Shield Hero Volume 16 ... Volume 17 (same spelling) dropped kept

The last row is the divergence #2921's author flagged.

Change

filterRelevantDetailed is now the single implementation. It returns the kept results plus a reason for every drop, and filterRelevant and filterRelevantDebug are thin wrappers over it, so a guard can't land in one path and miss the other again. #2863's author guard, #2921's volumeNumberAgrees, #2763's elision fallback and the identity gate all live in that one function. The drop record tells the interactive panel which check fired: a different author for this title, title words present but with other words or numbers among them, or the plain keyword miss.

Zero padding

Putting the identity gate on the production path at first dropped correct releases that pad a number ("Vol 07" for "Volume 7", "Book 07" for "Book 7"), because the gate compared numbers as literal words. Fixed in this PR (cc9d92b):

  • keywordPattern renders an all digit keyword as 0* plus the number without padding, so the phrase, keyword and in order matchers and ContainsPhrase (the identity gate) treat "7", "07" and "007" as one number, both directions. Word boundaries keep "7" from matching "17" or "70". volumeNumberAgrees already ignored padding.
  • conflictingTitleAuthor compares whole normalized strings, so foldVolumeMarkers became foldTitleTokens and strips padding too. Without that, "Dungeon Crawler Carl Book 07 - Some Other Writer" slips past the attribution check (a test pins it).

Remaining known gap

A digit and a spelled out number are still different words: "The Seven Habits of Highly Effective People" is dropped for "The 7 Habits of Highly Effective People", which production keeps on main. Pinned with a KNOWN GAP comment in TestFilterRelevantNumberPadding. No word to number equivalence here on purpose.

Comparison against main's production filter

271 distinct (title, author, release) triples: everything the indexer, api and scheduler suites feed the filter, plus 67 hand written realistic release names. Each was run through filterRelevantDebug on origin/main and on this branch. Differences:

Title Release main this PR
Power Down (x2) The Power of Writing It Down ... kept dropped wrong release
12 Rules for Life (x2) (Beyond Order:) 12 More Rules for Life ... kept dropped wrong release
Shield Hero Volume 17 ... Volume 16 kept dropped wrong volume
Shield Hero 7 ... 17 / ... 70 kept dropped wrong volume
Dungeon Crawler Carl Book 7 ... Book 17 / ... Book 017 kept dropped wrong book
Dungeon Crawler Carl Book 7 ... Book 07 - Some Other Writer kept dropped wrong author
Agent 007 Some Author - Agent 7 dropped kept right release
Fahrenheit 451 ... Fahrenheit 0451 dropped kept right release (contrived)
The 7 Habits of Highly Effective People ... The Seven Habits ... kept dropped right release, known gap

Every other triple, including the three padding rows, is decided the same way as on main.

Tests

  • fix(indexer): reject conflicting release title and author evidence #2502's two tables run through both wrappers.
  • TestRelevanceGuardsReachEveryProductionEntrypoint drives SearchBook, SearchBookWithDebug and SearchBookWithOutcomes against one fake indexer serving wrong releases beside the right ones; it now includes the same spelling Volume 16 case.
  • TestFilterRelevantDebugDropReasons pins the reason text for each check.
  • TestFilterRelevantNumberPadding (both wrappers) and TestContainsPhraseNumberPadding: padded and unpadded numbers match both ways; 17, 70, 017 and Volume 16 do not; a padded release naming another author is dropped; 7 versus Seven is the known gap.
  • Fail before for the padding commit: with release.go and title_identity.go at the previous commit, the three padding rows, the reverse and triple padded cases, the fix(indexer): match "Volume" in a title against "Vol" in a release name #2921 zero padded case and the matcher test fail. Reverting only title_identity.go fails the other author case.
  • TestFilterRelevantVolumeMarkerSpelling (from fix(indexer): match "Volume" in a title against "Vol" in a release name #2921) had separate wantPlain and wantDebug columns documenting the divergence. Changed deliberately: they collapse to one want per case, so the test fails if the two wrappers ever disagree. For production, same spelling Volume 16 for Volume 17 is now dropped (intended); zero padded Vol 07 for Volume 7 stays kept.
  • Fail before: with main's debug.go and searcher.go restored (plus a test shim for filterRelevantDetailed), the entry point test fails for Power Down, 12 Rules and Volume 17 on both SearchBookWithDebug and SearchBookWithOutcomes, while SearchBook passes, which is the blind spot. The identity tables, the reason test and the two flipped volume cases fail too.

Checklist

Test plan

  • go build ./... && go vet ./...
  • go test ./internal/indexer/... ./internal/api/... ./internal/scheduler/... -count=1
  • golangci-lint run ./internal/indexer/...

🤖 Generated with Claude Code

https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9

@codecov

codecov Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/indexer/title_identity.go 75.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

… grab

filterRelevant and filterRelevantDebug were two copies of the relevance
filter. #2502 added its guards, conflictingTitleAuthor and the title
identity phrase gate, to filterRelevant only, and pinned them with tests
that call filterRelevant directly.

Production never runs filterRelevant. The scheduler calls
SearchBookWithOutcomes, which projects SearchBookWithDebug, and the
interactive search calls SearchBookWithDebug; both run
filterRelevantDebug. filterRelevant is reached only through the
scheduler's fallback for searchers that cannot report outcomes, which in
practice is test stubs. So every release #2502 was written to reject was
still kept, and automatic search could still grab it.

filterRelevantDetailed is now the one implementation and returns the
kept results plus a reason for each drop. filterRelevant and
filterRelevantDebug are wrappers over it. The drop record now
distinguishes a conflicting author and a split title phrase from a plain
keyword miss.

The #2502 tables run through both wrappers, and a new test drives
SearchBook, SearchBookWithDebug and SearchBookWithOutcomes against one
fake indexer serving the wrong releases beside the right ones. On main
both production entry points keep all four wrong releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
@vavallee
vavallee force-pushed the fix/relevance-filter-single-implementation branch from 90c12f5 to c74573f Compare September 27, 2026 02:01
@vavallee

Copy link
Copy Markdown
Owner Author

Rebased onto main now that #2763 has landed. Its elision fallback is folded into filterRelevantDetailed, #2763's own elision tests pass against the single implementation, and the entry point test still shows 8 wrong releases kept on current main without this change.

vavallee added a commit that referenced this pull request Sep 29, 2026
…release is rejected (#2863) (#2872)

conflictingTitleAuthor only read an attribution directly after the title
("<title> by <author>") or after a spaced dash. A series label in between,
as in "Heart of Glass, Skye Druids (03) by Donna Grant EPUB", hid it, so the
release was accepted on its title alone and grabbed for Jennifer Hillier's
"Heart of Glass".

When the release starts with the requested title and neither existing form
applies, look for the first "by" in the part after the title, within a short
series label, and read the words after it (up to a format, release marker or
year) as the attribution for the existing author comparison. A "by" after a
narrator, translator or uploader word is ignored, and a release with no
parsable attribution is accepted as before.

filterRelevantDebug, which both production search paths run
(SearchBookWithOutcomes for automatic search, SearchBookWithDebug for the
interactive panel), never called conflictingTitleAuthor, so the fix would
not have reached a real search. It now runs the same guard and records the
drop reason. Draft #2812 folds both filters into one implementation; this
is the minimal slice of that needed for #2863.



Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
main gained #2863's hand copied author guard and #2921's volume number
guard in filterRelevantDebug. Both are folded into filterRelevantDetailed,
which both wrappers now share, so the production path also runs the
title identity gate it never had.

#2921 pinned the divergence in TestFilterRelevantVolumeMarkerSpelling
with separate plain and debug expectations. They collapse to one
expectation per case. Two cases change for production search: a same
spelling "Volume 16" release for a "Volume 17" title is now dropped
(intended), and a zero padded "Vol 07" release for a "Volume 7" title is
now dropped too (a known gap in the identity gate, marked in the test).

The drop reason for a keyword match that fails the identity gate or the
volume guard now reads "title words appear, but with other words or
numbers among them", which covers both. The changelog fragment is
renamed with the PR number and rewritten: the author half of the fix
reached production in v1.39.0 through #2863.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
@vavallee
vavallee marked this pull request as ready for review October 2, 2026 04:48
…matching

Putting the title identity gate on the production path (previous commits)
started dropping correct releases that pad a position: "Vol 07" for
"Volume 7", "Shield Hero 07" for "Shield Hero 7", "Book 07" for
"Book 7". The gate compared numbers as literal words.

keywordPattern now renders an all digit keyword as 0* followed by the
number without its padding, so the phrase, keyword and in order matchers
and the identity gate (ContainsPhrase) all accept "7", "07" and "007" as
one number, in either direction. The callers' word boundaries keep "7"
from matching "17" or "70". volumeNumberAgrees already ignored padding
through sameVolumeNumber.

conflictingTitleAuthor compares whole normalized strings, so it would
have read "Book 07 - Other Author" as a different title and skipped the
attribution. foldVolumeMarkers becomes foldTitleTokens and also strips
padding, so that release is still caught.

A digit and a spelled out number ("7" and "Seven") stay different; that
case is pinned as a known gap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root-cause analysis is confirmed by the diff: the deleted filterRelevantDebug body computed fullOK = tryMatch && volumeNumberAgrees — no identityOK — while filterRelevant had all three. The unification approach is the right fix; the PR description, test design, and changelog are all solid.

One blocker before merge:

Zero-padding regression — the PR flags it and asks for a decision, so this review is that decision point. titleIdentityWords treats "7" and "07" as different tokens, so "Dungeon Crawler Carl Book 07" is now dropped for "Book 7", "Shield Hero Vol 07" for "Volume 7", etc. Zero-padded volume numbers are extremely common in Usenet release names (scene-style zero padding, SABnzbd defaults). The PR correctly scopes the fix: normalising leading zeros in titleIdentityWords would close it and the test case is already marked KNOWN GAP / want: false. I'd rather see that normalization land here than ship a regression that silently stops grabbing a class of correct releases. The want: false row in volume_marker_test.go:109 is the concrete marker.

Everything else looks good:

  • filterRelevantDetailed is the correct single-implementation shape; filterRelevant and filterRelevantDebug are clean thin wrappers.
  • continue added at searcher.go after filtered = append(filtered, r) is necessary — without it a kept result would also push a drop record. Correct.
  • TestRelevanceGuardsReachEveryProductionEntrypoint is the right kind of test: drives real entry points, not just the filter functions in isolation. This is what was missing from the #2502 test suite.
  • TestFilterRelevantDebugDropReasons pinning reason strings is useful for the interactive panel.
  • relevanceFilters table in title_identity_test.go / TestFilterRelevantVolumeMarkerSpelling collapsing to a single want is an improvement — the split columns were documenting a bug, not an intended design divergence.
  • No security concerns; pure filter logic with no auth/tenancy surface.

— 🤖 Bindery triage bot (automated). Reply to correct me; a human will see it.

vavallee added a commit that referenced this pull request Oct 2, 2026
…ixed-title-relevance

Brings in #2812's zero padding fix: a number matches the same number with
any zero padding in the phrase, keyword and identity checks, and the
attribution guard folds padding too. The series position reading goes
through the same matchers, so it gets the rule without changes here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
@vavallee
vavallee merged commit 624ec30 into main Oct 2, 2026
33 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant