fix(indexer): run the #2502 relevance guards on the search paths that grab - #2812
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… grab filterRelevant and filterRelevantDebug were two copies of the relevance filter. #2502 added its guards, conflictingTitleAuthor and the title identity phrase gate, to filterRelevant only, and pinned them with tests that call filterRelevant directly. Production never runs filterRelevant. The scheduler calls SearchBookWithOutcomes, which projects SearchBookWithDebug, and the interactive search calls SearchBookWithDebug; both run filterRelevantDebug. filterRelevant is reached only through the scheduler's fallback for searchers that cannot report outcomes, which in practice is test stubs. So every release #2502 was written to reject was still kept, and automatic search could still grab it. filterRelevantDetailed is now the one implementation and returns the kept results plus a reason for each drop. filterRelevant and filterRelevantDebug are wrappers over it. The drop record now distinguishes a conflicting author and a split title phrase from a plain keyword miss. The #2502 tables run through both wrappers, and a new test drives SearchBook, SearchBookWithDebug and SearchBookWithOutcomes against one fake indexer serving the wrong releases beside the right ones. On main both production entry points keep all four wrong releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee <vavallee@protonmail.com>
90c12f5 to
c74573f
Compare
…release is rejected (#2863) (#2872) conflictingTitleAuthor only read an attribution directly after the title ("<title> by <author>") or after a spaced dash. A series label in between, as in "Heart of Glass, Skye Druids (03) by Donna Grant EPUB", hid it, so the release was accepted on its title alone and grabbed for Jennifer Hillier's "Heart of Glass". When the release starts with the requested title and neither existing form applies, look for the first "by" in the part after the title, within a short series label, and read the words after it (up to a format, release marker or year) as the attribution for the existing author comparison. A "by" after a narrator, translator or uploader word is ignored, and a release with no parsable attribution is accepted as before. filterRelevantDebug, which both production search paths run (SearchBookWithOutcomes for automatic search, SearchBookWithDebug for the interactive panel), never called conflictingTitleAuthor, so the fix would not have reached a real search. It now runs the same guard and records the drop reason. Draft #2812 folds both filters into one implementation; this is the minimal slice of that needed for #2863. Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee <vavallee@protonmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
main gained #2863's hand copied author guard and #2921's volume number guard in filterRelevantDebug. Both are folded into filterRelevantDetailed, which both wrappers now share, so the production path also runs the title identity gate it never had. #2921 pinned the divergence in TestFilterRelevantVolumeMarkerSpelling with separate plain and debug expectations. They collapse to one expectation per case. Two cases change for production search: a same spelling "Volume 16" release for a "Volume 17" title is now dropped (intended), and a zero padded "Vol 07" release for a "Volume 7" title is now dropped too (a known gap in the identity gate, marked in the test). The drop reason for a keyword match that fails the identity gate or the volume guard now reads "title words appear, but with other words or numbers among them", which covers both. The changelog fragment is renamed with the PR number and rewritten: the author half of the fix reached production in v1.39.0 through #2863. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee <vavallee@protonmail.com>
…matching
Putting the title identity gate on the production path (previous commits)
started dropping correct releases that pad a position: "Vol 07" for
"Volume 7", "Shield Hero 07" for "Shield Hero 7", "Book 07" for
"Book 7". The gate compared numbers as literal words.
keywordPattern now renders an all digit keyword as 0* followed by the
number without its padding, so the phrase, keyword and in order matchers
and the identity gate (ContainsPhrase) all accept "7", "07" and "007" as
one number, in either direction. The callers' word boundaries keep "7"
from matching "17" or "70". volumeNumberAgrees already ignored padding
through sameVolumeNumber.
conflictingTitleAuthor compares whole normalized strings, so it would
have read "Book 07 - Other Author" as a different title and skipped the
attribution. foldVolumeMarkers becomes foldTitleTokens and also strips
padding, so that release is still caught.
A digit and a spelled out number ("7" and "Seven") stay different; that
case is pinned as a known gap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
There was a problem hiding this comment.
The root-cause analysis is confirmed by the diff: the deleted filterRelevantDebug body computed fullOK = tryMatch && volumeNumberAgrees — no identityOK — while filterRelevant had all three. The unification approach is the right fix; the PR description, test design, and changelog are all solid.
One blocker before merge:
Zero-padding regression — the PR flags it and asks for a decision, so this review is that decision point. titleIdentityWords treats "7" and "07" as different tokens, so "Dungeon Crawler Carl Book 07" is now dropped for "Book 7", "Shield Hero Vol 07" for "Volume 7", etc. Zero-padded volume numbers are extremely common in Usenet release names (scene-style zero padding, SABnzbd defaults). The PR correctly scopes the fix: normalising leading zeros in titleIdentityWords would close it and the test case is already marked KNOWN GAP / want: false. I'd rather see that normalization land here than ship a regression that silently stops grabbing a class of correct releases. The want: false row in volume_marker_test.go:109 is the concrete marker.
Everything else looks good:
filterRelevantDetailedis the correct single-implementation shape;filterRelevantandfilterRelevantDebugare clean thin wrappers.continueadded atsearcher.goafterfiltered = append(filtered, r)is necessary — without it a kept result would also push a drop record. Correct.TestRelevanceGuardsReachEveryProductionEntrypointis the right kind of test: drives real entry points, not just the filter functions in isolation. This is what was missing from the #2502 test suite.TestFilterRelevantDebugDropReasonspinning reason strings is useful for the interactive panel.relevanceFilterstable intitle_identity_test.go/TestFilterRelevantVolumeMarkerSpellingcollapsing to a singlewantis an improvement — the split columns were documenting a bug, not an intended design divergence.- No security concerns; pure filter logic with no auth/tenancy surface.
— 🤖 Bindery triage bot (automated). Reply to correct me; a human will see it.
…ixed-title-relevance Brings in #2812's zero padding fix: a number matches the same number with any zero padding in the phrase, keyword and identity checks, and the attribution guard folds padding too. The series position reading goes through the same matchers, so it gets the rule without changes here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee <vavallee@protonmail.com>
Summary
The relevance filter existed as two copies,
filterRelevantandfilterRelevantDebug. #2502 added its two guards (conflictingTitleAuthorand the title identity phrase gate) tofilterRelevantonly, and pinned them with tests that callfilterRelevantdirectly.Production never runs
filterRelevant. The scheduler callsSearchBookWithOutcomes, which projectsSearchBookWithDebug, and the interactive search callsSearchBookWithDebug. Both runfilterRelevantDebug.filterRelevantis reached only by the scheduler's fallback for searchers that cannot report outcomes, which in practice is test stubs.Since this was drafted, #2863 copied the author guard across by hand (shipped in v1.39.0), and #2921 added a volume number guard to both copies. The title identity gate was never copied, so on current main the production path still keeps:
filterRelevantThe last row is the divergence #2921's author flagged.
Change
filterRelevantDetailedis now the single implementation. It returns the kept results plus a reason for every drop, andfilterRelevantandfilterRelevantDebugare thin wrappers over it, so a guard can't land in one path and miss the other again. #2863's author guard, #2921'svolumeNumberAgrees, #2763's elision fallback and the identity gate all live in that one function. The drop record tells the interactive panel which check fired: a different author for this title, title words present but with other words or numbers among them, or the plain keyword miss.Zero padding
Putting the identity gate on the production path at first dropped correct releases that pad a number ("Vol 07" for "Volume 7", "Book 07" for "Book 7"), because the gate compared numbers as literal words. Fixed in this PR (cc9d92b):
keywordPatternrenders an all digit keyword as0*plus the number without padding, so the phrase, keyword and in order matchers andContainsPhrase(the identity gate) treat "7", "07" and "007" as one number, both directions. Word boundaries keep "7" from matching "17" or "70".volumeNumberAgreesalready ignored padding.conflictingTitleAuthorcompares whole normalized strings, sofoldVolumeMarkersbecamefoldTitleTokensand strips padding too. Without that, "Dungeon Crawler Carl Book 07 - Some Other Writer" slips past the attribution check (a test pins it).Remaining known gap
A digit and a spelled out number are still different words: "The Seven Habits of Highly Effective People" is dropped for "The 7 Habits of Highly Effective People", which production keeps on main. Pinned with a KNOWN GAP comment in
TestFilterRelevantNumberPadding. No word to number equivalence here on purpose.Comparison against main's production filter
271 distinct (title, author, release) triples: everything the indexer, api and scheduler suites feed the filter, plus 67 hand written realistic release names. Each was run through
filterRelevantDebugon origin/main and on this branch. Differences:Every other triple, including the three padding rows, is decided the same way as on main.
Tests
TestRelevanceGuardsReachEveryProductionEntrypointdrivesSearchBook,SearchBookWithDebugandSearchBookWithOutcomesagainst one fake indexer serving wrong releases beside the right ones; it now includes the same spelling Volume 16 case.TestFilterRelevantDebugDropReasonspins the reason text for each check.TestFilterRelevantNumberPadding(both wrappers) andTestContainsPhraseNumberPadding: padded and unpadded numbers match both ways; 17, 70, 017 and Volume 16 do not; a padded release naming another author is dropped; 7 versus Seven is the known gap.release.goandtitle_identity.goat the previous commit, the three padding rows, the reverse and triple padded cases, the fix(indexer): match "Volume" in a title against "Vol" in a release name #2921 zero padded case and the matcher test fail. Reverting onlytitle_identity.gofails the other author case.TestFilterRelevantVolumeMarkerSpelling(from fix(indexer): match "Volume" in a title against "Vol" in a release name #2921) had separatewantPlainandwantDebugcolumns documenting the divergence. Changed deliberately: they collapse to onewantper case, so the test fails if the two wrappers ever disagree. For production, same spelling Volume 16 for Volume 17 is now dropped (intended); zero padded Vol 07 for Volume 7 stays kept.debug.goandsearcher.gorestored (plus a test shim forfilterRelevantDetailed), the entry point test fails for Power Down, 12 Rules and Volume 17 on bothSearchBookWithDebugandSearchBookWithOutcomes, whileSearchBookpasses, which is the blind spot. The identity tables, the reason test and the two flipped volume cases fail too.Checklist
docs/Troubleshooting-Wiki.mdalready describes for fix(indexer): reject conflicting release title and author evidence #2502)changelog.d/2812-relevance-filter-single-implementation.md)Test plan
go build ./... && go vet ./...go test ./internal/indexer/... ./internal/api/... ./internal/scheduler/... -count=1golangci-lint run ./internal/indexer/...🤖 Generated with Claude Code
https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9