feat(author): surface duplicate-title candidates for human review (#1970) - #2678
Conversation
| key={rule} | ||
| className="rounded bg-slate-200 px-1.5 py-0.5 text-[11px] text-slate-600 dark:bg-zinc-800 dark:text-zinc-400" | ||
| > | ||
| {t(`duplicateCandidates.rules.${rule}`, ruleDefaults[rule])} |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
eb3779b to
8570bc4
Compare
) The modal tracked one busy book id at a time, so starting a second row's toggle re-enabled the first row's button; a second click could fire an overlapping flip that cancels the first out silently. Track in-flight rows per book instead. The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass, but only the substring rule is gated above 2000 books. State what the loop actually does.
) The modal tracked one busy book id at a time, so starting a second row's toggle re-enabled the first row's button; a second click could fire an overlapping flip that cancels the first out silently. Track in-flight rows per book instead. The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass, but only the substring rule is gated above 2000 books. State what the loop actually does. Signed-off-by: gch ahcg <gchahcg@proton.me>
451a6ad to
4aa4967
Compare
) The modal tracked one busy book id at a time, so starting a second row's toggle re-enabled the first row's button; a second click could fire an overlapping flip that cancels the first out silently. Track in-flight rows per book instead. The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass, but only the substring rule is gated above 2000 books. State what the loop actually does. Signed-off-by: gch ahcg <gchahcg@proton.me>
4aa4967 to
a8216f5
Compare
vavallee
left a comment
There was a problem hiding this comment.
This is the shape I asked for on #1970: read only, `loadOwnedAuthor` on the route so it's tenancy scoped, the aggressive fold kept in its own package with a source scan test holding the boundary, and each group saying which rule fired. It builds and its tests pass once the docs conflict is resolved.
One thing blocks it, and it's about the only action the modal offers being Exclude.
The substring rule groups separate books of a series. I ran a batch of real titles through `duplicates.Scan`:
| Grouped | Rule | Actually |
|---|---|---|
| Foundation, Foundation and Empire, Second Foundation | substring | three different Asimov novels |
| Mistborn, Mistborn: The Final Empire | substring | same book, correct |
| The Hobbit, The Hobbit (Unabridged) | edition-suffix | correct |
| The Martian, Martian | article-strip | correct |
| Dune, Dune Messiah, Children of Dune | not grouped | correct, but only because "dune" is under the length floor |
So the guard is length based, and a long first book title that doubles as the series name (Foundation, Outlander, Discworld style) trips it. Someone trusting the report excludes Foundation and Empire.
The cleanest fix I can see: suppress a substring match when both books are linked to the same series at different positions, since two different positions in one series are by definition different works. That keeps Mistborn (usually one series position) and drops Foundation. If series links aren't available at that point, dropping substring from the default rule set and keeping the other three is also fine.
Two small things alongside:
- It conflicts with main in `docs/User-Guide-Wiki.md` only.
- It needs a changelog fragment under `changelog.d/` (`### Added`, no dashes in the text).
…vallee#1970) The same book often reaches a catalogue twice under slightly different titles ("The Martian" / "Martian", "Dune" / "Dune (Unabridged)"), and there was no way to see the collisions. This adds a read-only report that groups an author's titles that look like the same book, and a review modal whose only action is the existing exclude route. Detection lives in internal/duplicates, a package nothing else may import (a source-scan test pins the boundary): an aggressive title fold (case, punctuation, diacritics; & expanded to "and") feeds four rules — alnum-equal, article-strip, edition-suffix, and a guarded substring — and matches are linked transitively into groups. Each group and member reports which rule(s) matched, so the UI can explain every group in plain language. The aggressive fold is separate from the dedup key on purpose: it may be aggressive, the dedup key may not. GET /author/{id}/duplicate-candidates writes nothing; groups whose members are all already excluded are omitted. The author page's More menu gains "Review duplicates…", and the modal lists each group with its rules and an Exclude button per row, which calls PUT /book/{id}/exclude and re-fetches. Excluded members are shown struck through, and a group disappears once every member is excluded. API.md documents the endpoint and response; the user guide explains the review flow. Signed-off-by: gch ahcg <gchahcg@proton.me>
) The modal tracked one busy book id at a time, so starting a second row's toggle re-enabled the first row's button; a second click could fire an overlapping flip that cancels the first out silently. Track in-flight rows per book instead. The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass, but only the substring rule is gated above 2000 books. State what the loop actually does. Signed-off-by: gch ahcg <gchahcg@proton.me>
a8216f5 to
5ee8ab8
Compare
|
Fixed — thanks for running real titles through it. Added Verified against your table:
Added unit tests in |
…vavallee#1970) The substring rule alone grouped separate books in a series whenever a long series-opener title doubled as a prefix of a sequel's title ("Foundation" vs "Foundation and Empire", "Second Foundation") — found in review by running real titles through duplicates.Scan. DuplicateCandidates now also loads the author's series memberships and passes them to Scan, which drops a substring-only match between two books that are known, different positions in the same series — by definition different works. Same-position pairs (a genuine subtitle variant, e.g. Mistborn / Mistborn: The Final Empire) are unaffected. Missing series data (h.series nil, or no membership recorded) falls back to today's behaviour, per the review's accepted fallback. Signed-off-by: gch ahcg <gchahcg@proton.me>
5ee8ab8 to
b9f15a7
Compare
…lee#1970) The series position guard only helps when series links exist, and they are often missing. Without them the substring rule still grouped a series opener with every sequel carrying its name. Real catalogues with no series data: Asimov "Foundation" pulled in nine other titles as one group, King's "The Dark Tower" merged true and false pairs into a single transitive group, and "The Science of Discworld" grouped with its two sequels, as did "Carrie" with "Carrie Soto Is Back". The shorter title must now equal a whole part of the longer one as it is punctuated (split at a colon, semicolon, bracket, spaced hyphen, or en or em dash), compared with and without a leading article. Mistborn, The Gunslinger, Hogfather (Discworld, vavallee#20) and the other real subtitle cases still group; the series guard still covers "Mistborn" against "Mistborn: The Well of Ascension". The aggressive key also drops Latin diacritics after the umlaut expansion, so "Les Misérables" meets "Les Miserables", which is what the alnum-equal rule text in the UI already claimed. The handler series test gains a position 2 sequel that the separator rule matches, so removing the guard now fails it; with the old Foundation fixture it passed either way. Signed-off-by: vavallee <vavallee@protonmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Drop the dashes, describe the separator rule, and credit gchahcg. Signed-off-by: vavallee <vavallee@protonmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
|
Thanks for turning the series fix around so fast, the guard works exactly as described. I pushed two commits on top:
Rest looks good to me. |
Summary
Duplicate titles for one author ("The Martian" / "Martian", "Dune" / "Dune
(Unabridged)") were invisible. This adds a read-only candidate report per
author, plus a review modal whose only action is the existing exclude route.
Closes #1970.
Implementation notes
internal/duplicatesis a new package that nothing else may import — asource-scan test (
TestIngestionPackagesNeverImportThis) pins the boundary.The aggressive fold (case, punctuation, diacritics;
&→ "and") isdeliberately separate from the dedup key, which must stay conservative.
Four rules, each reported per group and per member:
alnum-equalarticle-stripedition-suffixsubstringGroups are linked transitively (A≈B, B≈C → one group); output is
deterministic (groups by key, members by id).
GET /author/{id}/duplicate-candidatesreturns only groups with ≥2non-excluded members; it writes nothing.
Where it's mounted / how it's wired
GET /author/{id}/duplicate-candidates(cmd/bindery/main.go).internal/api/authors_duplicate_candidates.go(one file perfeature, matching the repo convention).
DuplicateCandidatesModal(fetches on open, per-row Exclude callsPUT /book/{id}/exclude, re-fetches; excluded members struck through).docs/API.mdendpoint + response shape;docs/User-Guide-Wiki.mdreview-flow paragraph.
Why minimal / scope decisions
rewritten automatically. The endpoint is read-only by design.
residual suppressed) so "Works of X" vs "Complete Works of X" does not
light up; accepted false-positive classes are pinned in tests.
author (a 1000-book author scans in well under 2 s).
opt-in, so it is computed on demand.
Suggested review order
internal/duplicates/duplicates.go— fold, keys, rules,Scan.internal/duplicates/duplicates_test.go— rule matrix + pinned FPs.internal/duplicates/imports_test.go— the import-boundary pin.internal/api/authors_duplicate_candidates.go+ tests — handler contract.web/src/components/DuplicateCandidatesModal.tsx+ test — review flow.docs/API.md,docs/User-Guide-Wiki.md.Checklist
commitsskill —docs/,README.md, godoc, Helm values)Test plan
go test ./cmd/... ./internal/...cd web && npm run buildmake check(build, vet, golangci-lint, govulncheck, test, test-race, web ci/typecheck/lint/build/test)go test ./internal/duplicates/(L1 rule matrix)go test ./internal/api/ -run TestDuplicateCandidates(handler: groups, 404, empty, exclusion suppression, read-only, tenancy, 1000-book scale)cd web && npx vitest run src/components/DuplicateCandidatesModal.test.tsx(5 tests)cd web && npx vitest run(full web suite, 1007 tests)Follow-ups (not in this PR)