Skip to content

feat(author): surface duplicate-title candidates for human review (#1970) - #2678

Merged
vavallee merged 5 commits into
vavallee:mainfrom
gchahcg:feature/duplicate-candidates
Sep 28, 2026
Merged

vavallee merged 5 commits into
vavallee:mainfrom
gchahcg:feature/duplicate-candidates

Conversation

@gchahcg

@gchahcg gchahcg commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Duplicate titles for one author ("The Martian" / "Martian", "Dune" / "Dune
(Unabridged)") were invisible. This adds a read-only candidate report per
author, plus a review modal whose only action is the existing exclude route.
Closes #1970.

Implementation notes

  • internal/duplicates is a new package that nothing else may import — a
    source-scan test (TestIngestionPackagesNeverImportThis) pins the boundary.
    The aggressive fold (case, punctuation, diacritics; & → "and") is
    deliberately separate from the dedup key, which must stay conservative.

  • Four rules, each reported per group and per member:

    Rule Match
    alnum-equal identical after the aggressive fold
    article-strip identical after dropping a leading article (The/A/An, der/die/das, le/la, …)
    edition-suffix identical after dropping a trailing edition marker (Unabridged, Audiobook, …)
    substring one folded title contained in the other (min-length + residual guards)
  • Groups are linked transitively (A≈B, B≈C → one group); output is
    deterministic (groups by key, members by id).

  • GET /author/{id}/duplicate-candidates returns only groups with ≥2
    non-excluded members; it writes nothing.

Where it's mounted / how it's wired

  • Route: GET /author/{id}/duplicate-candidates (cmd/bindery/main.go).
  • Handler: internal/api/authors_duplicate_candidates.go (one file per
    feature, matching the repo convention).
  • UI: "Review duplicates…" in the author page's More menu →
    DuplicateCandidatesModal (fetches on open, per-row Exclude calls
    PUT /book/{id}/exclude, re-fetches; excluded members struck through).
  • Docs: docs/API.md endpoint + response shape; docs/User-Guide-Wiki.md
    review-flow paragraph.

Why minimal / scope decisions

  • Detection is aggressive, action is human: nothing is merged, deleted, or
    rewritten automatically. The endpoint is read-only by design.
  • Substring rule has guards (min key length 6, min residual 6, marker-only
    residual suppressed) so "Works of X" vs "Complete Works of X" does not
    light up; accepted false-positive classes are pinned in tests.
  • Pairwise scan is O(n²) and the substring pass is capped at 2000 books per
    author (a 1000-book author scans in well under 2 s).
  • The UI entry point is a menu item, not a page-load fetch: the report is
    opt-in, so it is computed on demand.

Suggested review order

  1. internal/duplicates/duplicates.go — fold, keys, rules, Scan.
  2. internal/duplicates/duplicates_test.go — rule matrix + pinned FPs.
  3. internal/duplicates/imports_test.go — the import-boundary pin.
  4. internal/api/authors_duplicate_candidates.go + tests — handler contract.
  5. web/src/components/DuplicateCandidatesModal.tsx + test — review flow.
  6. docs/API.md, docs/User-Guide-Wiki.md.

Checklist

  • Tests added or updated
  • Doc-update gate cleared (see commits skill — docs/, README.md, godoc, Helm values)
  • Wiki pages updated if user-facing behaviour changed

Test plan

  • go test ./cmd/... ./internal/...
  • cd web && npm run build
  • make check (build, vet, golangci-lint, govulncheck, test, test-race, web ci/typecheck/lint/build/test)
  • go test ./internal/duplicates/ (L1 rule matrix)
  • go test ./internal/api/ -run TestDuplicateCandidates (handler: groups, 404, empty, exclusion suppression, read-only, tenancy, 1000-book scale)
  • cd web && npx vitest run src/components/DuplicateCandidatesModal.test.tsx (5 tests)
  • cd web && npx vitest run (full web suite, 1007 tests)

Follow-ups (not in this PR)

  • Cross-author duplicate detection (same book under two author rows).
  • A "keep this one, exclude the rest" bulk action in the modal.

key={rule}
className="rounded bg-slate-200 px-1.5 py-0.5 text-[11px] text-slate-600 dark:bg-zinc-800 dark:text-zinc-400"
>
{t(`duplicateCandidates.rules.${rule}`, ruleDefaults[rule])}
@github-actions github-actions Bot added the bindery-notified Discord notification already sent for this PR label Sep 18, 2026
@codecov

codecov Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.99099% with 20 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/duplicates/duplicates.go 93.51% 6 Missing and 6 partials ⚠️
internal/api/authors_duplicate_candidates.go 78.37% 5 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@gchahcg
gchahcg force-pushed the feature/duplicate-candidates branch from eb3779b to 8570bc4 Compare September 19, 2026 14:03
gchahcg pushed a commit to gchahcg/bindery that referenced this pull request Sep 19, 2026
)

The modal tracked one busy book id at a time, so starting a second
row's toggle re-enabled the first row's button; a second click could
fire an overlapping flip that cancels the first out silently. Track
in-flight rows per book instead.

The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass,
but only the substring rule is gated above 2000 books. State what the
loop actually does.
gchahcg pushed a commit to gchahcg/bindery that referenced this pull request Sep 21, 2026
)

The modal tracked one busy book id at a time, so starting a second
row's toggle re-enabled the first row's button; a second click could
fire an overlapping flip that cancels the first out silently. Track
in-flight rows per book instead.

The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass,
but only the substring rule is gated above 2000 books. State what the
loop actually does.

Signed-off-by: gch ahcg <gchahcg@proton.me>
@gchahcg
gchahcg force-pushed the feature/duplicate-candidates branch from 451a6ad to 4aa4967 Compare September 21, 2026 01:47
gchahcg pushed a commit to gchahcg/bindery that referenced this pull request Sep 24, 2026
)

The modal tracked one busy book id at a time, so starting a second
row's toggle re-enabled the first row's button; a second click could
fire an overlapping flip that cancels the first out silently. Track
in-flight rows per book instead.

The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass,
but only the substring rule is gated above 2000 books. State what the
loop actually does.

Signed-off-by: gch ahcg <gchahcg@proton.me>
@gchahcg
gchahcg force-pushed the feature/duplicate-candidates branch from 4aa4967 to a8216f5 Compare September 24, 2026 22:29

@vavallee vavallee left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the shape I asked for on #1970: read only, `loadOwnedAuthor` on the route so it's tenancy scoped, the aggressive fold kept in its own package with a source scan test holding the boundary, and each group saying which rule fired. It builds and its tests pass once the docs conflict is resolved.

One thing blocks it, and it's about the only action the modal offers being Exclude.

The substring rule groups separate books of a series. I ran a batch of real titles through `duplicates.Scan`:

Grouped Rule Actually
Foundation, Foundation and Empire, Second Foundation substring three different Asimov novels
Mistborn, Mistborn: The Final Empire substring same book, correct
The Hobbit, The Hobbit (Unabridged) edition-suffix correct
The Martian, Martian article-strip correct
Dune, Dune Messiah, Children of Dune not grouped correct, but only because "dune" is under the length floor

So the guard is length based, and a long first book title that doubles as the series name (Foundation, Outlander, Discworld style) trips it. Someone trusting the report excludes Foundation and Empire.

The cleanest fix I can see: suppress a substring match when both books are linked to the same series at different positions, since two different positions in one series are by definition different works. That keeps Mistborn (usually one series position) and drops Foundation. If series links aren't available at that point, dropping substring from the default rule set and keeping the other three is also fine.

Two small things alongside:

  • It conflicts with main in `docs/User-Guide-Wiki.md` only.
  • It needs a changelog fragment under `changelog.d/` (`### Added`, no dashes in the text).

gch ahcg added 2 commits September 27, 2026 10:20
…vallee#1970)

The same book often reaches a catalogue twice under slightly different
titles ("The Martian" / "Martian", "Dune" / "Dune (Unabridged)"), and
there was no way to see the collisions. This adds a read-only report
that groups an author's titles that look like the same book, and a
review modal whose only action is the existing exclude route.

Detection lives in internal/duplicates, a package nothing else may
import (a source-scan test pins the boundary): an aggressive title
fold (case, punctuation, diacritics; & expanded to "and") feeds four
rules — alnum-equal, article-strip, edition-suffix, and a guarded
substring — and matches are linked transitively into groups. Each
group and member reports which rule(s) matched, so the UI can explain
every group in plain language. The aggressive fold is separate from
the dedup key on purpose: it may be aggressive, the dedup key may not.

GET /author/{id}/duplicate-candidates writes nothing; groups whose
members are all already excluded are omitted. The author page's More
menu gains "Review duplicates…", and the modal lists each group with
its rules and an Exclude button per row, which calls
PUT /book/{id}/exclude and re-fetches. Excluded members are shown
struck through, and a group disappears once every member is excluded.

API.md documents the endpoint and response; the user guide explains
the review flow.

Signed-off-by: gch ahcg <gchahcg@proton.me>
)

The modal tracked one busy book id at a time, so starting a second
row's toggle re-enabled the first row's button; a second click could
fire an overlapping flip that cancels the first out silently. Track
in-flight rows per book instead.

The Scan godoc claimed maxPairwiseBooks caps the O(n^2) pairwise pass,
but only the substring rule is gated above 2000 books. State what the
loop actually does.

Signed-off-by: gch ahcg <gchahcg@proton.me>
@gchahcg
gchahcg force-pushed the feature/duplicate-candidates branch from a8216f5 to 5ee8ab8 Compare September 27, 2026 14:28
@gchahcg

gchahcg commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Fixed — thanks for running real titles through it.

Added duplicates.SeriesSlot and a sameSeriesDifferentPosition guard: Scan now takes each book's series memberships and drops a substring-only match between two books that are known, different positions in the same series. DuplicateCandidates loads them via h.series.ListBookSeriesMembershipsByAuthor (nil-guarded for tests/deployments without a series repo, which falls back to today's behaviour).

Verified against your table:

  • Foundation / Foundation and Empire / Second Foundation (positions 1/2/3, same series) → no longer grouped
  • Mistborn / Mistborn: The Final Empire (both position 1) → still grouped, unaffected
  • The Hobbit / Hobbit (Unabridged), The Martian / Martian → unaffected (different rules)

Added unit tests in duplicates_test.go for both cases plus an end-to-end handler test wiring real series_books rows. Also rebased onto main (post-#2673) and fixed the docs/User-Guide-Wiki.md conflict, and added the changelog.d fragment.

…vavallee#1970)

The substring rule alone grouped separate books in a series whenever a
long series-opener title doubled as a prefix of a sequel's title
("Foundation" vs "Foundation and Empire", "Second Foundation") —
found in review by running real titles through duplicates.Scan.

DuplicateCandidates now also loads the author's series memberships and
passes them to Scan, which drops a substring-only match between two
books that are known, different positions in the same series — by
definition different works. Same-position pairs (a genuine subtitle
variant, e.g. Mistborn / Mistborn: The Final Empire) are unaffected.
Missing series data (h.series nil, or no membership recorded) falls
back to today's behaviour, per the review's accepted fallback.

Signed-off-by: gch ahcg <gchahcg@proton.me>
@gchahcg
gchahcg force-pushed the feature/duplicate-candidates branch from 5ee8ab8 to b9f15a7 Compare September 27, 2026 16:44
vavallee and others added 2 commits September 28, 2026 18:11
…lee#1970)

The series position guard only helps when series links exist, and they
are often missing. Without them the substring rule still grouped a
series opener with every sequel carrying its name. Real catalogues with
no series data: Asimov "Foundation" pulled in nine other titles as one
group, King's "The Dark Tower" merged true and false pairs into a single
transitive group, and "The Science of Discworld" grouped with its two
sequels, as did "Carrie" with "Carrie Soto Is Back".

The shorter title must now equal a whole part of the longer one as it is
punctuated (split at a colon, semicolon, bracket, spaced hyphen, or en
or em dash), compared with and without a leading article. Mistborn,
The Gunslinger, Hogfather (Discworld, vavallee#20) and the other real subtitle
cases still group; the series guard still covers "Mistborn" against
"Mistborn: The Well of Ascension".

The aggressive key also drops Latin diacritics after the umlaut
expansion, so "Les Misérables" meets "Les Miserables", which is what
the alnum-equal rule text in the UI already claimed.

The handler series test gains a position 2 sequel that the separator
rule matches, so removing the guard now fails it; with the old Foundation
fixture it passed either way.

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Drop the dashes, describe the separator rule, and credit gchahcg.

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
@vavallee

Copy link
Copy Markdown
Owner

Thanks for turning the series fix around so fast, the guard works exactly as described. I pushed two commits on top:

  • Subtitle matches now need a separator. The series guard only helps when series links exist, and plenty of authors don't have them. With no series data, Asimov's "Foundation" still pulled nine titles into one group (Prelude to Foundation, Foundation and Chaos, The Foundation Trilogy...), and "The Science of Discworld" grouped with its sequels, as did "Carrie" with "Carrie Soto Is Back". Now the shorter title has to be a whole part of the longer one as it's punctuated (colon, bracket, spaced dash). Mistborn, The Gunslinger, Hogfather (Discworld, release: v0.6.1 #20) and friends still group, and your series guard still earns its keep on "Mistborn" vs "Mistborn: The Well of Ascension". Across about 215 real Asimov, King and Pratchett titles the report went from 16 groups (roughly half wrong) to 10 (nine clearly right).
  • Accents fold now. "Les Misérables" and "Les Miserables" weren't grouping, even though the UI text says diacritics are ignored.
  • The handler series test gets a position 2 sequel, since the Foundation fixture passed with the guard deleted once the separator rule landed.
  • Docs and the UI rule text now describe the new rule, and the changelog fragment dropped its dashes and credits you.

Rest looks good to me.

@vavallee
vavallee merged commit c2831bb into vavallee:main Sep 28, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bindery-notified Discord notification already sent for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature proposal: opt-in, human-reviewed duplicate-title report

3 participants