Skip to content

feat: adopt unmatched library books from the Import page (#2547) - #2671

Merged
vavallee merged 19 commits into
mainfrom
feature/library-adoption
Sep 18, 2026
Merged

vavallee merged 19 commits into
mainfrom
feature/library-adoption

Conversation

@vavallee

@vavallee vavallee commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Summary

Library adoption: the books a library scan could not match stop being a read only, 1000 file table buried in Settings and become decisions on the Import page. /import opens on In your library, one row per book, with suggested matches (a one click Confirm only for a strong match), an in place editor, one group row per missing author with a single Add author, ignore, and Undo. Adoption is the scanner's own reconcile with a person supplying the match: files are registered where they are, nothing moves, nothing is queued or searched.

Stacks on #2662 and must merge after it. It reuses addBookCore from that PR; the base of this draft is refactor/add-book-core.

Closes #2547 (193 tracks are one row).

Design

Layout of /import (default view), desktop:

Import                                                     [ In your library | From a folder ]
Bring books you already have into Bindery.

+-------------------------------------------------------------------------------------------+
|  11  books need a decision      58 files   last scan 2 minutes ago   1 adopted  [Scan now] |
+-------------------------------------------------------------------------------------------+

FOLDERS              [Search titles, authors and folders]   (Needs a decision (11)) Ignored  Adopted
 All folders     11  [Filters]
 • Becky Chambers 4  +-----------------------------------------------------------------------------+
 • Andy Weir      3  | BOOK ⇅                          | BEST MATCH ⌄                  |           |
 • Brandon S.     2  |---------------------------------+-------------------------------+-----------|
 • Ann Leckie     1  | Artemis [Ebook]                 | [Strong match] Artemis        | [Confirm] More |
                     | 1 file · 508 KB · Andy Weir/... |                               |           |
                     | A Martyrs Tale [Ebook]          | [Possible match] The Martian  | [Choose book] More |
                     | Provenance [Ebook]              | No close title by Ann Leckie  | [Choose book] More |
                     | > Becky Chambers                | Author not in your library yet| [Add author] More |
                     |   4 books, 28 files · Becky ... |                               |           |
                     |     A Psalm for the Wild-Built  |                               | [Choose book] More |
                     +-----------------------------------------------------------------------------+

Principles:

  • One decision is shown once. Pending books on the page that share an author folder and failed because the author is not in the library become one group row: author, book and file counts, one line, a single primary Add author, and More (Show books, Ignore folder). Opening the group lists its books, which keep Choose book and Ignore in their own More menu and do not repeat the sentence.
  • Confirm is earned. STRONG_MATCH_SCORE = 0.92 in adoptionMatch.ts, and the author the scan read must match the book's author. A strong suggestion shows a green "Strong match" chip and a primary Confirm. Anything weaker shows a neutral "Possible match" chip and the title as a quiet link that opens the editor with it preselected; the row's primary action is Choose book.
  • Rows are one height. Every row has the same fixed width action cell: one primary button and a More menu (Choose book, Show files, Ignore). The middle cell is one truncated line of fact ("No close title by Ann Leckie"); the full sentence, which ends in the thing to do, is the tooltip and the first line of the editor, with the scanner's reason code only in the tooltip.
  • The rail is navigation. One clickable item per folder filtering the list, count right aligned, an amber dot for a folder whose author is missing. No actions.
  • Numbers are quiet. Counts and sizes use tabular numerals in the muted colour; the summary strip has one large number (books needing a decision) and secondary stats.

Row states:

State Middle cell Action cell
Strong suggestion Strong match chip, title Confirm, More (Choose book, Show files, Ignore)
Possible suggestion Possible match chip, title link (opens editor preselected) Choose book, More
No suggestion one line fact Choose book, More
Author missing, alone "X is not in your library" Add author, More (Choose book, Ignore)
Group "Author not in your library yet" Add author, More (Show books, Ignore folder)
Book inside an open group empty Choose book, More
Expanded (in place, no modal) editor below: full sentence, suggestions as radio rows with score bars, library search focused, collapsed metadata search asked only on Search, Adopt as Ebook/Audiobook, file list, Cancel / Adopt as X
Adopting / adopted / ignored quiet status line Undo once decided, until the next fetch; if Undo kept a book in use, a quiet line says so
Failed line returns with the server's sentence as an alert actions return
Ignored and Adopted lists fact / "Adopted as X" linked Unignore / Undo

Designed empty and status states: never scanned (explains a scan, Scan library), scan running (pulsing dot, polling every 3 s only while running), everything matched, truncated, scan error, a scan that found no files ("your decisions were kept"), nothing ignored, nothing adopted, filters match nothing (Clear filters). Mobile: rows and groups stack as cards with the same primary action and More menu, and the rail scrolls as chips. Keyboard: roving tabindex over rows and group rows, arrows / Home / End, Enter opens a book's editor or a group's books, Esc closes and returns focus to the row, i ignores, u undoes, / focuses search. The table has a caption naming the list and the keys.

Screenshots were taken from a local instance seeded with a demo library in headless Chromium with Inter installed (the wide word spacing is the headless renderer): list light and dark, weak suggestion opened in the editor, a strong suggestion confirmed, a group expanded, a row's More menu, 390 px mobile, and the folder view. They are local files, listed in the builder's report, not attached here.

Implementation notes

Layer What
088_unmatched_units.sql unmatched_units, unit_path UNIQUE, states pending/adopting/adopted/undoing/ignored, created_book_id / created_author_id / registered_paths_json for exact undo, member_paths_json, candidates, indexes on (state, top_score DESC, rel_path), (state, author_folder, rel_path), (state, reason), (book_id); book and author FKs ON DELETE SET NULL
internal/db/unmatched_units*.go ReconcileScan (chunks of 500, never moves an ignored row, adopted row back to pending only if adopted before the scan started, deletes unseen pending, purges decisions unseen 30 days, deletes nothing on zero files or error, releases claims stale for 15 min), ClaimState compare and swap, list with switch mapped sort, facets, one query book hydration
internal/importer/unmatched_units.go groupUnmatched (audio by folder, disc folders by the book folder, root level audio alone, ebooks by folder + stem), S10 eligibility, top 3 candidates from booksByAuthor with the reconcile's Jaro-Winkler, threshold 0.60, WithUnmatchedUnits, ScanRunning, audioTrackedByFolder
internal/importer/unit_boundaries.go discFolderRe (+ IsDiscFolderName), AllDiscFolders, SameStem moved from scan_walk.go; dirSubtreeHasAudio moves with AllDiscFolders because it is its only caller
internal/importer/scanner.go call sites only: walk keeps size and mode, collector instead of the 1000 cap, record units after the loop, blob gains unmatched_units, ignored_units, units_truncated, keeps unmatched_files: [] with a dated comment; the disc tracked check calls audioTrackedByFolder
internal/api/adoption.go, adoption_adopt.go list, summary, adopt, undo, ignore, unignore, bulk ignore; route group registerAdoptionRoutes under RequireAdmin
web pages/import/ImportPage, FolderImportView + FolderImportRow (split from ManualImportPage), AdoptionView, AdoptionSummary, AdoptionRail, AdoptionFacets, AdoptionTable, AdoptionRow, AdoptionEditor, useAdoptionList + adoptionReducer (named actions), adoptionHint; components/import/BookPicker, CatalogueAdder; components/SortHeader, useUnmatchedCount; api/adoption.ts. Largest new component 170 lines

Registration. AddBookFileIfMissing, which is AddBookFile (the scanner's write, scanner.go reconcile tiers) plus whether the row was inserted, so undo removes only what adoption added. An audiobook folder unit registers its folder, the shape an imported audiobook download already has (SetFormatFilePath(destDir)); a disc set registers the book folder, and the scan's tracked check now treats a disc folder's tracks as tracked when the folder above is. Ebook units register each member. The walked path is registered, not the resolved one, so the next scan recognises it; containment is still proven by ResolveContained. Undo uses BookRepo.UntrackFilePath, skipping any path that has since moved to another book.

SkipCatalogueSync: true. Adding a book never pulls the author's bibliography anyway (#1816), so the flag only affects the single work fallback. With it set, a provider failure answers at once instead of polling for 15 seconds, there is no second provider call, and no background sync can create the book after the adopt request has given up and compensated, which would strand a row undo does not know about. A person adopting a file by a new author gets that one book and that author, which is what they asked for.

BookCreated race. When the add created the book, every file on it at the end of the adopt is recorded as registered (the add's own library lookup can attach one), and undo deletes the created book only if it has no files left and no other row references it; same for a created author with no books.

Overlap with #2480 (contributor PR, changes requested)

File #2480 hunks This PR
internal/api/scan_walk.go @@ -51,7 +51,15 (walk limits), @@ -86,6 +94,9 (inside enumerateImportUnits) removes regexp/strings imports and discFolderRe above the limits, rewrites the two helper calls to importer.AllDiscFolders / importer.SameStem in the unit heuristic, deletes the three helpers at the end of the file
web/src/pages/ManualImportPage.tsx 8 hunks file deleted; its body is pages/import/FolderImportView.tsx + FolderImportRow.tsx verbatim apart from the CatalogueAdder props, so #2480's hunks port by path
web/src/pages/ManualImportPage.test.tsx 4 hunks renamed to pages/import/FolderImportView.test.tsx, import paths only
web/src/pages/settings/ImportTab.tsx, FolderScanSection.test.tsx FolderScanSection changes FolderScanSection retired to a link card and its test deleted, so those hunks drop
internal/db/books.go @@ -749,6 +749,21 untouched here

useFolderScan does not exist on this base (it arrives with #2480), so C5's "delete it if unused" has nothing to delete yet; if #2480 lands first, its only caller is FolderScanSection, which this PR removes.

Second backend review fixes (3cb3bdde, web note 63750f6e)

Item Fix Test
1 disc grouping, safe direction subfolders with no audio beneath them and hidden or system folders (leading ., @, #) are ignored; a book folder directly under a root can group CD/Disc folders, the root itself never; bare numbers are no longer disc names; Part N stays apart TestGroupUnmatched_DiscSetsFollowTheWalkerRule: every layout from the review table plus @eaDir, Book N, a non disc audio subfolder and Root/CD1,CD2
2 reversal errors were swallowed reverse returns its first error; Undo answers 503 and keeps the claim and record; a failed adopt whose reversal fails answers 503 and keeps its claim; recovery skips and retries next pass; the record is cleared only after every step succeeded TestUndo_KeepsTheRecordWhenReversalFails, TestAdopt_KeepsTheClaimWhenItsReversalFails (both then finished by recovery)
3 writes not scoped to the claim Claim returns a token (claim time plus random nonce) stored in claimed_at; progress, completion, reset, undo completion and release all require claimed_at IS token; recovery uses the stale row's token TestClaimToken_ScopesEveryWrite (A claims, recovery releases, B claims, every late A write refused)
4 fingerprint misses adds blocklist entries, calibre_id, editions and provider identifiers; the known false positives (status refresh, cover writes, author refresh moving updated_at) are documented on BookFingerprint as the safe direction and left alone TestUndo_KeepsACreatedBookSomeoneStartedUsing gains blocklist and calibre id cases
5 ignored rows under a removed root ConfiguredRoots option; old ignored rows purge under a root that produced files or a root no longer configured; a configured root that produced nothing keeps them TestReconcileScan_PurgesIgnoresUnderARemovedRoot

Fail before, on 63750f6e:

--- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule (0.04s)
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/Author/Series/1,2 (0.00s)
        unmatched_discs_test.go:91: units = 1 [Brandon Sanderson/Mistborn], want 2
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/Root/Book/CD1,CD2 (0.00s)
        unmatched_discs_test.go:91: units = 2 [Artemis/CD1 Artemis/CD2], want 1
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/Author/Book/CD1,CD2_+_Artwork_with_only_a_jpg (0.00s)
        unmatched_discs_test.go:91: units = 2 [Andy Weir/Artemis/CD1 Andy Weir/Artemis/CD2], want 1
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/Author/Book/CD1,CD2_+_empty_hidden_folder (0.00s)
        unmatched_discs_test.go:91: units = 2 [Andy Weir/Artemis/CD1 Andy Weir/Artemis/CD2], want 1
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/Author/Book/CD1,CD2_+_Synology_@eaDir (0.00s)
        unmatched_discs_test.go:91: units = 2 [Andy Weir/Artemis/CD1 Andy Weir/Artemis/CD2], want 1
FAIL
FAIL	github.com/vavallee/bindery/internal/importer	0.054s
FAIL
--- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing (0.36s)
    --- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing/blocklist_entry (0.09s)
        adoption_undo_safety_test.go:156: created book deleted although it was in use
    --- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing/calibre_id (0.09s)
        adoption_undo_safety_test.go:156: created book deleted although it was in use
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.369s
FAIL

Checked by reverting each fix in place (these tests need the new seams):

## 2 undo keeps the record when reversal fails: fix reverted in place
--- FAIL: TestUndo_KeepsTheRecordWhenReversalFails (0.09s)
    adoption_recovery_test.go:117: undo with a failing untrack = 200 {...state pending...}
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.095s
FAIL

## 3 claim token scopes writes: fix reverted in place
--- FAIL: TestClaimToken_ScopesEveryWrite (0.08s)
    unmatched_units_purge_test.go:121: A's progress landed on B's claim
    unmatched_units_purge_test.go:124: A completed B's claim
    unmatched_units_purge_test.go:134: row after A's late writes = &{... State:adopted ...}, want B's untouched claim
FAIL
FAIL	github.com/vavallee/bindery/internal/db	0.086s
FAIL

## 5 removed root purge: fix reverted in place
--- FAIL: TestReconcileScan_PurgesIgnoresUnderARemovedRoot (0.08s)
    unmatched_units_purge_test.go:87: purged = 0, want only the row under the removed root
FAIL
FAIL	github.com/vavallee/bindery/internal/db	0.084s
FAIL

## 2 adopt keeps the claim when its reversal fails: fix reverted in place
--- FAIL: TestAdopt_KeepsTheClaimWhenItsReversalFails (0.09s)
    adoption_recovery_test.go:163: adopt = 500 {"error":"internal server error"}
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.094s
FAIL

Backend review fixes (e8d7e1a1, b3e8287e)

Item Fix Test
1 Undo removed another book's file each registered path records its book; BookRepo.UntrackFilePathForBook deletes only path AND book_id; a gone book means nothing matches, the unit just returns to pending TestUndo_LeavesAFileThatNowBelongsToAnotherBook
2 author delete cascaded to excluded books ListByAuthorIncludingExcluded; the only other "unused" check (bookIsOnlyOurs) reads book_files and unmatched rows, no excluded filter TestUndo_KeepsACreatedAuthorWithAnExcludedBook
3 disc grouping merged separate books folder import walker rule (AllDiscFolders over every subfolder) plus CD / Disc / bare number names only, never a root or author folder; disc sets register each disc folder; scanner disc tracked rule reverted TestGroupUnmatched_DiscSetsFollowTheWalkerRule, TestScanLibrary_TrackedFolderDoesNotHideALaterSibling, TestAdopt_RegistersEachDiscFolder
4 Undo deleted a book in use fingerprint (monitored, updated_at, history, downloads, pending releases, series links) recorded at adopt, compared before untracking; used book and author kept, message says so TestUndo_KeepsACreatedBookSomeoneStartedUsing (monitored, series link)
5 crash mid adopt stranded rows created ids and each registered file written to the row before the next side effect; RecoverStaleClaims reverses like Undo, then resets: at startup for any claim, on list for claims over 15 min; the scan no longer releases claims TestRecoverStaleClaims_ReversesADeadAdopt
6 adopted rows purged after 30 days adopted rows kept while the book exists, purged once book_id is NULL TestReconcileScan_KeepsAdoptionsWhileTheirBookExists
7 zero file guard untested, SkipDeletion unwired scanner test added; SkipDeletion now set for a truncated scan and SkipPurge removed TestScanLibrary_ZeroFileScanKeepsUnits
8a claim dated by updated_at claimed_at column in 088 TestStaleClaims_DatedByClaimedAt
8b truncated scan deleted pending SkipDeletion: truncated TestScanLibrary_TruncatedScanRemovesNothing
8c lost undo completion reported success CompleteUndo false answers 409 TestUndo_ReportsALostCompletion
8d unmounted root lost ignores ignored purge only under RootsWithFiles TestReconcileScan_PurgesOldDecisions
8e contradictory format 400; the editor's Ebook/Audiobook toggle removed since it could only error TestAdopt_RejectsAFormatThatContradictsTheFiles

Decided differently from the review, with reasons:

  • Item 3, "mirror the walker's rule exactly". Exactly mirrored, the rule still merges the reviewer's first case: AllDiscFolders accepts Book N, Part N, Vol N and Chapter N, so a series folder holding Book 1, Book 2, Book 3 with audio is one unit to the folder import walker too. Adoption uses the walker's structure check through the moved helper unchanged, and adds two narrowing conditions of its own: the subfolder names must be CD, Disc or a bare number, and the folder must not be a root or directly under one. IsDiscFolderName and AllDiscFolders are untouched, so the folder import behaviour and the helper move are unchanged.
  • Item 1, "untrack only rows matching both path and that book id". Done at the SQL level with a new BookRepo.UntrackFilePathForBook, rather than UntrackFilePath plus a separate ownership read, so there is no gap between the check and the delete.

Fail before, on 270739a6 (the tree before these fixes) with the new tests copied in:

--- FAIL: TestUndo_LeavesAFileThatNowBelongsToAnotherBook (0.09s)
    adoption_undo_safety_test.go:58: book C files after undo = [], want it to keep Ann Leckie/Ancillary.epub
--- FAIL: TestUndo_KeepsACreatedAuthorWithAnExcludedBook (0.08s)
    adoption_undo_safety_test.go:93: created author deleted although it still has an excluded book
--- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing (0.17s)
    --- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing/monitored (0.09s)
        adoption_undo_safety_test.go:144: created book deleted although it was in use
    --- FAIL: TestUndo_KeepsACreatedBookSomeoneStartedUsing/series_link (0.08s)
        adoption_undo_safety_test.go:144: created book deleted although it was in use
--- FAIL: TestAdopt_RejectsAFormatThatContradictsTheFiles (0.08s)
    adoption_undo_safety_test.go:168: ebook adopted as audiobook = 200 {...adopted...}
--- FAIL: TestAdopt_RegistersEachDiscFolder (0.08s)
    adoption_undo_safety_test.go:191: registered [Andy Weir/Artemis], want the two disc folders
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.524s
FAIL
--- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule (0.01s)
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/series_of_numbered_books_stays_apart (0.00s)
        unmatched_discs_test.go:64: units = 1 [Brandon Sanderson/Mistborn], want 3
    --- FAIL: TestGroupUnmatched_DiscSetsFollowTheWalkerRule/numbered_folders_under_an_author_do_not_merge_the_author (0.00s)
        unmatched_discs_test.go:64: units = 1 [Some Author], want 2
--- FAIL: TestScanLibrary_TrackedFolderDoesNotHideALaterSibling (0.08s)
    unmatched_discs_test.go:94: counters = {FilesFound:2 Reconciled:0 Unmatched:0 AlreadyTracked:2 UnmatchedFiles:[] UnmatchedUnits:0xc0f66c353a8 UnitsTruncated:false}, want the folder's own track tracked and Book 4 considered
FAIL
FAIL	github.com/vavallee/bindery/internal/importer	0.102s
FAIL
--- FAIL: TestReconcileScan_KeepsAdoptionsWhileTheirBookExists (0.09s)
    unmatched_units_purge_test.go:47: adoption of an existing book purged after 90 days: <nil>
FAIL
FAIL	github.com/vavallee/bindery/internal/db	0.090s
FAIL

Tests that need the new seams (the recovery entry point, the unit cap variable, the new options) were checked by reverting each fix in place:

## 5 recovery reverses: mutation reverts the fix
--- FAIL: TestRecoverStaleClaims_ReversesADeadAdopt (0.09s)
    adoption_recovery_test.go:56: registered file still tracked after recovery
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.093s
FAIL

## 7 zero file guard: mutation reverts the fix
--- FAIL: TestScanLibrary_ZeroFileScanKeepsUnits (0.09s)
    unmatched_guards_test.go:54: a scan that found no files removed a stored unit
FAIL
FAIL	github.com/vavallee/bindery/internal/importer	0.090s
FAIL

## 8b truncated scan: mutation reverts the fix
--- FAIL: TestScanLibrary_TruncatedScanRemovesNothing (0.09s)
    unmatched_guards_test.go:82: a truncated scan removed a unit it did not list
FAIL
FAIL	github.com/vavallee/bindery/internal/importer	0.090s
FAIL

## 8a claimed_at: mutation reverts the fix
--- FAIL: TestStaleClaims_DatedByClaimedAt (0.08s)
    unmatched_units_test.go:278: stale claims = [], want only the hour old claim
FAIL
FAIL	github.com/vavallee/bindery/internal/db	0.086s
FAIL

## 8d root with files: mutation reverts the fix
--- FAIL: TestReconcileScan_PurgesOldDecisions (0.08s)
    unmatched_units_test.go:150: reconcile with the row's root producing no files = {Generation:2 Upserted:0 RemovedPending:0 Purged:1 Pending:0 Ignored:1}, want no purge
FAIL
FAIL	github.com/vavallee/bindery/internal/db	0.088s
FAIL

## 8c lost completion: mutation reverts the fix
--- FAIL: TestUndo_ReportsALostCompletion (0.09s)
    adoption_recovery_test.go:86: undo with a lost completion = 200 {...state undoing...}
FAIL
FAIL	github.com/vavallee/bindery/internal/api	0.095s
FAIL

Security

Item How
S9 list parameters sort and dir go through unmatchedOrderBy's switch; search is folded then LIKE escaped and bound; authorFolder, reason, format are bound. TestUnmatchedList_FiltersSortsAndEscapes feeds SQL in sort and authorFolder
S9 state changes adopt, undo, ignore, unignore claim the row with a single conditional UPDATE; TestClaimState_ExactlyOneWinner and TestAdopt_DoubleAdoptAndDoubleUndo (6 concurrent requests, one 200, five 409, under -race)
S9 undo deletes only created_book_id / created_author_id the row recorded, only with no files / books left and no other row referencing them
S9 nav count GET /library/unmatched/summary, admin only; the nav asks only when isAdmin (App test asserts no call for a non admin)
S10 rows only mode.IsRegular() from the walk's Lstat info, and a folder whose resolved path is under a resolved scanned root; TestEligibleUnmatched_SymlinkOutsideRoot, TestScanLibrary_SymlinkedBookIsNotListed
S10 adopt re-Lstat and LibraryRoots.ResolveContained (library roots only, not download dirs) on every member and the folder; TestAdopt_RejectsSymlinkEscape (file swapped for a symlink, and a symlinked folder leaving the library) answer 422 and register nothing
No path input no request field is a path; rows are addressed by id
Routes registerAdoptionRoutes in RequireAdmin; TestAdoptionRoutesEnumerated walks the router so an unlisted route fails, TestAdoptionRoutesRequireAdmin (role user and no role: 403, handler not run), TestAdoptionRoutesAllowAdmin
Tenancy rows are unowned and admin only; a created book or author is owned by the acting admin through addBookCore's context; choosing an existing book leaves owner and monitored as they were (asserted in TestAdopt_ExistingBookRegistersInPlaceAndUndoIsExact)

Performance

Item Result
P2 size and mode come from the walk's os.FileInfo; grouping does no stat. S10 resolves each folder once, not each file
P3 upserts committed every 500 rows with one prepared statement; deletes and purge in one short final transaction
P4 TestUnmatchedList_QueryPlansUseIndexes: every sort searches a state index, default and folder sorts need no temp b-tree. Facets only with facets=1, which the hook sends when state or search changes
P5 no polling except every 3 s while scan.running; nav badge reads once, then from an event after adopt, ignore or scan
T5 TestAdoptionList_NoProviderCallsAndOneHydrationQuery: 5 rows with 2 candidates each, with and without facets, one BookRefs call and zero provider calls. TestScanLibrary_MakesNoOutboundCalls: http.DefaultTransport spy sees nothing during a scan. Web: asks a metadata provider only when the search is submitted spies fetch through render, expand, open and typing, then sees exactly one /search/book on submit
T6 BenchmarkGroupUnmatched (50,000 files) 46 ms/op
T6 BenchmarkReconcileScan (20,000 units) 769 ms/op

Fail before evidence

Blob level tests copied onto the base (58292dea) and run there:

--- FAIL: TestScanLibrary_TrackFolderIsOneUnmatchedBook (0.14s)
    scanner_unmatched_blob_test.go:65: unmatched_files has 193 per file entries, want none: the list is units now
    scanner_unmatched_blob_test.go:68: unmatched_units = <nil>, want 1 book for the 193 tracks
--- FAIL: TestScanLibrary_UnmatchedListHasNoThousandCap (0.22s)
    scanner_unmatched_blob_test.go:93: unmatched_units = <nil> (truncated false), want all 1200 stored; unmatched_files held 1000
FAIL

Web tests copied onto the base:

× links to the Import page views, not to Settings tabs
× counts unmatched books and links to the Import page instead of listing files
Tests  2 failed | 66 passed (68)

The adopt, undo, ignore, reconcile, grouping and view tests exercise code that does not exist on the base, so they do not compile there. The provider on submit test was checked by mutation: making CatalogueAdder search on every keystroke fails it.

Deviations and follow ups

  • T7 asks for userEvent; @testing-library/user-event is not installed and no new npm package was added, so the keyboard path uses fireEvent.keyDown on real focus.
  • The list does not show rows in the transient adopting / undoing states, so every list stays a single state equality in index order; a crashed request's claim is released by the next scan.
  • SortHeader is the third copy of the sortable header; moving BooksPage and AuthorDetailPage onto it is left for a follow up to keep this diff reviewable (C6).
  • Settings links to /import with a plain anchor because the Settings page tree has no router in its tests.
  • Deferred per plan: bulk adopt above a score, single folder rescan, series from path (Pull Series From File Path #1430), embedded EPUB metadata as candidates.

Suggested review order

  1. 9825f2e6 db and migration
  2. 85868532, 8f330151 scanner grouping and candidates
  3. dceb018b API and routes, then e8d7e1a1 and 3cb3bdde review fixes
  4. ae8293af web, then 21126ad6 design pass, then 92cb5d12 removals
  5. 4a531701 docs

Checklist

Test plan

  • go build ./..., go vet ./..., GOOS=windows go build ./..., GOOS=darwin go build ./...
  • golangci-lint run ./... (0 issues)
  • go test ./cmd/... ./internal/... (exit 0, after the review fixes)
  • -race: ./cmd/bindery whole package ok; ./internal/api -run 'TestAdopt|TestUndo|TestIgnore_|TestAdoptionList|TestRecoverStaleClaims|TestEnumerateImportUnits|TestAddBookCore' ok; ./internal/db -run 'Unmatched|ReconcileScan|ClaimState|ClaimToken|BookRefs|Migrate088|StaleClaims|KeepsAdoptions|RemovedRoot' ok; ./internal/importer -run 'GroupUnmatched|EligibleUnmatched|SymlinkedBook|RanksCandidates|NoOutboundCalls|AudioTrackedByFolder|TrackFolderIsOne|NoThousandCap|UnmatchedReason|ClaimRank' ok. No race reported. The whole internal/db and internal/importer packages exceed the 10 minute test timeout under -race on this machine (as internal/api already does, make test: internal/api race package hits the 30-minute timeout #2293), so they were run by filter.
  • cd web && npx tsc --noEmit, npx vitest run (1025 passed), npm run build, npx eslint . (0 errors; warnings are pre-existing files)
  • Local instance seeded with a demo library: scan, list, confirm, editor, dark mode and 390 px layout checked in headless Chromium

🤖 Generated with Claude Code

https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9

vavallee and others added 12 commits September 17, 2026 01:12
…eusable cores

Library adoption and request approval both need to add a book or an
author without an HTTP request. Move the bodies of AddBook and Create
into addBookCore (add_book_core.go) and createAuthorCore
(create_author_core.go). The handlers now decode the body, call the
core, and map its errors back to the exact statuses and bodies they
returned before. Ownership still comes from the context.

This is a pure move: no behaviour change, and the existing tests pass
unmodified. addBookResult reports whether the book and the author were
created by the call, which the later callers need for compensation.
authors.go shrinks by about 480 lines (plan item C1).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…dd cores

Library adoption creates books unmonitored and must not pull in a
catalogue behind them; request approval needs the same control over
the author sync. Both overrides default to what the handlers already do,
and the handlers never set them.

addBookParams.Monitored nil keeps the forced monitored flag.
addBookParams.SkipCatalogueSync skips the single work fallback sync and
looks for the row once instead of polling, since nothing the core
started could still create it. createAuthorParams.SkipCatalogueSync
skips the catalogue sync on both the create and the relink path.

New tests call the cores directly: default params match the handler,
Monitored=false stores an unmonitored book, SkipCatalogueSync makes no
author works call (with a control case that does), and the handler's
error table matches the literal bodies.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Migration 088 adds unmatched_units, one row per book a library scan
could not match, keyed by path so an ignore or an adoption survives the
next scan. UnmatchedUnitRepo.ReconcileScan upserts in chunks of 500 and
never moves an ignored row; it removes unseen pending rows and purges
decisions unseen for 30 days, and does neither when the scan found no
files. Adopt, undo and ignore claim a row with a single conditional
UPDATE so two requests cannot both act on it. The list maps sort keys
through a switch, binds every filter, and hydrates the books a page
points at in one query.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…ts (#2547)

The library scan used to put its first 1000 unmatched files into the
lastScan blob, one entry per file, so a 193 track audiobook filled 193
rows and a large library lost everything past the cap. The scan now
groups unmatched files into book units (audio by folder, disc folders by
the book folder above, same stem ebooks together), ranks up to three
suggestions per unit from the catalogue already in memory with the
reconcile's own title measure, and stores them through
WithUnmatchedUnits. Bounds are 50,000 files and 20,000 units with a
truncated flag. Only regular files whose folder resolves inside a
scanned root become rows (S10); size and mode come from the walk.

The blob keeps its counters and the #1436 invariant, gains
unmatched_units, ignored_units and units_truncated, and writes
unmatched_files as an empty list for one release.

The disc folder regex, AllDiscFolders and SameStem move from
internal/api/scan_walk.go into importer so both scans share one
definition.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…thout sync

Review found that changing the 422 and 404 add book statuses, or the 502
create author status, left the whole internal/api suite green. The
error mapping tests now cover every sentinel and typed error of both
cores with the exact status and the exact JSON body, including the
409 bodies that carry the existing book or canonical author.

Documents what SkipCatalogueSync skips on each core. On the add book
core it removes only the single work fallback; SearchOnAdd still runs.
On the create author core it skips the whole sync, and the search on
add only ever runs inside that sync, so SearchOnAdd with
SkipCatalogueSync would accept a search and never perform it.
createAuthorCore now refuses that pair with
errCreateAuthorSearchNeedsSync. No HTTP request can set
SkipCatalogueSync, so the handler's behaviour is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Adds the admin only /library/unmatched routes. The list filters, sorts
through a fixed switch, returns facets only when asked, and hydrates
every book a page points at in one query. Adopt registers a unit's files
in place against a book already in the library, or against a metadata
result added through addBookCore unmonitored, in the adopted format,
with no search and SkipCatalogueSync set. No route accepts a path.

A request claims its row with a compare and swap, re-checks every file
with Lstat and LibraryRoots.ResolveContained, refuses with 409 when a
file already belongs to a book, and compensates on failure. Undo
untracks exactly what the adoption registered and removes a book or
author only when the row recorded creating it and nothing else holds it.

The scanner is wired to store unmatched units, and ScanRunning exposes
the in flight flag for the page's scan status.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
A multi disc audiobook's tracks parse their title from the disc folder,
so the unit read "CD 1". The unit is the book folder above the discs,
and it is named after that folder now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
/import becomes one page with two views: "In your library", the new
default, and "From a folder" (?view=folder), the existing folder import.
The library view has a summary strip with Scan now, a rail of the author
folders holding the most undecided books with the one action each calls
for, and a list whose rows say in one sentence why the book is here and
end in the thing to do. A suggestion is one click to confirm; any row
opens an editor in place with the suggestions and their scores, a
library search, and a collapsed metadata search that asks a provider
only on submit. Decisions show at once with Undo and revert inline on
error. Rows are keyboard operable (arrows, Enter, Esc, i, u, /) and
stack as cards on narrow screens. Admins see the pending count on the
Import nav entry.

ManualImportPage is split into ImportPage, FolderImportView and
FolderImportRow; BookPicker and CatalogueAdder move to
components/import, and SortHeader is extracted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…rt page

Settings > General no longer lists up to 1000 unmatched files with reason
hints nobody could act on; it shows how many books need a decision and a
link to review them. The bulk folder scan in Settings > Import becomes a
link card to /import?view=folder, and its test goes with it. The empty
Queue and Wanted hints point at the two Import views. The per file
reason and hint strings are removed from en.json and the one stale
translation of them from ko.json.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
A new user guide section, "Adopting files already in your library", the
/library/unmatched routes and their answers in API.md, a troubleshooting
entry for files a scan leaves unmatched, a README feature line, and the
changelog fragment. References to Manual Import, the bulk folder import
and the Settings unmatched table point at the Import page views now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

vavallee and others added 4 commits September 17, 2026 12:18
… actions

Books whose author is not in the library and share a folder are one
group row with a single Add author and a disclosure for the books, and
the rail is plain folder navigation. Only a strong suggestion (title
similarity of at least 0.92 with a matching author, one constant in
adoptionMatch.ts) gets a one click Confirm; anything weaker is a quiet
"Possible match" link that opens the editor preselected. Every row has
the same fixed action cell, one primary button and a More menu, and a
one line fact instead of a wrapping sentence, with the full sentence in
the tooltip and the editor. The summary is one number with quiet stats,
sort arrows are SVG, counts use tabular numerals, and the library search
no longer repeats the metadata search prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…es not own

From the backend review of library adoption:

- Undo removes a book_files row only while it still belongs to the book
  the adoption registered it to (each registered path now records its
  book), so a path since deleted with its book and registered to another
  keeps its new owner.
- A created author counts as unused only with no books at all,
  excluded ones included, since deleting it cascades to them.
- A created book is fingerprinted when the adoption finishes (monitored
  flag, updated_at, history, downloads, pending releases, series links);
  if that changed, Undo untracks the files but keeps the book and author
  and says so in the response message.
- An adopt writes its created ids and each registered file into the row
  before the next side effect. Abandoned adopting or undoing claims,
  dated by a new claimed_at column, are reversed the same way Undo
  reverses and then returned to pending: at startup for any claim, and
  on list for claims older than 15 minutes. A scan no longer releases
  claims itself.
- Adopted rows are kept while their book exists and purged once it is
  gone; ignored rows are purged after 30 days only under a root that
  produced files in the scan.
- A truncated scan removes no pending row (SkipDeletion, now wired).
- A lost undo completion answers 409; a format that contradicts the
  files answers 400.
- Disc sets follow the folder import walker's rule (every subfolder is a
  disc folder with audio, via AllDiscFolders) with CD, Disc or bare
  number names only, and never at a root or author folder, so
  Mistborn/Book 1..3 and Author/1, Author/2 stay separate. A disc set is
  registered disc folder by disc folder, and the scanner's disc tracked
  rule is reverted.

Migration 088 is unreleased and gains claimed_at and
created_book_fingerprint in place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Adopt no longer accepts a format that contradicts the files, so the
editor's Ebook or Audiobook toggle could only ever produce an error. The
user guide and API docs describe what Undo keeps, disc set grouping,
recovery after a crash, and how long adopted and ignored rows last.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
vavallee and others added 3 commits September 17, 2026 12:51
When Undo answers with a message because the book the adoption added
has been used since, the row that returned to pending shows a quiet
line: Files removed. The book stayed because it is now in use. It clears
with the next fetch. Also replaces two NUL bytes that had been written
into a template string in useAdoptionList.ts, which made git treat the
file as binary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
…tes to the claim

From the second backend review of library adoption:

- Disc sets: subfolders with no audio beneath them, and hidden or system
  folders (a leading dot, at sign or hash), no longer stop a folder from
  grouping; a book folder directly under a library root can group its
  CD or Disc folders (the root itself still cannot); bare numbers are no
  longer disc names, so Series/1 and Series/2 stay separate books.
- reverse returns its first error instead of logging it. Undo then
  answers 503 and keeps the claim and the record; a failed adopt whose
  reversal fails answers 503 the same way; recovery leaves the row for
  its next pass. The record is only cleared once every step succeeded.
- Claims carry a token (claim time plus a random nonce) stored in
  claimed_at. Adopt progress, completion, reset and undo completion all
  match it, so a request whose claim was recovered and retaken cannot
  write into the new claim.
- The created book fingerprint also covers blocklist entries, the
  Calibre id, editions and provider identifiers. Its known false
  positives are documented as the safe direction.
- Old ignored rows under a root that is no longer configured are purged;
  a configured root that produced no files still keeps its rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Adopt and bulk ignore decoded their JSON leniently, so a body carrying a
field the route does not take, such as "path": "/etc/passwd", was
silently ignored. No adoption route takes a path; both now answer 400
naming the unknown field, and nothing is changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
@vavallee
vavallee marked this pull request as ready for review September 17, 2026 16:37

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff. Here is what I checked and what I found.

Auth / tenancyregisterAdoptionRoutes wraps every route in r.Use(auth.RequireAdmin) (sensitive_routes.go). TestAdoptionRoutesRequireAdmin tests role user and no-role for all seven routes; TestAdoptionRoutesEnumerated catches any route added without a row in the table. Clean.

SQL injectionsort/dir go through the unmatchedOrderBy switch (fixed strings only); state, reason, authorFolder, format are all bound as ? parameters; search goes through FoldForSearchescapeLike → LIKE bound parameter. TestUnmatchedList_FiltersSortsAndEscapes sends "score); DROP TABLE unmatched_units; --" as the sort and "Other' OR '1'='1" as the folder and confirms neither matches extra rows. The nolint:gosec G202 lines are correctly annotated — where contains only fixed predicates and bound values, and ORDER BY comes from the switch. Clean.

Path traversal / symlinks (S10)registrationPaths calls os.Lstat on each member (catches a symlink swapped in after the scan) and then ResolveContained on every file and derived folder. TestAdopt_RejectsSymlinkEscape covers both cases: a file replaced by a symlink, and a path reached through a symlinked directory that leaves the library. Clean.

No path in request bodydecodeAdoptionBody uses json.Decoder.DisallowUnknownFields. The comment at adoption.go:963 names the intent. adoption_undo_safety_test.go line 2475 sends {"bookId": ..., "path": "/etc/passwd"} and expects a 400 that names the field. Clean.

TOCTOU / concurrent state changesClaim returns a random-nonce token stored in claimed_at; every subsequent write (progress, complete, release, undo complete) requires claimed_at IS token. TestAdopt_DoubleAdoptAndDoubleUndo races six concurrent requests and expects exactly one 200 and five 409s. TestClaimToken_ScopesEveryWrite (listed in the PR body) shows that a late write from a dead request cannot land on a new claimant's row. Clean.

Crash / partial-adopt recovery — Every side effect is persisted to the row before the next step. RecoverStaleClaims(ctx, 0) at startup reverses any claim from a previous process. TestRecoverStaleClaims_ReversesADeadAdopt simulates a panic after the first of two files and confirms recovery untracks the file, deletes the created book and author, and resets the row to pending. TestAdopt_KeepsTheClaimWhenItsReversalFails confirms a failed reversal leaves the claim intact for the next recovery pass. Clean.

Undo safetyUntrackFilePathForBook deletes only the row matching both path and book_id (no gap between check and delete). bookIsOnlyOurs runs after untracking, so the file count it reads is post-untrack. ListByAuthorIncludingExcluded prevents cascading into excluded books. BookFingerprint is read before untracking (untracking itself updates updated_at). All five reviewer items from the first pass and all five from the second have named tests.

No new deps — Checklist confirms no new Go module or npm package. The diff contains no go.mod / package.json changes.

Backward compatunmatched_files is kept as an empty list in the scan blob for one release so an older cached web bundle still parses.

rootPath in API response — The absolute server path is intentional, admin-only, and consistent with the existing /library/scan/status gate that the PR description explicitly references. Not a concern.

One note on the IgnoreMany guard(len(req.IDs) == 0) == (req.AuthorFolder == "") is correct but reads as a double-negation; something like noIDs && noFolder || hasIDs && hasFolder would be easier to reason about at a glance. Not a blocker.

Overall: the change is correct, the security surface is covered by tests, and no blocking defect was found.

— 🤖 Bindery triage bot (automated). Reply to correct me; a human will see it.

@vavallee
vavallee changed the base branch from refactor/add-book-core to main September 18, 2026 01:06
@vavallee
vavallee merged commit 51ff430 into main Sep 18, 2026
39 of 40 checks passed
trevorswanson pushed a commit to trevorswanson/bindery that referenced this pull request Sep 20, 2026
…kip-tracked

Resolves the manual-import wizard's move from web/src/pages/ManualImportPage.tsx
to web/src/pages/import/FolderImportView.tsx (upstream vavallee#2671's Import-page
refactor split it into FolderImportView/FolderImportRow/folderImport.ts and
added the "In your library" adoption view alongside it): the skip-tracked
scan and the "Show already imported" toggle now live in the new files, and
the equivalent Settings > Import bulk-scan UI is gone there too, replaced by
upstream's link out to the Import page.

internal/api/manual_import.go's Scan handler keeps our resolveImportFolder
extraction, now using upstream's outsideRootsMessage for the 403 body.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L3vXCo9PwmffPWUik6Gwvh
Signed-off-by: Trevor Swanson <83826109+trevorswanson@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Library scan: numbered-track audiobook filenames parsed as title '001-190' instead of the real title

2 participants