Skip to content

fix(metadata): preserve concurrent edits during hydration (#2758) - #2794

Merged
vavallee merged 2 commits into
vavallee:mainfrom
magrhino:codex/2758-hydration-concurrent-edits
Oct 2, 2026
Merged

vavallee merged 2 commits into
vavallee:mainfrom
magrhino:codex/2758-hydration-concurrent-edits

Conversation

@magrhino

Copy link
Copy Markdown
Contributor

Summary

Split out of #2759 at the maintainer's request. Hardcover edition hydration no longer overwrites book edits made while editions are being fetched, and it no longer silently drops its own results on rows whose updated_at was stored in a legacy shape.

Closes #2758.

Stacked on #2759. This branch comes from a fork, so it also carries the #2755 commits until #2759 merges. Only the last two commits are in scope:

  1. fix(metadata): preserve concurrent edits during hydration (#2758)
  2. fix(metadata): compare raw updated_at text in the hydration guard (#2758)

What changed

  • Guarded metadata write. BookRepo.UpdateHydratedMetadata writes only the hydrated columns, and only if the row hasn't changed since the snapshot. When a concurrent edit wins, the book is reloaded, so the caller can't act on a discarded ASIN or discarded metadata.
  • Guarded edition upsert. EditionRepo.UpsertMetadata checks that the parent still has the foreign ID and provider seen before the fetch. The check runs in the same statement as the insert or conflict update.
  • Reload on every path. The caller's book is refreshed on the duration-only, empty, unchanged, and failed-fetch paths before callers search.
  • SetLanguage now bumps updated_at, so a pending hydration sees an imported EPUB language.

How the precondition compares updated_at

The first version of the guard compared updated_at against a reformatted time.Time. books.updated_at holds several historical shapes, and for the two legacy ones the guard never matched: it returned updated=false, err=nil, so every refresh discarded its results.

The guard now keeps the column text exactly as stored, in Book.UpdatedAtRaw, and compares that. It reads the column through CAST(updated_at AS TEXT), because the SQLite driver turns DATETIME columns into timestamps even when the scan target is a string. There is no schema change.

Stored updated_at shape Where it comes from Before After
2026-09-01 10:00:00 CURRENT_TIMESTAMP (migrations 082, 084) never matched; results silently dropped matches
2026-09-01 10:00:00.123456789 +0000 UTC Go time.String() (#914, still written by author merge, #2792) never matched; results silently dropped matches
RFC3339Nano current writers matched matches
unparseable corrupt rows hard invalid book snapshot error matches, and the write normalises the value

I chose this over a row-version column because it needs no migration. Like any timestamp guard, two writes in the same clock tick would look identical. A version column is the only full fix for that, and that would be a separate change.

Tests

  • TestBookRepoUpdateHydratedMetadataLegacyUpdatedAtShapes covers all four stored shapes:
    • an unmodified row accepts the hydration write;
    • a concurrent edit through each writer that bumps updated_at still wins: SetLanguage, SetImageURL, MarkWantedMonitored, SetExcluded, and author merge. It fails for a writer that stops bumping the timestamp.
  • TestHydrateHardcoverEditionsPersistsOverLegacyUpdatedAt runs hydration end to end on both legacy shapes and checks that the result flags and persisted metadata survive. It fails against the old reformatted comparison.
  • TestBookRepoFillMissingAudiobookDurationRequiresMatchingIdentity (from fix(metadata): retain hardcover audiobook runtime before search (#2755) #2759) now also checks that the reloaded duration comes with its own identity, not a stale one.

Follow-ups (not in this PR)

Checklist

  • Commits signed off with git commit -s
  • Tests added or updated
  • docs/DEPLOYMENT.md updated if env vars, config, or upgrade path changed (not applicable)
  • Added a changelog fragment under changelog.d/ (2758-hydration-concurrent-edits.md)
  • Wiki pages updated if user-facing behaviour changed (docs/User-Guide-Wiki.md)

Test plan

  • go test -count=1 ./cmd/... ./internal/...
  • go vet ./internal/...
  • cd web && npm run build (not run; no web changes)

@github-actions github-actions Bot added the bindery-notified Discord notification already sent for this PR label Sep 25, 2026
@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.54054% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/db/books.go 86.79% 4 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@vavallee

Copy link
Copy Markdown
Owner

Early look since it's still a draft, no action needed yet. Reading `CAST(updated_at AS TEXT)` and comparing against that same raw text is the right fix for the problem that sank the earlier version: it's correct whatever shape the row holds, so none of the nine historical layouts can make it silently refuse.

The guard is only as good as "every writer of a guarded column bumps `updated_at`", so I went through main's `UPDATE books` statements that don't:

Writer Column Matters here?
`SetFilePath` fallback, `books.go:1118` `status` yes, `status` is in your guarded set. That's #2793, so it wants to land before or with this
`SetLanguage`, `books.go:1134` `language` yes, and you've already fixed it here
`SetCalibreID`, `books.go:1162` `calibre_id` no, not in the guarded set
`db.go:647`, `:1064` `dedup_key`, `sort_key` no, startup backfills

The main import path is fine: `AddBookFile` goes through `refreshBookStatus` into `Update`, which bumps it.

@magrhino

Copy link
Copy Markdown
Contributor Author

Thanks for walking through the UPDATE books writers, that table pinned down the gap.

The SetFilePath fallback from #2793 is fixed in this PR as of 4588a40. It now bumps updated_at alongside status and writes through r.exec. The concurrent-writer table in TestBookRepoUpdateHydratedMetadataLegacyUpdatedAtShapes has a new SetFilePathFallback case. It forces the fallback by storing an unscannable calibre_id so GetByID fails, then checks that a pending hydration write loses. Without the fix it fails on all four stored updated_at shapes.

Verifying that turned up one more pre-existing problem in the same fallback, now open as #2819. The fallback takes the legacy write on any GetByID error and drops that error, never adds a book_files row, and reports success for a missing book. It doesn't affect the hydration guard, so I've kept it out of this PR.

#2759 has merged, so I've rebased onto main. The diff is now just this PR's three commits. I'll mark it ready for review once CI is green.

@vavallee

Copy link
Copy Markdown
Owner

Thanks! Heads up that #2824 landed after your fix and removed the SetFilePath fallback entirely: a load error is now returned and a missing book is reported. So the fallback hunk and the SetFilePathFallback subtest can go when you rebase onto main. Once it's rebased I'll take another look.

)

Guard metadata writes with the original update timestamp and edition
writes with the parent identity observed before fetching. Advance the
book timestamp when imported EPUB language changes.

Reload current book state before callers search, including duration-only,
empty, unchanged, and failed fetch outcomes. Preserve ingestion provider
context only while the work identity still matches.

Keep the repository interfaces, hydration callers, concurrency regression
tests, and documented behavior together.

Signed-off-by: magrhino <12738722+magrhino@users.noreply.github.com>
…vallee#2758)

books.updated_at holds several historical shapes: CURRENT_TIMESTAMP from
migrations 082 and 084, and Go's time.String form from vavallee#914. Comparing
the column against a reformatted time.Time never matched those rows, so
every refresh silently discarded the hydrated metadata, and an
unparseable value failed with an invalid-snapshot error.

Carry the stored text on the book as UpdatedAtRaw, read through
CAST(updated_at AS TEXT) because the driver normalises DATETIME columns,
and compare that text in UpdateHydratedMetadata. No schema change.

Cover all four stored shapes, a concurrent edit through every writer
that bumps updated_at, and the identity the duration reload returns.

Signed-off-by: magrhino <12738722+magrhino@users.noreply.github.com>
@magrhino
magrhino force-pushed the codex/2758-hydration-concurrent-edits branch from 4588a40 to 3763daa Compare September 28, 2026 23:00
@magrhino

Copy link
Copy Markdown
Contributor Author

Rebased onto main and dropped the fallback hunk and subtest now that #2824 removed it.

@vavallee
vavallee merged commit 9124e65 into vavallee:main Oct 2, 2026
41 checks passed
vavallee added a commit that referenced this pull request Oct 5, 2026
#2926) (#3008)

* fix(metadata): guard audnex enrichment writes against concurrent edits (#2926)

Three paths made an Audnex call and then wrote the whole book row from
the snapshot taken before it, so an edit saved during the call was lost:
the ABS importer, the Hardcover list syncer and the manual enrich
endpoint. They now write through the same updated_at guard hydration
uses since #2794.

The background paths use UpdateHydratedMetadata, which writes only the
enrichment columns and, on a lost guard, reloads the row so the rest of
the import or sync works from the user's edit. The manual endpoint
retries once from a fresh read (the Audnex answer is cached, so this
normally costs no second request) and returns 409 if the row changes
again. When the endpoint also maps the book onto the canonical work for
its ASIN it needs the whole row, so BookRepo gains UpdateIfUnchanged,
the guarded sibling of Update.

Rebind had the same shape around its provider fetch and now uses
UpdateIfUnchanged too, answering 409 on a lost guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(web): tell a changed book apart from a duplicate in rebind conflicts (#2926)

The rebind modal turned every 409 without force_required into "This
metadata record conflicts with an existing book", so a user whose book
was edited during the rebind was told the target id was taken. Both new
409s (rebind and enrich) now carry reason "changed", and the modal shows
its own string for it. The enrich button already showed the server's
message; a test now pins that.

List sync logs a dropped enrichment at INFO, since it is not retried on
later syncs, and the comment no longer claims the reloaded row reaches
the widen path's queued search.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

---------

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
vavallee added a commit that referenced this pull request Oct 8, 2026
#2423) (#3107)

* fix(db): refresh a book's status without rewriting the whole row (#2375)

refreshBookStatus read the book, derived its status and file paths, then
wrote every column back through Update from that snapshot, so a user edit
that landed between the read and the write was reverted. It now writes
only the five columns it derives, guarded on the media_type and status it
derived them from, and derives again when either changed.

The second call site #2758 found (HydrateHardcoverEditions) was already
moved onto the guarded UpdateHydratedMetadata in #2794.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): queue a library scan requested while one is running (#3014)

POST /library/scan answered 409 while a scan was walking and dropped the
request, so a file placed in a folder the walk had already passed waited
for the next scheduled scan. A request made during a scan now sets a
pending flag and gets 202 with queued=true; the running scan runs exactly
one follow-up when it finishes, however many requests arrived. The gate
and the flag change together under a mutex so a request racing the end of
a scan is never lost. The cron path still skips rather than queues, but
runs a manual request queued during it.

Reported by foobarbigtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): reconcile an untracked audiobook beside a tracked one in a flat layout (#1985)

scanLibrary marked the parent folder of every tracked audio file as tracked
so the sibling tracks of a multi-track audiobook count as already tracked
(#1436). In a flat Author/Title.mp3 layout that parent is the author
folder, so every other audiobook by the author was absorbed and never
reached the matching tiers or Unmatched.

An audio file with no book folder of its own (directly in a root or in an
author folder) no longer marks its parent. The folder instead remembers the
track stems of its tracked files (the name with part, track and chapter
numbers removed), and only a sibling with the same stem, or a bare track
number, is absorbed. Book folder layouts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(web): refresh the book page while its download is importing (#2423)

The book page's live refresh (#1161) armed on book statuses nothing ever
wrote, so it never ran, and #2374 removed it. GET /book/{id} now carries
importInFlight, set while a download row for the book is grabbed,
downloading, completed, importPending or importing. The parked hand-off
states are left out so a page does not poll for hours. The page polls the
book every 5s while the flag is set and refreshes the history on the
reload that clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): let Fix match correct the link without moving the file (#2055)

Fix match ran the full import, so correcting which book a file belongs to
also moved it into the target's folder and renamed it from the naming
template. #2224 added the warning and destination preview; this adds the
other half the issue asked for.

POST /queue/manual-import/reassign takes relocate. false moves the
book_files row to the target book through MoveBookFile, keeps the stored
path spelling, records bookFileMoved in history and answers 200; no import
runs and nothing on disk changes. true or absent is the old import, so API
callers are unchanged. The modal now asks what should happen to the file
and defaults to Correct the match only; the move option keeps the
destination preview and the no undo warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): compare only the text before a track number in a flat folder (#1985)

Review of the first pass: audioTrackStem stripped numbers anywhere, so
track names carrying chapter titles (01 - Chapter One, Title - 02 - The
Middle), spelled parts (Part Two), n of m and a glued CD1 got different
stems and became unmatched rows where main had absorbed them. The stem is
now the text before the first track number (digits, or a spelled number
after a track label), with track labels removed; a name that starts with
its number is a bare track and is absorbed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(api): scope the book page's import in flight flag to the queue's owner (#2423)

HasImportInFlight counted every download for the book, so under tenancy a
user viewing a book nobody owns could learn that another user had a grab in
flight. It now takes the same QueryScope owner filter as the queue, with
ListScopeUserID from the request (admins and API key unscoped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): name the scan a library scan request will be answered by (#3014)

The Settings page matched its scan's result by comparing the browser clock
with ran_at, which can pick up the scan it queued behind, and gave up after
two minutes even while the server was still scanning. Each scan now gets an
id (boot id plus sequence), the 202 returns the id the requested scan's
result will carry (scanId), and the stored result carries scan_id. The
status endpoint adds the live running and queued state. The page waits for
its own scan_id and only gives up after two minutes when the server reports
nothing walking or queued.

Also tests the scheduled path: a scheduled scan walking, a manual request
queued behind it, a second scheduled tick skipping, and exactly one
follow-up carrying the manual request's id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(db): guard the status refresh on the stored file paths too (#2375)

The guarded write compared only media_type and status, so a concurrent
refresh that moved a path without changing either (a reorganize running
UpdateBookFilePath) could be overwritten by an older derivation, putting
the old path back. The guard now also compares the raw ebook_file_path and
audiobook_file_path columns, read directly rather than through the
bookColumns book_files view, with IS so a NULL read still matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(importer): record history when Fix match links a file nothing tracked (#2055)

RelinkFile wrote a bookFileMoved row only when it took the file from
another book, so linking an untracked file changed the book with nothing
on its History. It now writes the row in that case too, without the from
fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* fix(web): translate the library scan give up message (#3014)

The message the scan poll shows when it stops waiting was a hard coded
English string with a dash; it is now settings.general.scanCheckBack,
read through a ref so the poll does not restart when t changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* test(api): expect the live scan state in the admin scan status body (#3014)

TestLibraryScanStatusRouteAllowsAdmin compared the admin body byte for
byte with the stored blob, which no longer holds now that the status adds
the live running and queued flags. It now checks every stored field comes
back unchanged and that the only additions are those two booleans. The
non admin case is unchanged and still refused before the handler with no
paths in the body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

---------

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bindery-notified Discord notification already sent for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hardcover edition hydration can overwrite concurrent book edits during metadata promotion

2 participants