Skip to content

Import publication fails on Unraid user shares with a cache pool: "Destination mount changed before publication" #61

Open

Description

@ArnoldDeRuiter

Version: v0.3.8 (ghcr.io/logabell/dewarr:latest), Unraid 7.3.2, library on /mnt/user (shfs) mounted into the container.

What happens
An ebook import stages fine, then publication fails with "Destination mount changed before
publication". Retry Import then fails with "Publication settings or destination identity
changed" because the journal keeps the old destination_identity.

Cause (reproduced)
Unraid's shfs (fuse_useino=yes) reports a merged directory's inode from one of its branches,
preferring the cache pool. The share uses a cache pool, so the first write into the library
(the staging dir / new author folder) creates a cache branch and the destination root's
st_ino changes within seconds:

folder only on disk2 -> /mnt/user st_ino 649081296301245283
after writing a file via /mnt/user (lands on cache)
-> /mnt/user st_ino 12384899292958279

publication.py compares object_id(destination) = (st_dev, st_ino) recorded at plan time with
the root right before publishing, so it fails. Same class as #13 (SMB source identity), but
for the publication destination.

Expected
Either tolerate a changed destination root inode on FUSE/union filesystems (e.g. verify by
path + a marker file in the root, as the staging side already uses markers), or document
that Unraid shares must not use a cache pool. Also: a way to re-plan when the stored
destination_identity is stale, instead of Retry Import failing permanently.

Workaround
Set the library share to array-only (no cache pool), cancel the import and import again.


Proposed fix: stable destination identity on union/FUSE filesystems (Unraid shfs, mergerfs)

Problem: object_id(fd) = (st_dev, st_ino). On Unraid shfs (fuse_useino=yes) and mergerfs
(inodecalc=hybrid-hash), a library root's st_ino changes when a new branch (cache pool /
another disk) starts holding part of the directory - e.g. right after Dewarr's own first write.
Five call sites compare it: publication.py, combine.py, recovery.py, cancel_files.py,
bookdrop.py. Staged items already fall back to a marker (same_object(...) or has_publication_marker(...)); the destination root has no such fallback.

1. New helper module services/app/importing/root_identity.py

"""Identity of a library destination root that survives st_ino changes on union filesystems."""
import os
from uuid import uuid4

ROOT_MARKER = ".dewarr-library-root"


def _read_marker(root_fd):
    try:
        fd = os.open(ROOT_MARKER, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=root_fd)
    except FileNotFoundError:
        return None
    try:
        return os.read(fd, 64).decode().strip() or None
    finally:
        os.close(fd)


def ensure_root_marker(root_fd):
    """Create the hidden marker once (race-safe with O_EXCL); return its token."""
    token = _read_marker(root_fd)
    if token:
        return token
    token = uuid4().hex
    try:
        fd = os.open(ROOT_MARKER, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o644, dir_fd=root_fd)
    except FileExistsError:
        return _read_marker(root_fd)
    try:
        os.write(fd, (token + "\n").encode())
        os.fsync(fd)
    finally:
        os.close(fd)
    return token


def destination_identity(root_fd):
    st = os.fstat(root_fd)
    return {"device": st.st_dev, "inode": st.st_ino, "root_marker": ensure_root_marker(root_fd)}


def same_destination(root_fd, expected):
    """Inode match (old journals) OR same device + same root marker (union filesystems)."""
    st = os.fstat(root_fd)
    if st.st_dev == expected.get("device") and st.st_ino == expected.get("inode"):
        return True
    marker = expected.get("root_marker")
    return bool(marker) and st.st_dev == expected.get("device") and _read_marker(root_fd) == marker

2. Call sites (same change in all five modules)

-from .publication import object_id, same_object
+from .publication import object_id, same_object
+from .root_identity import destination_identity, same_destination

 # recording the destination when a receipt is created
-                    "destination_identity": object_id(destination),
+                    "destination_identity": destination_identity(destination),

 # the early receipt consistency check
-            if receipt.get("spec_hash") != spec_hash or receipt.get(
-                "destination_identity"
-            ) != object_id(destination):
+            if receipt.get("spec_hash") != spec_hash or not same_destination(
+                destination, receipt.get("destination_identity") or {}
+            ):
                 raise PublicationError("Publication settings or destination identity changed")

 # the check right before publishing (publication.py ~line 1407, combine.py ~line 505)
                         with directory(spec.destination_root) as current:
-                            if not same_object(current, receipt["destination_identity"]):
+                            if not same_destination(current, receipt["destination_identity"]):
                                 raise PublicationError(
                                     "Destination mount changed before publication"
                                 )

Also: publication_lock(staging, json.dumps(object_id(destination), ...)) keys the lock on the
inode - key it on (st_dev, root_marker) instead, otherwise two imports could take different
locks for the same root after an inode change.

3. Safety this keeps

  • Still refuses a genuinely different directory: another mount (st_dev differs) or another
    root (no marker / different token) - which is what the check is for.
  • Existing journals without root_marker keep the strict inode behaviour (no silent change
    for in-flight imports); a re-plan writes the new field.
  • The marker is hidden (dot-file) like .book-search-staging; Audiobookshelf/Grimmory skip it.
    Route verification could create it up front so the first import needs no write.

4. Re-plan path for stale journals

"Retry Import" currently fails forever with "Publication settings or destination identity
changed". When the only mismatch is the destination identity (spec_hash equal, receipt state
preparing/prepared, nothing published yet), offer "re-plan": rewrite destination_identity
and continue, instead of requiring Cancel + new import.

5. Tests

  • Fake os.fstat / tmpdir test: record identity, change the root's reported st_ino,
    publication still succeeds via the marker.
  • Different root dir with no marker -> still raises "Destination mount changed".
  • Old-format receipt (no root_marker) + changed inode -> still raises (unchanged behaviour).

Activity

  1. linear-code commented on Oct 6, 2026

    @linear-code
  2. self-assigned this
    on Oct 8, 2026
  3. ArnoldDeRuiter commented on Nov 4, 2026

    @ArnoldDeRuiter
    Author

    Hi @logabell I've seen #73 for which thanks!

    I validated v0.3.9 on Unraid 7.3.2, library share with a cache pool (/mnt/user via shfs), running the shipped destination_identity()/same_destination() inside the container as uid 99 against a test root:

    • Root only on disk7, marker on disk7, identity recorded → a write via /mnt/user creates a cache branch, root st_ino changes, same_destination = True (the old inode-only check = False). The original failure is fixed, thanks!
    • Remaining edge case: the first import creates .book-search-root on the cache pool. When the mover moves it to the array nightly), the marker's own inode changes, so same_destination = False for any receipt recorded before the move. It's a one-time window per root (the marker stays on the array afterwards), but an import that spans that first mover run gets held for review.

    Possible mitigations: document running the mover once after "Save & verify folder", or accept a marker whose token matches while only its inode changed, if the file is still a single-link regular file under the same root/device. Happy to test either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions