Skip to content

sort: never lose or overwrite data when filing releases - #3

Merged
Harrsn merged 9 commits into
mainfrom
fix/sorter-data-safety
Sep 16, 2026
Merged

Harrsn merged 9 commits into
mainfrom
fix/sorter-data-safety

Conversation

@Harrsn

@Harrsn Harrsn commented Sep 16, 2026

Copy link
Copy Markdown
Owner

What was wrong

Production runs MEDIASORT_MODE=move + REMOVE_ON_COMPLETE=1, with /downloads and /library as separate bind mounts, so every rename() fails with EXDEV and every "move" is really copy-then-delete. In that setup the sorter could destroy data in several ways, all reproduced locally before fixing:

  • F1 – unfiled content deleted with the torrent. The sorter exited 0 when it filed nothing (e.g. [Grp] Show - 13 [1080p], absolute anime numbering). The hook took rc 0 as success and removed the torrent with its data.
  • F3 / F9 – non-atomic overwrite. Placing over an existing library file wrote directly to the final path. A dropped CIFS link or a --force-recreate mid-copy left a truncated file where a good one used to be. It also overwrote regardless of quality, so a worse release could replace a better one.
  • F2 – samples/extras overwrote the feature. Every video in a movie release planned onto the same Title (Year).mkv. Sample/, Featurettes/ and CD2 files overwrote each other in walk order.
  • F4 – subtitle collisions. .en.srt, .es.srt and .en.forced.srt all mapped to one sidecar name, and RARBG Subs/ folders were never filed.
  • F6 – movie ISOs and RAR'd TV filed as games. Any .iso/.bin/archive made a release a "game".
  • Also found: faucet.classify never imported in production. The hook runs python /app/faucet/sort.py, so sys.path[0] is the package dir and from faucet.classify import … failed silently. Game detection has been running on file extensions alone.

What changed

Decisions: replace only if better, Plex extras folders, quarantine to _failed/.

  • Atomic placement. Copies go to .<name>.<pid>.faucet-partial beside the destination, are fsynced and size-checked, then os.replaced in. Per-process partials mean a hook/sweep race produces one intact winner (the loser exits 2). Partials idle >30 min are reaped.
  • Replace only if provably better. New library_files table records resolution and cam status for every file the sorter places. The sorter strips quality tags from filenames, so this is the only record of what a file is. faucet/quality.py is the shared, dependency-free ranking; library.py now uses it too. Unknown quality on either side keeps the existing file.
  • Samples skipped; extras routed. Extras go to Plex folders (Featurettes/, Trailers/, …) or use -trailer-style suffixes. Suffix-only matching keeps Trailer Park Boys safe. CD1/CD2 stack as - pt1/- pt2; otherwise the largest file wins and the rest are quarantined.
  • Subtitles keep language/flag tags, and Subs/ and per-episode Subs/<stem>/ layouts are filed.
  • Quarantine instead of delete. When a release is being consumed (move mode or REMOVE_ON_COMPLETE), anything not filed is moved intact to <release parent>/_failed/, which the sweep already excludes. QUARANTINE_DIR overrides the location. Junk is now an explicit extension list (the old <5 MB rule classed small ISOs/RARs as junk). Deletion and quarantine refuse library roots and shallow paths.
  • Movies require a year. A yearless "movie" was usually a misparse: ep.mkv inside an episode folder was being filed as movies/ep/.
  • Games: console/ROM formats always count as games. Disc images and archives count only when the classifier says game. An existing game is quarantined, not deleted.
  • hardlink mode falls back to copy, never to move, so seeding survives.
  • Exit-code contract. 0 = filed, 4 = filed with some content quarantined, 2 = I/O error (release left for retry), 1 = fatal. hook.py removes the torrent only on 0/4 and records a quarantined event. sweep.py counts 4 as swept.

Testing

  • tests/test_sort_safety.py: 38 new regression tests covering F1/F2/F3/F4/F6/F9, the hook's removal contract, and the guard rails. One test runs sort.py exactly as the hook does (no PYTHONPATH) to lock in the import fix. Full suite: 144 passed (106 existing + 38) on 3.12; CI covers 3.10/3.11.
  • End-to-end on a real EXDEV pair (tmpfs → disk, 600–800 MB files):
    • SIGKILL mid-upgrade left the library file byte-identical and the source intact, and a rerun completed the upgrade.
    • Two concurrent sorters on one release produced a correct file, no partials, and rc 0/2.

Before/after deploying

  • library_files starts empty. Files Faucet sorted before this PR have no recorded quality, so a new release landing on one of those exact paths is quarantined, not swapped in. This is conservative by design. Upgrades that target a differently-named (tagged) file are unaffected.
  • F14 interaction. Movie upgrades that never clear (e.g. the Zootopia 2 TS loop) will now pile copies into _failed/ every ~48h instead of overwriting. The upgrade/wants PR fixes the root cause; until then, check _failed/ occasionally.
  • Nothing in prod config needs to change. _failed/ is created on first use under /downloads/complete.

Follow-ups (not in this PR)

  • Run lock across hook/sweep/Check now (F5/F12).
  • Upgrade/wants correctness (F14/F19/F20).
  • The README claim that ".torrent URLs that 302 to magnets just work" is false (F29); it gets fixed in the hunter PR.
  • Consider mounting /mnt/nas once for both paths so moves are real renames again (documented in HOOKS.md).

Nine commits because they were pushed through the API one file at a time; please squash-merge.

@Harrsn
Harrsn merged commit 78ce5a5 into main Sep 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant