sort: never lose or overwrite data when filing releases - #3
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
Production runs
MEDIASORT_MODE=move+REMOVE_ON_COMPLETE=1, with/downloadsand/libraryas separate bind mounts, so everyrename()fails with EXDEV and every "move" is really copy-then-delete. In that setup the sorter could destroy data in several ways, all reproduced locally before fixing:[Grp] Show - 13 [1080p], absolute anime numbering). The hook took rc 0 as success and removed the torrent with its data.--force-recreatemid-copy left a truncated file where a good one used to be. It also overwrote regardless of quality, so a worse release could replace a better one.Title (Year).mkv.Sample/,Featurettes/and CD2 files overwrote each other in walk order..en.srt,.es.srtand.en.forced.srtall mapped to one sidecar name, and RARBGSubs/folders were never filed..iso/.bin/archive made a release a "game".faucet.classifynever imported in production. The hook runspython /app/faucet/sort.py, sosys.path[0]is the package dir andfrom faucet.classify import …failed silently. Game detection has been running on file extensions alone.What changed
Decisions: replace only if better, Plex extras folders, quarantine to
_failed/..<name>.<pid>.faucet-partialbeside the destination, are fsynced and size-checked, thenos.replaced in. Per-process partials mean a hook/sweep race produces one intact winner (the loser exits 2). Partials idle >30 min are reaped.library_filestable records resolution and cam status for every file the sorter places. The sorter strips quality tags from filenames, so this is the only record of what a file is.faucet/quality.pyis the shared, dependency-free ranking;library.pynow uses it too. Unknown quality on either side keeps the existing file.Featurettes/,Trailers/, …) or use-trailer-style suffixes. Suffix-only matching keeps Trailer Park Boys safe. CD1/CD2 stack as- pt1/- pt2; otherwise the largest file wins and the rest are quarantined.Subs/and per-episodeSubs/<stem>/layouts are filed.<release parent>/_failed/, which the sweep already excludes.QUARANTINE_DIRoverrides the location. Junk is now an explicit extension list (the old <5 MB rule classed small ISOs/RARs as junk). Deletion and quarantine refuse library roots and shallow paths.ep.mkvinside an episode folder was being filed asmovies/ep/.game. An existing game is quarantined, not deleted.hook.pyremoves the torrent only on 0/4 and records aquarantinedevent.sweep.pycounts 4 as swept.Testing
tests/test_sort_safety.py: 38 new regression tests covering F1/F2/F3/F4/F6/F9, the hook's removal contract, and the guard rails. One test runssort.pyexactly as the hook does (no PYTHONPATH) to lock in the import fix. Full suite: 144 passed (106 existing + 38) on 3.12; CI covers 3.10/3.11.Before/after deploying
library_filesstarts empty. Files Faucet sorted before this PR have no recorded quality, so a new release landing on one of those exact paths is quarantined, not swapped in. This is conservative by design. Upgrades that target a differently-named (tagged) file are unaffected._failed/every ~48h instead of overwriting. The upgrade/wants PR fixes the root cause; until then, check_failed/occasionally._failed/is created on first use under/downloads/complete.Follow-ups (not in this PR)
/mnt/nasonce for both paths so moves are real renames again (documented in HOOKS.md).Nine commits because they were pushed through the API one file at a time; please squash-merge.