Repository navigation
fix: move resume manifests and probe temps out of the download dir - #553
Merged
Merged
Conversation
Resume manifests lived beside the finished download as .{id}.manifest.json
and probe info JSONs as .grab-info-{id}.json/.grab-live-{id}.json, so a
user's Downloads folder collected Grab bookkeeping. Move them to XDG
locations:
- manifests: $XDG_STATE_HOME/grab/video/{id}-{dest-key}.manifest.json,
keyed by a hash of the canonical destination so an id reused elsewhere
cannot adopt another row's parts. VideoManifest records dest_dir at the
single writer choke point so the startup sweep can find its files.
- probe temps: $XDG_RUNTIME_DIR/grab/, falling back to state/run. Both
unified and live attempts share one Drop guard, so early returns cannot
leak them.
Cleanup now covers the new locations at every ownership boundary:
startup sweeps reclaim orphan state manifests and private temps, ResumePlan
::Finished and successful live delivery sweep the spent manifest, and row
discard removes private temps. Legacy dest-dir manifests and old probe
names are still read and reclaimed for one upgrade cycle.
Payload (.part/.ytdl/final.*) stays in the destination for the atomic
rename.
- Exclude Done rows from the dest-dir sweep keep-set: a finished row has delivered, so legacy manifests beside its download are debris, not live state. This also matches the state-dir keep-set. - Keep an unreadable or destination-less state manifest while any queued row claims its id, instead of deleting it before the keep check. A crash can leave a truncated manifest, and deleting it would discard a live row's resume state. - Write manifests and probe JSONs to a sibling temp and rename into place, so an interrupted write never truncates a good manifest. Startup sweeps clear the temps. - Reclaim from the manifest that was actually opened, not a path recomputed from the destination: a moved/symlinked dest no longer strands the file. - Drop the unified probe JSON as soon as yt-dlp exits, instead of holding signed URLs through merge/remux.
Owner
Author
|
Follow-up in 06acd6a addresses the review:
|
Per review decision: keep no permanent compatibility path for manifests left inside the destination dir. read_manifest now reads only the state manifest, so a row started by an older version restarts instead of resuming. Legacy manifest files are still deleted by name at startup / row removal (they are Grab bookkeeping), and old probe JSONs are still swept. Their owned .part files are no longer reclaimable without a state manifest; the row removal path still clears the JSONs. Tests that used dest-dir manifests to pin ownership cleanup now write the state manifest; the state sweep, prefix ownership, no-ownership survival, bystander, and row-removal tests keep their coverage.
Owner
Author
|
Update per review discussion: the legacy read fallback is removed in 7ff67c5.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Grab was writing bookkeeping beside the finished download:
.{id}.manifest.json(resume manifest) — left behind after a live capture delivered, and never reclaimed for finished rows..grab-info-{id}.json/.grab-live-{id}.json(probe info JSONs, which can contain signed media URLs and headers) — the unified path had no Drop guard, andResumePlan::Finishedskipped cleanup.A user's download folder should contain downloads, not application state.
Change
New locations
$XDG_STATE_HOME/grab/video/{id}-{dest-key}.manifest.json$XDG_RUNTIME_DIR/grab/info-{id}.json/live-{id}.jsonXDG_RUNTIME_DIRis unsetstate/grab/rundest-keyis FNV-1a 64 hex over the canonicalized destination (lexical normalization when it does not exist yet), so symlinked spellings of the same folder converge and an item id reused against another destination cannot adopt the wrong parts.Bookkeeping
VideoManifestgainsdest_dir: Option<PathBuf>(#[serde(default)]), injected at the singlewrite_manifestchoke point so the startup sweep can find owned files without reversing the hash.read_manifestresolves the state manifest first, then falls back to the legacy dest-dir names (canonical + three older spellings) for one upgrade cycle.InfoJsonGuardremoves probe temps on every return path in both the unified and live runners.Cleanup coverage
sweep_state_stagingreclaims state manifests whose(dest, id)no longer has a live queue row (done rows are swept), drops unusable/corrupt ones, and clears stale runtime temps.sweep_dest_stagingadditionally removes old.grab-info-*/.grab-live-*files.ResumePlan::Finishedsweeps its spent staging/manifest before returning.clean_staging_files) also removes runtime temps and legacy dest info JSONs.Payload (
.part,.ytdl,final.*) stays in the destination directory: the final delivery is an atomicrename_noreplace, and staging is the destination.Tests
.jsonremains in the destination (or its staging dir) and no manifest survives a completed download.Serial suite ordering is unchanged.
GRAB_STATE_DIR/GRAB_RUNTIME_DIRare debug-only seams and are ignored in release builds.