Skip to content

fix: move resume manifests and probe temps out of the download dir - #553

Merged
Linuxuser67 merged 6 commits into
mainfrom
refactor/xdg-bookkeeping
Oct 27, 2026
Merged

Linuxuser67 merged 6 commits into
mainfrom
refactor/xdg-bookkeeping

Conversation

@Linuxuser67

Copy link
Copy Markdown
Owner

Problem

Grab was writing bookkeeping beside the finished download:

  • .{id}.manifest.json (resume manifest) — left behind after a live capture delivered, and never reclaimed for finished rows.
  • .grab-info-{id}.json / .grab-live-{id}.json (probe info JSONs, which can contain signed media URLs and headers) — the unified path had no Drop guard, and ResumePlan::Finished skipped cleanup.

A user's download folder should contain downloads, not application state.

Change

New locations

Data Location Mode
Resume manifest $XDG_STATE_HOME/grab/video/{id}-{dest-key}.manifest.json 0600 file, 0700 dirs
Probe info JSON $XDG_RUNTIME_DIR/grab/info-{id}.json / live-{id}.json 0600
Fallback when XDG_RUNTIME_DIR is unset state/grab/run 0600

dest-key is FNV-1a 64 hex over the canonicalized destination (lexical normalization when it does not exist yet), so symlinked spellings of the same folder converge and an item id reused against another destination cannot adopt the wrong parts.

Bookkeeping

  • VideoManifest gains dest_dir: Option<PathBuf> (#[serde(default)]), injected at the single write_manifest choke point so the startup sweep can find owned files without reversing the hash.
  • read_manifest resolves the state manifest first, then falls back to the legacy dest-dir names (canonical + three older spellings) for one upgrade cycle.
  • A shared InfoJsonGuard removes probe temps on every return path in both the unified and live runners.

Cleanup coverage

  • Startup: sweep_state_staging reclaims state manifests whose (dest, id) no longer has a live queue row (done rows are swept), drops unusable/corrupt ones, and clears stale runtime temps. sweep_dest_staging additionally removes old .grab-info-*/.grab-live-* files.
  • ResumePlan::Finished sweeps its spent staging/manifest before returning.
  • Successful live delivery sweeps after finalize; failure paths keep the manifest as ownership proof for a salvaged recording.
  • Row discard (clean_staging_files) also removes runtime temps and legacy dest info JSONs.

Payload (.part, .ytdl, final.*) stays in the destination directory: the final delivery is an atomic rename_noreplace, and staging is the destination.

Tests

  • New: private ancestor creation, destination-scoped manifests, state sweep keep/orphan behavior, row-discard temp cleanup.
  • Tightened: unified and live success tests now assert no .json remains in the destination (or its staging dir) and no manifest survives a completed download.
  • Backward-compat tests still pin legacy dest-dir manifests and the recording-preserving sweep.

Serial suite ordering is unchanged. GRAB_STATE_DIR/GRAB_RUNTIME_DIR are debug-only seams and are ignored in release builds.

Resume manifests lived beside the finished download as .{id}.manifest.json
and probe info JSONs as .grab-info-{id}.json/.grab-live-{id}.json, so a
user's Downloads folder collected Grab bookkeeping. Move them to XDG
locations:

- manifests: $XDG_STATE_HOME/grab/video/{id}-{dest-key}.manifest.json,
  keyed by a hash of the canonical destination so an id reused elsewhere
  cannot adopt another row's parts. VideoManifest records dest_dir at the
  single writer choke point so the startup sweep can find its files.
- probe temps: $XDG_RUNTIME_DIR/grab/, falling back to state/run. Both
  unified and live attempts share one Drop guard, so early returns cannot
  leak them.

Cleanup now covers the new locations at every ownership boundary:
startup sweeps reclaim orphan state manifests and private temps, ResumePlan
::Finished and successful live delivery sweep the spent manifest, and row
discard removes private temps. Legacy dest-dir manifests and old probe
names are still read and reclaimed for one upgrade cycle.

Payload (.part/.ytdl/final.*) stays in the destination for the atomic
rename.
- Exclude Done rows from the dest-dir sweep keep-set: a finished row has
  delivered, so legacy manifests beside its download are debris, not live
  state. This also matches the state-dir keep-set.
- Keep an unreadable or destination-less state manifest while any queued row
  claims its id, instead of deleting it before the keep check. A crash can
  leave a truncated manifest, and deleting it would discard a live row's
  resume state.
- Write manifests and probe JSONs to a sibling temp and rename into place, so
  an interrupted write never truncates a good manifest. Startup sweeps clear
  the temps.
- Reclaim from the manifest that was actually opened, not a path recomputed
  from the destination: a moved/symlinked dest no longer strands the file.
- Drop the unified probe JSON as soon as yt-dlp exits, instead of holding
  signed URLs through merge/remux.
@Linuxuser67

Copy link
Copy Markdown
Owner Author

Follow-up in 06acd6a addresses the review:

  • Finding 1 (legacy litter from finished rows): the dest-dir sweep keep-set now excludes Done rows, matching the state-dir keep-set. Legacy .{id}.manifest.json files for delivered rows are reclaimed on restore.
  • Finding 2 (crash can drop live resume state): an unreadable or destination-less state manifest is kept while any queued row still claims its id; only orphaned ones are removed. Manifest/probe writes now go to a sibling .tmp and are renamed into place, so an interrupted write cannot truncate a good manifest. Startup sweeps clear leftover temps.
  • Finding 3: the sweep deletes owned files from the manifest it opened (manifest_ownership_of) and removes that exact file, instead of recomputing dest_key from the destination.
  • Finding 4: corrected the code comment; legacy names stay readable until the fallback is dropped in a later release (they are no longer written).
  • Finding 5: the unified runner drops InfoJsonGuard as soon as yt-dlp exits, before merge/remux.
  • Finding 6: .ytdl is intentionally kept beside .part: it is yt-dlp's own resume bookkeeping and must share the payload's directory for atomic delivery; it is removed on success.
  • Finding 7: added a regression test for a queued row's unreadable manifest. The legacy-Done path and moved-canonical-path cases still lack direct tests; noting them as follow-ups.

Per review decision: keep no permanent compatibility path for manifests
left inside the destination dir. read_manifest now reads only the state
manifest, so a row started by an older version restarts instead of
resuming.

Legacy manifest files are still deleted by name at startup / row removal
(they are Grab bookkeeping), and old probe JSONs are still swept. Their
owned .part files are no longer reclaimable without a state manifest; the
row removal path still clears the JSONs.

Tests that used dest-dir manifests to pin ownership cleanup now write the
state manifest; the state sweep, prefix ownership, no-ownership survival,
bystander, and row-removal tests keep their coverage.
@Linuxuser67

Copy link
Copy Markdown
Owner Author

Update per review discussion: the legacy read fallback is removed in 7ff67c5.

  • read_manifest reads only $XDG_STATE_HOME/grab/video/.... A download started by an older version restarts instead of resuming.
  • Legacy manifest files are still deleted by name at startup and on row removal (they are Grab bookkeeping), and old .grab-info-* / .grab-live-* probe JSONs are still swept.
  • The .part files once owned by a legacy manifest are no longer reclaimable without a state manifest; that is the accepted cost of dropping compatibility.
  • Tests that used dest-dir manifests to pin ownership now write the state manifest.

@Linuxuser67
Linuxuser67 merged commit f0cac9d into main Oct 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant