fix(download): isolate temp files and reject duplicate download targets - #333
Open
Kaushik2003 wants to merge 1 commit into
Open
fix(download): isolate temp files and reject duplicate download targets#333Kaushik2003 wants to merge 1 commit into
Kaushik2003 wants to merge 1 commit into
Conversation
httpDownload wrote every attempt to `${destPath}.tmp`. The download
pipeline step runs items in parallel (3 by default), so two items whose
filename template or generateFilename() fallback produced the same name
shared one temp file: both write streams truncated and appended to it,
both renamed it to the destination, and one attempt's cleanupTempFile()
could delete the other's in-flight data. The surviving file was a blend
of two downloads, or the second rename failed with ENOENT.
Give each attempt its own temp path (pid + random suffix, beside the
destination so the rename stays atomic), matching the temp-write naming
already used in hosted/files.ts.
That stops the corruption but still leaves two items silently racing for
one path, so the step now resolves all destinations up front, lets the
lowest-index item claim each path, and fails the rest with a named
conflict instead of overwriting. Resolving up front also removes the
duplicate URL renders in the yt-dlp cookie pre-scan.
Contributor
🟠 Maintainer review suggested — low confidenceThe automated review could not reach a fully supported conclusion. Limitations
This review is advisory and does not block merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
httpDownloadwrote every attempt to${destPath}.tmp. Because thedownloadpipeline step runs items in parallel (3 by default), two items whose
filenametemplate — or
generateFilename()fallback — resolves to the same name shared asingle temp file. Both opened a write stream on it, so the second truncated what
the first was still writing; both then renamed it to the destination. The result
was either a file blended from two downloads or, for whichever attempt renamed
second, an
ENOENTfailure. A failing attempt'scleanupTempFile()could alsodelete the other download's in-flight file.
This PR makes two changes:
src/download/index.ts— each attempt gets its own temp path via a newcreateTempPath()helper:${destPath}.${pid}.${random}.tmp. The temp filestays beside the destination so the final
renameremains same-filesystem andatomic. The naming mirrors the existing temp-write convention in
src/hosted/files.ts:401andsrc/site-memory/local-store.ts:195.src/pipeline/steps/download.ts— unique temp files stop the corruptionbut leave two items still racing for one destination, with the loser silently
discarded. The step now resolves every item's URL and destination up front,
lets the lowest-index item claim each path, and fails the rest with a named
conflict rather than overwriting:
The issue offered "serialized or reported as a conflict"; reporting was chosen
because serializing still leaves last-writer-wins on one path, just without the
interleaving — the user loses a download either way and never hears about it.
The check runs after the
skip_existingcheck, so re-runs against an alreadydownloaded file keep skipping as before instead of turning into conflicts.
Resolving destinations up front also removed the duplicate
render()passes overthe URL template in the yt-dlp cookie pre-scan — it now reads the resolved plans
instead of re-rendering every item twice.
Related issue: #
Type of Change
Checklist
Checks run, per
TESTING.md:Two regression tests were added, and both fail on
mainbefore the fix:src/download/index.test.ts— "gives each concurrent download of the samedestination its own temp file". Holds one response body open, starts a second
download to the same path while the first still owns a temp file, and asserts
both succeed, that the destination holds one whole body, and that no
.tmpfile is left behind. On
mainthe first download fails withENOENTon rename.src/pipeline/steps/download.test.ts— "reports duplicate destinations insteadof racing them onto one path". Three items with a fixed
filename: report.pdf;asserts one download runs and the other two fail with the conflict message. On
mainall three download.No documentation change: no page under
docs/,skills/, orREADME.mddescribes the pipeline
downloadstep or itsfilenamebehaviour. The step'sown header comment claimed "deduplication", which was not implemented; it now
reads "duplicate-target detection", which is what the code does.
Adapter Notes
CliErrorsubclasses instead of rawErrorScreenshots / Output
Before — the two new tests against unmodified
src/:After:
$ npx vitest run --project unit src/download/index.test.ts src/pipeline/steps/download.test.ts Test Files 2 passed (2) Tests 15 passed (15) $ npm test Test Files 442 passed (442) Tests 5651 passed | 2 skipped (5653) Duration 40.68s