Skip to content

fix(download): isolate temp files and reject duplicate download targets - #333

Open
Kaushik2003 wants to merge 1 commit into
agentrhq:mainfrom
Kaushik2003:fix/download-temp-file-collision
Open

fix(download): isolate temp files and reject duplicate download targets#333
Kaushik2003 wants to merge 1 commit into
agentrhq:mainfrom
Kaushik2003:fix/download-temp-file-collision

Conversation

@Kaushik2003

Copy link
Copy Markdown

Description

httpDownload wrote every attempt to ${destPath}.tmp. Because the download
pipeline step runs items in parallel (3 by default), two items whose filename
template — or generateFilename() fallback — resolves to the same name shared a
single temp file. Both opened a write stream on it, so the second truncated what
the first was still writing; both then renamed it to the destination. The result
was either a file blended from two downloads or, for whichever attempt renamed
second, an ENOENT failure. A failing attempt's cleanupTempFile() could also
delete the other download's in-flight file.

This PR makes two changes:

  1. src/download/index.ts — each attempt gets its own temp path via a new
    createTempPath() helper: ${destPath}.${pid}.${random}.tmp. The temp file
    stays beside the destination so the final rename remains same-filesystem and
    atomic. The naming mirrors the existing temp-write convention in
    src/hosted/files.ts:401 and src/site-memory/local-store.ts:195.

  2. src/pipeline/steps/download.ts — unique temp files stop the corruption
    but leave two items still racing for one destination, with the loser silently
    discarded. The step now resolves every item's URL and destination up front,
    lets the lowest-index item claim each path, and fails the rest with a named
    conflict rather than overwriting:

    Duplicate download target "report.pdf" (already claimed by item 0); give each item a unique filename
    

    The issue offered "serialized or reported as a conflict"; reporting was chosen
    because serializing still leaves last-writer-wins on one path, just without the
    interleaving — the user loses a download either way and never hears about it.
    The check runs after the skip_existing check, so re-runs against an already
    downloaded file keep skipping as before instead of turning into conflicts.

Resolving destinations up front also removed the duplicate render() passes over
the URL template in the yt-dlp cookie pre-scan — it now reads the resolved plans
instead of re-rendering every item twice.

Related issue: #

Type of Change

  • 🐛 Bug fix
  • ✨ New feature
  • 🌐 New site adapter
  • 📝 Documentation
  • ♻️ Refactor
  • 🔧 CI / build / tooling

Checklist

  • I ran the checks relevant to this PR
  • I updated tests or docs if needed
  • I included output or screenshots when useful

Checks run, per TESTING.md:

npm run typecheck   # clean
npm run build       # clean
npm test            # 442 files, 5651 passed, 2 skipped

Two regression tests were added, and both fail on main before the fix:

  • src/download/index.test.ts — "gives each concurrent download of the same
    destination its own temp file". Holds one response body open, starts a second
    download to the same path while the first still owns a temp file, and asserts
    both succeed, that the destination holds one whole body, and that no .tmp
    file is left behind. On main the first download fails with ENOENT on rename.
  • src/pipeline/steps/download.test.ts — "reports duplicate destinations instead
    of racing them onto one path". Three items with a fixed filename: report.pdf;
    asserts one download runs and the other two fail with the conflict message. On
    main all three download.

No documentation change: no page under docs/, skills/, or README.md
describes the pipeline download step or its filename behaviour. The step's
own header comment claimed "deduplication", which was not implemented; it now
reads "duplicate-target detection", which is what the code does.

Adapter Notes

  • Updated generated or lean docs when command discoverability changed
  • Used positional args for the command's primary subject unless a named flag is clearly better
  • Normalized expected adapter failures to CliError subclasses instead of raw Error

Screenshots / Output

Before — the two new tests against unmodified src/:

$ npx vitest run --project unit src/download/index.test.ts src/pipeline/steps/download.test.ts

 FAIL  |unit| src/download/index.test.ts > download helpers > gives each concurrent
                download of the same destination its own temp file
- Expected
+ Received
  {
-   "size": 9,
-   "success": true,
+   "error": "ENOENT: no such file or directory, rename
+             '/tmp/webcmd-dl-SV7Zt1/same-name.bin.tmp' -> '/tmp/webcmd-dl-SV7Zt1/same-name.bin'",
+   "size": 0,
+   "success": false,
  }

 FAIL  |unit| src/pipeline/steps/download.test.ts > stepDownload > reports duplicate
                destinations instead of racing them onto one path
AssertionError: expected "vi.fn()" to be called 1 times, but got 3 times

 Test Files  2 failed (2)
      Tests  2 failed | 13 passed (15)

After:

$ npx vitest run --project unit src/download/index.test.ts src/pipeline/steps/download.test.ts

 Test Files  2 passed (2)
      Tests  15 passed (15)

$ npm test

 Test Files  442 passed (442)
      Tests  5651 passed | 2 skipped (5653)
   Duration  40.68s

httpDownload wrote every attempt to `${destPath}.tmp`. The download
pipeline step runs items in parallel (3 by default), so two items whose
filename template or generateFilename() fallback produced the same name
shared one temp file: both write streams truncated and appended to it,
both renamed it to the destination, and one attempt's cleanupTempFile()
could delete the other's in-flight data. The surviving file was a blend
of two downloads, or the second rename failed with ENOENT.

Give each attempt its own temp path (pid + random suffix, beside the
destination so the rename stays atomic), matching the temp-write naming
already used in hosted/files.ts.

That stops the corruption but still leaves two items silently racing for
one path, so the step now resolves all destinations up front, lets the
lowest-index item claim each path, and fails the rest with a named
conflict instead of overwriting. Resolving up front also removes the
duplicate URL renders in the yt-dlp cookie pre-scan.
@github-actions

Copy link
Copy Markdown
Contributor

🟠 Maintainer review suggested — low confidence

The automated review could not reach a fully supported conclusion.

Limitations

  • The automated review returned an invalid structured result.

This review is advisory and does not block merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant