Description
While downloading, the file is written to a temporary path built only from the
destination:
// src/download/index.ts:119
const tempPath = `${destPath}.tmp`;
The download pipeline step runs several items at the same time (3 by default,
src/pipeline/steps/download.ts:131), and two items can easily end up with the
same filename — either because the filename template produces the same name,
or because generateFilename() falls back to something derived from the URL.
When that happens, both downloads write to the same .tmp file. Each one
opens its own fs.createWriteStream(tempPath), so the second truncates the file
the first is still filling, and then both rename it to the final name.
Two things go wrong, depending on the timing:
- The file that survives may hold bytes from two different downloads.
- Whichever attempt renames second fails with
ENOENT: no such file or directory, rename '.../file.bin.tmp' -> '.../file.bin',
because the other attempt already moved the shared temp file away.
The cleanup path makes it worse: if one download fails, cleanupTempFile()
(src/download/index.ts:121-127) deletes ${destPath}.tmp — which may be the
other download's file, still in progress.
Steps to Reproduce
- Write an adapter with a
download step whose filename template produces the
same name for more than one item, for example a fixed filename: report.pdf
with several items.
- Leave
concurrency at its default of 3.
- Run the command against two or more items with different URLs.
- Only one file is written, and its contents are not reliably from either URL.
The clobbering can also be seen directly against httpDownload, with no adapter
involved: start one download whose response body is held open, and while it is
still streaming, start a second download to the same destination path. The
second one truncates and renames the first one's temp file, and the first fails
on rename.
Expected Behavior
Each download attempt should use its own temporary file, so parallel downloads
never touch each other's data. A unique suffix would be enough, for example
${destPath}.${process.pid}-${counter}.tmp.
Downloads that genuinely target the same destination path should either be
serialized or reported as a conflict — not silently interleaved.
webcmd Version
0.7.1
Node.js Version
Other
Operating System
Linux
Logs / Screenshots
Description
While downloading, the file is written to a temporary path built only from the
destination:
The
downloadpipeline step runs several items at the same time (3 by default,src/pipeline/steps/download.ts:131), and two items can easily end up with thesame filename — either because the
filenametemplate produces the same name,or because
generateFilename()falls back to something derived from the URL.When that happens, both downloads write to the same
.tmpfile. Each oneopens its own
fs.createWriteStream(tempPath), so the second truncates the filethe first is still filling, and then both rename it to the final name.
Two things go wrong, depending on the timing:
ENOENT: no such file or directory, rename '.../file.bin.tmp' -> '.../file.bin',because the other attempt already moved the shared temp file away.
The cleanup path makes it worse: if one download fails,
cleanupTempFile()(
src/download/index.ts:121-127) deletes${destPath}.tmp— which may be theother download's file, still in progress.
Steps to Reproduce
downloadstep whosefilenametemplate produces thesame name for more than one item, for example a fixed
filename: report.pdfwith several items.
concurrencyat its default of 3.The clobbering can also be seen directly against
httpDownload, with no adapterinvolved: start one download whose response body is held open, and while it is
still streaming, start a second download to the same destination path. The
second one truncates and renames the first one's temp file, and the first fails
on rename.
Expected Behavior
Each download attempt should use its own temporary file, so parallel downloads
never touch each other's data. A unique suffix would be enough, for example
${destPath}.${process.pid}-${counter}.tmp.Downloads that genuinely target the same destination path should either be
serialized or reported as a conflict — not silently interleaved.
webcmd Version
0.7.1
Node.js Version
Other
Operating System
Linux
Logs / Screenshots