Summary
test/e2e/flow.e2e.test.js's test "e2e: --flow-json never writes a temp flow file to disk" intermittently fails in CI (observed once so far, on the main-push Checks workflow after merging #18) with a false-positive "leftover temp dir" assertion, due to test isolation timing issues under node --test's default parallel-file execution.
Reproduction
Not reliably reproducible locally (3/3 local full-suite runs passed). Observed once in CI:
https://github.com/tbrandenburg/node-red-cli/actions/runs/34035223292/job/... (Checks / Tests step on the chore: release v0.2.12 push to main)
✖ e2e: --flow-json never writes a temp flow file to disk (1184.163048ms)
AssertionError [ERR_ASSERTION]: no leftover node-red-cli temp dirs/files expected
+ actual - expected
+ [
+ 'node-red-cli-nm-integration-kw3zQ0'
+ ]
- []
Root cause hypothesis
The test scans os.tmpdir() globally for any node-red-cli-*-prefixed entries before and after its own single CLI invocation (test/e2e/flow.e2e.test.js:228-242). Because node --test runs test files concurrently by default, a different, unrelated test file (test/integration/node-modules.integration.test.js, whose temp dir is prefixed node-red-cli-nm-integration-*) can be mid-run — having created but not yet cleaned up its own temp dir — at the exact moment this test captures its "after" snapshot. The check has no way to distinguish "a temp dir this test itself created" from "a temp dir some unrelated concurrently-running test created", so it flags the latter as a false positive.
Expected behavior
The test should only ever fail when the CLI invocation it itself made leaves behind a temp file/dir, regardless of what other test files are doing concurrently.
Actual behavior
The test can spuriously fail based on unrelated concurrent test files' temp-dir lifecycle, purely due to timing.
Proposed fix (not investigated in depth — for whoever picks this up)
- Track only temp entries whose names correlate to a marker unique to this specific test's own invocation (e.g. inject a per-test-run marker into
runCli's env/args if the CLI's temp-dir naming scheme allows it), rather than diffing the entire os.tmpdir() directory listing.
- Alternatively, force this specific test to run in isolation (e.g.
node --test --test-concurrency=1 just for this file, or Node's test() { concurrency: false } option scoped narrowly) if a more targeted fix isn't quick.
- Alternatively, only fail if the leftover dir's name matches a prefix this test's own code path could have produced (e.g. filter to names NOT containing
-nm-integration- or other known unrelated test-file prefixes) as a minimal stopgap.
Scope
Test-only change; no production code affected. Low priority — did not block the #17/#18 release (already merged/published as v0.2.12), flagged here since it's a false-positive CI flake that could confuse future release verification if it recurs.
Summary
test/e2e/flow.e2e.test.js's test"e2e: --flow-json never writes a temp flow file to disk"intermittently fails in CI (observed once so far, on themain-push Checks workflow after merging #18) with a false-positive "leftover temp dir" assertion, due to test isolation timing issues undernode --test's default parallel-file execution.Reproduction
Not reliably reproducible locally (3/3 local full-suite runs passed). Observed once in CI:
https://github.com/tbrandenburg/node-red-cli/actions/runs/34035223292/job/... (Checks / Tests step on the
chore: release v0.2.12push tomain)Root cause hypothesis
The test scans
os.tmpdir()globally for anynode-red-cli-*-prefixed entries before and after its own single CLI invocation (test/e2e/flow.e2e.test.js:228-242). Becausenode --testruns test files concurrently by default, a different, unrelated test file (test/integration/node-modules.integration.test.js, whose temp dir is prefixednode-red-cli-nm-integration-*) can be mid-run — having created but not yet cleaned up its own temp dir — at the exact moment this test captures its "after" snapshot. The check has no way to distinguish "a temp dir this test itself created" from "a temp dir some unrelated concurrently-running test created", so it flags the latter as a false positive.Expected behavior
The test should only ever fail when the CLI invocation it itself made leaves behind a temp file/dir, regardless of what other test files are doing concurrently.
Actual behavior
The test can spuriously fail based on unrelated concurrent test files' temp-dir lifecycle, purely due to timing.
Proposed fix (not investigated in depth — for whoever picks this up)
runCli's env/args if the CLI's temp-dir naming scheme allows it), rather than diffing the entireos.tmpdir()directory listing.node --test --test-concurrency=1just for this file, or Node'stest(){ concurrency: false }option scoped narrowly) if a more targeted fix isn't quick.-nm-integration-or other known unrelated test-file prefixes) as a minimal stopgap.Scope
Test-only change; no production code affected. Low priority — did not block the #17/#18 release (already merged/published as v0.2.12), flagged here since it's a false-positive CI flake that could confuse future release verification if it recurs.