Skip to content

Flaky e2e test: '--flow-json never writes a temp flow file to disk' fails under node --test's parallel file execution #19

Description

@tbrandenburg

Summary

test/e2e/flow.e2e.test.js's test "e2e: --flow-json never writes a temp flow file to disk" intermittently fails in CI (observed once so far, on the main-push Checks workflow after merging #18) with a false-positive "leftover temp dir" assertion, due to test isolation timing issues under node --test's default parallel-file execution.

Reproduction

Not reliably reproducible locally (3/3 local full-suite runs passed). Observed once in CI:
https://github.com/tbrandenburg/node-red-cli/actions/runs/34035223292/job/... (Checks / Tests step on the chore: release v0.2.12 push to main)

✖ e2e: --flow-json never writes a temp flow file to disk (1184.163048ms)
  AssertionError [ERR_ASSERTION]: no leftover node-red-cli temp dirs/files expected
  + actual - expected
  + [
  +   'node-red-cli-nm-integration-kw3zQ0'
  + ]
  - []

Root cause hypothesis

The test scans os.tmpdir() globally for any node-red-cli-*-prefixed entries before and after its own single CLI invocation (test/e2e/flow.e2e.test.js:228-242). Because node --test runs test files concurrently by default, a different, unrelated test file (test/integration/node-modules.integration.test.js, whose temp dir is prefixed node-red-cli-nm-integration-*) can be mid-run — having created but not yet cleaned up its own temp dir — at the exact moment this test captures its "after" snapshot. The check has no way to distinguish "a temp dir this test itself created" from "a temp dir some unrelated concurrently-running test created", so it flags the latter as a false positive.

Expected behavior

The test should only ever fail when the CLI invocation it itself made leaves behind a temp file/dir, regardless of what other test files are doing concurrently.

Actual behavior

The test can spuriously fail based on unrelated concurrent test files' temp-dir lifecycle, purely due to timing.

Proposed fix (not investigated in depth — for whoever picks this up)

  • Track only temp entries whose names correlate to a marker unique to this specific test's own invocation (e.g. inject a per-test-run marker into runCli's env/args if the CLI's temp-dir naming scheme allows it), rather than diffing the entire os.tmpdir() directory listing.
  • Alternatively, force this specific test to run in isolation (e.g. node --test --test-concurrency=1 just for this file, or Node's test() { concurrency: false } option scoped narrowly) if a more targeted fix isn't quick.
  • Alternatively, only fail if the leftover dir's name matches a prefix this test's own code path could have produced (e.g. filter to names NOT containing -nm-integration- or other known unrelated test-file prefixes) as a minimal stopgap.

Scope

Test-only change; no production code affected. Low priority — did not block the #17/#18 release (already merged/published as v0.2.12), flagged here since it's a false-positive CI flake that could confuse future release verification if it recurs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions