Track intermittent CI test failures (code/CI/environment triage)

[hjmjohnson]

Important

These started as triage candidates, not confirmed bugs. A 30-day automated
review surfaced recurring intermittent CI failures as a starting point for
investigation. Each candidate could resolve to a code issue, a test
issue, or a CI/environment issue. As candidates are root-caused, this
issue is updated with the final disposition — see Resolution status below.
Several "looked numerically unstable" candidates turned out to have concrete
non-tolerance root causes; the term "flaky" is deliberately avoided for those.

The environment is often the root cause (e.g. timeouts under Valgrind), but
not always — see ShapeLabelMapFixture, which masqueraded as tolerance
instability and was in fact a sign-arbitrary algorithmic result.

Summary

A 30-day review (2026-05-25 → 06-25) of ITK CI failures across GitHub Actions
(ITK.Pixi, ITK.Arm64) and CDash (Azure DevOps + Kitware nightlies;
~18,920 failed-test rows, 676 distinct test names) found that most red CI is
CI-environment/infrastructure, with a small set of tests that fail
intermittently across unrelated commits and multiple platforms. This issue
tracks both so fixes/mitigations can be prioritized.

Methodology & how to read the tables

Failures were aggregated by test name × platform × commit/day. The signal that
separates a likely-recurring failure from a one-off breakage:

• Cross-commit (fails on several unrelated source revisions) and/or
• Cross-platform + multi-day (fails on Linux and Windows and macOS over
  many distinct days).

A test failing many times on a single day / single build is almost always a
transient breakage that was then fixed — those are excluded below.

Resolution status (updated as candidates are root-caused)

Candidate	Final classification	Disposition
ShapeLabelMapFixture	Two causes, both concrete (not tolerance instability). *_Direction cases: OBB origin is a corner of a sign-arbitrary eigenvector frame → two equally-correct answers; latent defect disabled since 2017 (0d9a5ed6), re-exposed (not introduced) by the VNL→Eigen backend swap #6475. Non-Direction cases: genuine FP-model scalar precision, Windows-MSVC-only.	In progress in #6521: full-precision baselines for the scalar group + invariant-geometry assertions (frame-invariant OBB center + vertex-set membership) for the sign-arbitrary group. Origin corner to be documented non-portable.
ParallelSparseFieldLevelSetRobustness.*	Real test/code issue, surfaced under Valgrind (per @blowekamp: Valgrind defects reproduce locally; not a deadlock — #6519 closed after 770+ runs showed zero hangs). Earlier deadlock hypothesis withdrawn.	Open: address the Valgrind-reported defects; the 300 s watchdog self-aborts under CI load are a secondary CI-load symptom.
itkComposeBigVectorImageFilterTest	CTest timeout under memory pressure (100-channel VectorImage, no explicit TIMEOUT, RESOURCE_LOCK MEMORY_SIZE).	Open: set an explicit generous TIMEOUT.
PythonLazyImportTime	Test-design assertion: 5000 ms wall-clock ceiling (load-sensitive) + 750-module ceiling (deterministic).	Open: raise/relax the wall-clock ceiling or make it non-blocking on loaded runners.
MaskedAssignFixture	Assertion failure (Completed (Failed)); remote-module test (*-Favorite-Remotes nightly), not in core tree.	Resolved: addressed by @blowekamp in #6542.
itkMeshFileReadWriteTest01	Originally hypothesized as round-trip comparison (VTK ASCII FP formatting/precision). 2026-07-10 evidence points earlier in the test: two windows-2022 Pixi-Cxx failures in 0.04 s with no diagnostic output, on unrelated PRs (#6569, #6577); one passed on rerun of an identical tree.	Open: capture the test's stderr in the Pixi workflow to disambiguate startup/first-read failure from FP precision; still intermittent as of 2026-07-10.
itkMCI_DSC_case_10_binary	New candidate 2026-07-10: SEGFAULT on macos-15 Pixi-Cxx (PR #6576, diff unrelated to the MorphologicalContourInterpolation remote module); sibling case_2 passed in 210 s on the same run. RUNS_LONG under a loaded runner fits the memory/load-pressure family, but one occurrence does not yet separate code from environment.	Open: track for cross-commit recurrence.

A. CI-environment / infrastructure failure modes (dominate red runs)

These are not test bugs; they prevent runs from producing a meaningful result.

Mode	Observed	Likely layer
ExternalData download failures (CID/tarball fetch)	~39% of red ITK.Pixi runs	CI/network/cache
Dashboard exit 255 "warnings-as-fatal" (0 real test failures)	~74% of red ITK.Arm64 runs	CI scripting (ci_completed_successfully)
Runner setup / apt-disk / tool-setup (exit 127)	several ITK.Pixi runs	CI/runner
One Windows nightly config red for many days (Windows11-VS22x64-RelWithDebInfo-Favorite-Remotes)	~32 tests × 15 days	single machine/config

B. Tests that fail intermittently across commits/platforms

These began as candidates; current understanding is in Resolution status above.
The original evidence is retained here for reference.

Candidate	Evidence
ShapeLabelMapFixture (5 3D_* cases)	187× / 30 days. Two modes: 3 non-Direction cases Windows-MSVC-only (105×); 2 _Direction cases cross-platform (Win+macOS+Linux).
ParallelSparseFieldLevelSetRobustness.SweepRepeat / .ConcurrentMultiPipeline	132× / 102× over 18 days; Linux+Win+macOS incl. TSan/Valgrind.
itkComposeBigVectorImageFilterTest	92× / 18 days; mostly coverage/memory-pressure builds.
PythonLazyImportTime	115× / 18 days; Win+Linux Python.
MaskedAssignFixture.SetGetPrint	276× / 18 days; concentrated in valgrind/coverage.
itkMeshFileReadWriteTest01	4 unrelated commits; ubuntu+windows. +2 windows-2022 Pixi occurrences 2026-07-10 (see comment thread); rerun-green on identical tree.

Explicitly excluded (single-day transients — NOT recurring)

ComputeImageSpectralDensityTest (06-12), GeneralizedEigenDecomposition.IdentityPencilMatchesStandard
(06-19), itkImageFileWriterStreamingTest1_2/_3 (06-14), and a 120-test mass
red on one Windows nightly (06-25) — each a single bad build/commit that was
fixed, not recurring.

Lesson learned

A test that flaps with the texture of numerical instability is not necessarily
a tolerance problem. When two backends can each return a different correct
answer (sign of an eigenvector, ordering of a degenerate decomposition), the
test is asserting an arbitrary representative — the fix is an invariant
assertion, not a looser bound. ShapeLabelMapFixture is the worked example:
every signal above the ITK test interface pointed at FP tolerance; the real
cause (algorithmic backend swap producing a sign-flipped but equally-correct
result) was only found by tracing below the test interface.

Suggested next steps

• Treat Section A as infrastructure work (ExternalData fetch retry/caching;
  reconsider warnings-as-fatal on ARM) — likely the biggest reliability payoff.
• Treat Section B per the Resolution status table; confirm/deny each
  before any code change.

Happy to break any individual candidate into its own focused issue.

[hjmjohnson]

Update — ParallelSparseFieldLevelSetRobustness.* is now root-caused and has a fix in #6519.

This candidate turned out to be a real deadlock in ParallelSparseFieldLevelSetImageFilter (not a test/environment flake): the apply-update phase blocks each work unit on its boundary neighbors (SignalNeighborsAndWait/WaitForNeighbor), which requires gang scheduling that ParallelizeArray does not guarantee — so a work-pool/TBB backend can leave a neighbor queued behind a blocked unit and deadlock. The runWithDeadline watchdog then aborts, surfacing as the intermittent CI failures recorded here.

Fix in #6519 dispatches only that one phase on a dedicated thread per work unit. The race is rare (380 oversubscribed local runs produced 0 hangs), so the mechanism was confirmed with a deterministic minimal reproducer (barrier on a bounded executor deadlocks; dedicated-thread gang completes); the filter's baseline output is bit-identical with the fix.

The other candidates in this issue remain open for triage.

[blowekamp]

There are two ParallelSparseFieldLevelSetRobustness tests failing on the nightly valgrind build due to time out.

I am not sure if there is an underlying problem with the code and the time out is due to a dead lock or if it just runs too slow.

[hjmjohnson]

Correction to my earlier comment on ParallelSparseFieldLevelSetRobustness.*. A decisive follow-up experiment did not confirm a filter deadlock, so #6519 has been closed (not merged).

Summary (full detail in #6519): the deadlock mechanism is provable in the abstract (a barrier on a bounded executor deadlocks; a dedicated-thread gang doesn't), but the filter itself completed in 770 robustness runs plus forced count > P probes on 32³ and 128³ images (up to 64 work units vs 16 TBB threads) — zero deadlocks. Its neighbor sync uses counting semaphores and skips empty boundaries, so it does not require gang scheduling in practice.

Most likely the intermittent failures here are runWithDeadline(300 s) watchdog timeouts under CI load, not deadlocks. Recommended direction: treat this candidate as a test-robustness / CI-load item (widen/relax the watchdog, shrink the work-unit sweep), not a filter bug. Re-classifying it accordingly under this issue.

[hjmjohnson]

Failure-mode breakdown for the Section B candidates (last 30 days, from the CDash details field — which distinguishes timeout/abort/assertion; the status field alone records all three as Failed).

Candidate	Fails	details mode	Class
itkComposeBigVectorImageFilterTest	33	33× Completed (Timeout)	CTest timeout
ParallelSparseFieldLevelSetRobustness	131	108× Subprocess aborted (300 s watchdog), 22× Failed, 1× SEGFAULT	deadline/watchdog (timeout-natured)
ShapeLabelMapFixture	187	187× Completed (Failed)	assertion (FP/ordering)
MaskedAssignFixture	99	99× Completed (Failed)	assertion
PythonLazyImportTime	56	56× Completed (Failed)	assertion (timing-threshold)
itkMeshFileReadWriteTest01	14	14× Completed (Failed)	comparison

Timeout count: 1 strict CTest timeout (itkComposeBigVectorImageFilterTest); 2 if the 300 s watchdog self-aborts on ParallelSparseFieldLevelSetRobustness are counted as timeout-natured. The other 4 are genuine assertion/comparison failures, not timeouts.

Corrections to my earlier hypotheses above: MaskedAssignFixture and PythonLazyImportTime are assertion failures (Completed (Failed)), not instrumentation/timing timeouts as I'd guessed.

Suggested triage split:

• Timeout / load → itkComposeBigVectorImageFilterTest (raise its CTest TIMEOUT) and ParallelSparseFieldLevelSetRobustness (watchdog-under-load; widen/relax the 300 s watchdog or shrink the work-unit sweep — see BUG: Fix ParallelSparseFieldLevelSet deadlock under work-pool backends #6519 analysis: not a confirmed deadlock).
• Assertion / nondeterminism (investigate the test/code) → ShapeLabelMapFixture (highest volume, 187), MaskedAssignFixture (99), PythonLazyImportTime (56, fragile wall-clock assertion), itkMeshFileReadWriteTest01 (14).

[hjmjohnson]

Catalog of the assertion/threshold types behind the non-timeout candidates (extracted from the test sources, so triage can see what each one checks and why it flaps).

Candidate	Threshold type	Value(s)	Why it can flap
ShapeLabelMapFixture	Absolute FP tolerance on computed geometry (EXPECT_NEAR / ITK_EXPECT_VECTOR_NEAR)	1e-4 on ~25 "resulting value" asserts: Perimeter, Centroid, EquivalentSphericalPerimeter/Radius, EquivalentEllipsoidDiameter, FeretDiameter, Flatness, Elongation, OrientedBoundingBoxOrigin (file also uses 1e-10 ×15, 1e-99/exact ×11)	Perimeter/moment/OBB estimation differs across platform & compiler in the 4th–5th significant digit; 1e-4 absolute on values like Perimeter≈28.39 is ~3.5e-6 relative — tight. Top-volume candidate (187).
PythonLazyImportTime	Wall-clock ceiling	MAX_CUMULATIVE_IMPORT_MS = 5000.0 (fails if import itk cumulative time > 5 s)	Slow/loaded CI runner pushes import past 5 s. Load-sensitive timing assertion.
PythonLazyImportTime	Module-count ceiling (not timing)	MAX_SYS_MODULES_AFTER_IMPORT_ITK = 750	Eager-loading regression — deterministic, not load-related.
itkComposeBigVectorImageFilterTest	CTest timeout + memory lock	no explicit TIMEOUT (CTest default) + RESOURCE_LOCK MEMORY_SIZE; test builds 100-channel VectorImage<unsigned char,3> of size³	Memory-bound; under contention it exceeds the default timeout → Completed (Timeout) (this is the 1 true CTest-timeout candidate).
itkMeshFileReadWriteTest01	Round-trip value comparison (no explicit numeric tolerance in source)	exact / format-sensitive	VTK ASCII FP formatting/precision on write→read round-trip.
MaskedAssignFixture	(remote-module test — not in the core ITK tree on this checkout; from the *-Favorite-Remotes nightly)	n/a here	Threshold not extractable from core tree.

Threshold-type summary

• Absolute FP tolerance too tight for cross-platform geometry → ShapeLabelMapFixture (1e-4, ~25 asserts). Most impactful fix: switch the "resulting value" asserts to a relative tolerance, or loosen to ~1e-3.
• Wall-clock ceiling → PythonLazyImportTime (5000 ms) — already commented as "intentionally loose… tightened over time"; raise or make non-blocking on loaded runners.
• CTest default timeout under memory pressure → itkComposeBigVectorImageFilterTest — set an explicit generous TIMEOUT.
• Exact round-trip comparison → itkMeshFileReadWriteTest01 — verify the writer/reader precision; may need a tolerance.

[hjmjohnson]

Draft fix for the top assertion candidate (ShapeLabelMapFixture, 187 fails) in #6521: converts the "resulting value" computed-geometry asserts from 1e-4 absolute to a magnitude-relative tolerance (max(1e-4, 1e-3·|expected|)). Loosen-only; verified/exact asserts unchanged.

[blowekamp]

Correction to my earlier comment on ParallelSparseFieldLevelSetRobustness.*. A decisive follow-up experiment did not confirm a filter deadlock, so #6519 has been closed (not merged).

Summary (full detail in #6519): the deadlock mechanism is provable in the abstract (a barrier on a bounded executor deadlocks; a dedicated-thread gang doesn't), but the filter itself completed in 770 robustness runs plus forced count > P probes on 32³ and 128³ images (up to 64 work units vs 16 TBB threads) — zero deadlocks. Its neighbor sync uses counting semaphores and skips empty boundaries, so it does not require gang scheduling in practice.

Most likely the intermittent failures here are runWithDeadline(300 s) watchdog timeouts under CI load, not deadlocks. Recommended direction: treat this candidate as a test-robustness / CI-load item (widen/relax the watchdog, shrink the work-unit sweep), not a filter bug. Re-classifying it accordingly under this issue.

@hjmjohnson This is partially in accurate. As I posted earlier the tests are failing in the valgrind builds. Running them locally, they do not time out locally. They do produce valgrid defects. There are real issues with the test and code.

[hjmjohnson]

Update — ShapeLabelMapFixture (top candidate, 187 fails) is now root-caused, and it was not the tolerance/numerical-instability flake it appeared to be.

The visible symptom — intermixed passing/failing runs that looked exactly like numerical instability against a tight tolerance boundary — was a red herring. It took a long time to track down precisely because every signal above the ITK test interface pointed at FP tolerance. Thanks to @blowekamp's historical knowledge of this code, it was eventually traced below the test interface to an algorithmic change, not a stability margin: the VNL→Eigen migration (#6475) swapped the eigensolver backend, and the two backends return equally-correct but sign-flipped principal axes. The oriented-bounding-box origin is one corner of that sign-arbitrary eigenvector frame, so it flips between two valid answers depending on which backend computed it — the box geometry is identical, only which corner is labeled "origin" differs.

This is a latent defect, not a regression: the affected assertions were disabled back in 2017 (commit 0d9a5ed6); #6475 merely re-exposed them. No numerical stability problem and no tolerance value would have fixed it — the answer is genuinely two valid solutions, and the test was asserting one arbitrary representative of them.

Two distinct failure modes under this one candidate

• Two *_Direction cases — cross-platform (Windows + macOS + Linux): the principal-axes / OBB-origin sign arbitrariness above. Not fixable by tolerance; needs an invariant assertion.
• Three non-Direction cases — Windows-MSVC-only (105×): a separate, genuine FP-model effect on rotation-invariant scalars (Perimeter, spherical metrics, Roundness). This is a precision/baseline issue and is addressed by regenerating baselines at full 17-digit double precision.

Fix direction (PR #6521)

Assert invariant geometry instead of the sign-arbitrary origin corner: vertex-set membership plus a new frame-invariant OBB center accessor (GetOrientedBoundingBoxCenter()), and document the origin corner as non-portable across solvers. Combined with the full-precision baselines for the Windows-only scalar group, both failure modes are resolved at root rather than papered over with a wider tolerance window.

The general lesson for this issue's triage: a test that flaps with the texture of numerical instability is not necessarily a tolerance problem. When two backends can each return a different correct answer (sign of an eigenvector, ordering of a degenerate decomposition), the test is asserting an arbitrary representative — the fix is an invariant assertion, not a looser bound.

[blowekamp]

@hjmjohnson I am working on the MaskedAssignFixture.SetGetPrint failure.

[blowekamp]

The MaskedAssignFixture was addressed as part of #6542

[hjmjohnson]

Fresh occurrence data from 2026-07-09/10 PR triage (PRs #6569, #6576–#6581): two new itkMeshFileReadWriteTest01 hits (both windows-2022 Pixi-Cxx, both cleared by rerun with no code change), and one new candidate — itkMCI_DSC_case_10_binary SEGFAULT on macos-15 Pixi-Cxx.

Occurrence details

itkMeshFileReadWriteTest01 (already tracked above; adds two windows-2022 data points)

When (UTC)	PR / head	Job	Note
2026-07-10 01:35	#6569 @ e7ba724	Pixi-Cxx windows-2022	failed in 0.04 s, no diagnostic output captured; 6174/6175 others passed
2026-07-10 03:36	#6577 @ 985e080	Pixi-Cxx windows-2022	same signature, unrelated diff (DisplacementField)

Both PRs touch nothing near Mesh IO. The #6569 occurrence passed on gh run rerun --failed with an identical tree (doc-only delta from the previously-green head), confirming intermittency rather than regression. The 0.04 s wall time suggests the failure happens at startup/first-read rather than in the FP round-trip comparison itself — possibly worth capturing the test's stderr in the Pixi workflow to disambiguate from the ASCII-precision hypothesis.

itkMCI_DSC_case_10_binary — new candidate (SEGFAULT)

When (UTC)	PR / head	Job	Note
2026-07-10 03:36	#6576 @ 5528d27	Pixi-Cxx macos-15	itkMCI_DSC_case_10_binary (SEGFAULT), label MorphologicalContourInterpolation RUNS_LONG; 6179/6180 others passed; sibling itkMCI_DSC_case_2_binary passed in 210 s on the same run

Single occurrence so far, on a PR whose diff (DisplacementField) cannot plausibly affect the MorphologicalContourInterpolation remote module. A segfault (not a comparison failure) in a RUNS_LONG test under a loaded runner fits the memory/load-pressure family already catalogued here, but one data point does not yet distinguish code vs environment — filing for cross-commit tracking.