Skip to content

[Flaky] AOT Selftests: process dies mid-run at fixture 481/1459 with no TAP trailer and no not ok #1152

Description

What happened?

The AOT Selftests CI leg intermittently dies mid-run rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037 ExitTr_Removed, #990 Issue487_SV2*), both of which run the corpus to completion and fail on an assertion.

Observed on PR #1149 (job 98975322325):

failing run re-run, identical commit
fixtures started (# Running:) 481 1459
# Total failures: trailer absent # Total failures: 0
not ok lines none none
exit code 1 0

The last output is # Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22; the process dies between fixtures, before # Running: ThreadSafe_RapidBackgroundSetState is printed. In the passing run that fixture is index 481, immediately after ConnectedAnimation_OrphanOnlyPassDoesNotCrash at 480, so the ordering is the same and the run simply stops there.

Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries no diagnostic at all — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:

  1. the host really died between fixtures, or
  2. the host produced more output that was lost, since it ends its run with a teardown-free TerminateProcess (Residual: full selftest suite 0xC0000005 at final process exit (distinct from #647 cross-fixture poisoning) #680) and the job pipes stdout through Tee-Object, so anything still buffered at termination is discarded.

If it is (2), the same truncation could in principle drop a genuine not ok on a future run.

Not a watchdog kill: the death is 32.9s after the first fixture (20:34:30.749 → 20:35:03.638), far inside the suite budget, and none of the StartHangWatchdog / per-fixture timeout paths emit their diagnostics.

ThreadSafe_RapidBackgroundSetState is not in SelfTestRunner.DefaultAotSkipPatterns and is a plain registry entry (SelfTestFixtureRegistry.cs:571, :2278).

Steps to reproduce

Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.

  1. dotnet publish tests/Reactor.AppTests.Host -p:PublishAotInternal=true -p:Platform=x64 -r win-x64 -c Release -o <out>
  2. & <out>/Reactor.AppTests.Host.exe --self-test 2>&1 | Tee-Object -FilePath aot-selftest-output.tap
  3. Compare (Select-String '# Running: ' aot-selftest-output.tap).Count against 1459 and check for the # Total failures: trailer. A short count with no trailer is this bug.

A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.

Reactor version / commit

f390d8f (PR #1149; the crash is independent of that PR — its diff only touches tests/Reactor.PackagedTests/*.cs, which the AOT job never compiles, since Reactor.AppTests.Host.csproj has no reference to Reactor.PackagedTests)

Platform

x64

.NET SDK version

10.0.x (windows-latest GitHub-hosted runner)

Windows version

windows-latest GitHub-hosted runner

Windows App SDK version

As resolved by main at the commit above.

Logs / stack trace

# Running: ConnectedAnimation_OrphanOnlyPassDoesNotCrash
ok ConnectedAnimOrphanOnly_DestinationMounted
ok ConnectedAnimOrphanOnly_NothingStarted
ok ConnectedAnimOrphanOnly_SurvivesRepeatedRoundTrips
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22
##[error]Process completed with exit code 1.

(Expected next line, per the passing run: # Running: ThreadSafe_RapidBackgroundSetState.)

Confirmation

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions