Flaky tests: Task.WhenAny + Assert.Same timing races fail intermittently in CI

[pedrosakuma]

Summary

Two tests failed intermittently in CI today (2026-08-29) while resolving Dependabot PRs (#237, #238, #243), each time with the same symptom: Assert.Same() Failure: Values are not the same instance. A re-run of the failed job fixed it in every case, confirming these are flaky rather than genuine regressions — the dependency bumps in those PRs were unrelated to the failing code paths.

Observed failures

1. B3.EntryPoint.Client.Tests.Fixp.KeepAliveSchedulerPeriodicTests.Start_WithBoundTransport_InvokesSendCallbackPeriodically

   • File: tests/B3.EntryPoint.Client.Tests/Fixp/KeepAliveSchedulerTests.cs:100
   • Seen in: PR Bump the xunit group with 1 update #243 run https://github.com/pedrosakuma/B3EntryPointClient/actions/runs/32327661354
   • Error: Assert.Same() Failure: Values are not the same instance

2. B3.EntryPoint.Conformance.Spec_4_7_Retransmit.ReconnectRetransmitTests.Reconnect_With_Persisted_State_Resumes_Outbound_SeqNum_After_Drop

   • File: tests/B3.EntryPoint.Conformance/Spec_4_7_Retransmit/ReconnectRetransmitTests.cs:165
   • Seen in: PR Bump Microsoft.Extensions.Diagnostics.Testing from 10.8.0 to 10.9.0 #237 run https://github.com/pedrosakuma/B3EntryPointClient/actions/runs/33254756293, and again in PR Bump Microsoft.Extensions.Hosting from 10.0.10 to 10.0.11 #238 run https://github.com/pedrosakuma/B3EntryPointClient/actions/runs/33255430764 (two separate occurrences within ~20 minutes)
   • Error: Assert.Same() Failure: Values are not the same instance

Root cause hypothesis

Both failures come from the same pattern:

var completed = await Task.WhenAny(signalTask, Task.Delay(TimeSpan.FromSeconds(N)));
Assert.Same(signalTask, completed);

Assert.Same(signalTask, completed) fails whenever the wall-clock Task.Delay wins the race instead of the expected event/tick task — i.e. the awaited condition didn't happen within the fixed timeout. This is inherently timing-sensitive and vulnerable to CI runner contention/slow scheduling (GitHub-hosted runners can have noisy-neighbor CPU throttling). The KeepAliveSchedulerTests case is especially tight: it waits for two 40ms scheduler ticks within a 5s budget, which should normally be generous, but any startup delay or thread-pool starvation can still exhaust it.

The same Task.WhenAny(...) + Assert.Same pattern (fixed timeout race) appears in at least 6 test files, so any of them could flake under the same conditions:

• tests/B3.EntryPoint.Client.Tests/ColdStartResumeTests.cs:267 (2s timeout)
• tests/B3.EntryPoint.Client.Tests/Fixp/KeepAliveSchedulerTests.cs:100 (5s timeout) — flaked
• tests/B3.EntryPoint.Conformance/Spec_4_7_Retransmit/ReconnectRetransmitTests.cs:165 (3s timeout) — flaked
• tests/B3.EntryPoint.Conformance/Spec_4_7_Retransmit/NotAppliedTests.cs:32 (3s timeout)
• tests/B3.EntryPoint.Conformance/Spec_4_7_Retransmit/RetransmitRejectTests.cs:35 (3s timeout)
• tests/B3.EntryPoint.Conformance/Spec_4_7_Retransmit/RetransmitTests.cs:38 (3s timeout)

Suggested follow-up

• Increase timeouts and/or add a small CI-only multiplier for these fixed-delay races, especially KeepAliveSchedulerTests (40ms interval may be too tight for hosted runners).
• Consider a shared test helper (e.g. await AssertCompletesAsync(task, timeout)) that produces a clearer failure message (including elapsed time) instead of a bare Assert.Same, to make future flakes easier to diagnose from CI logs alone.
• If flakiness continues, consider marking affected tests with a retry attribute or investigating scheduler/thread-pool warm-up costs on GitHub-hosted runners.

Impact

With repository auto-merge now enabled (see discussion in the session that produced this issue), these tests are required checks and can transiently block Dependabot (and other) PRs from merging automatically, requiring a manual CI re-run.