You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix flaky timing tests that fail CI on unrelated PRs #929
E2E.Tests.Openai.CallbackServerTests.Survives_connections_reset_before_they_are_accepted and Ignores_a_method_other_than_GET_and_still_takes_a_later_code
Cancels_an_in_flight_readiness_probe(startup) and Spends_the_preflight_probe_from_startupTimeout...
Likely causes: real clocks, tight deadlines, thread-pool starvation under parallel test load, and port races.
Plan: find the cause of each, prove it by looping the test under CPU load, and make it deterministic without changing what it checks.
Tracker
Fix these in the issues below. Close this issue when all four are closed.
CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budget
Also untouched: Cancels_an_in_flight_readiness_probe(startup) and Spends_the_preflight_probe.... Note: E2ESession takes no TimeProvider, although Screen does.
More timing flakes seen under load on PR #945's agent run (pass on rerun): ExpectPollTests.Accepts_an_asynchronous_read, DispatchLocatorActionTests.Fails_as_NOT_ACTIONABLE_when_the_click_opens_no_file_chooser, CoreTests.Replay_waits_for_an_end_state_that_shows_up_late.
Another timing flake in CI on PR #943 (passes on rerun expected): E2E.Mobile.Tests.Engine.ObservationTests.Lets_a_slow_snapshot_run_for_the_whole_timeout_the_locate_gets (failed in 691 ms).
Another timing flake in CI (PR #950, passes on rerun expected): E2E.Tests.ScreenshotAssertions.ToHaveScreenshotTests.Reports_the_mismatch_it_saw_when_the_deadline_cuts_off_a_later_capture_and_a_capture_that_hung_as_a_timeout (3 s).
Flaked again on PR #962 CI (run 41634287195, unrelated to that PR): DispatchPointerActionTests.Partial_Stops_dispatching_moves_when_aborted_and_releases_the_mouse got EngineException: swipeTo at point timed out before the page answered where it expected OperationCanceledException (the 2 s action budget ran out before the test's abort on a slow runner). Same test fixed once in #950. Rerun once; if it keeps failing the test needs a larger budget or a fake clock.
Flaked on PR #980 CI (run 41642798323, unrelated change): CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budget failed with EngineException: navigate to .../login: net::ERR_ABORTED (the test page navigation was aborted on a slow runner). Round 2 (#956) already touched this test. Rerun once.
Not flakes: run 41542494777 (move 0,0 extra in the pointer tests) was a real bug in that PR. ManagedProcess*, SharedAppProcesses, RunApps and AppPortAllocation fail only in the local sandbox (exit 137), not on CI, so no issue.
This issue is now a tracker. Close it when the four issues are closed.
Activity
Make the flaky timing tests deterministic (#930)
kblok commentedon Nov 2, 2026
PR #930 fixed five of these. Still flaky in CI, not fixed yet:
LocatorEngineResolveRetryContractTests.Ends_a_wait_whose_slow_last_resolve_found_nothing_as_LOCATOR_NOT_FOUND_not_a_timeoutLocatorEnginePerformContractTests.Never_sends_an_action_whose_target_resolved_past_the_deadlineCdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budgetAlso untouched:
Cancels_an_in_flight_readiness_probe(startup)andSpends_the_preflight_probe.... Note:E2ESessiontakes noTimeProvider, althoughScreendoes.kblok commentedon Nov 2, 2026
More timing flakes seen under load on PR #945's agent run (pass on rerun):
ExpectPollTests.Accepts_an_asynchronous_read,DispatchLocatorActionTests.Fails_as_NOT_ACTIONABLE_when_the_click_opens_no_file_chooser,CoreTests.Replay_waits_for_an_end_state_that_shows_up_late.kblok commentedon Nov 2, 2026
Another timing flake in CI on PR #943 (passes on rerun expected):
E2E.Mobile.Tests.Engine.ObservationTests.Lets_a_slow_snapshot_run_for_the_whole_timeout_the_locate_gets(failed in 691 ms).kblok commentedon Nov 3, 2026
Another timing flake in CI (PR #950, passes on rerun expected):
E2E.Tests.ScreenshotAssertions.ToHaveScreenshotTests.Reports_the_mismatch_it_saw_when_the_deadline_cuts_off_a_later_capture_and_a_capture_that_hung_as_a_timeout(3 s).kblok commentedon Nov 3, 2026
Flaked again on PR #962 CI (run 41634287195, unrelated to that PR):
DispatchPointerActionTests.Partial_Stops_dispatching_moves_when_aborted_and_releases_the_mousegotEngineException: swipeTo at point timed out before the page answeredwhere it expectedOperationCanceledException(the 2 s action budget ran out before the test's abort on a slow runner). Same test fixed once in #950. Rerun once; if it keeps failing the test needs a larger budget or a fake clock.kblok commentedon Nov 3, 2026
Flaked on PR #980 CI (run 41642798323, unrelated change):
CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budgetfailed withEngineException: navigate to .../login: net::ERR_ABORTED(the test page navigation was aborted on a slow runner). Round 2 (#956) already touched this test. Rerun once.kblok commentedon Nov 3, 2026
Triage of the last 30 failed CI runs. The flakes group into four causes, one issue each:
TimeProvider(LocatorEngine contract tests first), gate the slow step.DispatchPointerActionTests.Partial_Stops_dispatching_moves_when_aborted..., the most frequent one. Fix: abort from inside the first dispatched move, huge budget.CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budget,net::ERR_ABORTEDon the setup navigation.TinySite.StartAsyncport race (Address already in use).Not flakes: run 41542494777 (
move 0,0extra in the pointer tests) was a real bug in that PR. ManagedProcess*, SharedAppProcesses, RunApps and AppPortAllocation fail only in the local sandbox (exit 137), not on CI, so no issue.This issue is now a tracker. Close it when the four issues are closed.
Run the mobile input-budget tests on a clock the test moves (#1129)
kblok commentedon Nov 10, 2026
All four tracked issues are closed: #1007, #1008 and #1009 earlier, and #1006 by PR #1533 (merged). Closing this tracker.