Skip to content

Fix flaky timing tests that fail CI on unrelated PRs #929

Closed

Description

@kblok

Some timing tests fail in CI on pull requests that do not touch them. A rerun fixes each one. Each failure costs a rerun and hides real failures.

Tests seen failing:

  • E2E.Tests.ScreenshotAssertions.ToHaveScreenshotTests.Passes_a_not_whose_screen_differed_long_enough_when_its_last_capture_runs_past_the_deadline (failed on 4 PRs, 1 s)
  • E2E.Tests.TestFilters.AFilterPastTheDeadlineTests.Partial_warns_at_once_before_the_next_filter_is_called
  • ObservationTests.Takes_no_second_snapshot_for_a_capture_the_budget_abandoned
  • E2E.Mobile.Tests.Engine.ObservationTests.Ends_an_input_hung_on_its_device_command_at_its_budget_as_possibly_committed (503 ms)
  • E2E.Tests.Openai.CallbackServerTests.Survives_connections_reset_before_they_are_accepted and Ignores_a_method_other_than_GET_and_still_takes_a_later_code
  • Cancels_an_in_flight_readiness_probe(startup) and Spends_the_preflight_probe_from_startupTimeout...

Likely causes: real clocks, tight deadlines, thread-pool starvation under parallel test load, and port races.

Plan: find the cause of each, prove it by looping the test under CPU load, and make it deterministic without changing what it checks.

Tracker

Fix these in the issues below. Close this issue when all four are closed.

Activity

  1. kblok commented on Nov 2, 2026

    @kblok
    MemberAuthor

    PR #930 fixed five of these. Still flaky in CI, not fixed yet:

    • LocatorEngineResolveRetryContractTests.Ends_a_wait_whose_slow_last_resolve_found_nothing_as_LOCATOR_NOT_FOUND_not_a_timeout
    • LocatorEnginePerformContractTests.Never_sends_an_action_whose_target_resolved_past_the_deadline
    • CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budget
      Also untouched: Cancels_an_in_flight_readiness_probe(startup) and Spends_the_preflight_probe.... Note: E2ESession takes no TimeProvider, although Screen does.
  2. kblok commented on Nov 2, 2026

    @kblok
    MemberAuthor

    More timing flakes seen under load on PR #945's agent run (pass on rerun): ExpectPollTests.Accepts_an_asynchronous_read, DispatchLocatorActionTests.Fails_as_NOT_ACTIONABLE_when_the_click_opens_no_file_chooser, CoreTests.Replay_waits_for_an_end_state_that_shows_up_late.

  3. kblok commented on Nov 2, 2026

    @kblok
    MemberAuthor

    Another timing flake in CI on PR #943 (passes on rerun expected): E2E.Mobile.Tests.Engine.ObservationTests.Lets_a_slow_snapshot_run_for_the_whole_timeout_the_locate_gets (failed in 691 ms).

  4. kblok commented on Nov 3, 2026

    @kblok
    MemberAuthor

    Another timing flake in CI (PR #950, passes on rerun expected): E2E.Tests.ScreenshotAssertions.ToHaveScreenshotTests.Reports_the_mismatch_it_saw_when_the_deadline_cuts_off_a_later_capture_and_a_capture_that_hung_as_a_timeout (3 s).

  5. kblok commented on Nov 3, 2026

    @kblok
    MemberAuthor

    Flaked again on PR #962 CI (run 41634287195, unrelated to that PR): DispatchPointerActionTests.Partial_Stops_dispatching_moves_when_aborted_and_releases_the_mouse got EngineException: swipeTo at point timed out before the page answered where it expected OperationCanceledException (the 2 s action budget ran out before the test's abort on a slow runner). Same test fixed once in #950. Rerun once; if it keeps failing the test needs a larger budget or a fake clock.

  6. kblok commented on Nov 3, 2026

    @kblok
    MemberAuthor

    Flaked on PR #980 CI (run 41642798323, unrelated change): CdpSessionRecoveryTests.Counts_reconnect_time_against_the_operation_budget failed with EngineException: navigate to .../login: net::ERR_ABORTED (the test page navigation was aborted on a slow runner). Round 2 (#956) already touched this test. Rerun once.

  7. kblok commented on Nov 3, 2026

    @kblok
    MemberAuthor

    Triage of the last 30 failed CI runs. The flakes group into four causes, one issue each:

    Not flakes: run 41542494777 (move 0,0 extra in the pointer tests) was a real bug in that PR. ManagedProcess*, SharedAppProcesses, RunApps and AppPortAllocation fail only in the local sandbox (exit 137), not on CI, so no issue.

    This issue is now a tracker. Close it when the four issues are closed.

  8. self-assigned this
    on Nov 10, 2026
  9. kblok commented on Nov 10, 2026

    @kblok
    MemberAuthor

    All four tracked issues are closed: #1007, #1008 and #1009 earlier, and #1006 by PR #1533 (merged). Closing this tracker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions