Skip to content

test(core): de-flake the scheduler timing tests and the real-Docker container tests - #494

Merged
edspencer merged 3 commits into
mainfrom
test/deflake-scheduler-timing
Nov 14, 2026
Merged

edspencer merged 3 commits into
mainfrom
test/deflake-scheduler-timing

Conversation

@edspencer

@edspencer edspencer commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

De-flakes scheduler.test.ts, which has been turning CI red, including both attempts of the version PR #492.

The flake

Many tests here sleep for a fixed time, then assert the scheduler has already checked, triggered, run, or persisted state.yaml. On a busy runner that guess comes up short. Three tests failed this way on #492 and on e852a99. The first commit fixed those three, and CI promptly failed a fourth: triggers cron schedules when time is within trigger window, whose wait(500) had already been raised once "for slow CI runners". So the flake is the pattern, not any one test.

The scheduler logic itself is deterministic. Cron due-checks read the mockable Date.now(), and temp files are named randomly. What varies is how long start-up and file I/O take on the runner.

The fix

  • A waitUntil(predicate) helper polls every 10ms up to a 4s deadline. That's under vitest's 5s test timeout, so a real failure reports the helper's message rather than a bare "Test timed out".
  • Every sleep followed by a "something happened" assertion now polls: 24 in all. Assertions are unchanged.
  • Sleeps that prove an absence stay fixed, because you can't poll for something not happening: no second catch-up trigger, no re-trigger while running, nothing after stop, max-concurrency never exceeded. Each now starts after a waitUntil for the event it measures from, so the window is real rather than a guess.
  • The deadline uses performance.now(), not Date.now(). Several tests freeze Date.now to pin the cron clock, so a Date.now deadline would never expire.

Verification

  • scheduler.test.ts: 51/51 on 5 plain runs and 3 runs on one core shared with 8 busy-loops. biome check is clean.
  • Mutation check: with the cron trigger window disabled (if (false && …)), both window tests fail in 4s with waitUntil: condition not met within 4000ms. They neither hang nor pass, so polling hasn't made them toothless.
  • Caveat: I couldn't reproduce the original flake locally. The unchanged file passed under CPU contention and in three full-suite runs on 2 cores, because slowing the CPU slows the test's own timers in step. The changes only ever wait longer for the same conditions, so they can't introduce failures. The proof that the flake is gone is CI on this PR and chore: version packages #492.

Also: the real-Docker container tests

#492's rerun turned up a second flake. docker-security.test.ts › Container Configuration shells out to the real Docker daemon (docker create / inspect / rm), and two of its tests, sets memory limits and sets user to non-root, hit vitest's 5s default with the other 3772 tests green. One docker create on a cold runner can take that long by itself. That block now gets { timeout: 30_000 }; the pure-config tests elsewhere in the file keep the default. It passes 40/40 locally against a real daemon.

Test-only, so no changeset.

🤖 Generated with Claude Code

Three scheduler tests waited a fixed wait(300)/wait(100)/wait(50) and then
asserted the scheduler had already checked, triggered, run and persisted
state.yaml. On a loaded CI runner the guess is sometimes short: all three
failed on the #492 version PR (both attempts) and on e852a99, and
scheduler.test.ts failures recur in older CI runs too, while the behaviour
under test was fine.

Replace those sleeps with waitUntil(), which polls the same condition up to
a 5s deadline. The deadline uses performance.now(): two of these tests freeze
Date.now to pin the cron clock, so a Date.now deadline would never expire.
Assertions are unchanged; the tests only wait as long as they need to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 40 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1b5c689f-344a-4005-98e0-950aad3d456e
Commits

Reviewing files that changed from the base of the PR and between 82a4997 and 914dc39.

Files selected for processing: 2

  • packages/core/src/runner/runtime/__tests__/docker-security.test.ts
  • packages/core/src/scheduler/__tests__/scheduler.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Deploying herdctl with  Cloudflare Pages  Cloudflare Pages

Latest commit: 914dc39
Status: ✅  Deploy successful!
Preview URL: https://a235f531.herdctl.pages.dev
Branch Preview URL: https://test-deflake-scheduler-timin.herdctl.pages.dev

View logs

…happen

The first pass fixed the three tests seen failing on #492, and CI promptly
failed a fourth: "triggers cron schedules when time is within trigger
window", whose wait(500) had already been raised once "for slow CI runners".
The flake is the pattern, not those three tests: any fixed sleep followed by
an assertion that the scheduler got somewhere can lose on a busy runner.

Every positive-condition sleep in scheduler.test.ts now polls (21 more).
Sleeps that prove an ABSENCE (no second trigger, nothing after stop, max
concurrency never exceeded) stay fixed, now after a waitUntil for the event
they measure from. waitUntil's deadline drops to 4s, under vitest's 5s test
timeout, so a real failure reports the helper's message.

Mutation-checked: disabling the cron trigger window fails both window tests
in 4s with "waitUntil: condition not met"; they neither hang nor pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edspencer edspencer changed the title test(core): poll instead of sleeping in three scheduler timing tests test(core): poll instead of sleeping in scheduler tests that wait for something to happen Nov 14, 2026
docker-security.test.ts "Container Configuration" shells out to the real
Docker daemon (docker create / inspect / rm). On a cold or busy CI runner a
single `docker create` can exceed vitest's 5s default: "sets memory limits"
and "sets user to non-root" both timed out that way on #492, with the other
3772 tests green. Size the per-test timeout for real Docker on that block
only; the pure-config tests elsewhere in the file keep the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edspencer edspencer changed the title test(core): poll instead of sleeping in scheduler tests that wait for something to happen test(core): de-flake the scheduler timing tests and the real-Docker container tests Nov 14, 2026
@edspencer
edspencer merged commit 848fcb0 into main Nov 14, 2026
8 checks passed
@edspencer
edspencer deleted the test/deflake-scheduler-timing branch November 14, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant