Skip to content

fix(run): wait for the runner's count before the next test under --max-failures - #1749

Open
bardrop wants to merge 3 commits into
e2e-inc:mainfrom
bardrop-oss:fix/max-failures-worker-wait
Open

bardrop wants to merge 3 commits into
e2e-inc:mainfrom
bardrop-oss:fix/max-failures-worker-wait

Conversation

@bardrop

@bardrop bardrop commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What and why

  • The scheduler is now the only owner of the failure count. RunUnitMessage.failureLimit is just a flag that the run has a limit; the snapshot ({ limit, failures }) and the worker's own countFailure arithmetic are gone.
  • While the run has a limit, a worker that reports a counted failure waits before its next test (TargetExecutorOptions.beforePair, also before each setup pair) until the runner answers. The answer is a new continue message when the run is still below the limit, or the existing interrupt with the failure-limit skip once it is reached. One round-trip per failure, no timing.
  • The worker counts the failure before it emits the result, because an in-process runner answers inside emit.
  • Patch changeset for e2e. No docs change: docs/reference/cli.mdx already describes the fixed behavior.

Issue

Fixes #1745

Verified

Ran it locally: yes

  • New 2-worker, maxFailures: 2 test in parallel-workers.test.ts: file a fails at once, file b fails 500 ms later, so b's worker trips the limit. It asserts the rest of the tripping file is skipped with failure-limit (no interrupted), and at most one interrupted overall. It fails 3 of 3 runs against a main build and passes 5 of 5 here.
main this branch
b next interrupted (started after the limit, then cancelled) b next and b last skipped, failure-limit
  • vitest run tests/unit: 3653 pass. vitest run tests/integration: 972 pass, 3 fail in cli-init.test.ts only because @e2e-dev/mobile was not built locally (same on main).
  • pnpm check green.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

View guided diff

@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile label Nov 13, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] automated review on behalf of @okwasniewski

Thanks! Shape matches the issue: scheduler owns the count, a worker with a counted failure waits for continue or interrupt, no timer, no extra round-trip without a limit. No deadlock found (every result reaches the reply, the stop path always broadcasts interrupt), retries only count the final result, serial groups and shards behave, runner and worker are always the same version. New test fails 3/3 on main and passes 7/7 here; the issue repro now ends 2 failed + failure-limit skips.

Must fix:

  1. packages/e2e/tests/integration/parallel-workers.test.ts:805 - nothing fast covers continue. Dropping the continue reply keeps the new test green (the second failure's interrupt releases the first worker); only runner-lifecycle catches it, by hanging to its 120 s timeout. A lost continue hangs the run forever. Add a 2-worker case below the limit (e.g. maxFailures: 3, both files fail once) asserting both next tests pass. Optional: a scheduler.test.ts unit for continue-below-limit / none-at-limit.
  2. packages/e2e/src/run/worker/session.ts:324 - the await this.failuresAnswered() in the setup loop is dead: a setup unit holds one setup pair and the count is 0 when a unit starts. If it ever did gate after an interrupt, runSetupUnit has no interrupt check and would report failed with 0 attempts. Drop it and keep the single gate in the executor.
  3. .changeset/max-failures-worker-wait.md:6 - second sentence is internals. Keep only the user-facing first sentence.

Nits:

  • scheduler.ts:928-933 repeats countsTowardFailureLimit next to report(). Have report() return whether the run continues and answer from that; the 3-line comment goes away.
  • execute.ts:140-145 - beforePair JSDoc describes the worker/runner protocol; keep it to "awaited before each pair starts, before the interrupt check".
  • session.ts:225 - Math.max(0, unanswered - 1) hides an unasked-for continue; a plain decrement is fine.
  • protocol.ts:95 - failureLimit as a boolean reads like a number; e.g. answersFailures.
  • New test picks the tripping file by attempt end time; a short note on why the 500 ms gap exists helps.

@bardrop

bardrop commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. All fixed in 60737c3:

Must fix

  1. New integration test below --max-failures, each worker gets the runner's answer to its failure and runs its next test: 2 workers, maxFailures: 3, each file fails once, and both next tests must pass. I also added a unit test in scheduler.test.ts. It checks for continue, continue below the limit, continue, interrupt at the limit, and nothing without a limit. I checked both by removing the continue send: the unit test fails, and the integration test hangs until its timeout.
  2. Removed the await this.failuresAnswered() from the setup loop. The executor's beforePair is now the only gate.
  3. The changeset keeps only the first, user-facing sentence.

Nits

  • report() now returns whether the result was a counted failure that the run continues after. The result handler is now if (this.report(result) && this.answersFailures) send continue, and the extra comment is gone.
  • The beforePair JSDoc is now just "awaited before each pair starts, before the interrupt check".
  • continue is now a plain decrement.
  • Renamed the flag to answersFailures (protocol, scheduler, session).
  • Added a note on the 500 ms gap in the first test: it orders the two failures, so the tripping file is the one whose failure ended second.

Unit 3654 passed, integration 976 passed, and pnpm check is green.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 7 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@okwasniewski

Copy link
Copy Markdown
Member

[claude-opus-5-5] automated review on behalf of @okwasniewski

Re-reviewed at 60737c3. All asks are done:

  • The continue path is now covered. Removing the continue send in the scheduler fails the new scheduler.test.ts unit right away, and the new 2-worker integration test fails at its 120 s timeout, so the hang is bounded.
  • The setup gate in session.ts is gone. beforePair is the only gate left, and that's safe because the scheduler answers a result before that worker's unit-done.
  • The changeset is user-facing only, and all 5 nits are handled. scheduler.test.ts passes 56/56 and pnpm check is green.

One question: packages/e2e/tests/integration/parallel-workers.test.ts:618,655,689,725 raises the timeouts of 4 tests this PR doesn't touch (blocked-worker, blocked-retry, watchdog-window, serial-member) from 60 s to 120 s. Here they run in 6.6 to 13 s. Please revert them, or say why they need it.

Labeling Ready for Human Review. The two open questions from the first review, about bounding how long a waiting worker can wait and about how setup after an interrupt is reported, are for a maintainer.

@bardrop

bardrop commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Reverted in 616c401: the four tests (blocked-worker, blocked-retry, watchdog-window, serial-member) are back to 60 s. I had no good reason for raising them; they aren't affected by this change. parallel-workers.test.ts passes 31/31 locally with the old values.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@okwasniewski

Copy link
Copy Markdown
Member

[claude-opus-5-5] automated review on behalf of @okwasniewski

Re-checked at 616c401. The only change is the revert of the four 120 s timeouts back to 60 s in parallel-workers.test.ts, as asked. CI is green. Labeling Ready for Human Review.

@bardrop

bardrop commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on main (517ecc6) to clear the conflict with #1750's step-hold changes in execute.ts and session.ts. No change to this PR's logic: heldScreens and beforePair now sit side by side. scheduler.test.ts and parallel-workers.test.ts pass, and so do the full e2e suite, typecheck and lint.

@bardrop
bardrop force-pushed the fix/max-failures-worker-wait branch from 517ecc6 to 5516411 Compare November 15, 2026 18:23
@bardrop

bardrop commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on main after #1736. The only conflict was in TargetExecutorOptions: debug moved to EngineSessionsOptions, so this branch now adds just beforePair there. The worker still passes it from worker/session.ts. Locally: scheduler + parallel-workers tests, the full e2e suite (4670 passed), typecheck and lint are green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L Up to 499 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--max-failures with several workers reports unstarted tests as interrupted instead of skipped

2 participants