Skip to content

ci: a cancelled test job still reads as a failure — #1993's second acceptance item was answered with headroom only #2076

Description

@lex00

#1993 had two acceptance items. The first (headroom that does not depend on runner variance) was met by raising timeout-minutes from 10 to 20 in PRs #2008 and #2012. The second was: "A cancelled job is distinguishable from a failed one in the PR checks summary, or cannot occur." Nothing addressed it, and the growth curve #1993 described continues.

Test job durations on the last four green main runs (2026-09-02 to 2026-09-03):

run test job
33668723471 11m13s
33669978619 8m08s
33703664206 10m22s
33704689614 10m18s

The suite already runs past the old 10-minute ceiling on a normal day. At the growth rate since #1993 was filed (9m50s on 2026-08-29), the 20-minute ceiling is a matter of months, and when it is hit the job will again show as a red X that reviewers cannot tell from a real failure without opening it.

Ask

One of:

  • Shard test across a matrix of parallel jobs (vitest --shard), so wall time stops tracking suite size and each shard has real headroom. ci: the test job repeatedly hits its 10m ceiling and is cancelled, producing false red PRs #1993 listed this as the option that addresses the curve rather than the symptom.
  • Or make a timeout distinguishable: a final step with if: cancelled() that posts a commit status or check named test (timed out) so the checks summary says what happened, and a continue-on-error shape that keeps the cancelled job from masquerading as a test failure.

Acceptance: the second item from #1993 holds. A deliberately induced timeout (temporary timeout-minutes: 1 on a branch) shows in the checks summary as a timeout, not a failure, or the sharded jobs each finish under half their ceiling across five consecutive runs.

Follow-up to #1993 / PRs #2008, #2012. Related: the vitest onTaskUpdate flake in #1693, which is a different red with the same reviewer cost.

🤖 Claude Code — https://claude.ai/code/session_015HFrNGLJa12qojpfwbDwyD

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciCI, release, publishenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions