Skip to content

CI: a UI shard's limit sums its tests' slowest passes over its longer list, plus the run's start and stop (gatefold#260) - #326

Open
arimendelow wants to merge 4 commits into
mainfrom
gf-260-shard-budget
Open

arimendelow wants to merge 4 commits into
mainfrom
gf-260-shard-budget

Conversation

@arimendelow

@arimendelow arimendelow commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #260

Change: each UI shard's time limit now comes from a checked file of per-test times, scripts/ci-shards.times, instead of hand-written sums. scripts/ci-shards.py check fails unless each shard's timeout is 30 + 2 x (its tests' slowest passes, summed over whichever of its main or PR lists runs longer) + 5 + 12, with a comment that shows that sum. It also fails when a test CI runs has no time in the file, and the error names the line to add.

Why the undo shards went red with every test passing. The 2026-11-08 to 11-12 history has 4,568 shard test steps from 500 App runs. Of those, 53 were stopped at their limit, across 18 shards (undo, place, rotate, taps, reviewer and Duo), so the limits were too tight for a whole class of shards, not just undo:

  • The PR list wasn't counted. ipad-undo-edge sized its limit on its main list (4.8 min), but on a pull request it runs a heavier list (The rotation test waits for the spread's page count in reads, not a fixed 45 s #318: 148 + 183 + 71 s).
  • Fixed costs came out of the doubled budget. ci-within stops 2 min before the step's end, and xcodebuild needs 72 s at the median and 159 s at p99 to start the first test and finish. A 9-min budget left about 6 min of tests, so "twice the slowest" was really about 1.4 times.
  • The sums were stale. They came from a quieter week, and the catch-alls had gained tests since (ipad-undo-rest "94" now runs three tests).

What I chose and why. I made the limit follow the tests rather than split shards:

  • Splitting adds macOS jobs to the 18 lanes we already queue on, and still leaves each budget hand-written.
  • Each time in the file is the 95th percentile of that test's passing or skipped times over those runs (load up to 800), so one freak run doesn't set it.
  • With "twice" on top, a shard's budget covers its tests at about 3x their median.

What changes.

  • No new jobs, and lanes are unchanged.
  • The undo limits go from 52/51 to 67/66 min, and most other shards rise by 1 to 24 min. Two that sat above their need come down to it (ipad-reviewer-rest 92 to 85, duo-reviewer-reopen 85 to 77).
  • A limit may now sit at most a minute over its need, so a hang fails as soon as a green run can't still be running.
  • The smoke shard that also runs a pull request's edited UI test classes no longer gets a flat 108 min. changes now works out that limit with ci-shards.py smoke from the same times: 65 min with no edited classes, 169 with RotationRepro. A class not in the file counts 10 min per test.

For PRs that add a UI test (#307, #322, #324): after merging main, add a device|Class/test<TAB>seconds line to scripts/ci-shards.times (an estimate until CI has run it). Then rerun python3 -I scripts/ci-shards.py check --expect scripts/ci-shards.expect, and it prints the timeout line for any shard that grew.

A hang still fails. The step's limit and ci-within's stop are unchanged in kind. Probe run: 43296671477, on probe-260-hang, not for merge. It runs only ipad-undo-edge, with its first test sleeping forever.

CI-only change: no-changelog.

…ed times file, over its longer list, plus the run's start and stop (gatefold#260)

The undo shards ran out of time with every test passing: the limit counted only the main list (ipad-undo-edge's PR
list runs longer), the hand-written sums were from a quiet week, and the 2 min ci-within grace and xcodebuild's
start and end came out of the tests' doubled budget. scripts/ci-shards.times now holds each test's 95th-percentile
pass (176 App runs, 2026-11-08 to 11-12), and scripts/ci-shards.py check fails when a shard's timeout or its comment
isn't 30 + 2 x <sum over its longer list> + 5 + 12, or when a test CI runs has no time.
@arimendelow arimendelow added the no-changelog An app change a tester would not notice: no changelog fragment needed label Nov 12, 2026
…a limit sits at most a minute over its need, and a missing time is one round to fix (gatefold#260 review)
…list run keeps the shard's own limit (gatefold#260 review)
@arimendelow
arimendelow enabled auto-merge (squash) November 12, 2026 20:11
@arimendelow

Copy link
Copy Markdown
Contributor Author

Hang probe: run 43296671477, job 131456778736. The probe ran ipad-undo-edge alone at its new 67-min limit, with its first test sleeping forever. It got a 25-min test budget after 10 min of setup, ci-within stopped it 23 min 59 s in, and the job ended red with exit 124 after its logs were written (job 35 min). A hang still fails.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog An app change a tester would not notice: no changelog fragment needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI undo: ipad-undo-edge shard ran out of time in testATouchOnTheScrubberNeverJumps (random)

1 participant