Repository navigation
CI: a UI shard's limit sums its tests' slowest passes over its longer list, plus the run's start and stop (gatefold#260) - #326
Open
arimendelow wants to merge 4 commits into
Open
arimendelow wants to merge 4 commits into
arimendelow wants to merge 4 commits into
Conversation
…ed times file, over its longer list, plus the run's start and stop (gatefold#260) The undo shards ran out of time with every test passing: the limit counted only the main list (ipad-undo-edge's PR list runs longer), the hand-written sums were from a quiet week, and the 2 min ci-within grace and xcodebuild's start and end came out of the tests' doubled budget. scripts/ci-shards.times now holds each test's 95th-percentile pass (176 App runs, 2026-11-08 to 11-12), and scripts/ci-shards.py check fails when a shard's timeout or its comment isn't 30 + 2 x <sum over its longer list> + 5 + 12, or when a test CI runs has no time.
…a limit sits at most a minute over its need, and a missing time is one round to fix (gatefold#260 review)
…list run keeps the shard's own limit (gatefold#260 review)
arimendelow
enabled auto-merge (squash)
November 12, 2026 20:11
Contributor
Author
|
Hang probe: run 43296671477, job 131456778736. The probe ran |
…and ipad-undo-rest its limit
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #260
Change: each UI shard's time limit now comes from a checked file of per-test times,
scripts/ci-shards.times, instead of hand-written sums.scripts/ci-shards.py checkfails unless each shard'stimeoutis 30 + 2 x (its tests' slowest passes, summed over whichever of its main or PR lists runs longer) + 5 + 12, with a comment that shows that sum. It also fails when a test CI runs has no time in the file, and the error names the line to add.Why the undo shards went red with every test passing. The 2026-11-08 to 11-12 history has 4,568 shard test steps from 500 App runs. Of those, 53 were stopped at their limit, across 18 shards (undo, place, rotate, taps, reviewer and Duo), so the limits were too tight for a whole class of shards, not just undo:
ipad-undo-edgesized its limit on its main list (4.8 min), but on a pull request it runs a heavier list (The rotation test waits for the spread's page count in reads, not a fixed 45 s #318: 148 + 183 + 71 s).ci-withinstops 2 min before the step's end, and xcodebuild needs 72 s at the median and 159 s at p99 to start the first test and finish. A 9-min budget left about 6 min of tests, so "twice the slowest" was really about 1.4 times.ipad-undo-rest"94" now runs three tests).What I chose and why. I made the limit follow the tests rather than split shards:
What changes.
ipad-reviewer-rest92 to 85,duo-reviewer-reopen85 to 77).changesnow works out that limit withci-shards.py smokefrom the same times: 65 min with no edited classes, 169 with RotationRepro. A class not in the file counts 10 min per test.For PRs that add a UI test (#307, #322, #324): after merging main, add a
device|Class/test<TAB>secondsline toscripts/ci-shards.times(an estimate until CI has run it). Then rerunpython3 -I scripts/ci-shards.py check --expect scripts/ci-shards.expect, and it prints the timeout line for any shard that grew.A hang still fails. The step's limit and ci-within's stop are unchanged in kind. Probe run: 43296671477, on
probe-260-hang, not for merge. It runs onlyipad-undo-edge, with its first test sleeping forever.CI-only change:
no-changelog.