Skip to content

Refresh the test durations the backend shards split by - #591

Closed
Babissimo wants to merge 1 commit into
mainfrom
ci/refresh-test-durations
Closed

Babissimo wants to merge 1 commit into
mainfrom
ci/refresh-test-durations

Conversation

@Babissimo

Copy link
Copy Markdown
Contributor

Refreshes backend/.test_durations, which decides where pytest-split cuts the backend suite into CI's three shards.

ClickUp: Cut the backend suite's per-process fixed cost.

Why

The file was last written at #547, before #569 and about 200 new tests. Group 3 has run lighter than the other two since then: on a warm bench (run 35881134795) the three groups took 39.6, 42.0 and 34.8 s at -n 4, and CI's pytest steps show the same order.

How it was recorded

On a CI runner (run 35890081094, branch bench/shard-workers, on 1a88f7b): serially, so no xdist worker's first test absorbs the session's fixture setup; under coverage, as the shards run; with --clean-durations, so removed tests drop out. 4,862 entries, 221 s in all. The old file summed to 529 s, recorded under different conditions.

Expected effect

Costing today's collection with the new durations, the old file's slowest third is 4.8% over an even split (76.1 / 77.2 / 67.6 s serial); the new file cuts all three at 73.6 s. On CI's ~25 s of divisible test time per shard, that is roughly 1.5–3 s off the slowest shard. Small, and it costs nothing at run time.

A new grouping also changes which tests share a process, which this suite is sensitive to (see the comment above the pytest step in ci.yml), so each of the three new groups was run locally exactly as CI runs it.

Verification

  • On 1a88f7b, the file's keys matched the collection exactly (4,862, none missing, no stale keys, nothing negative or implausible), and each of the three new groups run locally at -n 4 --dist worksteal --splits 3 --group k, as the shards do, exited 0 (1,509 / 1,263 / 2,090 tests).
  • Rebased onto 79615af, whose node-tracks stack adds 130 tests (2.6%) the file has not seen; pytest-split costs those at the mean. This PR's own run exercises the resulting groups.
  • pre-commit run --all-files clean.
  • Not put through /code-review: the diff is 9,500 lines of recorded durations, so it was checked by the properties above instead.

🤖 Generated with Claude Code

The file was last written at #547, before #569 and about 200 new tests, and group 3 has run lighter than the other two since: 34.8 s against 39.6 and 42.0 s at -n 4 on a warm CI bench. Costed by fresh durations, the old file's slowest third is 4.8% over an even split and the new one cuts all three at 73.6 s serial, roughly 1.5-3 s off the slowest shard.

Recorded on a CI runner (run 35890081094) rather than a laptop: serially, so no xdist worker's first test absorbs the session fixture setup; under coverage, as the shards run; and with --clean-durations so removed tests drop out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Claude finished @Babissimo's task in 1m 29s —— View job


Claude's review of #591

  • Gather context (PR body, CLAUDE.md, ONBOARDING.md)
  • Inspect the changed file (backend/.test_durations)
  • Check consistency with pytest-split config and CI shard setup
  • Verify claims in PR description where feasible
  • Post review feedback

This is a data-only refresh of backend/.test_durations, and the checks I could run locally line up with the PR description's claims:

  • Valid JSON, matches claimed shape. 4,862 entries, no negative durations, no duplicate keys (raw key-line count matches jq's parsed length exactly, so the one apparent "duplicate" from a naive grep was just a parametrized test ID containing an embedded quoted string).
  • Sum matches. Total is ≈220.9s, matching the PR's "221 s in all."
  • Matches CI's shard mechanics. .github/workflows/ci.yml confirms 3 shards (matrix.shard: [1,2,3]) and the pytest step (ci.yml:176) uses --splits/--group against this file, with the working directory correctly set to backend so the relative path resolves. The comment above that step (ci.yml:162-169) explicitly notes: "A stale durations file costs balance, not correctness; it only moves the two boundaries" — so this is inherently low-risk even if some entries were off.
  • Pre-commit scope. .pre-commit-config.yaml has no hook (e.g. end-of-file-fixer, JSON formatter) that touches this file, so the "clean pre-commit run --all-files" claim is unsurprising and doesn't tell us much either way — not a concern, just noting it's not a strong verification signal on its own.

Not independently verified: I didn't install backend dependencies to run pytest --collect-only and diff keys against actual collection, so I can't independently confirm "no missing/stale keys" or the exact reported per-shard timings/group sizes (1,509/1,263/2,090 tests) — that would require reproducing the recording environment, which the PR author already did on a CI runner and documented in the description. Given this is a pure generated-data file with the failure mode explicitly scoped to "moves the boundaries, not correctness," I don't see anything to flag.

No issues found. Nothing to fix here.

Babissimo added a commit that referenced this pull request Sep 23, 2026
PR #591's serially recorded durations cut the three groups evenly on paper, but its first run still had group 3 lightest. Serial durations miss contention: the runners have two cores with two threads each, so CPU-bound tests slow under four workers and I/O-bound ones do not. This records durations on the xdist controller under -n 4, then times each group as cut by the current, serial and parallel files on the same runner, six times, rotated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Babissimo

Copy link
Copy Markdown
Contributor Author

Closing: measured, this file does not deliver the balance the description predicts.

Paired on the same runner (bench runs 35896688206 and 35898020598, branch bench/shard-workers, six runners each), timing each group at -n 4 as the shards run:

durations file groups (median wall) slowest group vs today's file
today's (#547) 41.0 / 42.9 / 34.0 s –
this PR's (recorded serially) 41.2 / 42.2 / 36.3 s −0.3 s
recorded under -n 4 40.4 / 39.1 / 40.0 s −2.2 s, then −2.4 s on a repeat

Serial durations miss contention. The runners have two cores with two threads each, so CPU-bound tests slow under four workers and I/O-bound ones do not, and a split that is even serially stays uneven at -n 4.

The -n 4 file does balance the groups, but for about 2.3 s. Its group 3 also failed once in 13 runs against none in about 90 for today's split, and that run printed no test name, so a grouping-induced failure is not ruled out. Neither seemed worth a new recording recipe. The file is pinned at .github/bench/durations-parallel.json on bench/shard-workers (78f9592) if the gate comes to matter more.

Separately, ONBOARDING says --store-durations under -n records only one worker's tests. With pytest-split 0.11 it recorded every test, since the controller writes last, but each worker's first test absorbs the session's fixture setup.

@Babissimo Babissimo closed this Sep 23, 2026
@Babissimo
Babissimo deleted the ci/refresh-test-durations branch September 24, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant