Skip to content

Shard the dashboard's tests two ways - #579

Merged
Babissimo merged 1 commit into
mainfrom
ci/shard-dashboard-tests
Sep 24, 2026
Merged

Babissimo merged 1 commit into
mainfrom
ci/shard-dashboard-tests

Conversation

@Babissimo

Copy link
Copy Markdown
Contributor

Splits Web (dashboard tests) into two vitest --shard entries, so the dashboard stops sitting level with the backend path and further backend savings can reach the gate.

ClickUp: Shard the dashboard's tests two ways (split out of Cut the backend suite's per-process fixed cost).

Why now

Since #569 the dashboard's tests entry has been the backend path's equal. Queue-free job durations on CI runs since #549 split it out:

before #569 (n=36) after #569 (n=7)
backend path (slowest shard + backend-coverage) 98 s 92 s
Web (dashboard tests) 91 s 88 s

The dashboard entry was the longer of the two in 2 of the 7 post-#569 runs, and it set the gate outright on main run 35876349605 (107 s against backend-coverage's 102 s). Its test step is a 75 s median; checkout, setup-node and npm ci add about 11 s.

Why a shard divides here

A vitest shard loads only its own files, unlike a pytest-split group, which collects the whole suite and discards the rest. At three workers locally the whole suite takes 100 s and a third 37–41 s, so about 8 s is fixed. The two shards partition the suite exactly (58 files each; 497 + 857 = 1,354 tests, the whole suite's count) and split its CPU time about evenly (roughly 245 s each). Vitest assigns files by a hash of their relative path, so the split is the same on every machine.

Expected: each entry's test step near 40 s and its job near 55 s, about 35 s under the backend path. To be confirmed on this PR's run.

The pinning test

backend/tests/test_web_build_matrix.py pinned each workspace script to exactly one entry. It now accepts a script that runs once per shard, provided the shards are 1..N of N once each, and it checks that the test step hands each entry its shard after npm's --. Without the --, npm keeps --shard as its own config (a warning), and both entries would run the whole suite while staying green.

Nine mutations each turn it red: dropping a shard, duplicating one, shards of 3 across two entries, the step dropping the shard, the step dropping --, one entry unsharded, a label without its shard, a malformed shard, and a shard on the checks entry. npm run test in place of npm test stays green.

Verification

  • pre-commit run --all-files clean.
  • Full backend suite with CI's flags (-m "not external" -n 4 --dist worksteal): 4,861 passed, 2 skipped, exit 0.
  • /code-review at xhigh: the -- gap above and a stale ONBOARDING line were fixed; the rest needed no change.

🤖 Generated with Claude Code

Since #569 shortened the backend path, the dashboard's tests entry has been its equal: on CI runs since then its job took a median 88 s against 92 s for the slowest backend shard plus backend-coverage, queue-free, and it was the longer of the two in 2 of 7 runs. It set the gate outright on main run 35876349605. So no further backend saving reaches the gate while this entry sits beside it.

A vitest shard loads only its own files, so unlike pytest's the split divides nearly all of the suite's time (about 8 s fixed against 92 s divisible, measured at three workers). Two shards partition the suite exactly, 58 files each and 497 + 857 = 1354 tests, and split its CPU time about evenly, which should put each entry near 55 s.

test_web_build_matrix.py pinned each workspace script to exactly one entry. It now accepts a script that runs once per shard, provided the shards are 1..N of N exactly once each, and checks that the test step hands each entry its shard after npm's --. Without the separator npm keeps --shard as its own config and both entries run the whole suite, green. Nine mutations of the shard list, the step and the labels each turn it red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

This comment has been minimized.

@claude

claude Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Claude finished @Babissimo's task in 1m 57s —— View job


Review: Shard the dashboard's tests two ways

  • Gather context (PR diff, CLAUDE.md, ONBOARDING.md)
  • Review the diff (.github/workflows/ci.yml, ONBOARDING.md, backend/tests/test_web_build_matrix.py)
  • Post review findings

No correctness issues found. This is a tight, self-contained change and the diff matches its own description closely.

What I checked:

  • ci.yml: the matrix now has dashboard tests 1/2 and dashboard tests 2/2 entries, and the test step passes -- --shard={0} only when matrix.shard is set (format('-- --shard={0}', matrix.shard)), leaving the -- in place so npm forwards --shard to vitest instead of swallowing it as its own flag — this is the exact gap the PR description calls out as fixed.
  • vitest version: dashboard/package.json pins vitest: ^4.1.4, which supports --shard. dashboard/package.json's test script is plain vitest run, so -- --shard=1/2 composes correctly into vitest run --shard=1/2.
  • File count sanity check: find dashboard/src -iname "*.test.*" -o -iname "*.spec.*" returns 116 files, matching the PR's claimed 58+58 split.
  • test_web_build_matrix.py: traced _shard, test_each_script_a_workspace_defines_runs_once_or_once_per_shard, and test_the_test_step_hands_each_entry_its_shard by hand against the matrix in ci.yml:
    • Unsharded workspaces (packages/shared, e2e) still take the shards == [None] fast path.
    • The dashboard's test script now produces shards = [(1,2), (2,2)], which satisfies sorted(shards) == [(1,2),(2,2)] — this correctly rejects the "shards of 3 across two entries" and "duplicate shard" mutations described in the PR body.
    • test_a_split_entry_is_named_for_its_part correctly derives "dashboard tests 1/2" / "dashboard tests 2/2" from (workspace, part, shard), matching the matrix labels.
  • ONBOARDING.md: the "checks that run on any PR" line was updated to say "in two shards," consistent with the new matrix — no other stale references to the old single "dashboard tests" entry found elsewhere in the repo (docs/, workflows).

One thing I couldn't verify directly: I wasn't able to run pytest or pre-commit in this sandboxed environment (no .venv present and command execution needed approval I don't have here), so I relied on manually tracing the test logic against the workflow YAML rather than executing it. The PR description states pre-commit run --all-files was run clean and the pinning test's nine mutations were verified, which I have no reason to doubt given the code traces correctly by hand.

Nothing else to flag — the change is narrowly scoped to CI sharding, the pinning test is extended sensibly rather than weakened, and docs were kept in sync.

@Babissimo
Babissimo merged commit 3bf6268 into main Sep 24, 2026
21 of 22 checks passed
@Babissimo
Babissimo deleted the ci/shard-dashboard-tests branch September 24, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant