Skip to content

Run the dashboard's tests on their own, and close three coverage-gate gaps - #549

Merged
Babissimo merged 2 commits into
mainfrom
ci/split-dashboard-tests
Sep 23, 2026
Merged

Babissimo merged 2 commits into
mainfrom
ci/split-dashboard-tests

Conversation

@Babissimo

@Babissimo Babissimo commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Follows #547 (backend sharding). Second half of ClickUp Shard the backend suite and split the dashboard tests, plus three follow-ups to #547's coverage gate that were parked on that ticket. Two commits, one purpose each.

1. Run the dashboard's tests in an entry of their own

Why. Now that #547 has sharded the backend suite, Web (dashboard) sets the gate. On main's run for #547 itself (35848199811) the backend path finished at 95 s and Web (dashboard) at 119 s. That job runs lint (13 s), typecheck (7 s), tests (39–75 s) and build (5 s) in sequence, so the tests wait on about 25 s of work they don't depend on.

What.

  • web-build becomes an explicit include matrix. The dashboard gets two entries, Web (dashboard checks) (lint, typecheck, build) and Web (dashboard tests); packages/shared and e2e keep one entry each and run everything as before. Step if:s on matrix.part choose each entry's half.
  • The job id is still web-build, so staging's needs: doesn't change. There is no branch protection or ruleset on main, so no required check refers to the old Web (dashboard) name.
  • backend/tests/test_web_build_matrix.py pins the assignment: every workspace has an entry, each lint/typecheck/test/build script runs in exactly one entry, the dashboard is split in two, each half runs only its own scripts, and a split entry's name says which half it is. Eleven mutations of the workflow all turn it red, including the two halves trading conditions (which keeps every count right).

Fixed versus divisible. Timed locally at CI's three workers, the whole dashboard suite took 100 s and each --shard third 37–41 s. That fits about 8 s fixed against 92 s divisible, the opposite of pytest's ~47 s fixed.

Expected effect. This PR's first run: Web (dashboard tests) finished at 63 s (46 s test step) and Web (dashboard checks) at 46 s, against a backend path of 107 s. On a slow runner the tests entry is about 13 s of setup plus a 75 s step, around 88 s, against a backend path of 94–121 s across the sharded runs so far. So the gate becomes the backend path: ~95 s rather than 119 s on #547's main run. The margin is thin on a slow runner. vitest --shard isn't included: it buys nothing while the tests entry finishes under the backend path. Given the split above, it is the cheap next step if the margin closes.

Deliberate trade-off: a lint or typecheck failure no longer skips the tests, so a broken tsc now costs a full test run. Runner minutes are free on this public repo, and wall time is what the gate measures.

2. Close three gaps in the backend coverage gate's pins

  • test_a_tmp_path_fixture_is_omitted_from_the_report checked only test_deploy_scope.py's shallow path shape. It now also asserts test_migrations.py's deeper popen-gwN/…/old_image/core/… shape. Two plausible narrowings of the glob (fixed depth, serial basetemp layout) matched the shallow path and would have passed while backend-coverage died; both now go red.
  • --splits 3 → --splits ${{ strategy.job-total }}, so the matrix alone sets the shard count. The test pins that --splits and --group follow the matrix, and that shard is its only axis: job-total counts every combination, so a second axis would leave part of the suite unrun.
  • The fragment count is parsed with _threshold() rather than a whitespace-sensitive substring.

Checks

  • pre-commit run --all-files (ruff, actionlint, vulture) passes.
  • Workflow-reading tests (test_deploy_workflows, test_rollback_script, test_smoke_tally_shared, test_nginx_origin_marker, test_backend_coverage_gate, test_web_build_matrix): 108 passed.

🤖 Generated with Claude Code

@claude

This comment has been minimized.

@Babissimo Babissimo changed the title Run the dashboard's tests in a web-build entry of their own Run the dashboard's tests on their own, and close three coverage-gate gaps Sep 23, 2026
@Babissimo
Babissimo force-pushed the ci/split-dashboard-tests branch from 7c9f94f to 5fcc101 Compare September 23, 2026 10:50
@claude

This comment has been minimized.

@Babissimo
Babissimo force-pushed the ci/split-dashboard-tests branch from 5fcc101 to 1af13a6 Compare September 23, 2026 11:00
@claude

This comment has been minimized.

Babissimo and others added 2 commits September 23, 2026 13:16
With the backend suite sharded (#547), Web (dashboard) became the job that
sets the gate: on main's run for #547 the backend path finished at 95 s and
the dashboard at 119 s. It runs lint (13 s), typecheck (7 s), tests (39-75 s)
and build (5 s) one after another, so the tests wait on 25 s of work they do
not depend on.

The tests now have their own matrix entry, beside one for lint, typecheck
and build. On a slow runner that entry is about 13 s of setup plus the test
step, which is below the backend path and takes the dashboard off the
critical path. The other workspaces keep one entry each; their scripts take
20 s together and splitting them would buy nothing.

Splitting rather than sharding with vitest --shard, because the split alone
is enough while the backend path stays above this entry, and sharding would
add a runner for no gain today. Unlike pytest, vitest's cost does divide by
shard: timed at three workers, the whole suite took 100 s and each third
37-41 s, which fits about 8 s fixed against 92 s divisible. So --shard is
the next step if the backend path ever drops below this entry.

The job keeps its name, web-build, so staging's needs list is unchanged.
Which entry runs which script is now decided by step conditions, and a
mistake there fails nothing: a script left out runs nowhere, one run twice
silently undoes the split, and the two halves trading conditions keeps
every count right while the tests land back on the checks entry.
tests/test_web_build_matrix.py pins that every workspace in package.json has
an entry, that each lint, typecheck, test and build script it defines runs
in exactly one, that the dashboard is split in two, that each half runs only
its own scripts, and that a split entry's name says which half it is. All
eleven mutations tried against the workflow turn it red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three follow-ups to #547, where the shards' coverage is combined and the threshold applied on a runner that measured nothing.

The omit test checked only test_deploy_scope.py's shallow tmp_path shape, a repo's top-level app.py. test_migrations.py also puts measured files in the report, a copied core/ two levels deeper and under an xdist worker's popen-gwN directory. A glob narrowed to a fixed depth, or to the serial basetemp layout, still matched the shallow path and would have passed while backend-coverage died on "No source for code". The test now asserts both shapes; both of those narrowings turn it red.

--splits named the shard count as a literal beside the matrix list it has to equal. It now reads strategy.job-total, so the matrix alone sets the count, and the test pins that --splits and --group follow the matrix. job-total counts every combination, so the test also pins that shard is the matrix's only axis: a second one would multiply --splits while --group still spanned only the shards, leaving part of the suite unrun.

The fragment count was checked by a whitespace-sensitive substring, where the rest of the file parses numbers with _threshold(). It uses that now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Babissimo
Babissimo force-pushed the ci/split-dashboard-tests branch from 1af13a6 to c0a9ce1 Compare September 23, 2026 12:17
@claude

claude Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Claude finished @Babissimo's task in 5m 46s —— View job


Claude finished the review

Review

Re-ran /code-review against the current HEAD. It traced every changed line in .github/workflows/ci.yml, backend/tests/test_web_build_matrix.py, and backend/tests/test_backend_coverage_gate.py, plus the callers/consumers of each (staging's needs: [... web-build ...], matrix.shard in backend-tests, the omit pattern in backend/pyproject.toml), and found no correctness bugs, broken invariants, cross-file breakage, or CLAUDE.md violations:

  • The web-build matrix split is sound: the two dashboard entries' if: matrix.part != 'tests' / != 'checks' partition lint/typecheck/build vs. test correctly, and packages/shared/e2e (no part set) still run all four steps since both conditions evaluate true for them.
  • The job id web-build is unchanged, so staging's needs: isn't affected by the entry split.
  • --splits ${{ strategy.job-total }} is consistent with the unchanged shard: [1, 2, 3] matrix, and the new test_web_build_matrix.py/test_backend_coverage_gate.py assertions pin these invariants (script-to-entry assignment, shard-count consistency, single-axis matrix, coverage-omit glob shape) so future drift fails loud.
  • Result matches the prior review pass on this PR (the -ne 3 hardcoded literal in backend-coverage's fragment-count guard is a pre-existing nitpick, not a new defect — it's now guarded by _threshold("-ne", ...) == len(shards), so it'll go red rather than silently pass if the shard count ever changes without updating it).

One unrelated CI observation: this run's backend-tests (1) job failed on sqlite3.OperationalError: database is locked in tests/test_blah2_poller.py::test_the_poller_follows_the_registry — that's contention flakiness under -n logical --dist worksteal, not something this PR's diff touches (it only changes ci.yml, ONBOARDING.md, and the two coverage/matrix test files). Worth a rerun rather than a code fix here.

No changes recommended.

@Babissimo
Babissimo merged commit e6f2375 into main Sep 23, 2026
38 of 39 checks passed
@Babissimo
Babissimo deleted the ci/split-dashboard-tests branch September 23, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant