Problem
We want to add substantially more tests and more groups, but the current
matrix already makes the PR cycle slow, and the groups that are defined but
not wired would multiply it rather than extend coverage.
test/tap/groups/lint_group_coverage.py reports:
group coverage lint: 94 groups | phantom-infra=0 | missing-workflow NEW=1 known=41
The 41 allowlisted families are real, intentional gaps. The largest are four
config-variant families:
| family |
groups |
tests |
mysql-auto_increment_delay_multiplex=0 |
4 |
274 |
mysql-multiplexing=false |
4 |
274 |
mysql-query_digests=0 |
4 |
273 |
mysql-query_digests_keep_comment=1 |
4 |
273 |
plus mysql90 (5 groups), mysql95 (5), mysql93, the *-binlog families,
mysql91-gr, mysql92-gr, mysqlx-e2e, pgsql-repl, pgsql17-repl, todo.
Wiring these naively is not a marginal cost. Today CI executes ~756 test-runs;
groups.json describes 2481. Enabling everything is 3.3x, and the four
config families alone are ~1094 test-runs -- more than the entire current load
-- for zero new tests. They are the same ~274 tests replayed with one
ProxySQL variable flipped each.
Proposal
1. Enable the variant families nightly, not per-PR.
Zero PR-latency cost, and they start producing signal immediately.
2. Promote selectively, driven by evidence.
Almost certainly only a minority of the 274 tests behave differently under
multiplexing=false or query_digests=0. Any test that fails under a variant
while passing under default gets promoted into a per-PR variant group. The
nightly keeps covering the rest.
This makes variant membership opt-in per test instead of blanket, so the four
families cost on the order of 200 runs rather than 1094 -- and, more importantly,
a newly added test then costs 1 run instead of 5. That is the property that makes
the suite affordable to grow.
Related, tracked separately
Not doing now
Deliberately deferred until the timeout and coverage work above settles.
Problem
We want to add substantially more tests and more groups, but the current
matrix already makes the PR cycle slow, and the groups that are defined but
not wired would multiply it rather than extend coverage.
test/tap/groups/lint_group_coverage.pyreports:The 41 allowlisted families are real, intentional gaps. The largest are four
config-variant families:
mysql-auto_increment_delay_multiplex=0mysql-multiplexing=falsemysql-query_digests=0mysql-query_digests_keep_comment=1plus
mysql90(5 groups),mysql95(5),mysql93, the*-binlogfamilies,mysql91-gr,mysql92-gr,mysqlx-e2e,pgsql-repl,pgsql17-repl,todo.Wiring these naively is not a marginal cost. Today CI executes ~756 test-runs;
groups.jsondescribes 2481. Enabling everything is 3.3x, and the fourconfig families alone are ~1094 test-runs -- more than the entire current load
-- for zero new tests. They are the same ~274 tests replayed with one
ProxySQL variable flipped each.
Proposal
1. Enable the variant families nightly, not per-PR.
Zero PR-latency cost, and they start producing signal immediately.
2. Promote selectively, driven by evidence.
Almost certainly only a minority of the 274 tests behave differently under
multiplexing=falseorquery_digests=0. Any test that fails under a variantwhile passing under default gets promoted into a per-PR variant group. The
nightly keeps covering the rest.
This makes variant membership opt-in per test instead of blanket, so the four
families cost on the order of 200 runs rather than 1094 -- and, more importantly,
a newly added test then costs 1 run instead of 5. That is the property that makes
the suite affordable to grow.
Related, tracked separately
TEST_TAP_TIMEOUTstill defaults to 0 (no per-test timeout) -- worth revisiting.no-infra-g1is the one NEW unwired group (5 tests, incl. the Monitoring failing after enabling caching_sha2_password globally #5363 regressiontest); it needs a caller + callee workflow pair.
legacy-g4~58 min vslegacy-g8~18 min. Wall clock is setby the slowest shard, so generating the shard assignment from measured runtimes
(and enforcing a max/median ratio in CI-lint-groups-json) would cut feedback
time at no coverage cost -- and, unlike a one-off rebalance, stays balanced as
tests are added.
Not doing now
Deliberately deferred until the timeout and coverage work above settles.