run_tests v1.0.0 can't express QA-only-for-GROUP==QA (QA excluded from "All"); blocks single-package rollout

[ChrisRackauckas-Claude]

Summary

run_tests v1.0.0 cannot reproduce the very common SciML pattern where the QA group runs only for GROUP=="QA" and is excluded from "All" (the default local ]test path). This blocks the SciMLTesting rollout for every single-package repo that follows that pattern.

Affected repos found during the rollout fan-out (all four run QA only under GROUP=="QA", never under "All"):

• GlobalSensitivity.jl
• HighDimPDE.jl
• IRKGaussLegendre.jl
• Integrals.jl

These repos' hand-written test/runtests.jl all have a standalone if GROUP == "QA" ... end branch plus an if GROUP == "All" || GROUP == "Core" (or bare else) branch that does not include QA. So under a plain local ]test (GROUP unset → "All"), QA does not run today.

The gap

In v1.0.0 there are exactly two ways to wire QA, and neither preserves "QA only for GROUP==QA, never under All":

1. qa= kwarg — runs QA for GROUP=="QA" and under "All" (src line 494: qa === nothing || _run_group_spec(...) inside the group == "All" branch). Using this changes behavior: ]test would newly run QA.

2. groups["QA"] with a declared env — correctly excluded from "All" (a groups entry with env !== nothing is skipped under All), but unreachable when GROUP=="QA": the elseif group == "QA" branch (line 499) short-circuits before the elseif haskey(group_table, group) branch (line 504) and throws ArgumentError("run_tests: GROUP=\"QA\" was requested but no \qa` body was provided")`.

Reproduction

using SciMLTesting

# (1) qa= kwarg: QA runs under "All" (behavior change vs. the repos above)
ran = String[]
withenv("GROUP" => "All") do
    run_tests(; core = () -> push!(ran, "core"),
              qa = (; env = qa_env_dir, body = () -> push!(ran, "QA")))
end
@assert ran == ["core", "QA"]   # QA leaked into All

# (2) groups["QA"] with env: unreachable for GROUP=="QA"
withenv("GROUP" => "QA") do
    run_tests(; core = () -> nothing,
              groups = Dict("QA" => (; env = qa_env_dir, body = () -> nothing)))
end
# ERROR: ArgumentError: run_tests: GROUP="QA" was requested but no `qa` body was provided

Suggested fix

Allow a QA body that is selectable by GROUP=="QA" yet excluded from "All". Options:

• Add an opt-out flag to the qa= spec, e.g. qa = (; env, body, in_all = false), so the group == "All" branch skips it when in_all == false. Default could stay true for back-compat.
• OR reorder/extend the GROUP=="QA" branch so a groups["QA"] entry is honored (i.e. check group_table for "QA" before erroring on qa === nothing). This makes the documented "declared-env group is excluded from All" mechanism work uniformly for QA too.

Either fix unblocks the four repos above (and the broader class). Until then those conversions are correctly held as blocked to avoid a behavior change to the local ]test path.

Verified against SciMLTesting v1.0.0 on Julia 1.11.9.

[ChrisRackauckas-Claude]

More repos hitting this blocker (rollout fan-out batch, 2026-06-13)

Four more single-package repos verified blocked by the same QA-excluded-from-"All" gap. All have the grouped-tests layer (GROUP dispatch + test/qa/ + test/test_groups.toml) already merged on their default branch — only the run_tests harness refactor is blocked.

Same shape as the original four (QA runs only for GROUP=="QA", never under "All"):

• ADTypes.jl — if GROUP=="All"||"Core" {core} + separate if GROUP=="QA" {activate qa; include qa.jl}. (qa.jl is a custom Aqua/JET body, not run_qa defaults.)
• BipartiteGraphs.jl — if GROUP=="QA" {activate qa; include qa.jl} else {core}. QA goes through else→core under "All". (qa.jl uses a custom Aqua subset + JET with @test_broken, tracked in BipartiteGraphs#36.)
• BlackBoxOptim.jl — if GROUP=="QA" {include qa/qa.jl; exit()} then the core test module. Extra wrinkle: its QA body runs in the main test env (no Pkg.activate(qa)), so even the qa=-with-env form would change the env it runs in.

Additional, independent reason (this one is blocked even beyond the QA-in-All gap):

• BaseModelica.jl — QA group is if (GROUP=="NoPre" || GROUP=="Quality") && isempty(VERSION.prerelease). Two issues run_tests v1.0.0 can't express:
  1. Two group names → one body (NoPre and Quality both trigger the same QA body); groups/qa have no multi-key alias.
  2. Conditional sub-env activation: the env (test/nopre) must be activated only when isempty(VERSION.prerelease). A declared-env group activates unconditionally once selected; CI runs version: pre × group: NoPre (the gate intentionally makes NoPre a no-op on prerelease — that's the point of the name), so an unconditional activate is a real behavior change on the pre matrix entry.

All four held as blocked pending the suggested fix (the in_all=false qa-spec flag, or honoring groups["QA"] for GROUP=="QA"). BaseModelica additionally needs a multi-name-alias + conditional-env mechanism.

[ChrisRackauckas-Claude]

Four more single-package repos hit this in a 2026-06-13 rollout batch; all run QA only for GROUP=="QA" and exclude it from "All":

• MATLABDiffEq.jl (default GROUP="Core")
• MathML.jl
• MaybeInplace.jl

For these three I built the groups["QA"]-with-declared-env conversion and opened draft PRs (SciML/MATLABDiffEq.jl#81, SciML/MathML.jl#105, SciML/MaybeInplace.jl#35), then empirically reproduced the throw and closed them as blocked on this issue:

ENV["GROUP"] = "QA"; run_tests(; core = ..., groups = Dict("QA" => (; env = ..., body = ...)))
# ArgumentError: run_tests: GROUP="QA" was requested but no `qa` body was provided

The elseif group == "QA" branch (src line 499) short-circuits and throws when qa === nothing before reaching the haskey(group_table, group) branch (line 504), so a groups["QA"] entry is unreachable for GROUP="QA". The alternative — the qa= kwarg — runs QA under "All" (line 494), which these repos' "All" does not do.

• MethodOfLines.jl hits this plus a second, independent blocker. Its test/qa/qa.jl self-activates the QA env with Pkg.activate(@__DIR__); Pkg.instantiate() and no Pkg.develop of the repo root. run_tests' only "exclude-from-All" mechanism is a declared env, which forces activate_group_env(develop=true) → on Julia 1.10/lts (where [sources] is ignored) this changes QA from analyzing the registered MethodOfLines release to the local PR-branch code — a real lts behavior change. GroupSpec exposes no develop/develop_sources flag to suppress it, and parent=[] still develops via the develop_sources! 1.10 [sources] backport. No PR opened.

The proposed in_all=false qa-spec flag (or honoring groups["QA"] for GROUP=="QA") would unblock the first three; MethodOfLines additionally needs a way to pass develop=false / develop_sources=false through the QA group spec (or its qa.jl should add the canonical Pkg.develop(root)).