Full libtest suite deadlocks: ABBA inversion between env barrier and workshop guard (headless_reload x workshop test)

[Hmbown]

Symptom

sh scripts/with-hermetic-test-home.sh cargo test -p codewhale-tui --lib --locked on main (8cd5bf199) freezes mid-run and never completes: completions stop (5285/12390 in the witnessed full run), all 14 libtest workers park inside runtime_api::tests::*, CPU goes to ~0, and every worker eventually trips libtest's "running for over 60 seconds" warning. Three independent full-suite processes on this machine are frozen at the same ~29 s CPU point (two full-lib runs plus one older orphan of the same binary), so this is deterministic, not a slow machine.

CI is unaffected (it runs nextest, one process per test), but the libtest full-lib run from AGENTS.md — which .config/nextest.toml still calls the authoritative release gate — cannot complete on main.

Bisection (all hermetic, same binary codewhale_tui-d15f71b25dca2897)

• Full lib: 5285 ok, then 14/14 workers stuck, all in runtime_api::tests.
• runtime_api::tests module alone: 34 ok, then 14/14 stuck. Prior global state not required.
• Memory subgroup (7 tests): 7 pass in 0.33 s.
• The 14 stuck tests by name: 3 escape, 11 wedge (60 s+ warnings).
• Halves: A (api_surfaces x2, events, fleet x2, get_config) 6/6 pass in 0.72 s; B (headless_reload, workshop, memory x3): memory 3 pass, then headless_reload + workshop wedge with no other test threads left alive — a 2-thread cycle.

Root cause (stacks via lldb -p, which attaches fine on this machine)

Classic ABBA lock-order inversion between the process-global env barrier (test_env_lock.rs) and the process-global workshop guard (tools/large_output_router.rs):

• headless_catalog::headless_reload_... seals the env first (let _env = lock_test_env() + guards, headless_catalog.rs:138-141), then blocks acquiring the workshop guard at headless_catalog.rs:142 (active_workshop_test_guard, large_output_router.rs:72). Holds ENV, wants WORKSHOP.
• http_and_web_server_thread_manager_installs_configured_workshop_byte_budgets takes the workshop guard first (tests.rs:141, held for the whole body), then blocks on the env barrier at tests.rs:154 via open_runtime_threads_for_server (runtime_api.rs:848) -> maybe_load_persisted_cache_for_config (provider_catalog_live.rs:1013) -> deepseek_base_url -> base_url_for_route_identity -> first_nonempty_env (config.rs:8483, blocking with_test_env_lock). Holds WORKSHOP, wants ENV.

Once these two wedge, the env barrier never releases; every other test needing it queues, all libtest workers park, and the suite freezes. Each test passes alone (verified: memory_list 0.05 s, memory subgroup, group A), so isolation runs and nextest never see it.

Fix direction

Establish one global lock order. Smallest change: take the workshop guard before the env seal in headless_reload (WORKSHOP -> ENV, matching the workshop test's order), and audit every other active_workshop_test_guard caller for the same inversion. Alternative: make the catalog-cache env read use the existing with_test_env_lock_if_uncontended deadlock-breaker — but that helper's contract is LazyLock initializers, and this read sits on a hot product path, so the reorder is the smaller blast radius.

Suggested verification: the headless_reload+workshop pair co-run (hangs pre-fix), the 14-name set, then the full lib suite to completion.

Suggest v0.9.13: it blocks the documented libtest gate, and tonight's release verification runs all died on it.

[Hmbown]

Fixed on main in f2f5d46.

Headless catalog tests now take active_workshop_test_guard before lock_test_env, matching the workshop byte-budget test's order (WORKSHOP → ENV). Co-ran the wedge pair under libtest --test-threads=2 (3/3 ok); runtime_api::tests:: with 14 threads completed without parking (no ABBA freeze).