Skip to content

test: consolidate the workspace's 71 test binaries into at most 8 categories (soldr layout) #1577

Description

@zackees

Problem

soldr cargo test --workspace built and linked 71 test binaries in the ubuntu PR gate (run 36525259298). Every binary links its crate's whole dependency graph. Test runtime is now ~50 s of the 423 s Test step, so almost all of the rest is compiling and linking those binaries. This is the shape zackees/ci.yml RUST-005 / RUST-012 target.

#1574 made a first cut: fbuild-build (18 files) and fbuild-daemon (10 files) now build one tests/it/main.rs binary each. That's 71 → 47, and locally cargo test --no-run -p fbuild-build -p fbuild-daemon went from 59–79 s to 20–31 s after a clean of those crates. This issue covers the rest.

Target: soldr's layout

soldr keeps at most 8 category binaries, each a directory with a main.rs that declares its modules, plus a shared common/ module (crates/soldr-cli/tests/{broker,cache_gc,cargo_front_door,cook_dylint,daemon,fetch_tools,guards,save_roundtrip,toolchain_env}/main.rs). A new test goes into an existing category, never into a new top-level tests/*.rs.

Current inventory (47 after #1574)

crate test binaries
fbuild-core lib + 6 integration (cpu_profiling, dep_identity, 4× platformio_*_resolution)
fbuild-cli bin fbuild + 4 integration (ci_command, daemon_crash_recovery, lib_select, test_emu_exit_code)
fbuild-build lib + it + avr_build + eh_frame_strip_esp32
fbuild-daemon lib + it + bins fbuild-daemon, containment_harness
fbuild-python cdylib _native + pyo3_policy, python_facades
fbuild-packages lib + disk_cache_schema_migration, lnk_e2e
fbuild-library-select lib + perf_tdd, teensy41_ldf_diag
fbuild-config lib + bin enrich_boards
fbuild-toolchain lib + qemu_linux_runtime
fbuild-build-engine lib + dev_daemon_namespace_isolation
fastled-examples bins bench-build-comparison, bench-fastled-examples
10 more crates lib only

Harnesses that ran 0 tests in CI but were still built and linked: unittests src/main.rs, src/bin/containment_harness.rs, src/bin/enrich_boards.rs, tests/disk_cache_schema_migration.rs, tests/qemu_linux_runtime.rs (the last two are presumably all #[ignore]d or cfg-gated).

Proposed categories (≤ 8 integration binaries across the workspace)

Integration tests can only see their own crate's public API, so each category lives in whichever crate already depends on everything it needs. That's usually fbuild-cli or fbuild-build, both of which sit high in the graph.

  1. build: fbuild-build/tests/it (done), plus build-engine namespace isolation
  2. daemon: fbuild-daemon/tests/it (done), plus cli daemon_crash_recovery
  3. cli: ci_command, lib_select, test_emu_exit_code
  4. config_resolution: fbuild-core platformio_*_resolution, dep_identity, cpu_profiling
  5. packages: disk_cache_schema_migration, lnk_e2e, toolchain qemu_linux_runtime
  6. library_select: perf_tdd, teensy41_ldf_diag
  7. python: pyo3_policy, python_facades (kept separate because they need libpython at link time)
  8. env_isolated: avr_build + eh_frame_strip_esp32. Their #[ignore]d download tests mutate FBUILD_* env vars, so they must not share a process with the other ignored tests. Either give them a category of their own, or make them take a shared env lock and then fold them into build.

Also

  • Set test = false on bins and libs whose harness has zero #[test] (fbuild bin, containment_harness, enrich_boards, the fastled-examples bench bins), per RUST-012's "enabled harness with zero tests".
  • Update the Dylint allowlists (dylints/*/src/allowlist.txt), --test <name> references in workflows (acceptance-205.yml, esp32s3-size-parity.yml) and run instructions in test doc comments. ci: compile fbuild once, path-selected board dispatch (badges stay current), main-only cache saves #1574 shows each place that breaks: --test X becomes --test <category> -- X::….
  • Add a guard (CI check or crate_guard-style hook) that fails on a new top-level crates/*/tests/*.rs, so the count stays at ≤ 8.

Acceptance

  • soldr cargo test --workspace --no-run produces ≤ 8 integration-test binaries, plus lib harnesses only for crates that actually have unit tests.
  • Pass/ignored counts are unchanged (compare --list before and after).
  • The ubuntu Test step time is recorded before and after in the PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions