fix: Integration tests cannot link with --features cuda,xla-iree

[inureyes]

Problem / Background

Any integration test built with the default features plus cuda,xla-iree fails at link time on a GB10 host (aarch64, sm_121) with IREE_CUDA_HOME provisioned by scripts/iree/setup-cuda.sh:

/usr/bin/ld: .../libiree_runtime_unified.a(call.c.o): undefined reference to symbol '__stack_chk_guard@@GLIBC_2.17'
/usr/bin/ld: /lib/ld-linux-aarch64.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

This is not specific to one test. The failure reproduces identically on tests/molmo2_xla_vision_parity.rs (which declares required-features = ["xla-iree"], Cargo.toml:332-333) and on the unrelated tests/cli_help_consistency.rs, which has no XLA involvement at all. The whole integration-test target class is affected, because the link recipe is emitted once by the root build.rs and applies to every target rustc links.

Found while validating PR #916. It is unrelated to that PR's changes: the same failure reproduces on an unrelated test target on the same tree.

Current Behavior

The root build.rs emits the IREE link recipe. The CUDA branch (build.rs:101-141, keyed on IREE_CUDA_HOME) emits this argument list at build.rs:117-129:

-Wl,--whole-archive -l:libiree_runtime_unified.a -Wl,--no-whole-archive
-Wl,--start-group -l:libiree_hal_drivers_cuda_registration_registration.a
-l:libflatcc_parsing.a -lgcc -lm -lpthread -ldl -Wl,--end-group

rustc links these test binaries with -nodefaultlibs, and the group does not include -lc, so libc symbols the IREE runtime references (__stack_chk_guard in the observed failure) have nothing to resolve against. libprintf_printf.a is conditionally inserted at build.rs:130-136 when the source build produced it, which does not change the libc situation.

Two other link recipes live in the same file and share the same shape of risk:

• The macOS branch at build.rs:38-86, keyed on IREE_MACOS_HOME. It uses -Wl,-force_load rather than --whole-archive, relies on Apple ld being multi-pass (so no --start-group), links driver archives collected by collect_driver_archives, and has no libgcc.
• The IREE_DIST branch at build.rs:143-176, which emits a group very close to the CUDA one (libflatcc_runtime.a plus libflatcc_parsing.a, -lgcc -lm -lpthread -ldl) and likewise has no -lc.

Why it has not been noticed

cargo check does not link, so cargo check --features cuda,xla-iree --all-targets passes cleanly on the same tree. The existing Molmo2 diagnostics workflow uses --no-default-features --features xla-diagnostics, which links successfully. The failure only appears once someone builds a test binary with the default feature set (which includes surgery, Cargo.toml:70) plus cuda,xla-iree. Establishing whether the surgery default feature is what shifts the link set is part of diagnosing this, not an established fact.

Impact

cuda,xla-iree is the natural production feature set for the OpenXLA backend on a CUDA host, and today no integration test can be built with it. Unit tests (--lib) and binaries are unaffected. Only test binaries that pull the IREE archives through this recipe fail.

Reproduce

eval "$(bash scripts/iree/setup-cuda.sh --env)"
export MLX_CUDA_ARCHITECTURES=121
cargo test --release --features cuda,xla-iree --test cli_help_consistency --no-run

Proposed Solution

Adding -lc inside the --start-group block of the CUDA recipe is the obvious candidate, but it is a hypothesis, not a settled fix. Before landing it, check the interaction with the other two recipes in the same file: the macOS IREE_MACOS_HOME branch (build.rs:38-86, Apple ld, -force_load, multi-pass, no libgcc, so a GNU-ld-shaped fix does not transfer) and the IREE_DIST branch (build.rs:143-176, which has the same missing -lc). Whether the IREE_DIST branch should be changed in the same way depends on whether it reproduces the failure when actually linked; do not change it blind.

Whatever lands must be verified by actually linking a test binary in each of the three configurations a developer can reach, not only by cargo check.

Scope

In scope: build.rs link-argument recipes (CUDA branch at 101-141, IREE_DIST branch at 143-176, macOS branch at 38-86 only if a link check shows it needs a change), plus the explanatory comments around them.

Out of scope: changes to scripts/iree/setup-cuda.sh or scripts/iree/setup-macos.sh, the IREE source build itself, and any behavior change in mlxcel-xla.

Implementation Notes

• Reuse: keep the existing single-emission structure in build.rs. Do not add a second place that emits link args.
• Constraints: GNU ld is single-pass and left to right; anything added must sit where the group re-scan can resolve it. Apple ld does not accept --start-group or -lgcc, so the macOS arm must stay distinct.
• Edge cases: source builds that do produce libprintf_printf.a and those that do not (build.rs:130-136) must both still link. The IREE_DIST branch must still link when IREE_CUDA_HOME is unset.
• Error handling: the existing assert!/expect diagnostics that point at the setup scripts must be preserved.

Acceptance Criteria

• cargo test --release --features cuda,xla-iree --test <any> --no-run links successfully on a GB10 host with IREE_CUDA_HOME set, including at least one test with required-features = ["xla-iree"] and one without.
• The --no-default-features --features xla-diagnostics path still links.
• The macOS and IREE_DIST recipes are either unchanged or verified by an actual link, not by cargo check.
• A comment in build.rs records why each library in the group is present, so the next edit does not drop one again.

Verification

eval "$(bash scripts/iree/setup-cuda.sh --env)"
export MLX_CUDA_ARCHITECTURES=121

# The failing case; must now link.
cargo test --release --features cuda,xla-iree --test cli_help_consistency --no-run
cargo test --release --features cuda,xla-iree --test molmo2_xla_vision_parity --no-run

# The path that already works; must not regress.
cargo test --release --no-default-features --features xla-diagnostics --no-run

A pass is a clean exit with the Executable tests/... line printed for each target and no ld returned 1 exit status. On macOS, the equivalent check is a link of a test target with IREE_MACOS_HOME set via scripts/iree/setup-macos.sh; report the result rather than assuming parity with Linux.

Technical Considerations

Related: PR #916 (where this was observed, unaffected by it). The xla-diagnostics feature composes cuda and xla-iree (Cargo.toml:103), so a fix must hold for both the composed feature and the direct cuda,xla-iree combination.

---

Correction from implementation (PR #1275)

Two claims in the analysis above were wrong, and are corrected here rather than silently in the PR.

The surgery hypothesis is unresolved, not confirmed. The failure is both target-dependent and feature-dependent: chat_template_kwargs links under cuda,xla-iree while molmo2_xla_vision_parity does not, and the same molmo2_xla_vision_parity links under xla-diagnostics. Both working binaries carry ld-linux-aarch64.so.1 as an explicit DT_NEEDED and the failing links do not. What puts it there in the working cases was not identified.

-lc is the fix, but not for the reason given. The suggestion was right; the mechanism was not. libc.so.6 does not define __stack_chk_guard (it is UND there), so adding libc does not supply the symbol. It works because rustc-link-arg can only append: the IREE archives land after rustc's own -lc, so their references have no libc after them. Repeating -lc after the archives restores the ordering, and ld then resolves through libc's DT_NEEDED to the dynamic linker.

Ablation on the real link, so the shipped change is the minimal one:

configuration	result
trailing -lc only	links
-Wl,--copy-dt-needed-entries only	fails, same error
both	links

The policy flag is therefore not shipped. It is a global relaxation that can hide a genuinely missing -l in the same link, and it buys nothing here.