scripts: serialize PyPy/Rust install and verify toolchain is usable

[avi-starkware]

Summary

Fixes a flaky Docker build failure of the form:

#14 60.73 [install_build_tools] Installing sccache 0.14.0...
#14 60.92 error: 'cargo' is not installed for the toolchain '1.92-x86_64-unknown-linux-gnu'.
#14 ERROR: process "/bin/sh -c scripts/install_build_tools.sh" did not complete successfully: exit code: 1

What we observed

• 'cargo' is not installed for the toolchain '1.9x-…' on several runs (PR scripts: install project Rust toolchain before cargo tools #13795's reproducer, remove-mock-forest-storage's sequencer docker build, and the most recent report on einat/proof-flow-1-state-reader-fixes).
• rustup panicking mid-install with thread 'CloseHandle' panicked at src/diskio/mod.rs:…: called Result::unwrap() on an Err value: RecvError on two separate runs (one on this branch, one on main's native-blockifier-artifacts-push). In both cases, buildx then hung for ~50 minutes until its gRPC keepalive timed out.

Both observations share a root pattern: rustup is asked to install the rust-toolchain.toml-pinned channel, the install is interrupted mid-flight, and rustup is left with a stub entry for the channel (marked "installed" / "active") while its components (rustc, cargo, …) never finished downloading. On the next invocation, cargo install … fails with "cargo is not installed for the toolchain".

I don't have a single log that shows the panic directly causing a subsequent "cargo not installed" error — the buildx hang usually masks what would have surfaced — but the panic is at least one plausible mid-install failure mode, and any mid-install exit (panic, OOM, timeout, etc.) produces the same partial-install state.

Fixes

1. Serialize install_pypy and install_rust. Concurrent disk I/O from a sibling installer is a known trigger for rustup's multi-threaded downloader panic (rust-lang/rustup#2926). Serial execution removes that trigger. The extra wall-time cost is ~10s per fresh base-image build (PyPy's runtime), and the step is Docker-cached most of the time.
2. Probe the pinned toolchain after install; force-reinstall on failure. This is the real correctness guarantee — it catches any partial-install state regardless of what caused it. cargo --version in the project dir (where rust-toolchain.toml is the active override) is a ~10ms no-op on a healthy install, so devs who already have the toolchain fully installed pay nothing. --force is only invoked when the check actually fails.

Why not unconditional rustup toolchain install --force?

It works in Docker (layer cache absorbs the cost) but is disruptive on a developer machine: --force re-downloads ~200 MB even when the toolchain is perfectly fine. The conditional probe avoids that.

Test plan

• CI (Docker builds in particular) passes on this branch
• On a dev machine with 1.92 already installed, the verification step short-circuits (no unnecessary reinstall in logs)

[cursor]

PR Summary

Low Risk
Changes are limited to build tooling scripts and primarily add defensive checks; main risk is longer install time or unexpected --force reinstall if the probe misfires.

Overview
Stops running PyPy and Rust setup in parallel in scripts/install_build_tools.sh, switching to serial installation to avoid flaky rustup partial installs.

Adds a post-install probe (cargo --version in the repo root) and conditionally forces rustup toolchain install --force when the project-pinned toolchain is missing components, then logs the resolved rustc version before proceeding.

^{Reviewed by Cursor Bugbot for commit 416eff2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is 

[github-actions]

Artifacts upload workflows:

• Started at 2026-04-20T08:01:31Z
• Started at 2026-04-20T15:28:44Z
• Started at 2026-04-20T15:38:24Z

[dorimedini-starkware]

@dorimedini-starkware reviewed 1 file and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on avi-starkware, idan-starkware, and yoavGrs).

---

scripts/install_build_tools.sh line 106 at r1 (raw file):

log_step "install_build_tools" "Installing PyPy and Rust..."
# Intentionally serial: rustup-init's multi-threaded component downloader has a
# known race (`thread 'CloseHandle' panicked at src/diskio/mod.rs: RecvError`)

is there some github issue you can link to here?

Code quote:

known race

---

scripts/install_build_tools.sh line 112 at r1 (raw file):

install_pypy
install_rust
log_step "install_build_tools" "PyPy and Rust installed"

Suggestion:

log_step "install_build_tools" "Installing PyPy..."
# Intentionally serial: rustup-init's multi-threaded component downloader has a
# known race (`thread 'CloseHandle' panicked at src/diskio/mod.rs: RecvError`)
# that's triggered or aggravated by concurrent I/O from parallel installers.
# When the race fires, rustup leaves a partial toolchain stub behind and later
# `cargo install` calls fail with "'cargo' is not installed for the toolchain".
install_pypy
log_step "install_build_tools" "PyPy installed. Installing Rust..."
install_rust
log_step "install_build_tools" "Rust installed."

[dorimedini-starkware]

@dorimedini-starkware reviewed 1 file and all commit messages, and resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on idan-starkware and yoavGrs).

[avi-starkware]

@avi-starkware made 2 comments.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on idan-starkware and yoavGrs).

---

scripts/install_build_tools.sh line 106 at r1 (raw file):

Previously, dorimedini-starkware wrote…

is there some github issue you can link to here?

Done.

---

scripts/install_build_tools.sh line 112 at r1 (raw file):

install_pypy
install_rust
log_step "install_build_tools" "PyPy and Rust installed"

Done.

[dorimedini-starkware]

Reviewable status: complete! all files reviewed, all discussions resolved (waiting on idan-starkware and yoavGrs).