Skip to content

Sign GCS service-account JWTs with aws-lc-rs [SECURITY] - #2734

Open
MarcusSorealheis wants to merge 4 commits into
TraceMachina:mainfrom
MarcusSorealheis:fix-vuln-2023-0071
Open

Sign GCS service-account JWTs with aws-lc-rs [SECURITY]#2734
MarcusSorealheis wants to merge 4 commits into
TraceMachina:mainfrom
MarcusSorealheis:fix-vuln-2023-0071

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Sep 5, 2026

Copy link
Copy Markdown
Member

What and why

Removes the rsa crate from the dependency graph to clear
RUSTSEC-2023-0071 (Marvin attack:
non-constant-time RSA that can leak private-key material through timing).
The advisory has no fixed version, so the only way to make the OpenSSF
Scorecard vulnerabilities check stop flagging it is to stop depending on
rsa altogether.

The single consumer is jsonwebtoken's rust_crypto backend, which
gcloud-auth uses to sign Google service-account JWTs. This switches
nativelink-store's gcloud-auth feature from jwt-rust-crypto to
jwt-aws-lc-rs, which is gcloud-auth's own default backend. That drops
rsa plus 22 other RustCrypto crates (p256, p384, ed25519-dalek,
num-bigint-dig, pkcs1, ...) and adds aws-lc-rs, aws-lc-sys,
cmake, fs_extra and untrusted.

Scope of the crypto change: aws-lc-rs is only linked for JWT signing.
Every rustls stack in the binary still runs on ring via the process
default provider installed in common_s3_utils and nativelink.rs; the
rustls crate's aws-lc-rs feature stays off, so there is no second rustls
provider to race with. The comments in nativelink-store/Cargo.toml are
updated to say so. The rules_rs crate metadata in MODULE.bazel.lock was
regenerated for the new graph.

Bazel: hermetic aws-lc instead of the cargo build script

The first CI run of this PR failed every Linux Bazel job at link time with
mold: error: undefined symbol: __isoc23_sscanf (referenced from
aws-lc-sys's bcm.c). aws-lc-sys's cargo build script compiles its C
with the toolchain's glibc 2.42 headers, which rename sscanf to
__isoc23_sscanf, while the LRE Rust toolchain links against musl, which
has no such symbol. #2452 hit the same wall and #2472 avoided it by dropping
aws-lc entirely.

rules_rs ships a supported answer for exactly this: it prints a warning
recommending the well-known annotation that turns the build script off and
links the crate against the BCR aws-lc module (built by Bazel's own C++
toolchain for the target platform) with bindgen-generated bindings.
MODULE.bazel now carries that recipe verbatim: bazel_dep(aws-lc), the
prebuilt bindgen toolchain from rules_rs, gen_build_script = "off" for
aws-lc-rs/aws-lc-sys, and inject_repo(crate, "aws-lc"). Cargo builds
are unaffected.

That moved the C compile into Bazel, but the Nix clang used for Linux builds
(LRE-CC under Bazel, TARGET_CC for the cargo/Nix packages) still compiles
against glibc headers while the Rust side links musl, so the second CI run
failed on the same class of symbol, and the Nix cargo llvm-cov build in
the Coverage job failed the same way: fopen64, __isoc23_sscanf and
__isoc23_strtol, the complete list mold reports. A blanket
--defsym link flag was tried next and broke every lld-based job (lld
refuses an alias whose target is absent from the link).

What is in the PR now: nativelink-store/aws_lc_musl_shims.c, three
forwarders that give those glibc names a definition on musl targets. It is
compiled by a new build.rs for cargo (only when CARGO_CFG_TARGET_ENV is
musl) and by an aws_lc_musl_shims cc_library for Bazel (only under
@local-remote-execution//libc:musl), which the aws-lc-sys annotation links
in with alwayslink. On glibc targets both are empty. The crane source
filter in flake.nix now lets that one .c file through, since it only
passes cargo sources by default. The proper fix is a
musl C toolchain for the musl Rust platforms in the Nix/LRE setup; that is
out of scope for a dependency bump, and this PR is the first thing that
makes the mismatch visible.

Worth weighing before merging. This is the third build-system
accommodation for one advisory: the hermetic aws-lc module for Bazel, the
musl shims for Bazel and cargo, and the NASM env for Windows. #2452 and
#2472 both backed away from aws-lc for the same musl pain. The Marvin attack
needs an attacker who can time RSA private-key operations on chosen inputs;
here the only RSA operation is signing a JWT NativeLink builds itself, so
the practical exposure is close to nil. Accepting RUSTSEC-2023-0071 as a
known, unexploitable finding is a legitimate alternative to this PR.

How was this verified?

  • cargo update -w after the feature change; cargo tree -i rsa now
    returns nothing.
  • cargo check --all passes.
  • cargo test -p nativelink-store --profile=smol --test gcs_client_test --test gcs_store_test passes (35 tests) with the new backend linked in.
  • bazel build --nobuild //:nativelink regenerated the crate metadata and
    pulled the aws-lc module; bazel build @crates//:aws-lc-sys-0.45.0 @crates//:aws-lc-rs-1.18.1, a full bazel build //:nativelink, and
    bazel test //nativelink-store:integration_tests/gcs_client_test_test
    all succeed on aarch64 macOS with the repository's LLVM toolchain. The
    resulting binary links AWS-LC's aws_lc_bazel_-prefixed symbols and
    aws-lc-sys no longer has a _bs build-script target.
  • The first CI run failed on windows-2022: aws-lc-sys needs NASM to
    assemble on x86_64 Windows and the runner image does not ship it. The
    Cargo Native workflow now sets AWS_LC_SYS_PREBUILT_NASM=1, which makes
    the crate use the prebuilt NASM objects it carries (a no-op on every other
    target). Anyone building on Windows locally needs NASM on PATH or that
    variable set.
  • Not verified locally: the Linux and musl LRE builds. Improve security scorecard #2452 previously
    needed extra C flags for aws-lc under musl and Migrate Azure Blob store to azure_storage_blob 1.0 #2472 later moved the
    tree to ring-only, so the LRE and native-bazel CI jobs are the real test
    for this PR.

Risk

Moderate. This adds a C crypto library to the build for every platform that
compiles nativelink-store, which is where a regression would show first
(build breakage in the Linux/musl CI jobs, or a larger binary), not at
runtime. Runtime behavior only changes for GCS deployments authenticating
with a service-account key file: the JWT is now signed by aws-lc instead of
the rsa crate. Deployments using workload identity or the metadata server
never sign a JWT and are unaffected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RW4DfubnMqK4UXHmG1J4vE


This change is Reviewable

@vercel

vercel Bot commented Sep 5, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 7, 2026 7:12am UTC
nativelink-aidm Ready Ready Preview Sep 7, 2026 7:12am UTC

Request Review

Fixes RUSTSEC-2023-0071 (Marvin attack) by dropping the `rsa` crate,
which has no patched release. Its only consumer was jsonwebtoken's
`rust_crypto` backend behind gcloud-auth's `jwt-rust-crypto` feature;
switch to `jwt-aws-lc-rs`, gcloud-auth's default backend.

aws-lc-rs is linked only for JWT signing. Every rustls stack still runs
on ring through the process-default provider, and rustls's aws-lc-rs
feature stays off.

Under Bazel, aws-lc-sys's cargo build script compiles against glibc
headers while the LRE toolchain links musl, leaving `__isoc23_sscanf`
undefined. Adopt the rules_rs recipe instead: build script off, link
against the BCR `aws-lc` module with bindgen-generated bindings.

The Cargo Native workflow sets AWS_LC_SYS_PREBUILT_NASM=1 so the
x86_64 Windows job builds aws-lc-sys from the crate's prebuilt NASM
objects instead of requiring NASM on the runner.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RW4DfubnMqK4UXHmG1J4vE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant