Support sharing Cargo crates between bzlmod modules

[gkgoat1]

I want to support bazel for my crates with bzlmod, but, currently (afaik), Cargo dependencies have to be re-added per module, requiring unique naming of extension modules and creating many copies of the same crate. If crate_universe could merge all of the used modules' Cargo manifests/specs, I would be able to support Bazel in my crates.

Apologies if this already has been implemented, just let me know.

[gkgoat1]

Bump, I am planning to support Bazel in my tooling and this is a clear blocker for that.

[UebelAndre]

This is something we should absolutely support. It currently works with vendored crates as that's how extensions like rules_rust_prost or rules_rust_pyo3 work but definitions in MODULE.bazel are not as thought out.

[bmclarnon]

+1 that this would be a very helpful feature, and I'm glad it's something you're open to supporting. If contributions would be welcome, I'm willing to help design and implement a solution. (I currently use the very hacky work-around of having one module define a crate universe and using a patch to add extra dependencies and features needed by other modules. I'd definitely like something more maintainable...)

A "simple" fix would be to merge crate.spec and crate.annotation info for repos with the same name, then call cargo_bazel once. But that would only work for crate.from_specs, and I assume you'd want the feature to work for crate.from_cargo as well -- as well as for a mix of the two.

I'm less familiar with cargo_bazel's implementation, but would it be feasible to have (or generate for crate.from_specs) a Cargo.toml file per repo and module, but pass all Cargo.toml files for the same repo to the splicing step? cargo_bazel could define a new top-level Cargo.toml file with dependencies on all of them, leading them to participate in version resolution together. BUILD aliases would then be generated for each of them, with an error generated if multiple versions were found.

Edit: It might also be possible to eventually use some some proposed bazel 9 features (see bazelbuild/bazel#20186 (comment)) to provide each module with a view of a shared crate_universe that only exposes the dependencies it specifically listed. This would also allow different versions of the same crate to be used across different modules -- but backed by the same crate universe.

Please let me know if this is something I should start investigating -- and any suggestions you might have. Thanks!

[gkgoat1]

+1 that this would be a very helpful feature, and I'm glad it's something you're open to supporting. If contributions would be welcome, I'm willing to help design and implement a solution. (I currently use the very hacky work-around of having one module define a crate universe and using a patch to add extra dependencies and features needed by other modules. I'd definitely like something more maintainable...)

A "simple" fix would be to merge crate.spec and crate.annotation info for repos with the same name, then call cargo_bazel once. But that would only work for crate.from_specs, and I assume you'd want the feature to work for crate.from_cargo as well -- as well as for a mix of the two.

I'm less familiar with cargo_bazel's implementation, but would it be feasible to have (or generate for crate.from_specs) a Cargo.toml file per repo and module, but pass all Cargo.toml files for the same repo to the splicing step? cargo_bazel could define a new top-level Cargo.toml file with dependencies on all of them, leading them to participate in version resolution together. BUILD aliases would then be generated for each of them, with an error generated if multiple versions were found.

Edit: It might also be possible to eventually use some some proposed bazel 9 features (see bazelbuild/bazel#20186 (comment)) to provide each module with a view of a shared crate_universe that only exposes the dependencies it specifically listed. This would also allow different versions of the same crate to be used across different modules -- but backed by the same crate universe.

Please let me know if this is something I should start investigating -- and any suggestions you might have. Thanks!

My idea was to merge all crate specs in cargo-bazel across extension instances, but that would require significant refactoring (a single, global, crate_universe, with some way of creating views for individual modules)

[bmclarnon]

That's similar to what I was thinking, except I was only planning to merge extension instances with the same name. While uncommon, it's still useful to support separate crate_universe instances when incompatible crate features are needed (e.g., a single workspace building both std and no_std binaries). It might also make it easier to roll out the change in a non-breaking (or minimally-breaking) way.

That said, in the common case all extensions should use the same name -- maybe a default would help. (Though I vaguely recall some bazel recommendations to always require repository rules to have explicit names.)

[anforowicz]

I'd like to provide some concrete real-world data and thoughts on how lockfile enforcement could work here, as we are encountering this issue in production when consuming @protobuf//rust:protobuf from Crubit under Bzlmod. For example - we had to patch rules_rust as a workaround here in google/crubit#1925 (see https://github.com/google/crubit/pull/1925/changes#diff-62834dc0816bd66efdf86fdbef97692f207b1c69b6a360a1b28a011ed20e0e06).

This comment was generated with the help of an AI agent. I hope this is ok. I read through the whole comment to check for correctness and edit for clarity. I do think that this comment provides new relevant information that I hope will be helpful.

The Real-World Impact Today

Currently, Protobuf defines its crates via crate.from_specs() (generating repo @crates), while Crubit uses crate.from_cargo() (generating repo @crate_index). This results in:

1. Duplicate Compilation of Identical Crates: Even though both modules depend on the exact same versions (syn 2.0.119, proc-macro2 1.0.107, quote 1.0.47), Bazel treats them as distinct repos (@crates__syn-2.0.119 vs
   @crate_index__syn-2.0.119). They are compiled twice, wasting CI time and bloating remote caches.
2. Type Identity & Feature Hazards: In Rust, types are keyed to the crate target identity. If types or traits from shared dependencies ever cross Bazel module boundaries, rustc rejects them as mismatched types. Furthermore, neither
   module sees the other's required features, breaking Cargo's workspace feature-unification.
3. Lockfile Friction: Because extensions.bzl disallows unpinned transitive extensions (Transitive crate_universe repositories must ship a lockfile), every upstream library is forced to ship a static Cargo.Bazel.lock (e.g. see Provide pre-generated lockfiles for Protobuf Rust under Bzlmod. protocolbuffers/protobuf#29533). Downstream consumers cannot repin or reconcile dependencies across module boundaries without maintaining local patch files on rules_rust.

How Lockfiles and Splicing Fit Together

The existing discussion above suggested merging crate.from_cargo and crate.from_specs instances sharing the same repository name. A few observations on how that can work cleanly with lockfiles:

1. The Splicer already supports this: In cargo-bazel, SplicingManifest (splicing.rs) already accepts both manifests (paths to Cargo.toml) and direct_packages (crate.spec). The extension already has the machinery to
   combine them into a single synthetic workspace before running Cargo resolution—it just currently invokes _generate_hub_and_spokes per module tag in isolation.
2. Follow Cargo's lockfile ownership model: In Cargo, transitive Cargo.lock files are always ignored; only the root workspace lockfile governs. We should adopt the same principle in Bzlmod:
   • When a module like protobuf is built standalone in its own workspace, its local Cargo.Bazel.lock is used.
   • When consumed transitively in a root-unified universe, the root module's lockfile locks the entire spliced closure.
   • The check if not is_root and not lockfile: fail(...) should only apply if the transitive module is configured as isolated. If its requirements are merged into the root's unified universe, the requirement for the transitive module
     to ship its own lockfile is eliminated.
3. Prior art in Bzlmod and polyglot build systems:
   • Bazel / Gazelle (go_deps): Follows this exact model—the extension collects go_deps.module tags across all transitive Bzlmod modules, executes Go's MVS resolution once globally into a single shared set of repos (@com_github_...), and leaves pinning/overrides to the root module.
   • Buck2 / Reindeer & Cargo Workspaces: Other multi-package Rust setups (such as Buck2's Reindeer or Cargo workspaces analyzed via Guppy/cargo-hakari) consistently enforce a single, unified dependency graph at the workspace root to guarantee feature unification and eliminate redundant crate builds.
4. Hub Repositories may preserve call-site compatibility: If module A calls use_repo(crate, "crates") and module B calls use_repo(crate, "crate_index"), the extension can generate hub repositories for each requested name that alias
   to the unified underlying @crates__<pkg>-<version> targets.
5. Opt-in / Configurable: To prevent breaking workflows that legitimately require separate universes (e.g. std vs no_std / firmware targets with conflicting crate features), unification could be active by default for modules
   sharing a universe name (e.g. "crates"), while allowing isolated = True or an explicit custom universe name when strict isolation is desired.