Load crates.io dependencies of the standard library

[jonas-schievink]

The standard library may depend on crates from crates.io, and some of them can be required to properly analyze libstd, like cfg_if, which caused #6038 in the past. We currently use a hard-coded dependency graph for the sysroot, but have no mechanism to detect or obtain the sources of crates.io dependencies.

We should figure out a way to properly solve this. There are a few options:

• The rust-src component could vendor the sources of crates.io dependencies (this seems insufficient though, we might still need to address one of the points below).
• We can use the Cargo.lock of the rust-lang/rust monorepo to resolve libstd dependencies via cargo metadata, but this requires creating a temporary directory or mutating rustup-installed directories, which is a bad idea.
• The compiler and standard library could be split into 2 workspaces, each with their own Cargo.lock. Then we could run cargo metadata --locked without requiring any disk operations besides index and crate downloads.

[jyn514]

We can use the Cargo.lock of the rust-lang/rust monorepo to resolve libstd dependencies via cargo metadata, but this requires creating a temporary directory or mutating rustup-installed directories

Why does this require mutating rustup directories? Why doesn't cargo metadata --locked work?

[jonas-schievink]

Because the lockfile is in $SYSROOT/lib/rustlib/src/rust/Cargo.lock, while the manifest of libstd is in $SYSROOT/lib/rustlib/src/rust/library/std/Cargo.toml. Running cargo metadata --locked expects the lockfile to be next to Cargo.toml (because the workspace configuration is not shipped with rust-src either).

[jyn514]

Would it help if rust-src shipped with a symlink from Cargo.lock to library/std/Cargo.lock? That would be a lot easier than separating the standard library into its own workspace.

[jonas-schievink]

The lockfile is currently shipped as part of the rustc-src component, so I think we'd have to duplicate that. It seems that even then, cargo metadata --locked will fail, since the lockfile contains all compiler dependencies too and would have to be updated to just contain libstd dependencies.

[Aaron1011]

@jonas-schievink What's the current status of this issue? Setting RustcSource::Discover, it looks like rust-analyzer was able to run cargo metadata for rustc-src successfully.

[bjorn3]

That only works for rustc-src, not rust-src as most people will have.