'cargo metadata' fails on read-only file system

[RalfJung]

Problem

I first reported this with rust-analyzer at rust-lang/rust-analyzer#10792: when running cargo metadata on a read-only file system, it fails saying

rust-analyzer failed to load workspace: Failed to read Cargo metadata for Rust sources: Failed to run `cargo metadata --manifest-path /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml`: `cargo metadata` exited with an error:     Updating crates.io index
error: failed to write /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.lock

Caused by:
  failed to open: /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.lock

Caused by:
  Read-only file system (os error 30)
rust-analyzer failed to load workspace: Failed to read Cargo metadata for Rust sources: Failed to run `cargo metadata --manifest-path /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml`: `cargo metadata` exited with an error:     Updating crates.io index
error: failed to write /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.lock

Caused by:
  failed to open: /home/r/.rustup/toolchains/miri/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.lock

Caused by:
  Read-only file system (os error 30)

However, the action RA is trying to perform is fundamentally a read-only operation (querying for information about the rustc compiler crates).

I don't think the currently available flags provide any way to make this work on a read-only file system either (I tried --frozen but as expected it refuses to download required crates from the network). Even if my toolchain was on a read-write file system I would not want RA's cargo metadata to change toolchain files, after all.

Proposed Solution

cargo metadata should either handle read-only file systems by falling back to not creating/updating the lock file -- or (IMO the right fix) it should not even attempt to mutate the workspace when what it is doing is a read-only operation like just determining the metadata.

Notes

No response

[ehuss]

Can you say more about your use case about why the Cargo.lock is out of date? You wouldn't be able to run any other cargo command, either, so just allowing cargo metadata to work with an out-of-date lockfile may not help much.

[bjorn3]

The rustc-src rustup component doesn't have the root Cargo.toml so the root Cargo.lock isn't used. Rust-analyzer runs cargo metadata for the compiler/rustc crate which normally doesn't need it's own Cargo.lock as it uses the workspace's version, but because the root Cargo.toml is missing, this isn't possible.

[RalfJung]

You wouldn't be able to run any other cargo command, either, so just allowing cargo metadata to work with an out-of-date lockfile may not help much.

AFAIK cargo metadata is the only command that needs to work for RA to be able to proceed here -- but I am not sure and I wouldn't know how to test this.

[aos]

Hi all - I ran into this issue while loading stdlib from a read-only location (attempting to access it from Nix store). I dug into this a bit and we can trace it down to resolve_with_registry writing the lock file:

cargo/src/cargo/ops/resolve.rs

Lines 206 to 208 in b816d82

	if !ws.is_ephemeral() && ws.require_optional_deps() {
	ops::write_pkg_lockfile(ws, &mut resolve)?;
	}

Since we don't write the lock file if the workspace is already marked as ephemeral, can we make the assumption that a read-only file system is inherently "ephemeral" and mark the workspace as such when creating it? The only other place where we check for is_ephemeral is when preloading a provided registry here:

cargo/src/cargo/core/workspace.rs

Lines 1029 to 1037 in b816d82

	pub fn preload(&self, registry: &mut PackageRegistry<'cfg>) {
	// These can get weird as this generally represents a workspace during
	// `cargo install`. Things like git repositories will actually have a
	// `PathSource` with multiple entries in it, so the logic below is
	// mostly just an optimization for normal `cargo build` in workspaces
	// during development.
	if self.is_ephemeral {
	return;
	}

Perhaps the metadata command can be instantiated with an ephemeral workspace? Or checking to see if the manifest_path is read-only, and then doing some operation to not allow the lock file to be written. Unfortunately, this is my only usage of this command, so I don't know other use cases. Either way, I'm happy to work on this one!

[bjorn3]

I don't think a read-only fs ahould be considered ephemeral. It can be compiled (or get metadata retrieved) twice in whoch case the same dependency versions should be used. I see rust-analyzer only reloading cargo metadata when Cargo.* changes as an optimization. cargo metadata should be an idempotent operation. That is running it twice will result in the same result. If there is no Cargo.lock that will not be true. The real fix would be to ship a correct Cargo.lock file as part of the rustc-src component.

[aos]

That makes sense - thanks for the insight.

[RalfJung]

I would say even more important than being idempotent is it being a read-only operation. It is merely querying some information and state from a crate. If that information depends on things like the crate registry, I am not surprised by multiple queries returning different results. (I would not expect curl to be idempotent, either.) However, I am severely surprised by such a command changing the on-disk state of my project, and even more surprised when there is no fallback for when performing that unexpected mutation fails.

[leoluk]

I would say even more important than being idempotent is it being a read-only operation.

Most cargo commands will mutate the lockfile by default - you need to specify --frozen or --locked to prevent that.

[RalfJung]

Yeah but those will just fail in that case, so that is not really helpful either.

I honestly don't think there is a good reason that a query like cargo metadata requires read-write access to the project files, no matter whether the lockfile is somehow out-of-date or not. Sure we could work around this issue by shipping a slightly different lockfile, but IMO that's a hack, no a solution.

[bjorn3]

Sure we could work around this issue by shipping a slightly different lockfile, but IMO that's a hack, no a solution.

The problem is that the lockfile is outdated. It is the solution. Ignoring lockfile write errors is a hack IMO. cargo metadata --frozen is the only mode of operation I would expect to be entirely side effect free. Anything else may need to download crates from crates.io and store them in ~/.cargo. In addition cargo build is conceptually just cargo metadata followed by deriving a list of build commands from the result. If cargo metadata didn't write the lockfile, cargo build wouldn't either.

[RalfJung]

In addition cargo build is conceptually just cargo metadata followed by deriving a list of build commands from the result. If cargo metadata didn't write the lockfile, cargo build wouldn't either.

I see no good reason why it has to be like that. It certainly does not match my mental model at all.

cargo build has good reasons to fail when called on a crate stored on a read-only FS. cargo metadata IMO does not.

Anything else may need to download crates from crates.io and store them in ~/.cargo.

Populating a cache is (conceptually) not a side-effect (formally speaking: a memoized function is observably pure). So I don't mind it changing ~/.cargo. That is categorically different from mutating the crate it is working on.