MODULE.bazel.lock is diff-unfriendly for large crate universes

[jblebrun]

We recently transitioned to using bzlmod to manage our 3 rather large crate universes. It's working great, and I really like not having to worry about the individual lockfiles.

There's one problem: The MODULE.bazel.lock file has become huge; it takes about 2x the space that the original lockfiles did. But worse, it's very diff-unfriendly. It contains entire BUILD files, and changings one crate will cause the entire master BUILD file to change, resulting in a pages-long inscrutable diff. This makes it challenging to review changes that include crate universe modifications, and often overloads diff viewer tools.

I have put together a proof of concept that reduces the MODULE.bazel.lock size to be comparable to the old lockfiles it replaces, but more importantly, it removes the very long BUILD content lines, so that changes affecting only one or two crates don't change the MODULE.bazel.lock file so dramatically.

Proof of concept is here: main...jblebrun:rules_rust:lockfile-opt

I'm not ready to make a PR with this, since I got quite a bit of help from AI to generate it, and I want to fully understand what it's doing before expecting others to do the same.

But I figured I'd open this issue, in case anyone else is struggling with the issue as well, is exploring possible resolutions already, or is interested in helping to review and refine my proof-of-concept solution.

[mgeisler]

Thanks for creating this issue! I have also recently started with Bazel and Rust and was wondering if others have the same problem 🙂

In my case, I have these files:

• Cargo.lock: 650 crates, 160 kB.
• MODULE.bazel.lock: 2.3 MB, 10k lines, wc -L MODULE.bazel.lock tells me the longest single line is 280k characters.

[mgeisler]

I see #3900 is discussing the size part of this problem, but not the long lines.

[mgeisler]

I asked an AI about this, and found a work-around that I had not considered: adding

MODULE.bazel.lock -diff

to .git/info/attributes makes Git stop computing a diff. The diff is still there, of course, but you no longer see it in your tools.

You can also use .gitattributes to make it apply to everyone who uses the repository.

[jblebrun]

I asked an AI about this, and found a work-around that I had not considered: adding

MODULE.bazel.lock -diff

to .git/info/attributes makes Git stop computing a diff. The diff is still there, of course, but you no longer see it in your tools.

You can also use .gitattributes to make it apply to everyone who uses the repository.

Ah OK nice, that is handy.

Unfortunately, many of us on this project have start using jj for my day-to-day workflows, and it seems that it does not respect this setting, and doesn't yet have a comparable setting of its own. But this helps in at least some scenarios, thanks!

[jblebrun]

I see #3900 is discussing the size part of this problem, but not the long lines.

I did see this one, but it was discussing Cargo.bazel.lock, not MODULE.bazel.lock. My proposed change gets MODULE.bazel.lock back to a size comparable to the Cargo.bazel.lock that we used to have (~2MB instead of ~4MB for our project), I'd be happy even with that.

Further reductions would be nice too, so I will keep an eye on that issue.

[jblebrun]

(If I understand it correctly, I think that roughly, my proposed change ends up storing the Cargo.bazel.lock-style info int he MODULE.bazel.lock, rather than storing the build files directly)

[hofbi]

Could be a duplicate of #3522, or at least be related. I haven't really found a good solution for rules_rust yet. Maybe you want to check out https://github.com/hermeticbuild/rules_rs.

[UebelAndre]

For everyone's awareness, the rules_rust maintainers are going to focus on Bazel 9 support and deprecating WORKSPACE (#3818) before picking up crate_universe work. By removing WORKSPACE support we will be able to clean up a ton of complexity in the crate_universe implementation that makes making changes difficult. We plan to have this done in the near future.

[jblebrun]

@UebelAndre That is exciting news, thanks for the update! I will sit on this for a bit then. :-)

[mgeisler]

Yes, great to hear!

I found #2505 today and this made me wonder: should we even version the MODULE.bazel.lock file? At work, we version a Cargo.lock file and this might actually be enough to keep things consistent for my team?