Move rust-analyzer from a git submodule to a git subtree

[fasterthanlime]

(Related to: long-term proc macro support plan)

Before (submodule)

RA is a git submodule in src/tools. When a "sync" happens, rust-lang/rust is updated to point to a different commit (example).

The changes suggested in #12803 (use the sysroot proc_macro crate) make RA/rustc incompatibilities more visible in that the submodule could then stop building.

Fixing that would involve two PRs:

• One to rust-analyzer, fixing proc-macro-srv to follow proc_macro bridge changes
• One to rust, bumping rust-analyzer

If the sync cycle isn't changed, this means the rust-analyzer RA component could be missing for up to a week, or even more, if the bridge changes haven't been followed in the rust-analyzer repo.

After (subtree)

Just like clippy, rust-analyzer would be a subtree (also in src/tools).

Feature development / bug fixes to RA happen as usual in the rust-lang/rust-analyzer repository. These can be synced to rust-lang/rust whenever convenient (clippy->rust example PR).

Changes to the proc_macro bridge are followed in the rust-lang/rust repository. These can be synced to rust-lang/rust-analyzer whenever convenient (rust->clippy example commit)

proc-macro-srv never breaks again (in the rustup component version of rust-analyzer), because it's immediately fixed as part of any PRs to rust-lang/rust.

Why does this matter?

A PR was just merged to "add support for the 1.64 ABI", but there's no such thing as "the 1.64 ABI", and it's going to be instabroken, see #12806.

Prior art

Similar discussion for rustfmt: rust-lang/rust#82385 (converted to a subtree in May 2021)

---

Tagging:

• @mystor who's been making proc_macro bridge changes, to confirm that rust-lang/rust folks are happy to keep the proc-macro-srv crate up-to-date with bridge changes (pinging other maintainers as needed)
• @jonas-schievink who's been keeping the "multi ABI support" approach up-to-date by manually vendoring+tuning several copies of the proc_macro bridge in the rust-analyzer repo
• @lnicola who's been doing most of the syncs I've seen in rust-lang/rust history
• @jyn514 who discussed using subtrees on Zulip
• @flodiebold, @Veykril, @bjorn3 who interacted with Resolving proc-macro ABI breakage for rustc nightly users: one way forward #12803
• @oli-obk who offered to connect with clippy folks in case RA folks have any questions about git subtree
• @cuviper who started work on getting RA tests to run on rust CI

[fasterthanlime]

Proposed course of action:

• S100: Convert rust-analyzer to a git subtree
• S200: Tackle Resolving proc-macro ABI breakage for rustc nightly users: one way forward #12803 ("sysroot ABI") in the rust-lang/rust tree, taking extra care to cfg-gate it properly
• S300: Sync in both directions, make sure rust-lang/rust-analyzer still builds on its own, on stable
• S400: Discuss next steps required to improve the ergonomics of Resolving proc-macro ABI breakage for rustc nightly users: one way forward #12803 further, like:
  • Having the rust-analyzer LSP discover/use the "rustup rust-analyzer component's proc-macro subcommand"
  • Splitting proc-macro-srv into its own rustup component
  • Reaching out to IntelliJ Rust folks who currently pull the proc-macro-srv crate directly from r-a's git repo

[lnicola]

I haven't used subtrees yet, so I don't know how involved the switch would be, but I don't like git submodules very much. So this seems like a good idea, assuming the rust-lang/rust contributors are fine with doing the extra work for RA.

But will all breaking changes show up on CI? I'm not sure if the RA tests are being run in rust-lang/rust.

[fasterthanlime]

But will all breaking changes show up on CI? I'm not sure if the RA tests are being run in rust-lang/rust.

@cuviper has done some work re: that in rust-lang/rust#99444, see also #12803 (comment)

The intent was to track rust-analyzer status in rust-toolstate, but I believe it could be repurposed to run the proc-macro-srv tests on rust CI, blocking PR merges until they pass. (More tests could be run, but those are the important ones imho)

[fasterthanlime]

I haven't used subtrees yet, so I don't know how involved the switch would be, but I don't like git submodules very much.

The rustc dev guide has a section on git subtree: https://rustc-dev-guide.rust-lang.org/contributing.html#external-dependencies-subtree

@oli-obk told me to quite literally "send y'all to clippy folks" to ask any questions you might have, they've been doing this for a while.

[fasterthanlime]

So this seems like a good idea, assuming the rust-lang/rust contributors are fine with doing the extra work for RA.

@mystor seemed okay with it "provided it's relatively easy". I don't know about other proc_macro bridge maintainers.

I've done it once myself to get an idea of how much work it is (and @jonas-schievink can confirm/deny). It's mostly:

• Removing functions that were removed from the bridge interface (in the various traits like FreeFunctions, Span, MultiSpan etc.)
• Adding functions that were added to the bridge interface (a lot of things in proc-macro-srv are stubs right now, so this is relatively easy).
• Following type/struct shape changes, see this diff for the recent move from Group being an assoc type to it being a struct.

[fasterthanlime]

For research purposes, I opened a draft PR on which rust-analyzer is a git subtree and ./x.py test rust-analyzer runs RA tests:

• Re-add rust-analyzer as a subtree rust#99465

It shows how the move would happen. All tests pass through ./x.py test rust-analyzer except for tidy::check_merge_commits — it might not make sense to run that one from the rust copy of rust-analyzer.

I also found clippy's actual git subtree guide, which was immensely helpful (cc @lnicola):

• https://github.com/rust-lang/rust-clippy/blob/master/book/src/development/infrastructure/sync.md