Skip to content

Fetch a crate from any forge, not only github - #52

Merged
jackmarsh merged 2 commits into
mainfrom
issue-21-git-forges
Aug 24, 2026
Merged

Fetch a crate from any forge, not only github#52
jackmarsh merged 2 commits into
mainfrom
issue-21-git-forges

Conversation

@jackmarsh

Copy link
Copy Markdown
Collaborator

Closes #21.

Where we already were

rust_repo covers both sources in one rule, which is the go_repo shape:

if git_repo:       download = rust_git_download(...)     # a forge
elif not download: download = rust_crate_download(...)   # crates.io

So the gap was never the declaration shape, only which forges the fetch understood.

What was wrong

The archive URL was https://{host}/{repo}/archive/{rev}.tar.gz with host defaulting to github, and rust_repo never passed host at all. The docstring claimed "github.com, gitlab works too". That was never true, checked rather than assumed:

forge scheme
github.com /archive/<rev>.tar.gz 200
codeberg (gitea/forgejo) /archive/<rev>.tar.gz 200
git.sr.ht /archive/<rev>.tar.gz 200
gitlab.com /-/archive/<rev>/<proj>-<rev>.tar.gz 200
gitlab.com /archive/<rev>.tar.gz 403

A gitlab repo declared as though it were github failed at download with nothing to say why.

sync --import was worse than silent: it warned and skipped, telling the reader to write a download= rule by hand.

The change

git_repo takes a full URL as well as the owner/repo shorthand, and the archive URL is built per forge scheme. The host picks the scheme, which cannot work for gitlab running somewhere its name does not say, so git_forge = "gitlab" says it outright.

sync --import records any other forge as the URL it was cloned from.

No git. This is remote_file and a URL, as it was before, so there is no host tooling and no new hermeticity question. A forge serving neither scheme still needs download=, which is the same escape hatch go-rules offers.

Why we cannot do what the neighbours do

Worth recording, since it came up: both neighbours delegate forge support to the language toolchain.

  • go-rules shells out to go mod download, which resolves any module path through the Go module proxy. It never sees a forge, a URL or a git command.
  • rules_rust shells out to cargo: cargo update, cargo fetch, cargo generate-lockfile, cargo vendor.

Neither is available here. Cargo is ruled out, and unlike Go's toolchain it has no standalone "fetch this repo at this revision" command anyway. So this is ours to build, and the per-forge table is the cost of not invoking cargo.

Testing

test/forge asserts the URLs while the package is parsed, so a change to the scheme fails the build rather than a fetch, and no network is involved.

It caught the first version of this change: split(sep, 1) does not take a maxsplit in this dialect, so every URL input raised index out of range. Every URL form was broken and the unit test found it before anything was pushed.

Covered: the shorthand, a github URL, a .git suffix as a Cargo.lock spells it, a gitlab group path (the tarball is named for the project, not the path), an explicit git_forge override, and the same host without one.

The --import half has a test asserting a git+https://gitlab.example.com/x/y#deadbeef source now round-trips instead of being skipped. That fixture already existed, asserting it was skipped.

189 tests pass, clippy clean, fmt clean.

Not covered

No CI step fetches from a real non-github forge, because no rule here fetches over the network in CI. The URLs are verified by hand against live forges (table above) and asserted at parse time. Next step is a real declaration in rust-corpus.

rust_repo already covered both sources in one rule: a crate comes from
crates.io, or from a git forge with git_repo and git_revision. The forge
half was github only, and the docstring's claim that gitlab worked too
was never true. GitLab's /archive/ path answers 403; the tarball is at
/-/archive/<rev>/<project>-<rev>.tar.gz.

git_repo now takes a full URL as well as the owner/repo shorthand, and
the archive URL is built per forge scheme. Verified against each forge
rather than assumed:

    github.com  /archive/<rev>.tar.gz                       200
    codeberg    /archive/<rev>.tar.gz                       200
    sr.ht       /archive/<rev>.tar.gz                       200
    gitlab.com  /-/archive/<rev>/<proj>-<rev>.tar.gz        200
    gitlab.com  /archive/<rev>.tar.gz                       403

The scheme is inferred from the host, which cannot work for gitlab
running somewhere its name does not say, so git_forge names it outright.

sync --import records any other forge as the URL it was cloned from
rather than warning and telling the reader to write the rule by hand.

test/forge asserts the URLs while the package is parsed, so a change to
the scheme fails the build rather than a fetch. It caught the first
version of this: split(sep, 1) does not take a maxsplit in this dialect,
so every URL input was broken.

Nothing here needs git. A forge serving neither scheme still needs
download=, which is the escape hatch go-rules has for the same case.
On gitlab the revision is its own path segment followed by the archive
name, so a tag containing a slash has to be escaped or gitlab reads the
part before the slash as the whole ref. sequoia tags releases
openpgp/v2.4.1 because one repository holds several crates.

Unescaped it resolves today, but only because no branch is named
openpgp; one appearing would silently fetch a different tree. The
escaped URL is verified against gitlab and returns the same 9,468,234
bytes.

Github takes the ref as the last segment before the extension, where a
slash needs no escaping and escaping it would break the URL, so this
applies to the gitlab scheme only.
@jackmarsh
jackmarsh merged commit b961592 into main Aug 24, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generic git fetcher for forges other than GitHub

1 participant