Fetch a crate from any forge, not only github - #52
Merged
Conversation
rust_repo already covered both sources in one rule: a crate comes from
crates.io, or from a git forge with git_repo and git_revision. The forge
half was github only, and the docstring's claim that gitlab worked too
was never true. GitLab's /archive/ path answers 403; the tarball is at
/-/archive/<rev>/<project>-<rev>.tar.gz.
git_repo now takes a full URL as well as the owner/repo shorthand, and
the archive URL is built per forge scheme. Verified against each forge
rather than assumed:
github.com /archive/<rev>.tar.gz 200
codeberg /archive/<rev>.tar.gz 200
sr.ht /archive/<rev>.tar.gz 200
gitlab.com /-/archive/<rev>/<proj>-<rev>.tar.gz 200
gitlab.com /archive/<rev>.tar.gz 403
The scheme is inferred from the host, which cannot work for gitlab
running somewhere its name does not say, so git_forge names it outright.
sync --import records any other forge as the URL it was cloned from
rather than warning and telling the reader to write the rule by hand.
test/forge asserts the URLs while the package is parsed, so a change to
the scheme fails the build rather than a fetch. It caught the first
version of this: split(sep, 1) does not take a maxsplit in this dialect,
so every URL input was broken.
Nothing here needs git. A forge serving neither scheme still needs
download=, which is the escape hatch go-rules has for the same case.
On gitlab the revision is its own path segment followed by the archive name, so a tag containing a slash has to be escaped or gitlab reads the part before the slash as the whole ref. sequoia tags releases openpgp/v2.4.1 because one repository holds several crates. Unescaped it resolves today, but only because no branch is named openpgp; one appearing would silently fetch a different tree. The escaped URL is verified against gitlab and returns the same 9,468,234 bytes. Github takes the ref as the last segment before the extension, where a slash needs no escaping and escaping it would break the URL, so this applies to the gitlab scheme only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #21.
Where we already were
rust_repocovers both sources in one rule, which is thego_reposhape:So the gap was never the declaration shape, only which forges the fetch understood.
What was wrong
The archive URL was
https://{host}/{repo}/archive/{rev}.tar.gzwithhostdefaulting to github, andrust_reponever passedhostat all. The docstring claimed "github.com, gitlab works too". That was never true, checked rather than assumed:/archive/<rev>.tar.gz/archive/<rev>.tar.gz/archive/<rev>.tar.gz/-/archive/<rev>/<proj>-<rev>.tar.gz/archive/<rev>.tar.gzA gitlab repo declared as though it were github failed at download with nothing to say why.
sync --importwas worse than silent: it warned and skipped, telling the reader to write adownload=rule by hand.The change
git_repotakes a full URL as well as theowner/reposhorthand, and the archive URL is built per forge scheme. The host picks the scheme, which cannot work for gitlab running somewhere its name does not say, sogit_forge = "gitlab"says it outright.sync --importrecords any other forge as the URL it was cloned from.No git. This is
remote_fileand a URL, as it was before, so there is no host tooling and no new hermeticity question. A forge serving neither scheme still needsdownload=, which is the same escape hatch go-rules offers.Why we cannot do what the neighbours do
Worth recording, since it came up: both neighbours delegate forge support to the language toolchain.
go mod download, which resolves any module path through the Go module proxy. It never sees a forge, a URL or a git command.cargo update,cargo fetch,cargo generate-lockfile,cargo vendor.Neither is available here. Cargo is ruled out, and unlike Go's toolchain it has no standalone "fetch this repo at this revision" command anyway. So this is ours to build, and the per-forge table is the cost of not invoking cargo.
Testing
test/forgeasserts the URLs while the package is parsed, so a change to the scheme fails the build rather than a fetch, and no network is involved.It caught the first version of this change:
split(sep, 1)does not take a maxsplit in this dialect, so every URL input raisedindex out of range. Every URL form was broken and the unit test found it before anything was pushed.Covered: the shorthand, a github URL, a
.gitsuffix as aCargo.lockspells it, a gitlab group path (the tarball is named for the project, not the path), an explicitgit_forgeoverride, and the same host without one.The
--importhalf has a test asserting agit+https://gitlab.example.com/x/y#deadbeefsource now round-trips instead of being skipped. That fixture already existed, asserting it was skipped.189 tests pass, clippy clean, fmt clean.
Not covered
No CI step fetches from a real non-github forge, because no rule here fetches over the network in CI. The URLs are verified by hand against live forges (table above) and asserted at parse time. Next step is a real declaration in rust-corpus.