Skip to content

Build in version-pinned paths for well-known download hosts - #459

Merged
andrew merged 2 commits into
git-pkgs:mainfrom
gilesw:feat/generic-builtin-artifacts
Oct 13, 2026
Merged

andrew merged 2 commits into
git-pkgs:mainfrom
gilesw:feat/generic-builtin-artifacts

Conversation

@gilesw

@gilesw gilesw commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #424. upstream.generic_artifacts made non-GitHub downloads cacheable as artifacts, but every user caching mise's Maven, Node or Go downloads has to write the same regexes. Without them, those tarballs stay on the metadata route: revalidated after metadata_ttl, buffered against metadata_max_size, and not listed as packages.

This adds built-in patterns keyed by upstream base URL, so naming the host is enough:

upstream:
  generic:
    apache: "https://archive.apache.org"
    nodejs: "https://nodejs.org"
    golang: "https://dl.google.com/go"
Upstream URL Cached as artifacts Package name
https://archive.apache.org dist/{project dirs}/{version}/[binaries/]{file}, {version} starting with a digit the project dirs, e.g. maven/maven-3, maven/mvnd
https://nodejs.org dist/v{version}/{file} the upstream name
https://dl.google.com/go go{version}.{os}-{arch}.tar.gz, .zip and their .sha256 the upstream name
  • Built-ins apply whatever the upstream is named; the URL is compared after trimming a trailing /.
  • generic_artifacts patterns add to the built-ins, so existing configs keep working, and duplicates of the built-in patterns become redundant.
  • Index and listing paths (dist/index.json, dist/maven/maven-3/) don't match and stay on the metadata route.
  • The docs list the built-in hosts. The generic_artifacts example now uses a host that isn't built in, and the README mise section gets a Maven/Node/Go recipe with no patterns.

Tested with go test ./... and go vet, and by running the proxy with only the generic: block above against the real hosts. The first fetch of apache-maven-3.9.9-bin.tar.gz took 79s and the second 8ms from cache. mvnd, Node SHASUMS256.txt and Go .sha256 files were also served from the artifact cache on the second fetch. /api/search?ecosystem=generic lists maven/maven-3, maven/mvnd, nodejs and golang. golangci-lint was not run locally.

Generic upstreams pointed at archive.apache.org, nodejs.org or
dl.google.com/go now cache their versioned downloads as artifacts without
any generic_artifacts patterns. Configured patterns add to the built-in ones.

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The built-in patterns are added before the configured generic_artifacts in NewGenericHandler, and parseArtifact returns the first match, so a built-in overrides an overlapping user pattern. The current README recipe (dist/(?P<name>maven)/maven-3/... on https://archive.apache.org) will now resolve to maven/maven-3 instead of maven. The artifact cache is keyed on the version PURL, so everything already cached as pkg:generic/maven@... misses and is downloaded again, the old package stays listed alongside the new one, and any version deny or cooldown rule written against pkg:generic/maven stops matching.

Please compile the configured patterns first and the built-ins after them, so explicit config wins, and add a test case where a configured pattern overlaps a built-in and its name is the one returned.

The first matching pattern names the package, so an upstream's own
patterns now take precedence over overlapping built-ins and keep the
package names, cache entries and rules already keyed on them.
@gilesw

gilesw commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. Configured generic_artifacts are now compiled before the built-ins, so explicit config wins on overlap. TestGenericHandler_BuiltinArtifactPatterns now configures the README maven recipe on the apache upstream and checks that maven-3 paths resolve to maven, while maven-4 and mvnd paths still fall through to the built-in. The configuration docs now say configured patterns are tried first.

@andrew
andrew merged commit a23ced2 into git-pkgs:main Oct 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants