Add package routes for private Composer and npm registries

[simonchrz]

Add package routes for private Composer and npm registries

Problem

Teams that point Composer or npm at the proxy as their only registry can't install private packages through it. Today each ecosystem has a single upstream. Private packages therefore need a second repository in composer.json / .npmrc, which brings two problems:

• Private packages bypass the cache.
• Composer is open to dependency confusion: with the proxy listed first, a public package with the same name as a private one takes precedence.

Solution

Add upstream.composer_routes and upstream.npm_routes. These map package name patterns to private registries:

upstream:
  composer_routes:
    "example/*": "https://packages.example.com/composer"
  npm_routes:
    "@example/*": "https://packages.example.com/npm"
  allow_private_hosts: ["packages.example.com"]
  auth:
    "https://packages.example.com":
      type: bearer
      token: "${PRIVATE_REGISTRY_TOKEN}"

• Strict routing. A matching package is fetched only from its route. If the route returns 404, 401 or 5xx, the client gets that error (404 or 502). The proxy never falls back to the default upstream, so a public package can't stand in for a private one.
• Separate cache identity. Metadata cache keys and artifact cache filenames of routed packages include a digest of the route URL. Entries cached from the public registry before a route was added are never served for a routed package, not even as a stale fallback. Changing a route's URL invalidates its entries too.
• Composer dev versions. A route also covers the package's {name}~dev.json file. Patterns are matched against the name without ~dev, while the upstream URL and cache key keep it.
• npm cooldown. The publish time of a routed version always comes from the route's metadata. The shared versions row, keyed by the plain version PURL, is neither read nor written for routed packages, so public and private times of the same name never mix, and a route's previous upstream doesn't lend its times to the new one.
• Downloads. Composer dist URLs and npm tarball URLs of routed packages are rewritten through the proxy, as before, and resolved against the route. npm tarballs are validated against the route's host and path.
• No name leak via Composer notifications. Routed Composer versions get an empty notification-url, set so that every version has it once Composer expands the minified format. Otherwise Composer reports their installs to the default repository's notify-batch (Packagist).
• Patterns. Patterns use path.Match syntax, case-insensitive. * does not cross /, and the longest pattern wins. Routes are validated at startup.
• Credentials. They reuse the existing upstream.auth and allow_private_hosts settings.

Not covered (documented)

• Composer search.json / packages/list.json and npm audit / signing-key requests still go to the default upstream. An npm audit request contains the routed package names, which matches what npm does today when a scope points to another registry.
• Routes are file-only, like the other map-valued upstream settings (helm, generic, …).

Tests

• Config validation for both route settings.
• Matcher: wildcards, longest match, case handling, cache-key separation.
• Composer and npm handler tests:
  • routed metadata comes only from the route
  • no fallback on 404/401/500
  • pre-cached public metadata and artifacts are ignored
  • unrouted packages are unaffected
  • downloads resolve against the route, also when npm metadata is unavailable
  • an exact Composer route covers the ~dev metadata file (regression)
  • every expanded version of minified routed Composer metadata has an empty notification-url, across inheritance, ~dev resets and an upstream URL that is set and unset
  • npm cooldown uses the route's publish times: a route added after a public download, the reverse order, and a route moved to another upstream (regression)
• Server test that the config reaches both handlers.

go test ./..., go test -race ./internal/handler/, go vet ./... and golangci-lint run ./... (v2.13.1) all pass.

[andrew]

Two routing and cache-isolation failures need fixing. Both were reproduced through the HTTP handlers.

[andrew]

An exact route for example/library does not match /p2/example/library~dev.json, because packageName still contains ~dev. This sends the private package's development metadata request to the default public registry. A handler-level reproduction returned public metadata without contacting the private registry. Strip ~dev for route matching while preserving it in the upstream URL and cache key, and add a regression test for an exact package route.

[simonchrz]

Thanks, confirmed. Fixed in fef12ac: sourceFor now matches routes against the package name without ~dev, and keeps the suffix in the upstream URL and the cache key, so the stable and dev files stay apart.

TestComposerExactRouteCoversDevMetadata uses an exact route for example/library and requests /p2/example/library~dev.json. Before the fix it got the public document without contacting the route. Now only the route is queried and the dev versions are rewritten through the proxy. The route docs mention the ~dev file too.

[andrew]

The metadata cache key is route-qualified here, but versionInCooldown still reads and writes publish times using the unqualified package/version PURL. I reproduced this by downloading a 30-day-old public release, then adding a private route whose same package/version was published one hour ago: with a seven-day cooldown, the private tarball returned HTTP 200. Scope stored publish times to the selected upstream, and cover adding or changing a route with existing version rows.

[simonchrz]

Thanks, confirmed, and it went both ways: a private time written to the shared row could also hold back the public version. Fixed in 8553245.

For routed packages, versionInCooldown no longer reads or writes the versions row. The publish time always comes from the route's metadata, which is already cached under the route's key. I went with that rather than scoping the row itself, because a route-qualified version row would also show up in the UI and enrichment. The cost is that an uncached routed download with cooldown on reads the route's cached metadata again instead of one row.

TestNPMRoutedCooldownUsesRoutePublishTimes covers your reproduction (month-old public release, hour-old private one, 7-day cooldown), the reverse order, and a route moved to another upstream. All three failed before the fix. It's the only place that reads stored publish times for cooldown.

I also rebased onto main for #418. The routed notification-url is now set so that every version has it after Composer expands the minified format, with a test across inheritance and ~dev resets.