Skip to content

Extract portable npm metadata transformations - #612

Merged
andrew merged 2 commits into
git-pkgs:mainfrom
simonchrz:refactor/npm-portable-metadata
Oct 26, 2026
Merged

andrew merged 2 commits into
git-pkgs:mainfrom
simonchrz:refactor/npm-portable-metadata

Conversation

@simonchrz

Copy link
Copy Markdown
Contributor

Supersedes #519, which @andrew wrote and said we could take over. His commit is kept as the first commit, rebased onto main; the second is mine. Part of #501.

As in #519, the npm metadata rewrite moves out of internal/handler into internal/metadata/npm, and the JSON scanner into internal/jsonscan, shared by npm and Composer. The rewrite takes everything as explicit inputs (proxy URL, cooldown policy, the time cooldown is evaluated at, denied versions, pushed versions) and reports what it did through callbacks the handler logs. It doesn't depend on handler.Proxy, so it pulls in no SQL drivers, cloud SDKs, metrics, HTTP or logging. Fetching, storage, the rewrite cache and HTTP responses stay in the handler.

Main changed a lot of this code after #519 branched. Where each change went:

Changes from #519:

  • The tarball callback reports a count instead of one call per version (see above).
  • npm.Abbreviated is new.
  • imports_test.go in internal/metadata/npm runs go list -deps and fails on any non-standard import outside an allowlist (semver, cooldown, purl, vers, packageurl-go, internal/jsonscan, internal/packageurl), or on net, net/http, log/slog or database/sql. database/sql/driver comes in through semver, as it already did in Extract portable npm metadata transformations #519. It's only interfaces, with no driver.
  • jsonscan has a package comment.

Cooldown is checked with Evaluate at one EvaluatedAt per rewrite instead of IsAllowed per version, as in #519. The handler passes time.Now(), so results are the same unless a rewrite runs across a cooldown boundary.

Portability: GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/npm ./internal/jsonscan works, and both packages' tests pass when compiled for wasip1 and run under Node's WASI. The imports test skips there because it has no go tool. I didn't try TinyGo.

Benchmarks (M5 Pro, internal/handler, main vs this branch, test binaries run alternately, n=12, benchstat):

main branch
NPMRewriteMetadata 45.20 ms 44.26 ms ~ (p=0.068)
NPMVersionTarball 16.74 ms 16.60 ms ~ (p=0.29)
ComposerRewriteMetadata 9.80 ms 9.75 ms ~ (p=0.89)
ComposerFindDownloadURL 3.43 ms 3.54 ms +3% (p=0.02)

Bytes and allocations per op are unchanged. A first, non-alternating run had ComposerFindDownloadURL at −7.5% and NPMVersionTarball at +6%, so I read these differences as noise. The scanner code is the same apart from the renames.

The npm rewrite unit tests moved to internal/metadata/npm with the code. The handler tests stay, adjusted to the new names. go test ./..., go test -race on handler, metadata and jsonscan, and golangci-lint pass.

andrew and others added 2 commits October 25, 2026 22:21
npm metadata rewriting depended on handler.Proxy, which pulls SQL
drivers, cloud SDKs and metrics into its import graph. Move the
transformation into internal/metadata/npm with explicit policy,
evaluation time, public URL and hosted-version inputs, and share the
JSON scanner between npm and Composer through internal/jsonscan.

Rebased onto main: rewritten tarball URLs are still logged once per
document, now through an OnTarballsRewritten callback that reports the
count.
Whether a packument is the abbreviated format is a property of the
document, so it sits with the rest of the npm metadata code; the
handler keeps only the error it turns that into. Add a test that keeps
the package's imports to the portable set, give jsonscan a package
comment, and describe the new packages in the architecture docs.
@andrew
andrew merged commit 2998a97 into git-pkgs:main Oct 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants