Repository navigation
Extract portable npm metadata transformations - #612
Merged
Merged
Conversation
npm metadata rewriting depended on handler.Proxy, which pulls SQL drivers, cloud SDKs and metrics into its import graph. Move the transformation into internal/metadata/npm with explicit policy, evaluation time, public URL and hosted-version inputs, and share the JSON scanner between npm and Composer through internal/jsonscan. Rebased onto main: rewritten tarball URLs are still logged once per document, now through an OnTarballsRewritten callback that reports the count.
Whether a packument is the abbreviated format is a property of the document, so it sits with the rest of the npm metadata code; the handler keeps only the error it turns that into. Add a test that keeps the package's imports to the portable set, give jsonscan a package comment, and describe the new packages in the architecture docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #519, which @andrew wrote and said we could take over. His commit is kept as the first commit, rebased onto main; the second is mine. Part of #501.
As in #519, the npm metadata rewrite moves out of
internal/handlerintointernal/metadata/npm, and the JSON scanner intointernal/jsonscan, shared by npm and Composer. The rewrite takes everything as explicit inputs (proxy URL, cooldown policy, the time cooldown is evaluated at, denied versions, pushed versions) and reports what it did through callbacks the handler logs. It doesn't depend onhandler.Proxy, so it pulls in no SQL drivers, cloud SDKs, metrics, HTTP or logging. Fetching, storage, the rewrite cache and HTTP responses stay in the handler.Main changed a lot of this code after #519 branched. Where each change went:
OnTarballRewrite(version, old, new)becameOnTarballsRewritten(count), called at most once per document. The handler still logs onerewrote tarball URLsdebug line with the count.npm.Abbreviated.packumentAccept,packumentValidatorand theerrAbbreviatedPackumentthey return stay in the handler, since they use the proxy's config and metadata cache.mustFilterMetadata, 09e41d0), the stale warning on filtered version metadata (994a936), the cooldown rewrite cache with its one-minute TTL (f4cab2d),storedRewrite/storedRewriteVariant(Cache Helm and PyPI metadata rewrites #600), stale fallback (Flag stale Composer and Chef metadata #541), metrics (Separate artifact and metadata cache metrics on /metrics and the analytics page #463, Count cached metadata against max_size and evict it LRU #507), the tarball and cooldown download checks (Don't fetch npm tarballs whose packument can't be read #582, 224bce8), npm search routing and debug-level request logs: all depend on the DB, cache, metrics or HTTP, so they stay in the handler as main has them. The cached rewrite now calls into the package.jsonscan.Changes from #519:
npm.Abbreviatedis new.imports_test.goininternal/metadata/npmrunsgo list -depsand fails on any non-standard import outside an allowlist (semver, cooldown, purl, vers, packageurl-go,internal/jsonscan,internal/packageurl), or onnet,net/http,log/slogordatabase/sql.database/sql/drivercomes in through semver, as it already did in Extract portable npm metadata transformations #519. It's only interfaces, with no driver.Cooldown is checked with
Evaluateat oneEvaluatedAtper rewrite instead ofIsAllowedper version, as in #519. The handler passestime.Now(), so results are the same unless a rewrite runs across a cooldown boundary.Portability:
GOOS=wasip1 GOARCH=wasm go build ./internal/metadata/npm ./internal/jsonscanworks, and both packages' tests pass when compiled for wasip1 and run under Node's WASI. The imports test skips there because it has nogotool. I didn't try TinyGo.Benchmarks (M5 Pro,
internal/handler, main vs this branch, test binaries run alternately, n=12, benchstat):Bytes and allocations per op are unchanged. A first, non-alternating run had ComposerFindDownloadURL at −7.5% and NPMVersionTarball at +6%, so I read these differences as noise. The scanner code is the same apart from the renames.
The npm rewrite unit tests moved to
internal/metadata/npmwith the code. The handler tests stay, adjusted to the new names.go test ./...,go test -raceon handler, metadata and jsonscan, and golangci-lint pass.