Apply cooldown to Maven

README.md

@@ -51,7 +51,7 @@ The proxy never uploads artifact bytes to a scanner. Each scanner is notified wi
 | Hex | Elixir | Yes* | ✓ |
 | pub.dev | Dart | Yes | ✓ |
 | PyPI | Python | Yes | ✓ |
-| Maven | Java | | ✓ |
+| Maven | Java | Yes‡ | ✓ |
 | Gradle Build Cache | Java/Kotlin | | ✓ |
 | NuGet | .NET | Yes | ✓ |
 | Composer | PHP | Yes | ✓ |
@@ -79,6 +79,8 @@ Cooldown requires publish timestamps in metadata. Registries without a "Yes" in
 
 † Go cooldown uses the time in each version's `.info`, which is the commit time of the tagged revision, not when the tag was pushed.
 
+‡ Maven cooldown uses the `Last-Modified` header of each version's `.pom`. On Maven Central that is the deploy time; a Nexus or Artifactory upstream may report when it cached the file instead. SNAPSHOT versions are not held back.
+
 ## Install
 
 ```bash

docs/configuration.md

@@ -980,10 +980,19 @@ Exact `packages` entries take precedence over patterns. When several patterns ma
 
 Resolution order: exact package override, then package pattern, then ecosystem override, then global default. This lets you set a conservative default while exempting trusted package families.
 
-Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex and Go. These ecosystems include publish timestamps in their metadata.
+Currently supported for npm, PyPI, pub.dev, Composer, Cargo, NuGet, Conda, RubyGems, Hex, Go and Maven. Most of these ecosystems include publish timestamps in their metadata; Go and Maven are described below.
 
 Go has no publish time. The module proxy's `.info` gives the commit time of the revision a version points at, so a version tagged on an old commit counts as old. The proxy reads each listed version's `.info` once, stores the time, and leaves versions in cooldown out of `@v/list` and `@latest`. When upstream's `@latest` is in cooldown, it serves the newest release out of cooldown instead. Requests for a version in cooldown, including its `.info`, `.mod` and `.zip`, and a branch or commit query that resolves to one, get 403 rather than 404, because the go command moves on to the next `GOPROXY` entry, such as `direct`, after a 404. If a version's `.info` can't be fetched, the list or download fails with 502 instead of letting the version through.
 
+Maven metadata has no publish time per version. The proxy uses the `Last-Modified` header of each version's `.pom`, which on Maven Central is the time the version was deployed. It reads the header once per version and stores the time, for each repository under `upstream.maven_repositories` separately: from the response when the `.pom` or Gradle `.module` itself is downloaded, which clients do before a version's other files, and otherwise with a `HEAD` request. A `.pom` or `.module` refused this way is still cached, so it counts toward `max_size` before the version leaves cooldown. To keep requests to the upstream down, it only checks versions that can still be in cooldown: none when the metadata's `<lastUpdated>` is older than the cooldown, and otherwise the versions from the end of the list back to the first one that is old enough. Versions in cooldown are left out of `maven-metadata.xml`, `<latest>` and `<release>` are set to the last versions left, and the metadata's `.md5`, `.sha1`, `.sha256` and `.sha512` files are computed from what is served. Requests for any file of a version in cooldown get 403, as for a version on the denylist. If a `.pom` can't be checked, the metadata or download fails with 502 instead of letting the version through, or with 503 and `Retry-After` when the upstream rate-limits the proxy; a version without a `.pom` is let through.
+
+Maven caveats:
+
+- A Nexus or Artifactory upstream may report when it cached the `.pom` rather than when it was published, so a version can stay in cooldown longer than it should.
+- On Google's Maven Central mirror, `Last-Modified` is when the file reached the mirror's bucket, a few hours after it was published, so versions stay in cooldown a little longer. The upload time is in `x-goog-meta-last-modified`, which the proxy does not read. It is missing for files from the mirror's 2019 bulk import.
+- Maven Central lists versions in version order, not in the order they were published. A patch to an older release line published after a newer release that is out of cooldown stays listed while it is in cooldown, but its files are still refused.
+- SNAPSHOT versions are not held back.
+
 With cooldown on, the proxy asks npm for full packuments, since the abbreviated ones it uses otherwise carry no publish times. Abbreviated packuments cached before cooldown was turned on are fetched again in full rather than served, including as a stale copy when the upstream is down.
 
 Note: Hex cooldown requires disabling registry signature verification since the proxy re-encodes the protobuf payload without the original signature. Set `HEX_NO_VERIFY_REPO_ORIGIN=1` or configure your repo with `no_verify: true`.

internal/handler/handler.go

@@ -1965,7 +1965,7 @@ func (p *Proxy) getOrFetchArtifactFromURLWithCachePURLs(ctx context.Context, eco
 	if p.StreamArtifacts {
 		return p.streamFromUpstream(ctx, ecosystem, name, version, filename, versionPURL, downloadURL, upstreamHash, declared,
 			func(fetchCtx context.Context) (*fetch.Artifact, error) {
-				return p.Fetcher.FetchWithHeaders(fetchCtx, downloadURL, headers)
+				return p.fetchUpstream(fetchCtx, downloadURL, headers)
 			})
 	}
 	if cached, err := p.getCachedArtifactWithUpstreamHash(ctx, pkgPURL, versionPURL, filename, upstreamHash); err != nil {
@@ -2040,7 +2040,7 @@ func (p *Proxy) fetchAndCacheFromURL(ctx context.Context, ecosystem, name, versi
 		"ecosystem", ecosystem, "name", name, "version", version, "url", downloadURL)
 
 	fetchStart := time.Now()
-	artifact, err := p.Fetcher.FetchWithHeaders(ctx, downloadURL, headers)
+	artifact, err := p.fetchUpstream(ctx, downloadURL, headers)
 	metrics.RecordUpstreamFetch(ecosystem, time.Since(fetchStart))
 	if err != nil {
 		metrics.RecordUpstreamError(ecosystem, "fetch_failed", metrics.FailureDetail{
@@ -2054,6 +2054,43 @@ func (p *Proxy) fetchAndCacheFromURL(ctx context.Context, ecosystem, name, versi
 	return p.storeArtifact(ctx, ecosystem, name, version, filename, pkgPURL, versionPURL, downloadURL, upstreamHash, declared, artifact)
 }
 
+type upstreamLastModifiedKey struct{}
+
+// WithUpstreamLastModified returns a context under which an artifact fetched
+// from upstream reports its Last-Modified to fn, so a handler can learn a
+// version's publish time from the download itself rather than with a
+// request of its own. fn is not called for a cache hit, or when the
+// response has no usable Last-Modified.
+func WithUpstreamLastModified(ctx context.Context, fn func(time.Time)) context.Context {
+	return context.WithValue(ctx, upstreamLastModifiedKey{}, fn)
+}
+
+// observedFetcher is the part of fetch.Fetcher and its circuit breaker that
+// reports response headers.
+type observedFetcher interface {
+	FetchObservedWithHeaders(ctx context.Context, url string, headers http.Header) (*fetch.ObservedArtifact, error)
+}
+
+// fetchUpstream fetches downloadURL, reporting its Last-Modified when ctx
+// asks for it.
+func (p *Proxy) fetchUpstream(ctx context.Context, downloadURL string, headers http.Header) (*fetch.Artifact, error) {
+	report, _ := ctx.Value(upstreamLastModifiedKey{}).(func(time.Time))
+	observer, ok := p.Fetcher.(observedFetcher)
+	if report == nil || !ok {
+		return p.Fetcher.FetchWithHeaders(ctx, downloadURL, headers)
+	}
+	observed, err := observer.FetchObservedWithHeaders(ctx, downloadURL, headers)
+	if err != nil {
+		return nil, err
+	}
+	if observed.Observation != nil {
+		if t, err := http.ParseTime(observed.Observation.Headers.Get(headerLastModified)); err == nil {
+			report(t)
+		}
+	}
+	return observed.Artifact, nil
+}
+
 // streamFromUpstream fetches an artifact when StreamArtifacts is set and hands its
 // body to the caller without storing it.
 //

internal/handler/maven.go

@@ -1,6 +1,7 @@
 package handler
 
 import (
+	"context"
 	"crypto/sha256"
 	"encoding/hex"
 	"errors"
@@ -170,32 +171,48 @@ func (s mavenSource) cacheFilename(filename string) string {
 const mavenUpstreamHashBytes = 8
 
 func (h *MavenHandler) handleMetadata(w http.ResponseWriter, r *http.Request, source mavenSource, urlPath string) {
-	cacheKey := source.metadataCacheKey(urlPath)
+	if h.cooldownEnabled() && h.serveCooldownMetadata(w, r, source, urlPath) {
+		return
+	}
+
+	body, contentType, cacheKey, err := h.fetchMetadata(r.Context(), source, urlPath)
+	if err != nil {
+		h.writeMetadataError(w, err)
+		return
+	}
+
+	h.proxy.writeMetadataCachedResponse(w, r, "maven", cacheKey, body, contentType)
+}
+
+// fetchMetadata fetches a metadata file from the source's upstream, or from
+// its fallback when the upstream doesn't have it, and returns it with the
+// key it is cached under.
+func (h *MavenHandler) fetchMetadata(ctx context.Context, source mavenSource, urlPath string) (body []byte, contentType, cacheKey string, err error) {
+	cacheKey = source.metadataCacheKey(urlPath)
 	upstreamURL := fmt.Sprintf("%s/%s", source.upstreamURL, urlPath)
 
-	body, contentType, err := h.proxy.FetchOrCacheMetadata(r.Context(), "maven", cacheKey, upstreamURL, "*/*")
+	body, contentType, err = h.proxy.FetchOrCacheMetadata(ctx, "maven", cacheKey, upstreamURL, "*/*")
 	if err != nil && source.fallbackURL != "" {
 		if errors.Is(err, ErrUpstreamNotFound) {
 			pluginPortalURL := fmt.Sprintf("%s/%s", source.fallbackURL, urlPath)
 			h.proxy.Logger.Info("maven metadata unavailable in primary upstream, trying Gradle Plugin Portal",
 				"path", urlPath)
-			body, contentType, err = h.proxy.FetchOrCacheMetadata(r.Context(), "maven", cacheKey, pluginPortalURL, "*/*")
+			body, contentType, err = h.proxy.FetchOrCacheMetadata(ctx, "maven", cacheKey, pluginPortalURL, "*/*")
 		}
 	}
-	if err != nil {
-		if errors.Is(err, ErrUpstreamNotFound) {
-			http.Error(w, "not found", http.StatusNotFound)
-			return
-		}
-		if writeUpstreamHeld(w, err) {
-			return
-		}
-		h.proxy.Logger.Error("metadata fetch failed", "error", err)
-		http.Error(w, "failed to fetch from upstream", http.StatusBadGateway)
+	return body, contentType, cacheKey, err
+}
+
+func (h *MavenHandler) writeMetadataError(w http.ResponseWriter, err error) {
+	if errors.Is(err, ErrUpstreamNotFound) {
+		http.Error(w, "not found", http.StatusNotFound)
 		return
 	}
-
-	h.proxy.writeMetadataCachedResponse(w, r, "maven", cacheKey, body, contentType)
+	if writeUpstreamHeld(w, err) {
+		return
+	}
+	h.proxy.Logger.Error("metadata fetch failed", "error", err)
+	http.Error(w, "failed to fetch from upstream", http.StatusBadGateway)
 }
 
 // handleDownload serves an artifact file, fetching and caching from upstream if needed.
@@ -208,6 +225,14 @@ func (h *MavenHandler) handleDownload(w http.ResponseWriter, r *http.Request, so
 		return
 	}
 
+	// The .pom of a version whose time isn't known yet carries it in its
+	// own Last-Modified, so it is checked once fetched, without a request
+	// of its own. Maven fetches the .pom before a version's other files.
+	ctx, checkAfter := h.cooldownFromDownload(r.Context(), source, urlPath, version)
+	if !checkAfter && h.checkCooldown(w, r, source, urlPath, version) {
+		return
+	}
+
 	// Maven uses group:artifact as the package name
 	name := fmt.Sprintf("%s:%s", group, artifact)
 
@@ -217,19 +242,23 @@ func (h *MavenHandler) handleDownload(w http.ResponseWriter, r *http.Request, so
 	upstreamURL := fmt.Sprintf("%s/%s", source.upstreamURL, urlPath)
 	cacheFilename := source.cacheFilename(filename)
 
-	result, err := h.proxy.GetOrFetchArtifactFromURL(r.Context(), "maven", name, version, cacheFilename, upstreamURL)
+	result, err := h.proxy.GetOrFetchArtifactFromURL(ctx, "maven", name, version, cacheFilename, upstreamURL)
 	if err != nil && source.fallbackURL != "" {
 		if errors.Is(err, ErrUpstreamNotFound) {
 			pluginPortalURL := fmt.Sprintf("%s/%s", source.fallbackURL, urlPath)
 			h.proxy.Logger.Info("maven artifact not found in primary upstream, trying Gradle Plugin Portal",
 				"group", group, "artifact", artifact, "version", version, "filename", filename)
-			result, err = h.proxy.GetOrFetchArtifactFromURL(r.Context(), "maven", name, version, cacheFilename, pluginPortalURL)
+			result, err = h.proxy.GetOrFetchArtifactFromURL(ctx, "maven", name, version, cacheFilename, pluginPortalURL)
 		}
 	}
 	if err != nil {
 		h.proxy.serveArtifactError(w, err, "failed to fetch artifact")
 		return
 	}
+	if checkAfter && h.checkCooldown(w, r, source, urlPath, version) {
+		_ = result.Reader.Close()
+		return
+	}
 
 	ServeArtifactRequest(w, r, result)
 }