npm metadata fetch fails for large packages (renovate ~105 MB) - make maxMetadataSize configurable #149

Summary

The hardcoded 100 MB cap on upstream npm metadata responses in internal/handler/handler.go:56 makes some packages unservable through the proxy. I ran into this with renovate (~105 MB metadata document, 8,956 versions). It looks like it's the same for drizzle-orm.

const maxMetadataSize = 100 << 20  // 100 MB

Introduced by PR-111 (Verify cached artifacts on read).

Reproduction

# Against a v0.3.2 instance:
curl -i https://<your-proxy>/npm/renovate
# HTTP/2 502
# {"error":"failed to fetch from upstream"}

# Server log:
# ERROR npm metadata fetch failed
#   error="upstream failed and no cached metadata: reading response:
#          metadata response exceeds size limit"

# Tarballs are unaffected:
curl -i https://<your-proxy>/npm/renovate/-/renovate-43.0.0.tgz
# HTTP/2 200

# Upstream document size for reference:
curl -sL https://registry.npmjs.org/renovate | wc -c
# ~105000000

The proxy successfully serves every other npm package we've used it for so far (lodash, express, left-pad, thousands more). It only 502s when the upstream document exceeds 100 MB.

Fixes on similar issue don't help

I spotted issue #78 and PR-79 with a similar case for drizzle-orm - with your two fixes:

1. The cap was raised from 50 MB to 100 MB, and silent truncation became a loud ErrMetadataTooLarge. renovate is just slightly over the new ceiling rather than a magnitude past it.
2. When cooldown is disabled, the npm handler switches its Accept header to application/vnd.npm.install-v1+json (abbreviated metadata), which strips READMEs & bulk fields and keeps drizzle-orm under the cap.

The proble is fix 2. is gated on cooldown being disabled - the abbreviated metadata format lacks the time map that cooldown filtering relies on, so the npm handler can only switch to it when cooldown is off. We need cooldown on so the full-metadata path is the only path, and renovate is currently unserveable through the proxy. Thus, I think the cap just needs more configurable headroom.

I understand the need for different values for this for different deployments (memory budgets), hence suggesting a configurable value instead of just raising the constant.
The current Config already has similar variables like metadata_ttl and storage.max_size parsed via ParseSize ("100MB", "500MB", "2GB"); I think a metadata_max_size would fit in here.

Action

I have an idea how this could be done and I've read CONTRIBUTING.md. Would you like me to open a PR for this?