Skip to content

fix: cache 404 responses for remote modules#35138

Open
bartlomieju wants to merge 1 commit into
mainfrom
fix/cache-404-module-fetches
Open

fix: cache 404 responses for remote modules#35138
bartlomieju wants to merge 1 commit into
mainfrom
fix/cache-404-module-fetches

Conversation

@bartlomieju

Copy link
Copy Markdown
Member

A remote module that responds with 404 is never written to the HTTP cache,
so the URL is re-requested during graph resolution on every process start.
This shows up with published jsr packages whose JSDoc type references point
at non-existent files (every run of an installed CLI re-downloads the same
handful of 404ing URLs) and with dynamic imports that 404.

This stores a body-less cache entry marked with a synthetic
x-deno-not-found header, the same pattern used for redirects (body-less
entries detected via their location header), and serves "not found" from
the cache for 15 minutes. The TTL keeps a resource that appears on the
remote server later (e.g. a freshly published package version that is
still propagating) from being missed for long, while eliminating the
re-fetch on every run. Authenticated requests are not negatively cached
because a 404 can depend on the provided credentials (private registries
respond with 404 for unauthorized requests). --reload bypasses the
negative cache like it does everything else.

Closes #25404

…very run

A remote module that responded with 404 was never written to the HTTP
cache, so the URL was re-requested during graph resolution on every
process start (ex. JSDoc type references in published jsr packages that
point at non-existent files, or dynamic imports that 404).

Store a body-less cache entry marked with a synthetic x-deno-not-found
header (the same pattern as redirects, which are stored body-less and
detected via their location header) and serve "not found" from the
cache for 15 minutes. Authenticated requests are not negatively cached
since a 404 may depend on credentials. --reload bypasses the negative
cache as usual.

Closes #25404
@nathanwhit

Copy link
Copy Markdown
Member

Need to make sure this has a TTL wired up so that it picks up on files if they no longer 404

@bartlomieju

Copy link
Copy Markdown
Member Author

This is wired up. There's a 15-minute TTL (NOT_FOUND_CACHE_TTL in libs/cache_dir/file_fetcher/mod.rs):

  • a 404 is cached as a body-less entry with an x-deno-not-found header, and the cache records the download time on set
  • on read, both the global and local cache paths check is_not_found_entry_fresh: within the TTL the cached "not found" is served, once expired it's treated as a cache miss and the URL is re-requested
  • --reload bypasses it, and authenticated requests are never negatively cached (a 404 there can depend on credentials)

The expiry path is covered by test_fetch_not_found_is_negatively_cached: it caches a 404, flips the server to respond 200, confirms the cached 404 is still served while fresh, then advances the clock past the TTL and asserts the URL is re-requested and now resolves successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

deno install'd script re-downloads jsr typechecking dependencies on every run

2 participants