Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 179 additions & 7 deletions fetch.bs
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,19 @@ Translate IDs: typedefdef-bodyinit bodyinit,dictdef-requestinit requestinit,type
urlPrefix:https://httpwg.org/specs/rfc5861.html#;type:dfn;spec:stale-while-revalidate
url:n-the-stale-while-revalidate-cache-control-extension;text:stale-while-revalidate lifetime

urlPrefix:https://httpwg.org/specs/rfc9842.html#;type:dfn;spec:rfc9842
url:use-as-dictionary;text:Use-As-Dictionary
url:available-dictionary;text:Available-Dictionary
url:dictionary-id;text:Dictionary-ID
url:rfc.section.2.2;text:finding the best matching dictionary

urlPrefix:https://httpwg.org/specs/rfc9651.html#;type:dfn;spec:rfc9651
url:rfc.section.2;text:structured field value
url:text-serialize;text:serializing structured fields
url:text-parse;text:parsing structured fields
url:;text:structured header
url:token;text:structured field token
url:parse-bare-item;text:bare item

urlPrefix:https://httpwg.org/specs/rfc9110.html#;type:dfn;spec:http
url:method.overview;text:method
Expand Down Expand Up @@ -67,6 +74,11 @@ urlPrefix:https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-layered-cooki
url:name-serialize-cookies;text:serialize cookies
url:name-garbage-collect-cookies;text:garbage collect cookies

urlPrefix:https://urlpattern.spec.whatwg.org/#;type:dfn;spec:urlpattern
url:url-pattern-create;text:creating a URL pattern
url:url-pattern-has-regexp-groups;text:has regexp groups


urlPrefix:https://www.rfc-editor.org/rfc/rfc6454;type:dfn;spec:RFC6454
url:section-7.1;text:serialized-origin
</pre>
Expand Down Expand Up @@ -1822,6 +1834,7 @@ device to assist defining CSP and Mixed Content. It is not exposed to JavaScript
the empty string,
"<code>audio</code>",
"<code>audioworklet</code>",
"<code>compression-dictionary</code>",
"<code>document</code>",
"<code>embed</code>",
"<code>font</code>",
Expand Down Expand Up @@ -1878,7 +1891,7 @@ not always relevant and might require different behavior.
<th>CSP directive
<th>Features
<tr>
<td rowspan=22>""
<td rowspan=23>""
<td>"<code>report</code>"
<td rowspan=2>&mdash;
<td>CSP, NEL reports.
Expand Down Expand Up @@ -1970,6 +1983,10 @@ not always relevant and might require different behavior.
<td>"<code>video</code>"
<td><code>media-src</code>
<td>HTML's <code>&lt;video></code> element
<tr>
<td>"<code>compression-dictionary</code>"
<td><code>connect-src</code>
<td>HTML's <code>&lt;link rel=compression-dictionary></code>
<tr>
<td>"<code>download</code>"
<td>""
Expand Down Expand Up @@ -2326,9 +2343,10 @@ When a request is [=request/cloned=], the created request gets a unique

<p>A <dfn export>subresource request</dfn> is a <a for=/>request</a>
whose <a for=request>destination</a> is "<code>audio</code>", "<code>audioworklet</code>",
"<code>font</code>", "<code>image</code>", "<code>json</code>", "<code>manifest</code>",
"<code>paintworklet</code>", "<code>script</code>", "<code>style</code>", "<code>text</code>",
"<code>track</code>", "<code>video</code>", "<code>xslt</code>", or the empty string.
"<code>compression-dictionary</code>", "<code>font</code>", "<code>image</code>",
"<code>json</code>", "<code>manifest</code>", "<code>paintworklet</code>", "<code>script</code>",
"<code>style</code>", "<code>text</code>", "<code>track</code>", "<code>video</code>",
"<code>xslt</code>", or the empty string.

<p>A <dfn export>non-subresource request</dfn> is a <a for=/>request</a>
whose <a for=request>destination</a> is "<code>document</code>", "<code>embed</code>",
Expand Down Expand Up @@ -3326,6 +3344,22 @@ or an <a>implementation-defined</a> value.
</div>


<h3 id=compression-dictionary-cache-partitions>Compression-dictionary cache partitions</h3>

<div algorithm>
<p>To <dfn>determine the compression-dictionary cache partition</dfn>, given a <a for=/>request</a> <var>request</var>:

<ol>
<li><p>Let <var>key</var> be the result of <a for=request>determining the network partition key</a>
given <var>request</var>.

<li><p>If <var>key</var> is null, then return null.

<li><p>Return the unique compression-dictionary cache associated with <var>key</var>. [[!RFC9842]]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should make compression-dictionary cache a link.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mirrors the section above on HTTP cache partitions where it does "Return the unique HTTP cache associated with...". The RFC itself doesn't go into detail about the cache itself and how it operates. Is that something that needs to be fully defined somewhere?

</ol>
</div>


<h3 id=port-blocking>Port blocking</h3>

<p class=note>New protocols can avoid the need for blocking ports by negotiating the protocol
Expand Down Expand Up @@ -6435,8 +6469,9 @@ run these steps:
<li><p>If <var>httpRequest</var>'s <a for=request>cache mode</a> is
"<code>only-if-cached</code>", then return a <a>network error</a>.

<li><p>Let <var>forwardResponse</var> be the result of running <a>HTTP-network fetch</a> given
<var>httpFetchParams</var>, <var>includeCredentials</var>, and <var>isNewConnectionFetch</var>.
<li><p>Let <var>forwardResponse</var> be the result of running
<a>HTTP-network compression-dictionary fetch</a> given <var>httpFetchParams</var>,
<var>includeCredentials</var>, and <var>isNewConnectionFetch</var>.

<li><p>If <var>httpRequest</var>'s <a for=request>method</a> is <a>unsafe</a> and
<var>forwardResponse</var>'s <a for=response>status</a> is in the range 200 to 399, inclusive,
Expand Down Expand Up @@ -6585,12 +6620,148 @@ run these steps:
<li><p>If <var>isAuthenticationFetch</var> is true, then create an <a>authentication entry</a> for
<var>request</var> and the given realm.

<li>
<p>If <var>request</var>'s <a for=request>response tainting</a> is not "<code>opaque</code>"
and <var>response</var>'s <a for=response>header list</a> <a for="header list">contains</a>
<a><code>Use-As-Dictionary</code></a>:
<!-- This is defined in [[!RFC9842]] -->

<ol>
<li><p>Let <var>dictionaryValue</var> be the result of
<a for="header list">getting a structured field value</a> given <a><code>Use-As-Dictionary</code></a>,
"<code>dictionary</code>", and <var>response</var>'s <a for=response>header list</a>.

<li><p>If <var>dictionaryValue</var> is null or <var>dictionaryValue</var>["<code>match</code>"]
does not <a for=map>exist</a>, then return <var>response</var>.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about other parameters?

For example https://www.rfc-editor.org/rfc/rfc9842#name-type says a client should not deal with a type value it does not understand, so I guess we should bail out if that happens.

For id what happens if it exceeds the 1024 characters limit?

For match-dest, I guess clients would just ignore unknown destinations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added details on how to handle invalid values for the other parameters. I don't explicitly call out that unknown keys should be ignored but I can add an explicit step to delete unknown keys if you think it would help.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was not sure what is the common way to handle this, however https://httpwg.org/specs/rfc9651.html#preserving-extensibility mentions this:

Specifications that use Dictionaries can also allow for forward compatibility by requiring that the presence of -- as well as value and type associated with -- unknown keys be ignored. Subsequent specifications can then add additional keys, specifying constraints on them as appropriate.

So maybe the spec should be explicit about ignoring unknown keys.

Incidentally, https://httpwg.org/specs/rfc9651.html#error-handling says ignoring the entire field is the default behavior when field-specific constraints are violated so it's indeed good the spec is now explicit for id, type, etc that we only ignore the corresponding key.

(note: probably rfc9651 should also have similar wording)


<li><p>If <var>dictionaryValue</var>["<code>type</code>"] <a for=map>exists</a> and its
<a>bare item</a> is not an <a>implementation-defined</a> supported dictionary type, then return
<var>response</var>.

<li><p>If <var>dictionaryValue</var>["<code>id</code>"] <a for=map>exists</a> and its
<a>bare item</a>'s <a for=string>length</a> is greater than 1024, then <a for=map>remove</a>
<var>dictionaryValue</var>["<code>id</code>"].

<li>
<p>If <var>dictionaryValue</var>["<code>match-dest</code>"] <a for=map>exists</a>:

<ol>
<li><p>Let <var>matchDestList</var> be <var>dictionaryValue</var>["<code>match-dest</code>"][0].

<li><p>For each <var>dest</var> of <var>matchDestList</var>: if <var>dest</var>'s <a>bare item</a>
is not a <a for=request>destination</a> supported by the user agent, then remove <var>dest</var>
from <var>matchDestList</var>.

<li><p>If <var>matchDestList</var> is empty, then return <var>response</var>.
</ol>

<li><p>Let <var>compressionDictionaryCache</var> be the result of
<a>determining the compression-dictionary cache partition</a> given <var>request</var>.

<li><p>If <var>compressionDictionaryCache</var> is null, then return <var>response</var>.
Comment thread
pmeenan marked this conversation as resolved.

<li><p>Let <var>pattern</var> be the result of
<a for=/>creating a URL pattern</a> given the <a>bare item</a> of <var>dictionaryValue</var>["<code>match</code>"],
the <a lt="URL serializer">serialization</a> of <var>request</var>'s <a for=request>current URL</a>,
and an empty map. If this throws an exception, then return <var>response</var>.

<li><p>If <var>pattern</var> is failure or <var>pattern</var> <a for=/>has regexp groups</a>,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"creating a URL pattern" uses algos from https://url.spec.whatwg.org/ that can return failure, but when that happens it actually throw an exception, rather than returning failure?

I wanted to ask about these potential exceptions the other day, but couldn't find an obvious input here that would make the algo throw... Anyway, I find this spec indeed deals with the case when an exception is thrown so I guess we probably want the same here: https://wicg.github.io/connection-allowlists/#abstract-opdef-parse-a-connection-allowlist-header

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also seems we lack tests for this. We should have somewhat exhaustive tests for error conditions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I added similar language that it should return response if the URL Pattern creation throws an exception. I'll add WPT tests for that and the rest of the edge cases that aren't currently covered now (will take a few days to work their way through).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just landed a (hopefully comprehensive) set of WPT tests to cover all of the edge cases I could think of: web-platform-tests/wpt#60164

  • Invalid and out-of-scope match properties (syntax failures and cross-origin patterns).
  • Invalid/unsupported dictionary type tokens.
  • Comprehensive match-dest parsing (unknown destinations, matching/non-matching destinations, and explicit wildcards).
  • Dictionary id maximum length validation (exactly 1024 characters vs exceeding 1025 characters).
  • Robustness against unknown additional dictionary parameters.
  • Rejection of entirely malformed structured headers (Use-As-Dictionary: ?0) and non-cacheable responses (max-age=0).
  • Opaque response tainting resulting from cross-origin redirects under no-cors mode correctly ignoring registration.
  • Evaluation of Available-Dictionary on matching redirect targets across redirect chains.
  • Precedence evaluation when matching overlapping pattern scopes.
  • Decoding failures resulting from dictionary hash mismatches throwing clean network errors for both dcb and dcz.

then return <var>response</var>.

<li><p>Let <var>expirationTime</var> be the time at which the <var>response</var> becomes
a <a>stale response</a>.

<li><p>If <var>expirationTime</var> is not in the future, then return <var>response</var>.

<li><p>Store <var>response</var> in <var>compressionDictionaryCache</var> with its associated
<var>pattern</var>, <var>dictionaryValue</var>, and <var>expirationTime</var>.
</ol>

<li><p>Return <var>response</var>. <span class=note>Typically <var>response</var>'s
<a for=response>body</a>'s <a for=body>stream</a> is still being enqueued to after
returning.</span>
</ol>
</div>

<h3 id=http-network-compression-dictionary-fetch>HTTP-network compression-dictionary fetch</h3>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if here and below we also want a hyphen between network and compression. cc @sideshowbarker

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I was copying how HTTP-network fetch is handled which has a space in the string but hyphens in the ID (<h3 id=http-network-fetch>HTTP-network fetch</h3>). Happy to go either way though.


<div algorithm>
<p>To <dfn id=concept-http-network-compression-dictionary-fetch>HTTP-network compression-dictionary fetch</dfn>,
given a <a for=/>fetch params</a> <var>fetchParams</var>, an optional boolean
<var>includeCredentials</var> (default false), and an optional boolean <var>forceNewConnection</var>
(default false):

<ol>
<li><p>Let <var>request</var> be <var>fetchParams</var>'s <a for="fetch params">request</a>.

<li>
<p>Let <var>fallback</var> be the following steps:

<ol>
<li><p>Return the result of running <a>HTTP-network fetch</a> given <var>fetchParams</var>,
<var>includeCredentials</var>, and <var>forceNewConnection</var>.
</ol>

<li><p>If <var>request</var>'s <a for=request>mode</a> is "<code>no-cors</code>", then return the
result of running <var>fallback</var>.

<li><p>If the user agent is configured to block cookies for <var>request</var>, then return the
result of running <var>fallback</var>.

<li><p>Let <var>compressionDictionaryCache</var> be the result of
<a>determining the compression-dictionary cache partition</a> given <var>request</var>.

<li><p>If <var>compressionDictionaryCache</var> is null, then return the result of running
<var>fallback</var>.

<li><p>Let <var>bestMatch</var> be the result of <a>finding the best matching dictionary</a> in
<var>compressionDictionaryCache</var> for <var>request</var>.

<li><p>If <var>bestMatch</var> is null, then return the result of running <var>fallback</var>.

<li><p>Add the <a><code>Available-Dictionary</code></a> and <a><code>Dictionary-ID</code></a>
(if applicable) headers to <var>request</var> using <var>bestMatch</var>.

<li><p><a for="header list">append</a> (`<code>Accept-Encoding</code>`, `<code>dcb</code>`)
to <var>request</var>'s <a for=request>header list</a>.

<li><p><a for="header list">append</a> (`<code>Accept-Encoding</code>`, `<code>dcz</code>`)
to <var>request</var>'s <a for=request>header list</a>.

<li><p>Let <var>response</var> be the result of running <var>fallback</var>.

<li><p>Let <var>codings</var> be the result of <a>extracting header list values</a> given
`<code>Content-Encoding</code>` and <var>response</var>'s <a for=response>header list</a>.

<li><p>If <var>codings</var> is null or does not contain `<code>dcb</code>` or `<code>dcz</code>`,
then return <var>response</var>.

<li><p>If <var>request</var>'s <a for=request>response tainting</a> is "<code>opaque</code>",
then return a <a>network error</a>.

<li><p>Let <var>availableDictionaryItem</var> be the result of
<a for="header list">getting a structured field value</a> given <a><code>Available-Dictionary</code></a>,
"<code>item</code>", and <var>request</var>'s <a for=request>header list</a>.

<li><p>If <var>availableDictionaryItem</var> is null, then return a <a>network error</a>.

<li><p>Let <var>availableDictionaryHash</var> be the <a>bare item</a> of <var>availableDictionaryItem</var>.

<li><p>Let <var>newBody</var> be a new <a for=/>body</a> whose <a for=body>stream</a> is the
result of transforming <var>response</var>'s <a for=response>body</a>'s <a for=body>stream</a>
with an algorithm that verifies that the dictionary hash in the stream matches
<var>availableDictionaryHash</var> and decodes the rest of the stream with the applicable
algorithm as defined in [[!RFC9842]]. If verification or decoding fails,
error the transformed stream.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what's implied by this "error the transformed stream". Shouldn't we return a network error if decoding/verification fails? If not what do we expect for newBody in that case?

Also, about the hash verification, I don't see a lot from RFC 9842... Looking for "hash" or "available-dictionary", I find these paragraphs:

and in particular "The dictionary is validated using an SHA-256 hash of the content to make sure that the client and server are both using the same dictionary." from which one can infer the server is expected to send back the same hash if it has found the same on its side.

So can we actually just add a previous step that checks availableDictionaryHash matches the Available-Dictionary in request's header at step 9 and return a network error otherwise. Or Am I missing something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

availableDictionaryHash and Available-Dictionary are both generated by the client and attributes of the request. The issue we need to protect against is a server responding with a dcb or dcz stream that was compressed with a dictionary other than the one requested (has happened to more than a few people when deploying because of incorrectly-configured Vary response headers).

I'm happy to change it to be a network error of some kind if there's a sensible way to plumb that. Where it gets a bit complicated is that it's a problem with the stream, not necessarily the HTTP-level response container and is closer to being a corrupt payload (like sending a brotli payload with Content-Encoding: zstd). It also won't show up until the body starts being processed/read.

That feels like it's a problem at the stream level rather than the network level but I'm happy to plumb it however it best fits into the spec.


<li><p>Set <var>response</var>'s <a for=response>body</a> to <var>newBody</var>.

<li><p><a for="header list">delete</a> `<code>Content-Encoding</code>` from <var>response</var>'s
<a for=response>header list</a>.

<li><p>Return <var>response</var>.
</ol>
</div>

<h3 id=http-network-fetch>HTTP-network fetch</h3>

Expand Down Expand Up @@ -6662,6 +6833,7 @@ optional boolean <var>forceNewConnection</var> (default false), run these steps:

<ul>
<li><p>Follow the relevant requirements from HTTP. [[!HTTP]] [[!HTTP-CACHING]]
[[!RFC9842]]

<li>
<p>If <var>request</var>'s <a for=request>body</a> is non-null, and <var>request</var>'s
Expand Down Expand Up @@ -8527,7 +8699,7 @@ dictionary RequestInit {
any window; // can only be set to null
};

enum RequestDestination { "", "audio", "audioworklet", "document", "embed", "font", "frame", "iframe", "image", "json", "manifest", "object", "paintworklet", "report", "script", "sharedworker", "style", "text", "track", "video", "worker", "xslt" };
enum RequestDestination { "", "audio", "audioworklet", "compression-dictionary", "document", "embed", "font", "frame", "iframe", "image", "json", "manifest", "object", "paintworklet", "report", "script", "sharedworker", "style", "text", "track", "video", "worker", "xslt" };
enum RequestMode { "navigate", "same-origin", "no-cors", "cors" };
enum RequestCredentials { "omit", "same-origin", "include" };
enum RequestCache { "default", "no-store", "reload", "no-cache", "force-cache", "only-if-cached" };
Expand Down