Skip to content

Add KV Connect protocol v4: multiplexed watch channel over WebSocket - #166

Draft
piscisaureus wants to merge 4 commits into
mainfrom
watch-channel-v4
Draft

piscisaureus wants to merge 4 commits into
mainfrom
watch-channel-v4

Conversation

@piscisaureus

Copy link
Copy Markdown
Member

Adds KV Connect protocol version 4: a multiplexed watch channel.

Today every watch() call opens its own long-lived streaming request,
so a client watching many key sets holds many concurrent streams, and
every reconnect re-delivers state the client already has. Version 4
multiplexes all watches for a database over one WebSocket connection:

  • Keys are added/removed in-band (WatchChannelClientMessage), so the
    watched set changes without new connections. The add message
    carries an optional baseline (last seen versionstamp, or "absent");
    the server only sends state that differs from it. Re-adding all keys
    with their baselines after a reconnect makes resumption lossless and
    duplicate-free.
  • Server messages are key-tagged and sparse (only changed keys),
    unlike the positional v3 WatchOutput.
  • RemoteTransport gains optional WebSocket support via defaulted
    methods, so this is semver-compatible: existing transports keep
    compiling, keep advertising [1, 2, 3], and keep the per-watch v3
    path. Only transports that implement websocket() negotiate v4.
    Nothing changes for any current consumer until its transport opts
    in.
  • The denokv server gains the /watch_channel endpoint (axum ws)
    behind the same auth, diffs updates against per-client known state,
    enforces a per-channel key cap (1024, close code 1008), and pings
    every 5s. Metadata negotiation now prefers v4.
  • The watch() stream contract is identical under both protocols:
    first item is a full snapshot, later items mark untouched keys
    Unchanged, in request key order — covered by integration tests
    that exercise the channel path (shared channel across watches,
    initial snapshots, per-key updates, re-subscription) next to the
    existing v3 test.

Spec for the new endpoint and messages is in proto/kv-connect.md
("Watch Channel (version 4)").

@piscisaureus

Copy link
Copy Markdown
Member Author

Pushed a revision addressing findings from a deeper review pass:

  • Fallback to per-watch streaming: if the channel repeatedly cannot
    be established or keeps dying young (five consecutive failed or
    short-lived connections — e.g. an intermediary that strips the
    Upgrade header, or the server closing with a policy violation),
    subscriptions now fail with a marker error and watch() falls back
    to the version 3 streaming path instead of silently retrying
    forever. Spec updated to permit this. Covered by a new integration
    test with a transport whose websocket connections always fail.
  • Subscribe/reconnect race: registering a subscription now resets
    the stored baseline of its keys (online and offline paths share one
    code path), so a reconnect that swallows the in-flight add still
    re-requests the state the new subscription needs; previously its
    initial snapshot could hang until the next write.
  • Permission tightening: the channel driver only connects to
    endpoint URLs that passed the caller's check_net_url in watch();
    endpoints introduced later by metadata refreshes trigger the
    fallback (where the v3 path re-checks permissions per request)
    instead of being connected to unchecked. The check also no longer
    silently no-ops when metadata is briefly unavailable.
  • v3 behavior parity: watch(vec![]) yields one empty initial
    snapshot (it used to hang on v4); watches with more than
    MAX_WATCHED_KEYS keys use the v3 path so server-side limit behavior
    is identical across versions; a malformed server frame now causes a
    lossless reconnect instead of failing every subscription on the
    channel.
  • Efficiency: entries are shared via Arc during fan-out instead
    of deep-copied per subscriber; the server batches bursts of
    add/remove messages into one watcher rebuild, skips rebuilds on
    remove entirely, and validates key sizes on add; both protocol
    paths now share a single entry-conversion helper; the hand-rolled
    abort guard was replaced with tokio_util::task::AbortOnDropHandle.

One known follow-up not addressed here: the npm client throws
"Unsupported version: 4" if a user explicitly passes
supportedVersions: [1,2,3,4] against an upgraded server — the TS
client should clamp its advertised versions to those it implements.

@piscisaureus
piscisaureus force-pushed the watch-channel-v4 branch 2 times, most recently from 3bb0b70 to 7b708ab Compare June 10, 2026 22:55
Watching n keys currently costs one long-lived streaming request per
watch() call. Protocol version 4 adds a watch channel: a single
WebSocket connection per database that key watches can be added to
and removed from at any time, with support for resuming after a
reconnect without duplicate deliveries.

* proto: add WatchChannelClientMessage (add/remove a watched key; add
  carries an optional resume baseline: the last seen versionstamp or
  "absent") and WatchChannelServerMessage (key-tagged outputs carrying
  only state that differs from what the client has) to
  datapath.proto. Spec the endpoint and message flow in kv-connect.md
  and list version 4 in the protocol versions section. The limits
  module of denokv_proto is now public.
* remote: RemoteTransport gains optional WebSocket support through
  defaulted methods, so existing transport implementations keep
  compiling and keep negotiating version 3 with the per-watch code
  path. Transports that implement websocket() advertise version 4
  during the metadata exchange. A per-database channel manager
  refcounts watched keys across watch() calls, demultiplexes updates
  to the per-call streams (entries shared via Arc, copied only at the
  caller-facing conversion), and reconnects with jittered backoff,
  re-adding every key with its last seen state as the baseline.
* remote: when the channel cannot be made to work, watch() falls back
  to per-watch streaming instead of retrying forever: after five
  consecutive failed or short-lived connections (e.g. an intermediary
  strips the Upgrade header, or the server closes the channel with a
  policy violation), subscriptions fail with a marker error that
  watch() converts into the version 3 code path. The same fallback
  covers endpoints introduced by metadata refreshes that have not
  passed the caller's permission check: the driver only connects to
  endpoints approved in watch(), and the per-watch path re-checks
  permissions on every request. Watches with more than
  MAX_WATCHED_KEYS keys or an empty key list also use version 3
  semantics.
* server: add the /watch_channel WebSocket endpoint, gated on
  x-denokv-version: 4. The server diffs watcher output against what
  each channel client is known to have, validates key sizes, enforces
  a per-channel key limit (close code 1008), batches bursts of
  add/remove messages into a single watcher rebuild (removals need no
  rebuild at all), and pings every 5s. The metadata endpoint now
  negotiates up to version 4.
* The watch() stream contract is unchanged regardless of negotiated
  version: the first item is a full snapshot and later items mark
  untouched keys as Unchanged, in request key order. Both protocol
  paths share one entry conversion.
@piscisaureus

Copy link
Copy Markdown
Member Author

Review findings addressed in the latest commits:

  • The channel reconnect loop now measures health by stream lifetime (30s) rather than received frames, so a server that sends a poison frame after the health window can no longer induce a zero-backoff reconnect loop; protocol errors count toward the failure limit.
  • Keys larger than 2048 bytes are rejected at add time by the shared state machine instead of killing the channel.
  • ClientMessageEffect::KeyAdded lets servers do a targeted read of the added key instead of a full sweep.
  • Subscribing while the channel is already connected re-sends baselines correctly (no missed-update window), and an empty key set no longer hangs the driver.
  • A 1008 policy close now surfaces as a channel failure that triggers the v3 fallback rather than dying silently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant