Skip to content

feat(control-api): add the GFS watch registry, listener and process runtime - #1499

Open
zach88 wants to merge 5 commits into
feat/gfs-event-subscriptions-api-basefrom
feat/gfs-event-subscriptions-api-registry
Open

zach88 wants to merge 5 commits into
feat/gfs-event-subscriptions-api-basefrom
feat/gfs-event-subscriptions-api-registry

Conversation

@zach88

@zach88 zach88 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What

The per-process GFS watch registry in control-api, plus the process-level pieces it runs on. Not mounted on any route yet (#1500 does that).

  • runtime.ts: a single cancel() form for timers (callers clear the handle when a timer fires); guarded / guardedTick / bestEffort covering both sync throws and async rejections; total(), raceAbort, and an injectable monotonic clock.
  • readSlots.ts: process-wide FIFO semaphore of 64 concurrent gfs-controller reads, with a demand counter and response-rate buckets feeding a backlogMs() estimate used to pace retries.
  • watchPool.ts: a dedicated pg pool for the watch, with client-side query timeouts, keepalive and a permanent error listener.
  • listener.ts: a dedicated LISTEN client for the resource, ping and command channels. Liveness is checked by the echo of its own ping; on loss it tears down its own state first, then reconnects with capped backoff. Dispatch is total per channel, and an unreadable payload on a revocation channel fails closed through onExpansionFailed.
  • recentDirty.ts: ring buffer of recent ancestry changes, used to flag an authorization decision that may have raced a move.
  • registry.ts:
    • Connections with a single socket writer gated by phase, backpressure (64 KiB or 5 s pending, then slow_consumer), heartbeat, max lifetime and token expiry.
    • closeConn never throws: registry memory is released first, side effects run best-effort, and the client's retry delay is spread.
    • The authorization op: ancestry CTE, a read token per drive minted per op, bounded gfs-controller reads, a gate after every await, auth and server deadlines, suspicion on concurrent ancestry changes, and deferred keys on transient failures.
    • A mutable subscription set through the gfs_watch_cmd channel: binding check (userId, authGeneration), a seq window and a single queued command.
    • gfs_resource_changed delivery with a 250 ms fixed window that only emits accepted keys outside the op in flight.
    • open() as the API for the GET handler, with key and drive ceilings enforced before admission.
  • readGfsc caps the gfs-controller response body at 64 KiB and replaces a non-uuid request id, which gfs-controller rejects.
  • Metrics gfs_watch_*, including gfs_watch_cmd_dropped_total{reason}.

The authorization and session push hooks (resetWindows, onExpansionFailed) are required dependencies of the listener and are no-ops here. #1502 fills them in.

Why

A watch stream has to keep three things true while requests, NOTIFYs, timers and socket events interleave across awaits: a client only ever receives events for keys it was authorized for, a change is not lost while a subscription is being re-authorized, and a stuck client or a burst of changes cannot exhaust the process. The registry is written as an explicit state machine so each of those can be checked at rest.

How

  • One writer per socket, gated by the connection's phase, so a late op result or a timer from a previous phase cannot write to a stream that has already moved on.
  • Every await in an op is followed by a gate that drops the result if the connection closed or a newer op replaced it.
  • Holds (connections whose user is being re-authorized) live in a structure of their own and are released by iterating that structure, never the per-user index.
  • Subscription changes cross replicas through gfs_watch_cmd, so a PUT can land on a replica that does not hold the stream.

Testing

  • Model-based: gfsWatch.registry.property.realPostgres drives the real registry with fast-check commands and fc.scheduler() for promise ordering, against real Postgres, the real trigger and real session/mint, with a fake gfs-controller built from its own envelope helpers. Invariants are checked at rest against a reference model.
  • Scenarios: gfsWatch.registry.realPostgres (45 cases) asserts the frames the client observes.
  • Unit and real-Postgres suites for runtime, readSlots, recentDirty, watchPool and listener (including listener loss through a controllable TCP proxy).
  • At this head: tsc --noEmit clean; npm test 634 files / 8951 tests passed (real-Postgres suites skip in this lane); real-Postgres run of the watch suites (gfsWatch*, gfs.externalGfsRead*): 15 files / 206 tests passed. db.realPostgresMigration fails 41 of 75 cases on the test host because its production access verifier shells out to psql, which is not installed there; the same 41 fail on dev.

Risks / follow-ups

Stack

Live GFS subscriptions ship as a stack. The server half and the desktop half must reach dev together: the server PRs retire the route the current desktop uses, and the desktop PR needs the new watch route.

# PR Base Scope
1 #1496 dev gfs-controller: drive name validation, principal_denied reason
2 #1497 dev host-context-controller: drive name validation (merge after 1)
3 #1498 dev control-api: shared building blocks, per-resource change trigger
4 #1499 3 control-api: watch registry, listener, process runtime
5 #1500 4 control-api: GET/PUT /external/gfs/watch
6 #1501 5 external-rest-api: GET/PUT /api/v1/me/gfs/watch
7 #1502 6 control-api + ERA: authorization and session push, retire the per-user entity-change stream
8 #1503 7 desktop-app: watch client, live previews and folders

1 and 2 are independent of the chain and safe to merge on their own (1 before 2, since the controller check in 2 mirrors the one in 1).

3 to 8 are one unit and should not be deployed partially:

  • 3 and 4 add groundwork; the only new runtime behavior is the gfs_resource_changed NOTIFY trigger, which nothing listens to yet.
  • 5 and 6 expose the route, but until 7 lands an authorization or session change does not cut the changed events of a stream that is idle (no op in flight). Do not ship 5/6 without 7.
  • 7 removes the per-user entity-change stream (ERA /api/v1/entity-changes/stream and control-api /external/entity-changes/stream), which the current desktop still uses. Do not ship 7 without 8.

zach88 and others added 5 commits November 12, 2026 00:33
Process-level building blocks for the live GFS watch, not wired yet:

- runtime: timers whose handle is cleared on fire and cancelled through a
  single cancel form, guarded/guardedTick/bestEffort covering sync and
  async failures, total(), raceAbort and an injectable monotonic clock.
- readSlots: process-wide FIFO semaphore of 64 gfsc reads with demand
  buckets and backlogMs for retry pacing.
- recentDirty ring buffer and a dedicated watch pg pool with client-side
  query timeouts, keepalive and a permanent error listener.
- listener: dedicated LISTEN client for the resource, ping and command
  channels; liveness by echo of its own ping, own-state-first teardown on
  loss, capped reconnect backoff, total per-channel dispatch, and
  fail-closed handling of unreadable revocation payloads through
  onExpansionFailed.
- gfs_watch_* metrics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the per-process watch registry that the HTTP routes will mount on:

- Connections with a single socket writer gated by phase, backpressure
  (64 KiB / 5 s -> slow_consumer), heartbeat, max lifetime and token exp.
- closeConn that never throws: registry memory is released first, side
  effects run best-effort, and the client retry is spread.
- The authorization op (ancestry CTE, per-op read token, bounded gfsc
  reads) with a gate after every await, auth and server deadlines,
  suspicion on concurrent ancestry changes, and deferred keys on
  transient failures.
- Mutable subscription set via gfs_watch_cmd: binding check
  (userId, authGeneration), seq window, and a single queued command.
- gfs_resource_changed delivery with a 250 ms fixed window that only
  emits accepted keys outside the op in flight.
- open() as the registry API for the GET handler, with key and drive
  ceilings enforced before admission.

readGfsc now caps the gfsc body at 64 KiB and replaces a non-uuid
request id, which gfsc rejects. Stream headers are exported as a
constant. New metric gfs_watch_cmd_dropped_total{reason}.

Revocation and session push hooks are left as no-ops for the follow-up
authz/session change; both ship together.

Tests: model-based run (fast-check commands + scheduler) against real
Postgres, the real trigger and real session/mint, with a fake gfsc built
from its own envelope helpers; 45 scenarios asserting observable frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…is set

The import sorter run by the pre-commit hook placed the clock helper,
which reaches src/db.ts through the watch runtime, ahead of the module
that points the core pool at the suite's database. The model-based
registry suite then bound the core pool to the default connection string
and failed in beforeAll on every run. The scenario suite only passed
because its env import happened to sort first.

Both suites now import only the env helper and types statically, and
load the harness and clock with a dynamic import from beforeAll, which
is the pattern the other real-Postgres suites already use.

Class-check: `process.env.CONTROL_API_PG_CONNECTION_STRING =` and
`gfsWatchRegistryEnv` under control-api/test → 3 sites (the two registry
suites via the env helper, gfsWatch.rateLimitCheck) → fix the two
registry suites; rateLimitCheck already sets the env and then uses
`await import('../src/db.js')`.
Repro-test: control-api/test/gfsWatch.registry.property.realPostgres.integration.test.ts
beforeAll (fails in f445e5a: "Connection terminated due to connection
timeout" from createUser, 5/5 runs on an isolated Postgres 16)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ent-subscriptions-api-registry

Brings origin/dev (22dd421) through the synced parent branch.
Conflicts: control-api package.json/package-lock.json version only;
kept the higher version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant