Skip to content

feat(control-api): add GFS watch building blocks and per-resource change notifications - #1498

Open
zach88 wants to merge 4 commits into
devfrom
feat/gfs-event-subscriptions-api-base
Open

zach88 wants to merge 4 commits into
devfrom
feat/gfs-event-subscriptions-api-base

Conversation

@zach88

@zach88 zach88 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What

First control-api slice of the live GFS watch. Two commits:

  1. Shared building blocks, no behavior change for existing callers (refactor):
    • isTransientInfraError: one classifier for pg, pg-pool, network and abort failures, including the session backend-unavailable wrapper.
    • setStreamHeaders, now shared by the entity-change and notification streams.
    • createRateLimitCheck: the rate-limit decision as a pure function plus a separate header writer. createRateLimitEnforcer is recomposed from both.
    • The externalWatchRateLimit module: the entity-change limiter config moves out of its route, next to the new per-user open/PUT limiters for the watch.
    • The external GFS ingress limiter and edge handler move to module level and are exported with attachExternalGfsRequestId.
    • resolveExternalGfsAuthorityForSession, now used by attachExternalGfsAuthority.
    • mintExternalGfsReadToken, shared with proxyReadToGfsc, and readGfsc, a typed gfs-controller read that never rejects.
  2. Per-resource change notifications and watch keys (feat):
    • Migration 0139 adds the gfs_resource_changed_notify trigger on gfs_resources. Every row write emits pg_notify('gfs_resource_changed', 'rid,old,new,h,a,op').
    • keys.ts: canonical watch keys gfs://<drive>/<rid32> and the normalizers for values that come from other layers.
    • notifySchemas.ts / resourceChangedSchema.ts: total parsers for the resource, ping and command channel payloads.

Nothing listens to the new channel yet; the registry comes in #1499.

Why

Today the desktop learns about GFS changes from a per-user stream that sends a generic "something may have changed" notice every 4 seconds by default, with or without changes. Each one makes the app re-resolve open previews and refetch active GFS queries, whether or not anything changed, and each re-resolve remounts the preview body (scroll and video position are lost, "Loading preview" flashes).

This stack replaces it with a watch stream: the client subscribes to the resources it is showing and only hears about the ones that changed. This PR adds the pieces the watch is built on, and extracts the ones it shares with existing routes so the logic is not duplicated.

How

  • The trigger casts uuids to text before coalescing, so a NULL column can never abort a GFS write. search_path is pinned and EXECUTE is revoked from PUBLIC, like the existing outbox trigger functions.
  • The trigger lives in a new migration and not inside the existing entity-change schema function: that function's version is already applied on existing databases, so a new migration is the only way to reach them.
  • The payload stays far below the 8000-byte NOTIFY limit: fixed-width fields only, no names or paths.
  • readGfsc never rejects: it returns a typed result (2xx with the raw kind, 401, 403 with principalDenied, 404, 410, or transient), so every await in the watch classifies its outcome in one place. 409, 429, 5xx, unparseable 2xx bodies, network failures, aborts and its deadline are all transient.

Testing

Real-Postgres suites use the repo's integration harness (CONTROL_API_REAL_PG_ADMIN_URL).

  • gfsWatch.resourceChangedTrigger.realPostgres: one parseable payload per insert, rename, move (with both parents), content update, soft delete, restore and hard delete; boundary values never abort a write and notify only on commit; pinned search_path, no PUBLIC execute, and the trigger still fires for the gfs-controller writer role.
  • gfsWatch.keys.property: fast-check properties for key canonicalization (round trip and rejection of non-canonical input).
  • gfs.externalGfsRead (unit and real Postgres): every gfs-controller response class maps to one tagged outcome; the shared mint matches the old proxy path for the linked-admin case.
  • gfsWatch.transientInfra (unit and real Postgres): classification against real pg / pool / socket failures.
  • gfsWatch.rateLimitCheck.realPostgres: the pure check and the shared header writer produce the allowed, denied and unavailable decisions and headers.
  • Migration order and policy tests updated for 0139.
  • At this head: tsc --noEmit clean; npm test 630 files / 8888 tests passed (real-Postgres suites skip in this lane); real-Postgres run of the watch suites (gfsWatch*, gfs.externalGfsRead*): 7 files / 80 tests passed. db.realPostgresMigration fails 41 of 75 cases on the test host because its production access verifier shells out to psql, which is not installed there; the same 41 fail on dev.

Risks / follow-ups

Stack

Live GFS subscriptions ship as a stack. The server half and the desktop half must reach dev together: the server PRs retire the route the current desktop uses, and the desktop PR needs the new watch route.

# PR Base Scope
1 #1496 dev gfs-controller: drive name validation, principal_denied reason
2 #1497 dev host-context-controller: drive name validation (merge after 1)
3 #1498 dev control-api: shared building blocks, per-resource change trigger
4 #1499 3 control-api: watch registry, listener, process runtime
5 #1500 4 control-api: GET/PUT /external/gfs/watch
6 #1501 5 external-rest-api: GET/PUT /api/v1/me/gfs/watch
7 #1502 6 control-api + ERA: authorization and session push, retire the per-user entity-change stream
8 #1503 7 desktop-app: watch client, live previews and folders

1 and 2 are independent of the chain and safe to merge on their own (1 before 2, since the controller check in 2 mirrors the one in 1).

3 to 8 are one unit and should not be deployed partially:

  • 3 and 4 add groundwork; the only new runtime behavior is the gfs_resource_changed NOTIFY trigger, which nothing listens to yet.
  • 5 and 6 expose the route, but until 7 lands an authorization or session change does not cut the changed events of a stream that is idle (no op in flight). Do not ship 5/6 without 7.
  • 7 removes the per-user entity-change stream (ERA /api/v1/entity-changes/stream and control-api /external/entity-changes/stream), which the current desktop still uses. Do not ship 7 without 8.

zach88 and others added 4 commits November 11, 2026 22:54
Groundwork for the live GFS watch stream, with no behaviour change for
existing callers:

- isTransientInfraError: single classifier for pg / pg-pool / network /
  abort failures, including the session backend-unavailable wrapper.
- setStreamHeaders shared by the entity-change and notification streams.
- createRateLimitCheck: pure rate-limit decision plus a separate header
  writer; createRateLimitEnforcer is recomposed from both.
- externalWatchRateLimit: entity-change limiter config moved out of its
  route, plus per-user open/PUT limiters for the watch.
- Module-level external GFS ingress limiter and edge handler, exported
  together with attachExternalGfsRequestId.
- resolveExternalGfsAuthorityForSession, now used by
  attachExternalGfsAuthority (same generation for V1, none for V2).
- mintExternalGfsReadToken shared with proxyReadToGfsc, and readGfsc, a
  typed gfsc read that never rejects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the gfs_resource_changed_notify trigger on gfs_resources (migration
0139). Every row write emits pg_notify('gfs_resource_changed',
'rid,old,new,h,a,op'): uuids are cast to text before coalescing so the
trigger can never abort a GFS write, search_path is pinned and EXECUTE is
revoked from PUBLIC like the existing outbox trigger functions.

Add the watch key helpers (canonical gfs://<drive>/<rid32> keys, the
normalisers for values from other layers) and the NOTIFY payload schemas
for the resource, ping and command channels. Nothing listens yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ptions-api-base

Resolve control-api package.json/package-lock.json version conflict by
keeping the higher version (0.1.944).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
const app = express()
app.get(
'/authority',
(req, res, next) => void mod.authority.attachExternalGfsAuthority(req, res, next)

let typeError: unknown
try {
;(undefined as unknown as { x: () => void }).x()
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import express, { type NextFunction, type Request, type Response } from 'express'
import { createPublicKey, randomBytes, randomUUID } from 'node:crypto'
import { once } from 'node:events'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants