Skip to content

feat(control-api): push authorization and session changes over the GFS watch and retire the per-user entity-change stream - #1502

Open
zach88 wants to merge 8 commits into
feat/gfs-event-subscriptions-erafrom
feat/gfs-event-subscriptions-authz
Open

zach88 wants to merge 8 commits into
feat/gfs-event-subscriptions-erafrom
feat/gfs-event-subscriptions-authz

Conversation

@zach88

@zach88 zach88 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Completes the server side of the GFS watch: authorization and session changes are pushed to open streams, and the per-user 4-second entity-change stream is retired. Control-api and ERA, plus test updates in control-ui.

  1. Change notifications for authorization and session (migrations 0140 to 0145, one per table):
    • gfs_authz_changed: one pg_notify per key on gfs_grants, gfs_shares, gfs_desktop_operator_links, control_admin_users and team_members. Subject keys carry the id only for user/team subjects of uuid length.
    • gfs_watch_session: users UPDATE (when the lifecycle changed) and DELETE.
    • The migration runner retries a lost lock wait (55P03) for migrations that declare lockRetries (validated to 0..3); only team_members and users do.
    • Total parsers parseAuthzKey and parseSessionHint.
  2. Reauthorize on authorization changes:
    • A 250 ms window collects direct user keys and expands team and admin keys with one query per kind on the watch pool, filtered against live connections at flush time.
    • A failed expansion is retried once. A second failure, an unreadable or throwing hint, or more than 10,000 ids in one window fails closed: every connected user is held at once and reauthorized, at most once every 5 s per replica, with the closes spread.
    • A move or soft delete of an ancestor reauthorizes user connections whose accepted keys sit on that chain. Linked operators keep their access.
  3. Root listings and session:
    • A change visible in a root listing sends roots_changed {} to connected users with read through a direct or team grant (one query per 250 ms window) and to every linked operator, at most once every 2 s per connection with a trailing notice. Inserts skip the grantee query. A held connection never receives it.
    • Each gfs_watch_session NOTIFY is a hint: the affected user is held, the window confirms against users in one query, and closes with session_expired only when the row is gone, retired or carries another lifecycle_version. A failed or unreadable confirmation closes with reauthorize. A forged hint over an active row closes nothing.
    • Listener loss and shutdown cancel every window and release every hold.
  4. Retire the per-user entity-change stream:
    • control-api: /external/entity-changes/stream, its per-user open limiter and entityChangeUserVisibilityRefreshMs are removed. streamEntityChanges becomes operator-only. The operator stream (/gfs/entity-changes/stream) keeps its limiters, admin auth and periodic recheck.
    • external-rest-api: /entity-changes/stream is removed.

Why

The per-user stream is a security fix that became the refresh mechanism: it sends a generic invalidation every 4 seconds by default with no ids, plus a session check per tick, so users are isolated from the timing of the global feed. With the watch carrying per-resource changes, revocations and session changes, that stream has no remaining job, and keeping it would leave two paths with different guarantees.

How

  • One migration per table keeps each transaction to a single table lock, so applying them cannot deadlock against an application writer that touches two of these tables.
  • Revocation never waits on a successful query: every failure path ends in reauthorize, which makes the client reopen and re-authorize from scratch. Holds are released only by iterating their own structure.
  • A session hint is never trusted on its own; it is confirmed against users before any session_expired.

Testing

All server suites run against real Postgres with the real triggers and writers.

  • gfsWatch.authzSessionTriggers.realPostgres: one NOTIFY per key for every writer of every table, payload parsing, trigger WHEN conditions.
  • gfsWatch.authzWindow.realPostgres: expansion per kind, retry, fail-closed (second failure, unreadable hint, id ceiling) with its rate limit, ancestor chain revocation, linked operators.
  • gfsWatch.rootsSession.realPostgres: root notices, their rate limit and trailing notice, session confirmation outcomes, forged hints.
  • Model-based: gfsWatch.authz.property.realPostgres and gfsWatch.rootsSession.property.realPostgres (fast-check commands and scheduler against a reference model). The registry property test now checks the read and kind of every decision on every connection, closed streams included, and anchors the suspicion window at the op's real start.
  • migrationExecutionPolicy: lockRetries validation and the retry on 55P03.
  • entityChangeStreamRetirement.realPostgres and routes.entityChangesRetired (ERA): both retired routes answer 404 and the operator stream stays live.
  • externalSessionCurrentness.realPostgres rewritten against the session observer and the watch.
  • At this head: tsc --noEmit clean; npm test 635 files / 8961 tests passed (real-Postgres suites skip in this lane); real-Postgres run of the watch suites (gfsWatch*, gfs.externalGfsRead*, entityChangeStreamRetirement, externalSessionCurrentness, routes.externalGfsRateLimitFailClosed): 27 files / 333 tests passed. db.realPostgresMigration fails 41 of 75 cases on the test host because its production access verifier shells out to psql, which is not installed there; the same 41 fail on dev. external-rest-api at this head: tsc --noEmit clean, npm test 56 files / 576 tests passed.

Risks / follow-ups

  • Merge together with feat(desktop-app): refresh GFS previews and folders live over the watch stream #1503. The current desktop still calls /entity-changes/stream, which now answers 404.
  • Migrations 0140 to 0145 add triggers on hot tables (team_members, users). Each one runs with a 10 s lock_timeout; the two hot ones retry a lost lock up to their declared count.
  • The root-listing window has no ceiling and is not single-flight: a mass move by a legitimate user can saturate the watch pool. A failed roots query only drops that notice (the client refetches roots on every ready and on focus); the session and authorization queries that share the pool fail safe (reauthorize, reopen), so no revocation is lost. Follow-up: GFS watch: cap the root-listing window and make it single-flight #1505.
  • Restoring an ancestor while a key under it is being authorized can leave that key accepted with a cut chain, and a later soft delete or move of the ancestor does not revoke it. No GFS writer restores today; this must be fixed before one is added. Follow-up: GFS watch: a restore during authorization leaves a key with a cut ancestor chain #1506.
  • File budget: 45 files in 3 services (control-api, ERA, and 3 control-ui test files updated for the retirement); 25 are test files (23 suites, 2 helpers) and 6 are deletions. The retirement cannot be split from the push it relies on.

Stack

Live GFS subscriptions ship as a stack. The server half and the desktop half must reach dev together: the server PRs retire the route the current desktop uses, and the desktop PR needs the new watch route.

# PR Base Scope
1 #1496 dev gfs-controller: drive name validation, principal_denied reason
2 #1497 dev host-context-controller: drive name validation (merge after 1)
3 #1498 dev control-api: shared building blocks, per-resource change trigger
4 #1499 3 control-api: watch registry, listener, process runtime
5 #1500 4 control-api: GET/PUT /external/gfs/watch
6 #1501 5 external-rest-api: GET/PUT /api/v1/me/gfs/watch
7 #1502 6 control-api + ERA: authorization and session push, retire the per-user entity-change stream
8 #1503 7 desktop-app: watch client, live previews and folders

1 and 2 are independent of the chain and safe to merge on their own (1 before 2, since the controller check in 2 mirrors the one in 1).

3 to 8 are one unit and should not be deployed partially:

  • 3 and 4 add groundwork; the only new runtime behavior is the gfs_resource_changed NOTIFY trigger, which nothing listens to yet.
  • 5 and 6 expose the route, but until 7 lands an authorization or session change does not cut the changed events of a stream that is idle (no op in flight). Do not ship 5/6 without 7.
  • 7 removes the per-user entity-change stream (ERA /api/v1/entity-changes/stream and control-api /external/entity-changes/stream), which the current desktop still uses. Do not ship 7 without 8.

zach88 and others added 8 commits November 12, 2026 16:20
…FS watch

Add migrations 0140-0145, one per table so each transaction holds a single
table lock and cannot deadlock against an application writer:
- gfs_authz_changed: one pg_notify per key on gfs_grants, gfs_shares,
  gfs_desktop_operator_links, control_admin_users and team_members. Subject
  keys carry the id only for user/team subjects of uuid length.
- gfs_watch_session: users UPDATE (WHEN lifecycle changed) and DELETE.

The migration runner retries a lost lock wait (55P03) for descriptors that
declare lockRetries (validated to 0..3); only team_members and users do.

Add total parsers parseAuthzKey and parseSessionHint. Listening on the new
channels comes with their handlers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nges

Listen on gfs_authz_changed and close affected streams with reauthorize:
- a 250 ms authorization window collects direct user keys and expands team
  and admin keys with one query per kind on the watch pool, filtered against
  live connections at flush time;
- a failed expansion is retried once; a second failure, an unreadable or
  throwing revocation hint, or more than 10,000 ids in a window fails closed:
  every connected user is held at once and reauthorized, at most once every
  5 s per replica, spread over the closes;
- a move or soft delete of an ancestor reauthorizes user connections whose
  accepted keys sit on that chain; linked operators keep their access;
- listener loss and shutdown drop pending windows and release every hold.

Tests run against real Postgres with the real triggers and writers, plus a
model-based property test. The shared harness now finds the listener backend
by its ping as well as its LISTEN, so the registry property test actually
induces listener loss instead of skipping it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… watch

Roots: a change visible to a root listing (h=1) sends roots_changed {} to
the connected direct grantees with read (one query per 250 ms window, with
subject_id compared as text) and to every linked operator, at most once
every 2 s per connection with a trailing notice. Inserts skip the grantee
query. A held connection never receives it.

Session: listen on gfs_watch_session and treat each NOTIFY as a hint. An
affected user is held, the window confirms against users in one query and
closes with session_expired only when the row is gone, retired or carries
another lifecycle_version; a failed or unreadable confirmation closes with
reauthorize instead. A forged hint over an active row closes nothing.

Listener loss and shutdown cancel both windows and release every hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anchor the 6 s suspicion window at the op's real start instead of at the
instant the test sent the command. A delta queued behind an op in flight, or
an open still authenticating, starts later; once more than 6 s have passed
since the move its snapshot of recent ancestry changes is rightly empty, and
the model demanded a suspicion the registry has no reason to raise. The start
is the admission instant for an open (FakeSocket.admittedVt) and srvDeadline
minus the delta deadline for a delta, read while the op is still the
connection's.

Check S4 on every decision, closed streams included. It was only checked on
connections still ready at rest, so a suspect acceptance on a stream that
later closed was never evaluated: a registry ignoring recentDirty, or with
suspicion disabled in finishOp, passed T-M2. Whether a key's chain holds P
comes from the fixture, since the indexed chain is gone once a stream closes.

Class-check: windows anchored at a command's instant instead of its op's
effect, in test/*property*.ts (grep "clock.now()" and "vt -") -> 2 sites,
both in this file (openVt, puts[].vt) -> both fixed; the other oracles anchor
at effect instants. Decision checks gated on "conn.phase === 'ready'" at rest
in checkModel -> 2 (S4 and the I6 read check) -> S4 fixed; the I6 read check
stays ready-only, left as a follow-up.
Repro-test: control-api/test/gfsWatch.registry.property.realPostgres.integration.test.ts:178
(fails at ff18a96 with seed -1676849103 path "11", 3/3, and seed
1022089507 path "22", 2/3; passes 3/3 each with the fix). Negative control at
:477: a registry ignoring recentDirty or with suspicion disabled in finishOp
passes at ff18a96 and fails on I6/S4 with this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The GFS watch (`/external/gfs/watch`) now carries live entity changes,
authorization and session changes for users, so the per-user 4-second
entity-change stream is removed end to end:

- control-api: drop `/external/entity-changes/stream`, its per-user
  open limiter and `entityChangeUserVisibilityRefreshMs`;
  `streamEntityChanges` becomes operator-only with the signature
  `(req, res, initialCursor, isAuthorized, principalId)`. The operator
  stream (`/gfs/entity-changes/stream`) keeps its limiters, admin auth
  and periodic recheck; `principal_kind` stays as a constant label.
- external-rest-api: drop `/entity-changes/stream` and its tests.
- Tests: session-currentness coverage rewritten against the observer and
  the watch; the stream-open fail-closed case now targets the watch and
  asserts the 503 on response headers (shared `openWatch` helper);
  control-ui GfsBrowser tests follow the new signature. New retirement
  tests pin 404 on both retired routes and a live operator stream.

The desktop client still calls the retired route and moves to the watch
in the follow-up desktop change; both must land together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…treams included

The I6 read check (a key may be accepted only after its op read the resource
as a 2xx under `allow`) and the I6/I11 kind check were evaluated only on
connections still ready at rest. An improper acceptance on a stream that later
closed (a listener loss, a reauthorize close, the client leaving) was never
evaluated: a registry that accepted a key after a deny or an unavailable read
and then closed the stream passed T-M2.

Both checks now run, next to S4, over every decision frame of every
connection, with the attribution S4 already used: a key a frame accepts that
the previous frame did not was decided by that frame's op (seq 0 = the open,
otherwise the put's requestId). That holds in this model because a key is only
accepted by the op that added it, and the one op that writes no frame (a
retained user's delta) belongs to a stream that only leaves retention by
closing. The ready-only block keeps I19/I21 and a single rest check: every key
accepted at rest is in the last decision frame.

The read rule changes from "the op's last read of the resource was allow" to
"one of the op's reads of it was allow". gfsc sees the resource, not the key,
so an op holding dir:P and file:P reads P twice; if P's decision turns to deny
between the reads, dir:P is rightly accepted with a deny as the op's last
read of P. The old rule flagged that legitimate case (the head oracle fails on
it too, 2/2, while the connection is still ready). It is pinned as a third
fixed example; with a single read per resource the two rules are the same.

Class-check: decision checks gated on "conn.phase === 'ready'" at rest in
test/*property*.ts (grep "phase !== 'ready'" / "phase === 'ready'") -> 2 sites
in this file: checkModel (I6 read, I6/I11 kind, I19, I21) and connAt (command
targeting, not an oracle) -> I6 read and kind moved to every decision; I19/I21
and the accepted-in-last-frame check stay ready-only (they are about rest
state); connAt is not a check. "last read decides" (grep "reads.at(-1)") -> 1
site -> replaced as above. The other property suites have no I6 read check.
Repro-test: control-api/test/gfsWatch.registry.property.realPostgres.integration.test.ts:491
(negative control: a registry that accepts on a non-principal 403 and closes
the stream with 'reauthorize' after the frame passes 3/3 x 202 runs at
6cea012 and fails with this change, seed -1989312804 path "5"; the same with
an unavailable (503) read: passes 3/3 at 6cea012, fails 2/2, seeds
-1292543513 path "16" and -713675540 path "6". Dual-read example at :620
fails 3/3 with the last-read rule and passes 3/3 with this one.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ubscriptions-authz

Brings origin/dev (22dd421) through the synced parent branch.
Conflicts: control-api package.json/package-lock.json version only,
kept the higher version; external-rest-api releaseManifest.ts, took
the v0.12.1 release coordinates from dev and kept this branch's
externalRestApiVersion, which tracks external-rest-api/package.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant