Skip to content

feat(platform): move the platform off Convex onto Postgres - #3107

Merged
larryro merged 172 commits into
mainfrom
0.5
Aug 30, 2026
Merged

feat(platform): move the platform off Convex onto Postgres#3107
larryro merged 172 commits into
mainfrom
0.5

Conversation

@larryro

@larryro larryro commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Replaces the Convex runtime with a Node + Hono + Postgres backend. The
running stack has no Convex process, service, control channel, or client
runtime; the app has its own wire contract; the reused 0.4 handler bodies
run against a SQL-backed ctx shim.

services/platform/backend/MIGRATION.md is the campaign's source of
truth — one row per domain, one row per teardown step, each naming what
landed and what was found en route. 94 done / 5 dropped / 1 pending.

What this is

main 0.5
Backend Convex deployment 213 modules under backend/, 57 SQL migrations applied at boot under an advisory lock
convex/ tree 1593 .ts 535 .ts — the reused handler bodies and what they reach
Convex runtime the stack none: no service, no image base, no CLI lane, no client
Jobs Convex scheduler pg-boss
Live updates Convex WS subscriptions SSE hint bus

The port is domain-by-domain, not a rewrite: where a 0.4 handler body was
correct, it is REUSED verbatim behind a ctx shim that dispatches a
function NAME to a SQL handler. That is why the diff deletes twice what
it adds — the decision logic mostly survives; the runtime under it does
not.

Gates

Run on the tip, each with its own exit code checked:

  • tsc --noEmit — 0
  • oxlint --type-aware — 0
  • server suite — 72 212 tests, 364 files
  • UI suite — 3436 tests, 445 files
  • integration — 256/256 against a real Postgres + MinIO

The integration run is the one that matters: it exercises every door
end-to-end against a throwaway database, including the machine journey
(presigned upload → bind → listing → content roundtrip), the automation
stepper, the task-agent arc, SSO/SCIM, governance holds and retention,
the WebDAV lane, and the in-sandbox bridges.

What is deliberately NOT done

The one pending row. convex is still in package.json, used as a
LIBRARY, not a runtime: v / Infer / GenericId from convex/values
type the reused handlers' argument contracts across 32 modules. Nothing
imports it from backend/ or lib/. Those validators become plain
TypeScript or zod as each domain ports, and then the reuse set moves out
of convex/ and the package leaves.

Also priced rather than hidden, in the ledger: WebDAVBackend's three
methods return any — exactly what the retired client's generic
resolved to for a name-addressed call, so nothing about strictness
changed here. Tightening it means declaring shapes at 66 call sites and
updating handler stubs that return partial ones; that is its own change.

Review notes

The ledger rows carry the reasoning, including the traps worth knowing
before touching this code:

  • The second dependency graph. Reused bodies call each other by NAME
    through the shim (internal.x.y.z) with no import to follow. Any
    closure analysis must walk static imports, dynamic await import(),
    AND api-object references — a naive import closure was wrong by 350
    files.
  • A wrapper can carry behaviour its impl does not. One real loss
    found and fixed this way: the connectors bridge's forensic audit lived
    in the 0.4 wrapper, and the 0.5 door calls the impl directly.
  • _generated/ is hand-maintained now. Adding a name there without
    giving the shim a handler for it is a runtime refusal waiting to
    happen.

larryro added 30 commits August 30, 2026 18:03
transactSerializable runs a postgres.js transaction at SERIALIZABLE and
re-executes the callback on SQLSTATE 40001/40P01 with jittered backoff,
layered on the existing connection-level withRetry (which only covers
08/57P/53-class transient errors). This is write-constitution rule 1 of
the 0.5 Postgres backend: every mutation-shaped handler goes through it,
so callbacks must stay pure apart from their database writes.
The Convex-to-Postgres port, increments 01-05 on the campaign ledger
(services/platform/backend/MIGRATION.md):

- Runtime: the platform image gains TALE_ROLE api/worker/all dispatch
  (Node runs backend/*.ts directly; compose profile 'backend'); a Node
  resolve hook + --experimental-transform-types let the backend import
  runtime-clean 0.4 modules unchanged instead of fork-copying them.
- Constitution: serializable writes, pg-boss 12 jobs enqueued inside the
  caller's transaction (LISTEN/NOTIFY wake, singletonKey dedupe, per-job
  batch results), and the Tier-2 hint outbox fanned out over SSE with
  Last-Event-ID replay. postgres.js json/jsonb serialization is aligned
  with node-postgres semantics (strings pass through) so pg-boss's
  pre-stringified parameters are not double-encoded.
- App DB: tale_app + an advisory-lock-guarded boot migrator; Better Auth
  and pg-boss own their schemas.
- Auth at 0.4 parity: email+password with the login-throttle gate
  (per-IP flood guard, per-account exponential lockout, timing jitter),
  the organization plugin (teams, access-control role matrix, slug/name
  hooks, scaffold-on-create), apiKey (+suffix column), twoFactor and
  passkey plugins; membership reads are direct SQL - the 0.4 mirror
  apparatus is gone.
- Ported domains: audit_logs (per-org chain head locked FOR UPDATE
  replaces the OCC genesis trick), login_attempts, notifications (the
  org-audience bell at the 0.4 shape), users, user_preferences,
  organizations (reads, org-switch record, deletion door, scaffold and
  cleanup jobs), members (list/context + guarded mutations), the PG
  rate limiter carrying the full 0.4 rule catalog, and file-direct org
  config reads (the configCache mirror dies).
- Proof: backend/integration-check.ts runs 22 checks against a real
  throwaway Postgres, including lockout flow and audit-chain
  re-verification.
Core CRUD, settings (identity/sharing/instructions/knowledge/agents/
models/connectors), pin/archive/restore/delete (confirm-gated cascade
mode), duplicate, project agents (create/update/delete with the 0.4
validation caps and admin-only secret grants), search, sidebar and
overview reads. The pure access matrix, key derivation, and audit action
constants are reused from 0.4 unchanged; per-org key/externalItemId
uniqueness moves into partial unique indexes. Cross-domain touchpoints
(document attach, delete-cascade walks, overdue rollups, automations
guard, REST v1 surface) land with their domains - tracked in
MIGRATION.md. Also fixes the token-bucket rate limiter's ratePerMs
parameter type under bigint expression context (::float8).
Tier A of the tasks domain: create/update with label resolution and
per-project numbering, status choreography with the subtask close-guard
and completedAt semantics, polymorphic assignee validation (humans need
project access, agents must be instances of the project), claim, board
drag moves on the reused LexoRank module, the label catalog
(create/rename/delete with task detach), advisory dependencies with
write-time cycle rejection, saved board views, the per-task activity
timeline, and the denormalized project rollup transitions. Project
creation now seeds the default labels in the same transaction (0.4
parity). Tier B (agent runs, review arc, comments/mentions, notify and
event fan-outs, REST, crons) lands with its infrastructure - tracked in
MIGRATION.md.
S3-compatible object storage becomes THE blob backend (Convex _storage
dies with the component): resolution prefers the org's BYO
object-storage/connection.json and falls back to the deployment default
under the 'default' config tree, failing closed when neither exists.
The aws4fetch signing, presign lanes, and key scheme are reused from
0.4 unchanged, so existing BYO-org configs work verbatim. Adds the
app.file_metadata ledger (RAG/transcription pipeline columns shipped
nullable for the knowledge/tts ports) and the files surface: a two-step
upload handshake with server-minted keys and HEAD-verified registration,
presigned GET serving, and org-scoped delete. Integration exercises the
full lane against a real MinIO (gated on ITEST_S3_ENDPOINT, visibly
skipped otherwise).
pg-boss schedule() registry (worker-boot idempotent upsert, UTC) with
the first maintenance sweeps - idle rate-limit rows (7d) and the
loginAttempts 30-day TTL plus 90-day block counters. Ports the platform
event seam at its exact 0.4 contract: dispatch stays a deliberate no-op
until the automations domain rebuilds the subscription fan-out, and the
producing domains (task created/status changed, project created) call
it inside their transactions so the swap-in is one function body.
Documents Tier A: bind a registered upload as a document, hub vs
project scoping with the 0.4 access-module semantics (project docs
never surface in hub listings; team rules with org-wide default),
rename/move/team edits, trash/restore soft lifecycle, project
attach/detach with the mutual-exclusivity rule (closing the projects
ledger gap), mention search, and presigned serving. Folders: tree CRUD
with the depth cap, sibling-name uniqueness, scope inheritance and
conflict rules, breadcrumbs - deleting refuses while any document
lives in the subtree until the trash-cascade lands. Integration drives
the whole vertical against real MinIO including the hub-visibility
flips on attach/trash.
The 0.5 replacement for the @convex-dev/agent component's thread and
message tables: app.threads + app.messages with the component's
(order, step_order) turn model, AI-SDK-shaped jsonb parts plus derived
plain text, and the thread_metadata sidecar (generation lifecycle
columns shipped for the chat engine). Task discussion comments ride it
end to end - lazily minted task_discussion threads, lockstep meta rows
(author, mentions, editedAt, locale snapshots), comment counts,
activity and audit trails, author-or-admin edit/delete. The mention
directory and its fan-outs land with collab/agents/automations.
Contacts: the per-org correspondent directory with role-matrix gating,
normalized emails, filterable keyset listing, find-or-create for the
conversations lane, soft trash, and contact.* events. Message feedback:
per-(message,user) upsert votes with toggle semantics and the org
insights feed with rating stats. Support cases: the org-scoped ticket
lifecycle with status timestamps, escalation levels, SLA fields,
comments that stamp first response, the activity feed, and archive.
Integration exercises all three verticals; the audit chain now verifies
36 rows across nine domains.
CRUD with the 0.4 field caps and role-matrix gating, per-org
case-insensitive name uniqueness as a real expression index (the
0.4 full-table probe becomes a constraint, kept only for the friendly
error), per-language translation upserts, and a filterable keyset
listing. REST and connector ingest lanes land with the machine door.
The credential RESOLUTION path - api-key decryption via secret_box, the
TALE_PROVIDER_KEY_ env gate re-checked at read time, and the
subscription-broker pool fetch/pick - reuses the 0.4 module verbatim: a
new ctx shim (lib/convex-shim.ts) re-points its two row lookups at
app.provider_credentials by function name and fails loud on any
un-shimmed ctx call. Admin surface: create (encrypt + masked preview,
first row becomes the default), status/default/allowlist edits with the
default swap enforced by a partial unique index, delete - all
audit-trailed, secrets never leaving the server. The node loader also
learns bare-specifier .js retries for CJS deep paths (validator/lib/*).
Search and fetch reuse the 0.4 modules verbatim - the ctx shim
re-points their three seams (the org-row lookup, credential loads, and
the retrievable-file access filter) at Postgres; the Tier-A filter
covers document and chat-thread scopes and denies conversation refs
until that domain lands. Ingest composes the same exported pieces
(extractText, the org embedder behind provider credentials,
indexDocument with the PII gate and scope stamps) into the
rag.index_file job, enqueued transactionally when an upload binds as a
document, with status writes on file_metadata. The deployment-default
corpus bootstraps at worker boot. Integration drives the full loop
against the real corpus database with a local OpenAI-compatible
embedding endpoint (base64 Float32, the SDK default).
larryro added 14 commits August 30, 2026 20:52
`jsonb_build_object` takes `"any"`, so the uncast parameter left Postgres
nothing to infer from and the statement failed to PARSE — 42P18, before
it ever ran. The endpoint answered 200 and wrote nothing, for as long as
it has existed, filling the dev log with the error a user finally
noticed. One `::numeric` fixes it; the sibling site in knowledge_entries
already casts, so this was an omission, not a pattern.

Nothing caught it because the lane had no integration coverage at all.
It has one now: post a wait against a real assistant message, assert the
value lands, then post a second one and assert the row is still stamped
with the first — the once-only guard the query carries.
A turn that spent 27 seconds and seven searches showed nothing until it
settled, then painted the whole trace at once.

The engine was never at fault: `runTurn` persists the settled parts at
each tool-round boundary, so the tool calls were in `app.messages.parts`
as they happened. The transport dropped them. The progress lane carried
text, reasoning and the cancel flag — `app.generations`, the row it
polls, has no column for parts — and the client read only those three.
The transcript itself is refetched at settle, which is why the trace
arrived in one lump.

Three edges close it. A parts write now bumps the generation clock: the
lane watches that clock, and a parts-only write moved nothing, so a tool
step was literally not progress. `progress` carries the row's current
parts, sent only when they changed — a text tick fires at the store's
250ms throttle while a tool result can be a whole RAG page. And the view
prefers the streamed parts when they outnumber the row's, which keeps
the settle write authoritative the moment it lands. The renderer needed
nothing; it already builds from `parts`.

Guarded against the shape of the bug rather than its symptom: the
integration check opens the stream on a fresh thread, runs a real tool
round, and asserts a `tool-call` part appears in a frame BEFORE
`event: settled` — finding it only in the settled payload fails. The
fake provider pauses before its final round so the assertion turns on
behaviour, not on whether the whole turn fit inside one poll.
The ENTITY_ID pattern is 16+ chars, so waitForURL resolved on the
wizard path and every later dashboard navigation hit the wizard again.
Forced rotation 400ed as voluntary without currentPassword, so invite
specs never left /forced-change-password. The automations list shows
the title-cased name, not the slug.
Typecheck failed on the false branch reading init.body when init is
undefined.
Every new thread's first message fires a second model call — the title
generator — and it spent real tokens (one 48-output-token call per
thread on the provider's bill) while writing nothing to usage_events:
generate_title.ts contained no usage write at all. The parse layer had
the numbers all along; nothing read them.

The attempt now carries its spend out, and the impl books it through an
injected recorder — the same ledger the turn writes through, handed in
by the job door, because the ctx shim has no ledger of its own. Booked
under its own `thread-title` agent slug so analytics can tell what the
conversation cost from what naming it cost; booked whether or not the
title was usable, because the tokens were spent either way; best-effort,
so a ledger failure never costs the title.

Deliberately unchanged: cost stays the catalog estimate everywhere.
Provider-reported cost would make spend a number governance can only
know after the fact — budget pre-checks and the ledger must price by
the same rule. The alias-routing under-count is a property of estimating
a routed request, not a config error.

The guard books through a real call: the integration check pins the
probe user's chat model to the fake provider (the fallback scan would
take the shipped openai connector, whose fake env key an earlier block
plants, and die on an unreachable host — masking the booking), then
asserts both the usage row and the AI title itself landed. Found en
route: the fake provider only spoke SSE, so the title lane's
non-streaming JSON call always fell back in this harness — it now
answers both dialects, like the providers it stands in for.
client.action skipped the write adapter's invalidate, so a successful
package upload left the listing on its cached empty page. saveVersion
now emits the automation hint the same-tab and SSE paths both need.
Clicking a task deliverable answered "Failed to load document": the
preview asked GET /files/<ref>/url and got a 404.

The app's file-identifier vocabulary is mixed by contract, not by
accident. Listings hand out the row's own id, the POST upload lane's
storageId IS the blob ref, task deliverables carry the ref, and the 0.4
getFileUrl query took the ref and resolved the row by storageId. The
0.5 port kept only the row-id half — /files/:fileId, its /url twin, the
batch /urls and the delete all resolved WHERE id — so every
ref-addressed read 404ed, while /files/statuses, ported separately,
already resolved by ref: the tell that the vocabulary was mixed all
along.

One resolver closes it. The s3: prefix makes the two identifiers
unambiguous, and a ref is org-scoped twice over — the WHERE, then the
key's own org prefix at presign time. All four routes go through it.

The guard round-trips the same upload through both identifiers, and the
ref lane's body must match byte-for-byte.
`tale update` is line-pinned — a 0.4.x instance never moves to 0.5.x on
its own — but the deploy guard is what stands behind the explicit move
the update hint suggests, and its baseline still said 0.4.0. With that,
`tale update --version 0.5.0 && tale deploy` on a 0.4 instance sailed
through: both versions sit above the old baseline, so the guard had
nothing to say while the deploy booted an empty Postgres and left every
chat, user and document orphaned in the retired store. 0.5 is exactly
the cutover the guard exists for — the application database moved from
Convex to Postgres and no importer bridges them — and the constant's
own contract says to bump it at the next one.

The refusal text, the update-path downgrade warnings (which still told
operators to run the deleted `tale migrate down`), and the upgrade docs
in all three locales now describe the 0.4 → 0.5 cutover: what carries
forward (the org config tree, on the shared volume), what does not (the
database), and the fresh-deployment path. The org-config carry-forward
is stated because it is the one asymmetry operators will probe.
The release workflow still built, gated, manifested and verified a
`convex` image whose Dockerfile was deleted with the service: the build
matrix would fail on the missing file, the container-test gate's pull
loop on the missing image, and the manifest/verify loops behind them —
the first tag push after the cutover would have died in four places.

The gate tests were worse than stale: compose.test.yml overrode a
`convex` service the base file no longer declares, which makes compose
reject the WHOLE project — the smoke test would fail at `up` before
probing anything. The overlay now wires the backend tiers and the
object store into the test env instead, and the smoke test probes what
actually ships: health-waits the 0.5 tier, `curl /ping` inside the
backend container, platform → backend-api connectivity (every /api lane
rides it), and backend → object-store health (an unreachable store
means every upload 503s).

The image test drops entries for services deleted long ago — convex,
and the crawler/rag pair that consolidated into the platform — which it
silently skipped as "image not found" while implying coverage; the
vulnerability scan list loses the same two ghosts.

Smoke has never actually completed on this branch (always skipped or
cancelled), so these are proven locally against freshly built images
rather than by CI history.
The opt-in docker-socket sidecar behind one-click "Apply & restart" was
built to bounce rag+convex after a deployment-config change; both are
gone, and the lane itself was broken against 0.5 — the app sent no
services, the backend defaulted to ['convex'], and the controller's own
allowlist stopped accepting convex at the teardown. Every operator who
may edit deployment config is named in TALE_DEPLOYMENT_CONFIG_ADMINS, a
host-side env var, so the manual path is always available.

The whole lane goes: services/controller/, POST /deployment/restart and
its wire-contract entry, adapter, hook and header button, the restart
strings (en/de/fr), the CLI's compose emitter and deploy/restore hooks,
the compose.yml block (the stack's last compose profile), the dev-loop
sibling spawn, the CONTROLLER_* env docs, the build/cleanup CI matrices,
and the trivy + commitlint entries. The post-save banner and the docs
now name the real apply path: docker compose restart backend-api
backend-worker, or tale deploy. cleanup-pr-images keeps a controller
row so leftover PR image tags still get GC'd; release.yml's controller
rows landed with the release-pipeline fix.

Integration 260/260; docs suite 194/194; cli tests green; i18n parity
green.
The backend entrypoint's egress firewall REJECTed all of RFC1918, but
the compose bridge networks are RFC1918 too and the container-netns
OUTPUT chain filters traffic to same-bridge peers — the claim that such
traffic bypasses OUTPUT via the bridge driver was wrong. Both backend
roles hold NET_ADMIN, so api and worker fenced themselves off from db
and crash-looped on CONNECT_TIMEOUT (TCP treats the ICMP net-prohibited
reject as a soft error and retries SYN until the client's timeout),
leaving the smoke stack permanently 'starting'. The web tier survived
only because it never gets NET_ADMIN.

ACCEPT the container's directly-connected subnets (from ip -4 route)
ahead of the RFC1918 rejects — IMDS and link-local stay rejected before
everything — and ship iproute2 in the runner image. Without iproute2
the RFC1918 fence is skipped with a warning instead of self-severing.

Verified in a bookworm-slim container with NET_ADMIN on a compose-like
bridge: peer connects instantly, external RFC1918 and IMDS stay fenced,
public egress unaffected. The CLI compose generator grants the same cap
in production, so the deployed stack hit the identical loop — the fix
rides in the image for both lanes.
Review follow-ups on the same-compose fence fix: derive the pass-list
from scope-link routes only, so a via-learned route (a pushed
10.0.0.0/8 through some gateway) can never widen it past genuinely
attached subnets; and when no subnet is derivable — iproute2 missing OR
an empty scope-link table — skip the RFC1918 fence entirely instead of
self-severing, keeping the IMDS + link-local rejects in every scenario.

A guard test now locks the rule program: it runs the shipped
install_ssrf_firewall against PATH-stubbed ip/iptables and asserts
ACCEPTs cover exactly the scope-link subnets ordered between the
IMDS/link-local rejects and the RFC1918 fence, and that both
no-derivable-subnet paths emit no fence. The ip stub serves the via
route to any non-scope-link query, so dropping the filter leaks a
10.0.0.0/8 ACCEPT and fails the guard (verified by mutation).

Live-checked on bookworm-slim with NET_ADMIN and an injected via
route: peer connects, via-routed and IMDS targets stay fenced.
@larryro
larryro merged commit fd5c009 into main Aug 30, 2026
62 checks passed
@larryro
larryro deleted the 0.5 branch August 30, 2026 15:35
larryro added a commit that referenced this pull request Sep 4, 2026
KNOWLEDGE_ENTRIES_FOLDER ("Knowledge entries") has had no importer since
the 0.5 rewrite (#3107): knowledge-entry backing documents are listed at
the Documents root, not filed into a reserved folder. knip never flagged
it because every backend/**/*.ts file is a knip entry for the platform
workspace, so backend exports are never reported unused. No message key,
helper, test, or docs sentence existed only for it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant