Bug: demo backend micro-stalls blow the 5s function-execution cap on trivial auth-path functions — better-auth adapter, getCurrentUser, 2FA getStatus (3 stall windows on 08-28, no deploy)

[larryro]

Source: GlitchTip sweep of demo.tale.dev, 2026-08-28 (automated). Tracked on the board as TAL-23.

Distinct from #3091 (deploy-window Postgres restarts — no deploy today), #3095 (15 s syscall ceiling inside the workpool loop — different limit, different functions), and #3088 (UNAUTHENTICATED refusals — these are timeouts on an authenticated path).

Evidence — two new GlitchTip groups born today

• TALE-PROJECT-125 — Error: Function execution timed out (maximum duration: 5s) — 5 events, 2026-08-28T03:19:02Z → 04:44:56Z, backend-observed, func_runtime: default, cached: false.
• TALE-PROJECT-126 — the client-observed twin (Uncaught Error: …) — users/queries:getCurrentUser, 03:47:03Z, an authenticated demo user.
• Precedent: TALE-PROJECT-10V — 4 events in one burst 2026-08-12T09:45:40–45Z, all prompts/queries:getSavedSourceMessageIds (private overlay) for one user — four live subscriptions dying together, stack through the public lib/rls/organization/get_user_organizations.ts:82 (paginated better-auth member findMany).

Per-event timeline of 125/126 — three distinct stall moments, each killing unrelated in-flight functions in the same instant:

stall moment (UTC)	victims
03:19:02.627 / .629	adapter:findOne (betterAuth component) + sandbox/session_mutations:upsertSessionOp — a query AND a mutation
03:47:02.104 / :03.396	two_factor/queries:getStatus + users/queries:getCurrentUser
04:44:06.6 / :56.6	adapter:findMany (betterAuth) ×2

Read

All implicated functions are trivial at HEAD 93d49d5 — none can burn 5 s of its own compute:

• users/queries.ts#L38-L77 — getCurrentUser: auth identity + one adapter.findOne by _id.
• two_factor/queries.ts#L170 — 2FA getStatus: single-row reads (+ a backup-codes decrypt).
• sandbox/session_mutations.ts#L958 — upsertSessionOp: single-row upsert with a bounded timeline payload.
• betterAuth/adapter.ts — the component adapter (@convex-dev/better-auth createApi); the hot member-by-userId findMany paginates at 100 items.

Unrelated trivial UDFs cannot each exceed 5 s at the same wall-clock instant. The demo convex-backend (or its Postgres) stalls ≥5 s, and whatever default-runtime UDF is in flight blows the execution cap.

No deploy correlation: the last release, 0.4.17, registered 2026-08-27T02:36Z (GlitchTip releases API), and none of #3091's Postgres-shutdown groups fired today. Leading suspect: colocated load — a sandbox agent session was live during the 03:19 window (upsertSessionOp is the sandbox run-ledger flush; the nightly triage automation runs at 03:00Z), so sandbox/agent workloads starving the demo host fits the timing. The 04:19 Unauthenticated burst (#3088's groups) sits between stall windows.

Impact: the failures land on the session/auth path — a real signed-in user had getCurrentUser + 2FA getStatus fail; better-auth adapter timeouts mean session validation fails during a stall. Low volume today, but this is the second distinct date with this shape (08-12, 08-28×3) and it scales with load.

Observability wrinkle: 125 vs 126 are the same 03:47 stall double-reported (backend-observed vs client-observed) — the same pattern as #3095's NM/O0 pair.

Proposed direction

1. Attribute the stalls: host CPU/IO + Postgres (checkpoints, autovacuum, lock waits) + convex-backend timings across the three 08-28 windows; the 03:19 upsertSessionOp event gives an exact overlap anchor with sandbox session activity.
2. Isolate the suspect: cgroup/CPU limits on sandbox agent containers colocated on the demo box, or move that workload off-box.
3. Auth-path resilience: one-shot retry on system-timeout for session-critical reads (getCurrentUser, better-auth adapter calls) so a 5 s stall doesn't surface as a failed session; verify convex/react subscription recovery after a system error.
4. Single-report per stall (backend + client twins), aligned with Bug: workflow workpool main loop hits Convex's 15 s syscall ceiling under backlog — SystemTimeoutError ×468, double-reported #3095's dedup item.

Acceptance

• Stall source identified and mitigated: a week of demo operation (incl. nightly automation runs) accrues no new Function execution timed out (maximum duration: 5s) events.
• A signed-in user riding out an induced stall keeps their session — no auth-path hard failure.
• GlitchTip groups 7975897 / 7975971 / 7774175 stay flat; any new occurrence reports once, not twice.