Skip to content

fix(platform): chat stall guard, atomic turn claim, sharing lifecycle - #3182

Draft
larryro wants to merge 5 commits into
mainfrom
fix/chat-integrity
Draft

fix(platform): chat stall guard, atomic turn claim, sharing lifecycle#3182
larryro wants to merge 5 commits into
mainfrom
fix/chat-integrity

Conversation

@larryro

@larryro larryro commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Seven verified medium-severity chat-domain defects from the backend deep review, all still live on main at 899fcc0 (post-#3157/#3158/#3164). Five atomic commits, one per seam.

Per-finding outcome

# Finding Outcome Evidence
1 Fixed 180s whole-request abort kills healthy long streams mid-reply fixed core/chat/turn_action.ts:648 had AbortSignal.timeout(180_000) on the fetch, covering body streaming. Replaced by a silence clock (core/chat/stream_stall.ts).
2 Trashed thread keeps serving its share link; unshare on it is a silent no-op fixed getSharedThread had no tm.status filter; unshareThread went through the active-only read. Base harness probe: trashed link answered 200, unshare on trashed reported ok while doing nothing.
3 Moving a project-shared thread to another project silently shares it with the new audience fixed moveThreadToProject updated project_id only. Base probe: shared=true in the new project, no audit row.
4 Arena column B runs without the project context and a winning B loses the project filing fixed ensureArenaPair copied agent + capabilities but not project_id; settle graduated B unfiled. Base probe: B.project=false, winner unfiled + unpinned.
5 Chat search tool surfaces trashed contacts fixed Shim contact query had no lifecycle predicate. Base probe: ["Zelda Trashed","Zelda Current"].
6 beginTurn is not the one transaction its contract requires; crash strands pending placeholder fixed Four pool statements, no sql.begin. Base probe (trigger-simulated crash at the generation insert): 2 stranded rows.
7 Busy gate is check-then-act; racing sends rebind and delete each other's generation row fixed Route read + ON CONFLICT (thread_id) DO UPDATE. Base probes: two concurrent opens both won (pending,pending), two concurrent HTTP sends both completed with 4 rows. Not addressed by #3164.

The abort-timeout approach (1)

The model round's only clock is now a silence clock, never a whole-request deadline: createStallGuard(180_000) aborts only after 180s with no bytes, and streamSse calls touch() on every chunk and races every reader.read() against the guard, so an actively producing stream is never aborted however long it runs, while a genuinely hung one still ends — with The model provider stopped sending data — the reply timed out after 180 seconds of silence., which the chat-error classifier buckets as transient (timed out). Headers count as first life; the first byte gets the same 180s allowance (a slow-thinking model is not a hung one). No env var, no migration.

The claim (6, 7)

beginTurn runs in one sql.begin; the generation insert is ON CONFLICT (thread_id) DO NOTHING RETURNING, and no row means another turn holds the thread → ThreadBusyError (new, lib/chat/turn.ts) rolls the open back. It is raised before the pipeline's settle block, so the loser never runs the endGeneration that would close the winner. Callers: send route → 409 {status:'refused'} (same shape as its fast-path read, which stays as a fast path); arena side → refused without an error row; REST turn → its existing busy error row; deferred send → re-parks the row and re-polls instead of deleting it. endGeneration is one transaction too and fails any placeholder still pending (a failed finalize is the one remaining orphan path). No migration (0070 is taken by sibling branches; none needed here).

Sharing lifecycle (2, 3) and arena (4)

  • getSharedThread requires tm.status = 'active'; unshareThread matches the owned row regardless of lifecycle and returns whether it did — the route answers { ok } honestly. Trash does not clear is_shared (the gate hides the link while trashed; restore returns the conversation as it was, and unshare is available throughout).
  • A move that changes the project sets shared_with_project = false and audits project.thread.unshared on the project the thread left (newState.movedToProjectId). Docs platform/projects/concepts.md (en/de/fr) name the rule.
  • Arena B is born with A's project_id + reasoning_effort (still hidden — never lists in the project Chats tab); a B verdict carries A's filing (coalesce, so pairs opened before this fix land filed), pin, and read watermark onto the survivor.

Tests

Colocated vitest (all red on base — run against a git archive 899fcc08a tree: 14 failed / 47 passed, the two stall files failing at import):

  • core/chat/stream_stall.test.ts (5): silence clock never fires under activity, fires after a silent window, measures from the last byte, dispose/late-touch.
  • core/chat/turn_action.test.ts (3): a healthy stream 3× longer than the window survives intact; a silent stream ends with the stall error; a user cancel passes through as itself.
  • domains/chat/store.test.ts (+3): beginTurn inside ONE begin with NOTIFY after commit; DO NOTHING claim → ThreadBusyError + rollback; endGeneration one tx failing a pending placeholder.
  • lib/chat/turn.test.ts (+1): a busy open propagates and the loser never calls endGeneration.
  • domains/chat/threads.test.ts (7): active-only share lookup; unshare without a lifecycle gate + honest boolean; move ends the share and audits (and not when the project is unchanged / never shared).
  • domains/chat/arena.test.ts (2): B's birth carries project_id/reasoning_effort; the B-wins graduation copies filing/pin/read state.
  • domains/chat/shim.test.ts (1): contacts query carries the lifecycle predicate.

Integration harness (backend/integration-check.ts), 7 new probes: share link 404 in trash + unshare works in trash + restored thread unshared; move ends share + audit row + new project's tab shows unshared; arena B project_id/effort + hidden from tab + B-wins filed and pinned; trashed contact absent from the shim query; trigger-simulated crash at the generation insert strands nothing; two concurrent beginTurn → one wins, one ThreadBusyError, 2 rows, endGeneration fails the orphan placeholder; two concurrent HTTP sends on a slow-drip provider → completed/refused, 200/409, exactly one full exchange.

Verification (observed)

  • bunx tsc --noEmit (platform): exit 0. bun run --filter @tale/platform lint: exit 0.
  • bun run --filter @tale/platform test: 439/442 files, 72,742 tests pass; the 3 failing files are app/routes/** suites failing to load with Denied ID …@fontsource…woff2 — reproduced identically on the base tree (symlinked node_modules worktree artefact, unrelated).
  • bun run --filter @tale/docs test: 30 files / 194 tests pass.
  • backend:integration on fresh tale-db + MinIO per run, SANDBOX_LLM_GATEWAY_ADMIN_PASSWORD set: branch 386 PASS / 0 FAIL (exit 0); base 379 PASS / 0 FAIL on its own probe set; base code + this branch's harness: 379 PASS / 7 FAIL — the seven failures are exactly the new probes.

Cross-class discoveries (not in this batch)

  • app.messages has no UNIQUE (thread_id, "order", step_order); with the claim in place the turn path can no longer race, but out-of-turn appenders (REST failure rows, arena side-error rows) still compute max(order)+1 unguarded. Adding the index needs a dedupe backfill of existing ties first → own migration (0071+).
  • Historical pending placeholders orphaned before this fix (no generation row) are not swept; a periodic sweep would want a partial index on app.messages (status) WHERE status = 'pending'.
  • Vitest server project in a symlinked-node_modules worktree cannot load 3 app/routes/** suites (@fontsource woff2 Denied ID) — environmental, present on base.

The direct model call ran under one whole-request AbortSignal.timeout of
180s that also covered body streaming, so every reply still streaming at
minute three — a high reasoning effort with a large output ceiling — was
cut mid-sentence and surfaced as a generic provider error.

Replace the deadline with a silence clock (core/chat/stream_stall.ts):
the guard restarts on every byte, so an actively producing stream is
never aborted however long it runs, while a provider that stops sending
for 180s still ends the round — with an error that names the stall and
classifies as transient. The first byte gets the same allowance, and
streamSse races every read against the clock so a hung read cannot
outlive it.
beginTurn's contract is one transaction — user row, placeholder, and
generation row commit together — but the Postgres store ran them as four
pool statements: a crash between them left a question with no reply or a
'pending' bubble no watchdog would ever fail. The same insert also
rebound an existing generation row (ON CONFLICT DO UPDATE), so two sends
racing through the route's check-then-act busy gate both ran, streamed
into one row, and the first to finish deleted it from under the other.

Wrap the open in sql.begin and make the generation insert the claim:
ON CONFLICT DO NOTHING, and no row means another turn holds the thread —
the open throws ThreadBusyError and rolls back, so the loser leaves no
trace and, raised before the pipeline's settle block, never runs the
endGeneration that would close the winner. The send route answers it as
the same 409 refusal its fast-path read gives, the arena side records no
error row, the REST turn writes its busy row, and a deferred send parks
itself again instead of deleting its tray row. endGeneration is one
transaction too and fails a placeholder still pending after a failed
finalize, so a settled turn never leaves an eternal pending bubble.
Two lifecycle leaks in thread sharing. A share link kept resolving for a
thread its owner had moved to Trash (or that had aged to 'expired'):
getSharedThread applied no status filter, while unshare went through the
active-thread read and was a silent no-op on a trashed thread — the route
still answered ok. And moving a project-shared thread to another project
carried shared_with_project along, granting the new project's members
the whole history with no consent and no audit row.

Gate the share-link read on status = 'active' like every other read;
match unshare on the owned row regardless of lifecycle and report whether
it matched. A move that changes the project switches the project share
off and audits the implicit unshare on the project the thread left; the
owner re-shares in the new project deliberately. Docs (en/de/fr) name
the rule.
ensureArenaPair copied A's agent and capabilities into column B but not
its project filing, so in a project chat column A's turns carried the
project instructions and knowledge while B's did not — a comparison
between prompts, not models — and a winning B graduated with no project,
dropping the conversation out of it and losing A's pin.

Give B the project_id and reasoning_effort at birth (still hidden, so it
never lists in the project's Chats tab), and on a B verdict carry A's
filing, pin, and read watermark onto the survivor — project_id via
coalesce so a pair opened before this fix still lands filed.
The chat shim's contact query — behind rag_search's contact leg and the
assistant's list lane — selected from app.contacts with no lifecycle
filter, while every read in the contacts domain excludes trashed rows, so
a contact the user had deleted kept resurfacing in answers as current.
Apply the same predicate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant