Skip to content

fix(platform): queue serializable task-comment writes per task - #3268

Merged
larryro merged 6 commits into
mainfrom
fix/task-comment-retry-queue
Sep 6, 2026
Merged

fix(platform): queue serializable task-comment writes per task#3268
larryro merged 6 commits into
mainfrom
fix/task-comment-retry-queue

Conversation

@larryro

@larryro larryro commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Defect

addTaskComment runs under transactSerializable (task comment route, connector task store) and every comment on a task bumps the same rows: the task's comment_count and the discussion thread's next message slot. Serializable commenters that overlap therefore all lose but the first, and a plain retry loses again whenever another commits first. transactSerializable gives up after five attempts — a count any burst larger than itself defeats — so a burst of comments on one task (several agents reporting on one task, a fan-out) fails legitimate comments with a 40001. This is the same storm #3246 fixed for the audit chain head, and the serializable twin of the count bound #3267 removes from the message slot under READ COMMITTED. Surfaced by the adversarial review of #3267 and priced there as a separate change.

Fix

domains/tasks/comments.ts: comment appends and deletes run through queuedOnTask, the twin of the audit head's lockChainHead:

  • a transaction-level advisory lock on task-comment:<taskId> (the shared RETRY_QUEUE_LOCK_CLASS) is taken before the write's first read, so a first attempt queues behind a retry that holds the key as a session lock from before its BEGIN;
  • a 40001/40P01 raised anywhere in the write is marked with the key (markRetryQueueKey), which makes the caller's next attempt take that session lock first (transactSerializable's retry queue).

A commenter wastes at most one attempt under contention. READ COMMITTED callers (the date nudge, the sandbox shim) pay only the lock, which orders one task's comments and marks nothing. Edits touch no shared row and stay as they were. Public signatures are unchanged.

Proof

  • New integration lane tasks: a burst of serializable commenters on one task all land: twelve serializable commenters on one task with the overlap widened → landed=12/12 rejected=0 commentCount=12 distinctOrders=12 attempts=23 — exactly N + (N−1): every loser retried once, queued, none twice.
  • Real-Postgres integration run on this branch: 525/526 across 139/139 lanes. The one red is messages: concurrent appends each take their own slot (appends=12, failed=1) — the READ COMMITTED slot-count defect that fix(platform): claim message slots until a deadline, not a fixed count #3267 fixes; this branch is cut from main 4df487e, which predates it. With fix(platform): claim message slots until a deadline, not a fixed count #3267 merged the run is 526/526.
  • tsc --noEmit, oxlint --type-aware, knip:check, oxfmt clean; vitest --project server backend/domains/tasks: 13 files / 80 tests green.

Review round 1 (adversarial review: changes requested) — repaired in 9dba83b, b53b7a0, de4857a

Two real blockers, both fixed:

  1. The task key clobbered the chain-head key. queuedOnTask re-marked every 40001 with the task key, overwriting the mark lockChainHead had put on a loss at the org's audit chain head, so a comment retry queued on the task key alone and met the other tasks' commenters at the chain head again — the fix(platform): harden erasure, legal-hold and audit-log paths #3246 storm returned for cross-task bursts in one org. The shared retry queue now carries a key list: marks nest (an outer mark is prepended to the inner ones, never twice), beginQueued takes the session locks in that order before BEGIN and unlocks in reverse, and the retry loop adopts the whole list. First attempts take their transaction-level locks in the same order (task, then chain head), so no two transactions ever hold the two keys in opposite orders. retryQueueKeysOf exposes the list; retryQueueKeyOf still answers the outermost key.
  2. Lock-order inversion in the overdue nudge. The level-2 rung of date-notifications.ts stamped the task row and then commented, while every commenter takes the task key first and writes the row second — a deadlock pair resolved only after deadlock_timeout. The claim now takes lockTaskCommentQueue as its first statement; the unit test pins key-before-claim.

Also from the review: CommentMeta named instead of an inline type; queuedOnTask and the retry-queue note now describe the nesting; new unit suite comments.queue.test.ts (lock first, task key in front of an inner audit key, other failures untouched) and serializable.test.ts cases for nested keys (19/19).

Proof on the repaired tree: real-Postgres integration run 527/527 across 140/140 lanes (the message-slot lane happened to pass this time). New cross-task lane tasks: a burst of serializable commenters across one org's tasks all land: landed=12/12 rejected=0 chainValid=true attempts=23 — every loser at the chain head retried exactly once, queued on task + chain head; the same-task lane again attempts=23; the audit lane unchanged (8/8, attempts=15). Full vitest --project server: 581 files / 6325 tests green; shared + platform tsc, oxlint, knip, oxfmt clean.

Review round 2 (adversarial re-review: changes requested) — repaired in 3b12792

One real defect the round-1 repair introduced: with several keys, beginQueued locked them OUTSIDE its unlock guard, so a failure on a later pg_advisory_lock (a cancelled statement, lock-table exhaustion) returned the pooled connection with the first key still held as a session lock — every later lock on that key would wait until that connection died. The lock loop now runs inside the guard: the keys already held are unlocked in reverse on every exit. Unit test: a 57014 on the second lock → lock, lock, unlock(first) — no BEGIN, no second attempt, connection released.

Also hardened (advisory): the retry loop keeps its key list when a marked failure carries a subset of it (a queued attempt that lost to a writer outside its inner queues), so a retry never drops the inner key; unit test covers it. serializable.test.ts is now 21/21.

Review round 3 (adversarial re-review): approve — nits applied in 1aced44

Round 2's repair holds on every exit path (lock failure on any key, BEGIN failure, callback failure, COMMIT answering ROLLBACK, COMMIT throwing, unlock failure): exactly the acquired keys are unlocked in reverse and the connection is released once; the single-key (audit) statement sequence is byte-for-byte unchanged. Two nits fixed: the subset helper had landed between transactSerializable's doc and the function; the cancelled-lock test now also asserts a single reservation. Reviewer's conclusion on the key-list rule: only two mark sites exist (lockChainHead, queuedOnTask) and both queuedOnTask callers queue on one task per transaction, so the reachable transitions are superset → replace and subset → keep; a disjoint mark is unreachable today and replace stays the conservative choice.

Every comment on a task bumps the same rows — the task's comment_count
and the discussion thread's next message slot — so serializable
commenters that overlap all lose but the first, and a plain retry loses
again whenever another commits first. `transactSerializable` gives up
after five attempts, a count any burst larger than itself defeats; the
audit chain head had the same storm (#3246) and the message slot under
READ COMMITTED had its count-bound twin (#3267).

Comment appends and deletes now run through `queuedOnTask`: the
transaction-level advisory lock on `task-comment:<taskId>` queues a first
attempt behind a retry that holds the key as a session lock from before
its BEGIN, and a 40001/40P01 raised anywhere in the write is marked with
the key so the caller's next attempt takes that session lock first. A
commenter therefore wastes at most one attempt; READ COMMITTED callers
pay only the lock. Edits touch no shared row and stay as they were.

The integration proof gains a lane: twelve serializable commenters on one
task, overlap widened, all land with one comment_count of twelve, twelve
distinct message slots, and between thirteen and twenty-four attempts —
every loser retried exactly once, queued.
A retry queued on one key drops the others: a task comment that lost at
the org's audit chain head was re-marked with the task key by its outer
queue, so its retry took the task's session lock alone, met the other
tasks' commenters at the chain head again and lost there — the storm
#3246 removed came back for cross-task bursts.

Marks now nest. `markRetryQueueKey` prepends an outer key to the keys an
inner callback already put on the error (never twice), so the list reads
in the callback's own acquisition order; `beginQueued` takes the session
locks in that order before BEGIN and unlocks in reverse; the retry loop
adopts the whole list whenever a marked failure arrives. Every first
attempt takes its transaction-level locks in the same order, so a queued
retry and a first attempt never hold two keys in opposite orders.
`retryQueueKeysOf` exposes the list; `retryQueueKeyOf` keeps answering
the outermost key.
…laim

The overdue ladder's level-2 rung stamps the task row and then comments
on the task, while every commenter takes the task's comment key first
and writes the row second — two lock orders for one pair of resources,
which Postgres resolves after deadlock_timeout with a 40P01 that the
READ COMMITTED commenters (agents, workflows, REST) surface as an error.
The claim now takes the key as its first statement, the order everyone
else uses; the unit test pins key-before-claim.
`queuedOnTask` marks a serialization failure with the task key on top of
the chain-head key the audit write inside it already carries, and the
shared retry queue now honours both, so a retry holds the task AND the
org's chain head from before its BEGIN. The lock statement is exported
(`lockTaskCommentQueue`) for transactions that write the task row before
commenting, the meta row shape has a name, and unit tests pin the
contract: lock first, task key in front of an inner audit key, other
failures untouched.

The integration proof gains the cross-task lane: twelve serializable
commenters on twelve tasks of one org meet only at the chain head, all
land, the chain verifies, and attempts stay within twice the burst.
The key list made `beginQueued` take several session locks in a row, but
the loop sat outside the unlock guard: a failure on the second key (a
cancelled statement, lock-table exhaustion) skipped the unlocks and
returned the connection to the pool still holding the first key — every
later transaction-level lock on that key would then wait until that
pooled connection died. The loop now runs inside the guard, so the keys
already held are unlocked in reverse on every exit and the connection is
released clean.

The retry loop also keeps its key list when a marked failure carries a
subset of it: a queued attempt that loses to a writer outside its inner
queues is re-marked with the outer key alone, and dropping the inner key
is how a retry loses at that resource again.
The subset helper had landed between transactSerializable's JSDoc and the function; it now sits above with its own line. The cancelled-lock test also asserts that no second reservation happens, so a future widening of the transient table is caught here rather than trusted.
@larryro
larryro merged commit a32d61e into main Sep 6, 2026
64 of 65 checks passed
@larryro
larryro deleted the fix/task-comment-retry-queue branch September 6, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant