[Fix] Sessions report an outdated status after a queued update arrives late - #2596
Open
roomote-roomote[bot] wants to merge 10 commits into
Open
[Fix] Sessions report an outdated status after a queued update arrives late#2596roomote-roomote[bot] wants to merge 10 commits into
roomote-roomote[bot] wants to merge 10 commits into
Conversation
Contributor
|
No new code issues found. See task
Reviewed e8cab9e |
roomote-roomote
Bot
requested review from
brunobergher,
daniel-lxs and
mrubens
as code owners
September 12, 2026 17:44
Queue admission recorded what an input claims and the executing turn derived it again, so the same canonical row had two authors and the upsert needed a jsonb guard to stop the second from overwriting the first. One builder now describes a Session input, the turn reuses the admitted record instead of restating the observation as execution time, and the guard is gone. A read that fails writes nothing for the key, so a queued event's current-state assertion cannot be replaced by the weaker shape a human input has. Canonical history also no longer falls back to the legacy transcript just because rendering came out empty. Eligibility is explicit: any canonical event other than the input being executed makes canonical history authoritative, and only a session that never recorded one reads as pre-canonical. Restored attachments had two preparation rules, and the retry's own rule dropped them under a helper delivery. Both paths share one preparation that keeps the direct, helper, and unsupported modes and the per-rebuild bound, holding at most once per turn so a retry cannot reserve a second set of IDs for the same bytes. The provenance header the current input and rebuilt history both emit is shared too. Ranking a kind or an authority also used `in`, which accepts inherited members: `toString` or `constructor` passed validation and then ranked as a function, making the ordering comparison NaN and destroying the total order the reducer depends on. Both checks are own-property tests.
…anonical-state-2hxyo73zu311l
A clean retry rebuilds the whole conversation, so attachments an earlier turn provided have to travel with it. The retry only prepared them under a delivery mode the turn had already resolved, and a text-only turn never resolves one, so those rebuilds dropped every restored image. The premise behind that guard was wrong: `prepareRetry` is declared `() => Promise<void> | void` and the retry driver already awaits it, so the branch can resolve delivery itself. It now does, only when restored attachments exist, which keeps the lookup off turns that never retry. Awaiting there is safe because holding and building the prompt still run without interruption between them. Regressions cover the reviewed path directly: a text-only warm turn that falls back to a clean retry attaches the restored file under a direct delivery, and holds and announces it for `inspect_images` under a helper delivery. Both fail against the previous guard.
The reported symptom is a delayed delegated-task report making a Session speak as though the conversation rolled back. The existing coverage for that used three synthetic versioned setup assertions, which travel a different path entirely: a child report is admitted as a `historical_observation`, and the reducer deliberately never subject-projects those, so supersession cannot apply to it and that test proved nothing about this behavior. These tests use the real child-report shape instead, arranged the way the symptom occurs: the report keeps the lower conversation sequence it was admitted with while the parent was busy, and newer human and assistant turns that corrected the picture sit above it. They assert the assembled context rather than the outcome, so the three candidate failure modes are separable. Missing context is ruled out by requiring both newer turns to survive the rebuild. An ordering error is ruled out by requiring them to precede the report being executed. Bad authority is left observable by requiring the report's provenance to survive, including the earlier observation time that distinguishes it from the newer turns. Warm continuation asserts the native transcript holding those turns is not invalidated, and a later human turn asserts it still sees the correction, its reply, and the report together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
A Session could contradict itself about the same thing. Authority over which status was current lived in prompt wording, and a Session that resumed after a restart reconstructed its history from a different source than one that kept running.
Sessions now keep one ordered record of everything the agent reads, and decide what is current in code:
fast_agent_messages, including events the transcript deliberately hides, such as delegated-task reports, pull-request updates, artifacts, wakeups, automations, and setup state. Admission happens in the same transaction as queue admission, so an accepted event is never model-invisible.conversation_seqallocated under the existing per-conversation advisory lock, so admission order is total and stable.observed_atrecords when the source was observed whilecreated_atremains when Roomote admitted it.One owner per rule
A later pass removed duplicated decision-making rather than adding to it. Queue admission and the executing turn both wrote the same canonical row from independently derived semantics, which is why the upsert needed a
jsonbguard to stop the second writer from overwriting the first. One builder now describes a Session input, the turn reuses the admitted record instead of restating the observation as execution time, and the guard is gone; a read that fails writes nothing for that key, so a queued event's current-state assertion can never be replaced by the weaker shape a human input has. Canonical history no longer falls back to the legacy transcript merely because rendering came out empty — eligibility is explicit, so a legitimately empty projection stays empty. Restored history attachments had two preparation rules, and the retry's own rule dropped them under a helper delivery; both paths now share one preparation that keeps the direct, helper, and unsupported modes and the per-rebuild bound. The provenance header the current input and rebuilt history both emit is shared too.Ranking a kind or an authority used the
inoperator, which accepts inherited members:toStringorconstructorpassed validation and then ranked as a function, making the ordering comparisonNaNand destroying the total order the reducer depends on. Both checks are own-property tests now.A clean retry also rebuilds the whole conversation, so attachments an earlier turn provided have to travel with it. That path only prepared them under a delivery mode the turn had already resolved, and a text-only turn never resolves one, so those rebuilds dropped every restored image. The retry now resolves delivery itself when restored attachments exist, which keeps the lookup off turns that never retry.
Migrations
This branch originally shipped
0086and0087.0086was claimed ondevelopin the meantime, so the work is now a single migration,0087, applied after it. It adds nullableconversation_seqandobserved_atplus the session projection watermark, backfills existing rows deterministically by(created_at, ts, turn_seq, id), keeps the previous ordering index underfast_agent_messages_legacy_order_idx, and adds a partial unique index over non-null sequences. Nothing is dropped or made required: a previous release keeps writing rows without a sequence, the partial index cannot reject those rows, and the next admission repairs them under the conversation lock. Note for reviewers: the deterministic backfill is hand-written anddrizzle-kit generatewill not reproduce it, so regenerating this file would silently drop it.How it was tested
Continuity when a delayed task report is consumed late. This is the behavior the change exists for, so it is covered directly on the real child-report path. A
child_messageis admitted as ahistorical_observation, and the reducer deliberately never subject-projects those, so supersession does not apply to it — the earlier reducer scenario built from three versioned setup assertions travels a different path and is not acceptance evidence for this behavior. The new tests instead arrange the report the way the symptom occurs, keeping the lower conversation sequence it was admitted with while the parent was busy, with newer human and assistant turns above it. They assert the assembled context rather than the outcome, so the candidate failure modes stay separable: both newer turns must survive the rebuild (ruling out missing context), they must precede the report being executed (ruling out an ordering error), and the report's provenance including its earlier observation time must survive (leaving authority judgement observable). Warm continuation asserts the native transcript holding those turns is not invalidated; a later human turn asserts it still sees the correction, its reply, and the report together.Reducer and freshness. Yellow v10 → queued green v11 → yellow v12 leaves only v12 current and drops green from rendered history; source versions beat admission order for late arrivals; unversioned assertions fall back to observation time; same-version disagreement from equal authority is conflicting; cold rendering equals the warm prefix plus its canonical suffix. A terminal pull-request status stays current when a later task re-emits an opening event for the same pull request, and closed → reopened and ready → draft are not pinned by the earlier state. Validation rejects inherited object members as a kind or an authority, and one unrankable claim cannot reorder the claims around it.
Ownership. The admitted semantics are offered for the turn to reuse and are preserved when a later writer carries none; a failed read writes nothing for that key; canonical history is not abandoned for the legacy transcript when it renders empty; restored attachments reach a helper model as announced attachment IDs and a direct model as files, on both the rebuild and the retry, including the text-only warm turn that never resolved a delivery mode.
Concurrency, against real Postgres admission and the real durable drain. Events are consumed in admission order with older states superseded and no stale state in the surviving turn's context; a turn held at its projection boundary is not retroactively suppressed; duplicate admission collapses to one queue row and one canonical row; a cold replay of the durable log reproduces the warm turn's context exactly.
Migrations. All 88 apply cleanly to a fresh database with every expected index present, and the deterministic backfill was exercised against rows left unsequenced the way a previous release leaves them, producing dense sequences ordered by
created_atwithobserved_atderived fromts.Real-preview smoke with live inference. On the pushed head, against real Postgres, Redis, BullMQ, and OpenCode: mixed text+image admission, warm reuse on the same native session with cache reads matching the cold write, durable queued supersession suppressing both older assertions, and cold image recovery restoring a historical attachment after the native session was cleared. For the delayed-report behavior, a real report was queued while the drain was down, a human correction ran inline and was answered, and the report was then consumed last; the model did not restate the stale claim, and a following human turn answered correctly both warm and after a forced cold rebuild that read context only from the canonical log.
pnpm lint:fast,oxlint,check-types:fastacross 27 packages,knip, andformat:checkall pass. cloud-agents Fast suites are at 819 tests, the SDKsrc/server/libsuite at 1620, andpackages/dbat 722. Two pre-existing AgentMail stranger-refusal failures reproduce identically on a cleandevelopcheckout and are unrelated to this change.Limits worth stating
The original report was observed behavior, not established causality, and it did not reproduce in the arrangement above, so no root cause is claimed for it and this change should not be read as proving that behavior fixed. What is demonstrated is continuity: the context and provenance a delayed report is judged against are complete and correctly ordered, warm and cold. A
child_messagecarries no producer timestamp, so itsobserved_atis admission time andoccurred_atis absent; a report genuinely produced long before admission would still look fresh. Service-path tests mock the inference call, so they prove which files and prompt text reach OpenCode, not provider acceptance. The clean-retry image path is covered by those mocked-inference tests rather than a live provider failure.Checklist
[Fix],[Feat],[Improve],[Refactor],[Docs], or[Chore]followed by a user-facing descriptionpnpm lintandpnpm check-typespass locallypnpm changeset