Skip to content

[Fix] Sessions report an outdated status after a queued update arrives late - #2596

Open
roomote-roomote[bot] wants to merge 10 commits into
developfrom
fix/fast-session-canonical-state-2hxyo73zu311l
Open

[Fix] Sessions report an outdated status after a queued update arrives late#2596
roomote-roomote[bot] wants to merge 10 commits into
developfrom
fix/fast-session-canonical-state-2hxyo73zu311l

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

​Opened on behalf of @daniel-lxs. Follow up by mentioning @roomote-roomote, in the web UI, or in Telegram.

What changed

A Session could contradict itself about the same thing. Authority over which status was current lived in prompt wording, and a Session that resumed after a restart reconstructed its history from a different source than one that kept running.

Sessions now keep one ordered record of everything the agent reads, and decide what is current in code:

  • One canonical log. Every model-relevant input is recorded in fast_agent_messages, including events the transcript deliberately hides, such as delegated-task reports, pull-request updates, artifacts, wakeups, automations, and setup state. Admission happens in the same transaction as queue admission, so an accepted event is never model-invisible.
  • Deterministic order and two distinct times. Each event receives a per-Session conversation_seq allocated under the existing per-conversation advisory lock, so admission order is total and stable. observed_at records when the source was observed while created_at remains when Roomote admitted it.
  • Typed state semantics. Events declare whether they are a historical observation, a state change, or a current-state assertion, along with the authority that produced them and the subject they describe. A version travels with an event only where the source genuinely provides one. Nothing invents a version for lifecycles providers do not number.
  • A reducer instead of prompt rules. Before inference, a pure projection classifies every event per subject as current, historically relevant, superseded and irrelevant, or conflicting. A superseded queued assertion never reaches the model at all. Equally authoritative disagreement at the same evidence strength and version is surfaced as a conflict rather than silently resolved.
  • Freshness decided by evidence, not arrival. For one subject the order is authority, then what the event claims about the present, then a comparable monotonic version, then observation time, then admission order. A merged or closed pull request stays authoritative even when a later task re-emits an opening event for it, while a genuine reopen still wins as soon as a producer asserts it as current state.
  • Warm and cold parity. The projection is hashed and the hash plus its sequence watermark are stored with the native OpenCode session on every turn, so ordinary follow-ups stay warm. A session is rebuilt from canonical history only when semantic events are genuinely ahead of its watermark or the state it was built on changed. Cold recovery replays the canonical log, including hidden-event provenance.
  • Claim provenance. Each assistant reply records, internally, the reducer version, projection hash, sequence watermark, and contributing event IDs behind it.

One owner per rule

A later pass removed duplicated decision-making rather than adding to it. Queue admission and the executing turn both wrote the same canonical row from independently derived semantics, which is why the upsert needed a jsonb guard to stop the second writer from overwriting the first. One builder now describes a Session input, the turn reuses the admitted record instead of restating the observation as execution time, and the guard is gone; a read that fails writes nothing for that key, so a queued event's current-state assertion can never be replaced by the weaker shape a human input has. Canonical history no longer falls back to the legacy transcript merely because rendering came out empty — eligibility is explicit, so a legitimately empty projection stays empty. Restored history attachments had two preparation rules, and the retry's own rule dropped them under a helper delivery; both paths now share one preparation that keeps the direct, helper, and unsupported modes and the per-rebuild bound. The provenance header the current input and rebuilt history both emit is shared too.

Ranking a kind or an authority used the in operator, which accepts inherited members: toString or constructor passed validation and then ranked as a function, making the ordering comparison NaN and destroying the total order the reducer depends on. Both checks are own-property tests now.

A clean retry also rebuilds the whole conversation, so attachments an earlier turn provided have to travel with it. That path only prepared them under a delivery mode the turn had already resolved, and a text-only turn never resolves one, so those rebuilds dropped every restored image. The retry now resolves delivery itself when restored attachments exist, which keeps the lookup off turns that never retry.

Migrations

This branch originally shipped 0086 and 0087. 0086 was claimed on develop in the meantime, so the work is now a single migration, 0087, applied after it. It adds nullable conversation_seq and observed_at plus the session projection watermark, backfills existing rows deterministically by (created_at, ts, turn_seq, id), keeps the previous ordering index under fast_agent_messages_legacy_order_idx, and adds a partial unique index over non-null sequences. Nothing is dropped or made required: a previous release keeps writing rows without a sequence, the partial index cannot reject those rows, and the next admission repairs them under the conversation lock. Note for reviewers: the deterministic backfill is hand-written and drizzle-kit generate will not reproduce it, so regenerating this file would silently drop it.

How it was tested

Continuity when a delayed task report is consumed late. This is the behavior the change exists for, so it is covered directly on the real child-report path. A child_message is admitted as a historical_observation, and the reducer deliberately never subject-projects those, so supersession does not apply to it — the earlier reducer scenario built from three versioned setup assertions travels a different path and is not acceptance evidence for this behavior. The new tests instead arrange the report the way the symptom occurs, keeping the lower conversation sequence it was admitted with while the parent was busy, with newer human and assistant turns above it. They assert the assembled context rather than the outcome, so the candidate failure modes stay separable: both newer turns must survive the rebuild (ruling out missing context), they must precede the report being executed (ruling out an ordering error), and the report's provenance including its earlier observation time must survive (leaving authority judgement observable). Warm continuation asserts the native transcript holding those turns is not invalidated; a later human turn asserts it still sees the correction, its reply, and the report together.

Reducer and freshness. Yellow v10 → queued green v11 → yellow v12 leaves only v12 current and drops green from rendered history; source versions beat admission order for late arrivals; unversioned assertions fall back to observation time; same-version disagreement from equal authority is conflicting; cold rendering equals the warm prefix plus its canonical suffix. A terminal pull-request status stays current when a later task re-emits an opening event for the same pull request, and closed → reopened and ready → draft are not pinned by the earlier state. Validation rejects inherited object members as a kind or an authority, and one unrankable claim cannot reorder the claims around it.

Ownership. The admitted semantics are offered for the turn to reuse and are preserved when a later writer carries none; a failed read writes nothing for that key; canonical history is not abandoned for the legacy transcript when it renders empty; restored attachments reach a helper model as announced attachment IDs and a direct model as files, on both the rebuild and the retry, including the text-only warm turn that never resolved a delivery mode.

Concurrency, against real Postgres admission and the real durable drain. Events are consumed in admission order with older states superseded and no stale state in the surviving turn's context; a turn held at its projection boundary is not retroactively suppressed; duplicate admission collapses to one queue row and one canonical row; a cold replay of the durable log reproduces the warm turn's context exactly.

Migrations. All 88 apply cleanly to a fresh database with every expected index present, and the deterministic backfill was exercised against rows left unsequenced the way a previous release leaves them, producing dense sequences ordered by created_at with observed_at derived from ts.

Real-preview smoke with live inference. On the pushed head, against real Postgres, Redis, BullMQ, and OpenCode: mixed text+image admission, warm reuse on the same native session with cache reads matching the cold write, durable queued supersession suppressing both older assertions, and cold image recovery restoring a historical attachment after the native session was cleared. For the delayed-report behavior, a real report was queued while the drain was down, a human correction ran inline and was answered, and the report was then consumed last; the model did not restate the stale claim, and a following human turn answered correctly both warm and after a forced cold rebuild that read context only from the canonical log.

pnpm lint:fast, oxlint, check-types:fast across 27 packages, knip, and format:check all pass. cloud-agents Fast suites are at 819 tests, the SDK src/server/lib suite at 1620, and packages/db at 722. Two pre-existing AgentMail stranger-refusal failures reproduce identically on a clean develop checkout and are unrelated to this change.

Limits worth stating

The original report was observed behavior, not established causality, and it did not reproduce in the arrangement above, so no root cause is claimed for it and this change should not be read as proving that behavior fixed. What is demonstrated is continuity: the context and provenance a delayed report is judged against are complete and correctly ordered, warm and cold. A child_message carries no producer timestamp, so its observed_at is admission time and occurred_at is absent; a report genuinely produced long before admission would still look fresh. Service-path tests mock the inference call, so they prove which files and prompt text reach OpenCode, not provider acceptance. The clean-retry image path is covered by those mocked-inference tests rather than a live provider failure.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

@roomote-community

roomote-community Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

No new code issues found. See task

  • packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts:4885 Persist the projection watermark for all canonical history so ordinary follow-up turns can reuse their native OpenCode session.
  • packages/sdk/src/server/lib/fast-agent-parent-event-queue.ts:267 Preserve terminal PR-state precedence: the reducer now ranks a current-state assertion above a later opening transition for the same subject and authority.
  • packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts:3320 Route restored historical images through the non-vision delivery path so cold rebuilds can inspect them instead of attaching them to an unsupported native model.
  • packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts:5512 Preserve historical images when a text-only warm turn falls back to a clean retry.

Reviewed e8cab9e

Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts Outdated
Comment thread packages/sdk/src/server/lib/fast-agent-parent-event-queue.ts Outdated
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review September 12, 2026 17:44
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts
Queue admission recorded what an input claims and the executing turn
derived it again, so the same canonical row had two authors and the
upsert needed a jsonb guard to stop the second from overwriting the
first. One builder now describes a Session input, the turn reuses the
admitted record instead of restating the observation as execution time,
and the guard is gone. A read that fails writes nothing for the key, so
a queued event's current-state assertion cannot be replaced by the
weaker shape a human input has.

Canonical history also no longer falls back to the legacy transcript
just because rendering came out empty. Eligibility is explicit: any
canonical event other than the input being executed makes canonical
history authoritative, and only a session that never recorded one reads
as pre-canonical.

Restored attachments had two preparation rules, and the retry's own
rule dropped them under a helper delivery. Both paths share one
preparation that keeps the direct, helper, and unsupported modes and
the per-rebuild bound, holding at most once per turn so a retry cannot
reserve a second set of IDs for the same bytes. The provenance header
the current input and rebuilt history both emit is shared too.

Ranking a kind or an authority also used `in`, which accepts inherited
members: `toString` or `constructor` passed validation and then ranked
as a function, making the ordering comparison NaN and destroying the
total order the reducer depends on. Both checks are own-property tests.
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts Outdated
A clean retry rebuilds the whole conversation, so attachments an earlier
turn provided have to travel with it. The retry only prepared them under
a delivery mode the turn had already resolved, and a text-only turn never
resolves one, so those rebuilds dropped every restored image.

The premise behind that guard was wrong: `prepareRetry` is declared
`() => Promise<void> | void` and the retry driver already awaits it, so
the branch can resolve delivery itself. It now does, only when restored
attachments exist, which keeps the lookup off turns that never retry.
Awaiting there is safe because holding and building the prompt still run
without interruption between them.

Regressions cover the reviewed path directly: a text-only warm turn that
falls back to a clean retry attaches the restored file under a direct
delivery, and holds and announces it for `inspect_images` under a helper
delivery. Both fail against the previous guard.
The reported symptom is a delayed delegated-task report making a Session
speak as though the conversation rolled back. The existing coverage for
that used three synthetic versioned setup assertions, which travel a
different path entirely: a child report is admitted as a
`historical_observation`, and the reducer deliberately never
subject-projects those, so supersession cannot apply to it and that test
proved nothing about this behavior.

These tests use the real child-report shape instead, arranged the way
the symptom occurs: the report keeps the lower conversation sequence it
was admitted with while the parent was busy, and newer human and
assistant turns that corrected the picture sit above it.

They assert the assembled context rather than the outcome, so the three
candidate failure modes are separable. Missing context is ruled out by
requiring both newer turns to survive the rebuild. An ordering error is
ruled out by requiring them to precede the report being executed. Bad
authority is left observable by requiring the report's provenance to
survive, including the earlier observation time that distinguishes it
from the newer turns. Warm continuation asserts the native transcript
holding those turns is not invalidated, and a later human turn asserts
it still sees the correction, its reply, and the report together.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant