Skip to content

Two quick confirmations in one chat start parallel agent runs that repeat the same create_event #929

Open

Description

@alex-mextner

What

Observed 2026-11-01 (revision bb7a9d4): two «Да» user turns about 6 s apart in one private chat, each answering one of two bot confirmation prompts. The two agent runs overlapped (started about 17:06:30 and 17:06:33 UTC, ended about 17:08:01 and 17:08:03), and each ran create_event for both requested events with identical arguments, so both exist twice (all four rows local_only: no Google link, no sync_log, no invitations found). Source trace: production runs in webhook mode (src/bootstrap/launch.ts) and src/web/server.ts states that 200 is sent before processing, so Telegram can deliver the next update while the previous one is still running; src/bot/pipeline/ai-agent-layer.ts then awaits runAgent with no per-chat gate, and CalendarBotAgent.inFlight in src/services/ai/agent.ts is only a shutdown drain set. Work: decide between a per-chat queue and atomic idempotency for create_event (a duplicate check that reads then writes still races), implement, prove it. Every run must record its reply delivery outcome (sent, failed or unknown).

Verified 2026-11-01 from the pinned dependency source (gramio 0.7.0, @gramio/composer 0.3.4, node_modules/gramio/dist/index.js and node_modules/.bun/@Gramio+composer@0.3.4/.../dist/index.js): webhookHandler(bot, 'Bun.serve', {secretToken}) without shouldWait calls bot.updates.queue.add(update) and returns the 200 response at once. EventQueue.process() shifts every queued event and calls the handler without awaiting the previous one (while (queue.length > 0 && active) { handler(event)... }), so updates are handled concurrently, one handleUpdate per update, with no ordering and no serialization per chat or per user; long polling does the same through queue.addBatch(updates). Two updates of one chat therefore do run overlapping middleware chains and pipelines in both modes; only per-chat serialization in the bot (not the framework) can order them. Also found: the chat-logging middleware writes the user turn to chat_history at ingress, and chatHistoryIds (userId -> latest row) is a single slot per user, so a run built after a second message arrived would link its tool actions to the wrong chat_history row.

Why (motivation)

On 2026-11-01 around 17:06 UTC one private chat sent two confirmation messages 6 s apart; each started its own agent run in parallel with no per-chat serialization, both ran create_event with identical arguments, and every requested event now exists twice. Both runs ended error/unverified after a validator timeout and render jobs taking 73-80 s against a 45 s wait. Delivery of a final reply is unknown from the logs (deliveryActionMs 131 and 192 only). No data was deleted or replayed. Related: duplicate check in create_event.

User impact

Any user who answers a confirmation twice or sends a quick follow-up while the bot is slow gets duplicated calendar entries and possibly no clear answer; slow hosts make the window wider.

Cost of inaction

Duplicate events keep appearing whenever the bot is slow, and users cannot tell whether their message was handled.

Acceptance criteria

  • A regression test with two overlapping runs in one chat (synthetic data) creates each event once and fails before the fix — forced: no screenshot applies (non-visual). Evidence, PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 head 705cd9f: test/bot/pipeline/ai-agent-layer-overlap.test.ts 8 pass 0 fail incl. 'the second confirmation waits, is handled on the first run history and creates nothing' and 'a queued confirmation creates nothing the first run already created'; test/services/ai/tool-handlers/events-duplicate-create.test.ts 34 pass 0 fail; written red first; hosted PR CI test job green on this head (run 41760085024 stage-qa 6 cases passed). Not proof of production behaviour after deploy.
  • A second message that arrives while a run is in flight waits for it or is rejected explicitly, and every run records its reply delivery outcome (sent, failed or unknown) or an honest failure notice attempt
  • The 'Routing to AI agent' log line records the Telegram update_id and message_id so redelivery can be told from a second message — forced: no screenshot applies (non-visual). Evidence, PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 head 705cd9f: test/bot/pipeline/ai-agent-layer-overlap.test.ts 8 pass 0 fail incl. 'the routing log line carries the Telegram update id and message id next to the message text it already had'; test/bot/middleware/chat-logging.test.ts 3 pass 0 fail. Not proof of the deployed line; checked after deploy.
  • Sibling note for the write-receipt rewrite (the files write-outcomes and write-receipt on the secretary-invites branch are not edited here): create_event now returns ToolResult.alreadyExists = { entity: 'event', id } with success: true and no created-event evidence; the receipt fold must read result.alreadyExists (not the output prose) and report 'already in the calendar' with that event id, not count it as a created write; the intent path already carries it via ToolAnswer.alreadyExists. A same-day conflict is success: false, mutationState: not_applied, error starting SAME_DAY_EVENT, naming the existing id
  • Deferred, from review: per-chat queue depth is uncapped (one member flooding a group delays the chat, each queued message runs its own 4 s typing timer); next action: cap the queue and share one typing timer per chat

Screenshots

(none)

Deploy

Deployed: no

Links

Skipped gates

  • acceptance-criteria: adds a deferred-finding criterion, none removed
  • acceptance-edit: unchecked criterion 2 dropped or reworded
  • links: no external entity named
  • post-merge-acceptance: Criteria 1 and 3 are checked with test evidence. Criterion 2 is only half delivered by PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 (a second message waits for the run ahead of it; the per-run reply delivery outcome is NOT recorded yet), 4 is a hand-over note for the write-receipt rewrite and 5 is a deferred finding (queue depth cap); none of the three is closed by this merge and none is claimed closed. After the deploy the ticket gets the live evidence (exact revision, health, logs), and task done will need real proofs, so the ticket stays open.

Activity

  1. alex-mextner commented on Nov 1, 2026

    @alex-mextner
    OwnerAuthor

    Fix in branch fix/chat-serialization-20261101 (not merged, not deployed):

    • Per-chat serialization: CalendarBotAgent.run goes through ChatRunGate (src/services/ai/chat-run-gate.ts): one run at a time per chat id, FIFO, other chats parallel. A hung run holds the chat for at most its request budget plus 30 s, then the next run goes on and the log says so. Queued runs are in CalendarBotAgent.inFlight, so the shutdown drain waits for them; at shutdown the wait ends at once and the run takes its normal fail-fast path (user notice, stored retry), so no queued message is dropped.
    • A queued run re-reads the user and the schedule window, saves its own user turn again after what the earlier run wrote (ChatHistoryRepository.moveUserTurnToEnd, only when the row verifiably is its message) and is not shown the messages queued behind it, so a second «Да» reads: request, created events, «Да». The logging middleware now stamps each update with its own chat_history row id (the per-user map is overwritten by the next message).
    • The Routing to AI agent line now carries updateId and messageId (criterion 3).
    • Regression tests with synthetic data through the real layer and agent: test/bot/pipeline/ai-agent-layer-overlap.test.ts (fails on main with every event twice), test/services/ai/chat-run-gate.test.ts, test/services/ai/agent-chat-queue.test.ts, plus a drain test for a queued run (criterion 1 and the wait half of criterion 2).
    • NOT done in this slice: criterion 2's second half, a recorded reply delivery outcome (sent, failed or unknown) per run. It lives in the turn-finish and delivery path next to the validator/render work and is left open here.
    • Companion: create_event returns the existing event for a repeat within 10 minutes (create_event saves exact and near-duplicate events: no check for the same title on the same day #520), so a second run that does create again still yields one row.
  2. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Deferred from the single-seat review (Opus) of the staged work, each with a next action:

    • /cal and voice runs queued behind an active run get no reservation, so their user row can stay ahead of the earlier run's output; next action: reserve at the point those handlers save their row (or pass the row to run) and test queued voice and /cal.
    • Recurring creates skip the repeat check (findSameEvent), so a retried create_event of a series makes a second series; next action: match a live series by normalized title, start and recurrence_rule within the window.
    • Mixed multi-create workflows with a closing respond show the raw receipt of a new create beside the rendered "already exists" line; next action: store the ToolAnswer of confirmed writes and render it with renderToolAnswer.
    • Hung-run notice: a notice is now used up only by a run that can show it; a hung run is announced once. Runbook line "every run beside an unfinished earlier run is told" needs rewording to once per hung run.
      Single-seat review: only one reviewer seat was available (Sol quota exhausted).
  3. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Deferred from the final single-seat review of db0c087 (not fixed before commit; the pre-commit gate accepted the review run):

    • HIGH: a queued «Да» is moved after the earlier run's ask_user/SAME_DAY_EVENT question (turn_order placement plus created_at), so its model may read it as answering a question the user never saw and call create_event allow_duplicate / force / delete confirm. Next action: keep the original created_at, and when slot.waited tell the model the message predates the previous reply and answers none of its questions; add an overlap test whose first run ends in ask_user.
    • Medium: stale snapshot when the earlier run finishes just before B reaches the gate (waited=false skips the refresh); next action: refresh whenever the history changed since the snapshot.
  4. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Status 2026-11-02 for #942 (not merged, not deployed): the follow-up for the HIGH finding (a queued «Да» read as the answer to a question asked after it was sent) is written and staged in the worktree but NOT committed or pushed. The pre-commit review gate needs a review of the final staged diff; the only reviewer seat (Opus: single-seat review, Sol quota out) is refused by the account 5-hour usage cap (92% against the 80% limit) after about 80 minutes of retries. The PR head is still db0c087, whose hosted checks test and type-bans are red; the follow-up fixes both.

    Follow-up content: created_at stays the time the user sent the message and placement is turn_order only; a placed turn is shown to the model as [queued: ...]; the day-reference guard ignores rows saved after the message; the stale-snapshot refresh also runs when placement found newer rows; voice transcripts reserve their place in the chat line; a repeated series create is matched; migration 069 doc, migration-order tests, tool catalog budget, handler fixtures, no as never, KPI gate re-measured on the new engine digest (server-run; whole-repo tsc passed there).

    New tickets: #952 (mixed create receipts) and #953 (voice transcript sent-at). Queue-depth cap and per-run delivery outcome stay open here.

  5. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Status 2026-11-03T22:30Z (Ship942): PR #942 (head 1abbe94) is NOT merged and NOT deployed; production is still ff93220. All hosted checks incl. stage-qa (run 41696776659, passed cases=6) are green on that head. gh ship 942 --dry-run refuses only because pyproject.toml/uv.lock carry no version bump. The bump 0.3.0 -> 0.3.1 (#942 first; #941 would then need 0.3.2) is staged, uncommitted, in .worktrees/chat-serialization-20261101.

    Why it is not committed: the git pre-commit review-gate has no applicable provision (the trivial-delta stamp review-stamp-diff is keyed to HEAD 17f022d, a pre-rebase commit; current HEAD is 1abbe94), and review diff --staged --pool 1 --task '#929' was refused 4 times (21:46Z, 21:46Z, 22:06Z, 22:26Z), nothing dispatched: the Anthropic accounts have no complete 5h+7d usage pair (review-cli account cap), Sol is out until 2026-11-09. Blocker: review-cli #786. Nothing here is verified post-merge (deploy, odroid health, logs): acceptance stays open.

    Next: when review capacity-check --json shows Opus usable, run review diff --staged --task '#929' in that worktree, commit, push, wait for Stage build/QA + hosted checks, gh ship 942, then verify the exact deployed revision on the odroid and tell session 84833 the merged SHA (it rebases PR #940).

  6. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Review-gate probe, one call, no retries. Started 2026-11-03T23:31:05+01:00, finished 23:31:12+01:00. Command: review diff --staged --pool 1 --task '#929' in .worktrees/chat-serialization-20261101 (staged: the pyproject.toml and uv.lock 0.3.0 to 0.3.1 bump). Exit 0, nothing dispatched. Output, verbatim:

    [review-cli] warning: --pool 1 is below the default ship floor of 3 distinct board roles; this run cannot satisfy `review task CODE --check` / gh ship.
    [review-cli] auto-selected level standard (Laya standard 0.45; tier_not_acted:standard/uncertain:floor_standard); pass --level to choose it yourself
    [review-cli] level 'standard' is a fixed roster (Opus (claude:claude-opus-5-5)) and does not run without all of it: a blocked seat is never replaced by another or a cheaper model. Nothing was dispatched.
    
    Blocked seats:
      - Opus (claude:claude-opus-5-5): omp:anthropic/claude-opus-5-5: no omp anthropic account may be used (3): mextner@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); alex.mex.ux@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); invntrm@gmail.com: EXCLUDED (config.yaml omp_accounts.exclude); claude:claude-opus-5-5: usage неизвестен для аккаунта invntrm@gmail.com: нет подтверждённой пары 5ч/7д от omp usage; консервативно не используется (ограничение review-cli, config account_usage_cap_percent=80): a complete shared 5-hour and 7-day observation is missing
    
    Providers missing (0 of 3 roles live: none):
      - anthropic: Opus (claude:claude-opus-5-5): [same four account reasons as above]; back: unknown (no account usage window names a reset)
    
    Wait until the capacity is back (or free the account), or choose deliberately: `--level LEVEL` for another level, `-m MODEL` to name the model yourself.
    

    Result: the gate still refuses, same cause as before (the 5-hour plus 7-day usage pair is missing for the Anthropic accounts; review-cli fix is in review-cli issue 786). Not committed, not pushed, not shipped. PR 942 stays at head 1abbe94 without the version bump; production is unchanged by this work. The roster-review debt is tracked in #961.

  7. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #942.
    Merge commit: d0bc076
    This does NOT close the ticket — acceptance still needs proof. Run task read #929 for the outstanding criteria, then task check/task done.

  8. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped and deployed: PR #942 merged to main as d0bc0769d20897b5441b1e3f568611b8bdef4deb (version 0.3.1), deployed by the hosted CI/CD run 41762104573 (test, build, deploy all success).

    Review disclosure (once for this ticket): single-seat review via omp Opus (review diff --staged --task 929 -m omp:anthropic/claude-opus-5-5, three role invocations, account mextner). The version-bump commit got no findings. The default Opus+Sol roster refuses until Sol returns 2026-11-09T10:20Z; the full-roster review stays owed in #961.

    Deploy verification of the exact revision (odroid, 2026-11-04 03:32 UTC, container started 03:04:12 UTC, 28 minutes of logs):

    • /opt/hypercal/releases/current.json: revision d0bc0769d20897b5441b1e3f568611b8bdef4deb, image_id sha256:525ec763..., verified 03:05:37 UTC.
    • docker inspect hypercal-bot: Image sha256:525ec763... equals the loaded image id for the tag ...:d0bc0769d208..., label revision d0bc0769d208..., restarts 0, OOMKilled false, health healthy.
    • /health ok, /ready ok (unverified) (the same value the previous release's receipt recorded).
    • Migration 069_chat_history_turn_order applied once at start; no migration error.
    • Logs since start: 0 lines of level >= 50 (the old container had 5 in its last 24 h).
    • Memory: cgroup 431 MB (bun RSS 345 MB plus the render browser processes) 30 minutes after start, against 138 MiB seen on the old container after 28 h of uptime; load average 1.3. A fresh container is not yet swapped out the way a 28 h one is on this host; I will sample again and record it, I did not establish the cause.

    What is NOT live-tested: the overlapping-confirmation path itself. No real user's calendar was written, and the stage QA cases never change the calendar. The proof for it is the regression tests (checked on criteria 1 and 3 above) and the green stage-qa run 41760085024 (6 cases) on the exact head. No Agent run started after waiting for the previous run in this chat line has appeared in production yet (no overlapping traffic in this window). Criteria 2 (reply delivery outcome), 4 (hand-over note) and 5 (queue depth cap) stay open; this ticket is not done.

  9. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Follow-up recorded after the merge of PR 940 (merge 4b2372d, on top of the per-chat serialization change d0bc076): the structured duplicate-create outcome ('Event already exists') still needs a fold into the readable write receipt, in src/services/ai/write-outcomes.ts and src/services/ai/write-receipt.ts. The receipt introduced by PR 940 does not know that outcome yet; until it does, a duplicate create would be shown as an ordinary refusal. Not implemented here because the outcome shape only became available on main with d0bc076.

  10. alex-mextner commented on Nov 7, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #974.
    Merge commit: 128846b
    This does NOT close the ticket — acceptance still needs proof. Run task read #929 for the outstanding criteria, then task check/task done.

  11. alex-mextner commented on Nov 7, 2026

    @alex-mextner
    OwnerAuthor

    Shipped the receipt side of the idempotent create (the hand-over item of this ticket): PR #974, merge commit 128846b.

    What changed: the write receipt reads create_event's ToolResult.alreadyExists = { entity: 'event', id } (never the output prose) as the same event. A creation and its repeat are one created line, no "Not created", no guard when nothing failed; a repeat of an event the run did not create is told once as "Already in the calendar" / "Уже есть в календаре". A refused try followed by the repeat counts as created; an earlier try of unknown result keeps its duplicate warning; a different event, a deliberate allow_duplicate copy and group-chat target hiding are unchanged. The receipt-only confirmation after a validator outage follows the same rule. Version 0.3.7.

    Evidence: new tests drive the real handler into the real ledger in ru and en (write-receipt-existing-event.test.ts, 20 tests, red first against main) plus receipt-confirmation.test.ts. Hosted CI on the exact head 24c73cb passed (test, lint, type-bans, leftover-grep, review-threads, gitleaks, CodeQL, dependency-review, stage-qa 6 cases). The build server's own run had 6 failures that are environment (stage-QA Python installer suites: no system git and a nearly full disk, #955; Chromium libs), so the hosted run is the validator, not the server.

    Deployed: the production container runs the image tagged with the merge SHA; /health ok, /ready ok (unverified), releases/current.json revision equals the merge SHA, no level 50/60 and no level 40 log records since the container start (about a minute of uptime when checked, not a live conversation test).

    Still open in this ticket: a second message waiting for the run ahead of it is shipped (#942) but the per-run reply delivery outcome is not recorded yet; the queue depth cap; live acceptance of the overlap case after deploy. Not closing the ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions