Repository navigation
Description
What
Observed 2026-11-01 (revision bb7a9d4): two «Да» user turns about 6 s apart in one private chat, each answering one of two bot confirmation prompts. The two agent runs overlapped (started about 17:06:30 and 17:06:33 UTC, ended about 17:08:01 and 17:08:03), and each ran create_event for both requested events with identical arguments, so both exist twice (all four rows local_only: no Google link, no sync_log, no invitations found). Source trace: production runs in webhook mode (src/bootstrap/launch.ts) and src/web/server.ts states that 200 is sent before processing, so Telegram can deliver the next update while the previous one is still running; src/bot/pipeline/ai-agent-layer.ts then awaits runAgent with no per-chat gate, and CalendarBotAgent.inFlight in src/services/ai/agent.ts is only a shutdown drain set. Work: decide between a per-chat queue and atomic idempotency for create_event (a duplicate check that reads then writes still races), implement, prove it. Every run must record its reply delivery outcome (sent, failed or unknown).
Verified 2026-11-01 from the pinned dependency source (gramio 0.7.0, @gramio/composer 0.3.4, node_modules/gramio/dist/index.js and node_modules/.bun/@Gramio+composer@0.3.4/.../dist/index.js): webhookHandler(bot, 'Bun.serve', {secretToken}) without shouldWait calls bot.updates.queue.add(update) and returns the 200 response at once. EventQueue.process() shifts every queued event and calls the handler without awaiting the previous one (while (queue.length > 0 && active) { handler(event)... }), so updates are handled concurrently, one handleUpdate per update, with no ordering and no serialization per chat or per user; long polling does the same through queue.addBatch(updates). Two updates of one chat therefore do run overlapping middleware chains and pipelines in both modes; only per-chat serialization in the bot (not the framework) can order them. Also found: the chat-logging middleware writes the user turn to chat_history at ingress, and chatHistoryIds (userId -> latest row) is a single slot per user, so a run built after a second message arrived would link its tool actions to the wrong chat_history row.
Why (motivation)
On 2026-11-01 around 17:06 UTC one private chat sent two confirmation messages 6 s apart; each started its own agent run in parallel with no per-chat serialization, both ran create_event with identical arguments, and every requested event now exists twice. Both runs ended error/unverified after a validator timeout and render jobs taking 73-80 s against a 45 s wait. Delivery of a final reply is unknown from the logs (deliveryActionMs 131 and 192 only). No data was deleted or replayed. Related: duplicate check in create_event.
User impact
Any user who answers a confirmation twice or sends a quick follow-up while the bot is slow gets duplicated calendar entries and possibly no clear answer; slow hosts make the window wider.
Cost of inaction
Duplicate events keep appearing whenever the bot is slow, and users cannot tell whether their message was handled.
Acceptance criteria
- A regression test with two overlapping runs in one chat (synthetic data) creates each event once and fails before the fix — forced: no screenshot applies (non-visual). Evidence, PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 head 705cd9f: test/bot/pipeline/ai-agent-layer-overlap.test.ts 8 pass 0 fail incl. 'the second confirmation waits, is handled on the first run history and creates nothing' and 'a queued confirmation creates nothing the first run already created'; test/services/ai/tool-handlers/events-duplicate-create.test.ts 34 pass 0 fail; written red first; hosted PR CI test job green on this head (run 41760085024 stage-qa 6 cases passed). Not proof of production behaviour after deploy.
- A second message that arrives while a run is in flight waits for it or is rejected explicitly, and every run records its reply delivery outcome (sent, failed or unknown) or an honest failure notice attempt
- The 'Routing to AI agent' log line records the Telegram update_id and message_id so redelivery can be told from a second message — forced: no screenshot applies (non-visual). Evidence, PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 head 705cd9f: test/bot/pipeline/ai-agent-layer-overlap.test.ts 8 pass 0 fail incl. 'the routing log line carries the Telegram update id and message id next to the message text it already had'; test/bot/middleware/chat-logging.test.ts 3 pass 0 fail. Not proof of the deployed line; checked after deploy.
- Sibling note for the write-receipt rewrite (the files write-outcomes and write-receipt on the secretary-invites branch are not edited here): create_event now returns ToolResult.alreadyExists = { entity: 'event', id } with success: true and no created-event evidence; the receipt fold must read result.alreadyExists (not the output prose) and report 'already in the calendar' with that event id, not count it as a created write; the intent path already carries it via ToolAnswer.alreadyExists. A same-day conflict is success: false, mutationState: not_applied, error starting SAME_DAY_EVENT, naming the existing id
- Deferred, from review: per-chat queue depth is uncapped (one member flooding a group delays the chat, each queued message runs its own 4 s typing timer); next action: cap the queue and share one typing timer per chat
Screenshots
(none)
Deploy
Deployed: no
Links
- Relates to create_event saves exact and near-duplicate events: no check for the same title on the same day #520: create_event saves exact and near-duplicate events: no check for the same title on the same day #520
- Relates to Full-roster review of the integration diff of the incident fixes (PRs 941, 942, async pictures) is owed #961: Full-roster review of the integration diff of the incident fixes (PRs 941, 942, async pictures) is owed #961
- PR: fix(ai): the write receipt reads create_event's 'already exists' result as the same event, not a second creation or a failure (Refs #929, #520) #974
- Merge commit: 128846b
Skipped gates
- acceptance-criteria: adds a deferred-finding criterion, none removed
- acceptance-edit: unchecked criterion 2 dropped or reworded
- links: no external entity named
- post-merge-acceptance: Criteria 1 and 3 are checked with test evidence. Criterion 2 is only half delivered by PR fix(ai): serialize agent runs per chat; idempotent same-day-safe create_event #942 (a second message waits for the run ahead of it; the per-run reply delivery outcome is NOT recorded yet), 4 is a hand-over note for the write-receipt rewrite and 5 is a deferred finding (queue depth cap); none of the three is closed by this merge and none is claimed closed. After the deploy the ticket gets the live evidence (exact revision, health, logs), and task done will need real proofs, so the ticket stays open.
Activity
alex-mextner commentedon Nov 1, 2026
Fix in branch fix/chat-serialization-20261101 (not merged, not deployed):
alex-mextner commentedon Nov 2, 2026
Deferred from the single-seat review (Opus) of the staged work, each with a next action:
Single-seat review: only one reviewer seat was available (Sol quota exhausted).
alex-mextner commentedon Nov 2, 2026
Deferred from the final single-seat review of db0c087 (not fixed before commit; the pre-commit gate accepted the review run):
alex-mextner commentedon Nov 2, 2026
Status 2026-11-02 for #942 (not merged, not deployed): the follow-up for the HIGH finding (a queued «Да» read as the answer to a question asked after it was sent) is written and staged in the worktree but NOT committed or pushed. The pre-commit review gate needs a review of the final staged diff; the only reviewer seat (Opus: single-seat review, Sol quota out) is refused by the account 5-hour usage cap (92% against the 80% limit) after about 80 minutes of retries. The PR head is still db0c087, whose hosted checks
testandtype-bansare red; the follow-up fixes both.Follow-up content: created_at stays the time the user sent the message and placement is turn_order only; a placed turn is shown to the model as [queued: ...]; the day-reference guard ignores rows saved after the message; the stale-snapshot refresh also runs when placement found newer rows; voice transcripts reserve their place in the chat line; a repeated series create is matched; migration 069 doc, migration-order tests, tool catalog budget, handler fixtures, no
as never, KPI gate re-measured on the new engine digest (server-run; whole-repo tsc passed there).New tickets: #952 (mixed create receipts) and #953 (voice transcript sent-at). Queue-depth cap and per-run delivery outcome stay open here.
fix(ai): a message that waited in its chat's line is its own turn, no…
fix(intent): re-measure the KPI gate after formatting intent-executor…
fix(ai): serialize agent runs per chat and make create_event idempote…
fix(ai): a message that waited in its chat's line is its own turn, no…
fix(intent): re-measure the KPI gate after formatting intent-executor…
alex-mextner commentedon Nov 3, 2026
Status 2026-11-03T22:30Z (Ship942): PR #942 (head 1abbe94) is NOT merged and NOT deployed; production is still ff93220. All hosted checks incl. stage-qa (run 41696776659, passed cases=6) are green on that head.
gh ship 942 --dry-runrefuses only because pyproject.toml/uv.lock carry no version bump. The bump 0.3.0 -> 0.3.1 (#942 first; #941 would then need 0.3.2) is staged, uncommitted, in .worktrees/chat-serialization-20261101.Why it is not committed: the git pre-commit review-gate has no applicable provision (the trivial-delta stamp review-stamp-diff is keyed to HEAD 17f022d, a pre-rebase commit; current HEAD is 1abbe94), and
review diff --staged --pool 1 --task '#929'was refused 4 times (21:46Z, 21:46Z, 22:06Z, 22:26Z), nothing dispatched: the Anthropic accounts have no complete 5h+7d usage pair (review-cli account cap), Sol is out until 2026-11-09. Blocker: review-cli #786. Nothing here is verified post-merge (deploy, odroid health, logs): acceptance stays open.Next: when
review capacity-check --jsonshows Opus usable, runreview diff --staged --task '#929'in that worktree, commit, push, wait for Stage build/QA + hosted checks,gh ship 942, then verify the exact deployed revision on the odroid and tell session 84833 the merged SHA (it rebases PR #940).alex-mextner commentedon Nov 3, 2026
Review-gate probe, one call, no retries. Started 2026-11-03T23:31:05+01:00, finished 23:31:12+01:00. Command:
review diff --staged --pool 1 --task '#929'in .worktrees/chat-serialization-20261101 (staged: the pyproject.toml and uv.lock 0.3.0 to 0.3.1 bump). Exit 0, nothing dispatched. Output, verbatim:Result: the gate still refuses, same cause as before (the 5-hour plus 7-day usage pair is missing for the Anthropic accounts; review-cli fix is in review-cli issue 786). Not committed, not pushed, not shipped. PR 942 stays at head 1abbe94 without the version bump; production is unchanged by this work. The roster-review debt is tracked in #961.
chore(release): bump the version to 0.3.1 for the per-chat serializat…
fix(ai): serialize agent runs per chat; idempotent same-day-safe crea…
alex-mextner commentedon Nov 4, 2026
Shipped: merged via #942.
Merge commit: d0bc076
This does NOT close the ticket — acceptance still needs proof. Run
task read #929for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 4, 2026
Shipped and deployed: PR #942 merged to main as
d0bc0769d20897b5441b1e3f568611b8bdef4deb(version 0.3.1), deployed by the hosted CI/CD run 41762104573 (test, build, deploy all success).Review disclosure (once for this ticket): single-seat review via omp Opus (
review diff --staged --task 929 -m omp:anthropic/claude-opus-5-5, three role invocations, account mextner). The version-bump commit got no findings. The default Opus+Sol roster refuses until Sol returns 2026-11-09T10:20Z; the full-roster review stays owed in #961.Deploy verification of the exact revision (odroid, 2026-11-04 03:32 UTC, container started 03:04:12 UTC, 28 minutes of logs):
/opt/hypercal/releases/current.json: revisiond0bc0769d20897b5441b1e3f568611b8bdef4deb, image_idsha256:525ec763..., verified 03:05:37 UTC.docker inspect hypercal-bot: Imagesha256:525ec763...equals the loaded image id for the tag...:d0bc0769d208..., label revisiond0bc0769d208..., restarts 0, OOMKilled false, health healthy./healthok,/readyok (unverified)(the same value the previous release's receipt recorded).069_chat_history_turn_orderapplied once at start; no migration error.What is NOT live-tested: the overlapping-confirmation path itself. No real user's calendar was written, and the stage QA cases never change the calendar. The proof for it is the regression tests (checked on criteria 1 and 3 above) and the green stage-qa run
41760085024(6 cases) on the exact head. NoAgent run started after waiting for the previous run in this chatline has appeared in production yet (no overlapping traffic in this window). Criteria 2 (reply delivery outcome), 4 (hand-over note) and 5 (queue depth cap) stay open; this ticket is not done.alex-mextner commentedon Nov 4, 2026
Follow-up recorded after the merge of PR 940 (merge 4b2372d, on top of the per-chat serialization change d0bc076): the structured duplicate-create outcome ('Event already exists') still needs a fold into the readable write receipt, in src/services/ai/write-outcomes.ts and src/services/ai/write-receipt.ts. The receipt introduced by PR 940 does not know that outcome yet; until it does, a duplicate create would be shown as an ordinary refusal. Not implemented here because the outcome shape only became available on main with d0bc076.
chore(release): bump the version to 0.3.7 for the receipt change (Refs …
fix(ai): the write receipt reads create_event's 'already exists' resu…
alex-mextner commentedon Nov 7, 2026
Shipped: merged via #974.
Merge commit: 128846b
This does NOT close the ticket — acceptance still needs proof. Run
task read #929for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 7, 2026
Shipped the receipt side of the idempotent create (the hand-over item of this ticket): PR #974, merge commit 128846b.
What changed: the write receipt reads
create_event'sToolResult.alreadyExists = { entity: 'event', id }(never the output prose) as the same event. A creation and its repeat are one created line, no "Not created", no guard when nothing failed; a repeat of an event the run did not create is told once as "Already in the calendar" / "Уже есть в календаре". A refused try followed by the repeat counts as created; an earlier try of unknown result keeps its duplicate warning; a different event, a deliberateallow_duplicatecopy and group-chat target hiding are unchanged. The receipt-only confirmation after a validator outage follows the same rule. Version 0.3.7.Evidence: new tests drive the real handler into the real ledger in ru and en (
write-receipt-existing-event.test.ts, 20 tests, red first against main) plusreceipt-confirmation.test.ts. Hosted CI on the exact head 24c73cb passed (test, lint, type-bans, leftover-grep, review-threads, gitleaks, CodeQL, dependency-review, stage-qa 6 cases). The build server's own run had 6 failures that are environment (stage-QA Python installer suites: no system git and a nearly full disk, #955; Chromium libs), so the hosted run is the validator, not the server.Deployed: the production container runs the image tagged with the merge SHA;
/healthok,/readyok (unverified),releases/current.jsonrevision equals the merge SHA, no level 50/60 and no level 40 log records since the container start (about a minute of uptime when checked, not a live conversation test).Still open in this ticket: a second message waiting for the run ahead of it is shipped (#942) but the per-run reply delivery outcome is not recorded yet; the queue depth cap; live acceptance of the overlap case after deploy. Not closing the ticket.