Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12,740 changes: 12,740 additions & 0 deletions docs/intents/benchmarks/20261102T053443Z-real-current.json

Large diffs are not rendered by default.

12,195 changes: 12,195 additions & 0 deletions docs/intents/benchmarks/20261102T053634Z-synthetic-current.json

Large diffs are not rendered by default.

12,740 changes: 12,740 additions & 0 deletions docs/intents/benchmarks/20261102T094538Z-real-current.json

Large diffs are not rendered by default.

12,195 changes: 12,195 additions & 0 deletions docs/intents/benchmarks/20261102T094731Z-synthetic-current.json

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions docs/intents/kpi-gate.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
"version": "1",
"checksum": "98447f88cd71fa57404321b1da6fad4df63a2784d8ace86de0aa3eb84d1f64f8"
},
"benchmarkResult": "docs/intents/benchmarks/20261029T120052Z-synthetic-current.json"
"benchmarkResult": "docs/intents/benchmarks/20261102T094731Z-synthetic-current.json"
},
"informational": {
"intervals95": {
Expand Down Expand Up @@ -100,7 +100,7 @@
"labelsChecksum": "19ee899c60362252cd89a08b71fc5621f417f2ece970dbf58222792ffff05f7e",
"splitChecksum": "310707bb8c752d46692ceb9b50cc86bda217b681ac54867263ca3e8dda8354f0"
},
"engineDigest": "de45d59076375ddabf818004177c1d8834e9a5b3da1bffed8a38f9e451b147b2",
"benchmarkResult": "docs/intents/benchmarks/20261029T115902Z-real-current.json",
"engineDigest": "6935a57b68b65dd6f414637a134039f280b5c1ff1b2344d609fa21a6ecd1112b",
"benchmarkResult": "docs/intents/benchmarks/20261102T094538Z-real-current.json",
"autoActivate": true
}
19 changes: 19 additions & 0 deletions docs/reference/deploy-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,25 @@ Known limits: the probe runs on the PC, so a PC that is off or offline raises no

Also known: (1) an outage seen by the PC probe and, separately, by a queued alert (the DO watchdog's POST, a crash alert) can start two sessions 30 minutes apart: dedupe is per alert id and per probe incident, not across sources; the second session's brief tells it to check `omp-steer list` first. (2) The unit starts at login or boot (linger); if the keyring is still locked, `keychain exec` fails with its own exit code, which `RestartPreventExitStatus=78` does not cover: the unit restarts every 30 s without being able to tell you. After a reboot check `systemctl --user is-active hcb-alert-watcher`; `keychain unlock` fixes it. (3) Bun loads the script once: after a merge that changes `scripts/hcb-alert-watcher.ts` run `systemctl --user restart hcb-alert-watcher` (the installer restarts only when the unit text changes). (4) A pop whose response times out before any header arrives may have consumed an alert on the server; only the journal line "queue step failed" shows it. (5) The queue poll refuses redirects so the token never travels to another host.

## Agent runs per chat and repeated creates (#929, #520)

GramIO hands every Telegram update to its handler without waiting for the previous one (`EventQueue.process` in `@gramio/composer` 0.3.4, webhook and long polling alike), so two messages of one chat used to run two agents at once; on 2026-11-01 two quick «Да» created every event twice. Now `CalendarBotAgent.run` goes through `ChatRunGate` (`src/services/ai/chat-run-gate.ts`): one run at a time per chat id, first come first served, other chats unaffected. This covers every caller (message pipeline, `/cal`, button continuations, voice, scheduled AI).

- The place in a chat's line is reserved when the message is saved, in the same synchronous step (`chat-logging` middleware → `CalendarBotAgent.reserveChatTurn` → `ChatRunGate.reserve`), not when its handler reaches the agent: a handler that waits on a scene or retry lookup first cannot be overtaken by a message saved after it. The update's handling runs inside that reservation (`withChatTurn`, an async context, `src/services/ai/chat-turn-context.ts`), so the agent run it starts (text pipeline, `/cal`, a button) takes the place over with no id passed along; a run started elsewhere (scheduled, retry) finds none and joins the end without a message of its own, whatever row id it borrows for the action log; the reservation is released when the update's handling ends (an intent answered it, a command, an error) or after 60 s unused (`Not waiting any longer` / `A reserved place in the chat was never used` in the log).
- A queued run starts after the earlier one has written its turn. It re-reads the user and the ±2 week schedule and reads its own message after the earlier run's rows (`ChatHistoryRepository.placeUserTurnAfterLatest` sets `chat_history.turn_order`; the row keeps its id, so the action log, the per-user `chatHistoryIds` map and built contexts stay valid). Only a run that took over a reserved place has an own message to place (and only after a bot row of the same conversation, private or group); scheduled/synthetic runs borrow the user's latest row id for the action log and never move it. It is not shown turns queued behind it, in its history or through `get_history` (`ctx.deferredTurnRowIds`), so its model reads «request, what was done, this message». The request budget (90 s) counts from the start of the run, not from arrival.
- A turn that waited is moved by reading position only, never in time: its `chat_history.created_at` stays the time the user sent it, `turn_order` records the move, and the history readers order a row by when it reads (the time of the newest row it was placed behind), then `turn_order`, then id. The model is shown such a turn with `[queued: …]` after its timestamp (`QUEUED_TURN_MARKER`, explained in the system prompt) and the day-reference guard ignores every row saved after the message when it looks for the `ask_user` question the message answers, so a message sent while an earlier run was still working is never read as the answer to a question that run asked afterwards (a queued «Да» could otherwise confirm a duplicate, a forced create or a delete nobody offered it). A button answer, which explicitly belongs to the question it was tapped under, still binds to that question. A voice message takes its place in the line when its transcript row is saved (`handleVoiceMessage`), like a text message.
- Known limits of the queued-turn rule (#929, #953, #957): (1) a button tap that waited behind a run started by another user message, or a reply that reached the chat before the asking run wrote its turn, is also read as queued and the guard loses the question's day; the model is then told to ask what it refers to, it is never executed on a guess. The guard infers the question from row order, it is not carried on the button. (2) The refusal to treat a queued «Да» as consent for `allow_duplicate`, `force` or a delete confirmation is the system prompt plus the guard: no tool handler checks it (a delete still needs the bot's own button tap). (3) A voice transcript can be bound to a question saved while it was being transcribed (#953).
- A run that never finishes holds the chat for at most its budget plus 30 s (`CHAT_RUN_GRACE_MS`); then the next run goes on and the log says `Not waiting any longer for the earlier agent run in this chat`. A queued run is in `CalendarBotAgent.inFlight`, so the shutdown drain waits for it; at shutdown the wait ends at once and the run takes its normal fail-fast path (notice and stored retry), so no queued message is dropped.
- Log lines to look for: `Agent run started after waiting for the previous run in this chat` (`waitedMs`, `queuedBehind`) and `Routing to AI agent` with `updateId`/`messageId`: the same ids twice mean a Telegram redelivery, new ids a second message.
- While a message waits the chat shows the typing indicator (refreshed every 4 s). If the run ahead outlives its lease (`CHAT_RUN_GRACE_MS` after the 90 s budget, so about 120 s of waiting for the run directly ahead; the lease bounds the wait for that one run only, so the Nth message in a long queue of slow runs waits for every run ahead of it in turn, up to about (N-1) leases), the waiting message goes on beside it and the user is told once per hung run (`agent_queue_lease_expired`, ru and en): the first later run that has a message of its own to show it announces that the earlier run was given up on, and a run without one (scheduled, or a supplement) never uses the notice up for the next. `Agent run started` and `Agent run complete` carry `updateId`/`messageId` too.
- Every kind of run uses the gate, keyed by its chat id: supplement runs (silent, optional text: no typing, no notice, so a supplement can sit behind a main run for up to the same ~120 s and then speaks to nobody) and scheduled/system runs (their chat is the user's private chat, so a scheduled reminder run waits behind an active conversation and vice versa). The worst case for any run is that wait plus its own 90 s budget.
- `create_event` (`EventRepository.createUnlessDuplicate`, one immediate SQLite transaction, so it holds across connections and restarts) compares with the live events of the same calendar (a one-off only with one-offs, a series only with series of the same recurrence rule; the owner's own events with no Google calendar, the `primary` alias or the id registered as the user's primary calendar, a secretary's `owner_id`, or the same group; events imported from a secondary Google calendar never count) that have the same title (trimmed, whitespace collapsed, case folded) and all-day flag. A series is matched only as an exact repeat within the window (`duplicate`), never by `same_day`:
- `duplicate`: same start and end (all-day: the same floating dates, as `allDayDates` reads them), the same description, location and reminders the caller stated, created within 10 minutes (`RECENT_DUPLICATE_WINDOW_MS`). It comes back as `Event already exists …`: success, `alreadyExists` on the tool result, no second row, reminders, notifications or calendar push.
- `same_day`: any other candidate on the same local day, whatever its age (the «move by creating» case of #520): `SAME_DAY_EVENT` error, `mutationState: not_applied`, pointing at `update_event` with the existing id.
- The model makes a deliberate second copy with `allow_duplicate: true`.
- The event row and its reminders are written in the same transaction (also for `allow_duplicate` and every other `EventService.createEvent` caller); a failed reminder write leaves no event, so a retry starts clean. All-day dates are read in the zone each event was saved with.
- The result of a repeat is `success: true`, `mutationState: not_applied`, `alreadyExists`. Intent answers: it is carried through `intent-executor` into `ToolAnswer`, and both the renderer and a closing `respond:` step (`basis.event.create`, `basis.event.create_invite`) say the event is already in the calendar instead of «created». The run-failure receipt wording («Completed: Create event») lives in `write-outcomes.ts` and is left to the receipt rewrite tracked in #929.

## Public ingress (Cloudflare tunnel)

`hypercal.mextner.com` is a proxied CNAME in the `mextner.com` Cloudflare zone (account `invntrm@gmail.com`) to the tunnel `odroid-home` (`872e1f2e-3488-4e80-955e-8aba2fada806`). cloudflared on the odroid (`cloudflared.service`, `--token-file /etc/cloudflared/token`) runs it with a remotely managed configuration, so the hostname rules are edited in Cloudflare, not on the odroid. Bot traffic is the rule `hypercal.mextner.com` -> `http://localhost:3001`; every path reaches the bot, which serves only its own routes (`/health`, `/ready`, `/webhook/telegram`, `/webhooks/google-calendar`, `/oauth/google/callback`, `/admin/alerts*` behind the bearer token). While the container is being replaced the tunnel answers 502; Telegram and Google retry, but there is no proxy holding requests as the retired DO Caddy did (`lb_try_duration`).
Expand Down
66 changes: 66 additions & 0 deletions docs/reference/migrations/069_chat_history_turn_order.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
migration: 069_chat_history_turn_order
rollback-compatible: yes
data-deletion: no
---

# Migration 069: where a queued user turn is read in the conversation

`src/database/migrations.ts`, migration `069_chat_history_turn_order`. Refs #929 (per-chat run
serialization) and #520.

## What it does

```sql
ALTER TABLE chat_history ADD COLUMN turn_order REAL;
```

One nullable column. Two messages of one chat used to run two agents on overlapping history; a run now
waits for the run ahead of it in its chat (`src/services/ai/chat-run-gate.ts`). The message that waited was
saved before everything the earlier run wrote, so by row order it sits in the middle of that run's output.
When its run starts, `ChatHistoryRepository.placeUserTurnAfterLatest` sets `turn_order` to the newest row's id
plus 0.5 and the history readers order a row by the time of the row it was placed behind and then by
`COALESCE(turn_order, id)`. The row keeps its id (the action log and the per-user row map hold ids) and its
`created_at` (the time the user sent it). A non-NULL `turn_order` also tells the reader that the turn was
sent before the rows it now follows, so the model is shown it as queued and never as the answer to a question
asked after it was sent.

NULL, which every row written before this migration has, means the row reads at its own id and time exactly
as before. No existing column or row is read, changed or deleted.

## Order in the file

Appended after `067_ai_usage_role_preset_cost`, the last entry. Nothing is moved or renumbered. The number
069 leaves 068 to the other open changes that claim it (the AI speed metrics and the image deliveries).

## Why it is safe to deploy automatically

- Additive DDL only: `ALTER TABLE ... ADD COLUMN` of one nullable column; existing rows read back NULL.
- The image before this migration names the columns it reads, or selects `*` and ignores the extra one, and
inserts with an explicit column list, so it keeps working on the migrated database. It orders by
`created_at, id`: for a queued turn that is the order the messages were sent in, which is only a different
reading position, never a lost row.

## Rollback

An image rollback needs no schema step: the previous image ignores `turn_order` and applied migrations it does
not know. Keep the column and the `migrations` row.

Dropping the column is never required. If it ever is, it deletes only the reading positions of queued turns:

```sql
BEGIN IMMEDIATE;
ALTER TABLE chat_history DROP COLUMN turn_order;
DELETE FROM migrations WHERE name = '069_chat_history_turn_order';
COMMIT;
```

## Verifying after deploy

```sql
SELECT name FROM migrations WHERE name = '069_chat_history_turn_order';
SELECT count(*) FROM chat_history WHERE turn_order IS NOT NULL;
```

Expect one migration row. The count stays 0 until two messages of one chat arrive while an earlier agent run
of that chat is still working.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "hypercalendarbot"
version = "0.3.0"
version = "0.3.1"
requires-python = ">=3.12"
dependencies = [
"pyrofork>=2.3",
Expand Down
11 changes: 10 additions & 1 deletion src/bot/handlers/message.handler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,16 @@ export function createMessageHandler(deps: MessageHandlerDeps) {
);

const run = () =>
runPipeline(ctx, messageText, layers, groupContext, incomingMsgId, admission.wasExplicitInvocation);
runPipeline(
ctx,
messageText,
layers,
groupContext,
incomingMsgId,
admission.wasExplicitInvocation,
undefined,
ctx.chatHistoryId,
);
if (admission.isGroupSessionMessage && deps.botToken) {
await runPipelineWithTyping(run, chatId, deps.botToken);
} else {
Expand Down
10 changes: 3 additions & 7 deletions src/bot/handlers/message/agent-context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,11 @@
// Builds the per-request AgentContext (sender, repositories, chat-history ids) for the message handler.

import type { EventOccurrence, User } from '../../../database/types.ts';
import { scheduleWindowAround } from '../../../services/ai/agent/queued-turn.ts';
import type { AgentContext } from '../../../services/ai/types.ts';
import { cmdLogger } from '../../../utils/logger.ts';
import type { MessageHandlerDeps } from './types.ts';

const SCHEDULE_CONTEXT_WINDOW_MS = 14 * 24 * 60 * 60 * 1000;

interface AgentGroupInfo {
isGroup: boolean;
groupChatId?: number;
Expand All @@ -29,13 +28,10 @@ function describeSecretaryAccess(deps: MessageHandlerDeps, user: User): string |
.join(', ');
}

/** Events two weeks either side of now, so the model knows the user's nearby schedule; empty on a read failure. */
/** The nearby schedule the model is shown; empty on a read failure. */
function recentEventsWindow(deps: MessageHandlerDeps, user: User): EventOccurrence[] {
try {
const now = new Date();
const start = new Date(now.getTime() - SCHEDULE_CONTEXT_WINDOW_MS).toISOString();
const end = new Date(now.getTime() + SCHEDULE_CONTEXT_WINDOW_MS).toISOString();
return deps.eventService.getEventsInRange(user.telegram_id, start, end);
return scheduleWindowAround(deps.eventService, user.telegram_id);
} catch (err) {
cmdLogger.error({ err, userId: user.telegram_id }, 'Failed to build schedule context');
return [];
Expand Down
28 changes: 18 additions & 10 deletions src/bot/handlers/message/voice-message.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { mkdir } from 'node:fs/promises';
import { InlineKeyboard } from 'gramio';
import { t } from '../../../config/constants.ts';
import type { User } from '../../../database/types.ts';
import { withChatTurn } from '../../../services/ai/chat-turn-context.ts';
import type { AgentContext } from '../../../services/ai/types.ts';
import {
fixDateOrdinals,
Expand Down Expand Up @@ -150,16 +151,23 @@ export async function handleVoiceMessage(

const logChatId = Number(chatId) !== user.telegram_id ? Number(chatId) : undefined;
const chatHistoryId = deps.conversationLogger.logUserMessage(user.telegram_id, transcription, logChatId);

const agentContext: AgentContext = {
...buildAgentContextFactory(deps)(user, Number(chatId), transcription),
inputMode: 'voice_message',
incomingMessageId: ctx.id,
chatHistoryId,
voiceFileId: voice.file_id,
};

const { responseText } = await deps.agent.run(agentContext);
// The transcript is saved here, not by the logging middleware (which saw no text): its place in the chat's
// line is reserved in the same step, so a voice message sent behind a running turn is answered after it.
const reservation = deps.agent.reserveChatTurn(Number(chatId), chatHistoryId);

let responseText: string | undefined;
try {
const agentContext: AgentContext = {
...buildAgentContextFactory(deps)(user, Number(chatId), transcription),
inputMode: 'voice_message',
incomingMessageId: ctx.id,
chatHistoryId,
voiceFileId: voice.file_id,
};
({ responseText } = await withChatTurn(reservation, () => deps.agent.run(agentContext)));
} finally {
reservation.release();
}

// Send voice reply if TTS is available and user has opted in
if (responseText && user.voice_response_enabled === 1 && deps.sendVoice) {
Expand Down
1 change: 1 addition & 0 deletions src/bot/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,7 @@ export function createBot(token: string, db: DatabaseService, aiConfig: AgentCon
conversationLogger,
actionLog: db.actionLog,
chatHistoryIds,
reserveChatTurn: (chatId, rowId) => agent.reserveChatTurn(chatId, rowId),
isConnectWizardInput: connectWizardGuard.isConnectWizardInput,
}),
)
Expand Down
Loading
Loading