Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ Each layer returns `{ handled: true }` to stop propagation, or `{ handled: false

- `agent.ts` — `CalendarBotAgent`: Anthropic SDK streaming, max 15 rounds, 90s timeout, 2 retries.
- `agent/` — what `agent.ts` composes: `round-loop` (model round → answer or tool calls), `tool-round` (one round's tool calls, dedup, ledger), `response-validation` + `validation-retry` (final-prose check and its one corrective retry), `turn-finish` (evidence guard, history writes, delivery), `measured-stream` (request deadline, late-producer guard, metrics), `failure-notices` / `failure-announcement` / `retry-store` (what the user is told after a failed run and the bounded retry store), `message-sanitizer` / `history-rows` / `conversation` (history repair and parsing).
- Validator outage: the response validator runs on its own `VALIDATION_TIMEOUT_MS` clock once started, and returns `approved` / `rejected` / `unavailable`: an outage is never a rejection. After calendar writes an unavailable verdict runs no retry-with-tools pass and the reply is `buildReceiptConfirmation` (`agent/receipt-confirmation.ts`) (exact titles, local dates and times, no model prose). A run that ends error / unverified / limit / with an "interrupted" notice pushes one rate-limited alert (`agent/run-incident.ts`, source `agent-run-incident`) to the `alerts` queue the auto-fix watcher reads; it carries request id, counts and outcomes, never user text, titles or ids.
- Timeouts: every deadline the bot arms (`AbortSignal.timeout`, `setTimeout`, `Bun.sleep` poll loops, `Promise.race`, BullMQ `waitUntilFinished`, a client `timeout` option, `busy_timeout`) reports through `reportTimeout()` (`src/services/incident/timeout-reporter.ts`; `recovered: true` when a fallback absorbed it) or, for a locked SQLite database, `reportIfDatabaseBusy()` (`src/services/incident/database-busy.ts`). `test/services/incident/timeout-sites.test.ts` fails per callable, not per file, when a new deadline neither reports nor sits on its exemption list with a reason. Alerts are one per kind per 10 minutes and carry kinds, durations and counts only.
- `streaming.ts` — `aiStreamRound`, the public import path of the provider chain; its pieces (types, errors, stream reader, slots, chains, failure handling, round budget) live in `streaming/`.
- `tools.ts` — 40+ tool definitions (the Claude API schema).
- `tool-executor.ts` — dispatches tool name → handler function.
Expand Down
20 changes: 20 additions & 0 deletions docs/intents/response-quality.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,23 @@ friendly invention.

The reference implementation of the empty-calendar wording is
`src/services/ai/empty-agenda.ts`, called from the `get_events` tool handler.

## Receipt-only confirmation after a validator outage

When the response validator cannot reach a verdict (timeout, provider error, spent request budget)
and the run already changed the calendar, the bot does not send the model's prose and does not
retry with tools. It sends a confirmation built only from the run's own write results
(`buildReceiptConfirmation`, `src/services/ai/agent/receipt-confirmation.ts`). Required wording:

- Russian, direct, one line per change: the verb of what happened (Создано / Изменено / Удалено),
the exact title in «», and the local weekday, day and clock time (or "весь день") taken from the
write result in the user's timezone, e.g. `Создано: «Synthetic standup» — вторник, 6 марта, 09:00`.
- Honest scope: it says these facts come from the calendar itself and that the written answer could
not be checked, so it is not sent, and that repeating the request is unnecessary. It never says
"Запрос прерван" or "Часть действий могла уже выполниться" when every write succeeded; an
outage is not a failure of the run.
- Nothing the model invented: no free windows, no counts, no advice. A write that failed or whose
outcome is unknown keeps the guard notice with the failure named instead of this confirmation.

Bad: "Выполнено: Создание события / Выполнено: Создание события / Запрос прерван…" (two lines that
cannot be told apart, and a claim that the request was cut short after every tool succeeded).
12 changes: 12 additions & 0 deletions docs/reference/deploy-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ What runs: the systemd **user** service `hcb-alert-watcher` on the owner's Ubunt

The brief the session gets (`~/.local/state/hcb-alert-watcher/briefs/`) fences the alert text as untrusted data, makes the first phase read-only, forbids restart/stop/rollback/redeploy, database writes, deletions and merging without the owner (#596), and requires a Russian `tg report` with the root cause first. The watcher itself tells the owner with `tg --tag problem` when it is blind: the launcher fails (alerts stay pending), or the queue answers 401/403/404 three times in a row (token rotated or endpoint disabled); each notice is sent once per streak. Every helper process it spawns (launcher, tmux, ctmux, tg) has a 30 s deadline and runs without the alert token; the tmux server and window are started in their own systemd scope so restarting the unit never kills the owner's tmux.

Agent-run incidents (`source = agent-run-incident`, since 2026-11-01): a user-facing agent run that ends badly returns no 5xx and leaves `/health` and `/ready` green, so neither the PC probe nor the DO watchdog sees it. On 2026-11-01 at about 17:06 UTC a run ended `unverified` after a validator timeout, the user got near-useless notices and nothing woke an investigation. The bot now pushes one row to the `alerts` table (`db.alerts.push`, the table this watcher pops) when a user-facing run ends in one of four classes: `error` (the run threw), `unverified` (the answer could not be checked), `limit` (the rounds or the request budget ran out) or `interrupted` (the user was shown the "request interrupted" notice). Supplement turns and silent runs raise nothing. The bot rate-limits itself to one alert per class per 10 minutes (in memory, so a restart resets it); incidents held back are counted and the next alert of that class carries `suppressedSince: <n> earlier incident(s) of this class were not alerted in the last 10 minutes`. On the PC these alerts travel like every other: spooled, one session per 30 minutes for everything pending, so an incident that arrives inside the window waits and shares the next session. The text is plain lines with identifiers and counts only, never user text, event titles, chat ids or user ids: `Agent run incident (<class>)`, `requestId`, `termination`, `elapsedMs`, `tools` (`<tool> ok=<n> fail=<n>` per tool, or `none`), `writeSucceeded: yes|no`, `provider: modelCalls=<n> attempts=<n> fallbacks=<n>`, `validator: not_run|approved|rejected|unavailable(timeout|provider_error|aborted|budget_expired)`, `delivery: delivered|fallback|discarded`, an optional `error: <error class name>`, the optional `suppressedSince` and a last line saying how to look the run up. The investigation finds the run by `requestId` in `docker logs hypercal-bot` and, while `AI_DEBUG_LOGS=true`, in the per-chat debug log: `docker exec hypercal-bot grep -rl 'REQUEST ID: <requestId>' /app/logs/chats` names the file (one `REQUEST ID:` line per run). A run with debug logging off, or older than that line, is found through the `chatId` on its start line in the container log and read from `/app/logs/chats/<chatId>/` by time. For this source the brief adds a checklist (the alert's `source` switches it on, never its text; the class on the first line only picks an extra block): every requestId (one chat can have two overlapping runs), the write receipts against the `events` rows including duplicates (the same title and start created twice), what the user actually saw, host load (`/proc/pressure`, swap) and the `image-render` queue, and the validator and provider timings in the `AI model call metric` lines. It reports only: it never replays a write, resends a message, or edits or deletes a row (unknown completion is not failure, AGENTS.md), and it reuses an existing ticket (`task find`) before opening one.

Install (idempotent; the token lives only in the keychain as `hcb-alert-watcher/admin-alert-token` and reaches the service through `keychain exec --caller hcb-alert-watcher`):

```bash
Expand All @@ -125,6 +127,16 @@ Known limits: the probe runs on the PC, so a PC that is off or offline raises no

Also known: (1) an outage seen by the PC probe and, separately, by a queued alert (the DO watchdog's POST, a crash alert) can start two sessions 30 minutes apart: dedupe is per alert id and per probe incident, not across sources; the second session's brief tells it to check `omp-steer list` first. (2) The unit starts at login or boot (linger); if the keyring is still locked, `keychain exec` fails with its own exit code, which `RestartPreventExitStatus=78` does not cover: the unit restarts every 30 s without being able to tell you. After a reboot check `systemctl --user is-active hcb-alert-watcher`; `keychain unlock` fixes it. (3) Bun loads the script once: after a merge that changes `scripts/hcb-alert-watcher.ts` run `systemctl --user restart hcb-alert-watcher` (the installer restarts only when the unit text changes). (4) A pop whose response times out before any header arrives may have consumed an alert on the server; only the journal line "queue step failed" shows it. (5) The queue poll refuses redirects so the token never travels to another host.

Probe failures under host load (2026-11-01): `state.json` showed one failed probe at 17:30:38 UTC (`lastProbeFailureAt`, the consecutive count back to 0 afterwards) and the journal had no line for it, because a single failed probe was not logged. It was a timeout, not an outage: cloudflared on the odroid logged `Incoming request ended abruptly: context canceled` for `hypercal.mextner.com/ready` at 17:30:38, the instant the watcher's 20 s `HCB_ALERT_PROBE_TIMEOUT_SECONDS` deadline fired (a second `/ready` request from an unidentified client ended the same way at 17:30:34). The bot's own log for that window holds only a Google pull-sync job that needed 46 s to apply zero items (it also waits on Google, so that alone proves little). About 18:00 UTC the host still showed load average 11-12 on 6 cores, 3 GB of 6 GB swap in use on the microSD, `/proc/pressure/io` `full avg300` above 50%, and `/ready` answered from this PC in 13.6 s, 1.9 s and (once) not within 25 s: a swapping odroid answers late, not wrongly (memory pressure on the odroid: #800). Each failed probe before the incident threshold is now logged with its reason, `readiness probe failed (1/3): no answer within 20s (probe timeout)` against `no response (<error>)` or `HTTP <status>`, so a blip shows in `journalctl --user -u hcb-alert-watcher`. Three timeouts in a row still raise a `down-<epoch>` incident, on purpose: a bot that needs more than 20 s to say it is ready is not serving users either, and the investigation reads the host load. Raise `HCB_ALERT_PROBE_TIMEOUT_SECONDS` only to judge a degraded host differently, not to hide it. Timestamps: the PC journal and the odroid journal print local time (CET, UTC+1 since 2026-10-25), so give `journalctl --since '2026-11-01 17:25:00 UTC'` and `docker logs --since 2026-11-01T17:25:00Z` an explicit zone. The same limit makes known limit (4) likelier: a queue pop that times out at the same 20 s may have consumed an alert on the server (the pop is not acknowledged), so an incident raised while the host is starved can be the one that is lost; only the journal line `queue step failed` shows it.

### Validator outage after writes

The final answer of an agent run is checked by a response validator (a model call). When it cannot reach a verdict (`validator: unavailable(<cause>)`: timeout, provider error, aborted or budget expired) and the run already executed calendar writes, the bot sends a deterministic receipt-only confirmation built from the write receipts (exact titles, dates and times). It never runs the retry-with-tools pass, because that pass could replay writes the user already has, and it never says "request interrupted" when every tool succeeded. A validator outage is not evidence that the answer is wrong: the run ends `unverified`, one `agent-run-incident` alert is queued, and the investigation reads what the user actually received. The validator runs on its own 16 s clock; once the request budget is spent it launches nothing and only the local checks decide.

### Timeout alerts

Every timeout the bot hits goes through `reportTimeout()` (`src/services/incident/timeout-reporter.ts`) and reaches the alerts queue as source `timeout`: at most one alert per kind (`validator`, `provider`, `agent_request`, `http`, `render`, `model_listing`, `claude_credential`, `chat_run_lease` (a message stopped waiting for a hung run ahead of it in its chat, counted once per hung run), `chat_run_reservation` (a message's handler outlived its reserved place in the chat's line), `agent_drain`, `google_refresh_lock`, `db_busy`, ...) per 10 minutes, with the `unrecovered` and `recovered` counts since the last alert. A timeout a provider fallback absorbed is `recovered`: counted, never dropped, so a trend is visible. The alert holds kinds, durations and counts only, no message text, user ids or titles. The investigation treats `unrecovered` ones as incidents and a burst of `recovered` ones as a capacity signal (host load, swap). A locked SQLite database has no timer to hook: the central error handlers and the places that swallow a database error hand what they caught to `reportIfDatabaseBusy()` (`src/services/incident/database-busy.ts`), which reports a `SQLITE_BUSY` as `db_busy`. `test/services/incident/timeout-sites.test.ts` checks every callable in `src` (not every file) that arms a deadline (`AbortSignal.timeout`, `setTimeout`, `Bun.sleep` poll loops, `Promise.race`, BullMQ `waitUntilFinished`, a client `timeout` option, a `TimeoutError`, `busy_timeout`): it must call `reportTimeout()` / `reportIfDatabaseBusy()` itself or sit on that file's exemption list with a reason, and an exemption that is reported elsewhere must name a callable that really reports. A callable that arms two deadlines and reports once is the one case the check cannot tell apart.

## Agent runs per chat and repeated creates (#929, #520)

GramIO hands every Telegram update to its handler without waiting for the previous one (`EventQueue.process` in `@gramio/composer` 0.3.4, webhook and long polling alike), so two messages of one chat used to run two agents at once; on 2026-11-01 two quick «Да» created every event twice. Now `CalendarBotAgent.run` goes through `ChatRunGate` (`src/services/ai/chat-run-gate.ts`): one run at a time per chat id, first come first served, other chats unaffected. This covers every caller (message pipeline, `/cal`, button continuations, voice, scheduled AI).
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "hypercalendarbot"
version = "0.3.2"
version = "0.3.3"
requires-python = ">=3.12"
dependencies = [
"pyrofork>=2.3",
Expand Down
Loading
Loading