You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Route every bot timeout through one reportTimeout reporter and finish landing the run-outcome alert work #935
Follow-up of #931. The worktree .worktrees/run-outcome-alerts-20261101 holds staged, uncommitted work (the staged review queue was 34-47 min): validator unavailable verdict, receipt-only confirmation, agent-run-incident alerts, delivery outcome with Telegram message ids, watcher checklist. Left to do: (1) land it (review, commit, ship); (2) src/services/incident/timeout-reporter.ts exists with one call site (validator); inventory every other timeout site (provider stream timeouts, Google and HTTP calls, DB and Redis waits, scheduler jobs, BullMQ waits) and route each through reportTimeout, recovered true when a fallback absorbed it, plus a static guard test over src; extend the watcher brief for the timeout source; (3) the receipt code lives in write-outcomes.ts, which another branch rewrites: move it to a new module; (4) the run-incident debug-log request-id test passes locally but fails on the server-run host (reads an empty log): find why.
Why (motivation)
Owner decision 2026-11-01: every timeout must reach the auto-fix flow, not only failed user runs.
User impact
Until then slow-host timeouts are visible only in docker logs, and the owner learns of them from users.
Cost of inaction
Timeouts that do not fail a user run stay invisible to the repair flow.
Acceptance criteria
The staged work is committed and landed through gh ship with Refs to issue 931
Every timeout site reports through reportTimeout and a static guard test fails on a new unreported timeout site — proof:
The run-incident debug-log test passes both locally and on the server — proof:
links: branch/path names in the text are false positives; the issue link is a markdown link
post-merge-acceptance: Criterion 1 is about landing through gh ship, which this merge is. Criterion 2 is covered by test/services/incident/timeout-sites.test.ts (10 pass on the server for this head) but is not ticked here. Criterion 3 (run-incident debug-log test locally and on the server) is not proven by me. None of the three is closed by this merge; the ticket stays open, task done needs real proofs after the deploy.
Review-gate classification (recorded once, 2026-11-02 03:3x CET): the default review diff --staged is refused host-wide. The codex seat (Sol) is quota-exhausted until 2026-11-08 02:05Z (exit 10), and Claude/Opus is refused by the 80% five-hour account cap until 05:09 CET. The owner-authorized reduced route is a single Opus seat after the cap resets: review diff --staged --task 935 --pool 1. A single-seat pass is the best available review here; the orchestrator adds a full-roster review of the integration diff later. No --no-verify, no config edits.
State of the branch fix/run-outcome-alerts-20261101 before that review: all touched test files pass locally one at a time; server-run tsc --noEmit and biome check . are clean. Changes since the previous report: the receipt builder moved out of write-outcomes.ts into agent/receipt-confirmation.ts (the sibling branch rewrites write-outcomes.ts); the timeout guard now checks each callable (test/helpers/timeout-site-scan.ts) instead of each file, which exposed and fixed four unreported timeouts: the models-listing probe, the agent shutdown drain, the Google token-refresh lock wait and swallowed SQLITE_BUSY in the provider-circuit sidecar; render-service now reports its queue-wait timeout too.
Commits on fix/run-outcome-alerts-20261101 (2026-11-02, single-seat Opus review, three passes each, recorded above): a417cea (reportTimeout reporter) and 529d1f4 (validator verdict, receipt, delivery outcome, incident alerts, watcher brief, per-callable timeout guard with 30 callables found / 16 routed / 14 exempt / 0 unaccounted). Not pushed, not shipped; a full-roster review of the integration diff is still owed.
Open findings from the reviews, none a data-integrity regression, each needs a follow-up change:
Receipt outro lacks the "no need to repeat" sentence that docs/intents/response-quality.md requires; one pass asked to drop it because a partly-done request might not be complete. Owner decision.
Receipt falls back to the ledger summary for the whole run when one write names no event (build named lines per write).
A normal run whose Telegram delivery failed (deliveryOutcome failed) raises no agent-run-incident; an interrupted supplement that wrote to the calendar raises none.
Recovered-only timeout alerts can start an investigation session; launch only when unrecovered > 0 or a burst threshold passes.
Logs, chat logs and database rows named in the incident checklist are not marked untrusted; make the duplicate query return ids only.
timeout alerts / run-incident alerts ignore the ADMIN_ALERT_TOKEN gate (rows pile up undrained without a token).
agent_request is reported for any error after the deadline, not only aborts; most timeout kinds carry no requestId; runbook delivery-line format and watcher fixture are stale.
An unavailable validator still triggers the corrective retry on a run without writes.
No test drives a real provider-chain timeout through reportProviderTimeouts, nor a delete_event / non-UTC receipt.
Fix prepared and staged in the run-outcome-alerts worktree, not committed yet: reportTimeout for claude-credential-pool (kind claude_credential, recovered when a later key saved the request), model-registry waitForCatalog and the probe's own budget (kind model_listing), two guard-test exemptions with reportedBy, docs kind list, regression tests (red before the change, green after). Server: biome and tsc green. The server full suite is not usable: its tmpfs is out of inodes (other agents' directories) and the account hit EDQUOT, so the full suite relies on hosted CI.
Blocker: the commit hook demands a review and review-cli dispatches nothing (omp usage has no 5-hour window for any Anthropic account, Sol out until 2026-11-09). tg#27037 sent to the owner.
Second blocker, 2026-11-03: review-cli admits no seat (no 5h usage window for any Anthropic account), so neither the #941 fix commit (3 new #928 deadline sites, staged) nor any version-bump commit can be made. The omp hatch only covers the bridge; the git review-gate needs a stamp. Owner informed (tg#27037, tg final 27082). Continue once review-cli #786 is installed.
Review-gate probe, one call, no retries. Started 2026-11-03T23:31:05+01:00, finished 23:31:12+01:00. Command: review diff --staged --pool 1 --task '#929' in .worktrees/chat-serialization-20261101 (staged: the pyproject.toml and uv.lock 0.3.0 to 0.3.1 bump). Exit 0, nothing dispatched. Output, verbatim:
[review-cli] warning: --pool 1 is below the default ship floor of 3 distinct board roles; this run cannot satisfy `review task CODE --check` / gh ship.
[review-cli] auto-selected level standard (Laya standard 0.45; tier_not_acted:standard/uncertain:floor_standard); pass --level to choose it yourself
[review-cli] level 'standard' is a fixed roster (Opus (claude:claude-opus-5-5)) and does not run without all of it: a blocked seat is never replaced by another or a cheaper model. Nothing was dispatched.
Blocked seats:
- Opus (claude:claude-opus-5-5): omp:anthropic/claude-opus-5-5: no omp anthropic account may be used (3): mextner@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); alex.mex.ux@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); invntrm@gmail.com: EXCLUDED (config.yaml omp_accounts.exclude); claude:claude-opus-5-5: usage неизвестен для аккаунта invntrm@gmail.com: нет подтверждённой пары 5ч/7д от omp usage; консервативно не используется (ограничение review-cli, config account_usage_cap_percent=80): a complete shared 5-hour and 7-day observation is missing
Providers missing (0 of 3 roles live: none):
- anthropic: Opus (claude:claude-opus-5-5): [same four account reasons as above]; back: unknown (no account usage window names a reset)
Wait until the capacity is back (or free the account), or choose deliberately: `--level LEVEL` for another level, `-m MODEL` to name the model yourself.
Result: the gate still refuses, same cause as before (the 5-hour plus 7-day usage pair is missing for the Anthropic accounts; review-cli fix is in review-cli issue 786). Not committed, not pushed, not shipped. PR 942 stays at head 1abbe94 without the version bump; production is unchanged by this work. The roster-review debt is tracked in #961.
PR #941 rebased onto main 4b2372d (0.3.3), head da6a6fd. Review: single-seat review via omp Opus (4 rounds, three role invocations each); full-roster review stays owed in #961. Fixed from the rounds: a key timeout lost when a later key failed without timing out; one hung catalogue listing counted twice; wrong limitMs; a lease counted once per waiting message (now once per hung run); the reservation expiry now reported (chat_run_reservation); runbook kind list. Left open (recorded, not fixed): (1) the lease report is recovered: true; a reviewer argues a hung run should be unrecovered, decision needed; (2) the new credential-pool tests use real 60 ms timers with little slack and can flake on a busy host; (3) the joiner timer and the provider deadline can both report a coinciding expiry in model-registry (rare); (4) a runbook line still quotes the old receipt wording; (5) no test that shutdown is not counted as a lease timeout.
Shipped: merged via #941.
Merge commit: e4185dd
This does NOT close the ticket — acceptance still needs proof. Run task read #935 for the outstanding criteria, then task check/task done.
PR #941 merged as e4185dd (0.3.3), deployed by the hosted CI/CD run and verified on the odroid at 2026-11-04 10:46 UTC: receipt revision and the container's image id (sha256:225e9c98...) match the merged SHA, restarts 0, not OOM-killed, healthy, /health ok, /ready ok (unverified), 0 lines of level >= 50 in the first minutes, memory 433 MiB, load 1.3. Single-seat review via omp Opus; full-roster review owed in #961. hcb-alert-watcher restarted on the canonical main checkout (it loads its script only at start). NOT done: the synthetic alert test through the real queue, and the #949 watcher fix PR; both remain open, as does this ticket (criteria 2 and 3 unproven by me).
Auto-fix chain proven end to end once on 2026-11-04 (status: implemented, merged and deployed as 0.3.3 / e4185dd; live-tested with ONE synthetic alert; the #949 brief fix is not in main yet).
At 10:51:41Z one alert, clearly labelled SYNTHETIC PIPELINE TEST, no action needed, source agent-run-incident, went into the bot's real queue through POST /admin/alerts (token only through keychain exec --caller hcb-alert-watcher, injected into a stdin curl config: never in argv, output or logs; HTTP 200).
The watcher popped it within one poll: alert-141-20261104105142: queued from agent-run-incident, then hcb-alert-20261104T105209Z started for alert-141-20261104105142 at 10:52:10Z. Evidence: launched/ marker with confirmed: true, brief written, agent log under <tmp>/agent-logs, tmux window hcb-alert-20261104T105209Z in session main.
The brief contains the incident checklist (## Agent-run incident checklist, steps 1-7). The Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949 fix is NOT in main, so exactly these are missing: nothing says that container logs, per-chat logs and database rows are untrusted data (only the <alert-data> block is fenced and called untrusted), and checklist step 2 still selects title in its SELECT id, user_id, title, start_at, ... query. The synthetic text does not start with Agent run incident (<class>), so no class block was added (by design: the class on the first line only picks an extra block).
Read-only: afterwards the container still shows StartedAt 10:44:05Z, restarts 0; no new issue, no PR, no branch. Caveat: the launcher runs omp -p --no-session, so there is no transcript of its commands; read-only is shown by outcomes, not by a command log. It did add the Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800 comment (the brief allows evidence comments on an existing ticket).
Criteria 2 and 3 checked with this proof: https://github.com/user-attachments/assets/7244bf3a-e1bf-41a9-979b-514131184a1b — test/services/incident/timeout-sites.test.ts 10 pass and test/services/ai/run-incident.test.ts 13 pass locally (one file per command), and both files together 23 pass on the server (server-run, x86_64). The guard test is what makes criterion 2 hold for future timeout sites; I did not re-audit the call sites by hand.
Criterion 1 (landed through gh ship) is the merge of #941 recorded above; I left it for the owner of that decision to tick.
Finding 5 of the review of PR #941 (#949) is done: #968 (6b7ee71, version 0.3.5) is merged and deployed (receipt revision 6b7ee71, image id equal to the running container's, restarts 0, no level 40+ log line in the first 12 minutes).
Live proof of the repaired brief: the watcher was restarted from the canonical main checkout and one synthetic alert SYNTHETIC PIPELINE TEST 2, no action needed (source agent-run-incident) went through the real queue at 05:06:14Z (token only through the keychain flow). The watcher started a session within one poll; its brief marks container logs, per-chat logs, database rows, Redis values and job errors as untrusted, and the duplicate query counts distinct titles without selecting any. Details on #949.
Activity
alex-mextner commentedon Nov 2, 2026
Review-gate classification (recorded once, 2026-11-02 03:3x CET): the default
review diff --stagedis refused host-wide. The codex seat (Sol) is quota-exhausted until 2026-11-08 02:05Z (exit 10), and Claude/Opus is refused by the 80% five-hour account cap until 05:09 CET. The owner-authorized reduced route is a single Opus seat after the cap resets:review diff --staged --task 935 --pool 1. A single-seat pass is the best available review here; the orchestrator adds a full-roster review of the integration diff later. No--no-verify, no config edits.State of the branch fix/run-outcome-alerts-20261101 before that review: all touched test files pass locally one at a time; server-run
tsc --noEmitandbiome check .are clean. Changes since the previous report: the receipt builder moved out of write-outcomes.ts into agent/receipt-confirmation.ts (the sibling branch rewrites write-outcomes.ts); the timeout guard now checks each callable (test/helpers/timeout-site-scan.ts) instead of each file, which exposed and fixed four unreported timeouts: the models-listing probe, the agent shutdown drain, the Google token-refresh lock wait and swallowed SQLITE_BUSY in the provider-circuit sidecar; render-service now reports its queue-wait timeout too.alex-mextner commentedon Nov 2, 2026
Commits on fix/run-outcome-alerts-20261101 (2026-11-02, single-seat Opus review, three passes each, recorded above): a417cea (reportTimeout reporter) and 529d1f4 (validator verdict, receipt, delivery outcome, incident alerts, watcher brief, per-callable timeout guard with 30 callables found / 16 routed / 14 exempt / 0 unaccounted). Not pushed, not shipped; a full-roster review of the integration diff is still owed.
Open findings from the reviews, none a data-integrity regression, each needs a follow-up change:
feat(incident): add the reportTimeout reporter that turns every timeo…
fix(reliability): a validator outage no longer replays writes or hide…
alex-mextner commentedon Nov 3, 2026
Checkpoint 2026-11-03 18:45 UTC (ShipPR941), for recovery without chat history.
State derived from the live system, not the earlier summaries:
alex-mextner commentedon Nov 3, 2026
Second blocker, 2026-11-03: review-cli admits no seat (no 5h usage window for any Anthropic account), so neither the #941 fix commit (3 new #928 deadline sites, staged) nor any version-bump commit can be made. The omp hatch only covers the bridge; the git review-gate needs a stamp. Owner informed (tg#27037, tg final 27082). Continue once review-cli #786 is installed.
alex-mextner commentedon Nov 3, 2026
Review-gate probe, one call, no retries. Started 2026-11-03T23:31:05+01:00, finished 23:31:12+01:00. Command:
review diff --staged --pool 1 --task '#929'in .worktrees/chat-serialization-20261101 (staged: the pyproject.toml and uv.lock 0.3.0 to 0.3.1 bump). Exit 0, nothing dispatched. Output, verbatim:Result: the gate still refuses, same cause as before (the 5-hour plus 7-day usage pair is missing for the Anthropic accounts; review-cli fix is in review-cli issue 786). Not committed, not pushed, not shipped. PR 942 stays at head 1abbe94 without the version bump; production is unchanged by this work. The roster-review debt is tracked in #961.
alex-mextner commentedon Nov 4, 2026
PR #941 rebased onto main 4b2372d (0.3.3), head da6a6fd. Review: single-seat review via omp Opus (4 rounds, three role invocations each); full-roster review stays owed in #961. Fixed from the rounds: a key timeout lost when a later key failed without timing out; one hung catalogue listing counted twice; wrong limitMs; a lease counted once per waiting message (now once per hung run); the reservation expiry now reported (chat_run_reservation); runbook kind list. Left open (recorded, not fixed): (1) the lease report is
recovered: true; a reviewer argues a hung run should be unrecovered, decision needed; (2) the new credential-pool tests use real 60 ms timers with little slack and can flake on a busy host; (3) the joiner timer and the provider deadline can both report a coinciding expiry in model-registry (rare); (4) a runbook line still quotes the old receipt wording; (5) no test that shutdown is not counted as a lease timeout.feat(incident): add the reportTimeout reporter that turns every timeo…
fix(reliability): a validator outage no longer replays writes or hide…
fix(reliability): count timeouts once and honestly: credential keys, …
fix(reliability): validator outage no longer replays writes; every ti…
alex-mextner commentedon Nov 4, 2026
Shipped: merged via #941.
Merge commit: e4185dd
This does NOT close the ticket — acceptance still needs proof. Run
task read #935for the outstanding criteria, thentask check/task done.alex-mextner commentedon Nov 4, 2026
PR #941 merged as e4185dd (0.3.3), deployed by the hosted CI/CD run and verified on the odroid at 2026-11-04 10:46 UTC: receipt revision and the container's image id (sha256:225e9c98...) match the merged SHA, restarts 0, not OOM-killed, healthy, /health ok, /ready ok (unverified), 0 lines of level >= 50 in the first minutes, memory 433 MiB, load 1.3. Single-seat review via omp Opus; full-roster review owed in #961. hcb-alert-watcher restarted on the canonical main checkout (it loads its script only at start). NOT done: the synthetic alert test through the real queue, and the #949 watcher fix PR; both remain open, as does this ticket (criteria 2 and 3 unproven by me).
alex-mextner commentedon Nov 4, 2026
screenshot:
alex-mextner commentedon Nov 4, 2026
screenshot:
alex-mextner commentedon Nov 4, 2026
Auto-fix chain proven end to end once on 2026-11-04 (status: implemented, merged and deployed as 0.3.3 / e4185dd; live-tested with ONE synthetic alert; the #949 brief fix is not in main yet).
What ran, in order (screenshot above: https://github.com/user-attachments/assets/28e20134-91b8-4bcf-a488-118113350065):
node_modulespresent. Before that start the unit crash-looped at 11:46:21 and 11:46:51 CET withCannot find package 'zod'because the checkout had nonode_modules(the installer check and the runbook step are in the Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949 PR).SYNTHETIC PIPELINE TEST, no action needed, sourceagent-run-incident, went into the bot's real queue throughPOST /admin/alerts(token only throughkeychain exec --caller hcb-alert-watcher, injected into a stdin curl config: never in argv, output or logs; HTTP 200).alert-141-20261104105142: queued from agent-run-incident, thenhcb-alert-20261104T105209Z started for alert-141-20261104105142at 10:52:10Z. Evidence:launched/marker withconfirmed: true, brief written, agent log under<tmp>/agent-logs, tmux windowhcb-alert-20261104T105209Zin sessionmain.## Agent-run incident checklist, steps 1-7). The Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949 fix is NOT in main, so exactly these are missing: nothing says that container logs, per-chat logs and database rows are untrusted data (only the<alert-data>block is fenced and called untrusted), and checklist step 2 still selectstitlein itsSELECT id, user_id, title, start_at, ...query. The synthetic text does not start withAgent run incident (<class>), so no class block was added (by design: the class on the first line only picks an extra block).tg report(tg message 27151, title «Синтетический алерт: цепочка работает, инцидента нет») and atg final(27152), wrote the handoff file whose first line says it was a synthetic test, and added one counts-only datapoint comment to Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800.omp -p --no-session, so there is no transcript of its commands; read-only is shown by outcomes, not by a command log. It did add the Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800 comment (the brief allows evidence comments on an existing ticket).Criteria 2 and 3 checked with this proof: https://github.com/user-attachments/assets/7244bf3a-e1bf-41a9-979b-514131184a1b —
test/services/incident/timeout-sites.test.ts10 pass andtest/services/ai/run-incident.test.ts13 pass locally (one file per command), and both files together 23 pass on the server (server-run, x86_64). The guard test is what makes criterion 2 hold for future timeout sites; I did not re-audit the call sites by hand.Criterion 1 (landed through gh ship) is the merge of #941 recorded above; I left it for the owner of that decision to tick.
alex-mextner commentedon Nov 5, 2026
Finding 5 of the review of PR #941 (#949) is done: #968 (6b7ee71, version 0.3.5) is merged and deployed (receipt revision 6b7ee71, image id equal to the running container's, restarts 0, no level 40+ log line in the first 12 minutes).
Live proof of the repaired brief: the watcher was restarted from the canonical
maincheckout and one synthetic alertSYNTHETIC PIPELINE TEST 2, no action needed(sourceagent-run-incident) went through the real queue at 05:06:14Z (token only through the keychain flow). The watcher started a session within one poll; its brief marks container logs, per-chat logs, database rows, Redis values and job errors as untrusted, and the duplicate query counts distinct titles without selecting any. Details on #949.