Skip to content

Route every bot timeout through one reportTimeout reporter and finish landing the run-outcome alert work #935

Open

Description

@alex-mextner

What

Follow-up of #931. The worktree .worktrees/run-outcome-alerts-20261101 holds staged, uncommitted work (the staged review queue was 34-47 min): validator unavailable verdict, receipt-only confirmation, agent-run-incident alerts, delivery outcome with Telegram message ids, watcher checklist. Left to do: (1) land it (review, commit, ship); (2) src/services/incident/timeout-reporter.ts exists with one call site (validator); inventory every other timeout site (provider stream timeouts, Google and HTTP calls, DB and Redis waits, scheduler jobs, BullMQ waits) and route each through reportTimeout, recovered true when a fallback absorbed it, plus a static guard test over src; extend the watcher brief for the timeout source; (3) the receipt code lives in write-outcomes.ts, which another branch rewrites: move it to a new module; (4) the run-incident debug-log request-id test passes locally but fails on the server-run host (reads an empty log): find why.

Why (motivation)

Owner decision 2026-11-01: every timeout must reach the auto-fix flow, not only failed user runs.

User impact

Until then slow-host timeouts are visible only in docker logs, and the owner learns of them from users.

Cost of inaction

Timeouts that do not fail a user run stay invisible to the repair flow.

Acceptance criteria

  • The staged work is committed and landed through gh ship with Refs to issue 931
  • Every timeout site reports through reportTimeout and a static guard test fails on a new unreported timeout site — proof: proof
  • The run-incident debug-log test passes both locally and on the server — proof: proof

Screenshots

  • implementation: implementation

Deploy

Deployed: no

Links

Skipped gates

  • links: branch/path names in the text are false positives; the issue link is a markdown link
  • post-merge-acceptance: Criterion 1 is about landing through gh ship, which this merge is. Criterion 2 is covered by test/services/incident/timeout-sites.test.ts (10 pass on the server for this head) but is not ticked here. Criterion 3 (run-incident debug-log test locally and on the server) is not proven by me. None of the three is closed by this merge; the ticket stays open, task done needs real proofs after the deploy.

Activity

  1. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Review-gate classification (recorded once, 2026-11-02 03:3x CET): the default review diff --staged is refused host-wide. The codex seat (Sol) is quota-exhausted until 2026-11-08 02:05Z (exit 10), and Claude/Opus is refused by the 80% five-hour account cap until 05:09 CET. The owner-authorized reduced route is a single Opus seat after the cap resets: review diff --staged --task 935 --pool 1. A single-seat pass is the best available review here; the orchestrator adds a full-roster review of the integration diff later. No --no-verify, no config edits.

    State of the branch fix/run-outcome-alerts-20261101 before that review: all touched test files pass locally one at a time; server-run tsc --noEmit and biome check . are clean. Changes since the previous report: the receipt builder moved out of write-outcomes.ts into agent/receipt-confirmation.ts (the sibling branch rewrites write-outcomes.ts); the timeout guard now checks each callable (test/helpers/timeout-site-scan.ts) instead of each file, which exposed and fixed four unreported timeouts: the models-listing probe, the agent shutdown drain, the Google token-refresh lock wait and swallowed SQLITE_BUSY in the provider-circuit sidecar; render-service now reports its queue-wait timeout too.

  2. alex-mextner commented on Nov 2, 2026

    @alex-mextner
    OwnerAuthor

    Commits on fix/run-outcome-alerts-20261101 (2026-11-02, single-seat Opus review, three passes each, recorded above): a417cea (reportTimeout reporter) and 529d1f4 (validator verdict, receipt, delivery outcome, incident alerts, watcher brief, per-callable timeout guard with 30 callables found / 16 routed / 14 exempt / 0 unaccounted). Not pushed, not shipped; a full-roster review of the integration diff is still owed.

    Open findings from the reviews, none a data-integrity regression, each needs a follow-up change:

    1. Receipt outro lacks the "no need to repeat" sentence that docs/intents/response-quality.md requires; one pass asked to drop it because a partly-done request might not be complete. Owner decision.
    2. Receipt falls back to the ledger summary for the whole run when one write names no event (build named lines per write).
    3. A normal run whose Telegram delivery failed (deliveryOutcome failed) raises no agent-run-incident; an interrupted supplement that wrote to the calendar raises none.
    4. Recovered-only timeout alerts can start an investigation session; launch only when unrecovered > 0 or a burst threshold passes.
    5. Logs, chat logs and database rows named in the incident checklist are not marked untrusted; make the duplicate query return ids only.
    6. timeout alerts / run-incident alerts ignore the ADMIN_ALERT_TOKEN gate (rows pile up undrained without a token).
    7. db_busy and run-incident alerts are written into the locked database (see SQLITE_BUSY lock timeouts are reported only from the bot error handler and worker failure handler #939).
    8. agent_request is reported for any error after the deadline, not only aborts; most timeout kinds carry no requestId; runbook delivery-line format and watcher fixture are stale.
    9. An unavailable validator still triggers the corrective retry on a run without writes.
    10. No test drives a real provider-chain timeout through reportProviderTimeouts, nor a delete_event / non-UTC receipt.
  3. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Checkpoint 2026-11-03 18:45 UTC (ShipPR941), for recovery without chat history.

    State derived from the live system, not the earlier summaries:

  4. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Second blocker, 2026-11-03: review-cli admits no seat (no 5h usage window for any Anthropic account), so neither the #941 fix commit (3 new #928 deadline sites, staged) nor any version-bump commit can be made. The omp hatch only covers the bridge; the git review-gate needs a stamp. Owner informed (tg#27037, tg final 27082). Continue once review-cli #786 is installed.

  5. alex-mextner commented on Nov 3, 2026

    @alex-mextner
    OwnerAuthor

    Review-gate probe, one call, no retries. Started 2026-11-03T23:31:05+01:00, finished 23:31:12+01:00. Command: review diff --staged --pool 1 --task '#929' in .worktrees/chat-serialization-20261101 (staged: the pyproject.toml and uv.lock 0.3.0 to 0.3.1 bump). Exit 0, nothing dispatched. Output, verbatim:

    [review-cli] warning: --pool 1 is below the default ship floor of 3 distinct board roles; this run cannot satisfy `review task CODE --check` / gh ship.
    [review-cli] auto-selected level standard (Laya standard 0.45; tier_not_acted:standard/uncertain:floor_standard); pass --level to choose it yourself
    [review-cli] level 'standard' is a fixed roster (Opus (claude:claude-opus-5-5)) and does not run without all of it: a blocked seat is never replaced by another or a cheaper model. Nothing was dispatched.
    
    Blocked seats:
      - Opus (claude:claude-opus-5-5): omp:anthropic/claude-opus-5-5: no omp anthropic account may be used (3): mextner@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); alex.mex.ux@gmail.com: USAGE UNKNOWN (review-cli account cap: a complete shared 5-hour and 7-day observation is missing); invntrm@gmail.com: EXCLUDED (config.yaml omp_accounts.exclude); claude:claude-opus-5-5: usage неизвестен для аккаунта invntrm@gmail.com: нет подтверждённой пары 5ч/7д от omp usage; консервативно не используется (ограничение review-cli, config account_usage_cap_percent=80): a complete shared 5-hour and 7-day observation is missing
    
    Providers missing (0 of 3 roles live: none):
      - anthropic: Opus (claude:claude-opus-5-5): [same four account reasons as above]; back: unknown (no account usage window names a reset)
    
    Wait until the capacity is back (or free the account), or choose deliberately: `--level LEVEL` for another level, `-m MODEL` to name the model yourself.
    

    Result: the gate still refuses, same cause as before (the 5-hour plus 7-day usage pair is missing for the Anthropic accounts; review-cli fix is in review-cli issue 786). Not committed, not pushed, not shipped. PR 942 stays at head 1abbe94 without the version bump; production is unchanged by this work. The roster-review debt is tracked in #961.

  6. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    PR #941 rebased onto main 4b2372d (0.3.3), head da6a6fd. Review: single-seat review via omp Opus (4 rounds, three role invocations each); full-roster review stays owed in #961. Fixed from the rounds: a key timeout lost when a later key failed without timing out; one hung catalogue listing counted twice; wrong limitMs; a lease counted once per waiting message (now once per hung run); the reservation expiry now reported (chat_run_reservation); runbook kind list. Left open (recorded, not fixed): (1) the lease report is recovered: true; a reviewer argues a hung run should be unrecovered, decision needed; (2) the new credential-pool tests use real 60 ms timers with little slack and can flake on a busy host; (3) the joiner timer and the provider deadline can both report a coinciding expiry in model-registry (rare); (4) a runbook line still quotes the old receipt wording; (5) no test that shutdown is not counted as a lease timeout.

  7. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #941.
    Merge commit: e4185dd
    This does NOT close the ticket — acceptance still needs proof. Run task read #935 for the outstanding criteria, then task check/task done.

  8. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    PR #941 merged as e4185dd (0.3.3), deployed by the hosted CI/CD run and verified on the odroid at 2026-11-04 10:46 UTC: receipt revision and the container's image id (sha256:225e9c98...) match the merged SHA, restarts 0, not OOM-killed, healthy, /health ok, /ready ok (unverified), 0 lines of level >= 50 in the first minutes, memory 433 MiB, load 1.3. Single-seat review via omp Opus; full-roster review owed in #961. hcb-alert-watcher restarted on the canonical main checkout (it loads its script only at start). NOT done: the synthetic alert test through the real queue, and the #949 watcher fix PR; both remain open, as does this ticket (criteria 2 and 3 unproven by me).

  9. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    screenshot: c3-tests.png

  10. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    screenshot: c2-chain.png

  11. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Auto-fix chain proven end to end once on 2026-11-04 (status: implemented, merged and deployed as 0.3.3 / e4185dd; live-tested with ONE synthetic alert; the #949 brief fix is not in main yet).

    What ran, in order (screenshot above: https://github.com/user-attachments/assets/28e20134-91b8-4bcf-a488-118113350065):

    1. The watcher unit started 11:47:08 CET (10:47:08Z), after the merge commit (11:14:44 CET). Canonical checkout at e4185dd, node_modules present. Before that start the unit crash-looped at 11:46:21 and 11:46:51 CET with Cannot find package 'zod' because the checkout had no node_modules (the installer check and the runbook step are in the Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949 PR).
    2. At 10:51:41Z one alert, clearly labelled SYNTHETIC PIPELINE TEST, no action needed, source agent-run-incident, went into the bot's real queue through POST /admin/alerts (token only through keychain exec --caller hcb-alert-watcher, injected into a stdin curl config: never in argv, output or logs; HTTP 200).
    3. The watcher popped it within one poll: alert-141-20261104105142: queued from agent-run-incident, then hcb-alert-20261104T105209Z started for alert-141-20261104105142 at 10:52:10Z. Evidence: launched/ marker with confirmed: true, brief written, agent log under <tmp>/agent-logs, tmux window hcb-alert-20261104T105209Z in session main.
    4. The brief contains the incident checklist (## Agent-run incident checklist, steps 1-7). The Incident checklist in the watcher brief does not mark logs, chat logs and database rows as untrusted data, and its duplicate query returns row content #949 fix is NOT in main, so exactly these are missing: nothing says that container logs, per-chat logs and database rows are untrusted data (only the <alert-data> block is fenced and called untrusted), and checklist step 2 still selects title in its SELECT id, user_id, title, start_at, ... query. The synthetic text does not start with Agent run incident (<class>), so no class block was added (by design: the class on the first line only picks an extra block).
    5. The session ended after about 4 minutes. It sent a Russian tg report (tg message 27151, title «Синтетический алерт: цепочка работает, инцидента нет») and a tg final (27152), wrote the handoff file whose first line says it was a synthetic test, and added one counts-only datapoint comment to Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800.
    6. Read-only: afterwards the container still shows StartedAt 10:44:05Z, restarts 0; no new issue, no PR, no branch. Caveat: the launcher runs omp -p --no-session, so there is no transcript of its commands; read-only is shown by outcomes, not by a command log. It did add the Odroid host memory pressure makes every HyperCalendarBot image render 6-15 s instead of ~1-2 s #800 comment (the brief allows evidence comments on an existing ticket).

    Criteria 2 and 3 checked with this proof: https://github.com/user-attachments/assets/7244bf3a-e1bf-41a9-979b-514131184a1b — test/services/incident/timeout-sites.test.ts 10 pass and test/services/ai/run-incident.test.ts 13 pass locally (one file per command), and both files together 23 pass on the server (server-run, x86_64). The guard test is what makes criterion 2 hold for future timeout sites; I did not re-audit the call sites by hand.

    Criterion 1 (landed through gh ship) is the merge of #941 recorded above; I left it for the owner of that decision to tick.

  12. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Finding 5 of the review of PR #941 (#949) is done: #968 (6b7ee71, version 0.3.5) is merged and deployed (receipt revision 6b7ee71, image id equal to the running container's, restarts 0, no level 40+ log line in the first 12 minutes).

    Live proof of the repaired brief: the watcher was restarted from the canonical main checkout and one synthetic alert SYNTHETIC PIPELINE TEST 2, no action needed (source agent-run-incident) went through the real queue at 05:06:14Z (token only through the keychain flow). The watcher started a session within one poll; its brief marks container logs, per-chat logs, database rows, Redis values and job errors as untrusted, and the duplicate query counts distinct titles without selecting any. Details on #949.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions