Skip to content

A Telegram getMe reset at boot is logged as an unhandled rejection at level 60 with the bot token inside the URL path, and the container restarts #972

Open

Description

@alex-mextner

What

On the 2026-11-05 15:16 UTC deploy of 78b6343 the first start logged 'Unhandled promise rejection' (level 60, module bot) for ECONNRESET on gramio getMe during start(); the container exited and Docker restarted it (RestartCount 1), and the second start was fine. The log record carries err.path with the full api.telegram.org URL including the bot token. Retry getMe on a transient reset instead of dying, and redact err.path (token) in the logger serializer.

Why (motivation)

A one-off network reset at boot must not cost a restart, and the token must not reach logs that agents read (it was shown on screen during deploy verification).

User impact

Deploys can show restarts 1 and a fatal log line for a transient network blip, and the bot token sits in container logs.

Cost of inaction

The next boot reset repeats the restart, and the token stays readable in logs.

Acceptance criteria

  • A getMe ECONNRESET at start is retried and does not exit the process (test)
  • A logged TelegramError or fetch error never contains the bot token (test with a synthetic token)

Screenshots

(none)

Deploy

Deployed: no

Links

  • (none)

Skipped gates

Activity

  1. alex-mextner commented on Nov 5, 2026

    @alex-mextner
    OwnerAuthor

    Corroborating production evidence from alert session hcb-alert-20261105T151713Z (read-only investigation; nothing on the server was changed).

    Timeline (UTC, 2026-11-05):

    • Outgoing container (image 6b7ee719, up since 05:03) healthy through its last log line 15:15:09 (15-min sync ticks all day). The deploy of 78b63430 stopped it ~15:15:10–15:16:00 (evidence dir release-6b7ee719ac9e-2026-11-05_15-16-00).
    • New container created 15:16:02 → web server 15:16:17 → gramio start() 15:16:51 → level-60 unhandled rejection on getMe (ECONNRESET; token confirmed present in err.path) 15:16:52 → Docker restart policy restarted the same container 15:16:53 (RestartCount 1) → second start fully up 15:17:22 (Bot started, getMe ok).
    • Release accepted by the gate at 15:17:44, receipt ready: ok (unverified). Stable since: RestartCount still 1, healthcheck healthy, zero level≥40 log lines after 15:17:23.

    Two notes for the fix:

    1. deploy-health.py never sees this failure class: _stable_verdict scans logs only since the container's current StartedAt (refreshed on every poll), and a single restart is deliberately not a crash loop (≥2 above baseline). A first-boot fatal + auto-restart therefore always passes the release verdict. Worth deciding whether the fatal scan should anchor to the original container start or count pre-restart fatals.
    2. The readiness alert down-1793891767 (3×30 s failed probes, last 503 bot not started) is the same event: /ready answers 503 while botStarted === false during bootstrap (livenessFailure, src/web/server.ts). Total user-visible gap ≈ 2 min 10 s (15:15:10 → 15:17:22): ~95 s of it is the normal stop+backup+recreate+bootstrap window, ~30 s from this crash. Telegram retries webhook delivery across the gap, so no lost updates are expected.
  2. alex-mextner commented on Nov 6, 2026

    @alex-mextner
    OwnerAuthor

    Progress, 2026-11-06 (PR #973, Refs #972; not merged, not deployed yet):

    • Implemented: launchBot awaits bot.start() and retries transient failures (ECONNRESET, ETIMEDOUT, ECONNREFUSED, 429, 5xx, non-JSON gateway page) with 1, 2, 4, 8, 15, 15, 15 s pauses over 8 attempts; a 401/404 or unknown error fails at once with one level-60 line; a failed launch alerts, runs the graceful shutdown and exits 1. The onStart menu hook no longer rejects unhandled. Every pino line, library console output and alert text is redacted (Telegram bot token, sk-ant keys, Bearer values, registered config secrets including the Redis password).
    • Tested: criterion 1 by test/bootstrap/launch-start-retry.test.ts and launch-real-gramio.test.ts (the real gramio Bot against a local socket that resets, answers an HTML 502, a 429 and a 401); criterion 2 by test/utils/logger-redaction.test.ts, redact-secrets.test.ts and the crash-handler / alert-repository / env tests, all with synthetic tokens. Removing the streamWrite hook makes the real-gramio test fail.
    • Reviewed: single-seat review diff --staged --pool 1 --task 972 -m omp:anthropic/claude-opus-5-5, six rounds, the last with no actionable findings. Full-roster review is owed in Full-roster review of the integration diff of the incident fixes (PRs 941, 942, async pictures) is owed #961.
    • Hosted CI: test, lint, type-bans, CodeQL, gitleaks, Stage QA all green on the head (Stage QA drove the stage bot through /start, /ping, /help, /today, /search, a free-text question).
    • Finding for the owner: the bot token of the 2026-11-05 15:16 UTC boot is in the production container json log (1 line; count only, not printed), and the next deploy copies that log unredacted into /opt/hypercal/logs/release-*/container.log.gz. Redaction cannot remove it; only reissuing BOT_TOKEN in BotFather (production and stage) makes it safe. Truncating/rotating those logs is the owner's call; no log was touched.
  3. alex-mextner commented on Nov 6, 2026

    @alex-mextner
    OwnerAuthor

    Pending, 2026-11-06: PR #973 is green (hosted CI, CodeQL, Stage QA on head 5f40d99, version 0.3.7) but NOT merged: gh ship 973 refuses at the review-quorum gate (0/3 distinct roles, the Sol roles are unavailable until 2026-11-09). The one-time hatch was requested once and not approved; the owner has the choice: approve the hatch, or wait for Sol on 2026-11-09. After the merge still to do: verify the deploy (receipt, image id = merged SHA, health/ready, restarts 0, 10 min of logs without level>=40), count token-shape matches per log file (counts only), and update #799/#938. The #972 acceptance criteria stay unchecked until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions