Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ set in `src/database/db.ts`. Never assume the filename — always check the code
- Use **pino** for all logging. The `err` key triggers pino's error serializer (stack trace included).
- Always pass errors as `{ err: error }`, never `{ error: String(error) }` or `{ error: err.message }`.
- `String(error)` and `.message` lose the stack trace and are not acceptable.
- Credentials never reach a log line or the alert queue (#972): `src/utils/logger.ts` passes every pino line through `redactSecrets` (`src/utils/redact-secrets.ts`: Telegram bot tokens, `sk-ant-…` keys, `Bearer` values, plus the secret values `loadConfig()` registers) right before it is written, `installConsoleRedaction()` does the same for `console` output of libraries, and `AlertRepository.push` redacts alert text. A fetch error's `err.path` is the full request URL, bot token included, so log `{ err }` freely; a new secret with no recognisable shape goes into the `registerSecretValues` list in `loadConfig()`.

## Linting

Expand Down
6 changes: 6 additions & 0 deletions docs/reference/deploy-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,12 @@ rm -f "$f"
ssh root@odroidn2 'docker logs --since 3h hypercal-bot 2>&1 | grep -E "\"level\":(40|50|60)|AI request metric|Delivered via|deep-link fallback attempted"'
```

### Telegram start at boot (#972)

`launchBot` (`src/bootstrap/launch.ts`) awaits `bot.start()`. A transient failure of the first Telegram call (`getMe`: ECONNRESET, ETIMEDOUT, ECONNREFUSED, a 429 or 5xx answer, a gateway page that is not JSON) logs `Telegram start failed on a transient network error; will retry` at level 40 (fields `attempt`, `maxAttempts`, `delayMs`, `code`) and tries again, up to 8 attempts with 1, 2, 4, 8, 15, 15, 15 s pauses (about a minute; a 429 on `getMe` waits at least its `retry_after`, at most 120 s). A shutdown that begins during a pause or an attempt ends the launch without another attempt or a crash alert (a start that finished by then is stopped again). In webhook mode (production) gramio sends `setWebhook` through its own retry loop inside one `start()` call: a 429 there waits out the full `retry_after`, uncapped, and the launcher sees no error until that call fails. After the last attempt it logs one level-60 `Telegram start failed after 8 attempts…` line, queues the alert `Bot failed to start: …` (the wrapper error with the failure's name and code, never the URL), runs the graceful shutdown (agents drained, queues closed, 8 s deadline: the workers started before the launch may hold jobs) and exits 1, so the supervisor restarts the process. A rejected token (401/404) or an error of unknown kind is not retried: one level-60 `Telegram start failed permanently and is not retried: check BOT_TOKEN…` line, the same alert, shutdown and exit. The command menu published by the `onStart` hook is cosmetic: a failed `setMyCommands` logs `Bot command menu was not updated` at level 40 and the bot keeps running. A boot with `RestartCount 1` and an `Unhandled promise rejection` for `getMe` is therefore a bug again, not a network blip.

Logs, console output of libraries and alert texts are redacted (`src/utils/redact-secrets.ts`): `[redacted-telegram-token]`, `[redacted-anthropic-key]`, `Bearer [redacted]` or `[redacted-secret]` replaces the credential. Raw copies of an old container log exist beyond `docker logs`, and replacing the container does not remove them: every deploy runs `deploy-evidence.py release-logs` before the swap and keeps the outgoing container's complete log, unredacted, as `/opt/hypercal/logs/release-<sha12>-<stamp>/container.log.gz` (mode 0700, root; the newest one is never deleted by `log-retention.py`), and `failed-*` bundles keep a raw `container.log.gz` as well. So the 2026-11-05 15:16 UTC boot of 78b63430 (level-60 record with the bot token in `err.path`) lives in the container's json log (`docker inspect --format '{{.LogPath}}' hypercal-bot`) and, after the next deploy, in `release-78b63430…/container.log.gz`. Count, never print: `grep -cE '[0-9]{5,}:[A-Za-z0-9_-]{30,}'` on the json log, `zgrep -cE …` on each `container.log.gz`. Truncating or rotating them is the owner's decision; an agent does not delete logs. Only reissuing the token makes it safe: a token that reached a log or a transcript is compromised for as long as it is valid, and revoking and reissuing `BOT_TOKEN` in BotFather (production and stage) is the owner's decision, not an agent action.

## Shared Server

Production moved on 2026-10-09 22:30 UTC (#784) from the DigitalOcean droplet `104.248.84.190` (amd64, 1 CPU) to the home ODROID-N2+ `root@odroidn2` (Tailscale MagicDNS name; aarch64, Armbian trixie, 6 cores A53+A73, 3.7 GB RAM, root filesystem on a microSD card). The bot and redis run there from `/opt/hypercal`.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "hypercalendarbot"
version = "0.3.6"
version = "0.3.7"
requires-python = ">=3.12"
dependencies = [
"pyrofork>=2.3",
Expand Down
19 changes: 14 additions & 5 deletions src/bootstrap/bot-commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,11 +108,20 @@ export function createOnStartHandler(
): (context: { info: { username?: string } }) => Promise<void> {
return async ({ info }) => {
webServerDeps.botStarted = true;
await api.setMyCommands({ commands: menu.en });
await api.setMyCommands({
commands: menu.ru,
language_code: 'ru',
});
// gramio runs this hook without awaiting it, so a rejection here is an unhandled one and the crash
// handler exits the process (#972: a connection reset right after a successful start). The menu is
// cosmetic, so each call failing is logged and the bot keeps serving.
const languages: Array<{ commands: BotCommand[]; language_code?: string }> = [
{ commands: menu.en },
{ commands: menu.ru, language_code: 'ru' },
];
for (const params of languages) {
try {
await api.setMyCommands(params);
} catch (err) {
botLogger.warn({ err, language: params.language_code ?? 'default' }, 'Bot command menu was not updated');
}
}
botLogger.info({ username: info.username }, 'Bot started');
};
}
13 changes: 11 additions & 2 deletions src/bootstrap/crash-handlers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
// (it writes to db.alerts), so a crash before that point is logged and exits without an alert.

import { botLogger } from '../utils/logger.ts';
import { redactSecrets } from '../utils/redact-secrets.ts';

export interface CrashLogger {
fatal(obj: object, message: string): void;
Expand All @@ -22,6 +23,11 @@ export interface CrashHandlerDeps {
export interface CrashHandlers {
/** Best-effort and synchronous: db.alerts.push() does not await. */
setAlertSink(sink: ((message: string) => void) | undefined): void;
/**
* Queues the alert for a failure the caller handles itself, with no exit: a start failure that must
* first run the graceful shutdown, which closes the database this alert is written to.
*/
alertFatal(error: Error): void;
onUncaughtException(error: Error): void;
onUnhandledRejection(reason: unknown): void;
}
Expand All @@ -41,9 +47,12 @@ export function createCrashHandlers(deps: CrashHandlerDeps): CrashHandlers {
setAlertSink(sink) {
alertSink = sink;
},
alertFatal(error) {
alertSink?.(redactSecrets(`Bot failed to start: ${error.stack ?? error.message}`));
},
onUncaughtException(error) {
deps.logger.fatal({ err: error }, 'Uncaught exception');
alertSink?.(`Bot crashed: ${error.stack ?? error.message}`);
alertSink?.(redactSecrets(`Bot crashed: ${error.stack ?? error.message}`));
deps.exit(1);
},
onUnhandledRejection(reason) {
Expand All @@ -53,7 +62,7 @@ export function createCrashHandlers(deps: CrashHandlerDeps): CrashHandlers {
return;
}
deps.logger.fatal({ err }, 'Unhandled promise rejection');
alertSink?.(`Bot unhandled rejection: ${err.stack ?? err.message}`);
alertSink?.(redactSecrets(`Bot unhandled rejection: ${err.stack ?? err.message}`));
deps.exit(1);
},
};
Expand Down
227 changes: 218 additions & 9 deletions src/bootstrap/launch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@
// on `webServerDeps` BEFORE bot.start() tells Telegram the URL, so the first delivery is served.
// - Pending updates are kept, not dropped: a message sent while the bot restarted is answered now,
// and createStaleUpdateGuard skips ones that waited too long.
// - `bot.start()` is awaited. It is async (getMe, then setWebhook or the polling kick-off); left
// floating, a getMe reset on 2026-11-05 was an unhandled rejection that killed the process (#972).
// A transient network failure is retried with bounded backoff; a permanent one (the token is
// rejected, an unexpected error) fails at once. Both log one clear line and end by throwing, and
// src/index.ts turns the throw into the crash handlers' alert and exit. Retrying start() is safe:
// getMe is a read, setWebhook and deleteWebhook are idempotent.

import type { EnvConfig } from '../config/env.ts';
import { publicUrl, TELEGRAM_WEBHOOK_PATH } from '../config/public-url.ts';
Expand All @@ -22,6 +28,167 @@ export interface WebhookStartOptions {

export interface LaunchableBot {
start(options?: WebhookStartOptions): unknown;
/** Called when a start finished after shutdown had begun, so polling that just began does not run on. */
stop?(): unknown;
}

/** How many times start() is tried before the launch gives up and the supervisor restarts the process. */
export const START_MAX_ATTEMPTS = 8;
const START_FIRST_DELAY_MS = 1_000;
const START_MAX_DELAY_MS = 15_000;
/** The longest single pause, reached only by a Telegram flood wait (`retry_after`). */
const START_MAX_FLOOD_WAIT_MS = 120_000;

/** Bun and Node network failure codes that a retry can outlast. */
const TRANSIENT_NETWORK_CODES: ReadonlySet<string> = new Set([
'ECONNRESET',
'ETIMEDOUT',
'ECONNREFUSED',
'ECONNABORTED',
'EPIPE',
'ENOTFOUND',
'EAI_AGAIN',
'ENETUNREACH',
'EHOSTUNREACH',
'ConnectionClosed',
'ConnectionRefused',
'FailedToOpenSocket',
'UND_ERR_SOCKET',
'UND_ERR_CONNECT_TIMEOUT',
]);

/**
* Whether a start() failure is worth another try: a network failure (a string `code` such as
* ECONNRESET, a timeout or abort, the same wrapped in `cause`), a Telegram answer that is the server's or
* the rate limit's problem (gramio's TelegramError carries a numeric `code` and the failing `method`;
* 429 and 5xx), or an answer body that is not JSON (gramio parses every response as JSON, and a gateway
* error page or an empty body from Telegram's front end fails there as a SyntaxError). A 401/404 token
* rejection, any other 4xx and an error of unknown kind are not transient.
*/
export function isTransientStartError(error: unknown): boolean {
if (!(error instanceof Error)) return false;
const { code, method } = error as Error & { code?: unknown; method?: unknown };
if (typeof code === 'string' && TRANSIENT_NETWORK_CODES.has(code)) return true;
if (typeof code === 'number' && typeof method === 'string') return code === 429 || code >= 500;
if (error instanceof SyntaxError || error.name === 'AbortError' || error.name === 'TimeoutError') return true;
return error.cause !== undefined && isTransientStartError(error.cause);
}

/** What a log line or error message may show of a failure: stable identifiers, never its message (which can quote a URL). */
function failureKind(error: unknown): { code?: string | number; name?: string } {
if (!(error instanceof Error)) return {};
const { code } = error as Error & { code?: unknown };
return { code: typeof code === 'string' || typeof code === 'number' ? code : undefined, name: error.name };
}

function describeFailure(error: unknown): string {
const { code, name } = failureKind(error);
return [name, code].filter((part) => part !== undefined).join(' ') || 'unknown error';
}

/** A start that gave up: the retry loop already logged its fatal line. */
export class TelegramStartError extends Error {
constructor(message: string, options?: { cause?: unknown }) {
super(message, options);
this.name = 'TelegramStartError';
}
}

export interface StartLogger {
warn(fields: object, message: string): void;
fatal(fields: object, message: string): void;
info(fields: object, message: string): void;
}

/** Seams for tests: the pause between attempts and where the lines go. */
export interface StartEnvironment {
sleep?: (ms: number) => Promise<void>;
logger?: StartLogger;
}

/**
* The pause after failed attempt `attempt` (1-based): doubling from one second, capped, and never shorter
* than the flood wait Telegram asked for (`payload.retry_after` of a 429), itself capped so one answer
* cannot park the process for hours. A restart straight after giving up would only lengthen a ban.
*/
function startRetryDelayMs(attempt: number, error: unknown): number {
const backoff = Math.min(START_FIRST_DELAY_MS * 2 ** (attempt - 1), START_MAX_DELAY_MS);
const { payload } = error as { payload?: { retry_after?: unknown } };
const retryAfter = payload?.retry_after;
const floodWaitMs = typeof retryAfter === 'number' && retryAfter > 0 ? retryAfter * 1_000 : 0;
return Math.max(backoff, Math.min(floodWaitMs, START_MAX_FLOOD_WAIT_MS));
}

async function startWithRetry(
bot: LaunchableBot,
start: () => unknown,
environment: Required<StartEnvironment>,
shuttingDown: () => boolean,
): Promise<boolean> {
const { sleep, logger } = environment;
for (let attempt = 1; ; attempt++) {
// A shutdown that begins while a start is pending or paused must not leave the bot running: it is
// closing the queues and the database, and a late start would take updates it can no longer handle
// and confirm their offset. The shutdown's own bot.stop() ran before polling began, so it is repeated.
if (shuttingDown()) {
logger.info({ attempt }, 'Telegram start abandoned: shutdown in progress');
return false;
}
try {
await start();
if (!shuttingDown()) return true;
logger.info({ attempt }, 'Telegram start finished during shutdown; stopping the bot again');
await stopQuietly(bot, logger);
return false;
} catch (error) {
if (shuttingDown()) {
logger.info({ attempt, ...failureKind(error) }, 'Telegram start failed during shutdown; not retried');
return false;
}
throwIfStartFailureIsFinal(error, attempt, logger);
const delayMs = startRetryDelayMs(attempt, error);
logger.warn(
{ ...failureKind(error), attempt, maxAttempts: START_MAX_ATTEMPTS, delayMs },
'Telegram start failed on a transient network error; will retry',
);
await sleep(delayMs);
}
}
}

/**
* Logs one fatal line and throws when a start failure ends the launch: a permanent one (a rejected token,
* an error of unknown kind) at once, a transient one on the last attempt. Returns when a retry is due.
*/
function throwIfStartFailureIsFinal(error: unknown, attempt: number, logger: StartLogger): void {
if (!isTransientStartError(error)) {
logger.fatal(
{ err: error, attempt },
'Telegram start failed permanently and is not retried: check BOT_TOKEN and the Telegram API answer',
);
throw new TelegramStartError(`Telegram start failed permanently (not retried): ${describeFailure(error)}`, {
cause: error,
});
}
if (attempt < START_MAX_ATTEMPTS) return;
logger.fatal(
{ err: error, attempt },
`Telegram start failed after ${START_MAX_ATTEMPTS} attempts on network errors; giving up so the supervisor restarts the process`,
);
throw new TelegramStartError(
`Telegram start failed after ${START_MAX_ATTEMPTS} attempts: ${describeFailure(error)}`,
{
cause: error,
},
);
}

async function stopQuietly(bot: LaunchableBot, logger: StartLogger): Promise<void> {
try {
await bot.stop?.();
} catch (err) {
logger.warn({ err }, 'Stopping the bot after a start that finished during shutdown failed');
}
}

export interface LaunchCollaborators<TBot extends LaunchableBot> {
Expand All @@ -43,12 +210,22 @@ export async function realLaunchCollaborators<TBot extends LaunchableBot>(): Pro

export async function launchBot<TBot extends LaunchableBot>(
config: Pick<EnvConfig, 'PUBLIC_DOMAIN'>,
webServerDeps: Pick<WebServerDeps, 'telegramWebhookHandler'>,
webServerDeps: Pick<WebServerDeps, 'telegramWebhookHandler' | 'telegramUpdatesClosed'>,
bot: TBot,
collaborators?: LaunchCollaborators<TBot>,
environment: StartEnvironment = {},
): Promise<void> {
const resolved: Required<StartEnvironment> = {
sleep: environment.sleep ?? ((ms) => Bun.sleep(ms)),
logger: environment.logger ?? botLogger,
};
if (!config.PUBLIC_DOMAIN) {
bot.start();
await startWithRetry(
bot,
() => bot.start(),
resolved,
() => webServerDeps.telegramUpdatesClosed === true,
);
return;
}
const { createWebhookHandler, randomSecret } = collaborators ?? (await realLaunchCollaborators<TBot>());
Expand All @@ -57,11 +234,43 @@ export async function launchBot<TBot extends LaunchableBot>(

webServerDeps.telegramWebhookHandler = createWebhookHandler(bot, webhookSecret);

bot.start({
webhook: {
url: webhookUrl,
secret_token: webhookSecret,
},
});
botLogger.info({ webhookUrl }, 'Bot started (webhook mode)');
const started = await startWithRetry(
bot,
() => bot.start({ webhook: { url: webhookUrl, secret_token: webhookSecret } }),
resolved,
() => webServerDeps.telegramUpdatesClosed === true,
);
if (started) resolved.logger.info({ webhookUrl }, 'Bot started (webhook mode)');
}

export interface LaunchFailureDeps {
logger: Pick<StartLogger, 'fatal' | 'warn'>;
/** Queues the crash alert; may throw (a locked database). */
alertFatal(error: Error): void;
/** The graceful shutdown, with its own deadline. */
shutdown(): Promise<void>;
exit(code: number): void;
}

/**
* The exit after a launch that threw. By then the workers started before the launch may hold jobs (the
* retries last a minute and more), so the process shuts down gracefully before exiting 1 and the
* supervisor restarts it. The alert goes first because shutdown closes the database that holds the alert
* queue; a failing alert or shutdown never skips the exit. The retry loop logged its own fatal line; an
* error from anywhere else in the launch (the webhook adapter) is logged here, or it would leave no trace.
*/
export async function handleLaunchFailure(error: unknown, deps: LaunchFailureDeps): Promise<void> {
const failure = error instanceof Error ? error : new Error(String(error));
if (!(failure instanceof TelegramStartError)) deps.logger.fatal({ err: failure }, 'Bot launch failed');
try {
deps.alertFatal(failure);
} catch (err) {
deps.logger.warn({ err }, 'The launch-failure alert could not be queued');
}
try {
await deps.shutdown();
} catch (err) {
deps.logger.warn({ err }, 'Shutdown after a failed launch threw');
}
deps.exit(1);
}
Loading
Loading