diff --git a/CLAUDE.md b/CLAUDE.md index 9c23af14..daf44da2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -210,6 +210,7 @@ set in `src/database/db.ts`. Never assume the filename — always check the code - Use **pino** for all logging. The `err` key triggers pino's error serializer (stack trace included). - Always pass errors as `{ err: error }`, never `{ error: String(error) }` or `{ error: err.message }`. - `String(error)` and `.message` lose the stack trace and are not acceptable. +- Credentials never reach a log line or the alert queue (#972): `src/utils/logger.ts` passes every pino line through `redactSecrets` (`src/utils/redact-secrets.ts`: Telegram bot tokens, `sk-ant-…` keys, `Bearer` values, plus the secret values `loadConfig()` registers) right before it is written, `installConsoleRedaction()` does the same for `console` output of libraries, and `AlertRepository.push` redacts alert text. A fetch error's `err.path` is the full request URL, bot token included, so log `{ err }` freely; a new secret with no recognisable shape goes into the `registerSecretValues` list in `loadConfig()`. ## Linting diff --git a/docs/reference/deploy-runbook.md b/docs/reference/deploy-runbook.md index 6e1c7e5e..db2b7380 100644 --- a/docs/reference/deploy-runbook.md +++ b/docs/reference/deploy-runbook.md @@ -71,6 +71,12 @@ rm -f "$f" ssh root@odroidn2 'docker logs --since 3h hypercal-bot 2>&1 | grep -E "\"level\":(40|50|60)|AI request metric|Delivered via|deep-link fallback attempted"' ``` +### Telegram start at boot (#972) + +`launchBot` (`src/bootstrap/launch.ts`) awaits `bot.start()`. A transient failure of the first Telegram call (`getMe`: ECONNRESET, ETIMEDOUT, ECONNREFUSED, a 429 or 5xx answer, a gateway page that is not JSON) logs `Telegram start failed on a transient network error; will retry` at level 40 (fields `attempt`, `maxAttempts`, `delayMs`, `code`) and tries again, up to 8 attempts with 1, 2, 4, 8, 15, 15, 15 s pauses (about a minute; a 429 on `getMe` waits at least its `retry_after`, at most 120 s). A shutdown that begins during a pause or an attempt ends the launch without another attempt or a crash alert (a start that finished by then is stopped again). In webhook mode (production) gramio sends `setWebhook` through its own retry loop inside one `start()` call: a 429 there waits out the full `retry_after`, uncapped, and the launcher sees no error until that call fails. After the last attempt it logs one level-60 `Telegram start failed after 8 attempts…` line, queues the alert `Bot failed to start: …` (the wrapper error with the failure's name and code, never the URL), runs the graceful shutdown (agents drained, queues closed, 8 s deadline: the workers started before the launch may hold jobs) and exits 1, so the supervisor restarts the process. A rejected token (401/404) or an error of unknown kind is not retried: one level-60 `Telegram start failed permanently and is not retried: check BOT_TOKEN…` line, the same alert, shutdown and exit. The command menu published by the `onStart` hook is cosmetic: a failed `setMyCommands` logs `Bot command menu was not updated` at level 40 and the bot keeps running. A boot with `RestartCount 1` and an `Unhandled promise rejection` for `getMe` is therefore a bug again, not a network blip. + +Logs, console output of libraries and alert texts are redacted (`src/utils/redact-secrets.ts`): `[redacted-telegram-token]`, `[redacted-anthropic-key]`, `Bearer [redacted]` or `[redacted-secret]` replaces the credential. Raw copies of an old container log exist beyond `docker logs`, and replacing the container does not remove them: every deploy runs `deploy-evidence.py release-logs` before the swap and keeps the outgoing container's complete log, unredacted, as `/opt/hypercal/logs/release--/container.log.gz` (mode 0700, root; the newest one is never deleted by `log-retention.py`), and `failed-*` bundles keep a raw `container.log.gz` as well. So the 2026-11-05 15:16 UTC boot of 78b63430 (level-60 record with the bot token in `err.path`) lives in the container's json log (`docker inspect --format '{{.LogPath}}' hypercal-bot`) and, after the next deploy, in `release-78b63430…/container.log.gz`. Count, never print: `grep -cE '[0-9]{5,}:[A-Za-z0-9_-]{30,}'` on the json log, `zgrep -cE …` on each `container.log.gz`. Truncating or rotating them is the owner's decision; an agent does not delete logs. Only reissuing the token makes it safe: a token that reached a log or a transcript is compromised for as long as it is valid, and revoking and reissuing `BOT_TOKEN` in BotFather (production and stage) is the owner's decision, not an agent action. + ## Shared Server Production moved on 2026-10-09 22:30 UTC (#784) from the DigitalOcean droplet `104.248.84.190` (amd64, 1 CPU) to the home ODROID-N2+ `root@odroidn2` (Tailscale MagicDNS name; aarch64, Armbian trixie, 6 cores A53+A73, 3.7 GB RAM, root filesystem on a microSD card). The bot and redis run there from `/opt/hypercal`. diff --git a/pyproject.toml b/pyproject.toml index 82b39787..3d3c9759 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "hypercalendarbot" -version = "0.3.6" +version = "0.3.7" requires-python = ">=3.12" dependencies = [ "pyrofork>=2.3", diff --git a/src/bootstrap/bot-commands.ts b/src/bootstrap/bot-commands.ts index d0c25e5b..fae7f1f0 100644 --- a/src/bootstrap/bot-commands.ts +++ b/src/bootstrap/bot-commands.ts @@ -108,11 +108,20 @@ export function createOnStartHandler( ): (context: { info: { username?: string } }) => Promise { return async ({ info }) => { webServerDeps.botStarted = true; - await api.setMyCommands({ commands: menu.en }); - await api.setMyCommands({ - commands: menu.ru, - language_code: 'ru', - }); + // gramio runs this hook without awaiting it, so a rejection here is an unhandled one and the crash + // handler exits the process (#972: a connection reset right after a successful start). The menu is + // cosmetic, so each call failing is logged and the bot keeps serving. + const languages: Array<{ commands: BotCommand[]; language_code?: string }> = [ + { commands: menu.en }, + { commands: menu.ru, language_code: 'ru' }, + ]; + for (const params of languages) { + try { + await api.setMyCommands(params); + } catch (err) { + botLogger.warn({ err, language: params.language_code ?? 'default' }, 'Bot command menu was not updated'); + } + } botLogger.info({ username: info.username }, 'Bot started'); }; } diff --git a/src/bootstrap/crash-handlers.ts b/src/bootstrap/crash-handlers.ts index 8a867bb7..45e9733f 100644 --- a/src/bootstrap/crash-handlers.ts +++ b/src/bootstrap/crash-handlers.ts @@ -8,6 +8,7 @@ // (it writes to db.alerts), so a crash before that point is logged and exits without an alert. import { botLogger } from '../utils/logger.ts'; +import { redactSecrets } from '../utils/redact-secrets.ts'; export interface CrashLogger { fatal(obj: object, message: string): void; @@ -22,6 +23,11 @@ export interface CrashHandlerDeps { export interface CrashHandlers { /** Best-effort and synchronous: db.alerts.push() does not await. */ setAlertSink(sink: ((message: string) => void) | undefined): void; + /** + * Queues the alert for a failure the caller handles itself, with no exit: a start failure that must + * first run the graceful shutdown, which closes the database this alert is written to. + */ + alertFatal(error: Error): void; onUncaughtException(error: Error): void; onUnhandledRejection(reason: unknown): void; } @@ -41,9 +47,12 @@ export function createCrashHandlers(deps: CrashHandlerDeps): CrashHandlers { setAlertSink(sink) { alertSink = sink; }, + alertFatal(error) { + alertSink?.(redactSecrets(`Bot failed to start: ${error.stack ?? error.message}`)); + }, onUncaughtException(error) { deps.logger.fatal({ err: error }, 'Uncaught exception'); - alertSink?.(`Bot crashed: ${error.stack ?? error.message}`); + alertSink?.(redactSecrets(`Bot crashed: ${error.stack ?? error.message}`)); deps.exit(1); }, onUnhandledRejection(reason) { @@ -53,7 +62,7 @@ export function createCrashHandlers(deps: CrashHandlerDeps): CrashHandlers { return; } deps.logger.fatal({ err }, 'Unhandled promise rejection'); - alertSink?.(`Bot unhandled rejection: ${err.stack ?? err.message}`); + alertSink?.(redactSecrets(`Bot unhandled rejection: ${err.stack ?? err.message}`)); deps.exit(1); }, }; diff --git a/src/bootstrap/launch.ts b/src/bootstrap/launch.ts index 036b2562..d691bd9c 100644 --- a/src/bootstrap/launch.ts +++ b/src/bootstrap/launch.ts @@ -10,6 +10,12 @@ // on `webServerDeps` BEFORE bot.start() tells Telegram the URL, so the first delivery is served. // - Pending updates are kept, not dropped: a message sent while the bot restarted is answered now, // and createStaleUpdateGuard skips ones that waited too long. +// - `bot.start()` is awaited. It is async (getMe, then setWebhook or the polling kick-off); left +// floating, a getMe reset on 2026-11-05 was an unhandled rejection that killed the process (#972). +// A transient network failure is retried with bounded backoff; a permanent one (the token is +// rejected, an unexpected error) fails at once. Both log one clear line and end by throwing, and +// src/index.ts turns the throw into the crash handlers' alert and exit. Retrying start() is safe: +// getMe is a read, setWebhook and deleteWebhook are idempotent. import type { EnvConfig } from '../config/env.ts'; import { publicUrl, TELEGRAM_WEBHOOK_PATH } from '../config/public-url.ts'; @@ -22,6 +28,167 @@ export interface WebhookStartOptions { export interface LaunchableBot { start(options?: WebhookStartOptions): unknown; + /** Called when a start finished after shutdown had begun, so polling that just began does not run on. */ + stop?(): unknown; +} + +/** How many times start() is tried before the launch gives up and the supervisor restarts the process. */ +export const START_MAX_ATTEMPTS = 8; +const START_FIRST_DELAY_MS = 1_000; +const START_MAX_DELAY_MS = 15_000; +/** The longest single pause, reached only by a Telegram flood wait (`retry_after`). */ +const START_MAX_FLOOD_WAIT_MS = 120_000; + +/** Bun and Node network failure codes that a retry can outlast. */ +const TRANSIENT_NETWORK_CODES: ReadonlySet = new Set([ + 'ECONNRESET', + 'ETIMEDOUT', + 'ECONNREFUSED', + 'ECONNABORTED', + 'EPIPE', + 'ENOTFOUND', + 'EAI_AGAIN', + 'ENETUNREACH', + 'EHOSTUNREACH', + 'ConnectionClosed', + 'ConnectionRefused', + 'FailedToOpenSocket', + 'UND_ERR_SOCKET', + 'UND_ERR_CONNECT_TIMEOUT', +]); + +/** + * Whether a start() failure is worth another try: a network failure (a string `code` such as + * ECONNRESET, a timeout or abort, the same wrapped in `cause`), a Telegram answer that is the server's or + * the rate limit's problem (gramio's TelegramError carries a numeric `code` and the failing `method`; + * 429 and 5xx), or an answer body that is not JSON (gramio parses every response as JSON, and a gateway + * error page or an empty body from Telegram's front end fails there as a SyntaxError). A 401/404 token + * rejection, any other 4xx and an error of unknown kind are not transient. + */ +export function isTransientStartError(error: unknown): boolean { + if (!(error instanceof Error)) return false; + const { code, method } = error as Error & { code?: unknown; method?: unknown }; + if (typeof code === 'string' && TRANSIENT_NETWORK_CODES.has(code)) return true; + if (typeof code === 'number' && typeof method === 'string') return code === 429 || code >= 500; + if (error instanceof SyntaxError || error.name === 'AbortError' || error.name === 'TimeoutError') return true; + return error.cause !== undefined && isTransientStartError(error.cause); +} + +/** What a log line or error message may show of a failure: stable identifiers, never its message (which can quote a URL). */ +function failureKind(error: unknown): { code?: string | number; name?: string } { + if (!(error instanceof Error)) return {}; + const { code } = error as Error & { code?: unknown }; + return { code: typeof code === 'string' || typeof code === 'number' ? code : undefined, name: error.name }; +} + +function describeFailure(error: unknown): string { + const { code, name } = failureKind(error); + return [name, code].filter((part) => part !== undefined).join(' ') || 'unknown error'; +} + +/** A start that gave up: the retry loop already logged its fatal line. */ +export class TelegramStartError extends Error { + constructor(message: string, options?: { cause?: unknown }) { + super(message, options); + this.name = 'TelegramStartError'; + } +} + +export interface StartLogger { + warn(fields: object, message: string): void; + fatal(fields: object, message: string): void; + info(fields: object, message: string): void; +} + +/** Seams for tests: the pause between attempts and where the lines go. */ +export interface StartEnvironment { + sleep?: (ms: number) => Promise; + logger?: StartLogger; +} + +/** + * The pause after failed attempt `attempt` (1-based): doubling from one second, capped, and never shorter + * than the flood wait Telegram asked for (`payload.retry_after` of a 429), itself capped so one answer + * cannot park the process for hours. A restart straight after giving up would only lengthen a ban. + */ +function startRetryDelayMs(attempt: number, error: unknown): number { + const backoff = Math.min(START_FIRST_DELAY_MS * 2 ** (attempt - 1), START_MAX_DELAY_MS); + const { payload } = error as { payload?: { retry_after?: unknown } }; + const retryAfter = payload?.retry_after; + const floodWaitMs = typeof retryAfter === 'number' && retryAfter > 0 ? retryAfter * 1_000 : 0; + return Math.max(backoff, Math.min(floodWaitMs, START_MAX_FLOOD_WAIT_MS)); +} + +async function startWithRetry( + bot: LaunchableBot, + start: () => unknown, + environment: Required, + shuttingDown: () => boolean, +): Promise { + const { sleep, logger } = environment; + for (let attempt = 1; ; attempt++) { + // A shutdown that begins while a start is pending or paused must not leave the bot running: it is + // closing the queues and the database, and a late start would take updates it can no longer handle + // and confirm their offset. The shutdown's own bot.stop() ran before polling began, so it is repeated. + if (shuttingDown()) { + logger.info({ attempt }, 'Telegram start abandoned: shutdown in progress'); + return false; + } + try { + await start(); + if (!shuttingDown()) return true; + logger.info({ attempt }, 'Telegram start finished during shutdown; stopping the bot again'); + await stopQuietly(bot, logger); + return false; + } catch (error) { + if (shuttingDown()) { + logger.info({ attempt, ...failureKind(error) }, 'Telegram start failed during shutdown; not retried'); + return false; + } + throwIfStartFailureIsFinal(error, attempt, logger); + const delayMs = startRetryDelayMs(attempt, error); + logger.warn( + { ...failureKind(error), attempt, maxAttempts: START_MAX_ATTEMPTS, delayMs }, + 'Telegram start failed on a transient network error; will retry', + ); + await sleep(delayMs); + } + } +} + +/** + * Logs one fatal line and throws when a start failure ends the launch: a permanent one (a rejected token, + * an error of unknown kind) at once, a transient one on the last attempt. Returns when a retry is due. + */ +function throwIfStartFailureIsFinal(error: unknown, attempt: number, logger: StartLogger): void { + if (!isTransientStartError(error)) { + logger.fatal( + { err: error, attempt }, + 'Telegram start failed permanently and is not retried: check BOT_TOKEN and the Telegram API answer', + ); + throw new TelegramStartError(`Telegram start failed permanently (not retried): ${describeFailure(error)}`, { + cause: error, + }); + } + if (attempt < START_MAX_ATTEMPTS) return; + logger.fatal( + { err: error, attempt }, + `Telegram start failed after ${START_MAX_ATTEMPTS} attempts on network errors; giving up so the supervisor restarts the process`, + ); + throw new TelegramStartError( + `Telegram start failed after ${START_MAX_ATTEMPTS} attempts: ${describeFailure(error)}`, + { + cause: error, + }, + ); +} + +async function stopQuietly(bot: LaunchableBot, logger: StartLogger): Promise { + try { + await bot.stop?.(); + } catch (err) { + logger.warn({ err }, 'Stopping the bot after a start that finished during shutdown failed'); + } } export interface LaunchCollaborators { @@ -43,12 +210,22 @@ export async function realLaunchCollaborators(): Pro export async function launchBot( config: Pick, - webServerDeps: Pick, + webServerDeps: Pick, bot: TBot, collaborators?: LaunchCollaborators, + environment: StartEnvironment = {}, ): Promise { + const resolved: Required = { + sleep: environment.sleep ?? ((ms) => Bun.sleep(ms)), + logger: environment.logger ?? botLogger, + }; if (!config.PUBLIC_DOMAIN) { - bot.start(); + await startWithRetry( + bot, + () => bot.start(), + resolved, + () => webServerDeps.telegramUpdatesClosed === true, + ); return; } const { createWebhookHandler, randomSecret } = collaborators ?? (await realLaunchCollaborators()); @@ -57,11 +234,43 @@ export async function launchBot( webServerDeps.telegramWebhookHandler = createWebhookHandler(bot, webhookSecret); - bot.start({ - webhook: { - url: webhookUrl, - secret_token: webhookSecret, - }, - }); - botLogger.info({ webhookUrl }, 'Bot started (webhook mode)'); + const started = await startWithRetry( + bot, + () => bot.start({ webhook: { url: webhookUrl, secret_token: webhookSecret } }), + resolved, + () => webServerDeps.telegramUpdatesClosed === true, + ); + if (started) resolved.logger.info({ webhookUrl }, 'Bot started (webhook mode)'); +} + +export interface LaunchFailureDeps { + logger: Pick; + /** Queues the crash alert; may throw (a locked database). */ + alertFatal(error: Error): void; + /** The graceful shutdown, with its own deadline. */ + shutdown(): Promise; + exit(code: number): void; +} + +/** + * The exit after a launch that threw. By then the workers started before the launch may hold jobs (the + * retries last a minute and more), so the process shuts down gracefully before exiting 1 and the + * supervisor restarts it. The alert goes first because shutdown closes the database that holds the alert + * queue; a failing alert or shutdown never skips the exit. The retry loop logged its own fatal line; an + * error from anywhere else in the launch (the webhook adapter) is logged here, or it would leave no trace. + */ +export async function handleLaunchFailure(error: unknown, deps: LaunchFailureDeps): Promise { + const failure = error instanceof Error ? error : new Error(String(error)); + if (!(failure instanceof TelegramStartError)) deps.logger.fatal({ err: failure }, 'Bot launch failed'); + try { + deps.alertFatal(failure); + } catch (err) { + deps.logger.warn({ err }, 'The launch-failure alert could not be queued'); + } + try { + await deps.shutdown(); + } catch (err) { + deps.logger.warn({ err }, 'Shutdown after a failed launch threw'); + } + deps.exit(1); } diff --git a/src/config/env.ts b/src/config/env.ts index ed9955fe..036b3b5d 100644 --- a/src/config/env.ts +++ b/src/config/env.ts @@ -11,6 +11,7 @@ import { } from '../services/ai/role-routing.ts'; import { TOOL_SCHEMA_MODES, type ToolSchemaMode } from '../services/ai/tool-schema-mode.ts'; import { logger, logOnce } from '../utils/logger.ts'; +import { registerSecretValues } from '../utils/redact-secrets.ts'; import { GOOGLE_OAUTH_CALLBACK_PATH, publicUrl } from './public-url.ts'; /** Owner-chosen order for main requests: z.ai, then Claude, then Hugging Face. @@ -235,6 +236,24 @@ function parseTelegramSessionMasterKey(): string | undefined { return undefined; } +/** + * The password in `redis://:@host:port`, as written in the URL (percent-encoded: what a quoted URL + * shows) and decoded (what a client prints). It has no recognisable shape, and a connection error can quote the URL. + */ +function redisPasswords(redisUrl: string): string[] { + try { + const { password } = new URL(redisUrl); + if (password === '') return []; + try { + return [password, decodeURIComponent(password)]; + } catch { + return [password]; + } + } catch { + return []; + } +} + export function loadConfig(): EnvConfig { const BOT_TOKEN = requireEnv('BOT_TOKEN'); const toolSchemaMode = parseToolSchemaMode(); @@ -256,7 +275,7 @@ export function loadConfig(): EnvConfig { const INTENT_LEARNER_DAILY_LIMIT = parseIntegerEnv('INTENT_LEARNER_DAILY_LIMIT') ?? 100; const telegramSessionMasterKey = parseTelegramSessionMasterKey(); - return { + const config: EnvConfig = { BOT_TOKEN, DATABASE_PATH: process.env.DATABASE_PATH || './data/calendar.db', NODE_ENV: (process.env.NODE_ENV as EnvConfig['NODE_ENV']) || 'development', @@ -312,4 +331,27 @@ export function loadConfig(): EnvConfig { GOOGLE_API_KEY: optionalEnv('GOOGLE_API_KEY'), TELEGRAM_SESSION_MASTER_KEY: telegramSessionMasterKey, }; + + // The log and alert redaction knows the SHAPE of a Telegram token or an Anthropic key; these have none + // (an operator-chosen ADMIN_ALERT_TOKEN) or are worth matching exactly, so the loaded values are registered. + registerSecretValues([ + ...redisPasswords(config.REDIS_URL), + config.BOT_TOKEN, + config.INLINE_BOT_TOKEN, + config.ADMIN_ALERT_TOKEN, + config.CLAUDE_API_TOKEN, + config.CLAUDE_API_TOKEN2, + config.MTPROTO_API_HASH, + config.ZAI_API_KEY, + config.HF_TOKEN, + config.GROQ_API_KEY, + config.OPENROUTER_API_KEY, + config.DEEPGRAM_API_KEY, + config.OPENWEATHER_API_KEY, + config.GOOGLE_API_KEY, + config.GOOGLE_CLIENT_SECRET, + config.ENCRYPTION_KEY, + config.TELEGRAM_SESSION_MASTER_KEY, + ]); + return config; } diff --git a/src/database/repositories/alert.repository.ts b/src/database/repositories/alert.repository.ts index ac2da400..01b34fd4 100644 --- a/src/database/repositories/alert.repository.ts +++ b/src/database/repositories/alert.repository.ts @@ -1,5 +1,6 @@ // src/database/repositories/alert.repository.ts import type { Database } from 'bun:sqlite'; +import { redactSecrets } from '../../utils/redact-secrets.ts'; export interface Alert { id: number; @@ -11,8 +12,9 @@ export interface Alert { export class AlertRepository { constructor(private readonly db: Database) {} + /** Stores the alert with credentials removed: the queue is read by agents and sent to the admin chat. */ push(text: string, source: string): void { - this.db.run('INSERT INTO alerts (text, source) VALUES (?, ?)', [text, source]); + this.db.run('INSERT INTO alerts (text, source) VALUES (?, ?)', [redactSecrets(text), source]); } /** Returns the oldest unconsumed alert and marks it consumed. Returns null if none. */ diff --git a/src/index.ts b/src/index.ts index 2e0ec12e..cff5cef0 100644 --- a/src/index.ts +++ b/src/index.ts @@ -30,7 +30,7 @@ import { createDomainEventBus } from './bootstrap/domain-events.ts'; import { initEventMentionStore } from './bootstrap/event-mentions.ts'; import { initGoogleSync } from './bootstrap/google-sync.ts'; import { initImageRender, recoverImageDeliveries } from './bootstrap/image-render.ts'; -import { launchBot } from './bootstrap/launch.ts'; +import { handleLaunchFailure, launchBot } from './bootstrap/launch.ts'; import { initLocationVerification } from './bootstrap/location.ts'; import { initNotificationScheduler } from './bootstrap/notifications.ts'; import { initScheduledAi } from './bootstrap/scheduled-ai.ts'; @@ -42,9 +42,12 @@ import { createBot } from './bot/index.ts'; import { createTelegramSender } from './services/ai/telegram-sender.ts'; import { createServiceTier } from './services/telegram-session/service-tier.ts'; import { botLogger } from './utils/logger.ts'; +import { installConsoleRedaction } from './utils/redact-secrets.ts'; import { startWebServer } from './web/server.ts'; const crashHandlers = installCrashHandlers(); +// Libraries print with console, outside pino (gramio's polling loop); keep credentials out of those lines too. +installConsoleRedaction(); const core = await initCore(crashHandlers); const { config } = core; @@ -152,6 +155,15 @@ Object.freeze(msgDeps); bot.onStart(createOnStartHandler(buildBotCommands(config), bot.api, ctx.webServerDeps)); -installShutdownSignals(createProcessShutdown(ctx, { bot, agent, webServerHandle })); +const shutdown = createProcessShutdown(ctx, { bot, agent, webServerHandle }); +installShutdownSignals(shutdown); -await launchBot(config, ctx.webServerDeps, bot); +// A launch that threw ends in the graceful shutdown and exit 1 (handleLaunchFailure). +await launchBot(config, ctx.webServerDeps, bot).catch((error: unknown) => + handleLaunchFailure(error, { + logger: botLogger, + alertFatal: (failure) => crashHandlers.alertFatal(failure), + shutdown, + exit: (code) => process.exit(code), + }), +); diff --git a/src/utils/logger.ts b/src/utils/logger.ts index 923babea..50c59740 100644 --- a/src/utils/logger.ts +++ b/src/utils/logger.ts @@ -1,10 +1,33 @@ // src/utils/logger.ts -import pino from 'pino'; +import pino, { type DestinationStream } from 'pino'; +import { redactSecrets } from './redact-secrets.ts'; -export const logger = pino({ - level: process.env.NODE_ENV === 'production' ? 'info' : 'debug', - transport: process.env.NODE_ENV !== 'production' ? { target: 'pino-pretty', options: { colorize: true } } : undefined, -}); +interface LoggerOptions { + level?: string; + /** Where lines go. Omitted: stdout in production, a pretty-printing transport otherwise. */ + destination?: DestinationStream; +} + +/** + * Every serialized line passes `redactSecrets` right before it is written (pino's `streamWrite` hook), so + * a bot token in `err.path`, an error message, a stack or any field never reaches the container log (#972). + */ +export function createLogger(options: LoggerOptions = {}): pino.Logger { + const production = process.env.NODE_ENV === 'production'; + return pino( + { + level: options.level ?? (production ? 'info' : 'debug'), + hooks: { streamWrite: redactSecrets }, + transport: + options.destination === undefined && !production + ? { target: 'pino-pretty', options: { colorize: true } } + : undefined, + }, + options.destination, + ); +} + +export const logger = createLogger(); export const botLogger = logger.child({ module: 'bot' }); export const dbLogger = logger.child({ module: 'db' }); diff --git a/src/utils/redact-secrets.ts b/src/utils/redact-secrets.ts new file mode 100644 index 00000000..8e5d1671 --- /dev/null +++ b/src/utils/redact-secrets.ts @@ -0,0 +1,90 @@ +// src/utils/redact-secrets.ts +// +// One place that removes credentials from text on its way to a log line, a console line or the admin +// alert queue. A leaf module: it imports nothing from the bot, so the logger can use it before config. +// +// Why it exists (#972): Bun's fetch puts the full request URL in `err.path`, and a Telegram request URL +// is `https://api.telegram.org/bot:/`, so a failed getMe printed the bot token in a +// level-60 record that agents read. Redaction works on the finished line, not per call site, so a new +// log statement cannot forget it. +// +// Known shapes: a Telegram bot token, an Anthropic API key or OAuth access token (`sk-ant-…`), an +// `Authorization: Bearer …` value, and any value registered from the loaded config (a secret with no +// recognisable shape, like ADMIN_ALERT_TOKEN). + +import { inspect } from 'node:util'; + +const TELEGRAM_BOT_TOKEN = /(?(); +/** The registry, longest first; rebuilt after a registration. */ +let longestFirst: string[] = []; + +/** + * Remembers config secrets, so a value with no recognisable shape is still removed. Empty and short values + * are ignored. The JSON-escaped form is remembered too: pino writes a value with a quote or a backslash + * in it escaped, and the exact value would not match the line. + */ +export function registerSecretValues(values: ReadonlyArray): void { + for (const value of values) { + if (value === undefined || value.length < MIN_REGISTERED_LENGTH) continue; + registered.add(value); + registered.add(JSON.stringify(value).slice(1, -1)); + } + longestFirst = [...registered].sort((a, b) => b.length - a.length); +} + +/** For tests: the registry is process-global. */ +export function resetRegisteredSecrets(): void { + registered.clear(); + longestFirst = []; +} + +/** + * Registered values go first and longest first: an exact value is the most precise thing known, and a + * shape pattern that ran before it could cut it at the first character outside its alphabet and leave the + * tail readable. + */ +export function redactSecrets(text: string): string { + let out = text; + for (const value of longestFirst) out = out.replaceAll(value, '[redacted-secret]'); + return out + .replace(TELEGRAM_BOT_TOKEN, '[redacted-telegram-token]') + .replace(ANTHROPIC_KEY, '[redacted-anthropic-key]') + .replace(BEARER_VALUE, '$1[redacted]'); +} + +const CONSOLE_METHODS = ['log', 'info', 'warn', 'error', 'debug'] as const; +const installed = new WeakSet(); + +/** + * Libraries print with `console` behind pino's back (gramio's long-polling loop does + * `console.error('Error received when fetching updates', error)`, and an error object prints its `path`). + * Each argument is rendered the way console would, redacted, and replaced by the text only when that + * changed something, so a line without a secret prints exactly as before. + */ +export function installConsoleRedaction(target: Console = console): void { + if (installed.has(target)) return; + installed.add(target); + for (const method of CONSOLE_METHODS) { + const original = target[method]; + if (typeof original !== 'function') continue; + target[method] = (...args: unknown[]) => { + original.apply( + target, + args.map((arg) => { + if (typeof arg === 'string') return redactSecrets(arg); + if (arg === null || typeof arg !== 'object') return arg; + const rendered = inspect(arg, { depth: 6 }); + const clean = redactSecrets(rendered); + return clean === rendered ? arg : clean; + }), + ); + }; + } +} diff --git a/test/bootstrap/bot-commands.test.ts b/test/bootstrap/bot-commands.test.ts index 64a6851d..7bd884d7 100644 --- a/test/bootstrap/bot-commands.test.ts +++ b/test/bootstrap/bot-commands.test.ts @@ -73,18 +73,38 @@ describe('createOnStartHandler', () => { expect(events).toEqual(['default:true', 'ru:true']); }); - test('botStarted is set even when publishing the menu fails', async () => { + test('a menu call that fails does not reject: botStarted stays set, the failure is logged and the bot keeps running (#972)', async () => { const deps = { botStarted: false }; const handler = createOnStartHandler( { en: [], ru: [] }, { setMyCommands: async () => { - throw new Error('telegram down'); + throw Object.assign(new Error('ECONNRESET: socket closed'), { code: 'ECONNRESET' }); }, }, deps, ); - await expect(handler({ info: {} })).rejects.toThrow('telegram down'); + + await expect(handler({ info: {} })).resolves.toBeUndefined(); + expect(deps.botStarted).toBe(true); }); + + test('a failed default-language menu does not skip the Russian one', async () => { + const languages: Array = []; + const handler = createOnStartHandler( + { en: [], ru: [] }, + { + setMyCommands: async (params) => { + languages.push(params.language_code); + if (params.language_code === undefined) throw new Error('telegram down'); + }, + }, + { botStarted: false }, + ); + + await handler({ info: {} }); + + expect(languages).toEqual([undefined, 'ru']); + }); }); diff --git a/test/bootstrap/crash-handlers.test.ts b/test/bootstrap/crash-handlers.test.ts index 6b1cb17e..edbdf0d2 100644 --- a/test/bootstrap/crash-handlers.test.ts +++ b/test/bootstrap/crash-handlers.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from 'bun:test'; import { createCrashHandlers, isTransientAbort } from '../../src/bootstrap/crash-handlers.ts'; +import { FAKE_GETME_URL, FAKE_TELEGRAM_TOKEN, SECRET_FRAGMENTS } from '../helpers/fake-secrets.ts'; interface LogCall { level: 'fatal' | 'warn'; @@ -86,3 +87,61 @@ describe('createCrashHandlers', () => { expect(alerts).toEqual([]); }); }); + +describe('alertFatal (#972)', () => { + test('raises a redacted alert without exiting, so the caller can run the graceful shutdown first', () => { + const { logs, exits, alerts, handlers } = harness(); + handlers.setAlertSink((msg) => alerts.push(msg)); + const failure = new Error(`Telegram start failed after 8 attempts: TypeError ECONNRESET ${FAKE_GETME_URL}`); + + handlers.alertFatal(failure); + + expect(alerts).toHaveLength(1); + expect(alerts[0]).toStartWith('Bot failed to start: '); + expect(alerts[0]).toContain('ECONNRESET'); + expect(alerts[0]).not.toContain(FAKE_TELEGRAM_TOKEN); + expect(exits).toEqual([]); + expect(logs).toEqual([]); + }); + + test('without an alert sink it does nothing', () => { + const { exits, handlers } = harness(); + handlers.alertFatal(new Error('x')); + expect(exits).toEqual([]); + }); +}); + +describe('crash alerts carry no secrets (#972)', () => { + /** What Bun's fetch throws on a reset: the URL, bot token included, sits in `path` and often in the stack text. */ + function resetWithToken(): Error { + const err = Object.assign(new TypeError(`ECONNRESET: request to ${FAKE_GETME_URL} was closed`), { + code: 'ECONNRESET', + path: FAKE_GETME_URL, + }); + err.stack = `TypeError: ECONNRESET\n at fetch (${FAKE_GETME_URL}:1:1)`; + return err; + } + + test('an unhandled rejection pushes an alert without the bot token in the message or the stack', () => { + const { alerts, handlers } = harness(); + handlers.setAlertSink((msg) => alerts.push(msg)); + + handlers.onUnhandledRejection(resetWithToken()); + + expect(alerts).toHaveLength(1); + expect(alerts[0]).toStartWith('Bot unhandled rejection: '); + expect(alerts[0]).not.toContain(FAKE_TELEGRAM_TOKEN); + for (const fragment of SECRET_FRAGMENTS) expect(alerts[0]).not.toContain(fragment); + expect(alerts[0]).toContain('ECONNRESET'); + }); + + test('an uncaught exception pushes an alert without the bot token', () => { + const { alerts, handlers } = harness(); + handlers.setAlertSink((msg) => alerts.push(msg)); + + handlers.onUncaughtException(resetWithToken()); + + expect(alerts[0]).toStartWith('Bot crashed: '); + expect(alerts[0]).not.toContain(FAKE_TELEGRAM_TOKEN); + }); +}); diff --git a/test/bootstrap/launch-real-gramio.test.ts b/test/bootstrap/launch-real-gramio.test.ts new file mode 100644 index 00000000..a475c2bc --- /dev/null +++ b/test/bootstrap/launch-real-gramio.test.ts @@ -0,0 +1,148 @@ +// #972: the retry classifier reads two things this repository does not own, the `code` strings of Bun's +// fetch errors and the shape of gramio's TelegramError. These tests run the REAL gramio Bot through the +// real launchBot against a local socket that resets, answers an HTML gateway page, floods or rejects, so a +// Bun or gramio upgrade that changes either shape fails here instead of reviving the boot loop. +import { afterEach, describe, expect, test } from 'bun:test'; +import net from 'node:net'; +import { Bot } from 'gramio'; +import { launchBot } from '../../src/bootstrap/launch.ts'; +import { createLogger } from '../../src/utils/logger.ts'; +import { FAKE_TELEGRAM_TOKEN, SECRET_FRAGMENTS } from '../helpers/fake-secrets.ts'; + +type Step = 'reset' | 'html502' | 'flood429' | 'unauthorized401' | 'ok'; + +const BODIES: Record, { status: string; type: string; body: string }> = { + html502: { status: '502 Bad Gateway', type: 'text/html', body: '502 Bad Gateway' }, + flood429: { + status: '429 Too Many Requests', + type: 'application/json', + body: '{"ok":false,"error_code":429,"description":"Too Many Requests: retry after 2","parameters":{"retry_after":2}}', + }, + unauthorized401: { + status: '401 Unauthorized', + type: 'application/json', + body: '{"ok":false,"error_code":401,"description":"Unauthorized"}', + }, +}; + +const OK_BY_METHOD: Record = { + getMe: '{"ok":true,"result":{"id":424242,"is_bot":true,"first_name":"Test","username":"test_bot"}}', + setWebhook: '{"ok":true,"result":true}', +}; + +/** A tiny HTTP server that answers each request from a script and counts the Telegram methods it saw. */ +async function scriptedTelegram(steps: Step[]) { + const methods: string[] = []; + const paths: string[] = []; + const server = net.createServer((socket) => { + socket.on('error', () => undefined); + let received = ''; + socket.on('data', (chunk) => { + received += chunk.toString('latin1'); + const head = received.split('\r\n\r\n')[0] ?? ''; + const requestLine = head.split('\r\n')[0] ?? ''; + const requestPath = requestLine.split(' ')[1] ?? ''; + if (!received.includes('\r\n\r\n')) return; + const method = requestPath.split('/').at(-1) ?? ''; + methods.push(method); + paths.push(requestPath); + const isGetMe = method === 'getMe'; + const step: Step = isGetMe ? (steps.shift() ?? 'ok') : 'ok'; + received = ''; + if (step === 'reset') { + socket.resetAndDestroy(); + return; + } + const reply = + step === 'ok' + ? { status: '200 OK', type: 'application/json', body: OK_BY_METHOD[method] ?? '{"ok":true,"result":true}' } + : BODIES[step]; + socket.end( + `HTTP/1.1 ${reply.status}\r\nContent-Type: ${reply.type}\r\nContent-Length: ${Buffer.byteLength(reply.body)}\r\nConnection: close\r\n\r\n${reply.body}`, + ); + }); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve())); + const { port } = server.address() as net.AddressInfo; + return { port, methods, paths, close: () => server.close() }; +} + +const open: Array<() => void> = []; +afterEach(() => { + for (const close of open.splice(0)) close(); +}); + +async function launchAgainst(steps: Step[]) { + const telegram = await scriptedTelegram(steps); + open.push(telegram.close); + const bot = new Bot(FAKE_TELEGRAM_TOKEN, { api: { baseURL: `http://127.0.0.1:${telegram.port}/bot` } }); + const sleeps: number[] = []; + const lines: string[] = []; + const logger = createLogger({ level: 'debug', destination: { write: (line: string) => void lines.push(line) } }); + const outcome = await launchBot({ PUBLIC_DOMAIN: 'bot.example.com' }, {}, bot, undefined, { + sleep: async (ms) => { + sleeps.push(ms); + }, + logger, + }).then( + () => ({ failed: false as const }), + (error: unknown) => ({ failed: true as const, error }), + ); + return { telegram, sleeps, lines, outcome }; +} + +function expectNoSecretIn(lines: string[]): void { + const text = lines.join(''); + expect(text).not.toContain(FAKE_TELEGRAM_TOKEN); + for (const fragment of SECRET_FRAGMENTS) expect(text).not.toContain(fragment); +} + +describe('launchBot against the real gramio Bot', () => { + test('two connection resets on getMe are retried and the webhook is then set', async () => { + const { telegram, sleeps, lines, outcome } = await launchAgainst(['reset', 'reset']); + + expect(outcome.failed).toBe(false); + expect(telegram.methods).toEqual(['getMe', 'getMe', 'getMe', 'setWebhook']); + expect(sleeps).toHaveLength(2); + expect(lines.filter((line) => JSON.parse(line).level === 40)).toHaveLength(2); + expectNoSecretIn(lines); + }); + + test('eight real resets end in a level-60 line whose real fetch error keeps code and path but not the bot token', async () => { + const { telegram, lines, outcome } = await launchAgainst(Array.from({ length: 8 }, () => 'reset' as const)); + + expect(outcome.failed).toBe(true); + // The request path the server saw holds the token, which is what Bun's error puts in `err.path`. + expect(telegram.paths[0]).toContain(FAKE_TELEGRAM_TOKEN); + const fatal = lines.map((line) => JSON.parse(line)).filter((record) => record.level === 60); + expect(fatal).toHaveLength(1); + expect(fatal[0].err.code).toBe('ECONNRESET'); + expect(fatal[0].err.path).toContain('/getMe'); + expect(fatal[0].err.path).not.toContain(FAKE_TELEGRAM_TOKEN); + expectNoSecretIn(lines); + }); + + test('an HTML 502 gateway page from Telegram is retried', async () => { + const { telegram, outcome } = await launchAgainst(['html502']); + + expect(outcome.failed).toBe(false); + expect(telegram.methods.filter((m) => m === 'getMe')).toHaveLength(2); + }); + + test('a 429 flood wait is waited out for the time Telegram asked', async () => { + const { sleeps, outcome } = await launchAgainst(['flood429']); + + expect(outcome.failed).toBe(false); + expect(sleeps).toEqual([2_000]); + }); + + test('a 401 is permanent: one request, no pause, one fatal line without the token', async () => { + const { telegram, sleeps, lines, outcome } = await launchAgainst(['unauthorized401']); + + expect(outcome.failed).toBe(true); + expect(telegram.methods).toEqual(['getMe']); + expect(sleeps).toEqual([]); + expect(lines.filter((line) => JSON.parse(line).level === 60)).toHaveLength(1); + expectNoSecretIn(lines); + }); +}); diff --git a/test/bootstrap/launch-start-retry.test.ts b/test/bootstrap/launch-start-retry.test.ts new file mode 100644 index 00000000..925713c8 --- /dev/null +++ b/test/bootstrap/launch-start-retry.test.ts @@ -0,0 +1,399 @@ +// #972: on 2026-11-05 a Telegram getMe reset during bot.start() was an unhandled rejection at level 60 +// and the container exited. A transient network failure at start is retried with bounded backoff; a +// permanent one (the token is rejected) still fails fast with one honest fatal line. +import { describe, expect, test } from 'bun:test'; +import { + handleLaunchFailure, + isTransientStartError, + type LaunchableBot, + launchBot, + START_MAX_ATTEMPTS, + TelegramStartError, + type WebhookStartOptions, +} from '../../src/bootstrap/launch.ts'; +import { createLogger } from '../../src/utils/logger.ts'; +import { FAKE_GETME_URL, FAKE_TELEGRAM_TOKEN, SECRET_FRAGMENTS } from '../helpers/fake-secrets.ts'; + +/** What Bun's fetch throws on a reset: the failing URL, token included, is in `path`. */ +function networkError(code: string): Error { + return Object.assign(new TypeError(`${code}: The socket connection was closed unexpectedly`), { + code, + path: FAKE_GETME_URL, + errno: 0, + }); +} + +/** gramio's TelegramError: a numeric Bot API `code` and the failing `method`. */ +function telegramError(code: number, description: string): Error { + return Object.assign(new Error(description), { name: 'getMe', method: 'getMe', code, params: {} }); +} + +function harness(failures: Array) { + const starts: Array = []; + const bot: LaunchableBot = { + start: async (options) => { + starts.push(options); + const failure = failures[starts.length - 1]; + if (failure) throw failure; + return undefined; + }, + }; + const sleeps: number[] = []; + const lines: string[] = []; + const logger = createLogger({ level: 'debug', destination: { write: (line: string) => void lines.push(line) } }); + const records = (): Array<{ level: number; msg: string; [key: string]: unknown }> => + lines.map((line) => JSON.parse(line)); + const control = { + sleep: async (ms: number) => { + sleeps.push(ms); + }, + logger, + }; + return { bot, starts, sleeps, lines, records, control }; +} + +const POLLING = { PUBLIC_DOMAIN: undefined }; + +describe('launchBot start retry', () => { + test.each([ + 'ECONNRESET', + 'ETIMEDOUT', + 'ECONNREFUSED', + 'EAI_AGAIN', + 'ConnectionClosed', + ])('a getMe %s is retried and the bot starts without an exception', async (code) => { + const { bot, starts, sleeps, records, control } = harness([networkError(code), networkError(code)]); + + await launchBot(POLLING, {}, bot, undefined, control); + + expect(starts).toHaveLength(3); + expect(sleeps).toHaveLength(2); + const warnings = records().filter((r) => r.level === 40); + expect(warnings).toHaveLength(2); + expect(String(warnings[0]?.msg)).toContain('retry'); + expect(warnings[0]).toMatchObject({ attempt: 1, maxAttempts: START_MAX_ATTEMPTS, code }); + expect(records().some((r) => r.level >= 50)).toBe(false); + }); + + test('a Telegram 502 and a 429 are retried, an AbortError and a wrapped reset too', async () => { + const abort = Object.assign(new Error('The operation was aborted'), { name: 'AbortError' }); + const wrapped = new Error('start failed', { cause: networkError('ECONNRESET') }); + const { bot, starts, control } = harness([ + telegramError(502, 'Bad Gateway'), + telegramError(429, 'Too Many Requests'), + abort, + wrapped, + ]); + + await launchBot(POLLING, {}, bot, undefined, control); + + expect(starts).toHaveLength(5); + }); + + test('the wait between attempts grows, is capped and the number of attempts is bounded', async () => { + const always = Array.from({ length: START_MAX_ATTEMPTS + 3 }, () => networkError('ECONNRESET')); + const { bot, starts, sleeps, records, control } = harness(always); + + await expect(launchBot(POLLING, {}, bot, undefined, control)).rejects.toThrow(/Telegram start failed/); + + expect(starts).toHaveLength(START_MAX_ATTEMPTS); + expect(sleeps).toHaveLength(START_MAX_ATTEMPTS - 1); + expect(sleeps[0]).toBeGreaterThan(0); + for (let i = 1; i < sleeps.length; i++) expect(sleeps[i]).toBeGreaterThanOrEqual(sleeps[i - 1] ?? 0); + expect(Math.max(...sleeps)).toBeLessThanOrEqual(30_000); + expect(sleeps.reduce((a, b) => a + b, 0)).toBeLessThanOrEqual(120_000); + const fatals = records().filter((r) => r.level === 60); + expect(fatals).toHaveLength(1); + expect(String(fatals[0]?.msg)).toContain(`${START_MAX_ATTEMPTS} attempts`); + }); + + test('a rejected token (401) fails at once with one honest fatal line and no retry', async () => { + const { bot, starts, sleeps, records, control } = harness([telegramError(401, 'Unauthorized')]); + + await expect(launchBot(POLLING, {}, bot, undefined, control)).rejects.toThrow(/Telegram start failed/); + + expect(starts).toHaveLength(1); + expect(sleeps).toEqual([]); + const fatals = records().filter((r) => r.level === 60); + expect(fatals).toHaveLength(1); + expect(String(fatals[0]?.msg)).toMatch(/not retried|permanent/i); + expect(records().filter((r) => r.level === 40)).toHaveLength(0); + }); + + test('a getMe 404 and an error with no network code or Telegram status are permanent too', async () => { + for (const failure of [ + telegramError(404, 'The bot token is incorrect. Check it in BotFather.'), + new TypeError('x is not a function'), + ]) { + const { bot, starts, control } = harness([failure]); + await expect(launchBot(POLLING, {}, bot, undefined, control)).rejects.toBeInstanceOf(Error); + expect(starts).toHaveLength(1); + } + }); + + test('neither the retry warnings nor the fatal line carry the bot token', async () => { + const reset = harness([ + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + networkError('ECONNRESET'), + ]); + await launchBot(POLLING, {}, reset.bot, undefined, reset.control).catch(() => undefined); + const rejected = harness([ + Object.assign(telegramError(401, `Unauthorized ${FAKE_GETME_URL}`), { path: FAKE_GETME_URL }), + ]); + await launchBot(POLLING, {}, rejected.bot, undefined, rejected.control).catch(() => undefined); + + for (const text of [...reset.lines, ...rejected.lines]) { + expect(text).not.toContain(FAKE_TELEGRAM_TOKEN); + for (const fragment of SECRET_FRAGMENTS) expect(text).not.toContain(fragment); + } + expect(reset.lines.length).toBeGreaterThan(0); + expect(reset.lines.join('')).toContain('ECONNRESET'); + }); + + test('webhook mode registers the handler once, retries start, and logs "started" only after it succeeded', async () => { + const { bot, starts, records, control } = harness([networkError('ECONNRESET')]); + const registered: string[] = []; + const deps: { telegramWebhookHandler?: (req: Request) => Response } = {}; + + await launchBot( + { PUBLIC_DOMAIN: 'bot.example.com' }, + deps, + bot, + { + randomSecret: () => 'secret-1', + createWebhookHandler: (_bot, secret) => { + registered.push(secret); + return () => new Response('ok'); + }, + }, + control, + ); + + expect(registered).toEqual(['secret-1']); + expect(starts).toHaveLength(2); + expect(starts[1]?.webhook.secret_token).toBe('secret-1'); + const messages = records().map((r) => r.msg); + expect(messages.filter((m) => m === 'Bot started (webhook mode)')).toHaveLength(1); + expect(messages.at(-1)).toBe('Bot started (webhook mode)'); + }); + + test('a start that rejects asynchronously is awaited, not left as an unhandled rejection', async () => { + const rejections: unknown[] = []; + const onRejection = (reason: unknown) => rejections.push(reason); + process.on('unhandledRejection', onRejection); + try { + const { bot, control } = harness([telegramError(401, 'Unauthorized')]); + await launchBot(POLLING, {}, bot, undefined, control).catch(() => undefined); + await new Promise((resolve) => setImmediate(resolve)); + } finally { + process.off('unhandledRejection', onRejection); + } + expect(rejections).toEqual([]); + }); +}); + +describe('launchBot start retry: edge cases', () => { + test('a 5xx answer whose body is not JSON (an HTML gateway page, an empty body) is retried', async () => { + const htmlBody = new SyntaxError('JSON Parse error: Unrecognized token <'); + const { bot, starts, control } = harness([htmlBody, htmlBody]); + + await launchBot(POLLING, {}, bot, undefined, control); + + expect(starts).toHaveLength(3); + }); + + test('a flood wait longer than the backoff is waited out, not hammered', async () => { + const flood = Object.assign(telegramError(429, 'Too Many Requests: retry after 40'), { + payload: { retry_after: 40 }, + }); + const { bot, sleeps, control } = harness([flood]); + + await launchBot(POLLING, {}, bot, undefined, control); + + expect(sleeps).toEqual([40_000]); + }); + + test('a flood wait is capped so one answer cannot park the process for hours', async () => { + const flood = Object.assign(telegramError(429, 'Too Many Requests'), { payload: { retry_after: 86_400 } }); + const { bot, sleeps, control } = harness([flood]); + + await launchBot(POLLING, {}, bot, undefined, control); + + expect(sleeps[0]).toBeLessThanOrEqual(120_000); + expect(sleeps[0]).toBeGreaterThanOrEqual(15_000); + }); + + test('a shutdown that starts during the pause stops the launch: no further attempt, no error', async () => { + const deps: { telegramUpdatesClosed?: boolean } = {}; + const { bot, starts, records, control } = harness([networkError('ECONNRESET')]); + const sleepingControl = { + ...control, + sleep: async () => { + deps.telegramUpdatesClosed = true; + }, + }; + + await launchBot(POLLING, deps, bot, undefined, sleepingControl); + + expect(starts).toHaveLength(1); + expect(records().some((r) => r.level >= 50)).toBe(false); + expect(records().some((r) => String(r.msg).includes('shutdown'))).toBe(true); + }); + + test('a start that succeeds after shutdown began is stopped again and not announced as started', async () => { + const deps: { telegramUpdatesClosed?: boolean } = {}; + const { starts, records, control } = harness([]); + let stops = 0; + const bot: LaunchableBot = { + start: async (options) => { + starts.push(options); + deps.telegramUpdatesClosed = true; + }, + stop: async () => { + stops++; + }, + }; + + await launchBot( + { PUBLIC_DOMAIN: 'bot.example.com' }, + deps, + bot, + { + randomSecret: () => 'secret-1', + createWebhookHandler: () => () => new Response('ok'), + }, + control, + ); + + expect(starts).toHaveLength(1); + expect(stops).toBe(1); + expect(records().map((r) => r.msg)).not.toContain('Bot started (webhook mode)'); + }); + + test('a start that fails during shutdown is abandoned quietly, not raised as a crash', async () => { + for (const failure of [telegramError(401, 'Unauthorized'), networkError('ECONNRESET')]) { + const deps: { telegramUpdatesClosed?: boolean } = {}; + const { starts, records, control } = harness([]); + const bot: LaunchableBot = { + start: async (options) => { + starts.push(options); + deps.telegramUpdatesClosed = true; + throw failure; + }, + }; + + await launchBot(POLLING, deps, bot, undefined, control); + + expect(starts).toHaveLength(1); + expect(records().some((r) => r.level >= 50)).toBe(false); + } + }); + + test('a shutdown that began before the first attempt starts nothing', async () => { + const { bot, starts, control } = harness([]); + + await launchBot(POLLING, { telegramUpdatesClosed: true }, bot, undefined, control); + + expect(starts).toHaveLength(0); + }); + + test('the thrown error names the cause so the crash alert says why, without the URL', async () => { + const permanent = harness([telegramError(401, 'Unauthorized')]); + const error = await launchBot(POLLING, {}, permanent.bot, undefined, permanent.control).catch((e: unknown) => e); + expect(error).toBeInstanceOf(TelegramStartError); + expect((error as Error).message).toContain('401'); + expect((error as Error).cause).toBeInstanceOf(Error); + + const exhausted = harness(Array.from({ length: START_MAX_ATTEMPTS }, () => networkError('ECONNRESET'))); + const last = await launchBot(POLLING, {}, exhausted.bot, undefined, exhausted.control).catch((e: unknown) => e); + expect((last as Error).message).toContain('ECONNRESET'); + }); +}); + +describe('isTransientStartError', () => { + test('classifies network failures and server-side Telegram codes as transient only', () => { + expect(isTransientStartError(networkError('ECONNRESET'))).toBe(true); + expect(isTransientStartError(telegramError(500, 'Internal'))).toBe(true); + expect(isTransientStartError(telegramError(503, 'Unavailable'))).toBe(true); + expect(isTransientStartError(telegramError(401, 'Unauthorized'))).toBe(false); + expect(isTransientStartError(telegramError(400, 'Bad Request'))).toBe(false); + expect(isTransientStartError(new Error('plain'))).toBe(false); + expect(isTransientStartError(new SyntaxError('JSON Parse error'))).toBe(true); + expect(isTransientStartError('ECONNRESET')).toBe(false); + }); +}); + +describe('handleLaunchFailure', () => { + function failureHarness(options: { alertThrows?: boolean; shutdownThrows?: boolean } = {}) { + const events: string[] = []; + const lines: Array<{ level: string; msg: string }> = []; + const deps = { + logger: { + fatal: (_fields: object, msg: string) => void lines.push({ level: 'fatal', msg }), + warn: (_fields: object, msg: string) => void lines.push({ level: 'warn', msg }), + }, + alertFatal: () => { + events.push('alert'); + if (options.alertThrows) throw new Error('SQLITE_BUSY'); + }, + shutdown: async () => { + events.push('shutdown'); + if (options.shutdownThrows) throw new Error('shutdown broke'); + }, + exit: (code: number) => void events.push(`exit ${code}`), + }; + return { deps, events, lines }; + } + + test('alerts first, then runs the graceful shutdown, then exits 1', async () => { + const { deps, events } = failureHarness(); + + await handleLaunchFailure( + new TelegramStartError('Telegram start failed permanently', { cause: new Error('x') }), + deps, + ); + + expect(events).toEqual(['alert', 'shutdown', 'exit 1']); + }); + + test('a launch error that did not come from the retry loop is logged fatal, one from it is not logged twice', async () => { + const stray = failureHarness(); + await handleLaunchFailure(new TypeError('The webhook adapter needs a GramIO Bot'), stray.deps); + expect(stray.lines.filter((l) => l.level === 'fatal')).toHaveLength(1); + + const fromLoop = failureHarness(); + await handleLaunchFailure(new TelegramStartError('Telegram start failed after 8 attempts'), fromLoop.deps); + expect(fromLoop.lines.filter((l) => l.level === 'fatal')).toHaveLength(0); + }); + + test('an alert that cannot be queued does not skip the shutdown or the exit', async () => { + const { deps, events, lines } = failureHarness({ alertThrows: true }); + + await handleLaunchFailure(new TelegramStartError('x'), deps); + + expect(events).toEqual(['alert', 'shutdown', 'exit 1']); + expect(lines.some((l) => l.level === 'warn')).toBe(true); + }); + + test('a shutdown that throws still ends in exit 1', async () => { + const { deps, events } = failureHarness({ shutdownThrows: true }); + + await handleLaunchFailure(new TelegramStartError('x'), deps); + + expect(events.at(-1)).toBe('exit 1'); + }); + + test('a non-Error rejection is wrapped, not lost', async () => { + const { deps, events } = failureHarness(); + + await handleLaunchFailure('plain string', deps); + + expect(events).toEqual(['alert', 'shutdown', 'exit 1']); + }); +}); diff --git a/test/config/env.test.ts b/test/config/env.test.ts index 73db0fe4..4c5c867f 100644 --- a/test/config/env.test.ts +++ b/test/config/env.test.ts @@ -2,6 +2,14 @@ import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; import { loadConfig } from '../../src/config/env.ts'; import type { ProviderId } from '../../src/services/ai/provider-ids.ts'; +import { redactSecrets, resetRegisteredSecrets } from '../../src/utils/redact-secrets.ts'; +import { + FAKE_ADMIN_ALERT_TOKEN, + FAKE_MTPROTO_API_HASH, + FAKE_REDIS_PASSWORD, + FAKE_SECOND_CLAUDE_TOKEN, + FAKE_TELEGRAM_TOKEN, +} from '../helpers/fake-secrets.ts'; describe('loadConfig', () => { const originalEnv = { ...process.env }; @@ -34,6 +42,53 @@ describe('loadConfig', () => { setAiVars(); }); + test('the configured bot token and ADMIN_ALERT_TOKEN are registered so logs and alerts never print them (#972)', () => { + resetRegisteredSecrets(); + process.env.BOT_TOKEN = FAKE_TELEGRAM_TOKEN; + process.env.ADMIN_ALERT_TOKEN = FAKE_ADMIN_ALERT_TOKEN; + process.env.MTPROTO_API_HASH = FAKE_MTPROTO_API_HASH; + process.env.CLAUDE_API_TOKEN2 = FAKE_SECOND_CLAUDE_TOKEN; + + loadConfig(); + + const line = redactSecrets(`token=${FAKE_ADMIN_ALERT_TOKEN} bot=${FAKE_TELEGRAM_TOKEN}`); + expect(line).not.toContain(FAKE_ADMIN_ALERT_TOKEN); + expect(line).not.toContain(FAKE_TELEGRAM_TOKEN); + const others = redactSecrets(`hash=${FAKE_MTPROTO_API_HASH} second=${FAKE_SECOND_CLAUDE_TOKEN}`); + expect(others).not.toContain(FAKE_MTPROTO_API_HASH); + expect(others).not.toContain(FAKE_SECOND_CLAUDE_TOKEN); + resetRegisteredSecrets(); + }); + + test('the password inside REDIS_URL is registered, so a logged connection error cannot print it', () => { + resetRegisteredSecrets(); + process.env.REDIS_URL = `redis://:${FAKE_REDIS_PASSWORD}@redis:6379`; + + loadConfig(); + + expect(redactSecrets(`connect ECONNREFUSED ${FAKE_REDIS_PASSWORD}`)).not.toContain(FAKE_REDIS_PASSWORD); + resetRegisteredSecrets(); + }); + + test('a password written percent-encoded in REDIS_URL is registered in that form too, as a quoted URL shows it', () => { + resetRegisteredSecrets(); + process.env.REDIS_URL = 'redis://:Pa%24%24-0123456789abcdef@redis:6379'; + + loadConfig(); + + expect(redactSecrets('connect redis://:Pa%24%24-0123456789abcdef@redis:6379')).not.toContain('0123456789abcdef'); + expect(redactSecrets('password Pa$$-0123456789abcdef')).not.toContain('0123456789abcdef'); + resetRegisteredSecrets(); + }); + + test('a REDIS_URL with no password or an escape that does not decode still loads', () => { + for (const url of ['redis://localhost:6379', 'redis://:bad%zzescape-0123456789@redis:6379']) { + process.env.REDIS_URL = url; + expect(() => loadConfig()).not.toThrow(); + } + resetRegisteredSecrets(); + }); + test('AI_TOOL_SCHEMA_MODE defaults to hybrid, accepts full and lazy as the rollback switch, rejects misspellings', () => { delete process.env.AI_TOOL_SCHEMA_MODE; expect(loadConfig().AI_TOOL_SCHEMA_MODE).toBe('hybrid'); diff --git a/test/database/repositories/alert.repository.test.ts b/test/database/repositories/alert.repository.test.ts index c4d411c0..e0927163 100644 --- a/test/database/repositories/alert.repository.test.ts +++ b/test/database/repositories/alert.repository.test.ts @@ -4,6 +4,7 @@ import { beforeEach, describe, expect, test } from 'bun:test'; import { migrations } from '../../../src/database/migrations'; import { AlertRepository } from '../../../src/database/repositories/alert.repository'; import { runMigrations } from '../../../src/database/schema'; +import { FAKE_GETME_URL, FAKE_TELEGRAM_TOKEN } from '../../helpers/fake-secrets.ts'; function createTestDb(): Database { const db = new Database(':memory:'); @@ -55,4 +56,13 @@ describe('AlertRepository', () => { expect(typeof alert!.created_at).toBe('string'); expect(alert!.created_at.length).toBeGreaterThan(0); }); + + test('push stores the text without a Telegram bot token, whoever produced the alert (#972)', () => { + repo.push(`Bot unhandled rejection: TypeError: request to ${FAKE_GETME_URL} failed`, 'bot-crash'); + + const alert = repo.pop(); + expect(alert!.text).not.toContain(FAKE_TELEGRAM_TOKEN); + expect(alert!.text).toContain('Bot unhandled rejection'); + expect(alert!.text).toContain('/getMe'); + }); }); diff --git a/test/helpers/fake-secrets.ts b/test/helpers/fake-secrets.ts new file mode 100644 index 00000000..ed48cdc6 --- /dev/null +++ b/test/helpers/fake-secrets.ts @@ -0,0 +1,32 @@ +// Synthetic credentials for redaction tests. Each value is assembled at run time so the source holds +// no literal that looks like a real token to a secret scanner, and none of them is a live credential. + +/** Bot id and token body shaped like a Telegram bot token: digits, a colon, 35 URL-safe characters starting with "A". */ +export const FAKE_TELEGRAM_BOT_ID = ['4', '2', '4', '2', '4', '2', '4', '2', '4'].join(''); +export const FAKE_TELEGRAM_SECRET = `A${'Zq9_-'.repeat(7)}`; +export const FAKE_TELEGRAM_TOKEN = `${FAKE_TELEGRAM_BOT_ID}:${FAKE_TELEGRAM_SECRET}`; + +/** An Anthropic API key and an Anthropic OAuth access token, synthetic bodies. */ +export const FAKE_ANTHROPIC_API_KEY = ['sk', 'ant', 'api03', 'Zq9Zq9Zq9Zq9Zq9Zq9Zq9_-Zq9'].join('-'); +export const FAKE_ANTHROPIC_OAUTH_TOKEN = ['sk', 'ant', 'oat01', 'Ab1Ab1Ab1Ab1Ab1Ab1Ab1_-Ab1'].join('-'); + +/** An opaque token with no recognisable prefix, like the operator-chosen ADMIN_ALERT_TOKEN. */ +export const FAKE_ADMIN_ALERT_TOKEN = 'opaque-admin-alert-value-7f3k9x2m4q'; + +/** The MTProto API hash: 32 hex characters with no prefix, and a second Claude pool key with no `sk-ant-` prefix. */ +export const FAKE_MTPROTO_API_HASH = 'c0ffee15badc0de5c0ffee15badc0de5'; +export const FAKE_SECOND_CLAUDE_TOKEN = 'second-pool-key-without-a-prefix-91xk2'; + +/** The password in `redis://:@host:6379`. */ +export const FAKE_REDIS_PASSWORD = 'redis-password-without-shape-5588aa'; + +/** The URL a failed getMe puts in `err.path`. */ +export const FAKE_GETME_URL = `https://api.telegram.org/bot${FAKE_TELEGRAM_TOKEN}/getMe`; + +/** The distinctive part of each secret: none of these strings may reach a log line or the alert queue. */ +export const SECRET_FRAGMENTS: readonly string[] = [ + FAKE_TELEGRAM_SECRET, + FAKE_ANTHROPIC_API_KEY, + FAKE_ANTHROPIC_OAUTH_TOKEN, + FAKE_ADMIN_ALERT_TOKEN, +]; diff --git a/test/services/incident/timeout-sites.test.ts b/test/services/incident/timeout-sites.test.ts index 0ee467e6..89421fab 100644 --- a/test/services/incident/timeout-sites.test.ts +++ b/test/services/incident/timeout-sites.test.ts @@ -105,6 +105,10 @@ const EXEMPT: { [site: string]: Exemption } = { "two waits at shutdown. The upload grace: a picture whose upload is still running ends unknown and raises the UncertainSend alert (reportUncertain). The shutdown note budget: deliberately NOT reported, it is best effort at exit (a stuck Telegram call only logs 'shutdown note not sent'); the whole shutdown deadline is already bounded and reported in bootstrap/shutdown.ts", }, // Timers that are not deadlines: nothing is cut off or abandoned when they fire. + 'src/bootstrap/launch.ts#launchBot': { + reason: + 'the backoff pause between two Telegram start attempts after a transient network failure; nothing is cut off or abandoned when it fires, and giving up is a fatal log plus the crash alert', + }, 'src/services/image/render-failure-reporter.ts#defaultSchedule': { reason: 'the end of the 10 min aggregation window of the render-failure alert; it sends what was held back', }, diff --git a/test/utils/logger-redaction.test.ts b/test/utils/logger-redaction.test.ts new file mode 100644 index 00000000..662def05 --- /dev/null +++ b/test/utils/logger-redaction.test.ts @@ -0,0 +1,118 @@ +// Every pino line passes through the secret redaction before it reaches the container log. The +// 2026-11-05 incident (#972) printed the bot token in `err.path` of a level-60 record; these tests +// log that exact error shape, and a real production-mode logger in a child process. +import { afterEach, describe, expect, test } from 'bun:test'; +import { createLogger } from '../../src/utils/logger.ts'; +import { registerSecretValues, resetRegisteredSecrets } from '../../src/utils/redact-secrets.ts'; +import { + FAKE_ADMIN_ALERT_TOKEN, + FAKE_ANTHROPIC_API_KEY, + FAKE_ANTHROPIC_OAUTH_TOKEN, + FAKE_GETME_URL, + FAKE_TELEGRAM_TOKEN, + SECRET_FRAGMENTS, +} from '../helpers/fake-secrets.ts'; + +afterEach(() => { + resetRegisteredSecrets(); +}); + +function capture() { + const lines: string[] = []; + const log = createLogger({ level: 'debug', destination: { write: (line: string) => void lines.push(line) } }); + return { log, lines }; +} + +function expectNoSecret(text: string): void { + expect(text).not.toContain(FAKE_TELEGRAM_TOKEN); + for (const fragment of SECRET_FRAGMENTS) expect(text).not.toContain(fragment); +} + +/** What Bun's fetch throws when Telegram resets the connection: the URL, token included, lands in `path`. */ +function resetError(): Error { + return Object.assign( + new TypeError( + 'ECONNRESET: The socket connection was closed unexpectedly. For more information, pass `verbose: true`', + ), + { code: 'ECONNRESET', path: FAKE_GETME_URL, errno: 0 }, + ); +} + +describe('createLogger redaction', () => { + test('a fetch reset logged as { err } keeps its code and URL but not the bot token in path', () => { + const { log, lines } = capture(); + + log.fatal({ err: resetError() }, 'Unhandled promise rejection'); + + const record = JSON.parse(lines.join('')); + expectNoSecret(lines.join('')); + expect(record.err.code).toBe('ECONNRESET'); + expect(record.err.path).toContain('https://api.telegram.org/bot'); + expect(record.err.path).toContain('/getMe'); + expect(record.msg).toBe('Unhandled promise rejection'); + }); + + test('a token in err.message, err.stack, a plain field, the message text and a child binding is redacted', () => { + const { log, lines } = capture(); + const failing = new Error(`request to ${FAKE_GETME_URL} failed`); + + log + .child({ module: 'bot', url: FAKE_GETME_URL }) + .error({ err: failing, params: { value: FAKE_TELEGRAM_TOKEN } }, `boom ${FAKE_GETME_URL}`); + + expect(lines).toHaveLength(1); + expectNoSecret(lines.join('')); + expect(JSON.parse(lines.join('')).module).toBe('bot'); + }); + + test('Anthropic keys, an OAuth bearer and a registered ADMIN_ALERT_TOKEN never reach the line', () => { + registerSecretValues([FAKE_ADMIN_ALERT_TOKEN]); + const { log, lines } = capture(); + const header = `Bearer ${FAKE_ANTHROPIC_OAUTH_TOKEN}`; + + log.error( + // The synthetic values below are logged on purpose: the test proves the logger removes them. + // codeql[js/clear-text-logging] fake credentials built in test/helpers/fake-secrets.ts, never live ones. + { + err: new Error(`401 invalid x-api-key ${FAKE_ANTHROPIC_API_KEY}`), + headers: { authorization: header, 'x-admin': FAKE_ADMIN_ALERT_TOKEN }, + }, + 'provider rejected the request', + ); + + expectNoSecret(lines.join('')); + }); + + test('a line with no secret keeps every field', () => { + const { log, lines } = capture(); + + log.info({ outcome: 'sent', turnMs: 1234 }, 'Image delivery outcome'); + + const record = JSON.parse(lines.join('')); + expect(record).toMatchObject({ outcome: 'sent', turnMs: 1234, msg: 'Image delivery outcome' }); + }); +}); + +describe('the production logger', () => { + test('a module-level logger in production mode prints the reset error without the token', async () => { + const script = ` + import { botLogger } from './src/utils/logger.ts'; + const err = Object.assign(new TypeError('ECONNRESET: closed'), { code: 'ECONNRESET', path: process.env.FAKE_URL }); + botLogger.fatal({ err }, 'Unhandled promise rejection'); + `; + const proc = Bun.spawn(['bun', '--no-env-file', '-e', script], { + cwd: `${import.meta.dir}/../..`, + env: { PATH: process.env.PATH ?? '', NODE_ENV: 'production', FAKE_URL: FAKE_GETME_URL }, + stdout: 'pipe', + stderr: 'pipe', + }); + const [stdout, stderr] = await Promise.all([new Response(proc.stdout).text(), new Response(proc.stderr).text()]); + await proc.exited; + + expectNoSecret(stdout + stderr); + const record = JSON.parse(stdout.trim().split('\n').at(-1) ?? '{}'); + expect(record.level).toBe(60); + expect(record.err.code).toBe('ECONNRESET'); + expect(record.err.path).toContain('/getMe'); + }); +}); diff --git a/test/utils/redact-secrets.test.ts b/test/utils/redact-secrets.test.ts new file mode 100644 index 00000000..1ef6f286 --- /dev/null +++ b/test/utils/redact-secrets.test.ts @@ -0,0 +1,176 @@ +import { afterEach, describe, expect, test } from 'bun:test'; +import { + installConsoleRedaction, + redactSecrets, + registerSecretValues, + resetRegisteredSecrets, +} from '../../src/utils/redact-secrets.ts'; +import { + FAKE_ADMIN_ALERT_TOKEN, + FAKE_ANTHROPIC_API_KEY, + FAKE_ANTHROPIC_OAUTH_TOKEN, + FAKE_GETME_URL, + FAKE_TELEGRAM_BOT_ID, + FAKE_TELEGRAM_SECRET, + FAKE_TELEGRAM_TOKEN, + SECRET_FRAGMENTS, +} from '../helpers/fake-secrets.ts'; + +afterEach(() => { + resetRegisteredSecrets(); +}); + +function expectNoSecret(text: string): void { + for (const fragment of SECRET_FRAGMENTS) expect(text).not.toContain(fragment); +} + +describe('redactSecrets: Telegram bot tokens', () => { + test('the token inside a request URL path is removed and the rest of the URL stays readable', () => { + const out = redactSecrets(`fetch failed: ${FAKE_GETME_URL}`); + expectNoSecret(out); + expect(out).toContain('https://api.telegram.org/bot'); + expect(out).toContain('/getMe'); + }); + + test('a token in an error message, a stack line and a JSON params blob is removed', () => { + const err = new Error(`request to ${FAKE_GETME_URL} failed`); + const json = JSON.stringify({ err: { message: err.message, stack: err.stack, path: FAKE_GETME_URL } }); + expectNoSecret(redactSecrets(json)); + expectNoSecret(redactSecrets(`at fetch (${FAKE_GETME_URL}:1:1)`)); + }); + + test('a URL-encoded colon does not hide the token', () => { + expectNoSecret( + redactSecrets(`https://api.telegram.org/bot${FAKE_TELEGRAM_BOT_ID}%3A${FAKE_TELEGRAM_SECRET}/getMe`), + ); + }); + + test('a bare token outside a URL is removed too', () => { + expectNoSecret(redactSecrets(`token=${FAKE_TELEGRAM_TOKEN}`)); + }); + + test('the same token twice in one string is removed both times', () => { + const out = redactSecrets(`${FAKE_GETME_URL} then ${FAKE_GETME_URL}`); + expectNoSecret(out); + }); + + test('times, durations and short ratios are not mistaken for a token', () => { + const text = 'started 15:16:52 boot 1:2 ratio 12345:6 slot 123456:abc'; + expect(redactSecrets(text)).toBe(text); + }); +}); + +describe('redactSecrets: other known secret shapes', () => { + test('an Anthropic API key and an Anthropic OAuth access token are removed', () => { + const out = redactSecrets(`invalid x-api-key ${FAKE_ANTHROPIC_API_KEY}; oauth ${FAKE_ANTHROPIC_OAUTH_TOKEN}`); + expectNoSecret(out); + expect(out).toContain('invalid x-api-key'); + }); + + test('an Authorization bearer value is removed whatever its shape', () => { + const out = redactSecrets(`headers: {"authorization":"Bearer ${FAKE_ADMIN_ALERT_TOKEN}"}`); + expectNoSecret(out); + expect(out).toContain('Bearer'); + }); + + test('a registered secret value is removed wherever it appears, with no recognisable prefix', () => { + registerSecretValues([FAKE_ADMIN_ALERT_TOKEN]); + const out = redactSecrets(`POST /admin/alerts?t=${FAKE_ADMIN_ALERT_TOKEN} failed`); + expectNoSecret(out); + expect(out).toContain('POST /admin/alerts'); + }); + + test('an unregistered opaque value is left alone, a short or empty registration changes nothing', () => { + registerSecretValues([undefined, '', 'short']); + const text = `value ${FAKE_ADMIN_ALERT_TOKEN} and short and more`; + expect(redactSecrets(text)).toBe(text); + }); +}); + +describe('redactSecrets: registered values beat the shape patterns', () => { + const ODD_SECRET = 'Xk2Lm9Pq4Rt7Wb3N#tail-after-a-hash-9z8y7x'; + + test('a registered value with characters outside the bearer alphabet is removed whole, not cut at the first odd character', () => { + registerSecretValues([ODD_SECRET]); + + const out = redactSecrets(`authorization: Bearer ${ODD_SECRET}`); + + expect(out).not.toContain('tail-after-a-hash'); + expect(out).not.toContain('Xk2Lm9Pq4Rt7Wb3N'); + }); + + test('a registered value that pino writes JSON-escaped (a quote or backslash inside) is removed from the line', () => { + const quoted = 'pass"word\\with-quote-and-backslash-4411'; + registerSecretValues([quoted]); + + const line = JSON.stringify({ msg: `failed with ${quoted}` }); + + expect(line).not.toContain(quoted); + expect(redactSecrets(line)).not.toContain('with-quote-and-backslash'); + }); + + test('when one registered value is a prefix of another the longer one is removed whole', () => { + registerSecretValues(['shared-prefix-0123456789', 'shared-prefix-0123456789-and-a-long-tail']); + + expect(redactSecrets('x shared-prefix-0123456789-and-a-long-tail y')).not.toContain('long-tail'); + }); +}); + +describe('redactSecrets: behaviour', () => { + test('text without a secret is returned unchanged', () => { + const text = 'Image delivery outcome {"outcome":"sent","turnMs":1234}'; + expect(redactSecrets(text)).toBe(text); + }); + + test('redaction is idempotent', () => { + const once = redactSecrets(`${FAKE_GETME_URL} ${FAKE_ANTHROPIC_API_KEY}`); + expect(redactSecrets(once)).toBe(once); + }); +}); + +describe('installConsoleRedaction', () => { + function recordingConsole(): { target: Console; lines: string[] } { + const lines: string[] = []; + const record = (...args: unknown[]): void => { + lines.push(args.map((a) => (typeof a === 'string' ? a : Bun.inspect(a))).join(' ')); + }; + const target = { log: record, info: record, warn: record, error: record, debug: record } as unknown as Console; + return { target, lines }; + } + + test('a string, an error carrying the token in its path, and a plain object are all redacted', () => { + const { target, lines } = recordingConsole(); + installConsoleRedaction(target); + const fetchError = Object.assign(new TypeError('ECONNRESET: The socket connection was closed unexpectedly'), { + code: 'ECONNRESET', + path: FAKE_GETME_URL, + }); + + target.error('Error received when fetching updates', fetchError); + target.warn(`poll ${FAKE_GETME_URL}`); + target.log({ url: FAKE_GETME_URL }); + + expect(lines).toHaveLength(3); + for (const line of lines) expectNoSecret(line); + expect(lines[0]).toContain('ECONNRESET'); + }); + + test('installing twice does not wrap twice', () => { + const { target, lines } = recordingConsole(); + installConsoleRedaction(target); + const once = target.error; + installConsoleRedaction(target); + expect(target.error).toBe(once); + target.error('plain'); + expect(lines).toEqual(['plain']); + }); + + test('an argument without a secret is passed through as the very same value', () => { + const seen: unknown[] = []; + const target = { log: (...args: unknown[]) => seen.push(...args) } as unknown as Console; + installConsoleRedaction(target); + const payload = { turnMs: 12 }; + target.log(payload); + expect(seen[0]).toBe(payload); + }); +}); diff --git a/uv.lock b/uv.lock index 5895ba5d..755e9fc9 100644 --- a/uv.lock +++ b/uv.lock @@ -335,7 +335,7 @@ wheels = [ [[package]] name = "hypercalendarbot" -version = "0.3.6" +version = "0.3.7" source = { virtual = "." } dependencies = [ { name = "numpy" },