Reintentar el login para que un fallo de DNS no mate al bot - #34
Merged
Merged
Conversation
On 2026-09-11 at 06:18 UTC an unattended-upgrade replaced glibc — which *is*
the DNS resolver — and needrestart restarted the bot in the middle of that
window. `getaddrinfo("discord.com")` returned EAI_AGAIN, the error propagated
out of `main()`, and the process exited 1. Two starts died that way before a
third one landed; each one sent an OnFailure alert to Discord.
`Client.start()` is `login()` followed by `connect()`, and only the first half
was exposed. `connect(reconnect=True)` already catches `aiohttp.ClientError`,
`OSError`, `GatewayNotFound` and friends and retries with its own backoff, so a
gateway that drops mid-song recovers on its own. `login()` has no retry at all,
which is why a few seconds without a resolver was fatal.
So `main()` now calls the two halves separately: the login is retried with
backoff (5s, 15s, 30s, 60s, 120s), and the gateway is handed to discord.py
untouched rather than wrapped in a second retry policy.
Only failures that waiting can fix are retried. A `LoginFailure` or a
`PrivilegedIntentsRequired` is a deployment mistake that will fail identically
forever, and retrying it invites a rate-limit on the login endpoint on top; a
4xx is ours and stays ours. DNS, refused connections, timeouts and Discord's
own 5xx are retried. After the delays run out the last error is re-raised, so a
lasting outage still reaches systemd instead of leaving a bot that looks alive
to `systemctl` and never trips OnFailure.
Retrying the login is not as simple as calling it twice, and one of the tests
here found out why. `HTTPClient.static_login` builds a new `aiohttp.ClientSession`
on every call and abandons the previous one, so a naive retry leaks a session
per attempt. Closing it is not enough either: the session is handed the
HTTPClient's connector with aiohttp's default `connector_owner=True`, so closing
the session closes that shared connector — and because `ClientSession.closed`
reports the connector's state, every later session built on it is born closed
and the next attempt dies with "Session is closed" instead of retrying. The
connector is cleared alongside the session so a fresh pair is built.
That bug was invisible to the tests using a fake HTTP client; it only appeared
in the one that drives a real `discord.Client` with real aiohttp sessions
against a resolver patched to fail. That test also asserts the invariant that
matters on a 768 MB box: no aiohttp session is left open behind the retries.
Verified end to end against the real API: with DNS made to fail twice and then
recover, the login retries and gets through, and an invalid token is then
rejected on the first attempt without a retry.
Tests: 319 passing, up from 293.
Closes #33
This was referenced Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cierra #33.
El bot murió el 2026-09-11 a las 06:18 UTC con
ClientConnectorDNSErrorydisparó la alerta de Discord.
unattended-upgradesreemplazó glibc — quees el resolver de DNS — y
needrestartreinició el bot dentro de esa ventana.getaddrinfo("discord.com")devolvióEAI_AGAIN, la excepción salió demain()y el proceso terminó con 1. Dos arranques murieron así antes de que eltercero entrara.
319 tests pasando (antes 293). +406 líneas, −1.
Dónde estaba realmente el agujero
Client.start()eslogin()seguido deconnect(). Leyendo discord.py, solola primera mitad estaba expuesta:
connect(reconnect=True)ya es resiliente: su bucle capturaaiohttp.ClientError,OSError,HTTPException,GatewayNotFound,ConnectionClosedyasyncio.TimeoutError, y reintenta con backoff propio. Ungateway que se cae a mitad de canción ya se recuperaba solo.
login()no tiene reintento ninguno. Por eso unos segundos sin resolvereran fatales, y por eso el traceback apuntaba a
static_login→/users/@me.Así que
main()llama a las dos mitades por separado: el login se reintenta, yel gateway se entrega a discord.py sin envolver — apilar nuestra política de
reintentos sobre la suya duplicaría cada espera.
Qué se reintenta y qué no
ClientConnectorDNSError),OSError, timeouts,GatewayNotFound, y los 5xx de DiscordLoginFailure,PrivilegedIntentsRequired, los 4xx,TypeErrorUn token equivocado o un intent que falta es un error de despliegue: va a fallar
igual para siempre, y machacar el endpoint de login encima invita a un
rate-limit. Un 4xx es culpa nuestra y lo sigue siendo.
Backoff de 5s, 15s, 30s, 60s, 120s. Agotados los intentos se relanza el último
error, a propósito: systemd sigue siendo la red de seguridad. Reintentar para
siempre dejaría un bot que
systemctlve vivo, que nunca funciona y que nuncadispara
OnFailure. Un test fija ese presupuesto entre 60 y 600 segundos.El bug que encontró el test de integración
Reintentar el login no es llamarlo dos veces, y uno de los tests lo demostró.
HTTPClient.static_logincrea unaaiohttp.ClientSessionnueva en cadallamada y abandona la anterior. Un reintento ingenuo filtra una sesión por
intento — justo el tipo de fuga lenta que el brief prohíbe en una caja de
768 MB.
Pero cerrarla tampoco basta. A la sesión se le pasa el connector del HTTPClient,
y el
connector_owner=Truepor defecto de aiohttp significa que cerrar lasesión cierra el connector compartido. Como
ClientSession.closedconsultael estado del connector:
...toda sesión posterior construida sobre él nace cerrada, y el siguiente
intento muere con
RuntimeError: Session is closeden vez de reintentar. Selimpia el connector junto con la sesión para que se construya un par nuevo.
Ese bug era invisible para los tests con un HTTP falso. Solo apareció en el
que maneja un
discord.Clientreal con sesiones aiohttp reales contra unresolver parcheado para fallar, y que además afirma el invariante que importa:
ninguna sesión aiohttp queda abierta tras los reintentos.
Cómo probarlo
Verificación de extremo a extremo ya hecha contra la API real — DNS forzado a
fallar dos veces y luego restaurado:
Reproducir el incidente entero en ilserver4, tras desplegar:
Lo que no se toca
La unidad de systemd. Un
After=nss-lookup.targetno habría evitado esto:quien reinició el servicio fue
needrestarta mitad del upgrade, con la máquinaya arrancada hacía horas.
Restart=on-failuresigue siendo el respaldo paratodo lo que no sea el login.