Conversation
…op run_until_complete tg_client.api_client.loop is the same loop object python-telegram-bot's Application runs via run_forever() on the main thread (both call asyncio.get_event_loop() there). Calling run_until_complete on that loop from executor/scheduler threads only "worked" because nest_asyncio patches away the reentrancy guard, which meant concurrent calls actually raced on the loop's internal ready/scheduled/selector state with no synchronization -- a plausible mechanism for the total, crash-free freezes we've seen. Telethon calls now run as real async/await coroutines bridged onto that same loop via the actual thread-safe primitive for it, with explicit per-call timeouts. Also drops get_main_chat_users/_get_chat_users (dead code, no callers).
Same underlying issue as the tg_client fix: sender.py fetched/patched a loop with nest_asyncio on every send from whatever thread called it, instead of running as a coroutine on PTB's actual loop. Rewritten to bridge into that loop via run_coroutine_threadsafe with explicit timeouts on every outbound call, same as tg_client. bot.py now captures that loop once and passes it to TelegramSender. Fixes two pre-existing bugs surfaced by the rewrite: - self.bot.send_photo(...) was called without await, so .png attachments were silently never sent (coroutine created, never scheduled). - The error-fallback path caught telegram.error.TelegramError, but the main send path raised requests.exceptions.HTTPError -- a different exception hierarchy entirely, so the retry-as-plain-text-on-parse-error logic could never actually fire. Now everything goes through self.bot, so the exception types line up and the fallback works.
send_important_event() (and the AppContext-init-failure error log) run during app.py's startup, before run_polling() has ever called run_forever() on the shared loop. run_coroutine_threadsafe schedules work onto a loop but doesn't run it -- with nothing pumping the loop yet and the only thread that could ever do so blocked on future.result(), every startup send deadlocked until the 60s timeout, crash-looping the bot. Both TelegramSender and TgClient now check loop.is_running() first: if running, bridge via run_coroutine_threadsafe as before; if not (we're on the same thread that owns the loop and nothing else is using it yet), run the coroutine directly instead of scheduling it into a void.
logger.usage()/logger.error() calls made synchronously from code already running as a coroutine on the shared loop (e.g. bot.py's add_usage_logging wrapper, which runs directly on PTB's loop rather than through the handler executor) flow through ErrorBroadcastHandler into TelegramSender, which then tried to block-wait on a coroutine scheduled onto that same loop. The loop can't advance to run it while the very callback that would need to keep it moving is stuck waiting - a self-deadlock, only ever broken by the outer timeout. Confirmed live: a local run's /help usage log hung for exactly SEND_BRIDGE_TIMEOUT_SEC before giving up. Both bridges now check asyncio.get_running_loop() against the target loop first. TelegramSender schedules a fire-and-forget task in that case (its return value isn't consumed by any caller). TgClient raises instead, since its callers do consume the result and a silent wrong-shaped fallback would just crash later with a confusing error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Prod (
sysblokbot-prod, built frommaster) has been intermittently freezing for 75-155 seconds at a time, several times a day - no Telegram replies, no scheduled jobs - matching a real incident where the bot was reported as "really slow to reply".Root cause: on
master,TelegramSender.send_to_chat_idsends via a barerequests.get(...)with no timeout, and all handler/job code runs synchronously inline on the bot's single asyncio event loop (via anest_asynciohack) rather than on a worker thread. Any single slow outbound call (Telegram API, Google Sheets, the WordPress health-check request) blocks that one shared loop, freezing the entire bot until the call returns.What this brings in
b7654fefix: run sync handlers on a thread pool instead of the event loopeecd47ffix: rewrite TelegramSender on native async/await, drop nest_asyncio180b25cfix: drive Telethon via run_coroutine_threadsafe instead of shared-loop run_until_complete37c003b/897ab1cfix: two deadlock edge cases in the new bridging code4755ecefix: disable auto_reconnect in tg clientdevfixes/chores accumulated since the last sync (Google API discovery fixes, uv migration, FB analytics fixes)Note on CI
The
Unit testsworkflow has been red onmastersince 2026-05-25 (afreeze_time-based scheduler test,test_scheduler.py::test_jobs_executed), well before any of the above commits existed - this is pre-existing and not a regression from this change.Test plan
sysblokbot-prodviadoppler run -- make deploy SERVICE=sysblokbot/shrug,/startand other commands reply promptly during aSendRemindersJob/SiteHealthCheckJob/SheetReportJobrun