Repository navigation
Faster request counts, cancel abandoned page reads, and a Background work setting - #85
Open
TheMightyTRex wants to merge 2 commits into
Open
TheMightyTRex wants to merge 2 commits into
TheMightyTRex wants to merge 2 commits into
Conversation
Leaving a page while a slow request is still running kept its database query busy until it finished, and moving back and forth piled the queries up. Reads (GET and HEAD) on the requests, following and library pages are now cancelled when the browser disconnects before the response starts; the database driver then cancels the running query. Writes, reads that call outside services, and clean-up after a response always run to the end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Request counts and background jobs looked operations up by kind and status, and by the request id inside the command payload, with no index for either, so each lookup read the whole operations table. On a library with 52,000 operations the request counts took about 4 minutes; with the two indexes in migration 0081 they take under a second. JIT is turned off for the app's connections, because compiling the large status queries cost more than it saved (17 seconds on one count). Settings -> Background work chooses how often scheduled jobs do their work: Normal (unchanged), Relaxed (x5), Light (x20) or Minimal (x50). Each job keeps its schedule and skips the runs that are not due at the chosen pace, so the setting applies within a minute without a restart. Migration 0082 stores it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
On my server (about 1,500 requests, 52,000 operations, a modest home server), Dewarr used a lot of CPU and database time while idle, and some pages were very slow:
payload['command'], and by kind and status. Neither had an index, so every lookup read the whole operations table: 2.1 million full scans since the database started. Repeated polling used up every database connection, and Dewarr stopped responding.The changes
Two indexes on
operations(migration0081_operation_lookups):(kind, status, created_at)((payload['command'] ->> 'intent_id'), kind, created_at), written with subscripts exactly as the app's queries compile, or Postgres ignores it.On a copy of my database, the request counts went from 4 min 2 s to 0.26 s, with identical results.
JIT off for the app's database connections (
-cjit=offin the connect options).Abandoned page reads are cancelled. A small ASGI middleware cancels GET and HEAD requests on the requests, following and library pages when the browser disconnects before the response starts. Cancelling the task makes psycopg cancel the running query. It never cancels writes, reads that call outside services (MAM, Prowlarr, Hardcover), or clean-up after a response has started.
Settings → Background work (admins): Normal, Relaxed (×5), Light (×20) or Minimal (×50).
background_work_settingstable (migration0082_background_work), likeonboarding_settings.Database changes and how to undo them
0081_operation_lookupsoperations0082_background_workbackground_work_settingstableBoth run on start like every Dewarr migration, and nobody needs to do anything after updating. To go back:
alembic downgrade 0080_member_setup. Both directions are tested, including on a real Dewarr v0.3.9 install.Tests
tests/unit/test_background_pace.py,tests/integration/test_background_work.py(API, admin only, both migrations down and up, the planner uses the new index),tests/unit/test_disconnect.py,apps/web/tests/ui/background-work.spec.ts.downloader-simpletimed out once while the server suite ran alongside, then passed twice alone).alembic downgrade 0080_member_setup→ Dewarr v0.3.9 starts with no errors.🤖 Generated with Claude Code