Repository navigation
Don't hold a DashMap shard lock across the database await in the memory sweep - #106
Merged
genusistimelord merged 1 commit intoOct 29, 2026
Conversation
…ry sweep The memory-unload sweep in `runner` iterated `session_store.inner` and awaited `store_session` inside the loop, so the iterator's shard read lock was held across the database write. While that future was suspended, any request whose session hashed to the same shard called `inner.insert` / `get_mut` / `remove` in `SessionService`, which takes the shard's write lock synchronously and blocks the tokio worker thread. When that worker was the one driving the I/O reactor, the database write could never complete, the read lock was never released, and the whole server stopped (0% CPU, every thread parked, listen backlog growing). The sweep's save step is now `save_due_sessions`: it clones the due, unexpired sessions out of the map first and awaits each save after the iterator is gone. The filter, the stop-at-first-error behaviour and the log lines are unchanged. The new test runs the save step on a one-worker runtime while another task writes the session being saved, with a watchdog thread: it times out on the old loop and passes in ~0.2 s with the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
|
Very good find. The aes_gcm issue is more of idk how to do a similar style with the newer changes yet. I don't want to enforce a database sweep if the changes break all the encrypted sessions. |
genusistimelord
merged commit Oct 29, 2026
77ab266
into
AscendingCreations:main
2 of 3 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
handler::runner's memory sweep awaitsstore_sessioninside aDashMapiterator:The iterator holds the current shard's read lock across the database write. While that future is suspended, a request whose session id hashes to the same shard reaches
SessionService, which callsinner.insert/get_mut/remove(service.rs) — a synchronous write lock on that shard — and blocks its tokio worker thread. If that worker is the one driving the I/O reactor, the database write the sweep is waiting on can never complete, the read lock is never released, and the whole server stops: 0% CPU, every thread parked in a futex, the listen backlog growing, database connections idle.We hit this three times under sustained test traffic (many logins, so many in-memory sessions become due for unloading while requests keep arriving) on 0.21/0.22 with
axum_session_sqlx. A core dump showed one runtime worker indashmap::lock::RawRwLock::lock_exclusive_slowunder theSessionServicefuture and every other worker parked.Fix
The sweep's save step moves into
save_due_sessions, which clones the due, unexpired sessions out of the map first and awaits eachstore_sessionafter the iterator is dropped. The filter, the stop-at-first-error behaviour and the debug logs are unchanged.Test
handler::tests::save_due_sessions_does_not_hold_a_shard_lock_across_the_store_awaitruns the save step on a one-worker multi-thread runtime with a slowDatabasePool(itsstoreawaits a timer), while a second task writes the very session being saved; a watchdog on a separate std thread turns a hang into a failure. With the old loop it times out (5 s); with the fix it passes in ~0.2 s.cargo fmt --checkis clean;cargo clippy -p axum_session --features key-store --all-targetsreports only the existingaes_gcmdeprecation warnings.Unrelated, noticed while testing: on current
main,--features advanceddoesn't compile (Utcis not in scope atsession_store.rs:523,539).🤖 Generated with Claude Code