Skip to content

Don't hold a DashMap shard lock across the database await in the memory sweep - #106

Merged
genusistimelord merged 1 commit into
AscendingCreations:mainfrom
sfried:fix/sweep-shard-lock-across-await
Oct 29, 2026
Merged

genusistimelord merged 1 commit into
AscendingCreations:mainfrom
sfried:fix/sweep-shard-lock-across-await

Conversation

@sfried

@sfried sfried commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Problem

handler::runner's memory sweep awaits store_session inside a DashMap iterator:

for session in session_store.inner.iter().filter(|r| r.autoremove < current_time) {
    if !session.expired() {
        if let Err(err) = session_store.store_session(&session).await {

The iterator holds the current shard's read lock across the database write. While that future is suspended, a request whose session id hashes to the same shard reaches SessionService, which calls inner.insert / get_mut / remove (service.rs) — a synchronous write lock on that shard — and blocks its tokio worker thread. If that worker is the one driving the I/O reactor, the database write the sweep is waiting on can never complete, the read lock is never released, and the whole server stops: 0% CPU, every thread parked in a futex, the listen backlog growing, database connections idle.

We hit this three times under sustained test traffic (many logins, so many in-memory sessions become due for unloading while requests keep arriving) on 0.21/0.22 with axum_session_sqlx. A core dump showed one runtime worker in dashmap::lock::RawRwLock::lock_exclusive_slow under the SessionService future and every other worker parked.

Fix

The sweep's save step moves into save_due_sessions, which clones the due, unexpired sessions out of the map first and awaits each store_session after the iterator is dropped. The filter, the stop-at-first-error behaviour and the debug logs are unchanged.

Test

handler::tests::save_due_sessions_does_not_hold_a_shard_lock_across_the_store_await runs the save step on a one-worker multi-thread runtime with a slow DatabasePool (its store awaits a timer), while a second task writes the very session being saved; a watchdog on a separate std thread turns a hang into a failure. With the old loop it times out (5 s); with the fix it passes in ~0.2 s.

cargo fmt --check is clean; cargo clippy -p axum_session --features key-store --all-targets reports only the existing aes_gcm deprecation warnings.

Unrelated, noticed while testing: on current main, --features advanced doesn't compile (Utc is not in scope at session_store.rs:523,539).

🤖 Generated with Claude Code

…ry sweep

The memory-unload sweep in `runner` iterated `session_store.inner` and awaited
`store_session` inside the loop, so the iterator's shard read lock was held across
the database write. While that future was suspended, any request whose session
hashed to the same shard called `inner.insert` / `get_mut` / `remove` in
`SessionService`, which takes the shard's write lock synchronously and blocks the
tokio worker thread. When that worker was the one driving the I/O reactor, the
database write could never complete, the read lock was never released, and the
whole server stopped (0% CPU, every thread parked, listen backlog growing).

The sweep's save step is now `save_due_sessions`: it clones the due, unexpired
sessions out of the map first and awaits each save after the iterator is gone.
The filter, the stop-at-first-error behaviour and the log lines are unchanged.

The new test runs the save step on a one-worker runtime while another task writes
the session being saved, with a watchdog thread: it times out on the old loop and
passes in ~0.2 s with the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@genusistimelord

Copy link
Copy Markdown
Member

Very good find. The aes_gcm issue is more of idk how to do a similar style with the newer changes yet. I don't want to enforce a database sweep if the changes break all the encrypted sessions.

@genusistimelord
genusistimelord merged commit 77ab266 into AscendingCreations:main Oct 29, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants