Skip to content

fix(#114): wait for async touch/eviction goroutines before closing badger DB#115

Merged
Reiers merged 1 commit into
mainfrom
fix/114-cache-close-race
Jul 11, 2026
Merged

fix(#114): wait for async touch/eviction goroutines before closing badger DB#115
Reiers merged 1 commit into
mainfrom
fix/114-cache-close-race

Conversation

@Reiers

@Reiers Reiers commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Closes #114.

What

state/cache.Store.Get bumps LRU asynchronously via go s.touch(c); Store.Put fires go s.evictToTarget() when live bytes exceed the soft cap. Neither goroutine was tracked, so Store.Close could return while one of them was mid-transaction against the underlying Badger DB. Once db.Close() finished, the async goroutine panicked with a nil-deref inside Badger's memTable traversal (getMemTables → memTable.IncrRef, badger db.go:731).

This has been latent on main since the persistent block cache landed. go test -count=1 ./state/cache/ reliably reproduces the panic without -race. With -race the goroutine scheduling makes it flakier but the underlying race still exists.

Fix

Track background goroutines with sync.WaitGroup. New spawnBG() gates the bg.Add(1) under a lifecycle mutex so a spawn racing Close is either counted (Close waits for it) or rejected (spawnBG returns nil). Close flips closed under the same mutex, then bg.Wait()s outside the lock, then closes the DB. Both async paths (touch and evictToTarget) use spawnBG uniformly.

Costs: one short mutex acquisition per Get (already atomic-checked before) and per over-cap Put. Happy-path behavior is unchanged.

Regression test

TestCloseWaitsForAsyncTouchAndEviction runs 32 iterations of a Get+Put+Close hammer: seed a CID, fan out 24 concurrent Gets + 24 concurrent Puts (each triggering the async LRU/eviction spawn paths at tiny soft cap), then Close. Reproduces the pre-fix panic ~5/5 without -race; passes 10/10 with the fix.

Evidence

  • CGO_ENABLED=0 go build ./... — clean
  • CGO_ENABLED=0 go vet ./... — clean
  • go test -count=1 -timeout 300s ./... — all 55 packages ok, including state/cache in 3.7s
  • go test -race -count=1 ./state/cache/ — 3/3 ok
  • Old flake (go test -count=1 ./state/cache/ on pre-fix main): panicked 3/3 in local repro before the fix; 10/10 clean after.

Non-scope

  • The unused evicting atomic.Bool field in Store is still dead code; leaving as-is (unrelated cleanup).
  • No behavior change on the Get fast path, Put fast path, or eviction pass itself.

…dger DB

Store.Get bumped LRU asynchronously via 'go s.touch(c)', and Store.Put
spawned 'go s.evictToTarget()' on over-cap. Neither goroutine was
tracked, so Store.Close could return while they were mid-transaction
against the underlying badger DB. Once db.Close() finished, the async
goroutines panicked with a nil-deref inside badger's memTable
traversal (getMemTables -> memTable.IncrRef).

Fix: track background goroutines with sync.WaitGroup. spawnBG() gates
Add() under a lifecycle mutex so a spawn racing Close is either
counted (Close waits for it) or rejected (spawnBG returns nil).
Close flips closed under the same mutex, then bg.Wait()s outside
the lock, then closes the DB. Both async paths (touch and
evictToTarget) use spawnBG uniformly.

Adds TestCloseWaitsForAsyncTouchAndEviction, 32 iterations of the
racy Get+Put+Close pattern the original test suite tripped on.
Reproduces the pre-fix panic ~5/5 without -race; passes 10/10 with
the fix.

Closes #114.
@Reiers Reiers merged commit 4ef3130 into main Jul 11, 2026
1 check passed
@Reiers Reiers deleted the fix/114-cache-close-race branch July 11, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

state/cache: nil-deref in async touch/evict goroutines outliving Store.Close

1 participant