fix(#114): wait for async touch/eviction goroutines before closing badger DB#115
Merged
Conversation
…dger DB Store.Get bumped LRU asynchronously via 'go s.touch(c)', and Store.Put spawned 'go s.evictToTarget()' on over-cap. Neither goroutine was tracked, so Store.Close could return while they were mid-transaction against the underlying badger DB. Once db.Close() finished, the async goroutines panicked with a nil-deref inside badger's memTable traversal (getMemTables -> memTable.IncrRef). Fix: track background goroutines with sync.WaitGroup. spawnBG() gates Add() under a lifecycle mutex so a spawn racing Close is either counted (Close waits for it) or rejected (spawnBG returns nil). Close flips closed under the same mutex, then bg.Wait()s outside the lock, then closes the DB. Both async paths (touch and evictToTarget) use spawnBG uniformly. Adds TestCloseWaitsForAsyncTouchAndEviction, 32 iterations of the racy Get+Put+Close pattern the original test suite tripped on. Reproduces the pre-fix panic ~5/5 without -race; passes 10/10 with the fix. Closes #114.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #114.
What
state/cache.Store.Getbumps LRU asynchronously viago s.touch(c);Store.Putfiresgo s.evictToTarget()when live bytes exceed the soft cap. Neither goroutine was tracked, soStore.Closecould return while one of them was mid-transaction against the underlying Badger DB. Oncedb.Close()finished, the async goroutine panicked with a nil-deref inside Badger's memTable traversal (getMemTables → memTable.IncrRef, badgerdb.go:731).This has been latent on
mainsince the persistent block cache landed.go test -count=1 ./state/cache/reliably reproduces the panic without-race. With-racethe goroutine scheduling makes it flakier but the underlying race still exists.Fix
Track background goroutines with
sync.WaitGroup. NewspawnBG()gates thebg.Add(1)under a lifecycle mutex so a spawn racingCloseis either counted (Close waits for it) or rejected (spawnBG returns nil).Closeflipsclosedunder the same mutex, thenbg.Wait()s outside the lock, then closes the DB. Both async paths (touch and evictToTarget) usespawnBGuniformly.Costs: one short mutex acquisition per Get (already atomic-checked before) and per over-cap Put. Happy-path behavior is unchanged.
Regression test
TestCloseWaitsForAsyncTouchAndEvictionruns 32 iterations of a Get+Put+Close hammer: seed a CID, fan out 24 concurrent Gets + 24 concurrent Puts (each triggering the async LRU/eviction spawn paths at tiny soft cap), then Close. Reproduces the pre-fix panic ~5/5 without-race; passes 10/10 with the fix.Evidence
CGO_ENABLED=0 go build ./...— cleanCGO_ENABLED=0 go vet ./...— cleango test -count=1 -timeout 300s ./...— all 55 packages ok, includingstate/cachein 3.7sgo test -race -count=1 ./state/cache/— 3/3 okgo test -count=1 ./state/cache/on pre-fix main): panicked 3/3 in local repro before the fix; 10/10 clean after.Non-scope
evicting atomic.Boolfield inStoreis still dead code; leaving as-is (unrelated cleanup).Getfast path,Putfast path, or eviction pass itself.