Repository navigation
rpcv2: wait for the hot DB destroys before the lifecycle e2e shuts down - #1008
Merged
Merged
Conversation
TestE2E_DaemonLifecycle_FirstStartIngestFreezeLookupRestartPrune waited for the discard counter, which rises when a chunk is demoted, and then cancelled the daemon and asserted that the chunks' hot keys were gone. The key goes with the deferred destroy at the end of the run, after the grace wait, and a shutdown during that wait skips the destroy by design, so the assertion raced the tick and failed once in a while under -race. The test now also waits for both hot DB directories to be removed before shutting down, the proof the prune step already uses for its index file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M23ZvEkyrFjyUUbm7zLwob
There was a problem hiding this comment.
Pull request overview
This PR fixes a timing race in the lifecycle E2E test by waiting for deferred hot-chunk destruction before shutdown.
Changes:
- Waits for both hot DB directories to be removed.
- Prevents cancellation during deferred destruction.
Note
Copilot is running an experiment and ran this review at Lite.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
leighmcculloch
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TestE2E_DaemonLifecycle_FirstStartIngestFreezeLookupRestartPrunefails now and then under-raceone2e_test.go:374, "chunk 00000001 hot key is discarded". It passed on rerun on #968 and fails about one run in four locally on the base branch with the race detector, so it is a timing race in the test, not a regression.The race. The test waits for the discard counter to reach two and then cancels the daemon. That counter rises when the lifecycle demotes a hot chunk. The hot key it then asserts on is removed by the deferred destroy at the end of the same tick, after the grace wait, and a cancellation during that wait skips the destroy by design and leaves the demoted key for the next run. The test's cancel lands at a random point up to 50 ms after the counter moved, so when it falls inside the prune scan or the grace wait, the assertion sees the key still there.
The fix. Before shutting down, the test also waits for both chunks' hot DB directories to be removed, which is what the destroy does. This is the same proof the test's prune step already uses for its index file. No production code changes.
Checked. Six consecutive
-raceruns of the test pass on this branch; on the base branch the same loop fails about one run in four.🤖 Generated with Claude Code