Repository navigation
Fix JsonMapStore Small→Medium transition resurrecting deleted keys - #1
Closed
mrcompator wants to merge 2 commits into
Closed
mrcompator wants to merge 2 commits into
mrcompator wants to merge 2 commits into
Conversation
When a JsonMapStore grows from the Small to the Medium storage model, both the commit() and retainTx() transition paths rewrite in-memory history values to the OffHeap marker. They did this unconditionally -- including for Deleted tombstones. Because OffHeap != Deleted, every key that had been removed became visible to keysAt() again, while sizeAt() (which tracks sizeByTx) still correctly excluded it. keysAt()'s `assert keys.size == size` then fails, and because the in-memory history stays corrupted the failure is sticky and cascades to every later read of that map. Guard both conversions with a `!= Deleted` check so tombstones are left intact. Adds a deterministic reproducer (JsonMapStoreTest.shouldKeepSizeAndKeysConsistentUnderConcurrentTransactions): model thresholds are injected low so the store crosses Small->Medium mid-run, then several Clients drive overlapping insert/delete/rollback transactions on one map while a reader checks size vs the iterated key count in a read transaction. Fails as `keys.size=520, size=412` before the fix; passes after. Full jsondb_test stays green (55/55).
mrcompator
force-pushed
the
fix/jsondb-medium-model-tombstone-resurrection
branch
from
August 28, 2026 14:22
348137d to
b035c88
Compare
Owner
Author
|
Superseded — re-opening against xtclang/xvm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
A
JsonMapStoreholds its data in memory in one of three "storage models" — Small, Medium, Large — and grows between them as the map gets bigger. When it grows Small → Medium, it moves the real values out of memory and leaves anOffHeapmarker in the history in their place.Both places that perform this conversion —
commit()(for the batch that triggered the growth) andretainTx()(for the rest of the history, during maintenance) — rewrote every history entry toOffHeapwithout checking whether the entry was aDeletedtombstone.Because
OffHeap != Deleted, this made every key that had ever beenremove()d look present again:keysAt()walks the history, so it counted those resurrected keys;sizeAt()reads a separate size index (sizeByTx), so it correctly did not;keysAt()asserts the two agree → it threw"keys.size == size".And since the in-memory history stayed corrupted, every later read of that map failed too — the error is sticky and cascades.
Observed in the field as
"keys.size == size": keys.size=296, size=284on a busy DBMap during a long accumulated test run. The durable transaction log andsizeByTxboth agreed at 284; the in-memory history reported 296 — the 12-key gap was exactly the number of deleted keys that had been resurrected by a Small→Medium transition.The fix
Skip
Deletedentries when converting history values toOffHeap, in bothcommit()andretainTx(). Two small guards.Reproducer
JsonMapStoreTest.shouldKeepSizeAndKeysConsistentUnderConcurrentTransactions— deterministic, ~1m40s:@TestInjectablesdrops the model thresholds so the store crosses to Medium partway through the run;Clients drive overlapping insert / delete / rollback transactions against onemapDatastore;mapData.sizeequals the iterated key count (keysAt()'s own internal assertion also surfaces here).Fails as
keys.size=520, size=412without the fix; passes with it. Fulljsondb_teststays green (55 / 55).