Skip to content

Fix JsonMapStore Small→Medium transition resurrecting deleted keys - #1

Closed
mrcompator wants to merge 2 commits into
masterfrom
fix/jsondb-medium-model-tombstone-resurrection
Closed

mrcompator wants to merge 2 commits into
masterfrom
fix/jsondb-medium-model-tombstone-resurrection

Conversation

@mrcompator

Copy link
Copy Markdown
Owner

What was wrong

A JsonMapStore holds its data in memory in one of three "storage models" — Small, Medium, Large — and grows between them as the map gets bigger. When it grows Small → Medium, it moves the real values out of memory and leaves an OffHeap marker in the history in their place.

Both places that perform this conversion — commit() (for the batch that triggered the growth) and retainTx() (for the rest of the history, during maintenance) — rewrote every history entry to OffHeap without checking whether the entry was a Deleted tombstone.

Because OffHeap != Deleted, this made every key that had ever been remove()d look present again:

  • keysAt() walks the history, so it counted those resurrected keys;
  • sizeAt() reads a separate size index (sizeByTx), so it correctly did not;
  • keysAt() asserts the two agree → it threw "keys.size == size".

And since the in-memory history stayed corrupted, every later read of that map failed too — the error is sticky and cascades.

Observed in the field as "keys.size == size": keys.size=296, size=284 on a busy DBMap during a long accumulated test run. The durable transaction log and sizeByTx both agreed at 284; the in-memory history reported 296 — the 12-key gap was exactly the number of deleted keys that had been resurrected by a Small→Medium transition.

The fix

Skip Deleted entries when converting history values to OffHeap, in both commit() and retainTx(). Two small guards.

Reproducer

JsonMapStoreTest.shouldKeepSizeAndKeysConsistentUnderConcurrentTransactions — deterministic, ~1m40s:

  • @TestInjectables drops the model thresholds so the store crosses to Medium partway through the run;
  • several Clients drive overlapping insert / delete / rollback transactions against one mapData store;
  • a reader repeatedly checks, inside one read transaction, that mapData.size equals the iterated key count (keysAt()'s own internal assertion also surfaces here).

Fails as keys.size=520, size=412 without the fix; passes with it. Full jsondb_test stays green (55 / 55).

lagergren and others added 2 commits August 28, 2026 08:28
When a JsonMapStore grows from the Small to the Medium storage model, both
the commit() and retainTx() transition paths rewrite in-memory history
values to the OffHeap marker. They did this unconditionally -- including
for Deleted tombstones. Because OffHeap != Deleted, every key that had been
removed became visible to keysAt() again, while sizeAt() (which tracks
sizeByTx) still correctly excluded it. keysAt()'s `assert keys.size ==
size` then fails, and because the in-memory history stays corrupted the
failure is sticky and cascades to every later read of that map.

Guard both conversions with a `!= Deleted` check so tombstones are left
intact.

Adds a deterministic reproducer
(JsonMapStoreTest.shouldKeepSizeAndKeysConsistentUnderConcurrentTransactions):
model thresholds are injected low so the store crosses Small->Medium
mid-run, then several Clients drive overlapping insert/delete/rollback
transactions on one map while a reader checks size vs the iterated key
count in a read transaction. Fails as `keys.size=520, size=412` before the
fix; passes after. Full jsondb_test stays green (55/55).
@mrcompator
mrcompator force-pushed the fix/jsondb-medium-model-tombstone-resurrection branch from 348137d to b035c88 Compare August 28, 2026 14:22
@mrcompator

Copy link
Copy Markdown
Owner Author

Superseded — re-opening against xtclang/xvm.

@mrcompator mrcompator closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants