While writing tests for #103 I found that define versionField on collection (test/versions.js) passes by accident, and the feature it covers has a data-format problem.
The bug
A collection's row bytes have two mutually exclusive layouts (builder/codegen.js):
- without
versionField: [type][version-stamp][fields] - the stamp is a prefix outside the record
- with
versionField: [fields] only - the stamp lives inside the record's own field
So adding versionField to a collection that already has rows on disk changes the layout under them. Old rows then decode as garbage - silently, no error. Instrumenting the insert path in the test shows prevVersion come out as 16 and 12: those are the members' ages, field bytes read as the version stamp.
The visible failure: with a garbage prevVersion (16), idx.version <= prevVersion is true for the new index, so its "previous keys" get computed from the garbage previous doc - whose name is still correct because it comes from the key - so prev and next keys match, the diff is empty, and entries for the new index are never written. find correctly returns 0 over an empty index.
Why the test passes today
The test helper's build() refreshes the generated index.js between builds but not messages.js, so generation 3 decodes rows through generation 1's cached codec - which still matches the old on-disk layout. One-line repro on main:
try {
delete require.cache[require.resolve(id)]
+ delete require.cache[require.resolve(path.join(id, 'messages.js'))]
} catch {}
With that, define versionField on collection fails on main (find returns 0 instead of 2). Note the passing state also isn't one a real deployment can reach: it reads rows with one codec generation and writes them with another.
Possible directions
- Keep writing the
[type][stamp] prefix even with versionField - old rows stay readable, at the cost of storing the version twice.
- Sniff the layout on decode - ambiguous in general, probably not worth it.
- Declare
versionField valid only on collections that never had rows without it, document that, and point the test at a fresh dir.
Happy to PR the test-helper fix plus whichever direction you pick. @mafintosh @chm-diederichs
While writing tests for #103 I found that
define versionField on collection(test/versions.js) passes by accident, and the feature it covers has a data-format problem.The bug
A collection's row bytes have two mutually exclusive layouts (builder/codegen.js):
versionField:[type][version-stamp][fields]- the stamp is a prefix outside the recordversionField:[fields]only - the stamp lives inside the record's own fieldSo adding
versionFieldto a collection that already has rows on disk changes the layout under them. Old rows then decode as garbage - silently, no error. Instrumenting the insert path in the test showsprevVersioncome out as 16 and 12: those are the members' ages, field bytes read as the version stamp.The visible failure: with a garbage
prevVersion(16),idx.version <= prevVersionis true for the new index, so its "previous keys" get computed from the garbage previous doc - whosenameis still correct because it comes from the key - so prev and next keys match, the diff is empty, and entries for the new index are never written.findcorrectly returns 0 over an empty index.Why the test passes today
The test helper's
build()refreshes the generatedindex.jsbetween builds but notmessages.js, so generation 3 decodes rows through generation 1's cached codec - which still matches the old on-disk layout. One-line repro on main:try { delete require.cache[require.resolve(id)] + delete require.cache[require.resolve(path.join(id, 'messages.js'))] } catch {}With that,
define versionField on collectionfails on main (findreturns 0 instead of 2). Note the passing state also isn't one a real deployment can reach: it reads rows with one codec generation and writes them with another.Possible directions
[type][stamp]prefix even withversionField- old rows stay readable, at the cost of storing the version twice.versionFieldvalid only on collections that never had rows without it, document that, and point the test at a fresh dir.Happy to PR the test-helper fix plus whichever direction you pick. @mafintosh @chm-diederichs