Skip to content

Adding versionField to an existing collection silently breaks old rows #104

Description

@shavtvalishvili

While writing tests for #103 I found that define versionField on collection (test/versions.js) passes by accident, and the feature it covers has a data-format problem.

The bug

A collection's row bytes have two mutually exclusive layouts (builder/codegen.js):

  • without versionField: [type][version-stamp][fields] - the stamp is a prefix outside the record
  • with versionField: [fields] only - the stamp lives inside the record's own field

So adding versionField to a collection that already has rows on disk changes the layout under them. Old rows then decode as garbage - silently, no error. Instrumenting the insert path in the test shows prevVersion come out as 16 and 12: those are the members' ages, field bytes read as the version stamp.

The visible failure: with a garbage prevVersion (16), idx.version <= prevVersion is true for the new index, so its "previous keys" get computed from the garbage previous doc - whose name is still correct because it comes from the key - so prev and next keys match, the diff is empty, and entries for the new index are never written. find correctly returns 0 over an empty index.

Why the test passes today

The test helper's build() refreshes the generated index.js between builds but not messages.js, so generation 3 decodes rows through generation 1's cached codec - which still matches the old on-disk layout. One-line repro on main:

     try {
       delete require.cache[require.resolve(id)]
+      delete require.cache[require.resolve(path.join(id, 'messages.js'))]
     } catch {}

With that, define versionField on collection fails on main (find returns 0 instead of 2). Note the passing state also isn't one a real deployment can reach: it reads rows with one codec generation and writes them with another.

Possible directions

  1. Keep writing the [type][stamp] prefix even with versionField - old rows stay readable, at the cost of storing the version twice.
  2. Sniff the layout on decode - ambiguous in general, probably not worth it.
  3. Declare versionField valid only on collections that never had rows without it, document that, and point the test at a fresh dir.

Happy to PR the test-helper fix plus whichever direction you pick. @mafintosh @chm-diederichs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions