docs: how to sync the store between machines

[mspinola]

Norgate needs Windows, so a research Mac or a Linux dashboard is usually a read-only replica of a store produced elsewhere. Nothing documented how to move the files, and the obvious approaches get two things wrong.

What NOT to sync is the load-bearing part

Two of these are correctness issues rather than savings:

Directory	Why exclude
_cache/, _raw/	databento producer-internal and rebuildable. On one real store, 188 MB of 270 MB
citpy/	written by cotmetrics on the consumer, not by cotdata. A mirroring sync deletes it or overwrites locally-derived output with nothing
manifest.json	the legacy aggregate. Nothing writes it, and it held both producer halves in ONE file, exactly the shape a file sync resolves last-writer-wins

Excluding them takes the payload from 270 MB to about 82 MB.

Recommendations

One producer writes everything. Two producers writing two stores that a sync later merges means every shared file is last-writer-wins.

Give COT its own task. It does not need Norgate, so it should not inherit NDU's "run only when user is logged on" constraint.

Avoid Dropbox and Google Drive, for three specific reasons rather than general dislike:

• conflict copies (something (conflicted copy).parquet) land inside the store, where any directory scan treats them as real
• Files On-Demand / Smart Sync placeholders break read_parquet on the machine doing research
• both sync continuously, so they will replicate mid-producer-run

Example scripts

examples/windows/sync-store.cmd — robocopy /MIR with the exclusions.

It normalises robocopy's exit codes, which is the gotcha here: robocopy returns 0-7 for success (1 = files copied, 3 = copied plus extras) and 8+ for failure. A wrapper that does not handle that makes Task Scheduler report every successful sync as a failure, and "restart on failure" then loops on it.

examples/mac/pull-store.sh — rsync over SSH for a consumer-pull setup, in two passes so a manifest never lands before the data it describes.

Also documented

What is already safe: every parquet is committed with an atomic os.replace, so a concurrent sync sees either the old file or the new one, never a partial. What a mid-run sync can catch is a partially updated set, which is why the sync belongs after the producer task rather than on its own timer.

And how to verify a sync landed, including reading the lag column correctly now that it measures write time.

Checks

Angle-bracket-free .cmd invariant preserved, bash -n clean on the shell script, doc links resolve, suite 134 passed, ruff clean. Docs only.