docs: how to sync the store between machines

README.md

@@ -28,7 +28,7 @@ cotdata separates *fetching* data (a "producer" that talks to vendors) from *usi
 
 ## Contents
 
-- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license)
+- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Syncing the store](docs/SYNCING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license)
 
 ## Quickstart
 
@@ -212,6 +212,23 @@ Full setup, including wrapper scripts, the crontab entries (nightly prices, dail
 
 The short version: a databento server schedules the same way as the Windows/Norgate producer — prices nightly, COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run.
 
+### Syncing the store between machines
+
+Norgate needs Windows, so a research Mac or a Linux dashboard is usually a **read-only
+replica** of a store produced elsewhere. Prefer one producer writing everything and a
+strictly one-directional sync.
+
+Two directories must be **excluded**, and both are correctness issues rather than savings:
+`_cache/` and `_raw/` are databento producer-internal (on one real store, ~70% of the
+bytes), and `citpy/` is written by *cotmetrics* on the consumer, so mirroring would delete
+or clobber locally-derived output. The legacy `manifest.json` should be excluded too.
+
+Consumer cloud sync (Dropbox, Google Drive) is a poor fit here: conflict copies land
+inside the store, and on-demand placeholder files break `read_parquet` on the machine
+doing research.
+
+Full guidance, exclusion table and example scripts: **[docs/SYNCING.md](docs/SYNCING.md)**.
+
 ## Operations
 
 Read-only and maintenance commands, all cross-platform (they work off the store, no network):

docs/SYNCING.md

@@ -0,0 +1,134 @@
+# Syncing the store between machines
+
+The store is a producer/consumer contract: one machine writes, others only read. When
+the producer and the consumers are different machines, something has to move the files.
+
+This page is about **what** to move and what to leave behind. The transport is the easy
+part and comes last.
+
+## Prefer one producer
+
+The simplest topology by a wide margin is **one machine writes everything, the rest are
+read-only replicas**. Norgate needs Windows, so if you have a Windows box, give it both
+jobs and let the sync be strictly one-directional.
+
+That removes an entire class of problem. Two producers writing into two stores that are
+later merged by a file sync means every shared file is resolved last-writer-wins, and
+whichever side syncs second wins silently.
+
+COT does not need Norgate, so do not make it inherit the Norgate Data Updater's
+interactive-session requirement. Give it its own task:
+
+| Task | Command | Task Scheduler "General" |
+|---|---|---|
+| prices | `cotdata-prices --prices --metadata --require-final` | Run only when user is logged on (NDU needs a desktop session) |
+| COT | `cotdata-cot --cot-all` | Run whether user is logged in or not |
+
+## What NOT to sync
+
+This is the part that matters, and on a real store it is most of the bytes.
+
+| Directory | Sync? | Why |
+|---|---|---|
+| `prices/` | **yes** | the data |
+| `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data |
+| `metadata/` | **yes** | contract specs |
+| `manifests/` | **yes** | per-half bookkeeping |
+| `status.json` | yes | the producer's own view, useful on the replica |
+| `_cache/` | **NO** | databento provider cache, producer-internal, rebuildable |
+| `_raw/` | **NO** | databento append-only raw store, producer-internal |
+| `citpy/` | **NO** | written by **cotmetrics**, not cotdata (see below) |
+| `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) |
+
+On one real store those exclusions dropped the payload from 270 MB to about 82 MB, and
+two of them are correctness issues rather than savings.
+
+### `citpy/` is not cotdata's
+
+`cotmetrics` writes it, defaulting to `$COTDATA_STORE/citpy` when `COTMETRICS_CITPY` is
+unset. It is derived on the *consumer* machine and does not exist on a pure producer. A
+mirroring sync would therefore either delete it or overwrite locally-derived output with
+nothing. Exclude it, or point `COTMETRICS_CITPY` somewhere outside the store.
+
+### `manifest.json` is legacy
+
+Nothing writes it any more (see ADR-0007). It held both producer halves in one file,
+which is exactly the shape a file sync resolves last-writer-wins: a producer pushing an
+aggregate containing only its own half would silently drop the other half's entries on
+arrival. The per-half files under `manifests/` are disjoint and merge correctly.
+
+Run `cotdata-update --migrate-manifests` once per store, then delete `manifest.json`.
+
+## Ordering: data before manifests
+
+If a manifest arrives before the parquet it describes, a consumer briefly sees an entry
+pointing at data that has not landed. The other order is harmless: data present but not
+yet announced.
+
+In practice this is a nicety rather than a hazard, because `get_prices` and `get_cot`
+read parquet directly and the manifest is status only. But if your transport lets you
+control ordering, sync the data directories first and `manifests/` last.
+
+Individual writes are already safe: the store commits every parquet with an atomic
+`os.replace`, so a sync copying concurrently sees either the old file or the new one,
+never a partial. What a mid-run sync *can* catch is a partially updated **set** of files,
+which is why running the sync after the producer task rather than on a timer is better.
+
+## Transports
+
+### Avoid Dropbox and Google Drive
+
+Three specific problems, all of which put junk inside the store:
+
+- **Conflict copies.** Both create `something (conflicted copy).parquet` when they see
+  divergence. Those land in the store, where any directory scan treats them as real.
+- **Placeholder files.** Files On-Demand and Smart Sync leave non-local stubs.
+  `read_parquet` on a stub either stalls fetching or fails, on the machine doing research.
+- **Continuous sync.** They will happily replicate the store mid-producer-run.
+
+### Recommended: push after the producer runs
+
+Run the sync as a step immediately after the producer task, so it fires at a
+known-consistent moment rather than on a timer that might land mid-run.
+
+See [`examples/windows/sync-store.cmd`](examples/windows/sync-store.cmd), which uses
+`robocopy /MIR` with the exclusions above. `robocopy` is built into Windows, is
+incremental, and its destination can be a mapped drive, a UNC path, or a local folder
+that something else (Syncthing, an rclone remote) watches.
+
+**Watch the exit code.** `robocopy` returns 0-7 for success (1 means files were copied,
+3 means copied plus extras) and 8 or above for failure. A wrapper that does not normalise
+that will make Task Scheduler report every successful sync as a failure. The example
+script handles it.
+
+### Alternative: pull from the consumer
+
+If you would rather the consumer decide when to fetch, see
+[`examples/mac/pull-store.sh`](examples/mac/pull-store.sh), which uses `rsync` over SSH.
+
+Note `rsync` needs to exist on **both** ends, so on Windows that means WSL or a packaged
+rsync. If you only have OpenSSH, `scp -r` works and the payload is small enough that
+non-incremental copying is tolerable, but you lose deletion handling.
+
+### Alternative: Syncthing
+
+Peer-to-peer, no cloud middleman, no placeholder files, handles many-file datasets, and
+its conflict handling is explicit rather than silently duplicating. Put the exclusions in
+a `.stignore` at the store root. Reasonable if you want something continuous and
+unattended, at the cost of a daemon on both machines.
+
+## Verifying a sync worked
+
+On the replica:
+
+```bash
+cotdata-update --check
+```
+
+Compare `newest data` and `last write (UTC)` against the producer's output. The lag
+column is measured on write time, so an entry the producer skipped shows up as behind
+even when its data looks fine.
+
+If the replica shows entries the producer does not have, the sync is additive rather than
+mirroring and stale files are accumulating. If it shows *fewer*, the sync has not
+completed or an exclusion is too broad.

docs/examples/mac/pull-store.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Pull the cotdata store from the producer machine onto a read-only replica.
+# The consumer decides when to fetch, so this is the alternative to the producer
+# pushing (see examples/windows/sync-store.cmd).
+#
+# Copy this next to your launchd/cron config and overwrite the markers below:
+#   REPLACE_WITH_PRODUCER_HOST = ssh target        e.g. matt@windows-box
+#   REPLACE_WITH_REMOTE_STORE  = store ON that host e.g. /c/Users/matt/cotdata_store
+#   REPLACE_WITH_LOCAL_STORE   = store on THIS mac  e.g. /Users/you/code/cotdata_store
+# (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be
+# read as a shell redirection.)
+#
+# rsync must exist on BOTH ends. On Windows that means WSL or a packaged rsync. With
+# only OpenSSH, `scp -r` works and the payload is small, but you lose deletion
+# handling. See docs/SYNCING.md.
+set -euo pipefail
+
+SRC="REPLACE_WITH_PRODUCER_HOST:REPLACE_WITH_REMOTE_STORE"
+DEST="REPLACE_WITH_LOCAL_STORE"
+
+# What each exclusion is for:
+#   _cache, _raw   databento producer-internal, rebuildable, most of the bytes
+#   citpy          written by COTMETRICS on THIS machine, not by cotdata — a
+#                  --delete sync would remove locally-derived output
+#   manifest.json  legacy aggregate, nothing writes it, and it is the one file a
+#                  sync resolves last-writer-wins across both producer halves
+EXCLUDES=(
+  --exclude '_cache/'
+  --exclude '_raw/'
+  --exclude 'citpy/'
+  --exclude 'manifest.json'
+)
+
+# Two passes so a manifest never arrives before the data it describes. Harmless if
+# reversed (get_prices reads parquet directly), but free to get right.
+rsync -az --delete "${EXCLUDES[@]}" --exclude 'manifests/' "$SRC/" "$DEST/"
+rsync -az "$SRC/manifests/" "$DEST/manifests/"
+
+# Confirm what landed. Compare against the producer's own --check output: the lag
+# column is measured on WRITE time, so an entry the producer skipped shows as behind
+# even when its data looks fine.
+export COTDATA_STORE="$DEST"
+cotdata-update --check

docs/examples/windows/sync-store.cmd

@@ -0,0 +1,35 @@
+@echo off
+REM cotdata store sync wrapper for Windows Task Scheduler.
+REM Run this AFTER the producer tasks, not on its own timer, so it fires at a
+REM known-consistent moment rather than possibly mid-run.
+REM
+REM Copy this file into your scheduler folder and overwrite the two markers below.
+REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and the
+REM file fails with "The syntax of the command is incorrect" even on comment lines.
+REM   REPLACE_WITH_STORE_PATH = your local store   e.g. C:\Users\you\cotdata_store
+REM   REPLACE_WITH_DEST_PATH  = the sync target    e.g. Z:\cotdata_store  or
+REM                             \\mac\code\cotdata_store  or a Syncthing folder
+REM See docs/SYNCING.md for what each exclusion is for.
+
+setlocal
+
+REM /MIR mirrors (copies new + deletes removed). /XD excludes directories:
+REM   _cache, _raw  databento producer-internal, rebuildable, ~70%% of the bytes
+REM   citpy         written by COTMETRICS on the consumer, not by cotdata —
+REM                 mirroring would delete or clobber locally-derived output
+REM /XF excludes the legacy aggregate manifest: nothing writes it, and it is the one
+REM file a sync would resolve last-writer-wins across two halves.
+robocopy "REPLACE_WITH_STORE_PATH" "REPLACE_WITH_DEST_PATH" /MIR /R:2 /W:5 /NFL /NDL /NP ^
+  /XD _cache _raw citpy ^
+  /XF manifest.json
+
+REM robocopy uses exit codes 0-7 for SUCCESS (1 = files copied, 2 = extras present,
+REM 3 = both, and so on) and 8+ for failure. Task Scheduler treats any non-zero as a
+REM failure, so without this every successful sync would be reported as an error and
+REM "restart on failure" would loop.
+if %ERRORLEVEL% GEQ 8 (
+  echo sync FAILED with robocopy code %ERRORLEVEL%
+  exit /b %ERRORLEVEL%
+)
+echo sync ok ^(robocopy code %ERRORLEVEL%^)
+exit /b 0