From 1111f6ce787d1976978d6bb42b599885812d35a8 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Sun, 26 Jul 2026 17:16:01 -0400 Subject: [PATCH] docs: how to sync the store between machines Norgate needs Windows, so a research Mac or a Linux dashboard is usually a read-only replica of a store produced elsewhere. Nothing documented how to move the files, and the obvious approaches get two things wrong. WHAT NOT TO SYNC is the load-bearing part, and two of the exclusions are correctness issues rather than savings: _cache/ and _raw/ databento producer-internal and rebuildable. On one real store these are 188MB of 270MB, so excluding them takes the payload from 270MB to about 82MB. citpy/ written by COTMETRICS on the consumer, not by cotdata. A mirroring sync would delete it or overwrite locally-derived output with nothing. manifest.json the legacy aggregate. Nothing writes it, and it held both producer halves in ONE file, which is exactly the shape a file sync resolves last-writer-wins. Recommends one producer writing everything, with COT given its own task so it does not inherit NDU's interactive-session requirement (it does not need Norgate). Advises against Dropbox and Google Drive for three specific reasons: conflict copies land inside the store where any directory scan treats them as real, on-demand placeholder files break read_parquet on the research machine, and both sync continuously so they will replicate mid-producer-run. Two example scripts: examples/windows/sync-store.cmd robocopy /MIR with the exclusions. Normalises robocopy's exit codes, which use 0-7 for success and 8+ for failure, so Task Scheduler does not report every successful sync as an error and retry-loop on it. examples/mac/pull-store.sh rsync over SSH for a consumer-pull setup, in two passes so a manifest never lands before the data it describes. Also notes what is already safe: every parquet is committed with an atomic os.replace, so a concurrent sync sees the old file or the new one, never a partial. What a mid-run sync can catch is a partially updated SET, which is why the sync belongs after the producer task rather than on its own timer. Suite 134 passed. Co-Authored-By: Claude Opus 5 --- README.md | 19 +++- docs/SYNCING.md | 134 +++++++++++++++++++++++++++ docs/examples/mac/pull-store.sh | 43 +++++++++ docs/examples/windows/sync-store.cmd | 35 +++++++ 4 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 docs/SYNCING.md create mode 100755 docs/examples/mac/pull-store.sh create mode 100644 docs/examples/windows/sync-store.cmd diff --git a/README.md b/README.md index adbf707..8fae68e 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ cotdata separates *fetching* data (a "producer" that talks to vendors) from *usi ## Contents -- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license) +- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Syncing the store](docs/SYNCING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license) ## Quickstart @@ -212,6 +212,23 @@ Full setup, including wrapper scripts, the crontab entries (nightly prices, dail The short version: a databento server schedules the same way as the Windows/Norgate producer — prices nightly, COT gets a daily morning catch-up plus a tight Friday-afternoon poll around its ~3:30pm ET release, all idempotent and safe to over-run. +### Syncing the store between machines + +Norgate needs Windows, so a research Mac or a Linux dashboard is usually a **read-only +replica** of a store produced elsewhere. Prefer one producer writing everything and a +strictly one-directional sync. + +Two directories must be **excluded**, and both are correctness issues rather than savings: +`_cache/` and `_raw/` are databento producer-internal (on one real store, ~70% of the +bytes), and `citpy/` is written by *cotmetrics* on the consumer, so mirroring would delete +or clobber locally-derived output. The legacy `manifest.json` should be excluded too. + +Consumer cloud sync (Dropbox, Google Drive) is a poor fit here: conflict copies land +inside the store, and on-demand placeholder files break `read_parquet` on the machine +doing research. + +Full guidance, exclusion table and example scripts: **[docs/SYNCING.md](docs/SYNCING.md)**. + ## Operations Read-only and maintenance commands, all cross-platform (they work off the store, no network): diff --git a/docs/SYNCING.md b/docs/SYNCING.md new file mode 100644 index 0000000..30bdf2b --- /dev/null +++ b/docs/SYNCING.md @@ -0,0 +1,134 @@ +# Syncing the store between machines + +The store is a producer/consumer contract: one machine writes, others only read. When +the producer and the consumers are different machines, something has to move the files. + +This page is about **what** to move and what to leave behind. The transport is the easy +part and comes last. + +## Prefer one producer + +The simplest topology by a wide margin is **one machine writes everything, the rest are +read-only replicas**. Norgate needs Windows, so if you have a Windows box, give it both +jobs and let the sync be strictly one-directional. + +That removes an entire class of problem. Two producers writing into two stores that are +later merged by a file sync means every shared file is resolved last-writer-wins, and +whichever side syncs second wins silently. + +COT does not need Norgate, so do not make it inherit the Norgate Data Updater's +interactive-session requirement. Give it its own task: + +| Task | Command | Task Scheduler "General" | +|---|---|---| +| prices | `cotdata-prices --prices --metadata --require-final` | Run only when user is logged on (NDU needs a desktop session) | +| COT | `cotdata-cot --cot-all` | Run whether user is logged in or not | + +## What NOT to sync + +This is the part that matters, and on a real store it is most of the bytes. + +| Directory | Sync? | Why | +|---|---|---| +| `prices/` | **yes** | the data | +| `cot_legacy/`, `cot_disagg/`, `cot_tff/` | **yes** | the data | +| `metadata/` | **yes** | contract specs | +| `manifests/` | **yes** | per-half bookkeeping | +| `status.json` | yes | the producer's own view, useful on the replica | +| `_cache/` | **NO** | databento provider cache, producer-internal, rebuildable | +| `_raw/` | **NO** | databento append-only raw store, producer-internal | +| `citpy/` | **NO** | written by **cotmetrics**, not cotdata (see below) | +| `manifest.json` | **NO** | legacy aggregate, nothing writes it (see below) | + +On one real store those exclusions dropped the payload from 270 MB to about 82 MB, and +two of them are correctness issues rather than savings. + +### `citpy/` is not cotdata's + +`cotmetrics` writes it, defaulting to `$COTDATA_STORE/citpy` when `COTMETRICS_CITPY` is +unset. It is derived on the *consumer* machine and does not exist on a pure producer. A +mirroring sync would therefore either delete it or overwrite locally-derived output with +nothing. Exclude it, or point `COTMETRICS_CITPY` somewhere outside the store. + +### `manifest.json` is legacy + +Nothing writes it any more (see ADR-0007). It held both producer halves in one file, +which is exactly the shape a file sync resolves last-writer-wins: a producer pushing an +aggregate containing only its own half would silently drop the other half's entries on +arrival. The per-half files under `manifests/` are disjoint and merge correctly. + +Run `cotdata-update --migrate-manifests` once per store, then delete `manifest.json`. + +## Ordering: data before manifests + +If a manifest arrives before the parquet it describes, a consumer briefly sees an entry +pointing at data that has not landed. The other order is harmless: data present but not +yet announced. + +In practice this is a nicety rather than a hazard, because `get_prices` and `get_cot` +read parquet directly and the manifest is status only. But if your transport lets you +control ordering, sync the data directories first and `manifests/` last. + +Individual writes are already safe: the store commits every parquet with an atomic +`os.replace`, so a sync copying concurrently sees either the old file or the new one, +never a partial. What a mid-run sync *can* catch is a partially updated **set** of files, +which is why running the sync after the producer task rather than on a timer is better. + +## Transports + +### Avoid Dropbox and Google Drive + +Three specific problems, all of which put junk inside the store: + +- **Conflict copies.** Both create `something (conflicted copy).parquet` when they see + divergence. Those land in the store, where any directory scan treats them as real. +- **Placeholder files.** Files On-Demand and Smart Sync leave non-local stubs. + `read_parquet` on a stub either stalls fetching or fails, on the machine doing research. +- **Continuous sync.** They will happily replicate the store mid-producer-run. + +### Recommended: push after the producer runs + +Run the sync as a step immediately after the producer task, so it fires at a +known-consistent moment rather than on a timer that might land mid-run. + +See [`examples/windows/sync-store.cmd`](examples/windows/sync-store.cmd), which uses +`robocopy /MIR` with the exclusions above. `robocopy` is built into Windows, is +incremental, and its destination can be a mapped drive, a UNC path, or a local folder +that something else (Syncthing, an rclone remote) watches. + +**Watch the exit code.** `robocopy` returns 0-7 for success (1 means files were copied, +3 means copied plus extras) and 8 or above for failure. A wrapper that does not normalise +that will make Task Scheduler report every successful sync as a failure. The example +script handles it. + +### Alternative: pull from the consumer + +If you would rather the consumer decide when to fetch, see +[`examples/mac/pull-store.sh`](examples/mac/pull-store.sh), which uses `rsync` over SSH. + +Note `rsync` needs to exist on **both** ends, so on Windows that means WSL or a packaged +rsync. If you only have OpenSSH, `scp -r` works and the payload is small enough that +non-incremental copying is tolerable, but you lose deletion handling. + +### Alternative: Syncthing + +Peer-to-peer, no cloud middleman, no placeholder files, handles many-file datasets, and +its conflict handling is explicit rather than silently duplicating. Put the exclusions in +a `.stignore` at the store root. Reasonable if you want something continuous and +unattended, at the cost of a daemon on both machines. + +## Verifying a sync worked + +On the replica: + +```bash +cotdata-update --check +``` + +Compare `newest data` and `last write (UTC)` against the producer's output. The lag +column is measured on write time, so an entry the producer skipped shows up as behind +even when its data looks fine. + +If the replica shows entries the producer does not have, the sync is additive rather than +mirroring and stale files are accumulating. If it shows *fewer*, the sync has not +completed or an exclusion is too broad. diff --git a/docs/examples/mac/pull-store.sh b/docs/examples/mac/pull-store.sh new file mode 100755 index 0000000..d90bbf6 --- /dev/null +++ b/docs/examples/mac/pull-store.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Pull the cotdata store from the producer machine onto a read-only replica. +# The consumer decides when to fetch, so this is the alternative to the producer +# pushing (see examples/windows/sync-store.cmd). +# +# Copy this next to your launchd/cron config and overwrite the markers below: +# REPLACE_WITH_PRODUCER_HOST = ssh target e.g. matt@windows-box +# REPLACE_WITH_REMOTE_STORE = store ON that host e.g. /c/Users/matt/cotdata_store +# REPLACE_WITH_LOCAL_STORE = store on THIS mac e.g. /Users/you/code/cotdata_store +# (Plain-text markers, not angle-bracket placeholders: an unedited <...> would be +# read as a shell redirection.) +# +# rsync must exist on BOTH ends. On Windows that means WSL or a packaged rsync. With +# only OpenSSH, `scp -r` works and the payload is small, but you lose deletion +# handling. See docs/SYNCING.md. +set -euo pipefail + +SRC="REPLACE_WITH_PRODUCER_HOST:REPLACE_WITH_REMOTE_STORE" +DEST="REPLACE_WITH_LOCAL_STORE" + +# What each exclusion is for: +# _cache, _raw databento producer-internal, rebuildable, most of the bytes +# citpy written by COTMETRICS on THIS machine, not by cotdata — a +# --delete sync would remove locally-derived output +# manifest.json legacy aggregate, nothing writes it, and it is the one file a +# sync resolves last-writer-wins across both producer halves +EXCLUDES=( + --exclude '_cache/' + --exclude '_raw/' + --exclude 'citpy/' + --exclude 'manifest.json' +) + +# Two passes so a manifest never arrives before the data it describes. Harmless if +# reversed (get_prices reads parquet directly), but free to get right. +rsync -az --delete "${EXCLUDES[@]}" --exclude 'manifests/' "$SRC/" "$DEST/" +rsync -az "$SRC/manifests/" "$DEST/manifests/" + +# Confirm what landed. Compare against the producer's own --check output: the lag +# column is measured on WRITE time, so an entry the producer skipped shows as behind +# even when its data looks fine. +export COTDATA_STORE="$DEST" +cotdata-update --check diff --git a/docs/examples/windows/sync-store.cmd b/docs/examples/windows/sync-store.cmd new file mode 100644 index 0000000..0de2920 --- /dev/null +++ b/docs/examples/windows/sync-store.cmd @@ -0,0 +1,35 @@ +@echo off +REM cotdata store sync wrapper for Windows Task Scheduler. +REM Run this AFTER the producer tasks, not on its own timer, so it fires at a +REM known-consistent moment rather than possibly mid-run. +REM +REM Copy this file into your scheduler folder and overwrite the two markers below. +REM Do NOT put angle brackets in a .cmd file: cmd reads them as redirection and the +REM file fails with "The syntax of the command is incorrect" even on comment lines. +REM REPLACE_WITH_STORE_PATH = your local store e.g. C:\Users\you\cotdata_store +REM REPLACE_WITH_DEST_PATH = the sync target e.g. Z:\cotdata_store or +REM \\mac\code\cotdata_store or a Syncthing folder +REM See docs/SYNCING.md for what each exclusion is for. + +setlocal + +REM /MIR mirrors (copies new + deletes removed). /XD excludes directories: +REM _cache, _raw databento producer-internal, rebuildable, ~70%% of the bytes +REM citpy written by COTMETRICS on the consumer, not by cotdata — +REM mirroring would delete or clobber locally-derived output +REM /XF excludes the legacy aggregate manifest: nothing writes it, and it is the one +REM file a sync would resolve last-writer-wins across two halves. +robocopy "REPLACE_WITH_STORE_PATH" "REPLACE_WITH_DEST_PATH" /MIR /R:2 /W:5 /NFL /NDL /NP ^ + /XD _cache _raw citpy ^ + /XF manifest.json + +REM robocopy uses exit codes 0-7 for SUCCESS (1 = files copied, 2 = extras present, +REM 3 = both, and so on) and 8+ for failure. Task Scheduler treats any non-zero as a +REM failure, so without this every successful sync would be reported as an error and +REM "restart on failure" would loop. +if %ERRORLEVEL% GEQ 8 ( + echo sync FAILED with robocopy code %ERRORLEVEL% + exit /b %ERRORLEVEL% +) +echo sync ok ^(robocopy code %ERRORLEVEL%^) +exit /b 0