Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/SYNCING.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,17 @@ maintenance than a per-symbol roll-rule table.
rsync on Windows (cwRsync or WSL). robocopy cannot speak SSH, and SMB must never be
exposed over the internet, so the Mac's SMB path does not carry here. See
[`examples/windows/push-to-server.cmd`](examples/windows/push-to-server.cmd). The
exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`), so the
producer-internal databento bronze under `_raw/databento/` never leaves the Windows box.
exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`, plus
`*.tmp` for partial-write temps), so the producer-internal databento bronze under
`_raw/databento/` never leaves the Windows box.
- **Auth:** key-based SSH only. A scheduled task cannot type a passphrase, so use a
dedicated key with `ssh -o BatchMode=yes`, never a password prompt.
- **cwRsync gotcha:** a Cygwin rsync (what `choco install rsync` gives you) must drive the
ssh that *ships with it*, not the native Windows OpenSSH. Native ssh corrupts rsync's
binary stream and fails with `connection unexpectedly closed (0 bytes received so far)`,
even though a plain `ssh host echo ok` works fine. That Cygwin ssh also has no HOME, so
give it an explicit writable `-o UserKnownHostsFile=`, and use cygdrive (`/cygdrive/c/…`)
paths throughout, including the key. The example script wires all three.

**Provider cutover (one-time).** The server previously held a databento-built store, so its
`prices/` and `manifests/prices.json` carry databento data under the very keys the Norgate
Expand Down
49 changes: 35 additions & 14 deletions docs/examples/windows/push-to-server.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,40 +5,61 @@ REM known-consistent moment rather than on a timer that might land mid-run.
REM
REM Why not robocopy here: robocopy cannot speak SSH, and the dash server is a
REM remote VPS, so SMB is off the table (never expose SMB over the internet).
REM This uses rsync, which needs a packaged rsync ON WINDOWS. Two common ones:
REM cwRsync -> cygwin-style paths, e.g. /cygdrive/c/Users/you/cotdata_store
REM WSL -> /mnt/c-style paths, e.g. /mnt/c/Users/you/cotdata_store
REM This file is written for cwRsync. For WSL, prefix the rsync lines with `wsl `,
REM swap /cygdrive/c for /mnt/c, and drop the RSYNC= path (use bare `rsync`).
REM This uses rsync, which needs a packaged rsync ON WINDOWS. cwRsync (a Cygwin
REM build) is the tested one; `choco install rsync` installs exactly that, to
REM rsync.exe at C:\ProgramData\chocolatey\bin\rsync.exe
REM ssh.exe at C:\ProgramData\chocolatey\lib\rsync\tools\bin\ssh.exe
REM (WSL also works: prefix the rsync lines with `wsl `, use /mnt/c paths, and a
REM native rsync/ssh inside the distro. The Cygwin gotchas below do not apply.)
REM
REM -- Three Cygwin-rsync gotchas this file already handles ---------------------
REM 1. Use the ssh that SHIPS WITH rsync, never the native Windows OpenSSH
REM (C:\Windows\System32\OpenSSH\ssh.exe). A Cygwin rsync driving native ssh
REM corrupts rsync's binary stream and dies with
REM "connection unexpectedly closed (0 bytes received so far)".
REM Point SSH_EXE at the bundled Cygwin ssh instead.
REM 2. That Cygwin ssh has no HOME, so it cannot write the default known_hosts and
REM warns "Failed to add the host ... (/known_hosts)". Give it an explicit
REM writable UserKnownHostsFile (created on first connect).
REM 3. All local paths are cygdrive form: /cygdrive/c/... , including the key.
REM ----------------------------------------------------------------------------
REM
REM Overwrite the markers below. Do NOT use angle brackets in a .cmd file: cmd
REM reads them as redirection and the file fails even on comment lines.
REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form
REM e.g. /cygdrive/c/Users/you/cotdata_store
REM REPLACE_WITH_SSH_EXE_CYG = the ssh that ships with rsync, cygdrive form
REM e.g. /cygdrive/c/ProgramData/chocolatey/lib/rsync/tools/bin/ssh.exe
REM REPLACE_WITH_SSH_KEY_CYG = batch SSH private key, cygdrive form
REM e.g. /cygdrive/c/Users/you/.ssh/cotdata_push
REM REPLACE_WITH_KNOWN_HOSTS_CYG= a writable known_hosts, cygdrive form
REM e.g. /cygdrive/c/Users/you/.ssh/known_hosts
REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form
REM e.g. /cygdrive/c/Users/you/cotdata_store
REM REPLACE_WITH_REMOTE = user@host:/path/to/store (no trailing slash)
REM e.g. deploy@dash.example.com:/srv/cotdata_store
REM See docs/SYNCING.md ("Dash store") for the exclusions and the one-time cutover.

setlocal
set "RSYNC=C:\Program Files\cwRsync\bin\rsync.exe"
set "SRC=REPLACE_WITH_STORE_PATH_CYG"
set "RSYNC=C:\ProgramData\chocolatey\bin\rsync.exe"
set "SSH_EXE=REPLACE_WITH_SSH_EXE_CYG"
set "KEY=REPLACE_WITH_SSH_KEY_CYG"
set "KNOWN=REPLACE_WITH_KNOWN_HOSTS_CYG"
set "SRC=REPLACE_WITH_STORE_PATH_CYG"
set "DEST=REPLACE_WITH_REMOTE"
set "SSH=ssh -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new"
set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%KNOWN%"

REM Data first, manifests last, so a manifest never announces parquet that has not
REM landed (harmless if reversed; get_prices reads parquet directly). --delete makes
REM this a true mirror. The exclusions match the Mac push:
REM _cache, _raw producer-internal; _raw/databento (the paid databento bronze)
REM rides under _raw and so is excluded, per ADR-0006.
REM citpy consumer-owned notes on the server; excluding it from --delete
REM is what stops the mirror from wiping them.
REM citpy consumer-owned on the server; excluding it from --delete is
REM what stops the mirror from wiping it.
REM manifest.json legacy aggregate, resolved last-writer-wins across halves.
REM *.tmp a producer's partial-write temp (atomic write via os.replace);
REM never propagate a half-written file.
"%RSYNC%" -az --delete ^
--exclude "_cache/" --exclude "_raw/" --exclude "citpy/" --exclude "manifest.json" ^
--exclude "manifests/" ^
--exclude "_cache/" --exclude "_raw/" --exclude "citpy/" ^
--exclude "manifest.json" --exclude "*.tmp" --exclude "manifests/" ^
-e "%SSH%" "%SRC%/" "%DEST%/"
if %ERRORLEVEL% NEQ 0 ( echo push FAILED, rsync code %ERRORLEVEL% & exit /b %ERRORLEVEL% )

Expand Down
Loading