diff --git a/docs/SYNCING.md b/docs/SYNCING.md index dcb2afc..e5593a8 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -51,10 +51,17 @@ maintenance than a per-symbol roll-rule table. rsync on Windows (cwRsync or WSL). robocopy cannot speak SSH, and SMB must never be exposed over the internet, so the Mac's SMB path does not carry here. See [`examples/windows/push-to-server.cmd`](examples/windows/push-to-server.cmd). The - exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`), so the - producer-internal databento bronze under `_raw/databento/` never leaves the Windows box. + exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`, plus + `*.tmp` for partial-write temps), so the producer-internal databento bronze under + `_raw/databento/` never leaves the Windows box. - **Auth:** key-based SSH only. A scheduled task cannot type a passphrase, so use a dedicated key with `ssh -o BatchMode=yes`, never a password prompt. +- **cwRsync gotcha:** a Cygwin rsync (what `choco install rsync` gives you) must drive the + ssh that *ships with it*, not the native Windows OpenSSH. Native ssh corrupts rsync's + binary stream and fails with `connection unexpectedly closed (0 bytes received so far)`, + even though a plain `ssh host echo ok` works fine. That Cygwin ssh also has no HOME, so + give it an explicit writable `-o UserKnownHostsFile=`, and use cygdrive (`/cygdrive/c/…`) + paths throughout, including the key. The example script wires all three. **Provider cutover (one-time).** The server previously held a databento-built store, so its `prices/` and `manifests/prices.json` carry databento data under the very keys the Norgate diff --git a/docs/examples/windows/push-to-server.cmd b/docs/examples/windows/push-to-server.cmd index 79a11f9..b65c975 100644 --- a/docs/examples/windows/push-to-server.cmd +++ b/docs/examples/windows/push-to-server.cmd @@ -5,40 +5,61 @@ REM known-consistent moment rather than on a timer that might land mid-run. REM REM Why not robocopy here: robocopy cannot speak SSH, and the dash server is a REM remote VPS, so SMB is off the table (never expose SMB over the internet). -REM This uses rsync, which needs a packaged rsync ON WINDOWS. Two common ones: -REM cwRsync -> cygwin-style paths, e.g. /cygdrive/c/Users/you/cotdata_store -REM WSL -> /mnt/c-style paths, e.g. /mnt/c/Users/you/cotdata_store -REM This file is written for cwRsync. For WSL, prefix the rsync lines with `wsl `, -REM swap /cygdrive/c for /mnt/c, and drop the RSYNC= path (use bare `rsync`). +REM This uses rsync, which needs a packaged rsync ON WINDOWS. cwRsync (a Cygwin +REM build) is the tested one; `choco install rsync` installs exactly that, to +REM rsync.exe at C:\ProgramData\chocolatey\bin\rsync.exe +REM ssh.exe at C:\ProgramData\chocolatey\lib\rsync\tools\bin\ssh.exe +REM (WSL also works: prefix the rsync lines with `wsl `, use /mnt/c paths, and a +REM native rsync/ssh inside the distro. The Cygwin gotchas below do not apply.) +REM +REM -- Three Cygwin-rsync gotchas this file already handles --------------------- +REM 1. Use the ssh that SHIPS WITH rsync, never the native Windows OpenSSH +REM (C:\Windows\System32\OpenSSH\ssh.exe). A Cygwin rsync driving native ssh +REM corrupts rsync's binary stream and dies with +REM "connection unexpectedly closed (0 bytes received so far)". +REM Point SSH_EXE at the bundled Cygwin ssh instead. +REM 2. That Cygwin ssh has no HOME, so it cannot write the default known_hosts and +REM warns "Failed to add the host ... (/known_hosts)". Give it an explicit +REM writable UserKnownHostsFile (created on first connect). +REM 3. All local paths are cygdrive form: /cygdrive/c/... , including the key. +REM ---------------------------------------------------------------------------- REM REM Overwrite the markers below. Do NOT use angle brackets in a .cmd file: cmd REM reads them as redirection and the file fails even on comment lines. -REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form -REM e.g. /cygdrive/c/Users/you/cotdata_store +REM REPLACE_WITH_SSH_EXE_CYG = the ssh that ships with rsync, cygdrive form +REM e.g. /cygdrive/c/ProgramData/chocolatey/lib/rsync/tools/bin/ssh.exe REM REPLACE_WITH_SSH_KEY_CYG = batch SSH private key, cygdrive form REM e.g. /cygdrive/c/Users/you/.ssh/cotdata_push +REM REPLACE_WITH_KNOWN_HOSTS_CYG= a writable known_hosts, cygdrive form +REM e.g. /cygdrive/c/Users/you/.ssh/known_hosts +REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form +REM e.g. /cygdrive/c/Users/you/cotdata_store REM REPLACE_WITH_REMOTE = user@host:/path/to/store (no trailing slash) REM e.g. deploy@dash.example.com:/srv/cotdata_store REM See docs/SYNCING.md ("Dash store") for the exclusions and the one-time cutover. setlocal -set "RSYNC=C:\Program Files\cwRsync\bin\rsync.exe" -set "SRC=REPLACE_WITH_STORE_PATH_CYG" +set "RSYNC=C:\ProgramData\chocolatey\bin\rsync.exe" +set "SSH_EXE=REPLACE_WITH_SSH_EXE_CYG" set "KEY=REPLACE_WITH_SSH_KEY_CYG" +set "KNOWN=REPLACE_WITH_KNOWN_HOSTS_CYG" +set "SRC=REPLACE_WITH_STORE_PATH_CYG" set "DEST=REPLACE_WITH_REMOTE" -set "SSH=ssh -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new" +set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%KNOWN%" REM Data first, manifests last, so a manifest never announces parquet that has not REM landed (harmless if reversed; get_prices reads parquet directly). --delete makes REM this a true mirror. The exclusions match the Mac push: REM _cache, _raw producer-internal; _raw/databento (the paid databento bronze) REM rides under _raw and so is excluded, per ADR-0006. -REM citpy consumer-owned notes on the server; excluding it from --delete -REM is what stops the mirror from wiping them. +REM citpy consumer-owned on the server; excluding it from --delete is +REM what stops the mirror from wiping it. REM manifest.json legacy aggregate, resolved last-writer-wins across halves. +REM *.tmp a producer's partial-write temp (atomic write via os.replace); +REM never propagate a half-written file. "%RSYNC%" -az --delete ^ - --exclude "_cache/" --exclude "_raw/" --exclude "citpy/" --exclude "manifest.json" ^ - --exclude "manifests/" ^ + --exclude "_cache/" --exclude "_raw/" --exclude "citpy/" ^ + --exclude "manifest.json" --exclude "*.tmp" --exclude "manifests/" ^ -e "%SSH%" "%SRC%/" "%DEST%/" if %ERRORLEVEL% NEQ 0 ( echo push FAILED, rsync code %ERRORLEVEL% & exit /b %ERRORLEVEL% )