From 3c6acadb635128c44d5d9e821ef75f477cb14619 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Mon, 27 Jul 2026 12:07:00 -0400 Subject: [PATCH] docs: fix push-to-server.cmd for Cygwin rsync + real-world gotchas Rolling out the Windows->Linux dash push surfaced three issues the example script would have walked the next person straight into: - A Cygwin rsync (`choco install rsync`) driving the native Windows OpenSSH corrupts rsync's binary stream and dies with "connection unexpectedly closed (0 bytes received so far)", even though plain `ssh host echo ok` works. It must use the ssh that ships with rsync. Add an SSH_EXE marker and say why. - That bundled Cygwin ssh has no HOME, so it can't write the default known_hosts ("Failed to add the host ... (/known_hosts)"). Add an explicit writable UserKnownHostsFile. - Exclude *.tmp so a producer's partial-write temp (atomic write via os.replace) never propagates. Also correct the install path to the choco layout (C:\ProgramData\chocolatey\...) and note the matching bundled ssh location, and add a "cwRsync gotcha" bullet to SYNCING.md's Dash store section. Docs only; no code changes. Co-Authored-By: Claude Opus 4.8 --- docs/SYNCING.md | 11 +++++- docs/examples/windows/push-to-server.cmd | 49 +++++++++++++++++------- 2 files changed, 44 insertions(+), 16 deletions(-) diff --git a/docs/SYNCING.md b/docs/SYNCING.md index dcb2afc..e5593a8 100644 --- a/docs/SYNCING.md +++ b/docs/SYNCING.md @@ -51,10 +51,17 @@ maintenance than a per-symbol roll-rule table. rsync on Windows (cwRsync or WSL). robocopy cannot speak SSH, and SMB must never be exposed over the internet, so the Mac's SMB path does not carry here. See [`examples/windows/push-to-server.cmd`](examples/windows/push-to-server.cmd). The - exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`), so the - producer-internal databento bronze under `_raw/databento/` never leaves the Windows box. + exclusions match the Mac push (`_cache/`, `_raw/`, `citpy/`, `manifest.json`, plus + `*.tmp` for partial-write temps), so the producer-internal databento bronze under + `_raw/databento/` never leaves the Windows box. - **Auth:** key-based SSH only. A scheduled task cannot type a passphrase, so use a dedicated key with `ssh -o BatchMode=yes`, never a password prompt. +- **cwRsync gotcha:** a Cygwin rsync (what `choco install rsync` gives you) must drive the + ssh that *ships with it*, not the native Windows OpenSSH. Native ssh corrupts rsync's + binary stream and fails with `connection unexpectedly closed (0 bytes received so far)`, + even though a plain `ssh host echo ok` works fine. That Cygwin ssh also has no HOME, so + give it an explicit writable `-o UserKnownHostsFile=`, and use cygdrive (`/cygdrive/c/…`) + paths throughout, including the key. The example script wires all three. **Provider cutover (one-time).** The server previously held a databento-built store, so its `prices/` and `manifests/prices.json` carry databento data under the very keys the Norgate diff --git a/docs/examples/windows/push-to-server.cmd b/docs/examples/windows/push-to-server.cmd index 79a11f9..b65c975 100644 --- a/docs/examples/windows/push-to-server.cmd +++ b/docs/examples/windows/push-to-server.cmd @@ -5,40 +5,61 @@ REM known-consistent moment rather than on a timer that might land mid-run. REM REM Why not robocopy here: robocopy cannot speak SSH, and the dash server is a REM remote VPS, so SMB is off the table (never expose SMB over the internet). -REM This uses rsync, which needs a packaged rsync ON WINDOWS. Two common ones: -REM cwRsync -> cygwin-style paths, e.g. /cygdrive/c/Users/you/cotdata_store -REM WSL -> /mnt/c-style paths, e.g. /mnt/c/Users/you/cotdata_store -REM This file is written for cwRsync. For WSL, prefix the rsync lines with `wsl `, -REM swap /cygdrive/c for /mnt/c, and drop the RSYNC= path (use bare `rsync`). +REM This uses rsync, which needs a packaged rsync ON WINDOWS. cwRsync (a Cygwin +REM build) is the tested one; `choco install rsync` installs exactly that, to +REM rsync.exe at C:\ProgramData\chocolatey\bin\rsync.exe +REM ssh.exe at C:\ProgramData\chocolatey\lib\rsync\tools\bin\ssh.exe +REM (WSL also works: prefix the rsync lines with `wsl `, use /mnt/c paths, and a +REM native rsync/ssh inside the distro. The Cygwin gotchas below do not apply.) +REM +REM -- Three Cygwin-rsync gotchas this file already handles --------------------- +REM 1. Use the ssh that SHIPS WITH rsync, never the native Windows OpenSSH +REM (C:\Windows\System32\OpenSSH\ssh.exe). A Cygwin rsync driving native ssh +REM corrupts rsync's binary stream and dies with +REM "connection unexpectedly closed (0 bytes received so far)". +REM Point SSH_EXE at the bundled Cygwin ssh instead. +REM 2. That Cygwin ssh has no HOME, so it cannot write the default known_hosts and +REM warns "Failed to add the host ... (/known_hosts)". Give it an explicit +REM writable UserKnownHostsFile (created on first connect). +REM 3. All local paths are cygdrive form: /cygdrive/c/... , including the key. +REM ---------------------------------------------------------------------------- REM REM Overwrite the markers below. Do NOT use angle brackets in a .cmd file: cmd REM reads them as redirection and the file fails even on comment lines. -REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form -REM e.g. /cygdrive/c/Users/you/cotdata_store +REM REPLACE_WITH_SSH_EXE_CYG = the ssh that ships with rsync, cygdrive form +REM e.g. /cygdrive/c/ProgramData/chocolatey/lib/rsync/tools/bin/ssh.exe REM REPLACE_WITH_SSH_KEY_CYG = batch SSH private key, cygdrive form REM e.g. /cygdrive/c/Users/you/.ssh/cotdata_push +REM REPLACE_WITH_KNOWN_HOSTS_CYG= a writable known_hosts, cygdrive form +REM e.g. /cygdrive/c/Users/you/.ssh/known_hosts +REM REPLACE_WITH_STORE_PATH_CYG = source store, cygdrive form +REM e.g. /cygdrive/c/Users/you/cotdata_store REM REPLACE_WITH_REMOTE = user@host:/path/to/store (no trailing slash) REM e.g. deploy@dash.example.com:/srv/cotdata_store REM See docs/SYNCING.md ("Dash store") for the exclusions and the one-time cutover. setlocal -set "RSYNC=C:\Program Files\cwRsync\bin\rsync.exe" -set "SRC=REPLACE_WITH_STORE_PATH_CYG" +set "RSYNC=C:\ProgramData\chocolatey\bin\rsync.exe" +set "SSH_EXE=REPLACE_WITH_SSH_EXE_CYG" set "KEY=REPLACE_WITH_SSH_KEY_CYG" +set "KNOWN=REPLACE_WITH_KNOWN_HOSTS_CYG" +set "SRC=REPLACE_WITH_STORE_PATH_CYG" set "DEST=REPLACE_WITH_REMOTE" -set "SSH=ssh -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new" +set "SSH=%SSH_EXE% -i %KEY% -o BatchMode=yes -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%KNOWN%" REM Data first, manifests last, so a manifest never announces parquet that has not REM landed (harmless if reversed; get_prices reads parquet directly). --delete makes REM this a true mirror. The exclusions match the Mac push: REM _cache, _raw producer-internal; _raw/databento (the paid databento bronze) REM rides under _raw and so is excluded, per ADR-0006. -REM citpy consumer-owned notes on the server; excluding it from --delete -REM is what stops the mirror from wiping them. +REM citpy consumer-owned on the server; excluding it from --delete is +REM what stops the mirror from wiping it. REM manifest.json legacy aggregate, resolved last-writer-wins across halves. +REM *.tmp a producer's partial-write temp (atomic write via os.replace); +REM never propagate a half-written file. "%RSYNC%" -az --delete ^ - --exclude "_cache/" --exclude "_raw/" --exclude "citpy/" --exclude "manifest.json" ^ - --exclude "manifests/" ^ + --exclude "_cache/" --exclude "_raw/" --exclude "citpy/" ^ + --exclude "manifest.json" --exclude "*.tmp" --exclude "manifests/" ^ -e "%SSH%" "%SRC%/" "%DEST%/" if %ERRORLEVEL% NEQ 0 ( echo push FAILED, rsync code %ERRORLEVEL% & exit /b %ERRORLEVEL% )