From 07d9aed9c16186ed63c5e44b904992ff993ca96d Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Tue, 4 Aug 2026 22:41:26 -0400 Subject: [PATCH] Deprecate push_data_cache_to_server.sh; the Windows box pushes everything It pushed from the MAC, which cotdata/docs/SYNCING.md documents as a read-only replica rather than a producer. One Windows server produces everything and feeds two replicas, the Mac over SMB and the dash server over SSH, so this was a replica pushing to a replica and whichever side synced second won silently. Deprecated rather than deleted, per the maintainers call and for a reason: this file is referenced in four places in server-side/README.md, and a reader following an obsolete-but-present script that PRINTS its replacement fails more loudly than one following a dangling reference. It now exits 2 unless CROWDMON_ALLOW_LEGACY_PUSH=1. The crowdmon damage panel briefly rode here as an optional fourth payload (#14). That was DEAD CODE: wired into a script that does not run, so the panel had no path to the server at all. Removed here; it now ships via crowdmon/docs/examples/windows/push-panel.cmd, a Windows template modelled on cotdatas push-to-server.cmd. CROWDMON_STORE is a separate store root, so no exclusion-list change to cotdatas push could have carried it. #15, merged an hour ago, told people to run this script from the Windows box. That was wrong: it is not run-it-elsewhere, it is do-not-run-it. Corrected. server-side/README.md section 6 rewritten around the two Windows templates, and the stale data_cache push removed from the release-day block. data_cache and cot_data.db are pushed by nothing: PR #12 moved them to .local-state/cot-analyzer/ and the server rebuilds the cache. If that turns out to be wrong the fix is a third Windows template, not a revival of this script. Co-Authored-By: Claude Opus 5 --- scripts/push_data_cache_to_server.sh | 99 +++++++++++++++------------- server-side/README.md | 58 ++++++++-------- 2 files changed, 85 insertions(+), 72 deletions(-) diff --git a/scripts/push_data_cache_to_server.sh b/scripts/push_data_cache_to_server.sh index cd4914b..97a7419 100755 --- a/scripts/push_data_cache_to_server.sh +++ b/scripts/push_data_cache_to_server.sh @@ -1,8 +1,43 @@ #!/bin/bash # +# ============================================================================ +# DEPRECATED 2026-08-04. Do not use. It is kept, not deleted, because +# server-side/README.md leans on it in four places and a reader following an +# obsolete-but-present script fails more loudly than one following a dangling +# reference. It now refuses to run. +# +# WHY. This pushes from the MAC, and the Mac is a read-only replica rather than a +# producer. cotdata/docs/SYNCING.md documents the real topology: one Windows +# server produces everything and feeds two replicas, the Mac over SMB +# (robocopy /MIR, docs/examples/windows/sync-store.cmd) and this Linux dash +# server over SSH (rsync, docs/examples/windows/push-to-server.cmd). A Mac push +# is a replica pushing to a replica, and whichever side syncs second wins +# silently. +# +# It had also been unable to run since PR #12 routed runtime state out of the +# repo: the preflight still demanded ./data/cot_data.db, which moved to +# .local-state/cot-analyzer/, so it exited 1. The stale ./data_cache was the +# quieter half, a Jul 28 copy that would have been pushed over the server's +# without comment had the preflight passed. Both are symptoms of a script nobody +# had run in a while, not bugs to fix. +# +# WHAT REPLACES IT, per payload: +# cotdata_store cotdata/docs/examples/windows/push-to-server.cmd +# crowdmon panel crowdmon/docs/examples/windows/push-panel.cmd +# data_cache/, db regenerated on the server by cotmetrics. If that turns out +# to be wrong, the fix is a Windows-side push modelled on the +# two above, not a revival of this file. +# +# The crowdmon damage panel briefly rode here as an optional fourth payload +# (#14). That was dead code: it was wired into a script that does not run. The +# payload is gone from here and lives in crowdmon's own Windows template. +# ============================================================================ +# +# Original header follows. +# # Push the data the server cannot generate for itself. # -# Three REQUIRED payloads and one optional: +# Exactly three payloads: # # cotdata_store Futures prices and contract specs. Norgate is Windows-only, so # the server can never produce these. This lives OUTSIDE the repo, @@ -11,25 +46,10 @@ # data_cache/ Derived per-instrument parquet, plus the options snapshots. # data/cot_data.db The SQLite database the app reads. # -# Optional, synced only when CROWDMON_STORE is set and present: -# -# crowdmon_store crowdmon's damage panel, built by crowdmon/bin/publish_damage.py on -# THIS machine before this script runs. Optional rather than required -# because the app -# runs perfectly well without it: the /damage page renders a "not -# available" card and every other page is unaffected. Making it a hard -# requirement would block a routine data sync on a publisher that had not -# run, which is the wrong trade for a page nothing else depends on. -# # Deliberately NOT shipped: the rest of data/, about 780M of CFTC archives that the ETL # downloads from cftc.gov itself (xls_data, cot_data) and CSV exports that the app # writes rather than reads (csv_data). The earlier version sent all of it. # -# RUN THIS FROM THE WINDOWS/NORGATE PRODUCER. That is where the prices and the contract -# specs exist, and it is therefore where every payload below has to be built before it can be -# shipped. Running it from a machine that merely READS a synced store will push whatever that -# machine happens to have, which is a copy of an older sync. -# # Dry run by default. Nothing moves until you pass --push. # # Usage: @@ -40,10 +60,27 @@ # HOST=user@example.com ssh target (empty means a local copy, used by the tests) # REMOTE_ROOT=/path workspace root on the server # COTDATA_STORE=/path source store; defaults to the app's own env var -# CROWDMON_STORE=/path crowdmon damage panel; skipped entirely when unset # set -euo pipefail +# Refuse rather than warn. A deprecated sync that still runs is one that will be run. +cat >&2 <<'DEPRECATED' +push_data_cache_to_server.sh is DEPRECATED and does nothing. + +The Mac is a read-only replica, not a producer. Everything the server needs is +pushed from the Windows box: + + cotdata store cotdata/docs/examples/windows/push-to-server.cmd + crowdmon panel crowdmon/docs/examples/windows/push-panel.cmd + +See cotdata/docs/SYNCING.md for the topology and cotdata/docs/WINDOWS_SCHEDULING.md +for how those are scheduled. Set CROWDMON_ALLOW_LEGACY_PUSH=1 to run this anyway, +which you almost certainly do not want: it would push a replica over a replica. +DEPRECATED +if [ "${CROWDMON_ALLOW_LEGACY_PUSH:-0}" != "1" ]; then + exit 2 +fi + HOST="${HOST:?set HOST to the deploy target, e.g. HOST=user@your-server}" REMOTE_ROOT="${REMOTE_ROOT:-/root/trading_workspace}" @@ -56,10 +93,6 @@ cd "$PROJECT_ROOT" STORE_SRC="${COTDATA_STORE:-$(cd "$PROJECT_ROOT/../.." && pwd)/cotdata_store}" # On the server the store is a sibling of the workspace, not inside it. STORE_DEST="$(dirname "$REMOTE_ROOT")/cotdata_store" -# Sibling of the workspace, exactly like cotdata_store, which is where -# src/components/crowdmon_artifact.py looks by default. -CROWDMON_SRC="${CROWDMON_STORE:-}" -CROWDMON_DEST="$(dirname "$REMOTE_ROOT")/crowdmon_store" APP_DEST="$REMOTE_ROOT/cot-analyzer" PUSH=0 @@ -87,13 +120,6 @@ if [ "$missing" -ne 0 ]; then exit 1 fi -# The optional payload is checked separately and never blocks: an unset or absent -# CROWDMON_STORE is a normal state, not a broken sync. -if [ -n "$CROWDMON_SRC" ] && [ ! -d "$CROWDMON_SRC" ]; then - echo "note: CROWDMON_STORE=$CROWDMON_SRC does not exist; skipping the damage panel." >&2 - CROWDMON_SRC="" -fi - # .DS_Store rides along from macOS otherwise; it is noise on a Linux box. RSYNC_OPTS=(-avz --no-o --no-g --human-readable --exclude='.DS_Store') if [ "$PUSH" -eq 1 ]; then @@ -119,20 +145,11 @@ echo " data_cache ./data_cache" echo " -> $(remote "$APP_DEST/data_cache")" echo " database ./data/cot_data.db" echo " -> $(remote "$APP_DEST/data/")" -if [ -n "$CROWDMON_SRC" ]; then - echo " crowdmon $CROWDMON_SRC" - echo " -> $(remote "$CROWDMON_DEST")" -else - echo " crowdmon (skipped: CROWDMON_STORE unset or absent)" -fi echo if [ "$PUSH" -eq 1 ] && [ -n "$HOST" ]; then # rsync will not create missing parent directories on its own. ssh "$HOST" "mkdir -p '$STORE_DEST' '$APP_DEST/data_cache' '$APP_DEST/data'" - if [ -n "$CROWDMON_SRC" ]; then - ssh "$HOST" "mkdir -p '$CROWDMON_DEST'" - fi fi # Trailing slash on the sources: copy the CONTENTS into the destination, so a rename @@ -146,14 +163,6 @@ rsync "${RSYNC_OPTS[@]}" ./data_cache/ "$(remote "$APP_DEST/data_cache/")" echo "--- database ---" rsync "${RSYNC_OPTS[@]}" ./data/cot_data.db "$(remote "$APP_DEST/data/")" -if [ -n "$CROWDMON_SRC" ]; then - # --delete so a pruned week disappears on the server too. The publisher keeps a rolling - # window and the manifest names the current one, so an orphaned old directory is dead - # weight rather than a fallback. - echo "--- crowdmon damage panel ---" - rsync "${RSYNC_OPTS[@]}" --delete "$CROWDMON_SRC/" "$(remote "$CROWDMON_DEST/")" -fi - echo if [ "$PUSH" -eq 1 ]; then echo "[done] Sync complete." diff --git a/server-side/README.md b/server-side/README.md index a936c7f..e452f3d 100644 --- a/server-side/README.md +++ b/server-side/README.md @@ -230,25 +230,33 @@ default price path. Three things are gitignored and must be copied from the machine that produced them. Run these **from the producer**, not the server. -Use the script, which knows all three payloads and dry-runs by default: +> **`scripts/push_data_cache_to_server.sh` is DEPRECATED and refuses to run.** It pushed +> from the **Mac**, which is a read-only replica rather than a producer, so it was a replica +> pushing to a replica. `cotdata/docs/SYNCING.md` documents the real topology: one Windows +> server produces everything and feeds two replicas, the Mac over SMB and this dash server +> over SSH. The script is kept rather than deleted so a reader who finds it here gets a +> message naming its replacement instead of a dangling reference. + +Everything is pushed **from the Windows producer**, by Task Scheduler running scheduler +copies of two templates. See `cotdata/docs/WINDOWS_SCHEDULING.md` for how they are wired +and chained behind `errorlevel` guards. + +| payload | pushed by | +|---|---| +| the cotdata store (~234M) | `cotdata/docs/examples/windows/push-to-server.cmd` | +| the crowdmon damage panel (~35M) | `crowdmon/docs/examples/windows/push-panel.cmd` | + +**`data_cache/` and `data/cot_data.db` are not pushed by either.** They are cotmetrics +runtime state, and PR #12 moved them out of this repo into `.local-state/cot-analyzer/`, +so the deprecated script could not have shipped them anyway. The server regenerates the +cache on first use. **If that turns out to be wrong**, the fix is a third Windows-side +template modelled on the two above, not a revival of the Mac script. + +The store push by hand, for reference, is: ```bash -./scripts/push_data_cache_to_server.sh # show what would move -./scripts/push_data_cache_to_server.sh --push # transfer -``` - -It sends the store (~234M), `data_cache/` (~85M) and `data/cot_data.db` (~37M), and -refuses to run at all if any of the three is missing locally, rather than leaving the -server with a partial set it cannot boot from. Override `HOST`, `REMOTE_ROOT` or -`COTDATA_STORE` if your layout differs. - -The equivalent by hand: - -```bash -rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ - ~/code/cotdata_store/ USER@HOST:/root/cotdata_store/ rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ - ./data_cache/ USER@HOST:/root/trading_workspace/cot-analyzer/data_cache/ + /path/to/cotdata_store/ USER@HOST:/root/cotdata_store/ ``` `data/` is a third gitignored directory, and most of it does **not** need to travel: @@ -257,14 +265,9 @@ rsync -avz --no-o --no-g --progress --exclude='.DS_Store' \ `cftc.gov` and extracts on its own. - `csv_data` (221M) is export output the app writes, not an input it reads. -The part worth syncing is the SQLite database: - -```bash -rsync -avz --no-o --no-g --progress ./data/cot_data.db USER@HOST:/root/trading_workspace/cot-analyzer/data/ -``` - -Syncing the whole of `data/` works but ships about 800M to save a download the server -does anyway. +None of it is pushed today. The SQLite database moved to `.local-state/cot-analyzer/` +in PR #12 and the server maintains its own; syncing the whole of `data/` would ship about +800M to save a download the server does anyway. ## 7. Install the service @@ -344,12 +347,13 @@ ssh USER@HOST ' cd /root/trading_workspace/cotmetrics && git pull && cd /root/trading_workspace/cot-analyzer && git pull ' -# only if the release changed derived-cache contents -rsync -avz --no-o --no-g ./data_cache/ USER@HOST:/root/trading_workspace/cot-analyzer/data_cache/ - ssh USER@HOST 'systemctl restart cot-analyzer && systemctl status cot-analyzer' ``` +> A release that changes derived-cache contents no longer ships a cache from here: the +> server rebuilds it. The `data_cache` push that used to sit in this block went with +> `push_data_cache_to_server.sh`, per ยง6. + Editable installs mean a `git pull` in `cotmetrics` or `cotdata` needs no reinstall, but it **does** need the restart, because the running process already imported them.