From d9eafbc119b92a4ef19c3f44fce57689eee527df Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Sat, 8 Aug 2026 23:44:45 -0400 Subject: [PATCH 1/2] Fail at startup when MARKETDATA_STORE is missing, not at the first price read ADR-0007 step 4 landed in cotmetrics, so bars now arrive through marketdata.get_bars and a price read needs a second store. Without it the failure is quiet in the worst way: the import succeeds, the app binds :5001, every COT page renders off the other store, and only the first chart wanting bars raises. That reads as a UI regression rather than a missing environment variable, which is exactly the confusion the COTDATA_STORE check upstairs exists to prevent. Same check, same shape, same reason. The second branch is the collision case. The two stores may share a synced parent but never one directory, because each producer rewrites its own manifest.json read-modify-write and a shared manifest loses entries. Pointing both variables at one path is the easy way to arrive there, so refuse it here rather than let a producer discover it later. Verified all three branches by hand: unset exits 1 naming the variable and the .env path, equal paths exit 1 naming the manifest reason, distinct paths fall through to the app. bash -n clean, ruff clean on src and tests, 131 tests pass. Co-Authored-By: Claude Opus 5 --- run-local.sh | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/run-local.sh b/run-local.sh index 9f6a542..c84845f 100755 --- a/run-local.sh +++ b/run-local.sh @@ -51,6 +51,26 @@ if [ -z "$COTDATA_STORE" ]; then exit 1 fi +# Same check for the second store. ADR-0007 moved bars out of cotdata, so since +# cotmetrics repointed at marketdata.get_bars a price read needs MARKETDATA_STORE and +# raises without it. That failure is quiet in the worst way: the import succeeds, the +# app binds its port, every COT page renders, and only the first chart wanting bars +# dies. Check it here so a run that cannot serve prices says so before it starts. +# It is a SEPARATE root, never COTDATA_STORE: both producers write manifest.json with +# a read-modify-write, so pointing them at one directory loses entries. +if [ -z "$MARKETDATA_STORE" ]; then + echo "run-local.sh: MARKETDATA_STORE is not set." >&2 + echo " Add it to $REPO/.env (gitignored), e.g." >&2 + echo " MARKETDATA_STORE=/path/to/marketdata_store" >&2 + echo " or export it before running. It must NOT be COTDATA_STORE." >&2 + exit 1 +elif [ "$MARKETDATA_STORE" = "$COTDATA_STORE" ]; then + echo "run-local.sh: MARKETDATA_STORE and COTDATA_STORE are the same path." >&2 + echo " They are two stores. Sharing one manifest.json loses entries, because" >&2 + echo " each producer rewrites it read-modify-write. Point them at separate dirs." >&2 + exit 1 +fi + # CIT PY research notes (dated .md/.txt the /citpy page links) are produced by a separate # tool and only read here. They used to default to $COTDATA_STORE/citpy, which was wrong # twice over: the store is a producer/consumer artifact that a sync mirrors, so a --delete From 0fa852f75984826a5cce09876ec8c0d078e0e3c0 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Sat, 8 Aug 2026 23:44:59 -0400 Subject: [PATCH 2/2] Three siblings to clone and two stores to point at, and where the bars are not The install instructions were a step behind the code and would have led the next clone into the error that prompted this. README named two siblings and two editable installs; there are three. Missing ../marketdata does not fail at install time, it fails at import, because cotmetrics/__init__.py re-exports signals and signals imports marketdata at module level, so the error arrives as a ModuleNotFoundError from a package nothing in this repo mentions. That traceback is now in README verbatim. The same lag ran through the rest. README said prices are read through cotdata.get_prices; cotmetrics/src no longer calls it anywhere. ARCHITECTURE said cotmetrics depends on cotdata alone, and that caches are busted by two counters. There are three, and marketdata is the one that carries the weight: the case the guard was written for, reconstructed volume promoted to default, is a PRICE schema bump, so watching cotdata alone would leave uncovered the only case it exists for. data-path.md needed more than a correction, because the two ADR-0007 steps are out of step with each other and that is worth recording rather than smoothing over. Step 4 (repoint consumers) is done. Step 2 (move the producer and the bars) is on ice. So the read points at $MARKETDATA_STORE/bars/futures/ while 99 price parquets sit in $COTDATA_STORE/prices/ where the producer keeps writing them, and a futures read against a store holding only equities returns an empty frame with no exception. The old diagram is kept and marked superseded, since its vendor-abstraction argument still holds; the measured state goes in a new section beside it with a one-line check a reader can run. Measured 2026-08-08 against the working tree and both live stores, not read off the docs, which is how the staleness surfaced. ruff clean on src and tests, 131 tests pass. Co-Authored-By: Claude Opus 5 --- README.md | 59 +++++++++++++++++++++++++++++--------------- docs/ARCHITECTURE.md | 14 ++++++++--- docs/data-path.md | 44 +++++++++++++++++++++++++++++++-- 3 files changed, 91 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index a26644e..dcacbee 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ A Dash/Plotly web app for exploring [CFTC](https://www.cftc.gov/MarketReports/CommitmentsofTraders/index.htm) Commitments of Traders positioning. It's the UI layer on top of the -[`cotmetrics`](https://github.com/mspinola/cotmetrics) metrics library and the -[`cotdata`](https://github.com/mspinola/cotdata) store; it computes no metrics of its -own. Given the CFTC legacy report, it will: +[`cotmetrics`](https://github.com/mspinola/cotmetrics) metrics library, which reads +positioning from the [`cotdata`](https://github.com/mspinola/cotdata) store and daily bars +from the [`marketdata`](https://github.com/mspinola/marketdata) store; it computes no +metrics of its own. Given the CFTC legacy report, it will: * generate CSV files with per-symbol data * generate RealTest data as an Event List for each symbol @@ -19,9 +20,9 @@ own. Given the CFTC legacy report, it will: ![Example Positioning Table](./docs/cot-positioning-table.png) > **This is a source application, not a PyPI package.** Clone it and run it. It expects -> the `cotdata` and `cotmetrics` siblings checked out alongside it (`../cotdata`, -> `../cotmetrics`) — `requirements.txt` installs them editable — and a populated -> `COTDATA_STORE`. See Setup below. +> three siblings checked out alongside it (`../cotdata`, `../marketdata`, `../cotmetrics`), +> which `requirements.txt` installs editable, and **two** populated stores, +> `COTDATA_STORE` and `MARKETDATA_STORE`. See Setup below. ## COT Data @@ -34,30 +35,44 @@ Check out the sibling repos alongside this one first, so the editable installs i ```bash git clone https://github.com/mspinola/cotdata +git clone https://github.com/mspinola/marketdata git clone https://github.com/mspinola/cotmetrics git clone https://github.com/mspinola/cot-analyzer cd cot-analyzer uv venv source .venv/bin/activate -uv pip install -r requirements.txt # includes the editable siblings (-e ../cotdata, -e ../cotmetrics) +uv pip install -r requirements.txt # editable siblings: -e ../cotdata, -e ../marketdata, -e ../cotmetrics ``` -`requirements.txt` pulls in two local sibling packages, both editable: +`requirements.txt` pulls in three local sibling packages, all editable: * **cotmetrics** (`-e ../cotmetrics[options]`) is the data and metrics layer. It - owns the indexer, the COT index and signals, and options data. COT/price data is - produced by cotdata and read from the shared store. This repo is the Dash + owns the indexer, the COT index and signals, and options data. This repo is the Dash application on top of it and computes no metrics of its own. -* **cotdata** (`-e ../cotdata`) is the store beneath that. Prices and COT are read through - `cotdata.get_prices` / `cotdata.get_cot`, backed by a shared file store. +* **cotdata** (`-e ../cotdata`) is CFTC positioning, read through `cotdata.get_cot`. +* **marketdata** (`-e ../marketdata`) is daily bars, read through `marketdata.get_bars`. + ADR-0007 moved prices out of cotdata, so this is a separate package **and** a separate + store. The distribution name on PyPI is `crucible-marketdata` (plain `marketdata` there + is an unrelated abandoned project); the import is still `marketdata`. -Point `COTDATA_STORE` at that store. Put it in `.env` at the repo root (gitignored) so both -the launcher and the systemd unit pick it up: +Skipping the `marketdata` clone does not fail at install time. It fails at import, because +`cotmetrics/__init__.py` re-exports `signals`, which imports `marketdata` at module level: + +``` +ModuleNotFoundError: No module named 'marketdata' +``` + +Point `COTDATA_STORE` and `MARKETDATA_STORE` at the two stores. Put both in `.env` at the +repo root (gitignored) so the launcher and the systemd unit pick them up: ```bash -echo 'COTDATA_STORE=~/code/cotdata_store' >> .env +printf 'COTDATA_STORE=~/code/cotdata_store\nMARKETDATA_STORE=~/code/marketdata_store\n' >> .env ``` +They must be **different** directories. Each store's producer rewrites its own +`manifest.json` read-modify-write, so one shared manifest silently loses entries. They may +sit under a common synced parent. + ## Run ```bash @@ -70,9 +85,9 @@ Use the launcher rather than `python src/main.py` directly. It changes to the re `data_cache/`, points `COTMETRICS_PARAMS` and `COT_VIZ_CONFIG` at the sibling `../cotmetrics-config/params.yaml` when that private config repo is checked out alongside (otherwise the metrics and viz layers fall back to the packaged sample and warn), sources -`.env`, and fails loudly if `COTDATA_STORE` is unset rather than surfacing later as -confusing empty data. You can still run `python src/main.py` directly if you export those -variables yourself. +`.env`, and fails loudly if `COTDATA_STORE` or `MARKETDATA_STORE` is unset (or if they are +set to the same path) rather than surfacing later as confusing empty data. You can still +run `python src/main.py` directly if you export those variables yourself. ## Configuration @@ -98,6 +113,9 @@ login sudo apt install git pip pip install uv curl -LsSf https://astral.sh/uv/install.sh | sh +git clone git@github.com:mspinola/cotdata.git +git clone git@github.com:mspinola/marketdata.git +git clone git@github.com:mspinola/cotmetrics.git git clone git@github.com:mspinola/cot-analyzer.git cd cot-analyzer @@ -119,8 +137,9 @@ all: do the heavy data work on the producer machine and sync the binaries over S rsync -avz --no-o --no-g --progress ./data_cache/ user@your-server-ip:/path/to/server/cot-analyzer/data_cache/ ``` -The server also needs `COTDATA_STORE` synced. Most of `./data` does **not** need to -travel, since the CFTC archives in it are downloaded and the CSV exports are generated. +The server also needs both stores synced, `COTDATA_STORE` and `MARKETDATA_STORE`. Most of +`./data` does **not** need to travel, since the CFTC archives in it are downloaded and the +CSV exports are generated. **See [server-side/README.md](server-side/README.md)** for the full setup: which repos the server needs, the complete environment variable list, exactly what to sync, and the diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 534c24e..044bcfc 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -30,8 +30,10 @@ The system is decoupled into logical modules, ensuring data engineering pipeline ### The repo boundary: cot-analyzer and cotmetrics The data and metrics layer lives in a **separate package, `cotmetrics`**, installed as an -editable sibling (`requirements.txt` carries `-e ../cotmetrics[options]`, which -in turn depends on `-e ../cotdata`). This repo holds the Dash application and nothing else. +editable sibling (`requirements.txt` carries `-e ../cotmetrics[options]`, which in turn +depends on `-e ../cotdata` for CFTC positioning and, since ADR-0007 moved bars out of +cotdata, on `-e ../marketdata` for daily prices). This repo holds the Dash application and +nothing else. The split matters when reading the rest of this document, so as a map: @@ -76,9 +78,13 @@ the notification rather than the modelling. with generic alias columns (`comms_idx`, `comms_zscore`, `willco`, `oi_zscore` and so on) already resolved for the requested lookback and basis. Every alias moves together, so a caller can never pair a normalized index with a raw z-score inside one condition set. -- Caches are busted by two counters: the upstream `cotdata` schema version, and +- Caches are busted by three counters, kept as separate keys in the cache marker rather + than collapsed: the `cotdata` schema version, the `marketdata` schema version, and `cotmetrics.constants.METRICS_CACHE_VERSION` for value-only changes in our own indicator - logic, which the per-symbol column-presence guards cannot see. + logic, which the per-symbol column-presence guards cannot see. Both stores are watched + because the case the guard was written for, reconstructed volume promoted to default, is + a **price** schema bump, and prices live in `marketdata` since ADR-0007. A cache written + before that split carries no marketdata key, reads as 0, and rebuilds once. ### E. The User Interface (`src/app_cot.py` & `src/pages/`) - A multi-page Single Page Application (SPA) built on Plotly Dash. diff --git a/docs/data-path.md b/docs/data-path.md index 1421f30..6989c47 100644 --- a/docs/data-path.md +++ b/docs/data-path.md @@ -4,9 +4,12 @@ cot-analyzer never talks to a data vendor. It never even calls the price API dir Every number on the dashboard arrives through two layers of indirection, and the store is the boundary where vendor identity stops mattering. -Verified 2026-07-26 by reading the source, not the docs. +Verified 2026-07-26 by reading the source, not the docs. **Re-verified 2026-08-08, and the +read path changed**: ADR-0007 step 4 landed in `cotmetrics` (commit `1290ec0`), so bars now +arrive through `marketdata.get_bars` and `cotdata.get_prices` is no longer called from +`cotmetrics/src` at all. See "The read path today" below before trusting the older diagram. -## Current path +## Current path (superseded 2026-08-08, kept for the vendor argument) ```mermaid flowchart TD @@ -99,6 +102,43 @@ reconstructed volume on one store and front-month volume on the other, same call not on databento at all and fall back to Yahoo, so a databento deployment is really a databento-plus-Yahoo deployment. +## The read path today + +Measured 2026-08-08 against the working tree and both live stores. + +**The consumers have moved. The bytes have not.** ADR-0007 is shipping as separate steps and +these two are out of step with each other: + +| | state | +|---|---| +| step 4, repoint consumers | **done**. `cotmetrics` reads `marketdata.get_bars(symbol, "backadj")` at `CotIndexer.py:752`, `signals.py:1030`, `options_data.py:366`. `grep get_prices cotmetrics/src` returns nothing | +| step 2, move the producer code and the bars | **on ice**. `providers/databento.py` has no owner | + +So the read now points at `$MARKETDATA_STORE/bars/futures/`, while 99 price parquets still +sit in `$COTDATA_STORE/prices/` where the producer keeps writing them. + +**This fails silently, which is the part to remember.** A futures read against a store with +no `bars/futures/` does not raise. It returns an empty frame: + +```python +marketdata.get_bars("GC", "backadj") # -> 0 rows, no error +``` + +The app still boots, binds :5001, and renders every COT page, because positioning comes from +the other store and is unaffected. Only price overlays and the price-dependent signals go +blank. Both failure modes below therefore look like a UI regression rather than a data one: + +- `MARKETDATA_STORE` unset: raises, but late, on the first bar read rather than at startup. + `run-local.sh` checks it upfront for this reason. +- `MARKETDATA_STORE` set and populated with `bars/equities/` only: no error anywhere. + +Until the futures bars are imported into the marketdata store, a local checkout renders +positioning without prices. Confirm before trusting a chart: + +```bash +python -c "import marketdata; print(len(marketdata.get_bars('GC','backadj')))" +``` + ## Target path after ADR-0007 `cotdata` keeps CFTC positioning only. All bars move to `marketdata`, which already holds