diff --git a/README.md b/README.md index a26644e..dcacbee 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ A Dash/Plotly web app for exploring [CFTC](https://www.cftc.gov/MarketReports/CommitmentsofTraders/index.htm) Commitments of Traders positioning. It's the UI layer on top of the -[`cotmetrics`](https://github.com/mspinola/cotmetrics) metrics library and the -[`cotdata`](https://github.com/mspinola/cotdata) store; it computes no metrics of its -own. Given the CFTC legacy report, it will: +[`cotmetrics`](https://github.com/mspinola/cotmetrics) metrics library, which reads +positioning from the [`cotdata`](https://github.com/mspinola/cotdata) store and daily bars +from the [`marketdata`](https://github.com/mspinola/marketdata) store; it computes no +metrics of its own. Given the CFTC legacy report, it will: * generate CSV files with per-symbol data * generate RealTest data as an Event List for each symbol @@ -19,9 +20,9 @@ own. Given the CFTC legacy report, it will: ![Example Positioning Table](./docs/cot-positioning-table.png) > **This is a source application, not a PyPI package.** Clone it and run it. It expects -> the `cotdata` and `cotmetrics` siblings checked out alongside it (`../cotdata`, -> `../cotmetrics`) — `requirements.txt` installs them editable — and a populated -> `COTDATA_STORE`. See Setup below. +> three siblings checked out alongside it (`../cotdata`, `../marketdata`, `../cotmetrics`), +> which `requirements.txt` installs editable, and **two** populated stores, +> `COTDATA_STORE` and `MARKETDATA_STORE`. See Setup below. ## COT Data @@ -34,30 +35,44 @@ Check out the sibling repos alongside this one first, so the editable installs i ```bash git clone https://github.com/mspinola/cotdata +git clone https://github.com/mspinola/marketdata git clone https://github.com/mspinola/cotmetrics git clone https://github.com/mspinola/cot-analyzer cd cot-analyzer uv venv source .venv/bin/activate -uv pip install -r requirements.txt # includes the editable siblings (-e ../cotdata, -e ../cotmetrics) +uv pip install -r requirements.txt # editable siblings: -e ../cotdata, -e ../marketdata, -e ../cotmetrics ``` -`requirements.txt` pulls in two local sibling packages, both editable: +`requirements.txt` pulls in three local sibling packages, all editable: * **cotmetrics** (`-e ../cotmetrics[options]`) is the data and metrics layer. It - owns the indexer, the COT index and signals, and options data. COT/price data is - produced by cotdata and read from the shared store. This repo is the Dash + owns the indexer, the COT index and signals, and options data. This repo is the Dash application on top of it and computes no metrics of its own. -* **cotdata** (`-e ../cotdata`) is the store beneath that. Prices and COT are read through - `cotdata.get_prices` / `cotdata.get_cot`, backed by a shared file store. +* **cotdata** (`-e ../cotdata`) is CFTC positioning, read through `cotdata.get_cot`. +* **marketdata** (`-e ../marketdata`) is daily bars, read through `marketdata.get_bars`. + ADR-0007 moved prices out of cotdata, so this is a separate package **and** a separate + store. The distribution name on PyPI is `crucible-marketdata` (plain `marketdata` there + is an unrelated abandoned project); the import is still `marketdata`. -Point `COTDATA_STORE` at that store. Put it in `.env` at the repo root (gitignored) so both -the launcher and the systemd unit pick it up: +Skipping the `marketdata` clone does not fail at install time. It fails at import, because +`cotmetrics/__init__.py` re-exports `signals`, which imports `marketdata` at module level: + +``` +ModuleNotFoundError: No module named 'marketdata' +``` + +Point `COTDATA_STORE` and `MARKETDATA_STORE` at the two stores. Put both in `.env` at the +repo root (gitignored) so the launcher and the systemd unit pick them up: ```bash -echo 'COTDATA_STORE=~/code/cotdata_store' >> .env +printf 'COTDATA_STORE=~/code/cotdata_store\nMARKETDATA_STORE=~/code/marketdata_store\n' >> .env ``` +They must be **different** directories. Each store's producer rewrites its own +`manifest.json` read-modify-write, so one shared manifest silently loses entries. They may +sit under a common synced parent. + ## Run ```bash @@ -70,9 +85,9 @@ Use the launcher rather than `python src/main.py` directly. It changes to the re `data_cache/`, points `COTMETRICS_PARAMS` and `COT_VIZ_CONFIG` at the sibling `../cotmetrics-config/params.yaml` when that private config repo is checked out alongside (otherwise the metrics and viz layers fall back to the packaged sample and warn), sources -`.env`, and fails loudly if `COTDATA_STORE` is unset rather than surfacing later as -confusing empty data. You can still run `python src/main.py` directly if you export those -variables yourself. +`.env`, and fails loudly if `COTDATA_STORE` or `MARKETDATA_STORE` is unset (or if they are +set to the same path) rather than surfacing later as confusing empty data. You can still +run `python src/main.py` directly if you export those variables yourself. ## Configuration @@ -98,6 +113,9 @@ login sudo apt install git pip pip install uv curl -LsSf https://astral.sh/uv/install.sh | sh +git clone git@github.com:mspinola/cotdata.git +git clone git@github.com:mspinola/marketdata.git +git clone git@github.com:mspinola/cotmetrics.git git clone git@github.com:mspinola/cot-analyzer.git cd cot-analyzer @@ -119,8 +137,9 @@ all: do the heavy data work on the producer machine and sync the binaries over S rsync -avz --no-o --no-g --progress ./data_cache/ user@your-server-ip:/path/to/server/cot-analyzer/data_cache/ ``` -The server also needs `COTDATA_STORE` synced. Most of `./data` does **not** need to -travel, since the CFTC archives in it are downloaded and the CSV exports are generated. +The server also needs both stores synced, `COTDATA_STORE` and `MARKETDATA_STORE`. Most of +`./data` does **not** need to travel, since the CFTC archives in it are downloaded and the +CSV exports are generated. **See [server-side/README.md](server-side/README.md)** for the full setup: which repos the server needs, the complete environment variable list, exactly what to sync, and the diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 534c24e..044bcfc 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -30,8 +30,10 @@ The system is decoupled into logical modules, ensuring data engineering pipeline ### The repo boundary: cot-analyzer and cotmetrics The data and metrics layer lives in a **separate package, `cotmetrics`**, installed as an -editable sibling (`requirements.txt` carries `-e ../cotmetrics[options]`, which -in turn depends on `-e ../cotdata`). This repo holds the Dash application and nothing else. +editable sibling (`requirements.txt` carries `-e ../cotmetrics[options]`, which in turn +depends on `-e ../cotdata` for CFTC positioning and, since ADR-0007 moved bars out of +cotdata, on `-e ../marketdata` for daily prices). This repo holds the Dash application and +nothing else. The split matters when reading the rest of this document, so as a map: @@ -76,9 +78,13 @@ the notification rather than the modelling. with generic alias columns (`comms_idx`, `comms_zscore`, `willco`, `oi_zscore` and so on) already resolved for the requested lookback and basis. Every alias moves together, so a caller can never pair a normalized index with a raw z-score inside one condition set. -- Caches are busted by two counters: the upstream `cotdata` schema version, and +- Caches are busted by three counters, kept as separate keys in the cache marker rather + than collapsed: the `cotdata` schema version, the `marketdata` schema version, and `cotmetrics.constants.METRICS_CACHE_VERSION` for value-only changes in our own indicator - logic, which the per-symbol column-presence guards cannot see. + logic, which the per-symbol column-presence guards cannot see. Both stores are watched + because the case the guard was written for, reconstructed volume promoted to default, is + a **price** schema bump, and prices live in `marketdata` since ADR-0007. A cache written + before that split carries no marketdata key, reads as 0, and rebuilds once. ### E. The User Interface (`src/app_cot.py` & `src/pages/`) - A multi-page Single Page Application (SPA) built on Plotly Dash. diff --git a/docs/data-path.md b/docs/data-path.md index 1421f30..6989c47 100644 --- a/docs/data-path.md +++ b/docs/data-path.md @@ -4,9 +4,12 @@ cot-analyzer never talks to a data vendor. It never even calls the price API dir Every number on the dashboard arrives through two layers of indirection, and the store is the boundary where vendor identity stops mattering. -Verified 2026-07-26 by reading the source, not the docs. +Verified 2026-07-26 by reading the source, not the docs. **Re-verified 2026-08-08, and the +read path changed**: ADR-0007 step 4 landed in `cotmetrics` (commit `1290ec0`), so bars now +arrive through `marketdata.get_bars` and `cotdata.get_prices` is no longer called from +`cotmetrics/src` at all. See "The read path today" below before trusting the older diagram. -## Current path +## Current path (superseded 2026-08-08, kept for the vendor argument) ```mermaid flowchart TD @@ -99,6 +102,43 @@ reconstructed volume on one store and front-month volume on the other, same call not on databento at all and fall back to Yahoo, so a databento deployment is really a databento-plus-Yahoo deployment. +## The read path today + +Measured 2026-08-08 against the working tree and both live stores. + +**The consumers have moved. The bytes have not.** ADR-0007 is shipping as separate steps and +these two are out of step with each other: + +| | state | +|---|---| +| step 4, repoint consumers | **done**. `cotmetrics` reads `marketdata.get_bars(symbol, "backadj")` at `CotIndexer.py:752`, `signals.py:1030`, `options_data.py:366`. `grep get_prices cotmetrics/src` returns nothing | +| step 2, move the producer code and the bars | **on ice**. `providers/databento.py` has no owner | + +So the read now points at `$MARKETDATA_STORE/bars/futures/`, while 99 price parquets still +sit in `$COTDATA_STORE/prices/` where the producer keeps writing them. + +**This fails silently, which is the part to remember.** A futures read against a store with +no `bars/futures/` does not raise. It returns an empty frame: + +```python +marketdata.get_bars("GC", "backadj") # -> 0 rows, no error +``` + +The app still boots, binds :5001, and renders every COT page, because positioning comes from +the other store and is unaffected. Only price overlays and the price-dependent signals go +blank. Both failure modes below therefore look like a UI regression rather than a data one: + +- `MARKETDATA_STORE` unset: raises, but late, on the first bar read rather than at startup. + `run-local.sh` checks it upfront for this reason. +- `MARKETDATA_STORE` set and populated with `bars/equities/` only: no error anywhere. + +Until the futures bars are imported into the marketdata store, a local checkout renders +positioning without prices. Confirm before trusting a chart: + +```bash +python -c "import marketdata; print(len(marketdata.get_bars('GC','backadj')))" +``` + ## Target path after ADR-0007 `cotdata` keeps CFTC positioning only. All bars move to `marketdata`, which already holds diff --git a/run-local.sh b/run-local.sh index 9f6a542..c84845f 100755 --- a/run-local.sh +++ b/run-local.sh @@ -51,6 +51,26 @@ if [ -z "$COTDATA_STORE" ]; then exit 1 fi +# Same check for the second store. ADR-0007 moved bars out of cotdata, so since +# cotmetrics repointed at marketdata.get_bars a price read needs MARKETDATA_STORE and +# raises without it. That failure is quiet in the worst way: the import succeeds, the +# app binds its port, every COT page renders, and only the first chart wanting bars +# dies. Check it here so a run that cannot serve prices says so before it starts. +# It is a SEPARATE root, never COTDATA_STORE: both producers write manifest.json with +# a read-modify-write, so pointing them at one directory loses entries. +if [ -z "$MARKETDATA_STORE" ]; then + echo "run-local.sh: MARKETDATA_STORE is not set." >&2 + echo " Add it to $REPO/.env (gitignored), e.g." >&2 + echo " MARKETDATA_STORE=/path/to/marketdata_store" >&2 + echo " or export it before running. It must NOT be COTDATA_STORE." >&2 + exit 1 +elif [ "$MARKETDATA_STORE" = "$COTDATA_STORE" ]; then + echo "run-local.sh: MARKETDATA_STORE and COTDATA_STORE are the same path." >&2 + echo " They are two stores. Sharing one manifest.json loses entries, because" >&2 + echo " each producer rewrites it read-modify-write. Point them at separate dirs." >&2 + exit 1 +fi + # CIT PY research notes (dated .md/.txt the /citpy page links) are produced by a separate # tool and only read here. They used to default to $COTDATA_STORE/citpy, which was wrong # twice over: the store is a producer/consumer artifact that a sync mirrors, so a --delete