From fb7a1a5b20254bd5217b065c006ff9a79d8c78c6 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Tue, 4 Aug 2026 21:45:26 -0400 Subject: [PATCH] handoff: claim ADR-0007 step 2, with a re-measurement that moved the wrong way Claimed, not started. A work order and a measurement, no code. ADR-0007 accepted the direction on 2026-07-27 and step 2 has not started since. crowdmon CLAUDE.md records it as on ice with nobody owning it, and says not to start without confirming that changed. It changed: asked for directly on 2026-08-04 while deciding which machine publishes the crowdmon damage panel. Re-measured, and step 2 grew 11% while sitting on ice: 1,721 lines to 1,901, with both price providers and prices.py growing and neither CFTC provider moving. The ADR predicted exactly that and the drift is now measured rather than anticipated. It is also not a file move. marketdata/providers/ holds only yfinance.py; the futures domain is declared in adjust.DOMAIN_TIERS so error messages are right, but nothing implements it. Step 2 WRITES a Norgate futures provider using cotdata as the reference. Consumers enumerated: crowdmon is 6 modules and 10 call sites, and contract_master.py:34 already documents itself as the site ADR-0007 will repoint. Section 4 is the finding the handoff exists for, and nothing had composed it before. crowdmon needs unadj AND propadj; propadj is derived on read from unadj + backadj; databento owes exactly one series per symbol by ADR-0007 scoping. So a databento-backed store cannot feed crowdmon at all, and only the Windows box can write a store it reads. Not for a Python reason: the Mac runs 3.11 under uv and builds the full panel today. That makes ADR-0007 open question 3 answerable. A synced store is not a temporary state pending step 2, it is the answer, unless the servers interpreter is raised AND a full-tier non-Norgate vendor appears. Co-Authored-By: Claude Opus 5 --- ...6-08-04-adr7-step2-price-producer-split.md | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md diff --git a/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md new file mode 100644 index 0000000..571c4a1 --- /dev/null +++ b/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md @@ -0,0 +1,194 @@ +# Handoff: ADR-0007 step 2, move price production out of `cotdata` + +**Status:** **CLAIMED, NOT STARTED.** This is a work order and a re-measurement, no code +**Date:** 2026-08-04 +**Lives at:** `cotdata/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md` +**Target:** a Claude Code session in a `cotdata` worktree, with `marketdata` beside it +**Decision it executes:** [crucible-stack ADR-0007](https://github.com/mspinola/crucible-stack/blob/main/docs/adr/ADR-0007-cotdata-is-cot-only-bars-live-in-marketdata.md), +**accepted 2026-07-27**. This proposes no new decision and asks for none +**Blocked on:** nothing technical. Three questions in §5 want an answer before code, and two +of them are ADR-0007's own open questions rather than new ones +**Companion:** `crowdmon/docs/design/amendments-2026-08-04.md` §D14 — the consumer-side +record of §4, filed there so the constraint is discoverable from the repo it binds +**Not this handoff:** step 3 (`livebook`'s six `contract_specs()` calls at `required=False`), +which ADR-0007 defers deliberately because it is a live book + +> Announced before the first line of code, per the neighbouring repo's convention. If you +> were about to start it, say so and I will drop it. + +--- + +## 0. Why now, and why this is a re-measurement rather than a proposal + +ADR-0007 settled the direction on 2026-07-27: `cotdata` becomes CFTC positioning only, and +all bars, futures included, move to `marketdata`. Step 1 shipped. Step 3's prerequisite is +half built. **Step 2 has not started**, and `crowdmon`'s `CLAUDE.md` records it as "on ice… +nobody owns it. Do not start it without confirming that has changed." + +That has now changed: the maintainer asked for it directly on 2026-08-04, in the course of +deciding which machine publishes the crowdmon damage panel. + +The one thing this handoff adds to the ADR is a **re-measurement, and the number moved the +wrong way.** + +## 1. Step 2 has grown ~11% while sitting on ice + +Measured 2026-08-04 against `wc -l`, on `main` at `49ebc45`: + +| file | ADR-0007 recorded (2026-07-27) | today | drift | +|---|---:|---:|---:| +| `src/cotdata/providers/databento.py` | 1,015 | **1,099** | +84 | +| `src/cotdata/providers/norgate.py` | 469 | **546** | +77 | +| `src/cotdata/prices.py` | 167 | **186** | +19 | +| `src/cotdata/providers/yfinance.py` | 70 | 70 | — | +| **total** | **1,721** ("roughly 1,700") | **1,901** | **+180** | + +Plus `metadata/contract_specs.parquet` and the `--metadata` action, unchanged in scope. + +ADR-0007 predicted this in as many words: *"`databento.py` arriving after this ADR was +written makes `cotdata` more price-heavy than the measurements above, which strengthens the +case and enlarges step 2 at the same time."* It was right, and the drift is now measured +rather than anticipated: **+180 lines in eight days, both price providers growing, `prices.py` +growing, and neither CFTC provider moving.** + +That is the argument for doing it sooner rather than the argument for doing it at all. The +ADR already made the second one. + +## 2. It is not a file move: `marketdata` has no futures provider + +`marketdata/src/marketdata/providers/` contains `__init__.py` and `yfinance.py`. That is all. + +The `futures` domain **is** declared in `adjust.DOMAIN_TIERS`, which ADR-0007 notes was done +so error messages are correct from the first day, but nothing implements it. So step 2 is +*write the Norgate futures provider into `marketdata`*, using `cotdata`'s as the reference, +and not *move a file across a seam*. + +Whoever takes this should price that difference in before estimating. 1,901 lines of source +to read; the delivered artifact is a provider that did not exist. + +## 3. The consumers, enumerated + +Measured 2026-08-04. Every one of these is a read that step 2 repoints. + +**`crowdmon`** — 6 modules, 10 call sites: + +| module | call | +|---|---| +| `futures/riskunits.py:138` | `cotdata.get_prices(symbol, adjustment="propadj")` | +| `futures/notional.py:88` | `cotdata.get_prices(symbol, adjustment="unadj")` | +| `futures/volume.py:106,117` | `cotdata.get_prices(..., volume="front")` | +| `futures/alignment.py:106` | `cotdata.get_prices(...)["Close"]` | +| `futures/clustering.py:116` | `cotdata.get_prices(...)["Close"]` | +| `futures/contract_master.py:97,100,117,122` | `all_symbols`, `load_manifest`, `store.read_metadata` | + +`contract_master.py:34` already documents itself as the call site ADR-0007 will repoint, and +`store.read_metadata()` is flagged there as the one non-`__all__` cotdata symbol the package +reaches for. That comment is the pre-existing marker for this work. + +**`cotmetrics` / `cot-analyzer`** — ADR-0007 §"measured facts" records that nothing in +`cot-analyzer` calls a price vendor, every live read is `cotdata.get_prices`, and every one of +those is `adjustment="backadj"`. + +**`npf`** — reads futures through `cotdata.get_prices`; ADR-0007's open question about +`propadj` records that a month-end Treasury verdict was **voided** by using `backadj` and +fixed by `propadj`, so this consumer has already been bitten by a tier choice. + +## 4. The finding this handoff exists to record + +Stated nowhere before today, and it is the constraint that decides which machine can produce +a store the newest consumer can use. + +**`crowdmon` needs `unadj` AND `propadj`. `propadj` is derived on read from `unadj` + +`backadj` (`cotdata.prices._ratio_adjust`), so BOTH stored tiers are its precondition.** +`contract_master.coverage()` already encodes this: `joinable` means specs plus `unadj` plus +`backadj`, and its docstring says the back-adjusted requirement is right "not for the reason +an earlier version of this docstring gave" — it is the precondition for the derived tier, not +a claim that `backadj` returns are correct. They are not; `riskunits` refuses them. + +Now compose that with ADR-0007's scoping of databento: + +> databento exists for exactly one reason: the `cot-analyzer` dashboard runs on a Linux +> server that cannot run Norgate. It is **not** a second research vendor, and its coverage +> should not be broadened toward parity with Norgate. + +and its measured consequence: + +> **Every one of those reads is `adjustment="backadj"`.** So databento owes exactly one +> series per symbol, not the full tier set. + +**Therefore: a databento-backed futures store, as ADR-0007 scopes it, cannot feed `crowdmon` +at all.** One stored tier cannot produce `propadj`, `riskunits` refuses anything else, and the +refusal is a `raise` rather than a warning because the error is invisible to a spot check +(`backadj` percent vol is 201x too high for soybeans and **0.47x for gold**, which never goes +negative and passes every implausibility screen). + +The practical consequence, which is what the maintainer was actually asking about: + +**Only the Windows box can produce a store `crowdmon` can consume.** Not because of Python +(`cotdata` and `crowdmon` both declare `>=3.10`, and the Mac runs 3.11 under uv and builds the +full panel today), and not really because of Windows. Because it is the only machine with a +vendor that supplies **all the tiers**. `norgatedata` talks to a locally installed Norgate +Data Updater rather than an API (`providers/norgate.py:207,325`, both commented "Windows +producer only"), so the macOS box has no path to it at any Python version. + +| box | Python >=3.10 | Norgate | can produce a crowdmon-consumable store | +|---|---|---|---| +| Windows | yes | **yes** | **yes** | +| macOS | yes, runs 3.11 today | no (no NDU for macOS) | no | +| Linux server | **no, runs 3.9** | no | no | + +**What step 2 must not do:** narrow this further by accident. If the `marketdata` futures +producer ships databento-first, or ships Norgate but stores only `backadj` to match the +server's needs, `crowdmon` stops working and the failure is a `raise` inside `riskunits` on a +weekly scheduled job. Whoever writes the provider needs `unadj` and `backadj` **both stored** +as a hard requirement, not as a coverage nicety. + +## 5. Three questions to answer before writing code + +Two are ADR-0007's own open questions. The third is new and is the one this handoff was +written to surface. + +**5.1 Does `propadj` stay derived-on-read?** ADR-0007 leaves this open but records that +evidence arrived on 2026-07-27 and pointed one way: the month-end Treasury seasonal was voided +on `backadj` and fixed on `propadj`, and `marketdata` already derives every tier on read. +**Recommendation: yes, and treat it as settled by that evidence.** §4 above is a second, +independent reason. + +**5.2 Do the two store roots converge?** `COTDATA_STORE` and `MARKETDATA_STORE` are both set +from the shell profile and neither is set by any launcher, so ADR-0007 notes converging is +still cheap. **That is now less true than it was**: `crowdmon`'s `bin/publish_damage.sh` and +its launchd agent both default `COTDATA_STORE`, and `CROWDMON_STORE` joined the family on +2026-08-04 as a third root. Three roots is the moment to decide, not after. + +**5.3 NEW: does the Linux server ever run a futures producer, or is a synced store +permanent?** ADR-0007 asks this and leaves it open. **2026-08-04 answered it de facto for at +least one consumer**: the crowding page ships as a synced artifact because that box runs +Python 3.9 against `crowdmon`'s `>=3.10` floor and cannot run the producer at any price +(`crowdmon` ADR-0001). Combined with §4, the server cannot produce a crowdmon-consumable store +even if its interpreter were raised, because it has no Norgate either. + +**So "synced store" is not a temporary state pending step 2. It is the answer**, unless either +the server's interpreter is raised AND a full-tier non-Norgate vendor appears. Worth writing +into ADR-0007's open-questions section as resolved rather than leaving it reading as undecided. + +## 6. What this handoff deliberately does not decide + +- **Whether to split price production into its own repo** rather than into `marketdata`. + ADR-0007 says `marketdata`, and ADR-0007 is accepted. A session that wants a different + answer writes a superseding ADR; it does not quietly build something else. +- **Whether databento's coverage should broaden.** ADR-0007 says no, twice. §4 is a reason to + revisit *someday* and is not a licence to. +- **Anything about `livebook`.** Step 3, deferred on purpose, live book. + +## 7. Suggested order + +1. Write `marketdata/src/marketdata/providers/norgate.py` against the `futures` domain already + declared in `adjust.DOMAIN_TIERS`, storing `unadj` and `backadj` as a hard requirement (§4). +2. Port `contract_specs` and `--metadata`. +3. Repoint `crowdmon`'s ten call sites (§3). It is the smallest consumer and the one whose + tier requirements are strictest, so it fails loudest if the provider is wrong. **Do it + first, not last.** +4. Repoint `cotmetrics` / `cot-analyzer`, which are `backadj`-only and therefore easy. +5. Delete from `cotdata`, and update `crucible-stack` ADR-0007's "Status of the work". + +Step 3 stays out. `npf` and `livebook` are a separate pass with a live book behind them.