Skip to content

Handoff: claim ADR-0007 step 2, with a re-measurement that moved the wrong way #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 1 commit into from
Aug 5, 2026
Merged

Handoff: claim ADR-0007 step 2, with a re-measurement that moved the wrong way #101

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
194 changes: 194 additions & 0 deletions docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Handoff: ADR-0007 step 2, move price production out of `cotdata`

**Status:** **CLAIMED, NOT STARTED.** This is a work order and a re-measurement, no code
**Date:** 2026-08-04
**Lives at:** `cotdata/docs/handoffs/2026-08-04-adr7-step2-price-producer-split.md`
**Target:** a Claude Code session in a `cotdata` worktree, with `marketdata` beside it
**Decision it executes:** [crucible-stack ADR-0007](https://github.com/mspinola/crucible-stack/blob/main/docs/adr/ADR-0007-cotdata-is-cot-only-bars-live-in-marketdata.md),
**accepted 2026-07-27**. This proposes no new decision and asks for none
**Blocked on:** nothing technical. Three questions in §5 want an answer before code, and two
of them are ADR-0007's own open questions rather than new ones
**Companion:** `crowdmon/docs/design/amendments-2026-08-04.md` §D14 — the consumer-side
record of §4, filed there so the constraint is discoverable from the repo it binds
**Not this handoff:** step 3 (`livebook`'s six `contract_specs()` calls at `required=False`),
which ADR-0007 defers deliberately because it is a live book

> Announced before the first line of code, per the neighbouring repo's convention. If you
> were about to start it, say so and I will drop it.

---

## 0. Why now, and why this is a re-measurement rather than a proposal

ADR-0007 settled the direction on 2026-07-27: `cotdata` becomes CFTC positioning only, and
all bars, futures included, move to `marketdata`. Step 1 shipped. Step 3's prerequisite is
half built. **Step 2 has not started**, and `crowdmon`'s `CLAUDE.md` records it as "on ice…
nobody owns it. Do not start it without confirming that has changed."

That has now changed: the maintainer asked for it directly on 2026-08-04, in the course of
deciding which machine publishes the crowdmon damage panel.

The one thing this handoff adds to the ADR is a **re-measurement, and the number moved the
wrong way.**

## 1. Step 2 has grown ~11% while sitting on ice

Measured 2026-08-04 against `wc -l`, on `main` at `49ebc45`:

| file | ADR-0007 recorded (2026-07-27) | today | drift |
|---|---:|---:|---:|
| `src/cotdata/providers/databento.py` | 1,015 | **1,099** | +84 |
| `src/cotdata/providers/norgate.py` | 469 | **546** | +77 |
| `src/cotdata/prices.py` | 167 | **186** | +19 |
| `src/cotdata/providers/yfinance.py` | 70 | 70 | — |
| **total** | **1,721** ("roughly 1,700") | **1,901** | **+180** |

Plus `metadata/contract_specs.parquet` and the `--metadata` action, unchanged in scope.

ADR-0007 predicted this in as many words: *"`databento.py` arriving after this ADR was
written makes `cotdata` more price-heavy than the measurements above, which strengthens the
case and enlarges step 2 at the same time."* It was right, and the drift is now measured
rather than anticipated: **+180 lines in eight days, both price providers growing, `prices.py`
growing, and neither CFTC provider moving.**

That is the argument for doing it sooner rather than the argument for doing it at all. The
ADR already made the second one.

## 2. It is not a file move: `marketdata` has no futures provider

`marketdata/src/marketdata/providers/` contains `__init__.py` and `yfinance.py`. That is all.

The `futures` domain **is** declared in `adjust.DOMAIN_TIERS`, which ADR-0007 notes was done
so error messages are correct from the first day, but nothing implements it. So step 2 is
*write the Norgate futures provider into `marketdata`*, using `cotdata`'s as the reference,
and not *move a file across a seam*.

Whoever takes this should price that difference in before estimating. 1,901 lines of source
to read; the delivered artifact is a provider that did not exist.

## 3. The consumers, enumerated

Measured 2026-08-04. Every one of these is a read that step 2 repoints.

**`crowdmon`** — 6 modules, 10 call sites:

| module | call |
|---|---|
| `futures/riskunits.py:138` | `cotdata.get_prices(symbol, adjustment="propadj")` |
| `futures/notional.py:88` | `cotdata.get_prices(symbol, adjustment="unadj")` |
| `futures/volume.py:106,117` | `cotdata.get_prices(..., volume="front")` |
| `futures/alignment.py:106` | `cotdata.get_prices(...)["Close"]` |
| `futures/clustering.py:116` | `cotdata.get_prices(...)["Close"]` |
| `futures/contract_master.py:97,100,117,122` | `all_symbols`, `load_manifest`, `store.read_metadata` |

`contract_master.py:34` already documents itself as the call site ADR-0007 will repoint, and
`store.read_metadata()` is flagged there as the one non-`__all__` cotdata symbol the package
reaches for. That comment is the pre-existing marker for this work.

**`cotmetrics` / `cot-analyzer`** — ADR-0007 §"measured facts" records that nothing in
`cot-analyzer` calls a price vendor, every live read is `cotdata.get_prices`, and every one of
those is `adjustment="backadj"`.

**`npf`** — reads futures through `cotdata.get_prices`; ADR-0007's open question about
`propadj` records that a month-end Treasury verdict was **voided** by using `backadj` and
fixed by `propadj`, so this consumer has already been bitten by a tier choice.

## 4. The finding this handoff exists to record

Stated nowhere before today, and it is the constraint that decides which machine can produce
a store the newest consumer can use.

**`crowdmon` needs `unadj` AND `propadj`. `propadj` is derived on read from `unadj` +
`backadj` (`cotdata.prices._ratio_adjust`), so BOTH stored tiers are its precondition.**
`contract_master.coverage()` already encodes this: `joinable` means specs plus `unadj` plus
`backadj`, and its docstring says the back-adjusted requirement is right "not for the reason
an earlier version of this docstring gave" — it is the precondition for the derived tier, not
a claim that `backadj` returns are correct. They are not; `riskunits` refuses them.

Now compose that with ADR-0007's scoping of databento:

> databento exists for exactly one reason: the `cot-analyzer` dashboard runs on a Linux
> server that cannot run Norgate. It is **not** a second research vendor, and its coverage
> should not be broadened toward parity with Norgate.

and its measured consequence:

> **Every one of those reads is `adjustment="backadj"`.** So databento owes exactly one
> series per symbol, not the full tier set.

**Therefore: a databento-backed futures store, as ADR-0007 scopes it, cannot feed `crowdmon`
at all.** One stored tier cannot produce `propadj`, `riskunits` refuses anything else, and the
refusal is a `raise` rather than a warning because the error is invisible to a spot check
(`backadj` percent vol is 201x too high for soybeans and **0.47x for gold**, which never goes
negative and passes every implausibility screen).

The practical consequence, which is what the maintainer was actually asking about:

**Only the Windows box can produce a store `crowdmon` can consume.** Not because of Python
(`cotdata` and `crowdmon` both declare `>=3.10`, and the Mac runs 3.11 under uv and builds the
full panel today), and not really because of Windows. Because it is the only machine with a
vendor that supplies **all the tiers**. `norgatedata` talks to a locally installed Norgate
Data Updater rather than an API (`providers/norgate.py:207,325`, both commented "Windows
producer only"), so the macOS box has no path to it at any Python version.

| box | Python >=3.10 | Norgate | can produce a crowdmon-consumable store |
|---|---|---|---|
| Windows | yes | **yes** | **yes** |
| macOS | yes, runs 3.11 today | no (no NDU for macOS) | no |
| Linux server | **no, runs 3.9** | no | no |

**What step 2 must not do:** narrow this further by accident. If the `marketdata` futures
producer ships databento-first, or ships Norgate but stores only `backadj` to match the
server's needs, `crowdmon` stops working and the failure is a `raise` inside `riskunits` on a
weekly scheduled job. Whoever writes the provider needs `unadj` and `backadj` **both stored**
as a hard requirement, not as a coverage nicety.

## 5. Three questions to answer before writing code

Two are ADR-0007's own open questions. The third is new and is the one this handoff was
written to surface.

**5.1 Does `propadj` stay derived-on-read?** ADR-0007 leaves this open but records that
evidence arrived on 2026-07-27 and pointed one way: the month-end Treasury seasonal was voided
on `backadj` and fixed on `propadj`, and `marketdata` already derives every tier on read.
**Recommendation: yes, and treat it as settled by that evidence.** §4 above is a second,
independent reason.

**5.2 Do the two store roots converge?** `COTDATA_STORE` and `MARKETDATA_STORE` are both set
from the shell profile and neither is set by any launcher, so ADR-0007 notes converging is
still cheap. **That is now less true than it was**: `crowdmon`'s `bin/publish_damage.sh` and
its launchd agent both default `COTDATA_STORE`, and `CROWDMON_STORE` joined the family on
2026-08-04 as a third root. Three roots is the moment to decide, not after.

**5.3 NEW: does the Linux server ever run a futures producer, or is a synced store
permanent?** ADR-0007 asks this and leaves it open. **2026-08-04 answered it de facto for at
least one consumer**: the crowding page ships as a synced artifact because that box runs
Python 3.9 against `crowdmon`'s `>=3.10` floor and cannot run the producer at any price
(`crowdmon` ADR-0001). Combined with §4, the server cannot produce a crowdmon-consumable store
even if its interpreter were raised, because it has no Norgate either.

**So "synced store" is not a temporary state pending step 2. It is the answer**, unless either
the server's interpreter is raised AND a full-tier non-Norgate vendor appears. Worth writing
into ADR-0007's open-questions section as resolved rather than leaving it reading as undecided.

## 6. What this handoff deliberately does not decide

- **Whether to split price production into its own repo** rather than into `marketdata`.
ADR-0007 says `marketdata`, and ADR-0007 is accepted. A session that wants a different
answer writes a superseding ADR; it does not quietly build something else.
- **Whether databento's coverage should broaden.** ADR-0007 says no, twice. §4 is a reason to
revisit *someday* and is not a licence to.
- **Anything about `livebook`.** Step 3, deferred on purpose, live book.

## 7. Suggested order

1. Write `marketdata/src/marketdata/providers/norgate.py` against the `futures` domain already
declared in `adjust.DOMAIN_TIERS`, storing `unadj` and `backadj` as a hard requirement (§4).
2. Port `contract_specs` and `--metadata`.
3. Repoint `crowdmon`'s ten call sites (§3). It is the smallest consumer and the one whose
tier requirements are strictest, so it fails loudest if the provider is wrong. **Do it
first, not last.**
4. Repoint `cotmetrics` / `cot-analyzer`, which are `backadj`-only and therefore easy.
5. Delete from `cotdata`, and update `crucible-stack` ADR-0007's "Status of the work".

Step 3 stays out. `npf` and `livebook` are a separate pass with a live book behind them.
Loading