Skip to content

Inherit the durable data facts from crowdmon #102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mspinola merged 1 commit into from
Aug 7, 2026
Merged

Inherit the durable data facts from crowdmon #102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 165 additions & 0 deletions docs/design/cross-report-comparability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Cross-report comparability: what carries between the four reports, and what does not

**Living document.** It exists because the same class of error has now been made, measured and
recorded three separate times, in three different repos, against three different pairs of
reports. The pattern is always the same: two reports describe the same market and the same
week, a category in one looks like a category in the other, and the difference or ratio
between them is computed. **The result is not interpretable, and nothing in the data says so.**

`cotdata` serves all four reports, so this is its fact to hold. Consumers that computed one of
these quantities did not do anything careless; they read two columns with plausible names.

---

## 1. The rule, before the cases

> **Two categories from two different reports may only be compared when both reports partition
> open interest the same way AND count spreading the same way. In practice that is almost
> never, and the exceptions are enumerated below rather than inferred.**

Three things can differ independently, and each alone is enough to break a comparison:

1. **The basis.** Futures-only against futures-and-options-combined.
2. **The partition.** Which traders land in which bucket.
3. **The spreading convention.** Which categories have spreading broken out, and which have it
netted into long and short.

---

## 2. Legacy against TFF: exactly two quantities carry

Measured over **6,279 overlapping market-weeks**.

| quantity | agreement |
|---|---|
| `open_interest` | **100.0000%**, maximum difference **0** |
| `nonreportable` long | **100.0000%** |
| `nonreportable` short | **100.0000%** |
| TFF category sum == Legacy category sum | 34.6% |
| `dealer` == `commercial` | 15.3% long, 14.5% short |
| `asset_manager + leveraged + other_reportable` == `noncommercial` | 15.6% long, 15.8% short |

So the two reports describe the same pool of contracts and agree exactly on where the
reportable line falls. **Above that line the obvious mapping fails about 85% of the time**, for
two compounding reasons.

**Spreading is counted differently.** Legacy breaks spreading out only for non-commercial
traders and nets commercial spreading into long and short; TFF breaks it out for every
category. Canadian dollar, 2026-07-28, open interest 372,447:

```
TFF long 348,849 + spread 23,598 = 372,447 residual 0
Legacy long 362,728 + spread 0 = 362,728 residual 9,719
gap between the two long totals = 13,879 = 23,598 - 9,719
```

Exact to the contract.

**The traders in each bucket are different people**, which is why no spreading correction
recovers the mapping. If Legacy's `noncommercial` held the same traders as TFF's buy side,
their spreading would match:

```
Legacy non-commercial spreading (derived) = 9,719
TFF asset_manager + leveraged + other_reportable = 15,278
```

### The consequence

**Any quantity built by subtracting one report's category from another's mixes a
classification difference with a spreading-convention difference, and neither is recoverable
from published data. The only quantities that carry across are open interest and
`nonreportable`.**

---

## 3. The `spread_contracts` trap, which is ours

**`canonicalize_legacy` sets `spread_contracts` to `NA` on every row.** Summing an all-null
column returns **0**, which prints as a measurement of zero spreading and is not one. The
9,719 in the worked example above is **derived as the identity residual, not read**.

Store-wide, the identity `long + spread == open_interest` closes on:

| report | closes |
|---|---|
| TFF | **99.984%** of market-weeks |
| Legacy | **19.857%** of market-weeks, median residual 912 |

That 19.857% is not a data quality problem. It is the spreading convention showing through a
column we populate with nulls. A caller that aggregates `spread_contracts` for Legacy and gets
0 has measured our null-handling, not the market.

---

## 4. Supplemental against everything else: a different basis

The Supplemental (Commodity Index Trader) report is **futures-and-options COMBINED**, where
Legacy, Disaggregated and TFF are all futures-only. Its `Open_Interest_All` is therefore a
different quantity for the same market and week: WHEAT-SRW on 2026-07-28 is **566,358
combined against 463,502 futures-only**.

**Nothing in the file says so.** It is the one report with no `FutOnly_or_Combined` column, so
`cotdata` asserts the flag rather than reading it, established by matching open interest
against both Legacy series 390/390 and 0/390.

**Its Index Traders does NOT nest inside Disaggregated's Swap Dealer.** The taxonomy is
Legacy, not Disaggregated, so the two cannot be differenced to isolate levered swap flow. The
index book is carved out of commercial, non-commercial **and non-commercial spreading**, three
buckets where CFTC's own prose names two.

**Coverage is 12 markets, then 13 from 2013**, when Soybean Meal entered. Both counts circulate
because both are right for part of the history, and six markets were renamed without changing
code. `cotdata-vintage coverage` derives the covered set from the data and prints every entry
and exit.

Detail: [`../analysis/2026-08-03-cit-supplemental-measurements.md`](../analysis/2026-08-03-cit-supplemental-measurements.md).

---

## 5. What this permits

The enumerated safe comparisons, so that a consumer does not have to infer them from the
prohibitions:

- **Open interest, Legacy against TFF**: identical, exactly, always. Safe.
- **`nonreportable`, Legacy against TFF**: identical, exactly, always. Safe.
- **Open interest, any futures-only report against the Supplemental**: **not** safe, different
basis.
- **Anything above the reportable line, across any two reports**: not safe.
- **Within one report, across weeks**: safe, subject to the gap rule (§6).

---

## 6. A related trap that is not about reports but bites the same callers

**Gaps in a market's series come from thin markets falling out of the report**, not from data
loss. Oats (`004603`) has a **294-day interval ending 2025-09-09** and five more over 50 days:
a thin market drops below the reporting threshold and reappears when it recovers. Any weekly
differencing needs an explicit gap rule, or one delta spans the whole absence and enters every
ranking as the largest flow in the sample.

**The Oct-Nov 2025 shutdown is the opposite case and is often confused with it.** It left COT
**report** dates intact and broke only **release** dates, which are all `derived` in that
window. Anything indexed by report date is unaffected; anything indexed by release date in
that window is resting on a guess.

---

## Provenance

Sections 2, 3 and 6 were measured in `crowdmon` and are restated here because `cotdata` owns
the facts and crowdmon is deprecated. The original sections, with their reproducers and the
dates they were established, are:

- §2, §3: `crowdmon/docs/design/amendments-2026-08-04.md` §D7, reproducer
`crowdmon/docs/analysis/reproduce_single_number.py::d7_legacy_and_tff_share_two_things`
- §6, gaps: `crowdmon/docs/design/amendments-2026-08-01.md` §A2
- §6, shutdown: `crowdmon/docs/design/amendments-2026-08-01.md` §A1

They are **restated rather than moved**: those files are point-in-time records under that
repo's doc lifecycle and are not edited by this harvest. See
`crowdmon/docs/HARVEST.md` for the full map of what was ported, what was already resolved, and
what was parked with the hypothesis.

Section 4 was authored here from the start and is summarised rather than moved.
157 changes: 157 additions & 0 deletions docs/design/reading-the-store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Reading the store: universe, coverage, continuity and price tiers

**Living document.** Four things a consumer of this store needs before it computes anything,
each of which has already cost a session that reasoned from a plausible-looking column or a
plausible-looking count. None of them is about report comparability, which has its own
document: [`cross-report-comparability.md`](cross-report-comparability.md).

---

## 1. The Disaggregated universe is mostly power and gas basis

Of the 279 markets in the Disaggregated report, **213 (76%) are ICE Futures Energy Division or
Nodal Exchange**: power hubs, gas basis, carbon allowances, renewable energy certificates. Not
classic outrights.

The consequence is specific rather than decorative. **Any cross-market statistic computed over
"all Disaggregated markets" is mostly a statement about ERCOT and PJM.** A principal component
over positioning changes across the full universe produces a PC1 describing power basis, not
the macro book, and anything downstream of it inherits that population.

State which universe a result is over, before computing it.

---

## 2. A coverage ratio whose denominator nobody chose is not a measurement

This has now been reported as a shortfall and then withdrawn **twice**, in two different
packages, for the same reason:

- "42 of 95 joinable, 44%", withdrawn once it turned out all 42 deployed markets joined.
- "25 of 279 markets", framed as a 9% sample and a limitation to be noted beside every result.
It is not a sample. The other 254 are markets **nobody trades**: no liquidity, no access, or
they are the power and gas basis contracts of §1. A join reaching every market in the
tradeable universe is complete, not 9% complete.

The rule that generalises: **the denominator has to be a set someone chose.** "Every market
the CFTC publishes" is not a target any consumer has, so a ratio against it measures nothing.
Where a coverage figure is genuinely useful, derive it from the data at read time rather than
quoting a number from prose. Counts here move: covered markets have been 25, then 45, then 47
as specs were added, and every one of those figures was correct on the day it was written and
stale within a fortnight.

---

## 3. A market code is not an instrument, and a hole is two different things

A gap in a code's weekly series has two unrelated causes, and only one of them is a migration.
Telling them apart requires joining a code's **internal continuity** to its **siblings**,
which nothing does automatically.

Over 51 codes in the two current-state panels, **46 have a longest inter-week gap of 8 days**,
which is a holiday shift. The five that do not split into two causes.

**Cause one: a migration, where a sibling code fills the hole.** Russell 2000:

| code | venue | weeks | span | longest internal gap |
|---|---|---|---|---|
| `239742` | CME | 587 | 2006-06-13 to 2026-07-28 | **3,255 days, 8.9 years**, ending 2017-08-15 |
| `23977A` | ICE | 516 | 2008-07-22 to 2018-06-05 | none |

The two are **complementary, not redundant**: the ICE code covers the CME code's hole almost
exactly, and together they are a continuous twenty-year market. Read apart, the CME code looks
like it began in 2017. A downstream percentile with a three-year warm-up then does not reach
it until 2023, and anything scoring an event in that window scores it on the **retiring**
venue against a much shorter reference series than its peers.

Lumber is the clean contrast: `058643` hands off to `058644` in 2023 with a two-month overlap
and no holes on either side.

**Cause two: a thin market falling out of the report.** Oats (`004603`) has a **294-day
interval ending 2025-09-09** and five more over 50 days. A market below the reporting
threshold disappears and returns when it recovers. No sibling fills these, because nothing is
missing: the market genuinely was not reported.

**Consequences.** Any weekly differencing needs an explicit gap rule, or one delta spans the
whole absence and enters every ranking as the largest flow in the sample. Any per-code history
length is a statement about the code, not the market. And a merge across sibling codes must
precede any differencing, because the other order fails silently.

---

## 4. Volume: the fuller-sounding parameter is the narrower series

`get_prices` takes `volume="front"` or `volume="reconstructed"`, documented as "continuous
front-month volume" and "true market volume (first + second expiring contract)".

**The second reads like the fuller series and is the narrower one.**
`Volume_Reconstructed = FirstVolume + SecondVolume`, exactly two expiries, while the plain
`Volume` field behind `front` spans the whole curve.

Two independent measurements establish that `front` is whole-market:

1. **Open interest matches the CFTC to the contract.** The price files carry an
`Open Interest` column Norgate collects from the exchange; the CFTC collects its own from
clearing members. Against COT total-market open interest for the same Tuesday: **exact
agreement on 25 of 26 markets**, palladium at 0.998, median ratio **1.000**. Two vendors
and two collection paths cannot agree that precisely unless both measure the whole market.
Front-month data would be a fraction.
2. **Curve concentration orders exactly as contract structure predicts.**

This is recorded because the naming cost real time: several places across two repos stated
that whole-market volume was absent from the store, and all of them were describing a naming
problem as a data gap.

---

## 5. Price tiers, and who can produce a store that carries them

Three facts that were each written down separately and never composed:

1. **`propadj` is derived on read** from `unadj` + `backadj`. It is not a stored tier.
2. **Norgate is the only vendor supplying all tiers, and it is Windows-only by mechanism
rather than by licence**: `norgatedata` talks to a locally installed Norgate Data Updater
application rather than to an API
([`../../src/cotdata/providers/norgate.py`](../../src/cotdata/providers/norgate.py)).
3. **databento owes exactly one series per symbol**, `backadj`, because crucible-stack
ADR-0007 scopes it to the Linux dashboard's needs and says its coverage "should not be
broadened toward parity with Norgate".

**Composed: a databento-backed futures store cannot produce `propadj` at all**, so any
consumer needing correct percentage returns cannot be served by one. This is a live constraint
on ADR-0007 step 2 rather than a historical note, and it is a **tier fact, not an operating
system fact**, a distinction that has been got wrong more than once:

| box | Python >= 3.10 | Norgate | can produce a store carrying all tiers |
|---|---|---|---|
| Windows | yes | **yes** | **yes** |
| macOS | yes | no | no |
| Linux server | no, runs 3.9 | no | no |

The reason a wrong tier is a `raise` and not a warning is that the error survives a spot check:
`backadj` percentage volatility is 201x too high for soybeans and **0.47x for gold**, which
never goes negative and passes every implausibility screen.

---

## Provenance

Every section here was measured in `crowdmon` and is restated because `cotdata` owns the facts
and `crowdmon` is deprecated. Restated, never moved: those files are point-in-time records
under that repo's doc lifecycle and are not edited by the harvest.

| section | source |
|---|---|
| §1 | `crowdmon/docs/design/amendments-2026-08-01.md` §A5 |
| §2 | `crowdmon/docs/design/amendments-2026-08-01.md` §A14, and §C12 of the 08-03 file for the moving count |
| §3 | `crowdmon/docs/design/amendments-2026-08-02.md` §B26, §B27, §B30; oats from §A2 |
| §4 | `crowdmon/docs/design/amendments-2026-08-01.md` §A13 |
| §5 | `crowdmon/docs/design/amendments-2026-08-04.md` §D14 |

`crowdmon/docs/HARVEST.md` is the full map of what was ported, what was already resolved
upstream, and what was parked with the hypothesis.

**One deliberate omission.** §5's source composes its argument partly through
`crowdmon.futures.contract_master.ContractMaster.coverage()`, and that API goes with the
package. Nothing here points at it; the three facts above stand on `cotdata`'s own providers
and on ADR-0007.
Loading