From 01ca8cba3f6c7f46089c003c78e08bd63ec551fd Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 6 Aug 2026 20:25:14 -0400 Subject: [PATCH] Inherit the durable data facts from crowdmon, which is being deprecated crowdmon's hypothesis is unproven and out of testable evidence, but several measurements made along the way are durable facts about data this package owns. They would be orphaned by that repo going quiet rather than by being wrong. Two new living documents, docs only, no code and no behaviour change. cross-report-comparability.md states the rule that has now been broken, measured and recorded three separate times in three repos against three different pairs of reports: two categories from two reports may only be compared when both partition open interest the same way AND count spreading the same way. The Legacy/TFF case is new here. Over 6,279 overlapping market-weeks the two reports agree on exactly two quantities, open_interest and nonreportable, both at 100.0000%. Above the reportable line the obvious mapping fails about 85% of the time, for two compounding reasons neither of which any correction recovers: spreading is counted differently (Legacy breaks it out for non-commercial only, TFF for every category) and the buckets hold different traders. The Canadian dollar worked example closes to the contract. The second half of that is a live trap in our own API rather than a crowdmon concern, and it is confirmed still live: canonicalize_legacy sets spread_contracts to NA on every row, so summing that column returns 0, which prints as a measurement of zero spreading and is not one. The identity long + spread == open_interest closes on 99.984% of TFF market-weeks and 19.857% of Legacy ones. The 19.857% is the spreading convention showing through a column we populate with nulls. reading-the-store.md carries four more: - 76% of the 279-market Disaggregated universe is ICE Energy Div and Nodal power and gas basis, so a cross-market statistic over the full universe is mostly about ERCOT and PJM. - A coverage ratio whose denominator nobody chose is not a measurement. Reported as a shortfall and withdrawn twice now, in two packages. Ported as the rule plus the warning that the count itself moves (25, then 45, then 47) rather than as any one number. - The fuller-sounding volume parameter is the narrower series. reconstructed is exactly two expiries; front is whole-market, established by open interest matching the CFTC exactly on 25 of 26 markets at a median ratio of 1.000. - A hole in a code's series is two different things and only one is a migration. RTY's CME code has an 8.9-year internal gap that the ICE code fills almost exactly; oats has a 294-day interval because a thin market fell out of the report and nothing is missing. The merge across sibling codes must precede any differencing, because the other order fails silently. Plus the composed tier fact: propadj is derived on read, Norgate is the only vendor supplying all tiers and is Windows-only by mechanism rather than licence, and databento owes only backadj per ADR-0007. So a databento-backed store cannot produce propadj at all, which is a live constraint on ADR-0007 step 2 and a tier fact rather than an OS fact. Restated, never moved: crowdmon's amendments are point-in-time records under that repo's doc lifecycle and are not edited by the harvest. Every section cites its source. Nothing here points at a crowdmon API, which is noted explicitly where the source argument leaned on ContractMaster.coverage(). 289 tests pass; this change is documentation only. Co-Authored-By: Claude Opus 5 --- docs/design/cross-report-comparability.md | 165 ++++++++++++++++++++++ docs/design/reading-the-store.md | 157 ++++++++++++++++++++ 2 files changed, 322 insertions(+) create mode 100644 docs/design/cross-report-comparability.md create mode 100644 docs/design/reading-the-store.md diff --git a/docs/design/cross-report-comparability.md b/docs/design/cross-report-comparability.md new file mode 100644 index 0000000..f49bfb9 --- /dev/null +++ b/docs/design/cross-report-comparability.md @@ -0,0 +1,165 @@ +# Cross-report comparability: what carries between the four reports, and what does not + +**Living document.** It exists because the same class of error has now been made, measured and +recorded three separate times, in three different repos, against three different pairs of +reports. The pattern is always the same: two reports describe the same market and the same +week, a category in one looks like a category in the other, and the difference or ratio +between them is computed. **The result is not interpretable, and nothing in the data says so.** + +`cotdata` serves all four reports, so this is its fact to hold. Consumers that computed one of +these quantities did not do anything careless; they read two columns with plausible names. + +--- + +## 1. The rule, before the cases + +> **Two categories from two different reports may only be compared when both reports partition +> open interest the same way AND count spreading the same way. In practice that is almost +> never, and the exceptions are enumerated below rather than inferred.** + +Three things can differ independently, and each alone is enough to break a comparison: + +1. **The basis.** Futures-only against futures-and-options-combined. +2. **The partition.** Which traders land in which bucket. +3. **The spreading convention.** Which categories have spreading broken out, and which have it + netted into long and short. + +--- + +## 2. Legacy against TFF: exactly two quantities carry + +Measured over **6,279 overlapping market-weeks**. + +| quantity | agreement | +|---|---| +| `open_interest` | **100.0000%**, maximum difference **0** | +| `nonreportable` long | **100.0000%** | +| `nonreportable` short | **100.0000%** | +| TFF category sum == Legacy category sum | 34.6% | +| `dealer` == `commercial` | 15.3% long, 14.5% short | +| `asset_manager + leveraged + other_reportable` == `noncommercial` | 15.6% long, 15.8% short | + +So the two reports describe the same pool of contracts and agree exactly on where the +reportable line falls. **Above that line the obvious mapping fails about 85% of the time**, for +two compounding reasons. + +**Spreading is counted differently.** Legacy breaks spreading out only for non-commercial +traders and nets commercial spreading into long and short; TFF breaks it out for every +category. Canadian dollar, 2026-07-28, open interest 372,447: + +``` +TFF long 348,849 + spread 23,598 = 372,447 residual 0 +Legacy long 362,728 + spread 0 = 362,728 residual 9,719 +gap between the two long totals = 13,879 = 23,598 - 9,719 +``` + +Exact to the contract. + +**The traders in each bucket are different people**, which is why no spreading correction +recovers the mapping. If Legacy's `noncommercial` held the same traders as TFF's buy side, +their spreading would match: + +``` +Legacy non-commercial spreading (derived) = 9,719 +TFF asset_manager + leveraged + other_reportable = 15,278 +``` + +### The consequence + +**Any quantity built by subtracting one report's category from another's mixes a +classification difference with a spreading-convention difference, and neither is recoverable +from published data. The only quantities that carry across are open interest and +`nonreportable`.** + +--- + +## 3. The `spread_contracts` trap, which is ours + +**`canonicalize_legacy` sets `spread_contracts` to `NA` on every row.** Summing an all-null +column returns **0**, which prints as a measurement of zero spreading and is not one. The +9,719 in the worked example above is **derived as the identity residual, not read**. + +Store-wide, the identity `long + spread == open_interest` closes on: + +| report | closes | +|---|---| +| TFF | **99.984%** of market-weeks | +| Legacy | **19.857%** of market-weeks, median residual 912 | + +That 19.857% is not a data quality problem. It is the spreading convention showing through a +column we populate with nulls. A caller that aggregates `spread_contracts` for Legacy and gets +0 has measured our null-handling, not the market. + +--- + +## 4. Supplemental against everything else: a different basis + +The Supplemental (Commodity Index Trader) report is **futures-and-options COMBINED**, where +Legacy, Disaggregated and TFF are all futures-only. Its `Open_Interest_All` is therefore a +different quantity for the same market and week: WHEAT-SRW on 2026-07-28 is **566,358 +combined against 463,502 futures-only**. + +**Nothing in the file says so.** It is the one report with no `FutOnly_or_Combined` column, so +`cotdata` asserts the flag rather than reading it, established by matching open interest +against both Legacy series 390/390 and 0/390. + +**Its Index Traders does NOT nest inside Disaggregated's Swap Dealer.** The taxonomy is +Legacy, not Disaggregated, so the two cannot be differenced to isolate levered swap flow. The +index book is carved out of commercial, non-commercial **and non-commercial spreading**, three +buckets where CFTC's own prose names two. + +**Coverage is 12 markets, then 13 from 2013**, when Soybean Meal entered. Both counts circulate +because both are right for part of the history, and six markets were renamed without changing +code. `cotdata-vintage coverage` derives the covered set from the data and prints every entry +and exit. + +Detail: [`../analysis/2026-08-03-cit-supplemental-measurements.md`](../analysis/2026-08-03-cit-supplemental-measurements.md). + +--- + +## 5. What this permits + +The enumerated safe comparisons, so that a consumer does not have to infer them from the +prohibitions: + +- **Open interest, Legacy against TFF**: identical, exactly, always. Safe. +- **`nonreportable`, Legacy against TFF**: identical, exactly, always. Safe. +- **Open interest, any futures-only report against the Supplemental**: **not** safe, different + basis. +- **Anything above the reportable line, across any two reports**: not safe. +- **Within one report, across weeks**: safe, subject to the gap rule (§6). + +--- + +## 6. A related trap that is not about reports but bites the same callers + +**Gaps in a market's series come from thin markets falling out of the report**, not from data +loss. Oats (`004603`) has a **294-day interval ending 2025-09-09** and five more over 50 days: +a thin market drops below the reporting threshold and reappears when it recovers. Any weekly +differencing needs an explicit gap rule, or one delta spans the whole absence and enters every +ranking as the largest flow in the sample. + +**The Oct-Nov 2025 shutdown is the opposite case and is often confused with it.** It left COT +**report** dates intact and broke only **release** dates, which are all `derived` in that +window. Anything indexed by report date is unaffected; anything indexed by release date in +that window is resting on a guess. + +--- + +## Provenance + +Sections 2, 3 and 6 were measured in `crowdmon` and are restated here because `cotdata` owns +the facts and crowdmon is deprecated. The original sections, with their reproducers and the +dates they were established, are: + +- §2, §3: `crowdmon/docs/design/amendments-2026-08-04.md` §D7, reproducer + `crowdmon/docs/analysis/reproduce_single_number.py::d7_legacy_and_tff_share_two_things` +- §6, gaps: `crowdmon/docs/design/amendments-2026-08-01.md` §A2 +- §6, shutdown: `crowdmon/docs/design/amendments-2026-08-01.md` §A1 + +They are **restated rather than moved**: those files are point-in-time records under that +repo's doc lifecycle and are not edited by this harvest. See +`crowdmon/docs/HARVEST.md` for the full map of what was ported, what was already resolved, and +what was parked with the hypothesis. + +Section 4 was authored here from the start and is summarised rather than moved. diff --git a/docs/design/reading-the-store.md b/docs/design/reading-the-store.md new file mode 100644 index 0000000..eb5ba7c --- /dev/null +++ b/docs/design/reading-the-store.md @@ -0,0 +1,157 @@ +# Reading the store: universe, coverage, continuity and price tiers + +**Living document.** Four things a consumer of this store needs before it computes anything, +each of which has already cost a session that reasoned from a plausible-looking column or a +plausible-looking count. None of them is about report comparability, which has its own +document: [`cross-report-comparability.md`](cross-report-comparability.md). + +--- + +## 1. The Disaggregated universe is mostly power and gas basis + +Of the 279 markets in the Disaggregated report, **213 (76%) are ICE Futures Energy Division or +Nodal Exchange**: power hubs, gas basis, carbon allowances, renewable energy certificates. Not +classic outrights. + +The consequence is specific rather than decorative. **Any cross-market statistic computed over +"all Disaggregated markets" is mostly a statement about ERCOT and PJM.** A principal component +over positioning changes across the full universe produces a PC1 describing power basis, not +the macro book, and anything downstream of it inherits that population. + +State which universe a result is over, before computing it. + +--- + +## 2. A coverage ratio whose denominator nobody chose is not a measurement + +This has now been reported as a shortfall and then withdrawn **twice**, in two different +packages, for the same reason: + +- "42 of 95 joinable, 44%", withdrawn once it turned out all 42 deployed markets joined. +- "25 of 279 markets", framed as a 9% sample and a limitation to be noted beside every result. + It is not a sample. The other 254 are markets **nobody trades**: no liquidity, no access, or + they are the power and gas basis contracts of §1. A join reaching every market in the + tradeable universe is complete, not 9% complete. + +The rule that generalises: **the denominator has to be a set someone chose.** "Every market +the CFTC publishes" is not a target any consumer has, so a ratio against it measures nothing. +Where a coverage figure is genuinely useful, derive it from the data at read time rather than +quoting a number from prose. Counts here move: covered markets have been 25, then 45, then 47 +as specs were added, and every one of those figures was correct on the day it was written and +stale within a fortnight. + +--- + +## 3. A market code is not an instrument, and a hole is two different things + +A gap in a code's weekly series has two unrelated causes, and only one of them is a migration. +Telling them apart requires joining a code's **internal continuity** to its **siblings**, +which nothing does automatically. + +Over 51 codes in the two current-state panels, **46 have a longest inter-week gap of 8 days**, +which is a holiday shift. The five that do not split into two causes. + +**Cause one: a migration, where a sibling code fills the hole.** Russell 2000: + +| code | venue | weeks | span | longest internal gap | +|---|---|---|---|---| +| `239742` | CME | 587 | 2006-06-13 to 2026-07-28 | **3,255 days, 8.9 years**, ending 2017-08-15 | +| `23977A` | ICE | 516 | 2008-07-22 to 2018-06-05 | none | + +The two are **complementary, not redundant**: the ICE code covers the CME code's hole almost +exactly, and together they are a continuous twenty-year market. Read apart, the CME code looks +like it began in 2017. A downstream percentile with a three-year warm-up then does not reach +it until 2023, and anything scoring an event in that window scores it on the **retiring** +venue against a much shorter reference series than its peers. + +Lumber is the clean contrast: `058643` hands off to `058644` in 2023 with a two-month overlap +and no holes on either side. + +**Cause two: a thin market falling out of the report.** Oats (`004603`) has a **294-day +interval ending 2025-09-09** and five more over 50 days. A market below the reporting +threshold disappears and returns when it recovers. No sibling fills these, because nothing is +missing: the market genuinely was not reported. + +**Consequences.** Any weekly differencing needs an explicit gap rule, or one delta spans the +whole absence and enters every ranking as the largest flow in the sample. Any per-code history +length is a statement about the code, not the market. And a merge across sibling codes must +precede any differencing, because the other order fails silently. + +--- + +## 4. Volume: the fuller-sounding parameter is the narrower series + +`get_prices` takes `volume="front"` or `volume="reconstructed"`, documented as "continuous +front-month volume" and "true market volume (first + second expiring contract)". + +**The second reads like the fuller series and is the narrower one.** +`Volume_Reconstructed = FirstVolume + SecondVolume`, exactly two expiries, while the plain +`Volume` field behind `front` spans the whole curve. + +Two independent measurements establish that `front` is whole-market: + +1. **Open interest matches the CFTC to the contract.** The price files carry an + `Open Interest` column Norgate collects from the exchange; the CFTC collects its own from + clearing members. Against COT total-market open interest for the same Tuesday: **exact + agreement on 25 of 26 markets**, palladium at 0.998, median ratio **1.000**. Two vendors + and two collection paths cannot agree that precisely unless both measure the whole market. + Front-month data would be a fraction. +2. **Curve concentration orders exactly as contract structure predicts.** + +This is recorded because the naming cost real time: several places across two repos stated +that whole-market volume was absent from the store, and all of them were describing a naming +problem as a data gap. + +--- + +## 5. Price tiers, and who can produce a store that carries them + +Three facts that were each written down separately and never composed: + +1. **`propadj` is derived on read** from `unadj` + `backadj`. It is not a stored tier. +2. **Norgate is the only vendor supplying all tiers, and it is Windows-only by mechanism + rather than by licence**: `norgatedata` talks to a locally installed Norgate Data Updater + application rather than to an API + ([`../../src/cotdata/providers/norgate.py`](../../src/cotdata/providers/norgate.py)). +3. **databento owes exactly one series per symbol**, `backadj`, because crucible-stack + ADR-0007 scopes it to the Linux dashboard's needs and says its coverage "should not be + broadened toward parity with Norgate". + +**Composed: a databento-backed futures store cannot produce `propadj` at all**, so any +consumer needing correct percentage returns cannot be served by one. This is a live constraint +on ADR-0007 step 2 rather than a historical note, and it is a **tier fact, not an operating +system fact**, a distinction that has been got wrong more than once: + +| box | Python >= 3.10 | Norgate | can produce a store carrying all tiers | +|---|---|---|---| +| Windows | yes | **yes** | **yes** | +| macOS | yes | no | no | +| Linux server | no, runs 3.9 | no | no | + +The reason a wrong tier is a `raise` and not a warning is that the error survives a spot check: +`backadj` percentage volatility is 201x too high for soybeans and **0.47x for gold**, which +never goes negative and passes every implausibility screen. + +--- + +## Provenance + +Every section here was measured in `crowdmon` and is restated because `cotdata` owns the facts +and `crowdmon` is deprecated. Restated, never moved: those files are point-in-time records +under that repo's doc lifecycle and are not edited by the harvest. + +| section | source | +|---|---| +| §1 | `crowdmon/docs/design/amendments-2026-08-01.md` §A5 | +| §2 | `crowdmon/docs/design/amendments-2026-08-01.md` §A14, and §C12 of the 08-03 file for the moving count | +| §3 | `crowdmon/docs/design/amendments-2026-08-02.md` §B26, §B27, §B30; oats from §A2 | +| §4 | `crowdmon/docs/design/amendments-2026-08-01.md` §A13 | +| §5 | `crowdmon/docs/design/amendments-2026-08-04.md` §D14 | + +`crowdmon/docs/HARVEST.md` is the full map of what was ported, what was already resolved +upstream, and what was parked with the hypothesis. + +**One deliberate omission.** §5's source composes its argument partly through +`crowdmon.futures.contract_master.ContractMaster.coverage()`, and that API goes with the +package. Nothing here points at it; the three facts above stand on `cotdata`'s own providers +and on ADR-0007.