diff --git a/docs/design/cross-report-comparability.md b/docs/design/cross-report-comparability.md new file mode 100644 index 0000000..f49bfb9 --- /dev/null +++ b/docs/design/cross-report-comparability.md @@ -0,0 +1,165 @@ +# Cross-report comparability: what carries between the four reports, and what does not + +**Living document.** It exists because the same class of error has now been made, measured and +recorded three separate times, in three different repos, against three different pairs of +reports. The pattern is always the same: two reports describe the same market and the same +week, a category in one looks like a category in the other, and the difference or ratio +between them is computed. **The result is not interpretable, and nothing in the data says so.** + +`cotdata` serves all four reports, so this is its fact to hold. Consumers that computed one of +these quantities did not do anything careless; they read two columns with plausible names. + +--- + +## 1. The rule, before the cases + +> **Two categories from two different reports may only be compared when both reports partition +> open interest the same way AND count spreading the same way. In practice that is almost +> never, and the exceptions are enumerated below rather than inferred.** + +Three things can differ independently, and each alone is enough to break a comparison: + +1. **The basis.** Futures-only against futures-and-options-combined. +2. **The partition.** Which traders land in which bucket. +3. **The spreading convention.** Which categories have spreading broken out, and which have it + netted into long and short. + +--- + +## 2. Legacy against TFF: exactly two quantities carry + +Measured over **6,279 overlapping market-weeks**. + +| quantity | agreement | +|---|---| +| `open_interest` | **100.0000%**, maximum difference **0** | +| `nonreportable` long | **100.0000%** | +| `nonreportable` short | **100.0000%** | +| TFF category sum == Legacy category sum | 34.6% | +| `dealer` == `commercial` | 15.3% long, 14.5% short | +| `asset_manager + leveraged + other_reportable` == `noncommercial` | 15.6% long, 15.8% short | + +So the two reports describe the same pool of contracts and agree exactly on where the +reportable line falls. **Above that line the obvious mapping fails about 85% of the time**, for +two compounding reasons. + +**Spreading is counted differently.** Legacy breaks spreading out only for non-commercial +traders and nets commercial spreading into long and short; TFF breaks it out for every +category. Canadian dollar, 2026-07-28, open interest 372,447: + +``` +TFF long 348,849 + spread 23,598 = 372,447 residual 0 +Legacy long 362,728 + spread 0 = 362,728 residual 9,719 +gap between the two long totals = 13,879 = 23,598 - 9,719 +``` + +Exact to the contract. + +**The traders in each bucket are different people**, which is why no spreading correction +recovers the mapping. If Legacy's `noncommercial` held the same traders as TFF's buy side, +their spreading would match: + +``` +Legacy non-commercial spreading (derived) = 9,719 +TFF asset_manager + leveraged + other_reportable = 15,278 +``` + +### The consequence + +**Any quantity built by subtracting one report's category from another's mixes a +classification difference with a spreading-convention difference, and neither is recoverable +from published data. The only quantities that carry across are open interest and +`nonreportable`.** + +--- + +## 3. The `spread_contracts` trap, which is ours + +**`canonicalize_legacy` sets `spread_contracts` to `NA` on every row.** Summing an all-null +column returns **0**, which prints as a measurement of zero spreading and is not one. The +9,719 in the worked example above is **derived as the identity residual, not read**. + +Store-wide, the identity `long + spread == open_interest` closes on: + +| report | closes | +|---|---| +| TFF | **99.984%** of market-weeks | +| Legacy | **19.857%** of market-weeks, median residual 912 | + +That 19.857% is not a data quality problem. It is the spreading convention showing through a +column we populate with nulls. A caller that aggregates `spread_contracts` for Legacy and gets +0 has measured our null-handling, not the market. + +--- + +## 4. Supplemental against everything else: a different basis + +The Supplemental (Commodity Index Trader) report is **futures-and-options COMBINED**, where +Legacy, Disaggregated and TFF are all futures-only. Its `Open_Interest_All` is therefore a +different quantity for the same market and week: WHEAT-SRW on 2026-07-28 is **566,358 +combined against 463,502 futures-only**. + +**Nothing in the file says so.** It is the one report with no `FutOnly_or_Combined` column, so +`cotdata` asserts the flag rather than reading it, established by matching open interest +against both Legacy series 390/390 and 0/390. + +**Its Index Traders does NOT nest inside Disaggregated's Swap Dealer.** The taxonomy is +Legacy, not Disaggregated, so the two cannot be differenced to isolate levered swap flow. The +index book is carved out of commercial, non-commercial **and non-commercial spreading**, three +buckets where CFTC's own prose names two. + +**Coverage is 12 markets, then 13 from 2013**, when Soybean Meal entered. Both counts circulate +because both are right for part of the history, and six markets were renamed without changing +code. `cotdata-vintage coverage` derives the covered set from the data and prints every entry +and exit. + +Detail: [`../analysis/2026-08-03-cit-supplemental-measurements.md`](../analysis/2026-08-03-cit-supplemental-measurements.md). + +--- + +## 5. What this permits + +The enumerated safe comparisons, so that a consumer does not have to infer them from the +prohibitions: + +- **Open interest, Legacy against TFF**: identical, exactly, always. Safe. +- **`nonreportable`, Legacy against TFF**: identical, exactly, always. Safe. +- **Open interest, any futures-only report against the Supplemental**: **not** safe, different + basis. +- **Anything above the reportable line, across any two reports**: not safe. +- **Within one report, across weeks**: safe, subject to the gap rule (§6). + +--- + +## 6. A related trap that is not about reports but bites the same callers + +**Gaps in a market's series come from thin markets falling out of the report**, not from data +loss. Oats (`004603`) has a **294-day interval ending 2025-09-09** and five more over 50 days: +a thin market drops below the reporting threshold and reappears when it recovers. Any weekly +differencing needs an explicit gap rule, or one delta spans the whole absence and enters every +ranking as the largest flow in the sample. + +**The Oct-Nov 2025 shutdown is the opposite case and is often confused with it.** It left COT +**report** dates intact and broke only **release** dates, which are all `derived` in that +window. Anything indexed by report date is unaffected; anything indexed by release date in +that window is resting on a guess. + +--- + +## Provenance + +Sections 2, 3 and 6 were measured in `crowdmon` and are restated here because `cotdata` owns +the facts and crowdmon is deprecated. The original sections, with their reproducers and the +dates they were established, are: + +- §2, §3: `crowdmon/docs/design/amendments-2026-08-04.md` §D7, reproducer + `crowdmon/docs/analysis/reproduce_single_number.py::d7_legacy_and_tff_share_two_things` +- §6, gaps: `crowdmon/docs/design/amendments-2026-08-01.md` §A2 +- §6, shutdown: `crowdmon/docs/design/amendments-2026-08-01.md` §A1 + +They are **restated rather than moved**: those files are point-in-time records under that +repo's doc lifecycle and are not edited by this harvest. See +`crowdmon/docs/HARVEST.md` for the full map of what was ported, what was already resolved, and +what was parked with the hypothesis. + +Section 4 was authored here from the start and is summarised rather than moved. diff --git a/docs/design/reading-the-store.md b/docs/design/reading-the-store.md new file mode 100644 index 0000000..eb5ba7c --- /dev/null +++ b/docs/design/reading-the-store.md @@ -0,0 +1,157 @@ +# Reading the store: universe, coverage, continuity and price tiers + +**Living document.** Four things a consumer of this store needs before it computes anything, +each of which has already cost a session that reasoned from a plausible-looking column or a +plausible-looking count. None of them is about report comparability, which has its own +document: [`cross-report-comparability.md`](cross-report-comparability.md). + +--- + +## 1. The Disaggregated universe is mostly power and gas basis + +Of the 279 markets in the Disaggregated report, **213 (76%) are ICE Futures Energy Division or +Nodal Exchange**: power hubs, gas basis, carbon allowances, renewable energy certificates. Not +classic outrights. + +The consequence is specific rather than decorative. **Any cross-market statistic computed over +"all Disaggregated markets" is mostly a statement about ERCOT and PJM.** A principal component +over positioning changes across the full universe produces a PC1 describing power basis, not +the macro book, and anything downstream of it inherits that population. + +State which universe a result is over, before computing it. + +--- + +## 2. A coverage ratio whose denominator nobody chose is not a measurement + +This has now been reported as a shortfall and then withdrawn **twice**, in two different +packages, for the same reason: + +- "42 of 95 joinable, 44%", withdrawn once it turned out all 42 deployed markets joined. +- "25 of 279 markets", framed as a 9% sample and a limitation to be noted beside every result. + It is not a sample. The other 254 are markets **nobody trades**: no liquidity, no access, or + they are the power and gas basis contracts of §1. A join reaching every market in the + tradeable universe is complete, not 9% complete. + +The rule that generalises: **the denominator has to be a set someone chose.** "Every market +the CFTC publishes" is not a target any consumer has, so a ratio against it measures nothing. +Where a coverage figure is genuinely useful, derive it from the data at read time rather than +quoting a number from prose. Counts here move: covered markets have been 25, then 45, then 47 +as specs were added, and every one of those figures was correct on the day it was written and +stale within a fortnight. + +--- + +## 3. A market code is not an instrument, and a hole is two different things + +A gap in a code's weekly series has two unrelated causes, and only one of them is a migration. +Telling them apart requires joining a code's **internal continuity** to its **siblings**, +which nothing does automatically. + +Over 51 codes in the two current-state panels, **46 have a longest inter-week gap of 8 days**, +which is a holiday shift. The five that do not split into two causes. + +**Cause one: a migration, where a sibling code fills the hole.** Russell 2000: + +| code | venue | weeks | span | longest internal gap | +|---|---|---|---|---| +| `239742` | CME | 587 | 2006-06-13 to 2026-07-28 | **3,255 days, 8.9 years**, ending 2017-08-15 | +| `23977A` | ICE | 516 | 2008-07-22 to 2018-06-05 | none | + +The two are **complementary, not redundant**: the ICE code covers the CME code's hole almost +exactly, and together they are a continuous twenty-year market. Read apart, the CME code looks +like it began in 2017. A downstream percentile with a three-year warm-up then does not reach +it until 2023, and anything scoring an event in that window scores it on the **retiring** +venue against a much shorter reference series than its peers. + +Lumber is the clean contrast: `058643` hands off to `058644` in 2023 with a two-month overlap +and no holes on either side. + +**Cause two: a thin market falling out of the report.** Oats (`004603`) has a **294-day +interval ending 2025-09-09** and five more over 50 days. A market below the reporting +threshold disappears and returns when it recovers. No sibling fills these, because nothing is +missing: the market genuinely was not reported. + +**Consequences.** Any weekly differencing needs an explicit gap rule, or one delta spans the +whole absence and enters every ranking as the largest flow in the sample. Any per-code history +length is a statement about the code, not the market. And a merge across sibling codes must +precede any differencing, because the other order fails silently. + +--- + +## 4. Volume: the fuller-sounding parameter is the narrower series + +`get_prices` takes `volume="front"` or `volume="reconstructed"`, documented as "continuous +front-month volume" and "true market volume (first + second expiring contract)". + +**The second reads like the fuller series and is the narrower one.** +`Volume_Reconstructed = FirstVolume + SecondVolume`, exactly two expiries, while the plain +`Volume` field behind `front` spans the whole curve. + +Two independent measurements establish that `front` is whole-market: + +1. **Open interest matches the CFTC to the contract.** The price files carry an + `Open Interest` column Norgate collects from the exchange; the CFTC collects its own from + clearing members. Against COT total-market open interest for the same Tuesday: **exact + agreement on 25 of 26 markets**, palladium at 0.998, median ratio **1.000**. Two vendors + and two collection paths cannot agree that precisely unless both measure the whole market. + Front-month data would be a fraction. +2. **Curve concentration orders exactly as contract structure predicts.** + +This is recorded because the naming cost real time: several places across two repos stated +that whole-market volume was absent from the store, and all of them were describing a naming +problem as a data gap. + +--- + +## 5. Price tiers, and who can produce a store that carries them + +Three facts that were each written down separately and never composed: + +1. **`propadj` is derived on read** from `unadj` + `backadj`. It is not a stored tier. +2. **Norgate is the only vendor supplying all tiers, and it is Windows-only by mechanism + rather than by licence**: `norgatedata` talks to a locally installed Norgate Data Updater + application rather than to an API + ([`../../src/cotdata/providers/norgate.py`](../../src/cotdata/providers/norgate.py)). +3. **databento owes exactly one series per symbol**, `backadj`, because crucible-stack + ADR-0007 scopes it to the Linux dashboard's needs and says its coverage "should not be + broadened toward parity with Norgate". + +**Composed: a databento-backed futures store cannot produce `propadj` at all**, so any +consumer needing correct percentage returns cannot be served by one. This is a live constraint +on ADR-0007 step 2 rather than a historical note, and it is a **tier fact, not an operating +system fact**, a distinction that has been got wrong more than once: + +| box | Python >= 3.10 | Norgate | can produce a store carrying all tiers | +|---|---|---|---| +| Windows | yes | **yes** | **yes** | +| macOS | yes | no | no | +| Linux server | no, runs 3.9 | no | no | + +The reason a wrong tier is a `raise` and not a warning is that the error survives a spot check: +`backadj` percentage volatility is 201x too high for soybeans and **0.47x for gold**, which +never goes negative and passes every implausibility screen. + +--- + +## Provenance + +Every section here was measured in `crowdmon` and is restated because `cotdata` owns the facts +and `crowdmon` is deprecated. Restated, never moved: those files are point-in-time records +under that repo's doc lifecycle and are not edited by the harvest. + +| section | source | +|---|---| +| §1 | `crowdmon/docs/design/amendments-2026-08-01.md` §A5 | +| §2 | `crowdmon/docs/design/amendments-2026-08-01.md` §A14, and §C12 of the 08-03 file for the moving count | +| §3 | `crowdmon/docs/design/amendments-2026-08-02.md` §B26, §B27, §B30; oats from §A2 | +| §4 | `crowdmon/docs/design/amendments-2026-08-01.md` §A13 | +| §5 | `crowdmon/docs/design/amendments-2026-08-04.md` §D14 | + +`crowdmon/docs/HARVEST.md` is the full map of what was ported, what was already resolved +upstream, and what was parked with the hypothesis. + +**One deliberate omission.** §5's source composes its argument partly through +`crowdmon.futures.contract_master.ContractMaster.coverage()`, and that API goes with the +package. Nothing here points at it; the three facts above stand on `cotdata`'s own providers +and on ADR-0007.