diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bd4f50..7f5e2f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,89 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added +- **The CFTC Supplemental (Commodity Index Trader) report, as a fourth `report_type`.** + 13 select agricultural markets with the index-trader book split out of the commercial and + non-commercial buckets. New producer action `cotdata-cot --cot-supplemental` (also in + `--cot-all`) writing `cot_supplemental/{symbol}_{code}.parquet`, new + `cotdata.get_cot(sym, report="supplemental")`, and `supplemental` added to the vintage + capture set from 2006 onward with `canonicalize_supplemental` behind it. Scope recorded + in [ADR-0002](docs/adr/ADR-0002-supplemental-report-is-in-scope.md); it is CFTC + positioning, so it sits inside ADR-0007's narrowed boundary on the same argument that + admitted vintage provenance. + + **Purely additive.** Nothing was added to the natural key, `VALUE_FIELDS` or + `ALL_COLUMNS`, so no stored `row_sha256` moves and no existing observation is touched. + A store that never runs the new action is unchanged. + + Four things came back different from what the brief and the CFTC prose said, all + measured against the real 2006-2026 archives + ([docs/analysis/2026-08-03-cit-supplemental-measurements.md](docs/analysis/2026-08-03-cit-supplemental-measurements.md)): + + - **Coverage is 12 markets, then 13 from 2013**, when Soybean Meal entered. Both counts + circulate because both are right for part of the history. That is the only entry and + there are no exits, and **six** markets were renamed without changing code (four NYBOT + to ICE in 2007, the two wheats relabelled in 2013, including 001612 changing exchange + from KCBT to CBOT and keeping its code), which is why coverage keys on `market_code` + and reports the name at the latest report date rather than the lexicographic max. + - **The report is futures-and-options combined and the file cannot say so** — it carries + no `FutOnly_or_Combined` column, so the existing `_combined_flag` would have defaulted + it to `False` and put a guessed value in the natural key. Established by matching open + interest against both Legacy series: **390/390** against combined, **0/390** against + futures-only. `canonicalize_supplemental` asserts it and takes no override. + - **Index traders are carved out of three buckets, not the two CFTC describes**: the + third is non-commercial *spreading*, ~0% of the index book in 2006 and ~9% now. + - **The fetch pattern differs from the other three reports.** No 2006-2016 history bundle + (404), and every year from 2006 returns 200 where disagg/TFF 404 before 2010, so + `annual_sources` needed its own floor. + + **The open-interest identity is exact on only ~55% of market-weeks, and that is + rounding.** The residual never exceeds 2 contracts against a tolerance of 4, and the rate + is flat across 21 years (51.6% to 59.5% by year, sd 2.1pp; the *breach* rate, either side + off, is 67.7% and ranges 62.6% to 71.7%). Control: on the same weeks, + Legacy *futures-only* is exact on 99.7% of rows while Legacy *combined* shows the + identical +/-1 pattern on 10%. Combined reports publish delta-weighted option equivalents + rounded to whole contracts, independently per category, which is the same n-addends + mechanism `rounding_tolerance` was already derived from. `validate()` raises zero + warnings across the full history. + + **The category vocabulary check must stay per-`report_type`.** Supplemental reuses + `commercial` and `noncommercial`, and they do not mean what they mean under Legacy: they + are net of index traders, and this report is combined where Legacy here is futures-only. + Reusing the labels rather than minting `non_commercial`-style spellings is deliberate — a + new spelling would make `category == "nonreportable"` silently miss every Supplemental + row while leaving `commercial`, the genuinely confusable label, identical anyway. + `report_type` and `combined` are both in the natural key, so the series cannot merge. + + **Index Traders does NOT nest inside Disaggregated's Swap Dealer.** The taxonomy is + Legacy, not Disaggregated, so the two cannot be differenced to isolate levered swap flow. + Anything relating them is an inference across differently-partitioned reports. +- **`cotdata-vintage coverage`** — emits which markets a report actually covered per year, + derived from the stored observations rather than from a list in source, and prints every + entry and exit. A covered set treated as constant is how a consumer reports on 12 markets + believing it has 13. It compares **consecutive years only** and emits a `gap` record + otherwise: coverage is derived from what was INGESTED, so a store holding 2006 and 2026 + and nothing between would otherwise report Soybean Meal as entering in 2026, presenting + an ingest artifact as a fact about the market. + +### Changed +- `vintage_ingest._resolve` now tolerates CFTC's `Postions`/`Positions`, `Spead`/`Spread` + and `NComm`/`NonComm` header variants alongside the single/double-underscore one it + already handled, and **composes up to three of them**. All are real spellings in shipped + CFTC files, three of them typos. Composition is what makes the tolerance useful on the + column that needs it most: `NComm_Postions_Spread_All_NoCIT` carries two defects at once, + so the realistic upstream cleanup — fix the typo and normalise the prefix in one pass — + is exactly the case a single-substitution search cannot reach. + + This widens a lookup that previously RAISED on anything it did not recognise, so the + regression to worry about is a variant silently landing on a different field's column. + It cannot: across every column name the four canonicalisers ask for, no candidate + spelling of one field is a candidate of another, and no target matches two real columns + in the shipped 2026 Legacy, Disaggregated, TFF or Supplemental headers. Both properties + are asserted, not argued, by + `tests/test_cit_supplemental.py::test_header_variants_cannot_resolve_to_another_field`. + A genuine rename still raises. + ## [0.3.0] - 2026-08-02 **The first release actually published to PyPI since 0.1.0.** 0.2.0 was tagged earlier diff --git a/README.md b/README.md index b6f64b1..79fedcb 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,7 @@ The **store is the API boundary** — not Python imports. Producers write Parque ┌────────────────────────────────────────────────────────────┐ │ CANONICAL STORE ($COTDATA_STORE) │ │ prices/ cot_legacy/ cot_disagg/ cot_tff/ │ + │ cot_supplemental/ │ │ metadata/ manifest.json status.json │ └────────────────────────────────────────────────────────────┘ │ read (offline, any OS) @@ -82,6 +83,7 @@ The store layout: - `cot_legacy/{symbol}_{code}.parquet` — weekly CFTC Legacy positioning. - `cot_disagg/{symbol}_{code}.parquet` — weekly CFTC Disaggregated positioning. - `cot_tff/{symbol}_{code}.parquet` — weekly CFTC Traders in Financial Futures positioning. +- `cot_supplemental/{symbol}_{code}.parquet` — weekly CFTC Supplemental (Commodity Index Trader) positioning, 13 agricultural markets. **Futures-and-options combined**, unlike the three above. - `metadata/contract_specs.parquet` — Norgate contract specifications (tick size, point value, margin). - `manifest.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`. - `status.json` — machine-readable new-data signal for downstream tools (see [Operations](#operations)). @@ -101,12 +103,18 @@ signals = cotdata.get_prices("ES", adjustment="backadj") # signals + stops (gap sizing = cotdata.get_prices("ES", adjustment="unadj") # position sizing (true dollar prices) milk = cotdata.get_prices("DC", adjustment="propadj") # ratio-adjusted: strictly positive, %-return preserving -# COT — three CFTC report families: +# COT — four CFTC report families: legacy = cotdata.get_cot("ES", report="legacy") # Commercial / Non-Commercial disagg = cotdata.get_cot("ES", report="disagg") # Managed Money, Swap Dealers, ... (commodities) tff = cotdata.get_cot("ES", report="tff") # Leveraged Funds, Asset Managers, ... (financials) +cit = cotdata.get_cot("CC", report="supplemental") # Index Traders (13 ags, combined basis) ``` +> [!WARNING] +> `supplemental` is **futures-and-options combined** and the other three are futures-only, so +> its `Open_Interest_All` is a different quantity for the same market and week. Do not +> difference or ratio across reports without accounting for that. + A price frame (`get_prices("ES", adjustment="backadj").tail(3)`): ``` @@ -130,7 +138,8 @@ COTDATA_STORE=/store cotdata-update --metadata # Norgate contr COTDATA_STORE=/store cotdata-update --cot-legacy # CFTC Legacy (any OS) COTDATA_STORE=/store cotdata-update --cot-disagg # CFTC Disaggregated (any OS) COTDATA_STORE=/store cotdata-update --cot-tff # CFTC Traders in Financial Futures (any OS) -COTDATA_STORE=/store cotdata-update --cot-all # all three CFTC COT reports +COTDATA_STORE=/store cotdata-update --cot-supplemental # CFTC Supplemental / index traders (any OS) +COTDATA_STORE=/store cotdata-update --cot-all # all four CFTC COT reports COTDATA_STORE=/store cotdata-vintage fetch # optional: capture as-published COT (any OS) ``` @@ -269,7 +278,7 @@ Every producer run writes `$COTDATA_STORE/status.json` (atomically, beside the d { "generated_at": "2026-07-15T10:15:24Z", "schema_version": 2, - "newest_data": { "prices": "2026-07-14", "cot_legacy": "2026-07-07", "cot_disagg": "2026-07-07", "cot_tff": "2026-07-07" }, + "newest_data": { "prices": "2026-07-14", "cot_legacy": "2026-07-07", "cot_disagg": "2026-07-07", "cot_tff": "2026-07-07", "cot_supplemental": "2026-07-07" }, "domains": { "prices": { "newest_data": "2026-07-14", "last_write": "2026-07-15T10:15:24Z", "entries": 84, "rows": 829096, "lagging": 0 }, "...": {} }, "last_run": { "kinds": ["prices"], "ok": ["ES", "..."], "symbols_failed": [], "rows": 1658000, "seconds": 88, "at": "2026-07-15T10:15:24Z" } } @@ -342,7 +351,8 @@ cotdata-vintage fetch --all # every year 1986-present (see below) cotdata-vintage ingest --pending # parse retained raw -> observations + revisions cotdata-vintage diff --since 2026-01-01 # field-level revisions, with revision depth cotdata-vintage asof --as-of 2026-07-24T18:00:00 --report-date 2026-07-21 -cotdata-vintage flow --market 088691 # weekly flow decomposition (see below) +cotdata-vintage zero-sum --market 088691 # long/short/open-interest identity per week +cotdata-vintage coverage # which markets a report covers, per year cotdata-schedule sync # CFTC Special Announcements cotdata-schedule published # true publication dates from retained weekly statics cotdata-schedule backfill # resolve release_date + its provenance @@ -353,11 +363,17 @@ How it works: - **Immutable landing zone.** Every fetch is recorded (including 304s) and raw bytes are retained permanently under `vintage/raw/`, written atomically and never rewritten. A byte-identical regeneration is deduped — a changed download is not itself a revision. -- **All three reports.** Legacy, Disaggregated and TFF canonicalise into one long schema. - Disagg and TFF also populate per-category spreading, per-category trader counts and - CR4/CR8 concentration, none of which the Legacy file carries, and they are where - **Managed Money** and **Leveraged Funds** live. A suppressed trader count (CFTC writes - `.`) canonicalises to null rather than to a string. +- **All four reports.** Legacy, Disaggregated, TFF and Supplemental canonicalise into one + long schema. Disagg and TFF also populate per-category spreading, per-category trader + counts and CR4/CR8 concentration, none of which the Legacy file carries, and they are + where **Managed Money** and **Leveraged Funds** live. A suppressed trader count (CFTC + writes `.`) canonicalises to null rather than to a string. Supplemental is where **Index + Traders** live, and it is the one report where `combined` is `True`: the category + vocabulary is checked per `report_type`, so its `commercial` (which is net of index + traders) can never be mistaken for Legacy's. +- **Coverage is not constant, and `cotdata-vintage coverage` says so.** A report's market + set is derived from the stored observations and every entry or exit is printed. The live + case is Supplemental, which covered 12 markets until Soybean Meal entered in 2013. - **Change-only observations.** A row is written only when its value hash differs from the latest for its natural key `(report_date, market_code, report_type, combined, category)`, so storage grows with actual revisions rather than with time. @@ -503,13 +519,26 @@ Entity-specific positioning and trader counts for financial markets (CFTC TFF Fu > [!NOTE] > **Lossless Image**: Like Disaggregated, TFF parquets are a **lossless image** of the source CFTC `txt` files — the financial entity groups (`Dealer`, `Asset_Mgr`, `Lev_Money`, `Other_Rept`) and their `Traders_*` counts. +### COT Supplemental Data (`cot_supplemental/{symbol}_{code}.parquet`) +Index-trader positioning for 13 select agricultural markets (CFTC Supplemental / Commodity Index Trader Report). **History starts in January 2006**, and the covered set was **12 markets until Soybean Meal entered in 2013**. Indexed by tz-naive `Report_Date_as_MM_DD_YYYY` (renamed at parse from CFTC's `As_of_Date_In_Form_*`, whose own name changed in 2013). + +> [!WARNING] +> **Combined basis, and the file does not say so.** This is the only report with no futures-only variant, and unlike Disaggregated and TFF it carries no `FutOnly_or_Combined` column. Verified by matching open interest against both Legacy series: 390/390 against futures-and-options combined, 0/390 against futures-only. Its open interest is therefore **not** comparable with the other three for the same market and week. + +> [!NOTE] +> **Positions are net of index traders.** `Comm_Positions_Long_All_NoCIT` is commercial *minus* the index book, not Legacy's commercial. The index book is carved out of commercial, non-commercial **and non-commercial spreading** — three buckets, where CFTC's own prose names two. Non-Reportable is untouched, because index traders are reportable by definition. + +> [!NOTE] +> **Lossless Image**: Like Disaggregated and TFF, these parquets are a lossless image of the source CFTC `txt` files, including CFTC's own header typos (`NComm_Postions_Spread_All_NoCIT`). + ## Reference: COT formats explained -The CFTC publishes positioning data in three formats; `cotdata` manages all three for complete coverage and the deepest history. +The CFTC publishes positioning data in four formats; `cotdata` manages all four for complete coverage and the deepest history. 1. **Legacy (1986–Present)** — *all markets.* Divides traders into **Commercial** (hedgers) and **Non-Commercial** (large speculators). The only format with pre-2006 data, so it's essential for long-term backtesting. 2. **Disaggregated / DIS (2006–Present)** — *physical commodities only* (Agriculture, Energy, Metals). Splits traders into **Producer/Merchant**, **Swap Dealers**, **Managed Money**, and **Other Reportables** — a clearer view of "smart money" (Managed Money) in commodities. 3. **Traders in Financial Futures / TFF (2006–Present)** — *financial markets only* (Equities, Rates, Currencies). Splits traders into **Dealer/Intermediary**, **Asset Manager**, **Leveraged Funds**, and **Other Reportables** — the definitive source for speculative flow (Leveraged Funds) in financials. +4. **Supplemental / CIT (2006–Present)** — *13 select agricultural markets only.* Takes the Legacy split and carves **Index Traders** out of it, the only public source that separates index flow. Futures-and-options **combined** only. The taxonomy is Legacy, not Disaggregated, so Index Traders does **not** nest inside Disaggregated's Swap Dealer and the two cannot be differenced to isolate levered swap flow. ## Diagnostics diff --git a/docs/adr/ADR-0002-supplemental-report-is-in-scope.md b/docs/adr/ADR-0002-supplemental-report-is-in-scope.md new file mode 100644 index 0000000..ab4f359 --- /dev/null +++ b/docs/adr/ADR-0002-supplemental-report-is-in-scope.md @@ -0,0 +1,69 @@ +# ADR-0002: the Supplemental (CIT) report is inside narrowed cotdata's boundary + +**Status:** accepted, 2026-08-03 +**Context:** [ADR-0001](ADR-0001-cot-vintage-provenance-in-parquet.md) (local pointer to +crucible-stack ADR-0008), crucible-stack ADR-0007 +**Supersedes:** nothing + +## Decision, in one line + +The CFTC **Supplemental (Commodity Index Trader)** report is CFTC positioning, so it is in +scope for the narrowed `cotdata`, on exactly the argument that admitted vintage provenance. + +## Context + +crucible-stack ADR-0007 narrows `cotdata` to **CFTC positioning only** and moves all price +series, futures included, out to `marketdata`. ADR-0008 then had to answer whether vintage +provenance — the raw snapshots, the bitemporal observations, the revision log — was inside +that narrowed boundary, and concluded it was: provenance about CFTC positioning is CFTC +positioning. + +Adding a fourth report type raises the same question in a milder form, and it should not be +left implicit. Three reports were already here when ADR-0007 was accepted, so their being +in scope is inherited rather than decided. + +## Decision + +In scope. The Supplemental is a weekly CFTC positioning report on the same release +schedule, the same Tuesday as-of convention, and the same distribution channel as the three +already ingested. Refusing it would draw the boundary at "the reports we happened to have +in July 2026" rather than at "CFTC positioning", which is not a boundary anyone could apply +to the next case. + +Nothing about it pulls price data back across the ADR-0007 seam. It carries no price series +and no contract specification. Its one novel property, being futures-and-options combined, +is a property of CFTC's own aggregation, not of anything `marketdata` owns. + +## Consequences + +- A fourth `report_type` in the canonical vintage schema, a fourth `cot_*` store domain + (`cot_supplemental`), and a fourth producer action (`--cot-supplemental`), all on the + `cot` half of the manifest seam. See the 0.4.0 changelog entry. +- The **category vocabulary check must stay per-`report_type`**. Supplemental reuses the + labels `commercial` and `noncommercial`, and they do not mean what they mean under + Legacy: they are net of index traders, and the report is combined where Legacy here is + futures-only. `report_type` and `combined` are both in the natural key, so the two can + never merge into one series, but a consumer summing across report types would still be + wrong. +- **No change is proposed to ADR-0007 or ADR-0008.** This is an application of the rule + ADR-0007 already states, not a new decision about the boundary, and those documents live + in the `crucible-stack` checkout where this session has no business editing a shared + working tree. Recording it here keeps it discoverable from the repo it constrains, which + is the same reason ADR-0001 exists as a local pointer. + +## The reason this is worth a file rather than a comment + +The report exists to separate index flow from the rest, and the requesting consumer +(`crowdmon`) wants it because a single Swap Dealer fragility weight has been measured doing +incoherent work: on cocoa the largest net long is the swap dealer, so **fragile** capital +sits at w=0.4, while on gold the immovable physical hedger is a swap dealer with +Producer/Merchant at a tenth of the swap book, so **robust** capital sits at the same 0.4. +Opposite errors, one weight. + +That makes it tempting to reason about the index/non-index split here. Do not: this package +ingests and validates, and the analysis is the companion handoff in `crowdmon`. The one +thing worth carrying across the seam is a warning the data supports and the CFTC prose does +not — Supplemental's Index Traders is a **Legacy-taxonomy** carve-out and does **not** nest +inside Disaggregated's Swap Dealer, so the two cannot be differenced to isolate levered +swap flow. Measured detail in +[docs/analysis/2026-08-03-cit-supplemental-measurements.md](../analysis/2026-08-03-cit-supplemental-measurements.md) §5. diff --git a/docs/analysis/2026-08-03-cit-supplemental-measurements.md b/docs/analysis/2026-08-03-cit-supplemental-measurements.md new file mode 100644 index 0000000..e63a69c --- /dev/null +++ b/docs/analysis/2026-08-03-cit-supplemental-measurements.md @@ -0,0 +1,210 @@ +# Supplemental (CIT) report: what the files actually say + +**Date:** 2026-08-03. Point-in-time. Per the doc lifecycle, `analysis/` is never amended: +if a later measurement contradicts this, write a new dated file rather than editing this +one. Corrected once before merge, on the same day, while it existed only on the branch that +introduced it (§3's rename count, and §4's exact-vs-breach rates, both caught in +pre-merge review). Nothing was published under the wrong numbers; from here the file is +frozen. + +Every number here comes from the real CFTC archives, `dea_cit_txt_2006.zip` through +`dea_cit_txt_2026.zip` (21 files, 13,584 market-weeks, 2006-01-03 to 2026-07-28), +downloaded live on 2026-08-03, plus `dea_fut_xls_*.zip` (Legacy futures-only) and +`dea_com_xls_*.zip` (Legacy futures-and-options combined) as controls. Reproducers are +the probe scripts described in §7. + +This exists because `docs/handoffs/2026-08-03-cit-supplemental.md` asked for the report's +properties to be **derived from the data rather than assumed**, and four of them came back +different from the handoff or from the CFTC prose. + +--- + +## 1. It is futures-and-options combined, and the file does not say so + +The Supplemental has no `FutOnly_or_Combined` column. Disaggregated and TFF both do, which +is what `vintage_ingest._combined_flag` reads, so the Supplemental would have defaulted to +`combined=False` and put a guessed value into the natural key. + +Established instead by matching against the two Legacy series on identical +(market_code, report_date) keys, 2026, 390 market-weeks, all 13 markets: + +| compared against | `Open_Interest_All` match | `NonRept_Positions_Long_All` match | +|---|---|---| +| Legacy futures-only (`annual.xls`) | **0 / 390** | 0 / 390 | +| Legacy futures-and-options combined (`annualof.xls`) | **390 / 390** | 390 / 390 | + +Wheat-SRW on 2026-07-28: Supplemental OI 566,358, Legacy combined 566,358, Legacy +futures-only 463,502. + +`canonicalize_supplemental` therefore asserts `combined=True` and takes no override, and +raises if a `FutOnly_or_Combined` column ever appears saying otherwise. + +## 2. The date column is named two different things, and the rename was cosmetic + +| years | column name | +|---|---| +| 2006-2012 | `As_of_Date_In_Form_MM/DD/YYYY` | +| 2013-2026 | `As_of_Date_In_Form_YYYY-MM-DD` | + +The **values are ISO `YYYY-MM-DD` in every year**, including the ones whose header claims +MM/DD/YYYY: the 2006 file's first row carries `2006-12-26` under the MM/DD/YYYY heading. +So 2013 fixed a mislabelled header, not a format. `providers/cftc_cit._parse_zip` accepts +either and renames to the repo's `Report_Date_as_MM_DD_YYYY`; it raises if it finds +neither, rather than letting `_canonicalize` fall back to the frame's positional index. + +Two other columns were renamed at the same boundary, both unused here: +`Pct_OI_Tot_Rept_{Long,Short}_All_NoCIT` and `Traders_Tot_Rept_{Long,Short}_All_NoCIT` +lost the `_NoCIT` suffix in **2012**, a year earlier than the date column. Column count is +54 in every year. + +Report dates are Tuesdays (9,114 of 9,192 distinct market-weeks in the 2013+ sample), with +78 Mondays where a holiday shifted the week. That matches the rest of the COT family. + +## 3. Coverage is 12 markets, then 13 from 2013 + +Both counts circulate because both are right for part of the history. + +| years | markets | +|---|---| +| 2006-2012 | 12 | +| 2013-2026 | 13 | + +**Soybean Meal (026603) entered in 2013**, first report date 2013-01-08. That is the only +entry and there are no exits in 21 years. Nothing else changed: no market code appears or +disappears, and the twelve originals run unbroken 2006-2026. + +**Six** codes were **renamed** without changing code, in two waves, which is why +market-name matching would have produced spurious entries and exits and market_code +matching does not: + +| code | old name | current name | renamed | +|---|---|---|---| +| 033661 | COTTON NO. 2 - NEW YORK BOARD OF TRADE | COTTON NO. 2 - ICE FUTURES U.S. | 2007 | +| 073732 | COCOA - NEW YORK BOARD OF TRADE | COCOA - ICE FUTURES U.S. | 2007 | +| 080732 | SUGAR NO. 11 - NEW YORK BOARD OF TRADE | SUGAR NO. 11 - ICE FUTURES U.S. | 2007 | +| 083731 | COFFEE C - NEW YORK BOARD OF TRADE | COFFEE C - ICE FUTURES U.S. | 2007 | +| 001602 | WHEAT - CHICAGO BOARD OF TRADE | WHEAT-SRW - CHICAGO BOARD OF TRADE | 2013 | +| 001612 | WHEAT - KANSAS CITY BOARD OF TRADE | WHEAT-HRW - CHICAGO BOARD OF TRADE | 2013 | + +001612 is the sharp one: the market changed exchange (KCBT to CBOT, after CME Group +acquired the Kansas City Board of Trade) and kept its CFTC code. Market names in the +2006-2012 files also carry a trailing space. + +**The 2007 wave is also why "the latest name" cannot be the lexicographic max**, which is +what a first pass of `derive_coverage` used: `'I' < 'N'`, so a max-of-strings rule reports +the pre-2007 NYBOT name for all four ICE markets in every year after the rename. The wheat +pair comes out right under either rule, which is exactly how it survived a first reading. +Caught in pre-merge review; the shipped rule takes the name at the latest report date. + +All 13 codes resolve in `registry.yaml`: ZW, KE, ZC, ZM, CT, HE, ZS, LE, GF, CC, ZL, SB, KC. + +## 4. The open-interest identity, and why the exception rate is 45% rather than 0% + +Summing the four canonical categories per side and adding non-commercial spreading: + +| | exact | per-year exact | residual | +|---|---|---|---| +| long side | 55.2% of 13,584 market-weeks | 51.6% - 59.5%, sd 2.1pp | never more than **2 contracts** | +| short side | 55.4% | — | never more than 2 contracts | + +Counting a market-week as a breach if **either** side is off, the **breach** rate is 67.7% +(per-year 62.6% in 2012 to 71.7% in 2023, sd 2.7pp). Two different rates, so keep the +qualifier attached: 55% is how often the identity is *exact on one side*, 68% is how often +*either side* is off by a contract or two. + +Both are **stable across the whole history** — no trend, no regime break, nothing at the +2013 coverage change and nothing at 2018. + +Using CFTC's own published `Tot_Rept_Positions_*` columns instead of summing categories, +the identity `Tot_Rept + NonRept == OI` is exact on 74.9% of rows and off by **at most 1** +on the rest. + +**The cause is combined reporting, not the Supplemental.** Control, on the same 2026 weeks +and the same identity: + +| file | exact | +|---|---| +| Legacy **futures-only** | 99.7% (31 breaches in 10,719 rows) | +| Legacy **combined** | 90.1%, residual +/-1 (a handful at +/-2) | +| Supplemental (combined) | 55% | + +A futures-and-options report publishes delta-weighted option equivalents rounded to whole +contracts, independently per category. Summing `n` independently rounded values admits at +most `n` contracts of error, which is exactly the bound `rounding_tolerance` already +returns (4 for the Supplemental's four categories). The Supplemental sits lowest of the +three because it has one more rounded addend than Legacy combined. + +This is worth stating plainly because it looks alarming and is not: **this is rounding, not +a data-quality problem**, the residual never approaches the tolerance, and `validate()` +raises zero soft warnings across all 21 years. + +## 5. Index traders are carved out of three buckets, not two + +CFTC's prose says index traders are drawn from the commercial and non-commercial +categories. Differencing the Supplemental against Legacy combined on the same keys shows a +third source: **non-commercial spreading**. + + CIT_Long = (Comm_Long - Comm_Long_NoCIT) + + (NonComm_Long - NonComm_Long_NoCIT) + + (NonComm_Spread - NonComm_Spread_NoCIT) + +Exact on 59.5% of 2026 rows and within +/-1 on essentially all of the rest, which is the +same rounding as §4 (three rounded differences). Wheat-SRW 2026-07-28: +76,743 + 45,210 + 24,157 = 146,110 = `CIT_Positions_Long_All`, exactly. + +No component is ever negative in any sampled year, so this is a partition rather than a +netting. + +**The composition has drifted substantially**, which matters for anything treating the +index book as one stable population (median share of `CIT_Long` by source): + +| year | from commercial | from non-commercial | from NC spreading | +|---|---|---|---| +| 2006 | 0.886 | 0.113 | 0.000 | +| 2010 | 0.808 | 0.188 | 0.003 | +| 2015 | 0.716 | 0.270 | 0.007 | +| 2020 | 0.622 | 0.274 | 0.094 | +| 2026 | 0.555 | 0.339 | 0.088 | + +Non-Reportable is untouched: index traders are reportable by definition, which is why it +has no `_NoCIT` variant and why it matches Legacy combined exactly (§1). + +## 6. Header spellings + +CFTC's header row carries two long-standing typos and one internal inconsistency. All are +tolerated by `_resolve` rather than normalised at parse, so a future correction is not a +breakage: + +- `NComm_Postions_Spread_All_NoCIT` — "Postions", missing the `i` +- `Change_NonComm_Spead_All_NoCIT` — "Spead" +- position columns say `NComm_`, while `Pct_OI_` and `Traders_` columns say `NonComm_` +- `Change_Comm_Short_All_NoCIT ` carries a trailing space + +## 7. Distribution details + +- Every year 2006-2026 is an individual zip; **there is no `dea_cit_txt_hist_2006_2016.zip`** + (404). Disaggregated and TFF are bundled that way, the Supplemental is not. +- Unlike disagg/TFF, whose annual zips 404 before 2010, **all 21 Supplemental years return + 200**, so `annual_sources` needs a separate 2006 floor rather than sharing the 2010 one. +- Archive member is `annualci.txt` in every year except 2006, which is `annualci2006.txt`. + Nothing here depends on the name (the parser takes `namelist()[0]`, as the other three do). +- A weekly static exists at `https://www.cftc.gov/dea/newcot/deacit.txt` (200). It is + **not** in the capture set: the Legacy weekly static already supplies the true + publication `Last-Modified` that `vintage_schedule.sync_published` resolves release dates + from, and all COT reports publish together, so a second one adds a source to the daily + sweep and no information. + +### Reproducers + +Scripts used, in the session scratchpad rather than committed (they download ~30 MB and +depend on nothing in this repo but `pandas`): `probe_cit.py` (schema stability, coverage, +identities), `probe2.py` (per-year breakdown), `probe3.py` (futures-only vs combined), +`probe4.py` (carve-out identity, control). The committed equivalent is +`tests/test_cit_supplemental.py`, which pins every property above that can be pinned from +the trimmed fixtures, and `tests/_gen_cit_fixtures.py`, which rebuilds those fixtures from +the real archives. + +Full-history figures were reproduced end to end through the shipped code path: 21 raw +snapshots captured, ingested to **54,336 observations across 13,584 market-weeks with zero +revisions and zero validation warnings**, and a full replay of all 21 snapshots wrote zero +further observations. diff --git a/docs/handoffs/2026-08-03-cit-supplemental.md b/docs/handoffs/2026-08-03-cit-supplemental.md new file mode 100644 index 0000000..adb4e8a --- /dev/null +++ b/docs/handoffs/2026-08-03-cit-supplemental.md @@ -0,0 +1,237 @@ +# Handoff: CIT Supplemental ingestion + +**Status:** complete (branch `claude/cit-supplemental-ingestion-8686f0`) +**Date:** 2026-08-03 +**Lives at:** `cotdata/docs/handoffs/2026-08-03-cit-supplemental.md` +**Target:** Claude Code session, `cotdata` worktree +**Depends on:** vintage subsystem (PR #78, merged) +**Deliverable:** Supplemental report ingested, vintage-captured, exposed through the canonical schema +**Companion:** `crowdmon/docs/handoffs/2026-08-03-index-share.md` — unblocked once this merges + +> Executed 2026-08-03. The original brief is kept below **verbatim**; every place the data +> contradicted it is corrected inline in a blockquote marked **CORRECTION**, and the +> report-back is §7. Measurements: [docs/analysis/2026-08-03-cit-supplemental-measurements.md](../analysis/2026-08-03-cit-supplemental-measurements.md). + +--- + +## 0. Why + +`crowdmon`'s fragility weights assign Swap Dealer a single `w = 0.4`. Two measurements have shown that one number is doing incoherent work: + +- **Cocoa** — the largest net long is the Swap Dealer, so the *fragile* capital sits at 0.4 rather than 1.0. +- **Gold** — the immovable physical-hedger side is a swap dealer, with Producer/Merchant at a tenth of the swap book, so the *robust* capital sits at 0.4 rather than 0.1. + +Opposite errors, same weight. A swap dealer's net is the residual of hedged client positions, so its fragility depends on the clients: index funds are unlevered and sticky, levered macro accounts are not. The Supplemental report is the only public source that separates index flow. + +**Scope.** The Supplemental report is CFTC positioning, so it falls inside ADR-0007's narrowed boundary on the same argument that admitted vintage provenance. Record this in the existing vintage ADR or a short new one — do not leave it implicit. + +> **Done:** [ADR-0002](../adr/ADR-0002-supplemental-report-is-in-scope.md), a new short +> local ADR rather than an edit to ADR-0001. ADR-0001 is a pointer to crucible-stack +> ADR-0008, which lives in a sibling checkout; the workspace working agreement says to +> record such things in the consumer's own docs rather than editing a shared working tree. +> No change is proposed to ADR-0007 or ADR-0008, because this is an application of the rule +> ADR-0007 already states rather than a new boundary decision. + +--- + +## 1. What this report is + +| Property | Value | +|---|---| +| Coverage | 13 select agricultural contracts | +| Basis | **Futures-and-options combined only** — there is no futures-only variant | +| History | January 2006 → present, annual compressed archive | +| Cadence | Weekly, same Tuesday as-of / Friday release as the rest | +| Categories | Non-Commercial (long / short / spreading), Commercial (long / short), **Index Traders** (long / short), Non-Reportable | + +**The taxonomy is Legacy, not Disaggregated.** Index Traders is carved out of *both* the commercial and non-commercial buckets. It does **not** nest inside Disaggregated's Swap Dealer, and the two cannot be differenced to isolate levered swap flow. Any relationship between them is an inference across differently-partitioned reports. + +**Verify the contract list from the data rather than assuming it.** Sources give both 12 and 13; the count has changed over the report's life. Derive the covered set per year from the files and report it, including any market that enters or leaves. + +> **CORRECTION, three items.** +> +> 1. **"Coverage: 13" is right only from 2013.** It is 12 markets 2006-2012 and 13 from +> 2013, when Soybean Meal (026603) entered. That is the only entry, and there are no +> exits in 21 years. Both circulating counts are correct for part of the history. +> 2. **"Carved out of *both* the commercial and non-commercial buckets" understates it by +> one bucket.** Differencing against Legacy combined shows a third source, +> non-commercial **spreading**: +> `CIT_Long = ΔComm_Long + ΔNonComm_Long + ΔNonComm_Spread`, exact to the rounding in +> §4. Spreading was ~0% of the index book in 2006 and is ~9% now, so the omission was +> harmless when the report started and is not now. The rest of the paragraph stands and +> is the single most important line in this handoff: Index Traders does **not** nest +> inside Disaggregated's Swap Dealer. +> 3. **"There is no futures-only variant" is true, and the file cannot tell you so.** +> Unlike Disaggregated and TFF the Supplemental carries no `FutOnly_or_Combined` column, +> so the existing `_combined_flag` would have defaulted it to `False`. Established +> instead by matching open interest against both Legacy series: 390/390 against +> futures-and-options combined, 0/390 against futures-only. +> +> Confirmed as written: January 2006 start, annual compressed archive, Tuesday as-of with +> holiday shifts to Monday, and the category list. + +--- + +## 2. Ingestion + +Add `supplemental` as a fourth `report_type` alongside `legacy`, `disaggregated`, `tff`. + +- Annual compressed archive, same fetch pattern as the existing three +- **`combined` is always `True`** for this report — assert it, never infer it +- Route through the vintage layer from the first fetch: raw snapshot capture, change-only observation writes, field-level revisions. No exceptions. + +> **CORRECTION: "same fetch pattern as the existing three" is not quite the pattern.** +> Disaggregated and TFF are served as a 2006-2016 bundle plus individual years, and their +> individual years 404 before 2010. The Supplemental has **no history bundle** +> (`dea_cit_txt_hist_2006_2016.zip` is a 404) and **every year from 2006 returns 200**, so +> `annual_sources` needed its own 2006 floor rather than sharing the existing 2010 one. +> The date column also differs and its name changed in 2013; see the measurements doc §2. + +### Schema + +The canonical observation schema has Disaggregated/TFF category columns. Supplemental introduces `spread_contracts` on a non-commercial category and has no direct analogue for the disaggregated splits. + +**Preferred approach:** keep one observation table, extend the category vocabulary with `non_commercial`, `commercial`, `index_traders`, `non_reportable`, and let `report_type` disambiguate. `spread_contracts` already exists and is nullable. + +**Do not** map Supplemental categories onto Disaggregated names. `commercial` here is not `producer_merchant`, and conflating them will silently corrupt every downstream sum. The category vocabulary check must be per-`report_type`. + +If the existing schema cannot carry this cleanly, say so and propose an alternative before implementing one. + +> **The schema absorbed it with no changes.** One observation table, the natural key +> untouched, `spread_contracts` populated on `noncommercial` only, cr4/cr8 null because the +> report publishes no concentration ratios. Nothing was added to `VALUE_FIELDS` or +> `ALL_COLUMNS`, so no stored `row_sha256` moves. +> +> **DEVIATION on the category spellings**, deliberate. Shipped as `commercial`, +> `noncommercial`, `index_trader`, `nonreportable` rather than the brief's +> `non_commercial` / `index_traders` / `non_reportable`. The repo's existing vocabulary +> spells these `noncommercial` and `nonreportable` in all three other report types, so a +> new spelling would make `category == "nonreportable"` silently miss every Supplemental +> row — a real footgun — while leaving `commercial`, the genuinely confusable label, +> identical either way. The distinguishing-by-spelling argument fails on exactly the +> category that needed it, so consistency wins and `report_type` does the disambiguating, +> as the brief itself specifies. + +--- + +## 3. Why vintage matters more here than elsewhere + +The 2018 trader reclassification — the only documented instance of positions being moved between categories, and the failure mode §4.5 of the vintage handoff exists to detect — **affected the Supplemental report specifically**. Some traders were removed from and others added to the index classification; individually small, noticeable in aggregate. + +So this is the series with the strongest known reclassification history, and it is also the one where reclassification does the most analytical damage, since the whole point of the report is the index/non-index boundary. + +Capture from the first fetch. Category-migration detection (vintage handoff §4.5, currently deferred) should be un-deferred for this report type once a quarter of snapshots exists. + +> **Done, with a limit worth stating.** All 21 years are captured and ingested, and every +> future fetch is captured from the first one. But **the 2018 reclassification is not +> recoverable from this**, and nothing in this work makes it so: CFTC serves current state +> only, so the 2018 archive we hold today is the post-reclassification restatement. The +> vintage store detects the *next* one; it cannot reconstruct the last one. +> +> Nothing in the data marks 2018 either. The identity exception rate (§4) is flat across +> the whole history with no break at 2018, which is consistent with a reclassification that +> moved traders between categories without disturbing totals — that is precisely why it is +> invisible to every check except a vintage diff. +> +> Category-migration detection stays deferred, as instructed: a quarter of snapshots does +> not exist yet. Earliest sensible date is **2026-11-01**, the same date §7.8 of the +> crowdmon spec already waits on. + +--- + +## 4. Validation + +Existing rules apply, plus: + +- `combined == True` on every row +- Category vocabulary enforced per `report_type` — an unknown label raises +- The Legacy identity: reportable long + non-reportable long should reconcile to open interest within tolerance. **Report the exception rate rather than suppressing it.** +- Covered-market set per year emitted as a coverage artifact, so entries and exits are visible rather than silent + +> **All four in place.** Exception rate reported in §7 and in the measurements doc §4, not +> suppressed: it is 45% of rows off by a contract or two, which is combined-report rounding +> rather than a data problem, and the control experiment that establishes that is in the +> measurements doc. Coverage artifact is `vintage/coverage/supplemental.parquet`, written +> by `cotdata-vintage coverage`, derived from the observations rather than from a list in +> source. + +--- + +## 5. Tests + +| Test | Assertion | +|---|---| +| Combined flag | Every Supplemental row has `combined == True` | +| Vocabulary isolation | Disaggregated category names rejected under `report_type = supplemental`, and vice versa | +| No cross-report mixing | A query for one report type never returns rows from another | +| Vintage round-trip | Snapshot → ingest → re-ingest is a no-op; a changed field emits one revision row | +| Coverage | Covered-market set derived from data matches the emitted artifact | + +Fixtures from real files trimmed to two or three markets, committed for offline runs. + +> **All five, in `tests/test_cit_supplemental.py` (20 tests), plus parse-level tests for +> the date-column rename and the assert-don't-infer path.** Fixtures are the real 2026 and +> 2012 archives trimmed to three markets and four weeks. Two years rather than one because +> the pair *is* the header rename. `tests/_gen_cit_fixtures.py` regenerates them and selects +> whole source lines rather than round-tripping through pandas, which would have dropped +> the zero padding on `"001602"` and the space padding on `"CBT "` — the byte-level quirks +> the parser is being tested against. + +--- + +## 6. Report back + +- The covered-market set per year, and any entries or exits over the history +- Whether the canonical schema absorbed the taxonomy cleanly or needed changes +- The Legacy-identity exception rate, and whether it is stable across history +- Confirmation that the first Supplemental fetch is vintage-captured +- Anything the data contradicted in this handoff, corrected in place + +**Do not do the analysis.** That is the companion handoff in `crowdmon`, and it should run against a merged, tested ingestion rather than alongside one. + +--- + +## 7. Report back (answers) + +**Covered-market set.** 12 markets 2006-2012, 13 from 2013. One entry, no exits: +Soybean Meal (026603) from 2013-01-08. **Six** renames without a code change, in two waves +(four NYBOT to ICE in 2007, the two wheats relabelled in 2013, including 001612 changing +exchange from KCBT to CBOT and keeping its code), which is why the coverage artifact keys +on `market_code` and not on name — and why it takes the name at the latest report date +rather than the lexicographic max, since `'I' < 'N'` makes max-of-strings report the +pre-2007 NYBOT name forever. Full table: measurements doc §3. + +**Schema.** Absorbed cleanly, no changes to the natural key, `VALUE_FIELDS` or +`ALL_COLUMNS`. One deliberate deviation on category spellings, §2 above. + +**Legacy-identity exception rate.** Summing the four canonical categories per side, the +identity is exact on 55.2% (long) and 55.4% (short) of 13,584 market-weeks, and the +residual is **never more than 2 contracts** against a tolerance of 4. Counting a +market-week as a breach if either side is off gives a **breach** rate of 67.7%. Two +different rates; keep the qualifier attached. + +Both are **stable across the whole history**: exact-rate per-year 51.6% to 59.5% (sd +2.1pp), breach-rate per-year 62.6% to 71.7% (sd 2.7pp), no trend and no break at 2013 or +2018. + +The cause is combined reporting, established by control: on the same 2026 weeks and the +same identity, Legacy **futures-only** is exact on 99.7% of rows and Legacy **combined** +shows the identical +/-1 pattern on 10%. Combined reports publish delta-weighted option +equivalents rounded to whole contracts, independently per category, so `n` categories admit +at most `n` contracts of error — which is the bound `rounding_tolerance` already returned. +`validate()` raises **zero** soft warnings across all 21 years. + +**Vintage capture.** Confirmed end to end through the shipped code path: 21 raw snapshots +captured, ingested to 54,336 observations across 13,584 market-weeks, zero revisions and +zero warnings on first ingest, and a full replay of all 21 snapshots wrote zero further +observations. `supplemental` is in the default `fetch` set from 2006 onward, so every +weekly release is captured from now on without further action. + +**Contradictions.** Four, all corrected in place above: the market count (§1), the +carve-out being three-way rather than two-way (§1), the combined flag being unreadable from +the file (§1), and the fetch pattern differing from the other three reports (§2). + +**Not done, deliberately.** No analysis: the index/non-index question is the companion +handoff's. No category-migration detection: it stays deferred until a quarter of snapshots +exists, earliest 2026-11-01. The 2018 reclassification is **not** recoverable and this work +does not make it so (§3). diff --git a/src/cotdata/config.py b/src/cotdata/config.py index 26147ca..742f1df 100644 --- a/src/cotdata/config.py +++ b/src/cotdata/config.py @@ -40,6 +40,10 @@ def cot_tff_dir() -> Path: return store_root() / "cot_tff" +def cot_supplemental_dir() -> Path: + return store_root() / "cot_supplemental" + + def manifest_path() -> Path: """The legacy aggregate manifest, written by BOTH halves. diff --git a/src/cotdata/cot.py b/src/cotdata/cot.py index 546a1a8..ec38d8b 100644 --- a/src/cotdata/cot.py +++ b/src/cotdata/cot.py @@ -16,14 +16,21 @@ def get_cot(name: str, report: str = "legacy") -> pd.DataFrame: """Read a symbol's weekly COT history with predecessor code stitching. name: the internal pipeline symbol, e.g. 'ES' or 'GC'. - report: 'legacy' (default), 'disagg' (commodity futures only), or 'tff' (financial futures only). + report: 'legacy' (default), 'disagg' (commodity futures only), 'tff' (financial + futures only), or 'supplemental' (13 agricultural markets, index traders split out). Returns: DataFrame indexed by date. The columns depend on the `report` requested: - legacy: Open_Interest_All, NonComm_Positions_Long_All, Comm_Positions_Long_All, etc. - disagg: Open_Interest_All, Prod_Merc_Positions_Long_All, M_Money_Positions_Long_All, etc. - tff: Open_Interest_All, Dealer_Positions_Long_All, Lev_Money_Positions_Long_All, etc. + - supplemental: Open_Interest_All, CIT_Positions_Long_All, + NComm_Positions_Long_All_NoCIT, Comm_Positions_Long_All_NoCIT, etc. If no data exists, returns an empty DataFrame. + + 'supplemental' is futures-and-options COMBINED and the other three are futures-only, + so its Open_Interest_All is a different quantity for the same market and week. Do not + difference or ratio across reports without accounting for that. """ sym = REGISTRY.get(name) if sym is None: # allow lookup by primary CFTC code, not just symbol @@ -33,9 +40,11 @@ def get_cot(name: str, report: str = "legacy") -> pd.DataFrame: "legacy": store.read_cot_legacy, "disagg": store.read_cot_disagg, "tff": store.read_cot_tff, + "supplemental": store.read_cot_supplemental, }.get(report) if not read_fn: - raise ValueError(f"Unknown report type: {report}. Expected 'legacy', 'disagg', or 'tff'") + raise ValueError(f"Unknown report type: {report}. Expected 'legacy', 'disagg', " + f"'tff', or 'supplemental'") if sym is None or not sym.cftc_code: return read_fn(name) diff --git a/src/cotdata/providers/cftc_cit.py b/src/cotdata/providers/cftc_cit.py new file mode 100644 index 0000000..49f5a4b --- /dev/null +++ b/src/cotdata/providers/cftc_cit.py @@ -0,0 +1,196 @@ +"""CFTC COT Supplemental (Commodity Index Trader) producer — cross-platform. + +Downloads dea_cit_txt_{year}.zip, parses losslessly, and writes per-code weekly +positioning tables to the store via store.write_cot_supplemental(). + +Three properties of this report that the other three producers do not have to think +about, all MEASURED against the real 2006-2026 archives (docs/analysis/2026-08-03-cit- +supplemental-measurements.md) rather than taken from the CFTC prose: + +1. **It is futures-and-options-COMBINED, and there is no futures-only variant.** Its + ``Open_Interest_All`` matches the Legacy *combined* file (annualof.xls) on 390/390 + 2026 market-weeks and the Legacy *futures-only* file on 0/390. Nothing in the file + itself says so — unlike Disaggregated and TFF it carries no ``FutOnly_or_Combined`` + column — which is exactly why ``canonicalize_supplemental`` asserts the flag instead + of inferring it. + +2. **The date column is named differently, and the name changed mid-history.** It is + ``As_of_Date_In_Form_YYYY-MM-DD`` from 2013 and ``As_of_Date_In_Form_MM/DD/YYYY`` + from 2006 to 2012. The rename is cosmetic: the VALUES are ISO ``YYYY-MM-DD`` in + every year including the ones whose header claims MM/DD/YYYY, so the 2013 change + fixed a mislabelled header rather than a format. Both are renamed to the repo's + ``Report_Date_as_MM_DD_YYYY`` here so downstream code sees one convention. + +3. **There is no 2006-2016 history bundle.** Disaggregated and TFF are served that way; + every Supplemental year 2006 through the current one is an individual zip. +""" +import datetime as dt +import zipfile +from email.utils import parsedate_to_datetime +from pathlib import Path + +import pandas as pd +import requests + +from .. import config, store +from ..registry import all_symbols, hist_code_scales + +URL_PREFIX = "https://www.cftc.gov/files/dea/history/dea_cit_txt_" +FIRST_YEAR = 2006 # Supplemental history start (January 2006) + +REPORT_DATE = "Report_Date_as_MM_DD_YYYY" +CONTRACT_CODE = "CFTC_Contract_Market_Code" + +# The two names CFTC has given the as-of date column. Order matters only in that both +# are accepted; a file carrying both would be a schema change worth failing on, which is +# why _parse_zip raises rather than picking one. +_DATE_COLS = ("As_of_Date_In_Form_YYYY-MM-DD", "As_of_Date_In_Form_MM/DD/YYYY") + + +def _cache_dir() -> Path: + d = config.store_root() / "_cache" / "cot_supplemental" + d.mkdir(parents=True, exist_ok=True) + return d + + +def _standardize_code(val) -> str: + """CFTC contract codes → 6-digit zero-padded string (matches CotSymbolCodeMap).""" + s = str(val).strip() + return s.zfill(6) if s.isdigit() else s + + +def _download_url(url: str, filename: str): + """Download a zip to the cache; skip if the server copy isn't newer.""" + zip_path = _cache_dir() / filename + try: + if zip_path.exists(): + head = requests.head(url, timeout=30) + server_mtime = head.headers.get("Last-Modified") + if server_mtime and zip_path.stat().st_mtime >= parsedate_to_datetime(server_mtime).timestamp(): + return zip_path # up to date + r = requests.get(url, timeout=180) + r.raise_for_status() + zip_path.write_bytes(r.content) + return zip_path + except Exception as e: # noqa: BLE001 + print(f" {filename} (supplemental): download failed — {e}") + return zip_path if zip_path.exists() else None + + +def _parse_zip(zip_path: Path) -> pd.DataFrame: + """Extract the .txt (CSV) from a year zip → full lossless DataFrame.""" + with zipfile.ZipFile(zip_path) as zf: + with zf.open(zf.namelist()[0]) as fh: + df = pd.read_csv(fh, low_memory=False) + + # Strip trailing whitespace from column names BEFORE accessing them. CFTC's own + # header row carries stray spaces (and two long-standing typos, "Postions" and + # "Spead", which are left alone — they are the real names). + df.columns = df.columns.str.strip() + + present = [c for c in _DATE_COLS if c in df.columns] + if len(present) != 1: + raise ValueError( + f"expected exactly one of {list(_DATE_COLS)} in the Supplemental file, found " + f"{present or 'none'}. CFTC changed the date column again: map it explicitly " + f"rather than letting the report date fall back to a positional index.") + df.rename(columns={present[0]: REPORT_DATE}, inplace=True) + + df[CONTRACT_CODE] = df[CONTRACT_CODE].apply(_standardize_code) + df[REPORT_DATE] = pd.to_datetime(df[REPORT_DATE], format="mixed").dt.tz_localize(None) + if not df.empty and df[REPORT_DATE].max() > pd.Timestamp.today().normalize() + pd.Timedelta(days=7): + raise ValueError(f"Date parsing sanity check failed: found future date {df[REPORT_DATE].max()}") + + # Parquet cannot serialize mixed-type object columns (CFTC_Market_Code and + # CFTC_Commodity_Code arrive space-padded, Contract_Units is free text). + for col in df.select_dtypes(include=["object"]).columns: + if col not in [CONTRACT_CODE, REPORT_DATE]: + df[col] = df[col].astype(str) + + return df + + +def coverage(frame: pd.DataFrame) -> pd.DataFrame: + """Which markets the Supplemental actually covered, per report year. + + Derived from the data, never from a hardcoded list. The covered set is NOT constant: + Soybean Meal (026603) entered in 2013, taking the count from 12 to 13, which is why + both counts are quoted in the wild. Returns one row per (report_year, market_code) + with the market name and the weeks observed. + + Aggregation is shared with ``vintage_ingest.derive_coverage`` rather than reimplemented. + That one reads the stored OBSERVATIONS and this one reads the downloaded FILES, so the + two answer slightly different questions and must not answer them differently: an + earlier pair of copies drifted on the name rule alone, reporting the pre-2007 NYBOT + name for four ICE markets. Their agreement is pinned by + tests/test_cit_supplemental.py::test_coverage_artifact_matches_the_set_derived_from_the_data. + """ + from ..vintage_ingest import _coverage_from + df = frame[[REPORT_DATE, CONTRACT_CODE, "Market_and_Exchange_Names"]].rename( + columns={REPORT_DATE: "report_date", CONTRACT_CODE: "market_code", + "Market_and_Exchange_Names": "market_name"}) + return _coverage_from(df, "supplemental") + + +def update(codes=None, first_year: int = FIRST_YEAR, last_year=None) -> dict: + """Download + parse the CFTC Supplemental COT; write full per-code history. + + codes: iterable of CFTC codes; default = all registry codes. Only 13 markets are + ever present, so most registry codes simply produce nothing — that is coverage, not + an error. Rebuilds the complete per-code table each run. Returns + ``{"kind", "ok", "wrote", "coverage"}``; ``ok`` is False only on a hard failure to + fetch the current year. + """ + code_to_sym = {} + for s in all_symbols(): + if s.cftc_code: + code_to_sym[s.cftc_code] = s.internal + for hc, _ in hist_code_scales(s.hist_codes): + code_to_sym[hc] = s.internal + + last_year = last_year or dt.date.today().year + want = set(codes) if codes else set(code_to_sym.keys()) + + frames = [] + latest_ok = True + for year in range(max(FIRST_YEAR, first_year), last_year + 1): + zp = _download_url(f"{URL_PREFIX}{year}.zip", f"dea_cit_txt_{year}.zip") + if not zp: + if year == last_year: + latest_ok = False # couldn't fetch current year — may have missed a release + continue + try: + frames.append(_parse_zip(zp)) + except Exception as e: # noqa: BLE001 + print(f" {year} (supplemental): parse failed — {e}") + if year == last_year: + latest_ok = False + + if not frames: + print("cftc_cit: no data parsed") + return {"kind": "cot_supplemental", "ok": False, "wrote": 0, "coverage": None} + + allrows = pd.concat(frames, ignore_index=True) + for col in allrows.select_dtypes(include=["object"]).columns: + if col not in [CONTRACT_CODE, REPORT_DATE]: + allrows[col] = allrows[col].astype(str) + + # Coverage over what this run actually WROTE, not over everything in the files. All + # 13 Supplemental codes resolve in the registry today so the two coincide, but a + # coverage report that silently described markets the store does not hold would be + # exactly the kind of "reports on 12 believing it has 13" error the artifact exists + # to prevent, pointed the other way. + cov = coverage(allrows[allrows[CONTRACT_CODE].isin(want)]) + + wrote = 0 + for code in sorted(want): + sub = allrows[allrows[CONTRACT_CODE] == code].copy() + if sub.empty: + continue + sub = sub.sort_values(REPORT_DATE).set_index(REPORT_DATE) + sym_name = code_to_sym.get(code) + file_name = f"{sym_name}_{code}" if sym_name else code + store.write_cot_supplemental(file_name, sub, source="cftc_cit") + wrote += 1 + print(f"{file_name}: {len(sub):5d} weeks (supplemental) -> store") + return {"kind": "cot_supplemental", "ok": latest_ok, "wrote": wrote, "coverage": cov} diff --git a/src/cotdata/status.py b/src/cotdata/status.py index 4e7e188..ce80f35 100644 --- a/src/cotdata/status.py +++ b/src/cotdata/status.py @@ -18,7 +18,8 @@ def status_path(): return config.store_root() / STATUS_FILENAME # Domains shown by --check, in report order. -_DOMAINS = ["prices", "metadata", "cot", "cot_legacy", "cot_disagg", "cot_tff"] +_DOMAINS = ["prices", "metadata", "cot", "cot_legacy", "cot_disagg", "cot_tff", + "cot_supplemental"] # An entry whose producer last touched it more than this many days behind its # domain's newest write probably failed while its peers succeeded (a partial run). _LAG_DAYS = 3 diff --git a/src/cotdata/store.py b/src/cotdata/store.py index 42d6932..35b034d 100644 --- a/src/cotdata/store.py +++ b/src/cotdata/store.py @@ -102,6 +102,20 @@ def read_cot_tff(name: str) -> pd.DataFrame: return pd.read_parquet(p) if p.exists() else pd.DataFrame() +# ── COT Supplemental (Commodity Index Traders) ──────────────────────────── +# The one report with no futures-only variant: every row is futures-and-options +# combined, so these tables are NOT comparable with cot_legacy/cot_disagg/cot_tff +# even for the same market and week. See providers/cftc_cit.py. +def write_cot_supplemental(name: str, df: pd.DataFrame, source: str) -> None: + _atomic_write_parquet(df, config.cot_supplemental_dir() / f"{name}.parquet") + _touch_manifest("cot_supplemental", name, df, source) + + +def read_cot_supplemental(name: str) -> pd.DataFrame: + p = config.cot_supplemental_dir() / f"{name}.parquet" + return pd.read_parquet(p) if p.exists() else pd.DataFrame() + + # ── Manifest ────────────────────────────────────────────────────────────── # ── The COT / price seam ────────────────────────────────────────────────── @@ -117,6 +131,7 @@ def read_cot_tff(name: str) -> pd.DataFrame: "cot_legacy": "cot", "cot_disagg": "cot", "cot_tff": "cot", + "cot_supplemental": "cot", } @@ -134,7 +149,7 @@ def half_for(kind: str) -> str: def _empty_manifest() -> dict: return {"schema_version": config.SCHEMA_VERSION, "metadata": {}, "prices": {}, - "cot_legacy": {}, "cot_disagg": {}, "cot_tff": {}} + "cot_legacy": {}, "cot_disagg": {}, "cot_tff": {}, "cot_supplemental": {}} def _read_json(p) -> dict: @@ -299,6 +314,7 @@ def _write_manifest(m: dict) -> None: "cot_legacy": config.cot_legacy_dir, "cot_disagg": config.cot_disagg_dir, "cot_tff": config.cot_tff_dir, + "cot_supplemental": config.cot_supplemental_dir, } diff --git a/src/cotdata/update.py b/src/cotdata/update.py index a25d9a8..62f507b 100644 --- a/src/cotdata/update.py +++ b/src/cotdata/update.py @@ -15,7 +15,7 @@ # become a second COT producer racing the first, which is the failure the split manifests # exist to contain. Read-only actions (--check, --reconcile) belong to both. _HALF_ACTIONS = { - "cot": ("cot_legacy", "cot_disagg", "cot_tff", "cot_all"), + "cot": ("cot_legacy", "cot_disagg", "cot_tff", "cot_supplemental", "cot_all"), "prices": ("prices", "metadata", "prices_yahoo", "ingest_databento", "build_databento"), } @@ -63,7 +63,12 @@ def main(argv=None, half=None) -> None: p.add_argument("--cot-legacy", action="store_true", help="Update CFTC COT Legacy (cross-platform).") p.add_argument("--cot-disagg", action="store_true", help="Update CFTC COT Disaggregated Futures-Only (cross-platform).") p.add_argument("--cot-tff", action="store_true", help="Update Traders in Financial Futures (TFF) COT (cross-platform).") - p.add_argument("--cot-all", action="store_true", help="Update all COT pipelines (Legacy, Disagg, TFF).") + p.add_argument("--cot-supplemental", action="store_true", + help="Update the CFTC Supplemental (Commodity Index Trader) COT " + "(cross-platform). 13 agricultural markets, and futures-and-options " + "COMBINED: its open interest is not comparable with the other three.") + p.add_argument("--cot-all", action="store_true", + help="Update all COT pipelines (Legacy, Disagg, TFF, Supplemental).") p.add_argument("--symbols", nargs="+", default=None, help="Internal symbols; default = all in registry.") p.add_argument("--full", action="store_true", help="Full rebuild of reconstructed volume (ignore the incremental " @@ -157,9 +162,11 @@ def main(argv=None, half=None) -> None: return if not (args.prices or args.metadata or args.prices_yahoo or args.ingest_databento - or args.build_databento or args.cot_legacy or args.cot_disagg or args.cot_tff or args.cot_all): + or args.build_databento or args.cot_legacy or args.cot_disagg or args.cot_tff + or args.cot_supplemental or args.cot_all): p.error("nothing to do — pass --check, --prices, --prices-yahoo, --ingest-databento, " - "--build-databento, --metadata, --cot-legacy, --cot-disagg, --cot-tff, or --cot-all") + "--build-databento, --metadata, --cot-legacy, --cot-disagg, --cot-tff, " + "--cot-supplemental, or --cot-all") if args.prices or args.metadata: from .providers import norgate @@ -231,6 +238,21 @@ def main(argv=None, half=None) -> None: if not (r or {}).get("ok", True): failed_kinds.append("cot_tff") + if args.cot_supplemental or args.cot_all: + from .providers import cftc_cit + r = cftc_cit.update() + kinds.append("cot_supplemental") + if not (r or {}).get("ok", True): + failed_kinds.append("cot_supplemental") + # Coverage is printed rather than merely written, because a market entering or + # leaving is the kind of change that is invisible in a per-market read and + # breaks every pooled statistic computed over the universe. + cov = (r or {}).get("coverage") + if cov is not None and not cov.empty: + per_year = cov.groupby("report_year")["market_code"].nunique() + print(f"cot_supplemental coverage: {int(per_year.min())}-{int(per_year.max())} " + f"markets per year over {len(per_year)} year(s).") + # Structured heartbeat for downstream tools: rebuild status.json from the now- # updated manifest. Pollers detect new data via newest_data[]. from . import status diff --git a/src/cotdata/vintage.py b/src/cotdata/vintage.py index 77a959e..29b3d3f 100644 --- a/src/cotdata/vintage.py +++ b/src/cotdata/vintage.py @@ -99,7 +99,7 @@ def manifest_path() -> Path: @dataclass(frozen=True) class Source: """One fetchable CFTC file. ``report_year`` is None for the current-week static.""" - report_type: str # legacy | disaggregated | tff + report_type: str # legacy | disaggregated | tff | supplemental source_kind: str # annual_zip | weekly_static ext: str # zip | txt url: str @@ -122,6 +122,12 @@ def _legacy_zip_url(year: int) -> str: # record eight permanent failure snapshots that could never succeed. _DISAGG_TFF_FIRST_YEAR = 2010 +# The Supplemental (Commodity Index Trader) report begins in January 2006, and UNLIKE +# disagg/TFF every year from 2006 onward is served as its own zip — verified live +# 2026-08-03, all 21 years returned 200. There is no dea_cit_txt_hist_2006_2016.zip +# (404), so there is no bundle special case to write here. +_SUPPLEMENTAL_FIRST_YEAR = 2006 + def annual_sources(year: int) -> list[Source]: out = [Source("legacy", "annual_zip", "zip", _legacy_zip_url(year), year)] @@ -130,6 +136,9 @@ def annual_sources(year: int) -> list[Source]: f"https://www.cftc.gov/files/dea/history/fut_disagg_txt_{year}.zip", year)) out.append(Source("tff", "annual_zip", "zip", f"https://www.cftc.gov/files/dea/history/fut_fin_txt_{year}.zip", year)) + if year >= _SUPPLEMENTAL_FIRST_YEAR: + out.append(Source("supplemental", "annual_zip", "zip", + f"https://www.cftc.gov/files/dea/history/dea_cit_txt_{year}.zip", year)) return out diff --git a/src/cotdata/vintage_cli.py b/src/cotdata/vintage_cli.py index 44768ed..b21abc7 100644 --- a/src/cotdata/vintage_cli.py +++ b/src/cotdata/vintage_cli.py @@ -70,7 +70,7 @@ def _snapshot_path(local_path: str): # column, and the canonicalisers accept either. def _canonicalisers(): from . import vintage_ingest - from .providers import cftc, cftc_disagg, cftc_tff + from .providers import cftc, cftc_cit, cftc_disagg, cftc_tff def legacy(path): wide = cftc._parse_zip(path).set_index(cftc.REPORT_DATE) @@ -80,6 +80,8 @@ def legacy(path): "legacy": legacy, "disaggregated": lambda p: vintage_ingest.canonicalize_disagg(cftc_disagg._parse_zip(p)), "tff": lambda p: vintage_ingest.canonicalize_tff(cftc_tff._parse_zip(p)), + "supplemental": lambda p: vintage_ingest.canonicalize_supplemental( + cftc_cit._parse_zip(p)), } @@ -270,6 +272,44 @@ def _cmd_asof(args) -> int: return 0 +def _cmd_coverage(args) -> int: + from . import vintage_ingest + + # Derive BEFORE writing. Writing first meant that pointing COTDATA_STORE at the wrong + # root truncated a perfectly good artifact to empty and then printed "nothing ingested + # yet" — destroying the thing it was asked to report on. + cov = vintage_ingest.derive_coverage(args.report_type) + if cov.empty: + print(f"coverage: no {args.report_type} observations ingested yet, so nothing to " + f"emit (the existing artifact, if any, is left alone). Run " + f"'cotdata-vintage fetch' then 'cotdata-vintage ingest' first.") + return 0 + path, cov = vintage_ingest.write_coverage(args.report_type) + print(f"coverage: {args.report_type} -> {path}") + per_year = cov.groupby("report_year")["market_code"].nunique() + lo, hi = int(per_year.min()), int(per_year.max()) + print(f" {len(cov)} (year, market) pairs across {len(per_year)} report year(s); " + f"{lo}-{hi} markets per year.") + changes = vintage_ingest.coverage_changes(cov) + if not changes: + print(" covered set is constant across the ingested history.") + else: + events = [c for c in changes if c["change"] != "gap"] + gaps = [c for c in changes if c["change"] == "gap"] + if events: + print(f" {len(events)} entry/exit event(s) — each one breaks any statistic " + f"pooled across the universe:") + for c in events: + print(f" {c['report_year']} {c['change'].upper():<5} " + f"{c['market_code']} {c['market_name']}") + for c in gaps: + print(f" {c['report_year']} GAP {c['market_name']}") + if args.detail: + print() + print(cov.to_string(index=False)) + return 0 + + def _cmd_zero_sum(args) -> int: """The schema check that was the first half of `flow`. @@ -362,6 +402,18 @@ def main(argv=None) -> int: a.add_argument("--market", default=None) a.set_defaults(func=_cmd_asof) + cvg = sub.add_parser("coverage", + help="Emit which markets a report actually covers, per year, " + "derived from the ingested observations.") + cvg.add_argument("--report-type", default="supplemental", dest="report_type", + choices=("legacy", "disaggregated", "tff", "supplemental"), + help="Default 'supplemental', the one report whose covered set has " + "changed (Soybean Meal entered in 2013, 12 markets -> 13).") + cvg.add_argument("--detail", action="store_true", + help="Print the full per-(year, market) table, not just entries " + "and exits.") + cvg.set_defaults(func=_cmd_coverage) + zs = sub.add_parser("zero-sum", help="Check the long/short/open-interest identity per market-week.") zs.add_argument("--market", required=True, help="CFTC market code, e.g. 088691.") diff --git a/src/cotdata/vintage_ingest.py b/src/cotdata/vintage_ingest.py index 47c5eb5..dec5eb1 100644 --- a/src/cotdata/vintage_ingest.py +++ b/src/cotdata/vintage_ingest.py @@ -50,12 +50,26 @@ "row_sha256", "is_tombstone"] # Controlled vocabulary per report type (validation §5). "nonreportable" is common -# to all three; the rest come from each report's reporting categories. +# to all four; the rest come from each report's reporting categories. +# +# The vocabulary is deliberately checked PER REPORT TYPE and never globally. Supplemental +# reuses "commercial" and "noncommercial", and those labels do NOT mean what they mean +# under Legacy: they are net of index traders, and the whole report is futures-and-options +# combined where Legacy here is futures-only. Nothing can silently merge the two, because +# both ``report_type`` and ``combined`` are in the natural key, but a consumer summing +# across report types would still be wrong, so keep the report type attached. +# +# Reusing the labels rather than minting "non_commercial"/"non_reportable" spellings is a +# deliberate choice against the source handoff: an alternative spelling would have made +# ``category == "nonreportable"`` silently miss every Supplemental row while leaving the +# genuinely confusable label, "commercial", identical anyway. Consistency loses nothing +# the natural key was not already carrying. CATEGORIES = { "legacy": {"commercial", "noncommercial", "nonreportable"}, "disaggregated": {"producer_merchant", "swap", "managed_money", "other_reportable", "nonreportable"}, "tff": {"dealer", "asset_manager", "leveraged", "other_reportable", "nonreportable"}, + "supplemental": {"commercial", "noncommercial", "index_trader", "nonreportable"}, } @@ -230,28 +244,112 @@ def canonicalize_legacy(wide: pd.DataFrame, *, combined: bool = False) -> pd.Dat "cr8_net_short": "Conc_Net_LE_8_TDR_Short_All", } +# Supplemental (Commodity Index Trader). Same tuple shape as the two above: +# (long, short, spread, traders_long, traders_short). +# +# Two things the other reports do not do. First, every position column except +# Non-Reportable carries a ``_NoCIT`` suffix, because the index-trader book has been +# carved OUT of it: this report's "commercial" is commercial-minus-index-traders, not +# Legacy's commercial. Second, the spelling is CFTC's own and carries two long-standing +# typos, "Postions" (missing the i) on the spread column and "NComm" rather than +# "NonComm" on the position columns while the Pct_/Traders_ columns use "NonComm". Both +# spellings are handled below rather than normalised, so the day CFTC fixes them is not +# the day this breaks: ``_resolve`` already tries the underscore variants, and the +# NComm/NonComm pair is listed explicitly. +# +# Non-Reportable has no ``_NoCIT`` variant because index traders are reportable by +# definition, so nothing is carved out of it. Verified: its values match the Legacy +# COMBINED file exactly on 390/390 2026 market-weeks. +_SUPPLEMENTAL_CATEGORIES = { + "noncommercial": ("NComm_Positions_Long_All_NoCIT", "NComm_Positions_Short_All_NoCIT", + "NComm_Postions_Spread_All_NoCIT", + "Traders_NonComm_Long_All_NoCIT", "Traders_NonComm_Short_All_NoCIT"), + "commercial": ("Comm_Positions_Long_All_NoCIT", "Comm_Positions_Short_All_NoCIT", + None, "Traders_Comm_Long_All_NoCIT", "Traders_Comm_Short_All_NoCIT"), + "index_trader": ("CIT_Positions_Long_All", "CIT_Positions_Short_All", + None, "Traders_CIT_Long_All", "Traders_CIT_Short_All"), + "nonreportable": ("NonRept_Positions_Long_All", "NonRept_Positions_Short_All", + None, None, None), +} + _REPORT_DATE_COL = "Report_Date_as_MM_DD_YYYY" +# CFTC's header spellings that have varied, or are outright typos, across the four +# reports. Each is a substitution tried on a missing column name before giving up. +# "Postions" and "Spead" are typos in CFTC's own header row; "NComm" vs "NonComm" is an +# inconsistency WITHIN the Supplemental file, whose position columns say NComm while its +# Pct_/Traders_ columns say NonComm. +_HEADER_VARIANTS = ( + ("__", "_"), ("_Positions", "__Positions"), + ("Positions", "Postions"), ("Postions", "Positions"), + ("Spread", "Spead"), ("Spead", "Spread"), + ("NonComm", "NComm"), ("NComm", "NonComm"), +) + +# How many substitutions may COMPOSE. One is not enough for the column that needs it +# most: ``NComm_Postions_Spread_All_NoCIT`` carries two defects at once, so the realistic +# CFTC cleanup (fix the typo and normalise the prefix in one pass, giving +# ``NonComm_Positions_Spread_All_NoCIT``) is exactly the case a single-substitution +# search cannot reach. Raising there would be safe but would still be an outage on a +# Friday release. Bounded rather than unbounded because the substitutions are not +# confluent — ``_Positions``/``__Positions`` and ``__``/``_`` invert each other — so an +# unbounded closure would loop. +_MAX_HEADER_SUBSTITUTIONS = 3 + + +def _header_candidates(name: str) -> list[str]: + """Every spelling of ``name`` reachable in at most _MAX_HEADER_SUBSTITUTIONS steps. + + Breadth-first so the fewest-edits spellings are tried first, which matters because + two candidates could in principle both exist in one file. Deterministic: the variant + tuple's order fixes the enumeration order. + + VERIFIED not to collide: across every column name the four canonicalisers ask for, + no candidate of one target is a candidate of another, and no candidate resolves to a + real column of a DIFFERENT field in the shipped 2026 Legacy, Disaggregated, TFF and + Supplemental headers. Pinned by test_header_variants_cannot_resolve_to_another_field. + """ + seen = {name} + frontier = [name] + out = [] + for _ in range(_MAX_HEADER_SUBSTITUTIONS): + nxt = [] + for cur in frontier: + for old, new in _HEADER_VARIANTS: + alt = cur.replace(old, new) + if alt not in seen: + seen.add(alt) + out.append(alt) + nxt.append(alt) + frontier = nxt + return out + + def _resolve(wide: pd.DataFrame, name: str | None) -> pd.Series | None: - """Look a column up, tolerating CFTC's single/double-underscore inconsistency. + """Look a column up, tolerating CFTC's known header spelling variants. Raises rather than silently producing a null column. A canonicaliser that quietly returns nulls for Managed Money because a header was renamed would write those nulls as OBSERVATIONS, and the next real value would then be recorded as a revision. A renamed column must fail loudly at ingest, not decay into fake revision history. + + A spelling variant is NOT a rename. The variants are a closed, enumerated list of + spellings CFTC has actually shipped for the same field; anything outside it still + raises, which is what keeps this a tolerance rather than a fuzzy match. """ if name is None: return None if name in wide.columns: return wide[name] - for alt in (name.replace("__", "_"), name.replace("_Positions", "__Positions")): + for alt in _header_candidates(name): if alt in wide.columns: return wide[alt] raise ValidationError( - f"column {name!r} not found (nor an underscore variant). CFTC changed a header: " - f"map it explicitly rather than letting this category ingest as nulls, which would " - f"be recorded as real observations and then as revisions when it came back.") + f"column {name!r} not found (nor a known spelling variant). CFTC changed a " + f"header: map it explicitly rather than letting this category ingest as nulls, " + f"which would be recorded as real observations and then as revisions when it " + f"came back.") def _combined_flag(wide: pd.DataFrame, override: bool | None) -> bool: @@ -348,6 +446,37 @@ def canonicalize_tff(wide: pd.DataFrame, *, combined: bool | None = None) -> pd. categories=_TFF_CATEGORIES, combined=combined) +def canonicalize_supplemental(wide: pd.DataFrame) -> pd.DataFrame: + """Supplemental (Commodity Index Trader) report to canonical long rows. + + ``combined`` is ASSERTED True rather than read, and takes no override. There is no + futures-only Supplemental to select between, so an override could only ever be wrong, + and every other path into this schema decides ``combined`` from the file. Making the + one report that cannot say so in its own bytes look like the others would put a + guessed value into the natural key. + + Measured rather than assumed (docs/analysis/2026-08-03-cit-supplemental-measurements.md): + this report's ``Open_Interest_All`` matches the Legacy futures-and-options-combined + file on 390 of 390 2026 market-weeks and the Legacy futures-only file on 0 of 390. + ``NonRept_Positions_Long_All`` matches combined on the same 390. + + The file carries no ``FutOnly_or_Combined`` column today. If CFTC ever adds one, this + raises unless it agrees, so the assertion cannot rot into a stale constant. + """ + if "FutOnly_or_Combined" in wide.columns: + claimed = {str(v).strip().lower() + for v in wide["FutOnly_or_Combined"].dropna().unique()} + if claimed - {"combined"}: + raise ValidationError( + f"the Supplemental file now carries FutOnly_or_Combined={sorted(claimed)}, " + f"but this report has only ever been futures-and-options combined. Either " + f"CFTC began publishing a futures-only variant, in which case the fetch " + f"list and this assertion both need updating, or the wrong file was " + f"passed to canonicalize_supplemental.") + return _canonicalize(wide, report_type="supplemental", + categories=_SUPPLEMENTAL_CATEGORIES, combined=True) + + # ── Validation (§5) ───────────────────────────────────────────────────────── class ValidationError(ValueError): pass @@ -480,11 +609,22 @@ def rounding_tolerance(n_categories: int) -> int: admits at most ``n`` contracts of error, which is the bound returned here. Measured against the 2026 files, the worst observed breach is well inside it: Legacy 1 - contract (bound 3), TFF 2 (bound 5), Disaggregated 0. Every single breach in all three + contract (bound 3), TFF 2 (bound 5), Disaggregated 0. Every breach in those three reports falls in one of those three Consolidated markets and nowhere else, so the tolerance costs no real sensitivity. Without it, 48 off-by-one warnings fire on the 2026 files alone, which is exactly the cry-wolf rate the previous version of this check was corrected for. + + **Supplemental breaks the "only in Consolidated markets" half of that**, and the + mechanism above is why the bound still holds. It is futures-and-options COMBINED, so + the published figures are delta-weighted option equivalents rounded to whole + contracts, and every category is rounded independently — the same n-addends argument, + applying to every market rather than to three. Measured over all 13,584 market-weeks + from 2006 to 2026, the derived category total misses open interest by at most 2 + contracts, against a bound of 4, and does so on 45% of rows rather than 0.4%. The + control is decisive: on the SAME 2026 weeks the Legacy futures-only file is exact on + 99.7% of rows while the Legacy combined file shows the identical +/-1 pattern on 10%. + Combined reporting is the cause, not the Supplemental's extra category. """ return max(1, int(n_categories)) @@ -694,6 +834,163 @@ def asof(t, *, report_date=None, market_code=None, report_type=None) -> pd.DataF return _latest_by_key(obs).reset_index(drop=True) +# ── Coverage (which markets a report could be scored on at all) ───────────── +# A report's covered market set is NOT a constant, and treating it as one is how a +# consumer silently reports on 12 markets while believing it has 13. The Supplemental is +# the live case: Soybean Meal (026603) entered in 2013 and the other twelve have run +# since 2006, which is why both counts circulate in the wild. Derived from the stored +# observations rather than from a list in source, so it cannot drift from the data. +COVERAGE_COLUMNS = ["report_type", "report_year", "market_code", "market_name", + "first_report_date", "last_report_date", "weeks"] + + +def _coverage_dir() -> Path: + return vintage.vintage_root() / "coverage" + + +def coverage_path(report_type: str) -> Path: + return _coverage_dir() / f"{report_type}.parquet" + + +def clean_market_name(v) -> str | None: + """One market name, whitespace-stripped, with every flavour of null mapped to None. + + ``astype(str)`` is NOT good enough and the difference is version-dependent: under + pandas 2 it turns a missing name into the literal ``"nan"``, and under pandas 3's str + dtype it leaves the float NaN in place. The first silently wins any lexicographic + comparison (lowercase sorts above every real CFTC name, which are uppercase); the + second raises ``TypeError`` comparing str to float. Both were reachable, because + ``market_name`` is descriptive and ``validate`` does not require it. + """ + try: + if pd.isna(v): + return None + except (TypeError, ValueError): + pass + s = str(v).strip() + return s or None + + +def latest_market_name(names, dates) -> str | None: + """The name carried at the LATEST report date, skipping nulls. + + Deliberately not ``max(names)``. Lexicographic max is not "current" and picks the + STALE name on four real markets: Cocoa, Cotton, Sugar and Coffee all renamed from + "... - NEW YORK BOARD OF TRADE" to "... - ICE FUTURES U.S.", and ``'I' < 'N'``, so + the max is the pre-rename name for every year after the change. Wheat happens to + come out right under either rule, which is how it went unnoticed. + """ + pairs = [(d, n) for d, n in zip(dates, map(clean_market_name, names)) if n is not None] + if not pairs: + return None + return max(pairs, key=lambda t: t[0])[1] + + +def _coverage_from(df: pd.DataFrame, report_type: str) -> pd.DataFrame: + """Shared aggregation behind both coverage entry points. + + ``df`` carries report_date / market_code / market_name. One implementation because + the producer derives coverage from the downloaded FILES and the vintage layer derives + it from the stored OBSERVATIONS, and two copies of the name rule drifted apart once + already. + """ + if df.empty: + return pd.DataFrame(columns=COVERAGE_COLUMNS) + df = df.copy() + df["report_date"] = pd.to_datetime(df["report_date"]) + df["report_year"] = df["report_date"].dt.year + rows = [] + for (year, code), g in df.groupby(["report_year", "market_code"], sort=True): + rows.append({ + "report_type": report_type, + "report_year": int(year), + "market_code": code, + "market_name": latest_market_name(g["market_name"], g["report_date"]), + "first_report_date": g["report_date"].min(), + "last_report_date": g["report_date"].max(), + "weeks": int(g["report_date"].nunique()), + }) + return pd.DataFrame(rows, columns=COVERAGE_COLUMNS) + + +def derive_coverage(report_type: str) -> pd.DataFrame: + """Which markets appear in the observation store for ``report_type``, per report year. + + Change-only writes do not lose coverage: a market-week's first sighting is always + written, so every (report_year, market_code) that was ever ingested has at least one + row here, and ``weeks`` counts DISTINCT report dates so later revisions do not + inflate it. + + This is the LIVE read. ``read_coverage`` returns whatever the artifact last recorded, + which is a snapshot with no freshness guarantee: prefer this one when correctness + matters and the artifact when you want what was published. + """ + obs = read_observations() + if obs.empty: + return pd.DataFrame(columns=COVERAGE_COLUMNS) + obs = obs[obs["report_type"] == report_type] + return _coverage_from(obs[["report_date", "market_code", "market_name"]], report_type) + + +def write_coverage(report_type: str) -> tuple[Path, pd.DataFrame]: + """Emit the coverage artifact for ``report_type``. Rewritten in full each time, + because it is a derived view of the observations and never a log. + + Takes the same single-writer lock as ingest. Not for the write, which is atomic, but + for the READ underneath it: ``read_observations`` concats one parquet per report year + while ``ingest_canonical`` rewrites them one at a time, so an unlocked derive can see + a run part-way through and emit a torn coverage set. Failing loudly is the repo's + answer everywhere else the two writers could interleave. + """ + with _WriteLock(vintage.vintage_root()): + cov = derive_coverage(report_type) + path = coverage_path(report_type) + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(".parquet.tmp") + cov.to_parquet(tmp, index=False) + tmp.replace(path) + return path, cov + + +def read_coverage(report_type: str) -> pd.DataFrame: + p = coverage_path(report_type) + return pd.read_parquet(p) if p.exists() else pd.DataFrame(columns=COVERAGE_COLUMNS) + + +def coverage_changes(cov: pd.DataFrame) -> list[dict]: + """Markets entering or leaving the covered set, year over year. + + The point of the artifact: an entry or exit is a break in every pooled statistic + computed across the universe, and it is invisible in a per-market read. + + **Only CONSECUTIVE years are compared**, and a gap emits a ``gap`` record instead. + Coverage is derived from what was INGESTED, not from what CFTC published, so a store + holding 2006 and 2026 and nothing between would otherwise report Soybean Meal as + entering in 2026 — an ingest artifact presented as a fact about the market. Refusing + to answer across a gap is the only honest option: the years that would settle it are + the ones that are missing. + """ + if cov.empty: + return [] + by_year = {int(y): dict(zip(g["market_code"], g["market_name"])) + for y, g in cov.groupby("report_year", sort=True)} + years = sorted(by_year) + out = [] + for prev, cur in zip(years, years[1:]): + if cur != prev + 1: + out.append({"report_year": cur, "change": "gap", "market_code": "", + "market_name": f"no observations for {prev + 1}-{cur - 1}, so " + f"entries and exits across the gap are unknowable"}) + continue + for code in sorted(set(by_year[cur]) - set(by_year[prev])): + out.append({"report_year": cur, "change": "enter", "market_code": code, + "market_name": by_year[cur][code]}) + for code in sorted(set(by_year[prev]) - set(by_year[cur])): + out.append({"report_year": cur, "change": "exit", "market_code": code, + "market_name": by_year[prev][code]}) + return out + + def read_revisions(detected_years=None) -> pd.DataFrame: d = _rev_dir() if not d.exists(): diff --git a/tests/_gen_cit_fixtures.py b/tests/_gen_cit_fixtures.py new file mode 100644 index 0000000..bf92f80 --- /dev/null +++ b/tests/_gen_cit_fixtures.py @@ -0,0 +1,67 @@ +"""Regenerate the committed Supplemental (CIT) fixtures from the real CFTC archives. + +Run manually, with network access, after a CFTC schema change: + + python tests/_gen_cit_fixtures.py /path/to/downloaded/zips + +The fixtures are REAL CFTC bytes trimmed to three markets and a few weeks, not synthetic +rows, because the properties the tests pin are properties of CFTC's own formatting: the +"Postions"/"Spead" header typos, the space-padded code columns, the ``_NoCIT`` suffixes, +and the 2013 rename of the as-of date column. A synthetic fixture would encode whatever +this module happened to believe about those, which is exactly what the tests exist to +check. + +Two years are kept for one reason: 2012 carries ``As_of_Date_In_Form_MM/DD/YYYY`` and +2026 carries ``As_of_Date_In_Form_YYYY-MM-DD``, so the pair is the header rename. +""" +import csv +import sys +import zipfile +from pathlib import Path + +import pandas as pd + +FIXTURES = Path(__file__).parent / "fixtures" / "cit" +# Cocoa and Wheat-SRW because they are the two markets the crowdmon fragility question +# actually turns on; Corn as a third, larger book. +KEEP_CODES = {"073732", "001602", "002602"} +KEEP_WEEKS = 4 + + +def trim(src_zip: Path, dest_zip: Path) -> None: + """Select whole SOURCE LINES and re-emit them verbatim. + + Deliberately not a pandas read/write round trip. That path re-quotes every field and + drops the zero padding on ``"001602"`` and the space padding on ``"CBT "``, which are + the exact byte-level quirks the parser is being tested against — the fixture would + then agree with the parser about a file CFTC does not publish. + """ + with zipfile.ZipFile(src_zip) as zf: + member = zf.namelist()[0] + text = zf.read(member).decode("utf-8", errors="strict") + lines = text.splitlines(keepends=True) + header = [h.strip().strip('"') for h in next(csv.reader([lines[0]]))] + code_i = header.index("CFTC_Contract_Market_Code") + date_i = next(i for i, h in enumerate(header) if h.startswith("As_of_Date_In_Form_") + and h != "As_of_Date_In_Form_YYMMDD") + + def fields(line): + return next(csv.reader([line])) + + parsed = [(fields(ln), ln) for ln in lines[1:] if ln.strip()] + dates = sorted({pd.Timestamp(f[date_i].strip()) for f, _ in parsed})[-KEEP_WEEKS:] + kept = [ln for f, ln in parsed + if f[code_i].strip().zfill(6) in KEEP_CODES + and pd.Timestamp(f[date_i].strip()) in dates] + + dest_zip.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(dest_zip, "w", zipfile.ZIP_DEFLATED) as zf: + zf.writestr(member, lines[0] + "".join(kept)) + print(f"{dest_zip.name}: {len(kept)} rows, {len(dates)} weeks, " + f"{len({fields(ln)[code_i].strip() for ln in kept})} markets, member={member}") + + +if __name__ == "__main__": + src = Path(sys.argv[1] if len(sys.argv) > 1 else ".") + for year in (2012, 2026): + trim(src / f"dea_cit_txt_{year}.zip", FIXTURES / f"dea_cit_txt_{year}.zip") diff --git a/tests/fixtures/cit/dea_cit_txt_2012.zip b/tests/fixtures/cit/dea_cit_txt_2012.zip new file mode 100644 index 0000000..4f0e7ca Binary files /dev/null and b/tests/fixtures/cit/dea_cit_txt_2012.zip differ diff --git a/tests/fixtures/cit/dea_cit_txt_2026.zip b/tests/fixtures/cit/dea_cit_txt_2026.zip new file mode 100644 index 0000000..abc45d7 Binary files /dev/null and b/tests/fixtures/cit/dea_cit_txt_2026.zip differ diff --git a/tests/test_cit_supplemental.py b/tests/test_cit_supplemental.py new file mode 100644 index 0000000..7644fa9 --- /dev/null +++ b/tests/test_cit_supplemental.py @@ -0,0 +1,497 @@ +"""CFTC Supplemental (Commodity Index Trader) ingestion. + +Offline. The fixtures under ``tests/fixtures/cit/`` are REAL CFTC bytes trimmed to three +markets and four weeks (see ``tests/_gen_cit_fixtures.py``), because most of what can go +wrong here is a property of CFTC's own formatting rather than of our schema: the +``NComm_Postions_Spread_All_NoCIT`` typo, the space-padded ``"CBT "`` code columns, the +``_NoCIT`` suffixes, and the 2013 rename of the as-of date column. Two years are kept so +the rename itself is under test. + +Handoff: docs/handoffs/2026-08-03-cit-supplemental.md §5. +""" +import datetime as dt +import zipfile +from pathlib import Path + +import pandas as pd +import pytest + +FIXTURES = Path(__file__).parent / "fixtures" / "cit" +CIT_2026 = FIXTURES / "dea_cit_txt_2026.zip" +CIT_2012 = FIXTURES / "dea_cit_txt_2012.zip" + + +@pytest.fixture() +def store_env(tmp_path, monkeypatch): + monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) + return tmp_path + + +def _canon(path=CIT_2026): + from cotdata.providers import cftc_cit + from cotdata.vintage_ingest import canonicalize_supplemental + return canonicalize_supplemental(cftc_cit._parse_zip(path)) + + +# ── Parse ─────────────────────────────────────────────────────────────────── +def test_parse_is_lossless_and_normalises_the_date_column(store_env): + """54 CFTC columns in, 54 out — same contract as the disagg provider. The as-of date + is renamed to the repo's Report_Date_as_MM_DD_YYYY, which is what makes the canonical + melt read a real date rather than falling back to the frame's positional index.""" + from cotdata.providers import cftc_cit + df = cftc_cit._parse_zip(CIT_2026) + assert len(df.columns) == 54 + assert cftc_cit.REPORT_DATE in df.columns + # the source date column is CONSUMED by the rename, not left beside its replacement + assert "As_of_Date_In_Form_YYYY-MM-DD" not in df.columns + # ...while the YYMMDD twin survives, because the parse is lossless + assert "As_of_Date_In_Form_YYMMDD" in df.columns + assert df[cftc_cit.CONTRACT_CODE].iloc[0] == "001602" # zero-padded, not 1602 + assert df[cftc_cit.REPORT_DATE].dt.year.eq(2026).all() + # CFTC's own header typos survive: they are the real column names. + assert "NComm_Postions_Spread_All_NoCIT" in df.columns + + +def test_both_spellings_of_the_as_of_date_column_parse(store_env): + """CFTC renamed As_of_Date_In_Form_MM/DD/YYYY to ...YYYY-MM-DD in 2013. The rename was + cosmetic — the VALUES were already ISO in 2012 — so both must land on the same + normalised column rather than one silently going missing.""" + from cotdata.providers import cftc_cit + old = cftc_cit._parse_zip(CIT_2012) + new = cftc_cit._parse_zip(CIT_2026) + assert cftc_cit.REPORT_DATE in old.columns and cftc_cit.REPORT_DATE in new.columns + assert old[cftc_cit.REPORT_DATE].dt.year.eq(2012).all() + assert str(old[cftc_cit.REPORT_DATE].dtype).startswith("datetime64") + + +def test_an_unrecognised_date_column_raises_rather_than_falling_back(store_env, tmp_path): + """Silently falling back to the frame index would date every row by position, which + reads as data rather than as a schema break.""" + from cotdata.providers import cftc_cit + with zipfile.ZipFile(CIT_2026) as zf: + member = zf.namelist()[0] + text = zf.read(member).decode() + broken = tmp_path / "broken.zip" + with zipfile.ZipFile(broken, "w") as zf: + zf.writestr(member, text.replace("As_of_Date_In_Form_YYYY-MM-DD", "Report_Week")) + with pytest.raises(ValueError, match="date column"): + cftc_cit._parse_zip(broken) + + +# ── The combined flag ─────────────────────────────────────────────────────── +def test_every_supplemental_row_is_combined(store_env): + """There is no futures-only Supplemental. Measured: its Open_Interest_All matches the + Legacy futures-and-options-combined file on 390/390 2026 market-weeks and the + futures-only file on 0/390.""" + c = _canon() + assert set(c["combined"]) == {True} + assert set(c["report_type"]) == {"supplemental"} + assert set(_canon(CIT_2012)["combined"]) == {True} + + +def test_combined_is_asserted_not_inferred(store_env): + """The file carries no FutOnly_or_Combined column, so _combined_flag's default would + silently produce combined=False — a guessed value in the natural key. If CFTC ever + adds the column saying otherwise, this must fail loudly instead of rotting.""" + from cotdata import vintage_ingest as vi + from cotdata.providers import cftc_cit + wide = cftc_cit._parse_zip(CIT_2026) + assert "FutOnly_or_Combined" not in wide.columns # the reason the assert exists + + wide = wide.copy() + wide["FutOnly_or_Combined"] = "FutOnly" + with pytest.raises(vi.ValidationError, match="only ever been futures-and-options"): + vi.canonicalize_supplemental(wide) + + +# ── Vocabulary ────────────────────────────────────────────────────────────── +def test_vocabulary_is_isolated_per_report_type(store_env): + """Supplemental's 'commercial' is not Disaggregated's producer_merchant, and its + categories must not validate under another report type in either direction.""" + from cotdata import vintage_ingest as vi + + c = _canon() + vi.validate(c) # valid as supplemental + + mislabelled = c.copy() + mislabelled["report_type"] = "disaggregated" + with pytest.raises(vi.ValidationError, match="outside vocabulary for 'disaggregated'"): + vi.validate(mislabelled) + + # ...and the reverse: a disaggregated category name under report_type=supplemental + intruder = c.copy() + intruder.loc[intruder.index[0], "category"] = "swap" + with pytest.raises(vi.ValidationError, match="outside vocabulary for 'supplemental'"): + vi.validate(intruder) + + +def test_index_trader_is_not_mapped_onto_a_disaggregated_name(store_env): + from cotdata import vintage_ingest as vi + assert vi.CATEGORIES["supplemental"] == { + "commercial", "noncommercial", "index_trader", "nonreportable"} + assert not (vi.CATEGORIES["supplemental"] & {"swap", "producer_merchant", + "managed_money", "other_reportable"}) + assert set(_canon()["category"]) == vi.CATEGORIES["supplemental"] + + +# ── Values ────────────────────────────────────────────────────────────────── +def test_categories_carry_the_no_cit_columns_not_the_legacy_ones(store_env): + """The whole point of the report: 'commercial' here is commercial NET of index + traders, so it must read Comm_Positions_Long_All_NoCIT. Reading a Legacy-named + column would double-count the index book.""" + from cotdata.providers import cftc_cit + wide = cftc_cit._parse_zip(CIT_2026) + c = _canon() + row = wide.iloc[0] + key = (pd.Timestamp(row[cftc_cit.REPORT_DATE]).normalize(), "001602") + sel = c[(c["report_date"] == key[0]) & (c["market_code"] == key[1])] + got = dict(zip(sel["category"], sel["long_contracts"])) + assert got["commercial"] == row["Comm_Positions_Long_All_NoCIT"] + assert got["noncommercial"] == row["NComm_Positions_Long_All_NoCIT"] + assert got["index_trader"] == row["CIT_Positions_Long_All"] + assert got["nonreportable"] == row["NonRept_Positions_Long_All"] + # only non-commercial has a spreading column in this report + spread = dict(zip(sel["category"], sel["spread_contracts"])) + assert spread["noncommercial"] == row["NComm_Postions_Spread_All_NoCIT"] + assert all(pd.isna(spread[k]) for k in ("commercial", "index_trader", "nonreportable")) + + +def test_concentration_ratios_are_null_because_the_report_omits_them(store_env): + c = _canon() + for f in ("cr4_net_long", "cr4_net_short", "cr8_net_long", "cr8_net_short"): + assert c[f].isna().all() + + +def test_open_interest_identity_stays_inside_the_rounding_tolerance(store_env): + """Combined reporting publishes delta-weighted option equivalents rounded to whole + contracts, independently per category, so the category total misses open interest by + a contract or two on roughly half the rows. That is rounding, not a breach, and must + not produce a warning.""" + from cotdata import vintage_ingest as vi + c = _canon() + assert vi.validate(c) == [] + g = c.groupby(["report_date", "market_code"], sort=False) + agg = g.agg(oi=("open_interest", "max"), L=("long_contracts", "sum"), + S=("short_contracts", "sum"), SP=("spread_contracts", "sum")) + tol = vi.rounding_tolerance(4) + assert ((agg.L + agg.SP - agg.oi).abs() <= tol).all() + assert ((agg.S + agg.SP - agg.oi).abs() <= tol).all() + + +# ── Vintage capture + ingest ──────────────────────────────────────────────── +def test_capture_includes_supplemental_from_2006(store_env): + """History starts in January 2006, and unlike disagg/TFF there is no 2006-2016 bundle + and no 404 window: every year from 2006 is served as its own zip.""" + from cotdata import vintage + assert [s.report_type for s in vintage.annual_sources(2005)] == ["legacy"] + assert "supplemental" in [s.report_type for s in vintage.annual_sources(2006)] + src = next(s for s in vintage.annual_sources(2026) if s.report_type == "supplemental") + assert src.url.endswith("dea_cit_txt_2026.zip") + assert src.source_kind == "annual_zip" and src.report_year == 2026 + + +def test_vintage_round_trip_is_a_no_op_and_a_change_emits_one_revision(store_env): + from cotdata import vintage_ingest as vi + c = _canon() + first = vi.ingest_canonical(c, snapshot_id="cit-1") + assert first["observations"] == 4 * 12 and first["revisions"] == 0 + assert first["warnings"] == [] + + again = vi.ingest_canonical(c, snapshot_id="cit-1-regenerated") + assert again["observations"] == 0 and again["revisions"] == 0 + + revised = c.copy() + i = revised.index[(revised["category"] == "index_trader").to_numpy().argmax()] + old = revised.loc[i, "long_contracts"] + revised.loc[i, "long_contracts"] = old + 500 + r = vi.ingest_canonical(revised, snapshot_id="cit-2") + assert r["observations"] == 1 and r["revisions"] == 1 + rev = vi.read_revisions().iloc[-1] + assert rev["report_type"] == "supplemental" and rev["category"] == "index_trader" + assert rev["field"] == "long_contracts" and rev["delta"] == 500.0 + + +def test_the_cli_ingests_a_supplemental_snapshot(store_env): + """Drives the real capture->ingest path: a recorded snapshot pointing at retained raw + bytes must find a canonicaliser and drain to parse_status=ok, not to 'skipped'.""" + from cotdata import vintage, vintage_cli + from cotdata import vintage_ingest as vi + + raw = store_env / "vintage" / "raw" / "annual_zip" / "2026" + raw.mkdir(parents=True, exist_ok=True) + (raw / "cit.zip").write_bytes(CIT_2026.read_bytes()) + vintage._write_manifest({"schema_version": 1, "snapshots": [{ + "snapshot_id": "cit-s1", "report_type": "supplemental", + "source_kind": "annual_zip", + "local_path": "vintage/raw/annual_zip/2026/cit.zip", "parse_status": "pending", + "retrieved_at": "2026-07-31T21:00:00Z", "report_year": 2026, + }]}) + assert vintage_cli.main(["ingest", "--pending"]) == 0 + assert vintage.read_snapshots()[0]["parse_status"] == "ok" + + obs = vi.read_observations() + assert len(obs) == 48 + assert set(obs["report_type"]) == {"supplemental"} + assert obs["observed_at"].eq(pd.Timestamp("2026-07-31T21:00:00")).all() + + +# ── No cross-report mixing ────────────────────────────────────────────────── +def test_a_query_for_one_report_type_never_returns_another(store_env): + """Supplemental and Legacy share the market code 001602, share report dates, and + share the category label 'commercial'. Only report_type separates them, and it is in + the natural key precisely so they cannot collapse into one series.""" + from cotdata import vintage_ingest as vi + + legacy_wide = pd.DataFrame({ + "Market_and_Exchange_Names": ["WHEAT-SRW - CHICAGO BOARD OF TRADE"], + "CFTC_Contract_Market_Code": ["001602"], + "Open_Interest_All": [463502], + "Comm_Positions_Long_All": [111111], "Comm_Positions_Short_All": [122222], + "NonComm_Positions_Long_All": [33333], "NonComm_Positions_Short_All": [44444], + "NonRept_Positions_Long_All": [5555], "NonRept_Positions_Short_All": [6666], + "Traders_Comm_Long_All": [50], "Traders_Comm_Short_All": [55], + "Traders_NonComm_Long_All": [60], "Traders_NonComm_Short_All": [45], + }, index=pd.DatetimeIndex(["2026-07-28"], name="Report_Date_as_MM_DD_YYYY")) + + vi.ingest_canonical(vi.canonicalize_legacy(legacy_wide), snapshot_id="lg-1") + vi.ingest_canonical(_canon(), snapshot_id="cit-1") + + both = vi.asof("2026-12-31", report_date="2026-07-28", market_code="001602") + assert set(both["report_type"]) == {"legacy", "supplemental"} + + only_sup = vi.asof("2026-12-31", report_date="2026-07-28", market_code="001602", + report_type="supplemental") + assert set(only_sup["report_type"]) == {"supplemental"} + assert set(only_sup["combined"]) == {True} + # the shared 'commercial' label carries different values and a different combined flag + lg = both[(both["report_type"] == "legacy") & (both["category"] == "commercial")] + sp = both[(both["report_type"] == "supplemental") & (both["category"] == "commercial")] + assert lg["long_contracts"].iloc[0] == 111111 + assert sp["long_contracts"].iloc[0] != 111111 + assert bool(lg["combined"].iloc[0]) is False and bool(sp["combined"].iloc[0]) is True + + +# ── Coverage ──────────────────────────────────────────────────────────────── +def test_coverage_artifact_matches_the_set_derived_from_the_data(store_env): + from cotdata import vintage_ingest as vi + from cotdata.providers import cftc_cit + + vi.ingest_canonical(_canon(CIT_2012), snapshot_id="cit-2012", + observed_at=dt.datetime(2013, 1, 4, tzinfo=dt.timezone.utc)) + vi.ingest_canonical(_canon(CIT_2026), snapshot_id="cit-2026", + observed_at=dt.datetime(2026, 7, 31, tzinfo=dt.timezone.utc)) + + path, cov = vi.write_coverage("supplemental") + assert path.exists() + pd.testing.assert_frame_equal(vi.read_coverage("supplemental"), cov) + + # The artifact must agree with the covered set read straight off the parsed files. + # Compared in FULL, names included: the producer path and the vintage path are two + # entry points onto one aggregation, and when they were two implementations they + # disagreed on exactly the name column. + from_files = pd.concat([cftc_cit._parse_zip(CIT_2012), cftc_cit._parse_zip(CIT_2026)], + ignore_index=True) + expected = cftc_cit.coverage(from_files) + pd.testing.assert_frame_equal(cov, expected) + # ...and the names are the real, stripped CFTC ones, not "nan" and not a stale rename + assert set(cov[cov["report_year"] == 2012]["market_name"]) == { + "WHEAT - CHICAGO BOARD OF TRADE", "CORN - CHICAGO BOARD OF TRADE", + "COCOA - ICE FUTURES U.S."} + + +def test_the_coverage_cli_does_not_truncate_a_good_artifact_on_an_empty_store(store_env): + """Writing before checking meant that pointing COTDATA_STORE at the wrong root + overwrote a real artifact with an empty one and then said 'nothing ingested yet'.""" + from cotdata import vintage_cli + from cotdata import vintage_ingest as vi + + vi.ingest_canonical(_canon(), snapshot_id="cit-1") + vintage_cli.main(["coverage", "--report-type", "supplemental"]) + before = vi.coverage_path("supplemental").read_bytes() + + # a report type with no observations must leave the file it is not about alone... + assert vintage_cli.main(["coverage", "--report-type", "tff"]) == 0 + assert not vi.coverage_path("tff").exists() + # ...and re-running against an emptied observation store must not destroy the artifact + for p in (vi.vintage.vintage_root() / "observations").glob("*/observations.parquet"): + p.unlink() + assert vintage_cli.main(["coverage", "--report-type", "supplemental"]) == 0 + assert vi.coverage_path("supplemental").read_bytes() == before + + +def test_coverage_names_a_market_from_its_latest_report_date(store_env): + """Lexicographic max is not 'current'. Cocoa, Cotton, Sugar and Coffee all renamed + from '... - NEW YORK BOARD OF TRADE' to '... - ICE FUTURES U.S.', and 'I' < 'N', so + a max-of-strings rule reports the pre-rename name forever after.""" + from cotdata import vintage_ingest as vi + dates = pd.to_datetime(["2007-01-02", "2007-06-05", "2007-12-04"]) + names = ["COCOA - NEW YORK BOARD OF TRADE", "COCOA - NEW YORK BOARD OF TRADE", + "COCOA - ICE FUTURES U.S."] + assert vi.latest_market_name(names, dates) == "COCOA - ICE FUTURES U.S." + assert sorted(set(names))[-1] != "COCOA - ICE FUTURES U.S." # the rule that was wrong + + cov = vi._coverage_from( + pd.DataFrame({"report_date": dates, "market_code": ["073732"] * 3, + "market_name": names}), "supplemental") + assert cov["market_name"].tolist() == ["COCOA - ICE FUTURES U.S."] + + +def test_a_missing_market_name_neither_crashes_nor_wins(store_env): + """market_name is descriptive, so validate() does not require it and a canonical frame + can carry nulls. Under pandas 2 astype(str) turns those into the literal 'nan', which + is lowercase and beats every real (uppercase) CFTC name; under pandas 3 it leaves a + float in place and the comparison raises TypeError. Neither may happen.""" + from cotdata import vintage_ingest as vi + dates = pd.to_datetime(["2026-01-06", "2026-01-13"]) + got = vi.latest_market_name(["CORN - CHICAGO BOARD OF TRADE", None], dates) + assert got == "CORN - CHICAGO BOARD OF TRADE" + assert vi.latest_market_name([None, float("nan")], dates) is None + assert vi.clean_market_name(" WHEAT ") == "WHEAT" + assert vi.clean_market_name(" ") is None + + cov = vi._coverage_from( + pd.DataFrame({"report_date": dates, "market_code": ["002602"] * 2, + "market_name": [None, None]}), "supplemental") + assert cov["market_name"].tolist() == [None] + + +def test_coverage_refuses_to_call_an_ingest_gap_a_market_event(store_env): + """Coverage is derived from what was INGESTED, not from what CFTC published. A store + holding 2006 and 2026 and nothing between must not report Soybean Meal as entering in + 2026 — that is an ingest artifact dressed up as a fact about the market.""" + from cotdata import vintage_ingest as vi + cov = pd.DataFrame([ + {"report_type": "supplemental", "report_year": 2006, "market_code": "001602", + "market_name": "WHEAT", "first_report_date": pd.Timestamp("2006-01-03"), + "last_report_date": pd.Timestamp("2006-12-26"), "weeks": 52}, + {"report_type": "supplemental", "report_year": 2026, "market_code": "001602", + "market_name": "WHEAT-SRW", "first_report_date": pd.Timestamp("2026-01-06"), + "last_report_date": pd.Timestamp("2026-07-28"), "weeks": 30}, + {"report_type": "supplemental", "report_year": 2026, "market_code": "026603", + "market_name": "SOYBEAN MEAL", "first_report_date": pd.Timestamp("2026-01-06"), + "last_report_date": pd.Timestamp("2026-07-28"), "weeks": 30}, + ]) + changes = vi.coverage_changes(cov) + assert [c["change"] for c in changes] == ["gap"] + assert "2007-2025" in changes[0]["market_name"] + assert not [c for c in changes if c["change"] == "enter"] + + +def test_coverage_reports_an_entry_rather_than_leaving_it_silent(store_env): + """Soybean Meal entered in 2013, taking the real universe from 12 markets to 13. An + entry is a break in anything pooled across the universe, so it has to be visible.""" + from cotdata import vintage_ingest as vi + + cov = pd.DataFrame([ + {"report_type": "supplemental", "report_year": 2012, "market_code": "001602", + "market_name": "WHEAT", "first_report_date": pd.Timestamp("2012-01-03"), + "last_report_date": pd.Timestamp("2012-12-31"), "weeks": 52}, + {"report_type": "supplemental", "report_year": 2013, "market_code": "001602", + "market_name": "WHEAT-SRW", "first_report_date": pd.Timestamp("2013-01-08"), + "last_report_date": pd.Timestamp("2013-12-31"), "weeks": 52}, + {"report_type": "supplemental", "report_year": 2013, "market_code": "026603", + "market_name": "SOYBEAN MEAL", "first_report_date": pd.Timestamp("2013-01-08"), + "last_report_date": pd.Timestamp("2013-12-31"), "weeks": 52}, + ]) + assert vi.coverage_changes(cov) == [ + {"report_year": 2013, "change": "enter", "market_code": "026603", + "market_name": "SOYBEAN MEAL"}] + + # a market LEAVING is reported too, and named from the year it was last seen + gone = cov[cov["report_year"] == 2012].copy() + gone = pd.concat([gone, cov[(cov["report_year"] == 2013) + & (cov["market_code"] == "026603")]]) + assert vi.coverage_changes(gone) == [ + {"report_year": 2013, "change": "enter", "market_code": "026603", + "market_name": "SOYBEAN MEAL"}, + {"report_year": 2013, "change": "exit", "market_code": "001602", + "market_name": "WHEAT"}] + + +def test_coverage_cli_writes_the_artifact(store_env, capsys): + from cotdata import vintage_cli + from cotdata import vintage_ingest as vi + vi.ingest_canonical(_canon(), snapshot_id="cit-1") + assert vintage_cli.main(["coverage", "--report-type", "supplemental"]) == 0 + assert vi.coverage_path("supplemental").exists() + out = capsys.readouterr().out + assert "13 markets per year" not in out # the fixture holds 3 + assert "3-3 markets per year" in out + assert "covered set is constant" in out + + +# ── Header spelling tolerance ─────────────────────────────────────────────── +# _resolve sits on the path that produces the permanent row_sha256, and the variant list +# is the only thing between a CFTC header correction and a failed Friday ingest. It +# shipped untested. +def test_a_corrected_cftc_header_still_resolves(store_env): + from cotdata import vintage_ingest as vi + want = "NComm_Postions_Spread_All_NoCIT" # two defects in one name + for corrected in ("NComm_Positions_Spread_All_NoCIT", # typo fixed + "NonComm_Postions_Spread_All_NoCIT", # prefix normalised + "NonComm_Positions_Spread_All_NoCIT"): # both, in one pass + got = vi._resolve(pd.DataFrame({corrected: [7]}), want) + assert got.iloc[0] == 7, corrected + + +def test_a_genuine_rename_still_raises(store_env): + """A spelling variant is not a rename. Anything outside the enumerated list must fail + loudly rather than ingest as nulls that later read as revisions.""" + from cotdata import vintage_ingest as vi + with pytest.raises(vi.ValidationError, match="known spelling variant"): + vi._resolve(pd.DataFrame({"IndexTraderLong": [1]}), "CIT_Positions_Long_All") + + +def test_header_variants_cannot_resolve_to_another_field(store_env): + """The regression the widening could have introduced: before it, a renamed column + raised. It must not now silently land on a DIFFERENT field's column.""" + import itertools + + from cotdata import vintage_ingest as vi + + targets = {} + for cats in (vi._SUPPLEMENTAL_CATEGORIES, vi._DISAGG_CATEGORIES, vi._TFF_CATEGORIES): + for cat, cols in cats.items(): + for i, col in enumerate(cols): + if col: + targets.setdefault(col, set()).add((cat, i)) + for field, col in vi._CONCENTRATION.items(): + targets.setdefault(col, set()).add(("conc", field)) + targets.setdefault("Open_Interest_All", set()).add(("oi", 0)) + + cands = {t: {t, *vi._header_candidates(t)} for t in targets} + for a, b in itertools.combinations(cands, 2): + if targets[a] == targets[b]: + continue # the same field, asked for by several reports + assert not (cands[a] & cands[b]), f"{a} and {b} share a candidate spelling" + + # and against a real file: no target may match two of its columns at once + from cotdata.providers import cftc_cit + header = set(cftc_cit._parse_zip(CIT_2026).columns) + for t, cl in cands.items(): + assert len(cl & header) <= 1, f"{t} matches several real columns" + + +# ── Store / producer plumbing ─────────────────────────────────────────────── +def test_supplemental_is_declared_on_the_cot_half(store_env): + from cotdata import store + assert store.half_for("cot_supplemental") == "cot" + assert "cot_supplemental" in store._empty_manifest() + + +def test_store_round_trip_and_get_cot(store_env): + from cotdata import cot, store + from cotdata.providers import cftc_cit + df = cftc_cit._parse_zip(CIT_2026).set_index(cftc_cit.REPORT_DATE) + store.write_cot_supplemental("ZW_001602", df, source="cftc_cit") + assert len(cot.get_cot("ZW", report="supplemental")) == len(df) + assert cot.get_cot("ZW", report="legacy").empty # different domain, not mixed + with pytest.raises(ValueError, match="Unknown report type"): + cot.get_cot("ZW", report="cit") + + +def test_cotdata_prices_refuses_the_supplemental_action(store_env): + """One host does one job: the price half must not become a second COT producer.""" + from cotdata import update + with pytest.raises(SystemExit): + update.main_prices(["--cot-supplemental"]) diff --git a/tests/test_manifest_seam.py b/tests/test_manifest_seam.py index 466e373..66a440d 100644 --- a/tests/test_manifest_seam.py +++ b/tests/test_manifest_seam.py @@ -187,7 +187,8 @@ def test_every_action_flag_is_assigned_to_a_half(): from cotdata import update assigned = set(update._HALF_ACTIONS["cot"]) | set(update._HALF_ACTIONS["prices"]) actions = {"prices", "metadata", "prices_yahoo", "ingest_databento", - "build_databento", "cot_legacy", "cot_disagg", "cot_tff", "cot_all"} + "build_databento", "cot_legacy", "cot_disagg", "cot_tff", + "cot_supplemental", "cot_all"} assert actions == assigned diff --git a/tests/test_vintage_capture.py b/tests/test_vintage_capture.py index cc1faf7..1418cdb 100644 --- a/tests/test_vintage_capture.py +++ b/tests/test_vintage_capture.py @@ -649,8 +649,15 @@ def http(url, *, etag=None, last_modified=None): def test_disagg_and_tff_start_in_2010_not_2006(): """cftc.gov serves 404 for fut_disagg_txt_2006..2009 and fut_fin_txt_2006..2009 (verified live), so 2006 made every `fetch --all` record eight permanent failure - snapshots that could never succeed.""" + snapshots that could never succeed. + + Supplemental is the counter-example, and the reason each report gets its own floor + rather than one shared constant: dea_cit_txt_2006..2009 all return 200 (verified live + 2026-08-03), so the same year that is a permanent failure for disagg/TFF is real data + for the Supplemental.""" from cotdata import vintage - assert [s.report_type for s in vintage.annual_sources(2009)] == ["legacy"] + assert {s.report_type for s in vintage.annual_sources(2009)} == { + "legacy", "supplemental"} + assert {s.report_type for s in vintage.annual_sources(2005)} == {"legacy"} assert {s.report_type for s in vintage.annual_sources(2010)} == { - "legacy", "disaggregated", "tff"} + "legacy", "disaggregated", "tff", "supplemental"}