[mspinola]

Description

Adds the CFTC Supplemental (Commodity Index Trader) report as a fourth report_type alongside legacy, disaggregated and tff: producer action, store domain, consumer reader, and vintage capture + canonicalisation from 2006 onward.

Executes docs/handoffs/2026-08-03-cit-supplemental.md, which is closed in this PR with every correction inline. Unblocks the companion handoff in crowdmon, which needs the index/non-index split to fix a Swap Dealer fragility weight that measurement showed doing opposite work on cocoa and gold.

Scope recorded in ADR-0002: the Supplemental is CFTC positioning, so it sits inside ADR-0007's narrowed boundary on the same argument that admitted vintage provenance. No change is proposed to ADR-0007 or ADR-0008 — this applies a rule they already state.

Purely additive. Nothing joins the natural key, VALUE_FIELDS or ALL_COLUMNS, so no stored row_sha256 moves and no existing observation is touched. A store that never runs the new action is unchanged.

Four things the data contradicted

All measured against the real 2006-2026 archives (21 files, 13,584 market-weeks). Full detail in docs/analysis/2026-08-03-cit-supplemental-measurements.md.

• Coverage is 12 markets, then 13 from 2013, when Soybean Meal entered. Both circulating counts are right for part of the history. Six markets were renamed without changing code (four NYBOT to ICE in 2007, the two wheats in 2013), which is why coverage keys on market_code.
• It is futures-and-options combined and the file cannot say so — no FutOnly_or_Combined column, so the existing _combined_flag would have defaulted it to False and put a guessed value in the natural key. Established by matching open interest against both Legacy series: 390/390 against combined, 0/390 against futures-only. Now asserted, with no override.
• Index traders are carved out of three buckets, not the two CFTC's prose names. The third is non-commercial spreading: ~0% of the index book in 2006, ~9% now.
• The fetch pattern differs from the other three. No 2006-2016 history bundle (404), and every year from 2006 returns 200 where disagg/TFF 404 before 2010, so annual_sources needed its own floor.

The identity exception rate, and why it is not alarming

Exact on ~55% of market-weeks, residual never above 2 contracts against a tolerance of 4, and flat across 21 years (per-year 51.6-59.5%, sd 2.1pp).

That is combined-report rounding, established by control rather than asserted: on the same 2026 weeks, Legacy futures-only is exact on 99.7% of rows while Legacy combined shows the identical +/-1 pattern. A combined report publishes delta-weighted option equivalents rounded to whole contracts, independently per category — the same n-addends mechanism rounding_tolerance was already derived from. validate() raises zero warnings across the full history.

Worth your eyes

The one deviation from written instructions. The brief asked for categories non_commercial / index_traders / non_reportable; this ships commercial / noncommercial / index_trader / nonreportable. A new spelling would make category == "nonreportable" silently miss every Supplemental row, while commercial — the genuinely confusable label — is identical either way, so the distinguish-by-spelling argument fails on exactly the category that needed it. report_type does the disambiguating, as the brief itself specifies. Reasoning in the closed handoff §2.

A cold-context review ran before this PR and found three real defects, all in the new coverage code (second commit): a lexicographic-max name rule that reported the stale pre-2007 NYBOT name on four live markets, a null-handling path that produced the literal "nan" on pandas 2 and a TypeError on pandas 3, and coverage_changes reporting an ingest gap as a market entry. It also found that _resolve — which sits on the permanent-hash path — had shipped untested.

Limits, stated rather than implied

• The 2018 reclassification is not recoverable and this does not make it so. CFTC serves current state only, so the 2018 archive we hold today is already the restatement. The vintage store detects the next one. Category-migration detection stays deferred; earliest sensible date 2026-11-01.
• Index Traders is a Legacy-taxonomy carve-out and does not nest inside Disaggregated's Swap Dealer. The two cannot be differenced to isolate levered swap flow.
• No analysis here, deliberately. That is the companion handoff's job, against a merged and tested ingestion.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have run tests (e.g. pytest) and they pass locally
• Any dependent changes have been merged and published in downstream modules

Verified locally: 289 tests pass on both pandas 2 and pandas 3 (Python 3.11 only — the 3.10/3.12/3.13/3.14 legs are what this PR's CI is for), ruff check src tests scripts clean, and a full green run against an empty store with no network. Separately reproduced end to end against the live CFTC archives: 21 raw snapshots captured, 54,336 observations, zero revisions, zero warnings, and a full 21-snapshot replay wrote nothing.

🤖 Generated with Claude Code