Inherit the durable data facts from crowdmon

[mspinola]

crowdmon is being deprecated. Its hypothesis is unproven and out of testable evidence, but several measurements made along the way are durable facts about data this package owns, and they would be orphaned by that repo going quiet rather than by being wrong.

Docs only. No code, no behaviour change. 289 tests pass.

cross-report-comparability.md

States the rule that has now been broken, measured and recorded three separate times, in three repos, against three different pairs of reports:

Two categories from two different reports may only be compared when both partition open interest the same way and count spreading the same way.

The Legacy/TFF case is new here. Over 6,279 overlapping market-weeks the two reports agree on exactly two quantities, open_interest and nonreportable, both at 100.0000%. Above the reportable line the obvious mapping fails ~85% of the time, for two compounding reasons neither of which any correction recovers: spreading is counted differently (Legacy breaks it out for non-commercial only, TFF for every category), and the buckets hold different traders. The Canadian dollar worked example closes to the contract.

The second half is a live trap in our own API, confirmed still live at harvest time: canonicalize_legacy sets spread_contracts to NA on every row, so summing that column returns 0, which prints as a measurement of zero spreading and is not one. The identity long + spread == open_interest closes on 99.984% of TFF market-weeks and 19.857% of Legacy ones. That 19.857% is the spreading convention showing through a column we populate with nulls.

reading-the-store.md

Four more, plus a composed tier fact:

• 76% of the 279-market Disaggregated universe is ICE Energy Div and Nodal power and gas basis, so a cross-market statistic over the full universe is mostly about ERCOT and PJM.
• A coverage ratio whose denominator nobody chose is not a measurement. Reported as a shortfall and withdrawn twice, in two packages. Ported as the rule plus the warning that the count itself moves (25, then 45, then 47), rather than as any one number.
• The fuller-sounding volume parameter is the narrower series. reconstructed is exactly two expiries; front is whole-market, established by open interest matching the CFTC exactly on 25 of 26 markets at a median ratio of 1.000.
• A hole in a code's series is two different things and only one is a migration. RTY's CME code has an 8.9-year internal gap that the ICE code fills almost exactly; oats has a 294-day interval because a thin market fell out of the report and nothing is missing. The merge across sibling codes must precede any differencing, because the other order fails silently.
• A databento-backed store cannot produce propadj at all. propadj is derived on read, Norgate is the only vendor supplying all tiers and is Windows-only by mechanism rather than licence, and databento owes only backadj per ADR-0007. A live constraint on step 2, and a tier fact rather than an OS fact.

Method

Restated, never moved. crowdmon's amendments are point-in-time records under that repo's doc lifecycle; rewriting them would destroy the record of when each thing was learned. Every section cites its source section and reproducer.

Both documents were read in full before restating, not summarised from headers. Facts already fixed upstream were checked and excluded rather than re-filed: A9's docstring correction is live in prices.py, A8 landed in ff2b755, B29 in #93.

Nothing here points at a crowdmon API. Where the source argument leaned on ContractMaster.coverage(), that is noted explicitly and the fact restated on cotdata's own providers and ADR-0007.

The full map of what was ported, what was already resolved, and what was parked with the hypothesis is crowdmon/docs/HARVEST.md (crowdmon PR, filed alongside this one).

Plain-language bottom line

A neighbouring project is being shelved. Along the way it measured a number of things about the COT data and the price store that are true regardless of whether its own idea worked, and a few of them are traps that would cost the next person real time. This moves those into the package that owns them, in full, with citations back to where they were established. Nothing about how cotdata behaves changes.

🤖 Generated with Claude Code