feat(vintage): disagg/TFF canonicalisers, frozen-year tripwire, flow decomposition

README.md

@@ -337,11 +337,12 @@ This is **opt-in and purely additive**: if you never run it, the store behaves e
 before. Enabling it adds a `vintage/` subtree.
 
 ```bash
-cotdata-vintage fetch                      # capture current year + weekly static (daily)
+cotdata-vintage fetch                      # capture current + prior year + weekly static (daily)
 cotdata-vintage fetch --all                # every year 1986-present (see below)
 cotdata-vintage ingest --pending           # parse retained raw -> observations + revisions
 cotdata-vintage diff --since 2026-01-01    # field-level revisions, with revision depth
 cotdata-vintage asof --as-of 2026-07-24T18:00:00 --report-date 2026-07-21
+cotdata-vintage flow --market 088691       # weekly flow decomposition (see below)
 cotdata-schedule sync                      # CFTC Special Announcements
 cotdata-schedule published                 # true publication dates from retained weekly statics
 cotdata-schedule backfill                  # resolve release_date + its provenance
@@ -352,6 +353,11 @@ How it works:
 - **Immutable landing zone.** Every fetch is recorded (including 304s) and raw bytes are
   retained permanently under `vintage/raw/`, written atomically and never rewritten. A
   byte-identical regeneration is deduped — a changed download is not itself a revision.
+- **All three reports.** Legacy, Disaggregated and TFF canonicalise into one long schema.
+  Disagg and TFF also populate per-category spreading, per-category trader counts and
+  CR4/CR8 concentration, none of which the Legacy file carries, and they are where
+  **Managed Money** and **Leveraged Funds** live. A suppressed trader count (CFTC writes
+  `.`) canonicalises to null rather than to a string.
 - **Change-only observations.** A row is written only when its value hash differs from the
   latest for its natural key `(report_date, market_code, report_type, combined, category)`,
   so storage grows with actual revisions rather than with time.
@@ -369,13 +375,28 @@ How it works:
   (that file holds one week and is overwritten), so weeks predating capture fall back
   down the chain.
 
+- **Flow decomposition.** `cotdata-vintage flow` labels each week's ΔLong versus ΔShort as
+  `new_longs` / `short_covering` / `new_shorts` / `long_liquidation`, by dominant leg, with
+  no parameters to tune. A rally driven by short covering has a finite fuel supply and one
+  driven by fresh longs does not, and they are indistinguishable on a price chart. It also
+  emits `days_elapsed`, because COT was **fortnightly until 1992-10-13** and holidays shift
+  the rest, so a "weekly" change is not always weekly.
+
 **Run capture on the producer, not a replica**, and schedule it **daily**: nearly every
 request returns 304, so a daily run is close to free while catching holiday-shifted and
-backlog releases with no schedule logic. `--all` is a **restatement tripwire** rather than
-a backfill — closed years are byte-frozen, so a checksum change on one is the retroactive-
-restatement signature; monthly or quarterly is the right cadence, and it is cheap because
-almost everything 304s. Full design notes, including the measured CFTC caching behaviour,
-are in [docs/design/cot_vintage.md](docs/design/cot_vintage.md).
+backlog releases with no schedule logic.
+
+**The frozen-year tripwire.** CFTC regenerates a rolling two-year window (current plus the
+immediately-prior year) and nothing older. The prior year is therefore re-served every week
+but byte-identical, which is the one place a **content** check on closed data comes free.
+It is in the default fetch set for that reason: it costs one roughly 7 MB transfer per week
+and zero bytes on disk, and it is the only automated retroactive-restatement detector here.
+Anything other than `unchanged bytes (deduped)` on it raises an alert, which `ingest`
+re-raises as a non-zero exit and a `REVISIONS_<date>.txt` marker file. `--no-prior-year`
+turns it off. `--all` extends the same check to every year, but since nothing older is ever
+re-served it mostly confirms 304s; monthly or quarterly is the right cadence for that. Full
+design notes, including the measured CFTC caching behaviour, are in
+[docs/design/cot_vintage.md](docs/design/cot_vintage.md).
 
 > **Replica warning.** The vintage tree must not be written on a machine whose store is
 > mirrored (`robocopy /MIR`, `rsync --delete`) from a producer: the mirror deletes

docs/design/crowdmon_futures_cot_module.md

@@ -166,6 +166,21 @@ class CotSource(Protocol):
 
 Two implementations: `LocalCotData` wrapping the existing module, and `CftcApiCotData` as a fallback and for backfill. Validation on every load — schema conformance, `long + short + spread <= OI` sanity, non-null keys, category vocabulary check.
 
+> **Amended 2026-07-30, the OI sanity check as written is wrong.** `long + short + spread
+> <= OI` is not a bound on a single category row: the two sides of a market are counted
+> separately, so a category holding 80k long and 294k short in a 383k-OI market sums to
+> 374k with nothing wrong. Implemented literally it fired on 811 of 5,778 gold rows (14%).
+>
+> The invariant that does hold is **per side, summed across every category**, since every
+> long contract in the market belongs to exactly one category: `Σ long <= OI` and
+> `Σ short <= OI`. That is now what `vintage_ingest.validate` checks, and store-wide
+> warnings went from thousands to zero.
+>
+> Stronger still, and free: **`Σ long == Σ short`**, because futures are closed and
+> zero-sum. Verified on **149,412 / 149,412 weeks** across all 95 Legacy markets, 1986 to
+> 2026. Exposed as `vintage_flow.zero_sum_check` and worth running on every adapter load,
+> because a dropped, duplicated or misrouted category column breaks it on the first week.
+
 *Open items to resolve at implementation:* the existing module's output schema and index conventions, whether history is already backfilled and from what date, and whether revisions are currently retained or overwritten. Revision retention is the one that matters most — see §5.3.
 
 > **Resolved 2026-07-30** by the vintage build (branch `claude/cot-revision-snapshots-9b196f`,
@@ -184,6 +199,24 @@ Two implementations: `LocalCotData` wrapping the existing module, and `CftcApiCo
 > - **Only futures-only is fetched**, so `combined` is constant-`False` today: the column is
 >   present and correct but not yet discriminating, and half the reportable universe is absent
 >   until the combined files are added.
+> - **On LEGACY only, `spread_contracts` is null and `NonComm_Positions_Spread_All` is not
+>   captured at all.** It is absent from `providers/cftc.py`'s `TARGET_COLS`, so it never
+>   reaches the stored parquet, the canonical rows, or any vintage observation. Measured as
+>   the exact, equal gap between each side total and open interest: gold on 2026-07-21 had
+>   OI 383,368 against 351,385 on both sides, a gap of 31,983 (8% of OI). Net positioning
+>   is unaffected, since spreading is long and short in equal measure, but **anything
+>   denominated as a share of open interest (§5.2 step 2) has a denominator containing
+>   contracts its numerator cannot see.**
+>   **Disaggregated and TFF do not have this defect** (canonicalisers added 2026-07-30):
+>   both publish spreading per category, and the identity closes exactly, 7,847 of 7,847
+>   Disaggregated weeks with an `oi_gap` of zero. Since the categories this document
+>   actually uses are Disaggregated and TFF categories, §5.2 rung 2 is sound for them and
+>   compromised only for Legacy.
+> - **Trader counts and CR4/CR8 are populated for Disaggregated and TFF**, per category and
+>   per market respectively, which is what §6.2's breadth-depth quadrant needs. They are
+>   absent on Legacy. Trader counts carry `.` for a suppressed value and canonicalise to
+>   null: on the 2026 Disaggregated file, 3,578 of 7,847 Managed Money long counts are
+>   suppressed, so "null" is a routine and meaningful state, not a data error.
 > - **`vintage: int` is not how it was built.** The implementation is bitemporal
 >   (`observed_at` plus change-only rows), so a point-in-time read is "greatest
 >   `observed_at <= t` per natural key". An integer vintage ordinal can be derived from that
@@ -293,6 +326,37 @@ Weekly change in net position decomposes into four states, which have different
 
 A rally driven by short covering and a rally driven by new longs look identical on a chart and are entirely different setups. This decomposition is one line of code and is among the highest-value outputs in the system.
 
+> **Built 2026-07-30** as `cotdata/vintage_flow.py` (`cotdata-vintage flow`), directly on
+> the canonical schema with no prices and no contract master. It lives in `cotdata` as a
+> read-side function that writes no store domain and changes no producer/consumer contract;
+> the positioning ENGINE stays here in crowdmon, because that one needs prices, a contract
+> master and configured weights. Three things the table above does not say, all of which
+> only appear against real data:
+>
+> - **"~0" never happens.** Both legs always move. Resolved by DOMINANT LEG: whichever of
+>   |ΔLong|, |ΔShort| is larger names the state and its sign gives the direction, exact
+>   ties to the long leg. Parameter-free, so nothing to tune and nothing to overfit. An
+>   optional `min_frac_oi` dead zone adds a `quiet` state and defaults to 0.0, since any
+>   other value is a judgement of the same kind as §6.3's fragility weights.
+> - **Open interest corroborates the label, and disagrees often enough to matter.** Fresh
+>   positioning should create contracts and exits should destroy them; where it does not,
+>   the label describes a transfer between categories rather than new or closed risk.
+>   Emitted as `oi_corroborates` rather than folded into the state, because open interest
+>   here is the market total (that is what CFTC reports), so it checks a per-category label
+>   against a market-level quantity.
+> - **The weekly change is not always weekly. COT was FORTNIGHTLY until 1992-10-13.**
+>   Across the store 415,908 of 447,951 intervals are 7 days; the 14 and 15-day intervals
+>   are almost all pre-October-1992, and the rest are holiday shifts. `days_elapsed` is
+>   emitted as a column so a caller filters on it instead of discovering it in a result.
+>   This is §5.3's holiday hazard showing up in a second place: not just in the release
+>   date, but in the differencing interval.
+>
+> Worked example, gold non-commercial, week ending 2026-06-02: Δnet was **+21,760**, which
+> reads as heavy fresh buying. The decomposition says ΔShort **-16,368** against ΔLong
+> +5,392 with open interest **down 27,437**: short covering, a rally with a finite fuel
+> supply. That is the distinction this section exists to make, on the first real week
+> anyone looked at.
+
 ---
 
 ## 7. Cross-market engine