feat(vintage): disagg/TFF canonicalisers, frozen-year tripwire, flow decomposition

[mspinola]

Adversarial review, then six fixes (9a34750). Reviewed by an agent given the spec
and diff and no prior context, because every earlier review was by the author and
shared its blind spots. It found six confirmed defects, four in code written the same
day, all reproduced before fixing and each now covered by a regression test. It also
confirmed hash purity empirically rather than by reading, which is the property the whole
design rests on. Full write-up in docs/design/cot_vintage.md §9.

#	Defect	Consequence
1	A fetch failure poisoned the dedupe comparison	One network blip produced a false restatement alert and re-retained megabytes already on disk
2	An absolute local_path was re-rooted under the store	On any replica using COTDATA_VINTAGE_ROOT (where it is mandatory), one run drained the whole backlog to failed with no way to reset
3	A bare ingest replayed every snapshot with observed_at = now	Forged reversed revisions and inflated age_days, corrupting the primary artifact
4	schedule backfill bypassed the write lock	Dropped a concurrent ingest's rows, leaving a revision asserting a change to a value no longer in observations/
5	Every flat week was labelled long_liquidation	11.1% of real transitions; made it the modal state with a third of its bucket being weeks where nothing happened
6	validate() accepted duplicate natural keys and had no null-rate band	Append order decided stored values; a changed value format would coerce to nulls and be read as revisions

Not fixed, recorded as a real gap: the announced release-date tier is unreachable in
production and acceptance criterion 5 is unmet. Nothing writes source="announced"
rows, so the Oct-Dec 2025 backlog weeks resolve only to derived. The plumbing and
precedence are correct; what is missing is an extractor pulling a
(report_date, release_date) pair out of free-text announcement prose, and writing that
against no corpus of real announcements would be guessing. A release date resolved by a
guess is worse than a derived one, because it carries a flag claiming it was announced.

---

Four commits, in the order they were asked for. Everything below was measured against the real store, and several measurements changed the plan.

Commit	What
1d8e1c9	frozen-year tripwire, flow decomposition (§6.4), corrected OI validation
38d6f90	step 2 proposal (contract master + normalisation), no step-2 code
b1ade42	Disaggregated and TFF canonicalisers

Disaggregated and TFF canonicalisers

The step 2 proposal identified this as the real prerequisite rather than step 2 itself. Every engine in the module spec keys on Managed Money and Leveraged Funds, which exist only in these two reports, and ingest wired Legacy only. asof() now returns both point-in-time.

Verified against the real first production capture (2026-07-31 01:15Z), not fixtures:

Report	Canonical rows, 2026	Weeks	Re-ingest
Legacy	31,041	10,347	0 rows
Disaggregated	39,235	7,847	0 rows
TFF	12,500	2,500	0 rows

All three ingest in about 5 seconds, and re-ingesting writes nothing, which is the strongest available confirmation that the permanent row_sha256 artifact is stable across the new code path.

These two reports populate three fields Legacy never does: per-category spreading, per-category trader counts, and CR4/CR8 net concentration. So the zero-sum identity closes completely for Disaggregated:

Report	Exact	Within tolerance	oi_gap
Legacy	10,321 / 10,347	10,347 / 10,347	never zero (the uncaptured spreading)
Disaggregated	7,847 / 7,847	7,847 / 7,847	zero everywhere
TFF	2,463 / 2,500	2,500 / 2,500	zero on 98.6%

Disaggregated closing exactly, with a zero gap, is the confirming counterpart to the Legacy finding: that gap really is the missing spreading column and nothing else.

The rounding tolerance is derived, not fitted

Every off-by-one-or-two row in both Legacy and TFF falls in exactly three markets:

Code	Name	Legacy	TFF	Worst
13874+	S&P 500 Consolidated	12	17	2
20974+	NASDAQ-100 Consolidated	11	15	1
12460+	DJIA Consolidated	3	5	1

The + suffix is CFTC's own marker for a Consolidated contract, which aggregates several contract sizes onto a common unit and so involves a division. Summing n independently rounded category figures admits at most n contracts of error, which is what rounding_tolerance() returns. Without it, 48 off-by-one warnings fire on the 2026 files alone, the same cry-wolf rate the OI check was corrected for in the first commit.

Four implementation points

• CFTC's header has a typo, and it is load-bearing. Swap__Positions_Short_All and _Spread_All carry a double underscore; _Long_All does not. Both spellings resolve, so the day CFTC fixes it is not the day Swap Dealer positions ingest as nulls.
• An unresolvable column raises. Silent nulls would be written as real observations, and the next genuine value recorded as a revision that never happened, polluting the exact artifact this subsystem exists to produce.
• Trader counts arrive as strings, because CFTC writes . for a suppressed count (3,578 of 7,847 Managed Money long counts on the 2026 file, so "null" is routine, not an error). Every value field is coerced so a marker never reaches row_sha256 as a literal string.
• combined is now read from the file's FutOnly_or_Combined column, and a file mixing the two is refused. Constant-FutOnly today, so nothing changes, but adding the combined files becomes purely a fetch-list change.

canonicalize_legacy is deliberately not routed through the new shared helper. Its output feeds row_sha256 over rows already stored in production, and rewriting that path to save a little duplication risks registering every stored row as revised at once.

Fixed in passing: raw paths were unreadable off the producer

snapshots.json records local_path as written by the capturing machine, and the producer is Windows, so the real store carries vintage\raw\annual_zip\2026\....zip. On macOS or Linux that is one filename containing backslashes, so ingest on either replica failed with "no such file" on a file plainly sitting there. Normalised on read, so every snapshot already recorded on the producer stays readable.

---

Three things, all measured against the real store rather than assumed.

1. The frozen-year tripwire is now automatic

The rolling two-year regeneration window (measured in #78) means CFTC re-serves the prior year every week, byte-identical. That is the one place a content check on closed data is free, and it is the only automated detector for the failure mode this subsystem exists to guard against (July 2008: reports restated back to 2007-07-03).

The standing decision was "run --all monthly or quarterly". A tripwire that only fires when somebody remembers to run it by hand is an intention, not a detector, so the prior year moves into the default fetch set: one ~7 MB transfer per week (the other six days 304) and zero bytes on disk, since the content dedupes.

Regime	Files	Expected weekly outcome	Violation
churn	current year	new bytes	nothing, that is the data arriving
frozen_in_window	prior year	unchanged bytes (deduped)	new sha (restatement), or 304/failure (check went blind)
frozen_out_of_window	2+ years back	304 not-modified	a 200 with new bytes

Two alert shapes, deliberately triggered differently:

• Content changed on a frozen year: always alerts. A discrete, diffable event; two restatements in consecutive weeks are two things worth knowing. January says so explicitly, since year-end finalisation is the one benign way it fires.
• The detector went blind (frozen-in-window 304 or failure): alerts on the TRANSITION only. A standing condition, so level-triggering would fire every day forever, and an alert that never clears is one that gets ignored.

fetch still exits zero on purpose: the Windows wrapper aborts the run on a non-zero fetch, which would skip the ingest that turns a restatement into readable revision rows. ingest re-raises it and already writes the REVISIONS_<date>.txt marker.

2. Flow decomposition (module spec section 6.4)

vintage_flow.py plus cotdata-vintage flow. Weekly dLong vs dShort per market/category into new_longs / short_covering / new_shorts / long_liquidation. No prices, no contract master, so it is a clean smoke test of the canonical schema.

Classification is by dominant leg, because the spec's "~0" leg never happens in real data. Parameter-free by default, with an optional min_frac_oi dead zone that stays off, since any value for it is a judgement in the same class as the fragility weights.

Read-side only: writes no store domain, changes no producer/consumer contract. ADR-0007 narrows cotdata by instrument domain, not derived-vs-raw, and ADR-0008 settles that COT-derived-from-COT stays inside that boundary. The positioning engine (prices, contract master, configured weights) stays in crowdmon.

Worked example, gold non-commercial, week ending 2026-06-02: dNet +21,760, which reads as heavy fresh buying. Decomposed it is dShort -16,368 against dLong +5,392 with OI down 27,437. Short covering: a rally with a finite fuel supply. That distinction is the whole point of section 6.4, and it showed up on the first real week anyone looked at.

Swept over the whole store

95 Legacy markets, 1986 to 2026-07-21:

Measure	Result
Zero-sum identity (sum of long across categories == sum of short)	149,412 / 149,412 weeks balanced
Flow rows	447,951
Exceptions	0
validate() warnings	0 (after the fix below)

Finding: non-commercial SPREADING is not captured, and never was

NonComm_Positions_Spread_All is absent from providers/cftc.py's TARGET_COLS, so every side total falls short of OI by exactly the spreading. Gold on 2026-07-21: OI 383,368, both sides 351,385, gap 31,983 (8%). 64 of 95 markets have at least one week where the gap is zero, which is the confirming case.

Net positioning is unaffected (spreading is long and short in equal measure). Anything denominated as a share of open interest is not, and that is section 5.2 step 2, the next build step.

Not fixed here: adding the column changes providers/cftc.py output, which breaks the current/ byte-identical guarantee and the golden baseline that exists to protect it. It is its own change, not a drive-by.

Finding: COT was FORTNIGHTLY until 1992-10-13

415,908 of 447,951 intervals are 7 days; the 15-day (9,057) and 14-day (5,775) intervals are almost all pre-October-1992. days_elapsed is emitted as a column rather than assumed, so callers filter on it instead of discovering it in a result. This is section 5.3's holiday hazard showing up in a second place: not only in the release date, but in the differencing interval.

3. Corrected validate()'s open-interest warning

It compared one category's long + short + spread against total OI, which is not a bound: the two sides of a market are counted separately, so a category holding 80k long and 294k short in a 383k-OI market sums to 374k with nothing wrong. Measured, it fired on 811 of 5,778 gold rows (14%), and a soft warning at that rate is one nobody reads.

Replaced with the invariant that does hold, per side summed across categories. Store-wide warnings went from thousands to zero. The module spec asserted the wrong form too, and is amended in the same pass per the working agreement on specs.

Verification

• 226 tests pass (16 new), ruff check src tests scripts clean
• Whole-store sweep above, run live against ~/code/cotdata_store
• cotdata-vintage flow --market 088691 --source current end to end

Generated with Claude Code