From 1d8e1c95a093aea2abb864ccff1bcd20dc322289 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 22:36:20 -0400 Subject: [PATCH 1/7] feat(vintage): frozen-year tripwire, flow decomposition, corrected OI validation Three things, all measured against the real store rather than assumed. 1. THE FROZEN-YEAR TRIPWIRE IS NOW AUTOMATIC The rolling two-year regeneration window (measured in #78) means CFTC re-serves the PRIOR year every week, byte-identical. That is the one place a content check on closed data is free, and it is the only automated detector for the failure mode this subsystem exists to guard against (July 2008: reports restated back to 2007-07-03). The standing decision was "run --all monthly or quarterly". A tripwire that only fires when somebody remembers to run it by hand is an intention, not a detector, so the prior year moves into the DEFAULT fetch set: one ~7 MB transfer per week (the other six days 304) and zero bytes on disk, since the content dedupes. Three regimes now carry different expectations, recorded on every snapshot as expectation / outcome / tripwire_alert: churn current year, new bytes are the data arriving frozen_in_window prior year, expect exactly "unchanged bytes (deduped)" frozen_out_of_window older, expect 304; content is never verified there Two alert shapes, deliberately triggered differently. Content changed on a frozen year always alerts: it is a discrete, diffable event. The detector going blind (a frozen-in-window 304 or fetch failure) alerts on the TRANSITION only, because it is a standing condition and a level-triggered alert would fire daily forever. fetch still exits zero on purpose: the Windows wrapper aborts the run on a non-zero fetch, which would skip the ingest that turns a restatement into readable revision rows. ingest re-raises it and already writes the marker file. 2. FLOW DECOMPOSITION (module spec section 6.4) vintage_flow.py plus `cotdata-vintage flow`. Weekly dLong vs dShort per market/category into new_longs / short_covering / new_shorts / long_liquidation. No prices, no contract master, so it is a clean smoke test of the canonical schema. Classification is by DOMINANT leg, because the spec's "~0" leg never happens in real data; parameter-free by default, with an optional min_frac_oi dead zone that stays off since any value for it is a judgement. Read-side only: writes no store domain, changes no producer/consumer contract. ADR-0007 narrows cotdata by instrument domain, not derived-vs-raw, and ADR-0008 settles that COT-derived-from-COT stays inside that boundary. The positioning engine (prices, contract master, configured weights) stays in crowdmon. Swept over all 95 Legacy markets, 1986 to 2026: 149,412 of 149,412 weeks satisfy the zero-sum identity (sum of long across categories == sum of short), 447,951 flow rows, zero exceptions. Two findings recorded in the docs: - NonComm_Positions_Spread_All is not captured and never was, so every side total falls short of OI by exactly the spreading (gold: 31,983 of 383,368, 8%). Net positioning is unaffected; anything denominated as a SHARE of open interest is not. Not fixed here: it would change providers/cftc.py output and break the current/ byte-identical guarantee, so it is its own change. - COT was FORTNIGHTLY until 1992-10-13. days_elapsed is emitted as a column instead of assuming seven, so callers filter rather than discover it later. 3. CORRECTED validate()'s OPEN-INTEREST WARNING It compared one category's long+short+spread against total OI, which is not a bound: the two sides are counted separately. It fired on 811 of 5,778 gold rows (14%), pure noise. Replaced with the invariant that holds, per side summed across categories. Store-wide warnings went from thousands to zero. The module spec asserted the wrong form too, and is amended in the same pass per the working agreement on specs. Co-Authored-By: Claude Opus 5 --- README.md | 28 ++- docs/design/cot_vintage.md | 136 +++++++++++++ docs/design/crowdmon_futures_cot_module.md | 55 +++++ docs/examples/windows/run-vintage.cmd | 11 +- src/cotdata/vintage.py | 191 +++++++++++++---- src/cotdata/vintage_cli.py | 96 ++++++++- src/cotdata/vintage_flow.py | 226 +++++++++++++++++++++ src/cotdata/vintage_ingest.py | 41 +++- tests/test_vintage_capture.py | 142 +++++++++++++ tests/test_vintage_flow.py | 124 +++++++++++ 10 files changed, 993 insertions(+), 57 deletions(-) create mode 100644 src/cotdata/vintage_flow.py create mode 100644 tests/test_vintage_flow.py diff --git a/README.md b/README.md index e95e7a4..ca8d2cd 100644 --- a/README.md +++ b/README.md @@ -337,11 +337,12 @@ This is **opt-in and purely additive**: if you never run it, the store behaves e before. Enabling it adds a `vintage/` subtree. ```bash -cotdata-vintage fetch # capture current year + weekly static (daily) +cotdata-vintage fetch # capture current + prior year + weekly static (daily) cotdata-vintage fetch --all # every year 1986-present (see below) cotdata-vintage ingest --pending # parse retained raw -> observations + revisions cotdata-vintage diff --since 2026-01-01 # field-level revisions, with revision depth cotdata-vintage asof --as-of 2026-07-24T18:00:00 --report-date 2026-07-21 +cotdata-vintage flow --market 088691 # weekly flow decomposition (see below) cotdata-schedule sync # CFTC Special Announcements cotdata-schedule published # true publication dates from retained weekly statics cotdata-schedule backfill # resolve release_date + its provenance @@ -369,13 +370,28 @@ How it works: (that file holds one week and is overwritten), so weeks predating capture fall back down the chain. +- **Flow decomposition.** `cotdata-vintage flow` labels each week's ΔLong versus ΔShort as + `new_longs` / `short_covering` / `new_shorts` / `long_liquidation`, by dominant leg, with + no parameters to tune. A rally driven by short covering has a finite fuel supply and one + driven by fresh longs does not, and they are indistinguishable on a price chart. It also + emits `days_elapsed`, because COT was **fortnightly until 1992-10-13** and holidays shift + the rest, so a "weekly" change is not always weekly. + **Run capture on the producer, not a replica**, and schedule it **daily**: nearly every request returns 304, so a daily run is close to free while catching holiday-shifted and -backlog releases with no schedule logic. `--all` is a **restatement tripwire** rather than -a backfill — closed years are byte-frozen, so a checksum change on one is the retroactive- -restatement signature; monthly or quarterly is the right cadence, and it is cheap because -almost everything 304s. Full design notes, including the measured CFTC caching behaviour, -are in [docs/design/cot_vintage.md](docs/design/cot_vintage.md). +backlog releases with no schedule logic. + +**The frozen-year tripwire.** CFTC regenerates a rolling two-year window (current plus the +immediately-prior year) and nothing older. The prior year is therefore re-served every week +but byte-identical, which is the one place a **content** check on closed data comes free. +It is in the default fetch set for that reason: it costs one roughly 7 MB transfer per week +and zero bytes on disk, and it is the only automated retroactive-restatement detector here. +Anything other than `unchanged bytes (deduped)` on it raises an alert, which `ingest` +re-raises as a non-zero exit and a `REVISIONS_.txt` marker file. `--no-prior-year` +turns it off. `--all` extends the same check to every year, but since nothing older is ever +re-served it mostly confirms 304s; monthly or quarterly is the right cadence for that. Full +design notes, including the measured CFTC caching behaviour, are in +[docs/design/cot_vintage.md](docs/design/cot_vintage.md). > **Replica warning.** The vintage tree must not be written on a machine whose store is > mirrored (`robocopy /MIR`, `rsync --delete`) from a producer: the mirror deletes diff --git a/docs/design/cot_vintage.md b/docs/design/cot_vintage.md index af5d81c..d61d43a 100644 --- a/docs/design/cot_vintage.md +++ b/docs/design/cot_vintage.md @@ -252,6 +252,142 @@ Still deferred: `vintage stats` (needs a quarter of data to measure), category-migration detection (needs revisions to exist), tombstone *logic* (column present, needs a real disappearing key to design against). +## 6. The frozen-year tripwire is now automated (2026-07-30) + +§3b established that CFTC regenerates a **rolling two-year window**, and §12.3 of the +handoff recorded the consequence as a decision ("`--all` is a restatement tripwire, not a +backfill, monthly or quarterly"). That decision had a hole: a tripwire that only fires when +somebody remembers to run `--all` by hand is not a detector, it is an intention. This pass +closes it. + +Three regeneration regimes, each with a different expected outcome: + +| Regime | Files | Expected weekly outcome | What violates it | +|---|---|---|---| +| `churn` | current year | new bytes | nothing, that is the data arriving | +| `frozen_in_window` | **prior year** | `unchanged bytes (deduped)` | a new sha (restatement), or a 304 / failure (the check went blind) | +| `frozen_out_of_window` | 2 or more years back | `304 not-modified` | a 200 carrying new bytes | +| `weekly` | the weekly static | new content each Friday | n/a | + +The prior-year slot is the load-bearing one, and it is the only place CFTC hands over a +free weekly **content** check on closed data. Everything older is never re-served, so its +content is not verified at all: a 304 there proves only that a header did not move. + +**Changed: the prior year is now in the DEFAULT fetch set.** It costs one roughly 7 MB +transfer per week (the other six days return 304, because CFTC re-touches `Last-Modified` +weekly) and zero bytes on disk, because the content is byte-identical and dedupes. That is +the entire price of the only automated retroactive-restatement detector in the stack. +`--no-prior-year` turns it off. A targeted `--year N` is still taken at face value with no +prior-year companion, because that is a targeted capture rather than the scheduled sweep. + +Two alert shapes, deliberately triggered differently: + +- **Content changed on a frozen year: always alerts.** It is a discrete event with new + bytes sitting beside the old ones, so it is diffable, and two restatements in + consecutive weeks are two things worth knowing rather than one. In January the message + says so explicitly, since year-end finalisation is the one benign way this fires. +- **The detector went blind (frozen-in-window 304 or fetch failure): alerts on the + TRANSITION only.** This is a standing condition. If CFTC stopped re-touching the prior + year, a level-triggered alert would fire every day forever, and an alert that never + clears is one that gets ignored. Same reasoning that scopes the ingest revision alert to + the current run's snapshots. + +`fetch` deliberately still exits zero. The Windows wrapper aborts the whole run on a +non-zero fetch, so alerting there would skip the `ingest` that turns a restatement into +readable revision rows, which is exactly what you want when it fires. The alert is +persisted on the snapshot (`expectation`, `outcome`, `tripwire_alert`) and re-raised by +`ingest`, which is the step that can exit non-zero safely because everything downstream of +it has already run. That path already writes the `REVISIONS_.txt` marker file. + +## 7. Flow decomposition, and what it found in the canonical schema + +Module spec §6.4 built as `vintage_flow.py`. Weekly ΔLong versus ΔShort per +market/category, labelled `new_longs` / `short_covering` / `new_shorts` / +`long_liquidation`. No prices, no contract master, no multiplier: every input is a column +the canonical schema already stores, which is what makes it a clean smoke test of that +schema rather than of anything else. + +Two design points that are not in the spec because they only appear against real data: + +- **Classification is by dominant leg.** The spec's table has a "~0" leg, which never + happens: both legs always move a little. Whichever of |ΔLong|, |ΔShort| is larger names + the state and its sign picks the direction, with exact ties going to the long leg so the + result is deterministic. This is parameter-free, so there is nothing to tune and nothing + to overfit. An optional `min_frac_oi` dead zone adds a `quiet` state; it defaults to 0.0 + because any other value is a judgement in the same class as the fragility weights and + belongs in a caller's config where it can be swept. +- **`oi_corroborates`.** Futures are closed and zero-sum, so contracts exist only because + somebody opened them: fresh positioning should coincide with rising open interest and + exits with falling. Where it does not, the label is describing a transfer of an existing + position between categories rather than new or closed risk. Kept as a separate column + rather than folded into the state, because open interest in the canonical schema is the + MARKET total (that is what CFTC reports), so it corroborates a per-category label + against a market-level quantity. A real check, not a proof. + +### Measured over the whole store (2026-07-30) + +95 Legacy markets, 1986 to 2026-07-21, canonicalised and decomposed: + +| Measure | Result | +|---|---| +| Zero-sum identity (Σ long across categories == Σ short) | **149,412 / 149,412 weeks balanced** | +| Flow rows produced | 447,951 | +| Exceptions | 0 | +| `validate()` warnings | 0 (after the fix below) | + +The zero-sum result is the strongest available validation of `canonicalize_legacy`: if the +category mapping had dropped, duplicated or misrouted a column, the identity would break on +the first week. It does not break on any week of any market in forty years. + +### Finding 1: non-commercial SPREADING is not captured, and never was + +The side totals reconcile with each other but fall short of open interest by a constant, +equal amount on both sides. That gap is `NonComm_Positions_Spread_All`, which is absent +from `providers/cftc.py`'s `TARGET_COLS` and therefore from the stored parquet, from the +canonical rows, and from every vintage observation. Gold on 2026-07-21: OI 383,368, both +sides 351,385, gap 31,983. 64 of 95 markets have at least one week where the gap is zero +(no spreading that week), which is the confirming case. + +Spreading is a matched long and short held by one trader, so it cancels out of the +long-versus-short identity while still counting toward open interest, which is exactly why +the gap is equal on both sides. **Consequences to know before relying on this:** + +- Net positioning is unaffected, because spreading is long and short in equal measure. +- Anything denominated as a **share of open interest** (module spec §5.2 step 2, which is + the next build step) is measuring a numerator against a denominator that includes + contracts the numerator cannot see. In gold that is 8% of OI. +- Trader counts are likewise absent from the stored table, though the vintage ingest path + reads them from the zip directly rather than from the parquet. + +Not fixed here. Adding the column changes `providers/cftc.py` output, which breaks the +byte-identical guarantee and the `current/` golden baseline that exists to protect it, so +it is a deliberate separate change and not a drive-by. + +### Finding 2: `validate()`'s open-interest warning was mis-specified + +It compared one category's `long + short + spread` against total open interest. That is +not a real bound: the two sides of a market are counted separately, so a category holding +80k long and 294k short in a 383k-OI market sums to 374k with nothing wrong. Measured, it +fired on **811 of 5,778 gold rows (14%)**, which is noise, and a soft warning at that rate +is one nobody reads. + +Corrected to the invariant that does hold: per side, summed across every category, since +every long contract in the market belongs to exactly one category. Store-wide warnings +went from thousands to **zero**. The module spec's §4 adapter contract said +`long + short + spread <= OI`, so the spec was wrong too and is amended in the same pass. + +### Finding 3: COT was FORTNIGHTLY until 1992-10-13 + +`days_elapsed` is emitted as a column rather than assumed to be seven, and that +immediately showed the reason. Across the store, 415,908 of 447,951 intervals are 7 days; +the 15-day (9,057) and 14-day (5,775) intervals are almost entirely pre-October-1992, +when the report was published twice a month. Gold's first 7-day interval is 1992-10-13. + +So a "weekly change" computed over pre-1992 history is a fortnightly change and is not +comparable to the rest of the series. The remaining off-7 intervals are holiday shifts +(gold has 11-day gaps at 2002-01-08 and 2003-02-25). The column is the fix: callers filter +on it rather than discovering this in a result. + ## Bottom line Discovery and the spike are done. The genuine null on historical recovery holds: no diff --git a/docs/design/crowdmon_futures_cot_module.md b/docs/design/crowdmon_futures_cot_module.md index c4d1af7..076f8d4 100644 --- a/docs/design/crowdmon_futures_cot_module.md +++ b/docs/design/crowdmon_futures_cot_module.md @@ -166,6 +166,21 @@ class CotSource(Protocol): Two implementations: `LocalCotData` wrapping the existing module, and `CftcApiCotData` as a fallback and for backfill. Validation on every load — schema conformance, `long + short + spread <= OI` sanity, non-null keys, category vocabulary check. +> **Amended 2026-07-30, the OI sanity check as written is wrong.** `long + short + spread +> <= OI` is not a bound on a single category row: the two sides of a market are counted +> separately, so a category holding 80k long and 294k short in a 383k-OI market sums to +> 374k with nothing wrong. Implemented literally it fired on 811 of 5,778 gold rows (14%). +> +> The invariant that does hold is **per side, summed across every category**, since every +> long contract in the market belongs to exactly one category: `Σ long <= OI` and +> `Σ short <= OI`. That is now what `vintage_ingest.validate` checks, and store-wide +> warnings went from thousands to zero. +> +> Stronger still, and free: **`Σ long == Σ short`**, because futures are closed and +> zero-sum. Verified on **149,412 / 149,412 weeks** across all 95 Legacy markets, 1986 to +> 2026. Exposed as `vintage_flow.zero_sum_check` and worth running on every adapter load, +> because a dropped, duplicated or misrouted category column breaks it on the first week. + *Open items to resolve at implementation:* the existing module's output schema and index conventions, whether history is already backfilled and from what date, and whether revisions are currently retained or overwritten. Revision retention is the one that matters most — see §5.3. > **Resolved 2026-07-30** by the vintage build (branch `claude/cot-revision-snapshots-9b196f`, @@ -184,6 +199,15 @@ Two implementations: `LocalCotData` wrapping the existing module, and `CftcApiCo > - **Only futures-only is fetched**, so `combined` is constant-`False` today: the column is > present and correct but not yet discriminating, and half the reportable universe is absent > until the combined files are added. +> - **`spread_contracts` is always null, and `NonComm_Positions_Spread_All` is not +> captured at all.** It is absent from `providers/cftc.py`'s `TARGET_COLS`, so it never +> reaches the stored parquet, the canonical rows, or any vintage observation. Measured as +> the exact, equal gap between each side total and open interest: gold on 2026-07-21 had +> OI 383,368 against 351,385 on both sides, a gap of 31,983 (8% of OI). Net positioning +> is unaffected, since spreading is long and short in equal measure, but **anything +> denominated as a share of open interest (§5.2 step 2) has a denominator containing +> contracts its numerator cannot see.** Trader counts are likewise missing from the +> stored table, though the vintage ingest path reads them straight from the zip. > - **`vintage: int` is not how it was built.** The implementation is bitemporal > (`observed_at` plus change-only rows), so a point-in-time read is "greatest > `observed_at <= t` per natural key". An integer vintage ordinal can be derived from that @@ -293,6 +317,37 @@ Weekly change in net position decomposes into four states, which have different A rally driven by short covering and a rally driven by new longs look identical on a chart and are entirely different setups. This decomposition is one line of code and is among the highest-value outputs in the system. +> **Built 2026-07-30** as `cotdata/vintage_flow.py` (`cotdata-vintage flow`), directly on +> the canonical schema with no prices and no contract master. It lives in `cotdata` as a +> read-side function that writes no store domain and changes no producer/consumer contract; +> the positioning ENGINE stays here in crowdmon, because that one needs prices, a contract +> master and configured weights. Three things the table above does not say, all of which +> only appear against real data: +> +> - **"~0" never happens.** Both legs always move. Resolved by DOMINANT LEG: whichever of +> |ΔLong|, |ΔShort| is larger names the state and its sign gives the direction, exact +> ties to the long leg. Parameter-free, so nothing to tune and nothing to overfit. An +> optional `min_frac_oi` dead zone adds a `quiet` state and defaults to 0.0, since any +> other value is a judgement of the same kind as §6.3's fragility weights. +> - **Open interest corroborates the label, and disagrees often enough to matter.** Fresh +> positioning should create contracts and exits should destroy them; where it does not, +> the label describes a transfer between categories rather than new or closed risk. +> Emitted as `oi_corroborates` rather than folded into the state, because open interest +> here is the market total (that is what CFTC reports), so it checks a per-category label +> against a market-level quantity. +> - **The weekly change is not always weekly. COT was FORTNIGHTLY until 1992-10-13.** +> Across the store 415,908 of 447,951 intervals are 7 days; the 14 and 15-day intervals +> are almost all pre-October-1992, and the rest are holiday shifts. `days_elapsed` is +> emitted as a column so a caller filters on it instead of discovering it in a result. +> This is §5.3's holiday hazard showing up in a second place: not just in the release +> date, but in the differencing interval. +> +> Worked example, gold non-commercial, week ending 2026-06-02: Δnet was **+21,760**, which +> reads as heavy fresh buying. The decomposition says ΔShort **-16,368** against ΔLong +> +5,392 with open interest **down 27,437**: short covering, a rally with a finite fuel +> supply. That is the distinction this section exists to make, on the first real week +> anyone looked at. + --- ## 7. Cross-market engine diff --git a/docs/examples/windows/run-vintage.cmd b/docs/examples/windows/run-vintage.cmd index f17a60a..0dc99be 100644 --- a/docs/examples/windows/run-vintage.cmd +++ b/docs/examples/windows/run-vintage.cmd @@ -80,7 +80,16 @@ REM ---------------------------------------------------------------------------- REM Optional: identify yourself to CFTC. Defaults to the repo URL if unset. REM set COTDATA_USER_AGENT=cotdata-vintage/0.1 (+contact you@example.com) -REM Default path = current year's three annual reports + the Legacy weekly static. +REM Default path = current year's three annual reports, the PRIOR year's three, and the +REM Legacy weekly static. +REM +REM The prior year is the FROZEN-YEAR TRIPWIRE. CFTC regenerates a rolling two-year window +REM and nothing older, so the prior year is re-served weekly but byte-identical: the one +REM place a content check on closed data is free. Expected result every week is exactly +REM "unchanged bytes (deduped)". Anything else is the retroactive-restatement signature and +REM raises an alert that `ingest` below turns into a non-zero exit and a marker file. +REM Costs about 7 MB of transfer per week (the other six days 304) and nothing on disk. +REM Pass --no-prior-year to turn it off. REM The weekly static is fetched for its HTTP Last-Modified, which is a true REM publication timestamp rather than a polling-interval approximation. "REPLACE_WITH_VENV_PATH\Scripts\cotdata-vintage.exe" fetch >> "%VINTAGE_LOG%" 2>&1 diff --git a/src/cotdata/vintage.py b/src/cotdata/vintage.py index 425d38f..e0bb8ac 100644 --- a/src/cotdata/vintage.py +++ b/src/cotdata/vintage.py @@ -258,6 +258,115 @@ def update_snapshot(snapshot_id: str, **fields) -> bool: return hit +# ── The frozen-year tripwire ──────────────────────────────────────────────── +# CFTC's weekly regeneration job touches a ROLLING TWO-YEAR WINDOW: the current year and +# the immediately-prior year, nothing older (measured 2026-07-30: 2025 and 2026 shared an +# identical Last-Modified while 2022/2023/2024 sat on a single January bulk re-touch; see +# docs/design/cot_vintage_store_handoff.md §12.2). That gives three regimes, and each has a +# DIFFERENT expected outcome, which is what makes a violation meaningful rather than noise: +# +# current year new bytes every week. Real new data. Nothing to assert. +# prior year (frozen) re-served every week but byte-IDENTICAL, so the expected record +# is exactly "unchanged bytes (deduped)". This is the only regime +# where CFTC hands us a free weekly content check on closed data. +# older (frozen) never re-touched, so If-Modified-Since 304s and no bytes arrive. +# Content is not verified at all; a 200 with new bytes is the alert. +# +# The prior-year slot is the load-bearing one. It is the sole automated detector for the +# failure mode this subsystem exists to guard against (July 2008: reports restated back to +# 2007-07-03), and it costs one ~7 MB weekly transfer that dedupes to zero bytes on disk. +EXPECT_CHURN = "churn" # current year: new bytes are normal +EXPECT_FROZEN_IN_WINDOW = "frozen_in_window" # prior year: re-served, must be identical +EXPECT_FROZEN_OUT_OF_WINDOW = "frozen_out_of_window" # older: 304, content unverified +EXPECT_WEEKLY = "weekly" # the weekly static: new content every Friday + + +def capture_expectation(report_year: int | None, current_year: int) -> str: + """Which regeneration regime a source sits in. See the block comment above.""" + if report_year is None: + return EXPECT_WEEKLY + if report_year >= current_year: + return EXPECT_CHURN + if report_year == current_year - 1: + return EXPECT_FROZEN_IN_WINDOW + return EXPECT_FROZEN_OUT_OF_WINDOW + + +# What actually happened on this fetch, as a single controlled token. Recorded on every +# snapshot so the tripwire can be EDGE-triggered off the previous record rather than +# re-firing on a standing condition. +OUTCOME_FIRST = "first" # no prior snapshot for this url, a baseline +OUTCOME_CHANGED = "changed" # 200, bytes differ from the last retained copy +OUTCOME_DEDUPED = "deduped" # 200, bytes byte-identical to the last retained copy +OUTCOME_NOT_MODIFIED = "not_modified" # 304, server declined to re-send +OUTCOME_FAILED = "failed" + + +def _tripwire_alert(expectation: str, outcome: str, prev_outcome: str | None) -> str | None: + """The alert reason for one capture, or None. + + Two kinds of violation, deliberately triggered differently: + + **Content changed on a frozen year** always alerts. It is a discrete event with new + bytes sitting next to the old ones, so it is diffable, and two restatements in + consecutive weeks are two things worth knowing about, not one. + + **The detector went blind** (a frozen-in-window year that 304s or fails) alerts only on + the TRANSITION into that state. It is a standing condition: if CFTC stopped re-touching + the prior year's Last-Modified, a level-triggered alert would fire every single day + forever, and an alert that never clears is one that gets ignored. Same reasoning + that scopes the ingest revision alert to this run's snapshots. Edge-triggering means it + is reported once, loudly, and then stays quiet until the pattern changes back. + + Note what is NOT an alert: a frozen year outside the two-year window returning 304. + That is the correct, expected outcome there, and it is also the reason the prior-year + slot matters: it is the only closed data whose CONTENT is checked at all. + """ + if expectation == EXPECT_FROZEN_IN_WINDOW: + if outcome == OUTCOME_CHANGED: + return ("content changed on the frozen prior year: CFTC re-serves this file " + "weekly and it has been byte-identical, so a new sha here is the " + "retroactive-restatement signature") + if outcome in (OUTCOME_NOT_MODIFIED, OUTCOME_FAILED) and prev_outcome == OUTCOME_DEDUPED: + return (f"the frozen prior year stopped being re-served ({outcome}) after " + f"previously deduping. The weekly content check has gone blind, so a " + f"restatement would no longer be detected here") + return None + if expectation == EXPECT_FROZEN_OUT_OF_WINDOW and outcome == OUTCOME_CHANGED: + return ("content changed on a closed year outside CFTC's regeneration window, " + "which should never be re-touched at all") + return None + + +def _annotate(rec: dict, *, prev: dict | None, now: dt.datetime) -> dict: + """Attach expectation / outcome / tripwire_alert to a capture record.""" + expectation = capture_expectation(rec.get("report_year"), now.year) + note = rec.get("note") or "" + if note.startswith("fetch failed"): + outcome = OUTCOME_FAILED + elif rec.get("http_status") == 304: + outcome = OUTCOME_NOT_MODIFIED + elif prev is None: + outcome = OUTCOME_FIRST + elif rec.get("note") == "unchanged bytes (deduped)": + outcome = OUTCOME_DEDUPED + else: + outcome = OUTCOME_CHANGED + alert = _tripwire_alert(expectation, outcome, (prev or {}).get("outcome")) + # Year-end finalisation is the one benign way a frozen year legitimately changes, and + # it happens on a known schedule. Say so in the message rather than teaching the reader + # to discount every January alert, which would train them to discount all of them. + if alert and outcome == OUTCOME_CHANGED and now.month == 1: + alert += (". NOTE: it is January, when CFTC finalises the year just ended, so this " + "is the one time of year a frozen-year change is plausibly routine") + rec["expectation"] = expectation + rec["outcome"] = outcome + rec["tripwire_alert"] = alert + if alert: + print(f" *** TRIPWIRE: {rec.get('report_type')} {rec.get('report_year')}: {alert}.") + return rec + + # ── Capture ───────────────────────────────────────────────────────────────── def _utcnow() -> dt.datetime: return dt.datetime.now(dt.timezone.utc).replace(microsecond=0) @@ -293,7 +402,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d if res.status == 304 or res.content is None: # Server says unchanged: record the check, retain nothing new, reuse prior file. - return { + return _annotate({ **base, "snapshot_id": _snapshot_id(retrieved_at, source.url, "304"), "http_status": 304, @@ -303,7 +412,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": None, "local_path": (prev or {}).get("local_path"), "note": "304 not-modified", - } + }, prev=prev, now=now) floor = _MIN_BYTES.get(source.source_kind, _MIN_BYTES_DEFAULT) if len(res.content) < floor: @@ -315,7 +424,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d sha = hashlib.sha256(res.content).hexdigest() if prev and prev.get("content_sha256") == sha: # Byte-identical to what we already retained (zips regenerate): dedupe, no rewrite. - return { + return _annotate({ **base, "snapshot_id": _snapshot_id(retrieved_at, source.url, sha[:8]), "http_status": res.status, @@ -325,7 +434,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": len(res.content), "local_path": prev.get("local_path"), "note": "unchanged bytes (deduped)", - } + }, prev=prev, now=now) # New bytes: retain immutably, via a .part file + atomic replace. A crash during a # plain write_bytes would leave a TRUNCATED file already carrying the full sha in its @@ -349,22 +458,17 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d rel = str(dest) # COTDATA_VINTAGE_ROOT points outside the store # ── Restatement tripwire ──────────────────────────────────────────────── - # A CLOSED year's content is frozen (measured: 2020-2024 static, and 2025 re-touched - # weekly by the rolling regeneration window but byte-identical since January). So a - # closed year whose sha CHANGES is the 2008-style retroactive-restatement signature — - # the failure mode this whole subsystem exists to detect — and it costs no extra - # machinery: it falls out of the ordinary dedupe path. 2025 in particular should - # produce exactly one "unchanged bytes (deduped)" record per week, indefinitely; the - # week it does not is the alert. - restatement_suspect = False - if prev is not None and source.report_year is not None and source.report_year < now.year: - restatement_suspect = True - print(f" *** RESTATEMENT SUSPECT: {source.report_type} {source.report_year} " - f"content changed (sha {(prev.get('content_sha256') or '')[:8]} -> {sha[:8]}). " - f"A closed year should be frozen. In January this may be ordinary year-end " - f"finalization; otherwise treat as a retroactive restatement and diff it.") - - return { + # A CLOSED year's content is frozen, so a closed year whose sha CHANGES is the + # 2008-style retroactive-restatement signature, the failure mode this whole subsystem + # exists to detect, and it costs no extra machinery: it falls out of the ordinary + # dedupe path. ``_annotate`` below classifies which frozen regime this is and writes + # the human-readable reason; this flag stays because it is the field ``ingest`` and + # every stored snapshot already key off. + restatement_suspect = ( + prev is not None and source.report_year is not None + and source.report_year < now.year) + + return _annotate({ **base, "snapshot_id": _snapshot_id(retrieved_at, source.url, sha[:8]), "restatement_suspect": restatement_suspect, @@ -375,24 +479,41 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": len(res.content), "local_path": rel, "note": None, - } + }, prev=prev, now=now) def fetch(year: int | None = None, *, all_years: bool = False, - include_weekly: bool = True, sources: list[Source] | None = None, + include_weekly: bool = True, include_prior_year: bool = True, + sources: list[Source] | None = None, http_get=_http_get, rate_limit_s: float = _RATE_LIMIT_S, now_fn=_utcnow) -> dict: """Capture the release-critical CFTC files into the immutable landing zone. - Default: the current year's three annual zips (which carry the newest release) plus - the Legacy weekly static (for its true-publication Last-Modified). ``all_years`` - walks 1986→current for a cold-start backfill of raw bytes. An explicit ``sources`` - list overrides the year/weekly derivation (targeted capture; used in tests). - - Returns ``{"records": [...], "new_files": n, "checks": n}``. + Default: the current year's three annual zips (which carry the newest release), the + PRIOR year's three (the frozen-year tripwire, see the block comment above + ``capture_expectation``), plus the Legacy weekly static (for its true-publication + Last-Modified). ``all_years`` walks 1986→current for a cold-start backfill of raw + bytes. An explicit ``sources`` list overrides the year/weekly derivation (targeted + capture; used in tests). + + The prior year is in the DEFAULT set rather than only on ``--all`` because the + tripwire is worth nothing if it only runs when someone remembers to run it manually. + It costs one ~7 MB transfer per week (CFTC re-touches Last-Modified weekly, so the + other six days 304) and zero bytes on disk, because the content is byte-identical and + dedupes. That is the whole price of the only automated retroactive-restatement + detector in the stack. An explicit ``year`` is taken at face value, with no prior-year + companion: that is a targeted capture, not the scheduled sweep. + + Returns ``{"records": [...], "new_files": n, "checks": n, "failed": n, + "tripwire_alerts": [...]}``. """ if sources is None: this_year = now_fn().year - years = range(1986, this_year + 1) if all_years else [year or this_year] + if all_years: + years = list(range(1986, this_year + 1)) + elif year is not None: + years = [year] + else: + years = [this_year] + ([this_year - 1] if include_prior_year else []) sources = [] for y in years: sources.extend(annual_sources(y)) @@ -414,7 +535,7 @@ def fetch(year: int | None = None, *, all_years: bool = False, # and dies at the first naming variant or missing disagg year otherwise. The # failure is RECORDED (so it is visible and retryable), then the run goes on — # matching what the ingest path already does. - rec = _failure_record(src, now, e) + rec = _failure_record(src, now, e, prev=_latest_for_url(snapshots, src.url)) failed += 1 print(f" {src.report_type}/{src.source_kind} {src.report_year or ''}: " f"fetch failed — {e}") @@ -431,12 +552,14 @@ def fetch(year: int | None = None, *, all_years: bool = False, time.sleep(rate_limit_s) return {"records": records, "new_files": new_files, "checks": len(records), - "failed": failed} + "failed": failed, + "tripwire_alerts": [r for r in records if r.get("tripwire_alert")]} -def _failure_record(source: Source, now: dt.datetime, error: Exception) -> dict: +def _failure_record(source: Source, now: dt.datetime, error: Exception, + *, prev: dict | None = None) -> dict: retrieved_at = _iso(now) - return { + return _annotate({ "source_url": source.url, "source_kind": source.source_kind, "report_type": source.report_type, @@ -452,4 +575,4 @@ def _failure_record(source: Source, now: dt.datetime, error: Exception) -> dict: "parse_status": "pending", "parse_error": None, "note": f"fetch failed: {error}", - } + }, prev=prev, now=now) diff --git a/src/cotdata/vintage_cli.py b/src/cotdata/vintage_cli.py index 165f1e8..01bb0dd 100644 --- a/src/cotdata/vintage_cli.py +++ b/src/cotdata/vintage_cli.py @@ -18,11 +18,22 @@ def _cmd_fetch(args) -> int: from . import vintage res = vintage.fetch(year=args.year, all_years=args.all, - include_weekly=not args.no_weekly) + include_weekly=not args.no_weekly, + include_prior_year=not args.no_prior_year) print(f"vintage fetch: {res['checks']} source(s) checked, " f"{res['new_files']} new raw file(s) retained.") for rec in res["records"]: - print(f" {rec['report_type']:<13} {rec['source_kind']:<13} {rec.get('note') or 'NEW'}") + print(f" {rec['report_type']:<13} {rec['source_kind']:<13} " + f"{rec.get('report_year') or '':<6} {rec.get('expectation') or '':<22} " + f"{rec.get('outcome') or '':<13} {rec.get('note') or 'NEW'}") + # Deliberately NOT a non-zero exit. The wrapper aborts the whole run on a non-zero + # fetch, so alerting here would skip the ingest that turns a restatement into readable + # revision rows, precisely the thing you want when the tripwire fires. The alert is + # persisted on the snapshot and re-raised by `ingest`, which is the step that can exit + # non-zero safely because everything downstream of it has already run. + for rec in res["tripwire_alerts"]: + print(f" *** TRIPWIRE {rec['report_type']} {rec.get('report_year')}: " + f"{rec['tripwire_alert']}") return 0 @@ -77,18 +88,33 @@ def _cmd_ingest(args) -> int: # A store-wide scan would keep firing on every subsequent run once a single # restatement had been seen, and an alert that never clears is one that gets ignored. suspects = [s for s in snaps if s.get("restatement_suspect")] - if total_rev or suspects: - _report_revisions(total_rev, suspects) + # Tripwire alerts overlap with `suspects` on the content-changed case and add the + # "detector went blind" case, which has no changed sha and so no suspect flag. Both + # need the same exit path: this is the only step that can exit non-zero without + # skipping downstream work. + alerts = [s for s in snaps if s.get("tripwire_alert")] + if total_rev or suspects or alerts: + _report_revisions(total_rev, suspects, alerts) + parts = [f"{total_rev} revision(s)"] + if suspects: + parts.append(f"{len(suspects)} closed-year restatement suspect(s)") + if alerts: + parts.append(f"{len(alerts)} frozen-year tripwire alert(s)") raise SystemExit( - f"cotdata-vintage: {total_rev} revision(s)" - + (f", {len(suspects)} closed-year restatement suspect(s)" if suspects else "") + "cotdata-vintage: " + ", ".join(parts) + " — review with 'cotdata-vintage diff'. Non-zero so a scheduler surfaces this; " "the data IS committed, this is a notification, not a failure.") return 0 -def _report_revisions(total_rev: int, suspects: list) -> None: +def _report_revisions(total_rev: int, suspects: list, alerts: list = ()) -> None: from . import vintage_ingest + if alerts: + print("\n*** FROZEN-YEAR TRIPWIRE ***") + for s in alerts[-5:]: + print(f" {s.get('report_type')} {s.get('report_year')} " + f"[{s.get('expectation')} -> {s.get('outcome')}] at {s.get('retrieved_at')}") + print(f" {s.get('tripwire_alert')}") if suspects: print("\n*** CLOSED-YEAR RESTATEMENT SUSPECT ***") for s in suspects[-5:]: @@ -144,6 +170,44 @@ def _cmd_asof(args) -> int: return 0 +def _cmd_flow(args) -> int: + from . import vintage_flow + + if args.source == "current": + canonical = vintage_flow.from_current_store(args.market) + print(f"vintage flow: {args.market} from the CURRENT-STATE store. NOT " + f"point-in-time. Revisions are already applied, so this is for looking at " + f"history and validating the schema, never for evaluating a rule.") + else: + canonical = vintage_flow.from_vintage(as_of=args.as_of, market_code=args.market) + if canonical.empty: + print("vintage flow: no vintage observations yet. The vintage series begins " + "at first capture; use --source current for history.") + return 0 + + z = vintage_flow.zero_sum_check(canonical) + unbalanced = int((~z["balanced"]).sum()) + print(f" zero-sum: {len(z) - unbalanced}/{len(z)} weeks balanced" + + (f" *** {unbalanced} UNBALANCED ***" if unbalanced else "")) + + fl = vintage_flow.decompose(canonical, min_frac_oi=args.min_frac_oi) + if args.category: + fl = fl[fl["category"] == args.category] + if fl.empty: + print(" no weeks to decompose.") + return 0 + off = fl[fl["days_elapsed"] != 7] + if len(off): + print(f" {len(off)} of {len(fl)} intervals are not 7 days (COT was FORTNIGHTLY " + f"before 1992-10-13, and holidays shift the rest). Those rows are not a " + f"weekly change and are not comparable to one.") + print(f"\n{fl['state'].value_counts().to_string()}\n") + cols = ["report_date", "category", "d_long", "d_short", "d_net", "d_oi", + "days_elapsed", "state", "oi_corroborates"] + print(fl.tail(args.last)[cols].to_string(index=False)) + return 0 + + def main(argv=None) -> int: p = argparse.ArgumentParser( prog="cotdata-vintage", @@ -154,6 +218,9 @@ def main(argv=None) -> int: f.add_argument("--year", type=int, default=None, help="Report year (default: current).") f.add_argument("--all", action="store_true", help="Every year 1986→present (cold-start).") f.add_argument("--no-weekly", action="store_true", help="Skip the current-week static.") + f.add_argument("--no-prior-year", action="store_true", dest="no_prior_year", + help="Skip the prior year, disabling the frozen-year restatement " + "tripwire. Saves one ~7 MB weekly transfer and nothing on disk.") f.set_defaults(func=_cmd_fetch) ing = sub.add_parser("ingest", help="Parse retained raw files into change-only observations.") @@ -174,6 +241,21 @@ def main(argv=None) -> int: a.add_argument("--market", default=None) a.set_defaults(func=_cmd_asof) + fw = sub.add_parser("flow", help="Weekly flow decomposition per market/category.") + fw.add_argument("--market", required=True, help="CFTC market code, e.g. 088691.") + fw.add_argument("--source", choices=("vintage", "current"), default="vintage", + help="'vintage' is point-in-time and starts at first capture; " + "'current' reads the existing store back to 1986 but has " + "revisions already applied, so it is NOT point-in-time.") + fw.add_argument("--as-of", default=None, dest="as_of", + help="Point-in-time timestamp (vintage source only).") + fw.add_argument("--category", default=None, help="e.g. noncommercial.") + fw.add_argument("--min-frac-oi", type=float, default=0.0, dest="min_frac_oi", + help="Dead zone as a fraction of prior open interest; weeks under it " + "on both legs are 'quiet'. Default 0.0 (no dead zone).") + fw.add_argument("--last", type=int, default=20, help="Rows to print (default 20).") + fw.set_defaults(func=_cmd_flow) + args = p.parse_args(argv) config.store_root() # fail fast if COTDATA_STORE unset return args.func(args) diff --git a/src/cotdata/vintage_flow.py b/src/cotdata/vintage_flow.py new file mode 100644 index 0000000..5f1ce04 --- /dev/null +++ b/src/cotdata/vintage_flow.py @@ -0,0 +1,226 @@ +"""Flow decomposition over canonical COT rows, for crowdmon-futures module spec §6.4. + +Weekly ΔLong versus ΔShort per market/category, labelled into the four states that look +identical on a price chart and are entirely different setups: + + ΔLong +, ΔShort ~0 new longs fresh conviction buying + ΔLong ~0, ΔShort − short covering a rally with a FINITE fuel supply + ΔLong ~0, ΔShort + new shorts fresh bearish conviction + ΔLong −, ΔShort ~0 long liquidation position exit, not fresh selling + +Runs on ingested COT alone: no prices, no contract master, no multiplier. That is what +makes it the right first consumer of the canonical schema: every input is a column the +vintage layer already stores, so a failure here is a schema failure and nothing else. + +**Why this lives in cotdata.** It is a READ-side function over cotdata's own canonical +rows: it writes no store domain, adds no manifest entry, and changes no producer/consumer +contract. ADR-0007 narrows cotdata along the axis of instrument domain (COT versus bars), +not derived-versus-raw, and ADR-0008 settles that provenance for COT data stays inside +that boundary. This is the same shape of thing: COT in, COT out, nothing else consulted. +The positioning ENGINE of the module spec (extremity, fragility, cross-market PCA) is a +different matter, because it needs prices, a contract master and configured weights, and +belongs in the crowdmon package where the spec puts it. + +See docs/design/crowdmon_futures_cot_module.md §6.4. +""" +from __future__ import annotations + +import pandas as pd + +from . import config, vintage_ingest + +# The grouping key: one series per market/category, walked in report_date order. +SERIES_KEY = ["market_code", "report_type", "combined", "category"] + +NEW_LONGS = "new_longs" +SHORT_COVERING = "short_covering" +NEW_SHORTS = "new_shorts" +LONG_LIQUIDATION = "long_liquidation" +QUIET = "quiet" +FLOW_STATES = (NEW_LONGS, SHORT_COVERING, NEW_SHORTS, LONG_LIQUIDATION, QUIET) + +OUT_COLUMNS = SERIES_KEY + [ + "report_date", "days_elapsed", + "long_contracts", "short_contracts", "open_interest", + "d_long", "d_short", "d_net", "d_oi", + "state", "oi_corroborates", +] + + +class FlowError(ValueError): + """The input is not a clean one-row-per-key-per-date panel.""" + + +def _require_panel(canonical: pd.DataFrame) -> None: + missing = [c for c in SERIES_KEY + ["report_date", "long_contracts", "short_contracts"] + if c not in canonical.columns] + if missing: + raise FlowError(f"missing columns for flow decomposition: {missing}") + dup = canonical.duplicated(subset=SERIES_KEY + ["report_date"]) + if dup.any(): + sample = canonical.loc[dup, SERIES_KEY + ["report_date"]].head(3).to_dict("records") + raise FlowError( + f"{int(dup.sum())} duplicate (key, report_date) rows, e.g. {sample}. " + f"A diff over a key with two rows for one week silently compares two VINTAGES " + f"of the same week and calls the revision a flow. Pass a point-in-time slice " + f"(vintage_ingest.asof) rather than raw observations.") + + +def decompose(canonical: pd.DataFrame, *, min_frac_oi: float = 0.0) -> pd.DataFrame: + """Label each week's positioning change per market/category. + + ``canonical`` is the long-form schema (``vintage_ingest.ALL_COLUMNS``), one row per + natural key per report date. Raises rather than guessing if a key has two rows for one + date, because that means two vintages and a diff across them is a revision, not a flow. + + **Classification is by dominant leg**, which is how the spec's "~0" resolves against + real data where both legs always move a little: whichever of |ΔLong|, |ΔShort| is + larger names the state, and its sign picks the direction. Exact ties go to the long + leg, so the result is deterministic. This is parameter-free by default and therefore + reproducible with nothing to tune. + + ``min_frac_oi`` optionally adds a dead zone: a week where BOTH legs move less than + that fraction of the prior week's open interest is labelled ``quiet`` instead of being + forced into a direction by noise. It defaults to 0.0 (no dead zone) deliberately: any + non-zero value is a judgement, in the same class as the module spec's fragility + weights, and belongs in a config the caller owns and can run sensitivity over, not + baked in here as a fake default. + + The first observation of each series has no predecessor and is dropped: there is no + such thing as its weekly change. + """ + _require_panel(canonical) + df = canonical.copy() + df["report_date"] = pd.to_datetime(df["report_date"]) + df = df.sort_values(SERIES_KEY + ["report_date"], kind="mergesort") + + g = df.groupby(SERIES_KEY, dropna=False, sort=False) + for src, dst in (("long_contracts", "d_long"), ("short_contracts", "d_short")): + df[dst] = g[src].diff() + df["d_oi"] = g["open_interest"].diff() if "open_interest" in df.columns else pd.NA + df["d_net"] = df["d_long"] - df["d_short"] + # Elapsed calendar days, not "one week". COT weeks are NOT uniformly seven days apart: + # holiday shifts move them, and a capture gap or an early-history hole makes a single + # diff span months. A caller comparing flow magnitudes across weeks must be able to + # see that, so it is a column rather than an assumption. + df["days_elapsed"] = g["report_date"].diff().dt.days + # Prior-week OI is the dead-zone denominator: a threshold has to be knowable BEFORE + # the week it judges, or it is fitted to the outcome it is classifying. + prior_oi = g["open_interest"].shift(1) if "open_interest" in df.columns else None + + df = df[df["d_long"].notna() & df["d_short"].notna()].copy() + df["state"] = _classify(df, prior_oi=None if prior_oi is None else prior_oi.loc[df.index], + min_frac_oi=min_frac_oi) + df["oi_corroborates"] = _corroborate(df) + return df.reindex(columns=OUT_COLUMNS).reset_index(drop=True) + + +def _classify(df: pd.DataFrame, *, prior_oi, min_frac_oi: float) -> pd.Series: + d_long = df["d_long"].astype("float64") + d_short = df["d_short"].astype("float64") + mag_l, mag_s = d_long.abs(), d_short.abs() + + long_dominates = mag_l >= mag_s # ties -> long leg, so the label is deterministic + state = pd.Series(pd.NA, index=df.index, dtype="object") + state = state.mask(long_dominates & (d_long > 0), NEW_LONGS) + state = state.mask(long_dominates & (d_long <= 0), LONG_LIQUIDATION) + state = state.mask(~long_dominates & (d_short > 0), NEW_SHORTS) + state = state.mask(~long_dominates & (d_short <= 0), SHORT_COVERING) + + if min_frac_oi and prior_oi is not None: + dead = prior_oi.astype("float64") * float(min_frac_oi) + state = state.mask((mag_l <= dead) & (mag_s <= dead), QUIET) + return state + + +def _corroborate(df: pd.DataFrame) -> pd.Series: + """Does market open interest agree with the label? + + Futures are a closed zero-sum system, so contracts only exist because someone opened + them. Fresh positioning (new longs, new shorts) should therefore coincide with RISING + open interest, and exits (short covering, long liquidation) with falling open interest. + When it does not, the label is describing a transfer of an existing position between + categories rather than new or closed risk, which is a materially different event. + + Note the asymmetry this cannot escape: ``open_interest`` in the canonical schema is + the MARKET total, repeated on every category row, because that is what the CFTC file + reports. So this corroborates a per-category label against a market-level quantity. It + is a real check and not a proof, which is why it is a separate column rather than + something folded into the state. + """ + if "d_oi" not in df.columns: + return pd.Series(pd.NA, index=df.index, dtype="object") + d_oi = pd.to_numeric(df["d_oi"], errors="coerce") + opening = df["state"].isin([NEW_LONGS, NEW_SHORTS]) + closing = df["state"].isin([SHORT_COVERING, LONG_LIQUIDATION]) + out = pd.Series(pd.NA, index=df.index, dtype="object") + out = out.mask(opening & d_oi.notna(), d_oi > 0) + out = out.mask(closing & d_oi.notna(), d_oi < 0) + return out + + +# ── Schema smoke test ─────────────────────────────────────────────────────── +def zero_sum_check(canonical: pd.DataFrame) -> pd.DataFrame: + """Per (report_date, market): do the category rows reconcile against open interest? + + The strongest check the canonical schema admits, and it needs no external data. In a + futures market every long is somebody's short, so summing ``long_contracts`` across + every category must equal the same sum of ``short_contracts``. If the category mapping + dropped, duplicated or misrouted a column, that identity breaks immediately. + + ``oi_gap`` is what the sides fall short of open interest by. It is expected to be + NON-ZERO and EQUAL on both sides for the Legacy report as stored today: the + non-commercial SPREADING column is not in ``providers/cftc.py``'s ``TARGET_COLS``, so + it never reaches the canonical rows. Spreading is a matched long and short held by one + trader, which is exactly why it cancels out of the long-versus-short identity while + still counting toward open interest. So ``long_total == short_total`` is the assertion, + and ``oi_gap`` is a measurement to read, not a failure, until spreading is captured, + at which point it should go to zero. + + Returns one row per (report_date, market_code, report_type, combined) with + ``long_total``, ``short_total``, ``open_interest``, ``oi_gap`` and ``balanced``. + """ + keys = ["report_date", "market_code", "report_type", "combined"] + missing = [c for c in keys + ["long_contracts", "short_contracts"] + if c not in canonical.columns] + if missing: + raise FlowError(f"missing columns for zero-sum check: {missing}") + g = canonical.groupby(keys, dropna=False, sort=False) + out = g.agg(long_total=("long_contracts", "sum"), + short_total=("short_contracts", "sum"), + open_interest=("open_interest", "max")).reset_index() + out["balanced"] = out["long_total"] == out["short_total"] + out["oi_gap"] = out["open_interest"] - out["long_total"] + return out + + +# ── Sources ───────────────────────────────────────────────────────────────── +def from_vintage(*, as_of=None, market_code=None, report_type="legacy") -> pd.DataFrame: + """Canonical rows out of the vintage store, as a point-in-time slice. + + This is the correct source: ``asof`` returns exactly one row per natural key, so the + diff below can never straddle two vintages of one week. With no ``as_of`` it means + "everything known now", which is still a consistent single vintage per key. + """ + t = pd.Timestamp.max if as_of is None else as_of + return vintage_ingest.asof(t, market_code=market_code, report_type=report_type) + + +def from_current_store(market_code: str) -> pd.DataFrame: + """Canonical rows built from the EXISTING current-state parquet for one market. + + Present because the vintage series begins at first capture and is therefore short, + while the current-state store holds Legacy history back to 1986. That makes this the + only way to exercise the canonical schema over four decades and 95 markets today. + + **Not point-in-time.** These are current values with revisions already applied, so a + result computed from here is contaminated by hindsight and must not be used to + evaluate anything. It is for validating the schema and for looking at history, which + is why it is a separate function with a separate name rather than a flag. + """ + from . import store + wide = store.read_cot_legacy(str(market_code)) + if wide.empty: + raise FlowError(f"no current-state Legacy table for market_code {market_code!r} " + f"(store: {config.cot_legacy_dir()})") + return vintage_ingest.canonicalize_legacy(wide) diff --git a/src/cotdata/vintage_ingest.py b/src/cotdata/vintage_ingest.py index 0038598..c7b86ad 100644 --- a/src/cotdata/vintage_ingest.py +++ b/src/cotdata/vintage_ingest.py @@ -175,8 +175,23 @@ def validate(canonical: pd.DataFrame) -> list[str]: """Fail loudly (raise) on structural problems; return a list of soft warnings. Raises rather than partially ingesting on: missing natural-key columns, null - natural-key values, or categories outside the controlled vocabulary. Warns (does - not fail) on ``long+short+spread > open_interest`` — definitional edge cases exist. + natural-key values, or categories outside the controlled vocabulary. Warns (does not + fail) when a SIDE total breaches open interest. + + **The side total is the invariant, not the per-row sum.** An earlier version compared + one category's ``long + short + spread`` against total open interest, which is not a + real bound: the two sides of a market are counted separately, so a category holding + 80k long and 294k short in a 383k-OI market sums to 374k without anything being wrong. + Measured against the real store it fired on 811 of 5,778 gold rows (14%), pure noise, + and a soft warning that cries wolf at that rate is one nobody reads. + + What actually must hold is per side, summed across every category: every long + contract in the market belongs to exactly one category, so ``Σ long <= OI`` and + ``Σ short <= OI``. Verified against 1,926 weeks of gold: the two side totals matched + each other on every single week. (They fall SHORT of OI by a constant, equal amount on + both sides, which is the non-commercial spreading column that ``providers/cftc.py`` + does not capture (see ``vintage_flow.zero_sum_check``). That gap is expected and is + not warned about.) """ missing = [c for c in NATURAL_KEY if c not in canonical.columns] if missing: @@ -195,13 +210,21 @@ def validate(canonical: pd.DataFrame) -> list[str]: raise ValidationError(f"categories {bad} outside vocabulary for {rt!r}") warnings = [] - for _, r in canonical.iterrows(): - oi = r.get("open_interest") - parts = [r.get("long_contracts"), r.get("short_contracts"), r.get("spread_contracts")] - s = sum(p for p in parts if p is not None and not pd.isna(p)) - if oi is not None and not pd.isna(oi) and s > oi: - warnings.append(f"{r['report_date'].date()} {r['market_code']} " - f"{r['category']}: long+short+spread ({s}) > OI ({oi})") + if "open_interest" not in canonical.columns: + return warnings + keys = ["report_date", "market_code", "report_type", "combined"] + for key, grp in canonical.groupby(keys, dropna=False, sort=False): + oi = grp["open_interest"].max() + if pd.isna(oi): + continue + for side in ("long_contracts", "short_contracts"): + total = pd.to_numeric(grp.get(side), errors="coerce").sum() + spread = pd.to_numeric(grp.get("spread_contracts"), errors="coerce").sum() + if total + spread > oi: + report_date, market_code = key[0], key[1] + warnings.append( + f"{pd.Timestamp(report_date).date()} {market_code}: " + f"{side} summed across categories ({total + spread:.0f}) > OI ({oi})") return warnings diff --git a/tests/test_vintage_capture.py b/tests/test_vintage_capture.py index 0f9d309..9e70adc 100644 --- a/tests/test_vintage_capture.py +++ b/tests/test_vintage_capture.py @@ -291,3 +291,145 @@ def test_changed_bytes_writes_second_immutable_snapshot(store_env): raws = sorted((store_env / "vintage" / "raw").rglob("*.zip")) assert len(raws) == 2 # both vintages retained; neither overwritten assert {p.read_bytes() for p in raws} == {_body(b"week1"), _body(b"week2-revised")} + + +# ── Frozen-year tripwire ──────────────────────────────────────────────────── +# CFTC regenerates a ROLLING TWO-YEAR window (current + prior). The prior year is +# therefore re-served every week but byte-identical, which is the one place we get a free +# weekly CONTENT check on closed data, and the only automated detector for a 2008-style +# retroactive restatement. These tests pin the three regimes and the edge-trigger rule. + +LEGACY_2025 = "https://www.cftc.gov/files/dea/history/dea_fut_xls_2025.zip" +LEGACY_2019 = "https://www.cftc.gov/files/dea/history/dea_fut_xls_2019.zip" + + +def _one(url, year): + from cotdata.vintage import Source + return [Source("legacy", "annual_zip", "zip", url, year)] + + +def test_expectation_follows_the_rolling_two_year_regeneration_window(): + from cotdata import vintage + assert vintage.capture_expectation(2026, 2026) == vintage.EXPECT_CHURN + assert vintage.capture_expectation(2025, 2026) == vintage.EXPECT_FROZEN_IN_WINDOW + assert vintage.capture_expectation(2024, 2026) == vintage.EXPECT_FROZEN_OUT_OF_WINDOW + assert vintage.capture_expectation(None, 2026) == vintage.EXPECT_WEEKLY + + +def test_prior_year_dedupe_is_the_expected_outcome_and_never_alerts(store_env): + """The nominal weekly result for the frozen prior year: CFTC re-touches Last-Modified, + re-serves identical bytes, we dedupe. Silence here is the tripwire working.""" + from cotdata import vintage + body = _body(b"2025-final") + http = _FakeHttp({LEGACY_2025: [(200, body, None, "lm1"), (200, body, None, "lm2")]}) + now = _clock() + vintage.fetch(sources=_one(LEGACY_2025, 2025), http_get=http, rate_limit_s=0, now_fn=now) + r2 = vintage.fetch(sources=_one(LEGACY_2025, 2025), http_get=http, rate_limit_s=0, now_fn=now) + + rec = r2["records"][0] + assert rec["expectation"] == vintage.EXPECT_FROZEN_IN_WINDOW + assert rec["outcome"] == vintage.OUTCOME_DEDUPED + assert rec["tripwire_alert"] is None + assert r2["tripwire_alerts"] == [] + + +def test_prior_year_content_change_alerts(store_env): + """Anything other than a dedupe on the frozen prior year is the alert the user asked + for. A changed sha there is the restatement signature.""" + from cotdata import vintage + http = _FakeHttp({LEGACY_2025: [(200, _body(b"2025-final"), None, "lm1"), + (200, _body(b"2025-RESTATED"), None, "lm2")]}) + now = _clock() + vintage.fetch(sources=_one(LEGACY_2025, 2025), http_get=http, rate_limit_s=0, now_fn=now) + r2 = vintage.fetch(sources=_one(LEGACY_2025, 2025), http_get=http, rate_limit_s=0, now_fn=now) + + rec = r2["records"][0] + assert rec["outcome"] == vintage.OUTCOME_CHANGED + assert rec["tripwire_alert"] and "restatement" in rec["tripwire_alert"] + assert len(r2["tripwire_alerts"]) == 1 + + +def test_prior_year_going_quiet_alerts_once_not_forever(store_env): + """If CFTC stops re-serving the prior year we get a 304 and the content check has gone + BLIND. Worth knowing, but it is a standing condition. A level-triggered alert would + fire every day forever, which is how an alert gets ignored, so it is edge-triggered.""" + from cotdata import vintage + body = _body(b"2025-final") + http = _FakeHttp({LEGACY_2025: [(200, body, None, "lm1"), (200, body, None, "lm2"), + (304, None, None, None), (304, None, None, None)]}) + now = _clock() + src = _one(LEGACY_2025, 2025) + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # first + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # deduped + r3 = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # goes quiet + r4 = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # still quiet + + assert r3["records"][0]["outcome"] == vintage.OUTCOME_NOT_MODIFIED + assert "blind" in r3["records"][0]["tripwire_alert"] + assert r4["records"][0]["tripwire_alert"] is None # edge, not level + + +def test_older_frozen_year_304_is_expected_and_silent(store_env): + """Outside the regeneration window a 304 is correct. Alerting on it would bury the + prior-year signal under ~35 years of noise on every --all sweep.""" + from cotdata import vintage + http = _FakeHttp({LEGACY_2019: [(200, _body(b"2019"), None, "lm1"), + (304, None, None, None)]}) + now = _clock() + src = _one(LEGACY_2019, 2019) + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + r2 = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + + assert r2["records"][0]["expectation"] == vintage.EXPECT_FROZEN_OUT_OF_WINDOW + assert r2["records"][0]["tripwire_alert"] is None + + +def test_current_year_new_bytes_never_alert(store_env): + """New data every week is the whole point of the current year.""" + from cotdata import vintage + http = _FakeHttp(_only(LEGACY_2026, (200, _body(b"w1"), None, "lm1"), + (200, _body(b"w2"), None, "lm2"))) + now = _clock() + _fetch(vintage, http, now) + r2 = _fetch(vintage, http, now) + assert r2["records"][0]["expectation"] == vintage.EXPECT_CHURN + assert r2["tripwire_alerts"] == [] + + +def test_default_fetch_includes_the_prior_year_so_the_tripwire_actually_runs(store_env): + """The tripwire is worth nothing if it only fires when someone remembers to run + --all by hand, so the prior year is in the DEFAULT scheduled set.""" + from cotdata import vintage + got = {(s.report_year, s.report_type) for s in _default_sources(vintage, 2026)} + assert (2026, "legacy") in got + assert (2025, "legacy") in got + assert (2024, "legacy") not in got + assert (None, "legacy") in got # the weekly static + + +def test_explicit_year_is_taken_at_face_value(store_env): + """A targeted capture is not the scheduled sweep: --year 2019 means 2019.""" + from cotdata import vintage + years = {s.report_year for s in _default_sources(vintage, 2026, year=2019)} + assert years == {2019, None} + + +def _default_sources(vintage, this_year, **kw): + """Capture the source list `fetch` derives, without any network at all.""" + seen = [] + + def spy(url, *, etag=None, last_modified=None): + raise AssertionError("no request expected") + + import datetime as dt + + def now(): + return dt.datetime(this_year, 7, 30, tzinfo=dt.timezone.utc) + + # fetch() records a failure rather than raising, so the spy's AssertionError still + # lets every derived source land in the returned records. + res = vintage.fetch(http_get=spy, rate_limit_s=0, now_fn=now, **kw) + for rec in res["records"]: + seen.append(vintage.Source(rec["report_type"], rec["source_kind"], "zip", + rec["source_url"], rec["report_year"])) + return seen diff --git a/tests/test_vintage_flow.py b/tests/test_vintage_flow.py new file mode 100644 index 0000000..0bb867c --- /dev/null +++ b/tests/test_vintage_flow.py @@ -0,0 +1,124 @@ +"""Flow decomposition (module spec §6.4) and the canonical-schema smoke test. + +Offline and synthetic, matching the repo's test idiom. The real-data sweep that these +encode is recorded in docs/design/cot_vintage.md §7: 95 markets, 149,412 weeks, every +one balanced. +""" +import pandas as pd +import pytest + + +def _rows(dates, *, category="noncommercial", longs=(), shorts=(), oi=None, + market="088691"): + oi = oi if oi is not None else [1_000_000] * len(dates) + return pd.DataFrame([{ + "report_date": pd.Timestamp(d), "market_code": market, "report_type": "legacy", + "combined": False, "category": category, + "long_contracts": lo, "short_contracts": sh, "open_interest": o, + } for d, lo, sh, o in zip(dates, longs, shorts, oi)]) + + +WEEKS = ["2026-01-06", "2026-01-13", "2026-01-20", "2026-01-27", "2026-02-03"] + + +def test_the_four_states_are_labelled_by_the_dominant_leg(): + """The spec's table is written with a '~0' leg, which never happens in real data. + Dominant-leg resolution is what makes it decidable, and it must pick the right one.""" + from cotdata import vintage_flow as vf + # start 100/100, then: +5000 long | -4000 short | +6000 short | -3000 long + can = _rows(WEEKS, + longs=[100, 5100, 5100, 5100, 2100], + shorts=[100, 100, -3900, 2100, 2100]) + fl = vf.decompose(can) + assert list(fl["state"]) == [vf.NEW_LONGS, vf.SHORT_COVERING, + vf.NEW_SHORTS, vf.LONG_LIQUIDATION] + + +def test_first_observation_of_a_series_is_dropped(): + """There is no such thing as the weekly change of the first week ever seen.""" + from cotdata import vintage_flow as vf + fl = vf.decompose(_rows(WEEKS, longs=[1, 2, 3, 4, 5], shorts=[1, 1, 1, 1, 1])) + assert len(fl) == len(WEEKS) - 1 + assert fl["report_date"].min() == pd.Timestamp(WEEKS[1]) + + +def test_series_do_not_leak_across_markets_or_categories(): + """A diff that crossed a group boundary would invent an enormous fake flow.""" + from cotdata import vintage_flow as vf + a = _rows(WEEKS[:2], longs=[10, 20], shorts=[10, 10], market="088691") + b = _rows(WEEKS[:2], longs=[9000, 9010], shorts=[10, 10], market="001602") + fl = vf.decompose(pd.concat([a, b], ignore_index=True)) + assert set(fl["d_long"]) == {10.0} # not 8980, which is the cross-market diff + + +def test_duplicate_vintages_for_one_week_are_refused(): + """Two rows for one (key, date) means two VINTAGES. Diffing them would report a + revision as though it were a flow, which is the exact confusion the vintage layer + exists to prevent.""" + from cotdata import vintage_flow as vf + can = pd.concat([_rows(WEEKS[:2], longs=[10, 20], shorts=[10, 10]), + _rows(WEEKS[1:2], longs=[25], shorts=[10])], ignore_index=True) + with pytest.raises(vf.FlowError, match="duplicate"): + vf.decompose(can) + + +def test_days_elapsed_exposes_a_gap_rather_than_assuming_seven(): + """COT was FORTNIGHTLY until 1992-10-13, and holiday weeks shift the rest. A caller + comparing flow magnitudes must be able to see that a 'weekly' change spans 15 days.""" + from cotdata import vintage_flow as vf + can = _rows(["1992-01-15", "1992-01-31", "1992-02-14"], + longs=[100, 200, 300], shorts=[100, 100, 100]) + fl = vf.decompose(can) + assert list(fl["days_elapsed"]) == [16, 14] + + +def test_dead_zone_is_off_by_default_and_scales_with_prior_open_interest(): + from cotdata import vintage_flow as vf + can = _rows(WEEKS[:2], longs=[1000, 1005], shorts=[1000, 1000], oi=[100_000, 100_000]) + assert vf.decompose(can)["state"].iloc[0] == vf.NEW_LONGS # 5 contracts, but no + # dead zone by default + quiet = vf.decompose(can, min_frac_oi=0.001) # 0.1% of 100k = 100 + assert quiet["state"].iloc[0] == vf.QUIET + + +def test_open_interest_corroborates_or_contradicts_the_label(): + """New positioning should create contracts; exits should destroy them. When it does + not, the label describes a transfer between categories, not new or closed risk.""" + from cotdata import vintage_flow as vf + rising = _rows(WEEKS[:2], longs=[100, 5100], shorts=[100, 100], oi=[1000, 9000]) + falling = _rows(WEEKS[:2], longs=[100, 5100], shorts=[100, 100], oi=[9000, 1000]) + assert vf.decompose(rising)["oi_corroborates"].iloc[0] is True + assert vf.decompose(falling)["oi_corroborates"].iloc[0] is False + + +def test_zero_sum_identity_holds_and_the_gap_is_spreading(): + """Every long is somebody's short, so the side totals must match. They fall short of + open interest by the non-commercial SPREADING column, which providers/cftc.py does + not capture: an expected, measurable gap, not a break.""" + from cotdata import vintage_flow as vf + can = pd.concat([ + _rows(WEEKS[:1], category="noncommercial", longs=[600], shorts=[100], oi=[1000]), + _rows(WEEKS[:1], category="commercial", longs=[300], shorts=[700], oi=[1000]), + _rows(WEEKS[:1], category="nonreportable", longs=[50], shorts=[150], oi=[1000]), + ], ignore_index=True) + z = vf.zero_sum_check(can) + assert len(z) == 1 + assert bool(z["balanced"].iloc[0]) is True # 950 long == 950 short + assert int(z["oi_gap"].iloc[0]) == 50 # the uncaptured spreading + + +def test_zero_sum_check_catches_a_broken_category_mapping(): + from cotdata import vintage_flow as vf + can = pd.concat([ + _rows(WEEKS[:1], category="noncommercial", longs=[600], shorts=[100], oi=[1000]), + _rows(WEEKS[:1], category="commercial", longs=[300], shorts=[700], oi=[1000]), + _rows(WEEKS[:1], category="nonreportable", longs=[50], shorts=[999], oi=[1000]), + ], ignore_index=True) + assert bool(vf.zero_sum_check(can)["balanced"].iloc[0]) is False + + +def test_missing_columns_raise_rather_than_silently_producing_nothing(): + from cotdata import vintage_flow as vf + can = _rows(WEEKS[:2], longs=[1, 2], shorts=[1, 1]).drop(columns=["short_contracts"]) + with pytest.raises(vf.FlowError, match="missing columns"): + vf.decompose(can) From 38d6f9099e4f9e262ff37577acd054a6f9e107f4 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 22:41:08 -0400 Subject: [PATCH 2/7] docs: propose the step 2 approach (contract master + normalisation) Proposal only, no step-2 code. Three measurements against the real store change what step 2 can be: 1. Step 2 does not belong in cotdata. Normalisation is by definition a joiner (COT x prices x contract specs), and ADR-0007's whole argument was that almost nothing joins the domains. It belongs in the consumer, which means creating crowdmon-futures as a sibling. 2. The registry declares 41 disagg + 8 TFF and ZERO legacy symbols, and the module spec's engines all key on Managed Money / Leveraged Funds, which exist only in those two reports. Vintage ingest wires Legacy only, so a point-in-time Managed Money series does not exist yet. That is the real step-2 prerequisite, and it is a cotdata change. 3. Notional off back-adjusted prices is wrong by +294% (gold 2002) and +257% (crude 2004), and crude's back-adjusted series reaches -27.52, which is not a price. The error is EXACTLY ZERO at the present date and grows monotonically backwards, so it passes every spot check anyone would run while corrupting the whole evaluation history. Volatility must come from back-adjusted returns, so the two factors of net_notional x sigma come from two different price series. Also measured: all 47 contract specs are USD (no FX layer needed), unadjusted prices exist for all 47, and the COT-to-specs-to-price join covers 42 of 95 Legacy markets (44%). Co-Authored-By: Claude Opus 5 --- docs/design/crowdmon_step2_normalisation.md | 207 ++++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 docs/design/crowdmon_step2_normalisation.md diff --git a/docs/design/crowdmon_step2_normalisation.md b/docs/design/crowdmon_step2_normalisation.md new file mode 100644 index 0000000..c6bee28 --- /dev/null +++ b/docs/design/crowdmon_step2_normalisation.md @@ -0,0 +1,207 @@ +# Step 2: contract master and normalisation — PROPOSAL + +**Status: proposed, not accepted.** Written before any step-2 code, per the request to +propose the approach first. Module spec: [crowdmon_futures_cot_module.md](crowdmon_futures_cot_module.md) +§5.1, §5.2, §13 step 2. Everything measured below was measured against the real store on +2026-07-30, and several measurements change what step 2 can be. + +--- + +## Headline: step 2 should not be built in `cotdata` + +Step 1 (vintage) belonged here, and [ADR-0008](https://github.com/mspinola/crucible-stack/blob/main/docs/adr/ADR-0008-cot-vintage-provenance-in-parquet.md) +argued why: it is CFTC-positioning provenance, it sits beside the fetch it records, and it +never crosses the instrument-domain axis [ADR-0007](https://github.com/mspinola/crucible-stack/blob/main/docs/adr/ADR-0007-cotdata-is-cot-only-bars-live-in-marketdata.md) +draws. Flow decomposition (§6.4, just built) is the same shape: COT in, COT out. + +**Normalisation is not that shape. It is by definition a joiner**: net contracts times a +multiplier times a price, then scaled by a volatility estimated from returns. It needs COT +*and* bars *and* contract specs in one function. + +ADR-0007's central measurement was that almost nothing joins the two domains, exactly two +files, which is what made the split cheap. A normalisation layer inside `cotdata` would +drag bars back in at precisely the moment ADR-0007 is pushing them out. Inside `marketdata` +it would drag COT in, and `marketdata` deliberately imports nothing from `cotdata`. The +only home that does not violate the seam is **the consumer**, which is the crowdmon package +the module spec already describes in §12 and which does not exist yet. + +So step 2 begins by creating `crowdmon-futures` as a workspace sibling, with a boundary +test in the shape of `crucible-stack/tests/test_boundaries.py`: crowdmon may import +`cotdata` and `marketdata`; neither may import crowdmon. + +This is a recommendation, not a fait accompli. The alternative is prototyping inside `npf`, +which is faster to start and worse to unpick later, since npf is the strategy repo and this +is infrastructure. + +--- + +## What is actually available (measured 2026-07-30) + +| Input | Present? | Detail | +|---|---|---| +| Multiplier | **yes** | `metadata/contract_specs.parquet`, `Point Value`, 47 rows | +| Tick size / value | yes | same table | +| Currency | **yes, and all 47 are USD** | so no FX conversion layer is needed at all | +| Exchange margin | yes | `Margin` column, so §8's margin sensitivity is buildable | +| Unadjusted prices | **yes** | `prices/_unadj.parquet`, 47 symbols | +| Back-adjusted prices | yes | `prices/_backadj.parquet`, 47 symbols | +| Roll calendar | **no** | not in the specs table, and there is no per-expiry source | +| First notice date | **no** | same | +| Daily price limits | **no** | spec §3 says "manually maintained", and nothing maintains it | + +The all-USD result is a genuine simplification worth banking: §5.2 rung 3 says "net notional +USD", and for this universe that is a multiplication with no FX join. + +### Coverage ceiling: 42 of 95 markets + +| Set | Count | +|---|---| +| Legacy COT market codes in the store | 95 | +| Registry symbols | 49 | +| Registry symbols with contract specs | 47 | +| Registry symbols with unadjusted prices | 47 (`MME`, `MFS` have none) | +| **COT markets joinable to both specs and price** | **42** | + +This is not a bug, it is the registry universe: `cotdata.registry` only names instruments +somebody wanted prices for. But it means **any "cross-market" result from step 2 onward +covers 44% of the CFTC market list**, and §7's PCA and trend-alignment panels inherit that. +State it in output rather than letting a reader assume 95. + +--- + +## Three findings that change the plan + +### 1. The categories the spec cares about are not in the report that is wired + +The registry declares **41 disaggregated and 8 TFF symbols, and zero legacy**. That matches +the module spec: the positioning engine, the fragility weights (§6.3), the cross-market +panel (§7) and the CTA calibration (§9.2) all key on **Managed Money** and **Leveraged +Funds**, which exist only in the Disaggregated and TFF reports. Legacy's `noncommercial` is +the pre-2006 proxy for them, not the same category. + +The vintage layer has a canonicaliser for **Legacy only**. Disagg and TFF are captured as +raw bytes and drain as `skipped`. + +So: **a point-in-time Managed Money series does not exist today, and step 2 has nothing +correct to normalise.** This is the real step-2 prerequisite, and it is a `cotdata` change +rather than a crowdmon one. The change-only and revision machinery is already report-type +agnostic, the controlled vocabularies are already declared, and the raw bytes are already +retained, so this is writing two canonicalisers and their tests, not new architecture. + +**Recommendation: do the disagg and TFF canonicalisers before step 2, in `cotdata`.** + +### 2. Notional must use unadjusted prices, and getting it wrong is invisible today + +§5.1 says the unadjusted series is "retained separately for notional and margin +calculations". That is right, and the size of the error is worth pinning down, because it +has a property that makes it dangerous: + +| Market | Date | Back-adjusted close | Unadjusted close | Notional error | +|---|---|---|---|---| +| GC | 2002-05-30 | 1282.00 | 325.50 | **+294%** | +| CL | 2004-12-13 | 146.48 | 41.01 | **+257%** | +| ZC | 2002-04-10 | 587.50 | 199.75 | **+194%** | +| GC | 2026-07-30 | 4100.10 | 4100.10 | +0.0% | + +**The error is exactly zero at the present date and grows monotonically backwards**, because +back-adjustment anchors on the most recent contract. So a notional computed from the +back-adjusted series passes every spot check anyone would actually run, and silently +corrupts the entire history a backtest is evaluated over. Crude's back-adjusted series even +reaches **-27.52**, which is not a price. + +Meanwhile volatility must come from the **back-adjusted** series, because that is the one +with correct returns; unadjusted returns carry fake roll gaps. + +So `net_notional × σ_daily` draws its two factors from two different price series. That is +not an implementation detail to discover in review, it is the central correctness fact of +step 2, and it should be pinned by a test that fails if either leg is swapped. + +### 3. The COT side now has point-in-time discipline; the price side does not + +`livebook/docs/OPERATIONS.md` records that Norgate back-adjusted series **restate history on +every roll**. `marketdata` captures no vintages and has no `asof`. + +So a release-date-indexed, vol-scaled notional series is point-in-time on its COT leg and +current-state on its price leg. That asymmetry is not fixable inside step 2, and it is not +a reason to stop, but it should be stated in output (a `pit_complete: False` flag or +equivalent) rather than left for someone to infer. §5.3's warning about `derived` release +dates has the same character: a provenance flag is worth more than a clean-looking number. + +--- + +## Proposed build + +``` +crowdmon_futures/ + ingest/cot_adapter.py CotSource over cotdata.vintage_ingest.asof; release_date + indexed, `derived` excludable, zero_sum_check on every load + normalize/ + contract_master.py registry x contract_specs, and a COVERAGE REPORT as its + first-class output rather than a silent inner join + notional.py net_contracts x Point Value x UNADJUSTED close + riskunits.py x sigma_daily from BACK-ADJUSTED returns + tests/ + test_boundaries.py crowdmon imports cotdata/marketdata; neither imports crowdmon + test_price_series_split.py fails if notional uses backadj or sigma uses unadj +``` + +### The normalisation ladder, rung by rung + +| Rung (spec §5.2) | Additional input | Status | +|---|---|---| +| 1. net contracts | COT only | available. Spec says do not report it, and it is right | +| 2. net / open interest | COT only | available, but the denominator is missing spreading (see below) | +| 3. net notional USD | Point Value + unadjusted price | available, 42 markets | +| 4. vol-scaled notional | σ from back-adjusted returns | available, 42 markets. The default for every cross-market comparison | + +Rung 2 carries a defect found while building §6.4: `NonComm_Positions_Spread_All` is not +captured by `providers/cftc.py`, so open interest includes spreading contracts the numerator +cannot see (gold: 8% of OI). Rung 2 is therefore biased low by a market-and-time-varying +amount. Rungs 3 and 4 are unaffected, since they never divide by OI. **Either fix the +capture first or do not use rung 2**, and the module spec's preference for rung 4 as the +default makes the second option cheap. + +### Explicitly out of step 2, and why + +- **Roll calendar, first notice date, roll congestion (§8).** No per-expiry price source + exists in the stack and none is being built (workspace CLAUDE.md, ADR-0007 step 2 on ice). + Blocked on data, not code. +- **Daily price limits (§8).** Would be a hand-maintained table. Worth doing eventually; + not on step 2's critical path. +- **Seasonal adjustment (§5.4).** The spec puts it at step 7. It is tempting to pull + forward because ag commercial z-scores are visibly seasonal, but it is a modelling choice + and step 2 is meant to be units. + +--- + +## Recommended sequencing + +| # | Work | Where | Blocking? | +|---|---|---|---| +| 0 | Disaggregated + TFF vintage canonicalisers | `cotdata` | **yes**, step 2 has nothing correct to normalise without them | +| 1 | `crowdmon-futures` skeleton + boundary test | new sibling | yes | +| 2 | `contract_master.py`, coverage report as its first output | crowdmon | | +| 3 | `notional.py`, with the unadj/backadj split pinned by test | crowdmon | | +| 4 | `riskunits.py` (vol-scaled) | crowdmon | | +| 5 | Fix spreading capture, if rung 2 is wanted | `cotdata` | no, rung 4 does not need it | + +## Three decisions this needs before code + +1. **Create `crowdmon-futures` as a sibling now, or prototype inside `npf`?** Recommend the + sibling, on the ADR-0007 seam argument above. +2. **Do the disagg/TFF canonicalisers first?** Recommend yes. Without them step 2 normalises + Legacy `noncommercial`, which is not the category any downstream engine wants. +3. **Accept the 42-of-95 coverage ceiling, or widen the registry first?** Recommend accept + and report it, since widening it means sourcing prices for 53 more markets and that is a + `marketdata` project, not a crowdmon one. + +## Bottom line + +Step 2 is the first build step that crosses the seam between COT and prices, so it belongs +in a new consumer package rather than in `cotdata`. Most of what it needs is already in the +store and in USD, which is better than expected. Two things are not: the Disaggregated and +TFF reports have no vintage canonicaliser, so the Managed Money and Leveraged Funds series +the whole system is built around cannot yet be read point-in-time, and that is a genuine +prerequisite rather than a detail. And the single correctness trap, using back-adjusted +prices for notional, is one whose error is exactly zero today and reaches nearly 300% in +2002, so it will not be caught by looking at the output. From 8ecedc0d0ff369a4751b213b6cbd892087ebc489 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 22:41:18 -0400 Subject: [PATCH 3/7] docs: drop the stray em dash from the step 2 proposal title --- docs/design/crowdmon_step2_normalisation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/crowdmon_step2_normalisation.md b/docs/design/crowdmon_step2_normalisation.md index c6bee28..ac58992 100644 --- a/docs/design/crowdmon_step2_normalisation.md +++ b/docs/design/crowdmon_step2_normalisation.md @@ -1,4 +1,4 @@ -# Step 2: contract master and normalisation — PROPOSAL +# Step 2: contract master and normalisation: PROPOSAL **Status: proposed, not accepted.** Written before any step-2 code, per the request to propose the approach first. Module spec: [crowdmon_futures_cot_module.md](crowdmon_futures_cot_module.md) From b1ade423a4b79f43745956a693891cecdbef4a85 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 22:57:11 -0400 Subject: [PATCH 4/7] feat(vintage): disaggregated and TFF canonicalisers The step 2 proposal identified this as the real prerequisite. Every engine in the module spec keys on Managed Money and Leveraged Funds, which exist only in these two reports, and ingest wired Legacy only. asof() now returns both point-in-time. Verified against the REAL first production capture (2026-07-31 01:15Z), not fixtures: 39,235 Disaggregated and 12,500 TFF canonical rows for 2026, all three reports ingesting in about 5 seconds, and re-ingest writing zero rows. These two reports populate three fields Legacy never does: per-category spreading, per-category trader counts, and CR4/CR8 net concentration. So the zero-sum identity closes COMPLETELY for Disaggregated (7,847/7,847 weeks, oi_gap zero everywhere), which is the confirming counterpart to the Legacy finding that its gap really is the uncaptured spreading column. ROUNDING TOLERANCE. The residual imbalance is CFTC's own and fully localised: every off-by-one-or-two row in both Legacy and TFF falls in exactly three markets, S&P 500 / NASDAQ-100 / DJIA Consolidated, whose codes carry CFTC's own '+' suffix for a unit-converted aggregate. Tolerance is derived from that mechanism rather than fitted: summing n independently rounded category figures admits at most n contracts of error. Without it, 48 off-by-one warnings fire on the 2026 files alone, the same cry-wolf rate the OI check was corrected for in the previous commit. Implementation points that will otherwise cost someone an afternoon: - CFTC's header has a typo. Swap__Positions_Short_All and _Spread_All carry a double underscore, _Long_All does not. Both spellings resolve, so the day CFTC fixes it is not the day Swap Dealer ingests as nulls. - An unresolvable column RAISES. Silent nulls would be written as real observations, and the next genuine value recorded as a revision that never happened, polluting the exact artifact this subsystem produces. - Trader counts arrive as strings because CFTC writes '.' for a suppressed count (3,578 of 7,847 Managed Money long counts on the 2026 file). Every value field is coerced so a marker never reaches row_sha256 as a literal string. - combined is now READ from the file's FutOnly_or_Combined column, and a file mixing the two is refused. Constant-FutOnly today, so nothing changes, but adding the combined files becomes a fetch-list change. - canonicalize_legacy is deliberately NOT routed through the new shared helper. Its output feeds row_sha256 over rows already stored in production; the duplication is cheaper than risking a mass revision. Also fixes a real cross-platform bug found in the production snapshot index: local_path is written by the Windows producer with backslashes, so ingest on either replica failed with "no such file" on a file plainly present. Normalised on read, so snapshots already recorded stay readable. Co-Authored-By: Claude Opus 5 --- README.md | 5 + docs/design/cot_vintage.md | 96 +++++++++ docs/design/crowdmon_futures_cot_module.md | 15 +- docs/design/crowdmon_step2_normalisation.md | 10 +- src/cotdata/vintage_cli.py | 61 ++++-- src/cotdata/vintage_flow.py | 57 ++++-- src/cotdata/vintage_ingest.py | 201 ++++++++++++++++++- tests/test_vintage_canonicalize.py | 205 ++++++++++++++++++++ tests/test_vintage_ingest.py | 26 ++- 9 files changed, 629 insertions(+), 47 deletions(-) create mode 100644 tests/test_vintage_canonicalize.py diff --git a/README.md b/README.md index ca8d2cd..249d3f3 100644 --- a/README.md +++ b/README.md @@ -353,6 +353,11 @@ How it works: - **Immutable landing zone.** Every fetch is recorded (including 304s) and raw bytes are retained permanently under `vintage/raw/`, written atomically and never rewritten. A byte-identical regeneration is deduped — a changed download is not itself a revision. +- **All three reports.** Legacy, Disaggregated and TFF canonicalise into one long schema. + Disagg and TFF also populate per-category spreading, per-category trader counts and + CR4/CR8 concentration, none of which the Legacy file carries, and they are where + **Managed Money** and **Leveraged Funds** live. A suppressed trader count (CFTC writes + `.`) canonicalises to null rather than to a string. - **Change-only observations.** A row is written only when its value hash differs from the latest for its natural key `(report_date, market_code, report_type, combined, category)`, so storage grows with actual revisions rather than with time. diff --git a/docs/design/cot_vintage.md b/docs/design/cot_vintage.md index d61d43a..99f0465 100644 --- a/docs/design/cot_vintage.md +++ b/docs/design/cot_vintage.md @@ -388,6 +388,102 @@ comparable to the rest of the series. The remaining off-7 intervals are holiday (gold has 11-day gaps at 2002-01-08 and 2003-02-25). The column is the fix: callers filter on it rather than discovering this in a result. +## 8. Disaggregated and TFF canonicalisers (2026-07-30) + +The step-2 proposal identified this as the real prerequisite, not step 2 itself: the +registry declares 41 disaggregated and 8 TFF symbols and zero legacy, and every engine in +the module spec keys on **Managed Money** and **Leveraged Funds**, which exist only in +those two reports. Ingest wired Legacy only, so no point-in-time series existed for the +categories the whole system is built around. Now it does. + +Both were built and verified against the **real captured snapshots** from the first +production capture (2026-07-31 01:15Z), not fixtures. + +| Report | Canonical rows for 2026 | Weeks | Categories | +|---|---|---|---| +| Legacy | 31,041 | 10,347 | commercial, noncommercial, nonreportable | +| Disaggregated | 39,235 | 7,847 | producer_merchant, swap, managed_money, other_reportable, nonreportable | +| TFF | 12,500 | 2,500 | dealer, asset_manager, leveraged, other_reportable, nonreportable | + +Ingest of all three takes about 5 seconds. 418 distinct market codes, which is far more +than the 95 in the current-state Legacy store, because the vintage layer canonicalises +everything CFTC published rather than the registry universe. That is deliberate: capture +everything, filter on read. + +### These two reports populate three fields Legacy never does + +- **Per-category spreading.** So the identity closes completely, which the Legacy defect + (§7 finding 1) prevents there. +- **Per-category trader counts.** §6.2's breadth-depth quadrant needs these and cannot be + built from Legacy. +- **CR4 / CR8 net concentration**, per market, repeated on each category row exactly as + open interest is. Only the net ratios have canonical columns; the gross ones are + published too and have nowhere to land. + +### The zero-sum identity by report, measured + +| Report | Exact | Within tolerance | `oi_gap` | +|---|---|---|---| +| Legacy | 10,321 / 10,347 | **10,347 / 10,347** | never zero (the uncaptured spreading) | +| Disaggregated | **7,847 / 7,847** | 7,847 / 7,847 | **zero everywhere** | +| TFF | 2,463 / 2,500 | **2,500 / 2,500** | zero on 98.6% | + +Disaggregated closing exactly, with a zero gap, is the confirming counterpart to the Legacy +finding: the gap there really is the missing spreading column and nothing else. + +**The residual is CFTC's own rounding, and it is fully localised.** Every off-by-one-or-two +row in *both* Legacy and TFF falls in exactly three markets: + +| Market code | Name | Legacy rows | TFF rows | Worst | +|---|---|---|---|---| +| `13874+` | S&P 500 Consolidated | 12 | 17 | 2 | +| `20974+` | NASDAQ-100 Consolidated | 11 | 15 | 1 | +| `12460+` | DJIA Consolidated | 3 | 5 | 1 | + +The `+` suffix is CFTC's own marker for a Consolidated contract, which aggregates several +contract sizes onto a common unit and therefore involves a division. So the tolerance is +derived from the mechanism rather than fitted: summing `n` independently rounded category +figures admits at most `n` contracts of error, which is what `rounding_tolerance()` +returns. Without it, 48 off-by-one warnings fire on the 2026 files alone, which is the +same cry-wolf rate §7 finding 2 was corrected for. + +### Three implementation points worth knowing before debugging them + +1. **CFTC's header row has a typo, and it is load-bearing.** + `Swap__Positions_Short_All` and `Swap__Positions_Spread_All` have a double underscore; + `Swap_Positions_Long_All` has one. Both spellings resolve, so the day CFTC fixes it is + not the day Swap Dealer positions start ingesting as nulls. +2. **A column that cannot be resolved raises.** Silently returning nulls is the worst + available outcome: they get written as real observations, and the next genuine value is + then recorded as a revision that never happened, permanently polluting the revision + history this subsystem exists to produce. +3. **Trader counts arrive as strings**, because CFTC writes `.` for a suppressed count. + Every value field is coerced with `errors="coerce"` so a suppression marker or a stray + pad can never reach `row_sha256` as a literal string. + +### `combined` is now read from the file + +Both reports carry a `FutOnly_or_Combined` column. It is read rather than hardcoded, and a +file mixing the two is refused. Today it is constant-`FutOnly`, so nothing changes, but +adding the combined files becomes purely a fetch-list change with the canonicaliser already +correct. §3c's carry-forward stands; the code no longer assumes it. + +### `canonicalize_legacy` was deliberately left alone + +The new code shares one vectorised helper; Legacy still uses its original per-row loop. +That duplication is intentional. Legacy's output feeds `row_sha256`, which is a permanent +artifact over rows already stored in production, and rewriting the code path that produced +those hashes to save a little duplication risks registering every stored row as revised at +once. The saving was not worth the risk. + +### Fixed in passing: raw paths were unreadable off the producer + +`snapshots.json` records `local_path` as written by the capturing machine, and the producer +is Windows, so the real store carries `vintage\raw\annual_zip\2026\....zip`. On macOS or +Linux that is a single filename containing backslashes, so ingest on either replica failed +with "no such file" on a file plainly sitting there. Normalised on read rather than on +write, so every snapshot already recorded on the producer stays readable. + ## Bottom line Discovery and the spike are done. The genuine null on historical recovery holds: no diff --git a/docs/design/crowdmon_futures_cot_module.md b/docs/design/crowdmon_futures_cot_module.md index 076f8d4..b13799d 100644 --- a/docs/design/crowdmon_futures_cot_module.md +++ b/docs/design/crowdmon_futures_cot_module.md @@ -199,15 +199,24 @@ Two implementations: `LocalCotData` wrapping the existing module, and `CftcApiCo > - **Only futures-only is fetched**, so `combined` is constant-`False` today: the column is > present and correct but not yet discriminating, and half the reportable universe is absent > until the combined files are added. -> - **`spread_contracts` is always null, and `NonComm_Positions_Spread_All` is not +> - **On LEGACY only, `spread_contracts` is null and `NonComm_Positions_Spread_All` is not > captured at all.** It is absent from `providers/cftc.py`'s `TARGET_COLS`, so it never > reaches the stored parquet, the canonical rows, or any vintage observation. Measured as > the exact, equal gap between each side total and open interest: gold on 2026-07-21 had > OI 383,368 against 351,385 on both sides, a gap of 31,983 (8% of OI). Net positioning > is unaffected, since spreading is long and short in equal measure, but **anything > denominated as a share of open interest (§5.2 step 2) has a denominator containing -> contracts its numerator cannot see.** Trader counts are likewise missing from the -> stored table, though the vintage ingest path reads them straight from the zip. +> contracts its numerator cannot see.** +> **Disaggregated and TFF do not have this defect** (canonicalisers added 2026-07-30): +> both publish spreading per category, and the identity closes exactly, 7,847 of 7,847 +> Disaggregated weeks with an `oi_gap` of zero. Since the categories this document +> actually uses are Disaggregated and TFF categories, §5.2 rung 2 is sound for them and +> compromised only for Legacy. +> - **Trader counts and CR4/CR8 are populated for Disaggregated and TFF**, per category and +> per market respectively, which is what §6.2's breadth-depth quadrant needs. They are +> absent on Legacy. Trader counts carry `.` for a suppressed value and canonicalise to +> null: on the 2026 Disaggregated file, 3,578 of 7,847 Managed Money long counts are +> suppressed, so "null" is a routine and meaningful state, not a data error. > - **`vintage: int` is not how it was built.** The implementation is bitemporal > (`observed_at` plus change-only rows), so a point-in-time read is "greatest > `observed_at <= t` per natural key". An integer vintage ordinal can be derived from that diff --git a/docs/design/crowdmon_step2_normalisation.md b/docs/design/crowdmon_step2_normalisation.md index ac58992..ff86016 100644 --- a/docs/design/crowdmon_step2_normalisation.md +++ b/docs/design/crowdmon_step2_normalisation.md @@ -90,6 +90,14 @@ retained, so this is writing two canonicalisers and their tests, not new archite **Recommendation: do the disagg and TFF canonicalisers before step 2, in `cotdata`.** +> **Done, 2026-07-30.** Both canonicalisers shipped and were verified against the real +> first production capture: 39,235 Disaggregated and 12,500 TFF canonical rows for 2026, +> ingesting in about 5 seconds. `asof()` now returns Managed Money and Leveraged Funds +> point-in-time, with per-category spreading, per-category trader counts and CR4/CR8, none +> of which Legacy carries. Detail and the measured zero-sum results are in +> [cot_vintage.md](cot_vintage.md) §8. **This section's blocker is cleared**; the rest of +> the proposal stands as written. + ### 2. Notional must use unadjusted prices, and getting it wrong is invisible today §5.1 says the unadjusted series is "retained separately for notional and margin @@ -178,7 +186,7 @@ default makes the second option cheap. | # | Work | Where | Blocking? | |---|---|---|---| -| 0 | Disaggregated + TFF vintage canonicalisers | `cotdata` | **yes**, step 2 has nothing correct to normalise without them | +| 0 | ~~Disaggregated + TFF vintage canonicalisers~~ | `cotdata` | **done 2026-07-30** | | 1 | `crowdmon-futures` skeleton + boundary test | new sibling | yes | | 2 | `contract_master.py`, coverage report as its first output | crowdmon | | | 3 | `notional.py`, with the unadj/backadj split pinned by test | crowdmon | | diff --git a/src/cotdata/vintage_cli.py b/src/cotdata/vintage_cli.py index 01bb0dd..06b20be 100644 --- a/src/cotdata/vintage_cli.py +++ b/src/cotdata/vintage_cli.py @@ -37,39 +37,68 @@ def _cmd_fetch(args) -> int: return 0 -def _cmd_ingest(args) -> int: +def _snapshot_path(local_path: str): + """Resolve a recorded raw-file path on ANY platform. + + ``local_path`` is stored relative to the store root, but it is written by whichever + machine captured it, and the producer is Windows: the real store carries + ``vintage\\raw\\annual_zip\\2026\\....zip``. Passed to ``Path`` on macOS or Linux that + is a single filename containing backslashes, which does not exist, so ingest on a + replica fails with a confusing "no such file" on a file that is plainly there. + Normalising here rather than at write time keeps every snapshot already recorded on + the Windows producer readable. + """ from pathlib import Path + return config.store_root().joinpath(*str(local_path).replace("\\", "/").split("/")) \ + if local_path else Path() + + +# Each report type maps to (parser, canonicaliser). Legacy is indexed by report date +# because that is the shape ``providers/cftc.py`` emits; disagg and TFF keep it as a +# column, and the canonicalisers accept either. +def _canonicalisers(): + from . import vintage_ingest + from .providers import cftc, cftc_disagg, cftc_tff + def legacy(path): + wide = cftc._parse_zip(path).set_index(cftc.REPORT_DATE) + return vintage_ingest.canonicalize_legacy(wide) + + return { + "legacy": legacy, + "disaggregated": lambda p: vintage_ingest.canonicalize_disagg(cftc_disagg._parse_zip(p)), + "tff": lambda p: vintage_ingest.canonicalize_tff(cftc_tff._parse_zip(p)), + } + + +def _cmd_ingest(args) -> int: from . import vintage, vintage_ingest - from .providers import cftc snaps = vintage.read_snapshots() if args.snapshot: snaps = [s for s in snaps if s.get("snapshot_id") == args.snapshot] elif args.pending: snaps = [s for s in snaps if s.get("parse_status") == "pending"] - # Only the Legacy annual zip is wired end-to-end this pass; disagg/TFF canonicalizers - # are a follow-on. Skip (don't fail) snapshots we can't yet parse. + canon = _canonicalisers() total_obs = total_rev = 0 for s in snaps: - if s.get("report_type") != "legacy" or s.get("source_kind") != "annual_zip": - # No canonicaliser for this report type yet. Mark it SKIPPED rather than - # leaving it pending: a snapshot that never drains is re-selected by - # --pending on every future run, and if it ever carried restatement_suspect - # the alert below would then re-fire forever, which is how an alert gets - # ignored. Skipped snapshots surface once and then go quiet. - # Raw bytes are retained, so adding a canonicaliser later just means - # re-marking these pending and re-running ingest — nothing is lost. + fn = canon.get(s.get("report_type")) + if fn is None or s.get("source_kind") != "annual_zip": + # No canonicaliser for this source. Today that is only the weekly static, + # whose 129 positional columns are a genuinely larger job than the annual + # zips. Mark it SKIPPED rather than leaving it pending: a snapshot that never + # drains is re-selected by --pending on every future run, and if it ever + # carried restatement_suspect the alert below would re-fire forever, which is + # how an alert gets ignored. Raw bytes are retained, so adding a canonicaliser + # later just means re-marking these pending and re-running: nothing is lost. vintage.update_snapshot( s["snapshot_id"], parse_status="skipped", parse_error=f"no canonicaliser for {s.get('report_type')}/" f"{s.get('source_kind')} yet") continue - path = config.store_root() / s["local_path"] + path = _snapshot_path(s["local_path"]) try: - wide = cftc._parse_zip(Path(path)) - wide = wide.set_index(cftc.REPORT_DATE) - canonical = vintage_ingest.canonicalize_legacy(wide) + canonical = fn(path) res = vintage_ingest.ingest_canonical(canonical, snapshot_id=s["snapshot_id"]) vintage.update_snapshot(s["snapshot_id"], parse_status="ok", parse_error=None) total_obs += res["observations"] diff --git a/src/cotdata/vintage_flow.py b/src/cotdata/vintage_flow.py index 5f1ce04..b3bf3c7 100644 --- a/src/cotdata/vintage_flow.py +++ b/src/cotdata/vintage_flow.py @@ -168,30 +168,57 @@ def zero_sum_check(canonical: pd.DataFrame) -> pd.DataFrame: every category must equal the same sum of ``short_contracts``. If the category mapping dropped, duplicated or misrouted a column, that identity breaks immediately. - ``oi_gap`` is what the sides fall short of open interest by. It is expected to be - NON-ZERO and EQUAL on both sides for the Legacy report as stored today: the - non-commercial SPREADING column is not in ``providers/cftc.py``'s ``TARGET_COLS``, so - it never reaches the canonical rows. Spreading is a matched long and short held by one - trader, which is exactly why it cancels out of the long-versus-short identity while - still counting toward open interest. So ``long_total == short_total`` is the assertion, - and ``oi_gap`` is a measurement to read, not a failure, until spreading is captured, - at which point it should go to zero. + Spreading is a matched long and short held by one trader, so it is added to BOTH side + totals: it cancels out of the long-versus-short identity but still counts toward open + interest. ``oi_gap`` is therefore what the sides fall short of open interest by, and + what it should be depends on the report: + + | Report | Expected ``balanced`` | Expected ``oi_gap`` | Measured 2026 | + |---|---|---|---| + | Legacy | always | **non-zero**, equals the uncaptured non-commercial spreading | 149,412/149,412 balanced, gap never zero | + | Disaggregated | always | **zero**, spreading is captured per category | 7,847/7,847 balanced, gap 0 everywhere | + | TFF | to within a contract or two | zero, same reason | 2,463/2,500 exact, worst case 2 contracts | + + The Legacy gap is the known defect: ``NonComm_Positions_Spread_All`` is not in + ``providers/cftc.py``'s ``TARGET_COLS``, so it never reaches the canonical rows. It is + a measurement to read, not a failure. + + The TFF residual is CFTC's own rounding, not a mapping error: every one of the 37 + off-by-one-or-two rows falls in the three "Consolidated" equity index markets (S&P 500, + NASDAQ-100, DJIA), which aggregate several contract sizes into a common unit and so + involve a division. Treat a TFF imbalance of 1 or 2 contracts in a Consolidated market + as expected, and anything larger or anywhere else as a real break. Returns one row per (report_date, market_code, report_type, combined) with - ``long_total``, ``short_total``, ``open_interest``, ``oi_gap`` and ``balanced``. + ``long_total``, ``short_total``, ``spread_total``, ``open_interest``, ``oi_gap``, + ``imbalance`` and ``balanced``. """ keys = ["report_date", "market_code", "report_type", "combined"] missing = [c for c in keys + ["long_contracts", "short_contracts"] if c not in canonical.columns] if missing: raise FlowError(f"missing columns for zero-sum check: {missing}") - g = canonical.groupby(keys, dropna=False, sort=False) - out = g.agg(long_total=("long_contracts", "sum"), - short_total=("short_contracts", "sum"), - open_interest=("open_interest", "max")).reset_index() - out["balanced"] = out["long_total"] == out["short_total"] + df = canonical.copy() + if "spread_contracts" not in df.columns: + df["spread_contracts"] = pd.NA + for c in ("long_contracts", "short_contracts", "spread_contracts", "open_interest"): + df[c] = pd.to_numeric(df.get(c), errors="coerce") + g = df.groupby(keys, dropna=False, sort=False) + out = g.agg(long_side=("long_contracts", "sum"), + short_side=("short_contracts", "sum"), + spread_total=("spread_contracts", "sum"), + open_interest=("open_interest", "max"), + n_categories=("long_contracts", "size")).reset_index() + out["long_total"] = out["long_side"] + out["spread_total"].fillna(0) + out["short_total"] = out["short_side"] + out["spread_total"].fillna(0) + out["imbalance"] = out["long_total"] - out["short_total"] + out["balanced"] = out["imbalance"] == 0 + # Exact balance is the assertion; within_tolerance is what an automated check should + # use, so the three Consolidated markets do not produce a permanent red light. + tol = out["n_categories"].map(vintage_ingest.rounding_tolerance) + out["within_tolerance"] = out["imbalance"].abs() <= tol out["oi_gap"] = out["open_interest"] - out["long_total"] - return out + return out.drop(columns=["long_side", "short_side"]) # ── Sources ───────────────────────────────────────────────────────────────── diff --git a/src/cotdata/vintage_ingest.py b/src/cotdata/vintage_ingest.py index c7b86ad..e17506e 100644 --- a/src/cotdata/vintage_ingest.py +++ b/src/cotdata/vintage_ingest.py @@ -154,6 +154,181 @@ def canonicalize_legacy(wide: pd.DataFrame, *, combined: bool = False) -> pd.Dat return pd.DataFrame(rows) +# ── Disaggregated / TFF canonicalisation ──────────────────────────────────── +# Both reports are far richer than Legacy, and the canonical schema was designed for them +# rather than for Legacy. Three columns that are permanently null on Legacy are populated +# here: per-category SPREADING, per-category TRADER COUNTS, and the CR4/CR8 concentration +# ratios. Everything the module spec's positioning engine actually keys on (Managed Money, +# Leveraged Funds) lives only in these two reports. +# +# Each entry is (long, short, spread, traders_long, traders_short); None means the report +# genuinely does not publish that field for that category, not that it is missing. +# Producer/Merchant and Non-Reportable have no spreading column by design, and +# Non-Reportable has no trader counts because it is defined as everyone below the +# reporting threshold. +_DISAGG_CATEGORIES = { + "producer_merchant": ("Prod_Merc_Positions_Long_All", "Prod_Merc_Positions_Short_All", + None, "Traders_Prod_Merc_Long_All", "Traders_Prod_Merc_Short_All"), + # NOTE the double underscore in three of these. It is a typo in CFTC's own header row, + # present on Short and Spread but NOT on Long, and it has been there for years. Both + # spellings are accepted below so that the day they fix it is not the day this breaks. + "swap": ("Swap_Positions_Long_All", "Swap__Positions_Short_All", + "Swap__Positions_Spread_All", "Traders_Swap_Long_All", "Traders_Swap_Short_All"), + "managed_money": ("M_Money_Positions_Long_All", "M_Money_Positions_Short_All", + "M_Money_Positions_Spread_All", "Traders_M_Money_Long_All", + "Traders_M_Money_Short_All"), + "other_reportable": ("Other_Rept_Positions_Long_All", "Other_Rept_Positions_Short_All", + "Other_Rept_Positions_Spread_All", "Traders_Other_Rept_Long_All", + "Traders_Other_Rept_Short_All"), + "nonreportable": ("NonRept_Positions_Long_All", "NonRept_Positions_Short_All", + None, None, None), +} + +_TFF_CATEGORIES = { + "dealer": ("Dealer_Positions_Long_All", "Dealer_Positions_Short_All", + "Dealer_Positions_Spread_All", "Traders_Dealer_Long_All", + "Traders_Dealer_Short_All"), + "asset_manager": ("Asset_Mgr_Positions_Long_All", "Asset_Mgr_Positions_Short_All", + "Asset_Mgr_Positions_Spread_All", "Traders_Asset_Mgr_Long_All", + "Traders_Asset_Mgr_Short_All"), + "leveraged": ("Lev_Money_Positions_Long_All", "Lev_Money_Positions_Short_All", + "Lev_Money_Positions_Spread_All", "Traders_Lev_Money_Long_All", + "Traders_Lev_Money_Short_All"), + "other_reportable": ("Other_Rept_Positions_Long_All", "Other_Rept_Positions_Short_All", + "Other_Rept_Positions_Spread_All", "Traders_Other_Rept_Long_All", + "Traders_Other_Rept_Short_All"), + "nonreportable": ("NonRept_Positions_Long_All", "NonRept_Positions_Short_All", + None, None, None), +} + +# Concentration ratios are per MARKET, not per category, so they repeat on every category +# row exactly as open_interest does. Only the NET ratios are in the canonical schema; the +# gross ones (Conc_Gross_LE_*) are published too but have no column to land in. +_CONCENTRATION = { + "cr4_net_long": "Conc_Net_LE_4_TDR_Long_All", + "cr4_net_short": "Conc_Net_LE_4_TDR_Short_All", + "cr8_net_long": "Conc_Net_LE_8_TDR_Long_All", + "cr8_net_short": "Conc_Net_LE_8_TDR_Short_All", +} + +_REPORT_DATE_COL = "Report_Date_as_MM_DD_YYYY" + + +def _resolve(wide: pd.DataFrame, name: str | None) -> pd.Series | None: + """Look a column up, tolerating CFTC's single/double-underscore inconsistency. + + Raises rather than silently producing a null column. A canonicaliser that quietly + returns nulls for Managed Money because a header was renamed would write those nulls + as OBSERVATIONS, and the next real value would then be recorded as a revision. A + renamed column must fail loudly at ingest, not decay into fake revision history. + """ + if name is None: + return None + if name in wide.columns: + return wide[name] + for alt in (name.replace("__", "_"), name.replace("_Positions", "__Positions")): + if alt in wide.columns: + return wide[alt] + raise ValidationError( + f"column {name!r} not found (nor an underscore variant). CFTC changed a header: " + f"map it explicitly rather than letting this category ingest as nulls, which would " + f"be recorded as real observations and then as revisions when it came back.") + + +def _combined_flag(wide: pd.DataFrame, override: bool | None) -> bool: + """Read ``combined`` from the file instead of hardcoding it. + + Disagg and TFF both carry a ``FutOnly_or_Combined`` column stating which series the + file is. Only futures-only files are fetched today, so this is constant-``FutOnly``, + but reading it means that adding the combined files later is purely a fetch-list + change: the canonicaliser is already correct, and the two series can never be silently + merged into one time series because ``combined`` is in the natural key. + """ + if override is not None: + return override + if "FutOnly_or_Combined" not in wide.columns: + return False + vals = {str(v).strip().lower() for v in wide["FutOnly_or_Combined"].dropna().unique()} + if not vals: + return False + if len(vals) > 1: + raise ValidationError( + f"file mixes futures-only and combined rows ({sorted(vals)}). These are " + f"different series and must not share a time series (module spec §3).") + return vals.pop() == "combined" + + +def _canonicalize(wide: pd.DataFrame, *, report_type: str, categories: dict, + combined: bool | None) -> pd.DataFrame: + """Melt a wide Disagg/TFF frame into canonical long rows, one per category. + + Vectorised per category rather than per row: these files are 190 columns wide and a + per-row loop over a cold-start backfill is minutes of pure overhead. + ``canonicalize_legacy`` is deliberately NOT routed through here. Its output feeds + ``row_sha256``, which is a permanent artifact, and rewriting the code path that + produces already-stored hashes to save a little duplication would risk registering + every stored row as revised at once. + """ + if _REPORT_DATE_COL in wide.columns: + report_date = pd.to_datetime(wide[_REPORT_DATE_COL]) + else: + report_date = pd.to_datetime(pd.Series(wide.index, index=wide.index)) + report_date = report_date.dt.normalize() + is_combined = _combined_flag(wide, combined) + + base = { + "report_date": report_date.to_numpy(), + "market_code": wide["CFTC_Contract_Market_Code"].astype(str).to_numpy(), + "market_name": wide.get("Market_and_Exchange_Names"), + "report_type": report_type, + "combined": is_combined, + "open_interest": _resolve(wide, "Open_Interest_All").to_numpy(), + } + for field, col in _CONCENTRATION.items(): + base[field] = wide[col].to_numpy() if col in wide.columns else pd.NA + + frames = [] + for category, (lc, sc, sp, tl, ts) in categories.items(): + part = pd.DataFrame(base) + part["category"] = category + part["long_contracts"] = _resolve(wide, lc).to_numpy() + part["short_contracts"] = _resolve(wide, sc).to_numpy() + for field, col in (("spread_contracts", sp), ("trader_count_long", tl), + ("trader_count_short", ts)): + s = _resolve(wide, col) + part[field] = pd.NA if s is None else s.to_numpy() + frames.append(part) + out = pd.concat(frames, ignore_index=True) + + # Trader-count columns arrive as STRINGS, because CFTC writes "." for a suppressed + # count (published counts are withheld where too few traders would be identifiable). + # Measured on the 2026 Disaggregated file: 3,578 of 7,847 Managed Money long counts + # are suppressed, so this is a routine state and not an error. Coercing to null is the + # correct reading of ".", and it must happen HERE rather than being left to parquet, + # which refuses the mixed column outright. + # + # Coercing every value field, not just the trader counts, is deliberate: the whole + # point is that a suppression marker or a stray pad must never reach row_sha256 as the + # literal string, because a later numeric value for the same key would then be + # recorded as a revision that never happened. + for f in VALUE_FIELDS: + if f in out.columns: + out[f] = pd.to_numeric(out[f], errors="coerce") + return out.reindex(columns=[c for c in ALL_COLUMNS if c in out.columns]) + + +def canonicalize_disagg(wide: pd.DataFrame, *, combined: bool | None = None) -> pd.DataFrame: + """Disaggregated futures report to canonical long rows (five categories per market).""" + return _canonicalize(wide, report_type="disaggregated", + categories=_DISAGG_CATEGORIES, combined=combined) + + +def canonicalize_tff(wide: pd.DataFrame, *, combined: bool | None = None) -> pd.DataFrame: + """Traders in Financial Futures report to canonical long rows.""" + return _canonicalize(wide, report_type="tff", + categories=_TFF_CATEGORIES, combined=combined) + + # ── Validation (§5) ───────────────────────────────────────────────────────── class ValidationError(ValueError): pass @@ -217,17 +392,39 @@ def validate(canonical: pd.DataFrame) -> list[str]: oi = grp["open_interest"].max() if pd.isna(oi): continue + tol = rounding_tolerance(len(grp)) for side in ("long_contracts", "short_contracts"): total = pd.to_numeric(grp.get(side), errors="coerce").sum() spread = pd.to_numeric(grp.get("spread_contracts"), errors="coerce").sum() - if total + spread > oi: + if total + spread > oi + tol: report_date, market_code = key[0], key[1] warnings.append( f"{pd.Timestamp(report_date).date()} {market_code}: " - f"{side} summed across categories ({total + spread:.0f}) > OI ({oi})") + f"{side} summed across categories ({total + spread:.0f}) > OI ({oi}) " + f"by more than the {tol}-contract rounding tolerance") return warnings +def rounding_tolerance(n_categories: int) -> int: + """Contracts by which the category totals may miss open interest without it meaning + anything. + + Derived from the mechanism, not fitted to the residual. CFTC publishes a handful of + **Consolidated** contracts (market codes carrying a ``+`` suffix: S&P 500, NASDAQ-100 + and DJIA Consolidated) which aggregate several contract sizes onto one unit, so each + category figure is independently rounded. Summing ``n`` independently rounded values + admits at most ``n`` contracts of error, which is the bound returned here. + + Measured against the 2026 files, the worst observed breach is well inside it: Legacy 1 + contract (bound 3), TFF 2 (bound 5), Disaggregated 0. Every single breach in all three + reports falls in one of those three Consolidated markets and nowhere else, so the + tolerance costs no real sensitivity. Without it, 48 off-by-one warnings fire on the + 2026 files alone, which is exactly the cry-wolf rate the previous version of this check + was corrected for. + """ + return max(1, int(n_categories)) + + # ── Observation store I/O ─────────────────────────────────────────────────── def read_observations(report_years=None) -> pd.DataFrame: d = _obs_dir() diff --git a/tests/test_vintage_canonicalize.py b/tests/test_vintage_canonicalize.py new file mode 100644 index 0000000..5fe4a11 --- /dev/null +++ b/tests/test_vintage_canonicalize.py @@ -0,0 +1,205 @@ +"""Disaggregated and TFF canonicalisation. + +These two reports, not Legacy, carry the categories the whole system keys on (Managed +Money, Leveraged Funds), plus three fields Legacy never populates: per-category spreading, +per-category trader counts, and CR4/CR8 concentration. Synthetic frames, matching the +repo's test idiom; the real-data numbers are recorded in docs/design/cot_vintage.md. +""" +import pandas as pd +import pytest + + +def _disagg_wide(**over): + """One market, one week, shaped like providers/cftc_disagg._parse_zip output. + + Balanced by construction: each side sums to 900 and open interest is 900 + spreading, + which is the identity CFTC's own files satisfy exactly. + """ + row = { + "Report_Date_as_MM_DD_YYYY": "2026-07-21", + "CFTC_Contract_Market_Code": "088691", + "Market_and_Exchange_Names": "GOLD - COMMODITY EXCHANGE INC.", + "FutOnly_or_Combined": "FutOnly", + "Open_Interest_All": 1000, + "Prod_Merc_Positions_Long_All": 100, "Prod_Merc_Positions_Short_All": 400, + "Traders_Prod_Merc_Long_All": "12", "Traders_Prod_Merc_Short_All": "14", + # CFTC's own double-underscore typo on Short and Spread but not Long. + "Swap_Positions_Long_All": 200, "Swap__Positions_Short_All": 150, + "Swap__Positions_Spread_All": 30, + "Traders_Swap_Long_All": "5", "Traders_Swap_Short_All": "6", + "M_Money_Positions_Long_All": 400, "M_Money_Positions_Short_All": 100, + "M_Money_Positions_Spread_All": 50, + "Traders_M_Money_Long_All": "40", "Traders_M_Money_Short_All": ".", + "Other_Rept_Positions_Long_All": 150, "Other_Rept_Positions_Short_All": 200, + "Other_Rept_Positions_Spread_All": 20, + "Traders_Other_Rept_Long_All": "7", "Traders_Other_Rept_Short_All": "8", + "NonRept_Positions_Long_All": 50, "NonRept_Positions_Short_All": 50, + "Conc_Net_LE_4_TDR_Long_All": 20.5, "Conc_Net_LE_4_TDR_Short_All": 18.0, + "Conc_Net_LE_8_TDR_Long_All": 31.0, "Conc_Net_LE_8_TDR_Short_All": 27.5, + } + row.update(over) + return pd.DataFrame([row]) + + +def _tff_wide(**over): + row = { + "Report_Date_as_MM_DD_YYYY": "2026-07-21", + "CFTC_Contract_Market_Code": "13874+", + "Market_and_Exchange_Names": "S&P 500 Consolidated - CHICAGO MERCANTILE EXCHANGE", + "FutOnly_or_Combined": "FutOnly", + "Open_Interest_All": 500, + "Dealer_Positions_Long_All": 50, "Dealer_Positions_Short_All": 200, + "Dealer_Positions_Spread_All": 10, + "Traders_Dealer_Long_All": "9", "Traders_Dealer_Short_All": "11", + "Asset_Mgr_Positions_Long_All": 200, "Asset_Mgr_Positions_Short_All": 60, + "Asset_Mgr_Positions_Spread_All": 5, + "Traders_Asset_Mgr_Long_All": "20", "Traders_Asset_Mgr_Short_All": "8", + "Lev_Money_Positions_Long_All": 120, "Lev_Money_Positions_Short_All": 150, + "Lev_Money_Positions_Spread_All": 15, + "Traders_Lev_Money_Long_All": "51", "Traders_Lev_Money_Short_All": "44", + "Other_Rept_Positions_Long_All": 30, "Other_Rept_Positions_Short_All": 40, + "Other_Rept_Positions_Spread_All": 8, + "Traders_Other_Rept_Long_All": "4", "Traders_Other_Rept_Short_All": "5", + "NonRept_Positions_Long_All": 20, "NonRept_Positions_Short_All": 20, + "Conc_Net_LE_4_TDR_Long_All": 15.0, "Conc_Net_LE_4_TDR_Short_All": 22.0, + "Conc_Net_LE_8_TDR_Long_All": 25.0, "Conc_Net_LE_8_TDR_Short_All": 41.9, + } + row.update(over) + return pd.DataFrame([row]) + + +def test_disagg_maps_every_category_in_the_controlled_vocabulary(): + from cotdata import vintage_ingest as vi + c = vi.canonicalize_disagg(_disagg_wide()) + assert set(c["category"]) == vi.CATEGORIES["disaggregated"] + mm = c[c.category == "managed_money"].iloc[0] + assert (mm.long_contracts, mm.short_contracts, mm.spread_contracts) == (400, 100, 50) + assert mm.report_type == "disaggregated" + assert mm.market_code == "088691" + + +def test_tff_maps_every_category_in_the_controlled_vocabulary(): + from cotdata import vintage_ingest as vi + c = vi.canonicalize_tff(_tff_wide()) + assert set(c["category"]) == vi.CATEGORIES["tff"] + lev = c[c.category == "leveraged"].iloc[0] + assert (lev.long_contracts, lev.short_contracts, lev.spread_contracts) == (120, 150, 15) + assert lev.report_type == "tff" + + +def test_cftc_double_underscore_typo_is_tolerated_both_ways(): + """Swap__Positions_Short_All has two underscores in CFTC's header and Long has one. + The day they fix it must not be the day Swap Dealer positions start ingesting as + nulls, which would then read as a revision when the value came back.""" + from cotdata import vintage_ingest as vi + fixed = _disagg_wide().rename(columns={ + "Swap__Positions_Short_All": "Swap_Positions_Short_All", + "Swap__Positions_Spread_All": "Swap_Positions_Spread_All"}) + swap = vi.canonicalize_disagg(fixed) + swap = swap[swap.category == "swap"].iloc[0] + assert (swap.long_contracts, swap.short_contracts, swap.spread_contracts) == (200, 150, 30) + + +def test_a_renamed_column_raises_instead_of_ingesting_nulls(): + """Silent nulls are the worst outcome: they get WRITTEN as observations, and the next + real value is then recorded as a revision that never happened.""" + from cotdata import vintage_ingest as vi + broken = _disagg_wide().rename( + columns={"M_Money_Positions_Long_All": "MMoney_Long_All_v2"}) + with pytest.raises(vi.ValidationError, match="M_Money_Positions_Long_All"): + vi.canonicalize_disagg(broken) + + +def test_suppressed_trader_count_becomes_null_not_the_literal_dot(): + """CFTC writes '.' where a count would identify traders. Measured on the 2026 file, + 3,578 of 7,847 Managed Money long counts are suppressed, so this is routine.""" + from cotdata import vintage_ingest as vi + c = vi.canonicalize_disagg(_disagg_wide()) + mm = c[c.category == "managed_money"].iloc[0] + assert mm.trader_count_long == 40 + assert pd.isna(mm.trader_count_short) + assert vi._norm(mm.trader_count_short) == "" # and never reaches the hash as a string + + +def test_concentration_ratios_and_spreading_populate_unlike_legacy(): + from cotdata import vintage_ingest as vi + c = vi.canonicalize_disagg(_disagg_wide()) + assert (c["cr4_net_long"] == 20.5).all() # per market, repeated per category row + assert (c["cr8_net_short"] == 27.5).all() + assert c["spread_contracts"].notna().sum() == 3 # not prod_merc, not nonreportable + + +def test_combined_is_read_from_the_file_not_hardcoded(): + from cotdata import vintage_ingest as vi + assert vi.canonicalize_disagg(_disagg_wide())["combined"].eq(False).all() + combined = _disagg_wide(FutOnly_or_Combined="Combined") + assert vi.canonicalize_disagg(combined)["combined"].eq(True).all() + + +def test_a_file_mixing_futures_only_and_combined_is_refused(): + """They are different series and must never share a time series (module spec section 3).""" + from cotdata import vintage_ingest as vi + mixed = pd.concat([_disagg_wide(), _disagg_wide(FutOnly_or_Combined="Combined")], + ignore_index=True) + with pytest.raises(vi.ValidationError, match="futures-only and combined"): + vi.canonicalize_disagg(mixed) + + +def test_disagg_satisfies_the_zero_sum_identity_exactly(): + """Unlike Legacy, spreading IS captured here, so the sides reconcile to open interest + with no gap at all. Real data: 7,847 of 7,847 weeks, gap zero everywhere.""" + from cotdata import vintage_flow as vf + from cotdata import vintage_ingest as vi + z = vf.zero_sum_check(vi.canonicalize_disagg(_disagg_wide())) + assert bool(z["balanced"].iloc[0]) is True + assert int(z["oi_gap"].iloc[0]) == 0 + + +def test_rounding_tolerance_is_derived_from_the_category_count(): + """CFTC's Consolidated contracts (market codes ending '+') aggregate several contract + sizes onto one unit, so each category figure is independently rounded. Summing n of + them admits at most n contracts of error. Not a fitted constant.""" + from cotdata import vintage_ingest as vi + assert vi.rounding_tolerance(3) == 3 # legacy + assert vi.rounding_tolerance(5) == 5 # disagg / tff + off_by_one = _tff_wide(Open_Interest_All=500 + 38 + 1) # sides sum to 500+38 + assert vi.validate(vi.canonicalize_tff(off_by_one)) == [] + + +def test_a_breach_larger_than_the_tolerance_still_warns(): + from cotdata import vintage_ingest as vi + broken = _tff_wide(Lev_Money_Positions_Long_All=99999) + assert vi.validate(vi.canonicalize_tff(broken)) + + +def test_change_only_and_revisions_work_for_the_new_report_types(tmp_path, monkeypatch): + """The bitemporal machinery is report-type agnostic, but nothing had exercised it for + anything except Legacy until now.""" + monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) + from cotdata import vintage_ingest as vi + first = vi.canonicalize_disagg(_disagg_wide()) + r1 = vi.ingest_canonical(first, snapshot_id="s1") + assert (r1["observations"], r1["revisions"]) == (5, 0) + + r2 = vi.ingest_canonical(vi.canonicalize_disagg(_disagg_wide()), snapshot_id="s2") + assert (r2["observations"], r2["revisions"]) == (0, 0) # idempotent + + revised = _disagg_wide(M_Money_Positions_Long_All=444) + r3 = vi.ingest_canonical(vi.canonicalize_disagg(revised), snapshot_id="s3") + assert r3["observations"] == 1 + rev = vi.read_revisions() + assert list(rev["category"]) == ["managed_money"] + assert list(rev["field"]) == ["long_contracts"] + assert (rev["old_value"].iloc[0], rev["new_value"].iloc[0]) == ("400", "444") + + +def test_tff_and_disagg_rows_never_collide_in_the_natural_key(tmp_path, monkeypatch): + """report_type is in the key, so the same market code under two reports stays two + series. S&P 500 appears in both Legacy and TFF.""" + monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) + from cotdata import vintage_ingest as vi + same_code = _disagg_wide(CFTC_Contract_Market_Code="13874+") + vi.ingest_canonical(vi.canonicalize_disagg(same_code), snapshot_id="s1") + r = vi.ingest_canonical(vi.canonicalize_tff(_tff_wide()), snapshot_id="s2") + assert (r["observations"], r["revisions"]) == (5, 0) # 5 new, nothing overwritten + assert len(vi.read_observations()) == 10 diff --git a/tests/test_vintage_ingest.py b/tests/test_vintage_ingest.py index 7b1151e..75aee13 100644 --- a/tests/test_vintage_ingest.py +++ b/tests/test_vintage_ingest.py @@ -225,24 +225,30 @@ def test_cli_ingest_exits_nonzero_when_revisions_are_recorded(store_env, capsys) assert "RESTATEMENT SUSPECT" in capsys.readouterr().out -def test_unparseable_report_types_drain_out_of_pending(store_env): - """disagg/TFF/weekly-static have no canonicaliser yet. They must be marked SKIPPED, - not left pending: a snapshot that never drains is re-selected forever, and a suspect - one would then re-alert on every future run.""" +def test_unparseable_sources_drain_out_of_pending(store_env): + """A source with no canonicaliser must be marked SKIPPED, not left pending: one that + never drains is re-selected forever, and a suspect one would then re-alert on every + future run. + + Since the disagg and TFF canonicalisers landed, the ONLY unparseable source left is + the weekly static (129 positional columns, a genuinely larger job than the annual + zips). A source whose parse FAILS must drain too, by a different route: it lands on + parse_status=failed, which --pending also does not select.""" from cotdata import vintage, vintage_cli vintage._write_manifest({"schema_version": 1, "snapshots": [ - {"snapshot_id": "d1", "report_type": "disaggregated", "source_kind": "annual_zip", - "local_path": "x", "parse_status": "pending", "retrieved_at": "2026-07-31T00:00:00Z"}, {"snapshot_id": "w1", "report_type": "legacy", "source_kind": "weekly_static", "local_path": "y", "parse_status": "pending", "retrieved_at": "2026-07-31T00:00:00Z"}, + {"snapshot_id": "d1", "report_type": "disaggregated", "source_kind": "annual_zip", + "local_path": "gone.zip", "parse_status": "pending", + "retrieved_at": "2026-07-31T00:00:00Z"}, ]}) assert vintage_cli.main(["ingest", "--pending"]) == 0 after = {s["snapshot_id"]: s for s in vintage.read_snapshots()} - assert after["d1"]["parse_status"] == "skipped" assert after["w1"]["parse_status"] == "skipped" - assert "no canonicaliser" in after["d1"]["parse_error"] - # and a second run selects nothing, so it stays quiet + assert "no canonicaliser" in after["w1"]["parse_error"] + assert after["d1"]["parse_status"] == "failed" # has a canonicaliser; file is missing + # and a second run selects neither, so it stays quiet assert vintage_cli.main(["ingest", "--pending"]) == 0 @@ -251,7 +257,7 @@ def test_suspect_on_an_unparseable_type_alerts_once_then_drains(store_env): every run forever.""" from cotdata import vintage, vintage_cli vintage._write_manifest({"schema_version": 1, "snapshots": [ - {"snapshot_id": "d1", "report_type": "disaggregated", "source_kind": "annual_zip", + {"snapshot_id": "d1", "report_type": "legacy", "source_kind": "weekly_static", "local_path": "x", "parse_status": "pending", "restatement_suspect": True, "report_year": 2025, "retrieved_at": "2026-07-31T00:00:00Z"}, ]}) From 017ba563833658a45ee8ba608b3f8f557efed5b4 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 23:19:49 -0400 Subject: [PATCH 5/7] docs: check cotmetrics for step 2 prior art, and correct the coverage finding Four questions, all answered by reading ~/code/trading_workspace/cotmetrics. 1. NO notional or multiplier code. point_value / Point Value / contract_specs / read_metadata all return zero hits; every `multiplier` hit is sigma_multiplier, a z-score threshold. The one apparent exception is options_data.calculate_intrinsic_curve, which is EQUITY ETF option chains with a hardcoded *100 shares per contract, not a futures multiplier. Step 2 writes the notional path fresh, and no existing consumer needs fixing. 2. NO market-code resolver either. cftc_code / CotSymbolCodeMap / contract_market_code return zero hits, and cotmetrics never imports cotdata.registry, confirming ADR-0007 observation #3 against the code. Its whole cotdata surface is get_cot / get_prices / schema_version, each taking the INTERNAL SYMBOL and letting cotdata resolve privately. cotmetrics never holds a market code. crowdmon's contract_master runs the other direction (canonical rows are keyed by market_code), so it becomes the first consumer of cotdata.registry. 3. COVERAGE WAS OVERSTATED, and the headline is withdrawn. "42 of 95, 44%" was measured against the Legacy store and framed against the full CFTC list, which step 2 does not aim at. Against the deployed universe: params.yaml is 47 (42 Role: deploy + 5 Role: heldout), 45 of 47 are joinable, and ALL 42 deploy markets are. The only two failures are MME and MFS, both heldout, both without Norgate specs or prices. Also caught: the two 42s (legacy-joinable, and Role: deploy) are the same number AND the same set for unrelated reasons, and EMD/KE/NKD were excluded only for having no LEGACY table, so the canonicalisers just shipped move the total to 45. 4. BACKADJ VERIFIED. Three production call sites, signals.py:1028, CotIndexer.py:655, options_data.py:366, all explicitly backadj. The ADR claim is true of the code, not just of the ADR. Stronger result from reading what each DOES with the series: none multiplies a historical price by a quantity (two use returns or bar shape, the third only the latest close, where the two series are equal by construction). So the +294% error is NOT shipping anywhere and there is nothing to go back and fix. Co-Authored-By: Claude Opus 5 --- docs/design/crowdmon_step2_normalisation.md | 153 +++++++++++++++++--- 1 file changed, 130 insertions(+), 23 deletions(-) diff --git a/docs/design/crowdmon_step2_normalisation.md b/docs/design/crowdmon_step2_normalisation.md index ff86016..ec14034 100644 --- a/docs/design/crowdmon_step2_normalisation.md +++ b/docs/design/crowdmon_step2_normalisation.md @@ -52,20 +52,39 @@ is infrastructure. The all-USD result is a genuine simplification worth banking: §5.2 rung 3 says "net notional USD", and for this universe that is a multiplication with no FX join. -### Coverage ceiling: 42 of 95 markets +### Coverage: not a constraint. Every deployed market is joinable + +**Corrected 2026-07-30.** An earlier draft of this section made "42 of 95 markets, 44%" a +headline finding. That was measured against the Legacy store only, and it framed the +denominator as the full CFTC market list, which step 2 does not aim at. Measured against +the deployed universe instead: | Set | Count | |---|---| -| Legacy COT market codes in the store | 95 | -| Registry symbols | 49 | -| Registry symbols with contract specs | 47 | -| Registry symbols with unadjusted prices | 47 (`MME`, `MFS` have none) | -| **COT markets joinable to both specs and price** | **42** | - -This is not a bug, it is the registry universe: `cotdata.registry` only names instruments -somebody wanted prices for. But it means **any "cross-market" result from step 2 onward -covers 44% of the CFTC market list**, and §7's PCA and trend-alignment panels inherit that. -State it in output rather than letting a reader assume 95. +| `cotmetrics-config/params.yaml` universe | 47 (**42 `Role: deploy`**, 5 `Role: heldout`) | +| Joinable to contract specs **and** an unadjusted price, matching COT in **any** report | **45 of 47** | +| Of the 42 `deploy` markets | **42 of 42** | +| Not joinable | `MME`, `MFS` (MSCI Emerging Markets, MSCI EAFE) | + +So the ceiling is not binding on anything anyone actually trades. Both failures are +`Role: heldout`, which params.yaml defines as "never in deploy selection", and both fail +for the same simple reason: Norgate carries neither contract specs nor prices for them. + +Two traps in these numbers worth naming, because both are easy to fall into: + +- **The two 42s are not the same fact.** "42 markets joinable against the Legacy store" + and "42 markets with `Role: deploy`" happen to be the same number *and* the same set, + but for unrelated reasons. Three symbols (`EMD`, `KE`, `NKD`) have full specs and price + coverage and are absent from the joinable set only because they have no **Legacy** COT + table at all; they are Disaggregated and TFF markets. Since the disagg and TFF + canonicalisers landed they are joinable, which is what moves the total from 42 to 45. +- **The registry's `report_type` is a preference, not an existence claim.** It declares 41 + disagg and 8 TFF and zero legacy, yet all 42 `deploy` markets do have a Legacy table. + Legacy covers everything back to 1986; `report_type` records which report cotmetrics + prefers to read, not which reports exist for that market. + +*(Aside, outside this document's scope: the workspace `CLAUDE.md` says params.yaml holds +"47 across 9 asset classes (7 heldout)". The file has 5 heldout, not 7.)* --- @@ -120,6 +139,14 @@ reaches **-27.52**, which is not a price. Meanwhile volatility must come from the **back-adjusted** series, because that is the one with correct returns; unadjusted returns carry fake roll gaps. +> **Nothing in the stack currently ships this error** (verified 2026-07-30). Every one of +> cotmetrics' three price reads is `backadj`, but none of them multiplies a historical +> price by a quantity: two consume returns or bar shape, where back-adjusted is correct, +> and the third reads only the latest close, where the two series are equal by +> construction. Step 2 is the first thing in the stack that needs a *level* in currency, +> so this is a trap to avoid rather than a bug to go and fix. See "What cotmetrics does and +> does not give us" below. + So `net_notional × σ_daily` draws its two factors from two different price series. That is not an implementation detail to discover in review, it is the central correctness fact of step 2, and it should be pinned by a test that fails if either leg is swapped. @@ -137,6 +164,70 @@ dates has the same character: a provenance flag is worth more than a clean-looki --- +## What cotmetrics does and does not give us + +Checked directly against `~/code/trading_workspace/cotmetrics` on 2026-07-30, because +"there is probably nothing to reuse" is the kind of assumption that costs a rewrite if it +is wrong in either direction. + +### Nothing to reuse for notional, and the reason is structural + +`point_value`, `Point Value`, `contract_specs` and `read_metadata` return **zero hits** in +cotmetrics. The only `multiplier` hits are `sigma_multiplier`, a z-score threshold in +`conditions.py`, which is unrelated. So the assumption holds: the COT Index is unitless, +and cotmetrics never converts contracts to currency. + +The single exception is not an exception. `options_data.calculate_intrinsic_curve` computes +a "notional intrinsic value" for **equity ETF option chains** using a hardcoded +`* 100` shares-per-contract, for max-pain plots on proxy ETFs. It is an equity options +multiplier, not a futures contract multiplier, and there is nothing in it step 2 can use. + +**Consequence: step 2 writes the notional path fresh.** There is no tested prior art to +inherit, and equally no existing consumer that would need fixing. + +### It never resolves a market code, so there is no resolver to reuse + +`cftc_code`, `CotSymbolCodeMap` and `contract_market_code` also return **zero hits**, and +cotmetrics never imports `cotdata.registry`. ADR-0007's observation #3 is confirmed against +the code, not just remembered. + +What it does instead is the interesting part: its entire cotdata surface is three +functions, `get_cot`, `get_prices` and `schema_version`, each taking the **internal symbol** +(`GC`, `ES`) and letting cotdata resolve the market code privately. cotmetrics never holds +a CFTC market code at all. Its only symbol table, `market_data._SYMBOL_TO_NAME`, is a +params.yaml-derived symbol-to-display-name map for chart labels. + +That is why there is nothing to reuse, and the direction of travel is the reason: +**cotmetrics starts from a symbol; crowdmon's `contract_master` starts from a market code**, +because the canonical vintage rows are keyed by `market_code`. Going code to symbol needs +`cotdata.registry`, which is exported in `__all__` and which crowdmon would become the +first consumer of. Nothing in cotmetrics can help with a lookup it never performs. + +### Every price read really is `backadj`, and the error is not shipping + +Three production call sites, all explicit: + +| Call site | Adjustment | What it does with the series | +|---|---|---| +| `signals.py:1028` | `"backadj"` | wick-rejection scores, shape-based | +| `CotIndexer.py:655` | `'backadj'` | OHLC for the index and charts | +| `options_data.py:366` | `"backadj"` | `Close.iloc[-1]` only, for a live price | + +(Plus two `monkeypatch.setattr` stubs in `tests/test_signals.py`, which set no adjustment.) + +ADR-0007's load-bearing claim is therefore true of the code as it stands, not only of the +ADR. And a second, stronger result falls out of reading what each one *does* with the +series: **none of them multiplies a historical price by a quantity.** Two consume returns +or bar shape, where back-adjusted is the correct choice; the third reads only the most +recent close, where back-adjusted and unadjusted are equal by construction. So the +294% +error described above is **not currently shipping anywhere**, and there is nothing to go +back and fix. + +The advice not to copy cotmetrics' price access stands, but for a narrower reason than +"it uses the wrong series". It uses the right series for what it does. Step 2 is simply +the first thing in the stack that needs a *level* in currency rather than a return, and +`unadj` is a different call that no existing code makes. + ## Proposed build ``` @@ -159,8 +250,8 @@ crowdmon_futures/ |---|---|---| | 1. net contracts | COT only | available. Spec says do not report it, and it is right | | 2. net / open interest | COT only | available, but the denominator is missing spreading (see below) | -| 3. net notional USD | Point Value + unadjusted price | available, 42 markets | -| 4. vol-scaled notional | σ from back-adjusted returns | available, 42 markets. The default for every cross-market comparison | +| 3. net notional USD | Point Value + unadjusted price | available for all 42 `deploy` markets | +| 4. vol-scaled notional | σ from back-adjusted returns | available for all 42 `deploy` markets. The default for every cross-market comparison | Rung 2 carries a defect found while building §6.4: `NonComm_Positions_Spread_All` is not captured by `providers/cftc.py`, so open interest includes spreading contracts the numerator @@ -199,17 +290,33 @@ default makes the second option cheap. sibling, on the ADR-0007 seam argument above. 2. **Do the disagg/TFF canonicalisers first?** Recommend yes. Without them step 2 normalises Legacy `noncommercial`, which is not the category any downstream engine wants. -3. **Accept the 42-of-95 coverage ceiling, or widen the registry first?** Recommend accept - and report it, since widening it means sourcing prices for 53 more markets and that is a - `marketdata` project, not a crowdmon one. +3. ~~**Accept the 42-of-95 coverage ceiling, or widen the registry first?**~~ **Withdrawn.** + Measured, and there is no decision here: all 42 `deploy` markets are joinable, and the + only two failures are held-out markets Norgate does not cover. Sourcing prices for the + 53 Legacy codes outside the registry would be a `marketdata` project, and nothing in + the module spec asks for it. ## Bottom line +**Updated 2026-07-30 after the canonicalisers landed and cotmetrics was checked.** + Step 2 is the first build step that crosses the seam between COT and prices, so it belongs -in a new consumer package rather than in `cotdata`. Most of what it needs is already in the -store and in USD, which is better than expected. Two things are not: the Disaggregated and -TFF reports have no vintage canonicaliser, so the Managed Money and Leveraged Funds series -the whole system is built around cannot yet be read point-in-time, and that is a genuine -prerequisite rather than a detail. And the single correctness trap, using back-adjusted -prices for notional, is one whose error is exactly zero today and reaches nearly 300% in -2002, so it will not be caught by looking at the output. +in a new consumer package rather than in `cotdata`. That is still the recommendation and it +is the only open question of substance. + +Everything else got better on measurement. The prerequisite is cleared: Managed Money and +Leveraged Funds are readable point-in-time. Coverage turned out not to be a constraint at +all, since all 42 `deploy` markets join cleanly and the only two failures are held-out +markets nobody trades. The inputs are all present and all in USD, so there is no FX layer. + +There is nothing to inherit from cotmetrics, and that is a clean answer rather than a +disappointing one: it has no notional path because the COT Index is unitless, and no market +code resolver because it never holds a market code, passing internal symbols to `get_cot` +and `get_prices` and letting cotdata resolve privately. Step 2 writes both fresh, and +becomes the first consumer of `cotdata.registry`. + +The one real trap remains. Notional must come from the unadjusted series and volatility +from the back-adjusted one, and getting that backwards is wrong by nearly 300% in 2002 +while being exactly right today. Nothing currently ships that error, so it is a thing to +avoid rather than a thing to repair, and it should be pinned by a test rather than a +comment. From 9a3475061b50a4444b15f4ea4e2cef443a4fed55 Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Thu, 30 Jul 2026 23:48:17 -0400 Subject: [PATCH 6/7] fix(vintage): six defects found by adversarial review Reviewed by an agent given the spec and diff and NO prior context, because every earlier review was done by the author and shared its blind spots. Six confirmed defects, four in code written the same day. All reproduced before fixing, and each now has a regression test naming the failure. 1. A FETCH FAILURE POISONED THE DEDUPE COMPARISON. Failure records carry no sha, etag or Last-Modified, so the next fetch sent no If-Modified-Since, could not match the dedupe test, and was classified as changed content. One network blip on a frozen year therefore produced a FALSE RESTATEMENT ALERT, the single alarm this subsystem exists to raise, and re-retained megabytes already on disk under a second filename carrying the identical sha. Split into _latest_with_content (content questions: is this the same file?) and _latest_for_url (outcome questions: what happened last time?). 2. AN ABSOLUTE local_path WAS RE-ROOTED UNDER THE STORE, turning /Volumes/ext/... into /Volumes/ext/... COTDATA_VINTAGE_ROOT is MANDATORY on a mirrored replica, so on exactly those hosts every ingest raised FileNotFoundError, was swallowed, and marked the snapshot failed, which --pending never re-selects. One run drained the entire backlog with no CLI able to reset it. 3. A BARE `ingest` REPLAYED EVERY SNAPSHOT EVER RECORDED with observed_at = now. Replaying an older snapshot after a revision wrote the superseded value back with a newer timestamp, emitting a reversed revision (35 -> 30) then a re-revision (30 -> 35) and inflating age_days on both. revisions/ is the primary artifact and age_days is what tells a consumer whether a restatement reached its calibration window. Now defaults to pending, --all-snapshots opts into a replay, and observed_at comes from the snapshot's own retrieved_at, which is what it always meant. 4. `schedule backfill` BYPASSED THE WRITE LOCK while read-modify-writing every observations partition, so running it beside an ingest silently dropped that ingest's rows and left a revision row asserting a change to a value no longer present in observations/. Per-file atomicity was never the missing piece. 5. EVERY FLAT WEEK WAS LABELLED long_liquidation, because d_long <= 0 swallows zero and the dead zone is off by default. Measured on the real 2026 Legacy file: 3,308 of 29,787 transitions (11.1%), making long_liquidation the modal state with a third of its bucket being weeks where nothing happened, and that value_counts() is the CLI's headline. Zero is now `quiet` unconditionally: it is the absence of a change, not a small number, so recognising it needs no parameter. 6. validate() ACCEPTED DUPLICATE NATURAL KEYS and had NO NULL-RATE BAND despite spec section 5 requiring one. Duplicates get identical (observed_at, snapshot_id), exhausting the tie-break so append order decides the stored value. The null band closes the hole errors="coerce" opened: a changed value FORMAT in a column whose NAME never moved would coerce to nulls, get written as observations, and be read as revisions when the values returned. Checked per category, since one broken column is only 1/n of a melted frame and a frame-wide band would wave it past. NOT FIXED, recorded as a real gap in cot_vintage.md section 9: the `announced` release-date tier is unreachable in production and acceptance criterion 5 is UNMET. Nothing writes source="announced" rows, so the Oct-Dec 2025 backlog weeks can only resolve to `derived`. The plumbing and precedence are correct; what is missing is an extractor pulling a (report_date, release_date) pair out of free-text announcement prose, and writing that against no corpus of real announcements would be guessing. A release date resolved by a guess is worse than a derived one, because it carries a provenance flag claiming it was announced. Also corrected two places where the code contradicted its own docs: the comment claiming an interrupted fetch "leaves a consistent store" (it can leave an orphan blob, which is the right direction to fail in but not what was claimed), and cot_vintage.md still describing the planned manifests/cot.json block rather than the self-owned snapshots.json. Verified: 249 tests pass, ruff clean, and the full ingest re-run against the real captured snapshots is unchanged at 82,776 observations with a second run a no-op. Co-Authored-By: Claude Opus 5 --- docs/design/cot_vintage.md | 73 +++++++++++++++++++++++++++++- src/cotdata/vintage.py | 61 ++++++++++++++++++++----- src/cotdata/vintage_cli.py | 41 ++++++++++++++--- src/cotdata/vintage_flow.py | 13 ++++++ src/cotdata/vintage_ingest.py | 50 ++++++++++++++++++++ src/cotdata/vintage_schedule.py | 12 +++++ tests/test_vintage_canonicalize.py | 35 ++++++++++++++ tests/test_vintage_capture.py | 60 ++++++++++++++++++++++++ tests/test_vintage_flow.py | 20 ++++++++ tests/test_vintage_ingest.py | 46 +++++++++++++++++++ 10 files changed, 392 insertions(+), 19 deletions(-) diff --git a/docs/design/cot_vintage.md b/docs/design/cot_vintage.md index 99f0465..1eb070a 100644 --- a/docs/design/cot_vintage.md +++ b/docs/design/cot_vintage.md @@ -192,9 +192,18 @@ vintage/release_schedule.parquet vintage/announcements.parquet ``` -Manifest: a new `vintage` block on the **cot-half** file (`manifests/cot.json`), so it +Manifest: ~~a new `vintage` block on the **cot-half** file (`manifests/cot.json`), so it stays on the CFTC producer's side of the ADR-0007 seam. `store._DOMAIN_HALF` gains -`vintage -> cot`. +`vintage -> cot`.~~ + +**Corrected 2026-07-30** (caught in adversarial review; this paragraph described the plan, +not what shipped). Provenance lives in a **self-owned `vintage/snapshots.json`**, and +`store._DOMAIN_HALF` has no `vintage` key. Two reasons, both discovered during the build: +`store.reconcile_manifest` prunes any manifest entry lacking a matching `{name}.parquet`, +and raw snapshot ids are not parquet files, so they would have been ghost-pruned; and both +deployed sync scripts exclude the name `manifest.json` unanchored, so a +`vintage/manifest.json` would have been stripped in transit. The handoff's §12.4 recorded +this deviation correctly, so this file was the stale one. ### §4.6 amendment (confirmed, not contingent) The spike upgrades the `observed` mechanism: capturing the weekly-static @@ -484,6 +493,66 @@ Linux that is a single filename containing backslashes, so ingest on either repl with "no such file" on a file plainly sitting there. Normalised on read rather than on write, so every snapshot already recorded on the producer stays readable. +## 9. Adversarial review, 2026-07-30 + +The subsystem was reviewed by an agent given the spec and the diff and **no prior context**, +specifically because every earlier review had been done by its author and would have shared +its blind spots. That was worth doing: it found six confirmed defects, four of them in code +written the same day, and one of them fires on an event as ordinary as a dropped network +connection. It also confirmed the hash-purity property empirically rather than by reading, +which is the one property the whole design rests on. + +### Fixed in this pass + +| # | Defect | Why it mattered | +|---|---|---| +| 1 | A **fetch failure poisoned the dedupe comparison**. Failure records carry no sha, etag or Last-Modified, so the next fetch sent no `If-Modified-Since`, could not match the dedupe test, and was classified as changed content. | One network blip on a frozen year produced a **false restatement alert**, which is the single alarm this subsystem exists to raise, and re-retained megabytes already on disk. Split into `_latest_with_content` (content questions) versus `_latest_for_url` (outcome questions). | +| 2 | An **absolute `local_path` was re-rooted under the store**, turning `/Volumes/ext/...` into `/Volumes/ext/...`. | `COTDATA_VINTAGE_ROOT` is *mandatory* on a mirrored replica, so on exactly those hosts every ingest failed, was swallowed, and marked the snapshot `failed`, which `--pending` never re-selects. One run drained the entire backlog with no CLI able to reset it. | +| 3 | A bare `ingest` with no flags **replayed every snapshot ever recorded** with `observed_at = now`. | Replaying an older snapshot after a revision wrote the superseded value back with a newer timestamp, emitting a reversed revision (35 to 30) then a re-revision (30 to 35) and inflating `age_days` on both. `revisions/` is the primary artifact and `age_days` is what tells a consumer whether a restatement reached its calibration window. Now defaults to pending, `--all-snapshots` opts into a replay, and `observed_at` comes from the snapshot's own `retrieved_at`. | +| 4 | **`schedule backfill` bypassed the write lock** while doing a read-modify-write of every observations partition. | Run beside an ingest it silently dropped that ingest's appended rows, leaving a revision row asserting a change to a value no longer present in `observations/`. Per-file atomicity was never the missing piece. | +| 5 | **Every flat week was labelled `long_liquidation`**, because `d_long <= 0` swallows zero and the dead zone is off by default. | Measured on the real 2026 Legacy file: 3,308 of 29,787 transitions (11.1%), which made `long_liquidation` the modal state with a third of its bucket being weeks where nothing happened. That `value_counts()` is the CLI's headline output. Zero is now `quiet` unconditionally, with no threshold involved. | +| 6 | **`validate()` accepted duplicate natural keys**, and **had no null-rate band** despite spec §5 requiring one. | Duplicates get identical `(observed_at, snapshot_id)`, exhausting the tie-break so append order decides the stored value. The null band closes the gap `errors="coerce"` opened: a changed value *format* in a column whose *name* never moved would coerce to nulls, which get written as observations and then read as revisions when the values return. Checked per category, since one broken column is only 1/n of a melted frame. | + +### Confirmed clean, by measurement rather than by reading + +Hash purity, the property everything else depends on, was exercised directly: mutating +`observed_at`, `snapshot_id`, `release_date`, `release_date_source`, `market_name`, +`is_tombstone`, every natural-key column and an unknown extra column all leave +`row_sha256` unchanged, while each of the ten `VALUE_FIELDS` changes it. Scalar stability +holds across `int`, `np.int64/32`, `float`, `np.float64/32`, `str`, `Decimal` and pandas +nullable `Int64`. A release-date backfill was confirmed empirically not to move any hash. + +Atomicity was clean everywhere it was checked: `_write_manifest`, `_append_parquet`, +`vintage_schedule._write`, backfill's per-file write and the raw `.part` write all use +temp plus `os.replace`, and `_WriteLock` uses `O_CREAT|O_EXCL` and fails loudly. + +### NOT fixed, recorded as a real gap: the `announced` tier is unreachable + +**Acceptance criterion 5 is unmet.** `sync()` scrapes the Special Announcements page into +`announcements.parquet`, but nothing ever writes a `source="announced"` row into +`release_schedule.parquet`, and `backfill` reads only the schedule table plus +`published_from_snapshots()`. So the Oct-Dec 2025 backlog weeks, the named target of that +criterion and §6's "single largest PIT hole", can only ever resolve to `derived` in +production. The precedence logic itself is correct and tested; the tests pass because they +inject a schedule frame directly. + +The plumbing is not the missing piece: `write_release_schedule` already accepts a `source` +column, so an `announced` row would flow through correctly. What is missing is a producer, +and building one means extracting a `(report_date, release_date)` pair from free-text +announcement prose. `_parse_announcements` deliberately never raises and captures only +`announcement_date` plus `raw_text`. **Writing that extractor against no corpus of real +announcements would be guessing**, and a release date resolved by a guess is worse than a +`derived` one, because it carries a provenance flag claiming it was announced. Left +unbuilt, and recorded here rather than quietly assumed to work. + +### Known and accepted + +`ingest_canonical` reloads every observation and builds a per-key dict with `iterrows()` +on each call, about 2.3 seconds against a 31k-row store. Fine for the daily path (three +snapshots), but a cold-start `fetch --all` plus `ingest` is roughly 120 snapshots against a +store growing into the millions of rows, so a full historical rebuild is hours rather than +minutes. Not on the daily path, so not fixed here. + ## Bottom line Discovery and the spike are done. The genuine null on historical recovery holds: no diff --git a/src/cotdata/vintage.py b/src/cotdata/vintage.py index e0bb8ac..8838b95 100644 --- a/src/cotdata/vintage.py +++ b/src/cotdata/vintage.py @@ -226,6 +226,31 @@ def _latest_for_url(snapshots: list[dict], url: str) -> dict | None: return max(prev, key=lambda t: (t[0], t[1]))[2] +def _latest_with_content(snapshots: list[dict], url: str) -> dict | None: + """Most recent snapshot for a URL that actually carries content. + + Distinct from ``_latest_for_url`` on purpose, and the distinction is load-bearing. + A FAILURE record has ``content_sha256``, ``http_etag`` and ``http_last_modified`` all + None, so comparing against it means the next fetch sends no If-Modified-Since, cannot + match the dedupe test, and is classified as changed content. + + On a frozen year that produces a FALSE RESTATEMENT ALERT from a single network blip: + the one alarm this subsystem exists to raise, fired by an event that says nothing + about the data. It also re-retains bytes already on disk under a second filename + carrying the identical sha. Both were reproduced before this was split out. + + Content questions ("is this the same file?") must therefore look past failures to the + last snapshot that actually saw bytes. Outcome questions ("what happened last time?") + must not, which is why the edge-trigger still uses ``_latest_for_url``. + """ + prev = [(s.get("retrieved_at") or "", i, s) + for i, s in enumerate(snapshots) + if s.get("source_url") == url and s.get("content_sha256")] + if not prev: + return None + return max(prev, key=lambda t: (t[0], t[1]))[2] + + def _snapshot_id(retrieved_at: str, url: str, tag: str) -> str: """Unique per (retrieval second, url, content-state). @@ -385,10 +410,15 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d changed sha does not imply changed data, and an unchanged sha means nothing new to retain). Raw bytes, once written, are never rewritten. """ + # Two different "previous" snapshots, and conflating them fires false restatement + # alerts. `content` is the last snapshot that actually saw bytes and answers "is this + # the same file?"; `prev` is the last snapshot of any kind and answers "what happened + # last time?", which is what the tripwire edge-triggers on. See _latest_with_content. + content = _latest_with_content(snapshots, source.url) prev = _latest_for_url(snapshots, source.url) res = http_get(source.url, - etag=(prev or {}).get("http_etag"), - last_modified=(prev or {}).get("http_last_modified")) + etag=(content or {}).get("http_etag"), + last_modified=(content or {}).get("http_last_modified")) retrieved_at = _iso(now) base = { "source_url": source.url, @@ -406,11 +436,11 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d **base, "snapshot_id": _snapshot_id(retrieved_at, source.url, "304"), "http_status": 304, - "http_etag": (prev or {}).get("http_etag"), - "http_last_modified": (prev or {}).get("http_last_modified"), - "content_sha256": (prev or {}).get("content_sha256"), + "http_etag": (content or {}).get("http_etag"), + "http_last_modified": (content or {}).get("http_last_modified"), + "content_sha256": (content or {}).get("content_sha256"), "byte_size": None, - "local_path": (prev or {}).get("local_path"), + "local_path": (content or {}).get("local_path"), "note": "304 not-modified", }, prev=prev, now=now) @@ -422,7 +452,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d f"legitimate snapshot.") sha = hashlib.sha256(res.content).hexdigest() - if prev and prev.get("content_sha256") == sha: + if content and content.get("content_sha256") == sha: # Byte-identical to what we already retained (zips regenerate): dedupe, no rewrite. return _annotate({ **base, @@ -432,7 +462,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "http_last_modified": res.last_modified, "content_sha256": sha, "byte_size": len(res.content), - "local_path": prev.get("local_path"), + "local_path": content.get("local_path"), "note": "unchanged bytes (deduped)", }, prev=prev, now=now) @@ -465,7 +495,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d # the human-readable reason; this flag stays because it is the field ``ingest`` and # every stored snapshot already key off. restatement_suspect = ( - prev is not None and source.report_year is not None + content is not None and source.report_year is not None and source.report_year < now.year) return _annotate({ @@ -545,8 +575,17 @@ def fetch(year: int | None = None, *, all_years: bool = False, new_files += 1 # Write after EVERY source, not once at the end: the manifest is small and its # replace is atomic, so this costs nothing measurable and shrinks the crash window - # from a whole run to a single source. Combined with the atomic raw write, an - # interrupted run leaves a consistent store rather than unrecorded blobs. + # from a whole run to a single source. + # + # It does NOT eliminate that window, and an earlier version of this comment + # claimed it did (caught in adversarial review). Raw bytes are written and + # os.replace'd inside capture_source, then the manifest entry is appended here, so + # a crash in between leaves a retained blob with no index entry. That is the + # deliberate direction to fail in: the reverse ordering would leave an index entry + # pointing at a file that does not exist, which is a lie rather than an omission. + # The blob is also self-describing, since its filename carries the sha8 prefix, and + # the only cost of the orphan is that the next fetch re-downloads and retains a + # duplicate copy of bytes already on disk. _write_manifest(m) if rate_limit_s and i < len(sources) - 1: time.sleep(rate_limit_s) diff --git a/src/cotdata/vintage_cli.py b/src/cotdata/vintage_cli.py index 06b20be..600a23f 100644 --- a/src/cotdata/vintage_cli.py +++ b/src/cotdata/vintage_cli.py @@ -48,9 +48,19 @@ def _snapshot_path(local_path: str): Normalising here rather than at write time keeps every snapshot already recorded on the Windows producer readable. """ - from pathlib import Path - return config.store_root().joinpath(*str(local_path).replace("\\", "/").split("/")) \ - if local_path else Path() + from pathlib import Path, PureWindowsPath + if not local_path: + return Path() + raw = str(local_path) + # An ABSOLUTE local_path means COTDATA_VINTAGE_ROOT pointed outside the store, which + # vintage_root() calls mandatory on a mirrored replica. Blindly re-rooting it under + # store_root() turns /Volumes/ext/vintage/... into /Volumes/ext/vintage/..., + # so every ingest raises FileNotFoundError, gets swallowed, and marks the snapshot + # parse_status="failed", which --pending never re-selects. One run would drain the + # entire backlog to failed with no CLI able to reset it. + if PureWindowsPath(raw).is_absolute() or Path(raw).is_absolute(): + return Path(raw.replace("\\", "/")) + return config.store_root().joinpath(*raw.replace("\\", "/").split("/")) # Each report type maps to (parser, canonicaliser). Legacy is indexed by report date @@ -77,7 +87,15 @@ def _cmd_ingest(args) -> int: snaps = vintage.read_snapshots() if args.snapshot: snaps = [s for s in snaps if s.get("snapshot_id") == args.snapshot] - elif args.pending: + elif not args.all_snapshots: + # Default to pending even with no flag. A bare `ingest` used to select EVERY + # snapshot ever recorded and re-ingest each one, which is not a no-op: replaying + # an older snapshot after a revision writes the superseded value back with a + # NEWER observed_at, emitting a reversed revision (35 -> 30) followed by a + # re-revision (30 -> 35), and inflating age_days on both. revisions/ is the + # primary artifact here and age_days is what tells a consumer whether a + # restatement reached into its calibration window, so forging either is worse + # than doing nothing. --all-snapshots still allows a deliberate full replay. snaps = [s for s in snaps if s.get("parse_status") == "pending"] canon = _canonicalisers() total_obs = total_rev = 0 @@ -99,7 +117,14 @@ def _cmd_ingest(args) -> int: path = _snapshot_path(s["local_path"]) try: canonical = fn(path) - res = vintage_ingest.ingest_canonical(canonical, snapshot_id=s["snapshot_id"]) + # observed_at is the snapshot's OWN retrieval time, not the clock at ingest. + # "When did we observe this value" is a property of the capture, so a late or + # repeated ingest of retained bytes must not move it: doing so would let a + # replay of an old snapshot outrank the newer one in the point-in-time read + # and hand back the pre-revision value for a timestamp after the revision. + res = vintage_ingest.ingest_canonical( + canonical, snapshot_id=s["snapshot_id"], + observed_at=s.get("retrieved_at") or None) vintage.update_snapshot(s["snapshot_id"], parse_status="ok", parse_error=None) total_obs += res["observations"] total_rev += res["revisions"] @@ -255,7 +280,11 @@ def main(argv=None) -> int: ing = sub.add_parser("ingest", help="Parse retained raw files into change-only observations.") g = ing.add_mutually_exclusive_group() g.add_argument("--snapshot", default=None, help="Ingest one snapshot id.") - g.add_argument("--pending", action="store_true", help="Ingest all parse_status=pending.") + g.add_argument("--pending", action="store_true", + help="Ingest all parse_status=pending. This is also the default.") + g.add_argument("--all-snapshots", action="store_true", dest="all_snapshots", + help="Replay EVERY recorded snapshot, including already-ingested ones. " + "Only for rebuilding a store from retained raw bytes.") ing.set_defaults(func=_cmd_ingest) d = sub.add_parser("diff", help="Show recorded field-level revisions.") diff --git a/src/cotdata/vintage_flow.py b/src/cotdata/vintage_flow.py index b3bf3c7..af7d17b 100644 --- a/src/cotdata/vintage_flow.py +++ b/src/cotdata/vintage_flow.py @@ -127,6 +127,19 @@ def _classify(df: pd.DataFrame, *, prior_oi, min_frac_oi: float) -> pd.Series: state = state.mask(~long_dominates & (d_short > 0), NEW_SHORTS) state = state.mask(~long_dominates & (d_short <= 0), SHORT_COVERING) + # A week where NEITHER leg moved is `quiet` unconditionally, with no threshold + # involved. Without this it falls through `long_dominates & (d_long <= 0)` and is + # labelled long_liquidation, which is not a judgement call gone wrong but a plain + # misstatement: nothing was liquidated. Measured over the real 2026 Legacy file it is + # 3,308 of 29,787 transitions (11.1%), and it made long_liquidation the modal state + # with 36% of that bucket being weeks where nothing happened. The CLI prints exactly + # that value_counts() as its headline. + # + # Deliberately NOT folded into min_frac_oi. Zero is not a small number, it is the + # absence of a change, so recognising it needs no parameter and must not be switched + # off by leaving the dead zone at its default. + state = state.mask((d_long == 0) & (d_short == 0), QUIET) + if min_frac_oi and prior_oi is not None: dead = prior_oi.astype("float64") * float(min_frac_oi) state = state.mask((mag_l <= dead) & (mag_s <= dead), QUIET) diff --git a/src/cotdata/vintage_ingest.py b/src/cotdata/vintage_ingest.py index e17506e..39f50ad 100644 --- a/src/cotdata/vintage_ingest.py +++ b/src/cotdata/vintage_ingest.py @@ -24,6 +24,12 @@ # ── Schema ────────────────────────────────────────────────────────────────── NATURAL_KEY = ["report_date", "market_code", "report_type", "combined", "category"] +# Ceiling on how much of a POSITION column may be null before ingest refuses (spec §5's +# "null-rate per column within a sane band"). Generous on purpose: it exists to catch a +# format change that coerced a whole column away, not to police the odd blank cell. +# Measured on the real 2026 files, the true null rate on these columns is 0%. +_MAX_NULL_RATE = 0.20 + # Fields that constitute the *value* of an observation. row_sha256 is computed over # these only: not over provenance (observed_at/snapshot_id), not over release_date # (resolved separately, so a release-date backfill must NOT read as a data revision), @@ -384,7 +390,51 @@ def validate(canonical: pd.DataFrame) -> list[str]: if bad: raise ValidationError(f"categories {bad} outside vocabulary for {rt!r}") + # Duplicate natural keys within ONE frame. The read side already refuses these + # (vintage_flow._require_panel); the write side must too, and for a worse reason. + # Ingesting both writes two rows sharing an identical (observed_at, snapshot_id), which + # exhausts _latest_by_key's tie-break and leaves the winner decided by append order. + # Both also diff against the same prior row, so revisions/ gains two contradictory + # entries for one detection. Raising is the only safe answer: there is no principled + # way to pick which of two rows claiming the same key is the real one. + dup = canonical.duplicated(subset=NATURAL_KEY, keep=False) + if dup.any(): + sample = canonical.loc[dup, NATURAL_KEY].head(3).to_dict("records") + raise ValidationError( + f"{int(dup.sum())} rows share a natural key within one frame, e.g. {sample}. " + f"Two rows for one key have no defined ordering, so the stored 'latest' would " + f"be whichever happened to be appended last.") + warnings = [] + + # NULL-RATE BAND (spec §5). The canonicalisers coerce every value field with + # errors="coerce", which is correct for CFTC's "." suppression marker but is exactly + # the mechanism that would silently swallow a CHANGED VALUE FORMAT in a column whose + # name never moved (thousands separators appearing, a unit suffix, a footnote glyph). + # _resolve catches a renamed column; nothing caught a renamed *format*, and the + # consequence is worse than a crash: the nulls get written as real observations, and + # the next genuine value is then recorded as a revision that never happened. Raising + # on a mass-null column turns that into a loud failure at the point of ingest. + # Checked PER CATEGORY, not over the whole frame. Each canonical category is melted + # from its own source column, so one broken column is only 1/n of the rows: with five + # disaggregated categories a wholly-coerced Managed Money column is a 20% frame-wide + # null rate, which any band loose enough to be safe would wave straight through. Per + # category the same failure is 100%, which is unmissable. + fields = [f for f in ("long_contracts", "short_contracts", "open_interest") + if f in canonical.columns] + for (rt, cat), grp in canonical.groupby(["report_type", "category"], + dropna=False, sort=False): + for f in fields: + null_rate = grp[f].isna().mean() + if null_rate > _MAX_NULL_RATE: + raise ValidationError( + f"{null_rate:.0%} of {f!r} is null for {rt}/{cat}, above the " + f"{_MAX_NULL_RATE:.0%} band. A position column is never mostly blank " + f"in a real CFTC file, so this is a parse or format change, not data. " + f"Refusing to write nulls as observations: they would be recorded as " + f"revisions when the values return. (Trader counts are excluded, " + f"since CFTC genuinely suppresses roughly half of them.)") + if "open_interest" not in canonical.columns: return warnings keys = ["report_date", "market_code", "report_type", "combined"] diff --git a/src/cotdata/vintage_schedule.py b/src/cotdata/vintage_schedule.py index aa50e98..14f9bed 100644 --- a/src/cotdata/vintage_schedule.py +++ b/src/cotdata/vintage_schedule.py @@ -328,6 +328,18 @@ def backfill(*, schedule: pd.DataFrame | None = None, if not obs_dir.exists(): return counts + # Under the SAME lock ingest uses. This is a read-modify-write of every observations + # partition, so running it concurrently with an ingest silently drops that ingest's + # appended rows: backfill reads the file, the ingest appends and writes, backfill + # writes back what it read. The store is then internally inconsistent in the worst + # way available here, holding a revision row asserting a change to a value that no + # longer exists in observations/. Each file's write is already atomic; atomicity per + # file was never the missing piece. + with vi._WriteLock(vintage.vintage_root()): + return _backfill_locked(obs_dir, smap, counts, observed_window_days) + + +def _backfill_locked(obs_dir, smap, counts, observed_window_days: int) -> dict: for part in sorted(obs_dir.glob("report_year=*/observations.parquet")): df = pd.read_parquet(part) if df.empty: diff --git a/tests/test_vintage_canonicalize.py b/tests/test_vintage_canonicalize.py index 5fe4a11..b366c3e 100644 --- a/tests/test_vintage_canonicalize.py +++ b/tests/test_vintage_canonicalize.py @@ -203,3 +203,38 @@ def test_tff_and_disagg_rows_never_collide_in_the_natural_key(tmp_path, monkeypa r = vi.ingest_canonical(vi.canonicalize_tff(_tff_wide()), snapshot_id="s2") assert (r["observations"], r["revisions"]) == (5, 0) # 5 new, nothing overwritten assert len(vi.read_observations()) == 10 + + +def test_duplicate_natural_keys_in_one_frame_are_refused(): + """Found by adversarial review. Two rows for one key get identical + (observed_at, snapshot_id), which exhausts _latest_by_key's tie-break and leaves the + stored 'latest' decided by append order. Both also diff against the same prior row, so + revisions/ gains two contradictory entries for one detection.""" + from cotdata import vintage_ingest as vi + doubled = pd.concat([_disagg_wide(), _disagg_wide(Open_Interest_All=1001)], + ignore_index=True) + with pytest.raises(vi.ValidationError, match="natural key"): + vi.validate(vi.canonicalize_disagg(doubled)) + + +def test_a_column_coerced_entirely_to_null_raises_instead_of_being_stored(): + """Spec section 5 requires a null-rate band, and nothing implemented it. errors="coerce" + is right for CFTC's '.' suppression marker but would silently swallow a changed VALUE + FORMAT in a column whose name never moved. The nulls would be written as real + observations and the next genuine value recorded as a revision that never happened.""" + from cotdata import vintage_ingest as vi + reformatted = _disagg_wide(M_Money_Positions_Long_All="4,00") # thousands separator + with pytest.raises(vi.ValidationError, match="null"): + vi.validate(vi.canonicalize_disagg(reformatted)) + + +def test_suppressed_trader_counts_do_not_trip_the_null_band(): + """CFTC genuinely suppresses about half of all trader counts, so the band must apply + to position columns only or it would fire on every real file.""" + from cotdata import vintage_ingest as vi + allsupp = _disagg_wide(**{c: "." for c in + ("Traders_Prod_Merc_Long_All", "Traders_Prod_Merc_Short_All", + "Traders_Swap_Long_All", "Traders_Swap_Short_All", + "Traders_M_Money_Long_All", "Traders_M_Money_Short_All", + "Traders_Other_Rept_Long_All", "Traders_Other_Rept_Short_All")}) + assert vi.validate(vi.canonicalize_disagg(allsupp)) == [] diff --git a/tests/test_vintage_capture.py b/tests/test_vintage_capture.py index 9e70adc..09ed008 100644 --- a/tests/test_vintage_capture.py +++ b/tests/test_vintage_capture.py @@ -433,3 +433,63 @@ def now(): seen.append(vintage.Source(rec["report_type"], rec["source_kind"], "zip", rec["source_url"], rec["report_year"])) return seen + + +def test_a_fetch_failure_does_not_defeat_dedupe_or_fire_a_false_tripwire(store_env): + """Found by adversarial review, reproduced before fixing. + + A failure record carries no sha, no etag and no Last-Modified. Comparing the next + fetch against it means no If-Modified-Since is sent, the dedupe test cannot match, and + byte-identical content is classified as CHANGED. On a frozen year that turns one + network blip into a false restatement alert, which is the single alarm this whole + subsystem exists to raise, and re-retains megabytes already on disk.""" + from cotdata import vintage + body = _body(b"2025-frozen") + + class _Flaky: + def __init__(self): + self.n = 0 + + def __call__(self, url, *, etag=None, last_modified=None): + from cotdata.vintage import HttpResult + self.n += 1 + if self.n == 3: + raise ConnectionError("network down") + if self.n == 2: + return HttpResult(304, None) + return HttpResult(200, body, etag=None, last_modified="lm1") + + http, now, src = _Flaky(), _clock(), _one(LEGACY_2025, 2025) + for _ in range(3): + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + recovered = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + + rec = recovered["records"][0] + assert rec["outcome"] == vintage.OUTCOME_DEDUPED # not "changed" + assert rec["tripwire_alert"] is None # no false restatement alert + assert not rec.get("restatement_suspect") # absent on a dedupe, which is falsy + assert recovered["new_files"] == 0 # nothing re-retained + assert len(list((store_env / "vintage" / "raw").rglob("*.zip"))) == 1 + + +def test_a_304_after_a_failure_still_resolves_its_prior_file(store_env): + """The 304 branch reuses the previous snapshot's sha and local_path. Taken from a + failure record those are both None, leaving a snapshot that points nowhere.""" + from cotdata import vintage + from cotdata.vintage import HttpResult + body = _body(b"frozen") + seq = [lambda: HttpResult(200, body, etag=None, last_modified="lm1"), + lambda: (_ for _ in ()).throw(ConnectionError("down")), + lambda: HttpResult(304, None)] + n = [0] + + def http(url, *, etag=None, last_modified=None): + n[0] += 1 + return seq[n[0] - 1]() + + now, src = _clock(), _one(LEGACY_2025, 2025) + for _ in range(3): + res = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + rec = res["records"][0] + assert rec["content_sha256"] is not None + assert rec["local_path"] is not None diff --git a/tests/test_vintage_flow.py b/tests/test_vintage_flow.py index 0bb867c..92afa38 100644 --- a/tests/test_vintage_flow.py +++ b/tests/test_vintage_flow.py @@ -122,3 +122,23 @@ def test_missing_columns_raise_rather_than_silently_producing_nothing(): can = _rows(WEEKS[:2], longs=[1, 2], shorts=[1, 1]).drop(columns=["short_contracts"]) with pytest.raises(vf.FlowError, match="missing columns"): vf.decompose(can) + + +def test_a_week_where_nothing_moved_is_quiet_not_long_liquidation(): + """Found by adversarial review. `long_dominates & (d_long <= 0)` swallows d_long == 0, + so every flat week was labelled long_liquidation. Measured over the real 2026 Legacy + file that was 3,308 of 29,787 transitions (11.1%), which made long_liquidation the + modal state with a third of its bucket being weeks where nothing happened. + + Zero is not a small number, it is the absence of a change, so this must NOT depend on + the min_frac_oi dead zone, which is off by default.""" + from cotdata import vintage_flow as vf + can = _rows(WEEKS[:3], longs=[100, 100, 100], shorts=[50, 50, 50]) + assert list(vf.decompose(can)["state"]) == [vf.QUIET, vf.QUIET] + + +def test_a_one_contract_move_is_still_classified(): + """The quiet rule must catch only exact zero, or it becomes an unstated threshold.""" + from cotdata import vintage_flow as vf + can = _rows(WEEKS[:2], longs=[100, 101], shorts=[50, 50]) + assert vf.decompose(can)["state"].iloc[0] == vf.NEW_LONGS diff --git a/tests/test_vintage_ingest.py b/tests/test_vintage_ingest.py index 75aee13..19ef861 100644 --- a/tests/test_vintage_ingest.py +++ b/tests/test_vintage_ingest.py @@ -298,3 +298,49 @@ def test_oi_over_sum_warns_not_raises(store_env): res = vi.ingest_canonical(c, snapshot_id="s1") assert res["warnings"] # soft warning emitted assert res["observations"] == 3 # still ingested + + +def test_an_absolute_local_path_is_not_re_rooted_under_the_store(store_env): + """Found by adversarial review. COTDATA_VINTAGE_ROOT outside the store records an + ABSOLUTE local_path, and vintage_root() calls that override mandatory on a mirrored + replica. Re-rooting it turned /Volumes/ext/... into /Volumes/ext/..., so every + ingest raised FileNotFoundError, got swallowed, and marked the snapshot failed, which + --pending never re-selects. One run drained the whole backlog with no way to reset.""" + from cotdata import vintage_cli + assert str(vintage_cli._snapshot_path("/ext/vintage/raw/a/x.zip")) == "/ext/vintage/raw/a/x.zip" + assert str(vintage_cli._snapshot_path(r"D:\vintage\raw\a\x.zip")) == "D:/vintage/raw/a/x.zip" + rel = vintage_cli._snapshot_path(r"vintage\raw\annual_zip\2026\x.zip") + assert rel == store_env / "vintage" / "raw" / "annual_zip" / "2026" / "x.zip" + + +def test_backfill_takes_the_same_write_lock_as_ingest(store_env): + """Found by adversarial review. backfill read-modify-writes every observations + partition, so running it beside an ingest drops that ingest's appended rows and leaves + a revision row asserting a change to a value no longer present in observations/.""" + import pytest as _pytest + + from cotdata import vintage, vintage_ingest, vintage_schedule + vintage_ingest.ingest_canonical(_canon("2026-07-21"), snapshot_id="s1") + with vintage_ingest._WriteLock(vintage.vintage_root()): + with _pytest.raises(RuntimeError, match="single-writer"): + vintage_schedule.backfill() + vintage_schedule.backfill() # and succeeds once the lock is released + + +def test_a_replayed_snapshot_keeps_its_own_observed_at(store_env): + """observed_at is a property of the CAPTURE, not of when ingest happened. Taking it + from the clock let a replay of an older snapshot outrank a newer one in the + point-in-time read and hand back the pre-revision value.""" + import pandas as pd + + from cotdata import vintage_ingest as vi + early = _canon("2026-07-21", comm_long=100) + late = _canon("2026-07-21", comm_long=200) + vi.ingest_canonical(early, snapshot_id="s1", observed_at="2026-07-24T19:00:00Z") + vi.ingest_canonical(late, snapshot_id="s2", observed_at="2026-07-31T19:00:00Z") + # replay the OLD snapshot after the new one, exactly as `ingest --all-snapshots` would + vi.ingest_canonical(early, snapshot_id="s1", observed_at="2026-07-24T19:00:00Z") + + now = vi.asof(pd.Timestamp("2026-08-01"), report_date="2026-07-21") + comm = now[now.category == "commercial"] + assert list(comm["long_contracts"]) == [200] # the newer value still wins From 5e492ac9dc32769f695fe4ca18e78bf9f5e8f5ac Mon Sep 17 00:00:00 2001 From: Matt Spinola Date: Fri, 31 Jul 2026 08:41:02 -0400 Subject: [PATCH 7/7] fix(vintage): finish two incomplete fixes, plus four from a second review The six fixes from 9a34750 were reviewed by a second cold agent, since they were written by the same person who wrote the code being fixed and nobody had looked at them. It downloaded EVERY real CFTC annual file (41 Legacy years, 17 Disaggregated, 17 TFF, about 1.6M canonical rows) and put all of it through the two new raising checks. That sweep is the most valuable result of either review: NO legitimate duplicate natural key exists in 40 years of any report type, and the null rate on long/short/open_interest is exactly 0.0 in every (report_type, category) group of every year. Neither new raise can hard-fail a historical backfill. Fixes 4 and 5 came back sound, with fix 5's 11.1% flat-week figure reproducing to the row. TWO FIXES CLOSED THEIR REPRODUCER BUT NOT THEIR CLASS. 1. Fix 1 re-pointed restatement_suspect at the content-bearing snapshot but left the FIRST/CHANGED classification keyed to `prev`. So when the FIRST record for a URL was a fetch failure, the recovering fetch still fired the false restatement alert, with outcome and restatement_suspect disagreeing. _annotate now takes `content` and classifies FIRST off it. 2. Fix 3 corrected the observed_at stamp but not the COMPARISON: ingest_canonical still diffed against whatever was newest in the store, so --all-snapshots fabricated a reversed revision (300 -> 100) plus a re-revision and grew revisions/ without bound on every pass. The diff is now filtered to observed_at <= observed_ts, which makes it genuinely bitemporal. A replay is a true no-op, verified three passes deep on the real captured snapshots. FOUR FURTHER FINDINGS, ALL FIXED. - The blind-detector alert COULD NOT FIRE AT THE YEAR ROLLOVER. It needed prev_outcome == deduped, but a year that churned through 2025 has prev_outcome == changed on the January morning it becomes frozen-in-window. The rollover is the likeliest moment for CFTC's window to shift, so that was the worst possible blind spot. Now keys on whether the previous fetch saw bytes at all. - A FETCH FAILURE WAS A TRIPWIRE CONDITION. Connectivity is not provenance: a dropped connection says nothing about whether CFTC restated anything, and routing it there turned one blip into a frozen-year restatement alarm. Failures are counted and printed by fetch, where an operational problem belongs. - A RUN WHERE EVERY SNAPSHOT FAILED TO PARSE EXITED 0, reporting success to Task Scheduler while the store gained nothing and `failed` stayed terminal. Now non-zero, in ONE combined message: a first draft raised on the failure and silently swallowed a restatement suspect in the same run, which is the more serious of the two. `ingest --retry-failed` is the way back, and it is the only one, since nothing else ever wrote parse_status back to pending and two separate defects had stranded whole backlogs there. - DISAGG AND TFF WERE FETCHED FROM 2006, but cftc.gov 404s for fut_disagg_txt_2006..2009 and fut_fin_txt_2006..2009 (verified live). Every `fetch --all` recorded eight permanent, unfixable failure snapshots. Corrected to 2010. - THE NULL BAND WAS VACUOUS FOR LEGACY, because canonicalize_legacy never coerced: a value arriving as "200,000" stayed an object column, passed every check, and hashed differently from the numeric form. Exactly the fabricated-revision failure the band prevents, on the one report type where it could not see it. Legacy now coerces like the other two, and it is PROVED not to move any stored hash: across 95 markets and 448,236 canonical rows the real values are already int64, so it is an identity. Verified: 257 tests pass, ruff clean, and a full re-ingest of the real captured snapshots is unchanged at 82,776 observations with both a plain re-run and an --all-snapshots replay writing zero rows. Co-Authored-By: Claude Opus 5 --- docs/design/cot_vintage.md | 55 ++++++++++++++++++ src/cotdata/vintage.py | 77 ++++++++++++++++++------- src/cotdata/vintage_cli.py | 64 ++++++++++++++++----- src/cotdata/vintage_ingest.py | 30 +++++++++- tests/test_vintage_canonicalize.py | 21 +++++++ tests/test_vintage_capture.py | 91 ++++++++++++++++++++++++++++++ tests/test_vintage_ingest.py | 68 +++++++++++++++++++++- 7 files changed, 367 insertions(+), 39 deletions(-) diff --git a/docs/design/cot_vintage.md b/docs/design/cot_vintage.md index 1eb070a..7428c93 100644 --- a/docs/design/cot_vintage.md +++ b/docs/design/cot_vintage.md @@ -545,6 +545,61 @@ announcements would be guessing**, and a release date resolved by a guess is wor `derived` one, because it carries a provenance flag claiming it was announced. Left unbuilt, and recorded here rather than quietly assumed to work. +### Second review pass, on the six fixes themselves + +The fixes were then reviewed by a second cold agent, because they had been written by the +same person who wrote the code being fixed and nobody had looked at them. It downloaded +**every real CFTC annual file, 41 Legacy years plus 17 Disaggregated and 17 TFF years, +roughly 1.6 million canonical rows**, and put all of it through the two new raising checks. + +Its verdict on fix 6 is the most valuable result in either review: **no legitimate +duplicate natural key exists in 40 years of any report type, and the null rate on +`long_contracts` / `short_contracts` / `open_interest` is exactly 0.0 in every +`(report_type, category)` group of every year.** So neither new raise can hard-fail a +historical backfill. Fixes 4 and 5 came back sound, with fix 5's 11.1% figure reproducing +to the row. Fix 2 is sound for every path this deployment can reach. + +Two fixes closed their reproducer but not their class, and both are now finished: + +| Was | Now | +|---|---| +| Fix 1 re-pointed `restatement_suspect` at the content-bearing snapshot but left the FIRST/CHANGED classification keyed to `prev`, so a fetch failure as the **first** record for a URL still fired the false restatement alert, with the two signals disagreeing | `_annotate` takes `content` and classifies FIRST from it | +| Fix 3 corrected the `observed_at` stamp but not the comparison: `ingest_canonical` still diffed against whatever was newest, so `--all-snapshots` fabricated a reversed revision plus a re-revision and grew `revisions/` without bound on every pass | The diff is filtered to `observed_at <= observed_ts`, making the comparison genuinely bitemporal. A replay is now a true no-op, verified three passes deep on real data | + +Four further findings, all fixed: + +- **The blind-detector alert could not fire at the year rollover.** It required + `prev_outcome == deduped`, but a year that churned all through 2025 has + `prev_outcome == changed` on the January morning it becomes frozen-in-window. The + rollover is the most likely moment for CFTC's window to shift, which made it the worst + possible blind spot. Now keys on whether the previous fetch saw bytes at all. +- **A fetch failure was a tripwire condition.** Connectivity is not provenance: a dropped + connection says nothing about whether CFTC restated anything, and routing it here turned + one blip into a frozen-year restatement alarm. Failures are now counted and printed by + `fetch`, where an operational problem belongs. +- **A run where every snapshot failed to parse exited 0**, reporting success to Task + Scheduler while the store gained nothing, with `failed` being terminal. Now non-zero, in + a single combined message: an earlier draft raised on the failure first and silently + swallowed a restatement suspect in the same run, which is the more serious of the two. + **`ingest --retry-failed`** is the way back, and it is the only one: nothing else ever + wrote `parse_status` back to `pending`, and two separate defects had stranded whole + backlogs there. +- **Disaggregated and TFF were fetched from 2006**, but cftc.gov serves 404 for + `fut_disagg_txt_2006..2009` and `fut_fin_txt_2006..2009` (verified live in both the + review and here). Every `fetch --all` recorded eight permanent failure snapshots that + could never succeed. First year corrected to 2010. +- **The null band was vacuous for Legacy**, because `canonicalize_legacy` never coerced. A + value arriving as `"200,000"` stayed an object column, passed every check, and hashed + differently from the numeric form: precisely the fabricated-revision failure the band + exists to prevent, on the one report type where it could not see it. Legacy now coerces + like the other two. **Proved not to move any stored hash**: across 95 markets and 448,236 + canonical rows the real values are already `int64`, so the coercion is an identity. + +One caveat carried forward from the review rather than fixed: fix 5's headline 11.1% is +measured over all 418 markets in the 2026 file. Over the 51 markets in the current-state +store the flat-week rate is 0.07%. Both numbers are honest; they describe different +populations, and the tracked universe is much less affected than the full CFTC file. + ### Known and accepted `ingest_canonical` reloads every observation and builds a per-key dict with `iterrows()` diff --git a/src/cotdata/vintage.py b/src/cotdata/vintage.py index 8838b95..a764004 100644 --- a/src/cotdata/vintage.py +++ b/src/cotdata/vintage.py @@ -115,9 +115,17 @@ def _legacy_zip_url(year: int) -> str: return f"https://www.cftc.gov/files/dea/history/dea_fut_xls_{year}.zip" +# Disaggregated and TFF history at these URLs starts in 2010, NOT 2006. The reports +# themselves begin in 2006, which is what the providers record and what an earlier version +# of this list used, but cftc.gov serves 404 for fut_disagg_txt_2006..2009 and +# fut_fin_txt_2006..2009 (verified live in review). Using 2006 made every `fetch --all` +# record eight permanent failure snapshots that could never succeed. +_DISAGG_TFF_FIRST_YEAR = 2010 + + def annual_sources(year: int) -> list[Source]: out = [Source("legacy", "annual_zip", "zip", _legacy_zip_url(year), year)] - if year >= 2006: + if year >= _DISAGG_TFF_FIRST_YEAR: out.append(Source("disaggregated", "annual_zip", "zip", f"https://www.cftc.gov/files/dea/history/fut_disagg_txt_{year}.zip", year)) out.append(Source("tff", "annual_zip", "zip", @@ -326,6 +334,10 @@ def capture_expectation(report_year: int | None, current_year: int) -> str: OUTCOME_NOT_MODIFIED = "not_modified" # 304, server declined to re-send OUTCOME_FAILED = "failed" +# Outcomes where bytes actually arrived and could therefore be hashed. "Did we see content +# last time?" is the question the blind-detector transition turns on. +_SAW_BYTES = (OUTCOME_FIRST, OUTCOME_CHANGED, OUTCOME_DEDUPED) + def _tripwire_alert(expectation: str, outcome: str, prev_outcome: str | None) -> str | None: """The alert reason for one capture, or None. @@ -336,12 +348,25 @@ def _tripwire_alert(expectation: str, outcome: str, prev_outcome: str | None) -> bytes sitting next to the old ones, so it is diffable, and two restatements in consecutive weeks are two things worth knowing about, not one. - **The detector went blind** (a frozen-in-window year that 304s or fails) alerts only on - the TRANSITION into that state. It is a standing condition: if CFTC stopped re-touching - the prior year's Last-Modified, a level-triggered alert would fire every single day - forever, and an alert that never clears is one that gets ignored. Same reasoning - that scopes the ingest revision alert to this run's snapshots. Edge-triggering means it - is reported once, loudly, and then stays quiet until the pattern changes back. + **The detector went blind** (a frozen-in-window year that starts returning 304) alerts + only on the TRANSITION into that state. It is a standing condition: if CFTC stopped + re-touching the prior year's Last-Modified, a level-triggered alert would fire every + single day forever, and an alert that never clears is one that gets ignored. Same + reasoning that scopes the ingest revision alert to this run's snapshots. + + The transition is "last time we received bytes, this time we did not", which is why it + keys on ``_SAW_BYTES`` rather than on ``deduped`` alone. Requiring ``deduped`` + specifically left a hole at the YEAR ROLLOVER, found in review: a year that churned all + through 2025 has ``prev_outcome == changed`` on the January morning it becomes + frozen-in-window, so if CFTC stopped re-serving it at exactly that boundary the + detector went blind and never said so. The rollover is the single most likely moment + for CFTC's regeneration window to shift, which made it the worst possible blind spot. + + A FETCH FAILURE is deliberately NOT a tripwire condition. Connectivity is not + provenance: a dropped connection says nothing about whether CFTC restated anything, and + routing it here turned one blip into a frozen-year restatement alarm on the daily run. + Failures are counted and printed by ``fetch`` itself, which is where an operational + problem belongs. Note what is NOT an alert: a frozen year outside the two-year window returning 304. That is the correct, expected outcome there, and it is also the reason the prior-year @@ -352,10 +377,10 @@ def _tripwire_alert(expectation: str, outcome: str, prev_outcome: str | None) -> return ("content changed on the frozen prior year: CFTC re-serves this file " "weekly and it has been byte-identical, so a new sha here is the " "retroactive-restatement signature") - if outcome in (OUTCOME_NOT_MODIFIED, OUTCOME_FAILED) and prev_outcome == OUTCOME_DEDUPED: - return (f"the frozen prior year stopped being re-served ({outcome}) after " - f"previously deduping. The weekly content check has gone blind, so a " - f"restatement would no longer be detected here") + if outcome == OUTCOME_NOT_MODIFIED and prev_outcome in _SAW_BYTES: + return ("the frozen prior year stopped being re-served (304) after previously " + "returning content. The weekly content check has gone blind, so a " + "restatement would no longer be detected here") return None if expectation == EXPECT_FROZEN_OUT_OF_WINDOW and outcome == OUTCOME_CHANGED: return ("content changed on a closed year outside CFTC's regeneration window, " @@ -363,15 +388,25 @@ def _tripwire_alert(expectation: str, outcome: str, prev_outcome: str | None) -> return None -def _annotate(rec: dict, *, prev: dict | None, now: dt.datetime) -> dict: - """Attach expectation / outcome / tripwire_alert to a capture record.""" +def _annotate(rec: dict, *, prev: dict | None, content: dict | None, + now: dt.datetime) -> dict: + """Attach expectation / outcome / tripwire_alert to a capture record. + + ``prev`` is the previous snapshot of any kind and answers "what happened last time?". + ``content`` is the previous snapshot that actually carried bytes and answers "have we + ever seen this file?". Classifying FIRST off ``prev`` was the half of the earlier + dedupe fix that got missed: when the first record for a URL is a fetch failure, the + recovering fetch found ``prev`` non-None, fell through to CHANGED, and fired the same + false restatement alert the fix existed to remove. It also left ``outcome`` and + ``restatement_suspect`` disagreeing, since only the latter had been re-pointed. + """ expectation = capture_expectation(rec.get("report_year"), now.year) note = rec.get("note") or "" if note.startswith("fetch failed"): outcome = OUTCOME_FAILED elif rec.get("http_status") == 304: outcome = OUTCOME_NOT_MODIFIED - elif prev is None: + elif content is None: outcome = OUTCOME_FIRST elif rec.get("note") == "unchanged bytes (deduped)": outcome = OUTCOME_DEDUPED @@ -442,7 +477,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": None, "local_path": (content or {}).get("local_path"), "note": "304 not-modified", - }, prev=prev, now=now) + }, prev=prev, content=content, now=now) floor = _MIN_BYTES.get(source.source_kind, _MIN_BYTES_DEFAULT) if len(res.content) < floor: @@ -464,7 +499,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": len(res.content), "local_path": content.get("local_path"), "note": "unchanged bytes (deduped)", - }, prev=prev, now=now) + }, prev=prev, content=content, now=now) # New bytes: retain immutably, via a .part file + atomic replace. A crash during a # plain write_bytes would leave a TRUNCATED file already carrying the full sha in its @@ -509,7 +544,7 @@ def capture_source(source: Source, *, snapshots: list[dict], http_get, now: dt.d "byte_size": len(res.content), "local_path": rel, "note": None, - }, prev=prev, now=now) + }, prev=prev, content=content, now=now) def fetch(year: int | None = None, *, all_years: bool = False, @@ -565,7 +600,9 @@ def fetch(year: int | None = None, *, all_years: bool = False, # and dies at the first naming variant or missing disagg year otherwise. The # failure is RECORDED (so it is visible and retryable), then the run goes on — # matching what the ingest path already does. - rec = _failure_record(src, now, e, prev=_latest_for_url(snapshots, src.url)) + rec = _failure_record(src, now, e, + prev=_latest_for_url(snapshots, src.url), + content=_latest_with_content(snapshots, src.url)) failed += 1 print(f" {src.report_type}/{src.source_kind} {src.report_year or ''}: " f"fetch failed — {e}") @@ -596,7 +633,7 @@ def fetch(year: int | None = None, *, all_years: bool = False, def _failure_record(source: Source, now: dt.datetime, error: Exception, - *, prev: dict | None = None) -> dict: + *, prev: dict | None = None, content: dict | None = None) -> dict: retrieved_at = _iso(now) return _annotate({ "source_url": source.url, @@ -614,4 +651,4 @@ def _failure_record(source: Source, now: dt.datetime, error: Exception, "parse_status": "pending", "parse_error": None, "note": f"fetch failed: {error}", - }, prev=prev, now=now) + }, prev=prev, content=content, now=now) diff --git a/src/cotdata/vintage_cli.py b/src/cotdata/vintage_cli.py index 600a23f..ac3390c 100644 --- a/src/cotdata/vintage_cli.py +++ b/src/cotdata/vintage_cli.py @@ -21,7 +21,8 @@ def _cmd_fetch(args) -> int: include_weekly=not args.no_weekly, include_prior_year=not args.no_prior_year) print(f"vintage fetch: {res['checks']} source(s) checked, " - f"{res['new_files']} new raw file(s) retained.") + f"{res['new_files']} new raw file(s) retained" + + (f", {res['failed']} FAILED." if res.get("failed") else ".")) for rec in res["records"]: print(f" {rec['report_type']:<13} {rec['source_kind']:<13} " f"{rec.get('report_year') or '':<6} {rec.get('expectation') or '':<22} " @@ -87,6 +88,18 @@ def _cmd_ingest(args) -> int: snaps = vintage.read_snapshots() if args.snapshot: snaps = [s for s in snaps if s.get("snapshot_id") == args.snapshot] + elif args.retry_failed: + # The ONLY route back from parse_status="failed", which is otherwise terminal: + # --pending does not select it and nothing else ever writes the field back. Two + # separate defects have reached that state en masse (a stale lock, and a path that + # would not resolve), and in both cases the entire backlog was stranded with no + # command able to recover it. Raw bytes are retained, so a retry is always safe. + for s_ in [x for x in snaps if x.get("parse_status") == "failed"]: + vintage.update_snapshot(s_["snapshot_id"], parse_status="pending", + parse_error=None) + snaps = vintage.read_snapshots() + snaps = [x for x in snaps if x.get("parse_status") == "pending"] + print(f"vintage ingest: reset {len(snaps)} failed snapshot(s) to pending.") elif not args.all_snapshots: # Default to pending even with no flag. A bare `ingest` used to select EVERY # snapshot ever recorded and re-ingest each one, which is not a no-op: replaying @@ -98,7 +111,7 @@ def _cmd_ingest(args) -> int: # than doing nothing. --all-snapshots still allows a deliberate full replay. snaps = [s for s in snaps if s.get("parse_status") == "pending"] canon = _canonicalisers() - total_obs = total_rev = 0 + total_obs = total_rev = n_failed = 0 for s in snaps: fn = canon.get(s.get("report_type")) if fn is None or s.get("source_kind") != "annual_zip": @@ -131,8 +144,10 @@ def _cmd_ingest(args) -> int: print(f" {s['snapshot_id']}: +{res['observations']} obs, +{res['revisions']} rev") except Exception as e: # noqa: BLE001 — record the failure, don't abort the batch vintage.update_snapshot(s["snapshot_id"], parse_status="failed", parse_error=str(e)) - print(f" {s['snapshot_id']}: FAILED — {e}") - print(f"vintage ingest: {total_obs} new observation(s), {total_rev} revision(s).") + n_failed += 1 + print(f" {s['snapshot_id']}: FAILED: {e}") + print(f"vintage ingest: {total_obs} new observation(s), {total_rev} revision(s)" + + (f", {n_failed} FAILED." if n_failed else ".")) # Surface revisions rather than only recording them. A scheduled run's stdout goes # nowhere, so a silent exit-0 after detecting a retroactive restatement would defeat @@ -147,18 +162,34 @@ def _cmd_ingest(args) -> int: # need the same exit path: this is the only step that can exit non-zero without # skipping downstream work. alerts = [s for s in snaps if s.get("tripwire_alert")] + if not (total_rev or suspects or alerts or n_failed): + return 0 + + # ONE exit describing everything noteworthy. A parse failure and a restatement suspect + # in the same run are two separate things the operator needs, and an earlier draft + # raised on the failure first, which silently swallowed the suspect: the more serious + # of the two, reported by the less serious one arriving first. + _report_revisions(total_rev, suspects, alerts) + parts = [] if total_rev or suspects or alerts: - _report_revisions(total_rev, suspects, alerts) - parts = [f"{total_rev} revision(s)"] - if suspects: - parts.append(f"{len(suspects)} closed-year restatement suspect(s)") - if alerts: - parts.append(f"{len(alerts)} frozen-year tripwire alert(s)") - raise SystemExit( - "cotdata-vintage: " + ", ".join(parts) - + " — review with 'cotdata-vintage diff'. Non-zero so a scheduler surfaces this; " - "the data IS committed, this is a notification, not a failure.") - return 0 + parts.append(f"{total_rev} revision(s)") + if suspects: + parts.append(f"{len(suspects)} closed-year restatement suspect(s)") + if alerts: + parts.append(f"{len(alerts)} frozen-year tripwire alert(s)") + # A run where every snapshot failed to parse used to exit 0, because non-zero was + # reserved for revisions. On an unattended producer that reports success to Task + # Scheduler while the store gains nothing, and 'failed' is terminal, so it was a silent + # data-loss route rather than a cosmetic one. + if n_failed: + parts.append(f"{n_failed} snapshot(s) FAILED to parse (raw bytes retained, so " + f"nothing is lost: fix the cause and re-run with " + f"'cotdata-vintage ingest --retry-failed', which is the only way back " + f"since 'failed' is not 'pending')") + raise SystemExit( + "cotdata-vintage: " + "; ".join(parts) + + ". Review with 'cotdata-vintage diff'. Non-zero so a scheduler surfaces this; " + "any data shown above IS committed, so this is a notification, not a rollback.") def _report_revisions(total_rev: int, suspects: list, alerts: list = ()) -> None: @@ -282,6 +313,9 @@ def main(argv=None) -> int: g.add_argument("--snapshot", default=None, help="Ingest one snapshot id.") g.add_argument("--pending", action="store_true", help="Ingest all parse_status=pending. This is also the default.") + g.add_argument("--retry-failed", action="store_true", dest="retry_failed", + help="Reset parse_status=failed snapshots to pending and re-ingest " + "them. The only way back from a failed parse.") g.add_argument("--all-snapshots", action="store_true", dest="all_snapshots", help="Replay EVERY recorded snapshot, including already-ingested ones. " "Only for rebuilding a store from retained raw bytes.") diff --git a/src/cotdata/vintage_ingest.py b/src/cotdata/vintage_ingest.py index 39f50ad..a5c41e8 100644 --- a/src/cotdata/vintage_ingest.py +++ b/src/cotdata/vintage_ingest.py @@ -157,7 +157,20 @@ def canonicalize_legacy(wide: pd.DataFrame, *, combined: bool = False) -> pd.Dat "cr4_net_long": pd.NA, "cr4_net_short": pd.NA, "cr8_net_long": pd.NA, "cr8_net_short": pd.NA, }) - return pd.DataFrame(rows) + out = pd.DataFrame(rows) + # Coerce exactly as the disagg/TFF path does. Without this the null-rate band in + # validate() is VACUOUS for Legacy (found in review): a value arriving as "200,000" + # after a CFTC format change stays an object column, passes every check, and hashes + # differently from the numeric form, which is precisely the fabricated-revision failure + # the band exists to prevent, on the one report type where it could not see it. + # + # Confirmed not to move any stored hash: across all 95 markets and 40 years the real + # values are already int64, so to_numeric is an identity and row_sha256 is unchanged. + # _norm normalises int/float/numpy alike, so the dtype could not have mattered anyway. + for f in VALUE_FIELDS: + if f in out.columns: + out[f] = pd.to_numeric(out[f], errors="coerce") + return out # ── Disaggregated / TFF canonicalisation ──────────────────────────────────── @@ -574,7 +587,20 @@ def ingest_canonical(canonical: pd.DataFrame, *, snapshot_id: str, observed_at = dt.datetime.now(dt.timezone.utc).replace(microsecond=0) observed_ts = _naive_utc(observed_at) - prior = _latest_by_key(read_observations()) + # Diff against what was known AS OF this snapshot's observed_at, not against whatever + # is newest in the store. Without the filter the comparison is not bitemporal at all: + # re-ingesting an older snapshot compares it to a LATER value and emits a reversed + # revision (300 -> 100) plus a re-revision, growing revisions/ without bound on every + # replay. Found in review, after a first fix corrected only the observed_at stamp. + # + # On the forward path this changes nothing, because observed_at is then the newest + # timestamp in the store and the filter admits every row. On a replay it makes the + # operation a true no-op: the row the snapshot itself wrote is the latest as of its own + # observed_at, so the hash matches and nothing is written. + obs = read_observations() + if not obs.empty: + obs = obs[obs["observed_at"] <= observed_ts] + prior = _latest_by_key(obs) prior_by_key = {} for _, r in prior.iterrows(): prior_by_key[tuple(r[k] for k in NATURAL_KEY)] = r diff --git a/tests/test_vintage_canonicalize.py b/tests/test_vintage_canonicalize.py index b366c3e..81dc00c 100644 --- a/tests/test_vintage_canonicalize.py +++ b/tests/test_vintage_canonicalize.py @@ -238,3 +238,24 @@ def test_suppressed_trader_counts_do_not_trip_the_null_band(): "Traders_M_Money_Long_All", "Traders_M_Money_Short_All", "Traders_Other_Rept_Long_All", "Traders_Other_Rept_Short_All")}) assert vi.validate(vi.canonicalize_disagg(allsupp)) == [] + + +def test_the_null_band_is_not_vacuous_for_legacy(): + """Second review pass. canonicalize_legacy did not coerce, so a Legacy value arriving + as "200,000" stayed an object column, passed every check, and hashed differently from + the numeric form: exactly the fabricated-revision failure the band exists to prevent, + on the one report type where it could not see it.""" + import pandas as _pd + + from cotdata import vintage_ingest as vi + wide = _pd.DataFrame([{ + "CFTC_Contract_Market_Code": "088691", "Market_and_Exchange_Names": "GOLD", + "Open_Interest_All": 1000, "Comm_Positions_Long_All": "200,000", + "Comm_Positions_Short_All": 400, "NonComm_Positions_Long_All": 300, + "NonComm_Positions_Short_All": 100, "NonRept_Positions_Long_All": 50, + "NonRept_Positions_Short_All": 50, "Traders_Comm_Long_All": 5, + "Traders_Comm_Short_All": 5, "Traders_NonComm_Long_All": 5, + "Traders_NonComm_Short_All": 5, + }], index=_pd.DatetimeIndex([_pd.Timestamp("2026-07-21")])) + with pytest.raises(vi.ValidationError, match="null"): + vi.validate(vi.canonicalize_legacy(wide)) diff --git a/tests/test_vintage_capture.py b/tests/test_vintage_capture.py index 09ed008..e257148 100644 --- a/tests/test_vintage_capture.py +++ b/tests/test_vintage_capture.py @@ -493,3 +493,94 @@ def http(url, *, etag=None, last_modified=None): rec = res["records"][0] assert rec["content_sha256"] is not None assert rec["local_path"] is not None + + +def test_a_failure_as_the_FIRST_record_does_not_fire_a_false_tripwire(store_env): + """Second review pass. The first fix re-pointed restatement_suspect at the + content-bearing snapshot but left the FIRST/CHANGED classification keyed to `prev`, so + when the very first record for a URL was a fetch failure the recovering fetch still + fired the false restatement alert, with outcome and restatement_suspect disagreeing.""" + from cotdata import vintage + from cotdata.vintage import HttpResult + body = _body(b"2025-frozen") + n = [0] + + def http(url, *, etag=None, last_modified=None): + n[0] += 1 + if n[0] == 1: + raise ConnectionError("network down on the very first attempt") + return HttpResult(200, body, etag=None, last_modified="lm1") + + now, src = _clock(), _one(LEGACY_2025, 2025) + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + res = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) + + rec = res["records"][0] + assert rec["outcome"] == vintage.OUTCOME_FIRST # never seen bytes before, so FIRST + assert rec["tripwire_alert"] is None + assert not rec["restatement_suspect"] # and the two signals agree + + +def test_the_blind_alert_fires_at_the_year_rollover(store_env): + """Second review pass. The blind trigger required prev_outcome == deduped, but a year + that churned all through 2025 has prev_outcome == changed on the January morning it + becomes frozen-in-window. The rollover is the single most likely moment for CFTC's + regeneration window to shift, so it was the worst possible blind spot.""" + import datetime as dt + + from cotdata import vintage + from cotdata.vintage import HttpResult + n = [0] + + def http(url, *, etag=None, last_modified=None): + n[0] += 1 + if n[0] == 1: + return HttpResult(200, _body(b"dec"), etag=None, last_modified="lm1") + return HttpResult(304, None) + + src = _one(LEGACY_2025, 2025) + dec = lambda: dt.datetime(2025, 12, 26, 17, tzinfo=dt.timezone.utc) # noqa: E731 + jan = lambda: dt.datetime(2026, 1, 2, 17, tzinfo=dt.timezone.utc) # noqa: E731 + r1 = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=dec) + assert r1["records"][0]["expectation"] == vintage.EXPECT_CHURN # 2025 in 2025 + r2 = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=jan) + + rec = r2["records"][0] + assert rec["expectation"] == vintage.EXPECT_FROZEN_IN_WINDOW # 2025 in 2026 + assert rec["outcome"] == vintage.OUTCOME_NOT_MODIFIED + assert rec["tripwire_alert"] and "gone blind" in rec["tripwire_alert"] + + +def test_a_fetch_failure_is_not_a_tripwire_condition(store_env): + """Connectivity is not provenance. A dropped connection says nothing about whether + CFTC restated anything, and routing it into the frozen-year alarm turned one blip into + a restatement alert on the daily run. Failures are counted by fetch instead.""" + from cotdata import vintage + from cotdata.vintage import HttpResult + body = _body(b"frozen") + n = [0] + + def http(url, *, etag=None, last_modified=None): + n[0] += 1 + if n[0] <= 2: + return HttpResult(200, body, etag=None, last_modified="lm1") + raise ConnectionError("blip") + + now, src = _clock(), _one(LEGACY_2025, 2025) + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # first + vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # deduped + res = vintage.fetch(sources=src, http_get=http, rate_limit_s=0, now_fn=now) # fails + + assert res["records"][0]["outcome"] == vintage.OUTCOME_FAILED + assert res["tripwire_alerts"] == [] # no alarm + assert res["failed"] == 1 # but it IS counted + + +def test_disagg_and_tff_start_in_2010_not_2006(): + """cftc.gov serves 404 for fut_disagg_txt_2006..2009 and fut_fin_txt_2006..2009 + (verified live), so 2006 made every `fetch --all` record eight permanent failure + snapshots that could never succeed.""" + from cotdata import vintage + assert [s.report_type for s in vintage.annual_sources(2009)] == ["legacy"] + assert {s.report_type for s in vintage.annual_sources(2010)} == { + "legacy", "disaggregated", "tff"} diff --git a/tests/test_vintage_ingest.py b/tests/test_vintage_ingest.py index 19ef861..def2f64 100644 --- a/tests/test_vintage_ingest.py +++ b/tests/test_vintage_ingest.py @@ -221,7 +221,12 @@ def test_cli_ingest_exits_nonzero_when_revisions_are_recorded(store_env, capsys) vintage_cli.main(["ingest", "--pending"]) msg = str(exc.value) assert "restatement suspect" in msg and "cotdata-vintage diff" in msg - assert "notification, not a failure" in msg # the data IS committed + # the unparseable fixture ALSO fails, and both must appear: an earlier draft raised on + # the failure first and swallowed the suspect, which is the more serious of the two + assert "FAILED to parse" in msg + # wording is "not a rollback" rather than "not a failure": with a parse failure in + # the same message, calling the whole run not-a-failure would be untrue + assert "notification, not a rollback" in msg # the data IS committed assert "RESTATEMENT SUSPECT" in capsys.readouterr().out @@ -242,7 +247,9 @@ def test_unparseable_sources_drain_out_of_pending(store_env): "local_path": "gone.zip", "parse_status": "pending", "retrieved_at": "2026-07-31T00:00:00Z"}, ]}) - assert vintage_cli.main(["ingest", "--pending"]) == 0 + # the missing file makes d1 fail, and a failed parse is itself now a non-zero exit + with pytest.raises(SystemExit, match="FAILED to parse"): + vintage_cli.main(["ingest", "--pending"]) after = {s["snapshot_id"]: s for s in vintage.read_snapshots()} assert after["w1"]["parse_status"] == "skipped" @@ -344,3 +351,60 @@ def test_a_replayed_snapshot_keeps_its_own_observed_at(store_env): now = vi.asof(pd.Timestamp("2026-08-01"), report_date="2026-07-21") comm = now[now.category == "commercial"] assert list(comm["long_contracts"]) == [200] # the newer value still wins + + +def test_replaying_an_old_snapshot_is_a_true_no_op(store_env): + """Second review pass. Taking observed_at from the snapshot fixed the TIMESTAMP but not + the COMPARISON: ingest_canonical still diffed against whatever was newest in the store, + so a replay emitted a reversed revision (300 -> 100) plus a re-revision, and grew + revisions/ without bound on every pass. The diff is now bitemporal.""" + from cotdata import vintage_ingest as vi + caps = [("s1", 100, "2026-07-24T19:00:00Z"), ("s2", 200, "2026-07-31T19:00:00Z"), + ("s3", 300, "2026-08-07T19:00:00Z")] + for sid, v, at in caps: + vi.ingest_canonical(_canon("2026-07-21", comm_long=v), snapshot_id=sid, + observed_at=at) + n_obs, n_rev = len(vi.read_observations()), len(vi.read_revisions()) + assert n_rev == 2 # 100 -> 200 -> 300, both forward + + for _ in range(3): # three full replays + for sid, v, at in caps: + vi.ingest_canonical(_canon("2026-07-21", comm_long=v), snapshot_id=sid, + observed_at=at) + assert (len(vi.read_observations()), len(vi.read_revisions())) == (n_obs, n_rev) + rev = vi.read_revisions() + assert list(rev["old_value"]) == ["100", "200"] # no reversed pair was invented + assert list(rev["new_value"]) == ["200", "300"] + + +def test_retry_failed_is_the_way_back_from_a_failed_parse(store_env): + """`failed` is otherwise terminal: --pending does not select it and nothing else writes + the field back. Two separate defects stranded whole backlogs there.""" + import pytest as _pytest + + from cotdata import vintage, vintage_cli + vintage._write_manifest({"schema_version": 1, "snapshots": [ + {"snapshot_id": "d1", "report_type": "legacy", "source_kind": "annual_zip", + "local_path": "gone.zip", "parse_status": "pending", + "retrieved_at": "2026-07-31T00:00:00Z"}]}) + with _pytest.raises(SystemExit, match="FAILED to parse"): + vintage_cli.main(["ingest", "--pending"]) + assert vintage.read_snapshots()[0]["parse_status"] == "failed" + + assert vintage_cli.main(["ingest", "--pending"]) == 0 # terminal: not re-selected + with _pytest.raises(SystemExit): # but --retry-failed does + vintage_cli.main(["ingest", "--retry-failed"]) + + +def test_a_run_where_every_snapshot_failed_does_not_report_success(store_env): + """It used to exit 0, because non-zero was reserved for revisions. On an unattended + producer that reports success to Task Scheduler while the store gains nothing.""" + import pytest as _pytest + + from cotdata import vintage, vintage_cli + vintage._write_manifest({"schema_version": 1, "snapshots": [ + {"snapshot_id": f"d{i}", "report_type": "legacy", "source_kind": "annual_zip", + "local_path": f"gone{i}.zip", "parse_status": "pending", + "retrieved_at": "2026-07-31T00:00:00Z"} for i in range(3)]}) + with _pytest.raises(SystemExit, match="3 snapshot"): + vintage_cli.main(["ingest", "--pending"])