mspinola · Jul 31, 2026 · Jul 30, 2026 · Jul 30, 2026 · Jul 30, 2026 · Jul 30, 2026
diff --git a/.gitignore b/.gitignore
@@ -9,6 +9,9 @@ dist/
 # never commit the data store itself
 /store/
 *.parquet
+# ...except the tiny committed golden fixture that guards current/ byte-identity
+# (tests/test_current_baseline.py). Regenerate with tests/_gen_golden.py.
+!tests/fixtures/golden/*.parquet
 
 # local Task Scheduler / cron wrapper scripts (machine-specific paths, and on
 # Linux a Databento API key). Copy templates from docs/examples/ into here.

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,24 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 ## [Unreleased]
 
 ### Added
+- **COT vintage capture** (`cotdata-vintage fetch`) — an immutable, hashed landing
+  zone for as-published CFTC files under `$COTDATA_STORE/vintage/raw/`, with
+  provenance (etag, last-modified, sha256, size, retrieved-at) recorded in a
+  self-owned `vintage/manifest.json`. Purely additive: the current-state store is
+  byte-identical (guarded by `tests/test_current_baseline.py`). This is step 1 of
+  the vintage/revision-tracking subsystem — capture must start now because an
+  uncaptured weekly release is irrecoverable; ingest/diff/PIT land next and run
+  retroactively over retained raw bytes. Decision recorded in crucible-stack
+  ADR-0008; design in `docs/design/cot_vintage.md`.
+- **COT vintage ingest + revision tracking** (`cotdata-vintage ingest|diff|asof`,
+  `cotdata-schedule sync|backfill`) — parses retained raw snapshots into a change-only
+  bitemporal `observations/` table (a row is written only when its value hash differs
+  from the latest for its natural key, so storage grows with revisions not with time),
+  emits field-level `revisions/` with `age_days` revision depth, and answers
+  point-in-time `asof(t)` reads (greatest `observed_at <= t` per key). Release dates are
+  resolved with explicit provenance (`observed > announced > scheduled > derived`),
+  including a backfill that flags the Oct–Dec 2025 appropriations-lapse backlog weeks as
+  `announced` rather than silently `derived`. All pandas/pyarrow, no database.
 - **`propadj` price adjustment** — a proportional (ratio) back-adjusted view
   derived on read from the stored `unadj` + `backadj` series via
   `get_prices(symbol, adjustment="propadj")`. It preserves daily percentage

diff --git a/README.md b/README.md
@@ -28,7 +28,7 @@ cotdata separates *fetching* data (a "producer" that talks to vendors) from *usi
 
 ## Contents
 
-- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Syncing the store](docs/SYNCING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license)
+- [Quickstart](#quickstart) · [How it works](#how-it-works) · [Reading data](#reading-data-consumer) · [Producing data](#producing-data-producer) · [Windows setup](docs/WINDOWS_SETUP.md) · [Scheduling on Windows](docs/WINDOWS_SCHEDULING.md) · [Scheduling on Linux](docs/LINUX_SCHEDULING.md) · [Syncing the store](docs/SYNCING.md) · [Operations](#operations) · [Concepts & design](#concepts--design) · [COT vintage tracking](#cot-vintage-tracking-as-published-history) · [Reference: schemas](#reference-data-schemas) · [Reference: COT formats](#reference-cot-formats-explained) · [Diagnostics](#diagnostics) · [Development](#development) · [Contributing](#contributing) · [License](#license)
 
 ## Quickstart
 
@@ -85,6 +85,9 @@ The store layout:
 - `metadata/contract_specs.parquet` — Norgate contract specifications (tick size, point value, margin).
 - `manifest.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`.
 - `status.json` — machine-readable new-data signal for downstream tools (see [Operations](#operations)).
+- `vintage/` — optional as-published (vintage) capture: retained raw CFTC downloads plus
+  change-only observations and field-level revisions. Purely additive; the tables above are
+  unchanged whether or not it is enabled. See [COT vintage tracking](#cot-vintage-tracking-as-published-history).
 
 ## Reading data (consumer)
 
@@ -128,6 +131,7 @@ COTDATA_STORE=/store  cotdata-update --cot-legacy                # CFTC Legacy (
 COTDATA_STORE=/store  cotdata-update --cot-disagg                # CFTC Disaggregated (any OS)
 COTDATA_STORE=/store  cotdata-update --cot-tff                   # CFTC Traders in Financial Futures (any OS)
 COTDATA_STORE=/store  cotdata-update --cot-all                   # all three CFTC COT reports
+COTDATA_STORE=/store  cotdata-vintage fetch                      # optional: capture as-published COT (any OS)
 ```
 
 `--prices` with no `--symbols` updates every symbol in the registry; add `--symbols` to scope it. Each run prints a per-symbol line with the date advance (e.g. `ES: … [2026-07-13 -> 2026-07-14]`) and a summary footer (OK/failed counts, rows written, elapsed, newest date). A run **exits non-zero** if a fetch hard-fails (Norgate/CFTC unreachable), so a scheduler can retry — see [Scheduling on Windows](#scheduling-on-windows-task-scheduler).
@@ -226,6 +230,12 @@ Anything a consumer put in the store by hand is a **correctness** issue rather t
 saving. No producer creates it, so a mirroring sync deletes it. Exclude it, but the real
 fix is to keep it out of the store: the store belongs to its producer.
 
+The same rule bites hardest on `vintage/` if you enable it, because that data cannot be
+re-fetched: capture it on the **producer** so it syncs outward, or keep it outside the
+mirrored store with `COTDATA_VINTAGE_ROOT`. Its provenance index is deliberately named
+`snapshots.json`, since the usual `manifest.json` exclusion matches by name at any depth
+and would otherwise strip it in transit, delivering raw archives with no index.
+
 Consumer cloud sync (Dropbox, Google Drive) is a poor fit here: conflict copies land
 inside the store, and on-demand placeholder files break `read_parquet` on the machine
 doing research.
@@ -311,6 +321,67 @@ The supported futures contracts are defined in a YAML registry, so adding a mark
 
 The store uses **atomic writes** (write-temp-then-rename). Consumers can safely query via `get_prices` / `get_cot` even while `cotdata-update` is actively downloading and writing.
 
+### COT vintage tracking (as-published history)
+
+CFTC revises COT data after publication — most consequentially through **trader
+reclassification**, which moves positions between categories retroactively. Because
+downstream signals are rolling z-scores and percentiles against years of history, a
+restatement silently rewrites the baseline every historical reading was computed against.
+There is precedent: in July 2008 the Commission revised reports back to July 3, 2007.
+
+**CFTC serves current state only.** There is no vintage archive and no as-published
+endpoint, so vintage data can only be accumulated going forward — every uncaptured week is
+a permanent blind spot in the part of the series most likely to have been revised.
+
+This is **opt-in and purely additive**: if you never run it, the store behaves exactly as
+before. Enabling it adds a `vintage/` subtree.
+
+```bash
+cotdata-vintage fetch                      # capture current year + weekly static (daily)
+cotdata-vintage fetch --all                # every year 1986-present (see below)
+cotdata-vintage ingest --pending           # parse retained raw -> observations + revisions
+cotdata-vintage diff --since 2026-01-01    # field-level revisions, with revision depth
+cotdata-vintage asof --as-of 2026-07-24T18:00:00 --report-date 2026-07-21
+cotdata-schedule sync                      # CFTC Special Announcements
+cotdata-schedule published                 # true publication dates from retained weekly statics
+cotdata-schedule backfill                  # resolve release_date + its provenance
+```
+
+How it works:
+
+- **Immutable landing zone.** Every fetch is recorded (including 304s) and raw bytes are
+  retained permanently under `vintage/raw/`, written atomically and never rewritten. A
+  byte-identical regeneration is deduped — a changed download is not itself a revision.
+- **Change-only observations.** A row is written only when its value hash differs from the
+  latest for its natural key `(report_date, market_code, report_type, combined, category)`,
+  so storage grows with actual revisions rather than with time.
+- **Field-level revisions** carry `age_days` (revision depth): whether revisions stay in
+  recent weeks or reach back into the calibration window determines how much the rest of
+  a system has to care.
+- **Point-in-time reads.** `asof(t)` returns each key's latest value observed at or before
+  `t`, reconstructing what was actually knowable then.
+- **Release dates with provenance.** `report_date` is stored exactly as reported (never
+  normalized to Tuesday), and `release_date` is resolved through
+  `published > observed > announced > scheduled > derived`, with the source recorded — a
+  release date without provenance is worse than none, since indexing on `report_date`
+  embeds a lookahead (three days normally, weeks during a backlog). `published` is the
+  weekly static's HTTP `Last-Modified`, a true publication timestamp; it is forward-only
+  (that file holds one week and is overwritten), so weeks predating capture fall back
+  down the chain.
+
+**Run capture on the producer, not a replica**, and schedule it **daily**: nearly every
+request returns 304, so a daily run is close to free while catching holiday-shifted and
+backlog releases with no schedule logic. `--all` is a **restatement tripwire** rather than
+a backfill — closed years are byte-frozen, so a checksum change on one is the retroactive-
+restatement signature; monthly or quarterly is the right cadence, and it is cheap because
+almost everything 304s. Full design notes, including the measured CFTC caching behaviour,
+are in [docs/design/cot_vintage.md](docs/design/cot_vintage.md).
+
+> **Replica warning.** The vintage tree must not be written on a machine whose store is
+> mirrored (`robocopy /MIR`, `rsync --delete`) from a producer: the mirror deletes
+> destination-only files and the data is irreplaceable. Capture on the producer, or set
+> `COTDATA_VINTAGE_ROOT` to a path outside the mirrored store. See [docs/SYNCING.md](docs/SYNCING.md).
+
 ## Local development
 
 ```bash

diff --git a/docs/SYNCING.md b/docs/SYNCING.md
@@ -158,6 +158,38 @@ arrival. The per-half files under `manifests/` are disjoint and merge correctly.
 
 Run `cotdata-update --migrate-manifests` once per store, then delete `manifest.json`.
 
+### `vintage/` is irreplaceable, so where it is WRITTEN matters
+
+The vintage tree (`vintage/raw/`, `observations/`, `revisions/`, `snapshots.json`) records
+CFTC data *as published*. CFTC serves current state only and there is no vintage archive,
+so a deleted vintage snapshot can never be re-fetched. Treat it as write-once.
+
+**Capture on the producer, never on a replica.** Vintage capture fetches from CFTC, so it
+is a producer action and belongs beside the COT half (`run-vintage.cmd`, chained after
+`run-cot.cmd`, daily). Written on the producer it propagates outward like any other store
+content. Written on a **replica** it is destroyed by the next `/MIR` or `--delete` pass,
+for the same reason `citpy` is (below): the source has no such directory, so the mirror
+removes it. `citpy` is regenerable; vintage data is not.
+
+If a replica genuinely must capture, point `COTDATA_VINTAGE_ROOT` at a path **outside**
+the mirrored store. That box then holds the only copy, so give it its own backup.
+
+Per replica in this deployment:
+
+| Target | Carries `vintage/`? | Why |
+|---|---|---|
+| Mac (research) | **Yes, in full** | Natural second copy of irreplaceable bytes, ~1 GB/yr, and research may query revisions |
+| Linux dash VPS | **No** | cot-analyzer reads prices and COT only; it would carry ~1 GB/yr of archives it never opens |
+
+**Naming gotcha, already handled:** the vintage provenance index is `snapshots.json`, not
+`manifest.json`. Both sync scripts exclude `manifest.json` *unanchored* (robocopy `/XF`
+and rsync `--exclude` both match by name at any depth), so a `vintage/manifest.json` would
+have been stripped in transit and the replica would receive raw archives with no index.
+Do not rename it back.
+
+Exclude `*.part` alongside `*.tmp`: raw downloads land via a `.part` file plus an atomic
+replace, and a sync running mid-capture must not carry the partial.
+
 ## Check before you mirror
 
 `--delete` and `/MIR` are silent when they destroy something. Run the preflight first:

diff --git a/docs/WINDOWS_SCHEDULING.md b/docs/WINDOWS_SCHEDULING.md
@@ -35,9 +35,22 @@ set COTDATA_STORE=REPLACE_WITH_STORE_PATH
 
 Using the full venv `\Scripts\cotdata-prices.exe` / `\Scripts\cotdata-cot.exe` path (rather than relying on the command being on `PATH`) matters here: Task Scheduler runs with a different, often bare, environment than your interactive shell, so a bare command name that resolves fine in Command Prompt can fail to resolve under the scheduler.
 
+`run-vintage.cmd` — **optional**, the as-published (vintage) capture. Copy
+[`docs/examples/windows/run-vintage.cmd`](examples/windows/run-vintage.cmd); it runs
+`cotdata-vintage fetch` then `ingest --pending`, both exit-code guarded. Two things to know
+before enabling it:
+
+- **It belongs on the producer.** Capture fetches from CFTC, so it is a producer action —
+  and a vintage tree written on a mirrored replica is deleted by the next sync, which is
+  unrecoverable because CFTC serves current state only. See [SYNCING.md](SYNCING.md).
+- **Schedule it daily, not weekly.** Nearly every request returns 304, so a daily run costs
+  almost nothing while catching holiday-shifted and backlog releases with no schedule logic.
+
+
+
 ## Creating the tasks
 
-Create three tasks — times are the **machine's local** time; convert from ET if it isn't on Eastern:
+Create three tasks (plus an optional fourth if you enable vintage capture) — times are the **machine's local** time; convert from ET if it isn't on Eastern:
 
 ```bat
 :: 1) Prices — fire at the Continuous Futures Final (~8:55pm ET); --require-final + restart
@@ -46,6 +59,11 @@ schtasks /Create /TN "cotdata prices" /TR "<DIR>\run-prices.cmd" /SC DAILY /ST 2
 
 :: 2) COT — daily morning catch-up for holiday-delayed releases and as a safety net
 schtasks /Create /TN "cotdata COT (catch-up)" /TR "<DIR>\run-cot.cmd" /SC DAILY /ST 08:10
+
+:: 3) Vintage (OPTIONAL) — as-published capture, ~90 min after the 15:30 ET release.
+::    Daily is deliberate: almost every request 304s, so it is nearly free, and it
+::    tightens the observed release date from a 7-day bound to a 1-day one.
+schtasks /Create /TN "cotdata vintage" /TR "<DIR>\run-vintage.cmd" /SC DAILY /ST 17:00
 ```
 
 > **Substitute `<DIR>` before running these** — with the real folder holding your `.cmd` files, e.g. `C:\Users\you\code\cotdata\scheduler`. `schtasks` takes the quoted `/TR` value as a literal string and **does not check the file exists**, so a leftover `"<DIR>\run-cot.cmd"` is accepted without error and creates a task that fails only when it fires. Verify each task points somewhere real:

diff --git a/docs/adr/ADR-0001-cot-vintage-provenance-in-parquet.md b/docs/adr/ADR-0001-cot-vintage-provenance-in-parquet.md
@@ -0,0 +1,13 @@
+# ADR stub: COT vintage provenance
+
+> **This decision is recorded as crucible-stack ADR-0008**, beside ADR-0007, because
+> its scope half — *is vintage provenance inside narrowed cotdata's boundary?* — is an
+> interpretation of ADR-0007 and must be discoverable from ADR-0007's own directory.
+> This file is a local pointer so the cotdata worktree still surfaces the decision.
+
+**Decision, in one line:** vintage provenance IS in scope for the narrowed `cotdata`
+(it is CFTC-positioning provenance), and it persists in the existing Parquet + manifest
+contract — no database.
+
+- Full ADR: `crucible-stack/docs/adr/ADR-0008-cot-vintage-provenance-in-parquet.md`
+- Design/working notes: [../design/cot_vintage.md](../design/cot_vintage.md)