Correct the Open Interest label: whole-market, not front-month - #27
Conversation
README's futures-bar table called the stored `Open Interest` column "front-month open interest". It is whole-market, and the mislabel has now cost time twice: it was one of two written notes that nearly led to the conclusion that the data a between-reports positioning estimator needs was missing from the store, when it is present on all 41 npf-universe markets back to a median of 1979. Three independent measurements say whole-market: - cotdata/docs/design/reading-the-store.md section 4: exact agreement with CFTC total-market OI on 25 of 26 markets, palladium at 0.998, median ratio 1.000. - npf (private) re-measured across the full 41-market universe at the current vintage: per-market median ratio 0.9999 to 1.0000 on 41 of 41, median within-market IQR 0.0000, median 1,872 pairs per market. PA is exact at this vintage too. - Spot-checked again here on 14 markets (~24,000 pairs): median ratio 1.0000 on 14 of 14, zero IQR on 12 of them. The README now carries a runnable reproducer for CL, verified to print `1926 1.0`. The same mislabel sat on the neighbouring `Volume` row, which is the identical naming trap the `get_bars` docstring already warns about from the other end: `volume="front"` spans the whole curve while `reconstructed` sums exactly two expiries. Measured on the current store, the median `reconstructed / front` ratio is 0.55 in NG, 0.59 in CL, 0.67 in HE and 0.77 in ZC, so the fuller-sounding parameter is the narrower series. Corrected in the table, in the `Volume_Reconstructed` fall-back row, and in two code comments that stated the direction backwards. Scoped the claim to the Norgate producer, which is what writes this store. The databento producer reads OI from the `.n.0` continuous contract's own statistics (stat_type 9), so its column is that contract's rather than the market's, and the CFTC agreement should not be carried across vendors. docs/design.md was checked and carries no occurrence of the mislabel. Documentation only: no code, no stored data, no behaviour change. `ruff check src tests` clean, 242 passed / 12 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`test_live_store_agrees_with_the_last_declared_regime` already meant to skip when there is no store to check, and said so in its docstring. It could not. `store.read_metadata()` resolves the store root, which raises by design when `MARKETDATA_STORE` is unset, so the guard never got to run: in a plain shell the two parametrized cases failed with a RuntimeError two frames down that reads like a real breakage on an untouched tree. CI hid this. `python-test.yml` sets MARKETDATA_STORE to an empty directory, so `read_metadata` returns an empty frame and the written guard fires on its second arm. The unset case is invisible to CI by construction, which is why a green run never surfaced it. The guard now checks the variable first and skips naming it, then falls through to the existing empty-specs arm. This is the same shape as the npf skip guards that keyed on COTDATA_STORE alone after the price read moved to MARKETDATA_STORE, and produced 20 tracebacks where 26 skips were wanted. Added a test that the guard fires, on this file's own principle that a guard which has never fired is indistinguishable from one that is not wired in. It asserts on the skip REASON rather than merely that a skip happened, since the old RuntimeError also named the variable and still read as a breakage. A blank value is covered alongside an unset one, because `config.store_root` strips before checking and the two are the same mistake. Verified by deleting the guard: the new test fails. Measured in three environments, `ruff check src tests` clean in all: no MARKETDATA_STORE 243 passed, 14 skipped (was 2 failed) empty store, as CI sets it 243 passed, 14 skipped live store 245 passed, 12 skipped (tripwire runs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Pushed
CI hid it. The guard now checks the variable first and skips naming it, then falls through to the existing empty-specs arm. Added a test that the guard fires, on this file's own stated principle that a guard which has never fired is indistinguishable from one that is not wired in. It asserts on the skip reason, not merely that a skip happened, because the old Measured in three environments,
|
1 participant
README.md's "What a futures bar carries" table described the storedOpen Interestcolumn as front-month open interest. It is whole-market, and the label has now cost real time: it was one of two written notes that nearly led to the conclusion that the data a between-reports positioning estimator needs was missing from this store, when it is present on all 41 npf-universe markets back to a median start of 1979.Documentation only. No code, no stored data, no behaviour change.
The evidence
Norgate collects open interest from the exchange; the CFTC collects its own from clearing members. Two vendors, two collection paths, matched on the COT report Tuesday. They agree, three times over:
cotdata/docs/design/reading-the-store.md§4scripts/nowcast_data_check.pyanddocs/npf/nowcast_data_check.mdTwo independent collection paths cannot agree to four decimals on anything but the whole market. A front-month series would sit far below 1.0.
The README now carries a runnable reproducer for CL, verified to print
1926 1.0.The same mislabel on the neighbouring row
The
Volumerow said "front-month volume", which is the identical naming trap theget_barsdocstring already warns about from the other end:volume="front"spans the whole curve, whilereconstructedsounds fuller and isFirstVolume + SecondVolume, exactly two expiries. Measured on the current store (medianreconstructed / frontover the last 1,500 bars):Corrected in the table, in the
Volume_Reconstructedfall-back row, and in two code comments that stated the direction backwards (bars.py::_reconstructed_volumeandproviders/norgate.py::_reconstruct_volumeboth called the wider series "front-month").Scoping
The whole-market reading is a fact about the Norgate producer, which is what writes this store. The databento producer reads OI from the
.n.0continuous contract's own statistics (stat_type9), so itsOpen Interestis that contract's rather than the market's. Stated explicitly so the correction is not carried across vendors into a new error.Checked and unchanged
docs/design.mdcarries no occurrence of the mislabel. Its only mention of open interest (line 235, the full-history-rewrite argument) is correct as written.Verification
ruff check src tests(ruff 0.15.22): cleanpytest tests/ -qwithMARKETDATA_STOREset: 242 passed, 12 skippedBottom line, in plain language: the store has always held whole-market open interest on every futures market. Only the label was wrong, and it was wrong in the one direction that makes a reader conclude the data is unusable. Three separate measurements, one of them run fresh for this PR, put the agreement with the CFTC's own total at four decimal places. This is a genuine and settled correction, not a marginal call.
🤖 Generated with Claude Code