Skip to content

fix: --check measures lag on write time, not on data age - #58

Merged
mspinola merged 1 commit into
mainfrom
fix/lag-check-uses-write-time
Jul 26, 2026
Merged

mspinola merged 1 commit into
mainfrom
fix/lag-check-uses-write-time

Conversation

@mspinola

Copy link
Copy Markdown
Owner

--check flagged six entries on your real store, none of which were a problem. All six were old-but-correct, and the check had no way to tell that apart from a failure.

The wrong question

The check exists to catch a partial run: one entry failing while its peers succeed. It compared each entry's last_date against the domain's newest last_date, which measures how recent the data is. That is a different question.

Two legitimate classes of old-but-correct entry were flagged permanently:

Thin markets that drop out of the CFTC report. Below the reporting threshold a contract stops appearing and returns months later. Measured over 19 years of weekly data:

NKD_240741: 19 gaps >10d, longest 168d  (recent: 168d, 147d)   currently 140d
ZO_004603:  22 gaps >10d, longest 294d  (recent: 294d, 105d)   currently  49d

Both are comfortably inside their own normal.

Retired hist_code contracts, frozen by design. These were suppressed by name via hist_code_names(), except the store also holds bare-code duplicates (23977A alongside RTY_23977A) written before the naming convention gained a symbol prefix. The suppression set only ever generated the prefixed form, so the bare ones warned forever.

The right question

Both classes are rewritten on every producer pass, so both carry a current updated_at:

NKD_240741  last_date 2026-03-03  updated_at 2026-07-26T16:13:07Z
ES_13874A   last_date 2026-07-21  updated_at 2026-07-26T16:13:07Z   <- identical

Comparing updated_at against the domain's newest write separates "the source had nothing newer" from "the producer skipped this". It also makes the bare-duplicate naming mismatch moot, rather than needing the suppression set to cover two conventions.

A hole found while testing

When no entry had a parseable updated_at, the new shape initially reported everything as current. Turning "cannot check" into "all clear" is the precise failure this check exists to prevent, so unusable timestamps are now flagged. There is a test for it.

Verification

On the real store: 6 false positives to 0.

✓ every entry was written by the latest producer pass.

And a simulated skip is still caught:

⚠ cot_legacy: 1 entry NOT WRITTEN in the last run (>3d behind the domain's newest write):
    ES_13874A         last written 2026-07-01T04:00:00Z  (25d behind)

Suite 129 passed, ruff clean.

Note

hist_code_names() and the ignore_lag parameter are kept and still work. They are no longer load-bearing for this case, but they remain the escape hatch for suppressing an entry by name.

--check flagged six entries on the real store, none of which were a problem. All
six were old-but-correct, and the check had no way to tell that from a failure.

The check exists to catch a PARTIAL RUN: one entry failing while its peers
succeed. It was comparing each entry's last_date against the domain's newest
last_date, which measures how recent the DATA is. That is a different question,
and it is the wrong one.

Two legitimate classes of old-but-correct entry were flagged permanently:

  - Thin markets that fall below the CFTC reporting threshold, drop out of the
    report, and reappear months later. NKD_240741 and ZO_004603 each have around
    20 gaps over 10 days across 19 years of weekly data, the longest 168d and
    294d. Their current 140d and 49d are well inside normal.
  - Retired hist_code contracts, frozen by design. Suppressed by name via
    hist_code_names(), except that the store also holds bare-code duplicates
    (23977A alongside RTY_23977A) written before the naming convention included
    a symbol prefix, which the suppression set never matched.

Both classes are rewritten on every producer pass, so both carry a current
updated_at. Comparing updated_at against the domain's newest write separates
"the source had nothing newer" from "the producer skipped this", which is
exactly the distinction the check needed. It also makes the bare-duplicate
naming mismatch moot rather than requiring the suppression set to cover two
conventions.

Also closes a hole found while testing: when no entry had a parseable
updated_at, the old shape silently reported everything as current. "Cannot
check" now renders as flagged, not as clean, since turning an unknown into a
pass is the precise failure this check exists to prevent.

Verified on the real store: 6 false positives to 0, and a simulated skipped
entry is still caught (25d behind). Suite 129 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mspinola
mspinola merged commit 61a0fb3 into main Jul 26, 2026
5 checks passed
@mspinola
mspinola deleted the fix/lag-check-uses-write-time branch July 26, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant