Repository navigation
fix: --check measures lag on write time, not on data age - #58
Merged
Merged
Conversation
--check flagged six entries on the real store, none of which were a problem. All
six were old-but-correct, and the check had no way to tell that from a failure.
The check exists to catch a PARTIAL RUN: one entry failing while its peers
succeed. It was comparing each entry's last_date against the domain's newest
last_date, which measures how recent the DATA is. That is a different question,
and it is the wrong one.
Two legitimate classes of old-but-correct entry were flagged permanently:
- Thin markets that fall below the CFTC reporting threshold, drop out of the
report, and reappear months later. NKD_240741 and ZO_004603 each have around
20 gaps over 10 days across 19 years of weekly data, the longest 168d and
294d. Their current 140d and 49d are well inside normal.
- Retired hist_code contracts, frozen by design. Suppressed by name via
hist_code_names(), except that the store also holds bare-code duplicates
(23977A alongside RTY_23977A) written before the naming convention included
a symbol prefix, which the suppression set never matched.
Both classes are rewritten on every producer pass, so both carry a current
updated_at. Comparing updated_at against the domain's newest write separates
"the source had nothing newer" from "the producer skipped this", which is
exactly the distinction the check needed. It also makes the bare-duplicate
naming mismatch moot rather than requiring the suppression set to cover two
conventions.
Also closes a hole found while testing: when no entry had a parseable
updated_at, the old shape silently reported everything as current. "Cannot
check" now renders as flagged, not as clean, since turning an unknown into a
pass is the precise failure this check exists to prevent.
Verified on the real store: 6 false positives to 0, and a simulated skipped
entry is still caught (25d behind). Suite 129 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--checkflagged six entries on your real store, none of which were a problem. All six were old-but-correct, and the check had no way to tell that apart from a failure.The wrong question
The check exists to catch a partial run: one entry failing while its peers succeed. It compared each entry's
last_dateagainst the domain's newestlast_date, which measures how recent the data is. That is a different question.Two legitimate classes of old-but-correct entry were flagged permanently:
Thin markets that drop out of the CFTC report. Below the reporting threshold a contract stops appearing and returns months later. Measured over 19 years of weekly data:
Both are comfortably inside their own normal.
Retired hist_code contracts, frozen by design. These were suppressed by name via
hist_code_names(), except the store also holds bare-code duplicates (23977AalongsideRTY_23977A) written before the naming convention gained a symbol prefix. The suppression set only ever generated the prefixed form, so the bare ones warned forever.The right question
Both classes are rewritten on every producer pass, so both carry a current
updated_at:Comparing
updated_atagainst the domain's newest write separates "the source had nothing newer" from "the producer skipped this". It also makes the bare-duplicate naming mismatch moot, rather than needing the suppression set to cover two conventions.A hole found while testing
When no entry had a parseable
updated_at, the new shape initially reported everything as current. Turning "cannot check" into "all clear" is the precise failure this check exists to prevent, so unusable timestamps are now flagged. There is a test for it.Verification
On the real store: 6 false positives to 0.
And a simulated skip is still caught:
Suite 129 passed, ruff clean.
Note
hist_code_names()and theignore_lagparameter are kept and still work. They are no longer load-bearing for this case, but they remain the escape hatch for suppressing an entry by name.