feat(vintage): seed the CFTC release calendar so derived rows get real dates - #82
Merged
Merged
Conversation
…l dates
cotdata-schedule sync now also parses the published CFTC release schedule, which lists
one year of RELEASE dates month by month and marks holiday-delayed ones with an asterisk
('*Delayed release date due to a federal holiday.').
What this is actually worth, stated precisely: on a normal week 'derived' (report_date +
3, weekend-adjusted) already lands on the right Friday, so most dates do not move. It
buys two things. It CORRECTS the six 2026 weeks delayed by a federal holiday, where
derived is wrong by three days (New Year, Juneteenth, Independence Day, Veterans Day,
Thanksgiving, Christmas). And it upgrades the PROVENANCE of every matched row from
derived (a guess) to scheduled (a published fact), which is what lets strict
point-in-time evaluation trust the date at all.
report_date is derived from each release as the latest Tuesday at least three days
earlier. That one rule covers both the normal Friday release and a holiday-delayed
Monday one, which reports the previous week's Tuesday, without needing a holiday
calendar of our own.
Parsed against a trimmed copy of the live page committed as a fixture, so the parser is
pinned to real markup rather than an idealised shape. It reads the year from tag-stripped
text because it sits inside nested markup, and it RAISES rather than returning an empty
frame on a layout change: a silent empty parse looks exactly like 'CFTC published
nothing' and would quietly leave every row on derived.
Cross-validated: the calendar says report 2026-07-21 released 2026-07-24, which is the
same date the published mechanism independently derives from the weekly static's
Last-Modified header. Two unrelated sources agree, and a test pins that.
Also fixes a pandas FutureWarning in the announcements merge, where concatenating into an
empty store is deprecated and shifts dtype inference.
Coverage limit: the page only ever shows the current year, and CFTC does not publish past
calendars there, so earlier years cannot be seeded this way.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cotdata-schedule syncnow also parses the published CFTC release schedule, which lists one year of release dates month by month and marks holiday-delayed ones with an asterisk — "Delayed release date due to a federal holiday."What this is actually worth
Stated precisely, because it is less than it sounds: on a normal week
derived(report_date + 3, weekend-adjusted) already lands on the right Friday. Most dates will not move. Seeding buys two things:derivedis wrong by three days:derived(a guess) toscheduled(a published fact) — which is what lets strict PIT evaluation trust the date at all. Per the design notes,derivedfails on exactly the weeks that matter, and these are those weeks.Design
report_dateis derived from each release as the latest Tuesday at least three days earlier. That single rule covers both the normal Friday release and a holiday-delayed Monday one (which reports the previous week's Tuesday), with no holiday calendar of our own to maintain.Parsed against a trimmed copy of the live page, committed as a fixture, so the parser is pinned to real markup. It reads the year from tag-stripped text (it sits inside
<h3><strong>…) and raises rather than returning an empty frame on a layout change — a silent empty parse looks identical to "CFTC published nothing" and would quietly leave every row onderived.Cross-validation
The calendar says report
2026-07-21→ release2026-07-24. That is the same date thepublishedmechanism independently derives from the weekly static'sLast-Modifiedheader. Two unrelated sources agree, and a test pins it.Sanity checks in tests: 52 releases (one per week), every report_date a Tuesday, every unflagged release a Friday, all six delays landing on Mondays.
Limits
The page only ever shows the current year; CFTC does not publish past calendars there, so earlier years cannot be seeded this way and stay on
announced/derived.Also fixes a pandas
FutureWarningin the announcements merge (concatenating into an empty store is deprecated and shifts dtype inference).208 tests pass,
ruff check src tests scriptsclean. Live sync verified end to end.🤖 Generated with Claude Code