Repository navigation
docs: correct stale v2 migration status in config and README - #13
Merged
Merged
Conversation
The store was migrated to schema v2 by a full producer pass (2026-07-14), but two doc statements still described the pre-migration world: - config.py claimed the store "does not actually carry v2 shape until a full producer pass re-writes it" — no longer true. Reword to note the migration is done and that schema_version() reflects on-disk truth. - README described Volume_Reconstructed as "systematically higher than raw Volume." Verified against the store this is symbol-dependent: higher for products whose rolls spread volume across contracts, but roughly equal or lower for near-empty-back-month symbols like crypto (BTC/ETH reconstruct to ~0.98-1.00x raw). Reword to reflect that. Docs-only; no code behavior change. Store verified: schema_version=2, 84/84 price files carry reconstructed volume. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 30, 2026
Moves findings out of commit messages and chat and into the docs, so the docs are the source of truth. The specs above the new sections are left unedited: they are the documents as written before implementation, and the amendments record where they were wrong rather than quietly correcting them. handoff §12 "Outcome" (following the ADR-0006 house pattern): - The Last-Modified spike as an explicit NEGATIVE result: historical release dates cannot be recovered from headers, so §4.6's fallback chain is the confirmed path. - Header sweep 2020-2026 and the ROLLING TWO-YEAR WINDOW: 2025 and 2026 share an identical Last-Modified, so CFTC regenerates current + prior year and everything older is static. Corrects an earlier claim that all closed years were re-touched weekly, and explains why weekly downloads have always grabbed both years. Also: no ETag is served on any file, so If-None-Match can never fire, while If-Modified-Since does 304. - Decisions: retention (keep everything, no pruning), --all as a restatement TRIPWIRE rather than a backfill, daily producer-side capture, and the futures-only carry-forward that leaves `combined` constant-False with half the reportable universe absent. - Deviations table (polars->pandas, manifest.json->snapshots.json, synthetic fixtures, no --from-git, and `published` shipping despite §10 deferring it). - Answers to §11, including that backfill coverage is legitimately empty because no production vintages exist yet. crowdmon_futures_cot_module.md: resolves the §4 open items (schema, backfill span, revisions were overwritten, no release_date, futures-only) and notes that `vintage: int` is not how it was built — the implementation is bitemporal. §5.3 gains the two findings that change what it can assume: release date is resolved with provenance rather than read, and `derived` fails on exactly the weeks that matter; and vintage history is forward-only, so PIT protection cannot be backfilled. Links to cot_vintage.md and ADR-0008 dangle on main until PRs #78 and #13 merge; called out inline rather than left to surprise a reader. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
added a commit
that referenced
this pull request
Jul 30, 2026
The warning was written while these docs sat on main and cot_vintage.md did not. They now travel in the same PR, so that link never dangles. Only the cross-repo ADR-0008 reference still depends on a separate merge (crucible-stack #13). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mspinola
pushed a commit
that referenced
this pull request
Aug 9, 2026
Written against a stale clone. `--require-final` was wired into `marketdata-update` by PR #12, which merged while this branch was being built, and a duplicate written here converged on the same design independently hours apart. #12 is the better of the two -- it also keeps `--final-cutoff` accepted and ignored, so a scheduler still carrying cotdata's flag does not break on it, and it prints a per-reference defer table. Worth recording rather than quietly fixing: two passes reaching for the same gate is evidence the deletion could not proceed without it, which is the point §8.5 was making. #13 is now only the ported producer tests, which nothing else covers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two docs-only corrections. The schema v1→v2 reconstructed-volume migration completed via a full producer pass on 2026-07-14, but two statements still described the pre-migration world:
config.pyclaimed the store "does not actually carry v2 shape until a full producer pass re-writes it." No longer true — reworded to note the migration is done and thatschema_version()reflects on-disk manifest truth (a fresh/partial store can still read <2).README.mddescribedVolume_Reconstructedas "systematically higher than raw Volume." Verified against the live store: it's symbol-dependent — higher for products whose rolls spread volume across contracts, but roughly equal or lower for near-empty-back-month symbols like crypto (BTC/ETH reconstruct to ~0.98–1.00× raw). Reworded accordingly.Verification
Checked against the actual store (
schema_version=2):Volume_Reconstructed; 829k rows allVolume_Source=reconstructed.No code behavior change.
🤖 Generated with Claude Code