Intraday ohlcv-1m: schema, batch puller with cost preview, and the 6-symbol pilot - #28
Merged
Merged
Conversation
Adds ohlcv-1m support for the news-failure path work, which the trigger studies identified as the one real data gap: his news failure is an intraday path (opened weaker on the number, closed positive) and daily OHLC is a lossy projection of it. Two things this gets right that the previous pull did not. Cost is known in advance and for free. metadata.get_cost prices any request exactly without transferring data, so sampling to estimate is unnecessary. Priced 2026-09-06 for 30 CME symbols over 2022-01 to 2026-09: ohlcv-1h $7.51, ohlcv-1m $144.07, ohlcv-1s $1,767.64, trades $2,832.72. Schema choice dominates by more than two orders of magnitude, and minute bars are the granularity the question needs. The transport changes. providers/databento.py::_fetch uses timeseries.get_range, which streams one symbol at a time and is why the last large pull ran for days. The new puller uses the batch api: submit a job, databento prepares files server-side, download compressed. Same data, same cost, very different wall clock. The nightly producer is untouched. INTRADAY_SCHEMA is deliberately NOT added to _SCHEMAS, because the two-stage path is a daily-bar pipeline a scheduled task depends on and the volumes differ by three orders of magnitude. Intraday gets its own raw namespace via intraday_raw_path, kept out of any consumer sync like the rest of the producer-internal raw store. The puller refuses to submit a paid job without --yes in a non-interactive shell, and prints the cost before asking either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ing it The 6-symbol ohlcv-1m pilot completed: job GLBX-20260906-4L6PBNH6G8, 56 monthly files, 138 MB, $35.67, 9.77M records. Wall clock was about nine minutes against days for the streaming path, which is the batch api doing what it is for. Data verified readable: correct schema, continuous symbology, 2022-01 through 2026-08. It landed in the wrong place, and the reason is worth recording. raw_root() falls back to $MARKETDATA_STORE/_raw/databento when MARKETDATA_DATABENTO_RAW is unset. That variable is exported from ~/.zshrc but NOT from ~/.bash_profile, so anything launched from a bash context takes the fallback silently. The configured store is ~/code/databento_raw/databento (1.5G of daily raw), outside the mirrored tree; the fallback is inside it. marketdata_store is the destination of the Windows producer's nightly robocopy /MIR, and /MIR purges whatever the source lacks. So 138 MB of paid data sitting there was at risk of a delayed-action delete, the same failure mode CLAUDE.md documents for equity symbols seeded on the Mac alone. Whether the sync's exclusions would have covered _raw could not be verified because the Windows share was not mounted, so this is a real risk rather than a confirmed loss. The data was moved to the configured location by hand. Documents the hazard on raw_root rather than changing the fallback, which is deliberate and load-bearing elsewhere. Anyone adding a caller should set the variable explicitly and check where raw_root resolved before a long fetch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 2, and it costs nothing: local files only, no API calls. 9,770,671 rows across ES NQ RTY GC CL 6E, 2022-01-02 to 2026-08-31, roughly 1.6M minute bars each. Price ranges check out (ES 3502-7838, GC 1618-5627, CL 54.7-130.5, 6E 1.0-1.2). Two conversions the raw CSV needs, both done once at the boundary. Prices arrive as fixed-point integers at 1e-9, so databento's 4771000000000 is 4771.0; left as integers they produce nonsense a long way downstream. ts_event is UTC nanoseconds and is stored tz-aware UTC. Timezone policy stated rather than assumed: the store keeps one canonical timezone and consumers convert. News-failure work wants US/Eastern because the releases that matter are scheduled in ET, but that belongs to the consumer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ohlcv-1msupport for the news-failure path work, which the npf trigger studies identified as the one real data gap: his news failure is an intraday path (opened weaker on the number, closed positive) and daily OHLC is a lossy projection of it.Two things this gets right that the previous pull did not
Cost is known in advance, for free.
metadata.get_costprices any request exactly without transferring data, so sampling to estimate is unnecessary. Priced 2026-09-06 for 30 CME symbols over 2022-01 to 2026-09:ohlcv-1hohlcv-1mohlcv-1stradesSchema choice dominates by more than two orders of magnitude, and minute bars are the granularity the question needs.
The transport changes.
providers/databento.py::_fetchusestimeseries.get_range, which streams one symbol at a time and is why the last large pull ran for days. The new puller uses the batch API: submit a job, databento prepares files server-side, download compressed. Same data, same cost. The pilot completed in about nine minutes.The nightly producer is untouched
INTRADAY_SCHEMAis deliberately not added to_SCHEMAS. The two-stage path is a daily-bar pipeline a scheduled task depends on, and the volumes differ by three orders of magnitude. Intraday gets its own raw namespace viaintraday_raw_path, kept out of consumer sync like the rest of the producer-internal raw store.The puller refuses to submit a paid job without
--yesin a non-interactive shell, and prints the cost before asking either way.What was pulled
Job
GLBX-20260906-4L6PBNH6G8: ES NQ RTY GC CL 6E, 2022-01 to 2026-09, $35.67, 9,770,671 records, 56 monthly files. Normalized to per-symbol parquet: ~1.6M minute bars each, price ranges verified (ES 3502-7838, GC 1618-5627, CL 54.7-130.5, 6E 1.0-1.2).Two conversions done once at the boundary: prices arrive as fixed-point integers at 1e-9, and
ts_eventis UTC nanoseconds, stored tz-aware. Timezone policy is stated rather than assumed, with the store keeping one canonical zone and consumers converting.A hazard documented, not silently fixed
The pilot landed in the wrong place.
raw_root()falls back to$MARKETDATA_STORE/_raw/databentowhenMARKETDATA_DATABENTO_RAWis unset, and that variable is exported from~/.zshrcbut not from~/.bash_profile, so anything launched from a bash context takes the fallback silently.marketdata_storeis the destination of the Windows producer's nightlyrobocopy /MIR, and/MIRpurges whatever the source lacks. So 138 MB of paid data was at risk of the same delayed-action deleteCLAUDE.mddocuments for equity symbols. Whether the sync's exclusions cover_rawcould not be verified (the Windows share was not mounted), so this is a real risk rather than a confirmed loss. The data was moved to the configured location by hand and the hazard is documented onraw_rootrather than changing a fallback that is deliberate and load-bearing elsewhere.🤖 Generated with Claude Code