feat(norgate): data-driven --require-final gate (drop the fragile wall-clock cutoff) - #77
Merged
Merged
Conversation
…lock cutoff) The --require-final gate uses a fixed local-clock cutoff (--final-cutoff, default 20:55). It broke on 2026-07-27: Norgate finalized the Futures DB at 8:49 PM, the check requires both Futures and Continuous >= 20:55, so 8:49 < 8:55 deferred all night and prices went stale. A wall-clock cutoff is fragile by construction (must sit below the earliest evening final yet above any daytime interim; Norgate drifts). norgatedata has no calendar API but exposes last_quoted_date(symbol) — the latest bar date — which lets finals_ready ask "has the latest bar advanced to the expected session?" instead of "was the file touched after a magic minute?", immune to publish-time drift. Adds: - docs/design/finals_ready_data_driven.md — problem, available API, proposed data + session design, the calendar-source options, and the one open question (does Norgate expose a provisional current-day bar pre-settlement?). - scripts/probe_norgate_finals.py — Windows probe to answer that question and confirm no calendar function in the installed build. Spec only; implementation follows once the probe result is in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… tests
Daytime probe settled the design: norgatedata has no calendar API,
last_quoted_date is None for continuous symbols, and Norgate (EOD) never shows an
in-progress session's bar (at 11am Tue the latest continuous bar was Mon 7/27,
final OHLC, OI=0 marking the newest bar). So the finals gate needs neither a
trading calendar nor a wall-clock cutoff:
finals_ready := norgate_latest_bar_date > store_latest_bar_date
Add the pure, norgatedata-free core `_finals_ready_by_date(norgate_last, store_last)`
with unit tests (newer session ready; store-current not ready; empty store ready;
no-bar defers; datetimes normalized to date). Weekends/holidays produce no new bar
so no capture; publish-time drift is absorbed by retries; the 2026-07-27 stall
could not recur.
Not yet wired into the CLI: pending a two-point evening probe to confirm Norgate
never shows the session bar provisionally pre-settlement, then finals_ready() I/O
+ keep --final-cutoff as a deprecated fallback. Spec: docs/design/finals_ready_data_driven.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k cutoff --require-final now defers until Norgate holds a NEWER settled continuous bar than the store already has, across a liquid reference quorum (ES, CL, ZC), instead of waiting for a fixed local-clock cutoff on the DB refresh time. Why: the clock cutoff (--final-cutoff, default 20:55) is fragile by construction and broke on 2026-07-27 — Norgate finalized the Futures DB at 8:49pm, the check requires both Futures and Continuous >= 20:55, so 8:49 < 8:55 deferred all night and prices went stale. A wall-clock cutoff must sit below the earliest evening final yet above any daytime interim, and Norgate's publish time drifts. The daytime probe showed norgatedata has no calendar API, last_quoted_date is None for continuous symbols, and Norgate (EOD) never exposes an in-progress session's bar — so "newer settled bar than the store" is a clean, robust signal. - finals_ready(): thin norgatedata I/O (_norgate_last_bar_date via a short trailing price_timeseries window; _store_last_bar_date from the prices manifest) delegating to the pure _finals_ready_quorum over _finals_ready_by_date. Guards on _require_norgate_service(). - CLI: --require-final uses it; --final-cutoff accepted-but-ignored (deprecated) so no scheduler breaks. Legacy _finals_ready clock core retained, unused, for rollback. - Tests (norgatedata-free): pure core, quorum (all-advance vs one lagging), wiring via mocks, ignored cutoff arg. 23 passing; ruff clean. Immune to publish-time drift (early → ready early, late → a retry catches it) and needs no trading calendar (weekends/holidays produce no new bar). Live validation on the Windows producer is the remaining rollout step. Spec: docs/design/finals_ready_data_driven.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the fragile wall-clock
--require-finalgate with a data-driven one: defer until Norgate has a newer settled continuous bar than the store already holds, across a liquid reference quorum (ES, CL, ZC).Why
The clock cutoff (
--final-cutoff, default20:55) required both theFuturesandContinuous Futuresdatabases to be refreshed at/after a fixed local time. That is fragile by construction — the cutoff must sit below the earliest evening final yet above any daytime interim, and Norgate's publish time drifts night to night.It broke in production on 2026-07-27: Norgate finalized the
FuturesDB at 8:49pm and never touched it again that night, so8:49 < 20:55never satisfied the gate, every retry deferred, the chained store sync was skipped, and prices went stale on Friday's bar. It also has no single safe value — finalize at 20:44 and a20:45cutoff misses it; lower the cutoff and you risk capturing a pre-settlement bar.Investigation (probe on the Windows producer)
scripts/probe_norgate_finals.pyestablished:norgatedataexposes no holiday/trading-calendar API.last_quoted_dateisNonefor continuous symbols (it's for expiring instruments), so the signal must come fromprice_timeseries.OI=0marking the newest bar). No Tuesday bar existed intraday.So a bar date newer than the store's is a clean "a new session settled" signal, needing no clock and no calendar.
Change
_finals_ready_by_date(pure per-symbol core) +_finals_ready_quorum(pure combiner, requires the whole quorum so a session is captured once and complete) +finals_ready(thin norgatedata I/O: trailingprice_timeserieswindow for the Norgate date, prices manifest for the store date).--require-finaluses it;--final-cutoffis accepted but ignored (deprecated) so existing schedulers don't break. Legacy_finals_readyclock core retained, unused, for rollback.Tests
norgatedata-free: pure core, quorum (all-advance vs one lagging),finals_readywiring via mocks, and the ignoredcutoffarg. 23 passing, ruff clean. The--require-finalCLI-exit tests (which mockfinals_ready) are unchanged and pass.Rollout
Data-driven is the default (the old path is broken, so there's nothing to fall back to). Remaining step: a few nights of live validation on the Windows producer — confirm it flips to ready when the new settled bar lands and stays not-ready on a same-session re-run. Spec:
docs/design/finals_ready_data_driven.md.🤖 Generated with Claude Code