feat(norgate): data-driven --require-final gate (drop the fragile wall-clock cutoff)

CHANGELOG.md

@@ -24,6 +24,22 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   off ETF proxies. Adds the MSCI EM (MME→EEM) and EAFE (MFS→EFA) held-out
   generalization markets. ([#24](https://github.com/mspinola/cotdata/pull/24))
 
+### Changed
+- **`--require-final` price gate is now data-driven, not a wall-clock cutoff.**
+  `finals_ready` previously deferred until Norgate's `Futures` and `Continuous
+  Futures` databases were both refreshed at/after a fixed local time
+  (`--final-cutoff`, default 20:55). That is fragile by construction: the cutoff
+  must sit below the earliest evening final yet above any daytime interim, and
+  Norgate's publish time drifts. On 2026-07-27 Norgate finalized the Futures DB at
+  8:49pm, so the `>= 20:55` check never turned true and prices went stale for the
+  day. The gate now asks the robust question — does Norgate hold a **newer settled
+  continuous bar** than the store already has? — across a liquid reference quorum
+  (ES, CL, ZC). It is immune to publish-time drift (early publish → ready early,
+  late publish → a retry catches it) and needs no trading calendar (weekends and
+  holidays simply produce no new bar). `--final-cutoff` is accepted but ignored
+  (deprecated) so existing schedulers do not break. See
+  `docs/design/finals_ready_data_driven.md`.
+
 ### Fixed
 - **`databento.fetch_daily_ohlc`'s `start_date` parameter now actually does
   something** (dormant provider). It was previously silently ignored on every

docs/design/finals_ready_data_driven.md

@@ -0,0 +1,168 @@
+# Data-driven `finals_ready` (replace the fixed-clock cutoff)
+
+Status: IMPLEMENTED, pending live validation on the Windows producer. The daytime probe
+answered the open question (Norgate never shows an in-progress session's bar); the gate is
+wired data-driven with unit tests. What remains is a few nights confirming it flips ready at
+the right moment against the live feed.
+
+## Problem
+
+`--require-final` gates the nightly Norgate price capture so it never stores an
+interim (non-settled) bar. Today it does so with a fixed **local-clock cutoff**
+(`--final-cutoff`, default `20:55`): `finals_ready` is true only once *both* the
+`Futures` and `Continuous Futures` databases were refreshed at/after today's cutoff
+(`providers/norgate.py::_finals_ready`).
+
+This is calibration, not robustness, and it broke in production on 2026-07-27:
+
+- Norgate's evening publish finalized the **Futures** database at **8:49 PM** and did
+  not touch it again that night; **Continuous Futures** finalized at 8:55 PM.
+- The check requires *both* `>= 20:55`. Futures at 8:49 `< 8:55` → never ready → the
+  task deferred (exit 1) on every attempt, the chained store sync was skipped, and
+  prices went stale (stuck on the prior Friday's bar).
+
+The failure mode is intrinsic to a wall-clock cutoff: it must sit *below* the earliest
+evening-final refresh and *above* any daytime interim refresh, and Norgate's publish
+time drifts night to night. If Norgate finalizes at 20:44, a `20:45` cutoff misses it;
+if you lower the cutoff to catch early nights, you risk accepting a pre-settlement bar.
+There is no single clock value that is safe.
+
+## Available Norgate API (v1.0.77)
+
+`norgatedata` exposes **no** holiday / trading-calendar / session function (confirmed on
+PyPI). It does expose data-content signals we are not yet using:
+
+- `last_quoted_date(symbol, datetimeformat='iso')` — the latest trading date that has a
+  bar for `symbol`. For a continuous symbol (never expires) this is the newest available
+  daily bar.
+- `second_last_quoted_date(symbol)` — the prior trading day's date.
+- `last_price_update_time(symbol)` — local PC time the symbol was last refreshed.
+- `last_database_update_time(db)` — local PC time the database was refreshed (current
+  approach).
+- `status()` — is NDU running.
+
+`last_quoted_date` is the key: it lets us ask "**has Norgate's latest bar advanced to the
+session we expect?**" instead of "was the file touched after a magic minute?" That is
+immune to publish-time drift.
+
+## Proposed design
+
+Replace the clock-threshold with a **data + session** check:
+
+```
+finals_ready(now) := last_quoted_date(ref) >= expected_last_session(now)
+```
+
+- `ref` = one (or a small quorum) of liquid continuous symbols that trade every US
+  futures session (e.g. `&ES`, `&CL`). Use the min across the quorum so a single lagging
+  symbol cannot green-light the whole book.
+- `expected_last_session(now)` = the most recent trading day whose session has **closed**
+  as of `now`. Weekend/holiday aware.
+
+Why this is robust where the cutoff is not:
+
+- **Publish-time drift disappears.** If Norgate finalizes early, `last_quoted_date`
+  advances early and we are ready early. If it finalizes late, the bar is simply not
+  there yet and the run defers — retries then catch it whenever it lands. The gate no
+  longer depends on Norgate hitting a target minute.
+- **The only remaining clock element is coarse and stable.** `expected_last_session`
+  needs to know a session has *closed*, i.e. "we are past the exchange's daily settlement
+  window." That threshold has hours of slack (settlement is done well before the evening
+  publish), unlike the razor-thin publish cutoff. It is a property of the market, not of
+  Norgate's schedule.
+
+### Two pieces to nail down
+
+1. **`expected_last_session` / the trading calendar.** `norgatedata` has no calendar API,
+   so options are:
+   - `pandas_market_calendars` (mature, has CME/ICE calendars) — adds a dependency to a
+     public package. Cleanest correctness.
+   - A maintained holiday list in-repo (the NDU app shows NYSE/exchange holidays 6 months
+     each way; not exposed to Python) — no dependency, but must be maintained.
+   - Derive from Norgate: treat `expected_last_session` as "the max `last_quoted_date`
+     across a broad basket," and require the ref quorum to have caught up to it. Avoids a
+     calendar entirely, at the cost of a softer definition.
+   A wrong calendar only causes a **false defer** (safe: it self-heals next session), never
+   a false capture, so this is a robustness/convenience tradeoff, not a correctness risk.
+
+2. **Interim/preliminary bars — the open question below.**
+
+## Open question (Windows probe required)
+
+Does Norgate ever expose a **provisional current-day bar before final settlement**?
+- If **no** (today's bar appears only once settled): `last_quoted_date == today` is a
+  sufficient, clean "finals are in" signal, and the design above is complete.
+- If **yes** (staged/preliminary then final settlement): bar *presence* is not enough,
+  because the value can still be revised. We would then keep a **coarse** "past settlement
+  window" guard (e.g. only trust today's bar after ~18:00 ET) on top of the date check —
+  still far more forgiving than the current publish-time cutoff.
+
+The probe (`scripts/probe_norgate_finals.py`, run on the Windows producer) answers this by
+sampling `last_quoted_date` and the tail of `price_timeseries` for a couple of continuous
+symbols during the day and again after the evening publish, and by dumping `dir(norgatedata)`
+to confirm no calendar function exists in the installed build.
+
+## Compatibility / rollout
+
+- `cotdata-prices` is a public CLI. Keep `--final-cutoff` accepted (as the fallback
+  "settlement window" guard, or deprecated-but-honored) so no caller breaks.
+- Keep the pure-core split: `_finals_ready(...)` stays a norgatedata-free, unit-tested
+  function operating on plain dates/times; `finals_ready(...)` does the norgatedata I/O.
+- Default behavior change (clock → data) is the point, but gate it so it can be rolled
+  back with a flag if a first-night surprise appears.
+
+## Test plan
+
+Unit-test the pure core on any OS (no norgatedata): weekend/holiday rollover, early vs
+late publish, ref-quorum lag, and the interim guard if needed. Live-validate `finals_ready`
+against the real feed on Windows across a few evenings before flipping the default.
+
+## Probe results (Windows, 2026-07-28 daytime) — design simplified
+
+The daytime run (`scripts/probe_norgate_finals.py`) settled the open questions and made the
+design simpler than the draft above:
+
+- **No calendar API.** The only `*session*` functions (`futures_market_session_*`,
+  `session_type`) describe which contracts trade in a session, not holidays. Confirmed.
+- **`last_quoted_date` is unusable for the gate** — it returns `None` for continuous
+  symbols (it is meant for expiring instruments). The signal must come from
+  `price_timeseries`.
+- **Norgate (EOD) does not publish an in-progress session's bar.** At 11 AM Tue the latest
+  continuous bar was **Mon 7/27** (final OHLC), with **`Open Interest == 0`** while every
+  prior day was ~1.9M — `OI==0` simply marks the newest bar (OI publishes T+1). No Tuesday
+  bar existed intraday.
+
+So the gate needs **neither a trading calendar nor a wall-clock cutoff**:
+
+```
+finals_ready := norgate_latest_bar_date  >  store_latest_bar_date
+```
+
+Implemented as the pure core `_finals_ready_by_date(norgate_last, store_last)` (unit-tested,
+no norgatedata). Weekends/holidays produce no new bar (correctly no capture); publish-time
+drift is absorbed by retries; the 2026-07-27 failure could not recur (Monday's bar was
+available all along — only the clock cutoff blocked it).
+
+### What shipped
+
+- `_finals_ready_by_date(norgate_last, store_last)` — pure per-symbol core.
+- `_finals_ready_quorum(norgate_dates, store_dates)` — pure combiner; ready only when
+  EVERY reference symbol (`_FINALS_REF_SYMBOLS = ES, CL, ZC`) has a newer settled bar than
+  the store, so a session is captured once and complete.
+- `finals_ready()` — thin norgatedata I/O: reads each ref's latest continuous bar date
+  (`_norgate_last_bar_date`, a short trailing `price_timeseries` window) and the store's
+  last captured date (`_store_last_bar_date`, from the prices manifest), then delegates to
+  the quorum. Guards on `_require_norgate_service()`.
+- CLI: `--require-final` now uses this; `--final-cutoff` is accepted-but-ignored
+  (deprecated) so no scheduler breaks. The legacy `_finals_ready` clock core is retained,
+  unused, for reference/rollback.
+- Unit tests cover the core, the quorum (all-advance vs one lagging), the wiring, and the
+  ignored `cutoff` arg — all norgatedata-free.
+
+### Remaining: live validation (Windows)
+
+A couple of evenings confirming `finals_ready` flips to ready when the new settled bar lands
+(and that a same-session re-run stays not-ready). The optional two-point evening probe
+(before ~8:45 PM, after ~9:15 PM) is a nice confirmation of the transition but is no longer a
+blocker — the daytime probe already showed Norgate does not expose an in-progress session's
+bar.

scripts/probe_norgate_finals.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+"""Probe norgatedata's data-content signals for the data-driven finals_ready redesign.
+
+Run on the WINDOWS producer (needs norgatedata + a live Norgate subscription).
+
+Run it TWICE and compare the output:
+  1. during the trading day (before Norgate's evening publish), and
+  2. after the evening publish (e.g. ~9:30 PM ET).
+
+The key question: does `last_quoted_date` for a continuous symbol show TODAY's date
+only after settlement, or already during the day (a provisional bar)? That decides
+whether bar-presence alone is a sufficient "finals are in" signal. See
+docs/design/finals_ready_data_driven.md.
+"""
+import datetime as dt
+
+import norgatedata
+
+REF_SYMBOLS = ["&ES", "&CL", "&ZC"]  # liquid continuous: S&P, WTI, corn — trade every session
+FINAL_DATABASES = ["Futures", "Continuous Futures"]
+
+print(f"# probe at {dt.datetime.now().isoformat()}  (local PC time)")
+print(f"norgatedata version: {getattr(norgatedata, '__version__', '?')}")
+try:
+    print(f"NDU running (status()): {norgatedata.status()}")
+except Exception as e:  # noqa: BLE001
+    print(f"status() ERROR: {e!r}")
+
+# 1) Confirm whether this build has ANY calendar/holiday/session function.
+api = sorted(x for x in dir(norgatedata) if not x.startswith("_"))
+cal = [x for x in api if any(k in x.lower() for k in
+       ("holiday", "calendar", "session", "business", "market_day", "trading_day", "busday"))]
+print(f"\ncalendar-ish functions in this build: {cal or 'NONE'}")
+print(f"full API: {api}")
+
+# 2) Database refresh times — the current (fragile) signal.
+print("\n-- last_database_update_time --")
+for db in FINAL_DATABASES:
+    try:
+        print(f"  {db}: {norgatedata.last_database_update_time(db)}")
+    except Exception as e:  # noqa: BLE001
+        print(f"  {db}: ERROR {e!r}")
+
+# 3) Data-content signals per ref symbol — the proposed signal.
+print("\n-- per-symbol date/time signals --")
+for s in REF_SYMBOLS:
+    try:
+        lq = norgatedata.last_quoted_date(s, datetimeformat="iso")
+        slq = norgatedata.second_last_quoted_date(s, datetimeformat="iso")
+        lpu = norgatedata.last_price_update_time(s)
+        print(f"  {s}: last_quoted_date={lq}  second_last={slq}  last_price_update_time={lpu}")
+    except Exception as e:  # noqa: BLE001
+        print(f"  {s}: ERROR {e!r}")
+
+# 4) The actual tail — does today's bar exist yet, and do its values look settled?
+print("\n-- price_timeseries tail (last 4 bars) --")
+for s in REF_SYMBOLS:
+    try:
+        df = norgatedata.price_timeseries(
+            s,
+            padding_setting=norgatedata.PaddingType.NONE,
+            timeseriesformat="pandas-dataframe",
+            start_date="2026-07-20",
+        )
+        print(f"  {s}:")
+        print("    " + df.tail(4).to_string().replace("\n", "\n    "))
+    except Exception as e:  # noqa: BLE001
+        print(f"  {s}: ERROR {e!r}")

src/cotdata/providers/norgate.py

@@ -28,6 +28,9 @@
 # Default local-time cutoff after which Norgate's "Final" futures prices are in
 # (≈ Continuous Futures Final at ~8:55pm ET per Norgate's update schedule).
 DEFAULT_FINAL_CUTOFF = "20:55"
+# Liquid continuous reference symbols for the data-driven finals gate: each trades every
+# US futures session, so a newer settled bar on all of them means a session has landed.
+_FINALS_REF_SYMBOLS = ("ES", "CL", "ZC")
 # If roll-day overnight moves exceed this multiple of the normal-day median, the
 # series looks UNADJUSTED (calendar-spread gaps not stitched). Self-calibrating
 # per symbol, so it works across products with different spread magnitudes.
@@ -253,7 +256,11 @@ def _to_naive_local(t):
 
 
 def _finals_ready(db_times: dict, cutoff: str = DEFAULT_FINAL_CUTOFF, now=None):
-    """Pure core of :func:`finals_ready` — testable without norgatedata.
+    """LEGACY wall-clock finals core, superseded by the data-driven gate
+    (:func:`_finals_ready_by_date` / :func:`finals_ready`). Retained for reference and
+    rollback; no CLI path uses it. It broke in production because a fixed clock cutoff
+    must sit below the earliest evening final yet above any daytime interim, and Norgate's
+    publish time drifts (see docs/design/finals_ready_data_driven.md).
 
     db_times maps database name → its last-update datetime (tz-aware or naive local,
     or None). Ready when every database was refreshed at/after today's `cutoff`
@@ -271,15 +278,85 @@ def _finals_ready(db_times: dict, cutoff: str = DEFAULT_FINAL_CUTOFF, now=None):
     return ready, detail
 
 
-def finals_ready(cutoff: str = DEFAULT_FINAL_CUTOFF, now=None):
-    """True once Norgate has this day's FINAL futures prices — i.e. it has refreshed
-    both the 'Futures' and 'Continuous Futures' databases at/after today's local
-    `cutoff`. Uses norgatedata.last_database_update_time (the local PC time of the
-    last DB refresh). Lets a scheduled run avoid capturing interim (non-final) bars.
-    Returns (ready: bool, detail: dict)."""
+def _finals_ready_by_date(norgate_last, store_last):
+    """Data-driven finals gate (pure core): ready when Norgate's latest continuous bar is
+    a NEWER completed session than the store already holds.
+
+    Both args are dates (or datetimes, normalized to their date; or None). Norgate is
+    end-of-day and only publishes a session's bar once that session is complete, so a
+    bar date newer than the store's is a new *settled* session to capture. This needs no
+    trading calendar (weekends/holidays simply produce no new bar) and no wall-clock
+    cutoff (early publish → ready early; late publish → not there yet → a retry catches
+    it), which is what makes it immune to Norgate's publish-time drift. Returns
+    (ready: bool, detail: dict)."""
+    def _d(x):
+        if x is None:
+            return None
+        return x.date() if isinstance(x, dt.datetime) else x
+
+    nl, sl = _d(norgate_last), _d(store_last)
+    detail = {
+        "norgate_last": nl.isoformat() if nl else None,
+        "store_last": sl.isoformat() if sl else None,
+    }
+    if nl is None:
+        return False, detail  # Norgate has no bar to offer → defer
+    return (sl is None or nl > sl), detail
+
+
+def _finals_ready_quorum(norgate_dates: dict, store_dates: dict):
+    """Combine per-reference results into the finals gate (pure/testable): ready only when
+    EVERY reference symbol has a newer settled bar in Norgate than the store already holds.
+    Requiring the whole quorum means a session is captured once and complete, and one
+    lagging reference cannot green-light a partial capture. Returns (ready, detail)."""
+    per, ready_all = {}, True
+    for sym in norgate_dates:
+        r, d = _finals_ready_by_date(norgate_dates.get(sym), store_dates.get(sym))
+        per[sym] = {**d, "ready": r}
+        ready_all = ready_all and r
+    return ready_all, {"mode": "data", "per_symbol": per}
+
+
+def _norgate_last_bar_date(sym: str):
+    """Latest continuous (back-adjusted) bar date Norgate holds for internal `sym`, or None.
+    Pulls a short trailing window (cheap) and takes the last index. Norgate is end-of-day
+    and only publishes a session's bar once settled, so this date advances exactly when a
+    new final session lands — the signal the gate keys on."""
     import norgatedata  # Windows producer only
-    times = {db: norgatedata.last_database_update_time(db) for db in FINAL_DATABASES}
-    return _finals_ready(times, cutoff, now)
+    ng_sym = REGISTRY[sym].norgate + CCB_SUFFIX
+    start = (dt.date.today() - dt.timedelta(days=10)).isoformat()
+    df = norgatedata.price_timeseries(
+        ng_sym,
+        padding_setting=norgatedata.PaddingType.NONE,
+        timeseriesformat="pandas-dataframe",
+        start_date=start,
+    )
+    if df is None or len(df) == 0:
+        return None
+    return pd.to_datetime(df.index[-1]).tz_localize(None).normalize().date()
+
+
+def _store_last_bar_date(sym: str):
+    """Latest date already captured in the store for internal `sym` (back-adjusted), or
+    None if the store has never seen it. Read from the prices manifest — no price I/O."""
+    prices = store.load_manifest().get("prices", {})
+    ld = (prices.get(f"{sym}_backadj") or {}).get("last_date")
+    return pd.to_datetime(ld).date() if ld else None
+
+
+def finals_ready(cutoff=None, now=None, ref_symbols=_FINALS_REF_SYMBOLS):
+    """Ready once Norgate has a NEWER settled continuous bar than the store, for a quorum of
+    liquid reference symbols (data-driven finals gate). Replaces the old wall-clock cutoff:
+    immune to Norgate's publish-time drift (early publish → ready early; late publish → not
+    there yet → a retry catches it) and needs no trading calendar (weekends and holidays
+    simply produce no new bar). See docs/design/finals_ready_data_driven.md.
+
+    `cutoff` and `now` are accepted for backward compatibility and IGNORED — the clock gate
+    is deprecated. Returns (ready: bool, detail: dict)."""
+    _require_norgate_service()  # NDU-down guard: norgatedata calls bare sys.exit otherwise
+    norgate_dates = {s: _norgate_last_bar_date(s) for s in ref_symbols}
+    store_dates = {s: _store_last_bar_date(s) for s in ref_symbols}
+    return _finals_ready_quorum(norgate_dates, store_dates)
 
 
 def _norgate_covered(symbols):