feat(preflight): add sf_preflight.py — Saturday SF dry-rehearsal#136
Merged
Conversation
Predicts whether the Saturday SF would succeed BEFORE launching a spot. Today's recovery cycle (5 SF redrives, ~5 polygon API calls each) burned free-tier quota and operator hours discovering bugs sequentially. This module simulates the critical pre-Phase-1 path against real S3 + ArcticDB state and reports per-step pass/fail in ~30s with 1 polygon call total. Eight independent checks, mapped to today's incident stack: PR #130 (backfill regression) → check_backfill_source_freshness PR #131 (polygon coverage flake) → check_polygon_grouped_coverage PR #132 (missing-from-closes scoping) → check_predicted_missing_from_closes PR #133 (freshness scan scoping) → check_universe_sample_freshness PR #134 (workflow ordering) → check_universe_drift PR #135 (return shape) → check_constituents_fetch Postflight contracts → check_postflight_contracts ArcticDB reachability → check_arctic_connectivity Each check is a pure function taking a PreflightContext, returning a CheckResult. The orchestrator runs them all (catching per-check exceptions so one fail doesn't abort the suite) and emits human or JSON output. Exit code 1 on any failure. Two macOS-specific design notes: 1. ArcticDB libs are initialized once in check_arctic_connectivity and reused across downstream checks via the context — re-initializing adb.Arctic() crashes Aws::S3::S3Client::S3Client on macOS. 2. Checks are ordered with arctic_connectivity FIRST so its bundled AWS SDK loads before boto3 (which gets pulled in by collectors imports). Polygon check skips gracefully (WARN, not FAIL) when POLYGON_API_KEY is unset — supports laptop-side preflight where the .env isn't loaded. On the spot the key is present and the check fires. 18 tests in tests/test_sf_preflight.py — happy path + each failure mode each check is designed to catch + orchestrator isolation. 394 tests total. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI runs without POLYGON_API_KEY in env, so the no-key skip-to-WARN guard short-circuited the 3 polygon-coverage tests before they reached the mocked client. Set the env var via monkeypatch so the guard passes through to the polygon mock. Also add explicit test for the no-key path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merged
4 tasks
cipher813
added a commit
that referenced
this pull request
May 28, 2026
…across all artifacts (#339) Closes the gap surfaced 2026-05-28: current-state probe answers 'is the artifact present now?' but operators also need 'did it land last weekend? are there gaps in the producer's history?' Filed per the same feedback memory observe_mode_unconditional_gates — absence-of-artifact is the failure mode, and a single-cycle absence could be a false-positive where a multi-cycle gap is a real producer regression. Adds: - event['mode']='historical' dispatch in handler(). Routes to a new _handle_historical(s3, now, started_at, lookback_overrides) path that walks the registry, probes the last N cycles per artifact, and writes _freshness_monitor/history.json (page 26 will surface per-row history expanders + gap counts). - New EB cron alpha-engine-freshness-monitor-historical-cron (daily 04:00 UTC, off-peak) wired in deploy.sh --bootstrap. - Default lookback: 12 saturday_sf + 30 weekday_sf/eod_sf cycles (~3 months each). continuous skipped (current-state covers). Tunable via event['lookback'] override. 403/404/NoSuchKey normalization: S3 returns 403 (not 404) for missing keys when the Lambda lacks s3:ListBucket. Treat both as cleanly-absent (no error_code in output) so page 26 doesn't show spurious '403 errors' on legitimately-absent historical cycles. 9 new unit tests cover: saturday/weekday/eod cycle-date resolution, continuous skip, zero-count short-circuit, date/trading_day/no-placeholder template rendering, and handler mode-dispatch. Live smoke (post-deploy + manual invoke): n_artifacts=51, n_cycles_probed=474, duration=10.08s Surfaced 1 real finding for follow-up: several artifacts use calendar-vs-trading-day-anchored templates that don't match producer behavior. research_signals registered as signals/{date}/signals.json with cadence=saturday_sf, but producer writes to mostly Friday trading-day keys (2026-05-22, 2026-05-15, etc.). The historical probe correctly reports the Saturday keys as absent — which IS the right answer given the registry template. ROADMAP follow-up filed separately to audit all registry templates for calendar-vs-trading-day mismatch. Calendar-naive by design — NYSE holidays surface as false-positive absent days but operators can interpret in context. Calendar-aware backfill is a P3 follow-up if the noise becomes worth the dependency lift. Composes with the OBSERVATION_REGISTRY arc (#349/#351/#352/#355 + #135/#136/#137). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check_backfill_source_freshness↔ PR fix(backfill): apply daily_closes delta + regression preflight #130check_polygon_grouped_coverage↔ PR fix(polygon): per-ticker fallback for grouped-daily coverage gaps #131check_predicted_missing_from_closes↔ PR fix(daily_append): scope missing-from-closes check to caller's request list #132check_universe_sample_freshness↔ PR fix(daily_append): scope post-write freshness scan to expected_tickers #133check_universe_drift↔ PR fix(morning_enrich): refresh constituents + prune stragglers before write #134check_constituents_fetch↔ PR fix(constituents): include tickers in collect() return dict #135check_postflight_contracts↔ existing postflightcheck_arctic_connectivity↔ ArcticDB cluster reachabilitySample output (run against current state, post-merge of #130-#135)
Design notes (macOS workarounds)
check_arctic_connectivityand reused viaPreflightContext— re-initializingadb.Arctic()crashesAws::S3::S3Client::S3Clienton macOS.arctic_connectivityfirst so its bundled AWS SDK loads before boto3 (which gets pulled in by collectors imports).POLYGON_API_KEYis unset — supports laptop preflight without the spot's .env.Test plan
test_sf_preflight.py: happy path + each failure mode each check catches + orchestrator isolationFollowups (not in this PR)
expected_tickersbandages in fix(daily_append): scope missing-from-closes check to caller's request list #132/fix(daily_append): scope post-write freshness scan to expected_tickers #133 once 2-3 clean Saturday SFs validate the architectural reorder🤖 Generated with Claude Code