Skip to content

feat(data): spot_drift_detection.sh --preflight-only (Friday shell-run dry path — closes DriftDetection skip-exception)#261

Merged
cipher813 merged 1 commit into
mainfrom
feat/spot-drift-detection-preflight-only
May 18, 2026
Merged

feat(data): spot_drift_detection.sh --preflight-only (Friday shell-run dry path — closes DriftDetection skip-exception)#261
cipher813 merged 1 commit into
mainfrom
feat/spot-drift-detection-preflight-only

Conversation

@cipher813
Copy link
Copy Markdown
Owner

Adds a --preflight-only modifier to infrastructure/spot_drift_detection.sh, mirroring the merged alpha-engine-data #259 (spot_data_weekly.sh) / predictor #175 / backtester #224 pattern. Closes the DriftDetection skip-exception in ROADMAP "Friday shell-run — per-module dry-path activation" — the one per-module SF step still SKIPPED rather than dry-run on the Friday shell_run.

Insertion point

  • PREFLIGHT_ONLY=0 modifier var, initialised before the arg-parse loop (orthogonal to RUN_MODE, set -u safe).
  • --preflight-only) PREFLIGHT_ONLY=1; shift ;; added to the case loop; echo line added to the run banner.
  • Guard block inserted after the smoke-only block and strictly before the # ── Full drift detection ── section (the run_remote bash -s <<DRIFT heredoc) and before the trailing aws cloudwatch put-metric-data heartbeat.

No-scan / no-write proof

monitoring.drift_detector (lives in alpha-engine-predictor, reached via the sibling-clone PYTHONPATH) is the sole code path doing any S3 get_object/put_object of the drift report or SNS publish on alert; the launcher's CloudWatch put-metric-data heartbeat trails it. The PREFLIGHT_ONLY guard exit 0s strictly before the <<DRIFT heredoc, so the drift scan, the SNS publish, the S3 put_object, and the CloudWatch emit are all statically unreachable. The preflight itself runs only BasePreflight.check_env_vars (env read) + BasePreflight.check_s3_bucket (bucket HEAD) + an importlib.import_module of the drift module (import-only — the boto3 clients + check_drift()/main() sit behind if __name__ == "__main__", which an import does not trigger). Hard invariant holds: zero external API data fetch, zero S3/CW/SNS/config mutation; early exit 0 because a passed preflight is a healthy outcome (SSM/SF report Success).

Preflight substrate reused

The drift workload binary lives in alpha-engine-predictor (no --preflight-only of its own; out of scope to modify here) and this repo's preflight.py DataPreflight modes (daily/morning_enrich/phase1/phase2) are data-collection scoped — none maps to drift. Per the canonical-lib fallback the preflight composes alpha_engine_lib.preflight.BasePreflight directly (env-vars + S3 HEAD, both read-only). No bespoke preflight scaffolding duplicated.

Verbatim flag name

--preflight-only

Tests

New tests/test_spot_drift_detection_preflight_only.py — 5 static grep / source-position assertions, mirroring tests/test_preflight_only_dry_path.py:

  1. flag parses as a modifier (PREFLIGHT_ONLY=1) + PREFLIGHT_ONLY=0 init present
  2. guard precedes the DRIFT section and the CloudWatch heartbeat
  3. guard block exit 0s before the DRIFT section
  4. no drift scan / S3 / CW / SNS write inside the block
  5. canonical BasePreflight reused (env-vars + S3 HEAD) + import-only drift smoke, no scaffolding

bash -n infrastructure/spot_drift_detection.sh clean. Full data suite: 1342 passed, 1 skipped (pre-existing), 5 pre-existing warnings (unrelated daily_append.py pandas FutureWarning).

Relationship to #260

Independent: #260 touches spot_data_weekly.sh + the Lambda --dry-run keystone (a different file). The Saturday/Friday SF rewire to route the DriftDetection state at this --preflight-only flag under the Friday shell_run is a separate follow-on — no step_function.json change here.

🤖 Generated with Claude Code

…n dry path — closes DriftDetection skip-exception)

Adds a `--preflight-only` modifier to infrastructure/spot_drift_detection.sh,
mirroring the merged #259 (spot_data_weekly.sh) / predictor #175 /
backtester #224 pattern. Closes the DriftDetection skip-exception in
ROADMAP "Friday shell-run — per-module dry-path activation" — the one
per-module SF step still SKIPPED rather than dry-run on the Friday shell_run.

Insertion point
---------------
`PREFLIGHT_ONLY=0` modifier var initialised before the arg-parse loop
(orthogonal to RUN_MODE, `set -u` safe); `--preflight-only) PREFLIGHT_ONLY=1`
added to the case loop. The guard block is inserted AFTER the smoke-only
block and strictly BEFORE the "# ── Full drift detection ──" section (the
`run_remote bash -s <<DRIFT` heredoc) and before the trailing
`aws cloudwatch put-metric-data` heartbeat.

No-scan / no-write proof
------------------------
`monitoring.drift_detector` (in alpha-engine-predictor, on the sibling-clone
PYTHONPATH) is the SOLE code path that does any S3 get_object/put_object of
the drift report or SNS publish on alert; the launcher's CloudWatch
put-metric-data heartbeat trails it. The PREFLIGHT_ONLY guard `exit 0`s
strictly before the `<<DRIFT` heredoc, so the scan, the SNS publish, the S3
put_object, and the CloudWatch emit are all statically unreachable. The
preflight itself runs only BasePreflight.check_env_vars (env read) +
BasePreflight.check_s3_bucket (bucket HEAD) + an `importlib.import_module`
of the drift module (import-only — boto3 clients + check_drift()/main()
sit behind `if __name__ == "__main__"`, which an import does not trigger).
Zero external API data fetch, zero S3/CW/SNS/config mutation; exit 0
because a passed preflight is a healthy outcome (SSM/SF report Success).

Preflight substrate reused
--------------------------
The drift workload binary lives in alpha-engine-predictor (no
--preflight-only of its own; out of scope to modify here) and this repo's
preflight.py DataPreflight modes (daily/morning_enrich/phase1/phase2) are
data-collection scoped — none maps to drift. Per the canonical-lib
fallback the preflight composes `alpha_engine_lib.preflight.BasePreflight`
DIRECTLY (env-vars + S3 HEAD) — no bespoke preflight scaffolding duplicated.

Verbatim flag name: `--preflight-only`

Tests
-----
New tests/test_spot_drift_detection_preflight_only.py (5 static
greps/source-position assertions, mirroring
tests/test_preflight_only_dry_path.py): flag parses as a modifier;
guard precedes DRIFT + heartbeat; exit 0 before DRIFT; no scan/S3/CW/SNS
in block; canonical BasePreflight reused (no scaffolding). `bash -n`
clean. Full data suite: 1342 passed, 1 skipped (pre-existing), 5
pre-existing warnings.

Independent of #260: that PR touches spot_data_weekly.sh + the Lambda
dry-run keystone (a different file); the Saturday/Friday SF rewire to
route the DriftDetection state at this `--preflight-only` flag under the
Friday shell_run is a separate follow-on (no step_function.json change here).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cipher813 cipher813 merged commit d1e7d9d into main May 18, 2026
1 check passed
@cipher813 cipher813 deleted the feat/spot-drift-detection-preflight-only branch May 18, 2026 22:11
cipher813 added a commit that referenced this pull request May 18, 2026
…ht + EvalJudge/Rationale/Replay/CF dry_run_llm) (#263)

Closes the keystone gap: the 5 documented shell-run skip-exceptions are
flipped skip→dry. Under shell_run EVERY substantive workload now boots +
runs dry; ZERO skip-exceptions remain. All prerequisite dry flags were
already MERGED on origin/main of their repos.

Per-state mechanism:

| State                       | Type   | Mechanism (under shell_run)                          |
|-----------------------------|--------|------------------------------------------------------|
| DriftDetection              | spot   | commands.$ States.Format($.preflight_args) → ` --preflight-only` (data #261) |
| EvalJudgeSubmitFirstSaturday| Lambda | Payload "dry_run_llm.$": "$.research_dry" (research #202) |
| EvalJudgeSubmitWeekly       | Lambda | Payload "dry_run_llm.$": "$.research_dry" (research #202) |
| EvalJudgePoll               | Lambda | Payload "dry_run_llm.$": "$.research_dry" (research #202) |
| EvalJudgeProcess            | Lambda | Payload "dry_run_llm.$": "$.research_dry" (research #202) |
| RationaleClustering         | Lambda | Payload "dry_run_llm.$": "$.research_dry" (research #202) |
| ReplayConcordance           | Lambda | Payload "dry_run_llm.$": "$.research_dry" (backtester #225) |
| Counterfactual              | Lambda | Payload "dry_run_llm.$": "$.research_dry" (backtester #225) |

Exact canonical dry var: $.research_dry. It is THE canonical shell-run
LLM-dry signal — InitializeInput seeds it false on every run (so the
absent path / real Sat 02:00 PT firing is unchanged); ApplyShellRunDefaults
already sets it true under shell_run (it backed Research from the
keystone). No new var invented — research #202 / backtester #225 PR bodies
specify dry_run_llm, and reusing $.research_dry keeps the absent-path
guarantee automatic (no extra seeding needed; the seed already exists).

Changes:
- ApplyShellRunDefaults: removed skip_drift_detection / skip_eval_judge /
  skip_rationale_clustering / skip_replay_concordance / skip_counterfactual
  from the force-set JsonMerge blob. It now force-sets ZERO skip_*.
  Per-flag user overrides still win (merge order unchanged). The
  Choice-gated CheckSkip<State> gates are LEFT INTACT (still valid for
  targeted operator skips — verified by test_skip_gates_still_intact).
- DriftDetection: literal `commands` array → `commands.$` States.Array
  whose final entry is States.Format('bash infrastructure/
  spot_drift_detection.sh{} 2>&1 | tee /var/log/drift-detection.log',
  $.preflight_args). {} sits immediately after the script token with no
  literal space; preflight_args carries its leading space inside the var,
  so preflight_args="" reproduces the origin/main command char-for-char
  and " --preflight-only" yields exactly one separating space.
- 7 eval Lambda Payloads: added "dry_run_llm.$": "$.research_dry".
  EvalRollingMean (alpha-engine-research-eval-rolling-mean) was NOT touched
  — it has no skip gate, was never a keystone exception, and is a pure
  historical-metric reader (out of scope).

Byte-identical proof approach:
- shell_run absent ⇒ CheckShellRun.Default = CheckSkipMorningEnrich
  (unchanged); InitializeInput seeds preflight_args="", research_dry=false.
  Every spot States.Format resolves char-for-char to the frozen
  origin/main literal; every eval Lambda dry_run_llm.$ resolves to false
  (handlers default it false ⇒ behaviourally identical to pre-rewire).
- The frozen baseline fixture tests/fixtures/sf_prekeystone_spot_commands
  .json now INCLUDES DriftDetection's pre-rewire origin/main literal
  command (regenerated via the established generator at preflight_args="";
  the existing 7 entries are unchanged). The byte-identical test asserts
  DriftDetection's resolved command at preflight_args="" equals that
  frozen baseline and carries --preflight-only (single space) under
  shell_run.
- CI-safe: tests read only the committed fixture (no `git show
  origin/main` shell-out — that was the #260 CI failure).

Tests:
- _SPOT_STATES grew to 8 (added DriftDetection); _DRY_LAMBDA_STATES grew
  to 11 (added the 7 eval states); _KEYSTONE_SKIP_EXCEPTIONS = empty set.
- test_shell_defaults_force_set_ZERO_skip_exceptions asserts the blob
  force-sets no skip_* and none of the 16 workload skips (incl. the 5
  ex-exceptions) appear.
- TestHappyPathTraversal: under shell_run nothing is skipped (skipped ==
  set()); DriftDetection is VISITED (runs dry), not jumped past.
- Module + class docstrings updated to the rewire semantics.

JSON valid (58 top-level states, 91 incl. parallel branches). Full
alpha-engine-data suite: 1351 passed, 1 skipped, 0 failed.

Zero skip-exceptions remain — every substantive task runs dry under
shell_run (spots → --preflight-only, Lambdas → dry_run_llm).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant