feat(backtester): shell-run dry path for replay concordance + counterfactual (closes SF skip-exceptions)

[cipher813]

Summary

Closes the last two Saturday-SF shell-run keystone skip-exceptions (ReplayConcordance / Counterfactual) by giving both replay Image-Lambda handlers a verified clean no-write dry path, so the keystone can route them dry (boot + imports for real, benign success before the scan) instead of pure-skipping them.

Dry event-key (verbatim)

"dry_run_llm" (boolean) — reuses the canonical keystone key established verbatim by the Research Lambda in alpha-engine-data step_function.json ("dry_run_llm.$": "$.research_dry"). No invented key. Distinct from the handlers' pre-existing dry_run event key, whose compute-but-don't-emit-metrics semantic is intentionally left untouched (backward compatible — proven by retained legacy tests).

The SF rewire (the ReplayConcordance/Counterfactual states must pass "dry_run_llm": true under shell_run, and stop hard-skipping them) is a separate alpha-engine-data follow-on PR — not in this repo.

Shared helper (canonical, no copy-paste)

replay/__init__.py — the single implementation imported by both handlers:

• SHELL_RUN_DRY_EVENT_KEY = "dry_run_llm"
• is_shell_run_dry(event) — tolerant bool / str / None coercion
• shell_run_dry_response(handler_name, t0) — benign {"status": "DRY_RUN", ...} envelope

Per-handler short-circuit proof

lambda_concordance/handler.py — the dry check sits after _ensure_init() and after the deferred from replay.batch import compute_and_emit_concordance (boot + module imports run for real — the keystone's whole point), but before the compute_and_emit_concordance(...) call. That call is the sole entry to the replay.batch decision_artifacts S3 scan, the langchain_anthropic / target-model replay calls, the CloudWatch agent_cheap_model_concordance emit, and the S3 summary persist. The dry path returns before all of them → zero external/LLM calls, zero S3/CW writes.

lambda_counterfactual/handler.py — symmetric: dry check after _ensure_init() + from replay.counterfactual import compute_and_emit, before that call (the sole entry to the decision_artifacts S3 scan + sklearn fit + CloudWatch agent_counterfactual_rule_fit emit + S3 per-agent persist). No LLM on this path regardless.

Both return a status the SF (Catch-wrapped, non-blocking) treats as success.

Known SEPARATE issue (OUT OF SCOPE)

Counterfactual times out at 600s on real Saturday runs from corpus growth (~32,740 artifacts; last success 2026-05-13; the SF Catch swallows it). This PR does not touch the scan logic and does not attempt to fix that timeout — it remains a distinct bug to be tracked separately. Aside: because the dry path skips the scan entirely, it incidentally avoids the timeout under shell_run only; the real-Saturday timeout is unaffected and still owed.

Tests

• +15 handler tests (TestShellRunDryPath ×2): dry short-circuits before scan (compute never called, _ensure_init still runs, success status), string-true coercion, dry=false / absent take the real path, legacy dry_run still takes the real path with emit_metrics=False.
• New tests/test_replay_shell_run_dry.py for the shared helper.
• Full backtester suite: 1684 passed, 5 skipped, 1 deselected (parity deselected as usual), 0 failed. No new deps, no secrets.

🤖 Generated with Claude Code