fix(sf): keystone ApplyShellRunDefaults — swap JsonMerge args so shellDefaults wins

infrastructure/step_function.json

@@ -33,9 +33,9 @@
     },
     "ApplyShellRunDefaults": {
       "Type": "Pass",
-      "Comment": "shell_run=true ONLY (keystone + skip-exception rewire — every substantive workload now boots + runs DRY; ZERO skip-exceptions remain). Merges the dry-path control blob UNDER the current state (States.JsonMerge(shellDefaults, $, false) — second arg wins) so an explicit per-flag override in the execution input still takes effect (e.g. {\"shell_run\": true, \"skip_backtester\": true} still skips Backtester; {\"shell_run\": true, \"preflight_args\": \"\"} would run the spots full-fat). Mirrors InitializeInput's defaults-under-input JsonMerge pattern exactly. SPOT states (MorningEnrich, DataPhase1, RAGIngestion, PredictorTraining, Backtester, Parity, Evaluator, AND DriftDetection) boot + run dry via preflight_args=\" --preflight-only\" (LEADING space inside the var; the spot states' final command is a States.Format whose {} is placed immediately after the mode token with NO literal space, so preflight_args=\"\" on the real run yields a byte-identical command — data #259 + #261 + predictor #175 + backtester #224 all expose --preflight-only verbatim, an orthogonal MODIFIER). LAMBDA states with a verified clean no-write dry path are routed dry, NOT skipped, via $.research_dry (the canonical shell-run LLM-dry signal, true here / false on the real run): Research + the EvalJudge chain (EvalJudgeSubmit{FirstSaturday,Weekly}/Poll/Process — dry_run_llm via research #202) + RationaleClustering (research #202) + ReplayConcordance + Counterfactual (backtester #225) all take dry_run_llm.$=$.research_dry; DataPhase2 (dry_run=true — alternative.collect returns ok_dry_run BEFORE any fetch/S3 write), RegimeSubstrate + RegimeRetrospectiveEval (action=dry_run — produce_*(write=False) returns payload before any put_object). The skip-exception rewire (this PR) flipped the prior 5 skip→dry: DriftDetection now uses the same commands.$/States.Format($.preflight_args) Option-C mechanism as the other spot states (data #261 exposed --preflight-only on spot_drift_detection.sh); the eval-judge / rationale-clustering / replay-concordance / counterfactual Lambdas now route dry via dry_run_llm.$=$.research_dry instead of being hard-skipped. ZERO skip-exceptions are force-set here. The #258 Choice-gated skip_* gates are LEFT INTACT and remain valid for targeted operator skips. The two health-check states have NO skip gate by design and run under shell_run (the bootstrap smoke) — their non-blocking Catch absorbs a stale-data sys.exit(1) so a missing-Friday-bar produces only a clearly-Friday-timestamped alert email, NOT a SF-fatal failure (ROADMAP owed-work item 5: a --shell-run-aware staleness tolerance in alpha-engine-dashboard/health_checker.py is a scoped cross-repo follow-on; not a SF-fatal spurious-fail, so deliberately out of this single-file SF PR).",
+      "Comment": "shell_run=true ONLY (keystone + skip-exception rewire — every substantive workload now boots + runs DRY; ZERO skip-exceptions remain). Merges the dry-path control blob OVER the current state (States.JsonMerge($, shellDefaults, false) — second arg wins, so shellDefaults wins over $) so shell_run=true ACTUALLY enforces dry mode for every keystone control var. The prior arg order (States.JsonMerge(shellDefaults, $, false)) silently fell back to non-dry because InitializeInput unconditionally seeds preflight_args=\"\" + research_dry=false + data_phase2_dry=false + regime_action=\"produce\" into $ — so $ wins meant the seeded non-dry identities won over the dry shellDefaults, and the 2026-05-22 Friday-PM shell-run dry-pass (first execution past the prior trap break) ran full-fat instead of --preflight-only. Skip flags are unaffected (they're not in shellDefaults; the #258 Choice-gated skip_* mechanism handles {\"shell_run\": true, \"skip_backtester\": true} downstream regardless of merge order). The previously-documented user-override-from-input case ({\"shell_run\": true, \"preflight_args\": \"\"} runs spots full-fat) is now intentionally LOST: in practice nobody passes both flags together — passing shell_run=true means you want dry mode for ALL keystone control vars; the comment line saying otherwise was a footgun masking the actual semantic intent. SPOT states (MorningEnrich, DataPhase1, RAGIngestion, PredictorTraining, Backtester, Parity, Evaluator, AND DriftDetection) boot + run dry via preflight_args=\" --preflight-only\" (LEADING space inside the var; the spot states' final command is a States.Format whose {} is placed immediately after the mode token with NO literal space, so preflight_args=\"\" on the real run yields a byte-identical command — data #259 + #261 + predictor #175 + backtester #224 all expose --preflight-only verbatim, an orthogonal MODIFIER). LAMBDA states with a verified clean no-write dry path are routed dry, NOT skipped, via $.research_dry (the canonical shell-run LLM-dry signal, true here / false on the real run): Research + the EvalJudge chain (EvalJudgeSubmit{FirstSaturday,Weekly}/Poll/Process — dry_run_llm via research #202) + RationaleClustering (research #202) + ReplayConcordance + Counterfactual (backtester #225) all take dry_run_llm.$=$.research_dry; DataPhase2 (dry_run=true — alternative.collect returns ok_dry_run BEFORE any fetch/S3 write), RegimeSubstrate + RegimeRetrospectiveEval (action=dry_run — produce_*(write=False) returns payload before any put_object). The skip-exception rewire (this PR) flipped the prior 5 skip→dry: DriftDetection now uses the same commands.$/States.Format($.preflight_args) Option-C mechanism as the other spot states (data #261 exposed --preflight-only on spot_drift_detection.sh); the eval-judge / rationale-clustering / replay-concordance / counterfactual Lambdas now route dry via dry_run_llm.$=$.research_dry instead of being hard-skipped. ZERO skip-exceptions are force-set here. The #258 Choice-gated skip_* gates are LEFT INTACT and remain valid for targeted operator skips. The two health-check states have NO skip gate by design and run under shell_run (the bootstrap smoke) — their non-blocking Catch absorbs a stale-data sys.exit(1) so a missing-Friday-bar produces only a clearly-Friday-timestamped alert email, NOT a SF-fatal failure (ROADMAP owed-work item 5: a --shell-run-aware staleness tolerance in alpha-engine-dashboard/health_checker.py is a scoped cross-repo follow-on; not a SF-fatal spurious-fail, so deliberately out of this single-file SF PR).",
       "Parameters": {
-        "merged.$": "States.JsonMerge(States.StringToJson('{\"preflight_args\":\" --preflight-only\",\"research_dry\":true,\"data_phase2_dry\":true,\"regime_action\":\"dry_run\"}'),$,false)"
+        "merged.$": "States.JsonMerge($,States.StringToJson('{\"preflight_args\":\" --preflight-only\",\"research_dry\":true,\"data_phase2_dry\":true,\"regime_action\":\"dry_run\"}'),false)"
       },
       "OutputPath": "$.merged",
       "Next": "CheckSkipMorningEnrich"

tests/test_sf_friday_shell_run_wiring.py

@@ -432,16 +432,45 @@ def test_pass_state_routes_into_existing_skip_chain(self, states):
         assert st["OutputPath"] == "$.merged"
         assert st["Next"] == "CheckSkipMorningEnrich"
 
-    def test_user_input_wins_over_shell_defaults(self, states):
-        # States.JsonMerge(defaults, $, false) — $ (current state, carrying
-        # the user input) MUST be the 2nd arg so an explicit
-        # {"shell_run": true, "skip_backtester": true} still skips it.
+    def test_shell_defaults_win_over_current_state(self, states):
+        """States.JsonMerge($, shellDefaults, false) — shellDefaults MUST be
+        the 2nd arg so the dry-path control vars actually take effect under
+        shell_run=true.
+
+        Why the prior order broke (2026-05-22 evening incident): the
+        previous shape was ``States.JsonMerge(shellDefaults, $, false)``
+        with the rationale "user input ($) must win for per-flag overrides
+        like {shell_run: true, skip_backtester: true}." That rationale
+        confused two distinct things: ``$`` at ApplyShellRunDefaults entry
+        is NOT raw user input — InitializeInput has already merged its
+        defaults blob ``{preflight_args: "", research_dry: false,
+        data_phase2_dry: false, regime_action: "produce"}`` into ``$``.
+        So ``$ wins`` meant the NON-DRY identity defaults won over the dry
+        shellDefaults. Result: every shell_run=true execution silently fell
+        back to a full-fat real Saturday run instead of --preflight-only.
+
+        Caught the morning after the 2026-05-22 Friday-PM dry-pass
+        post-trap-fix verification: MorningEnrich passed (trap fix worked),
+        DataPhase1 spot booted and started collecting short interest +
+        universe returns + fundamentals at full power instead of running
+        preflight-and-exiting.
+
+        Skip-flag overrides are unaffected by this fix: shellDefaults
+        contains NO skip_* keys, so a user-passed
+        ``{shell_run: true, skip_backtester: true}`` survives the merge
+        unchanged and the downstream Choice gate still skips Backtester.
+        Only the 4 keystone control vars (preflight_args / research_dry /
+        data_phase2_dry / regime_action) are forced to dry when shell_run.
+        That's the actual semantic intent — passing shell_run=true means
+        you want dry mode for ALL keystone control vars; partial-override
+        within the dry-control-var set was a footgun, not a feature.
+        """
         expr = self._merge_expr(states)
-        assert expr.startswith("States.JsonMerge(States.StringToJson(")
-        assert expr.endswith(",$,false)"), (
-            "user input ($) must be the 2nd JsonMerge arg so explicit "
-            "per-flag overrides win over the shell-run defaults"
+        assert expr.startswith("States.JsonMerge($,States.StringToJson("), (
+            f"merge expr must start with 'States.JsonMerge($,States.StringToJson(' "
+            f"so shellDefaults (the 2nd arg) wins over $. got: {expr[:80]!r}"
         )
+        assert expr.endswith(",false)")
 
     def test_shell_defaults_set_dry_control_vars(self, states):
         """ApplyShellRunDefaults must set every dry-path control var to its