Skip to content

feat(spot): add --rag-only mode (skip DataPhase1, run RAG alone)#63

Merged
cipher813 merged 1 commit into
mainfrom
feat/spot-rag-only-mode
Apr 20, 2026
Merged

feat(spot): add --rag-only mode (skip DataPhase1, run RAG alone)#63
cipher813 merged 1 commit into
mainfrom
feat/spot-rag-only-mode

Conversation

@cipher813
Copy link
Copy Markdown
Owner

Summary

  • Adds `--rag-only` flag to `spot_data_weekly.sh` for post-incident surgical reruns where DataPhase1 succeeded but RAG failed downstream.
  • Fetches 4 RAG secrets from SSM, runs `rag/pipelines/run_weekly_ingestion.sh` (real, non-dry-run), emits `Process=rag-ingestion` heartbeat only (skips `data-phase1`).
  • Motivation: tonight's Sunday recovery. Last Saturday's SF cleared DataPhase1 at 2026-04-18T03:47 UTC (verified `s3://alpha-engine-research/health/data_phase1.json`) then RAG hung on the `.env`-sourced `RAG_DATABASE_URL` truncation. PR feat(spot): fetch RAG secrets from SSM instead of sourcing .env #60 root-fixed the bash-metachar bug; this PR gives us the operational verb to re-run the failed step alone.

Modes (now 4)

  • `` / `full` — DataPhase1 + RAG, default scheduled-run shape
  • `--smoke-only` — bootstrap + Phase1 `--dry-run`
  • `--rag-smoke-only` — bootstrap + RAG secrets + submodule imports + RAG `--dry-run`
  • `--rag-only` (new) — bootstrap + RAG secrets + RAG real ingestion; skips DataPhase1 entirely

Test plan

  • `bash -n` syntax check passes
  • After merge: `bash infrastructure/spot_data_weekly.sh --rag-only` from laptop — should fetch 2026-04-12 signals via the hardened `--from-signals` resolver (fix(universe_returns): unstick NULL return_5d rows + drop weekend signals #38) and ingest into Neon pgvector
  • Heartbeat metric `AlphaEngine/Heartbeat{Process=rag-ingestion}` emits; no `data-phase1` heartbeat

🤖 Generated with Claude Code

…lone)

When DataPhase1 completes cleanly but RAG fails downstream (e.g. 2026-04-17
Saturday SF: DataPhase1 OK at 03:47 UTC, RAG hung on .env-sourced
RAG_DATABASE_URL truncation — PR #60 root-fix), rerunning the whole Saturday
SF burns ~30 minutes of spot time + polygon API quota on idempotent
DataPhase1 overwrite.

New mode skips DataPhase1 entirely:
  ./infrastructure/spot_data_weekly.sh --rag-only

Fetches the 4 RAG secrets from SSM (same Phase 2 SSM path as the full run),
invokes rag/pipelines/run_weekly_ingestion.sh (real, non-dry-run), emits
only the rag-ingestion CloudWatch heartbeat so alarm state reflects what
actually ran.

Use-case: post-incident surgical re-run that unblocks Research (the SF's
next downstream step) without re-executing already-successful upstream work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cipher813 cipher813 merged commit 0878cb7 into main Apr 20, 2026
1 check passed
@cipher813 cipher813 deleted the feat/spot-rag-only-mode branch April 20, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant