Adopt alpha-engine-lib v0.1.0 for preflight + logging#28
Merged
Conversation
Replaces the inline log_config.py (committed 2026-04-14 in PR #26) and the proposed-then-closed inline _preflight (PR #27) with the shared library. alpha-engine-lib is now the single source of truth for these cross-cutting concerns; drift across consumer repos becomes impossible by construction. Changes - requirements.txt: add alpha-engine-lib @ git+…@v0.1.0 with [arcticdb,flow_doctor] extras. Drop the direct flow-doctor pin — pulled in transitively by the extra. - preflight.py: new module with DataPreflight(BasePreflight). Composes mode-specific check sequences (daily / phase1 / phase2) on top of the shared primitives. ~40 LOC. - weekly_collector.py: - Import setup_logging from alpha_engine_lib.logging (was local log_config). Pass flow-doctor.yaml path explicitly since the lib version is path-parametric (each consumer has its own yaml). - Call DataPreflight(...).run() at the top of main(), after config load, before run_weekly(). - log_config.py: deleted. The lib version is now the sole copy. Test plan - [x] pytest tests/ — 41 pass - [x] Import smoke: `from preflight import DataPreflight` works against locally-installed alpha-engine-lib v0.1.0 - [ ] EC2: pip install -r requirements.txt must succeed. The git+ URL requires the existing ~/.netrc PAT to have Contents: read on cipher813/alpha-engine-lib. If it doesn't, the install fails loud and visibly (no silent fall-through). - [ ] Next weekday DailyData run: confirm "Pre-flight OK (mode=daily)" appears before "COLLECTING: daily closes". - [ ] Forced failure test: `FRED_API_KEY= python weekly_collector.py --phase 1 --dry-run` should raise at preflight. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI runner has no ~/.netrc, so the git+https pip install from the private alpha-engine-lib repo fails with "could not read Username for 'https://github.com'". Added a pre-install step that rewrites the alpha-engine-lib URL to include an x-access-token, using a PAT stored as the ALPHA_ENGINE_LIB_TOKEN GitHub Actions secret. Hard-fails with an actionable error if the secret isn't set, rather than letting pip fail with a confusing subprocess error further down. Prerequisite before merge: add ALPHA_ENGINE_LIB_TOKEN as a repo secret on cipher813/alpha-engine-data. Any fine-grained PAT with Contents: read on cipher813/alpha-engine-lib works. Can reuse an existing PAT if it already has (or can be extended to include) that scope. Scope only the lib URL (not a global github.com rewrite) so other git operations — including actions/checkout of the current repo via GITHUB_TOKEN — continue to work unchanged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This was referenced Apr 14, 2026
cipher813
added a commit
that referenced
this pull request
Apr 14, 2026
Applies the PR #24/#25/#28 pattern to the Saturday RAG ingestion path. The shell script was silently swallowing failures in all 5 pipelines, making "RAG Weekly Ingestion Complete" a lie whenever any step failed. Shell script (rag/pipelines/run_weekly_ingestion.sh) - Removed `|| echo "WARNING: ... (non-fatal)"` from all 5 ingestion steps, the CloudWatch heartbeat, and the completion email. set -e was already active but these swallowers defeated it. - Removed the runtime `if [ -n "$FINNHUB_API_KEY" ]; then ... else echo SKIPPED fi` branch. All required env vars are now hard-failed by preflight (step 0) before any ingestion runs. A silently-skipped earnings transcript step defeats the purpose of having transcripts at all. - Added `Step 0/5: python -m rag.preflight` at the top. - The hardcoded 'status: ok' completion email is now truthful rather than aspirational — with set -e active and no swallowers, reaching the email means all 5 pipelines actually succeeded. New file: rag/preflight.py - RAGPreflight(BasePreflight) subclass — composes check_env_vars (AWS_REGION, VOYAGE_API_KEY, FINNHUB_API_KEY, EDGAR_IDENTITY, RAG_DATABASE_URL) + check_s3_bucket. - main() uses alpha-engine-lib's setup_logging with the shared flow-doctor.yaml path, so a preflight failure fires email + issue via the existing dispatch. rag/db.py - is_available(): log.debug → log.warning for the exception path. The function was otherwise unchanged — it's a non-raising probe for future retrieval-side consumers. Flagged as unused inside alpha-engine-data (zero callers); defer deletion until cross-repo audit completes, since predictor / research / backtester may import from it. rag/pipelines/ingest_8k_filings.py - Per-URL download failure: log.debug → log.warning. Caller still treats None as "skip this filing" (aggregate counts are reported), so no behavior change; the failure rate is just visible now. Dead code flagged (no change in this PR) - rag/db.py::is_available — zero local callers. Keep for now, flag for future cross-repo sweep. Out of scope (tracked) - Adopt alpha-engine-lib setup_logging in each ingestion script's main() for consistent log formatting + flow-doctor capture of per-pipeline errors. Currently only preflight.py uses the lib; ingestion scripts still use Python's default root logger. Minor follow-up. - Date-parsing `except ValueError: continue` patterns in ingest_sec_filings, ingest_8k_filings, ingest_theses, ingest_earnings_transcripts. Reviewed case-by-case — all are legitimate "skip this malformed entry" flows with aggregate counts upstream. Not silent fails. Test plan - [x] pytest tests/ — 41 pass - [x] Syntax check on all modified Python files - [x] bash -n on run_weekly_ingestion.sh - [ ] Next Saturday Step Function run exercises the hardened path. Forced failure test: unset FINNHUB_API_KEY on EC2 and re-run — must fail at preflight (step 0), not silently skip step 3. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5 tasks
cipher813
added a commit
that referenced
this pull request
Apr 15, 2026
Root-cause fix for the silent-fail that let 2026-04-15 weekday pipeline runs report SUCCEEDED while macro/SPY sat 5 days stale (4/10 → 4/15) until the inference Lambda preflight finally caught it via the PR #28 deploy canary. ## The silent-fail Old macro update loop: for key in macro_keys: bar = closes.get(key) if bar and not np.isnan(...): macro_lib.update(key, new_row) If `closes.get(key)` returned None (upstream daily_closes collection gap for that ticker), the update silently skipped. No log. No raise. daily_append returned status="ok". weekly_collector returned status="ok". Python exited 0. Step Function marked SUCCEEDED. macro never updated. Combined with the "all stocks skipped because backfill wrote today's row" case, an entire daily_append invocation could produce ZERO writes and still report success. ## The fix 1. Track macro_missing_from_closes list. Any key absent from closes (or with NaN Close) goes in the list instead of being silently skipped. 2. After both loops, if the list is non-empty, raise RuntimeError naming the missing keys. Macro inputs are not optional — SPY feeds return_vs_spy_5d, VIX feeds vix_level, sector ETFs feed sector-relative features. Upstream collection must produce them. 3. Sector ETF iteration now uses an explicit ["XLB","XLC",...] list rather than filtering closes.keys() with startswith("XL"). Missing sector ETFs now surface as rejects instead of silently-not-iterating. 4. After update() calls complete, read back every updated key and verify its last date matches today. A pure belt-and-suspenders check — update() has no return value to confirm success, so a silent no-op in ArcticDB's commit layer could still slip through the no-exception path without this readback. If verification fails, raise with the specific (key, reason) tuples. 5. Summary log line now surfaces macro_updated + sector_updated counts alongside the stock counts, so "ok with zero writes" is visible in the logs instead of hidden behind stock-only metrics. ## Test plan tests/test_daily_append_semantics.py — 3 new source-level regression tests locking the hard-fail semantics: - Missing macro keys raise - Verification readback exists - Sector ETFs iterate explicit list Full suite: 46 passed (was 43 + 3 new). ## Follow-up Investigate WHY the upstream daily_closes collector was missing macro keys in the 2026-04-15 08:39 PT rerun despite loading 920 tickers with SPY/VIX/etc verified present in the parquet. The missing-keys branch should not trigger in a healthy run — its purpose is to loudly halt the pipeline when upstream collection is broken, so we find out and fix it instead of running on stale data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 tasks
cipher813
added a commit
that referenced
this pull request
Apr 15, 2026
…37) Root-cause fix for the silent-fail that let 2026-04-15 weekday pipeline runs report SUCCEEDED while macro/SPY sat 5 days stale (4/10 → 4/15) until the inference Lambda preflight finally caught it via the PR #28 deploy canary. ## The silent-fail Old macro update loop: for key in macro_keys: bar = closes.get(key) if bar and not np.isnan(...): macro_lib.update(key, new_row) If `closes.get(key)` returned None (upstream daily_closes collection gap for that ticker), the update silently skipped. No log. No raise. daily_append returned status="ok". weekly_collector returned status="ok". Python exited 0. Step Function marked SUCCEEDED. macro never updated. Combined with the "all stocks skipped because backfill wrote today's row" case, an entire daily_append invocation could produce ZERO writes and still report success. ## The fix 1. Track macro_missing_from_closes list. Any key absent from closes (or with NaN Close) goes in the list instead of being silently skipped. 2. After both loops, if the list is non-empty, raise RuntimeError naming the missing keys. Macro inputs are not optional — SPY feeds return_vs_spy_5d, VIX feeds vix_level, sector ETFs feed sector-relative features. Upstream collection must produce them. 3. Sector ETF iteration now uses an explicit ["XLB","XLC",...] list rather than filtering closes.keys() with startswith("XL"). Missing sector ETFs now surface as rejects instead of silently-not-iterating. 4. After update() calls complete, read back every updated key and verify its last date matches today. A pure belt-and-suspenders check — update() has no return value to confirm success, so a silent no-op in ArcticDB's commit layer could still slip through the no-exception path without this readback. If verification fails, raise with the specific (key, reason) tuples. 5. Summary log line now surfaces macro_updated + sector_updated counts alongside the stock counts, so "ok with zero writes" is visible in the logs instead of hidden behind stock-only metrics. ## Test plan tests/test_daily_append_semantics.py — 3 new source-level regression tests locking the hard-fail semantics: - Missing macro keys raise - Verification readback exists - Sector ETFs iterate explicit list Full suite: 46 passed (was 43 + 3 new). ## Follow-up Investigate WHY the upstream daily_closes collector was missing macro keys in the 2026-04-15 08:39 PT rerun despite loading 920 tickers with SPY/VIX/etc verified present in the parquet. The missing-keys branch should not trigger in a healthy run — its purpose is to loudly halt the pipeline when upstream collection is broken, so we find out and fix it instead of running on stale data. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the inline `log_config.py` (#26) and the proposed-then-closed inline `_preflight` (#27) with the shared library `alpha-engine-lib` v0.1.0 (https://github.com/cipher813/alpha-engine-lib).
Single source of truth for these cross-cutting concerns; drift across consumer repos becomes impossible by construction once all 5 modules adopt the lib.
Changes
Test plan
PAT prerequisite
If the EC2 `~/.netrc` PAT doesn't already include `alpha-engine-lib`, add it before merging:
```
On EC2 — PAT token value never visible in logs as long as you paste
it directly into ~/.netrc and don't echo it
machine github.com
login ec2-user
password
```
(Note: never `cat ~/.netrc` or `env` to stdout — that would expose the token.)
Rollout after this merges
🤖 Generated with Claude Code