test(live-smoke): paths-filtered Polygon + FRED + ArcticDB smokes (L258)#327
Merged
Conversation
Closes the mocked-test scope-shape gap for the three external services alpha-engine-data depends on. Mirrors the morning-signal #34 pattern shipped 2026-05-26 for the same class of bug: unit tests mock the external client, so payload-shape drift (field renames, schema deprecations, status-code semantics) is invisible to CI until production fires. Each smoke is its own paths-filtered workflow + skip-on-no-credentials script, so PRs that don't touch the relevant module skip the workflow entirely and forks without secrets get a clean skip rather than a failing CI status. Smokes: - Polygon: get_grouped_daily for the most recent US weekday; asserts every bar carries the {open, high, low, close, volume, vwap} keys the consumer reads. ~$0.01/run, gated on POLYGON_API_KEY. - FRED: fetch_fred_history("DGS2", period_years=1); asserts >=50 observations and "value" column present. Free tier, gated on FRED_API_KEY. - ArcticDB: read tail of SPY from universe library; asserts the canonical OHLCV_COLS + PROVENANCE_COL schema. Read-only, no writes. Gated on OIDC role assumption (github-actions-lambda-deploy). IAM grant: adds two scoped Statements to infrastructure/iam/github-actions-lambda-deploy.json: - ArcticDBSmokeReadObject: s3:GetObject on arcticdb/* (read-only) - ArcticDBSmokeListBucket: s3:ListBucket with prefix condition Operator-step on merge: `./infrastructure/iam/apply.sh github-actions-lambda-deploy` to push the new policy to AWS. The IAM drift check will fail until that runs. Secrets to add in GHA repo settings: POLYGON_API_KEY, FRED_API_KEY. Composes with morning-signal #34, alpha-engine-lib #78 (anthropic_payload chokepoint), and the L258 P0-retrospective entry in ROADMAP. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 27, 2026
cipher813
added a commit
that referenced
this pull request
May 27, 2026
alpha_engine_lib.alerts.publish's S3-backed dedup uses HeadObject + conditional PUT semantics that require s3:ListBucket on the bucket itself (scoped via the s3:prefix condition). The watchdog role previously had only s3:GetObject + s3:PutObject on the dedup-marker prefix; the missing ListBucket caused dedup probes to error with AccessDenied, and the lib's fail-safe-to-publish path correctly fired the alert anyway — but with dedup non-functional, every cron firing during a persistent outage re-paged the operator instead of collapsing under the 12h dedup window. The grant is scoped by an s3:prefix Condition to ``_alerts/_dedup`` + ``_alerts/_dedup/*`` so the watchdog can't enumerate other prefixes on the bucket. Mirrors the alpha-engine-data #327 (L258) precedent where ArcticDBSmokeListBucket added the same Condition-scoped grant for the OIDC role's ArcticDB smoke. Applied via ``aws iam put-role-policy`` against ``alpha-engine-pipeline-watchdog-role`` (verified live). The deploy.sh already re-applies the policy on every run, so the codified change is in lockstep with the live state. ROADMAP L295 (P2, 2026-05-26 PM audit finding). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the mocked-test scope-shape gap for the three external services alpha-engine-data depends on (ROADMAP L258 P0 retrospective). Mirrors the morning-signal #34 pattern shipped 2026-05-26 for the same class of bug: unit tests mock the external client, so payload-shape drift is invisible to CI until production fires.
Each smoke is its own paths-filtered workflow + skip-on-no-credentials script, so PRs that don't touch the relevant module skip the workflow entirely and forks without secrets get a clean skip rather than a failing CI status.
polygon_client.py,collectors/daily_closes.py,collectors/nasdaq_snapshot.pyPOLYGON_API_KEYcollectors/fred_history.py,collectors/daily_closes.pyFRED_API_KEYstore/arctic_store.py,builders/{daily_append,backfill,_price_cache_writeboth}.pyEach smoke skips cleanly when its credential is absent — verified locally.
IAM grant (operator step on merge)
Two new scoped Statements added to
infrastructure/iam/github-actions-lambda-deploy.json:ArcticDBSmokeReadObject—s3:GetObjectonarcticdb/*(read-only)ArcticDBSmokeListBucket—s3:ListBucketwith prefix condition scoped toarcticdbOn merge, the IAM drift check will fail until the operator runs:
```bash
./infrastructure/iam/apply.sh github-actions-lambda-deploy
```
Secrets to add in GHA repo settings
Until both secrets are added, those two workflows take the clean-skip path on every PR; once added, the smoke fires on the next PR touching the path-filtered files.
Test plan
POLYGON_API_KEY not set; skipping./FRED_API_KEY not set; skipping./no AWS credentials in env; skipping.)test_*.pymatch, no collection)POLYGON_API_KEY+FRED_API_KEYGHA secrets./infrastructure/iam/apply.sh github-actions-lambda-deployto push the new policypolygon_client.pyexercises the live-Polygon smoke greenstore/arctic_store.pyexercises the live-ArcticDB smoke greenComposes with morning-signal #34, alpha-engine-lib #78 (anthropic_payload chokepoint), and the L258 P0-retrospective entry in ROADMAP.
🤖 Generated with Claude Code