Skip to content

Emit daily_data module health stamp after daily_closes.collect#41

Merged
cipher813 merged 1 commit into
mainfrom
feat/daily-data-health-stamp
Apr 16, 2026
Merged

Emit daily_data module health stamp after daily_closes.collect#41
cipher813 merged 1 commit into
mainfrom
feat/daily-data-health-stamp

Conversation

@cipher813
Copy link
Copy Markdown
Owner

Summary

Adds `health/daily_data.json` emission after `daily_closes.collect()` in `_run_daily()`, using the schema consumed by the executor's `check_upstream_health()` (`alpha-engine/executor/health_status.py:91`).

Belt-and-suspenders with the direct `LastModified` gates in alpha-engine#53 and alpha-engine-predictor#33:

Check Catches Catches that
Direct `LastModified` (already in #53/#33) "stamp green, blob stale"
Module health stamp (this PR) "ran and failed" with structured reason partial-success `degraded` state, observability, post-mortem metadata

The stamp alone would leave the silent-upstream-staleness hole — stamp says green while the blob is yesterday's. That's why both PRs exist.

Change

  • New `_write_module_health()` helper matching executor's `write_health()` schema (`module`, `status`, `last_success`, `run_date`, `duration_seconds`, `summary`, `warnings`, `error`).
  • Called in `_run_daily()` immediately after `daily_closes.collect()`, scoped to that collector alone (feature store + ArcticDB append have their own lifecycle; the executor reads `daily_closes/{date}.parquet`, so the stamp reflects only that).
  • Emitted on both success AND failure — `last_success: null` when status=failed lets readers distinguish "ran and failed today" from "hasn't run in N hours."
  • Legacy `_write_health_marker()` phase stamps (`health/data_phase0.json` etc.) kept untouched for backward compat.

Test plan

  • 5 new unit tests in `tests/test_module_health.py` — ok/failed/degraded/key pattern/defaults
  • Full suite: 71 passed / 0 failed
  • Post-merge: first weekday run (Mon 13:05 UTC) emits `health/daily_data.json` with status=ok, tickers_captured≈909

Deploy sequencing

  1. Merge this PR first — data stamps start emitting on next weekday run
  2. Then merge follow-up alpha-engine PR that adds `"daily_data": 24` to `_UPSTREAM_MAX_AGE_H`
  3. If executor PR merges first, every executor run fails with "no health data returned" until data PR deploys

Follow-up (not in this PR)

  • Migrate Phase 1 and Phase 2 stamp writes from `_write_health_marker(phase=N)` to `_write_module_health(module_name=...)`. Out of scope for belt-and-suspenders on `daily_data`.
  • Partial-success semantics (status=degraded when N < threshold tickers captured). Placeholder in schema; policy decision for a later PR.

🤖 Generated with Claude Code

Adds a module-scoped health stamp (health/daily_data.json) that matches
the schema consumed by alpha-engine's executor upstream gate
(executor/health_status.py::check_upstream_health). Complements the
direct LastModified check added in alpha-engine#53 — stamp catches
"ran and failed" state with structured reason/summary; direct check
catches "ran ok by the stamp but the artifact is stale."

Scoped to daily_closes only, not the whole daily pipeline — the
executor consumes predictor/daily_closes/{date}.parquet, which is
written by this collector. Feature store + ArcticDB append run
after and have their own concerns; their failures don't invalidate
the daily_closes write for the executor's purposes.

Emitted on BOTH success and failure paths so downstream can
distinguish "ran and failed today" (last_success=null, status=failed)
from "hasn't run in N hours" (stamp absent).

- new _write_module_health() helper using executor's schema
- kept _write_health_marker() phase stamps for backward compat
- 5 tests covering ok / failed / degraded / key pattern / defaults

Pairs with:
- cipher813/alpha-engine#53 (executor-side LastModified gate)
- cipher813/alpha-engine-predictor#33 (predictor-side LastModified gate)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@cipher813 cipher813 merged commit 4a14d0b into main Apr 16, 2026
1 check passed
@cipher813 cipher813 deleted the feat/daily-data-health-stamp branch April 16, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant