fix(daily_closes): source-priority coalesce — never regress a cell to a less-informative value

[cipher813]

Why

The 2026-06-01 weekday Step Function failed at MorningEnrich. Root cause:

• A FRED 429 rate-limit storm meant TNX (DGS10, the 10Y yield) was never collected for the 5/29 target date.
• polygon_only mode overwrote the existing daily-closes parquet wholesale (canonical_existing_rows empty by construction). Polygon never serves ^TNX and yfinance was refused, so the rewrite blanked the TNX value the prior (Friday EOD) parquet already held.
• daily_append correctly hard-failed on the missing critical macro key → pipeline halt.

This violated the 2026-05-10 decision ("a cell is only updated if the data exists in the authoritative source, else the prior datapoint is retained"). The retain mechanism existed (canonical_existing_rows) but was gated to yfinance_only/auto; polygon_only opted out and did a destructive overwrite.

SOTA / institutional alignment

This implements the source-of-record waterfall that institutional market-data masters use (provenance per cell + priority-ranked COALESCE upsert + bitemporal revisions-not-blanking + loud degradation events). Verified against that pattern before implementing.

What changed (collectors/daily_closes.py)

• _coalesce_by_source_priority — for each ticker, keep the row from the highest-priority source across {prior parquet, this run}. _SOURCE_PRIORITY = {polygon:3, fred:3, yfinance:1} (polygon/fred are co-primary over disjoint domains — equities vs ^indices). Subsumes:
  • retain-on-empty — a ticker the live pass can't refresh keeps its prior row (fixes the TNX blank).
  • restatement wins — same-or-higher source overwrites (ties → fresh), so polygon's corporate-action-adjusted closes still land.
  • no source-downgrade — a yfinance backstop can't clobber a prior polygon close + true VWAP (prevents the 2026-04-17 VWAP=None contamination class).
  • null/NaN Close treated as missing (never wins, never written as an empty cell).
• polygon_only now coalesces before write instead of destructive overwrite. Runs after the coverage gate (computed on the fresh fetch), so a genuine polygon outage still hard-fails and is not masked by retained rows.
• Macro yfinance backstop — the equity-specific no-silent-fails refusal no longer blanket-blocks the FRED-index macro tickers (^TNX/^VIX/^IRX/^VIX3M). Polygon never serves them and they carry no VWAP, so FRED → yfinance is their legitimate chain. Loud WARN on fallback (records the FRED degradation; satisfies feedback_no_silent_fails). Equities still refuse yfinance.

no_silent_fails deviation rationale (per CLAUDE.md)

The macro yfinance backstop is a loud fallback, not a silent one: (a) failure mode = FRED rate-limit/outage on a FRED-only macro ticker; (b) primary deliverable (the parquet + macro keys) survives via the documented FRED→yfinance chain; (c) recording surface = explicit WARN log naming the ticker(s) + the polygon_only coalesce WARN counting retained/downgrade-blocked cells. Equity universe semantics are unchanged.

Tests

• New tests/test_daily_closes_source_priority_merge.py — 9 tests on the coalesce invariant (retain-on-empty, restatement, downgrade-block, null handling, unknown-source prior, mixed-scenario stats).
• tests/test_daily_closes_source_modes.py — added macro-backstop wiring test + end-to-end TNX-retain regression; tightened 2 docstrings that over-claimed the old destructive-overwrite contract.
• Full suite: 1742 passed, 1 skipped.

Out-of-scope follow-ups (filing to ROADMAP)

1. FRED rate-limit hardening — this PR makes the system resilient to FRED 429s but does not stop them. The windowed backfill fires ~N×2 FRED calls in a tight burst with no backoff/jitter. A FRED retry/backoff-with-jitter (SOTA retry primitive) is separate hardening.
2. Waterfall consolidation (P3) — polygon_only (coalesce) and yfinance_only/auto (canonical_existing_rows) are parallel preservation mechanisms; consider unifying on the priority-waterfall primitive.
3. Operator note — today's 6/1 weekday SF did not trade (halted pre-MorningEnrich-completion); no recovery re-run fired (pre-open window had closed).

🤖 Generated with Claude Code