Skip to content

META: Homeostasis — drive surface area to 100% terminal, a freshness layer that works, and demand-driven refresh (supersedes #1090) #1108

Open
Open
META: Homeostasis — drive surface area to 100% terminal, a freshness layer that works, and demand-driven refresh (supersedes #1090)#1108
Labels
enhancementNew feature or requestepicLong-running operating-map / meta tracker

Description

@holden
holden
opened on Jun 10, 2026

Supersedes #1090. That tracker did its job: it got us from "scattered backfills" to a measured surface area with a working freshness substrate. This issue is the reframe around the real end goal — homeostasis — with the current state laid out source-by-source, an honest evaluation of what's left, and the one build that actually matters next.

0. The goal, in one sentence

Homeostasis = the system keeps its own data complete and fresh with no babysitting.

Three layers, in order. Each is a prerequisite for the next:

  1. Surface area → 100% terminal. Every eligible data point is either fetched or confirmed source-absent (a durable "we tried, the source has nothing" marker). NOT "100% of every field" — much data doesn't exist at the source.
  2. A freshness layer that works. A uniform ledger that knows, for every data point: where it came from, when we fetched it, and whether it's due for a refresh.
  3. Demand-driven refresh. The system refreshes what's stale — proactively (capped background sweepers) and on access (you load a movie page → its stale data self-freshens). ← this is the prize.

Two numbers we never conflate:

  • Coverage % — we have the value.
  • Terminal % — we have it or confirmed the source doesn't. ← this is what we drive to 100%.

1. Current state — the whole surface area (📸 snapshot 2026-06-10 — regenerate via §9)

Totals: 1,165,578 movies · 680,710 people · freshness ledger = ~1.74M rows across 6 sources.

Layer 1 — fetchable sources (have a terminal goal)

Source Terminal Coverage / note Due now State
tmdb_details 100% every movie imported 131,411 ✅ terminal · ⏳ refreshing
tmdb_metrics 100% 934,797 rated + 230,200 voteless (ceiling) — ✅
omdb 99.98% 637,533 fetched + 17,455 source-absent (3,027 mat-debt + 137 needs-fetch = noise) 141,176 ✅ terminal · ⏳ refreshing
canonical_lists 100% 10/10 active lists — ✅
people_profile_path 100% 28,426 + 801 source-absent — ✅
people_biography ~100% 29,208 fetched, 0 pending, 4 due; coverage plateaus ~59% (rest have no TMDb bio) 4 ✅ just finished draining
festival_person_link 67% 8,486 / 12,623; gap is empty-payload noms → needs reimport — ⏳ draining (#1104), target 95%
watch_providers ~87% canonical / 15% full catalog ok 75,624 · empty 67,085 · error 5,639 35,999 ⏳ catalog crawl (capped)
imdb_id 56% 655,125 / 1,165,578 — TMDb returns null for ~510k — 🔒 structural ceiling

Derived (via OMDb/TMDb — coverage is source-absence, not backlog)

rotten_tomatoes 37,843 · metacritic 20,912 · tmdb_budget 68,954 · tmdb_revenue 26,115 — all at source ceiling.

Computed / supplemental / ephemeral (not fetch backlogs)

collaborations (graph) · person_quality_scores (weekly) · stock_images (on-demand) · wikidata (verification) · now_playing (ephemeral, continuously refreshed — working).

Read this table as: Layer 1 is essentially done. Everything is terminal or at a documented source-ceiling except three things still moving: festival (67%), watch-providers (catalog crawl), and making the imdb_id ceiling explicit (mark the ~510k as source-absent so 56% reads as 100% terminal).

2. How the freshness layer works (Layer 2 — ✅ DONE & live)

The substrate shipped (was #1090 Phase 1 / #1096). It is the foundation everything else stands on:

  • data_refreshes ledger — one polymorphic table keyed (entity_type, entity_id, source) storing fetched_at, stale_after, status (pending|ok|empty|error|ineligible), error_reason, attempt_count, last_checked_at. It stores freshness metadata only — the values still live in external_metrics / blobs.
  • Uniform API — Freshness.touch/5 (every fetch worker calls it), Freshness.stale?/3, Freshness.due/2. One vocabulary for all sources.
  • Policy registry — per-(entity_type, source) staleness rules: :age_tiered (recent movies refresh more often than classics) and :fixed_cadence, with empty-result and error backoff.
  • Live: ~1.74M ledger rows across all 6 sources. Cinegraph.Health.SurfaceArea.report() renders the table in §1 on demand.

What this buys us: for any data point we can now answer "where from / when / is it due?" — which is exactly the signal Layer 3 consumes to decide what to refresh.

3. Demand-driven refresh (Layer 3 — the end goal)

Two halves. One is shipping; the other is the real prize and isn't built.

3a. Proactive (background, capped) — 🟡 shipping now

#1106 — the unified per-movie TMDb refresh: one append_to_response call re-hydrates a movie's details + metrics + credits + watch-providers and touches the ledger for all of them at once (replacing the old siloed, single-purpose fetches). Driven by Freshness.due/2.

3b. On-access (read-through) — ❌ NOT built — the prize

A movie/person page view checks the ledger; if a source is stale, it enqueues the same unified refresh worker (debounced) so the page self-freshens. This is what makes the catalog stay current without any cron babysitting — true homeostasis. Design lives in #1010.

This is the most important remaining work. Everything else is cleanup or autopilot.

4. The "API budget governor" — what it is, and do we actually need it?

You asked what this is and where it came from. Straight answer:

What it was specced as (old #1090 Phase 4 / #1010 §6): a heavyweight subsystem — a per-source daily API-spend ledger (refresh_budgets table with atomic counters), hard per-source caps (OMDb 90k/day, TMDb 40k/day…), and a Budget.allow?(source) arbiter that every fetch worker calls before each API hit, plus queue backpressure.

Why it was proposed: the fear that once read-through (3b) is on AND the sweepers are uncapped, total API volume becomes unbounded and spiky — a traffic spike or a bot crawling 50k pages could trigger 50k refreshes in minutes and blow OMDb's hard 100k/day quota or hammer the shared 16 GB DB.

Honest evaluation — we do NOT need the heavyweight version. We already have most of the throttling for free:

The only genuinely unbounded path is read-through, because external traffic drives it. And even that is tamed by lightweight means, not a new subsystem:

  1. Ledger-gated enqueue — only enqueue if due? (a fresh page does nothing).
  2. Oban uniqueness — collapse duplicate enqueues.
  3. Queue-depth backpressure — skip enqueue if the queue is already deep.
  4. A per-source daily ceiling constant — checked against a cheap count of today's completed jobs. No new table, no arbiter.

Verdict / decision: ❌ Drop the heavyweight budget governor. Fold a minimal spend-guard (the 4 bullets above) into the read-through work itself, and only revisit a real budget ledger if we ever observe read-through actually approaching OMDb's 100k/day under real traffic. This removes a large speculative build from the roadmap.

5. Wireframes — the admin homeostasis dashboard

Surface the report() we already compute. New page /admin/homeostasis.

5a. Overview

┌─ Surface Area · Homeostasis ───────────────────────── grade: A- ─┐
│  Terminal 97.8%        Fresh 86%        1.17M movies · 681k people │
├───────────────────────────────────────────────────────────────────┤
│  SOURCE             TERMINAL  COVERAGE   FRESH    DUE     STATE     │
│  tmdb_details       ████ 100%   100%      86%    131,411  ✅ ⏳     │
│  tmdb_metrics       ████ 100%    80%¹      —        —     ✅        │
│  omdb               ████ 99.98%  97%      78%    141,176  ✅ ⏳     │
│  watch_providers    ██·· ~87%c   15%f     51%     35,999  ⏳        │
│  people_biography   ████ ~100%   59%¹    ~100%        4   ✅        │
│  people_profile     ████ 100%    100%      —        —     ✅        │
│  festival_link      ██▌· 67%      —        —        —     ⏳ #1104  │
│  imdb_id            ██·· 56%     ceiling   —        —     🔒        │
│  canonical_lists    ████ 100%    10/10     —        —     ✅        │
│  rt / metacritic    ···· derived (source ceiling)         🔒       │
├───────────────────────────────────────────────────────────────────┤
│  ¹ ceiling = source has no more to give   Legend ✅ terminal       │
│  ⏳ working · 🔒 source ceiling                                     │
└───────────────────────────────────────────────────────────────────┘

5b. Per-source drill-down (click a row)

┌─ tmdb_details ──────────────────────────────────────────────┐
│  Terminal 100% (1,165,578)     Fresh 86%     Due 131,411     │
│  TTL: age-tiered — recent 30d · classic 180d                 │
│                                                              │
│  Refresh activity (7d):  ▁▁▁▁▁▅▅   floor 5,000/day           │
│  Worker: TMDbMovieRefreshWorker · cron 04:00 UTC             │
│  Last run: 2026-06-11 04:00 → 4,981 ok · 19 err · 0 stuck    │
│                                                              │
│  [ Run now ]   [ Adjust cap ]    read-through: OFF (planned) │
└──────────────────────────────────────────────────────────────┘

5c. Refresh-spend monitor (the lightweight guard from §4)

┌─ Refresh spend · today ─────────────────────────────────────┐
│  TMDb   ████████░░░░░░░░  12,430 / 40,000 calls              │
│  OMDb   ██░░░░░░░░░░░░░░   3,100 / 90,000 calls              │
│  Queue depth:  tmdb 480 · omdb 0 · movie_availability 1,200  │
│  Read-through: OFF        Last governor trip: never          │
└──────────────────────────────────────────────────────────────┘

6. Roadmap — what's next, in order

  1. Soak Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106 → close on its benchmark. (in flight — reads tomorrow 04:00 UTC)
  2. Drain festival to ~95% via Festival person-link reimport rollout (waves) — execute the #873 repair #1104 waves. (autopilot)
  3. Make the imdb_id ceiling explicit — write source-absent markers for the ~510k TMDb-null movies so 56% reads as 100% terminal. (small, finishes Layer 1)
  4. Build read-through (Data freshness architecture: read-through (access-triggered) refresh + age-tiered TTLs to keep every source current #1010) + the minimal spend-guard (§4). ← the next real build / Layer 3 prize.
  5. Ship the /admin/homeostasis dashboard (§5) — surface report() for at-a-glance ongoing visibility.
  6. Heavyweight budget governor — dropped (§4); revisit only if evidence demands it.

7. Issue map — roped in, superseded, out of scope

Superseded & closed by this issue (their content is folded in above):

Active children (open, roped in):

Correctly OUT of scope (own trackers — this issue does not own them):

8. Definition of Done (the epic)

This epic is done when all four hold:

  1. Layer 1 — terminal: every fetchable source is 100% terminal or a documented source-ceiling with explicit source-absent markers — including the ~510k TMDb-null imdb_id rows marked, so the dashboard reads 100% terminal, not 56%.
  2. Layer 2 — freshness: the ledger covers every source (✅ already true) and is rendered by SurfaceArea.report() + /admin/homeostasis.
  3. Layer 3 — refresh: read-through is live (page view → debounced refresh of stale sources) behind the lightweight spend-guard, and the proactive floor sweeper (Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106) has soaked clean.
  4. Visibility: /admin/homeostasis (§5) shipped; the spend-guard is active and observable; a source that starts drifting re-opens as a child issue rather than silently rotting.

9. How to regenerate the §1 numbers

§1 is a point-in-time snapshot (2026-06-10). To refresh from prod (read-only):

# In `kamal console`, or locally via Cinegraph.ProdRpc.eval_json/1
# (must IO.puts(Jason.encode!(...)); convert tuples → lists for JSON).
Cinegraph.Health.SurfaceArea.report()
# → %{sources: [%{source:, terminal_pct:, eligible:, fetched:, source_absent:, needs_fetch:, ...}], ...}

# "Due now" per source = what the refresh engine will pick up:
import Ecto.Query; alias Cinegraph.Repo
now = DateTime.utc_now()
due = fn src ->
  from(r in "data_refreshes",
    where: r.source == ^src and r.stale_after < type(^now, :utc_datetime_usec) and r.status != "ineligible",
    select: count(r.id)) |> Repo.one()
end
Enum.map(~w(tmdb_details watch_providers omdb tmdb_person), &{&1, due.(&1)})

10. Acceptance criteria for the next slices

Per the review: #1108 is the operating map; each slice below ships as its own PR-sized child. These are the bolts to tighten before coding.

10a. Make the imdb_id ceiling explicit — small, finishes Layer 1, no API

  • Where markers live: data_refreshes rows (entity_type: "movie", source: "imdb_id", status: :empty) — reuse the existing source-absent convention; no new table.
  • How to compute: movies with a tmdb_id but imdb_id IS NULL after a TMDb external_ids fetch has run (i.e. we asked, TMDb returned null). Gate on "we actually checked" — never mark an unchecked movie absent.
  • Idempotency: upsert on (entity_type, entity_id, source); re-runs touch last_checked_at, never duplicate.
  • Tests: null-after-fetch → :empty; never-checked → untouched; re-run is a no-op; surface_area imdb_id terminal_pct → ~100%.
  • Done: dashboard imdb_id reads 100% terminal (fetched + source-absent), not 56%.

10b. Read-through refresh — Layer 3, the prize (do after #1106 soaks clean)

  • Trigger: movie/person show mount → check the ledger for that entity's sources; for each due? source, enqueue the unified worker (Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106's TMDbMovieRefreshWorker / the person equivalent).
  • Debounce: the ledger is the debounce — enqueue only if Freshness.due?; Oban unique (period ≥ 3600s, keys [:movie_id]) collapses dupes. A fresh page enqueues nothing.
  • Backpressure: skip enqueue if the target queue's available + scheduled > threshold (start 1,000).
  • Spend-guard (the lightweight §4 guard): per-source daily ceiling checked against a cheap count of today's completed jobs — start TMDb 40k / OMDb 90k; over cap → skip (log, don't fail). No new table.
  • Observability: per-source counts of enqueued vs skipped (reason: dup / cap / backpressure) → §5c spend monitor.
  • Flagging: off by default behind config; enable only after Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106's soak benchmark passes.
  • Tests: stale-on-view → 1 enqueue; fresh → 0; over-cap → skip; deep queue → skip; dup-in-window → collapsed.

10c. /admin/homeostasis dashboard (§5)

  • Route/auth: /admin/homeostasis, behind existing admin auth (same gate as /admin/health).
  • Data source: SurfaceArea.report() + ledger due-counts, cached (the report is heavy — reuse the 35-min health-cache + warmer pattern; never compute in the request path).
  • Refresh behavior: server-rendered from cache; a manual "recompute" action busts the cache.
  • Buttons: v1 is read-only — the "Run now / Adjust cap" controls in §5b are placeholders until wired to the maintenance modules; label them as such or omit in v1.
  • Done: the §5a grid renders live per-source terminal / coverage / fresh / due with the overall grade.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestepicLong-running operating-map / meta tracker

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions