[Impl] Finish Layer 1: complete TMDb discovery stubs via the unified refresh + retire the no-op RepairImdbIds (child of #1108)

[holden]

Child of #1108 — finishes Layer 1. Completes the TMDb discovery stubs (movie rows that never got a full /movie fetch) by reusing the proven #1106 unified-refresh worker, retires the no-op RepairImdbIds path, and closes the imdb_id arm of #760. After this, every movie has a tmdb_details ledger row and imdb_id reaches ~100% terminal.

---

1. Baseline — the residual after #1109 (prod, 2026-06-10)

#1109 marked 349,324 checked-but-null movies source-absent → imdb_id terminal 56% → 86.18%. What's left:

Set	Count	Why
imdb_id needs_fetch (null, no :empty marker)	161,129	never had a full /movie fetch → imdb_id was never actually checked
movies with no tmdb_details ledger row	~250,678	superset: discovery stubs (incl. the 161k) + ~89k fetched-pre-backfill

These are discovery stubs: a movie row created from a bulk/discovery payload (which does not carry imdb_id) that never received a full /movie/{id} fetch. They are genuine backlog, not source-absent.

Why nothing currently fixes them:

• RepairImdbIds → TMDbDetailsWorker is a no-op — process_tmdb_movie/3 (tmdb_details_worker.ex:94) skips any movie that already exists. (Visible in prod: TMDbDetailsWorker shows ~5,194 cancelled.)
• Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106's unified refresh misses them — Freshness.due("tmdb_details") only selects movies that already have a ledger row; stubs have none.

This is exactly the dynamic #760 documents (sweepers that don't actually resolve the bar).

---

2. The fix — complete stubs via the unified refresh worker

TMDbMovieRefreshWorker.refresh/1 works on any existing movie (verified tmdb_movie_refresh_worker.ex:37 — Repo.get + full get_movie_for_refresh + refresh_movie_from_tmdb). Unlike TMDbDetailsWorker, it refreshes rather than skips. So a sustained capped sweeper that enqueues it for movies lacking a tmdb_details ledger row will, per movie:

• do a real /movie fetch → imdb_id resolves: present → :ok (fetched); null-after-real-fetch → :empty (source-absent) via the [Impl] Make the imdb_id ceiling explicit — mark source-absent in the freshness ledger so Layer 1 reads 100% terminal (child of #1108) #1109 inline touch,
• bootstrap a tmdb_details ledger row (+ metrics, credits, watch, imdb_id) → the movie enters the normal age-tiered refresh cycle,
• — which lets us retire the no-op RepairImdbIds/ImdbIdRepairSweeper entirely.

Watch-region fan-out is cheap here: long-tail stubs almost always have no providers (watch_providers → :empty), so store_tmdb_watch_providers writes little.

---

3. Changes (file-by-file)

3a. New maintenance module — Cinegraph.Maintenance.CompleteTmdbStubs

lib/cinegraph/maintenance/complete_tmdb_stubs.ex (mirror maintenance/refresh_biographies.ex):

• Eligible set — movies with no tmdb_details ledger row, canonical-first:

  from m in "movies",
    where: fragment(
      "NOT EXISTS (SELECT 1 FROM data_refreshes dr WHERE dr.entity_type='movie' AND dr.entity_id=? AND dr.source='tmdb_details')",
      m.id),
    order_by: [desc: fragment("? != '{}'::jsonb", m.canonical_sources), desc: m.id],
    select: m.id

• Enqueues TMDbMovieRefreshWorker.new(%{movie_id: id}) in chunked Oban.insert_all (the worker is uniqueness-keyed on :movie_id/1h, so re-runs collapse).
• run(opts) :: {:ok, %{found, enqueued, failed, dry_run}}; :limit + :dry_run.

3b. New sweeper — Cinegraph.Workers.StubCompletionSweeper

lib/cinegraph/workers/stub_completion_sweeper.ex (mirror BiographyRefreshSweeper): queue: :maintenance, max_attempts: 1, priority: 3, @per_run_limit 5_000, perform → CompleteTmdbStubs.run(limit:). Sustained (drains the ~250k and keeps pace with new discovery stubs — the #760 fix).

3c. Cron — config/config.exs

Add {"0 7 * * *", Cinegraph.Workers.StubCompletionSweeper} in the slot vacated by ImdbIdRepairSweeper (3e). Gating: reuses TMDbMovieRefreshWorker, so enable only once #1106's soak has proven that worker. (By the time this slice deploys, the soak — 2026-06-11 — will have passed; if implementing earlier, land the cron commented and flip it on post-soak.)

3d. Retire the no-op repair path

• Remove the {"0 7 * * *", ImdbIdRepairSweeper} cron from config/config.exs.
• In lib/cinegraph/admin/job_registry.ex: move :imdb_id_repair_sweeper from a scheduled entry to an on-demand entry (schedule: nil) — keeps it manually runnable, drops it from cron parity. Add the :stub_completion_sweeper scheduled entry (queue :maintenance, schedule "0 7 * * *"). Net scheduled count stays 27 (−1 +1) → parity test green.
• Add a @deprecated-style moduledoc note to RepairImdbIds/ImdbIdRepairSweeper pointing at CompleteTmdbStubs (full deletion optional — keep callable for now).

3e. (no surface_area change)

imdb_id_row/0 already counts :empty markers as terminal (#1109); as stubs complete, fetched/source_absent rise and needs_fetch falls automatically. tmdb_details_row/0 already keys on tmdb_data, so its terminal stays ~100% — but the ledger coverage (which drives refresh) is what this backfills.

---

4. Capping & DB load

• 5,000/day floor (matches the other sweepers); ~250k drains in ~50 days, then it idles except for new stubs. Can raise to 10k if the Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106 soak shows headroom.
• Each completion = 1 TMDb call + the usual refresh writes; watch fan-out is minimal for the (mostly provider-less) long tail. Monitor DB pool alongside the Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106 floor + availability sweeper.

---

5. Tests

• complete_tmdb_stubs_test.exs (new): movie with no tmdb_details ledger → enqueues one TMDbMovieRefreshWorker; movie with a tmdb_details ledger → excluded; dry_run enqueues nothing, found > 0; :limit caps; canonical-first ordering.
• job_registry_test.exs parity: green after the −imdb_id_repair / +stub_completion swap (count stays 27).
• (Retirement) keep/adjust repair_imdb_ids_test.exs if the module stays; remove if deleted.

6. Verification (prod, read-only)

# stubs remaining (no tmdb_details ledger)
import Ecto.Query; alias Cinegraph.Repo
Repo.one(from m in "movies",
  where: fragment("NOT EXISTS (SELECT 1 FROM data_refreshes dr WHERE dr.entity_type='movie' AND dr.entity_id=? AND dr.source='tmdb_details')", m.id),
  select: count(m.id))            # → ~250,678 before; trends to ~0
Cinegraph.Maintenance.CompleteTmdbStubs.run(dry_run: true)   # sizes the set
Cinegraph.Health.SurfaceArea.report().sources |> Enum.find(& &1.source == "imdb_id")
# imdb_id terminal_pct trends 86 → ~100; needs_fetch → ~0

7. Rollout & soak

1. Deploy (cron on only post-Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106-soak). 2. Sweeper drains 5k/day. 3. Soak/checkpoints (multi-week): tmdb_details ledger rows climb toward ~1.16M; imdb_id terminal 86% → ~100%; needs_fetch → ~0; TMDbDetailsWorker cancelled count stops growing (repair retired). 4. Watch DB pool/box memory vs the combined sweeper load.

8. Definition of Done

• Every movie has a tmdb_details ledger row (or :ineligible); the no-tmdb_details-ledger set → ~0.
• imdb_id terminal ~100% (residual resolved to fetched-or-source-absent).
• ImdbIdRepairSweeper retired (off cron); CompleteTmdbStubs sustained.
• Tests + parity green; prod verified. Layer 1 of META: Homeostasis — drive surface area to 100% terminal, a freshness layer that works, and demand-driven refresh (supersedes #1090) #1108 is complete.

9. Relation

• Parent: META: Homeostasis — drive surface area to 100% terminal, a freshness layer that works, and demand-driven refresh (supersedes #1090) #1108 — finishes Layer 1.
• Closes the imdb_id arm of Sweeper caps don't outpace catalog growth — OMDb/IMDb-id/profile_path drift bars stay flat or grow #760 (the sustained sweeper + real-fetch completion is the resolution; recommend scoping Sweeper caps don't outpace catalog growth — OMDb/IMDb-id/profile_path drift bars stay flat or grow #760 down to its remaining OMDb/profile_path arms or closing if subsumed).
• Reuses Unified per-movie TMDb refresh (one call, all sources) + near-term availability throughput bump #1106's TMDbMovieRefreshWorker — gated on its soak (2026-06-11).
• Out of scope: read-through (META: Homeostasis — drive surface area to 100% terminal, a freshness layer that works, and demand-driven refresh (supersedes #1090) #1108 §10b), the /admin/homeostasis dashboard (§10c).