AppSignal → Starter tier (250K/mo): ignore Oban + bot-filter + sample anonymous traffic, then measure the drop

[holden]

Goal

Drive AppSignal request-billing on cinegraph/prod down to the Starter tier (250K req/mo ≈ 8,300 req/day) and hold it there for the rolling-30-day window required to downgrade.

Baseline + full audit: #1098. This issue is the implementation playbook — ship all levers in one pass, then watch the drop over several days and tune.

Where we are (measured 2026-06-09, 7-day window): ~18.4M req/mo · oban 54.5% · web 45.4% · graphql 0.06%. One open exception: DBConnection.ConnectionError × 40,916 (replica pool, #1018).
Target: 250K/mo = 8,300/day. That's a 73× cut — it requires sampling, not just filtering. This is a deliberate trade: keep all errors + all admin/api/auth + a representative sample of anonymous traffic; stop paying to record every bot page-view.

---

Why sampling is mandatory (the decisive data)

web broken down by action over 7 days — this is the entire ~279K/day web bill:

Action group	7-day	% of web
People detail pages (/people/:slug, /classic, /legacy, /.../movies, /directors/:id, /.../movies/acting)	1,303,418	66.8%
Movie detail pages (/movies/:slug + /legacy)	476,192	24.4%
Collaborations (/collaborations)	132,322	6.8%
Companies (/companies/:slug)	31,227	1.6%
Auth + admin + api + index + health + sitemap (the stuff worth keeping)	~7,000	0.4%

Two findings that set the strategy:

1. ~98% of the web bill is anonymous content-detail-page hits on a dead-flat 24/7 curve = crawler traffic indexing the slug space (1.15M movies + all people/directors). Real interactive traffic is a rounding error (~7K/wk for auth/admin/api combined).
2. Zero Elixir.*Live actions appear → LiveView connected (WebSocket) mounts are not separately billed. All web billing is plain HTTP route transactions. (This kills the LiveView-leak worry from AppSignal: cut request-billing (~18.4M/mo measured — Oban 54%, bot-heavy web, replica-pool error storm) #1098 Step 2 — no mount/3 guard needed.)

Starter's 8,300/day budget is less than the real + uncaught crawler residue on these routes even after a bot filter, so we must sample anonymous traffic. Errors are unaffected: CinegraphWeb.Router already does use Honeybadger.Plug (router.ex:3), so dropping a transaction from AppSignal billing never loses error visibility.

---

The plan — 4 levers, shipped together (Phase A), tuned over days (Phase B)

#	Lever	Mechanism	Expected effect
1	Ignore oban namespace + noise actions	config	−54.5% + health/sitemap/redirect noise → frees the whole budget for web
2	Drop bot/crawler User-Agents	endpoint plug, Tracer.ignore/0	web is ~all crawler; large cut
3	Sample remaining anonymous content	same plug, runtime-tunable sample_rate	brings web under 8,300/day
4	Fix the error sources anyway	code (#1018 replica pool, dead-slug 404s)	cuts both real volume + lets us sample less aggressively

Levers 1–3 ship in one PR ("do all of them in one, see how it drops" — @razrfly). Lever 4 can follow but reduces how hard #3 must sample.

---

Phase A — implementation (one PR)

A1. AppSignal config — ignore Oban + noise actions

config/config.exs currently has a bare block at lines 328-331. Extend it (action names verified against the router):

config :appsignal, :config,
  otp_app: :cinegraph,
  name: "cinegraph",
  env: config_env(),
  # Oban errors are already captured via Honeybadger.Plug (router.ex:3) — nothing lost. ~54.5% of volume.
  ignore_namespaces: ["oban"],
  ignore_actions: [
    "CinegraphWeb.HealthController#index",
    "CinegraphWeb.HealthController#database",
    "CinegraphWeb.HealthController#metrics",
    "CinegraphWeb.PageController#redirect_to_movies",
    "CinegraphWeb.PageController#manifesto",
    "CinegraphWeb.SitemapController#index",
    "CinegraphWeb.SitemapController#show"
  ]

runtime.exs:87 separately merges push_api_key: into the same config key — that's fine, just don't overwrite the keyword list.

A2. Sampler plug — bot filter + anonymous sampling

lib/cinegraph_web/plugs/appsignal_sampler.ex:

defmodule CinegraphWeb.Plugs.AppsignalSampler do
  @moduledoc """
  Keeps AppSignal request-billing under the Starter tier (250K/mo). See #1098.

  Runs right after Plug.Telemetry (endpoint.ex:42), before the router.
  Order of decisions:
    1. Operational surfaces (/admin, /api, /health) -> always tracked, never sampled.
    2. Bot / crawler / empty User-Agent              -> dropped (Tracer.ignore/0).
    3. Everything else (anonymous public traffic)    -> keep only `sample_rate` of it.

  Errors are unaffected: they're captured by Honeybadger (router.ex:3), so dropping a
  transaction from AppSignal billing never loses error visibility.
  """
  @behaviour Plug
  import Plug.Conn, only: [get_req_header: 2]

  @excluded_prefixes ["/admin", "/api", "/health"]

  @bot_pattern ~r/bot|crawler|spider|slurp|googlebot|bingbot|duckduckbot|yandexbot|baiduspider|applebot|petalbot|sogou|facebookexternalhit|twitterbot|linkedinbot|slackbot|discordbot|telegrambot|whatsapp|pinterest|redditbot|embedly|mastodon|ia_archiver|archive\.org_bot|gptbot|chatgpt-user|oai-searchbot|claudebot|anthropic-ai|perplexitybot|amazonbot|bytespider|dataforseo|semrushbot|ahrefsbot|mj12bot|dotbot|headlesschrome|phantomjs|python-requests|go-http-client|node-fetch|axios|okhttp|libwww-perl|curl|wget|scrapy|uptimerobot|pingdom|statuscake/i

  @impl Plug
  def init(opts), do: opts

  @impl Plug
  def call(conn, _opts) do
    cfg = Application.get_env(:cinegraph, __MODULE__, [])

    cond do
      not Keyword.get(cfg, :enabled, false) -> conn
      not filterable_path?(conn.request_path) -> conn                  # keep ops surfaces 100%
      bot_request?(conn) -> ignore(conn)                               # drop all bots
      :rand.uniform() <= Keyword.get(cfg, :sample_rate, 1.0) -> conn   # keep sampled fraction
      true -> ignore(conn)                                             # drop the rest
    end
  end

  defp ignore(conn) do
    Appsignal.Tracer.ignore()
    conn
  end

  defp filterable_path?(path),
    do: not Enum.any?(@excluded_prefixes, &(path == &1 or String.starts_with?(path, &1 <> "/")))

  defp bot_request?(conn) do
    case get_req_header(conn, "user-agent") do
      [ua | _] -> ua == "" or Regex.match?(@bot_pattern, ua)
      [] -> true
    end
  end
end

A3. Endpoint wiring

lib/cinegraph_web/endpoint.ex — immediately after Plug.Telemetry (line 42):

plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]
plug CinegraphWeb.Plugs.AppsignalSampler   # <-- add

A4. Config switches (off in dev/test, on in prod with a conservative initial rate)

# config/config.exs
config :cinegraph, CinegraphWeb.Plugs.AppsignalSampler, enabled: false, sample_rate: 1.0

# config/prod.exs  — start conservative; tune in Phase B
config :cinegraph, CinegraphWeb.Plugs.AppsignalSampler, enabled: true, sample_rate: 0.2

Why sample_rate is config, not hardcoded: we don't yet know the true bot-capture rate, so we ship at 0.2, measure the real post-filter number, then tune (one-line change, no code).

A5. Test

test/cinegraph_web/plugs/appsignal_sampler_test.exs — UA matrix (real browser passes, known bots dropped, empty/missing UA dropped), excluded-path pass-through (/admin, /api, /health never ignored), and sample_rate: 0.0 drops a non-bot anonymous request while 1.0 keeps it. Appsignal.Tracer.ignore/0 is a safe no-op when the agent isn't running, so the plug is testable without AppSignal active.

---

Phase B — measure & tune (the multi-day part)

Deploy A, then watch for 3–5 days (crawler traffic is steady, so a couple of full days is enough to read the new floor). Use the AppSignal Usage tab, or the MCP / dashboard query:

• transaction_duration count, tag namespace=* → confirm oban is gone and read the new web total.
• transaction_duration count, tags action=*, namespace=web → confirm the content routes collapsed and nothing unexpected is leaking.

Tuning rule: after the bot filter settles, read web_per_day. Set
sample_rate ≈ 7000 / (web_per_day_at_sample_rate_1.0).
Since A ships at 0.2, the un-sampled anon volume ≈ (observed_anon_per_day) / 0.2; solve for the rate that lands total ≈ 7,000/day (leaves headroom under 8,300). Adjust the one config line and redeploy.

Decision gates:

• After A: oban namespace shows ~0 billable. (Confirms Lever 1.)
• After A: web/day dropped sharply vs the ~279K/day baseline. (Confirms Lever 2.)
• After tune: total /day sits ≤ ~7,500 with headroom. (Confirms Lever 3.)
• Spot-check: a real logged-in session + an /admin page + a GraphQL call still appear as tracked transactions. (Confirms we didn't over-sample real signal.)

---

Phase C — fix the error sources (reduces required sampling + fixes real bugs)

• DBConnection.ConnectionError × 40,916 — replica pool exhaustion (Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018). Billed web transactions and a real reliability bug. Fixing it removes the Decision: Supabase Integration Approach - Client Library vs Direct API #1 pollutant at the source.
• Dead-slug 404 floods — unknown/deleted /people/:slug, /movies/:slug that raise instead of returning a clean 404. Return 404; removes billed exceptions and lets us raise sample_rate (better real-traffic visibility for the same bill).
• Bulk-close stale Oban performance tombstones once Lever 1 confirms they've stopped recurring.

---

Phase D — downgrade

AppSignal allows downgrade only after usage is below the lower plan's allowance for 30+ days (Usage docs).

• Once the Usage tab shows rolling-30d ≤ 250K, flip the plan to Starter.

---

Checklist

• A1 config: ignore_namespaces: ["oban"] + health/sitemap/redirect ignore_actions
• A2 AppsignalSampler plug (bot filter + anonymous sampling)
• A3 endpoint wiring after Plug.Telemetry
• A4 config switches (dev/test off; prod on @ sample_rate: 0.2)
• A5 plug test
• B deploy, measure 3–5 days, tune sample_rate to land ≤ ~7,500/day
• C fix replica DBConnection.ConnectionError (Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018) + dead-slug 404s; bulk-close stale incidents
• D after rolling-30d ≤ 250K, downgrade to Starter

Honest trade: at Starter we are sampling anonymous traffic — we keep 100% of errors (via Honeybadger), 100% of admin/api/auth, and a representative sample of anonymous page-views, while no longer paying to record every individual bot/anon hit. Given that ~98% of web volume is crawler traffic, the lost signal is negligible.

Refs: audit #1098 · replica pool #1018 · precedent razrfly/eventasaurus#5483.

[holden]

📊 Baseline snapshot — PRE-DEPLOY (locked for before/after)

Measured from AppSignal MCP (cinegraph/prod), transaction_duration COUNT, 7-day window Jun 2 → Jun 9 00:00 UTC (entirely before the Phase A deploy on Jun 9 afternoon). This is the reference we compare against.

Volume by namespace

Namespace	7-day	per-day	per-hour	~per-month
oban	2,341,641	334,520	13,938	~10.0M
web	1,950,156	278,594	11,608	~8.4M
graphql	2,506	358	15	~11K
TOTAL	4,294,303	613,472	25,561	~18.4M

• Starter target: 250K/mo = 8,300/day. Baseline is ~74× over.
• web is dead-flat 24/7 (Jun 8 hourly range 9,629–11,938) → machine/crawler traffic, not humans.

web by action (the ~279K/day, top routes)

Action	7-day	% web
/people/:slug	536,549	27.5%
/movies/:slug	383,450	19.7%
/people/:slug/classic	277,112	14.2%
/people/:slug/legacy	208,133	10.7%
/people/:slug/movies	143,443	7.4%
/collaborations	132,322	6.8%
/directors/:id	98,308	5.0%
/movies/:slug/legacy	92,742	4.8%
(people detail variants total)	1,303,418	66.8%
auth + admin + api + index + health + sitemap	~7,000	0.4%

Open exception (pre-deploy)

• 🔴 DBConnection.ConnectionError (Repo.Replica pool exhaustion) — 40,916 occurrences, namespace web. Phase C target (Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018).

Expected after Phase A (sample_rate = 1.0, pass-through)

• oban → ~0/day (ignore_namespaces). −54.5%.
• web → bot-filtered floor (unknown until measured — that's what the soak reveals).
• graphql unchanged (~358/day, kept).
• Projected total after Lever 1+2 ≈ web-floor + ~360/day. Sampling (Lever 3) then closes the gap to ≤8,300/day.

How to reproduce at +24h (same queries)

get_metrics_list · metric transaction_duration · type count:

1. namespace=* over the post-deploy 24h → confirm oban ≈ 0, read new web/day.
2. action=*, namespace=web over the post-deploy 24h → confirm content routes collapsed.
   Then set sample_rate ≈ 7000 / web_per_day_observed in config/prod.exs and redeploy.

[holden]

✅ Video Clerk shelved — ready to soak (2nd deploy)

Soak-readiness validation (local)

• mix compile --warnings-as-errors — clean
• mix format --check-formatted — clean (all 10 changed files)
• Changed/added tests — 22/22 pass (sampler 17, video_clerk_live 4, show_v2_video_clerk 1)
• Route table confirms gating: GET /admin/video-clerk exists; public /video-clerk is gone (404)
• Remaining suite failures are the pre-existing baseline set (ShowV2RouteTest fixtures, home_live disabled-redirect) — none introduced by this change

What we did

Theory correction: the homepage is currently disabled (router.ex:49, / → /movies), so the homepage CTA was not the live discovery vector. The real driver of the ~9K/day was the "Ask the Video Clerk" section on every crawled movie detail page (show_v2.ex), linking to /video-clerk?seed=…. Movie pages are the most-crawled surface (~383K/wk), so crawlers followed it.

Change	Effect
/video-clerk → /admin/video-clerk behind admin basic auth	conn-level gate: bots get 401 before any billable render; page + code preserved, reachable for dev
Movie-page "Ask the Video Clerk" link → /now-playing	kills the real public discovery vector
Homepage hero CTA / card / section links → /now-playing; footer link removed	correct-but-dormant until homepage is re-enabled
Internal self-nav (push_patch, share URLs) → /admin/video-clerk	page still works for admins

What we expect from this deploy

• /video-clerk drops out of public billing: ~9,000/day → ~0 (gated + no public links to discover it). Residual 404/401 from bots that already know the URL will be cheap and fade.
• web: ~23,300/day → ~14,300/day (−39%).
• Total: ~712K/mo → ~445K/mo. Still ~1.8× over Starter (250K), comfortably under the 3M/€49 tier.

Cumulative picture (where the levers have landed)

Stage	Total/day	~Total/mo	vs Starter (8,300/day)
Baseline (pre-Phase-A)	613,472	~18.4M	74× over
Phase A live (ignore oban + bot filter, sample_rate 1.0)	~23,700	~712K	2.9× over
+ this deploy (video-clerk gated)	~14,600	~445K	1.8× over
+ Phase B (sample_rate ≈ 0.5) — projected	~7,400	~224K	under ✅

How long to soak

24 hours. This is a fresh deploy, so we want a full daily cycle to (a) confirm /video-clerk flatlines to ~0, and (b) read the clean new web/day floor before computing the Phase B rate. An optional ~3h peek will confirm /video-clerk dropped, but don't set sample_rate off a partial cycle.

Next (after 24h)

1. Pull transaction_duration count: namespace=* and action=*, namespace=web.
2. Confirm /video-clerk ≈ 0 and read web/day.
3. Set sample_rate ≈ 7000 / observed_web_per_day (~0.5 expected) in config/prod.exs → Phase B → under Starter.