AppSignal: cut request-billing (~18.4M/mo measured — Oban 54%, bot-heavy web, replica-pool error storm)

[holden]

Goal

Get AppSignal request-billing under control on Cinegraph. AppSignal bills per monitored request (each web request and each background job = 1 request). This issue ports the proven playbook from razrfly/eventasaurus#5483 (Wombie), but the baseline measurement below shows Cinegraph is in a completely different magnitude of trouble than Wombie was, so the tier expectations are reset accordingly.

Tier reference (request-based billing): Free 50K/mo (€0) · Starter 250K/mo (€18.25/mo) · 3M/mo (€49/mo).

---

Step 0 — Baseline (MEASURED from AppSignal MCP, cinegraph/prod, 2026-06-09)

Pulled transaction_duration COUNT by namespace over 7 days (Jun 2 → Jun 9):

Namespace	7-day requests	Share	~Monthly run-rate
oban (background jobs)	2,341,641	54.5%	~10.0M/mo
web	1,950,156	45.4%	~8.4M/mo
graphql	2,506	0.06%	~11K/mo
TOTAL	4,294,303	100%	~18.4M/mo

Reality check on tiers:

• Current run-rate ~18.4M/month = 368× the Free tier (50K), 73× Starter (250K), and ~6× over the €49/3M tier.
• 🚫 Free (50K) is not achievable for this app — drop that expectation. It would require a 99.7% cut.
• 🎯 Realistic landing spots: 3M tier (€49) is reachable with Steps 1–2. Starter (250K) is only reachable if we also sample/limit anonymous web traffic (Step 2b), not just filter bots.

Key signal — web traffic is machine traffic, not humans:
web throughput is dead-flat ~9,600–11,900 req/hour, 24/7, with no day/night curve (hourly timeseries Jun 8). Real users produce a diurnal curve; a flat round-the-clock line = bots + health pings + reconnect/machine traffic. This strongly confirms the bot-filter thesis and means most of the ~8.4M/mo web volume is filterable.

Exception incidents (signal-quality audit — MEASURED)

Only one open exception incident, and it's huge:

• 🔴 DBConnection.ConnectionError — 40,916 occurrences, namespace web, action "Unknown action", last seen Jun 09 13:28.
  [Cinegraph.Repo.Replica] connection not available and request was dropped from queue — i.e. read-replica pool exhaustion (the shared-Postgres-no-pooler infra, see Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018). This is both a billing pollutant (40K+ billed failed transactions) and genuine reliability signal. It is NOT bot noise — do not just ignore it. Fix the root cause (pool size / pooler / query load), which removes the noise at source.

All other recent incidents are performance incidents (sampled), not exceptions. Notable Oban offenders by occurrence (these vanish from billing entirely after Step 1):

• ConnectionMonitorWorker#perform — 2,299 occ (cron every 5 min)
• TMDbDetailsWorker#perform — 2,131 occ
• MovieAvailabilityRefreshWorker#perform — 442 occ
• DataRepairWorker#perform — 528 occ
• NominationPersonResolver#perform — 188 occ, PersonQualityScoreWorker — 194, CollaborationWorker — 116 …

---

Current code state (audit — 2026-06-09)

Stack: Phoenix + AppSignal (appsignal_phoenix ~> 2.0) + Oban. AppSignal active in prod via APPSIGNAL_PUSH_API_KEY (config/runtime.exs:87).

• ❌ AppSignal config is bare — config/config.exs:328-331 is just otp_app/name/env. No ignore_namespaces, no ignore_actions. All 3 namespaces (web, oban, graphql) fully billed.
• ❌ No bot-filter plug. Plug.Telemetry is at lib/cinegraph_web/endpoint.ex:42; nothing filters crawler traffic after it.
• ✅ Oban errors captured elsewhere — use Honeybadger.Plug in router.ex:3. So ignoring the oban namespace in AppSignal loses zero error visibility.
• ✅ Verified action names against the router (see Step 1).

---

Step 1 — Ignore oban namespace + noise actions (config only — biggest single win, ~54.5%)

Removes ~10.0M/mo immediately. In config/config.exs, extend the existing block (currently lines 328-331):

config :appsignal, :config,
  otp_app: :cinegraph,
  name: "cinegraph",
  env: config_env(),
  # All Oban job tracking — errors already captured via Honeybadger.Plug (router.ex:3),
  # so nothing is lost by removing them from AppSignal billing. ~54.5% of all volume.
  ignore_namespaces: ["oban"],
  ignore_actions: [
    # Fly health pings — no diagnostic value (verified routes: router.ex:41-43)
    "CinegraphWeb.HealthController#index",
    "CinegraphWeb.HealthController#database",
    "CinegraphWeb.HealthController#metrics",
    # Pure redirect (root "/" -> /movies) + static SEO files
    "CinegraphWeb.PageController#redirect_to_movies",
    "CinegraphWeb.PageController#manifesto",
    "CinegraphWeb.SitemapController#index",
    "CinegraphWeb.SitemapController#show"
  ]

⚠️ ignore_actions drops BOTH performance traces AND exceptions for those actions — fine for health/sitemap/redirect; do NOT add real content routes here (use Step 2).
⚠️ Note runtime.exs:87 also sets config :appsignal, :config, push_api_key: — that's a separate merge key and is fine; just don't accidentally overwrite the keyword list.

• Add ignore_namespaces + ignore_actions
• Deploy, re-measure namespace split — expect web (~8.4M/mo) to be ~100% of remaining billable volume

Projected after Step 1: ~18.4M → ~8.4M/mo. Still ~3× over the 3M tier — Step 2 is mandatory, not optional.

---

Step 2 — Bot-filter plug (port from Wombie)

Drop crawler/unfurl-bot transactions while keeping 100% of real-user transactions. AppSignal Phoenix is telemetry-based: the span is created on [:phoenix, :endpoint, :start] (emitted by Plug.Telemetry), and Appsignal.Tracer.ignore/0 deletes the current request's spans + blocks later ones, so the transaction is never billed. Mount right after Plug.Telemetry (endpoint.ex:42).

defmodule CinegraphWeb.Plugs.AppsignalBotFilter do
  @moduledoc "Drop AppSignal transactions for crawler/bot UAs to stay under billing quota. See razrfly/eventasaurus#5483."
  import Plug.Conn, only: [get_req_header: 2]
  @behaviour Plug

  @excluded_prefixes ["/admin", "/api", "/health"]

  @bot_pattern ~r/bot|crawler|spider|slurp|googlebot|bingbot|duckduckbot|yandexbot|baiduspider|applebot|petalbot|sogou|facebookexternalhit|twitterbot|linkedinbot|slackbot|discordbot|telegrambot|whatsapp|pinterest|redditbot|embedly|mastodon|ia_archiver|archive\.org_bot|gptbot|chatgpt-user|oai-searchbot|claudebot|anthropic-ai|perplexitybot|amazonbot|bytespider|dataforseo|semrushbot|ahrefsbot|mj12bot|dotbot|headlesschrome|phantomjs|python-requests|go-http-client|node-fetch|axios|okhttp|libwww-perl|curl|wget|scrapy|uptimerobot|pingdom|statuscake/i

  @impl Plug
  def init(opts), do: opts

  @impl Plug
  def call(conn, _opts) do
    if filterable_path?(conn.request_path) and bot_request?(conn), do: Appsignal.Tracer.ignore()
    conn
  end

  defp filterable_path?(path), do: not Enum.any?(@excluded_prefixes, &(path == &1 or String.starts_with?(path, &1 <> "/")))

  defp bot_request?(conn) do
    case get_req_header(conn, "user-agent") do
      [ua | _] -> ua == "" or Regex.match?(@bot_pattern, ua)
      [] -> true
    end
  end
end

# lib/cinegraph_web/endpoint.ex — right after Plug.Telemetry (line 42)
plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]
plug CinegraphWeb.Plugs.AppsignalBotFilter   # <-- add

Tracer.ignore/0 is a safe no-op in dev/test. Add a unit test (UA matrix + excluded-path pass-through).

• Add plug + endpoint wiring + test
• Deploy, soak 24–48h, re-measure

⚠️ LiveView caveat (Cinegraph is LiveView-heavy — 53 mount/3)

The HTTP plug catches the disconnected (initial HTTP) mount — the billed page-load request. A LiveView's connected (WebSocket) mount may be a separate billed transaction the HTTP plug can't reach. During the soak, verify whether live/channel transactions are billed separately; if bots drive them, add a mount/3-level guard (ignore when connect_info UA is a bot).

Projected after Step 1+2 (assuming Wombie-like ~74% bot share of web): ~8.4M → ~2.2M/mo. → under the 3M tier (€49), still ~9× over Starter.

---

Step 2b — Reach Starter (250K) [stretch — only if €49 → €18.25 is worth it]

Getting from ~2.2M/mo to ≤250K (≈8.3K/day) cannot be done by bot-filtering alone — the flat 24/7 web profile means a large floor of non-bot machine/anonymous traffic remains. Options, in order of preference:

• Fix the DBConnection.ConnectionError storm (40,916 occ) first — see Step 3. If a chunk of web volume is retry/reconnect amplification from pool exhaustion, fixing it cuts both billing and the error.
• Add sampling for anonymous (non-logged-in, non-admin) web transactions, keeping 100% of authenticated + admin + API. AppSignal supports Appsignal.Tracer.ignore/0 probabilistically in the same plug.
• Re-measure. Decide whether Starter is realistically holdable or whether the 3M/€49 tier is the right home for an app at this traffic level.

---

Step 3 — Signal-quality audit (don't pay for bullshit)

• Fix DBConnection.ConnectionError (40,916 occ) at the source — replica pool exhaustion (Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018). Returns clean behavior instead of dropped-from-queue raises; removes the Decision: Supabase Integration Approach - Client Library vs Direct API #1 billing pollutant AND a real reliability problem. This is the highest-value signal item.
• Confirm Oban perf incidents stop appearing after Step 1 (they should drop out of billing entirely).
• Keep genuine signal (real DB errors, real bugs); bulk-close stale tombstones.

---

Step 4 — Downgrade (gated on 30-day rolling window)

AppSignal allows a downgrade only after usage is below the lower plan's allowance for 30+ days (Usage docs).

• Once rolling-30d ≤ target tier, downgrade.

---

Checklist summary

• Step 0 — baseline measured: ~18.4M/mo (oban 54.5% / web 45.4% / graphql ~0%); 1 open exception = 40,916× DBConnection.ConnectionError
• Step 1 — ignore_namespaces: ["oban"] + health/sitemap/redirect ignore_actions (≈ −54.5%, → ~8.4M/mo)
• Step 2 — AppsignalBotFilter plug + test; soak 24–48h (→ ~2.2M/mo, under 3M tier)
• Step 2 LiveView check — verify connected-mount transactions aren't a separate billed leak
• Step 2b — (stretch) sample anonymous web traffic to reach Starter 250K
• Step 3 — fix the DBConnection.ConnectionError replica-pool storm (Cinegraph → PgBouncer (16GB tuning done; sibling apps deferred to separate follow-ups) #1018); bulk-close stale incidents
• Step 4 — wait out 30-day window, downgrade

Bottom line: Free (50K) is off the table at this scale. Step 1 alone halves the bill; Step 1+2 should land us under the €49/3M tier; Starter (250K) is only reachable with sampling + fixing the replica-pool error storm.

Reference: razrfly/eventasaurus#5483 (Wombie) — same stack, 550K→143K/mo. Cinegraph starts ~33× higher in volume, so expectations are scaled accordingly.

[holden]

Implementation playbook split out into #1100 — AppSignal → Starter tier: ship Lever 1 (ignore Oban) + Lever 2 (bot filter) + Lever 3 (sample anonymous traffic) in one PR, then measure the drop over 3–5 days and tune sample_rate. This issue (#1098) stays as the measured audit/baseline.