[Twixes]

Problem

Users have been reporting that their Signals inbox goes empty shortly after enabling sources, and stays that way - showing "Inbox is warming up…". Others see reports briefly appear as "researching" then vanish with no retry. These reports fail, and one issue is we have been hiding these in the UI completely, which I'm fixing in PostHog/code#1950. But, there's the core issue of failures too.

Handing it over to Claude:

Investigation of production data over the last 7 days shows ~70% of signal report research runs are failing, with the failure rate climbing:

Day	Failed	Ready	Fail %
Apr 24	1,585	610	71%
Apr 25	3,853	1,037	77%
Apr 27	1,285	758	58%
Apr 28	2,403	883	69%
Apr 29	1,524	264	81%

(Source: signal_report_completed events, grouped by properties.result)

Session replay signals are hit hardest — 3,693 failed vs 448 ready (89% failure rate) over the last 48h, compared to ~50% for error_tracking reports. 68 unique orgs are affected by failures vs only 9 getting ready reports.

The dominant error across all failures is a single RuntimeError:

custom_prompt - drain_final_log: TaskRun reached terminal status=failed — no agent message

(Seen on every error log from temporal-worker-video-export with activity_type=run_agentic_report_activity, e.g. report 019ddb56-9b63-7e72-9430-368da7915f10 for team 222600, report 019ddb62-d0f6-7bd5-bd38-66f177caa64b for team 61853)

Three compounding issues cause this:

1. The SSE relay crashes on sandbox 404s instead of handling them

When a Modal sandbox dies, its /events URL returns HTTP 404. The relay's reconnect loop in relay_sandbox_events.py catches httpx.TransportError and httpx_sse.SSEError — but httpx.HTTPStatusError is a sibling class (inherits from HTTPError, not TransportError), so 404s escape the reconnect loop entirely.

This is confirmed by 2,333 HTTPStatusError occurrences in error tracking (issue c95785db) and production logs like:

HTTPStatusError: Client error '404 Not Found' for url
'https://a-ta-01kqdf3ydmj4f7bea0gs50stxf-ktj3ha6xovyx8j0dw8zdkkqiu.w.modal.host/events?_modal_connect_token=...'

(From temporal-worker-tasks-agent, activity_type=relay_sandbox_events, e.g. run a5256b26-4d64-45a8-be39-26af4805dbef at 2026-04-29T22:33)

Once the relay crashes, the workflow loses its heartbeat source and the 5-minute inactivity timer starts counting down silently.

2. Zero retries on sandbox activities

Per @sortafreel, keeping no retries for now to avoid a thundering herd, particularly till repo selection is stable via other changes.

Both select_repository_activity and run_agentic_report_activity use RetryPolicy(maximum_attempts=1). Any transient sandbox failure — Modal container eviction, image build timeout (ExecutionError: Timed out waiting for image to be created, 102 occurrences in error tracking issue c3950dfd), OOM — permanently kills the report with no retry.

3. Opaque error messages when sandbox runs fail

The _drain_final_log RuntimeError says only "no agent message" with no hint of the underlying cause. The TaskRun.error_message field (e.g. "Run timed out due to inactivity", "Follow-up delivery failed: ...") is available at the call site but not passed through, making triage harder than it needs to be.

Changes

Handle httpx.HTTPStatusError in the relay reconnect loop (relay_sandbox_events.py):

• 404 → mark error and return cleanly (sandbox is gone, reconnect won't help)
• 5xx → reconnect with backoff (transient server errors), same treatment as transport errors

Add retry policies to the two sandbox activities (summary.py):
Per @sortafreel, keeping no retries for now to avoid a thundering herd, particularly till repo selection is stable via other changes.

• select_repository_activity: 3 attempts, 10s initial backoff, 2x coefficient, 10min timeout per attempt (was 30min × 1 attempt)
• run_agentic_report_activity: 3 attempts, 30s initial backoff, 2x coefficient, 20min timeout per attempt (was 4h × 1 attempt)
• Non-retryable business errors (safety rejection, not actionable, requires human input) are handled by explicit conditionals that return early — they never reach the retry mechanism

Surface the TaskRun cause in _drain_final_log (custom_prompt_runner.py):

• Error now reads e.g. "TaskRun reached terminal status=failed (cause: Run timed out due to inactivity) — no agent message" instead of just "no agent message"

How did you test this code?

I'm an agent. I ran ruff check on all 3 changed files (all passed) and verified there are no import or type errors. The relay 404 fix was validated by tracing the httpx class hierarchy (HTTPStatusError → HTTPError; TransportError → RequestError → HTTPError), confirming the existing catch blocks don't cover it.

The retry policy changes are to Temporal workflow configuration and don't have dedicated unit tests — they're verified by production behavior.

Publish to changelog?

No.

🤖 Agent context

• Agent: Claude Code (Opus 4.6)
• How the investigation was done: Queried PostHog production analytics (signal_report_completed events grouped by properties.result and properties.source_products), searched temporal-worker-video-export and temporal-worker-tasks-agent logs via the logs MCP tool, read the full process-task workflow/relay/sandbox code path, and correlated error tracking issues (c95785db HTTPStatusError 2333 occ, c3950dfd ExecutionError 102 occ, 25a722fd RuntimeError 956 occ).
• Related context: The inactivity timeout was reduced from 30min to 5min in commit 447fa2b453c on Apr 22 — that's not reverted here (policy decision for the Tasks team), but the relay fix and retries should significantly reduce failures under the current timeout.
• Companion change: PostHog Code changes (default status filter including "failed", failed report UI) in the code repo — separate PR.

[greptile-apps]

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
products/tasks/backend/temporal/process_task/activities/relay_sandbox_events.py:327-335
**Non-404 4xx errors treated as retryable**

The comment says "Retryable server errors (502, 503, etc.)" but the fallthrough also catches other permanent 4xx codes — e.g. `401 Unauthorized`, `403 Forbidden`, `410 Gone` — and retries them up to `MAX_RECONNECT_ATTEMPTS` times. None of those will recover on reconnect, so it would be cleaner to only retry on 5xx (or at least also `return` early for other 4xx codes the way 404 does).

### Issue 2 of 2
products/tasks/backend/temporal/process_task/activities/relay_sandbox_events.py:328-334
**Missing `error` field in non-404 HTTP error log**

The `httpx.ReadTimeout` and `httpx.TransportError` handlers both include `error=str(e)` in their structured log entries, but the new 5xx reconnect path omits it. Including the response body or status message would help distinguish e.g. a `502 Bad Gateway` from a `503 Service Unavailable` when diagnosing relay failures.

```suggestion
                # Retryable server errors (502, 503, etc.)
                reconnect_count += 1
                logger.warning(
                    "relay_sandbox_events_http_error",
                    run_id=run_id,
                    status_code=e.response.status_code,
                    error=str(e),
                    reconnect_count=reconnect_count,
                )
                await asyncio.sleep(min(reconnect_count * 2, 10))
```

_{Reviews (1): Last reviewed commit: "fix(signals): reduce inbox report failur..." | Re-trigger Greptile}

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 3a6713e while workflow was testing 9adc86f.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

Query snapshots: Backend query snapshots updated

Changes: 1 snapshots (1 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-04-30 11:31 UTC	Run
prod-us	✅ Deployed	2026-04-30 11:52 UTC	Run
prod-eu	✅ Deployed	2026-04-30 11:56 UTC	Run