sre: exclude probe routes from 5xx gate, surface SRE report, stop asking questions in CI

.github/prompts/sre.md

@@ -4,14 +4,18 @@ You are an **SRE (Site Reliability Engineer) agent** for Telegram bots deployed
 
 **Bot identity is provided in your prompt.** You are invoked directly by `_bot-deploy.yml` (or by hand via `sre-manual.yml`) with explicit fields: `Bot`, `ArgoCD app`, `Container label`, `GHCR image`, `Commit`, and `Workflow run`, plus a deploy-failure issue number when one exists. Use these values for `APP_NAME`, `CONTAINER`, and `IMAGE_NAME` throughout this runbook — there is no issue body to read for bot identity anymore. Do not assume which bot failed — every bot in this monorepo opts into SRE coverage by default, including future ones.
 
-**A `Failure class` field is also provided** (`infra`/`app`/`unknown`), pre-computed by `verify-deploy.sh` itself: `infra` means it could not reach ArgoCD/Loki/Prometheus (network/DNS/VPN — see the 2026-07-23 `argo.internal` incident, issue #251); `app` means the control plane was reachable and reported the app itself unhealthy (bad readiness, error logs — never 5xx, which is structurally always 0, see Step 1); `unknown` means it was reachable but inconclusive (e.g. sync never completed for a reason that could be either side). **This strongly suggests, but does not prove, the severity** — `infra` is usually P3/transient. Before opening any escalation issue for an `infra`-classed failure: run the Step 1 health check yourself and confirm ArgoCD actually converged (Synced + Healthy) and there's no active Loki `level="Error"` burst. If it converged fine, close as transient with a short note (VPN/connectivity blip, deploy itself succeeded) instead of investigating further — do not treat `infra` as a free pass to skip verification, just as a strong prior that saves you from over-investigating a CI runner network hiccup.
+**A `Failure class` field is also provided** (`infra`/`app`/`unknown`), pre-computed by `verify-deploy.sh` itself: `infra` means it could not reach ArgoCD/Loki/Prometheus (network/DNS/VPN — see the 2026-07-23 `argo.internal` incident, issue #251); `app` means the control plane was reachable and reported the app itself unhealthy (bad readiness, error logs — 5xx is never a real webhook signal but CAN be a phantom rate from frozen counters, see Step 1); `unknown` means it was reachable but inconclusive (e.g. sync never completed for a reason that could be either side). **This strongly suggests, but does not prove, the severity** — `infra` is usually P3/transient. Before opening any escalation issue for an `infra`-classed failure: run the Step 1 health check yourself and confirm ArgoCD actually converged (Synced + Healthy) and there's no active Loki `level="Error"` burst. If it converged fine, close as transient with a short note (VPN/connectivity blip, deploy itself succeeded) instead of investigating further — do not treat `infra` as a free pass to skip verification, just as a strong prior that saves you from over-investigating a CI runner network hiccup.
 
 If a deploy-failure issue number was provided, that issue is still the incident record — comment your findings on it and close it once resolved (Step 7). If none was provided (e.g. a `sre-manual.yml` dispatch with no related issue), skip the issue comment/close steps and report your findings in the workflow run output only.
 
 ## Your outputs
 
 Your deliverables are **issue comments** with structured incident analysis, **rollback actions** when production is down, **one-liner code mitigations** for simple bugs, and **escalation issues** for complex bugs requiring human attention.
 
+## Non-interactive run
+
+This run is unattended CI — nobody is watching to answer a question. Never end your report with a question, an offer, or a menu of follow-up options to choose from. Every recommendation you make must be either executed now (Path A one-liner, Step 6) or filed as an issue for a human (Path B, Step 6) — there is no third "ask and wait" option. Your final message must be the finished incident report itself (the Step 7 template), written so it reads correctly as a standalone artifact with no further input from anyone.
+
 ## Prerequisites
 
 - VPN is pre-established by the workflow (WireGuard to `*.internal` hosts)
@@ -24,7 +28,7 @@ Your deliverables are **issue comments** with structured incident analysis, **ro
 
 You already have the workflow run link and commit SHA from your prompt. Determine severity.
 
-**The central fact for every bot in this fleet: pod health and HTTP status code are not usable "is it broken" signals.** A runtime failure in update handling does not crash the process — `/healthz` keeps returning OK and readiness keeps passing. The `/bot` webhook always returns HTTP 200 to Telegram regardless of internal exceptions (deliberately, to avoid Telegram retry storms), so the 5xx rate (`sum(rate(http_server_request_duration_seconds_count{status_code=~"5.."}))`) is **structurally always zero, for every bot, forever** — it is dead logic, not evidence of anything. A bot can be completely non-functional — throwing on every update, replying to nobody — while ArgoCD reports it `Healthy` and every HTTP response is a 200. **Pod health is therefore a necessary-but-not-sufficient signal, never proof of a working bot.** The real "is it actually broken" signal is Loki `level="Error"` on the update-handling path (`level` is a real indexed Loki label, values `Information`/`Warning`/`Error`), plus each bot's own business metrics (Step 1c).
+**The central fact for every bot in this fleet: pod health and HTTP status code are not usable "is it broken" signals.** A runtime failure in update handling does not crash the process — `/healthz` keeps returning OK and readiness keeps passing. The `/bot` webhook always returns HTTP 200 to Telegram regardless of internal exceptions (deliberately, to avoid Telegram retry storms), so a genuine 5xx from that route never happens — but the aggregate 5xx rate is **not** "structurally always zero": `/ready` (the DB-ping readiness check, `src/BotInfra/Readiness.fs`) returns 503 by design whenever the DB ping fails, and with 2 replicas per bot and no per-pod label on the Prometheus scrape, `rate()`/`increase()` on that job's counters can report a constant phantom non-zero value once the two pods' raw values diverge and alternate in one series (each alternation looks like a counter reset). **2026-09-08 worked example:** the shared Postgres restarted at 23:58:49Z, `/ready` 503'd on both coupon-bot and vahter-bot for ~2 minutes, the raw counters then froze (29 coupon-pod, 30/29 vahter-pods) — yet `rate()` kept reporting a constant 0.5557/s (coupon) and 0.4466/s (vahter) for the next 9 hours, and `verify-deploy.sh` failed a real deploy on `5xx_rate=0.63`. **Diagnostic:** query the raw counter series and `changes(...[5m])`/`resets(...[5m])` over the window — a non-zero rate on a counter whose raw value never moves is this collision artifact, not traffic. `verify-deploy.sh` now excludes `/ready`/`/healthz` from its 5xx gate, but the same phantom-rate mechanism can surface on any other frozen counter on a 2-replica job — never assume 5xx is dead logic. A bot can be completely non-functional — throwing on every update, replying to nobody — while ArgoCD reports it `Healthy` and every HTTP response is a 200. **Pod health is therefore a necessary-but-not-sufficient signal, never proof of a working bot.** The real "is it actually broken" signal is Loki `level="Error"` on the update-handling path (`level` is a real indexed Loki label, values `Information`/`Warning`/`Error`), plus each bot's own business metrics (Step 1c).
 
 | Severity | Criteria | Response |
 |----------|----------|----------|
@@ -114,7 +118,7 @@ Use `gh` CLI to read the failed workflow run logs. The `verify-deploy.sh` script
 | Phase 2 | `FAILED: Pod is not healthy after` | Pod readiness probes failed beyond the 3-minute grace period |
 | Phase 3 (Loki) | `FAILED: Error logs detected` | Application is producing Error/Fatal log entries |
 
-Note: `verify-deploy.sh` also emits a `FAILED: 5xx error rate is non-zero` marker for a Prometheus 5xx check, but that check is dead logic — the `/bot` webhook always returns HTTP 200 regardless of internal exceptions, so the underlying rate is structurally always 0 and this marker cannot occur in practice. Do not use it as a signal; Loki `level="Error"` (above) is the real one.
+Note: `verify-deploy.sh` also emits a `FAILED: 5xx error rate is non-zero` marker for a Prometheus 5xx check; the `/bot` webhook always returns HTTP 200 regardless of internal exceptions, but the marker CAN still fire from a phantom rate on frozen probe-route counters — never treat 5xx rate as proof of anything without the Step 1 raw-counter check. Do not use it as a signal; Loki `level="Error"` (above) is the real one.
 
 ### Step 3: Query Observability Services
 
@@ -191,6 +195,8 @@ curl -s -G http://loki.internal/loki/api/v1/query_range \
 | **Code bug** | Application crash, unhandled exception, regression from recent commit | Escalate to coding agent (Step 6) |
 | **Configuration** | Missing env var, wrong secret, migration failure | Document in issue, label as `infra` |
 
+**When `verify-deploy.sh` failed on a metric threshold, "transient race" is not an acceptable root cause on its own.** The report must state the metric's actual value, which series/route produced it, and since when — pull the raw series (not just the aggregate) and check whether it's actually moving before naming a cause.
+
 ### Step 5: Rollback (if production is impacted)
 
 **Only rollback for genuine P1 incidents.** If old replicas are still serving (P2), skip rollback. P1 is now reachable through the update-handling error path (Step 1a), not only "no healthy pods" — so this section will fire more often than it used to. Follow the disable → rollback → re-enable order below exactly; don't skip a step because pods "look" fine.
@@ -389,4 +395,4 @@ gh issue close "$DEPLOY_FAILURE_ISSUE_NUMBER"
 | Waiting reason | `kube_pod_container_status_waiting_reason{container="CONTAINER"}` |
 | Memory usage | `container_memory_working_set_bytes{container="CONTAINER"}` |
 
-None of the above are "is it doing its job" signals by themselves — a pod can post all-green values on every row here while the update path throws on every message (drill 3). **Never use the 5xx HTTP rate as a health metric — it is structurally always 0** (Step 1). The primary "is it actually broken" signal is the Loki query `sum(count_over_time({container="CONTAINER"} | json | level="Error"[15m]))` (Step 1a), backed by the per-bot business metrics in Step 1c.
+None of the above are "is it doing its job" signals by themselves — a pod can post all-green values on every row here while the update path throws on every message (drill 3). **Never use the 5xx HTTP rate as a health metric without the Step 1 raw-counter check — it is never a real webhook signal but CAN be a phantom rate from frozen counters** (Step 1). The primary "is it actually broken" signal is the Loki query `sum(count_over_time({container="CONTAINER"} | json | level="Error"[15m]))` (Step 1a), backed by the per-bot business metrics in Step 1c.

.github/workflows/_sre-agent.yml

@@ -87,6 +87,7 @@ jobs:
         } >> "$GITHUB_ENV"
 
     - name: Run SRE agent
+      id: run-agent
       uses: openai/codex-action@v1
       with:
         openai-api-key: ${{ secrets.AZURE_OPENAI_API_KEY }}
@@ -96,6 +97,7 @@ jobs:
         safety-strategy: drop-sudo
         effort: high
         codex-home: ${{ github.workspace }}/.github/codex
+        output-file: /tmp/codex-output/sre-report.md
         prompt: |
           <instructions>
           ${{ env.AGENT_PROMPT }}
@@ -120,6 +122,27 @@ jobs:
         GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }}
 
+    - name: Upload SRE agent report
+      # codex-action's output-file is the only transcript it exposes.
+      if: always()
+      uses: actions/upload-artifact@v4
+      with:
+        name: sre-report-${{ inputs.bot }}
+        path: /tmp/codex-output/
+        if-no-files-found: ignore
+        retention-days: 14
+
+    - name: Post SRE agent report to job summary
+      # Makes the report visible on the run's Summary page without opening logs.
+      if: always()
+      run: |
+        echo "## SRE agent report — ${{ inputs.bot }}" >> "$GITHUB_STEP_SUMMARY"
+        if [ -f /tmp/codex-output/sre-report.md ]; then
+          cat /tmp/codex-output/sre-report.md >> "$GITHUB_STEP_SUMMARY"
+        else
+          echo "no report file produced (agent step failed or was cancelled)" >> "$GITHUB_STEP_SUMMARY"
+        fi
+
     - name: Re-enable ArgoCD auto-sync (safety net — a killed run must not leave it off)
       # Idempotent: a no-op if the agent never touched auto-sync. Runs even if
       # earlier steps failed/were cancelled, so a killed investigation can't

scripts/verify-deploy.sh

@@ -352,9 +352,10 @@ RESTARTS=$(curl -sf -G "${PROMETHEUS_URL}/api/v1/query" \
     2>/dev/null || echo '{"data":{"result":[]}}')
 RESTART_COUNT=$(echo "$RESTARTS" | jq -r '[.data.result[].value[1] | tonumber] | add // 0')
 
-# Check 5xx rate
+# Check 5xx rate. Readiness (/ready, /healthz) 503s are excluded: those
+# counters are expected to be non-zero after any DB blip; Phase 2 covers readiness.
 ERRORS_5XX=$(curl -sf -G "${PROMETHEUS_URL}/api/v1/query" \
-    --data-urlencode "query=sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~\"5..\",job=\"${CONTAINER}\"}[5m]))" \
+    --data-urlencode "query=sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~\"5..\",http_route!~\"/ready|/healthz\",job=\"${CONTAINER}\"}[5m]))" \
     2>/dev/null || echo '{"data":{"result":[]}}')
 ERROR_5XX_RATE=$(echo "$ERRORS_5XX" | jq -r '[.data.result[].value[1] | tonumber] | add // 0')
 
@@ -363,9 +364,9 @@ log "  Prometheus: restarts=${RESTART_COUNT}, 5xx_rate=${ERROR_5XX_RATE}"
 # Note: restart count is cumulative, so we only fail if it's unexpectedly high.
 # For a fresh deployment, a single restart might be acceptable.
 if [ "$(echo "$ERROR_5XX_RATE > 0" | bc -l 2>/dev/null || echo 0)" = "1" ]; then
-    log "FAILED: 5xx error rate is non-zero: ${ERROR_5XX_RATE}"
+    log "FAILED: 5xx error rate is non-zero: ${ERROR_5XX_RATE} (probe routes excluded)"
     summary "### ❌ Phase 3: Log & Metrics Check FAILED"
-    summary "- **5xx Error Rate:** ${ERROR_5XX_RATE} (expected: 0)"
+    summary "- **5xx Error Rate:** ${ERROR_5XX_RATE} (expected: 0, probe routes excluded)"
     summary "- **Container Restarts:** ${RESTART_COUNT}"
     emit_failure_class "app"
     exit 1