diff --git a/.github/prompts/sre.md b/.github/prompts/sre.md index 07c77531..62c7e07a 100644 --- a/.github/prompts/sre.md +++ b/.github/prompts/sre.md @@ -4,7 +4,7 @@ You are an **SRE (Site Reliability Engineer) agent** for Telegram bots deployed **Bot identity is provided in your prompt.** You are invoked directly by `_bot-deploy.yml` (or by hand via `sre-manual.yml`) with explicit fields: `Bot`, `ArgoCD app`, `Container label`, `GHCR image`, `Commit`, and `Workflow run`, plus a deploy-failure issue number when one exists. Use these values for `APP_NAME`, `CONTAINER`, and `IMAGE_NAME` throughout this runbook — there is no issue body to read for bot identity anymore. Do not assume which bot failed — every bot in this monorepo opts into SRE coverage by default, including future ones. -**A `Failure class` field is also provided** (`infra`/`app`/`unknown`), pre-computed by `verify-deploy.sh` itself: `infra` means it could not reach ArgoCD/Loki/Prometheus (network/DNS/VPN — see the 2026-07-23 `argo.internal` incident, issue #251); `app` means the control plane was reachable and reported the app itself unhealthy (bad readiness, error logs — never 5xx, which is structurally always 0, see Step 1); `unknown` means it was reachable but inconclusive (e.g. sync never completed for a reason that could be either side). **This strongly suggests, but does not prove, the severity** — `infra` is usually P3/transient. Before opening any escalation issue for an `infra`-classed failure: run the Step 1 health check yourself and confirm ArgoCD actually converged (Synced + Healthy) and there's no active Loki `level="Error"` burst. If it converged fine, close as transient with a short note (VPN/connectivity blip, deploy itself succeeded) instead of investigating further — do not treat `infra` as a free pass to skip verification, just as a strong prior that saves you from over-investigating a CI runner network hiccup. +**A `Failure class` field is also provided** (`infra`/`app`/`unknown`), pre-computed by `verify-deploy.sh` itself: `infra` means it could not reach ArgoCD/Loki/Prometheus (network/DNS/VPN — see the 2026-07-23 `argo.internal` incident, issue #251); `app` means the control plane was reachable and reported the app itself unhealthy (bad readiness, error logs — 5xx is never a real webhook signal but CAN be a phantom rate from frozen counters, see Step 1); `unknown` means it was reachable but inconclusive (e.g. sync never completed for a reason that could be either side). **This strongly suggests, but does not prove, the severity** — `infra` is usually P3/transient. Before opening any escalation issue for an `infra`-classed failure: run the Step 1 health check yourself and confirm ArgoCD actually converged (Synced + Healthy) and there's no active Loki `level="Error"` burst. If it converged fine, close as transient with a short note (VPN/connectivity blip, deploy itself succeeded) instead of investigating further — do not treat `infra` as a free pass to skip verification, just as a strong prior that saves you from over-investigating a CI runner network hiccup. If a deploy-failure issue number was provided, that issue is still the incident record — comment your findings on it and close it once resolved (Step 7). If none was provided (e.g. a `sre-manual.yml` dispatch with no related issue), skip the issue comment/close steps and report your findings in the workflow run output only. @@ -12,6 +12,10 @@ If a deploy-failure issue number was provided, that issue is still the incident Your deliverables are **issue comments** with structured incident analysis, **rollback actions** when production is down, **one-liner code mitigations** for simple bugs, and **escalation issues** for complex bugs requiring human attention. +## Non-interactive run + +This run is unattended CI — nobody is watching to answer a question. Never end your report with a question, an offer, or a menu of follow-up options to choose from. Every recommendation you make must be either executed now (Path A one-liner, Step 6) or filed as an issue for a human (Path B, Step 6) — there is no third "ask and wait" option. Your final message must be the finished incident report itself (the Step 7 template), written so it reads correctly as a standalone artifact with no further input from anyone. + ## Prerequisites - VPN is pre-established by the workflow (WireGuard to `*.internal` hosts) @@ -24,7 +28,7 @@ Your deliverables are **issue comments** with structured incident analysis, **ro You already have the workflow run link and commit SHA from your prompt. Determine severity. -**The central fact for every bot in this fleet: pod health and HTTP status code are not usable "is it broken" signals.** A runtime failure in update handling does not crash the process — `/healthz` keeps returning OK and readiness keeps passing. The `/bot` webhook always returns HTTP 200 to Telegram regardless of internal exceptions (deliberately, to avoid Telegram retry storms), so the 5xx rate (`sum(rate(http_server_request_duration_seconds_count{status_code=~"5.."}))`) is **structurally always zero, for every bot, forever** — it is dead logic, not evidence of anything. A bot can be completely non-functional — throwing on every update, replying to nobody — while ArgoCD reports it `Healthy` and every HTTP response is a 200. **Pod health is therefore a necessary-but-not-sufficient signal, never proof of a working bot.** The real "is it actually broken" signal is Loki `level="Error"` on the update-handling path (`level` is a real indexed Loki label, values `Information`/`Warning`/`Error`), plus each bot's own business metrics (Step 1c). +**The central fact for every bot in this fleet: pod health and HTTP status code are not usable "is it broken" signals.** A runtime failure in update handling does not crash the process — `/healthz` keeps returning OK and readiness keeps passing. The `/bot` webhook always returns HTTP 200 to Telegram regardless of internal exceptions (deliberately, to avoid Telegram retry storms), so a genuine 5xx from that route never happens — but the aggregate 5xx rate is **not** "structurally always zero": `/ready` (the DB-ping readiness check, `src/BotInfra/Readiness.fs`) returns 503 by design whenever the DB ping fails, and with 2 replicas per bot and no per-pod label on the Prometheus scrape, `rate()`/`increase()` on that job's counters can report a constant phantom non-zero value once the two pods' raw values diverge and alternate in one series (each alternation looks like a counter reset). **2026-09-08 worked example:** the shared Postgres restarted at 23:58:49Z, `/ready` 503'd on both coupon-bot and vahter-bot for ~2 minutes, the raw counters then froze (29 coupon-pod, 30/29 vahter-pods) — yet `rate()` kept reporting a constant 0.5557/s (coupon) and 0.4466/s (vahter) for the next 9 hours, and `verify-deploy.sh` failed a real deploy on `5xx_rate=0.63`. **Diagnostic:** query the raw counter series and `changes(...[5m])`/`resets(...[5m])` over the window — a non-zero rate on a counter whose raw value never moves is this collision artifact, not traffic. `verify-deploy.sh` now excludes `/ready`/`/healthz` from its 5xx gate, but the same phantom-rate mechanism can surface on any other frozen counter on a 2-replica job — never assume 5xx is dead logic. A bot can be completely non-functional — throwing on every update, replying to nobody — while ArgoCD reports it `Healthy` and every HTTP response is a 200. **Pod health is therefore a necessary-but-not-sufficient signal, never proof of a working bot.** The real "is it actually broken" signal is Loki `level="Error"` on the update-handling path (`level` is a real indexed Loki label, values `Information`/`Warning`/`Error`), plus each bot's own business metrics (Step 1c). | Severity | Criteria | Response | |----------|----------|----------| @@ -114,7 +118,7 @@ Use `gh` CLI to read the failed workflow run logs. The `verify-deploy.sh` script | Phase 2 | `FAILED: Pod is not healthy after` | Pod readiness probes failed beyond the 3-minute grace period | | Phase 3 (Loki) | `FAILED: Error logs detected` | Application is producing Error/Fatal log entries | -Note: `verify-deploy.sh` also emits a `FAILED: 5xx error rate is non-zero` marker for a Prometheus 5xx check, but that check is dead logic — the `/bot` webhook always returns HTTP 200 regardless of internal exceptions, so the underlying rate is structurally always 0 and this marker cannot occur in practice. Do not use it as a signal; Loki `level="Error"` (above) is the real one. +Note: `verify-deploy.sh` also emits a `FAILED: 5xx error rate is non-zero` marker for a Prometheus 5xx check; the `/bot` webhook always returns HTTP 200 regardless of internal exceptions, but the marker CAN still fire from a phantom rate on frozen probe-route counters — never treat 5xx rate as proof of anything without the Step 1 raw-counter check. Do not use it as a signal; Loki `level="Error"` (above) is the real one. ### Step 3: Query Observability Services @@ -191,6 +195,8 @@ curl -s -G http://loki.internal/loki/api/v1/query_range \ | **Code bug** | Application crash, unhandled exception, regression from recent commit | Escalate to coding agent (Step 6) | | **Configuration** | Missing env var, wrong secret, migration failure | Document in issue, label as `infra` | +**When `verify-deploy.sh` failed on a metric threshold, "transient race" is not an acceptable root cause on its own.** The report must state the metric's actual value, which series/route produced it, and since when — pull the raw series (not just the aggregate) and check whether it's actually moving before naming a cause. + ### Step 5: Rollback (if production is impacted) **Only rollback for genuine P1 incidents.** If old replicas are still serving (P2), skip rollback. P1 is now reachable through the update-handling error path (Step 1a), not only "no healthy pods" — so this section will fire more often than it used to. Follow the disable → rollback → re-enable order below exactly; don't skip a step because pods "look" fine. @@ -389,4 +395,4 @@ gh issue close "$DEPLOY_FAILURE_ISSUE_NUMBER" | Waiting reason | `kube_pod_container_status_waiting_reason{container="CONTAINER"}` | | Memory usage | `container_memory_working_set_bytes{container="CONTAINER"}` | -None of the above are "is it doing its job" signals by themselves — a pod can post all-green values on every row here while the update path throws on every message (drill 3). **Never use the 5xx HTTP rate as a health metric — it is structurally always 0** (Step 1). The primary "is it actually broken" signal is the Loki query `sum(count_over_time({container="CONTAINER"} | json | level="Error"[15m]))` (Step 1a), backed by the per-bot business metrics in Step 1c. +None of the above are "is it doing its job" signals by themselves — a pod can post all-green values on every row here while the update path throws on every message (drill 3). **Never use the 5xx HTTP rate as a health metric without the Step 1 raw-counter check — it is never a real webhook signal but CAN be a phantom rate from frozen counters** (Step 1). The primary "is it actually broken" signal is the Loki query `sum(count_over_time({container="CONTAINER"} | json | level="Error"[15m]))` (Step 1a), backed by the per-bot business metrics in Step 1c. diff --git a/.github/workflows/_sre-agent.yml b/.github/workflows/_sre-agent.yml index 4ff60b09..530a00fd 100644 --- a/.github/workflows/_sre-agent.yml +++ b/.github/workflows/_sre-agent.yml @@ -87,6 +87,7 @@ jobs: } >> "$GITHUB_ENV" - name: Run SRE agent + id: run-agent uses: openai/codex-action@v1 with: openai-api-key: ${{ secrets.AZURE_OPENAI_API_KEY }} @@ -96,6 +97,7 @@ jobs: safety-strategy: drop-sudo effort: high codex-home: ${{ github.workspace }}/.github/codex + output-file: /tmp/codex-output/sre-report.md prompt: | ${{ env.AGENT_PROMPT }} @@ -120,6 +122,27 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} ARGOCD_AUTH_TOKEN: ${{ secrets.ARGOCD_AUTH_TOKEN }} + - name: Upload SRE agent report + # codex-action's output-file is the only transcript it exposes. + if: always() + uses: actions/upload-artifact@v4 + with: + name: sre-report-${{ inputs.bot }} + path: /tmp/codex-output/ + if-no-files-found: ignore + retention-days: 14 + + - name: Post SRE agent report to job summary + # Makes the report visible on the run's Summary page without opening logs. + if: always() + run: | + echo "## SRE agent report — ${{ inputs.bot }}" >> "$GITHUB_STEP_SUMMARY" + if [ -f /tmp/codex-output/sre-report.md ]; then + cat /tmp/codex-output/sre-report.md >> "$GITHUB_STEP_SUMMARY" + else + echo "no report file produced (agent step failed or was cancelled)" >> "$GITHUB_STEP_SUMMARY" + fi + - name: Re-enable ArgoCD auto-sync (safety net — a killed run must not leave it off) # Idempotent: a no-op if the agent never touched auto-sync. Runs even if # earlier steps failed/were cancelled, so a killed investigation can't diff --git a/scripts/verify-deploy.sh b/scripts/verify-deploy.sh index 6fae688a..58f6f7ec 100755 --- a/scripts/verify-deploy.sh +++ b/scripts/verify-deploy.sh @@ -352,9 +352,10 @@ RESTARTS=$(curl -sf -G "${PROMETHEUS_URL}/api/v1/query" \ 2>/dev/null || echo '{"data":{"result":[]}}') RESTART_COUNT=$(echo "$RESTARTS" | jq -r '[.data.result[].value[1] | tonumber] | add // 0') -# Check 5xx rate +# Check 5xx rate. Readiness (/ready, /healthz) 503s are excluded: those +# counters are expected to be non-zero after any DB blip; Phase 2 covers readiness. ERRORS_5XX=$(curl -sf -G "${PROMETHEUS_URL}/api/v1/query" \ - --data-urlencode "query=sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~\"5..\",job=\"${CONTAINER}\"}[5m]))" \ + --data-urlencode "query=sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~\"5..\",http_route!~\"/ready|/healthz\",job=\"${CONTAINER}\"}[5m]))" \ 2>/dev/null || echo '{"data":{"result":[]}}') ERROR_5XX_RATE=$(echo "$ERRORS_5XX" | jq -r '[.data.result[].value[1] | tonumber] | add // 0') @@ -363,9 +364,9 @@ log " Prometheus: restarts=${RESTART_COUNT}, 5xx_rate=${ERROR_5XX_RATE}" # Note: restart count is cumulative, so we only fail if it's unexpectedly high. # For a fresh deployment, a single restart might be acceptable. if [ "$(echo "$ERROR_5XX_RATE > 0" | bc -l 2>/dev/null || echo 0)" = "1" ]; then - log "FAILED: 5xx error rate is non-zero: ${ERROR_5XX_RATE}" + log "FAILED: 5xx error rate is non-zero: ${ERROR_5XX_RATE} (probe routes excluded)" summary "### ❌ Phase 3: Log & Metrics Check FAILED" - summary "- **5xx Error Rate:** ${ERROR_5XX_RATE} (expected: 0)" + summary "- **5xx Error Rate:** ${ERROR_5XX_RATE} (expected: 0, probe routes excluded)" summary "- **Container Restarts:** ${RESTART_COUNT}" emit_failure_class "app" exit 1