feat(oncall-agent): read-only Slack-triggered cluster triage agent

[manan164]

feat(oncall-agent): read-only Slack-triggered cluster triage agent

Draft. The scaffold, the read-only investigation loop, and a per-alert-type triage
playbook for all 17 health-check alert types now work. What remains is a live
end-to-end run against ah5r-prod and a hypothesis-quality eval.

What this is

An Agentspan agent that triages Orkes SaaS cluster health-check alerts. It polls the Slack
alert channel (Web API, no Socket Mode), parses the executionId from the failing
health_check execution URL, runs read-only agent-handler commands against the ah5r-prod
Conductor API to investigate, and replies in-thread with a root-cause hypothesis.

It is strictly read-only and advisory — it never takes a remediating action. SQL is gated by
a deterministic SELECT-only guard (sql_guard.py), not by trusting the model. The agent reads
organizationId / clusterName / cloudEnvironmentTag off the failing execution, so the LLM
only threads the executionId into each tool — it cannot target the wrong cluster and no secrets
pass through tool args.

Done ✅

• Slack ingestion — Web API poller (conversations.history + chat.postMessage), bot token
  only, run-once / --loop, state-file dedup. Matches sdk/python/examples/91_slack_autofix_agent.py.
• Alert parsing (alert.py) — extracts execution id + severity + org/cluster from the alert text.
• Conductor dispatch (conductor_client.py) — starts agent-handler command workflows on
  ah5r-prod via conductor-python (app key/secret), polls to completion, derives + caches cluster context.
• Read-only tool set (tools.py) — get_incident_details, get_cluster_metrics,
  get_infrastructure_metrics, get_pods_data, get_deployments_info, get_pod_events,
  get_top_output, pull_pod_logs, get_ingress_info, run_sql_select (SELECT-guarded).
• SQL safety guard (sql_guard.py) — SELECT/WITH/EXPLAIN/SHOW only; rejects every mutation,
  multi-statement, and comment-smuggling case before it reaches the DB.
• Per-alert-type triage playbook for all 17 health-check alert types (agent.py) — symptom →
  evidence-to-gather → what-to-cite, matched off the issues text. Keeps the strong Redis /
  decider-queue / CPU / heap guidance and extends to component-down, pod, networking, and
  self-describing alerts (table below).
• Tests (deterministic, no LLM in the assertion path, per CLAUDE.md) — test_sql_guard.py,
  test_alert.py, test_poller.py, test_conductor_client.py, test_tools_readonly.py
  (read-only safety guard, pinned to the real AgentHandlerCommand enum names),
  test_tools_dispatch.py (each tool dispatches its expected agent-handler command). 35 passing.
• Local run path (python -m oncall_agent.main [triage <execId>]), .env.example, README.

To do 🚧

• End-to-end run against ah5r-prod — needs Conductor app key/secret, Slack app tokens, and the
  Agentspan server running with ANTHROPIC_API_KEY. Not yet exercised live.
• Hypothesis-quality eval — run dry-run on real alerts and score the root-cause hypotheses
  before trusting it. Tune the playbook from real misses.
• Remediation — deliberately out of scope for v1; when added it must go behind the Agentspan
  HITL approval gate.

The 17 health-check alert types — all now have a playbook

Source of truth: HealthIssue enum in
orkes-saas/.../worker/HealthCheckIssuesWorker.java. "Approach" = how agent.py triages it.

#	Alert type	Sev	Triage approach
1	REDIS_CRITICAL_USAGE	CRITICAL	decider-queue backlog → server/worker logs
2	REDIS_HIGH_USAGE	MAJOR	decider-queue backlog → server/worker logs
3	CONDUCTOR_HIGH_HEAP_USAGE	MAJOR	top + infra metrics → logs grep OutOfMemory/GC
4	CONDUCTOR_HIGH_CPU_USAGE	MAJOR	top + infra metrics → hot-pod logs
5	CONDUCTOR_ERROR_LOGS_COUNT_EXCEEDED_THRESHOLD	MAJOR	server logs → name dominant exception
6	CONDUCTOR_WARN_LOGS_COUNT_EXCEEDED_THRESHOLD	MINOR	server logs → name dominant warning
7	CONDUCTOR_HEALTHY (failed)	CRITICAL	conductor pod events + logs (crashloop/OOM/image)
8	WORKERS_HEALTHY (failed)	CRITICAL	worker pod events + logs
9	PROMETHEUS_NOT_RUNNING	MAJOR	prometheus pod events + logs (note: metrics may be stale)
10	POD_NOT_RUNNING	MAJOR	pod events (schedule/image/OOM) + logs
11	POD_RESTARTED	MAJOR	pod events for reason + pre-crash logs
12	DNS_HEALTHY (failed)	CRITICAL	get_ingress_info — no address ⇒ LB unprovisioned; else external → infra
13	DOMAIN_RESOLUTION	CRITICAL	get_ingress_info → resolve vs escalate to infra
14	DOMAIN_REACHABILITY	CRITICAL	get_ingress_info → reachable vs escalate to infra
15	AUTH_STALE	MAJOR	self-describing → relay + rotate cluster API key (remediation)
16	DOMAIN_CERTIFICATE_WILL_EXPIRE	MINOR	self-describing (domain + days in msg) → renew cert
17	RESPONSE_TIME	MINOR	optionally correlate CPU/heap/restarts, else relay latency

On testing the playbook: the playbook is prompt text — by CLAUDE.md rule 1 it can't be
LLM-judged in unit tests and isn't deterministically assertable, so it has no unit test by design.
What is tested deterministically: the read-only safety guard and the per-tool dispatch contract
(get_ingress_info → GET_INGRESS_INFO, mutating commands stay unreachable). Playbook quality is
validated in the eval step above.

Testing

cd oncall-agent
PYTHONPATH=src python -m pytest -q   # 35 passing

Out of scope (v1)

Triage + read-only investigation only. No remediation (restart/scale/rollback).