Run inspection API: design and rollout plan #390

Right now debugging a slow or stuck job means jumping between Redis, kubectl, Prometheus, and Cloud Logging by hand. We want to put a read-only inspection API in front of all of that on the API server, with a lightweight dashboard and agent-friendly summary endpoints on top.

Design

Where the data comes from

• Redis: OpenRL's own state for each model (metadata, live sessions, per-step op records, worker heartbeats, and placements).
• Kubernetes: pods, nodes, Workloads, and the GPUs in each node's ResourceSlices.
• DCGM + Prometheus: GPU activity from a DCGM exporter running on every GPU node, stored alongside vLLM metrics in Prometheus (Managed Prometheus on GKE).
• Cloud Logging: worker logs, labeled by model.

What we want to expose

Everything lives as read-only routes under /api/v1/inspect on the API server:

• Sessions: live sessions and the models created under each one.
• Models: lifecycle state (active, waiting, placed, failed, ended), config/settings, and step count.
• Steps: the ops in each step with their timing and metrics, plus any client-reported metrics.
• Workers & placements: which pod and GPU served a model, and when.
• GPUs & nodes: cluster capacity, which GPUs are currently held, and how busy each one is.
• Logs: worker and client logs for a model.
• Summaries (for agents): a compact per-model and per-session rollup covering pace, failures, lost or stuck requests, and where to pull logs.

Open question: the API already says run today, while the server codebase calls it a model. We should pick one name and stick to it before landing the rest of these routes.

Rollout plan

• 1. Inventory from Redis and Kubernetes (feat(dashboard): serve run inventory from Redis at /api/v1/inspect/inventory #388, feat(dashboard): read pods, nodes, Workloads and DRA GPUs from Kubernetes #389)
• 2. Deploy setup: split out separate service accounts for the API server and scheduler, add a DCGM exporter to every GPU bundle, and write up the IAM setup docs.
• 3. Cluster view and logs: wire the Kubernetes data into /inventory and add /logs.
• 4. Placements: record placements from Workload status.
• 5. GPU metrics and activity: add /query, /activity, and per-GPU duty.
• 6. Recorder and sampler telemetry: buffered writes, worker heartbeats, and per-batch sampler records. Gated on no regression in sampler throughput.
• 7. Trainer telemetry: op records, queue wait time, step counter, and /steps. Gated on trainer step time.
• 8. Agent summaries: model and session summary endpoints.
• 9. UI: ship the whole dashboard in one PR (static assets, no build step).