Repository navigation
feat(dashboard): read pods, nodes, Workloads and DRA GPUs from Kubernetes - #389
Conversation
| name: open-rl-dashboard | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: open-rl-dashboard |
There was a problem hiding this comment.
I am still thinking if this should be a separate component. I am able to argue both ways ....
There was a problem hiding this comment.
My reasoning was that I wanted to isolate any like tinker specific server stuff from dashboard stuff. I agree it should be core to our service though I am also able to argue both ways
There was a problem hiding this comment.
thought more about it, since the inspect route is different from the post training api, there will be clear branching in the http router layer. I am leaning towards keeping them all in single api-server to simplify the deployment.
Not in this PR (or may be), we can have separate kubernetes SA for just the api-server with restricted permissions. WDYT ?
There was a problem hiding this comment.
Removed standalone deployment and included the routes in the main api_server
…etes server.dashboard.kubernetes lists pods, Workloads, nodes and NVIDIA ResourceSlices, and turns a cluster that cannot be read into an error. A node's GPU count includes the devices in its newest ResourceSlice generation. The API server's open-rl-sa account can already list all four through the base RBAC. Nothing calls this module yet.
696fbf4 to
1b407c8
Compare
Why
The inventory needs to show where each run's workers are and what is wrong with them. This PR adds the module that reads that from Kubernetes. It does not depend on #388. A small follow-up wires it into
/inventory.What changed
server.dashboard.kuberneteslists pods, Workloads, nodes, and NVIDIA ResourceSlices. It names each pod's problem, such asCrashLoopBackOfforUnschedulable.Scope
No new RBAC. The API server's
open-rl-saaccount can already list all four resources through the base roles (open-rl-fleet-reader,open-rl-worker-launcher).Verification
make testandmake lintpass, alone and merged with feat(dashboard): serve run inventory from Redis at /api/v1/inspect/inventory #388.