Skip to content

k8s/deploy/demo: live lights, an idle fleet view, and the recipe on t… - #438

Draft
droot wants to merge 1 commit into
openrl-dev:mainfrom
droot:pr/demo-sweep-polish
Draft

droot wants to merge 1 commit into
openrl-dev:mainfrom
droot:pr/demo-sweep-polish

Conversation

@droot

@droot droot commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

…he Sweep tab

Live state

  • The Fleet chips and the Timeline's current turn come from /api/v1/inspect/now, polled twice a second, instead of replaying the activity history about 8 s behind. A turn, op or batch of sampler requests that just ended is kept as a finished span until the history has it. On an API server without the endpoint (404), the page falls back to the replay, so it works with or without it.
  • On a shared LoRA runtime, each job's chip is lit only while that job's op or sampler requests run.

Fleet

  • A sweep selector entry, "no sweep": the pool with no job's counts or chips, any worker holding a GPU drawn as a neutral chip. ?sweep=none opens on it; starting a sweep switches back to the active sweeps.
  • A time-sliced worker is drawn on the GPU only in its turns, even before its first one, so a worker waiting for the GPU no longer shows as holding it. The timeline's tick step follows its width, and its legend counts holders that do not fit.

Sweep

  • Cards under the spec show the recipe every job runs: GSM8K, GRPO, 8 x 8 answers per step, 512 tokens, learning rates and LoRA rank, and the spec's totals as it is edited. The batch settings are constants in app.py that the Jobs are launched with, so the cards show what runs.
  • The job list is collapsed until opened, so the page before a launch shows the spec and not earlier sweeps.
  • "demo" is gone from the header and title, runs are named sweep-HHMMSS, the keynote-20 preset is mixed-20, and the qwen-8 preset is gone.

Run on open-rl-dra-use5 (8 x H100) with 16- and 20-job sweeps.

@droot
droot force-pushed the pr/demo-sweep-polish branch 4 times, most recently from e25acec to 0ef5e37 Compare November 6, 2026 19:00
…he Sweep tab

Live state
- The Fleet chips and the Timeline's current turn come from /api/v1/inspect/now, polled twice a second, instead of
  replaying the activity history about 8 s behind. A turn, op or batch of sampler requests that just ended is kept as
  a finished span until the history has it. On an API server without the endpoint (404), the page falls back to the
  replay, so it works with or without it.
- On a shared LoRA runtime, each job's chip is lit only while that job's op or sampler requests run.

Fleet
- The Fleet tab opens on "no sweep": the pool with no job's counts or chips, any worker holding a GPU drawn as a
  neutral chip. Starting a sweep switches it to the active sweeps at the click, holding on the empty pool until the
  new sweep's first Job appears rather than showing the previous sweep; ?sweep=active, ?sweep=all or ?sweep=<run> opens on
  those instead.
- A time-sliced worker is drawn on the GPU only in its turns, even before its first one, so a worker waiting for the
  GPU no longer shows as holding it. Work that runs at the same time shares the trace's height, one band per job,
  so a LoRA sampler serving five jobs in one batch shows all five instead of the selected job hiding the rest. The
  timeline's tick step follows its width, and its legend counts holders that do not fit.
- The pool shows only nodes labelled openrl.io/enabled (the ones the scheduler places workers on), or every GPU node
  when none is; GPU names still resolve on every node, so older sweeps' timelines keep them.
- No end card when a sweep finishes: the headline, the done tile and the clock already say so.

Sweep
- Cards under the spec show the recipe every job runs: GSM8K, GRPO, 8 x 8 answers per step, 512 tokens, learning
  rates and LoRA rank, and the spec's totals as it is edited. The batch settings are constants in app.py that the
  Jobs are launched with, so the cards show what runs.
- The job list is collapsed until opened, so the page before a launch shows the spec and not earlier sweeps.
- "demo" is gone from the header and title, runs are named sweep-HHMMSS, the keynote-20 preset is mixed-20, and the
  qwen-8 preset is gone.

Run on open-rl-dra-use5 (8 x H100) with 16- and 20-job sweeps.
@droot
droot force-pushed the pr/demo-sweep-polish branch from 0ef5e37 to fdb70ff Compare November 7, 2026 00:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant