Skip to content

perf(deploy): pre-pull the worker image on every GPU node - #366

Closed
hodaaaaaaaaaa wants to merge 1 commit into
mainfrom
perf/worker-image-prepull
Closed

hodaaaaaaaaaa wants to merge 1 commit into
mainfrom
perf/worker-image-prepull

Conversation

@hodaaaaaaaaaa

Copy link
Copy Markdown
Collaborator

Problem

A new worker on a node without the worker image pulls it first, several GB that take minutes, and a cold job pays it for its trainer and its sampler.

Change

The base bundle adds the open-rl-worker-prepull DaemonSet, which pulls the worker image onto every GPU node with openrl.io/enabled=true at deploy time and only sleeps. The kustomize images transformer rewrites its image like every other OpenRL image, so a release that bumps the tag re-pulls on every node. The kind overlays drop it, since their workers pull from a cluster-local registry.

Based on droot's prototype in #343.

Results

Measured on GKE with eight GSM8K GRPO jobs launched together (gemma-4-E2B and Qwen3-0.6B, LoRA and full fine-tuning, A100 80GB and L4), with the worker image removed from every node before each run, two runs each:

  • A cold worker image pull took about 3 minutes per worker.
  • With the DaemonSet rolled out first, every job reached its first training step 330–400 s sooner, and the whole workload took 749 s instead of 1111 s.

@hodaaaaaaaaaa
hodaaaaaaaaaa marked this pull request as draft October 22, 2026 16:25
@hodaaaaaaaaaa
hodaaaaaaaaaa deleted the perf/worker-image-prepull branch October 23, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant