feat: sample through sampling pools

Makefile

@@ -19,7 +19,7 @@ HOST           ?= 127.0.0.1
 PORT           ?= 9003
 # The fully qualified base URL used by local CLI tools and clients
 BASE_URL       ?= http://$(HOST):$(PORT)
-UNIT_TESTS ?= tests.test_session_lifecycle tests.test_proto_codec tests.test_forward_only tests.test_fft_batch_failure tests.test_api_server_paths tests.test_accel_timeslicer tests.test_trainer_optimizer_correctness tests.test_trainer_logprob_chunks tests.test_worker_manager tests.test_scheduler_worker_manager tests.test_estimator tests.test_redis_store tests.test_cluster_eval_script tests.test_perf_compare_script tests.test_delta_weight_sync tests.test_diffing_backends tests.test_sampler_weight_rotation tests.test_commands tests.test_weight_sync_config tests.test_exclusive_trainer tests.test_delete_model tests.test_automodel_worker tests.test_worker_thread_device tests.test_trainer_data_parallel tests.test_sampler_batch_limits tests.test_openrl_status tests.test_skill_docs tests.test_trainer_ready tests.test_sampling_store tests.test_lora_snapshots tests.test_pool_accept tests.test_sampling_dispatcher
+UNIT_TESTS ?= tests.test_session_lifecycle tests.test_proto_codec tests.test_forward_only tests.test_fft_batch_failure tests.test_api_server_paths tests.test_accel_timeslicer tests.test_trainer_optimizer_correctness tests.test_trainer_logprob_chunks tests.test_worker_manager tests.test_scheduler_worker_manager tests.test_estimator tests.test_redis_store tests.test_cluster_eval_script tests.test_perf_compare_script tests.test_delta_weight_sync tests.test_diffing_backends tests.test_sampler_weight_rotation tests.test_sampler_batch_limits tests.test_commands tests.test_weight_sync_config tests.test_exclusive_trainer tests.test_delete_model tests.test_automodel_worker tests.test_worker_thread_device tests.test_trainer_data_parallel tests.test_openrl_status tests.test_skill_docs tests.test_trainer_ready tests.test_sampling_store tests.test_lora_snapshots tests.test_pool_accept tests.test_sampling_dispatcher tests.test_vllm_sampler tests.test_vllm_options
 # Only forward BASE_URL to e2e when the user supplied it. The Makefile default
 # is for local CLI usage; e2e should start its own backend by default.
 TRAINING_TEST_BASE_URL ?= $(if $(filter environment command line,$(origin BASE_URL)),$(BASE_URL),)
@@ -115,7 +115,7 @@ test:
 
 .PHONY: test-weight-transfer
 test-weight-transfer:
-	uv run --frozen --extra vllm --extra gpu --extra cluster python -m unittest tests.test_delta_weight_transfer_engine tests.test_delta_weight_sync tests.test_weight_sync_config tests.test_sampler_patch tests.test_weight_transfer_gpu
+	uv run --frozen --extra vllm --extra gpu --extra cluster python -m unittest tests.test_delta_weight_transfer_engine tests.test_delta_weight_sync tests.test_weight_sync_config tests.test_weight_transfer_gpu
 
 lint:
 	uv run --extra dev ruff check .

docs/configuration.md

@@ -48,15 +48,15 @@ REDIS_URL=redis://127.0.0.1:6379/0 VLLM_ARCHITECTURE_OVERRIDE=Gemma4ForCausalLM
 | Env var | Default | What it does |
 | --- | --- | --- |
 | `BASE_MODEL` | unset | Hugging Face model id loaded by the trainer and, when using vLLM, by the sampler. |
-| `SAMPLING_BACKEND` | `torch` in one process, `vllm` when `REDIS_URL` is set | Sampling backend selector. `torch` samples in the training process. `vllm` queues sampling requests for a vLLM sampler worker and needs `REDIS_URL`. |
+| `SAMPLING_BACKEND` | `torch` in one process, `vllm` when `REDIS_URL` is set | Sampling backend selector. `torch` samples in the training process. `vllm` samples through the sampling pools described below and needs `REDIS_URL`. |
 | `REDIS_URL` | unset | Enables distributed mode by switching the request store to Redis. Leave unset for a single-machine run. |
 | `OPEN_RL_FUTURE_TTL_S` | `300` | How long resolved request results stay readable by `retrieve_future` after a worker resolves them. |
 
 ## Server paths
 
 | Env var | Default | What it does |
 | --- | --- | --- |
-| `OPEN_RL_TMP_DIR` | `/tmp/open-rl` | Root directory for adapter snapshots under `peft/` and saved states under `checkpoints/`. |
+| `OPEN_RL_TMP_DIR` | `/tmp/open-rl` | Root directory for adapter snapshots under `peft/` and saved states under `checkpoints/`. Required with `OPEN_RL_WORKER_MANAGER=scheduler`, set to the shared volume every worker pod mounts. |
 | `OPEN_RL_TRAIN_TOKEN_BUDGET` | `auto` | Maximum `batch_size * max_sequence_length` for padded trainer chunks inside one `forward_backward` request. `auto` uses the smaller of 16384 and the tokens whose activations fit in half of the free device memory, read at each request. A number is used as given; `0` runs one datum per pass. |
 | `TRAINER_CUDA_VISIBLE_DEVICES`, `SAMPLER_CUDA_VISIBLE_DEVICES` | unset | GPUs given to the trainer and sampler worker processes the local worker manager launches. |
 
@@ -100,3 +100,25 @@ On Kubernetes the release bundles set these variables. The API server runs with
 each trainer and sampler, and the OpenRL scheduler starts their pods. The API
 server writes each worker's environment into the `Workload`'s pod template, so
 there are no worker deployments to configure by hand.
+
+## Sampling pools
+
+Sampling goes through sampling pools. A pool serves one FFT job, one exclusive
+LoRA job, or every shared LoRA job on one base model. Pending requests stay in a
+Redis stream until a result is saved, so the Redis in `k8s/deploy/base` keeps an
+append-only file on a persistent volume and never evicts keys.
+
+A pool that one job owns runs vLLM in the sampler's own process when it has one
+replica. A pool with two or more replicas (`openrl.sampler_replicas`), and every
+pool that shared LoRA jobs use, runs `vllm serve` Workloads behind one CPU
+Deployment, named after the pool, that runs the dispatcher, Envoy and the llm-d
+endpoint picker (configured by the `open-rl-sampler-router` ConfigMap in
+`k8s/deploy/base`). A shared pool grows to the most replicas any of its jobs asks
+for. On one machine every pool has one replica, in process. See
+[the sampling pools design](designs/017-sampling-pools.md).
+
+| Env var | Default | What it does |
+| --- | --- | --- |
+| `OPEN_RL_DISPATCH_CONCURRENCY` | `256` | Requests one pool keeps in flight. |
+| `OPEN_RL_SAMPLE_DEADLINE_SECONDS` | `3600` | How long a sampling request waits for a result before it fails. Set it on the API server. Worker pods and pool dispatchers get the same value. |
+| `OPEN_RL_DISPATCHER_IMAGE` | the worker image | Image for each pool's dispatcher container. The release sets it to the API-server image, which carries the dispatcher without the GPU stack and pulls in seconds. |

docs/designs/016-agent-friendly-openrl.md

@@ -218,13 +218,13 @@ that every launcher reports:
 |---|---|---|---|
 | `waiting_for_placement` | no accelerator assigned yet | Workload without a `nodeName` (`Pending`, early `Placing`) | never |
 | `starting` | placed; pulling the image, mounting volumes, loading the model | `nodeName` set, phase not yet `Running`; or `Running` but the worker has not announced itself | process alive, worker not announced |
-| `ready` | serving requests | Workload `Running` and the worker announced itself: `open_rl:sampler_ready:<runtime>` once vLLM can serve, `open_rl:trainer_ready:<runtime>` once the trainer holds a loaded model and drains its queue | process alive and announced |
+| `ready` | serving requests | Workload `Running`, and for a sampler the pod `Ready`, which its readiness probe allows only while vLLM serves the weights the pool expects; a trainer announces itself with `open_rl:trainer_ready:<runtime>` once it holds a loaded model and drains its queue | process alive, and a trainer announced |
 | `failed` | exited with an error, or the request cannot be satisfied | Workload `Failed` | exit code not zero |
 | `stopped` | exited normally, or being released | Workload `Succeeded`, or a deletion timestamp | exit code zero |
 
-Both announcements are TTL keys the worker refreshes each minute and
-deletes on exit, so a dead process stops looking ready on its own. The
-trainer's comes from the request processor asking the worker whether it
+The announcement is a TTL key the trainer refreshes each minute and
+deletes on exit, so a dead process stops looking ready on its own. It
+comes from the request processor asking the worker whether it
 holds a model (`is_ready()`), so a LoRA trainer is `starting` during its
 preload and a full fine-tuning trainer until its `create_model` has
 loaded the weights.

docs/fft/fft.md

@@ -112,7 +112,7 @@ Open-RL segregates operations into a **Control Plane** (infrastructure provision
 API Server Control Queue (open_rl:worker_launch_queue) ---> WorkerLaunchProcessor (launches/gracefully stops PIDs)
 
 [Data Plane Operations]
-API Server Data Queues (queue:<model_id>, sampler_queue:<model_id>) ---> Workers (runs steps/computes tokens)
+API Server Data Queues (queue:<model_id>, sampling:<pool>) ---> Workers (runs steps/computes tokens)
 ```
 
 ### Queue Division & Protocols
@@ -124,16 +124,16 @@ API Server Data Queues (queue:<model_id>, sampler_queue:<model_id>) ---> Workers
 | **Control Plane** | `create_sampling_session` | Registry Metadata Key (Redis) | API Server |
 | **Data Plane** | `forward_backward`, `optim_step` | `open_rl:queue:<model_id>` (Isolated) | PyTorch Trainer Worker |
 | **Data Plane** | `save_weights_for_sampler` | `open_rl:queue:<model_id>` (Isolated) | PyTorch Trainer Worker |
-| **Data Plane** | `sample`, `asample` | `open_rl:sampler_queue:<model_id>` (Isolated) | vLLM Sampler Worker |
+| **Data Plane** | `sample`, `asample` | `open_rl:sampling:<pool>` (Redis stream, one per pool) | The pool's dispatcher |
 
 ### Provisioning & Activation Mechanics
 * **Trainer Provisioning:** When `/api/v1/create_model` is invoked, the API Server enqueues a launch command. `WorkerLaunchProcessor` pops the request and invokes `FFTWorkerManager.launch_trainer(model_id)` to provision the pod or subprocess.
-* **Sampler Provisioning:** When `/api/v1/create_sampling_session` is invoked, `WorkerLaunchProcessor` invokes `FFTWorkerManager.launch_sampler(model_id)`. The sampler compiles CUDA graphs and posts `open_rl:sampler_ready:<model_id> = "1"` to Redis. The API Server polls this key, blocking session return until physical readiness is verified.
+* **Sampler Provisioning:** When `/api/v1/create_sampling_session` is invoked, the API Server makes sure the model's sampling pool exists and returns at once. Samples wait in the pool's Redis stream until a sampler is up to serve them.
 
 ### Graceful Queue-Based Teardown (Sentinel Pattern)
 To prevent dropping in-flight rollouts during de-provisioning, teardown uses the **Sentinel Pattern**:
 1. When `/api/v1/delete_model` is invoked, the API Server enqueues a `shutdown_workers` command to `open_rl:worker_launch_queue`.
-2. `WorkerLaunchProcessor` enqueues a **Shutdown Sentinel (Poison Pill)** (`{"request_id": "SHUTDOWN_SENTINEL"}`) to the tails of both the trainer (`queue:<model_id>`) and sampler (`sampler_queue:<model_id>`) FIFO data queues.
+2. `WorkerLaunchProcessor` enqueues a **Shutdown Sentinel (Poison Pill)** (`{"request_id": "SHUTDOWN_SENTINEL"}`) to the tail of the trainer's FIFO data queue (`queue:<model_id>`). The job's pending samples fail at once with `<model_id> was deleted`.
 3. Workers continue popping and processing pending tasks. Upon popping the sentinel, they halt polling, complete active asyncio tasks, unregister from the time-slicer daemon, and exit gracefully.
 
 ---

k8s/deploy/base/rbac.yaml

@@ -11,10 +11,10 @@ rules:
   verbs: ["create", "get", "list", "watch", "delete"]
 - apiGroups: ["apps"]
   resources: ["deployments"]
-  verbs: ["create", "get", "delete"]
+  verbs: ["create", "get", "list", "delete"]
 - apiGroups: ["networking.k8s.io"]
   resources: ["networkpolicies"]
-  verbs: ["create", "delete"]
+  verbs: ["create", "list", "delete"]
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["list"]

skills/openrl/references/lifecycle.md

@@ -8,8 +8,8 @@ an image pull (the worker image is several GiB) and a model download.
 
 | Your call | What starts | Shared with |
 |---|---|---|
-| `create_lora_training_client` (LoRA, default) | one trainer for the base model, Workload `lora-<base>-0-trainer` | every other LoRA job on the same base model; each job is one adapter in it |
-| first `sample` or `save_weights_and_get_sampling_client`, or already at `create_lora_training_client` with `openrl.recipe=rl` | one sampler, `lora-<base>-0-sampler` (plus `-1`, `-2`... with `openrl.sampler_replicas`) | same |
+| `create_lora_training_client` (LoRA, default) | one trainer for the base model, Workload `lora-<base>-<hash>-0-trainer` | every other LoRA job on the same base model; each job is one adapter in it |
+| first `sample` or `save_weights_and_get_sampling_client`, or already at `create_lora_training_client` with `openrl.recipe=rl` | one sampler, `lora-<base>-<hash>-0-sampler` (plus `-1`, `-2`... up to the largest `openrl.sampler_replicas` any job asked for), behind a CPU dispatcher pod | same |
 | `create_lora_training_client` with `openrl.fine_tuning_type=full`, or `openrl.exclusive=true`, or `openrl.trainer_gpus>1` | a trainer of its own, `fft-<model_id>-trainer`, and later `fft-<model_id>-sampler` | nothing; full fine-tuning workers take turns on a GPU with other jobs' workers unless exclusive |
 
 The workers that serve one job appear together in the job's model view,
@@ -20,14 +20,17 @@ The workers that serve one job appear together in the job's model view,
 Each worker in the model view has a `state`, the same five values on every server, and a
 `detail` in the scheduler's own words (Workload phase and reason, node). Branch on `state`;
 show `detail` to a human. `kubectl -n openrl-system get workloads` shows the same facts
-`detail` quotes; the pods behind them are the scheduler's and the view does not read them.
+`detail` quotes. The pods behind them are the scheduler's; the view reads only whether a
+running sampler's pod is Ready, and names the reason when its container is waiting, for
+example `CrashLoopBackOff`. A sampler is also `starting` while its pool's dispatcher has no
+pod available, since no sample is sent without it.
 
 | State | Meaning | What `detail` says | Normal duration |
 |---|---|---|---|
 | `waiting_for_placement` | no accelerator assigned yet | `Pending (NoCapacity: ...)`: nothing in the cluster fits; `Pending (WaitingForCapacity)`: something fits but is busy | seconds; longer only when the cluster is full |
 | `starting` | placed on a node; pulling the image, mounting volumes, or loading the model | `Placing (PodCreated) on <node>` is the image pull and volume mount; `Running ..., loading the model` is the trainer loading the base model or vLLM starting | under a minute on a warm node; 5 to 10 minutes on a cold node. Trainer: `ready` about 2 minutes after `create_model`; sampler: 3 to 4 minutes more while vLLM starts |
 | `ready` | serving requests: the worker has its model loaded and is draining its queue | node name | |
-| `failed` | the process exited with an error, or the request cannot be satisfied | the scheduler's reason, for example `Failed (PodFailed)`. A container that cannot pull its image or crash-loops is not reported as failed yet; it stays `starting` or `ready` past the normal duration, see the runbook | |
+| `failed` | the process exited with an error, or the request cannot be satisfied | the scheduler's reason, for example `Failed (PodFailed)`. A container that cannot pull its image or crash-loops is not reported as failed yet; it stays `starting` or `ready` past the normal duration (a crash-looping sampler is `starting` with the reason in `detail`), see the runbook | |
 | `stopped` | exited normally, or being released | `being released; ...` while the reaper or `delete_model` tears it down | |
 
 While a call blocks, ask the server about the model from another shell; the examples