[hardware, fsdp, vllm, trainer, ray] feat: Intel GPU support for e2e GRPO/PPO/SFT training (FSDP + vLLM rollout) by kahlun · Pull Request #1 · kahlun/verl

kahlun · 2026-06-03T07:00:12Z

What does this PR do?

Adds direct Intel GPU (Pytorch xpu variant) support to verl core for end-to-end GRPO, PPO, and SFT training using FSDP/FSDP2 + vLLM rollout (colocated mode).

Validated on: Intel Arc Pro B60 (Battlemage, 2×24 GB):

Config	timing_s/step	throughput	weight sync
1-GPU colocated GRPO	51.2 s	148.2 tok/s	5.1 s
2-GPU colocated GRPO	93.0 s	41.3 tok/s*	5.3 s

*2-GPU throughput with same 16-prompt batch (not scaled up); full throughput benefit requires larger batch size.

Model: Qwen2.5-0.5B-Instruct, dataset: GSM8K, 16 prompts/batch.

Changes

New device utilities (verl/utils/device.py):

is_torch_xpu_available(), is_xpu_available flag
get_device_name() → "xpu" branch
get_nccl_backend() → "xccl" for XPU (oneCCL)
get_visible_devices_keyword() → "ZE_AFFINITY_MASK" (Level Zero)

Distributed (verl/utils/distributed.py):

Composite backend "cpu:gloo,xpu:xccl" for Intel GPU process group only
all_reduce_avg(): SUM + divide workaround (ReducedOp.AVG will be supported by PyTorch 2.13)

FSDP (verl/utils/fsdp_utils.py):

set_force_sum_reduction_for_comms(True) for FSDP2 on Intel GPU

Ray resource mapping (verl/single_controller/ray/base.py):

Map device "xpu" → Ray resource "GPU" (Ray's IntelGPUAccelerator registers XPU under the "GPU" key, same as CUDA)

Engine registry (verl/workers/engine/fsdp/transformer_impl.py):

Add "xpu" to @EngineRegistry.register(device=[...]) for language and value model engines

vLLM rollout (verl/workers/rollout/vllm_rollout/vllm_async_server.py):

Force uni executor for TP=1 on Intel GPU (avoids Level Zero multi-context OOM)
Pop ONEAPI_DEVICE_SELECTOR before vLLM's EngineCore subprocess spawns (prevents oneDNN OpenCL init crash)
Apply Intel GPU-specific vLLM patches before engine init

vLLM patches (verl/utils/vllm/intel_gpu_patches.py, new file):

Skip false-OOM preflight check on Intel GPU(vllm.v1.worker.utils.request_memory)
Wrap profiling assert in GPUWorker.determine_num_available_blocks

Runtime env (verl/trainer/constants_ppo.py):

Propagate ZE_AFFINITY_MASK to Ray workers; explicitly drop ONEAPI_DEVICE_SELECTOR

Worker (verl/single_controller/base/worker.py):

XPU device init and get_device_name() routing

Replica (verl/workers/rollout/replica.py):

Use get_device_name() in init_colocated / init_standalone (covers XPU, CUDA, NPU generically)

Docker + tests (new files):

docker/intel_gpu/Dockerfile.intel_gpu — reproducible build (oneAPI 2025.3, compute-runtime 26.09, torch 2.11.0+xpu, vLLM 0.17.1)
docker/intel_gpu/README.md — setup guide
requirements-intel-gpu.txt — intel GPU-specific pip deps
tests/special_intel_gpu/run_grpo_intel_gpu.sh — e2e GRPO (1-GPU and 2-GPU validated)
tests/special_intel_gpu/run_ppo_intel_gpu.sh — e2e PPO smoke test
tests/special_intel_gpu/run_sft__intel_gpu.sh — e2e SFT smoke test

Documentation (new files):

docs/intel_gpu_tutorial/intel_gpu_build_dockerfile_page.rst — Docker build guide, software stack table, host hardware check, full env var reference
docs/intel_gpu_tutorial/intel_gpu_quick_start.rst — environment check, feature support matrix with perf numbers, known limitations, GRPO/PPO/SFT examples

Test

Validated on Intel Arc Pro B60 (2×24 GB VRAM):

NUM_GPUS=1 bash tests/special_intel_gpu/run_grpo_intel_gpu.sh
# timing_s/step: 51.2,  throughput: 148.2 tok/s  ✓

NUM_GPUS=2 bash tests/special_intel_gpu/run_grpo_intel_gpu.sh
# timing_s/step: 93.0,  completes without OOM     ✓

CI: Intel GPU hardware not available in GitHub Actions. Tests are in tests/special_intel_gpu/ for manual validation.

Checklist Before Submitting

Read the Contribute Guide.
Applied pre-commit checks: all hooks pass on changed files.
Documentation added: docs/intel_gpu_tutorial/ with build guide and quick-start.
Tests added in tests/special_intel_gpu/ covering GRPO (1-GPU, 2-GPU), PPO, SFT.
AI assistance used (Claude). All changes reviewed and validated end-to-end on real Intel GPU hardware.
Not duplicating any existing open PR (no existing PR covers FSDP + vLLM direct Intel GPU integration).

Update `Dockerfile.stable.sglang` so the stable SGLang image builds on GB200/aarch64 by keeping the SGLang base image's ARM `flash-attn-4`, `sglang-kernel`, and `transformers` deps instead of reinstalling incompatible pins; x86 likely should be simplified the same way, but I left it mostly unchanged because I did not test x86. Validated container: `verlai/verl:sgl0512.aarch64.dev1` Sanity passed on a GB200 node: build succeeded, `sglang-kernel` includes SM100 kernels, CUDA matmul/Apex smoke/SGLang CLI all passed. More tests are welcome, especially x86 build validation. Co-authored-by: Codex <codex@openai.com>

…+ vLLM rollout) Adds end-to-end Intel XPU (Arc Pro / Data Center GPU Max) support for GRPO, PPO, and SFT training. Validated on 2× Intel Arc Pro B60 (24 GB, Battlemage, driver 26.09). No behavioral changes to existing CUDA/NPU code paths. New files: - docker/xpu/Dockerfile.xpu, requirements-xpu.txt - tests/special_xpu/ (run_grpo_xpu.sh, run_ppo_xpu.sh, run_sft_xpu.sh) - verl/utils/vllm/xpu_patches.py Core changes: - verl/utils/device.py: XPU detection, device name, nccl backend (xccl), ZE_AFFINITY_MASK keyword, is_support_ipc=False (torch-xpu-ops#1678) - verl/single_controller/ray/base.py, worker.py: IntelGPUAccelerator mapping - verl/utils/distributed.py, fsdp_utils.py: all_reduce_avg SUM/N workaround (torch-xpu-ops#3020), composite cpu:gloo,xpu:xccl backend - verl/workers/rollout/vllm_rollout/vllm_async_server.py: enable_sleep_mode=False, pop ONEAPI_DEVICE_SELECTOR before AsyncLLM init, distributed_executor_backend=uni - verl/utils/vllm/xpu_patches.py: idempotent patches for false OOM check (vllm#37149) and profiling assert (vllm#36720) - verl/utils/model.py: read attn_implementation from config, not hardcoded fa2 - verl/workers/rollout/replica.py: get_device_name() replaces hardcoded cuda/npu - verl/workers/rollout/vllm_rollout/bucketed_weight_transfer.py: hasattr guard - verl/workers/engine/fsdp/transformer_impl.py: register xpu in EngineRegistry - verl/trainer/constants_ppo.py: propagate ONEAPI_DEVICE_SELECTOR to Ray workers - verl/utils/attention_utils.py: pure-PyTorch SDPA for XPU (no flash_attn dep)

…PU count Fixes discovered during 4× Arc Pro B60 PPO validation: - requirements-xpu.txt: add trl<0.18 (PPO critic needs AutoModelForCausalLMWithValueHead) - run_ppo_xpu.sh: add +override_config.attn_implementation=sdpa for actor and critic (was only in GRPO script), add RAY_NUM_PRESTART_PYTHON_WORKERS=0, add enforce_eager/free_cache_engine/ref offload params, add ray_kwargs.num_gpus - run_grpo_xpu.sh + run_ppo_xpu.sh: add CCL_TOPO_ALGO=0 — ZE_AFFINITY_MASK renumbers worker devices to 0, causing oneCCL oversubscription detection to fail with >2 GPUs - Dockerfile.xpu: remove hardcoded ONEAPI_DEVICE_SELECTOR=level_zero:0,1; sitecustomize.py now auto-detects GPU count at Python startup Tested: 4-GPU PPO completes step 1/1 (176s), actor/loss=0.0036, critic/vf_loss=11.8

Removed upstream in the 'migrate Diffusion RL stack to verl-omni' refactor. Never used in replica.py — import-only stale reference.

After a forward pass on XPU, the SYCL runtime may remap flat_param.data to a different storage than _local_shard due to Level Zero buffer aliasing. The two data_ptr/id assertions in offload_fsdp_model_to_cpu are CUDA-specific invariants that do not hold on XPU. Guard them with `not is_xpu_available`. Fixes AssertionError crash in GRPO/PPO training with param_offload=True.

The conda 'lun' activate script sets ONEAPI_DEVICE_SELECTOR=level_zero:0,1. When propagated to Ray workers, oneDNN's SDPA kernel fails to find its OpenCL device (oneDNN uses OpenCL alongside Level Zero for sdpa primitives), crashing with "could not create a primitive". Fix get_ppo_ray_runtime_env() to propagate ZE_AFFINITY_MASK instead — the correct device restriction mechanism for Level Zero. Explicitly remove any ONEAPI_DEVICE_SELECTOR from the Ray runtime env so workers always have a clean selector state. Also update run_grpo_xpu.sh to: - unset ONEAPI_DEVICE_SELECTOR after conda activate - switch attn_implementation to eager (SDPA oneDNN path is broken on B60 driver 1.14.37435 without OpenCL userspace)

Level Zero maps XPU device memory into each process's CPU virtual address space (~2.2 TB VmPeak per process). Ray's memory monitor reads VmPeak and concludes the system is OOM during training-step peak, killing workers. Setting RAY_memory_monitor_refresh_ms=0 disables the monitor. Actual RAM usage stays well below available limits — the VmPeak spike is a driver artifact, not real allocation. Affects all XPU runs (1-GPU and 2-GPU) whenever FSDP workers and vLLM EngineCore subprocess coexist in the same Ray job. Co-authored-by: Claude

…ming Two pre-commit issues found when preparing the PR: 1. verl/workers/rollout/replica.py init_standalone() references `use_gpu` before assignment. Add `use_gpu = self.rollout_worker_use_gpu()` to match the pattern in init_colocated(). (ruff F821) 2. requirements-xpu.txt comment used "veRL" instead of "verl". (naming hook) Co-authored-by: Claude

Two new RST pages under docs/xpu_tutorial/: - xpu_build_dockerfile_page.rst: software stack table (compute-runtime, IGC, oneCCL, PyTorch 2.11.0+xpu, vLLM 0.17.1), host hardware check (lspci / renderD* / render group), build + run instructions, full env var table with accurate explanations (ONEAPI_DEVICE_SELECTOR 3-layer suppression, RAY_memory_monitor_refresh_ms L0 VmPeak rationale). - xpu_quick_start.rst: environment check with expected output, feature support matrix with measured perf numbers (1-GPU: 51.2 s/step 148 tok/s, 2-GPU: 93 s/step), known limitations (L0 VA pressure, SGLang IPC, ONEAPI_DEVICE_SELECTOR), GRPO/PPO/SFT example workflow, manual launch snippet. All commands verified against actual test runs. Both pages wired into docs/index.rst Hardware Support toctree. docker/xpu/README.md corrected: GPU counts now reflect validated configs (1-GPU and 2-GPU GRPO confirmed; PPO/SFT note 4-GPU default, show 2-GPU override). Co-authored-by: Claude

requirements-xpu.txt and Dockerfile.xpu comments referenced the old torch==2.10.0+xpu pin. The actual installed wheel (and the docs) are 2.11.0+xpu, which is what vllm/requirements/xpu.txt resolves to with vLLM 0.17.x. Update the two comment strings to match reality. Co-authored-by: Claude Signed-off-by: Kah Lun Teoh <kahlun@github.com>

- Updated index.rst to include Intel GPU tutorial links. - Added intel_gpu_build_dockerfile_page.rst for building Intel GPU Docker images. - Created intel_gpu_quick_start.rst for quick-start instructions on Intel GPU usage. - Updated requirements-xpu.txt to reflect the new Dockerfile location for PyTorch with XPU support. - Introduced .env.intel_gpu.example for optional Intel GPU test runtime overrides. - Added docker_run_ppo.sh for running PPO tests in Docker with Intel GPU. - Created intel_gpu_env.sh for shared environment setup in Intel GPU tests. - Implemented run_grpo_intel_gpu.sh for end-to-end GRPO testing on Intel XPU. - Added run_ppo_intel_gpu.sh for end-to-end PPO testing on Intel XPU. - Introduced run_sft_intel_gpu.sh for SFT smoke testing on Intel XPU. - Updated existing XPU test scripts to utilize the new environment configuration for Intel GPU.

…accuracy