Skip to content

feat(xpu): Add Intel GPU support for veRL training - #2

Open
kahlun wants to merge 26 commits into
mainfrom
xpu/clean-pr2
Open

feat(xpu): Add Intel GPU support for veRL training#2
kahlun wants to merge 26 commits into
mainfrom
xpu/clean-pr2

Conversation

@kahlun

@kahlun kahlun commented Jun 4, 2026

Copy link
Copy Markdown
Owner

What does this PR do?

Adds direct Intel GPU (Pytorch xpu variant) support to verl core for end-to-end GRPO, PPO, and SFT training using FSDP/FSDP2 + vLLM rollout (colocated mode).

Validated on: Intel Arc Pro B70 (Battlemage, 2×24 GB):

Validation (2× Arc Pro B70)

Test Result Performance
GRPO 1-GPU 172 tok/s
GRPO 2-GPU 41.6 tok/s
PPO 2-GPU 27.6 tok/s
SFT 2-GPU PASS

*2-GPU throughput with same 16-prompt batch (not scaled up); full throughput benefit requires larger batch size.

Model: Qwen2.5-0.5B-Instruct, dataset: GSM8K, 16 prompts/batch.


Changes

Distributed (verl/utils/distributed.py):

  • Composite backend "cpu:gloo,xpu:xccl" for Intel GPU process group init
  • all_reduce_avg(): SUM + divide workaround (oneCCL doesn't support ReduceOp.AVG)

FSDP (verl/utils/fsdp_utils.py):

  • set_force_sum_reduction_for_comms(True) for FSDP2 on Intel GPU

Ray resource mapping (verl/single_controller/ray/base.py):

  • Map device "xpu" → Ray resource "GPU" (Ray's IntelGPUAccelerator registers XPU under the "GPU" key, same as CUDA)

Engine registry (verl/workers/engine/fsdp/transformer_impl.py):

  • Add "xpu" to @EngineRegistry.register(device=[...]) for language and value model engines

vLLM rollout (verl/workers/rollout/vllm_rollout/vllm_async_server.py):

  • Force uni executor for TP=1 on Intel GPU(avoids Level Zero multi-context OOM)
  • Pop ONEAPI_DEVICE_SELECTOR before vLLM's EngineCore subprocess spawns (prevents oneDNN OpenCL init crash)
  • Apply Intel GPU-specific vLLM patches before engine init

vLLM patches (verl/utils/vllm/intel_gpu_patches.py, new file):

  • Skip false-OOM preflight check on Intel GPU (vllm.v1.worker.utils.request_memory)
  • Wrap profiling assert in GPUWorker.determine_num_available_blocks

Runtime env (verl/trainer/constants_ppo.py):

  • Propagate ZE_AFFINITY_MASK to Ray workers; explicitly drop ONEAPI_DEVICE_SELECTOR

Worker (verl/single_controller/base/worker.py):

  • Intel GPU device init and get_device_name() routing

Replica (verl/workers/rollout/replica.py):

  • Use get_device_name() in init_colocated / init_standalone (covers Intel GPU, CUDA, NPU generically)

Docker + tests (new files):

  • docker/intel_gpu/Dockerfile.intel_gpu — reproducible build (oneAPI 2025.3, compute-runtime 26.09, torch 2.11.0+xpu, vLLM 0.17.1)
  • docker/intel_gpu/README.md — setup guide
  • requirements-intel-gpu.txt — intel GPU-specific pip deps
  • tests/special_intel_gpu/run_grpo_intel_gpu.sh — e2e GRPO (1-GPU and 2-GPU validated)
  • tests/special_intel_gpu/run_ppo_intel_gpu.sh — e2e PPO smoke test
  • tests/special_intel_gpu/run_sft__intel_gpu.sh — e2e SFT smoke test

Documentation (new files):

  • docs/intel_gpu_tutorial/intel_gpu_build_dockerfile_page.rst — Docker build guide, software stack table, host hardware check, full env var reference
  • docs/intel_gpu_tutorial/intel_gpu_quick_start.rst — environment check, feature support matrix with perf numbers, known limitations, GRPO/PPO/SFT examples

All tests passing with Qwen2.5-0.5B-Instruct on GSM8K.

NUM_GPUS=1 bash tests/special_intel_gpu/run_grpo_intel_gpu.sh
# timing_s/step: 51.2,  throughput: 148.2 tok/s  ✓

NUM_GPUS=2 bash tests/special_intel_gpu/run_grpo_intel_gpu.sh
# timing_s/step: 93.0,  completes without OOM     ✓

CI: Intel GPU hardware not available in GitHub Actions. Tests are in tests/special_intel_gpu/ for manual validation.


Checklist Before Submitting

  • Read the Contribute Guide.
  • Applied pre-commit checks: all hooks pass on changed files.
  • Documentation added: docs/intel_gpu_tutorial/ with build guide and quick-start.
  • Tests added in tests/special_intel_gpu/ covering GRPO (1-GPU, 2-GPU), PPO, SFT.
  • AI assistance used (Claude). All changes reviewed and validated end-to-end on real Intel GPU hardware.
  • Not duplicating any existing open PR (no existing PR covers FSDP + vLLM direct Intel GPU integration).

kahlun and others added 22 commits July 7, 2026 19:23
Add comprehensive Intel GPU (Arc/Arc Pro B-series) support for veRL:

- Docker image with Intel compute-runtime 26.09, IGC 2.30.1, oneCCL 2021.15
- FSDP + vLLM rollout support for GRPO, PPO, and SFT algorithms
- Runtime patches for vLLM XPU platform and Intel GPU device detection
- Intel GPU-specific test scripts and environment configuration
- Documentation for Docker build and getting started guide

Validated on 2× Intel Arc Pro B60 (Battlemage):
- GRPO: 41.6 tok/s (2-GPU), 172 tok/s (1-GPU)
- PPO: 27.6 tok/s (2-GPU GAE with critic)
- SFT: Multi-GPU FSDP training with checkpointing

Co-authored-by: Claude <claude@anthropic.com>
Signed-off-by: Kah Lun Teoh <kah.lun.teoh@intel.com>
…ne environment setup, and remove deprecated oneCCL workarounds
…ated environment scripts, and streamline memory reporting
kahlun added 4 commits August 7, 2026 01:51
# Conflicts:
#	verl/workers/rollout/replica.py
#	verl/workers/rollout/sglang_rollout/async_sglang_server.py
#	verl/workers/rollout/vllm_rollout/bucketed_weight_transfer.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant