Skip to content

perf(h3): overlap the 2-GPU recipe rollout with engine concurrency 2 #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Rockdu merged 1 commit into from
Aug 21, 2026
Merged

perf(h3): overlap the 2-GPU recipe rollout with engine concurrency 2 #197

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion scripts/run_diffusion_grpo_h3_t2va_2gpu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
noise_level 0.7, 10 inference steps, SDE window size 2, flow_shift 12.0 /
audio_flow_shift 3.0, PickScore over 8 frames.

Engine concurrency is 2, not 1: at 1 the engine idles through the encode and handoff
of a finished sample before the next denoise starts, and 2 overlaps them. Higher does not
help -- the extra requests queue inside the engine and only stretch per-sample latency.

The router health-check window is widened well past its 30s x 3 default: H3 denoising
blocks the engine's uvicorn event loop for ~43s per sample, so /health cannot answer
while a request runs. A 16-sample eval keeps the loop busy for ~800s straight, which
Expand Down Expand Up @@ -156,7 +160,7 @@ def execute(args: ScriptArgs, prompt_dir: str) -> None:

sglang_args = (
"--use-miles-router "
"--sglang-server-concurrency 1 "
"--sglang-server-concurrency 2 "
"--sglang-tp-size 2 "
"--sglang-sp-degree 1 "
"--sglang-ulysses-degree 1 "
Expand Down
Loading