Skip to content

[FEAT] Add optional prewarm before trace replay benchmarks - #39

Merged
VincentG1234 merged 3 commits into
mainfrom
FEAT/trace-replay-prewarm
Jul 21, 2026
Merged

VincentG1234 merged 3 commits into
mainfrom
FEAT/trace-replay-prewarm

Conversation

@VincentG1234

@VincentG1234 VincentG1234 commented Jul 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add optional benchmark.prewarm for guidellm_trace_replay: a short concurrent GuideLLM run before the replay benchmark so vLLM kernel autotuners (CUDA graphs, torch.compile, etc.) can settle before timed trace requests.

Why

Trace replay measures inter-arrival timing faithfully, but the first requests after vLLM startup can pay one-time kernel compilation and graph-capture costs. That skews early replay metrics unless the server is warmed up separately. benchmark.warmup does not apply to the replay profile, so this adds a dedicated pre-run phase.

What changed

  • auto_tune_vllm/benchmarks/config.py — benchmark.prewarm field with validation (duration, concurrency; trace replay only)
  • auto_tune_vllm/benchmarks/profiles.py — derive prompt/output token mean+stdev from local trace files; render_prewarm_args() for synthetic concurrent prewarm data
  • auto_tune_vllm/benchmarks/trace_replay.py — run prewarm subprocess before replay; fail the trial on prewarm error/timeout/cancellation
  • auto_tune_vllm/benchmarks/providers.py — optional log_path on _launch_subprocess() for separate prewarm logs
  • tests/benchmarks/test_guidellm_trace_replay.py — config validation, token-stat derivation, CLI args, failure/timeout behavior
  • docs/configuration.md — Prewarm section under trace replay
  • examples/study_config_trace_replay.yaml — commented prewarm example

How tested

  • ruff check .
  • pytest -v tests/ (108 passed)
  • Manual E2E (maintainer): auto-tune-vllm optimize with examples/study_config_trace_replay.yaml and benchmark.prewarm enabled

Risks / limitations

  • Prewarm adds wall time per trial (duration seconds + subprocess overhead).
  • Token statistics are auto-derived from the local trace file; for hf:// datasets or missing files, falls back to benchmark.prompt_tokens / benchmark.output_tokens with a warning.

Links

@VincentG1234
VincentG1234 force-pushed the FEAT/trace-replay-prewarm branch from 0df18b6 to ce9a54d Compare July 21, 2026 09:44
@VincentG1234
VincentG1234 merged commit b135411 into main Jul 21, 2026
7 checks passed
@VincentG1234
VincentG1234 deleted the FEAT/trace-replay-prewarm branch July 21, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant