[VincentG1234]

Summary

Introduces a new guidellm_trace_replay benchmark type that runs GuideLLM in
kind=replay mode, replaying inter-arrival intervals from a JSONL/CSV/Parquet
trace file. Refactors GuideLLM CLI construction behind a small BenchmarkProfile
abstraction (concurrent vs. replay) so the existing concurrent path is unchanged.

Why

benchmark_type: "guidellm" only exercises the concurrent profile (fixed
streams=rate). Realistic tuning against production-like traffic needs replay
of recorded timestamps + token lengths, not a synthetic constant-concurrency
load. This PR wires GuideLLM's replay profile into auto-tune-vllm with a
dedicated provider, config schema, and baseline handling.

What changed

• auto_tune_vllm/benchmarks/profiles.py — new file: BenchmarkProfile,
  ConcurrentProfile, ReplayProfile, profile_from_dict,
  render_replay_data_cli.
• auto_tune_vllm/benchmarks/trace_replay.py — new
  GuideLLMTraceReplayBenchmark provider that reuses the shared base command
  and appends replay-specific --data / --data-loader args.
• auto_tune_vllm/benchmarks/providers.py — split _build_guidellm_command
  into _build_guidellm_base_command + _append_standard_data_args so the
  replay provider can share the base and swap the data section.
• auto_tune_vllm/benchmarks/config.py — add benchmark.profile field, add
  _normalize_benchmark_profile (single source of truth: benchmark_type →
  profile.kind), validate rate > 0, widen rate to Optional[float] for
  fractional time_scale support, and add profile-specific defaults via
  _apply_default_rate: 50 for concurrent profiles, 1.0 (real-time)
  for trace replay when rate is omitted.
• auto_tune_vllm/benchmarks/__init__.py — export new profile / provider
  symbols.
• auto_tune_vllm/core/config.py — new _default_baseline_concurrency helper
  so baselines default to [1] for trace replay (concurrency doesn't apply)
  and [int(rate)] otherwise.
• auto_tune_vllm/execution/trial_controller.py — dispatch
  guidellm_trace_replay → GuideLLMTraceReplayBenchmark.
• tests/benchmarks/test_guidellm_trace_replay.py — new unit tests covering
  config normalization, validation errors, CLI construction, fractional
  rate / time_scale precedence, Mooncake fields, data_samples, and
  replay default rate.
• tests/benchmarks/test_guidellm_command.py — add test for concurrent
  default rate=50 when omitted.
• docs/configuration.md — document benchmark_type: "guidellm_trace_replay",
  the benchmark.profile schema, replay-specific field table, rate-as-
  time_scale fallback, and profile-specific default rate values.
• examples/study_config_trace_replay.yaml — end-to-end example study using
  the new profile.
• examples/trace_replay/sample.jsonl — 247-row (~200 s) synthetic trace for
  the example.

How tested

• ruff check .
• pytest -v tests/benchmarks/ (32 passed)
• Manual E2E (maintainer): auto-tune-vllm optimize --study-config examples/study_config_trace_replay.yaml against a real vLLM server.

Risks / limitations

• Type change: benchmark.rate widens from int to Optional[float]
  to support fractional time_scale in replay. Concurrent studies that set
  rate explicitly are unaffected; omitted rate still defaults to 50
  for concurrent profiles and 1.0 for trace replay.

[Copilot]

Pull request overview

Adds a new GuideLLM trace replay benchmark mode to auto-tune-vllm, introducing a small “benchmark profile” abstraction so the existing concurrent GuideLLM path can remain unchanged while enabling production-like traffic replay from trace files.

Changes:

• Introduces BenchmarkProfile abstraction with ConcurrentProfile and ReplayProfile, plus a new GuideLLMTraceReplayBenchmark provider that appends replay-specific --data / --data-loader args.
• Updates benchmark config schema to support benchmark.profile, normalizes benchmark_type → profile kind, and widens benchmark.rate to Optional[float] with profile-specific defaults.
• Adds tests, documentation, and an end-to-end example config + sample trace dataset.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/benchmarks/test_guidellm_trace_replay.py	Adds unit tests for replay config normalization/validation and CLI construction.
tests/benchmarks/test_guidellm_command.py	Adds a test for concurrent default rate when omitted.
examples/trace_replay/sample.jsonl	Adds a synthetic JSONL trace used by the replay example.
examples/study_config_trace_replay.yaml	Adds an end-to-end study config example using guidellm_trace_replay.
docs/configuration.md	Documents the new benchmark type, profile schema, and replay-specific fields/semantics.
auto_tune_vllm/execution/trial_controller.py	Dispatches guidellm_trace_replay to the new provider.
auto_tune_vllm/core/config.py	Adjusts baseline concurrency defaults for replay vs concurrent benchmarks.
auto_tune_vllm/benchmarks/trace_replay.py	Implements the GuideLLMTraceReplayBenchmark provider.
auto_tune_vllm/benchmarks/providers.py	Refactors GuideLLM command construction into base + data-args appenders.
auto_tune_vllm/benchmarks/profiles.py	Adds profile classes and replay --data CLI rendering.
auto_tune_vllm/benchmarks/config.py	Adds benchmark.profile, profile normalization, and rate defaulting/validation.
auto_tune_vllm/benchmarks/init.py	Exports new profile/provider symbols.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.