llm-slo-probe

Blackbox SLI/SLO prober for OpenAI-compatible LLM serving. Measures TTFT/ITL/throughput/error-rate from the client side and grades them against an SLO spec.

Why (the reliability framing)

This tool applies the SRE golden-signals model to LLM serving, from the outside:

Golden signal	This tool's measurement
Latency	TTFT (time-to-first-token) and ITL (inter-token latency), p50/p95/p99, measured client-side
Traffic	Throughput in tokens/sec and requests/sec, sustained over the probe window
Errors	Hard errors (non-200 / transport failures) and cutoffs (finish_reason == "length", i.e. truncated generations), rolled into a success rate
Saturation	Deliberately out of scope. Saturation (GPU memory pressure, KV-cache occupancy, queue depth inside the serving engine) needs engine-internal metrics — vLLM/TGI/Triton exporters, GPU telemetry — not blackbox HTTP probing. This tool tells you what the client experiences; it cannot tell you why the engine is degraded. Pair it with an engine-side exporter for the full picture.

The SLO spec + error-budget math below is a direct application of the SRE error-budget model: you declare an objective (e.g. "99% of requests succeed"), the tool computes how much of that budget you've burned, and reports pass/fail per objective plus an overall verdict. That verdict is what should gate a release or page an on-call — not a raw latency number in isolation.

Install

pip install -e .

Requires Python 3.11+. One runtime dependency: httpx (for async SSE streaming). Everything else is stdlib — including the TOML parsing (tomllib) and the demo backend (http.server).

Usage

llm-slo-probe --url URL [--model MODEL] [--api-key KEY] [--prompt TEXT]
              [--concurrency N] [--duration SECONDS] [--timeout SECONDS]
              [--slo PATH] [--prom-out PATH]

Flag	Default	Meaning
--url	(required)	OpenAI-compatible /v1/chat/completions endpoint
--model	gpt-3.5-turbo	model name sent in the request body
--api-key	none	bearer token, if the endpoint needs one
--prompt	"Write one sentence about reliability."	prompt sent on every request
--concurrency	4	number of concurrent in-flight requests
--duration	30.0	probe window, in seconds
--timeout	30.0	per-request timeout, in seconds
--slo	none	path to a TOML SLO spec (see below); omit to skip grading
--prom-out	none	path to write a Prometheus textfile-collector-compatible .prom file

Exit code is 0 if all declared SLO objectives passed, 1 otherwise — safe to use as a CI/deploy gate.

Sample run

This is a real captured run against the bundled stdlib mock backend (not fabricated numbers). Mock started as a throwaway backend:

python -c "from llm_slo_probe import mock_server; import time; mock_server.serve('127.0.0.1', 8200, token_delay_s=0.01, n_tokens=40); time.sleep(600)" &
python -m llm_slo_probe --url http://127.0.0.1:8200/v1/chat/completions \
  --model mock --concurrency 8 --duration 15 --slo slo.example.toml --prom-out /tmp/llm_slo.prom

Output:

 REQUESTS  224   ok 224   req/s 14.6
 TTFT      p50 17ms   p95 20ms   p99 22ms
 ITL       p50 14ms   p95 15ms
 THRU      583 tok/s
 ERRORS    hard 0   cutoffs 0   success 100.0%
 SLO
   ttft_p95      <= 500ms     obs 20ms     PASS
   itl_p95       <= 50ms      obs 15ms     PASS
   success_rate  >= 99.0%     obs 100.0%   PASS   budget 100%
 RESULT    PASS

Note: this is against a loopback mock with a fixed 10ms per-token delay and 40 tokens/response — the numbers reflect the mock's synthetic timing plus local scheduling/network overhead, not a real model. They demonstrate the tool's mechanics (concurrency, percentiles, SLO grading, error budget), not real-world LLM serving latency. Point it at a real OpenAI-compatible endpoint (vLLM, TGI, Ollama, a hosted API) for numbers that mean something.

Real run — Qwen 2.5 0.5B on Apple M3 Pro (via Ollama)

A concurrency sweep against a real model (--slo = TTFT p95 ≤ 500ms, ITL p95 ≤ 50ms, success ≥ 99%), 18s per level:

concurrency	req/s	TTFT p50	TTFT p95	ITL p95	throughput	success	SLO
1	3.5	108ms	121ms	7ms	99 tok/s	100%	PASS
4	3.7	799ms	1096ms	11ms	104 tok/s	100%	FAIL
16	3.5	4149ms	4527ms	12ms	103 tok/s	100%	FAIL
64	3.9	14474ms	18374ms	13ms	107 tok/s	100%	FAIL

What the probe surfaces that a health check wouldn't:

• Throughput is flat (~100 tok/s) no matter the concurrency. Ollama serializes decode on the single Metal device — there's no continuous batching, so piling on load doesn't buy throughput.
• TTFT collapses ~150× (121ms → 18s) while ITL stays flat (7–13ms). The degradation is entirely queue wait, not generation speed. This is the golden-signals split in one table: latency isn't one number — saturation shows up in TTFT, and only in TTFT.
• Success rate stays 100% the whole time. Nothing errored. A 5xx-only monitor would call this fleet perfectly healthy while p95 first-token latency crossed 18 seconds. That gap is the entire reason this tool measures TTFT client-side.

The SLO passes at concurrency 1 and fails from 4 up — a concrete error-budget line, not a vibe. A batching server (vLLM) would scale throughput and hold TTFT far longer under the same load; this run is a baseline that makes that difference measurable.

SLO spec

# slo.example.toml
[slo]
ttft_p95_ms = 500
itl_p95_ms  = 50
success_rate = 0.99

• ttft_p95_ms — the 95th-percentile time-to-first-token must be at or below this many milliseconds. This is the signal users feel as "did it hang."
• itl_p95_ms — the 95th-percentile inter-token latency (gap between successive streamed tokens) must be at or below this. This is the signal users feel as "is it typing smoothly."
• success_rate — the fraction of requests that complete with a 2xx response and a non-truncated finish_reason (i.e. not cut off by a length limit) must be at or above this.

Each objective independently reports PASS/FAIL. success_rate additionally reports an error-budget percentage remaining: if your SLO allows 1% failures (success_rate = 0.99) and you observed 0.3% failures, you've burned 30% of your error budget and have 70% left (budget = max(0, 1 - observed_error / allowed_error) * 100). Latency objectives (TTFT/ITL) are graded pass/fail only — the observed p95 already is the budget check; there's no accumulating "burn" to report for a single point-in-time percentile the way there is for a rate over the window.

Prometheus output

--prom-out PATH writes a plain-text file in Prometheus exposition format:

llm_slo_ttft_seconds{quantile="0.5"} 0.0165115
llm_slo_ttft_seconds{quantile="0.95"} 0.0202766
llm_slo_ttft_seconds{quantile="0.99"} 0.0217465
llm_slo_itl_seconds{quantile="0.95"} 0.0153478
llm_slo_throughput_tokens_per_second 583.449
llm_slo_success_rate 1
llm_slo_hard_errors_total 0
llm_slo_cutoffs_total 0
llm_slo_objective_passed{name="ttft_p95"} 1
llm_slo_objective_passed{name="itl_p95"} 1
llm_slo_objective_passed{name="success_rate"} 1
llm_slo_error_budget_percent{name="success_rate"} 100

This is hand-written (no prometheus-client dependency — the format is simple enough not to justify one). Drop the output path under a directory scraped by the node_exporter textfile collector (e.g. --prom-out /var/lib/node_exporter/textfile_collector/llm_slo.prom) to get these as first-class Prometheus metrics without running a separate exporter process.

Honesty caveat

Two things about these numbers that matter more than the numbers themselves:

1. Token count is chunk-based, not tokenizer-based. Each SSE content chunk from the streaming response is counted as one token (1 chunk ≈ 1 token). This is an approximation — real tokenizers can pack more or less than one token per chunk depending on the serving engine's streaming granularity. Treat throughput numbers as "chunks/sec of observable progress," which is close enough to tokens/sec to be useful, not exact enough to cite in a paper.
2. All measurements are client-side. TTFT and ITL include this process's network round-trip, TLS handshake (if any), local event-loop scheduling, and whatever else sits between this machine and the endpoint — not just the model's internal generation time. That's deliberate: a blackbox prober should measure what a real client experiences, including the parts of the stack the serving engine doesn't control. But it means these numbers are not directly comparable to an engine's own internal (server-side) latency metrics, and will look worse than server-side numbers under network jitter or an underpowered probing host.

Roadmap

llm-slo-probe is the load-generation and measurement client for a larger project, reliable-llm-gateway-lab — a gateway that will sit in front of one or more LLM backends and use this tool's SLO grading to drive routing/fallback decisions and demonstrate error-budget-based release gating end to end.