[tang-vu]

Why this matters

Benchmarking a loaded model while another client generates tokens currently records the server-wide counter delta as if it belonged to the benchmark. A request that generated 64 tokens in two seconds can be reported and persisted as 400 tokens / two seconds (200 tok/s) because unrelated requests incremented the counters.

The root cause is treating aggregate Prometheus counters as a per-request measurement and mixing counters, response timings, usage, and even whitespace word counts. The invariant is that the token numerator and duration describe the same request and measurement basis.

Use a complete timings.predicted_n / timings.predicted_ms pair when supplied; otherwise use the request's integer usage.completion_tokens over its measured wall duration. Return the basis in method. Incomplete timings never mix with a different token count. A completion without an actual valid request token count returns 502 and writes no performance samples. Remove the now-unused counter scraper and word-count estimator.

Overlap check

Searched all open/closed upstream PR titles, bodies, and changes to routers/models.py, plus benchmark, comparison, latency, counters and request measurements. #1194 introduced the performance oracle. #2471 validates model introspection before benchmarking; it does not change measurement. #3843 compares catalog models before download. #4064 overlaps independent usage scrapes. This PR fixes benchmark attribution and preserves discovery, activation, authentication and the existing performance storage boundary.

Validation

• Eleven new authenticated HTTP regression cases failed before implementation. They exercise a 400-token server delta during a 64-token request, complete/incomplete timing pairs, usage-only runtimes, no token count, booleans/fractions and malformed response shapes. Assertions cover the HTTP receipt and performance writer, plus absence of metric scrapes.
• pytest -q tests/test_benchmark_measurement.py tests/test_models.py tests/test_performance_oracle.py — 169 passed, Python 3.11.16; one dependency deprecation warning.
• Scoped pre-commit hooks and git diff --check passed.

The shipped NVIDIA default is llama.cpp b9014. Its server contract documents request timing fields; /metrics documents aggregate generation counters. Tests use those response contracts and a controlled clock; they do not claim a measured GPU throughput improvement or live Lemonade validation.

Wall-time measurements include prompt processing, queueing and transport and are explicitly labelled; decode timings describe generation. Concurrent load can still slow the request itself. Previously saved samples are not rewritten, and this does not solve model replacement during an in-flight request. Revert restores the previous measurement behavior without a schema migration. Shared Python covers the API on Linux/macOS/Windows; executed evidence is Linux/WSL. Batch baseline gate limitations remain root/WSL fixtures and unchanged dummy private-key fixtures.