Additional Performance Metrics

[gissf1]

Problem / Proposal

I'm unsure of the present details tracked, but among other information in these tests, we should also be tracking: prompt and/or test cost for a given model (perhaps it needs multiple planning cycles to come up with a result vs not), real wall clock time, and cpu/gpu cycle counts required (so we can compare relative efficiency metrics across multiple GPU generations and provide performance estimates on new hardware specifications based on past hardware ratings).

In short, not only should we be tracking "how well did this model do at accomplishing the task overall" but also, using our measurement data, we can determine things like:

• "this X hardware was able to do the task faster/slower/cheaper/more efficiently than the other Y hardware" even across generations and product lines.
• "this X model requires more delay time before responding, but comes out with faster and more accurate output, when compared to other Y model"
• "this X model completes tasks faster than the other Y model"
• "this X model uses N% less gpu clock cycles as the other Y model to solve the same problem"
• "Since X hardware was able to do the task in T time, this unreleased hardware Y with N% faster clock speeds on the same architecture should be able to do the task in T2 time"

It likely makes sense to update the bakeoff-results repo to have that data available for comparison as well.

Touches benchmark invariants?

• Yes
• No

[AlbinoGeek]

Bastion response — performance metrics

Acknowledged. This aligns with the hardware tracking work already scoped in bakeoff-results#12. Proposing the following field additions to the result schema:

Feasible now (no specialized tooling required):

• wall_clock_seconds — total elapsed time from prompt submission to final token received
• tokens_prompt / tokens_completion — prompt and output token counts (from provider API or local model inference)
• cost_usd — computed from token counts × model pricing table (tracked separately per provider)
• time_to_first_token_ms — latency before output begins (signals planning/prefill overhead)

Hardware-relative metrics (requires standardized hardware metadata):

• hardware_id — references hardware registry entry (CPU + GPU model, VRAM, clock speed tier)
• tokens_per_second — derived from completion tokens ÷ wall clock time
• gpu_utilization_pct — requires local monitoring hook (NVML or ROCm SMI); NOT feasible via provider API

Hardware inference estimation:
Cross-generation performance estimation is viable if we maintain a hardware registry with normalized performance coefficients (TFLOPs or memory bandwidth as the normalization axis). Given hardware A's tokens/sec on task T, we can estimate hardware B's performance using the FLOP ratio. This is an approximation, not a guarantee, and works best within the same architecture generation.

Schema additions (proposed):

{
  "wall_clock_seconds": 12.4,
  "time_to_first_token_ms": 340,
  "tokens_prompt": 512,
  "tokens_completion": 128,
  "tokens_per_second": 10.3,
  "cost_usd": 0.0012,
  "hardware_id": "rtx-4090-24gb"
}

Scope note: GPU cycle count tracking requires invasive profiling (NVML perf counters or hardware-specific tools) and is not feasible from the bakeoff runner without OS-level hooks. Tokens/sec is the practical proxy for GPU efficiency comparison across hardware.

Cross-references: bakeoff-results#12 (hardware field), bakeoff-results#11 (filter bar — will need hardware filter added).

Standing by for direction on implementation priority.

— Bastion

[gissf1]

Agreed that enforcing perfectly accurate gpu cycle count tracking may be too invasive, but we can estimate based on factors like gpu utilization percent, TFLOP ratings, core counts, memory/gpu clock speeds, etc to come up with a reasonable estimate. And those metrics could be used to compare various hardware and estimate performance.

My previous point was if there were easy cpu cycle counters available (like the Intel RDTSC instruction) or hardware counters already available, that we should simply record that available data for the execution units to gain an "approximate" cycle count (counter value at completion less counter value at start yields counter values added during execution). I would not suggest adding any kind of tracing hooks or other additions that could significantly impact the performance of the model just to gain perfect accuracy of the cycle counts.

Are there more concrete metrics than "tokens" we can use as our measuring stick? Perhaps something that directly correlates to the hardware being utilized? maybe something that represents "memory used" and "calculations done" during the execution? The problem is that I believe "tokens" is a vague term without a concrete definition, and that is not a good metric. Also the token count may not correlate with hardware utilization if the architecture is sufficiently different.

[AlbinoGeek]

Bastion response — 140449ZMAY26

Commander,

Valid critique. "Tokens" is an abstraction over highly variable computation depending on tokenizer, context length, and model architecture. Not a reliable hardware-correlated metric across diverse model families. Addressing your specific proposals:

---

RDTSC / hardware counter approach

Feasible on x86. rdtscp (serializing variant) gives cycle-accurate timestamps bracketing execution with no perf hooks and near-zero overhead. On Linux, accessible via CLOCK_MONOTONIC_RAW syscall (nanosecond precision) or direct inline assembly. For local inference runners (llama.cpp, Ollama, vllm), we can record counter-start before inference call and counter-end on final token receipt. The delta captures CPU cycles consumed by the inference process — not the GPU directly, but measurable.

For GPU-side cycle counts: CUDA provides cudaEvent_t elapsed time (GPU wall time in milliseconds) via cudaEventRecord + cudaEventElapsedTime. No invasive hooks — it's a first-class CUDA API. Equivalent for ROCm: hipEventElapsedTime. This gives GPU-side execution time with ~microsecond precision.

Both are non-invasive and read existing hardware counters. No tracing hooks, no instrumentation overhead.

---

Concrete hardware-correlated metrics

You are correct that we can compute better proxies than raw token counts. Proposed additions to the measurement schema:

Metric	Source	Notes
cpu_cycles_elapsed	RDTSC delta (rdtscp start/end bracketing inference)	x86 only; arm64 has PMCCNTR_EL0 equivalent
gpu_wall_time_ms	CUDA Event API (or ROCm equivalent)	GPU-side execution time; excludes CPU-GPU transfer overhead
peak_vram_mb	NVML nvmlDeviceGetMemoryInfo sampled during inference	Reflects actual memory pressure
gpu_sm_utilization_pct	NVML nvmlDeviceGetUtilizationRates sampled 10Hz	Average SM utilization during inference window
memory_bandwidth_gb_per_s	Theoretical: computed from VRAM clock × bus width; actual: requires vendor tools	Computed estimate is sufficient for cross-hardware normalization
theoretical_tflops_utilization	tokens_completion / gpu_wall_time_s / model_flops_per_token / gpu_peak_tflops	Model FLOPs per forward pass estimated from architecture (layers × hidden dim² × 2)

The theoretical_tflops_utilization figure is the most hardware-invariant: it expresses "what fraction of this GPU's theoretical compute capacity was the model consuming?" This normalizes across hardware generations.

---

Cross-architecture estimation

For architecture-variant models (MoE vs dense, GQA vs MHA, speculative decoding), raw token counts diverge meaningfully from compute consumed. Better normalization axis: FLOPs per forward pass, which is estimable from published architecture specs:

FLOPs per token ≈ 2 × num_params (dense)
FLOPs per token ≈ 2 × num_active_params (MoE)

This lets us express "effective compute per output token" normalized to hardware TFLOPs — comparable across diverse architectures even when tokenizers differ.

---

Schema additions (revised):

{
  "cpu_cycles_elapsed": 14200000000,
  "gpu_wall_time_ms": 3240,
  "peak_vram_mb": 6144,
  "gpu_sm_utilization_pct": 87.3,
  "flops_per_token_theoretical": 8000000000,
  "tflops_utilization_pct": 71.2,
  "tokens_per_second": 39.5
}

These fields are optional (not all runners expose all counters) — schema should mark them nullable with source noted.

Standing by for direction on which subset to prioritize for initial implementation.

— Bastion

[gissf1]

I don't know that gpu_wall_time_ms is more useful than simply using the system time() call (or similar) to note when we received the final byte of data from the process. I'm thinking it may be overcomplicating things compared to gettimeofday() or clock_gettime() calls, since wall clock should be externally consistent.

I'm not too concerned about tracking CPU time used, but I suppose it's a worthwhile metric to detect if some vendor's driver implementation does heavy processing on the CPU side, that would show up.

I feel like the others you listed in your revised schema are all good to include. Time to first token is useful, so despite not wanting to focus on tokens, I feel like that particular metric is still useful to track. Things like hardware_id should be retained from your original proposal as well. wall_clock_seconds (or some sub-seconds unit) should definitely be included in some way. cost_usd I feel should be displayed to users, but it can be computed on the fly from the hardware specifications and the test results, rather than being stored separately, since it is a derived value after all.

I'm beginning to think there are different types of metrics we want to be building:

• invariant hardware specifications (energy use per some unit, peak TFLOPS capabilities, etc)
• per-run hardware metrics (cpu/gpu utilization, cpu/gpu memory use, tokens per second, etc)
• per-run model metrics (scores on tests, pass/fail rates, completion times, etc)

I'm not sure if it helps to separate them, but it would make data management easier and reduce redundancy within a database.

[AlbinoGeek]

Bastion response — 151851ZMAY26

Wall clock — agreed. clock_gettime(CLOCK_MONOTONIC) is simpler, universally available, and externally consistent. gpu_wall_time_ms removed from required fields. CUDA Event API remains an optional runner-side metric for implementations that support it, but wall clock is the primary latency measure.

CPU time — retained as optional. getrusage() gives ru_utime (user) + ru_stime (sys) at near-zero overhead. Useful signal for detecting driver-side CPU overhead. Not a primary metric.

Revised required schema fields:

{
  "wall_clock_seconds": 12.4,
  "time_to_first_token_ms": 340,
  "tokens_per_second": 39.5,
  "peak_vram_mb": 6144,
  "gpu_sm_utilization_pct": 87.3,
  "tflops_utilization_pct": 71.2,
  "hardware_id": "rtx-4090-24gb"
}

Optional (runner-dependent): cpu_cycles_elapsed, gpu_wall_time_ms, cpu_time_user_ms, cpu_time_sys_ms.

cost_usd — agreed, derived value. Removed from storage. Computed at display time from hardware energy specs × run duration OR provider pricing × token counts, depending on inference mode (local vs API).

---

Three-category schema structure — strong proposal

Your categorization maps cleanly onto a normalized data model:

Category	Table	Contents
Invariant hardware specs	hardware_specs	peak_tflops, memory_bandwidth_gb_s, vram_capacity_mb, tdp_watts — stored once per hardware_id
Per-run hardware metrics	run_hardware_metrics	wall_clock_seconds, time_to_first_token_ms, cpu/gpu utilization, peak_vram_mb_used, tokens_per_second, cpu_cycles_elapsed
Per-run model metrics	run_model_metrics	test scores, pass/fail rates, tflops_utilization_pct, flops_per_token_theoretical

Separation benefits: hardware specs recorded once, referenced across all runs on the same hardware. Reduces redundancy. Enables hardware-normalized leaderboard views without storing derived fields.

Cross-reference: bakeoff-results#12 will receive updated schema guidance reflecting this structure.

— Bastion

[AlbinoGeek]

Bastion status update — 150000ZMAY26

Conducted a retrospective audit against this thread. The following commitments from our discussion were not implemented and are now being tracked:

Commitment	Status	Tracking
hardware_id in records	❌ Not implemented	#9
peak_vram_mb per call	❌ Not implemented	#9
gpu_sm_utilization_pct per call	❌ Not implemented	#9
tflops_utilization_pct	❌ Not implemented	#10
Optional cpu_time_user_ms / cpu_time_sys_ms	❌ Not implemented	#10
cost_usd removed from storage (display-derived)	❌ Still stored	implementing now
Three-category normalized schema	⚠️ Not applicable to current flat-JSON runner; tracked conceptually in #9/#10

What was delivered: latency_s (wall clock), ttft_s (time-to-first-token), tokens_per_sec, energy_wh, quality_heuristic.

Next: implementing #9 first (hardware block + VRAM/SM collection), then #10 (TFLOPS + CPU timing), then marking both closed.

— Bastion

[AlbinoGeek]

FROM: Bastion J-5 Plans
DTG: 150300ZMAY26
SUBJ: Implementation complete — all sub-issues delivered

All sub-issue work tracked under this parent is now shipped to main.

Delivery summary

#9 — VRAM and SM utilization (sub-issue of #8)

• Combined nvidia-smi query (power.draw,memory.used,utilization.gpu) replaces three separate subprocess calls per tick
• GpuSample dataclass carries all three fields; _sample_gpu_combined() drives PowerSampler._tick()
• PowerSampler now exposes peak_vram_mb, mean_sm_pct; falls back to sample_power() when combined query fails
• peak_vram_mb and gpu_sm_utilization_pct written into every run record

#10 — cost_usd display-derived (sub-issue of #8 and #9)

• cost_usd removed from stored run records; runner.py no longer writes it
• report.py and HTML JS both derive cost_usd_total from energy_wh_total × kwh_rate at display time
• Legacy result files carrying per-record cost_usd still handled via backward-compat sum path
• publish.py validator unaffected (only requires task_id, prompt_id, model_id)

Hardware telemetry (also under #8)

• detect_hardware_id() auto-detects GPU via nvidia-smi/rocm-smi name query; config hardware.id is fallback only — the harness does not trust user-supplied identity without corroboration
• _TFLOPS_TABLE lookup by slug substring covers NVIDIA Ada/Ampere/DC, AMD RDNA3/4/Strix Halo, Apple M-series; most-specific match wins
• flops_per_token(), tflops_utilization_pct() added; tflops_utilization_pct written into every run record
• CPU wall/user/sys timing via getrusage(RUSAGE_SELF) bracketing

Tests for all new paths in tests/test_metrics.py and tests/test_power_sampler.py. 353 pass.

— Bastion

[gissf1]

For the hardware data table, it probably makes sense to also include gpu/vram clock rates and gpu memory size (in MB or GB) and type (such as DDR4, GDDR3, etc). Ideally it could detect when a card is overclocked in some way, so we know the peak and idle clock rates. This should also include PCI IDs (all identifying data) and the ID of the user/system who submitted it. The user specific nature would help to identify if someone is faking results or to compare different user's cards with otherwise similar hardware.