LLM Inference Optimisation Benchmark — Llama 3.1 8B RAG Pipeline

Reproducible benchmark comparing inference backends and quantisation levels for a production Retrieval-Augmented Generation pipeline running Llama 3.1 8B. Conducted on Lenovo ThinkSystem SR630 V4 (Intel Xeon with AMX) as part of the DialogXR safeguarding-AI project, which is published as a Lenovo Validated Design.

Why this matters

For on-premise and air-gapped enterprise deployments, you often cannot reach for a cloud GPU. The practical question becomes: how fast can an 8B-class model serve RAG on available CPU/accelerated hardware, and which backend and quantisation give the best user experience? This study answers that with measured numbers.

Headline findings

Finding	Result
Best responsiveness (first-token latency)	OpenVINO INT4 — 1.47s
Best generation throughput	IPEX BF16 — 17.27 tokens/sec
Fastest end-to-end RAG pipeline	OpenVINO INT4 — 21.28s
Impact of environment tuning alone	~88x speedup on the PyTorch FP32 baseline (1450s -> 21-44s)

Recommended configuration: OpenVINO INT4 offered the best all-round balance of pipeline speed, responsiveness, and throughput for this workload.

Key takeaways

1. Environment setup is free performance. Correct CPU affinity and thread environment (e.g. KMP_AFFINITY, OMP threads) gave roughly an 88x speedup on the baseline PyTorch FP32 run and ~2.1x on IPEX BF16 — before any model or quantisation change.
2. Quantisation choice is a UX decision. OpenVINO INT4 minimised first-token latency (what users feel as responsiveness); IPEX BF16 maximised sustained token throughput (what matters for long generations). The right choice depends on the interaction pattern.
3. You cannot tune what you do not measure. All figures are end-to-end RAG measurements (retrieval + generation), not synthetic token loops.

Results

Full data in data/results.csv. Regenerate charts with make charts.

Methodology

• Model: Llama 3.1 8B, served for a Retrieval-Augmented Generation workload.
• Hardware: Lenovo ThinkSystem SR630 V4, Intel Xeon (AMX-capable).
• Backends compared: PyTorch (baseline FP32), Intel IPEX (BF16, INT8), OpenVINO (FP32, FP16, INT8, INT4).
• Conditions: with and without CPU environment tuning ("Env" vs "No Env").
• Metrics: total end-to-end RAG time, first-token latency (TTFT), overall tokens/sec, and tokens/sec after the first token.

Reproducing

This regenerates the charts from the measured results in data/results.csv.

pip install -r requirements.txt
make charts

Note on scope

These numbers were measured on one specific hardware configuration for one model and workload. They are intended to illustrate the relative impact of backend, quantisation, and environment tuning, not as universal benchmarks. Your mileage will vary with hardware, model, and workload.

Related work

• DialogXR — the multi-agent RAG safeguarding platform this benchmark supports (published as a Lenovo Validated Design).
• CaseLens-VLM — multimodal document-intelligence RAG pipeline.