From a018395cdaee0ef66f49642e868ffd14eb703b14 Mon Sep 17 00:00:00 2001 From: Jake Stevens Date: Tue, 11 Aug 2026 07:29:03 -0700 Subject: [PATCH] Slice logits to the last position before argmax in parallel prefill Parallel prefill returns logits for the whole prompt, and the sampling step argmaxed that entire tensor before indexing the last row: next_token = torch.argmax(logits, dim=-1)[0, -1].item() For Llama 3.2 1B at a 1984-token prompt that scans [1, 1984, 128256] to obtain a single token -- seq_len times the necessary work. Slice first instead. Same result, and the cost stops scaling with prompt length. Measured on an AMD EPYC Genoa devserver, 16 cores, MKL/OMP=1: prompt dtype before after 1984 float32 267.89 ms 0.14 ms 1984 bfloat16 1150.24 ms 0.59 ms 8128 float32 1094.65 ms 0.15 ms 8128 bfloat16 4718.06 ms 0.59 ms It lands hardest on reduced precision: torch's CPU argmax is roughly 4x slower on bfloat16 than float32, so a bf16 model paid ~1.15 s of the 1.42 s it took to pick one token at a 1984-token prompt. This also distorts reported prefill throughput, because on_prompt_eval_end() is recorded after the sampling step. Splitting forward() from the argmax on Llama 3.2 1B showed the ExecuTorch forward gap between bf16 and fp32 was -19 ms (bf16 faster) at a 1984-token prompt while the argmax gap was +886 ms; at 8128 tokens, forward +856 ms against argmax +3651 ms. Nearly all of an apparent bf16 regression was this line. Only the parallel-prefill path is affected. The sequential-prefill fallback and the decode loop call argmax on single-position logits, where the two forms are equivalent, and the other call sites in this file already slice. --- optimum/executorch/modeling.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimum/executorch/modeling.py b/optimum/executorch/modeling.py index f119d94a..819cfa68 100644 --- a/optimum/executorch/modeling.py +++ b/optimum/executorch/modeling.py @@ -750,7 +750,8 @@ def generate( cache_position=torch.arange(len(prompt_tokens), dtype=torch.long, device=self.device), ) self.stats.on_sampling_end() - next_token = torch.argmax(logits, dim=-1)[0, -1].item() + # Slice first: prefill returns logits for every position, we need one. + next_token = torch.argmax(logits[:, -1, :], dim=-1).item() else: # Sequential prefill is preserved for backwards compatibility in order to run PTE generated w/o dynamic shapes. # TODO: We can remove this block once the executorch runtime supports `cache_position`.