Skip to content

Commit 79de1f5

Browse files
committed
model : fix k-pool indexer softmax gridDim.y overflow
Reshape the k-pool gate logits before softmax so n_new does not map to gridDim.y, which is capped at 65535 on CUDA. Fixes prompt processing abort at n_kv >= 262144 reported in ggml-org#27773. Assisted-by: Cursor
1 parent de25343 commit 79de1f5

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

src/models/glm5-next.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -617,7 +617,9 @@ ggml_tensor * llama_model_glm5_next::graph::build_kpool_select(
617617

618618
ggml_tensor * logits = ggml_add(ctx0, pg, layer.indexer_kpool_ape);
619619
logits = ggml_cont(ctx0, ggml_permute(ctx0, logits, 1, 0, 2, 3)); // [kpool, head_dim, n_new]
620+
logits = ggml_reshape_2d(ctx0, logits, kpool, n_embd_indexer * n_new);
620621
ggml_tensor * probs = ggml_soft_max(ctx0, logits);
622+
probs = ggml_reshape_3d(ctx0, probs, kpool, n_embd_indexer, n_new);
621623

622624
pk = ggml_cont(ctx0, ggml_permute(ctx0, pk, 1, 0, 2, 3));
623625
pooled_new = ggml_sum_rows(ctx0, ggml_mul(ctx0, probs, pk)); // [1, head_dim, n_new]

0 commit comments

Comments
 (0)