Skip to content

model : fix k-pool indexer softmax gridDim.y overflow - #28730

Closed
AIalliAI wants to merge 48 commits into
ggml-org:masterfrom
AIalliAI:fix/glm5-kpool-softmax-grid-y
Closed

model : fix k-pool indexer softmax gridDim.y overflow#28730
AIalliAI wants to merge 48 commits into
ggml-org:masterfrom
AIalliAI:fix/glm5-kpool-softmax-grid-y

Conversation

@AIalliAI

Copy link
Copy Markdown

Overview

Reshape k-pool gate logits before ggml_soft_max so n_new does not map to gridDim.y, which is capped at 65535 on CUDA. During k-pool indexer prefill at n_kv >= 262144, the gate softmax is shaped [kpool, head_dim, n_new] and trips the launch at kpool = 4.

Reshape to [kpool, head_dim * n_new] before softmax, reshape back after. Same pattern as the comment in qwen4exp.cpp.

Depends on #27773. Review the scoped 2-commit diff here:
timkhronos:GLM5.3-Flash...AIalliAI:fix/glm5-kpool-softmax-grid-y

Intended merge path: timkhronos#11 into GLM5.3-Flash, then #27773 to master. Do not merge this PR directly to master.

Additional information

Related: #27754, unslothai#214, #27901, #28175

Fork copies: timkhronos#11, AIalliAI#2

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - assisted implementation and review; I understand the change and can maintain it.

@github-actions github-actions Bot added model Model specific testing Everything test related mtmd Related to multimodal functionality (video/image/audio) conversion labels Sep 11, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

Hi @AIalliAI, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

  • Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.


Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@AIalliAI

Copy link
Copy Markdown
Author

Hi @AIalliAI, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.
  • Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

As I've replied on the other comments you made. Your instructions are conflicting and I will be combining all of these int one pr to support GLM5. @maintainers

@AIalliAI AIalliAI closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion model Model specific mtmd Related to multimodal functionality (video/image/audio) testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants