Skip to content

docs: describe what the release does, not how the kernels are written - #555

Merged
bong-water-water-bong merged 1 commit into
mainfrom
buildsecrecy-docs-method
Oct 24, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
buildsecrecy-docs-method

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

docs: describe what the release does, not how the kernels are written

The performance table and the two method paragraphs around it described our own kernel work in the vocabulary
it is written in — cache types and head groupings per case, the plan shape picked per token count, the
attention split and combine tree, tiled window sizes. Those are results pages and user documentation, so they
now say what changed and what it bought, in the terms a reader needs to decide whether to use the release:

  • docs/releases.md: "speculative verify pass rework", "batched MoE decode", "decode attention for gated
    attention at long context" — every number kept as measured, with the thermal label untouched.
  • docs/serve.md (--precise): the reason a follow-up turn is not bit-identical stays (a different token
    count gets a different plan, so the last bits differ and a near-tie can flip); the plan's internals are gone.
  • docs/hw-gate.md: the long-context decode-attention check keeps what it covers and what it caught, without
    the per-case parameter list or the private reference.

No reason given for the change, on purpose: it is housekeeping, and the numbers that sell the release are
untouched.

🤖 Generated with Claude Code

The performance table and the two method paragraphs around it described our own kernel work in the vocabulary
it is written in — cache types and head groupings per case, the plan shape picked per token count, the
attention split and combine tree, tiled window sizes. Those are results pages and user documentation, so they
now say what changed and what it bought, in the terms a reader needs to decide whether to use the release:

- `docs/releases.md`: "speculative verify pass rework", "batched MoE decode", "decode attention for gated
  attention at long context" — every number kept as measured, with the thermal label untouched.
- `docs/serve.md` (`--precise`): the reason a follow-up turn is not bit-identical stays (a different token
  count gets a different plan, so the last bits differ and a near-tie can flip); the plan's internals are gone.
- `docs/hw-gate.md`: the long-context decode-attention check keeps what it covers and what it caught, without
  the per-case parameter list or the private reference.

No reason given for the change, on purpose: it is housekeeping, and the numbers that sell the release are
untouched.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 377ff8b)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Overly Specific Implementation Details

The updated documentation still includes overly specific implementation details about kernel parameters and internal behaviors (e.g., "K q8_0 with V f16 (-ctk q8_0 alone) and with V q8_0, Hq/Hkv 16/8 and 32/8, n_kv 1031, 2048 and 4103"). While the PR description states that such details should be removed, this change still retains them, which contradicts the stated intent of the PR.

| fa-decode-q8 | the add-on's own long-context decode-attention check against a double-precision reference, quantised K with both cache value types and two head groupings, six runs in one hold | a mixed cache-type bug `test-backend-ops` cannot see: it has no `FLASH_ATTN_EXT` case with K and V of different types |
Inconsistent Performance Claims

The performance improvements listed in the release notes are inconsistent in their presentation. Some entries refer to specific PR numbers and performance gains, while others lack clear performance metrics or are vague in their descriptions (e.g., "batched MoE decode" lacks a specific performance gain). This inconsistency makes it harder for users to understand the actual impact of the changes.

| speculative verify pass rework | `#487` | 27B `--mtp` **+31–39 %** |
| batched MoE decode | `#521`, `#526` | pin `744bd713d968`: batching pays from five tokens per window; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) |
| decode attention for gated attention at long context | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** |
Vague Explanation of Bit-Identical Behavior

The explanation for why follow-up turns are not bit-identical has been simplified but still lacks clarity. The original text provided more detailed reasoning about kernel families, activation quantization, and other technical aspects. The new version omits these details, which may leave users confused about why bit-identical behavior cannot be achieved.

where a cold run prefills the conversation in 512-token ubatches - and HRX serves a different token count with a
different plan, so the last bits of the logits differ and a near-tie can flip. Measured on
2026-10-22, 10 greedy two-turn chats per model, follow-up against a cold run of the same conversation.

@bong-water-water-bong
bong-water-water-bong marked this pull request as ready for review October 24, 2026 11:41
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Review (orchestrator): numbers and labels kept, the method vocabulary gone from the three pages. Merging on green CI.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 377ff8b

@bong-water-water-bong
bong-water-water-bong merged commit 84444f8 into main Oct 24, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the buildsecrecy-docs-method branch October 24, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant