Repository navigation
fix: bound sliding KV re-RoPE to physical cache span - #792
Open
clumsylad21 wants to merge 1 commit into
Open
clumsylad21 wants to merge 1 commit into
clumsylad21 wants to merge 1 commit into
Conversation
Signed-off-by: Bhavyansh Sharma <66562679+clumsylad21@users.noreply.github.com>
Contributor
Author
Contributor
Author
|
Hii @jakmro, Did you get a chance to look at it?? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #777
Summary
This fixes an out-of-bounds access during KeyDiff KV-cache compaction on Metal sliding-window caches.
The crash reported in #777 occurs inside
rerope_recent_int8()after long-context rolling compaction. The underlying issue is that KeyDiff was treatingCacheHeader::current_seq_lenas the number of physically allocated cache rows for every layer.That assumption is valid for ordinary/non-sliding caches, but not for Metal sliding-window KV caches.
Root cause
Metal sliding KV caches use a ring buffer.
For these caches,
current_seq_lenis a logical, monotonically increasing cursor used to determine the next ring slot. It is therefore allowed to grow beyond the physical cache capacity.For example, with:
max_seq_len = 517the Metal cache may have:
This is valid: the 600 value is the logical position, while the recent KV rows are physically stored in a bounded ring.
The usable physical span of the sliding cache is:
which is 512 rows in this example.
During KeyDiff compaction,
Model::compress_kv_cache_keydiff()previously used:when re-RoPEing non-compressible/sliding key caches.
For a Metal sliding cache this could therefore call
rerope_recent_int8()with a range extending beyond the physical INT8/scales buffers.For the example above, KeyDiff could attempt to process rows up to 600 even though only 512 physical ring rows are valid.
This also matches the reported failure location:
rerope_recent_int8()eventually dequantizes an INT8 row and accesses its corresponding per-group scale, so an out-of-range row index can produce the observed invalid scale/data load.Importantly,
current_seq_lenitself should not be clamped or reset here. Metal uses it as the logical ring cursor, and changing it would alter future ring-slot selection.Fix
Add
rerope_physical_end()to distinguish logical cache length from the number of physical rows that may safely be accessed.For non-sliding caches it preserves the existing behavior:
For sliding caches it bounds the re-RoPE range to:
Model::compress_kv_cache_keydiff()now uses this physical end when re-RoPEing sliding-layer keys.This leaves the Metal logical cursor untouched while preventing re-RoPE from accessing rows outside the physical ring buffer.
Regression coverage
Added a graph-managed INT8 KV-cache regression test covering the mixed lifecycle involved in this bug:
The test specifically verifies the important case:
and exercises the INT8 SIMD/NEON re-RoPE path without accessing outside the physical cache.
Validated on GitHub Actions using the macOS ARM64 runner; the KV compression test suite passes with all 30 tests.