Skip to content

fix: bound sliding KV re-RoPE to physical cache span - #792

Open
clumsylad21 wants to merge 1 commit into
cactus-compute:mainfrom
clumsylad21:fix/issue-777-keydiff-kv-crash
Open

clumsylad21 wants to merge 1 commit into
cactus-compute:mainfrom
clumsylad21:fix/issue-777-keydiff-kv-crash

Conversation

@clumsylad21

Copy link
Copy Markdown
Contributor

Fixes #777

Summary

This fixes an out-of-bounds access during KeyDiff KV-cache compaction on Metal sliding-window caches.

The crash reported in #777 occurs inside rerope_recent_int8() after long-context rolling compaction. The underlying issue is that KeyDiff was treating CacheHeader::current_seq_len as the number of physically allocated cache rows for every layer.

That assumption is valid for ordinary/non-sliding caches, but not for Metal sliding-window KV caches.

Root cause

Metal sliding KV caches use a ring buffer.

For these caches, current_seq_len is a logical, monotonically increasing cursor used to determine the next ring slot. It is therefore allowed to grow beyond the physical cache capacity.

For example, with:

  • sliding window = 512
  • sink = 4
  • max_seq_len = 517

the Metal cache may have:

current_seq_len = 600
max_seq_len     = 517

This is valid: the 600 value is the logical position, while the recent KV rows are physically stored in a bounded ring.

The usable physical span of the sliding cache is:

max_seq_len - sink_size - 1

which is 512 rows in this example.

During KeyDiff compaction, Model::compress_kv_cache_keydiff() previously used:

hi = current_seq_len;

when re-RoPEing non-compressible/sliding key caches.

For a Metal sliding cache this could therefore call rerope_recent_int8() with a range extending beyond the physical INT8/scales buffers.

For the example above, KeyDiff could attempt to process rows up to 600 even though only 512 physical ring rows are valid.

This also matches the reported failure location: rerope_recent_int8() eventually dequantizes an INT8 row and accesses its corresponding per-group scale, so an out-of-range row index can produce the observed invalid scale/data load.

Importantly, current_seq_len itself should not be clamped or reset here. Metal uses it as the logical ring cursor, and changing it would alter future ring-slot selection.

Fix

Add rerope_physical_end() to distinguish logical cache length from the number of physical rows that may safely be accessed.

For non-sliding caches it preserves the existing behavior:

physical_end = current_seq_len

For sliding caches it bounds the re-RoPE range to:

min(current_seq_len, max_seq_len - sink_size - 1)

Model::compress_kv_cache_keydiff() now uses this physical end when re-RoPEing sliding-layer keys.

This leaves the Metal logical cursor untouched while preventing re-RoPE from accessing rows outside the physical ring buffer.

Regression coverage

Added a graph-managed INT8 KV-cache regression test covering the mixed lifecycle involved in this bug:

  1. append KV rows to both global and sliding caches
  2. allow the Metal sliding cache logical cursor to grow beyond its physical capacity
  3. compact the global KeyDiff cache
  4. shrink the global cache
  5. re-RoPE the sliding INT8 cache
  6. repeat across multiple reset/compaction cycles

The test specifically verifies the important case:

logical current_seq_len = 600
physical sliding span   = 512
re-RoPE end             = 512

and exercises the INT8 SIMD/NEON re-RoPE path without accessing outside the physical cache.

Validated on GitHub Actions using the macOS ARM64 runner; the KV compression test suite passes with all 30 tests.

Signed-off-by: Bhavyansh Sharma <66562679+clumsylad21@users.noreply.github.com>
@clumsylad21

Copy link
Copy Markdown
Contributor Author

@ncylich @jakmro Please review this.

@clumsylad21

Copy link
Copy Markdown
Contributor Author

Hii @jakmro, Did you get a chance to look at it??

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Long-context KeyDiff KV-cache compaction crashes with EXC_BAD_ACCESS in cactus::kvcompress::rerope_recent_int8 (multi-turn session, ~900 decode tokens)

1 participant