Skip to content

Add native chunked prefill for transpiled LLM bundles - #659

Merged
HenryNdubuaku merged 12 commits into
v2from
v2-chunked-prefill
May 25, 2026
Merged

HenryNdubuaku merged 12 commits into
v2from
v2-chunked-prefill

Conversation

@ncylich

@ncylich ncylich commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds native chunked prefill support for transpiled multimodal LLM bundles and keeps the implementation aligned with the current v2 native C++ runtime path.

Runtime changes

  • Loads only the component graphs needed for the selected decode route.
  • Adds chunked text prefill through lm_encoder_text_chunk and decoder_prefill_chunk components.
  • Copies compatible prefill cache state into step decode components before decode.
  • Supports direct decoder-step and full-context text decode routes for bundles without the full chunked path.
  • Preserves cache metadata on reset and shifts KV cache rows when thinking-token ranges are removed.
  • Caches CACTUS_KV_CACHE_FP16 lookup after startup so cache mode stays stable during a process.

Transpiler changes

  • Emits chunk prefill components for Gemma4, Qwen3-VL, and LFM2-VL pipelines.
  • Lowers shared internal KV-cache attention paths for prefill/query reuse.
  • Adds Qwen3-VL model-type/profile handling and safer Qwen layer-type mask selection.
  • Preserves Gemma4 vision projection precision and adjusts Gemma4 audio bias handling.
  • Uses 128-token chunk defaults for the supported multimodal profiles.

Build and platform changes

  • Serializes local Cactus build directory updates to avoid overlapping build races.
  • Improves Android NDK discovery for SDK and Homebrew installs.
  • Respects the current Android CPU affinity mask when pinning worker threads.

Tests

  • ./cactus/build.sh
  • PYTHONPATH=python ./venv/bin/python -m pytest python/tests/test_cli_transpile_defaults.py -q
  • PYTHONPATH=python ../cactus/venv/bin/python -m pytest python/tests/test_cli_transpile_defaults.py -q
  • Earlier validation on the same patch set included the focused transpiler suite, graph tests, and kernel tests.

Implement native chunked prefill support for transpiled multimodal LLM bundles and align the generated component pipelines with the C++ runtime loader.

Runtime changes:
- Load only the required component graphs for each supported decode route.
- Add chunked text prefill routing through lm_encoder_text_chunk and decoder_prefill_chunk components.
- Copy chunked prefill cache state into step decode components when compatible.
- Support direct decoder-step and full-context text decode routes for bundles that do not expose the full chunked path.
- Preserve cache metadata on reset and shift cache rows when thinking-token ranges are removed.
- Keep FP16 KV cache configuration stable after process startup.

Transpiler changes:
- Emit chunk prefill components for Gemma4, Qwen3-VL, and LFM2-VL component pipelines.
- Lower shared internal KV-cache attention paths for prefill/query reuse.
- Add Qwen3-VL model-type/profile handling and safer Qwen layer-type mask selection.
- Preserve Gemma4 vision projection precision and adjust Gemma4 audio bias handling.
- Prefer 128-token chunk defaults for supported multimodal profiles.

Build and platform changes:
- Make local Cactus builds less racy by serializing build directory updates.
- Improve Android NDK discovery for Homebrew and SDK installs.
- Respect existing Android CPU affinity masks when pinning worker threads.

Tests:
- Update transpile default expectations for multimodal component pipelines.
- Update graph serialization tests for persisted node ids.
- Add KV-cache full-window eviction coverage.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the v2-chunked-prefill branch from f9766c1 to ef08202 Compare May 20, 2026 19:42
ncylich added 2 commits May 20, 2026 17:52
Add dynamic Gemma4 media merge for prompts with audio so runtime prefill uses the actual prompt token layout instead of a static captured lm_encoder merge plan. This keeps image-only and non-audio paths on the existing chunked path while routing Gemma4 audio/image token rows through lm_encoder_media_step before decoder_prefill_chunk.

Fix Gemma4 audio conversion policy by keeping audio bias tensors in FP16, and preserve the vision/runtime fixes needed for native Gemma4 image quality. Add local integration probes and investigation docs covering Gemma image/audio/mixed, Qwen image, Parakeet file/PCM, and smoke validation.

Also ignore generated local integration and investigation outputs while preserving benchmark source/results and reproducible test harnesses.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Release graph temporaries after their final use while retaining declared component outputs and cache state buffers.

Unload transient media, encoder, and chunk-prefill graphs after copying their outputs so Gemma4 image/audio requests do not keep large intermediate buffers resident.

Validation: focused Gemma4, Qwen3-VL, and LFM2-VL integration checks pass locally.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@justinl66
justinl66 force-pushed the v2-chunked-prefill branch from f877825 to 68a422e Compare May 21, 2026 03:42
justinl66 and others added 8 commits May 20, 2026 21:29
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Add a first-token path that can sample directly after cache priming, track chunked prefill cache-copy and tail metrics, and allow sufficiently large tails to run as padded chunks.

When padded chunks are used for first-token sampling, rewind the copied decode cache to the final real prompt token and run that final token through the step decoder so logits are taken from real context instead of pad context. Keep LFM2-VL on scalar tails because its recurrent cached state is not safe to rewind after executing padded tokens.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the v2-chunked-prefill branch from 71b9b8b to c044fbc Compare May 22, 2026 22:40
@HenryNdubuaku
HenryNdubuaku merged commit 4e592ff into v2 May 25, 2026
5 checks passed
ncylich added a commit that referenced this pull request Jun 14, 2026
The gemma4 FP16 weight scaler keyed the 1/16 down-scaling on the filename
embed_vision_embedding.weights, but the vision embedding projection is saved
as embed_vision_proj.weights, so it was stored unscaled. The gemma4 runtime
operates in a 1/16-scaled residual space (token embeddings, norms, etc. are
all down-scaled to keep activations in fp16 range), so an unscaled projection
left image soft-tokens ~16x too large in the residual stream. This drowned out
the per-layer refinement of image features and degraded multimodal reasoning:
gemma-4-e4b-it confidently misidentified a monkey as a toucan, while e2b was
unaffected only because its projection was converted before the filename was
dropped from the scale set (regression in #659). Add the real filename so the
projection is down-scaled like every other gemma4 embedding-space weight.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
jakmro pushed a commit that referenced this pull request Jun 14, 2026
* cactus-graph: preserve cache-node descriptor across steal_cache_buffer

Any second chunked prefill on a gated-deltanet bundle threw a state rank
error because moving the cache buffer into the decoder graph left the
prefill node's BufferDesc empty. Restore the source to a fresh unallocated
descriptor with its original shape and precision so the next execute
re-initializes the state exactly as on first run.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-engine: clear tool-constraint bias on deactivation

Every generation after a force_tools turn on the same handle degenerated
into <|tool_call> loops on gemma models, surviving cactus_reset. The
deactivating init left the previously computed force bias in place while
update() no-ops when inactive, so decode kept force-picking the
call-start token. Clear the bias in init and drop the redundant reset in
clear_tool_constraints that recomputed it just before deactivation.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* engine+python: golden chat-template render tests vs HF reference

Adds a cactus_render_prompt FFI call that exposes the fully rendered
chat prompt, plus a ctypes wrapper and export. A golden pytest suite
checks engine renders and tokenizations against the HF reference chat
templates for gemma-4, qwen3, and lfm2 across system, tool-call,
parallel-tool-call, and thinking conversations, plus the gemma-4 image
expansion. Aligns the gemma-4 formatter with the reference template:
<|think|> gains its trailing newline and the image block drops its
surrounding blank lines.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* transpiler: stable cache layer keys across components

The internal KV-cache layer key fell through to the node id whenever
attention_layer_index was 0, and decoder_media_step graphs never
received attention_layer_index hints at all, so components of one
bundle disagreed on layer keys and the engine's positional
cache-state pairing failed at load time with
"prefill and step cache layer mismatch".

Derive the key with explicit None checks before falling back to the
node id, and include decoder_media_step in the attention-hint
assignment.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-engine: disable cloud handoff for session after auth failure

A stale or revoked API key made every low-confidence turn re-attempt the
cloud handoff, burning up to cloud_timeout_ms per turn before falling back
to local output. Track http_401/http_403 completion failures on the model
handle and drop handoff eligibility for the rest of the session once one
is seen; timeouts, 5xx, and connection errors keep retrying as before.
The pre-streaming trigger now also respects handoff eligibility, and the
live test drives a forced handoff against a local 401 server to verify
the second turn stays local.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* server: send OpenAI-wrapped tool definitions to the engine

The engine's parse_tools_json scans for a literal "function" key, so
the flat {name, description, parameters} dicts _translate_tools emitted
were parsed as zero tools and requests silently lost their tool
definitions. Build the wrapped form once and filter it for the
tool_choice object branch.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-engine: render gemma3 tool calls with function special tokens

Gemma3-family prompts now follow the FunctionGemma chat template:
system text and tool declarations render in a developer turn, prior
assistant tool calls as <start_function_call>call:name{...} blocks,
and tool results as <start_function_response> blocks, with <escape>
quoting instead of the gemma4 pipe tags. The tool-call constrainer
keys its tags on the model family and forces multi-token tags one
token at a time, so force_tools no longer degenerates into marker
loops on bundles whose tags are not single special tokens. DSL-to-
JSON argument conversion escapes string content so malformed model
output cannot corrupt the response payload.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-engine: render qwen think markup only for thinking chat templates

format_qwen_style appended the Qwen3-hybrid "<think>\n\n</think>\n\n"
generation scaffold (and think-wrapped trailing assistant prefills) for
every qwen-family bundle. Non-thinking bundles such as
qwen3-vl-2b-instruct have no think handling in their chat template, so
the scaffold pushed the model off-distribution: at temperature 0 the
second turn of a conversation greedily replayed the previous assistant
message byte-for-byte instead of answering the new question. The
dequantized bundle reproduces both behaviors in the reference stack:
with the scaffold it emits the identical replay, without it it answers
the question.

Gate the think emission sites on the bundle's chat template actually
containing <think>, which keeps hybrid-thinking renders byte-identical,
and add a multi-turn distinctness regression test.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* tests: share bundle helpers and deduplicate suite scaffolding

Moves bundle discovery into tests/bundles.py, folds the repeated
chunked-prefill check into the llm suite reusing its existing helper,
drops import-time HF env mutation for local_files_only, and extracts
the shared tool-capture setup in the server tests.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* tests: target a non-embedding bundle for the embedding-rejection live test

The rejection test ran against the default LLM fixture (gemma-4-e2b-it),
which carries a decoder_embed_chunk component and therefore legitimately
serves hidden-state embeddings, so the endpoint returned 200 instead of
the expected 400. Discover a bundle that lacks both text_embedding and
decoder_embed_chunk, skipping when none is present, and share the server
startup via a context manager.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-convert: down-scale the gemma4 vision embedding projection

The gemma4 FP16 weight scaler keyed the 1/16 down-scaling on the filename
embed_vision_embedding.weights, but the vision embedding projection is saved
as embed_vision_proj.weights, so it was stored unscaled. The gemma4 runtime
operates in a 1/16-scaled residual space (token embeddings, norms, etc. are
all down-scaled to keep activations in fp16 range), so an unscaled projection
left image soft-tokens ~16x too large in the residual stream. This drowned out
the per-layer refinement of image features and degraded multimodal reasoning:
gemma-4-e4b-it confidently misidentified a monkey as a toucan, while e2b was
unaffected only because its projection was converted before the filename was
dropped from the scale set (regression in #659). Add the real filename so the
projection is down-scaled like every other gemma4 embedding-space weight.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* cactus-convert: down-scale the gemma4 audio embedding projection

Same bug class as the vision embedding projection: the gemma4 audio feature
projection (model.embed_audio.embedding_projection.weight -> embed_audio_proj.weights)
maps audio soft-tokens into the LM's 1/16-scaled residual space but was stored
unscaled (scale_factor 1.0, confirmed: stored norm == HF norm), leaving audio
soft-tokens ~16x too large in the residual stream. Add embed_audio_proj and
embed_audio_embedding to the gemma4 down-scale set so audio embeddings are scaled
like vision embeddings and every other embedding-space weight.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>

* Revert "cactus-convert: down-scale the gemma4 audio embedding projection"

This reverts commit f892740.

---------

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants