Repository navigation
Add native chunked prefill for transpiled LLM bundles - #659
Merged
Merged
Conversation
Implement native chunked prefill support for transpiled multimodal LLM bundles and align the generated component pipelines with the C++ runtime loader. Runtime changes: - Load only the required component graphs for each supported decode route. - Add chunked text prefill routing through lm_encoder_text_chunk and decoder_prefill_chunk components. - Copy chunked prefill cache state into step decode components when compatible. - Support direct decoder-step and full-context text decode routes for bundles that do not expose the full chunked path. - Preserve cache metadata on reset and shift cache rows when thinking-token ranges are removed. - Keep FP16 KV cache configuration stable after process startup. Transpiler changes: - Emit chunk prefill components for Gemma4, Qwen3-VL, and LFM2-VL component pipelines. - Lower shared internal KV-cache attention paths for prefill/query reuse. - Add Qwen3-VL model-type/profile handling and safer Qwen layer-type mask selection. - Preserve Gemma4 vision projection precision and adjust Gemma4 audio bias handling. - Prefer 128-token chunk defaults for supported multimodal profiles. Build and platform changes: - Make local Cactus builds less racy by serializing build directory updates. - Improve Android NDK discovery for Homebrew and SDK installs. - Respect existing Android CPU affinity masks when pinning worker threads. Tests: - Update transpile default expectations for multimodal component pipelines. - Update graph serialization tests for persisted node ids. - Add KV-cache full-window eviction coverage. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
force-pushed
the
v2-chunked-prefill
branch
from
May 20, 2026 19:42
f9766c1 to
ef08202
Compare
Add dynamic Gemma4 media merge for prompts with audio so runtime prefill uses the actual prompt token layout instead of a static captured lm_encoder merge plan. This keeps image-only and non-audio paths on the existing chunked path while routing Gemma4 audio/image token rows through lm_encoder_media_step before decoder_prefill_chunk. Fix Gemma4 audio conversion policy by keeping audio bias tensors in FP16, and preserve the vision/runtime fixes needed for native Gemma4 image quality. Add local integration probes and investigation docs covering Gemma image/audio/mixed, Qwen image, Parakeet file/PCM, and smoke validation. Also ignore generated local integration and investigation outputs while preserving benchmark source/results and reproducible test harnesses. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Release graph temporaries after their final use while retaining declared component outputs and cache state buffers. Unload transient media, encoder, and chunk-prefill graphs after copying their outputs so Gemma4 image/audio requests do not keep large intermediate buffers resident. Validation: focused Gemma4, Qwen3-VL, and LFM2-VL integration checks pass locally. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
justinl66
force-pushed
the
v2-chunked-prefill
branch
from
May 21, 2026 03:42
f877825 to
68a422e
Compare
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Signed-off-by: justinl66 <justinfinity7@gmail.com>
Add a first-token path that can sample directly after cache priming, track chunked prefill cache-copy and tail metrics, and allow sufficiently large tails to run as padded chunks. When padded chunks are used for first-token sampling, rewind the copied decode cache to the final real prompt token and run that final token through the step decoder so logits are taken from real context instead of pad context. Keep LFM2-VL on scalar tails because its recurrent cached state is not safe to rewind after executing padded tokens. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
force-pushed
the
v2-chunked-prefill
branch
from
May 22, 2026 22:40
71b9b8b to
c044fbc
Compare
# Conflicts: # python/cactus/cli/run.py
ncylich
added a commit
that referenced
this pull request
Jun 14, 2026
The gemma4 FP16 weight scaler keyed the 1/16 down-scaling on the filename embed_vision_embedding.weights, but the vision embedding projection is saved as embed_vision_proj.weights, so it was stored unscaled. The gemma4 runtime operates in a 1/16-scaled residual space (token embeddings, norms, etc. are all down-scaled to keep activations in fp16 range), so an unscaled projection left image soft-tokens ~16x too large in the residual stream. This drowned out the per-layer refinement of image features and degraded multimodal reasoning: gemma-4-e4b-it confidently misidentified a monkey as a toucan, while e2b was unaffected only because its projection was converted before the filename was dropped from the scale set (regression in #659). Add the real filename so the projection is down-scaled like every other gemma4 embedding-space weight. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
jakmro
pushed a commit
that referenced
this pull request
Jun 14, 2026
* cactus-graph: preserve cache-node descriptor across steal_cache_buffer
Any second chunked prefill on a gated-deltanet bundle threw a state rank
error because moving the cache buffer into the decoder graph left the
prefill node's BufferDesc empty. Restore the source to a fresh unallocated
descriptor with its original shape and precision so the next execute
re-initializes the state exactly as on first run.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-engine: clear tool-constraint bias on deactivation
Every generation after a force_tools turn on the same handle degenerated
into <|tool_call> loops on gemma models, surviving cactus_reset. The
deactivating init left the previously computed force bias in place while
update() no-ops when inactive, so decode kept force-picking the
call-start token. Clear the bias in init and drop the redundant reset in
clear_tool_constraints that recomputed it just before deactivation.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* engine+python: golden chat-template render tests vs HF reference
Adds a cactus_render_prompt FFI call that exposes the fully rendered
chat prompt, plus a ctypes wrapper and export. A golden pytest suite
checks engine renders and tokenizations against the HF reference chat
templates for gemma-4, qwen3, and lfm2 across system, tool-call,
parallel-tool-call, and thinking conversations, plus the gemma-4 image
expansion. Aligns the gemma-4 formatter with the reference template:
<|think|> gains its trailing newline and the image block drops its
surrounding blank lines.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* transpiler: stable cache layer keys across components
The internal KV-cache layer key fell through to the node id whenever
attention_layer_index was 0, and decoder_media_step graphs never
received attention_layer_index hints at all, so components of one
bundle disagreed on layer keys and the engine's positional
cache-state pairing failed at load time with
"prefill and step cache layer mismatch".
Derive the key with explicit None checks before falling back to the
node id, and include decoder_media_step in the attention-hint
assignment.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-engine: disable cloud handoff for session after auth failure
A stale or revoked API key made every low-confidence turn re-attempt the
cloud handoff, burning up to cloud_timeout_ms per turn before falling back
to local output. Track http_401/http_403 completion failures on the model
handle and drop handoff eligibility for the rest of the session once one
is seen; timeouts, 5xx, and connection errors keep retrying as before.
The pre-streaming trigger now also respects handoff eligibility, and the
live test drives a forced handoff against a local 401 server to verify
the second turn stays local.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* server: send OpenAI-wrapped tool definitions to the engine
The engine's parse_tools_json scans for a literal "function" key, so
the flat {name, description, parameters} dicts _translate_tools emitted
were parsed as zero tools and requests silently lost their tool
definitions. Build the wrapped form once and filter it for the
tool_choice object branch.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-engine: render gemma3 tool calls with function special tokens
Gemma3-family prompts now follow the FunctionGemma chat template:
system text and tool declarations render in a developer turn, prior
assistant tool calls as <start_function_call>call:name{...} blocks,
and tool results as <start_function_response> blocks, with <escape>
quoting instead of the gemma4 pipe tags. The tool-call constrainer
keys its tags on the model family and forces multi-token tags one
token at a time, so force_tools no longer degenerates into marker
loops on bundles whose tags are not single special tokens. DSL-to-
JSON argument conversion escapes string content so malformed model
output cannot corrupt the response payload.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-engine: render qwen think markup only for thinking chat templates
format_qwen_style appended the Qwen3-hybrid "<think>\n\n</think>\n\n"
generation scaffold (and think-wrapped trailing assistant prefills) for
every qwen-family bundle. Non-thinking bundles such as
qwen3-vl-2b-instruct have no think handling in their chat template, so
the scaffold pushed the model off-distribution: at temperature 0 the
second turn of a conversation greedily replayed the previous assistant
message byte-for-byte instead of answering the new question. The
dequantized bundle reproduces both behaviors in the reference stack:
with the scaffold it emits the identical replay, without it it answers
the question.
Gate the think emission sites on the bundle's chat template actually
containing <think>, which keeps hybrid-thinking renders byte-identical,
and add a multi-turn distinctness regression test.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* tests: share bundle helpers and deduplicate suite scaffolding
Moves bundle discovery into tests/bundles.py, folds the repeated
chunked-prefill check into the llm suite reusing its existing helper,
drops import-time HF env mutation for local_files_only, and extracts
the shared tool-capture setup in the server tests.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* tests: target a non-embedding bundle for the embedding-rejection live test
The rejection test ran against the default LLM fixture (gemma-4-e2b-it),
which carries a decoder_embed_chunk component and therefore legitimately
serves hidden-state embeddings, so the endpoint returned 200 instead of
the expected 400. Discover a bundle that lacks both text_embedding and
decoder_embed_chunk, skipping when none is present, and share the server
startup via a context manager.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-convert: down-scale the gemma4 vision embedding projection
The gemma4 FP16 weight scaler keyed the 1/16 down-scaling on the filename
embed_vision_embedding.weights, but the vision embedding projection is saved
as embed_vision_proj.weights, so it was stored unscaled. The gemma4 runtime
operates in a 1/16-scaled residual space (token embeddings, norms, etc. are
all down-scaled to keep activations in fp16 range), so an unscaled projection
left image soft-tokens ~16x too large in the residual stream. This drowned out
the per-layer refinement of image features and degraded multimodal reasoning:
gemma-4-e4b-it confidently misidentified a monkey as a toucan, while e2b was
unaffected only because its projection was converted before the filename was
dropped from the scale set (regression in #659). Add the real filename so the
projection is down-scaled like every other gemma4 embedding-space weight.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* cactus-convert: down-scale the gemma4 audio embedding projection
Same bug class as the vision embedding projection: the gemma4 audio feature
projection (model.embed_audio.embedding_projection.weight -> embed_audio_proj.weights)
maps audio soft-tokens into the LM's 1/16-scaled residual space but was stored
unscaled (scale_factor 1.0, confirmed: stored norm == HF norm), leaving audio
soft-tokens ~16x too large in the residual stream. Add embed_audio_proj and
embed_audio_embedding to the gemma4 down-scale set so audio embeddings are scaled
like vision embeddings and every other embedding-space weight.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
* Revert "cactus-convert: down-scale the gemma4 audio embedding projection"
This reverts commit f892740.
---------
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds native chunked prefill support for transpiled multimodal LLM bundles and keeps the implementation aligned with the current v2 native C++ runtime path.
Runtime changes
lm_encoder_text_chunkanddecoder_prefill_chunkcomponents.CACTUS_KV_CACHE_FP16lookup after startup so cache mode stays stable during a process.Transpiler changes
Build and platform changes
Tests
./cactus/build.shPYTHONPATH=python ./venv/bin/python -m pytest python/tests/test_cli_transpile_defaults.py -qPYTHONPATH=python ../cactus/venv/bin/python -m pytest python/tests/test_cli_transpile_defaults.py -q