Skip to content

Support full-context cache transpilation - #670

Open
ncylich wants to merge 4 commits into
v2from
fix-full-context
Open

ncylich wants to merge 4 commits into
v2from
fix-full-context

Conversation

@ncylich

@ncylich ncylich commented May 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add explicit cache context sizing for cached component transpilation so Gemma4, Qwen, and LFM decoder graphs can reserve the model context length instead of deriving capacity from the capture prompt.
  • Keep Gemma4 sliding-attention layers capped to the configured sliding window while full-attention layers use the requested/model context length.
  • Add cactus convert --cache-context-length and --retranspile so graph artifacts can be rebuilt from existing converted weights.
  • Add exact token-file input for cactus run and the chat test binary, plus a context-scaling benchmark tool that writes CSV/JSONL summaries and PNG plots.

Validation

  • source ./venv/bin/activate && python -m py_compile python/tests/transpile/tools/benchmark_gemma4_context_scaling.py && pytest -q python/tests/test_transpile_cache_context.py python/tests/test_cli_run.py python/tests/test_bindings.py::TestTranspileOptions python/tests/test_bindings.py::TestCliParser::test_run_command_chunked_bundle_flags python/tests/test_cli_transpile_defaults.py python/tests/test_optimize_gemma4_attention.py
  • source ./venv/bin/activate && cactus build
  • source ./venv/bin/activate && cactus run weights/gemma-4-e4b-it --input-ids-file /tmp/gemma4_cactus_verify_tokens.txt --max-new-tokens 4 --result-json /tmp/gemma4_cactus_verify_result.json
  • Previously verified retranspiled Gemma4 E4B IR with full-attention cache length 131072 and sliding-attention cache length 512, then ran live full-context generation with 131040 prompt tokens and 32 generated tokens.

@ncylich
ncylich force-pushed the fix-full-context branch from fc794f4 to 5469e19 Compare May 28, 2026 06:36
@ncylich ncylich changed the title Support full-context Gemma4 cache transpilation Support full-context cache transpilation May 28, 2026
@ncylich
ncylich force-pushed the fix-full-context branch from 5469e19 to 96d71a8 Compare May 28, 2026 06:39
Add explicit cache context sizing for cached component transpilation so decoder graphs can reserve the model context length instead of deriving capacity from the capture prompt. Gemma4, Qwen, and LFM cached decode builders now share the same config-driven resolver while preserving their existing capture-derived fallback sizes when no model context field is available.

For Gemma4, sliding-attention layers keep the configured sliding-window cache size while full-attention layers receive the requested full cache length. Add a retranspile path that reuses existing converted weights, wire the cache context option through the CLI and transpile entrypoint, and make lowering honor per-attention cache length metadata.

Add token-file input support for exact-token live generation and a context-scaling benchmark tool that records CSV/JSONL summaries and plots. Cover the new CLI plumbing, retranspile behavior, context parsing, and per-layer cache metadata with focused tests.

Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the fix-full-context branch from 96d71a8 to cb9adda Compare May 28, 2026 06:54
ncylich added 2 commits May 29, 2026 09:49
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the fix-full-context branch from ab7109a to e00859d Compare May 29, 2026 16:51
Signed-off-by: Noah Cylich <noahcylich@gmail.com>

# Conflicts:
#	python/tests/test_cli_transpile_defaults.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant