Skip to content

Fix transpiled LLM runtime and optimize chunked prefill - #654

Closed
ncylich wants to merge 13 commits into
v2from
v2-perf
Closed

ncylich wants to merge 13 commits into
v2from
v2-perf

Conversation

@ncylich

@ncylich ncylich commented May 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes the native transpiled LLM runtime path and optimizes chunked prefill for Cactus component bundles. It keeps cactus run on the raw C++/FFI-backed chat binary, adds token-id benchmark support, and makes c128 chunked prefill work across Gemma, LFM, and Qwen-style component bundles.

Key Changes

  • Adds a native cactus_benchmark_tokens FFI entry point and wires cactus run / chat to support --input-ids, --max-new-tokens, and --result-json.
  • Adds cached/chunked prefill routing for component bundles with lm_encoder_text_chunk and decoder_prefill_chunk.
  • Updates the default prefill chunk size to 128.
  • Fixes LFM chunked prefill by allowing compatible conv cache states instead of forcing token-by-token fallback.
  • Fixes Qwen decode handoff by copying INT8 KV cache rows and scale data correctly when prefill and step cache capacities differ.
  • Adds precision conversion for component tensor handoff where needed for scalar/int-style inputs such as position_ids and attention_mask.
  • Adds prefill-only benchmark behavior that can skip decode-cache preparation when no decode tokens are requested.
  • Adds graph/runtime support needed for cache-only attention and chunked prefill lowering.
  • Updates transpiler profiles/adapters for Gemma, LFM, and Qwen component selection, including Qwen VL chunked prefill components.
  • Adds build and CLI plumbing needed for native transpiled runtime execution, including safer local build locking and Android NDK discovery.
  • Updates .gitignore to keep local benchmark outputs, generated Android test artifacts, and local third-party runtime drops out of status.

Validation

  • Ran python -m py_compile for the touched CLI and transpiler modules.
  • Ran native cactus run smoke tests with 1024-token fixtures for Gemma, LFM, and Qwen.
  • Verified LFM and Qwen generation smoke tests with multi-token decode after chunked prefill.
  • Verified Qwen no longer crashes during chunked-prefill decode handoff after INT8 KV cache scale copying was fixed.

Risks / Notes

  • The PR changes the native cactus run behavior to require a local transpiled component bundle rather than falling back through older download/native-loader paths.
  • Prefill-only benchmark rows intentionally skip decode-cache preparation when max_new_tokens=0, so those timings should not be compared directly to TTFT-style rows that include decode handoff.
  • The Android benchmark harness source is not included in this PR; benchmark outputs and generated local artifacts remain ignored.

Replace the Pixel Cactus LFM and Qwen full-core c128 prefill rows with fresh rerun measurements, including the focused Qwen 4096 retry row.

Keep the corrected Mac, Samsung, and Pixel full-core rows in the main aggregate CSVs, and remove redundant handoff/per-run artifacts from the tracked branch after verifying the Samsung per-run rows are already present in matrix_galaxy_s24_or_s25_full_core.csv.

Add the Android cactus_llm_bench target wiring needed to build the standalone benchmark runner, including the mbedTLS archive links and expected output path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant