Skip to content

Fix image completions hanging in chunked media prefill - #811

Open
ayushmk7 wants to merge 2 commits into
cactus-compute:mainfrom
ayushmk7:fix/media-prefill-sequence-length
Open

ayushmk7 wants to merge 2 commits into
cactus-compute:mainfrom
ayushmk7:fix/media-prefill-sequence-length

Conversation

@ayushmk7

Copy link
Copy Markdown
Contributor

Fixes #802

This isn't a Metal deadlock. Chunked media prefill runs about 1000 chunks instead of about 9, so Metal sits in waitUntilCompleted while the GPU works through them, and the CPU backend spins at full load.

1. Sequence length

lm_encoder outputs are reported at full context, e.g. inputs_embeds is [1, 128000, 1024]. set_sequence_length_in_shape only rewrote a sequence dimension equal to 1, so the stored shape kept 128000. run_chunk_prefill_path then set full_seq = 128000 for a 1043-token prompt. The fix writes the same dimension that sequence_length_from_shape reads.

2. Published LFM2-VL bundles

With the hang gone, image answers were still wrong. The v2.0 bundles on HF have no media_image_preprocess_strategy key in their manifest; the current transpiler writes it, but these bundles are older. Without the key, the engine used the Gemma 4 preprocessing path, never spliced image features, and tokenized the image as one 1024-token tile. The fix falls back to lfm2_vl when the key is missing and family is lfm2_vl.

Testing (Cactus-Compute/LFM2-VL-450M cq4 @ v2.0, M-series Mac, assets/app.png, "Describe this image in one sentence.")

main this PR
Metal no output after 90s, stuck in waitUntilCompleted "A phone screen with a question about the square root of 64." (TTFT 7.0s, 2076 prompt tokens)
CPU no output after 180s, ~1000% CPU same caption (TTFT 9.8s)
Metal, text only "Paris" "Paris"

v2.0.1 gives the same caption with 2076 prompt tokens and a 7.0s TTFT.

set_sequence_length_in_shape only rewrote a sequence dimension equal to 1.
lm_encoder outputs are now reported at full context (e.g. [1, 128000, 1024]),
so the stored inputs_embeds shape kept 128000 and run_chunk_prefill_path ran
about 1000 decoder_prefill_chunk passes for a ~1k-token image prompt. On
Metal this looks like a hang in waitUntilCompleted; on CPU it spins at full
load.

Write the dimension that sequence_length_from_shape reads.

Fixes cactus-compute#802

Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>
Published LFM2-VL bundles (weights tag v2.0) predate the
media_image_preprocess_strategy manifest key. With it empty the engine
preprocessed images through the Gemma 4 path, skipped the vision projector
and never spliced image features, and the tokenizer fell back to a single
1024-token tile. Completions ran but ignored the image.

Fall back to lfm2_vl when the key is missing and the manifest family is
lfm2_vl.

Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v2.1.0: cactus_complete with an image never returns — Metal backend deadlocks in waitUntilCompleted during chunked media prefill (macOS + iOS)

1 participant