Repository navigation
Conversation
set_sequence_length_in_shape only rewrote a sequence dimension equal to 1. lm_encoder outputs are now reported at full context (e.g. [1, 128000, 1024]), so the stored inputs_embeds shape kept 128000 and run_chunk_prefill_path ran about 1000 decoder_prefill_chunk passes for a ~1k-token image prompt. On Metal this looks like a hang in waitUntilCompleted; on CPU it spins at full load. Write the dimension that sequence_length_from_shape reads. Fixes cactus-compute#802 Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>
Published LFM2-VL bundles (weights tag v2.0) predate the media_image_preprocess_strategy manifest key. With it empty the engine preprocessed images through the Gemma 4 path, skipped the vision projector and never spliced image features, and the tokenizer fell back to a single 1024-token tile. Completions ran but ignored the image. Fall back to lfm2_vl when the key is missing and the manifest family is lfm2_vl. Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #802
This isn't a Metal deadlock. Chunked media prefill runs about 1000 chunks instead of about 9, so Metal sits in
waitUntilCompletedwhile the GPU works through them, and the CPU backend spins at full load.1. Sequence length
lm_encoderoutputs are reported at full context, e.g.inputs_embedsis[1, 128000, 1024].set_sequence_length_in_shapeonly rewrote a sequence dimension equal to 1, so the stored shape kept 128000.run_chunk_prefill_paththen setfull_seq = 128000for a 1043-token prompt. The fix writes the same dimension thatsequence_length_from_shapereads.2. Published LFM2-VL bundles
With the hang gone, image answers were still wrong. The
v2.0bundles on HF have nomedia_image_preprocess_strategykey in their manifest; the current transpiler writes it, but these bundles are older. Without the key, the engine used the Gemma 4 preprocessing path, never spliced image features, and tokenized the image as one 1024-token tile. The fix falls back tolfm2_vlwhen the key is missing andfamilyislfm2_vl.Testing (
Cactus-Compute/LFM2-VL-450Mcq4 @v2.0, M-series Mac,assets/app.png, "Describe this image in one sentence.")waitUntilCompletedv2.0.1 gives the same caption with 2076 prompt tokens and a 7.0s TTFT.