feat(inference): parse Qwen3.5 vision config fields (ADR-069 S1)#670
feat(inference): parse Qwen3.5 vision config fields (ADR-069 S1)#670ohdearquant wants to merge 1 commit into
Conversation
Parse and store the vision-language checkpoint fields that were previously dropped by qwen35_config.rs, without wiring them into the forward pass: - New VisionConfig struct (ViT depth/hidden_size/heads/patch geometry etc.), parsed from the top-level vision_config object; None for text-only checkpoints. - Four vision token ids (image/video/vision_start/vision_end), threaded from the top-level config.json the same way tie_word_embeddings already is. - RopeParams gains mrope_section and mrope_interleaved, parsed from text_config.rope_parameters alongside the existing rope_theta and partial_rotary_factor handling (unchanged). All five new Qwen35Config fields default to None and are additive; every preset constructor and existing config.json parse path is unaffected. Adding them to the struct required a mechanical None-field addition at existing exhaustive struct-literal test helpers across the crate (no logic changes) so the crate keeps compiling under default, metal-gpu, and all-features builds. Round-trips the real 0.8B checkpoint fixture (vision_config dims, all four token ids, mrope_section == [11, 11, 10], mrope_interleaved == true) and adds a text-only-config negative test proving the plain decoder path is unaffected. Consumption by the forward pass (ViT encoder, merger, decoder M-RoPE) lands in later ADR-069 stages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E2E Parity ReportPASS: all 4 prompts match within their respective match windows
|
print(fib
print(fib
|
E2E Parity ReportPASS: 3/4 gating prompts match; 1 known divergence (#535) excluded from the verdict
|
|
) * docs(adr): ADR-078 — multimodal/vision serving priority (deferral) Eighth/final sequencing ADR (bundle I). Records the PRIORITY placement of multimodal serving (deferred-not-ranked, gated on the vision dev lane) — distinct from ADR-069's already-accepted technical vision design, which it does not re-litigate or amend. Verdict rests on source/issue state at origin/main@594b557be: serve fails closed on image_url (PR #656), vision module inert scaffold (called only from tests), zero vision fields in the runtime config, ADR-069 merged (#669), S1 idle as draft PR #670, tracking #564/#565/#566 open. Explicit non-deprioritization: vision is an active tracked dev lane. No experiment needed; amends nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(adr): address ADR-078 codex R1 — precise R2/R3 grep claims + named re-entry trigger R1 mediums, TBV'd vs origin/main@594b557be: - M1 (R2): generate_multimodal has more test callers than listed (:22396,:22417 in metal_generate_multimodal_honors_stop_string); rephrased to 'every live caller is a test fn' + VisionEncoder::new sole call site vision/mod.rs:265. Conclusion (no binary/non-test path) unchanged. - M2 (R3): 'zero Metal references' overstated — 5 doc-comment Metal mentions exist (vit.rs:12,14; mod.rs:19; multimodal.rs:7,38). Rephrased to 'no Metal implementation/API; only doc comments'. Leo press-time gate: added a NAMED re-entry trigger to the gated-deferral decision (ADR-069 S5-S6 served-parity milestone, or explicit #564 lane scheduling) so the deferral has a concrete re-entry condition, not an open-ended park. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
ADR-069 stage S1: parse the vision-language checkpoint fields that
qwen35_config.rswas previously dropping, without wiring them into theforward pass.
VisionConfigstruct (ViTdepth,hidden_size,num_heads,patch_size,spatial_merge_size,out_hidden_size,temporal_patch_size,num_position_embeddings,in_channels,deepstack_visual_indexes), parsed from the top-levelvision_configobject in
config.json.NoneonQwen35Configfor text-onlycheckpoints.
image_token_id,video_token_id,vision_start_token_id,vision_end_token_id), threaded from thetop-level
config.jsonthe same waytie_word_embeddingsalready is.RopeParamsgainsmrope_section: Option<Vec<usize>>andmrope_interleaved: Option<bool>, parsed fromtext_config.rope_parametersalongside the existingrope_thetaandpartial_rotary_factorhandling (unchanged).All five new
Qwen35Configfields are additive and default toNone;every preset constructor and the existing
config.jsonparse path areotherwise unaffected. Consumption by the forward pass (ViT encoder,
merger, decoder M-RoPE) lands in later ADR-069 stages -- these fields are
parsed and stored only.
Note on scope
Qwen35Configis a 31(+5)-field struct built via fully-exhaustive structliterals (no
..Default::default()spread) in several test helperfunctions across the crate. Adding new fields to the struct is a source
change that, by Rust's own construction-exhaustiveness rules, requires a
matching field at every such literal. This PR adds the 5 new fields (all
None) at each of those existing test-helper literals -- purelymechanical, no logic changes -- so the crate keeps compiling under
default,
metal-gpu, and all-features builds. The real feature work isentirely contained in
qwen35_config.rs.Verification
(also clean under
--all-targets,--features metal-gpu, and--all-features)cargo fmt -p lattice-inference -- --checkis clean.The round-trip test parses the real released 0.8B
config.jsonfixture(already committed at
crates/inference/tests/fixtures/qwen35_0_8b_config.json)and asserts every vision dimension, all four token ids, and
mrope_section == [11, 11, 10]/mrope_interleaved == true. A secondtest asserts a text-only config.json (no
vision_config, norope_parameters) still parses cleanly with every new field resolving toNone, proving the plain-text decoder path is unaffected.Follow-up
Consuming these fields (ViT forward pass, patch merger, image-token
injection, decoder M-RoPE) is out of scope here and lands in later
ADR-069 stages.