Skip to content

feat(inference): parse Qwen3.5 vision config fields (ADR-069 S1)#670

Draft
ohdearquant wants to merge 1 commit into
feat/visionfrom
s1-vision-config
Draft

feat(inference): parse Qwen3.5 vision config fields (ADR-069 S1)#670
ohdearquant wants to merge 1 commit into
feat/visionfrom
s1-vision-config

Conversation

@ohdearquant

Copy link
Copy Markdown
Owner

Authored by Claude (Anthropic agent) on behalf of @ohdearquant.

Summary

ADR-069 stage S1: parse the vision-language checkpoint fields that
qwen35_config.rs was previously dropping, without wiring them into the
forward pass.

  • New VisionConfig struct (ViT depth, hidden_size, num_heads,
    patch_size, spatial_merge_size, out_hidden_size,
    temporal_patch_size, num_position_embeddings, in_channels,
    deepstack_visual_indexes), parsed from the top-level vision_config
    object in config.json. None on Qwen35Config for text-only
    checkpoints.
  • Four vision token ids (image_token_id, video_token_id,
    vision_start_token_id, vision_end_token_id), threaded from the
    top-level config.json the same way tie_word_embeddings already is.
  • RopeParams gains mrope_section: Option<Vec<usize>> and
    mrope_interleaved: Option<bool>, parsed from
    text_config.rope_parameters alongside the existing rope_theta and
    partial_rotary_factor handling (unchanged).

All five new Qwen35Config fields are additive and default to None;
every preset constructor and the existing config.json parse path are
otherwise unaffected. Consumption by the forward pass (ViT encoder,
merger, decoder M-RoPE) lands in later ADR-069 stages -- these fields are
parsed and stored only.

Note on scope

Qwen35Config is a 31(+5)-field struct built via fully-exhaustive struct
literals (no ..Default::default() spread) in several test helper
functions across the crate. Adding new fields to the struct is a source
change that, by Rust's own construction-exhaustiveness rules, requires a
matching field at every such literal. This PR adds the 5 new fields (all
None) at each of those existing test-helper literals -- purely
mechanical, no logic changes -- so the crate keeps compiling under
default, metal-gpu, and all-features builds. The real feature work is
entirely contained in qwen35_config.rs.

Verification

cargo test -p lattice-inference qwen35_config::
test model::qwen35_config::tests::test_qwen35_0_8b_config_json_fixture_parses ... ok
test model::qwen35_config::tests::test_text_only_config_has_no_vision_fields ... ok
test result: ok. 52 passed; 0 failed; 0 ignored; 0 measured; 1456 filtered out; finished in 0.00s
cargo clippy -p lattice-inference -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.03s

(also clean under --all-targets, --features metal-gpu, and
--all-features)

cargo test -p lattice-inference --lib
test result: ok. 1500 passed; 0 failed; 8 ignored; 0 measured; 0 filtered out; finished in 44.67s

cargo fmt -p lattice-inference -- --check is clean.

The round-trip test parses the real released 0.8B config.json fixture
(already committed at crates/inference/tests/fixtures/qwen35_0_8b_config.json)
and asserts every vision dimension, all four token ids, and
mrope_section == [11, 11, 10] / mrope_interleaved == true. A second
test asserts a text-only config.json (no vision_config, no
rope_parameters) still parses cleanly with every new field resolving to
None, proving the plain-text decoder path is unaffected.

Follow-up

Consuming these fields (ViT forward pass, patch merger, image-token
injection, decoder M-RoPE) is out of scope here and lands in later
ADR-069 stages.

Parse and store the vision-language checkpoint fields that were previously
dropped by qwen35_config.rs, without wiring them into the forward pass:

- New VisionConfig struct (ViT depth/hidden_size/heads/patch geometry etc.),
  parsed from the top-level vision_config object; None for text-only
  checkpoints.
- Four vision token ids (image/video/vision_start/vision_end), threaded from
  the top-level config.json the same way tie_word_embeddings already is.
- RopeParams gains mrope_section and mrope_interleaved, parsed from
  text_config.rope_parameters alongside the existing rope_theta and
  partial_rotary_factor handling (unchanged).

All five new Qwen35Config fields default to None and are additive; every
preset constructor and existing config.json parse path is unaffected.
Adding them to the struct required a mechanical None-field addition at
existing exhaustive struct-literal test helpers across the crate (no
logic changes) so the crate keeps compiling under default, metal-gpu, and
all-features builds.

Round-trips the real 0.8B checkpoint fixture (vision_config dims, all four
token ids, mrope_section == [11, 11, 10], mrope_interleaved == true) and
adds a text-only-config negative test proving the plain decoder path is
unaffected.

Consumption by the forward pass (ViT encoder, merger, decoder M-RoPE) lands
in later ADR-069 stages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

E2E Parity Report

PASS: all 4 prompts match within their respective match windows

Prompt Window Agreement First Diff HF tok/s Lattice tok/s Verdict
The capital of France is 3 15/15 none 0.5 2.8 PASS
In the year 2024, artificial intelligence 3 15/15 none 0.4 2.4 PASS
`def fibonacci(n):
if n <= 1:
    return n
return` | 3 | 15/15 | none | 0.4 | 2.0 | PASS |

| def merge_sort(arr): """ Merge sort implementation. | 2 | 15/15 | none | 0.4 | 0.2 | PASS |

The capital of France is

  • HF: Paris.
    The capital of France is Paris.
    The capital of France
  • Lattice: Paris.
    The capital of France is Paris.
    The capital of France

In the year 2024, artificial intelligence

  • HF: (AI) has become a significant part of the global economy. It is
  • Lattice: (AI) has become a significant part of the global economy. It is

def fibonacci(n): if n <= 1: return n return

  • HF: fibonacci(n-1) + fibonacci(n-2)

print(fib

  • Lattice: fibonacci(n-1) + fibonacci(n-2)

print(fib

def merge_sort(arr): """ Merge sort implementation.

  • HF: merge_sort(arr):
    """
    Merge sort implementation.

  • Lattice: merge_sort(arr):
    """
    Merge sort implementation.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

E2E Parity Report

PASS: 3/4 gating prompts match; 1 known divergence (#535) excluded from the verdict

Prompt Window Agreement First Diff HF tok/s Lattice tok/s Verdict
The capital of France is 3 4/4 none 0.4 3.9 PASS
In the year 2024, artificial intelligence 3 4/4 none 0.2 7.1 PASS
`def fibonacci(n):
if n <= 1:
    return n
return` | 3 | 4/4 | none | 0.1 | 5.9 | PASS |

| def merge_sort(arr): """ Merge sort implementation. | 2 | 0/4 | pos 0 | 0.1 | 0.8 | KNOWN-DIVERGENT (#535) |

The capital of France is

  • HF: Paris.
    The
  • Lattice: Paris.
    The

In the year 2024, artificial intelligence

  • HF: (AI) has
  • Lattice: (AI) has

def fibonacci(n): if n <= 1: return n return

  • HF: fibonacci(n-1
  • Lattice: fibonacci(n-1

def merge_sort(arr): """ Merge sort implementation.

  • HF: merge_sort(arr):
  • Lattice: main():

@ohdearquant ohdearquant changed the base branch from main to feat/vision July 5, 2026 17:11
oceanwaves630 added a commit that referenced this pull request Jul 9, 2026
)

* docs(adr): ADR-078 — multimodal/vision serving priority (deferral)

Eighth/final sequencing ADR (bundle I). Records the PRIORITY placement of multimodal serving (deferred-not-ranked, gated on the vision dev lane) — distinct from ADR-069's already-accepted technical vision design, which it does not re-litigate or amend. Verdict rests on source/issue state at origin/main@594b557be: serve fails closed on image_url (PR #656), vision module inert scaffold (called only from tests), zero vision fields in the runtime config, ADR-069 merged (#669), S1 idle as draft PR #670, tracking #564/#565/#566 open. Explicit non-deprioritization: vision is an active tracked dev lane. No experiment needed; amends nothing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(adr): address ADR-078 codex R1 — precise R2/R3 grep claims + named re-entry trigger

R1 mediums, TBV'd vs origin/main@594b557be:
- M1 (R2): generate_multimodal has more test callers than listed (:22396,:22417 in metal_generate_multimodal_honors_stop_string); rephrased to 'every live caller is a test fn' + VisionEncoder::new sole call site vision/mod.rs:265. Conclusion (no binary/non-test path) unchanged.
- M2 (R3): 'zero Metal references' overstated — 5 doc-comment Metal mentions exist (vit.rs:12,14; mod.rs:19; multimodal.rs:7,38). Rephrased to 'no Metal implementation/API; only doc comments'.
Leo press-time gate: added a NAMED re-entry trigger to the gated-deferral decision (ADR-069 S5-S6 served-parity milestone, or explicit #564 lane scheduling) so the deferral has a concrete re-entry condition, not an open-ended park.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant