Skip to content

fix: InvalidConfig labels non-LLaVA loader errors as invalid LLaVA config #986

Description

@inureyes

Problem / Background

HostPreprocessorError::InvalidConfig in src/multimodal/host_preprocessor.rs:1074 is declared as:

#[error("invalid LLaVA host-preprocessor config: {0}")]
InvalidConfig(String),

Several of the messages constructed with that variant have nothing to do with a LLaVA checkpoint (line numbers against main at c3e134ba5):

  1. host_preprocessor.rs:95, in XlaVisionBackendPolicy::from_value: MLXCEL_XLA_VISION_BACKEND must be auto, host, or iree; got {other:?}. This is family independent, and on the Qwen2-VL path it fires from from_env() after the family has been identified as Qwen2-VL, never LLaVA.
  2. host_preprocessor.rs:158: Qwen2-VL XLA vision has no MLX fallback; MLXCEL_XLA_VISION_BACKEND=host is unsupported.
  3. host_preprocessor.rs:178: Qwen2-VL XLA image execution requires the xla-iree feature (the #[cfg(not(feature = "xla-iree"))] arm).

Two related cases found while checking the above:

  1. host_preprocessor.rs:150: failed to identify model family from {path}: {error}. This one fires before any family is known, so claiming LLaVA is guaranteed wrong.
  2. host_preprocessor.rs:237: MLXCEL_XLA_VISION_BACKEND=iree requires the xla-iree feature. This is reachable only under load_llava_image_preprocessor, so the LLaVA prefix is not false, but the message is about an env var and a build feature rather than a config field, and the combined string reads oddly.

Impact is diagnostic only, but it is user facing. Both surfaces that load the preprocessor stringify the error:

  • src/backend/xla.rs:111 wraps it inside create_session as OpenXLA image preprocessor load failed: {error}.
  • src/server/batch/xla_preprocess.rs:74 maps the loader error with error.to_string() and passes it through the worker ready channel, which fails server startup with that text.

So for a Qwen2-VL checkpoint an operator sees:

OpenXLA image preprocessor load failed: invalid LLaVA host-preprocessor config: Qwen2-VL XLA image execution requires the xla-iree feature

They are told a LLaVA config is invalid while loading a Qwen2-VL checkpoint, which sends them looking in the wrong place.

Proposed Solution

Either of:

  • Make the variant text family neutral, for example invalid OpenXLA host-preprocessor config: {0}, or
  • Split a family-specific variant off so LLaVA weight and config failures keep the LLaVA wording while the policy, family-detection, and Qwen2-VL arms get their own variant.

Prefer whichever keeps the LLaVA weight-loading and config messages as informative as they are today. WeightLoad (failed to load LLaVA host-preprocessor weights: {0}) is only produced from the LLaVA loader paths in src/loading/vlm_llava.rs, so it can stay as is unless the split makes a matching rename natural.

Acceptance Criteria

  • The variant text no longer claims LLaVA for family-independent messages (MLXCEL_XLA_VISION_BACKEND policy parsing, model-family detection failure) or for Qwen2-VL messages.
  • The actionable content of each message is preserved verbatim: the env var name MLXCEL_XLA_VISION_BACKEND and its accepted values, and the --features xla-iree rebuild instruction.
  • LLaVA-specific config and weight failures still name LLaVA, so their diagnostics do not regress.
  • cargo test --release --lib --features metal,accelerate multimodal::host_preprocessor stays green.
  • Any assertion that matches on the message text in src/multimodal/host_preprocessor_tests.rs or the src/server/batch/xla_preprocess.rs tests is updated. Current state: the host-preprocessor tests match on the variant with matches! (host_preprocessor_tests.rs:300 and :314) rather than on the display string, and the batch tests assert on other substrings, so no text assertion is expected to break. Re-check after the change.

Technical Considerations

This was split out of #966 to keep that PR surgical. #966 deliberately left the wrapper text alone and only touched the Qwen2-VL loader contract; it does extend the message at host_preprocessor.rs:178 with the rebuild instruction (; rebuild mlxcel with --features xla-iree (this family has no MLX vision fallback)), so whoever picks this up should rebase on the merged #966 and preserve whatever final text landed there.

No behavior change is intended: the same conditions must still fail, with the same variant discriminant unless a new variant is introduced, and only the rendered prefix changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelsModel architectures, weights, loading, metadatagood first issueGood for newcomerspriority:lowLow prioritystatus:doneCompletedtype:bugBug fixes, error corrections, or issue resolutions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions