Skip to content

feat: add MiniCPM-V 4.6 image-to-text support - #455

Open
Dayuxiaoshui wants to merge 1 commit into
inclusionAI:mainfrom
Dayuxiaoshui:OP
Open

feat: add MiniCPM-V 4.6 image-to-text support#455
Dayuxiaoshui wants to merge 1 commit into
inclusionAI:mainfrom
Dayuxiaoshui:OP

Conversation

@Dayuxiaoshui

Copy link
Copy Markdown

What does this PR do?

This PR enables image-to-text training and serving for MiniCPM-V 4.6 checkpoints through AReno's existing MiniCPM adapter.

The implementation:

  • aligns MiniCPM-V processor outputs (pixel_values, target_sizes, and patch grouping metadata) with AReno's multimodal feature schema;
  • loads and saves the MiniCPM-V NaViT vision tower and merger/projector weights alongside the language model;
  • executes the variable-resolution vision encoder, window merger, and final visual-to-language projection inside the AReno model adapter;
  • replaces image placeholder token embeddings with projected image features;
  • preserves the hybrid Gated DeltaNet/full-attention paths during training, prefill, eager decode, and CUDA Graph decode;
  • adds dense recurrent-cache slot management so hybrid attention state is independent of paged KV-cache block IDs and is cleared before slot reuse;
  • supports multimodal SFT, GSPO/GRPO-style rollout training, checkpoint round trips, tensor parallelism, and OpenAI-compatible serving; and
  • prevents the Qwen3.5-VL adapter from claiming nested MiniCPM-V text configs.

Related issue

Fixes #137

Parent issue: #132

Type of change

  • Bug fix
  • New feature
  • Breaking change (public API / CLI behavior changes in a non-backward-compatible way)
  • Documentation update
  • Refactoring
  • Performance improvement
  • Test coverage improvement

How was it tested?

CPU coverage:

pytest -q tests/test_inference_scheduler_cpu.py tests/test_minicpmv46_multimodal_cpu.py
# 34 passed

pytest -q tests/test_registry_cpu.py tests/test_registry_discovery_cpu.py
# 7 passed

CUDA_VISIBLE_DEVICES= pytest -q tests/test_policy_tensor_sync_cpu.py::test_real_gloo_collectives_reshard_train_tp2_to_rollout_tp1 tests/test_policy_tensor_sync_cpu.py::test_real_gloo_collectives_reshard_train_tp1_to_rollout_tp2
# 2 passed

pytest tests/ -k cpu completed with 450 passing tests. Three socket/Gloo failures from the restricted test environment passed when rerun outside the sandbox (with GPUs hidden for the two CPU policy-sync cases). Two existing Qwen3.5-VL CPU tests still fail on the prerequisite Qwen3.5 image-to-text baseline and are unrelated to the MiniCPM-V changes.

Real-model validation used the ModelScope checkpoint OpenBMB/MiniCPM-V-4.6 on NVIDIA H200 GPUs:

  • loaded all 779 checkpoint tensors, including 453 vision-tower tensors and 6 merger tensors;
  • matched sampled text/vision checkpoint mappings exactly (max_abs_diff=0);
  • matched the Transformers reference projected image embeddings exactly (max_abs_diff=0, mean_abs_diff=0);
  • served red and blue image questions through the OpenAI-compatible API and returned Red and Blue;
  • generated a 16-token image response through CUDA Graph replay, with runtime logs reporting cuda_graph=True;
  • completed two consecutive image SFT steps with finite loss and gradient norms, then saved and reloaded the checkpoint successfully;
  • confirmed finite, nonzero gradients for all 453 vision-tower gradient tensors and all 6 merger gradient tensors;
  • completed a TP=2 image SFT step; and
  • completed one end-to-end image GSPO step covering rollout, reward, multimodal training, and policy role switching.

The installed flash_attn package in the validation environment did not export the required attention functions, so GPU validation used --attn-backend native. This is an environment limitation rather than a MiniCPM-V adapter failure.

Checklist

  • The PR title summarizes the contribution.
  • Linked the related issue in the description (if any).
  • Existing tests pass (pytest tests/ -k cpu).
  • New behavior is covered by tests.
  • Described the test commands run and any hardware limitations.
  • Public API / CLI changes are additive and backward-compatible (see CONTRIBUTING.md).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add MiniCPM-V 4.6 image-to-text support

1 participant