Skip to content

Add source-based multimodal projector loading#267

Merged
leehack merged 2 commits into
mainfrom
issue-133-mmproj-source-api
Jul 4, 2026
Merged

Add source-based multimodal projector loading#267
leehack merged 2 commits into
mainfrom
issue-133-mmproj-source-api

Conversation

@leehack

@leehack leehack commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Summary

Adds source-based multimodal projector loading for issue #133:

  • adds LlamaEngine.loadMultimodalProjectorSource(...)
  • routes projector sources through the same ModelSource / resolver / download-cache layer used by loadModelSource(...)
  • serializes source resolution, download/cache work, and projector loading through the multimodal lifecycle queue
  • documents native/file-backed and URL-loading backend behavior, including unsupported auth/checksum/cache options on URL backends
  • updates README, website lifecycle/multimodal docs, support matrix, and changelog surfaces

Fixes #133.

Validation

  • dart test test/unit/core/engine/engine_test.dart passed on VM and Chrome during implementation
  • after lifecycle review fix, dart test -p vm test/unit/core/engine/engine_test.dart passed
  • dart analyze lib/src/core/engine/engine.dart test/unit/core/engine/engine_test.dart passed
  • dart format --output=none --set-exit-if-changed lib/src/core/engine/engine.dart test/unit/core/engine/engine_test.dart passed
  • git diff --check passed
  • Direct real model + source-based mmproj E2E passed:
    • served local mmproj-LFM2-VL-450M-Q8_0.gguf over http://127.0.0.1:7359/
    • dart run .dart_tool/e2e/mmproj_source_real_model_smoke.dart /opt/UnitySrc/personal/llama/llamadart/models/LFM2-VL-450M-Q4_0.gguf http://127.0.0.1:7359/mmproj-LFM2-VL-450M-Q8_0.gguf /private/tmp/llamadart-ticket-133/.dart_tool/e2e/mmproj-cache
    • result: RESULT mmproj_source_real_model {"supportsVision":true,...}

Follow-up

A higher-level atomic loadModelBundle(...) API with rollback semantics and independent model/projector cache options remains separate from this scoped projector-source API.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-267.

@leehack leehack marked this pull request as ready for review July 4, 2026 17:59
Copilot AI review requested due to automatic review settings July 4, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new public engine API to load GGUF multimodal projector (“mmproj”) assets via ModelSource, aligning projector loading with the existing model source resolution + download/cache pipeline and documenting backend-specific limitations (notably URL-loading web backends).

Changes:

  • Added LlamaEngine.loadMultimodalProjectorSource(...) with multimodal lifecycle queue serialization and URL-backend option gating.
  • Expanded unit tests to cover native/web behaviors, option forwarding, and serialization guarantees.
  • Updated README + website docs + support matrix + changelogs to document the new API and platform constraints.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
lib/src/core/engine/engine.dart Implements loadMultimodalProjectorSource(...) and extends URL-backend option rejection messaging.
test/unit/core/engine/engine_test.dart Adds coverage for projector source loading across native/web, options/progress forwarding, and serialization behavior.
README.md Documents how/when to use loadMultimodalProjectorSource(...) and web limitations.
website/docs/guides/multimodal.md Adds guidance + web notes for source-based projector loading.
website/docs/guides/model-lifecycle.md Documents lifecycle semantics and backend limitations for projector source loading.
website/docs/platforms/support-matrix.md Notes platform support/limitations for projector sources on URL-loading backends.
CHANGELOG.md Adds an Unreleased entry for the new API.
website/docs/changelog/recent-releases.md Adds an Unreleased entry mirroring the changelog note.
Comments suppressed due to low confidence (1)

lib/src/core/engine/engine.dart:1208

  • For non-model assets (e.g. assetType: 'multimodal projector'), the cancellation-token error message is still generic and doesn’t mention the asset type, unlike the other checks in this helper. This makes URL-backend projector failures harder to diagnose from logs.
    if (options.cancelToken != null) {
      throw LlamaUnsupportedException(
        'Cancellation tokens require the native download/cache manager.',
      );

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@leehack leehack force-pushed the issue-133-mmproj-source-api branch from 89b92f1 to 89dac86 Compare July 4, 2026 21:53
@leehack leehack merged commit 425d66b into main Jul 4, 2026
11 checks passed
@leehack leehack deleted the issue-133-mmproj-source-api branch July 4, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): consider source-based multimodal projector loading API

2 participants