Add source-based multimodal projector loading#267
Merged
Conversation
Contributor
|
Chat app preview removed for |
There was a problem hiding this comment.
Pull request overview
Adds a new public engine API to load GGUF multimodal projector (“mmproj”) assets via ModelSource, aligning projector loading with the existing model source resolution + download/cache pipeline and documenting backend-specific limitations (notably URL-loading web backends).
Changes:
- Added
LlamaEngine.loadMultimodalProjectorSource(...)with multimodal lifecycle queue serialization and URL-backend option gating. - Expanded unit tests to cover native/web behaviors, option forwarding, and serialization guarantees.
- Updated README + website docs + support matrix + changelogs to document the new API and platform constraints.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
lib/src/core/engine/engine.dart |
Implements loadMultimodalProjectorSource(...) and extends URL-backend option rejection messaging. |
test/unit/core/engine/engine_test.dart |
Adds coverage for projector source loading across native/web, options/progress forwarding, and serialization behavior. |
README.md |
Documents how/when to use loadMultimodalProjectorSource(...) and web limitations. |
website/docs/guides/multimodal.md |
Adds guidance + web notes for source-based projector loading. |
website/docs/guides/model-lifecycle.md |
Documents lifecycle semantics and backend limitations for projector source loading. |
website/docs/platforms/support-matrix.md |
Notes platform support/limitations for projector sources on URL-loading backends. |
CHANGELOG.md |
Adds an Unreleased entry for the new API. |
website/docs/changelog/recent-releases.md |
Adds an Unreleased entry mirroring the changelog note. |
Comments suppressed due to low confidence (1)
lib/src/core/engine/engine.dart:1208
- For non-model assets (e.g.
assetType: 'multimodal projector'), the cancellation-token error message is still generic and doesn’t mention the asset type, unlike the other checks in this helper. This makes URL-backend projector failures harder to diagnose from logs.
if (options.cancelToken != null) {
throw LlamaUnsupportedException(
'Cancellation tokens require the native download/cache manager.',
);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
89b92f1 to
89dac86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds source-based multimodal projector loading for issue #133:
LlamaEngine.loadMultimodalProjectorSource(...)ModelSource/ resolver / download-cache layer used byloadModelSource(...)Fixes #133.
Validation
dart test test/unit/core/engine/engine_test.dartpassed on VM and Chrome during implementationdart test -p vm test/unit/core/engine/engine_test.dartpasseddart analyze lib/src/core/engine/engine.dart test/unit/core/engine/engine_test.dartpasseddart format --output=none --set-exit-if-changed lib/src/core/engine/engine.dart test/unit/core/engine/engine_test.dartpassedgit diff --checkpassedmmproj-LFM2-VL-450M-Q8_0.ggufoverhttp://127.0.0.1:7359/dart run .dart_tool/e2e/mmproj_source_real_model_smoke.dart /opt/UnitySrc/personal/llama/llamadart/models/LFM2-VL-450M-Q4_0.gguf http://127.0.0.1:7359/mmproj-LFM2-VL-450M-Q8_0.gguf /private/tmp/llamadart-ticket-133/.dart_tool/e2e/mmproj-cacheRESULT mmproj_source_real_model {"supportsVision":true,...}Follow-up
A higher-level atomic
loadModelBundle(...)API with rollback semantics and independent model/projector cache options remains separate from this scoped projector-source API.