diff --git a/CHANGELOG.md b/CHANGELOG.md index 200f109f5..f169b2932 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,14 @@ - Add a notebook in `example/laya_tetris/training/` that fine-tunes a Laya decision head for the Tetris example and exports it for `DecisionEngine` ([#604](https://github.com/leehack/llamadart/issues/604)). +- Run `DecisionEngine` on WebGPU through the bridge decision API + (apiVersion 1), which bridge assets `v0.1.47+` include + ([#604](https://github.com/leehack/llamadart/issues/604)). +* Aligned the default WebGPU bridge assets to `v0.1.47` for the decision API, + retaining Web/native llama.cpp + `v0.4.1@b29c606e28a01b1bc8c1351026a0fa6e616bf6c4` parity and Web + `@litert-lm/core@0.15.0`. Immutable Web asset manifest: + `9c5e9008d187690e283f37b2b892396da03e3c71bf7c3434d7c87ceca8e6a4bc`. - Extend the GGUF speech-to-text validation pack with four synthetic edge fixtures built in-process, so no extra audio is stored: generated digital silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second diff --git a/README.md b/README.md index d5f29b921..ab80d75b7 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,9 @@ models through LiteRT-LM. - Experimental Laya-style decision models on native llama.cpp through `DecisionEngine`: typed choice, score, and yes/no answers from a ModernBERT encoder GGUF and a safetensors head, one encoder pass per question; validated - on macOS (Metal, CPU), other native platforms untested. + on macOS (Metal, CPU), other native platforms untested. Web runs it through + WebGPU bridge assets `v0.1.47+`, which the default Web pin includes; checked + only in headless Chromium on macOS. Unsupported runtime/option combinations are rejected explicitly instead of silently degrading. Check the support matrix before relying on a capability for @@ -158,7 +160,7 @@ Current default runtime pins: | --- | --- | | Native llama.cpp / GGUF | `leehack/llamadart-native@v0.4.1-1` | | Native LiteRT-LM / `.litertlm` | `leehack/litert-lm-native@v0.17.0-6` | -| Web llama.cpp / GGUF | `leehack/llama-web-bridge-assets@v0.1.44` | +| Web llama.cpp / GGUF | `leehack/llama-web-bridge-assets@v0.1.47` | | Web LiteRT-LM / `.litertlm` | `@litert-lm/core@0.15.0` | Native overrides accept stable `vMAJOR.MINOR.PATCH` releases and preserve diff --git a/doc/decision_engine.md b/doc/decision_engine.md index a6fc2c6d5..c14f49b3b 100644 --- a/doc/decision_engine.md +++ b/doc/decision_engine.md @@ -46,6 +46,8 @@ DecisionEngine (core, pure Dart) BackendDecision (backend.dart, web-safe value types) NativeAutoBackend -> NativeLlamaBackend -> worker isolate -> LlamaCppService private encoder llama_context + safetensors head + ggml head graph + WebAutoBackend -> WebGpuLlamaBackend -> WebGpuDecisionHeads -> llama-web-bridge + bridge decision API 1: private encoder context + head in the WASM core raw marker logits + raw act logits -> decoder (core) -> DecisionResult ``` @@ -89,16 +91,14 @@ abstract class BackendDecision { - `BackendDecisionOutput`: per sequence, raw marker `logits` and raw `actLogits` (`Float32List`). -`NativeAutoBackend` implements and forwards it; the LiteRT-LM delegate reports -unsupported. `WebAutoBackend` does not implement it until the bridge ships the -module, so the engine hook reports unsupported and `DecisionEngine.load` throws -`LlamaUnsupportedException`. +`NativeAutoBackend` and `WebAutoBackend` implement and forward it; their +LiteRT-LM delegates report unsupported. ### Engine hooks (`lib/src/core/engine/engine.dart`) Plain public methods documented as low-level integration hooks, like the TTS trio. The capabilities hook checks `is! BackendDecision` before readiness, so -Web reports a stable reason without a model. +a backend without the contract reports a stable reason without a model. Backend handles are not unique over an engine's life: the worker numbers handles from 1, and a new worker starts after `LlamaEngine.dispose` followed by @@ -184,6 +184,12 @@ static helper that unit tests cover: The run path rejects a sequence longer than `llama_n_ubatch` before `llama_encode`, whose `GGML_ASSERT` would abort the process. +`validateDecisionSequences` checks every sequence before the first encoder +pass: 1 to `n_ubatch` tokens inside the vocabulary, and 1 to token-count +markers inside the sequence. The bridge core runs the same checks with the same +messages, plus a question type check that `DecisionQuestionType` always passes, +so both runtimes reject the same input; the marker-count bound comes from the +bridge, whose head graph sizes its buffers by marker count. Windows: `llama.dll` exports no `ggml_*` graph symbols; they live in `ggml-base.dll` (ops, graph, sched, buffers) and `ggml.dll` (registry). The head @@ -195,6 +201,63 @@ and the generated bindings on other platforms. The bindings leave out on their default asset, `package:llamadart/llamadart`, there too. Generated bindings are not edited. +### Web (`lib/src/backends/webgpu/`) + +`WebGpuLlamaBackend` implements `BackendDecision` through `WebGpuDecisionHeads` +(`webgpu_decision.dart`), which calls the llama-web-bridge decision API: +`getDecisionCapabilities`, `loadDecisionHead`, `runDecision` and +`freeDecisionHead` (bridge `docs/api.md`, "Decision heads"). The bridge runs the +head on WebGPU when the model loaded with GPU layers and on the CPU otherwise, +and reports which as `deviceName`. + +- Capability probe: a bridge object without all four methods reports + unsupported with "Web decision models need llama-web-bridge assets v0.1.47+ + with the decision API (apiVersion 1)", from the + `webGpuDecisionBridgeRequirement` constant. A capability or head response + with an `apiVersion` other than 1 is unsupported too, and such a head is + freed first. Bridge assets `v0.1.47+`, the default pin among them, have the + API. +- Paths are URLs, resolved in Dart against `document.baseURI` before any + fetch, so a page's `` applies to both in both bridge modes. The + bridge fetches `headPath`. It takes the config only as text, so `configPath` + is fetched in the page with `fetch`, before the head, and passed as + `configJson`; with both a missing config and a bad head, Web reports the + config where native reports the head. A failed fetch or an HTTP error is + `LlamaModelException` "Cannot read the decision head config at ." with + the status or error in `details`. URLs in error messages and details drop + user info, query and fragment, including URLs that browser and bridge + errors quote. +- Handles: the backend numbers heads itself, never reusing a number, and maps + each to the bridge instance and bridge handle that loaded it. `modelFree` and + `dispose` dispose the bridge, and a model load on the same bridge frees every + bridge head, so the backend forgets all heads at each. A run with a forgotten + head, or with a head whose bridge is no longer active, throws + `LlamaStateException` without calling the bridge; a free does nothing. +- Errors: the bridge rejects with plain `Error`s that carry the core's message + and no status code, so the mapping reads the message after stripping the + bridge's `Failed to load decision head: ` or `Decision run failed: ` prefix. + "Load the decision head again" (a freed head, or one lost to a worker + failure, which also forgets the head), "No model loaded", "Bridge has been + disposed", "was cancelled" and "during active generation" map to + `LlamaStateException`, from the capability probe too; "decision encoder + context" to `LlamaContextException`; anything else to unsupported for the + probe, `LlamaModelException` for a load (head URL in `details`), + `LlamaInferenceException` for a run and `LlamaStateException` for a free. + Load errors that ask bridge callers to pass `configJson` name `configPath` + or the config URL instead. Without an active bridge, the probe reports + unsupported and a load throws `LlamaStateException`, as native does for an + unloaded model. A malformed head description or output is + `LlamaDecisionException`, like native's unexpected worker responses. +- Numbers: Web numbers cannot tell `30.0` from `30`, so `pythonJsonDumps` + writes an integral double in a non-`String` state, instructions, criteria or + levels as an int (`30` where Python writes `30.0`). The tokens then differ + from native and Laya; the guide's Web section tells users to pass such + values as `String`s when parity matters. +- The bridge serializes decision calls with its other operations and cannot + cancel a run. When its worker fails during a run, it reloads the model on the + main thread and rejects the run; the engine keeps its model, and the + `DecisionEngine` must be loaded again. + ## Parity rules Sequence (`build_sequence`, `max_len` 512, `head_max_len` 192): @@ -261,7 +324,8 @@ JSON-like (null, bool, num, String, List, Map with String keys). | Linux | CPU, Vulkan, CUDA | expected, untested | | Windows | CPU, Vulkan, CUDA | expected through the `ggml-base` twins, untested | | Native LiteRT-LM | - | `LlamaUnsupportedException` | -| Web (WebGPU bridge) | - | `LlamaUnsupportedException` until the bridge module ships | +| Web (WebGPU bridge) | WebGPU or CPU (WASM) | bridge assets `v0.1.47+` (apiVersion 1), the default pin among them; older assets report `LlamaUnsupportedException`. CI uses a fake bridge; checked locally with a real model ([Web check](#web-check)) | +| LiteRT-LM Web | - | `LlamaUnsupportedException` | Real-model evidence is macOS only. The CPU head unit tests carry no `local-only` tag, so CI's Linux VM job and its macOS and Windows native test @@ -317,6 +381,32 @@ On Metal, disposing the engine with a head still loaded exits cleanly; skipping the head frees in `freeModel` and `dispose` makes the same exit abort in `ggml_metal_rsets_free`. +### Web check + +Local only, not in CI: `DecisionEngine` through `LlamaEngine(LlamaBackend())` +in Playwright's headless Chromium on the same machine, with the pinned bridge +assets (bridge source `64ba8250`), the 24 fixture rows, +`laya-head.safetensors` and the tolerances of `decision-model-smoke`. Token ids +and markers matched on every row. + +| Backbone | Bridge runtime | Head device | Logit diff | Probability diff | Score diff | +| --- | --- | --- | --- | --- | --- | +| `laya-Q8_0.gguf` | WebGPU; worker and main thread on wasm64 and wasm32 | WebGPU | 0.1636 | 0.0436 | 0.0247 | +| F16 (local conversion) | WebGPU; worker and main thread on wasm64 | WebGPU | 0.0169 | 0.0046 | 0.0013 | +| F16 (local conversion) | WASM CPU; worker and main thread on wasm64 | CPU | 0.0149 | 0.0039 | 0.0028 | +| `laya-Q8_0.gguf` | WASM CPU; worker and main thread on wasm64 | CPU | 0.2326 | 0.0628 | 0.1224 | + +Q8_0 on the WASM CPU misses the probability and score tolerances on one row, +`plain_text/urgency5`, with the same top option. The bridge's own smoke, which +calls the bridge directly, gets the same worst logit difference on wasm32 and +wasm64 in both bridge modes, so the drift comes from the bridge's WASM CPU +Q8_0 path rather than llamadart. +Typed key reads with the question identity check, sequence validation +messages, error mapping, URL redaction, `` resolution, and heads +freed or bridges disposed behind the engine's back were checked against the +same assets. The previous pin, `v0.1.44`, which lacks the API, reported +unsupported with the actionable reason in both bridge modes. + ## Known limits User-facing limits are listed under @@ -346,7 +436,21 @@ guide. order; the service's load-time check helpers, head device choice, sequence validation, and run order through a substituted encoder; worker, backend-client and router routing with fakes; engine hooks and facade with a - fake backend; Web unsupported path under `@TestOn('browser')`. + fake backend. +- Unit (Chrome): `WebGpuDecisionHeads` against a fake bridge + (`test/support/fake_webgpu_decision_bridge.dart`): the capability probe for + old assets, API version skew, bridge reasons and state rejections; head + loading with page-fetched configs, unreadable ones, URLs resolved against a + ``, and credentials and queries kept out of errors; error mapping, + including the `configJson` wording; handle scoping to the loading bridge; + malformed responses. `WebGpuLlamaBackend` without an active bridge, and + forgetting heads on `modelFree`, a same-bridge model load and `dispose`; + `WebAutoBackend` forwarding and LiteRT-LM Web reporting unsupported; the + engine hook without a model. +- Integration (Chrome, fake bridge): `DecisionEngine` through `LlamaEngine`, + `WebAutoBackend` and `WebGpuLlamaBackend`: answers, typed key reads with the + question identity check, sequence layout, page-fetched config, old assets, + API version skew, a cancelled capability probe, and a model unload. - Integration (VM, CI's `stories15M.gguf`): a llama-architecture model is reported unsupported and `DecisionEngine.load` fails before reading the head. - Local-only E2E `test/e2e/backends/decision_engine_e2e_test.dart`: real GGUF diff --git a/doc/webgpu_bridge.md b/doc/webgpu_bridge.md index bf89a372a..84a59bc38 100644 --- a/doc/webgpu_bridge.md +++ b/doc/webgpu_bridge.md @@ -19,21 +19,22 @@ pipelines. `https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@/llama_webgpu_bridge.js` 2. Local fallback: `./webgpu_bridge/llama_webgpu_bridge.js` -Default pinned tag in the example is `v0.1.44`. +Default pinned tag in the example is `v0.1.47`. That release embeds llama.cpp `v0.4.1`, matching the `hook/build.dart` native pin (`v0.4.1-1`, both built from upstream llama.cpp `v0.4.1@b29c606e28a01b1bc8c1351026a0fa6e616bf6c4`) -even though the bridge asset tag `v0.1.44` differs from the native runtime tag -`v0.4.1-1`. Provenance for this immutable consumer artifact: release `389783936`, -tag commit `fdafd9f8cbdb9bf99c359536595eff9a23095379`, bridge source -`89178be67c3c84300bc1b129182bd5bc5a8e21fc`, and manifest SHA-256 -`8d61f453753ac7a7d839ac12318b70986a814748d86029993118c19454293aa9`. The bridge +even though the bridge asset tag `v0.1.47` differs from the native runtime tag +`v0.4.1-1`. Provenance for this immutable consumer artifact: release `394986324`, +tag commit `ee45e864641648a99411128f9bb82b7897fad221`, bridge source +`64ba8250871bf2472cc2064c6a00fff050783f02`, and manifest SHA-256 +`9c5e9008d187690e283f37b2b892396da03e3c71bf7c3434d7c87ceca8e6a4bc`. The bridge assets were qualified against native `v0.4.1`; `v0.4.1-1` rebuilds it from the -same upstream commit. It retains the Qwen3-ASR typed speech-to-text contract -introduced in `v0.1.30` and provisions the explicit 1 MiB Wasm stack needed for -memory64 context construction in direct and worker modes. The chat bootstrap opts -`SpeechToTextEngine` into that contract from the immutable tag; older or custom -assets remain disabled unless the host explicitly sets +same upstream commit. The assets add the decision API (apiVersion 1), retain the +Qwen3-ASR typed speech-to-text contract introduced in `v0.1.30`, and provision +the explicit 1 MiB Wasm stack needed for memory64 context construction in +direct and worker modes. The chat bootstrap opts `SpeechToTextEngine` into that +contract from the immutable tag; older or custom assets remain disabled unless +the host explicitly sets `window.__llamadartBridgeSpeechToTextSupported = true` after equivalent validation. @@ -48,7 +49,7 @@ model bytes. To vendor pinned assets into local app web files: ```bash -WEBGPU_BRIDGE_ASSETS_TAG=v0.1.44 ./scripts/fetch_webgpu_bridge_assets.sh +WEBGPU_BRIDGE_ASSETS_TAG=v0.1.47 ./scripts/fetch_webgpu_bridge_assets.sh ``` Optional compatibility env vars: @@ -129,7 +130,7 @@ You can override CDN source/version before the bridge loader runs: ```html ``` @@ -179,6 +180,10 @@ window.LlamaWebGpuBridge = class LlamaWebGpuBridge { - `cancel()` - `dispose()` - `applyChatTemplate(messages, addAssistant, customTemplate)` +- `getDecisionCapabilities()` +- `loadDecisionHead(url, { configJson })` +- `runDecision(handle, sequences)` +- `freeDecisionHead(handle)` - `isGpuActive()` - `getBackendName()` @@ -187,6 +192,8 @@ window.LlamaWebGpuBridge = class LlamaWebGpuBridge { - Web backend remains GGUF URL-based (`modelLoadFromUrl`). - If bridge activation fails, model loading fails (no alternate web backend). - Embeddings on web require bridge assets with embedding APIs (`v0.1.7+`). +- `DecisionEngine` on web requires bridge assets with the decision API + (apiVersion 1, `v0.1.47+`). - State persistence on web requires bridge assets with state APIs (`v0.1.15+`); paths are bridge WASMFS virtual paths and are not durable across page reloads. Durable browser storage currently requires app-level export/import outside the diff --git a/example/chat_app/web/index.html b/example/chat_app/web/index.html index 43d5afe96..7024d56fb 100644 --- a/example/chat_app/web/index.html +++ b/example/chat_app/web/index.html @@ -211,7 +211,7 @@ typeof configuredRepo === 'string' && configuredRepo.length > 0 ? configuredRepo : 'leehack/llama-web-bridge-assets'; - const defaultBridgeAssetsTag = 'v0.1.44'; + const defaultBridgeAssetsTag = 'v0.1.47'; const defaultBridgeLlamaCppTag = 'v0.4.1'; const bridgeAssetsTag = typeof configuredTag === 'string' && configuredTag.length > 0 diff --git a/lib/src/backends/llama_cpp/llama_cpp_service.dart b/lib/src/backends/llama_cpp/llama_cpp_service.dart index 0bb36e476..9d3c07843 100644 --- a/lib/src/backends/llama_cpp/llama_cpp_service.dart +++ b/lib/src/backends/llama_cpp/llama_cpp_service.dart @@ -7859,8 +7859,10 @@ class LlamaCppService { /// Checks [sequences] against a decision head's limits. /// /// Each sequence needs 1 to [tokenLimit] tokens, each in `[0, vocabSize)`, - /// at least one marker, and every marker a position in its tokens. Throws - /// [LlamaInferenceException] naming the first sequence that fails. + /// 1 to token-count markers, and every marker a position in its tokens. + /// Throws [LlamaInferenceException] naming the first sequence that fails. + /// The llama-web-bridge decision core runs these checks with the same + /// messages, so both runtimes reject the same sequences. static void validateDecisionSequences( List sequences, { required int tokenLimit, @@ -7888,6 +7890,13 @@ class LlamaCppService { 'Decision sequence $i has no option markers.', ); } + if (sequence.markers.length > tokens.length) { + throw LlamaInferenceException( + 'Decision sequence $i has ${sequence.markers.length} markers for ' + 'its ${tokens.length} tokens; a sequence holds at most one marker ' + 'per token.', + ); + } for (final marker in sequence.markers) { if (marker < 0 || marker >= tokens.length) { throw LlamaInferenceException( diff --git a/lib/src/backends/web/web_backend.dart b/lib/src/backends/web/web_backend.dart index f9e4b8e6e..982e4a879 100644 --- a/lib/src/backends/web/web_backend.dart +++ b/lib/src/backends/web/web_backend.dart @@ -21,8 +21,13 @@ class WebAutoBackend BackendGrammarConstraintsSupport, BackendDeferredEngineCreation, BackendTextToSpeech, + BackendDecision, BackendStatePersistence, BackendStatePersistenceSupport { + static const String _decisionUnsupportedMessage = + 'The active Web runtime does not run decision models. Load a ' + 'ModernBERT encoder GGUF, which uses the llama.cpp WebGPU bridge.'; + final LlamaBackend Function() _webGpuFactory; final LlamaBackend Function() _liteRtLmFactory; @@ -216,6 +221,58 @@ class WebAutoBackend } } + @override + Future decisionCapabilities(int modelHandle) { + final delegate = _requireDelegate(); + if (delegate is! BackendDecision) { + return Future.value( + const BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: _decisionUnsupportedMessage, + ), + ); + } + return (delegate as BackendDecision).decisionCapabilities(modelHandle); + } + + @override + Future decisionHeadLoad( + int modelHandle, + String headPath, { + String? configPath, + }) { + final delegate = _requireDelegate(); + if (delegate is! BackendDecision) { + throw LlamaUnsupportedException(_decisionUnsupportedMessage); + } + return (delegate as BackendDecision).decisionHeadLoad( + modelHandle, + headPath, + configPath: configPath, + ); + } + + @override + Future> decisionRun( + int headHandle, + List sequences, + ) { + final delegate = _requireDelegate(); + if (delegate is! BackendDecision) { + throw LlamaUnsupportedException(_decisionUnsupportedMessage); + } + return (delegate as BackendDecision).decisionRun(headHandle, sequences); + } + + @override + Future decisionHeadFree(int headHandle) { + final delegate = _delegate; + if (delegate is BackendDecision) { + return (delegate as BackendDecision).decisionHeadFree(headHandle); + } + return Future.value(); + } + @override Future> embed( int contextHandle, diff --git a/lib/src/backends/webgpu/interop.dart b/lib/src/backends/webgpu/interop.dart index 10fe2d5e2..1d216a2ee 100644 --- a/lib/src/backends/webgpu/interop.dart +++ b/lib/src/backends/webgpu/interop.dart @@ -53,6 +53,24 @@ extension type LlamaWebGpuBridge._(JSObject _) implements JSObject { WebGpuTextToSpeechOptions options, ); + /// Returns decision-head support for the loaded model. + external JSPromise? getDecisionCapabilities(); + + /// Loads a decision head from a URL for the loaded model. + external JSPromise? loadDecisionHead( + String url, [ + WebGpuDecisionHeadOptions? options, + ]); + + /// Runs decision sequences through the encoder and the head [handle]. + external JSPromise? runDecision( + int handle, + JSArray sequences, + ); + + /// Frees the decision head [handle]; unknown handles are ignored. + external JSPromise? freeDecisionHead(int handle); + /// Tokenizes text. external JSPromise? tokenize(String text, [bool? addSpecial]); @@ -225,3 +243,86 @@ extension type WebGpuTextToSpeechOptions._(JSObject _) implements JSObject { JSFunction? onProgress, }); } + +/// Decision-head support reported by `getDecisionCapabilities`. +@JS() +@anonymous +extension type WebGpuDecisionCapabilities._(JSObject _) implements JSObject { + /// Decision API version of the bridge. + external JSAny? get apiVersion; + + /// Whether the loaded model can run decision heads. + external JSAny? get supported; + + /// Why the loaded model cannot run decision heads. + external JSAny? get reason; +} + +/// Decision-head load options. +@JS() +@anonymous +extension type WebGpuDecisionHeadOptions._(JSObject _) implements JSObject { + /// Creates head load options. + /// + /// [configJson] is Laya's `rl_agent_config.json` text; without it the + /// bridge reads the head's `laya.config` metadata. + external factory WebGpuDecisionHeadOptions({ + @JS('configJson') String? configJson, + @JS('onProgress') JSFunction? onProgress, + }); +} + +/// A decision head loaded by `loadDecisionHead`. +@JS() +@anonymous +extension type WebGpuDecisionHeadInfo._(JSObject _) implements JSObject { + /// Decision API version of the bridge. + external JSAny? get apiVersion; + + /// Bridge handle of the head. + external JSAny? get handle; + + /// Hidden size shared by the encoder and the head. + external JSAny? get hiddenSize; + + /// Token that starts every sequence. + external JSAny? get clsToken; + + /// Token that separates sequence parts. + external JSAny? get sepToken; + + /// Token placed before each option. + external JSAny? get maskToken; + + /// Text of the mask token. + external JSAny? get maskText; + + /// The head's Laya config as JSON text. + external JSAny? get configJson; + + /// Name of the device the head runs on. + external JSAny? get deviceName; +} + +/// Encoder input for one question, as `runDecision` takes it. +@JS() +@anonymous +extension type WebGpuDecisionSequence._(JSObject _) implements JSObject { + /// Creates an encoder input. + external factory WebGpuDecisionSequence({ + required JSInt32Array tokens, + required JSInt32Array markers, + @JS('questionType') required int questionType, + }); +} + +/// Raw head outputs for one sequence, as `runDecision` returns them. +@JS() +@anonymous +extension type WebGpuDecisionOutput._(JSObject _) implements JSObject { + /// One raw logit per marker. + external JSAny? get logits; + + /// Action-head logits. + external JSAny? get actLogits; +} diff --git a/lib/src/backends/webgpu/webgpu_backend.dart b/lib/src/backends/webgpu/webgpu_backend.dart index d32987cd0..d6a79afc2 100644 --- a/lib/src/backends/webgpu/webgpu_backend.dart +++ b/lib/src/backends/webgpu/webgpu_backend.dart @@ -17,6 +17,7 @@ import '../../core/models/inference/generation_params.dart'; import '../../core/models/inference/model_params.dart'; import '../backend.dart'; import 'interop.dart'; +import 'webgpu_decision.dart'; import 'webgpu_load_retry_policy.dart'; @JS('Object.keys') @@ -30,6 +31,7 @@ class WebGpuLlamaBackend BackendBatchEmbeddings, BackendPromptSpeechToTextSupport, BackendTextToSpeech, + BackendDecision, BackendStatePersistence, BackendStatePersistenceSupport { static const Duration _bridgeReadyTimeout = Duration(seconds: 12); @@ -73,6 +75,7 @@ class WebGpuLlamaBackend bool _webGpuMultimodalWarmupAttempted = false; bool? _preferMemory64Override; bool? _forceRemoteFetchBackendOverride; + final WebGpuDecisionHeads _decisionHeads = WebGpuDecisionHeads(); /// Creates a bridge-backed web backend. WebGpuLlamaBackend({ @@ -313,6 +316,7 @@ class WebGpuLlamaBackend final abortController = _abortController; _bridge = null; _abortController = null; + _decisionHeads.clear(); abortController?.abort(); bridge?.cancel(); if (bridge == null) { @@ -1124,6 +1128,7 @@ class WebGpuLlamaBackend _isReady = true; _mmContextActive = false; + _decisionHeads.clear(); _resetWebGpuMultimodalWarmupState(); return 1; } catch (e) { @@ -2045,6 +2050,44 @@ class WebGpuLlamaBackend _bridge?.cancel(); } + /// Probes the active bridge for decision heads. + /// + /// Reports unsupported without an active bridge, and for bridge assets + /// without the decision API or with a decision API version other than 1. + @override + Future decisionCapabilities(int modelHandle) { + return _decisionHeads.capabilities(_activeBridge); + } + + /// Loads a decision head into the active bridge. + /// + /// [headPath] and [configPath] are URLs that resolve against the document + /// base URL. The config is fetched here and passed to the bridge as text; + /// the bridge fetches the head. + @override + Future decisionHeadLoad( + int modelHandle, + String headPath, { + String? configPath, + }) { + return _decisionHeads.load(_activeBridge, headPath, configUrl: configPath); + } + + @override + Future> decisionRun( + int headHandle, + List sequences, + ) { + return _decisionHeads.run(_activeBridge, headHandle, sequences); + } + + @override + Future decisionHeadFree(int headHandle) { + return _decisionHeads.free(_activeBridge, headHandle); + } + + LlamaWebGpuBridge? get _activeBridge => _usingBridge ? _bridge : null; + @override Future> embed( int contextHandle, diff --git a/lib/src/backends/webgpu/webgpu_decision.dart b/lib/src/backends/webgpu/webgpu_decision.dart new file mode 100644 index 000000000..dbeac6e74 --- /dev/null +++ b/lib/src/backends/webgpu/webgpu_decision.dart @@ -0,0 +1,459 @@ +import 'dart:js_interop'; +import 'dart:js_interop_unsafe'; + +import 'package:web/web.dart' show Response, URL, document, window; + +import '../../core/exceptions.dart'; +import '../backend.dart'; +import 'interop.dart'; + +/// Decision API version that [WebGpuDecisionHeads] speaks. +const int webGpuDecisionApiVersion = 1; + +/// Bridge assets that [WebGpuDecisionHeads] needs, as named in errors. +const String webGpuDecisionBridgeRequirement = + 'llama-web-bridge assets v0.1.47+ with the decision API ' + '(apiVersion $webGpuDecisionApiVersion)'; + +/// Decision heads loaded through the llama.cpp WebGPU bridge. +/// +/// Handles are this object's own and never reused. Each head belongs to the +/// bridge instance that loaded it: once the backend replaces or disposes that +/// bridge, or calls [clear], [run] throws [LlamaStateException] and [free] +/// does nothing. +class WebGpuDecisionHeads { + final Map _heads = {}; + int _nextHandle = 1; + + /// Probes [bridge] for decision heads on its loaded model. + /// + /// [bridge] is the backend's active bridge, or null when it has none, which + /// reports that no model is loaded. Bridges without the decision methods, a + /// capability response with an `apiVersion` other than + /// [webGpuDecisionApiVersion], and a failed probe report unsupported with an + /// actionable reason. Throws [LlamaStateException] when the bridge rejects + /// the probe for its state: disposed, cancelled, or without a model. + Future capabilities( + LlamaWebGpuBridge? bridge, + ) async { + if (bridge == null) { + return const BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: + 'No model is loaded on the Web bridge. Load a ModernBERT encoder ' + 'GGUF first.', + ); + } + if (!_exposesDecisionApi(bridge)) { + return const BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: + 'Web decision models need $webGpuDecisionBridgeRequirement; the ' + 'loaded bridge does not expose it.', + ); + } + final JSAny? raw; + try { + raw = await _settle(bridge.getDecisionCapabilities()); + } catch (error) { + final exception = _bridgeException( + _errorText(error), + fallback: LlamaUnsupportedException.new, + ); + if (exception is LlamaStateException) throw exception; + return BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: + 'The Web decision capability probe failed: ${exception.message}', + ); + } + if (raw == null || !raw.isA()) { + return const BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: 'The Web decision capability response is invalid.', + ); + } + final value = raw as WebGpuDecisionCapabilities; + final apiVersion = _int(value.apiVersion); + if (apiVersion != webGpuDecisionApiVersion) { + return BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: _apiVersionSkew(apiVersion), + ); + } + if (_bool(value.supported) != true) { + return BackendDecisionCapabilities( + isSupported: false, + unsupportedReason: + _string(value.reason) ?? + 'The loaded Web model does not support decision heads.', + ); + } + return const BackendDecisionCapabilities(isSupported: true); + } + + /// Loads the decision head at [headUrl] into [bridge]. + /// + /// [bridge] is the backend's active bridge, or null when it has none. Both + /// URLs resolve against the document base URL. [configUrl], when given, is + /// fetched here before the head and passed to the bridge as text; the + /// bridge fetches the head. URLs in errors drop user info, query and + /// fragment. Throws [LlamaStateException] when [bridge] is null or rejects + /// the probe or load for its state: disposed, busy, cancelled, or without a + /// model; [LlamaUnsupportedException] when [capabilities] reports + /// unsupported or the head reports another decision API version; + /// [LlamaModelException] when the head or config cannot be fetched, is + /// malformed, or does not fit the encoder; [LlamaContextException] when the + /// head's encoder context cannot be created; and [LlamaDecisionException] + /// for a malformed bridge response. + Future load( + LlamaWebGpuBridge? bridge, + String headUrl, { + String? configUrl, + }) async { + if (bridge == null) { + throw LlamaStateException( + 'No model is loaded on the Web bridge. Load the decision encoder ' + 'before its head.', + ); + } + final capabilities = await this.capabilities(bridge); + if (!capabilities.isSupported) { + throw LlamaUnsupportedException( + capabilities.unsupportedReason ?? + 'The loaded Web model does not support decision heads.', + ); + } + final resolvedHeadUrl = _resolveUrl(headUrl); + final resolvedConfigUrl = configUrl == null ? null : _resolveUrl(configUrl); + final configJson = resolvedConfigUrl == null + ? null + : await _fetchConfigText(resolvedConfigUrl); + + final JSAny? raw; + try { + raw = await _settle( + bridge.loadDecisionHead( + resolvedHeadUrl, + WebGpuDecisionHeadOptions(configJson: configJson), + ), + ); + } catch (error) { + var message = _errorText( + error, + ).replaceAll('Pass configJson ', 'Pass configPath '); + if (resolvedConfigUrl != null) { + message = message.replaceAll( + 'config in configJson ', + 'config in ${_displayUrl(resolvedConfigUrl)} ', + ); + } + throw _bridgeException( + message, + fallback: (message) => + LlamaModelException(message, _displayUrl(resolvedHeadUrl)), + ); + } + + final info = raw != null && raw.isA() + ? raw as WebGpuDecisionHeadInfo + : null; + final bridgeHandle = info == null ? null : _int(info.handle); + Future release() async { + if (bridgeHandle == null || bridgeHandle <= 0) return; + try { + await _settle(bridge.freeDecisionHead(bridgeHandle)); + } catch (_) {} + } + + final apiVersion = info == null ? null : _int(info.apiVersion); + if (info != null && apiVersion != webGpuDecisionApiVersion) { + await release(); + throw LlamaUnsupportedException(_apiVersionSkew(apiVersion)); + } + final hiddenSize = info == null ? null : _int(info.hiddenSize); + final clsToken = info == null ? null : _int(info.clsToken); + final sepToken = info == null ? null : _int(info.sepToken); + final maskToken = info == null ? null : _int(info.maskToken); + final maskText = info == null ? null : _rawString(info.maskText); + final configText = info == null ? null : _rawString(info.configJson); + final deviceName = info == null ? null : _rawString(info.deviceName); + if (bridgeHandle == null || + bridgeHandle <= 0 || + hiddenSize == null || + clsToken == null || + sepToken == null || + maskToken == null || + maskText == null || + configText == null || + deviceName == null) { + await release(); + throw LlamaDecisionException( + 'The Web decision runtime returned a malformed head description.', + ); + } + + final handle = _nextHandle++; + _heads[handle] = _WebGpuDecisionHead(bridge, bridgeHandle); + return BackendDecisionHeadInfo( + handle: handle, + hiddenSize: hiddenSize, + clsToken: clsToken, + sepToken: sepToken, + maskToken: maskToken, + maskText: maskText, + configJson: configText, + deviceName: deviceName, + ); + } + + /// Runs [sequences] through the head [handle] on [bridge], in order. + /// + /// [bridge] is the backend's active bridge, or null when it has none. + /// Throws [LlamaStateException] when the head is not loaded on [bridge], + /// including when the bridge lost it, which also forgets the head, and when + /// the bridge has no model or is busy; [LlamaInferenceException] when a + /// sequence is invalid or the encoder or head pass fails; and + /// [LlamaDecisionException] for malformed outputs. + Future> run( + LlamaWebGpuBridge? bridge, + int handle, + List sequences, + ) async { + final head = _heads[handle]; + if (head == null || bridge == null || !identical(head.bridge, bridge)) { + _heads.remove(handle); + throw LlamaStateException( + 'Decision head $handle is not loaded on this Web runtime; it was ' + 'freed, its model was unloaded, or the bridge restarted. Load the ' + 'decision head again.', + ); + } + final input = [ + for (final sequence in sequences) + WebGpuDecisionSequence( + tokens: sequence.tokens.toJS, + markers: sequence.markers.toJS, + questionType: sequence.questionType.index, + ), + ].toJS; + + final JSAny? raw; + try { + raw = await _settle(bridge.runDecision(head.bridgeHandle, input)); + } catch (error) { + final exception = _bridgeException( + _errorText(error), + fallback: LlamaInferenceException.new, + ); + if (exception.message.contains(_reloadHint)) { + _heads.remove(handle); + } + throw exception; + } + return _parseOutputs(raw); + } + + /// Frees the head [handle] on [bridge]. + /// + /// Does nothing when the head is not loaded on [bridge]. Bridge failures + /// throw a [LlamaException]. + Future free(LlamaWebGpuBridge? bridge, int handle) async { + final head = _heads.remove(handle); + if (head == null || bridge == null || !identical(head.bridge, bridge)) { + return; + } + try { + await _settle(bridge.freeDecisionHead(head.bridgeHandle)); + } catch (error) { + throw _bridgeException( + _errorText(error), + fallback: LlamaStateException.new, + ); + } + } + + /// Forgets every head, for when the bridge freed them all. + void clear() => _heads.clear(); + + static const String _reloadHint = 'Load the decision head again'; + static final RegExp _absoluteUrl = RegExp( + r'[A-Za-z][A-Za-z0-9+.-]*://[^\s"<>]+', + ); + + static bool _exposesDecisionApi(LlamaWebGpuBridge bridge) { + for (final name in const [ + 'getDecisionCapabilities', + 'loadDecisionHead', + 'runDecision', + 'freeDecisionHead', + ]) { + if (!bridge.getProperty(name.toJS).isA()) { + return false; + } + } + return true; + } + + static String _apiVersionSkew(int? apiVersion) => + 'The Web bridge implements decision API version ' + '${apiVersion ?? 'unknown'}; llamadart needs ' + '$webGpuDecisionBridgeRequirement.'; + + static List _parseOutputs(JSAny? raw) { + LlamaDecisionException malformed() => LlamaDecisionException( + 'The Web decision runtime returned malformed outputs.', + ); + if (raw == null || !raw.isA()) throw malformed(); + final outputs = []; + for (final item in (raw as JSArray).toDart) { + final output = item != null && item.isA() + ? _parseOutput(item as WebGpuDecisionOutput) + : null; + if (output == null) throw malformed(); + outputs.add(output); + } + return outputs; + } + + static BackendDecisionOutput? _parseOutput(WebGpuDecisionOutput output) { + final logits = output.logits; + final actLogits = output.actLogits; + if (logits == null || + actLogits == null || + !logits.isA() || + !actLogits.isA()) { + return null; + } + return BackendDecisionOutput( + logits: (logits as JSFloat32Array).toDart, + actLogits: (actLogits as JSFloat32Array).toDart, + ); + } + + static Future _fetchConfigText(String url) async { + final message = + 'Cannot read the decision head config at ${_displayUrl(url)}.'; + final Response response; + try { + response = await window.fetch(url.toJS).toDart; + } catch (error) { + throw LlamaModelException(message, _errorText(error)); + } + if (!response.ok) { + throw LlamaModelException( + message, + 'HTTP ${response.status} ${response.statusText}'.trim(), + ); + } + try { + return (await response.text().toDart).toDart; + } catch (error) { + throw LlamaModelException(message, _errorText(error)); + } + } + + static LlamaException _bridgeException( + String message, { + required LlamaException Function(String message) fallback, + }) { + if (message.contains(_reloadHint) || + message.startsWith('No model loaded') || + message.contains('Bridge has been disposed') || + message.contains('was cancelled') || + message.contains('during active generation')) { + return LlamaStateException(message); + } + if (message.contains('decision encoder context')) { + return LlamaContextException(message); + } + return fallback(message); + } + + static String _errorText(Object error) => _coreMessage( + _bridgeErrorMessage(error), + ).replaceAllMapped(_absoluteUrl, (match) => _displayUrl(match[0]!)); + + static String _resolveUrl(String url) { + if (url.isEmpty) return url; + try { + return URL(url, document.baseURI).href; + } catch (_) { + return url; + } + } + + static String _coreMessage(String message) { + for (final prefix in const [ + 'Failed to load decision head: ', + 'Decision run failed: ', + ]) { + if (message.startsWith(prefix)) { + return message.substring(prefix.length); + } + } + return message; + } + + static Future _settle(JSAny? value) async { + if (value != null && value.isA()) { + return (value as JSPromise).toDart; + } + return value; + } + + static int? _int(JSAny? value) { + if (value == null || !value.isA()) return null; + final number = (value as JSNumber).toDartDouble; + if (!number.isFinite || number != number.truncateToDouble()) return null; + return number.toInt(); + } + + static bool? _bool(JSAny? value) => value != null && value.isA() + ? (value as JSBoolean).toDart + : null; + + static String? _rawString(JSAny? value) => + value != null && value.isA() + ? (value as JSString).toDart + : null; + + static String? _string(JSAny? value) { + final text = _rawString(value); + return text == null || text.isEmpty ? null : text; + } +} + +String _bridgeErrorMessage(Object error) { + try { + final message = (error as JSObject).getProperty('message'.toJS); + if (message != null && message.isA()) { + return (message as JSString).toDart; + } + } catch (_) {} + return error.toString(); +} + +String _displayUrl(String url) { + final uri = Uri.tryParse(url); + if (uri == null) { + final end = url.indexOf(RegExp('[?#]')); + return (end < 0 ? url : url.substring(0, end)).replaceFirst( + RegExp('//[^/]*@'), + '//', + ); + } + return Uri( + scheme: uri.hasScheme ? uri.scheme : null, + host: uri.hasAuthority ? uri.host : null, + port: uri.hasPort ? uri.port : null, + path: uri.path, + ).toString(); +} + +class _WebGpuDecisionHead { + _WebGpuDecisionHead(this.bridge, this.bridgeHandle); + + final LlamaWebGpuBridge bridge; + final int bridgeHandle; +} diff --git a/lib/src/core/decision/decision_engine.dart b/lib/src/core/decision/decision_engine.dart index 27583a999..5d5907835 100644 --- a/lib/src/core/decision/decision_engine.dart +++ b/lib/src/core/decision/decision_engine.dart @@ -61,8 +61,10 @@ class DecisionModelInfo { /// recommended because the decision path does not use the engine's own /// context. /// -/// Supported on native llama.cpp backends. On Web and with the native -/// LiteRT-LM backend, [load] throws [LlamaUnsupportedException]. +/// Supported on native llama.cpp backends, and on Web with llama-web-bridge +/// assets that include the decision API (apiVersion 1). With bridge assets +/// without decision API version 1, and with the LiteRT-LM backends, [load] +/// throws [LlamaUnsupportedException]. /// /// ```dart /// final engine = LlamaEngine(LlamaBackend()); @@ -138,13 +140,16 @@ class DecisionEngine { /// Loads the decision head at [headPath] for the model loaded in [engine]. /// /// [configPath] names Laya's `rl_agent_config.json` for head files without - /// `laya.config` metadata, such as the official checkpoint. Throws - /// [LlamaUnsupportedException] when the backend or model cannot run - /// decision heads; [LlamaModelException] when the head file or its config - /// cannot be read, is malformed, or does not fit the encoder; + /// `laya.config` metadata, such as the official checkpoint. On Web both are + /// URLs resolved against the document base URL: the bridge fetches the + /// head, and the config is fetched in the page and passed to the bridge as + /// text. Throws [LlamaUnsupportedException] when the backend or model + /// cannot run decision heads; [LlamaModelException] when the head file or + /// its config cannot be read, is malformed, or does not fit the encoder; /// [LlamaContextException] when the head's encoder context cannot be /// created; and [LlamaStateException] when the model is unloaded during the - /// load. When a backend returns a head whose config or mask text fails + /// load, or on Web when the bridge rejects the load as disposed, busy or + /// cancelled. When a backend returns a head whose config or mask text fails /// validation, the head is freed and [LlamaDecisionException] is thrown. static Future load( LlamaEngine engine, { @@ -211,7 +216,9 @@ class DecisionEngine { /// encoding escapes it in non-string states. Throws [LlamaStateException] /// after [dispose] or once the engine's model is unloaded. A call running /// during an unload throws it too, unless its sequences already reached the - /// backend; that call returns answers from the unloaded model. + /// backend; that call returns answers from the unloaded model. On Web it is + /// also thrown once the bridge restarts its runtime, which frees the head; + /// load the DecisionEngine again. /// /// To read answers as typed values, build [questions] with /// [DecisionKey.questionsOf] and read them with diff --git a/lib/src/core/engine/engine.dart b/lib/src/core/engine/engine.dart index 07e7d9e36..e4c8e0037 100644 --- a/lib/src/core/engine/engine.dart +++ b/lib/src/core/engine/engine.dart @@ -1359,7 +1359,8 @@ class LlamaEngine { /// metadata. The returned [BackendDecisionHeadInfo.handle] is an engine /// handle that this engine never reuses, not the backend's own handle; pass /// it to [runDecisionBackend] and [freeDecisionHeadBackend]. The head stays - /// usable until it is freed or the model is unloaded. + /// usable until it is freed or the model is unloaded; on Web, a bridge that + /// restarts its runtime frees it too. Future loadDecisionHeadBackend( String headPath, { String? configPath, @@ -1401,7 +1402,7 @@ class LlamaEngine { /// builds the sequences and decodes the outputs. [headHandle] is a handle /// returned by [loadDecisionHeadBackend]. Throws [LlamaStateException] when /// it is not loaded on this engine, such as after it was freed or its model - /// was unloaded. + /// was unloaded, and on Web when a bridge runtime restart freed it. Future> runDecisionBackend( int headHandle, List sequences, diff --git a/scripts/fetch_webgpu_bridge_assets.sh b/scripts/fetch_webgpu_bridge_assets.sh index 15ae2d28e..f285625af 100755 --- a/scripts/fetch_webgpu_bridge_assets.sh +++ b/scripts/fetch_webgpu_bridge_assets.sh @@ -4,7 +4,7 @@ set -euo pipefail ROOT_DIR="$(git rev-parse --show-toplevel)" OUT_DIR="${WEBGPU_BRIDGE_OUT_DIR:-$ROOT_DIR/example/chat_app/web/webgpu_bridge}" ASSETS_REPO="${WEBGPU_BRIDGE_ASSETS_REPO:-leehack/llama-web-bridge-assets}" -ASSETS_TAG="${WEBGPU_BRIDGE_ASSETS_TAG:-v0.1.44}" +ASSETS_TAG="${WEBGPU_BRIDGE_ASSETS_TAG:-v0.1.47}" CDN_BASE="${WEBGPU_BRIDGE_CDN_BASE:-https://cdn.jsdelivr.net/gh/${ASSETS_REPO}@${ASSETS_TAG}}" PATCH_SAFARI_COMPAT="${WEBGPU_BRIDGE_PATCH_SAFARI_COMPAT:-1}" MIN_SAFARI_VERSION="${WEBGPU_BRIDGE_MIN_SAFARI_VERSION:-170400}" @@ -21,7 +21,7 @@ if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then Downloads prebuilt WebGPU bridge assets into the chat_app web directory. Default source: - https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@v0.1.44 + https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@v0.1.47 Environment variables: WEBGPU_BRIDGE_ASSETS_REPO Asset repo in owner/repo format @@ -35,7 +35,7 @@ Usage: ./scripts/fetch_webgpu_bridge_assets.sh Examples: - WEBGPU_BRIDGE_ASSETS_TAG=v0.1.44 ./scripts/fetch_webgpu_bridge_assets.sh + WEBGPU_BRIDGE_ASSETS_TAG=v0.1.47 ./scripts/fetch_webgpu_bridge_assets.sh WEBGPU_BRIDGE_ASSETS_REPO=acme/llama-web-bridge-assets WEBGPU_BRIDGE_ASSETS_TAG=v2 ./scripts/fetch_webgpu_bridge_assets.sh USAGE exit 0 diff --git a/test/integration/backends/webgpu/webgpu_engine_decision_browser_integration_test.dart b/test/integration/backends/webgpu/webgpu_engine_decision_browser_integration_test.dart new file mode 100644 index 000000000..8308d0303 --- /dev/null +++ b/test/integration/backends/webgpu/webgpu_engine_decision_browser_integration_test.dart @@ -0,0 +1,245 @@ +@TestOn('browser') +library; + +import 'dart:js_interop'; + +import 'package:llamadart/llamadart.dart'; +import 'package:llamadart/src/backends/web/web_backend.dart'; +import 'package:llamadart/src/backends/webgpu/webgpu_backend.dart'; +import 'package:test/test.dart'; +import 'package:web/web.dart' show Blob, BlobPropertyBag, URL; + +import '../../../support/fake_webgpu_decision_bridge.dart'; + +void main() { + late List bridges; + late bool withDecisionApi; + late LlamaEngine engine; + + setUp(() { + bridges = []; + withDecisionApi = true; + engine = LlamaEngine( + WebAutoBackend( + webGpuFactory: () => WebGpuLlamaBackend( + bridgeFactory: ([config]) { + final fake = FakeDecisionBridge( + withDecisionApi: withDecisionApi, + withModelApi: true, + ); + bridges.add(fake); + return fake.bridge; + }, + ), + ), + ); + }); + + tearDown(() => engine.dispose()); + + Future loadModel() => engine.loadModel( + 'laya-Q8_0.gguf', + modelParams: const ModelParams(contextSize: 512), + ); + + final questions = { + 'department': DecisionQuestion.choice( + 'Which department?', + criteria: {'billing': null, 'technical': null, 'other': null}, + ), + 'refund': DecisionQuestion.noul('Refund requested?'), + }; + + test('answers questions through the WebGPU bridge', () async { + await loadModel(); + + final capabilities = await DecisionEngine.capabilitiesFor(engine); + final decisions = await DecisionEngine.load( + engine, + headPath: 'laya-head.safetensors', + ); + final result = await decisions.systemOne( + state: 'Billed twice.', + questions: questions, + ); + await decisions.dispose(); + + final fake = bridges.single; + expect(capabilities.isSupported, isTrue); + expect(capabilities.backendName, 'WebGPU (Fake)'); + expect(decisions.info.maxTokens, 32); + expect(decisions.info.headMaxTokens, 16); + expect(decisions.info.deviceName, 'WebGPU'); + expect(fake.lastSequences, hasLength(2)); + for (final sequence in fake.lastSequences) { + expect(sequence.typedArrays, isTrue); + expect(sequence.tokens.first, 1); + expect(sequence.tokens.last, 2); + expect( + [for (final m in sequence.markers) sequence.tokens[m]], + [for (final _ in sequence.markers) 3], + ); + } + expect(fake.lastSequences.map((s) => s.markers.length), [3, 2]); + expect(fake.lastSequences.map((s) => s.questionType), [0, 2]); + expect(result.choices['department']!.choice, 'billing'); + expect(result.nouls['refund']!.noul, lessThan(0.5)); + expect( + result.usage.inputTokens, + fake.lastSequences.fold(0, (n, s) => n + s.tokens.length), + ); + expect(fake.calls.last, 'free 7'); + expect(fake.liveHandles, isEmpty); + }); + + test('reads typed keys from Web results', () async { + await loadModel(); + final decisions = await DecisionEngine.load( + engine, + headPath: 'laya-head.safetensors', + ); + addTearDown(decisions.dispose); + final department = ChoiceKey.enumOf( + 'department', + 'Which department?', + criteria: { + _Department.billing: null, + _Department.technical: null, + _Department.other: null, + }, + ); + final urgency = ScoreKey.of( + 'urgency', + 'How urgent?', + levels: ['low', 'high'], + ); + final refund = NoulKey.of('refund', 'Refund requested?'); + + final result = await decisions.systemOne( + state: 'Billed twice.', + questions: DecisionKey.questionsOf([department, urgency, refund]), + ); + + expect(bridges.single.lastSequences.map((s) => s.questionType), [0, 1, 2]); + expect(result.questions!['department'], same(department.question)); + expect(result.answerOf(department).value, _Department.billing); + expect(result.answerOf(urgency).levelProbabilities, hasLength(2)); + expect(result.answerOf(refund).noul, result.nouls['refund']!.noul); + expect( + () => result.answerOf(NoulKey.of('refund', 'Refund requested?')), + throwsA( + isA().having( + (error) => error.message, + 'message', + startsWith('Result question "refund" is not this key\'s question'), + ), + ), + ); + }); + + test('fetches configPath in the page', () async { + const config = '{"max_len": 48, "head_max_len": 24}'; + final url = URL.createObjectURL( + Blob([config.toJS].toJS, BlobPropertyBag(type: 'text/plain')), + ); + addTearDown(() => URL.revokeObjectURL(url)); + await loadModel(); + + final decisions = await DecisionEngine.load( + engine, + headPath: 'model.safetensors', + configPath: url, + ); + + expect(bridges.single.loadedConfigs, [config]); + expect(decisions.info.maxTokens, 48); + expect(decisions.info.headMaxTokens, 24); + }); + + test('reports bridge assets without the decision API', () async { + withDecisionApi = false; + await loadModel(); + + final capabilities = await DecisionEngine.capabilitiesFor(engine); + + expect(capabilities.isSupported, isFalse); + expect( + capabilities.unsupportedReason, + 'Web decision models need llama-web-bridge assets v0.1.47+ with the ' + 'decision API (apiVersion 1); the loaded bridge does not expose it.', + ); + await expectLater( + DecisionEngine.load(engine, headPath: 'laya-head.safetensors'), + throwsA(isA()), + ); + }); + + test('reports decision API version skew', () async { + await loadModel(); + bridges.single.capabilitiesApiVersion = 2; + + await expectLater( + DecisionEngine.load(engine, headPath: 'laya-head.safetensors'), + throwsA( + isA().having( + (error) => error.message, + 'message', + contains('decision API version 2'), + ), + ), + ); + expect( + bridges.single.calls.where((call) => call.startsWith('load ')), + isEmpty, + ); + }); + + test( + 'a cancelled capability probe fails load with LlamaStateException', + () async { + await loadModel(); + bridges.single.capabilitiesError = + 'Decision capability probe was cancelled.'; + + final capabilities = await DecisionEngine.capabilitiesFor(engine); + + expect(capabilities.isSupported, isFalse); + expect(capabilities.unsupportedReason, contains('was cancelled')); + await expectLater( + DecisionEngine.load(engine, headPath: 'laya-head.safetensors'), + throwsA( + isA().having( + (error) => error.message, + 'message', + 'Decision capability probe was cancelled.', + ), + ), + ); + }, + ); + + test( + 'a head fails with LlamaStateException after the model unloads', + () async { + await loadModel(); + final decisions = await DecisionEngine.load( + engine, + headPath: 'laya-head.safetensors', + ); + + await engine.unloadModel(); + + await expectLater( + decisions.systemOne(state: 'Billed twice.', questions: questions), + throwsA(isA()), + ); + await decisions.dispose(); + final fake = bridges.single; + expect(fake.disposeCalls, 1); + expect(fake.calls.where((call) => call.startsWith('free')), isEmpty); + expect(fake.calls.where((call) => call.startsWith('run')), isEmpty); + }, + ); +} + +enum _Department { billing, technical, other } diff --git a/test/support/fake_webgpu_decision_bridge.dart b/test/support/fake_webgpu_decision_bridge.dart new file mode 100644 index 000000000..88ef38bd2 --- /dev/null +++ b/test/support/fake_webgpu_decision_bridge.dart @@ -0,0 +1,236 @@ +import 'dart:js_interop'; +import 'dart:js_interop_unsafe'; +import 'dart:typed_data'; + +import 'package:llamadart/src/backends/webgpu/interop.dart'; + +@JS('Promise.reject') +external JSPromise _rejectPromise(JSAny? reason); + +/// A promise rejected with a JS error object carrying [message]. +JSPromise rejectWithMessage(String message) => + _rejectPromise(JSObject()..setProperty('message'.toJS, message.toJS)); + +/// One `runDecision` sequence as the fake bridge received it. +typedef FakeDecisionSequence = ({ + List tokens, + List markers, + int questionType, + bool typedArrays, +}); + +/// A fake llama-web-bridge instance with the decision API. +/// +/// Follows the bridge's documented behavior: handles are never reused, +/// `freeDecisionHead` ignores unknown handles, and `dispose` or a model load +/// frees every head. Handles start at 7, so tests can tell them from backend +/// handles. While an `*Error` field is set, matching calls reject with it. +class FakeDecisionBridge { + /// Creates the fake. Without [withDecisionApi] it models bridge assets that + /// predate the decision API; [withModelApi] adds the model-load, tokenizer + /// and lifecycle methods `WebGpuLlamaBackend` needs. + FakeDecisionBridge({bool withDecisionApi = true, bool withModelApi = false}) { + if (withDecisionApi) _installDecisionApi(); + if (withModelApi) _installModelApi(); + } + + /// The JS object handed to the backend. + final JSObject object = JSObject(); + + /// [object] as the interop type. + LlamaWebGpuBridge get bridge => object as LlamaWebGpuBridge; + + /// `apiVersion` reported by `getDecisionCapabilities`. + int capabilitiesApiVersion = 1; + + /// `supported` reported by `getDecisionCapabilities`. + bool supported = true; + + /// `reason` reported by `getDecisionCapabilities`, when set. + String? reason; + + /// Replaces the whole `getDecisionCapabilities` result when set. + JSAny? capabilitiesResult; + + /// Fields that replace or, when null, remove head info fields. + Map headInfoOverrides = {}; + + /// Replaces the `runDecision` result when set. + JSAny? Function()? runResult; + + /// Rejection messages for matching calls. + String? capabilitiesError, loadError, runError, freeError; + + /// Every decision call, in order. + final List calls = []; + + /// `configJson` of each `loadDecisionHead` call, null when absent. + final List loadedConfigs = []; + + /// Sequences of the last `runDecision` call. + List lastSequences = const []; + + /// Bridge handles of loaded, unfreed heads. + final Set liveHandles = {}; + + /// Number of `dispose` calls. + int disposeCalls = 0; + + int _nextHandle = 7; + + void _installDecisionApi() { + object.setProperty( + 'getDecisionCapabilities'.toJS, + (() { + calls.add('capabilities'); + final error = capabilitiesError; + if (error != null) return rejectWithMessage(error); + final result = + capabilitiesResult ?? + (JSObject() + ..setProperty('apiVersion'.toJS, capabilitiesApiVersion.toJS) + ..setProperty('supported'.toJS, supported.toJS) + ..setProperty('reason'.toJS, reason?.toJS)); + return Future.value(result).toJS; + }).toJS, + ); + object.setProperty( + 'loadDecisionHead'.toJS, + ((JSAny? source, JSObject? options) { + final url = (source as JSString).toDart; + final config = options?.getProperty('configJson'.toJS); + final configJson = config != null && config.isA() + ? (config as JSString).toDart + : null; + calls.add('load $url'); + loadedConfigs.add(configJson); + final error = loadError; + if (error != null) return rejectWithMessage(error); + final handle = _nextHandle++; + liveHandles.add(handle); + final info = { + 'apiVersion': 1, + 'handle': handle, + 'hiddenSize': 4, + 'clsToken': 1, + 'sepToken': 2, + 'maskToken': 3, + 'maskText': '[MASK]', + 'configJson': configJson ?? '{"max_len": 32, "head_max_len": 16}', + 'deviceName': 'WebGPU', + ...headInfoOverrides, + }; + final result = JSObject(); + for (final MapEntry(:key, :value) in info.entries) { + if (value != null) result.setProperty(key.toJS, value.jsify()); + } + return Future.value(result).toJS; + }).toJS, + ); + object.setProperty( + 'runDecision'.toJS, + ((JSNumber rawHandle, JSArray sequences) { + final handle = rawHandle.toDartInt; + lastSequences = [ + for (final sequence in sequences.toDart) _sequenceOf(sequence), + ]; + calls.add('run $handle ${lastSequences.length}'); + final error = runError; + if (error != null) return rejectWithMessage(error); + if (!liveHandles.contains(handle)) { + return rejectWithMessage( + 'Decision head $handle is not loaded; it was freed, its model ' + 'was unloaded, or the bridge runtime restarted. Load the decision ' + 'head again.', + ); + } + final custom = runResult; + if (custom != null) return Future.value(custom()).toJS; + final outputs = [ + for (final sequence in lastSequences) + JSObject() + ..setProperty( + 'logits'.toJS, + Float32List.fromList([ + for (var i = sequence.markers.length; i > 0; i--) + i.toDouble(), + ]).toJS, + ) + ..setProperty( + 'actLogits'.toJS, + Float32List.fromList([1.5, -0.5]).toJS, + ), + ]; + return Future.value(outputs.toJS).toJS; + }).toJS, + ); + object.setProperty( + 'freeDecisionHead'.toJS, + ((JSNumber rawHandle) { + final handle = rawHandle.toDartInt; + calls.add('free $handle'); + final error = freeError; + if (error != null) return rejectWithMessage(error); + liveHandles.remove(handle); + return Future.value().toJS; + }).toJS, + ); + } + + void _installModelApi() { + object.setProperty( + 'loadModelFromUrl'.toJS, + ((String url, JSObject? options) { + calls.add('loadModel $url'); + liveHandles.clear(); + return Future.value().toJS; + }).toJS, + ); + object.setProperty( + 'tokenize'.toJS, + ((String text, bool? addSpecial) { + final ids = Uint32List.fromList([ + for (final unit in text.codeUnits.take(3)) 10 + unit % 90, + ]); + return Future.value(ids.toJS).toJS; + }).toJS, + ); + object.setProperty( + 'dispose'.toJS, + (() { + disposeCalls++; + liveHandles.clear(); + return Future.value().toJS; + }).toJS, + ); + object.setProperty('cancel'.toJS, (() {}).toJS); + object.setProperty('setLogLevel'.toJS, ((JSAny? level) {}).toJS); + object.setProperty('getBackendName'.toJS, (() => 'WebGPU (Fake)').toJS); + object.setProperty('getContextSize'.toJS, (() => 512).toJS); + object.setProperty('isGpuActive'.toJS, (() => true).toJS); + object.setProperty( + 'getModelMetadata'.toJS, + (() => JSObject() + ..setProperty('general.architecture'.toJS, 'modern-bert'.toJS)) + .toJS, + ); + } + + static FakeDecisionSequence _sequenceOf(JSObject sequence) { + final tokens = sequence.getProperty('tokens'.toJS); + final markers = sequence.getProperty('markers'.toJS); + final typed = + tokens != null && + markers != null && + tokens.isA() && + markers.isA(); + return ( + tokens: typed ? (tokens as JSInt32Array).toDart.toList() : const [], + markers: typed ? (markers as JSInt32Array).toDart.toList() : const [], + questionType: sequence + .getProperty('questionType'.toJS) + .toDartInt, + typedArrays: typed, + ); + } +} diff --git a/test/unit/backends/llama_cpp/llama_cpp_service_test.dart b/test/unit/backends/llama_cpp/llama_cpp_service_test.dart index c17afbe17..95a1c3553 100644 --- a/test/unit/backends/llama_cpp/llama_cpp_service_test.dart +++ b/test/unit/backends/llama_cpp/llama_cpp_service_test.dart @@ -2000,6 +2000,26 @@ void main() { rejects('marker -1'), ); }); + + test('rejects more markers than tokens, as the Web bridge does', () { + validate([ + sequence(tokens: const [1, 2], markers: const [0, 1]), + ]); + expect( + () => validate([ + sequence(), + sequence(tokens: const [1, 4, 2], markers: const [0, 1, 2, 1]), + ]), + throwsA( + isA().having( + (error) => error.message, + 'message', + 'Decision sequence 1 has 4 markers for its 3 tokens; a ' + 'sequence holds at most one marker per token.', + ), + ), + ); + }); }); group('resolveDecisionHeadConfigText', () { diff --git a/test/unit/backends/web/web_backend_test.dart b/test/unit/backends/web/web_backend_test.dart index d7e7b115d..ecd2cb16c 100644 --- a/test/unit/backends/web/web_backend_test.dart +++ b/test/unit/backends/web/web_backend_test.dart @@ -5,6 +5,7 @@ import 'dart:typed_data'; import 'package:llamadart/src/backends/backend.dart'; import 'package:llamadart/src/backends/web/web_backend.dart'; +import 'package:llamadart/src/core/decision/decision_question.dart'; import 'package:llamadart/src/core/engine/chat_completion_request_planner.dart'; import 'package:llamadart/src/core/engine/engine.dart'; import 'package:llamadart/src/core/exceptions.dart'; @@ -29,6 +30,7 @@ void main() { expect(backend, isA()); expect(backend, isA()); expect(backend, isA()); + expect(backend, isA()); expect((backend as WebAutoBackend).supportsStatePersistence, isFalse); expect(backend.supportsEmbeddings, isFalse); }); @@ -179,6 +181,82 @@ void main() { ); }); + test('WebAutoBackend forwards decision calls to its delegate', () async { + final delegate = _DecisionBackend(); + final backend = WebAutoBackend(webBackend: delegate); + final sequence = BackendDecisionSequence( + tokens: Int32List.fromList([1, 3, 2]), + markers: Int32List.fromList([1]), + questionType: DecisionQuestionType.noul, + ); + + final capabilities = await backend.decisionCapabilities(1); + final head = await backend.decisionHeadLoad( + 1, + 'laya-head.safetensors', + configPath: 'rl_agent_config.json', + ); + final outputs = await backend.decisionRun(head.handle, [sequence]); + await backend.decisionHeadFree(head.handle); + + expect(capabilities.isSupported, isTrue); + expect(outputs.single.logits, [0.5]); + expect(delegate.calls, [ + 'capabilities 1', + 'load 1 laya-head.safetensors rl_agent_config.json', + 'run 9 1', + 'free 9', + ]); + }); + + test('WebAutoBackend reports decision models unsupported on LiteRT-LM ' + 'Web', () async { + const reason = + 'The active Web runtime does not run decision models. Load a ' + 'ModernBERT encoder GGUF, which uses the llama.cpp WebGPU bridge.'; + final liteRtLm = _RecordingBackend('litert'); + final backend = WebAutoBackend( + webGpuFactory: _DecisionBackend.new, + liteRtLmFactory: () => liteRtLm, + ); + await backend.modelLoadFromUrl( + 'https://example.com/gemma-4-E2B-it-web.litertlm', + const ModelParams(), + ); + + final capabilities = await backend.decisionCapabilities(1); + + expect(capabilities.isSupported, isFalse); + expect(capabilities.unsupportedReason, reason); + expect( + () => backend.decisionHeadLoad(1, 'laya-head.safetensors'), + throwsA( + isA().having( + (error) => error.message, + 'message', + reason, + ), + ), + ); + expect( + () => backend.decisionRun(1, const []), + throwsA(isA()), + ); + await backend.decisionHeadFree(1); + }); + + test('WebAutoBackend rejects decision calls before a model load', () async { + final backend = WebAutoBackend(webGpuFactory: _DecisionBackend.new); + + expect(() => backend.decisionCapabilities(1), throwsStateError); + expect( + () => backend.decisionHeadLoad(1, 'laya-head.safetensors'), + throwsStateError, + ); + expect(() => backend.decisionRun(1, const []), throwsStateError); + await backend.decisionHeadFree(1); + }); + test('WebAutoBackend routes .litertlm URLs to LiteRT-LM delegate', () async { final webGpu = _RecordingBackend('webgpu'); final liteRtLm = _RecordingBackend('litert'); @@ -386,3 +464,53 @@ class _TextToSpeechBackend extends _NoStateBackend cancelCalls += 1; } } + +class _DecisionBackend extends _NoStateBackend implements BackendDecision { + final List calls = []; + + @override + Future decisionCapabilities( + int modelHandle, + ) async { + calls.add('capabilities $modelHandle'); + return const BackendDecisionCapabilities(isSupported: true); + } + + @override + Future decisionHeadLoad( + int modelHandle, + String headPath, { + String? configPath, + }) async { + calls.add('load $modelHandle $headPath $configPath'); + return const BackendDecisionHeadInfo( + handle: 9, + hiddenSize: 4, + clsToken: 1, + sepToken: 2, + maskToken: 3, + maskText: '[MASK]', + configJson: '{}', + deviceName: 'WebGPU', + ); + } + + @override + Future> decisionRun( + int headHandle, + List sequences, + ) async { + calls.add('run $headHandle ${sequences.length}'); + return [ + BackendDecisionOutput( + logits: Float32List.fromList([0.5]), + actLogits: Float32List.fromList([1, 0]), + ), + ]; + } + + @override + Future decisionHeadFree(int headHandle) async { + calls.add('free $headHandle'); + } +} diff --git a/test/unit/backends/webgpu/webgpu_backend_test.dart b/test/unit/backends/webgpu/webgpu_backend_test.dart index d860fec7a..87e327fdd 100644 --- a/test/unit/backends/webgpu/webgpu_backend_test.dart +++ b/test/unit/backends/webgpu/webgpu_backend_test.dart @@ -11,7 +11,9 @@ import 'package:llamadart/llamadart.dart'; import 'package:llamadart/src/backends/webgpu/interop.dart'; import 'package:llamadart/src/backends/webgpu/webgpu_backend.dart'; import 'package:test/test.dart'; -import 'package:web/web.dart' show Response, window; +import 'package:web/web.dart' show Response, document, window; + +import '../../../support/fake_webgpu_decision_bridge.dart'; @JS('Promise.reject') external JSPromise _rejectPromise(JSAny? reason); @@ -3615,4 +3617,176 @@ void main() { }, ); }); + + group('WebGpuLlamaBackend decision heads', () { + late List bridges; + late bool withDecisionApi; + late WebGpuLlamaBackend backend; + + setUp(() { + bridges = []; + withDecisionApi = true; + backend = WebGpuLlamaBackend( + bridgeFactory: ([config]) { + final fake = FakeDecisionBridge( + withDecisionApi: withDecisionApi, + withModelApi: true, + ); + bridges.add(fake); + return fake.bridge; + }, + ); + }); + + tearDown(() => backend.dispose()); + + Future loadModel() => backend.modelLoadFromUrl( + 'laya-Q8_0.gguf', + const ModelParams(contextSize: 512), + ); + + final sequence = BackendDecisionSequence( + tokens: Int32List.fromList([1, 3, 20, 2]), + markers: Int32List.fromList([1]), + questionType: DecisionQuestionType.choice, + ); + + test('reports no model before a bridge is active', () async { + expect(backend, isA()); + final capabilities = await backend.decisionCapabilities(1); + + expect(capabilities.isSupported, isFalse); + expect( + capabilities.unsupportedReason, + 'No model is loaded on the Web bridge. Load a ModernBERT encoder GGUF ' + 'first.', + ); + await expectLater( + backend.decisionHeadLoad(1, 'laya-head.safetensors'), + throwsA( + isA().having( + (error) => error.message, + 'message', + 'No model is loaded on the Web bridge. Load the decision encoder ' + 'before its head.', + ), + ), + ); + await expectLater( + backend.decisionRun(1, [sequence]), + throwsA(isA()), + ); + await backend.decisionHeadFree(1); + expect(bridges, isEmpty); + }); + + test('reports bridge assets without the decision API', () async { + withDecisionApi = false; + await loadModel(); + + final capabilities = await backend.decisionCapabilities(1); + + expect(capabilities.isSupported, isFalse); + expect( + capabilities.unsupportedReason, + contains( + 'llama-web-bridge assets v0.1.47+ with the decision API ' + '(apiVersion 1)', + ), + ); + await expectLater( + backend.decisionHeadLoad(1, 'laya-head.safetensors'), + throwsA(isA()), + ); + }); + + test('loads, runs and frees heads on the active bridge', () async { + await loadModel(); + final fake = bridges.single; + + final capabilities = await backend.decisionCapabilities(1); + final head = await backend.decisionHeadLoad(1, 'laya-head.safetensors'); + final outputs = await backend.decisionRun(head.handle, [sequence]); + await backend.decisionHeadFree(head.handle); + + expect(capabilities.isSupported, isTrue); + expect(head.handle, 1); + expect(outputs.single.logits, [1.0]); + expect(fake.calls, [ + 'loadModel laya-Q8_0.gguf', + 'capabilities', + 'capabilities', + 'load ${Uri.parse(document.baseURI).resolve('laya-head.safetensors')}', + 'run 7 1', + 'free 7', + ]); + expect(fake.liveHandles, isEmpty); + }); + + test('frees heads with the model and never reuses handles', () async { + await loadModel(); + final first = await backend.decisionHeadLoad(1, 'laya-head.safetensors'); + + await backend.modelFree(1); + await expectLater( + backend.decisionRun(first.handle, [sequence]), + throwsA(isA()), + ); + await backend.decisionHeadFree(first.handle); + + await loadModel(); + await expectLater( + backend.decisionRun(first.handle, [sequence]), + throwsA(isA()), + ); + final second = await backend.decisionHeadLoad(1, 'laya-head.safetensors'); + + expect(bridges, hasLength(2)); + expect(bridges.first.disposeCalls, 1); + expect( + bridges.first.calls.where((call) => call.startsWith('free')), + isEmpty, + ); + expect(second.handle, 2); + expect( + bridges.last.calls.where((call) => call.startsWith('run')), + isEmpty, + ); + }); + + test('forgets heads when a model reloads on the same bridge', () async { + await loadModel(); + final head = await backend.decisionHeadLoad(1, 'laya-head.safetensors'); + + await loadModel(); + + expect(bridges, hasLength(1)); + await expectLater( + backend.decisionRun(head.handle, [sequence]), + throwsA(isA()), + ); + expect( + bridges.single.calls.where((call) => call.startsWith('run')), + isEmpty, + ); + }); + + test('forgets heads on dispose', () async { + await loadModel(); + final head = await backend.decisionHeadLoad(1, 'laya-head.safetensors'); + + await backend.dispose(); + + await expectLater( + backend.decisionRun(head.handle, [sequence]), + throwsA(isA()), + ); + await backend.decisionHeadFree(head.handle); + expect(bridges.single.disposeCalls, 1); + expect( + bridges.single.calls.where((call) => call.startsWith('free')), + isEmpty, + ); + }); + }); } diff --git a/test/unit/backends/webgpu/webgpu_decision_test.dart b/test/unit/backends/webgpu/webgpu_decision_test.dart new file mode 100644 index 000000000..a5fa95823 --- /dev/null +++ b/test/unit/backends/webgpu/webgpu_decision_test.dart @@ -0,0 +1,735 @@ +@TestOn('browser') +library; + +import 'dart:js_interop'; +import 'dart:js_interop_unsafe'; +import 'dart:typed_data'; + +import 'package:llamadart/src/backends/backend.dart'; +import 'package:llamadart/src/backends/webgpu/webgpu_decision.dart'; +import 'package:llamadart/src/core/decision/decision_question.dart'; +import 'package:llamadart/src/core/exceptions.dart'; +import 'package:test/test.dart'; +import 'package:web/web.dart' + show Blob, BlobPropertyBag, HTMLBaseElement, URL, document; + +import '../../../support/fake_webgpu_decision_bridge.dart'; + +void main() { + late FakeDecisionBridge fake; + late WebGpuDecisionHeads heads; + + setUp(() { + fake = FakeDecisionBridge(); + heads = WebGpuDecisionHeads(); + }); + + BackendDecisionSequence sequence( + List tokens, + List markers, [ + DecisionQuestionType questionType = DecisionQuestionType.choice, + ]) => BackendDecisionSequence( + tokens: Int32List.fromList(tokens), + markers: Int32List.fromList(markers), + questionType: questionType, + ); + + Matcher throwsTyped(Object? message) => + throwsA(isA().having((error) => error.message, 'message', message)); + + String blobUrl(String text) => URL.createObjectURL( + Blob([text.toJS].toJS, BlobPropertyBag(type: 'application/json')), + ); + + String pageUrl(String path) => + Uri.parse(document.baseURI).resolve(path).toString(); + + void useBaseHref(String href) { + final base = document.createElement('base') as HTMLBaseElement..href = href; + document.head!.append(base); + addTearDown(() => base.remove()); + } + + group('capabilities', () { + test('reports supported for decision API version 1', () async { + final capabilities = await heads.capabilities(fake.bridge); + + expect(capabilities.isSupported, isTrue); + expect(capabilities.unsupportedReason, isNull); + expect(fake.calls, ['capabilities']); + }); + + test('names the required assets for bridges without the API', () async { + final old = FakeDecisionBridge(withDecisionApi: false); + final partial = FakeDecisionBridge(); + partial.object.delete('freeDecisionHead'.toJS); + + for (final bridge in [old, partial]) { + final capabilities = await heads.capabilities(bridge.bridge); + + expect(capabilities.isSupported, isFalse); + expect( + capabilities.unsupportedReason, + 'Web decision models need llama-web-bridge assets v0.1.47+ with ' + 'the decision API (apiVersion 1); the loaded bridge does not expose ' + 'it.', + ); + } + expect(partial.calls, isEmpty); + }); + + test('reports another decision API version as unsupported', () async { + fake.capabilitiesApiVersion = 2; + final skewed = await heads.capabilities(fake.bridge); + fake.capabilitiesResult = JSObject() + ..setProperty('supported'.toJS, true.toJS); + final unversioned = await heads.capabilities(fake.bridge); + + expect(skewed.isSupported, isFalse); + expect( + skewed.unsupportedReason, + 'The Web bridge implements decision API version 2; llamadart needs ' + 'llama-web-bridge assets v0.1.47+ with the decision API (apiVersion 1).', + ); + expect(unversioned.isSupported, isFalse); + expect(unversioned.unsupportedReason, contains('version unknown')); + }); + + test('passes the bridge reason through', () async { + fake + ..supported = false + ..reason = 'Decision heads need a ModernBERT encoder GGUF.'; + final withReason = await heads.capabilities(fake.bridge); + fake.reason = ''; + final withoutReason = await heads.capabilities(fake.bridge); + + expect(withReason.isSupported, isFalse); + expect( + withReason.unsupportedReason, + 'Decision heads need a ModernBERT encoder GGUF.', + ); + expect( + withoutReason.unsupportedReason, + 'The loaded Web model does not support decision heads.', + ); + }); + + test('throws LlamaStateException for state rejections', () async { + for (final message in [ + 'Bridge has been disposed.', + 'Decision capability probe was cancelled.', + 'No model loaded. Call loadModelFromUrl first.', + ]) { + fake.capabilitiesError = message; + + await expectLater( + heads.capabilities(fake.bridge), + throwsTyped(message), + reason: message, + ); + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsTyped(message), + reason: message, + ); + } + expect(fake.calls.where((call) => call.startsWith('load')), isEmpty); + }); + + test('reports invalid responses and failed probes', () async { + fake.capabilitiesResult = 'yes'.toJS; + final invalid = await heads.capabilities(fake.bridge); + fake + ..capabilitiesResult = null + ..capabilitiesError = 'WebGPU core is not initialized'; + final failed = await heads.capabilities(fake.bridge); + + expect(invalid.isSupported, isFalse); + expect( + invalid.unsupportedReason, + 'The Web decision capability response is invalid.', + ); + expect(failed.isSupported, isFalse); + expect( + failed.unsupportedReason, + 'The Web decision capability probe failed: WebGPU core is not ' + 'initialized', + ); + }); + }); + + group('load', () { + test('returns the head under a backend handle', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + + expect(head.handle, 1); + expect(head.hiddenSize, 4); + expect(head.clsToken, 1); + expect(head.sepToken, 2); + expect(head.maskToken, 3); + expect(head.maskText, '[MASK]'); + expect(head.configJson, '{"max_len": 32, "head_max_len": 16}'); + expect(head.deviceName, 'WebGPU'); + expect(fake.calls, [ + 'capabilities', + 'load ${pageUrl('laya-head.safetensors')}', + ]); + expect(fake.loadedConfigs, [null]); + expect(fake.liveHandles, {7}); + }); + + test('fetches configPath and passes its text to the bridge', () async { + const config = '{"max_len": 64, "head_max_len": 24}'; + final url = blobUrl(config); + addTearDown(() => URL.revokeObjectURL(url)); + + final head = await heads.load( + fake.bridge, + 'model.safetensors', + configUrl: url, + ); + + expect(fake.loadedConfigs, [config]); + expect(head.configJson, config); + }); + + test('rejects unsupported models before loading', () async { + fake + ..supported = false + ..reason = 'The loaded model reports architecture "llama".'; + + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsTyped( + 'The loaded model reports architecture "llama".', + ), + ); + await expectLater( + heads.load( + FakeDecisionBridge(withDecisionApi: false).bridge, + 'laya-head.safetensors', + ), + throwsTyped( + contains('decision API (apiVersion 1)'), + ), + ); + expect(fake.calls, ['capabilities']); + }); + + test('fails with LlamaModelException for unreadable configs', () async { + final revoked = blobUrl('{}'); + URL.revokeObjectURL(revoked); + + await expectLater( + heads.load( + fake.bridge, + 'laya-head.safetensors', + configUrl: 'missing_decision_config.json?token=secret#frag', + ), + throwsA( + isA() + .having( + (error) => error.message, + 'message', + 'Cannot read the decision head config at ' + '${pageUrl('missing_decision_config.json')}.', + ) + .having( + (error) => '${error.details}', + 'details', + startsWith('HTTP 404'), + ), + ), + ); + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors', configUrl: revoked), + throwsA( + isA().having( + (error) => error.message, + 'message', + startsWith('Cannot read the decision head config at blob:'), + ), + ), + ); + expect(fake.calls.where((call) => call.startsWith('load')), isEmpty); + }); + + test('fails with LlamaModelException when a config body fails', () async { + final fetch = globalContext.getProperty('fetch'.toJS); + addTearDown(() => globalContext.setProperty('fetch'.toJS, fetch)); + globalContext.setProperty( + 'fetch'.toJS, + ((JSAny? url) => Future.value( + JSObject() + ..setProperty('ok'.toJS, true.toJS) + ..setProperty('status'.toJS, 200.toJS) + ..setProperty( + 'text'.toJS, + (() => rejectWithMessage('network error')).toJS, + ), + ).toJS).toJS, + ); + + await expectLater( + heads.load( + fake.bridge, + 'laya-head.safetensors', + configUrl: 'https://example.com/rl_agent_config.json', + ), + throwsA( + isA() + .having( + (error) => error.message, + 'message', + 'Cannot read the decision head config at ' + 'https://example.com/rl_agent_config.json.', + ) + .having((error) => error.details, 'details', 'network error'), + ), + ); + expect(fake.calls.where((call) => call.startsWith('load')), isEmpty); + }); + + test('resolves head and config URLs against the document base', () async { + useBaseHref(pageUrl('decision-base/')); + + await heads.load(fake.bridge, 'models/laya-head.safetensors'); + await expectLater( + heads.load( + fake.bridge, + 'models/model.safetensors', + configUrl: 'models/rl_agent_config.json', + ), + throwsTyped( + 'Cannot read the decision head config at ' + '${pageUrl('models/rl_agent_config.json')}.', + ), + ); + final blob = blobUrl('{}'); + addTearDown(() => URL.revokeObjectURL(blob)); + await heads.load(fake.bridge, blob); + + expect(document.baseURI, endsWith('/decision-base/')); + expect(fake.calls.where((call) => call.startsWith('load')), [ + 'load ${pageUrl('models/laya-head.safetensors')}', + 'load $blob', + ]); + }); + + test('keeps credentials and queries out of load errors', () async { + const secretConfig = + 'https://alice:s3cret@example.com/rl_agent_config.json?sig=xyz#frag'; + const secretHead = + 'https://alice:s3cret@example.com/laya-head.safetensors?sig=xyz'; + Matcher redacted(String message) => throwsA( + isA() + .having((error) => error.message, 'message', message) + .having( + (error) => '$error', + 'toString', + allOf( + isNot(contains('s3cret')), + isNot(contains('alice')), + isNot(contains('hunter2')), + isNot(contains('bob')), + isNot(contains('sig=')), + isNot(contains('token=')), + isNot(contains('frag')), + ), + ), + ); + + await expectLater( + heads.load( + fake.bridge, + 'laya-head.safetensors', + configUrl: secretConfig, + ), + redacted( + 'Cannot read the decision head config at ' + 'https://example.com/rl_agent_config.json.', + ), + ); + fake.loadError = + "Failed to execute 'fetch' on 'Window': Request cannot be " + 'constructed from a URL that includes credentials: $secretHead'; + await expectLater( + heads.load(fake.bridge, secretHead), + redacted( + "Failed to execute 'fetch' on 'Window': Request cannot be " + 'constructed from a URL that includes credentials: ' + 'https://example.com/laya-head.safetensors', + ), + ); + fake.loadError = + 'Failed to fetch decision head from ' + 'https://bob:hunter2@cdn.example.com/head?token=abc (timeout)'; + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + redacted( + 'Failed to fetch decision head from https://cdn.example.com/head ' + '(timeout)', + ), + ); + fake.loadError = + "Failed to execute 'fetch' on 'Window': Failed to parse URL from " + 'https://bob:hunter2@[cdn/head?token=abc'; + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + redacted( + "Failed to execute 'fetch' on 'Window': Failed to parse URL from " + 'https://[cdn/head', + ), + ); + }); + + test('names configPath in bridge config errors', () async { + final config = blobUrl('{"max_len": "long"}'); + addTearDown(() => URL.revokeObjectURL(config)); + fake.loadError = + 'Failed to load decision head: The decision head at ' + '"model.safetensors" has no "laya.config" metadata. Pass configJson ' + "with the head's rl_agent_config.json."; + + await expectLater( + heads.load(fake.bridge, 'model.safetensors'), + throwsTyped( + 'The decision head at "model.safetensors" has no "laya.config" ' + "metadata. Pass configPath with the head's rl_agent_config.json.", + ), + ); + fake.loadError = + 'Failed to load decision head: The decision head config in ' + 'configJson is invalid: Decision head "max_len" must be a positive ' + 'integer, got "long".'; + await expectLater( + heads.load(fake.bridge, 'model.safetensors', configUrl: config), + throwsTyped( + 'The decision head config in $config is invalid: Decision head ' + '"max_len" must be a positive integer, got "long".', + ), + ); + }); + + test('maps bridge load failures to typed exceptions', () async { + final cases = <(String, Matcher)>[ + ( + 'Failed to load decision head: The decision head at ' + '"laya-head.safetensors" is 768 wide but the loaded encoder has ' + 'hidden size 1024. Use the head trained for this encoder.', + throwsA( + isA() + .having( + (error) => error.message, + 'message', + startsWith('The decision head at "laya-head.safetensors"'), + ) + .having( + (error) => error.details, + 'details', + 'https://example.com/laya-head.safetensors', + ), + ), + ), + ( + 'Failed to fetch decision head: 404 Not Found', + throwsTyped( + 'Failed to fetch decision head: 404 Not Found', + ), + ), + ( + 'Failed to load decision head: Failed to create the decision ' + 'encoder context of 512 tokens.', + throwsTyped( + 'Failed to create the decision encoder context of 512 tokens.', + ), + ), + ( + 'No model loaded. Call loadModelFromUrl first.', + throwsTyped( + 'No model loaded. Call loadModelFromUrl first.', + ), + ), + ( + 'Bridge has been disposed.', + throwsTyped('Bridge has been disposed.'), + ), + ( + 'Decision head load was cancelled.', + throwsTyped('Decision head load was cancelled.'), + ), + ]; + for (final (message, matcher) in cases) { + fake.loadError = message; + await expectLater( + heads.load( + fake.bridge, + 'https://user:pass@example.com/laya-head.safetensors?sig=abc', + ), + matcher, + reason: message, + ); + } + }); + + test('frees a head that reports another API version', () async { + fake.headInfoOverrides = {'apiVersion': 2}; + + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsTyped( + 'The Web bridge implements decision API version 2; llamadart needs ' + 'llama-web-bridge assets v0.1.47+ with the decision API (apiVersion 1).', + ), + ); + expect(fake.calls.last, 'free 7'); + expect(fake.liveHandles, isEmpty); + }); + + test('frees a head with a malformed description', () async { + for (final field in [ + 'maskText', + 'configJson', + 'deviceName', + 'sepToken', + ]) { + fake.headInfoOverrides = {field: null}; + + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsTyped( + 'The Web decision runtime returned a malformed head description.', + ), + reason: field, + ); + } + fake.headInfoOverrides = {'hiddenSize': 1.5}; + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsA(isA()), + ); + expect(fake.liveHandles, isEmpty); + + for (final handle in [0, -1]) { + fake.headInfoOverrides = {'handle': handle}; + await expectLater( + heads.load(fake.bridge, 'laya-head.safetensors'), + throwsA(isA()), + reason: '$handle', + ); + } + expect(fake.calls.where((call) => call.startsWith('free')), [ + for (final handle in [7, 8, 9, 10, 11]) 'free $handle', + ]); + }); + }); + + group('run', () { + test('sends typed sequences to the bridge handle', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + + final outputs = await heads.run(fake.bridge, head.handle, [ + sequence([1, 3, 20, 3, 21, 2], [1, 3], DecisionQuestionType.score), + sequence([1, 3, 2], [1], DecisionQuestionType.noul), + ]); + + expect(fake.calls.last, 'run 7 2'); + expect(fake.lastSequences.map((s) => s.typedArrays), [true, true]); + expect(fake.lastSequences.first.tokens, [1, 3, 20, 3, 21, 2]); + expect(fake.lastSequences.first.markers, [1, 3]); + expect(fake.lastSequences.map((s) => s.questionType), [1, 2]); + expect(outputs, hasLength(2)); + expect(outputs.first.logits, isA()); + expect(outputs.first.logits, [2.0, 1.0]); + expect(outputs.last.logits, [1.0]); + expect(outputs.first.actLogits, [1.5, -0.5]); + }); + + test('keeps heads scoped to the bridge that loaded them', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + final other = FakeDecisionBridge(); + + await expectLater( + heads.run(other.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsTyped( + 'Decision head 1 is not loaded on this Web runtime; it was freed, ' + 'its model was unloaded, or the bridge restarted. Load the decision ' + 'head again.', + ), + ); + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsA(isA()), + ); + await expectLater( + heads.run(null, 99, const []), + throwsA(isA()), + ); + expect(other.calls, isEmpty); + expect(fake.calls.where((call) => call.startsWith('run')), isEmpty); + }); + + test('forgets every head on clear', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + heads.clear(); + + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsA(isA()), + ); + await heads.free(fake.bridge, head.handle); + expect(fake.calls.where((call) => !call.startsWith('capab')), [ + 'load ${pageUrl('laya-head.safetensors')}', + ]); + }); + + test( + 'maps bridge validation failures to LlamaInferenceException', + () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + fake.runError = + 'Decision run failed: Decision sequence 0 has 4 markers for its 3 ' + 'tokens; a sequence holds at most one marker per token.'; + + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1, 3, 2], [0, 1, 2, 1]), + ]), + throwsTyped( + 'Decision sequence 0 has 4 markers for its 3 tokens; a sequence ' + 'holds at most one marker per token.', + ), + ); + fake.runError = null; + final outputs = await heads.run(fake.bridge, head.handle, [ + sequence([1, 3, 2], [1]), + ]); + expect(outputs, hasLength(1)); + }, + ); + + test('keeps a head when the bridge is busy', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + fake.runError = + 'Decision run failed: Decision heads cannot be loaded or run during ' + 'active generation or text-to-speech synthesis'; + + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsTyped( + 'Decision heads cannot be loaded or run during active generation or ' + 'text-to-speech synthesis', + ), + ); + fake.runError = null; + final outputs = await heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]); + expect(outputs, hasLength(1)); + }); + + test('drops a head the bridge lost', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + fake.runError = + 'Decision head 1 was lost when the bridge worker failed (boom). ' + 'Load the decision head again.'; + + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsTyped(contains('was lost')), + ); + fake.runError = null; + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsTyped(contains('on this Web runtime')), + ); + expect(fake.calls.where((call) => call.startsWith('run')), hasLength(1)); + }); + + test('rejects malformed outputs with LlamaDecisionException', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + final results = [ + () => JSObject(), + () => [null].toJS, + () => [ + JSObject() + ..setProperty('logits'.toJS, Float32List(1).toJS) + ..setProperty('actLogits'.toJS, [1.toJS].toJS), + ].toJS, + () => [ + JSObject()..setProperty('logits'.toJS, Float32List(1).toJS), + ].toJS, + () => [ + JSObject() + ..setProperty('logits'.toJS, [1.toJS].toJS) + ..setProperty('actLogits'.toJS, Float32List(2).toJS), + ].toJS, + ]; + for (final result in results) { + fake.runResult = result; + await expectLater( + heads.run(fake.bridge, head.handle, [ + sequence([1], [0]), + ]), + throwsTyped( + 'The Web decision runtime returned malformed outputs.', + ), + ); + } + }); + }); + + group('free', () { + test('frees once and never reuses handles', () async { + final first = await heads.load(fake.bridge, 'a.safetensors'); + await heads.free(fake.bridge, first.handle); + await heads.free(fake.bridge, first.handle); + await heads.free(fake.bridge, 42); + final second = await heads.load(fake.bridge, 'b.safetensors'); + + expect(second.handle, 2); + expect(fake.calls.where((call) => call.startsWith('free')), ['free 7']); + expect(fake.liveHandles, {8}); + }); + + test('ignores heads of another bridge and maps failures', () async { + final head = await heads.load(fake.bridge, 'laya-head.safetensors'); + final other = FakeDecisionBridge(); + final otherHead = await heads.load(other.bridge, 'laya-head.safetensors'); + expect(other.liveHandles, {7}); + + await heads.free(other.bridge, head.handle); + await heads.free(null, otherHead.handle); + + expect(other.calls.where((call) => call.startsWith('free')), isEmpty); + expect(other.liveHandles, {7}); + expect(fake.calls.where((call) => call.startsWith('free')), isEmpty); + expect(fake.liveHandles, {7}); + + for (final message in [ + 'Bridge has been disposed.', + 'Decision head handle must be a positive integer, got 0.', + ]) { + final loaded = await heads.load(fake.bridge, 'laya-head.safetensors'); + fake.freeError = message; + await expectLater( + heads.free(fake.bridge, loaded.handle), + throwsTyped(message), + reason: message, + ); + } + }); + }); +} diff --git a/test/unit/core/decision/decision_engine_web_test.dart b/test/unit/core/decision/decision_engine_web_test.dart index dc7ed0892..ec8b3b391 100644 --- a/test/unit/core/decision/decision_engine_web_test.dart +++ b/test/unit/core/decision/decision_engine_web_test.dart @@ -5,9 +5,9 @@ import 'package:llamadart/llamadart.dart'; import 'package:test/test.dart'; void main() { - const reason = 'The active backend does not expose decision models.'; + const reason = 'Load a model first.'; - test('the Web backend reports decision models as unsupported', () async { + test('the Web backend asks for a model before probing', () async { final engine = LlamaEngine(LlamaBackend()); addTearDown(engine.dispose); @@ -17,7 +17,7 @@ void main() { expect(capabilities.unsupportedReason, reason); }); - test('load throws LlamaUnsupportedException on Web', () async { + test('load without a model throws LlamaUnsupportedException', () async { final engine = LlamaEngine(LlamaBackend()); addTearDown(engine.dispose); diff --git a/test/unit/tooling/check_webgpu_bridge_tag_test.dart b/test/unit/tooling/check_webgpu_bridge_tag_test.dart index 751ac5568..15cd1e3fa 100644 --- a/test/unit/tooling/check_webgpu_bridge_tag_test.dart +++ b/test/unit/tooling/check_webgpu_bridge_tag_test.dart @@ -67,37 +67,37 @@ const String _approvedManifestJson = ''' { "artifacts": { "llama_webgpu_bridge.d.ts": { - "sha256": "be584430457c76cf991c39ebfd19f771424f86f6304899840bf9e50f1b32cafc", - "size_bytes": 6330 + "sha256": "d8fa58ab587bb79c44dff40714fa8fa130da8ead160c8aa33f9254ed75055837", + "size_bytes": 7810 }, "llama_webgpu_bridge.js": { - "sha256": "a704115fe87d3defff4a02c5a2b1d1bf0ab3bc1f00de3f40ed2c9b2d5983fd73", - "size_bytes": 210601 + "sha256": "eaa670759e559ca7ee7182fe8d5440872b6861aa6c3a5e8b7ec7f6e55c604254", + "size_bytes": 227520 }, "llama_webgpu_bridge_worker.js": { "sha256": "47bbfa0fe897e708455b9497e9aa41d2e94489c329300c33b12936fb3169deca", "size_bytes": 257 }, "llama_webgpu_core.js": { - "sha256": "67bade52aad19471ee96a7180db691e1f6b4e87d4254adaa3e5d9a31f6206ed1", - "size_bytes": 113847 + "sha256": "7e0617766b459bd1d1e398705716730e472bdfc0147f54212a5eb113b75b8a43", + "size_bytes": 114678 }, "llama_webgpu_core.wasm": { - "sha256": "f643e79520ac97bc150db6806735b9b73a98b07eb1b2fa4146bb773944315c71", - "size_bytes": 8918014 + "sha256": "4c2e81f5c44808367aa3f484776e7e4faa9f3841511d6a7ad2a39e6ec063d4a9", + "size_bytes": 9170142 }, "llama_webgpu_core_mem64.js": { - "sha256": "6575880ad6a631b6a8f9aaaebb9a5c3f1cc2910f59700ce529d3de6077276337", - "size_bytes": 130764 + "sha256": "3f9c7e8738588b043ba69688df836feefd1a95a211a281d9ae5ba6ac181f3bb2", + "size_bytes": 131595 }, "llama_webgpu_core_mem64.wasm": { - "sha256": "aaf399050af09af0c44b55ecf9d665e8f6e18df6f6deada3feab07677ddd7233", - "size_bytes": 9145549 + "sha256": "a6de570070c13c73d90666e903c4770f5eeb2d24be7fc630444de6d6609133a9", + "size_bytes": 9410357 } }, "assets_repository": "leehack/llama-web-bridge-assets", - "bridge_assets_tag": "v0.1.44", - "bridge_commit": "89178be67c3c84300bc1b129182bd5bc5a8e21fc", + "bridge_assets_tag": "v0.1.47", + "bridge_commit": "64ba8250871bf2472cc2064c6a00fff050783f02", "bridge_repository": "leehack/llama-web-bridge", "capabilities": { "memory64": true, @@ -128,43 +128,43 @@ const String _approvedManifestJson = ''' "emscripten_version": "6.0.8", "files": { "llama_webgpu_bridge.d.ts": { - "sha256": "be584430457c76cf991c39ebfd19f771424f86f6304899840bf9e50f1b32cafc", - "size_bytes": 6330 + "sha256": "d8fa58ab587bb79c44dff40714fa8fa130da8ead160c8aa33f9254ed75055837", + "size_bytes": 7810 }, "llama_webgpu_bridge.js": { - "sha256": "a704115fe87d3defff4a02c5a2b1d1bf0ab3bc1f00de3f40ed2c9b2d5983fd73", - "size_bytes": 210601 + "sha256": "eaa670759e559ca7ee7182fe8d5440872b6861aa6c3a5e8b7ec7f6e55c604254", + "size_bytes": 227520 }, "llama_webgpu_bridge_worker.js": { "sha256": "47bbfa0fe897e708455b9497e9aa41d2e94489c329300c33b12936fb3169deca", "size_bytes": 257 }, "llama_webgpu_core.js": { - "sha256": "67bade52aad19471ee96a7180db691e1f6b4e87d4254adaa3e5d9a31f6206ed1", - "size_bytes": 113847 + "sha256": "7e0617766b459bd1d1e398705716730e472bdfc0147f54212a5eb113b75b8a43", + "size_bytes": 114678 }, "llama_webgpu_core.wasm": { - "sha256": "f643e79520ac97bc150db6806735b9b73a98b07eb1b2fa4146bb773944315c71", - "size_bytes": 8918014 + "sha256": "4c2e81f5c44808367aa3f484776e7e4faa9f3841511d6a7ad2a39e6ec063d4a9", + "size_bytes": 9170142 }, "llama_webgpu_core_mem64.js": { - "sha256": "6575880ad6a631b6a8f9aaaebb9a5c3f1cc2910f59700ce529d3de6077276337", - "size_bytes": 130764 + "sha256": "3f9c7e8738588b043ba69688df836feefd1a95a211a281d9ae5ba6ac181f3bb2", + "size_bytes": 131595 }, "llama_webgpu_core_mem64.wasm": { - "sha256": "aaf399050af09af0c44b55ecf9d665e8f6e18df6f6deada3feab07677ddd7233", - "size_bytes": 9145549 + "sha256": "a6de570070c13c73d90666e903c4770f5eeb2d24be7fc630444de6d6609133a9", + "size_bytes": 9410357 } }, - "github_run_id": "35075283754", - "github_run_url": "https://github.com/leehack/llama-web-bridge/actions/runs/35075283754", + "github_run_id": "35904377401", + "github_run_url": "https://github.com/leehack/llama-web-bridge/actions/runs/35904377401", "llama_cpp_commit": "b29c606e28a01b1bc8c1351026a0fa6e616bf6c4", "llama_cpp_tag": "v0.4.1", "native_commit": "a4ee6b9fa71127d6cdf625e26d82ca5ab7b1d102", "native_manifest_sha256": "d8dc86fcb55e566ee04aa9ed235716bd0e7a48cdaecb17822c098b074ec33d3e", "native_release_tag": "v0.4.1", "native_repository": "leehack/llamadart-native", - "orchestrator_correlation_id": "auto-stable-v0.4.1-d8dc86fcb55e566e-build-89178be67c3c8430", + "orchestrator_correlation_id": "auto-stable-v0.4.1-d8dc86fcb55e566e-build-64ba8250871bf247", "qualification_gates": { "multimodal": "passed", "speech_to_text": "required-automated-qualification", @@ -173,9 +173,9 @@ const String _approvedManifestJson = ''' }, "release_channel": "stable", "release_rebuild": 0, - "release_tag": "v0.1.44", + "release_tag": "v0.1.47", "schema_version": 2, - "source_commit": "89178be67c3c84300bc1b129182bd5bc5a8e21fc", + "source_commit": "64ba8250871bf2472cc2064c6a00fff050783f02", "source_repository": "leehack/llama-web-bridge", "unproven_capabilities": { "hardware_gpu_acceleration": "unavailable-on-hosted-runners", @@ -196,7 +196,7 @@ Future> _verifyManifestJson( }) { final bytes = utf8.encode(manifestJson); return verifyManifest( - expectedTag: 'v0.1.44', + expectedTag: 'v0.1.47', expectedLlamaCppTag: bridgeLlamaCppTag, expectedLlamaCppCommit: bridgeLlamaCppCommit, expectedBridgeCommit: bridgeSourceCommit, @@ -255,7 +255,7 @@ void main() { 'website/docs/changelog/recent-releases.md', ]) { final current = _currentReleaseNotes(path); - expect(current, contains('`v0.1.44`'), reason: path); + expect(current, contains('`v0.1.47`'), reason: path); expect(current, contains(bridgeManifestSha256), reason: path); expect( current, @@ -842,8 +842,8 @@ void main() { }); test('immutable release identity stays pinned to the approved release', () { - expect(bridgeAssetsReleaseId, '389783936'); - expect(bridgeAssetsTagCommit, 'fdafd9f8cbdb9bf99c359536595eff9a23095379'); + expect(bridgeAssetsReleaseId, '394986324'); + expect(bridgeAssetsTagCommit, 'ee45e864641648a99411128f9bb82b7897fad221'); }); test('accepts equivalent CRLF documentation passages', () { diff --git a/tool/testing/check_webgpu_bridge_tag.dart b/tool/testing/check_webgpu_bridge_tag.dart index c06108ac7..547f74520 100644 --- a/tool/testing/check_webgpu_bridge_tag.dart +++ b/tool/testing/check_webgpu_bridge_tag.dart @@ -372,7 +372,7 @@ const String bridgeLlamaCppTag = 'v0.4.1'; const String bridgeLlamaCppCommit = 'b29c606e28a01b1bc8c1351026a0fa6e616bf6c4'; /// The exact bridge source commit used to build the pinned bridge assets. -const String bridgeSourceCommit = '89178be67c3c84300bc1b129182bd5bc5a8e21fc'; +const String bridgeSourceCommit = '64ba8250871bf2472cc2064c6a00fff050783f02'; /// Canonical repository identities recorded in the approved manifest. const String bridgeAssetsRepository = 'leehack/llama-web-bridge-assets'; @@ -389,14 +389,14 @@ const String bridgeNativeReleaseTag = 'v0.4.1'; const String bridgeApprovedNativePin = 'v0.4.1-1'; /// The asset repository release that published the pinned bridge assets. -const String bridgeAssetsReleaseId = '389783936'; +const String bridgeAssetsReleaseId = '394986324'; /// The asset repository commit the pinned bridge asset tag points at. -const String bridgeAssetsTagCommit = 'fdafd9f8cbdb9bf99c359536595eff9a23095379'; +const String bridgeAssetsTagCommit = 'ee45e864641648a99411128f9bb82b7897fad221'; /// SHA-256 hash of the exact approved published manifest.json. const String bridgeManifestSha256 = - '8d61f453753ac7a7d839ac12318b70986a814748d86029993118c19454293aa9'; + '9c5e9008d187690e283f37b2b892396da03e3c71bf7c3434d7c87ceca8e6a4bc'; /// Where the native runtime's llama.cpp build is pinned. const String nativeLlamaCppTagPath = 'lib/src/hook/native_release_pins.dart'; diff --git a/website/docs/changelog/recent-releases.md b/website/docs/changelog/recent-releases.md index 442c1c457..fcb6fac07 100644 --- a/website/docs/changelog/recent-releases.md +++ b/website/docs/changelog/recent-releases.md @@ -33,6 +33,14 @@ For canonical full release notes, use: - Add a notebook in `example/laya_tetris/training/` that fine-tunes a Laya decision head for the Tetris example and exports it for `DecisionEngine` ([#604](https://github.com/leehack/llamadart/issues/604)). +- Run `DecisionEngine` on WebGPU through the bridge decision API + (apiVersion 1), which bridge assets `v0.1.47+` include + ([#604](https://github.com/leehack/llamadart/issues/604)). +- Aligned default WebGPU bridge assets to `v0.1.47` for the decision API, + retaining Web/native llama.cpp + `v0.4.1@b29c606e28a01b1bc8c1351026a0fa6e616bf6c4` parity and Web + `@litert-lm/core@0.15.0`. Immutable Web asset manifest: + `9c5e9008d187690e283f37b2b892396da03e3c71bf7c3434d7c87ceca8e6a4bc`. - Extend the GGUF speech-to-text validation pack with four synthetic edge fixtures built in-process, so no extra audio is stored: generated digital silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second diff --git a/website/docs/guides/decision-models.md b/website/docs/guides/decision-models.md index 2e759d5e0..889e5ef5f 100644 --- a/website/docs/guides/decision-models.md +++ b/website/docs/guides/decision-models.md @@ -1,6 +1,6 @@ --- title: Decision Models -description: Answer typed choice, score, and yes/no questions about a state with Laya-style encoder decision models on native llama.cpp. +description: Answer typed choice, score, and yes/no questions about a state with Laya-style encoder decision models on llama.cpp. --- `DecisionEngine` answers typed questions about a state with a Laya-style @@ -25,13 +25,14 @@ in a Flutter app. | Runtime | `DecisionEngine` | | --- | --- | | Native llama.cpp / GGUF | Experimental: ModernBERT (`modern-bert`) encoder GGUF plus a Laya decision head; validated on macOS (Metal, CPU), other native platforms untested | -| WebGPU / GGUF | Unsupported: `DecisionEngine.load` throws `LlamaUnsupportedException` | +| WebGPU / GGUF | Experimental, with bridge assets `v0.1.47+` (apiVersion 1), which the default pin includes; checked only in headless Chromium on macOS. Older assets report unsupported. See [Web](#web) | | Native LiteRT-LM / `.litertlm` | Unsupported: `DecisionEngine.load` throws `LlamaUnsupportedException` | | LiteRT-LM Web | Unsupported: `DecisionEngine.load` throws `LlamaUnsupportedException` | The head runs on CPU when the model is loaded on CPU, and on the model's GPU when a device of its backend is available, otherwise on CPU. -`decisions.info.deviceName` names that device, such as `CPU` or `MTL0`. +`decisions.info.deviceName` names that device, such as `CPU` or `MTL0`; on Web, +the bridge reports its own device name. ## Load a decision model @@ -39,8 +40,8 @@ The reference assets are the community GGUF conversion [`fr0stbit3/laya-gguf`](https://huggingface.co/fr0stbit3/laya-gguf): the `laya-Q8_0.gguf` backbone (421 MB) and the `laya-head.safetensors` head (106 MB, F32). Load the backbone into a `LlamaEngine`, fetch the head through -the engine's model download manager, then load the head with -`DecisionEngine.load`: +the engine's model download manager (native only; on Web, pass a URL as shown +in [Web](#web)), then load the head with `DecisionEngine.load`: ```dart final engine = LlamaEngine(LlamaBackend()); @@ -356,9 +357,10 @@ for (final MapEntry(key: id, value: answer) in result.answers.entries) { ## Capabilities and model info `DecisionEngine.capabilitiesFor(engine)` reports whether a head can load on the -engine now. Probe it after the backbone is loaded: without a model, native -llama.cpp reports that a model must be loaded first. Web backends report -unsupported with or without a model. +engine now. Probe it after the backbone is loaded: without a model, it reports +that a model must be loaded first. With a model on Web, bridge assets without +the decision API, or with another decision API version, report unsupported and +name the assets needed. `decisions.info` describes the loaded model: `hiddenSize`, the sequence limit `maxTokens`, the question-and-options budget `headMaxTokens`, and the @@ -400,6 +402,56 @@ final official = await DecisionEngine.load( ); ``` +## Web + +On Web, `DecisionEngine` runs through the decision API (apiVersion 1) of the +llama.cpp WebGPU bridge, which `llama-web-bridge-assets` `v0.1.47+` and the +default pin include. With older assets, `capabilitiesFor` reports unsupported +and `DecisionEngine.load` throws `LlamaUnsupportedException`. LiteRT-LM Web +models report unsupported too. + +- `headPath` and `configPath` are URLs, resolved against the document base + URL, so a `` applies. The engine's model download manager is not + available on Web; pass the head's URL instead: + + ```dart + final head = ModelSource.huggingFace( + repoId: 'fr0stbit3/laya-gguf', + revision: 'ce2afdc0a8766af56a29a22dcf4a781e1f5c7d3c', + filePath: 'laya-head.safetensors', + ); + final decisions = await DecisionEngine.load( + engine, + headPath: head.resolvedUri!.toString(), + ); + ``` + +- The bridge downloads the head into its in-memory file system, so peak memory + includes the whole head file. The page fetches `configPath` and passes its + text to the bridge; a config that cannot be fetched throws + `LlamaModelException`. +- The head runs on WebGPU when the model loaded with GPU layers and on the + bridge CPU otherwise. +- Web numbers cannot tell `30.0` from `30`. In a state, instructions, criteria, + levels or descriptions that are not a `String`, an integral `double` is + written as an `int`: `{'seats': 30.0}` becomes `{"seats": 30}`, where native + and Laya write `{"seats": 30.0}`. The model reads different tokens, so + answers can differ from native. When that matters, pass the value as a + `String` you encode yourself. +- A bridge that restarts its runtime, for example when its worker fails during + a call, frees its heads. Calls then throw `LlamaStateException`; load the + `DecisionEngine` again. +- On the bridge CPU (no GPU layers), `laya-Q8_0.gguf` differs from Laya by + more than 0.05 in probability and 0.10 in score (0.0628 and 0.1224) on one of + the 24 questions in Laya's parity fixture, with the same top option. The + bridge's own smoke gets the same worst logit difference, so the drift comes + from the bridge's WASM CPU Q8_0 path, not llamadart. On the same fixture, a + locally converted F16 backbone, or GPU layers with either backbone, stays + within those bounds; [Accuracy and speed](#accuracy-and-speed) shows native + Q8_0 changing decisions on 187 random questions. The design doc's + [Web check](https://github.com/leehack/llamadart/blob/main/doc/decision_engine.md#web-check) + has the Web accuracy numbers. + ## Accuracy and speed On an Apple M4 Max (macOS), over Laya's 24-question parity fixture with @@ -412,8 +464,8 @@ including a yes/no answer that went from 0.694 to 0.457 on the CPU. An F16 conversion matched F32 on Metal and flipped two near-ties on the CPU. Use an F32 backbone, or F16 on Metal, when answers must match Laya. The design doc's [Measured](https://github.com/leehack/llamadart/blob/main/doc/decision_engine.md#measured) -section has the full tables and method. Other platforms and GPU backends have -not been measured. +section has the full tables and method. Other native platforms and GPU +backends have not been measured; [Web](#web) covers the bridge. ## Known limits @@ -443,3 +495,5 @@ not been measured. - **No U+0000.** A state, question or option text that contains U+0000 throws `LlamaDecisionException`, because native tokenization would cut the text there. A state that is not a `String` is sent as JSON, which escapes it. +- **Web numbers.** On Web, an integral `double` in JSON text, such as `30.0`, + is written as `30`, unlike native and Laya; see [Web](#web). diff --git a/website/docs/guides/text-to-speech.md b/website/docs/guides/text-to-speech.md index a4c2c883f..02e0f3daa 100644 --- a/website/docs/guides/text-to-speech.md +++ b/website/docs/guides/text-to-speech.md @@ -175,13 +175,13 @@ memory-constrained mobile devices. - Web requires published bridge assets `v0.1.33+`, WebAssembly memory64 for the pinned roughly 1.48 GB model/projector pair, and a browser/device with enough memory. Older bridge assets fail capability discovery clearly. -- The chat example pins `v0.1.44`, which retains the `v0.1.34` worker +- The chat example pins `v0.1.47`, which retains the `v0.1.34` worker recovery. Eligible WebGPU errors and generic worker timeouts retry once on the main thread using cached model/projector bytes with CPU-only settings. The exact `worker request timeout` and `worker init timeout` errors preserve the original GPU offload settings on that retry. Already CPU-only models are not retried, and cancellation takes precedence over recovery. Other errors - propagate unchanged. See the [bridge recovery contract](https://github.com/leehack/llama-web-bridge/blob/89178be67c3c84300bc1b129182bd5bc5a8e21fc/docs/api.md#synthesizespeechoptions). + propagate unchanged. See the [bridge recovery contract](https://github.com/leehack/llama-web-bridge/blob/64ba8250871bf2472cc2064c6a00fff050783f02/docs/api.md#synthesizespeechoptions). Recovery is slower and does not replace sufficient browser memory or validation of queue-watchdog timeouts on the target browser/device. - Web speaker references are selected-file bytes only; microphone speaker diff --git a/website/docs/platforms/support-matrix.md b/website/docs/platforms/support-matrix.md index 0c307c9ab..6ae5c1878 100644 --- a/website/docs/platforms/support-matrix.md +++ b/website/docs/platforms/support-matrix.md @@ -30,10 +30,12 @@ supports experimental CPU-only streaming ASR through isolate. LiteRT-LM Web does not expose typed speech. See the [speech recognition support matrix](../guides/speech-to-text#current-support-matrix). -Experimental Laya-style decision models run only on native llama.cpp: +Experimental Laya-style decision models run on native llama.cpp and WebGPU: [`DecisionEngine`](../guides/decision-models) pairs a ModernBERT encoder GGUF with a safetensors decision head. It is validated on macOS (Metal, CPU); other -native platforms are untested. On WebGPU, native LiteRT-LM, and LiteRT-LM Web, +native platforms are untested. WebGPU needs bridge assets `v0.1.47+`, which the +default pin includes; it was checked only in headless Chromium on macOS. On +native LiteRT-LM and LiteRT-LM Web, `DecisionEngine.load` throws `LlamaUnsupportedException`. Available override tags are published on the diff --git a/website/docs/platforms/webgpu-bridge.md b/website/docs/platforms/webgpu-bridge.md index b526144a1..51abc985b 100644 --- a/website/docs/platforms/webgpu-bridge.md +++ b/website/docs/platforms/webgpu-bridge.md @@ -146,13 +146,13 @@ development validation, and CDN-first loading for normal hosted deployments: 1. On localhost: local asset first, then CDN fallback. 2. On hosted deployments: CDN asset first, then local fallback. -The example currently pins bridge assets to `v0.1.44`, with local vendored assets -identified as `v0.1.44-local-v0.4.1`. +The example currently pins bridge assets to `v0.1.47`, with local vendored assets +identified as `v0.1.47-local-v0.4.1`. Fetch pinned local assets with: ```bash -WEBGPU_BRIDGE_ASSETS_TAG=v0.1.44 ./scripts/fetch_webgpu_bridge_assets.sh +WEBGPU_BRIDGE_ASSETS_TAG=v0.1.47 ./scripts/fetch_webgpu_bridge_assets.sh ``` To verify the loaded runtime in a browser console, inspect: @@ -201,13 +201,16 @@ cannot report success before the bridge exposes `prefetchModelToCache(...)`. physical playback, intelligibility, or speaker-reference fidelity. wasm32 TTS remains unsupported; use memory64. - `v0.1.39+` remains the compatibility floor for bridge asset capabilities. -- The pinned `v0.1.44` bridge assets embed llama.cpp `v0.4.1`, matching the native runtime +- `v0.1.47+` bridge assets include the decision API (apiVersion 1) that + [`DecisionEngine`](../guides/decision-models#web) needs; older assets report + decision models as unsupported. +- The pinned `v0.1.47` bridge assets embed llama.cpp `v0.4.1`, matching the native runtime (`v0.4.1-1`, both built from upstream `v0.4.1@b29c606e28a01b1bc8c1351026a0fa6e616bf6c4`) - even though the bridge asset tag `v0.1.44` differs from the native runtime tag - `v0.4.1-1`. Pinned artifact provenance: release `389783936`, tag commit - `fdafd9f8cbdb9bf99c359536595eff9a23095379`, bridge source - `89178be67c3c84300bc1b129182bd5bc5a8e21fc`, manifest SHA-256 - `8d61f453753ac7a7d839ac12318b70986a814748d86029993118c19454293aa9`. The + even though the bridge asset tag `v0.1.47` differs from the native runtime tag + `v0.4.1-1`. Pinned artifact provenance: release `394986324`, tag commit + `ee45e864641648a99411128f9bb82b7897fad221`, bridge source + `64ba8250871bf2472cc2064c6a00fff050783f02`, manifest SHA-256 + `9c5e9008d187690e283f37b2b892396da03e3c71bf7c3434d7c87ceca8e6a4bc`. The bridge assets were qualified against native `v0.4.1`; `v0.4.1-1` rebuilds it from the same upstream commit. The bridge assets provision an explicit 1 MiB stack for both wasm32 and memory64, preventing graph-parameter growth from @@ -343,7 +346,7 @@ You can override bridge asset source/version before loader startup: ```html