Off-isolate frame worklets (setFrameWorklet) — 0.1.0 - #5
Merged
Conversation
Run heavy per-frame work (ML/CV) on a background isolate, main thread untouched: - setFrameWorklet(entry, args:) spawns a worker; the NativeCallable.listener is created ON the worker, so the native camera thread delivers frames there (proven on-device). Frames cross as the same zero-copy FFI pointers. - FrameWorklet context: onFrame / send / args. Results stream via frameResults. - Entry is a top-level fn (Dart can't ship closures across isolates); a worker can't read rootBundle, so model bytes come via args. - Race-free cross-isolate teardown; shared Frame.fromNative factory. - setFrameProcessor (main isolate) stays for light work.
EfficientDet-Lite0 inference + YUV->RGB now run in a worklet on a worker isolate; the UI isolate only maps boxes (previewRectFromFrame) and paints. A live main-isolate UI-tick counter shows the main thread stays free under heavy load. Model bytes are loaded on main and passed via args (Interpreter.fromBuffer).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The headline differentiator's last gap.
setFrameWorklet(entry, args:)runs heavy per-frame work (ML/CV) on a background isolate with the main thread untouched; results stream viaframeResults.Mechanism (device-proven on Pixel 8): the
NativeCallable.listeneris created on the worker isolate, so the native camera thread delivers frames there — frames cross as the same zero-copy FFI pointers. The entry is a top-level function (Dart can't ship closures across isolates); model bytes come viaargs(a worker can't read rootBundle).Example: the Object Detector now runs EfficientDet-Lite0 fully off-isolate, with a live main-isolate UI-tick counter proving the main thread stays free. Both platforms build; 14 tests pass;
setFrameProcessorstays for light work.