Off-isolate frame worklets (setFrameWorklet) — 0.1.0

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.1.0
+
+* **Off-isolate frame worklets** — the headline feature. `CameraController.setFrameWorklet(entry, args:)`
+  runs heavy per-frame work (ML inference, CV) on a **background isolate**, with the main thread untouched;
+  results stream back via `CameraController.frameResults`. The worklet entry is a top-level function (Dart
+  can't ship closures across isolates) and receives a `FrameWorklet` context (`onFrame`, `send`, `args`).
+  Frames cross to the worker as the same zero-copy FFI pointers. The simpler main-isolate `setFrameProcessor`
+  stays for light work.
+* **Example:** the Object Detector now runs entirely off-isolate — EfficientDet-Lite0 inference on a worker,
+  with a live main-isolate UI-tick counter proving the main thread stays free.
+* **Internal:** shared `Frame.fromNative` factory; race-free cross-isolate worklet teardown.
+
 ## 0.0.6
 
 * **New:** `CameraController.previewRectFromFrame(rect, sourceRotationDegrees:)` maps a normalized

README.md

@@ -190,13 +190,16 @@ Run code for every camera frame — the package's headline feature.
 
 ### Threading model — read this
 
-- **Dart frame callback:** delivered **asynchronously on the main isolate's
-  event loop** (via `dart:ffi` `NativeCallable.listener`). It does **not** run
-  on a background isolate and does **not** block the camera thread. Keep the work
-  light, or copy data out and hand it to your own isolate. (A true off-isolate
-  worklet model is on the roadmap.)
-- **Native C/C++ hook:** runs **synchronously on the camera thread** with zero
-  added latency — use this for the heaviest work.
+Three options, lightest to heaviest:
+
+- **`setFrameProcessor`** — a closure delivered **asynchronously on the main
+  isolate** (via `dart:ffi` `NativeCallable.listener`). Easiest; keep the work
+  light (FPS, brightness) or it competes with your UI.
+- **`setFrameWorklet`** — a top-level function that runs on a **background
+  isolate** for heavy work (ML/CV), with the main thread untouched. Results
+  stream back via `frameResults`. See [below](#off-isolate-worklet-heavy-mlcv).
+- **Native C/C++ hook** — runs **synchronously on the camera thread** with zero
+  added latency, for the very heaviest work.
 
 ```dart
 await controller.setFrameProcessor((frame) {
@@ -212,6 +215,40 @@ await controller.setFrameProcessor((frame) {
 > padded). `computeLuminance` is **YUV/Android-only** — on iOS (BGRA) it returns
 > `0.0`; read `getPlaneData(0)` instead.
 
+### Off-isolate worklet (heavy ML/CV)
+
+For heavy per-frame work, run it on a **background isolate** so the UI never
+janks. The entry is a **top-level/static function** (Dart can't ship a closure
+across isolates). Pass model bytes via `args` — a worker isolate can't read
+`rootBundle`, so load assets on the main isolate. Register `onFrame`, and `send`
+results back:
+
+```dart
+// Top-level — runs on the worker isolate.
+void detectorWorklet(FrameWorklet w) {
+  final args = w.args as ({Uint8List model, String labels});
+  final interpreter = Interpreter.fromBuffer(args.model); // tflite_flutter
+  w.onFrame((frame) {
+    final boxes = runModel(interpreter, frame); // heavy inference, off-main
+    w.send(boxes);                              // sendable result → main
+  });
+}
+
+// Main isolate:
+final model =
+    (await rootBundle.load('assets/model.tflite')).buffer.asUint8List();
+final labels = await rootBundle.loadString('assets/labels.txt');
+controller.frameResults.listen((r) => setState(() => _boxes = r as List));
+await controller.setFrameWorklet(
+  detectorWorklet,
+  args: (model: model, labels: labels),
+);
+```
+
+Frames cross to the worker as the **same zero-copy FFI pointers** (no buffer
+copy). Map detection boxes onto the preview with `previewRectFromFrame`. The
+example's **Object Detector** is a full EfficientDet-Lite0 worklet.
+
 ### Keeping a frame past the callback
 
 The `Frame` and its buffers are valid **only during the callback**. To use the
@@ -283,6 +320,7 @@ working example.
 | Video recording (+ audio) | ✅ CameraX | ✅ AVAssetWriter |
 | Barcode / QR scanning | ✅ MLKit | ✅ Vision |
 | Frame processor (Dart, main isolate) | ✅ YUV planes | ✅ BGRA |
+| Frame worklet (Dart, background isolate) | ✅ | ✅ |
 | Native C/C++ plugin (camera thread) | ✅ | ✅ |
 | Zoom / torch / exposure / tap-focus | ✅ | ✅ |
 | Manual focus distance | ⬜ | ✅ |
@@ -296,8 +334,9 @@ between MLKit (Android) and Vision (iOS).
 ## Limitations & Roadmap
 
 - **No web / desktop** — Android + iOS only.
-- **Frame processor runs on the main isolate** today; a true off-isolate worklet
-  model is planned.
+- **Off-isolate worklets** (`setFrameWorklet`) run heavy work on a background
+  isolate; the simpler `setFrameProcessor` stays main-isolate by design (light
+  work). Worklet entries must be top-level functions (a Dart isolate constraint).
 - **Recording options** (`RecordVideoOptions`, codec/HDR) and **pause/resume**
   are not yet wired on all platforms; `startRecording` takes a path string.
 - **Mirror** is set at `initialize` time (no runtime toggle yet).

android/build.gradle

@@ -1,7 +1,7 @@
 // The Android Gradle Plugin builds the native code with the Android NDK.
 
 group = "dev.jentejan.flutter_native_vision_camera"
-version = "0.0.6"
+version = "0.1.0"
 
 buildscript {
     repositories {

example/lib/main.dart

@@ -87,7 +87,7 @@ class _HomePageState extends State<HomePage> {
             _buildFeatureCard(
               context,
               title: 'Object Detector',
-              subtitle: 'Frame processor → EfficientDet (90 classes) + boxes',
+              subtitle: 'Off-isolate worklet → EfficientDet (90 classes)',
               icon: Icons.center_focus_strong,
               color: Colors.deepPurple,
               onTap: () {