diff --git a/CHANGELOG.md b/CHANGELOG.md index 6ae2fd3..37f2b53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ # Changelog +## 0.1.0 + +* **Off-isolate frame worklets** — the headline feature. `CameraController.setFrameWorklet(entry, args:)` + runs heavy per-frame work (ML inference, CV) on a **background isolate**, with the main thread untouched; + results stream back via `CameraController.frameResults`. The worklet entry is a top-level function (Dart + can't ship closures across isolates) and receives a `FrameWorklet` context (`onFrame`, `send`, `args`). + Frames cross to the worker as the same zero-copy FFI pointers. The simpler main-isolate `setFrameProcessor` + stays for light work. +* **Example:** the Object Detector now runs entirely off-isolate — EfficientDet-Lite0 inference on a worker, + with a live main-isolate UI-tick counter proving the main thread stays free. +* **Internal:** shared `Frame.fromNative` factory; race-free cross-isolate worklet teardown. + ## 0.0.6 * **New:** `CameraController.previewRectFromFrame(rect, sourceRotationDegrees:)` maps a normalized diff --git a/README.md b/README.md index 8586f3b..be721e4 100644 --- a/README.md +++ b/README.md @@ -190,13 +190,16 @@ Run code for every camera frame — the package's headline feature. ### Threading model — read this -- **Dart frame callback:** delivered **asynchronously on the main isolate's - event loop** (via `dart:ffi` `NativeCallable.listener`). It does **not** run - on a background isolate and does **not** block the camera thread. Keep the work - light, or copy data out and hand it to your own isolate. (A true off-isolate - worklet model is on the roadmap.) -- **Native C/C++ hook:** runs **synchronously on the camera thread** with zero - added latency — use this for the heaviest work. +Three options, lightest to heaviest: + +- **`setFrameProcessor`** — a closure delivered **asynchronously on the main + isolate** (via `dart:ffi` `NativeCallable.listener`). Easiest; keep the work + light (FPS, brightness) or it competes with your UI. +- **`setFrameWorklet`** — a top-level function that runs on a **background + isolate** for heavy work (ML/CV), with the main thread untouched. Results + stream back via `frameResults`. See [below](#off-isolate-worklet-heavy-mlcv). +- **Native C/C++ hook** — runs **synchronously on the camera thread** with zero + added latency, for the very heaviest work. ```dart await controller.setFrameProcessor((frame) { @@ -212,6 +215,40 @@ await controller.setFrameProcessor((frame) { > padded). `computeLuminance` is **YUV/Android-only** — on iOS (BGRA) it returns > `0.0`; read `getPlaneData(0)` instead. +### Off-isolate worklet (heavy ML/CV) + +For heavy per-frame work, run it on a **background isolate** so the UI never +janks. The entry is a **top-level/static function** (Dart can't ship a closure +across isolates). Pass model bytes via `args` — a worker isolate can't read +`rootBundle`, so load assets on the main isolate. Register `onFrame`, and `send` +results back: + +```dart +// Top-level — runs on the worker isolate. +void detectorWorklet(FrameWorklet w) { + final args = w.args as ({Uint8List model, String labels}); + final interpreter = Interpreter.fromBuffer(args.model); // tflite_flutter + w.onFrame((frame) { + final boxes = runModel(interpreter, frame); // heavy inference, off-main + w.send(boxes); // sendable result → main + }); +} + +// Main isolate: +final model = + (await rootBundle.load('assets/model.tflite')).buffer.asUint8List(); +final labels = await rootBundle.loadString('assets/labels.txt'); +controller.frameResults.listen((r) => setState(() => _boxes = r as List)); +await controller.setFrameWorklet( + detectorWorklet, + args: (model: model, labels: labels), +); +``` + +Frames cross to the worker as the **same zero-copy FFI pointers** (no buffer +copy). Map detection boxes onto the preview with `previewRectFromFrame`. The +example's **Object Detector** is a full EfficientDet-Lite0 worklet. + ### Keeping a frame past the callback The `Frame` and its buffers are valid **only during the callback**. To use the @@ -283,6 +320,7 @@ working example. | Video recording (+ audio) | ✅ CameraX | ✅ AVAssetWriter | | Barcode / QR scanning | ✅ MLKit | ✅ Vision | | Frame processor (Dart, main isolate) | ✅ YUV planes | ✅ BGRA | +| Frame worklet (Dart, background isolate) | ✅ | ✅ | | Native C/C++ plugin (camera thread) | ✅ | ✅ | | Zoom / torch / exposure / tap-focus | ✅ | ✅ | | Manual focus distance | ⬜ | ✅ | @@ -296,8 +334,9 @@ between MLKit (Android) and Vision (iOS). ## Limitations & Roadmap - **No web / desktop** — Android + iOS only. -- **Frame processor runs on the main isolate** today; a true off-isolate worklet - model is planned. +- **Off-isolate worklets** (`setFrameWorklet`) run heavy work on a background + isolate; the simpler `setFrameProcessor` stays main-isolate by design (light + work). Worklet entries must be top-level functions (a Dart isolate constraint). - **Recording options** (`RecordVideoOptions`, codec/HDR) and **pause/resume** are not yet wired on all platforms; `startRecording` takes a path string. - **Mirror** is set at `initialize` time (no runtime toggle yet). diff --git a/android/build.gradle b/android/build.gradle index 32b1113..3752487 100644 --- a/android/build.gradle +++ b/android/build.gradle @@ -1,7 +1,7 @@ // The Android Gradle Plugin builds the native code with the Android NDK. group = "dev.jentejan.flutter_native_vision_camera" -version = "0.0.6" +version = "0.1.0" buildscript { repositories { diff --git a/example/lib/main.dart b/example/lib/main.dart index 34c5c99..c96697f 100644 --- a/example/lib/main.dart +++ b/example/lib/main.dart @@ -87,7 +87,7 @@ class _HomePageState extends State { _buildFeatureCard( context, title: 'Object Detector', - subtitle: 'Frame processor → EfficientDet (90 classes) + boxes', + subtitle: 'Off-isolate worklet → EfficientDet (90 classes)', icon: Icons.center_focus_strong, color: Colors.deepPurple, onTap: () { diff --git a/example/lib/object_detector_page.dart b/example/lib/object_detector_page.dart index aabce73..934c7fc 100644 --- a/example/lib/object_detector_page.dart +++ b/example/lib/object_detector_page.dart @@ -6,20 +6,14 @@ import 'package:flutter/services.dart' show rootBundle; import 'package:flutter_native_vision_camera/flutter_native_vision_camera.dart'; import 'package:tflite_flutter/tflite_flutter.dart'; -/// A real-time multi-class object detector built on the package's FFI frame -/// pipeline. +/// A real-time multi-class object detector that runs entirely **off the main +/// isolate** via the package's frame-worklet API. /// -/// Pipeline: [CameraController.setFrameProcessor] delivers each frame → we read -/// the raw YUV/BGRA buffers over FFI and resize+rotate them into the model's -/// input tensor → an **EfficientDet-Lite0** detector (90 COCO classes) runs in a -/// background isolate → we draw a labelled box for every object. -/// -/// On top of the general detection it keeps a **single-class certainty trigger** -/// for "cat": temporal voting turns the jittery per-frame score into a stable -/// "certain" verdict — the technique you'd use for a reliable real-time alert. -/// -/// The frame pipeline is generic, so swapping EfficientDet for your own -/// `.tflite` is a one-line change — that's the whole point of the FFI access. +/// The heavy work — YUV→RGB preprocessing and EfficientDet-Lite0 inference (90 +/// COCO classes) — happens in [objectDetectorWorklet] on a worker isolate. It +/// `send`s plain detection data back; the UI isolate only maps boxes into +/// preview space (cheap) and paints. The main thread never touches a frame, so +/// the preview and UI stay smooth no matter how heavy the model is. class ObjectDetectorPage extends StatefulWidget { const ObjectDetectorPage({super.key}); @@ -30,43 +24,39 @@ class ObjectDetectorPage extends StatefulWidget { class _ObjectDetectorPageState extends State with WidgetsBindingObserver { final CameraController _controller = CameraController(); - final FrameProcessorThrottler _throttler = FrameProcessorThrottler( - targetFps: 5, - ); - - Interpreter? _interpreter; - IsolateInterpreter? _isolate; - List _labels = []; - int _inputSize = 320; - List> _outShapes = []; - - static const double _displayGate = 0.40; // min score to draw a box - // Temporal voting for the "cat" trigger. + + // Cat-certainty voting (lives on the UI isolate — it's trivial). final List _catWindow = []; static const int _windowSize = 8; static const int _needHits = 4; + static const double _gate = 0.40; - bool _isInitialized = false; - bool _busy = false; - int _lastFrameDeg = -1; // diagnostics: log rotation only when it changes List<_Obj> _objects = []; double _catScore = 0; bool _catCertain = false; + bool _isInitialized = false; String? _error; + int _uiTicks = 0; // proves the main isolate stays free @override void initState() { super.initState(); WidgetsBinding.instance.addObserver(this); _init(); + _spin(); + } + + Future _spin() async { + while (mounted) { + await Future.delayed(const Duration(milliseconds: 16)); + if (mounted) setState(() => _uiTicks++); + } } @override void dispose() { WidgetsBinding.instance.removeObserver(this); _controller.dispose(); - _isolate?.close(); - _interpreter?.close(); super.dispose(); } @@ -88,8 +78,6 @@ class _ObjectDetectorPageState extends State setState(() => _error = 'Camera permission denied.'); return; } - await _loadModel(); - final devices = await CameraDevices.getAvailableCameraDevices(); if (devices.isEmpty) { setState(() => _error = 'No camera found.'); @@ -100,7 +88,17 @@ class _ObjectDetectorPageState extends State devices.first; await _controller.initialize(device, pixelFormat: PixelFormat.yuv); - await _controller.setFrameProcessor(_onFrame); + _controller.frameResults.listen(_onResult); + // A worker isolate can't read rootBundle, so load the model + labels on + // the main isolate and hand them to the worklet via `args`. + final model = (await rootBundle.load( + 'assets/efficientdet.tflite', + )).buffer.asUint8List(); + final labels = await rootBundle.loadString('assets/coco_labels.txt'); + await _controller.setFrameWorklet( + objectDetectorWorklet, + args: (model: model, labels: labels), + ); await _controller.setActive(true); setState(() => _isInitialized = true); } catch (e) { @@ -108,226 +106,31 @@ class _ObjectDetectorPageState extends State } } - Future _loadModel() async { - final interpreter = await Interpreter.fromAsset( - 'assets/efficientdet.tflite', - ); - _interpreter = interpreter; - _isolate = await IsolateInterpreter.create(address: interpreter.address); - - _inputSize = interpreter.getInputTensor(0).shape[1]; - _outShapes = interpreter - .getOutputTensors() - .map((t) => t.shape) - .toList(growable: false); - - final raw = await rootBundle.loadString('assets/coco_labels.txt'); - _labels = raw - .split('\n') - .map((e) => e.trim()) - .where((e) => e.isNotEmpty) - .toList(); - - debugPrint( - 'ObjectDetector: input=${interpreter.getInputTensor(0).shape}, ' - 'outputs=$_outShapes, labels=${_labels.length}', - ); - } - - void _onFrame(Frame frame) { - if (_isolate == null) return; - if (!_throttler.shouldProcess((frame.timestamp * 1000).toInt())) return; - if (_busy) return; - _busy = true; - - // DIAGNOSTIC: how do the frame's orientation and the preview's rotation - // relate as the phone turns? (Logged only when the frame orientation flips.) - final fdeg = _degreesOf(frame.orientation); - if (fdeg != _lastFrameDeg) { - _lastFrameDeg = fdeg; - debugPrint( - 'ObjRot: frame.orientation=$fdeg° previewQuarterTurns=' - '${_controller.previewRotation} mirrored=${_controller.previewMirrored} ' - 'displaySize=${_controller.displayPreviewSize}', + // Runs on the UI isolate: map the worker's frame-space boxes into preview + // space and update the overlay. This is the only per-frame main-isolate work. + void _onResult(Object? msg) { + if (msg is! _DetResult || !mounted) return; + final objs = <_Obj>[]; + var catScore = 0.0; + for (final d in msg.dets) { + final box = _controller.previewRectFromFrame( + Rect.fromLTRB(d.xmin, d.ymin, d.xmax, d.ymax), + sourceRotationDegrees: msg.rotation, ); + final isCat = d.label == 'cat'; + objs.add(_Obj(box, d.label, d.score, isCat)); + if (isCat && d.score > catScore) catScore = d.score; } - - // Read+resize+rotate the frame into the model input NOW (the FFI buffer is - // only valid inside this callback). - final Uint8List input; - try { - input = _frameToRgb(frame, _inputSize, fdeg); - } catch (_) { - _busy = false; - return; - } - - _detect(input, fdeg).then(_apply).whenComplete(() => _busy = false); - } - - Future<_Detection> _detect(Uint8List rgb, int frameDeg) async { - try { - final s = _inputSize; - final inputs = [rgb.reshape([1, s, s, 3])]; - final outputs = { - for (var i = 0; i < _outShapes.length; i++) i: _alloc(_outShapes[i]), - }; - await _isolate!.runForMultipleInputs(inputs, outputs); - - // EfficientDet's 4 outputs come back in a converter-dependent order, so - // classify them by shape (and value range) instead of assuming. - List>? boxes; // [N][4] ymin,xmin,ymax,xmax - final twoDim = >[]; // candidate scores/classes - for (var i = 0; i < _outShapes.length; i++) { - final shape = _outShapes[i]; - if (shape.length == 3 && shape.last == 4) { - boxes = _flatten2(outputs[i]); - } else if (shape.length == 2) { - twoDim.add(_flatten1(outputs[i])); - } - } - if (boxes == null || twoDim.length < 2) return const _Detection(0, []); - - // Of the two [1,N] tensors, classes holds indices (0..89) → larger max; - // scores are probabilities in [0,1]. - twoDim.sort((a, b) => _max(b).compareTo(_max(a))); - final classes = twoDim[0]; - final scores = twoDim[1]; - - var catScore = 0.0; - final objs = <_Obj>[]; - for (var j = 0; j < scores.length; j++) { - if (scores[j] < _displayGate) continue; - final ci = classes[j].round(); - if (ci < 0 || ci >= _labels.length) continue; - final label = _labels[ci]; - if (label == '???') continue; // COCO index gaps - final b = boxes[j]; // ymin, xmin, ymax, xmax (in the upright frame) - final isCat = label == 'cat'; - // Map the box into preview-display space with the package helper so it - // tracks the preview at any device orientation. - final box = _controller.previewRectFromFrame( - Rect.fromLTRB(b[1], b[0], b[3], b[2]), - sourceRotationDegrees: frameDeg, - ); - objs.add(_Obj(box, label, scores[j], isCat)); - if (isCat && scores[j] > catScore) catScore = scores[j]; - } - return _Detection(catScore, objs); - } catch (e) { - debugPrint('ObjectDetector inference error: $e'); - return const _Detection(0, []); - } - } - - void _apply(_Detection d) { - if (!mounted) return; - _catWindow.add(d.catScore); + _catWindow.add(catScore); if (_catWindow.length > _windowSize) _catWindow.removeAt(0); - final hits = _catWindow.where((s) => s >= _displayGate).length; + final hits = _catWindow.where((s) => s >= _gate).length; setState(() { - _objects = d.objects; - _catScore = d.catScore; + _objects = objs; + _catScore = catScore; _catCertain = _catWindow.length >= _windowSize && hits >= _needHits; }); } - /// Samples the YUV_420_888 frame into a packed RGB `uint8` buffer of - /// [size]x[size], applying [rotationDegrees] so the model sees an upright - /// image. Stride-correct (honours row/pixel strides) — the reason the package - /// exposes [Frame.planeBytesPerRow] / [Frame.planePixelStride]. - Uint8List _frameToRgb(Frame frame, int size, int rotationDegrees) { - final fw = frame.width, fh = frame.height; - final y = frame.getPlaneData(0); - final u = frame.getPlaneData(1); - final v = frame.getPlaneData(2); - final yRow = frame.planeBytesPerRow(0); - final uRow = frame.planeBytesPerRow(1); - final vRow = frame.planeBytesPerRow(2); - final uPix = frame.planePixelStride(1); - final vPix = frame.planePixelStride(2); - - final out = Uint8List(size * size * 3); - var o = 0; - for (var oy = 0; oy < size; oy++) { - final tv = (oy + 0.5) / size; // normalized in upright image - for (var ox = 0; ox < size; ox++) { - final tu = (ox + 0.5) / size; - // Map upright (tu,tv) back to the sensor-oriented frame. - final double nu, nv; - switch (rotationDegrees) { - case 90: - nu = tv; - nv = 1 - tu; - break; - case 180: - nu = 1 - tu; - nv = 1 - tv; - break; - case 270: - nu = 1 - tv; - nv = tu; - break; - default: - nu = tu; - nv = tv; - } - var fx = (nu * fw).toInt(); - var fy = (nv * fh).toInt(); - if (fx < 0) fx = 0; - if (fx >= fw) fx = fw - 1; - if (fy < 0) fy = 0; - if (fy >= fh) fy = fh - 1; - - final yy = y[fy * yRow + fx]; - final cx = fx >> 1, cy = fy >> 1; - final uu = u[cy * uRow + cx * uPix] - 128; - final vv = v[cy * vRow + cx * vPix] - 128; - - out[o++] = _clip(yy + ((1436 * vv) >> 10)); // R - out[o++] = _clip(yy - ((352 * uu + 731 * vv) >> 10)); // G - out[o++] = _clip(yy + ((1814 * uu) >> 10)); // B - } - } - return out; - } - - static int _clip(int v) => v < 0 ? 0 : (v > 255 ? 255 : v); - - static int _degreesOf(Orientation o) { - switch (o) { - case Orientation.landscapeLeft: - return 90; - case Orientation.portraitUpsideDown: - return 180; - case Orientation.landscapeRight: - return 270; - case Orientation.portrait: - return 0; - } - } - - // Output-buffer helpers. - static Object _alloc(List shape) { - if (shape.length == 1) return List.filled(shape[0], 0); - return List.generate(shape[0], (_) => _alloc(shape.sublist(1))); - } - - static List _flatten1(Object? o) => - ((o as List)[0] as List).cast().map((e) => e.toDouble()).toList(); - - static List> _flatten2(Object? o) => ((o as List)[0] as List) - .map((r) => (r as List).cast().map((e) => e.toDouble()).toList()) - .toList(); - - static double _max(List l) { - var m = double.negativeInfinity; - for (final v in l) { - if (v > m) m = v; - } - return m; - } - @override Widget build(BuildContext context) { return Scaffold( @@ -435,7 +238,7 @@ class _ObjectDetectorPageState extends State minHeight: 8, backgroundColor: Colors.white24, valueColor: AlwaysStoppedAnimation( - _catScore >= _displayGate + _catScore >= _gate ? Colors.greenAccent : Colors.amberAccent, ), @@ -453,9 +256,9 @@ class _ObjectDetectorPageState extends State ], ), const SizedBox(height: 6), - const Text( - 'EfficientDet-Lite0 · 90 COCO classes · cat-certainty by voting', - style: TextStyle(color: Colors.white70, fontSize: 11), + Text( + 'EfficientDet-Lite0 off-isolate · UI ticks $_uiTicks (smooth ⇒ main free)', + style: const TextStyle(color: Colors.white70, fontSize: 11), ), ], ), @@ -463,26 +266,191 @@ class _ObjectDetectorPageState extends State } } -/// One frame's detections. -class _Detection { - final double catScore; // best cat score this frame (drives the cat trigger) - final List<_Obj> objects; - const _Detection(this.catScore, this.objects); +// --------------------------------------------------------------------------- +// Worklet — everything below runs on the WORKER isolate (top-level only). +// --------------------------------------------------------------------------- + +typedef _Det = ({ + String label, + double score, + double ymin, + double xmin, + double ymax, + double xmax, +}); +typedef _DetResult = ({int rotation, List<_Det> dets}); + +/// Frame worklet: loads EfficientDet-Lite0 once, then runs YUV→RGB + inference +/// per frame on the worker isolate and sends back frame-space detections. +void objectDetectorWorklet(FrameWorklet w) { + final args = w.args as ({Uint8List model, String labels}); + final interpreter = Interpreter.fromBuffer(args.model); + final inputSize = interpreter.getInputTensor(0).shape[1]; + final outShapes = interpreter + .getOutputTensors() + .map((t) => t.shape) + .toList(growable: false); + final labels = args.labels + .split('\n') + .map((e) => e.trim()) + .where((e) => e.isNotEmpty) + .toList(); + final throttler = FrameProcessorThrottler(targetFps: 8); + + w.onFrame((frame) { + if (!throttler.shouldProcess((frame.timestamp * 1000).toInt())) return; + final rotation = frame.orientation.degrees; + final rgb = _frameToRgb(frame, inputSize, rotation); + final dets = _runDetection(interpreter, outShapes, labels, rgb, inputSize); + w.send((rotation: rotation, dets: dets)); + }); } +List<_Det> _runDetection( + Interpreter interpreter, + List> outShapes, + List labels, + Uint8List rgb, + int size, +) { + final inputs = [ + rgb.reshape([1, size, size, 3]), + ]; + final outputs = { + for (var i = 0; i < outShapes.length; i++) i: _alloc(outShapes[i]), + }; + interpreter.runForMultipleInputs(inputs, outputs); + + List>? boxes; // [N][4] ymin,xmin,ymax,xmax + final twoDim = >[]; + for (var i = 0; i < outShapes.length; i++) { + final shape = outShapes[i]; + if (shape.length == 3 && shape.last == 4) { + boxes = _flatten2(outputs[i]); + } else if (shape.length == 2) { + twoDim.add(_flatten1(outputs[i])); + } + } + if (boxes == null || twoDim.length < 2) return const []; + + // Of the two [1,N] tensors, classes holds indices (larger max); scores [0,1]. + twoDim.sort((a, b) => _max(b).compareTo(_max(a))); + final classes = twoDim[0], scores = twoDim[1]; + + final dets = <_Det>[]; + for (var j = 0; j < scores.length; j++) { + if (scores[j] < 0.40) continue; + final ci = classes[j].round(); + if (ci < 0 || ci >= labels.length) continue; + final label = labels[ci]; + if (label == '???') continue; + final b = boxes[j]; + dets.add(( + label: label, + score: scores[j], + ymin: b[0], + xmin: b[1], + ymax: b[2], + xmax: b[3], + )); + } + return dets; +} + +/// Samples a YUV_420_888 frame into a packed RGB `uint8` buffer of [size]×[size], +/// applying [rotationDegrees] (stride-correct — honours row/pixel strides). +Uint8List _frameToRgb(Frame frame, int size, int rotationDegrees) { + final fw = frame.width, fh = frame.height; + final y = frame.getPlaneData(0); + final u = frame.getPlaneData(1); + final v = frame.getPlaneData(2); + final yRow = frame.planeBytesPerRow(0); + final uRow = frame.planeBytesPerRow(1); + final vRow = frame.planeBytesPerRow(2); + final uPix = frame.planePixelStride(1); + final vPix = frame.planePixelStride(2); + + final out = Uint8List(size * size * 3); + var o = 0; + for (var oy = 0; oy < size; oy++) { + final tv = (oy + 0.5) / size; + for (var ox = 0; ox < size; ox++) { + final tu = (ox + 0.5) / size; + final double nu, nv; + switch (rotationDegrees) { + case 90: + nu = tv; + nv = 1 - tu; + break; + case 180: + nu = 1 - tu; + nv = 1 - tv; + break; + case 270: + nu = 1 - tv; + nv = tu; + break; + default: + nu = tu; + nv = tv; + } + var fx = (nu * fw).toInt(); + var fy = (nv * fh).toInt(); + if (fx < 0) fx = 0; + if (fx >= fw) fx = fw - 1; + if (fy < 0) fy = 0; + if (fy >= fh) fy = fh - 1; + + final yy = y[fy * yRow + fx]; + final cx = fx >> 1, cy = fy >> 1; + final uu = u[cy * uRow + cx * uPix] - 128; + final vv = v[cy * vRow + cx * vPix] - 128; + + out[o++] = _clip(yy + ((1436 * vv) >> 10)); // R + out[o++] = _clip(yy - ((352 * uu + 731 * vv) >> 10)); // G + out[o++] = _clip(yy + ((1814 * uu) >> 10)); // B + } + } + return out; +} + +int _clip(int v) => v < 0 ? 0 : (v > 255 ? 255 : v); + +Object _alloc(List shape) { + if (shape.length == 1) return List.filled(shape[0], 0); + return List.generate(shape[0], (_) => _alloc(shape.sublist(1))); +} + +List _flatten1(Object? o) => + ((o as List)[0] as List).cast().map((e) => e.toDouble()).toList(); + +List> _flatten2(Object? o) => ((o as List)[0] as List) + .map((r) => (r as List).cast().map((e) => e.toDouble()).toList()) + .toList(); + +double _max(List l) { + var m = double.negativeInfinity; + for (final v in l) { + if (v > m) m = v; + } + return m; +} + +// --------------------------------------------------------------------------- +// UI-isolate types. +// --------------------------------------------------------------------------- + class _Obj { - final Rect box; // normalized 0..1 in the upright image + final Rect box; // preview-display space final String label; final double score; final bool isCat; const _Obj(this.box, this.label, this.score, this.isCat); } -/// Draws each detection, mapping a normalized box in the upright image onto the -/// `contain`-fitted preview rect. Cats use [catColor]; other objects are cyan. class _BoxPainter extends CustomPainter { final List<_Obj> objects; - final Size? previewSize; // upright preview dimensions + final Size? previewSize; final Color catColor; _BoxPainter(this.objects, this.previewSize, this.catColor); @@ -503,7 +471,6 @@ class _BoxPainter extends CustomPainter { for (final obj in objects) { final color = obj.isCat ? catColor : Colors.cyanAccent; - // obj.box is already in preview-display space (see previewRectFromFrame). final r = Rect.fromLTRB( img.left + obj.box.left * img.width, img.top + obj.box.top * img.height, diff --git a/example/pubspec.lock b/example/pubspec.lock index 5cefceb..861741d 100644 --- a/example/pubspec.lock +++ b/example/pubspec.lock @@ -140,7 +140,7 @@ packages: path: ".." relative: true source: path - version: "0.0.5" + version: "0.1.0" flutter_plugin_android_lifecycle: dependency: transitive description: diff --git a/ios/flutter_native_vision_camera.podspec b/ios/flutter_native_vision_camera.podspec index 8e1ad81..495ae0d 100644 --- a/ios/flutter_native_vision_camera.podspec +++ b/ios/flutter_native_vision_camera.podspec @@ -4,7 +4,7 @@ # Pod::Spec.new do |s| s.name = 'flutter_native_vision_camera' - s.version = '0.0.6' + s.version = '0.1.0' s.summary = 'High-performance Flutter FFI camera plugin with zero-copy preview and real-time frame access.' s.description = <<-DESC A high-performance camera plugin for Flutter built on AVFoundation (iOS) and CameraX (Android), diff --git a/lib/flutter_native_vision_camera.dart b/lib/flutter_native_vision_camera.dart index e2ea25c..6880e33 100644 --- a/lib/flutter_native_vision_camera.dart +++ b/lib/flutter_native_vision_camera.dart @@ -17,6 +17,7 @@ export 'src/camera_permissions.dart'; export 'src/camera_preview.dart'; export 'src/frame.dart'; export 'src/frame_processor.dart'; +export 'src/frame_worklet.dart' show FrameWorklet, FrameWorkletEntry; export 'src/types/types.dart'; export 'src/types/code_scanner.dart'; diff --git a/lib/src/camera_controller.dart b/lib/src/camera_controller.dart index b95490e..c5c7e61 100644 --- a/lib/src/camera_controller.dart +++ b/lib/src/camera_controller.dart @@ -1,9 +1,11 @@ import 'dart:async'; +import 'dart:isolate'; import 'package:flutter/foundation.dart'; import 'package:flutter/services.dart'; import '../flutter_native_vision_camera.dart'; +import 'frame_worklet.dart'; /// The method channel used for communication with native platform code. const MethodChannel _channel = MethodChannel( @@ -45,6 +47,13 @@ class CameraController extends ValueNotifier { CameraDevice? _device; FrameProcessorPipeline? _frameProcessorPipeline; + + // Off-isolate frame worklet state. + Isolate? _workletIsolate; + SendPort? _workletControl; + ReceivePort? _workletReceive; + Completer? _workletStopped; + StreamController? _frameResults; int? _previewWidth; int? _previewHeight; int? _previewRotationDegrees; @@ -236,6 +245,7 @@ class CameraController extends ValueNotifier { // Tear down any frame processor from a previous session before re-init / // device-switch, so frames already queued from the old session don't read // buffers the native side is about to recycle (use-after-free). + await _stopWorklet(); _frameProcessorPipeline?.stop(); _frameProcessorPipeline = null; @@ -399,6 +409,7 @@ class CameraController extends ValueNotifier { /// own isolate. For the heaviest work, register a native C/C++ plugin, which /// runs synchronously on the camera thread. Pass `null` to disable. Future setFrameProcessor(FrameProcessorCallback? callback) async { + await _stopWorklet(); _frameProcessorPipeline?.stop(); _frameProcessorPipeline = null; @@ -412,6 +423,76 @@ class CameraController extends ValueNotifier { }); } + /// A broadcast stream of results sent from a frame worklet via + /// `FrameWorklet.send`. Listen here to drive the UI from worklet output. + Stream get frameResults => + (_frameResults ??= StreamController.broadcast()).stream; + + /// Runs a frame worklet on a **background isolate** — the high-performance + /// path for heavy per-frame work (ML inference, CV) that would jank the UI if + /// run on the main isolate via [setFrameProcessor]. + /// + /// [entry] must be a **top-level or static** function (Dart can't ship a + /// closure that captures state to another isolate). It runs once on the worker + /// to set up state (e.g. load a model), registers a per-frame handler with + /// `worklet.onFrame`, and returns results with `worklet.send` — which surface + /// on [frameResults]. + /// + /// Mutually exclusive with [setFrameProcessor] (the native layer has one + /// callback slot). Pass `null` to stop the worklet. Frames cross to the worker + /// as the same zero-copy FFI pointers — no buffer copy. + /// + /// [args] is a sendable value handed to the worklet as `FrameWorklet.args` — + /// use it for initialization data the worker can't load itself. A background + /// isolate **can't read assets** (`rootBundle` needs the main isolate), so + /// load your model bytes on the main isolate and pass them here. + Future setFrameWorklet(FrameWorkletEntry? entry, {Object? args}) async { + await _stopWorklet(); + _frameProcessorPipeline?.stop(); + _frameProcessorPipeline = null; + + if (entry != null) { + _frameResults ??= StreamController.broadcast(); + final rp = ReceivePort(); + _workletReceive = rp; + rp.listen((msg) { + if (msg is SendPort) { + _workletControl = msg; + } else if (msg == frameWorkletStoppedSignal) { + _workletStopped?.complete(); + } else { + final out = _frameResults; + if (out != null && !out.isClosed) out.add(msg); + } + }); + _workletIsolate = await spawnFrameWorklet( + entry, + rp.sendPort, + RootIsolateToken.instance, + args, + ); + } + + await _channel.invokeMethod('setFrameProcessor', { + 'enabled': entry != null, + }); + } + + /// Stops the frame worklet, waiting for the worker to detach the native + /// callback before returning (so a subsequent processor can't race it). + Future _stopWorklet() async { + if (_workletIsolate == null) return; + final done = _workletStopped = Completer(); + _workletControl?.send('stop'); + await done.future.timeout(const Duration(seconds: 2), onTimeout: () {}); + _workletStopped = null; + _workletControl = null; + _workletReceive?.close(); + _workletReceive = null; + _workletIsolate?.kill(priority: Isolate.beforeNextEvent); + _workletIsolate = null; + } + /// Updates the code scanner configuration. Future setCodeScanner(CodeScannerConfiguration? configuration) async { await _channel.invokeMethod('setCodeScanner', { @@ -455,6 +536,13 @@ class CameraController extends ValueNotifier { // Stop any active frame processing immediately _frameProcessorPipeline?.stop(); _frameProcessorPipeline = null; + _workletControl?.send('stop'); + _workletReceive?.close(); + _workletReceive = null; + _workletIsolate?.kill(priority: Isolate.beforeNextEvent); + _workletIsolate = null; + _frameResults?.close(); + _frameResults = null; // Notify native side to shut down camera hardware _channel.invokeMethod('dispose'); diff --git a/lib/src/frame.dart b/lib/src/frame.dart index 2f6b39d..278639b 100644 --- a/lib/src/frame.dart +++ b/lib/src/frame.dart @@ -62,6 +62,55 @@ class Frame { required this.timestamp, }); + /// Builds a [Frame] from a native handle + metadata struct, mapping the raw + /// platform format/orientation codes to the Dart enums. Shared by the + /// main-isolate pipeline and the worklet runtime. + factory Frame.fromNative(Pointer handle, FrameMetadataNative metadata) { + return Frame( + handle, + width: metadata.width, + height: metadata.height, + pixelFormat: _mapPixelFormat(metadata.pixelFormat), + orientation: _mapOrientation(metadata.orientation), + timestamp: metadata.timestamp, + ); + } + + static PixelFormat _mapPixelFormat(int nativeFormat) { + switch (nativeFormat) { + case 35: // android.graphics.ImageFormat.YUV_420_888 + case 842094169: // android.graphics.ImageFormat.YV12 + return PixelFormat.yuv; + case 1: // BGRA (iOS) / RGB family + case 22: // android.graphics.ImageFormat.RGBA_8888 + return PixelFormat.rgb; + default: + if (nativeFormat >= 0 && nativeFormat < PixelFormat.values.length) { + return PixelFormat.values[nativeFormat]; + } + return PixelFormat.unknown; + } + } + + static Orientation _mapOrientation(int nativeOrientation) { + switch (nativeOrientation) { + case 0: + return Orientation.portrait; + case 90: + return Orientation.landscapeLeft; + case 180: + return Orientation.portraitUpsideDown; + case 270: + return Orientation.landscapeRight; + default: + if (nativeOrientation >= 0 && + nativeOrientation < Orientation.values.length) { + return Orientation.values[nativeOrientation]; + } + return Orientation.portrait; + } + } + /// The number of bytes in each row of the frame's image data. int get bytesPerRow => _getBytesPerRow(_pointer); diff --git a/lib/src/frame_processor.dart b/lib/src/frame_processor.dart index aaa8575..6413e4d 100644 --- a/lib/src/frame_processor.dart +++ b/lib/src/frame_processor.dart @@ -4,8 +4,6 @@ import 'dart:ffi'; import 'package:flutter/foundation.dart'; import 'frame.dart'; -import 'types/orientation.dart'; -import 'types/pixel_format.dart'; /// The signature of a frame processor callback. /// @@ -52,14 +50,7 @@ class FrameProcessorPipeline { /// pipeline was stopped between dispatch and delivery, or the user callback /// throws. void _onNativeFrame(Pointer handle, FrameMetadataNative metadata) { - final frame = Frame( - handle, - width: metadata.width, - height: metadata.height, - pixelFormat: _mapPixelFormat(metadata.pixelFormat), - orientation: _mapOrientation(metadata.orientation), - timestamp: metadata.timestamp, - ); + final frame = Frame.fromNative(handle, metadata); try { if (!_stopped) callback(frame); } catch (e, stack) { @@ -82,41 +73,6 @@ class FrameProcessorPipeline { _nativeCallable?.close(); _nativeCallable = null; } - - static PixelFormat _mapPixelFormat(int nativeFormat) { - switch (nativeFormat) { - case 35: // android.graphics.ImageFormat.YUV_420_888 - case 842094169: // android.graphics.ImageFormat.YV12 - return PixelFormat.yuv; - case 1: // BGRA (iOS) / RGB family - case 22: // android.graphics.ImageFormat.RGBA_8888 - return PixelFormat.rgb; - default: - if (nativeFormat >= 0 && nativeFormat < PixelFormat.values.length) { - return PixelFormat.values[nativeFormat]; - } - return PixelFormat.unknown; - } - } - - static Orientation _mapOrientation(int nativeOrientation) { - switch (nativeOrientation) { - case 0: - return Orientation.portrait; - case 90: - return Orientation.landscapeLeft; - case 180: - return Orientation.portraitUpsideDown; - case 270: - return Orientation.landscapeRight; - default: - if (nativeOrientation >= 0 && - nativeOrientation < Orientation.values.length) { - return Orientation.values[nativeOrientation]; - } - return Orientation.portrait; - } - } } /// Helper to throttle frame processing to a target FPS. diff --git a/lib/src/frame_worklet.dart b/lib/src/frame_worklet.dart new file mode 100644 index 0000000..b0a8a3b --- /dev/null +++ b/lib/src/frame_worklet.dart @@ -0,0 +1,141 @@ +import 'dart:async'; +import 'dart:ffi'; +import 'dart:io'; +import 'dart:isolate'; + +import 'package:flutter/foundation.dart'; +import 'package:flutter/services.dart'; + +import 'frame.dart'; + +/// A top-level (or static) function that sets up a frame worklet. It runs +/// **once on the worker isolate**: initialise any persistent state (load a +/// model, etc.), then register a handler with [FrameWorklet.onFrame] and return +/// results with [FrameWorklet.send]. +/// +/// It MUST be a top-level or static function — Dart cannot send a closure that +/// captures state to another isolate. Pass it to +/// `CameraController.setFrameWorklet`. +typedef FrameWorkletEntry = FutureOr Function(FrameWorklet worklet); + +/// The context handed to a [FrameWorkletEntry], living on the worker isolate. +class FrameWorklet { + FrameWorklet._(this._send, this.args); + + final void Function(Object? result) _send; + + /// The sendable initialization value passed to `setFrameWorklet(entry, + /// args:)`. Use it for data the worklet can't obtain itself — most importantly + /// **assets**: a background isolate can't read `rootBundle`, so load your model + /// bytes on the main isolate and pass them here (then `Interpreter.fromBuffer`). + final Object? args; + + void Function(Frame frame)? _handler; + + /// Registers the per-frame [handler]. It runs on the worker isolate for every + /// dispatched camera frame; the [Frame] (and its buffers) are valid only for + /// the duration of the call. + void onFrame(void Function(Frame frame) handler) => _handler = handler; + + /// Sends a **sendable** [result] back to the main isolate, where it surfaces + /// on `CameraController.frameResults`. Use plain data (numbers, strings, + /// lists, maps, records) — not platform handles or widgets. + void send(Object? result) => _send(result); +} + +/// Opens the plugin's dynamic library on the current isolate. The binding +/// globals in `frame.dart` are per-isolate `late` fields, so a worker isolate +/// must open the library and call [initializeFrameBindings] itself. +DynamicLibrary _openDylib() { + const name = 'flutter_native_vision_camera'; + if (Platform.isMacOS || Platform.isIOS) { + return DynamicLibrary.open('$name.framework/$name'); + } + if (Platform.isAndroid || Platform.isLinux) { + return DynamicLibrary.open('lib$name.so'); + } + if (Platform.isWindows) return DynamicLibrary.open('$name.dll'); + throw UnsupportedError('Unknown platform: ${Platform.operatingSystem}'); +} + +class _WorkletBootstrap { + const _WorkletBootstrap(this.entry, this.toMain, this.rootToken, this.args); + final FrameWorkletEntry entry; + final SendPort toMain; + final RootIsolateToken? rootToken; + final Object? args; +} + +/// Spawns the worker isolate that runs [entry]. The worker sends its control +/// [SendPort] (first message) and then each `send()` result to [toMain]. +Future spawnFrameWorklet( + FrameWorkletEntry entry, + SendPort toMain, + RootIsolateToken? rootToken, + Object? args, +) { + return Isolate.spawn( + _workletMain, + _WorkletBootstrap(entry, toMain, rootToken, args), + debugName: 'FrameWorklet', + ); +} + +Future _workletMain(_WorkletBootstrap b) async { + // Enable platform channels / rootBundle on this isolate (so the worklet can + // load assets, e.g. a .tflite model). + final token = b.rootToken; + if (token != null) { + BackgroundIsolateBinaryMessenger.ensureInitialized(token); + } + initializeFrameBindings(_openDylib()); + + final control = ReceivePort(); + b.toMain.send(control.sendPort); // hand main a stop channel + keep us alive + + final ctx = FrameWorklet._(b.toMain.send, b.args); + late final NativeCallable callable; + + void dispatch(Pointer handle, FrameMetadataNative metadata) { + final frame = Frame.fromNative(handle, metadata); + try { + ctx._handler?.call(frame); + } catch (e, s) { + if (kDebugMode) debugPrint('FrameWorklet handler threw: $e\n$s'); + } finally { + // Release the reference the native dispatcher took on our behalf. + frame.decrementRefCount(); + } + } + + callable = NativeCallable.listener( + dispatch, + ); + + // Run the user's setup (may await, e.g. loading a model). Frames don't flow + // until we register the native callback below, so there's no early race. + try { + await b.entry(ctx); + } catch (e, s) { + if (kDebugMode) debugPrint('FrameWorklet entry failed: $e\n$s'); + } + setNativeFrameProcessorCallback(callable.nativeFunction); + + control.listen((msg) { + if (msg == 'stop') { + // Detach the native side first (mutex-guarded in C, so no dispatch is in + // flight), then tear down and confirm before this isolate exits. + setNativeFrameProcessorCallback(nullptr); + callable.close(); + b.toMain.send(_workletStopped); + control.close(); + } + }); +} + +/// Sentinel the worker sends once it has cleared the native callback, so the +/// main isolate can safely start a new processor without a teardown race. +const String _workletStopped = '__frame_worklet_stopped__'; + +/// Exposed so the controller recognises the stop-confirmation sentinel. +const String frameWorkletStoppedSignal = _workletStopped; diff --git a/pubspec.yaml b/pubspec.yaml index 3a50c0e..a2e22c7 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -1,6 +1,6 @@ name: flutter_native_vision_camera description: "High-performance Flutter FFI camera plugin with zero-copy preview textures, integrated MLKit/Vision barcode scanning, and low-latency native frame access for real-time on-device vision." -version: 0.0.6 +version: 0.1.0 homepage: https://github.com/JenteJan/flutter_native_vision_camera repository: https://github.com/JenteJan/flutter_native_vision_camera issue_tracker: https://github.com/JenteJan/flutter_native_vision_camera/issues