Problem
Portrait blur and background replace mangle hands when they're held away from the body,
against the background. The selfie-segmentation mask punches blocky holes through the
palm, chops fingers off at the knuckles, or drops a hand entirely. Hands resting on the
face or body seem ok, they're already inside the person region.
Cause
The bundled selfie_segmentation model (MediaPipe Selfie Segmentation, 256², ~460 KB) was
trained on selfie framing and doesn't reliably classify hands and forearms extended away
from the torso.
For what it's worth I checked whether swapping the segmenter is viable, and it isn't.
| model |
res |
median (Ryzen 5 PRO 5675U, 4 threads) |
selfie_segmentation (current) |
256² |
2.0 ms |
| MODNet |
512×288 |
92 ms |
| RobustVideoMatting (mnv3) |
512×288 |
88 ms |
Both alternatives land at 8–11 fps which is not good.
Proposal: use the hand models already bundled
palm_detection and hand_landmark are already shipped and already loaded for Reactions.
Their 21 landmarks can be rasterised into a hand silhouette and merged into the
segmentation mask, so hands stop depending on the segmenter recognising them.
Cost (same machine, 4 threads):
| stage |
median |
palm_detection 192² |
4.6 ms |
hand_landmark 224², per hand |
1.1 ms |
selfie_segmentation 256² |
2.0 ms |
Palm detection + two hands + segmentation ≈ 9 ms of a 33 ms frame at 30 fps — and the
existing REDETECT_INTERVAL / GESTURE_INTERVAL amortisation applies just as well here.
Measurements
Prototyped in Python against the bundled ONNX models: palm detection with the
opencv_zoo decode, landmarks per detected palm, silhouette from a filled palm hull plus
capsules along the bone connections, merged into the selfie mask.
40 frames at 1280×720, 8 fps, me gesticulating wildly with both hands raised:
Detection reliability : both hands detected, landmark confidence above threshold, in
40/40 frames. No dropouts (a flickering hand would be worse than the original artifact).
Temporal stability — mean per-frame |Δalpha|, lower is steadier:
| mask |
mean |
worst |
| stock selfie mask |
0.0414 |
0.1525 |
| hand silhouette alone |
0.0286 |
0.0806 |
| stock ∪ hands |
0.0430 |
0.1644 |
| stock ∪ hands, EMA α=0.5 |
0.0398 |
0.1475 |
The landmark geometry is steadier than the segmentation it's correcting, so merging it in
doesn't add jitter and with a simple EMA it comes out slightly better than stock on both mean
and worst case.
Existing is left, proposed new is right.
Implementation note
The better shape for this in your architecture is almost certainly to feed the hand alpha
as an additional prior into the existing fast-guided-filter step rather than merging
after the fact, so the edge snaps to the actual hand boundary against the frame luma the
same way the body edge already does.
Two limits: the v1 landmark model doesn't rotation-normalise its crop,
so heavily twisted wrists will degrade; and MAX_TRACKS = 2 caps this at two hands.
Environment
- OpenEffects 0.1.4 (released Arch packages), Omarchy / Arch, Hyprland (Wayland)
- Ryzen 5 PRO 5675U (6c/12t), integrated graphics, no discrete GPU
- Logitech C920 at 1280×720, PipeWire 1.6.8, GStreamer 1.28.6
Happy to share the prototype script if it's useful.
Problem
Portrait blur and background replace mangle hands when they're held away from the body,
against the background. The selfie-segmentation mask punches blocky holes through the
palm, chops fingers off at the knuckles, or drops a hand entirely. Hands resting on the
face or body seem ok, they're already inside the person region.
Cause
The bundled
selfie_segmentationmodel (MediaPipe Selfie Segmentation, 256², ~460 KB) wastrained on selfie framing and doesn't reliably classify hands and forearms extended away
from the torso.
For what it's worth I checked whether swapping the segmenter is viable, and it isn't.
selfie_segmentation(current)Both alternatives land at 8–11 fps which is not good.
Proposal: use the hand models already bundled
palm_detectionandhand_landmarkare already shipped and already loaded for Reactions.Their 21 landmarks can be rasterised into a hand silhouette and merged into the
segmentation mask, so hands stop depending on the segmenter recognising them.
Cost (same machine, 4 threads):
palm_detection192²hand_landmark224², per handselfie_segmentation256²Palm detection + two hands + segmentation ≈ 9 ms of a 33 ms frame at 30 fps — and the
existing
REDETECT_INTERVAL/GESTURE_INTERVALamortisation applies just as well here.Measurements
Prototyped in Python against the bundled ONNX models: palm detection with the
opencv_zoo decode, landmarks per detected palm, silhouette from a filled palm hull plus
capsules along the bone connections, merged into the selfie mask.
40 frames at 1280×720, 8 fps, me gesticulating wildly with both hands raised:
Detection reliability : both hands detected, landmark confidence above threshold, in
40/40 frames. No dropouts (a flickering hand would be worse than the original artifact).
Temporal stability — mean per-frame
|Δalpha|, lower is steadier:The landmark geometry is steadier than the segmentation it's correcting, so merging it in
doesn't add jitter and with a simple EMA it comes out slightly better than stock on both mean
and worst case.
Implementation note
The better shape for this in your architecture is almost certainly to feed the hand alpha
as an additional prior into the existing fast-guided-filter step rather than merging
after the fact, so the edge snaps to the actual hand boundary against the frame luma the
same way the body edge already does.
Two limits: the v1 landmark model doesn't rotation-normalise its crop,
so heavily twisted wrists will degrade; and
MAX_TRACKS = 2caps this at two hands.Environment
Happy to share the prototype script if it's useful.