From a5121bd6a8cc060a53c5e2b1054fca137e6a850c Mon Sep 17 00:00:00 2001 From: Bastian Machek <16717398+bmachek@users.noreply.github.com> Date: Wed, 4 Nov 2026 08:17:18 +0100 Subject: [PATCH 1/4] feat(cull): read eyes and gaze the way real photographs show them The face state thresholds were pinned on one portrait and on lids moved shut on the mesh. Measured on photographs (examples/face_state_probe.rs, 285 faces from Wikimedia Commons to choose by, 292 from a disjoint sample to check by, not committed), neither reading meant what the ranking took it to mean. Eyes. eyeBlink tops out low: eyes shut read 0.54-0.78 on the more closed eye and 0.41-0.72 as a mean, so the mean at 0.6 caught 6 of 22 shut or lowered pairs. The more closed eye now speaks, with thresholds where the readings are: 19 of 22, about 17 of 24 on the second sample (was about 8), and 1 of 178 open-eyed portraits over the label. Gaze. eyeLook* is the eye inside the head, so the label had it backwards: people turned 25-30 degrees looking straight at the camera read 0.85-0.99, a head turned 39 degrees with the eyes following it 0.14. It is now degrees off the lens, head and eyes together, from a head pose read off a frame of bone landmarks (within 1 degree of a full Procrustes fit on yaw and roll, 2 on pitch, at no linear algebra dependency). looking_away in portraits drops from 25-31% to 10-12% and rises from 53% to 67% in People looking away. Shut eyes abstain; their eyeLook* is noise. Squint is read at last, only on a face that is not smiling. All 52 blendshapes are stored per face, so a later signal need not decode every original again. The version check read the key's presence, so a bump would have changed nothing; its value counts now. #471 is in no release, so only development databases are analysed again. Also from the live test: shut eyes were reported as the best open eyes of their group, a single always being its own best. Limits, in the wiki: a slit overlaps eyes shut, sunglasses read closed (the relative rules hold inside a burst), a head on its side reads open. The sports and street thresholds are scaled by judgement, not measured. A hand-culled shoot still has to confirm all of it. Verified: cargo fmt/clippy/test (workspace) with face and facestate goldens on the real models, check-docs, and live against the release binary by path. Docs-Reviewed: Help-Analyze-and-Index.md Docs-Reviewed: Help-Find-Similar.md Docs-Reviewed: Credits.md Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy --- docs/wiki/Dev-Backend-API.md | 27 +- .../Dev-Image-Culling-Implementation-Plan.md | 60 ++- docs/wiki/Help-Cull-Photos.md | 19 +- .../crates/lrg-analysis/src/culling_config.rs | 41 +- .../crates/lrg-analysis/src/face_aggregate.rs | 86 ++++- server-rs/crates/lrg-analysis/src/grouping.rs | 146 +++++++- .../lrg-api/src/routes/group_similar.rs | 4 +- server-rs/crates/lrg-api/src/routes/index.rs | 2 +- .../crates/lrg-api/src/routes/index_upload.rs | 32 +- server-rs/crates/lrg-api/tests/contract.rs | 35 +- .../crates/lrg-imaging/src/cull_config.rs | 24 +- server-rs/crates/lrg-ml/Cargo.toml | 3 + .../lrg-ml/examples/face_state_probe.rs | 89 +++++ server-rs/crates/lrg-ml/src/face_state.rs | 350 +++++++++++++++--- .../crates/lrg-ml/tests/face_state_goldens.rs | 12 +- 15 files changed, 771 insertions(+), 159 deletions(-) create mode 100644 server-rs/crates/lrg-ml/examples/face_state_probe.rs diff --git a/docs/wiki/Dev-Backend-API.md b/docs/wiki/Dev-Backend-API.md index 4a692cf4..d4477f56 100644 --- a/docs/wiki/Dev-Backend-API.md +++ b/docs/wiki/Dev-Backend-API.md @@ -267,12 +267,18 @@ on disk, MediaPipe's face mesh and blendshapes run on the prominent faces (at most 12, largest first) of the same full-size image, or of the working copy when there is no larger one (`lrg_ml::face_state`). Per face the row gets `face_eye_state` (`open`/`half`/`closed`, `unknown` when the face is turned too -far, barely a face or too small), `face_eye_closed`, `face_smile`, -`face_gaze_away` and `face_yaw` (null when the model did not run for it). Per +far, barely a face or too small), `face_eye_closed` (the more closed eye), +`face_smile`, `face_eye_squint`, `face_gaze_off_lens` (degrees from the lens, head +turn included), `face_yaw`, `face_pitch`, `face_roll` (degrees) and +`face_blendshapes` (all 52, in the order of `lrg_ml::face_state::BLENDSHAPE_NAMES`); +all null when the model did not run for it. These rows exist only where the pass +writes faces, not on a pass that only prepares culling. Per photo: `cull_eye_closed` (the most closed prominent eyes), `cull_eye_state`, -`cull_smile` (the weakest smile among the prominent faces), `cull_gaze_away` — -absent when no prominent face could be judged — plus `cull_face_state_version` -and `cull_face_state_model` when the model ran on this pass. Without the model +`cull_smile` (the weakest smile among the prominent faces), `cull_eye_squint` (the +strongest squint among the prominent faces not smiling), `cull_gaze_off_lens` (the +prominent face with open eyes looking furthest from the lens) — absent when no +prominent face could be judged — plus `cull_face_state_version` and `cull_face_state_model` when +the model ran on this pass. Without the model the response says once that a rough eye estimate was used, with the fix (`"Photos were analysed without eye state, smile and gaze: …"`). @@ -608,10 +614,11 @@ abstain. They act only when every frame of a group with a face carries `portrait` weights and gates focus most, `sports` least, `street` not at all. -**Eye state, smile and gaze** (`cull_eye_closed`, `cull_smile`, `cull_gaze_away`) -replace the eye-openness proxy where every frame of a group with a face -carries `cull_face_state_version` (all or nothing): the blink penalty is then -the model's `eyeBlink`, judged at `reason_eye_closed_threshold` / +**Eye state, smile and gaze** (`cull_eye_closed`, `cull_smile`, `cull_eye_squint`, +`cull_gaze_off_lens`) replace the eye-openness proxy where every frame of a group +with a face carries the current `cull_face_state_version` (all or nothing; an older +version counts as not analysed, and `/index/unprocessed` queues the photo again): the +blink penalty is then the model's `eyeBlink` of the more closed eye, judged at `reason_eye_closed_threshold` / `reject_eye_closed_threshold` (plus the relative rules) instead of the proxy's thresholds, and a face it could not judge is never a blink. Reasons: `eyes_closed`, `eyes_half_closed` (a label, never a defect), @@ -627,7 +634,7 @@ against the group. | `summary` | object | `group_count`, `pick_count`, `alternate_count`, `reject_candidate_count`, `near_duplicate_group_count`, `intentional_set_group_count`, `intentional_set_photo_count`, `culling_preset`, `unindexed_count` | | `groups[].winner_photo_id` | string or null | The best-ranked frame that is not a reject candidate. **`null`** when every frame is one: a blurred single, or a burst blurred throughout | | `groups[].alternate_photo_ids`, `groups[].reject_candidate_photo_ids` | string[] | The rest of the group | -| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`gaze_away` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | +| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`eye_squint`/`gaze_off_lens` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | | `warnings` | string[] | Every caveat for the run, most fundamental first: unanalyzed photos skipped, no embeddings, AI signals whose prompt vectors could not be prepared (one message naming every lost signal and the fix), photos with faces prepared before eye focus or eye state were measured (one message when "Prepare now" fixes both), photos whose eyes the face state model could not analyse because it is not downloaded (with that fix), photos whose moment was not judged. Empty when there is nothing to say | | `warning` | string or null | The same messages joined, for clients older than `warnings` | diff --git a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md index 0a42a8c7..61d16842 100644 --- a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md +++ b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md @@ -1185,17 +1185,20 @@ read "closed" on 767 of 769 frames. portrait (`tests/face_state_goldens.rs`): landmarks within ~1% of the interocular distance, blink and smile within 0.01; the closed-lid path exactly, by moving MediaPipe's own upper-lid landmarks onto the lower ones. -- **Per face**: `eye_closed` (mean `eyeBlink`, only the near eye from 35° of - head turn), `eye_state` open/half/closed/unknown (unknown below 0.5 presence, - beyond 60° of yaw, or with the eyes under 24 px apart), smile, gaze, yaw. +- **Per face**: `eye_closed` (`eyeBlink` of the more closed eye, only the near + eye from 35° of head turn), `eye_state` open/half/closed/unknown (unknown below + 0.5 presence, beyond 60° of yaw, or with the eyes under 24 px apart), smile, + squint, `gaze_off_lens` in degrees, head pose (yaw, pitch, roll) and all 52 + blendshapes, so that a later signal need not decode every original again. Faces under 64 px are not run — the model card warns against rescaling them. Prominent faces only (the aggregation's gate), at most 12, on the full-size original from the focus step when there is one. - **Per photo**: the most closed prominent eyes, the weakest smile, the - furthest gaze, and `cull_face_state_version`. + strongest squint on a face not smiling, the furthest gaze among open eyes, and + `cull_face_state_version`, whose value counts: an older one is analysed again. - **Ranking**: where the group has the version throughout, the model's reading becomes blink penalty and eye openness, judged at its own thresholds (closed - ≥ 0.6, label ≥ 0.5; sports 0.8/0.7, street 0.85/0.75, portrait 0.55/0.45); + > 0.55, label > 0.5; sports 0.68/0.6, street 0.72/0.62, portrait 0.5/0.45); `eyes_closed`, `eyes_half_closed` (label only), `looking_away` (portrait), `best_smile`; a smile of 0.6 or more with the eyes shut is a laugh; smile weight 0.15 portrait, 0.10 event. An unknown face is never a blink. @@ -1207,9 +1210,46 @@ read "closed" on 767 of 769 frames. There is no photograph with closed eyes under a free license in the repository, so the closed path is pinned synthetically (lids moved shut: -0.81/0.76) and the open path on the reference portrait (0.04/0.03). The -thresholds sit between those with room on both sides, and the relative rules -still apply on top, but **they are hypotheses** until a hand-culled shoot with -real blinks has been scored with `cull_eval --ablate` — the same caveat as every -other threshold here. +0.81/0.76) and the open path on the reference portrait (0.04/0.03). + +Those two points were a poor guide. Measured with `examples/face_state_probe.rs` +on portraits from Wikimedia Commons (not committed; 285 usable faces to choose the +thresholds, judged by eye, then 292 from a disjoint sample to check them): + +| Reading | On the first sample | On the second sample | +|---|---|---| +| Eyes shut or lowered, caught by the first rule (mean ≥ 0.6) | 6 of 22 | about 8 of 24 | +| The same, caught by the more closed eye > 0.55 | 19 of 22 | about 17 of 24 | +| Faces with the eyes open and a reading > 0.5 | 0 of 213 | 1 of 178 portraits | +| Labelled `looking_away` in portraits, old (`eyeLook` > 0.5) → new (> 25°) | | 25–31% → 10–12% | +| The same in Commons' *People looking away* | | 53% → 67% | + +- **`eyeBlink` tops out low.** Eyes shut read 0.54–0.78 on the more closed eye and + 0.41–0.72 as a mean. No open eye reached 0.35. +- **`eyeLook*` is the eye inside the head.** People turned 25–30° and looking + straight at the camera read 0.85–0.99, and a head turned 39° with the eyes + following it 0.14, so the old reading had it backwards. Added to the head's turn + at 28° per unit (where the two cancel for people looking back at the lens) it + separates well: portraits a median 9°, people looking away 31°. Up and down is + weaker: `eyeLookUp` barely moves when a lowered head looks up at the lens, so + selfies from above read about 20°. That is why the label sits at 25°. Shut eyes + abstain: their `eyeLook*` is noise, 31° on one such portrait. +- **The head pose** is a frame of bone landmarks, within about 1° of a full + Procrustes fit to MediaPipe's canonical face model on yaw and roll and 2° on pitch. +- **`eyeSquint` runs high for everyone** (median 0.43) and rises on a smile, so it + counts only from 0.78 on a face smiling under 0.3. It fired on no face of the + second sample; it adds little beyond the lids, and no false alarms. +- **Several coefficients do not move on photographs at all**: `cheekSquint`, + `noseSneer`, `cheekPuff` and `jawForward` stay at 0.00 even at the 99th percentile, + and `mouthFrown` and `mouthFunnel` reach about 0.1. Don't build on them: a genuine + smile cannot be read from `cheekSquint`. `jawOpen`, `mouthPucker`, `browDown` and + `mouthPress` do move. + +What this does not solve: a slit narrowed against the sun overlaps eyes shut (0.5 +to 0.65), and neither `eyeSquint` nor the lid gap in the landmarks separates them; +sunglasses read as closed (0.47–0.74); a head lying on its side reads open with the +eyes shut. The relative rules catch the first two inside a burst, since every frame +reads alike there. And portraits from Commons are not a shoot, so **the thresholds +stay hypotheses** until a hand-culled shoot with real blinks has been scored with +`cull_eval --ablate` — the same caveat as every other threshold here. diff --git a/docs/wiki/Help-Cull-Photos.md b/docs/wiki/Help-Cull-Photos.md index 76d14c6a..f41bfa52 100644 --- a/docs/wiki/Help-Cull-Photos.md +++ b/docs/wiki/Help-Cull-Photos.md @@ -209,17 +209,24 @@ blink. the frame. - **One blink spoils the group.** The most closed eyes among the people that matter in the frame decide, so a group shot with one person blinking is not - "eyes open". Small faces in the background do not count. + "eyes open". One shut eye counts as well, even when the other is open. Small + faces in the background do not count. - **Compared with the burst.** A frame becomes a reject candidate for closed eyes only when another frame of the same burst has them open. Half-closed eyes (a squint into the sun, a mid-blink) are named in the explanation but - never make a frame a reject candidate. + never make a frame a reject candidate. The model cannot always tell eyes + narrowed to a slit from eyes shut, so a hard squint can read as closed, and + sunglasses can too. Comparing with the burst keeps either from costing a frame + when the whole burst looks the same. - **A laugh is not a blink.** A broad smile with the eyes shut counts as a laugh, like a kiss already did. - **The smile counts in Portrait and Event.** Between otherwise equal frames, the one where everyone smiles more wins, and its explanation says - "best smile in group". **Portrait** also notes when someone is looking away - from the camera. + "best smile in group". +- **Looking away means from the lens.** **Portrait** notes when someone looks + away from the camera. Head and eyes count together: a head turned to the side + with the eyes back on the camera is looking at it, and a head turned with the + eyes following it is not. It is a note, never a reason to reject. Like focus, the face is read on the full-size original where there is one. The model is part of **Download AI models** (about 7 MB). Without it culling @@ -280,8 +287,8 @@ For each photo, the backend stores culling-related fields such as: `cull_blink_penalty`, `cull_occlusion`, and `cull_eye_focus` / `cull_face_focus` (focus on the eyes and on the face, measured on the full-size original; absent when the faces were too small to judge), and - `cull_eye_state` / `cull_eye_closed`, `cull_smile`, `cull_gaze_away` (from - the face model; absent when no face could be judged) + `cull_eye_state` / `cull_eye_closed`, `cull_smile`, `cull_eye_squint`, + `cull_gaze_off_lens` (from the face model; absent when no face could be judged) - aesthetic scores: `cull_aesthetic` (a contrast-and-colour heuristic) and the photo's **Aesthetic score** from the AestheticSigLIP model, stored at indexing time (`aesthetic_score`) and read by culling as `cull_aesthetic_model` diff --git a/server-rs/crates/lrg-analysis/src/culling_config.rs b/server-rs/crates/lrg-analysis/src/culling_config.rs index b8543e08..d7958e83 100644 --- a/server-rs/crates/lrg-analysis/src/culling_config.rs +++ b/server-rs/crates/lrg-analysis/src/culling_config.rs @@ -302,13 +302,13 @@ pub struct RankingConfig { /// A pick within this of the group's best eye focus is reported as having /// the sharpest eyes of the group (`eyes_sharpest_in_group`). pub reason_eye_focus_delta: f64, - /// With the face state model (`cull_eye_closed`, MediaPipe's `eyeBlink`), - /// above this a frame is labelled `eyes_closed`; above - /// [`Self::reject_eye_closed_threshold`] it is a blink defect. They replace - /// `reason_possible_blink_threshold`/`reject_blink_penalty_threshold`, + /// With the face state model (`cull_eye_closed`, MediaPipe's `eyeBlink` of + /// the more closed eye), above this a frame is labelled `eyes_closed`; + /// above [`Self::reject_eye_closed_threshold`] it is a blink defect. They + /// replace `reason_possible_blink_threshold`/`reject_blink_penalty_threshold`, /// which were tuned against a proxy that read "closed" on almost every - /// small face; the relative rules still apply on top. On the reference - /// portrait an open eye reads 0.03–0.04 and shut lids 0.76–0.81. + /// small face; the relative rules still apply on top. On real photographs + /// judged by hand, eyes shut read 0.54–0.78 and no open eye reached 0.35. /// **Hypotheses until a hand-culled fixture says otherwise.** pub reason_eye_closed_threshold: f64, pub reject_eye_closed_threshold: f64, @@ -316,6 +316,10 @@ pub struct RankingConfig { /// squint, a mid-blink — and the frame is labelled `eyes_half_closed`. /// Never a defect. 0 switches the label off. pub reason_eyes_half_closed_threshold: f64, + /// `cull_eye_squint` at or above this gets the same label, for eyes + /// narrowed against the sun with the lids reading open. Half of all faces + /// read above 0.43, so it has to sit far up. + pub reason_eye_squint_threshold: f64, /// A smile at least this broad with the eyes shut is a laugh, not a blink /// (per face, next to the scene-level CLIP gate). 0 switches it off. pub laugh_smile_threshold: f64, @@ -326,9 +330,9 @@ pub struct RankingConfig { /// A pick within this of the group's broadest smile is reported as having /// the best smile (`best_smile`). pub reason_smile_delta: f64, - /// Above this `eyeLook*` a frame is labelled `looking_away`. 0 switches the - /// label off; it is never a defect. - pub reason_gaze_away_threshold: f64, + /// Above this many degrees from the lens (`cull_gaze_off_lens`) a frame is + /// labelled `looking_away`. 0 switches the label off; it is never a defect. + pub reason_gaze_off_lens_degrees: f64, } /// Which deliberately-captured multi-frame set a group is, if any. @@ -462,7 +466,7 @@ impl CullingConfig { cfg.ranking.smile_weight = 0.0; cfg.ranking.laugh_smile_threshold = 0.0; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; - cfg.ranking.reason_gaze_away_threshold = 0.0; + cfg.ranking.reason_gaze_off_lens_degrees = 0.0; cfg } } @@ -534,12 +538,13 @@ const BASE: CullingConfig = CullingConfig { relative_focus_margin: 0.08, reason_eye_focus_delta: 0.02, reason_eye_closed_threshold: 0.5, - reject_eye_closed_threshold: 0.6, + reject_eye_closed_threshold: 0.55, reason_eyes_half_closed_threshold: 0.35, + reason_eye_squint_threshold: 0.78, laugh_smile_threshold: 0.6, smile_weight: 0.0, reason_smile_delta: 0.05, - reason_gaze_away_threshold: 0.0, + reason_gaze_off_lens_degrees: 0.0, }, sets: SetDetectionConfig { enabled: true, @@ -593,9 +598,9 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { // smile per face is worth scoring, and eye contact is a question // worth answering. cfg.ranking.reason_eye_closed_threshold = 0.45; - cfg.ranking.reject_eye_closed_threshold = 0.55; + cfg.ranking.reject_eye_closed_threshold = 0.5; cfg.ranking.smile_weight = 0.15; - cfg.ranking.reason_gaze_away_threshold = 0.5; + cfg.ranking.reason_gaze_off_lens_degrees = 25.0; } "street" => { cfg.ranking.face_group_weight_technical = 0.70; @@ -625,8 +630,8 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reason_face_focus_threshold = 0.0; cfg.ranking.reject_face_focus_threshold = 0.0; // Closed eyes are often the moment in street work. - cfg.ranking.reason_eye_closed_threshold = 0.75; - cfg.ranking.reject_eye_closed_threshold = 0.85; + cfg.ranking.reason_eye_closed_threshold = 0.62; + cfg.ranking.reject_eye_closed_threshold = 0.72; } "event" => { cfg.grouping.time_window_default_seconds = 2; @@ -698,8 +703,8 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reason_face_focus_threshold = 0.32; cfg.ranking.reject_face_focus_threshold = 0.25; // Athletes squint and grimace; only shut eyes count. - cfg.ranking.reason_eye_closed_threshold = 0.7; - cfg.ranking.reject_eye_closed_threshold = 0.8; + cfg.ranking.reason_eye_closed_threshold = 0.6; + cfg.ranking.reject_eye_closed_threshold = 0.68; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; } _ => {} diff --git a/server-rs/crates/lrg-analysis/src/face_aggregate.rs b/server-rs/crates/lrg-analysis/src/face_aggregate.rs index 6f3e194f..b5b2a567 100644 --- a/server-rs/crates/lrg-analysis/src/face_aggregate.rs +++ b/server-rs/crates/lrg-analysis/src/face_aggregate.rs @@ -41,8 +41,11 @@ pub struct FaceMetricsInput { pub eye_closed: Option, /// `mouthSmile`, `0..1`; `None` like `eye_closed`. pub smile: Option, - /// The strongest `eyeLook*` coefficient, `0..1`; 0 is looking into the lens. - pub gaze_away: Option, + /// `eyeSquint`, `0..1`; `None` like `eye_closed`. + pub eye_squint: Option, + /// How far from the lens the eyes point, in degrees, head turn included; + /// `None` like `eye_closed`. + pub gaze_off_lens: Option, } #[derive(Debug, Clone, PartialEq)] @@ -69,8 +72,11 @@ pub struct AggregatedFaceMetrics { /// The weakest smile among the prominent faces: a group only smiles when /// everyone in it does. pub cull_smile: Option, - /// The prominent face looking furthest away from the lens. - pub cull_gaze_away: Option, + /// The strongest squint among the prominent faces that are not smiling; a + /// broad smile narrows the eyes too, and that is not a squint. + pub cull_eye_squint: Option, + /// The prominent face looking furthest from the lens, in degrees. + pub cull_gaze_off_lens: Option, } impl AggregatedFaceMetrics { @@ -90,7 +96,8 @@ impl AggregatedFaceMetrics { cull_eye_closed: None, cull_eye_state: None, cull_smile: None, - cull_gaze_away: None, + cull_eye_squint: None, + cull_gaze_off_lens: None, } } } @@ -198,7 +205,24 @@ pub fn aggregate_face_culling_metrics( }; let eye_closed_agg = measured(|f| f.eye_closed).reduce(f64::max); let smile_agg = measured(|f| f.smile).reduce(f64::min); - let gaze_away_agg = measured(|f| f.gaze_away).reduce(f64::max); + let eye_squint_agg = prominent + .iter() + .map(|&i| &faces[i]) + .filter(|f| f.smile.is_some_and(|s| s < cfg.face_state_squint_max_smile)) + .filter_map(|f| f.eye_squint.map(unit)) + .reduce(f64::max); + // Degrees, so not through `measured`'s clamp to `0..1`. Shut eyes look + // nowhere: their `eyeLook*` is noise (31° on a portrait with the eyes shut), + // so such a face abstains. + let gaze_off_lens_agg = prominent + .iter() + .map(|&i| &faces[i]) + .filter(|f| { + f.eye_closed + .is_none_or(|c| c < cfg.face_state_closed_threshold) + }) + .filter_map(|f| f.gaze_off_lens) + .reduce(f64::max); let eye_state_agg = eye_closed_agg.map(|v| { if v >= cfg.face_state_closed_threshold { "closed" @@ -236,7 +260,8 @@ pub fn aggregate_face_culling_metrics( cull_eye_closed: eye_closed_agg.map(round4), cull_eye_state: eye_state_agg, cull_smile: smile_agg.map(round4), - cull_gaze_away: gaze_away_agg.map(round4), + cull_eye_squint: eye_squint_agg.map(round4), + cull_gaze_off_lens: gaze_off_lens_agg.map(round4), } } @@ -270,7 +295,8 @@ mod tests { face_focus: Some(0.55), eye_closed: Some(0.03), smile: Some(0.9), - gaze_away: Some(0.1), + eye_squint: Some(0.4), + gaze_off_lens: Some(4.0), }]; let m = aggregate_face_culling_metrics(&faces, &fm()); assert_eq!(m.cull_face_count, 1); @@ -413,14 +439,14 @@ mod tests { area_ratio: 0.20, eye_closed: Some(0.04), smile: Some(0.9), - gaze_away: Some(0.1), + gaze_off_lens: Some(32.5), ..Default::default() }, FaceMetricsInput { area_ratio: 0.18, eye_closed: Some(0.81), smile: Some(0.6), - gaze_away: Some(0.2), + gaze_off_lens: Some(40.0), ..Default::default() }, // Turned away: the model could not judge the eyes. @@ -433,7 +459,7 @@ mod tests { area_ratio: 0.01, eye_closed: Some(0.99), smile: Some(0.0), - gaze_away: Some(0.9), + gaze_off_lens: Some(70.0), ..Default::default() }, ]; @@ -441,7 +467,7 @@ mod tests { assert_eq!(m.cull_eye_closed, Some(0.81)); assert_eq!(m.cull_eye_state, Some("closed")); assert_eq!(m.cull_smile, Some(0.6)); - assert_eq!(m.cull_gaze_away, Some(0.2)); + assert_eq!(m.cull_gaze_off_lens, Some(32.5)); let open = aggregate_face_culling_metrics(&faces[..1], &fm()); assert_eq!(open.cull_eye_state, Some("open")); @@ -452,6 +478,42 @@ mod tests { ); } + #[test] + fn eyes_that_are_shut_do_not_look_away() { + let face = |closed, off| FaceMetricsInput { + area_ratio: 0.2, + eye_closed: Some(closed), + gaze_off_lens: Some(off), + ..Default::default() + }; + let shut = aggregate_face_culling_metrics(&[face(0.66, 31.0)], &fm()); + assert_eq!(shut.cull_gaze_off_lens, None); + let group = [face(0.66, 31.0), face(0.05, 12.0)]; + let group = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(group.cull_gaze_off_lens, Some(12.0)); + } + + /// `eyeSquint` runs high on a broad smile, so only an unsmiling face's + /// counts as a squint; a face the smile was not read on says nothing. + #[test] + fn a_squint_only_counts_on_a_face_that_is_not_smiling() { + let face = |squint, smile: Option| FaceMetricsInput { + area_ratio: 0.2, + eye_squint: Some(squint), + smile, + ..Default::default() + }; + let smiling = aggregate_face_culling_metrics(&[face(0.9, Some(0.8))], &fm()); + assert_eq!(smiling.cull_eye_squint, None); + let squinting = aggregate_face_culling_metrics(&[face(0.84, Some(0.1))], &fm()); + assert_eq!(squinting.cull_eye_squint, Some(0.84)); + let group = [face(0.9, Some(0.8)), face(0.6, Some(0.1))]; + let group = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(group.cull_eye_squint, Some(0.6)); + let unread = aggregate_face_culling_metrics(&[face(0.9, None)], &fm()); + assert_eq!(unread.cull_eye_squint, None); + } + #[test] fn eye_focus_is_the_least_sharp_measured_prominent_face() { let faces = vec![ diff --git a/server-rs/crates/lrg-analysis/src/grouping.rs b/server-rs/crates/lrg-analysis/src/grouping.rs index 4298f4f2..02d7ddd9 100644 --- a/server-rs/crates/lrg-analysis/src/grouping.rs +++ b/server-rs/crates/lrg-analysis/src/grouping.rs @@ -248,10 +248,12 @@ struct Scored<'a> { eyes_soft: bool, face_soft: bool, /// From the face state model (`cull_eye_closed`, `cull_smile`, - /// `cull_gaze_away`); `None` when no prominent face could be judged. + /// `cull_eye_squint`, `cull_gaze_off_lens`); `None` when no prominent face + /// could be judged. The last is in degrees, the others `0..1`. cull_eye_closed: Option, cull_smile: Option, - cull_gaze_away: Option, + cull_eye_squint: Option, + cull_gaze_off_lens: Option, /// The photo went through the face state pass (`cull_face_state_version` /// is stored), whether or not its faces could be judged. state_measured: bool, @@ -303,8 +305,19 @@ pub const FOCUS_VERSION: i64 = 1; pub const FACE_STATE_VERSION_KEY: &str = "cull_face_state_version"; /// The value stored under [`FACE_STATE_VERSION_KEY`]. Bump it when stored -/// readings become incomparable with new ones. -pub const FACE_STATE_VERSION: i64 = 1; +/// readings become incomparable with new ones; `/index/unprocessed` then has +/// those photos analysed again, and until then they rank like photos without +/// the model. +/// +/// 2: `cull_eye_closed` is the more closed eye instead of the mean of both, and +/// `cull_gaze_off_lens` replaces `cull_gaze_away`. +pub const FACE_STATE_VERSION: i64 = 2; + +/// Whether a photo's face state was read by this version of the analysis. The +/// key's value counts, not its presence, or a bump would change nothing. +pub fn face_state_is_current(metadata: &serde_json::Map) -> bool { + metadata.get(FACE_STATE_VERSION_KEY).and_then(Value::as_i64) == Some(FACE_STATE_VERSION) +} /// How much eye focus one point of the 0–10 in the explanation is worth. /// @@ -536,12 +549,14 @@ fn rank_group_records( let cull_face_focus = m.get("cull_face_focus").and_then(Value::as_f64).map(unit); let focus_measured = m.contains_key(FOCUS_VERSION_KEY); let read = |key: &str| m.get(key).and_then(Value::as_f64).map(unit); - let (cull_eye_closed, cull_smile, cull_gaze_away) = ( + let (cull_eye_closed, cull_smile, cull_eye_squint) = ( read("cull_eye_closed"), read("cull_smile"), - read("cull_gaze_away"), + read("cull_eye_squint"), ); - let state_measured = m.contains_key(FACE_STATE_VERSION_KEY); + // Degrees, so not through `read`'s clamp to `0..1`. + let cull_gaze_off_lens = m.get("cull_gaze_off_lens").and_then(Value::as_f64); + let state_measured = face_state_is_current(m); // Injected by the API layer only when the preset asks the gate // question, so an absent score never lifts a blink. let intent_threshold = cfg.ranking.eyes_closed_intent_threshold; @@ -575,7 +590,8 @@ fn rank_group_records( face_soft: false, cull_eye_closed, cull_smile, - cull_gaze_away, + cull_eye_squint, + cull_gaze_off_lens, state_measured, half_closed: false, hard_defect: false, @@ -862,9 +878,12 @@ fn rank_group_records( && !item.blink && !item.eyes_closed_intended && rk.reason_eyes_half_closed_threshold > 0.0 - && item + && (item .cull_eye_closed - .is_some_and(|v| v >= rk.reason_eyes_half_closed_threshold && v < reason_blink); + .is_some_and(|v| v >= rk.reason_eyes_half_closed_threshold && v < reason_blink) + || item + .cull_eye_squint + .is_some_and(|v| v >= rk.reason_eye_squint_threshold)); item.occluded = group_has_faces && item.cull_face_count > 0 && item.cull_occlusion > rk.reason_occlusion_threshold @@ -1053,9 +1072,13 @@ fn rank_group_records( // With the face state model, a frame whose eyes it could not judge // has won nothing on them. let eyes_judged = !state_active || item.cull_eye_closed.is_some(); + // The most open eyes of a group can still be shut, and a single is + // always its own best: only the model can say so. + let eyes_open = !state_active || item.cull_blink_penalty <= reason_blink; if item.cull_eye_openness >= (max_eye_openness - rk.reason_eyes_open_delta).max(0.0) && is_winner && eyes_judged + && eyes_open { reason_codes.push("eyes_open_best".to_string()); } else if item.eyes_closed_intended && item.cull_blink_penalty > reason_blink { @@ -1071,10 +1094,10 @@ fn rank_group_records( reason_codes.push("eyes_half_closed".to_string()); } if state_active - && rk.reason_gaze_away_threshold > 0.0 + && rk.reason_gaze_off_lens_degrees > 0.0 && item - .cull_gaze_away - .is_some_and(|g| g > rk.reason_gaze_away_threshold) + .cull_gaze_off_lens + .is_some_and(|g| g > rk.reason_gaze_off_lens_degrees) { reason_codes.push("looking_away".to_string()); } @@ -1229,7 +1252,8 @@ fn rank_group_records( for (key, value) in [ ("eye_closed", item.cull_eye_closed), ("smile", item.cull_smile), - ("gaze_away", item.cull_gaze_away), + ("eye_squint", item.cull_eye_squint), + ("gaze_off_lens", item.cull_gaze_off_lens), ] { if let Some(v) = value { metrics.insert(key.into(), Value::from(round4(v))); @@ -2568,8 +2592,10 @@ mod tests { r.metadata .insert("cull_blink_penalty".into(), Value::from(0.9)); if measured { - r.metadata - .insert(FACE_STATE_VERSION_KEY.into(), Value::from(1)); + r.metadata.insert( + FACE_STATE_VERSION_KEY.into(), + Value::from(FACE_STATE_VERSION), + ); } if let Some(v) = closed { r.metadata.insert("cull_eye_closed".into(), Value::from(v)); @@ -2659,6 +2685,94 @@ mod tests { assert_eq!(squint.metrics["clear_defect"], false); } + #[test] + fn shut_eyes_are_never_the_best_open_eyes() { + let groups = cull( + vec![state_frame("shut", 1000.0, Some(0.66), Some(0.2), true)], + "portrait", + ); + let shut = photo(&groups[0], "shut"); + assert!( + !shut.reason_codes.contains(&"eyes_open_best".to_string()), + "{:?}", + shut.reason_codes + ); + let groups = cull( + vec![state_frame("open", 1000.0, Some(0.05), Some(0.2), true)], + "portrait", + ); + assert!(photo(&groups[0], "open") + .reason_codes + .contains(&"eyes_open_best".to_string())); + } + + /// Eyes narrowed against the sun with the lids reading open: the label, and + /// still no defect. + #[test] + fn a_squint_with_the_lids_reading_open_is_labelled() { + let mut squint = state_frame("squint", 1000.5, Some(0.25), Some(0.1), true); + squint + .metadata + .insert("cull_eye_squint".into(), Value::from(0.82)); + let groups = cull( + vec![ + state_frame("open", 1000.0, Some(0.04), Some(0.1), true), + squint, + ], + "event", + ); + let squint = photo(&groups[0], "squint"); + assert!(squint + .reason_codes + .contains(&"eyes_half_closed".to_string())); + assert_eq!(squint.metrics["eye_squint"], 0.82); + assert_eq!(squint.metrics["clear_defect"], false); + let open = photo(&groups[0], "open"); + assert!(!open.reason_codes.contains(&"eyes_half_closed".to_string())); + } + + /// Degrees from the lens, not clamped to `0..1`, and only where the preset + /// asks. + #[test] + fn looking_away_is_judged_in_degrees_from_the_lens() { + let frame = |id, t, off: f64| { + let mut r = state_frame(id, t, Some(0.04), Some(0.5), true); + r.metadata + .insert("cull_gaze_off_lens".into(), Value::from(off)); + r + }; + let records = || vec![frame("at_lens", 1000.0, 6.0), frame("away", 1000.5, 38.0)]; + let groups = cull(records(), "portrait"); + let away = photo(&groups[0], "away"); + assert!(away.reason_codes.contains(&"looking_away".to_string())); + assert_eq!(away.metrics["gaze_off_lens"], 38.0); + assert_eq!(away.metrics["clear_defect"], false); + let at_lens = photo(&groups[0], "at_lens"); + assert!(!at_lens.reason_codes.contains(&"looking_away".to_string())); + + let groups = cull(records(), "event"); + let away = photo(&groups[0], "away"); + assert!(!away.reason_codes.contains(&"looking_away".to_string())); + } + + /// The version's value counts: a reading from before a bump means something + /// else, and must not be ranked as if it were today's. + #[test] + fn a_reading_from_an_older_analysis_counts_as_unanalysed() { + let mut stale = state_frame("stale", 1000.0, Some(0.04), Some(0.5), true); + stale.metadata.insert( + FACE_STATE_VERSION_KEY.into(), + Value::from(FACE_STATE_VERSION - 1), + ); + assert!(!face_state_is_current(&stale.metadata)); + let current = state_frame("current", 1000.5, Some(0.04), Some(0.5), true); + assert!(face_state_is_current(¤t.metadata)); + let groups = cull(vec![stale, current], "event"); + for p in &groups[0].photos { + assert!(!p.metrics.contains_key("eye_closed"), "{}", p.photo_id); + } + } + /// A burst with one frame from before the model ran keeps the proxy for /// all of them, rather than judging two frames on two instruments. #[test] diff --git a/server-rs/crates/lrg-api/src/routes/group_similar.rs b/server-rs/crates/lrg-api/src/routes/group_similar.rs index 1597b36a..85ed163c 100644 --- a/server-rs/crates/lrg-api/src/routes/group_similar.rs +++ b/server-rs/crates/lrg-api/src/routes/group_similar.rs @@ -647,9 +647,7 @@ async fn compute_groups( .and_then(Value::as_i64) .unwrap_or(0) > 0 - && !r - .metadata - .contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY) + && !lrg_analysis::grouping::face_state_is_current(&r.metadata) }) .count(); diff --git a/server-rs/crates/lrg-api/src/routes/index.rs b/server-rs/crates/lrg-api/src/routes/index.rs index 71db0152..5b300487 100644 --- a/server-rs/crates/lrg-api/src/routes/index.rs +++ b/server-rs/crates/lrg-api/src/routes/index.rs @@ -283,7 +283,7 @@ async fn check_unprocessed( .and_then(Value::as_i64) .unwrap_or(0) > 0 - && !m.contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY); + && !lrg_analysis::grouping::face_state_is_current(&m); // A photo is done for `species` only if it was checked *and* // checked by the head currently on disk. Widening the taxa // whitelist changes what BioCLIP can answer, so results from an diff --git a/server-rs/crates/lrg-api/src/routes/index_upload.rs b/server-rs/crates/lrg-api/src/routes/index_upload.rs index d2523d81..d0565688 100644 --- a/server-rs/crates/lrg-api/src/routes/index_upload.rs +++ b/server-rs/crates/lrg-api/src/routes/index_upload.rs @@ -2034,7 +2034,7 @@ async fn finish_one( // prepared before it was downloaded is analysed on the next cull pass. let state_missing = options.face_pass == FacePass::QualityOnly && state.face_state.is_cached() - && !main_metadata.contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY); + && !lrg_analysis::grouping::face_state_is_current(&main_metadata); if options.regenerate_metadata || !already_checked || focus_missing || state_missing { // Decoded here rather than carried from phase 1: full RGB is by far // the largest per-photo allocation, and holding one per photo in @@ -2152,7 +2152,8 @@ async fn finish_one( face_focus: f.face_focus, eye_closed: judged(f).map(|s| s.eye_closed), smile: judged(f).map(|s| s.smile), - gaze_away: judged(f).map(|s| s.gaze_away), + eye_squint: judged(f).map(|s| s.eye_squint), + gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), }) .collect(); apply_face_aggregate(&mut main_metadata, &inputs, state_ran); @@ -2274,8 +2275,25 @@ async fn finish_one( ); m.insert("face_eye_closed".into(), json!(st.map(|s| s.eye_closed))); m.insert("face_smile".into(), json!(st.map(|s| s.smile))); - m.insert("face_gaze_away".into(), json!(st.map(|s| s.gaze_away))); - m.insert("face_yaw".into(), json!(st.map(|s| s.yaw_degrees))); + m.insert("face_eye_squint".into(), json!(st.map(|s| s.eye_squint))); + m.insert( + "face_gaze_off_lens".into(), + json!(st.map(|s| s.gaze_off_lens)), + ); + m.insert("face_yaw".into(), json!(st.map(|s| s.pose.yaw))); + m.insert("face_pitch".into(), json!(st.map(|s| s.pose.pitch))); + m.insert("face_roll".into(), json!(st.map(|s| s.pose.roll))); + // All 52, in `BLENDSHAPE_NAMES` order, to three + // places: a later signal can then read them + // without decoding every original again. + m.insert( + "face_blendshapes".into(), + json!(st.map(|s| s + .blendshapes + .iter() + .map(|v| (*v as f64 * 1000.0).round() / 1000.0) + .collect::>())), + ); m.insert("face_model".into(), json!(face_model)); StoreRecord { id: format!("{photo_id}_{i}"), @@ -2303,7 +2321,8 @@ async fn finish_one( face_focus: f.face_focus, eye_closed: judged(f).map(|s| s.eye_closed), smile: judged(f).map(|s| s.smile), - gaze_away: judged(f).map(|s| s.gaze_away), + eye_squint: judged(f).map(|s| s.eye_squint), + gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), }) .collect(); apply_face_aggregate(&mut main_metadata, &inputs, state_ran); @@ -2716,7 +2735,8 @@ fn apply_face_aggregate( ("cull_face_focus", agg.cull_face_focus), ("cull_eye_closed", agg.cull_eye_closed), ("cull_smile", agg.cull_smile), - ("cull_gaze_away", agg.cull_gaze_away), + ("cull_eye_squint", agg.cull_eye_squint), + ("cull_gaze_off_lens", agg.cull_gaze_off_lens), ] { match value { Some(v) => { diff --git a/server-rs/crates/lrg-api/tests/contract.rs b/server-rs/crates/lrg-api/tests/contract.rs index 12f700af..a6d386ff 100644 --- a/server-rs/crates/lrg-api/tests/contract.rs +++ b/server-rs/crates/lrg-api/tests/contract.rs @@ -3975,15 +3975,15 @@ async fn the_face_state_model_is_its_own_downloadable_family() { /// would change nothing. #[tokio::test] async fn check_unprocessed_asks_for_face_state_only_with_the_model() { - let row = |id: &str, faces: i64, state: bool| { + let row = |id: &str, faces: i64, state: Option| { let mut m = serde_json::Map::new(); m.insert("cull_phash".into(), serde_json::json!("abcd0123abcd0123")); m.insert("cull_faces_present".into(), serde_json::json!(faces > 0)); m.insert("cull_face_count".into(), serde_json::json!(faces)); m.insert("capture_time".into(), serde_json::json!(1_700_000_000.0)); m.insert("cull_focus_version".into(), serde_json::json!(1)); - if state { - m.insert("cull_face_state_version".into(), serde_json::json!(1)); + if let Some(version) = state { + m.insert("cull_face_state_version".into(), serde_json::json!(version)); } lrg_store::StoreRecord { id: id.into(), @@ -3992,9 +3992,18 @@ async fn check_unprocessed_asks_for_face_state_only_with_the_model() { } }; let rows = [ - row("faces_without_state", 2, false), - row("faces_with_state", 1, true), - row("faceless", 0, false), + row("faces_without_state", 2, None), + row( + "faces_with_state", + 1, + Some(lrg_analysis::grouping::FACE_STATE_VERSION), + ), + row( + "faces_with_older_state", + 1, + Some(lrg_analysis::grouping::FACE_STATE_VERSION - 1), + ), + row("faceless", 0, None), ]; let ask = |app: axum::Router, db: String| async move { let response = app @@ -4003,7 +4012,12 @@ async fn check_unprocessed_asks_for_face_state_only_with_the_model() { .header("content-type", "application/json") .body(Body::from( serde_json::json!({ - "photo_ids": ["faces_without_state", "faces_with_state", "faceless"], + "photo_ids": [ + "faces_without_state", + "faces_with_state", + "faces_with_older_state", + "faceless", + ], "tasks": "cull", "regenerate_metadata": "false", "db_path": db, @@ -4052,7 +4066,7 @@ async fn check_unprocessed_asks_for_face_state_only_with_the_model() { .unwrap(); assert_eq!( ask(app(state.clone()), db).await, - serde_json::json!(["faces_without_state"]) + serde_json::json!(["faces_without_state", "faces_with_older_state"]) ); } @@ -4082,7 +4096,10 @@ async fn cull_names_a_real_blink_from_the_face_state() { m.insert("cull_blink_penalty".into(), serde_json::json!(0.9)); m.insert("cull_occlusion".into(), serde_json::json!(0.1)); m.insert("cull_focus_version".into(), serde_json::json!(1)); - m.insert("cull_face_state_version".into(), serde_json::json!(1)); + m.insert( + "cull_face_state_version".into(), + serde_json::json!(lrg_analysis::grouping::FACE_STATE_VERSION), + ); m.insert("cull_eye_closed".into(), serde_json::json!(closed)); m.insert("cull_smile".into(), serde_json::json!(0.4)); m.insert("aesthetic_score".into(), serde_json::json!(6.0)); diff --git a/server-rs/crates/lrg-imaging/src/cull_config.rs b/server-rs/crates/lrg-imaging/src/cull_config.rs index 4a4255bf..5ae7d58a 100644 --- a/server-rs/crates/lrg-imaging/src/cull_config.rs +++ b/server-rs/crates/lrg-imaging/src/cull_config.rs @@ -206,14 +206,25 @@ pub struct FaceMetricsConfig { /// Smallest interocular distance, in pixels of the image analysed, at /// which the eye state is judged. pub face_state_min_eye_px: f64, - /// `eyeBlink` at or above this is closed eyes. On the reference portrait - /// an open eye reads 0.03–0.04 and the same landmarks with the lids moved - /// shut 0.76–0.81. **A hypothesis until a hand-culled fixture says - /// otherwise.** + /// `eyeBlink` of the more closed eye at or above this is closed eyes. On + /// real photographs, judged by hand, eyes shut read 0.54–0.78 and no open + /// eye reached 0.35; a slit narrowed against the sun overlaps the shut + /// range, and no coefficient separates the two. **A hypothesis until a + /// hand-culled fixture says otherwise.** pub face_state_closed_threshold: f64, /// `eyeBlink` at or above this, but below the closed threshold, is half /// closed: a squint, a mid-blink, a sleepy look. pub face_state_half_threshold: f64, + /// How many degrees of turn inside the head one unit of `eyeLook*` is + /// worth, sideways and up or down. Sideways from people turned away and + /// looking back at the lens, where the eyes' reading cancels the head's + /// turn at about this; up and down from the portraits' pitch against + /// `eyeLookUp/Down`, the rougher of the two. + pub face_state_gaze_across_degrees: f64, + pub face_state_gaze_up_degrees: f64, + /// `eyeSquint` rises on a broad smile as well, so it only counts as a + /// squint on a face smiling less than this. + pub face_state_squint_max_smile: f64, } impl FaceMetricsConfig { @@ -250,8 +261,11 @@ impl FaceMetricsConfig { face_state_max_yaw_degrees: 60.0, face_state_near_eye_yaw_degrees: 35.0, face_state_min_eye_px: 24.0, - face_state_closed_threshold: 0.6, + face_state_closed_threshold: 0.55, face_state_half_threshold: 0.35, + face_state_gaze_across_degrees: 28.0, + face_state_gaze_up_degrees: 20.0, + face_state_squint_max_smile: 0.3, } } } diff --git a/server-rs/crates/lrg-ml/Cargo.toml b/server-rs/crates/lrg-ml/Cargo.toml index b31ba54d..d47da19e 100644 --- a/server-rs/crates/lrg-ml/Cargo.toml +++ b/server-rs/crates/lrg-ml/Cargo.toml @@ -26,5 +26,8 @@ ort = { version = "2.0.0-rc.13", features = ["coreml"] } [[example]] name = "bench_image" +[[example]] +name = "face_state_probe" + [dev-dependencies] tempfile.workspace = true diff --git a/server-rs/crates/lrg-ml/examples/face_state_probe.rs b/server-rs/crates/lrg-ml/examples/face_state_probe.rs new file mode 100644 index 00000000..d23c0706 --- /dev/null +++ b/server-rs/crates/lrg-ml/examples/face_state_probe.rs @@ -0,0 +1,89 @@ +//! Prints what the face state pipeline reads on the photos given, one JSON +//! object per face, for measuring a new signal or threshold against real +//! photographs before it goes into the ranking. +//! +//! ```bash +//! cargo run --release -p lrg-ml --example face_state_probe -- a.jpg b.jpg > faces.jsonl +//! ``` +//! +//! Needs the YuNet detector and the two face state models, found where the +//! server finds them (`LRG_FACE_MESH_ONNX` / `LRG_FACE_BLENDSHAPES_ONNX`, else +//! the models directory). Faces are detected at the size the file has; the +//! server detects on a copy of at most 2048 px and then analyses the original. + +use lrg_imaging::cull_config::FaceMetricsConfig; +use lrg_ml::face_state::{FaceGeometry, FaceStateModel, BLENDSHAPE_NAMES}; +use lrg_ml::faces::{FaceModel, FacePass}; +use serde_json::{json, Map, Value}; + +fn main() { + let cfg = FaceMetricsConfig::defaults(); + let faces = FaceModel::new(lrg_ml::model_paths::resolve_face()); + let state = FaceStateModel::new(lrg_ml::model_paths::resolve_face_state()); + assert!(faces.is_cached(), "face detection models not found"); + assert!(state.is_cached(), "face state models not found"); + + for path in std::env::args().skip(1) { + let img = match image::open(&path) { + Ok(img) => img.to_rgb8(), + Err(e) => { + eprintln!("{path}: {e}"); + continue; + } + }; + let (w, h) = (img.width() as usize, img.height() as usize); + let rgb = img.into_raw(); + let detected = match faces.detect_faces(&rgb, w, h, &cfg, FacePass::QualityOnly) { + Ok(detected) => detected, + Err(e) => { + eprintln!("{path}: {e}"); + continue; + } + }; + let geometry: Vec = detected + .iter() + .map(|f| FaceGeometry { + bbox: f.bbox.map(|v| v as f64), + kps: f.kps, + }) + .collect(); + let states = match state.analyze(&rgb, w, h, &geometry, &cfg) { + Ok(states) => states, + Err(e) => { + eprintln!("{path}: {e}"); + continue; + } + }; + for (i, (face, st)) in detected.iter().zip(states).enumerate() { + let Some(st) = st else { continue }; + let shapes: Map = BLENDSHAPE_NAMES + .iter() + .zip(st.blendshapes) + .map(|(name, v)| (name.to_string(), json!(v))) + .collect(); + println!( + "{}", + json!({ + "file": path, + "face": i, + "width": w, + "height": h, + "bbox": face.bbox, + "det_score": face.det_score, + "area_ratio": face.area_ratio, + "presence": st.presence, + "eye_state": st.eye_state.as_str(), + "eye_closed": st.eye_closed, + "eye_squint": st.eye_squint, + "smile": st.smile, + "gaze_off_lens": st.gaze_off_lens, + "yaw": st.pose.yaw, + "pitch": st.pose.pitch, + "roll": st.pose.roll, + "blendshapes": shapes, + "landmarks": st.landmarks, + }) + ); + } + } +} diff --git a/server-rs/crates/lrg-ml/src/face_state.rs b/server-rs/crates/lrg-ml/src/face_state.rs index d43259d0..d8411121 100644 --- a/server-rs/crates/lrg-ml/src/face_state.rs +++ b/server-rs/crates/lrg-ml/src/face_state.rs @@ -29,6 +29,15 @@ //! than 80° away, less than half visible, or too small to rescale to the input //! without loss. Such a face gets [`EyeState::Unknown`] and abstains from //! every decision that reads the eyes. +//! +//! Two readings needed real photographs to get right (`examples/ +//! face_state_probe.rs`, about 300 portraits from Wikimedia Commons; numbers in +//! `docs/wiki/Dev-Image-Culling-Implementation-Plan.md`). `eyeBlink` tops out +//! well below 1: eyes shut in a photograph read 0.54–0.78 on the more closed +//! eye, where lids moved shut on the mesh read 0.8. And `eyeLook*` measures the +//! eye inside the head, so it says nothing about the lens until the head's own +//! turn is added: someone turned 30° and looking straight at the camera reads +//! 0.99, someone turned 40° with the eyes following the head 0.14. use std::path::PathBuf; use std::sync::atomic::{AtomicI64, Ordering}; @@ -42,7 +51,7 @@ use lrg_imaging::cull_config::FaceMetricsConfig; /// Stamped on every stored face state (`cull_face_state_model`). Bump when a /// change would make new readings incomparable with stored ones. -pub const MODEL_ID: &str = "mediapipe-facemesh-v2+blendshape-v2/1"; +pub const MODEL_ID: &str = "mediapipe-facemesh-v2+blendshape-v2/2"; pub const MESH_INPUT_SIZE: usize = 256; pub const LANDMARK_COUNT: usize = 478; @@ -122,8 +131,14 @@ pub const BLENDSHAPE_NAMES: [&str; BLENDSHAPE_COUNT] = [ const EYE_BLINK_LEFT: usize = 9; const EYE_BLINK_RIGHT: usize = 10; -const EYE_LOOK_FIRST: usize = 11; -const EYE_LOOK_LAST: usize = 18; +const EYE_LOOK_DOWN_LEFT: usize = 11; +const EYE_LOOK_DOWN_RIGHT: usize = 12; +const EYE_LOOK_IN_LEFT: usize = 13; +const EYE_LOOK_IN_RIGHT: usize = 14; +const EYE_LOOK_OUT_LEFT: usize = 15; +const EYE_LOOK_OUT_RIGHT: usize = 16; +const EYE_LOOK_UP_LEFT: usize = 17; +const EYE_LOOK_UP_RIGHT: usize = 18; const EYE_SQUINT_LEFT: usize = 19; const EYE_SQUINT_RIGHT: usize = 20; const MOUTH_SMILE_LEFT: usize = 44; @@ -135,8 +150,22 @@ const RIGHT_EYE_OUTER: usize = 33; const RIGHT_EYE_INNER: usize = 133; const LEFT_EYE_INNER: usize = 362; const LEFT_EYE_OUTER: usize = 263; -const RIGHT_FACE_EDGE: usize = 234; -const LEFT_FACE_EDGE: usize = 454; + +// What the head pose is measured on: landmark pairs spanning the face across +// (the subject's right to left) and upwards, all on bone rather than on the jaw +// or the lips, so that an expression barely moves them. +const POSE_ACROSS: [(usize, usize); 5] = [ + (RIGHT_EYE_OUTER, LEFT_EYE_OUTER), + (RIGHT_EYE_INNER, LEFT_EYE_INNER), + (234, 454), + (127, 356), + (70, 300), +]; +const POSE_UP: [(usize, usize); 4] = [(195, 10), (197, 151), (195, 9), (6, 10)]; +/// How far the [`POSE_UP`] pairs lean back from the face's own vertical on +/// MediaPipe's canonical face model (`canonical_face_model.obj`). With the face +/// symmetric about x, this one angle is all of that model the pose needs. +const POSE_UP_LEAN_DEGREES: f64 = 16.1844; #[derive(Debug, thiserror::Error)] pub enum FaceStateError { @@ -199,19 +228,19 @@ pub struct FaceState { /// Probability that the region holds a face at all (the mesh's own /// presence output, through a sigmoid as MediaPipe applies it). pub presence: f64, - /// How closed the eyes are, `0..1`: the mean of `eyeBlinkLeft/Right`, or - /// only the near eye's when the head is turned so far that the far one is - /// hidden ([`FaceMetricsConfig::face_state_near_eye_yaw_degrees`]). + /// How closed the eyes are, `0..1`: the more closed of `eyeBlinkLeft/Right`, + /// or only the near eye's when the head is turned so far that the far one + /// is hidden ([`FaceMetricsConfig::face_state_near_eye_yaw_degrees`]). pub eye_closed: f64, + /// Mean of `eyeSquintLeft/Right`. High on a broad smile too, so it means a + /// squint only without one. pub eye_squint: f64, /// Mean of `mouthSmileLeft/Right`. pub smile: f64, - /// The strongest of the eight `eyeLook*` coefficients: 0 is looking into - /// the lens. - pub gaze_away: f64, - /// Head turn in degrees from frontal, from the depth of the two face - /// edges; positive turns the subject's left cheek away. - pub yaw_degrees: f64, + /// How far from the lens the eyes point, in degrees: the head's turn and + /// the eyes' own turn inside it ([`gaze_off_lens_degrees`]). + pub gaze_off_lens: f64, + pub pose: HeadPose, pub eye_state: EyeState, pub blendshapes: [f32; BLENDSHAPE_COUNT], /// The 478 landmarks in pixels of the image analysed (`z` scaled like @@ -219,6 +248,18 @@ pub struct FaceState { pub landmarks: Vec<[f64; 3]>, } +/// Which way the head points, in degrees from facing the camera. +#[derive(Debug, Clone, Copy, PartialEq)] +pub struct HeadPose { + /// Positive turns the nose to the right of the image, and the subject's + /// left cheek away. + pub yaw: f64, + /// Positive lifts the chin. + pub pitch: f64, + /// Positive raises the eye on the right of the image. + pub roll: f64, +} + /// The square, rotated region the mesh model sees, in image pixels. #[derive(Debug, Clone, Copy, PartialEq)] pub struct Roi { @@ -336,14 +377,78 @@ fn dist2(a: [f64; 3], b: [f64; 3]) -> f64 { ((a[0] - b[0]).powi(2) + (a[1] - b[1]).powi(2)).sqrt() } -/// Head turn from frontal, in degrees, from the depth difference of the two -/// face edges against their horizontal distance. -pub fn yaw_degrees(landmarks: &[[f64; 3]]) -> f64 { - let r = landmarks[RIGHT_FACE_EDGE]; - let l = landmarks[LEFT_FACE_EDGE]; - (l[2] - r[2]) - .atan2((l[0] - r[0]).abs().max(1e-6)) - .to_degrees() +fn dot(a: [f64; 3], b: [f64; 3]) -> f64 { + a[0] * b[0] + a[1] * b[1] + a[2] * b[2] +} + +fn along(a: [f64; 3], scale: f64) -> [f64; 3] { + [a[0] * scale, a[1] * scale, a[2] * scale] +} + +fn plus(a: [f64; 3], b: [f64; 3]) -> [f64; 3] { + [a[0] + b[0], a[1] + b[1], a[2] + b[2]] +} + +fn normalized(a: [f64; 3]) -> [f64; 3] { + along(a, 1.0 / dot(a, a).sqrt().max(1e-9)) +} + +/// The head's pose from a frame of its landmarks: an axis across the face, one +/// up it, and the one straight out of it. On about 280 real faces this stays +/// within 1° of a full Procrustes fit of all 468 points to MediaPipe's canonical +/// face model on yaw and roll, and within 2° on pitch, at no linear algebra +/// dependency. +pub fn head_pose(landmarks: &[[f64; 3]]) -> HeadPose { + // Into the canonical model's axes: x right, y up, z towards the viewer. + let at = |i: usize| { + let p = landmarks[i]; + [p[0], -p[1], -p[2]] + }; + let span = |pairs: &[(usize, usize)]| { + pairs.iter().fold([0.0; 3], |sum, &(from, to)| { + plus(sum, plus(at(to), along(at(from), -1.0))) + }) + }; + let across = normalized(span(&POSE_ACROSS)); + let up = span(&POSE_UP); + let up = normalized(plus(up, along(across, -dot(across, up)))); + let out = [ + across[1] * up[2] - across[2] * up[1], + across[2] * up[0] - across[0] * up[2], + across[0] * up[1] - across[1] * up[0], + ]; + let (sin, cos) = POSE_UP_LEAN_DEGREES.to_radians().sin_cos(); + let forward = plus(along(up, -sin), along(out, cos)); + HeadPose { + yaw: forward[0].atan2(forward[2]).to_degrees(), + pitch: forward[1].atan2(forward[0].hypot(forward[2])).to_degrees(), + roll: across[1].atan2(across[0]).to_degrees(), + } +} + +/// How far from the lens the eyes point, in degrees: the head's turn plus the +/// eyes' own turn inside it, read off `eyeLook*`. Someone with the head turned +/// who looks back at the camera comes out near 0; the eyes' turn cancels the +/// head's. +/// +/// The up-and-down part is the weaker of the two: `eyeLookUp` barely moves when +/// a lowered head looks up at the lens, so such a face reads 15–20° off it. +pub fn gaze_off_lens_degrees( + pose: HeadPose, + blendshapes: &[f32; BLENDSHAPE_COUNT], + cfg: &FaceMetricsConfig, +) -> f64 { + let b = |i: usize| blendshapes[i] as f64; + // Towards the subject's left, the right of the image. + let across = (b(EYE_LOOK_OUT_LEFT) - b(EYE_LOOK_IN_LEFT) + b(EYE_LOOK_IN_RIGHT) + - b(EYE_LOOK_OUT_RIGHT)) + / 2.0; + let up = (b(EYE_LOOK_UP_LEFT) + b(EYE_LOOK_UP_RIGHT) + - b(EYE_LOOK_DOWN_LEFT) + - b(EYE_LOOK_DOWN_RIGHT)) + / 2.0; + (pose.yaw + cfg.face_state_gaze_across_degrees * across) + .hypot(pose.pitch + cfg.face_state_gaze_up_degrees * up) } /// Turns the raw model outputs into [`FaceState`]. `eye_px` is the @@ -356,10 +461,12 @@ pub fn derive_state( cfg: &FaceMetricsConfig, ) -> FaceState { let b = |i: usize| blendshapes[i] as f64; - let yaw = yaw_degrees(&landmarks); + let pose = head_pose(&landmarks); // Turned far enough that one eye is hidden: only the near one says - // anything. The near eye is the wider one in the image. - let eye_closed = if yaw.abs() >= cfg.face_state_near_eye_yaw_degrees { + // anything. The near eye is the wider one in the image. Otherwise the more + // closed eye speaks: the mean let one shut eye hide behind an open one, and + // read 0.56 on eyes shut that the worse eye put at 0.65. + let eye_closed = if pose.yaw.abs() >= cfg.face_state_near_eye_yaw_degrees { let right_w = dist2(landmarks[RIGHT_EYE_OUTER], landmarks[RIGHT_EYE_INNER]); let left_w = dist2(landmarks[LEFT_EYE_OUTER], landmarks[LEFT_EYE_INNER]); // MediaPipe's "left" blendshapes are the subject's left eye. @@ -369,15 +476,13 @@ pub fn derive_state( b(EYE_BLINK_RIGHT) } } else { - (b(EYE_BLINK_LEFT) + b(EYE_BLINK_RIGHT)) / 2.0 + b(EYE_BLINK_LEFT).max(b(EYE_BLINK_RIGHT)) }; let eye_squint = (b(EYE_SQUINT_LEFT) + b(EYE_SQUINT_RIGHT)) / 2.0; let smile = (b(MOUTH_SMILE_LEFT) + b(MOUTH_SMILE_RIGHT)) / 2.0; - let gaze_away = (EYE_LOOK_FIRST..=EYE_LOOK_LAST) - .map(b) - .fold(0.0f64, f64::max); + let gaze_off_lens = gaze_off_lens_degrees(pose, &blendshapes, cfg); let eye_state = if presence < cfg.face_state_min_presence - || yaw.abs() > cfg.face_state_max_yaw_degrees + || pose.yaw.abs() > cfg.face_state_max_yaw_degrees || eye_px < cfg.face_state_min_eye_px { EyeState::Unknown @@ -393,8 +498,8 @@ pub fn derive_state( eye_closed: eye_closed.clamp(0.0, 1.0), eye_squint: eye_squint.clamp(0.0, 1.0), smile: smile.clamp(0.0, 1.0), - gaze_away: gaze_away.clamp(0.0, 1.0), - yaw_degrees: yaw, + gaze_off_lens, + pose, eye_state, blendshapes, landmarks, @@ -708,17 +813,86 @@ mod tests { assert!((p[0][2] - 16.0).abs() < 1e-5, "{}", p[0][2]); } - fn frontal_landmarks() -> Vec<[f64; 3]> { + /// The points [`head_pose`] reads, from MediaPipe's canonical face model, + /// turned by the given pose and placed in the image as the mesh would. + fn posed_landmarks(yaw: f64, pitch: f64, roll: f64) -> Vec<[f64; 3]> { + const CANONICAL: [(usize, [f64; 3]); 16] = [ + (6, [0.0, 2.473255, 5.788627]), + (9, [0.0, 4.885979, 5.385258]), + (10, [0.0, 8.261778, 4.481535]), + (33, [-4.445859, 2.663991, 3.173422]), + (70, [-5.720968, 4.254584, 2.830852]), + (127, [-7.743095, 2.364999, -2.005167]), + (133, [-1.856432, 2.585245, 3.757904]), + (151, [0.0, 6.54539, 5.027311]), + (195, [0.0, 1.059413, 6.774605]), + (197, [0.0, 1.728369, 6.31675]), + (234, [-7.664182, 0.673132, -2.435867]), + (263, [4.445859, 2.663991, 3.173422]), + (300, [5.720968, 4.254584, 2.830852]), + (356, [7.743095, 2.364999, -2.005167]), + (362, [1.856432, 2.585245, 3.757904]), + (454, [7.664182, 0.673132, -2.435867]), + ]; + let (sy, cy) = yaw.to_radians().sin_cos(); + let (sp, cp) = pitch.to_radians().sin_cos(); + let (sr, cr) = roll.to_radians().sin_cos(); let mut lm = vec![[0.0f64; 3]; LANDMARK_COUNT]; - lm[RIGHT_EYE_OUTER] = [100.0, 100.0, 0.0]; - lm[RIGHT_EYE_INNER] = [130.0, 100.0, 0.0]; - lm[LEFT_EYE_INNER] = [170.0, 100.0, 0.0]; - lm[LEFT_EYE_OUTER] = [200.0, 100.0, 0.0]; - lm[RIGHT_FACE_EDGE] = [70.0, 130.0, 0.0]; - lm[LEFT_FACE_EDGE] = [230.0, 130.0, 0.0]; + for (i, [x, y, z]) in CANONICAL { + // Pitch about x, so +z goes towards +y; then yaw about y, so +z goes + // towards +x; then roll about the viewing axis. The order + // [`head_pose`] reads them back in. + let (y, z) = (y * cp + z * sp, z * cp - y * sp); + let (x, z) = (x * cy + z * sy, z * cy - x * sy); + let (x, y) = (x * cr - y * sr, x * sr + y * cr); + // Back into the image: y down, z away from the viewer. + lm[i] = [200.0 + 10.0 * x, 200.0 - 10.0 * y, -10.0 * z]; + } lm } + fn frontal_landmarks() -> Vec<[f64; 3]> { + posed_landmarks(0.0, 0.0, 0.0) + } + + #[test] + fn the_head_pose_recovers_a_known_pose() { + for (yaw, pitch, roll) in [ + (0.0, 0.0, 0.0), + (30.0, 0.0, 0.0), + (-30.0, 0.0, 0.0), + (0.0, 20.0, 0.0), + (0.0, -20.0, 0.0), + (0.0, 0.0, 15.0), + (-40.0, -15.0, 0.0), + ] { + let got = head_pose(&posed_landmarks(yaw, pitch, roll)); + assert!((got.yaw - yaw).abs() < 0.1, "yaw {yaw}: {got:?}"); + assert!((got.pitch - pitch).abs() < 0.1, "pitch {pitch}: {got:?}"); + assert!((got.roll - roll).abs() < 0.1, "roll {roll}: {got:?}"); + } + } + + /// Which way each sign points, against the geometry and not the formula: a + /// nose moved right, a chin lifted, the right-hand eye raised. + #[test] + fn the_head_pose_signs_mean_what_they_say() { + let turned = posed_landmarks(30.0, 0.0, 0.0); + let (nose, left, right) = (turned[195], turned[454], turned[234]); + assert!(nose[0] > (left[0] + right[0]) / 2.0, "nose to image right"); + assert!(left[2] > right[2], "the subject's left cheek goes away"); + + let lifted = posed_landmarks(0.0, 20.0, 0.0); + let (level, up) = (frontal_landmarks()[195][1], lifted[195][1]); + assert!(up < level, "the nose rises in the image: {up} {level}"); + + let rolled = posed_landmarks(0.0, 0.0, 15.0); + assert!( + rolled[LEFT_EYE_OUTER][1] < rolled[RIGHT_EYE_OUTER][1], + "the eye on the right of the image is the higher one" + ); + } + fn shapes(blink: (f32, f32), smile: f32) -> [f32; BLENDSHAPE_COUNT] { let mut b = [0f32; BLENDSHAPE_COUNT]; b[EYE_BLINK_LEFT] = blink.0; @@ -735,7 +909,22 @@ mod tests { assert_eq!(state((0.04, 0.03)).eye_state, EyeState::Open); assert_eq!(state((0.45, 0.40)).eye_state, EyeState::Half); assert_eq!(state((0.81, 0.76)).eye_state, EyeState::Closed); - assert!((state((0.81, 0.76)).eye_closed - 0.785).abs() < 1e-6); + assert!((state((0.81, 0.76)).eye_closed - 0.81).abs() < 1e-6); + } + + /// One eye shut on a frontal face: the mean would have read this as open. + #[test] + fn the_more_closed_eye_speaks_for_a_frontal_face() { + let c = cfg(); + let s = derive_state( + 0.99, + shapes((0.62, 0.11), 0.0), + frontal_landmarks(), + 100.0, + &c, + ); + assert!((s.eye_closed - 0.62).abs() < 1e-6); + assert_eq!(s.eye_state, EyeState::Closed); } #[test] @@ -747,40 +936,81 @@ mod tests { // Eyes too small in the image. let s = derive_state(0.99, shapes((0.9, 0.9), 0.0), frontal_landmarks(), 8.0, &c); assert_eq!(s.eye_state, EyeState::Unknown); - // Turned almost to profile: the left edge far deeper than the right. - let mut lm = frontal_landmarks(); - lm[LEFT_FACE_EDGE][2] = 1000.0; + // Turned almost to profile. + let lm = posed_landmarks(70.0, 0.0, 0.0); let s = derive_state(0.99, shapes((0.9, 0.9), 0.0), lm, 100.0, &c); - assert!(s.yaw_degrees > 80.0, "{}", s.yaw_degrees); + assert!(s.pose.yaw > 60.0, "{:?}", s.pose); assert_eq!(s.eye_state, EyeState::Unknown); } #[test] fn a_turned_head_is_judged_on_the_near_eye() { let c = cfg(); - // About 40 degrees turned, the subject's left eye (on the right of - // the image) seen wide, the right one foreshortened and reading closed. - let mut lm = frontal_landmarks(); - lm[LEFT_FACE_EDGE][2] = 134.0; - lm[RIGHT_EYE_INNER] = [112.0, 100.0, 0.0]; - let s = derive_state(0.99, shapes((0.05, 0.9), 0.0), lm, 100.0, &c); - assert!( - s.yaw_degrees > 35.0 && s.yaw_degrees < 60.0, - "{}", - s.yaw_degrees - ); + // 40 degrees to the right of the image: the subject's right eye, on + // the left of the image, is the near one and seen wide; the far one is + // foreshortened and reads closed. + let lm = posed_landmarks(40.0, 0.0, 0.0); + let s = derive_state(0.99, shapes((0.9, 0.05), 0.0), lm, 100.0, &c); assert_eq!(s.eye_state, EyeState::Open); assert!((s.eye_closed - 0.05).abs() < 1e-6); } + fn eyes_turned(towards_image_right: f32, up: f32) -> [f32; BLENDSHAPE_COUNT] { + let mut b = [0f32; BLENDSHAPE_COUNT]; + if towards_image_right >= 0.0 { + b[EYE_LOOK_OUT_LEFT] = towards_image_right; + b[EYE_LOOK_IN_RIGHT] = towards_image_right; + } else { + b[EYE_LOOK_IN_LEFT] = -towards_image_right; + b[EYE_LOOK_OUT_RIGHT] = -towards_image_right; + } + if up >= 0.0 { + b[EYE_LOOK_UP_LEFT] = up; + b[EYE_LOOK_UP_RIGHT] = up; + } else { + b[EYE_LOOK_DOWN_LEFT] = -up; + b[EYE_LOOK_DOWN_RIGHT] = -up; + } + b + } + + /// The portrait pose that used to be labelled "looking away": the head + /// turned, the eyes back at the camera. Real readings from the probe. + #[test] + fn eyes_that_come_back_to_the_lens_cancel_the_head_turn() { + let c = cfg(); + let gaze = |yaw, eyes| { + derive_state(0.99, eyes, posed_landmarks(yaw, 0.0, 0.0), 100.0, &c).gaze_off_lens + }; + assert!(gaze(-29.0, eyes_turned(0.99, 0.0)) < 5.0); + assert!(gaze(25.0, eyes_turned(-0.84, 0.0)) < 5.0); + // The head turned and the eyes following it: that is looking away, + // though the eyes themselves are centred. + assert!(gaze(39.0, eyes_turned(0.0, 0.0)) > 35.0); + // A frontal head with the eyes off to the side is looking away too. + assert!(gaze(0.0, eyes_turned(0.9, 0.0)) > 20.0); + // The same turn the other way round does not cancel, it adds up. + assert!(gaze(25.0, eyes_turned(0.5, 0.0)) > 35.0); + } + + #[test] + fn eyes_looking_down_from_a_level_head_are_off_the_lens() { + let c = cfg(); + let s = derive_state(0.99, eyes_turned(0.0, -0.8), frontal_landmarks(), 100.0, &c); + assert!(s.gaze_off_lens > 15.0, "{}", s.gaze_off_lens); + } + #[test] - fn smile_and_gaze_come_from_their_blendshapes() { + fn the_smile_comes_from_its_blendshapes() { let c = cfg(); - let mut b = shapes((0.0, 0.0), 0.9); - b[EYE_LOOK_FIRST + 3] = 0.7; - let s = derive_state(0.99, b, frontal_landmarks(), 100.0, &c); + let s = derive_state( + 0.99, + shapes((0.0, 0.0), 0.9), + frontal_landmarks(), + 100.0, + &c, + ); assert!((s.smile - 0.9).abs() < 1e-6); - assert!((s.gaze_away - 0.7).abs() < 1e-6); } #[test] diff --git a/server-rs/crates/lrg-ml/tests/face_state_goldens.rs b/server-rs/crates/lrg-ml/tests/face_state_goldens.rs index c68bd83b..5b8e8630 100644 --- a/server-rs/crates/lrg-ml/tests/face_state_goldens.rs +++ b/server-rs/crates/lrg-ml/tests/face_state_goldens.rs @@ -183,8 +183,14 @@ fn the_reference_portrait_reads_like_mediapipe() { state.smile ); assert!( - state.yaw_degrees.abs() < 30.0, - "near frontal: {}", - state.yaw_degrees + state.pose.yaw.abs() < 30.0, + "near frontal: {:?}", + state.pose + ); + // She looks straight into the lens, with her head slightly turned. + assert!( + state.gaze_off_lens < 25.0, + "looking at the camera: {}", + state.gaze_off_lens ); } From 0a6e27ef30060b85476ad928463950257e7f6f8b Mon Sep 17 00:00:00 2001 From: Bastian Machek <16717398+bmachek@users.noreply.github.com> Date: Wed, 4 Nov 2026 08:57:56 +0100 Subject: [PATCH 2/4] feat(people,search,cull,edit): ask the face state what people ask of a photo People and search can now ask what a face shows, culling names the mouth and the light on the face, and AI Edit reads the faces it is about to edit. All of it measured on photographs from Wikimedia Commons first (not committed), the same way as the eyes and gaze before it. People and search. lrg_analysis::face_expression answers eyes open, smiling (mouthSmile from 0.5) and looking at the camera (within 20 degrees), per person on GET /v1/faces/persons//photos and for everyone in the frame on /v1/search, both pages and the MCP tools. A reading the model did not make is unknown, not a no: it is counted and the fix named. A person's card shows their best portrait, no longer the sharpest blink, and "best N" takes each person's best photos. A catalog indexed before the model was on disk has rows with no readings, and a cull pass never touched them. It now brings its readings to the rows a full pass wrote, matched by box, leaving identities alone, and stamps the photo current only once they are saved. /index/unprocessed queues a face pass for readings that are out of date or rows the cull pass could not match (face_rows_behind). Checked live: indexed without the model, a person named, cull pass with it: the person keeps the name and the filters answer. Culling gains labels, which move no score until a hand-culled shoot says what they are worth: mouth_open (portrait, event), face_darker, face_hard_light, face_blown. What the measuring changed: - mouthPucker reads up to 0.99 on full lips at rest and mouthFunnel barely moves, so the mouth is jawOpen and mouthLowerDown (from 0.2: 10% of people speaking, 0.6% of portraits). - Blown is the middle channel, not the top: warm light clips red on skin that holds its detail, and that flagged 16% of faces. It is now 2.6% over 0.05. - Luminance is only ever compared with the same people in a sibling frame, since skin ranges over several stops. AI Edit: face_hard_light takes clarity and most contrast, face_blown holds the white point, face_below_surroundings holds contrast. That last reading is lowest on people with dark skin in front of something bright, correctly exposed, so it never calls a face underexposed or backlit: it can only hold contrast back, and says what it measured. A failed read reaches the user; missing models skip the rules, as the comment there explains. Left out on the evidence: keeping turned heads out of cluster formation scores the same at every limit from 60 to 30 degrees on 370 faces of 18 public figures, since the occlusion gate already holds them back, and raising that gate lowers the score. Clustering is unchanged; examples/face_cluster_eval.rs is what measured it. Verified: cargo fmt/clippy/test (workspace) with face and facestate goldens on the real models, check-docs, live against the release binary. busted has one failure, aesthetic_refine_spec.lua:76, which fails the same without this change. Not yet run inside Lightroom. Docs-Reviewed: Credits.md Docs-Reviewed: Dev-Agent-Bridge.md Docs-Reviewed: Help-Find-Similar.md Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy --- docs/wiki/Dev-Backend-API.md | 58 ++- .../Dev-Image-Culling-Implementation-Plan.md | 44 ++- docs/wiki/Help-AI-Agents.md | 4 +- docs/wiki/Help-AI-Edit.md | 5 +- docs/wiki/Help-Advanced-Search.md | 4 + docs/wiki/Help-Analyze-and-Index.md | 2 + docs/wiki/Help-Cull-Photos.md | 12 +- docs/wiki/Help-People-Faces.md | 18 +- .../LrGeniusAI.lrdevplugin/APISearchIndex.lua | 28 +- plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua | 71 +++- plugin/spec/api_search_index_spec.lua | 19 + server-rs/Cargo.lock | 1 + .../examples/face_cluster_eval.rs | 131 +++++++ .../crates/lrg-analysis/src/culling_config.rs | 39 ++ .../lrg-analysis/src/edit_guardrails.rs | 196 +++++++++- .../crates/lrg-analysis/src/face_aggregate.rs | 72 ++++ .../lrg-analysis/src/face_expression.rs | 368 ++++++++++++++++++ server-rs/crates/lrg-analysis/src/grouping.rs | 181 +++++++++ server-rs/crates/lrg-analysis/src/lib.rs | 1 + server-rs/crates/lrg-api/src/edit_budget.rs | 214 +++++++++- server-rs/crates/lrg-api/src/routes/edit.rs | 15 +- server-rs/crates/lrg-api/src/routes/faces.rs | 106 ++++- server-rs/crates/lrg-api/src/routes/index.rs | 15 +- .../crates/lrg-api/src/routes/index_upload.rs | 182 +++++++-- server-rs/crates/lrg-api/src/routes/search.rs | 33 +- .../crates/lrg-api/src/routes/style_edit.rs | 11 +- server-rs/crates/lrg-api/src/ui/people.html | 24 ++ server-rs/crates/lrg-api/src/ui/search.html | 9 + server-rs/crates/lrg-api/tests/contract.rs | 285 ++++++++++++++ .../crates/lrg-imaging/src/cull_config.rs | 4 + server-rs/crates/lrg-mcp/Cargo.toml | 1 + server-rs/crates/lrg-mcp/src/tools_index.rs | 136 ++++++- .../lrg-ml/examples/face_state_probe.rs | 25 +- server-rs/crates/lrg-ml/src/face_state.rs | 342 +++++++++++++++- 34 files changed, 2525 insertions(+), 131 deletions(-) create mode 100644 server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs create mode 100644 server-rs/crates/lrg-analysis/src/face_expression.rs diff --git a/docs/wiki/Dev-Backend-API.md b/docs/wiki/Dev-Backend-API.md index d4477f56..9211f3a8 100644 --- a/docs/wiki/Dev-Backend-API.md +++ b/docs/wiki/Dev-Backend-API.md @@ -268,16 +268,20 @@ most 12, largest first) of the same full-size image, or of the working copy when there is no larger one (`lrg_ml::face_state`). Per face the row gets `face_eye_state` (`open`/`half`/`closed`, `unknown` when the face is turned too far, barely a face or too small), `face_eye_closed` (the more closed eye), -`face_smile`, `face_eye_squint`, `face_gaze_off_lens` (degrees from the lens, head -turn included), `face_yaw`, `face_pitch`, `face_roll` (degrees) and -`face_blendshapes` (all 52, in the order of `lrg_ml::face_state::BLENDSHAPE_NAMES`); -all null when the model did not run for it. These rows exist only where the pass -writes faces, not on a pass that only prepares culling. Per +`face_smile`, `face_eye_squint`, `face_mouth_open`, `face_gaze_off_lens` (degrees from +the lens, head turn included), `face_yaw`, `face_pitch`, `face_roll` (degrees), the light +on the skin as `face_luminance`, `face_against_surround_stops`, `face_across_stops` and +`face_blown`, and `face_blendshapes` (all 52, in the order of +`lrg_ml::face_state::BLENDSHAPE_NAMES`); all null when the model did not run for it. A +pass that only prepares culling writes no rows of its own, but brings these readings +to the rows an earlier full pass wrote, matched by box, and touches nothing else on them. Per photo: `cull_eye_closed` (the most closed prominent eyes), `cull_eye_state`, `cull_smile` (the weakest smile among the prominent faces), `cull_eye_squint` (the strongest squint among the prominent faces not smiling), `cull_gaze_off_lens` (the -prominent face with open eyes looking furthest from the lens) — absent when no -prominent face could be judged — plus `cull_face_state_version` and `cull_face_state_model` when +prominent face with open eyes looking furthest from the lens), `cull_mouth_open` (the +most open mouth among those not smiling), `cull_face_luminance` (the darkest), +`cull_face_across_stops` (the hardest light) and `cull_face_blown` (the most blown skin) — +absent when no prominent face could be judged — plus `cull_face_state_version` and `cull_face_state_model` when the model ran on this pass. Without the model the response says once that a rough eye estimate was used, with the fix (`"Photos were analysed without eye state, smile and gaze: …"`). @@ -404,7 +408,7 @@ producer this build uses*, not merely present: | task | outstanding when | | --- | --- | | `embeddings` | no embedding, or a stored `embedding_model` that is not this build's | -| `faces` | no face rows and no `faces_checked`, or a `face_model` that is not this build's | +| `faces` | no face rows and no `faces_checked`, a `face_model` that is not this build's, or — once its model is on disk — a photo with faces whose `cull_face_state_version` is missing or older, or that a culling pass left with face rows it could not match (`face_rows_behind`) | | `aesthetic` (also implied by `embeddings` and `cull`) | no `aesthetic_score`, or an `aesthetic_model` that is not this build's — and only when the scorer is on disk, since a missing model cannot be fixed by running again | | genre (with `embeddings`, `cull` and `aesthetic`) | an embedding but no `genre`, or a `genre_version` that is not this build's (the genre list or its prompts changed), and only when the search model is on disk. Made again from the stored embedding, so the photo is not re-embedded | | `species` | no `species_checked`, or a `species_model` that is not the head on disk | @@ -508,6 +512,7 @@ Runs a semantic search query against indexed photos. | `sort` | string | `relevance`, `aesthetic` or `combined`. Default `relevance` with a term, `aesthetic` without. `relevance` and `combined` without a term are a **400** ("need a search term"), as is an unknown value | | `min_aesthetic` | float | Only photos whose current aesthetic score is at least this (1–9). A photo with no score from the current export cannot be judged and is left out | | `genres` | array | Only photos whose `genre` is one of these ids (see `GET /v1/search/genres`). A photo with no genre is left out | +| `eyes_open`, `smiling`, `looking_at_camera` | bool | Only photos where this holds for everyone who matters in the frame, read off `cull_eye_state`, `cull_smile` (0.5 or more) and `cull_gaze_off_lens` (20° or less). A photo measured to have no faces is simply no match; one with no current reading cannot be judged and is left out and counted | | `aesthetic_weight` | float | `combined` only, 0–1, default 0.5: the share of the ordering that is the aesthetic score rather than relevance | | `catalog_id` | string | Restrict to a specific catalog | | `photo_ids` | array | Restrict search to specific `photo_id` values (`uuids` is accepted as an alias) | @@ -536,7 +541,7 @@ match in the title, caption or keywords, which has no distance — counts as ful matches without a current score go **last**, in their earlier order: they are not on the scale, and a warning says how many. -`min_aesthetic` and `genres` are applied **after** the relevance cut, so they never move the cut +`min_aesthetic`, `genres` and the face questions are applied **after** the relevance cut, so they never move the cut (it is computed over the whole candidate list) and can leave fewer than `max_results`. What they drop for want of data, rather than for its value, is counted in `warnings`, since that is a problem the user can fix by indexing; a photo that was judged and found short of the minimum is simply not a @@ -624,6 +629,16 @@ thresholds, and a face it could not judge is never a blink. Reasons: `eyes_closed`, `eyes_half_closed` (a label, never a defect), `looking_away` (`portrait`), `best_smile`; a smile at `laugh_smile_threshold` with the eyes shut counts as `eyes_closed_intentional`. + +**Mouth and the light on the face** are labels only and move no score, until a +hand-culled shoot says what they are worth: `mouth_open` (`portrait` and `event`: +`cull_mouth_open` at `reason_mouth_open_threshold`, with a sibling clearly less +open), `face_darker` (the frame's darkest face at `reason_face_darker_stops` below the +group's brightest), `face_hard_light` (`cull_face_across_stops` at +`reason_face_hard_light_stops`, with a sibling clearly softer) and `face_blown` +(`cull_face_blown` at `reason_face_blown_threshold`, also on a single). `street` and +`sports` ask none of them. `cull_face_luminance` is only ever compared within a group, +since skin ranges over several stops between people. `smile_weight` (portrait 0.15, event 0.10) takes a share of the face term, against the group. @@ -634,7 +649,7 @@ against the group. | `summary` | object | `group_count`, `pick_count`, `alternate_count`, `reject_candidate_count`, `near_duplicate_group_count`, `intentional_set_group_count`, `intentional_set_photo_count`, `culling_preset`, `unindexed_count` | | `groups[].winner_photo_id` | string or null | The best-ranked frame that is not a reject candidate. **`null`** when every frame is one: a blurred single, or a burst blurred throughout | | `groups[].alternate_photo_ids`, `groups[].reject_candidate_photo_ids` | string[] | The rest of the group | -| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`eye_squint`/`gaze_off_lens` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | +| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`eye_squint`/`gaze_off_lens`/`mouth_open`/`face_luminance`/`face_across_stops`/`face_blown` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | | `warnings` | string[] | Every caveat for the run, most fundamental first: unanalyzed photos skipped, no embeddings, AI signals whose prompt vectors could not be prepared (one message naming every lost signal and the fix), photos with faces prepared before eye focus or eye state were measured (one message when "Prepare now" fixes both), photos whose eyes the face state model could not analyse because it is not downloaded (with that fix), photos whose moment was not judged. Empty when there is nothing to say | | `warning` | string or null | The same messages joined, for clients older than `warnings` | @@ -715,7 +730,7 @@ mask list, plus: | Field | Type | Description | |---|---|---| -| `guardrail_reasons` | string[] | Machine-readable codes for what the frame allowed or refused, e.g. `hard_light_no_added_contrast`, `flat_light_contrast_allowed`. Empty when the budget changed nothing | +| `guardrail_reasons` | string[] | Machine-readable codes (with the face and expression models on disk, also `face_hard_light`, `face_below_surroundings`, `face_blown`, read off the faces in the frame; if that read fails, `warning` says so and the edit goes on without them) for what the frame allowed or refused, e.g. `hard_light_no_added_contrast`, `flat_light_contrast_allowed`. Empty when the budget changed nothing | | `guardrail_explanations` | string[] | The same, as sentences for the UI | | `edit_warnings` | string[] | Unrelated problems worth surfacing (e.g. no training examples found) | @@ -900,8 +915,10 @@ returned as warnings rather than failing the request. Returns all detected persons with `person_id`, `name`, `face_count`, `photo_count`, `manual_count` (how many of the faces a human placed) and a representative `thumbnail` (base64 JPEG, empty when the person has none). The -representative is the highest-scoring face by `det_score + sharpness`, not the -first row found. +representative is the person's best portrait (`lrg_analysis::face_expression::portrait_score`): +detector confidence and sharpness, less a shut eye, a turned head and a gaze off +the lens, plus a little for a smile. A row without a reading pays a fixed price, so +it yields to a portrait known to be good. Every face without a person — an empty `person_id` from a row that was never clustered, or `person_unassigned` written by the clusterer — is reported as a @@ -946,13 +963,22 @@ already has — as its merge gesture: it asks, then either posts to really are different people with the same name. ### `GET /v1/faces/persons//photos` -Returns the list of `photo_id` values associated with a specific person. +The ids of the photos showing this person, in `photo_ids` (`photo_uuids` is the same list). `_unassigned` opens the faces without a person. + +| Query | Meaning | +|---|---| +| `eyes_open`, `smiling`, `looking_at_camera` | `true` keeps only photos where this person's face answers yes (`lrg_analysis::face_expression`). A person twice on a photo counts when either face does | +| `sort` | `best` puts the person's best portrait first; otherwise by photo id | +| `limit` | At most this many | + +`not_judged` counts the photos left out because the face has no reading, and `warnings` says so with the fix, since that is a problem the user can solve, where a frown is simply not a smile. ### `GET /v1/faces/persons//faces` Returns the individual faces in one person, which is what the People page's detail view is built on. Query parameters: `limit` (default 200, max 500), `offset`, and `sort` — `typical` (closest to the cluster centroid first), -`outlier` (furthest first, which surfaces wrong faces) or `quality`. +`outlier` (furthest first, which surfaces wrong faces), `quality` or `portrait` (the +best picture of them first, as for the person's card). Only this person's vectors are fetched, so the endpoint is cheap enough to page through. Each face carries `face_id`, `photo_id`, `thumbnail`, `distance` from @@ -1078,7 +1104,7 @@ nothing will ever run: | page | action | needs | |---|---|---| -| `people` | `show_in_library` | a non-empty `person_ids` array | +| `people` | `show_in_library` | a non-empty `person_ids` array; `eyes_open`, `smiling`, `looking_at_camera`, `best` are passed on to `GET /v1/faces/persons//photos` | | `search` | `show_search_results` | a non-empty `photo_ids` array of strings, in rank order; `term`, `sort`, `min_aesthetic`, `genre_label`, `max_results` name the collection | | `search` | `refresh_search_context` | nothing | diff --git a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md index 61d16842..a37ba26a 100644 --- a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md +++ b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md @@ -1245,7 +1245,49 @@ thresholds, judged by eye, then 292 from a disjoint sample to check them): smile cannot be read from `cheekSquint`. `jawOpen`, `mouthPucker`, `browDown` and `mouthPress` do move. -What this does not solve: a slit narrowed against the sun overlaps eyes shut (0.5 +### Mouth and the light on the face + +Both went in as labels and move no score: there is still no hand-culled shoot to say +what they are worth. Measured the same way, on 692 usable faces with the second +sample and 190 photographs of people speaking added. + +- **The mouth is `jawOpen` and `mouthLowerDown`.** `mouthPucker` reads up to 0.99 on + full lips at rest and `mouthFunnel` barely moves, so neither is used. From 0.2 the + mouth is visibly open on nearly every face judged: 10% of people speaking, 0.6% of + portraits. A laugh is open too, so only a face smiling under 0.4 counts. +- **Blown means two channels.** The top channel at 250 flagged 16% of faces, most of + them warm light clipping red on skin that holds its detail. The middle channel at + 245 leaves 2.6% over 0.05, where surgical masks and white headbands still sit, and + from 0.15 what is left is blown skin. +- **Light across the face holds up.** It compares a face with itself, so it does not + care about skin. 10% of faces reach 2 stops, nearly all real side light; hair across + a cheek is the main false reading. +- **A face against its surroundings does not mean what it looks like.** The lowest + readings are almost all people with dark skin in front of something bright, and + correctly exposed. So it never produces a label. Its only use is to hold back added + contrast in AI Edit, which costs that face nothing, and that explanation says what + was measured rather than calling the frame backlit. +- **The same goes for luminance alone**, which is only ever compared with the same + people in another frame of the group. + +### People, and what clustering did not need + +`lrg_analysis::face_expression` answers what People and search ask: eyes open, +smiling (`mouthSmile` from 0.5, where most faces judged by eye show a smile and a +snarl can still reach it), looking at the camera (within 20°, so a face between that +and the 25° of `looking_away` is claimed by neither), and a person's best portrait. + +Keeping turned heads out of cluster formation looked like the obvious next step and +is not in: `examples/face_cluster_eval.rs`, on 370 faces of 18 public figures from +Commons, scores it the same at every limit from 60° to 30°, because the occlusion gate +already holds those faces back. That gate holds back over half of them, which looked +wrong, but raising it from 0.55 lowers the score at every threshold tried. Recall in +this sample is low (0.29) whichever way, with few photographs per person, so it says +what these two changes do and not how good clustering is. + +### What this does not solve + +A slit narrowed against the sun overlaps eyes shut (0.5 to 0.65), and neither `eyeSquint` nor the lid gap in the landmarks separates them; sunglasses read as closed (0.47–0.74); a head lying on its side reads open with the eyes shut. The relative rules catch the first two inside a burst, since every frame diff --git a/docs/wiki/Help-AI-Agents.md b/docs/wiki/Help-AI-Agents.md index 7ef679da..6dc1886f 100644 --- a/docs/wiki/Help-AI-Agents.md +++ b/docs/wiki/Help-AI-Agents.md @@ -40,9 +40,9 @@ Behind those requests the agent uses these tools: | Tool | What it does | |---|---| | `get_status` | Whether Lightroom is connected, the open catalog, the selection, how much is indexed | -| `search_photos` | Semantic search: describe what is in the picture. Can also rank by aesthetic score and keep only some genres; with no description it returns your best photos (see [Advanced Search](Help-Advanced-Search#ranking)) | +| `search_photos` | Semantic search: describe what is in the picture. Can also rank by aesthetic score, keep only some genres, and ask of the faces (eyes open, smiling, looking at the camera); with no description it returns your best photos (see [Advanced Search](Help-Advanced-Search#ranking)) | | `find_similar_photos` | Photos that look alike, or near-duplicates | -| `list_people`, `find_photos_of_people` | People from LrGeniusAI's face recognition | +| `list_people`, `find_photos_of_people` | People from LrGeniusAI's face recognition, also narrowed by what their face shows (eyes open, smiling, looking at the camera) or to their best portraits | | `find_photos_in_catalog` | Lightroom's own filters: rating, flag, label, keywords, folder, dates, camera, lens | | `cull_photos` | Groups bursts and duplicates, names winners and reject candidates | | `get_photo_details`, `view_photo` | Metadata (including aesthetic score and genre) — on request every field Lightroom keeps, IPTC, EXIF, GPS and other plug-ins' fields included — and the photo as Lightroom renders it now | diff --git a/docs/wiki/Help-AI-Edit.md b/docs/wiki/Help-AI-Edit.md index 23efb4b8..19ad9a8f 100644 --- a/docs/wiki/Help-AI-Edit.md +++ b/docs/wiki/Help-AI-Edit.md @@ -133,7 +133,10 @@ thin style profile from one caused by an unusual photo. When enabled, a **review dialog** opens for each photo before the edit is applied. You see: - The proposed develop values, plus how confident the style match was and which of your saved edits it drew on. -- Any guardrail explanations — what the frame allowed, and what got capped. +- Any guardrail explanations — what the frame allowed, and what got capped. With + the face models downloaded these include the faces in the frame: hard light across + a face leaves clarity alone, a face well below the frame around it keeps contrast + from pushing it darker, and blown skin holds the white point back. - Options to **Apply** or **Skip**. **Recommended for first use.** Disable only after you've validated the results for your shooting style. diff --git a/docs/wiki/Help-Advanced-Search.md b/docs/wiki/Help-Advanced-Search.md index 59e69215..f22b6387 100644 --- a/docs/wiki/Help-Advanced-Search.md +++ b/docs/wiki/Help-Advanced-Search.md @@ -152,6 +152,10 @@ keep it, so open the search from Lightroom each time. **Keywords and captions** (the metadata half of the search, all four fields) — the dialog's controls of the same name. They start where you left them in the dialog. +- **Faces** — *eyes open*, *smiling*, *looking at the camera*: only photos where + that holds for everyone who matters in the frame. It is read off the faces, so it + is more reliable than searching for "smiling". Photos without faces are left out, + and the page says how many could not be judged and what to run on them. Sliders and menus search again as you move them. Notes from the backend — no search model, photos without a score left out by the minimum, and the like — diff --git a/docs/wiki/Help-Analyze-and-Index.md b/docs/wiki/Help-Analyze-and-Index.md index 3473de03..cd132e7f 100644 --- a/docs/wiki/Help-Analyze-and-Index.md +++ b/docs/wiki/Help-Analyze-and-Index.md @@ -88,6 +88,8 @@ needed for metadata: embeddings, faces and species run without one. [culling](Help-Cull-Photos.md#focus-on-the-eyes) and — with the eye and expression model downloaded — whether the eyes are open, the smile and the gaze; a photo without faces is not. + Photos whose faces were indexed before that model was on disk are read again the + next time you run this with face detection on them, and keep the people you named. - **Identify animal and plant species** — runs BioCLIP 2 and writes a taxonomic identification. Greyed out until the models have been downloaded in the Plug-in Manager, under *On-device AI models* — one **Download AI diff --git a/docs/wiki/Help-Cull-Photos.md b/docs/wiki/Help-Cull-Photos.md index f41bfa52..229aaad9 100644 --- a/docs/wiki/Help-Cull-Photos.md +++ b/docs/wiki/Help-Cull-Photos.md @@ -223,6 +223,14 @@ blink. - **The smile counts in Portrait and Event.** Between otherwise equal frames, the one where everyone smiles more wins, and its explanation says "best smile in group". +- **The mouth, in Portrait and Event.** A frame with the mouth caught open mid-word + says so when another frame of the burst has it closed. A note only; a broad + smile or a laugh does not count. +- **The light on the face.** The explanation names a face darker than in another + frame of the burst, harder light across the face than in another frame, and skin + that is blown out. Not at **Street** or **Sports**. These are notes and move no + score. A face is only ever compared with itself in another frame, never with a + standard, because skin differs by several stops between people. - **Looking away means from the lens.** **Portrait** notes when someone looks away from the camera. Head and eyes count together: a head turned to the side with the eyes back on the camera is looking at it, and a head turned with the @@ -288,7 +296,9 @@ For each photo, the backend stores culling-related fields such as: `cull_face_focus` (focus on the eyes and on the face, measured on the full-size original; absent when the faces were too small to judge), and `cull_eye_state` / `cull_eye_closed`, `cull_smile`, `cull_eye_squint`, - `cull_gaze_off_lens` (from the face model; absent when no face could be judged) + `cull_gaze_off_lens`, `cull_mouth_open`, `cull_face_luminance`, + `cull_face_across_stops`, `cull_face_blown` (from the face model; absent when no + face could be judged) - aesthetic scores: `cull_aesthetic` (a contrast-and-colour heuristic) and the photo's **Aesthetic score** from the AestheticSigLIP model, stored at indexing time (`aesthetic_score`) and read by culling as `cull_aesthetic_model` diff --git a/docs/wiki/Help-People-Faces.md b/docs/wiki/Help-People-Faces.md index c3f0ca91..8583b7ce 100644 --- a/docs/wiki/Help-People-Faces.md +++ b/docs/wiki/Help-People-Faces.md @@ -35,7 +35,8 @@ does not keep it, so open People from Lightroom each time. A grid of persons from your indexed photos. Each card has: -- A thumbnail of the representative face. +- A thumbnail of their best portrait: eyes open and facing the camera, where the + model could read that. - A name field, or *Unassigned faces* for the faces not yet grouped into a person. - The number of photos this person appears in and how many faces are in the @@ -117,8 +118,9 @@ actually right. Each face shows how much it looks like the rest of the group. A face further from the group than the clustering threshold is marked as an outlier, and a face that failed the quality gate says which measurement was bad. The **sort** -control offers *least like this person first*, *most like this person first* -and *best quality first* — the first of those is the fast way to spot a face +control offers *least like this person first*, *most like this person first*, +*best quality first* and *best portrait first* (eyes open, facing the camera, +sharp) — the first of those is the fast way to spot a face that does not belong, so it is what the view opens on. Click faces to select them (shift-click for a range), then: @@ -153,8 +155,16 @@ several people selected, the dropdown decides what you get: together (the default) - **photos with any selected person** — the union +**Only** narrows that down by what the person's face shows: *eyes open*, +*smiling*, *looking at the camera*. With several people selected, each of them is +asked. **Photos** takes only each person's best 20 or 50 portraits instead of all +of them. A photo whose face could not be judged is left out, and Lightroom says how +many that was and what to do about it: their faces were indexed before the eye and +expression model was downloaded. Running **Analyze & Index Photos** with face +detection reads them, and keeps the people you named. + The collection goes into the `People` collection set and is named after the -selected people plus a timestamp. People without a name appear in it once, as +selected people, what you asked for, and a timestamp. People without a name appear in it once, as "Unnamed person"; a name list longer than 100 characters becomes `People ()`. diff --git a/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua b/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua index 5f147bc0..750b188e 100644 --- a/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua +++ b/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua @@ -4397,15 +4397,39 @@ function SearchIndexAPI.setPersonName(personId, name) return true, result or {} end +--- +-- The query string asking for only some of a person's photos. +-- @param options table|nil eyes_open, smiling, looking_at_camera (booleans), best (a count) +-- @return string "" when it asks nothing, else starting with "?" +function SearchIndexAPI.personPhotosQuery(options) + options = options or {} + local parts = {} + for _, key in ipairs({ "eyes_open", "smiling", "looking_at_camera" }) do + if options[key] == true then + parts[#parts + 1] = key .. "=true" + end + end + if type(options.best) == "number" and options.best > 0 then + parts[#parts + 1] = "sort=best" + parts[#parts + 1] = string.format("limit=%d", options.best) + end + if #parts == 0 then + return "" + end + return "?" .. table.concat(parts, "&") +end + --- -- Get photo UUIDs that contain this person. -- @param personId string e.g. "person_0" --- @return table|nil { status, person_id, photo_uuids } or nil, err -function SearchIndexAPI.getPhotosForPerson(personId) +-- @param options table|nil see personPhotosQuery +-- @return table|nil { status, person_id, photo_uuids, not_judged, warnings } or nil, err +function SearchIndexAPI.getPhotosForPerson(personId, options) if not personId or personId == "" then return nil, "person_id required" end local url = SearchIndexAPI.url("FACES_PERSON_PHOTOS", personId, "photos") + .. SearchIndexAPI.personPhotosQuery(options) local result, err = _request("GET", url, {}) if err then log:error("getPhotosForPerson failed: " .. err) diff --git a/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua b/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua index ddff2d9d..88d87f97 100644 --- a/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua +++ b/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua @@ -57,23 +57,51 @@ local function photoIdsFromPersonResponse(resp) or (type(resp.photo_uuids) == "table" and resp.photo_uuids or {}) end +--- What a selection on the People page asked for, in words: "best 20, eyes open". +local function describePhotoOptions(options) + options = options or {} + local parts = {} + if type(options.best) == "number" then + parts[#parts + 1] = string.format("best %d", options.best) + end + if options.eyes_open then + parts[#parts + 1] = "eyes open" + end + if options.smiling then + parts[#parts + 1] = "smiling" + end + if options.looking_at_camera then + parts[#parts + 1] = "looking at the camera" + end + return table.concat(parts, ", ") +end + +--- Collects what a person-photos response says went short, in either field. +local function addResponseWarnings(warnings, resp) + if resp.warning then + log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) + addWarning(warnings, resp.warning) + end + for _, warning in ipairs(type(resp.warnings) == "table" and resp.warnings or {}) do + log:warn("GetPhotosForPerson warning: " .. tostring(warning)) + addWarning(warnings, warning) + end +end + --- Union: jedes Foto, das mindestens eine der Personen enthält (Reihenfolge: erstes Auftreten). -local function unionPhotoIdsForEntries(entries, warnings) +local function unionPhotoIdsForEntries(entries, warnings, options) local seen = {} local photoIdsOrdered = {} for _, ent in ipairs(entries) do local pid = ent and ent.person_id if pid and pid ~= "" then - local resp, err = SearchIndexAPI.getPhotosForPerson(pid) + local resp, err = SearchIndexAPI.getPhotosForPerson(pid, options) if err or type(resp) ~= "table" then ErrorHandler.handleError("Could not get photos for person", err or "No data") return nil end - if resp and resp.warning then - log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) - addWarning(warnings, resp.warning) - end + addResponseWarnings(warnings, resp) local ids = photoIdsFromPersonResponse(resp) for _, photoId in ipairs(ids) do @@ -88,7 +116,7 @@ local function unionPhotoIdsForEntries(entries, warnings) end --- Schnittmenge: Fotos, in denen alle Personen gemeinsam vorkommen (Reihenfolge wie erste Person). -local function intersectPhotoIdsForEntries(entries, warnings) +local function intersectPhotoIdsForEntries(entries, warnings, options) if #entries == 0 then return {} end @@ -99,15 +127,12 @@ local function intersectPhotoIdsForEntries(entries, warnings) if not pid or pid == "" then return {} end - local resp, err = SearchIndexAPI.getPhotosForPerson(pid) + local resp, err = SearchIndexAPI.getPhotosForPerson(pid, options) if err or type(resp) ~= "table" then ErrorHandler.handleError("Could not get photos for person", err or "No data") return nil end - if resp.warning then - log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) - addWarning(warnings, resp.warning) - end + addResponseWarnings(warnings, resp) local ids = photoIdsFromPersonResponse(resp) if not firstIds then firstIds = ids @@ -142,7 +167,7 @@ local function intersectPhotoIdsForEntries(entries, warnings) end --- Führt "Show in Library" aus. entries: { person_id, person_name? }[]; matchMode: "union" | "intersection". -local function doShowInLibrary(entries, matchMode) +local function doShowInLibrary(entries, matchMode, options) if not entries or #entries == 0 then LrDialogs.message("No people selected", "Select one or more people on the People page, then try again.", "info") return @@ -152,9 +177,9 @@ local function doShowInLibrary(entries, matchMode) local warnings = {} local photoIdsOrdered if mode == "intersection" and #entries >= 2 then - photoIdsOrdered = intersectPhotoIdsForEntries(entries, warnings) + photoIdsOrdered = intersectPhotoIdsForEntries(entries, warnings, options) else - photoIdsOrdered = unionPhotoIdsForEntries(entries, warnings) + photoIdsOrdered = unionPhotoIdsForEntries(entries, warnings, options) end if photoIdsOrdered == nil then @@ -166,7 +191,10 @@ local function doShowInLibrary(entries, matchMode) reportWarnings(warnings) if #photoIdsOrdered == 0 then - if mode == "intersection" and #entries >= 2 then + local asked = describePhotoOptions(options) + if asked ~= "" then + LrDialogs.message("No photos", string.format("No photos of this selection match: %s.", asked), "info") + elseif mode == "intersection" and #entries >= 2 then LrDialogs.message("No photos", "No photos contain all selected people together.", "info") else LrDialogs.message("No photos", "No photos found for this person.", "info") @@ -196,6 +224,10 @@ local function doShowInLibrary(entries, matchMode) if #label > 100 then label = string.format("People (%d)", #entries) end + local asked = describePhotoOptions(options) + if asked ~= "" then + label = string.format("%s (%s)", label, asked) + end local collectionName = string.format("%s @ %s", label, LrDate.timeToW3CDate(LrDate.currentTime())) local collectionSet, collection @@ -249,7 +281,12 @@ local function runAction(action) ) return end - doShowInLibrary(entries, action.match_mode) + doShowInLibrary(entries, action.match_mode, { + eyes_open = action.eyes_open == true, + smiling = action.smiling == true, + looking_at_camera = action.looking_at_camera == true, + best = type(action.best) == "number" and action.best or nil, + }) return end diff --git a/plugin/spec/api_search_index_spec.lua b/plugin/spec/api_search_index_spec.lua index 60b6d044..0a540310 100644 --- a/plugin/spec/api_search_index_spec.lua +++ b/plugin/spec/api_search_index_spec.lua @@ -101,6 +101,25 @@ describe("SearchIndexAPI.interpretIndexResponse", function() end) end) +describe("SearchIndexAPI.personPhotosQuery", function() + it("asks nothing when nothing was asked", function() + assert.are.equal("", SearchIndexAPI.personPhotosQuery(nil)) + assert.are.equal("", SearchIndexAPI.personPhotosQuery({ eyes_open = false, best = 0 })) + end) + + it("asks only for what was switched on", function() + assert.are.equal("?eyes_open=true", SearchIndexAPI.personPhotosQuery({ eyes_open = true, smiling = false })) + assert.are.equal( + "?smiling=true&looking_at_camera=true", + SearchIndexAPI.personPhotosQuery({ smiling = true, looking_at_camera = true }) + ) + end) + + it("turns a count into the best photos first", function() + assert.are.equal("?sort=best&limit=20", SearchIndexAPI.personPhotosQuery({ best = 20 })) + end) +end) + describe("SearchIndexAPI.interpretEditResponse", function() it("accepts a success response and returns the whole body", function() local response = { status = "success", recipe = { Exposure2012 = 0.3 } } diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index f0955973..4c48ba86 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -4489,6 +4489,7 @@ dependencies = [ "futures-util", "http-body-util", "log", + "lrg-analysis", "lrg-api", "lrg-common", "lrg-store", diff --git a/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs b/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs new file mode 100644 index 00000000..01104dc4 --- /dev/null +++ b/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs @@ -0,0 +1,131 @@ +//! Scores face clustering against known identities, for a change to its policy. +//! +//! ```text +//! cargo run --release -p lrg-ml --example face_state_probe -- --embed people/*/*.jpg > faces.jsonl +//! cargo run --release -p lrg-analysis --example face_cluster_eval -- faces.jsonl +//! ``` +//! +//! A face's identity is the folder its photo is in, and only the largest face of +//! each photo counts, so a photo of two people does not read as a mistake. The +//! score is BCubed: precision is how much of a face's cluster is the same person, +//! recall how much of that person is in the face's cluster. A face left unassigned +//! is a cluster of its own, which is what it is to whoever browses the People page. + +use std::collections::HashMap; +use std::path::Path; + +use lrg_analysis::face_cluster::{cluster_faces, ClusterParams, FaceInput}; +use serde_json::Value; + +struct Scored { + precision: f64, + recall: f64, + clusters: usize, + unassigned: usize, + held_back: usize, + attached: usize, +} + +fn score(faces: &[FaceInput], identity: &[String], params: &ClusterParams) -> Scored { + let outcome = cluster_faces(faces, params); + let label = |i: usize| { + outcome.labels[i] + .map(|l| l as i64) + .unwrap_or(-(i as i64) - 1) + }; + let (mut precision, mut recall) = (0.0, 0.0); + for i in 0..faces.len() { + let same_cluster = |j: &usize| label(*j) == label(i); + let same_person = |j: &usize| identity[*j] == identity[i]; + let in_cluster = (0..faces.len()).filter(same_cluster).count() as f64; + let of_person = (0..faces.len()).filter(same_person).count() as f64; + let both = (0..faces.len()) + .filter(|j| same_cluster(j) && same_person(j)) + .count() as f64; + precision += both / in_cluster; + recall += both / of_person; + } + let n = faces.len().max(1) as f64; + Scored { + precision: precision / n, + recall: recall / n, + clusters: outcome.clusters.len(), + unassigned: outcome.labels.iter().filter(|l| l.is_none()).count(), + held_back: outcome.stats.low_quality, + attached: outcome.stats.attached, + } +} + +fn main() { + let path = std::env::args().nth(1).expect("faces.jsonl"); + let text = std::fs::read_to_string(path).expect("readable file"); + let mut largest: HashMap = HashMap::new(); + for line in text.lines() { + let row: Value = serde_json::from_str(line).expect("one JSON object per line"); + let file = row["file"].as_str().unwrap_or_default().to_string(); + let area = |v: &Value| v["area_ratio"].as_f64().unwrap_or(0.0); + if largest + .get(&file) + .is_none_or(|kept| area(kept) < area(&row)) + { + largest.insert(file, row); + } + } + let mut rows: Vec<(String, Value)> = largest.into_iter().collect(); + rows.sort_by(|a, b| a.0.cmp(&b.0)); + + let identity: Vec = rows + .iter() + .map(|(file, _)| { + Path::new(file) + .parent() + .map(|p| p.display().to_string()) + .unwrap_or_default() + }) + .collect(); + let number = |v: &Value, key: &str| v[key].as_f64(); + let faces: Vec = rows + .iter() + .map(|(file, row)| FaceInput { + id: file.clone(), + embedding: row["embedding"] + .as_array() + .expect("run the probe with --embed") + .iter() + .filter_map(|v| v.as_f64().map(|x| x as f32)) + .collect(), + det_score: number(row, "det_score"), + area_ratio: number(row, "area_ratio"), + sharpness: number(row, "sharpness"), + occlusion: number(row, "occlusion"), + pinned: None, + }) + .collect(); + println!( + "{} faces of {} people", + faces.len(), + identity + .iter() + .collect::>() + .len(), + ); + println!("threshold max_occl clusters unassigned held attached precision recall f1"); + for threshold in [0.4, 0.5, 0.6] { + for max_occlusion in [0.55, 0.7, 0.85, 1.0] { + let mut params = ClusterParams::defaults(); + params.cosine_threshold = threshold; + params.gate.max_occlusion = max_occlusion; + let s = score(&faces, &identity, ¶ms); + println!( + "{threshold:9.2} {max_occlusion:8.2} {:8} {:10} {:4} {:8} {:9.3} {:6.3} {:.3}", + s.clusters, + s.unassigned, + s.held_back, + s.attached, + s.precision, + s.recall, + 2.0 * s.precision * s.recall / (s.precision + s.recall).max(1e-9), + ); + } + } +} diff --git a/server-rs/crates/lrg-analysis/src/culling_config.rs b/server-rs/crates/lrg-analysis/src/culling_config.rs index d7958e83..b41feed2 100644 --- a/server-rs/crates/lrg-analysis/src/culling_config.rs +++ b/server-rs/crates/lrg-analysis/src/culling_config.rs @@ -333,6 +333,27 @@ pub struct RankingConfig { /// Above this many degrees from the lens (`cull_gaze_off_lens`) a frame is /// labelled `looking_away`. 0 switches the label off; it is never a defect. pub reason_gaze_off_lens_degrees: f64, + /// `cull_mouth_open` from here up, on a frame with a sibling whose mouth is + /// [`Self::reason_mouth_open_delta`] less open, is labelled `mouth_open`. + /// Measured: 10% of photographs of people speaking reach 0.2, and 0.6% of + /// portraits. 0 switches the label off; it is never a defect. + pub reason_mouth_open_threshold: f64, + pub reason_mouth_open_delta: f64, + /// A frame whose darkest face is this many stops below the same reading in + /// its group's brightest frame is labelled `face_darker`. Only within a + /// group, since skin ranges over several stops between people and this + /// must never call dark skin underexposed. 0 switches the label off. + pub reason_face_darker_stops: f64, + /// `cull_face_blown` from here up is labelled `face_blown`. 0 switches it + /// off. Measured: 2.6% of faces pass 0.05, where surgical masks and white + /// headbands still sit, and from 0.15 what is left is blown skin. + pub reason_face_blown_threshold: f64, + /// `cull_face_across_stops` from here up, on a frame with a sibling whose + /// light is [`Self::reason_face_hard_light_delta`] stops softer, is labelled + /// `face_hard_light`. 10% of faces reach 2 stops, nearly all real side + /// light. 0 switches the label off. + pub reason_face_hard_light_stops: f64, + pub reason_face_hard_light_delta: f64, } /// Which deliberately-captured multi-frame set a group is, if any. @@ -467,6 +488,10 @@ impl CullingConfig { cfg.ranking.laugh_smile_threshold = 0.0; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; cfg.ranking.reason_gaze_off_lens_degrees = 0.0; + cfg.ranking.reason_mouth_open_threshold = 0.0; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; cfg } } @@ -545,6 +570,12 @@ const BASE: CullingConfig = CullingConfig { smile_weight: 0.0, reason_smile_delta: 0.05, reason_gaze_off_lens_degrees: 0.0, + reason_mouth_open_threshold: 0.0, + reason_mouth_open_delta: 0.15, + reason_face_darker_stops: 0.75, + reason_face_blown_threshold: 0.15, + reason_face_hard_light_stops: 2.0, + reason_face_hard_light_delta: 1.0, }, sets: SetDetectionConfig { enabled: true, @@ -601,6 +632,7 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reject_eye_closed_threshold = 0.5; cfg.ranking.smile_weight = 0.15; cfg.ranking.reason_gaze_off_lens_degrees = 25.0; + cfg.ranking.reason_mouth_open_threshold = 0.2; } "street" => { cfg.ranking.face_group_weight_technical = 0.70; @@ -632,6 +664,9 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { // Closed eyes are often the moment in street work. cfg.ranking.reason_eye_closed_threshold = 0.62; cfg.ranking.reject_eye_closed_threshold = 0.72; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; } "event" => { cfg.grouping.time_window_default_seconds = 2; @@ -665,6 +700,7 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.eyes_closed_intent_threshold = 0.7; // Everyone smiling is half of a group picture. cfg.ranking.smile_weight = 0.10; + cfg.ranking.reason_mouth_open_threshold = 0.2; } "sports" => { cfg.grouping.time_window_default_seconds = 3; @@ -706,6 +742,9 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reason_eye_closed_threshold = 0.6; cfg.ranking.reject_eye_closed_threshold = 0.68; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; } _ => {} } diff --git a/server-rs/crates/lrg-analysis/src/edit_guardrails.rs b/server-rs/crates/lrg-analysis/src/edit_guardrails.rs index 3150fab6..e30b5116 100644 --- a/server-rs/crates/lrg-analysis/src/edit_guardrails.rs +++ b/server-rs/crates/lrg-analysis/src/edit_guardrails.rs @@ -14,10 +14,16 @@ //! surprising edit that explains itself is debuggable, and one that does not is //! indistinguishable from a bug. //! +//! The rules about faces read [`FaceConditions`], which the caller measures on +//! the same frame. Every one of them compares a face with itself or with its own +//! frame, and none calls a face too dark: skin ranges over several stops between +//! people, so a dark face in bright surroundings is just as often exposed +//! perfectly. All such a reading may do here is hold contrast back, which costs +//! that face nothing. +//! //! What is **not** here yet, because it needs signals this crate cannot reach: -//! the face-differential rule (needs face boxes), mixed-light detection (needs a -//! regional illuminant estimate) and the backlight rule (needs a subject -//! region). [`EditBudget`] is shaped so those slot in without changing callers. +//! mixed-light detection (needs a regional illuminant estimate). [`EditBudget`] +//! is shaped so it slots in without changing callers. use lrg_imaging::scene::SceneState; @@ -35,6 +41,21 @@ pub struct CaptureConditions { pub is_raw: bool, } +/// What was measured about the faces in the frame, as far as it limits an edit. +/// The worst of the faces that matter in the frame speaks for it, and `None` is +/// "not measured": no faces, or no face model on this computer. +#[derive(Debug, Clone, Copy, Default)] +pub struct FaceConditions { + /// Between the two cheeks of the hardest-lit face, in stops. + pub across_face_stops: Option, + /// The darkest face against the frame around it, in stops; negative is + /// darker. A backlit face reads low, and so does dark skin in bright + /// surroundings, which is why this only ever holds contrast back. + pub against_surround_stops: Option, + /// The largest share of a face's skin that is blown out, `0..1`. + pub blown: Option, +} + /// Tunables for [`derive_budget`]. /// /// Every threshold here is a starting guess, chosen to be defensible rather @@ -66,6 +87,17 @@ pub struct GuardrailConfig { pub contrast_budget_flat: f64, /// Largest clarity value on an ordinary frame. pub clarity_budget_default: f64, + /// Light this many stops apart between the two cheeks is hard light on the + /// face. 10% of faces in a sample of 600 portraits reach 2, nearly all of + /// them real side light. + pub face_hard_light_stops: f64, + /// A face this many stops below the frame around it. 2.4% of that sample. + pub face_below_surround_stops: f64, + /// This share of a face's skin blown out; below it sit surgical masks and + /// white headbands. + pub face_blown_share: f64, + /// Largest total of contrast-increasing moves on a face that needs it. + pub contrast_budget_face: f64, } impl GuardrailConfig { @@ -81,6 +113,10 @@ impl GuardrailConfig { contrast_budget_default: 25.0, contrast_budget_flat: 40.0, clarity_budget_default: 20.0, + face_hard_light_stops: 2.0, + face_below_surround_stops: 2.0, + face_blown_share: 0.15, + contrast_budget_face: 10.0, } } } @@ -107,6 +143,12 @@ pub enum GuardrailReason { HighIsoShadowsLimited, /// Shadows are already blocked; lifting them cannot recover detail. ShadowsAlreadyClipped, + /// Hard light across a face: clarity and contrast would deepen the split. + FaceHardLight, + /// A face well below the frame around it: contrast would push it further. + FaceBelowSurroundings, + /// Blown skin on a face: the white point must not push it further. + FaceBlown, } impl GuardrailReason { @@ -120,6 +162,9 @@ impl GuardrailReason { GuardrailReason::HighlightsUnrecoverable => "highlights_unrecoverable", GuardrailReason::HighIsoShadowsLimited => "high_iso_shadows_limited", GuardrailReason::ShadowsAlreadyClipped => "shadows_already_clipped", + GuardrailReason::FaceHardLight => "face_hard_light", + GuardrailReason::FaceBelowSurroundings => "face_below_surroundings", + GuardrailReason::FaceBlown => "face_blown", } } @@ -143,6 +188,15 @@ impl GuardrailReason { GuardrailReason::ShadowsAlreadyClipped => { "Shadows are already blocked, so lifting them further would not recover detail." } + GuardrailReason::FaceHardLight => { + "Hard light falls across a face, so clarity was left alone and contrast limited: both would deepen the split." + } + GuardrailReason::FaceBelowSurroundings => { + "A face sits well below the frame around it, so contrast was limited to keep it from going darker." + } + GuardrailReason::FaceBlown => { + "Skin on a face is blown out, so the white point was held back." + } } } } @@ -286,6 +340,43 @@ pub fn derive_budget( budget } +/// What this frame can absorb, faces included. See [`FaceConditions`] for why +/// these rules only ever remove, and never call a face too dark. +pub fn derive_budget_with_faces( + scene: &SceneState, + capture: &CaptureConditions, + faces: &FaceConditions, + cfg: &GuardrailConfig, +) -> EditBudget { + let mut budget = derive_budget(scene, capture, cfg); + if faces + .across_face_stops + .is_some_and(|s| s >= cfg.face_hard_light_stops) + { + budget.clarity_budget = 0.0; + budget.contrast_budget = budget.contrast_budget.min(cfg.contrast_budget_face); + budget.reasons.push(GuardrailReason::FaceHardLight); + } + if faces + .against_surround_stops + .is_some_and(|s| s <= -cfg.face_below_surround_stops) + { + budget.contrast_budget = budget.contrast_budget.min(cfg.contrast_budget_face); + budget.reasons.push(GuardrailReason::FaceBelowSurroundings); + } + // The same split as highlights in general: raw usually holds something above + // the rendering, a JPEG never does. + if faces.blown.is_some_and(|b| b >= cfg.face_blown_share) { + budget.whites_budget = if capture.is_raw { + budget.whites_budget.min(cfg.contrast_budget_default * 0.5) + } else { + 0.0 + }; + budget.reasons.push(GuardrailReason::FaceBlown); + } + budget +} + /// The contrast-increasing moves in a recipe, summed. /// /// Only the increases count. A profile that pulls contrast down and raises @@ -504,4 +595,103 @@ mod tests { assert!(!reason.explanation().is_empty()); } } + + fn with_faces(faces: FaceConditions, is_raw: bool) -> EditBudget { + derive_budget_with_faces( + &scene(0.3, 5.0), + &CaptureConditions { iso: None, is_raw }, + &faces, + &cfg(), + ) + } + + #[test] + fn no_face_reading_leaves_the_budget_as_it_was() { + let plain = derive_budget(&scene(0.3, 5.0), &CaptureConditions::default(), &cfg()); + let budget = with_faces(FaceConditions::default(), false); + assert_eq!(budget.contrast_budget, plain.contrast_budget); + assert_eq!(budget.clarity_budget, plain.clarity_budget); + assert_eq!(budget.whites_budget, plain.whites_budget); + assert!(budget.reasons.is_empty()); + } + + #[test] + fn hard_light_across_a_face_takes_clarity_and_most_of_the_contrast() { + let budget = with_faces( + FaceConditions { + across_face_stops: Some(3.2), + ..Default::default() + }, + false, + ); + assert_eq!(budget.clarity_budget, 0.0); + assert_eq!(budget.contrast_budget, cfg().contrast_budget_face); + assert_eq!(budget.reasons, vec![GuardrailReason::FaceHardLight]); + + let soft = with_faces( + FaceConditions { + across_face_stops: Some(0.8), + ..Default::default() + }, + false, + ); + assert!(soft.reasons.is_empty()); + } + + /// This reading is as low on dark skin in bright surroundings as on a + /// backlit face, so it may only ever hold contrast back: never the shadows a + /// dim face could use, never the clarity. + #[test] + fn a_face_below_its_surroundings_only_holds_contrast_back() { + let plain = with_faces(FaceConditions::default(), false); + let budget = with_faces( + FaceConditions { + against_surround_stops: Some(-3.1), + ..Default::default() + }, + false, + ); + assert_eq!(budget.contrast_budget, cfg().contrast_budget_face); + assert_eq!(budget.shadow_lift_budget, plain.shadow_lift_budget); + assert_eq!(budget.clarity_budget, plain.clarity_budget); + assert_eq!(budget.reasons, vec![GuardrailReason::FaceBelowSurroundings]); + + let brighter = with_faces( + FaceConditions { + against_surround_stops: Some(1.5), + ..Default::default() + }, + false, + ); + assert!(brighter.reasons.is_empty()); + } + + #[test] + fn blown_skin_holds_the_white_point_back_by_what_the_source_can_recover() { + let blown = FaceConditions { + blown: Some(0.4), + ..Default::default() + }; + assert_eq!(with_faces(blown, false).whites_budget, 0.0); + assert!(with_faces(blown, true).whites_budget > 0.0); + assert_eq!( + with_faces(blown, false).reasons, + vec![GuardrailReason::FaceBlown] + ); + } + + /// A ceiling never rises: a flat frame's wider contrast still gives way. + #[test] + fn face_rules_only_ever_remove() { + let flat = derive_budget_with_faces( + &scene(0.1, 3.0), + &CaptureConditions::default(), + &FaceConditions { + across_face_stops: Some(3.0), + ..Default::default() + }, + &cfg(), + ); + assert_eq!(flat.contrast_budget, cfg().contrast_budget_face); + } } diff --git a/server-rs/crates/lrg-analysis/src/face_aggregate.rs b/server-rs/crates/lrg-analysis/src/face_aggregate.rs index b5b2a567..30e97e9d 100644 --- a/server-rs/crates/lrg-analysis/src/face_aggregate.rs +++ b/server-rs/crates/lrg-analysis/src/face_aggregate.rs @@ -46,6 +46,13 @@ pub struct FaceMetricsInput { /// How far from the lens the eyes point, in degrees, head turn included; /// `None` like `eye_closed`. pub gaze_off_lens: Option, + /// How far the mouth is open, `0..1`; `None` like `eye_closed`. + pub mouth_open: Option, + /// The light on the skin (`lrg_ml::face_state::FaceLight`); `None` like + /// `eye_closed`, or when no skin patch fell inside the image. + pub luminance: Option, + pub across_face_stops: Option, + pub blown: Option, } #[derive(Debug, Clone, PartialEq)] @@ -77,6 +84,15 @@ pub struct AggregatedFaceMetrics { pub cull_eye_squint: Option, /// The prominent face looking furthest from the lens, in degrees. pub cull_gaze_off_lens: Option, + /// The most open mouth among the prominent faces not smiling. + pub cull_mouth_open: Option, + /// The darkest prominent face. Only ever compared with this same frame's + /// siblings: skin ranges over several stops between people. + pub cull_face_luminance: Option, + /// The hardest light across a prominent face, in stops. + pub cull_face_across_stops: Option, + /// The largest share of a prominent face's skin that is blown out. + pub cull_face_blown: Option, } impl AggregatedFaceMetrics { @@ -98,6 +114,10 @@ impl AggregatedFaceMetrics { cull_smile: None, cull_eye_squint: None, cull_gaze_off_lens: None, + cull_mouth_open: None, + cull_face_luminance: None, + cull_face_across_stops: None, + cull_face_blown: None, } } } @@ -223,6 +243,19 @@ pub fn aggregate_face_culling_metrics( }) .filter_map(|f| f.gaze_off_lens) .reduce(f64::max); + let mouth_open_agg = prominent + .iter() + .map(|&i| &faces[i]) + .filter(|f| f.smile.is_some_and(|s| s < cfg.face_state_mouth_max_smile)) + .filter_map(|f| f.mouth_open.map(unit)) + .reduce(f64::max); + let luminance_agg = measured(|f| f.luminance).reduce(f64::min); + let blown_agg = measured(|f| f.blown).reduce(f64::max); + // Stops, so not through `measured`'s clamp to `0..1`. + let across_agg = prominent + .iter() + .filter_map(|&i| faces[i].across_face_stops) + .reduce(f64::max); let eye_state_agg = eye_closed_agg.map(|v| { if v >= cfg.face_state_closed_threshold { "closed" @@ -262,6 +295,10 @@ pub fn aggregate_face_culling_metrics( cull_smile: smile_agg.map(round4), cull_eye_squint: eye_squint_agg.map(round4), cull_gaze_off_lens: gaze_off_lens_agg.map(round4), + cull_mouth_open: mouth_open_agg.map(round4), + cull_face_luminance: luminance_agg.map(round4), + cull_face_across_stops: across_agg.map(round4), + cull_face_blown: blown_agg.map(round4), } } @@ -297,6 +334,10 @@ mod tests { smile: Some(0.9), eye_squint: Some(0.4), gaze_off_lens: Some(4.0), + mouth_open: Some(0.02), + luminance: Some(0.3), + across_face_stops: Some(0.4), + blown: Some(0.0), }]; let m = aggregate_face_culling_metrics(&faces, &fm()); assert_eq!(m.cull_face_count, 1); @@ -478,6 +519,37 @@ mod tests { ); } + /// A laugh is open too, and not what the mouth reading is after. + #[test] + fn an_open_mouth_only_counts_on_a_face_that_is_not_smiling() { + let face = |open, smile| FaceMetricsInput { + area_ratio: 0.2, + mouth_open: Some(open), + smile: Some(smile), + ..Default::default() + }; + let laugh = aggregate_face_culling_metrics(&[face(0.6, 0.8)], &fm()); + assert_eq!(laugh.cull_mouth_open, None); + let word = aggregate_face_culling_metrics(&[face(0.3, 0.1)], &fm()); + assert_eq!(word.cull_mouth_open, Some(0.3)); + } + + #[test] + fn the_light_takes_the_worst_prominent_face() { + let face = |luminance, across, blown| FaceMetricsInput { + area_ratio: 0.2, + luminance: Some(luminance), + across_face_stops: Some(across), + blown: Some(blown), + ..Default::default() + }; + let group = [face(0.4, 0.5, 0.0), face(0.2, 3.1, 0.3)]; + let m = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(m.cull_face_luminance, Some(0.2)); + assert_eq!(m.cull_face_across_stops, Some(3.1)); + assert_eq!(m.cull_face_blown, Some(0.3)); + } + #[test] fn eyes_that_are_shut_do_not_look_away() { let face = |closed, off| FaceMetricsInput { diff --git a/server-rs/crates/lrg-analysis/src/face_expression.rs b/server-rs/crates/lrg-analysis/src/face_expression.rs new file mode 100644 index 00000000..23aa85be --- /dev/null +++ b/server-rs/crates/lrg-analysis/src/face_expression.rs @@ -0,0 +1,368 @@ +//! What the stored face state says about a face or a photo, for the workflows +//! that ask a person's question of it: "her with the eyes open", "the best +//! portrait of him", "everyone smiling at the camera". +//! +//! Culling compares frames of one burst and can lean on the group. These +//! questions are asked of a whole catalog, so there is no group to lean on and +//! each answer has to hold alone. That is why a reading the model did not make +//! is [`Verdict::Unknown`] and not a no: a photo indexed before the model was on +//! disk is a data problem the caller reports, where a frown is simply not a +//! smile. + +use serde_json::{Map, Value}; + +use crate::grouping::face_state_is_current; + +/// `mouthSmile` from here up reads as smiling. In portraits judged by eye, 0.35 to +/// 0.5 is at most the hint of one, and from 0.5 most faces show a smile, closed +/// mouth included. A snarl can reach it too. +pub const SMILING_FROM: f64 = 0.5; + +/// Eyes this near the lens read as looking at the camera, head turn included. +/// Portraits have a median of 9°; a high-angle selfie reads 15–20° for looking +/// straight at the lens, because `eyeLookUp` barely moves. `looking_away` starts +/// further out (25°), so a face between the two is claimed by neither. +pub const LOOKING_AT_CAMERA_WITHIN_DEGREES: f64 = 20.0; + +/// What [`portrait_score`] charges a row with no reading. +const UNREAD_COST: f64 = 0.3; + +/// Answer to one question about a face or a photo. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Verdict { + Yes, + No, + /// The model made no such reading, so the question cannot be answered. + Unknown, +} + +impl Verdict { + fn and(self, other: Verdict) -> Verdict { + match (self, other) { + (Verdict::No, _) | (_, Verdict::No) => Verdict::No, + (Verdict::Unknown, _) | (_, Verdict::Unknown) => Verdict::Unknown, + _ => Verdict::Yes, + } + } + + fn of(reading: Option) -> Verdict { + match reading { + Some(true) => Verdict::Yes, + Some(false) => Verdict::No, + None => Verdict::Unknown, + } + } +} + +/// What a caller asks of a face or a photo. Empty asks nothing. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ExpressionFilter { + pub eyes_open: bool, + pub smiling: bool, + pub looking_at_camera: bool, +} + +impl ExpressionFilter { + pub fn is_empty(&self) -> bool { + !(self.eyes_open || self.smiling || self.looking_at_camera) + } + + /// In plain words, for a set's label or a message: "eyes open and smiling". + pub fn describe(&self) -> String { + let mut parts = Vec::new(); + if self.eyes_open { + parts.push("eyes open"); + } + if self.smiling { + parts.push("smiling"); + } + if self.looking_at_camera { + parts.push("looking at the camera"); + } + parts.join(" and ") + } + + fn judge( + &self, + eye_state: Option<&str>, + smile: Option, + gaze_off_lens: Option, + ) -> Verdict { + // "unknown" is the model saying it cannot tell, the same as no reading. + let eye_state = eye_state.filter(|s| *s != "unknown"); + let mut verdict = Verdict::Yes; + if self.eyes_open { + verdict = verdict.and(Verdict::of(eye_state.map(|s| s == "open"))); + } + if self.smiling { + verdict = verdict.and(Verdict::of(smile.map(|s| s >= SMILING_FROM))); + } + if self.looking_at_camera { + // Shut eyes look nowhere, whatever `eyeLook*` says, and that is an + // answer rather than a missing reading. + let looking = if eye_state == Some("closed") { + Some(false) + } else { + gaze_off_lens.map(|g| g <= LOOKING_AT_CAMERA_WITHIN_DEGREES) + }; + verdict = verdict.and(Verdict::of(looking)); + } + verdict + } + + /// Judges one row of the face table. + pub fn judge_face(&self, face: &Map) -> Verdict { + self.judge( + face.get("face_eye_state").and_then(Value::as_str), + face.get("face_smile").and_then(Value::as_f64), + face.get("face_gaze_off_lens").and_then(Value::as_f64), + ) + } + + /// Judges a photo by its stored aggregates, which speak for the people who + /// matter in the frame together: everyone's eyes open, everyone smiling. + pub fn judge_photo(&self, photo: &Map) -> Verdict { + // Measured, and no faces: not a smile, and nothing the user could fix. + if photo.get("cull_face_count").and_then(Value::as_i64) == Some(0) { + return Verdict::No; + } + if !face_state_is_current(photo) { + return Verdict::Unknown; + } + self.judge( + photo.get("cull_eye_state").and_then(Value::as_str), + photo.get("cull_smile").and_then(Value::as_f64), + photo.get("cull_gaze_off_lens").and_then(Value::as_f64), + ) + } +} + +/// How good a portrait of its person a face row makes, higher better: for the +/// picture on a person's card, and for "the best photos of her". +/// +/// Starts from what the detector and the sharpness measure say, which is all a +/// row from before the face state has. Then a shut eye costs more than anything +/// else can earn: a person's card showing them blinking is the failure this +/// exists for. A row with no reading at all pays a fixed price: nobody has seen +/// its eyes, so it should not outrank a portrait known to be good, and among such +/// rows the order stays what it always was. +pub fn portrait_score(face: &Map) -> f64 { + let number = |key: &str| face.get(key).and_then(Value::as_f64); + let mut score = + number("face_det_score").unwrap_or(0.0) + number("face_sharpness").unwrap_or(0.0); + match face.get("face_eye_state").and_then(Value::as_str) { + Some("closed") => score -= 1.0, + Some("half") => score -= 0.4, + Some(_) => {} + None => score -= UNREAD_COST, + } + if let Some(yaw) = number("face_yaw") { + score -= 0.5 * (yaw.abs() / 60.0).clamp(0.0, 1.0); + } + if let Some(gaze) = number("face_gaze_off_lens") { + score -= 0.3 * (gaze / 45.0).clamp(0.0, 1.0); + } + if let Some(smile) = number("face_smile") { + score += 0.2 * smile.clamp(0.0, 1.0); + } + score +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::grouping::{FACE_STATE_VERSION, FACE_STATE_VERSION_KEY}; + use serde_json::json; + + fn row(value: Value) -> Map { + value.as_object().unwrap().clone() + } + + fn filter(eyes_open: bool, smiling: bool, looking_at_camera: bool) -> ExpressionFilter { + ExpressionFilter { + eyes_open, + smiling, + looking_at_camera, + } + } + + #[test] + fn an_empty_filter_asks_nothing_and_passes_anything() { + assert!(ExpressionFilter::default().is_empty()); + assert_eq!( + ExpressionFilter::default().judge_face(&row(json!({}))), + Verdict::Yes + ); + } + + #[test] + fn a_face_answers_each_question_from_its_own_reading() { + let face = row(json!({ + "face_eye_state": "open", + "face_smile": 0.8, + "face_gaze_off_lens": 6.0, + })); + assert_eq!(filter(true, true, true).judge_face(&face), Verdict::Yes); + + let frowning = row(json!({"face_eye_state": "open", "face_smile": 0.1})); + assert_eq!( + filter(false, true, false).judge_face(&frowning), + Verdict::No + ); + + let blinking = row(json!({"face_eye_state": "closed"})); + assert_eq!( + filter(true, false, false).judge_face(&blinking), + Verdict::No + ); + + let away = row(json!({"face_gaze_off_lens": 38.0})); + assert_eq!(filter(false, false, true).judge_face(&away), Verdict::No); + } + + /// A photo that could not be judged is a data problem the caller reports; + /// one judged and found wanting is simply not a match. + #[test] + fn a_missing_reading_is_unknown_and_a_no_outranks_it() { + let empty = row(json!({})); + assert_eq!( + filter(true, false, false).judge_face(&empty), + Verdict::Unknown + ); + let unsure = row(json!({"face_eye_state": "unknown"})); + assert_eq!( + filter(true, false, false).judge_face(&unsure), + Verdict::Unknown + ); + // Shut eyes settle it, whatever else is missing. + let shut = row(json!({"face_eye_state": "closed"})); + assert_eq!(filter(true, true, false).judge_face(&shut), Verdict::No); + } + + #[test] + fn a_photo_speaks_for_everyone_in_it() { + let current = |extra: Value| { + let mut m = row(extra); + m.insert(FACE_STATE_VERSION_KEY.into(), json!(FACE_STATE_VERSION)); + m.insert("cull_face_count".into(), json!(2)); + m + }; + let group = current(json!({ + "cull_eye_state": "open", + "cull_smile": 0.7, + "cull_gaze_off_lens": 12.0, + })); + assert_eq!(filter(true, true, true).judge_photo(&group), Verdict::Yes); + let one_blink = current(json!({"cull_eye_state": "closed"})); + assert_eq!( + filter(true, false, false).judge_photo(&one_blink), + Verdict::No + ); + } + + #[test] + fn a_photo_without_faces_is_no_and_one_not_analysed_is_unknown() { + let landscape = row(json!({"cull_face_count": 0})); + assert_eq!( + filter(false, true, false).judge_photo(&landscape), + Verdict::No + ); + + let never_prepared = row(json!({})); + assert_eq!( + filter(false, true, false).judge_photo(&never_prepared), + Verdict::Unknown + ); + + // An older analysis means something else by the same fields. + let stale = row(json!({ + "cull_face_count": 1, + "cull_smile": 0.9, + FACE_STATE_VERSION_KEY: FACE_STATE_VERSION - 1, + })); + assert_eq!( + filter(false, true, false).judge_photo(&stale), + Verdict::Unknown + ); + } + + /// Shut eyes look nowhere: a plain no, not a reading to go and fetch, and not + /// a yes when `eyeLook*` happens to read near the lens. + #[test] + fn shut_eyes_are_not_looking_at_the_camera() { + let face = row(json!({"face_eye_state": "closed", "face_gaze_off_lens": 4.0})); + assert_eq!(filter(false, false, true).judge_face(&face), Verdict::No); + let photo = row(json!({ + "cull_face_count": 1, + "cull_eye_state": "closed", + FACE_STATE_VERSION_KEY: FACE_STATE_VERSION, + })); + assert_eq!(filter(false, false, true).judge_photo(&photo), Verdict::No); + } + + #[test] + fn the_filter_says_what_it_asks() { + assert_eq!( + filter(true, true, false).describe(), + "eyes open and smiling" + ); + assert_eq!( + filter(false, false, true).describe(), + "looking at the camera" + ); + } + + /// The card showing someone blinking is what this exists to prevent: a + /// sharper, better detected frame with the eyes shut still loses. + #[test] + fn a_sharp_blink_loses_to_a_softer_frame_with_the_eyes_open() { + let blink = row(json!({ + "face_det_score": 0.98, + "face_sharpness": 0.9, + "face_eye_state": "closed", + "face_yaw": 2.0, + "face_gaze_off_lens": 5.0, + })); + let open = row(json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_yaw": 8.0, + "face_gaze_off_lens": 8.0, + })); + assert!(portrait_score(&open) > portrait_score(&blink)); + } + + #[test] + fn a_frontal_face_beats_a_turned_one_and_a_smile_breaks_the_tie() { + let base = json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_gaze_off_lens": 8.0, + }); + let at = |key: &str, value: f64| { + let mut m = row(base.clone()); + m.insert(key.into(), json!(value)); + m + }; + assert!(portrait_score(&at("face_yaw", 3.0)) > portrait_score(&at("face_yaw", 45.0))); + assert!(portrait_score(&at("face_smile", 0.9)) > portrait_score(&at("face_smile", 0.1))); + } + + /// Nobody has seen its eyes: it yields to a portrait known to be good, and + /// such rows keep the order among themselves that they always had. + #[test] + fn a_row_without_a_reading_yields_to_a_good_portrait_and_keeps_its_order() { + let old = |sharpness: f64| row(json!({"face_det_score": 0.9, "face_sharpness": sharpness})); + assert!(portrait_score(&old(0.8)) > portrait_score(&old(0.5))); + let known_good = row(json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_yaw": 5.0, + "face_gaze_off_lens": 8.0, + })); + assert!(portrait_score(&known_good) > portrait_score(&old(0.5))); + } +} diff --git a/server-rs/crates/lrg-analysis/src/grouping.rs b/server-rs/crates/lrg-analysis/src/grouping.rs index 02d7ddd9..04e3d0de 100644 --- a/server-rs/crates/lrg-analysis/src/grouping.rs +++ b/server-rs/crates/lrg-analysis/src/grouping.rs @@ -254,6 +254,13 @@ struct Scored<'a> { cull_smile: Option, cull_eye_squint: Option, cull_gaze_off_lens: Option, + /// Labels only, until a hand-culled shoot says what they are worth to the + /// ranking: `cull_mouth_open`, `cull_face_luminance`, + /// `cull_face_across_stops` (in stops) and `cull_face_blown`. + cull_mouth_open: Option, + cull_face_luminance: Option, + cull_face_across_stops: Option, + cull_face_blown: Option, /// The photo went through the face state pass (`cull_face_state_version` /// is stored), whether or not its faces could be judged. state_measured: bool, @@ -382,6 +389,10 @@ fn explanation_from_reason_codes(codes: &[String]) -> String { "eyes_half_closed" => "eyes half closed or squinting".to_string(), "looking_away" => "looking away from the camera".to_string(), "best_smile" => "best smile in group".to_string(), + "mouth_open" => "mouth open, caught mid-word — a sibling has it closed".to_string(), + "face_darker" => "the face is darker than in a sibling frame".to_string(), + "face_blown" => "the skin on the face is blown out".to_string(), + "face_hard_light" => "harder light across the face than in a sibling frame".to_string(), "near_duplicate_weaker" => "weaker duplicate or burst alternative".to_string(), other => other.replace('_', " "), } @@ -556,6 +567,7 @@ fn rank_group_records( ); // Degrees, so not through `read`'s clamp to `0..1`. let cull_gaze_off_lens = m.get("cull_gaze_off_lens").and_then(Value::as_f64); + let cull_face_across_stops = m.get("cull_face_across_stops").and_then(Value::as_f64); let state_measured = face_state_is_current(m); // Injected by the API layer only when the preset asks the gate // question, so an absent score never lifts a blink. @@ -592,6 +604,10 @@ fn rank_group_records( cull_smile, cull_eye_squint, cull_gaze_off_lens, + cull_mouth_open: read("cull_mouth_open"), + cull_face_luminance: read("cull_face_luminance"), + cull_face_across_stops, + cull_face_blown: read("cull_face_blown"), state_measured, half_closed: false, hard_defect: false, @@ -692,6 +708,20 @@ fn rank_group_records( .filter(|s| s.cull_face_count > 0) .all(|s| s.cull_smile.is_some()); let best_smile = smile_values.iter().cloned().reduce(f64::max); + // The group's best face readings for the labels below, which only ever say + // that one frame is worse than a sibling. + let least_open_mouth = scored + .iter() + .filter_map(|s| s.cull_mouth_open) + .reduce(f64::min); + let brightest_face = scored + .iter() + .filter_map(|s| s.cull_face_luminance) + .reduce(f64::max); + let softest_light = scored + .iter() + .filter_map(|s| s.cull_face_across_stops) + .reduce(f64::min); // The group's best face readings, for the relative defect rules: a proxy // that reads "closed" on every frame of a burst says nothing about any one @@ -1110,6 +1140,39 @@ fn rank_group_records( { reason_codes.push("best_smile".to_string()); } + if rk.reason_mouth_open_threshold > 0.0 + && !item.eyes_closed_intended + && item.cull_mouth_open.is_some_and(|open| { + open >= rk.reason_mouth_open_threshold + && least_open_mouth.is_some_and(|l| open >= l + rk.reason_mouth_open_delta) + }) + { + reason_codes.push("mouth_open".to_string()); + } + if rk.reason_face_darker_stops > 0.0 + && item.cull_face_luminance.is_some_and(|l| { + brightest_face.is_some_and(|b| { + (b.max(1e-4) / l.max(1e-4)).log2() >= rk.reason_face_darker_stops + }) + }) + { + reason_codes.push("face_darker".to_string()); + } + if rk.reason_face_blown_threshold > 0.0 + && item + .cull_face_blown + .is_some_and(|b| b >= rk.reason_face_blown_threshold) + { + reason_codes.push("face_blown".to_string()); + } + if rk.reason_face_hard_light_stops > 0.0 + && item.cull_face_across_stops.is_some_and(|a| { + a >= rk.reason_face_hard_light_stops + && softest_light.is_some_and(|s| a >= s + rk.reason_face_hard_light_delta) + }) + { + reason_codes.push("face_hard_light".to_string()); + } } // Explains the pick a technical reading would not have made. Emitted // for the winner only, and only when the moment is what carried it — a @@ -1254,6 +1317,10 @@ fn rank_group_records( ("smile", item.cull_smile), ("eye_squint", item.cull_eye_squint), ("gaze_off_lens", item.cull_gaze_off_lens), + ("mouth_open", item.cull_mouth_open), + ("face_luminance", item.cull_face_luminance), + ("face_across_stops", item.cull_face_across_stops), + ("face_blown", item.cull_face_blown), ] { if let Some(v) = value { metrics.insert(key.into(), Value::from(round4(v))); @@ -2706,6 +2773,120 @@ mod tests { .contains(&"eyes_open_best".to_string())); } + fn frame_with(id: &str, t: f64, readings: &[(&str, f64)]) -> GroupingInput { + let mut r = state_frame(id, t, Some(0.04), Some(0.2), true); + for (key, value) in readings { + r.metadata.insert((*key).into(), Value::from(*value)); + } + r + } + + #[test] + fn an_open_mouth_is_named_only_against_a_sibling_and_where_the_preset_asks() { + let records = || { + vec![ + frame_with("closed", 1000.0, &[("cull_mouth_open", 0.02)]), + frame_with("word", 1000.5, &[("cull_mouth_open", 0.4)]), + ] + }; + let groups = cull(records(), "event"); + let word = photo(&groups[0], "word"); + assert!(word.reason_codes.contains(&"mouth_open".to_string())); + // A label, never a defect. + assert_eq!(word.metrics["clear_defect"], false); + let closed = photo(&groups[0], "closed"); + assert!(!closed.reason_codes.contains(&"mouth_open".to_string())); + + let groups = cull(records(), "default"); + let word = photo(&groups[0], "word"); + assert!(!word.reason_codes.contains(&"mouth_open".to_string())); + + // Speaking all the way through a burst says nothing about any one frame. + let groups = cull( + vec![ + frame_with("a", 1000.0, &[("cull_mouth_open", 0.4)]), + frame_with("b", 1000.5, &[("cull_mouth_open", 0.45)]), + ], + "event", + ); + for id in ["a", "b"] { + assert!(!photo(&groups[0], id) + .reason_codes + .contains(&"mouth_open".to_string())); + } + } + + /// Dark skin is not an underexposed face: the reading only ever compares a + /// frame with its siblings, so a burst that is uniformly dark says nothing. + #[test] + fn a_dark_face_is_named_only_against_a_brighter_sibling() { + let groups = cull( + vec![ + frame_with("even1", 1000.0, &[("cull_face_luminance", 0.05)]), + frame_with("even2", 1000.5, &[("cull_face_luminance", 0.052)]), + ], + "event", + ); + for id in ["even1", "even2"] { + assert!(!photo(&groups[0], id) + .reason_codes + .contains(&"face_darker".to_string())); + } + let groups = cull( + vec![ + frame_with("bright", 1000.0, &[("cull_face_luminance", 0.2)]), + frame_with("dark", 1000.5, &[("cull_face_luminance", 0.08)]), + ], + "event", + ); + let dark = photo(&groups[0], "dark"); + assert!(dark.reason_codes.contains(&"face_darker".to_string())); + assert_eq!(dark.metrics["clear_defect"], false); + } + + #[test] + fn blown_skin_is_named_even_on_a_single() { + let groups = cull( + vec![frame_with("hot", 1000.0, &[("cull_face_blown", 0.3)])], + "default", + ); + assert!(photo(&groups[0], "hot") + .reason_codes + .contains(&"face_blown".to_string())); + let groups = cull( + vec![frame_with("mask", 1000.0, &[("cull_face_blown", 0.08)])], + "default", + ); + assert!(!photo(&groups[0], "mask") + .reason_codes + .contains(&"face_blown".to_string())); + } + + #[test] + fn hard_light_is_named_against_a_softer_sibling_in_stops() { + let groups = cull( + vec![ + frame_with("soft", 1000.0, &[("cull_face_across_stops", 0.5)]), + frame_with("hard", 1000.5, &[("cull_face_across_stops", 3.5)]), + ], + "default", + ); + let hard = photo(&groups[0], "hard"); + assert!(hard.reason_codes.contains(&"face_hard_light".to_string())); + assert_eq!(hard.metrics["face_across_stops"], 3.5); + let groups = cull( + vec![frame_with( + "alone", + 1000.0, + &[("cull_face_across_stops", 3.5)], + )], + "default", + ); + assert!(!photo(&groups[0], "alone") + .reason_codes + .contains(&"face_hard_light".to_string())); + } + /// Eyes narrowed against the sun with the lids reading open: the label, and /// still no defect. #[test] diff --git a/server-rs/crates/lrg-analysis/src/lib.rs b/server-rs/crates/lrg-analysis/src/lib.rs index a571ec0b..7533eb14 100644 --- a/server-rs/crates/lrg-analysis/src/lib.rs +++ b/server-rs/crates/lrg-analysis/src/lib.rs @@ -8,6 +8,7 @@ pub mod edit_guardrails; pub mod eval; pub mod face_aggregate; pub mod face_cluster; +pub mod face_expression; pub mod grouping; pub mod keywords; pub mod persons; diff --git a/server-rs/crates/lrg-api/src/edit_budget.rs b/server-rs/crates/lrg-api/src/edit_budget.rs index 302b4e16..8b358f56 100644 --- a/server-rs/crates/lrg-api/src/edit_budget.rs +++ b/server-rs/crates/lrg-api/src/edit_budget.rs @@ -16,11 +16,18 @@ //! which is what a harsh frame actually wants — is never scaled. use lrg_analysis::edit_guardrails::{ - derive_budget, fit_contrast_moves, CaptureConditions, EditBudget, GuardrailConfig, + derive_budget_with_faces, fit_contrast_moves, CaptureConditions, EditBudget, FaceConditions, + GuardrailConfig, }; +use lrg_imaging::cull_config::FaceMetricsConfig; use lrg_imaging::scene::{scene_state, SceneState, SceneStateConfig}; +use lrg_ml::face_state::{EyeState, FaceGeometry, FaceStateModel}; +use lrg_ml::faces::{FaceModel, FacePass}; use serde_json::{json, Map, Value}; +/// The most faces of one frame the guardrails read, largest first. +const MAX_FACES_READ: usize = 12; + /// The plain recipe fields that add contrast, in the order /// [`fit_contrast_moves`] sees them. The tone curve is budgeted alongside them /// but is not a single field, so it rides in a fourth slot as a derived @@ -161,28 +168,116 @@ pub fn apply_budget(recipe: &mut Value, budget: &EditBudget) -> Vec { .collect() } +/// The models that read faces, when both are on this computer. +pub struct FaceReaders<'a> { + pub faces: &'a FaceModel, + pub state: &'a FaceStateModel, +} + +impl<'a> FaceReaders<'a> { + /// `None` without the models. The edit goes on without the face rules and + /// without a word: the user did not ask for them, an edit does not need + /// them, and the settings already offer the download where it does matter. + pub fn when_downloaded( + faces: &'a FaceModel, + state: &'a FaceStateModel, + ) -> Option> { + (faces.is_cached() && state.is_cached()).then_some(FaceReaders { faces, state }) + } + + /// What the faces that matter in the frame show. A failure here leaves the + /// recipe under the frame's other rules, and the caller says so. + fn read(&self, pixels: &[u8], width: usize, height: usize) -> Result { + let cfg = FaceMetricsConfig::defaults(); + let detected = self + .faces + .detect_faces(pixels, width, height, &cfg, FacePass::QualityOnly) + .map_err(|e| e.to_string())?; + let largest = detected.iter().map(|f| f.area_ratio).fold(0.0f64, f64::max); + let cutoff = largest * cfg.prominent_face_area_fraction; + let mut prominent: Vec<_> = detected.iter().filter(|f| f.area_ratio >= cutoff).collect(); + prominent.sort_by(|a, b| b.area_ratio.total_cmp(&a.area_ratio)); + prominent.truncate(MAX_FACES_READ); + let geometry: Vec = prominent + .iter() + .map(|f| FaceGeometry { + bbox: f.bbox.map(|v| v as f64), + kps: f.kps, + }) + .collect(); + let states = self + .state + .analyze(pixels, width, height, &geometry, &cfg) + .map_err(|e| e.to_string())?; + let lights: Vec<_> = states + .iter() + .flatten() + .filter(|s| s.eye_state != EyeState::Unknown) + .filter_map(|s| s.light) + .collect(); + Ok(FaceConditions { + across_face_stops: lights + .iter() + .filter_map(|l| l.across_face_stops) + .reduce(f64::max), + against_surround_stops: lights + .iter() + .filter_map(|l| l.against_surround_stops) + .reduce(f64::min), + blown: lights.iter().map(|l| l.blown).reduce(f64::max), + }) + } +} + +/// What [`measure_and_apply`] did to a recipe. +#[derive(Debug, Default)] +pub struct Applied { + /// The reason codes for whatever fired, empty when nothing did. + pub reasons: Vec, + /// Something that went short and the user should hear about. + pub warning: Option, +} + /// Measure the frame and constrain the recipe to what it can absorb. /// -/// Returns the reason codes for whatever fired, empty when nothing did. A -/// decode failure is not an error: the recipe is returned unconstrained, which +/// A decode failure is not an error: the recipe is returned unconstrained, which /// is exactly how the edit path behaved before guardrails existed. pub fn measure_and_apply( recipe: &mut Value, image_bytes: &[u8], capture: &CaptureConditions, -) -> Vec { + readers: Option>, +) -> Applied { if !recipe_needs_budget(recipe) { - return Vec::new(); + return Applied::default(); } let Ok(decoded) = image::load_from_memory(image_bytes) else { log::warn!("Edit guardrails: could not decode the image; recipe left unconstrained."); - return Vec::new(); + return Applied::default(); }; let rgb = decoded.to_rgb8(); let (width, height) = (rgb.width() as usize, rgb.height() as usize); let pixels = rgb.into_raw(); let scene = scene_state(&pixels, width, height, &SceneStateConfig::defaults()); - apply_with_scene(recipe, &scene, capture) + let (faces, warning) = match readers.map(|r| r.read(&pixels, width, height)) { + Some(Err(e)) => { + log::warn!("Edit guardrails: faces not read ({e})."); + ( + FaceConditions::default(), + Some(format!( + "The faces in this photo could not be read ({e}), so the edit was not limited \ + by the light on them. If this keeps happening, run \"Download AI models\" in \ + Plug-in Manager again." + )), + ) + } + Some(Ok(faces)) => (faces, None), + None => (FaceConditions::default(), None), + }; + Applied { + reasons: apply_with_faces(recipe, &scene, capture, &faces), + warning, + } } /// [`measure_and_apply`] with the measurement already done. @@ -191,7 +286,17 @@ pub fn apply_with_scene( scene: &SceneState, capture: &CaptureConditions, ) -> Vec { - let budget = derive_budget(scene, capture, &GuardrailConfig::defaults()); + apply_with_faces(recipe, scene, capture, &FaceConditions::default()) +} + +/// [`apply_with_scene`] with what was read off the faces in the frame. +pub fn apply_with_faces( + recipe: &mut Value, + scene: &SceneState, + capture: &CaptureConditions, + faces: &FaceConditions, +) -> Vec { + let budget = derive_budget_with_faces(scene, capture, faces, &GuardrailConfig::defaults()); apply_budget(recipe, &budget) } @@ -246,6 +351,73 @@ mod tests { ); } + /// The canonical portrait case: an ordinary frame, a habitual `+20 clarity`, + /// and hard light across the face it would deepen. + #[test] + fn hard_light_on_a_face_takes_the_clarity_out_of_the_recipe() { + let mut recipe = recipe_with(json!({"contrast": 8.0, "clarity": 20.0})); + let faces = FaceConditions { + across_face_stops: Some(3.0), + ..Default::default() + }; + let reasons = apply_with_faces( + &mut recipe, + &ordinary(), + &CaptureConditions::default(), + &faces, + ); + let global = global_of(&recipe); + assert_eq!(global["clarity"].as_f64().unwrap(), 0.0); + assert!(global["contrast"].as_f64().unwrap() <= 10.0); + assert_eq!(reasons, vec!["face_hard_light".to_string()]); + } + + /// A frame without a face reading edits as it always did. + #[test] + fn no_faces_read_changes_nothing() { + let mut with = recipe_with(json!({"contrast": 15.0, "clarity": 10.0})); + let mut without = with.clone(); + apply_with_scene(&mut with, &ordinary(), &CaptureConditions::default()); + apply_with_faces( + &mut without, + &ordinary(), + &CaptureConditions::default(), + &FaceConditions::default(), + ); + assert_eq!(with, without); + } + + /// A failed read costs the edit its face rules, and the user hears about it: + /// the models are there, so they would otherwise assume the rules applied. + #[test] + fn faces_that_could_not_be_read_are_reported_and_the_edit_goes_on() { + let dir = tempfile::tempdir().unwrap(); + let path = |name: &str| dir.path().join(name); + let faces = FaceModel::new(lrg_ml::faces::FaceModelPaths { + det_onnx: path("det.onnx"), + rec_onnx: path("rec.onnx"), + auraface_onnx: path("aura.onnx"), + }); + let state = FaceStateModel::new(lrg_ml::face_state::FaceStatePaths { + mesh_onnx: path("mesh.onnx"), + blendshapes_onnx: path("blend.onnx"), + }); + let readers = FaceReaders { + faces: &faces, + state: &state, + }; + let mut recipe = recipe_with(json!({"contrast": 5.0})); + let applied = measure_and_apply( + &mut recipe, + &flat_soft_jpeg(), + &CaptureConditions::default(), + Some(readers), + ); + let warning = applied.warning.expect("the failed read is reported"); + assert!(warning.contains("Download AI models"), "{warning}"); + assert_eq!(global_of(&recipe)["contrast"].as_f64().unwrap(), 5.0); + } + /// A frame can be harshly lit without the recipe ever asking for contrast. /// Explaining that contrast "was not increased" there would be noise next /// to an edit that was returned exactly as generated. @@ -380,8 +552,13 @@ mod tests { // No decode is attempted, so deliberately invalid image bytes are fine. let mut recipe = recipe_with(json!({"temperature": 5600.0, "contrast": -10.0})); assert!(!recipe_needs_budget(&recipe)); - let reasons = - measure_and_apply(&mut recipe, b"not an image", &CaptureConditions::default()); + let reasons = measure_and_apply( + &mut recipe, + b"not an image", + &CaptureConditions::default(), + None, + ) + .reasons; assert!(reasons.is_empty()); assert_eq!(global_of(&recipe)["temperature"].as_f64().unwrap(), 5600.0); } @@ -441,7 +618,9 @@ mod tests { &mut recipe, &harshly_lit_jpeg(), &CaptureConditions::default(), - ); + None, + ) + .reasons; let g = global_of(&recipe); assert_eq!(g["contrast"].as_f64().unwrap(), 0.0, "reasons: {reasons:?}"); @@ -464,7 +643,9 @@ mod tests { &mut recipe, &flat_soft_jpeg(), &CaptureConditions::default(), - ); + None, + ) + .reasons; assert_eq!(recipe, before, "flat light should widen, never narrow"); assert!(reasons.is_empty(), "got {reasons:?}"); @@ -474,8 +655,13 @@ mod tests { fn an_undecodable_image_leaves_the_recipe_alone() { let mut recipe = recipe_with(json!({"contrast": 90.0})); let before = recipe.clone(); - let reasons = - measure_and_apply(&mut recipe, b"not an image", &CaptureConditions::default()); + let reasons = measure_and_apply( + &mut recipe, + b"not an image", + &CaptureConditions::default(), + None, + ) + .reasons; assert!(reasons.is_empty()); assert_eq!(recipe, before, "a decode failure must not change the edit"); } diff --git a/server-rs/crates/lrg-api/src/routes/edit.rs b/server-rs/crates/lrg-api/src/routes/edit.rs index 4b8fdd55..88d2d714 100644 --- a/server-rs/crates/lrg-api/src/routes/edit.rs +++ b/server-rs/crates/lrg-api/src/routes/edit.rs @@ -611,11 +611,19 @@ pub(crate) async fn generate_edit_recipe_for_photo( // switched off must not first be scaled and then discarded, which // would spend budget on a field that never reaches the photo and // shrink the ones that do. - response.guardrail_reasons = crate::edit_budget::measure_and_apply( + let applied = crate::edit_budget::measure_and_apply( &mut recipe, image_bytes, &capture_conditions(options, filename, image_bytes), + crate::edit_budget::FaceReaders::when_downloaded(&state.face, &state.face_state), ); + response.guardrail_reasons = applied.reasons; + if let Some(msg) = applied.warning { + response.warning = Some(match response.warning.take() { + Some(existing) => format!("{existing}\n{msg}"), + None => msg, + }); + } if !response.guardrail_reasons.is_empty() { log::info!( "Photo {photo_id}: edit constrained by the frame ({}).", @@ -813,13 +821,16 @@ pub(crate) fn success_payload( /// a newer backend should show nothing rather than a bare identifier. fn guardrail_explanations(codes: &[String]) -> Vec { use lrg_analysis::edit_guardrails::GuardrailReason::*; - const ALL: [lrg_analysis::edit_guardrails::GuardrailReason; 6] = [ + const ALL: [lrg_analysis::edit_guardrails::GuardrailReason; 9] = [ HardLightNoAddedContrast, NoTonalHeadroom, FlatLightContrastAllowed, HighlightsUnrecoverable, HighIsoShadowsLimited, ShadowsAlreadyClipped, + FaceHardLight, + FaceBelowSurroundings, + FaceBlown, ]; codes .iter() diff --git a/server-rs/crates/lrg-api/src/routes/faces.rs b/server-rs/crates/lrg-api/src/routes/faces.rs index 450e5fe8..3a4fff8c 100644 --- a/server-rs/crates/lrg-api/src/routes/faces.rs +++ b/server-rs/crates/lrg-api/src/routes/faces.rs @@ -31,6 +31,7 @@ use serde::Deserialize; use serde_json::{json, Map, Value}; use lrg_analysis::face_cluster::{self, ClusterParams, FaceInput, QualityGate}; +use lrg_analysis::face_expression::{portrait_score, ExpressionFilter, Verdict}; use lrg_analysis::{clustering, persons}; use lrg_store::{StoreRecord, FACE_TABLE}; @@ -320,6 +321,16 @@ fn number_at(meta: &Map, key: &str) -> f64 { meta.get(key).and_then(Value::as_f64).unwrap_or(0.0) } +/// Of two faces of one person on one photo, the answer that counts for the +/// photo: a yes settles it, and only when no face answers is it unknown. +fn best_verdict(a: Verdict, b: Verdict) -> Verdict { + match (a, b) { + (Verdict::Yes, _) | (_, Verdict::Yes) => Verdict::Yes, + (Verdict::Unknown, _) | (_, Verdict::Unknown) => Verdict::Unknown, + _ => Verdict::No, + } +} + /// The same, but keeping "the row does not have this" distinct from "the row /// says zero". The quality gate needs the difference: a face indexed before /// these measurements existed must not be judged as if every one of them came @@ -758,7 +769,7 @@ async fn list_persons(State(state): State>) -> Response { entry.manual_count += 1; } entry.photo_ids.insert(photo_of(meta).to_string()); - let quality = number_at(meta, "face_det_score") + number_at(meta, "face_sharpness"); + let quality = portrait_score(meta); if entry.thumbnail.is_none() || quality > entry.thumbnail_quality { if let Some(thumb) = meta .get("thumbnail") @@ -815,7 +826,13 @@ async fn person_thumbnail( let thumb = match store.scan_meta(FACE_TABLE).await { Ok(rows) => rows .into_iter() - .find(|(_, m)| person_of(m) == person_id) + .filter(|(_, m)| person_of(m) == person_id) + .filter(|(_, m)| { + m.get("thumbnail") + .and_then(Value::as_str) + .is_some_and(|t| !t.is_empty()) + }) + .max_by(|(_, a), (_, b)| portrait_score(a).total_cmp(&portrait_score(b))) .and_then(|(_, m)| { m.get("thumbnail") .and_then(Value::as_str) @@ -947,9 +964,24 @@ async fn confirm_person_faces(state: &AppState, person_id: &str) -> Result, + limit: Option, +} + async fn person_photos( State(state): State>, UrlPath(person_id): UrlPath, + Query(params): Query, ) -> Response { log::info!( "Get photos for person request received for person_id={}", @@ -972,26 +1004,64 @@ async fn person_photos( } }; let wants_unassigned = is_unassigned(&person_id); - let mut photo_ids: Vec = rows - .into_iter() - .filter(|(_, m)| { - if wants_unassigned { - is_unassigned(person_of(m)) - } else { - person_of(m) == person_id - } - }) - .map(|(_, m)| photo_of(&m).to_string()) - .filter(|id| !id.is_empty()) - .collect::>() + let filter = ExpressionFilter { + eyes_open: params.eyes_open, + smiling: params.smiling, + looking_at_camera: params.looking_at_camera, + }; + // A person can be on a photo twice (a mirror, a poster): the photo counts + // when any of those faces answers yes, and scores as the best of them. + let mut photos: HashMap = HashMap::new(); + for (_, m) in &rows { + let mine = if wants_unassigned { + is_unassigned(person_of(m)) + } else { + person_of(m) == person_id + }; + let photo = photo_of(m); + if !mine || photo.is_empty() { + continue; + } + let (verdict, score) = (filter.judge_face(m), portrait_score(m)); + let entry = photos.entry(photo.to_string()).or_insert((verdict, score)); + entry.0 = best_verdict(entry.0, verdict); + entry.1 = entry.1.max(score); + } + let not_judged = photos + .values() + .filter(|(v, _)| *v == Verdict::Unknown) + .count(); + let mut matching: Vec<(String, f64)> = photos .into_iter() + .filter(|(_, (v, _))| *v == Verdict::Yes) + .map(|(id, (_, score))| (id, score)) .collect(); - photo_ids.sort(); + if params.sort.as_deref() == Some("best") { + matching.sort_by(|a, b| b.1.total_cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + } else { + matching.sort_by(|a, b| a.0.cmp(&b.0)); + } + if let Some(limit) = params.limit { + matching.truncate(limit); + } + let photo_ids: Vec = matching.into_iter().map(|(id, _)| id).collect(); + let mut warnings = Vec::new(); + if not_judged > 0 { + warnings.push(format!( + "{not_judged} photo(s) of this person could not be judged for {} and were left out. A \ + face too small or turned too far cannot be read. One indexed before the eye and \ + expression model was on disk can: run 'Analyze & Index Photos' with face detection \ + on those photos, after \"Download AI models\" in Plug-in Manager.", + filter.describe() + )); + } Json(json!({ "status": "ok", "person_id": person_id, "photo_ids": photo_ids.clone(), "photo_uuids": photo_ids, + "not_judged": not_judged, + "warnings": warnings, })) .into_response() } @@ -1003,7 +1073,8 @@ struct PersonFacesQuery { limit: Option, offset: Option, /// `typical` (closest to the person's average face first), `outlier` (the - /// reverse — where wrong faces surface), or `quality`. + /// reverse — where wrong faces surface), `quality`, or `portrait` (the best + /// picture of them first: eyes open, facing the camera). sort: Option, } @@ -1115,6 +1186,9 @@ async fn person_faces( + 0.1 * (number_at(m, "face_area_ratio") / 0.12).clamp(0.0, 1.0) }; match sort.as_str() { + "portrait" => { + mine.sort_by(|(_, a), (_, b)| portrait_score(b).total_cmp(&portrait_score(a))) + } "quality" => mine.sort_by(|(_, a), (_, b)| { quality_score(b) .partial_cmp(&quality_score(a)) diff --git a/server-rs/crates/lrg-api/src/routes/index.rs b/server-rs/crates/lrg-api/src/routes/index.rs index 5b300487..d54daaa4 100644 --- a/server-rs/crates/lrg-api/src/routes/index.rs +++ b/server-rs/crates/lrg-api/src/routes/index.rs @@ -249,8 +249,19 @@ async fn check_unprocessed( || has_flag("alt_text") || has_flag("keywords"); let needs_metadata = compute_metadata && (regenerate_metadata || !has_any_metadata); - let needs_faces = - compute_faces && (regenerate_metadata || !faces_checked.contains(*pid)); + // A photo with faces whose readings are missing or from an older + // analysis goes through the face pass again once the model is on + // disk. Person assignments carry over, and this is how a catalog + // indexed earlier gets what People's filters ask about. + let needs_face_state = face_state_available + && m.get("cull_face_count") + .and_then(Value::as_i64) + .unwrap_or(0) + > 0 + && (!lrg_analysis::grouping::face_state_is_current(&m) + || has_flag("face_rows_behind")); + let needs_faces = compute_faces + && (regenerate_metadata || !faces_checked.contains(*pid) || needs_face_state); let needs_cull_phash = any_task && (regenerate_metadata || !has_flag("cull_phash")); // `cull_faces_present` is legitimately `false` for a photo with // no faces, so completion is "the key exists", not "it is diff --git a/server-rs/crates/lrg-api/src/routes/index_upload.rs b/server-rs/crates/lrg-api/src/routes/index_upload.rs index d0565688..765329bc 100644 --- a/server-rs/crates/lrg-api/src/routes/index_upload.rs +++ b/server-rs/crates/lrg-api/src/routes/index_upload.rs @@ -2031,10 +2031,18 @@ async fn finish_one( let focus_missing = options.face_pass == FacePass::QualityOnly && !main_metadata.contains_key(lrg_analysis::grouping::FOCUS_VERSION_KEY); // The same for the face state, once its model is on disk: a photo - // prepared before it was downloaded is analysed on the next cull pass. - let state_missing = options.face_pass == FacePass::QualityOnly + // prepared before it was downloaded is analysed on the next pass. A full + // pass only when the photo has faces, since it re-runs identity for them. + let has_faces = main_metadata + .get("cull_face_count") + .and_then(Value::as_i64) + .unwrap_or(0) + > 0; + let rows_behind = + options.face_pass == FacePass::Full && main_metadata.contains_key(FACE_ROWS_BEHIND_KEY); + let state_missing = (options.face_pass == FacePass::QualityOnly || has_faces) && state.face_state.is_cached() - && !lrg_analysis::grouping::face_state_is_current(&main_metadata); + && (rows_behind || !lrg_analysis::grouping::face_state_is_current(&main_metadata)); if options.regenerate_metadata || !already_checked || focus_missing || state_missing { // Decoded here rather than carried from phase 1: full RGB is by far // the largest per-photo allocation, and holding one per photo in @@ -2154,9 +2162,40 @@ async fn finish_one( smile: judged(f).map(|s| s.smile), eye_squint: judged(f).map(|s| s.eye_squint), gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), + mouth_open: judged(f).map(|s| s.mouth_open), + luminance: judged_light(f).map(|l| l.luminance), + across_face_stops: judged_light(f).and_then(|l| l.across_face_stops), + blown: judged_light(f).map(|l| l.blown), }) .collect(); - apply_face_aggregate(&mut main_metadata, &inputs, state_ran); + // Before the readings are stamped as current: a photo whose rows + // could not be saved must be asked for again, or People could + // never read it. + let mut stamped = state_ran; + if state_ran { + match merge_face_state_into_rows(store, photo_id, &faces).await { + Ok(0) => { + main_metadata.remove(FACE_ROWS_BEHIND_KEY); + } + // Nothing to tell the user yet: People says so when + // asked, and names the full pass this queues. + Ok(unmatched) => { + log::debug!( + "Photo {photo_id}: {unmatched} face row(s) match no face." + ); + main_metadata.insert(FACE_ROWS_BEHIND_KEY.into(), json!(true)); + } + Err(e) => { + log::warn!("Photo {photo_id}: face rows not updated: {e}"); + stamped = false; + warnings.push(format!( + "{filename}: eyes, smile and gaze could not be saved ({e}). \ + Choose 'Prepare now' in 'Cull Similar Photos' to try again." + )); + } + } + } + apply_face_aggregate(&mut main_metadata, &inputs, stamped); log::debug!( "Photo {photo_id}: cull pass scored {} face(s), no identity written.", faces.len() @@ -2265,35 +2304,7 @@ async fn finish_one( m.insert("face_focus".into(), json!(f.face_focus)); m.insert("face_eye_px".into(), json!(f.eye_px)); m.insert("face_focus_scale".into(), json!(f.focus_scale)); - // Null when the face state did not run for this - // face; `face_eye_state` is "unknown" when it ran - // and could not judge the eyes. - let st = f.state.as_ref(); - m.insert( - "face_eye_state".into(), - json!(st.map(|s| s.eye_state.as_str())), - ); - m.insert("face_eye_closed".into(), json!(st.map(|s| s.eye_closed))); - m.insert("face_smile".into(), json!(st.map(|s| s.smile))); - m.insert("face_eye_squint".into(), json!(st.map(|s| s.eye_squint))); - m.insert( - "face_gaze_off_lens".into(), - json!(st.map(|s| s.gaze_off_lens)), - ); - m.insert("face_yaw".into(), json!(st.map(|s| s.pose.yaw))); - m.insert("face_pitch".into(), json!(st.map(|s| s.pose.pitch))); - m.insert("face_roll".into(), json!(st.map(|s| s.pose.roll))); - // All 52, in `BLENDSHAPE_NAMES` order, to three - // places: a later signal can then read them - // without decoding every original again. - m.insert( - "face_blendshapes".into(), - json!(st.map(|s| s - .blendshapes - .iter() - .map(|v| (*v as f64 * 1000.0).round() / 1000.0) - .collect::>())), - ); + insert_face_state(&mut m, f.state.as_ref()); m.insert("face_model".into(), json!(face_model)); StoreRecord { id: format!("{photo_id}_{i}"), @@ -2323,6 +2334,11 @@ async fn finish_one( smile: judged(f).map(|s| s.smile), eye_squint: judged(f).map(|s| s.eye_squint), gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), + mouth_open: judged(f).map(|s| s.mouth_open), + luminance: judged_light(f).map(|l| l.luminance), + across_face_stops: judged_light(f) + .and_then(|l| l.across_face_stops), + blown: judged_light(f).map(|l| l.blown), }) .collect(); apply_face_aggregate(&mut main_metadata, &inputs, state_ran); @@ -2337,6 +2353,7 @@ async fn finish_one( // used to disappear on a plain re-index. main_metadata.insert("faces_checked".into(), json!(true)); main_metadata.insert("face_model".into(), json!(face_model)); + main_metadata.remove(FACE_ROWS_BEHIND_KEY); } Err(e) => { log::warn!("Face detection/indexing failed for {photo_id}: {e}"); @@ -2701,6 +2718,10 @@ fn carry_over_person_ids_by_box( /// The face state of a face, when it ran and could judge the eyes; a face it /// could not judge abstains from every eye, smile and gaze aggregate. +fn judged_light(face: &lrg_ml::faces::FaceResult) -> Option { + judged(face).and_then(|s| s.light) +} + fn judged(face: &lrg_ml::faces::FaceResult) -> Option<&lrg_ml::face_state::FaceState> { face.state .as_ref() @@ -2737,6 +2758,10 @@ fn apply_face_aggregate( ("cull_smile", agg.cull_smile), ("cull_eye_squint", agg.cull_eye_squint), ("cull_gaze_off_lens", agg.cull_gaze_off_lens), + ("cull_mouth_open", agg.cull_mouth_open), + ("cull_face_luminance", agg.cull_face_luminance), + ("cull_face_across_stops", agg.cull_face_across_stops), + ("cull_face_blown", agg.cull_face_blown), ] { match value { Some(v) => { @@ -2779,6 +2804,97 @@ fn apply_face_aggregate( } } +/// A face's readings from the face state model on its row. Null when the model +/// did not run for this face; `face_eye_state` is "unknown" when it ran and could +/// not judge the eyes. +fn insert_face_state(m: &mut Map, st: Option<&lrg_ml::face_state::FaceState>) { + m.insert( + "face_eye_state".into(), + json!(st.map(|s| s.eye_state.as_str())), + ); + m.insert("face_eye_closed".into(), json!(st.map(|s| s.eye_closed))); + m.insert("face_smile".into(), json!(st.map(|s| s.smile))); + m.insert("face_eye_squint".into(), json!(st.map(|s| s.eye_squint))); + m.insert("face_mouth_open".into(), json!(st.map(|s| s.mouth_open))); + let light = st.and_then(|s| s.light); + m.insert("face_luminance".into(), json!(light.map(|l| l.luminance))); + m.insert( + "face_against_surround_stops".into(), + json!(light.and_then(|l| l.against_surround_stops)), + ); + m.insert( + "face_across_stops".into(), + json!(light.and_then(|l| l.across_face_stops)), + ); + m.insert("face_blown".into(), json!(light.map(|l| l.blown))); + m.insert( + "face_gaze_off_lens".into(), + json!(st.map(|s| s.gaze_off_lens)), + ); + m.insert("face_yaw".into(), json!(st.map(|s| s.pose.yaw))); + m.insert("face_pitch".into(), json!(st.map(|s| s.pose.pitch))); + m.insert("face_roll".into(), json!(st.map(|s| s.pose.roll))); + // All 52, in `BLENDSHAPE_NAMES` order, to three places: a later signal can + // then read them without decoding every original again. + m.insert( + "face_blendshapes".into(), + json!(st.map(|s| s + .blendshapes + .iter() + .map(|v| (*v as f64 * 1000.0).round() / 1000.0) + .collect::>())), + ); +} + +/// Marks a photo whose face rows a quality-only pass could not bring up to date, +/// so the next full pass re-detects them. Its own readings are current, so nothing +/// else would ever ask for that. +const FACE_ROWS_BEHIND_KEY: &str = "face_rows_behind"; + +/// Brings a quality-only pass's readings to the rows a full pass wrote earlier, +/// each to the row whose box it landed on, and touches nothing else on them. A +/// catalog whose faces were indexed before the model was on disk gets its readings +/// this way, so People can ask about them without re-running identity. Returns how +/// many rows no detected face landed on: those keep no reading. +async fn merge_face_state_into_rows( + store: &lrg_store::Store, + photo_id: &str, + faces: &[lrg_ml::faces::FaceResult], +) -> Result { + let mut rows = store + .get_by_id_prefix(FACE_TABLE, &format!("{photo_id}_")) + .await + .map_err(|e| e.to_string())?; + let mut changed = Vec::new(); + let mut unmatched = 0; + for row in rows.iter_mut() { + let best = stored_bbox(row).and_then(|old_box| { + faces + .iter() + .map(|f| (box_iou(f.bbox, old_box), f)) + .filter(|(iou, _)| *iou >= FACE_CARRY_OVER_MIN_IOU) + .max_by(|a, b| a.0.total_cmp(&b.0)) + }); + match best { + None => unmatched += 1, + // A bystander too small for the model has a box and no reading, + // which a re-detection would not change. + Some((_, face)) if face.state.is_some() => { + insert_face_state(&mut row.metadata, face.state.as_ref()); + changed.push(row.clone()); + } + Some(_) => {} + } + } + if !changed.is_empty() { + store + .upsert(FACE_TABLE, &changed) + .await + .map_err(|e| e.to_string())?; + } + Ok(unmatched) +} + /// At most this many faces of one photo get the face state model, largest /// first. Only prominent faces count towards the photo's eye state anyway, and /// a group of forty would otherwise cost forty mesh passes. diff --git a/server-rs/crates/lrg-api/src/routes/search.rs b/server-rs/crates/lrg-api/src/routes/search.rs index ab572d33..e5694ce0 100644 --- a/server-rs/crates/lrg-api/src/routes/search.rs +++ b/server-rs/crates/lrg-api/src/routes/search.rs @@ -17,6 +17,7 @@ use axum::extract::{Query, State}; use axum::response::{IntoResponse, Json, Response}; use serde_json::{json, Map, Value}; +use lrg_analysis::face_expression::{ExpressionFilter, Verdict}; use lrg_analysis::relevance_filter::{ clamp_max_results, clamp_strictness, smart_filter_by_relevance, ScoredResult, }; @@ -147,11 +148,12 @@ pub const AESTHETIC_RANGE: (f32, f32) = struct Restrictions { min_aesthetic: Option, genres: Option>, + expression: ExpressionFilter, } impl Restrictions { fn is_empty(&self) -> bool { - self.min_aesthetic.is_none() && self.genres.is_none() + self.min_aesthetic.is_none() && self.genres.is_none() && self.expression.is_empty() } } @@ -161,6 +163,7 @@ impl Restrictions { struct Dropped { no_score: usize, no_genre: usize, + no_face_state: usize, } /// Applies `restrictions` to `hits`, counting the photos it had to drop for want @@ -201,6 +204,16 @@ fn restrict( } } } + if !restrictions.expression.is_empty() { + match restrictions.expression.judge_photo(m) { + Verdict::Yes => {} + Verdict::No => return false, + Verdict::Unknown => { + dropped.no_face_state += 1; + return false; + } + } + } true }) .collect(); @@ -326,6 +339,7 @@ async fn search( .and_then(Value::as_f64) .unwrap_or(0.5) .clamp(0.0, 1.0); + let wants = |key: &str| body.get(key).and_then(Value::as_bool).unwrap_or(false); let restrictions = Restrictions { min_aesthetic: body .get("min_aesthetic") @@ -341,6 +355,11 @@ async fn search( .collect::>() }) .filter(|set| !set.is_empty()), + expression: ExpressionFilter { + eyes_open: wants("eyes_open"), + smiling: wants("smiling"), + looking_at_camera: wants("looking_at_camera"), + }, }; let photo_ids_to_search: Option> = body @@ -602,6 +621,17 @@ async fn search( dropped.no_genre )); } + if dropped.no_face_state > 0 { + warnings.push(format!( + "{} matching photo(s) could not be judged for {} and were left out. A face too small \ + or turned too far cannot be read. One analysed before the eye and expression model \ + was on disk can: choose 'Prepare now' in 'Cull Similar Photos' for those photos, or \ + run 'Analyze & Index Photos' with face detection, after \"Download AI models\" in \ + Plug-in Manager.", + dropped.no_face_state, + restrictions.expression.describe() + )); + } let mut hits = order(hits, sort, aesthetic_weight, &rows); if sort != SortMode::Relevance && term.is_some() { @@ -807,6 +837,7 @@ mod tests { let restrictions = Restrictions { min_aesthetic: Some(6.0), genres: Some(["macro".to_string()].into_iter().collect()), + ..Default::default() }; let (kept, dropped) = restrict(hits, &restrictions, &rows); assert_eq!(ids(&kept), ["ok"]); diff --git a/server-rs/crates/lrg-api/src/routes/style_edit.rs b/server-rs/crates/lrg-api/src/routes/style_edit.rs index 3f3f1134..f34effc1 100644 --- a/server-rs/crates/lrg-api/src/routes/style_edit.rs +++ b/server-rs/crates/lrg-api/src/routes/style_edit.rs @@ -423,11 +423,13 @@ async fn style_edit(State(state): State>, mut multipart: Multipart it is on the raw Kelvin scale and this photo is not raw." ); } - let guardrail_reasons = crate::edit_budget::measure_and_apply( + let applied = crate::edit_budget::measure_and_apply( &mut styled_recipe, &image_bytes, &crate::routes::edit::capture_conditions(&options, filename.as_deref(), &image_bytes), + crate::edit_budget::FaceReaders::when_downloaded(&state.face, &state.face_state), ); + let guardrail_reasons = applied.reasons; if !guardrail_reasons.is_empty() { log::info!( "Photo {photo_id}: style-engine edit constrained by the frame ({}).", @@ -446,11 +448,16 @@ async fn style_edit(State(state): State>, mut multipart: Multipart { log::error!("Failed to persist style_edit recipe for {photo_id}: {e}"); } + let warning = match (result.warning.as_deref(), applied.warning) { + (Some(a), Some(b)) => Some(format!("{a}\n{b}")), + (Some(a), None) => Some(a.to_string()), + (None, b) => b, + }; let mut payload = success_payload( &photo_id, &styled_recipe, &options, - result.warning.as_deref(), + warning.as_deref(), &guardrail_reasons, ); payload["engine"] = json!("style"); diff --git a/server-rs/crates/lrg-api/src/ui/people.html b/server-rs/crates/lrg-api/src/ui/people.html index be01c866..57fa42b3 100644 --- a/server-rs/crates/lrg-api/src/ui/people.html +++ b/server-rs/crates/lrg-api/src/ui/people.html @@ -355,6 +355,25 @@

People

+ + Only + + + + + @@ -376,6 +395,7 @@

Person

+ @@ -1569,6 +1589,10 @@

Person

person_ids: ids, person_names: names, match_mode: ids.length > 1 ? el("match-mode").value : "union", + eyes_open: el("only-eyes-open").checked, + smiling: el("only-smiling").checked, + looking_at_camera: el("only-looking").checked, + best: Number(el("photo-limit").value) || null, }); if (data.plugin_connected) { state.textContent = "Sent — switch to Lightroom."; diff --git a/server-rs/crates/lrg-api/src/ui/search.html b/server-rs/crates/lrg-api/src/ui/search.html index 2f3370ad..490a8e9c 100644 --- a/server-rs/crates/lrg-api/src/ui/search.html +++ b/server-rs/crates/lrg-api/src/ui/search.html @@ -176,6 +176,12 @@

Search

>Genre + + Faces + + + +