diff --git a/docs/wiki/Dev-Backend-API.md b/docs/wiki/Dev-Backend-API.md index 4a692cf4..0a2be2cd 100644 --- a/docs/wiki/Dev-Backend-API.md +++ b/docs/wiki/Dev-Backend-API.md @@ -266,13 +266,23 @@ focus was measured on the reduced working copy …"`). on disk, MediaPipe's face mesh and blendshapes run on the prominent faces (at most 12, largest first) of the same full-size image, or of the working copy when there is no larger one (`lrg_ml::face_state`). Per face the row gets -`face_eye_state` (`open`/`half`/`closed`, `unknown` when the face is turned too -far, barely a face or too small), `face_eye_closed`, `face_smile`, -`face_gaze_away` and `face_yaw` (null when the model did not run for it). Per +`face_eye_state` (`open`/`half`/`closed`, `down` for lids down on a lowered head, +`unknown` when the face is turned too far, barely a face or too small), `face_eye_closed` (the more closed eye), +`face_smile`, `face_eye_squint`, `face_mouth_open`, `face_gaze_off_lens` (degrees from +the lens, head turn included), `face_yaw`, `face_pitch`, `face_roll` (degrees), the light +on the skin as `face_luminance`, `face_against_surround_stops`, `face_across_stops` and +`face_blown`, and `face_blendshapes` (all 52, in the order of +`lrg_ml::face_state::BLENDSHAPE_NAMES`); all null when the model did not run for it. A +pass that only prepares culling writes no rows of its own, but brings these readings +to the rows an earlier full pass wrote, matched by box, and touches nothing else on them. Per photo: `cull_eye_closed` (the most closed prominent eyes), `cull_eye_state`, -`cull_smile` (the weakest smile among the prominent faces), `cull_gaze_away` — -absent when no prominent face could be judged — plus `cull_face_state_version` -and `cull_face_state_model` when the model ran on this pass. Without the model +`cull_smile` (the weakest smile among the prominent faces), `cull_eye_squint` (the +strongest squint among the prominent faces not smiling), `cull_gaze_off_lens` (the +prominent face with open eyes looking furthest from the lens), `cull_mouth_open` (the +most open mouth among those not smiling), `cull_face_luminance` (the darkest), +`cull_face_across_stops` (the hardest light) and `cull_face_blown` (the most blown skin) — +absent when no prominent face could be judged — plus `cull_face_state_version` and `cull_face_state_model` when +the model ran on this pass. Without the model the response says once that a rough eye estimate was used, with the fix (`"Photos were analysed without eye state, smile and gaze: …"`). @@ -398,7 +408,7 @@ producer this build uses*, not merely present: | task | outstanding when | | --- | --- | | `embeddings` | no embedding, or a stored `embedding_model` that is not this build's | -| `faces` | no face rows and no `faces_checked`, or a `face_model` that is not this build's | +| `faces` | no face rows and no `faces_checked`, a `face_model` that is not this build's, or — once its model is on disk — a photo with faces whose `cull_face_state_version` is missing or older, or that a culling pass left with face rows it could not match (`face_rows_behind`) | | `aesthetic` (also implied by `embeddings` and `cull`) | no `aesthetic_score`, or an `aesthetic_model` that is not this build's — and only when the scorer is on disk, since a missing model cannot be fixed by running again | | genre (with `embeddings`, `cull` and `aesthetic`) | an embedding but no `genre`, or a `genre_version` that is not this build's (the genre list or its prompts changed), and only when the search model is on disk. Made again from the stored embedding, so the photo is not re-embedded | | `species` | no `species_checked`, or a `species_model` that is not the head on disk | @@ -502,6 +512,7 @@ Runs a semantic search query against indexed photos. | `sort` | string | `relevance`, `aesthetic` or `combined`. Default `relevance` with a term, `aesthetic` without. `relevance` and `combined` without a term are a **400** ("need a search term"), as is an unknown value | | `min_aesthetic` | float | Only photos whose current aesthetic score is at least this (1–9). A photo with no score from the current export cannot be judged and is left out | | `genres` | array | Only photos whose `genre` is one of these ids (see `GET /v1/search/genres`). A photo with no genre is left out | +| `eyes_open`, `smiling`, `looking_at_camera` | bool | Only photos where this holds for everyone who matters in the frame, read off `cull_eye_state`, `cull_smile` (0.5 or more) and `cull_gaze_off_lens` (20° or less). A photo measured to have no faces is simply no match; one with no current reading cannot be judged and is left out and counted | | `aesthetic_weight` | float | `combined` only, 0–1, default 0.5: the share of the ordering that is the aesthetic score rather than relevance | | `catalog_id` | string | Restrict to a specific catalog | | `photo_ids` | array | Restrict search to specific `photo_id` values (`uuids` is accepted as an alias) | @@ -530,7 +541,7 @@ match in the title, caption or keywords, which has no distance — counts as ful matches without a current score go **last**, in their earlier order: they are not on the scale, and a warning says how many. -`min_aesthetic` and `genres` are applied **after** the relevance cut, so they never move the cut +`min_aesthetic`, `genres` and the face questions are applied **after** the relevance cut, so they never move the cut (it is computed over the whole candidate list) and can leave fewer than `max_results`. What they drop for want of data, rather than for its value, is counted in `warnings`, since that is a problem the user can fix by indexing; a photo that was judged and found short of the minimum is simply not a @@ -606,17 +617,31 @@ abstain. They act only when every frame of a group with a face carries sharpest. A single is never rejected for its focus; - as `eyes_sharpest_in_group` on a pick within `reason_eye_focus_delta` of the best. -`portrait` weights and gates focus most, `sports` least, `street` not at all. +`portrait` weights and gates focus most, `sports` least, `street` not at all. Eyes measured +at or above `reason_eye_focus_threshold` also overrule a low frame-wide sharpness: such a +frame is not a blur defect. -**Eye state, smile and gaze** (`cull_eye_closed`, `cull_smile`, `cull_gaze_away`) -replace the eye-openness proxy where every frame of a group with a face -carries `cull_face_state_version` (all or nothing): the blink penalty is then -the model's `eyeBlink`, judged at `reason_eye_closed_threshold` / +**Eye state, smile and gaze** (`cull_eye_closed`, `cull_smile`, `cull_eye_squint`, +`cull_gaze_off_lens`) replace the eye-openness proxy where every frame of a group +with a face carries the current `cull_face_state_version` (all or nothing; an older +version counts as not analysed, and `/index/unprocessed` queues the photo again): the +blink penalty is then the model's `eyeBlink` of the more closed eye, judged at `reason_eye_closed_threshold` / `reject_eye_closed_threshold` (plus the relative rules) instead of the proxy's thresholds, and a face it could not judge is never a blink. Reasons: `eyes_closed`, `eyes_half_closed` (a label, never a defect), -`looking_away` (`portrait`), `best_smile`; a smile at +`looking_away` (`portrait`), `looking_down` (a note, and such a face is left out of +the blink reading), `best_smile`; a smile at `laugh_smile_threshold` with the eyes shut counts as `eyes_closed_intentional`. + +**Mouth and the light on the face** are labels only and move no score, until a +hand-culled shoot says what they are worth: `mouth_open` (`portrait` and `event`: +`cull_mouth_open` at `reason_mouth_open_threshold`, with a sibling clearly less +open), `face_darker` (the frame's darkest face at `reason_face_darker_stops` below the +group's brightest), `face_hard_light` (`cull_face_across_stops` at +`reason_face_hard_light_stops`, with a sibling clearly softer) and `face_blown` +(`cull_face_blown` at `reason_face_blown_threshold`, also on a single). `street` and +`sports` ask none of them. `cull_face_luminance` is only ever compared within a group, +since skin ranges over several stops between people. `smile_weight` (portrait 0.15, event 0.10) takes a share of the face term, against the group. @@ -627,7 +652,7 @@ against the group. | `summary` | object | `group_count`, `pick_count`, `alternate_count`, `reject_candidate_count`, `near_duplicate_group_count`, `intentional_set_group_count`, `intentional_set_photo_count`, `culling_preset`, `unindexed_count` | | `groups[].winner_photo_id` | string or null | The best-ranked frame that is not a reject candidate. **`null`** when every frame is one: a blurred single, or a burst blurred throughout | | `groups[].alternate_photo_ids`, `groups[].reject_candidate_photo_ids` | string[] | The rest of the group | -| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`gaze_away` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | +| `groups[].photos[]` | object[] | Per photo: `rank`, `cull_score`, `winner`, `reject_candidate`, `reason_codes`, `explanation`, `metrics`. `metrics` carries `semantic`/`semantic_axis`, `emotion`/`emotion_axis` and `moment` only when computed, `eye_focus`/`face_focus` only when measured, `eye_closed`/`smile`/`eye_squint`/`gaze_off_lens`/`mouth_open`/`face_luminance`/`face_across_stops`/`face_blown` when the group was judged on the face state, `eyes_closed_intended` when true, and always `clear_defect`. Where at least two frames of the group had their eyes measured, `explanation` ends in `eye focus N/10 within the group`: 10 for the sharpest, one point less per 0.03 below it | | `warnings` | string[] | Every caveat for the run, most fundamental first: unanalyzed photos skipped, no embeddings, AI signals whose prompt vectors could not be prepared (one message naming every lost signal and the fix), photos with faces prepared before eye focus or eye state were measured (one message when "Prepare now" fixes both), photos whose eyes the face state model could not analyse because it is not downloaded (with that fix), photos whose moment was not judged. Empty when there is nothing to say | | `warning` | string or null | The same messages joined, for clients older than `warnings` | @@ -708,7 +733,7 @@ mask list, plus: | Field | Type | Description | |---|---|---| -| `guardrail_reasons` | string[] | Machine-readable codes for what the frame allowed or refused, e.g. `hard_light_no_added_contrast`, `flat_light_contrast_allowed`. Empty when the budget changed nothing | +| `guardrail_reasons` | string[] | Machine-readable codes (with the face and expression models on disk, also `face_hard_light`, `face_below_surroundings`, `face_blown`, read off the faces in the frame; if that read fails, `warning` says so and the edit goes on without them) for what the frame allowed or refused, e.g. `hard_light_no_added_contrast`, `flat_light_contrast_allowed`. Empty when the budget changed nothing | | `guardrail_explanations` | string[] | The same, as sentences for the UI | | `edit_warnings` | string[] | Unrelated problems worth surfacing (e.g. no training examples found) | @@ -893,8 +918,10 @@ returned as warnings rather than failing the request. Returns all detected persons with `person_id`, `name`, `face_count`, `photo_count`, `manual_count` (how many of the faces a human placed) and a representative `thumbnail` (base64 JPEG, empty when the person has none). The -representative is the highest-scoring face by `det_score + sharpness`, not the -first row found. +representative is the person's best portrait (`lrg_analysis::face_expression::portrait_score`): +detector confidence and sharpness, less a shut eye, a turned head and a gaze off +the lens, plus a little for a smile. A row without a reading pays a fixed price, so +it yields to a portrait known to be good. Every face without a person — an empty `person_id` from a row that was never clustered, or `person_unassigned` written by the clusterer — is reported as a @@ -939,13 +966,22 @@ already has — as its merge gesture: it asks, then either posts to really are different people with the same name. ### `GET /v1/faces/persons//photos` -Returns the list of `photo_id` values associated with a specific person. +The ids of the photos showing this person, in `photo_ids` (`photo_uuids` is the same list). `_unassigned` opens the faces without a person. + +| Query | Meaning | +|---|---| +| `eyes_open`, `smiling`, `looking_at_camera` | `true` keeps only photos where this person's face answers yes (`lrg_analysis::face_expression`). A person twice on a photo counts when either face does | +| `sort` | `best` puts the person's best portrait first; otherwise by photo id | +| `limit` | At most this many | + +`not_judged` counts the photos left out because the face has no reading, and `warnings` says so with the fix, since that is a problem the user can solve, where a frown is simply not a smile. ### `GET /v1/faces/persons//faces` Returns the individual faces in one person, which is what the People page's detail view is built on. Query parameters: `limit` (default 200, max 500), `offset`, and `sort` — `typical` (closest to the cluster centroid first), -`outlier` (furthest first, which surfaces wrong faces) or `quality`. +`outlier` (furthest first, which surfaces wrong faces), `quality` or `portrait` (the +best picture of them first, as for the person's card). Only this person's vectors are fetched, so the endpoint is cheap enough to page through. Each face carries `face_id`, `photo_id`, `thumbnail`, `distance` from @@ -1071,7 +1107,7 @@ nothing will ever run: | page | action | needs | |---|---|---| -| `people` | `show_in_library` | a non-empty `person_ids` array | +| `people` | `show_in_library` | a non-empty `person_ids` array; `eyes_open`, `smiling`, `looking_at_camera`, `best` are passed on to `GET /v1/faces/persons//photos` | | `search` | `show_search_results` | a non-empty `photo_ids` array of strings, in rank order; `term`, `sort`, `min_aesthetic`, `genre_label`, `max_results` name the collection | | `search` | `refresh_search_context` | nothing | diff --git a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md index 0a42a8c7..5fd2039c 100644 --- a/docs/wiki/Dev-Image-Culling-Implementation-Plan.md +++ b/docs/wiki/Dev-Image-Culling-Implementation-Plan.md @@ -1185,17 +1185,20 @@ read "closed" on 767 of 769 frames. portrait (`tests/face_state_goldens.rs`): landmarks within ~1% of the interocular distance, blink and smile within 0.01; the closed-lid path exactly, by moving MediaPipe's own upper-lid landmarks onto the lower ones. -- **Per face**: `eye_closed` (mean `eyeBlink`, only the near eye from 35° of - head turn), `eye_state` open/half/closed/unknown (unknown below 0.5 presence, - beyond 60° of yaw, or with the eyes under 24 px apart), smile, gaze, yaw. +- **Per face**: `eye_closed` (`eyeBlink` of the more closed eye, only the near + eye from 35° of head turn), `eye_state` open/half/closed/unknown (unknown below + 0.5 presence, beyond 60° of yaw, or with the eyes under 24 px apart), smile, + squint, `gaze_off_lens` in degrees, head pose (yaw, pitch, roll) and all 52 + blendshapes, so that a later signal need not decode every original again. Faces under 64 px are not run — the model card warns against rescaling them. Prominent faces only (the aggregation's gate), at most 12, on the full-size original from the focus step when there is one. - **Per photo**: the most closed prominent eyes, the weakest smile, the - furthest gaze, and `cull_face_state_version`. + strongest squint on a face not smiling, the furthest gaze among open eyes, and + `cull_face_state_version`, whose value counts: an older one is analysed again. - **Ranking**: where the group has the version throughout, the model's reading becomes blink penalty and eye openness, judged at its own thresholds (closed - ≥ 0.6, label ≥ 0.5; sports 0.8/0.7, street 0.85/0.75, portrait 0.55/0.45); + > 0.55, label > 0.5; sports 0.68/0.6, street 0.72/0.62, portrait 0.5/0.45); `eyes_closed`, `eyes_half_closed` (label only), `looking_away` (portrait), `best_smile`; a smile of 0.6 or more with the eyes shut is a laugh; smile weight 0.15 portrait, 0.10 event. An unknown face is never a blink. @@ -1207,9 +1210,118 @@ read "closed" on 767 of 769 frames. There is no photograph with closed eyes under a free license in the repository, so the closed path is pinned synthetically (lids moved shut: -0.81/0.76) and the open path on the reference portrait (0.04/0.03). The -thresholds sit between those with room on both sides, and the relative rules -still apply on top, but **they are hypotheses** until a hand-culled shoot with -real blinks has been scored with `cull_eval --ablate` — the same caveat as every -other threshold here. +0.81/0.76) and the open path on the reference portrait (0.04/0.03). + +Those two points were a poor guide. Measured with `examples/face_state_probe.rs` +on portraits from Wikimedia Commons (not committed; 285 usable faces to choose the +thresholds, judged by eye, then 292 from a disjoint sample to check them): + +| Reading | On the first sample | On the second sample | +|---|---|---| +| Eyes shut or lowered, caught by the first rule (mean ≥ 0.6) | 6 of 22 | about 8 of 24 | +| The same, caught by the more closed eye > 0.55 | 19 of 22 | about 17 of 24 | +| Faces with the eyes open and a reading > 0.5 | 0 of 213 | 1 of 178 portraits | +| Labelled `looking_away` in portraits, old (`eyeLook` > 0.5) → new (> 25°) | | 25–31% → 10–12% | +| The same in Commons' *People looking away* | | 53% → 67% | + +- **`eyeBlink` tops out low.** Eyes shut read 0.54–0.78 on the more closed eye and + 0.41–0.72 as a mean. No open eye reached 0.35. +- **`eyeLook*` is the eye inside the head.** People turned 25–30° and looking + straight at the camera read 0.85–0.99, and a head turned 39° with the eyes + following it 0.14, so the old reading had it backwards. Added to the head's turn + at 28° per unit (where the two cancel for people looking back at the lens) it + separates well: portraits a median 9°, people looking away 31°. Up and down is + weaker: `eyeLookUp` barely moves when a lowered head looks up at the lens, so + selfies from above read about 20°. That is why the label sits at 25°. Shut eyes + abstain: their `eyeLook*` is noise, 31° on one such portrait. +- **The head pose** is a frame of bone landmarks, within about 1° of a full + Procrustes fit to MediaPipe's canonical face model on yaw and roll and 2° on pitch. +- **`eyeSquint` runs high for everyone** (median 0.43) and rises on a smile, so it + counts only from 0.78 on a face smiling under 0.3. It fired on no face of the + second sample; it adds little beyond the lids, and no false alarms. +- **Several coefficients do not move on photographs at all**: `cheekSquint`, + `noseSneer`, `cheekPuff` and `jawForward` stay at 0.00 even at the 99th percentile, + and `mouthFrown` and `mouthFunnel` reach about 0.1. Don't build on them: a genuine + smile cannot be read from `cheekSquint`. `jawOpen`, `mouthPucker`, `browDown` and + `mouthPress` do move. + +### A lowered gaze is not a blink + +Commons did not show this; the first run on a real shoot did. In a test catalog of +197 photos, 36 faces read closed, nearly all a child looking down at candles with the +lids over the eyes. Lids down on a head lowered 10° or more now read `down`: + +| Faces reading closed (0.5 or more on the lids) | Called `down` | +|---|---| +| This catalog, nearly all really looking down | 30 of 39 | +| Commons, eyes really shut, mostly people asleep | 10 of 34 | + +The second row is the price, and the direction is the right one: a note where a reject +would have been. `eyeLookDown` does not help, since it runs high on shut eyes too +(0.45–0.80). After this the faces reading closed in the catalog fell from 36 to 7, and +all 18 of the highest `down` readings, checked by eye, really do look down. Of the 7, about +two are right. The others are lower gazes on a head barely lowered (under 10°), and lids +on a raised head behind glasses, which read 0.50–0.58 with the eyes open. The relative +rules hold those back inside a burst. + +### Sharp eyes in a soft frame + +The same catalog had ten frames of a child behind birthday candles read `blurred` +throughout, on a frame-wide sharpness of 0.03–0.05, while the eyes read 0.54–0.60 and were +visibly sharp. With every frame a defect the group had no pick, and all ten were reject +candidates. A frame whose eyes reach `reason_eye_focus_threshold` is no longer a blur defect. +Of the 13 frames with faces reading `blurred`, the 10 that pass include the 9 that are clearly sharp at the eyes +when checked by eye, and the one really smeared frame (0.42) stays. It is one catalog, and +the reading depends on magnification, so a slightly soft frame shot close can still pass. + +### Mouth and the light on the face + +Both went in as labels and move no score: there is still no hand-culled shoot to say +what they are worth. Measured the same way, on 692 usable faces with the second +sample and 190 photographs of people speaking added. + +- **The mouth is `jawOpen` and `mouthLowerDown`.** `mouthPucker` reads up to 0.99 on + full lips at rest and `mouthFunnel` barely moves, so neither is used. A lowered head + abstains: foreshortened lips read open when closed. From 0.2 the + mouth is visibly open on nearly every face judged: 10% of people speaking, 0.6% of + portraits. A laugh is open too, so only a face smiling under 0.4 counts. +- **Blown means two channels.** The top channel at 250 flagged 16% of faces, most of + them warm light clipping red on skin that holds its detail. The middle channel at + 245 leaves 2.6% over 0.05, where surgical masks and white headbands still sit, and + from 0.15 what is left is blown skin. +- **Light across the face holds up.** It compares a face with itself, so it does not + care about skin. 10% of faces reach 2 stops, nearly all real side light; hair across + a cheek is the main false reading. +- **A face against its surroundings does not mean what it looks like.** The lowest + readings are almost all people with dark skin in front of something bright, and + correctly exposed. So it never produces a label. Its only use is to hold back added + contrast in AI Edit, which costs that face nothing, and that explanation says what + was measured rather than calling the frame backlit. +- **The same goes for luminance alone**, which is only ever compared with the same + people in another frame of the group. + +### People, and what clustering did not need + +`lrg_analysis::face_expression` answers what People and search ask: eyes open, +smiling (`mouthSmile` from 0.5, where most faces judged by eye show a smile and a +snarl can still reach it), looking at the camera (within 20°, so a face between that +and the 25° of `looking_away` is claimed by neither), and a person's best portrait. + +Keeping turned heads out of cluster formation looked like the obvious next step and +is not in: `examples/face_cluster_eval.rs`, on 370 faces of 18 public figures from +Commons, scores it the same at every limit from 60° to 30°, because the occlusion gate +already holds those faces back. That gate holds back over half of them, which looked +wrong, but raising it from 0.55 lowers the score at every threshold tried. Recall in +this sample is low (0.29) whichever way, with few photographs per person, so it says +what these two changes do and not how good clustering is. + +### What this does not solve + +A slit narrowed against the sun overlaps eyes shut (0.5 +to 0.65), and neither `eyeSquint` nor the lid gap in the landmarks separates them; +sunglasses read as closed (0.47–0.74); a head lying on its side reads open with the +eyes shut. The relative rules catch the first two inside a burst, since every frame +reads alike there. And portraits from Commons are not a shoot, so **the thresholds +stay hypotheses** until a hand-culled shoot with real blinks has been scored with +`cull_eval --ablate` — the same caveat as every other threshold here. diff --git a/docs/wiki/Help-AI-Agents.md b/docs/wiki/Help-AI-Agents.md index 7ef679da..6dc1886f 100644 --- a/docs/wiki/Help-AI-Agents.md +++ b/docs/wiki/Help-AI-Agents.md @@ -40,9 +40,9 @@ Behind those requests the agent uses these tools: | Tool | What it does | |---|---| | `get_status` | Whether Lightroom is connected, the open catalog, the selection, how much is indexed | -| `search_photos` | Semantic search: describe what is in the picture. Can also rank by aesthetic score and keep only some genres; with no description it returns your best photos (see [Advanced Search](Help-Advanced-Search#ranking)) | +| `search_photos` | Semantic search: describe what is in the picture. Can also rank by aesthetic score, keep only some genres, and ask of the faces (eyes open, smiling, looking at the camera); with no description it returns your best photos (see [Advanced Search](Help-Advanced-Search#ranking)) | | `find_similar_photos` | Photos that look alike, or near-duplicates | -| `list_people`, `find_photos_of_people` | People from LrGeniusAI's face recognition | +| `list_people`, `find_photos_of_people` | People from LrGeniusAI's face recognition, also narrowed by what their face shows (eyes open, smiling, looking at the camera) or to their best portraits | | `find_photos_in_catalog` | Lightroom's own filters: rating, flag, label, keywords, folder, dates, camera, lens | | `cull_photos` | Groups bursts and duplicates, names winners and reject candidates | | `get_photo_details`, `view_photo` | Metadata (including aesthetic score and genre) — on request every field Lightroom keeps, IPTC, EXIF, GPS and other plug-ins' fields included — and the photo as Lightroom renders it now | diff --git a/docs/wiki/Help-AI-Edit.md b/docs/wiki/Help-AI-Edit.md index 23efb4b8..19ad9a8f 100644 --- a/docs/wiki/Help-AI-Edit.md +++ b/docs/wiki/Help-AI-Edit.md @@ -133,7 +133,10 @@ thin style profile from one caused by an unusual photo. When enabled, a **review dialog** opens for each photo before the edit is applied. You see: - The proposed develop values, plus how confident the style match was and which of your saved edits it drew on. -- Any guardrail explanations — what the frame allowed, and what got capped. +- Any guardrail explanations — what the frame allowed, and what got capped. With + the face models downloaded these include the faces in the frame: hard light across + a face leaves clarity alone, a face well below the frame around it keeps contrast + from pushing it darker, and blown skin holds the white point back. - Options to **Apply** or **Skip**. **Recommended for first use.** Disable only after you've validated the results for your shooting style. diff --git a/docs/wiki/Help-Advanced-Search.md b/docs/wiki/Help-Advanced-Search.md index 59e69215..f22b6387 100644 --- a/docs/wiki/Help-Advanced-Search.md +++ b/docs/wiki/Help-Advanced-Search.md @@ -152,6 +152,10 @@ keep it, so open the search from Lightroom each time. **Keywords and captions** (the metadata half of the search, all four fields) — the dialog's controls of the same name. They start where you left them in the dialog. +- **Faces** — *eyes open*, *smiling*, *looking at the camera*: only photos where + that holds for everyone who matters in the frame. It is read off the faces, so it + is more reliable than searching for "smiling". Photos without faces are left out, + and the page says how many could not be judged and what to run on them. Sliders and menus search again as you move them. Notes from the backend — no search model, photos without a score left out by the minimum, and the like — diff --git a/docs/wiki/Help-Analyze-and-Index.md b/docs/wiki/Help-Analyze-and-Index.md index 3473de03..cd132e7f 100644 --- a/docs/wiki/Help-Analyze-and-Index.md +++ b/docs/wiki/Help-Analyze-and-Index.md @@ -88,6 +88,8 @@ needed for metadata: embeddings, faces and species run without one. [culling](Help-Cull-Photos.md#focus-on-the-eyes) and — with the eye and expression model downloaded — whether the eyes are open, the smile and the gaze; a photo without faces is not. + Photos whose faces were indexed before that model was on disk are read again the + next time you run this with face detection on them, and keep the people you named. - **Identify animal and plant species** — runs BioCLIP 2 and writes a taxonomic identification. Greyed out until the models have been downloaded in the Plug-in Manager, under *On-device AI models* — one **Download AI diff --git a/docs/wiki/Help-Cull-Photos.md b/docs/wiki/Help-Cull-Photos.md index 76d14c6a..8ff2d378 100644 --- a/docs/wiki/Help-Cull-Photos.md +++ b/docs/wiki/Help-Cull-Photos.md @@ -161,6 +161,9 @@ eye and over the face as a whole. - **The sharper eye counts.** With a shallow depth of field the near eye is sharp and the far one is not, which is how it should be. +- **Sharp eyes overrule a soft-looking frame.** A small face against something dark + or out of focus can make the whole frame measure soft. Where the eyes were measured + sharp, the frame is not called blurred. - **In a couple or group shot, the least sharp prominent face decides**: the frame is only sharp when everyone in it is. Small faces in the background do not count. @@ -209,17 +212,36 @@ blink. the frame. - **One blink spoils the group.** The most closed eyes among the people that matter in the frame decide, so a group shot with one person blinking is not - "eyes open". Small faces in the background do not count. + "eyes open". One shut eye counts as well, even when the other is open. Small + faces in the background do not count. - **Compared with the burst.** A frame becomes a reject candidate for closed eyes only when another frame of the same burst has them open. Half-closed eyes (a squint into the sun, a mid-blink) are named in the explanation but - never make a frame a reject candidate. + never make a frame a reject candidate. The model cannot always tell eyes + narrowed to a slit from eyes shut, so a hard squint can read as closed, and + sunglasses can too. Comparing with the burst keeps either from costing a frame + when the whole burst looks the same. - **A laugh is not a blink.** A broad smile with the eyes shut counts as a laugh, like a kiss already did. +- **Looking down is not a blink.** Lids down on a lowered head, a child looking at + the candles say, read much like shut eyes. Such a frame says "looking down" and + is never a reject candidate for it. Someone really asleep with the head down gets + the same note, which costs a warning and no photo. - **The smile counts in Portrait and Event.** Between otherwise equal frames, the one where everyone smiles more wins, and its explanation says - "best smile in group". **Portrait** also notes when someone is looking away - from the camera. + "best smile in group". +- **The mouth, in Portrait and Event.** A frame with the mouth caught open mid-word + says so when another frame of the burst has it closed. A note only; a broad + smile or a laugh does not count. +- **The light on the face.** The explanation names a face darker than in another + frame of the burst, harder light across the face than in another frame, and skin + that is blown out. Not at **Street** or **Sports**. These are notes and move no + score. A face is only ever compared with itself in another frame, never with a + standard, because skin differs by several stops between people. +- **Looking away means from the lens.** **Portrait** notes when someone looks + away from the camera. Head and eyes count together: a head turned to the side + with the eyes back on the camera is looking at it, and a head turned with the + eyes following it is not. It is a note, never a reason to reject. Like focus, the face is read on the full-size original where there is one. The model is part of **Download AI models** (about 7 MB). Without it culling @@ -280,8 +302,10 @@ For each photo, the backend stores culling-related fields such as: `cull_blink_penalty`, `cull_occlusion`, and `cull_eye_focus` / `cull_face_focus` (focus on the eyes and on the face, measured on the full-size original; absent when the faces were too small to judge), and - `cull_eye_state` / `cull_eye_closed`, `cull_smile`, `cull_gaze_away` (from - the face model; absent when no face could be judged) + `cull_eye_state` / `cull_eye_closed`, `cull_smile`, `cull_eye_squint`, + `cull_gaze_off_lens`, `cull_mouth_open`, `cull_face_luminance`, + `cull_face_across_stops`, `cull_face_blown` (from the face model; absent when no + face could be judged) - aesthetic scores: `cull_aesthetic` (a contrast-and-colour heuristic) and the photo's **Aesthetic score** from the AestheticSigLIP model, stored at indexing time (`aesthetic_score`) and read by culling as `cull_aesthetic_model` diff --git a/docs/wiki/Help-People-Faces.md b/docs/wiki/Help-People-Faces.md index c3f0ca91..8583b7ce 100644 --- a/docs/wiki/Help-People-Faces.md +++ b/docs/wiki/Help-People-Faces.md @@ -35,7 +35,8 @@ does not keep it, so open People from Lightroom each time. A grid of persons from your indexed photos. Each card has: -- A thumbnail of the representative face. +- A thumbnail of their best portrait: eyes open and facing the camera, where the + model could read that. - A name field, or *Unassigned faces* for the faces not yet grouped into a person. - The number of photos this person appears in and how many faces are in the @@ -117,8 +118,9 @@ actually right. Each face shows how much it looks like the rest of the group. A face further from the group than the clustering threshold is marked as an outlier, and a face that failed the quality gate says which measurement was bad. The **sort** -control offers *least like this person first*, *most like this person first* -and *best quality first* — the first of those is the fast way to spot a face +control offers *least like this person first*, *most like this person first*, +*best quality first* and *best portrait first* (eyes open, facing the camera, +sharp) — the first of those is the fast way to spot a face that does not belong, so it is what the view opens on. Click faces to select them (shift-click for a range), then: @@ -153,8 +155,16 @@ several people selected, the dropdown decides what you get: together (the default) - **photos with any selected person** — the union +**Only** narrows that down by what the person's face shows: *eyes open*, +*smiling*, *looking at the camera*. With several people selected, each of them is +asked. **Photos** takes only each person's best 20 or 50 portraits instead of all +of them. A photo whose face could not be judged is left out, and Lightroom says how +many that was and what to do about it: their faces were indexed before the eye and +expression model was downloaded. Running **Analyze & Index Photos** with face +detection reads them, and keeps the people you named. + The collection goes into the `People` collection set and is named after the -selected people plus a timestamp. People without a name appear in it once, as +selected people, what you asked for, and a timestamp. People without a name appear in it once, as "Unnamed person"; a name list longer than 100 characters becomes `People ()`. diff --git a/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua b/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua index 5f147bc0..750b188e 100644 --- a/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua +++ b/plugin/LrGeniusAI.lrdevplugin/APISearchIndex.lua @@ -4397,15 +4397,39 @@ function SearchIndexAPI.setPersonName(personId, name) return true, result or {} end +--- +-- The query string asking for only some of a person's photos. +-- @param options table|nil eyes_open, smiling, looking_at_camera (booleans), best (a count) +-- @return string "" when it asks nothing, else starting with "?" +function SearchIndexAPI.personPhotosQuery(options) + options = options or {} + local parts = {} + for _, key in ipairs({ "eyes_open", "smiling", "looking_at_camera" }) do + if options[key] == true then + parts[#parts + 1] = key .. "=true" + end + end + if type(options.best) == "number" and options.best > 0 then + parts[#parts + 1] = "sort=best" + parts[#parts + 1] = string.format("limit=%d", options.best) + end + if #parts == 0 then + return "" + end + return "?" .. table.concat(parts, "&") +end + --- -- Get photo UUIDs that contain this person. -- @param personId string e.g. "person_0" --- @return table|nil { status, person_id, photo_uuids } or nil, err -function SearchIndexAPI.getPhotosForPerson(personId) +-- @param options table|nil see personPhotosQuery +-- @return table|nil { status, person_id, photo_uuids, not_judged, warnings } or nil, err +function SearchIndexAPI.getPhotosForPerson(personId, options) if not personId or personId == "" then return nil, "person_id required" end local url = SearchIndexAPI.url("FACES_PERSON_PHOTOS", personId, "photos") + .. SearchIndexAPI.personPhotosQuery(options) local result, err = _request("GET", url, {}) if err then log:error("getPhotosForPerson failed: " .. err) diff --git a/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua b/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua index ddff2d9d..88d87f97 100644 --- a/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua +++ b/plugin/LrGeniusAI.lrdevplugin/TaskPeople.lua @@ -57,23 +57,51 @@ local function photoIdsFromPersonResponse(resp) or (type(resp.photo_uuids) == "table" and resp.photo_uuids or {}) end +--- What a selection on the People page asked for, in words: "best 20, eyes open". +local function describePhotoOptions(options) + options = options or {} + local parts = {} + if type(options.best) == "number" then + parts[#parts + 1] = string.format("best %d", options.best) + end + if options.eyes_open then + parts[#parts + 1] = "eyes open" + end + if options.smiling then + parts[#parts + 1] = "smiling" + end + if options.looking_at_camera then + parts[#parts + 1] = "looking at the camera" + end + return table.concat(parts, ", ") +end + +--- Collects what a person-photos response says went short, in either field. +local function addResponseWarnings(warnings, resp) + if resp.warning then + log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) + addWarning(warnings, resp.warning) + end + for _, warning in ipairs(type(resp.warnings) == "table" and resp.warnings or {}) do + log:warn("GetPhotosForPerson warning: " .. tostring(warning)) + addWarning(warnings, warning) + end +end + --- Union: jedes Foto, das mindestens eine der Personen enthält (Reihenfolge: erstes Auftreten). -local function unionPhotoIdsForEntries(entries, warnings) +local function unionPhotoIdsForEntries(entries, warnings, options) local seen = {} local photoIdsOrdered = {} for _, ent in ipairs(entries) do local pid = ent and ent.person_id if pid and pid ~= "" then - local resp, err = SearchIndexAPI.getPhotosForPerson(pid) + local resp, err = SearchIndexAPI.getPhotosForPerson(pid, options) if err or type(resp) ~= "table" then ErrorHandler.handleError("Could not get photos for person", err or "No data") return nil end - if resp and resp.warning then - log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) - addWarning(warnings, resp.warning) - end + addResponseWarnings(warnings, resp) local ids = photoIdsFromPersonResponse(resp) for _, photoId in ipairs(ids) do @@ -88,7 +116,7 @@ local function unionPhotoIdsForEntries(entries, warnings) end --- Schnittmenge: Fotos, in denen alle Personen gemeinsam vorkommen (Reihenfolge wie erste Person). -local function intersectPhotoIdsForEntries(entries, warnings) +local function intersectPhotoIdsForEntries(entries, warnings, options) if #entries == 0 then return {} end @@ -99,15 +127,12 @@ local function intersectPhotoIdsForEntries(entries, warnings) if not pid or pid == "" then return {} end - local resp, err = SearchIndexAPI.getPhotosForPerson(pid) + local resp, err = SearchIndexAPI.getPhotosForPerson(pid, options) if err or type(resp) ~= "table" then ErrorHandler.handleError("Could not get photos for person", err or "No data") return nil end - if resp.warning then - log:warn("GetPhotosForPerson warning: " .. tostring(resp.warning)) - addWarning(warnings, resp.warning) - end + addResponseWarnings(warnings, resp) local ids = photoIdsFromPersonResponse(resp) if not firstIds then firstIds = ids @@ -142,7 +167,7 @@ local function intersectPhotoIdsForEntries(entries, warnings) end --- Führt "Show in Library" aus. entries: { person_id, person_name? }[]; matchMode: "union" | "intersection". -local function doShowInLibrary(entries, matchMode) +local function doShowInLibrary(entries, matchMode, options) if not entries or #entries == 0 then LrDialogs.message("No people selected", "Select one or more people on the People page, then try again.", "info") return @@ -152,9 +177,9 @@ local function doShowInLibrary(entries, matchMode) local warnings = {} local photoIdsOrdered if mode == "intersection" and #entries >= 2 then - photoIdsOrdered = intersectPhotoIdsForEntries(entries, warnings) + photoIdsOrdered = intersectPhotoIdsForEntries(entries, warnings, options) else - photoIdsOrdered = unionPhotoIdsForEntries(entries, warnings) + photoIdsOrdered = unionPhotoIdsForEntries(entries, warnings, options) end if photoIdsOrdered == nil then @@ -166,7 +191,10 @@ local function doShowInLibrary(entries, matchMode) reportWarnings(warnings) if #photoIdsOrdered == 0 then - if mode == "intersection" and #entries >= 2 then + local asked = describePhotoOptions(options) + if asked ~= "" then + LrDialogs.message("No photos", string.format("No photos of this selection match: %s.", asked), "info") + elseif mode == "intersection" and #entries >= 2 then LrDialogs.message("No photos", "No photos contain all selected people together.", "info") else LrDialogs.message("No photos", "No photos found for this person.", "info") @@ -196,6 +224,10 @@ local function doShowInLibrary(entries, matchMode) if #label > 100 then label = string.format("People (%d)", #entries) end + local asked = describePhotoOptions(options) + if asked ~= "" then + label = string.format("%s (%s)", label, asked) + end local collectionName = string.format("%s @ %s", label, LrDate.timeToW3CDate(LrDate.currentTime())) local collectionSet, collection @@ -249,7 +281,12 @@ local function runAction(action) ) return end - doShowInLibrary(entries, action.match_mode) + doShowInLibrary(entries, action.match_mode, { + eyes_open = action.eyes_open == true, + smiling = action.smiling == true, + looking_at_camera = action.looking_at_camera == true, + best = type(action.best) == "number" and action.best or nil, + }) return end diff --git a/plugin/spec/api_search_index_spec.lua b/plugin/spec/api_search_index_spec.lua index 60b6d044..0a540310 100644 --- a/plugin/spec/api_search_index_spec.lua +++ b/plugin/spec/api_search_index_spec.lua @@ -101,6 +101,25 @@ describe("SearchIndexAPI.interpretIndexResponse", function() end) end) +describe("SearchIndexAPI.personPhotosQuery", function() + it("asks nothing when nothing was asked", function() + assert.are.equal("", SearchIndexAPI.personPhotosQuery(nil)) + assert.are.equal("", SearchIndexAPI.personPhotosQuery({ eyes_open = false, best = 0 })) + end) + + it("asks only for what was switched on", function() + assert.are.equal("?eyes_open=true", SearchIndexAPI.personPhotosQuery({ eyes_open = true, smiling = false })) + assert.are.equal( + "?smiling=true&looking_at_camera=true", + SearchIndexAPI.personPhotosQuery({ smiling = true, looking_at_camera = true }) + ) + end) + + it("turns a count into the best photos first", function() + assert.are.equal("?sort=best&limit=20", SearchIndexAPI.personPhotosQuery({ best = 20 })) + end) +end) + describe("SearchIndexAPI.interpretEditResponse", function() it("accepts a success response and returns the whole body", function() local response = { status = "success", recipe = { Exposure2012 = 0.3 } } diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index f0955973..4c48ba86 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -4489,6 +4489,7 @@ dependencies = [ "futures-util", "http-body-util", "log", + "lrg-analysis", "lrg-api", "lrg-common", "lrg-store", diff --git a/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs b/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs new file mode 100644 index 00000000..01104dc4 --- /dev/null +++ b/server-rs/crates/lrg-analysis/examples/face_cluster_eval.rs @@ -0,0 +1,131 @@ +//! Scores face clustering against known identities, for a change to its policy. +//! +//! ```text +//! cargo run --release -p lrg-ml --example face_state_probe -- --embed people/*/*.jpg > faces.jsonl +//! cargo run --release -p lrg-analysis --example face_cluster_eval -- faces.jsonl +//! ``` +//! +//! A face's identity is the folder its photo is in, and only the largest face of +//! each photo counts, so a photo of two people does not read as a mistake. The +//! score is BCubed: precision is how much of a face's cluster is the same person, +//! recall how much of that person is in the face's cluster. A face left unassigned +//! is a cluster of its own, which is what it is to whoever browses the People page. + +use std::collections::HashMap; +use std::path::Path; + +use lrg_analysis::face_cluster::{cluster_faces, ClusterParams, FaceInput}; +use serde_json::Value; + +struct Scored { + precision: f64, + recall: f64, + clusters: usize, + unassigned: usize, + held_back: usize, + attached: usize, +} + +fn score(faces: &[FaceInput], identity: &[String], params: &ClusterParams) -> Scored { + let outcome = cluster_faces(faces, params); + let label = |i: usize| { + outcome.labels[i] + .map(|l| l as i64) + .unwrap_or(-(i as i64) - 1) + }; + let (mut precision, mut recall) = (0.0, 0.0); + for i in 0..faces.len() { + let same_cluster = |j: &usize| label(*j) == label(i); + let same_person = |j: &usize| identity[*j] == identity[i]; + let in_cluster = (0..faces.len()).filter(same_cluster).count() as f64; + let of_person = (0..faces.len()).filter(same_person).count() as f64; + let both = (0..faces.len()) + .filter(|j| same_cluster(j) && same_person(j)) + .count() as f64; + precision += both / in_cluster; + recall += both / of_person; + } + let n = faces.len().max(1) as f64; + Scored { + precision: precision / n, + recall: recall / n, + clusters: outcome.clusters.len(), + unassigned: outcome.labels.iter().filter(|l| l.is_none()).count(), + held_back: outcome.stats.low_quality, + attached: outcome.stats.attached, + } +} + +fn main() { + let path = std::env::args().nth(1).expect("faces.jsonl"); + let text = std::fs::read_to_string(path).expect("readable file"); + let mut largest: HashMap = HashMap::new(); + for line in text.lines() { + let row: Value = serde_json::from_str(line).expect("one JSON object per line"); + let file = row["file"].as_str().unwrap_or_default().to_string(); + let area = |v: &Value| v["area_ratio"].as_f64().unwrap_or(0.0); + if largest + .get(&file) + .is_none_or(|kept| area(kept) < area(&row)) + { + largest.insert(file, row); + } + } + let mut rows: Vec<(String, Value)> = largest.into_iter().collect(); + rows.sort_by(|a, b| a.0.cmp(&b.0)); + + let identity: Vec = rows + .iter() + .map(|(file, _)| { + Path::new(file) + .parent() + .map(|p| p.display().to_string()) + .unwrap_or_default() + }) + .collect(); + let number = |v: &Value, key: &str| v[key].as_f64(); + let faces: Vec = rows + .iter() + .map(|(file, row)| FaceInput { + id: file.clone(), + embedding: row["embedding"] + .as_array() + .expect("run the probe with --embed") + .iter() + .filter_map(|v| v.as_f64().map(|x| x as f32)) + .collect(), + det_score: number(row, "det_score"), + area_ratio: number(row, "area_ratio"), + sharpness: number(row, "sharpness"), + occlusion: number(row, "occlusion"), + pinned: None, + }) + .collect(); + println!( + "{} faces of {} people", + faces.len(), + identity + .iter() + .collect::>() + .len(), + ); + println!("threshold max_occl clusters unassigned held attached precision recall f1"); + for threshold in [0.4, 0.5, 0.6] { + for max_occlusion in [0.55, 0.7, 0.85, 1.0] { + let mut params = ClusterParams::defaults(); + params.cosine_threshold = threshold; + params.gate.max_occlusion = max_occlusion; + let s = score(&faces, &identity, ¶ms); + println!( + "{threshold:9.2} {max_occlusion:8.2} {:8} {:10} {:4} {:8} {:9.3} {:6.3} {:.3}", + s.clusters, + s.unassigned, + s.held_back, + s.attached, + s.precision, + s.recall, + 2.0 * s.precision * s.recall / (s.precision + s.recall).max(1e-9), + ); + } + } +} diff --git a/server-rs/crates/lrg-analysis/src/culling_config.rs b/server-rs/crates/lrg-analysis/src/culling_config.rs index b8543e08..b41feed2 100644 --- a/server-rs/crates/lrg-analysis/src/culling_config.rs +++ b/server-rs/crates/lrg-analysis/src/culling_config.rs @@ -302,13 +302,13 @@ pub struct RankingConfig { /// A pick within this of the group's best eye focus is reported as having /// the sharpest eyes of the group (`eyes_sharpest_in_group`). pub reason_eye_focus_delta: f64, - /// With the face state model (`cull_eye_closed`, MediaPipe's `eyeBlink`), - /// above this a frame is labelled `eyes_closed`; above - /// [`Self::reject_eye_closed_threshold`] it is a blink defect. They replace - /// `reason_possible_blink_threshold`/`reject_blink_penalty_threshold`, + /// With the face state model (`cull_eye_closed`, MediaPipe's `eyeBlink` of + /// the more closed eye), above this a frame is labelled `eyes_closed`; + /// above [`Self::reject_eye_closed_threshold`] it is a blink defect. They + /// replace `reason_possible_blink_threshold`/`reject_blink_penalty_threshold`, /// which were tuned against a proxy that read "closed" on almost every - /// small face; the relative rules still apply on top. On the reference - /// portrait an open eye reads 0.03–0.04 and shut lids 0.76–0.81. + /// small face; the relative rules still apply on top. On real photographs + /// judged by hand, eyes shut read 0.54–0.78 and no open eye reached 0.35. /// **Hypotheses until a hand-culled fixture says otherwise.** pub reason_eye_closed_threshold: f64, pub reject_eye_closed_threshold: f64, @@ -316,6 +316,10 @@ pub struct RankingConfig { /// squint, a mid-blink — and the frame is labelled `eyes_half_closed`. /// Never a defect. 0 switches the label off. pub reason_eyes_half_closed_threshold: f64, + /// `cull_eye_squint` at or above this gets the same label, for eyes + /// narrowed against the sun with the lids reading open. Half of all faces + /// read above 0.43, so it has to sit far up. + pub reason_eye_squint_threshold: f64, /// A smile at least this broad with the eyes shut is a laugh, not a blink /// (per face, next to the scene-level CLIP gate). 0 switches it off. pub laugh_smile_threshold: f64, @@ -326,9 +330,30 @@ pub struct RankingConfig { /// A pick within this of the group's broadest smile is reported as having /// the best smile (`best_smile`). pub reason_smile_delta: f64, - /// Above this `eyeLook*` a frame is labelled `looking_away`. 0 switches the - /// label off; it is never a defect. - pub reason_gaze_away_threshold: f64, + /// Above this many degrees from the lens (`cull_gaze_off_lens`) a frame is + /// labelled `looking_away`. 0 switches the label off; it is never a defect. + pub reason_gaze_off_lens_degrees: f64, + /// `cull_mouth_open` from here up, on a frame with a sibling whose mouth is + /// [`Self::reason_mouth_open_delta`] less open, is labelled `mouth_open`. + /// Measured: 10% of photographs of people speaking reach 0.2, and 0.6% of + /// portraits. 0 switches the label off; it is never a defect. + pub reason_mouth_open_threshold: f64, + pub reason_mouth_open_delta: f64, + /// A frame whose darkest face is this many stops below the same reading in + /// its group's brightest frame is labelled `face_darker`. Only within a + /// group, since skin ranges over several stops between people and this + /// must never call dark skin underexposed. 0 switches the label off. + pub reason_face_darker_stops: f64, + /// `cull_face_blown` from here up is labelled `face_blown`. 0 switches it + /// off. Measured: 2.6% of faces pass 0.05, where surgical masks and white + /// headbands still sit, and from 0.15 what is left is blown skin. + pub reason_face_blown_threshold: f64, + /// `cull_face_across_stops` from here up, on a frame with a sibling whose + /// light is [`Self::reason_face_hard_light_delta`] stops softer, is labelled + /// `face_hard_light`. 10% of faces reach 2 stops, nearly all real side + /// light. 0 switches the label off. + pub reason_face_hard_light_stops: f64, + pub reason_face_hard_light_delta: f64, } /// Which deliberately-captured multi-frame set a group is, if any. @@ -462,7 +487,11 @@ impl CullingConfig { cfg.ranking.smile_weight = 0.0; cfg.ranking.laugh_smile_threshold = 0.0; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; - cfg.ranking.reason_gaze_away_threshold = 0.0; + cfg.ranking.reason_gaze_off_lens_degrees = 0.0; + cfg.ranking.reason_mouth_open_threshold = 0.0; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; cfg } } @@ -534,12 +563,19 @@ const BASE: CullingConfig = CullingConfig { relative_focus_margin: 0.08, reason_eye_focus_delta: 0.02, reason_eye_closed_threshold: 0.5, - reject_eye_closed_threshold: 0.6, + reject_eye_closed_threshold: 0.55, reason_eyes_half_closed_threshold: 0.35, + reason_eye_squint_threshold: 0.78, laugh_smile_threshold: 0.6, smile_weight: 0.0, reason_smile_delta: 0.05, - reason_gaze_away_threshold: 0.0, + reason_gaze_off_lens_degrees: 0.0, + reason_mouth_open_threshold: 0.0, + reason_mouth_open_delta: 0.15, + reason_face_darker_stops: 0.75, + reason_face_blown_threshold: 0.15, + reason_face_hard_light_stops: 2.0, + reason_face_hard_light_delta: 1.0, }, sets: SetDetectionConfig { enabled: true, @@ -593,9 +629,10 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { // smile per face is worth scoring, and eye contact is a question // worth answering. cfg.ranking.reason_eye_closed_threshold = 0.45; - cfg.ranking.reject_eye_closed_threshold = 0.55; + cfg.ranking.reject_eye_closed_threshold = 0.5; cfg.ranking.smile_weight = 0.15; - cfg.ranking.reason_gaze_away_threshold = 0.5; + cfg.ranking.reason_gaze_off_lens_degrees = 25.0; + cfg.ranking.reason_mouth_open_threshold = 0.2; } "street" => { cfg.ranking.face_group_weight_technical = 0.70; @@ -625,8 +662,11 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reason_face_focus_threshold = 0.0; cfg.ranking.reject_face_focus_threshold = 0.0; // Closed eyes are often the moment in street work. - cfg.ranking.reason_eye_closed_threshold = 0.75; - cfg.ranking.reject_eye_closed_threshold = 0.85; + cfg.ranking.reason_eye_closed_threshold = 0.62; + cfg.ranking.reject_eye_closed_threshold = 0.72; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; } "event" => { cfg.grouping.time_window_default_seconds = 2; @@ -660,6 +700,7 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.eyes_closed_intent_threshold = 0.7; // Everyone smiling is half of a group picture. cfg.ranking.smile_weight = 0.10; + cfg.ranking.reason_mouth_open_threshold = 0.2; } "sports" => { cfg.grouping.time_window_default_seconds = 3; @@ -698,9 +739,12 @@ pub fn get_culling_config(preset: &str) -> CullingConfig { cfg.ranking.reason_face_focus_threshold = 0.32; cfg.ranking.reject_face_focus_threshold = 0.25; // Athletes squint and grimace; only shut eyes count. - cfg.ranking.reason_eye_closed_threshold = 0.7; - cfg.ranking.reject_eye_closed_threshold = 0.8; + cfg.ranking.reason_eye_closed_threshold = 0.6; + cfg.ranking.reject_eye_closed_threshold = 0.68; cfg.ranking.reason_eyes_half_closed_threshold = 0.0; + cfg.ranking.reason_face_darker_stops = 0.0; + cfg.ranking.reason_face_blown_threshold = 0.0; + cfg.ranking.reason_face_hard_light_stops = 0.0; } _ => {} } diff --git a/server-rs/crates/lrg-analysis/src/edit_guardrails.rs b/server-rs/crates/lrg-analysis/src/edit_guardrails.rs index 3150fab6..57e74b89 100644 --- a/server-rs/crates/lrg-analysis/src/edit_guardrails.rs +++ b/server-rs/crates/lrg-analysis/src/edit_guardrails.rs @@ -14,10 +14,16 @@ //! surprising edit that explains itself is debuggable, and one that does not is //! indistinguishable from a bug. //! +//! The rules about faces read [`FaceConditions`], which the caller measures on +//! the same frame. Every one of them compares a face with itself or with its own +//! frame, and none calls a face too dark: skin ranges over several stops between +//! people, so a dark face in bright surroundings is just as often exposed +//! perfectly. All such a reading may do here is hold contrast back, which costs +//! that face nothing. +//! //! What is **not** here yet, because it needs signals this crate cannot reach: -//! the face-differential rule (needs face boxes), mixed-light detection (needs a -//! regional illuminant estimate) and the backlight rule (needs a subject -//! region). [`EditBudget`] is shaped so those slot in without changing callers. +//! mixed-light detection (needs a regional illuminant estimate). [`EditBudget`] +//! is shaped so it slots in without changing callers. use lrg_imaging::scene::SceneState; @@ -35,6 +41,21 @@ pub struct CaptureConditions { pub is_raw: bool, } +/// What was measured about the faces in the frame, as far as it limits an edit. +/// The worst of the faces that matter in the frame speaks for it, and `None` is +/// "not measured": no faces, or no face model on this computer. +#[derive(Debug, Clone, Copy, Default)] +pub struct FaceConditions { + /// Between the two cheeks of the hardest-lit face, in stops. + pub across_face_stops: Option, + /// The darkest face against the frame around it, in stops; negative is + /// darker. A backlit face reads low, and so does dark skin in bright + /// surroundings, which is why this only ever holds contrast back. + pub against_surround_stops: Option, + /// The largest share of a face's skin that is blown out, `0..1`. + pub blown: Option, +} + /// Tunables for [`derive_budget`]. /// /// Every threshold here is a starting guess, chosen to be defensible rather @@ -66,6 +87,17 @@ pub struct GuardrailConfig { pub contrast_budget_flat: f64, /// Largest clarity value on an ordinary frame. pub clarity_budget_default: f64, + /// Light this many stops apart between the two cheeks is hard light on the + /// face. 10% of faces in a sample of 600 portraits reach 2, nearly all of + /// them real side light. + pub face_hard_light_stops: f64, + /// A face this many stops below the frame around it. 2.4% of that sample. + pub face_below_surround_stops: f64, + /// This share of a face's skin blown out; below it sit surgical masks and + /// white headbands. + pub face_blown_share: f64, + /// Largest total of contrast-increasing moves on a face that needs it. + pub contrast_budget_face: f64, } impl GuardrailConfig { @@ -81,6 +113,10 @@ impl GuardrailConfig { contrast_budget_default: 25.0, contrast_budget_flat: 40.0, clarity_budget_default: 20.0, + face_hard_light_stops: 2.0, + face_below_surround_stops: 2.0, + face_blown_share: 0.15, + contrast_budget_face: 10.0, } } } @@ -107,6 +143,12 @@ pub enum GuardrailReason { HighIsoShadowsLimited, /// Shadows are already blocked; lifting them cannot recover detail. ShadowsAlreadyClipped, + /// Hard light across a face: clarity and contrast would deepen the split. + FaceHardLight, + /// A face well below the frame around it: contrast would push it further. + FaceBelowSurroundings, + /// Blown skin on a face: the white point must not push it further. + FaceBlown, } impl GuardrailReason { @@ -120,6 +162,9 @@ impl GuardrailReason { GuardrailReason::HighlightsUnrecoverable => "highlights_unrecoverable", GuardrailReason::HighIsoShadowsLimited => "high_iso_shadows_limited", GuardrailReason::ShadowsAlreadyClipped => "shadows_already_clipped", + GuardrailReason::FaceHardLight => "face_hard_light", + GuardrailReason::FaceBelowSurroundings => "face_below_surroundings", + GuardrailReason::FaceBlown => "face_blown", } } @@ -143,6 +188,15 @@ impl GuardrailReason { GuardrailReason::ShadowsAlreadyClipped => { "Shadows are already blocked, so lifting them further would not recover detail." } + GuardrailReason::FaceHardLight => { + "Hard light falls across a face, so no clarity was added and contrast was limited: both would deepen the split." + } + GuardrailReason::FaceBelowSurroundings => { + "A face sits well below the frame around it, so contrast was limited to keep it from going darker." + } + GuardrailReason::FaceBlown => { + "Skin on a face is blown out, so the white point was held back." + } } } } @@ -286,6 +340,43 @@ pub fn derive_budget( budget } +/// What this frame can absorb, faces included. See [`FaceConditions`] for why +/// these rules only ever remove, and never call a face too dark. +pub fn derive_budget_with_faces( + scene: &SceneState, + capture: &CaptureConditions, + faces: &FaceConditions, + cfg: &GuardrailConfig, +) -> EditBudget { + let mut budget = derive_budget(scene, capture, cfg); + if faces + .across_face_stops + .is_some_and(|s| s >= cfg.face_hard_light_stops) + { + budget.clarity_budget = 0.0; + budget.contrast_budget = budget.contrast_budget.min(cfg.contrast_budget_face); + budget.reasons.push(GuardrailReason::FaceHardLight); + } + if faces + .against_surround_stops + .is_some_and(|s| s <= -cfg.face_below_surround_stops) + { + budget.contrast_budget = budget.contrast_budget.min(cfg.contrast_budget_face); + budget.reasons.push(GuardrailReason::FaceBelowSurroundings); + } + // The same split as highlights in general: raw usually holds something above + // the rendering, a JPEG never does. + if faces.blown.is_some_and(|b| b >= cfg.face_blown_share) { + budget.whites_budget = if capture.is_raw { + budget.whites_budget.min(cfg.contrast_budget_default * 0.5) + } else { + 0.0 + }; + budget.reasons.push(GuardrailReason::FaceBlown); + } + budget +} + /// The contrast-increasing moves in a recipe, summed. /// /// Only the increases count. A profile that pulls contrast down and raises @@ -504,4 +595,103 @@ mod tests { assert!(!reason.explanation().is_empty()); } } + + fn with_faces(faces: FaceConditions, is_raw: bool) -> EditBudget { + derive_budget_with_faces( + &scene(0.3, 5.0), + &CaptureConditions { iso: None, is_raw }, + &faces, + &cfg(), + ) + } + + #[test] + fn no_face_reading_leaves_the_budget_as_it_was() { + let plain = derive_budget(&scene(0.3, 5.0), &CaptureConditions::default(), &cfg()); + let budget = with_faces(FaceConditions::default(), false); + assert_eq!(budget.contrast_budget, plain.contrast_budget); + assert_eq!(budget.clarity_budget, plain.clarity_budget); + assert_eq!(budget.whites_budget, plain.whites_budget); + assert!(budget.reasons.is_empty()); + } + + #[test] + fn hard_light_across_a_face_takes_clarity_and_most_of_the_contrast() { + let budget = with_faces( + FaceConditions { + across_face_stops: Some(3.2), + ..Default::default() + }, + false, + ); + assert_eq!(budget.clarity_budget, 0.0); + assert_eq!(budget.contrast_budget, cfg().contrast_budget_face); + assert_eq!(budget.reasons, vec![GuardrailReason::FaceHardLight]); + + let soft = with_faces( + FaceConditions { + across_face_stops: Some(0.8), + ..Default::default() + }, + false, + ); + assert!(soft.reasons.is_empty()); + } + + /// This reading is as low on dark skin in bright surroundings as on a + /// backlit face, so it may only ever hold contrast back: never the shadows a + /// dim face could use, never the clarity. + #[test] + fn a_face_below_its_surroundings_only_holds_contrast_back() { + let plain = with_faces(FaceConditions::default(), false); + let budget = with_faces( + FaceConditions { + against_surround_stops: Some(-3.1), + ..Default::default() + }, + false, + ); + assert_eq!(budget.contrast_budget, cfg().contrast_budget_face); + assert_eq!(budget.shadow_lift_budget, plain.shadow_lift_budget); + assert_eq!(budget.clarity_budget, plain.clarity_budget); + assert_eq!(budget.reasons, vec![GuardrailReason::FaceBelowSurroundings]); + + let brighter = with_faces( + FaceConditions { + against_surround_stops: Some(1.5), + ..Default::default() + }, + false, + ); + assert!(brighter.reasons.is_empty()); + } + + #[test] + fn blown_skin_holds_the_white_point_back_by_what_the_source_can_recover() { + let blown = FaceConditions { + blown: Some(0.4), + ..Default::default() + }; + assert_eq!(with_faces(blown, false).whites_budget, 0.0); + assert!(with_faces(blown, true).whites_budget > 0.0); + assert_eq!( + with_faces(blown, false).reasons, + vec![GuardrailReason::FaceBlown] + ); + } + + /// A ceiling never rises: a flat frame's wider contrast still gives way. + #[test] + fn face_rules_only_ever_remove() { + let flat = derive_budget_with_faces( + &scene(0.1, 3.0), + &CaptureConditions::default(), + &FaceConditions { + across_face_stops: Some(3.0), + ..Default::default() + }, + &cfg(), + ); + assert_eq!(flat.contrast_budget, cfg().contrast_budget_face); + } } diff --git a/server-rs/crates/lrg-analysis/src/face_aggregate.rs b/server-rs/crates/lrg-analysis/src/face_aggregate.rs index 6f3e194f..d3077a4b 100644 --- a/server-rs/crates/lrg-analysis/src/face_aggregate.rs +++ b/server-rs/crates/lrg-analysis/src/face_aggregate.rs @@ -39,10 +39,23 @@ pub struct FaceMetricsInput { /// (`lrg_ml::face_state`). `None` when the face state did not run for this /// face or could not judge the eyes (turned away, too small). pub eye_closed: Option, + /// The lids are down on a lowered head (`EyeState::Down`). Such a face's + /// `eye_closed` is `None`: looking down is not a blink. + pub looking_down: bool, /// `mouthSmile`, `0..1`; `None` like `eye_closed`. pub smile: Option, - /// The strongest `eyeLook*` coefficient, `0..1`; 0 is looking into the lens. - pub gaze_away: Option, + /// `eyeSquint`, `0..1`; `None` like `eye_closed`. + pub eye_squint: Option, + /// How far from the lens the eyes point, in degrees, head turn included; + /// `None` like `eye_closed`. + pub gaze_off_lens: Option, + /// How far the mouth is open, `0..1`; `None` like `eye_closed`. + pub mouth_open: Option, + /// The light on the skin (`lrg_ml::face_state::FaceLight`); `None` like + /// `eye_closed`, or when no skin patch fell inside the image. + pub luminance: Option, + pub across_face_stops: Option, + pub blown: Option, } #[derive(Debug, Clone, PartialEq)] @@ -64,13 +77,29 @@ pub struct AggregatedFaceMetrics { /// Eyes of the prominent face whose eyes are the most closed, from the /// face state model; `None` when no prominent face's eyes could be judged. pub cull_eye_closed: Option, - /// That reading as `open`/`half`/`closed`, at the face-state thresholds. + /// That reading as `open`/`half`/`closed`, at the face-state thresholds, or + /// `down` when eyes that would read open are next to a face looking down, or + /// no other face could be judged. pub cull_eye_state: Option<&'static str>, + /// A prominent face is looking down. + pub cull_looking_down: bool, /// The weakest smile among the prominent faces: a group only smiles when /// everyone in it does. pub cull_smile: Option, - /// The prominent face looking furthest away from the lens. - pub cull_gaze_away: Option, + /// The strongest squint among the prominent faces that are not smiling; a + /// broad smile narrows the eyes too, and that is not a squint. + pub cull_eye_squint: Option, + /// The prominent face looking furthest from the lens, in degrees. + pub cull_gaze_off_lens: Option, + /// The most open mouth among the prominent faces not smiling. + pub cull_mouth_open: Option, + /// The darkest prominent face. Only ever compared with this same frame's + /// siblings: skin ranges over several stops between people. + pub cull_face_luminance: Option, + /// The hardest light across a prominent face, in stops. + pub cull_face_across_stops: Option, + /// The largest share of a prominent face's skin that is blown out. + pub cull_face_blown: Option, } impl AggregatedFaceMetrics { @@ -89,8 +118,14 @@ impl AggregatedFaceMetrics { cull_face_focus: None, cull_eye_closed: None, cull_eye_state: None, + cull_looking_down: false, cull_smile: None, - cull_gaze_away: None, + cull_eye_squint: None, + cull_gaze_off_lens: None, + cull_mouth_open: None, + cull_face_luminance: None, + cull_face_across_stops: None, + cull_face_blown: None, } } } @@ -198,16 +233,48 @@ pub fn aggregate_face_culling_metrics( }; let eye_closed_agg = measured(|f| f.eye_closed).reduce(f64::max); let smile_agg = measured(|f| f.smile).reduce(f64::min); - let gaze_away_agg = measured(|f| f.gaze_away).reduce(f64::max); - let eye_state_agg = eye_closed_agg.map(|v| { - if v >= cfg.face_state_closed_threshold { - "closed" - } else if v >= cfg.face_state_half_threshold { - "half" - } else { - "open" - } - }); + let eye_squint_agg = prominent + .iter() + .map(|&i| &faces[i]) + .filter(|f| f.smile.is_some_and(|s| s < cfg.face_state_squint_max_smile)) + .filter_map(|f| f.eye_squint.map(unit)) + .reduce(f64::max); + // Degrees, so not through `measured`'s clamp to `0..1`. Shut eyes look + // nowhere: their `eyeLook*` is noise (31° on a portrait with the eyes shut), + // so such a face abstains. + let gaze_off_lens_agg = prominent + .iter() + .map(|&i| &faces[i]) + .filter(|f| { + f.eye_closed + .is_none_or(|c| c < cfg.face_state_closed_threshold) + }) + .filter_map(|f| f.gaze_off_lens) + .reduce(f64::max); + let mouth_open_agg = prominent + .iter() + .map(|&i| &faces[i]) + // A lowered head foreshortens the lips, and a closed mouth reads open. + .filter(|f| !f.looking_down) + .filter(|f| f.smile.is_some_and(|s| s < cfg.face_state_mouth_max_smile)) + .filter_map(|f| f.mouth_open.map(unit)) + .reduce(f64::max); + let luminance_agg = measured(|f| f.luminance).reduce(f64::min); + let blown_agg = measured(|f| f.blown).reduce(f64::max); + // Stops, so not through `measured`'s clamp to `0..1`. + let across_agg = prominent + .iter() + .filter_map(|&i| faces[i].across_face_stops) + .reduce(f64::max); + let looking_down_agg = prominent.iter().any(|&i| faces[i].looking_down); + let eye_state_agg = match eye_closed_agg { + Some(v) if v >= cfg.face_state_closed_threshold => Some("closed"), + Some(v) if v >= cfg.face_state_half_threshold => Some("half"), + // Someone's eyes are not on the frame, so "everyone's eyes open" is out. + _ if looking_down_agg => Some("down"), + Some(_) => Some("open"), + None => None, + }; let weight_total = cfg.score_weight_sharpness + cfg.score_weight_prominence @@ -235,8 +302,14 @@ pub fn aggregate_face_culling_metrics( cull_face_focus: face_focus_agg.map(round4), cull_eye_closed: eye_closed_agg.map(round4), cull_eye_state: eye_state_agg, + cull_looking_down: looking_down_agg, cull_smile: smile_agg.map(round4), - cull_gaze_away: gaze_away_agg.map(round4), + cull_eye_squint: eye_squint_agg.map(round4), + cull_gaze_off_lens: gaze_off_lens_agg.map(round4), + cull_mouth_open: mouth_open_agg.map(round4), + cull_face_luminance: luminance_agg.map(round4), + cull_face_across_stops: across_agg.map(round4), + cull_face_blown: blown_agg.map(round4), } } @@ -269,8 +342,14 @@ mod tests { eye_focus: Some(0.7), face_focus: Some(0.55), eye_closed: Some(0.03), + looking_down: false, smile: Some(0.9), - gaze_away: Some(0.1), + eye_squint: Some(0.4), + gaze_off_lens: Some(4.0), + mouth_open: Some(0.02), + luminance: Some(0.3), + across_face_stops: Some(0.4), + blown: Some(0.0), }]; let m = aggregate_face_culling_metrics(&faces, &fm()); assert_eq!(m.cull_face_count, 1); @@ -413,14 +492,14 @@ mod tests { area_ratio: 0.20, eye_closed: Some(0.04), smile: Some(0.9), - gaze_away: Some(0.1), + gaze_off_lens: Some(32.5), ..Default::default() }, FaceMetricsInput { area_ratio: 0.18, eye_closed: Some(0.81), smile: Some(0.6), - gaze_away: Some(0.2), + gaze_off_lens: Some(40.0), ..Default::default() }, // Turned away: the model could not judge the eyes. @@ -433,7 +512,7 @@ mod tests { area_ratio: 0.01, eye_closed: Some(0.99), smile: Some(0.0), - gaze_away: Some(0.9), + gaze_off_lens: Some(70.0), ..Default::default() }, ]; @@ -441,7 +520,7 @@ mod tests { assert_eq!(m.cull_eye_closed, Some(0.81)); assert_eq!(m.cull_eye_state, Some("closed")); assert_eq!(m.cull_smile, Some(0.6)); - assert_eq!(m.cull_gaze_away, Some(0.2)); + assert_eq!(m.cull_gaze_off_lens, Some(32.5)); let open = aggregate_face_culling_metrics(&faces[..1], &fm()); assert_eq!(open.cull_eye_state, Some("open")); @@ -452,6 +531,110 @@ mod tests { ); } + #[test] + fn a_face_looking_down_is_no_blink_but_the_eyes_are_not_all_open() { + let down = FaceMetricsInput { + area_ratio: 0.2, + looking_down: true, + ..Default::default() + }; + let open = FaceMetricsInput { + area_ratio: 0.2, + eye_closed: Some(0.05), + ..Default::default() + }; + let shut = FaceMetricsInput { + area_ratio: 0.2, + eye_closed: Some(0.7), + ..Default::default() + }; + let alone = aggregate_face_culling_metrics(std::slice::from_ref(&down), &fm()); + assert!(alone.cull_looking_down); + assert_eq!(alone.cull_eye_closed, None); + assert_eq!(alone.cull_eye_state, Some("down")); + + let with_open = aggregate_face_culling_metrics(&[down, open], &fm()); + assert_eq!(with_open.cull_eye_closed, Some(0.05)); + assert_eq!(with_open.cull_eye_state, Some("down")); + + // A real blink next to it still speaks. + let with_shut = aggregate_face_culling_metrics(&[down, shut], &fm()); + assert_eq!(with_shut.cull_eye_state, Some("closed")); + } + + /// A laugh is open too, and not what the mouth reading is after. + #[test] + fn an_open_mouth_only_counts_on_a_face_that_is_not_smiling() { + let face = |open, smile| FaceMetricsInput { + area_ratio: 0.2, + mouth_open: Some(open), + smile: Some(smile), + ..Default::default() + }; + let laugh = aggregate_face_culling_metrics(&[face(0.6, 0.8)], &fm()); + assert_eq!(laugh.cull_mouth_open, None); + let word = aggregate_face_culling_metrics(&[face(0.3, 0.1)], &fm()); + assert_eq!(word.cull_mouth_open, Some(0.3)); + let lowered = FaceMetricsInput { + looking_down: true, + ..face(0.44, 0.0) + }; + let lowered = aggregate_face_culling_metrics(&[lowered], &fm()); + assert_eq!(lowered.cull_mouth_open, None); + } + + #[test] + fn the_light_takes_the_worst_prominent_face() { + let face = |luminance, across, blown| FaceMetricsInput { + area_ratio: 0.2, + luminance: Some(luminance), + across_face_stops: Some(across), + blown: Some(blown), + ..Default::default() + }; + let group = [face(0.4, 0.5, 0.0), face(0.2, 3.1, 0.3)]; + let m = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(m.cull_face_luminance, Some(0.2)); + assert_eq!(m.cull_face_across_stops, Some(3.1)); + assert_eq!(m.cull_face_blown, Some(0.3)); + } + + #[test] + fn eyes_that_are_shut_do_not_look_away() { + let face = |closed, off| FaceMetricsInput { + area_ratio: 0.2, + eye_closed: Some(closed), + gaze_off_lens: Some(off), + ..Default::default() + }; + let shut = aggregate_face_culling_metrics(&[face(0.66, 31.0)], &fm()); + assert_eq!(shut.cull_gaze_off_lens, None); + let group = [face(0.66, 31.0), face(0.05, 12.0)]; + let group = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(group.cull_gaze_off_lens, Some(12.0)); + } + + /// `eyeSquint` runs high on a broad smile, so only an unsmiling face's + /// counts as a squint; a face the smile was not read on says nothing. + #[test] + fn a_squint_only_counts_on_a_face_that_is_not_smiling() { + let face = |squint, smile: Option| FaceMetricsInput { + area_ratio: 0.2, + eye_squint: Some(squint), + smile, + ..Default::default() + }; + let smiling = aggregate_face_culling_metrics(&[face(0.9, Some(0.8))], &fm()); + assert_eq!(smiling.cull_eye_squint, None); + let squinting = aggregate_face_culling_metrics(&[face(0.84, Some(0.1))], &fm()); + assert_eq!(squinting.cull_eye_squint, Some(0.84)); + let group = [face(0.9, Some(0.8)), face(0.6, Some(0.1))]; + let group = aggregate_face_culling_metrics(&group, &fm()); + assert_eq!(group.cull_eye_squint, Some(0.6)); + let unread = aggregate_face_culling_metrics(&[face(0.9, None)], &fm()); + assert_eq!(unread.cull_eye_squint, None); + } + #[test] fn eye_focus_is_the_least_sharp_measured_prominent_face() { let faces = vec![ diff --git a/server-rs/crates/lrg-analysis/src/face_expression.rs b/server-rs/crates/lrg-analysis/src/face_expression.rs new file mode 100644 index 00000000..3d5b92c2 --- /dev/null +++ b/server-rs/crates/lrg-analysis/src/face_expression.rs @@ -0,0 +1,375 @@ +//! What the stored face state says about a face or a photo, for the workflows +//! that ask a person's question of it: "her with the eyes open", "the best +//! portrait of him", "everyone smiling at the camera". +//! +//! Culling compares frames of one burst and can lean on the group. These +//! questions are asked of a whole catalog, so there is no group to lean on and +//! each answer has to hold alone. That is why a reading the model did not make +//! is [`Verdict::Unknown`] and not a no: a photo indexed before the model was on +//! disk is a data problem the caller reports, where a frown is simply not a +//! smile. + +use serde_json::{Map, Value}; + +use crate::grouping::face_state_is_current; + +/// `mouthSmile` from here up reads as smiling. In portraits judged by eye, 0.35 to +/// 0.5 is at most the hint of one, and from 0.5 most faces show a smile, closed +/// mouth included. A snarl can reach it too. +pub const SMILING_FROM: f64 = 0.5; + +/// Eyes this near the lens read as looking at the camera, head turn included. +/// Portraits have a median of 9°; a high-angle selfie reads 15–20° for looking +/// straight at the lens, because `eyeLookUp` barely moves. `looking_away` starts +/// further out (25°), so a face between the two is claimed by neither. +pub const LOOKING_AT_CAMERA_WITHIN_DEGREES: f64 = 20.0; + +/// What [`portrait_score`] charges a row with no reading. +const UNREAD_COST: f64 = 0.3; + +/// Answer to one question about a face or a photo. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Verdict { + Yes, + No, + /// The model made no such reading, so the question cannot be answered. + Unknown, +} + +impl Verdict { + fn and(self, other: Verdict) -> Verdict { + match (self, other) { + (Verdict::No, _) | (_, Verdict::No) => Verdict::No, + (Verdict::Unknown, _) | (_, Verdict::Unknown) => Verdict::Unknown, + _ => Verdict::Yes, + } + } + + fn of(reading: Option) -> Verdict { + match reading { + Some(true) => Verdict::Yes, + Some(false) => Verdict::No, + None => Verdict::Unknown, + } + } +} + +/// What a caller asks of a face or a photo. Empty asks nothing. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ExpressionFilter { + pub eyes_open: bool, + pub smiling: bool, + pub looking_at_camera: bool, +} + +impl ExpressionFilter { + pub fn is_empty(&self) -> bool { + !(self.eyes_open || self.smiling || self.looking_at_camera) + } + + /// In plain words, for a set's label or a message: "eyes open and smiling". + pub fn describe(&self) -> String { + let mut parts = Vec::new(); + if self.eyes_open { + parts.push("eyes open"); + } + if self.smiling { + parts.push("smiling"); + } + if self.looking_at_camera { + parts.push("looking at the camera"); + } + parts.join(" and ") + } + + fn judge( + &self, + eye_state: Option<&str>, + smile: Option, + gaze_off_lens: Option, + ) -> Verdict { + // "unknown" is the model saying it cannot tell, the same as no reading. + let eye_state = eye_state.filter(|s| *s != "unknown"); + let mut verdict = Verdict::Yes; + if self.eyes_open { + verdict = verdict.and(Verdict::of(eye_state.map(|s| s == "open"))); + } + if self.smiling { + verdict = verdict.and(Verdict::of(smile.map(|s| s >= SMILING_FROM))); + } + if self.looking_at_camera { + // Shut eyes look nowhere, whatever `eyeLook*` says, and eyes going + // down are not on the lens. Both are answers, not missing readings. + let looking = if matches!(eye_state, Some("closed" | "down")) { + Some(false) + } else { + gaze_off_lens.map(|g| g <= LOOKING_AT_CAMERA_WITHIN_DEGREES) + }; + verdict = verdict.and(Verdict::of(looking)); + } + verdict + } + + /// Judges one row of the face table. + pub fn judge_face(&self, face: &Map) -> Verdict { + self.judge( + face.get("face_eye_state").and_then(Value::as_str), + face.get("face_smile").and_then(Value::as_f64), + face.get("face_gaze_off_lens").and_then(Value::as_f64), + ) + } + + /// Judges a photo by its stored aggregates, which speak for the people who + /// matter in the frame together: everyone's eyes open, everyone smiling. + pub fn judge_photo(&self, photo: &Map) -> Verdict { + // Measured, and no faces: not a smile, and nothing the user could fix. + if photo.get("cull_face_count").and_then(Value::as_i64) == Some(0) { + return Verdict::No; + } + if !face_state_is_current(photo) { + return Verdict::Unknown; + } + self.judge( + photo.get("cull_eye_state").and_then(Value::as_str), + photo.get("cull_smile").and_then(Value::as_f64), + photo.get("cull_gaze_off_lens").and_then(Value::as_f64), + ) + } +} + +/// How good a portrait of its person a face row makes, higher better: for the +/// picture on a person's card, and for "the best photos of her". +/// +/// Starts from what the detector and the sharpness measure say, which is all a +/// row from before the face state has. Then a shut eye costs more than anything +/// else can earn: a person's card showing them blinking is the failure this +/// exists for. A row with no reading at all pays a fixed price: nobody has seen +/// its eyes, so it should not outrank a portrait known to be good, and among such +/// rows the order stays what it always was. +pub fn portrait_score(face: &Map) -> f64 { + let number = |key: &str| face.get(key).and_then(Value::as_f64); + let mut score = + number("face_det_score").unwrap_or(0.0) + number("face_sharpness").unwrap_or(0.0); + match face.get("face_eye_state").and_then(Value::as_str) { + Some("closed") => score -= 1.0, + Some("half" | "down") => score -= 0.4, + Some(_) => {} + None => score -= UNREAD_COST, + } + if let Some(yaw) = number("face_yaw") { + score -= 0.5 * (yaw.abs() / 60.0).clamp(0.0, 1.0); + } + if let Some(gaze) = number("face_gaze_off_lens") { + score -= 0.3 * (gaze / 45.0).clamp(0.0, 1.0); + } + if let Some(smile) = number("face_smile") { + score += 0.2 * smile.clamp(0.0, 1.0); + } + score +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::grouping::{FACE_STATE_VERSION, FACE_STATE_VERSION_KEY}; + use serde_json::json; + + fn row(value: Value) -> Map { + value.as_object().unwrap().clone() + } + + fn filter(eyes_open: bool, smiling: bool, looking_at_camera: bool) -> ExpressionFilter { + ExpressionFilter { + eyes_open, + smiling, + looking_at_camera, + } + } + + #[test] + fn an_empty_filter_asks_nothing_and_passes_anything() { + assert!(ExpressionFilter::default().is_empty()); + assert_eq!( + ExpressionFilter::default().judge_face(&row(json!({}))), + Verdict::Yes + ); + } + + #[test] + fn a_face_answers_each_question_from_its_own_reading() { + let face = row(json!({ + "face_eye_state": "open", + "face_smile": 0.8, + "face_gaze_off_lens": 6.0, + })); + assert_eq!(filter(true, true, true).judge_face(&face), Verdict::Yes); + + let frowning = row(json!({"face_eye_state": "open", "face_smile": 0.1})); + assert_eq!( + filter(false, true, false).judge_face(&frowning), + Verdict::No + ); + + let blinking = row(json!({"face_eye_state": "closed"})); + assert_eq!( + filter(true, false, false).judge_face(&blinking), + Verdict::No + ); + + let away = row(json!({"face_gaze_off_lens": 38.0})); + assert_eq!(filter(false, false, true).judge_face(&away), Verdict::No); + } + + /// A photo that could not be judged is a data problem the caller reports; + /// one judged and found wanting is simply not a match. + #[test] + fn a_missing_reading_is_unknown_and_a_no_outranks_it() { + let empty = row(json!({})); + assert_eq!( + filter(true, false, false).judge_face(&empty), + Verdict::Unknown + ); + let unsure = row(json!({"face_eye_state": "unknown"})); + assert_eq!( + filter(true, false, false).judge_face(&unsure), + Verdict::Unknown + ); + // Shut eyes settle it, whatever else is missing. + let shut = row(json!({"face_eye_state": "closed"})); + assert_eq!(filter(true, true, false).judge_face(&shut), Verdict::No); + } + + #[test] + fn a_photo_speaks_for_everyone_in_it() { + let current = |extra: Value| { + let mut m = row(extra); + m.insert(FACE_STATE_VERSION_KEY.into(), json!(FACE_STATE_VERSION)); + m.insert("cull_face_count".into(), json!(2)); + m + }; + let group = current(json!({ + "cull_eye_state": "open", + "cull_smile": 0.7, + "cull_gaze_off_lens": 12.0, + })); + assert_eq!(filter(true, true, true).judge_photo(&group), Verdict::Yes); + let one_blink = current(json!({"cull_eye_state": "closed"})); + assert_eq!( + filter(true, false, false).judge_photo(&one_blink), + Verdict::No + ); + } + + #[test] + fn a_photo_without_faces_is_no_and_one_not_analysed_is_unknown() { + let landscape = row(json!({"cull_face_count": 0})); + assert_eq!( + filter(false, true, false).judge_photo(&landscape), + Verdict::No + ); + + let never_prepared = row(json!({})); + assert_eq!( + filter(false, true, false).judge_photo(&never_prepared), + Verdict::Unknown + ); + + // An older analysis means something else by the same fields. + let stale = row(json!({ + "cull_face_count": 1, + "cull_smile": 0.9, + FACE_STATE_VERSION_KEY: FACE_STATE_VERSION - 1, + })); + assert_eq!( + filter(false, true, false).judge_photo(&stale), + Verdict::Unknown + ); + } + + #[test] + fn eyes_going_down_are_neither_open_nor_on_the_camera() { + let face = row(json!({"face_eye_state": "down", "face_gaze_off_lens": 12.0})); + assert_eq!(filter(true, false, false).judge_face(&face), Verdict::No); + assert_eq!(filter(false, false, true).judge_face(&face), Verdict::No); + } + + /// Shut eyes look nowhere: a plain no, not a reading to go and fetch, and not + /// a yes when `eyeLook*` happens to read near the lens. + #[test] + fn shut_eyes_are_not_looking_at_the_camera() { + let face = row(json!({"face_eye_state": "closed", "face_gaze_off_lens": 4.0})); + assert_eq!(filter(false, false, true).judge_face(&face), Verdict::No); + let photo = row(json!({ + "cull_face_count": 1, + "cull_eye_state": "closed", + FACE_STATE_VERSION_KEY: FACE_STATE_VERSION, + })); + assert_eq!(filter(false, false, true).judge_photo(&photo), Verdict::No); + } + + #[test] + fn the_filter_says_what_it_asks() { + assert_eq!( + filter(true, true, false).describe(), + "eyes open and smiling" + ); + assert_eq!( + filter(false, false, true).describe(), + "looking at the camera" + ); + } + + /// The card showing someone blinking is what this exists to prevent: a + /// sharper, better detected frame with the eyes shut still loses. + #[test] + fn a_sharp_blink_loses_to_a_softer_frame_with_the_eyes_open() { + let blink = row(json!({ + "face_det_score": 0.98, + "face_sharpness": 0.9, + "face_eye_state": "closed", + "face_yaw": 2.0, + "face_gaze_off_lens": 5.0, + })); + let open = row(json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_yaw": 8.0, + "face_gaze_off_lens": 8.0, + })); + assert!(portrait_score(&open) > portrait_score(&blink)); + } + + #[test] + fn a_frontal_face_beats_a_turned_one_and_a_smile_breaks_the_tie() { + let base = json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_gaze_off_lens": 8.0, + }); + let at = |key: &str, value: f64| { + let mut m = row(base.clone()); + m.insert(key.into(), json!(value)); + m + }; + assert!(portrait_score(&at("face_yaw", 3.0)) > portrait_score(&at("face_yaw", 45.0))); + assert!(portrait_score(&at("face_smile", 0.9)) > portrait_score(&at("face_smile", 0.1))); + } + + /// Nobody has seen its eyes: it yields to a portrait known to be good, and + /// such rows keep the order among themselves that they always had. + #[test] + fn a_row_without_a_reading_yields_to_a_good_portrait_and_keeps_its_order() { + let old = |sharpness: f64| row(json!({"face_det_score": 0.9, "face_sharpness": sharpness})); + assert!(portrait_score(&old(0.8)) > portrait_score(&old(0.5))); + let known_good = row(json!({ + "face_det_score": 0.9, + "face_sharpness": 0.5, + "face_eye_state": "open", + "face_yaw": 5.0, + "face_gaze_off_lens": 8.0, + })); + assert!(portrait_score(&known_good) > portrait_score(&old(0.5))); + } +} diff --git a/server-rs/crates/lrg-analysis/src/grouping.rs b/server-rs/crates/lrg-analysis/src/grouping.rs index 4298f4f2..b75fd97c 100644 --- a/server-rs/crates/lrg-analysis/src/grouping.rs +++ b/server-rs/crates/lrg-analysis/src/grouping.rs @@ -248,10 +248,21 @@ struct Scored<'a> { eyes_soft: bool, face_soft: bool, /// From the face state model (`cull_eye_closed`, `cull_smile`, - /// `cull_gaze_away`); `None` when no prominent face could be judged. + /// `cull_eye_squint`, `cull_gaze_off_lens`); `None` when no prominent face + /// could be judged. The last is in degrees, the others `0..1`. cull_eye_closed: Option, cull_smile: Option, - cull_gaze_away: Option, + cull_eye_squint: Option, + cull_gaze_off_lens: Option, + /// Labels only, until a hand-culled shoot says what they are worth to the + /// ranking: `cull_mouth_open`, `cull_face_luminance`, + /// `cull_face_across_stops` (in stops) and `cull_face_blown`. + cull_mouth_open: Option, + cull_face_luminance: Option, + cull_face_across_stops: Option, + cull_face_blown: Option, + /// A prominent face has its lids down on a lowered head: looking down. + looking_down: bool, /// The photo went through the face state pass (`cull_face_state_version` /// is stored), whether or not its faces could be judged. state_measured: bool, @@ -303,8 +314,20 @@ pub const FOCUS_VERSION: i64 = 1; pub const FACE_STATE_VERSION_KEY: &str = "cull_face_state_version"; /// The value stored under [`FACE_STATE_VERSION_KEY`]. Bump it when stored -/// readings become incomparable with new ones. -pub const FACE_STATE_VERSION: i64 = 1; +/// readings become incomparable with new ones; `/index/unprocessed` then has +/// those photos analysed again, and until then they rank like photos without +/// the model. +/// +/// 2: `cull_eye_closed` is the more closed eye instead of the mean of both, and +/// `cull_gaze_off_lens` replaces `cull_gaze_away`. +/// 3: lids down on a lowered head read `down`, no longer closed. +pub const FACE_STATE_VERSION: i64 = 3; + +/// Whether a photo's face state was read by this version of the analysis. The +/// key's value counts, not its presence, or a bump would change nothing. +pub fn face_state_is_current(metadata: &serde_json::Map) -> bool { + metadata.get(FACE_STATE_VERSION_KEY).and_then(Value::as_i64) == Some(FACE_STATE_VERSION) +} /// How much eye focus one point of the 0–10 in the explanation is worth. /// @@ -368,7 +391,12 @@ fn explanation_from_reason_codes(codes: &[String]) -> String { "eyes_closed" => "eyes closed".to_string(), "eyes_half_closed" => "eyes half closed or squinting".to_string(), "looking_away" => "looking away from the camera".to_string(), + "looking_down" => "looking down, which is not counted as a blink".to_string(), "best_smile" => "best smile in group".to_string(), + "mouth_open" => "mouth open, caught mid-word — a sibling has it closed".to_string(), + "face_darker" => "the face is darker than in a sibling frame".to_string(), + "face_blown" => "the skin on the face is blown out".to_string(), + "face_hard_light" => "harder light across the face than in a sibling frame".to_string(), "near_duplicate_weaker" => "weaker duplicate or burst alternative".to_string(), other => other.replace('_', " "), } @@ -536,12 +564,15 @@ fn rank_group_records( let cull_face_focus = m.get("cull_face_focus").and_then(Value::as_f64).map(unit); let focus_measured = m.contains_key(FOCUS_VERSION_KEY); let read = |key: &str| m.get(key).and_then(Value::as_f64).map(unit); - let (cull_eye_closed, cull_smile, cull_gaze_away) = ( + let (cull_eye_closed, cull_smile, cull_eye_squint) = ( read("cull_eye_closed"), read("cull_smile"), - read("cull_gaze_away"), + read("cull_eye_squint"), ); - let state_measured = m.contains_key(FACE_STATE_VERSION_KEY); + // Degrees, so not through `read`'s clamp to `0..1`. + let cull_gaze_off_lens = m.get("cull_gaze_off_lens").and_then(Value::as_f64); + let cull_face_across_stops = m.get("cull_face_across_stops").and_then(Value::as_f64); + let state_measured = face_state_is_current(m); // Injected by the API layer only when the preset asks the gate // question, so an absent score never lifts a blink. let intent_threshold = cfg.ranking.eyes_closed_intent_threshold; @@ -575,7 +606,16 @@ fn rank_group_records( face_soft: false, cull_eye_closed, cull_smile, - cull_gaze_away, + cull_eye_squint, + cull_gaze_off_lens, + cull_mouth_open: read("cull_mouth_open"), + cull_face_luminance: read("cull_face_luminance"), + cull_face_across_stops, + cull_face_blown: read("cull_face_blown"), + looking_down: m + .get("cull_looking_down") + .and_then(Value::as_bool) + .unwrap_or(false), state_measured, half_closed: false, hard_defect: false, @@ -676,6 +716,20 @@ fn rank_group_records( .filter(|s| s.cull_face_count > 0) .all(|s| s.cull_smile.is_some()); let best_smile = smile_values.iter().cloned().reduce(f64::max); + // The group's best face readings for the labels below, which only ever say + // that one frame is worse than a sibling. + let least_open_mouth = scored + .iter() + .filter_map(|s| s.cull_mouth_open) + .reduce(f64::min); + let brightest_face = scored + .iter() + .filter_map(|s| s.cull_face_luminance) + .reduce(f64::max); + let softest_light = scored + .iter() + .filter_map(|s| s.cull_face_across_stops) + .reduce(f64::min); // The group's best face readings, for the relative defect rules: a proxy // that reads "closed" on every frame of a burst says nothing about any one @@ -709,6 +763,14 @@ fn rank_group_records( .iter() .filter(|s| s.cull_face_count > 0) .all(|s| s.focus_measured); + let eyes_in_focus = |item: &Scored| { + focus_active + && rk.reason_eye_focus_threshold > 0.0 + && item.cull_face_count > 0 + && item + .cull_eye_focus + .is_some_and(|v| v >= rk.reason_eye_focus_threshold) + }; let eye_focus_values: Vec = scored .iter() .filter(|s| s.cull_face_count > 0) @@ -841,7 +903,13 @@ fn rank_group_records( // When the sharpest frame is itself soft there is no usable sibling // to measure against, and the burst missed focus throughout: every // frame is a defect, as before. - let blur_defect = item.sharpness_effective < rk.reason_blur_threshold + // + // Unless the eyes were measured sharp, which settles it. The frame-wide + // reading also falls for a small face against something dark or out of + // focus: on a shoot, ten frames of a child behind birthday candles all + // read "blurred" with sharp eyes and lost the group its pick. + let blur_defect = !eyes_in_focus(item) + && item.sharpness_effective < rk.reason_blur_threshold && (!relative || !rk.relative_blur_rule || group_size == 1 @@ -862,9 +930,12 @@ fn rank_group_records( && !item.blink && !item.eyes_closed_intended && rk.reason_eyes_half_closed_threshold > 0.0 - && item + && (item .cull_eye_closed - .is_some_and(|v| v >= rk.reason_eyes_half_closed_threshold && v < reason_blink); + .is_some_and(|v| v >= rk.reason_eyes_half_closed_threshold && v < reason_blink) + || item + .cull_eye_squint + .is_some_and(|v| v >= rk.reason_eye_squint_threshold)); item.occluded = group_has_faces && item.cull_face_count > 0 && item.cull_occlusion > rk.reason_occlusion_threshold @@ -959,7 +1030,8 @@ fn rank_group_records( if intentional_set.is_some() { return false; } - let is_soft = item.sharpness_effective < rk.reason_blur_threshold; + let is_soft = + item.sharpness_effective < rk.reason_blur_threshold && !eyes_in_focus(item); if i == 0 { // Nothing better to compare with: a single, or the best of its // group. Only blur counts here; see @@ -1053,9 +1125,14 @@ fn rank_group_records( // With the face state model, a frame whose eyes it could not judge // has won nothing on them. let eyes_judged = !state_active || item.cull_eye_closed.is_some(); + // The most open eyes of a group can still be shut, and a single is + // always its own best: only the model can say so. + let eyes_open = + !state_active || (item.cull_blink_penalty <= reason_blink && !item.looking_down); if item.cull_eye_openness >= (max_eye_openness - rk.reason_eyes_open_delta).max(0.0) && is_winner && eyes_judged + && eyes_open { reason_codes.push("eyes_open_best".to_string()); } else if item.eyes_closed_intended && item.cull_blink_penalty > reason_blink { @@ -1070,11 +1147,13 @@ fn rank_group_records( } else if item.half_closed { reason_codes.push("eyes_half_closed".to_string()); } - if state_active - && rk.reason_gaze_away_threshold > 0.0 + if state_active && item.looking_down { + reason_codes.push("looking_down".to_string()); + } else if state_active + && rk.reason_gaze_off_lens_degrees > 0.0 && item - .cull_gaze_away - .is_some_and(|g| g > rk.reason_gaze_away_threshold) + .cull_gaze_off_lens + .is_some_and(|g| g > rk.reason_gaze_off_lens_degrees) { reason_codes.push("looking_away".to_string()); } @@ -1087,6 +1166,39 @@ fn rank_group_records( { reason_codes.push("best_smile".to_string()); } + if rk.reason_mouth_open_threshold > 0.0 + && !item.eyes_closed_intended + && item.cull_mouth_open.is_some_and(|open| { + open >= rk.reason_mouth_open_threshold + && least_open_mouth.is_some_and(|l| open >= l + rk.reason_mouth_open_delta) + }) + { + reason_codes.push("mouth_open".to_string()); + } + if rk.reason_face_darker_stops > 0.0 + && item.cull_face_luminance.is_some_and(|l| { + brightest_face.is_some_and(|b| { + (b.max(1e-4) / l.max(1e-4)).log2() >= rk.reason_face_darker_stops + }) + }) + { + reason_codes.push("face_darker".to_string()); + } + if rk.reason_face_blown_threshold > 0.0 + && item + .cull_face_blown + .is_some_and(|b| b >= rk.reason_face_blown_threshold) + { + reason_codes.push("face_blown".to_string()); + } + if rk.reason_face_hard_light_stops > 0.0 + && item.cull_face_across_stops.is_some_and(|a| { + a >= rk.reason_face_hard_light_stops + && softest_light.is_some_and(|s| a >= s + rk.reason_face_hard_light_delta) + }) + { + reason_codes.push("face_hard_light".to_string()); + } } // Explains the pick a technical reading would not have made. Emitted // for the winner only, and only when the moment is what carried it — a @@ -1229,7 +1341,12 @@ fn rank_group_records( for (key, value) in [ ("eye_closed", item.cull_eye_closed), ("smile", item.cull_smile), - ("gaze_away", item.cull_gaze_away), + ("eye_squint", item.cull_eye_squint), + ("gaze_off_lens", item.cull_gaze_off_lens), + ("mouth_open", item.cull_mouth_open), + ("face_luminance", item.cull_face_luminance), + ("face_across_stops", item.cull_face_across_stops), + ("face_blown", item.cull_face_blown), ] { if let Some(v) = value { metrics.insert(key.into(), Value::from(round4(v))); @@ -2568,8 +2685,10 @@ mod tests { r.metadata .insert("cull_blink_penalty".into(), Value::from(0.9)); if measured { - r.metadata - .insert(FACE_STATE_VERSION_KEY.into(), Value::from(1)); + r.metadata.insert( + FACE_STATE_VERSION_KEY.into(), + Value::from(FACE_STATE_VERSION), + ); } if let Some(v) = closed { r.metadata.insert("cull_eye_closed".into(), Value::from(v)); @@ -2659,6 +2778,265 @@ mod tests { assert_eq!(squint.metrics["clear_defect"], false); } + #[test] + fn shut_eyes_are_never_the_best_open_eyes() { + let groups = cull( + vec![state_frame("shut", 1000.0, Some(0.66), Some(0.2), true)], + "portrait", + ); + let shut = photo(&groups[0], "shut"); + assert!( + !shut.reason_codes.contains(&"eyes_open_best".to_string()), + "{:?}", + shut.reason_codes + ); + let groups = cull( + vec![state_frame("open", 1000.0, Some(0.05), Some(0.2), true)], + "portrait", + ); + assert!(photo(&groups[0], "open") + .reason_codes + .contains(&"eyes_open_best".to_string())); + } + + /// A child looking at the candles next to one looking up: a note, no defect, + /// and no claim about the best open eyes. + #[test] + fn looking_down_is_named_and_never_a_defect() { + let mut down = state_frame("down", 1000.0, None, Some(0.2), true); + down.metadata + .insert("cull_looking_down".into(), Value::Bool(true)); + down.metadata + .insert("cull_gaze_off_lens".into(), Value::from(40.0)); + let groups = cull( + vec![down, state_frame("up", 1000.5, Some(0.05), Some(0.2), true)], + "portrait", + ); + let down = photo(&groups[0], "down"); + assert!(down.reason_codes.contains(&"looking_down".to_string())); + assert!(!down.reason_codes.contains(&"looking_away".to_string())); + assert!(!down.reason_codes.contains(&"eyes_closed".to_string())); + assert!(!down.reason_codes.contains(&"eyes_open_best".to_string())); + assert!(!down.reject_candidate, "{:?}", down.reason_codes); + assert_eq!(down.metrics["clear_defect"], false); + } + + /// Candlelight: little detail anywhere in the frame, the child's eyes sharp. + /// Readings from a real group of ten, which used to lose its pick. + #[test] + fn eyes_measured_sharp_overrule_a_soft_frame() { + let groups = cull( + vec![ + focus_frame("a", 1000.0, 0.04, Some(0.60), Some(0.47), true), + focus_frame("b", 1000.5, 0.03, Some(0.57), Some(0.48), true), + ], + "portrait", + ); + let a = photo(&groups[0], "a"); + assert!(!a.reason_codes.contains(&"blurred".to_string())); + assert_eq!(a.metrics["clear_defect"], false); + // The group has its pick, and the best frame is no reject candidate. + assert!(a.winner, "{:?}", a.reason_codes); + assert!(!a.reject_candidate); + } + + /// Soft eyes are no rescue, and neither are eyes nobody measured. + #[test] + fn a_soft_frame_without_sharp_eyes_stays_blurred() { + let groups = cull( + vec![ + focus_frame("soft_eyes", 1000.0, 0.04, Some(0.42), Some(0.39), true), + focus_frame("no_reading", 1000.5, 0.04, None, Some(0.40), true), + ], + "portrait", + ); + for id in ["soft_eyes", "no_reading"] { + let p = photo(&groups[0], id); + assert!(p.reason_codes.contains(&"blurred".to_string()), "{id}"); + } + } + + fn frame_with(id: &str, t: f64, readings: &[(&str, f64)]) -> GroupingInput { + let mut r = state_frame(id, t, Some(0.04), Some(0.2), true); + for (key, value) in readings { + r.metadata.insert((*key).into(), Value::from(*value)); + } + r + } + + #[test] + fn an_open_mouth_is_named_only_against_a_sibling_and_where_the_preset_asks() { + let records = || { + vec![ + frame_with("closed", 1000.0, &[("cull_mouth_open", 0.02)]), + frame_with("word", 1000.5, &[("cull_mouth_open", 0.4)]), + ] + }; + let groups = cull(records(), "event"); + let word = photo(&groups[0], "word"); + assert!(word.reason_codes.contains(&"mouth_open".to_string())); + // A label, never a defect. + assert_eq!(word.metrics["clear_defect"], false); + let closed = photo(&groups[0], "closed"); + assert!(!closed.reason_codes.contains(&"mouth_open".to_string())); + + let groups = cull(records(), "default"); + let word = photo(&groups[0], "word"); + assert!(!word.reason_codes.contains(&"mouth_open".to_string())); + + // Speaking all the way through a burst says nothing about any one frame. + let groups = cull( + vec![ + frame_with("a", 1000.0, &[("cull_mouth_open", 0.4)]), + frame_with("b", 1000.5, &[("cull_mouth_open", 0.45)]), + ], + "event", + ); + for id in ["a", "b"] { + assert!(!photo(&groups[0], id) + .reason_codes + .contains(&"mouth_open".to_string())); + } + } + + /// Dark skin is not an underexposed face: the reading only ever compares a + /// frame with its siblings, so a burst that is uniformly dark says nothing. + #[test] + fn a_dark_face_is_named_only_against_a_brighter_sibling() { + let groups = cull( + vec![ + frame_with("even1", 1000.0, &[("cull_face_luminance", 0.05)]), + frame_with("even2", 1000.5, &[("cull_face_luminance", 0.052)]), + ], + "event", + ); + for id in ["even1", "even2"] { + assert!(!photo(&groups[0], id) + .reason_codes + .contains(&"face_darker".to_string())); + } + let groups = cull( + vec![ + frame_with("bright", 1000.0, &[("cull_face_luminance", 0.2)]), + frame_with("dark", 1000.5, &[("cull_face_luminance", 0.08)]), + ], + "event", + ); + let dark = photo(&groups[0], "dark"); + assert!(dark.reason_codes.contains(&"face_darker".to_string())); + assert_eq!(dark.metrics["clear_defect"], false); + } + + #[test] + fn blown_skin_is_named_even_on_a_single() { + let groups = cull( + vec![frame_with("hot", 1000.0, &[("cull_face_blown", 0.3)])], + "default", + ); + assert!(photo(&groups[0], "hot") + .reason_codes + .contains(&"face_blown".to_string())); + let groups = cull( + vec![frame_with("mask", 1000.0, &[("cull_face_blown", 0.08)])], + "default", + ); + assert!(!photo(&groups[0], "mask") + .reason_codes + .contains(&"face_blown".to_string())); + } + + #[test] + fn hard_light_is_named_against_a_softer_sibling_in_stops() { + let groups = cull( + vec![ + frame_with("soft", 1000.0, &[("cull_face_across_stops", 0.5)]), + frame_with("hard", 1000.5, &[("cull_face_across_stops", 3.5)]), + ], + "default", + ); + let hard = photo(&groups[0], "hard"); + assert!(hard.reason_codes.contains(&"face_hard_light".to_string())); + assert_eq!(hard.metrics["face_across_stops"], 3.5); + let groups = cull( + vec![frame_with( + "alone", + 1000.0, + &[("cull_face_across_stops", 3.5)], + )], + "default", + ); + assert!(!photo(&groups[0], "alone") + .reason_codes + .contains(&"face_hard_light".to_string())); + } + + /// Eyes narrowed against the sun with the lids reading open: the label, and + /// still no defect. + #[test] + fn a_squint_with_the_lids_reading_open_is_labelled() { + let mut squint = state_frame("squint", 1000.5, Some(0.25), Some(0.1), true); + squint + .metadata + .insert("cull_eye_squint".into(), Value::from(0.82)); + let groups = cull( + vec![ + state_frame("open", 1000.0, Some(0.04), Some(0.1), true), + squint, + ], + "event", + ); + let squint = photo(&groups[0], "squint"); + assert!(squint + .reason_codes + .contains(&"eyes_half_closed".to_string())); + assert_eq!(squint.metrics["eye_squint"], 0.82); + assert_eq!(squint.metrics["clear_defect"], false); + let open = photo(&groups[0], "open"); + assert!(!open.reason_codes.contains(&"eyes_half_closed".to_string())); + } + + /// Degrees from the lens, not clamped to `0..1`, and only where the preset + /// asks. + #[test] + fn looking_away_is_judged_in_degrees_from_the_lens() { + let frame = |id, t, off: f64| { + let mut r = state_frame(id, t, Some(0.04), Some(0.5), true); + r.metadata + .insert("cull_gaze_off_lens".into(), Value::from(off)); + r + }; + let records = || vec![frame("at_lens", 1000.0, 6.0), frame("away", 1000.5, 38.0)]; + let groups = cull(records(), "portrait"); + let away = photo(&groups[0], "away"); + assert!(away.reason_codes.contains(&"looking_away".to_string())); + assert_eq!(away.metrics["gaze_off_lens"], 38.0); + assert_eq!(away.metrics["clear_defect"], false); + let at_lens = photo(&groups[0], "at_lens"); + assert!(!at_lens.reason_codes.contains(&"looking_away".to_string())); + + let groups = cull(records(), "event"); + let away = photo(&groups[0], "away"); + assert!(!away.reason_codes.contains(&"looking_away".to_string())); + } + + /// The version's value counts: a reading from before a bump means something + /// else, and must not be ranked as if it were today's. + #[test] + fn a_reading_from_an_older_analysis_counts_as_unanalysed() { + let mut stale = state_frame("stale", 1000.0, Some(0.04), Some(0.5), true); + stale.metadata.insert( + FACE_STATE_VERSION_KEY.into(), + Value::from(FACE_STATE_VERSION - 1), + ); + assert!(!face_state_is_current(&stale.metadata)); + let current = state_frame("current", 1000.5, Some(0.04), Some(0.5), true); + assert!(face_state_is_current(¤t.metadata)); + let groups = cull(vec![stale, current], "event"); + for p in &groups[0].photos { + assert!(!p.metrics.contains_key("eye_closed"), "{}", p.photo_id); + } + } + /// A burst with one frame from before the model ran keeps the proxy for /// all of them, rather than judging two frames on two instruments. #[test] diff --git a/server-rs/crates/lrg-analysis/src/lib.rs b/server-rs/crates/lrg-analysis/src/lib.rs index a571ec0b..7533eb14 100644 --- a/server-rs/crates/lrg-analysis/src/lib.rs +++ b/server-rs/crates/lrg-analysis/src/lib.rs @@ -8,6 +8,7 @@ pub mod edit_guardrails; pub mod eval; pub mod face_aggregate; pub mod face_cluster; +pub mod face_expression; pub mod grouping; pub mod keywords; pub mod persons; diff --git a/server-rs/crates/lrg-api/src/edit_budget.rs b/server-rs/crates/lrg-api/src/edit_budget.rs index 302b4e16..8b358f56 100644 --- a/server-rs/crates/lrg-api/src/edit_budget.rs +++ b/server-rs/crates/lrg-api/src/edit_budget.rs @@ -16,11 +16,18 @@ //! which is what a harsh frame actually wants — is never scaled. use lrg_analysis::edit_guardrails::{ - derive_budget, fit_contrast_moves, CaptureConditions, EditBudget, GuardrailConfig, + derive_budget_with_faces, fit_contrast_moves, CaptureConditions, EditBudget, FaceConditions, + GuardrailConfig, }; +use lrg_imaging::cull_config::FaceMetricsConfig; use lrg_imaging::scene::{scene_state, SceneState, SceneStateConfig}; +use lrg_ml::face_state::{EyeState, FaceGeometry, FaceStateModel}; +use lrg_ml::faces::{FaceModel, FacePass}; use serde_json::{json, Map, Value}; +/// The most faces of one frame the guardrails read, largest first. +const MAX_FACES_READ: usize = 12; + /// The plain recipe fields that add contrast, in the order /// [`fit_contrast_moves`] sees them. The tone curve is budgeted alongside them /// but is not a single field, so it rides in a fourth slot as a derived @@ -161,28 +168,116 @@ pub fn apply_budget(recipe: &mut Value, budget: &EditBudget) -> Vec { .collect() } +/// The models that read faces, when both are on this computer. +pub struct FaceReaders<'a> { + pub faces: &'a FaceModel, + pub state: &'a FaceStateModel, +} + +impl<'a> FaceReaders<'a> { + /// `None` without the models. The edit goes on without the face rules and + /// without a word: the user did not ask for them, an edit does not need + /// them, and the settings already offer the download where it does matter. + pub fn when_downloaded( + faces: &'a FaceModel, + state: &'a FaceStateModel, + ) -> Option> { + (faces.is_cached() && state.is_cached()).then_some(FaceReaders { faces, state }) + } + + /// What the faces that matter in the frame show. A failure here leaves the + /// recipe under the frame's other rules, and the caller says so. + fn read(&self, pixels: &[u8], width: usize, height: usize) -> Result { + let cfg = FaceMetricsConfig::defaults(); + let detected = self + .faces + .detect_faces(pixels, width, height, &cfg, FacePass::QualityOnly) + .map_err(|e| e.to_string())?; + let largest = detected.iter().map(|f| f.area_ratio).fold(0.0f64, f64::max); + let cutoff = largest * cfg.prominent_face_area_fraction; + let mut prominent: Vec<_> = detected.iter().filter(|f| f.area_ratio >= cutoff).collect(); + prominent.sort_by(|a, b| b.area_ratio.total_cmp(&a.area_ratio)); + prominent.truncate(MAX_FACES_READ); + let geometry: Vec = prominent + .iter() + .map(|f| FaceGeometry { + bbox: f.bbox.map(|v| v as f64), + kps: f.kps, + }) + .collect(); + let states = self + .state + .analyze(pixels, width, height, &geometry, &cfg) + .map_err(|e| e.to_string())?; + let lights: Vec<_> = states + .iter() + .flatten() + .filter(|s| s.eye_state != EyeState::Unknown) + .filter_map(|s| s.light) + .collect(); + Ok(FaceConditions { + across_face_stops: lights + .iter() + .filter_map(|l| l.across_face_stops) + .reduce(f64::max), + against_surround_stops: lights + .iter() + .filter_map(|l| l.against_surround_stops) + .reduce(f64::min), + blown: lights.iter().map(|l| l.blown).reduce(f64::max), + }) + } +} + +/// What [`measure_and_apply`] did to a recipe. +#[derive(Debug, Default)] +pub struct Applied { + /// The reason codes for whatever fired, empty when nothing did. + pub reasons: Vec, + /// Something that went short and the user should hear about. + pub warning: Option, +} + /// Measure the frame and constrain the recipe to what it can absorb. /// -/// Returns the reason codes for whatever fired, empty when nothing did. A -/// decode failure is not an error: the recipe is returned unconstrained, which +/// A decode failure is not an error: the recipe is returned unconstrained, which /// is exactly how the edit path behaved before guardrails existed. pub fn measure_and_apply( recipe: &mut Value, image_bytes: &[u8], capture: &CaptureConditions, -) -> Vec { + readers: Option>, +) -> Applied { if !recipe_needs_budget(recipe) { - return Vec::new(); + return Applied::default(); } let Ok(decoded) = image::load_from_memory(image_bytes) else { log::warn!("Edit guardrails: could not decode the image; recipe left unconstrained."); - return Vec::new(); + return Applied::default(); }; let rgb = decoded.to_rgb8(); let (width, height) = (rgb.width() as usize, rgb.height() as usize); let pixels = rgb.into_raw(); let scene = scene_state(&pixels, width, height, &SceneStateConfig::defaults()); - apply_with_scene(recipe, &scene, capture) + let (faces, warning) = match readers.map(|r| r.read(&pixels, width, height)) { + Some(Err(e)) => { + log::warn!("Edit guardrails: faces not read ({e})."); + ( + FaceConditions::default(), + Some(format!( + "The faces in this photo could not be read ({e}), so the edit was not limited \ + by the light on them. If this keeps happening, run \"Download AI models\" in \ + Plug-in Manager again." + )), + ) + } + Some(Ok(faces)) => (faces, None), + None => (FaceConditions::default(), None), + }; + Applied { + reasons: apply_with_faces(recipe, &scene, capture, &faces), + warning, + } } /// [`measure_and_apply`] with the measurement already done. @@ -191,7 +286,17 @@ pub fn apply_with_scene( scene: &SceneState, capture: &CaptureConditions, ) -> Vec { - let budget = derive_budget(scene, capture, &GuardrailConfig::defaults()); + apply_with_faces(recipe, scene, capture, &FaceConditions::default()) +} + +/// [`apply_with_scene`] with what was read off the faces in the frame. +pub fn apply_with_faces( + recipe: &mut Value, + scene: &SceneState, + capture: &CaptureConditions, + faces: &FaceConditions, +) -> Vec { + let budget = derive_budget_with_faces(scene, capture, faces, &GuardrailConfig::defaults()); apply_budget(recipe, &budget) } @@ -246,6 +351,73 @@ mod tests { ); } + /// The canonical portrait case: an ordinary frame, a habitual `+20 clarity`, + /// and hard light across the face it would deepen. + #[test] + fn hard_light_on_a_face_takes_the_clarity_out_of_the_recipe() { + let mut recipe = recipe_with(json!({"contrast": 8.0, "clarity": 20.0})); + let faces = FaceConditions { + across_face_stops: Some(3.0), + ..Default::default() + }; + let reasons = apply_with_faces( + &mut recipe, + &ordinary(), + &CaptureConditions::default(), + &faces, + ); + let global = global_of(&recipe); + assert_eq!(global["clarity"].as_f64().unwrap(), 0.0); + assert!(global["contrast"].as_f64().unwrap() <= 10.0); + assert_eq!(reasons, vec!["face_hard_light".to_string()]); + } + + /// A frame without a face reading edits as it always did. + #[test] + fn no_faces_read_changes_nothing() { + let mut with = recipe_with(json!({"contrast": 15.0, "clarity": 10.0})); + let mut without = with.clone(); + apply_with_scene(&mut with, &ordinary(), &CaptureConditions::default()); + apply_with_faces( + &mut without, + &ordinary(), + &CaptureConditions::default(), + &FaceConditions::default(), + ); + assert_eq!(with, without); + } + + /// A failed read costs the edit its face rules, and the user hears about it: + /// the models are there, so they would otherwise assume the rules applied. + #[test] + fn faces_that_could_not_be_read_are_reported_and_the_edit_goes_on() { + let dir = tempfile::tempdir().unwrap(); + let path = |name: &str| dir.path().join(name); + let faces = FaceModel::new(lrg_ml::faces::FaceModelPaths { + det_onnx: path("det.onnx"), + rec_onnx: path("rec.onnx"), + auraface_onnx: path("aura.onnx"), + }); + let state = FaceStateModel::new(lrg_ml::face_state::FaceStatePaths { + mesh_onnx: path("mesh.onnx"), + blendshapes_onnx: path("blend.onnx"), + }); + let readers = FaceReaders { + faces: &faces, + state: &state, + }; + let mut recipe = recipe_with(json!({"contrast": 5.0})); + let applied = measure_and_apply( + &mut recipe, + &flat_soft_jpeg(), + &CaptureConditions::default(), + Some(readers), + ); + let warning = applied.warning.expect("the failed read is reported"); + assert!(warning.contains("Download AI models"), "{warning}"); + assert_eq!(global_of(&recipe)["contrast"].as_f64().unwrap(), 5.0); + } + /// A frame can be harshly lit without the recipe ever asking for contrast. /// Explaining that contrast "was not increased" there would be noise next /// to an edit that was returned exactly as generated. @@ -380,8 +552,13 @@ mod tests { // No decode is attempted, so deliberately invalid image bytes are fine. let mut recipe = recipe_with(json!({"temperature": 5600.0, "contrast": -10.0})); assert!(!recipe_needs_budget(&recipe)); - let reasons = - measure_and_apply(&mut recipe, b"not an image", &CaptureConditions::default()); + let reasons = measure_and_apply( + &mut recipe, + b"not an image", + &CaptureConditions::default(), + None, + ) + .reasons; assert!(reasons.is_empty()); assert_eq!(global_of(&recipe)["temperature"].as_f64().unwrap(), 5600.0); } @@ -441,7 +618,9 @@ mod tests { &mut recipe, &harshly_lit_jpeg(), &CaptureConditions::default(), - ); + None, + ) + .reasons; let g = global_of(&recipe); assert_eq!(g["contrast"].as_f64().unwrap(), 0.0, "reasons: {reasons:?}"); @@ -464,7 +643,9 @@ mod tests { &mut recipe, &flat_soft_jpeg(), &CaptureConditions::default(), - ); + None, + ) + .reasons; assert_eq!(recipe, before, "flat light should widen, never narrow"); assert!(reasons.is_empty(), "got {reasons:?}"); @@ -474,8 +655,13 @@ mod tests { fn an_undecodable_image_leaves_the_recipe_alone() { let mut recipe = recipe_with(json!({"contrast": 90.0})); let before = recipe.clone(); - let reasons = - measure_and_apply(&mut recipe, b"not an image", &CaptureConditions::default()); + let reasons = measure_and_apply( + &mut recipe, + b"not an image", + &CaptureConditions::default(), + None, + ) + .reasons; assert!(reasons.is_empty()); assert_eq!(recipe, before, "a decode failure must not change the edit"); } diff --git a/server-rs/crates/lrg-api/src/routes/edit.rs b/server-rs/crates/lrg-api/src/routes/edit.rs index 4b8fdd55..88d2d714 100644 --- a/server-rs/crates/lrg-api/src/routes/edit.rs +++ b/server-rs/crates/lrg-api/src/routes/edit.rs @@ -611,11 +611,19 @@ pub(crate) async fn generate_edit_recipe_for_photo( // switched off must not first be scaled and then discarded, which // would spend budget on a field that never reaches the photo and // shrink the ones that do. - response.guardrail_reasons = crate::edit_budget::measure_and_apply( + let applied = crate::edit_budget::measure_and_apply( &mut recipe, image_bytes, &capture_conditions(options, filename, image_bytes), + crate::edit_budget::FaceReaders::when_downloaded(&state.face, &state.face_state), ); + response.guardrail_reasons = applied.reasons; + if let Some(msg) = applied.warning { + response.warning = Some(match response.warning.take() { + Some(existing) => format!("{existing}\n{msg}"), + None => msg, + }); + } if !response.guardrail_reasons.is_empty() { log::info!( "Photo {photo_id}: edit constrained by the frame ({}).", @@ -813,13 +821,16 @@ pub(crate) fn success_payload( /// a newer backend should show nothing rather than a bare identifier. fn guardrail_explanations(codes: &[String]) -> Vec { use lrg_analysis::edit_guardrails::GuardrailReason::*; - const ALL: [lrg_analysis::edit_guardrails::GuardrailReason; 6] = [ + const ALL: [lrg_analysis::edit_guardrails::GuardrailReason; 9] = [ HardLightNoAddedContrast, NoTonalHeadroom, FlatLightContrastAllowed, HighlightsUnrecoverable, HighIsoShadowsLimited, ShadowsAlreadyClipped, + FaceHardLight, + FaceBelowSurroundings, + FaceBlown, ]; codes .iter() diff --git a/server-rs/crates/lrg-api/src/routes/faces.rs b/server-rs/crates/lrg-api/src/routes/faces.rs index 450e5fe8..3a4fff8c 100644 --- a/server-rs/crates/lrg-api/src/routes/faces.rs +++ b/server-rs/crates/lrg-api/src/routes/faces.rs @@ -31,6 +31,7 @@ use serde::Deserialize; use serde_json::{json, Map, Value}; use lrg_analysis::face_cluster::{self, ClusterParams, FaceInput, QualityGate}; +use lrg_analysis::face_expression::{portrait_score, ExpressionFilter, Verdict}; use lrg_analysis::{clustering, persons}; use lrg_store::{StoreRecord, FACE_TABLE}; @@ -320,6 +321,16 @@ fn number_at(meta: &Map, key: &str) -> f64 { meta.get(key).and_then(Value::as_f64).unwrap_or(0.0) } +/// Of two faces of one person on one photo, the answer that counts for the +/// photo: a yes settles it, and only when no face answers is it unknown. +fn best_verdict(a: Verdict, b: Verdict) -> Verdict { + match (a, b) { + (Verdict::Yes, _) | (_, Verdict::Yes) => Verdict::Yes, + (Verdict::Unknown, _) | (_, Verdict::Unknown) => Verdict::Unknown, + _ => Verdict::No, + } +} + /// The same, but keeping "the row does not have this" distinct from "the row /// says zero". The quality gate needs the difference: a face indexed before /// these measurements existed must not be judged as if every one of them came @@ -758,7 +769,7 @@ async fn list_persons(State(state): State>) -> Response { entry.manual_count += 1; } entry.photo_ids.insert(photo_of(meta).to_string()); - let quality = number_at(meta, "face_det_score") + number_at(meta, "face_sharpness"); + let quality = portrait_score(meta); if entry.thumbnail.is_none() || quality > entry.thumbnail_quality { if let Some(thumb) = meta .get("thumbnail") @@ -815,7 +826,13 @@ async fn person_thumbnail( let thumb = match store.scan_meta(FACE_TABLE).await { Ok(rows) => rows .into_iter() - .find(|(_, m)| person_of(m) == person_id) + .filter(|(_, m)| person_of(m) == person_id) + .filter(|(_, m)| { + m.get("thumbnail") + .and_then(Value::as_str) + .is_some_and(|t| !t.is_empty()) + }) + .max_by(|(_, a), (_, b)| portrait_score(a).total_cmp(&portrait_score(b))) .and_then(|(_, m)| { m.get("thumbnail") .and_then(Value::as_str) @@ -947,9 +964,24 @@ async fn confirm_person_faces(state: &AppState, person_id: &str) -> Result, + limit: Option, +} + async fn person_photos( State(state): State>, UrlPath(person_id): UrlPath, + Query(params): Query, ) -> Response { log::info!( "Get photos for person request received for person_id={}", @@ -972,26 +1004,64 @@ async fn person_photos( } }; let wants_unassigned = is_unassigned(&person_id); - let mut photo_ids: Vec = rows - .into_iter() - .filter(|(_, m)| { - if wants_unassigned { - is_unassigned(person_of(m)) - } else { - person_of(m) == person_id - } - }) - .map(|(_, m)| photo_of(&m).to_string()) - .filter(|id| !id.is_empty()) - .collect::>() + let filter = ExpressionFilter { + eyes_open: params.eyes_open, + smiling: params.smiling, + looking_at_camera: params.looking_at_camera, + }; + // A person can be on a photo twice (a mirror, a poster): the photo counts + // when any of those faces answers yes, and scores as the best of them. + let mut photos: HashMap = HashMap::new(); + for (_, m) in &rows { + let mine = if wants_unassigned { + is_unassigned(person_of(m)) + } else { + person_of(m) == person_id + }; + let photo = photo_of(m); + if !mine || photo.is_empty() { + continue; + } + let (verdict, score) = (filter.judge_face(m), portrait_score(m)); + let entry = photos.entry(photo.to_string()).or_insert((verdict, score)); + entry.0 = best_verdict(entry.0, verdict); + entry.1 = entry.1.max(score); + } + let not_judged = photos + .values() + .filter(|(v, _)| *v == Verdict::Unknown) + .count(); + let mut matching: Vec<(String, f64)> = photos .into_iter() + .filter(|(_, (v, _))| *v == Verdict::Yes) + .map(|(id, (_, score))| (id, score)) .collect(); - photo_ids.sort(); + if params.sort.as_deref() == Some("best") { + matching.sort_by(|a, b| b.1.total_cmp(&a.1).then_with(|| a.0.cmp(&b.0))); + } else { + matching.sort_by(|a, b| a.0.cmp(&b.0)); + } + if let Some(limit) = params.limit { + matching.truncate(limit); + } + let photo_ids: Vec = matching.into_iter().map(|(id, _)| id).collect(); + let mut warnings = Vec::new(); + if not_judged > 0 { + warnings.push(format!( + "{not_judged} photo(s) of this person could not be judged for {} and were left out. A \ + face too small or turned too far cannot be read. One indexed before the eye and \ + expression model was on disk can: run 'Analyze & Index Photos' with face detection \ + on those photos, after \"Download AI models\" in Plug-in Manager.", + filter.describe() + )); + } Json(json!({ "status": "ok", "person_id": person_id, "photo_ids": photo_ids.clone(), "photo_uuids": photo_ids, + "not_judged": not_judged, + "warnings": warnings, })) .into_response() } @@ -1003,7 +1073,8 @@ struct PersonFacesQuery { limit: Option, offset: Option, /// `typical` (closest to the person's average face first), `outlier` (the - /// reverse — where wrong faces surface), or `quality`. + /// reverse — where wrong faces surface), `quality`, or `portrait` (the best + /// picture of them first: eyes open, facing the camera). sort: Option, } @@ -1115,6 +1186,9 @@ async fn person_faces( + 0.1 * (number_at(m, "face_area_ratio") / 0.12).clamp(0.0, 1.0) }; match sort.as_str() { + "portrait" => { + mine.sort_by(|(_, a), (_, b)| portrait_score(b).total_cmp(&portrait_score(a))) + } "quality" => mine.sort_by(|(_, a), (_, b)| { quality_score(b) .partial_cmp(&quality_score(a)) diff --git a/server-rs/crates/lrg-api/src/routes/group_similar.rs b/server-rs/crates/lrg-api/src/routes/group_similar.rs index 1597b36a..85ed163c 100644 --- a/server-rs/crates/lrg-api/src/routes/group_similar.rs +++ b/server-rs/crates/lrg-api/src/routes/group_similar.rs @@ -647,9 +647,7 @@ async fn compute_groups( .and_then(Value::as_i64) .unwrap_or(0) > 0 - && !r - .metadata - .contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY) + && !lrg_analysis::grouping::face_state_is_current(&r.metadata) }) .count(); diff --git a/server-rs/crates/lrg-api/src/routes/index.rs b/server-rs/crates/lrg-api/src/routes/index.rs index 71db0152..d54daaa4 100644 --- a/server-rs/crates/lrg-api/src/routes/index.rs +++ b/server-rs/crates/lrg-api/src/routes/index.rs @@ -249,8 +249,19 @@ async fn check_unprocessed( || has_flag("alt_text") || has_flag("keywords"); let needs_metadata = compute_metadata && (regenerate_metadata || !has_any_metadata); - let needs_faces = - compute_faces && (regenerate_metadata || !faces_checked.contains(*pid)); + // A photo with faces whose readings are missing or from an older + // analysis goes through the face pass again once the model is on + // disk. Person assignments carry over, and this is how a catalog + // indexed earlier gets what People's filters ask about. + let needs_face_state = face_state_available + && m.get("cull_face_count") + .and_then(Value::as_i64) + .unwrap_or(0) + > 0 + && (!lrg_analysis::grouping::face_state_is_current(&m) + || has_flag("face_rows_behind")); + let needs_faces = compute_faces + && (regenerate_metadata || !faces_checked.contains(*pid) || needs_face_state); let needs_cull_phash = any_task && (regenerate_metadata || !has_flag("cull_phash")); // `cull_faces_present` is legitimately `false` for a photo with // no faces, so completion is "the key exists", not "it is @@ -283,7 +294,7 @@ async fn check_unprocessed( .and_then(Value::as_i64) .unwrap_or(0) > 0 - && !m.contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY); + && !lrg_analysis::grouping::face_state_is_current(&m); // A photo is done for `species` only if it was checked *and* // checked by the head currently on disk. Widening the taxa // whitelist changes what BioCLIP can answer, so results from an diff --git a/server-rs/crates/lrg-api/src/routes/index_upload.rs b/server-rs/crates/lrg-api/src/routes/index_upload.rs index d2523d81..7abcde73 100644 --- a/server-rs/crates/lrg-api/src/routes/index_upload.rs +++ b/server-rs/crates/lrg-api/src/routes/index_upload.rs @@ -2031,10 +2031,18 @@ async fn finish_one( let focus_missing = options.face_pass == FacePass::QualityOnly && !main_metadata.contains_key(lrg_analysis::grouping::FOCUS_VERSION_KEY); // The same for the face state, once its model is on disk: a photo - // prepared before it was downloaded is analysed on the next cull pass. - let state_missing = options.face_pass == FacePass::QualityOnly + // prepared before it was downloaded is analysed on the next pass. A full + // pass only when the photo has faces, since it re-runs identity for them. + let has_faces = main_metadata + .get("cull_face_count") + .and_then(Value::as_i64) + .unwrap_or(0) + > 0; + let rows_behind = + options.face_pass == FacePass::Full && main_metadata.contains_key(FACE_ROWS_BEHIND_KEY); + let state_missing = (options.face_pass == FacePass::QualityOnly || has_faces) && state.face_state.is_cached() - && !main_metadata.contains_key(lrg_analysis::grouping::FACE_STATE_VERSION_KEY); + && (rows_behind || !lrg_analysis::grouping::face_state_is_current(&main_metadata)); if options.regenerate_metadata || !already_checked || focus_missing || state_missing { // Decoded here rather than carried from phase 1: full RGB is by far // the largest per-photo allocation, and holding one per photo in @@ -2150,12 +2158,48 @@ async fn finish_one( occlusion: Some(f.occlusion), eye_focus: f.eye_focus, face_focus: f.face_focus, - eye_closed: judged(f).map(|s| s.eye_closed), + eye_closed: judged(f) + .filter(|s| s.eye_state != lrg_ml::face_state::EyeState::Down) + .map(|s| s.eye_closed), + looking_down: judged(f) + .is_some_and(|s| s.eye_state == lrg_ml::face_state::EyeState::Down), smile: judged(f).map(|s| s.smile), - gaze_away: judged(f).map(|s| s.gaze_away), + eye_squint: judged(f).map(|s| s.eye_squint), + gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), + mouth_open: judged(f).map(|s| s.mouth_open), + luminance: judged_light(f).map(|l| l.luminance), + across_face_stops: judged_light(f).and_then(|l| l.across_face_stops), + blown: judged_light(f).map(|l| l.blown), }) .collect(); - apply_face_aggregate(&mut main_metadata, &inputs, state_ran); + // Before the readings are stamped as current: a photo whose rows + // could not be saved must be asked for again, or People could + // never read it. + let mut stamped = state_ran; + if state_ran { + match merge_face_state_into_rows(store, photo_id, &faces).await { + Ok(0) => { + main_metadata.remove(FACE_ROWS_BEHIND_KEY); + } + // Nothing to tell the user yet: People says so when + // asked, and names the full pass this queues. + Ok(unmatched) => { + log::debug!( + "Photo {photo_id}: {unmatched} face row(s) match no face." + ); + main_metadata.insert(FACE_ROWS_BEHIND_KEY.into(), json!(true)); + } + Err(e) => { + log::warn!("Photo {photo_id}: face rows not updated: {e}"); + stamped = false; + warnings.push(format!( + "{filename}: eyes, smile and gaze could not be saved ({e}). \ + Choose 'Prepare now' in 'Cull Similar Photos' to try again." + )); + } + } + } + apply_face_aggregate(&mut main_metadata, &inputs, stamped); log::debug!( "Photo {photo_id}: cull pass scored {} face(s), no identity written.", faces.len() @@ -2264,18 +2308,7 @@ async fn finish_one( m.insert("face_focus".into(), json!(f.face_focus)); m.insert("face_eye_px".into(), json!(f.eye_px)); m.insert("face_focus_scale".into(), json!(f.focus_scale)); - // Null when the face state did not run for this - // face; `face_eye_state` is "unknown" when it ran - // and could not judge the eyes. - let st = f.state.as_ref(); - m.insert( - "face_eye_state".into(), - json!(st.map(|s| s.eye_state.as_str())), - ); - m.insert("face_eye_closed".into(), json!(st.map(|s| s.eye_closed))); - m.insert("face_smile".into(), json!(st.map(|s| s.smile))); - m.insert("face_gaze_away".into(), json!(st.map(|s| s.gaze_away))); - m.insert("face_yaw".into(), json!(st.map(|s| s.yaw_degrees))); + insert_face_state(&mut m, f.state.as_ref()); m.insert("face_model".into(), json!(face_model)); StoreRecord { id: format!("{photo_id}_{i}"), @@ -2301,9 +2334,20 @@ async fn finish_one( occlusion: Some(f.occlusion), eye_focus: f.eye_focus, face_focus: f.face_focus, - eye_closed: judged(f).map(|s| s.eye_closed), + eye_closed: judged(f) + .filter(|s| s.eye_state != lrg_ml::face_state::EyeState::Down) + .map(|s| s.eye_closed), + looking_down: judged(f).is_some_and(|s| { + s.eye_state == lrg_ml::face_state::EyeState::Down + }), smile: judged(f).map(|s| s.smile), - gaze_away: judged(f).map(|s| s.gaze_away), + eye_squint: judged(f).map(|s| s.eye_squint), + gaze_off_lens: judged(f).map(|s| s.gaze_off_lens), + mouth_open: judged(f).map(|s| s.mouth_open), + luminance: judged_light(f).map(|l| l.luminance), + across_face_stops: judged_light(f) + .and_then(|l| l.across_face_stops), + blown: judged_light(f).map(|l| l.blown), }) .collect(); apply_face_aggregate(&mut main_metadata, &inputs, state_ran); @@ -2318,6 +2362,7 @@ async fn finish_one( // used to disappear on a plain re-index. main_metadata.insert("faces_checked".into(), json!(true)); main_metadata.insert("face_model".into(), json!(face_model)); + main_metadata.remove(FACE_ROWS_BEHIND_KEY); } Err(e) => { log::warn!("Face detection/indexing failed for {photo_id}: {e}"); @@ -2682,6 +2727,10 @@ fn carry_over_person_ids_by_box( /// The face state of a face, when it ran and could judge the eyes; a face it /// could not judge abstains from every eye, smile and gaze aggregate. +fn judged_light(face: &lrg_ml::faces::FaceResult) -> Option { + judged(face).and_then(|s| s.light) +} + fn judged(face: &lrg_ml::faces::FaceResult) -> Option<&lrg_ml::face_state::FaceState> { face.state .as_ref() @@ -2716,7 +2765,12 @@ fn apply_face_aggregate( ("cull_face_focus", agg.cull_face_focus), ("cull_eye_closed", agg.cull_eye_closed), ("cull_smile", agg.cull_smile), - ("cull_gaze_away", agg.cull_gaze_away), + ("cull_eye_squint", agg.cull_eye_squint), + ("cull_gaze_off_lens", agg.cull_gaze_off_lens), + ("cull_mouth_open", agg.cull_mouth_open), + ("cull_face_luminance", agg.cull_face_luminance), + ("cull_face_across_stops", agg.cull_face_across_stops), + ("cull_face_blown", agg.cull_face_blown), ] { match value { Some(v) => { @@ -2733,6 +2787,11 @@ fn apply_face_aggregate( lrg_analysis::grouping::FOCUS_VERSION_KEY.into(), json!(lrg_analysis::grouping::FOCUS_VERSION), ); + if agg.cull_looking_down { + metadata.insert("cull_looking_down".into(), json!(true)); + } else { + metadata.remove("cull_looking_down"); + } match agg.cull_eye_state { Some(state) => { metadata.insert("cull_eye_state".into(), json!(state)); @@ -2759,6 +2818,97 @@ fn apply_face_aggregate( } } +/// A face's readings from the face state model on its row. Null when the model +/// did not run for this face; `face_eye_state` is "unknown" when it ran and could +/// not judge the eyes. +fn insert_face_state(m: &mut Map, st: Option<&lrg_ml::face_state::FaceState>) { + m.insert( + "face_eye_state".into(), + json!(st.map(|s| s.eye_state.as_str())), + ); + m.insert("face_eye_closed".into(), json!(st.map(|s| s.eye_closed))); + m.insert("face_smile".into(), json!(st.map(|s| s.smile))); + m.insert("face_eye_squint".into(), json!(st.map(|s| s.eye_squint))); + m.insert("face_mouth_open".into(), json!(st.map(|s| s.mouth_open))); + let light = st.and_then(|s| s.light); + m.insert("face_luminance".into(), json!(light.map(|l| l.luminance))); + m.insert( + "face_against_surround_stops".into(), + json!(light.and_then(|l| l.against_surround_stops)), + ); + m.insert( + "face_across_stops".into(), + json!(light.and_then(|l| l.across_face_stops)), + ); + m.insert("face_blown".into(), json!(light.map(|l| l.blown))); + m.insert( + "face_gaze_off_lens".into(), + json!(st.map(|s| s.gaze_off_lens)), + ); + m.insert("face_yaw".into(), json!(st.map(|s| s.pose.yaw))); + m.insert("face_pitch".into(), json!(st.map(|s| s.pose.pitch))); + m.insert("face_roll".into(), json!(st.map(|s| s.pose.roll))); + // All 52, in `BLENDSHAPE_NAMES` order, to three places: a later signal can + // then read them without decoding every original again. + m.insert( + "face_blendshapes".into(), + json!(st.map(|s| s + .blendshapes + .iter() + .map(|v| (*v as f64 * 1000.0).round() / 1000.0) + .collect::>())), + ); +} + +/// Marks a photo whose face rows a quality-only pass could not bring up to date, +/// so the next full pass re-detects them. Its own readings are current, so nothing +/// else would ever ask for that. +const FACE_ROWS_BEHIND_KEY: &str = "face_rows_behind"; + +/// Brings a quality-only pass's readings to the rows a full pass wrote earlier, +/// each to the row whose box it landed on, and touches nothing else on them. A +/// catalog whose faces were indexed before the model was on disk gets its readings +/// this way, so People can ask about them without re-running identity. Returns how +/// many rows no detected face landed on: those keep no reading. +async fn merge_face_state_into_rows( + store: &lrg_store::Store, + photo_id: &str, + faces: &[lrg_ml::faces::FaceResult], +) -> Result { + let mut rows = store + .get_by_id_prefix(FACE_TABLE, &format!("{photo_id}_")) + .await + .map_err(|e| e.to_string())?; + let mut changed = Vec::new(); + let mut unmatched = 0; + for row in rows.iter_mut() { + let best = stored_bbox(row).and_then(|old_box| { + faces + .iter() + .map(|f| (box_iou(f.bbox, old_box), f)) + .filter(|(iou, _)| *iou >= FACE_CARRY_OVER_MIN_IOU) + .max_by(|a, b| a.0.total_cmp(&b.0)) + }); + match best { + None => unmatched += 1, + // A bystander too small for the model has a box and no reading, + // which a re-detection would not change. + Some((_, face)) if face.state.is_some() => { + insert_face_state(&mut row.metadata, face.state.as_ref()); + changed.push(row.clone()); + } + Some(_) => {} + } + } + if !changed.is_empty() { + store + .upsert(FACE_TABLE, &changed) + .await + .map_err(|e| e.to_string())?; + } + Ok(unmatched) +} + /// At most this many faces of one photo get the face state model, largest /// first. Only prominent faces count towards the photo's eye state anyway, and /// a group of forty would otherwise cost forty mesh passes. diff --git a/server-rs/crates/lrg-api/src/routes/search.rs b/server-rs/crates/lrg-api/src/routes/search.rs index ab572d33..e5694ce0 100644 --- a/server-rs/crates/lrg-api/src/routes/search.rs +++ b/server-rs/crates/lrg-api/src/routes/search.rs @@ -17,6 +17,7 @@ use axum::extract::{Query, State}; use axum::response::{IntoResponse, Json, Response}; use serde_json::{json, Map, Value}; +use lrg_analysis::face_expression::{ExpressionFilter, Verdict}; use lrg_analysis::relevance_filter::{ clamp_max_results, clamp_strictness, smart_filter_by_relevance, ScoredResult, }; @@ -147,11 +148,12 @@ pub const AESTHETIC_RANGE: (f32, f32) = struct Restrictions { min_aesthetic: Option, genres: Option>, + expression: ExpressionFilter, } impl Restrictions { fn is_empty(&self) -> bool { - self.min_aesthetic.is_none() && self.genres.is_none() + self.min_aesthetic.is_none() && self.genres.is_none() && self.expression.is_empty() } } @@ -161,6 +163,7 @@ impl Restrictions { struct Dropped { no_score: usize, no_genre: usize, + no_face_state: usize, } /// Applies `restrictions` to `hits`, counting the photos it had to drop for want @@ -201,6 +204,16 @@ fn restrict( } } } + if !restrictions.expression.is_empty() { + match restrictions.expression.judge_photo(m) { + Verdict::Yes => {} + Verdict::No => return false, + Verdict::Unknown => { + dropped.no_face_state += 1; + return false; + } + } + } true }) .collect(); @@ -326,6 +339,7 @@ async fn search( .and_then(Value::as_f64) .unwrap_or(0.5) .clamp(0.0, 1.0); + let wants = |key: &str| body.get(key).and_then(Value::as_bool).unwrap_or(false); let restrictions = Restrictions { min_aesthetic: body .get("min_aesthetic") @@ -341,6 +355,11 @@ async fn search( .collect::>() }) .filter(|set| !set.is_empty()), + expression: ExpressionFilter { + eyes_open: wants("eyes_open"), + smiling: wants("smiling"), + looking_at_camera: wants("looking_at_camera"), + }, }; let photo_ids_to_search: Option> = body @@ -602,6 +621,17 @@ async fn search( dropped.no_genre )); } + if dropped.no_face_state > 0 { + warnings.push(format!( + "{} matching photo(s) could not be judged for {} and were left out. A face too small \ + or turned too far cannot be read. One analysed before the eye and expression model \ + was on disk can: choose 'Prepare now' in 'Cull Similar Photos' for those photos, or \ + run 'Analyze & Index Photos' with face detection, after \"Download AI models\" in \ + Plug-in Manager.", + dropped.no_face_state, + restrictions.expression.describe() + )); + } let mut hits = order(hits, sort, aesthetic_weight, &rows); if sort != SortMode::Relevance && term.is_some() { @@ -807,6 +837,7 @@ mod tests { let restrictions = Restrictions { min_aesthetic: Some(6.0), genres: Some(["macro".to_string()].into_iter().collect()), + ..Default::default() }; let (kept, dropped) = restrict(hits, &restrictions, &rows); assert_eq!(ids(&kept), ["ok"]); diff --git a/server-rs/crates/lrg-api/src/routes/style_edit.rs b/server-rs/crates/lrg-api/src/routes/style_edit.rs index 3f3f1134..f34effc1 100644 --- a/server-rs/crates/lrg-api/src/routes/style_edit.rs +++ b/server-rs/crates/lrg-api/src/routes/style_edit.rs @@ -423,11 +423,13 @@ async fn style_edit(State(state): State>, mut multipart: Multipart it is on the raw Kelvin scale and this photo is not raw." ); } - let guardrail_reasons = crate::edit_budget::measure_and_apply( + let applied = crate::edit_budget::measure_and_apply( &mut styled_recipe, &image_bytes, &crate::routes::edit::capture_conditions(&options, filename.as_deref(), &image_bytes), + crate::edit_budget::FaceReaders::when_downloaded(&state.face, &state.face_state), ); + let guardrail_reasons = applied.reasons; if !guardrail_reasons.is_empty() { log::info!( "Photo {photo_id}: style-engine edit constrained by the frame ({}).", @@ -446,11 +448,16 @@ async fn style_edit(State(state): State>, mut multipart: Multipart { log::error!("Failed to persist style_edit recipe for {photo_id}: {e}"); } + let warning = match (result.warning.as_deref(), applied.warning) { + (Some(a), Some(b)) => Some(format!("{a}\n{b}")), + (Some(a), None) => Some(a.to_string()), + (None, b) => b, + }; let mut payload = success_payload( &photo_id, &styled_recipe, &options, - result.warning.as_deref(), + warning.as_deref(), &guardrail_reasons, ); payload["engine"] = json!("style"); diff --git a/server-rs/crates/lrg-api/src/ui/people.html b/server-rs/crates/lrg-api/src/ui/people.html index be01c866..57fa42b3 100644 --- a/server-rs/crates/lrg-api/src/ui/people.html +++ b/server-rs/crates/lrg-api/src/ui/people.html @@ -355,6 +355,25 @@

People

+ + Only + + + + + @@ -376,6 +395,7 @@

Person

+ @@ -1569,6 +1589,10 @@

Person

person_ids: ids, person_names: names, match_mode: ids.length > 1 ? el("match-mode").value : "union", + eyes_open: el("only-eyes-open").checked, + smiling: el("only-smiling").checked, + looking_at_camera: el("only-looking").checked, + best: Number(el("photo-limit").value) || null, }); if (data.plugin_connected) { state.textContent = "Sent — switch to Lightroom."; diff --git a/server-rs/crates/lrg-api/src/ui/search.html b/server-rs/crates/lrg-api/src/ui/search.html index 2f3370ad..490a8e9c 100644 --- a/server-rs/crates/lrg-api/src/ui/search.html +++ b/server-rs/crates/lrg-api/src/ui/search.html @@ -176,6 +176,12 @@

Search

>Genre + + Faces + + + +