[bmachek]

Summary

Culling now judges faces by the criteria the commercial culling tools use. Lightroom's own Assisted Culling has Eye focus and Eyes open, including "can't tell". Narrative has per-face focus and eye traffic lights, Aftershoot key faces and expression, Excire eye sharpness and smile. Before this PR, face sharpness was a Laplacian variance over the whole face box on the ≤2048 px working copy, and blinks were guessed from an eye-openness gradient proxy.

This PR holds the first two of five planned packages (see Follow-ups):

1. Eye and face focus at full resolution (84b0d3b), outstanding item 3 in Dev-Image-Culling-Implementation-Plan.md.
2. Eye state, smile and gaze per face with MediaPipe (e13515d), outstanding item 1.

1. Eye and face focus at full resolution

• New focus measure (lrg-imaging/src/focus.rs): a port of scikit-image's blur_effect (Crété-Roffet 2007). It compares a patch's gradient energy before and after a re-blur, so contrast barely moves it; that is what comparing the frames of one burst needs. A golden test pins it to scikit-image (testdata/focus_goldens.json, made by make_focus_goldens.py).
• Measured on the full-size original. Detection stays on the working copy, so YuNet, the stored boxes and identities are unchanged. Only photos with faces get a second decode of the original, where the face step measures again (convert::decode_oriented shares its code with normalisation, so boxes map by one scale factor; faces::refine_focus). By-path, the file is read again, a raw through its embedded preview. Uploads are kept until the face step. At most two such decodes run at once.
• Per face: the sharper eye (a square of 0.5 × interocular distance) and the whole face. Per photo: the least sharp prominent face. Faces too small to judge (eyes under 40 px apart) abstain. cull_focus_version tells "too small" apart from "indexed before this existed".
• Ranking: eye focus takes a share of the face term, scored against the group. eyes_out_of_focus (or face_out_of_focus when the eyes are too small) becomes a defect only when the frame is clearly behind a sharper sibling; a single frame is never rejected for focus. All or nothing per group. Weights: portrait 0.55, default and event 0.35, sports 0.20, street off.
• Explanation: eye focus N/10 within the group, relative to the sharpest frame, because per-pixel sharpness depends on camera, ISO and magnification as much as on focus.
• Migration: /v1/index/unprocessed re-queues cull-prepared photos with faces but no focus version. The cull pass re-runs the face step for them, writing no identities. /v1/cull/grade warns about photos still without it. A failed full-size decode shows up in the index warnings.

2. Eye state, smile and gaze (MediaPipe)

• Models: MediaPipe Face Mesh V2 (478 landmarks) and Blendshape V2 (52 expression coefficients), both from Google's face_landmarker.task, pinned by URL and SHA-256. Both model cards state Apache-2.0. The uses they exclude, life-critical decisions and identifying people, do not include culling. server-rs/scripts/export_face_state.py (standalone, PEP 723) converts them with tf2onnx and checks every output against TFLite before anything is written: at most 4·10⁻⁴ px off on landmarks, 1.3·10⁻⁶ on blendshapes. Credited in Credits.md.
• Inference (lrg-ml/src/face_state.rs):
  • a crop 1.5× the face, square, rotated along the eye line, scaled to 256 px with a replicated border;
  • two passes: first from the detector box, then from the first pass's landmarks;
  • blendshapes from the 146-landmark subset MediaPipe uses.
• Per face:
  • eye state open / half / closed / unknown ("can't tell": not a face, head turned more than 60°, or eyes under 24 px);
  • from 35° of head turn, only the near eye counts;
  • also smile, gaze away and head turn.
• Indexing:
  • analyses up to 12 prominent faces, largest first, on the full-size original from the same decode the eye focus uses, otherwise on the working copy;
  • per face, FACE_TABLE gets face_eye_state, face_eye_closed, face_smile, face_gaze_away and face_yaw;
  • per photo: cull_eye_closed (worst prominent face), cull_eye_state, cull_smile (least smiling) and cull_gaze_away, plus cull_face_state_version and cull_face_state_model.
• Ranking:
  • the model's reading replaces the proxy, again all or nothing per group, so one group never mixes two scales;
  • eyes_closed replaces possible_blink when measured and is a defect from 0.6;
  • new labels eyes_half_closed (a label, not a reject), looking_away (portrait) and best_smile, with smile weights portrait 0.15 and event 0.10;
  • a laugh with the eyes shut (smile ≥ 0.6) is not a blink;
  • street and sports use higher thresholds, and python_parity turns all of it off.
• Model family, migration and warnings:
  • its own family face_state in /v1/models/assets and in /v1/server/health (face_state_model, face_state_error); Download AI models fetches it (~7 MB);
  • /v1/index/unprocessed re-queues cull-prepared photos with faces but no face-state version, only once the model is on disk;
  • /v1/cull/grade warns with the fix that applies: Prepare now when photos lack the reading, Download AI models when the model is missing;
  • indexing warns once per run when face detection ran without the model;
  • the plugin's model check asks for it before face and cull runs (Util.requiredModelFamilies).

Cost

Measured in release on a 4-core container:

• Eye focus: about 0.24 s more per photo with faces for a 24 MP JPEG original (0.21 s decode + 0.03 s measuring, examples/bench_cull_metrics.rs).
• Face state: about 26 ms per analysed face (two mesh passes and the blendshapes), plus about 40 ms once to load the models. It reuses that decode.
• Photos without faces cost nothing extra.

Merging publishes a model release

The push to main runs model-assets-face-state.yml, since its paths include the new export script. It publishes face-state-assets-v1: face_mesh_v2.onnx and face_blendshapes_v2.onnx, about 7 MB, Apache-2.0. Nothing was dispatched from this branch.

Please check that run after merging. Until the release exists, Download AI models fails on every installation that is missing the eye and expression model. download_release_assets resolves every missing file before it fetches any, so the other families are blocked too. Culling itself keeps working on the old proxy and says so in its warning.

PR CI does not require the facestate goldens yet, since there is no release to fetch them from. The nightly golden-tests-full.yml fetches them from that release.

Test plan

• cargo fmt --check, cargo clippy --workspace --all-targets (no warnings), cargo test --workspace
• Goldens with the real models (LRG_REQUIRE_GOLDENS=face,auraface,facestate, face-state models exported locally):
  • face detection and the existing metrics are unchanged;
  • landmarks on the reference portrait are about 1 % of the interocular distance off MediaPipe's;
  • blink and smile are within ±0.01;
  • synthetically closed eyelids through the blendshape model match MediaPipe to 1e-4.
• focus_goldens against scikit-image: synthetic cases exact to 1e-9; the astronaut's eyes and face to 1e-2 across JPEG decoders.
• Unit tests:
  • focus: eye-only blur, abstaining eyes, full-size refine mapping;
  • face state: rotation, warp and border, projection, eye-state rules incl. near eye and unknown, smile and gaze;
  • aggregation: worst judged prominent face;
  • ranking (all-or-nothing focus as well): the model finds a blink the proxy missed; a laugh is not a blink; an unjudged face is never a blink; half-closed is labelled, not rejected; a group with an unanalysed frame keeps the proxy; the broader smile wins a portrait burst;
  • the warnings name the fix that applies.
• Contract tests:
  • /index/unprocessed re-queue rules, for focus and for face state only with the model on disk;
  • face_state is its own downloadable family;
  • /cull/grade names a real blink, and its metrics, reasons, explanation and warnings.
• cull_eval: winner top-1 100 %, NDCG 1.000, reject recall 100 %, reject precision 85.7 %. The one false reject predates this PR, in the event fixture, where focus is not active.
• Live against the running binary, with curl and the access token:
  • with the model: three 3072 px portraits, by-path and multipart. Full-size focus and face state were measured, the sharp frame won, the eye-blurred frame got eyes_out_of_focus and a reject, the blurred one blurred. No false blinks, and best_smile went to the right frame.
  • without the model: the right warnings, the proxy as fallback, nothing re-queued.
  • migration: with the model, photos were re-queued, prepared again and then clean.
• Plugin: busted 614 successes / 0 failures, luacheck 0 warnings, StyLua clean.
• Not done:
  • There is no freely licensed real photo with closed eyes here. The blink path is pinned through synthetically closed eyelids instead, exactly against MediaPipe.
  • The thresholds (closed ≥ 0.6, half ≥ 0.35, laugh ≥ 0.6, eye-focus defect) stay documented hypotheses until a hand-culled real shoot runs through cull_eval.
  • Not tested inside Lightroom: no catalog was approved for this session.

Docs

• Help-Cull-Photos.md: new sections Focus on the eyes and Eyes open, smile and gaze; reason codes and fields.
• Help-People-Faces.md, Help-Analyze-and-Index.md, Troubleshooting.md.
• Getting-Started.md: the download size is now 4.2 GB. The old 3.3 GB had not counted the quality model since AestheticSigLIP score + genre at index time, and a trimmed set of catalog fields (part 2 of 4) #425.
• Dev-Backend-API.md: health, model families, /index/unprocessed, the index face fields, /cull/grade metrics and warnings, ranking.
• Credits.md (MediaPipe).
• server-rs/README.md (new Face state models section) and plugin/README.md, with the regenerated wiki copies.
• Dev-Image-Culling-Implementation-Plan.md (items 1 and 3 done) and Dev-Image-Culling-Signal-Analysis.md.
• CLAUDE.md, AGENTS.md and CONTRIBUTING.md (golden family facestate, the lrg-ml line), and doc-sources.toml.

Reviewed and still accurate (Docs-Reviewed trailers): Help-Find-Similar.md, Help-Choosing-AI-Model.md, Help-Other-AI-Server.md.

Follow-ups (planned packages)

3. Subject detection (D-FINE-S, license check first): body focus for sports, a wildlife preset, subjects cut off at the frame edge
4. Workflow: key people and a coverage report, an automatic preset by genre, strictness per criterion, a target number of picks, grouping sensitivity, accidental shots
5. A browser review page with face and eye close-ups and traffic lights

🤖 Generated with Claude Code

https://claude.ai/code/session_01HaMMvqLxjmxBq24sQiYQmn