Repository navigation
Face state: signals that hold on real photographs, for culling, People, search and AI Edit - #474
Merged
Merged
Conversation
The face state thresholds were pinned on one portrait and on lids moved shut on the mesh. Measured on photographs (examples/face_state_probe.rs, 285 faces from Wikimedia Commons to choose by, 292 from a disjoint sample to check by, not committed), neither reading meant what the ranking took it to mean. Eyes. eyeBlink tops out low: eyes shut read 0.54-0.78 on the more closed eye and 0.41-0.72 as a mean, so the mean at 0.6 caught 6 of 22 shut or lowered pairs. The more closed eye now speaks, with thresholds where the readings are: 19 of 22, about 17 of 24 on the second sample (was about 8), and 1 of 178 open-eyed portraits over the label. Gaze. eyeLook* is the eye inside the head, so the label had it backwards: people turned 25-30 degrees looking straight at the camera read 0.85-0.99, a head turned 39 degrees with the eyes following it 0.14. It is now degrees off the lens, head and eyes together, from a head pose read off a frame of bone landmarks (within 1 degree of a full Procrustes fit on yaw and roll, 2 on pitch, at no linear algebra dependency). looking_away in portraits drops from 25-31% to 10-12% and rises from 53% to 67% in People looking away. Shut eyes abstain; their eyeLook* is noise. Squint is read at last, only on a face that is not smiling. All 52 blendshapes are stored per face, so a later signal need not decode every original again. The version check read the key's presence, so a bump would have changed nothing; its value counts now. #471 is in no release, so only development databases are analysed again. Also from the live test: shut eyes were reported as the best open eyes of their group, a single always being its own best. Limits, in the wiki: a slit overlaps eyes shut, sunglasses read closed (the relative rules hold inside a burst), a head on its side reads open. The sports and street thresholds are scaled by judgement, not measured. A hand-culled shoot still has to confirm all of it. Verified: cargo fmt/clippy/test (workspace) with face and facestate goldens on the real models, check-docs, and live against the release binary by path. Docs-Reviewed: Help-Analyze-and-Index.md Docs-Reviewed: Help-Find-Similar.md Docs-Reviewed: Credits.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy
…a photo People and search can now ask what a face shows, culling names the mouth and the light on the face, and AI Edit reads the faces it is about to edit. All of it measured on photographs from Wikimedia Commons first (not committed), the same way as the eyes and gaze before it. People and search. lrg_analysis::face_expression answers eyes open, smiling (mouthSmile from 0.5) and looking at the camera (within 20 degrees), per person on GET /v1/faces/persons/<id>/photos and for everyone in the frame on /v1/search, both pages and the MCP tools. A reading the model did not make is unknown, not a no: it is counted and the fix named. A person's card shows their best portrait, no longer the sharpest blink, and "best N" takes each person's best photos. A catalog indexed before the model was on disk has rows with no readings, and a cull pass never touched them. It now brings its readings to the rows a full pass wrote, matched by box, leaving identities alone, and stamps the photo current only once they are saved. /index/unprocessed queues a face pass for readings that are out of date or rows the cull pass could not match (face_rows_behind). Checked live: indexed without the model, a person named, cull pass with it: the person keeps the name and the filters answer. Culling gains labels, which move no score until a hand-culled shoot says what they are worth: mouth_open (portrait, event), face_darker, face_hard_light, face_blown. What the measuring changed: - mouthPucker reads up to 0.99 on full lips at rest and mouthFunnel barely moves, so the mouth is jawOpen and mouthLowerDown (from 0.2: 10% of people speaking, 0.6% of portraits). - Blown is the middle channel, not the top: warm light clips red on skin that holds its detail, and that flagged 16% of faces. It is now 2.6% over 0.05. - Luminance is only ever compared with the same people in a sibling frame, since skin ranges over several stops. AI Edit: face_hard_light takes clarity and most contrast, face_blown holds the white point, face_below_surroundings holds contrast. That last reading is lowest on people with dark skin in front of something bright, correctly exposed, so it never calls a face underexposed or backlit: it can only hold contrast back, and says what it measured. A failed read reaches the user; missing models skip the rules, as the comment there explains. Left out on the evidence: keeping turned heads out of cluster formation scores the same at every limit from 60 to 30 degrees on 370 faces of 18 public figures, since the occlusion gate already holds them back, and raising that gate lowers the score. Clustering is unchanged; examples/face_cluster_eval.rs is what measured it. Verified: cargo fmt/clippy/test (workspace) with face and facestate goldens on the real models, check-docs, live against the release binary. busted has one failure, aesthetic_refine_spec.lua:76, which fails the same without this change. Not yet run inside Lightroom. Docs-Reviewed: Credits.md Docs-Reviewed: Dev-Agent-Bridge.md Docs-Reviewed: Help-Find-Similar.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy
The first run on a real shoot, a test catalog of 197 photos, showed what the Commons samples did not: 36 faces read closed, nearly all a child looking down at candles with the lids over the eyes. That made a false "eyes closed" and could make a reject candidate of a frame a photographer may well want. Lids down on a head lowered 10 degrees or more now read `down`. Such a face is left out of the blink reading and the frame says "looking down", never as a defect. Eyes going down count as neither open nor on the camera in the People and search filters, and cost a portrait what half-closed eyes do. Measured: 30 of 39 faces reading closed in that catalog become `down`, and all 18 of the highest, checked by eye, really do look down. 10 of 34 faces with the eyes really shut, mostly people asleep, get the note too, which costs a warning where a reject would have been, the right way round. eyeLookDown does not separate them. Faces reading closed fall from 36 to 7. The rest are mostly gazes on a barely lowered head and lids on a raised head behind glasses, which the relative rules hold back inside a burst. FACE_STATE_VERSION goes to 3, so those photos are analysed again. Also says what a guardrail did: "no clarity was added", not "left alone", for a value it had set to 0. Verified: cargo fmt/clippy/test (workspace) with face and facestate goldens on models downloaded from the published release, check-docs, and in Lightroom Classic 15.6 on the test catalog: 123 photos with faces queued again by the version, names kept. Docs-Reviewed: Help-Analyze-and-Index.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy
Found in Lightroom, on the test catalog. Ten frames of a child behind birthday candles all read `blurred`, on a frame-wide sharpness of 0.03-0.05, while the eyes read 0.54-0.60 and are visibly sharp. With every frame a defect the group had no pick and all ten were reject candidates, so deleting the candidates would have lost the whole series, next to explanations reading "noticeably blurred" and "eye focus 10/10". A frame whose eyes reach reason_eye_focus_threshold is no longer a blur defect, on the top frame's own check as well. In portrait every group now has its pick, and candidates fall from 74 to 69, the one really smeared frame still among them. Older than this PR, fixed here because these signals are what make it possible. It is one catalog, and the reading depends on magnification, so a slightly soft frame shot close can pass. Still false: frames with no measurable eyes, a sharp cat or a boy's shirt at 0.09, which default and event still reject as a whole group. The eyes cannot help there; the frame-wide threshold needs a hand-culled shoot. Also: a lowered head foreshortens the lips, so a closed mouth read open and got mouth_open. Such a face now abstains. Verified: cargo fmt/clippy/test (workspace) with goldens, cull_eval unchanged (winner top-1 7 of 7), check-docs, and re-graded live on the test catalog. Docs-Reviewed: Help-Analyze-and-Index.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures the face signals from #471 on real photographs, fixes what they got wrong, and takes them to People, search and AI Edit. Measured on portraits from Wikimedia Commons, which are not committed.
docs/wiki/Dev-Image-Culling-Implementation-Plan.mdhas the numbers and limits.examples/face_state_probe.rsandexamples/face_cluster_eval.rsreproduce them.1. Eyes and gaze
The thresholds were pinned on one portrait and on lids moved shut on the mesh.
looking_awayin portraits, second samplelooking_awayin Commons' People looking awayeyeLook*is the eye inside the head, so the old label had it backwards. Gaze is now degrees off the lens, head and eyes together.2. People and search
GET /v1/faces/persons/<id>/photosand for everyone in the frame on/v1/search. Available on both pages and in the MCP tools./index/unprocessedqueues a face pass for readings out of date or rows it could not match.Clustering is unchanged, on the evidence. Keeping turned heads out of cluster formation scores the same at every limit from 60° to 30° (370 faces of 18 public figures), because the occlusion gate already holds them back. Raising that gate lowers the score.
3. Mouth and the light on the face
mouth_open,face_darker,face_hard_light,face_blown. A hand-culled shoot has to say what they are worth first.face_hard_light,face_below_surroundings,face_blown.Face against surroundings is deliberately weak. Its lowest readings are mostly people with dark skin in front of something bright, correctly exposed. So it never produces a label and never calls a face underexposed. It can only hold back added contrast. Luminance is only compared with the same people in a sibling frame.
Found in Lightroom
Commons did not show these; the first run on a real shoot (197 photos) did.
down, with a note and no defect. Faces reading closed: 36 → 7. The price: 10 of 34 faces really shut, mostly people asleep, get the note instead of a reject.blurred, so the group had no pick and all ten were reject candidates. Eyes atreason_eye_focus_thresholdnow settle it. Inportraitevery group has its pick again.Known limits
blurredwhen they are sharp: a cat at 0.09 on the frame-wide reading.defaultandeventstill have 2 and 1 groups without a pick for this reason.Testing
cargo fmt,clippy, tests withLRG_REQUIRE_GOLDENS=face,facestate, on models downloaded from the publishedface-state-assets-v1check-docs,luacheck,styluabustedon Lua 5.1 in CI. Locally on Lua 5.5,aesthetic_refine_spec.lua:76fails because 5.5 prints1.0, unrelated to this PRTaskPeople.luato collections with all four outcomes, search page in headless Chrome, guardrails through the route with a fake LLM,run_plugin_tests28 of 28, live MCP suite 6 of 6, runbook phases 2, 3 and 5, and the phase 4 checks that need no handscull_eval --ablate🤖 Generated with Claude Code
https://claude.ai/code/session_01HaMMvsyz1bjNzJoq8hSGXy