Repository navigation
Culling: a public benchmark, and level with the established tools - #480
Merged
Merged
Conversation
…ore it Every culling threshold so far is a hypothesis: the fixtures under testdata/cull_eval are synthetic and were written by the person who wrote the code. This adds the tooling for a benchmark that needs no hand cull. - scripts/cull_bench.py takes free-licence Commons quality images per genre (event, portrait, sport, wildlife, landscape), turns each into a series with defects made on purpose (missed focus, focus on the background, eyes out of focus, camera shake, exposure, noise) and copies that must survive any cull (shallow depth of field, panning), indexes them on a backend of this build and writes one fixture per genre. Masks and eye positions come from torchvision's Mask R-CNN and Keypoint R-CNN, not from the shipped models. - lrg_analysis::eval reads the benchmark's labels (must_keep, reject_optional, defect, expect_reasons, forbid_reasons) and reports pair order, lost must-keep frames, reasons named and misnamed, per defect. Fixtures without them score as before. - cull_eval gains --by-defect and --tune (sweeps one knob at a time and suggests; changes nothing). - /v1/cull/grade takes include_embeddings and returns each photo's embedding packed as int8 (embedding_q8). Export Culling Fixture asks for it: without it a replayed fixture groups on pHash and time alone, and a darker or softer frame lands in a group of its own. - docs: what the established tools judge (Dev-Image-Culling-Competitive- Analysis.md), the benchmark's README, the new request field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
…ued for 120 Commons quality images, 24 per genre, made into series with defects on purpose, indexed on a backend of this build. What it found, and what changed: - A frame a stop and a third brighter took the pick in 21 of 120 series: the exposure term rewards a mean near mid-grey, and blown highlights were not in the recomputed technical score at all. Clipping beyond the group's least clipped frame now costs (clip_penalty_weight 0.2) and is named (highlights_clipped): 21 -> 5. - Originals were "blinks" and "occluded" against copies of themselves a stop brighter. relative_defect_margin 0.10 -> 0.14, relative_occlusion_margin 0.10 -> 0.20 (also what the real baptism shoot measured), relative_blur_rule on (the other candidate from that shoot). Same labels, before -> after: reject precision 82.3 -> 87.7 %, must-keep frames lost 25.4 -> 16.9 %, winner top-1 31.7 -> 33.3 %, recall 70.0 -> 68.7 %, pair order 77.1 -> 75.9 % (the clipped frame now also ranks below some blurred ones). Open, and documented: grain reads as detail in every focus measure, so the noisy twin takes 58 of 120 series; focus behind the subject is not named. Both are for the subject pass, which needs a new index pass. Also: cull_eval's defect table counts which defect won each series, and --ablate benchmark-calibration reproduces the column before; a CI test holds ranking at today's level on the benchmark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Step 1 of matching the established culling tools. Sharpness used to be the frame-wide reading and the sharpest tile; neither knows what the photo is of, so a shallow depth of field read as blur and focus behind the subject read as sharp. - RF-DETR-Seg Nano (Apache-2.0, COCO, ONNX, about 120 MB, new download family `subject`) finds the subject on the working copy. choose_subject weighs score, area and centrality; people, animals and vehicles that touch join it (the rider and the horse). Focus is the blur effect over tiles inside the subject and clear of it, plus how streaked the background is (a pan). - Ranking: subject focus carries the sharpness (subject_focus_weight 0.75, 1.0 for sports and wildlife), a sharp subject is never "blurred", and a soft one is named subject_out_of_focus / focus_on_background; a pan is named panning. - Only the subject's faces count: for a person only the faces on them, for an animal or a vehicle the faces on it or else every face, beside a thing every face. A soft bystander no longer rejects the keeper, which is what Lightroom's assisted culling is criticised for. cull_faces_detected keeps the full count for the face pass and People; cull_face_subject_version re-runs the face step when the subject is newer than the aggregates. - Grain at full working size (cull_noise_fine, metrics version 2): a frame clearly grainier than a sibling of equal frame-wide sharpness is a noisy twin, ranks below it and does not set the group's best focus, but is never a reject candidate for the grain alone. - A `wildlife` preset and prompt set. The benchmark, made again for this (sport subject with what carries the athlete; no subject defects where a person as large as one of the subject's is left out; eyes_soft only on the subject's eyes), on the same measurements without -> with these rules: winner top-1 31.7 -> 73.3 %, pair order 78.6 -> 85.9 %, reject precision 90.7 -> 93.0 %, recall 68.4 -> 69.2 %, must-keep lost 12.1 -> 9.1 % (16.9 % at the first run), wrong reason 5.2 -> 1.2 %. The noisy copy wins 3 series in 120, was 56; focus behind the subject is named 85 %, was 0 %. CI floors raised to match. Open, documented: a face found in one exposure only, the occlusion proxy, the face state on a soft-background twin, and who the subject is among people. Also: cull_eval --mistakes lists every keeper lost and wrong reason, and --ablate subject reproduces the column before; cull_bench.py make --genres. Docs-Reviewed: Help-Analyze-and-Index.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Step 2 of matching the established culling tools. The genre is stored for every photo, but the photographer still had to pick one preset for a whole selection — a wedding and the football match that afternoon were culled with the same one. - `culling_preset: "auto"`: each group is ranked with the preset most of its photos' genres call for (preset_for_genres; a tie goes to the preset that asks more). Grouping uses the preset most of the selection calls for, and each group's prompts are scored with its own preset's sets. - A guessed "street" is ranked as an event: the street preset leaves focus unjudged on purpose, and the genre model calls 13 of 24 portraits and 12 of 24 events on the benchmark "street". Mapped to the street preset, auto found 63.4 % of the rejects and 48 % of the eyes that missed focus; as an event, 69.2 % and 80 %. - grouping: group_and_sort_images_with_config is now partition_records + Partition::rank, so a caller can choose a config per group; grouping itself is unchanged (every existing test and golden passes as before). - API: groups[].preset, summary.presets (groups per chosen preset), and a warning counting photos without a genre, with the fix. Fixtures carry the genre; cull_eval --preset auto ranks the way the backend does. - Plugin: "Auto (each burst by its genre)" is the first option and the default for a new setup; the completion message says what it chose. The fixture export and the MCP cull tool know `auto`. Benchmark, same measurements, hand-chosen preset per genre -> auto: winner top-1 73.3 -> 77.5 %, pair order 85.9 -> 86.2 %, reject precision 93.0 -> 93.3 %, recall 69.2 -> 69.2 %, must-keep lost 9.1 -> 8.3 %. CI holds auto at or above the hand-chosen presets. Also fixes the subject weight the API page still gave from before step 1's calibration (0.75; sports and wildlife 1.0). Docs-Reviewed: Help-Find-Similar.md, Help-AI-Agents.md, Dev-Agent-Bridge.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Step 3 of matching the established culling tools.
Key people (`key_people: true`, opt-in in the dialog):
- Who counts: the persons named in People who are in the selection, and
anyone in at least a fifth of its photos with faces and in three at least
(lrg_analysis::key_people). Faces without a person in People are clustered
among the selection's own, so it works on a shoot never opened in People.
- In a group where any frame shows one, every frame's face aggregates are
summed again from its key faces alone, however small — the bride's blink
counts though a guest stands larger in front of her, and a frame where she
turned away has no face that matters. Reasons about eyes, focus and gaze
name the key person ("Anna: eyes closed"); the names never reach a fixture.
- Reads FACE_TABLE (Store::get_by_id_prefixes), so the prep pass adds the
faces task. Photos never through face recognition are counted in a warning
with the fix; the response lists the key people and the plugin's completion
message names them.
- Not measured: no public data says whom a photographer cared about. A
contract test holds the behaviour.
Small faces:
- YuNet looks again at four overlapping tiles of the working copy. A tile face
is kept when the first look could not have seen it (under 20 px in its
640 px view), lies whole in its tile and overlaps no face found; it comes
after the first pass's faces, which are unchanged (goldens pass). It counts
for culling and gets no identity, no row, no thumbnail, so People never
sees it.
- On the benchmark's 120 photographs it added 101 faces, about 92 of them real
by eye (spectators, the back of a group). Without the size limit the larger
tile faces were as often pots, an accordion or a horse as people.
- Re-indexed: recall 69.2 -> 70.8 %, precision 93.0 -> 92.8 %, keepers lost
unchanged at 9.1 %; eyes that missed focus found 80 -> 82 %.
A frame whose face was not found beside siblings whose was, with its subject
in focus, ranks below them but is no reject on the score margin alone: the
tiles had made that the one keeper they cost.
Docs-Reviewed: Help-People-Faces.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Step 4 of matching the established culling tools. Ranking answers which frame of a burst is best; a photographer delivering a gallery asks for "the best 300" or "a fifth of it", with every part of the day in it. - lrg_analysis::selection: consecutive groups close in time (90 s) and alike (picks within 0.35 cosine distance) are one scene. A keep value comparable between groups — the aesthetic score, blended with the moment, halved by a clear defect — chooses the best pick of every scene first, then fills the target by keep value less similarity to what is chosen (frames of one group count 0.9 alike, embedding or not) and less 0.1 per place below the group's pick, with bonuses for a group or a key person not yet in. A reject candidate only as the only frame of its scene, and said. Highlights: the best chosen, kept apart, at most one per group. - /v1/cull/grade: target_count or target_fraction, highlights_count; groups[].scene_id and selected_photo_ids, photos[].selected/highlight, summary.selection, warnings when fewer were worth keeping or a scene had only flawed frames. Without a target nothing changes. - Plugin: "Keep: the pick of every burst | a number | a share", "Highlights"; under a target the chosen photos are the picks (flags included) and a Highlights collection holds the highlights. - cull_eval --target N|P% --highlights N; on the benchmark one photo per series covers every series with its best frame 76.7 % of the time and a labelled reject 0.8 %; two per series 63.6 % / 2.9 % (the rank cost came from this: 56.9 / 3.3 % without it). CI holds the first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Step 6 of matching the established culling tools. A Lightroom catalog holds years of the photographer's flags and stars, which no stand-alone culling tool can read. - lrg_analysis::taste: inside each group the photographer culled (a mark of theirs, not one a cull wrote and they left), two frames marked differently are a decision. A pairwise logistic regression without bias, on the ranked signals scaled by their spread — the default ranking's score among them — learns a correction to culling. Five-fold cross-validation by group compares it with the default ranking on the same held-out decisions; it is used only from 30 decisions and when it agrees more often. Applied, it reorders the frames that are not reject candidates; a changed pick says so. - /v1/cull/taste: POST learns (422 with the reason under 30 decisions), GET reports, DELETE forgets; the model lives next to the database. /v1/cull/grade takes use_taste and says when no useful style exists. - Plugin: "Learn My Culling Style..." reads flags, stars and what the last cull wrote (Util.cullCurrentMarks, now shared with the cull), sends the decisions and reports how often the style agrees with the photographer against the default ranking; "Use my culling style" in the cull dialog. - Given the benchmark's labels as if they were a photographer's: 91.6 % agreement on held-out series, the default ranking 87.3 %. Also records step 5's licence gate: no animal-eye model has weights and training data clear for this project's use, so it stops there until the maintainer decides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Found while checking step 6 on the benchmark: 31 of its 120 series were read as focus stacks and kept whole, so none of their rejects was ever nominated. Each was a tripod series without people with one camera-shaken frame: sharp nowhere, its "sharpest region" lands wherever the noise is, and the region "travels" — the one sign the focus stack rule looks for. A real tripod burst with one shaken frame is read the same way. Every slice of a focus stack is sharp somewhere, at its own depth. A focus stack now needs every frame's sharpest part (cull_sharpness_peak) at focus_stack_min_peak_ratio (0.35) of the sharpest frame's, where it is known. Benchmark: reject recall 70.8 -> 86.9 % (auto 87.2 %), precision 92.8 -> 94.1 % (94.4 %), keepers lost unchanged; landscape recall 35 -> 67 %, wildlife 14 -> 77 % — most of what the first run put down to those genres finding few rejects. The labelled sets of testdata/cull_eval are still kept and recognised. CI floors raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
…pped Step 7 of the culling plan was to ship an optional vision-model tiebreak only if it decides close calls better than the ranking. Measured, it does not. - cull_eval --close-calls FILE [--gap G] exports the groups whose pick and runner-up score within G of each other, with their labels: 94 at 0.05 on the benchmark, 53 with differently labelled frames; the ranking chose the better one in 34 (64.2 %). - lrg-llama's tiebreak_probe example shows a local model both frames side by side (alternating sides) and asks which to keep, constrained to JSON. Qwen2.5-VL 3B (Q4_K_M): right in 32 (60.4 %), answering "the left one" 96 % of the time — by position, not by the photographs. Not shipped. The benchmark's close calls are technical (one moment per series), so a larger or cloud model, or a hand-culled shoot for moment and expression, can be measured with the same two commands. Documented in the benchmark and plan pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Culling measured against a public benchmark, and brought level with — in places past — the established culling tools (Lightroom Assisted Culling, Aftershoot, Narrative, FilterPixel, Imagen, Excire). The research and the plan are in
docs/wiki/Dev-Image-Culling-Competitive-Analysis.mdandDev-Image-Culling-Implementation-Plan.md("Matching the established tools"); every number is inDev-Image-Culling-Benchmark.md.What it does, step by step
cull_eval --by-defect/--tune/--mistakes/--target.subject): a shallow depth of field or a pan is not "blurred", focus behind the subject is named; only faces on the subject count (a soft bystander no longer rejects the keeper); noisy twinsculling_preset: "auto", the plugin's new default)use_taste), proven by cross-validation against the default ranking and used only when betterOverall on the benchmark (same labels,
auto): winner top-1 77.5 %, pair order 86.2 %, reject precision 94.4 %, recall 87.2 %, must-keep lost 8.3 %, wrong reason 1.2 %. CI (cull_bench_fixtures.rs) holds these floors.Step 7 (a vision model for close calls) was measured and not shipped: on the benchmark's 53 close calls with differently labelled frames, a local Qwen2.5-VL 3B chose right 60.4 % of the time against the ranking's 64.2 %, and said "the left one" 96 % of the time. The measuring tools stay (
cull_eval --close-calls,lrg-llama'stiebreak_probe), so a larger or cloud model can be checked the same way.API
/v1/cull/grade:culling_preset: "auto",key_people,target_count/target_fraction,highlights_count,use_taste;groups[].preset,scene_id,selected_photo_ids;photos[].selected/highlight;summary.presets,selection,key_people,taste_changed_picks; new warnings for each degradation. New:POST/GET/DELETE /v1/cull/taste. Index: subject fields,cull_subject_mask,cull_faces_detected,cull_face_subject_version,cull_noise_fine(metrics version 2) — older photos are re-queued by/v1/index/unprocessed. All inDev-Backend-API.md.Test plan
cargo fmt,cargo clippy --workspace --all-targets(zero warnings),cargo test --workspaceLRG_REQUIRE_GOLDENS=face,auraface,facestate,subjectgoldens with models presentbusted(one failure,aesthetic_refine_spec.lua, predates this branch and is a Lua 5.4 number-formatting difference)scripts/check-docs.py, pre-commit hooks/v1/cull/gradewithauto,key_people, a target and highlights,/v1/index/unprocessed,/v1/cull/tastelearn/status🤖 Generated with Claude Code
https://claude.ai/code/session_01HaMMvtmmaW5VfgA4b4aa5V