Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ Whether you prefer running local models to ensure maximum privacy or want to lev
- **Cull Similar Photos...** *(beta)* — burst grouping and auto-ranking
- **People...** — face clusters and named person collections
- **Find Similar Images...** — find near-duplicates or visually similar photos
- **Deduplicate Keyword Synonyms...** — clean up synonym sprawl in your catalog
6. For AI Edit, first teach it your style: edit a few photos by hand and run **Save Edits as AI Training Examples...**. AI Edit builds every recipe from those examples and needs at least five of them — it calls no LLM. Keep **Review each proposed edit before applying it** enabled while you validate the results.
- **Deduplicate Keyword Synonyms...** — clean up synonym sprawl in your catalog (in the **Help** menu)
6. For AI Edit, first teach it your style: edit a few photos by hand and run **Save Edits as AI Training Examples...** (in the **Help** menu). AI Edit builds every recipe from those examples and needs at least five of them — it calls no LLM. Keep **Review each proposed edit before applying it** enabled while you validate the results.

*For comprehensive details, model setup guides, and tips, please visit [lrgenius.com/help](http://lrgenius.com/help/).*

Expand Down
1 change: 1 addition & 0 deletions docs/wiki/Credits.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ LrGeniusAI is made possible by these amazing open-source projects and AI framewo
## Core Backend Dependencies
- **YuNet**: Fast, lightweight face detector from the OpenCV Model Zoo; provides the face boxes and five landmarks. [GitHub](https://github.com/opencv/opencv_zoo)
- **FaceNet (facenet-pytorch)**: Inception-ResNet-v1 with VGGFace2 weights; produces the face embeddings that group photos by person. [GitHub](https://github.com/timesler/facenet-pytorch)
- **AuraFace (fal.ai)**: ArcFace-style ResNet-100 trained on commercially usable data, Apache-2.0; the optional, more accurate face recognizer. [Hugging Face](https://huggingface.co/fal/AuraFace-v1)
- **SigLIP2 (NaFlex)** (Apache-2.0): Google's `siglip2-so400m-patch16-naflex` ([SigLIP 2, Tschannen et al., 2025](https://arxiv.org/abs/2502.14786)), the vision-language model behind semantic search and the photo genre; it keeps each photo's aspect ratio. Exported with Hugging Face Transformers. [Model](https://huggingface.co/google/siglip2-so400m-patch16-naflex)
- **AestheticSigLIP**: Gowthami Somepalli's `AestheticSigLIP`, a SigLIP 2 So400m NaFlex encoder fine-tuned to score photos from 1 to 9, which provides the *Aesthetic score*. Apache-2.0. [Model](https://huggingface.co/somepago/AestheticSigLIP)
- **CLIP-IQA**: The prompt-pair method (Wang, Chan & Loy, *Exploring CLIP for Assessing the Look and Feel of Images*, AAAI 2023) behind the moment, emotion and kiss-gate questions culling asks of a photo. [Paper](https://arxiv.org/abs/2207.12396)
Expand Down
91 changes: 84 additions & 7 deletions docs/wiki/Dev-Backend-API.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@ when it is `"failed"`.
| `clip_image_model` / `clip_image_error` | The SigLIP2 **image** tower alone — what indexing and the training path embed with. |
| `clip_text_model` / `clip_text_error` | The SigLIP2 **text** tower alone — what search, keyword lookup and the CLIP-IQA prompt sets embed with. |
| `aesthetic_model` / `aesthetic_error` | AestheticSigLIP, the photo quality scorer. Its own backbone, loaded and idle-unloaded on its own clock. |
| `face_model` / `face_error` | YuNet + FaceNet. |
| `face_model` / `face_error` | YuNet + the active recognizer (FaceNet or AuraFace). |
| `face_recognizer` | Which recognizer is active: `"facenet"` or `"auraface"` (see `GET /v1/faces/recognizer`). |
| `species_model` / `species_error` | BioCLIP 2. |

The two SigLIP2 towers load and idle-unload **independently**, so an
Expand Down Expand Up @@ -247,10 +248,11 @@ preview for the browser search page — …"`).

`/v1/index/unprocessed` does **not** count a missing preview as outstanding:
the cull pre-flight and the agent's indexing command ask it too, and would offer
to prepare photos that lack nothing they need. A photo indexed before previews
were stored gets one from the next run that *sends* it — any scope of Analyze &
Index but *New or unprocessed photos*, with *Skip photos with existing data*, so
nothing else is made again.
to prepare photos that lack nothing they need. Photos indexed before previews
were stored get theirs from *Add Missing Search Previews*
(`POST /v1/photos/thumbnails/missing`, then `POST /v1/photos/thumbnail` per
photo), which renders only the preview and touches nothing else; the next run
that *sends* a photo makes one too.

**The computer stays awake.** Both index routes run through `process_batch`,
which holds `AppState::keep_awake` for the batch (`crate::keep_awake`). On macOS
Expand Down Expand Up @@ -308,7 +310,7 @@ route; the search page fetches it with the header and shows the bytes as a
|---|---|---|
| 200 | the JPEG | a preview is stored |
| 400 | `{"error"}` | no `photo_id` |
| 404 | `{"error", "error_code": "no_thumbnail"}` | none is stored: the photo was indexed before previews were, or not at all. An answer, not a failure — the page draws a placeholder and counts these |
| 404 | `{"error", "error_code": "no_thumbnail"}` | none is stored: the photo was indexed before previews were, or not at all. An answer, not a failure — the page draws a placeholder and counts these, and names *Add Missing Search Previews* as the fix |

**Disk cost and lifetime.** One file per photo,
`<lance root>/thumbnails/<aa>/<sha256 of the photo_id>.jpg` (the lance root is
Expand All @@ -322,6 +324,38 @@ cleanup's purge (`POST /v1/photos/catalogs/cleanup`), only for rows that are
really gone; left out of `POST /v1/db/backups`, since any indexing run makes
them again.

### `POST /v1/photos/thumbnails/missing`
Which **indexed** photos have no search-page preview — what the plug-in's
*Help ▸ Add Missing Search Previews* asks before it renders
anything. Body (JSON, every field optional):

| Field | Meaning |
|---|---|
| `photo_ids` | Only these photos (the selection). Left out: every indexed photo. Not a list → 400 |
| `catalog_id` | Only photos this catalog has (`catalog_ids`) |

Returns `{"photo_ids": [...], "indexed": <n>, "missing": <n>}` — `indexed`
counts the photos the question covered that have an index row, `photo_ids`
those of them without a preview. A photo that is not indexed is never listed:
a preview belongs to an index row, and the next indexing run makes one anyway.
No database bound → the empty answer. Costs a metadata scan (like
`GET /v1/photos/ids`) plus a file stat per photo.

### `POST /v1/photos/thumbnail`
Stores the search-page preview of an already indexed photo, without indexing it
again. Multipart: `photo_id` and one `image` part — any JPEG/PNG; the plug-in
sends a small export (512 px). The backend scales it down
to the size indexing stores (256 px long edge, quality 75) and writes it where
`GET /v1/photos/thumbnail` reads it, replacing any earlier one. Nothing else
about the photo changes.

| Status | Body | When |
|---|---|---|
| 200 | `{"status": "success", "photo_id"}` | stored |
| 400 | `{"error"}` | no `photo_id`, no `image`, or an image that cannot be decoded |
| 404 | `{"error", "error_code": "not_indexed"}` | the photo has no index row — refused, since nothing would ever delete a preview of a photo the index does not know |
| 503 | `{"error"}` | no database is bound |

### `GET /v1/photos/ids`
Returns a list of all indexed `photo_id` values, optionally filtered by catalog.

Expand Down Expand Up @@ -694,7 +728,50 @@ them decides what a re-cluster is allowed to touch:
Detects faces in an uploaded image and stores their embeddings.

### `POST /v1/faces/search`
Finds photos containing faces similar to those in a reference image.
Finds photos containing faces similar to those in a reference image. Only
stored faces from the recognizer that embedded the query (`face_model`) are
ranked: after a FaceNet/AuraFace switch the other rows are vectors of another
space until their photos are re-indexed. How many were left out comes back in
`warnings`, so a search right after a switch (no results at all) says why
instead of looking like nobody matched.

### `GET /v1/faces/recognizer`
Which network makes face identity embeddings (`lrg_ml::faces::Recognizer`):

```json
{"active": "facenet", "saved": "facenet", "model_id": "yunet-2023mar/facenet-vggface2",
"recognizers": [{"id": "facenet", "name": "FaceNet", "description": "...", "ready": true,
"active": true, "model_id": "...", "approx_bytes": 0, "downloadable": false},
{"id": "auraface", "name": "AuraFace", "ready": false, "downloadable": true,
"approx_bytes": 261000000, ...}],
"warnings": []}
```

`saved` differs from `active` when the saved choice could not be restored at
start (AuraFace's file is gone); `warnings` then says so, for the Plug-in
Manager.

### `POST /v1/faces/recognizer`
`{"recognizer": "facenet" | "auraface"}` — switches, saves the choice (restored
at start) and answers like `GET`, plus `changed` and `warnings` for the user
(the faces of the other recognizer are set aside until re-indexed; names carry
over). 400 for an unknown name; 409 `error_code: "not_downloaded"` when the
recognizer's model is not on disk.

The switch changes `face_model` on every row written afterwards. Clustering,
`/v1/faces/search` and `/v1/index/unprocessed` only treat rows of the active
model as current, so the two spaces are never compared. On re-detection
(`POST /v1/index/photos` with faces), person assignments of previous rows from
the *other* recognizer are carried over by box overlap (IoU ≥ 0.6) — the
detector is the same, so a face lands on its old box — while rows of the same
recognizer keep matching by embedding.

### `POST /v1/faces/recognizer/downloads` / `GET /v1/faces/recognizer/downloads`
Starts the AuraFace download (`fal/AuraFace-v1`, `glintr100.onnx`, from
Hugging Face — `HF_ENDPOINT` is honoured) / reports its progress, in the same
shape as `/v1/models/assets/downloads` (`status`, `progress`, `total`,
`current_file`, `error`). Does not switch; the plugin calls
`POST /v1/faces/recognizer` once it completed.

### `POST /v1/faces/cluster`
Re-clusters all stored face embeddings into person groups. The policy lives in
Expand Down
12 changes: 10 additions & 2 deletions docs/wiki/Dev-Plugin-README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,10 @@ The plugin is designed to work with local and cloud providers, while keeping Lig
- Optional face detection and clustering
- Optional species identification (BioCLIP 2, on-device)
- Every processed photo also gets a small preview (256 px, ~13 KB, stored by the backend beside
the index) for the browser search page
the index) for the browser search page. Photos indexed before previews were stored get theirs
from *Add Missing Search Previews...* (`TaskFillSearchPreviews.lua`): it asks the backend which
indexed photos lack one, exports small JPEGs of those in batches, and touches nothing else.
(Not `requestJpegThumbnail`: Lightroom delivers a few previews and then stops answering.)

### Advanced Search

Expand All @@ -58,6 +61,10 @@ The plugin is designed to work with local and cloud providers, while keeping Lig
### People Workflows

- Cluster faces into persons
- Choose the face recognizer in Plug-in Manager (*Face recognition*): FaceNet (default) or
AuraFace (more accurate, ~260 MB download). `PluginInfoDialogSections.switchFaceRecognizer`
downloads AuraFace when needed and switches via `/v1/faces/recognizer`; names carry over when
the photos are re-indexed
- Rename persons
- Jump from a person directly to a Lightroom collection
- The People interface opens in your web browser (served by the backend at
Expand Down Expand Up @@ -313,7 +320,8 @@ In the plugin settings dialog you can configure:
(`LocalModelCatalog.engineFor`).
The section reports why it is unavailable when the host cannot use it — a
source build without the `llamacpp` feature, or a missing MLX helper.
- Export size and quality used for AI processing
- Export size and quality used for AI processing, photos per export session (indexing exports
in batches of that size, `Util.exportChunkSize`, so Lightroom frees its memory between them)
- Prompt presets — a preset is the *system* prompt: which expert is looking at
the photo, which vocabulary they use, how specific they may be, and what they
must never invent. **Default** (the analytical voice) and **Family &
Expand Down
4 changes: 2 additions & 2 deletions docs/wiki/Dev-Project-README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ Whether you prefer running local models to ensure maximum privacy or want to lev
- **Cull Similar Photos...** *(beta)* — burst grouping and auto-ranking
- **People...** — face clusters and named person collections
- **Find Similar Images...** — find near-duplicates or visually similar photos
- **Deduplicate Keyword Synonyms...** — clean up synonym sprawl in your catalog
6. For AI Edit, first teach it your style: edit a few photos by hand and run **Save Edits as AI Training Examples...**. AI Edit builds every recipe from those examples and needs at least five of them — it calls no LLM. Keep **Review each proposed edit before applying it** enabled while you validate the results.
- **Deduplicate Keyword Synonyms...** — clean up synonym sprawl in your catalog (in the **Help** menu)
6. For AI Edit, first teach it your style: edit a few photos by hand and run **Save Edits as AI Training Examples...** (in the **Help** menu). AI Edit builds every recipe from those examples and needs at least five of them — it calls no LLM. Keep **Review each proposed edit before applying it** enabled while you validate the results.

*For comprehensive details, model setup guides, and tips, please visit [lrgenius.com/help](http://lrgenius.com/help/).*

Expand Down
24 changes: 24 additions & 0 deletions docs/wiki/Dev-Server-README.md
Original file line number Diff line number Diff line change
Expand Up @@ -419,6 +419,30 @@ which is where `POST /v1/models/assets/downloads` puts them. Assets are publishe
by `.github/workflows/model-assets-face.yml` to the fixed `face-assets-v1`
tag, independent of the other two families.

**AuraFace (optional recognizer).** `fal/AuraFace-v1`'s `glintr100.onnx`
(Apache-2.0, ~260 MB) can replace FaceNet: `POST /v1/faces/recognizer/downloads`
fetches it from Hugging Face (`HF_ENDPOINT` is honoured) to
`~/.cache/lrgenius/models/auraface_glintr100.onnx` (or `LRG_AURAFACE_ONNX`),
and `POST /v1/faces/recognizer` switches. The choice is saved in
`~/.cache/lrgenius/models/face_recognizer.txt` (or `LRG_FACE_RECOGNIZER_FILE`)
and restored at start. It runs on the same YuNet landmarks and the same 112x112
ArcFace crop InsightFace makes, RGB, `(v - 127.5) / 127.5`
(`lrg-ml/src/auraface.rs`). The download is pinned to the Hugging Face commit
(`AURAFACE_REVISION` in `lrg-api/src/routes/face_recognizer.rs`) whose model
InsightFace's own embedding in `testdata/face_goldens.json` was made from; the
golden test checks the chain against it (cosine 0.9988, needs > 0.99). To move
the pin, record the reference for the new commit and run the test again:

```bash
uv run --no-project server-rs/scripts/auraface_reference.py --revision <commit> --output ~/.cache/lrgenius/models/auraface_glintr100.onnx
cargo test -p lrg-ml --test face_goldens -- auraface # LRG_REQUIRE_GOLDENS=auraface to fail instead of skip
```

The two recognizers' rows carry different `face_model` ids
(`yunet-2023mar/facenet-vggface2`, `yunet-2023mar/auraface-v1`), so nothing
compares across them; on re-detection, person assignments from the other
recognizer's rows are carried over by box overlap (same detector).

These replaced InsightFace's `buffalo_l` (SCRFD + ArcFace), which was
resolved from `INSIGHTFACE_ROOT` and could not be redistributed — face
detection was the one family the plugin could not download for you. Note the
Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ None. AI Edit does not call a language model at all — it interpolates the deve

### AI Edit says my style profile is not ready

It needs at least five saved training examples before it can produce anything. Edit some photos the way you like them and run **Save Edits as AI Training Examples** (*Library → Plug-in Extras*). See [Help: Train from Edits](Help-Train-From-Edits).
It needs at least five saved training examples before it can produce anything. Edit some photos the way you like them and run **Save Edits as AI Training Examples** (*Help* menu). See [Help: Train from Edits](Help-Train-From-Edits).

### I want to review edits before they are applied

Expand Down
11 changes: 5 additions & 6 deletions docs/wiki/Help-Advanced-Search.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,12 +164,11 @@ The previews are small copies (256 pixels on the long side) that **Analyze &
Index Photos** stores for each photo it processes, whatever you ask it to do;
they take about 13 KB a photo on the backend's disk, roughly 130 MB per 10,000
photos, beside the search index. Photos indexed before this version have none
yet: they show *No preview yet*, and the page says how many. To add them, click
**Show in Lightroom**, select the photos of the new collection and run **Analyze
& Index Photos** with the scope *Selected photos only* and *Skip photos with
existing data* — the previews are added and nothing that is already there is made again.
The scope *New or unprocessed photos* does not pick them up, because nothing
they need for searching is missing.
yet: they show *No preview yet*, and the page says how many. To add them, run
**Help ▸ Add Missing Search Previews...** — for the whole
catalog, or click **Show in Lightroom** first and choose *Selected photos*. It
only adds the previews, from a small export of each photo; nothing is analyzed
again.

A preview is deleted with its photo's index data (the index cleanup, or
removing the photo from the index), and is not part of a database backup —
Expand Down
13 changes: 7 additions & 6 deletions docs/wiki/Help-Analyze-and-Index.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,12 +107,13 @@ photos with existing data* skips everything else for a photo. If one cannot be
saved, the completion dialog lists it as a warning; the photo is indexed all the
same.

Photos indexed before previews existed get theirs the next time a run processes
them: with the scope *Selected photos only*, *Current view* or *All photos in
catalog* and *Skip photos with existing data*, nothing else about them is made
again. The scope
*New or unprocessed photos* leaves them out, because nothing a search, a cull or
a face lookup needs is missing.
Photos indexed before previews existed get theirs from **Help ▸ Add Missing
Search Previews...**: choose selected photos, the current
view or the whole catalog, and it adds a preview to every indexed photo that has
none, from a small export of each photo, without analyzing anything again. (A run of Analyze & Index
that processes a photo adds one too, but *New or unprocessed photos* leaves
these photos out, because nothing a search, a cull or a face lookup needs is
missing.)

### Turning off AI metadata makes the rest of the run faster

Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/Help-Keyword-Dedup-and-Declutter.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This is a **catalog-modifying operation**. Back up your catalog before running i

### How to run it

`Library -> Plug-in Extras -> Deduplicate Keyword Synonyms...`
`Help -> Deduplicate Keyword Synonyms...`

The workflow has five steps:

Expand Down
Loading
Loading