feat: individual pet recognition (Phase 2) - #843
Open
Deeds67 wants to merge 65 commits into
Open
Conversation
github-actions
Bot
added
the
documentation
Deeds67
added
rc-ml
Deeds67
force-pushed
the
feat/pet-recognition
branch
from
July 29, 2026 19:24
4c1f702 to
09d1760
Compare
🧪 Release candidate buildLatest RC Images published
How to run this RCIn the directory containing your services:
immich-server:
image: ghcr.io/open-noodle/gallery-server:pr-843-rc.4
immich-machine-learning:
image: ghcr.io/open-noodle/gallery-ml:pr-843-rc.4Then pull and restart: docker compose pull immich-server immich-machine-learning
docker compose up -dEach push publishes a new numbered tag, so update the image line to move to a newer RC. To roll back, point it at an earlier Previous builds (4)
Last updated Thu, 30 Jul 2026 08:49:01 GMT — every push while the |
Adds src/petid/metrics.py with verification_scores (all-pairs cosine ROC-AUC + EER), identification (leave-one-out top-1/mAP), and clustering_quality (agglomerative clustering, homogeneity/completeness). identification() excludes the self-index from the ranked candidate list rather than only deprioritizing it via -inf: leaving self in with -inf similarity still counted it as a "relevant" hit at the worst possible rank, capping mAP at ~0.83 on perfectly-separable data instead of ~1.0.
Adds petid.evaluate: embed_records() runs a checkpoint's embedder over records, evaluate() scores per-species (dog/cat) test-split metrics plus the DogFaceNet eval_only set and writes a markdown report, and a python -m petid.evaluate CLI. Also fixes PetEmbedder(pretrained=False)'s Dinov2Config to use image_size=518 (matching facebook/dinov2-small) instead of 224. The zeroshot/head/full checkpoints are always produced from the pretrained backbone (1370 position-embedding rows), so loading them into the non-pretrained eval embedder (previously 257 rows) hard-failed load_state_dict with a size mismatch. DINOv2 interpolates position encodings at runtime regardless of the configured image_size, so actual 224x224 inputs are unaffected.
… plan Slice 2 registered VectorIndex.Pet but left the reindex/drop call sites hardcoded to Clip/Face, since they live in service code outside that slice. Recording it in slice 6 so the pet index cannot silently rot.
Adds machineLearning.petRecognition (disabled by default, model pet-recognition-base, maxDistance 0.55, minFaces 1), the petRecognition queue, and its two jobs with real @onjob handlers that currently gate off — clustering lands in slice 5, reprocess/nightly in slice 6. The handlers ship now rather than later because JobRepository.setup() refuses to boot if any JobName lacks a handler. maxDistance 0.55 / minFaces 1 come from the phase-1 threshold sweep: cats need a looser threshold than dogs, and a low minFaces is what lets a pet photographed once still surface as its own individual. Two exhaustive Record<QueueName, …> sites beyond the plan's list needed the new member (queue-legacy.dto.ts, an ML repository fixture) — tsc caught both. Includes the regenerated TypeScript SDK, which the web slice needs to typecheck against the new config. 5143 server tests green, tsc and eslint clean.
Three repository primitives the pipeline needs: - detectPets takes an optional recognition model and, when given one, asks for DETECTION + RECOGNITION in a single /predict the way detectFaces does. Without it the request is byte-identical to today's, which is the recognition-disabled path and has its own regression test. - refreshPetFaces writes asset_face rows and their embeddings, returning the generated ids so the caller can queue one recognition job per face. - searchPets is searchFaces scoped to pet_search, minus minBirthDate. A test inserts a human face and asserts searchPets never returns it, which is what makes the separate-table decision hold. refreshPetFaces uses a transaction with two sequential inserts rather than refreshFaces' single CTE: that pattern only works because the face caller pre-generates ids client-side, whereas asset_face.id is DB-generated here, so the embedding rows genuinely depend on the first insert's output. Rows are paired positionally and a multi-face test proves the ordering. 5146 unit tests and 12 medium tests green; tsc and eslint clean.
Adds the petRecognition accordion (enable, model, maxDistance, minFaces) and wires the new queue through every admin surface. Three of those maps are Record<QueueName, …> and so exhaustive by type — check:typescript is the real gate here — but ADMIN_VISIBLE_QUEUES is a plain array, and omitting it would have silently hidden the queue from the UI. Forcing pet recognition deletes all pet people and re-detects, so it gets the same confirmation prompt pet detection has. DetailPanelPeople now badges pets, closing the gap where a pet in a photo read as a person. Guards included for older payloads with no `type` and for a null species (which must not render title="null"). 3826 web tests green, check:typescript clean. check:svelte reports 0 files locally, a known repo anomaly, so it is verified on CI rather than counted here.
handlePetDetection now requests embeddings when recognition is enabled, writes faces and embeddings together, and queues one recognition job per pet instead of bucketing by species. With recognition disabled it is byte-for-byte the old code path, guarded by an explicit regression test — that is what lets this ship without a destructive migration for existing users. handlePetRecognition mirrors handleRecognizeFaces: nearest-neighbour search, assign-or-create, a single deferred retry for non-core faces, and a face_identity link with type 'pet'. That link matters beyond bookkeeping — it activates the shared-space pet propagation path that until now was dead code, because nothing ever set identityId for a pet. Two things the plan got wrong, fixed here: - The plan used one search sized to minFaces for both the core test and the person lookup. At the shipped default (minFaces 1, so numResults 1) a face's own row is always the nearest match at distance 0, so the window is always itself and nothing could ever cluster — every pet would spawn its own person forever. Restored the second hasPerson-scoped search the face pipeline uses for exactly this reason. The medium test (two near-identical embeddings -> one person) only passes with it. - Species had nowhere to live: asset_face has no species column. The detected label now rides the job payload and is read only when creating a new person; every other path takes species from the existing person row. 5161 unit tests green, 3 new medium tests against a real DB, and the neighbouring face/shared-space medium suites re-run clean. tsc, eslint and prettier clean.
handleQueuePetRecognition is now real: - force purges pet people, their shared-space copies and every pet_search row, then requeues detection so assets are re-embedded with the current model. This is the path an admin takes after enabling recognition or switching model. A medium test pins its blast radius: in the same database a human person, their asset_face, their face_search embedding and their space copy all survive the purge untouched. - nightly skips when the recorded lastRun is newer than the newest pet face, mirroring facial recognition, and is gated by nightlyTasks.clusterNewPets (default on). - both paths record lastRun AND the model used, so a later change can detect a model switch rather than guessing. Also closes a gap slice 2 left open: VectorIndex.Pet was registered but the maintenance call sites still hardcoded Clip/Face, so pet_index would never be rebuilt, prewarmed, or replaced after a vector-extension switch. It is now wired into both, with a parity test asserting every VectorIndex member is covered — so the next index cannot be added without maintenance. 5172 unit tests and 5 medium tests green; neighbouring medium suites re-run clean; tsc, eslint and prettier clean.
Regenerates the Dart client (first pass for this feature) and picks up the clusterNewPets drift in the spec and TS SDK. Adds e2e coverage mirroring the pet-detection specs: config defaults, enable round-trip, validation bounds on maxDistance and minFaces, and the petRecognition queue appearing in GET /jobs and accepting commands. Error strings were taken from the running Zod schema rather than guessed. The specs typecheck but were not executed locally — they need the docker stack, so CI runs them. Also two whitespace-only prettier fixes on test files from earlier slices.
…ry methods The @GenerateSql decorators on the pet repository methods make sql-tools emit documented queries, and those generated files are a CI gate — SQL Schema Checks regenerates them and fails if the committed copies differ. Generated against a real database (running the task without one deletes every query file).
handlePetRecognition clusters faces with a read-then-create sequence (searchPets -> no match -> personRepository.create -> reassignFaces), exactly like handleRecognizeFaces. Nothing guards the gap between the search and the create, so with concurrency > 1 two faces of the same pet both search before either commits, both find no owning person and both create one -- leaving one pet split across duplicate people that only a manual merge can fix. The window is wider here than for faces: petRecognition ships minFaces: 1, so isCore is true on the very first face and a person is created immediately, where facialRecognition's minFaces: 3 requires three neighbours first. Clustering is also single-pass and accumulative -- face N+1 must see what face N assigned -- so running it in parallel makes results non-deterministic, not just duplicated. Drop PetRecognition from ConcurrentQueueName (and the duplicated isConcurrentQueue list, which is not derived from the type), and from SystemConfigJobSchema so the admin UI stops offering the slider. JobSettings.svelte needs no change: it already falls through to the disabled "This job is not concurrency-safe." field for queues absent from SystemConfigJobDto, which is how facial recognition renders. updateConcurrency hard-codes 1 for non-concurrent queues, so this also forces 1 on any instance that already stored a higher value.
The detector is YOLO11, a general COCO model that emits all ten COCO animal classes, but the re-ID model is a DINOv2 backbone trained on dog and cat identities only. Embedding a bird or a horse with it produces arbitrary neighbours, so those species were being clustered into individual "pets" on no meaningful signal. Measured on a real library: of 512 pet people, 278 (54%) were species the model was never trained on -- 168 birds, 45 horses, 23 cows, 19 bears, 10 elephants, 8 sheep, 3 zebras, 2 giraffes. It also amplified detector misfires. 58 of 2911 pet detections (2.0%, a lower bound -- it only counts boxes containing a *detected* human face) landed on a person, and 46 of those 58 were bird/horse/elephant. Before individual recognition a misfire quietly joined a shared "Bird" bucket; afterwards each one became its own named-able identity on the People page. One observed cluster held 33 faces, 9 of them human. So route only dog/cat to writeDetectedPetsForRecognition and send the rest down the species-bucket path they used before recognition existed. A misfire now costs one shared bucket again rather than an identity per hit, and dog/cat clustering is unaffected. Both writers are additive (refreshPetFaces only inserts, and no-ops on an empty list), so running both on one asset is order-independent. The bucket writer is only invoked when it has work, keeping the existing call shape -- and the 5.1 "byte-for-byte identical when recognition is off" guard -- intact. Existing pet people are not migrated; a forced Pet Recognition re-run purges and rebuilds them under the new routing.
Recognition-written faces that were never clustered have personId: null, so deleteAllPets()'s person-scoped delete could not see them — and the force purge's subsequent deleteAllPetSearch() truncate destroyed the pet_search row that was their only identity, leaving permanent orphan asset_face rows. Delete faces joined to pet_search first, inside the same transaction and before the truncate. Both force paths (recognition and detection) get the fix for free. Also scopes the medium spec's whole-DB face_search/pet_search count assertions by faceId — the medium database is shared across the file, so those counts broke as soon as another test seeded a face.
…paths Pet faces share sourceType 'machine-learning' with human faces, so every queue-level human operation reached them: a force recognition reset unassigned pet faces and unlinked their identities (person cleanup then deleted the now face-less pet people, losing names), the untyped shared-space wipe deleted every space pet copy, a force detection reset hard-deleted all pet faces and their embeddings, and both recognition fan-outs queued human FacialRecognition jobs over pet faces that can only ever fail. Adds petFacePredicate to src/utils/database.ts — a pet face has a pet_search row or is assigned to a type='pet' person — and threads excludePetFaces / excludePets options through unassignFaces, deleteFaces, getAllFaces, unlinkFacesBySourceType, deleteAllPersonFaces and deleteAllPersons. Only the human call sites opt in; pet-side paths are pet-scoped by construction. handlePersonCleanup deliberately stays generic: pet people now keep their faces through a human reset, and a pet person with genuinely zero faces should still be collected.
Pet faces are written without an explicit sourceType, so they take the same machine-learning default as human faces and every one of them landed in handleDetectFaces' mlFaceIds set. Two things followed: a pet face no detection box matched was hard-deleted as a stale ML face (taking its pet_search row with it), and where boxes did overlap, a second overlapping detection wrote a human face_search embedding straight onto the pet face. getForDetectFacesJob now selects a computed isPet column (petFacePredicate) via a local jsonArrayFrom — withFaces is left alone for its read-only callers — and handleDetectFaces skips pet faces both when collecting removal candidates and when scanning IoU match candidates.
…atch fallback - Resize crops with INTER_AREA on downscale and INTER_LINEAR on upscale, the closest match to the antialiased PIL bilinear used in training and eval; the previous default-interpolation resize was a train/serve skew. - Skip pets whose post-clamp crop is under 2px on either side: they now come back without an embedding key rather than carrying a garbage one. RecognizedPet's embedding becomes NotRequired to model that. - Mirror FaceRecognizer's execution-provider batch handling — the _batch_size_default property (MIGraphX/OpenVINO force 1), the settings.max_batch_size consumption and the chunking loop — plus a pet_recognition slot in MaxBatchSize. - Pair crops to embeddings with strict=True over the embeddable subset, so a row-count mismatch from a broken model raises instead of silently dropping pets. Zipping pets directly would have raised on the normal degenerate-skip path, so the strictness is applied where it actually guards a bug. - Clamp detector output boxes to the image bounds; they go over the wire straight into asset_face. - Pass the base class's ignore_patterns in both pet _download overrides.
… embedding-less pets refreshPetFaces paired embeddingsToAdd[i] with the i-th INSERT … RETURNING row, which is only correct while postgres happens to return insert order — nothing guarantees it, and a mispairing silently attaches one pet's embedding to another pet's face. Callers now pre-generate face ids and pair by explicit faceId, the way the human refreshFaces already does. The method throws on a count mismatch or an embedding naming a face it is not inserting, before opening the transaction. pet_search gains a nullable species column (migration 1785200000000, also registered in revert-to-immich.sql) written at embed time. The queue-all and nightly recognition paths carry no label in their job data, so they were creating pet people with species: null; person creation now falls back to the stored species. A recognizable pet that arrives without an embedding is routed to the species bucket rather than written as an unassigned face with no pet_search row. Such a face matched neither arm of petFacePredicate, so the human pipeline could not see it as a pet and would destroy it — this is what makes the predicate's coverage claim actually hold. Query docs regenerated (this also picks up slice 3's isPet column, which needs a server build before mise run sql sees it).
…nd scoped reprocess The model switch lifecycle existed only as a comment: state.modelName was written but never read back, there was no config hook, no per-job guard, and no validation, so a model change silently mixed two embedding spaces in pet_search and a typo'd model name broke pet detection at runtime (one /predict carries both tasks). - onConfigValidate rejects any model outside PET_RECOGNITION_MODEL_NAMES. - ConfigInit and ConfigUpdate both route to handleModelSwitch under a new advisory lock (DatabaseLock.PetRecognitionModelSwitch = 860). withLock serializes but does not dedupe, so the first thing under the lock is a re-read of the stored state: two deliveries of the same ConfigUpdate each carry the same stale oldConfig and would otherwise purge and requeue twice, double-detecting every asset. - The switch purge is SCOPED — it deletes embeddings and the individuals recognition created from them, and leaves species buckets alone. Buckets are detector output, are not model-coupled, and in every requeue-skipped configuration would have had no rebuild path. The admin Reset button keeps its full-purge semantics, where the requeue does rebuild them. - Requeue is gated on recognition AND detection both being enabled; recognition on with detection off stamps pendingReprocess and defers the force run to detection being re-enabled, because a non-force run skips petsDetectedAt assets and would rebuild nothing. - The drift check in the non-force queue-all runs before the nightly date-skip, so an idle library cannot mask an offline model switch forever. - handlePetDetection re-reads the config after the ML call and skips the write if the model changed mid-flight, mirroring the CLIP guard.
…lations - The pet-recognition accordion now warns when pet detection is disabled: recognition still applies to already-detected pets, but nothing new gets scanned. Soft dependency only — the server deliberately allows this. - Changing the model prompts for confirmation before the page saves, via SystemConfigButtonRow's onBeforeSave hook (there is no per-accordion save). The model select is disabled while recognition is off, so this covers the recognition-on switch paths; recognition-off switches arrive only via API or config file and are handled server-side. - The pet-recognition reset dialog told half the truth. It now says the purge always happens and that reprocessing only runs while detection is enabled. - The pet-recognition queue card gets mdiPawOutline so it is distinguishable from pet detection's mdiPaw, mirroring the face pair. - The paw badge in the asset viewer and on person tiles gets role="img", an aria-label and a translated species tooltip instead of the raw enum value, falling back to the raw value for species with no key.
…tring to $t svelte-check in CI rejected $t(getPetSpeciesI18nKey(species)): $t takes a typed Translations key, not an arbitrary string, and the helper's raw-value fallback made its return type string. Local check:svelte scans 0 files (it needs --workspace to resolve anything, and --no-tsconfig zeroes it out entirely), so this only surfaced on push. getPetSpeciesI18nKey now returns Translations | undefined for known species only, and a new getPetSpeciesLabel does the raw-value substitution at the call site where it belongs. Same rendered output: translated label for a mapped species, raw value for anything else.
…on gate The e2e stack has no ML service, so these cover config, queues and the force purge rather than the detect->embed->cluster flow (that seam is covered by the medium tests). - createPetWithEmbedding seeds a pet person with an assigned face and a pet_search row. The raw pg client has no notion of the vector column type, so the 512-d literal needs an explicit ::vector cast in the query text. - R9.9 pauses the petDetection queue, force-starts petRecognition, and asserts the pet people are gone and the requeued detection job is counted. A paused queue parks the job under paused rather than waiting, so the assertion sums both. - R9.10 pins that starting the queue with recognition disabled is a no-op even under force — the HTTP layer has no enabled guard, so this genuinely exercises the handler's check rather than being rejected by validation. - R9.11 replaces a vacuous assertion that never checked anything: the asset people list is now asserted to actually contain a type: 'pet' entry. Not executed locally — the e2e stack was not running and building the server image against another slice's in-progress tree would have been unrepresentative. CI runs the e2e API suite.
…y date compare - The non-force queue-all now skips when the PetRecognition queue already has pending work and prewarms the pet vector index before fanning out, matching the facial-recognition sibling. Precedence is enabled check -> drift check -> nightly date-skip -> pending-work skip -> prewarm -> fan-out: the pending-work skip must never gate the drift check, or a busy library would never notice an offline model switch. - An already-assigned pet face now queues shared-space face matching before skipping. The space may have been created after the face was first recognized, the same reason the human sibling does it. - getLatestPetDate returns a Date instead of ::text. The nightly guard compared an ISO-T string against pg's space-separated text, where 'T' > ' ' made any same-day lastRun look newer and skipped the run. Upstream's getLatestFaceDate keeps the identical quirk deliberately, for rebase hygiene. - detectPets defaults a missing pet-detection response key to [], instead of letting the caller's pets.filter throw. Closes the enumerated unit-test debt: pending-work/prewarm/space-match parity tests, eight handlePetRecognition pins (hasPerson fallback, numResults boundary, inclusive minFaces boundary, deferred-then-core, deferred-then-fallback, the no-person exit, null asset, spaceId dedupe), detection failure and case-insensitive species routing, the isRecognizablePetSpecies truth table, and the two machine-learning response-parsing cases. Nightly tests now key the system-metadata mock by argument rather than one shared blob.
…lustering seam The load-bearing recognition SQL had only ever run against mocks. - R9.1 getUnassignedPetFaces against a real database: returns embedded and unassigned faces, excludes assigned, soft-deleted, invisible and embedding-less ones — all four predicates asserted separately. - R9.2 getLatestPetDate returns a Date, and a lastRun earlier the same day no longer looks newer than petsDetectedAt (the F11 regression). - R9.3 searchPets' maxDistance boundary is inclusive: read a row's exact distance with a loose search, re-query with maxDistance equal to it, row still returned. - R9.4 a wrong-dimension embedding rejects and rolls the face insert back with it, leaving no orphan asset_face row. - R9.5 getPetFaceForRecognition excludes soft-deleted faces. - R9.7 minFaces: 2 end-to-end, which un-deads an override every other test leaves at the shipped default of 1: face A defers, face B becomes core and creates the person, and A's deferred run rejoins it through the hasPerson fallback rather than creating a second person. - R9.8 the detect -> embed -> cluster seam against a real database, with only the ML repository mocked: detection's queued recognition jobs are replayed through the real handler and must produce one pet person carrying its species and a pet-typed face identity. e2e cannot reach this seam (no ML service in that stack) and the unit tests stub the repository, so this is the only place the two handlers meet for real.
…rdown Vitest intermittently failed the whole web suite with an unhandled "ReferenceError: document is not defined" thrown from bits-ui's body-scroll-lock.svelte.js and attributed to SpaceEditModal.spec.ts — every test passing, the run still red. This is pre-existing on main (824e93a on 07-26, 24a83ca on 07-28) and unrelated to pet recognition; it just happened to land on this PR's first green-path run. bits-ui does not release the body style when the locking component unmounts. It defers the reset to a ~24ms window.setTimeout so a modal that closes and reopens in the same tick keeps its styles. A spec whose last render is a modal therefore finishes with that timer still pending, and when vitest tears the happy-dom environment down first the callback dereferences a document that no longer exists. It is a race, which is why it only surfaces on loaded CI runners. Wait the pending reset out in a global afterAll, while the DOM is still alive. The body carries `overflow: hidden` for exactly as long as a lock is outstanding and the deferred reset clears it, so a spec that never opens a modal pays nothing; the iteration cap stops a spec that sets that style for its own reasons from stalling the file.
…install End-to-End Lint died in 27s with [//:sdk:install] ENOENT: rename '.pnpm/lock.yaml.1741090523' -> '.pnpm/lock.yaml' [//:sdk:build] Command was killed with SIGTERM: pnpm install Callers list the two tasks as parallel `depends` (e2e's ci-unit and ci-setup), and `pnpm build` runs pnpm's own deps-status check, which shells out to `pnpm install`. So two installs ran at once against the same store and raced on node_modules/.pnpm/lock.yaml — one renamed the temp lockfile out from under the other, the loser took a SIGTERM. Giving sdk:build an explicit dependency on sdk:install makes mise order them instead of racing, everywhere rather than just in the e2e tasks. The duplicate sibling edge in those callers is deduplicated, so this adds no extra work. Verified with `mise ci-unit` in e2e/: sdk:install now completes before sdk:build starts, and no nested install is triggered.
…side effect
e2e's `check` type-checks src/specs/server/api/oauth.e2e-spec.ts, which imports
@immich/e2e-auth-server, so tsc follows it into packages/e2e-auth-server and
needs that package's own deps. `--filter immich-e2e` installs only immich-e2e,
so those were never installed by this task:
../packages/e2e-auth-server/auth-server.ts(7,8): error TS2307:
Cannot find module 'jose' or its corresponding type declarations.
../packages/e2e-auth-server/auth-server.ts(8,22): error TS2307:
Cannot find module 'oidc-provider' or its corresponding type declarations.
They resolved anyway because the unserialised `sdk:build` in the previous commit's
race ran pnpm's deps-status check over "all 12 workspace projects" — a full
workspace install that happened to cover packages/e2e-auth-server. When that
install lost the race the job died before reaching `check`; when it won, `check`
passed on borrowed deps. Serialising sdk:build removed the accident and left the
real gap visible.
The `...` suffix scopes the install to immich-e2e plus its workspace
dependencies (packages/sdk, packages/e2e-auth-server, packages/cli) — 4 of 12
projects — so the task installs what it type-checks.
Verified by deleting packages/e2e-auth-server/node_modules to reproduce CI's
state: `mise check` failed with the errors above, and `mise ci-unit` passes
end-to-end with this change.
Upstream v3.1.0 bumps eslint-plugin-unicorn 64 -> 72 and reorders tailwind
utility classes. Fork-only pet files were not in upstream's sweep, so they
now report:
- unicorn/consistent-conditional-object-spread (3x, server) — the
`...(cond ? { k } : {})` form becomes `...(cond && { k })`; spreading
`false`/`undefined` is a no-op, so behaviour is unchanged.
- unicorn/no-unnecessary-boolean-comparison (web) — `x === false` -> `!x`.
- better-tailwindcss/enforce-consistent-class-order (web) — `bottom-1 right-1`
-> `right-1 bottom-1` on the pet badge, matching the sibling in
person-tile.svelte that the rebase already reordered.
All four are eslint --fix output, applied per-file so upstream's own
pre-existing warnings are left alone.
Deeds67
force-pushed
the
feat/pet-recognition
branch
from
July 30, 2026 08:27
030e0a4 to
f7ba162
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Individual pet recognition: the "which pet is this" layer on top of the existing pet detector. Detected pets get an embedding, embeddings cluster into named individuals, and those individuals show up on the People page like any other person.
Phase 1 (the training spike) and Phase 2 §4.1 (model production) are already on this branch; this PR completes §4.2–§4.7 — the ML embedder, storage, server pipeline, and admin UI.
The models
Three selectable models, published to Hugging Face under
open-noodle, all emitting a uniform 512-d L2-normalized embedding so a model switch never needs a schema change:pet-recognition-smallpet-recognition-base(default)pet-recognition-largeA frozen DINOv2 backbone plus a trained linear projection — the projection's normalized output is the embedding. Fine-tuning the backbone was tried and rejected: it overfits the training identities and forgets DINOv2's general features. Trained only on CC0 (Dogs-World) and CC BY (Cat Individual Images) data, on an Apache-2.0 backbone, so the lineup is commercially clean.
Scored on the complete test splits — 16,469 held-out dog identities and 102 cat identities, not a sample. Phase 1's dog numbers came from a capped ~600-identity subset and were ~1.5× optimistic; the honest full-split figures are the ones above, and the Phase-1 results doc now says so at the top.
It is off by default, and that is the point
petRecognition.enableddefaults to false. While it is off, pet detection behaves exactly as it does today — per-species "Dog"/"Cat" bucket people, no embeddings, no recognition jobs. Upgrading users see no change and lose nothing. There is an explicit regression test guarding that path.Switching to individual pets only happens when an admin enables recognition, and that is a deliberate, purging reprocess: pet people and
pet_searchare cleared, then assets are re-detected and re-embedded with the current model. A medium test pins the blast radius — in the same database, human people, their faces, theirface_searchembeddings and their shared-space copies all survive untouched.How it works
Deliberately isolated from human faces: a separate
pet_searchtable and its ownsearchPetsquery, because the DINOv2-projection space is incompatible with ArcFace faces and the clustering NN is always single-type. A test inserts a human face and assertssearchPetsnever returns it.Assigning a pet face also writes a
face_identityrow withtype: 'pet'— which activates the shared-space pet propagation path that until now was dead code, because nothing ever setidentityIdfor a pet.Two bugs the tests caught before they shipped
minFacesfor both the "is this a core cluster" test and the "does a matching pet already exist" lookup. At the shipped default (minFaces: 1, sonumResults: 1) a face's own row is always the nearest match at distance 0 — so the window is always itself, and every pet would have spawned its own person forever. Fixed with the secondhasPerson-scoped search the face pipeline uses for exactly this reason; the medium test (two near-identical embeddings → one person) only passes with it.projection(normalize(pooler)).Defaults, and where they come from
maxDistance: 0.55andminFaces: 1are measured, not guessed: a clustering-threshold sweep over the full splits showed cats need a looser threshold than dogs (completeness 0.88 at 0.40 vs 0.93 at 0.50), and a lowminFacesis what lets a pet photographed once still surface as its own individual. Both are admin-tunable.Testing
pet_search,searchPets, clustering, and the purgecheck:typescriptclean, which is the real gate since three admin maps are exhaustiveRecord<QueueName, …>ruff+mypy --strictcleanAlso fixes ML test collection on this branch: bare
uv run pytest(what CI runs) recursed into the committedpet-recognition-training/subproject, which has its own venv, and died with 11 collection errors.Not in this PR
withSharedSpacespath is a fast-follow.Design:
docs/superpowers/specs/2026-07-24-pet-recognition-phase2-design.mdSlices + test plan:
docs/superpowers/specs/2026-07-25-pet-recognition-phase2-implementation-slices.mdModel results:
docs/superpowers/plans/2026-07-24-pet-recognition-phase2-model-production-RESULTS.md