Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions examples/dedup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,51 @@ The script uses `pooled` (`[cls || mean(patches)]`) when present and falls
back to `cls`. For large folders the pairwise pass is O(n^2) in Python; a
few thousand images is fine, hundreds of thousands will want a real index
(FAISS, sqlite-vec, ...).

Caveat: `.webp` is listed in `EXTS` but `dinov2-cli` cannot decode WebP
(`stb_image` has no WebP support). A single `.webp` file in the scanned
folder makes the CLI exit non-zero and the script aborts. Keep the input
folder to JPEG/PNG/BMP/TGA until that is fixed.

## Measured run (synthetic corpus, planted duplicates)

Corpus: 320 photos from `picsum.photos` (`/seed/<n>/400/300`, seeds 1..320,
all JPEG) plus 40 planted variants of 30 of them (every source got a JPEG
recompress at q=60; 10 sources also got an 80% resize, a 90% center crop,
or a +15% brightness shift). Picsum maps seeds onto a limited photo pool,
so the download itself already contained natural duplicates; ground truth
was therefore built from pixel-identical groups of the originals (46 groups
covering 98 files) plus the planted variants, giving 64 true duplicate
clusters and 130 true duplicate pairs across 360 files.

Command (Apple M1, Metal, `dinov2-small`):

```bash
/usr/bin/time -l python3 examples/dedup/dedup.py \
./build-release/bin/dinov2-cli models/dinov2-small/model.gguf \
/tmp/dedup-corpus/images -o report.html
```

Result: 360 images embedded in 124 s wall clock (~0.35 s/image), peak RSS
~335 MB, 64 clusters reported at the default `-t 0.95`.

Scored pair-level against ground truth (predicted pairs are all pairs
inside reported clusters):

| threshold | clusters | precision | recall | F1 |
|-----------|----------|-----------|--------|-------|
| 0.999 | 60 | 1.000 | 0.646 | 0.785 |
| 0.99 | 64 | 1.000 | 0.846 | 0.917 |
| 0.975 | 64 | 1.000 | 0.915 | 0.956 |
| 0.95 | 64 | 1.000 | 0.977 | 0.988 |
| 0.90 | 63 | 0.935 | 1.000 | 0.967 |
| 0.85 | 63 | 0.867 | 1.000 | 0.929 |

On this corpus the default `0.95` is the sweet spot: zero false merges,
and the only miss is one 90% center crop that lands at ~0.93 similarity.
Raising the cutoff toward `0.99` keeps precision perfect but drops recall
to 0.85 (crops and some resizes fall under the bar); lowering it to `0.90`
reaches perfect recall at the cost of 9 false pairs from merely similar
photos.

![sanitized sample of the HTML report](report-sample.png)
Binary file added examples/dedup/report-sample.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading