-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add evidence-bound photo duplicate audit #313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
seonghobae
wants to merge
31
commits into
fix/release-artifact-windows-namespace-v1
Choose a base branch
from
feat/photo-duplicate-quality-v1
base: fix/release-artifact-windows-namespace-v1
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
220494a
feat: add evidence-bound photo duplicate audit
seonghobae 2dab86f
feat: group exact decoded PNG pixels
seonghobae 0811ca6
fix: bind photo evidence to one stable snapshot
seonghobae fbbe2b1
test(photo): reject hard-link duplicate inflation
seonghobae e637738
fix(photo): deduplicate filesystem identities
seonghobae e6fbcab
test(photo): bound decoded PNG allocation
seonghobae 2d84088
fix(photo): bound decoded PNG memory
seonghobae e21f6fc
test(photo): reject unsupported exact-audit inputs
seonghobae 09d97f8
fix(photo): reject unsupported exact evidence
seonghobae 2281eb6
fix: keep photo audit portable and read-only
seonghobae bcaa609
test: fail closed without photo handle identity
seonghobae ceecac6
fix: bind photo hashing to opened file identity
seonghobae 95eb36d
test: exercise photo identity on Windows
seonghobae 04ae214
docs: align photo audit active-use contract
seonghobae d94057a
fix(ci): preserve main path filters and parse photo regression safely
seonghobae f03e6cf
test(photo): make decode-bound fixtures exceed the budget
seonghobae cf20a1d
test(photo): align APNG rejection contract
seonghobae 993d4fe
merge(main): adopt protected CI path filters for photo audit
seonghobae d761f75
fix(photo): fail closed when audit evidence is incomplete
seonghobae 1e6918a
test(photo): require nonzero exit for incomplete audit evidence
seonghobae 69c2a2c
fix(photo): bind bounded evidence to managed-path and raster contracts
seonghobae 55aaf73
docs(adr): keep photo audit decision proposed until integration
seonghobae c4c6369
docs(adr): mark photo audit decision proposed
seonghobae 9190c87
fix(changelog): preserve protected-main release history
seonghobae 461331f
fix(photo): align PNG metadata evidence with 0.18 API
seonghobae 59b4fe7
docs(photo): delegate product baseline to canonical owner
seonghobae 0d5ae6a
chore(stack): adopt canonical release prerequisite
seonghobae 85575d4
chore(stack): adopt exact release test repair
seonghobae 2448e06
chore: restack photo-audit owner on current release foundation
seonghobae 94a07ca
chore(stack): restack photo-audit owner on current release foundation
seonghobae 1676e1c
chore(stack): adopt current release Test foundation
seonghobae File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
101 changes: 101 additions & 0 deletions
101
docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # ADR 0012: Separate photo-duplicate and keeper evidence without composite scoring | ||
|
|
||
| - Status: Proposed | ||
| - Date: 2026-08-29 | ||
|
|
||
| ## Context | ||
|
|
||
| Pixel dimensions, bit depth, codec losslessness, edit lineage, metadata completeness, perceptual | ||
| similarity, and perceptual quality answer different questions. Combining them with undocumented | ||
| weights would turn descriptive evidence into deletion authority. Pixel count is also not a | ||
| validated substitute for spatial-frequency response or perceptual quality. A perceptual hash | ||
| distance requires a descriptor-specific, population-calibrated decision threshold; BRISQUE | ||
| requires the trained model used by the published method. | ||
|
|
||
| Photos libraries and File Provider trees have additional ownership and materialization semantics. | ||
| Reading package internals or a dataless placeholder can trigger provider activity and cannot | ||
| authorize cleanup. Encoded files and aggregate audit batches also need explicit work budgets before | ||
| any content allocation so hostile inputs cannot convert an audit into an availability failure. | ||
|
|
||
| ## Decision | ||
|
|
||
| DiskSage records byte identity with BLAKE3 and groups currently materialized PNG files only when | ||
| raster dimensions and normalized decoded RGBA16 pixels have the same domain-separated digest. | ||
| The decoder expands palette/low-depth samples and `tRNS` transparency before normalization, while | ||
| source bit depth remains separate keeper evidence. This permits semantically identical lossless | ||
| encodings with different compression or ancillary metadata to share a group without a | ||
| perceptual-distance threshold, while preventing different raster shapes or transparency semantics | ||
| from collapsing into one exact identity. | ||
|
|
||
| Before content allocation, the audit rejects symlinks, provider-managed paths, Photos library | ||
| packages, dataless objects, unsupported codecs, and encoded inputs above the fixed per-file budget. | ||
| The already-open handle is read through the same byte ceiling and filesystem-object identity, | ||
| length, and modification evidence are rechecked around hashing. Aggregate input count and declared | ||
| bytes are also bounded. Active-use evidence is collected fail-closed only by a fresh execution | ||
| preflight; read-only audit does not turn platform-specific process inspection into a discovery | ||
| prerequisite. | ||
|
|
||
| The public wire contract is `disksage.photo-duplicate-audit.v2`. Version 2 makes the evidence-state | ||
| serialization and keeper metadata semantics explicit and changes exact grouping to | ||
| `decoded-pixel-rgba16-raster-exact-v2`; consumers must not interpret it as the earlier draft v1 | ||
| shape. | ||
|
|
||
| The audit reports dimensions, source bit depth, losslessness, metadata-field count, lineage | ||
| availability, no-reference IQA availability, and perceptual-descriptor availability as separate | ||
| evidence. It does not calculate a composite score. Perceptual grouping remains unavailable until a | ||
| versioned descriptor and dataset-calibrated threshold artifact include provenance and a | ||
| cryptographic checksum. BRISQUE remains unavailable until its exact trained model artifact has | ||
| equivalent provenance and checksum. | ||
|
|
||
| Keeper evidence uses Pareto dominance only: losslessness, source bit depth, metadata completeness, | ||
| and original/edit lineage must all be no worse and at least one must be better for exactly one | ||
| member. File size, modification time, and filename have no quality authority. Ties and incomparable | ||
| members require customer selection; byte-identical members therefore never receive an arbitrary | ||
| automatic keeper. The audit may display a unique Pareto keeper but does not mutate files. Cleanup | ||
| execution and permanent deletion remain unavailable. A later execution decision must bind an exact | ||
| group identity, exact unique keeper identity, fresh approval, current inactive/materialized | ||
| evidence, reversible Trash or quarantine, and a durable journal with undo. | ||
|
|
||
| This ADR remains Proposed until the exact PR head passes the applicable Test/Release/Security/SAST | ||
| checks and all valid review findings are resolved. Merge is the acceptance boundary. | ||
|
|
||
| ## Consequences | ||
|
|
||
| Customers can establish bounded exact-duplicate evidence without intentionally hydrating Photos or | ||
| cloud-provider data. They are told why near-duplicate grouping and cleanup are unavailable and what | ||
| evidence must be installed next. This initial capability deliberately recovers no bytes by itself. | ||
| Automation receives a non-zero CLI exit when the supplied audit is incomplete while retaining the | ||
| structured JSON rejection evidence. | ||
|
|
||
| ## Rejected alternatives | ||
|
|
||
| - A hand-tuned weighted “quality score”: rejected because its weights and construct validity are | ||
| unsupported. | ||
| - A fixed perceptual-hash distance copied from examples: rejected because it is not calibrated to | ||
| the operating image population. | ||
| - Selecting the largest image automatically: rejected because dimensions alone do not establish | ||
| fidelity, originality, or perceptual quality. | ||
| - Substring matching for cloud-provider brands: rejected because it both misses canonical managed | ||
| roots and rejects ordinary local names; DiskSage reuses the filesystem/platform classifiers | ||
| owned by the cloud bounded context. | ||
| - Unbounded `read_to_end`: rejected because decode limits do not protect the preceding encoded-file | ||
| allocation. | ||
| - Direct Photos library mutation or permanent deletion: rejected because it bypasses provider and | ||
| reversible-recovery contracts. | ||
|
|
||
| ## References | ||
|
|
||
| Camera & Imaging Products Association. (2026). *Exchangeable image file format for digital still | ||
| camera: Exif Version 3.1 (CIPA DC-008-Translation-2026)*. | ||
| https://www.cipa.jp/e/std/std-sec.html | ||
|
|
||
| International Organization for Standardization. (2024). *Photography—Electronic still picture | ||
| imaging—Resolution and spatial frequency responses (ISO 12233:2024)*. | ||
| https://www.iso.org/standard/88626.html | ||
|
|
||
| Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012). No-reference image quality assessment in the | ||
| spatial domain. *IEEE Transactions on Image Processing, 21*(12), 4695–4708. | ||
| https://doi.org/10.1109/TIP.2012.2214050 | ||
|
|
||
| Zauner, C. (2010). *Implementation and benchmarking of perceptual image hash functions* [Master's | ||
| thesis, University of Applied Sciences Upper Austria]. https://www.phash.org/docs/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| use std::path::PathBuf; | ||
|
|
||
| fn main() { | ||
| let paths: Vec<PathBuf> = std::env::args_os().skip(1).map(PathBuf::from).collect(); | ||
| if paths.is_empty() { | ||
| eprintln!("다음 단계: 내보낸 로컬 PNG 파일 경로를 하나 이상 지정하세요. Photos 보관함과 클라우드 전용 파일은 열지 않습니다."); | ||
| std::process::exit(2); | ||
| } | ||
| let now = std::time::SystemTime::now() | ||
| .duration_since(std::time::UNIX_EPOCH) | ||
| .map(|value| value.as_millis() as u64) | ||
| .unwrap_or_default(); | ||
| let audit = disksage_lib::photo_duplicate::audit_photos(&paths, now); | ||
| println!("{}", serde_json::to_string_pretty(&audit).unwrap()); | ||
| if !audit.evidence_complete || audit.inspected_input_count == 0 { | ||
| std::process::exit(3); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.