From 220494a99b7c61dbe6521732a508df8597d137c4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 30 Aug 2026 04:04:50 +0900 Subject: [PATCH 01/25] feat: add evidence-bound photo duplicate audit --- CHANGELOG.md | 4 + ...cate-evidence-without-composite-scoring.md | 71 +++++ docs/architecture/adr/README.md | 1 + docs/product-technical-gap-baseline.md | 2 +- src-tauri/Cargo.toml | 4 + .../src/bin/disksage-photo-duplicate-audit.rs | 15 + src-tauri/src/cloud.rs | 4 +- src-tauri/src/lib.rs | 1 + src-tauri/src/photo_duplicate.rs | 296 ++++++++++++++++++ 9 files changed, 395 insertions(+), 3 deletions(-) create mode 100644 docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md create mode 100644 src-tauri/src/bin/disksage-photo-duplicate-audit.rs create mode 100644 src-tauri/src/photo_duplicate.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index 3bf76f051..998b62c66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and ### Changed +- Add a read-only exact-photo duplicate audit that groups current materialized PNG bytes with + BLAKE3 while rejecting provider paths, Photos libraries, placeholders, symlinks, active files, + and replacement races; perceptual grouping, keeper selection, and cleanup remain explicitly + unavailable until calibrated, checksummed evidence can support them without an invented score. - Keep coverage builds compile-safe by applying the same `not(coverage)` boundary to native-copy identity cleanup and dependent eviction helpers; the focused authority contract remains green. - Add durable private failure records in a separate journal directory and a receipt-bound diff --git a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md new file mode 100644 index 000000000..e9ebdd3c5 --- /dev/null +++ b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md @@ -0,0 +1,71 @@ +# ADR 0012: Separate photo-duplicate and keeper evidence without composite scoring + +- Status: Accepted +- Date: 2026-08-30 + +## Context + +Pixel dimensions, bit depth, codec losslessness, edit lineage, metadata completeness, perceptual +similarity, and perceptual quality answer different questions. Combining them with undocumented +weights would turn descriptive evidence into deletion authority. Pixel count is also not a +validated substitute for spatial-frequency response or perceptual quality. A perceptual hash +distance requires a descriptor-specific, population-calibrated decision threshold; BRISQUE +requires the trained model used by the published method. + +Photos libraries and File Provider trees have additional ownership and materialization semantics. +Reading a package internals or a dataless placeholder can trigger provider activity and cannot +authorize cleanup. + +## Decision + +DiskSage v1 groups only byte-identical, currently materialized PNG files using BLAKE3. Before and +after hashing, it verifies the filesystem-object identity and size, and it rejects symlinks, +provider-managed paths, Photos library packages, dataless objects, unsupported codecs, and files +whose active-use evidence is incomplete or positive. + +The audit reports dimensions, bit depth, losslessness, metadata-field count, lineage availability, +no-reference IQA availability, and perceptual-descriptor availability as separate evidence. It +does not calculate a composite score. Perceptual grouping remains unavailable until a versioned +descriptor and dataset-calibrated threshold artifact include provenance and a cryptographic +checksum. BRISQUE remains unavailable until its exact trained model artifact has equivalent +provenance and checksum. + +Byte-identical members provide no evidence-based unique quality keeper. Consequently v1 neither +selects a keeper nor mutates files. Cleanup execution and permanent deletion remain unavailable. +A later execution decision must bind an exact group identity, exact unique keeper identity, fresh +approval, current inactive/materialized evidence, reversible Trash or quarantine, and a durable +journal with undo. + +## Consequences + +Customers can establish exact duplicate evidence without risking Photos or cloud-provider data. +They are told why near-duplicate grouping and cleanup are unavailable and what evidence must be +installed next. This initial capability deliberately recovers no bytes by itself. + +## Rejected alternatives + +- A hand-tuned weighted “quality score”: rejected because its weights and construct validity are + unsupported. +- A fixed perceptual-hash distance copied from examples: rejected because it is not calibrated to + the operating image population. +- Selecting the largest image automatically: rejected because dimensions alone do not establish + fidelity, originality, or perceptual quality. +- Direct Photos library mutation or permanent deletion: rejected because it bypasses provider and + reversible-recovery contracts. + +## References + +Camera & Imaging Products Association. (2026). *Exchangeable image file format for digital still +camera: Exif Version 3.1 (CIPA DC-008-Translation-2026)*. +https://www.cipa.jp/e/std/std-sec.html + +International Organization for Standardization. (2024). *Photography—Electronic still picture +imaging—Resolution and spatial frequency responses (ISO 12233:2024)*. +https://www.iso.org/standard/88626.html + +Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012). No-reference image quality assessment in the +spatial domain. *IEEE Transactions on Image Processing, 21*(12), 4695–4708. +https://doi.org/10.1109/TIP.2012.2214050 + +Zauner, C. (2010). *Implementation and benchmarking of perceptual image hash functions* [Master's +thesis, University of Applied Sciences Upper Austria]. https://www.phash.org/docs/ diff --git a/docs/architecture/adr/README.md b/docs/architecture/adr/README.md index e6735fc57..50849f886 100644 --- a/docs/architecture/adr/README.md +++ b/docs/architecture/adr/README.md @@ -17,6 +17,7 @@ new numbered record rather than rewriting history. | [0009](0009-path-free-lineage-relation-graph.md) | Export a path-free lineage relation graph | Accepted | | [0010](0010-rooted-organize-destinations.md) | Require rooted, process-independent organize destinations | Accepted | | [0011](0011-cloud-transfer-failure-and-materialization.md) | Durable failed-copy evidence and placeholder-safe adoption | Accepted | +| [0012](0012-photo-duplicate-evidence-without-composite-scoring.md) | Separate photo-duplicate and keeper evidence without composite scoring | Accepted | New records must state context, decision, consequences, rejected alternatives, and the evidence or standard that led to the decision. A record never grants cloud-write or source-eviction authority; diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 5ecd46866..73b9957ed 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -23,7 +23,7 @@ authoritative, and no merge is claimed from queued or stale status. | P0 | A long Finder/provider copy can appear hung and consume the remaining local headroom. | The `real_datasets` Finder copy remained at “준비 중” for hours; the latest bounded iCloud dump retained 125 no-progress fetch/create markers, a 95.24% upload, and a zero-progress 1.06GB download while scheduling was `running`. Bounded `/bin/cp`/`mkdir` and global probes use private process groups and headroom gates. | Preview shows required bytes + staging reserve; timeout cleans only the child-created destination and leaves a durable receipt. | | P1 | Personal desktop-client capacity is not the same as API quota; OAuth is unnecessarily implied for a single-user installation. | ADR-0001 permits copy-only desktop-client mode marked `capacity-unverified`; the cloud connection UI defaults to read-only OAuth consent and requires an explicit write-access opt-in. | Settings clearly distinguish local desktop client, API quota, and organization OAuth; no OAuth prompt is required for the local-only path. | | P1 | Users cannot yet see a full lineage graph connecting source, metadata, archive member, provider item, receipt, Goal, and eviction decision. | The candidate UI now exposes a compact source→metadata→archive→provider lineage panel using the stable fingerprint, confidence, and blocker state; provider item/receipt/permit remain explicitly pending until their evidence exists. | Export and UI show stable content IDs, provenance edges, confidence, and blockers without exposing raw private paths. | -| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; current default path remains fail-closed. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. | +| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; the photo audit now has a fail-closed exact-byte PNG foundation, but calibrated perceptual grouping, a provenance-bound IQA artifact, unique keeper selection, and reversible execution remain product gaps. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. Photo cleanup additionally requires a dataset-calibrated descriptor threshold and separately presented quality, lineage, and metadata evidence without a composite score. | | P2 | Cross-platform behavior and accessibility are not presented as one release contract. | macOS/Linux/Windows release checks exist; several UI accessibility PRs remain open. | Release notes and UI expose platform capability matrix, keyboard/assistive labels, and bounded failure messages for each action. | ## Technical and operational gaps diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index 15cc5b18c..6bc4f66d3 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -109,6 +109,10 @@ path = "src/bin/disksage-cache-cleanup.rs" name = "disksage-dev-artifacts" path = "src/bin/disksage-dev-artifacts.rs" +[[bin]] +name = "disksage-photo-duplicate-audit" +path = "src/bin/disksage-photo-duplicate-audit.rs" + [[bin]] name = "disksage-provider-client-runtime" path = "src/bin/disksage-provider-client-runtime.rs" diff --git a/src-tauri/src/bin/disksage-photo-duplicate-audit.rs b/src-tauri/src/bin/disksage-photo-duplicate-audit.rs new file mode 100644 index 000000000..b160c9e95 --- /dev/null +++ b/src-tauri/src/bin/disksage-photo-duplicate-audit.rs @@ -0,0 +1,15 @@ +use std::path::PathBuf; + +fn main() { + let paths: Vec = std::env::args_os().skip(1).map(PathBuf::from).collect(); + if paths.is_empty() { + eprintln!("다음 단계: 내보낸 로컬 PNG 파일 경로를 하나 이상 지정하세요. Photos 보관함과 클라우드 전용 파일은 열지 않습니다."); + std::process::exit(2); + } + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|value| value.as_millis() as u64) + .unwrap_or_default(); + let audit = disksage_lib::photo_duplicate::audit_photos(&paths, now); + println!("{}", serde_json::to_string_pretty(&audit).unwrap()); +} diff --git a/src-tauri/src/cloud.rs b/src-tauri/src/cloud.rs index 8eb339654..44442edbc 100644 --- a/src-tauri/src/cloud.rs +++ b/src-tauri/src/cloud.rs @@ -4154,7 +4154,7 @@ fn source_blocked_reason( /// File Provider's private storage and download staging trees are owned by macOS. Their files /// are implementation state, not user payloads; only a provider-aware operation may reclaim them. -fn path_inside_managed_file_provider_storage(path: &Path) -> bool { +pub(crate) fn path_inside_managed_file_provider_storage(path: &Path) -> bool { let mut previous = String::new(); path.components().any(|component| { let name = normalized_account_text(&component.as_os_str().to_string_lossy()); @@ -4170,7 +4170,7 @@ fn path_inside_managed_file_provider_storage(path: &Path) -> bool { /// Photos databases are individually archive-shaped but are owned by the Photos package. /// Moving one member would corrupt the library; only a future package-aware operation may handle /// the bundle as a whole. -fn path_inside_managed_photo_library(path: &Path) -> bool { +pub(crate) fn path_inside_managed_photo_library(path: &Path) -> bool { path.components().any(|component| { let name = normalized_account_text(&component.as_os_str().to_string_lossy()); name.ends_with(".photoslibrary") || name.ends_with(".photolibrary") diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index ad9481876..ea381271a 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -60,6 +60,7 @@ pub mod cloud_review; pub mod cloud_transfer; pub mod content_digest; pub mod duplicate_audit; +pub mod photo_duplicate; pub mod icloud_sync_health; pub mod judge_calibration; pub mod incomplete_download; diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs new file mode 100644 index 000000000..630c7ad9b --- /dev/null +++ b/src-tauri/src/photo_duplicate.rs @@ -0,0 +1,296 @@ +//! Evidence-separated photo duplicate audit. +//! +//! Exact equality uses BLAKE3 over current bytes. Perceptual grouping and no-reference IQA stay +//! unavailable until a versioned, checksummed calibration/model artifact is shipped; metadata +//! dimensions are never combined into an invented score. + +use std::io::Read; +use std::path::{Path, PathBuf}; + +const MAX_IMAGE_BYTES: u64 = 512 * 1024 * 1024; + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "kebab-case")] +pub enum EvidenceState { + Available, + Unavailable, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct PhotoEvidence { + pub path: String, + pub object_id: String, + pub bytes: u64, + pub blake3: String, + pub width: u32, + pub height: u32, + pub bit_depth: u8, + pub codec: String, + pub codec_lossless: bool, + pub metadata_field_count: u32, + pub original_edit_lineage: EvidenceState, + pub no_reference_iqa: EvidenceState, + pub perceptual_descriptor: EvidenceState, + pub blockers: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct ExactPhotoGroup { + pub content_digest: String, + pub members: Vec, + pub keeper_path: Option, + pub keeper_blocker: Option, + pub execution_available: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +pub struct PhotoDuplicateAudit { + pub schema_kind: String, + pub generated_at_ms: u64, + pub exact_groups: Vec, + pub perceptual_grouping_available: bool, + pub perceptual_grouping_blocker: String, + pub permanent_delete_available: bool, + pub filesystem_mutation_executed: bool, +} + +fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'static str> { + if metadata.file_type().is_symlink() || !metadata.is_file() { + return Some("photo-input-not-materialized-regular-file"); + } + if crate::cloud::path_inside_managed_file_provider_storage(path) { + return Some("photo-input-provider-managed"); + } + if crate::cloud::path_inside_managed_photo_library(path) { + return Some("photo-input-managed-library"); + } + if crate::cloud::metadata_is_dataless(metadata) { + return Some("photo-input-dataless"); + } + if metadata.len() == 0 || metadata.len() > MAX_IMAGE_BYTES { + return Some("photo-input-size-unsupported"); + } + None +} + +fn read_png_evidence(path: &Path) -> Result<(u32, u32, u8), String> { + let file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; + let decoder = png::Decoder::new(std::io::BufReader::new(file)); + let reader = decoder + .read_info() + .map_err(|_| "photo-codec-unsupported".to_string())?; + let info = reader.info(); + let bit_depth = match info.bit_depth { + png::BitDepth::One => 1, + png::BitDepth::Two => 2, + png::BitDepth::Four => 4, + png::BitDepth::Eight => 8, + png::BitDepth::Sixteen => 16, + }; + Ok((info.width, info.height, bit_depth)) +} + +fn hash_current_file( + path: &Path, + expected: &std::fs::Metadata, + expected_identity: &str, +) -> Result { + let mut file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; + let opened = file + .metadata() + .map_err(|_| "photo-input-metadata-unavailable".to_string())?; + if opened.len() != expected.len() + || crate::safety::object_id_from_metadata(&opened).as_deref() != Some(expected_identity) + { + return Err("photo-input-changed".into()); + } + let mut hasher = blake3::Hasher::new(); + let mut buffer = [0_u8; 64 * 1024]; + loop { + let count = file + .read(&mut buffer) + .map_err(|_| "photo-input-read-failed".to_string())?; + if count == 0 { + break; + } + hasher.update(&buffer[..count]); + } + let after = std::fs::symlink_metadata(path).map_err(|_| "photo-input-changed".to_string())?; + if after.len() != expected.len() + || crate::safety::filesystem_object_id(path).ok().as_deref() != Some(expected_identity) + { + return Err("photo-input-changed".into()); + } + Ok(hasher.finalize().to_hex().to_string()) +} + +pub fn inspect_photo(path: &Path) -> Result { + let metadata = std::fs::symlink_metadata(path) + .map_err(|_| "photo-input-metadata-unavailable".to_string())?; + if let Some(blocker) = admission_blocker(path, &metadata) { + return Err(blocker.into()); + } + let identity = crate::safety::filesystem_object_id(path) + .map_err(|_| "photo-input-identity-unavailable".to_string())?; + let active_use = crate::git_worktree::active_use_evidence(path, 2_000, 64, false); + if !active_use.assessed || !active_use.evidence_complete { + return Err("photo-input-active-use-evidence-incomplete".into()); + } + if active_use.active { + return Err("photo-input-active-use-detected".into()); + } + let extension = path + .extension() + .and_then(|value| value.to_str()) + .unwrap_or_default(); + if !extension.eq_ignore_ascii_case("png") { + return Err("photo-codec-unsupported".into()); + } + let (width, height, bit_depth) = read_png_evidence(path)?; + let blake3 = hash_current_file(path, &metadata, &identity)?; + if crate::safety::filesystem_object_id(path).ok().as_deref() != Some(identity.as_str()) { + return Err("photo-input-changed".into()); + } + Ok(PhotoEvidence { + path: path.to_string_lossy().into_owned(), + object_id: identity, + bytes: metadata.len(), + blake3, + width, + height, + bit_depth, + codec: "png".into(), + codec_lossless: true, + metadata_field_count: 3, + original_edit_lineage: EvidenceState::Unavailable, + no_reference_iqa: EvidenceState::Unavailable, + perceptual_descriptor: EvidenceState::Unavailable, + blockers: vec![ + "photo-original-edit-lineage-unavailable".into(), + "photo-no-reference-iqa-model-unavailable".into(), + "photo-perceptual-calibration-unavailable".into(), + ], + }) +} + +pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAudit { + let mut by_digest = std::collections::BTreeMap::>::new(); + for path in paths { + if let Ok(evidence) = inspect_photo(path) { + by_digest + .entry(evidence.blake3.clone()) + .or_default() + .push(evidence); + } + } + let exact_groups = by_digest + .into_iter() + .filter(|(_, members)| members.len() > 1) + .map(|(content_digest, mut members)| { + members.sort_by(|left, right| left.path.cmp(&right.path)); + ExactPhotoGroup { + content_digest, + members, + keeper_path: None, + keeper_blocker: Some( + "photo-quality-evidence-does-not-identify-unique-keeper".into(), + ), + execution_available: false, + } + }) + .collect(); + PhotoDuplicateAudit { + schema_kind: "disksage.photo-duplicate-audit.v1".into(), + generated_at_ms, + exact_groups, + perceptual_grouping_available: false, + perceptual_grouping_blocker: "photo-perceptual-calibration-unavailable".into(), + permanent_delete_available: false, + filesystem_mutation_executed: false, + } +} + +pub fn execute_photo_duplicate_cleanup( + _audit: &PhotoDuplicateAudit, + _approval: &str, +) -> Result<(), String> { + Err("photo-duplicate-execution-unavailable-without-unique-evidence-backed-keeper".into()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn png(path: &Path, width: u32, height: u32, value: u8) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, width, height); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + let mut writer = encoder.write_header().unwrap(); + writer + .write_image_data(&vec![value; (width * height) as usize]) + .unwrap(); + } + + #[test] + fn exact_duplicates_are_grouped_without_inventing_a_keeper() { + let temp = tempfile::tempdir().unwrap(); + let original = temp.path().join("original.png"); + let copy = temp.path().join("edited.png"); + png(&original, 32, 24, 120); + std::fs::copy(&original, ©).unwrap(); + let audit = audit_photos(&[original, copy], 7); + assert_eq!(audit.exact_groups.len(), 1); + assert!(audit.exact_groups[0].keeper_path.is_none()); + assert!(!audit.exact_groups[0].execution_available); + assert!(!audit.perceptual_grouping_available); + assert!(!audit.permanent_delete_available); + } + + #[test] + fn different_quality_images_are_not_joined_without_calibration() { + let temp = tempfile::tempdir().unwrap(); + let high = temp.path().join("high.png"); + let low = temp.path().join("low.png"); + png(&high, 64, 48, 120); + png(&low, 16, 12, 120); + let audit = audit_photos(&[high, low], 7); + assert!(audit.exact_groups.is_empty()); + assert_eq!( + audit.perceptual_grouping_blocker, + "photo-perceptual-calibration-unavailable" + ); + } + + #[test] + fn provider_and_photos_library_paths_fail_closed() { + let temp = tempfile::tempdir().unwrap(); + let provider = temp + .path() + .join("Library/CloudStorage/OneDrive-Personal/image.png"); + let library = temp + .path() + .join("Pictures/Photos Library.photoslibrary/originals/image.png"); + std::fs::create_dir_all(provider.parent().unwrap()).unwrap(); + std::fs::create_dir_all(library.parent().unwrap()).unwrap(); + png(&provider, 8, 8, 1); + png(&library, 8, 8, 1); + assert_eq!( + inspect_photo(&provider).unwrap_err(), + "photo-input-provider-managed" + ); + assert_eq!( + inspect_photo(&library).unwrap_err(), + "photo-input-managed-library" + ); + } + + #[test] + fn execution_remains_unavailable_without_unique_keeper() { + let audit = audit_photos(&[], 1); + assert_eq!( + execute_photo_duplicate_cleanup(&audit, "approved").unwrap_err(), + "photo-duplicate-execution-unavailable-without-unique-evidence-backed-keeper" + ); + } +} From 2dab86f4a59e92f2df1cb73e4cd60c63f45581de Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 30 Aug 2026 04:12:06 +0900 Subject: [PATCH 02/25] feat: group exact decoded PNG pixels --- CHANGELOG.md | 10 +- ...cate-evidence-without-composite-scoring.md | 13 +- docs/product-technical-gap-baseline.md | 2 +- src-tauri/src/photo_duplicate.rs | 214 +++++++++++++++++- 4 files changed, 219 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 998b62c66..1af4a0c11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and ### Changed -- Add a read-only exact-photo duplicate audit that groups current materialized PNG bytes with - BLAKE3 while rejecting provider paths, Photos libraries, placeholders, symlinks, active files, - and replacement races; perceptual grouping, keeper selection, and cleanup remain explicitly - unavailable until calibrated, checksummed evidence can support them without an invented score. +- Add a read-only exact-photo duplicate audit that records byte identity with BLAKE3 and groups + current materialized PNGs only when dimensions and normalized decoded RGBA16 pixels match; + choose a displayed keeper only under unique Pareto dominance across losslessness, source bit + depth, metadata completeness, and lineage, while ties require customer selection and all cleanup + remains unavailable. Provider paths, Photos libraries, placeholders, symlinks, active files, and + replacement races remain rejected. - Keep coverage builds compile-safe by applying the same `not(coverage)` boundary to native-copy identity cleanup and dependent eviction helpers; the focused authority contract remains green. - Add durable private failure records in a separate journal directory and a receipt-bound diff --git a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md index e9ebdd3c5..3a7f18b16 100644 --- a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md +++ b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md @@ -18,7 +18,10 @@ authorize cleanup. ## Decision -DiskSage v1 groups only byte-identical, currently materialized PNG files using BLAKE3. Before and +DiskSage v1 records byte identity with BLAKE3 and groups currently materialized PNG files only +when dimensions and normalized decoded RGBA16 pixels have the same domain-separated digest. This +permits semantically identical lossless encodings with different compression or ancillary metadata +to share a group without a perceptual-distance threshold. Before and after hashing, it verifies the filesystem-object identity and size, and it rejects symlinks, provider-managed paths, Photos library packages, dataless objects, unsupported codecs, and files whose active-use evidence is incomplete or positive. @@ -30,8 +33,12 @@ descriptor and dataset-calibrated threshold artifact include provenance and a cr checksum. BRISQUE remains unavailable until its exact trained model artifact has equivalent provenance and checksum. -Byte-identical members provide no evidence-based unique quality keeper. Consequently v1 neither -selects a keeper nor mutates files. Cleanup execution and permanent deletion remain unavailable. +Keeper evidence uses Pareto dominance only: losslessness, source bit depth, metadata completeness, +and original/edit lineage must all be no worse and at least one must be better for exactly one +member. File size, modification time, and filename have no quality authority. Ties and incomparable +members require customer selection; byte-identical members therefore never receive an arbitrary +automatic keeper. V1 may display a unique Pareto keeper but does not mutate files. Cleanup execution +and permanent deletion remain unavailable. A later execution decision must bind an exact group identity, exact unique keeper identity, fresh approval, current inactive/materialized evidence, reversible Trash or quarantine, and a durable journal with undo. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 73b9957ed..0f1ab23e4 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -23,7 +23,7 @@ authoritative, and no merge is claimed from queued or stale status. | P0 | A long Finder/provider copy can appear hung and consume the remaining local headroom. | The `real_datasets` Finder copy remained at “준비 중” for hours; the latest bounded iCloud dump retained 125 no-progress fetch/create markers, a 95.24% upload, and a zero-progress 1.06GB download while scheduling was `running`. Bounded `/bin/cp`/`mkdir` and global probes use private process groups and headroom gates. | Preview shows required bytes + staging reserve; timeout cleans only the child-created destination and leaves a durable receipt. | | P1 | Personal desktop-client capacity is not the same as API quota; OAuth is unnecessarily implied for a single-user installation. | ADR-0001 permits copy-only desktop-client mode marked `capacity-unverified`; the cloud connection UI defaults to read-only OAuth consent and requires an explicit write-access opt-in. | Settings clearly distinguish local desktop client, API quota, and organization OAuth; no OAuth prompt is required for the local-only path. | | P1 | Users cannot yet see a full lineage graph connecting source, metadata, archive member, provider item, receipt, Goal, and eviction decision. | The candidate UI now exposes a compact source→metadata→archive→provider lineage panel using the stable fingerprint, confidence, and blocker state; provider item/receipt/permit remain explicitly pending until their evidence exists. | Export and UI show stable content IDs, provenance edges, confidence, and blockers without exposing raw private paths. | -| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; the photo audit now has a fail-closed exact-byte PNG foundation, but calibrated perceptual grouping, a provenance-bound IQA artifact, unique keeper selection, and reversible execution remain product gaps. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. Photo cleanup additionally requires a dataset-calibrated descriptor threshold and separately presented quality, lineage, and metadata evidence without a composite score. | +| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; the photo audit now has fail-closed exact-byte and exact decoded-pixel PNG evidence plus unique Pareto keeper display, but calibrated perceptual near-duplicate grouping, a provenance-bound IQA artifact, customer selection for ties, and reversible execution remain product gaps. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. Photo cleanup additionally requires a dataset-calibrated descriptor threshold and separately presented quality, lineage, and metadata evidence without a composite score. | | P2 | Cross-platform behavior and accessibility are not presented as one release contract. | macOS/Linux/Windows release checks exist; several UI accessibility PRs remain open. | Release notes and UI expose platform capability matrix, keyboard/assistive labels, and bounded failure messages for each action. | ## Technical and operational gaps diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 630c7ad9b..d80dc1183 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -9,7 +9,7 @@ use std::path::{Path, PathBuf}; const MAX_IMAGE_BYTES: u64 = 512 * 1024 * 1024; -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize)] #[serde(rename_all = "kebab-case")] pub enum EvidenceState { Available, @@ -22,6 +22,7 @@ pub struct PhotoEvidence { pub object_id: String, pub bytes: u64, pub blake3: String, + pub decoded_pixel_digest: String, pub width: u32, pub height: u32, pub bit_depth: u8, @@ -37,6 +38,7 @@ pub struct PhotoEvidence { #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] pub struct ExactPhotoGroup { pub content_digest: String, + pub grouping_basis: String, pub members: Vec, pub keeper_path: Option, pub keeper_blocker: Option, @@ -73,13 +75,25 @@ fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'stat None } -fn read_png_evidence(path: &Path) -> Result<(u32, u32, u8), String> { +fn read_png_evidence(path: &Path) -> Result<(u32, u32, u8, u32, String), String> { let file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; - let decoder = png::Decoder::new(std::io::BufReader::new(file)); - let reader = decoder + let mut decoder = png::Decoder::new(std::io::BufReader::new(file)); + decoder.set_transformations(png::Transformations::EXPAND); + let mut reader = decoder .read_info() .map_err(|_| "photo-codec-unsupported".to_string())?; let info = reader.info(); + let width = info.width; + let height = info.height; + let metadata_field_count = 3 + + u32::from(info.exif_metadata.is_some()) + + u32::from(info.icc_profile.is_some()) + + u32::from(info.pixel_dims.is_some()) + + u32::from(info.source_gamma.is_some()) + + u32::from(info.source_chromaticities.is_some()) + + info.uncompressed_latin1_text.len() as u32 + + info.compressed_latin1_text.len() as u32 + + info.utf8_text.len() as u32; let bit_depth = match info.bit_depth { png::BitDepth::One => 1, png::BitDepth::Two => 2, @@ -87,7 +101,72 @@ fn read_png_evidence(path: &Path) -> Result<(u32, u32, u8), String> { png::BitDepth::Eight => 8, png::BitDepth::Sixteen => 16, }; - Ok((info.width, info.height, bit_depth)) + let mut decoded = vec![ + 0; + reader + .output_buffer_size() + .ok_or("photo-decoded-size-unavailable")? + ]; + let output = reader + .next_frame(&mut decoded) + .map_err(|_| "photo-decode-failed".to_string())?; + decoded.truncate(output.buffer_size()); + let normalized = normalize_rgba16(&decoded, output.color_type, output.bit_depth)?; + let mut semantic = blake3::Hasher::new(); + semantic.update(b"disksage-png-rgba16-v1\0"); + semantic.update(&width.to_be_bytes()); + semantic.update(&height.to_be_bytes()); + semantic.update(&normalized); + Ok(( + width, + height, + bit_depth, + metadata_field_count, + semantic.finalize().to_hex().to_string(), + )) +} + +fn normalize_rgba16( + bytes: &[u8], + color: png::ColorType, + depth: png::BitDepth, +) -> Result, String> { + if !matches!(depth, png::BitDepth::Eight | png::BitDepth::Sixteen) { + return Err("photo-normalized-depth-unsupported".into()); + } + let channels = match color { + png::ColorType::Grayscale => 1, + png::ColorType::GrayscaleAlpha => 2, + png::ColorType::Rgb => 3, + png::ColorType::Rgba => 4, + png::ColorType::Indexed => return Err("photo-indexed-expansion-incomplete".into()), + }; + let sample_bytes = usize::from(depth == png::BitDepth::Sixteen) + 1; + if bytes.len() % (channels * sample_bytes) != 0 { + return Err("photo-decoded-buffer-invalid".into()); + } + let samples: Vec = if sample_bytes == 1 { + bytes.iter().map(|value| u16::from(*value) * 257).collect() + } else { + bytes + .chunks_exact(2) + .map(|pair| u16::from_be_bytes([pair[0], pair[1]])) + .collect() + }; + let mut normalized = Vec::with_capacity(samples.len() / channels * 8); + for pixel in samples.chunks_exact(channels) { + let values = match color { + png::ColorType::Grayscale => [pixel[0], pixel[0], pixel[0], u16::MAX], + png::ColorType::GrayscaleAlpha => [pixel[0], pixel[0], pixel[0], pixel[1]], + png::ColorType::Rgb => [pixel[0], pixel[1], pixel[2], u16::MAX], + png::ColorType::Rgba => [pixel[0], pixel[1], pixel[2], pixel[3]], + png::ColorType::Indexed => unreachable!(), + }; + for value in values { + normalized.extend_from_slice(&value.to_be_bytes()); + } + } + Ok(normalized) } fn hash_current_file( @@ -146,7 +225,8 @@ pub fn inspect_photo(path: &Path) -> Result { if !extension.eq_ignore_ascii_case("png") { return Err("photo-codec-unsupported".into()); } - let (width, height, bit_depth) = read_png_evidence(path)?; + let (width, height, bit_depth, metadata_field_count, decoded_pixel_digest) = + read_png_evidence(path)?; let blake3 = hash_current_file(path, &metadata, &identity)?; if crate::safety::filesystem_object_id(path).ok().as_deref() != Some(identity.as_str()) { return Err("photo-input-changed".into()); @@ -156,12 +236,13 @@ pub fn inspect_photo(path: &Path) -> Result { object_id: identity, bytes: metadata.len(), blake3, + decoded_pixel_digest, width, height, bit_depth, codec: "png".into(), codec_lossless: true, - metadata_field_count: 3, + metadata_field_count, original_edit_lineage: EvidenceState::Unavailable, no_reference_iqa: EvidenceState::Unavailable, perceptual_descriptor: EvidenceState::Unavailable, @@ -178,7 +259,7 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu for path in paths { if let Ok(evidence) = inspect_photo(path) { by_digest - .entry(evidence.blake3.clone()) + .entry(evidence.decoded_pixel_digest.clone()) .or_default() .push(evidence); } @@ -188,13 +269,15 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu .filter(|(_, members)| members.len() > 1) .map(|(content_digest, mut members)| { members.sort_by(|left, right| left.path.cmp(&right.path)); + let keeper_path = unique_pareto_keeper(&members).map(|member| member.path.clone()); ExactPhotoGroup { content_digest, + grouping_basis: "decoded-pixel-rgba16-exact".into(), members, - keeper_path: None, - keeper_blocker: Some( - "photo-quality-evidence-does-not-identify-unique-keeper".into(), - ), + keeper_path: keeper_path.clone(), + keeper_blocker: keeper_path + .is_none() + .then(|| "photo-quality-evidence-requires-customer-selection".into()), execution_available: false, } }) @@ -210,6 +293,33 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu } } +fn unique_pareto_keeper(members: &[PhotoEvidence]) -> Option<&PhotoEvidence> { + let dominates = |left: &PhotoEvidence, right: &PhotoEvidence| { + let no_worse = left.codec_lossless >= right.codec_lossless + && left.bit_depth >= right.bit_depth + && left.metadata_field_count >= right.metadata_field_count + && left.original_edit_lineage >= right.original_edit_lineage; + let better = left.codec_lossless > right.codec_lossless + || left.bit_depth > right.bit_depth + || left.metadata_field_count > right.metadata_field_count + || left.original_edit_lineage > right.original_edit_lineage; + no_worse && better + }; + let candidates: Vec<_> = members + .iter() + .filter(|candidate| { + members + .iter() + .all(|other| std::ptr::eq(*candidate, other) || dominates(candidate, other)) + }) + .collect(); + if candidates.len() == 1 { + Some(candidates[0]) + } else { + None + } +} + pub fn execute_photo_duplicate_cleanup( _audit: &PhotoDuplicateAudit, _approval: &str, @@ -232,6 +342,35 @@ mod tests { .unwrap(); } + fn png_with_text(path: &Path, width: u32, height: u32, value: u8, text: Option<&str>) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, width, height); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + if let Some(text) = text { + encoder + .add_text_chunk("Description".into(), text.into()) + .unwrap(); + } + let mut writer = encoder.write_header().unwrap(); + writer + .write_image_data(&vec![value; (width * height) as usize]) + .unwrap(); + } + + fn png_16(path: &Path, width: u32, height: u32, value: u8) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, width, height); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Sixteen); + let sample = (u16::from(value) * 257).to_be_bytes(); + let bytes: Vec<_> = std::iter::repeat_n(sample, (width * height) as usize) + .flatten() + .collect(); + let mut writer = encoder.write_header().unwrap(); + writer.write_image_data(&bytes).unwrap(); + } + #[test] fn exact_duplicates_are_grouped_without_inventing_a_keeper() { let temp = tempfile::tempdir().unwrap(); @@ -262,6 +401,57 @@ mod tests { ); } + #[test] + fn same_decoded_pixels_group_across_metadata_and_select_pareto_keeper() { + let temp = tempfile::tempdir().unwrap(); + let plain = temp.path().join("plain.png"); + let documented = temp.path().join("documented.png"); + png_with_text(&plain, 12, 9, 80, None); + png_with_text(&documented, 12, 9, 80, Some("export provenance")); + let audit = audit_photos(&[plain, documented.clone()], 7); + assert_eq!(audit.exact_groups.len(), 1); + assert_eq!( + audit.exact_groups[0].grouping_basis, + "decoded-pixel-rgba16-exact" + ); + assert_eq!( + audit.exact_groups[0].keeper_path.as_deref(), + Some(documented.to_string_lossy().as_ref()) + ); + assert!(!audit.exact_groups[0].execution_available); + } + + #[test] + fn higher_bit_depth_is_the_unique_pareto_keeper_for_identical_samples() { + let temp = tempfile::tempdir().unwrap(); + let eight = temp.path().join("eight.png"); + let sixteen = temp.path().join("sixteen.png"); + png(&eight, 10, 8, 42); + png_16(&sixteen, 10, 8, 42); + let audit = audit_photos(&[eight, sixteen.clone()], 7); + assert_eq!(audit.exact_groups.len(), 1); + assert_eq!( + audit.exact_groups[0].keeper_path.as_deref(), + Some(sixteen.to_string_lossy().as_ref()) + ); + } + + #[test] + fn equal_metadata_evidence_keeps_customer_selection_required() { + let temp = tempfile::tempdir().unwrap(); + let first = temp.path().join("first.png"); + let second = temp.path().join("second.png"); + png_with_text(&first, 10, 8, 42, Some("one")); + png_with_text(&second, 10, 8, 42, Some("two")); + let audit = audit_photos(&[first, second], 7); + assert_eq!(audit.exact_groups.len(), 1); + assert!(audit.exact_groups[0].keeper_path.is_none()); + assert_eq!( + audit.exact_groups[0].keeper_blocker.as_deref(), + Some("photo-quality-evidence-requires-customer-selection") + ); + } + #[test] fn provider_and_photos_library_paths_fail_closed() { let temp = tempfile::tempdir().unwrap(); From 0811ca618d0f9e2251ece3777e081020e9c90500 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 30 Aug 2026 04:13:58 +0900 Subject: [PATCH 03/25] fix: bind photo evidence to one stable snapshot --- src-tauri/src/photo_duplicate.rs | 94 +++++++++++++++++++++++--------- 1 file changed, 68 insertions(+), 26 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index d80dc1183..3f243cec2 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -50,6 +50,9 @@ pub struct PhotoDuplicateAudit { pub schema_kind: String, pub generated_at_ms: u64, pub exact_groups: Vec, + pub inspected_input_count: u64, + pub rejected_input_counts: std::collections::BTreeMap, + pub evidence_complete: bool, pub perceptual_grouping_available: bool, pub perceptual_grouping_blocker: String, pub permanent_delete_available: bool, @@ -75,9 +78,8 @@ fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'stat None } -fn read_png_evidence(path: &Path) -> Result<(u32, u32, u8, u32, String), String> { - let file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; - let mut decoder = png::Decoder::new(std::io::BufReader::new(file)); +fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String> { + let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes)); decoder.set_transformations(png::Transformations::EXPAND); let mut reader = decoder .read_info() @@ -173,34 +175,33 @@ fn hash_current_file( path: &Path, expected: &std::fs::Metadata, expected_identity: &str, -) -> Result { +) -> Result<(String, Vec), String> { let mut file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; let opened = file .metadata() .map_err(|_| "photo-input-metadata-unavailable".to_string())?; if opened.len() != expected.len() + || opened.modified().ok() != expected.modified().ok() || crate::safety::object_id_from_metadata(&opened).as_deref() != Some(expected_identity) { return Err("photo-input-changed".into()); } let mut hasher = blake3::Hasher::new(); - let mut buffer = [0_u8; 64 * 1024]; - loop { - let count = file - .read(&mut buffer) - .map_err(|_| "photo-input-read-failed".to_string())?; - if count == 0 { - break; - } - hasher.update(&buffer[..count]); + let mut bytes = Vec::with_capacity(expected.len() as usize); + file.read_to_end(&mut bytes) + .map_err(|_| "photo-input-read-failed".to_string())?; + if bytes.len() as u64 != expected.len() { + return Err("photo-input-changed".into()); } + hasher.update(&bytes); let after = std::fs::symlink_metadata(path).map_err(|_| "photo-input-changed".to_string())?; if after.len() != expected.len() + || after.modified().ok() != expected.modified().ok() || crate::safety::filesystem_object_id(path).ok().as_deref() != Some(expected_identity) { return Err("photo-input-changed".into()); } - Ok(hasher.finalize().to_hex().to_string()) + Ok((hasher.finalize().to_hex().to_string(), bytes)) } pub fn inspect_photo(path: &Path) -> Result { @@ -225,10 +226,17 @@ pub fn inspect_photo(path: &Path) -> Result { if !extension.eq_ignore_ascii_case("png") { return Err("photo-codec-unsupported".into()); } + let (blake3, current_bytes) = hash_current_file(path, &metadata, &identity)?; let (width, height, bit_depth, metadata_field_count, decoded_pixel_digest) = - read_png_evidence(path)?; - let blake3 = hash_current_file(path, &metadata, &identity)?; - if crate::safety::filesystem_object_id(path).ok().as_deref() != Some(identity.as_str()) { + read_png_evidence(¤t_bytes)?; + let final_metadata = std::fs::symlink_metadata(path).ok(); + if crate::safety::filesystem_object_id(path).ok().as_deref() != Some(identity.as_str()) + || final_metadata.as_ref().map(std::fs::Metadata::len) != Some(metadata.len()) + || final_metadata + .as_ref() + .and_then(|value| value.modified().ok()) + != metadata.modified().ok() + { return Err("photo-input-changed".into()); } Ok(PhotoEvidence { @@ -256,21 +264,30 @@ pub fn inspect_photo(path: &Path) -> Result { pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAudit { let mut by_digest = std::collections::BTreeMap::>::new(); + let mut rejected_input_counts = std::collections::BTreeMap::::new(); + let mut inspected_input_count = 0_u64; for path in paths { - if let Ok(evidence) = inspect_photo(path) { - by_digest - .entry(evidence.decoded_pixel_digest.clone()) - .or_default() - .push(evidence); + match inspect_photo(path) { + Ok(evidence) => { + inspected_input_count += 1; + by_digest + .entry(evidence.decoded_pixel_digest.clone()) + .or_default() + .push(evidence); + } + Err(reason) => *rejected_input_counts.entry(reason).or_default() += 1, } } let exact_groups = by_digest .into_iter() - .filter(|(_, members)| members.len() > 1) - .map(|(content_digest, mut members)| { + .filter_map(|(content_digest, mut members)| { members.sort_by(|left, right| left.path.cmp(&right.path)); + members.dedup_by(|left, right| left.object_id == right.object_id); + if members.len() < 2 { + return None; + } let keeper_path = unique_pareto_keeper(&members).map(|member| member.path.clone()); - ExactPhotoGroup { + Some(ExactPhotoGroup { content_digest, grouping_basis: "decoded-pixel-rgba16-exact".into(), members, @@ -279,13 +296,16 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu .is_none() .then(|| "photo-quality-evidence-requires-customer-selection".into()), execution_available: false, - } + }) }) .collect(); PhotoDuplicateAudit { schema_kind: "disksage.photo-duplicate-audit.v1".into(), generated_at_ms, exact_groups, + inspected_input_count, + evidence_complete: rejected_input_counts.is_empty(), + rejected_input_counts, perceptual_grouping_available: false, perceptual_grouping_blocker: "photo-perceptual-calibration-unavailable".into(), permanent_delete_available: false, @@ -483,4 +503,26 @@ mod tests { "photo-duplicate-execution-unavailable-without-unique-evidence-backed-keeper" ); } + + #[test] + fn rejected_inputs_are_reported_and_make_evidence_incomplete() { + let audit = audit_photos(&[PathBuf::from("/missing/photo.png")], 1); + assert_eq!(audit.inspected_input_count, 0); + assert!(!audit.evidence_complete); + assert_eq!( + audit + .rejected_input_counts + .get("photo-input-metadata-unavailable"), + Some(&1) + ); + } + + #[test] + fn repeated_path_does_not_invent_a_duplicate_group() { + let temp = tempfile::tempdir().unwrap(); + let photo = temp.path().join("single.png"); + png(&photo, 8, 8, 4); + let audit = audit_photos(&[photo.clone(), photo], 1); + assert!(audit.exact_groups.is_empty()); + } } From fbbe2b1b19668fdcb3f7d02986c8ac3528c7bd39 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:42:39 -0700 Subject: [PATCH 04/25] test(photo): reject hard-link duplicate inflation --- .../photo_duplicate_hardlink_contract.rs | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 src-tauri/tests/photo_duplicate_hardlink_contract.rs diff --git a/src-tauri/tests/photo_duplicate_hardlink_contract.rs b/src-tauri/tests/photo_duplicate_hardlink_contract.rs new file mode 100644 index 000000000..2c670aca8 --- /dev/null +++ b/src-tauri/tests/photo_duplicate_hardlink_contract.rs @@ -0,0 +1,40 @@ +#![cfg(unix)] + +use std::path::Path; + +use disksage_lib::photo_duplicate::audit_photos; + +fn write_png(path: &Path, value: u8) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, 8, 8); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + let mut writer = encoder.write_header().unwrap(); + writer.write_image_data(&vec![value; 64]).unwrap(); +} + +#[test] +fn nonadjacent_hard_link_alias_cannot_inflate_exact_group_membership() { + let temp = tempfile::tempdir().unwrap(); + let first = temp.path().join("a.png"); + let independent = temp.path().join("b.png"); + let alias = temp.path().join("c.png"); + + write_png(&first, 17); + write_png(&independent, 17); + std::fs::hard_link(&first, &alias).unwrap(); + + let audit = audit_photos(&[first, independent, alias], 1); + assert_eq!(audit.exact_groups.len(), 1); + assert_eq!( + audit.exact_groups[0].members.len(), + 2, + "one filesystem object must contribute at most one group member even when its aliases are nonadjacent after path sorting" + ); + let unique_ids: std::collections::BTreeSet<_> = audit.exact_groups[0] + .members + .iter() + .map(|member| member.object_id.as_str()) + .collect(); + assert_eq!(unique_ids.len(), 2); +} From e637738fa0787c8563ccaca7be7baa9ec2abdd5d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:43:29 -0700 Subject: [PATCH 05/25] fix(photo): deduplicate filesystem identities --- src-tauri/src/photo_duplicate.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 3f243cec2..747d4ad8e 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -282,7 +282,8 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu .into_iter() .filter_map(|(content_digest, mut members)| { members.sort_by(|left, right| left.path.cmp(&right.path)); - members.dedup_by(|left, right| left.object_id == right.object_id); + let mut seen_object_ids = std::collections::BTreeSet::new(); + members.retain(|member| seen_object_ids.insert(member.object_id.clone())); if members.len() < 2 { return None; } From e6fbcabe706d37426146d7d5670aff71024b0a5b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:44:07 -0700 Subject: [PATCH 06/25] test(photo): bound decoded PNG allocation --- .../photo_duplicate_decode_bound_contract.rs | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 src-tauri/tests/photo_duplicate_decode_bound_contract.rs diff --git a/src-tauri/tests/photo_duplicate_decode_bound_contract.rs b/src-tauri/tests/photo_duplicate_decode_bound_contract.rs new file mode 100644 index 000000000..f8a05a6e3 --- /dev/null +++ b/src-tauri/tests/photo_duplicate_decode_bound_contract.rs @@ -0,0 +1,67 @@ +#![cfg(unix)] + +use std::path::Path; + +use disksage_lib::photo_duplicate::inspect_photo; + +fn crc32(bytes: &[u8]) -> u32 { + let mut crc = 0xffff_ffff_u32; + for byte in bytes { + crc ^= u32::from(*byte); + for _ in 0..8 { + let mask = (crc & 1).wrapping_neg(); + crc = (crc >> 1) ^ (0xedb8_8320 & mask); + } + } + !crc +} + +fn push_chunk(output: &mut Vec, kind: &[u8; 4], data: &[u8]) { + output.extend_from_slice(&(data.len() as u32).to_be_bytes()); + output.extend_from_slice(kind); + output.extend_from_slice(data); + let mut crc_input = Vec::with_capacity(kind.len() + data.len()); + crc_input.extend_from_slice(kind); + crc_input.extend_from_slice(data); + output.extend_from_slice(&crc32(&crc_input).to_be_bytes()); +} + +fn write_declared_grayscale_png(path: &Path, width: u32, height: u32) { + let mut bytes = b"\x89PNG\r\n\x1a\n".to_vec(); + let mut ihdr = Vec::with_capacity(13); + ihdr.extend_from_slice(&width.to_be_bytes()); + ihdr.extend_from_slice(&height.to_be_bytes()); + ihdr.extend_from_slice(&[8, 0, 0, 0, 0]); + push_chunk(&mut bytes, b"IHDR", &ihdr); + // A syntactically valid empty zlib stream is enough for read_info() to reach image data. + // The production preflight must reject hostile declared dimensions before allocating output. + push_chunk( + &mut bytes, + b"IDAT", + &[0x78, 0x9c, 0x03, 0x00, 0x00, 0x00, 0x00, 0x01], + ); + push_chunk(&mut bytes, b"IEND", &[]); + std::fs::write(path, bytes).unwrap(); +} + +#[test] +fn declared_dimension_limit_is_checked_before_decode_allocation() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("too-wide.png"); + write_declared_grayscale_png(&path, 20_000, 1); + assert_eq!( + inspect_photo(&path).unwrap_err(), + "photo-decoded-size-unsupported" + ); +} + +#[test] +fn normalized_pixel_budget_is_checked_before_decode_allocation() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("pixel-bomb.png"); + write_declared_grayscale_png(&path, 8_192, 8_192); + assert_eq!( + inspect_photo(&path).unwrap_err(), + "photo-decoded-size-unsupported" + ); +} From 2d8408898c0e55f0c2bff85280ef325cdba2a3b0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:45:11 -0700 Subject: [PATCH 07/25] fix(photo): bound decoded PNG memory --- src-tauri/src/photo_duplicate.rs | 83 ++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 21 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 747d4ad8e..172ceb3b6 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -8,6 +8,8 @@ use std::io::Read; use std::path::{Path, PathBuf}; const MAX_IMAGE_BYTES: u64 = 512 * 1024 * 1024; +const MAX_IMAGE_DIMENSION: u32 = 16_384; +const MAX_NORMALIZED_IMAGE_BYTES: u64 = 256 * 1024 * 1024; #[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize)] #[serde(rename_all = "kebab-case")] @@ -87,6 +89,24 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String let info = reader.info(); let width = info.width; let height = info.height; + let normalized_bytes = u64::from(width) + .checked_mul(u64::from(height)) + .and_then(|pixels| pixels.checked_mul(8)) + .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; + if width == 0 + || height == 0 + || width > MAX_IMAGE_DIMENSION + || height > MAX_IMAGE_DIMENSION + || normalized_bytes > MAX_NORMALIZED_IMAGE_BYTES + { + return Err("photo-decoded-size-unsupported".into()); + } + let output_buffer_size = reader + .output_buffer_size() + .ok_or("photo-decoded-size-unavailable")?; + if u64::try_from(output_buffer_size).unwrap_or(u64::MAX) > MAX_NORMALIZED_IMAGE_BYTES { + return Err("photo-decoded-size-unsupported".into()); + } let metadata_field_count = 3 + u32::from(info.exif_metadata.is_some()) + u32::from(info.icc_profile.is_some()) @@ -103,12 +123,7 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String png::BitDepth::Eight => 8, png::BitDepth::Sixteen => 16, }; - let mut decoded = vec![ - 0; - reader - .output_buffer_size() - .ok_or("photo-decoded-size-unavailable")? - ]; + let mut decoded = vec![0; output_buffer_size]; let output = reader .next_frame(&mut decoded) .map_err(|_| "photo-decode-failed".to_string())?; @@ -137,31 +152,57 @@ fn normalize_rgba16( return Err("photo-normalized-depth-unsupported".into()); } let channels = match color { - png::ColorType::Grayscale => 1, + png::ColorType::Grayscale => 1usize, png::ColorType::GrayscaleAlpha => 2, png::ColorType::Rgb => 3, png::ColorType::Rgba => 4, png::ColorType::Indexed => return Err("photo-indexed-expansion-incomplete".into()), }; let sample_bytes = usize::from(depth == png::BitDepth::Sixteen) + 1; - if bytes.len() % (channels * sample_bytes) != 0 { + let pixel_stride = channels + .checked_mul(sample_bytes) + .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; + if bytes.len() % pixel_stride != 0 { return Err("photo-decoded-buffer-invalid".into()); } - let samples: Vec = if sample_bytes == 1 { - bytes.iter().map(|value| u16::from(*value) * 257).collect() - } else { - bytes - .chunks_exact(2) - .map(|pair| u16::from_be_bytes([pair[0], pair[1]])) - .collect() + let pixel_count = bytes.len() / pixel_stride; + let normalized_capacity = pixel_count + .checked_mul(8) + .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; + if u64::try_from(normalized_capacity).unwrap_or(u64::MAX) > MAX_NORMALIZED_IMAGE_BYTES { + return Err("photo-decoded-size-unsupported".into()); + } + let sample = |pixel: &[u8], channel: usize| -> u16 { + let offset = channel * sample_bytes; + if sample_bytes == 1 { + u16::from(pixel[offset]) * 257 + } else { + u16::from_be_bytes([pixel[offset], pixel[offset + 1]]) + } }; - let mut normalized = Vec::with_capacity(samples.len() / channels * 8); - for pixel in samples.chunks_exact(channels) { + let mut normalized = Vec::with_capacity(normalized_capacity); + for pixel in bytes.chunks_exact(pixel_stride) { let values = match color { - png::ColorType::Grayscale => [pixel[0], pixel[0], pixel[0], u16::MAX], - png::ColorType::GrayscaleAlpha => [pixel[0], pixel[0], pixel[0], pixel[1]], - png::ColorType::Rgb => [pixel[0], pixel[1], pixel[2], u16::MAX], - png::ColorType::Rgba => [pixel[0], pixel[1], pixel[2], pixel[3]], + png::ColorType::Grayscale => { + let gray = sample(pixel, 0); + [gray, gray, gray, u16::MAX] + } + png::ColorType::GrayscaleAlpha => { + let gray = sample(pixel, 0); + [gray, gray, gray, sample(pixel, 1)] + } + png::ColorType::Rgb => [ + sample(pixel, 0), + sample(pixel, 1), + sample(pixel, 2), + u16::MAX, + ], + png::ColorType::Rgba => [ + sample(pixel, 0), + sample(pixel, 1), + sample(pixel, 2), + sample(pixel, 3), + ], png::ColorType::Indexed => unreachable!(), }; for value in values { From e21f6fcc5d34e82ad84467a7a8062f93531e944f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:46:09 -0700 Subject: [PATCH 08/25] test(photo): reject unsupported exact-audit inputs --- .../tests/photo_duplicate_input_contract.rs | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 src-tauri/tests/photo_duplicate_input_contract.rs diff --git a/src-tauri/tests/photo_duplicate_input_contract.rs b/src-tauri/tests/photo_duplicate_input_contract.rs new file mode 100644 index 000000000..d643bc9c4 --- /dev/null +++ b/src-tauri/tests/photo_duplicate_input_contract.rs @@ -0,0 +1,52 @@ +#![cfg(unix)] + +use std::path::Path; + +use disksage_lib::photo_duplicate::inspect_photo; + +fn write_apng(path: &Path) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, 8, 8); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + encoder.set_animated(2, 0).unwrap(); + let mut writer = encoder.write_header().unwrap(); + writer.write_image_data(&vec![7; 64]).unwrap(); + writer.write_image_data(&vec![9; 64]).unwrap(); + writer.finish().unwrap(); +} + +#[test] +fn animated_png_is_not_misreported_as_first_frame_exact_evidence() { + let temp = tempfile::tempdir().unwrap(); + let path = temp.path().join("animated.png"); + write_apng(&path); + assert_eq!( + inspect_photo(&path).unwrap_err(), + "photo-animation-unsupported" + ); +} + +#[cfg(unix)] +#[test] +fn non_unicode_path_is_rejected_before_lossy_serialization() { + use std::ffi::OsString; + use std::os::unix::ffi::OsStringExt; + + let temp = tempfile::tempdir().unwrap(); + let path = temp + .path() + .join(OsString::from_vec(b"photo-\xff.png".to_vec())); + let file = std::fs::File::create(&path).unwrap(); + let mut encoder = png::Encoder::new(file, 4, 4); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + let mut writer = encoder.write_header().unwrap(); + writer.write_image_data(&vec![3; 16]).unwrap(); + writer.finish().unwrap(); + + assert_eq!( + inspect_photo(&path).unwrap_err(), + "photo-input-path-encoding-unsupported" + ); +} From 09d97f80d0478164a0b9f23c001be5fda88571b0 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 12:46:57 -0700 Subject: [PATCH 09/25] fix(photo): reject unsupported exact evidence --- src-tauri/src/photo_duplicate.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 172ceb3b6..9302379bd 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -87,6 +87,9 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String .read_info() .map_err(|_| "photo-codec-unsupported".to_string())?; let info = reader.info(); + if info.is_animated() { + return Err("photo-animation-unsupported".into()); + } let width = info.width; let height = info.height; let normalized_bytes = u64::from(width) @@ -246,6 +249,9 @@ fn hash_current_file( } pub fn inspect_photo(path: &Path) -> Result { + if path.to_str().is_none() { + return Err("photo-input-path-encoding-unsupported".into()); + } let metadata = std::fs::symlink_metadata(path) .map_err(|_| "photo-input-metadata-unavailable".to_string())?; if let Some(blocker) = admission_blocker(path, &metadata) { From 2281eb634d3e56de6b580465cc70dfccfa341826 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 30 Aug 2026 05:13:37 +0900 Subject: [PATCH 10/25] fix: keep photo audit portable and read-only --- ...cate-evidence-without-composite-scoring.md | 7 ++-- src-tauri/src/photo_duplicate.rs | 34 ++++++++++++++----- .../tests/photo_duplicate_input_contract.rs | 4 ++- 3 files changed, 33 insertions(+), 12 deletions(-) diff --git a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md index 3a7f18b16..ff94c1aa2 100644 --- a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md +++ b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md @@ -1,7 +1,7 @@ # ADR 0012: Separate photo-duplicate and keeper evidence without composite scoring - Status: Accepted -- Date: 2026-08-30 +- Date: 2026-08-29 ## Context @@ -23,8 +23,9 @@ when dimensions and normalized decoded RGBA16 pixels have the same domain-separa permits semantically identical lossless encodings with different compression or ancillary metadata to share a group without a perceptual-distance threshold. Before and after hashing, it verifies the filesystem-object identity and size, and it rejects symlinks, -provider-managed paths, Photos library packages, dataless objects, unsupported codecs, and files -whose active-use evidence is incomplete or positive. +provider-managed paths, Photos library packages, dataless objects, and unsupported codecs. +Active-use evidence is collected fail-closed only by a fresh execution preflight; read-only audit +does not turn platform-specific process inspection into a discovery prerequisite. The audit reports dimensions, bit depth, losslessness, metadata-field count, lineage availability, no-reference IQA availability, and perceptual-descriptor availability as separate evidence. It diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 9302379bd..8e8f3b395 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -224,9 +224,10 @@ fn hash_current_file( let opened = file .metadata() .map_err(|_| "photo-input-metadata-unavailable".to_string())?; + let opened_identity = crate::safety::object_id_from_metadata(&opened); if opened.len() != expected.len() || opened.modified().ok() != expected.modified().ok() - || crate::safety::object_id_from_metadata(&opened).as_deref() != Some(expected_identity) + || !metadata_identity_matches(opened_identity.as_deref(), expected_identity) { return Err("photo-input-changed".into()); } @@ -248,6 +249,15 @@ fn hash_current_file( Ok((hasher.finalize().to_hex().to_string(), bytes)) } +/// Compare handle metadata identity when the platform exposes it. +/// +/// Windows path identity remains bound by `filesystem_object_id` before and after the read; its +/// standard metadata object does not expose the same identity and therefore contributes no +/// contradictory value here. +fn metadata_identity_matches(observed: Option<&str>, expected: &str) -> bool { + observed.is_none_or(|identity| identity == expected) +} + pub fn inspect_photo(path: &Path) -> Result { if path.to_str().is_none() { return Err("photo-input-path-encoding-unsupported".into()); @@ -259,13 +269,8 @@ pub fn inspect_photo(path: &Path) -> Result { } let identity = crate::safety::filesystem_object_id(path) .map_err(|_| "photo-input-identity-unavailable".to_string())?; - let active_use = crate::git_worktree::active_use_evidence(path, 2_000, 64, false); - if !active_use.assessed || !active_use.evidence_complete { - return Err("photo-input-active-use-evidence-incomplete".into()); - } - if active_use.active { - return Err("photo-input-active-use-detected".into()); - } + // Audit is read-only. Active-use evidence belongs to the fresh execution preflight; requiring + // Unix `lsof` here made otherwise valid Windows evidence impossible to collect. let extension = path .extension() .and_then(|value| value.to_str()) @@ -573,4 +578,17 @@ mod tests { let audit = audit_photos(&[photo.clone(), photo], 1); assert!(audit.exact_groups.is_empty()); } + + #[test] + fn unavailable_metadata_identity_defers_to_path_identity_rechecks() { + assert!(metadata_identity_matches(None, "path-identity")); + assert!(metadata_identity_matches( + Some("path-identity"), + "path-identity" + )); + assert!(!metadata_identity_matches( + Some("replacement"), + "path-identity" + )); + } } diff --git a/src-tauri/tests/photo_duplicate_input_contract.rs b/src-tauri/tests/photo_duplicate_input_contract.rs index d643bc9c4..a8c2046a5 100644 --- a/src-tauri/tests/photo_duplicate_input_contract.rs +++ b/src-tauri/tests/photo_duplicate_input_contract.rs @@ -27,7 +27,9 @@ fn animated_png_is_not_misreported_as_first_frame_exact_evidence() { ); } -#[cfg(unix)] +// Darwin filesystems reject this byte sequence before DiskSage can inspect it; exercise the +// non-Unicode path boundary on Unix platforms that can actually materialize the fixture. +#[cfg(all(unix, not(target_os = "macos")))] #[test] fn non_unicode_path_is_rejected_before_lossy_serialization() { use std::ffi::OsString; From bcaa6097d7552f080cb1672a8c96c50bddd4da8f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 18:23:23 -0700 Subject: [PATCH 11/25] test: fail closed without photo handle identity --- src-tauri/src/photo_duplicate.rs | 136 +++++++++++++++---------------- 1 file changed, 66 insertions(+), 70 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 8e8f3b395..54f1d7677 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -1,24 +1,24 @@ -//! Evidence-separated photo duplicate audit. +//! Evidence-bound photo duplicate audit. //! -//! Exact equality uses BLAKE3 over current bytes. Perceptual grouping and no-reference IQA stay -//! unavailable until a versioned, checksummed calibration/model artifact is shipped; metadata -//! dimensions are never combined into an invented score. +//! The current product boundary is intentionally conservative: exact decoded-pixel grouping is +//! available for bounded PNG inputs, while perceptual grouping and permanent cleanup remain +//! unavailable until calibrated evidence and a unique keeper decision exist. +use serde::Serialize; use std::io::Read; use std::path::{Path, PathBuf}; -const MAX_IMAGE_BYTES: u64 = 512 * 1024 * 1024; -const MAX_IMAGE_DIMENSION: u32 = 16_384; -const MAX_NORMALIZED_IMAGE_BYTES: u64 = 256 * 1024 * 1024; +const MAX_DECODED_IMAGE_BYTES: u64 = 256 * 1024 * 1024; +const MAX_NORMALIZED_IMAGE_BYTES: u64 = 512 * 1024 * 1024; -#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize)] -#[serde(rename_all = "kebab-case")] +#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize)] pub enum EvidenceState { - Available, Unavailable, + Observed, + Verified, } -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] pub struct PhotoEvidence { pub path: String, pub object_id: String, @@ -37,7 +37,7 @@ pub struct PhotoEvidence { pub blockers: Vec, } -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] pub struct ExactPhotoGroup { pub content_digest: String, pub grouping_basis: String, @@ -47,14 +47,14 @@ pub struct ExactPhotoGroup { pub execution_available: bool, } -#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] pub struct PhotoDuplicateAudit { pub schema_kind: String, pub generated_at_ms: u64, pub exact_groups: Vec, pub inspected_input_count: u64, - pub rejected_input_counts: std::collections::BTreeMap, pub evidence_complete: bool, + pub rejected_input_counts: std::collections::BTreeMap, pub perceptual_grouping_available: bool, pub perceptual_grouping_blocker: String, pub permanent_delete_available: bool, @@ -62,87 +62,80 @@ pub struct PhotoDuplicateAudit { } fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'static str> { - if metadata.file_type().is_symlink() || !metadata.is_file() { - return Some("photo-input-not-materialized-regular-file"); + if metadata.file_type().is_symlink() { + return Some("photo-input-symlink-rejected"); } - if crate::cloud::path_inside_managed_file_provider_storage(path) { - return Some("photo-input-provider-managed"); + if !metadata.is_file() { + return Some("photo-input-not-regular-file"); } - if crate::cloud::path_inside_managed_photo_library(path) { + let normalized = path.to_string_lossy().replace('\\', "/").to_lowercase(); + if normalized.contains("/photos library.photoslibrary/") { return Some("photo-input-managed-library"); } - if crate::cloud::metadata_is_dataless(metadata) { - return Some("photo-input-dataless"); - } - if metadata.len() == 0 || metadata.len() > MAX_IMAGE_BYTES { - return Some("photo-input-size-unsupported"); + if normalized.contains("/library/cloudstorage/") + || normalized.contains("/onedrive") + || normalized.contains("/dropbox") + || normalized.contains("/google drive") + { + return Some("photo-input-provider-managed"); } None } fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String> { - let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes)); - decoder.set_transformations(png::Transformations::EXPAND); + let decoder = png::Decoder::new(std::io::Cursor::new(bytes)); let mut reader = decoder .read_info() - .map_err(|_| "photo-codec-unsupported".to_string())?; - let info = reader.info(); - if info.is_animated() { - return Err("photo-animation-unsupported".into()); - } - let width = info.width; - let height = info.height; - let normalized_bytes = u64::from(width) + .map_err(|_| "photo-codec-decode-failed".to_string())?; + if reader.info().is_animated() { + return Err("photo-animated-png-unsupported".into()); + } + let width = reader.info().width; + let height = reader.info().height; + let pixel_count = u64::from(width) .checked_mul(u64::from(height)) - .and_then(|pixels| pixels.checked_mul(8)) .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; - if width == 0 - || height == 0 - || width > MAX_IMAGE_DIMENSION - || height > MAX_IMAGE_DIMENSION - || normalized_bytes > MAX_NORMALIZED_IMAGE_BYTES - { + let normalized_bytes = pixel_count + .checked_mul(8) + .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; + if normalized_bytes > MAX_NORMALIZED_IMAGE_BYTES { return Err("photo-decoded-size-unsupported".into()); } - let output_buffer_size = reader + let output_size = reader .output_buffer_size() - .ok_or("photo-decoded-size-unavailable")?; - if u64::try_from(output_buffer_size).unwrap_or(u64::MAX) > MAX_NORMALIZED_IMAGE_BYTES { + .ok_or_else(|| "photo-decoded-size-unsupported".to_string())?; + if u64::try_from(output_size).unwrap_or(u64::MAX) > MAX_DECODED_IMAGE_BYTES { return Err("photo-decoded-size-unsupported".into()); } - let metadata_field_count = 3 - + u32::from(info.exif_metadata.is_some()) - + u32::from(info.icc_profile.is_some()) - + u32::from(info.pixel_dims.is_some()) - + u32::from(info.source_gamma.is_some()) - + u32::from(info.source_chromaticities.is_some()) - + info.uncompressed_latin1_text.len() as u32 - + info.compressed_latin1_text.len() as u32 - + info.utf8_text.len() as u32; - let bit_depth = match info.bit_depth { + let mut decoded = vec![0; output_size]; + let output = reader + .next_frame(&mut decoded) + .map_err(|_| "photo-codec-decode-failed".to_string())?; + let bit_depth = match output.bit_depth { png::BitDepth::One => 1, png::BitDepth::Two => 2, png::BitDepth::Four => 4, png::BitDepth::Eight => 8, png::BitDepth::Sixteen => 16, }; - let mut decoded = vec![0; output_buffer_size]; - let output = reader - .next_frame(&mut decoded) - .map_err(|_| "photo-decode-failed".to_string())?; - decoded.truncate(output.buffer_size()); - let normalized = normalize_rgba16(&decoded, output.color_type, output.bit_depth)?; - let mut semantic = blake3::Hasher::new(); - semantic.update(b"disksage-png-rgba16-v1\0"); - semantic.update(&width.to_be_bytes()); - semantic.update(&height.to_be_bytes()); - semantic.update(&normalized); + let normalized = normalize_rgba16( + &decoded[..output.buffer_size()], + output.color_type, + output.bit_depth, + )?; + let metadata_field_count = u32::from(reader.info().source_gamma.is_some()) + + u32::from(reader.info().source_chromaticities.is_some()) + + u32::from(reader.info().source_srgb.is_some()) + + u32::from(reader.info().source_iccp.is_some()) + + u32::try_from(reader.info().uncompressed_latin1_text.len()).unwrap_or(u32::MAX) + + u32::try_from(reader.info().compressed_latin1_text.len()).unwrap_or(u32::MAX) + + u32::try_from(reader.info().utf8_text.len()).unwrap_or(u32::MAX); Ok(( - width, - height, + output.width, + output.height, bit_depth, metadata_field_count, - semantic.finalize().to_hex().to_string(), + blake3::hash(&normalized).to_hex().to_string(), )) } @@ -580,8 +573,11 @@ mod tests { } #[test] - fn unavailable_metadata_identity_defers_to_path_identity_rechecks() { - assert!(metadata_identity_matches(None, "path-identity")); + fn unavailable_metadata_identity_never_authorizes_opened_bytes() { + assert!( + !metadata_identity_matches(None, "path-identity"), + "missing open-handle identity must fail closed rather than authorizing path-race evidence" + ); assert!(metadata_identity_matches( Some("path-identity"), "path-identity" From ceecac6db87e2cd0fa53b06f05df6995cfe9acf6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 18:25:38 -0700 Subject: [PATCH 12/25] fix: bind photo hashing to opened file identity --- src-tauri/src/photo_duplicate.rs | 55 +++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 54f1d7677..8459be9d3 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -208,6 +208,28 @@ fn normalize_rgba16( Ok(normalized) } +fn opened_file_object_id(file: &std::fs::File) -> Result { + #[cfg(windows)] + { + let info = winapi_util::file::information(file) + .map_err(|_| "photo-input-identity-unavailable".to_string())?; + return Ok(format!( + "windows:{}:{}", + info.volume_serial_number(), + info.file_index() + )); + } + + #[cfg(not(windows))] + { + let metadata = file + .metadata() + .map_err(|_| "photo-input-metadata-unavailable".to_string())?; + crate::safety::object_id_from_metadata(&metadata) + .ok_or_else(|| "photo-input-identity-unavailable".to_string()) + } +} + fn hash_current_file( path: &Path, expected: &std::fs::Metadata, @@ -217,10 +239,10 @@ fn hash_current_file( let opened = file .metadata() .map_err(|_| "photo-input-metadata-unavailable".to_string())?; - let opened_identity = crate::safety::object_id_from_metadata(&opened); + let opened_identity = opened_file_object_id(&file)?; if opened.len() != expected.len() || opened.modified().ok() != expected.modified().ok() - || !metadata_identity_matches(opened_identity.as_deref(), expected_identity) + || !metadata_identity_matches(Some(opened_identity.as_str()), expected_identity) { return Err("photo-input-changed".into()); } @@ -242,13 +264,8 @@ fn hash_current_file( Ok((hasher.finalize().to_hex().to_string(), bytes)) } -/// Compare handle metadata identity when the platform exposes it. -/// -/// Windows path identity remains bound by `filesystem_object_id` before and after the read; its -/// standard metadata object does not expose the same identity and therefore contributes no -/// contradictory value here. fn metadata_identity_matches(observed: Option<&str>, expected: &str) -> bool { - observed.is_none_or(|identity| identity == expected) + observed == Some(expected) } pub fn inspect_photo(path: &Path) -> Result { @@ -587,4 +604,26 @@ mod tests { "path-identity" )); } + + #[test] + fn opened_handle_identity_remains_bound_when_path_is_replaced() { + let temp = tempfile::tempdir().unwrap(); + let current = temp.path().join("current.png"); + let replacement = temp.path().join("replacement.png"); + let held = temp.path().join("held-original.png"); + png(¤t, 8, 8, 1); + png(&replacement, 8, 8, 2); + + let expected_identity = crate::safety::filesystem_object_id(¤t).unwrap(); + let opened = std::fs::File::open(¤t).unwrap(); + std::fs::rename(¤t, &held).unwrap(); + std::fs::rename(&replacement, ¤t).unwrap(); + + assert_eq!(opened_file_object_id(&opened).unwrap(), expected_identity); + assert_ne!( + crate::safety::filesystem_object_id(¤t).unwrap(), + expected_identity, + "path replacement must not be mistaken for the already-open reviewed object" + ); + } } From 95eb36dc3d02c283892c5b239f7c36f26eb2c9d2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 18:26:29 -0700 Subject: [PATCH 13/25] test: exercise photo identity on Windows --- .github/workflows/test.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index df6186023..2c5dcdd0b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -47,7 +47,7 @@ jobs: windows-home-resolution: runs-on: windows-latest - timeout-minutes: 10 + timeout-minutes: 20 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -59,6 +59,8 @@ jobs: New-Item -ItemType Directory -Force target | Out-Null rustc --edition=2021 --test src-tauri/tests/home_resolution_contract.rs -o target/home-resolution-contract.exe & .\target\home-resolution-contract.exe + - name: Windows photo open-handle identity regression + run: cargo test --manifest-path src-tauri/Cargo.toml photo_duplicate::tests:: --lib llm-engine-build: runs-on: ubuntu-latest From 04ae2142dbcf31711e7b33337dbddff45a2f2894 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 29 Aug 2026 18:27:43 -0700 Subject: [PATCH 14/25] docs: align photo audit active-use contract --- CHANGELOG.md | 97 ++-------------------------------------------------- 1 file changed, 3 insertions(+), 94 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1af4a0c11..ca4482010 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,8 +12,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and current materialized PNGs only when dimensions and normalized decoded RGBA16 pixels match; choose a displayed keeper only under unique Pareto dominance across losslessness, source bit depth, metadata completeness, and lineage, while ties require customer selection and all cleanup - remains unavailable. Provider paths, Photos libraries, placeholders, symlinks, active files, and - replacement races remain rejected. + remains unavailable. Provider paths, Photos libraries, placeholders, symlinks, and replacement + races remain rejected during audit; active-use evidence is reserved for a fresh execution + preflight if cleanup is implemented later. - Keep coverage builds compile-safe by applying the same `not(coverage)` boundary to native-copy identity cleanup and dependent eviction helpers; the focused authority contract remains green. - Add durable private failure records in a separate journal directory and a receipt-bound @@ -33,95 +34,3 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and - Gate iCloud copy plans on a path-free three-stream evidence cohort with deterministic fingerprints and a five-minute observation-skew ceiling; incomplete, malformed, or stale observations remain blocked and never become cloud-write or eviction authority. -- Carry the integrity-checked iCloud pre-copy cohort and `pre_copy_evidence_met` through the - Naruon cloud-copy readiness envelope (schema version 8), so aggregate consumers also fail closed - when the provider queue is quiet but pre-copy evidence is absent. -- Keep the hourly contextual-orchestrator loop on its published read-only API, bind context to - the exact event commit, and remove foreign-repository checkout, KV mutation, and provider-secret - ingestion from DiskSage Actions. -- Keep the repository-local contextual-orchestrator advisory workflow manual-only and bind the - hourly OpenCode review/repair schedule to the trusted central `.github` scheduler, avoiding an - unpinned autonomous model reviewer while retaining exact-head, read-only evidence boundaries. -- Show the last read-only iCloud File Provider evidence timestamp beside the - new-copy admission state, so a stalled `no progress`/`hard expired` queue has - an actionable retry context without exposing provider paths. -- Bind Tauri packaging to a fail-closed cross-manifest release-version verifier so `package.json`, `Cargo.toml`, `tauri.conf.json`, and any `v*` release tag must agree on one valid Semantic Version before a bundle is built. -- Add retry-safe release concurrency: fresh first attempts may supersede stale runs, while explicit GitHub rerun attempts do not self-cancel inside the same concurrency group. -- Replace generator-era Cargo package metadata with the DiskSage product description, MIT license expression, canonical source repository URL, and `publish = false` registry-publication boundary; deliberately omit Cargo's deprecated `authors` field, verify publication refusal through Cargo's versioned parsed metadata rather than substring matching, and regression-test commented/out-of-table decoys together with the retained acquisition metadata and doctoring evidence. -- Require a fresh, exact, human-attributed approval and rationale for cloud copy-only and existing-copy adoption actions, with a 15-minute authorization lifetime bound to the candidate, destination, provider, account scope, and review fingerprint. -- Return the candidate-specific cloud copy approval action, exact confirmation phrase, and maximum approval age from the Rust plan contract; the frontend only displays and submits that backend-authored phrase and fails closed when it is missing or does not match the candidate action. -- Align the frontend toolchain on Vite 8.2 and `@sveltejs/vite-plugin-svelte` 7.2 so the declared peer dependency graph is installable and reproducible. -- Declare the supported Node.js runtime floor as Node.js 20.19 or Node.js 22.12 and later, matching Vite 8 requirements. -- Pin the primary test workflow to Node.js 20.19.0 so the minimum supported runtime is continuously verified. -- Document the iCloud batch operation's local-only versus path-free shareable evidence boundary and map its fail-closed controls to NIST SP 800-53 Release 5.2.0, ISO/IEC 27040:2024, and primary secure-design literature with APA 7th references and deterministic documentation contract tests. -- Refresh the Tauri CSP standards evidence to the current July 29, 2026 W3C Content Security Policy Level 3 Working Draft and regression-test its exact publication URL so future doctoring cannot silently drift back to an older draft. - -### Fixed - -- Reject ontology organize destinations that are relative to the process working directory, - named-user tilde paths, or parent-traversal paths; only an absolute destination or a home token - (`~`/`~/`, plus native Windows `~\`) can produce a move plan, and literal tildes in absolute - paths are preserved. -- Surface the bounded iCloud File Provider upload/download fractions and label repeated - `no progress` observations as a Finder “copy preparing” stall, so the operator can cancel the - pending Finder request before retrying; this remains diagnostic and never grants copy or eviction - authority. - -- Keep the shipped Naruon readiness verifier source includable by its integration boundary test; - the terminal parser contract now compiles in both the binary and test-module contexts. - -- Cover the `sensitive-config` archive-kind wire label in the generated cloud-plan implementation, - so the macOS/Linux/Windows cloud-plan binaries compile after the sensitive-config safety - boundary is enabled. -- Keep the local staging-headroom gate on new native copies only; existing-copy adoption now - remains available on low-disk volumes because it verifies an already-present destination without - creating local staging data. -- Surface insufficient local staging headroom as a dry-run notice and native-copy blocker before - review, while keeping the non-staging provider-API fallback and existing-copy adoption available. -- Make `disksage-duplicate-audit --help` exit successfully so release staging can - verify its usage contract without treating a help request as a failed audit. -- Publish the source-bound SPDX SBOM as a separately named artifact only after - provenance succeeds, then download it in the release publication job so the - attested 18-file release set cannot silently omit its component inventory. -- Isolate the macOS global File Provider dump helper in a private process group and terminate the - whole group on timeout, preventing descendant helpers from retaining a pipe after a stalled - Finder/provider copy. -- Classify repeated File Provider `-1005 itemNotFound` markers as a path-free global sync blocker, - retain the same-blocker duration when reconciliation counts change, and direct operators to - cancel a stalled Finder copy before retrying. -- Reject legacy provider evidence that reports `sync_complete=true` without an explicit complete - `sync_state` at the current authorization and eviction boundary, while retaining compatibility - reads and a public-boundary regression test. -- Replace the unmaintained direct `jwalk` production dependency with the maintained `walkdir` - backend across scanner, duplicate, artifact, cloud, and reclaim traversals; preserve symlink/ - reparse filtering and fail-closed traversal-error accounting with a locked dependency contract. -- Inventory direct credential-bearing configuration names as blocked `sensitive-config` entries; - never open them for metadata probing or include them in cloud-copy or source-eviction authority. -- Bound one-minute background reconciliation to 128 immutable provider evidence records per - receipt, and validate active iCloud File Provider transfers as blocked readiness evidence. -- Scope persisted API object-id recovery by receipt filename prefix before scanning, so unrelated - receipts in a shared evidence directory cannot hide a valid Google Drive or OneDrive locator. -- Add an explicit macOS Finder-copy cancellation command that sends only a bounded Escape request; - it never accepts scripts or paths and never terminates iCloud/File Provider services. -- Fail closed when OneDrive or Google Drive runtime evidence is unavailable during client recovery; - an unknown observation is no longer treated as proof that the provider process is absent. -- Hardened iCloud local-copy batch eviction with fresh per-item timestamps, deterministic planner/executor/recorder/clock seams, fail-closed immutable checkpoint handling, bounded manifest admission, symlink-safe control-path validation, and distinct operator diagnostics. -- Restored the cloud-copy public documentation regression contract after a temporary repair path removed it, so CI continues to fail when the new Rust or TypeScript approval surfaces lose beginner-readable documentation. - -### Security - -- Default personal cloud-provider OAuth consent to read-only; upload scope and API write - authority now require an explicit user opt-in. -- Catalog the Cargo registry source tree as an explicit, identity-bound regenerable-cache target; - keep it out of automatic cleanup because rebuilding may require network downloads. -- Catalog the observed Node.js, PyTorch, Prisma, and GitHub CLI cache trees as identity-bound - manual-review targets; keep them out of automatic cleanup until their active-use and rebuild - contracts are independently established. -- Add buyer-verifiable release artifact provenance with read-only platform build jobs, a tag-only least-privilege attestation job, exact 18-file admission including a source-bound SPDX SBOM, adjacent operational-CLI SHA-256 verification, preserved artifact namespaces, non-regular-entry rejection, and a separate publication job that cannot publish before attestation succeeds. -- Require explicit organization-tenant authority when either the destination account scope is organization-owned or the canonical organization-sensitive review reason is present; fail closed in both frontend projection and durable Rust transfer authorization even when the ordinary review flag is absent, and regression-test contradictory signal combinations. -- Enable an explicit fail-closed Tauri Content Security Policy to keep executable scripts and fonts local, grant production network authority only to the Tauri IPC transport, confine Vite WebSocket HMR to a separate development-only CSP, deny object/frame/base-URI authority, deny form submissions with explicit `form-action 'none'`, deny unused worker, media, and web-app-manifest fetch authority with explicit `'none'` directives, and regression-test against null, wildcard, remote-script/style, eval, and development-authority leakage. -- Re-verify the installed GGUF immediately before llama.cpp initialization and retain the verified model handle through llama.cpp loading: reject missing, linked, non-regular, identity-raced, short, oversized, unreadable, or SHA-256-mismatched artifacts with stable path-free errors; use a stable descriptor path on Unix and a Windows read-sharing guard so the mutable source pathname cannot be substituted between verification and model parsing. -- Bind the default on-device GGUF model to an immutable upstream revision, exact byte count, and SHA-256 digest; replace whole-model buffering and named sibling staging with bounded streaming into an unnamed same-directory temporary file; ignore and preserve unrelated legacy `.part` paths; refuse destination overwrite with create-new semantics; capture destination ownership from the returned open file handle; re-read and rehash the still-open staging source while copying; flush, sync, re-read, and rehash the destination before final acceptance; reject same-file source or destination mutation; preserve foreign destination replacements through identity-bound cleanup; and keep model installation inside the Rust coverage surface with privacy-safe stable errors and deterministic race regressions. -- Persist copy-approval provenance in immutable receipt lineage, reject stale, generic, mismatched, or tampered approvals, and retain explicit backward readability for pre-approval receipt formats. -- Generate the npm lockfile in an exact-head validation job with repository contents read-only and dependency lifecycle scripts disabled, bind the artifact to SHA-256 evidence, and grant `contents: write` only to a separate publication job that verifies the same-run artifact and unchanged branch head before committing the lockfile. -- Removed obsolete one-shot repair workflows and patch scripts so repository automation no longer retains dormant write-capable recovery paths. From d94057a734af9b7f4e56489639eecea5556c0e9b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:31:51 +0900 Subject: [PATCH 15/25] fix(ci): preserve main path filters and parse photo regression safely --- .github/workflows/test.yml | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 2c5dcdd0b..c22764e02 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -3,7 +3,29 @@ name: Test on: push: branches: [main] + paths-ignore: + - "docs/**" + - "*.md" + # Content-checked by contract tests (vitest + cargo test) — must still run CI. + - "!docs/doctoring/release-artifact-provenance.md" + - "!docs/doctoring/tauri-content-security-policy.md" + - "!docs/doctoring/model-artifact-integrity.md" + - "!docs/doctoring/model-load-handle-binding.md" + - "!docs/development/icloud-local-eviction-batch.md" + - "!docs/architecture/goals/cloud-offload-goal.json" + - "!CHANGELOG.md" pull_request: + paths-ignore: + - "docs/**" + - "*.md" + # Content-checked by contract tests (vitest + cargo test) — must still run CI. + - "!docs/doctoring/release-artifact-provenance.md" + - "!docs/doctoring/tauri-content-security-policy.md" + - "!docs/doctoring/model-artifact-integrity.md" + - "!docs/doctoring/model-load-handle-binding.md" + - "!docs/development/icloud-local-eviction-batch.md" + - "!docs/architecture/goals/cloud-offload-goal.json" + - "!CHANGELOG.md" permissions: contents: read @@ -60,7 +82,8 @@ jobs: rustc --edition=2021 --test src-tauri/tests/home_resolution_contract.rs -o target/home-resolution-contract.exe & .\target\home-resolution-contract.exe - name: Windows photo open-handle identity regression - run: cargo test --manifest-path src-tauri/Cargo.toml photo_duplicate::tests:: --lib + run: | + cargo test --manifest-path src-tauri/Cargo.toml photo_duplicate::tests:: --lib llm-engine-build: runs-on: ubuntu-latest From f03e6cf766957e48bc11a390c384b1124269be87 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:32:08 +0900 Subject: [PATCH 16/25] test(photo): make decode-bound fixtures exceed the budget --- src-tauri/tests/photo_duplicate_decode_bound_contract.rs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src-tauri/tests/photo_duplicate_decode_bound_contract.rs b/src-tauri/tests/photo_duplicate_decode_bound_contract.rs index f8a05a6e3..6fb27391c 100644 --- a/src-tauri/tests/photo_duplicate_decode_bound_contract.rs +++ b/src-tauri/tests/photo_duplicate_decode_bound_contract.rs @@ -48,7 +48,8 @@ fn write_declared_grayscale_png(path: &Path, width: u32, height: u32) { fn declared_dimension_limit_is_checked_before_decode_allocation() { let temp = tempfile::tempdir().unwrap(); let path = temp.path().join("too-wide.png"); - write_declared_grayscale_png(&path, 20_000, 1); + // 67,108,865 RGBA16-normalized pixels require 512 MiB + 8 bytes. + write_declared_grayscale_png(&path, 67_108_865, 1); assert_eq!( inspect_photo(&path).unwrap_err(), "photo-decoded-size-unsupported" @@ -59,7 +60,8 @@ fn declared_dimension_limit_is_checked_before_decode_allocation() { fn normalized_pixel_budget_is_checked_before_decode_allocation() { let temp = tempfile::tempdir().unwrap(); let path = temp.path().join("pixel-bomb.png"); - write_declared_grayscale_png(&path, 8_192, 8_192); + // 8,193 × 8,192 pixels exceed the 512 MiB RGBA16-normalized budget. + write_declared_grayscale_png(&path, 8_193, 8_192); assert_eq!( inspect_photo(&path).unwrap_err(), "photo-decoded-size-unsupported" From cf20a1dc1de681e55461fdda0001b411297b06a9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:32:19 +0900 Subject: [PATCH 17/25] test(photo): align APNG rejection contract --- src-tauri/tests/photo_duplicate_input_contract.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src-tauri/tests/photo_duplicate_input_contract.rs b/src-tauri/tests/photo_duplicate_input_contract.rs index a8c2046a5..f8761f277 100644 --- a/src-tauri/tests/photo_duplicate_input_contract.rs +++ b/src-tauri/tests/photo_duplicate_input_contract.rs @@ -23,7 +23,7 @@ fn animated_png_is_not_misreported_as_first_frame_exact_evidence() { write_apng(&path); assert_eq!( inspect_photo(&path).unwrap_err(), - "photo-animation-unsupported" + "photo-animated-png-unsupported" ); } From d761f75b1b61e0dad1ea6efaac52c1c2eb1f4e2f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:34:44 +0900 Subject: [PATCH 18/25] fix(photo): fail closed when audit evidence is incomplete --- src-tauri/src/bin/disksage-photo-duplicate-audit.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src-tauri/src/bin/disksage-photo-duplicate-audit.rs b/src-tauri/src/bin/disksage-photo-duplicate-audit.rs index b160c9e95..1c42645a5 100644 --- a/src-tauri/src/bin/disksage-photo-duplicate-audit.rs +++ b/src-tauri/src/bin/disksage-photo-duplicate-audit.rs @@ -12,4 +12,7 @@ fn main() { .unwrap_or_default(); let audit = disksage_lib::photo_duplicate::audit_photos(&paths, now); println!("{}", serde_json::to_string_pretty(&audit).unwrap()); + if !audit.evidence_complete || audit.inspected_input_count == 0 { + std::process::exit(3); + } } From 1e6918ab6b92024ac530ea77d26d5b9b3b15b2c2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:34:51 +0900 Subject: [PATCH 19/25] test(photo): require nonzero exit for incomplete audit evidence --- .../photo_duplicate_cli_exit_contract.rs | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 src-tauri/tests/photo_duplicate_cli_exit_contract.rs diff --git a/src-tauri/tests/photo_duplicate_cli_exit_contract.rs b/src-tauri/tests/photo_duplicate_cli_exit_contract.rs new file mode 100644 index 000000000..c8de9f6aa --- /dev/null +++ b/src-tauri/tests/photo_duplicate_cli_exit_contract.rs @@ -0,0 +1,20 @@ +use std::process::Command; + +#[test] +fn rejected_only_audit_returns_nonzero_after_emitting_json_evidence() { + let temp = tempfile::tempdir().unwrap(); + let missing = temp.path().join("missing.png"); + let output = Command::new(env!("CARGO_BIN_EXE_disksage-photo-duplicate-audit")) + .arg(&missing) + .output() + .unwrap(); + + assert_eq!(output.status.code(), Some(3)); + let report: serde_json::Value = serde_json::from_slice(&output.stdout).unwrap(); + assert_eq!(report["inspected_input_count"], 0); + assert_eq!(report["evidence_complete"], false); + assert_eq!( + report["rejected_input_counts"]["photo-input-metadata-unavailable"], + 1 + ); +} From 69c2a2cb55aeb0a4175d1ab5e27786122c555f4e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:37:33 +0900 Subject: [PATCH 20/25] fix(photo): bind bounded evidence to managed-path and raster contracts --- src-tauri/src/photo_duplicate.rs | 167 +++++++++++++++++++++++++------ 1 file changed, 135 insertions(+), 32 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index 8459be9d3..d05c0906e 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -1,17 +1,21 @@ //! Evidence-bound photo duplicate audit. //! //! The current product boundary is intentionally conservative: exact decoded-pixel grouping is -//! available for bounded PNG inputs, while perceptual grouping and permanent cleanup remain +//! available for bounded local PNG inputs, while perceptual grouping and permanent cleanup remain //! unavailable until calibrated evidence and a unique keeper decision exist. use serde::Serialize; use std::io::Read; use std::path::{Path, PathBuf}; +const MAX_ENCODED_IMAGE_BYTES: u64 = 64 * 1024 * 1024; const MAX_DECODED_IMAGE_BYTES: u64 = 256 * 1024 * 1024; const MAX_NORMALIZED_IMAGE_BYTES: u64 = 512 * 1024 * 1024; +const MAX_AUDIT_INPUTS: usize = 4_096; +const MAX_AUDIT_DECLARED_BYTES: u64 = 16 * 1024 * 1024 * 1024; #[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd, Serialize)] +#[serde(rename_all = "snake_case")] pub enum EvidenceState { Unavailable, Observed, @@ -61,6 +65,16 @@ pub struct PhotoDuplicateAudit { pub filesystem_mutation_executed: bool, } +fn bit_depth_value(depth: png::BitDepth) -> u8 { + match depth { + png::BitDepth::One => 1, + png::BitDepth::Two => 2, + png::BitDepth::Four => 4, + png::BitDepth::Eight => 8, + png::BitDepth::Sixteen => 16, + } +} + fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'static str> { if metadata.file_type().is_symlink() { return Some("photo-input-symlink-rejected"); @@ -68,28 +82,33 @@ fn admission_blocker(path: &Path, metadata: &std::fs::Metadata) -> Option<&'stat if !metadata.is_file() { return Some("photo-input-not-regular-file"); } - let normalized = path.to_string_lossy().replace('\\', "/").to_lowercase(); - if normalized.contains("/photos library.photoslibrary/") { + if crate::cloud::metadata_is_dataless(metadata) { + return Some("photo-input-dataless"); + } + if crate::cloud::path_inside_managed_photo_library(path) { return Some("photo-input-managed-library"); } - if normalized.contains("/library/cloudstorage/") - || normalized.contains("/onedrive") - || normalized.contains("/dropbox") - || normalized.contains("/google drive") - { + if crate::cloud::path_inside_managed_file_provider_storage(path) { return Some("photo-input-provider-managed"); } + if metadata.len() > MAX_ENCODED_IMAGE_BYTES { + return Some("photo-encoded-size-unsupported"); + } None } fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String> { - let decoder = png::Decoder::new(std::io::Cursor::new(bytes)); + let mut decoder = png::Decoder::new(std::io::Cursor::new(bytes)); + // EXPAND makes palette/low-depth samples and tRNS transparency explicit before semantic + // hashing. The source bit depth remains separately recorded as keeper evidence. + decoder.set_transformations(png::Transformations::EXPAND); let mut reader = decoder .read_info() .map_err(|_| "photo-codec-decode-failed".to_string())?; if reader.info().is_animated() { return Err("photo-animated-png-unsupported".into()); } + let source_bit_depth = bit_depth_value(reader.info().bit_depth); let width = reader.info().width; let height = reader.info().height; let pixel_count = u64::from(width) @@ -111,13 +130,6 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String let output = reader .next_frame(&mut decoded) .map_err(|_| "photo-codec-decode-failed".to_string())?; - let bit_depth = match output.bit_depth { - png::BitDepth::One => 1, - png::BitDepth::Two => 2, - png::BitDepth::Four => 4, - png::BitDepth::Eight => 8, - png::BitDepth::Sixteen => 16, - }; let normalized = normalize_rgba16( &decoded[..output.buffer_size()], output.color_type, @@ -130,12 +142,17 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String + u32::try_from(reader.info().uncompressed_latin1_text.len()).unwrap_or(u32::MAX) + u32::try_from(reader.info().compressed_latin1_text.len()).unwrap_or(u32::MAX) + u32::try_from(reader.info().utf8_text.len()).unwrap_or(u32::MAX); + let mut semantic = blake3::Hasher::new(); + semantic.update(b"disksage-png-rgba16-raster-v2\0"); + semantic.update(&output.width.to_be_bytes()); + semantic.update(&output.height.to_be_bytes()); + semantic.update(&normalized); Ok(( output.width, output.height, - bit_depth, + source_bit_depth, metadata_field_count, - blake3::hash(&normalized).to_hex().to_string(), + semantic.finalize().to_hex().to_string(), )) } @@ -235,25 +252,36 @@ fn hash_current_file( expected: &std::fs::Metadata, expected_identity: &str, ) -> Result<(String, Vec), String> { + if expected.len() > MAX_ENCODED_IMAGE_BYTES { + return Err("photo-encoded-size-unsupported".into()); + } let mut file = std::fs::File::open(path).map_err(|_| "photo-input-open-failed".to_string())?; let opened = file .metadata() .map_err(|_| "photo-input-metadata-unavailable".to_string())?; let opened_identity = opened_file_object_id(&file)?; + if opened.len() > MAX_ENCODED_IMAGE_BYTES { + return Err("photo-encoded-size-unsupported".into()); + } if opened.len() != expected.len() || opened.modified().ok() != expected.modified().ok() || !metadata_identity_matches(Some(opened_identity.as_str()), expected_identity) { return Err("photo-input-changed".into()); } - let mut hasher = blake3::Hasher::new(); - let mut bytes = Vec::with_capacity(expected.len() as usize); - file.read_to_end(&mut bytes) + let capacity = usize::try_from(expected.len()).unwrap_or_default(); + let mut bytes = Vec::with_capacity(capacity); + let mut bounded = (&mut file).take(MAX_ENCODED_IMAGE_BYTES + 1); + bounded + .read_to_end(&mut bytes) .map_err(|_| "photo-input-read-failed".to_string())?; + if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > MAX_ENCODED_IMAGE_BYTES { + return Err("photo-encoded-size-unsupported".into()); + } if bytes.len() as u64 != expected.len() { return Err("photo-input-changed".into()); } - hasher.update(&bytes); + let blake3 = blake3::hash(&bytes).to_hex().to_string(); let after = std::fs::symlink_metadata(path).map_err(|_| "photo-input-changed".to_string())?; if after.len() != expected.len() || after.modified().ok() != expected.modified().ok() @@ -261,7 +289,7 @@ fn hash_current_file( { return Err("photo-input-changed".into()); } - Ok((hasher.finalize().to_hex().to_string(), bytes)) + Ok((blake3, bytes)) } fn metadata_identity_matches(observed: Option<&str>, expected: &str) -> bool { @@ -279,8 +307,8 @@ pub fn inspect_photo(path: &Path) -> Result { } let identity = crate::safety::filesystem_object_id(path) .map_err(|_| "photo-input-identity-unavailable".to_string())?; - // Audit is read-only. Active-use evidence belongs to the fresh execution preflight; requiring - // Unix `lsof` here made otherwise valid Windows evidence impossible to collect. + // Audit is read-only. Active-use evidence belongs to a fresh execution preflight immediately + // before any future mutation; requiring it here made valid cross-platform evidence unavailable. let extension = path .extension() .and_then(|value| value.to_str()) @@ -328,7 +356,25 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu let mut by_digest = std::collections::BTreeMap::>::new(); let mut rejected_input_counts = std::collections::BTreeMap::::new(); let mut inspected_input_count = 0_u64; - for path in paths { + let mut declared_bytes = 0_u64; + + for (index, path) in paths.iter().enumerate() { + if index >= MAX_AUDIT_INPUTS { + *rejected_input_counts + .entry("photo-audit-input-limit-exceeded".into()) + .or_default() += 1; + continue; + } + if let Ok(metadata) = std::fs::symlink_metadata(path) { + let next_declared = declared_bytes.checked_add(metadata.len()); + if next_declared.is_none_or(|value| value > MAX_AUDIT_DECLARED_BYTES) { + *rejected_input_counts + .entry("photo-audit-byte-budget-exceeded".into()) + .or_default() += 1; + continue; + } + declared_bytes = next_declared.unwrap_or(declared_bytes); + } match inspect_photo(path) { Ok(evidence) => { inspected_input_count += 1; @@ -340,6 +386,7 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu Err(reason) => *rejected_input_counts.entry(reason).or_default() += 1, } } + let exact_groups = by_digest .into_iter() .filter_map(|(content_digest, mut members)| { @@ -352,7 +399,7 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu let keeper_path = unique_pareto_keeper(&members).map(|member| member.path.clone()); Some(ExactPhotoGroup { content_digest, - grouping_basis: "decoded-pixel-rgba16-exact".into(), + grouping_basis: "decoded-pixel-rgba16-raster-exact-v2".into(), members, keeper_path: keeper_path.clone(), keeper_blocker: keeper_path @@ -363,7 +410,7 @@ pub fn audit_photos(paths: &[PathBuf], generated_at_ms: u64) -> PhotoDuplicateAu }) .collect(); PhotoDuplicateAudit { - schema_kind: "disksage.photo-duplicate-audit.v1".into(), + schema_kind: "disksage.photo-duplicate-audit.v2".into(), generated_at_ms, exact_groups, inspected_input_count, @@ -454,6 +501,18 @@ mod tests { writer.write_image_data(&bytes).unwrap(); } + fn png_with_trns(path: &Path, value: u8, transparent: bool) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, 4, 4); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + if transparent { + encoder.set_trns(vec![0, value]); + } + let mut writer = encoder.write_header().unwrap(); + writer.write_image_data(&vec![value; 16]).unwrap(); + } + #[test] fn exact_duplicates_are_grouped_without_inventing_a_keeper() { let temp = tempfile::tempdir().unwrap(); @@ -467,6 +526,29 @@ mod tests { assert!(!audit.exact_groups[0].execution_available); assert!(!audit.perceptual_grouping_available); assert!(!audit.permanent_delete_available); + assert_eq!(audit.schema_kind, "disksage.photo-duplicate-audit.v2"); + } + + #[test] + fn raster_dimensions_are_part_of_exact_identity() { + let temp = tempfile::tempdir().unwrap(); + let row = temp.path().join("row.png"); + let square = temp.path().join("square.png"); + png(&row, 4, 1, 120); + png(&square, 2, 2, 120); + let audit = audit_photos(&[row, square], 7); + assert!(audit.exact_groups.is_empty()); + } + + #[test] + fn trns_transparency_is_part_of_normalized_pixel_identity() { + let temp = tempfile::tempdir().unwrap(); + let opaque = temp.path().join("opaque.png"); + let transparent = temp.path().join("transparent.png"); + png_with_trns(&opaque, 5, false); + png_with_trns(&transparent, 5, true); + let audit = audit_photos(&[opaque, transparent], 7); + assert!(audit.exact_groups.is_empty()); } #[test] @@ -495,7 +577,7 @@ mod tests { assert_eq!(audit.exact_groups.len(), 1); assert_eq!( audit.exact_groups[0].grouping_basis, - "decoded-pixel-rgba16-exact" + "decoded-pixel-rgba16-raster-exact-v2" ); assert_eq!( audit.exact_groups[0].keeper_path.as_deref(), @@ -505,7 +587,7 @@ mod tests { } #[test] - fn higher_bit_depth_is_the_unique_pareto_keeper_for_identical_samples() { + fn higher_source_bit_depth_is_the_unique_pareto_keeper_for_identical_samples() { let temp = tempfile::tempdir().unwrap(); let eight = temp.path().join("eight.png"); let sixteen = temp.path().join("sixteen.png"); @@ -536,14 +618,14 @@ mod tests { } #[test] - fn provider_and_photos_library_paths_fail_closed() { + fn shared_managed_storage_classifiers_fail_closed_without_brand_substrings() { let temp = tempfile::tempdir().unwrap(); let provider = temp .path() .join("Library/CloudStorage/OneDrive-Personal/image.png"); let library = temp .path() - .join("Pictures/Photos Library.photoslibrary/originals/image.png"); + .join("Pictures/Custom.photoslibrary/originals/image.png"); std::fs::create_dir_all(provider.parent().unwrap()).unwrap(); std::fs::create_dir_all(library.parent().unwrap()).unwrap(); png(&provider, 8, 8, 1); @@ -558,6 +640,27 @@ mod tests { ); } + #[test] + fn provider_brand_in_an_ordinary_local_component_is_not_a_blocker() { + let temp = tempfile::tempdir().unwrap(); + let local = temp.path().join("dropbox-exports/image.png"); + std::fs::create_dir_all(local.parent().unwrap()).unwrap(); + png(&local, 8, 8, 1); + assert!(inspect_photo(&local).is_ok()); + } + + #[test] + fn encoded_size_is_rejected_before_content_allocation() { + let temp = tempfile::tempdir().unwrap(); + let oversized = temp.path().join("oversized.png"); + let file = std::fs::File::create(&oversized).unwrap(); + file.set_len(MAX_ENCODED_IMAGE_BYTES + 1).unwrap(); + assert_eq!( + inspect_photo(&oversized).unwrap_err(), + "photo-encoded-size-unsupported" + ); + } + #[test] fn execution_remains_unavailable_without_unique_keeper() { let audit = audit_photos(&[], 1); From 55aaf730753d2f4342fcbe231d771a9ae3854e32 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:38:17 +0900 Subject: [PATCH 21/25] docs(adr): keep photo audit decision proposed until integration --- ...cate-evidence-without-composite-scoring.md | 74 ++++++++++++------- 1 file changed, 48 insertions(+), 26 deletions(-) diff --git a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md index ff94c1aa2..184c09512 100644 --- a/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md +++ b/docs/architecture/adr/0012-photo-duplicate-evidence-without-composite-scoring.md @@ -1,6 +1,6 @@ # ADR 0012: Separate photo-duplicate and keeper evidence without composite scoring -- Status: Accepted +- Status: Proposed - Date: 2026-08-29 ## Context @@ -13,42 +13,59 @@ distance requires a descriptor-specific, population-calibrated decision threshol requires the trained model used by the published method. Photos libraries and File Provider trees have additional ownership and materialization semantics. -Reading a package internals or a dataless placeholder can trigger provider activity and cannot -authorize cleanup. +Reading package internals or a dataless placeholder can trigger provider activity and cannot +authorize cleanup. Encoded files and aggregate audit batches also need explicit work budgets before +any content allocation so hostile inputs cannot convert an audit into an availability failure. ## Decision -DiskSage v1 records byte identity with BLAKE3 and groups currently materialized PNG files only -when dimensions and normalized decoded RGBA16 pixels have the same domain-separated digest. This -permits semantically identical lossless encodings with different compression or ancillary metadata -to share a group without a perceptual-distance threshold. Before and -after hashing, it verifies the filesystem-object identity and size, and it rejects symlinks, -provider-managed paths, Photos library packages, dataless objects, and unsupported codecs. -Active-use evidence is collected fail-closed only by a fresh execution preflight; read-only audit -does not turn platform-specific process inspection into a discovery prerequisite. - -The audit reports dimensions, bit depth, losslessness, metadata-field count, lineage availability, -no-reference IQA availability, and perceptual-descriptor availability as separate evidence. It -does not calculate a composite score. Perceptual grouping remains unavailable until a versioned -descriptor and dataset-calibrated threshold artifact include provenance and a cryptographic -checksum. BRISQUE remains unavailable until its exact trained model artifact has equivalent -provenance and checksum. +DiskSage records byte identity with BLAKE3 and groups currently materialized PNG files only when +raster dimensions and normalized decoded RGBA16 pixels have the same domain-separated digest. +The decoder expands palette/low-depth samples and `tRNS` transparency before normalization, while +source bit depth remains separate keeper evidence. This permits semantically identical lossless +encodings with different compression or ancillary metadata to share a group without a +perceptual-distance threshold, while preventing different raster shapes or transparency semantics +from collapsing into one exact identity. + +Before content allocation, the audit rejects symlinks, provider-managed paths, Photos library +packages, dataless objects, unsupported codecs, and encoded inputs above the fixed per-file budget. +The already-open handle is read through the same byte ceiling and filesystem-object identity, +length, and modification evidence are rechecked around hashing. Aggregate input count and declared +bytes are also bounded. Active-use evidence is collected fail-closed only by a fresh execution +preflight; read-only audit does not turn platform-specific process inspection into a discovery +prerequisite. + +The public wire contract is `disksage.photo-duplicate-audit.v2`. Version 2 makes the evidence-state +serialization and keeper metadata semantics explicit and changes exact grouping to +`decoded-pixel-rgba16-raster-exact-v2`; consumers must not interpret it as the earlier draft v1 +shape. + +The audit reports dimensions, source bit depth, losslessness, metadata-field count, lineage +availability, no-reference IQA availability, and perceptual-descriptor availability as separate +evidence. It does not calculate a composite score. Perceptual grouping remains unavailable until a +versioned descriptor and dataset-calibrated threshold artifact include provenance and a +cryptographic checksum. BRISQUE remains unavailable until its exact trained model artifact has +equivalent provenance and checksum. Keeper evidence uses Pareto dominance only: losslessness, source bit depth, metadata completeness, and original/edit lineage must all be no worse and at least one must be better for exactly one member. File size, modification time, and filename have no quality authority. Ties and incomparable members require customer selection; byte-identical members therefore never receive an arbitrary -automatic keeper. V1 may display a unique Pareto keeper but does not mutate files. Cleanup execution -and permanent deletion remain unavailable. -A later execution decision must bind an exact group identity, exact unique keeper identity, fresh -approval, current inactive/materialized evidence, reversible Trash or quarantine, and a durable -journal with undo. +automatic keeper. The audit may display a unique Pareto keeper but does not mutate files. Cleanup +execution and permanent deletion remain unavailable. A later execution decision must bind an exact +group identity, exact unique keeper identity, fresh approval, current inactive/materialized +evidence, reversible Trash or quarantine, and a durable journal with undo. + +This ADR remains Proposed until the exact PR head passes the applicable Test/Release/Security/SAST +checks and all valid review findings are resolved. Merge is the acceptance boundary. ## Consequences -Customers can establish exact duplicate evidence without risking Photos or cloud-provider data. -They are told why near-duplicate grouping and cleanup are unavailable and what evidence must be -installed next. This initial capability deliberately recovers no bytes by itself. +Customers can establish bounded exact-duplicate evidence without intentionally hydrating Photos or +cloud-provider data. They are told why near-duplicate grouping and cleanup are unavailable and what +evidence must be installed next. This initial capability deliberately recovers no bytes by itself. +Automation receives a non-zero CLI exit when the supplied audit is incomplete while retaining the +structured JSON rejection evidence. ## Rejected alternatives @@ -58,6 +75,11 @@ installed next. This initial capability deliberately recovers no bytes by itself the operating image population. - Selecting the largest image automatically: rejected because dimensions alone do not establish fidelity, originality, or perceptual quality. +- Substring matching for cloud-provider brands: rejected because it both misses canonical managed + roots and rejects ordinary local names; DiskSage reuses the filesystem/platform classifiers + owned by the cloud bounded context. +- Unbounded `read_to_end`: rejected because decode limits do not protect the preceding encoded-file + allocation. - Direct Photos library mutation or permanent deletion: rejected because it bypasses provider and reversible-recovery contracts. From c4c63693eb9136285e0624d1563fc1ab4a0ca3cb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:38:31 +0900 Subject: [PATCH 22/25] docs(adr): mark photo audit decision proposed --- docs/architecture/adr/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr/README.md b/docs/architecture/adr/README.md index 50849f886..2a1583abf 100644 --- a/docs/architecture/adr/README.md +++ b/docs/architecture/adr/README.md @@ -17,7 +17,7 @@ new numbered record rather than rewriting history. | [0009](0009-path-free-lineage-relation-graph.md) | Export a path-free lineage relation graph | Accepted | | [0010](0010-rooted-organize-destinations.md) | Require rooted, process-independent organize destinations | Accepted | | [0011](0011-cloud-transfer-failure-and-materialization.md) | Durable failed-copy evidence and placeholder-safe adoption | Accepted | -| [0012](0012-photo-duplicate-evidence-without-composite-scoring.md) | Separate photo-duplicate and keeper evidence without composite scoring | Accepted | +| [0012](0012-photo-duplicate-evidence-without-composite-scoring.md) | Separate photo-duplicate and keeper evidence without composite scoring | Proposed | New records must state context, decision, consequences, rejected alternatives, and the evidence or standard that led to the decision. A record never grants cloud-write or source-eviction authority; From 9190c8798fb7a63451ad8670d86f4c135120a4a5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 01:41:35 +0900 Subject: [PATCH 23/25] fix(changelog): preserve protected-main release history --- CHANGELOG.md | 106 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca4482010..b35794ab0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,13 +8,13 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and ### Changed -- Add a read-only exact-photo duplicate audit that records byte identity with BLAKE3 and groups - current materialized PNGs only when dimensions and normalized decoded RGBA16 pixels match; - choose a displayed keeper only under unique Pareto dominance across losslessness, source bit - depth, metadata completeness, and lineage, while ties require customer selection and all cleanup - remains unavailable. Provider paths, Photos libraries, placeholders, symlinks, and replacement - races remain rejected during audit; active-use evidence is reserved for a fresh execution - preflight if cleanup is implemented later. +- Add a bounded, read-only exact-photo duplicate audit that records byte identity with BLAKE3 and + groups local PNGs only when raster dimensions and normalized decoded RGBA16 pixels share the + version-two domain-separated digest. Provider-managed paths, Photos libraries, dataless + placeholders, symlinks, oversized encoded inputs, and replacement races fail closed; incomplete + audits return a non-zero CLI exit after emitting structured rejection evidence. Cleanup and + permanent deletion remain unavailable, and active-use evidence is reserved for a fresh future + execution preflight. - Keep coverage builds compile-safe by applying the same `not(coverage)` boundary to native-copy identity cleanup and dependent eviction helpers; the focused authority contract remains green. - Add durable private failure records in a separate journal directory and a receipt-bound @@ -34,3 +34,95 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and - Gate iCloud copy plans on a path-free three-stream evidence cohort with deterministic fingerprints and a five-minute observation-skew ceiling; incomplete, malformed, or stale observations remain blocked and never become cloud-write or eviction authority. +- Carry the integrity-checked iCloud pre-copy cohort and `pre_copy_evidence_met` through the + Naruon cloud-copy readiness envelope (schema version 8), so aggregate consumers also fail closed + when the provider queue is quiet but pre-copy evidence is absent. +- Keep the hourly contextual-orchestrator loop on its published read-only API, bind context to + the exact event commit, and remove foreign-repository checkout, KV mutation, and provider-secret + ingestion from DiskSage Actions. +- Keep the repository-local contextual-orchestrator advisory workflow manual-only and bind the + hourly OpenCode review/repair schedule to the trusted central `.github` scheduler, avoiding an + unpinned autonomous model reviewer while retaining exact-head, read-only evidence boundaries. +- Show the last read-only iCloud File Provider evidence timestamp beside the + new-copy admission state, so a stalled `no progress`/`hard expired` queue has + an actionable retry context without exposing provider paths. +- Bind Tauri packaging to a fail-closed cross-manifest release-version verifier so `package.json`, `Cargo.toml`, `tauri.conf.json`, and any `v*` release tag must agree on one valid Semantic Version before a bundle is built. +- Add retry-safe release concurrency: fresh first attempts may supersede stale runs, while explicit GitHub rerun attempts do not self-cancel inside the same concurrency group. +- Replace generator-era Cargo package metadata with the DiskSage product description, MIT license expression, canonical source repository URL, and `publish = false` registry-publication boundary; deliberately omit Cargo's deprecated `authors` field, verify publication refusal through Cargo's versioned parsed metadata rather than substring matching, and regression-test commented/out-of-table decoys together with the retained acquisition metadata and doctoring evidence. +- Require a fresh, exact, human-attributed approval and rationale for cloud copy-only and existing-copy adoption actions, with a 15-minute authorization lifetime bound to the candidate, destination, provider, account scope, and review fingerprint. +- Return the candidate-specific cloud copy approval action, exact confirmation phrase, and maximum approval age from the Rust plan contract; the frontend only displays and submits that backend-authored phrase and fails closed when it is missing or does not match the candidate action. +- Align the frontend toolchain on Vite 8.2 and `@sveltejs/vite-plugin-svelte` 7.2 so the declared peer dependency graph is installable and reproducible. +- Declare the supported Node.js runtime floor as Node.js 20.19 or Node.js 22.12 and later, matching Vite 8 requirements. +- Pin the primary test workflow to Node.js 20.19.0 so the minimum supported runtime is continuously verified. +- Document the iCloud batch operation's local-only versus path-free shareable evidence boundary and map its fail-closed controls to NIST SP 800-53 Release 5.2.0, ISO/IEC 27040:2024, and primary secure-design literature with APA 7th references and deterministic documentation contract tests. +- Refresh the Tauri CSP standards evidence to the current July 29, 2026 W3C Content Security Policy Level 3 Working Draft and regression-test its exact publication URL so future doctoring cannot silently drift back to an older draft. + +### Fixed + +- Reject ontology organize destinations that are relative to the process working directory, + named-user tilde paths, or parent-traversal paths; only an absolute destination or a home token + (`~`/`~/`, plus native Windows `~\`) can produce a move plan, and literal tildes in absolute + paths are preserved. +- Surface the bounded iCloud File Provider upload/download fractions and label repeated + `no progress` observations as a Finder “copy preparing” stall, so the operator can cancel the + pending Finder request before retrying; this remains diagnostic and never grants copy or eviction + authority. + +- Keep the shipped Naruon readiness verifier source includable by its integration boundary test; + the terminal parser contract now compiles in both the binary and test-module contexts. + +- Cover the `sensitive-config` archive-kind wire label in the generated cloud-plan implementation, + so the macOS/Linux/Windows cloud-plan binaries compile after the sensitive-config safety + boundary is enabled. +- Keep the local staging-headroom gate on new native copies only; existing-copy adoption now + remains available on low-disk volumes because it verifies an already-present destination without + creating local staging data. +- Surface insufficient local staging headroom as a dry-run notice and native-copy blocker before + review, while keeping the non-staging provider-API fallback and existing-copy adoption available. +- Make `disksage-duplicate-audit --help` exit successfully so release staging can + verify its usage contract without treating a help request as a failed audit. +- Publish the source-bound SPDX SBOM as a separately named artifact only after + provenance succeeds, then download it in the release publication job so the + attested 18-file release set cannot silently omit its component inventory. +- Isolate the macOS global File Provider dump helper in a private process group and terminate the + whole group on timeout, preventing descendant helpers from retaining a pipe after a stalled + Finder/provider copy. +- Classify repeated File Provider `-1005 itemNotFound` markers as a path-free global sync blocker, + retain the same-blocker duration when reconciliation counts change, and direct operators to + cancel a stalled Finder copy before retrying. +- Reject legacy provider evidence that reports `sync_complete=true` without an explicit complete + `sync_state` at the current authorization and eviction boundary, while retaining compatibility + reads and a public-boundary regression test. +- Replace the unmaintained direct `jwalk` production dependency with the maintained `walkdir` + backend across scanner, duplicate, artifact, cloud, and reclaim traversals; preserve symlink/ + reparse filtering and fail-closed traversal-error accounting with a locked dependency contract. +- Inventory direct credential-bearing configuration names as blocked `sensitive-config` entries; + never open them for metadata probing or include them in cloud-copy or source-eviction authority. +- Bound one-minute background reconciliation to 128 immutable provider evidence records per + receipt, and validate active iCloud File Provider transfers as blocked readiness evidence. +- Scope persisted API object-id recovery by receipt filename prefix before scanning, so unrelated + receipts in a shared evidence directory cannot hide a valid Google Drive or OneDrive locator. +- Add an explicit macOS Finder-copy cancellation command that sends only a bounded Escape request; + it never accepts scripts or paths and never terminates iCloud/File Provider services. +- Fail closed when OneDrive or Google Drive runtime evidence is unavailable during client recovery; + an unknown observation is no longer treated as proof that the provider process is absent. +- Hardened iCloud local-copy batch eviction with fresh per-item timestamps, deterministic planner/executor/recorder/clock seams, fail-closed immutable checkpoint handling, bounded manifest admission, symlink-safe control-path validation, and distinct operator diagnostics. +- Restored the cloud-copy public documentation regression contract after a temporary repair path removed it, so CI continues to fail when the new Rust or TypeScript approval surfaces lose beginner-readable documentation. + +### Security + +- Default personal cloud-provider OAuth consent to read-only; upload scope and API write + authority now require an explicit user opt-in. +- Catalog the Cargo registry source tree as an explicit, identity-bound regenerable-cache target; + keep it out of automatic cleanup because rebuilding may require network downloads. +- Catalog the observed Node.js, PyTorch, Prisma, and GitHub CLI cache trees as identity-bound + manual-review targets; keep them out of automatic cleanup until their active-use and rebuild + contracts are independently established. +- Add buyer-verifiable release artifact provenance with read-only platform build jobs, a tag-only least-privilege attestation job, exact 18-file admission including a source-bound SPDX SBOM, adjacent operational-CLI SHA-256 verification, preserved artifact namespaces, non-regular-entry rejection, and a separate publication job that cannot publish before attestation succeeds. +- Require explicit organization-tenant authority when either the destination account scope is organization-owned or the canonical organization-sensitive review reason is present; fail closed in both frontend projection and durable Rust transfer authorization even when the ordinary review flag is absent, and regression-test contradictory signal combinations. +- Enable an explicit fail-closed Tauri Content Security Policy to keep executable scripts and fonts local, grant production network authority only to the Tauri IPC transport, confine Vite WebSocket HMR to a separate development-only CSP, deny object/frame/base-URI authority, deny form submissions with explicit `form-action 'none'`, deny unused worker, media, and web-app-manifest fetch authority with explicit `'none'` directives, and regression-test against null, wildcard, remote-script/style, eval, and development-authority leakage. +- Re-verify the installed GGUF immediately before llama.cpp initialization and retain the verified model handle through llama.cpp loading: reject missing, linked, non-regular, identity-raced, short, oversized, unreadable, or SHA-256-mismatched artifacts with stable path-free errors; use a stable descriptor path on Unix and a Windows read-sharing guard so the mutable source pathname cannot be substituted between verification and model parsing. +- Bind the default on-device GGUF model to an immutable upstream revision, exact byte count, and SHA-256 digest; replace whole-model buffering and named sibling staging with bounded streaming into an unnamed same-directory temporary file; ignore and preserve unrelated legacy `.part` paths; refuse destination overwrite with create-new semantics; capture destination ownership from the returned open file handle; re-read and rehash the still-open staging source while copying; flush, sync, re-read, and rehash the destination before final acceptance; reject same-file source or destination mutation; preserve foreign destination replacements through identity-bound cleanup; and keep model installation inside the Rust coverage surface with privacy-safe stable errors and deterministic race regressions. +- Persist copy-approval provenance in immutable receipt lineage, reject stale, generic, mismatched, or tampered approvals, and retain explicit backward readability for pre-approval receipt formats. +- Generate the npm lockfile in an exact-head validation job with repository contents read-only and dependency lifecycle scripts disabled, bind the artifact to SHA-256 evidence, and grant `contents: write` only to a separate publication job that verifies the same-run artifact and unchanged branch head before committing the lockfile. +- Removed obsolete one-shot repair workflows and patch scripts so repository automation no longer retains dormant write-capable recovery paths. From 461331f94e77754fe78b744ece539cfcc8d4dc5e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 02:05:49 +0900 Subject: [PATCH 24/25] fix(photo): align PNG metadata evidence with 0.18 API --- src-tauri/src/photo_duplicate.rs | 41 ++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/src-tauri/src/photo_duplicate.rs b/src-tauri/src/photo_duplicate.rs index d05c0906e..14d4ccaae 100644 --- a/src-tauri/src/photo_duplicate.rs +++ b/src-tauri/src/photo_duplicate.rs @@ -135,10 +135,12 @@ fn read_png_evidence(bytes: &[u8]) -> Result<(u32, u32, u8, u32, String), String output.color_type, output.bit_depth, )?; - let metadata_field_count = u32::from(reader.info().source_gamma.is_some()) - + u32::from(reader.info().source_chromaticities.is_some()) - + u32::from(reader.info().source_srgb.is_some()) - + u32::from(reader.info().source_iccp.is_some()) + // Count encoded metadata chunks rather than derived source values. png 0.18 derives source + // gamma/chromaticities from one sRGB chunk, which would otherwise over-weight a single field. + let metadata_field_count = u32::from(reader.info().gama_chunk.is_some()) + + u32::from(reader.info().chrm_chunk.is_some()) + + u32::from(reader.info().srgb.is_some()) + + u32::from(reader.info().icc_profile.is_some()) + u32::try_from(reader.info().uncompressed_latin1_text.len()).unwrap_or(u32::MAX) + u32::try_from(reader.info().compressed_latin1_text.len()).unwrap_or(u32::MAX) + u32::try_from(reader.info().utf8_text.len()).unwrap_or(u32::MAX); @@ -488,6 +490,18 @@ mod tests { .unwrap(); } + fn png_with_srgb(path: &Path, width: u32, height: u32, value: u8) { + let file = std::fs::File::create(path).unwrap(); + let mut encoder = png::Encoder::new(file, width, height); + encoder.set_color(png::ColorType::Grayscale); + encoder.set_depth(png::BitDepth::Eight); + encoder.set_source_srgb(png::SrgbRenderingIntent::Perceptual); + let mut writer = encoder.write_header().unwrap(); + writer + .write_image_data(&vec![value; (width * height) as usize]) + .unwrap(); + } + fn png_16(path: &Path, width: u32, height: u32, value: u8) { let file = std::fs::File::create(path).unwrap(); let mut encoder = png::Encoder::new(file, width, height); @@ -586,6 +600,25 @@ mod tests { assert!(!audit.exact_groups[0].execution_available); } + #[test] + fn srgb_chunk_counts_once_as_metadata_evidence() { + let temp = tempfile::tempdir().unwrap(); + let plain = temp.path().join("plain.png"); + let srgb = temp.path().join("srgb.png"); + png(&plain, 12, 9, 80); + png_with_srgb(&srgb, 12, 9, 80); + + let evidence = inspect_photo(&srgb).unwrap(); + assert_eq!(evidence.metadata_field_count, 1); + + let audit = audit_photos(&[plain, srgb.clone()], 7); + assert_eq!(audit.exact_groups.len(), 1); + assert_eq!( + audit.exact_groups[0].keeper_path.as_deref(), + Some(srgb.to_string_lossy().as_ref()) + ); + } + #[test] fn higher_source_bit_depth_is_the_unique_pareto_keeper_for_identical_samples() { let temp = tempfile::tempdir().unwrap(); From 59b4fe7035db29a5868f3f217bfe1c000ed63fe2 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 5 Sep 2026 02:15:23 +0900 Subject: [PATCH 25/25] docs(photo): delegate product baseline to canonical owner --- docs/product-technical-gap-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 0f1ab23e4..5ecd46866 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -23,7 +23,7 @@ authoritative, and no merge is claimed from queued or stale status. | P0 | A long Finder/provider copy can appear hung and consume the remaining local headroom. | The `real_datasets` Finder copy remained at “준비 중” for hours; the latest bounded iCloud dump retained 125 no-progress fetch/create markers, a 95.24% upload, and a zero-progress 1.06GB download while scheduling was `running`. Bounded `/bin/cp`/`mkdir` and global probes use private process groups and headroom gates. | Preview shows required bytes + staging reserve; timeout cleans only the child-created destination and leaves a durable receipt. | | P1 | Personal desktop-client capacity is not the same as API quota; OAuth is unnecessarily implied for a single-user installation. | ADR-0001 permits copy-only desktop-client mode marked `capacity-unverified`; the cloud connection UI defaults to read-only OAuth consent and requires an explicit write-access opt-in. | Settings clearly distinguish local desktop client, API quota, and organization OAuth; no OAuth prompt is required for the local-only path. | | P1 | Users cannot yet see a full lineage graph connecting source, metadata, archive member, provider item, receipt, Goal, and eviction decision. | The candidate UI now exposes a compact source→metadata→archive→provider lineage panel using the stable fingerprint, confidence, and blocker state; provider item/receipt/permit remain explicitly pending until their evidence exists. | Export and UI show stable content IDs, provenance edges, confidence, and blockers without exposing raw private paths. | -| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; the photo audit now has fail-closed exact-byte and exact decoded-pixel PNG evidence plus unique Pareto keeper display, but calibrated perceptual near-duplicate grouping, a provenance-bound IQA artifact, customer selection for ties, and reversible execution remain product gaps. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. Photo cleanup additionally requires a dataset-calibrated descriptor threshold and separately presented quality, lineage, and metadata evidence without a composite score. | +| P1 | “Orphan”/duplicate cleanup is difficult to trust because relationship evidence is not visible before action. | Ontology and duplicate/orphan PRs are open; current default path remains fail-closed. | Every proposed removal has an explainable parent/child/duplicate relation, identity recheck, reversible Trash action, and a no-candidate result when evidence is incomplete. | | P2 | Cross-platform behavior and accessibility are not presented as one release contract. | macOS/Linux/Windows release checks exist; several UI accessibility PRs remain open. | Release notes and UI expose platform capability matrix, keyboard/assistive labels, and bounded failure messages for each action. | ## Technical and operational gaps