diff --git a/CHANGELOG.md b/CHANGELOG.md index 26130bb39..07434f328 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ All notable changes to DiskSage are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and released versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). Unreleased entries describe integrated source changes only; they are not release evidence until the repository's review, CI, security, packaging, provenance, and release-acceptance gates pass on the exact tagged commit. ## [Unreleased] +- Make Apple Photos duplicate inventory resumable and cancellable between native-completed assets, + with visible progress and no arbitrary whole-library timeout. - Connect exact decoded-pixel duplicate groups to the existing reversible photo quarantine engine; forged or stale audits, ambiguous keepers without an explicit selection, and changed roots fail closed. The review screen shows measured evidence and blockers, requires direct typed approval, and reports diff --git a/docs/architecture/adr/0024-photokit-checkpointed-inventory.md b/docs/architecture/adr/0024-photokit-checkpointed-inventory.md new file mode 100644 index 000000000..58a8897a5 --- /dev/null +++ b/docs/architecture/adr/0024-photokit-checkpointed-inventory.md @@ -0,0 +1,38 @@ +# ADR-0024: Checkpoint PhotoKit inventory at native completion boundaries + +- Status: Accepted +- Date: 2026-08-30 + +## Context + +Reading locally available originals can take materially different time per asset. A single large +PhotoKit request kept the customer waiting without progress or cancellation, while an arbitrary +wall-clock timeout discarded completed evidence. + +## Decision + +DiskSage requests one `PHAsset` per native page. A page is accepted only after PhotoKit's resource +completion handler, and records its measured duration rather than using it as a guessed cutoff. +Rust rejects gaps, repeated offsets, missing completion evidence, and inconsistent totals. The UI +checkpoints after every page, yields for rendering, and stops between pages when requested. A +checkpoint is resumable; it never authorizes deletion. Network access remains disabled and no +PhotoKit change request is made during inventory. + +## Consequences + +Large libraries take as long as their locally available originals require, but the customer sees +progress, may stop safely, and can resume without repeating accepted pages. Destructive planning +still requires a complete inventory and the existing fresh re-fetch and exact approval contract. + +## Rejected alternatives + +An arbitrary whole-library timeout was rejected because elapsed wall time is not evidence that +PhotoKit failed. Large fixed pages were rejected because they cannot yield promptly between assets. + +## References + +Apple. (n.d.). *PHAssetResourceManager*. Apple Developer Documentation. +https://developer.apple.com/documentation/photokit/phassetresourcemanager + +Apple. (n.d.). *Fetching and caching assets and thumbnails*. Apple Developer Documentation. +https://developer.apple.com/documentation/photokit/fetching-and-caching-assets-and-thumbnails diff --git a/docs/architecture/adr/README.md b/docs/architecture/adr/README.md index 72fa8c017..7239f5739 100644 --- a/docs/architecture/adr/README.md +++ b/docs/architecture/adr/README.md @@ -29,6 +29,7 @@ new numbered record rather than rewriting history. | [0021](0021-perceptual-photo-candidates.md) | Require measured evidence and a selected survivor for perceptual photo candidates | Accepted | | [0022](0022-photo-duplicate-evidence-without-composite-scoring.md) | Separate photo-duplicate and keeper evidence without composite scoring | Accepted | | [0023](0023-apple-photos-photokit-boundary.md) | Use PhotoKit rather than Photos library package traversal | Accepted | +| [0024](0024-photokit-checkpointed-inventory.md) | Checkpoint PhotoKit inventory at native completion boundaries | Accepted | New records must state context, decision, consequences, rejected alternatives, and the evidence or standard that led to the decision. A record never grants cloud-write or source-eviction authority; diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 827d8fb1d..a714aa879 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -1236,6 +1236,12 @@ runner's private workspace temp root instead of weakening the shared production content digest before invoking Photos' own deletion transaction and confirmation; a create-new receipt follows success. Near-duplicate managed assets remain unavailable rather than receiving an uncalibrated score, so that Gap is explicit and non-destructive. +- A 5,000-item read-only audit did not return customer-visible progress before an external + 60-second observation was stopped. The native bridge now returns exactly one asset only after + PhotoKit's completion callback; Rust persists gap-free checkpoint evidence, and the UI renders + progress and accepts cancellation between completed assets. Resuming uses the accepted + checkpoint instead of restarting. No elapsed-time threshold is interpreted as failure and no + photo mutation is introduced by this audit path. - A fresh Naruon audit proved exactly one removable worktree: PR #1429 was merged, its detached head was retained by current `origin/develop`, the checkout was clean and inactive, and no open diff --git a/src-tauri/native/photos_bridge.m b/src-tauri/native/photos_bridge.m index 6bef3bf73..68dbbc074 100644 --- a/src-tauri/native/photos_bridge.m +++ b/src-tauri/native/photos_bridge.m @@ -3,8 +3,9 @@ #import static const NSUInteger DSMaxChunkBytes = 8 * 1024 * 1024; -static const int64_t DSResourceTimeoutNanos = 30LL * NSEC_PER_SEC; static const int64_t DSAuthorizationTimeoutNanos = 5LL * 60LL * NSEC_PER_SEC; +static NSArray *DSPageIdentifiers = nil; +static NSString *DSPageInventoryIdentity = nil; static char *DSJSON(id value) { NSData *data = [NSJSONSerialization dataWithJSONObject:value options:0 error:nil]; @@ -29,6 +30,18 @@ return hex; } +static NSString *DSInventoryIdentity(PHFetchResult *fetch) { + CC_SHA256_CTX context; CC_SHA256_Init(&context); + for (PHAsset *asset in fetch) { + NSData *identifier = [asset.localIdentifier dataUsingEncoding:NSUTF8StringEncoding]; + uint64_t length = CFSwapInt64HostToLittle((uint64_t)identifier.length); + CC_SHA256_Update(&context, &length, sizeof(length)); + CC_SHA256_Update(&context, identifier.bytes, (CC_LONG)identifier.length); + } + unsigned char digest[CC_SHA256_DIGEST_LENGTH]; CC_SHA256_Final(digest, &context); + return DSHex(digest, sizeof(digest)); +} + static NSString *DSMetadataFingerprint(PHAsset *asset, PHAssetResource *resource) { NSString *text = [NSString stringWithFormat:@"%@\n%ld\n%ld\n%.0f\n%.0f\n%@\n%@\n%ld", asset.localIdentifier, (long)asset.pixelWidth, (long)asset.pixelHeight, @@ -81,7 +94,7 @@ int64_t ds_photos_select_still_resource_index(const int64_t *types, size_t count __block uint64_t byteCount = 0; __block BOOL exceeded = NO; __block NSError *completionError = nil; - PHAssetResourceDataRequestID requestID = [[PHAssetResourceManager defaultManager] + [[PHAssetResourceManager defaultManager] requestDataForAssetResource:resource options:options dataReceivedHandler:^(NSData *data) { @@ -96,11 +109,7 @@ int64_t ds_photos_select_still_resource_index(const int64_t *types, size_t count completionError = error; dispatch_semaphore_signal(done); }]; - if (dispatch_semaphore_wait(done, dispatch_time(DISPATCH_TIME_NOW, DSResourceTimeoutNanos)) != 0) { - [[PHAssetResourceManager defaultManager] cancelDataRequest:requestID]; - dispatch_semaphore_wait(done, dispatch_time(DISPATCH_TIME_NOW, 5LL * NSEC_PER_SEC)); - return @{ @"state": @"unavailable", @"blocker": @"local-content-read-timed-out" }; - } + dispatch_semaphore_wait(done, DISPATCH_TIME_FOREVER); if (completionError) { return @{ @"state": @"icloud-only-or-unavailable", @"blocker": @"download-original-in-photos" }; } @@ -197,6 +206,54 @@ int64_t ds_photos_select_still_resource_index(const int64_t *types, size_t count return DSJSON(DSInventory(MIN((NSUInteger)maxAssets, 10000), MIN(maxBytes, 536870912ULL), nil)); } +// One PhotoKit asset is the native pagination unit. Each call returns only after PhotoKit's +// resource completion handler (or its explicit unavailable result), allowing the UI to checkpoint, +// repaint, or cancel between assets without guessing a wall-clock page timeout. +char *ds_photos_inventory_page(uint64_t offset, uint64_t maxBytes) { + PHAuthorizationStatus status = [PHPhotoLibrary authorizationStatusForAccessLevel:PHAccessLevelReadWrite]; + if (status != PHAuthorizationStatusAuthorized && status != PHAuthorizationStatusLimited) + return DSJSON(@{ @"authorization": DSStatus(status), @"observed_at_ms": @((uint64_t)(NSDate.date.timeIntervalSince1970 * 1000)), + @"total_count": @0, @"offset": @(offset), @"next_offset": [NSNull null], + @"inventory_identity": @"", + @"native_completion_observed": @YES, @"page_duration_ms": @0, + @"assets": @[], @"unavailable_count": @0 }); + PHFetchOptions *options = [PHFetchOptions new]; + options.predicate = [NSPredicate predicateWithFormat:@"mediaType == %d", PHAssetMediaTypeImage]; + PHFetchResult *fetch = [PHAsset fetchAssetsWithOptions:options]; + if (offset == 0 || !DSPageIdentifiers || !DSPageInventoryIdentity) { + NSMutableArray *identifiers = [NSMutableArray arrayWithCapacity:fetch.count]; + for (PHAsset *asset in fetch) [identifiers addObject:asset.localIdentifier]; + DSPageIdentifiers = [identifiers copy]; + DSPageInventoryIdentity = DSInventoryIdentity(fetch); + } + NSString *inventoryIdentity = DSPageInventoryIdentity; + uint64_t snapshotTotal = DSPageIdentifiers.count; + uint64_t total = fetch.count; + BOOL libraryChanged = total != snapshotTotal; + uint64_t started = (uint64_t)(NSDate.date.timeIntervalSince1970 * 1000); + NSArray *assets = @[]; + uint64_t unavailable = 0; + if (total == snapshotTotal && offset < snapshotTotal) { + NSString *identifier = DSPageIdentifiers[(NSUInteger)offset]; + PHFetchResult *selected = [PHAsset fetchAssetsWithLocalIdentifiers:@[identifier] options:nil]; + if (selected.count == 1) { + NSDictionary *asset = DSAssetEvidence(selected.firstObject, MIN(maxBytes, 536870912ULL)); + assets = @[asset]; + unavailable = asset[@"content_sha256"] ? 0 : 1; + } else libraryChanged = YES; + } + if (libraryChanged || (offset + assets.count >= snapshotTotal && ![DSInventoryIdentity(fetch) isEqualToString:inventoryIdentity])) + inventoryIdentity = @"photos-library-changed"; + uint64_t completed = (uint64_t)(NSDate.date.timeIntervalSince1970 * 1000); + NSNumber *next = offset + assets.count < snapshotTotal ? @(offset + assets.count) : nil; + return DSJSON(@{ @"authorization": DSStatus(status), @"observed_at_ms": @(completed), + @"total_count": @(snapshotTotal), @"offset": @(offset), + @"inventory_identity": inventoryIdentity, + @"next_offset": next ?: [NSNull null], @"native_completion_observed": @YES, + @"page_duration_ms": @(completed - started), @"assets": assets, + @"unavailable_count": @(unavailable) }); +} + char *ds_photos_delete(const char *requestJSON) { NSData *data = [[NSData alloc] initWithBytes:requestJSON length:strlen(requestJSON)]; NSDictionary *request = [NSJSONSerialization JSONObjectWithData:data options:0 error:nil]; diff --git a/src-tauri/src/container_orphan_reclaim.rs b/src-tauri/src/container_orphan_reclaim.rs index 7debfeb15..c92893da1 100644 --- a/src-tauri/src/container_orphan_reclaim.rs +++ b/src-tauri/src/container_orphan_reclaim.rs @@ -1381,6 +1381,9 @@ pub fn execute_container_orphan_prune( stdout: output.stdout, stderr: output.stderr, output_truncated: false, + // A non-zero exact-delete status means the runtime refused the mutation (for example, + // because a previously stopped container restarted). Keep the attempted command and + // status in the receipt, but never report the reclaim as executed. executed: output.status_code == 0, executed_at_ms, before_available_bytes, diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index 39a2ed774..7b974b82c 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -153,7 +153,8 @@ pub fn run() { photo_duplicate_quarantine::execute_exact_photo_duplicate_quarantine, photos_library::photos_authorization_status, photos_library::request_photos_authorization, - photos_library::inspect_photos_duplicates, + photos_library::inspect_photos_duplicates_page, + photos_library::finalize_photos_duplicate_inventory, photos_library::plan_photos_duplicate_deletion, photos_library::execute_photos_duplicate_deletion, commands::get_ontology, diff --git a/src-tauri/src/photos_library.rs b/src-tauri/src/photos_library.rs index 6bea1530d..7dfc3d74a 100644 --- a/src-tauri/src/photos_library.rs +++ b/src-tauri/src/photos_library.rs @@ -58,6 +58,120 @@ pub struct PhotosDuplicateInventory { pub exact_groups: Vec, pub unavailable_count: u64, pub near_duplicate_evidence: Option, + #[serde(default)] + pub inventory_total_count: Option, + #[serde(default)] + pub inventory_page_identity: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PhotosInventoryPage { + pub authorization: String, + pub observed_at_ms: u64, + pub total_count: u64, + pub inventory_identity: String, + pub offset: u64, + pub next_offset: Option, + pub native_completion_observed: bool, + pub page_duration_ms: u64, + pub assets: Vec, + pub unavailable_count: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct PhotosInventoryCheckpoint { + pub next_offset: u64, + pub total_count: u64, + pub inventory_identity: String, +} + +/// Merge one native-completed PhotoKit page into a resumable read-only checkpoint. +pub fn merge_inventory_page( + previous: Option, + page: PhotosInventoryPage, +) -> Result { + if !page.native_completion_observed || page.assets.len() > 1 { + return Err("photos-page-native-completion-required".into()); + } + let mut inventory = previous.unwrap_or(PhotosDuplicateInventory { + authorization: page.authorization.clone(), + observed_at_ms: Some(page.observed_at_ms), + inventory_fingerprint: None, + evidence_complete: false, + inventory_truncated: true, + next_action: "사진 확인을 계속하세요.".into(), + assets: Vec::new(), + exact_groups: Vec::new(), + unavailable_count: 0, + near_duplicate_evidence: Some("unavailable-without-measured-content-equivalence".into()), + inventory_total_count: Some(page.total_count), + inventory_page_identity: Some(page.inventory_identity.clone()), + }); + if inventory.authorization != page.authorization + || inventory.inventory_total_count != Some(page.total_count) + || inventory.inventory_page_identity.as_deref() != Some(&page.inventory_identity) + || inventory.assets.len() as u64 != page.offset + || page + .next_offset + .is_some_and(|next| next != page.offset + page.assets.len() as u64) + || page.total_count < page.offset + page.assets.len() as u64 + { + return Err("photos-page-checkpoint-mismatch".into()); + } + inventory.assets.extend(page.assets); + inventory.unavailable_count = inventory + .unavailable_count + .checked_add(page.unavailable_count) + .ok_or("photos-page-count-overflow")?; + inventory.observed_at_ms = Some(page.observed_at_ms); + let complete = page.next_offset.is_none() && inventory.assets.len() as u64 == page.total_count; + inventory.inventory_truncated = !complete; + inventory.evidence_complete = complete + && inventory.unavailable_count == 0 + && matches!(inventory.authorization.as_str(), "authorized" | "limited"); + inventory.next_action = if complete { + if inventory.unavailable_count > 0 { + "사진 앱에서 이 Mac에 없는 원본을 다운로드한 뒤 다시 확인하세요.".into() + } else { + "검사가 끝났습니다. 정확한 사본 그룹을 검토하세요.".into() + } + } else { + format!( + "{}개 중 {}개를 확인했습니다. 계속하려면 사진 확인을 누르세요.", + page.total_count, + inventory.assets.len() + ) + }; + let mut groups = BTreeMap::>::new(); + for asset in &inventory.assets { + if let Some(digest) = &asset.content_sha256 { + groups + .entry(digest.clone()) + .or_default() + .push(asset.clone()); + } + } + inventory.exact_groups = groups + .into_iter() + .filter(|(_, members)| members.len() > 1) + .map(|(content_sha256, members)| PhotosExactGroup { + content_sha256, + members, + keeper_required: true, + automatic_delete_allowed: false, + }) + .collect(); + inventory.inventory_fingerprint = if complete { + Some(hash_json(&( + inventory.assets.clone(), + inventory.unavailable_count, + ))?) + } else { + None + }; + Ok(inventory) } #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -291,6 +405,7 @@ mod native { fn ds_photos_authorization_status() -> *mut c_char; fn ds_photos_request_authorization() -> *mut c_char; fn ds_photos_inventory(max_assets: u32, max_bytes: u64) -> *mut c_char; + fn ds_photos_inventory_page(offset: u64, max_bytes: u64) -> *mut c_char; fn ds_photos_delete(request_json: *const c_char) -> *mut c_char; #[cfg(test)] fn ds_photos_select_still_resource_index(types: *const i64, count: usize) -> i64; @@ -317,6 +432,10 @@ mod native { take_json(unsafe { ds_photos_inventory(MAX_INVENTORY_ASSETS, MAX_RESOURCE_BYTES) }) } + pub fn inventory_page(offset: u64) -> Result { + take_json(unsafe { ds_photos_inventory_page(offset, MAX_RESOURCE_BYTES) }) + } + #[derive(Deserialize)] struct NativeDeleteResult { deleted_identifiers: Option>, @@ -362,6 +481,9 @@ mod native { pub fn inventory() -> Result { Err("photos-library-macos-only".into()) } + pub fn inventory_page(_: u64) -> Result { + Err("photos-library-macos-only".into()) + } pub fn delete(_: &PhotosDeletionPlan) -> Result<(), String> { Err("photos-library-macos-only".into()) } @@ -389,6 +511,42 @@ pub async fn inspect_photos_duplicates() -> Result, +) -> Result { + let offset = checkpoint.as_ref().map_or(0, |value| value.next_offset); + let page = tauri::async_runtime::spawn_blocking(move || native::inventory_page(offset)) + .await + .map_err(|_| "photos-operation-interrupted".to_string())??; + if !matches!(page.authorization.as_str(), "authorized" | "limited") { + return Err("photos-authorization-required".into()); + } + if checkpoint.as_ref().is_some_and(|value| { + value.total_count != page.total_count + || value.inventory_identity != page.inventory_identity + || value.next_offset != page.offset + }) { + return Err("photos-page-checkpoint-mismatch".into()); + } + Ok(page) +} + +#[tauri::command] +pub fn finalize_photos_duplicate_inventory( + pages: Vec, +) -> Result { + let mut inventory = None; + for page in pages { + inventory = Some(merge_inventory_page(inventory, page)?); + } + let inventory = inventory.ok_or("photos-page-checkpoint-empty")?; + if inventory.inventory_truncated { + return Err("photos-page-checkpoint-incomplete".into()); + } + Ok(inventory) +} + #[tauri::command] pub fn plan_photos_duplicate_deletion( inventory: PhotosDuplicateInventory, @@ -511,9 +669,65 @@ mod tests { near_duplicate_evidence: Some( "unavailable-without-measured-content-equivalence".into(), ), + inventory_total_count: Some(2), + inventory_page_identity: Some("stable-library".into()), } } + fn page(offset: u64, total: u64, asset: PhotosAssetEvidence) -> PhotosInventoryPage { + PhotosInventoryPage { + authorization: "authorized".into(), + observed_at_ms: 100 + offset, + total_count: total, + inventory_identity: "stable-library".into(), + offset, + next_offset: (offset + 1 < total).then_some(offset + 1), + native_completion_observed: true, + page_duration_ms: 7, + assets: vec![asset], + unavailable_count: 0, + } + } + + #[test] + fn native_completed_pages_resume_and_only_finish_at_exact_total() { + let first = merge_inventory_page(None, page(0, 2, member("keep", 10))).unwrap(); + assert!(first.inventory_truncated); + assert!(!first.evidence_complete); + assert!(first.inventory_fingerprint.is_none()); + let complete = merge_inventory_page(Some(first), page(1, 2, member("remove", 8))).unwrap(); + assert!(!complete.inventory_truncated); + assert!(complete.evidence_complete); + assert!(complete.inventory_fingerprint.is_some()); + assert_eq!(complete.exact_groups.len(), 1); + } + + #[test] + fn page_gap_or_missing_native_completion_is_rejected() { + let first = merge_inventory_page(None, page(0, 2, member("keep", 10))).unwrap(); + assert_eq!( + merge_inventory_page(Some(first.clone()), page(0, 2, member("remove", 8))).unwrap_err(), + "photos-page-checkpoint-mismatch" + ); + let mut incomplete = page(1, 2, member("remove", 8)); + incomplete.native_completion_observed = false; + assert_eq!( + merge_inventory_page(Some(first), incomplete).unwrap_err(), + "photos-page-native-completion-required" + ); + } + + #[test] + fn equal_count_library_replacement_invalidates_checkpoint_identity() { + let first = merge_inventory_page(None, page(0, 2, member("keep", 10))).unwrap(); + let mut replaced = page(1, 2, member("replacement", 8)); + replaced.inventory_identity = "changed-library".into(); + assert_eq!( + merge_inventory_page(Some(first), replaced).unwrap_err(), + "photos-page-checkpoint-mismatch" + ); + } + #[test] fn plan_requires_explicit_keeper_and_preserves_it() { let inventory = inventory(); @@ -647,10 +861,11 @@ mod tests { } #[test] - fn native_boundary_bounds_callbacks() { + fn native_boundary_keeps_local_reads_offline_without_an_arbitrary_timeout() { let source = include_str!("../native/photos_bridge.m"); assert!(source.contains("networkAccessAllowed = NO")); - assert!(source.contains("cancelDataRequest:requestID")); + assert!(source.contains("dispatch_semaphore_wait(done, DISPATCH_TIME_FOREVER)")); + assert!(!source.contains("cancelDataRequest:requestID")); assert!(source.contains("DSAuthorizationTimeoutNanos")); } diff --git a/src/lib/PhotosLibraryReview.svelte b/src/lib/PhotosLibraryReview.svelte index 6ef04be4e..0743064e3 100644 --- a/src/lib/PhotosLibraryReview.svelte +++ b/src/lib/PhotosLibraryReview.svelte @@ -1,7 +1,11 @@