Search before asking
Please describe the bug 🐞
There appears to be a discrepancy between how incremental‑sync safety is validated for Delta vs. Iceberg when historical metadata has been truncated. Iceberg correctly detects expired snapshots and triggers a full snapshot sync, but Delta’s isIncrementalSyncSafeFrom only checks whether a commit existed at the given instant—not whether the underlying log files still exist. As a result, incremental sync is incorrectly marked “safe,” leading to failures such as DeltaFileNotFoundException when attempting to reconstruct old commits. This issue proposes enhancing Delta’s safety check to verify that required commit files are still available before approving incremental sync.
Are you willing to submit PR?
Code of Conduct
Search before asking
Please describe the bug 🐞
There appears to be a discrepancy between how incremental‑sync safety is validated for Delta vs. Iceberg when historical metadata has been truncated. Iceberg correctly detects expired snapshots and triggers a full snapshot sync, but Delta’s
isIncrementalSyncSafeFromonly checks whether a commit existed at the given instant—not whether the underlying log files still exist. As a result, incremental sync is incorrectly marked “safe,” leading to failures such asDeltaFileNotFoundExceptionwhen attempting to reconstruct old commits. This issue proposes enhancing Delta’s safety check to verify that required commit files are still available before approving incremental sync.Are you willing to submit PR?
Code of Conduct