fix: return full commit log when a commit tail is shorter than the fetch page#5040
Open
anxkhn wants to merge 1 commit into
Open
fix: return full commit log when a commit tail is shorter than the fetch page#5040anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a paging bug in the NoSQL Commits.commitLog (most-recent-first) iterator where null-padded fetch pages could cause an early stop when a commit’s tail is shorter than the internal fetch page size. This prevents silent truncation of commit history (and downstream maintenance retention decisions) when polaris.persistence.reference-previous-head-count is configured below the fetch page size.
Changes:
- Size the
fetchManyID page array tomin(REVERSE_COMMIT_FETCH_SIZE, tail.length)to avoid null padding reachingfetchMany/iteration. - Add a parameterized regression test covering multiple
referencePreviousHeadCountvalues below the page size across all NoSQL backends. - Add required test-fixture dependency and document the fix in
CHANGELOG.md.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| persistence/nosql/persistence/impl/src/main/java/org/apache/polaris/persistence/nosql/impl/commits/CommitsImpl.java | Eliminates null padding in reverse-page fetches to prevent premature end-of-history in commitLog. |
| persistence/nosql/persistence/impl/src/testFixtures/java/org/apache/polaris/persistence/nosql/impl/commits/BaseTestCommitLogImpl.java | Adds a cross-backend regression test verifying commitLog returns full history with short tails. |
| persistence/nosql/persistence/impl/build.gradle.kts | Adds polaris-idgen-api to test fixtures to support constructing custom-config persistence instances in the new test. |
| CHANGELOG.md | Notes the commit log truncation fix under Unreleased Fixes. |
…tch page Commits.commitLog paged through history by building a fixed-size ObjRef array of REVERSE_COMMIT_FETCH_SIZE and filling only tail.length entries. fetchMany returns null at the positions of the trailing null ids, and the iterator treats the first null as end-of-history, so a commit whose recent-ancestor tail is shorter than the fetch page size truncated the natural-order commit log at that point. With the default reference-previous-head-count of 20 (equal to the fetch size) the trailing null always coincided with genuine end-of-history, so the bug was not observable. With a smaller reference-previous-head-count the log stopped early, and because maintenance walks this log to decide retained objects, still-referenced objects could be dropped. Size the fetch array to min(REVERSE_COMMIT_FETCH_SIZE, tail.length) so no null padding reaches the iterator, mirroring the growing-list approach already used by commitLogReversed. Add a parameterized regression test that commits deeper than the tail with reference-previous-head-count below the fetch size and asserts the full log is returned. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
anxkhn
force-pushed
the
fix/commitlog-null-padding-truncation
branch
from
July 16, 2026 08:23
9363f85 to
b90b049
Compare
Contributor
Author
|
Pushed a revision addressing the latest review/CI feedback. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Commits.commitLogallocated a fixed-size fetch array and populated onlytail.lengthentries. Whenpolaris.persistence.reference-previous-head-countwas below the internal page size, null padding reachedfetchMany, and the iterator interpreted the first null as end-of-history. This silently truncated the natural-order commit log and could affect maintenance retention decisions.This change sizes the fetch array to
min(REVERSE_COMMIT_FETCH_SIZE, tail.length), so no null padding reaches the iterator. It mirrors the bounded growing-list approach used bycommitLogReversed.A parameterized test in
BaseTestCommitLogImplcovers previous-head counts 2, 3, and 5 across the NoSQL persistence backends and verifies that both commit-log directions return the complete 50-commit history.No public interface or configuration changes.
Checklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(not needed, no API or configuration change)