Sync master-aligned delayed delivery & cursor persistence work into branch-as-4.0 (CI verification) - #98
Open
nodece wants to merge 211 commits into
Open
Sync master-aligned delayed delivery & cursor persistence work into branch-as-4.0 (CI verification)#98nodece wants to merge 211 commits into
nodece wants to merge 211 commits into
Conversation
…ust the log level to error (apache#25854) (cherry picked from commit 09035ff)
…m memorySize is a negative value (apache#25805) (cherry picked from commit 59d1495)
…26-45205 (apache#25844) (cherry picked from commit 7220158)
(cherry picked from commit a24a3b5)
…nd add neg-ack tests (apache#25867) Signed-off-by: Dream95 <zhou_8621@163.com> (cherry picked from commit a5c1029)
(cherry picked from commit 2e02b78)
…est (apache#25889) (cherry picked from commit a6af801)
…ties (apache#25862) (cherry picked from commit 9b15504)
(cherry picked from commit b406518)
…method may lose mark-delete properties in race condition (apache#25803) (cherry picked from commit 47eec87) Signed-off-by: Zixuan Liu <nodeces@gmail.com>
…y class does not match (apache#25825) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit b93fe9e)
…topic compaction (apache#25817) (cherry picked from commit 1fa9e35)
…n send failure when ackReceiptEnabled=false (apache#25525) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit fae3df9)
…acing LongAdder with DoubleAdder for sum accumulation (apache#25594) (cherry picked from commit 00577a5)
…ing (apache#25620) (cherry picked from commit 759a5f5)
…terFailover causing flaky test (apache#25388) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> (cherry picked from commit 3bc834f)
…LookupAutoClusterFailover (apache#25826) (cherry picked from commit 4229e19)
- master branch contains this change - the second parameter will also match a null value
…lishRateLimiting (apache#25365) (cherry picked from commit a3ae705)
- resolve a previous incorrect merge conflict resolution
…skipping-the-load-shedding (apache#23549) (cherry picked from commit 0f9ea18)
…ehavior (apache#25913) (cherry picked from commit be9f97a)
…ooKeeperClient.clientCreator (apache#25910) (cherry picked from commit 5627c01)
…tAutoClusterFailover() test (apache#25892) (cherry picked from commit ed09950)
(cherry picked from commit d57af8f)
…ache#25918) (cherry picked from commit df953e9)
…th ack timeout backoff (apache#25916) Signed-off-by: Dream95 <zhou_8621@163.com> (cherry picked from commit 756c03d)
…he replicator's remote topic creation (apache#26382) (cherry picked from commit bbc02f0)
…t explicitly (apache#26384) (cherry picked from commit 3be684a)
…ime (apache#26415) (cherry picked from commit 12b86b1)
…che#26361) (cherry picked from commit a25ae29)
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
…p usage to LongBitmap (apache#26117)
…lease level RandomGenerator.nextLong(long) is a JDK 17+ API and is not visible when pulsar-common is compiled at the client compiler release level. Use nextInt instead; all bounds fit in int and the differential test semantics are unchanged.
apache#26127 flips this default to true upstream. Deliberately keep the fork's existing default (false) to preserve upgrade compatibility for existing clusters; the new PositionRangeSet-backed format remains opt-in. Deviation from apache/master recorded during the 2026-08-27 alignment (PRs apache#25384/apache#26010/apache#26117/apache#26127).
Signed-off-by: Zixuan Liu <nodeces@gmail.com>
…[] (apache#26183) (cherry picked from commit 060b130)
…delivery trackers Promote the bucket-only dedup index introduced by apache#26251 to the shared delayed package as DelayedMessageIndex and use it in the in-memory tracker as well. addMessage now tracks positions idempotently, so a duplicate position is no longer enqueued twice (previously it would be scheduled and delivered twice). The manually maintained delayedMessagesCount counter is replaced by index.size(), making the counter an invariant of the bitmap instead of a discipline at every mutation site.
Squashed from the optimize-TripleLongPriorityQueue-heap-operations branch (10 commits on top of apache#26127) and adapted to the current PositionRangeSet-based code. Instead of serializing the full individualDeletedMessages snapshot on every flush, the cursor ledger becomes a structured checkpoint log: - CursorCheckpointLog: CursorLogEntry envelope; oversized checkpoints are transparently chunked (CursorCheckpointChunk), recovery scans back bounded by MAX_SCAN_BACK and falls back to legacy PositionInfo bytes - CursorCheckpointPersistence: per-msgLedger incremental writes — AckState carries one bitmap per changed ledger inline, unchanged ledgers only store an AckStateRef to their previously persisted entry; self-heals active ledgers without a checkpoint - Reset is consistency-first: in-memory state applies only after the tombstone is durable; on BK failure falls back to an md-only ZK tombstone (complete because a post-reset state has no holes) - shouldPersistUnackRangesToLedger always targets BK when enabled, so holes never collapse into ZK metadata - Cursor stats: individualDeletedMessagesCount / firstIndividualDeleted Message (ManagedCursorMXBean, metrics, internal stats), cardinality guard for markDeletePosition >= lastPosition, and truncation recording - Gated by persistentUnackedRangesWithPerLedgerEntryEnabled (default false); persistentUnackedRangesMaxEntrySize defaults to 5MB. Dirty-ledger tracking is tied to this flag alone; the legacy persistentUnackedRangesWithMultipleEntriesEnabled no longer has any runtime effect. Metrics aligned with apache/master and extended: - Truncation OTel counters match apache/master exactly, with its edge-triggered warning (lastCursorDataFullyPersistable); the long-array persistence path instruments the same counter via a toRanges callback, closing a silent-truncation blind spot upstream still has - New OTel metrics: cursor.ack.operation.count and cursor.recover.operation.count observables (zero hot-path cost) plus cursor.ack/persist/recover.latency DoubleHistograms (unit s); legacy brk_ml_cursor_* Prometheus gauges kept - Attribute resolution failures (non-topic-named managed ledgers) are contained so telemetry recording never breaks callback chains Adaptations for this branch: recoverIndividualDeletedMessages keeps the List-based entry point delegating to the (int, IntFunction) accessor form; test logging kept on the fork's explicit slf4j logger; restored the 5-arg MarkDeleteEntry convenience constructor used by NonDurableCursorImpl (missing on the source branch tip). Verification: build + checkstyle green; tests pass — CursorCheckpointLogRecoveryTest 13, PositionRangeSetDirtyTrackingTest 16, PositionRangeSetTest 22, PositionRangeSetCompatibilityTest 5, ManagedCursorTest#testCheckpoint* 27, OrderingTest 1.
nodece
force-pushed
the
branch-as-4.0-08282121
branch
from
August 28, 2026 13:38
43c054c to
8b87c1f
Compare
The ledger-delete failure hook added by the checkpoint persistence work was wired to io.netty.util.BooleanSupplier, which lacks getAsBoolean(). Use java.util.function.BooleanSupplier via its qualified name and drop the now-unused netty import.
…ePersistentEnabled (apache#21370) The part 2 of PIP-299: add config dispatcherPauseOnAckStatePersistentEnabled
The PositionRangeSetBenchmark introduced new microbench packages without package-info.java, failing the JavadocPackage checkstyle rule.
Move the AssertJUnit static imports before regular imports without a blank-line separation, as required by the ImportOrder checkstyle rule.
recordAckStats/recordPersistStats/recordRecoverStats assumed ledger.getFactory() is non-null; tests that construct ManagedCursorImpl with a mocked ManagedLedgerImpl (e.g. testRecoverCursorAheadOfLastPosition) hit an NPE in the recovery callback. Skip OTel recording when the factory is absent.
PersistentReplicator's constructor reads topic.getHierarchyTopicPolicies().getResourceGroupName(); the fixture's mocked PersistentTopic returned null there, failing the first data-provider case with an NPE and skipping the rest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan