From 753252de9a26ec6f7d406111801929521a1fb0dd Mon Sep 17 00:00:00 2001 From: Penghui Li Date: Mon, 8 Sep 2025 09:13:31 -0700 Subject: [PATCH 1/2] [improve][pip] PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PIP proposes adding two broker-level metrics to provide visibility into non-recoverable data skipping when autoSkipNonRecoverableData is enabled: - pulsar_broker_non_recoverable_ledgers_skipped_total: Count of ledgers skipped - pulsar_broker_non_recoverable_entries_skipped_total: Count of entries skipped The metrics enable operators to: - Set up alerts when data loss occurs - Monitor data durability SLAs - Distinguish between systematic (ledger-level) vs localized (entry-level) issues - Use broker logs for detailed investigation of specific topics affected Implementation adds counters to BrokerOperabilityMetrics with integration points in ManagedLedgerImpl.skipNonRecoverableLedger() and ManagedCursorImpl.skipNonRecoverableEntries(). 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- pip/pip-441.md | 115 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 pip/pip-441.md diff --git a/pip/pip-441.md b/pip/pip-441.md new file mode 100644 index 0000000000000..4f5e204bec272 --- /dev/null +++ b/pip/pip-441.md @@ -0,0 +1,115 @@ +# PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data + +# Background knowledge + +Pulsar's `autoSkipNonRecoverableData` feature allows brokers to skip corrupted data during disaster recovery to maintain topic availability. The system uses two skip strategies: + +1. **Ledger-level skipping**: Skips entire ledgers when completely unrecoverable +2. **Entry-level skipping**: Skips specific entries within a ledger when only partially corrupted + +The entry level skipping was introduced in PIP-327 and refined in [PR #17753](https://github.com/apache/pulsar/pull/17753) to handle scenarios like ledger corruption, bookie failures, and partial data loss. + +# Motivation + +Currently, there is no visibility into when and how frequently non-recoverable data is being skipped, creating operational challenges: + +- **No alerting capability** when data loss occurs +- **No audit trail** for compliance and data integrity requirements +- **Cannot distinguish** between healthy systems and those silently skipping data +- **Cannot determine** if data loss is wholesale (ledgers) or partial (entries) +- **Limited capacity planning** without understanding failure patterns + +# Goals + +## In Scope + +Add two broker-level metrics: +- `pulsar_broker_non_recoverable_ledgers_skipped_total` - Count of ledgers skipped +- `pulsar_broker_non_recoverable_entries_skipped_total` - Count of entries skipped + +## Out of Scope + +- Topic/subscription-level metrics (would burden metrics system with high cardinality) +- Historical tracking of specific ledgers/entries skipped +- Changes to existing `autoSkipNonRecoverableData` functionality + +# High Level Design + +Two broker-level counters will be added to `BrokerOperabilityMetrics`: + +- **Ledger counter**: Incremented in `ManagedLedgerImpl.skipNonRecoverableLedger()` +- **Entry counter**: Incremented in `ManagedCursorImpl.skipNonRecoverableEntries()` + +Both metrics are exposed via the existing Prometheus `/metrics` endpoint. + +# Detailed Design + +## Implementation Details + +**BrokerOperabilityMetrics Changes:** +```java +private final LongAdder nonRecoverableLedgersSkippedCount; +private final LongAdder nonRecoverableEntriesSkippedCount; + +public void recordNonRecoverableLedgerSkipped() { + this.nonRecoverableLedgersSkippedCount.increment(); +} + +public void recordNonRecoverableEntriesSkipped(long entriesCount) { + this.nonRecoverableEntriesSkippedCount.add(entriesCount); +} +``` + +**Integration Points:** +- `ManagedLedgerImpl.skipNonRecoverableLedger()` → calls `recordNonRecoverableLedgerSkipped()` +- `ManagedCursorImpl.skipNonRecoverableEntries()` → calls `recordNonRecoverableEntriesSkipped(count)` + +**OpenTelemetry Support:** +- `pulsar.broker.non_recoverable_ledger.skip.count` +- `pulsar.broker.non_recoverable_entries.skip.count` + +## Public-facing Changes + +### Metrics + +| Metric Name | Description | Type | +|-------------|-------------|------| +| `pulsar_broker_non_recoverable_ledgers_skipped_total` | Count of ledgers skipped when `autoSkipNonRecoverableData` enabled | Counter | +| `pulsar_broker_non_recoverable_entries_skipped_total` | Count of entries skipped when `autoSkipNonRecoverableData` enabled | Counter | + +**Labels:** `broker`, `cluster` + +# Monitoring + +**Use Cases:** +- **Alerting**: Get notified when data loss occurs +- **SLA Monitoring**: Track data durability metrics +- **Root Cause Analysis**: Compare metrics to understand if issues are systematic (ledger-level) or localized (entry-level) +- **Investigation**: Use metrics for alerting, then check broker logs for specific topic details + +**Example Alerts:** +```prometheus +# Alert on any data loss +increase(pulsar_broker_non_recoverable_ledgers_skipped_total[5m]) > 0 or +increase(pulsar_broker_non_recoverable_entries_skipped_total[5m]) > 0 +``` + +# Backward & Forward Compatibility + +- **Upgrade**: No special procedures required. Metrics start at 0. +- **Downgrade**: Fully backward compatible. Metrics simply stop being exposed. +- **Geo-Replication**: No impact. Each broker maintains its own counts. + +# Alternatives + +- **Topic-level metrics**: Rejected due to high cardinality burden on metrics system. Use broker metrics for alerting, then check logs for specific topics. +- **Single combined metric**: Rejected as separate ledger/entry metrics provide better operational insights. + +# Summary + +This PIP adds essential observability for the existing `autoSkipNonRecoverableData` feature without changing its behavior. The broker-level approach balances operational visibility with system performance by avoiding high-cardinality metrics. + +# Links + +* Mailing List discussion thread: [To be added] +* Mailing List voting thread: [To be added] \ No newline at end of file From 5b451941d233fd385543f5eb0fea96ef32cd9c91 Mon Sep 17 00:00:00 2001 From: Jiwe Guo Date: Wed, 17 Sep 2025 10:08:52 +0800 Subject: [PATCH 2/2] add discuss and vote thread --- pip/pip-441.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pip/pip-441.md b/pip/pip-441.md index 4f5e204bec272..58622a9e3b07c 100644 --- a/pip/pip-441.md +++ b/pip/pip-441.md @@ -111,5 +111,5 @@ This PIP adds essential observability for the existing `autoSkipNonRecoverableDa # Links -* Mailing List discussion thread: [To be added] -* Mailing List voting thread: [To be added] \ No newline at end of file +* Mailing List discussion thread: https://lists.apache.org/thread/b638towc7o4qb8dsozys4c14s00yflfj +* Mailing List voting thread: https://lists.apache.org/thread/t461899qf878j4wr7wfhomkhw67vorjp \ No newline at end of file