[improve][pip] PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data

pip/pip-441.md

@@ -0,0 +1,115 @@
+# PIP-441: Add Broker-Level Metrics for Skipped Non-Recoverable Data
+
+# Background knowledge
+
+Pulsar's `autoSkipNonRecoverableData` feature allows brokers to skip corrupted data during disaster recovery to maintain topic availability. The system uses two skip strategies:
+
+1. **Ledger-level skipping**: Skips entire ledgers when completely unrecoverable
+2. **Entry-level skipping**: Skips specific entries within a ledger when only partially corrupted
+
+The entry level skipping was introduced in PIP-327 and refined in [PR #17753](https://github.com/apache/pulsar/pull/17753) to handle scenarios like ledger corruption, bookie failures, and partial data loss.
+
+# Motivation
+
+Currently, there is no visibility into when and how frequently non-recoverable data is being skipped, creating operational challenges:
+
+- **No alerting capability** when data loss occurs
+- **No audit trail** for compliance and data integrity requirements  
+- **Cannot distinguish** between healthy systems and those silently skipping data
+- **Cannot determine** if data loss is wholesale (ledgers) or partial (entries)
+- **Limited capacity planning** without understanding failure patterns
+
+# Goals
+
+## In Scope
+
+Add two broker-level metrics:
+- `pulsar_broker_non_recoverable_ledgers_skipped_total` - Count of ledgers skipped
+- `pulsar_broker_non_recoverable_entries_skipped_total` - Count of entries skipped
+
+## Out of Scope
+
+- Topic/subscription-level metrics (would burden metrics system with high cardinality)
+- Historical tracking of specific ledgers/entries skipped
+- Changes to existing `autoSkipNonRecoverableData` functionality
+
+# High Level Design
+
+Two broker-level counters will be added to `BrokerOperabilityMetrics`:
+
+- **Ledger counter**: Incremented in `ManagedLedgerImpl.skipNonRecoverableLedger()`
+- **Entry counter**: Incremented in `ManagedCursorImpl.skipNonRecoverableEntries()`
+
+Both metrics are exposed via the existing Prometheus `/metrics` endpoint.
+
+# Detailed Design
+
+## Implementation Details
+
+**BrokerOperabilityMetrics Changes:**
+```java
+private final LongAdder nonRecoverableLedgersSkippedCount;
+private final LongAdder nonRecoverableEntriesSkippedCount;
+
+public void recordNonRecoverableLedgerSkipped() {
+    this.nonRecoverableLedgersSkippedCount.increment();
+}
+
+public void recordNonRecoverableEntriesSkipped(long entriesCount) {
+    this.nonRecoverableEntriesSkippedCount.add(entriesCount);
+}
+```
+
+**Integration Points:**
+- `ManagedLedgerImpl.skipNonRecoverableLedger()` → calls `recordNonRecoverableLedgerSkipped()`  
+- `ManagedCursorImpl.skipNonRecoverableEntries()` → calls `recordNonRecoverableEntriesSkipped(count)`
+
+**OpenTelemetry Support:**
+- `pulsar.broker.non_recoverable_ledger.skip.count` 
+- `pulsar.broker.non_recoverable_entries.skip.count`
+
+## Public-facing Changes
+
+### Metrics
+
+| Metric Name | Description | Type |
+|-------------|-------------|------|
+| `pulsar_broker_non_recoverable_ledgers_skipped_total` | Count of ledgers skipped when `autoSkipNonRecoverableData` enabled | Counter |
+| `pulsar_broker_non_recoverable_entries_skipped_total` | Count of entries skipped when `autoSkipNonRecoverableData` enabled | Counter |
+
+**Labels:** `broker`, `cluster`
+
+# Monitoring
+
+**Use Cases:**
+- **Alerting**: Get notified when data loss occurs 
+- **SLA Monitoring**: Track data durability metrics
+- **Root Cause Analysis**: Compare metrics to understand if issues are systematic (ledger-level) or localized (entry-level)
+- **Investigation**: Use metrics for alerting, then check broker logs for specific topic details
+
+**Example Alerts:**
+```prometheus
+# Alert on any data loss
+increase(pulsar_broker_non_recoverable_ledgers_skipped_total[5m]) > 0 or
+increase(pulsar_broker_non_recoverable_entries_skipped_total[5m]) > 0
+```
+
+# Backward & Forward Compatibility
+
+- **Upgrade**: No special procedures required. Metrics start at 0.
+- **Downgrade**: Fully backward compatible. Metrics simply stop being exposed.
+- **Geo-Replication**: No impact. Each broker maintains its own counts.
+
+# Alternatives
+
+- **Topic-level metrics**: Rejected due to high cardinality burden on metrics system. Use broker metrics for alerting, then check logs for specific topics.
+- **Single combined metric**: Rejected as separate ledger/entry metrics provide better operational insights.
+
+# Summary
+
+This PIP adds essential observability for the existing `autoSkipNonRecoverableData` feature without changing its behavior. The broker-level approach balances operational visibility with system performance by avoiding high-cardinality metrics.
+
+# Links
+
+* Mailing List discussion thread: https://lists.apache.org/thread/b638towc7o4qb8dsozys4c14s00yflfj
+* Mailing List voting thread: https://lists.apache.org/thread/t461899qf878j4wr7wfhomkhw67vorjp