From 52f47ace6aa648f087ede6080f4b71e9b1f1b604 Mon Sep 17 00:00:00 2001 From: namest504 Date: Wed, 15 Oct 2025 00:18:52 +0900 Subject: [PATCH 1/7] [improve][pip] PIP-445: Add Builder Methods to Create Message-based TableView --- pip/pip-445.md | 126 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 pip/pip-445.md diff --git a/pip/pip-445.md b/pip/pip-445.md new file mode 100644 index 0000000000000..bb613c4f244aa --- /dev/null +++ b/pip/pip-445.md @@ -0,0 +1,126 @@ +# PIP-445: Add Builder Methods to Create Message-based TableView + +# Background knowledge + +* **TableView**: In Pulsar, a `TableView` is a client-side abstraction that provides a key-value map interface over a Pulsar topic. It consumes messages from the topic (typically a compacted one) and maintains an in-memory view of the latest value for each key. This allows applications to easily query the current state of a key without managing a consumer manually. + +* **Pulsar `Message`**: A Pulsar message is not just its data payload. The `Message` object is a container that includes the deserialized **payload** (`T`) as well as important **metadata**, such as a message key, user-defined properties (a key-value map), event time, publish time, and more. + +# Motivation + +The current `TableView` API provides a `get(String key)` method that only returns the deserialized **value** (`T`) of the latest message for a given key. This limits its usefulness for applications that need access to the message's metadata. + +For instance, a user might need to inspect the message **properties** to get a trace-id or check the **event time** to determine if the data is recent. Currently, the only way to access this metadata is to create a separate, redundant `Consumer` on the same topic, which is inefficient and undermines the convenience of using a `TableView`. + +This proposal aims to solve this problem by providing a way to create a `TableView` that exposes the entire `Message` object. + +# Goals + +## In Scope + +* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder` interface. +* Allow users to create a `TableView>` instance, which provides access to the complete `Message` object for each key, including its payload, properties, and all other metadata. +* Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users. + +## Out of Scope + +* Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder. +* Changing the underlying topic compaction logic or any broker-side functionality. + +# High Level Design + +The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`. + +1. New methods, `TableView> createForMessages()` and `CompletableFuture>> createForMessagesAsync()`, will be added to the `TableViewBuilder` interface. +2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message` object for each key. +3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`). + +This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases. + +# Detailed Design + +## Design & Implementation Details + +The changes will be confined to the Pulsar client library. + +* **Interface `org.apache.pulsar.client.api.TableViewBuilder`**: + New methods will be added to this interface to create a `TableView` for messages. + +* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: + The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`. + +* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl`**: + A new class will be created that implements `TableView>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message` objects instead of just `T` values. Its `get(key)` method will return the full `Message` object. + +* **Class `org.apache.pulsar.client.impl.TableViewImpl`**: + This class will remain unchanged, ensuring no impact on existing users. + +## Public-facing Changes + +### Public API + +New methods will be added to the `org.apache.pulsar.client.api.TableViewBuilder` interface. + +* **Method Signatures**: + ```java + TableView> createForMessages() throws PulsarClientException; + + CompletableFuture>> createForMessagesAsync(); + ``` +* **Description**: Creates a `TableView` instance where the values in the map are the full `Message` objects, including payload and metadata. This allows access to message properties, event time, etc. +* **Return Value**: A `TableView>` instance. + +### Binary protocol + +No changes. + +### Configuration + +No changes. + +### CLI + +No changes. + +### Metrics + +No changes. + +# Monitoring + +No new metrics are introduced by this change. Existing client-side metrics are unaffected. + +# Security Considerations + +This proposal has no security implications. The new method exposes message metadata that the client is already authorized to receive by consuming the topic. It does not alter any authentication or authorization mechanisms. + +# Backward & Forward Compatibility + +This change is fully backward-compatible. + +* The addition of new methods to the builder interface is a non-breaking change. Existing code that uses `create()` or `createAsync()` will continue to function as before with no performance or behavioral changes. + +## Upgrade + +The upgrade process is seamless. Applications can update their client dependency to a version containing this feature and start using the new builder methods without any other changes. + +## Downgrade / Rollback + +A downgrade is also seamless. If an application that uses the new `createForMessages` methods is rolled back to an older client version, it will fail at compile time. Applications that do not use the new methods can be rolled back without any issues. + +## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations + +This is a client-side change and has no impact on geo-replication. + +# Alternatives + +## Add `getRawMessage(String key)` to `TableView` + +An alternative considered was to add a `getRawMessage(String key)` method directly to the `TableView` interface. This would have required modifying the existing `TableViewImpl` to store the entire `Message` object for all users. + +This approach was rejected because it would be a **breaking change in terms of performance**. It would increase memory and CPU consumption for all `TableView` users, even those who do not need access to the raw message. The proposed builder-based approach is superior as it makes this an opt-in feature, preserving the performance characteristics of the existing `TableView`. + +# Links + +* Mailing List discussion thread: TBD +* Mailing List voting thread: TBD \ No newline at end of file From 1f52c13ac987a94b0335d90e6bdfec64082aeb12 Mon Sep 17 00:00:00 2001 From: namest504 Date: Mon, 20 Oct 2025 11:16:41 +0900 Subject: [PATCH 2/7] Update pip-445.md --- pip/pip-445.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index bb613c4f244aa..81daed8251fd8 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -43,17 +43,17 @@ This opt-in design provides the new functionality efficiently without impacting The changes will be confined to the Pulsar client library. -* **Interface `org.apache.pulsar.client.api.TableViewBuilder`**: - New methods will be added to this interface to create a `TableView` for messages. +* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView`**: + * An abstract base class will be created to contain the common logic for `TableView` implementations, such as managing the underlying consumer and handling topic events. This prevents code duplication. -* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: - The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`. +* **Class `org.apache.pulsar.client.impl.TableViewImpl`**: + * This class will be refactored to extend the new `AbstractTableView`. While its internal implementation will change to use the base class, its public-facing functionality and behavior will remain exactly the same. This ensures that the change has no impact on existing users. * **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl`**: - A new class will be created that implements `TableView>`. It will be based on the existing `TableViewImpl` but its internal map will store `Message` objects instead of just `T` values. Its `get(key)` method will return the full `Message` object. + * A new class will be created that extends `AbstractTableView>` and implements the `TableView>` interface. It will be responsible for storing the full `Message` object for each key, which its `get(key)` method will return. -* **Class `org.apache.pulsar.client.impl.TableViewImpl`**: - This class will remain unchanged, ensuring no impact on existing users. +* **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: + * The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`. ## Public-facing Changes From a1ce3d59915afdcd6f661e2ef793ca03799979a5 Mon Sep 17 00:00:00 2001 From: namest504 Date: Mon, 20 Oct 2025 11:20:18 +0900 Subject: [PATCH 3/7] Update pip-445.md --- pip/pip-445.md | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index 81daed8251fd8..da668c689330a 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -18,24 +18,24 @@ This proposal aims to solve this problem by providing a way to create a `TableVi ## In Scope -* Add new methods, `createForMessages()` and `createForMessagesAsync()`, to the `TableViewBuilder` interface. -* Allow users to create a `TableView>` instance, which provides access to the complete `Message` object for each key, including its payload, properties, and all other metadata. +* Add new generic methods, `createMapped()` and `createMappedAsync()`, to the `TableViewBuilder` interface, which accept a mapping function. +* Allow users to create a `TableView` instance by providing a function that transforms a `Message` into a custom object `V`. This includes the ability to create a `TableView>` by passing an identity function. * Ensure the change is fully backward-compatible and does not impact the performance of existing `TableView` users. ## Out of Scope * Modifying the behavior of the existing `create()` and `createAsync()` methods in the builder. * Changing the underlying topic compaction logic or any broker-side functionality. +* Handling exceptions thrown by the user-provided mapper function within the `TableView` (e.g., "poison pill" message handling). # High Level Design -The proposed solution is a simple and non-breaking addition to the public client API. Instead of modifying the existing `TableView` implementation, we will introduce new methods to the `TableViewBuilder`. +The proposed solution is a simple and non-breaking addition to the public client API. Instead of adding a specific method for retrieving messages, we will introduce a more flexible, generic mapping mechanism. -1. New methods, `TableView> createForMessages()` and `CompletableFuture>> createForMessagesAsync()`, will be added to the `TableViewBuilder` interface. -2. These methods will create a new, specialized `TableView` implementation (`MessageTableViewImpl`) that stores the entire `Message` object for each key. -3. The existing `create()` and `createAsync()` methods will continue to create the standard `TableView` that stores only the message value (`T`). - -This opt-in design provides the new functionality efficiently without impacting the performance or behavior of existing `TableView` use cases. +1. New generic methods, ` TableView createMapped(...)` and ` CompletableFuture> createMappedAsync(...)`, will be added to the `TableViewBuilder` interface. +2. These methods will accept a `java.util.function.Function, V>` as a parameter. This `mapper` function defines how to transform an incoming raw `Message` into a value `V` to be stored in the `TableView`. +3. This approach provides maximum flexibility. Users who need the entire `Message` object can simply pass `Function.identity()` as the mapper. Other users can create custom, memory-efficient objects containing only the necessary data from the message payload and metadata. +4. The existing `create()` and `createAsync()` methods will remain unchanged, preserving behavior for all existing use cases. # Detailed Design @@ -55,20 +55,25 @@ The changes will be confined to the Pulsar client library. * **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: * The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`. -## Public-facing Changes +# Public-facing Changes -### Public API +## Public API -New methods will be added to the `org.apache.pulsar.client.api.TableViewBuilder` interface. +New generic methods will be added to the `org.apache.pulsar.client.api.TableViewBuilder` interface. * **Method Signatures**: ```java - TableView> createForMessages() throws PulsarClientException; + TableView createMapped(Function, V> mapper) throws PulsarClientException; - CompletableFuture>> createForMessagesAsync(); + CompletableFuture> createMappedAsync(Function, V> mapper); ``` -* **Description**: Creates a `TableView` instance where the values in the map are the full `Message` objects, including payload and metadata. This allows access to message properties, event time, etc. -* **Return Value**: A `TableView>` instance. +* **Description**: Creates a `TableView` instance where the values are the result of applying a user-defined `mapper` function to each message. This provides a flexible way to create a key-value view over a topic, allowing users to extract data from the message payload, properties, and other metadata into a custom object `V`. To get a view of the full `Message` objects, `java.util.function.Function.identity()` can be used as the mapper. +* **Parameters**: + * `mapper`: A function that takes a `Message` and returns a custom object of type `V`. +* **Return Value**: A `TableView` instance. +* **Behavior Notes**: + * If the `mapper` function returns `null`, it is treated as a tombstone message, and the corresponding key will be removed from the `TableView`. + * Exceptions thrown by the `mapper` function are not handled by the `TableView` itself. This may cause the consumer to get stuck and attempt to process the "poison pill" message repeatedly. Handling such failures is considered out of scope for this proposal. ### Binary protocol @@ -120,6 +125,12 @@ An alternative considered was to add a `getRawMessage(String key)` method direct This approach was rejected because it would be a **breaking change in terms of performance**. It would increase memory and CPU consumption for all `TableView` users, even those who do not need access to the raw message. The proposed builder-based approach is superior as it makes this an opt-in feature, preserving the performance characteristics of the existing `TableView`. +## Add specific `createForMessages()` methods + +Another alternative was to add specific, non-generic methods like `createForMessages()` that would always return a `TableView>`. + +This approach was rejected because it is less flexible than the mapper-based solution. The `createMapped` approach covers the `createForMessages` use case (via `Function.identity()`) while also empowering users to perform custom transformations, making it a more powerful and future-proof API. + # Links * Mailing List discussion thread: TBD From 1305faabd2316d65214a3283ba91a9ab6b59fa85 Mon Sep 17 00:00:00 2001 From: namest504 Date: Wed, 29 Oct 2025 15:08:18 +0900 Subject: [PATCH 4/7] update pip-445.mc --- pip/pip-445.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index da668c689330a..b9bd8910943d5 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -43,17 +43,17 @@ The proposed solution is a simple and non-breaking addition to the public client The changes will be confined to the Pulsar client library. -* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView`**: - * An abstract base class will be created to contain the common logic for `TableView` implementations, such as managing the underlying consumer and handling topic events. This prevents code duplication. +* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView`**: + * The existing `TableViewImpl` will be renamed and made abstract to become `AbstractTableView`. This base class will contain the common logic for different `TableView` implementations and will be generic on both the topic schema type `T` and the `TableView`'s value type `V`. -* **Class `org.apache.pulsar.client.impl.TableViewImpl`**: - * This class will be refactored to extend the new `AbstractTableView`. While its internal implementation will change to use the base class, its public-facing functionality and behavior will remain exactly the same. This ensures that the change has no impact on existing users. - -* **New Class `org.apache.pulsar.client.impl.MessageTableViewImpl`**: - * A new class will be created that extends `AbstractTableView>` and implements the `TableView>` interface. It will be responsible for storing the full `Message` object for each key, which its `get(key)` method will return. +* **New `TableView` implementation classes**: + * `org.apache.pulsar.client.impl.TableView`: A new class that extends `AbstractTableView` to provide the classic `TableView` behavior, mapping to the message's value. This ensures backward compatibility for existing users. + * `org.apache.pulsar.client.impl.MessageMapperTableView`: A new class that extends `AbstractTableView`. It implements the logic for the new `createMapped` methods, using a supplied `Function, V>` to transform messages into the values stored in the `TableView`. * **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: - * The new `createForMessages` methods will be implemented to instantiate a new `MessageTableViewImpl`. + * The builder's implementation will be updated to use the new classes. + * `create()` and `createAsync()` will now instantiate the new `org.apache.pulsar.client.impl.TableView`. + * The new `createMapped()` and `createMappedAsync()` methods will instantiate `MessageMapperTableView` with the user-provided mapper function. # Public-facing Changes From dc3f707c2c423df505b9be059bf42a94d05fc5c6 Mon Sep 17 00:00:00 2001 From: namest504 Date: Wed, 29 Oct 2025 15:12:58 +0900 Subject: [PATCH 5/7] update pip-445.md --- pip/pip-445.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index b9bd8910943d5..1c704f674f092 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -43,16 +43,17 @@ The proposed solution is a simple and non-breaking addition to the public client The changes will be confined to the Pulsar client library. -* **New Abstract Class `org.apache.pulsar.client.impl.AbstractTableView`**: - * The existing `TableViewImpl` will be renamed and made abstract to become `AbstractTableView`. This base class will contain the common logic for different `TableView` implementations and will be generic on both the topic schema type `T` and the `TableView`'s value type `V`. +* **Refactoring of `org.apache.pulsar.client.impl.TableViewImpl`**: + * The existing `TableViewImpl` class is refactored into a new inheritance structure. + * The core logic is moved to a new abstract base class, `org.apache.pulsar.client.impl.AbstractTableView`. + * To preserve the original functionality, a new simple class `org.apache.pulsar.client.impl.TableView` is introduced, which extends `AbstractTableView`. -* **New `TableView` implementation classes**: - * `org.apache.pulsar.client.impl.TableView`: A new class that extends `AbstractTableView` to provide the classic `TableView` behavior, mapping to the message's value. This ensures backward compatibility for existing users. +* **New `MessageMapperTableView` implementation**: * `org.apache.pulsar.client.impl.MessageMapperTableView`: A new class that extends `AbstractTableView`. It implements the logic for the new `createMapped` methods, using a supplied `Function, V>` to transform messages into the values stored in the `TableView`. * **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: * The builder's implementation will be updated to use the new classes. - * `create()` and `createAsync()` will now instantiate the new `org.apache.pulsar.client.impl.TableView`. + * `create()` and `createAsync()` will now instantiate the new `org.apache.pulsar.client.impl.TableView` to provide the classic `TableView` behavior. * The new `createMapped()` and `createMappedAsync()` methods will instantiate `MessageMapperTableView` with the user-provided mapper function. # Public-facing Changes From 58c926aa6452b73e9af40be1ea959038930fc306 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Feb 2026 12:32:37 +0200 Subject: [PATCH 6/7] Update mailing list links in pip-445.md --- pip/pip-445.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index 1c704f674f092..9bcb52f9cf39d 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -134,5 +134,5 @@ This approach was rejected because it is less flexible than the mapper-based sol # Links -* Mailing List discussion thread: TBD -* Mailing List voting thread: TBD \ No newline at end of file +* Mailing List discussion thread: https://lists.apache.org/thread/12jxo0n3njjpm17w2zdo2qvx1bs3y2dk +* Mailing List voting thread: https://lists.apache.org/thread/5qfjb8thvb9h8f81kg1xch67przd5w36 From 160f125cdadd0180949872c8912d34c808442b97 Mon Sep 17 00:00:00 2001 From: stlim Date: Fri, 10 Jul 2026 10:17:14 +0900 Subject: [PATCH 7/7] Align implementation details with review feedback - Use the TableViewImpl / AbstractTableViewImpl / MessageMapperTableViewImpl class names - Clarify message pooling behavior for mapped table views - Fix leftover createForMessages method name in the downgrade section --- pip/pip-445.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/pip/pip-445.md b/pip/pip-445.md index 9bcb52f9cf39d..8695a0d00d0db 100644 --- a/pip/pip-445.md +++ b/pip/pip-445.md @@ -45,16 +45,16 @@ The changes will be confined to the Pulsar client library. * **Refactoring of `org.apache.pulsar.client.impl.TableViewImpl`**: * The existing `TableViewImpl` class is refactored into a new inheritance structure. - * The core logic is moved to a new abstract base class, `org.apache.pulsar.client.impl.AbstractTableView`. - * To preserve the original functionality, a new simple class `org.apache.pulsar.client.impl.TableView` is introduced, which extends `AbstractTableView`. + * The core logic is moved to a new abstract base class, `org.apache.pulsar.client.impl.AbstractTableViewImpl`. + * To preserve the original functionality, the existing `org.apache.pulsar.client.impl.TableViewImpl` class becomes a simple subclass that extends `AbstractTableViewImpl`. -* **New `MessageMapperTableView` implementation**: - * `org.apache.pulsar.client.impl.MessageMapperTableView`: A new class that extends `AbstractTableView`. It implements the logic for the new `createMapped` methods, using a supplied `Function, V>` to transform messages into the values stored in the `TableView`. +* **New `MessageMapperTableViewImpl` implementation**: + * `org.apache.pulsar.client.impl.MessageMapperTableViewImpl`: A new class that extends `AbstractTableViewImpl`. It implements the logic for the new `createMapped` methods, using a supplied `Function, V>` to transform messages into the values stored in the `TableView`. * **Class `org.apache.pulsar.client.impl.TableViewBuilderImpl`**: * The builder's implementation will be updated to use the new classes. - * `create()` and `createAsync()` will now instantiate the new `org.apache.pulsar.client.impl.TableView` to provide the classic `TableView` behavior. - * The new `createMapped()` and `createMappedAsync()` methods will instantiate `MessageMapperTableView` with the user-provided mapper function. + * `create()` and `createAsync()` will keep instantiating `org.apache.pulsar.client.impl.TableViewImpl` to provide the classic `TableView` behavior. + * The new `createMapped()` and `createMappedAsync()` methods will instantiate `MessageMapperTableViewImpl` with the user-provided mapper function. # Public-facing Changes @@ -73,6 +73,7 @@ New generic methods will be added to the `org.apache.pulsar.client.api.TableView * `mapper`: A function that takes a `Message` and returns a custom object of type `V`. * **Return Value**: A `TableView` instance. * **Behavior Notes**: + * Message pooling is not used for mapped table views, so the `Message` instance passed to the `mapper` function can be safely retained (e.g. when `Function.identity()` is used as the mapper). * If the `mapper` function returns `null`, it is treated as a tombstone message, and the corresponding key will be removed from the `TableView`. * Exceptions thrown by the `mapper` function are not handled by the `TableView` itself. This may cause the consumer to get stuck and attempt to process the "poison pill" message repeatedly. Handling such failures is considered out of scope for this proposal. @@ -112,7 +113,7 @@ The upgrade process is seamless. Applications can update their client dependency ## Downgrade / Rollback -A downgrade is also seamless. If an application that uses the new `createForMessages` methods is rolled back to an older client version, it will fail at compile time. Applications that do not use the new methods can be rolled back without any issues. +A downgrade is also seamless. If an application that uses the new `createMapped` methods is rolled back to an older client version, it will fail at compile time. Applications that do not use the new methods can be rolled back without any issues. ## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations