[Go Functions] Most of ConsumerSpec is ignored: cryptoSpec, schemaProperties, consumerProperties, poolMessages

[david-streamlio]

Master Issue: #26404

Search before reporting

• I searched in the issues and found nothing similar.

Motivation

ConsumerSpec carries nine fields. The Go runtime reads two of them.

message ConsumerSpec {
    string schemaType = 1;                                  // read
    string serdeClassName = 2;                              // read (schema selection)
    bool isRegexPattern = 3;                                // read
    ReceiverQueueSize receiverQueueSize = 4;                // read
    map<string, string> schemaProperties = 5;               // NOT read
    map<string, string> consumerProperties = 6;             // NOT read
    CryptoSpec cryptoSpec = 7;                              // NOT read
    bool poolMessages = 8;                                  // NOT read
    MessagePayloadProcessorSpec messagePayloadProcessorSpec = 9;  // NOT read
}

git grep -in "CryptoSpec\|SchemaProperties\|ConsumerProperties\|PoolMessages\|MessagePayloadProcessor" -- 'pulsar-function-go/pf/*.go' returns nothing.

The Python runtime applies schemaProperties and cryptoSpec (python_instance.py builds a crypto key reader through get_crypto_reader(consumer_conf.cryptoSpec) and threads schemaProperties into get_schema), and the Java runtime applies all of them.

cryptoSpec is the one with real consequences: a Go function consuming from an encrypted topic gets no crypto key reader, so it cannot decrypt. The rest are quieter — consumerProperties are invisible to broker-side tooling that reads them, poolMessages leaves a performance option unavailable, and schemaProperties are dropped from schema resolution.

Solution

Apply the remaining ConsumerSpec fields in setupConsumer, mapping onto pulsar.ConsumerOptions. The Go client has an equivalent for most of them:

ConsumerSpec field	pulsar.ConsumerOptions
schemaProperties	schema properties on the resolved Schema
consumerProperties	Properties
cryptoSpec	Decryption (*MessageDecryptionInfo)
poolMessages	no direct equivalent — needs confirming
messagePayloadProcessorSpec	no equivalent — likely out of scope

Grouped into one issue because a single change to setupConsumer covers them, but they are separable and a partial fix is useful. Suggested order by impact: cryptoSpec, then consumerProperties and schemaProperties, then the two that may have no Go client equivalent.

Where a field genuinely cannot be supported, refusing it explicitly at startup — the way EFFECTIVELY_ONCE is refused at instanceConf.go:137 — would be better than continuing to ignore it. That is the point made in the master issue.

Alternatives

Leaving these unimplemented is defensible for messagePayloadProcessorSpec, which is a Java-centric extension point. It is not defensible for cryptoSpec, where the effect is that an encrypted topic cannot be consumed at all.

Anything else?

Verified against origin/master.

Are you willing to submit a PR?

• I'm willing to submit a PR!