[Bug] Sink inputSpecs schemaProperties is silently dropped by SinkConfigUtils in both conversion directions

[david-streamlio]

Search before asking

• I searched in the issues and found nothing similar.

Version

Reproduced on master (e4d106f6ee). The same omission is present on the 4.0.x line.

Minimal reproduce step

Create a sink whose inputSpecs entry carries schemaProperties:

{
  "tenant": "public",
  "namespace": "default",
  "name": "test-sink",
  "inputSpecs": {
    "persistent://public/default/in": {
      "schemaType": "avro",
      "schemaProperties": { "__alwaysAllowNull": "true" },
      "consumerProperties": { "application": "billing" }
    }
  }
}

Then read it back:

pulsar-admin sinks get --tenant public --namespace default --name test-sink

What did you expect to see?

schemaProperties preserved on the round trip, the same way consumerProperties is, and the same way it already works for Pulsar Functions.

What did you see instead?

schemaProperties is silently dropped. The API accepts the value, but it never reaches ConsumerSpec and is absent from the response. There is no error or warning, so the caller has no signal that the setting was discarded.

Anything else?

The proto field exists — Function.proto:111 defines map<string, string> schemaProperties = 5 on ConsumerSpec — and ConsumerConfig.schemaProperties exists on the model. Only the sink conversion path fails to use them, in both directions:

• SinkConfigUtils.convert() builds each ConsumerSpec from the inputSpecs entry and copies schemaType/serdeClassName, receiverQueueSize, cryptoSpec, messagePayloadProcessorSpec, consumerProperties and poolMessages — but never schemaProperties.
• SinkConfigUtils.convertFromDetails() restores the same set on the way back, and likewise never restores schemaProperties.

FunctionConfigUtils handles the field correctly in both directions (putSchemaProperties at lines 137/161/248, forEachSchemaProperties at line 432), which is what makes this sink-specific rather than a general ConsumerConfig limitation.

Practical impact. Any client that exposes schemaProperties on a sink input spec exposes a write-only field: the value is accepted, dropped, and then reads back absent, producing permanent drift for declarative tooling. This surfaced while reviewing a Terraform provider change (streamnative/terraform-provider-pulsar#224), where exposing the attribute had to be deferred for exactly this reason.

Suggested fix: copy schemaProperties in both convert() and convertFromDetails(), mirroring the consumerProperties handling, and add round-trip coverage in SinkConfigUtilsTest.

I am happy to put up a PR for this.

Are you willing to submit a PR?

• I'm willing to submit a PR!