Skip to content

fix(sink): set the producer compression codec once - #167

Merged
catinspace-au merged 2 commits into
mainfrom
fix/compression-alias
Nov 15, 2026
Merged

catinspace-au merged 2 commits into
mainfrom
fix/compression-alias

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

Vector writes sink.compression as compression.codec. The generator also wrote compression.type. librdkafka reads both as one setting. rdkafka applies a client's options in hash order.

With compression.codec: gzip in sink.librdkafka_options, 30 starts of Vector 0.61.0 on the generated sink gave 12 gzip and 18 zstd. The generated sink alone gave 30 zstd, because both spellings carried the same value.

The sink sets the codec once through Vector's compression field. It drops compression.codec and compression.type from the profile's librdkafka_options. Both spellings are derived options, so config validation rejects either in sink.librdkafka_options.

Vector writes sink.compression as compression.codec. The generator also wrote compression.type. librdkafka reads both as one setting. rdkafka applies a client's options in hash order.

With compression.codec: gzip in sink.librdkafka_options, 30 starts of Vector 0.61.0 on the generated sink gave 12 gzip and 18 zstd. The generated sink alone gave 30 zstd, because both spellings carried the same value.

The sink sets the codec once through Vector's compression field. It drops compression.codec and compression.type from the profile's librdkafka_options. Both spellings are derived options, so config validation rejects either in sink.librdkafka_options.
The profile constants file is in scalo-rs. The hyperi-ci doc-paths check treats the path in docs/librdkafka.md as a missing local file. Replace the path with a link to that file.
@catinspace-au
catinspace-au merged commit 3f83b17 into main Nov 15, 2026
13 checks passed
@catinspace-au
catinspace-au deleted the fix/compression-alias branch November 15, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant