[fix][sec] Upgrade Netty to 4.2.17 to address several CVEs and bugs

[lhotari]

Motivation

Netty 4.2.17.Final is a bug-fix and security
release, and upstream "strongly recommends" upgrading.

Security fixes. It resolves seven advisories (five of them still have their CVE id pending
assignment upstream, so the release notes show CVE-2026-XXXXX):

Advisory	Severity	Netty artifact	Summary
GHSA-p85m-gvr3-788c	High	netty-handler	Hostname verification disabled on the OpenSSL client path when trust-manager wrapping is unavailable (Java 25+)
GHSA-c4c3-7fpv-j4q5	High	netty-handler	SNI routing bypass via fragmented TLS ClientHello causing fallback to the default SslContext
GHSA-fccg-mwvh-qqg4	Medium	netty-handler	Fragmented ClientHello records trigger quadratic pre-handshake reassembly in default SNI parsing
GHSA-cc6x-ffm5-83wf	Medium	netty-codec-socks	SOCKS4/5 proxy encoder NUL byte and credential injection
CVE-2026-59903	Medium	netty-codec-http	Cache poisoning and information disclosure via CORS Vary header overwrite
CVE-2026-59902	High	netty-transport-sctp	Memory exhaustion in SctpMessageCompletionHandler
GHSA-43fm-7cxg-hf3j	Low	netty-codec-mqtt	MQTT topic name and client id validation bypass

netty-transport-sctp and netty-codec-mqtt are not bundled in Pulsar's distributions (they don't
appear in either LICENSE.bin.txt), so the last two are listed only for completeness.
netty-handler, netty-codec-socks and netty-codec-http are all shipped.

GHSA-p85m-gvr3-788c is the one worth calling out. Netty's earlier CVE-2026-50010 fix added hostname
verification to a plain X509TrustManager by wrapping it, and on the SslProvider.OPENSSL path that
wrapping is Unsafe-based reflection. Unsafe is not available by default on Java 25+, so the
wrapper degrades to a no-op and an OpenSSL client configured with a plain (non-extended)
X509TrustManager ends up doing no hostname verification. The two environmental preconditions are
ordinary for Pulsar on master: an unset tlsProvider resolves to OPENSSL_REFCNT whenever the
native engine is available (TlsFactorySupport), and master now defaults to JDK 25 (#26070).
Whether verification is actually lost additionally depends on a plain rather than extended trust
manager being in play, so this mainly concerns deployments supplying their own trust manager via a
custom PulsarTlsFactory; the upgrade removes the question either way.

Non-security fixes. 4.2.17.Final also carries several fixes in areas Pulsar leans on heavily:

• Fix buddy cache evicting chunks with live buffers (#17154)
• Fix direct memory OOM on low-core containers (#17166)
• Reject negative maxOrder in PooledByteBufAllocator (#17093)
• Fix AdaptiveByteBuf._setLongLE calling checked setLongLE (#17098)
• SslHandler: fix possible buffer leak when an OOME is thrown during allocation (#17059)
• Weakly reference engines from the OpenSSL engine map (#17199)
• Avoid classloader leak via GlobalEventExecutor terminationFuture failure (#17140)
• HttpObjectEncoder / DefaultHttp2FrameWriter: fix buffer leak when a Throwable is thrown during header encoding (#17089)
• AsciiString.cached(String) sanitization and the follow-up performance regression fix (#17007, #17074)
• io_uring: preserve readPending when rescheduling canceled reads and fix the recvmmsg emulation (#17087, #17187)

netty-tcnative. Bumped 2.0.78.Final → 2.0.81.Final to stay aligned with what
netty-parent:4.2.17.Final itself declares (<tcnative.version>2.0.81.Final</tcnative.version>).
The changes across those three releases
are:

• Clear the certificate chain in setKeyMaterial (netty-tcnative#987)
• Drop checks on SSL_CREDENTIAL support for older BoringSSL (netty-tcnative#980)
• Add SSL.getGroupName(...), returning the named group used by the most recently completed handshake (netty-tcnative#991) — this is what backs Netty's new OpenSslSession named-group accessor (netty#17058)
• Link the linux aarch_64 and x86_64 openssl-dynamic artifacts against OpenSSL 3.x (netty-tcnative#986, netty-tcnative#989) — not applicable to Pulsar, which bundles netty-tcnative-boringssl-static

Modifications

• gradle/libs.versions.toml: netty 4.2.16.Final → 4.2.17.Final, netty-tcnative 2.0.78.Final → 2.0.81.Final
• Update the bundled jar lists in distribution/server/src/assemble/LICENSE.bin.txt and
  distribution/shell/src/assemble/LICENSE.bin.txt to the new versions

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests. In addition to CI, ./gradlew checkBinaryLicense
was run locally for both the server and shell distributions to confirm the LICENSE.bin.txt entries
match the jars that are actually bundled.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Netty is upgraded from 4.2.16.Final to 4.2.17.Final and netty-tcnative from 2.0.78.Final to
2.0.81.Final.