Skip to content

[fix][fn] Honour retainOrdering and retainKeyOrdering in the Go function runtime - #26414

Merged
david-streamlio merged 2 commits into
apache:masterfrom
david-streamlio:fix-go-fn-ordering
Aug 25, 2026
Merged

[fix][fn] Honour retainOrdering and retainKeyOrdering in the Go function runtime#26414
david-streamlio merged 2 commits into
apache:masterfrom
david-streamlio:fix-go-fn-ordering

Conversation

@david-streamlio

Copy link
Copy Markdown
Contributor

Fixes #26405
Master Issue: #26404

Motivation

The Go runtime picked the subscription type from one field:

// pulsar-function-go/pf/instance.go:307-310
subscriptionType := pulsar.Shared
if int32(gi.context.instanceConf.funcDetails.Source.SubscriptionType) == pb.SubscriptionType_value["FAILOVER"] {
	subscriptionType = pulsar.Failover
}

RetainOrdering and RetainKeyOrdering appeared nowhere in pulsar-function-go outside the generated pb package, so both were accepted by pulsar-admin, reported back by functions get, carried into the instance in the protobuf, and dropped.

For retainKeyOrdering that is a correctness problem rather than a missing feature. This is accepted:

pulsar-admin functions create --go fn --retain-key-ordering ...

and the function runs on a Shared subscription, so messages sharing a key are distributed across instances and processed concurrently — precisely the guarantee the flag exists to provide. Nothing fails, nothing is logged, and the only symptom is out-of-order processing observed downstream.

The Python runtime applies both (python_instance.py:147-151), as does Java.

Modifications

Extract resolveSubscriptionType and apply the ordering flags after the explicit SubscriptionType:

  • retainOrderingFailover
  • retainKeyOrderingKeyShared
  • ordering wins when both are set, which is the precedence python_instance.py applies in its if/elif

EFFECTIVELY_ONCE needs no arm here: instanceConf.go:137 refuses it before an instance is built.

The helper takes the three fields rather than a FunctionDetails so it is directly testable and so it does not copy a protobuf message by value — go vet reports that as copying a lock, and the version of this change that passed the struct added three new warnings on top of the three already present on master.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Six table-driven cases in pf/subscriptionType_test.go: the Shared default; an explicit Failover; retainOrdering selecting Failover; retainKeyOrdering selecting KeyShared; ordering winning when both are set; and retainKeyOrdering overriding an explicit Failover.
  • Confirmed the tests are not vacuous: removing the two new arms fails the suite.
  • go build ./... and the full go test ./pf/ pass; go vet reports the same three pre-existing lock-copy warnings as master and no new ones.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • Anything that affects deployment

Deployment note. A Go function already deployed with --retain-ordering or --retain-key-ordering changes subscription type on the next restart — Shared to Failover or KeyShared respectively. That is the configured behaviour taking effect for the first time, but it is a change for anyone who set the flag, observed Shared behaviour and adapted to it. A function that sets neither flag is unaffected.

Documentation

  • doc-required
  • doc-not-needed
  • doc
  • doc-complete

Both options are already documented; this makes the Go runtime honour them.

…ion runtime

setupConsumer picked the subscription type from SubscriptionType alone.
RetainOrdering and RetainKeyOrdering appeared nowhere in
pulsar-function-go outside the generated protobuf, so both were accepted
by pulsar-admin, reported back by functions get, carried into the instance
and then dropped.

The consequence for retainKeyOrdering is a correctness one rather than a
missing feature: the function ran on a Shared subscription, so messages
sharing a key were distributed across instances and processed
concurrently - exactly the guarantee the flag exists to provide. Nothing
failed and nothing logged; the ordering was simply absent.

Extract resolveSubscriptionType and apply both flags after the explicit
SubscriptionType, matching the Java and Python runtimes. retainOrdering
selects Failover, retainKeyOrdering selects KeyShared, and ordering wins
when both are set - the precedence python_instance.py applies in its
if/elif.

EFFECTIVELY_ONCE needs no arm: instanceConf.go refuses it before an
instance is built.

The helper takes the three fields rather than FunctionDetails so it can be
tested directly and so it does not copy a protobuf message by value, which
go vet reports as copying a lock.

Fixes apache#26405
Master Issue: apache#26404
Resolves a conflict in pulsar-function-go/pf/instance.go with apache#26415
(negativeAckRedeliveryDelayMs), which landed on master and added
resolveNackRedeliveryDelay at the same insertion point ahead of
setupConsumer.

Both helpers are kept. setupConsumer now resolves the subscription type
through resolveSubscriptionType and the redelivery delay through
resolveNackRedeliveryDelay, and all four Subscribe call sites carry both.
@david-streamlio
david-streamlio merged commit 0897616 into apache:master Aug 25, 2026
44 checks passed
@david-streamlio
david-streamlio deleted the fix-go-fn-ordering branch August 25, 2026 14:34
@lhotari lhotari added this to the 5.0.0-M2 milestone Aug 27, 2026
lhotari pushed a commit that referenced this pull request Aug 27, 2026
lhotari pushed a commit that referenced this pull request Aug 27, 2026
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go Functions] retainOrdering and retainKeyOrdering are ignored, so key ordering is silently not preserved

3 participants