Skip to content

[improve][doc] Note runtime support for maxMessageRetries and deadLetterTopic - #1214

Open
david-streamlio wants to merge 2 commits into
apache:mainfrom
david-streamlio:docs-python-fn-dlq
Open

[improve][doc] Note runtime support for maxMessageRetries and deadLetterTopic#1214
david-streamlio wants to merge 2 commits into
apache:mainfrom
david-streamlio:docs-python-fn-dlq

Conversation

@david-streamlio

Copy link
Copy Markdown
Contributor

Motivation

The FunctionConfig table in docs/functions-cli.md documents maxMessageRetries and deadLetterTopic with no runtime qualification, so a reader has no way to tell that the Go runtime does not honour either field.

This matters now because apache/pulsar#26400 makes the Python runtime honour both. Before that PR the Python runtime ignored them silently — functions get echoed the configuration back faithfully and nothing was ever routed to the dead letter topic. Recording which runtimes support what is what keeps that class of surprise from recurring.

The note added when apache/pulsar#6084 was closed in 2020 — "This parameter is not supported in Python Functions" — was dropped from the live docs in the 2.11 functions-cli.md rewrite. It survives only in versioned_docs/version-2.3.0 through version-2.10.x, where it is accurate for those releases and is left untouched. So there is no incorrect statement to retract here, only a missing caveat to add.

Modifications

Add an inline **Note:** to both rows, in the style the table already uses for topicsPattern and autoAck:

  • maxMessageRetries — supported by Java and Python; Python requires a value of at least 1, because the Python client's ConsumerDeadLetterPolicy rejects a redelivery count below 1. Go does not support it.
  • deadLetterTopic — supported by Java and Python, and requires maxMessageRetries to be set (FunctionConfigUtils.doCommonChecks rejects a dead letter topic with retries unset: "Dead Letter Topic specified, however max retries is set to infinity"). Go does not support it.

Also correct one unrelated error in the same table, noticed while editing it: the maxPendingAsyncRequests row listed its Related Command Argument as --max-message-retries. That flag belongs to maxMessageRetries, two rows above. maxPendingAsyncRequests is a FunctionConfig field with no corresponding flag in CmdFunctions, so the column is now N/A, matching the other flagless rows. Happy to split this into its own PR if preferred.

Versioned docs

docs/ only, deliberately. Python support for these fields ships with apache/pulsar#26400 and is not in any released version, so back-porting the note to versioned_docs/ would describe behaviour those releases do not have.

Documentation

This is the documentation change. The corresponding code change is apache/pulsar#26400, which also updates the #Java#Java, Python runtime marker on both @Option descriptions in CmdFunctions, so the Support column of the generated pulsar-admin CLI reference stays in step with this table.

…terTopic

The FunctionConfig table documents both fields with no runtime qualification,
so a reader has no way to tell that Go functions do not honour them, or that
Python requires a redelivery count of at least 1.

apache/pulsar#26400 makes the Python runtime honour both, so record which
runtimes support them, in the same inline "**Note:**" style the table already
uses for topicsPattern and autoAck.

Also correct the maxPendingAsyncRequests row, which credited it with
`--max-message-retries`. That flag belongs to maxMessageRetries;
maxPendingAsyncRequests is a FunctionConfig field with no CLI flag, so its
Related Command Argument is N/A.
@david-streamlio

Copy link
Copy Markdown
Contributor Author

Cross-linking: this documents the runtime behaviour implemented in apache/pulsar#26400 (deadLetterTopic and maxMessageRetries support for Python functions), which currently carries doc-required pending this PR.

So the two are a pair — merging this clears the label on the code PR.

Motivation:

The Python function runtime passes a dead_letter_policy to
Client.subscribe() for every input topic. That parameter was added in
pulsar-client-python 3.3.0, so 3.3.0 is now the runtime's minimum; every
other argument the runtime passes is available in earlier releases.
Nothing on the site stated a minimum, and the Python packaging page told
users to install 2.10.0, which is below it.

Modifications:

- functions-package-python.md: install `pulsar-client>=3.3.0` rather than
  pinning 2.10.0, with a note explaining where the floor comes from and
  that it only matters when the reader supplies the Python environment
  themselves - the Docker images already install a newer client.

- functions-cli.md: note the same requirement on the maxMessageRetries
  and deadLetterTopic rows, and record that an empty deadLetterTopic
  leaves the client to derive `<topic>-<subscription>-DLQ`
  (RetryMessageUtil.getDLQTopic for Java, the documented
  ConsumerDeadLetterPolicy default for Python).
@david-streamlio

Copy link
Copy Markdown
Contributor Author

Pushed 000346a, which adds the pulsar-client requirement @lhotari asked for in
apache/pulsar#26400.

One thing that turned up while writing it: the requirement is not specific to maxMessageRetries /
deadLetterTopic. The Python runtime passes dead_letter_policy to Client.subscribe() for every
input topic — as None when nothing is configured — so pulsar-client 3.3.0 is the floor for all
Python functions, not just ones using a dead letter topic. I diffed every keyword the runtime passes
against the 3.2.0 signature and dead_letter_policy is the only one missing, so 3.3.0 is exactly the
floor and nothing else in the runtime was already above it.

That is why the main note went on Package Python functions rather than only into the CLI table.
While there I also changed the documented install from pulsar-client==2.10.0 to
pulsar-client>=3.3.0 — following the page as written would have produced a runtime that fails at
subscribe(). Happy to split that into its own PR if you would rather keep this one to the CLI table.

The functions-cli.md rows now also record that an empty deadLetterTopic leaves the client to
derive <topic>-<subscription>-DLQ. That holds for both runtimes the rows cover —
RetryMessageUtil.getDLQTopic() is topic + "-" + subscription + "-DLQ", and the Python client
documents the same default.

I have not rendered the site locally, so the note block and table are checked structurally rather
than visually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant