Skip to content

fix(eventhubs): fold checkpoint blob key to lowercase ASCII - #5109

Open
Johnathan W (j7nw4r) wants to merge 5 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-blob-key-lowercase
Open

Johnathan W (j7nw4r) wants to merge 5 commits into
Azure:mainfrom
j7nw4r:j7nw4r/fix-eventhubs-blob-key-lowercase

Conversation

@j7nw4r

@j7nw4r Johnathan W (j7nw4r) commented Aug 20, 2026

Copy link
Copy Markdown
Member

Summary

BlobCheckpointStore built the checkpoint and ownership blob key from the fully qualified namespace, the event hub name, and the consumer group without folding their case. Event Hubs treats the consumer group as case insensitive, so a deployment that spelled the group $Default on one run and $default on the next built two disjoint key sets and reprocessed events. The four key builders now fold those three segments to lowercase ASCII, and the partition id keeps its case.

Motivation

Nothing in the crate normalized the three segments, and the service accepts both spellings of a consumer group as the same group. Two runs of one program therefore owned two disjoint sets of checkpoints. Neither run saw the other's progress, and a restart resumed from the wrong place. This defect stands on its own and needs no second SDK.

The cross language parity question is separate, and this pull request does not settle it. Three SDKs fold all three segments, which are .NET (BlobCheckpointStoreInternal.cs lines 142, 204, and 429), JavaScript (blobCheckpointStore.ts lines 282 to 284), and Python (_blobstoragecs.py). Three do not, which are Go (checkpoints/blob_store.go lines 274 to 301 format the fields verbatim), Java, and Rust. Folding does not restore interoperability. It moves this crate from the second group into the first, and a maintainer must make that choice before merge.

Changes

  • Checkpoint::get_checkpoint_blob_prefix_name and Ownership::get_ownership_prefix_name fold the namespace, the event hub name, and the consumer group with azure_core::fmt::to_ascii_lowercase.
  • Checkpoint::get_checkpoint_blob_name and Ownership::get_ownership_name inherit the fold and append the partition id unchanged.
  • The fold is ASCII only, not str::to_lowercase, because the standard library applies the context dependent Greek final sigma rule and would produce a key that differs from the .NET key.
  • The doc comments on all four public functions state the folding rule and name the layout.
  • Each changelog carries one Breaking Changes entry that gives the old key, the new key, and the migration.
  • assets.json moves from tag ..._d7273c4b84 to ..._a1ab5fd8f3, which folds the key in 13 recordings.
  • in_memory_checkpoint_store.rs and checkpoint_store.rs are unchanged, because every call site already routed through the four key builders.

Migration

Records that an older Rust client wrote stay at the old key and become unreachable. This change adds no dual read and no fallback lookup. A processor that starts against an existing container resumes from its configured start position.

Test plan

  • 8 new plain tests. Six unit tests sit inline in models.rs, and two drive the public CheckpointStore trait in tests/eventhubs_checkpoint_store.rs.
  • Seven were proved red against the unchanged source. key_is_stable_across_input_case states the defect without naming a fold direction, and checkpoint_key_survives_consumer_group_case_change failed with left: 0, right: 1.
  • key_functions_reject_empty_parameters is a characterization test and is green by design. Deleting the consumer_group guard turns it red, which was proved and then reverted.
  • fold_is_ascii_only also fails a str::to_lowercase fix, because the Greek pair must pass through unchanged.
  • The recorded assets are re-recorded. The new tag folds the key in the request URI, in the prefix query parameter, and in the Name and Prefix elements of the list responses. The percent escapes and the Variables block keep their case.
  • The recorded suite passes in CI on this branch head, across the seven Build Test matrix jobs and Build Analyze.
  • Live mode against a real storage account passes the whole crate. checkpoint_unit_tests reports 7 passed and 0 failed, and ownership_unit_tests reports 14 passed and 0 failed.
  • cargo fmt --check and cargo clippy --all-targets --all-features exit 0 for both crates on the pinned 1.95 toolchain.

Open decisions for a maintainer

  • The parity choice between the .NET group and the Go group, stated under Motivation.
  • to_ascii_lowercase in the published azure_core 1.1.0 mixes a char index and a byte index, so to_ascii_lowercase("\u{00e9}A") panics with start byte index 1 is not a char boundary. Event Hubs restricts these three names to ASCII, so no supported input reaches it. The four functions are public and take &str, so an upstream fix in typespec_client_core would remove the risk.

Closes #5099

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added Event Hubs Storage Storage Service (Queues, Blobs, Files) labels Aug 20, 2026
@j7nw4r Johnathan W (j7nw4r) self-assigned this Aug 20, 2026
@j7nw4r
Johnathan W (j7nw4r) force-pushed the j7nw4r/fix-eventhubs-blob-key-lowercase branch from cb256cb to d904c86 Compare August 25, 2026 18:13
The checkpoint and ownership blob keys keep the case of the caller's
namespace, event hub name, and consumer group. Two callers that spell
the same Event Hub with a different case land on two key sets, so a
checkpoint that one caller writes is invisible to the other.

Add a unit test module to the event processor models. It pins that
each of the three key fields folds to lowercase on its own, that the
partition id keeps its case, that the key is stable across the input
case, and that the fold is an ASCII fold. One characterization test
holds the empty parameter errors in place.

Add two checkpoint store tests. They store with a mixed case triple,
list with a lowercase triple, and expect the record back with its
stored case.
Event Hubs treats the fully qualified namespace, the event hub name,
and the consumer group as case insensitive. The checkpoint and
ownership blob key kept the case of the caller, so one deployment
that spelled the consumer group `$Default` on one run and `$default`
on the next built two disjoint key sets and reprocessed events.

The four key functions on `Checkpoint` and `Ownership` now fold the
three key fields with `azure_core::fmt::to_ascii_lowercase`. The
partition id keeps its case. The fold applies to ASCII letters only.
No call site changes, because every caller routes through these four
functions and no caller parses the three segments back out of a key.

Records that an older Rust client wrote stay at the old key and
become unreachable. The change adds no dual read and no fallback
lookup.

The recorded playback assets for the blob checkpoint store pin the
old casing, so that suite needs a re-record.

Fixes Azure#5099
The checkpoint and the ownership blob keys now fold to lowercase ASCII.
The recorded requests still hold the mixed case keys, so the test proxy
cannot match the request URI and 13 recorded tests fail in playback.

Fold the consumer group and the namespace to lowercase in the recorded
blob keys. The change touches the request URI, the `prefix` query
parameter, and the `Name` and `Prefix` elements in the list responses.
The percent escapes and the `Variables` block keep their case, so each
test still reads back the value that it recorded.

Point assets.json at the new tag.

Refs Azure#5099
@j7nw4r
Johnathan W (j7nw4r) force-pushed the j7nw4r/fix-eventhubs-blob-key-lowercase branch 2 times, most recently from 95dc958 to 37f7e94 Compare September 1, 2026 19:59
@j7nw4r
Johnathan W (j7nw4r) marked this pull request as ready for review September 1, 2026 20:26
Copilot AI balanced review requested due to automatic review settings September 1, 2026 20:26
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
3 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The breaking checkpoint migration and cross-SDK compatibility choice require maintainer approval.

Pull request overview

Normalizes Event Hubs checkpoint and ownership keys to prevent case-based state divergence.

Changes:

  • Lowercases ASCII namespace, Event Hub, and consumer-group key segments.
  • Adds normalization and case-preservation tests.
  • Updates migration notes and playback assets.
File summaries
File Description
azure_messaging_eventhubs/src/event_processor/models.rs Normalizes key builders and adds unit tests.
azure_messaging_eventhubs/tests/eventhubs_checkpoint_store.rs Tests case-insensitive store access.
azure_messaging_eventhubs/CHANGELOG.md Documents the breaking key change.
azure_messaging_eventhubs_checkpointstore_blob/CHANGELOG.md Documents migration impact.
azure_messaging_eventhubs_checkpointstore_blob/assets.json References refreshed playback recordings.
Review details

Suppressed comments (1)

sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_checkpoint_store.rs:244

  • New test functions should not use the redundant test_ prefix; the test attribute already identifies this as a test. Rename it to keep test naming aligned with the repository guidance.
async fn test_ownership_key_survives_consumer_group_case_change() {
  • Files reviewed: 5/5 changed files
  • Comments generated: 4
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/eventhubs/azure_messaging_eventhubs/CHANGELOG.md Outdated
Comment thread sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_checkpoint_store.rs Outdated
Comment thread sdk/eventhubs/azure_messaging_eventhubs_checkpointstore_blob/CHANGELOG.md Outdated
One breaking behavior change occupied four bullets in each changelog.
The repository changelog guidance asks for a single line for each
change to a public API.

Fold the motivation, the old and new key, and the migration note into
one entry in each file. No wording that a reader needs is dropped.
AGENTS.md asks that a test function name does not begin with "test"
unless the prefix disambiguates it from the function under test. The
prefix does no work here.

Rename the two new checkpoint store tests. The inline unit tests in
models.rs already follow the rule.
@j7nw4r
Johnathan W (j7nw4r) enabled auto-merge (squash) September 4, 2026 16:35

- The `default` feature now selects `fe2o3_amqp_rustls`, so AMQP framed directly on TCP (`amqps://`, port 5671) runs on rustls with the aws-lc-rs provider where it ran on native-tls. Both stacks read the trust store of the operating system, so a namespace behind a private or an enterprise certificate authority keeps working. The stacks read that store through different platform APIs, and a deployment that tunes native-tls directly, such as one that sets OpenSSL environment variables, can still see a difference. To keep native-tls, turn off the default features, name `fe2o3_amqp`, and take a direct dependency on `fe2o3-amqp` with its `native-tls` feature. ([#4189](https://github.com/Azure/azure-sdk-for-rust/issues/4189))
- On the receive path, the `amqp:link:stolen` AMQP condition is no longer auto-retried. A receiver displaced by a higher-or-equal-epoch attacher now surfaces the error (translated to `EventHubsError::ConsumerDisconnected` by `EventReceiver::stream_events`) instead of silently re-attaching. Sender, CBS, and management operations retain the historical retry-on-stolen behavior.
- The checkpoint and ownership blob key that `Checkpoint::get_checkpoint_blob_prefix_name`, `Checkpoint::get_checkpoint_blob_name`, `Ownership::get_ownership_prefix_name`, and `Ownership::get_ownership_name` build now folds the fully qualified namespace, the event hub name, and the consumer group to lowercase ASCII. The partition id keeps its case, so `NS.ServiceBus.Windows.Net/My-Hub/$Default/checkpoint/0` becomes `ns.servicebus.windows.net/my-hub/$default/checkpoint/0`, and the ownership key changes in the same way. Event Hubs treats the consumer group as case insensitive, so a deployment that spelled the group `$Default` on one run and `$default` on the next built two disjoint key sets and reprocessed events. Records that an older Rust client wrote stay at the old key and become unreachable. The change adds no dual read and no fallback lookup, and a processor that starts against an existing container resumes from its configured start position. ([#5099](https://github.com/Azure/azure-sdk-for-rust/issues/5099))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we explicitly call out that this migration is not safe for rolling upgrades and that the default fallback can skip events? Existing clients use the mixed-case ownership/checkpoint tree while upgraded clients use the lowercase tree, so mixed versions will not coordinate ownership. Also, when the lowercase checkpoint is not found, StartPosition::default() is Latest, meaning retained events after the old checkpoint can be skipped (while an Earliest configuration would replay). Please either add a compatibility/migration path or document a stop-the-world rollout: stop processors, migrate checkpoint blobs and metadata, let ownership records be recreated, then start only upgraded clients.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Event Hubs Storage Storage Service (Queues, Blobs, Files)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Event Hubs] BlobCheckpointStore does not lowercase the blob key, so Rust and .NET processors do not interoperate

3 participants