fix(eventhubs): fold checkpoint blob key to lowercase ASCII - #5109
Johnathan W (j7nw4r) wants to merge 5 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
cb256cb to
d904c86
Compare
The checkpoint and ownership blob keys keep the case of the caller's namespace, event hub name, and consumer group. Two callers that spell the same Event Hub with a different case land on two key sets, so a checkpoint that one caller writes is invisible to the other. Add a unit test module to the event processor models. It pins that each of the three key fields folds to lowercase on its own, that the partition id keeps its case, that the key is stable across the input case, and that the fold is an ASCII fold. One characterization test holds the empty parameter errors in place. Add two checkpoint store tests. They store with a mixed case triple, list with a lowercase triple, and expect the record back with its stored case.
Event Hubs treats the fully qualified namespace, the event hub name, and the consumer group as case insensitive. The checkpoint and ownership blob key kept the case of the caller, so one deployment that spelled the consumer group `$Default` on one run and `$default` on the next built two disjoint key sets and reprocessed events. The four key functions on `Checkpoint` and `Ownership` now fold the three key fields with `azure_core::fmt::to_ascii_lowercase`. The partition id keeps its case. The fold applies to ASCII letters only. No call site changes, because every caller routes through these four functions and no caller parses the three segments back out of a key. Records that an older Rust client wrote stay at the old key and become unreachable. The change adds no dual read and no fallback lookup. The recorded playback assets for the blob checkpoint store pin the old casing, so that suite needs a re-record. Fixes Azure#5099
The checkpoint and the ownership blob keys now fold to lowercase ASCII. The recorded requests still hold the mixed case keys, so the test proxy cannot match the request URI and 13 recorded tests fail in playback. Fold the consumer group and the namespace to lowercase in the recorded blob keys. The change touches the request URI, the `prefix` query parameter, and the `Name` and `Prefix` elements in the list responses. The percent escapes and the `Variables` block keep their case, so each test still reads back the value that it recorded. Point assets.json at the new tag. Refs Azure#5099
95dc958 to
37f7e94
Compare
|
Azure Pipelines: Successfully started running 1 pipeline(s). 3 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🔵 Needs a closer look
The breaking checkpoint migration and cross-SDK compatibility choice require maintainer approval.
Pull request overview
Normalizes Event Hubs checkpoint and ownership keys to prevent case-based state divergence.
Changes:
- Lowercases ASCII namespace, Event Hub, and consumer-group key segments.
- Adds normalization and case-preservation tests.
- Updates migration notes and playback assets.
File summaries
| File | Description |
|---|---|
azure_messaging_eventhubs/src/event_processor/models.rs |
Normalizes key builders and adds unit tests. |
azure_messaging_eventhubs/tests/eventhubs_checkpoint_store.rs |
Tests case-insensitive store access. |
azure_messaging_eventhubs/CHANGELOG.md |
Documents the breaking key change. |
azure_messaging_eventhubs_checkpointstore_blob/CHANGELOG.md |
Documents migration impact. |
azure_messaging_eventhubs_checkpointstore_blob/assets.json |
References refreshed playback recordings. |
Review details
Suppressed comments (1)
sdk/eventhubs/azure_messaging_eventhubs/tests/eventhubs_checkpoint_store.rs:244
- New test functions should not use the redundant
test_prefix; the test attribute already identifies this as a test. Rename it to keep test naming aligned with the repository guidance.
async fn test_ownership_key_survives_consumer_group_case_change() {
- Files reviewed: 5/5 changed files
- Comments generated: 4
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
One breaking behavior change occupied four bullets in each changelog. The repository changelog guidance asks for a single line for each change to a public API. Fold the motivation, the old and new key, and the migration note into one entry in each file. No wording that a reader needs is dropped.
AGENTS.md asks that a test function name does not begin with "test" unless the prefix disambiguates it from the function under test. The prefix does no work here. Rename the two new checkpoint store tests. The inline unit tests in models.rs already follow the rule.
|
|
||
| - The `default` feature now selects `fe2o3_amqp_rustls`, so AMQP framed directly on TCP (`amqps://`, port 5671) runs on rustls with the aws-lc-rs provider where it ran on native-tls. Both stacks read the trust store of the operating system, so a namespace behind a private or an enterprise certificate authority keeps working. The stacks read that store through different platform APIs, and a deployment that tunes native-tls directly, such as one that sets OpenSSL environment variables, can still see a difference. To keep native-tls, turn off the default features, name `fe2o3_amqp`, and take a direct dependency on `fe2o3-amqp` with its `native-tls` feature. ([#4189](https://github.com/Azure/azure-sdk-for-rust/issues/4189)) | ||
| - On the receive path, the `amqp:link:stolen` AMQP condition is no longer auto-retried. A receiver displaced by a higher-or-equal-epoch attacher now surfaces the error (translated to `EventHubsError::ConsumerDisconnected` by `EventReceiver::stream_events`) instead of silently re-attaching. Sender, CBS, and management operations retain the historical retry-on-stolen behavior. | ||
| - The checkpoint and ownership blob key that `Checkpoint::get_checkpoint_blob_prefix_name`, `Checkpoint::get_checkpoint_blob_name`, `Ownership::get_ownership_prefix_name`, and `Ownership::get_ownership_name` build now folds the fully qualified namespace, the event hub name, and the consumer group to lowercase ASCII. The partition id keeps its case, so `NS.ServiceBus.Windows.Net/My-Hub/$Default/checkpoint/0` becomes `ns.servicebus.windows.net/my-hub/$default/checkpoint/0`, and the ownership key changes in the same way. Event Hubs treats the consumer group as case insensitive, so a deployment that spelled the group `$Default` on one run and `$default` on the next built two disjoint key sets and reprocessed events. Records that an older Rust client wrote stay at the old key and become unreachable. The change adds no dual read and no fallback lookup, and a processor that starts against an existing container resumes from its configured start position. ([#5099](https://github.com/Azure/azure-sdk-for-rust/issues/5099)) |
There was a problem hiding this comment.
Could we explicitly call out that this migration is not safe for rolling upgrades and that the default fallback can skip events? Existing clients use the mixed-case ownership/checkpoint tree while upgraded clients use the lowercase tree, so mixed versions will not coordinate ownership. Also, when the lowercase checkpoint is not found, StartPosition::default() is Latest, meaning retained events after the old checkpoint can be skipped (while an Earliest configuration would replay). Please either add a compatibility/migration path or document a stop-the-world rollout: stop processors, migrate checkpoint blobs and metadata, let ownership records be recreated, then start only upgraded clients.
Summary
BlobCheckpointStorebuilt the checkpoint and ownership blob key from the fully qualified namespace, the event hub name, and the consumer group without folding their case. Event Hubs treats the consumer group as case insensitive, so a deployment that spelled the group$Defaulton one run and$defaulton the next built two disjoint key sets and reprocessed events. The four key builders now fold those three segments to lowercase ASCII, and the partition id keeps its case.Motivation
Nothing in the crate normalized the three segments, and the service accepts both spellings of a consumer group as the same group. Two runs of one program therefore owned two disjoint sets of checkpoints. Neither run saw the other's progress, and a restart resumed from the wrong place. This defect stands on its own and needs no second SDK.
The cross language parity question is separate, and this pull request does not settle it. Three SDKs fold all three segments, which are .NET (
BlobCheckpointStoreInternal.cslines 142, 204, and 429), JavaScript (blobCheckpointStore.tslines 282 to 284), and Python (_blobstoragecs.py). Three do not, which are Go (checkpoints/blob_store.golines 274 to 301 format the fields verbatim), Java, and Rust. Folding does not restore interoperability. It moves this crate from the second group into the first, and a maintainer must make that choice before merge.Changes
Checkpoint::get_checkpoint_blob_prefix_nameandOwnership::get_ownership_prefix_namefold the namespace, the event hub name, and the consumer group withazure_core::fmt::to_ascii_lowercase.Checkpoint::get_checkpoint_blob_nameandOwnership::get_ownership_nameinherit the fold and append the partition id unchanged.str::to_lowercase, because the standard library applies the context dependent Greek final sigma rule and would produce a key that differs from the .NET key.assets.jsonmoves from tag..._d7273c4b84to..._a1ab5fd8f3, which folds the key in 13 recordings.in_memory_checkpoint_store.rsandcheckpoint_store.rsare unchanged, because every call site already routed through the four key builders.Migration
Records that an older Rust client wrote stay at the old key and become unreachable. This change adds no dual read and no fallback lookup. A processor that starts against an existing container resumes from its configured start position.
Test plan
models.rs, and two drive the publicCheckpointStoretrait intests/eventhubs_checkpoint_store.rs.key_is_stable_across_input_casestates the defect without naming a fold direction, andcheckpoint_key_survives_consumer_group_case_changefailed withleft: 0, right: 1.key_functions_reject_empty_parametersis a characterization test and is green by design. Deleting theconsumer_groupguard turns it red, which was proved and then reverted.fold_is_ascii_onlyalso fails astr::to_lowercasefix, because the Greek pair must pass through unchanged.prefixquery parameter, and in theNameandPrefixelements of the list responses. The percent escapes and theVariablesblock keep their case.checkpoint_unit_testsreports 7 passed and 0 failed, andownership_unit_testsreports 14 passed and 0 failed.cargo fmt --checkandcargo clippy --all-targets --all-featuresexit 0 for both crates on the pinned 1.95 toolchain.Open decisions for a maintainer
to_ascii_lowercasein the publishedazure_core1.1.0 mixes a char index and a byte index, soto_ascii_lowercase("\u{00e9}A")panics withstart byte index 1 is not a char boundary. Event Hubs restricts these three names to ASCII, so no supported input reaches it. The four functions are public and take&str, so an upstream fix intypespec_client_corewould remove the risk.Closes #5099