E2E certification tests - PR1 by FabianMeiswinkel · Pull Request #5277 · Azure/azure-sdk-for-rust

Fabian Meiswinkel (FabianMeiswinkel) · 2026-09-10T16:36:13Z

Summary

Introduces the foundation for functional Cosmos DB Rust SDK E2E testing through the public azure_data_cosmos API.

PR1 scope

Language-neutral scenario metadata and implementation references.
Reusable account, runtime, and client setup profiles.
Source-native Rust test implementations and assertions.
Hosted in-memory emulator capability discovery.
Gateway V1 and Gateway V2 execution.
Initial smoke coverage:
- client bootstrap;
- item CRUD lifecycle and upsert;
- conflicts, wrong partition keys, and ETag concurrency;
- parameterized and invalid queries;
- critical diagnostics.
Initial consistency and configuration-default matrices.
CI profile selection and scheduled extended matrices.

Follow-up PRs

PR2 — Core operations and emulator fidelity: control plane, PK/HPK, feed ranges, change feed, batch, patch, validation limits, atomicity, continuation, and selected live differential baselines.
PR3 — Configuration and resilience: authentication and bootstrap fallback, broader configuration precedence, binary/Gateway/connection settings, session behavior, retries, timeouts, hedging, fault injection, diagnostics, and representative OpenTelemetry validation.
PR4 — Dynamic topology and availability: region and account failover, partition migration, split/merge phases, replication controls, routing refresh, PPAF/PPCB, and failback.
PR5 — Live matrix and promotion: Azure Live profiles and required coverage, complete diagnostics/OpenTelemetry audit, CI sharding, reports, runtime calibration, quarantine policy, documentation, and cross-SDK portability review.

# Conflicts: # sdk/cosmos/ci.yml

azure-pipelines · 2026-09-10T16:36:38Z

Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Copilot review overview

🟡 Changes recommended

One matrix references nonexistent profiles, and consistency tests can pass without proving their intended transient behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 3

New issues introduced by this change (3)

Severity	Finding
	`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/catalog.rs` — Catalog validation is not exhaustive because the scenario inventory is hard-coded. A new JSON…
	`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_lifecycle.rs` — The delayed consistency cases can pass without exercising the behavior they are intended to…
	`sdk/cosmos/e2e-profile-matrix.json` — These values cannot be consumed by the new setup path: `New-CosmosE2eEmulatorConfig` resolves each…

What changed in this PR

Introduces a metadata-driven Cosmos DB E2E testing framework using the public Rust SDK and hosted emulator.

Changes:

Adds scenario/profile schemas, metadata, and Rust implementations.
Adds emulator capability discovery and profile-driven setup.
Adds Gateway V1/V2 CI matrices for consistency testing.

File	Description
`sdk/cosmos/eng/scripts/Invoke-CosmosTestSetup.ps1`	Validates metadata and generates emulator configs.
`sdk/cosmos/e2e-read-consistency-override-matrix.json`	Defines read-consistency override jobs.
`sdk/cosmos/e2e-profile-matrix.json`	Defines a currently invalid profile matrix.
`sdk/cosmos/e2e-consistency-matrix.json`	Defines account-consistency jobs.
`sdk/cosmos/e2e_tests/schema/scenario.v1.json`	Defines scenario metadata schema.
`sdk/cosmos/e2e_tests/schema/profile.v1.json`	Defines setup-profile schema.
`sdk/cosmos/e2e_tests/scenarios/queries/parameterized-filter.json`	Catalogs parameterized query coverage.
`sdk/cosmos/e2e_tests/scenarios/queries/invalid-syntax.json`	Catalogs invalid-query coverage.
`sdk/cosmos/e2e_tests/scenarios/management/capabilities.json`	Catalogs capability discovery.
`sdk/cosmos/e2e_tests/scenarios/items/upsert-create-update.json`	Catalogs upsert behavior.
`sdk/cosmos/e2e_tests/scenarios/items/optimistic-concurrency.json`	Catalogs ETag concurrency behavior.
`sdk/cosmos/e2e_tests/scenarios/items/not-found-wrong-partition-key.json`	Catalogs not-found behavior.
`sdk/cosmos/e2e_tests/scenarios/items/lifecycle.json`	Catalogs item lifecycle behavior.
`sdk/cosmos/e2e_tests/scenarios/items/create-conflict.json`	Catalogs duplicate-create behavior.
`sdk/cosmos/e2e_tests/scenarios/diagnostics/success-and-error.json`	Catalogs diagnostics coverage.
`sdk/cosmos/e2e_tests/scenarios/bootstrap/primary-success.json`	Catalogs client bootstrap behavior.
`sdk/cosmos/e2e_tests/README.md`	Documents the E2E framework.
`sdk/cosmos/e2e_tests/profiles/targetDefault.json`	Adds a target-default profile.
`sdk/cosmos/e2e_tests/profiles/readConsistencyOverrideMatrix.json`	Adds override combinations.
`sdk/cosmos/e2e_tests/profiles/lifecycleConsistencyMatrix.json`	Adds consistency configurations.
`sdk/cosmos/e2e_tests/profiles/legacyGatewayV1.json`	Adds a Gateway V1 profile.
`sdk/cosmos/e2e_tests/profiles/hostedEmulatorSmoke.json`	Adds the smoke-test profile.
`sdk/cosmos/e2e_tests/implementations/rust.json`	Maps scenarios to Rust tests.
`sdk/cosmos/ci.yml`	Schedules extended E2E matrices.
`sdk/cosmos/azure_data_cosmos/tests/e2e_tests.rs`	Registers the E2E test target.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/support.rs`	Adds shared test helpers.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/query_parameterized_filter.rs`	Tests parameterized queries.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/query_invalid_syntax.rs`	Tests invalid query errors.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/mod.rs`	Registers E2E modules and tests.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_upsert.rs`	Tests upsert semantics.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_optimistic_concurrency.rs`	Tests stale ETag rejection.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_not_found.rs`	Tests partition-key isolation.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_lifecycle.rs`	Tests lifecycle consistency matrices.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_create_conflict.rs`	Tests conflicts across key types.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/fixture.rs`	Adds reusable resource fixtures.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/diagnostics_success_and_error.rs`	Tests public diagnostics.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/catalog.rs`	Validates catalog relationships.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/capabilities.rs`	Tests capability discovery.
`sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/bootstrap_primary.rs`	Tests primary endpoint bootstrap.
`sdk/cosmos/azure_data_cosmos/Cargo.toml`	Registers the E2E integration target.
`sdk/cosmos/azure_data_cosmos/build.rs`	Allows the `e2e` test category.
`sdk/cosmos/azure_data_cosmos_emulator/src/management.rs`	Adds the capabilities endpoint.
`sdk/cosmos/azure_data_cosmos_emulator/README.md`	Documents capability discovery.
`sdk/cosmos/azure_data_cosmos_driver/src/options/read_consistency.rs`	Clarifies regional consistency semantics.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

…to users/fabianm/E2ETests_PR1 # Conflicts: # sdk/cosmos/azure_data_cosmos/build.rs

Fabian Meiswinkel (FabianMeiswinkel) · 2026-09-13T03:35:04Z

/azp run rust - cosmos - weekly

azure-pipelines · 2026-09-13T03:35:28Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Fabian Meiswinkel (FabianMeiswinkel) · 2026-09-13T04:40:26Z

/azp run rust - cosmos - weekly

azure-pipelines · 2026-09-13T04:40:50Z

Azure Pipelines: Successfully started running 1 pipeline(s).

azure-pipelines · 2026-09-13T18:05:34Z

Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Ashley Stanton-Nurse (analogrelay)

Haven't dug through the implementations yet, but the infrastructure looks sound. If others approve the implementations, go ahead and merge, I'll try to review them myself in a bit though.

Ashley Stanton-Nurse (analogrelay)

Nothing critical, and it can all be addressed in follow-up PRs if you want to avoid churn on this one.

Ashley Stanton-Nurse (analogrelay) · 2026-09-15T18:46:51Z

+    pub async fn run<F>(test: F) -> TestResult
+    where
+        F: AsyncFnOnce(&E2eTestFixture) -> TestResult,
+    {
+        Self::run_with_partition_key("/pk".into(), test).await
+    }
+
+    pub async fn run_with_partition_key<F>(
+        partition_key: PartitionKeyDefinition,
+        test: F,
+    ) -> TestResult
+    where
+        F: AsyncFnOnce(&E2eTestFixture) -> TestResult,
+    {
+        let client = build_client().await?;
+        Self::run_with_client(client, partition_key, test).await
+    }
+
+    pub async fn run_with_client<F>(
+        client: CosmosClient,
+        partition_key: PartitionKeyDefinition,
+        test: F,
+    ) -> TestResult


How about a builder pattern?

E2eTest::builder() .with_partition_key_definition(...) .run(|f| { });

That avoids the ever-expanding run_... constructors.

Good idea for readability,

Ashley Stanton-Nurse (analogrelay) · 2026-09-15T18:50:59Z

+    }
+}
+
+fn read_cases_for_account_consistency(account: &AccountDefinition) -> Vec<PostCreateReadCase> {


I'm not 100% sure this reads better than just having separate test-case functions with ample use of helpers. This requires reading and understanding the cases, understanding all the setup and verification helpers to know what they do with these enum values, and then finally reading the test itself. Having each case be a literal test function, but one that leverages a few very common helpers for truly common behaviour (critically, the test function contains the logic that handles session tokens, or checks if the read succeeds immediately, etc.

Not blocking on this, but it would be a refactor towards clarity IMO. I'd actually prefer more lines of code here if they made it easier to review individual cases in isolation.

Agreed - ultimately main "quality" for source code of tests is readability

My concerns have been addressed and all source is under sdk/cosmos now.

Fabian Meiswinkel (FabianMeiswinkel) added 4 commits September 8, 2026 18:55

Initial framework for describing test matrix

44c9afd

Change what json files capture vs. source code

15ee860

Merge remote-tracking branch 'origin/main' into users/fabianm/E2ETests

ffcd2fd

# Conflicts: # sdk/cosmos/ci.yml

Formatting fixes

c25f571

Fabian Meiswinkel (FabianMeiswinkel) requested a review from a team as a code owner September 10, 2026 16:36

Copilot AI balanced review requested due to automatic review settings September 10, 2026 16:36

github-actions Bot added the Cosmos The azure_cosmos crate label Sep 10, 2026

Copilot started reviewing on behalf of Fabian Meiswinkel (FabianMeiswinkel) September 10, 2026 16:37 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/catalog.rs

Comment thread sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/item_lifecycle.rs Outdated

Comment thread sdk/cosmos/e2e-profile-matrix.json Outdated

Fabian Meiswinkel (FabianMeiswinkel) added 7 commits September 10, 2026 21:17

Refactored test cases for better readability

40c9808

Reacting to code review comments

9582713

Addressing Code review feedback

7688360

Update catalog.rs

aba0f5d

Applying code review feedback

401fa86

Apply code review feedback

7f025d5

Applying formatting

35c02bd

Fabian Meiswinkel (FabianMeiswinkel) requested review from a team, Rick Winter (RickWinter), Anton Kolesnyk (antkmsft), Heath Stewart (heaths) and Jeff Fisher (xirzec) as code owners September 11, 2026 21:55

Heath Stewart (heaths) previously requested changes Sep 11, 2026

View reviewed changes

Comment thread eng/scripts/Test-Packages.ps1 Outdated

Comment thread eng/scripts/Test-Packages.ps1 Outdated

Fabian Meiswinkel (FabianMeiswinkel) added 5 commits September 11, 2026 22:01

Add spec

2eeb5f1

Reacting to code review feedback

64259e2

Revert eng/* changes to test execution script

db94ade

Merge branch 'main' of https://github.com/Azure/azure-sdk-for-rust in…

d3d9989

…to users/fabianm/E2ETests_PR1 # Conflicts: # sdk/cosmos/azure_data_cosmos/build.rs

Addressing code review feedback

d11c39c

Fabian Meiswinkel (FabianMeiswinkel) marked this pull request as draft September 12, 2026 21:54

Fabian Meiswinkel (FabianMeiswinkel) requested a review from Heath Stewart (heaths) September 12, 2026 21:54

Fix session token capturing being also gated on session consistency

e849d65

Fabian Meiswinkel (FabianMeiswinkel) added 2 commits September 13, 2026 04:22

Fix test flakiness

2954d3a

Fix test failures

f25504d

Fabian Meiswinkel (FabianMeiswinkel) marked this pull request as ready for review September 13, 2026 18:05

Ashley Stanton-Nurse (analogrelay) reviewed Sep 14, 2026

View reviewed changes

Comment thread sdk/cosmos/e2e_tests/profiles/smokeTests.json

Comment thread sdk/cosmos/azure_data_cosmos/tests/e2e_test_cases/fixture.rs Outdated

Addressing code review comments

20161e7

Ashley Stanton-Nurse (analogrelay) approved these changes Sep 15, 2026

View reviewed changes

Uh oh!

Conversation

Fabian Meiswinkel (FabianMeiswinkel) commented Sep 10, 2026

Summary

PR1 scope

Follow-up PRs

Uh oh!

azure-pipelines Bot commented Sep 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fabian Meiswinkel (FabianMeiswinkel) commented Sep 13, 2026

Uh oh!

azure-pipelines Bot commented Sep 13, 2026

Uh oh!

Fabian Meiswinkel (FabianMeiswinkel) commented Sep 13, 2026

Uh oh!

azure-pipelines Bot commented Sep 13, 2026

Uh oh!

azure-pipelines Bot commented Sep 13, 2026

Uh oh!

Ashley Stanton-Nurse (analogrelay) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Ashley Stanton-Nurse (analogrelay) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Ashley Stanton-Nurse (analogrelay) Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Fabian Meiswinkel (FabianMeiswinkel) Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Ashley Stanton-Nurse (analogrelay) Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Fabian Meiswinkel (FabianMeiswinkel) Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants