fix: make frontend cluster create idempotent to prevent CS duplicate collisions#6025
fix: make frontend cluster create idempotent to prevent CS duplicate collisions#6025cssjr wants to merge 1 commit into
Conversation
…collisions The frontend's createHCPCluster() performs a two-phase write: it creates the cluster in Cluster Service (CS) first, then writes to Cosmos DB. If the Cosmos write fails, CS has the cluster but the RP doesn't — a split-brain state. When the Azure SDK automatically retries the failed 500, the retry calls PostCluster again and CS rejects with "Duplicate ARO-HCP cluster name" (400 InvalidRequestContent). This was hitting ~7.2% of CI E2E runs (ARO-27951) and was confirmed in a Stage incident (ARO-17939). Fix: before calling PostCluster, search CS by Azure metadata (subscription, resource group, resource name, tenant, managed resource group) via a new shared FindClusterByAzureInfo function. If an orphaned CS cluster from a prior failed attempt is found, reuse it instead of creating a duplicate. This makes the create path idempotent on retry. Also refactors the backend's ClusterClusterServiceCreate controller to use the same shared function, eliminating duplicated search logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cssjr The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Skipping CI for Draft Pull Request. |
|
/test all |
|
/assign copilot |
|
@cssjr: GitHub didn't allow me to assign the following users: copilot. Note that only Azure members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cc copilot |
|
/retest |
|
@cssjr: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
createHCPCluster()path before callingPostCluster, preventing split-brain when Cosmos write fails after CS create succeedsFindClusterByAzureInfointo a sharedocmpackage function, deduplicating logic between frontend and backend controllerRoot Cause
The frontend's
createHCPCluster()performs a two-phase write that is not atomic:PostClusterto Cluster Service (CS) — creates the cluster recordIf Phase 2 fails (transient Cosmos error), the system enters a split-brain state: CS has the cluster, Cosmos does not. When the Azure SDK automatically retries the failed 500:
CreateOrUpdateHCPClusterchecks Cosmos → not found → routes tocreateHCPClusterPostClusterto CS → CS rejects: "Duplicate ARO-HCP cluster name" (HTTP 400)InvalidRequestContent→ test/request failsThis was confirmed by an identical Stage incident documented in ARO-17939 (comment by Mike Gahagan, 2026-05-20).
Fix
Before calling
PostCluster, the frontend now searches CS for an existing cluster with matching Azure metadata (subscription, resource group, resource name, tenant, managed resource group) viaocm.FindClusterByAzureInfo. If an orphaned CS cluster from a prior failed attempt is found, it is reused instead of creating a duplicate.The backend's
ClusterClusterServiceCreatecontroller already had this idempotency pattern — this PR extracts it into a shared function and applies it to the frontend as well.Test plan
go buildfor all affected packages (internal/ocm, frontend, backend)go test ./internal/ocm/...— passesgo test ./frontend/...— passesgo test ./backend/...— all 20+ controller test packages passmake lint— passesmake test-integration)branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelsuccess rate after merge🤖 Generated with Claude Code