fix(clouddriver-azure): retry VMSS resize on Azure 409 concurrent write conflicts

[sjungling]

Summary

• Add HTTP 409 (Conflict) to AzureBaseClient.canRetry() so the retry harness backs off and retries on Azure concurrent write conflicts
• Merge the get + apply in AzureComputeClient.resizeServerGroup() into one executeOp closure, matching the pattern already established by the disable methods — retries now re-fetch a fresh VMSS reference rather than reusing a stale one

Root cause

Investigated from pipeline execution 01KVE8QYGXK6TK349JW2E6SPVA (deploy-devaz for monitoringatlas2, 2026-06-18 14:03 PDT). The scaleDown stage failed terminally with:

Status code 409, ConflictingConcurrentWriteNotAllowed
"The operation was interrupted by a conflicting concurrent write on the same entity. Please retry later."

Azure holds an exclusive write lock on a VMSS for the duration of any update. The disableCluster step completed from Spinnaker's perspective but the underlying Azure VMSS update was still in flight when scaleDown immediately issued a resize — hitting the lock and getting a 409. Azure explicitly documents this as transient and instructs callers to retry.

Prior art — the single-closure pattern

Three existing methods in AzureNetworkClient already wrap the full read-modify-write sequence in one executeOp closure. resizeServerGroup now matches them:

• disableServerGroup (app gateway path)
• disableServerGroupWithLoadBalancer
• enableServerGroupWithAppGateway

All three wrap vmss.update().apply() inside the same closure as the getByResourceGroup call, so a failed write retries the entire read-modify-write sequence with a fresh VMSS reference.

Before

sequenceDiagram
    participant S as Spinnaker (Orca)
    participant CD as Clouddriver
    participant AZ as Azure VMSS API

    S->>CD: scaleDown(v054, capacity=0)
    CD->>AZ: getByResourceGroup(v054)
    Note over CD,AZ: inside executeOp() — retried on failure
    AZ-->>CD: 200 OK, vmss object
    CD->>AZ: vmss.update().withCapacity(0).apply()
    Note over CD,AZ: NOT inside executeOp() — no retry
    AZ-->>CD: 409 ConflictingConcurrentWriteNotAllowed
    Note over CD: ManagementException caught<br/>resourceNotFound? No → rethrow
    CD-->>S: AtomicOperationException
    S-->>S: Stage TERMINAL — pipeline fails

Loading

After

sequenceDiagram
    participant S as Spinnaker (Orca)
    participant CD as Clouddriver
    participant AZ as Azure VMSS API

    S->>CD: scaleDown(v054, capacity=0)

    rect rgb(220, 240, 255)
        Note over CD,AZ: executeOp() — attempt 1
        CD->>AZ: getByResourceGroup(v054)
        AZ-->>CD: 200 OK, vmss object
        CD->>AZ: vmss.update().withCapacity(0).apply()
        AZ-->>CD: 409 ConflictingConcurrentWriteNotAllowed
        Note over CD: canRetry(409)? Yes → sleep(backoff)
    end

    rect rgb(220, 240, 255)
        Note over CD,AZ: executeOp() — attempt 2 (fresh VMSS fetch)
        CD->>AZ: getByResourceGroup(v054)
        AZ-->>CD: 200 OK, vmss object
        CD->>AZ: vmss.update().withCapacity(0).apply()
        AZ-->>CD: 200 OK
    end

    CD-->>S: success
    S-->>S: Stage SUCCEEDED — pipeline continues

Loading

Test plan

• AzureBaseClientSpec — verifies executeOp retries on 409, 408, 5xx; does not retry on 404; throws after exhausting retries
• AzureComputeClientSpec — integration test verifies VMSS is re-fetched on each retry (proving get lives inside the closure); null guard verified; structural test verifies single executeOp call
• Deploy pipeline for an Azure tenant completes without manual intervention when disableCluster and scaleDown race