updateStorageMapping can silently break every task on a data plane it doesn't validate

[jwhartley]

Problem

update_storage_mapping runs its storage health checks against spec.data_planes (the proposed list) rather than against the data planes that actually host live specs under the prefix. Removing a data plane from that list therefore exempts it from validation, while its tasks keep running on it and keep needing storage access.

That interacts badly with republish. A change to stores broadcasts a re-publication to the whole prefix, which restarts every shard. A running shard tolerates a storage auth failure as a retryable warning (failed to refresh remote fragments (will retry)), but a restarting shard has to play back its recovery log from that same store first. So a single storage-mapping edit can convert a latent, invisible storage failure into every task on the unvalidated plane going down at once, with nothing surfaced at edit time.

Observed

A tenant with ~76 live specs on a public data plane that was not present in data_planes[] changed their primary store to an Azure container. That plane's Azure application had no grant on the container, so every request from it fails:

Code: InvalidAuthenticationInfo
RESPONSE Status: 400 Authentication information is not given in the correct format. Check the value of Authorization header.

The edit passed validation (both listed planes health-checked fine against the new store), republished the prefix, and restarted all shards. They have been stuck PENDING ever since, failing recovery-log playback. A subsequent data plane migration for the same prefix also wedges, because it runs the same connection test before migrating.

Code

crates/control-plane-api/src/server/public/graphql/storage_mappings.rs

Health checks are scoped to the proposed list only:

let data_planes = resolve_data_planes(&snapshot, &spec.data_planes)?;
let health_checks = run_all_health_checks(&catalog_prefix, &data_planes, &spec.stores).await;

Update only rejects failures on newly added stores/planes, and the comment acknowledges the gap:

// We only fail on health check errors for newly added stores or data planes.
// Tasks under this storage mapping will still be broken if there are any failing
// health checks, but we allow the update so long as the user isn't adding more
// problems than there already were.

create_storage_mapping does guard the analogous case:

// Check if any existing tasks or collections would be affected by this new storage mapping.
// We disallow creating storage mappings that would change the storage for existing specs.

On the UI side, removing a data plane is an unguarded per-row X with no live-spec check and no warning (estuary/ui, src/components/admin/Settings/StorageMappings/Dialog/DataPlanesCard/DataPlanesCard.tsx).

Open question

Which behaviour is intended. create and update disagree today about whether "would this affect existing specs" is blocking, and it isn't clear from the code whether that asymmetry is deliberate. Flagging it for whoever owns this path to decide.