control-plane: verify delete permission for user storage mappings - #3197
Merged
Conversation
Storage-mapping health checks only exercised read/connect, so a bucket lacking delete permission still passed. Set check_delete on the FragmentStoreHealth requests from the storage-mapping mutations, probing under "recovery/". Bumps go.gazette.dev/core for the new request fields (gazette/core#489).
williamhbaker
force-pushed
the
wb/storage-mapping-delete-probe
branch
from
July 22, 2026 21:26
2c99d60 to
2f75e0e
Compare
mdibaiee
approved these changes
Jul 23, 2026
williamhbaker
pushed a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Aug 12, 2026
## What's Changed * control-plane-api: fix storage-mapping suffix strip to respect path-segment boundaries by @GregorShear in estuary/flow#3239 * materialize: remove deprecated `constraints` from materialize protocol by @mdibaiee in estuary/flow#3104 * control-plane: verify delete permission for user storage mappings by @williamhbaker in estuary/flow#3197 * Gate invite-link private-DP grants on an existing private data plane by @GregorShear in estuary/flow#3234 * local: fix dashboard origin to the external UI (http://localhost:3000) by @jgraettinger in estuary/flow#3230 * data-plane-controller: add GitHub App git auth to replace SSH machine user by @skord in estuary/flow#3243 * data-plane-controller: fix broken deploy (comments in env/secrets block scalars) by @skord in estuary/flow#3250 * notifications: remove stale alert_notifications snapshot files by @jwhartley in estuary/flow#3079 * Docs: Link fixes, remove example pages by @aeluce in estuary/flow#3231 * data-plane-controller: cut git auth over to the GitHub App by @skord in estuary/flow#3249 * Docs: reference page for OpenSearch materialization by @aeluce in estuary/flow#3113 * vm: fix Lima development VMs on Linux hosts by @skord in estuary/flow#3267 * publisher: discard unauthorized recovered ACK intents by @jgraettinger in estuary/flow#3268 * docs: source-sqs by @Alex-Bair in estuary/flow#3201 * runtime-next: remove hardcoded materialization ser_policy fallback by @dgreer-dev in estuary/flow#3140 * runtime-v2: bind task terms to ShardSpec bytes and restart gracefully from Join-wait by @jgraettinger in estuary/flow#3264 * Extend `PrefixFilter` with `in` and add storage mapping query as a consumer by @GregorShear in estuary/flow#3242 * docs: add initial materialize-hubspot docs by @danielnelson in estuary/flow#3042 * shuffle: fix open-phase deadlock via EOF-cascade teardown by @jgraettinger in estuary/flow#3261 * docs: add dataMovementStalled threshold example and format by @jwhartley in estuary/flow#3251 * control-plane-api: add a root effectiveAlertConfig(catalogPrefixOrName:) query by @GregorShear in estuary/flow#3227 * runtime-next: stop constructing per-iteration timers in actor loops by @jgraettinger in estuary/flow#3273 * tests: update to projection constraints in ledger connector by @danielnelson in estuary/flow#3274 * docs: document _meta field selection and depth behavior by @jwhartley in estuary/flow#3209 * control-plane: refresh notification emails by @aeluce in estuary/flow#3179 * docs: source-braintree-native resources have configurable concurrency by @Alex-Bair in estuary/flow#3270 * docs: materialize-iceberg table_identifier_case and field_name_case options by @jacobmarble in estuary/flow#3262 * runtime-v2: fix shuffle recovery deadlock under split read cohorts by @jgraettinger in estuary/flow#3278 * 2781: Preemptively deploying states for Discovery and Publish by @bbartman in estuary/flow#3279 * Docs: Add Snowflake user type by @aeluce in estuary/flow#3280 * docs: rediscovery interval config setting for SQL capture connectors by @Alex-Bair in estuary/flow#3285 * docs: source-mailchimp-native by @nicolaslazo in estuary/flow#3288 * config-encryption: fix broken image and deploy from CI by @mdibaiee in estuary/flow#3308 * config-encryption: fix Cloud Run startup probe port by @mdibaiee in estuary/flow#3315 * Docs: Webhook capture organization and fixes by @aeluce in estuary/flow#3313 * Sync Schedule moved to runtime v2 by @dgreer-dev in estuary/flow#3269 * dekaf: fix avro schema when property type is invalid by @danielnelson in estuary/flow#3301 * docs: source-brevo rewrite by @Alex-Bair in estuary/flow#3316 * docs: AWS IAM auth cross-account guidance, correct Identity Provider ARN wording by @jwhartley in estuary/flow#3298 * Local-stack QA quality of life: arm64 connector containers, child-process cleanup, readiness by @jgraettinger in estuary/flow#3276 * control-plane-api: remove dead evolutions module by @jshearer in estuary/flow#3302 * Introduce an indirect form of built specifications by @jgraettinger in estuary/flow#3303 * docs: clarify BYOK KMS is decrypt-only and identity may not be in dashboard by @jwhartley in estuary/flow#3065 * shuffle: gate the causal-hint stall timeout on a requested checkpoint by @williamhbaker in estuary/flow#3323 * docs: source-shopify-native add markets stream by @Alex-Bair in estuary/flow#3331 * docs: source-commercetools by @Alex-Bair in estuary/flow#3332 **Full Changelog**: estuary/flow@v0.6.12...v0.6.13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Historically storage-mapping health checks have only exercised read/connect, so a bucket lacking the delete permission on its
recovery/path was allowed in, but later would cause problems with the shard pruner service that needs to delete objects from this location.gazette/core#489 adds an optional check for deleting from
recovery/, which is intended to be used for this case. It is optional so that it does not run on during the broker's normal on-going health checks, since a failure to delete would not necessarily impact an otherwise functioning data flow, and because there are pre-existing working configurations like this that we don't necessarily want to stop. The deletion check is only used for user-initiated changes to storage mappings, as a gate to ensure that the permissions exist for the shard pruner to work correctly.Separately but related, gazette/core#488 downgrades the shard pruner failure to delete due to permissions from a fatal error to a log, as this is an expected condition for pre-existing configurations.
This PR wires up the delete validation in the control plane API's
check_store_health.The gazette pin points to the merged gazette/core#489 (commit e54beb5c6e6452a94fe0c0bbe21a1695fa9fddd6). It is fine to merge this before new brokers are deployed with the new code, since pre-existing brokers will just ignore the extra parameters.
Closes #3073
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)