broker: add opt-in delete-permission probe to FragmentStoreHealth - #489
Merged
Conversation
williamhbaker
force-pushed
the
wb/health-check-delete-probe
branch
from
July 15, 2026 22:06
5dd174d to
4d020e5
Compare
The storage-mapping connection test never exercises a delete, so a bucket policy missing the delete action passes setup, and the problem only surfaces later during operations that require deletion, such as recovery log pruning. Add check_delete and check_delete_prefix to FragmentStoreHealthRequest. When check_delete is set, the broker writes and removes a throwaway probe object under the prefix (empty probes the store root) to verify delete permission at configuration time. The probe is on-demand and excluded from the periodic background check, and is intended to be invoked as a user-initiated check when storage mappings change.
williamhbaker
force-pushed
the
wb/health-check-delete-probe
branch
from
July 15, 2026 22:13
4d020e5 to
ec74045
Compare
dgreer-dev
reviewed
Jul 16, 2026
dgreer-dev
left a comment
dgreer-dev
left a comment
There was a problem hiding this comment.
Since check_delete makes the broker do a Put/Remove against the store, should this require more than Capability_READ?
Contributor
Author
|
I was thinking about that but we are already doing a PUT (write) as part of the normal health checks so it didn't seem like much more of a deviation than that. |
williamhbaker
added a commit
to estuary/flow
that referenced
this pull request
Jul 17, 2026
Storage-mapping health checks only exercised read/connect, so a bucket lacking delete permission still passed. Set check_delete on the FragmentStoreHealth requests from the storage-mapping mutations, probing under "recovery/". Bumps go.gazette.dev/core for the new request fields (gazette/core#489).
williamhbaker
added a commit
to estuary/flow
that referenced
this pull request
Jul 22, 2026
Storage-mapping health checks only exercised read/connect, so a bucket lacking delete permission still passed. Set check_delete on the FragmentStoreHealth requests from the storage-mapping mutations, probing under "recovery/". Bumps go.gazette.dev/core for the new request fields (gazette/core#489).
williamhbaker
added a commit
to estuary/flow
that referenced
this pull request
Jul 23, 2026
Storage-mapping health checks only exercised read/connect, so a bucket lacking delete permission still passed. Set check_delete on the FragmentStoreHealth requests from the storage-mapping mutations, probing under "recovery/". Bumps go.gazette.dev/core for the new request fields (gazette/core#489).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The storage-mapping connection test never exercises a delete, so a bucket policy missing the delete action passes setup and only fails later, when recovery-log pruning hits AccessDenied and fires recurring alerts.
Add check_delete and check_delete_prefix to FragmentStoreHealthRequest. When check_delete is set, the broker writes and removes a throwaway probe object under the prefix (empty probes the store root) to verify delete permission at configuration time. The probe is on-demand and excluded from the periodic background check, so existing stores lacking delete permission keep reporting healthy. It uses the generic Store interface, so S3, GCS, and Azure are all covered.
Note: Manually tested locally against an S3 emulator.