Extend PrefixFilter with in and add storage mapping query as a consumer - #3242
Merged
Conversation
filter param to the storageMappings queryfilter param to storageMappings and an in predicate to PrefixFilter
This was referenced Jul 23, 2026
GregorShear
force-pushed
the
greg/gql-storage-mappings-filter
branch
2 times, most recently
from
July 24, 2026 03:16
0b2ac58 to
bede83a
Compare
4 tasks
GregorShear
force-pushed
the
greg/gql-storage-mappings-filter
branch
from
July 24, 2026 18:24
bede83a to
572cb9c
Compare
…icate to `PrefixFilter` Composable prefix filtering shared across storageMappings, alertConfigs, and inviteLinks; the deprecated `by` maps onto the same PrefixFilter.
GregorShear
force-pushed
the
greg/gql-storage-mappings-filter
branch
from
July 24, 2026 20:38
572cb9c to
95c7343
Compare
filter param to storageMappings and an in predicate to PrefixFilterPrefixFilter with in and add consumer
jshearer
previously approved these changes
Jul 27, 2026
PrefixFilter with in and add consumerPrefixFilter with in and add storage mapping query as a consumer
…XES narrowing Bound the caller-controlled `in` set with a max_items=100 validator (min_items=1 already floored it), so narrow_to_exact_set and the per-request `= ANY(...)` binding can't be driven unbounded. The validator is runtime-only, but the accompanying doc-comment update regenerates the field's description in the SDL. Add coverage for the narrow_to_exact_set / MAX_PREFIXES interaction that was previously untested: a caller with more than MAX_PREFIXES readable prefixes is refused an unfiltered listing, but succeeds once an `in` filter narrows the authorized set back under the cap. Also assert an over-cap `in` is rejected, and correct the alert-config test comment that overclaimed this coverage.
GregorShear
force-pushed
the
greg/gql-storage-mappings-filter
branch
from
July 27, 2026 18:27
8b63592 to
7b9e18b
Compare
jshearer
approved these changes
Jul 27, 2026
williamhbaker
pushed a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Aug 12, 2026
## What's Changed * control-plane-api: fix storage-mapping suffix strip to respect path-segment boundaries by @GregorShear in estuary/flow#3239 * materialize: remove deprecated `constraints` from materialize protocol by @mdibaiee in estuary/flow#3104 * control-plane: verify delete permission for user storage mappings by @williamhbaker in estuary/flow#3197 * Gate invite-link private-DP grants on an existing private data plane by @GregorShear in estuary/flow#3234 * local: fix dashboard origin to the external UI (http://localhost:3000) by @jgraettinger in estuary/flow#3230 * data-plane-controller: add GitHub App git auth to replace SSH machine user by @skord in estuary/flow#3243 * data-plane-controller: fix broken deploy (comments in env/secrets block scalars) by @skord in estuary/flow#3250 * notifications: remove stale alert_notifications snapshot files by @jwhartley in estuary/flow#3079 * Docs: Link fixes, remove example pages by @aeluce in estuary/flow#3231 * data-plane-controller: cut git auth over to the GitHub App by @skord in estuary/flow#3249 * Docs: reference page for OpenSearch materialization by @aeluce in estuary/flow#3113 * vm: fix Lima development VMs on Linux hosts by @skord in estuary/flow#3267 * publisher: discard unauthorized recovered ACK intents by @jgraettinger in estuary/flow#3268 * docs: source-sqs by @Alex-Bair in estuary/flow#3201 * runtime-next: remove hardcoded materialization ser_policy fallback by @dgreer-dev in estuary/flow#3140 * runtime-v2: bind task terms to ShardSpec bytes and restart gracefully from Join-wait by @jgraettinger in estuary/flow#3264 * Extend `PrefixFilter` with `in` and add storage mapping query as a consumer by @GregorShear in estuary/flow#3242 * docs: add initial materialize-hubspot docs by @danielnelson in estuary/flow#3042 * shuffle: fix open-phase deadlock via EOF-cascade teardown by @jgraettinger in estuary/flow#3261 * docs: add dataMovementStalled threshold example and format by @jwhartley in estuary/flow#3251 * control-plane-api: add a root effectiveAlertConfig(catalogPrefixOrName:) query by @GregorShear in estuary/flow#3227 * runtime-next: stop constructing per-iteration timers in actor loops by @jgraettinger in estuary/flow#3273 * tests: update to projection constraints in ledger connector by @danielnelson in estuary/flow#3274 * docs: document _meta field selection and depth behavior by @jwhartley in estuary/flow#3209 * control-plane: refresh notification emails by @aeluce in estuary/flow#3179 * docs: source-braintree-native resources have configurable concurrency by @Alex-Bair in estuary/flow#3270 * docs: materialize-iceberg table_identifier_case and field_name_case options by @jacobmarble in estuary/flow#3262 * runtime-v2: fix shuffle recovery deadlock under split read cohorts by @jgraettinger in estuary/flow#3278 * 2781: Preemptively deploying states for Discovery and Publish by @bbartman in estuary/flow#3279 * Docs: Add Snowflake user type by @aeluce in estuary/flow#3280 * docs: rediscovery interval config setting for SQL capture connectors by @Alex-Bair in estuary/flow#3285 * docs: source-mailchimp-native by @nicolaslazo in estuary/flow#3288 * config-encryption: fix broken image and deploy from CI by @mdibaiee in estuary/flow#3308 * config-encryption: fix Cloud Run startup probe port by @mdibaiee in estuary/flow#3315 * Docs: Webhook capture organization and fixes by @aeluce in estuary/flow#3313 * Sync Schedule moved to runtime v2 by @dgreer-dev in estuary/flow#3269 * dekaf: fix avro schema when property type is invalid by @danielnelson in estuary/flow#3301 * docs: source-brevo rewrite by @Alex-Bair in estuary/flow#3316 * docs: AWS IAM auth cross-account guidance, correct Identity Provider ARN wording by @jwhartley in estuary/flow#3298 * Local-stack QA quality of life: arm64 connector containers, child-process cleanup, readiness by @jgraettinger in estuary/flow#3276 * control-plane-api: remove dead evolutions module by @jshearer in estuary/flow#3302 * Introduce an indirect form of built specifications by @jgraettinger in estuary/flow#3303 * docs: clarify BYOK KMS is decrypt-only and identity may not be in dashboard by @jwhartley in estuary/flow#3065 * shuffle: gate the causal-hint stall timeout on a requested checkpoint by @williamhbaker in estuary/flow#3323 * docs: source-shopify-native add markets stream by @Alex-Bair in estuary/flow#3331 * docs: source-commercetools by @Alex-Bair in estuary/flow#3332 **Full Changelog**: estuary/flow@v0.6.12...v0.6.13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the shared
PrefixFilterGraphQL input with an exact-setinpredicate — an alternative to the existingstartsWithsubtree match (the two are mutually exclusive), letting prefix-scoped queries filter by an explicit set of prefixes.storageMappingsgains a composablefilterargument built on it, matching the pattern already used byinviteLinks(and the in-flightdataPlanesfilter).Groundwork for moving the UI's storage-mappings table (estuary/ui#2031) and flowctl onto the shared filter pattern.
PrefixFilterTwo scoping modes, mutually exclusive across every consumer:
For example, scope a query to an explicit set of prefixes (intersected with what the caller can read):
startsWith) or an exact set (in), never a mix; supplying both is a request error.in: []is rejected at input validation, rather than ambiguously meaning "nothing" or "everything".storageMappings,alertConfigs,inviteLinks— resolve their scope through one combinator,filtered_authorized_prefixes, which chains the mutual-exclusion check (PrefixFilter::into_parts), the authorized-prefix lookup, and the exact-set narrowing (PrefixFilter::narrow_to_exact_set). The narrow-only invariant — a filter can only remove authorized prefixes, never add them — has a single owner.Deprecating
storageMappings(by:)storageMappingsgainsfilter: { catalogPrefix }, deprecatingbyin its favor (underPrefix → startsWith,exactPrefixes → in).bymaps onto the samePrefixFilterinternally and stays until flowctl migrates (#3238).Stack / follow-ups
by: { exactPrefixes }withfilter: { catalogPrefix: { in } }, stacked on this PR.filter: { catalogPrefix: { startsWith } }after a codegen refresh.