Skip to content

Extend PrefixFilter with in and add storage mapping query as a consumer - #3242

Merged
GregorShear merged 2 commits into
masterfrom
greg/gql-storage-mappings-filter
Jul 27, 2026
Merged

Extend PrefixFilter with in and add storage mapping query as a consumer#3242
GregorShear merged 2 commits into
masterfrom
greg/gql-storage-mappings-filter

Conversation

@GregorShear

@GregorShear GregorShear commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Extends the shared PrefixFilter GraphQL input with an exact-set in predicate — an alternative to the existing startsWith subtree match (the two are mutually exclusive), letting prefix-scoped queries filter by an explicit set of prefixes. storageMappings gains a composable filter argument built on it, matching the pattern already used by inviteLinks (and the in-flight dataPlanes filter).

Groundwork for moving the UI's storage-mappings table (estuary/ui#2031) and flowctl onto the shared filter pattern.

PrefixFilter

Two scoping modes, mutually exclusive across every consumer:

input PrefixFilter {
  startsWith: String     # subtree match: acmeCo/ matches acmeCo/, acmeCo/team/, ...
  in: [String!]          # exact-set match: only the listed prefixes
}

For example, scope a query to an explicit set of prefixes (intersected with what the caller can read):

storageMappings(filter: { catalogPrefix: { in: ["acmeCo/", "betaCo/"] } })
  • A prefix scope is always either a subtree (startsWith) or an exact set (in), never a mix; supplying both is a request error.
  • An empty in: [] is rejected at input validation, rather than ambiguously meaning "nothing" or "everything".
  • All three consumers — storageMappings, alertConfigs, inviteLinks — resolve their scope through one combinator, filtered_authorized_prefixes, which chains the mutual-exclusion check (PrefixFilter::into_parts), the authorized-prefix lookup, and the exact-set narrowing (PrefixFilter::narrow_to_exact_set). The narrow-only invariant — a filter can only remove authorized prefixes, never add them — has a single owner.

Deprecating storageMappings(by:)

storageMappings gains filter: { catalogPrefix }, deprecating by in its favor (underPrefix → startsWith, exactPrefixes → in). by maps onto the same PrefixFilter internally and stays until flowctl migrates (#3238).

Stack / follow-ups

@GregorShear GregorShear changed the title control-plane-api: add a filter param to the storageMappings query control-plane-api: add a filter param to storageMappings and an in predicate to PrefixFilter Jul 23, 2026
@GregorShear
GregorShear force-pushed the greg/gql-storage-mappings-filter branch 2 times, most recently from 0b2ac58 to bede83a Compare July 24, 2026 03:16
@GregorShear
GregorShear requested a review from jshearer July 24, 2026 03:32
@GregorShear
GregorShear force-pushed the greg/gql-storage-mappings-filter branch from bede83a to 572cb9c Compare July 24, 2026 18:24
…icate to `PrefixFilter`

Composable prefix filtering shared across storageMappings, alertConfigs, and inviteLinks; the deprecated `by` maps onto the same PrefixFilter.
@GregorShear
GregorShear force-pushed the greg/gql-storage-mappings-filter branch from 572cb9c to 95c7343 Compare July 24, 2026 20:38
@GregorShear GregorShear changed the title control-plane-api: add a filter param to storageMappings and an in predicate to PrefixFilter Extend PrefixFilter with in and add consumer Jul 27, 2026
jshearer
jshearer previously approved these changes Jul 27, 2026

@jshearer jshearer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % couple nits

Comment thread crates/control-plane-api/src/server/public/graphql/filters.rs Outdated
Comment thread crates/control-plane-api/src/server/public/graphql/alert_configs.rs Outdated
@GregorShear GregorShear changed the title Extend PrefixFilter with in and add consumer Extend PrefixFilter with in and add storage mapping query as a consumer Jul 27, 2026
…XES narrowing

Bound the caller-controlled `in` set with a max_items=100 validator (min_items=1 already floored it), so narrow_to_exact_set and the per-request `= ANY(...)` binding can't be driven unbounded. The validator is runtime-only, but the accompanying doc-comment update regenerates the field's description in the SDL.

Add coverage for the narrow_to_exact_set / MAX_PREFIXES interaction that was previously untested: a caller with more than MAX_PREFIXES readable prefixes is refused an unfiltered listing, but succeeds once an `in` filter narrows the authorized set back under the cap. Also assert an over-cap `in` is rejected, and correct the alert-config test comment that overclaimed this coverage.
@GregorShear
GregorShear force-pushed the greg/gql-storage-mappings-filter branch from 8b63592 to 7b9e18b Compare July 27, 2026 18:27
@GregorShear
GregorShear merged commit f097350 into master Jul 27, 2026
11 checks passed
@GregorShear
GregorShear deleted the greg/gql-storage-mappings-filter branch July 27, 2026 19:57
williamhbaker pushed a commit to estuary/homebrew-flowctl that referenced this pull request Aug 12, 2026
## What's Changed
* control-plane-api: fix storage-mapping suffix strip to respect path-segment boundaries by @GregorShear in estuary/flow#3239
* materialize: remove deprecated `constraints` from materialize protocol by @mdibaiee in estuary/flow#3104
* control-plane: verify delete permission for user storage mappings by @williamhbaker in estuary/flow#3197
* Gate invite-link private-DP grants on an existing private data plane by @GregorShear in estuary/flow#3234
* local: fix dashboard origin to the external UI (http://localhost:3000) by @jgraettinger in estuary/flow#3230
* data-plane-controller: add GitHub App git auth to replace SSH machine user by @skord in estuary/flow#3243
* data-plane-controller: fix broken deploy (comments in env/secrets block scalars) by @skord in estuary/flow#3250
* notifications: remove stale alert_notifications snapshot files by @jwhartley in estuary/flow#3079
* Docs: Link fixes, remove example pages by @aeluce in estuary/flow#3231
* data-plane-controller: cut git auth over to the GitHub App by @skord in estuary/flow#3249
* Docs: reference page for OpenSearch materialization by @aeluce in estuary/flow#3113
* vm: fix Lima development VMs on Linux hosts by @skord in estuary/flow#3267
* publisher: discard unauthorized recovered ACK intents by @jgraettinger in estuary/flow#3268
* docs: source-sqs by @Alex-Bair in estuary/flow#3201
* runtime-next: remove hardcoded materialization ser_policy fallback by @dgreer-dev in estuary/flow#3140
* runtime-v2: bind task terms to ShardSpec bytes and restart gracefully from Join-wait by @jgraettinger in estuary/flow#3264
* Extend `PrefixFilter` with `in` and add storage mapping query as a consumer by @GregorShear in estuary/flow#3242
* docs: add initial materialize-hubspot docs by @danielnelson in estuary/flow#3042
* shuffle: fix open-phase deadlock via EOF-cascade teardown by @jgraettinger in estuary/flow#3261
* docs: add dataMovementStalled threshold example and format by @jwhartley in estuary/flow#3251
* control-plane-api: add a root effectiveAlertConfig(catalogPrefixOrName:) query by @GregorShear in estuary/flow#3227
* runtime-next: stop constructing per-iteration timers in actor loops by @jgraettinger in estuary/flow#3273
* tests: update to projection constraints in ledger connector by @danielnelson in estuary/flow#3274
* docs: document _meta field selection and depth behavior by @jwhartley in estuary/flow#3209
* control-plane: refresh notification emails by @aeluce in estuary/flow#3179
* docs: source-braintree-native resources have configurable concurrency by @Alex-Bair in estuary/flow#3270
* docs: materialize-iceberg table_identifier_case and field_name_case options by @jacobmarble in estuary/flow#3262
* runtime-v2: fix shuffle recovery deadlock under split read cohorts by @jgraettinger in estuary/flow#3278
* 2781: Preemptively deploying states for Discovery and Publish by @bbartman in estuary/flow#3279
* Docs: Add Snowflake user type by @aeluce in estuary/flow#3280
* docs: rediscovery interval config setting for SQL capture connectors by @Alex-Bair in estuary/flow#3285
* docs: source-mailchimp-native by @nicolaslazo in estuary/flow#3288
* config-encryption: fix broken image and deploy from CI by @mdibaiee in estuary/flow#3308
* config-encryption: fix Cloud Run startup probe port by @mdibaiee in estuary/flow#3315
* Docs: Webhook capture organization and fixes by @aeluce in estuary/flow#3313
* Sync Schedule moved to runtime v2 by @dgreer-dev in estuary/flow#3269
* dekaf: fix avro schema when property type is invalid by @danielnelson in estuary/flow#3301
* docs: source-brevo rewrite by @Alex-Bair in estuary/flow#3316
* docs: AWS IAM auth cross-account guidance, correct Identity Provider ARN wording by @jwhartley in estuary/flow#3298
* Local-stack QA quality of life: arm64 connector containers, child-process cleanup, readiness by @jgraettinger in estuary/flow#3276
* control-plane-api: remove dead evolutions module by @jshearer in estuary/flow#3302
* Introduce an indirect form of built specifications by @jgraettinger in estuary/flow#3303
* docs: clarify BYOK KMS is decrypt-only and identity may not be in dashboard by @jwhartley in estuary/flow#3065
* shuffle: gate the causal-hint stall timeout on a requested checkpoint by @williamhbaker in estuary/flow#3323
* docs: source-shopify-native add markets stream by @Alex-Bair in estuary/flow#3331
* docs: source-commercetools by @Alex-Bair in estuary/flow#3332

**Full Changelog**: estuary/flow@v0.6.12...v0.6.13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants