2781: Preemptively deploying states for Discovery and Publish - #3279
Merged
Conversation
…tibility Define DiscoverState and PublicationState and wire them as the executors' automations::Executor::State, ahead of the stale-snapshot deferral changes which will actually read and write them. The automations framework decodes persisted task state with serde_json::from_str, and the prior `()` state only accepts JSON `null` — so this intermediate deploy is needed to keep in-flight tasks decodable across a deploy or rollback in either direction. Nothing reads the state fields yet, hence #[allow(dead_code)].
williamhbaker
pushed a commit
to estuary/homebrew-flowctl
that referenced
this pull request
Aug 12, 2026
## What's Changed * control-plane-api: fix storage-mapping suffix strip to respect path-segment boundaries by @GregorShear in estuary/flow#3239 * materialize: remove deprecated `constraints` from materialize protocol by @mdibaiee in estuary/flow#3104 * control-plane: verify delete permission for user storage mappings by @williamhbaker in estuary/flow#3197 * Gate invite-link private-DP grants on an existing private data plane by @GregorShear in estuary/flow#3234 * local: fix dashboard origin to the external UI (http://localhost:3000) by @jgraettinger in estuary/flow#3230 * data-plane-controller: add GitHub App git auth to replace SSH machine user by @skord in estuary/flow#3243 * data-plane-controller: fix broken deploy (comments in env/secrets block scalars) by @skord in estuary/flow#3250 * notifications: remove stale alert_notifications snapshot files by @jwhartley in estuary/flow#3079 * Docs: Link fixes, remove example pages by @aeluce in estuary/flow#3231 * data-plane-controller: cut git auth over to the GitHub App by @skord in estuary/flow#3249 * Docs: reference page for OpenSearch materialization by @aeluce in estuary/flow#3113 * vm: fix Lima development VMs on Linux hosts by @skord in estuary/flow#3267 * publisher: discard unauthorized recovered ACK intents by @jgraettinger in estuary/flow#3268 * docs: source-sqs by @Alex-Bair in estuary/flow#3201 * runtime-next: remove hardcoded materialization ser_policy fallback by @dgreer-dev in estuary/flow#3140 * runtime-v2: bind task terms to ShardSpec bytes and restart gracefully from Join-wait by @jgraettinger in estuary/flow#3264 * Extend `PrefixFilter` with `in` and add storage mapping query as a consumer by @GregorShear in estuary/flow#3242 * docs: add initial materialize-hubspot docs by @danielnelson in estuary/flow#3042 * shuffle: fix open-phase deadlock via EOF-cascade teardown by @jgraettinger in estuary/flow#3261 * docs: add dataMovementStalled threshold example and format by @jwhartley in estuary/flow#3251 * control-plane-api: add a root effectiveAlertConfig(catalogPrefixOrName:) query by @GregorShear in estuary/flow#3227 * runtime-next: stop constructing per-iteration timers in actor loops by @jgraettinger in estuary/flow#3273 * tests: update to projection constraints in ledger connector by @danielnelson in estuary/flow#3274 * docs: document _meta field selection and depth behavior by @jwhartley in estuary/flow#3209 * control-plane: refresh notification emails by @aeluce in estuary/flow#3179 * docs: source-braintree-native resources have configurable concurrency by @Alex-Bair in estuary/flow#3270 * docs: materialize-iceberg table_identifier_case and field_name_case options by @jacobmarble in estuary/flow#3262 * runtime-v2: fix shuffle recovery deadlock under split read cohorts by @jgraettinger in estuary/flow#3278 * 2781: Preemptively deploying states for Discovery and Publish by @bbartman in estuary/flow#3279 * Docs: Add Snowflake user type by @aeluce in estuary/flow#3280 * docs: rediscovery interval config setting for SQL capture connectors by @Alex-Bair in estuary/flow#3285 * docs: source-mailchimp-native by @nicolaslazo in estuary/flow#3288 * config-encryption: fix broken image and deploy from CI by @mdibaiee in estuary/flow#3308 * config-encryption: fix Cloud Run startup probe port by @mdibaiee in estuary/flow#3315 * Docs: Webhook capture organization and fixes by @aeluce in estuary/flow#3313 * Sync Schedule moved to runtime v2 by @dgreer-dev in estuary/flow#3269 * dekaf: fix avro schema when property type is invalid by @danielnelson in estuary/flow#3301 * docs: source-brevo rewrite by @Alex-Bair in estuary/flow#3316 * docs: AWS IAM auth cross-account guidance, correct Identity Provider ARN wording by @jwhartley in estuary/flow#3298 * Local-stack QA quality of life: arm64 connector containers, child-process cleanup, readiness by @jgraettinger in estuary/flow#3276 * control-plane-api: remove dead evolutions module by @jshearer in estuary/flow#3302 * Introduce an indirect form of built specifications by @jgraettinger in estuary/flow#3303 * docs: clarify BYOK KMS is decrypt-only and identity may not be in dashboard by @jwhartley in estuary/flow#3065 * shuffle: gate the causal-hint stall timeout on a requested checkpoint by @williamhbaker in estuary/flow#3323 * docs: source-shopify-native add markets stream by @Alex-Bair in estuary/flow#3331 * docs: source-commercetools by @Alex-Bair in estuary/flow#3332 **Full Changelog**: estuary/flow@v0.6.12...v0.6.13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
control-plane: carry discover/publication poll state for deploy compatibility
Define DiscoverState and PublicationState and wire them as the executors' automations::Executor::State, ahead of the stale-snapshot deferral changes which will actually read and write them. The automations framework decodes persisted task state with serde_json::from_str, and the prior
()state only accepts JSONnull— so this intermediate deploy is needed to keep in-flight tasks decodable across a deploy or rollback in either direction.Nothing reads the state fields yet, hence #[allow(dead_code)].
We are doing this first in order to prevent a deserialization issue when we deploy the next set of changes from here
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)