Repository navigation
fix(iac): apply refuses an MWAA environment it cannot create instead of reporting success - #715
Merged
Merged
Conversation
…of reporting success
An AWS contract with orchestration.engine airflow (or mwaa) and an
orchestration.schedule made the AWS planner plan mwaa.ensure_environment,
and the OpenTofu emitter's _emit_from_actions had no branch for it. The
generated main.tf.json held no aws_mwaa_environment, fluid apply reported
success, and no Airflow environment existed. OpenTofu is the only AWS
apply engine, so nothing else created it.
fluid apply and fluid generate iac now stop with unsupported_binding, kind
mwaa-environment-not-emitted, naming the environment and two ways out:
engine none keeps the schedule on EventBridge Scheduler, or push the DAG
to an MWAA environment you own with fluid schedule-sync. The OpenTofu
engine cannot create one itself: aws_mwaa_environment needs a VPC
network_configuration with private subnets, which the contract does not
carry (aws-ia/terraform-aws-mwaa requires private_subnet_ids with no
default; the CloudFormation NetworkConfiguration reference says the same).
Every other planned op without an emitter is accounted for too. An op no
list knows is refused by name (kind planned-op-not-emitted), the way dbt
raises MissingMaterializationError ("No materialization ... was found for
adapter", dbt-adapters exceptions/compilation.py) rather than skipping
the model. Data-plane ops emit() derives from exposes[] and build steps
pass quietly; known planner ops nothing provisions yet (IAM bindings from
metadata.policies, Athena views, Redshift schemas and tables,
execution.trigger schedules) and orchestration.tasks steps are named in
a warning. --shadow now maps an environment op to a logical resource, so
a dropped one would show as a native-only gap.
… MWAA contract is refused even when the planner fails The previous commit refused any planned op the AWS emitter did not know. native_actions plans with the first expose's provider, so on a contract whose first expose is on GCP or Snowflake, `--provider aws` (what the ambiguous-provider error tells the user to run) fed bq.* / sf.* ops to the AWS emitter and failed the AWS half with planned-op-not-emitted. An unknown op is now named in a warning (op and id rendered with repr, so a control character from a task's op never reaches the terminal) instead of refused. What kept the refusal honest is now a test: every "op" literal under fluid_build/providers/aws, and a mode="replace" plan, must be emitted (_ACTION_EMITTERS, which replaces the if/elif chain) or classified. It found aws.s3.copy_prefix, the pre-replace backup the rollback writer records, which is now owned elsewhere. dbt-core asserts adapter materialization coverage the same way. native_actions returns [] when the planner raises (a Glue table name its validator refuses, a cyclic orchestration.tasks), and the emitter then saw no mwaa.ensure_environment and wrote the data plane, exit 0. An empty plan for a contract that asks for MWAA (the schedule planner's own test: a schedule and engine airflow or mwaa, lower-cased, unstripped) is refused the same way. The remediation named `schedule-sync --environment-name <env>`, which the MWAA dispatch never reads: it syncs to --destination. It now gives the full generate-schedule / schedule-sync pair with --destination filled from orchestration.mwaa.dagS3Path, and tells a contract that names its environment to push to it rather than create one.
…loud's planner ran A multi-cloud contract whose first expose is on GCP or Snowflake, with orchestration.engine airflow and a schedule, still wrote an AWS module with no MWAA environment and exited 0 under `fluid generate iac --provider aws` (and `fluid apply --provider aws`). native_actions plans with the first expose's provider, so the AWS planner that plans mwaa.ensure_environment never ran; the emitter only refused when the action list was empty or held that op, and the foreign ops made it non-empty. FLUID_PROVIDER=local hit the same gap with LocalProvider's ops. The refusal now follows the contract: when it asks for MWAA (the same predicate the AWS schedule planner applies) and no planned action is the environment op, the emitter refuses with unsupported_binding, kind mwaa-environment-not-emitted, whichever planner produced the actions. Regression tests drive generate_iac with --provider aws on both multi-cloud shapes and the emitter with GCP, Snowflake and local ops.
The refusal printed `fluid generate schedule --output dags` and then `fluid schedule-sync --dags-dir dags`. schedule-sync's default (product delete scope) mirrors each top-level directory of --dags-dir and refuses loose files, so the printed pair failed with schedule_sync_dags_dir_not_product_scoped. The DAG is now generated into dags/<contract id>/, and a contract id that is not a valid product directory name prints a <product-id> placeholder instead. A test runs the printed output path through schedule-sync's own scope check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
plan.jsonpromised an MWAA environment thatapplynever created.providers/aws/plan/schedule.pyemits anmwaa.ensure_environmentaction for an AWS contract withorchestration.engine: airflow|mwaaplus a schedule or triggers.iac/providers/aws.py::_emit_from_actionshad no branch for it, somain.tf.jsonsilently lackedaws_mwaa_environment,fluid apply"succeeded", and--shadowdid not flag the drop.UnsupportedBindingError, with kindmwaa-environment-not-emittedformwaa.ensure_environmentandplanned-op-not-emittedfor any other unknown op. It is never silently dropped, following the design note iniac/base.py.orchestration.engine: none+ schedule), or deploy DAGs to an existing MWAA environment.--shadowreports it too.--provider awsrefuses an MWAA contract even when another cloud's planner ran, and a multi-cloud contract applies again.iam.bind_glue_database) is now logged instead of silent.CHANGELOG.md.Creating a real
aws_mwaa_environmentneeds VPCnetwork_configuration(private subnets, security groups) that the contract does not carry, so it is out of scope.Type of Change
orchestration.engine: airflow|mwaaplus a schedule or triggers now fail atgenerate iac/applyinstead of succeeding without the environment.Documentation
Checklist
tests/iac/test_iac_aws_unemitted_ops.py;test_iac_aws.pyandtest_iac_shadow.pyupdated)ruff check,black(24.10.0), import-linter and the mypy strict hotspots are cleanTesting
Real
fluidCLI infc-test:3.12with a faketofushim that records the module it is handed,origin/mainvs this branch:engine: airflow+ schedule (generate iac)main.tf.jsonwith 3 resources and no environment, exit 0unsupported_binding [ERR_UNSUPPORTED_BINDING] kind: mwaa-environment-not-emitted+ remediation; nomain.tf.jsonfluid applyengine: mwaa, uppercaseAIRFLOW, named shared environment--shadowengine: none+ schedule / triggers onlyaws_scheduler_scheduleemitted fornone+ schedule)