Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/advanced/airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,36 @@ fluid schedule-sync --scheduler airflow --dags-dir dist/artifacts/schedule --des

`--dags-dir` must contain one directory per product. With the default `--delete-scope product`, a loose file in `--dags-dir` is refused with `schedule_sync_dags_dir_not_product_scoped` and exit code 2, so one product's sync cannot delete another's DAGs. If you generated with `fluid generate schedule`, write into `-o <dags-dir>/<product-id>/` as in the quick start. `schedule-sync` takes no contract argument: it needs `--scheduler` and `--dags-dir`, plus `--destination` for `airflow` and `mwaa`.

## MWAA on AWS

*([forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715), unreleased)* `fluid apply` does not create an Amazon MWAA environment. Create the environment yourself, in your own VPC, or use one you already run, and push the DAGs to the S3 folder it reads. Schedule the builds with `builds[].execution.trigger`, and leave `orchestration.schedule` out:

```yaml
orchestration:
engine: airflow
builds:
- id: load_orders
execution:
trigger:
type: schedule
cron: "0 6 * * *"
```

```bash
fluid generate schedule contract.fluid.yaml -o dags/sales_orders_daily
fluid schedule-sync --scheduler mwaa --dags-dir dags --destination s3://acme-airflow-dags/dags/ --dry-run
```

```text
[schedule-sync] scheduler=mwaa dags-dir=.../dags env=dev delete-scope=product dry-run=True
[schedule-sync] → /usr/local/bin/aws s3 sync .../dags/sales_orders_daily/ s3://acme-airflow-dags/dags/sales_orders_daily/ --delete
[schedule-sync] ✔ mwaa sync complete (1 subprocess(es))
```

Drop `--dry-run` to push. `--destination` is the S3 folder the environment reads its DAGs from, in its source bucket. Each scheduled run executes `fluid apply` on the MWAA worker, which needs what [What the DAG does](#what-the-dag-does) lists.

An `aws` contract that keeps `orchestration.schedule` with `orchestration.engine: airflow` asks `fluid apply` for the environment itself, and is refused with `unsupported_binding`, kind `mwaa-environment-not-emitted`, before anything is applied; see [Scheduling on AWS](../providers/aws.md#scheduling-on-aws). Before this change the apply reported success and no environment existed. The refusal's own next step names `fluid generate schedule --scheduler airflow --output dags/<contract id>` and `fluid schedule-sync --scheduler mwaa --dags-dir dags`, the product-scoped layout above, so the two commands run as printed.

## `fluid generate-airflow` (deprecated)

`fluid generate-airflow` still runs, and prints a warning that points at its replacement:
Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/error-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,8 @@ As of 0.18.1 the second suggestion names `azure`, but `fluid providers` lists `a

The state refusals (`state_shared_with_another_provider`, `state_migration_ambiguous` and the other `state_*` events) are described on [OpenTofu state](../concepts/state.md). The refusal for an env with no overlay (`overlay_declared_but_missing`) and the `bundle_env_mismatch` refusal are on [Environments and overlays](../concepts/environments-and-overlays.md); `plan_env_mismatch` is in [Production troubleshooting](./production-troubleshooting.md).

*(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714) and [forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715))* `apply_contract_invalid`, `apply_plan_missing_contract` and `unsupported_binding` (kind `mwaa-environment-not-emitted`) are not catalogued, so they print no curated suggestions. [Production troubleshooting](./production-troubleshooting.md#environment-mode-and-plan-file-errors) explains the first two, and [Scheduling on AWS](../providers/aws.md#scheduling-on-aws) the third. `contract_version_unsupported`, catalogued [above](#contract-version-unsupported), is now raised by `fluid apply` on both engines and in every mode.

### planner_failed

`ERR_PLANNER_FAILED`. The documentation link lands on [Production troubleshooting](./production-troubleshooting.md).
Expand Down
5 changes: 5 additions & 0 deletions docs/advanced/production-troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ When the bundle is unrecoverable, `fluid apply --no-verify-plan-binding` skips t
| `apply_build_id_requires_build_mode` | `--build-id` selects which build runs, and the mode (for example `dry-run`) runs none | Use `--mode amend-and-build` or `replace-and-build`, or drop `--build-id` |
| `apply_plan_unreadable` | The plan file is missing, is not JSON, or is truncated | Regenerate it with `fluid plan --out` |
| `apply_plan_mode_mismatch` | The plan was made for a different `--mode` than apply was given | Re-run `fluid plan --mode <mode>` with the mode you will apply |
| `apply_contract_invalid` | *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* The contract, or the contract a `plan.json` embeds, is not valid against its schema; the `errors:` line lists why. `fluid apply` checks it on both engines and in every mode, so a cloud or build apply that used to exit 0 on such a contract now stops here | Run `fluid validate <contract> --env <env>`, fix what it reports, and apply again. No flag skips the check. See [Contract validation](../cli/apply.md#contract-validation) |
| `contract_version_unsupported` | The contract's `fluidVersion` is below 0.7. *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* `fluid apply` raises it on both engines and in every mode | Migrate the contract to a 0.7.x schema |
| `apply_plan_missing_contract` | *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* The `plan.json` embeds no contract, so there is nothing to validate. Refused with or without `--no-verify-plan-binding`; such a plan used to run its actions unchecked on the native engine | Re-run `fluid plan <contract> --out <plan>.json` and apply the new plan |
| `contract_tests_baseline_missing`, `contract_tests_bad_baseline` | `fluid contract-tests --baseline` names a file that does not exist, or whose content is not a baseline | Create one with `fluid contract-tests <contract> --write-baseline baseline.schema.json` |

### The federation check
Expand Down Expand Up @@ -203,6 +206,8 @@ The organization comes from `organization_id`, else `FLUID_CC_ORG_ID`; with neit
| `dbt_tests_refusing_overwrite` | `fluid generate dbt-tests --out` names an existing file that lacks fluid's managed-by header, so it is not overwritten | Delete the file, or pass a different `--out` |
| `refusing_to_overwrite_existing_file_pass_force_to_override` (exit 2) | `fluid scaffold-ide` would overwrite a file that exists | Pass `--force`; see [`fluid scaffold-ide`](../cli/scaffold-ide.md) |
| `principal-placeholder` (the kind of an `unsupported_binding` error) | On a `gcp` binding, a principal is a placeholder (a reserved top-level domain) or is not an IAM member and `binding.principals` does not map it | Map it in `binding.principals` to a real group or service account, or write the IAM member in `accessPolicy`; see [`fluid generate iac`](../cli/generate-iac.md) |
| `mwaa-environment-not-emitted` (the kind of an `unsupported_binding` error) | *(unreleased, [forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715))* An `aws` contract with `orchestration.engine: airflow` and an `orchestration.schedule` asks for an MWAA environment, which the OpenTofu module cannot create: it needs a VPC network the contract does not carry. `fluid apply` and `fluid generate iac` stop before anything is applied; the apply used to report success with no environment | To schedule without Airflow, set `orchestration.engine: none` and keep the schedule. To keep Airflow, run the DAGs on an MWAA environment you own and drop `orchestration.schedule`. See [Scheduling on AWS](../providers/aws.md#scheduling-on-aws) |
| `aws_planned_op_not_emitted`, `aws_orchestration_task_not_emitted`, `planned_op_not_emitted` (WARNING) | *(unreleased, [forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715))* The AWS planner planned an operation the OpenTofu module has no resource for, and the apply goes on without it. The line names the operation and its id | Read which operation it is in [Planned operations the module does not create](../providers/aws.md#planned-operations-the-module-does-not-create), and provision it another way if you need it |
| `schedule_sync_dags_dir_not_product_scoped` (exit 2) | `fluid schedule-sync` defaults to `--delete-scope product`, which mirrors each top-level directory of `--dags-dir` and never deletes another product's DAGs. The directory has loose files | Put the files in `<dags-dir>/<product-id>/`. `fluid generate artifacts` does; for `fluid generate schedule` use `-o <dags-dir>/<product-id>/`. Or pass `--delete-scope none` to copy without deleting |
| `lakeformation-grant-columns` | A Lake Formation grant gives a principal read access, but the contract's column restrictions let it read no column | Remove the grant, or allow the principal at least one column in `policy.authz.columnRestrictions` |
| `generate_iac_aws_account_required` | `fluid generate iac` for AWS needs the account id and does not look it up | Set `AWS_ACCOUNT_ID`. `fluid apply` resolves the account itself |
Expand Down
2 changes: 1 addition & 1 deletion docs/advanced/typed-cli-errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Each class has a `for_*` factory that builds the `what`, `why`, `fix` and `doc`

`SchemaValidationError` is also the name of a different class in the forge agent layer; see [Typed Errors](./typed-errors.md). The two are not the same type.

The route table also sends four catalogued events here: `invalid_schema_version`, `contract_version_unsupported`, `invalid_min_version` and `invalid_max_version`. Set `fluidVersion` to a version the installed CLI bundles. The latest stable contract schema is 0.7.5; 0.7.6 is a preview that a contract opts in to with `fluidVersion: "0.7.6"`. `fluid validate` reports the version it validated against.
The route table also sends four catalogued events here: `invalid_schema_version`, `contract_version_unsupported`, `invalid_min_version` and `invalid_max_version`. Set `fluidVersion` to a version the installed CLI bundles. The latest stable contract schema is 0.7.5; 0.7.6 is a preview that a contract opts in to with `fluidVersion: "0.7.6"`. `fluid validate` reports the version it validated against. *([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), unreleased)* `fluid apply` raises `contract_version_unsupported` before it plans, on both engines and in every mode; see [Contract validation](../cli/apply.md#contract-validation).

### Capability negotiation

Expand Down
51 changes: 51 additions & 0 deletions docs/cli/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ Two things differ between the engines, and the rest of this page says which one
- **Modes.** `--mode dry-run` stops after `tofu plan` on the OpenTofu engine. Beyond that the OpenTofu engine does not branch on `--mode`: `amend`, `create-only` and `replace` run the same `tofu apply` of the emitted module. What stays mode-specific is the [data-loss gate](#safety-gates) and the build phase of the `*-and-build` modes.
- **Options.** Some options are read only by the native engine. They are marked in the [options tables](#options).

*([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), unreleased)* The contract check does not differ: both engines validate the contract before they plan, in every mode. See [Contract validation](#contract-validation).

## Apply mode

| `--mode` | What it does |
Expand Down Expand Up @@ -152,6 +154,49 @@ amend-and-build (or replace-and-build) to run that build, or drop --build-id.

Before 0.16.3 the flag was accepted and dropped, so a DDL-only apply exited 0.

## Contract validation

*([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), unreleased)* `fluid apply` checks the contract against its schema before it plans anything, on both engines and in every mode, `--dry-run` and the build modes included. A GCP contract that still carries the deprecated `metadata.policies` mapping stops before `tofu` runs:

```bash
fluid apply contract.fluid.yaml --dry-run --yes
```

```text
❌ apply_contract_invalid [ERR_APPLY_CONTRACT_INVALID]
command: apply
error_count: 1
errors: ["metadata: Additional properties are not allowed ('policies' was unexpected)"]
message: Contract validation failed with 1 error(s); ``fluid apply`` will not proceed on a schema-invalid contract. Run ``fluid validate`` for the full report.
```

It is the check `fluid validate` and `fluid plan` run: a `fluidVersion` below 0.7 is refused, then the contract is validated against the schema its `fluidVersion` names.

| The input | Event | Exit |
|---|---|---|
| A contract whose `fluidVersion` is below 0.7 | `contract_version_unsupported` | 1 |
| A contract that is not valid against its schema | `apply_contract_invalid` | 1 |
| A `plan.json` that embeds no contract | `apply_plan_missing_contract` | 1 |

Before this change only the native engine ran the check, and not in the build modes. On the OpenTofu engine (`aws`, `gcp`, `snowflake`, `confluent` and IaC plugin clouds), and under `--mode amend-and-build` or `replace-and-build` on either engine, a contract `fluid validate` rejects reached `tofu` or the build runner and could apply with exit 0.

What is checked:

- **A contract file or a bundle.** The contract with its `--env` overlay, as `fluid validate <contract> --env <env>` sees it: `{{ env.* }}` placeholders are still literal.
- **A `plan.json`.** The contract embedded in the plan, the copy its `planDigest` covers. Every plan `fluid plan` writes embeds it (see [What `plan.json` records](./plan.md#what-plan-json-records)).

A plan whose `contract` is missing, empty or not an object is refused, with or without `--no-verify-plan-binding`. There is nothing to check, and the `planDigest` is an unkeyed hash: it proves the file has not changed since it was written, not who wrote it. Before this change such a plan fell back to the native engine and ran its actions unchecked.

```text
❌ apply_plan_missing_contract [ERR_APPLY_PLAN_MISSING_CONTRACT]
path: .../runtime/plan.json
hint: plan.json embeds no contract, so there is nothing to validate before applying it. Re-run ``fluid plan <contract> --out <plan>.json`` with a current forge-cli.
```

The check runs after the [plan binding](#plan-binding) checks and the [mode gate](#the-mode-gate), and before the [apply hooks](#extension-point-apply-hooks), `tofu`, the native provider and any build, so a hook never sees a refused contract. `fluid apply` reads its input once, and the engine choice, the hooks, `tofu`, the build runner and the native provider all take that copy: the contract that is applied is the contract that was checked, even if the file changes while the apply runs.

There is no flag to skip the check. Run `fluid validate <contract> --env <env>` to see every error, fix them, and apply again.

## Plan binding

When you pass a saved plan (`runtime/plan.json`) instead of a contract, `apply` runs three gates on it before it builds or provisions anything, on both engines:
Expand All @@ -174,6 +219,8 @@ This is the Terraform-style "apply consumes exact plan" guarantee, enforced cryp

Pass the bundle with `--bundle <path-to.tgz>`: the tgz this `plan.json` was generated against. When omitted, `apply` auto-discovers a single sibling `.tgz` / `.tar.gz` next to the `plan.json`. If several are present it logs a warning and leaves the bundle unresolved, so pass `--bundle` to disambiguate. A `plan.json` made from a raw contract has an empty `bundleDigest`, so only `planDigest` is verified; see [`fluid plan`](./plan.md#plan-binding).

*([forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714), unreleased)* After these gates the contract the plan embeds is validated against its schema, and a plan that embeds none is refused with `apply_plan_missing_contract`. See [Contract validation](#contract-validation).

### The plan's mode

`fluid plan --mode X` records `X` in `plan.json`. `fluid apply plan.json --mode Y` with a different `Y` is refused with `apply_plan_mode_mismatch` and exit 1, before any build runs. This is why every non-default example above plans with the same `--mode`. See [`fluid plan --mode`](./plan.md#key-options).
Expand Down Expand Up @@ -516,6 +563,9 @@ Each failure prints the typed event name and an `[ERR_<EVENT>]` slug, and exits
| --- | --- |
| `apply_plan_digest_plan_tamper`, `apply_plan_digest_bundle_mismatch`, `apply_plan_digest_bundle_missing` | A saved plan fails its digest checks. See [Plan binding](#plan-binding). |
| `apply_plan_mode_mismatch` | `--mode` differs from the mode the plan was made for. |
| `apply_contract_invalid` | *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* The contract, or the contract a `plan.json` embeds, is not valid against its schema. Raised on both engines and in every mode. See [Contract validation](#contract-validation). |
| `contract_version_unsupported` | *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* The contract's `fluidVersion` is below 0.7. Raised on both engines and in every mode. |
| `apply_plan_missing_contract` | *(unreleased, [forge-cli #714](https://github.com/Agenticstiger/forge-cli/pull/714))* A `plan.json` embeds no contract. Re-run `fluid plan` and apply the new plan. |
| `plan_env_mismatch` | A build mode is given an `--env` that differs from the plan's. |
| `bundle_env_mismatch` | `--env` differs from the env a bundle was built for. |
| `apply_build_id_requires_build_mode` | `--build-id` with a mode that runs no build. |
Expand All @@ -527,6 +577,7 @@ Each failure prints the typed event name and an `[ERR_<EVENT>]` slug, and exits
| `opentofu_dataset_access_unreconciled` | State holds a BigQuery dataset an older forge-cli applied with an authoritative access list, and every entry of it is a grant the contract no longer makes. Revoke those entries on the dataset by hand, or keep one of them in `accessPolicy` for one apply. |
| `apply_state_backend_invalid`, `state_shared_with_another_provider`, `state_migration_*` | See [Remote state](#remote-state). |
| `generate_iac_provider_mismatch` | `--provider` contradicts the contract's binding. |
| `unsupported_binding` | The OpenTofu module cannot provision part of the contract as declared, and refuses it rather than drop it; the `kind:` line says which part. *(unreleased, [forge-cli #715](https://github.com/Agenticstiger/forge-cli/pull/715))* Kind `mwaa-environment-not-emitted`: an `aws` contract with `orchestration.engine: airflow` and an `orchestration.schedule` asks for an MWAA environment, which the module cannot create. See [Scheduling on AWS](../providers/aws.md#scheduling-on-aws). |
| `opentofu_engine_no_tofu` | The OpenTofu engine needs `tofu` and it is not on `PATH`. Install it, or pass `--ensure-opentofu`. |

## Notes
Expand Down
Loading
Loading