Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/cli/scaffold-composer.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,13 @@ with DAG(
- Writes one `<dag_id>.py` file per contract, where `dag_id` comes from `id` (or `name`) with `.` replaced by `_`.
- The generated DAG has three sequential `BashOperator` tasks: `validate`, `plan`, `apply`. The `plan` and `apply` steps run against the GCP provider, and `apply` passes `--yes`.
- The tasks call `python -m fluid_build.cli` with the contract path as you gave it to `scaffold-composer`, relative to the Composer worker's working directory, so the contract has to be at that path on the worker. `--env` selects the overlay used to read the contract when the DAG is generated; the commands inside the DAG carry no `--env`.
- Schedule is read from `builds[].execution.trigger.cron` in the contract; if missing, defaults to `0 2 * * *` (daily at 02:00). `trigger.schedule`, the key [`generate schedule`](./generate.md#fluid-generate-schedule) reads, is not read here: a contract with `schedule: "15 3 * * *"` and no `cron` still gets `0 2 * * *`.
- Schedule is read from `builds[].execution.trigger.cron` in the contract; if missing, defaults to `0 2 * * *` (daily at 02:00). *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* It is the cron of the first build a scheduler starts: a build that runs outside forge-cli (`engine: flink`, `properties.language: flink_sql`) or whose trigger is `continuous` is skipped, so a Flink job's own cron does not schedule the DAG. `trigger.schedule`, the key [`generate schedule`](./generate.md#fluid-generate-schedule) reads, is not read here: a contract with `schedule: "15 3 * * *"` and no `cron` still gets `0 2 * * *`.
- *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* A contract whose every build runs outside forge-cli or continuously gets no DAG unless it declares `orchestration.tasks`, the test [`fluid generate schedule`](./generate.md#fluid-generate-schedule) applies before it writes no files. The command exits `0`:

```text
Every build runs outside forge-cli (a Flink job, for example) or continuously
(trigger.type: continuous), so there is nothing to schedule. No DAG written.
```

A DAG the command wrote for the product before stays where it is, and the command prints its path when it finds it under the name it writes now: delete it, and remove it from Cloud Composer if you deployed it there. A file of that name that it did not write is not mentioned.
- For non-Composer Airflow installs, see [`fluid generate-airflow`](./generate-airflow.md). For Airflow, Dagster and Prefect output from one command, see [`fluid generate schedule`](./generate.md#fluid-generate-schedule).
31 changes: 31 additions & 0 deletions docs/cli/validate.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,37 @@ AWS has one warning of its own: `accessPolicy.grants` on a contract whose AWS bi

Each message names its remedy. How the fields work, and what `fluid apply` emits for them on each cloud, is in [Governance](../advanced/governance.md), the [GCP provider](../providers/gcp.md) and the [AWS provider](../providers/aws.md).

## A continuous trigger on a batch build

*(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* `trigger.type: continuous` (schema 0.7.6, see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)) says the job's platform keeps it running, so no scheduler starts the build. On a build forge-cli runs as a batch run, which ends, `fluid validate` warns:

```text
✅ Valid FLUID contract (schema v0.7.6)
⚠️ 1 warning(s)
Validation completed in 0.002s

Validation Warnings:
====================
1. builds[0] (id 'customer360_build'): trigger.type continuous on engine 'dbt': forge-cli runs dbt builds as batch runs, which end, and its dbt runner refuses a continuous trigger: forge-cli never runs this build. Use trigger.type schedule or manual, or run the job on a platform that keeps it running.
```

Without `--strict` it exits `0`; with it, `1`. `--format json` lists it under `warnings`.

| Build | What forge-cli does with a continuous one | Warning |
| --- | --- | --- |
| `engine: dbt`, `dbt-<adapter>` or `python`, or any build the python runner takes: no engine, `spark`, `glue` or `custom` without a `runtime.platform` of its own, an ingestion engine without `pattern: acquisition` | Its runner refuses the trigger, so `fluid apply --mode amend-and-build` fails on the build (exit `1`) and nothing else runs it. | Yes |
| Inline SQL (`engine: sql`, or `pattern: embedded-logic` with `properties.sql`) on the local DuckDB engine or Snowflake, or a `duckdb`, `airbyte`, `meltano` or `dlt` acquisition | Runs only when `fluid apply` runs builds; no scheduler starts it. | Yes |
| A `kafka-connect` or `debezium` acquisition, `engine: flink`, or `properties.language: flink_sql` | The connector or job keeps running on its platform. | No |
| `engine: spark`, `glue` or `custom` with a `runtime.platform` other than `local`, or inline SQL on a platform with no embedded-SQL executor | forge-cli does not run it, so `fluid apply --mode amend-and-build` leaves it to that platform, as it does a Flink job, and records it as skipped with `reason: runs_outside_forge`. | No |

For a build of the last kind, `fluid apply --mode amend-and-build` prints:

```text
⏭️ Build 'ingest_events' - trigger.type continuous on engine 'spark': databricks keeps the job running, outside forge-cli; there is nothing for forge-cli to run.
```

The check runs on the contract file and on a bundle's resolved contract, not in [workspace mode](#validating-without-a-contract-path).

## Validating a bundle (`.tgz`)

Pass a bundle built with `fluid bundle --format tgz` and `fluid validate` checks the bundle itself and the contract inside it:
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/builds-exposes-bindings.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ builds:

It is not Airflow's `@continuous`, which starts batch runs back to back.
- **On 0.7.5**, `continuous` fails validation: `'continuous' is not one of ['schedule', 'event', 'manual', 'dependency', 'dataset', 'schedule_and_dataset', 'timetable']`.
- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. The dbt and python runners refuse it (`Unknown trigger type: continuous`), as they refuse `event` and `dataset`, and [`fluid scaffold-composer`](../cli/scaffold-composer.md) still writes its nightly DAG for any contract.
- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* On a build forge-cli runs as a batch run, [`fluid validate` warns](../cli/validate.md#a-continuous-trigger-on-a-batch-build), so `--strict` fails. The dbt and python runners refuse the trigger and say why (exit `1`), and the python runner takes every build no other runner does, such as one with no engine or a local `spark` script; an inline-SQL build, or a `duckdb`, `airbyte`, `meltano` or `dlt` acquisition, runs only when `fluid apply` runs builds. A continuous `spark`, `glue` or `custom` build that names a `runtime.platform` other than `local` gets no warning: `fluid apply --mode amend-and-build` leaves it to that platform, as it does a Flink job. [`fluid scaffold-composer`](../cli/scaffold-composer.md) writes no DAG for a contract whose every build is continuous or runs outside forge-cli.
- **Overlays.** An environment overlay cannot turn a scheduled build continuous: an overlay cannot delete the base's `cron` or `schedule`, and setting it to `null` is refused too. Declare the trigger in each environment's overlay instead of in the base.
- **A DAG deployed earlier stays.** When every build of a product becomes continuous, it gets no `schedule/` artifacts, so pipeline stage 11 skips `fluid schedule-sync`, and a DAG deployed for the product before stays in the scheduler: remove it by hand.

Expand Down
Loading