From f245bc66a051820c6cbcd55e50e4cced92d54bc1 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Wed, 21 Oct 2026 22:36:47 +0200 Subject: [PATCH] docs: a continuous trigger on a batch build warns; fluid apply leaves platform-run continuous builds; scaffold-composer schedules only scheduled builds (forge-cli #737) --- docs/cli/scaffold-composer.md | 10 +++++++- docs/cli/validate.md | 31 ++++++++++++++++++++++++ docs/concepts/builds-exposes-bindings.md | 2 +- 3 files changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/cli/scaffold-composer.md b/docs/cli/scaffold-composer.md index 83656f1..2973c2c 100644 --- a/docs/cli/scaffold-composer.md +++ b/docs/cli/scaffold-composer.md @@ -57,5 +57,13 @@ with DAG( - Writes one `.py` file per contract, where `dag_id` comes from `id` (or `name`) with `.` replaced by `_`. - The generated DAG has three sequential `BashOperator` tasks: `validate`, `plan`, `apply`. The `plan` and `apply` steps run against the GCP provider, and `apply` passes `--yes`. - The tasks call `python -m fluid_build.cli` with the contract path as you gave it to `scaffold-composer`, relative to the Composer worker's working directory, so the contract has to be at that path on the worker. `--env` selects the overlay used to read the contract when the DAG is generated; the commands inside the DAG carry no `--env`. -- Schedule is read from `builds[].execution.trigger.cron` in the contract; if missing, defaults to `0 2 * * *` (daily at 02:00). `trigger.schedule`, the key [`generate schedule`](./generate.md#fluid-generate-schedule) reads, is not read here: a contract with `schedule: "15 3 * * *"` and no `cron` still gets `0 2 * * *`. +- Schedule is read from `builds[].execution.trigger.cron` in the contract; if missing, defaults to `0 2 * * *` (daily at 02:00). *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* It is the cron of the first build a scheduler starts: a build that runs outside forge-cli (`engine: flink`, `properties.language: flink_sql`) or whose trigger is `continuous` is skipped, so a Flink job's own cron does not schedule the DAG. `trigger.schedule`, the key [`generate schedule`](./generate.md#fluid-generate-schedule) reads, is not read here: a contract with `schedule: "15 3 * * *"` and no `cron` still gets `0 2 * * *`. +- *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* A contract whose every build runs outside forge-cli or continuously gets no DAG unless it declares `orchestration.tasks`, the test [`fluid generate schedule`](./generate.md#fluid-generate-schedule) applies before it writes no files. The command exits `0`: + + ```text + Every build runs outside forge-cli (a Flink job, for example) or continuously + (trigger.type: continuous), so there is nothing to schedule. No DAG written. + ``` + + A DAG the command wrote for the product before stays where it is, and the command prints its path when it finds it under the name it writes now: delete it, and remove it from Cloud Composer if you deployed it there. A file of that name that it did not write is not mentioned. - For non-Composer Airflow installs, see [`fluid generate-airflow`](./generate-airflow.md). For Airflow, Dagster and Prefect output from one command, see [`fluid generate schedule`](./generate.md#fluid-generate-schedule). diff --git a/docs/cli/validate.md b/docs/cli/validate.md index e39ce1d..eed7c0a 100644 --- a/docs/cli/validate.md +++ b/docs/cli/validate.md @@ -260,6 +260,37 @@ AWS has one warning of its own: `accessPolicy.grants` on a contract whose AWS bi Each message names its remedy. How the fields work, and what `fluid apply` emits for them on each cloud, is in [Governance](../advanced/governance.md), the [GCP provider](../providers/gcp.md) and the [AWS provider](../providers/aws.md). +## A continuous trigger on a batch build + +*(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* `trigger.type: continuous` (schema 0.7.6, see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)) says the job's platform keeps it running, so no scheduler starts the build. On a build forge-cli runs as a batch run, which ends, `fluid validate` warns: + +```text +✅ Valid FLUID contract (schema v0.7.6) +⚠️ 1 warning(s) +Validation completed in 0.002s + +Validation Warnings: +==================== + 1. builds[0] (id 'customer360_build'): trigger.type continuous on engine 'dbt': forge-cli runs dbt builds as batch runs, which end, and its dbt runner refuses a continuous trigger: forge-cli never runs this build. Use trigger.type schedule or manual, or run the job on a platform that keeps it running. +``` + +Without `--strict` it exits `0`; with it, `1`. `--format json` lists it under `warnings`. + +| Build | What forge-cli does with a continuous one | Warning | +| --- | --- | --- | +| `engine: dbt`, `dbt-` or `python`, or any build the python runner takes: no engine, `spark`, `glue` or `custom` without a `runtime.platform` of its own, an ingestion engine without `pattern: acquisition` | Its runner refuses the trigger, so `fluid apply --mode amend-and-build` fails on the build (exit `1`) and nothing else runs it. | Yes | +| Inline SQL (`engine: sql`, or `pattern: embedded-logic` with `properties.sql`) on the local DuckDB engine or Snowflake, or a `duckdb`, `airbyte`, `meltano` or `dlt` acquisition | Runs only when `fluid apply` runs builds; no scheduler starts it. | Yes | +| A `kafka-connect` or `debezium` acquisition, `engine: flink`, or `properties.language: flink_sql` | The connector or job keeps running on its platform. | No | +| `engine: spark`, `glue` or `custom` with a `runtime.platform` other than `local`, or inline SQL on a platform with no embedded-SQL executor | forge-cli does not run it, so `fluid apply --mode amend-and-build` leaves it to that platform, as it does a Flink job, and records it as skipped with `reason: runs_outside_forge`. | No | + +For a build of the last kind, `fluid apply --mode amend-and-build` prints: + +```text +⏭️ Build 'ingest_events' - trigger.type continuous on engine 'spark': databricks keeps the job running, outside forge-cli; there is nothing for forge-cli to run. +``` + +The check runs on the contract file and on a bundle's resolved contract, not in [workspace mode](#validating-without-a-contract-path). + ## Validating a bundle (`.tgz`) Pass a bundle built with `fluid bundle --format tgz` and `fluid validate` checks the bundle itself and the contract inside it: diff --git a/docs/concepts/builds-exposes-bindings.md b/docs/concepts/builds-exposes-bindings.md index 37077dc..c83c1e1 100644 --- a/docs/concepts/builds-exposes-bindings.md +++ b/docs/concepts/builds-exposes-bindings.md @@ -169,7 +169,7 @@ builds: It is not Airflow's `@continuous`, which starts batch runs back to back. - **On 0.7.5**, `continuous` fails validation: `'continuous' is not one of ['schedule', 'event', 'manual', 'dependency', 'dataset', 'schedule_and_dataset', 'timetable']`. -- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. The dbt and python runners refuse it (`Unknown trigger type: continuous`), as they refuse `event` and `dataset`, and [`fluid scaffold-composer`](../cli/scaffold-composer.md) still writes its nightly DAG for any contract. +- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. *(unreleased, [forge-cli #737](https://github.com/Agenticstiger/forge-cli/pull/737))* On a build forge-cli runs as a batch run, [`fluid validate` warns](../cli/validate.md#a-continuous-trigger-on-a-batch-build), so `--strict` fails. The dbt and python runners refuse the trigger and say why (exit `1`), and the python runner takes every build no other runner does, such as one with no engine or a local `spark` script; an inline-SQL build, or a `duckdb`, `airbyte`, `meltano` or `dlt` acquisition, runs only when `fluid apply` runs builds. A continuous `spark`, `glue` or `custom` build that names a `runtime.platform` other than `local` gets no warning: `fluid apply --mode amend-and-build` leaves it to that platform, as it does a Flink job. [`fluid scaffold-composer`](../cli/scaffold-composer.md) writes no DAG for a contract whose every build is continuous or runs outside forge-cli. - **Overlays.** An environment overlay cannot turn a scheduled build continuous: an overlay cannot delete the base's `cron` or `schedule`, and setting it to `null` is refused too. Declare the trigger in each environment's overlay instead of in the base. - **A DAG deployed earlier stays.** When every build of a product becomes continuous, it gets no `schedule/` artifacts, so pipeline stage 11 skips `fluid schedule-sync`, and a DAG deployed for the product before stays in the scheduler: remove it by hand.