diff --git a/docs/advanced/airflow.md b/docs/advanced/airflow.md index 8912357..6045856 100644 --- a/docs/advanced/airflow.md +++ b/docs/advanced/airflow.md @@ -30,7 +30,7 @@ Generated 1 files (airflow scheduler): Tip: To regenerate after editing the contract: fluid generate schedule ``` -`fluid generate schedule` renders these DAGs when the contract's `orchestration.engine` is `airflow` and the contract declares no explicit `orchestration.tasks`. A contract with no `orchestration.engine` at all but a build with a cron trigger defaults to Airflow. A build with a trigger of another type (`event`, `timetable`, `schedule_and_dataset`) gets no DAG, because a plain cron DAG would silently drop what those types mean. `--env` is the overlay for the generation and the `--env` every scheduled run passes to `fluid apply`; leave it out for a contract with no overlays. +`fluid generate schedule` renders these DAGs when the contract's `orchestration.engine` is `airflow` and the contract declares no explicit `orchestration.tasks`. A contract with no `orchestration.engine` at all but a build with a cron trigger defaults to Airflow. A build with a trigger of another type (`event`, `timetable`, `schedule_and_dataset`) gets no DAG, because a plain cron DAG would silently drop what those types mean. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* Neither does a `continuous` build: its platform keeps it running (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). `--env` is the overlay for the generation and the `--env` every scheduled run passes to `fluid apply`; leave it out for a contract with no overlays. ## What the DAG does diff --git a/docs/cli/contract-validation.md b/docs/cli/contract-validation.md index 7521420..f6d42d4 100644 --- a/docs/cli/contract-validation.md +++ b/docs/cli/contract-validation.md @@ -40,7 +40,7 @@ fluid contract-validation contract.fluid.yaml --output-format json --output-file ## Notes - Connects to the live resource for the contract's provider (BigQuery, Snowflake, AWS, or local DuckDB) and verifies schema, bindings, quality specs, and metadata. -- Returns exit code `1` if any errors are found, or if `--strict` is set and any warnings are present. +- Returns exit code `1` if any errors are found, or if `--strict` is set and any warnings are present. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* With `--env`, an overlay patch that lands on another entry is such a warning (category `overlay`); see [`fluid validate`](./validate.md#env-and-environment-overlays). - Provider is auto-detected from `exposes[].binding.platform` (or `builds[].execution.runtime.platform`) — the `--provider`, `--project`, and `--region` flags only override the detected values. - Schema lookups are cached per `(table, provider)` for `--cache-ttl` seconds; use `--cache-clear` to force a refresh. - With `--check-drift`, results are compared against the recorded validation history for the same contract and resource. diff --git a/docs/cli/generate-artifacts.md b/docs/cli/generate-artifacts.md index 87d1943..596aae8 100644 --- a/docs/cli/generate-artifacts.md +++ b/docs/cli/generate-artifacts.md @@ -116,7 +116,7 @@ The `schedule` key writes DAGs when either holds: - `orchestration.engine` is set to anything except `none`. - No `orchestration.engine` is set, and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. The engine then defaults to Airflow. The file does nothing until `schedule-sync` delivers it, and stage 11 names the scheduler (`airflow`, `mwaa`, `composer` and `astronomer` run Airflow DAGs), so the default decides the file format only. -Each such build gets its own DAG that runs `fluid apply`, unless the contract hand-declares `orchestration.tasks`, which keep their declared operators. A build is left out when its own `execution.orchestration.engine` is `none` or not Airflow, or when its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a plain cron DAG cannot express. +Each such build gets its own DAG that runs `fluid apply`, unless the contract hand-declares `orchestration.tasks`, which keep their declared operators. A build is left out when its own `execution.orchestration.engine` is `none` or not Airflow, or when its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a plain cron DAG cannot express. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build, or one that runs outside forge-cli, gets no DAG and no task at all (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). ### Layout and naming diff --git a/docs/cli/test.md b/docs/cli/test.md index 0d90a7a..c520cdf 100644 --- a/docs/cli/test.md +++ b/docs/cli/test.md @@ -52,7 +52,7 @@ The local provider reads the data with DuckDB. If the `duckdb` package is missin | `--provider` | Override the provider (`gcp`, `snowflake`, `aws`, `local`). Like every provider name, it must be registered in your install. | | `--project`, `--region` | Override the project / account ID or region from the contract. | | `--server` | Provider connection string or identifier, such as a Snowflake account locator. | -| `--strict` | Exit `1` on any warning. In the example above `--strict` exits `1`. | +| `--strict` | Exit `1` on any warning. In the example above `--strict` exits `1`. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* With `--env`, an overlay patch that lands on another entry is such a warning, shown as an "Environment overlay" row; see [`fluid validate`](./validate.md#env-and-environment-overlays). `--engine soda` does not read `--strict`. | | `--no-data` | Skip live data validation: structure-only checks. Checks 4 to 7 print `not checked (--no-data)`. | | `--output` | `text` (default), `json`, or `junit`. | | `--output-file` | Write the report to a file instead of stdout. | diff --git a/docs/cli/validate.md b/docs/cli/validate.md index 9f7a9cc..e6f524d 100644 --- a/docs/cli/validate.md +++ b/docs/cli/validate.md @@ -157,6 +157,27 @@ Two environments are exempt from the refusal: `dev`, and an environment whose na The check lives in the contract loader, which `plan`, `apply`, `verify` and `publish` also use: `fluid apply --env prod` on a contract with no `prod` overlay prints the same warning. See [Per-environment overlays](../recipes/per-environment-overlays.md). +*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* An overlay patch that lands on another entry is a validation warning, so `--strict` fails on it. A contract that merges overlays by position (0.7.5 or older, or a 0.7.6 base whose entries lack unique ids) can do that: an overlay written against `builds: [ingest, transform]` patches the wrong build once the base gains one in front. + +```text +✅ Valid FLUID contract (schema v0.7.5) +⚠️ 1 warning(s) +Validation completed in 0.001s + +Validation Warnings: +==================== + 1. overlay_patch_retargeted: in network.service-instances, the overlay's builds[1] names id transform but patches the base's builds[1], id ingest, because lists merge by position. Put the overlay's entries in the base's order, or set fluidVersion 0.7.6 or later to merge builds by id. + +❌ Validation failed: warnings treated as errors (--strict) +``` + +- Without `--strict` it exits `0`; with it, `1`. `--format json` lists it under `warnings`. +- `--quiet` prints it as a `WARNING:` line, and with `--strict` as `ERROR (--strict): `. +- Workspace mode lists it under the product, and `--strict` fails the product. +- `fluid contract-validation --env` and `fluid test --env` (the native engine) report it too; `fluid test` shows it as an "Environment overlay" row. +- `overlay_entry_added`, an overlay entry the base does not have, stays a log line and does not fail `--strict`: a build only one environment runs is often intended. +- A bundle does not carry it: `fluid bundle --env` merged the overlay before, so `fluid validate .tgz --strict` (pipeline stage 2) passes. Validate the contract file with `--env` to catch it. + ## Contracts split into fragments A root contract can pull objects from other files with `$ref`. `fluid validate` resolves the pointers before it checks anything, so you pass the root file and validate the whole product. A `$ref` may only name a file inside the contract's own directory tree. One that climbs out of it is refused: diff --git a/docs/concepts/builds-exposes-bindings.md b/docs/concepts/builds-exposes-bindings.md index ac56997..37077dc 100644 --- a/docs/concepts/builds-exposes-bindings.md +++ b/docs/concepts/builds-exposes-bindings.md @@ -136,9 +136,43 @@ builds: - **Deployment settings go under `properties.parameters`.** `embeddedLogicPattern` is closed, so `parallelism` beside `sql` fails validation with `Additional properties are not allowed ('parallelism' was unexpected)`. Name them as the operator does: `parallelism` and `upgradeMode` (`savepoint`, `last-state`, `stateless`) from its job spec, and `flinkConfiguration`. - **`fluid apply --mode amend-and-build`** prints `Build 'reconcile' - engine 'flink' runs on kubernetes, outside forge-cli; there is nothing for forge-cli to run.` and counts the build on its own summary line, `Run outside forge-cli: 1`. A contract whose only builds run elsewhere exits 0; it is not "every build was skipped". The apply run report records the build as `skipped` with `reason: runs_outside_forge`. - **No planner runs its SQL.** The local, AWS, GCP and Snowflake planners never turn it into a DuckDB, Athena, BigQuery or Snowflake statement, and the local planner lands nothing for an expose the job writes. In a contract that mixes a Flink build with a local one, name each build's exposes in `outputs`: when it cannot tell which build writes an expose, the local planner lands neither and warns. -- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job. +- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job, or *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* say so with `trigger: {type: continuous}` (see [Continuous builds](#continuous-builds-trigger-type-continuous)). - **On 0.7.5**, `engine: flink` fails validation. A 0.7.5 contract declares the same job with `properties.language: flink_sql` (with `engine: sql`, say), and forge-cli treats it the same way. +### Continuous builds: `trigger.type: continuous` + +*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A build whose job its platform keeps running, such as a Flink job or a `kafka-connect` or `debezium` connector, can say so (schema 0.7.6 preview): + +```yaml +fluidVersion: "0.7.6" +builds: + - id: ingest + engine: kafka-connect + pattern: acquisition + execution: + trigger: + type: continuous +``` + +- **No scheduler starts it.** `fluid generate schedule`, `fluid schedule-sync` and the `fluid apply` DAGs leave the build out. A connector used to get a nightly `0 2 * * *` task. A contract whose every build is continuous or runs outside forge-cli has nothing to schedule: + + ```text + Every build runs outside forge-cli (a Flink job, for example) or continuously + (trigger.type: continuous), so there is nothing to schedule. No files written. + ``` + +- **It takes no other trigger.** `cron`, `schedule`, `datasets`, `event` and `timetable` are refused beside it, as Databricks Jobs refuses a continuous trigger beside any other, and the error names the field and the trigger type that takes it: + + ```text + 1. builds[0].execution.trigger: cron is not allowed with type continuous: a continuous job runs all the time, so its trigger takes no other trigger. Remove cron, or set type: schedule. + ``` + + It is not Airflow's `@continuous`, which starts batch runs back to back. +- **On 0.7.5**, `continuous` fails validation: `'continuous' is not one of ['schedule', 'event', 'manual', 'dependency', 'dataset', 'schedule_and_dataset', 'timetable']`. +- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. The dbt and python runners refuse it (`Unknown trigger type: continuous`), as they refuse `event` and `dataset`, and [`fluid scaffold-composer`](../cli/scaffold-composer.md) still writes its nightly DAG for any contract. +- **Overlays.** An environment overlay cannot turn a scheduled build continuous: an overlay cannot delete the base's `cron` or `schedule`, and setting it to `null` is refused too. Declare the trigger in each environment's overlay instead of in the base. +- **A DAG deployed earlier stays.** When every build of a product becomes continuous, it gets no `schedule/` artifacts, so pipeline stage 11 skips `fluid schedule-sync`, and a DAG deployed for the product before stays in the scheduler: remove it by hand. + ## `exposes[]` - the consumer-facing API ```yaml @@ -517,7 +551,7 @@ builds: timezone: UTC ``` -`trigger.type` is one of `schedule`, `event`, `manual`, `dependency`, `dataset`, `schedule_and_dataset`, `timetable`; `scheduled` is not a value, and the cron expression goes in `schedule` (`cron` is also accepted). `fluid apply --mode amend-and-build` runs the script with the contract's directory as its working directory. +`trigger.type` is one of `schedule`, `event`, `manual`, `dependency`, `dataset`, `schedule_and_dataset`, `timetable`, and in 0.7.6 `continuous` (see [Continuous builds](#continuous-builds-trigger-type-continuous)); `scheduled` is not a value, and the cron expression goes in `schedule` (`cron` is also accepted). `fluid apply --mode amend-and-build` runs the script with the contract's directory as its working directory. With a `schedule` trigger, `fluid generate schedule contract.fluid.yaml --out ` writes an Airflow DAG with `SCHEDULE = '0 * * * *'`. The DAG's task runs `fluid apply ... --mode amend-and-build --build-id --yes`. @@ -545,6 +579,7 @@ Schema `0.7.6` is a preview. It validates only when the file names it, and the s - **`exposes[].lifecycle.expire`** deletes stored data once it is older than `retention`; it is `false` unless you set it. - *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* **`builds[].engine: flink`** declares a job that runs outside forge-cli; see [Flink jobs](#flink-jobs-engine-flink). - *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* **Overlays merge `builds`, `exposes` and `consumes` by identity**; see [How an overlay merges](./environments-and-overlays.md#how-an-overlay-merges). +- *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* **`builds[].execution.trigger.type: continuous`** declares a job its platform keeps running, which no scheduler starts; see [Continuous builds](#continuous-builds-trigger-type-continuous). ## Where to look next diff --git a/docs/concepts/environments-and-overlays.md b/docs/concepts/environments-and-overlays.md index 61d7b5b..466f660 100644 --- a/docs/concepts/environments-and-overlays.md +++ b/docs/concepts/environments-and-overlays.md @@ -104,6 +104,9 @@ not name the file. - *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* Before 0.7.6 the merge is by position, and a patch that lands on an entry naming another id logs `overlay_patch_retargeted`: Put the overlay's entries in the base's order, or set `fluidVersion: "0.7.6"`. + *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* `fluid validate --env`, `fluid contract-validation --env` and + `fluid test --env` report it as a warning instead, and `--strict` fails on + it; see [`fluid validate`: `--env` and environment overlays](../cli/validate.md#env-and-environment-overlays). - Base keys an overlay does not set survive inside a patched map. A base `binding.location.path` is still present in the resolved binding of an overlay that sets `project`, `dataset` and `table`: `fluid bundle --env gcp` shows diff --git a/docs/reference/preview-fields.md b/docs/reference/preview-fields.md index 9917c7c..f30d977 100644 --- a/docs/reference/preview-fields.md +++ b/docs/reference/preview-fields.md @@ -89,6 +89,8 @@ The [`consumes`](./contract-0.7.6-preview.md#consumes) and [`exposes`](./contrac *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract that names 0.7.6, an overlay patches `builds`, `exposes` and `consumes` by identity instead of by position; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges). +*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* `builds[].execution.trigger.type: continuous` is a 0.7.6 value too, for a job its platform keeps running: no scheduler starts it, and it takes no other trigger; see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous). + ### Federated upstreams `upstreamWorkspace` and `upstreamDigest` travel together. This pair of fields on a `consumes[]` entry validates under 0.7.6: diff --git a/docs/walkthrough/airflow-declarative.md b/docs/walkthrough/airflow-declarative.md index 42be185..b55ad3c 100644 --- a/docs/walkthrough/airflow-declarative.md +++ b/docs/walkthrough/airflow-declarative.md @@ -139,7 +139,7 @@ The DAG is written for Airflow 3 (`airflow.sdk.DAG`, `BashOperator` from `apache Stage 3 (`fluid generate artifacts`) writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with `type: schedule` or no type. Airflow renders the second case. -Within that, a build gets the `fluid apply` DAG unless its own `execution.orchestration.engine` is `none` or not Airflow, or its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a cron DAG cannot express. A contract that hand-declares `orchestration.tasks` keeps those operators instead. +Within that, a build gets the `fluid apply` DAG unless its own `execution.orchestration.engine` is `none` or not Airflow, or its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a cron DAG cannot express. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build gets none either: its platform keeps it running (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). A contract that hand-declares `orchestration.tasks` keeps those operators instead. ### The DAG id and the environment