Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/advanced/airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Generated 1 files (airflow scheduler):
Tip: To regenerate after editing the contract: fluid generate schedule
```

`fluid generate schedule` renders these DAGs when the contract's `orchestration.engine` is `airflow` and the contract declares no explicit `orchestration.tasks`. A contract with no `orchestration.engine` at all but a build with a cron trigger defaults to Airflow. A build with a trigger of another type (`event`, `timetable`, `schedule_and_dataset`) gets no DAG, because a plain cron DAG would silently drop what those types mean. `--env` is the overlay for the generation and the `--env` every scheduled run passes to `fluid apply`; leave it out for a contract with no overlays.
`fluid generate schedule` renders these DAGs when the contract's `orchestration.engine` is `airflow` and the contract declares no explicit `orchestration.tasks`. A contract with no `orchestration.engine` at all but a build with a cron trigger defaults to Airflow. A build with a trigger of another type (`event`, `timetable`, `schedule_and_dataset`) gets no DAG, because a plain cron DAG would silently drop what those types mean. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* Neither does a `continuous` build: its platform keeps it running (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). `--env` is the overlay for the generation and the `--env` every scheduled run passes to `fluid apply`; leave it out for a contract with no overlays.

## What the DAG does

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/contract-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ fluid contract-validation contract.fluid.yaml --output-format json --output-file
## Notes

- Connects to the live resource for the contract's provider (BigQuery, Snowflake, AWS, or local DuckDB) and verifies schema, bindings, quality specs, and metadata.
- Returns exit code `1` if any errors are found, or if `--strict` is set and any warnings are present.
- Returns exit code `1` if any errors are found, or if `--strict` is set and any warnings are present. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* With `--env`, an overlay patch that lands on another entry is such a warning (category `overlay`); see [`fluid validate`](./validate.md#env-and-environment-overlays).
- Provider is auto-detected from `exposes[].binding.platform` (or `builds[].execution.runtime.platform`) — the `--provider`, `--project`, and `--region` flags only override the detected values.
- Schema lookups are cached per `(table, provider)` for `--cache-ttl` seconds; use `--cache-clear` to force a refresh.
- With `--check-drift`, results are compared against the recorded validation history for the same contract and resource.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/generate-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ The `schedule` key writes DAGs when either holds:
- `orchestration.engine` is set to anything except `none`.
- No `orchestration.engine` is set, and a build declares `execution.trigger.schedule` (or `cron`) with trigger `type: schedule` or no type. The engine then defaults to Airflow. The file does nothing until `schedule-sync` delivers it, and stage 11 names the scheduler (`airflow`, `mwaa`, `composer` and `astronomer` run Airflow DAGs), so the default decides the file format only.

Each such build gets its own DAG that runs `fluid apply`, unless the contract hand-declares `orchestration.tasks`, which keep their declared operators. A build is left out when its own `execution.orchestration.engine` is `none` or not Airflow, or when its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a plain cron DAG cannot express.
Each such build gets its own DAG that runs `fluid apply`, unless the contract hand-declares `orchestration.tasks`, which keep their declared operators. A build is left out when its own `execution.orchestration.engine` is `none` or not Airflow, or when its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a plain cron DAG cannot express. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build, or one that runs outside forge-cli, gets no DAG and no task at all (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)).

### Layout and naming

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ The local provider reads the data with DuckDB. If the `duckdb` package is missin
| `--provider` | Override the provider (`gcp`, `snowflake`, `aws`, `local`). Like every provider name, it must be registered in your install. |
| `--project`, `--region` | Override the project / account ID or region from the contract. |
| `--server` | Provider connection string or identifier, such as a Snowflake account locator. |
| `--strict` | Exit `1` on any warning. In the example above `--strict` exits `1`. |
| `--strict` | Exit `1` on any warning. In the example above `--strict` exits `1`. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* With `--env`, an overlay patch that lands on another entry is such a warning, shown as an "Environment overlay" row; see [`fluid validate`](./validate.md#env-and-environment-overlays). `--engine soda` does not read `--strict`. |
| `--no-data` | Skip live data validation: structure-only checks. Checks 4 to 7 print `not checked (--no-data)`. |
| `--output` | `text` (default), `json`, or `junit`. |
| `--output-file` | Write the report to a file instead of stdout. |
Expand Down
21 changes: 21 additions & 0 deletions docs/cli/validate.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,27 @@ Two environments are exempt from the refusal: `dev`, and an environment whose na

The check lives in the contract loader, which `plan`, `apply`, `verify` and `publish` also use: `fluid apply --env prod` on a contract with no `prod` overlay prints the same warning. See [Per-environment overlays](../recipes/per-environment-overlays.md).

*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* An overlay patch that lands on another entry is a validation warning, so `--strict` fails on it. A contract that merges overlays by position (0.7.5 or older, or a 0.7.6 base whose entries lack unique ids) can do that: an overlay written against `builds: [ingest, transform]` patches the wrong build once the base gains one in front.

```text
✅ Valid FLUID contract (schema v0.7.5)
⚠️ 1 warning(s)
Validation completed in 0.001s

Validation Warnings:
====================
1. overlay_patch_retargeted: in network.service-instances, the overlay's builds[1] names id transform but patches the base's builds[1], id ingest, because lists merge by position. Put the overlay's entries in the base's order, or set fluidVersion 0.7.6 or later to merge builds by id.

❌ Validation failed: warnings treated as errors (--strict)
```

- Without `--strict` it exits `0`; with it, `1`. `--format json` lists it under `warnings`.
- `--quiet` prints it as a `WARNING:` line, and with `--strict` as `ERROR (--strict): <warning>`.
- Workspace mode lists it under the product, and `--strict` fails the product.
- `fluid contract-validation --env` and `fluid test --env` (the native engine) report it too; `fluid test` shows it as an "Environment overlay" row.
- `overlay_entry_added`, an overlay entry the base does not have, stays a log line and does not fail `--strict`: a build only one environment runs is often intended.
- A bundle does not carry it: `fluid bundle --env` merged the overlay before, so `fluid validate <bundle>.tgz --strict` (pipeline stage 2) passes. Validate the contract file with `--env` to catch it.

## Contracts split into fragments

A root contract can pull objects from other files with `$ref`. `fluid validate` resolves the pointers before it checks anything, so you pass the root file and validate the whole product. A `$ref` may only name a file inside the contract's own directory tree. One that climbs out of it is refused:
Expand Down
39 changes: 37 additions & 2 deletions docs/concepts/builds-exposes-bindings.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,43 @@ builds:
- **Deployment settings go under `properties.parameters`.** `embeddedLogicPattern` is closed, so `parallelism` beside `sql` fails validation with `Additional properties are not allowed ('parallelism' was unexpected)`. Name them as the operator does: `parallelism` and `upgradeMode` (`savepoint`, `last-state`, `stateless`) from its job spec, and `flinkConfiguration`.
- **`fluid apply --mode amend-and-build`** prints `Build 'reconcile' - engine 'flink' runs on kubernetes, outside forge-cli; there is nothing for forge-cli to run.` and counts the build on its own summary line, `Run outside forge-cli: 1`. A contract whose only builds run elsewhere exits 0; it is not "every build was skipped". The apply run report records the build as `skipped` with `reason: runs_outside_forge`.
- **No planner runs its SQL.** The local, AWS, GCP and Snowflake planners never turn it into a DuckDB, Athena, BigQuery or Snowflake statement, and the local planner lands nothing for an expose the job writes. In a contract that mixes a Flink build with a local one, name each build's exposes in `outputs`: when it cannot tell which build writes an expose, the local planner lands neither and warns.
- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job.
- **Schedules and generated code.** `fluid generate schedule` gives the build no task or DAG, and a contract with no other build has nothing to schedule (exit 0). `fluid generate transformation` says there is no code to generate for it. Leave `execution.trigger` out for a continuous job, or *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* say so with `trigger: {type: continuous}` (see [Continuous builds](#continuous-builds-trigger-type-continuous)).
- **On 0.7.5**, `engine: flink` fails validation. A 0.7.5 contract declares the same job with `properties.language: flink_sql` (with `engine: sql`, say), and forge-cli treats it the same way.

### Continuous builds: `trigger.type: continuous`

*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A build whose job its platform keeps running, such as a Flink job or a `kafka-connect` or `debezium` connector, can say so (schema 0.7.6 preview):

```yaml
fluidVersion: "0.7.6"
builds:
- id: ingest
engine: kafka-connect
pattern: acquisition
execution:
trigger:
type: continuous
```

- **No scheduler starts it.** `fluid generate schedule`, `fluid schedule-sync` and the `fluid apply` DAGs leave the build out. A connector used to get a nightly `0 2 * * *` task. A contract whose every build is continuous or runs outside forge-cli has nothing to schedule:

```text
Every build runs outside forge-cli (a Flink job, for example) or continuously
(trigger.type: continuous), so there is nothing to schedule. No files written.
```

- **It takes no other trigger.** `cron`, `schedule`, `datasets`, `event` and `timetable` are refused beside it, as Databricks Jobs refuses a continuous trigger beside any other, and the error names the field and the trigger type that takes it:

```text
1. builds[0].execution.trigger: cron is not allowed with type continuous: a continuous job runs all the time, so its trigger takes no other trigger. Remove cron, or set type: schedule.
```

It is not Airflow's `@continuous`, which starts batch runs back to back.
- **On 0.7.5**, `continuous` fails validation: `'continuous' is not one of ['schedule', 'event', 'manual', 'dependency', 'dataset', 'schedule_and_dataset', 'timetable']`.
- **Where it belongs.** Declare it on a build that runs outside forge-cli or on a connector. The dbt and python runners refuse it (`Unknown trigger type: continuous`), as they refuse `event` and `dataset`, and [`fluid scaffold-composer`](../cli/scaffold-composer.md) still writes its nightly DAG for any contract.
- **Overlays.** An environment overlay cannot turn a scheduled build continuous: an overlay cannot delete the base's `cron` or `schedule`, and setting it to `null` is refused too. Declare the trigger in each environment's overlay instead of in the base.
- **A DAG deployed earlier stays.** When every build of a product becomes continuous, it gets no `schedule/` artifacts, so pipeline stage 11 skips `fluid schedule-sync`, and a DAG deployed for the product before stays in the scheduler: remove it by hand.

## `exposes[]` - the consumer-facing API

```yaml
Expand Down Expand Up @@ -517,7 +551,7 @@ builds:
timezone: UTC
```

`trigger.type` is one of `schedule`, `event`, `manual`, `dependency`, `dataset`, `schedule_and_dataset`, `timetable`; `scheduled` is not a value, and the cron expression goes in `schedule` (`cron` is also accepted). `fluid apply --mode amend-and-build` runs the script with the contract's directory as its working directory.
`trigger.type` is one of `schedule`, `event`, `manual`, `dependency`, `dataset`, `schedule_and_dataset`, `timetable`, and in 0.7.6 `continuous` (see [Continuous builds](#continuous-builds-trigger-type-continuous)); `scheduled` is not a value, and the cron expression goes in `schedule` (`cron` is also accepted). `fluid apply --mode amend-and-build` runs the script with the contract's directory as its working directory.

With a `schedule` trigger, `fluid generate schedule contract.fluid.yaml --out <dir>` writes an Airflow DAG with `SCHEDULE = '0 * * * *'`. The DAG's task runs `fluid apply ... --mode amend-and-build --build-id <id> --yes`.

Expand Down Expand Up @@ -545,6 +579,7 @@ Schema `0.7.6` is a preview. It validates only when the file names it, and the s
- **`exposes[].lifecycle.expire`** deletes stored data once it is older than `retention`; it is `false` unless you set it.
- *(unreleased, [forge-cli #731](https://github.com/Agenticstiger/forge-cli/pull/731))* **`builds[].engine: flink`** declares a job that runs outside forge-cli; see [Flink jobs](#flink-jobs-engine-flink).
- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* **Overlays merge `builds`, `exposes` and `consumes` by identity**; see [How an overlay merges](./environments-and-overlays.md#how-an-overlay-merges).
- *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* **`builds[].execution.trigger.type: continuous`** declares a job its platform keeps running, which no scheduler starts; see [Continuous builds](#continuous-builds-trigger-type-continuous).

## Where to look next

Expand Down
3 changes: 3 additions & 0 deletions docs/concepts/environments-and-overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ not name the file.
- *(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* Before 0.7.6 the merge is by position, and a patch that
lands on an entry naming another id logs `overlay_patch_retargeted`: Put the
overlay's entries in the base's order, or set `fluidVersion: "0.7.6"`.
*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* `fluid validate --env`, `fluid contract-validation --env` and
`fluid test --env` report it as a warning instead, and `--strict` fails on
it; see [`fluid validate`: `--env` and environment overlays](../cli/validate.md#env-and-environment-overlays).
- Base keys an overlay does not set survive inside a patched map. A base
`binding.location.path` is still present in the resolved binding of an overlay
that sets `project`, `dataset` and `table`: `fluid bundle --env gcp` shows
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/preview-fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ The [`consumes`](./contract-0.7.6-preview.md#consumes) and [`exposes`](./contrac

*(unreleased, [forge-cli #732](https://github.com/Agenticstiger/forge-cli/pull/732))* In a contract that names 0.7.6, an overlay patches `builds`, `exposes` and `consumes` by identity instead of by position; see [How an overlay merges](../concepts/environments-and-overlays.md#how-an-overlay-merges).

*(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* `builds[].execution.trigger.type: continuous` is a 0.7.6 value too, for a job its platform keeps running: no scheduler starts it, and it takes no other trigger; see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous).

### Federated upstreams

`upstreamWorkspace` and `upstreamDigest` travel together. This pair of fields on a `consumes[]` entry validates under 0.7.6:
Expand Down
2 changes: 1 addition & 1 deletion docs/walkthrough/airflow-declarative.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ The DAG is written for Airflow 3 (`airflow.sdk.DAG`, `BashOperator` from `apache

Stage 3 (`fluid generate artifacts`) writes schedule artifacts when the contract sets `orchestration.engine` to anything but `none`, or when it sets no engine and a build declares `execution.trigger.schedule` (or `cron`) with `type: schedule` or no type. Airflow renders the second case.

Within that, a build gets the `fluid apply` DAG unless its own `execution.orchestration.engine` is `none` or not Airflow, or its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a cron DAG cannot express. A contract that hand-declares `orchestration.tasks` keeps those operators instead.
Within that, a build gets the `fluid apply` DAG unless its own `execution.orchestration.engine` is `none` or not Airflow, or its trigger is `event`, `dataset`, `schedule_and_dataset` or `timetable`, which a cron DAG cannot express. *(unreleased, [forge-cli #734](https://github.com/Agenticstiger/forge-cli/pull/734))* A `continuous` build gets none either: its platform keeps it running (see [Continuous builds](../concepts/builds-exposes-bindings.md#continuous-builds-trigger-type-continuous)). A contract that hand-declares `orchestration.tasks` keeps those operators instead.

### The DAG id and the environment

Expand Down
Loading