Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The site is organised the way a reader uses it (get started, tutorials, how-to g

## Current Versioning

- Current CLI release documented here: `0.18.1`
- Current CLI release documented here: `0.23.0`
- Contract schema: `fluidVersion: 0.7.5` is the stable default; `0.7.6` is an opt-in preview

Those are different on purpose. `fluid version` reports the installed CLI release, while `fluidVersion` inside a contract selects the contract schema version.
Expand Down
4 changes: 2 additions & 2 deletions docs/.vuepress/cli-version.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"supportedCliVersion": "0.18.1",
"supportedCliInstallSpec": "data-product-forge==0.18.1",
"supportedCliVersion": "0.23.0",
"supportedCliInstallSpec": "data-product-forge==0.23.0",
"supportedFluidContractVersion": "0.7.5",
"quickstartScaffoldVersion": "0.7.5",
"supportedSdkVersion": "0.10.0",
Expand Down
9 changes: 4 additions & 5 deletions docs/advanced/project-dotenv-trust.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,12 @@ What it may not set is any variable that
project file has nothing to supply there. The full list is
`fluid_build/credentials/protected_env.py` in forge-cli.

::: tip Two more in the next release after `0.22.0`
::: tip Two more since `0.23.0`
`FLUID_ACQUISITION_SECRET_REF_ALLOWLIST`, which allows an acquisition source's `env://`
reference against a remote host, and `TESTCONTAINERS_HOST_OVERRIDE`, which rewrites a
contract-declared loopback host to a reachable one, join the list on forge-cli `main`
([#751](https://github.com/Agenticstiger/forge-cli/pull/751)). Both decide where a credential
may be sent, so neither may come from a file beside the contract. The release is a minor bump
from `0.22.0` and its number is not settled yet.
contract-declared loopback host to a reachable one, joined the list in `0.23.0`
([forge-cli #751](https://github.com/Agenticstiger/forge-cli/pull/751)). Both decide where a
credential may be sent, so neither may come from a file beside the contract.
:::

### An unlisted name fails closed
Expand Down
4 changes: 2 additions & 2 deletions docs/cli/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# CLI Reference

This section tracks the command surface of `forge-cli` `0.18.1`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands.
This section tracks the command surface of `forge-cli` `0.23.0`, the release `pip install data-product-forge` installs. `fluid --help` shows the promoted commands.

::: tip Looking up *how to do something* (not *what a command does*)?
The page you actually want is [**CLI by task →**](./tasks/) — narrative walkthroughs organized around things you might be trying to accomplish, like "deploy to a new cloud", "add quality rules", "debug a failed run", or "add agent governance". Each task page links to the relevant commands. **This** page is the command index, grouped by purpose — better for lookup, worse for onboarding.
:::

## Read this first

- CLI release examples in this section use `0.18.1`
- CLI release examples in this section use `0.18.1` <!-- cli-version: historical -->
- Contract examples use `fluidVersion: 0.7.5`, the stable schema, which `fluid init --quickstart` emits. `0.7.6` is a preview you opt into per contract
- `fluid version` and `fluidVersion` are different things
- The pinned CLI version is recorded in [`docs/.vuepress/cli-version.json`](https://github.com/Agenticstiger/forge_docs/blob/main/docs/.vuepress/cli-version.json) and enforced by the [`cli-consistency`](https://github.com/Agenticstiger/forge_docs/actions/workflows/cli-consistency.yml) workflow.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/contract-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The contract is loaded the way `fluid plan` loads it, with `$ref` fragments and
```yaml
- name: Schema gate
run: |
pip install data-product-forge==0.18.1
pip install data-product-forge==0.23.0
fluid contract-tests contract.fluid.yaml --baseline baseline.schema.json
```

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/mission.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ cosmetics ([`0.13.0` security fixes](../RELEASE_NOTES_0.13.0.md#security)).
```yaml
- name: Mission gate
run: |
pip install data-product-forge==0.18.1
pip install data-product-forge==0.23.0
fluid mission check quality-coverage contract.fluid.yaml
```

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/scaffold-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The generated file installs `data-product-forge` from PyPI (`pip install --quiet
| publish | On `main` with `DMM_API_URL` set: `fluid publish $CONTRACT --catalog $CATALOG`. |

::: warning Pin the version in a pipeline that holds credentials
An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.18.1"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file.
An unpinned `pip install data-product-forge` takes whatever release is latest when the job runs, and the same job holds the credentials the apply stage uses. After generating, edit the install line to pin the version you tested, for example `pip install --quiet "data-product-forge==0.23.0"`, and move the pin deliberately when you upgrade. [`fluid generate ci`](./generate.md#fluid-generate-ci) pins the install to the version that generated the file.
:::

The settings are `CONTRACT` (the path you passed), `PROVIDER` (default `default`), `BUILD_ID` (default empty; set it to a `builds[].id` to run that build), `CATALOG` (default `datamesh-manager`), `AIRFLOW_DAGS_DEST` and `DMM_API_URL`. GitLab reads them as CI/CD variables, GitHub Actions as `env`, repository `vars` and `secrets`, Jenkins from the `environment` block.
Expand Down
2 changes: 1 addition & 1 deletion docs/cli/version.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Python: 3.11.15 (darwin)
System: Darwin arm64
```

`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.18.1`.
`fluid --version` is the top-level flag and prints one line, `FLUID Forge CLI v0.23.0`.

## Options

Expand Down
2 changes: 1 addition & 1 deletion docs/faq/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ pip install --upgrade "data-product-forge[local]"
fluid version
```

Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.18.1`), not a range.
Then follow the [upgrade guide](../upgrading.md). It has a checklist for each version you cross, because some upgrades need action: `0.17.0`, for example, moves remote state to a per-provider key for a contract that uses a per-contract state key, and needs one `fluid schedule-sync` to retire old Airflow DAGs if you sync Airflow DAGs, and `0.16.3` changed some results on purpose. Pin an exact version in CI (`data-product-forge==0.23.0`), not a range.

Your contracts do not need a new `fluidVersion`: `0.18.1` reads `0.7.1` to `0.7.5`, and `0.7.6` as a preview.

Expand Down
2 changes: 1 addition & 1 deletion docs/providers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Fluid Forge uses one contract format across local and provider-backed execution

## Docs baseline

- CLI release covered by the primary docs: `0.18.1`
- CLI release covered by the primary docs: `0.23.0`
- Contract schema: `0.7.5` is the current stable version; `0.7.6` is a preview you opt into with `fluidVersion: "0.7.6"`. The GCP retention, encryption and `binding.principals` fields need `0.7.6`.
- Which `fluidVersion` each scaffolding command writes is listed in [`fluid init`](../cli/init.md#which-fluidversion-each-path-writes).

Expand Down
2 changes: 1 addition & 1 deletion docs/providers/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Deploy data products to Amazon Web Services (S3, Glue, Athena, Lake Formation) with the same contract and CLI commands you use for the other providers.

**Docs Baseline:** CLI `0.18.1`<br>
**Docs Baseline:** CLI `0.18.1`<br> <!-- cli-version: historical -->
**`fluid apply` creates:** S3 buckets, Glue databases and tables, Lake Formation registrations and grants, KMS keys, S3 lifecycle rules<br>
**`fluid verify` reads:** Glue, Athena, S3, KMS, Lake Formation

Expand Down
2 changes: 1 addition & 1 deletion docs/providers/gcp.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# GCP Provider

**Status:** ✅ Production Ready
**Docs Baseline:** CLI `0.18.1`<br>
**Docs Baseline:** CLI `0.18.1`<br> <!-- cli-version: historical -->
**Services:** BigQuery, Cloud Storage, Pub/Sub, IAM, Data Catalog policy tags, Cloud KMS

> **Why it matters**
Expand Down
2 changes: 1 addition & 1 deletion docs/providers/local.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Local Provider

**Status:** ✅ Production Ready
**Docs Baseline:** CLI `0.18.1`<br>
**Docs Baseline:** CLI `0.18.1`<br> <!-- cli-version: historical -->
**Engine:** DuckDB (1.5.0 or later)

> **Why it matters**
Expand Down
2 changes: 1 addition & 1 deletion docs/providers/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Deploy data products to Snowflake Data Cloud (databases, schemas and tables) using the same contract format and CLI commands as the other providers.

**Status:** ✅ Production
**Docs Baseline:** CLI `0.18.1`<br>
**Docs Baseline:** CLI `0.18.1`<br> <!-- cli-version: historical -->
**Tested Services:** Databases, Schemas, Tables

> **Why it matters**
Expand Down
2 changes: 1 addition & 1 deletion docs/recipes/per-environment-overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ jobs:
matrix:
env: [dev, staging, prod]
steps:
- run: pip install "data-product-forge==0.18.1"
- run: pip install "data-product-forge==0.23.0"
- run: fluid validate contract.fluid.yaml --env ${{ matrix.env }}
- run: fluid plan contract.fluid.yaml --env ${{ matrix.env }}
```
Expand Down
6 changes: 3 additions & 3 deletions docs/reference/contract-0.7.5.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ description: "The fields of the FLUID 0.7.5 contract schema (stable): path, type

Status: **stable**. Schema 0.7.5 is the newest stable schema bundled with the CLI. The newer 0.7.6 schema is a preview; see the [0.7.6 preview delta](./contract-0.7.6-preview.md).

Generated from `fluid_build/schemas/fluid-schema-0.7.5.json` in the `data-product-forge` 0.18.1 package (sha256 `763fb786cd73bc4f`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`.
Generated from `fluid_build/schemas/fluid-schema-0.7.5.json` in the `data-product-forge` 0.23.0 package (sha256 `a1c33dc003599649`). It is the schema `fluid validate` checks a contract against, rendered without edits. To regenerate: `python scripts/gen_contract_reference.py`.

Each table row is one field, addressed by its path from the contract root: `.` descends into an object, `[]` marks the items of a list, and `<key>` stands for any key of a map. **Required** means required inside its parent object; the parent can itself be optional. **Allowed values** lists the enum, pattern and range the schema enforces. An object typed plainly `object` rejects keys the schema does not list; `object (open)` accepts extra keys and `object (free-form)` has no listed keys. A row that begins "Only when ..." exists only under that condition. See [how to read this reference](./README.md#how-to-read-a-row).

Expand Down Expand Up @@ -309,7 +309,7 @@ Each table row is one field, addressed by its path from the contract root: `.` d
| `exposes[].binding.governance.lakeFormation.grants[].permissionsWithGrantOption` | array of string | no | unique items | | Subset of permissions the principal can re-grant. Maps to aws_lakeformation_permissions.permissions_with_grant_option. |
| `exposes[].binding.governance.lakeFormation.grants[].columns` | array of string | no | unique items | | Optional column-level restriction. When set, the grant is emitted against aws_lakeformation_permissions.table_with_columns instead of .table, allowing the principal to read only the listed columns. |
| `exposes[].binding.governance.lakeFormation.grants[].excludedColumns` | array of string | no | unique items | | Opposite of columns: every column EXCEPT these is granted. Maps to table_with_columns.excluded_column_names. Mutually exclusive with columns. |
| `exposes[].binding.governance.lakeFormation.tags` | object (map) | no | | | LF-tag (TBAC) associations applied to this table. Keys must reference tag definitions declared in the contract's top-level governance.lakeFormation.tagDefinitions. Emitted as aws_lakeformation_resource_lf_tags. |
| `exposes[].binding.governance.lakeFormation.tags` | object (map) | no | | | LF-tag (TBAC) associations applied to this table, as tag key to value. Emitted as aws_lakeformation_resource_lf_tags; a key whose value is empty is not associated. A key defined in the contract's top-level governance.lakeFormation.tagDefinitions is a tag this contract creates: the association waits for it, and the value must be one of the tag's values. A key the contract does not define is a tag owned outside the contract, such as a platform's central LF-tag ontology: it must already exist in the account, the principal that runs the apply needs the Lake Formation ASSOCIATE permission on it, and only Lake Formation checks its key and value, at apply. A key the contract defines must not also be created elsewhere. Refused when the binding names no Glue table (location.database and location.table). |
| `exposes[].binding.governance.lakeFormation.tags.<key>` | string | no | | | |
| `exposes[].binding.governance.lakeFormation.rowFilter` | object | no | | | Row-level (and optionally column-level) filter applied to all reads of this table via LF. Emitted as aws_lakeformation_data_cells_filter. The filter must then be granted to principals via grants\[\].permissions including SELECT — LF enforces both layers. |
| `exposes[].binding.governance.lakeFormation.rowFilter.name` | string | yes | | | Filter name (must be unique per table). |
Expand Down Expand Up @@ -1139,5 +1139,5 @@ Each table row is one field, addressed by its path from the contract root: `.` d
| `governance` | object | no | | | Contract-level governance: account/project-wide settings (e.g. AWS Lake Formation admins and LF-tag definitions). Per-resource grants and tag associations live under binding.governance. |
| `governance.lakeFormation` | object | no | | | AWS Lake Formation account-wide settings: data-lake admins and LF-tag (TBAC) definitions. Applies to the AWS account the contract's IaC apply targets. |
| `governance.lakeFormation.admins` | array of string | no | unique items<br>each item: matches `^arn:aws[a-z0-9-]*:iam::` | | IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op. |
| `governance.lakeFormation.tagDefinitions` | object (map) | no | | | LF-TBAC tag-key definitions. Each entry is a tag key mapped to its allowed string values. Emitted as one aws_lakeformation_lf_tag per key; values are the tag's allowed-values list. Field idiom borrowed from OpenMetadata's classification-label model — see ODCS roles + OpenMetadata data contracts spec. |
| `governance.lakeFormation.tagDefinitions` | object (map) | no | | | LF-TBAC tag-key definitions. Each entry is a tag key mapped to its allowed string values. Emitted as one aws_lakeformation_lf_tag per key; values are the tag's allowed-values list. The apply CREATES every tag defined here, so a key defined here must not also be created elsewhere: not by a platform bootstrap, another contract or by hand in the same account. A tag owned outside the contract is not defined here, and a binding's governance.lakeFormation.tags can still associate it. An association with a key defined here waits for the tag, and its value must be one of the tag's values, compared case-insensitively because Lake Formation stores keys and values lower-cased. A key with no values is refused, and so are two keys that would be one resource name, such as a-b and a_b. Field idiom borrowed from OpenMetadata's classification-label model — see ODCS roles + OpenMetadata data contracts spec. |
| `governance.lakeFormation.tagDefinitions.<key>` | array of string | no | min items 1<br>unique items | | |
Loading
Loading