Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/advanced/source-aligned-acquisition.md
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,8 @@ debezium.sink.iceberg.oauth2-server-uri=http://keycloak:8080/realms/iceberg/prot
debezium.sink.iceberg.scope=lakekeeper
```

The source password, and any other property whose value is a credential, is not written to the file. Its line holds an expression, and forge-cli passes the value to Debezium Server in that environment variable: the `secretRef` from the source connection becomes `debezium.source.database.password=${FLUID_DBZ_SOURCE_DATABASE_PASSWORD}`. Quarkus resolves the expression when the server starts. A `server_binary` wrapper that replaces the environment must pass the `FLUID_DBZ_*` variables through. The file is created readable by your user only (mode 0600). A sink value that starts with `env:` is Iceberg's own reference and is written as it is, as above.

`type: bearer` with `tokenEnv: UC_TOKEN` gives `iceberg.catalog.rest.auth.type=oauth2` and `iceberg.catalog.token=env:UC_TOKEN` on Kafka Connect, and `debezium.sink.iceberg.rest.auth.type=oauth2` and `debezium.sink.iceberg.token=env:UC_TOKEN` on Debezium Server. `audience: lakekeeper` adds `iceberg.catalog.audience=lakekeeper`, and `resource` adds `iceberg.catalog.resource` the same way.

#### Where to set the variable
Expand Down Expand Up @@ -658,7 +660,7 @@ With `iceberg_sink_enabled: false` and no hand-written config, the reason reads
1. iceberg sink (build 'stream_orders'): sink.catalogAuth and iceberg_catalog_overrides both set the catalog's auth (iceberg.catalog.credential), so which one reaches the catalog is ambiguous (Iceberg uses a token over a credential when it gets both). Remove those keys from iceberg_catalog_overrides, or remove sink.catalogAuth
```

**A literal secret in an override.** Where no `catalogAuth` applies, an override or a hand-written sink config that sets `credential` or `token` to a literal value is a warning (next to a `catalogAuth` that applies, the override is refused, above), because forge-cli would push the value to the Kafka Connect REST API or write it to `application.properties`. Under `--strict`, `fluid validate` exits `1` on it. A value that starts with `env:` or carries a `${...}` expression, such as a Kafka Connect config provider reference, draws no warning:
**A literal secret in an override.** Where no `catalogAuth` applies, an override or a hand-written sink config that sets `credential` or `token` to a literal value is a warning (next to a `catalogAuth` that applies, the override is refused, above), because the secret sits in the contract, and forge-cli would push it to the Kafka Connect REST API or pass it to Debezium Server. Under `--strict`, `fluid validate` exits `1` on it. A value that starts with `env:` or carries a `${...}` expression, such as a Kafka Connect config provider reference, draws no warning:

```text
✅ Valid FLUID contract (schema v0.7.6)
Expand All @@ -670,7 +672,7 @@ Validation Warnings:
1. iceberg sink (build 'stream_orders'): iceberg_catalog_overrides sets iceberg.catalog.credential to a literal value (not shown), so the secret is inlined into the sink config forge-cli pushes to the Kafka Connect REST API. Use sink.catalogAuth, or set iceberg.catalog.credential to env:<VARIABLE>, which Iceberg resolves from the environment of the process that runs the sink
```

On Debezium Server the warning names `debezium.server.sink.config` and says the value is written to `application.properties`. The runs log the warning and go on.
On Debezium Server the warning names `debezium.server.sink.config` and says the value is passed to Debezium Server. The runs log the warning and go on.

What `catalogAuth` does not cover:

Expand Down
2 changes: 1 addition & 1 deletion docs/cli/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ The next-steps panel stops at `plan`. To run a scaffold, `cd` into it and `fluid
└── forge-receipt.json # provenance record; ignored by the generated .gitignore
```

Commit `fluid.workspace.yaml`, `.gitignore`, `.fluid/team-memory.yaml` and everything in `my-project/` except the receipt. The generated `.gitignore` block excludes `.fluid/init-receipt.json`, `.fluid/forge-receipt.json`, `.fluid/copilot-memory.json`, `.fluid/logs/` and `runtime/`.
Commit `fluid.workspace.yaml`, `.gitignore`, `.fluid/team-memory.yaml` and everything in `my-project/` except the receipt. The generated `.gitignore` block excludes `.fluid/init-receipt.json`, `.fluid/forge-receipt.json`, `.fluid/copilot-memory.json`, `.fluid/logs/`, `**/.fluid/debezium/`, `**/.fluid/meltano/` and `runtime/` ([what to commit](./split.md)).

`--blank`, `--blueprint` and the provider starters write the same workspace files, and a product directory that holds `contract.fluid.yaml` and `.fluid/forge-receipt.json` only. Directory templates add their `README.md` and sample data.

Expand Down
2 changes: 2 additions & 0 deletions docs/cli/mission.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ Scorecard: 1/2 non-advisory checks passing — FAIL

The command exits `1`. With `--json` the same scorecard is one object with `mission`, `passed`, `gating_passed`, `gating_total`, `contract_sha256` and a `results` list; each result has `name`, `passed`, `advisory`, `detail` and `diagnostics`.

A failing `eq`, `ne`, `contains`, `lt`, `lte`, `gt` or `gte` predicate quotes the value it compared, for example `exposes[0].binding.location.database: expected == 'orders', got 'sales'`. A value at or under a key that holds a credential, such as `password`, `secrets`, `auth`, `connection_string` or `apiKey`, shows as `***REDACTED***` instead: in the terminal, in `--json`, in the stored scorecard, and in what `fluid mission run` sends to its LLM provider. A key that refers to a credential (`secretRef`, `tokenEnv`, `catalogAuth.type`) shows its value, as do `maxTokensPerDay`, `authz` and `author`.

## Built-in missions

Two flagship missions ship with the CLI.
Expand Down
9 changes: 9 additions & 0 deletions docs/cli/split.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ block, and where each file lives:
| `<workspace>/.fluid/skills.yaml` | the industry picker in interactive `fluid init` | Yes. The industry reference pack, shared by the team. |
| `<product>/.fluid/forge-receipt.json` | `fluid forge`, `fluid init` | No. |
| `<product>/.fluid/copilot-memory.json` | the AI copilot | No. |
| `<product>/.fluid/debezium/`, `<product>/.fluid/meltano/` | the embedded Debezium Server and Meltano runners, on each run | No (in the block since [forge-cli #722](https://github.com/Agenticstiger/forge-cli/pull/722)). Regenerated each run. Neither holds a credential the run resolved: Debezium Server gets those through its environment, and a Singer tap or target reads its config from a private temporary directory. |
| `<product>/.fluid/ci-state.json` | `fluid forge` when it scaffolds CI, for example `fluid forge --ci <provider>` | Yes. Records the inputs that produced the committed CI files. Files from `fluid generate ci` have no `ci-state.json` record. |
| `runtime/` | build outputs such as bundles, plans and apply reports | No (in the block). |
| `contract.bundled.fluid.yaml` | `fluid bundle` without `--out` on a fragment layout | No. A build output; regenerate it. |
Expand All @@ -206,9 +207,17 @@ The rules in the generated block are:
.fluid/forge-receipt.json
.fluid/copilot-memory.json
.fluid/logs/
**/.fluid/debezium/
**/.fluid/meltano/
runtime/
```

The two `**/` lines come with [forge-cli #722](https://github.com/Agenticstiger/forge-cli/pull/722) and match in every product directory. A
workspace whose `.gitignore` already has the block keeps its old copy: add the two
lines yourself, and check that no `application.properties`, `tap_config.json` or
`target_config.json` under those directories was committed. Earlier releases wrote the
resolved password into them, so rotate any credential you find there.

As of 0.18.1, the `.fluid/...` lines match only the `.fluid/` directory at
the workspace root, because a gitignore pattern with a `/` in the middle is
anchored to the `.gitignore` file's directory. When a product sits in its
Expand Down
Loading