From c1242031c13632fec8ac410aab0506e3d66720a8 Mon Sep 17 00:00:00 2001 From: Speculator55005 <50082482+fas89@users.noreply.github.com> Date: Tue, 13 Oct 2026 23:23:51 +0200 Subject: [PATCH] docs: credentials stay out of generated connector config and mission output (forge-cli #722) - split.md / init.md: the fluid init .gitignore block now ignores **/.fluid/debezium/ and **/.fluid/meltano/ in every product directory; existing workspaces add the lines and check for committed credentials. - source-aligned-acquisition.md: Debezium Server gets credential-valued properties through ${FLUID_DBZ_} environment variables, the file is 0600, and the inline-secret warning says the value is passed to Debezium Server. - mission.md: what a failing predicate quotes, and what it masks. --- docs/advanced/source-aligned-acquisition.md | 6 ++++-- docs/cli/init.md | 2 +- docs/cli/mission.md | 2 ++ docs/cli/split.md | 9 +++++++++ 4 files changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/advanced/source-aligned-acquisition.md b/docs/advanced/source-aligned-acquisition.md index 1dcba57..e86643e 100644 --- a/docs/advanced/source-aligned-acquisition.md +++ b/docs/advanced/source-aligned-acquisition.md @@ -546,6 +546,8 @@ debezium.sink.iceberg.oauth2-server-uri=http://keycloak:8080/realms/iceberg/prot debezium.sink.iceberg.scope=lakekeeper ``` +The source password, and any other property whose value is a credential, is not written to the file. Its line holds an expression, and forge-cli passes the value to Debezium Server in that environment variable: the `secretRef` from the source connection becomes `debezium.source.database.password=${FLUID_DBZ_SOURCE_DATABASE_PASSWORD}`. Quarkus resolves the expression when the server starts. A `server_binary` wrapper that replaces the environment must pass the `FLUID_DBZ_*` variables through. The file is created readable by your user only (mode 0600). A sink value that starts with `env:` is Iceberg's own reference and is written as it is, as above. + `type: bearer` with `tokenEnv: UC_TOKEN` gives `iceberg.catalog.rest.auth.type=oauth2` and `iceberg.catalog.token=env:UC_TOKEN` on Kafka Connect, and `debezium.sink.iceberg.rest.auth.type=oauth2` and `debezium.sink.iceberg.token=env:UC_TOKEN` on Debezium Server. `audience: lakekeeper` adds `iceberg.catalog.audience=lakekeeper`, and `resource` adds `iceberg.catalog.resource` the same way. #### Where to set the variable @@ -658,7 +660,7 @@ With `iceberg_sink_enabled: false` and no hand-written config, the reason reads 1. iceberg sink (build 'stream_orders'): sink.catalogAuth and iceberg_catalog_overrides both set the catalog's auth (iceberg.catalog.credential), so which one reaches the catalog is ambiguous (Iceberg uses a token over a credential when it gets both). Remove those keys from iceberg_catalog_overrides, or remove sink.catalogAuth ``` -**A literal secret in an override.** Where no `catalogAuth` applies, an override or a hand-written sink config that sets `credential` or `token` to a literal value is a warning (next to a `catalogAuth` that applies, the override is refused, above), because forge-cli would push the value to the Kafka Connect REST API or write it to `application.properties`. Under `--strict`, `fluid validate` exits `1` on it. A value that starts with `env:` or carries a `${...}` expression, such as a Kafka Connect config provider reference, draws no warning: +**A literal secret in an override.** Where no `catalogAuth` applies, an override or a hand-written sink config that sets `credential` or `token` to a literal value is a warning (next to a `catalogAuth` that applies, the override is refused, above), because the secret sits in the contract, and forge-cli would push it to the Kafka Connect REST API or pass it to Debezium Server. Under `--strict`, `fluid validate` exits `1` on it. A value that starts with `env:` or carries a `${...}` expression, such as a Kafka Connect config provider reference, draws no warning: ```text ✅ Valid FLUID contract (schema v0.7.6) @@ -670,7 +672,7 @@ Validation Warnings: 1. iceberg sink (build 'stream_orders'): iceberg_catalog_overrides sets iceberg.catalog.credential to a literal value (not shown), so the secret is inlined into the sink config forge-cli pushes to the Kafka Connect REST API. Use sink.catalogAuth, or set iceberg.catalog.credential to env:, which Iceberg resolves from the environment of the process that runs the sink ``` -On Debezium Server the warning names `debezium.server.sink.config` and says the value is written to `application.properties`. The runs log the warning and go on. +On Debezium Server the warning names `debezium.server.sink.config` and says the value is passed to Debezium Server. The runs log the warning and go on. What `catalogAuth` does not cover: diff --git a/docs/cli/init.md b/docs/cli/init.md index 6b5eace..6f478a0 100644 --- a/docs/cli/init.md +++ b/docs/cli/init.md @@ -70,7 +70,7 @@ The next-steps panel stops at `plan`. To run a scaffold, `cd` into it and `fluid └── forge-receipt.json # provenance record; ignored by the generated .gitignore ``` -Commit `fluid.workspace.yaml`, `.gitignore`, `.fluid/team-memory.yaml` and everything in `my-project/` except the receipt. The generated `.gitignore` block excludes `.fluid/init-receipt.json`, `.fluid/forge-receipt.json`, `.fluid/copilot-memory.json`, `.fluid/logs/` and `runtime/`. +Commit `fluid.workspace.yaml`, `.gitignore`, `.fluid/team-memory.yaml` and everything in `my-project/` except the receipt. The generated `.gitignore` block excludes `.fluid/init-receipt.json`, `.fluid/forge-receipt.json`, `.fluid/copilot-memory.json`, `.fluid/logs/`, `**/.fluid/debezium/`, `**/.fluid/meltano/` and `runtime/` ([what to commit](./split.md)). `--blank`, `--blueprint` and the provider starters write the same workspace files, and a product directory that holds `contract.fluid.yaml` and `.fluid/forge-receipt.json` only. Directory templates add their `README.md` and sample data. diff --git a/docs/cli/mission.md b/docs/cli/mission.md index 86e0ccd..5e87caf 100644 --- a/docs/cli/mission.md +++ b/docs/cli/mission.md @@ -103,6 +103,8 @@ Scorecard: 1/2 non-advisory checks passing — FAIL The command exits `1`. With `--json` the same scorecard is one object with `mission`, `passed`, `gating_passed`, `gating_total`, `contract_sha256` and a `results` list; each result has `name`, `passed`, `advisory`, `detail` and `diagnostics`. +A failing `eq`, `ne`, `contains`, `lt`, `lte`, `gt` or `gte` predicate quotes the value it compared, for example `exposes[0].binding.location.database: expected == 'orders', got 'sales'`. A value at or under a key that holds a credential, such as `password`, `secrets`, `auth`, `connection_string` or `apiKey`, shows as `***REDACTED***` instead: in the terminal, in `--json`, in the stored scorecard, and in what `fluid mission run` sends to its LLM provider. A key that refers to a credential (`secretRef`, `tokenEnv`, `catalogAuth.type`) shows its value, as do `maxTokensPerDay`, `authz` and `author`. + ## Built-in missions Two flagship missions ship with the CLI. diff --git a/docs/cli/split.md b/docs/cli/split.md index 83d2551..e892ecc 100644 --- a/docs/cli/split.md +++ b/docs/cli/split.md @@ -195,6 +195,7 @@ block, and where each file lives: | `/.fluid/skills.yaml` | the industry picker in interactive `fluid init` | Yes. The industry reference pack, shared by the team. | | `/.fluid/forge-receipt.json` | `fluid forge`, `fluid init` | No. | | `/.fluid/copilot-memory.json` | the AI copilot | No. | +| `/.fluid/debezium/`, `/.fluid/meltano/` | the embedded Debezium Server and Meltano runners, on each run | No (in the block since [forge-cli #722](https://github.com/Agenticstiger/forge-cli/pull/722)). Regenerated each run. Neither holds a credential the run resolved: Debezium Server gets those through its environment, and a Singer tap or target reads its config from a private temporary directory. | | `/.fluid/ci-state.json` | `fluid forge` when it scaffolds CI, for example `fluid forge --ci ` | Yes. Records the inputs that produced the committed CI files. Files from `fluid generate ci` have no `ci-state.json` record. | | `runtime/` | build outputs such as bundles, plans and apply reports | No (in the block). | | `contract.bundled.fluid.yaml` | `fluid bundle` without `--out` on a fragment layout | No. A build output; regenerate it. | @@ -206,9 +207,17 @@ The rules in the generated block are: .fluid/forge-receipt.json .fluid/copilot-memory.json .fluid/logs/ +**/.fluid/debezium/ +**/.fluid/meltano/ runtime/ ``` +The two `**/` lines come with [forge-cli #722](https://github.com/Agenticstiger/forge-cli/pull/722) and match in every product directory. A +workspace whose `.gitignore` already has the block keeps its old copy: add the two +lines yourself, and check that no `application.properties`, `tap_config.json` or +`target_config.json` under those directories was committed. Earlier releases wrote the +resolved password into them, so rotate any credential you find there. + As of 0.18.1, the `.fluid/...` lines match only the `.fluid/` directory at the workspace root, because a gitignore pattern with a `/` in the middle is anchored to the `.gitignore` file's directory. When a product sits in its