Skip to content

security: a contract could send a product's credential to any host it named - #758

Merged
fas89 merged 2 commits into
mainfrom
fix/acquisition-deployment-host-allowlist
Oct 27, 2026
Merged

fas89 merged 2 commits into
mainfrom
fix/acquisition-deployment-host-allowlist

Conversation

@fas89

@fas89 fas89 commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Closes the board's "[P0][Security][forge-cli] A contract's deployment.server_url receives the resolved database credential, with no host allowlist" (Trello UQHdgLWD), the gap documented as a known limitation when #751 shipped.

The defect

On the kafka-connect path, and on the default bring-your-own / managed modes of debezium and airbyte, forge-cli never connects to the source database. It resolves the contract's connection.secretRef, puts the plaintext into the connector config (database.password, connection.password) and POSTs the whole config to properties.<engine>.deployment.server_url — a URL the contract chooses. Nothing constrained that host, so the credential could be delivered anywhere, needing no allowlist entry and no operator mistake. A loopback connection.host even made the build look local, so #751's env:// policy cleared it as staying on this machine.

A public orchestrator host must now be named in FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST. Loopback and private hosts need no entry — an orchestrator on the cluster network is the ordinary deployment, and a credential that stays inside the organisation is not the leak this closes. Link-local and unspecified are refused outright, allowlist or not: that is where instance metadata lives.

The card's own suggestion was wrong

It said to mirror FLUID_COMMAND_CENTER_HOST_ALLOWLIST, FLUID_FEDERATION_HOST_ALLOWLIST and FLUID_WEBHOOK_HOST_ALLOWLIST. All three are SSRF allowlists: unset, they permit any public host and refuse private ones, and the Command Center gate still lets a credential reach any public https endpoint unlisted. Mirroring them would have reproduced the hole. The deny-unless-named shape comes from _secret_policy.py, which answers the same question — may this credential go where the contract says? — for connection.secretRef.

What the adversarial review caught, and it was a lot

Four lenses, every finding handed to a skeptic told to refute it: 23 raised, 10 refuted, 13 confirmed. Three HIGHs were the same bug, found independently, and it defeated this PR's own control:

  • A mixed DNS answer bypassed the policy completely. hostname_scope stopped at the first non-public address, so an attacker naming the host answers with one decoy private record beside their public one: the policy called the host internal, allowed it unlisted, and safe_http pinned the public address anyway. Reproduced with a faked resolver. The classifier now reports public / non_public / mixed / unresolved and examines every address; mixed needs an entry like any public host. _hostname_is_private keeps answering True for mixed and unresolved — its fail-closed direction, the opposite of this gate's, which is why the states cannot be collapsed.
  • "A refused deployment never reads the secret" was false, and the test pinning it could not fail. All three runners build the connector config — which resolves the secretRef — before constructing the client, so a refused deployment had already fetched the password from its secret store. The gate now also runs where each runner reads server_url. My test patched _acquisition_common.resolve_connection_secrets while the runner binds that name in its own namespace, and never drove the runner at all; it now drives execute_kafka_connect_build with the spy patched where the runner looks it up.
  • A Tailscale / CGNAT orchestrator was refused as "public". The tree's extended non-routable-range list lived in util/safe_http while this classifier used bare ipaddress flags, so 100.64.0.0/10 was public to one and private to the other. The predicate and the list move to the tier-0 _net leaf and safe_http re-exports them — a duplicate definition removed, not added. _hostname_is_private becomes stricter for CGNAT/6to4/NAT64/TEST-NET, which only ever refuses more.
  • A refusal was being swallowed. The kafka-connect status poll caught it as eventual consistency and retried to the deadline; the outer handler flattened it to str(exc), losing the why and the fix that names the variable. Both re-raise now.

The test findings were fair too: the file resolved example.com and skipped on failure, so six of eight DNS-dependent assertions could pass without testing anything, and the per-request guard was pinned only by len(event_hooks) >= 2, which a no-op satisfies.

One thing the live run taught

fluid import airbyte --server-url https://… was refused — but that URL is the operator's, typed on the command line, and this policy exists because a contract can choose the destination. It blocked a legitimate action and told the operator to set an allowlist variable to permit what they had just asked for. operator_supplied=True now marks that path, with a message that blames no contract and does not offer the allowlist as the remedy for a metadata address. The scheme and link-local rules still apply there.

Shape

One policy module; the check sits in the two REST client constructors (KafkaConnectRestClient, which debezium also builds, and AirbyteRestClient) and at the three runner server_url reads. A fourth engine costs one line. The variable is operator-only via credentials/protected_env.py, because fluid apply hydrates .env, .env.<env> and FLUID_SECRETS_FILE, which are committed beside the contract.

Validation

  • 56 tests, DNS faked throughout, so nothing depends on the network or can pass vacuously.
  • Five mechanisms mutation-checked: first-hit classification, the unconsulted deny-list, the missing mixed branch, the removed runner-level gate, and a no-op request guard each fail their own test.
  • Full suite, CI-shaped, both floors, sequential: 27989 passed / 4 failed (duckdb 2.0.0) and 27988 passed / 4 failed (1.5.6) — the same four that fail on clean base. The existing airbyte/debezium/kafka-connect suites pass unchanged.
  • Live, through the CLI: fluid import airbyte --server-url https://example.com passes the gate and fails on the real endpoint; the same against 169.254.169.254 is refused; a contract-chosen public host is refused with exit 1.
  • ruff, black==24.10.0, license headers, import-linter (6 kept / 0 broken), startup budget 6/6, detect-secrets (CI-pinned 1.5.0, baseline untouched) all clean.

Documentation

  • No docs needed — the refusal explains itself and names the variable to set; the upgrade note is in CHANGELOG.md.

… named

On the kafka-connect path, and on the DEFAULT `bring-your-own` / `managed`
modes of debezium and airbyte, forge-cli never connects to the source
database. It resolves the contract's `connection.secretRef`, puts the
plaintext into the connector config (`database.password`,
`connection.password`) and POSTs the whole config to
`properties.<engine>.deployment.server_url` — a URL the CONTRACT chooses.
Nothing constrained that host, so the credential could go anywhere, needing no
allowlist entry and no operator mistake; a loopback `connection.host` even made
the build look local, so the `env://` policy from #751 cleared it as staying on
this machine.

A public orchestrator host must now be named in
`FLUID_ACQUISITION_DEPLOYMENT_HOST_ALLOWLIST`. Loopback and private hosts need
no entry — the ordinary cluster deployment, and a credential that stays inside
the organisation is not the leak this closes. Link-local and unspecified are
refused outright, allowlist or not: that is where instance metadata lives.

**The three existing HOST_ALLOWLIST variables are the wrong model.** The card
said to mirror `FLUID_COMMAND_CENTER_HOST_ALLOWLIST`,
`FLUID_FEDERATION_HOST_ALLOWLIST` and `FLUID_WEBHOOK_HOST_ALLOWLIST`. All three
are SSRF allowlists: unset, they PERMIT any public host and refuse private
ones, and the Command Center gate still lets a credential reach any public
`https` endpoint unlisted. Mirroring them would have reproduced the hole. The
deny-unless-named shape comes from `_secret_policy.py`, which answers the same
question for `connection.secretRef`.

**A mixed DNS answer is public.** `_net.hostname_scope` reports `public`,
`non_public`, `mixed` or `unresolved`, and examines every address. Stopping at
the first non-public one — which the first version of this did — was a complete
bypass: an attacker naming the host answers with one private record beside
their public one, the policy calls the host internal and allows it unlisted,
and `safe_http` pins the public address anyway. `_hostname_is_private` keeps
answering True for `mixed` and `unresolved`, its fail-closed direction and the
opposite of this gate's, which is why the states cannot be collapsed.

**The address predicate and the non-routable-range list move to tier-0 `_net`
and `safe_http` re-exports them.** They existed in both places and disagreed:
`_hostname_is_private` judged on bare `ipaddress` flags, so a Tailscale/CGNAT
address (`100.64.0.0/10`) was public there and non-public in `safe_http`. This
policy therefore refused a CGNAT orchestrator as "public" — wrong answer, wrong
reason. One list now, consulted by both; `_hostname_is_private` gets stricter
for CGNAT/6to4/NAT64/TEST-NET, which only ever refuses more.

**The gate runs where each runner reads `server_url`,** before the connector
config is built — and building it is what resolves the `secretRef`. The client
constructors keep the same check as the chokepoint for any other caller, which
is also where an operator-supplied URL is distinguished. The constructor check
ALONE was not enough and the first version of this promised an ordering it did
not have: all three runners build the config first, so a refused deployment had
already fetched the password from Vault/KMS.

**A refusal is no longer swallowed.** The kafka-connect status poll caught it as
eventual consistency and retried to the deadline; the outer handler flattened it
to `str(exc)`, dropping the `why` and the `fix` that names the variable. Both
re-raise it now.

`fluid import airbyte --server-url` exposed an over-reach in the live run: that
URL is the OPERATOR's, and the policy exists because a CONTRACT can choose the
destination. It refused a legitimate action and told the operator to set an
allowlist variable to permit what they had just asked for.
`operator_supplied=True` marks that path, with a message that blames no contract
and does not offer the allowlist as the remedy for a metadata address.

Modular by request: one policy module, two client constructors (debezium shares
kafka-connect's), three runner call sites, so a fourth engine costs one line.
The variable is operator-only in `credentials/protected_env.py`, because `fluid
apply` hydrates `.env`, `.env.<env>` and `FLUID_SECRETS_FILE`, committed beside
the contract.

Validation. 56 tests, DNS faked throughout — the first version resolved
`example.com` and skipped on failure, so six of eight DNS-dependent assertions
could pass without testing anything. Five mechanisms are mutation-checked:
first-hit classification, the unconsulted deny-list, the missing `mixed`
branch, the removed runner-level gate and a no-op request guard each fail their
own test. Full suite CI-shaped on both floors, run sequentially: 27989 and
27988 passed, 4 failed — the same four that fail on clean base. Live:
`fluid import airbyte --server-url https://example.com` passes the gate and
fails on the real endpoint; the same command against 169.254.169.254 is
refused; a contract-chosen public host is refused with exit 1. ruff,
black==24.10.0, license headers, import-linter (6 kept / 0 broken), startup
budget and detect-secrets all clean.
@github-actions github-actions Bot added security Security-related changes or reports cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes labels Oct 27, 2026
Comment thread tests/build_runners/test_deployment_host_policy.py Fixed
The ordering test reached the runner module through a dotted `import ... as`
while every other test in the file used `from ... import`, which the quality
bot flagged on #758. The from-import of the module object keeps the
monkeypatch target (the runner's own namespace, which is the whole point of
that test) without a `sys.modules` lookup.
@fas89
fas89 merged commit 5f2ec50 into main Oct 27, 2026
37 checks passed
@fas89
fas89 deleted the fix/acquisition-deployment-host-allowlist branch October 27, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation security Security-related changes or reports tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant