Skip to content

fix(fabric-api): Refuse forbidden probe targets and redial for permits - #1141

Merged
privateip merged 5 commits into
mainfrom
fix/lg-qualify-findings
Oct 27, 2026
Merged

privateip merged 5 commits into
mainfrom
fix/lg-qualify-findings

Conversation

@privateip

@privateip privateip commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Running the Looking Glass service on real fabric routers found four defects that tests against fake nodes miss: the gateway charged and dispatched probes to forbidden destinations, logged callers' probe targets, and broke direct operator probes once permits were required, because node certificates lacked client auth and a node kept presenting a replaced certificate.

This change refuses forbidden targets before any charge, keeps targets out of service failure logs, issues node certificates for client auth when permits are required, and reconnects to the gateway whenever a node's credentials reload.

Each deployment should give the gateway the same denied platform ranges its nodes have, which the security guide describes, and the certificate change adds a restart of the certificate pods to the permits rollout but none of the routers.

Stack

First of two stacked PRs. The second, #1142, adds the qualification suite that found these defects, with its results. Rolling back needs no migration.

Test plan

  • Unit tests reproduce each defect, failing without the fix and passing with it
  • On the containerlab lab, the qualification suite's destination, confidentiality and direct-probe scenarios pass, including a direct probe after its node's certificate pod is replaced

Related to #1048

The gateway checked a Looking Glass probe's target only for form: a numeric address in the query's family. It admitted a ping or traceroute to a private, documentation, unique-local or multicast address, took its snapshot and dispatched it, and every node then refused it with DestinationNotAllowed. The caller got a complete stream reporting ExecutionFailed instead of an INVALID_ARGUMENT refusal, and the request cost a cell and a project request token, a node slot per selected router and a reservation of the probe's packets for traffic that was never sent. The service contract requires the public destination policy at both the gateway and the node, before the snapshot at the gateway.

The gateway now applies the same public policy the nodes do (special-use ranges, IPv6 outside 2000::/3) to every service probe while validating it, and refuses a forbidden target with DestinationNotAllowed before any charge. A new gateway flag, --deny-prefixes, carries the deployment's platform and management ranges, as the node flag of the same name does; operator allow exceptions never apply. The nodes keep their own check.

The gateway tests used documentation addresses as probe targets for valid requests; they now use public ones.
The interceptor logs every failed call at info level as "rpc failed" with the error's message. When the gateway refuses a malformed Looking Glass request, the message quotes the caller's target, for example an IPv4-mapped address or an address in the wrong family, so the gateway's routine log carried customer probe targets. The service contract keeps targets out of routine logs.

A failed Looking Glass service call is now logged with its error code and, for an invalid query, its query code, but not its message. The caller still receives the full message. Operator and FabricQuery calls keep the message in the log line, since their callers are operators diagnosing the fabric.
@privateip privateip self-assigned this Oct 27, 2026
@privateip privateip changed the title fix(fabric-api): Fix four Looking Glass defects found in qualification fix(fabric-api): Refuse forbidden probe targets and keep permits working Oct 27, 2026
With components/fabric-api-require-permits, a node's sidecar obtains a permit from the cell gateway before it runs a probe an operator sent it directly, and it does so as a TLS client. components/fabric-api issues node certificates for server auth only, and the gateway requires client auth of its callers, so the handshake failed and every direct operator probe on such a node was refused with LimiterUnavailable.

The require-permits Component now issues each node's certificate for client and server auth. Applying the Component already restarts the fabric-router pods; the certificate change adds a restart of the fabric-api-certs pods and none of the fabric-router pods. Without the Component, nodes never act as TLS clients and their certificates stay server-only.
A node's sidecar kept one connection to the cell gateway for probe permits. A TLS connection presents the certificate it was opened with for its whole life, while the gateway re-verifies the caller's certificate on every call. The node's certificate belongs to its fabric-api-certs pod, so when that pod was replaced (a DaemonSet rollout or an eviction) the sidecar loaded a certificate naming the new pod but kept asking for permits under the old one, and the gateway refused each request with Unauthorized until the connection happened to close. A renewal would have failed the same way once the old certificate expired.

The sidecar now opens a new connection each time its credentials load and closes the previous one once the requests running on it end, so a reload never interrupts a permit request.
The gateway's destination check applies the public probe policy, the special-use ranges and the IPv6 global-unicast rule, whether or not --deny-prefixes names any deployment ranges. The validation test configured a deny list, so nothing showed that a gateway deployed without one still refuses a unique-local target. It now checks that too.

Related to #1048
@privateip
privateip force-pushed the fix/lg-qualify-findings branch from b8fd549 to a48892a Compare October 27, 2026 18:03
@privateip privateip changed the title fix(fabric-api): Refuse forbidden probe targets and keep permits working fix(fabric-api): Refuse forbidden probe targets and redial for permits Oct 27, 2026
@privateip
privateip added this pull request to stack #1143 October 27, 2026 18:49
@privateip
privateip marked this pull request as ready for review October 27, 2026 19:01
@privateip
privateip requested a review from a team as a code owner October 27, 2026 19:01
@privateip

Copy link
Copy Markdown
Collaborator Author

Two independent reviews ran on this PR, then a second pass on the fixed head. The first pass split. One reviewer held on a manifest render it could not run and on a claim that an empty deny list makes the destination check inert. The render fixture in the repository shows the client auth key usage on the certificate DaemonSet, and a new unit test shows the gateway refuses a unique local target with no deny list. The second pass also split, holding on body wording about router restarts, which is now corrected in the body and the commit message. The open nits are left as is.

CI is green on the current head. This is the bottom of a native stack with #1142, and GitHub does not allow auto-merge on stacked pull requests, so the stack merges by hand from #1142 once approved.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Forbidden probe targets are now refused at the gateway before any charge or dispatch, the target is gone from the gateway's service-call logs, and the redial cannot loop. One log path still quotes the target on the nodes.

  • Refusal runs in serviceQuery during validate, ahead of the snapshot and budget charge. DestinationPolicy.Check unmaps v4-mapped addresses, rejects zones, and covers private, CGN, loopback, link-local, multicast, reserved, documentation and 6to4 for v4, plus everything outside 2000::/3 for v6. Table tests cover private, multicast, ULA, documentation, a configured deny range, and an empty --deny-prefixes.
  • Log leak remains on the node: logCall only drops the message for LookingGlassService methods. A service probe the gateway forwards to a node as FabricService/Execute still fails with "destination is in ..." (runProbe policy.Check), and the node logs that message at Info. It triggers whenever the node refuses what the gateway admitted, for example a gateway --deny-prefixes narrower than the nodes'. Omit the message for any request that carries a service-origin probe, or log by code alone on Execute failures from the gateway role.
  • Client-auth EKU does not widen what a node cert can do. Nodes only reach the gateway's RequestPermit/ReleasePermit per GatewayMethods, and node servers still accept only gateway and operator identities (AllowedCaller). The patch changes only the fabric-api-certs DaemonSet's CSI usages, and the require-permits component is opt-in.
  • Redial cannot spin: OnLoad fires only when the cert, key or CA digest changes (30 s poll), grpc.NewClient dials lazily, a failed redial keeps the old connection, and the old one closes after in-flight calls finish. A call with no deadline would hold the old connection open; permit calls carry a deadline from the request.
  • --deny-prefixes defaults to empty on the gateway and nothing under config/ sets it, so until each cell sets it the platform ranges are enforced only on the nodes, after the request is charged. The docs say so, and the gap is real for a new cell.

Follow-ups:

  • Wire the gateway's --deny-prefixes from the same source as the nodes' value, or fail the gateway start when nodes and gateway disagree, so the pre-charge refusal covers platform ranges by default.
  • Add a test that the node's failure log for a refused service probe omits the target.

@privateip
privateip merged commit 0e028cf into main Oct 27, 2026
23 checks passed
@privateip
privateip deleted the fix/lg-qualify-findings branch October 27, 2026 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants