Repository navigation
fix(fabric-api): Refuse forbidden probe targets and redial for permits - #1141
Conversation
The gateway checked a Looking Glass probe's target only for form: a numeric address in the query's family. It admitted a ping or traceroute to a private, documentation, unique-local or multicast address, took its snapshot and dispatched it, and every node then refused it with DestinationNotAllowed. The caller got a complete stream reporting ExecutionFailed instead of an INVALID_ARGUMENT refusal, and the request cost a cell and a project request token, a node slot per selected router and a reservation of the probe's packets for traffic that was never sent. The service contract requires the public destination policy at both the gateway and the node, before the snapshot at the gateway. The gateway now applies the same public policy the nodes do (special-use ranges, IPv6 outside 2000::/3) to every service probe while validating it, and refuses a forbidden target with DestinationNotAllowed before any charge. A new gateway flag, --deny-prefixes, carries the deployment's platform and management ranges, as the node flag of the same name does; operator allow exceptions never apply. The nodes keep their own check. The gateway tests used documentation addresses as probe targets for valid requests; they now use public ones.
The interceptor logs every failed call at info level as "rpc failed" with the error's message. When the gateway refuses a malformed Looking Glass request, the message quotes the caller's target, for example an IPv4-mapped address or an address in the wrong family, so the gateway's routine log carried customer probe targets. The service contract keeps targets out of routine logs. A failed Looking Glass service call is now logged with its error code and, for an invalid query, its query code, but not its message. The caller still receives the full message. Operator and FabricQuery calls keep the message in the log line, since their callers are operators diagnosing the fabric.
With components/fabric-api-require-permits, a node's sidecar obtains a permit from the cell gateway before it runs a probe an operator sent it directly, and it does so as a TLS client. components/fabric-api issues node certificates for server auth only, and the gateway requires client auth of its callers, so the handshake failed and every direct operator probe on such a node was refused with LimiterUnavailable. The require-permits Component now issues each node's certificate for client and server auth. Applying the Component already restarts the fabric-router pods; the certificate change adds a restart of the fabric-api-certs pods and none of the fabric-router pods. Without the Component, nodes never act as TLS clients and their certificates stay server-only.
A node's sidecar kept one connection to the cell gateway for probe permits. A TLS connection presents the certificate it was opened with for its whole life, while the gateway re-verifies the caller's certificate on every call. The node's certificate belongs to its fabric-api-certs pod, so when that pod was replaced (a DaemonSet rollout or an eviction) the sidecar loaded a certificate naming the new pod but kept asking for permits under the old one, and the gateway refused each request with Unauthorized until the connection happened to close. A renewal would have failed the same way once the old certificate expired. The sidecar now opens a new connection each time its credentials load and closes the previous one once the requests running on it end, so a reload never interrupts a permit request.
The gateway's destination check applies the public probe policy, the special-use ranges and the IPv6 global-unicast rule, whether or not --deny-prefixes names any deployment ranges. The validation test configured a deny list, so nothing showed that a gateway deployed without one still refuses a unique-local target. It now checks that too. Related to #1048
b8fd549 to
a48892a
Compare
|
Two independent reviews ran on this PR, then a second pass on the fixed head. The first pass split. One reviewer held on a manifest render it could not run and on a claim that an empty deny list makes the destination check inert. The render fixture in the repository shows the client auth key usage on the certificate DaemonSet, and a new unit test shows the gateway refuses a unique local target with no deny list. The second pass also split, holding on body wording about router restarts, which is now corrected in the body and the commit message. The open nits are left as is. CI is green on the current head. This is the bottom of a native stack with #1142, and GitHub does not allow auto-merge on stacked pull requests, so the stack merges by hand from #1142 once approved. |
ecv
left a comment
There was a problem hiding this comment.
Approve.
Forbidden probe targets are now refused at the gateway before any charge or dispatch, the target is gone from the gateway's service-call logs, and the redial cannot loop. One log path still quotes the target on the nodes.
- Refusal runs in
serviceQueryduringvalidate, ahead of the snapshot and budget charge.DestinationPolicy.Checkunmaps v4-mapped addresses, rejects zones, and covers private, CGN, loopback, link-local, multicast, reserved, documentation and 6to4 for v4, plus everything outside 2000::/3 for v6. Table tests cover private, multicast, ULA, documentation, a configured deny range, and an empty--deny-prefixes. - Log leak remains on the node:
logCallonly drops the message forLookingGlassServicemethods. A service probe the gateway forwards to a node asFabricService/Executestill fails with "destination is in ..." (runProbepolicy.Check), and the node logs that message at Info. It triggers whenever the node refuses what the gateway admitted, for example a gateway--deny-prefixesnarrower than the nodes'. Omit the message for any request that carries a service-origin probe, or log by code alone onExecutefailures from the gateway role. - Client-auth EKU does not widen what a node cert can do. Nodes only reach the gateway's
RequestPermit/ReleasePermitperGatewayMethods, and node servers still accept only gateway and operator identities (AllowedCaller). The patch changes only thefabric-api-certsDaemonSet's CSI usages, and the require-permits component is opt-in. - Redial cannot spin:
OnLoadfires only when the cert, key or CA digest changes (30 s poll),grpc.NewClientdials lazily, a failed redial keeps the old connection, and the old one closes after in-flight calls finish. A call with no deadline would hold the old connection open; permit calls carry a deadline from the request. --deny-prefixesdefaults to empty on the gateway and nothing underconfig/sets it, so until each cell sets it the platform ranges are enforced only on the nodes, after the request is charged. The docs say so, and the gap is real for a new cell.
Follow-ups:
- Wire the gateway's
--deny-prefixesfrom the same source as the nodes' value, or fail the gateway start when nodes and gateway disagree, so the pre-charge refusal covers platform ranges by default. - Add a test that the node's failure log for a refused service probe omits the target.
Summary
Running the Looking Glass service on real fabric routers found four defects that tests against fake nodes miss: the gateway charged and dispatched probes to forbidden destinations, logged callers' probe targets, and broke direct operator probes once permits were required, because node certificates lacked client auth and a node kept presenting a replaced certificate.
This change refuses forbidden targets before any charge, keeps targets out of service failure logs, issues node certificates for client auth when permits are required, and reconnects to the gateway whenever a node's credentials reload.
Each deployment should give the gateway the same denied platform ranges its nodes have, which the security guide describes, and the certificate change adds a restart of the certificate pods to the permits rollout but none of the routers.
Stack
First of two stacked PRs. The second, #1142, adds the qualification suite that found these defects, with its results. Rolling back needs no migration.
Test plan
Related to #1048