fix(ec2): enforce security-group egress and ingress as independent gates - #2525
Merged
Merged
Conversation
SG enforcement rendered every instance's rules into one nftables `forward` chain: an instance's egress allows and default-deny, then the next instance's ingress allows and default-deny. nftables evaluates a chain first-match, so for a packet A -> B, A's egress `ip saddr A accept` (the default security group allows all egress) ended evaluation before B's ingress `ip daddr B drop` was reached whenever A was emitted before B. Instances are keyed by random instance id, so traffic that B's security group denies got through about half the time. That is the cause of the `EC2 SG enforcement (privileged)` job failing intermittently on main and on unrelated PRs: the ruleset captured at failure shows `ip saddr 172.18.0.2 accept` ahead of `ip daddr 172.18.0.3 drop comment "default-deny ingress"`. On AWS both gates must permit a packet. Render them as two base chains on the forward hook, `egress` then `ingress`: an `accept` ends only its own chain and the packet still traverses the other, where a drop is final. NACL denies go in the chain for their direction. The bridge-family L2 table gets the same structure through a shared renderer. Test: a ruleset evaluator with nftables base-chain semantics checks that A -> B is dropped under the default security group in both instance orders, and allowed once B permits ingress from A, for both tables. It fails against the single-chain renderer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
EC2 SG enforcement (privileged)fails intermittently onmainand on PRs that don't touch EC2 (e.g.bb45b79eb,2137ed7b6on main; #2523). It was treated as runner flakiness. It is a real security-group bypass.Cause
SG enforcement rendered every instance's rules into one nftables
forwardchain, per instance in turn: egress allows + default-deny egress, ingress allows + default-deny ingress. nftables evaluates a chain first-match. For a packet A -> B:That is the ruleset captured by the job at failure. Whenever A was emitted before B, A's egress
acceptended evaluation and the packet reached B even though B's security group allows no ingress. Instances are keyed by random instance id, so the order, and the outcome, was a coin flip: the in-job 3x retry fails identically within a run because the ids and order are fixed once launched.The same structure was mirrored into the bridge-family L2 table and applies across subnets too.
Fix
On AWS a packet must be permitted by the sender's egress rules and the receiver's ingress rules. Both tables are now rendered with two base chains on the
forwardhook:egress(inet priority -6 / bridge -301): stateful accept, egress NACL denies, each instance's egress allows + default-deny egressingress(inet -5 / bridge -300): stateful accept, ingress NACL denies, each instance's ingress allows + default-deny ingressAn
acceptends only its own base chain; the packet still traverses the other, where adropis final. So the gates are independent and instance order no longer matters. Both tables share one renderer (render_table/render_chain), so they cannot drift.Surfaces
website/content/docs/services/ec2.mdstates the egress-and-ingress semantics.ec2_sg_enforcement_real.rs: comment and failure message updated (no assertion change; it still looks forip daddr <ip> dropininet fakecloud_ec2).Test plan
egress_and_ingress_are_independent_gates_whatever_the_instance_order: a small evaluator with nftables base-chain semantics (first match per chain,dropfinal,acceptends only its chain) runs against bothrender_rulesetandrender_bridge_ruleset. Under the default security group A -> B and B -> A are dropped in both instance orders; once B allows ingress from A, A -> B passes and B -> A is still dropped. Against the previous single-chain renderer it fails withA -> B must be dropped.fakecloud-ec2lib tests 263/263 (existing ruleset text tests unchanged and passing); clippy-D warningsand fmt clean;ec2_sg_enforcement_realcompiles.nft+CAP_NET_ADMIN; this PR'sEC2 SG enforcement (privileged)run is the end-to-end check.Summary by cubic
Fixes a real EC2 security-group bypass that caused
EC2 SG enforcement (privileged)to fail intermittently on main and on unrelated PRs. Egress and ingress rules used to share one nftables forward chain, so an egressacceptcould end evaluation before a later ingress default-deny; they are now independent base chains, and a packet must pass both, as on AWS.Bug Fixes
inet fakecloud_ec2andbridge fakecloud_ec2_l2through one shared renderer, so L2 enforcement cannot drift.A -> Bis dropped under the default security group and allowed once B permits ingress from A.Testing
nftenforcement.Written for commit 8f539d4. Summary will update on new commits.