Skip to content

fix(ec2): enforce security-group egress and ingress as independent gates - #2525

Merged
vieiralucas merged 1 commit into
mainfrom
fix/ec2-sg-independent-gates
Sep 14, 2026
Merged

vieiralucas merged 1 commit into
mainfrom
fix/ec2-sg-independent-gates

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Summary

EC2 SG enforcement (privileged) fails intermittently on main and on PRs that don't touch EC2 (e.g. bb45b79eb, 2137ed7b6 on main; #2523). It was treated as runner flakiness. It is a real security-group bypass.

Cause

SG enforcement rendered every instance's rules into one nftables forward chain, per instance in turn: egress allows + default-deny egress, ingress allows + default-deny ingress. nftables evaluates a chain first-match. For a packet A -> B:

ip saddr 172.18.0.2 accept                               <- A's egress allow (default SG: all egress)
ip saddr 172.18.0.2 drop comment "default-deny egress"
ip daddr 172.18.0.3 drop comment "default-deny ingress"  <- B's ingress deny, never reached

That is the ruleset captured by the job at failure. Whenever A was emitted before B, A's egress accept ended evaluation and the packet reached B even though B's security group allows no ingress. Instances are keyed by random instance id, so the order, and the outcome, was a coin flip: the in-job 3x retry fails identically within a run because the ids and order are fixed once launched.

The same structure was mirrored into the bridge-family L2 table and applies across subnets too.

Fix

On AWS a packet must be permitted by the sender's egress rules and the receiver's ingress rules. Both tables are now rendered with two base chains on the forward hook:

  • egress (inet priority -6 / bridge -301): stateful accept, egress NACL denies, each instance's egress allows + default-deny egress
  • ingress (inet -5 / bridge -300): stateful accept, ingress NACL denies, each instance's ingress allows + default-deny ingress

An accept ends only its own base chain; the packet still traverses the other, where a drop is final. So the gates are independent and instance order no longer matters. Both tables share one renderer (render_table / render_chain), so they cannot drift.

Surfaces

  • Docs: website/content/docs/services/ec2.md states the egress-and-ingress semantics.
  • ec2_sg_enforcement_real.rs: comment and failure message updated (no assertion change; it still looks for ip daddr <ip> drop in inet fakecloud_ec2).
  • No API, SDK, conformance or count change. The k8s backend (NetworkPolicies) is untouched.

Test plan

  • New egress_and_ingress_are_independent_gates_whatever_the_instance_order: a small evaluator with nftables base-chain semantics (first match per chain, drop final, accept ends only its chain) runs against both render_ruleset and render_bridge_ruleset. Under the default security group A -> B and B -> A are dropped in both instance orders; once B allows ingress from A, A -> B passes and B -> A is still dropped. Against the previous single-chain renderer it fails with A -> B must be dropped.
  • fakecloud-ec2 lib tests 263/263 (existing ruleset text tests unchanged and passing); clippy -D warnings and fmt clean; ec2_sg_enforcement_real compiles.
  • The real packet-drop test needs Linux + nft + CAP_NET_ADMIN; this PR's EC2 SG enforcement (privileged) run is the end-to-end check.

Summary by cubic

Fixes a real EC2 security-group bypass that caused EC2 SG enforcement (privileged) to fail intermittently on main and on unrelated PRs. Egress and ingress rules used to share one nftables forward chain, so an egress accept could end evaluation before a later ingress default-deny; they are now independent base chains, and a packet must pass both, as on AWS.

Bug Fixes

  • Applies the same two-chain structure to inet fakecloud_ec2 and bridge fakecloud_ec2_l2 through one shared renderer, so L2 enforcement cannot drift.
  • Adds a unit test that checks both instance orders and proves A -> B is dropped under the default security group and allowed once B permits ingress from A.
  • Updates the EC2 docs to describe the independent egress and ingress gates.
  • No API, SDK, conformance, or count changes; the k8s NetworkPolicy backend is untouched.

Testing

  • Existing ruleset text tests pass; clippy and fmt are clean.
  • The privileged end-to-end packet-drop test still covers real nft enforcement.

Written for commit 8f539d4. Summary will update on new commits.

Review in cubic

SG enforcement rendered every instance's rules into one nftables `forward`
chain: an instance's egress allows and default-deny, then the next
instance's ingress allows and default-deny. nftables evaluates a chain
first-match, so for a packet A -> B, A's egress `ip saddr A accept` (the
default security group allows all egress) ended evaluation before B's
ingress `ip daddr B drop` was reached whenever A was emitted before B.
Instances are keyed by random instance id, so traffic that B's security
group denies got through about half the time.

That is the cause of the `EC2 SG enforcement (privileged)` job failing
intermittently on main and on unrelated PRs: the ruleset captured at
failure shows `ip saddr 172.18.0.2 accept` ahead of
`ip daddr 172.18.0.3 drop comment "default-deny ingress"`.

On AWS both gates must permit a packet. Render them as two base chains on
the forward hook, `egress` then `ingress`: an `accept` ends only its own
chain and the packet still traverses the other, where a drop is final. NACL
denies go in the chain for their direction. The bridge-family L2 table gets
the same structure through a shared renderer.

Test: a ruleset evaluator with nftables base-chain semantics checks that
A -> B is dropped under the default security group in both instance orders,
and allowed once B permits ingress from A, for both tables. It fails against
the single-chain renderer.
@vieiralucas
vieiralucas merged commit 610fdb0 into main Sep 14, 2026
158 checks passed
@vieiralucas
vieiralucas deleted the fix/ec2-sg-independent-gates branch September 14, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant