Cilium policy refactor

[mskwierczynski982]

Description

1. Reverted fix: make default network policy compatible with Cilium dataplane #216 changes to NetworkPolicy - it was taken from microsoft migration to cilium guide but it is naive example in their docs - rule matching all namespaces per number of ports translates to huge amount of entries in BPF map
2. Implemented logic to deploy Cilium Network Policy only on clusters with Cilium CRD - classic Network Policy is reverted to last good state
3. Implemented new Cilium Network Policy

Justification of changes:

The problem

Cilium enforces policy through a per-pod BPF map
(cilium_policy_v2_<endpoint-id>, 16,384 entries) keyed on
(identity, port, protocol, direction). A selector matching every pod in the
cluster costs one entry per identity, per port.

The breaking line is namespaceSelector: {} in the outside-world egress rule
(network-policy.yaml:105):

    - to: #outside world - sentry, optimizely, etc
        - ipBlock:
            cidr: 0.0.0.0/0
        - namespaceSelector: {}      # <-- compiles to `namespace Exists`
        - podSelector: {}
      ports:
        - port: 443
        ...12 port entries, one being 11000-11999

Peers in a rule are OR'd and ports: applies to all of them, so that
cluster-wide peer inherits all 12 port entries — which expand to 17 map
tokens, because 11000-11999 is stored as 6 LPM prefixes, not 1.

644 identities x 17 tokens = ~10,900 entries from a single rule.

This peer was added for the Cilium migration, following the Microsoft
guidance. That advice is correct for the port-less example in the doc, and
fatal when applied to a rule carrying 12 ports — the doc never states that
cost.

Measured on aks-mgc-b-lab-aks01-weu-a99

Cluster pod identities	644
breaking-evtrek pod (endpoint 441)	14,796 / 16,384 — 90% full
Entries per cluster identity	17
Break-even	~910 identities
Policies deployed	69 across 6 namespaces

At 300 apps x 5 namespaces (1,500 identities) each selected pod's map reaches ~27,000 entries —
165% of the limit.

The fix

The single wide rule is split by destination type:

# in-cluster: narrow peer, NO ports -> 1 entry per identity
- toEndpoints:
    - matchLabels:
        k8s:io.cilium.k8s.namespace.labels.environment: <env>

# external: full port list, but the peer is ~2 identities
- toEntities: [world]
  toPorts:
    - ports: [443, 80, 8433, ... 11000-11999]

Peer	identities	x tokens	entries
Before: namespaceSelector: {} + ports	644	17	10,948
After: environment=<env>, port-less	~150	1	150
After: toEntities: world + ports	2	17	34

Two independent levers: narrowing the peer, and removing the port list from it.
The second is the larger win — dropping ports: takes the multiplier to 1
regardless of how many identities the peer matches.

Result: ~12,400 -> ~600 entries per pod (20x), and no cluster-wide selector
remains, so it no longer grows with cluster size.

Note the format change is incidental. The same fix works in a plain
NetworkPolicy. Only toEntities: [host, remote-node] genuinely requires CNP,
because Kubernetes NetworkPolicy cannot select a node.

Net effect on pod access

Removed — traffic the old policy allowed and the new one does not:

What	Assessment
Cross-environment pod egress on 443/80/8433/8000/8200/9243/5432/6380/1433/11000-11999	Never intended; incidental to ipBlock: 0.0.0.0/0 matching pod IPs under NPM
Cross-namespace ingress on 443/80/8443/8000 from unrelated namespaces	Ingress twin of the same pattern

Added — required for parity; without these Cilium is stricter than NPM:

What	Why
toEntities: [host, remote-node]	Node IPs resolve to node identities and are never matched by an ipBlock, even inside the VNet CIDR. NPM allowed pod -> own-node egress implicitly.
fromEntities: [host, remote-node]	LoadBalancer/NodePort on the default externalTrafficPolicy: Cluster SNATs the client to a node IP. Without this all LB ingress drops.
kube-apiserver	Its own reserved identity on managed AKS
Same-environment egress: 4 ports -> all ports	Ingress already allowed these peers on all ports; this makes egress symmetric. A real widening within the environment boundary.

Unchanged: own-namespace, same-environment ingress, DNS, all external/PaaS
egress, prometheus scrape.

Summary: traffic within an environment is unchanged or slightly wider;
traffic across environments on those ports is now denied — which is what the
policy always intended, but NPM could not express.

Chart

Select the chart that you are modifying:

• core
• dotnet-core
• cron-job
• job
• app-reverse-proxy
• pact-broker
• ado-build-agents
• ado-agent-cleaner
• event-worker

Checklist

• Description provided
• Linked issue
• Chart version bumped
• README.md updated with any new values or changes
• Updated template tests in ${CHART}/tests/*.py