Support switching "target_backend" on HAProxy

[theboringstuff]

Description

Currently KubeMarine supports switching between "nginx" and "envoy" on "services.loadbalancer.target_backend" cluster.yaml property, however, actual change happens not in HAProxy configuration, but in nginx/envoy configuration - they move hostPorts 20080/20443 between each other to determine who will be the target backend. This causes problems, since we want to be able to switch between nginx/envoy quickly and with minimal impact (and probably do it multiple times in both direction). Redeploying plugins is too impacting in this case. Thus, we need to implement this switching through HAProxy re-configuration, swapping between ports 20080/20443 (nginx) and 21080/21443 (envoy).

Solution

Describe solution. List all changes that you made during implementation.

• Added new fields envoy_http and envoy_https to cluster.yaml section services.loadbalancer.target_ports with default values 21080 and 21443. These are used by envoy plugin as hostPorts, similar to nginx 20080/20443 ports.
  • In all-in-one and similar schemas the default is 80/443, same as nginx. It means that if user enabled both envoy and nginx in such schema, he also should reconfigure ports appropriately so that envoy/nginx do not bind same hostPorts.
• target_backend is now used by HAProxy configuration to determine which pair of ports (nginx 20080/20443 or envoy 21080/21443) to use as backend
• Updated docs/tests and check_iaas procedure accordingly

How to apply

• to update previously installed Envoy to use ports 21080/21443, run install procedure task deploy.plugins

Test Cases

1. Install cluster without specifying services.loadbalancer target_backend and target_ports, with both ingress-nginx and envoy-gateway plugins enabled (do not provide custom ports configuration in cluster.yaml)
2. Go to the load balancer IP and make sure you get 404 response from nginx curl -kv https://$LB_IP
3. Now, set cluster.yaml services.loadbalancer.target_backend to envoy and run install task deploy.loadbalancer.haproxy.configure to reconfigure HAProxy specifically
4. Go to the load balancer IP again and make sure you again get 404 response, but this time not from nginx, but from envoy curl -kv https://$LB_IP

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• There is no breaking changes, or migration patch is provided
• Integration CI passed
• Unit tests. If Yes list of new/changed tests with brief description
• There is no merge conflicts

[alexarefev]

Do we need to check if the ingress and envoy target ports are intersecting?

[theboringstuff]

Do we need to check if the ingress and envoy target ports are intersecting?

If both nginx and envoy are installed, then specifying same ports for both of them definitely would be strange and could lead to issues. I will add some check which verifies that if both are installed, then target ports should be different

UPD: added check