Install Meridio on kind cluster: proxy-load-balancer-a1-xxx stuck failed Readiness probe

[dezenxi]

Hi,
I'm following up instructions https://meridio.nordix.org/docs/demo/multus-kind-ovs/#installation
to install Meridio on kind cluster.

However, some pods got not-ready state as below
ipam-trench-a-0 1/1 Running 0 6m41s
meridio-operator-548db54687-njlcw 1/1 Running 0 7m35s
nsp-trench-a-0 1/1 Running 0 6m41s
proxy-load-balancer-a1-9mfq6 0/1 Running 0 6m40s
proxy-load-balancer-a1-p5mrg 0/1 Running 0 6m39s
stateless-lb-frontend-attr-1-7b6d9d56f7-6gtd4 0/2 Init:CrashLoopBackOff 6 (35s ago) 6m41s
stateless-lb-frontend-attr-1-7b6d9d56f7-9x4sv 0/2 Init:CrashLoopBackOff 6 (42s ago) 6m41s

k describe pod proxy-load-balancer-a1-9mfq6 -n red
Warning Unhealthy 17s (x29 over 4m17s) kubelet Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

k describe pod stateless-lb-frontend-attr-1-7b6d9d56f7-6gtd4 -n red
Args:
-c
sysctl -w net.ipv4.conf.all.forwarding=1 ; sysctl -w net.ipv4.fib_multipath_hash_policy=1 ; sysctl -w net.ipv4.conf.all.rp_filter=0 ; sysctl -w net.ipv4.conf.default.rp_filter=0 ; sysctl -w net.ipv6.conf.all.forwarding=1 ; sysctl -w net.ipv6.fib_multipath_hash_policy=1
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 13 Jun 2023 23:35:43 +1000
Finished: Tue, 13 Jun 2023 23:35:43 +1000
Ready: False
Warning BackOff 11m (x4 over 12m) kubelet Back-off restarting failed container

I followed strictly the step in https://meridio.nordix.org/docs/demo/multus-kind-ovs/#installation
Still don't know why pod of proxy and frond-end are not up/running
I need your help to install Meridio on kind cluster.
Is there any prerequisite when installing Meridio on kind cluster?

Regards,
Duong

[LionelJouin]

I haven't tried these instructions recently. You can try this to setup your KinD cluster (Spire, NSM, GW/TG):

make -s -C ./docs/demo/scripts/kind

And then you can install Meridio like this:

helm install meridio-crds https://artifactory.nordix.org/artifactory/cloud-native/meridio/Meridio-CRDs-v1.0.6.tgz --create-namespace
helm install meridio https://artifactory.nordix.org/artifactory/cloud-native/meridio/Meridio-v1.0.6.tgz --create-namespace

And Multus you can install in the same way

kubectl apply -f https://raw.githubusercontent.com/Nordix/xcluster/master/ovl/multus/multus-install.yaml
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/whereabouts/master/doc/crds/daemonset-install.yaml
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/whereabouts/master/doc/crds/whereabouts.cni.cncf.io_ippools.yaml
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/whereabouts/master/doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml

[dezenxi]

Hi @LionelJouin ,

Thanks for responses,
I followed your instructions, but still failed at step
kubectl apply -f docs/demo/multus-meridio.yaml -n red

The pods proxy-load-balancer still not ready

Readiness probe failed: service unhealthy (responded with "NOT_SERVING")

Pod stateless-lb-frontend-attr-1 were stuck at INIT/crash state as before.

Regards,
Duong

[LionelJouin]

In that case the first thing to check is if the interface in the attractor instance exist and what is the bird status.

It seems similar to this tutorial case: https://meridio.nordix.org/training/troubleshooting-ctf/scenario-2

The logs of the frontend container could be also useful

[dezenxi]

Hi @LionelJouin ,

Do I need to create NetworkAttachmentDefinition meridio-nad like in ?
https://meridio.nordix.org/docs/demo/multus-kind-ovs/#installation

Regards,
Duong

[LionelJouin]

If you use a network-attachment type of interface in your attractor, then yes, you will need to create a NAD (NetworkAttachmentDefinition)

[dezenxi]

Hi @LionelJouin ,
I updated multus-meridio.yaml/Attractor to use type nsm-vlan

    name: ext-vlan0
    ipv4-prefix: 169.254.100.0/24
    ipv6-prefix: 100:100::/64
    type: nsm-vlan
    nsm-vlan:
      vlan-id: 100
      base-interface: eth0

Now, all pods are up and running

NAME                                            READY   STATUS    RESTARTS   AGE
ipam-trench-a-0                                 1/1     Running   0          36m
meridio-operator-598994c599-6mdjp               1/1     Running   0          46m
nse-vlan-attr-1-b4659bfb4-k672g                 1/1     Running   0          36m
nsp-trench-a-0                                  1/1     Running   0          36m
proxy-load-balancer-a1-lb5tx                    1/1     Running   0          36m
proxy-load-balancer-a1-n57vk                    1/1     Running   0          36m
stateless-lb-frontend-attr-1-5d4d576cc6-pmt78   3/3     Running   0          36m
stateless-lb-frontend-attr-1-5d4d576cc6-qqb97   3/3     Running   0          36m

I'll try to use network-attachment type later.
Thank you very much.

Regards,
Duong

[dezenxi]

Hi @LionelJouin ,
Do you think Frontend should check external interface before starting bird (bgp/bfd)?
I can see in case of static config that frontend reach ready state although the interface missing.
So, explicitly check and printout interface missing would help reduce a lot of time of troubleshooting, such as
"Missing interface XYZ , check attractor config or CNI...."

Regards,
Duong

Regards,
Duong