A bug when installing zookeeper in K8s using pulsar-helm

[luky116]

Phenomenon: I used Pulsar helm to install the Pulsar cluster, and found that occasionally some zookeeper Pods could not start normally, as shown below:

I checked the logs of this zk pod and found the following problems:

2024-03-19T08:24:13,855+0000 [ListenerHandler-pub-bjzdt-test-pulsar-cluster-zookeeper-1.pub-bjzdt-test-pulsar-cluster-zookeeper.test-pulsar-ns50.svc.cluster.local/11.53.72.241:3888] ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address pub-bjzdt-test-pulsar-cluster-zookeeper-1.pub-bjzdt-test-pulsar-cluster-zookeeper.test-pulsar-ns50.svc.cluster.local/11.53.72.241:3888
java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:555) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:544) ~[?:?]
	at sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:648) ~[?:?]
	at java.net.ServerSocket.bind(ServerSocket.java:388) ~[?:?]
	at java.net.ServerSocket.bind(ServerSocket.java:342) ~[?:?]
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.createNewServerSocket(QuorumCnxManager.java:1141) ~[org.apache.zookeeper-zookeeper-3.8.3.jar:3.8.3]
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1070) ~[org.apache.zookeeper-zookeeper-3.8.3.jar:3.8.3]
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039) ~[org.apache.zookeeper-zookeeper-3.8.3.jar:3.8.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]

I added some logs in the zookeeper-statefulset.yaml template:

It turns out that the value in zk's myid file is not the number of the current zk node:
The value of myid file is 2:

But judging from the zookeeper.conf file, myid should be 3:

I suspect it is a problem with the bin/generate-zookeeper-config.sh script. I tried running this script locally and the results were as expected.

I would like to ask everyone, how should I deal with this problem? Thank you very much

[lhotari]

Please share the Pulsar Helm Chart version that you are using and more details about your target Kubernetes environment.

Just a reminder that older Pulsar Helm Chart versions aren't supported. If you aren't running on the most recent version 3.3.0 of the Pulsar Helm Chart, please do an upgrade.

[luky116]

Please share the Pulsar Helm Chart version that you are using and more details about your target Kubernetes environment.

Just a reminder that older Pulsar Helm Chart versions aren't supported. If you aren't running on the most recent version 3.3.0 of the Pulsar Helm Chart, please do an upgrade.

3.3.0

[luky116]

Please share the Pulsar Helm Chart version that you are using and more details about your target Kubernetes environment.

Just a reminder that older Pulsar Helm Chart versions aren't supported. If you aren't running on the most recent version 3.3.0 of the Pulsar Helm Chart, please do an upgrade.

I seem to have found the cause of the problem. I added a log here:

It was found that the myid file already existed before the conf file was initialized:

However, the generate-zookeeper-config.sh script will determine that if the myid file exists, it will not change this file:

So, I added a line of command here to delete the myid file and the problem was solved:

If necessary, I can submit a PR to you to solve this problem.

Thank you very much

[lhotari]

If necessary, I can submit a PR to you to solve this problem.

I don't think that a PR is necessary. It's intentional that this file isn't modified. This file is present because of data consistency reasons, to prevent accidential usage on a different server. There seems to be a special case in your Kubernetes environment that this happened in the first place. What Kubernetes environment are you using?

[luky116]

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.13", GitCommit:"49433308be5b958856b6949df02b716e0a7cf0a3", GitTreeState:"clean", BuildDate:"2023-04-12T12:15:50Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.9", GitCommit:"d1483fdf7a0578c83523bc1e2212a606a44fd71d", GitTreeState:"clean", BuildDate:"2023-09-13T11:25:26Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.26) exceeds the supported minor version skew of +/-1

[luky116]

This K8s is my test environment. I have helm uninstall and helm install many times. The data is placed on the local disk of the node, and the path remains unchanged. This myid file may be left over from my last helm install

[lhotari]

This K8s is my test environment. I have helm uninstal and helm install many times. The data is placed on the local disk of the node, and the path remains unchanged. This myid file may be left over from my last helm install

To cleanup completely, you would have to delete the PVCs separately after uninstalling the Pulsar helm chart.

[luky116]

i see, thank you very much