One of the issues infra teams often have when monitoring Kafka cluster is around comparable measures of latency/throughput.
Latency and other performance metrics from Kafka can be heavily influenced by the Kafka client applications producing and consuming from the cluster. The quality and type of client application can vary considerably from one cluster to another and there are many ways an application can be mis-configured/written which can effect these metrics. This makes it hard to infra teams to get a clear signal of the client-view of a cluster and also makes it difficult to compare the performance of one cluster to another.
What is required is a way to objectively (as possible) compare performance and for that you need an independent data pipeline, controlled by the infra team. A canary in the coal mine. At its most basic, this would be a producer and consumer that would send standardised messages at a set rate to a topic that was configured to have a partition on each broker. This would provide you with per-broker client side metrics that you could trust and give early warning of issues.
This could be useful for the StreamsHub Console in that it would provide a standardised client-side view of cluster health across a multi-cluster setup. It would also provide per broker connectivity and client performance metrics.
There have been attempts at providing such a tool in the past, for example the Strimzi Canary and canary-java. We could look at potentially using these as a basis for a StreamsHub based tool or start from scratch. The key features would be:
- Standardised message payloads
- Configurable message rates
- Canary topics with partitions on each broker (this will be a particular challenge to maintain when the cluster scales)
One of the issues infra teams often have when monitoring Kafka cluster is around comparable measures of latency/throughput.
Latency and other performance metrics from Kafka can be heavily influenced by the Kafka client applications producing and consuming from the cluster. The quality and type of client application can vary considerably from one cluster to another and there are many ways an application can be mis-configured/written which can effect these metrics. This makes it hard to infra teams to get a clear signal of the client-view of a cluster and also makes it difficult to compare the performance of one cluster to another.
What is required is a way to objectively (as possible) compare performance and for that you need an independent data pipeline, controlled by the infra team. A canary in the coal mine. At its most basic, this would be a producer and consumer that would send standardised messages at a set rate to a topic that was configured to have a partition on each broker. This would provide you with per-broker client side metrics that you could trust and give early warning of issues.
This could be useful for the StreamsHub Console in that it would provide a standardised client-side view of cluster health across a multi-cluster setup. It would also provide per broker connectivity and client performance metrics.
There have been attempts at providing such a tool in the past, for example the Strimzi Canary and canary-java. We could look at potentially using these as a basis for a StreamsHub based tool or start from scratch. The key features would be: