Do we need a new Consumer Group lag exporter?

[tomncooper]

Currently, Strimzi uses the kafka_exporter project as a way to expose Consumer Group lag metrics to Prometheus.

The kafka_exporter project

The kafka_exporter is widely adopted, with around 2,500 stars and 680 forks, making it something of a de facto standard for getting Kafka metrics into Prometheus. The community is clearly active in terms of contributions, with people regularly opening PRs for new features, bug fixes, and dependency updates. The problem isn't the supply of contributions, it's the review bandwidth. The entire project runs through a single maintainer who merges PRs in sporadic bursts a few times a year, then goes quiet for months. The bus factor is effectively one.

This bottleneck shows up across the board. There are 54 open PRs, some dating back to 2018, and 76 open issues, over half of which have never received a single comment. The release cadence has been steadily slowing, from four releases in 2021 down to one per year recently, and the last release (v1.9.0) was back in February 2025, now 17 months ago. Several CVE-related dependency bumps have been merged to master since then but never shipped in a release, which means anyone pulling tagged releases rather than master is running with known vulnerabilities.

It's worth noting the project isn't abandoned. The maintainer does still show up and merge things periodically, and the Apache 2.0 license makes forking straightforward. But for a downstream consumer, the practical reality is that getting changes upstreamed and released is unpredictable, and security fixes can sit undelivered for extended periods.

Unmerged Enhancements

There are many enhancements, security and bug fixes that would improve the downstream users' experience of the kafka_exporter, but remain unmerged. For example:

• Security and CVE fixes

  • #524 (1 week old) - Dependency updates to fix HIGH/CRITICAL CVEs found by Trivy scan
  • #504 (20 months old) - Go crossbuild version update to resolve 16 stdlib CVEs

• Correctness bugs

  • #520 (2 months old) - Unavailable consumer offsets (-1) incorrectly summed into aggregate metric, producing wrong values
  • #402 (2 years 10 months old) - Data race under concurrent scrapes, confirmed with Go race detector

• Performance

  • #516 (5 months old) - Batch offset fetching to reduce API calls to Kafka brokers
  • #125 (6 years 9 months old) - Flags to suppress per-partition metrics in large environments where cardinality explodes

• Features with operational value

  • #525 (opened today by Strimzi maintainer scholzj) - OAuth token file support for rotation without restart
  • #96 (7 years 3 months old, 9 reviews) - Min insync replicas metric for ISR availability alerting
  • #195 (5 years 7 months old) - Consumer lag reported as time, not just offsets
  • #500 (10 months old) - Allow setting JSON format for logging output

We would like to enhance the StreamsHub Console with Consumer Group metrics including temporal lag metric such as those proposed in #195. However, given this PR hasn't been merged in over 5 years, we doubt that we will see such a feature (or any other significant improvements) any time soon.

We have to question whether this is a healthy project to base our Kafka infrastructure on.

Possible Solutions

Increase the Bus factor

We could attempt to gain committer status on the project and remove the single developer bottleneck. The problem here is that submitting PRs to fix bugs and add features, the usual way to gain committer status, will be difficult as the maintainer only approves in bursts of decreasing frequency. We could of course reach out privately to them and see if they would be willing to share the load.

Fork

We could Fork the project and attempt to bring the community with us, merging existing PRs and bug fixes and addressing comments. This obviously comes with the support burden of a largely unfamiliar codebase and is complicated by the fact that the original project is still sporadically active and so we risk splitting the community.

A new exporter project

We could create a new exporter, based on the modern upstream Apache Kafka Java client libraries. As these are actually part of the upstream Kafka project we gain much stronger compatibility with existing and new Kafka versions as well as security and performance fixes, compared to the Sarama client used by the kafka_exporter.

A project supported by StreamsHub would have multiple maintainers able to commit PRs and handle issues. Hopefully, over time this exporter could address the unmet needs of existing kafka_exporter users including Strimzi.

[ppatierno]

FYI, a similar issue was also opened within the Strimzi project strimzi/strimzi-kafka-operator#12922

[tomncooper]

I posted on that issue @ppatierno. My personal opinion is that we could do with a new Java based CG lag exporter hosted in an active community. In the short term getting more committers onto the kafka_exporter project or forking it is an option, but long-term I think it would be good to have something which uses the official clients and is actively maintained.

[tomncooper]

I also wonder if something like the KmetaDB prototype that @MikeEdgar created might provide another option, particularly for use with the Console.

[MikeEdgar]

I also wonder if something like the KmetaDB prototype that @MikeEdgar created might provide another option, particularly for use with the Console.

Something that is interesting about using kmetadb is that since it uses a SQL persistence layer, there are basically no build-time limits on what metrics you can choose to export. In other words, metrics could be dynamically defined with configuration using SQL.

[lfcastilhos]

It would be extremely valuable. As a kafka_exporter user, I have been facing significant challenges related to CVEs (security vulnerabilities). These are by far the most critical issues affecting us and require urgent attention.