Time lag softcaps at 30 minutes

[Cenness]

Here how klag_consumer_lag_ms > 0 looks like

Here is graph from naive exporter that substracts timestamps:

klag:
version 0.1.9
KAFKA_HEALTH_CHECK_INTERVAL_MS : 30000
KAFKA_REQUEST_TIMEOUT_MS : 10000
METRICS_INTERVAL_MS : 30000

Another screenshot from klag, showing rebound:

[themoah]

hi @Cenness , that's valid bug.
i'll try to tackle it in upcoming days.

[themoah]

Hi @Cenness would be interested in reviewing #31 ?
I've tested the fix locally it fixed for me.

[Cenness]

Built image on that branch, got very surprising results:

Same axis:

Timestamps for first group - 2026-05-27T05:29:42.823Z for consumergroup offset, 2026-05-27T08:27:30.018Z - high watermark

Klag also doesn't see third group with same consumer behaviour despite having no filters. Each group is on separate topic.

What's also interesting is that, according to your interpolation, the first group is lagging behind less than the second group, despite what naive metrics show.

[themoah]

would you mind sharing kafka cluster version and how these consumers run? Are they long running?
Also, what is naive ?
It's bit strange that klag doesn't see the third group

[Cenness]

kafka is 3.9.0

how these consumers run? Are they long running?

They are essentially a shell scripts with a trigger outside of kafka. They run as long as there are messages to process and exit after they are done. Could be a second to a few minutes, median is single digit seconds.

Also, what is naive ?

It is actually a klag-exporter in "message" mode - https://github.com/softwaremill/klag-exporter/blob/62d422458f3154925b1d520de3e79d141025d720/README.md?plain=1#L67

bit strange that klag doesn't see the third group

Maybe it's related to delay between collecting enought data to calculate interpolation and starting point where last offset is? Here an illustration of similar thing:

As you can see there is a delay between naive and yours metrics appearance. Steps are exactly when new messages were published to the topic