telegraf wrongly assumes metric is prometheus histogram and breaks itself

[DaveWK]

Relevant telegraf.conf

[global_tags]
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = false
[[inputs.influxdb_listener]]
  service_address = "127.0.0.1:8086"

[[outputs.opentelemetry]]

Logs from Telegraf

Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64
Nov 30 15:07:38 telegraf[13175]: 2023-11-30T20:07:38Z W! [outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64

System info

telegraf 1.28.5

Docker

No response

Steps to reproduce

1. Send some influxdb metrics that include a count field ending in i, in example count=1i
2. Attempt to export via outputs.opentelemetry
3. Observe telegraf complain about the incorrect assumption it made with the warning message: "[outputs.opentelemetry] Failed to add point: unsupported histogram count value type int64"

Expected behavior

The metric should be converted to a format that is compatible with opentelemetry's specification and sent to the specified output.

Actual behavior

telegraf freaks out about a type assumption that it made on it's own and refuses to output via opentelemetry

Additional info

I think the main problem is that the logic for assuming a prometheus histogram is faulty. This also came up in:
#12431

I would suggest either:

The logic shouldn't be assuming it's a prometheus histogram when the count field is of type int64.

The value should be converted to a supported type automatically

It should be possible in the configuration file to explicitly define the conversion for values of incompatible types

[powersj]

Hi,

Thanks for the report.

telegraf freaks out about a type assumption that it made on it's own and refuses to output via opentelemetry

The input, influxdb listener, does not make any assumptions nor does it set any types on metrics. InfluxDB does not have the concept of types, so everything gets recorded as "untyped" in Telegraf, not "histogram".

That error message is coming from a call to "AddPoint", which is from the influx2otel library. I believe the error returned here in convertHistogramV1. This is only called if the determined type is histogram, which would mean that a) the type was not untyped, cleary wrong, and also ensuring that both the count and sum fields exist, which I am guessing is also not the case.

I think that is actually happening here is our iota's are not lined up between telegraf and the otel library. It looks like Telegraf's Untyped, 2, is the otel library's Histogram.

@jacobmarble is this something you could please chime in on from the otel library perspective?

Thanks!

[seankhliao]

I ran into what I think is the same issue in influxdata/influxdb-observability#312, though I was using the opentelemetry collector (which uses the same underlying influx2otel conversion).

I don't think it's from mismatched value types, I think it's because influx2otel tries to infer a type incorrectly?
In https://github.com/influxdata/influxdb-observability/blob/32fff60a01f9bb7baa8e4c0aeda62fe03b1a6fb9/influx2otel/metrics_telegraf_prometheus_v1.go#L52-L58 it assumes if a count field is present then it's a histogram, but the metrics we get don't work that way.

Example line of metrics we get:

accounts_index_startup,host_id=HkAi2J2QTC7tLW9Y5cQi9RmMhy3TNmqwyjWn2vnSqaG6 estimate_mem_bytes=0i,flush_should_evict_us=0i,count_in_mem=0i,count=118606460i,bg_waiting_percent=0.0037496250588446856,bg_throttling_wait_percent=0,slot_list_len=0i,ref_count=0i,slot_list_cached=0i,min_in_bin_mem=0i,max_in_bin_mem=0i,count_from_bins_mem=0i,median_from_bins_mem=0i,min_in_bin_disk=2956i,max_in_bin_disk=1046676i,count_from_bins_disk=118606260i,median_from_bins_disk=3300i,gets_from_mem=0i,get_mem_us=0i,gets_missing=0i,get_missing_us=0i,entries_from_mem=0i,entry_mem_us=0i,load_disk_found_count=0i,load_disk_found_us=0i,load_disk_missing_count=0i,load_disk_missing_us=0i,entries_missing=0i,entry_missing_us=0i,failed_to_evict=0i,updates_in_mem=0i,get_range_us=0i,inserts=49989013i,deletes=0i,active_threads=32i,items=0i,keys=0i,ms_per_age=10001i,buckets_scanned=31992i,flush_scan_us=246i,flush_update_us=0i,flush_grow_us=0i,flush_evict_us=0i,disk_index_resizes=17561i,disk_index_failed_resizes=5344i,disk_index_max_size=2361002i,disk_index_new_file_us=820346i,disk_index_resize_us=264091959i,disk_index_flush_file_us=85i,disk_index_index_file_size=10319620656i,disk_index_data_file_size=0i,disk_index_data_file_count=0i,disk_index_find_index_entry_mut_us=117379712i,disk_index_flush_mmap_us=13152766i,disk_data_resizes=0i,disk_data_max_size=0i,disk_data_new_file_us=0i,disk_data_resize_us=0i,disk_data_flush_file_us=0i,disk_data_flush_mmap_us=0i,flush_entries_updated_on_disk=0i,flush_entries_evicted_from_mem=0i 1732619845882886927

I think the inference should only assume histogram if all fields match.

[DaveWK]

Seems like you've entered my circle of heck. I still haven't been able to get to the bottom of how to import (what looks to be the same as my use case) solana's metrics via opentelemetry because of this somehow "not an inference" behavior that we're both observing.

[seankhliao]

😭 we both suffer from solana's metrics.
I got this working last week by building a patched opentelemetry collector instance.
I just skipped inference by changing this line to use Gauge instead of Untyped https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/e4b2bbf0f22a33946e968703e9b6917cddd85c3f/receiver/influxdbreceiver/receiver.go#L184

[DaveWK]

thanks! I got depressed after dealing with both influx + otel blaming each other for this "definitely not inference" behavior -- much appreciated for the hotfix.

[skartikey]

I opened influxdata/influxdb-observability#346 for the specific error here ("unsupported histogram count value type int64").

Cause: in the V1 schema, convertHistogramV1/convertSummaryV1 only accepted float64 for the count, sum, and bucket/quantile fields, but integer line protocol fields like count=1i decode as int64, so integer-valued histograms and summaries were rejected and dropped. The PR accepts int64/uint64 too.

This only fixes the value-type rejection, not the inference that @seankhliao and @DaveWK raised, where an untyped metric with a count field is assumed to be a histogram. That inference (and @seankhliao's Gauge workaround) is a separate change.