Relevant telegraf.conf
[agent]
interval = "3s"
flush_interval = "3s"
omit_hostname = true
[[inputs.exec]]
commands = ["/path/to/emit.sh"]
data_format = "influx"
[[outputs.parquet]]
directory = "/path/to/out"
`emit.sh` — emits two fields for the first two calls, three thereafter:
#!/bin/sh
n=$(cat counter 2>/dev/null || echo 0); n=$((n+1)); echo $n > counter
if [ "$n" -le 2 ]; then echo "demo a=1i,b=2i"; else echo "demo a=1i,b=2i,c=3i"; fi
Logs from Telegraf
System info
Telegraf 1.39.2, Linux 6.18.40.
Docker
n/a
Steps to reproduce
outputs.parquet builds the Arrow schema from the metrics in the first flush and then holds it for the lifetime of the file. Fields that appear later are dropped. The README documents this:
"if additional fields are present after the first metric flush those fields are
omitted"
and, more severely:
"If a metric schema does not match the schema in the file it will be dropped."
In reality, a metric is quietly stripped of the fields after the first flush. An operator has no signal that the file they are accumulating is lossy until they query it and find a column missing, which may be weeks later.
This is easy to hit in ordinary operation, not just in contrived cases:
- a sensor that is absent at boot and appears once its driver loads (adding a hardware sensor to a host does exactly this)
- an input whose fields depend on state — GPU metrics that only appear once a process attaches, ECC counters that only appear post-error
- a plugin added or reconfigured while the current file is still open
- any input where a value is
N/A on the first scrape and numeric afterwards, since SetIfUsed-style helpers omit the field entirely in the former case
Expected behavior
One of, in rough order of preference:
- The file's schema is widened to accommodate the new field, with earlier rows null for it.
- A new file is started when the schema changes, so no data is lost.
- At minimum, a
W!-level log line naming the dropped field(s) and metric, once per distinct schema mismatch, so the loss is at least observable.
Actual behavior
Field c was present in 6 of the 8 emitted metrics and is absent from the output entirely:
$ duckdb -c "describe select * from read_parquet('out/*.parquet')"
┌───────────┬─────────┐
│ a │ bigint │
│ b │ bigint │
│ timestamp│ bigint │
└───────────┴─────────┘
$ duckdb -c "select * from read_parquet('out/*.parquet')"
a │ b │ timestamp
────┼─────┼─────────────────────
1 │ 2 │ 1785970881000000000
1 │ 2 │ 1785970884000000000
1 │ 2 │ 1785970887000000000 ← c=3 emitted from here on
1 │ 2 │ 1785970890000000000
1 │ 2 │ 1785970893000000000
1 │ 2 │ 1785970896000000000
1 │ 2 │ 1785970899000000000
1 │ 2 │ 1785970902000000000All 8 rows are present; the column is not. Telegraf logged nothing.
Additional info
While evaluating this plugin for a host-telemetry pipeline, the other blocker was that there is no partitioning. Output files are named <measurement>-<date>-<epoch>.parquet in a single flat directory, so there is no way to lay out host=…/date=… (or any tag-based) directory structure of the kind every Parquet query engine uses for partition pruning — DuckDB, DataFusion, Spark and Athena all read Hive-style partitioning natively, and without it a query over a year of data must open every file.
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.39.2, Linux 6.18.40.
Docker
n/a
Steps to reproduce
outputs.parquetbuilds the Arrow schema from the metrics in the first flush and then holds it for the lifetime of the file. Fields that appear later are dropped. The README documents this:and, more severely:
In reality, a metric is quietly stripped of the fields after the first flush. An operator has no signal that the file they are accumulating is lossy until they query it and find a column missing, which may be weeks later.
This is easy to hit in ordinary operation, not just in contrived cases:
N/Aon the first scrape and numeric afterwards, sinceSetIfUsed-style helpers omit the field entirely in the former caseExpected behavior
One of, in rough order of preference:
W!-level log line naming the dropped field(s) and metric, once per distinct schema mismatch, so the loss is at least observable.Actual behavior
Field
cwas present in 6 of the 8 emitted metrics and is absent from the output entirely:All 8 rows are present; the column is not. Telegraf logged nothing.
Additional info
While evaluating this plugin for a host-telemetry pipeline, the other blocker was that there is no partitioning. Output files are named
<measurement>-<date>-<epoch>.parquetin a single flat directory, so there is no way to lay outhost=…/date=…(or any tag-based) directory structure of the kind every Parquet query engine uses for partition pruning — DuckDB, DataFusion, Spark and Athena all read Hive-style partitioning natively, and without it a query over a year of data must open every file.