docs(telemetry): add docs/telemetry.md — metrics reference and backend setup guide

[mpecan]

Summary

Create docs/telemetry.md — a comprehensive reference for the OTel metrics exporter introduced in #85.

Motivation

The README currently has no telemetry section. Users cannot discover the feature, understand what is collected, or configure it without reading the source code. This violates tokf's transparency principle.

Contents

docs/telemetry.md should cover:

1. Quick-start

export TOKF_TELEMETRY_ENABLED=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
tokf run cargo build

2. Configuration reference

Variable	Default	Description
TOKF_TELEMETRY_ENABLED	false	Set to true, 1, or yes to enable
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4318	OTLP collector endpoint
OTEL_EXPORTER_OTLP_PROTOCOL	http	http or grpc (grpc requires --features otel-grpc)
OTEL_EXPORTER_OTLP_HEADERS	(none)	Comma-separated key=value headers
OTEL_RESOURCE_ATTRIBUTES	(none)	OTel resource attributes; service.name extracted
TOKF_OTEL_PIPELINE	(none)	Pipeline label added to every metric

Config file: ~/.config/tokf/config.toml ([telemetry] section).

3. Metrics reference

Metric	Type	Unit	Description
gen_ai.client.token.usage	Histogram	{token}	Token usage per invocation (split by gen_ai.token.type: input/output)
tokf.filter.input_lines	Counter	{line}	Lines before filtering
tokf.filter.output_lines	Counter	{line}	Lines after filtering
tokf.filter.lines_removed	Counter	{line}	Lines removed
tokf.compression.ratio	Gauge	1	output/input line ratio
tokf.tokens.saved	Counter	{token}	Estimated tokens saved
tokf.filter.duration	Histogram	s	Filter wall-clock time
tokf.filter.invocations	Counter	{invocation}	Total filter invocations

Token counts are estimates (bytes / 4) — not exact tokenizer output.

4. Attributes

Every metric carries: tokf.filter.name, tokf.command, tokf.exit_code, tokf.pipeline, service.name, service.version.

5. Guarantees

• Export is best-effort: tokf waits at most 200 ms for the OTLP flush (see ADR-0001).
• No data is lost: every invocation is recorded to SQLite before the OTel path. A future tokf telemetry sync command (see docs(telemetry): add docs/telemetry.md — metrics reference and backend setup guide #88) will replay unsynced events.

6. Backend setup examples

Docker Compose snippets for: OTel Collector → Prometheus, Grafana, Jaeger, Datadog, Honeycomb.

7. Troubleshooting

Common problems: endpoint unreachable, missing otel-grpc feature, gRPC/HTTP mismatch, header auth format.

Acceptance Criteria

• docs/telemetry.md created with all sections above
• README.md gets a short "Telemetry" section linking to docs/telemetry.md
• --otel-export flag documented in the CLI flags table in README