feat: add promtool, logcli and tempo-cli for querying the observability stack

[venkatamutyala]

toolbox covered argocd and bao but nothing for metrics, logs, traces or alerts — anything observability-shaped meant opening a browser. This adds the three native CLIs.

Requires GlueOps/k8s-monitoring-helm#353 (Grafana [auth.jwt]).

Why this needs no proxy

All three tools accept arbitrary headers, so each wrapper sends two:

header	read by
Authorization: Bearer <jwt>	oauth2-proxy at the edge
X-JWT-Assertion: <jwt>	Grafana's [auth.jwt]

Two are needed because the edge consumes Authorization for itself and Traefik's forwardauth deletes it before Grafana sees it. X-JWT-Assertion is in no authResponseHeaders list and passes through untouched.

So this needs neither a loopback proxy (as bao does) nor a bearer-preserving middleware (as argocd does). Queries go through Grafana's datasource proxy, so there's one edge host and one credential for everything.

Alerts come free

promtool reaches /api/v1/alerts and /api/v1/rules on the same datasource proxy, so alert state and rule listings need no extra tool. Verified: 4 alerts (2 firing, 2 pending), 34 rule groups / 239 rules.

Sharp edges the tempo-cli wrapper hides

1. Headers are Key=Value, not Key: Value
2. search takes a bare host + --path-prefix; trace-id takes a full URL and has no --path-prefix
3. --use-grpc is refused outright — headers would travel as gRPC metadata and never reach the edge

grafana-ds resolves datasource UIDs at run time, since Grafana generates them per cluster.

Backwards compatible — no toggle needed

On a cluster without [auth.jwt], these three get a 302 while argocd and bao keep working. Login is unaffected. This matches the existing precedent: toolbox-login already treats OpenBao login as non-fatal ("OpenBao: not logged in (argocd is unaffected)") because auth/jwt is equally new.

Verified from the built image, against a live cluster

grafana-ds prometheus|loki|tempo   →  prometheus / P8E80F9AEF21F6940 / de7lydl3hl9fkd
promtool  count(up)                →  {} => 110
promtool  count(ALERTS)            →  {} => 1
logcli    labels                   →  k8s_container_name, k8s_pod_name, service_name
tempo-cli query api search-tags    →  {"scopes":[{"name":"intrinsic",...
tempo-cli --use-grpc               →  refused, as intended

Known risks — documented, not blocked

The README gains a Known risks section recording that Grafana's datasource proxy is a full pass-through: it forwards POST/PUT/DELETE, and Grafana has no method-level control. So these credentials can write to the datasources — including Loki's ingestion endpoint, verified as reachable (Loki's own 422, not a Grafana block). Those writes are durable: all three datasources are backed by S3, so nothing is undone by restarting pods or rebuilding the cluster. There is also no per-user audit trail in Grafana OSS.

This was accepted knowingly. Token holders already have ArgoCD and OpenBao access. The AI-agent callout at the top of the README now says explicitly: do not delete or modify data — treat everything as read-only, and stop and ask if a task appears to need a destructive action.

If the convention ever needs enforcing, the point is a Traefik Method(GET) rule on /api/datasources/proxy/. Deliberately not done here.

Note on image size

The image grows to ~1.31 GB; the three binaries are ~260 MB (tempo-cli alone is 80 MB). Versions are pinned as build args. A COPY --from=grafana/tempo-cli could trim it if size becomes a concern.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrpnuCduw2KivngVRQmnG7