Skip to content

fix: fill in promtool's server argument, refuse the paths that cannot authenticate - #29

Merged
venkatamutyala merged 1 commit into
mainfrom
fix/promtool-datasource-url
Sep 5, 2026
Merged

venkatamutyala merged 1 commit into
mainfrom
fix/promtool-datasource-url

Conversation

@venkatamutyala

Copy link
Copy Markdown
Collaborator

Found while running the CLI-SSO sweep against a freshly rebuilt nonprod.jupiter.onglueops.rocks.

The bug

promtool is the only query wrapper that does not point itself at the cluster. logcli sets --addr and tempo-cli sets --path-prefix, both from grafana-ds — but bin/promtool only appended the two auth headers, so the bare form fails:

$ toolbox promtool query instant 'sum(up)'
promtool.real: error: required argument 'expr' not provided, try --help

The caller has to hand-build https://grafana.<domain>/api/datasources/proxy/uid/<uid>, which the README already told them they would not have to:

Queries go through Grafana’s datasource proxy rather than to Loki/Thanos/Tempo directly, so there is one edge host and one credential for everything. The wrappers resolve the datasource UID at run time (grafana-ds)

A second bug in the same branch

Chasing it turned up that the old wrapper applied --header to subcommands that do not accept it:

$ toolbox promtool debug pprof <server>
promtool.real: error: unknown long flag '--header', try --help

Only two subcommands take --header at all:

subcommand --header can reach the cluster
query instant yes yes
query range yes yes
query series no no
query labels no no
debug pprof / metrics / all no no

Without --header a request arrives at the edge carrying no credential and oauth2-proxy redirects it to the login page — so those paths cannot work here regardless of the flag error.

The change

  • Resolve the prometheus datasource UID and insert the <server> positional for query instant and query range.
  • Refuse query series, query labels and debug against this cluster with a message that says why, following the precedent of tempo-cli refusing --use-grpc.
  • Always pass an explicit http(s) server through untouched, so promtool still works against anything else.
  • --help and the local subcommands (check, push, test, tsdb) are unchanged.
  • README updated so the command table and the proxy section match the behaviour.

Verification

Built and run against a live cluster:

$ promtool query instant 'sum(up)'
{} => 110 @[1788626601.938]

$ promtool query instant 'count by (alertstate)(ALERTS)'
{alertstate="pending"} => 2 @[1788626604.322]
{alertstate="firing"} => 2 @[1788626604.322]

$ promtool query labels alertstate
toolbox: 'promtool query labels' takes no --header, so it cannot authenticate to this cluster. ...

$ promtool debug pprof
toolbox: 'promtool debug' takes no --header, so it cannot authenticate to this cluster. ...

Explicit server, query --help and check --help all still behave as before.

Not included

The logcli delete create guard discussed separately — deliberately left out, to be sorted later.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrpnuCduw2KivngVRQmnG7

… authenticate

`promtool` was the only query wrapper that did not point itself at the cluster.
`logcli` sets `--addr` and `tempo-cli` sets `--path-prefix` from `grafana-ds`,
but `bin/promtool` only appended the two auth headers, so the bare form failed:

    $ toolbox promtool query instant 'sum(up)'
    promtool.real: error: required argument 'expr' not provided

The caller had to hand-build `https://grafana.<domain>/api/datasources/proxy/uid/<uid>`,
which the README already said they would not have to.

Resolve the prometheus datasource UID and insert the `<server>` positional for
`query instant` and `query range`.

Those two are also the only subcommands that can work here at all: `query series`,
`query labels` and every `debug` subcommand take no `--header`, so the old wrapper
appended a flag they reject —

    $ toolbox promtool debug pprof <server>
    promtool.real: error: unknown long flag '--header'

— and even without the flag their requests would arrive at the edge carrying no
credential and be redirected to the login page. Refuse those against this cluster
with a message that says why, following the precedent of `tempo-cli` refusing
`--use-grpc`. An explicit `http(s)` server is always passed through untouched, so
promtool still works against anything else.

Verified against nonprod.jupiter.onglueops.rocks: bare `query instant`/`range`
return data, the refused paths print the reason, an explicit server still works,
and `--help` plus the local subcommands (`check`, `push`, `test`, `tsdb`) are
untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrpnuCduw2KivngVRQmnG7
@venkatamutyala
venkatamutyala merged commit d22018d into main Sep 5, 2026
3 checks passed
@venkatamutyala
venkatamutyala deleted the fix/promtool-datasource-url branch September 5, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant