diff --git a/README.md b/README.md index e9cfed0..dec1944 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,11 @@ public internet cannot reach. This repository is the complete source of what runs there. The image Perfloop publishes is built from it, by the workflow in this repository, and is tagged with the commit it was built from. -Sources the relay reads today: Prometheus, VictoriaMetrics, and Loki. Go -pprof and Pyroscope are next. Each kind is one table of read routes in -`routes.go`, and a kind the relay does not know is refused when it starts. +Sources the relay reads today: Prometheus, VictoriaMetrics, Loki, and Go +pprof. Each kind is one table of read routes in `routes.go`, and a kind the +relay does not know is refused when it starts. -756 lines of Go outside tests, three runtime dependencies (`coder/websocket`, +947 lines of Go outside tests, three runtime dependencies (`coder/websocket`, `go.yaml.in/yaml/v3`, `golang.org/x/net`): read it in an afternoon. ## What it does diff --git a/docs/configuration.md b/docs/configuration.md index 11b34f8..7984c9d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -28,12 +28,13 @@ upstreams: # leading or trailing -. Names must be unique in the file. name: vm # One of the kinds routes.go lists: today prometheus, victoriametrics, - # or loki. Selects the read routes + # loki, or pprof. Selects the read routes # (docs/security.md, "What the relay forwards"). kind: victoriametrics # The provider base: scheme, host, port, and an optional path prefix. # No user info, query, fragment, or trailing slash. The read's path is - # appended to it. + # appended to it. Required for every kind but pprof, which names + # targets instead. url: http://vmselect.monitoring.svc:8481/select/0/prometheus # Optional. Headers added to every request to this upstream; they # replace any header of the same name that Perfloop sent. Values are @@ -46,7 +47,37 @@ upstreams: - name: logs kind: loki url: https://loki-gateway.monitoring.svc + - name: go + kind: pprof + # Required for kind pprof, at least one. The Go processes this upstream + # may profile: target name to the http(s) base of that process's + # net/http/pprof handler, with the same URL rules as `url`. Each name is + # one lowercase DNS label. Perfloop names one target per read; a name not + # listed here is refused. + targets: + api-1: http://api-1.prod.svc:6060 + api-2: http://api-2.prod.svc:6060/internal + # Optional, kind pprof only. The longest `seconds` a read may ask for, + # 1 to 45. Default 30, net/http/pprof's own CPU profile length. + max_seconds: 10 + # Optional, kind pprof only. Profile reads in flight on this upstream, + # across all of its targets, per relay process; 1 to 8. Default 1. A + # further read is refused with status 429. + max_concurrent: 2 ``` +## pprof upstreams + +A `pprof` upstream reads `net/http/pprof` from Go processes you name. It has +`targets` instead of `url`; `targets`, `max_seconds`, and `max_concurrent` +are refused on every other kind. The relay forwards `GET` to exactly these +routes under each target's base: `/debug/pprof/profile`, `heap`, `allocs`, +`goroutine`, `mutex`, and `block`. `seconds` is the only query key the relay +accepts: the capture length for `profile`, which must carry it, and an +optional delta window for the other routes. It is one integer from 1 to +`max_seconds`. Every other key, `debug` and `gc` included, is refused, so only +the binary profile leaves the process. `-print-routes` prints the targets and +both caps. + The validation rules are in `config.go`, `Validate`; `config_test.go` lists -every refused shape. +every refused shape. The pprof query rules are in `routes.go`, `pprofQuery`. diff --git a/docs/operations.md b/docs/operations.md index fbc4b84..a56d7bc 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -18,7 +18,7 @@ perfloop-relay -config relay.yaml -print-routes ``` loads and validates the configuration and prints, per upstream, its kind, -its URL, the request headers it passes from Perfloop, the names of its +its URL (for `pprof`, its targets and caps), the request headers it passes from Perfloop, the names of its configured headers, and every request the relay will forward, then exits. An invalid configuration exits non-zero with the validation error. Header values and the token are not printed. Example: @@ -50,7 +50,7 @@ The relay writes one line per read, `msg: "relay read"`, with these fields: | `method`, `path` | The method and path as Perfloop sent them. The query string is not logged, in this field or in `error`. | | `request_id` | The Perfloop work behind the read, `/`, from the `Perfloop-Request-Id` header. Empty for a read no tool call made, such as a connect check from Setup. | | `status` | The status the relay answered. | -| `decision` | The read route that was forwarded (`query`, `labels`, ...), or why the read was refused: `unknown-upstream`, `method`, `path`, `route`, `upstream-error`, or `truncated`. | +| `decision` | The read route that was forwarded (`query`, `labels`, ...), or why the read was refused: `unknown-upstream`, `method`, `path`, `route`, `upstream-error`, or `truncated`; for `pprof` also `unknown-target`, `query`, or `busy` (the `max_concurrent` cap). | | `ms` | Time spent on the read. | | `error` | The upstream error, if any, without the request URL (`relay.go`, `forward`). | diff --git a/docs/security.md b/docs/security.md index a7aba5c..9c6abed 100644 --- a/docs/security.md +++ b/docs/security.md @@ -98,8 +98,13 @@ The documented read routes, per upstream kind: | --- | --- | --- | | `prometheus`, `victoriametrics` | `/api/v1/` | `query`, `query_range`, `labels`, `label/{name}/values`, `series`, `metadata` | | `loki` | `/loki/api/v1/` | `query`, `query_range`, `labels`, `label/{name}/values` | +| `pprof` | `//debug/pprof/` | `profile`, `heap`, `allocs`, `goroutine`, `mutex`, `block` | -`{name}` is one non-empty path segment. Admin, write, push, and delete routes +`{name}` is one non-empty path segment. `` is a name in the pprof +upstream's `targets`; any other answers `404 unknown target`. A pprof read's +query must be empty or exactly one `seconds` from 1 to the upstream's +`max_seconds`, and `profile` must carry it; anything else answers `400`. A +read past the upstream's `max_concurrent` answers `429`. Admin, write, push, and delete routes are not in the table and are refused before any upstream request; the tests in `routes_test.go` and `relay_test.go` include the cases `admin/tsdb/delete_series`, `admin/tsdb/snapshot`, `write`, `push`, and a `..` in the path. diff --git a/docs/setup.md b/docs/setup.md index ddd440a..7e8daec 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -13,8 +13,8 @@ Collect these before starting. Nothing else is needed. | `` | Your Perfloop tenant name, the segment after `/t/` in the app URL. | | `` | The Kubernetes namespace the relay will run in. | | `` | A name you choose for the source, one lowercase DNS label, for example `vm`. | -| `` | One of the kinds `routes.go` lists: today `prometheus`, `victoriametrics`, or `loki`. | -| `` | The provider base URL as reachable from inside the cluster, for example `http://vmselect.monitoring.svc:8481/select/0/prometheus`. No trailing slash. | +| `` | One of the kinds `routes.go` lists: today `prometheus`, `victoriametrics`, `loki`, or `pprof`. | +| `` | The provider base URL as reachable from inside the cluster, for example `http://vmselect.monitoring.svc:8481/select/0/prometheus`. No trailing slash. A `pprof` upstream has `targets` instead, one such URL per Go process ([configuration.md](configuration.md), "pprof upstreams"). | | `` | A read-only token for that upstream, if it needs one. Query-scoped. | | `` | The relay image digest to pin. See "Choose the image" below. | | A Perfloop admin | Creating the relay token and registering the source happen in Perfloop Setup, which needs an admin login in the browser. | diff --git a/examples/kubernetes/relay.yaml b/examples/kubernetes/relay.yaml index a847c2a..7568a54 100644 --- a/examples/kubernetes/relay.yaml +++ b/examples/kubernetes/relay.yaml @@ -18,8 +18,8 @@ data: token: ${PERFLOOP_RELAY_TOKEN} upstreams: - name: - kind: # one of the kinds routes.go lists: prometheus, victoriametrics, loki - url: http://..svc: + kind: # one of the kinds routes.go lists: prometheus, victoriametrics, loki, pprof + url: http://..svc: # a pprof upstream has targets instead: docs/configuration.md headers: Authorization: Bearer ${UPSTREAM_READ_TOKEN} ---