Repository navigation
feat: with --metrics, serve the tip, the step of the peer and the reconnects at GET /metrics, in the Prometheus text format (#79) - #95
Merged
Conversation
…econnects at `GET /metrics`, in the Prometheus text format (#79) `--metrics` takes a port on 127.0.0.1, or an address and a port. elo listens there before it dials, and logs the address in a `metrics` line. A failed bind is a fault of this host, so elo exits with 1. The HTTP code is our own and has no dependency: one request for each connection, a head of at most 8 KiB, one connection at a time, each in at most 10 s. #70 can use the same code. The metrics are the height and the work of the header tip, the step of the connection (`dial`, `sync`, `synced`, `wait`) and the reconnects. The work is in decimal, with all its digits, so `Work` now divides with a remainder. The values are in `Cell`s: elo runs on one thread, and the node and the server are two futures of one task.
…d, so that a head in many small reads costs no quadratic time (#79) The search starts at the last 4 bytes of the previous fill. Before, each read searched the full buffer again, so 8 KiB in one-byte reads was 33 million window compares on the thread of the node.
… of the peer state and that the tip does not lose work (#79) `reconnect` and `peer(Wait)` were two calls that `main.rs` had to keep together. Now the move into `wait` is the reconnect, so the counter cannot drift from the state. `peer` asserts the steps of a connection: `dial` to `sync` to `synced`, each of them to `wait`, and `wait` to `dial`. So `main.rs` sets `dial` after the wait, and `Metrics::new` starts in `dial`. The chain asserts more work at each new tip, so `tip` asserts that the work does not go down. The height can go down on a reorganization, so `tip` does not assert it.
…s label checked, into a body of its maximum capacity (#79) `Text::family` returns a `Family` that has the name, so `text()` writes each name once and a sample cannot get the name of another family. `sample_labelled` takes the label and its value apart, not an empty string for no label. It asserts the label bytes, and that the value has no quote, backslash or newline, which the text format must escape. The body starts at `BODY_LENGTH_MAX`, so one scrape allocates once.
…eck each part of the request line in its own condition (#79) `content_type` and `allow` come from our code, so a CR or LF in one is a bug that would put a header of its own in the response.
…clients in the queue cannot hold a scrape past its timeout (#79) With 10 s for each connection and the default backlog of 1024, k slow clients before a scrape held it for k * 10 s. One idle connection each 10 s was enough to fail each scrape, as the default scrape timeout of Prometheus is 10 s too. A head is at most 8 KiB, so 1 s is enough. Linux queues one connection more than the backlog, so a scrape waits for at most 5 connections and its own: 6 s. A const assert keeps this below the 10 s. When the queue is full, the kernel drops the connect, and the client tries again. `http::listen` binds with the backlog in our code. It sets `SO_REUSEADDR`, as `TcpListener::bind` did.
…o_metrics_failures_total` (#79) `serve` dropped the error of each connection, so a limit of file descriptors or a run of slow clients left no trace. Now `serve` gives each failure to a function of the caller: `accept` for an accept error, `io` for a read or write error, and `timeout` for no answer in 1 s. A client that closes before a head, or that sends a bad head, is not a failure. The new family makes the longest body 1184 bytes, so its maximum is now 2 KiB.
…nly a counter has a name that ends with `_total` (#79) The type of a family was free text, so a family of type `guage` was possible. Prometheus refuses a scrape with an unknown type. The text of the tests did not find it, because the same hand writes the text and its test. Now a `Kind` is `Gauge` or `Counter`, and one match gives its name. A counter name ends with `_total`, and a gauge name does not.
…that the body fits when each value is at its maximum (#79) A test proved the maximum of the body with numbers by hand: 6 values of one digit, 78 for the work, and 4 counters. A new family that did not change that test kept the test green, and then the assert in `end` could stop elo at a scrape, when a counter got large. Now each type of a value has `LENGTH_MAX`, and `Text` counts the bytes that each value can still grow. `end` asserts that the body fits with each value at its maximum, so each test that writes the text proves the maximum. The longest body is now 1192 bytes, as a state of `u8` can have 3 digits. `Work::decimal` gives a `Decimal`, whose maximum is 78, and its test asserts the 78 against `Work::MAX`.
…n 8 newer ones come before its answer, so that idle clients cannot hold a scrape (#79) elo answered one connection at a time. The listen backlog of 4 holds 5 connections, and when it is full, the kernel drops the SYN of a scrape, and the client sends it again after 1 s, then 3 s. So the bound of 6 s held only for a scrape in the queue. In a netns, 8 idle clients made a scrape take 7.8 s, and 16 made it take 10.8 s, past the scrape timeout of 10 s of Prometheus. Now `serve` accepts each connection when it comes, and keeps at most 8 exchanges, each with its own timeout of 1 s. A new connection goes into the next of 8 places in turn. When that place has an exchange with no answer, elo closes it and counts it as `evicted`. So a connection stays open until its answer, its timeout, or 8 newer connections. With 4, 8, 16 or 64 idle clients, a scrape now takes less than 1 ms. The exchanges are futures in an array, which one `poll_fn` polls in the task of `serve`, so nothing is spawned, as before. The const assert now only asserts that the timeout of an exchange is less than the scrape timeout, as no queue of connections waits before a scrape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #79.
What
elo --metrics [<ip>:]<port> <network> [<peer>]servesGET /metricsin the Prometheus text format (text/plain; version=0.0.4).elo_header_tip_height,elo_header_tip_work,elo_peer_state{state="dial|sync|synced|wait"},elo_reconnects_total,elo_metrics_failures_total{kind="accept|io|timeout"}.metrics address=<ip:port>, before the first dial.src/http.rs: our own HTTP/1.1 code for one request on each connection.src/metrics.rs: the state, the names and the text writer.Work::decimal: the work with all its digits.Worknow divides with a remainder.Why
Off by default, and a port alone means 127.0.0.1. The issue asks for localhost by default and a flag for the address. With no flag, elo opens no port. So two elos on one host do not collide, and the image still needs no port.
--metrics 9332listens on127.0.0.1:9332.--metrics 0.0.0.0:9332is for a scraper in another container. Port 0 takes a free port, and themetricsline gives it. The L3 tests use that. I did not pick a default port, because none is ours. Rejected: on by default on a fixed port.Our own HTTP code, no dependency. hyper brings many crates for one path. The code reads one request head of at most 8 KiB, answers, and closes. It checks only the request line: a token method, a target of visible ASCII,
HTTP/1.0orHTTP/1.1. It does not read the headers. #70 can usehttp::servewith its ownanswer. One listener for both, or two, waits for #70.One connection at a time, 1 s each, from a backlog of 4. First this was 10 s with the default backlog of 1024. Then k slow clients before a scrape held it for k * 10 s, and one idle connect each 10 s failed each scrape. Now Linux queues at most 5 connections, so a scrape gets its answer in at most 6 s. A const assert keeps that below the 10 s default scrape timeout of Prometheus. A head of 8 KiB needs much less than 1 s. When the queue is full, the kernel drops the connect and the client tries again. A slow client never delays the node: the node and the server are two futures of one task (
besideinmain.rs). Rejected: concurrent connections, as the serial server is simpler and the bound is enough for one scraper.Each failed connection is counted.
servegives each failure to a function of the caller:accept,io(a read or a write) ortimeout. An accept error then waits 1 s, so an fd limit does not end the node. A client that closes before a head, or sends a bad head, is not a failure.No
tokio::spawn. A panic in a spawned task does not end the process. A bug then would not exit with 101.besidepolls the server next to the node, so a panic in either one ends elo.Cell, not atomics. elo runs on one thread, so no lock and no atomic is necessary. The issue asks for atomics on the block path. That path does not exist yet. When a second thread comes, the counters become atomics.Work in decimal, with all digits. Prometheus reads a value as a float, but the text can hold the exact integer. So no
as f64cast and no rounding in elo. 5 chunks of 19 digits hold 2^256 - 1, and a test checks that.Peer state as a stateset, with its steps asserted. One gauge for each of the 4 states, with 1 for the current one. The label set is fixed, with no label for each peer address.
dialcovers the lookup, the connect and the handshake.syncis the sync from the locator, andsyncedcomes after it.Metrics::peerasserts each step:dialtosynctosynced, each of them towait,waittodial. The move intowaitis the reconnect, so the counter and the state cannot drift.Metrics::tipasserts that the work does not go down, as the chain asserts more work at each new tip.Bounds. The body is at most 2 KiB, with an assert. A test adds the longest value of each sample to the text and checks that it fits (now 1184 bytes). Each sample goes through the
Familyof its name, and a label value with a quote, a backslash or a newline is a bug.Checks:
promtool check metrics(Prometheus 3.16) accepts the text, with no lint warning. All gates pass locally, L3 in a netns. Mutants on the diff: 62 caught, 0 missed, 16 unviable.Not confident:
tipline does. During a long first sync, the height stays at the last tip until the sync ends. A metric for headers that come in during a sync is for the next PR, together with the messages and bytes of each command.--metricsmust come before the network. A flag after it is bad usage.POST /is 404, not 405.acceptfail. Thetimeoutandiokinds have a test, with a paused clock and a reset connection.elo-testgate run failedannouncement_on_a_branch_that_lost_to_ours_after_a_low_work_sync_gets_no_getheaders(low-work sync: the chain ended below the minimum work). That test does not use--metrics, and with no flagbesidepolls only apendingfuture. It passed 20 runs alone and 8 full runs, on this branch and onmaineach, and the gate passed again. So I think it is a timing flake of that test under load, but I did not find its cause.Next in #79: messages and bytes in and out for each command (a hook in
net.rs), then histograms when #62 and #64 have flushes to measure.