Skip to content

feat: with --metrics, serve the tip, the step of the peer and the reconnects at GET /metrics, in the Prometheus text format (#79) - #95

Merged
lorenzolfm merged 11 commits into
mainfrom
issue-79
Nov 15, 2026
Merged

lorenzolfm merged 11 commits into
mainfrom
issue-79

Conversation

@lorenzolfm

@lorenzolfm lorenzolfm commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Part of #79.

What

  • elo --metrics [<ip>:]<port> <network> [<peer>] serves GET /metrics in the Prometheus text format (text/plain; version=0.0.4).
  • First metrics: elo_header_tip_height, elo_header_tip_work, elo_peer_state{state="dial|sync|synced|wait"}, elo_reconnects_total, elo_metrics_failures_total{kind="accept|io|timeout"}.
  • New log event metrics address=<ip:port>, before the first dial.
  • src/http.rs: our own HTTP/1.1 code for one request on each connection. src/metrics.rs: the state, the names and the text writer.
  • Work::decimal: the work with all its digits. Work now divides with a remainder.

Why

Off by default, and a port alone means 127.0.0.1. The issue asks for localhost by default and a flag for the address. With no flag, elo opens no port. So two elos on one host do not collide, and the image still needs no port. --metrics 9332 listens on 127.0.0.1:9332. --metrics 0.0.0.0:9332 is for a scraper in another container. Port 0 takes a free port, and the metrics line gives it. The L3 tests use that. I did not pick a default port, because none is ours. Rejected: on by default on a fixed port.

Our own HTTP code, no dependency. hyper brings many crates for one path. The code reads one request head of at most 8 KiB, answers, and closes. It checks only the request line: a token method, a target of visible ASCII, HTTP/1.0 or HTTP/1.1. It does not read the headers. #70 can use http::serve with its own answer. One listener for both, or two, waits for #70.

One connection at a time, 1 s each, from a backlog of 4. First this was 10 s with the default backlog of 1024. Then k slow clients before a scrape held it for k * 10 s, and one idle connect each 10 s failed each scrape. Now Linux queues at most 5 connections, so a scrape gets its answer in at most 6 s. A const assert keeps that below the 10 s default scrape timeout of Prometheus. A head of 8 KiB needs much less than 1 s. When the queue is full, the kernel drops the connect and the client tries again. A slow client never delays the node: the node and the server are two futures of one task (beside in main.rs). Rejected: concurrent connections, as the serial server is simpler and the bound is enough for one scraper.

Each failed connection is counted. serve gives each failure to a function of the caller: accept, io (a read or a write) or timeout. An accept error then waits 1 s, so an fd limit does not end the node. A client that closes before a head, or sends a bad head, is not a failure.

No tokio::spawn. A panic in a spawned task does not end the process. A bug then would not exit with 101. beside polls the server next to the node, so a panic in either one ends elo.

Cell, not atomics. elo runs on one thread, so no lock and no atomic is necessary. The issue asks for atomics on the block path. That path does not exist yet. When a second thread comes, the counters become atomics.

Work in decimal, with all digits. Prometheus reads a value as a float, but the text can hold the exact integer. So no as f64 cast and no rounding in elo. 5 chunks of 19 digits hold 2^256 - 1, and a test checks that.

Peer state as a stateset, with its steps asserted. One gauge for each of the 4 states, with 1 for the current one. The label set is fixed, with no label for each peer address. dial covers the lookup, the connect and the handshake. sync is the sync from the locator, and synced comes after it. Metrics::peer asserts each step: dial to sync to synced, each of them to wait, wait to dial. The move into wait is the reconnect, so the counter and the state cannot drift. Metrics::tip asserts that the work does not go down, as the chain asserts more work at each new tip.

Bounds. The body is at most 2 KiB, with an assert. A test adds the longest value of each sample to the text and checks that it fits (now 1184 bytes). Each sample goes through the Family of its name, and a label value with a quote, a backslash or a newline is a bug.

Checks: promtool check metrics (Prometheus 3.16) accepts the text, with no lint warning. All gates pass locally, L3 in a netns. Mutants on the diff: 62 caught, 0 missed, 16 unviable.

Not confident:

  • The tip moves only when the tip line does. During a long first sync, the height stays at the last tip until the sync ends. A metric for headers that come in during a sync is for the next PR, together with the messages and bytes of each command.
  • --metrics must come before the network. A flag after it is bad usage.
  • No fuzz target for the request line. The listener is on localhost by default, and the parser is a table test.
  • An unknown path gets 404 before the method is checked, so POST / is 404, not 405.
  • No test makes accept fail. The timeout and io kinds have a test, with a paused clock and a reset connection.
  • One elo-test gate run failed announcement_on_a_branch_that_lost_to_ours_after_a_low_work_sync_gets_no_getheaders (low-work sync: the chain ended below the minimum work). That test does not use --metrics, and with no flag beside polls only a pending future. It passed 20 runs alone and 8 full runs, on this branch and on main each, and the gate passed again. So I think it is a timing flake of that test under load, but I did not find its cause.

Next in #79: messages and bytes in and out for each command (a hook in net.rs), then histograms when #62 and #64 have flushes to measure.

…econnects at `GET /metrics`, in the Prometheus text format (#79)

`--metrics` takes a port on 127.0.0.1, or an address and a port. elo
listens there before it dials, and logs the address in a `metrics`
line. A failed bind is a fault of this host, so elo exits with 1.

The HTTP code is our own and has no dependency: one request for each
connection, a head of at most 8 KiB, one connection at a time, each in
at most 10 s. #70 can use the same code.

The metrics are the height and the work of the header tip, the step of
the connection (`dial`, `sync`, `synced`, `wait`) and the reconnects.
The work is in decimal, with all its digits, so `Work` now divides with
a remainder. The values are in `Cell`s: elo runs on one thread, and the
node and the server are two futures of one task.
…d, so that a head in many small reads costs no quadratic time (#79)

The search starts at the last 4 bytes of the previous fill. Before,
each read searched the full buffer again, so 8 KiB in one-byte reads
was 33 million window compares on the thread of the node.
… of the peer state and that the tip does not lose work (#79)

`reconnect` and `peer(Wait)` were two calls that `main.rs` had to keep
together. Now the move into `wait` is the reconnect, so the counter
cannot drift from the state.

`peer` asserts the steps of a connection: `dial` to `sync` to
`synced`, each of them to `wait`, and `wait` to `dial`. So `main.rs`
sets `dial` after the wait, and `Metrics::new` starts in `dial`.

The chain asserts more work at each new tip, so `tip` asserts that the
work does not go down. The height can go down on a reorganization, so
`tip` does not assert it.
…s label checked, into a body of its maximum capacity (#79)

`Text::family` returns a `Family` that has the name, so `text()` writes
each name once and a sample cannot get the name of another family.
`sample_labelled` takes the label and its value apart, not an empty
string for no label. It asserts the label bytes, and that the value has
no quote, backslash or newline, which the text format must escape.

The body starts at `BODY_LENGTH_MAX`, so one scrape allocates once.
…eck each part of the request line in its own condition (#79)

`content_type` and `allow` come from our code, so a CR or LF in one is
a bug that would put a header of its own in the response.
…clients in the queue cannot hold a scrape past its timeout (#79)

With 10 s for each connection and the default backlog of 1024, k slow
clients before a scrape held it for k * 10 s. One idle connection each
10 s was enough to fail each scrape, as the default scrape timeout of
Prometheus is 10 s too.

A head is at most 8 KiB, so 1 s is enough. Linux queues one connection
more than the backlog, so a scrape waits for at most 5 connections and
its own: 6 s. A const assert keeps this below the 10 s. When the queue
is full, the kernel drops the connect, and the client tries again.

`http::listen` binds with the backlog in our code. It sets
`SO_REUSEADDR`, as `TcpListener::bind` did.
…o_metrics_failures_total` (#79)

`serve` dropped the error of each connection, so a limit of file
descriptors or a run of slow clients left no trace. Now `serve` gives
each failure to a function of the caller: `accept` for an accept
error, `io` for a read or write error, and `timeout` for no answer in
1 s. A client that closes before a head, or that sends a bad head, is
not a failure.

The new family makes the longest body 1184 bytes, so its maximum is now
2 KiB.
…nly a counter has a name that ends with `_total` (#79)

The type of a family was free text, so a family of type `guage` was
possible. Prometheus refuses a scrape with an unknown type. The text of
the tests did not find it, because the same hand writes the text and
its test. Now a `Kind` is `Gauge` or `Counter`, and one match gives
its name. A counter name ends with `_total`, and a gauge name does not.
…that the body fits when each value is at its maximum (#79)

A test proved the maximum of the body with numbers by hand: 6 values
of one digit, 78 for the work, and 4 counters. A new family that did not
change that test kept the test green, and then the assert in `end`
could stop elo at a scrape, when a counter got large.

Now each type of a value has `LENGTH_MAX`, and `Text` counts the bytes
that each value can still grow. `end` asserts that the body fits with
each value at its maximum, so each test that writes the text proves the
maximum. The longest body is now 1192 bytes, as a state of `u8` can have
3 digits. `Work::decimal` gives a `Decimal`, whose maximum is 78, and its
test asserts the 78 against `Work::MAX`.
…n 8 newer ones come before its answer, so that idle clients cannot hold a scrape (#79)

elo answered one connection at a time. The listen backlog of 4 holds 5
connections, and when it is full, the kernel drops the SYN of a scrape,
and the client sends it again after 1 s, then 3 s. So the bound of 6 s
held only for a scrape in the queue. In a netns, 8 idle clients made a
scrape take 7.8 s, and 16 made it take 10.8 s, past the scrape timeout
of 10 s of Prometheus.

Now `serve` accepts each connection when it comes, and keeps at most 8
exchanges, each with its own timeout of 1 s. A new connection goes into
the next of 8 places in turn. When that place has an exchange with no
answer, elo closes it and counts it as `evicted`. So a connection stays
open until its answer, its timeout, or 8 newer connections. With 4, 8,
16 or 64 idle clients, a scrape now takes less than 1 ms.

The exchanges are futures in an array, which one `poll_fn` polls in the
task of `serve`, so nothing is spawned, as before. The const assert now
only asserts that the timeout of an exchange is less than the scrape
timeout, as no queue of connections waits before a scrape.
@lorenzolfm
lorenzolfm merged commit 93afd0d into main Nov 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant