Connect the matching loop to a real node - #26
Merged
Merged
Conversation
Two SUB sockets, one per endpoint, drained into StreamIngest. The module is deliberately thin: decode a multipart message and hand it over. Everything that can be decided without a socket was decided elsewhere, and this is the one place in the package that gives up being testable with no network. Two sockets is the design rather than tidiness. A block message is ~1.7 MB, so sharing one queue with the transaction stream puts a block in front of thousands of transactions in a bounded queue -- the sharing is itself a cause of the drops everything else here is braced against. Blocks are drained before transactions when both are ready. A block is the only thing that writes the record; a transaction only alerts. Draining the writer first is what lets a spend published in the same breath as its funding be recognised at all. The receive buffer is raised well above the ZMQ default of 1000, which is the setting the measured loss was measured under. It is worth being precise about what that does NOT do: bitcoind drops at its own high-water mark before the bytes reach us, which is a node setting and issue #25, so no buffer here recovers those. The buffer reduces loss, the sequence tracker notices it, and repair is what fixes it. Reconnects are a gap and are meant to be. ZMQ reconnects silently, so what arrives is a jumped counter -- reported, and demanding reconciliation. The one case nothing here can cover is a *fresh* subscriber: the first message on a topic establishes a baseline, so a starting service cannot know what happened while it was not listening. That hole belongs to the enrolment baseline scan and to the chain follower, not to a subscriber trying harder. pyzmq is now a runtime dependency, and SECURITY.md's claim that there are none is replaced rather than deleted. What that sentence was really claiming -- a supply chain small enough to read -- is still the goal and should stay checkable, so it now names the single dependency, why it is unavoidable (bitcoind publishes over ZMQ; there is no stdlib client and reimplementing the wire protocol would be the worse risk), and what it can reach: two localhost sockets, no keys, no database, no destinations. A dependency finding is in scope if it is against pyzmq and comes with a reachable path. The tests drive real sockets on ephemeral loopback ports rather than a mock, because every property worth testing here belongs to the transport -- bounded queues, silent discards, multipart framing, slow joiners -- and a fake would get each of them politely wrong. The slow-joiner behaviour is real and is worked around in the test rather than in the service: a subscriber that has just connected genuinely has missed what came before. Not here: the service entrypoint. Wiring this to a config, a database and the chain follower is the deployment slice, and the induced-gap proof on the beta node is this issue's definition of done rather than something the suite can assert. Refs #24 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connects the matching loop to a real node. The module is deliberately thin — decode a multipart message and hand it over — because everything that could be decided without a socket was decided elsewhere, and this is the one place in the package that gives up being runnable with no network.
Design points that are load-bearing
Two sockets, one per endpoint. Not tidiness. A block message is ~1.7 MB, so sharing one queue with the transaction stream puts a block in front of thousands of transactions in a bounded queue — the sharing is itself a cause of the drops everything else here is braced against.
Blocks are drained before transactions when both are ready. A block is the only thing that writes the record; a transaction only alerts. Draining the writer first is what lets a spend published in the same breath as its funding be recognised at all.
The receive buffer is raised well above the ZMQ default of 1000 — the setting the measured loss was measured under. Worth being precise about what that does not do: bitcoind drops at its own high-water mark before the bytes reach us (that is #25, a node setting), so no buffer here recovers those. The buffer reduces loss, the sequence tracker notices it, and repair fixes it.
Reconnects are a gap, and are meant to be. ZMQ reconnects silently, so what arrives is a jumped counter — reported, and demanding reconciliation. The one case nothing here can cover is a fresh subscriber: the first message on a topic establishes a baseline, so a starting service cannot know what happened while it was not listening. That hole belongs to the enrolment baseline scan and to the chain follower, not to a subscriber trying harder.
pyzmq becomes the first runtime dependency
SECURITY.md's claim that the runtime has none is replaced rather than deleted. What that sentence was really claiming — a supply chain small enough to read — is still the goal and should stay checkable, so it now names the one dependency, why it is unavoidable (bitcoind publishes over ZMQ; there is no stdlib client, and reimplementing the wire protocol would be the worse risk), and what it can reach: two localhost sockets, no keys, no database, no destinations.A dependency finding is in scope if it is against
pyzmqand comes with a reachable path.Tested against real sockets, and against the real node
The suite drives real ZMQ on ephemeral loopback ports rather than a mock. Every property worth testing here belongs to the transport — bounded queues, silent discards, multipart framing, slow joiners — and a fake gets each of them politely wrong. The slow-joiner behaviour is worked around in the test, not in the service: a subscriber that has just connected genuinely has missed what came before.
Beyond the suite, this was run against the production node read-only over forwarded ports, writing nothing:
254 real mainnet transactions parsed, none malformed. Until now the transaction parser had only ever seen its own synthetic fixtures; this is the first evidence that the segwit path, the witness-stripped txid, and the 3-part envelope decoding are right against a real publisher rather than against our own generator.
What this does not close
#24 stays open. Its definition of done is an induced gap, repaired against the real node, and that is not done:
A repair path that has never run against a real node is not known to work, and this PR does not claim otherwise. It is the code the proof will run on.
Also not here: the service entrypoint. Wiring this to a config, a database and the chain follower is the deployment slice.
Checks
175 tests, ruff clean, mutation sweep 65/65 with 3 documented equivalents.
Refs #24
🤖 Generated with Claude Code