feedline

MoldUDP64 market-data feed handler in C++20: A/B multicast line arbitration, gap detection and retransmission recovery, a lock-free SPSC handoff, and a deterministic chaos harness that proves the output stream is exact whatever the network does.

Repo description: C++20 market-data feed handler: MoldUDP64 A/B line arbitration, gap recovery, lock-free SPSC ring, and a deterministic fault-injection harness (loss, bursts, reorder, dead line) proving exactly-once in-order delivery.

Topics: cpp20 market-data feed-handler moldudp64 multicast udp lock-free spsc-queue low-latency hft fault-injection chaos-engineering thread-sanitizer

Part 2 of a three-repo low-latency trading study: limitbook (book + matching) ← feedline (feeds it) → greekstream (real-time options risk).

---

The problem

Exchanges publish every packet twice, on two independent multicast lines, because UDP drops packets and switch buffers overflow during bursts. A feed handler must merge A and B into one gap-free, duplicate-free, in-order stream:

• fill a hole from the other line if it shows up quickly (the common case);
• otherwise request retransmission, and re-request if the retransmission is lost;
• detect a gap at the tail of the stream, where nothing arrives after the loss;
• hand messages to the strategy thread without locks or allocation.

Getting this right is easy to claim and hard to prove. feedline proves it with a discrete-event simulator that applies loss, burst loss, duplication, reordering and dead lines to both feeds, with lossy recovery on top, and checks every byte of every message.

Architecture

flowchart LR
    subgraph network thread
      A[line A<br/>multicast] --> ARB
      B[line B<br/>multicast] --> ARB
      R[recovery<br/>unicast] --> ARB
      ARB[Arbiter<br/>dedup · reorder window · gap timer]
      ARB -- request(first, count) --> RS[(retransmission<br/>server)]
      RS -. resend .-> R
    end
    ARB -- on_message --> Q[[SpscRing<br/>acquire/release · cached indices]]
    subgraph consumer thread
      Q --> C[book builder / strategy]
    end

Loading

Component	Design
mold::PacketView / PacketBuilder	Zero-copy MoldUDP64 parse/build: header, message blocks, heartbeat, end-of-session, retransmission request
Arbiter<Sink, Recovery, Window, MaxMessage>	seq < next drop, == next deliver and drain, > next buffer in a fixed slot window (seq & (Window-1)); gap timer → per-run recovery requests → retry timer
SpscRing<T, N>	Bounded lock-free ring; release/acquire publication; producer and consumer cache each other's index; separate cache lines (128 B on Apple Silicon)
sim::FaultInjector	Per-line drop / burst / duplicate / reorder / dead; deterministic from a seed
sim::run_chaos	Event-driven simulation of publisher → two faulty lines → arbiter → verifier, with a lossy, delayed recovery server; optional real consumer thread
UdpSocket	Non-blocking POSIX multicast receiver and sender, plus unicast for recovery

Design notes: docs/DESIGN.md · C++ feature policy: docs/FEATURE_POLICY.md

Quick start (macOS / Linux)

brew install cmake ninja      # or apt install cmake ninja-build g++-13
make run                      # in-process chaos run: 1M messages, loss + bursts + reorder, lossy recovery
make test                     # 26 tests incl. 40 randomized chaos seeds and a 2-thread SPSC stress
make tsan                     # the same tests under ThreadSanitizer

Real sockets, two terminals:

make listen                   # terminal 1: joins 239.255.1.1 / 239.255.1.2 on 127.0.0.1
make publish                  # terminal 2: 2M messages @ 500k/s with 1% loss per line

If nothing arrives, pass --iface <your en0 IPv4> to both tools: some hosts don't route multicast on loopback.

Results (first milestone)

Measured on a 2-vCPU cloud VM (Xeon @ 2.8 GHz, GCC 13, -O3). Rerun on your own machine. Cross-core numbers on a shared VM are the least representative of all.

Scenario	Result
Arbiter, A+B both delivering every packet	129 M msgs/s
Arbiter, 10% loss on A filled from B	129 M msgs/s
SPSC throughput, 64-byte items, 2 threads	106 M items/s
SPSC ping-pong round trip	1.06 µs on this VM (expect far lower on bare metal: measure yours)
Chaos: 1M msgs, 2% loss each line + bursts + 5% reorder + 20% recovery loss, threaded consumer	exact; 473 packets lost on both lines, recovered; consumer stall p50 1 µs, p99 205 µs (simulated)
Chaos: line A dead, 1% loss on B	exact; stall p50 1.2 ms, p99 6.8 ms ← see "known weakness"
Real UDP multicast on loopback: 300k msgs @ 300k/s, 2% loss per line	exact payload verification; 38 msgs recovered over unicast; stall p99 262 µs

Known weakness, found by the chaos test

The gap timer is per gap episode, not per hole. With line A dead, every loss on B is a hole nobody else will fill. If a new hole opens while an older one is still being recovered, the new hole waits for the retry timer (500 µs) instead of the gap timeout (50 µs). That's why the dead-line p99 stall is 6.8 ms. Fixing it is Milestone 2, with a number to beat.

Testing strategy

Test	What it proves
Chaos.RandomizedSeedsAlwaysExact	40 seeds × random fault profiles on both lines (incl. dead lines) and lossy recovery → output is exactly 1..N, in order, byte-identical
Chaos.CorrelatedLossNeedsRecovery	Packets lost on both lines are recovered via retransmission
Chaos.ThreadedConsumerSeesExactStream	The same guarantee across a real thread boundary through the SPSC ring
SpscRing.ConcurrentProducerConsumerPreservesSequence	2M items across two threads; under TSan it flags any missing acquire/release
Arbiter.*	Dedup, other-line fill before timeout, timeout → request → retry, multi-run requests, heartbeat-revealed tail gap, foreign session rejection, end-of-session completion
UdpLoopback.*	Real multicast and unicast round trips (skipped, not failed, if the host has no multicast route)

Mutation-tested ordering: downgrading the ring's release store and acquire load to relaxed makes ThreadSanitizer report a data race in the stress test. On x86 the corrupted build still produces correct output, because x86 orders stores strongly. Apple Silicon (ARM) does not. A passing test on x86 proves nothing about ARM; TSan does.

Project layout

include/feedline/
  wire.hpp  mold_udp64.hpp  arbiter.hpp  spsc_ring.hpp  latency_histogram.hpp  thread_util.hpp  udp.hpp
  sim/  rng.hpp  payload.hpp  fault_injector.hpp  chaos.hpp
tools/  feedline_chaos.cpp  feedline_publish.cpp  feedline_listen.cpp
tests/  test_mold_udp64  test_spsc_ring  test_arbiter  test_chaos  test_udp_loopback
bench/  bench_feedline.cpp

Roadmap: the parts left for you

Details in docs/ROADMAP.md.

• M2: Per-hole gap timers. Target: dead-line p99 stall under 300 µs (from 6.8 ms)
• M3: recvmmsg batching and SO_TIMESTAMPING hardware/kernel timestamps → true wire-to-consumer latency
• M4: Kernel bypass on Linux: AF_XDP or io_uring multishot receive; compare against the socket path
• M5: Snapshot + replay recovery when a gap exceeds the reorder window (the window_overflows path)
• M6: Feed limitbook: --itch consumer → BookBuilder, end-to-end with loss
• M7: CPU isolation playbook on Linux (isolcpus, IRQ affinity, SCHED_FIFO) with before/after histograms

Skills demonstrated

• Concurrency: lock-free SPSC ring, C++ memory model (memory_order_acquire / release / relaxed), false-sharing avoidance, cached remote indices, busy-polling with pause/yield, CPU pinning, ThreadSanitizer, mutation-tested ordering
• Networking: UDP multicast (IP_ADD_MEMBERSHIP, IP_MULTICAST_IF/LOOP/TTL), non-blocking sockets, SO_RCVBUF sizing, unicast request/response
• Market data protocols: NASDAQ MoldUDP64 sessions, sequence numbers, heartbeats, end-of-session, retransmission requests; A/B line arbitration
• Modern C++ (C++20): concepts (MessageSink, RecoveryClient), constrained class templates (requires power-of-two capacity), std::span/std::byte, in-place construction via try_push_with, if constexpr, feature-test macros, designated initializers
• Reliability engineering: deterministic discrete-event simulation, fault injection (loss, bursts, reorder, duplication, dead line), property-based tests over many seeds, byte-level payload verification
• Build and CI: CMake presets, GitHub Actions matrix (GCC/Clang × Release/ASan/TSan + AppleClang), -Werror -Wconversion

License

MIT © Udaykishore Resu