Skip to content

Commit 7e64e6e

Browse files
committed
feat: acceptor plus K reactor threads with socket hand-off inbox
Server::run now runs an acceptor (the calling thread) plus K reactor threads (threads=0 -> hardware_concurrency). Accepted sockets move to a reactor over a mutex-guarded inbox woken by net::Wakeup (POSIX socketpair; Windows loopback pair) - a Connection is owned by exactly one reactor, so connection state stays lock-free. Round-robin placement; deadlines, backpressure and drain-then- drop stop are per-reactor. Fix found on the Linux gate: SIGINT turns accept poll into EINTR; the error path used to join reactors and the normal stop path joined them again -> pthread_join EINVAL out of run(). Interrupted now just re-checks stop, and stopAll joins only joinable threads. Tests: cross-reactor keep-alive (12 conns / 4 reactors), stop-drain across all reactors, pipelined storm with interleaved fresh clients - all under CI TSan. Linux gates re-run under Docker Desktop after podman/VS disappeared from this machine; clang-format now invoked via LLVM path in the local gate.
1 parent 8ec38fd commit 7e64e6e

13 files changed

Lines changed: 425 additions & 74 deletions

File tree

‎CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ endif()
6161

6262
add_library(net STATIC
6363
src/net/socket_common.cpp
64+
src/net/wakeup.cpp
6465
${net_platform_srcs})
6566
target_include_directories(net PUBLIC src)
6667
if(WIN32)

‎CMakePresets.json‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@
22
"version": 3,
33
"cmakeMinimumRequired": {
44
"major": 3,
5-
"minor": 25,
6-
"rc": 0
5+
"minor": 25
76
},
87
"configurePresets": [
98
{

‎PLAN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ phase/step/milestone wording, ever.
174174
| ✓ | ~~`feature/posix-socket-layer`~~ | The seam: portable `net/` API + `raw_win`/`raw_posix` backends; `accept4` + `MSG_NOSIGNAL` + `EINTR`; cross-platform socket tests; Docker gate builds whole Linux tree | merged — both gates green |
175175
| C | `feature/ci-matrix` | Actions matrix: Linux job = sanitizer container (`tools/asan-inside.sh`), Windows = native MinGW + ctest | merged — PR #5 gated itself green on first run |
176176
| R | `feature/reactor-and-connection` | single-threaded poll reactor: connection state machine, timeouts (408), POLLOUT backpressure, signals (self-pipe / console ctrl), graceful drain | merged — 200/keep-alive/408/400/stop integration tests; ASan + TSan green; SIGINT-exit-0 smoke in the Linux gate |
177-
| T | `feature/reactor-threading` | acceptor thread + K reactors + hand-off inbox; `Connection` stays owned by one reactor | TSan green on concurrent tests; RSS stays flat |
177+
| ✓ | ~~`feature/reactor-threading`~~ | acceptor thread + K reactors + socket hand-off inbox (`net::Wakeup`) | merged — cross-reactor keep-alive/pipelining/stop-drain tests; TSan green; EINTR double-join fixed |
178178
| ✓ | ~~`feature/static-files`~~ | root mapping, traversal guard, MIME, index, 404/405, HEAD (`std::filesystem`) + per-connection write cap | merged — traversal/HEAD/405/slow-loris/pipelined-burst tests both OSes; ASan + TSan green |
179179
| G | `feature/caching-and-range` | ETag, Last-Modified, If-None-Match, Range/206/416 | 304 + 206 tests incl. multi-range rejection |
180180
| H | `feature/hardening` | slow-body limit, smuggling corpus, soak (pipelining + slow-client tests landed with S) | ASan clean; 60 s soak flat RSS on Linux |

‎docs/ARCHITECTURE.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,17 @@ until then.
2222
obsolete folding, duplicate `Content-Length`, `Content-Length`+`Transfer-Encoding`,
2323
non-token names/methods, unknown versions. URL decode/normalise and MIME live
2424
with the static handler (they are a serving concern, not a parsing one).
25-
- `server/` — single-threaded poll reactor (`Server::run`): accept batch, per-connection
26-
deadlines (408), graceful stop (drain queued responses, grace cap, then drop).
27-
`Connection` = socket + `RequestParser` + capped output buffer (`max_queued_bytes`):
28-
a slow reader throttles the drain — queued responses stop growing, remaining
29-
pipelined requests wait in the parser, POLLOUT or the next read resumes it.
25+
- `server/` — acceptor thread + K reactor threads (`Server::run`, `threads=0` →
26+
hardware_concurrency). The acceptor batches accepts and hands each socket to a
27+
reactor round-robin over a mutex-guarded inbox plus a `net::Wakeup` socketpair;
28+
a `Connection` (socket + `RequestParser` + capped output buffer
29+
`max_queued_bytes`) is owned by exactly one reactor forever — no connection
30+
state is ever shared or locked. Poll timeouts come from per-connection
31+
deadlines (408); graceful stop drains queued responses per reactor, grace cap,
32+
then drops. EINTR from poll is just a re-check (the stop path is the only
33+
joiner). A slow reader throttles its connection — queued responses stop
34+
growing, remaining pipelined requests wait in the parser, POLLOUT or the next
35+
read resumes the drain.
3036
Handlers return `Reply{status, content_type, body}`; HEAD keeps an honest
3137
`Content-Length` and no body; handler exceptions become 500 at the reactor seam.
3238
`StaticFiles` serves a canonical root: strict percent-decode before resolution,

‎src/main.cpp‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,16 +13,25 @@ int main(int argc, char** argv) {
1313
port = std::atoi(argv[1]);
1414
}
1515
if (port < 1 || port > 65535) {
16-
std::fprintf(stderr, "usage: http_server [port] [root]\n");
16+
std::fprintf(stderr, "usage: http_server [port] [root] [threads]\n");
1717
return 2;
1818
}
19+
unsigned threads = 0; // 0 = hardware_concurrency
20+
if (argc > 3) {
21+
threads = static_cast<unsigned>(std::atoi(argv[3]));
22+
if (threads > 128) {
23+
std::fprintf(stderr, "usage: threads must be 0-128\n");
24+
return 2;
25+
}
26+
}
1927

2028
try {
2129
net::Context ctx;
2230
app::ShutdownSignals signals;
2331
const std::string root = argc > 2 ? argv[2] : ".";
2432
server::StaticFiles files(root);
2533
server::Server server(static_cast<std::uint16_t>(port), files);
34+
server.threads = threads;
2635
std::printf("serving %s on http://127.0.0.1:%d/ (stop with Ctrl-C)\n",
2736
files.root().string().c_str(), port);
2837
std::fflush(stdout);

‎src/net/detail_raw.h‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,9 @@ namespace detail {
1010
// OS primitives. Error detail stays in the OS error slot (WSAGetLastError/errno);
1111
// read it via net::lastError() immediately after a failed call.
1212

13-
handle_t rawSocket(); // stream socket, IPv4 (CloExec where supported)
13+
handle_t rawSocket(); // stream socket, IPv4 (CloExec where supported)
14+
// Connected non-blocking bidirectional pipe for poll-set wake-ups (both valid).
15+
bool rawSocketPair(handle_t out[2]);
1416
int rawClose(handle_t); // 0 on success
1517
bool rawSetNonBlocking(handle_t, bool);
1618
handle_t rawAccept(handle_t listener); // raw; then call rawAfterAccept

‎src/net/raw_posix.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,13 @@ handle_t rawSocket() {
1212
return ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
1313
}
1414

15+
bool rawSocketPair(handle_t out[2]) {
16+
if (::socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC | SOCK_NONBLOCK, 0, out) != 0) {
17+
return false;
18+
}
19+
return true;
20+
}
21+
1522
int rawClose(handle_t handle) {
1623
return ::close(handle);
1724
}

‎src/net/raw_win.cpp‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,42 @@ handle_t rawSocket() {
1111
return ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
1212
}
1313

14+
// No socketpair() on Winsock: build the equivalent from a loopback listener.
15+
// Setup runs blocking (one-time); the pair leaves non-blocking.
16+
bool rawSocketPair(handle_t out[2]) {
17+
const handle_t l = ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
18+
if (l == kInvalidHandle) {
19+
return false;
20+
}
21+
sockaddr_in a{};
22+
a.sin_family = AF_INET;
23+
a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
24+
a.sin_port = 0;
25+
int len = sizeof(a);
26+
bool ok = ::bind(l, reinterpret_cast<sockaddr*>(&a), sizeof(a)) != SOCKET_ERROR &&
27+
::listen(l, 1) != SOCKET_ERROR &&
28+
::getsockname(l, reinterpret_cast<sockaddr*>(&a), &len) != SOCKET_ERROR;
29+
const handle_t c = ok ? ::socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) : kInvalidHandle;
30+
if (ok && c != kInvalidHandle) {
31+
ok = ::connect(c, reinterpret_cast<sockaddr*>(&a), sizeof(a)) != SOCKET_ERROR;
32+
}
33+
if (ok) {
34+
out[0] = ::accept(l, nullptr, nullptr);
35+
out[1] = c;
36+
ok = out[0] != kInvalidHandle;
37+
}
38+
::closesocket(l);
39+
if (!ok) {
40+
if (c != kInvalidHandle) {
41+
::closesocket(c);
42+
}
43+
return false;
44+
}
45+
rawSetNonBlocking(out[0], true);
46+
rawSetNonBlocking(out[1], true);
47+
return true;
48+
}
49+
1450
int rawClose(handle_t handle) {
1551
return ::closesocket(handle);
1652
}

‎src/net/wakeup.cpp‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#include "net/wakeup.h"
2+
3+
#include "net/detail_raw.h"
4+
5+
#include <cstdint>
6+
7+
namespace net {
8+
9+
Wakeup Wakeup::create() {
10+
detail::handle_t pair[2] = {detail::kInvalidHandle, detail::kInvalidHandle};
11+
if (!detail::rawSocketPair(pair)) {
12+
return Wakeup{};
13+
}
14+
return Wakeup{Socket{pair[0]}, Socket{pair[1]}};
15+
}
16+
17+
void Wakeup::notify() {
18+
if (!out_.valid()) {
19+
return;
20+
}
21+
const char byte = 'w';
22+
out_.sendSome(&byte, 1); // WouldBlock simply means a wake-up is already pending
23+
}
24+
25+
void Wakeup::drain() {
26+
if (!in_.valid()) {
27+
return;
28+
}
29+
char buf[64];
30+
while (in_.recvSome(buf, sizeof(buf)) == static_cast<int>(sizeof(buf))) {
31+
}
32+
}
33+
34+
} // namespace net

‎src/net/wakeup.h‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#pragma once
2+
3+
#include "net/socket.h"
4+
5+
#include <utility>
6+
7+
namespace net {
8+
9+
// Cross-thread wake-up for a poll loop: notify() (any thread) makes the thread
10+
// blocked in pollSockets() return by making readHandle() readable. POSIX uses a
11+
// socketpair; Windows uses a loopback pair — it is just a socket either way.
12+
class Wakeup {
13+
public:
14+
Wakeup() = default; // invalid on failure (lastError() has the reason)
15+
static Wakeup create();
16+
17+
detail::handle_t readHandle() const { return in_.handle(); }
18+
bool valid() const { return in_.valid() && out_.valid(); }
19+
20+
void notify(); // fire-and-forget; coalesces
21+
void drain(); // consume pending wake-ups
22+
23+
private:
24+
Wakeup(Socket in, Socket out) : in_(std::move(in)), out_(std::move(out)) {}
25+
Socket in_;
26+
Socket out_;
27+
};
28+
29+
} // namespace net

0 commit comments

Comments
 (0)