Skip to content

Commit d988462

Browse files
committed
feat: single-threaded poll reactor with graceful shutdown and per-OS signals
server::Server reactor (accept batch, deadline table -> 408, POLLOUT backpressure, drain-then-drop stop with grace cap) + server::Connection (socket + RequestParser + out buffer; protocol errors answered once best-effort then dropped). Connection stays reactor-owned; no shared state yet. app::ShutdownSignals: Linux self-pipe (pipe2, instant poll wake); Windows console-control handler sets flag and blocks until notifyStopped (a returning handler means exit-now), reactor pollCap 100ms. CMake TSAN option; CI linux-tsan job (native runner: kernel containers cannot set vm.mmap_rnd_bits TSan needs). SIGINT graceful-stop smoke added to the container gate. Traps: TSan FATALs on kernel 6.x containers (unexpected memory mapping) until vm.mmap_rnd_bits=28; accepted-socket index bug — new conns join next poll, not the live fd set; test race where stop beat accept (backlog RST != EOF) fixed by serving one request first.
1 parent 8668203 commit d988462

20 files changed

Lines changed: 961 additions & 16 deletions

‎.github/workflows/ci.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,22 @@ jobs:
1717
docker run --rm -v "$GITHUB_WORKSPACE:/src" -w /src http-cpp-asan \
1818
bash tools/asan-inside.sh
1919
20+
# Concurrency gate: reactor tests under ThreadSanitizer (Linux only; no TSan
21+
# runtime on the Windows MinGW toolchain). Native runner, not a container:
22+
# TSan needs vm.mmap_rnd_bits=28, which kernel 6.x containers cannot set.
23+
linux-tsan:
24+
runs-on: ubuntu-latest
25+
steps:
26+
- uses: actions/checkout@v4
27+
- name: Widen TSan address space
28+
run: sudo sysctl -w vm.mmap_rnd_bits=28
29+
- name: Install toolchain
30+
run: |
31+
sudo apt-get update
32+
sudo apt-get install -y --no-install-recommends g++ cmake ninja-build netcat-openbsd
33+
- name: Build and test under TSan
34+
run: SAN=thread bash tools/asan-inside.sh
35+
2036
# Windows native gate: MinGW gcc, warnings-as-errors, ctest.
2137
windows-native:
2238
runs-on: windows-latest

‎CMakeLists.txt‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ set(CMAKE_CXX_EXTENSIONS OFF)
77
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
88

99
option(HTTP_SERVER_ENABLE_SANITIZERS "Build with address and undefined sanitizers" OFF)
10+
option(HTTP_SERVER_ENABLE_TSAN "Build with ThreadSanitizer (Linux; reactor tests)" OFF)
1011

1112
include(CheckCXXSourceCompiles)
1213
set(CMAKE_REQUIRED_FLAGS "-fsanitize=address,undefined")
@@ -26,6 +27,14 @@ if(HTTP_SERVER_ENABLE_SANITIZERS)
2627
add_link_options(-fsanitize=address,undefined)
2728
endif()
2829

30+
if(HTTP_SERVER_ENABLE_TSAN)
31+
if(HTTP_SERVER_ENABLE_SANITIZERS)
32+
message(FATAL_ERROR "TSAN and ASAN/UBSAN sanitizer options are mutually exclusive.")
33+
endif()
34+
add_compile_options(-fsanitize=thread -fno-omit-frame-pointer)
35+
add_link_options(-fsanitize=thread)
36+
endif()
37+
2938
set(HTTP_WARNINGS -Wall -Wextra -Wpedantic -Wshadow -Wconversion -Werror)
3039

3140
# http layer is platform-neutral.
@@ -59,8 +68,21 @@ if(WIN32)
5968
endif()
6069
target_compile_options(net PRIVATE ${HTTP_WARNINGS})
6170

62-
add_executable(http_server src/main.cpp)
63-
target_link_libraries(http_server PRIVATE net http)
71+
# server: reactor + connection, platform-neutral; signals implementation is per OS.
72+
add_library(server STATIC
73+
src/server/server.cpp
74+
src/server/connection.cpp)
75+
target_link_libraries(server PUBLIC net http)
76+
target_compile_options(server PRIVATE ${HTTP_WARNINGS})
77+
78+
if(WIN32)
79+
set(app_signals_src src/app/signals_win.cpp)
80+
else()
81+
set(app_signals_src src/app/signals_posix.cpp)
82+
endif()
83+
84+
add_executable(http_server src/main.cpp ${app_signals_src})
85+
target_link_libraries(http_server PRIVATE net http server)
6486
target_compile_options(http_server PRIVATE ${HTTP_WARNINGS})
6587

6688
enable_testing()
@@ -69,11 +91,12 @@ add_executable(unit_tests
6991
tests/test_main.cpp
7092
tests/test_socket.cpp
7193
tests/test_request.cpp
72-
tests/test_response.cpp)
94+
tests/test_response.cpp
95+
tests/test_server.cpp)
7396
target_include_directories(unit_tests SYSTEM PRIVATE external/doctest)
7497
# doctest is a vendored header: warnings from it are not ours to fix,
7598
# hence SYSTEM above; our own test code still builds warning-clean.
7699
target_compile_options(unit_tests PRIVATE -Wall -Wextra -Werror)
77-
target_link_libraries(unit_tests PRIVATE net http)
100+
target_link_libraries(unit_tests PRIVATE net http server)
78101

79102
add_test(NAME unit_tests COMMAND unit_tests)

‎PLAN.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,14 +173,15 @@ phase/step/milestone wording, ever.
173173
| ✓ | ~~`feature/response-writer`~~ | Response, headers, status, chunk encoder | merged |
174174
| ✓ | ~~`feature/posix-socket-layer`~~ | The seam: portable `net/` API + `raw_win`/`raw_posix` backends; `accept4` + `MSG_NOSIGNAL` + `EINTR`; cross-platform socket tests; Docker gate builds whole Linux tree | merged — both gates green |
175175
| C | `feature/ci-matrix` | Actions matrix: Linux job = sanitizer container (`tools/asan-inside.sh`), Windows = native MinGW + ctest | merged — PR #5 gated itself green on first run |
176-
| R | `feature/reactor-and-connection` | poll loop, connection state machine, hand-off inbox, timeouts, buffering, signals (win + posix), graceful shutdown — all single-source except `app/signals_*` | integration: in-memory 200, keep-alive, 408 idle, Ctrl-C/SIGTERM exit 0; **TSan job added and green** |
176+
| R | `feature/reactor-and-connection` | single-threaded poll reactor: connection state machine, timeouts (408), POLLOUT backpressure, signals (self-pipe / console ctrl), graceful drain | merged — 200/keep-alive/408/400/stop integration tests; ASan + TSan green; SIGINT-exit-0 smoke in the Linux gate |
177+
| T | `feature/reactor-threading` | acceptor thread + K reactors + hand-off inbox; `Connection` stays owned by one reactor | TSan green on concurrent tests; RSS stays flat |
177178
| S | `feature/static-files` | root mapping, traversal guard, MIME, index, 404/405, error pages (`std::filesystem`) | status-by-status integration tests; traversal rejected on both OSes |
178179
| G | `feature/caching-and-range` | ETag, Last-Modified, If-None-Match, Range/206/416 | 304 + 206 tests incl. multi-range rejection |
179180
| H | `feature/hardening` | pipelining, slow-body limit, max-body/max-heads, request-smuggling corpus, soak | pipelined + slow-client tests; ASan clean; 60 s soak flat RSS on Linux |
180181
| D | `feature/docs-and-benchmark` | README (build/run per OS, limits), load script, numbers | documented; req/s + RSS recorded on both platforms |
181182

182183
Then `release/0.1.0` → `develop` + PR to `main` (approval required) → tag `v0.1.0`.
183-
C is next; R needs C.
184+
R merged single-threaded; S (static files) stacks on it — T (threading) can land after S.
184185

185186
## 8. Git workflow (gitflow)
186187

‎docs/ARCHITECTURE.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,20 @@ until then.
2121
push/incremental, enforces head/body/count limits, and rejects bare-LF endings,
2222
obsolete folding, duplicate `Content-Length`, `Content-Length`+`Transfer-Encoding`,
2323
non-token names/methods, unknown versions. Pending: URL decode/normalise, MIME.
24-
- `app/` — signal handling (pending, win + posix)
25-
- `server/` — listener, connection, static file handler (pending)
24+
- `server/` — single-threaded poll reactor (`Server::run`): accept batch, per-connection
25+
deadlines (408), backpressure via POLLOUT, graceful stop (drain queued responses,
26+
grace cap, then drop). `Connection` = socket + `RequestParser` + output buffer;
27+
protocol errors are answered once (best effort) then the connection drops —
28+
smuggling candidates never share a stream. `ShutdownSignal` is the pluggable stop
29+
trigger; tests drive it directly.
30+
- `app/` — shutdown signals, one paragraph precisely because behaviour differs:
31+
Linux uses a self-pipe (`pipe2`, handler writes one byte → poll wakes instantly,
32+
SIGINT drains and exits 0 — smoke-tested in the container gate); Windows cannot
33+
union a waitable event with WSAPoll, so the console-control handler sets a flag
34+
and blocks until the loop ends (`notifyStopped()`), with the reactor capped at a
35+
100 ms poll while idle. A returning handler would mean "exit now" — hence block.
2636

27-
Gates: `tools/ck.ps1` (Windows) and `tools/ck-asan-docker.sh` (whole tree, ASan+UBSan,
28-
Ubuntu 24.04 container). The socket test file is the same source on both platforms.
37+
Gates: `tools/ck.ps1` (Windows), `tools/ck-asan-docker.sh` (whole tree, ASan+UBSan,
38+
Ubuntu 24.04 container, plus SIGINT graceful-stop smoke), and TSan (`SAN=thread`,
39+
Linux runner — kernel containers can't set the `vm.mmap_rnd_bits` TSan needs).
40+
The socket and server test files are the same source on both platforms.

‎src/app/signals.h‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#pragma once
2+
3+
#include "server/shutdown.h"
4+
5+
#include <memory>
6+
7+
namespace app {
8+
9+
// Process-level SIGINT/SIGTERM (POSIX) or console-control (Windows) shutdown.
10+
// Installs OS handlers in the constructor — one instance per process.
11+
//
12+
// POSIX: self-pipe — the handler writes a byte, the reactor's poll wakes at once.
13+
// Windows: the handler sets a flag and blocks (a returning handler means "handled,
14+
// exit now" would kill the graceful drain); the reactor notices within pollCap().
15+
class ShutdownSignals final : public server::ShutdownSignal {
16+
public:
17+
struct Impl; // OS state, defined in the per-OS .cpp
18+
19+
ShutdownSignals();
20+
~ShutdownSignals() override;
21+
22+
bool stopRequested() const override;
23+
void preparePoll(net::PollSet& set) override;
24+
void afterPoll(const net::PollSet& set) override;
25+
std::chrono::milliseconds pollCap() const override;
26+
void notifyStopped() override;
27+
28+
private:
29+
std::unique_ptr<Impl> impl_;
30+
};
31+
32+
} // namespace app

‎src/app/signals_posix.cpp‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#include "app/signals.h"
2+
3+
#include "net/poll.h"
4+
5+
#include <csignal>
6+
#include <fcntl.h>
7+
#include <poll.h> // for the pollfd revents bits (net/poll.h pulls it too)
8+
#include <unistd.h>
9+
10+
#include <atomic>
11+
12+
struct app::ShutdownSignals::Impl {
13+
int pipe[2] = {-1, -1};
14+
};
15+
16+
namespace {
17+
app::ShutdownSignals::Impl* g_impl = nullptr;
18+
std::atomic_bool g_stop{false};
19+
20+
// Async-signal-safe: write() only. The atomic store is a fallback for a failed
21+
// self-pipe; atomic_bool is lock-free on every platform we target.
22+
void onSignal(int) {
23+
g_stop.store(true);
24+
if (g_impl != nullptr && g_impl->pipe[1] >= 0) {
25+
const char byte = 'x';
26+
const ssize_t ignored = ::write(g_impl->pipe[1], &byte, 1);
27+
(void)ignored;
28+
}
29+
}
30+
} // namespace
31+
32+
namespace app {
33+
34+
ShutdownSignals::ShutdownSignals() : impl_(std::make_unique<Impl>()) {
35+
if (::pipe2(impl_->pipe, O_CLOEXEC | O_NONBLOCK) != 0) {
36+
impl_->pipe[0] = impl_->pipe[1] = -1;
37+
return; // degraded: reactor still re-checks after every poll
38+
}
39+
g_impl = impl_.get();
40+
struct sigaction sa {};
41+
sa.sa_handler = onSignal;
42+
sigemptyset(&sa.sa_mask);
43+
sa.sa_flags = SA_RESTART; // send/poll EINTR handling exists, but avoid churn
44+
::sigaction(SIGINT, &sa, nullptr);
45+
::sigaction(SIGTERM, &sa, nullptr);
46+
::signal(SIGPIPE, SIG_IGN); // belt over MSG_NOSIGNAL's trousers
47+
}
48+
49+
ShutdownSignals::~ShutdownSignals() {
50+
if (impl_->pipe[0] >= 0) {
51+
::close(impl_->pipe[0]);
52+
::close(impl_->pipe[1]);
53+
}
54+
}
55+
56+
bool ShutdownSignals::stopRequested() const { return g_stop.load(); }
57+
58+
void ShutdownSignals::preparePoll(net::PollSet& set) {
59+
if (impl_->pipe[0] >= 0) {
60+
set.add(impl_->pipe[0], net::PollIn);
61+
}
62+
}
63+
64+
void ShutdownSignals::afterPoll(const net::PollSet& set) {
65+
for (std::size_t i = 0; i < set.size(); ++i) {
66+
if (set.socket(i) == impl_->pipe[0] && (set.revents(i) & net::PollIn)) {
67+
char drain[64];
68+
while (::read(impl_->pipe[0], drain, sizeof(drain)) > 0) {
69+
}
70+
g_stop.store(true);
71+
return;
72+
}
73+
}
74+
}
75+
76+
std::chrono::milliseconds ShutdownSignals::pollCap() const {
77+
return std::chrono::milliseconds(0); // self-pipe wakes poll immediately
78+
}
79+
80+
void ShutdownSignals::notifyStopped() {}
81+
82+
} // namespace app

‎src/app/signals_win.cpp‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#include "app/signals.h"
2+
3+
#include "net/poll.h"
4+
5+
#ifndef WIN32_LEAN_AND_MEAN
6+
#define WIN32_LEAN_AND_MEAN
7+
#endif
8+
#include <windows.h>
9+
10+
#include <atomic>
11+
12+
struct app::ShutdownSignals::Impl {};
13+
14+
namespace {
15+
std::atomic_bool g_stop{false};
16+
HANDLE g_done = nullptr; // set once the reactor loop has ended
17+
18+
BOOL WINAPI onConsoleEvent(DWORD type) {
19+
if (type != CTRL_C_EVENT && type != CTRL_BREAK_EVENT && type != CTRL_CLOSE_EVENT &&
20+
type != CTRL_LOGOFF_EVENT && type != CTRL_SHUTDOWN_EVENT) {
21+
return FALSE; // not ours; next handler / default
22+
}
23+
g_stop.store(true);
24+
// Returning TRUE means "handled, continue" — but returning at all races the
25+
// 5s close grace, so block until the reactor is done and let process exit
26+
// take us out.
27+
if (g_done != nullptr) {
28+
::WaitForSingleObject(g_done, 4000);
29+
} else {
30+
::Sleep(4000);
31+
}
32+
return TRUE;
33+
}
34+
} // namespace
35+
36+
namespace app {
37+
38+
ShutdownSignals::ShutdownSignals() : impl_(std::make_unique<Impl>()) {
39+
g_done = ::CreateEventW(nullptr, TRUE, FALSE, nullptr);
40+
::SetConsoleCtrlHandler(&onConsoleEvent, TRUE);
41+
}
42+
43+
ShutdownSignals::~ShutdownSignals() {
44+
::SetConsoleCtrlHandler(&onConsoleEvent, FALSE);
45+
if (g_done != nullptr) {
46+
::CloseHandle(g_done);
47+
g_done = nullptr;
48+
}
49+
}
50+
51+
bool ShutdownSignals::stopRequested() const { return g_stop.load(); }
52+
53+
void ShutdownSignals::preparePoll(net::PollSet& set) { (void)set; }
54+
void ShutdownSignals::afterPoll(const net::PollSet& set) { (void)set; }
55+
56+
std::chrono::milliseconds ShutdownSignals::pollCap() const {
57+
return std::chrono::milliseconds(100); // no waitable socket+event union on WSAPoll
58+
}
59+
60+
void ShutdownSignals::notifyStopped() {
61+
if (g_done != nullptr) {
62+
::SetEvent(g_done);
63+
}
64+
}
65+
66+
} // namespace app

‎src/main.cpp‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,36 @@
1-
// Placeholder entry point. Replaced as the socket and server layers land.
2-
int main() { return 0; }
1+
#include "app/signals.h"
2+
#include "http/request.h"
3+
#include "net/context.h"
4+
#include "server/server.h"
5+
6+
#include <cstdio>
7+
#include <cstdlib>
8+
#include <exception>
9+
10+
int main(int argc, char** argv) {
11+
int port = 8080;
12+
if (argc > 1) {
13+
port = std::atoi(argv[1]);
14+
}
15+
if (port < 1 || port > 65535) {
16+
std::fprintf(stderr, "usage: http_server [port]\n");
17+
return 2;
18+
}
19+
20+
try {
21+
net::Context ctx;
22+
app::ShutdownSignals signals;
23+
server::Server server(static_cast<std::uint16_t>(port),
24+
[](const http::Request&) {
25+
return std::string("Hello from http-server-cpp\n");
26+
});
27+
std::printf("listening on http://127.0.0.1:%d/ (stop with Ctrl-C)\n", port);
28+
std::fflush(stdout);
29+
server.run(signals);
30+
std::puts("stopped");
31+
} catch (const std::exception& e) {
32+
std::fprintf(stderr, "error: %s\n", e.what());
33+
return 1;
34+
}
35+
return 0;
36+
}

‎src/net/poll.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@
88

99
namespace net {
1010

11+
// Portable poll event bits (POLL* spellings differ across OS headers).
12+
inline constexpr short PollIn = POLLIN;
13+
inline constexpr short PollOut = POLLOUT;
14+
inline constexpr short PollErr = POLLERR;
15+
inline constexpr short PollHup = POLLHUP;
16+
inline constexpr short PollNval = POLLNVAL;
17+
1118
// Thin poll()/WSAPoll fd set. Only the first pollSockets() return value of entries
1219
// carry meaningful revents (Windows leaves the rest untouched).
1320
class PollSet {

‎src/net/socket.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,9 @@ class Socket {
4646
// WouldBlock failure returns false; check lastError().kind. The reactor never
4747
// uses it; it buffers per-connection instead.
4848
bool sendAll(const char* data, std::size_t len);
49+
// What the kernel accepts right now: >0 bytes, -1 error (WouldBlock possible on
50+
// a non-blocking socket — check lastError().kind). The reactor's write path.
51+
int sendSome(const char* data, std::size_t len);
4952
// >0 bytes read, 0 peer closed, -1 error (WouldBlock possible on non-blocking).
5053
int recvSome(void* buffer, std::size_t capacity);
5154

0 commit comments

Comments
 (0)