Skip to content

feat: fail only the request when a handler hits a runtime error - #1399

Merged
kacy merged 6 commits into
mainfrom
feat/failure-isolation
Nov 15, 2026
Merged

kacy merged 6 commits into
mainfrom
feat/failure-isolation

Conversation

@kacy

@kacy kacy commented Nov 14, 2026

Copy link
Copy Markdown
Owner

A runtime error inside an http handler (std.net.http, std.web, the http/2 server) now fails that request with a 500, and one inside a grpc method fails the call with INTERNAL. The server keeps serving. Everywhere else the process still exits with the same message and status 1. docs/failure_isolation.md has the design, the alternatives and the semantics.

The handler call runs behind a catch frame on the task's own stack (cranelift/runtime/src/failure.rs): a short naked routine for x86_64 and aarch64 saves the callee-saved registers, and a runtime error that finds a frame jumps back to it with its message. The runtime error sites in scope (index, key, value on none, division, substring split, and any pith_runtime_error code other than 3 to 5, so integer overflow is covered when it lands) now go through failure::fail and own nothing that needs a drop when they jump. Pith frames that are skipped leak what they held and their defers do not run; the bound is in the doc. Mutexes locked inside the call are recorded and unlocked on failure. The new builtin is isolate_call(fn() -> Int) -> String. A nested coroutine per call was measured too: about 8 ns per call against under 1 ns for the catch frame, and it would need every block site to forward parks through the inner coroutine.

The http/1.1 answer is a 500 with Connection: close; http/2 answers a 500 on the stream (a streaming handler gets RST_STREAM INTERNAL_ERROR); grpc answers status 13 with the message internal error. stderr gets pith runtime error in GET /path: <message>.

Tests: tests/cases/test_web_handler_failure.pith (each error kind over a real socket, a second request after each, pipelining behind a failure, and 20 failures while holding a Mutex), tests/cases/test_grpc_method_failure.pith (unary and streaming, then a good call on the same connection), three abort goldens showing that errors outside a handler still exit 1 with the same message (main after a caught failure, a spawned task, the os-thread backend), and tests/leaks/leak_isolated_failure.pith, which holds the machinery itself flat.

Benchmark: bench/http_sweep.sh, pith arm only, server on cpus 0-7,16-23 and wrk on 8-15,24-31, 10 s runs, four rounds with the arm order rotated each round. Other builds were running on the machine (load average 20 to 33), so single runs vary by 3% to 5%. Medians in req/s:

conns before after change before range after range
32 1,107,500 1,122,600 +1.4% 1.05M to 1.18M 0.78M to 1.15M
128 1,777,600 1,760,400 -1.0% 1.74M to 1.81M 1.74M to 1.79M
256 1,855,400 1,824,400 -1.7% 1.81M to 1.88M 1.79M to 1.88M
1024 1,896,800 1,888,800 -0.4% 1.87M to 1.94M 1.88M to 1.89M

The first version carried the response through a threadlocal, which marks it shared and made its counts atomic; that cost 4% to 7% at 256 and 1024 connections and is fixed in the last commit.

Not covered in phase 1: errors in tasks a handler spawns, stack overflow, allocation failure, streamed body producers and connection takers (they run after the handler returns), and semaphores or channels a failed handler was using.

@kacy
kacy merged commit d9613f1 into main Nov 15, 2026
4 checks passed
@kacy
kacy deleted the feat/failure-isolation branch November 15, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant