diff --git a/README.md b/README.md index 868b660..ac5a6d5 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,61 @@ No exception in the package should crash the app or leave a call hanging. Failur `NativeHttpClient.create` throws a `PlatformException` if the platform can't create a client, for example when Cronet's native library fails to load. +### Read-idle timeout + +A client can fail each of its calls once the call has received nothing from the server for a set time. Apart from +deadlines, this is the only way either client notices a dead HTTP/2 connection (see "Dead connections"): + +```dart +NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); +``` + +It applies to every call on the client. For per-call bounds use deadlines (connect-dart's `TimeoutSignal` or +`DeadlineSignal`), which can also move the client off a dead connection (see "Calls that run out of time"). Streams +that go quiet for long need a client without a read-idle timeout, or periodic messages. + +- **What it measures:** time since the call last received anything: headers, a body chunk or trailers. The timer starts + when the call does, so it also bounds the wait for the response headers. A unary call the server holds longer than + the timeout fails, and so does a stream that goes quiet for longer. +- **iOS:** `URLRequest.timeoutInterval`. URLSession treats a request's 60 s as unset and uses the session's value + instead, so the bridge sends 60.001 s for a 60 s timeout. +- **Android:** a timer in the bridge. +- **The failure:** `unavailable`, with a `NativeHttpException` whose `platformCode` is `NSURLErrorTimedOut` (-1001) on + iOS or `net::ERR_TIMED_OUT` (-7) on Android. On Android a timeout before `onStreamReady` is `notSent`, as with any + other failure there. +- **The connection:** see "Calls that run out of time" below. + +### Calls that run out of time + +A call that hits its read-idle timeout, or its deadline with `evictOnDeadline`, may be stuck on a dead connection. +The client keeps new calls off such a connection, so they don't wait on it too. + +```dart +NativeHttpConfig( + readIdleTimeout: const Duration(seconds: 30), // optional + deadlineStep: const Duration(seconds: 1), + evictOnDeadline: true, +); +``` + +- **Read-idle timeout:** the connection counts as dead if nothing has arrived from the call's origin (host and port), + on any call, since the call last received anything. If something has, the connection is alive and only that call + was slow, so nothing changes. +- **Deadline, with `evictOnDeadline`:** the connection counts as dead under the same check as a read-idle timeout. + Both options are off by default; use them together: + - **`deadlineStep`** takes a step off the deadline each hop passes on. connect-dart sends the call's deadline in + `connect-timeout-ms`, and the client sends the server that much less, or half the deadline when that is less, so + the server never gets zero. A reachable server then times the call out first and answers `deadline_exceeded` + before the client's own deadline fires. So when the client's deadline fires, the server's answer never came. + Servers that call further services should take their own step off the deadline they pass on. + - **Without a step,** the server enforces the same deadline, but its clock starts later. On a healthy connection the + client's deadline usually fires first, and with `evictOnDeadline` alone the connection would be evicted for + nothing. +- **A dead connection is evicted.** Neither stack can close a single connection, so the client moves its new calls to + a fresh `URLSession` or `CronetEngine`, which dials its own connections. Calls already on the old one are not + canceled. They finish there, and it is released after the last one ends. Several calls that run out of time on the + same connection evict it once. + ## Server and infrastructure requirements The clients only reach HTTP/3, and only keep bidi working, if the server side meets these requirements. Each was @@ -73,9 +128,11 @@ found or verified with the tests in this package. ([target group protocol version](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html)). - **Listener rules must route the Connect content types above to that target group.** A rule that matches only `application/grpc` sends Connect requests elsewhere. -- **No client-side dead-connection detection.** Neither stack sends HTTP/2 pings the caller controls, and server - pings stop at any load balancer that terminates HTTP/2. Idle timeouts on the path must exceed a stream's longest - quiet period, or the stream must carry periodic messages. See "QUIC idle timeout vs HTTP/2 keepalive pings". +- **No dead-connection detection apart from the read-idle timeout.** Neither stack sends HTTP/2 pings the caller + controls, and server pings stop at any load balancer that terminates HTTP/2. A read-idle timeout notices a dead + connection, but only if it is shorter than a stream's quiet periods. Idle timeouts on the path must exceed a + stream's longest quiet period, or the stream must carry periodic messages. See "QUIC idle timeout vs HTTP/2 + keepalive pings". **The HTTP/3 endpoint** @@ -111,7 +168,7 @@ found or verified with the tests in this package. | Certificates | System trust store | System trust store; QUIC only with certificates that chain to a public root | | Negotiated protocol | Not exposed (`NativeHttpResponseInfo.protocol` is `null`) | `h2`, `quic/1+spdy/3` (HTTP/3) or `http/1.1` | | QUIC idle timeout | No setting; the server's advertised value applies | `NativeHttpConfig.quicIdleTimeout`, in whole seconds | -| Request idle timer | A task that receives nothing for a day fails; a quiet stream otherwise stays open | None | +| Read-idle timeout | `NativeHttpConfig.readIdleTimeout`, through `URLRequest.timeoutInterval`. Without one, a task that receives nothing for a day fails | `NativeHttpConfig.readIdleTimeout`, through a timer in the bridge. None without one | | Redirects | Followed only for requests without a body; otherwise the 3xx response is returned, since a streamed body can't be resent | Not tested | ### Dead connections @@ -136,9 +193,16 @@ The integration tests measure both: Clients may enforce a floor on the peer's value. quic-go's client raises anything under 5 s to 5 s (`protocol.MinRemoteIdleTimeout`), so measure any other client before relying on a short server value. -This does nothing on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP -connection that dies without the interface going down goes unnoticed until a call's deadline or an -application-level heartbeat. +None of this works on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP +connection that dies without the interface going down goes unnoticed until a call on it runs out of time: a read-idle +timeout, or a deadline with `evictOnDeadline` (see "Calls that run out of time"). The integration tests check this: + +- Over a frozen HTTP/2 connection nothing arrives for 3 s, and a new call on the same client waits on it until its + deadline. With `evictOnDeadline`, the call after that goes out on a new connection. +- With a read-idle timeout, the call fails within the timeout, and the client's next call goes out on a new + connection. +- With `deadlineStep`, the server answers a deadline with `deadline_exceeded` a step before the client's own + deadline, and the client stays on the same connection. ### QUIC idle timeout vs HTTP/2 keepalive pings @@ -151,7 +215,7 @@ not over HTTP/2. | Who sets it | Both advertise a value; the connection uses the smaller (RFC 9000 §10.1) | Each side sets its own pings | | Can a server setting make the client notice? | Yes. A short server value becomes the client's timeout, and the server's pings stop arriving once the path dies. | No. A server that detects a dead connection can't tell the client, because nothing gets through. | | Through a load balancer | NLB QUIC passthrough is end to end, so the client sees the server's pings | An ALB ends HTTP/2 on each side, so the server's pings stop at the ALB. AWS doesn't document the ALB pinging clients, and nothing here relies on it. | -| Client control here | Android: `quicIdleTimeout`. iOS: none. | None. Neither URLSession nor Cronet exposes HTTP/2 ping settings. | +| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout, and `evictOnDeadline`. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout, and `evictOnDeadline`. | | What a client that sends its own pings gets, for comparison | — | `package:http2` with 3 s pings and a 3 s timeout notices in about 6 s | So for servers: @@ -161,8 +225,8 @@ So for servers: - Too long an idle timeout delays detection. - Too few pings closes quiet connections that are healthy. - **HTTP/2:** server pings, such as `http.HTTP2Config.SendPingTimeout` in Go, only clean up the server's side. - - On the client, a silently dead connection lasts until a call deadline, an application-level heartbeat, or TCP's - own retransmission timeout, which takes minutes. + - On the client, a silently dead connection lasts until a call deadline, a read-idle timeout, an + application-level heartbeat, or TCP's own retransmission timeout, which takes minutes. - Every idle timeout on the path, such as a load balancer's, must exceed the longest quiet period of a stream, or the stream must carry periodic messages. @@ -197,9 +261,14 @@ Neither platform finishes a stream on its own while the client still has its req - **`test_server/`:** a Go server for `connectrpc.eliza.v1.ElizaService` over HTTP/2 and HTTP/3, scripted for tests (see `eliza.go`). - `-max-idle` and `-keepalive` set its QUIC idle timeout and ping interval. - - `-admin-addr` serves `POST /freeze?for=10s` and `POST /unfreeze`, which drop all its UDP traffic so a test can - simulate a dead HTTP/3 path. - - Its own tests (`go test ./...`) check it over HTTP/3, including the idle timeout, pings and freezing. + - `-admin-addr` serves an endpoint that simulates dead paths: + - `POST /freeze?for=10s` drops all its UDP traffic, so every HTTP/3 connection. + - `POST /freeze?for=10s&path=tcp` black-holes the TCP connections open at that moment, so the HTTP/2 connections + clients already have. It drops what they send and what the server writes, then closes them. Connections opened + later work, so a client that dials again gets through. + - `POST /unfreeze` ends both. + - Its own tests (`go test ./...`) check it over HTTP/3, including the idle timeout, pings and freezing, and check + the TCP freeze over HTTP/2. - It creates a CA in `test_server/certs/` on first start and reuses it, so a device only has to trust it once. - With `QLOGDIR` set, it writes a qlog trace of each QUIC connection there, which shows the server's side of a handshake. @@ -225,13 +294,26 @@ Neither platform finishes a stream on its own while the client still has its req The script starts the server with a 3 s idle timeout and 1 s pings. Over HTTP/3 (iOS) the tests also check two things: a quiet stream outlives the idle timeout, and a frozen path fails an open stream within it. + + The read-idle timeout is tested over HTTP/2 on both platforms. On iOS each of these cases freezes the server's UDP + first, so URLSession falls back to TCP as it would where UDP is blocked. The server reports each call's client + address in `X-Remote-Addr`, which shows whether two calls shared a connection. The cases check five things: + - A quiet stream fails at its timeout. + - A stream that keeps receiving outlives its timeout. + - A frozen connection draws no error, and new calls go to it until one hits its deadline. With `evictOnDeadline`, + the call after that is on a new connection. + - With a read-idle timeout, the call fails and the client's next call is on a new connection. + - With `deadlineStep`, the server answers a deadline a step before the client's, and the client stays on the same + connection. - **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's bridge (`URLSessionBridge`, `CronetBridge`) has no Flutter dependency, and the tests call it directly, for behavior Dart can't reach: - **iOS (`ios/connect_native_http/Tests`, XCTest):** a write stuck on a full buffer returns once the call is canceled or its connection fails; a redirect of a request with a body is returned, not followed; shutting a - session down lets its calls finish and refuses new ones. The script runs them on the simulator. For a quick loop - on the Mac, against a running test server: + session down lets its calls finish and refuses new ones; a read-idle timeout fails a quiet call with + `NSURLErrorTimedOut`, and the engine's next call works; a call canceled at its deadline ends at once, and the + engine's next call works. The script runs them on the simulator. For a quick loop on the Mac, against a running + test server: ```bash cd ios/connect_native_http && CONNECT_NATIVE_HTTP_CORE_ONLY=1 SERVER_URL=https://localhost:8443 \ @@ -240,7 +322,9 @@ Neither platform finishes a stream on its own while the client still has its req - **Android (`android/src/androidTest`, instrumented):** writes sent before Cronet is ready go out once it is; a write after the server finished the call is dropped instead of crashing; shutting an engine down lets its calls - finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped. + finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped; a + read-idle timeout fails a quiet call with -7, once, and the engine's next call works; data arriving resets the + timer; a call canceled at its deadline ends at once, and the engine's next call works. - Both platforms' suites also check that malformed method calls get an error reply and leave the bridge working. - **`example/integration_test/public_server_test.dart`:** HTTP/3 on Android against a server with a public certificate. It also checks that `quicIdleTimeout` ends a quiet connection, against a control without it. It is diff --git a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt index af1789b..3abd162 100644 --- a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt +++ b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt @@ -91,6 +91,56 @@ class CronetBridgeTest { assertTrue(received(1).contains("say: hi")) } + @Test + fun aReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() { + val engine = createEngine() + // The timer also runs while the connection is set up, which through the emulator once took longer than the + // timeout (the TLS handshake was cut off). A call first leaves a connection for the timed one to reuse. + assertSayWorks(engine, 10) + open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 500) + await(1, "data") + val start = System.nanoTime() + val outcome = awaitOutcome(1) + val elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) + assertEquals("failed", outcome["type"]) + assertEquals(-7, outcome["internalErrorCode"]) + assertEquals(true, outcome["streamReady"]) + assertTrue("a 500 ms read-idle timeout took $elapsedMs ms", elapsedMs < 2000) + // Cronet's own cancel follows, and changes nothing. + Thread.sleep(200) + assertEquals(listOf(outcome), events.filter { it["callId"] == 1 && it["type"] in OUTCOMES } + drained(1)) + + // New calls go to a fresh CronetEngine. + assertSayWorks(engine, 2) + } + + @Test + fun receivingResetsTheReadIdleTimer() { + val engine = createEngine() + assertSayWorks(engine, 10) + open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 1000) + await(1, "data") + // Twice the timeout in all, 600 ms between replies. + for (i in 0 until 3) { + Thread.sleep(600) + invoke("write", mapOf("callId" to 1, "data" to envelope(sentence("ping $i")))) + awaitReceived(1, "echo: ping $i") + } + invoke("write", mapOf("callId" to 1, "data" to envelope(sentence("bye")))) + assertEquals("done", awaitOutcome(1)["type"]) + } + + // A call that hit its deadline is canceled at once, and moves the engine's new calls to a fresh CronetEngine. + @Test + fun aDeadlineCancelEndsTheCallAndTheEngineCarriesOn() { + val engine = createEngine() + open(engine, 1, CONVERSE, "application/connect+proto") + await(1, "data") + invoke("cancel", mapOf("callId" to 1, "evictConnection" to true)) + assertEquals("canceled", awaitOutcome(1)["type"]) + assertSayWorks(engine, 2) + } + @Test fun malformedArgumentsAreRefusedInsteadOfCrashing() { val engine = createEngine() @@ -101,6 +151,9 @@ class CronetBridgeTest { "end" to mapOf("callId" to null), "cancel" to emptyMap(), "shutdownEngine" to mapOf("engineId" to 1.5), + "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to 0), + "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to "soon"), + "cancel" to mapOf("callId" to 1, "evictConnection" to "yes"), ) for ((method, arguments) in malformed) { val reply = reply(method, arguments) @@ -139,17 +192,27 @@ class CronetBridgeTest { path: String, contentType: String, extraHeaders: List> = emptyList(), + readIdleTimeoutMs: Int? = null, ) { - invoke("open", mapOf( - "engineId" to engine, - "callId" to callId, - "url" to serverUrl + path, - "method" to "POST", - "headers" to listOf(listOf("content-type", contentType), listOf("connect-protocol-version", "1")) + - extraHeaders, - )) + invoke("open", openArguments(engine, callId, path, contentType, extraHeaders) + + ("readIdleTimeoutMs" to readIdleTimeoutMs)) } + private fun openArguments( + engine: Int, + callId: Int, + path: String, + contentType: String, + extraHeaders: List> = emptyList(), + ): Map = mapOf( + "engineId" to engine, + "callId" to callId, + "url" to serverUrl + path, + "method" to "POST", + "headers" to listOf(listOf("content-type", contentType), listOf("connect-protocol-version", "1")) + + extraHeaders, + ) + // Each case opens a fresh connection to the host through the emulator, which once took 3 s to come up on CI (the // request reached the server just as a 3 s wait gave up). 10 s, like the Dart cases, still fails a real hang fast. private fun await(callId: Int, vararg types: String): Map { @@ -165,7 +228,27 @@ class CronetBridgeTest { } } - private fun awaitOutcome(callId: Int) = await(callId, "done", "failed", "canceled") + private fun awaitOutcome(callId: Int) = await(callId, *OUTCOMES.toTypedArray()) + + // Outcomes for [callId] that arrived since the last wait. + private fun drained(callId: Int): List> { + val more = mutableListOf>() + incoming.drainTo(more) + events.addAll(more) + return more.filter { it["callId"] == callId && it["type"] in OUTCOMES } + } + + private fun awaitReceived(callId: Int, text: String) { + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10) + while (!received(callId).contains(text)) { + val left = deadline - System.nanoTime() + val next = if (left > 0) incoming.poll(left, TimeUnit.NANOSECONDS) else null + if (next == null) { + fail("timed out waiting for \"$text\" on call $callId; events: $events") + } + events.add(next!!) + } + } // The response body so far, as text; Connect framing and protobuf leave the sentences readable. private fun received(callId: Int): String = events @@ -175,6 +258,7 @@ class CronetBridgeTest { private companion object { const val CONVERSE = "/connectrpc.eliza.v1.ElizaService/Converse" const val SAY = "/connectrpc.eliza.v1.ElizaService/Say" + val OUTCOMES = setOf("done", "failed", "canceled") // A SayRequest or ConverseRequest: field 1, the sentence. fun sentence(text: String): ByteArray { diff --git a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt index 88c343a..8d60dc7 100644 --- a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt +++ b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt @@ -10,9 +10,13 @@ import org.chromium.net.CronetException import org.chromium.net.NetworkException import org.chromium.net.QuicOptions import org.chromium.net.UrlResponseInfo +import java.net.URI +import java.net.URISyntaxException import java.nio.ByteBuffer -import java.util.concurrent.ExecutorService import java.util.concurrent.Executors +import java.util.concurrent.ScheduledExecutorService +import java.util.concurrent.ScheduledFuture +import java.util.concurrent.TimeUnit // Bridges Cronet BidirectionalStream to lib/src/cronet/cronet_http_client.dart, one stream per call. Dart sends // request chunks as method calls; response headers, body chunks, trailers and the outcome go to [emit], from the @@ -29,20 +33,42 @@ class CronetBridge( private val main = Handler(Looper.getMainLooper()) - // Cronet callbacks and all state below run on this one thread, so none of it needs locking. - private val executor: ExecutorService = Executors.newSingleThreadExecutor() + // Cronet callbacks, read-idle timers and all state below run on this one thread, so none of it needs locking. + private val executor: ScheduledExecutorService = Executors.newSingleThreadScheduledExecutor() private val engines = HashMap() private val calls = HashMap() private var nextEngineId = 1 - private class Engine(val cronet: CronetEngine) { + private class EngineOptions(val quicHints: List>, val quicIdleTimeoutSeconds: Int?) + + // One CronetEngine and the calls on it. An engine whose call ran out of time on a quiet connection (see evict) is + // retired: new calls go to a fresh one, with its own connections, and the retired one shuts down once its last call + // ends. + private class Pool(val cronet: CronetEngine) { val callIds = HashSet() + + // When this engine last received anything from each origin ("host:port"), in System.nanoTime(). + val lastReceived = HashMap() + } + + private class Engine(val options: EngineOptions, var current: Pool) { + val retired = ArrayList() var shuttingDown = false } - private class Call(val callId: Int, val engineId: Int) { + private class Call( + val callId: Int, + val engineId: Int, + val pool: Pool, + val origin: String, + val readIdleTimeoutNanos: Long?, + ) { lateinit var stream: BidirectionalStream + // When the call last received anything from the server, or started, for its read-idle timeout. + var lastReceived = System.nanoTime() + var readIdleTimer: ScheduledFuture<*>? = null + // No request body is written before Cronet reports the stream ready, which is what lets Dart treat a // failure before this point as "never sent". var ready = false @@ -93,22 +119,14 @@ class CronetBridge( private fun dispatch(method: String, args: Map, result: (Reply) -> Unit) { when (method) { "createEngine" -> { - val quicHints = (args["quicHints"] as List>?).orEmpty() - val quicIdleTimeoutSeconds = args["quicIdleTimeoutSeconds"] as Int? + val options = EngineOptions( + (args["quicHints"] as List>?).orEmpty().map { it[0] as String to it[1] as Int }, + args["quicIdleTimeoutSeconds"] as Int?, + ) executor.execute { try { - val builder = CronetEngine.Builder(context).enableHttp2(true).enableQuic(true) - for (hint in quicHints) { - val port = hint[1] as Int - builder.addQuicHint(hint[0] as String, port, port) - } - if (quicIdleTimeoutSeconds != null) { - builder.setQuicOptions( - QuicOptions.builder().setIdleConnectionTimeoutSeconds(quicIdleTimeoutSeconds.toLong()), - ) - } val id = nextEngineId++ - engines[id] = Engine(builder.build()) + engines[id] = Engine(options, Pool(build(options))) result(Reply.Success(id)) } catch (e: Throwable) { // Throwable, not Exception: failing to load Cronet's native library is an UnsatisfiedLinkError. @@ -133,7 +151,10 @@ class CronetBridge( val url = args["url"] as String val httpMethod = args["method"] as String val headers = (args["headers"] as List>).map { it[0] to it[1] } - execute(callId) { open(engineId, callId, url, httpMethod, headers) } + // A Dart int arrives as an Int or, past 32 bits, a Long. + val readIdleTimeoutMs = (args["readIdleTimeoutMs"] as Number?)?.toLong() + require(readIdleTimeoutMs == null || readIdleTimeoutMs > 0) { "readIdleTimeoutMs: $readIdleTimeoutMs" } + execute(callId) { open(engineId, callId, url, httpMethod, headers, readIdleTimeoutMs) } result(Reply.Success(null)) } "write" -> { @@ -149,21 +170,51 @@ class CronetBridge( } "cancel" -> { val callId = args["callId"] as Int - execute(callId) { calls[callId]?.stream?.cancel() } + // Set when the call hit its deadline. The server was given a deadline a step earlier, so on a live + // connection its deadline_exceeded would have ended the call first. + val evictConnection = args["evictConnection"] as Boolean? ?: false + execute(callId) { + val c = calls[callId] ?: return@execute + if (evictConnection) { + evict(c) + } + c.stream.cancel() + } result(Reply.Success(null)) } else -> result(Reply.NotImplemented) } } - private fun open(engineId: Int, callId: Int, url: String, method: String, headers: List>) { + private fun build(options: EngineOptions): CronetEngine { + val builder = CronetEngine.Builder(context).enableHttp2(true).enableQuic(true) + for ((host, port) in options.quicHints) { + builder.addQuicHint(host, port, port) + } + if (options.quicIdleTimeoutSeconds != null) { + builder.setQuicOptions( + QuicOptions.builder().setIdleConnectionTimeoutSeconds(options.quicIdleTimeoutSeconds.toLong()), + ) + } + return builder.build() + } + + private fun open( + engineId: Int, + callId: Int, + url: String, + method: String, + headers: List>, + readIdleTimeoutMs: Long?, + ) { val engine = engines[engineId] if (engine == null || engine.shuttingDown) { emit(failure(callId, false, "engine $engineId is shut down", null)) return } - val c = Call(callId, engineId) - val builder = engine.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) + val pool = engine.current + val c = Call(callId, engineId, pool, origin(url), readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }) + val builder = pool.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) for ((name, value) in headers) { builder.addHeader(name, value) } @@ -172,7 +223,88 @@ class CronetBridge( // once it has started, so abort() then reports it as never sent, with no stream to release. c.stream.start() calls[callId] = c - engine.callIds.add(callId) + pool.callIds.add(callId) + c.lastReceived = System.nanoTime() + scheduleReadIdleCheck(c, c.readIdleTimeoutNanos) + } + + // "host:port" of [url], for telling which calls share connections. java.net.URI is stricter than Cronet, so a URL it + // rejects stands for itself rather than failing a call Cronet would carry. + private fun origin(url: String): String { + val target = try { + URI(url) + } catch (e: URISyntaxException) { + return url + } + val port = if (target.port != -1) target.port else if (target.scheme == "http") 80 else 443 + return "${target.host}:$port" + } + + // Read-idle timeout. Rather than rescheduling on every chunk received, each check finds when the call last received + // something and, unless the timeout has run out since, schedules the next check for when it would. + private fun scheduleReadIdleCheck(c: Call, delayNanos: Long?) { + if (delayNanos == null) { + return + } + val check = Runnable { + try { + checkReadIdle(c) + } catch (e: Exception) { + Log.e(TAG, "call ${c.callId} failed in the bridge", e) + abort(c.callId, e) + } + } + c.readIdleTimer = executor.schedule(check, delayNanos, TimeUnit.NANOSECONDS) + } + + private fun checkReadIdle(c: Call) { + val timeout = c.readIdleTimeoutNanos ?: return + if (c.reported || calls[c.callId] !== c) { + return + } + val idle = System.nanoTime() - c.lastReceived + if (idle < timeout) { + scheduleReadIdleCheck(c, timeout - idle) + return + } + val message = "read idle timeout: nothing received for ${TimeUnit.NANOSECONDS.toMillis(idle)} ms" + report(c, failure(c.callId, c.ready, message, ERR_TIMED_OUT)) + // Cronet's onCanceled then releases the call and the stream. + c.stream.cancel() + evict(c) + } + + // Anything from the server shows the call's connection is alive. + private fun received(c: Call) { + val now = System.nanoTime() + c.lastReceived = now + c.pool.lastReceived[c.origin] = now + } + + // A call that ran out of time (its read-idle timeout, or its deadline) may be stuck on a dead connection. + // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received + // anything; otherwise only that call was slow. Cronet has no call that closes one connection, so the engine's + // CronetEngine is retired: new calls go to a fresh one, which dials its own connections, and the old one shuts down + // once the calls still on it end. Calls that time out together on the same dead connection retire it once. + private fun evict(c: Call) { + val engine = engines[c.engineId] ?: return + if (engine.current !== c.pool || engine.shuttingDown) { + return + } + val heard = c.pool.lastReceived[c.origin] + if (heard != null && heard - c.lastReceived > 0) { + return + } + val fresh = try { + Pool(build(engine.options)) + } catch (e: Throwable) { + // Keep the engine there is; new calls on a dead connection fail on their own timeouts. + Log.e(TAG, "engine ${c.engineId}: could not replace the CronetEngine for call ${c.callId}", e) + return + } + engine.retired.add(engine.current) + engine.current = fresh + Log.i(TAG, "engine ${c.engineId}: call ${c.callId} ran out of time on a quiet connection; moved to a new engine") } // Fails a call the bridge itself could not carry on with, after an exception on its thread. @@ -224,8 +356,9 @@ class CronetBridge( // Cronet's final callback for a call: releases it and, unless the bridge has already failed it, reports [event]. private fun finish(c: Call, event: Map) { + c.readIdleTimer?.cancel(false) calls.remove(c.callId) - engines[c.engineId]?.callIds?.remove(c.callId) + c.pool.callIds.remove(c.callId) report(c, event) shutdownIfIdle(c.engineId) } @@ -237,16 +370,25 @@ class CronetBridge( } } + // Shuts down the engine's retired CronetEngines that no call uses any more, and the whole engine once it is shutting + // down and idle. private fun shutdownIfIdle(engineId: Int) { val engine = engines[engineId] ?: return - if (engine.shuttingDown && engine.callIds.isEmpty()) { + val idle = engine.retired.filter { it.callIds.isEmpty() } + engine.retired.removeAll(idle) + idle.forEach { shutdown(engineId, it) } + if (engine.shuttingDown && engine.current.callIds.isEmpty() && engine.retired.isEmpty()) { engines.remove(engineId) - try { - engine.cronet.shutdown() - } catch (e: Exception) { - // Cronet refuses while it still counts a request as running. The engine is unreachable either way. - Log.w(TAG, "engine $engineId did not shut down", e) - } + shutdown(engineId, engine.current) + } + } + + private fun shutdown(engineId: Int, pool: Pool) { + try { + pool.cronet.shutdown() + } catch (e: Exception) { + // Cronet refuses while it still counts a request as running. The engine is unreachable either way. + Log.w(TAG, "engine $engineId did not shut down", e) } } @@ -269,6 +411,7 @@ class CronetBridge( } override fun onResponseHeadersReceived(stream: BidirectionalStream, info: UrlResponseInfo) { + received(c) emit(mapOf( "callId" to c.callId, "type" to "headers", @@ -285,6 +428,7 @@ class CronetBridge( buffer: ByteBuffer, endOfStream: Boolean, ) { + received(c) buffer.flip() if (buffer.hasRemaining()) { val bytes = ByteArray(buffer.remaining()) @@ -315,6 +459,7 @@ class CronetBridge( info: UrlResponseInfo, trailers: UrlResponseInfo.HeaderBlock, ) { + received(c) emit(mapOf("callId" to c.callId, "type" to "trailers", "headers" to headerPairs(trailers.asList))) } @@ -341,5 +486,8 @@ class CronetBridge( private companion object { const val TAG = "connect_native_http" const val READ_BUFFER_BYTES = 32 * 1024 + + // Chromium's net::ERR_TIMED_OUT (net/base/net_error_list.h), reported for a read idle timeout. + const val ERR_TIMED_OUT = -7 } } diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index c41cadc..aad77a7 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -38,10 +38,22 @@ const introduceInterval = Duration(milliseconds: 100); final class Harness { Harness._(this.native, this.eliza); - static Future create(String baseUrl, {Set http3Origins = const {}}) async { + static Future create( + String baseUrl, { + Set http3Origins = const {}, + Duration? readIdleTimeout, + Duration? deadlineStep, + bool evictOnDeadline = false, + }) async { late final Harness harness; final native = await NativeHttpClient.create( - NativeHttpConfig(http3Origins: http3Origins, onResponse: (info) => harness.platformProtocols.add(info.protocol)), + NativeHttpConfig( + http3Origins: http3Origins, + readIdleTimeout: readIdleTimeout, + deadlineStep: deadlineStep, + evictOnDeadline: evictOnDeadline, + onResponse: (info) => harness.platformProtocols.add(info.protocol), + ), ); if (native == null) { throw StateError('no native client on this platform'); @@ -63,7 +75,13 @@ final class Harness { /// `X-Proto` of each response: the protocol the server saw. final serverProtocols = []; - void recordHeaders(Headers headers) => serverProtocols.add(headers.get('x-proto')?.join(',') ?? ''); + /// `X-Remote-Addr` of each response: the client's address as the server saw it, which differs between connections. + final remoteAddrs = []; + + void recordHeaders(Headers headers) { + serverProtocols.add(headers.get('x-proto')?.join(',') ?? ''); + remoteAddrs.add(headers.get('x-remote-addr')?.join(',') ?? ''); + } } /// Every case finishes in a few seconds against a local server (most well under one); a hang should fail fast. @@ -119,6 +137,34 @@ Future harnessNegotiating(String protocol) async { } } +/// A client on HTTP/2, with the server's UDP frozen for the rest of the case so that QUIC can't connect: URLSession +/// then falls back to TCP, as on a network that blocks UDP. Android is on HTTP/2 with this server anyway. +Future harnessOverHttp2({ + Duration? readIdleTimeout, + Duration? deadlineStep, + bool evictOnDeadline = false, +}) async { + addTearDown(() => admin('/unfreeze')); + await admin('/freeze?for=60s'); + final harness = await Harness.create( + serverUrl, + readIdleTimeout: readIdleTimeout, + deadlineStep: deadlineStep, + evictOnDeadline: evictOnDeadline, + ); + addTearDown(harness.native.close); + await harness.eliza.say( + SayRequest(sentence: 'which protocol'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: harness.recordHeaders, + ); + expect(harness.serverProtocols.last, 'HTTP/2.0'); + return harness; +} + +/// The platform code of a read-idle timeout: `NSURLErrorTimedOut` on iOS, `net::ERR_TIMED_OUT` from the Cronet bridge. +final readIdleTimeoutCode = io.Platform.isIOS ? -1001 : -7; + Future nativeCause(Future call) async { try { await call; @@ -316,6 +362,163 @@ void main() { expect(h3.serverProtocols.last, 'HTTP/3.0'); }, timeout: http3CaseTimeout); + // Over HTTP/2 neither platform notices a dead connection (no pings), so a read-idle timeout is what does. + bool skipUnlessAdmin() { + if (adminUrl.isEmpty) { + markTestSkipped('needs ADMIN_URL'); + return true; + } + return false; + } + + // Up to 8 s for the first call over HTTP/2 on top of the case itself. + const http2CaseTimeout = Duration(seconds: 25); + + testCase('a read-idle timeout ends a quiet HTTP/2 stream', (_) async { + if (skipUnlessAdmin()) { + return; + } + const timeout = Duration(seconds: 1); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); + expect(h2.serverProtocols.last, 'HTTP/2.0'); + + // Timed from the greeting, the last thing the stream received. + final stopwatch = Stopwatch()..start(); + final cause = await nativeCause(responses.moveNext()); + expect(cause.platformCode, readIdleTimeoutCode, reason: '$cause'); + expect(cause.failure, NativeHttpFailure.other, reason: '$cause'); + expect(stopwatch.elapsed, greaterThan(timeout * 0.5)); + expect(stopwatch.elapsed, lessThan(timeout + const Duration(seconds: 2))); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('an HTTP/2 stream that keeps receiving outlives its read-idle timeout', (_) async { + if (skipUnlessAdmin()) { + return; + } + const timeout = Duration(seconds: 1); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); + // Three times the timeout in all, but never more than 0.6 of it between replies. + for (var i = 0; i < 5; i++) { + await Future.delayed(timeout * 0.6); + requests.add(ConverseRequest(sentence: 'ping $i')); + expect(await responses.moveNext(), isTrue); + expect(responses.current.sentence, 'echo: ping $i'); + } + requests.add(ConverseRequest(sentence: 'bye')); + expect(await responses.moveNext(), isTrue); + expect(await responses.moveNext(), isFalse); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('a dead HTTP/2 connection is reused until a call runs out of time on it', (_) async { + // Nothing notices the dead connection by itself; with evictOnDeadline, the first call to run out of time, here on + // its deadline, moves the client off it. + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(deadlineStep: const Duration(seconds: 1), evictOnDeadline: true); + final (requests, responses) = await openQuietConverse(h2); + final deadAddr = h2.remoteAddrs.last; + await admin('/freeze?for=60s&path=tcp'); + + requests.add(ConverseRequest(sentence: 'anyone?')); + final reply = responses.moveNext(); + await expectLater(reply.timeout(const Duration(seconds: 3)), throwsA(isA())); + // A new call goes to the same pooled connection, and waits until its deadline. + await expectLater( + h2.eliza.say(SayRequest(sentence: 'hi'), signal: TimeoutSignal(const Duration(seconds: 2))), + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + // The server's deadline_exceeded never came, so the next call dials a new connection. + final res = await h2.eliza.say( + SayRequest(sentence: 'after'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: h2.recordHeaders, + ); + expect(res.sentence, 'say: after'); + expect(h2.remoteAddrs.last, isNot(deadAddr)); + unawaited(reply.then((_) {}, onError: (_) {})); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('the server answers a deadline a step early, and the connection is kept', (_) async { + // With deadlineStep, the server gets a deadline a second shorter than the client's, so its deadline_exceeded ends + // the call before the client's own deadline fires, and shows the connection is alive. + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(deadlineStep: const Duration(seconds: 1), evictOnDeadline: true); + final liveAddr = h2.remoteAddrs.last; + final stopwatch = Stopwatch()..start(); + await expectLater( + h2.eliza.say(SayRequest(sentence: 'sleep:5000'), signal: TimeoutSignal(const Duration(seconds: 3))), + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + // The server's 2 s, not the client's 3 s. + expect(stopwatch.elapsed, lessThan(const Duration(milliseconds: 2800))); + await h2.eliza.say(SayRequest(sentence: 'after'), onHeader: h2.recordHeaders); + expect(h2.remoteAddrs.last, liveAddr); + }, timeout: http2CaseTimeout); + + testCase('a read-idle timeout keeps a connection another call is still receiving on', (_) async { + // The quiet stream's timeout says nothing about the connection while the busy one keeps hearing from the server. + if (skipUnlessAdmin()) { + return; + } + const timeout = Duration(seconds: 1); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (quietRequests, quiet) = await openQuietConverse(h2); + final liveAddr = h2.remoteAddrs.last; + final (busyRequests, busy) = await openQuietConverse(h2); + expect(h2.remoteAddrs.last, liveAddr); + + final quietFailure = nativeCause(quiet.moveNext()); + for (var i = 0; i < 6; i++) { + await Future.delayed(timeout * 0.3); + busyRequests.add(ConverseRequest(sentence: 'ping $i')); + expect(await busy.moveNext(), isTrue); + } + expect((await quietFailure).platformCode, readIdleTimeoutCode); + + await h2.eliza.say(SayRequest(sentence: 'after'), onHeader: h2.recordHeaders); + expect(h2.remoteAddrs.last, liveAddr); + busyRequests.add(ConverseRequest(sentence: 'bye')); + expect(await busy.moveNext(), isTrue); + await busyRequests.close(); + await quietRequests.close(); + }, timeout: http2CaseTimeout); + + testCase('a read-idle timeout notices a dead HTTP/2 connection, and the next call dials a new one', (_) async { + if (skipUnlessAdmin()) { + return; + } + const timeout = Duration(seconds: 2); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); + await admin('/freeze?for=60s&path=tcp'); + + final deadAddr = h2.remoteAddrs.last; + requests.add(ConverseRequest(sentence: 'anyone?')); + final stopwatch = Stopwatch()..start(); + final cause = await nativeCause(responses.moveNext()); + expect(cause.platformCode, readIdleTimeoutCode, reason: '$cause'); + expect(stopwatch.elapsed, lessThan(timeout + const Duration(seconds: 2))); + await requests.close(); + + // The frozen connection is still open, so this only works on a new one. + final res = await h2.eliza.say( + SayRequest(sentence: 'after'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: h2.recordHeaders, + ); + expect(res.sentence, 'say: after'); + expect(h2.serverProtocols.last, 'HTTP/2.0'); + expect(h2.remoteAddrs.last, isNot(deadAddr)); + }, timeout: http2CaseTimeout); + testCase('a dead HTTP/3 path fails an open stream', (_) async { if (skipUnlessHttp3()) { return; diff --git a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift index da137a6..d5fa580 100644 --- a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift +++ b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift @@ -33,8 +33,11 @@ public final class URLSessionBridge { private var nextSessionId = 1 private final class Session { - let urlSession: URLSession + // Replaced after a call times out, so new calls dial a fresh connection; see `evictConnection`. + var urlSession: URLSession let http3Origins: Set + // When `urlSession` last received anything from each origin ("host:port"), in uptime nanoseconds. + var lastReceived: [String: UInt64] = [:] init(urlSession: URLSession, http3Origins: Set) { self.urlSession = urlSession @@ -44,6 +47,12 @@ public final class URLSessionBridge { private final class Call { let callId: Int + var engineId = 0 + // The URLSession the call's task runs in. + var urlSession: URLSession? + var origin = "" + // When the call last received anything, or started, in uptime nanoseconds. + var lastReceived = URLSessionBridge.now() var task: URLSessionTask! var input: InputStream? var output: OutputStream? @@ -90,17 +99,9 @@ public final class URLSessionBridge { return "\(origin[0]):\(origin[1])" } queue.async { - let configuration = URLSessionConfiguration.ephemeral - // URLSession fails a task that receives nothing for this long. Its default of 60 s would end a quiet stream - // or a long-running unary call; callers bound calls with deadlines instead. - configuration.timeoutIntervalForRequest = 24 * 60 * 60 - // Only a bridge with its own roots implements the trust challenge. With the system's roots there is nothing - // for the delegate to decide, so certificate checks are left entirely to the network stack. - let delegate = self.trustedRoots.isEmpty ? Delegate(bridge: self) : TrustingDelegate(bridge: self) - let session = URLSession(configuration: configuration, delegate: delegate, delegateQueue: self.delegateQueue) let id = self.nextSessionId self.nextSessionId += 1 - self.sessions[id] = Session(urlSession: session, http3Origins: Set(origins)) + self.sessions[id] = Session(urlSession: self.makeURLSession(), http3Origins: Set(origins)) reply(.success(id)) } case "shutdownEngine": @@ -117,7 +118,8 @@ public final class URLSessionBridge { let headers: [[String]] = try arg(args, "headers") guard headers.allSatisfy({ $0.count == 2 }) else { throw BadArgument(name: "headers") } let hasBody: Bool = try arg(args, "hasBody") - queue.async { self.open(engineId, callId, url, method, headers, hasBody) } + let readIdleTimeoutMs = try optionalPositiveInt(args, "readIdleTimeoutMs") + queue.async { self.open(engineId, callId, url, method, headers, hasBody, readIdleTimeoutMs) } reply(.success(nil)) case "write": let callId: Int = try arg(args, "callId") @@ -142,19 +144,62 @@ public final class URLSessionBridge { reply(.success(nil)) case "cancel": let callId: Int = try arg(args, "callId") - queue.async { self.calls[callId]?.task.cancel() } + // Set when the call hit its deadline. The server was given a deadline a step earlier, so on a live connection + // its deadline_exceeded would have ended the call first. + let value = args["evictConnection"] + guard value == nil || value is NSNull || value is Bool else { throw BadArgument(name: "evictConnection") } + let evict = value as? Bool ?? false + queue.async { + guard let c = self.calls[callId] else { return } + if evict { + self.evictConnection(of: c) + } + c.task.cancel() + } reply(.success(nil)) default: reply(.notImplemented) } } + /// An optional argument that must be a positive integer when present; absent or null for none. + private func optionalPositiveInt(_ args: [String: Any], _ name: String) throws -> Int? { + guard let value = args[name], !(value is NSNull) else { return nil } + guard let n = value as? Int, n > 0 else { throw BadArgument(name: name) } + return n + } + + static func now() -> UInt64 { + DispatchTime.now().uptimeNanoseconds + } + private func failure(_ callId: Int, domain: String, code: Int, message: String) -> [String: Any] { ["callId": callId, "type": "failed", "domain": domain, "code": code, "message": message] } + private func makeURLSession() -> URLSession { + let configuration = URLSessionConfiguration.ephemeral + // URLSession fails a task that receives nothing for this long. Its default of 60 s would end a quiet stream or a + // long-running unary call; callers bound calls with deadlines, or set the client's read-idle timeout. + configuration.timeoutIntervalForRequest = Self.sessionIdleTimeout + // Only a bridge with its own roots implements the trust challenge. With the system's roots there is nothing for + // the delegate to decide, so certificate checks are left entirely to the network stack. + let delegate = trustedRoots.isEmpty ? Delegate(bridge: self) : TrustingDelegate(bridge: self) + return URLSession(configuration: configuration, delegate: delegate, delegateQueue: delegateQueue) + } + + /// The `URLRequest.timeoutInterval` for a read-idle timeout of `ms` milliseconds, or nil for the session's. + static func requestTimeoutInterval(readIdleTimeoutMs ms: Int?) -> TimeInterval? { + guard let ms else { return nil } + let interval = TimeInterval(ms) / 1000 + // URLSession takes a request's 60 s, URLRequest's default, to mean "not set" and applies the session's value + // instead, however the request got it. One millisecond more is still honored. + return interval == 60 ? 60.001 : interval + } + private func open( - _ engineId: Int, _ callId: Int, _ url: String, _ method: String, _ headers: [[String]], _ hasBody: Bool + _ engineId: Int, _ callId: Int, _ url: String, _ method: String, _ headers: [[String]], _ hasBody: Bool, + _ readIdleTimeoutMs: Int? ) { guard let session = sessions[engineId], let target = URL(string: url) else { emit(failure( @@ -168,9 +213,18 @@ public final class URLSessionBridge { request.addValue(h[1], forHTTPHeaderField: h[0]) } let port = target.port ?? (target.scheme == "http" ? 80 : 443) - request.assumesHTTP3Capable = session.http3Origins.contains("\(target.host ?? ""):\(port)") + let origin = "\(target.host ?? ""):\(port)" + request.assumesHTTP3Capable = session.http3Origins.contains(origin) + // The request's read-idle timeout: URLSession fails the task once it has received nothing for this long, with + // NSURLErrorTimedOut. + if let interval = Self.requestTimeoutInterval(readIdleTimeoutMs: readIdleTimeoutMs) { + request.timeoutInterval = interval + } let c = Call(callId: callId) + c.engineId = engineId + c.urlSession = session.urlSession + c.origin = origin if hasBody { var input: InputStream? var output: OutputStream? @@ -218,14 +272,26 @@ public final class URLSessionBridge { callsByTask[task]?.input == nil } + // Anything from the server shows the call's connection is alive. + private func received(_ c: Call) { + let now = Self.now() + c.lastReceived = now + if let session = sessions[c.engineId], session.urlSession === c.urlSession { + session.lastReceived[c.origin] = now + } + } + fileprivate func didReceive(_ response: URLResponse, for task: URLSessionTask) { - guard let c = callsByTask[task], let http = response as? HTTPURLResponse else { return } + guard let c = callsByTask[task] else { return } + received(c) + guard let http = response as? HTTPURLResponse else { return } let headers = http.allHeaderFields.map { [String(describing: $0.key), String(describing: $0.value)] } emit(["callId": c.callId, "type": "headers", "status": http.statusCode, "protocol": NSNull(), "headers": headers]) } fileprivate func didReceive(_ data: Data, for task: URLSessionTask) { guard let c = callsByTask[task] else { return } + received(c) emit(["callId": c.callId, "type": "data", "data": data]) } @@ -248,9 +314,28 @@ public final class URLSessionBridge { emit(["callId": c.callId, "type": "canceled"]) return } + if error.domain == NSURLErrorDomain && error.code == NSURLErrorTimedOut { + evictConnection(of: c) + } emit(failure(c.callId, domain: error.domain, code: error.code, message: error.localizedDescription)) } + // A call that ran out of time (its read-idle timeout, or its deadline) may be stuck on a dead connection. + // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received + // anything; otherwise only that call was slow. URLSession pools connections per session and has no call that closes + // one, so the engine's new calls go to a fresh URLSession, which dials its own connections. The old one finishes the + // tasks it still has, and is then released. Calls that time out together on the same dead connection replace it + // once. + private func evictConnection(of c: Call) { + guard let session = sessions[c.engineId], let old = c.urlSession, session.urlSession === old else { return } + if let heard = session.lastReceived[c.origin], heard > c.lastReceived { + return + } + session.urlSession = makeURLSession() + session.lastReceived = [:] + old.finishTasksAndInvalidate() + } + fileprivate func serverTrust(_ challenge: URLAuthenticationChallenge) -> ( URLSession.AuthChallengeDisposition, URLCredential? ) { @@ -267,6 +352,9 @@ public final class URLSessionBridge { // 64 KiB, the bound pair's buffer: how much request body may wait for URLSession to read it. private static let bodyBufferBytes = 64 * 1024 + + // A day: in effect no read-idle timeout, for requests that don't set one. + private static let sessionIdleTimeout: TimeInterval = 24 * 60 * 60 } // URLSession keeps its delegate until the session is invalidated, so this holds the bridge weakly. diff --git a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift index 1978fa2..06bb7d5 100644 --- a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift +++ b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift @@ -89,14 +89,72 @@ final class URLSessionBridgeTests: XCTestCase { XCTAssertEqual(outcome(2)?["domain"] as? String, "ConnectNativeHttp") } + func testReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() async throws { + let engine = try await createEngine() + // The timeout also covers setting up the connection, which can outlast it on a loaded simulator. A call first + // leaves a connection for the timed one to reuse. + try await open(engine, callId: 10, path: Self.say, contentType: "application/proto") + try await write(10, sentence("warm up")) + try await invoke("end", ["callId": 10]) + try await waitUntil("the warm-up call ended", timeout: 10) { self.outcome(10) != nil } + try await open( + engine, callId: 1, path: Self.converse, contentType: "application/connect+proto", readIdleTimeoutMs: 500) + try await waitUntil("the greeting arrived") { self.received(1).contains("hello") } + let start = Date() + try await waitUntil("the call timed out") { self.outcome(1) != nil } + let elapsed = Date().timeIntervalSince(start) + XCTAssertEqual(outcome(1)?["type"] as? String, "failed", "\(String(describing: outcome(1)))") + XCTAssertEqual(outcome(1)?["code"] as? Int, NSURLErrorTimedOut) + XCTAssertLessThan(elapsed, 2, "a 500 ms read-idle timeout took \(elapsed) s") + + // New calls go to a fresh URLSession. + try await open(engine, callId: 2, path: Self.say, contentType: "application/proto") + try await write(2, sentence("after")) + try await invoke("end", ["callId": 2]) + try await waitUntil("the next call ended") { self.outcome(2) != nil } + XCTAssertEqual(outcome(2)?["type"] as? String, "done", "\(String(describing: outcome(2)))") + XCTAssertTrue(received(2).contains("say: after")) + } + + // A call that hit its deadline is canceled at once, and moves the engine's new calls to a fresh URLSession. + func testADeadlineCancelEndsTheCallAndTheEngineCarriesOn() async throws { + let engine = try await createEngine() + try await open(engine, callId: 1, path: Self.converse, contentType: "application/connect+proto") + try await waitUntil("the greeting arrived", timeout: 10) { self.received(1).contains("hello") } + try await invoke("cancel", ["callId": 1, "evictConnection": true]) + try await waitUntil("the call was canceled") { self.outcome(1) != nil } + XCTAssertEqual(outcome(1)?["type"] as? String, "canceled", "\(String(describing: outcome(1)))") + + try await open(engine, callId: 2, path: Self.say, contentType: "application/proto") + try await write(2, sentence("after")) + try await invoke("end", ["callId": 2]) + try await waitUntil("the next call ended", timeout: 10) { self.outcome(2) != nil } + XCTAssertEqual(outcome(2)?["type"] as? String, "done", "\(String(describing: outcome(2)))") + } + + func testRequestTimeoutIntervalAvoidsTheValueURLSessionIgnores() { + XCTAssertNil(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: nil)) + XCTAssertEqual(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: 1500), 1.5) + XCTAssertEqual(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: 60_000), 60.001) + } + func testMalformedArgumentsAreRefusedInsteadOfCrashing() async throws { let malformed: [(String, [String: Any])] = [ ("open", [:]), ("open", ["engineId": "1", "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true]), ("open", ["engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [["one"]], "hasBody": true]), + ("open", [ + "engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true, + "readIdleTimeoutMs": "soon", + ]), + ("open", [ + "engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true, + "readIdleTimeoutMs": 0, + ]), ("write", ["callId": 1, "data": "not bytes"]), ("end", ["callId": "1"]), ("cancel", [:]), + ("cancel", ["callId": 1, "evictConnection": "yes"]), ("shutdownEngine", ["engineId": NSNull()]), ("createEngine", ["http3Origins": [["localhost"]]]), ] @@ -144,11 +202,15 @@ final class URLSessionBridgeTests: XCTestCase { return try await invoke("createEngine", ["http3Origins": [origin]]) as! Int } - private func open(_ engine: Int, callId: Int, path: String, contentType: String) async throws { - try await invoke("open", [ + private func open( + _ engine: Int, callId: Int, path: String, contentType: String, readIdleTimeoutMs: Int? = nil + ) async throws { + var arguments: [String: Any] = [ "engineId": engine, "callId": callId, "url": serverURL + path, "method": "POST", "headers": [["content-type", contentType], ["connect-protocol-version", "1"]], "hasBody": true, - ]) + ] + arguments["readIdleTimeoutMs"] = readIdleTimeoutMs + try await invoke("open", arguments) } private func write(_ callId: Int, _ data: Data) async throws { diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index 1e7f1e8..ec6386a 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -1,4 +1,5 @@ import 'dart:async'; +import 'dart:math' as math; import 'package:connectrpc/connect.dart'; import 'package:flutter/foundation.dart'; @@ -60,6 +61,9 @@ final class _Call { /// Set when the call's signal fired, so the outcome reports the signal's error rather than the cancellation. Object? abortError; + /// Time since the call last received anything from the server, or started. + final sinceReceived = Stopwatch()..start(); + void fail(Object error) { finished = true; if (!response.isCompleted) { @@ -133,9 +137,16 @@ final class _Dispatcher { /// Asks the plugin to cancel call [id]; its `canceled` event then ends the call. Should the plugin fail to take the /// request, no event will come, so the call ends here instead. - Future cancel(int id) async { + /// + /// With [evictConnection], the call hit its deadline (see [NativeHttpConfig.evictOnDeadline]). The plugin then moves + /// new calls off the call's connection, unless something from the call's origin has arrived since the call last + /// received anything. + Future cancel(int id, {bool evictConnection = false}) async { try { - await bridge.methods.invokeMethod('cancel', {'callId': id}); + await bridge.methods.invokeMethod('cancel', { + 'callId': id, + if (evictConnection) 'evictConnection': true, + }); } catch (error, stack) { final call = remove(id); call?.fail(call.abortError ?? ConnectException(Code.canceled, '${bridge.name}: cancel failed: $error')); @@ -169,6 +180,9 @@ final class _Dispatcher { } void _onCallEvent(int id, _Call call, Map e) { + if (const {'headers', 'data', 'trailers'}.contains(e['type'])) { + call.sinceReceived.reset(); + } switch (e['type']) { case 'headers': final headers = bridge.responseHeaders([ @@ -231,12 +245,31 @@ final class _Dispatcher { } } +/// The `connect-timeout-ms` to send for a call whose deadline is [ms] away: [step] less, or half of [ms] when that is +/// less, so the server never gets zero. See [NativeHttpConfig.deadlineStep]. +int serverTimeoutMs(int ms, Duration step) => ms - math.min(step.inMilliseconds, ms ~/ 2); + +/// connect-dart's deadline header, which it sets from the call's signal. +const _connectTimeoutHeader = 'connect-timeout-ms'; + /// A connect-dart [HttpClient] on one engine of a [PlatformBridge] plugin. base class ChannelHttpClient implements NativeHttpClient { ChannelHttpClient(this._bridge, this._engineId, this._config); /// Starts the bridge's event routing and creates an engine with [arguments], returning its id. - static Future createEngine(PlatformBridge bridge, Map arguments) async { + static Future createEngine( + PlatformBridge bridge, + NativeHttpConfig config, + Map arguments, + ) async { + final readIdle = config.readIdleTimeout; + if (readIdle != null && readIdle.isNegative) { + throw ArgumentError.value(readIdle, 'readIdleTimeout', 'must not be negative'); + } + final step = config.deadlineStep; + if (step != null && step.isNegative) { + throw ArgumentError.value(step, 'deadlineStep', 'must not be negative'); + } _Dispatcher.of(bridge); return (await bridge.methods.invokeMethod('createEngine', arguments))!; } @@ -245,13 +278,36 @@ base class ChannelHttpClient implements NativeHttpClient { final int _engineId; final NativeHttpConfig _config; + /// [NativeHttpConfig.deadlineStep], or `null` for none. + late final Duration? _deadlineStep = _config.deadlineStep == Duration.zero ? null : _config.deadlineStep; + + /// [NativeHttpConfig.readIdleTimeout] in whole milliseconds, rounded up so a sub-millisecond timeout isn't none. + late final int? _readIdleTimeoutMs = switch (_config.readIdleTimeout) { + null || Duration.zero => null, + final timeout => (timeout.inMicroseconds / 1000).ceil(), + }; + @override HttpClient get httpClient => _send; Future _send(HttpRequest req) async { final dispatcher = _Dispatcher.of(_bridge); final methods = _bridge.methods; + final step = _deadlineStep; final call = _Call(Uri.parse(req.url)); + // How much earlier than the client the server times the call out. + var serverLead = Duration.zero; + final headers = >[]; + for (final (:name, :value) in req.header.entries) { + final ms = step != null && name == _connectTimeoutHeader ? int.tryParse(value) : null; + if (ms != null && ms > 0) { + final serverMs = serverTimeoutMs(ms, step!); + serverLead = Duration(milliseconds: ms - serverMs); + headers.add([name, '$serverMs']); + } else { + headers.add([name, value]); + } + } final id = dispatcher.register(call, _config.onResponse); call.body.onCancel = () { if (!call.finished) { @@ -259,10 +315,15 @@ base class ChannelHttpClient implements NativeHttpClient { } }; req.signal?.future.then((error) { - if (!call.finished) { - call.abortError = error; - unawaited(dispatcher.cancel(id)); + if (call.finished) { + return; } + call.abortError = error; + // A call that received something since the server's deadline was still hearing from the server, whatever the + // server made of the deadline, so its connection is alive. + final evict = + _config.evictOnDeadline && error.code == Code.deadlineExceeded && call.sinceReceived.elapsed >= serverLead; + unawaited(dispatcher.cancel(id, evictConnection: evict)); }); try { await methods.invokeMethod('open', { @@ -270,10 +331,9 @@ base class ChannelHttpClient implements NativeHttpClient { 'callId': id, 'url': req.url, 'method': req.method, - 'headers': [ - for (final (:name, :value) in req.header.entries) [name, value], - ], + 'headers': headers, 'hasBody': req.body != null, + 'readIdleTimeoutMs': _readIdleTimeoutMs, }); } catch (error) { // The plugin refused the call, or isn't there (a background isolate, say): nothing was sent. diff --git a/lib/src/cronet/cronet_http_client.dart b/lib/src/cronet/cronet_http_client.dart index a00e89a..267d662 100644 --- a/lib/src/cronet/cronet_http_client.dart +++ b/lib/src/cronet/cronet_http_client.dart @@ -36,7 +36,7 @@ final class CronetHttpClient extends ChannelHttpClient { if (idle != null && idle < const Duration(seconds: 1)) { throw ArgumentError.value(idle, 'quicIdleTimeout', 'Cronet takes whole seconds; use at least 1 s'); } - final engineId = await ChannelHttpClient.createEngine(cronetBridge, { + final engineId = await ChannelHttpClient.createEngine(cronetBridge, config, { 'quicHints': [ for (final origin in config.http3Origins) [origin.host, origin.port], ], diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index 1a145f8..c453e84 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -12,7 +12,14 @@ final class NativeHttpResponseInfo { /// Configuration shared by both platform clients. Server certificates are checked against the platform's trust /// store. final class NativeHttpConfig { - const NativeHttpConfig({this.http3Origins = const {}, this.quicIdleTimeout, this.onResponse}); + const NativeHttpConfig({ + this.http3Origins = const {}, + this.quicIdleTimeout, + this.readIdleTimeout, + this.deadlineStep, + this.evictOnDeadline = false, + this.onResponse, + }); /// `https://host:port` origins to try HTTP/3 with from the first request, before an `Alt-Svc` response header has /// advertised it. Both platforms fall back to HTTP/2 over TCP on their own when QUIC fails. @@ -29,6 +36,45 @@ final class NativeHttpConfig { /// Android only. URLSession exposes no QUIC settings, so on iOS only the server's advertised idle timeout applies. final Duration? quicIdleTimeout; + /// Fails each of this client's calls once it has received nothing from the server for this long. `null` or + /// [Duration.zero] means none. Each call has its own timer in the platform client, so it also works on the HTTP/2 + /// fallback, where nothing else notices a dead connection. + /// + /// It applies to every call, so leave it off on a client that carries streams which go quiet for long, or use a + /// second client for them. For per-call bounds use deadlines (connect-dart's `TimeoutSignal` or `DeadlineSignal`), + /// with [evictOnDeadline] to move new calls off a dead connection. + /// + /// - **What resets it:** anything the call receives. It runs from when the call starts, so it also bounds the wait + /// for the response headers, and a unary call the server holds longer than this fails. A server stream that may go + /// quiet needs a longer timeout, or periodic messages. + /// - **iOS:** `URLRequest.timeoutInterval`. URLSession fails the task with `NSURLErrorTimedOut` (-1001). + /// - **Android:** a timer in the bridge, which fails the call with `net::ERR_TIMED_OUT` (-7). + /// - **The failure:** a `ConnectException` with code `unavailable`, whose cause is a [NativeHttpException] with + /// that `platformCode`. + /// - **The connection:** if nothing has arrived from the call's origin, on any call, since the call last received + /// anything, the connection is taken to be dead. The client moves new calls to a fresh `URLSession` or Cronet + /// engine, so they dial a new connection, and lets the calls already on the old one finish there. + final Duration? readIdleTimeout; + + /// How much shorter a deadline the server gets than the call's own. connect-dart sends a call's deadline in + /// `connect-timeout-ms`; with this set, the client sends that much less, or half the deadline when that is less, so + /// the server never gets zero. `null` or [Duration.zero] sends the deadline as it is. + /// + /// A reachable server then times the call out first, and its `deadline_exceeded` arrives before the client's own + /// deadline fires. So a deadline that does fire on the client means the server's answer never came, which is what + /// makes [evictOnDeadline] reliable. Servers that call further services should take their own step off the deadline + /// they pass on. + final Duration? deadlineStep; + + /// Whether a call that hits its deadline on the client moves new calls off its connection, as a [readIdleTimeout] + /// does: if nothing has arrived from the call's origin, on any call, since the call last received anything, new + /// calls go to a fresh `URLSession` or Cronet engine. Calls already on the old connection are not canceled. + /// + /// Use it with [deadlineStep]. Without one, the server enforces the same deadline as the client, but its clock + /// starts later, so on a healthy connection the client's deadline usually fires before the server's + /// `deadline_exceeded` arrives, and the connection is evicted for nothing. + final bool evictOnDeadline; + /// Called with each response's headers, before its body is read. An exception it throws goes to /// `FlutterError.onError`, and the call goes on. final void Function(NativeHttpResponseInfo info)? onResponse; diff --git a/lib/src/url_session/url_session_http_client.dart b/lib/src/url_session/url_session_http_client.dart index f158e93..1817181 100644 --- a/lib/src/url_session/url_session_http_client.dart +++ b/lib/src/url_session/url_session_http_client.dart @@ -40,7 +40,7 @@ final class UrlSessionHttpClient extends ChannelHttpClient { UrlSessionHttpClient._(int engineId, NativeHttpConfig config) : super(urlSessionBridge, engineId, config); static Future create(NativeHttpConfig config) async { - final engineId = await ChannelHttpClient.createEngine(urlSessionBridge, { + final engineId = await ChannelHttpClient.createEngine(urlSessionBridge, config, { 'http3Origins': [ for (final origin in config.http3Origins) [origin.host, origin.port], ], diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 241f538..efd0539 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -248,6 +248,109 @@ void main() { await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.canceled))); }); + group('deadlines', () { + Future> cancelAfterDeadline(NativeHttpClient client) async { + final signal = TimeoutSignal(const Duration(milliseconds: 10)); + final response = client.httpClient(request(StreamController().stream, signal: signal)); + await settle(); + final id = plugin.lastCallId; + await Future.delayed(const Duration(milliseconds: 50)); + final cancel = plugin.callsTo('cancel').single.arguments as Map; + expect(cancel['callId'], id); + await plugin.emit({'callId': id, 'type': 'canceled'}); + await expectLater( + response, + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + return cancel; + } + + Future sentTimeout(NativeHttpClient client, String timeout) async { + final headers = Headers() + ..add('content-type', 'application/proto') + ..add('connect-timeout-ms', timeout); + final req = HttpRequest('https://example.test/svc/Method', 'POST', headers, null, null); + unawaited(client.httpClient(req).then((_) {}, onError: (_) {})); + await settle(); + final pairs = (plugin.callsTo('open').last.arguments as Map)['headers'] as List; + return pairs.cast>().firstWhere((pair) => pair[0] == 'connect-timeout-ms')[1] as String?; + } + + test('by default a deadline cancels the stream and leaves its connection alone', () async { + expect((await cancelAfterDeadline(client)).containsKey('evictConnection'), isFalse); + }); + + test("with evictOnDeadline, a deadline asks the plugin to leave the call's connection", () async { + final evicting = await CronetHttpClient.create(const NativeHttpConfig(evictOnDeadline: true)); + expect((await cancelAfterDeadline(evicting))['evictConnection'], isTrue); + }); + + test('a call still receiving within the server\'s head start does not evict', () async { + final both = await CronetHttpClient.create( + const NativeHttpConfig(deadlineStep: Duration(seconds: 1), evictOnDeadline: true), + ); + Future> cancelArgs({required bool receiveLate}) async { + plugin.calls.clear(); + // 400 ms: the server gets 200 ms, half, so its head start is 200 ms. + final headers = Headers() + ..add('content-type', 'application/proto') + ..add('connect-timeout-ms', '400'); + final req = HttpRequest( + 'https://example.test/svc/Method', + 'POST', + headers, + StreamController().stream, + TimeoutSignal(const Duration(milliseconds: 400)), + ); + final response = both.httpClient(req); + await settle(); + final id = plugin.lastCallId; + if (receiveLate) { + await Future.delayed(const Duration(milliseconds: 330)); + await plugin.headers(id); + } + await Future.delayed(const Duration(milliseconds: 500)); + final cancel = plugin.callsTo('cancel').single.arguments as Map; + await plugin.emit({'callId': id, 'type': 'canceled'}); + await response.then((r) => r.body.drain()).catchError((_) {}); + return cancel; + } + + expect((await cancelArgs(receiveLate: false))['evictConnection'], isTrue); + expect((await cancelArgs(receiveLate: true)).containsKey('evictConnection'), isFalse); + }); + + test('a signal that is not a deadline leaves the connection alone', () async { + final evicting = await CronetHttpClient.create(const NativeHttpConfig(evictOnDeadline: true)); + final signal = CancelableSignal(); + unawaited(evicting.httpClient(request(null, signal: signal)).then((_) {}, onError: (_) {})); + await settle(); + signal.cancel('stop'); + await settle(); + expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('evictConnection'), isFalse); + }); + + test('by default the server gets the deadline as it is', () async { + expect(await sentTimeout(client, '5000'), '5000'); + }); + + test("with deadlineStep, the server's deadline is a step shorter than the client's", () async { + final stepping = await CronetHttpClient.create(const NativeHttpConfig(deadlineStep: Duration(seconds: 1))); + expect(await sentTimeout(stepping, '5000'), '4000'); + // Under two steps, half: the server never gets zero. + expect(await sentTimeout(stepping, '1500'), '750'); + expect(await sentTimeout(stepping, '1'), '1'); + expect(await sentTimeout(stepping, 'soon'), 'soon'); + }); + + test('a negative step is refused', () async { + await expectLater( + CronetHttpClient.create(const NativeHttpConfig(deadlineStep: Duration(seconds: -1))), + throwsArgumentError, + ); + }); + }); + test('cancelling the response body cancels the native stream', () async { final responseFuture = client.httpClient(request(null)); await settle(); @@ -302,6 +405,52 @@ void main() { await plugin.emit({'callId': id, 'type': 'done'}); }); + group('read-idle timeout', () { + Future> openArgs(NativeHttpClient client) async { + unawaited(client.httpClient(request(null)).then((_) {}, onError: (_) {})); + await settle(); + return plugin.callsTo('open').last.arguments as Map; + } + + test('is off unless configured', () async { + expect((await openArgs(client))['readIdleTimeoutMs'], isNull); + }); + + test('applies to every call of a client configured with it', () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); + expect((await openArgs(configured))['readIdleTimeoutMs'], 3000); + expect((await openArgs(configured))['readIdleTimeoutMs'], 3000); + }); + + test('zero is none', () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration.zero)); + expect((await openArgs(configured))['readIdleTimeoutMs'], isNull); + }); + + test('rounds up to whole milliseconds', () async { + final configured = await CronetHttpClient.create( + const NativeHttpConfig(readIdleTimeout: Duration(microseconds: 1)), + ); + expect((await openArgs(configured))['readIdleTimeoutMs'], 1); + }); + + test('a negative timeout is refused', () async { + await expectLater( + CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: -1))), + throwsArgumentError, + ); + }); + + test('a timeout fails the call as unavailable, with the platform code', () async { + final response = client.httpClient(request(null)); + await settle(); + await plugin.failed(plugin.lastCallId, streamReady: true, internalErrorCode: -7); + final cause = await failureCause(response); + expect(cause.platformCode, -7); + expect(cause.failure, NativeHttpFailure.other); + }); + }); + test('close shuts the engine down', () async { await client.close(); expect(plugin.callsTo('shutdownEngine').single.arguments['engineId'], 7); diff --git a/test_server/freeze.go b/test_server/freeze.go index 06704ac..d67a7c7 100644 --- a/test_server/freeze.go +++ b/test_server/freeze.go @@ -5,6 +5,7 @@ import ( "fmt" "net" "net/http" + "sync" "sync/atomic" "time" ) @@ -63,13 +64,93 @@ func (c *freezablePacketConn) SetWriteBuffer(bytes int) error { return errors.New("the underlying conn can't set its send buffer") } -// adminHandler lets tests on a device freeze the HTTP/3 path over plain HTTP: +// freezableListener hands out TCP connections that Freeze can black-hole, so a test can kill the HTTP/2 path of the +// connections a client already has, as a network change that leaves both ends up would. Neither URLSession nor +// Cronet notices such a connection on its own: nothing fails, and nothing arrives. +type freezableListener struct { + net.Listener + + mu sync.Mutex + conns map[*freezableConn]struct{} +} + +func newFreezableListener(l net.Listener) *freezableListener { + return &freezableListener{Listener: l, conns: map[*freezableConn]struct{}{}} +} + +func (l *freezableListener) Accept() (net.Conn, error) { + conn, err := l.Listener.Accept() + if err != nil { + return nil, err + } + c := &freezableConn{Conn: conn, listener: l} + l.mu.Lock() + l.conns[c] = struct{}{} + l.mu.Unlock() + return c, nil +} + +// Freeze black-holes every connection open now for d, then closes it: the bytes it dropped leave the connection +// unusable. Connections accepted later are unaffected, so a client that dials again gets through. A zero d closes +// the frozen connections now. It returns how many connections it froze. +func (l *freezableListener) Freeze(d time.Duration) int { + l.mu.Lock() + defer l.mu.Unlock() + n := 0 + for c := range l.conns { + if d <= 0 { + if c.frozen.Load() { + _ = c.Conn.Close() + } + continue + } + if c.frozen.CompareAndSwap(false, true) { + n++ + time.AfterFunc(d, func() { _ = c.Conn.Close() }) + } + } + return n +} + +type freezableConn struct { + net.Conn + listener *freezableListener + frozen atomic.Bool +} + +// Read drops everything that arrives while the connection is frozen. +func (c *freezableConn) Read(p []byte) (int, error) { + for { + n, err := c.Conn.Read(p) + if err != nil || !c.frozen.Load() { + return n, err + } + } +} + +// Write drops everything while the connection is frozen, as a dead path would. +func (c *freezableConn) Write(p []byte) (int, error) { + if c.frozen.Load() { + return len(p), nil + } + return c.Conn.Write(p) +} + +func (c *freezableConn) Close() error { + c.listener.mu.Lock() + delete(c.listener.conns, c) + c.listener.mu.Unlock() + return c.Conn.Close() +} + +// adminHandler lets tests on a device kill the server's network paths over plain HTTP: // -// POST /freeze?for=10s drop all UDP traffic for 10 s -// POST /unfreeze stop dropping now +// POST /freeze?for=10s drop all UDP traffic, so every HTTP/3 connection, for 10 s +// POST /freeze?for=10s&path=tcp black-hole the TCP connections open now (HTTP/2) for 10 s, then close them +// POST /unfreeze stop dropping UDP now, and close the frozen TCP connections // -// TCP is never frozen, so HTTP/2 calls are unaffected. -func adminHandler(conn *freezablePacketConn) http.Handler { +// udp is nil when the server has no HTTP/3. +func adminHandler(udp *freezablePacketConn, tcp *freezableListener) http.Handler { mux := http.NewServeMux() mux.HandleFunc("POST /freeze", func(w http.ResponseWriter, r *http.Request) { d, err := time.ParseDuration(r.URL.Query().Get("for")) @@ -78,11 +159,25 @@ func adminHandler(conn *freezablePacketConn) http.Handler { http.StatusBadRequest) return } - conn.Freeze(d) - fmt.Fprintf(w, "frozen for %s\n", d) + switch path := r.URL.Query().Get("path"); path { + case "", "udp": + if udp == nil { + http.Error(w, "no HTTP/3 path to freeze", http.StatusConflict) + return + } + udp.Freeze(d) + fmt.Fprintf(w, "udp frozen for %s\n", d) + case "tcp": + fmt.Fprintf(w, "%d tcp connections frozen for %s\n", tcp.Freeze(d), d) + default: + http.Error(w, fmt.Sprintf("path=%q: want udp or tcp", path), http.StatusBadRequest) + } }) mux.HandleFunc("POST /unfreeze", func(w http.ResponseWriter, r *http.Request) { - conn.Freeze(0) + if udp != nil { + udp.Freeze(0) + } + tcp.Freeze(0) fmt.Fprintln(w, "unfrozen") }) return mux diff --git a/test_server/main.go b/test_server/main.go index 10db368..8bb0b16 100644 --- a/test_server/main.go +++ b/test_server/main.go @@ -8,7 +8,7 @@ // // -max-idle and -keepalive set the server's QUIC idle timeout and ping interval. QUIC uses the smaller of the two // endpoints' idle timeouts, so -max-idle also bounds how long a client takes to notice a dead path. -admin-addr -// serves a plain-HTTP endpoint that freezes the HTTP/3 path (see adminHandler). +// serves a plain-HTTP endpoint that freezes the HTTP/3 path, or the open HTTP/2 connections (see adminHandler). package main import ( @@ -69,33 +69,39 @@ func main() { handler := protocolHeaders(newElizaMux(), *altSvc) tlsConfig := newTLSConfig(cert) h2 := &http.Server{Addr: *addr, Handler: handler, TLSConfig: tlsConfig, ReadHeaderTimeout: 10 * time.Second} + tcp, err := net.Listen("tcp", *addr) + if err != nil { + log.Fatalf("http2: %v", err) + } + tcpConns := newFreezableListener(tcp) go func() { - if err := h2.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) { + if err := h2.ServeTLS(tcpConns, "", ""); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("http2: %v", err) } }() var h3 *http3.Server + var udpConn *freezablePacketConn if !*noH3 { udp, err := net.ListenPacket("udp", *h3Addr) if err != nil { log.Fatalf("http3: %v", err) } - conn := &freezablePacketConn{PacketConn: udp} + udpConn = &freezablePacketConn{PacketConn: udp} h3 = newHTTP3Server(handler, tlsConfig, quicConfig(*maxIdle, *keepAlive)) go func() { - if err := h3.Serve(conn); err != nil && !errors.Is(err, http.ErrServerClosed) { + if err := h3.Serve(udpConn); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("http3: %v", err) } }() - if *adminAddr != "" { - admin := &http.Server{Addr: *adminAddr, Handler: adminHandler(conn), ReadHeaderTimeout: 10 * time.Second} - go func() { - if err := admin.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - log.Fatalf("admin: %v", err) - } - }() - } + } + if *adminAddr != "" { + admin := &http.Server{Addr: *adminAddr, Handler: adminHandler(udpConn, tcpConns), ReadHeaderTimeout: 10 * time.Second} + go func() { + if err := admin.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + log.Fatalf("admin: %v", err) + } + }() } log.Printf("listening: tcp %s, udp %s (h3 %t, max idle %s, keepalive %s), admin %q, Alt-Svc %q, CA in %s", *addr, *h3Addr, !*noH3, *maxIdle, *keepAlive, *adminAddr, *altSvc, *certDir) @@ -142,11 +148,13 @@ func newElizaMux() *http.ServeMux { } // protocolHeaders logs each request and reports the protocol it arrived on in X-Proto, which is how tests check -// for HTTP/3 on clients that don't expose the negotiated protocol. HTTP/2 responses also advertise HTTP/3. +// for HTTP/3 on clients that don't expose the negotiated protocol, and the client's address in X-Remote-Addr, which +// tells a test whether two calls shared a connection. HTTP/2 responses also advertise HTTP/3. func protocolHeaders(next http.Handler, altSvc string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Printf("%s %s %s from %s", r.Proto, r.Method, r.URL.Path, r.RemoteAddr) w.Header().Set("X-Proto", r.Proto) + w.Header().Set("X-Remote-Addr", r.RemoteAddr) if r.ProtoMajor < 3 && altSvc != "" { w.Header().Set("Alt-Svc", altSvc) } diff --git a/test_server/server_test.go b/test_server/server_test.go index 756b383..cf5008e 100644 --- a/test_server/server_test.go +++ b/test_server/server_test.go @@ -5,6 +5,7 @@ import ( "crypto/tls" "crypto/x509" "errors" + "fmt" "io" "net" "net/http" @@ -168,7 +169,7 @@ func TestFrozenPathFailsOpenStreamWithinIdleTimeout(t *testing.T) { func TestAdminFreezeAndUnfreeze(t *testing.T) { conn := &freezablePacketConn{} - handler := adminHandler(conn) + handler := adminHandler(conn, newFreezableListener(nil)) post := func(target string) int { w := httptest.NewRecorder() handler.ServeHTTP(w, httptest.NewRequest(http.MethodPost, target, nil)) @@ -183,6 +184,96 @@ func TestAdminFreezeAndUnfreeze(t *testing.T) { if code := post("/freeze?for=nonsense"); code != http.StatusBadRequest || conn.frozen() { t.Fatalf("bad duration: status %d, frozen %t", code, conn.frozen()) } + if code := post("/freeze?for=1h&path=tcp"); code != http.StatusOK || conn.frozen() { + t.Fatalf("freeze tcp: status %d, udp frozen %t", code, conn.frozen()) + } + if code := post("/freeze?for=1h&path=sctp"); code != http.StatusBadRequest { + t.Fatalf("unknown path: status %d", code) + } +} + +// startHTTP2 serves the eliza handler over HTTP/2 on a free loopback TCP port, through a freezableListener, and +// returns a function that makes a client with its own connections. +func startHTTP2(t *testing.T) (func() elizav1connect.ElizaServiceClient, *freezableListener) { + t.Helper() + dir := t.TempDir() + cert, err := writeCertificates(dir, []string{"localhost", "127.0.0.1"}) + if err != nil { + t.Fatal(err) + } + caDER, err := os.ReadFile(filepath.Join(dir, "ca.der")) + if err != nil { + t.Fatal(err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + roots.AddCert(ca) + + tcp, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + listener := newFreezableListener(tcp) + server := &http.Server{Handler: protocolHeaders(newElizaMux(), ""), TLSConfig: newTLSConfig(cert)} + go func() { _ = server.ServeTLS(listener, "", "") }() + t.Cleanup(func() { _ = server.Close() }) + + baseURL := "https://" + tcp.Addr().String() + newClient := func() elizav1connect.ElizaServiceClient { + transport := &http.Transport{TLSClientConfig: &tls.Config{RootCAs: roots}, ForceAttemptHTTP2: true} + t.Cleanup(transport.CloseIdleConnections) + client := connect.NewClient(connecthttp.NewTransport(&http.Client{Transport: transport}, baseURL)) + return elizav1connect.NewElizaServiceClient(client) + } + return newClient, listener +} + +func TestFrozenTCPConnectionGoesSilentAndNewConnectionsWork(t *testing.T) { + newClient, listener := startHTTP2(t) + stream := openQuietConverse(t, newClient()) + if n := listener.Freeze(time.Minute); n != 1 { + t.Fatalf("froze %d connections, want 1", n) + } + t.Cleanup(func() { listener.Freeze(0) }) + + // Nothing fails on the frozen connection: the client's sentence and the server's echo both vanish. + received := make(chan error, 1) + go func() { + if err := stream.Send(&elizav1.ConverseRequest{Sentence: "anyone?"}); err != nil { + received <- err + return + } + res, err := stream.Receive() + if err == nil { + err = fmt.Errorf("received %q", res.GetSentence()) + } + received <- err + }() + select { + case err := <-received: + t.Fatalf("the frozen connection answered: %v", err) + case <-time.After(time.Second): + } + + // A connection dialed after the freeze works. + res, err := newClient().Say(timeoutContext(t), &elizav1.SayRequest{Sentence: "fresh"}) + if err != nil || res.GetSentence() != "say: fresh" { + t.Fatalf("Say on a new connection: %v, %v", res, err) + } + + // Unfreezing closes the frozen connection, which ends the stream. + listener.Freeze(0) + select { + case err := <-received: + if err == nil { + t.Fatal("the stream ended without an error") + } + case <-time.After(5 * time.Second): + t.Fatal("the stream outlived its closed connection") + } } func timeoutContext(t *testing.T) context.Context { @@ -379,5 +470,8 @@ func TestAltSvcOnlyOnTCPResponses(t *testing.T) { if got := w.Header().Get("Alt-Svc"); got != tc.wantAltSvc { t.Errorf("HTTP/%d: Alt-Svc = %q, want %q", tc.major, got, tc.wantAltSvc) } + if got := w.Header().Get("X-Remote-Addr"); got != req.RemoteAddr { + t.Errorf("HTTP/%d: X-Remote-Addr = %q, want %q", tc.major, got, req.RemoteAddr) + } } }