From a19af5788fa7d835e2aabac8295da36663aef03a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:43:30 +0000 Subject: [PATCH 1/7] Add a per-call read-idle timeout that evicts the dead connection A call fails as unavailable once it has received nothing from the server for its read-idle timeout. Set it per call with Headers.setReadIdleTimeout (carried in a header the client strips), or as a client default with NativeHttpConfig.readIdleTimeout. - iOS: URLRequest.timeoutInterval; NSURLErrorTimedOut (-1001). A 60 s value is sent as 60.001 s, since URLSession ignores a request's 60 s. - Android: a timer on the bridge's thread, reset by headers, data and trailers; net::ERR_TIMED_OUT (-7). - On a timeout the engine moves new calls to a fresh URLSession or CronetEngine, so they dial a new connection. The old one finishes its calls and is then released. The test server's /freeze?path=tcp black-holes the HTTP/2 connections open at the time. New integration cases use it to check, over HTTP/2 on both platforms, that a quiet stream fails at its timeout, a busy one doesn't, a dead connection goes unnoticed without a timeout, and with one the next call gets through on a new connection. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- README.md | 79 ++++++-- .../connect_native_http/CronetBridgeTest.kt | 86 ++++++++- .../connect_native_http/CronetBridge.kt | 169 ++++++++++++++---- example/integration_test/eliza_test.dart | 135 +++++++++++++- .../URLSessionBridge.swift | 70 ++++++-- .../URLSessionBridgeTests.swift | 45 ++++- lib/connect_native_http.dart | 1 + lib/src/channel/channel_http_client.dart | 21 ++- lib/src/cronet/cronet_http_client.dart | 2 +- lib/src/native_http_config.dart | 18 +- lib/src/read_idle_timeout.dart | 51 ++++++ .../url_session/url_session_http_client.dart | 2 +- test/cronet_http_client_test.dart | 75 +++++++- test_server/freeze.go | 111 +++++++++++- test_server/main.go | 30 ++-- test_server/server_test.go | 93 +++++++++- 16 files changed, 886 insertions(+), 102 deletions(-) create mode 100644 lib/src/read_idle_timeout.dart diff --git a/README.md b/README.md index 868b660..dfdaa07 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,34 @@ No exception in the package should crash the app or leave a call hanging. Failur `NativeHttpClient.create` throws a `PlatformException` if the platform can't create a client, for example when Cronet's native library fails to load. +### Read-idle timeout + +A call can fail once it has received nothing from the server for a set time. This is the only way either client +notices a dead HTTP/2 connection (see "Dead connections"). Set it per call, or as a default for every call: + +```dart +// One call. An interceptor can set it per procedure the same way. +await eliza.say(request, headers: Headers()..setReadIdleTimeout(const Duration(seconds: 10))); + +// Every call that doesn't set its own. Duration.zero on a call turns the default off for that call. +NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); +``` + +- **What it measures:** time since the call last received anything: headers, a body chunk or trailers. The timer starts + when the call does, so it also bounds the wait for the response headers. A unary call the server holds longer than + the timeout fails, and so does a stream that goes quiet for longer. +- **iOS:** `URLRequest.timeoutInterval`. URLSession treats a request's 60 s as unset and uses the session's value + instead, so the bridge sends 60.001 s for a 60 s timeout. +- **Android:** a timer in the bridge. +- **The failure:** `unavailable`, with a `NativeHttpException` whose `platformCode` is `NSURLErrorTimedOut` (-1001) on + iOS or `net::ERR_TIMED_OUT` (-7) on Android. On Android a timeout before `onStreamReady` is `notSent`, as with any + other failure there. +- **The connection:** a timeout is taken as a sign the connection is dead, and the connection is evicted. Neither stack + can close a single connection, so the client moves its new calls to a fresh `URLSession` or `CronetEngine`, which + dials its own connections. Calls already on the old one finish there, and it is released after the last one ends. +- **The header:** the value travels in a `native-http-read-idle-timeout-ms` request header (`readIdleTimeoutHeader`), + which the client removes before sending. connect-dart's `CallOptions` carry nothing else per call. + ## Server and infrastructure requirements The clients only reach HTTP/3, and only keep bidi working, if the server side meets these requirements. Each was @@ -73,9 +101,11 @@ found or verified with the tests in this package. ([target group protocol version](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html)). - **Listener rules must route the Connect content types above to that target group.** A rule that matches only `application/grpc` sends Connect requests elsewhere. -- **No client-side dead-connection detection.** Neither stack sends HTTP/2 pings the caller controls, and server - pings stop at any load balancer that terminates HTTP/2. Idle timeouts on the path must exceed a stream's longest - quiet period, or the stream must carry periodic messages. See "QUIC idle timeout vs HTTP/2 keepalive pings". +- **No dead-connection detection apart from the read-idle timeout.** Neither stack sends HTTP/2 pings the caller + controls, and server pings stop at any load balancer that terminates HTTP/2. A read-idle timeout notices a dead + connection, but only if it is shorter than a stream's quiet periods. Idle timeouts on the path must exceed a + stream's longest quiet period, or the stream must carry periodic messages. See "QUIC idle timeout vs HTTP/2 + keepalive pings". **The HTTP/3 endpoint** @@ -111,7 +141,7 @@ found or verified with the tests in this package. | Certificates | System trust store | System trust store; QUIC only with certificates that chain to a public root | | Negotiated protocol | Not exposed (`NativeHttpResponseInfo.protocol` is `null`) | `h2`, `quic/1+spdy/3` (HTTP/3) or `http/1.1` | | QUIC idle timeout | No setting; the server's advertised value applies | `NativeHttpConfig.quicIdleTimeout`, in whole seconds | -| Request idle timer | A task that receives nothing for a day fails; a quiet stream otherwise stays open | None | +| Read-idle timeout | Per call, through `URLRequest.timeoutInterval`. Without one, a task that receives nothing for a day fails | Per call, through a timer in the bridge. None without one | | Redirects | Followed only for requests without a body; otherwise the 3xx response is returned, since a streamed body can't be resent | Not tested | ### Dead connections @@ -136,9 +166,13 @@ The integration tests measure both: Clients may enforce a floor on the peer's value. quic-go's client raises anything under 5 s to 5 s (`protocol.MinRemoteIdleTimeout`), so measure any other client before relying on a short server value. -This does nothing on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP -connection that dies without the interface going down goes unnoticed until a call's deadline or an -application-level heartbeat. +None of this works on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP +connection that dies without the interface going down goes unnoticed until a call's deadline, an application-level +heartbeat, or a read-idle timeout. The integration tests check both sides: + +- Without a read-idle timeout, a frozen HTTP/2 connection draws no error for 3 s, and a new call on the same client + waits on it until its deadline. +- With one, the call fails within the timeout, and the client's next call goes out on a new connection. ### QUIC idle timeout vs HTTP/2 keepalive pings @@ -151,7 +185,7 @@ not over HTTP/2. | Who sets it | Both advertise a value; the connection uses the smaller (RFC 9000 §10.1) | Each side sets its own pings | | Can a server setting make the client notice? | Yes. A short server value becomes the client's timeout, and the server's pings stop arriving once the path dies. | No. A server that detects a dead connection can't tell the client, because nothing gets through. | | Through a load balancer | NLB QUIC passthrough is end to end, so the client sees the server's pings | An ALB ends HTTP/2 on each side, so the server's pings stop at the ALB. AWS doesn't document the ALB pinging clients, and nothing here relies on it. | -| Client control here | Android: `quicIdleTimeout`. iOS: none. | None. Neither URLSession nor Cronet exposes HTTP/2 ping settings. | +| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout per call. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout per call. | | What a client that sends its own pings gets, for comparison | — | `package:http2` with 3 s pings and a 3 s timeout notices in about 6 s | So for servers: @@ -161,8 +195,8 @@ So for servers: - Too long an idle timeout delays detection. - Too few pings closes quiet connections that are healthy. - **HTTP/2:** server pings, such as `http.HTTP2Config.SendPingTimeout` in Go, only clean up the server's side. - - On the client, a silently dead connection lasts until a call deadline, an application-level heartbeat, or TCP's - own retransmission timeout, which takes minutes. + - On the client, a silently dead connection lasts until a call deadline, a read-idle timeout, an + application-level heartbeat, or TCP's own retransmission timeout, which takes minutes. - Every idle timeout on the path, such as a load balancer's, must exceed the longest quiet period of a stream, or the stream must carry periodic messages. @@ -197,9 +231,14 @@ Neither platform finishes a stream on its own while the client still has its req - **`test_server/`:** a Go server for `connectrpc.eliza.v1.ElizaService` over HTTP/2 and HTTP/3, scripted for tests (see `eliza.go`). - `-max-idle` and `-keepalive` set its QUIC idle timeout and ping interval. - - `-admin-addr` serves `POST /freeze?for=10s` and `POST /unfreeze`, which drop all its UDP traffic so a test can - simulate a dead HTTP/3 path. - - Its own tests (`go test ./...`) check it over HTTP/3, including the idle timeout, pings and freezing. + - `-admin-addr` serves an endpoint that simulates dead paths: + - `POST /freeze?for=10s` drops all its UDP traffic, so every HTTP/3 connection. + - `POST /freeze?for=10s&path=tcp` black-holes the TCP connections open at that moment, so the HTTP/2 connections + clients already have. It drops what they send and what the server writes, then closes them. Connections opened + later work, so a client that dials again gets through. + - `POST /unfreeze` ends both. + - Its own tests (`go test ./...`) check it over HTTP/3, including the idle timeout, pings and freezing, and check + the TCP freeze over HTTP/2. - It creates a CA in `test_server/certs/` on first start and reuses it, so a device only has to trust it once. - With `QLOGDIR` set, it writes a qlog trace of each QUIC connection there, which shows the server's side of a handshake. @@ -225,12 +264,20 @@ Neither platform finishes a stream on its own while the client still has its req The script starts the server with a 3 s idle timeout and 1 s pings. Over HTTP/3 (iOS) the tests also check two things: a quiet stream outlives the idle timeout, and a frozen path fails an open stream within it. + + The read-idle timeout is tested over HTTP/2 on both platforms. On iOS each of these cases freezes the server's UDP + first, so URLSession falls back to TCP as it would where UDP is blocked. The cases check four things: + - A quiet stream fails at its timeout. + - A stream that keeps receiving outlives its timeout. + - Without a timeout, a frozen connection goes unnoticed. This is the control. + - With one, the call fails and the client's next call works, which it can only do on a new connection. - **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's bridge (`URLSessionBridge`, `CronetBridge`) has no Flutter dependency, and the tests call it directly, for behavior Dart can't reach: - **iOS (`ios/connect_native_http/Tests`, XCTest):** a write stuck on a full buffer returns once the call is canceled or its connection fails; a redirect of a request with a body is returned, not followed; shutting a - session down lets its calls finish and refuses new ones. The script runs them on the simulator. For a quick loop + session down lets its calls finish and refuses new ones; a read-idle timeout fails a quiet call with + `NSURLErrorTimedOut`, and the engine's next call works. The script runs them on the simulator. For a quick loop on the Mac, against a running test server: ```bash @@ -240,7 +287,9 @@ Neither platform finishes a stream on its own while the client still has its req - **Android (`android/src/androidTest`, instrumented):** writes sent before Cronet is ready go out once it is; a write after the server finished the call is dropped instead of crashing; shutting an engine down lets its calls - finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped. + finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped; a + read-idle timeout fails a quiet call with -7, once, and the engine's next call works; data arriving resets the + timer. - Both platforms' suites also check that malformed method calls get an error reply and leave the bridge working. - **`example/integration_test/public_server_test.dart`:** HTTP/3 on Android against a server with a public certificate. It also checks that `quicIdleTimeout` ends a quiet connection, against a control without it. It is diff --git a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt index af1789b..dc92ad5 100644 --- a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt +++ b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt @@ -91,6 +91,41 @@ class CronetBridgeTest { assertTrue(received(1).contains("say: hi")) } + @Test + fun aReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() { + val engine = createEngine() + open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 500) + await(1, "data") + val start = System.nanoTime() + val outcome = awaitOutcome(1) + val elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) + assertEquals("failed", outcome["type"]) + assertEquals(-7, outcome["internalErrorCode"]) + assertEquals(true, outcome["streamReady"]) + assertTrue("a 500 ms read-idle timeout took $elapsedMs ms", elapsedMs < 2000) + // Cronet's own cancel follows, and changes nothing. + Thread.sleep(200) + assertEquals(listOf(outcome), events.filter { it["callId"] == 1 && it["type"] in OUTCOMES } + drained(1)) + + // New calls go to a fresh CronetEngine. + assertSayWorks(engine, 2) + } + + @Test + fun receivingResetsTheReadIdleTimer() { + val engine = createEngine() + open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 1000) + await(1, "data") + // Twice the timeout in all, 600 ms between replies. + for (i in 0 until 3) { + Thread.sleep(600) + invoke("write", mapOf("callId" to 1, "data" to envelope(sentence("ping $i")))) + awaitReceived(1, "echo: ping $i") + } + invoke("write", mapOf("callId" to 1, "data" to envelope(sentence("bye")))) + assertEquals("done", awaitOutcome(1)["type"]) + } + @Test fun malformedArgumentsAreRefusedInsteadOfCrashing() { val engine = createEngine() @@ -101,6 +136,8 @@ class CronetBridgeTest { "end" to mapOf("callId" to null), "cancel" to emptyMap(), "shutdownEngine" to mapOf("engineId" to 1.5), + "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to 0), + "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to "soon"), ) for ((method, arguments) in malformed) { val reply = reply(method, arguments) @@ -139,17 +176,27 @@ class CronetBridgeTest { path: String, contentType: String, extraHeaders: List> = emptyList(), + readIdleTimeoutMs: Int? = null, ) { - invoke("open", mapOf( - "engineId" to engine, - "callId" to callId, - "url" to serverUrl + path, - "method" to "POST", - "headers" to listOf(listOf("content-type", contentType), listOf("connect-protocol-version", "1")) + - extraHeaders, - )) + invoke("open", openArguments(engine, callId, path, contentType, extraHeaders) + + ("readIdleTimeoutMs" to readIdleTimeoutMs)) } + private fun openArguments( + engine: Int, + callId: Int, + path: String, + contentType: String, + extraHeaders: List> = emptyList(), + ): Map = mapOf( + "engineId" to engine, + "callId" to callId, + "url" to serverUrl + path, + "method" to "POST", + "headers" to listOf(listOf("content-type", contentType), listOf("connect-protocol-version", "1")) + + extraHeaders, + ) + // Each case opens a fresh connection to the host through the emulator, which once took 3 s to come up on CI (the // request reached the server just as a 3 s wait gave up). 10 s, like the Dart cases, still fails a real hang fast. private fun await(callId: Int, vararg types: String): Map { @@ -165,7 +212,27 @@ class CronetBridgeTest { } } - private fun awaitOutcome(callId: Int) = await(callId, "done", "failed", "canceled") + private fun awaitOutcome(callId: Int) = await(callId, *OUTCOMES.toTypedArray()) + + // Outcomes for [callId] that arrived since the last wait. + private fun drained(callId: Int): List> { + val more = mutableListOf>() + incoming.drainTo(more) + events.addAll(more) + return more.filter { it["callId"] == callId && it["type"] in OUTCOMES } + } + + private fun awaitReceived(callId: Int, text: String) { + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10) + while (!received(callId).contains(text)) { + val left = deadline - System.nanoTime() + val next = if (left > 0) incoming.poll(left, TimeUnit.NANOSECONDS) else null + if (next == null) { + fail("timed out waiting for \"$text\" on call $callId; events: $events") + } + events.add(next!!) + } + } // The response body so far, as text; Connect framing and protobuf leave the sentences readable. private fun received(callId: Int): String = events @@ -175,6 +242,7 @@ class CronetBridgeTest { private companion object { const val CONVERSE = "/connectrpc.eliza.v1.ElizaService/Converse" const val SAY = "/connectrpc.eliza.v1.ElizaService/Say" + val OUTCOMES = setOf("done", "failed", "canceled") // A SayRequest or ConverseRequest: field 1, the sentence. fun sentence(text: String): ByteArray { diff --git a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt index 88c343a..f6cbb3e 100644 --- a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt +++ b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt @@ -11,8 +11,10 @@ import org.chromium.net.NetworkException import org.chromium.net.QuicOptions import org.chromium.net.UrlResponseInfo import java.nio.ByteBuffer -import java.util.concurrent.ExecutorService import java.util.concurrent.Executors +import java.util.concurrent.ScheduledExecutorService +import java.util.concurrent.ScheduledFuture +import java.util.concurrent.TimeUnit // Bridges Cronet BidirectionalStream to lib/src/cronet/cronet_http_client.dart, one stream per call. Dart sends // request chunks as method calls; response headers, body chunks, trailers and the outcome go to [emit], from the @@ -29,20 +31,32 @@ class CronetBridge( private val main = Handler(Looper.getMainLooper()) - // Cronet callbacks and all state below run on this one thread, so none of it needs locking. - private val executor: ExecutorService = Executors.newSingleThreadExecutor() + // Cronet callbacks, read-idle timers and all state below run on this one thread, so none of it needs locking. + private val executor: ScheduledExecutorService = Executors.newSingleThreadScheduledExecutor() private val engines = HashMap() private val calls = HashMap() private var nextEngineId = 1 - private class Engine(val cronet: CronetEngine) { + private class EngineOptions(val quicHints: List>, val quicIdleTimeoutSeconds: Int?) + + // One CronetEngine and the calls on it. An engine whose call hit its read-idle timeout is retired: new calls go to + // a fresh one, with its own connections, and the retired one shuts down once its last call ends. + private class Pool(val cronet: CronetEngine) { val callIds = HashSet() + } + + private class Engine(val options: EngineOptions, var current: Pool) { + val retired = ArrayList() var shuttingDown = false } - private class Call(val callId: Int, val engineId: Int) { + private class Call(val callId: Int, val engineId: Int, val pool: Pool, val readIdleTimeoutNanos: Long?) { lateinit var stream: BidirectionalStream + // When the call last received anything from the server, or started, for its read-idle timeout. + var lastReceived = System.nanoTime() + var readIdleTimer: ScheduledFuture<*>? = null + // No request body is written before Cronet reports the stream ready, which is what lets Dart treat a // failure before this point as "never sent". var ready = false @@ -93,22 +107,14 @@ class CronetBridge( private fun dispatch(method: String, args: Map, result: (Reply) -> Unit) { when (method) { "createEngine" -> { - val quicHints = (args["quicHints"] as List>?).orEmpty() - val quicIdleTimeoutSeconds = args["quicIdleTimeoutSeconds"] as Int? + val options = EngineOptions( + (args["quicHints"] as List>?).orEmpty().map { it[0] as String to it[1] as Int }, + args["quicIdleTimeoutSeconds"] as Int?, + ) executor.execute { try { - val builder = CronetEngine.Builder(context).enableHttp2(true).enableQuic(true) - for (hint in quicHints) { - val port = hint[1] as Int - builder.addQuicHint(hint[0] as String, port, port) - } - if (quicIdleTimeoutSeconds != null) { - builder.setQuicOptions( - QuicOptions.builder().setIdleConnectionTimeoutSeconds(quicIdleTimeoutSeconds.toLong()), - ) - } val id = nextEngineId++ - engines[id] = Engine(builder.build()) + engines[id] = Engine(options, Pool(build(options))) result(Reply.Success(id)) } catch (e: Throwable) { // Throwable, not Exception: failing to load Cronet's native library is an UnsatisfiedLinkError. @@ -133,7 +139,10 @@ class CronetBridge( val url = args["url"] as String val httpMethod = args["method"] as String val headers = (args["headers"] as List>).map { it[0] to it[1] } - execute(callId) { open(engineId, callId, url, httpMethod, headers) } + // A Dart int arrives as an Int or, past 32 bits, a Long. + val readIdleTimeoutMs = (args["readIdleTimeoutMs"] as Number?)?.toLong() + require(readIdleTimeoutMs == null || readIdleTimeoutMs > 0) { "readIdleTimeoutMs: $readIdleTimeoutMs" } + execute(callId) { open(engineId, callId, url, httpMethod, headers, readIdleTimeoutMs) } result(Reply.Success(null)) } "write" -> { @@ -156,14 +165,35 @@ class CronetBridge( } } - private fun open(engineId: Int, callId: Int, url: String, method: String, headers: List>) { + private fun build(options: EngineOptions): CronetEngine { + val builder = CronetEngine.Builder(context).enableHttp2(true).enableQuic(true) + for ((host, port) in options.quicHints) { + builder.addQuicHint(host, port, port) + } + if (options.quicIdleTimeoutSeconds != null) { + builder.setQuicOptions( + QuicOptions.builder().setIdleConnectionTimeoutSeconds(options.quicIdleTimeoutSeconds.toLong()), + ) + } + return builder.build() + } + + private fun open( + engineId: Int, + callId: Int, + url: String, + method: String, + headers: List>, + readIdleTimeoutMs: Long?, + ) { val engine = engines[engineId] if (engine == null || engine.shuttingDown) { emit(failure(callId, false, "engine $engineId is shut down", null)) return } - val c = Call(callId, engineId) - val builder = engine.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) + val pool = engine.current + val c = Call(callId, engineId, pool, readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }) + val builder = pool.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) for ((name, value) in headers) { builder.addHeader(name, value) } @@ -172,7 +202,68 @@ class CronetBridge( // once it has started, so abort() then reports it as never sent, with no stream to release. c.stream.start() calls[callId] = c - engine.callIds.add(callId) + pool.callIds.add(callId) + c.lastReceived = System.nanoTime() + scheduleReadIdleCheck(c, c.readIdleTimeoutNanos) + } + + // Read-idle timeout. Rather than rescheduling on every chunk received, each check finds when the call last received + // something and, unless the timeout has run out since, schedules the next check for when it would. + private fun scheduleReadIdleCheck(c: Call, delayNanos: Long?) { + if (delayNanos == null) { + return + } + val check = Runnable { + try { + checkReadIdle(c) + } catch (e: Exception) { + Log.e(TAG, "call ${c.callId} failed in the bridge", e) + abort(c.callId, e) + } + } + c.readIdleTimer = executor.schedule(check, delayNanos, TimeUnit.NANOSECONDS) + } + + private fun checkReadIdle(c: Call) { + val timeout = c.readIdleTimeoutNanos ?: return + if (c.reported || calls[c.callId] !== c) { + return + } + val idle = System.nanoTime() - c.lastReceived + if (idle < timeout) { + scheduleReadIdleCheck(c, timeout - idle) + return + } + val message = "read idle timeout: nothing received for ${TimeUnit.NANOSECONDS.toMillis(idle)} ms" + report(c, failure(c.callId, c.ready, message, ERR_TIMED_OUT)) + // Cronet's onCanceled then releases the call and the stream. + c.stream.cancel() + evict(c) + } + + private fun received(c: Call) { + c.lastReceived = System.nanoTime() + } + + // Moves the engine's new calls off the connection [c] timed out on. Cronet has no call that closes one + // connection, so the engine's CronetEngine is retired: new calls go to a fresh one, which dials its own + // connections, and the old one shuts down once the calls still on it end. Calls that time out together on the same + // dead connection retire it once. + private fun evict(c: Call) { + val engine = engines[c.engineId] ?: return + if (engine.current !== c.pool || engine.shuttingDown) { + return + } + val fresh = try { + Pool(build(engine.options)) + } catch (e: Throwable) { + // Keep the engine there is; new calls on a dead connection fail on their own timeouts. + Log.e(TAG, "engine ${c.engineId}: could not replace the CronetEngine after a read idle timeout", e) + return + } + engine.retired.add(engine.current) + engine.current = fresh + Log.i(TAG, "engine ${c.engineId}: call ${c.callId} hit its read idle timeout; new calls use a fresh CronetEngine") } // Fails a call the bridge itself could not carry on with, after an exception on its thread. @@ -224,8 +315,9 @@ class CronetBridge( // Cronet's final callback for a call: releases it and, unless the bridge has already failed it, reports [event]. private fun finish(c: Call, event: Map) { + c.readIdleTimer?.cancel(false) calls.remove(c.callId) - engines[c.engineId]?.callIds?.remove(c.callId) + c.pool.callIds.remove(c.callId) report(c, event) shutdownIfIdle(c.engineId) } @@ -237,16 +329,25 @@ class CronetBridge( } } + // Shuts down the engine's retired CronetEngines that no call uses any more, and the whole engine once it is shutting + // down and idle. private fun shutdownIfIdle(engineId: Int) { val engine = engines[engineId] ?: return - if (engine.shuttingDown && engine.callIds.isEmpty()) { + val idle = engine.retired.filter { it.callIds.isEmpty() } + engine.retired.removeAll(idle) + idle.forEach { shutdown(engineId, it) } + if (engine.shuttingDown && engine.current.callIds.isEmpty() && engine.retired.isEmpty()) { engines.remove(engineId) - try { - engine.cronet.shutdown() - } catch (e: Exception) { - // Cronet refuses while it still counts a request as running. The engine is unreachable either way. - Log.w(TAG, "engine $engineId did not shut down", e) - } + shutdown(engineId, engine.current) + } + } + + private fun shutdown(engineId: Int, pool: Pool) { + try { + pool.cronet.shutdown() + } catch (e: Exception) { + // Cronet refuses while it still counts a request as running. The engine is unreachable either way. + Log.w(TAG, "engine $engineId did not shut down", e) } } @@ -269,6 +370,7 @@ class CronetBridge( } override fun onResponseHeadersReceived(stream: BidirectionalStream, info: UrlResponseInfo) { + received(c) emit(mapOf( "callId" to c.callId, "type" to "headers", @@ -285,6 +387,7 @@ class CronetBridge( buffer: ByteBuffer, endOfStream: Boolean, ) { + received(c) buffer.flip() if (buffer.hasRemaining()) { val bytes = ByteArray(buffer.remaining()) @@ -315,6 +418,7 @@ class CronetBridge( info: UrlResponseInfo, trailers: UrlResponseInfo.HeaderBlock, ) { + received(c) emit(mapOf("callId" to c.callId, "type" to "trailers", "headers" to headerPairs(trailers.asList))) } @@ -341,5 +445,8 @@ class CronetBridge( private companion object { const val TAG = "connect_native_http" const val READ_BUFFER_BYTES = 32 * 1024 + + // Chromium's net::ERR_TIMED_OUT (net/base/net_error_list.h), reported for a read idle timeout. + const val ERR_TIMED_OUT = -7 } } diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index c41cadc..fc24f62 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -38,10 +38,14 @@ const introduceInterval = Duration(milliseconds: 100); final class Harness { Harness._(this.native, this.eliza); - static Future create(String baseUrl, {Set http3Origins = const {}}) async { + static Future create(String baseUrl, {Set http3Origins = const {}, Duration? readIdleTimeout}) async { late final Harness harness; final native = await NativeHttpClient.create( - NativeHttpConfig(http3Origins: http3Origins, onResponse: (info) => harness.platformProtocols.add(info.protocol)), + NativeHttpConfig( + http3Origins: http3Origins, + readIdleTimeout: readIdleTimeout, + onResponse: (info) => harness.platformProtocols.add(info.protocol), + ), ); if (native == null) { throw StateError('no native client on this platform'); @@ -82,9 +86,14 @@ Future admin(String path) async { } /// Opens Converse and reads the server's greeting. Nothing flows afterwards until the caller sends. -Future<(StreamController, StreamIterator)> openQuietConverse(Harness harness) async { +Future<(StreamController, StreamIterator)> openQuietConverse( + Harness harness, { + Headers? headers, +}) async { final requests = StreamController(); - final responses = StreamIterator(harness.eliza.converse(requests.stream, onHeader: harness.recordHeaders)); + final responses = StreamIterator( + harness.eliza.converse(requests.stream, headers: headers, onHeader: harness.recordHeaders), + ); expect(await responses.moveNext(), isTrue); expect(responses.current.sentence, 'hello'); return (requests, responses); @@ -119,6 +128,25 @@ Future harnessNegotiating(String protocol) async { } } +/// A client on HTTP/2, with the server's UDP frozen for the rest of the case so that QUIC can't connect: URLSession +/// then falls back to TCP, as on a network that blocks UDP. Android is on HTTP/2 with this server anyway. +Future harnessOverHttp2({Duration? readIdleTimeout}) async { + addTearDown(() => admin('/unfreeze')); + await admin('/freeze?for=60s'); + final harness = await Harness.create(serverUrl, readIdleTimeout: readIdleTimeout); + addTearDown(harness.native.close); + await harness.eliza.say( + SayRequest(sentence: 'which protocol'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: harness.recordHeaders, + ); + expect(harness.serverProtocols.last, 'HTTP/2.0'); + return harness; +} + +/// The platform code of a read-idle timeout: `NSURLErrorTimedOut` on iOS, `net::ERR_TIMED_OUT` from the Cronet bridge. +final readIdleTimeoutCode = io.Platform.isIOS ? -1001 : -7; + Future nativeCause(Future call) async { try { await call; @@ -316,6 +344,105 @@ void main() { expect(h3.serverProtocols.last, 'HTTP/3.0'); }, timeout: http3CaseTimeout); + // Over HTTP/2 neither platform notices a dead connection (no pings), so a read-idle timeout is what does. + bool skipUnlessAdmin() { + if (adminUrl.isEmpty) { + markTestSkipped('needs ADMIN_URL'); + return true; + } + return false; + } + + // Up to 8 s for the first call over HTTP/2 on top of the case itself. + const http2CaseTimeout = Duration(seconds: 25); + + testCase('a read-idle timeout ends a quiet HTTP/2 stream', (_) async { + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(); + const timeout = Duration(seconds: 1); + final (requests, responses) = await openQuietConverse(h2, headers: Headers()..setReadIdleTimeout(timeout)); + expect(h2.serverProtocols.last, 'HTTP/2.0'); + + // Timed from the greeting, the last thing the stream received. + final stopwatch = Stopwatch()..start(); + final cause = await nativeCause(responses.moveNext()); + expect(cause.platformCode, readIdleTimeoutCode, reason: '$cause'); + expect(cause.failure, NativeHttpFailure.other, reason: '$cause'); + expect(stopwatch.elapsed, greaterThan(timeout * 0.5)); + expect(stopwatch.elapsed, lessThan(timeout + const Duration(seconds: 2))); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('an HTTP/2 stream that keeps receiving outlives its read-idle timeout', (_) async { + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(); + const timeout = Duration(seconds: 1); + final (requests, responses) = await openQuietConverse(h2, headers: Headers()..setReadIdleTimeout(timeout)); + // Three times the timeout in all, but never more than 0.6 of it between replies. + for (var i = 0; i < 5; i++) { + await Future.delayed(timeout * 0.6); + requests.add(ConverseRequest(sentence: 'ping $i')); + expect(await responses.moveNext(), isTrue); + expect(responses.current.sentence, 'echo: ping $i'); + } + requests.add(ConverseRequest(sentence: 'bye')); + expect(await responses.moveNext(), isTrue); + expect(await responses.moveNext(), isFalse); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('without a read-idle timeout, a dead HTTP/2 connection goes unnoticed', (_) async { + // The control for the next case: what the read-idle timeout is for. + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(); + final (requests, responses) = await openQuietConverse(h2); + await admin('/freeze?for=60s&path=tcp'); + + requests.add(ConverseRequest(sentence: 'anyone?')); + final reply = responses.moveNext(); + await expectLater(reply.timeout(const Duration(seconds: 3)), throwsA(isA())); + // A new call goes to the same pooled connection, and waits until its deadline. + await expectLater( + h2.eliza.say(SayRequest(sentence: 'hi'), signal: TimeoutSignal(const Duration(seconds: 2))), + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + unawaited(reply.then((_) {}, onError: (_) {})); + await requests.close(); + }, timeout: http2CaseTimeout); + + testCase('a read-idle timeout notices a dead HTTP/2 connection, and the next call dials a new one', (_) async { + if (skipUnlessAdmin()) { + return; + } + // The client-wide default this time, rather than per call. + const timeout = Duration(seconds: 2); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); + await admin('/freeze?for=60s&path=tcp'); + + requests.add(ConverseRequest(sentence: 'anyone?')); + final stopwatch = Stopwatch()..start(); + final cause = await nativeCause(responses.moveNext()); + expect(cause.platformCode, readIdleTimeoutCode, reason: '$cause'); + expect(stopwatch.elapsed, lessThan(timeout + const Duration(seconds: 2))); + await requests.close(); + + // The frozen connection is still open, so this only works on a new one. + final res = await h2.eliza.say( + SayRequest(sentence: 'after'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: h2.recordHeaders, + ); + expect(res.sentence, 'say: after'); + expect(h2.serverProtocols.last, 'HTTP/2.0'); + }, timeout: http2CaseTimeout); + testCase('a dead HTTP/3 path fails an open stream', (_) async { if (skipUnlessHttp3()) { return; diff --git a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift index da137a6..b757374 100644 --- a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift +++ b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift @@ -33,7 +33,8 @@ public final class URLSessionBridge { private var nextSessionId = 1 private final class Session { - let urlSession: URLSession + // Replaced after a call times out, so new calls dial a fresh connection; see `evictConnection`. + var urlSession: URLSession let http3Origins: Set init(urlSession: URLSession, http3Origins: Set) { @@ -44,6 +45,9 @@ public final class URLSessionBridge { private final class Call { let callId: Int + var engineId = 0 + // The URLSession the call's task runs in. + var urlSession: URLSession? var task: URLSessionTask! var input: InputStream? var output: OutputStream? @@ -90,17 +94,9 @@ public final class URLSessionBridge { return "\(origin[0]):\(origin[1])" } queue.async { - let configuration = URLSessionConfiguration.ephemeral - // URLSession fails a task that receives nothing for this long. Its default of 60 s would end a quiet stream - // or a long-running unary call; callers bound calls with deadlines instead. - configuration.timeoutIntervalForRequest = 24 * 60 * 60 - // Only a bridge with its own roots implements the trust challenge. With the system's roots there is nothing - // for the delegate to decide, so certificate checks are left entirely to the network stack. - let delegate = self.trustedRoots.isEmpty ? Delegate(bridge: self) : TrustingDelegate(bridge: self) - let session = URLSession(configuration: configuration, delegate: delegate, delegateQueue: self.delegateQueue) let id = self.nextSessionId self.nextSessionId += 1 - self.sessions[id] = Session(urlSession: session, http3Origins: Set(origins)) + self.sessions[id] = Session(urlSession: self.makeURLSession(), http3Origins: Set(origins)) reply(.success(id)) } case "shutdownEngine": @@ -117,7 +113,13 @@ public final class URLSessionBridge { let headers: [[String]] = try arg(args, "headers") guard headers.allSatisfy({ $0.count == 2 }) else { throw BadArgument(name: "headers") } let hasBody: Bool = try arg(args, "hasBody") - queue.async { self.open(engineId, callId, url, method, headers, hasBody) } + // Absent or null for none. + var readIdleTimeoutMs: Int? + if let value = args["readIdleTimeoutMs"], !(value is NSNull) { + guard let ms = value as? Int, ms > 0 else { throw BadArgument(name: "readIdleTimeoutMs") } + readIdleTimeoutMs = ms + } + queue.async { self.open(engineId, callId, url, method, headers, hasBody, readIdleTimeoutMs) } reply(.success(nil)) case "write": let callId: Int = try arg(args, "callId") @@ -153,8 +155,29 @@ public final class URLSessionBridge { ["callId": callId, "type": "failed", "domain": domain, "code": code, "message": message] } + private func makeURLSession() -> URLSession { + let configuration = URLSessionConfiguration.ephemeral + // URLSession fails a task that receives nothing for this long. Its default of 60 s would end a quiet stream or a + // long-running unary call; callers bound calls with deadlines, or set their own read-idle timeout per request. + configuration.timeoutIntervalForRequest = Self.sessionIdleTimeout + // Only a bridge with its own roots implements the trust challenge. With the system's roots there is nothing for + // the delegate to decide, so certificate checks are left entirely to the network stack. + let delegate = trustedRoots.isEmpty ? Delegate(bridge: self) : TrustingDelegate(bridge: self) + return URLSession(configuration: configuration, delegate: delegate, delegateQueue: delegateQueue) + } + + /// The `URLRequest.timeoutInterval` for a read-idle timeout of `ms` milliseconds, or nil for the session's. + static func requestTimeoutInterval(readIdleTimeoutMs ms: Int?) -> TimeInterval? { + guard let ms else { return nil } + let interval = TimeInterval(ms) / 1000 + // URLSession takes a request's 60 s, URLRequest's default, to mean "not set" and applies the session's value + // instead, however the request got it. One millisecond more is still honored. + return interval == 60 ? 60.001 : interval + } + private func open( - _ engineId: Int, _ callId: Int, _ url: String, _ method: String, _ headers: [[String]], _ hasBody: Bool + _ engineId: Int, _ callId: Int, _ url: String, _ method: String, _ headers: [[String]], _ hasBody: Bool, + _ readIdleTimeoutMs: Int? ) { guard let session = sessions[engineId], let target = URL(string: url) else { emit(failure( @@ -169,8 +192,15 @@ public final class URLSessionBridge { } let port = target.port ?? (target.scheme == "http" ? 80 : 443) request.assumesHTTP3Capable = session.http3Origins.contains("\(target.host ?? ""):\(port)") + // The request's read-idle timeout: URLSession fails the task once it has received nothing for this long, with + // NSURLErrorTimedOut. + if let interval = Self.requestTimeoutInterval(readIdleTimeoutMs: readIdleTimeoutMs) { + request.timeoutInterval = interval + } let c = Call(callId: callId) + c.engineId = engineId + c.urlSession = session.urlSession if hasBody { var input: InputStream? var output: OutputStream? @@ -248,9 +278,22 @@ public final class URLSessionBridge { emit(["callId": c.callId, "type": "canceled"]) return } + if error.domain == NSURLErrorDomain && error.code == NSURLErrorTimedOut { + evictConnection(of: c) + } emit(failure(c.callId, domain: error.domain, code: error.code, message: error.localizedDescription)) } + // A call that timed out is taken as a sign its connection is dead. URLSession pools connections per session and has + // no call that closes one, so the engine's new calls go to a fresh URLSession, which dials its own connections. The + // old one finishes the tasks it still has, and is then released. Calls that time out together on the same dead + // connection replace it once. + private func evictConnection(of c: Call) { + guard let session = sessions[c.engineId], let old = c.urlSession, session.urlSession === old else { return } + session.urlSession = makeURLSession() + old.finishTasksAndInvalidate() + } + fileprivate func serverTrust(_ challenge: URLAuthenticationChallenge) -> ( URLSession.AuthChallengeDisposition, URLCredential? ) { @@ -267,6 +310,9 @@ public final class URLSessionBridge { // 64 KiB, the bound pair's buffer: how much request body may wait for URLSession to read it. private static let bodyBufferBytes = 64 * 1024 + + // A day: in effect no read-idle timeout, for requests that don't set one. + private static let sessionIdleTimeout: TimeInterval = 24 * 60 * 60 } // URLSession keeps its delegate until the session is invalidated, so this holds the bridge weakly. diff --git a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift index 1978fa2..d0f10ae 100644 --- a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift +++ b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift @@ -89,11 +89,46 @@ final class URLSessionBridgeTests: XCTestCase { XCTAssertEqual(outcome(2)?["domain"] as? String, "ConnectNativeHttp") } + func testReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() async throws { + let engine = try await createEngine() + try await open( + engine, callId: 1, path: Self.converse, contentType: "application/connect+proto", readIdleTimeoutMs: 500) + try await waitUntil("the greeting arrived") { self.received(1).contains("hello") } + let start = Date() + try await waitUntil("the call timed out") { self.outcome(1) != nil } + let elapsed = Date().timeIntervalSince(start) + XCTAssertEqual(outcome(1)?["type"] as? String, "failed", "\(String(describing: outcome(1)))") + XCTAssertEqual(outcome(1)?["code"] as? Int, NSURLErrorTimedOut) + XCTAssertLessThan(elapsed, 2, "a 500 ms read-idle timeout took \(elapsed) s") + + // New calls go to a fresh URLSession. + try await open(engine, callId: 2, path: Self.say, contentType: "application/proto") + try await write(2, sentence("after")) + try await invoke("end", ["callId": 2]) + try await waitUntil("the next call ended") { self.outcome(2) != nil } + XCTAssertEqual(outcome(2)?["type"] as? String, "done", "\(String(describing: outcome(2)))") + XCTAssertTrue(received(2).contains("say: after")) + } + + func testRequestTimeoutIntervalAvoidsTheValueURLSessionIgnores() { + XCTAssertNil(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: nil)) + XCTAssertEqual(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: 1500), 1.5) + XCTAssertEqual(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: 60_000), 60.001) + } + func testMalformedArgumentsAreRefusedInsteadOfCrashing() async throws { let malformed: [(String, [String: Any])] = [ ("open", [:]), ("open", ["engineId": "1", "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true]), ("open", ["engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [["one"]], "hasBody": true]), + ("open", [ + "engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true, + "readIdleTimeoutMs": "soon", + ]), + ("open", [ + "engineId": 1, "callId": 1, "url": serverURL, "method": "POST", "headers": [[String]](), "hasBody": true, + "readIdleTimeoutMs": 0, + ]), ("write", ["callId": 1, "data": "not bytes"]), ("end", ["callId": "1"]), ("cancel", [:]), @@ -144,11 +179,15 @@ final class URLSessionBridgeTests: XCTestCase { return try await invoke("createEngine", ["http3Origins": [origin]]) as! Int } - private func open(_ engine: Int, callId: Int, path: String, contentType: String) async throws { - try await invoke("open", [ + private func open( + _ engine: Int, callId: Int, path: String, contentType: String, readIdleTimeoutMs: Int? = nil + ) async throws { + var arguments: [String: Any] = [ "engineId": engine, "callId": callId, "url": serverURL + path, "method": "POST", "headers": [["content-type", contentType], ["connect-protocol-version", "1"]], "hasBody": true, - ]) + ] + arguments["readIdleTimeoutMs"] = readIdleTimeoutMs + try await invoke("open", arguments) } private func write(_ callId: Int, _ data: Data) async throws { diff --git a/lib/connect_native_http.dart b/lib/connect_native_http.dart index a20569e..2174d56 100644 --- a/lib/connect_native_http.dart +++ b/lib/connect_native_http.dart @@ -4,3 +4,4 @@ library; export 'src/native_http_client.dart' show NativeHttpClient; export 'src/native_http_config.dart' show NativeHttpConfig, NativeHttpResponseInfo; export 'src/native_http_exception.dart' show NativeHttpException, NativeHttpFailure; +export 'src/read_idle_timeout.dart' show ReadIdleTimeoutHeaders, readIdleTimeoutHeader; diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index 1e7f1e8..3c8f6f5 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -7,6 +7,7 @@ import 'package:flutter/services.dart'; import '../native_http_client.dart'; import '../native_http_config.dart'; import '../native_http_exception.dart'; +import '../read_idle_timeout.dart'; import 'connect_end_stream.dart'; /// A platform plugin that speaks this file's channel protocol: `createEngine`, `shutdownEngine`, `open`, `write`, @@ -236,7 +237,15 @@ base class ChannelHttpClient implements NativeHttpClient { ChannelHttpClient(this._bridge, this._engineId, this._config); /// Starts the bridge's event routing and creates an engine with [arguments], returning its id. - static Future createEngine(PlatformBridge bridge, Map arguments) async { + static Future createEngine( + PlatformBridge bridge, + NativeHttpConfig config, + Map arguments, + ) async { + final readIdle = config.readIdleTimeout; + if (readIdle != null && readIdle.isNegative) { + throw ArgumentError.value(readIdle, 'readIdleTimeout', 'must not be negative'); + } _Dispatcher.of(bridge); return (await bridge.methods.invokeMethod('createEngine', arguments))!; } @@ -251,6 +260,12 @@ base class ChannelHttpClient implements NativeHttpClient { Future _send(HttpRequest req) async { final dispatcher = _Dispatcher.of(_bridge); final methods = _bridge.methods; + final int? readIdleMs; + try { + readIdleMs = readIdleTimeoutMs(req.header, _config.readIdleTimeout); + } on FormatException catch (error) { + throw ConnectException(Code.internal, '${_bridge.name}: ${error.message}: ${error.source}', cause: error); + } final call = _Call(Uri.parse(req.url)); final id = dispatcher.register(call, _config.onResponse); call.body.onCancel = () { @@ -271,9 +286,11 @@ base class ChannelHttpClient implements NativeHttpClient { 'url': req.url, 'method': req.method, 'headers': [ - for (final (:name, :value) in req.header.entries) [name, value], + for (final (:name, :value) in req.header.entries) + if (name != readIdleTimeoutHeader) [name, value], ], 'hasBody': req.body != null, + 'readIdleTimeoutMs': readIdleMs, }); } catch (error) { // The plugin refused the call, or isn't there (a background isolate, say): nothing was sent. diff --git a/lib/src/cronet/cronet_http_client.dart b/lib/src/cronet/cronet_http_client.dart index a00e89a..267d662 100644 --- a/lib/src/cronet/cronet_http_client.dart +++ b/lib/src/cronet/cronet_http_client.dart @@ -36,7 +36,7 @@ final class CronetHttpClient extends ChannelHttpClient { if (idle != null && idle < const Duration(seconds: 1)) { throw ArgumentError.value(idle, 'quicIdleTimeout', 'Cronet takes whole seconds; use at least 1 s'); } - final engineId = await ChannelHttpClient.createEngine(cronetBridge, { + final engineId = await ChannelHttpClient.createEngine(cronetBridge, config, { 'quicHints': [ for (final origin in config.http3Origins) [origin.host, origin.port], ], diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index 1a145f8..4e7f804 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -12,7 +12,7 @@ final class NativeHttpResponseInfo { /// Configuration shared by both platform clients. Server certificates are checked against the platform's trust /// store. final class NativeHttpConfig { - const NativeHttpConfig({this.http3Origins = const {}, this.quicIdleTimeout, this.onResponse}); + const NativeHttpConfig({this.http3Origins = const {}, this.quicIdleTimeout, this.readIdleTimeout, this.onResponse}); /// `https://host:port` origins to try HTTP/3 with from the first request, before an `Alt-Svc` response header has /// advertised it. Both platforms fall back to HTTP/2 over TCP on their own when QUIC fails. @@ -29,6 +29,22 @@ final class NativeHttpConfig { /// Android only. URLSession exposes no QUIC settings, so on iOS only the server's advertised idle timeout applies. final Duration? quicIdleTimeout; + /// The read-idle timeout of calls that don't set their own with `Headers.setReadIdleTimeout`: a call fails once it + /// has received nothing from the server for this long. `null` or [Duration.zero] means none. Both platforms apply it + /// per call, so it also works on the HTTP/2 fallback, where nothing else notices a dead connection. + /// + /// - **What resets it:** anything the call receives. It runs from when the call starts, so it also bounds the wait + /// for the response headers, and a unary call the server holds longer than this fails. A server stream that may go + /// quiet needs a longer timeout, or periodic messages. + /// - **iOS:** `URLRequest.timeoutInterval`. URLSession fails the task with `NSURLErrorTimedOut` (-1001). + /// - **Android:** a timer in the bridge, which fails the call with `net::ERR_TIMED_OUT` (-7). + /// - **The failure:** a `ConnectException` with code `unavailable`, whose cause is a [NativeHttpException] with + /// that `platformCode`. + /// - **The connection:** a call that timed out is taken as a sign its connection is dead. The client moves new calls + /// to a fresh `URLSession` or Cronet engine, so they dial a new connection, and lets the calls already on the old + /// one finish there. + final Duration? readIdleTimeout; + /// Called with each response's headers, before its body is read. An exception it throws goes to /// `FlutterError.onError`, and the call goes on. final void Function(NativeHttpResponseInfo info)? onResponse; diff --git a/lib/src/read_idle_timeout.dart b/lib/src/read_idle_timeout.dart new file mode 100644 index 0000000..993ccc5 --- /dev/null +++ b/lib/src/read_idle_timeout.dart @@ -0,0 +1,51 @@ +import 'package:connectrpc/connect.dart'; + +/// The request header that carries one call's read-idle timeout, in whole milliseconds. `0` means none. +/// +/// The client reads it and removes it, so it never reaches the server. Set it with [setReadIdleTimeout], in a call's +/// `headers` or from an interceptor. +const readIdleTimeoutHeader = 'native-http-read-idle-timeout-ms'; + +/// Gives the call that sends [headers] a read-idle timeout: the call fails if it receives nothing from the server for +/// [timeout]. [Duration.zero] turns the timeout off for this call, overriding `NativeHttpConfig.readIdleTimeout`. +/// +/// ```dart +/// final res = await eliza.say( +/// SayRequest(sentence: 'hi'), +/// headers: Headers()..setReadIdleTimeout(const Duration(seconds: 10)), +/// ); +/// ``` +/// +/// See `NativeHttpConfig.readIdleTimeout` for what counts as receiving, and what happens on a timeout. +extension ReadIdleTimeoutHeaders on Headers { + void setReadIdleTimeout(Duration timeout) { + if (timeout.isNegative) { + throw ArgumentError.value(timeout, 'timeout', 'must not be negative'); + } + // At least 1 ms, so a sub-millisecond timeout doesn't turn into "none". + final ms = timeout == Duration.zero ? 0 : (timeout.inMicroseconds / 1000).ceil(); + this[readIdleTimeoutHeader] = '$ms'; + } +} + +/// The read-idle timeout for a request with [headers], in whole milliseconds, or `null` for none: the request's +/// [readIdleTimeoutHeader] if it has one, else [fallback]. Throws a [FormatException] on a malformed header. +int? readIdleTimeoutMs(Headers headers, Duration? fallback) { + final values = headers.get(readIdleTimeoutHeader); + if (values == null || values.isEmpty) { + if (fallback == null || fallback == Duration.zero) { + return null; + } + return (fallback.inMicroseconds / 1000).ceil(); + } + // Repeated headers: the last one set wins, as with [Headers.operator []=]. + final value = values.last.trim(); + // Digits only: int.parse would also take a sign or a 0x prefix. + final ms = _digits.hasMatch(value) ? int.tryParse(value) : null; + if (ms == null) { + throw FormatException('$readIdleTimeoutHeader must be a whole number of milliseconds', value); + } + return ms == 0 ? null : ms; +} + +final _digits = RegExp(r'^[0-9]{1,15}$'); diff --git a/lib/src/url_session/url_session_http_client.dart b/lib/src/url_session/url_session_http_client.dart index f158e93..1817181 100644 --- a/lib/src/url_session/url_session_http_client.dart +++ b/lib/src/url_session/url_session_http_client.dart @@ -40,7 +40,7 @@ final class UrlSessionHttpClient extends ChannelHttpClient { UrlSessionHttpClient._(int engineId, NativeHttpConfig config) : super(urlSessionBridge, engineId, config); static Future create(NativeHttpConfig config) async { - final engineId = await ChannelHttpClient.createEngine(urlSessionBridge, { + final engineId = await ChannelHttpClient.createEngine(urlSessionBridge, config, { 'http3Origins': [ for (final origin in config.http3Origins) [origin.host, origin.port], ], diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 241f538..f40657b 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -64,10 +64,12 @@ class FakePlugin { /// Lets queued microtasks and platform-channel replies run. Future settle() => Future.delayed(Duration.zero); -HttpRequest request(Stream? body, {AbortSignal? signal}) => HttpRequest( +HttpRequest request(Stream? body, {AbortSignal? signal, Headers? headers}) => HttpRequest( 'https://example.test/svc/Method', 'POST', - Headers()..add('content-type', 'application/proto'), + Headers() + ..add('content-type', 'application/proto') + ..addAll(headers ?? Headers()), body, signal, ); @@ -302,6 +304,75 @@ void main() { await plugin.emit({'callId': id, 'type': 'done'}); }); + group('read-idle timeout', () { + Future> openArgs(NativeHttpClient client, {Headers? headers}) async { + unawaited(client.httpClient(request(null, headers: headers)).then((_) {}, onError: (_) {})); + await settle(); + return plugin.callsTo('open').last.arguments as Map; + } + + test('is off unless configured', () async { + expect((await openArgs(client))['readIdleTimeoutMs'], isNull); + }); + + test("is the client's default for calls that don't set one", () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); + expect((await openArgs(configured))['readIdleTimeoutMs'], 3000); + }); + + test("a call's own replaces the default, and its header doesn't reach the server", () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); + final args = await openArgs( + configured, + headers: Headers()..setReadIdleTimeout(const Duration(milliseconds: 1500)), + ); + expect(args['readIdleTimeoutMs'], 1500); + expect(args['headers'], [ + ['content-type', 'application/proto'], + ]); + }); + + test('zero turns the default off for one call', () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); + final args = await openArgs(configured, headers: Headers()..setReadIdleTimeout(Duration.zero)); + expect(args['readIdleTimeoutMs'], isNull); + }); + + test('rounds up to whole milliseconds', () { + final headers = Headers()..setReadIdleTimeout(const Duration(microseconds: 1)); + expect(headers[readIdleTimeoutHeader], '1'); + }); + + test('a malformed header fails the call before it opens', () async { + for (final value in ['-1', '1.5', '0x10', '+5', 'soon']) { + final headers = Headers()..[readIdleTimeoutHeader] = value; + await expectLater( + client.httpClient(request(null, headers: headers)), + throwsA(isA().having((e) => e.code, 'code', Code.internal)), + reason: value, + ); + } + expect(plugin.callsTo('open'), isEmpty); + }); + + test('a negative timeout is refused', () async { + expect(() => Headers().setReadIdleTimeout(const Duration(seconds: -1)), throwsArgumentError); + await expectLater( + CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: -1))), + throwsArgumentError, + ); + }); + + test('a timeout fails the call as unavailable, with the platform code', () async { + final response = client.httpClient(request(null)); + await settle(); + await plugin.failed(plugin.lastCallId, streamReady: true, internalErrorCode: -7); + final cause = await failureCause(response); + expect(cause.platformCode, -7); + expect(cause.failure, NativeHttpFailure.other); + }); + }); + test('close shuts the engine down', () async { await client.close(); expect(plugin.callsTo('shutdownEngine').single.arguments['engineId'], 7); diff --git a/test_server/freeze.go b/test_server/freeze.go index 06704ac..d67a7c7 100644 --- a/test_server/freeze.go +++ b/test_server/freeze.go @@ -5,6 +5,7 @@ import ( "fmt" "net" "net/http" + "sync" "sync/atomic" "time" ) @@ -63,13 +64,93 @@ func (c *freezablePacketConn) SetWriteBuffer(bytes int) error { return errors.New("the underlying conn can't set its send buffer") } -// adminHandler lets tests on a device freeze the HTTP/3 path over plain HTTP: +// freezableListener hands out TCP connections that Freeze can black-hole, so a test can kill the HTTP/2 path of the +// connections a client already has, as a network change that leaves both ends up would. Neither URLSession nor +// Cronet notices such a connection on its own: nothing fails, and nothing arrives. +type freezableListener struct { + net.Listener + + mu sync.Mutex + conns map[*freezableConn]struct{} +} + +func newFreezableListener(l net.Listener) *freezableListener { + return &freezableListener{Listener: l, conns: map[*freezableConn]struct{}{}} +} + +func (l *freezableListener) Accept() (net.Conn, error) { + conn, err := l.Listener.Accept() + if err != nil { + return nil, err + } + c := &freezableConn{Conn: conn, listener: l} + l.mu.Lock() + l.conns[c] = struct{}{} + l.mu.Unlock() + return c, nil +} + +// Freeze black-holes every connection open now for d, then closes it: the bytes it dropped leave the connection +// unusable. Connections accepted later are unaffected, so a client that dials again gets through. A zero d closes +// the frozen connections now. It returns how many connections it froze. +func (l *freezableListener) Freeze(d time.Duration) int { + l.mu.Lock() + defer l.mu.Unlock() + n := 0 + for c := range l.conns { + if d <= 0 { + if c.frozen.Load() { + _ = c.Conn.Close() + } + continue + } + if c.frozen.CompareAndSwap(false, true) { + n++ + time.AfterFunc(d, func() { _ = c.Conn.Close() }) + } + } + return n +} + +type freezableConn struct { + net.Conn + listener *freezableListener + frozen atomic.Bool +} + +// Read drops everything that arrives while the connection is frozen. +func (c *freezableConn) Read(p []byte) (int, error) { + for { + n, err := c.Conn.Read(p) + if err != nil || !c.frozen.Load() { + return n, err + } + } +} + +// Write drops everything while the connection is frozen, as a dead path would. +func (c *freezableConn) Write(p []byte) (int, error) { + if c.frozen.Load() { + return len(p), nil + } + return c.Conn.Write(p) +} + +func (c *freezableConn) Close() error { + c.listener.mu.Lock() + delete(c.listener.conns, c) + c.listener.mu.Unlock() + return c.Conn.Close() +} + +// adminHandler lets tests on a device kill the server's network paths over plain HTTP: // -// POST /freeze?for=10s drop all UDP traffic for 10 s -// POST /unfreeze stop dropping now +// POST /freeze?for=10s drop all UDP traffic, so every HTTP/3 connection, for 10 s +// POST /freeze?for=10s&path=tcp black-hole the TCP connections open now (HTTP/2) for 10 s, then close them +// POST /unfreeze stop dropping UDP now, and close the frozen TCP connections // -// TCP is never frozen, so HTTP/2 calls are unaffected. -func adminHandler(conn *freezablePacketConn) http.Handler { +// udp is nil when the server has no HTTP/3. +func adminHandler(udp *freezablePacketConn, tcp *freezableListener) http.Handler { mux := http.NewServeMux() mux.HandleFunc("POST /freeze", func(w http.ResponseWriter, r *http.Request) { d, err := time.ParseDuration(r.URL.Query().Get("for")) @@ -78,11 +159,25 @@ func adminHandler(conn *freezablePacketConn) http.Handler { http.StatusBadRequest) return } - conn.Freeze(d) - fmt.Fprintf(w, "frozen for %s\n", d) + switch path := r.URL.Query().Get("path"); path { + case "", "udp": + if udp == nil { + http.Error(w, "no HTTP/3 path to freeze", http.StatusConflict) + return + } + udp.Freeze(d) + fmt.Fprintf(w, "udp frozen for %s\n", d) + case "tcp": + fmt.Fprintf(w, "%d tcp connections frozen for %s\n", tcp.Freeze(d), d) + default: + http.Error(w, fmt.Sprintf("path=%q: want udp or tcp", path), http.StatusBadRequest) + } }) mux.HandleFunc("POST /unfreeze", func(w http.ResponseWriter, r *http.Request) { - conn.Freeze(0) + if udp != nil { + udp.Freeze(0) + } + tcp.Freeze(0) fmt.Fprintln(w, "unfrozen") }) return mux diff --git a/test_server/main.go b/test_server/main.go index 10db368..0c79bfc 100644 --- a/test_server/main.go +++ b/test_server/main.go @@ -8,7 +8,7 @@ // // -max-idle and -keepalive set the server's QUIC idle timeout and ping interval. QUIC uses the smaller of the two // endpoints' idle timeouts, so -max-idle also bounds how long a client takes to notice a dead path. -admin-addr -// serves a plain-HTTP endpoint that freezes the HTTP/3 path (see adminHandler). +// serves a plain-HTTP endpoint that freezes the HTTP/3 path, or the open HTTP/2 connections (see adminHandler). package main import ( @@ -69,33 +69,39 @@ func main() { handler := protocolHeaders(newElizaMux(), *altSvc) tlsConfig := newTLSConfig(cert) h2 := &http.Server{Addr: *addr, Handler: handler, TLSConfig: tlsConfig, ReadHeaderTimeout: 10 * time.Second} + tcp, err := net.Listen("tcp", *addr) + if err != nil { + log.Fatalf("http2: %v", err) + } + tcpConns := newFreezableListener(tcp) go func() { - if err := h2.ListenAndServeTLS("", ""); err != nil && !errors.Is(err, http.ErrServerClosed) { + if err := h2.ServeTLS(tcpConns, "", ""); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("http2: %v", err) } }() var h3 *http3.Server + var udpConn *freezablePacketConn if !*noH3 { udp, err := net.ListenPacket("udp", *h3Addr) if err != nil { log.Fatalf("http3: %v", err) } - conn := &freezablePacketConn{PacketConn: udp} + udpConn = &freezablePacketConn{PacketConn: udp} h3 = newHTTP3Server(handler, tlsConfig, quicConfig(*maxIdle, *keepAlive)) go func() { - if err := h3.Serve(conn); err != nil && !errors.Is(err, http.ErrServerClosed) { + if err := h3.Serve(udpConn); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("http3: %v", err) } }() - if *adminAddr != "" { - admin := &http.Server{Addr: *adminAddr, Handler: adminHandler(conn), ReadHeaderTimeout: 10 * time.Second} - go func() { - if err := admin.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { - log.Fatalf("admin: %v", err) - } - }() - } + } + if *adminAddr != "" { + admin := &http.Server{Addr: *adminAddr, Handler: adminHandler(udpConn, tcpConns), ReadHeaderTimeout: 10 * time.Second} + go func() { + if err := admin.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + log.Fatalf("admin: %v", err) + } + }() } log.Printf("listening: tcp %s, udp %s (h3 %t, max idle %s, keepalive %s), admin %q, Alt-Svc %q, CA in %s", *addr, *h3Addr, !*noH3, *maxIdle, *keepAlive, *adminAddr, *altSvc, *certDir) diff --git a/test_server/server_test.go b/test_server/server_test.go index 756b383..2c9fefe 100644 --- a/test_server/server_test.go +++ b/test_server/server_test.go @@ -5,6 +5,7 @@ import ( "crypto/tls" "crypto/x509" "errors" + "fmt" "io" "net" "net/http" @@ -168,7 +169,7 @@ func TestFrozenPathFailsOpenStreamWithinIdleTimeout(t *testing.T) { func TestAdminFreezeAndUnfreeze(t *testing.T) { conn := &freezablePacketConn{} - handler := adminHandler(conn) + handler := adminHandler(conn, newFreezableListener(nil)) post := func(target string) int { w := httptest.NewRecorder() handler.ServeHTTP(w, httptest.NewRequest(http.MethodPost, target, nil)) @@ -183,6 +184,96 @@ func TestAdminFreezeAndUnfreeze(t *testing.T) { if code := post("/freeze?for=nonsense"); code != http.StatusBadRequest || conn.frozen() { t.Fatalf("bad duration: status %d, frozen %t", code, conn.frozen()) } + if code := post("/freeze?for=1h&path=tcp"); code != http.StatusOK || conn.frozen() { + t.Fatalf("freeze tcp: status %d, udp frozen %t", code, conn.frozen()) + } + if code := post("/freeze?for=1h&path=sctp"); code != http.StatusBadRequest { + t.Fatalf("unknown path: status %d", code) + } +} + +// startHTTP2 serves the eliza handler over HTTP/2 on a free loopback TCP port, through a freezableListener, and +// returns a function that makes a client with its own connections. +func startHTTP2(t *testing.T) (func() elizav1connect.ElizaServiceClient, *freezableListener) { + t.Helper() + dir := t.TempDir() + cert, err := writeCertificates(dir, []string{"localhost", "127.0.0.1"}) + if err != nil { + t.Fatal(err) + } + caDER, err := os.ReadFile(filepath.Join(dir, "ca.der")) + if err != nil { + t.Fatal(err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + roots.AddCert(ca) + + tcp, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + listener := newFreezableListener(tcp) + server := &http.Server{Handler: protocolHeaders(newElizaMux(), ""), TLSConfig: newTLSConfig(cert)} + go func() { _ = server.ServeTLS(listener, "", "") }() + t.Cleanup(func() { _ = server.Close() }) + + baseURL := "https://" + tcp.Addr().String() + newClient := func() elizav1connect.ElizaServiceClient { + transport := &http.Transport{TLSClientConfig: &tls.Config{RootCAs: roots}, ForceAttemptHTTP2: true} + t.Cleanup(transport.CloseIdleConnections) + client := connect.NewClient(connecthttp.NewTransport(&http.Client{Transport: transport}, baseURL)) + return elizav1connect.NewElizaServiceClient(client) + } + return newClient, listener +} + +func TestFrozenTCPConnectionGoesSilentAndNewConnectionsWork(t *testing.T) { + newClient, listener := startHTTP2(t) + stream := openQuietConverse(t, newClient()) + if n := listener.Freeze(time.Minute); n != 1 { + t.Fatalf("froze %d connections, want 1", n) + } + t.Cleanup(func() { listener.Freeze(0) }) + + // Nothing fails on the frozen connection: the client's sentence and the server's echo both vanish. + received := make(chan error, 1) + go func() { + if err := stream.Send(&elizav1.ConverseRequest{Sentence: "anyone?"}); err != nil { + received <- err + return + } + res, err := stream.Receive() + if err == nil { + err = fmt.Errorf("received %q", res.GetSentence()) + } + received <- err + }() + select { + case err := <-received: + t.Fatalf("the frozen connection answered: %v", err) + case <-time.After(time.Second): + } + + // A connection dialed after the freeze works. + res, err := newClient().Say(timeoutContext(t), &elizav1.SayRequest{Sentence: "fresh"}) + if err != nil || res.GetSentence() != "say: fresh" { + t.Fatalf("Say on a new connection: %v, %v", res, err) + } + + // Unfreezing closes the frozen connection, which ends the stream. + listener.Freeze(0) + select { + case err := <-received: + if err == nil { + t.Fatal("the stream ended without an error") + } + case <-time.After(5 * time.Second): + t.Fatal("the stream outlived its closed connection") + } } func timeoutContext(t *testing.T) context.Context { From 3fde613fc223745dec6a49badeb10a2a83e5bd87 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:50:04 +0000 Subject: [PATCH 2/7] Warm the connection before the native read-idle timeout tests On the Android emulator, the first call on a fresh Cronet engine spent longer than its 500 ms read-idle timeout in the TLS handshake, so the timer ended it before the greeting arrived. The timeout covers connection setup by design. The timed calls now reuse a connection an earlier call left behind. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- .../com/agtonomy/connect_native_http/CronetBridgeTest.kt | 4 ++++ .../ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt index dc92ad5..c447084 100644 --- a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt +++ b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt @@ -94,6 +94,9 @@ class CronetBridgeTest { @Test fun aReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() { val engine = createEngine() + // The timer also runs while the connection is set up, which through the emulator once took longer than the + // timeout (the TLS handshake was cut off). A call first leaves a connection for the timed one to reuse. + assertSayWorks(engine, 10) open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 500) await(1, "data") val start = System.nanoTime() @@ -114,6 +117,7 @@ class CronetBridgeTest { @Test fun receivingResetsTheReadIdleTimer() { val engine = createEngine() + assertSayWorks(engine, 10) open(engine, 1, CONVERSE, "application/connect+proto", readIdleTimeoutMs = 1000) await(1, "data") // Twice the timeout in all, 600 ms between replies. diff --git a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift index d0f10ae..a5f7ea1 100644 --- a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift +++ b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift @@ -91,6 +91,12 @@ final class URLSessionBridgeTests: XCTestCase { func testReadIdleTimeoutFailsAQuietCallAndTheEngineCarriesOn() async throws { let engine = try await createEngine() + // The timeout also covers setting up the connection, which can outlast it on a loaded simulator. A call first + // leaves a connection for the timed one to reuse. + try await open(engine, callId: 10, path: Self.say, contentType: "application/proto") + try await write(10, sentence("warm up")) + try await invoke("end", ["callId": 10]) + try await waitUntil("the warm-up call ended", timeout: 10) { self.outcome(10) != nil } try await open( engine, callId: 1, path: Self.converse, contentType: "application/connect+proto", readIdleTimeoutMs: 500) try await waitUntil("the greeting arrived") { self.received(1).contains("hello") } From e3609f1119c7485e1bc8b4a26f649c6bdd79e2d1 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:57:18 +0000 Subject: [PATCH 3/7] Evict a connection after a deadline too, unless it is still alive A call that hits its deadline may be stuck on a dead connection, and without eviction the next call would go to the same one. But the server enforces the same deadline (connect-dart sends connect-timeout-ms) and answers deadline_exceeded about a round trip after the client's own deadline, so the client firing first says nothing about the connection. - The call still fails at its deadline: Dart fails it immediately and tells the bridge to cancel it after a 1 s grace period. - During the grace period the stream stays open. If anything arrives from the call's origin, the server's answer included, the connection is alive. If not, new calls move to a fresh URLSession or CronetEngine, as after a read-idle timeout. - A read-idle timeout now only evicts under the same guard: nothing has arrived from the origin, on any call, since the call last received anything. The test server now reports each client's address in X-Remote-Addr, so the integration tests can check whether a call reused a connection. They check that a deadline on a frozen HTTP/2 connection moves the next call to a new one, that a read-idle timeout does too, and that a deadline the server answers keeps the connection. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- README.md | 47 ++++++++---- .../connect_native_http/CronetBridgeTest.kt | 16 ++++ .../connect_native_http/CronetBridge.kt | 74 +++++++++++++++--- example/integration_test/eliza_test.dart | 42 +++++++++- .../URLSessionBridge.swift | 76 ++++++++++++++++--- .../URLSessionBridgeTests.swift | 14 ++++ lib/src/channel/channel_http_client.dart | 31 ++++++-- lib/src/native_http_config.dart | 7 +- test/cronet_http_client_test.dart | 26 +++++++ test_server/main.go | 4 +- test_server/server_test.go | 3 + 11 files changed, 294 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index dfdaa07..5e52611 100644 --- a/README.md +++ b/README.md @@ -74,12 +74,28 @@ NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); - **The failure:** `unavailable`, with a `NativeHttpException` whose `platformCode` is `NSURLErrorTimedOut` (-1001) on iOS or `net::ERR_TIMED_OUT` (-7) on Android. On Android a timeout before `onStreamReady` is `notSent`, as with any other failure there. -- **The connection:** a timeout is taken as a sign the connection is dead, and the connection is evicted. Neither stack - can close a single connection, so the client moves its new calls to a fresh `URLSession` or `CronetEngine`, which - dials its own connections. Calls already on the old one finish there, and it is released after the last one ends. +- **The connection:** see "Calls that run out of time" below. - **The header:** the value travels in a `native-http-read-idle-timeout-ms` request header (`readIdleTimeoutHeader`), which the client removes before sending. connect-dart's `CallOptions` carry nothing else per call. +### Calls that run out of time + +A call that hits its read-idle timeout or its deadline may be stuck on a dead connection. The client keeps new calls +off such a connection, so they don't wait on it too. + +- **Read-idle timeout:** the connection counts as dead if nothing has arrived from the call's origin (host and port), + on any call, since the call last received anything. If something has, the connection is alive and only that call + was slow, so nothing changes. +- **Deadline:** the call fails at its deadline, as always. connect-dart sends the deadline in `connect-timeout-ms`, so + the server times the call out too and answers `deadline_exceeded`. Its clock starts later, so its answer arrives + about a round trip after the client's deadline. The bridge therefore keeps the stream open for 1 s more before + canceling it. If anything arrives from the origin in that time, the server's answer included, the connection is + alive. If nothing does, it counts as dead. +- **A dead connection is evicted.** Neither stack can close a single connection, so the client moves its new calls to + a fresh `URLSession` or `CronetEngine`, which dials its own connections. Calls already on the old one are not + canceled. They finish there, and it is released after the last one ends. Several calls that run out of time on the + same connection evict it once. + ## Server and infrastructure requirements The clients only reach HTTP/3, and only keep bidi working, if the server side meets these requirements. Each was @@ -167,12 +183,14 @@ Clients may enforce a floor on the peer's value. quic-go's client raises anythin (`protocol.MinRemoteIdleTimeout`), so measure any other client before relying on a short server value. None of this works on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP -connection that dies without the interface going down goes unnoticed until a call's deadline, an application-level -heartbeat, or a read-idle timeout. The integration tests check both sides: +connection that dies without the interface going down goes unnoticed until a call on it runs out of time: a deadline +or a read-idle timeout (see "Calls that run out of time"). The integration tests check this: -- Without a read-idle timeout, a frozen HTTP/2 connection draws no error for 3 s, and a new call on the same client - waits on it until its deadline. -- With one, the call fails within the timeout, and the client's next call goes out on a new connection. +- Over a frozen HTTP/2 connection nothing arrives for 3 s, and a new call on the same client waits on it until its + deadline. The call after that goes out on a new connection. +- With a read-idle timeout, the call fails within the timeout, and the client's next call goes out on a new + connection. +- A call the server answers with `deadline_exceeded` leaves the client on the same connection. ### QUIC idle timeout vs HTTP/2 keepalive pings @@ -266,18 +284,21 @@ Neither platform finishes a stream on its own while the client still has its req things: a quiet stream outlives the idle timeout, and a frozen path fails an open stream within it. The read-idle timeout is tested over HTTP/2 on both platforms. On iOS each of these cases freezes the server's UDP - first, so URLSession falls back to TCP as it would where UDP is blocked. The cases check four things: + first, so URLSession falls back to TCP as it would where UDP is blocked. The server reports each call's client + address in `X-Remote-Addr`, which shows whether two calls shared a connection. The cases check five things: - A quiet stream fails at its timeout. - A stream that keeps receiving outlives its timeout. - - Without a timeout, a frozen connection goes unnoticed. This is the control. - - With one, the call fails and the client's next call works, which it can only do on a new connection. + - A frozen connection draws no error, and new calls go to it until one hits its deadline. The call after that is + on a new connection. + - With a read-idle timeout, the call fails and the client's next call is on a new connection. + - A deadline the server answers leaves the client on the same connection. - **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's bridge (`URLSessionBridge`, `CronetBridge`) has no Flutter dependency, and the tests call it directly, for behavior Dart can't reach: - **iOS (`ios/connect_native_http/Tests`, XCTest):** a write stuck on a full buffer returns once the call is canceled or its connection fails; a redirect of a request with a body is returned, not followed; shutting a session down lets its calls finish and refuses new ones; a read-idle timeout fails a quiet call with - `NSURLErrorTimedOut`, and the engine's next call works. The script runs them on the simulator. For a quick loop + `NSURLErrorTimedOut`, and the engine's next call works; a deadline cancel waits out its grace period. The script runs them on the simulator. For a quick loop on the Mac, against a running test server: ```bash @@ -289,7 +310,7 @@ Neither platform finishes a stream on its own while the client still has its req write after the server finished the call is dropped instead of crashing; shutting an engine down lets its calls finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped; a read-idle timeout fails a quiet call with -7, once, and the engine's next call works; data arriving resets the - timer. + timer; a deadline cancel waits out its grace period. - Both platforms' suites also check that malformed method calls get an error reply and leave the bridge working. - **`example/integration_test/public_server_test.dart`:** HTTP/3 on Android against a server with a public certificate. It also checks that `quicIdleTimeout` ends a quiet connection, against a control without it. It is diff --git a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt index c447084..ec91847 100644 --- a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt +++ b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt @@ -130,6 +130,20 @@ class CronetBridgeTest { assertEquals("done", awaitOutcome(1)["type"]) } + // After a deadline the stream stays open for the grace period, for a sign that its connection is alive. + @Test + fun aDeadlineCancelWaitsOutTheGracePeriod() { + val engine = createEngine() + open(engine, 1, CONVERSE, "application/connect+proto") + await(1, "data") + val start = System.nanoTime() + invoke("cancel", mapOf("callId" to 1, "deadlineGraceMs" to 500)) + assertEquals("canceled", awaitOutcome(1)["type"]) + val elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) + assertTrue("canceled after $elapsedMs ms, before the 500 ms grace period", elapsedMs >= 450) + assertSayWorks(engine, 2) + } + @Test fun malformedArgumentsAreRefusedInsteadOfCrashing() { val engine = createEngine() @@ -142,6 +156,8 @@ class CronetBridgeTest { "shutdownEngine" to mapOf("engineId" to 1.5), "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to 0), "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to "soon"), + "cancel" to mapOf("callId" to 1, "deadlineGraceMs" to 0), + "cancel" to mapOf("callId" to 1, "deadlineGraceMs" to "soon"), ) for ((method, arguments) in malformed) { val reply = reply(method, arguments) diff --git a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt index f6cbb3e..7e8f1e1 100644 --- a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt +++ b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt @@ -10,6 +10,7 @@ import org.chromium.net.CronetException import org.chromium.net.NetworkException import org.chromium.net.QuicOptions import org.chromium.net.UrlResponseInfo +import java.net.URI import java.nio.ByteBuffer import java.util.concurrent.Executors import java.util.concurrent.ScheduledExecutorService @@ -43,6 +44,9 @@ class CronetBridge( // a fresh one, with its own connections, and the retired one shuts down once its last call ends. private class Pool(val cronet: CronetEngine) { val callIds = HashSet() + + // When this engine last received anything from each origin ("host:port"), in System.nanoTime(). + val lastReceived = HashMap() } private class Engine(val options: EngineOptions, var current: Pool) { @@ -50,7 +54,13 @@ class CronetBridge( var shuttingDown = false } - private class Call(val callId: Int, val engineId: Int, val pool: Pool, val readIdleTimeoutNanos: Long?) { + private class Call( + val callId: Int, + val engineId: Int, + val pool: Pool, + val origin: String, + val readIdleTimeoutNanos: Long?, + ) { lateinit var stream: BidirectionalStream // When the call last received anything from the server, or started, for its read-idle timeout. @@ -158,7 +168,17 @@ class CronetBridge( } "cancel" -> { val callId = args["callId"] as Int - execute(callId) { calls[callId]?.stream?.cancel() } + // Set when the call hit its deadline: see cancelAfterDeadline. + val deadlineGraceMs = (args["deadlineGraceMs"] as Number?)?.toLong() + require(deadlineGraceMs == null || deadlineGraceMs > 0) { "deadlineGraceMs: $deadlineGraceMs" } + execute(callId) { + val c = calls[callId] ?: return@execute + if (deadlineGraceMs != null) { + cancelAfterDeadline(c, deadlineGraceMs) + } else { + c.stream.cancel() + } + } result(Reply.Success(null)) } else -> result(Reply.NotImplemented) @@ -192,7 +212,15 @@ class CronetBridge( return } val pool = engine.current - val c = Call(callId, engineId, pool, readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }) + val target = URI(url) + val port = if (target.port != -1) target.port else if (target.scheme == "http") 80 else 443 + val c = Call( + callId, + engineId, + pool, + "${target.host}:$port", + readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }, + ) val builder = pool.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) for ((name, value) in headers) { builder.addHeader(name, value) @@ -241,19 +269,47 @@ class CronetBridge( evict(c) } + // Anything from the server shows the call's connection is alive. private fun received(c: Call) { - c.lastReceived = System.nanoTime() + val now = System.nanoTime() + c.lastReceived = now + c.pool.lastReceived[c.origin] = now } - // Moves the engine's new calls off the connection [c] timed out on. Cronet has no call that closes one - // connection, so the engine's CronetEngine is retired: new calls go to a fresh one, which dials its own - // connections, and the old one shuts down once the calls still on it end. Calls that time out together on the same - // dead connection retire it once. + // The call has hit its deadline and Dart has already failed it. The server enforces the same deadline and answers + // about a round trip later, so the stream stays open for the grace period: an answer, or anything else from the + // origin, shows the connection is alive. After it, the stream is canceled, and the connection evicted if it stayed + // quiet. + private fun cancelAfterDeadline(c: Call, graceMs: Long) { + val check = Runnable { + try { + // Gone: the server's answer ended it, or Cronet failed it. + if (calls[c.callId] === c) { + evict(c) + c.stream.cancel() + } + } catch (e: Exception) { + Log.e(TAG, "call ${c.callId} failed in the bridge", e) + abort(c.callId, e) + } + } + executor.schedule(check, graceMs, TimeUnit.MILLISECONDS) + } + + // A call that ran out of time (its read-idle timeout, or its deadline and grace) may be stuck on a dead connection. + // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received + // anything; otherwise only that call was slow. Cronet has no call that closes one connection, so the engine's + // CronetEngine is retired: new calls go to a fresh one, which dials its own connections, and the old one shuts down + // once the calls still on it end. Calls that time out together on the same dead connection retire it once. private fun evict(c: Call) { val engine = engines[c.engineId] ?: return if (engine.current !== c.pool || engine.shuttingDown) { return } + val heard = c.pool.lastReceived[c.origin] + if (heard != null && heard - c.lastReceived > 0) { + return + } val fresh = try { Pool(build(engine.options)) } catch (e: Throwable) { @@ -263,7 +319,7 @@ class CronetBridge( } engine.retired.add(engine.current) engine.current = fresh - Log.i(TAG, "engine ${c.engineId}: call ${c.callId} hit its read idle timeout; new calls use a fresh CronetEngine") + Log.i(TAG, "engine ${c.engineId}: call ${c.callId} ran out of time on a quiet connection; moved to a new engine") } // Fails a call the bridge itself could not carry on with, after an exception on its thread. diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index fc24f62..4512381 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -67,7 +67,13 @@ final class Harness { /// `X-Proto` of each response: the protocol the server saw. final serverProtocols = []; - void recordHeaders(Headers headers) => serverProtocols.add(headers.get('x-proto')?.join(',') ?? ''); + /// `X-Remote-Addr` of each response: the client's address as the server saw it, which differs between connections. + final remoteAddrs = []; + + void recordHeaders(Headers headers) { + serverProtocols.add(headers.get('x-proto')?.join(',') ?? ''); + remoteAddrs.add(headers.get('x-remote-addr')?.join(',') ?? ''); + } } /// Every case finishes in a few seconds against a local server (most well under one); a hang should fail fast. @@ -395,13 +401,15 @@ void main() { await requests.close(); }, timeout: http2CaseTimeout); - testCase('without a read-idle timeout, a dead HTTP/2 connection goes unnoticed', (_) async { - // The control for the next case: what the read-idle timeout is for. + testCase('a dead HTTP/2 connection is reused until a call runs out of time on it', (_) async { + // Nothing notices the dead connection by itself; the first call to run out of time, here on its deadline, moves + // the client off it. if (skipUnlessAdmin()) { return; } final h2 = await harnessOverHttp2(); final (requests, responses) = await openQuietConverse(h2); + final deadAddr = h2.remoteAddrs.last; await admin('/freeze?for=60s&path=tcp'); requests.add(ConverseRequest(sentence: 'anyone?')); @@ -412,10 +420,36 @@ void main() { h2.eliza.say(SayRequest(sentence: 'hi'), signal: TimeoutSignal(const Duration(seconds: 2))), throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), ); + // Nothing arrives in the grace period after the deadline, so the next call dials a new connection. + await Future.delayed(const Duration(milliseconds: 1500)); + final res = await h2.eliza.say( + SayRequest(sentence: 'after'), + signal: TimeoutSignal(const Duration(seconds: 8)), + onHeader: h2.recordHeaders, + ); + expect(res.sentence, 'say: after'); + expect(h2.remoteAddrs.last, isNot(deadAddr)); unawaited(reply.then((_) {}, onError: (_) {})); await requests.close(); }, timeout: http2CaseTimeout); + testCase('a deadline the server answers keeps the connection', (_) async { + // connect-dart sends the deadline, and the server answers deadline_exceeded about a round trip after the client's + // own deadline, within the grace period: the connection is alive. + if (skipUnlessAdmin()) { + return; + } + final h2 = await harnessOverHttp2(); + final liveAddr = h2.remoteAddrs.last; + await expectLater( + h2.eliza.say(SayRequest(sentence: 'sleep:5000'), signal: TimeoutSignal(const Duration(seconds: 1))), + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + await Future.delayed(const Duration(milliseconds: 1500)); + await h2.eliza.say(SayRequest(sentence: 'after'), onHeader: h2.recordHeaders); + expect(h2.remoteAddrs.last, liveAddr); + }, timeout: http2CaseTimeout); + testCase('a read-idle timeout notices a dead HTTP/2 connection, and the next call dials a new one', (_) async { if (skipUnlessAdmin()) { return; @@ -426,6 +460,7 @@ void main() { final (requests, responses) = await openQuietConverse(h2); await admin('/freeze?for=60s&path=tcp'); + final deadAddr = h2.remoteAddrs.last; requests.add(ConverseRequest(sentence: 'anyone?')); final stopwatch = Stopwatch()..start(); final cause = await nativeCause(responses.moveNext()); @@ -441,6 +476,7 @@ void main() { ); expect(res.sentence, 'say: after'); expect(h2.serverProtocols.last, 'HTTP/2.0'); + expect(h2.remoteAddrs.last, isNot(deadAddr)); }, timeout: http2CaseTimeout); testCase('a dead HTTP/3 path fails an open stream', (_) async { diff --git a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift index b757374..3b70fe5 100644 --- a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift +++ b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift @@ -36,6 +36,8 @@ public final class URLSessionBridge { // Replaced after a call times out, so new calls dial a fresh connection; see `evictConnection`. var urlSession: URLSession let http3Origins: Set + // When `urlSession` last received anything from each origin ("host:port"), in uptime nanoseconds. + var lastReceived: [String: UInt64] = [:] init(urlSession: URLSession, http3Origins: Set) { self.urlSession = urlSession @@ -48,6 +50,9 @@ public final class URLSessionBridge { var engineId = 0 // The URLSession the call's task runs in. var urlSession: URLSession? + var origin = "" + // When the call last received anything, or started, in uptime nanoseconds. + var lastReceived = URLSessionBridge.now() var task: URLSessionTask! var input: InputStream? var output: OutputStream? @@ -113,12 +118,7 @@ public final class URLSessionBridge { let headers: [[String]] = try arg(args, "headers") guard headers.allSatisfy({ $0.count == 2 }) else { throw BadArgument(name: "headers") } let hasBody: Bool = try arg(args, "hasBody") - // Absent or null for none. - var readIdleTimeoutMs: Int? - if let value = args["readIdleTimeoutMs"], !(value is NSNull) { - guard let ms = value as? Int, ms > 0 else { throw BadArgument(name: "readIdleTimeoutMs") } - readIdleTimeoutMs = ms - } + let readIdleTimeoutMs = try optionalPositiveInt(args, "readIdleTimeoutMs") queue.async { self.open(engineId, callId, url, method, headers, hasBody, readIdleTimeoutMs) } reply(.success(nil)) case "write": @@ -144,13 +144,33 @@ public final class URLSessionBridge { reply(.success(nil)) case "cancel": let callId: Int = try arg(args, "callId") - queue.async { self.calls[callId]?.task.cancel() } + // Set when the call hit its deadline: see `cancelAfterDeadline`. + let deadlineGraceMs = try optionalPositiveInt(args, "deadlineGraceMs") + queue.async { + guard let c = self.calls[callId] else { return } + if let deadlineGraceMs { + self.cancelAfterDeadline(c, graceMs: deadlineGraceMs) + } else { + c.task.cancel() + } + } reply(.success(nil)) default: reply(.notImplemented) } } + /// An optional argument that must be a positive integer when present; absent or null for none. + private func optionalPositiveInt(_ args: [String: Any], _ name: String) throws -> Int? { + guard let value = args[name], !(value is NSNull) else { return nil } + guard let n = value as? Int, n > 0 else { throw BadArgument(name: name) } + return n + } + + static func now() -> UInt64 { + DispatchTime.now().uptimeNanoseconds + } + private func failure(_ callId: Int, domain: String, code: Int, message: String) -> [String: Any] { ["callId": callId, "type": "failed", "domain": domain, "code": code, "message": message] } @@ -201,6 +221,7 @@ public final class URLSessionBridge { let c = Call(callId: callId) c.engineId = engineId c.urlSession = session.urlSession + c.origin = "\(target.host ?? ""):\(port)" if hasBody { var input: InputStream? var output: OutputStream? @@ -248,14 +269,26 @@ public final class URLSessionBridge { callsByTask[task]?.input == nil } + // Anything from the server shows the call's connection is alive. + private func received(_ c: Call) { + let now = Self.now() + c.lastReceived = now + if let session = sessions[c.engineId], session.urlSession === c.urlSession { + session.lastReceived[c.origin] = now + } + } + fileprivate func didReceive(_ response: URLResponse, for task: URLSessionTask) { - guard let c = callsByTask[task], let http = response as? HTTPURLResponse else { return } + guard let c = callsByTask[task] else { return } + received(c) + guard let http = response as? HTTPURLResponse else { return } let headers = http.allHeaderFields.map { [String(describing: $0.key), String(describing: $0.value)] } emit(["callId": c.callId, "type": "headers", "status": http.statusCode, "protocol": NSNull(), "headers": headers]) } fileprivate func didReceive(_ data: Data, for task: URLSessionTask) { guard let c = callsByTask[task] else { return } + received(c) emit(["callId": c.callId, "type": "data", "data": data]) } @@ -284,16 +317,35 @@ public final class URLSessionBridge { emit(failure(c.callId, domain: error.domain, code: error.code, message: error.localizedDescription)) } - // A call that timed out is taken as a sign its connection is dead. URLSession pools connections per session and has - // no call that closes one, so the engine's new calls go to a fresh URLSession, which dials its own connections. The - // old one finishes the tasks it still has, and is then released. Calls that time out together on the same dead - // connection replace it once. + // A call that ran out of time (its read-idle timeout, or its deadline and grace) may be stuck on a dead connection. + // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received + // anything; otherwise only that call was slow. URLSession pools connections per session and has no call that closes + // one, so the engine's new calls go to a fresh URLSession, which dials its own connections. The old one finishes the + // tasks it still has, and is then released. Calls that time out together on the same dead connection replace it + // once. private func evictConnection(of c: Call) { guard let session = sessions[c.engineId], let old = c.urlSession, session.urlSession === old else { return } + if let heard = session.lastReceived[c.origin], heard > c.lastReceived { + return + } session.urlSession = makeURLSession() + session.lastReceived = [:] old.finishTasksAndInvalidate() } + // The call has hit its deadline and Dart has already failed it. The server enforces the same deadline and answers + // about a round trip later, so the task stays open for the grace period: an answer, or anything else from the + // origin, shows the connection is alive. After it, the task is canceled, and the connection evicted if it stayed + // quiet. + private func cancelAfterDeadline(_ c: Call, graceMs: Int) { + queue.asyncAfter(deadline: .now() + .milliseconds(graceMs)) { + // Gone: the server's answer ended it, or the stack failed it. + guard self.calls[c.callId] === c else { return } + self.evictConnection(of: c) + c.task.cancel() + } + } + fileprivate func serverTrust(_ challenge: URLAuthenticationChallenge) -> ( URLSession.AuthChallengeDisposition, URLCredential? ) { diff --git a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift index a5f7ea1..31644ca 100644 --- a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift +++ b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift @@ -116,6 +116,18 @@ final class URLSessionBridgeTests: XCTestCase { XCTAssertTrue(received(2).contains("say: after")) } + // After a deadline the task stays open for the grace period, for a sign that its connection is alive. + func testADeadlineCancelWaitsOutTheGracePeriod() async throws { + let engine = try await createEngine() + try await open(engine, callId: 1, path: Self.converse, contentType: "application/connect+proto") + try await waitUntil("the greeting arrived", timeout: 10) { self.received(1).contains("hello") } + let start = Date() + try await invoke("cancel", ["callId": 1, "deadlineGraceMs": 500]) + try await waitUntil("the call was canceled") { self.outcome(1) != nil } + XCTAssertEqual(outcome(1)?["type"] as? String, "canceled", "\(String(describing: outcome(1)))") + XCTAssertGreaterThanOrEqual(Date().timeIntervalSince(start), 0.45, "canceled before the 500 ms grace period") + } + func testRequestTimeoutIntervalAvoidsTheValueURLSessionIgnores() { XCTAssertNil(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: nil)) XCTAssertEqual(URLSessionBridge.requestTimeoutInterval(readIdleTimeoutMs: 1500), 1.5) @@ -138,6 +150,8 @@ final class URLSessionBridgeTests: XCTestCase { ("write", ["callId": 1, "data": "not bytes"]), ("end", ["callId": "1"]), ("cancel", [:]), + ("cancel", ["callId": 1, "deadlineGraceMs": 0]), + ("cancel", ["callId": 1, "deadlineGraceMs": "soon"]), ("shutdownEngine", ["engineId": NSNull()]), ("createEngine", ["http3Origins": [["localhost"]]]), ] diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index 3c8f6f5..dd98348 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -134,9 +134,17 @@ final class _Dispatcher { /// Asks the plugin to cancel call [id]; its `canceled` event then ends the call. Should the plugin fail to take the /// request, no event will come, so the call ends here instead. - Future cancel(int id) async { + /// + /// With [deadlineGrace], the call has hit its deadline and already failed here. The plugin then waits up to that long + /// before canceling it, for anything from the call's origin: the server's own `deadline_exceeded`, typically, since + /// connect-dart sends the deadline and the server's clock starts later. If nothing arrives, the call is taken to be + /// stuck on a dead connection, and new calls move to a fresh one. + Future cancel(int id, {Duration? deadlineGrace}) async { try { - await bridge.methods.invokeMethod('cancel', {'callId': id}); + await bridge.methods.invokeMethod('cancel', { + 'callId': id, + if (deadlineGrace != null) 'deadlineGraceMs': deadlineGrace.inMilliseconds, + }); } catch (error, stack) { final call = remove(id); call?.fail(call.abortError ?? ConnectException(Code.canceled, '${bridge.name}: cancel failed: $error')); @@ -232,6 +240,10 @@ final class _Dispatcher { } } +/// How long the plugin keeps a call that hit its deadline, waiting for a sign its connection is alive. Covers the +/// server's own `deadline_exceeded`, which arrives about a round trip after the client's deadline. +const deadlineGrace = Duration(seconds: 1); + /// A connect-dart [HttpClient] on one engine of a [PlatformBridge] plugin. base class ChannelHttpClient implements NativeHttpClient { ChannelHttpClient(this._bridge, this._engineId, this._config); @@ -274,10 +286,19 @@ base class ChannelHttpClient implements NativeHttpClient { } }; req.signal?.future.then((error) { - if (!call.finished) { - call.abortError = error; - unawaited(dispatcher.cancel(id)); + if (call.finished) { + return; + } + call.abortError = error; + if (error.code == Code.deadlineExceeded) { + // The caller gets the deadline now; the plugin keeps the stream a little longer to tell whether its connection + // is dead (see _Dispatcher.cancel), and nothing it reports for the call matters any more. + dispatcher.remove(id); + call.fail(error); + unawaited(dispatcher.cancel(id, deadlineGrace: deadlineGrace)); + return; } + unawaited(dispatcher.cancel(id)); }); try { await methods.invokeMethod('open', { diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index 4e7f804..d5554b4 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -40,9 +40,10 @@ final class NativeHttpConfig { /// - **Android:** a timer in the bridge, which fails the call with `net::ERR_TIMED_OUT` (-7). /// - **The failure:** a `ConnectException` with code `unavailable`, whose cause is a [NativeHttpException] with /// that `platformCode`. - /// - **The connection:** a call that timed out is taken as a sign its connection is dead. The client moves new calls - /// to a fresh `URLSession` or Cronet engine, so they dial a new connection, and lets the calls already on the old - /// one finish there. + /// - **The connection:** if nothing has arrived from the call's origin, on any call, since the call last received + /// anything, the connection is taken to be dead. The client moves new calls to a fresh `URLSession` or Cronet + /// engine, so they dial a new connection, and lets the calls already on the old one finish there. A call that hits + /// its deadline is checked the same way, after a 1 s grace period for the server's own `deadline_exceeded`. final Duration? readIdleTimeout; /// Called with each response's headers, before its body is read. An exception it throws goes to diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index f40657b..9305f5b 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -250,6 +250,32 @@ void main() { await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.canceled))); }); + test('a deadline fails the call at once, and gives the plugin time to check the connection', () async { + final signal = TimeoutSignal(const Duration(milliseconds: 10)); + final response = client.httpClient(request(StreamController().stream, signal: signal)); + await settle(); + final id = plugin.lastCallId; + // No canceled event: the call ends without waiting for the plugin. + await expectLater( + response.timeout(const Duration(seconds: 1)), + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + final cancel = plugin.callsTo('cancel').single.arguments as Map; + expect(cancel['callId'], id); + expect(cancel['deadlineGraceMs'], 1000); + // The plugin's later events for the call change nothing. + await plugin.emit({'callId': id, 'type': 'canceled'}); + }); + + test('a signal that is not a deadline cancels at once', () async { + final signal = CancelableSignal(); + unawaited(client.httpClient(request(null, signal: signal)).then((_) {}, onError: (_) {})); + await settle(); + signal.cancel('stop'); + await settle(); + expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('deadlineGraceMs'), isFalse); + }); + test('cancelling the response body cancels the native stream', () async { final responseFuture = client.httpClient(request(null)); await settle(); diff --git a/test_server/main.go b/test_server/main.go index 0c79bfc..8bb0b16 100644 --- a/test_server/main.go +++ b/test_server/main.go @@ -148,11 +148,13 @@ func newElizaMux() *http.ServeMux { } // protocolHeaders logs each request and reports the protocol it arrived on in X-Proto, which is how tests check -// for HTTP/3 on clients that don't expose the negotiated protocol. HTTP/2 responses also advertise HTTP/3. +// for HTTP/3 on clients that don't expose the negotiated protocol, and the client's address in X-Remote-Addr, which +// tells a test whether two calls shared a connection. HTTP/2 responses also advertise HTTP/3. func protocolHeaders(next http.Handler, altSvc string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Printf("%s %s %s from %s", r.Proto, r.Method, r.URL.Path, r.RemoteAddr) w.Header().Set("X-Proto", r.Proto) + w.Header().Set("X-Remote-Addr", r.RemoteAddr) if r.ProtoMajor < 3 && altSvc != "" { w.Header().Set("Alt-Svc", altSvc) } diff --git a/test_server/server_test.go b/test_server/server_test.go index 2c9fefe..cf5008e 100644 --- a/test_server/server_test.go +++ b/test_server/server_test.go @@ -470,5 +470,8 @@ func TestAltSvcOnlyOnTCPResponses(t *testing.T) { if got := w.Header().Get("Alt-Svc"); got != tc.wantAltSvc { t.Errorf("HTTP/%d: Alt-Svc = %q, want %q", tc.major, got, tc.wantAltSvc) } + if got := w.Header().Get("X-Remote-Addr"); got != req.RemoteAddr { + t.Errorf("HTTP/%d: X-Remote-Addr = %q, want %q", tc.major, got, req.RemoteAddr) + } } } From 337005f4f08b5c2cc58155479760eae66ff95f0b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 00:58:59 +0000 Subject: [PATCH 4/7] Drop the per-call read-idle header; deadlines cover per-call timing The read-idle timeout is now only NativeHttpConfig.readIdleTimeout, applied to every call on the client. Per-call bounds come from a call's deadline, which already moves the client off a dead connection after its grace period. setReadIdleTimeout and readIdleTimeoutHeader are gone, along with the header the client stripped before sending. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- README.md | 18 ++++---- example/integration_test/eliza_test.dart | 18 +++----- lib/connect_native_http.dart | 1 - lib/src/channel/channel_http_client.dart | 18 ++++---- lib/src/native_http_config.dart | 10 +++-- lib/src/read_idle_timeout.dart | 51 ----------------------- test/cronet_http_client_test.dart | 53 +++++++----------------- 7 files changed, 43 insertions(+), 126 deletions(-) delete mode 100644 lib/src/read_idle_timeout.dart diff --git a/README.md b/README.md index 5e52611..7090941 100644 --- a/README.md +++ b/README.md @@ -54,17 +54,17 @@ native library fails to load. ### Read-idle timeout -A call can fail once it has received nothing from the server for a set time. This is the only way either client -notices a dead HTTP/2 connection (see "Dead connections"). Set it per call, or as a default for every call: +A client can fail each of its calls once the call has received nothing from the server for a set time. Apart from +deadlines, this is the only way either client notices a dead HTTP/2 connection (see "Dead connections"): ```dart -// One call. An interceptor can set it per procedure the same way. -await eliza.say(request, headers: Headers()..setReadIdleTimeout(const Duration(seconds: 10))); - -// Every call that doesn't set its own. Duration.zero on a call turns the default off for that call. NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); ``` +It applies to every call on the client. For per-call bounds use deadlines (connect-dart's `TimeoutSignal` or +`DeadlineSignal`), which also move the client off a dead connection (see "Calls that run out of time"). Streams that +go quiet for long need a client without a read-idle timeout, or periodic messages. + - **What it measures:** time since the call last received anything: headers, a body chunk or trailers. The timer starts when the call does, so it also bounds the wait for the response headers. A unary call the server holds longer than the timeout fails, and so does a stream that goes quiet for longer. @@ -75,8 +75,6 @@ NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); iOS or `net::ERR_TIMED_OUT` (-7) on Android. On Android a timeout before `onStreamReady` is `notSent`, as with any other failure there. - **The connection:** see "Calls that run out of time" below. -- **The header:** the value travels in a `native-http-read-idle-timeout-ms` request header (`readIdleTimeoutHeader`), - which the client removes before sending. connect-dart's `CallOptions` carry nothing else per call. ### Calls that run out of time @@ -157,7 +155,7 @@ found or verified with the tests in this package. | Certificates | System trust store | System trust store; QUIC only with certificates that chain to a public root | | Negotiated protocol | Not exposed (`NativeHttpResponseInfo.protocol` is `null`) | `h2`, `quic/1+spdy/3` (HTTP/3) or `http/1.1` | | QUIC idle timeout | No setting; the server's advertised value applies | `NativeHttpConfig.quicIdleTimeout`, in whole seconds | -| Read-idle timeout | Per call, through `URLRequest.timeoutInterval`. Without one, a task that receives nothing for a day fails | Per call, through a timer in the bridge. None without one | +| Read-idle timeout | `NativeHttpConfig.readIdleTimeout`, through `URLRequest.timeoutInterval`. Without one, a task that receives nothing for a day fails | `NativeHttpConfig.readIdleTimeout`, through a timer in the bridge. None without one | | Redirects | Followed only for requests without a body; otherwise the 3xx response is returned, since a streamed body can't be resent | Not tested | ### Dead connections @@ -203,7 +201,7 @@ not over HTTP/2. | Who sets it | Both advertise a value; the connection uses the smaller (RFC 9000 §10.1) | Each side sets its own pings | | Can a server setting make the client notice? | Yes. A short server value becomes the client's timeout, and the server's pings stop arriving once the path dies. | No. A server that detects a dead connection can't tell the client, because nothing gets through. | | Through a load balancer | NLB QUIC passthrough is end to end, so the client sees the server's pings | An ALB ends HTTP/2 on each side, so the server's pings stop at the ALB. AWS doesn't document the ALB pinging clients, and nothing here relies on it. | -| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout per call. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout per call. | +| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout, and eviction after a deadline. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout, and eviction after a deadline. | | What a client that sends its own pings gets, for comparison | — | `package:http2` with 3 s pings and a 3 s timeout notices in about 6 s | So for servers: diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index 4512381..6471130 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -92,14 +92,9 @@ Future admin(String path) async { } /// Opens Converse and reads the server's greeting. Nothing flows afterwards until the caller sends. -Future<(StreamController, StreamIterator)> openQuietConverse( - Harness harness, { - Headers? headers, -}) async { +Future<(StreamController, StreamIterator)> openQuietConverse(Harness harness) async { final requests = StreamController(); - final responses = StreamIterator( - harness.eliza.converse(requests.stream, headers: headers, onHeader: harness.recordHeaders), - ); + final responses = StreamIterator(harness.eliza.converse(requests.stream, onHeader: harness.recordHeaders)); expect(await responses.moveNext(), isTrue); expect(responses.current.sentence, 'hello'); return (requests, responses); @@ -366,9 +361,9 @@ void main() { if (skipUnlessAdmin()) { return; } - final h2 = await harnessOverHttp2(); const timeout = Duration(seconds: 1); - final (requests, responses) = await openQuietConverse(h2, headers: Headers()..setReadIdleTimeout(timeout)); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); expect(h2.serverProtocols.last, 'HTTP/2.0'); // Timed from the greeting, the last thing the stream received. @@ -385,9 +380,9 @@ void main() { if (skipUnlessAdmin()) { return; } - final h2 = await harnessOverHttp2(); const timeout = Duration(seconds: 1); - final (requests, responses) = await openQuietConverse(h2, headers: Headers()..setReadIdleTimeout(timeout)); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (requests, responses) = await openQuietConverse(h2); // Three times the timeout in all, but never more than 0.6 of it between replies. for (var i = 0; i < 5; i++) { await Future.delayed(timeout * 0.6); @@ -454,7 +449,6 @@ void main() { if (skipUnlessAdmin()) { return; } - // The client-wide default this time, rather than per call. const timeout = Duration(seconds: 2); final h2 = await harnessOverHttp2(readIdleTimeout: timeout); final (requests, responses) = await openQuietConverse(h2); diff --git a/lib/connect_native_http.dart b/lib/connect_native_http.dart index 2174d56..a20569e 100644 --- a/lib/connect_native_http.dart +++ b/lib/connect_native_http.dart @@ -4,4 +4,3 @@ library; export 'src/native_http_client.dart' show NativeHttpClient; export 'src/native_http_config.dart' show NativeHttpConfig, NativeHttpResponseInfo; export 'src/native_http_exception.dart' show NativeHttpException, NativeHttpFailure; -export 'src/read_idle_timeout.dart' show ReadIdleTimeoutHeaders, readIdleTimeoutHeader; diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index dd98348..6f14fee 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -7,7 +7,6 @@ import 'package:flutter/services.dart'; import '../native_http_client.dart'; import '../native_http_config.dart'; import '../native_http_exception.dart'; -import '../read_idle_timeout.dart'; import 'connect_end_stream.dart'; /// A platform plugin that speaks this file's channel protocol: `createEngine`, `shutdownEngine`, `open`, `write`, @@ -266,18 +265,18 @@ base class ChannelHttpClient implements NativeHttpClient { final int _engineId; final NativeHttpConfig _config; + /// [NativeHttpConfig.readIdleTimeout] in whole milliseconds, rounded up so a sub-millisecond timeout isn't none. + late final int? _readIdleTimeoutMs = switch (_config.readIdleTimeout) { + null || Duration.zero => null, + final timeout => (timeout.inMicroseconds / 1000).ceil(), + }; + @override HttpClient get httpClient => _send; Future _send(HttpRequest req) async { final dispatcher = _Dispatcher.of(_bridge); final methods = _bridge.methods; - final int? readIdleMs; - try { - readIdleMs = readIdleTimeoutMs(req.header, _config.readIdleTimeout); - } on FormatException catch (error) { - throw ConnectException(Code.internal, '${_bridge.name}: ${error.message}: ${error.source}', cause: error); - } final call = _Call(Uri.parse(req.url)); final id = dispatcher.register(call, _config.onResponse); call.body.onCancel = () { @@ -307,11 +306,10 @@ base class ChannelHttpClient implements NativeHttpClient { 'url': req.url, 'method': req.method, 'headers': [ - for (final (:name, :value) in req.header.entries) - if (name != readIdleTimeoutHeader) [name, value], + for (final (:name, :value) in req.header.entries) [name, value], ], 'hasBody': req.body != null, - 'readIdleTimeoutMs': readIdleMs, + 'readIdleTimeoutMs': _readIdleTimeoutMs, }); } catch (error) { // The plugin refused the call, or isn't there (a background isolate, say): nothing was sent. diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index d5554b4..9e99bf4 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -29,9 +29,13 @@ final class NativeHttpConfig { /// Android only. URLSession exposes no QUIC settings, so on iOS only the server's advertised idle timeout applies. final Duration? quicIdleTimeout; - /// The read-idle timeout of calls that don't set their own with `Headers.setReadIdleTimeout`: a call fails once it - /// has received nothing from the server for this long. `null` or [Duration.zero] means none. Both platforms apply it - /// per call, so it also works on the HTTP/2 fallback, where nothing else notices a dead connection. + /// Fails each of this client's calls once it has received nothing from the server for this long. `null` or + /// [Duration.zero] means none. Each call has its own timer in the platform client, so it also works on the HTTP/2 + /// fallback, where nothing else notices a dead connection. + /// + /// It applies to every call, so leave it off on a client that carries streams which go quiet for long, or use a + /// second client for them. A call's own deadline (connect-dart's `TimeoutSignal` or `DeadlineSignal`) also moves new + /// calls off a dead connection; see "Calls that run out of time" in the README. /// /// - **What resets it:** anything the call receives. It runs from when the call starts, so it also bounds the wait /// for the response headers, and a unary call the server holds longer than this fails. A server stream that may go diff --git a/lib/src/read_idle_timeout.dart b/lib/src/read_idle_timeout.dart deleted file mode 100644 index 993ccc5..0000000 --- a/lib/src/read_idle_timeout.dart +++ /dev/null @@ -1,51 +0,0 @@ -import 'package:connectrpc/connect.dart'; - -/// The request header that carries one call's read-idle timeout, in whole milliseconds. `0` means none. -/// -/// The client reads it and removes it, so it never reaches the server. Set it with [setReadIdleTimeout], in a call's -/// `headers` or from an interceptor. -const readIdleTimeoutHeader = 'native-http-read-idle-timeout-ms'; - -/// Gives the call that sends [headers] a read-idle timeout: the call fails if it receives nothing from the server for -/// [timeout]. [Duration.zero] turns the timeout off for this call, overriding `NativeHttpConfig.readIdleTimeout`. -/// -/// ```dart -/// final res = await eliza.say( -/// SayRequest(sentence: 'hi'), -/// headers: Headers()..setReadIdleTimeout(const Duration(seconds: 10)), -/// ); -/// ``` -/// -/// See `NativeHttpConfig.readIdleTimeout` for what counts as receiving, and what happens on a timeout. -extension ReadIdleTimeoutHeaders on Headers { - void setReadIdleTimeout(Duration timeout) { - if (timeout.isNegative) { - throw ArgumentError.value(timeout, 'timeout', 'must not be negative'); - } - // At least 1 ms, so a sub-millisecond timeout doesn't turn into "none". - final ms = timeout == Duration.zero ? 0 : (timeout.inMicroseconds / 1000).ceil(); - this[readIdleTimeoutHeader] = '$ms'; - } -} - -/// The read-idle timeout for a request with [headers], in whole milliseconds, or `null` for none: the request's -/// [readIdleTimeoutHeader] if it has one, else [fallback]. Throws a [FormatException] on a malformed header. -int? readIdleTimeoutMs(Headers headers, Duration? fallback) { - final values = headers.get(readIdleTimeoutHeader); - if (values == null || values.isEmpty) { - if (fallback == null || fallback == Duration.zero) { - return null; - } - return (fallback.inMicroseconds / 1000).ceil(); - } - // Repeated headers: the last one set wins, as with [Headers.operator []=]. - final value = values.last.trim(); - // Digits only: int.parse would also take a sign or a 0x prefix. - final ms = _digits.hasMatch(value) ? int.tryParse(value) : null; - if (ms == null) { - throw FormatException('$readIdleTimeoutHeader must be a whole number of milliseconds', value); - } - return ms == 0 ? null : ms; -} - -final _digits = RegExp(r'^[0-9]{1,15}$'); diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 9305f5b..3d61b84 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -64,12 +64,10 @@ class FakePlugin { /// Lets queued microtasks and platform-channel replies run. Future settle() => Future.delayed(Duration.zero); -HttpRequest request(Stream? body, {AbortSignal? signal, Headers? headers}) => HttpRequest( +HttpRequest request(Stream? body, {AbortSignal? signal}) => HttpRequest( 'https://example.test/svc/Method', 'POST', - Headers() - ..add('content-type', 'application/proto') - ..addAll(headers ?? Headers()), + Headers()..add('content-type', 'application/proto'), body, signal, ); @@ -331,8 +329,8 @@ void main() { }); group('read-idle timeout', () { - Future> openArgs(NativeHttpClient client, {Headers? headers}) async { - unawaited(client.httpClient(request(null, headers: headers)).then((_) {}, onError: (_) {})); + Future> openArgs(NativeHttpClient client) async { + unawaited(client.httpClient(request(null)).then((_) {}, onError: (_) {})); await settle(); return plugin.callsTo('open').last.arguments as Map; } @@ -341,48 +339,25 @@ void main() { expect((await openArgs(client))['readIdleTimeoutMs'], isNull); }); - test("is the client's default for calls that don't set one", () async { + test('applies to every call of a client configured with it', () async { final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); expect((await openArgs(configured))['readIdleTimeoutMs'], 3000); + expect((await openArgs(configured))['readIdleTimeoutMs'], 3000); }); - test("a call's own replaces the default, and its header doesn't reach the server", () async { - final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); - final args = await openArgs( - configured, - headers: Headers()..setReadIdleTimeout(const Duration(milliseconds: 1500)), - ); - expect(args['readIdleTimeoutMs'], 1500); - expect(args['headers'], [ - ['content-type', 'application/proto'], - ]); - }); - - test('zero turns the default off for one call', () async { - final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: 3))); - final args = await openArgs(configured, headers: Headers()..setReadIdleTimeout(Duration.zero)); - expect(args['readIdleTimeoutMs'], isNull); - }); - - test('rounds up to whole milliseconds', () { - final headers = Headers()..setReadIdleTimeout(const Duration(microseconds: 1)); - expect(headers[readIdleTimeoutHeader], '1'); + test('zero is none', () async { + final configured = await CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration.zero)); + expect((await openArgs(configured))['readIdleTimeoutMs'], isNull); }); - test('a malformed header fails the call before it opens', () async { - for (final value in ['-1', '1.5', '0x10', '+5', 'soon']) { - final headers = Headers()..[readIdleTimeoutHeader] = value; - await expectLater( - client.httpClient(request(null, headers: headers)), - throwsA(isA().having((e) => e.code, 'code', Code.internal)), - reason: value, - ); - } - expect(plugin.callsTo('open'), isEmpty); + test('rounds up to whole milliseconds', () async { + final configured = await CronetHttpClient.create( + const NativeHttpConfig(readIdleTimeout: Duration(microseconds: 1)), + ); + expect((await openArgs(configured))['readIdleTimeoutMs'], 1); }); test('a negative timeout is refused', () async { - expect(() => Headers().setReadIdleTimeout(const Duration(seconds: -1)), throwsArgumentError); await expectLater( CronetHttpClient.create(const NativeHttpConfig(readIdleTimeout: Duration(seconds: -1))), throwsArgumentError, From 25afc05c070d4222ab5cd2d5c66fc7ff416680d7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:04:32 +0000 Subject: [PATCH 5/7] Step the deadline down per hop instead of a grace timer after it The client now sends the server a connect-timeout-ms 1 s shorter than the call's own deadline (or half of it, under 2 s), so a reachable server answers deadline_exceeded before the client's deadline fires. When the client's deadline does fire, the server's answer never came: the bridge cancels the call at once and evicts its connection, under the same "nothing else arrived from the origin" check as a read-idle timeout. The separate 1 s timer after the deadline is gone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- README.md | 23 +++++----- .../connect_native_http/CronetBridgeTest.kt | 12 ++--- .../connect_native_http/CronetBridge.kt | 35 +++------------ example/integration_test/eliza_test.dart | 15 ++++--- .../URLSessionBridge.swift | 29 ++++-------- .../URLSessionBridgeTests.swift | 17 ++++--- lib/src/channel/channel_http_client.dart | 45 +++++++++++-------- lib/src/native_http_config.dart | 3 +- test/cronet_http_client_test.dart | 35 ++++++++++----- 9 files changed, 104 insertions(+), 110 deletions(-) diff --git a/README.md b/README.md index 7090941..bfb9363 100644 --- a/README.md +++ b/README.md @@ -84,11 +84,12 @@ off such a connection, so they don't wait on it too. - **Read-idle timeout:** the connection counts as dead if nothing has arrived from the call's origin (host and port), on any call, since the call last received anything. If something has, the connection is alive and only that call was slow, so nothing changes. -- **Deadline:** the call fails at its deadline, as always. connect-dart sends the deadline in `connect-timeout-ms`, so - the server times the call out too and answers `deadline_exceeded`. Its clock starts later, so its answer arrives - about a round trip after the client's deadline. The bridge therefore keeps the stream open for 1 s more before - canceling it. If anything arrives from the origin in that time, the server's answer included, the connection is - alive. If nothing does, it counts as dead. +- **Deadline:** each hop takes a step off the deadline it passes on. connect-dart sends the call's deadline in + `connect-timeout-ms`, and the client sends the server 1 s less, or half the deadline when that is less, so the + server never gets zero. A server that is reachable then times the call out first and answers `deadline_exceeded` + before the client's own deadline fires. So when the client's deadline fires, the server's answer never came, and the + connection counts as dead under the same check as a read-idle timeout. Servers that call further services should + take their own step off the deadline they pass on. - **A dead connection is evicted.** Neither stack can close a single connection, so the client moves its new calls to a fresh `URLSession` or `CronetEngine`, which dials its own connections. Calls already on the old one are not canceled. They finish there, and it is released after the last one ends. Several calls that run out of time on the @@ -188,7 +189,8 @@ or a read-idle timeout (see "Calls that run out of time"). The integration tests deadline. The call after that goes out on a new connection. - With a read-idle timeout, the call fails within the timeout, and the client's next call goes out on a new connection. -- A call the server answers with `deadline_exceeded` leaves the client on the same connection. +- The server answers a deadline with `deadline_exceeded` a step before the client's own deadline, and the client + stays on the same connection. ### QUIC idle timeout vs HTTP/2 keepalive pings @@ -289,15 +291,16 @@ Neither platform finishes a stream on its own while the client still has its req - A frozen connection draws no error, and new calls go to it until one hits its deadline. The call after that is on a new connection. - With a read-idle timeout, the call fails and the client's next call is on a new connection. - - A deadline the server answers leaves the client on the same connection. + - The server answers a deadline a step before the client's, and the client stays on the same connection. - **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's bridge (`URLSessionBridge`, `CronetBridge`) has no Flutter dependency, and the tests call it directly, for behavior Dart can't reach: - **iOS (`ios/connect_native_http/Tests`, XCTest):** a write stuck on a full buffer returns once the call is canceled or its connection fails; a redirect of a request with a body is returned, not followed; shutting a session down lets its calls finish and refuses new ones; a read-idle timeout fails a quiet call with - `NSURLErrorTimedOut`, and the engine's next call works; a deadline cancel waits out its grace period. The script runs them on the simulator. For a quick loop - on the Mac, against a running test server: + `NSURLErrorTimedOut`, and the engine's next call works; a call canceled at its deadline ends at once, and the + engine's next call works. The script runs them on the simulator. For a quick loop on the Mac, against a running + test server: ```bash cd ios/connect_native_http && CONNECT_NATIVE_HTTP_CORE_ONLY=1 SERVER_URL=https://localhost:8443 \ @@ -308,7 +311,7 @@ Neither platform finishes a stream on its own while the client still has its req write after the server finished the call is dropped instead of crashing; shutting an engine down lets its calls finish and refuses new ones; a header Cronet rejects fails its call as never sent; an empty write is skipped; a read-idle timeout fails a quiet call with -7, once, and the engine's next call works; data arriving resets the - timer; a deadline cancel waits out its grace period. + timer; a call canceled at its deadline ends at once, and the engine's next call works. - Both platforms' suites also check that malformed method calls get an error reply and leave the bridge working. - **`example/integration_test/public_server_test.dart`:** HTTP/3 on Android against a server with a public certificate. It also checks that `quicIdleTimeout` ends a quiet connection, against a control without it. It is diff --git a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt index ec91847..3abd162 100644 --- a/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt +++ b/android/src/androidTest/kotlin/com/agtonomy/connect_native_http/CronetBridgeTest.kt @@ -130,17 +130,14 @@ class CronetBridgeTest { assertEquals("done", awaitOutcome(1)["type"]) } - // After a deadline the stream stays open for the grace period, for a sign that its connection is alive. + // A call that hit its deadline is canceled at once, and moves the engine's new calls to a fresh CronetEngine. @Test - fun aDeadlineCancelWaitsOutTheGracePeriod() { + fun aDeadlineCancelEndsTheCallAndTheEngineCarriesOn() { val engine = createEngine() open(engine, 1, CONVERSE, "application/connect+proto") await(1, "data") - val start = System.nanoTime() - invoke("cancel", mapOf("callId" to 1, "deadlineGraceMs" to 500)) + invoke("cancel", mapOf("callId" to 1, "evictConnection" to true)) assertEquals("canceled", awaitOutcome(1)["type"]) - val elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) - assertTrue("canceled after $elapsedMs ms, before the 500 ms grace period", elapsedMs >= 450) assertSayWorks(engine, 2) } @@ -156,8 +153,7 @@ class CronetBridgeTest { "shutdownEngine" to mapOf("engineId" to 1.5), "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to 0), "open" to openArguments(engine, 1, SAY, "application/proto") + ("readIdleTimeoutMs" to "soon"), - "cancel" to mapOf("callId" to 1, "deadlineGraceMs" to 0), - "cancel" to mapOf("callId" to 1, "deadlineGraceMs" to "soon"), + "cancel" to mapOf("callId" to 1, "evictConnection" to "yes"), ) for ((method, arguments) in malformed) { val reply = reply(method, arguments) diff --git a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt index 7e8f1e1..fde3dd1 100644 --- a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt +++ b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt @@ -168,16 +168,15 @@ class CronetBridge( } "cancel" -> { val callId = args["callId"] as Int - // Set when the call hit its deadline: see cancelAfterDeadline. - val deadlineGraceMs = (args["deadlineGraceMs"] as Number?)?.toLong() - require(deadlineGraceMs == null || deadlineGraceMs > 0) { "deadlineGraceMs: $deadlineGraceMs" } + // Set when the call hit its deadline. The server was given a deadline a step earlier, so on a live + // connection its deadline_exceeded would have ended the call first. + val evictConnection = args["evictConnection"] as Boolean? ?: false execute(callId) { val c = calls[callId] ?: return@execute - if (deadlineGraceMs != null) { - cancelAfterDeadline(c, deadlineGraceMs) - } else { - c.stream.cancel() + if (evictConnection) { + evict(c) } + c.stream.cancel() } result(Reply.Success(null)) } @@ -276,27 +275,7 @@ class CronetBridge( c.pool.lastReceived[c.origin] = now } - // The call has hit its deadline and Dart has already failed it. The server enforces the same deadline and answers - // about a round trip later, so the stream stays open for the grace period: an answer, or anything else from the - // origin, shows the connection is alive. After it, the stream is canceled, and the connection evicted if it stayed - // quiet. - private fun cancelAfterDeadline(c: Call, graceMs: Long) { - val check = Runnable { - try { - // Gone: the server's answer ended it, or Cronet failed it. - if (calls[c.callId] === c) { - evict(c) - c.stream.cancel() - } - } catch (e: Exception) { - Log.e(TAG, "call ${c.callId} failed in the bridge", e) - abort(c.callId, e) - } - } - executor.schedule(check, graceMs, TimeUnit.MILLISECONDS) - } - - // A call that ran out of time (its read-idle timeout, or its deadline and grace) may be stuck on a dead connection. + // A call that ran out of time (its read-idle timeout, or its deadline) may be stuck on a dead connection. // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received // anything; otherwise only that call was slow. Cronet has no call that closes one connection, so the engine's // CronetEngine is retired: new calls go to a fresh one, which dials its own connections, and the old one shuts down diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index 6471130..e914c16 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -415,8 +415,7 @@ void main() { h2.eliza.say(SayRequest(sentence: 'hi'), signal: TimeoutSignal(const Duration(seconds: 2))), throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), ); - // Nothing arrives in the grace period after the deadline, so the next call dials a new connection. - await Future.delayed(const Duration(milliseconds: 1500)); + // The server's deadline_exceeded never came, so the next call dials a new connection. final res = await h2.eliza.say( SayRequest(sentence: 'after'), signal: TimeoutSignal(const Duration(seconds: 8)), @@ -428,19 +427,21 @@ void main() { await requests.close(); }, timeout: http2CaseTimeout); - testCase('a deadline the server answers keeps the connection', (_) async { - // connect-dart sends the deadline, and the server answers deadline_exceeded about a round trip after the client's - // own deadline, within the grace period: the connection is alive. + testCase('the server answers a deadline a step early, and the connection is kept', (_) async { + // The server gets a deadline a second shorter than the client's, so its deadline_exceeded ends the call before the + // client's own deadline fires, and shows the connection is alive. if (skipUnlessAdmin()) { return; } final h2 = await harnessOverHttp2(); final liveAddr = h2.remoteAddrs.last; + final stopwatch = Stopwatch()..start(); await expectLater( - h2.eliza.say(SayRequest(sentence: 'sleep:5000'), signal: TimeoutSignal(const Duration(seconds: 1))), + h2.eliza.say(SayRequest(sentence: 'sleep:5000'), signal: TimeoutSignal(const Duration(seconds: 3))), throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), ); - await Future.delayed(const Duration(milliseconds: 1500)); + // The server's 2 s, not the client's 3 s. + expect(stopwatch.elapsed, lessThan(const Duration(milliseconds: 2800))); await h2.eliza.say(SayRequest(sentence: 'after'), onHeader: h2.recordHeaders); expect(h2.remoteAddrs.last, liveAddr); }, timeout: http2CaseTimeout); diff --git a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift index 3b70fe5..1b2bfdf 100644 --- a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift +++ b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift @@ -144,15 +144,17 @@ public final class URLSessionBridge { reply(.success(nil)) case "cancel": let callId: Int = try arg(args, "callId") - // Set when the call hit its deadline: see `cancelAfterDeadline`. - let deadlineGraceMs = try optionalPositiveInt(args, "deadlineGraceMs") + // Set when the call hit its deadline. The server was given a deadline a step earlier, so on a live connection + // its deadline_exceeded would have ended the call first. + let value = args["evictConnection"] + guard value == nil || value is NSNull || value is Bool else { throw BadArgument(name: "evictConnection") } + let evict = value as? Bool ?? false queue.async { guard let c = self.calls[callId] else { return } - if let deadlineGraceMs { - self.cancelAfterDeadline(c, graceMs: deadlineGraceMs) - } else { - c.task.cancel() + if evict { + self.evictConnection(of: c) } + c.task.cancel() } reply(.success(nil)) default: @@ -317,7 +319,7 @@ public final class URLSessionBridge { emit(failure(c.callId, domain: error.domain, code: error.code, message: error.localizedDescription)) } - // A call that ran out of time (its read-idle timeout, or its deadline and grace) may be stuck on a dead connection. + // A call that ran out of time (its read-idle timeout, or its deadline) may be stuck on a dead connection. // It is taken to be if nothing has arrived from the call's origin, on any call, since the call last received // anything; otherwise only that call was slow. URLSession pools connections per session and has no call that closes // one, so the engine's new calls go to a fresh URLSession, which dials its own connections. The old one finishes the @@ -333,19 +335,6 @@ public final class URLSessionBridge { old.finishTasksAndInvalidate() } - // The call has hit its deadline and Dart has already failed it. The server enforces the same deadline and answers - // about a round trip later, so the task stays open for the grace period: an answer, or anything else from the - // origin, shows the connection is alive. After it, the task is canceled, and the connection evicted if it stayed - // quiet. - private func cancelAfterDeadline(_ c: Call, graceMs: Int) { - queue.asyncAfter(deadline: .now() + .milliseconds(graceMs)) { - // Gone: the server's answer ended it, or the stack failed it. - guard self.calls[c.callId] === c else { return } - self.evictConnection(of: c) - c.task.cancel() - } - } - fileprivate func serverTrust(_ challenge: URLAuthenticationChallenge) -> ( URLSession.AuthChallengeDisposition, URLCredential? ) { diff --git a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift index 31644ca..06bb7d5 100644 --- a/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift +++ b/ios/connect_native_http/Tests/ConnectNativeHttpCoreTests/URLSessionBridgeTests.swift @@ -116,16 +116,20 @@ final class URLSessionBridgeTests: XCTestCase { XCTAssertTrue(received(2).contains("say: after")) } - // After a deadline the task stays open for the grace period, for a sign that its connection is alive. - func testADeadlineCancelWaitsOutTheGracePeriod() async throws { + // A call that hit its deadline is canceled at once, and moves the engine's new calls to a fresh URLSession. + func testADeadlineCancelEndsTheCallAndTheEngineCarriesOn() async throws { let engine = try await createEngine() try await open(engine, callId: 1, path: Self.converse, contentType: "application/connect+proto") try await waitUntil("the greeting arrived", timeout: 10) { self.received(1).contains("hello") } - let start = Date() - try await invoke("cancel", ["callId": 1, "deadlineGraceMs": 500]) + try await invoke("cancel", ["callId": 1, "evictConnection": true]) try await waitUntil("the call was canceled") { self.outcome(1) != nil } XCTAssertEqual(outcome(1)?["type"] as? String, "canceled", "\(String(describing: outcome(1)))") - XCTAssertGreaterThanOrEqual(Date().timeIntervalSince(start), 0.45, "canceled before the 500 ms grace period") + + try await open(engine, callId: 2, path: Self.say, contentType: "application/proto") + try await write(2, sentence("after")) + try await invoke("end", ["callId": 2]) + try await waitUntil("the next call ended", timeout: 10) { self.outcome(2) != nil } + XCTAssertEqual(outcome(2)?["type"] as? String, "done", "\(String(describing: outcome(2)))") } func testRequestTimeoutIntervalAvoidsTheValueURLSessionIgnores() { @@ -150,8 +154,7 @@ final class URLSessionBridgeTests: XCTestCase { ("write", ["callId": 1, "data": "not bytes"]), ("end", ["callId": "1"]), ("cancel", [:]), - ("cancel", ["callId": 1, "deadlineGraceMs": 0]), - ("cancel", ["callId": 1, "deadlineGraceMs": "soon"]), + ("cancel", ["callId": 1, "evictConnection": "yes"]), ("shutdownEngine", ["engineId": NSNull()]), ("createEngine", ["http3Origins": [["localhost"]]]), ] diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index 6f14fee..d02b099 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -1,4 +1,5 @@ import 'dart:async'; +import 'dart:math' as math; import 'package:connectrpc/connect.dart'; import 'package:flutter/foundation.dart'; @@ -134,15 +135,14 @@ final class _Dispatcher { /// Asks the plugin to cancel call [id]; its `canceled` event then ends the call. Should the plugin fail to take the /// request, no event will come, so the call ends here instead. /// - /// With [deadlineGrace], the call has hit its deadline and already failed here. The plugin then waits up to that long - /// before canceling it, for anything from the call's origin: the server's own `deadline_exceeded`, typically, since - /// connect-dart sends the deadline and the server's clock starts later. If nothing arrives, the call is taken to be - /// stuck on a dead connection, and new calls move to a fresh one. - Future cancel(int id, {Duration? deadlineGrace}) async { + /// With [evictConnection], the call hit its deadline without the server's own `deadline_exceeded`, which a live + /// connection would have delivered first (see [serverTimeoutMs]). The plugin then moves new calls off the call's + /// connection, unless something else from the call's origin has arrived since the call last received anything. + Future cancel(int id, {bool evictConnection = false}) async { try { await bridge.methods.invokeMethod('cancel', { 'callId': id, - if (deadlineGrace != null) 'deadlineGraceMs': deadlineGrace.inMilliseconds, + if (evictConnection) 'evictConnection': true, }); } catch (error, stack) { final call = remove(id); @@ -239,9 +239,22 @@ final class _Dispatcher { } } -/// How long the plugin keeps a call that hit its deadline, waiting for a sign its connection is alive. Covers the -/// server's own `deadline_exceeded`, which arrives about a round trip after the client's deadline. -const deadlineGrace = Duration(seconds: 1); +/// What each hop takes off the deadline it passes on: the server gets a deadline this much shorter than the client's +/// own, so on a live connection its `deadline_exceeded` arrives before the client's deadline fires. +const deadlineStep = Duration(seconds: 1); + +/// The `connect-timeout-ms` to send for a call whose deadline is [ms] away: [deadlineStep] less, or half of [ms] when +/// that is less, so the server never gets zero. +int serverTimeoutMs(int ms) => ms - math.min(deadlineStep.inMilliseconds, ms ~/ 2); + +/// connect-dart's deadline header, which it sets from the call's signal. +const _connectTimeoutHeader = 'connect-timeout-ms'; + +/// A `connect-timeout-ms` value with [serverTimeoutMs] applied; anything but a positive number passes unchanged. +String _stepDown(String value) => switch (int.tryParse(value)) { + final ms? when ms > 0 => '${serverTimeoutMs(ms)}', + _ => value, +}; /// A connect-dart [HttpClient] on one engine of a [PlatformBridge] plugin. base class ChannelHttpClient implements NativeHttpClient { @@ -289,15 +302,8 @@ base class ChannelHttpClient implements NativeHttpClient { return; } call.abortError = error; - if (error.code == Code.deadlineExceeded) { - // The caller gets the deadline now; the plugin keeps the stream a little longer to tell whether its connection - // is dead (see _Dispatcher.cancel), and nothing it reports for the call matters any more. - dispatcher.remove(id); - call.fail(error); - unawaited(dispatcher.cancel(id, deadlineGrace: deadlineGrace)); - return; - } - unawaited(dispatcher.cancel(id)); + // The server's deadline was a step earlier, so a deadline that fires here never got the server's answer. + unawaited(dispatcher.cancel(id, evictConnection: error.code == Code.deadlineExceeded)); }); try { await methods.invokeMethod('open', { @@ -306,7 +312,8 @@ base class ChannelHttpClient implements NativeHttpClient { 'url': req.url, 'method': req.method, 'headers': [ - for (final (:name, :value) in req.header.entries) [name, value], + for (final (:name, :value) in req.header.entries) + [name, name == _connectTimeoutHeader ? _stepDown(value) : value], ], 'hasBody': req.body != null, 'readIdleTimeoutMs': _readIdleTimeoutMs, diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index 9e99bf4..ab5e6d9 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -47,7 +47,8 @@ final class NativeHttpConfig { /// - **The connection:** if nothing has arrived from the call's origin, on any call, since the call last received /// anything, the connection is taken to be dead. The client moves new calls to a fresh `URLSession` or Cronet /// engine, so they dial a new connection, and lets the calls already on the old one finish there. A call that hits - /// its deadline is checked the same way, after a 1 s grace period for the server's own `deadline_exceeded`. + /// its deadline is checked the same way: the server is sent a deadline a step shorter, so on a live connection its + /// `deadline_exceeded` arrives first. final Duration? readIdleTimeout; /// Called with each response's headers, before its body is read. An exception it throws goes to diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 3d61b84..77745f3 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -248,30 +248,45 @@ void main() { await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.canceled))); }); - test('a deadline fails the call at once, and gives the plugin time to check the connection', () async { + test("a deadline cancels the stream and asks the plugin to leave the call's connection", () async { final signal = TimeoutSignal(const Duration(milliseconds: 10)); final response = client.httpClient(request(StreamController().stream, signal: signal)); await settle(); final id = plugin.lastCallId; - // No canceled event: the call ends without waiting for the plugin. - await expectLater( - response.timeout(const Duration(seconds: 1)), - throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), - ); + await Future.delayed(const Duration(milliseconds: 50)); final cancel = plugin.callsTo('cancel').single.arguments as Map; expect(cancel['callId'], id); - expect(cancel['deadlineGraceMs'], 1000); - // The plugin's later events for the call change nothing. + expect(cancel['evictConnection'], isTrue); await plugin.emit({'callId': id, 'type': 'canceled'}); + await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded))); }); - test('a signal that is not a deadline cancels at once', () async { + test('a signal that is not a deadline leaves the connection alone', () async { final signal = CancelableSignal(); unawaited(client.httpClient(request(null, signal: signal)).then((_) {}, onError: (_) {})); await settle(); signal.cancel('stop'); await settle(); - expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('deadlineGraceMs'), isFalse); + expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('evictConnection'), isFalse); + }); + + test("the server's deadline is a step shorter than the client's", () async { + Future sent(String timeout) async { + final headers = Headers() + ..add('content-type', 'application/proto') + ..add('connect-timeout-ms', timeout); + final req = HttpRequest('https://example.test/svc/Method', 'POST', headers, null, null); + unawaited(client.httpClient(req).then((_) {}, onError: (_) {})); + await settle(); + final pairs = (plugin.callsTo('open').last.arguments as Map)['headers'] as List; + return pairs.cast>().firstWhere((pair) => pair[0] == 'connect-timeout-ms')[1] as String?; + } + + expect(await sent('5000'), '4000'); + // Under two steps, half: the server never gets zero. + expect(await sent('1500'), '750'); + expect(await sent('1'), '1'); + expect(await sent('soon'), 'soon'); }); test('cancelling the response body cancels the native stream', () async { From 3e1b1ce743f77c1bba56e47f170cdbb80de81110 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:32:27 +0000 Subject: [PATCH 6/7] Make the deadline step and eviction on a deadline opt-in Both are now NativeHttpConfig options, off by default: - deadlineStep: how much shorter a deadline the server gets in connect-timeout-ms than the call's own (half the deadline when that is less). Unset, the deadline is sent unchanged. - evictOnDeadline: whether a call that hits its deadline on the client moves new calls off its connection, under the usual guard. Unset, a deadline just cancels the call. The docs recommend using them together: without a step, the client's deadline usually beats the server's deadline_exceeded on a healthy connection. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- README.md | 51 +++++++++------ example/integration_test/eliza_test.dart | 35 +++++++--- lib/src/channel/channel_http_client.dart | 31 ++++----- lib/src/native_http_config.dart | 36 ++++++++-- test/cronet_http_client_test.dart | 83 ++++++++++++++++-------- 5 files changed, 159 insertions(+), 77 deletions(-) diff --git a/README.md b/README.md index bfb9363..ac5a6d5 100644 --- a/README.md +++ b/README.md @@ -62,8 +62,8 @@ NativeHttpConfig(readIdleTimeout: const Duration(seconds: 30)); ``` It applies to every call on the client. For per-call bounds use deadlines (connect-dart's `TimeoutSignal` or -`DeadlineSignal`), which also move the client off a dead connection (see "Calls that run out of time"). Streams that -go quiet for long need a client without a read-idle timeout, or periodic messages. +`DeadlineSignal`), which can also move the client off a dead connection (see "Calls that run out of time"). Streams +that go quiet for long need a client without a read-idle timeout, or periodic messages. - **What it measures:** time since the call last received anything: headers, a body chunk or trailers. The timer starts when the call does, so it also bounds the wait for the response headers. A unary call the server holds longer than @@ -78,18 +78,30 @@ go quiet for long need a client without a read-idle timeout, or periodic message ### Calls that run out of time -A call that hits its read-idle timeout or its deadline may be stuck on a dead connection. The client keeps new calls -off such a connection, so they don't wait on it too. +A call that hits its read-idle timeout, or its deadline with `evictOnDeadline`, may be stuck on a dead connection. +The client keeps new calls off such a connection, so they don't wait on it too. + +```dart +NativeHttpConfig( + readIdleTimeout: const Duration(seconds: 30), // optional + deadlineStep: const Duration(seconds: 1), + evictOnDeadline: true, +); +``` - **Read-idle timeout:** the connection counts as dead if nothing has arrived from the call's origin (host and port), on any call, since the call last received anything. If something has, the connection is alive and only that call was slow, so nothing changes. -- **Deadline:** each hop takes a step off the deadline it passes on. connect-dart sends the call's deadline in - `connect-timeout-ms`, and the client sends the server 1 s less, or half the deadline when that is less, so the - server never gets zero. A server that is reachable then times the call out first and answers `deadline_exceeded` - before the client's own deadline fires. So when the client's deadline fires, the server's answer never came, and the - connection counts as dead under the same check as a read-idle timeout. Servers that call further services should - take their own step off the deadline they pass on. +- **Deadline, with `evictOnDeadline`:** the connection counts as dead under the same check as a read-idle timeout. + Both options are off by default; use them together: + - **`deadlineStep`** takes a step off the deadline each hop passes on. connect-dart sends the call's deadline in + `connect-timeout-ms`, and the client sends the server that much less, or half the deadline when that is less, so + the server never gets zero. A reachable server then times the call out first and answers `deadline_exceeded` + before the client's own deadline fires. So when the client's deadline fires, the server's answer never came. + Servers that call further services should take their own step off the deadline they pass on. + - **Without a step,** the server enforces the same deadline, but its clock starts later. On a healthy connection the + client's deadline usually fires first, and with `evictOnDeadline` alone the connection would be evicted for + nothing. - **A dead connection is evicted.** Neither stack can close a single connection, so the client moves its new calls to a fresh `URLSession` or `CronetEngine`, which dials its own connections. Calls already on the old one are not canceled. They finish there, and it is released after the last one ends. Several calls that run out of time on the @@ -182,15 +194,15 @@ Clients may enforce a floor on the peer's value. quic-go's client raises anythin (`protocol.MinRemoteIdleTimeout`), so measure any other client before relying on a short server value. None of this works on the **HTTP/2 fallback** used where UDP is blocked. Neither stack exposes HTTP/2 pings, so a TCP -connection that dies without the interface going down goes unnoticed until a call on it runs out of time: a deadline -or a read-idle timeout (see "Calls that run out of time"). The integration tests check this: +connection that dies without the interface going down goes unnoticed until a call on it runs out of time: a read-idle +timeout, or a deadline with `evictOnDeadline` (see "Calls that run out of time"). The integration tests check this: - Over a frozen HTTP/2 connection nothing arrives for 3 s, and a new call on the same client waits on it until its - deadline. The call after that goes out on a new connection. + deadline. With `evictOnDeadline`, the call after that goes out on a new connection. - With a read-idle timeout, the call fails within the timeout, and the client's next call goes out on a new connection. -- The server answers a deadline with `deadline_exceeded` a step before the client's own deadline, and the client - stays on the same connection. +- With `deadlineStep`, the server answers a deadline with `deadline_exceeded` a step before the client's own + deadline, and the client stays on the same connection. ### QUIC idle timeout vs HTTP/2 keepalive pings @@ -203,7 +215,7 @@ not over HTTP/2. | Who sets it | Both advertise a value; the connection uses the smaller (RFC 9000 §10.1) | Each side sets its own pings | | Can a server setting make the client notice? | Yes. A short server value becomes the client's timeout, and the server's pings stop arriving once the path dies. | No. A server that detects a dead connection can't tell the client, because nothing gets through. | | Through a load balancer | NLB QUIC passthrough is end to end, so the client sees the server's pings | An ALB ends HTTP/2 on each side, so the server's pings stop at the ALB. AWS doesn't document the ALB pinging clients, and nothing here relies on it. | -| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout, and eviction after a deadline. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout, and eviction after a deadline. | +| Client control here | Android: `quicIdleTimeout`. iOS: none. Both: a read-idle timeout, and `evictOnDeadline`. | No pings: neither URLSession nor Cronet exposes HTTP/2 ping settings. Both: a read-idle timeout, and `evictOnDeadline`. | | What a client that sends its own pings gets, for comparison | — | `package:http2` with 3 s pings and a 3 s timeout notices in about 6 s | So for servers: @@ -288,10 +300,11 @@ Neither platform finishes a stream on its own while the client still has its req address in `X-Remote-Addr`, which shows whether two calls shared a connection. The cases check five things: - A quiet stream fails at its timeout. - A stream that keeps receiving outlives its timeout. - - A frozen connection draws no error, and new calls go to it until one hits its deadline. The call after that is - on a new connection. + - A frozen connection draws no error, and new calls go to it until one hits its deadline. With `evictOnDeadline`, + the call after that is on a new connection. - With a read-idle timeout, the call fails and the client's next call is on a new connection. - - The server answers a deadline a step before the client's, and the client stays on the same connection. + - With `deadlineStep`, the server answers a deadline a step before the client's, and the client stays on the same + connection. - **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's bridge (`URLSessionBridge`, `CronetBridge`) has no Flutter dependency, and the tests call it directly, for behavior Dart can't reach: diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index e914c16..4ecffa1 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -38,12 +38,20 @@ const introduceInterval = Duration(milliseconds: 100); final class Harness { Harness._(this.native, this.eliza); - static Future create(String baseUrl, {Set http3Origins = const {}, Duration? readIdleTimeout}) async { + static Future create( + String baseUrl, { + Set http3Origins = const {}, + Duration? readIdleTimeout, + Duration? deadlineStep, + bool evictOnDeadline = false, + }) async { late final Harness harness; final native = await NativeHttpClient.create( NativeHttpConfig( http3Origins: http3Origins, readIdleTimeout: readIdleTimeout, + deadlineStep: deadlineStep, + evictOnDeadline: evictOnDeadline, onResponse: (info) => harness.platformProtocols.add(info.protocol), ), ); @@ -131,10 +139,19 @@ Future harnessNegotiating(String protocol) async { /// A client on HTTP/2, with the server's UDP frozen for the rest of the case so that QUIC can't connect: URLSession /// then falls back to TCP, as on a network that blocks UDP. Android is on HTTP/2 with this server anyway. -Future harnessOverHttp2({Duration? readIdleTimeout}) async { +Future harnessOverHttp2({ + Duration? readIdleTimeout, + Duration? deadlineStep, + bool evictOnDeadline = false, +}) async { addTearDown(() => admin('/unfreeze')); await admin('/freeze?for=60s'); - final harness = await Harness.create(serverUrl, readIdleTimeout: readIdleTimeout); + final harness = await Harness.create( + serverUrl, + readIdleTimeout: readIdleTimeout, + deadlineStep: deadlineStep, + evictOnDeadline: evictOnDeadline, + ); addTearDown(harness.native.close); await harness.eliza.say( SayRequest(sentence: 'which protocol'), @@ -397,12 +414,12 @@ void main() { }, timeout: http2CaseTimeout); testCase('a dead HTTP/2 connection is reused until a call runs out of time on it', (_) async { - // Nothing notices the dead connection by itself; the first call to run out of time, here on its deadline, moves - // the client off it. + // Nothing notices the dead connection by itself; with evictOnDeadline, the first call to run out of time, here on + // its deadline, moves the client off it. if (skipUnlessAdmin()) { return; } - final h2 = await harnessOverHttp2(); + final h2 = await harnessOverHttp2(deadlineStep: const Duration(seconds: 1), evictOnDeadline: true); final (requests, responses) = await openQuietConverse(h2); final deadAddr = h2.remoteAddrs.last; await admin('/freeze?for=60s&path=tcp'); @@ -428,12 +445,12 @@ void main() { }, timeout: http2CaseTimeout); testCase('the server answers a deadline a step early, and the connection is kept', (_) async { - // The server gets a deadline a second shorter than the client's, so its deadline_exceeded ends the call before the - // client's own deadline fires, and shows the connection is alive. + // With deadlineStep, the server gets a deadline a second shorter than the client's, so its deadline_exceeded ends + // the call before the client's own deadline fires, and shows the connection is alive. if (skipUnlessAdmin()) { return; } - final h2 = await harnessOverHttp2(); + final h2 = await harnessOverHttp2(deadlineStep: const Duration(seconds: 1), evictOnDeadline: true); final liveAddr = h2.remoteAddrs.last; final stopwatch = Stopwatch()..start(); await expectLater( diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index d02b099..9cef5c9 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -135,9 +135,9 @@ final class _Dispatcher { /// Asks the plugin to cancel call [id]; its `canceled` event then ends the call. Should the plugin fail to take the /// request, no event will come, so the call ends here instead. /// - /// With [evictConnection], the call hit its deadline without the server's own `deadline_exceeded`, which a live - /// connection would have delivered first (see [serverTimeoutMs]). The plugin then moves new calls off the call's - /// connection, unless something else from the call's origin has arrived since the call last received anything. + /// With [evictConnection], the call hit its deadline (see [NativeHttpConfig.evictOnDeadline]). The plugin then moves + /// new calls off the call's connection, unless something from the call's origin has arrived since the call last + /// received anything. Future cancel(int id, {bool evictConnection = false}) async { try { await bridge.methods.invokeMethod('cancel', { @@ -239,20 +239,16 @@ final class _Dispatcher { } } -/// What each hop takes off the deadline it passes on: the server gets a deadline this much shorter than the client's -/// own, so on a live connection its `deadline_exceeded` arrives before the client's deadline fires. -const deadlineStep = Duration(seconds: 1); - -/// The `connect-timeout-ms` to send for a call whose deadline is [ms] away: [deadlineStep] less, or half of [ms] when -/// that is less, so the server never gets zero. -int serverTimeoutMs(int ms) => ms - math.min(deadlineStep.inMilliseconds, ms ~/ 2); +/// The `connect-timeout-ms` to send for a call whose deadline is [ms] away: [step] less, or half of [ms] when that is +/// less, so the server never gets zero. See [NativeHttpConfig.deadlineStep]. +int serverTimeoutMs(int ms, Duration step) => ms - math.min(step.inMilliseconds, ms ~/ 2); /// connect-dart's deadline header, which it sets from the call's signal. const _connectTimeoutHeader = 'connect-timeout-ms'; /// A `connect-timeout-ms` value with [serverTimeoutMs] applied; anything but a positive number passes unchanged. -String _stepDown(String value) => switch (int.tryParse(value)) { - final ms? when ms > 0 => '${serverTimeoutMs(ms)}', +String _stepDown(String value, Duration step) => switch (int.tryParse(value)) { + final ms? when ms > 0 => '${serverTimeoutMs(ms, step)}', _ => value, }; @@ -270,6 +266,10 @@ base class ChannelHttpClient implements NativeHttpClient { if (readIdle != null && readIdle.isNegative) { throw ArgumentError.value(readIdle, 'readIdleTimeout', 'must not be negative'); } + final step = config.deadlineStep; + if (step != null && step.isNegative) { + throw ArgumentError.value(step, 'deadlineStep', 'must not be negative'); + } _Dispatcher.of(bridge); return (await bridge.methods.invokeMethod('createEngine', arguments))!; } @@ -290,6 +290,7 @@ base class ChannelHttpClient implements NativeHttpClient { Future _send(HttpRequest req) async { final dispatcher = _Dispatcher.of(_bridge); final methods = _bridge.methods; + final step = _config.deadlineStep == Duration.zero ? null : _config.deadlineStep; final call = _Call(Uri.parse(req.url)); final id = dispatcher.register(call, _config.onResponse); call.body.onCancel = () { @@ -302,8 +303,8 @@ base class ChannelHttpClient implements NativeHttpClient { return; } call.abortError = error; - // The server's deadline was a step earlier, so a deadline that fires here never got the server's answer. - unawaited(dispatcher.cancel(id, evictConnection: error.code == Code.deadlineExceeded)); + final evict = _config.evictOnDeadline && error.code == Code.deadlineExceeded; + unawaited(dispatcher.cancel(id, evictConnection: evict)); }); try { await methods.invokeMethod('open', { @@ -313,7 +314,7 @@ base class ChannelHttpClient implements NativeHttpClient { 'method': req.method, 'headers': [ for (final (:name, :value) in req.header.entries) - [name, name == _connectTimeoutHeader ? _stepDown(value) : value], + [name, name == _connectTimeoutHeader && step != null ? _stepDown(value, step) : value], ], 'hasBody': req.body != null, 'readIdleTimeoutMs': _readIdleTimeoutMs, diff --git a/lib/src/native_http_config.dart b/lib/src/native_http_config.dart index ab5e6d9..c453e84 100644 --- a/lib/src/native_http_config.dart +++ b/lib/src/native_http_config.dart @@ -12,7 +12,14 @@ final class NativeHttpResponseInfo { /// Configuration shared by both platform clients. Server certificates are checked against the platform's trust /// store. final class NativeHttpConfig { - const NativeHttpConfig({this.http3Origins = const {}, this.quicIdleTimeout, this.readIdleTimeout, this.onResponse}); + const NativeHttpConfig({ + this.http3Origins = const {}, + this.quicIdleTimeout, + this.readIdleTimeout, + this.deadlineStep, + this.evictOnDeadline = false, + this.onResponse, + }); /// `https://host:port` origins to try HTTP/3 with from the first request, before an `Alt-Svc` response header has /// advertised it. Both platforms fall back to HTTP/2 over TCP on their own when QUIC fails. @@ -34,8 +41,8 @@ final class NativeHttpConfig { /// fallback, where nothing else notices a dead connection. /// /// It applies to every call, so leave it off on a client that carries streams which go quiet for long, or use a - /// second client for them. A call's own deadline (connect-dart's `TimeoutSignal` or `DeadlineSignal`) also moves new - /// calls off a dead connection; see "Calls that run out of time" in the README. + /// second client for them. For per-call bounds use deadlines (connect-dart's `TimeoutSignal` or `DeadlineSignal`), + /// with [evictOnDeadline] to move new calls off a dead connection. /// /// - **What resets it:** anything the call receives. It runs from when the call starts, so it also bounds the wait /// for the response headers, and a unary call the server holds longer than this fails. A server stream that may go @@ -46,11 +53,28 @@ final class NativeHttpConfig { /// that `platformCode`. /// - **The connection:** if nothing has arrived from the call's origin, on any call, since the call last received /// anything, the connection is taken to be dead. The client moves new calls to a fresh `URLSession` or Cronet - /// engine, so they dial a new connection, and lets the calls already on the old one finish there. A call that hits - /// its deadline is checked the same way: the server is sent a deadline a step shorter, so on a live connection its - /// `deadline_exceeded` arrives first. + /// engine, so they dial a new connection, and lets the calls already on the old one finish there. final Duration? readIdleTimeout; + /// How much shorter a deadline the server gets than the call's own. connect-dart sends a call's deadline in + /// `connect-timeout-ms`; with this set, the client sends that much less, or half the deadline when that is less, so + /// the server never gets zero. `null` or [Duration.zero] sends the deadline as it is. + /// + /// A reachable server then times the call out first, and its `deadline_exceeded` arrives before the client's own + /// deadline fires. So a deadline that does fire on the client means the server's answer never came, which is what + /// makes [evictOnDeadline] reliable. Servers that call further services should take their own step off the deadline + /// they pass on. + final Duration? deadlineStep; + + /// Whether a call that hits its deadline on the client moves new calls off its connection, as a [readIdleTimeout] + /// does: if nothing has arrived from the call's origin, on any call, since the call last received anything, new + /// calls go to a fresh `URLSession` or Cronet engine. Calls already on the old connection are not canceled. + /// + /// Use it with [deadlineStep]. Without one, the server enforces the same deadline as the client, but its clock + /// starts later, so on a healthy connection the client's deadline usually fires before the server's + /// `deadline_exceeded` arrives, and the connection is evicted for nothing. + final bool evictOnDeadline; + /// Called with each response's headers, before its body is read. An exception it throws goes to /// `FlutterError.onError`, and the call goes on. final void Function(NativeHttpResponseInfo info)? onResponse; diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 77745f3..319198d 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -248,30 +248,24 @@ void main() { await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.canceled))); }); - test("a deadline cancels the stream and asks the plugin to leave the call's connection", () async { - final signal = TimeoutSignal(const Duration(milliseconds: 10)); - final response = client.httpClient(request(StreamController().stream, signal: signal)); - await settle(); - final id = plugin.lastCallId; - await Future.delayed(const Duration(milliseconds: 50)); - final cancel = plugin.callsTo('cancel').single.arguments as Map; - expect(cancel['callId'], id); - expect(cancel['evictConnection'], isTrue); - await plugin.emit({'callId': id, 'type': 'canceled'}); - await expectLater(response, throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded))); - }); - - test('a signal that is not a deadline leaves the connection alone', () async { - final signal = CancelableSignal(); - unawaited(client.httpClient(request(null, signal: signal)).then((_) {}, onError: (_) {})); - await settle(); - signal.cancel('stop'); - await settle(); - expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('evictConnection'), isFalse); - }); + group('deadlines', () { + Future> cancelAfterDeadline(NativeHttpClient client) async { + final signal = TimeoutSignal(const Duration(milliseconds: 10)); + final response = client.httpClient(request(StreamController().stream, signal: signal)); + await settle(); + final id = plugin.lastCallId; + await Future.delayed(const Duration(milliseconds: 50)); + final cancel = plugin.callsTo('cancel').single.arguments as Map; + expect(cancel['callId'], id); + await plugin.emit({'callId': id, 'type': 'canceled'}); + await expectLater( + response, + throwsA(isA().having((e) => e.code, 'code', Code.deadlineExceeded)), + ); + return cancel; + } - test("the server's deadline is a step shorter than the client's", () async { - Future sent(String timeout) async { + Future sentTimeout(NativeHttpClient client, String timeout) async { final headers = Headers() ..add('content-type', 'application/proto') ..add('connect-timeout-ms', timeout); @@ -282,11 +276,44 @@ void main() { return pairs.cast>().firstWhere((pair) => pair[0] == 'connect-timeout-ms')[1] as String?; } - expect(await sent('5000'), '4000'); - // Under two steps, half: the server never gets zero. - expect(await sent('1500'), '750'); - expect(await sent('1'), '1'); - expect(await sent('soon'), 'soon'); + test('by default a deadline cancels the stream and leaves its connection alone', () async { + expect((await cancelAfterDeadline(client)).containsKey('evictConnection'), isFalse); + }); + + test("with evictOnDeadline, a deadline asks the plugin to leave the call's connection", () async { + final evicting = await CronetHttpClient.create(const NativeHttpConfig(evictOnDeadline: true)); + expect((await cancelAfterDeadline(evicting))['evictConnection'], isTrue); + }); + + test('a signal that is not a deadline leaves the connection alone', () async { + final evicting = await CronetHttpClient.create(const NativeHttpConfig(evictOnDeadline: true)); + final signal = CancelableSignal(); + unawaited(evicting.httpClient(request(null, signal: signal)).then((_) {}, onError: (_) {})); + await settle(); + signal.cancel('stop'); + await settle(); + expect((plugin.callsTo('cancel').single.arguments as Map).containsKey('evictConnection'), isFalse); + }); + + test('by default the server gets the deadline as it is', () async { + expect(await sentTimeout(client, '5000'), '5000'); + }); + + test("with deadlineStep, the server's deadline is a step shorter than the client's", () async { + final stepping = await CronetHttpClient.create(const NativeHttpConfig(deadlineStep: Duration(seconds: 1))); + expect(await sentTimeout(stepping, '5000'), '4000'); + // Under two steps, half: the server never gets zero. + expect(await sentTimeout(stepping, '1500'), '750'); + expect(await sentTimeout(stepping, '1'), '1'); + expect(await sentTimeout(stepping, 'soon'), 'soon'); + }); + + test('a negative step is refused', () async { + await expectLater( + CronetHttpClient.create(const NativeHttpConfig(deadlineStep: Duration(seconds: -1))), + throwsArgumentError, + ); + }); }); test('cancelling the response body cancels the native stream', () async { From 3c9f0cecb777679c5702b9ed1219ef65ee638cc8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 10 Nov 2026 01:46:49 +0000 Subject: [PATCH 7/7] Review fixes: keep connections a deadline call was still hearing on - With evictOnDeadline, a call that received anything since the server's deadline (the head start deadlineStep gave the server) no longer evicts its connection: the server was still answering, whatever it made of the deadline. - Android: a URL java.net.URI rejects no longer fails the call; it just stands for its own origin in the eviction guard. - iOS builds the origin string once; stale comments that tied engine retirement to read-idle timeouts only, or mentioned a per-request read-idle timeout, are updated. - New tests: the head-start window (Dart unit), and a read-idle timeout on one stream that leaves the connection to a stream still receiving (integration, HTTP/2, checked by X-Remote-Addr). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BqmpXX1R55toGUCQrbJY4i --- .../connect_native_http/CronetBridge.kt | 30 ++++++++------ example/integration_test/eliza_test.dart | 28 +++++++++++++ .../URLSessionBridge.swift | 7 ++-- lib/src/channel/channel_http_client.dart | 40 +++++++++++++------ test/cronet_http_client_test.dart | 35 ++++++++++++++++ 5 files changed, 113 insertions(+), 27 deletions(-) diff --git a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt index fde3dd1..8d60dc7 100644 --- a/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt +++ b/android/src/main/kotlin/com/agtonomy/connect_native_http/CronetBridge.kt @@ -11,6 +11,7 @@ import org.chromium.net.NetworkException import org.chromium.net.QuicOptions import org.chromium.net.UrlResponseInfo import java.net.URI +import java.net.URISyntaxException import java.nio.ByteBuffer import java.util.concurrent.Executors import java.util.concurrent.ScheduledExecutorService @@ -40,8 +41,9 @@ class CronetBridge( private class EngineOptions(val quicHints: List>, val quicIdleTimeoutSeconds: Int?) - // One CronetEngine and the calls on it. An engine whose call hit its read-idle timeout is retired: new calls go to - // a fresh one, with its own connections, and the retired one shuts down once its last call ends. + // One CronetEngine and the calls on it. An engine whose call ran out of time on a quiet connection (see evict) is + // retired: new calls go to a fresh one, with its own connections, and the retired one shuts down once its last call + // ends. private class Pool(val cronet: CronetEngine) { val callIds = HashSet() @@ -211,15 +213,7 @@ class CronetBridge( return } val pool = engine.current - val target = URI(url) - val port = if (target.port != -1) target.port else if (target.scheme == "http") 80 else 443 - val c = Call( - callId, - engineId, - pool, - "${target.host}:$port", - readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }, - ) + val c = Call(callId, engineId, pool, origin(url), readIdleTimeoutMs?.let { TimeUnit.MILLISECONDS.toNanos(it) }) val builder = pool.cronet.newBidirectionalStreamBuilder(url, Callback(c), executor).setHttpMethod(method) for ((name, value) in headers) { builder.addHeader(name, value) @@ -234,6 +228,18 @@ class CronetBridge( scheduleReadIdleCheck(c, c.readIdleTimeoutNanos) } + // "host:port" of [url], for telling which calls share connections. java.net.URI is stricter than Cronet, so a URL it + // rejects stands for itself rather than failing a call Cronet would carry. + private fun origin(url: String): String { + val target = try { + URI(url) + } catch (e: URISyntaxException) { + return url + } + val port = if (target.port != -1) target.port else if (target.scheme == "http") 80 else 443 + return "${target.host}:$port" + } + // Read-idle timeout. Rather than rescheduling on every chunk received, each check finds when the call last received // something and, unless the timeout has run out since, schedules the next check for when it would. private fun scheduleReadIdleCheck(c: Call, delayNanos: Long?) { @@ -293,7 +299,7 @@ class CronetBridge( Pool(build(engine.options)) } catch (e: Throwable) { // Keep the engine there is; new calls on a dead connection fail on their own timeouts. - Log.e(TAG, "engine ${c.engineId}: could not replace the CronetEngine after a read idle timeout", e) + Log.e(TAG, "engine ${c.engineId}: could not replace the CronetEngine for call ${c.callId}", e) return } engine.retired.add(engine.current) diff --git a/example/integration_test/eliza_test.dart b/example/integration_test/eliza_test.dart index 4ecffa1..aad77a7 100644 --- a/example/integration_test/eliza_test.dart +++ b/example/integration_test/eliza_test.dart @@ -463,6 +463,34 @@ void main() { expect(h2.remoteAddrs.last, liveAddr); }, timeout: http2CaseTimeout); + testCase('a read-idle timeout keeps a connection another call is still receiving on', (_) async { + // The quiet stream's timeout says nothing about the connection while the busy one keeps hearing from the server. + if (skipUnlessAdmin()) { + return; + } + const timeout = Duration(seconds: 1); + final h2 = await harnessOverHttp2(readIdleTimeout: timeout); + final (quietRequests, quiet) = await openQuietConverse(h2); + final liveAddr = h2.remoteAddrs.last; + final (busyRequests, busy) = await openQuietConverse(h2); + expect(h2.remoteAddrs.last, liveAddr); + + final quietFailure = nativeCause(quiet.moveNext()); + for (var i = 0; i < 6; i++) { + await Future.delayed(timeout * 0.3); + busyRequests.add(ConverseRequest(sentence: 'ping $i')); + expect(await busy.moveNext(), isTrue); + } + expect((await quietFailure).platformCode, readIdleTimeoutCode); + + await h2.eliza.say(SayRequest(sentence: 'after'), onHeader: h2.recordHeaders); + expect(h2.remoteAddrs.last, liveAddr); + busyRequests.add(ConverseRequest(sentence: 'bye')); + expect(await busy.moveNext(), isTrue); + await busyRequests.close(); + await quietRequests.close(); + }, timeout: http2CaseTimeout); + testCase('a read-idle timeout notices a dead HTTP/2 connection, and the next call dials a new one', (_) async { if (skipUnlessAdmin()) { return; diff --git a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift index 1b2bfdf..d5fa580 100644 --- a/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift +++ b/ios/connect_native_http/Sources/ConnectNativeHttpCore/URLSessionBridge.swift @@ -180,7 +180,7 @@ public final class URLSessionBridge { private func makeURLSession() -> URLSession { let configuration = URLSessionConfiguration.ephemeral // URLSession fails a task that receives nothing for this long. Its default of 60 s would end a quiet stream or a - // long-running unary call; callers bound calls with deadlines, or set their own read-idle timeout per request. + // long-running unary call; callers bound calls with deadlines, or set the client's read-idle timeout. configuration.timeoutIntervalForRequest = Self.sessionIdleTimeout // Only a bridge with its own roots implements the trust challenge. With the system's roots there is nothing for // the delegate to decide, so certificate checks are left entirely to the network stack. @@ -213,7 +213,8 @@ public final class URLSessionBridge { request.addValue(h[1], forHTTPHeaderField: h[0]) } let port = target.port ?? (target.scheme == "http" ? 80 : 443) - request.assumesHTTP3Capable = session.http3Origins.contains("\(target.host ?? ""):\(port)") + let origin = "\(target.host ?? ""):\(port)" + request.assumesHTTP3Capable = session.http3Origins.contains(origin) // The request's read-idle timeout: URLSession fails the task once it has received nothing for this long, with // NSURLErrorTimedOut. if let interval = Self.requestTimeoutInterval(readIdleTimeoutMs: readIdleTimeoutMs) { @@ -223,7 +224,7 @@ public final class URLSessionBridge { let c = Call(callId: callId) c.engineId = engineId c.urlSession = session.urlSession - c.origin = "\(target.host ?? ""):\(port)" + c.origin = origin if hasBody { var input: InputStream? var output: OutputStream? diff --git a/lib/src/channel/channel_http_client.dart b/lib/src/channel/channel_http_client.dart index 9cef5c9..ec6386a 100644 --- a/lib/src/channel/channel_http_client.dart +++ b/lib/src/channel/channel_http_client.dart @@ -61,6 +61,9 @@ final class _Call { /// Set when the call's signal fired, so the outcome reports the signal's error rather than the cancellation. Object? abortError; + /// Time since the call last received anything from the server, or started. + final sinceReceived = Stopwatch()..start(); + void fail(Object error) { finished = true; if (!response.isCompleted) { @@ -177,6 +180,9 @@ final class _Dispatcher { } void _onCallEvent(int id, _Call call, Map e) { + if (const {'headers', 'data', 'trailers'}.contains(e['type'])) { + call.sinceReceived.reset(); + } switch (e['type']) { case 'headers': final headers = bridge.responseHeaders([ @@ -246,12 +252,6 @@ int serverTimeoutMs(int ms, Duration step) => ms - math.min(step.inMilliseconds, /// connect-dart's deadline header, which it sets from the call's signal. const _connectTimeoutHeader = 'connect-timeout-ms'; -/// A `connect-timeout-ms` value with [serverTimeoutMs] applied; anything but a positive number passes unchanged. -String _stepDown(String value, Duration step) => switch (int.tryParse(value)) { - final ms? when ms > 0 => '${serverTimeoutMs(ms, step)}', - _ => value, -}; - /// A connect-dart [HttpClient] on one engine of a [PlatformBridge] plugin. base class ChannelHttpClient implements NativeHttpClient { ChannelHttpClient(this._bridge, this._engineId, this._config); @@ -278,6 +278,9 @@ base class ChannelHttpClient implements NativeHttpClient { final int _engineId; final NativeHttpConfig _config; + /// [NativeHttpConfig.deadlineStep], or `null` for none. + late final Duration? _deadlineStep = _config.deadlineStep == Duration.zero ? null : _config.deadlineStep; + /// [NativeHttpConfig.readIdleTimeout] in whole milliseconds, rounded up so a sub-millisecond timeout isn't none. late final int? _readIdleTimeoutMs = switch (_config.readIdleTimeout) { null || Duration.zero => null, @@ -290,8 +293,21 @@ base class ChannelHttpClient implements NativeHttpClient { Future _send(HttpRequest req) async { final dispatcher = _Dispatcher.of(_bridge); final methods = _bridge.methods; - final step = _config.deadlineStep == Duration.zero ? null : _config.deadlineStep; + final step = _deadlineStep; final call = _Call(Uri.parse(req.url)); + // How much earlier than the client the server times the call out. + var serverLead = Duration.zero; + final headers = >[]; + for (final (:name, :value) in req.header.entries) { + final ms = step != null && name == _connectTimeoutHeader ? int.tryParse(value) : null; + if (ms != null && ms > 0) { + final serverMs = serverTimeoutMs(ms, step!); + serverLead = Duration(milliseconds: ms - serverMs); + headers.add([name, '$serverMs']); + } else { + headers.add([name, value]); + } + } final id = dispatcher.register(call, _config.onResponse); call.body.onCancel = () { if (!call.finished) { @@ -303,7 +319,10 @@ base class ChannelHttpClient implements NativeHttpClient { return; } call.abortError = error; - final evict = _config.evictOnDeadline && error.code == Code.deadlineExceeded; + // A call that received something since the server's deadline was still hearing from the server, whatever the + // server made of the deadline, so its connection is alive. + final evict = + _config.evictOnDeadline && error.code == Code.deadlineExceeded && call.sinceReceived.elapsed >= serverLead; unawaited(dispatcher.cancel(id, evictConnection: evict)); }); try { @@ -312,10 +331,7 @@ base class ChannelHttpClient implements NativeHttpClient { 'callId': id, 'url': req.url, 'method': req.method, - 'headers': [ - for (final (:name, :value) in req.header.entries) - [name, name == _connectTimeoutHeader && step != null ? _stepDown(value, step) : value], - ], + 'headers': headers, 'hasBody': req.body != null, 'readIdleTimeoutMs': _readIdleTimeoutMs, }); diff --git a/test/cronet_http_client_test.dart b/test/cronet_http_client_test.dart index 319198d..efd0539 100644 --- a/test/cronet_http_client_test.dart +++ b/test/cronet_http_client_test.dart @@ -285,6 +285,41 @@ void main() { expect((await cancelAfterDeadline(evicting))['evictConnection'], isTrue); }); + test('a call still receiving within the server\'s head start does not evict', () async { + final both = await CronetHttpClient.create( + const NativeHttpConfig(deadlineStep: Duration(seconds: 1), evictOnDeadline: true), + ); + Future> cancelArgs({required bool receiveLate}) async { + plugin.calls.clear(); + // 400 ms: the server gets 200 ms, half, so its head start is 200 ms. + final headers = Headers() + ..add('content-type', 'application/proto') + ..add('connect-timeout-ms', '400'); + final req = HttpRequest( + 'https://example.test/svc/Method', + 'POST', + headers, + StreamController().stream, + TimeoutSignal(const Duration(milliseconds: 400)), + ); + final response = both.httpClient(req); + await settle(); + final id = plugin.lastCallId; + if (receiveLate) { + await Future.delayed(const Duration(milliseconds: 330)); + await plugin.headers(id); + } + await Future.delayed(const Duration(milliseconds: 500)); + final cancel = plugin.callsTo('cancel').single.arguments as Map; + await plugin.emit({'callId': id, 'type': 'canceled'}); + await response.then((r) => r.body.drain()).catchError((_) {}); + return cancel; + } + + expect((await cancelArgs(receiveLate: false))['evictConnection'], isTrue); + expect((await cancelArgs(receiveLate: true)).containsKey('evictConnection'), isFalse); + }); + test('a signal that is not a deadline leaves the connection alone', () async { final evicting = await CronetHttpClient.create(const NativeHttpConfig(evictOnDeadline: true)); final signal = CancelableSignal();