Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ found or verified with the tests in this package.
- **No client-side dead-connection detection.** Neither stack sends HTTP/2 pings the caller controls, and server
pings stop at any load balancer that terminates HTTP/2. Idle timeouts on the path must exceed a stream's longest
quiet period, or the stream must carry periodic messages. See "QUIC idle timeout vs HTTP/2 keepalive pings".
- **Register `grpc.health.v1.Health` on every server, if clients use `keepAlive`.** In connect-go:
`grpchealth.Register(server, grpchealth.NewStaticChecker())`. Route `/grpc.health.v1.Health/Check` through the
load balancer like the other procedures. See "Keepalive".

**The HTTP/3 endpoint**

Expand Down Expand Up @@ -140,6 +143,32 @@ This does nothing on the **HTTP/2 fallback** used where UDP is blocked. Neither
connection that dies without the interface going down goes unnoticed until a call's deadline or an
application-level heartbeat.

### Keepalive

`NativeHttpConfig.keepAlive` keeps a quiet connection carrying traffic, so nothing on the path closes it for being
idle: a load balancer's idle timeout (an AWS ALB's is 60 s by default), a NAT mapping, or the server's own. It is
opt-in, and the same on both platforms:

```dart
NativeHttpConfig(
http3Origins: {origin},
keepAlive: const NativeKeepAlive(interval: Duration(seconds: 15), timeout: Duration(seconds: 10)),
)
```

- While a client has calls open to an origin, it sends a `grpc.health.v1.Health/Check` request there every
`interval`: a Connect unary call with an empty `HealthCheckRequest`, over the same URLSession or Cronet engine. It
takes the same connection as the calls, which the integration tests check from the server's side.
- One request at a time per origin, starting one `interval` after the first call opens and stopping once none is
open. A request that hasn't ended after `timeout` is canceled.
- It only keeps the connection busy. It doesn't fail calls or notice a dead connection, and a request on another
stream isn't data on a call's own stream, so it doesn't satisfy a read timeout on that call. Any response counts,
since the request is for traffic, not the server's health. A response other than HTTP 200 is reported once per
origin through `FlutterError.onError`, since it usually means the server doesn't register the service.
- Unlike HTTP/2 PING frames, which a load balancer that ends HTTP/2 answers itself, these requests reach the server.
- Over HTTP/3 the server's QUIC pings already keep the path busy, so keepalive mostly matters on the HTTP/2 path.
- Keepalive requests don't reach `onResponse`.

### QUIC idle timeout vs HTTP/2 keepalive pings

The two mechanisms differ in who notices a dead path, which is why server settings protect the phone over HTTP/3 but
Expand Down Expand Up @@ -193,12 +222,13 @@ Neither platform finishes a stream on its own while the client still has its req
## Tests

- **`test/`:** host unit tests, runnable on Linux CI. They cover error classification, Connect end-stream detection,
URLSession response headers, and the shared channel adapter against a fake plugin.
URLSession response headers, and the shared channel adapter, keepalive included, against a fake plugin.
- **`test_server/`:** a Go server for `connectrpc.eliza.v1.ElizaService` over HTTP/2 and HTTP/3, scripted for tests
(see `eliza.go`).
- `-max-idle` and `-keepalive` set its QUIC idle timeout and ping interval.
- `-admin-addr` serves `POST /freeze?for=10s` and `POST /unfreeze`, which drop all its UDP traffic so a test can
simulate a dead HTTP/3 path.
simulate a dead HTTP/3 path, and `GET /health-checks`, the health checks it has received.
- It serves `grpc.health.v1.Health` for keepalive, and reports each request's client address in `X-Remote-Addr`.
- Its own tests (`go test ./...`) check it over HTTP/3, including the idle timeout, pings and freezing.
- It creates a CA in `test_server/certs/` on first start and reuses it, so a device only has to trust it once.
- With `QLOGDIR` set, it writes a qlog trace of each QUIC connection there, which shows the server's side of a
Expand All @@ -223,6 +253,8 @@ Neither platform finishes a stream on its own while the client still has its req
before any Alt-Svc, must reach HTTP/3 within a few fresh clients. A loaded simulator sometimes lets TCP win
URLSession's race even locally, so a single fallback is logged rather than failed.

It also checks that keepalives reach the server on the same connection as a quiet stream.

The script starts the server with a 3 s idle timeout and 1 s pings. Over HTTP/3 (iOS) the tests also check two
things: a quiet stream outlives the idle timeout, and a frozen path fails an open stream within it.
- **Native bridge tests:** the same script runs them after the Dart tests, against the same server. Each platform's
Expand Down
65 changes: 61 additions & 4 deletions example/integration_test/eliza_test.dart
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Runs connect_native_http's platform client against ../test_server on a device or simulator. Start it through
// ../run_integration_tests.sh, which starts the server, trusts its CA on the device and passes these defines.
import 'dart:async';
import 'dart:convert';
import 'dart:io' as io;

import 'package:connect_native_http/connect_native_http.dart';
Expand All @@ -26,7 +27,8 @@ const unreachableUrl = String.fromEnvironment('UNREACHABLE_URL');
/// The test server under a name its certificate does not cover.
const untrustedUrl = String.fromEnvironment('UNTRUSTED_URL');

/// The test server's plain-HTTP admin endpoint, which can freeze its HTTP/3 path (test_server/freeze.go).
/// The test server's plain-HTTP admin endpoint, which can freeze its HTTP/3 path and lists the health checks it
/// received (test_server/freeze.go).
const adminUrl = String.fromEnvironment('ADMIN_URL');

/// The QUIC idle timeout the test server advertises (its -max-idle).
Expand All @@ -38,10 +40,18 @@ const introduceInterval = Duration(milliseconds: 100);
final class Harness {
Harness._(this.native, this.eliza);

static Future<Harness> create(String baseUrl, {Set<Uri> http3Origins = const {}}) async {
static Future<Harness> create(
String baseUrl, {
Set<Uri> http3Origins = const {},
NativeKeepAlive? keepAlive,
}) async {
late final Harness harness;
final native = await NativeHttpClient.create(
NativeHttpConfig(http3Origins: http3Origins, onResponse: (info) => harness.platformProtocols.add(info.protocol)),
NativeHttpConfig(
http3Origins: http3Origins,
keepAlive: keepAlive,
onResponse: (info) => harness.platformProtocols.add(info.protocol),
),
);
if (native == null) {
throw StateError('no native client on this platform');
Expand All @@ -63,7 +73,13 @@ final class Harness {
/// `X-Proto` of each response: the protocol the server saw.
final serverProtocols = <String>[];

void recordHeaders(Headers headers) => serverProtocols.add(headers.get('x-proto')?.join(',') ?? '');
/// `X-Remote-Addr` of each response: the client's address as the server saw it, which identifies the connection.
final remoteAddrs = <String>[];

void recordHeaders(Headers headers) {
serverProtocols.add(headers.get('x-proto')?.join(',') ?? '');
remoteAddrs.add(headers.get('x-remote-addr')?.join(',') ?? '');
}
}

/// Every case finishes in a few seconds against a local server (most well under one); a hang should fail fast.
Expand All @@ -81,6 +97,19 @@ Future<void> admin(String path) async {
}
}

/// Every health check the test server has received, oldest first: each one's `proto` and `remoteAddr`.
Future<List<Map<String, Object?>>> healthChecks() async {
final client = io.HttpClient();
try {
final response = await (await client.getUrl(Uri.parse('$adminUrl/health-checks'))).close();
expect(response.statusCode, 200, reason: 'GET $adminUrl/health-checks');
final body = await response.transform(utf8.decoder).join();
return (jsonDecode(body) as List).cast<Map<String, Object?>>();
} finally {
client.close();
}
}

/// Opens Converse and reads the server's greeting. Nothing flows afterwards until the caller sends.
Future<(StreamController<ConverseRequest>, StreamIterator<ConverseResponse>)> openQuietConverse(Harness harness) async {
final requests = StreamController<ConverseRequest>();
Expand Down Expand Up @@ -245,6 +274,34 @@ void main() {
await requests.close();
});

testCase('keepalives reach the server over the connection of the quiet call they keep busy', (_) async {
if (adminUrl.isEmpty) {
markTestSkipped('needs ADMIN_URL');
return;
}
const interval = Duration(milliseconds: 500);
final kept = await Harness.create(
serverUrl,
http3Origins: {Uri.parse(serverUrl)},
keepAlive: const NativeKeepAlive(interval: interval, timeout: Duration(seconds: 2)),
);
addTearDown(kept.native.close);
final (requests, responses) = await openQuietConverse(kept);
final connection = kept.remoteAddrs.last;
expect(connection, isNotEmpty, reason: 'the server reports X-Remote-Addr');

await Future<void>.delayed(interval * 3 + const Duration(milliseconds: 300));
final onConnection = (await healthChecks()).where((check) => check['remoteAddr'] == connection).toList();
// One per interval while the stream is open, after the first interval.
expect(onConnection.length, greaterThanOrEqualTo(2), reason: 'health checks from $connection: $onConnection');
expect({for (final check in onConnection) check['proto']}, {kept.serverProtocols.last});

requests.add(ConverseRequest(sentence: 'still there'));
expect(await responses.moveNext(), isTrue);
expect(responses.current.sentence, 'echo: still there');
await requests.close();
});

testCase('a refused connection is reported as never sent', (_) async {
expect(unreachableUrl, isNotEmpty, reason: 'run with --dart-define=UNREACHABLE_URL=...');
final unreachable = await Harness.create(unreachableUrl);
Expand Down
2 changes: 1 addition & 1 deletion lib/connect_native_http.dart
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
library;

export 'src/native_http_client.dart' show NativeHttpClient;
export 'src/native_http_config.dart' show NativeHttpConfig, NativeHttpResponseInfo;
export 'src/native_http_config.dart' show NativeHttpConfig, NativeHttpResponseInfo, NativeKeepAlive;
export 'src/native_http_exception.dart' show NativeHttpException, NativeHttpFailure;
82 changes: 73 additions & 9 deletions lib/src/channel/channel_http_client.dart
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import '../native_http_client.dart';
import '../native_http_config.dart';
import '../native_http_exception.dart';
import 'connect_end_stream.dart';
import 'keep_alive.dart';

/// A platform plugin that speaks this file's channel protocol: `createEngine`, `shutdownEngine`, `open`, `write`,
/// `end` and `cancel` method calls, and `headers`, `data`, `trailers`, `responseComplete`, `done`, `failed` and
Expand Down Expand Up @@ -54,14 +55,29 @@ final class _Call {
final response = Completer<HttpResponse>();
final body = StreamController<Uint8List>();
final trailer = Headers();
bool finished = false;
ConnectEndStreamDetector? endStream;

/// Set when the call's signal fired, so the outcome reports the signal's error rather than the cancellation.
Object? abortError;

/// Called once, when the call finishes.
void Function()? onFinished;

bool get finished => _finished;
bool _finished = false;

void markFinished() {
if (_finished) {
return;
}
_finished = true;
final callback = onFinished;
onFinished = null;
callback?.call();
}

void fail(Object error) {
finished = true;
markFinished();
if (!response.isCompleted) {
response.completeError(error);
}
Expand All @@ -76,7 +92,7 @@ final class _Call {

/// The server sent the whole response, so nothing more is sent.
void completeResponse(String bridgeName) {
finished = true;
markFinished();
if (!response.isCompleted) {
response.completeError(ConnectException(Code.internal, '$bridgeName: response ended without headers'));
}
Expand Down Expand Up @@ -233,7 +249,12 @@ final class _Dispatcher {

/// A connect-dart [HttpClient] on one engine of a [PlatformBridge] plugin.
base class ChannelHttpClient implements NativeHttpClient {
ChannelHttpClient(this._bridge, this._engineId, this._config);
ChannelHttpClient(this._bridge, this._engineId, this._config) {
final keepAlive = _config.keepAlive;
if (keepAlive != null) {
_keepAlive = KeepAliveMonitor(keepAlive, ping: _ping);
}
}

/// Starts the bridge's event routing and creates an engine with [arguments], returning its id.
static Future<int> createEngine(PlatformBridge bridge, Map<String, Object?> arguments) async {
Expand All @@ -244,15 +265,55 @@ base class ChannelHttpClient implements NativeHttpClient {
final PlatformBridge _bridge;
final int _engineId;
final NativeHttpConfig _config;
KeepAliveMonitor? _keepAlive;

/// Origins whose keepalive got a response other than HTTP 200, which is reported only the first time.
final _keepAliveRefused = <Uri>{};

@override
HttpClient get httpClient => _send;
HttpClient get httpClient =>
(req) => _send(req, keepAlive: true);

/// One keepalive request to [origin]. It isn't itself tracked, so it doesn't keep its own loop going, and it ends
/// quietly whatever happens: noticing a dead connection is left to the calls' own deadlines and timeouts.
Future<void> _ping(Uri origin) async {
final keepAlive = _config.keepAlive!;
final url = origin.replace(path: healthCheckPath).toString();
final signal = CancelableSignal();
final timer = Timer(keepAlive.timeout, () => signal.cancel('${_bridge.name}: keepalive to $origin timed out'));
try {
final response = await _send(
HttpRequest(url, 'POST', healthCheckHeaders(keepAlive.timeout), null, signal),
keepAlive: false,
);
await response.body.drain<void>();
if (response.status != 200 && _keepAliveRefused.add(origin)) {
reportError(
StateError(
'${_bridge.name}: keepalive: $url answered HTTP ${response.status}. Does the server register '
'grpc.health.v1.Health? The request still carried traffic.',
),
StackTrace.current,
'while sending a keepalive to $origin',
);
}
} catch (_) {
// The request failed or timed out. Calls on a dead connection fail on their own.
} finally {
timer.cancel();
}
}

Future<HttpResponse> _send(HttpRequest req) async {
/// Sends [req]. With [keepAlive], the call counts as open for [NativeHttpConfig.keepAlive] until it finishes, and
/// [NativeHttpConfig.onResponse] sees its response.
Future<HttpResponse> _send(HttpRequest req, {required bool keepAlive}) async {
final dispatcher = _Dispatcher.of(_bridge);
final methods = _bridge.methods;
final call = _Call(Uri.parse(req.url));
final id = dispatcher.register(call, _config.onResponse);
final id = dispatcher.register(call, keepAlive ? _config.onResponse : null);
if (keepAlive) {
call.onFinished = _keepAlive?.track(call.url);
}
call.body.onCancel = () {
if (!call.finished) {
unawaited(dispatcher.cancel(id));
Expand All @@ -278,7 +339,7 @@ base class ChannelHttpClient implements NativeHttpClient {
} catch (error) {
// The plugin refused the call, or isn't there (a background isolate, say): nothing was sent.
dispatcher.remove(id);
call.finished = true;
call.markFinished();
final cause = NativeHttpException(NativeHttpFailure.notSent, '${_bridge.name}: open failed: $error');
throw ConnectException(Code.unavailable, cause.message, cause: cause);
}
Expand Down Expand Up @@ -313,5 +374,8 @@ base class ChannelHttpClient implements NativeHttpClient {
}

@override
Future<void> close() => _bridge.methods.invokeMethod<void>('shutdownEngine', {'engineId': _engineId});
Future<void> close() {
_keepAlive?.close();
return _bridge.methods.invokeMethod<void>('shutdownEngine', {'engineId': _engineId});
}
}
Loading
Loading