Support TCP Fast Open

[zonyitoo]

TcpListener

All of Linux, OS X and Windows are the same, call setsockopt with TCP_FASTOPEN option

// For Linux, value is the queue length of pending packets
int opt = 5;
// For the others, just a boolean value for enable and disable
int opt = 1;

// Call it before listen()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

TcpStream

This is a bit more complicated than TcpListener, because we have to send SYN with the first data packet to make TFO works. APIs are completely different on different platforms:

Linux

Before 4.11, Linux uses MSG_FASTOPEN flag for sendto(), doesn't need to call connect()

// Send SYN with data in buf
ssize_t n = sendto(socket, buf, buf_length, MSG_FASTOPEN, saddr, saddr_len);

After 4.11, Linux provides a TCP_FASTOPEN_CONNECT option

int enable = 1;

// Enable it before connect()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN_CONNECT, &enable, sizeof(enable));

// Call connect as usual
int ret = connect(socket, saddr, saddr_len);

BSDs (except Darwin)

Uses sendto() as Linux without MSG_FASTOPEN flag

// Set before sendto()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

// Call sendto() without flags
ssize_t n = sendto(socket, buf, buf_length, 0, saddr, saddr_len);

Darwin (OS X)

Darwin supports TFO after OS X 10.11, iOS 9.0, tvOS 9.0 and watchOS 2.0.

It supports with a new syscall connectx.

sa_endpoints_t endpoints;
memset(&endpoints, 0, sizeof(endpoints));
endpoints.sae_dstaddr = &saddr;
endpoints.sae_dstaddrlen = saddr_len;

int ret = connectx(socket,
                   &endpoints,
                   SAE_ASSOCID_ANY,
                   CONNECT_DATA_IDEMPOTENT | CONNECT_RESUME_ON_READ_WRITE,
                   NULL,
                   0,
                   NULL,
                   NULL);

SYN will be sent with the first write or send.

Windows

Windows supports with ConnectEx, since Windows 10. https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nc-mswsock-lpfn_connectex

// TCP_FASTOPEN is required
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

// Get LPFN_CONNECTEX
LPFN_CONNECTEX ConnectEx = ...;

// Call it with the first packet of data, SYN will send with this packet's data
LPDWORD bytesSent = 0;
int ret = ConnectEx(socket, saddr, saddr_len, buf, buf_length, &bytesSent, &overlapped);

Challenge

As we can see for those OSes' APIs, (except OS X and Linux > 4.10) connections are made with the first write(), which means that if we want to support TFO, we have to put those code in the first call of std::io::Write::write or tokio::AsyncWrite::poll_write.

There is no other way except customizing a TcpStream from socket() and call different APIs on different OSes while sending the first data packet.

I want to open a discussion here for how in Rust's world to support TFO gracefully.

Related API PRs

1. connectx for OS X: Add TCP FastOpen support for macOS libc#1635
2. TCP_FASTOPEN_CONNECT for Linux: Add TCP_FASTOPEN_CONNECT for Linux after 4.11 libc#1634
3. TCP_FASTOPEN for Windows: Missing TCP_FASTOPEN retep998/winapi-rs#856

[alexcrichton]

I don't know much about this myself, but PRs are most welcome to explore the design space here!

[Thomasdezeeuw]

@zonyitoo still interested in contributing this?

[zonyitoo]

I made a few steps but couldn't find a good design for it.

[Thomasdezeeuw]

We could just expose the system calls/flags if we can't get a nice cross-platform API.

[zonyitoo]

https://github.com/zonyitoo/tokio-tfo

Made a simple repo for this.

[Thomasdezeeuw]

@zonyitoo do you still think this should go into Socket2? Or is your library a better place to keep it?

[zonyitoo]

Well, setting the TCP_FASTOPEN flag could be in socket2, but the whole TFO process (client making connection, server setting up acceptor socket) may have to be put in tokio-tfo.

I am ok to make a PR for setting TCP_FASTOPEN for sockets.

[Thomasdezeeuw]

Well, setting the TCP_FASTOPEN flag could be in socket2, but the whole TFO process (client making connection, server setting up acceptor socket) may have to be put in tokio-tfo.

I am ok to make a PR for setting TCP_FASTOPEN for sockets.

Sounds good.

[zonyitoo]

https://github.com/zonyitoo/tokio-tfo/blob/3e23d00c4258884db73cba49753fa37f5f9fa7ca/src/sys/unix/linux.rs#L388-L419

There is a problem about how to design an unified API. Linux's TCP_FASTOPEN accepts an integer value reprecenting the length of TFO accept queue. On the other platforms, including Windows, macOS, FreeBSD, it is only a switch with 0 and 1.

[Thomasdezeeuw]

We could document the behaviour, i.e. 0 disables, >= 1 enables, on Linux it changes the queue length.