Add curl_connect(), curl_send(), curl_recv(): raw socket access via CURLOPT_CONNECT_ONLY

[allanvc]

libcurl can establish a connection (TCP, TLS handshake, certificate
verification, proxies) without speaking the protocol, through
CURLOPT_CONNECT_ONLY, and then hand the socket to the application via
curl_easy_send() / curl_easy_recv(). The package exposes the option
(connect_only), but not the two functions, and a handle that went through
curl_fetch_memory() cannot use them anyway: the fetch functions perform on
the shared multi handle and remove the easy handle afterwards
(src/interrupt.c), which detaches the connection. So this PR adds the
missing entry point as well:

h <- curl_connect("imaps://imap.example.com")   # TCP + TLS only, no protocol
curl_send(h, "A1 CAPABILITY\r\n")               # character or raw; returns bytes sent
curl_recv(h, timeout = 5000)                    # raw vector; empty on timeout,
                                                # attr "closed" when the peer closed

curl_connect() sets CURLOPT_CONNECT_ONLY and calls curl_easy_perform()
directly on the easy handle (the connect is quick and honors
connecttimeout); any handle option (timeouts, TLS, proxy) can be passed
through its handle argument. curl_send() loops over curl_easy_send()
handling CURLE_AGAIN by waiting on CURLINFO_ACTIVESOCKET with select();
curl_recv() waits up to timeout ms for readability and returns what one
curl_easy_recv() yields.

The concrete use case is IMAP in mRpostman: IDLE (RFC 2177) parks a
connection while the server pushes notifications, and commands carrying
literals (MULTIAPPEND, RFC 3502; REPLACE, RFC 8508) need a +
continuation dialogue; neither fits libcurl's request/response model, but
both are simple once the (TLS) socket is available. The same applies to any
line-based protocol libcurl can connect to (SMTP, POP3, NNTP, or plain TCP
via telnet://). mRpostman >= 2.0.0 ships this as its own compiled code
(src/imap_socket.c); with these functions in curl, that code goes away
and the package is back to pure R.

Notes:

• CURLOPT_CONNECT_ONLY exists since libcurl 7.15.2 and
  curl_easy_send/curl_easy_recv since 7.18.2, so there is no new build
  requirement.
• With CONNECT_ONLY, libcurl consumes the initial exchange of pingpong
  protocols (the IMAP greeting and capability banner, for instance) while
  connecting; the application should not expect to read it (documented in
  the man page).
• A connect-only handle cannot be reused for regular transfers (libcurl
  documents it as one-shot); after the peer closes, curl_recv() returns an
  empty raw vector with attr(, "closed") = TRUE.
• Tested on Linux against imaps:// (TLS) and imap:// (plain) servers: login,
  literal upload with continuation handling, timeout, and peer-close
  detection. tests/testthat/test-socket.R runs a round trip against
  imap.gmail.com (skipped offline/on CRAN).

Validation performed with the patch applied (Linux, libcurl 8.14.1, R 4.4.1): the full package test suite passes (no new failures or skips), and the round trip in tests/testthat/test-socket.R was exercised over TLS (imaps://) and against a plain imap:// server, including a literal upload with + continuation handling, the timeout path, and peer-close detection. roxygen2::roxygenize() regenerates NAMESPACE and man/curl_socket.Rd cleanly.

[jeroen]

What would you need this for in the context of R?

[allanvc]

Hi Jeroen, thanks for taking a look.

The use case is mRpostman, an IMAP client that runs on libcurl's native IMAP support. Most of the protocol maps fine to one request per command, but a few things don't fit that model. IDLE (RFC 2177) keeps the connection open while the server pushes notifications. MULTIAPPEND and CATENATE send multiple literals and the server paces the upload with continuation prompts. COMPRESS=DEFLATE switches the whole session to a deflate stream.

Right now mRpostman ships a small C layer that does exactly what this PR does: open the connection with CURLOPT_CONNECT_ONLY and then use curl_easy_send()/curl_easy_recv(). libcurl still does the TCP connect, the TLS handshake and the certificate verification, and the package talks the protocol on the socket it gets back. If curl exposed curl_connect()/curl_send()/curl_recv(), I could drop that compiled code entirely and the package would be pure R again.

It's not IMAP specific either. Any package that needs to speak a stateful text protocol (NNTP, custom TCP/TLS services, etc.) could use this instead of embedding its own socket code, and it gets libcurl's TLS, proxy and CA handling for free. This is basically the use case libcurl documents for connect-only mode, the PR just exposes it to R.

Happy to adjust the API or trim the surface if you'd prefer something smaller.

[jeroen]

Could you try if the approach from #448 works for you, which uses the curl::curl() connection interface? I think that may be a more natural match for what you are trying to do.

[allanvc]

Hi Jeroen, thanks for #449. I tried it with mRpostman and it works for everything the package does over the raw socket (IDLE, literals with continuations, MULTIAPPEND, CATENATE, BINARY, NOTIFY, COMPRESS=DEFLATE). I replaced the compiled socket code with a few lines of R over curl(url, "r+b") opened with blocking = FALSE and the full test suite passed unchanged. Also checked over TLS against Gmail with OAuth, and a 5 MB literal each way.

Two things I noticed:

1. Ctrl-C during a blocking read gives Failure in select() while waiting for data instead of a clean interrupt, because select() returns -1 with errno == EINTR and that is treated as a failure. The connection stays usable afterwards, so it's only the message. Treating EINTR as a timeout lets the existing pending_interrupt() check take over. In wait_on_socket():

   -  return select((int) sockfd + 1, &infd, &outfd, &errfd, &tv);
   +  int ret = select((int) sockfd + 1, &infd, &outfd, &errfd, &tv);
   +  if(ret < 0 && errno == EINTR)
   +    return 0; /* interrupted by a signal: let the caller check pending_interrupt() */
   +  return ret;

   plus #include <errno.h>. With that, Ctrl-C in a blocking readLines() on an idling IMAP connection is caught by tryCatch(..., interrupt = ) and the connection keeps working.

2. Nice side effect: use_ssl = 3 on an imap://host:143 URL does STARTTLS during the connect phase, so the socket you get back is already TLS. Might be worth a line in the docs.

Happy to close #448 when #449 is in.