Vine: Evaluate HTCondor Shared Port Setup

[dthain]

Within a large HTCondor pool (such as the OSPool), it may not always be possible for the worker to make a simple TCP connection back to the manager, due to a variety of network policies, firewalls, address translation, etc.

HTCondor itself has a "shared port" mechanism by which all processes connect to the same open port, and their connections are proxied to the correct process on that host. It may be possible for TaskVine to use this mechanism to proxy connections from the workers to the manager.

Briefly, we would create a new module htcondor_shared_link that would use the protocol below to connect to the proxy, handshake, and then return a normal link for the worker to use. On the manager side, we would have to add a piece to also accept connections from a unix domain socket, and then handle those normally. The unknown piece to work out is the binary format of the handshake between the worker and the proxy.

@bbockelm provided this longer overview:

By default, a HTCondor install runs the "shared port" daemon that provides a mechanism to multiplex the one incoming open TCP port amongst the different daemons running on the hosts.  For common functionality (particularly, "condor_ssh_to_job"), this multiplexing mechanism is made open by default to unprivileged users.

Let's say are three actors here:
- The remote client (e.g., the Globus Compute worker)
- The condor_shared_port daemon (serves as the multiplexer)
- The Globus Compute manager, running on the same host as the condor_shared_port but without incoming TCP connectivity.

Here's the sequence of events:
- Remote client opens a TCP connection to the condor_shared_port daemon (typically, port 9618; I can give Python pseudo-code to determine this port).
- Remote client sends a header to the condor_shared_port.  The header is a tuple, consisting of a (shared port ID, client name, deadline, unused args) followed by an end-of-message marker.  The tuple contents are:
  - Shared port ID: a unique identifier for the Globus Compute manager on the host; can be randomly-generated.
  - Client name: unique identifier for the Globus Compute worker; mostly, used for log messages.
  - Deadline: Unix timestamp at which point the request should be ignored (requests may queue under load).
  - Unused args: Arbitrary string, currently unused.
  Given about 4 hours, I could write down the binary encoding for the tuple.
- The condor_shared_port daemon looks into a directory, defaulting to /var/lock/condor/daemon_sock, for a Unix domain socket named $(Shared port ID).
- condor_shared_port opens the socket and binds to create a socket.
  - It is assumed the Globus Compute Manager has the Unix domain socket open and is listening.
- The condor_shared_port sends a fixed header to the Globus Compute Manager and sends the FD to the Globus Compute Manager using the SCM_RIGHTS mechanism.
  - On failure, the TCP socket is unceremoniously closed by the condor_shared_port.

At this point, the Globus Compute Manager has a file descriptor that is a TCP socket attached to the remote client.  The typical next step is to have the server hand off the connected socket to the communication library you're using (ZeroMQ in your case?).

On the remote client side, you need to open the socket, send the special header, and then pass the socket to the comm library.

I think a simple Python-based demo would take 2 days?