Add support for exiting validators through the Keymanager API

[zah]

The Keymanager API is a standardized REST API for managing validator keys, supported by all Ethereum consensus clients.

A recent addition to the API introduced support for obtaining signed voluntary exit messages:
ethereum/keymanager-APIs#58

The ejector could potentially use the new endpoint as a replacement for the custom VALIDATOR_EXIT_WEBHOOK mechanism, leading to a very simple integration setup for all operators.

[kolyasapphire]

Hey @zah!

Thanks for a feature request and sorry for a late response - just got back from vacation.

Started an internal discussion, will get back once we have a decision on this.

[isidorosp]

I think there's roughly four options here:

1. make it an option but heavily advise against it, and explain why (see below)
2. refuse to implement it because of security considerations
3. implement some configurable "cooldown" / rate-limiting / "manual input required" functionality into the ejector to catch potential cases of potentially dangerous execution
4. approach this in a different way (@avsetsin suggested a script wrapping keymanager API calls to generate the messages and store them alongside the ejector, instead of just directly calling the API).

What are the security considerations? If someone the ejector is accessed by an intruder (but e.g. the validator client itself is not), or there is some sort of serious bug in the ejector, then all validators associated with the validator client(s) that the ejector talks to could be immediately exited. The default behavior of the ejector mitigates total possible impact through required pre-signed messages (which should always be reasonably limited). Webhook implementations likewise may have additional workflow mechanics (e.g. queuing the commands for approval) or "checks" (such as rate-limiting, cooldown mechanics, requiring manual input over certain thresholds etc).

Would be good to get node operator input here since they're the intended audience :)

[PacoBits]

This implementation could offer significant benefits for integrations, although it does present certain security risks. Perhaps option 3, "Implement some configurable 'cooldown' / rate-limiting / 'manual input required'" might be the most interesting in my opinion.

I would add an execution mode enviroment variable to the validator ejector, by default in safe mode, limited to X requests per hour/minute. However, I would also provide the possibility to run it in dangerous mode (without limits), accompanied by a strong warning and acknowledgment of the associated risks.

[kolyasapphire]

@eddort and I agreed that implementing it in the Ejector itself will require touching its critical code, which we ideally would rather not do.

We could write a script to fetch keys from KAPI and generate and sign exit messages for them using the Keymanager API. However, Ejector will have to be restarted manually to pick up new messages (#55) and I feel that it's not the level of automation NOs are looking for.

We can discuss creating a separate service to be used alongside Ejector in webhook mode. With detailed security notices and built-in limits, of course.

[paulhauner]

In our setup, we run multiple VCs without a central key manager. I suspect that may be the case for other operators, too.

In such a scenario, if Ejector were to use the keymanager API, the Ejector would need to know a mapping of validator pubkey to keymanager API URL (e.g., the IP address of the VC). This is probably too complex for the Ejector to implement.

Seeing as the Ejector doesn't support the keymanager API and even if it did, it wouldn't support our setup, we produced a single, idempotent Ansible script that will ensure that the set of exit messages in the Ejector dir exactly matches the KAPI output.

The script works along these lines:

1. Generate a file which maps validator pubkeys to VC hosts (i.e., SSH to each host and use the keymanager API to list keys, append those (pubkey,hostname)'s to a file on the local machine).
2. Query KAPI for a list of exits to prepare.
3. Look at the list of exits already in the Ejector directory.
4. Delete any exits that exist in the Ejector dir that were not in the KAPI response.
5. For each exit that was in the KAPI response, but does not exist in the Ejector dir:
   1. For each pubkey, SSH to the host in the map file created in (1).
   2. Use the keymanager API to generate an exit.
6. Encrypt the newly created exits.
7. Put the new exits in the Ejector dir.
8. Restart the Ejector.

This means we've effectively implemented option (4) in @isidorosp's comment. I find it creates a nice balance where it's easy to run but it's not so automated that a bug/exploit in the Ejector could exit our entire validator set.

[zah]

I think all suggested concerns and approaches are quite reasonable, but there is still room for a reusable solution that all operators could deploy more easily. @paulhauner's approach could be implemented as a command-line tool that's invoked as a cron job and configured with a list of nodes offering the Keymanager API. The "human in the loop approach" could be implemented as a web application, backed by a simple database.