Support key-based ssh for droplets

[ehmatthes]

The initial POC work used a username/password combination for ssh to the droplet, because that's what I was most familiar with. That's not conducive to testing, because the password is sent by email when creating a new droplet through the CLI.

• Support key-based droplets in regular workflow.
• Document
• Merge
• Release
• Announce

[ehmatthes]

Working notes

• Add a CLI arg for key-based droplet creation.
• If available, use this for all server commands.
• Manual deployment test, starting with a droplet created by a doctl command. This should be the basis for an e2e test.

[ehmatthes]

CLI arg

• Add a CLI arg, --ssh-key, path to key for ssh into droplet.
  • Note: this is the first plugin to extend the core CLI.
  • Implement in deploy.py first.
  • Then move to separate cli.py file.
• Write validation for plugin CLI args.
• Update the plugin generator to include code for extending CLI.

[ehmatthes]

Access droplet

• Create a droplet using ssh keys in DO console.
• Parse the CLI arg to get path to ssh keys.
• Use the key to carry out deployment work.

[ehmatthes]

Resuming work 9/22

• Refactor plugin structure to be consistent with current plugin generator structure.
  • hooks
  • cli.py
  • plugin_config.py
  • platform_deployer
  • Verify with --help output, with currently does not show --ssh-key option.
  • Write a help test.
  • Copy the cli arg test for now.
  • Also --ssh-key is not recognized. After updating, is it recognized?
  • grep dsd-vps-demo, the empty plugin I used as a reference for current plugin structure.

[ehmatthes]

Goal: rewrite "Automated deployment" to work for SSH keys

Update the current Automated deployment section in the main README to use SSHkeys.

• Create droplet, using ssh keys (See notes in Write e2e test #15.)
• Get ip address of droplet.
• Update README as I go
• Update platform_deployer to use ssh key if --ssh-key passed

Notes

See offline notes, with some identifying info, for more specifics.

• Grab ID of droplet from creation call.
• Check status before grabbing IP address. If "new", it won't have an IP address.
• In _prep_automate_all(), create the droplet and get all info for SSH work.
• In other methods, use key as appropriate. Probably need a helper function to get correct call based on whether key is available, or un/pw.
• In platform_deployer._validate_platform(), check that --platform is provided if --automate-all is used. (I believe this happens earlier than _validate_cli().)
• prep automate all doesn't need to repeat validation that's done in cli.
• Validation should check that doctl is available if using automate-all and platform is do.
• Think through validation of automate all. If no existing resources and automate all, we need a --platform arg.
• Log full json output, but don't dump it all to console.

[ehmatthes]

Open questions

• In _prep_automate_all(), we need to make the resources. This is platform-specific! What does that mean for the generality of this plugin? Probably not too much, as there's very little else that should be platform-specific. But we will need a --platform flag.
• There are two automate-all modes. One where we need to create the instance. Another where we have a machine to point to, but we want to automate the rest of the push. Sort out these workflows, and how to accommodate them.

[ehmatthes]

Parking lot

• Which args in droplet creation should be exposed on CLI?
• Probably want a subgroup of options for each platform supported.
• An important appeal of this plugin is the ability to deploy more than one project to the same instance. With a PAAS, every new project multiplies costs, even for demo projects. Write an e2e test that deploys twice to the same instance, and make this point in PR.
• Document the plugin_config attributes that are set during a deployment run. For example, plugin_config.droplet_id. This would be useful in logs.
• Is there any DO output that should be suppressed from logs?
• This definitely needs its own documentation site.
• Should text=True be an option, or even the default, for run_quick_command()? It feels like I shouldn't have to use run_quick_command().stdout.decode() every time I want to grab the output.
• Document most significant plugin_utils better.
• Much refactoring can be done here, walrus, for-else when looping over ip_addresses.
• E2e test probably needs an env var for the key ID for creating droplets.
• Build support for multiple SSH key ids in utils.get_ssh_key_ids_digitalocean().
• Why is server_username on dsd_config and not on plugin_config?
• "root" -> dsd_config.server_username
• Set an appropriate password for the non-root user.
  • Prompt for password, preferably early (gather all info needed early, so we don't have random interruptions after long work starts).
  • Write a general dsd_utils.get_user_input() function.
• Make an isolated DO namespace for my testing work?
• Fix permissions on django_user setup?
• Set pw for django_user, so can use sudo when logged in.
• Move git config earlier, so actions that require interaction or confirmation all happen early on. Or, find way to make that fingerprint non-interactive.

[ehmatthes]

Warning

• Don't merge this until both un/pw and ssh key approaches can be used. Resolve the conflict around automate-all being overloaded currently.

[ehmatthes]

Using key-based commands

This should be a matter of updating the paramiko code to use the key instead of un/pw when building a client.

• Do I need to specify the key when creating the droplet? (Yes.)

(Pdb) client.connect(hostname=plugin_config.ip_address, username="root", key_filename="</path/to/key>")
*** paramiko.ssh_exception.SSHException: Invalid key

• Pass a key ID, or generate one.
  • Get list of keys.
  • If one, ask if okay to use that one.
  • If none, create one.
  • If existing, ask user which to use.
  • Also give user option to generate a new key.

[ehmatthes]

_connect_server()

• Update this to work appropriately with a key-based server.
• Why is there a try with no except in run_server_cmd_ssh()?
• Refactor all of this!

[ehmatthes]

Current progress

Gets all the way to:

...
Installing uv...
Running server command over SSH...
  command: curl -LsSf https://astral.sh/uv/install.sh | sh
    Trying to connect using ssh key...
no checksums to verify
installing to /root/.local/bin
  uv
  uvx
everything's installed!

To add $HOME/.local/bin to your PATH, either restart your shell or run:

    source $HOME/.local/bin/env (sh, bash, zsh)
    source $HOME/.local/bin/env.fish (fish)
downloading uv 0.8.21 x86_64-unknown-linux-gnu
Installing Python...
Running server command over SSH...
  command: /home/django_user/.local/bin/uv python install 3.12
    Trying to connect using ssh key...
bash: line 1: /home/django_user/.local/bin/uv: No such file or directory
Initializing Git project on server...
DSDCommandError: Git ssh keyfile already exists at /Users/eric/.ssh/id_rsa_git.
...

• How does this key interact with ssh key? (Don't create key if we're already using one.)

Gets to:

Looking for gunicorn...
  Added gunicorn to requirements file.
  Modified file: /Users/eric/projects/dsd-dev-project_qvu5p/blog/settings.py

  Looking in /Users/eric/projects/dsd-dev-project_qvu5p for serve_project.sh...
    File serve_project.sh not found. Generating file...

    Wrote serve_project.sh to /Users/eric/projects/dsd-dev-project_qvu5p/serve_project.sh
Copying /var/folders/gk/y2n2jsfj23g864pdr38rv4ch0000gn/T/tmpu9b_8azb to server.
Traceback (most recent call last):
  File "/Users/eric/projects/dsd-dev-project_qvu5p/manage.py", line 22, in <module>
    main()
    ~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 416, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/Users/eric/projects/django-simple-deploy/django_simple_deploy/management/commands/deploy.py", line 164, in handle
    pm.hook.dsd_deploy()
    ~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/Users/eric/projects/dsd-vps/dsd_vps/deploy.py", line 36, in dsd_deploy
    platform_deployer.deploy()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/platform_deployer.py", line 57, in deploy
    self._add_caddyfile()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/platform_deployer.py", line 240, in _add_caddyfile
    do_utils.copy_to_server(path_local, path_remote)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/utils.py", line 144, in copy_to_server
    client.connect(
    ~~~~~~~~~~~~~~^
        hostname = os.environ.get("DSD_HOST_IPADDR"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        timeout = timeout
        ^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/client.py", line 407, in connect
    raise NoValidConnectionsError(errors)

• What needs to change in _copy_to_server()?

[ehmatthes]

After updating _copy_to_server()

Wrote serve_project.sh to /Users/eric/projects/dsd-dev-project_qvu5p/serve_project.sh
Copying /var/folders/gk/y2n2jsfj23g864pdr38rv4ch0000gn/T/tmpg9pflal6 to server.
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Invalid key
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Error reading SSH protocol banner
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Error reading SSH protocol banner[Errno 54] Connection reset by peer
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Error reading SSH protocol banner[Errno 54] Connection reset by peer
      Attempt failed, waiting 20s...
    Trying to connect using ssh key...
Error reading SSH protocol banner
      Attempt failed, waiting 20s...
Traceback (most recent call last):
  File "/Users/eric/projects/dsd-dev-project_qvu5p/manage.py", line 22, in <module>
    main()
    ~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 416, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/Users/eric/projects/django-simple-deploy/django_simple_deploy/management/commands/deploy.py", line 164, in handle
    pm.hook.dsd_deploy()
    ~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_hooks.py", line 512, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
    raise exception
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/pluggy/_callers.py", line 121, in _multicall
    res = hook_impl.function(*args)
  File "/Users/eric/projects/dsd-vps/dsd_vps/deploy.py", line 36, in dsd_deploy
    platform_deployer.deploy()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/platform_deployer.py", line 57, in deploy
    self._add_caddyfile()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/platform_deployer.py", line 240, in _add_caddyfile
    do_utils.copy_to_server(path_local, path_remote)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/eric/projects/dsd-vps/dsd_vps/utils.py", line 163, in copy_to_server
    sftp = client.open_sftp()
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/client.py", line 606, in open_sftp
    return self._transport.open_sftp_client()
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/transport.py", line 1210, in open_sftp_client
    return SFTPClient.from_transport(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/sftp_client.py", line 164, in from_transport
    chan = t.open_session(
        window_size=window_size, max_packet_size=max_packet_size
    )
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/transport.py", line 988, in open_session
    return self.open_channel(
           ~~~~~~~~~~~~~~~~~^
        "session",
        ^^^^^^^^^^
    ...<2 lines>...
        timeout=timeout,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/eric/projects/dsd-dev-project_qvu5p/.venv/lib/python3.13/site-packages/paramiko/transport.py", line 1082, in open_channel
    raise SSHException("SSH session not active")
paramiko.ssh_exception.SSHException: SSH session not active

• What changes does copy_to_server() need to work with ssh keys?

[ehmatthes]

Fix non-root user

• Failing at first call that uses non-root user.
  • See how far process gets using root. (All the way, but deployment fails because it depends on the non-root user.)
  • Fix issue with configuring non-root user.
• Update check_server_available() using non-root user. (Not an issue in this function.)