fix(auth): survive a slow gcloud and a missing ssh-copy-id (0.22.1)

[johnyaku]

Three bugs found on a single dt auth setup run in a containerised module load dt environment, plus two follow-ups. Each of the three made a working setup look broken, and two sent the user off to fix something that was never wrong.

1. A slow gcloud was reported as "not authenticated"

gcloud auth list had a 10 s budget. gcloud is a large Python application and on NCI it lives on /g/data, so a cold start pays NFS latency for every module it imports:

cold	20.4 s
warm	5.3 s

Blowing the budget returned the same bare None as "nobody is logged in", so the run advised gcloud auth login to someone who had just done exactly that and whose account was active.

Budget raised to 30 s, and the tri-state None replaced with gcloud_auth_status() returning missing / timeout / error / unauthenticated / ok. Only the last failure now suggests re-authenticating. check_gcloud_authenticated() stays as the yes/no wrapper for callers that don't show the answer to a user.

2. ssh-copy-id was assumed to exist

It ships in openssh-clients as a separate file from ssh itself, so minimal container images routinely have one and not the other — which is what the containerised dt build hits.

The call was unguarded, unlike _deploy_key_forge directly above it which checks shutil.which. So FileNotFoundError escaped the whole SSH phase, and the caller's broad except Exception discarded every result — including hosts that had already succeeded. The failing run had just registered a key with GitHub and reported nothing about it.

Now guarded, with a fallback that appends to the remote authorized_keys over plain ssh. Three details that matter:

• The key travels as $0, never interpolated into the script body.
• Both script and key are shlex.quote'd — ssh joins its arguments and the remote shell parses them a second time, so one layer of quoting isn't enough.
• stdin is left alone, since the key isn't deployed yet and ssh may need to prompt for a password.

3. The GCP quota project was never set

secrets.gcp.project reached the secret resource path but never the credentials. Those are two different things called "the project", and nothing derived one from the other — so gcloud auth application-default login users got a google-auth UserWarning about "quota exceeded"/"API not enabled" on every call. Setting it once in dt beats every user running gcloud auth application-default set-quota-project on every machine they touch.

4. TestFetchUrlImport made a real network call

Both tests patched subprocess.run, but _fetch_url_import consults doctor.check_network_connectivity first for any http/https/s3/gs source — a socket probe, which a subprocess patch cannot touch. So they passed on a login node and failed on a compute node.

Every full-suite run through qxub reported 2 failures unrelated to whatever was being changed, which is the fastest way to teach people to ignore a red suite. Verified by forcing the ambient probe to return False: the tests' own mock wins, so they no longer depend on where they run.

5. The key-deployment loop existed twice

Byte-identical copies in ssh.ssh_setup and setup._do_ssh_setup. They had already drifted — one carried a verbose "could not deploy" message the other lacked — and fix #2 above had to be applied to both by hand, which is the argument for not leaving it that way.

Extracted to ssh._deploy_keys_and_report. setup.py loses 66 lines and no longer imports the deploy helpers at all; a test asserts they stay gone, since their return would mean the duplication returned with them. The existing suite already patched dt.auth.ssh._deploy_key_* — where the shared function resolves them — so those tests now cover both entry points rather than one.

Also

The "N host(s) need key deployment" list printed before the refinement pass that adds more hosts, so it announced a subset and then deployed to hosts it had never named — which reads as the connectivity check being wrong.

Tests

2138 passed, 2 skipped, 0 failed under PBS via qxub — the first fully green compute-node run.

35 new tests in tests/unit/test_auth_robustness.py, including the authorized_keys script executed for real against a throwaway HOME: it round-trips a key containing ", $, and backticks through both shell layers, is idempotent, sets 600/700, and preserves pre-existing entries.

Note

_DEFAULT_KEY_PATH is an unused import in setup.py. It was already unused on main, so it's left alone rather than widening this diff.

🤖 Generated with Claude Code