Skip to content

[CHEF-32139] Fix knife vcenter vm clone hanging/failing during SSH bootstrap - #67

Open
ashiqueps wants to merge 4 commits into
mainfrom
vcenter-vm-clone-ssh-bootstrap-fix
Open

[CHEF-32139] Fix knife vcenter vm clone hanging/failing during SSH bootstrap#67
ashiqueps wants to merge 4 commits into
mainfrom
vcenter-vm-clone-ssh-bootstrap-fix

Conversation

@ashiqueps

Copy link
Copy Markdown
Contributor

Summary

Fixes knife vcenter vm clone hanging or failing forever with Errno::EADDRNOTAVAIL during the SSH-bootstrap wait phase, even though the cloned VM is genuinely reachable over SSH the entire time. Previously customers had to manually Ctrl-C the clone and re-run knife bootstrap against the already-cloned VM.

Root Cause

knife-cloud's SshBootstrapProtocol#wait_for_server_ready computes the SSH probe port as config[:connection_port] || config[:ssh_port] with no fallback to the standard port 22 (unlike its own ssh_gateway branch a few lines later, which correctly falls back to || 22). When a user doesn't pass --connection-port/--ssh-port on the CLI (the common case), both values are nil, so the code calls TCPSocket.new(hostname, nil), which deterministically raises Errno::EADDRNOTAVAIL on every single attempt, forever — even though the VM is reachable on port 22 the whole time.

Confirmed via live reproduction against a real vCenter lab environment: TCPSocket.new(ip, nil) raises the exact error observed in the failing bootstrap, while TCPSocket.new(ip, 22) against the same VM at the same moment succeeds immediately.

Changes Made

  • lib/chef/knife/cloud/ssh_bootstrap_protocol_patch.rb (new) — reopens SshBootstrapProtocol to:
    • Override wait_for_server_ready to add the missing || 22 default (the actual root-cause fix)
    • Rescue Errno::EADDRNOTAVAIL as a transient, retryable error in tcp_test_ssh (same bucket as ECONNREFUSED/ENETUNREACH/EHOSTUNREACH), for genuinely transient local networking errors
    • Add a capped exponential backoff with jitter between probe attempts, reducing connection churn against the target
    • Add throttled diagnostic status output (attempt count, elapsed time, last error) so a wait is visible and explainable instead of a silent wall of dots
  • lib/chef/knife/vcenter_vm_clone.rb — requires the patch via the lazy deps block
  • lib/support/clone_vm.rb — robustify VM lookup by folder path and bootstrap IP extraction (ignore link-local addresses, bound the wait with a timeout instead of looping forever)
  • lib/chef/knife/cloud/vcenter_service.rb, lib/chef/knife/cloud/vcenter_service_helpers.rb — supporting service changes needed for the clone/bootstrap flow

Testing

  • bundle exec rake spec — 30 examples, 0 failures
  • bundle exec rake style — 0 offenses
  • Live end-to-end validation against a real vCenter lab environment: clone → instant SSH-readiness detection (no hang) → SSH connect → chef-client install → first Chef Infra Client run, all completing successfully with zero manual intervention (exit code 0)

Test Scenarios Covered

  • Happy path: wait_for_server_ready defaults the probed port to 22 when neither --connection-port nor --ssh-port is configured
  • Explicit --connection-port/--ssh-port values still take precedence over the default
  • Regression test reproducing the exact TCPSocket.new(host, nil) failure mode this patch prevents
  • Transient error handling: EADDRNOTAVAIL, ECONNREFUSED, non-readable timeout, successful SSH banner path
  • Backoff growth and cap, diagnostic status throttling

@ashiqueps
ashiqueps requested review from a team as code owners July 24, 2026 08:44
@ashiqueps ashiqueps added Type: Bug Does not work as expected. Aspect: Integration Works correctly with other projects or systems. labels Jul 24, 2026
ashiqueps added a commit that referenced this pull request Jul 28, 2026
The 'regression' test asserted that TCPSocket.new('127.0.0.1', nil)
raises Errno::EADDRNOTAVAIL specifically. This errno is OS/kernel
dependent for a nil-port connect(2): it raised EADDRNOTAVAIL locally
(macOS) but Errno::ECONNREFUSED in the Buildkite Linux/Docker CI
environment, causing PR #67's verify pipeline to fail on both Ruby
3.1 and 3.4 jobs even though the actual bug fix (the SSH bootstrap
protocol patch itself) is correct and unaffected.

Both errnos descend from SystemCallError, and both are already
treated as transient/retryable by #tcp_test_ssh (covered by separate
examples immediately below this one). Assert on SystemCallError
instead of a single platform-specific errno so the test is portable
across environments while still verifying the same root cause it was
written to document.

Verified locally: bundle exec rake -> 30 examples, 0 failures, 0
Chefstyle offenses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
@ashiqueps
ashiqueps force-pushed the vcenter-vm-clone-ssh-bootstrap-fix branch from 6f493aa to ca33fc1 Compare July 28, 2026 10:31
ashiqueps and others added 2 commits July 28, 2026 16:11
Root cause: knife-cloud's SshBootstrapProtocol#wait_for_server_ready
computes the SSH probe port as
'config[:connection_port] || config[:ssh_port]' with no fallback to
the standard port 22 (unlike its own ssh_gateway branch, which
correctly falls back to '|| 22'). When a user does not pass
--connection-port/--ssh-port on the CLI (the common case), this
results in TCPSocket.new(hostname, nil), which deterministically
raises Errno::EADDRNOTAVAIL on every single connection attempt,
forever, even though the VM is genuinely reachable on port 22 the
whole time. This is why 'knife vcenter vm clone' would hang/fail
during bootstrap even though manual ssh/nc to the same host worked
instantly, and why users had to Ctrl-C and manually re-run
'knife bootstrap'.

Confirmed via live reproduction against a real vCenter lab
environment: TCPSocket.new(ip, nil) raises the exact error observed,
while TCPSocket.new(ip, 22) against the same VM at the same moment
succeeds immediately.

Changes:
- lib/chef/knife/cloud/ssh_bootstrap_protocol_patch.rb (new): reopens
  SshBootstrapProtocol to override wait_for_server_ready with the
  missing '|| 22' default, rescue Errno::EADDRNOTAVAIL as a transient
  error in tcp_test_ssh, add capped exponential backoff with jitter
  between probe attempts, and add throttled diagnostic status output
  (attempt count, elapsed time, last error) so a wait is visible and
  explainable instead of a silent wall of dots.
- lib/chef/knife/vcenter_vm_clone.rb: require the patch via the lazy
  deps block.
- lib/support/clone_vm.rb: robustify VM lookup by folder path and
  bootstrap IP extraction (ignore link-local addresses, bound the
  wait with a timeout instead of looping forever).
- lib/chef/knife/cloud/vcenter_service.rb,
  lib/chef/knife/cloud/vcenter_service_helpers.rb: supporting service
  changes needed for the clone/bootstrap flow.
- spec/unit/ssh_bootstrap_protocol_patch_spec.rb (new): covers the
  port-default fix, transient-error handling, backoff, and
  diagnostic throttling.
- spec/unit/clone_vm_spec.rb, spec/unit/vcenter_service_spec.rb,
  spec/unit/vcenter_vm_clone_spec.rb (new): coverage for the
  supporting changes above.

Validated end-to-end against a live vCenter lab environment: clone
-> instant SSH-readiness detection (no hang) -> SSH connect -> chef-
client install -> first Chef Infra Client run, all completing
successfully with zero manual intervention.

bundle exec rake: 30 examples, 0 failures, 0 Chefstyle offenses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
The 'regression' test asserted that TCPSocket.new('127.0.0.1', nil)
raises Errno::EADDRNOTAVAIL specifically. This errno is OS/kernel
dependent for a nil-port connect(2): it raised EADDRNOTAVAIL locally
(macOS) but Errno::ECONNREFUSED in the Buildkite Linux/Docker CI
environment, causing PR #67's verify pipeline to fail on both Ruby
3.1 and 3.4 jobs even though the actual bug fix (the SSH bootstrap
protocol patch itself) is correct and unaffected.

Both errnos descend from SystemCallError, and both are already
treated as transient/retryable by #tcp_test_ssh (covered by separate
examples immediately below this one). Assert on SystemCallError
instead of a single platform-specific errno so the test is portable
across environments while still verifying the same root cause it was
written to document.

Verified locally: bundle exec rake -> 30 examples, 0 failures, 0
Chefstyle offenses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
@ashiqueps
ashiqueps force-pushed the vcenter-vm-clone-ssh-bootstrap-fix branch from ca33fc1 to 7bdcd5c Compare July 28, 2026 10:42
ashiqueps and others added 2 commits July 29, 2026 10:44
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
@ashiqueps ashiqueps changed the title Fix knife vcenter vm clone hanging/failing during SSH bootstrap [CHEF-32139] Fix knife vcenter vm clone hanging/failing during SSH bootstrap Jul 29, 2026
@ashiqueps ashiqueps added the Expeditor: Bump Version Minor Used by github.minor_bump_labels to bump the Minor version number. label Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Aspect: Integration Works correctly with other projects or systems. Expeditor: Bump Version Minor Used by github.minor_bump_labels to bump the Minor version number. Type: Bug Does not work as expected.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants