[CHEF-32139] Fix knife vcenter vm clone hanging/failing during SSH bootstrap - #67
Open
ashiqueps wants to merge 4 commits into
Open
[CHEF-32139] Fix knife vcenter vm clone hanging/failing during SSH bootstrap#67ashiqueps wants to merge 4 commits into
ashiqueps wants to merge 4 commits into
Conversation
ashiqueps
added a commit
that referenced
this pull request
Jul 28, 2026
The 'regression' test asserted that TCPSocket.new('127.0.0.1', nil)
raises Errno::EADDRNOTAVAIL specifically. This errno is OS/kernel
dependent for a nil-port connect(2): it raised EADDRNOTAVAIL locally
(macOS) but Errno::ECONNREFUSED in the Buildkite Linux/Docker CI
environment, causing PR #67's verify pipeline to fail on both Ruby
3.1 and 3.4 jobs even though the actual bug fix (the SSH bootstrap
protocol patch itself) is correct and unaffected.
Both errnos descend from SystemCallError, and both are already
treated as transient/retryable by #tcp_test_ssh (covered by separate
examples immediately below this one). Assert on SystemCallError
instead of a single platform-specific errno so the test is portable
across environments while still verifying the same root cause it was
written to document.
Verified locally: bundle exec rake -> 30 examples, 0 failures, 0
Chefstyle offenses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
ashiqueps
force-pushed
the
vcenter-vm-clone-ssh-bootstrap-fix
branch
from
July 28, 2026 10:31
6f493aa to
ca33fc1
Compare
Root cause: knife-cloud's SshBootstrapProtocol#wait_for_server_ready computes the SSH probe port as 'config[:connection_port] || config[:ssh_port]' with no fallback to the standard port 22 (unlike its own ssh_gateway branch, which correctly falls back to '|| 22'). When a user does not pass --connection-port/--ssh-port on the CLI (the common case), this results in TCPSocket.new(hostname, nil), which deterministically raises Errno::EADDRNOTAVAIL on every single connection attempt, forever, even though the VM is genuinely reachable on port 22 the whole time. This is why 'knife vcenter vm clone' would hang/fail during bootstrap even though manual ssh/nc to the same host worked instantly, and why users had to Ctrl-C and manually re-run 'knife bootstrap'. Confirmed via live reproduction against a real vCenter lab environment: TCPSocket.new(ip, nil) raises the exact error observed, while TCPSocket.new(ip, 22) against the same VM at the same moment succeeds immediately. Changes: - lib/chef/knife/cloud/ssh_bootstrap_protocol_patch.rb (new): reopens SshBootstrapProtocol to override wait_for_server_ready with the missing '|| 22' default, rescue Errno::EADDRNOTAVAIL as a transient error in tcp_test_ssh, add capped exponential backoff with jitter between probe attempts, and add throttled diagnostic status output (attempt count, elapsed time, last error) so a wait is visible and explainable instead of a silent wall of dots. - lib/chef/knife/vcenter_vm_clone.rb: require the patch via the lazy deps block. - lib/support/clone_vm.rb: robustify VM lookup by folder path and bootstrap IP extraction (ignore link-local addresses, bound the wait with a timeout instead of looping forever). - lib/chef/knife/cloud/vcenter_service.rb, lib/chef/knife/cloud/vcenter_service_helpers.rb: supporting service changes needed for the clone/bootstrap flow. - spec/unit/ssh_bootstrap_protocol_patch_spec.rb (new): covers the port-default fix, transient-error handling, backoff, and diagnostic throttling. - spec/unit/clone_vm_spec.rb, spec/unit/vcenter_service_spec.rb, spec/unit/vcenter_vm_clone_spec.rb (new): coverage for the supporting changes above. Validated end-to-end against a live vCenter lab environment: clone -> instant SSH-readiness detection (no hang) -> SSH connect -> chef- client install -> first Chef Infra Client run, all completing successfully with zero manual intervention. bundle exec rake: 30 examples, 0 failures, 0 Chefstyle offenses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
The 'regression' test asserted that TCPSocket.new('127.0.0.1', nil)
raises Errno::EADDRNOTAVAIL specifically. This errno is OS/kernel
dependent for a nil-port connect(2): it raised EADDRNOTAVAIL locally
(macOS) but Errno::ECONNREFUSED in the Buildkite Linux/Docker CI
environment, causing PR #67's verify pipeline to fail on both Ruby
3.1 and 3.4 jobs even though the actual bug fix (the SSH bootstrap
protocol patch itself) is correct and unaffected.
Both errnos descend from SystemCallError, and both are already
treated as transient/retryable by #tcp_test_ssh (covered by separate
examples immediately below this one). Assert on SystemCallError
instead of a single platform-specific errno so the test is portable
across environments while still verifying the same root cause it was
written to document.
Verified locally: bundle exec rake -> 30 examples, 0 failures, 0
Chefstyle offenses.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
ashiqueps
force-pushed
the
vcenter-vm-clone-ssh-bootstrap-fix
branch
from
July 28, 2026 10:42
ca33fc1 to
7bdcd5c
Compare
Signed-off-by: Ashique Saidalavi <ashique.saidalavi@progress.com>
nikhil2611
approved these changes
Jul 29, 2026
sanjain-progress
approved these changes
Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes
knife vcenter vm clonehanging or failing forever withErrno::EADDRNOTAVAILduring the SSH-bootstrap wait phase, even though the cloned VM is genuinely reachable over SSH the entire time. Previously customers had to manually Ctrl-C the clone and re-runknife bootstrapagainst the already-cloned VM.Root Cause
knife-cloud'sSshBootstrapProtocol#wait_for_server_readycomputes the SSH probe port asconfig[:connection_port] || config[:ssh_port]with no fallback to the standard port 22 (unlike its ownssh_gatewaybranch a few lines later, which correctly falls back to|| 22). When a user doesn't pass--connection-port/--ssh-porton the CLI (the common case), both values arenil, so the code callsTCPSocket.new(hostname, nil), which deterministically raisesErrno::EADDRNOTAVAILon every single attempt, forever — even though the VM is reachable on port 22 the whole time.Confirmed via live reproduction against a real vCenter lab environment:
TCPSocket.new(ip, nil)raises the exact error observed in the failing bootstrap, whileTCPSocket.new(ip, 22)against the same VM at the same moment succeeds immediately.Changes Made
lib/chef/knife/cloud/ssh_bootstrap_protocol_patch.rb(new) — reopensSshBootstrapProtocolto:wait_for_server_readyto add the missing|| 22default (the actual root-cause fix)Errno::EADDRNOTAVAILas a transient, retryable error intcp_test_ssh(same bucket asECONNREFUSED/ENETUNREACH/EHOSTUNREACH), for genuinely transient local networking errorslib/chef/knife/vcenter_vm_clone.rb— requires the patch via the lazydepsblocklib/support/clone_vm.rb— robustify VM lookup by folder path and bootstrap IP extraction (ignore link-local addresses, bound the wait with a timeout instead of looping forever)lib/chef/knife/cloud/vcenter_service.rb,lib/chef/knife/cloud/vcenter_service_helpers.rb— supporting service changes needed for the clone/bootstrap flowTesting
bundle exec rake spec— 30 examples, 0 failuresbundle exec rake style— 0 offensesTest Scenarios Covered
wait_for_server_readydefaults the probed port to 22 when neither--connection-portnor--ssh-portis configured--connection-port/--ssh-portvalues still take precedence over the defaultTCPSocket.new(host, nil)failure mode this patch preventsEADDRNOTAVAIL,ECONNREFUSED, non-readable timeout, successful SSH banner path