Skip to content

Fix launch timeout being applied 1000x longer than configured#562

Open
sombraSoft wants to merge 1 commit into
jenkinsci:developfrom
sombraSoft:fix-launch-timeout-units
Open

Fix launch timeout being applied 1000x longer than configured#562
sombraSoft wants to merge 1 commit into
jenkinsci:developfrom
sombraSoft:fix-launch-timeout-units

Conversation

@sombraSoft

Copy link
Copy Markdown

A configured launch timeout of 300 seconds actually waits ~3.5 days. InstanceConfiguration.provision() passes getLaunchTimeoutMillis() (seconds × 1000) into the node builder, but ComputeEngineInstance.launchTimeout is documented and consumed as seconds — its own getLaunchTimeoutMillis() multiplies by 1000 again. Every wait that uses node.getLaunchTimeoutMillis() (insert-operation wait, startup-script wait, SSH wait) runs with a timeout 1000× larger than configured.

In production we found 16 Computer.threadPoolForRemoting threads wedged for a week inside ComputeClient.waitForOperationCompletion, polling long-expired GCE operations (404 on every poll), with their nodes stuck offline in "connecting" state and the backing VMs kept alive indefinitely by CleanLostNodesWork label refreshes:

Computer.threadPoolForRemoting [#64] TIMED_WAITING
  com.google.cloud.graphite.platforms.plugin.client.ComputeClient.waitForOperationCompletion(ComputeClient.java:669)
  com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:195)
  hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)

The fix passes launchTimeoutSeconds to the builder, matching the field's documented unit.

Resolves #560

Testing done

  • New unit test InstanceConfigurationTest#testProvisionedNodeLaunchTimeoutIsInSeconds provisions a node from a configuration with a 100-second launch timeout and asserts the node reports 100 s / 100 000 ms. It fails before the fix (expected:<100> but was:<100000>) and passes after.
  • Full InstanceConfigurationTest class passes (28/28).
  • Verified in a production Jenkins instance (public OpenROAD CI): with the corrected timeout the launcher gives up after the configured 300 s and terminateNode() cleans up, instead of wedging threads for days.

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate the feature works or the issue is fixed

InstanceConfiguration.provision() passed getLaunchTimeoutMillis() into
ComputeEngineInstance.launchTimeout, which is documented and consumed as
seconds; ComputeEngineInstance.getLaunchTimeoutMillis() then multiplied by
1000 again. A configured 300-second launch timeout therefore became
~3.5 days, leaving launch threads polling expired GCE operations for days
and keeping dead nodes (and their VMs) alive indefinitely.
@sombraSoft
sombraSoft requested a review from a team as a code owner July 8, 2026 04:47

@das7pad das7pad left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

(We also discovered this at work today and would love to see this merged.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Launch timeout is applied 1000x longer than configured (seconds passed as milliseconds, multiplied again)

2 participants