Repository navigation
Measure whether the ready hook can exceed five minutes #629
Description
Tracker #605 carries this measurement follow-up from #598. A PT5M ready-hook failure reason may be a platform cap or a per-request timeout. Accepting a configured value above 300 seconds with a fast ready hook does not distinguish them.
Moved past 1.1.0 because resolving the platform limit requires a dedicated delayed-readiness experiment, separate from the release conformance coverage.
Acceptance: build an image whose ready hook deliberately takes more than 300 seconds with readyTimeoutInSeconds above that duration. Record the service outcome and duration, region, API version, and logs. Replace the current platform observation if the result contradicts it, and retain a named conformance check. Clean up the image, VM and service-created logs independently.
Activity
bonk-ai commentedon Oct 15, 2026
readyTimeoutInSecondsfromhooks_block(crates/microvms-app/src/control/image.rs:224-228), which writes oneBuildHookTimeoutto bothreadyandvalidate(crates/microvms-app/src/control/mod.rs:1055-1076).BuildHookTimeout::try_newaccepts 1 to 3600 (crates/microvms-domain/src/hooks.rs:82-97; ceiling atcrates/microvms-domain/src/constants.rs:312-313, service model2025-09-09at:54), andbuild --build-hook-timeout-secparses through it (crates/microvms-cli/src/cli.rs:1251-1256,:2716-2719). The client sets no bound at 300.docs/PLATFORM.md:381-393listsReady hook invocation timed out after PT5Mamong observed buildstateReasons and says that record lacks the configuredreadyTimeoutInSeconds, so it does not establish a five-minute platform limit.GetMicrovmImageVersionechoes the hooks (docs/PLATFORM.md:606-607), so the configured value can be read back beside the outcome, anddocs/PLATFORM.md:596-600records whatsnapshotBuildholds after a ready-hook timeout./readywith 200 straight away (crates/agentd/src/routes.rs:323-331) and has no workload handler for it (crates/agentd/src/hook_handlers.rs:46-47), so the delay has to come from the image.measurement_dockerfilealready replaces the stanza'sCMDwith a probe (conformance/lanes/platform_measurements.py:42-56)../conformance/run_rs.py --only platform_measurements --measurement-variant <name>(conformance/run_rs.py:118-136). Itsisolated_imagebuild passes no--build-hook-timeout-sectoday (conformance/lanes/isolated_image.py:49-65), andensure_image_cleanupdeletes the run's VMs, images, artifact objects and service-created log groups by prefix, reading each back (conformance/lanes/ensure_image.py:221-274).laithalsaadoon commentedon Oct 18, 2026
The second full main live run, 39088836602, did not resolve this measurement. On October 18, 2026 in
us-east-1, API2025-09-09, the request and service echo both specified a 600-second ready timeout. The image reachedCREATE_FAILED(ERR_PLATFORM) after about 711.79 seconds. The report recorded 60 begin events and 30 end events, with completed delays from 310.0002 to 310.1001 seconds, but classified no conclusive service outcome.The completed delay is logged before the fixture forwards to the daemon, so it does not establish successful readiness. The report reduced the underlying service diagnostic to
other build failure; it establishes neither a successful hook beyond five minutes nor a five-minute service cap. The named conformance check correctly failed, and #629 remains open. The current platform entry is unchanged.The lane's cleanup assertions passed, but the overall live suite failed. The release tracker records the separate suite failure and pending image deletion. Further live verification is stopped under the current full-run limit; a maintainer decision is required before another attempt.
laithalsaadoon commentedon Oct 18, 2026
Deferred past 1.2.0.
Full live run 39088836602 on main didn't settle this. The image reached
CREATE_FAILEDwithERR_PLATFORMafter about 712 seconds, with a requested and echoed ready timeout of 600 seconds, and the lane classified no conclusive outcome. A release shouldn't wait on an answer the service hasn't given yet, so #714 keeps the measurement in the live suite and records an inconclusive outcome as a skip with its full report. A conclusive outcome still passes, and the lane's cleanup checks still gate the run.This issue stays open, and
docs/PLATFORM.mdkeeps its current entry until a run produces a conclusive result.test(conformance): fail a #629 measurement whose build never reached …
fix(conformance): repair the four live-suite failures from run 390888…
laithalsaadoon commentedon Oct 18, 2026
The v1.2.0 tag's live run (run 39113559270) measured this again and recorded it as the suite's one expected skip: region us-east-1, API 2025-09-09, requested timeout 600, echoed timeouts
[600], stateCREATE_FAILED, errorERR_PLATFORM, outcome inconclusive, 710 seconds of wall time, 60 begin records and 30 end records. The two service reasons were outside the lane's known categories (other build failure). This matches run 39088836602 and the local run of #714, so the question stays open past 1.2.0: the lane still can't tell a five-minute cap from an unrelated build failure.