Skip to content

vmtime time-source task spins forever on a timeout half the ring past the start time #1069

Open

Description

@q5438722

Summary

TimerState::poll busy-loops forever, at 100% CPU and without ever yielding, when a VmTimeAccess timeout is circularly after the current VM time but circularly before the keeper's last start time. From then on no VM-time timeout fires, and stop() and save() on the keeper hang.

Repro

vmtime_half_ring_spin.txt

Add the attached vmtime_half_ring_spin.rs (.txt) to vm/vmcore/tests/ and run:

cargo test -p vmcore --test vmtime_half_ring_spin
  1. Create a keeper at VM time 0, build a source, and call start().
  2. Let at least 100ns of VM time pass.
  3. Call set_timeout_if_before(VmTime::from_100ns(1 << 63)) and poll once.
  4. Call keeper.stop().await.

The control test runs the same sequence with T = 1e9 and passes.

Expected

T is ~29,000 years ahead, so the timer sleeps and stop() completes.

Actual

thread 'timeout_half_ring_after_start_does_not_spin' panicked at vm/vmcore/tests/vmtime_half_ring_spin.rs:53:10:
vmtime source task did not yield within 10s (TimerState::poll spins): Timeout

The spinning thread used 9.99s of CPU in 10s of wall time:

pal_async::sys::epoll::...::set_deadline   support/pal/pal_async/src/unix/epoll.rs:517
vmcore::vmtime::TimerState::set_next       vm/vmcore/src/vmtime.rs:239
vmcore::vmtime::TimerState::wake           vm/vmcore/src/vmtime.rs:260
vmcore::vmtime::TimerState::poll           vm/vmcore/src/vmtime.rs:330
vmcore::vmtime::...::run::{async_fn}       vm/vmcore/src/vmtime.rs:724

Root cause

VmTime comparisons are circular and therefore not transitive, as the doc comment on is_before notes. The two halves of the timer logic use different reference points. With start = 0, now = d > 0 and T = 2^63:

  • wake() compares T with now. T - now = 2^63 - d > 0, so T is still pending and set_next(T) is called.
  • timestamp() measures T from start. T.checked_sub(start) returns None (2^63 is i64::MIN), so the deadline becomes the host time of the last start, which is in the past.
  • The host timer is ready immediately, wake() re-arms the same past deadline, and poll loops forever.

Any T in start + [2^63, 2^63 + d) triggers this, so the window grows with uptime.

Impact

  • The keeper task hangs, so no VM-time timeout in the VM fires again.
  • Possible guest path (a lead, not verified end to end): OpenHCL's VmTimeGuestTimer::timeout (openhcl/virt_mshv_vtl/src/processor/hardware_cvm/mod.rs:3354) derives the timeout from the guest's synthetic-timer deadline. A guest that programs a deadline ~29,000 years ahead may reach this; I haven't checked how that value is validated.
  • VmTimeAccess::host_time() also goes through timestamp(). The OpenHCL VP loop (openhcl/virt_mshv_vtl/src/processor/mod.rs:813) uses its result as a deadline and continues when that deadline is ready, so from reading the code it may busy-loop as well.
  • Restoring a saved state that has an armed timeout reaches the same code.

Suggested fix

Measure future times from now, the reference wake() uses. Clamp the distance to one day, like poll's default deadline, because a circular distance can exceed what the host clock can represent. Times that are not after now keep today's behavior. timestamp() takes now_os as a parameter so that the unit test can use synthetic host times.

fix.patch

Verification

  • The spin test hangs on 76edd46e2 and passes with the patch. The control test passes on both.
  • cargo test -p vmcore --lib (30 tests, including the new test_timestamp) passes, and so does the regression test for the "duration too large" panic.
  • rustfmt and clippy -D warnings are clean.
  • Replay of all 53 vmtime_v2_* timeout inputs:
build pass timeout other
76edd46e2 0 48 5 ("duration too large" panic)
76edd46e2 + since.min(86400s) 1 52 0
76edd46e2 + this patch 53 0 0

Alternative

Bound how far ahead set_timeout* may be set, for example by treating more than 2^62 units ahead as "never". That narrows the API rather than fixing the conversion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions