Repository navigation
Description
On a 9a kernel without CPU idle states, which is what the published kernel is for zumapro, a sleep longer than three minutes ends with systemd killing its own services, and after the one that took systemd-logind the lock screen was unusable until a reboot. Found on 2026-11-14 during the call tests for #556, on a test kernel (7.3.0-rc6-omamobile-push1-gc50fd8fdf42a, the published tip plus kernel #247, linux-omarchy-mobile#232 and its review patches).
What happened
Sleeps with the modem up, ended by a call, a text or the RTC. From one boot's journal:
05:44:41 PM: suspend entry (s2idle)
05:47:16 PM: suspend exit (2 min 35 s: nothing)
05:49:03 PM: suspend entry (s2idle)
05:54:03 systemd-homed.service: Watchdog timeout (limit 3min)!
05:54:03 systemd-coredumpd.service: Watchdog timeout (limit 3min)!
05:54:03 PM: suspend exit (5 min 0 s)
05:58:18 PM: suspend entry (s2idle)
06:01:59 systemd-journald.service: Watchdog timeout (limit 3min)!
06:01:59 PM: suspend exit (3 min 41 s)
06:04:12 PM: suspend entry (s2idle)
06:09:49 systemd-logind.service: Watchdog timeout (limit 3min)!
06:09:49 systemd-journald.service: Watchdog timeout (limit 3min)!
06:09:50 PM: suspend exit (5 min 38 s)
06:09:50 systemd-logind.service: Main process exited, code=dumped, status=6/ABRT
Every sleep of 3 min 41 s or more in that boot cost at least one service (four of four); none of the shorter ones did. After logind went, sddm's helper failed in a loop (Failed to take control of "/dev/tty1", VirtualTerminalAlreadyTaken, 5,962 times in 18 minutes), Hyprland kept running, and the owner could neither swipe to the PIN pad nor answer a ringing call. A reboot cleared it.
Why
systemd's service watchdogs run on CLOCK_MONOTONIC. In s2idle that clock stops only when the idle loop reaches tick_freeze(), which calls timekeeping_suspend(), and that is only called from enter_s2idle_proper() in drivers/cpuidle/cpuidle.c, for a cpuidle state that has an enter_s2idle callback. zumapro's device tree has no idle states, there is no cpuidle driver (/sys/devices/system/cpu/cpuidle/current_driver reads none), so the idle loop takes default_idle_call() and the clock runs through the sleep. The sleep plan already notes that timekeeping keeps running there (S1, "the s2idle loop doesn't freeze the tick"). The services are frozen and can't ping; at the thaw systemd sees more than WatchdogSec gone and aborts them. gs201's device tree has idle states, so I'd expect the 7 Pro not to see this, but I can't check that.
It didn't show before because every sleep series on a published-tip kernel so far used 60 or 120 s sleeps. The 300 sleeps of the night before were 60 s each; even so, systemd-userdbd was killed four times in that night (two short sleeps close together add up).
What I tried
A drop-in for every service, service.d/10-…conf with WatchdogSec=0, in place from boot (a running service keeps the watchdog it started with, so a runtime drop-in and daemon-reload did nothing): WatchdogUSec read 0 for journald, logind, homed, userdbd, udevd, resolved, timesyncd and coredumpd, and a boot with twelve sleeps, one of them 4 min 15 s, had no watchdog timeout, no restart and a working session after. One long sleep is a thin count; the mechanism is a timer, so I'd expect it to hold.
Ways out, as I see them
- The kernel stops the clock: idle states for zumapro in the published device tree, entered through s2idle, so
tick_freeze()runs. That is the real fix and it is tied to the open C2 question in the sleep plan (the no-wake with C2 entered through s2idle's own path), so it may not be soon. - A stopgap in
omamobile-device-tegu: no service watchdogs while the kernel has no cpuidle driver. A systemd generator could write theservice.ddrop-in only whencurrent_driverreadsnone, so it goes away by itself once idle states land, and it would cost the watchdogs' protection only where they are doing harm. I can send that as a pull request with a longer sleep series behind it if you want it. - Leave it and keep the 9a's sleeps short. I don't think that is an option once the palmtop sleeps on its own.
Until one of these is in, any 9a sleep test longer than three minutes on a kernel without idle states needs a reboot after it, and the 9a's Suspend cell can't honestly move, whatever #232 and #556 do for the modem.
Activity
Somnius commentedon Nov 15, 2026
One data point from the 8 Pro, in case it helps, @W4HO-ham: husky is in the same state.
/sys/devices/system/cpu/cpuidle/current_driverreadsnoneon my unit on the published kernel and on my test kernels (I wrote that up on derekross/linux-omarchy-mobile#217 (comment)).I did two long s2idle sleeps through
systemctl suspendon 2026-11-10, 756 s and 619 s, both woken by the power key, on a local test kernel (details in #724 (comment)). But I only kept the timing lines for those two, so I can't say whether systemd killed any service at the wake. I never looked forWatchdog timeoutlines. If that boot's journal is still on the phone I'll grep it and post what it says, either way.So for husky: same missing idle states, long sleeps done, the watchdog effect not checked. I haven't tried your drop-in.
W4HO-ham commentedon Nov 15, 2026
Thank you, @Somnius, that helps. If that boot's journal is still there, this finds it:
journalctl -b <boot> --no-pager | grep -E "Watchdog timeout \(limit|killed by the SIGABRT". On the 9a every sleep over about three minutes showed it, for journald and logind and whatever else hasWatchdogSec=, the first time the monotonic clock jumped after the wake. If husky shows the same, the drop-in in #766 would belong in husky's device package too (or in a shared one for the kernels without idle states), and I'd add it there.Somnius commentedon Nov 15, 2026
Thanks @W4HO-ham, I ran your grep on the 8 Pro. The 11-10 boot's journal is gone (the oldest journal on the phone starts 2026-11-12 05:43), so I can't answer for those two sleeps. Over all 15 boots that are still stored there are 0 matches for
Watchdog timeout (limitorkilled by the SIGABRT.That zero doesn't tell us anything, though: the only suspend lines in those journals are two
PM: suspend entry (deep)lines at 2026-11-12 05:44, with no exit line after them, so there is no s2idle sleep with a wake in them that could show it.So husky is still unchecked. Same missing idle states, so I'd expect the same thing. I'll do a timed s2idle sleep over four minutes on my unit when I run the attended suspend test, with the drop-in from #766 left out first, and post what the journal says either way. If it shows up, I agree it belongs in husky's device package or a shared one.
W4HO-ham commentedon Nov 15, 2026
Thanks, that plan sounds right. One thing I noticed in passing:
PM: suspend entry (deep)with no exit is the case AGENTS.md's "Only s2idle, neverdeep" note is about (on gs201deepdoesn't wake). If those two weren't on purpose, it may be worth checking/sys/power/mem_sleepreads[s2idle]before the timed test.Somnius commentedon Nov 15, 2026
Good catch, thanks @W4HO-ham. I looked at the journals still on the phone and at the log I saved that day. The two
PM: suspend entry (deep)lines are from two different boots on 2026-11-12. The one at 05:44:57 ended a boot that had run about 22 hours, and my saved log has logind writingsuspend requested from client PID 1648 ('upowerd')right before it, so that one was UPower's action and not a sleep I asked for (the case I wrote up in #724 (comment)). The other line ends a boot that lasted only 30 seconds, and I couldn't find in the journal what asked for that sleep, so for that one I don't know. The clock in those journals jumps, so I can't order the two by time.The kernel I run today has
mem_sleep_default=s2idleon its command line and/sys/power/mem_sleepreads[s2idle] deep. I'll read it again right before the timed test.I also listed which system services have a watchdog on this unit: journald, logind, udevd, resolved, oomd, userdbd, nsresourced, coredumpd, networkd, timesyncd and homed, all at 3 minutes, and there is no
service.ddrop-in for any of them. So the same thing could happen here if the monotonic clock runs through the sleep as on the 9a, and I'll check that too (monotonic against boottime before and after). The sleep itself will go throughsystemctl suspendwith the RTC alarm armed, as the husky runbook in #724 (comment) asks: one sleep of a bit over four minutes without the drop-in from #766, and one with it in place from boot. None of that has run yet.docs: s2idle run logs for derekross#765
Somnius commentedon Nov 15, 2026
Hi @W4HO-ham, the timed sleeps from above are done, one run each, and husky shows the same thing as your 9a.
Both runs on one 8 Pro, kernel
7.3.0-rc6-omamobile-push1-g596d44e5bd5e, cable in, Wi-Fi and the PCIe host unloaded as the husky runbook in #724 (comment) asks,systemctl suspendwith the RTC alarm armed at +270 s,mem_sleepreading[s2idle] deep, no cpuidle driver. Sleep length is from the RTC, not from the journal.Without a drop-in: it slept 270 s (suspend_stats success 0 to 1, fail 0 to 0, entry
(s2idle)with an exit, no UFS error), CLOCK_MONOTONIC and CLOCK_BOOTTIME both grew 270.4 s, so the monotonic clock ran through the sleep. Eleven system services had a 3 minute watchdog (journald, logind, udevd, resolved, oomd, userdbd, nsresourced, coredumpd, networkd, timesyncd, homed) and after the wake the journal hasWatchdog timeout (limit 3min)!forsystemd-logind,systemd-journaldandsystemd-nsresourced. journald and logind restarted (NRestarts 0 to 1), journald dumped core, and the journal files were renamed as "corrupted or uncleanly shut down". One more effect: NetworkManager kept every device unmanaged with the reason "networking is disabled or the system is suspended", because logind was restarted and it never saw the resume.nmcli networking offandondidn't clear it, restarting NetworkManager did.With
/etc/systemd/system/service.d/10-test-765.conf([Service],WatchdogSec=0) in place from boot (one reboot in between, same kernel, same alarm, Wi-Fi unloaded again): it slept 270 s again (success 0 to 1, fail 0 to 0, MONOTONIC +271.1 s), no watchdog line, no restarts, and no running system service had a watchdog when it started.The two logs are here, if you want to check my reading: https://raw.githubusercontent.com/Somnius/omarchy-mobile/docs-power-profiles-1115/docs/figures/power-1115/s2idle/wd-a.log and https://raw.githubusercontent.com/Somnius/omarchy-mobile/docs-power-profiles-1115/docs/figures/power-1115/s2idle/wd-b.log
What this doesn't show: one run each, on one unit, with the cable in, so no current figure from it; only system services were looked at, not user units; and I haven't tried a longer sleep or one with Wi-Fi loaded. The RTC alarm does wake this phone from s2idle, which I hadn't known before.
So for husky I'd say the drop-in is needed, as on the 9a. You asked where it should live: a shared place for kernels without idle states seems better than one copy per device, and I can send husky's copy if that is what you'd prefer.
W4HO-ham commentedon Nov 15, 2026
Thanks @Somnius, that's a clean pair: the same three-minute watchdog timeouts without the drop-in and none with it, and the monotonic clock running through the sleep on husky too. The NetworkManager part is new to me and worse than what the 9a showed: with logind restarted under it, NM never saw the resume and kept every device unmanaged until it was restarted itself. So on husky a sleep over three minutes costs the network as well, not only the lock screen.
On where it lives, @derekross: #766 has it in device-tegu only. Now that two devices need it, a shared copy for every device whose kernel has no idle states would do, and #766 could move it there, or husky gets its own copy as Somnius offers. Which would you rather?
Somnius commentedon Nov 15, 2026
@derekross, two things for your choice. I'd still take the shared copy, as I said in #765 (comment), but one per device works for me too.
Husky's copy is ready as a branch, Somnius@e4de14d : the same two settings as the drop-in in #766, with its test lines. The package's
test/allpasses on my desktop; the packaged file hasn't been on the phone yet. If you want one per device I'll open it as a pull request, and if #766 becomes the shared one I'll drop it.Heads-up: #806 also takes husky to 0.1.14 and edits the README row next to this one, so the second of the two will need a rebase.
omamobile-device-tegu 0.2.21: no service watchdogs on the 9a, whose s…