Skip to content

Pixel 9a, a kernel without idle states: a sleep over three minutes ends with systemd's watchdog killing journald and logind, and the lock screen stops working #765

Open

Description

@W4HO-ham

On a 9a kernel without CPU idle states, which is what the published kernel is for zumapro, a sleep longer than three minutes ends with systemd killing its own services, and after the one that took systemd-logind the lock screen was unusable until a reboot. Found on 2026-11-14 during the call tests for #556, on a test kernel (7.3.0-rc6-omamobile-push1-gc50fd8fdf42a, the published tip plus kernel #247, linux-omarchy-mobile#232 and its review patches).

What happened

Sleeps with the modem up, ended by a call, a text or the RTC. From one boot's journal:

05:44:41 PM: suspend entry (s2idle)
05:47:16 PM: suspend exit                                   (2 min 35 s: nothing)
05:49:03 PM: suspend entry (s2idle)
05:54:03 systemd-homed.service: Watchdog timeout (limit 3min)!
05:54:03 systemd-coredumpd.service: Watchdog timeout (limit 3min)!
05:54:03 PM: suspend exit                                   (5 min 0 s)
05:58:18 PM: suspend entry (s2idle)
06:01:59 systemd-journald.service: Watchdog timeout (limit 3min)!
06:01:59 PM: suspend exit                                   (3 min 41 s)
06:04:12 PM: suspend entry (s2idle)
06:09:49 systemd-logind.service: Watchdog timeout (limit 3min)!
06:09:49 systemd-journald.service: Watchdog timeout (limit 3min)!
06:09:50 PM: suspend exit                                   (5 min 38 s)
06:09:50 systemd-logind.service: Main process exited, code=dumped, status=6/ABRT

Every sleep of 3 min 41 s or more in that boot cost at least one service (four of four); none of the shorter ones did. After logind went, sddm's helper failed in a loop (Failed to take control of "/dev/tty1", VirtualTerminalAlreadyTaken, 5,962 times in 18 minutes), Hyprland kept running, and the owner could neither swipe to the PIN pad nor answer a ringing call. A reboot cleared it.

Why

systemd's service watchdogs run on CLOCK_MONOTONIC. In s2idle that clock stops only when the idle loop reaches tick_freeze(), which calls timekeeping_suspend(), and that is only called from enter_s2idle_proper() in drivers/cpuidle/cpuidle.c, for a cpuidle state that has an enter_s2idle callback. zumapro's device tree has no idle states, there is no cpuidle driver (/sys/devices/system/cpu/cpuidle/current_driver reads none), so the idle loop takes default_idle_call() and the clock runs through the sleep. The sleep plan already notes that timekeeping keeps running there (S1, "the s2idle loop doesn't freeze the tick"). The services are frozen and can't ping; at the thaw systemd sees more than WatchdogSec gone and aborts them. gs201's device tree has idle states, so I'd expect the 7 Pro not to see this, but I can't check that.

It didn't show before because every sleep series on a published-tip kernel so far used 60 or 120 s sleeps. The 300 sleeps of the night before were 60 s each; even so, systemd-userdbd was killed four times in that night (two short sleeps close together add up).

What I tried

A drop-in for every service, service.d/10-…conf with WatchdogSec=0, in place from boot (a running service keeps the watchdog it started with, so a runtime drop-in and daemon-reload did nothing): WatchdogUSec read 0 for journald, logind, homed, userdbd, udevd, resolved, timesyncd and coredumpd, and a boot with twelve sleeps, one of them 4 min 15 s, had no watchdog timeout, no restart and a working session after. One long sleep is a thin count; the mechanism is a timer, so I'd expect it to hold.

Ways out, as I see them

  1. The kernel stops the clock: idle states for zumapro in the published device tree, entered through s2idle, so tick_freeze() runs. That is the real fix and it is tied to the open C2 question in the sleep plan (the no-wake with C2 entered through s2idle's own path), so it may not be soon.
  2. A stopgap in omamobile-device-tegu: no service watchdogs while the kernel has no cpuidle driver. A systemd generator could write the service.d drop-in only when current_driver reads none, so it goes away by itself once idle states land, and it would cost the watchdogs' protection only where they are doing harm. I can send that as a pull request with a longer sleep series behind it if you want it.
  3. Leave it and keep the 9a's sleeps short. I don't think that is an option once the palmtop sleeps on its own.

Until one of these is in, any 9a sleep test longer than three minutes on a kernel without idle states needs a reboot after it, and the 9a's Suspend cell can't honestly move, whatever #232 and #556 do for the modem.

Activity

  1. Somnius commented on Nov 15, 2026

    @Somnius
    Contributor

    One data point from the 8 Pro, in case it helps, @W4HO-ham: husky is in the same state. /sys/devices/system/cpu/cpuidle/current_driver reads none on my unit on the published kernel and on my test kernels (I wrote that up on derekross/linux-omarchy-mobile#217 (comment)).

    I did two long s2idle sleeps through systemctl suspend on 2026-11-10, 756 s and 619 s, both woken by the power key, on a local test kernel (details in #724 (comment)). But I only kept the timing lines for those two, so I can't say whether systemd killed any service at the wake. I never looked for Watchdog timeout lines. If that boot's journal is still on the phone I'll grep it and post what it says, either way.

    So for husky: same missing idle states, long sleeps done, the watchdog effect not checked. I haven't tried your drop-in.

  2. W4HO-ham commented on Nov 15, 2026

    @W4HO-ham
    ContributorAuthor

    Thank you, @Somnius, that helps. If that boot's journal is still there, this finds it: journalctl -b <boot> --no-pager | grep -E "Watchdog timeout \(limit|killed by the SIGABRT". On the 9a every sleep over about three minutes showed it, for journald and logind and whatever else has WatchdogSec=, the first time the monotonic clock jumped after the wake. If husky shows the same, the drop-in in #766 would belong in husky's device package too (or in a shared one for the kernels without idle states), and I'd add it there.

  3. Somnius commented on Nov 15, 2026

    @Somnius
    Contributor

    Thanks @W4HO-ham, I ran your grep on the 8 Pro. The 11-10 boot's journal is gone (the oldest journal on the phone starts 2026-11-12 05:43), so I can't answer for those two sleeps. Over all 15 boots that are still stored there are 0 matches for Watchdog timeout (limit or killed by the SIGABRT.

    That zero doesn't tell us anything, though: the only suspend lines in those journals are two PM: suspend entry (deep) lines at 2026-11-12 05:44, with no exit line after them, so there is no s2idle sleep with a wake in them that could show it.

    So husky is still unchecked. Same missing idle states, so I'd expect the same thing. I'll do a timed s2idle sleep over four minutes on my unit when I run the attended suspend test, with the drop-in from #766 left out first, and post what the journal says either way. If it shows up, I agree it belongs in husky's device package or a shared one.

  4. W4HO-ham commented on Nov 15, 2026

    @W4HO-ham
    ContributorAuthor

    Thanks, that plan sounds right. One thing I noticed in passing: PM: suspend entry (deep) with no exit is the case AGENTS.md's "Only s2idle, never deep" note is about (on gs201 deep doesn't wake). If those two weren't on purpose, it may be worth checking /sys/power/mem_sleep reads [s2idle] before the timed test.

  5. Somnius commented on Nov 15, 2026

    @Somnius
    Contributor

    Good catch, thanks @W4HO-ham. I looked at the journals still on the phone and at the log I saved that day. The two PM: suspend entry (deep) lines are from two different boots on 2026-11-12. The one at 05:44:57 ended a boot that had run about 22 hours, and my saved log has logind writing suspend requested from client PID 1648 ('upowerd') right before it, so that one was UPower's action and not a sleep I asked for (the case I wrote up in #724 (comment)). The other line ends a boot that lasted only 30 seconds, and I couldn't find in the journal what asked for that sleep, so for that one I don't know. The clock in those journals jumps, so I can't order the two by time.

    The kernel I run today has mem_sleep_default=s2idle on its command line and /sys/power/mem_sleep reads [s2idle] deep. I'll read it again right before the timed test.

    I also listed which system services have a watchdog on this unit: journald, logind, udevd, resolved, oomd, userdbd, nsresourced, coredumpd, networkd, timesyncd and homed, all at 3 minutes, and there is no service.d drop-in for any of them. So the same thing could happen here if the monotonic clock runs through the sleep as on the 9a, and I'll check that too (monotonic against boottime before and after). The sleep itself will go through systemctl suspend with the RTC alarm armed, as the husky runbook in #724 (comment) asks: one sleep of a bit over four minutes without the drop-in from #766, and one with it in place from boot. None of that has run yet.

  6. Somnius commented on Nov 15, 2026

    @Somnius
    Contributor

    Hi @W4HO-ham, the timed sleeps from above are done, one run each, and husky shows the same thing as your 9a.

    Both runs on one 8 Pro, kernel 7.3.0-rc6-omamobile-push1-g596d44e5bd5e, cable in, Wi-Fi and the PCIe host unloaded as the husky runbook in #724 (comment) asks, systemctl suspend with the RTC alarm armed at +270 s, mem_sleep reading [s2idle] deep, no cpuidle driver. Sleep length is from the RTC, not from the journal.

    Without a drop-in: it slept 270 s (suspend_stats success 0 to 1, fail 0 to 0, entry (s2idle) with an exit, no UFS error), CLOCK_MONOTONIC and CLOCK_BOOTTIME both grew 270.4 s, so the monotonic clock ran through the sleep. Eleven system services had a 3 minute watchdog (journald, logind, udevd, resolved, oomd, userdbd, nsresourced, coredumpd, networkd, timesyncd, homed) and after the wake the journal has Watchdog timeout (limit 3min)! for systemd-logind, systemd-journald and systemd-nsresourced. journald and logind restarted (NRestarts 0 to 1), journald dumped core, and the journal files were renamed as "corrupted or uncleanly shut down". One more effect: NetworkManager kept every device unmanaged with the reason "networking is disabled or the system is suspended", because logind was restarted and it never saw the resume. nmcli networking off and on didn't clear it, restarting NetworkManager did.

    With /etc/systemd/system/service.d/10-test-765.conf ([Service], WatchdogSec=0) in place from boot (one reboot in between, same kernel, same alarm, Wi-Fi unloaded again): it slept 270 s again (success 0 to 1, fail 0 to 0, MONOTONIC +271.1 s), no watchdog line, no restarts, and no running system service had a watchdog when it started.

    The two logs are here, if you want to check my reading: https://raw.githubusercontent.com/Somnius/omarchy-mobile/docs-power-profiles-1115/docs/figures/power-1115/s2idle/wd-a.log and https://raw.githubusercontent.com/Somnius/omarchy-mobile/docs-power-profiles-1115/docs/figures/power-1115/s2idle/wd-b.log

    What this doesn't show: one run each, on one unit, with the cable in, so no current figure from it; only system services were looked at, not user units; and I haven't tried a longer sleep or one with Wi-Fi loaded. The RTC alarm does wake this phone from s2idle, which I hadn't known before.

    So for husky I'd say the drop-in is needed, as on the 9a. You asked where it should live: a shared place for kernels without idle states seems better than one copy per device, and I can send husky's copy if that is what you'd prefer.

  7. W4HO-ham commented on Nov 15, 2026

    @W4HO-ham
    ContributorAuthor

    Thanks @Somnius, that's a clean pair: the same three-minute watchdog timeouts without the drop-in and none with it, and the monotonic clock running through the sleep on husky too. The NetworkManager part is new to me and worse than what the 9a showed: with logind restarted under it, NM never saw the resume and kept every device unmanaged until it was restarted itself. So on husky a sleep over three minutes costs the network as well, not only the lock screen.

    On where it lives, @derekross: #766 has it in device-tegu only. Now that two devices need it, a shared copy for every device whose kernel has no idle states would do, and #766 could move it there, or husky gets its own copy as Somnius offers. Which would you rather?

  8. Somnius commented on Nov 15, 2026

    @Somnius
    Contributor

    @derekross, two things for your choice. I'd still take the shared copy, as I said in #765 (comment), but one per device works for me too.

    Husky's copy is ready as a branch, Somnius@e4de14d : the same two settings as the drop-in in #766, with its test lines. The package's test/all passes on my desktop; the packaged file hasn't been on the phone yet. If you want one per device I'll open it as a pull request, and if #766 becomes the shared one I'll drop it.

    Heads-up: #806 also takes husky to 0.1.14 and edits the README row next to this one, so the second of the two will need a rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions