Skip to content

scsi: ufs: exynos: re-apply the UFS pin state when resuming with the link down (Pixel 9a) - #224

Merged
derekross merged 2 commits into
derekross:omamobilefrom
W4HO-ham:tegu-suspend-fixes
Nov 12, 2026
Merged

derekross merged 2 commits into
derekross:omamobilefrom
W4HO-ham:tegu-suspend-fixes

Conversation

@W4HO-ham

@W4HO-ham W4HO-ham commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

One fix for suspend on the Pixel 9a (zumapro), the result of a long series on the palmtop, in two commits (the driver, then the zumapro device tree).

Dead storage after a wake. On about one wake in a hundred the UFS pads, gph5-0 (the device's reference clock) and gph5-1 (its reset), came back from a system suspend as plain inputs (CON 0x0 instead of the UFS function 0x2, read from the 13060000.pinctrl debugfs with the palmtop alive in that state). The device then gets no clock and no reset edge, the host reports no device present (HCS 0x8) through every link startup, the resume fails and the root filesystem goes read-only until a reboot reprograms the pads. Nothing on the host side reaches a device whose pads are disconnected: a host reset, a 1 ms reset pulse, a PHY power cycle, waits of up to 1.6 s between tries and a real VCC power cycle were each tried (kernels #90 to #94 in device/tegu/SLEEP-PLAN.md) and each failed the same way. The first commit re-applies the UFS node's pin state in exynos_ufs_resume() for a link-off resume, with a "sleep" pinctrl state on the node so the default is really written again.

The silent no-wake (every core in C2 during s2idle, the second failure of the series) is not in this PR any more: the google,no-s2idle idle-state property it carried doesn't do what its message said (with enter_s2idle cleared, do_idle() still takes C2 through the ordinary entry), the property had no binding, and the published zumapro tree has no idle states. The fix will be a sleep hook in omamobile-device-tegu that disables cpuidle/state1 for the sleep.

Evidence (device/tegu/SLEEP-PLAN.md on omarchy-mobile#409, logs under logs/tegu-linkwait-20261107/), all on the palmtop, kernel 7.3.0-rc6-omamobile-tegu-s2mpg15-m1-g991042e2eff3-dirty (the integration tree) with the UFS fix (and the cores kept out of C2 for the sleep):

  • 300 of 300 RTC-woken 60 s systemctl suspend cycles, WFI-only (2026-11-07/08);
  • 300 of 300 with C2 enabled while awake and the ordinary idle path in the sleep (2026-11-08);
  • the everyday path: auto-sleep through the shell, a wake by the Power key, a 30-minute and an hour's sleep, and a running 6-hour series of 30-minute auto-sleeps, all clean so far.
    Before the changes: one dead-storage wake in 15 to 215, and a silent no-wake after 6 to 135 cycles with C2.

Checks: W=1 on the touched objects and zumapro-tegu.dtb, 0 warnings; checkpatch --strict clean; tools/kernel-pr-check passes. What the series has not covered yet: this exact branch booted on the published tip (the runs were on the integration tree), and sleep with the modem on; both are on the plan. The watchdog-through-suspend diagnostic, the retry waits and the readback logging that found these are not part of this PR.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham. Tracking a one-in-a-hundred dead wake down to two pads reading CON 0x0, then the silent no-wake to "nothing after the suspend side" with the watchdog kept running through the sleep, is careful work, and 300 of 300 in both configurations is a strong result.

What I checked: the gate passes on today's omamobile (clean merge, both commits signed off, checkpatch quiet, W=1 on both files, the device trees with no new warnings), and CI's dtbs_check is green. On the Pixel 7 Pro side, nothing changes in behaviour, and I won't need a run on my phone. gs201's UFS node has no pin control at all, so dev->pins is NULL there and both pinctrl_pm_select_*() calls return at their first line; gs101 and zuma have only a "default" state, so the sleep call finds no state and the default call is a no-op for an unchanged state. Only zumapro.dtsi has a "sleep" state. The objects change only where you'd expect: exynos_ufs_suspend() and exynos_ufs_resume() (the two calls) and dt_init_idle_driver() (the property read), and the other 47 functions are identical.

The UFS fix is close. Two things before it merges:

  1. Please split the zumapro.dtsi hunk into its own commit (arm64: dts: exynos: google: zumapro: …, after the driver commit). Kernel PRs land commit by commit, and a driver change and a board change go in separate commits.
  2. As your description says, the branch hasn't booted on the published tip yet. The 300-wake runs were on the integration tree, whose ufs-exynos.c and ufshcd.c also had the retry and readback code. No wake in arm64: dts: exynos: google: zumapro-tegu: Add the haptics and the torch #96 needed a retry, so that code never ran and the result most likely carries over. Still, a short attended series on omamobile plus this branch (a few dozen RTC wakes and one Power-key wake, checking dmesg | grep -E "link startup failed|Host init failed|I/O error" after each), with uname -r in the PR, would close it.

Two wording points. The condition is !ufshcd_is_link_active(), which includes hibern8, so at rpm_lvl 3 every runtime suspend and resume switches the pin state too (your #96-#100 runs covered that). The comment and message say "link-off". And an upstream reviewer will first ask why pinctrl-samsung's own save and restore of the bank isn't enough. Your #95/#96 logs point to the answer: the controller restored con=0x22 at every wake, so the pads most likely lose their function after that restore. One sentence on that in the message would help.

The google,no-s2idle commit I'd take out of this PR, for three reasons:

  • When C2 is the only state above WFI, I don't think it does what the message says. With enter_s2idle cleared, cpuidle_enter_s2idle() finds no state and returns 0. do_idle() then falls through to cpuidle_find_deepest_state() (kernel/sched/idle.c, the s2idle branch), and that call ignores enter_s2idle and picks C2 again, entered through the ordinary ->enter and without tick_freeze(). So the cores probably still sit in C2 during the sleep, with timekeeping left running, and that may be what really made the difference. arm64: dts: exynos: google: gs201-lynx: enable Bluetooth on the APM UART #98's counters can't show which: state1's s2idle_usage can't rise once the state has no s2idle entry, and its usage also grows while awake. A read-only check would settle it: each CPU's cpuidle/state0 and state1 usage and time, read just before and just after one 60 s sleep. If state1's time grows by about 60 s per core, C2 ran through the sleep. (If PSCI ran in OSI mode with a domain hierarchy, cpuidle-psci would also set the deepest state's enter_s2idle again after dt_init_idle_driver().)
  • The property has no binding: idle-states.yaml closes its state nodes with additionalProperties: false, so the first DT that sets it fails dtbs_check. And zumapro has no idle-state nodes on omamobile or in an open PR yet, so nothing in our trees would use it. Upstream will also resist a vendor-prefixed property in the generic idle-state code.
  • What you measured in arm64: dts: exynos: google: zumapro-tegu: Add the haptics and the torch #96 is easy to get without a kernel change: a sleep hook in the 9a's device package that writes 1 to /sys/devices/system/cpu/cpu*/cpuidle/state1/disable before the sleep and 0 after keeps C2 while awake and gives real WFI in s2idle (both find_deepest_state() paths skip a disabled state). Like the NFC hook, it doesn't cover a sleep written straight to /sys/power/state.

For the root cause, one pointer. On gs101 and gs201, C2 runs with exynos-pmu's CPU PM hook (pmu_cpuhp, google,pmu-intr-gen-syscon: the CPU inform and interrupt-generator writes in __gs101_cpu_pmu_offline()), and the 7 Pro enters C2 in s2idle and wakes every time. zuma_pmu_data has no pmu_cpuhp, and zumapro.dtsi has no pmu-intr-gen node, so the 9a's C2 from the integration DT runs without that handshake. It would be worth comparing before the idle states come to a PR.

So: the UFS commit (split in two) plus the boot on the tip, and this can merge.

@W4HO-ham

W4HO-ham commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the close reading, and for the do_idle() walk: you're right that with enter_s2idle cleared the s2idle branch falls through to cpuidle_find_deepest_state(), which takes C2 through the ordinary entry, so #98's 300 of 300 most likely ran C2 with the tick running rather than WFI. I've dropped the cpuidle commit and the property; the shipping form will be the cpuidle/state1 sleep hook in omamobile-device-tegu (what #96 had for its 300), with the before-and-after state1 time read you suggest as its check, and the pmu_cpuhp comparison goes on the plan for when the idle states come to a PR.

The UFS fix is now two commits, as asked:

  • scsi: ufs: exynos: re-apply the UFS pin state when resuming with the link down: the condition and the message now say "with the link down (hibern8 included)", so at rpm_lvl 3 every runtime resume does it too, and the message says why pinctrl-samsung's restore isn't enough: it wrote the pads back as 0x22 at every wake and on the failing wakes they still read as inputs afterwards, so whatever takes the function away does it after that restore.
  • arm64: dts: exynos: google: zumapro: give the UFS node a sleep pin state.

Same code as before apart from the comment, same sign-off, gate clean (checkpatch --strict 0, W=1 on ufs-exynos.o 0, zumapro-tegu.dtb builds). The published-tip run is next: omamobile at 1a9d1d4 plus these two commits is built (7.3.0-rc6-omamobile-push1-gf693b36e62ce), and it goes on the palmtop as soon as today's auto-sleep series on the owner's kernel ends, for a few dozen RTC wakes and a Power-key wake with the UFS grep after each; the result comes here with uname -r.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham, for turning this around so quickly. The split is
exactly right, the "link down (hibern8 included)" wording and the paragraph
on why pinctrl-samsung's own restore isn't enough read well, and thank you
for taking the cpuidle reading the way you did.

I re-ran the checks on the new head: it merges cleanly onto today's
omamobile (1a882ca6916f), both commits are signed off, checkpatch is
quiet, the touched file builds with W=1, the device trees show no new
warnings, and CI's dtbs_check is green. On the Pixel 7 Pro nothing changes:
the driver code is the same as last round apart from the comment, and the
cpuidle file is now the base's own, so last round's check still holds and I
won't need a run on my phone.

One thing remains, the run on the published kernel, and one request about
it. Your build is on 1a9d1d4e9, and omamobile has moved since: it now
has the 9a's Wi-Fi on PCIe channel 1 and the modem link on channel 0, both
enabled in zumapro-tegu.dts. On 1a9d1d4e9 the 9a has no PCIe nodes at
all, so a run there would sleep without Wi-Fi, unlike your integration-tree
runs and unlike what will land. Could you rebuild on today's tip plus these
two commits for the run? A few dozen RTC wakes and one Power-key wake, with
dmesg | grep -E "link startup failed|Host init failed|I/O error" after
each and uname -r in the comment, attended for the first run as the 9a's
sleep rule asks. The published zumapro tree has no idle states yet, so this
run is WFI-only and tests the UFS fix on its own, which is what we want here.

Two small things that don't block: the driver message's 300 cycles were on
the integration tree with C2 off for the sleep, and saying so in that
sentence would be accurate; and the PR title and description still mention
the cpuidle commit.

Once the run is in, this can merge.

…link down

On the Pixel 9a (zumapro) the UFS pads, gph5-0 (the device's reference
clock) and gph5-1 (its reset), came back from a system suspend as plain
inputs (CON 0) on about one wake in a hundred. The device then gets no
clock and no reset, the host reports no device present through every
link startup (HCS 0x8, no DP), and the resume fails with the root
filesystem read-only until a reboot reprograms the pads. No host-side
retry reaches a device whose pads are disconnected: a host reset, a
longer reset pulse, a PHY power cycle, waits of up to 1.6 s between
tries and a real VCC power cycle were each tried and each failed the
same way.

pinctrl-samsung's own save and restore of the bank is not enough here:
its restore wrote the pads' CON back as 0x22 at every wake, and on the
failing wakes the pads still read as inputs afterwards, so whatever
takes their function away does it after that restore.

Select the UFS node's pin state again in the resume callback whenever
the link is down (hibern8 included, so at rpm_lvl 3 every runtime
resume does it too), once the host's clocks are on. The node needs a
"sleep" pin state for this, selected in the suspend callback, because
pinctrl_select_state() is a no-op for an unchanged state; the zumapro
device tree gets it in the next commit.

With this change 300 RTC-woken s2idle cycles completed on a Pixel 9a,
on the integration tree with the cores kept out of C2 for the sleep,
with no storage failure (2026-11-07/08); before it one wake in 15 to
215 lost the storage.

Signed-off-by: W4HO <87742609+W4HO-ham@users.noreply.github.com>
The UFS host re-applies its pin state when it resumes with the link
down, so that the pads come back as the UFS function after a system
suspend (on the Pixel 9a they read as plain inputs on about one wake in
a hundred). pinctrl_select_state() does nothing for an unchanged state,
so the node needs a second state to switch away to: the "sleep" state
names the same pin groups as the default.

Signed-off-by: W4HO <87742609+W4HO-ham@users.noreply.github.com>
@W4HO-ham W4HO-ham changed the title ufs: exynos, cpuidle: dt: the Pixel 9a wakes from suspend-to-idle every time scsi: ufs: exynos: re-apply the UFS pin state when resuming with the link down (Pixel 9a) Nov 9, 2026
@W4HO-ham

W4HO-ham commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

The published-tip run ran into something else first, worth knowing before the rebuild on today's tip goes on the palmtop. On omamobile at 1a9d1d4 plus the two commits (7.3.0-rc6-omamobile-push1-gf693b36e62ce, the published zumapro-tegu.dtb, no overlays) the boot was clean (root read-write, spm_lvl 5, no cpuidle directory, 0 UFS lines), but the first 60 s sleep never woke on its alarm: the palmtop slept 31 minutes until a key press woke it (pm_wakeup_irq 54, a GPA pin, not the PMIC's). The reason is in the boot log: sec-pmic-acpm firmware:power-management:pmic-1: no interrupt, so no RTC alarm, rtc0 has no wakealarm, and the series' rtcwake -m no -s 60 failed with set rtc wake alarm failed: Invalid argument. The published zumapro-tegu.dts gives pmic-1 no interrupt; the integration tree has interrupts = <GIC_SPI 74 IRQ_TYPE_LEVEL_HIGH> and wakeup-source on it (and a google,vgpio2ap-sysreg pointing at a syscon at 0x15090000), which is what the owner's runs woke on (irq 56). So the UFS fix wasn't exercised, and no RTC-woken series is possible on the published kernel until the main PMIC's interrupt is in its device tree. I'll propose that as its own small board commit (checking first what the published driver needs of it), rebuild on today's tip with it, and then run the series you asked for. The rebuild on today's tip (abcd6aac51cf) plus the two UFS commits is already built, 7.3.0-rc6-omamobile-push1-g0603df1c821e. After the press the palmtop also logged exynos-acpm-protocol ... Timeout! ch:2 and s5m-rtc: Failed to prepare registers for time reading: -62 on the second sleep; I'll keep that console. The two commits here are updated as you asked (the evidence sentence names the integration tree and C2 off for the sleep; title and description without the cpuidle commit).

@W4HO-ham

W4HO-ham commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Your read-only check, run on the palmtop (the owner's kernel #100, C2 states with the old google,no-s2idle): over one 60 s RTC-woken sleep every core spent 19.9 to 57.1 s in state1, in 99 to 210 entries, with no s2idle directory under the state. So yes: with enter_s2idle cleared the cores took C2 through the ordinary idle path for the sleep, and #98's 300 of 300 ran that way, not in WFI. With state1/disable set for the sleep: 0 entries and 0.0 s of state1, 61.8 to 62.0 s of state0. The failing case is therefore only C2 through s2idle's own entry, with the tick frozen. The sleep hook is omarchy-mobile#591 (tested on the palmtop: 0.7 to 2.3 s of state1 across the same sleep, the awake seconds inside the snapshots), and the numbers are in SLEEP-PLAN on omarchy-mobile#409.

On the published-tip run: with the main PMIC's interrupt added to the published tree (three small commits, coming as their own PR once signed off) the alarm works, and a first series gave 21 RTC wakes with no UFS line and then a 22nd sleep that never woke. I don't trust that boot as a published install, though: the kit pushed the Image and DTB only, so no module was loaded (no Wi-Fi, no GPU, no session, 16 failed units), and without a ramoops node it left no console. The rerun is going on the palmtop now, the same kernel (7.3.0-rc6-omamobile-push1-g6c9420bfa77f: today's tip abcd6aac51cf, this PR's two commits, the three PMIC commits) with its modules installed and the ramoops overlay alone. Result here when it's done.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

The published-tip run, done properly this time: 36 of 36.

uname -r: 7.3.0-rc6-omamobile-push1-g6c9420bfa77f, that is omamobile at abcd6aac51cf (today's tip, with the 9a's PCIe nodes), this PR's two commits, and three board commits that give the main PMIC its interrupt (without them the published tree has no RTC alarm on the 9a; they come as their own PR). Built with defconfig + kernel/configs/omamobile.config, pushed with bin/omamobile push together with its own modules, the published DTB with only the ramoops overlay on top. The boot: session up, GPU, display, haptics, torch, gauge, Bluetooth and NFC from the in-tree drivers; no idle states, so the sleep is WFI; the modem masked. Attended (I watched the first sleeps from the laptop, the owner at home).

The series: 36 systemctl suspend sleeps of 60 s (59.7 to 60.2 s by dmesg), each woken by the RTC alarm (pm_wakeup_irq 52, the PMIC), with dmesg | grep -E "link startup failed|Host init failed|I/O error" empty after each and over the whole boot, no oops, suspend_stats 36 and 0.

Two honest gaps. Wi-Fi was not up in this run: the shared config doesn't build the zuma PCIe PHY until omarchy-mobile#274 merges (now unblocked, since #187 is in), so omamobile-tegu-graphics-start failed its Wi-Fi step. Run 4 is on the palmtop now, the same sources built with #274's CONFIG_PHY_ZUMA_PCIE=m, so the sleeps run with the PCIe link and brcmfmac loaded, as it will land. And the Power-key wake comes at the end of run 4. Also for the record: the first attempt on this kernel (no modules loaded, 16 failed units, no console) went silent at its 22nd sleep; that didn't recur here and I can't explain it from what that boot left behind. The log of the 36 is in SLEEP-PLAN on omarchy-mobile#409.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham, for all of this in one afternoon, and for reporting it
as it happened, including the correction that the second boot had no modules
loaded. The commit message now says where the 300 cycles ran, and the title
and description match the two commits.

I re-ran the checks on this head: it merges cleanly onto today's omamobile
(abcd6aac51cf), both commits are signed off, checkpatch is quiet, the
touched file builds with W=1, the device trees show no new warnings, and CI's
dtbs_check is green. On the Pixel 7 Pro nothing changes: its UFS node has no
pin states, so the two new calls return at once. I checked that on today's
tree again.

And thank you for the counters across one sleep. 19.9 to 57.1 s of state1
per core with no s2idle directory answers my question completely: #98 ran
C2 through the ordinary idle path. #591 is the right follow-up.

On the run. The missing PMIC interrupt is a real find: without it the
published kernel has no RTC alarm at all on the 9a, which nobody had noticed.
Please do send those three commits as their own PR, a draft is fine, so the
tree you're testing exists somewhere before its results go on record.

The 22nd sleep is the part I need answered before this merges. Twenty-one
wakes without a UFS line look good for these two commits, and a dead-storage
wake would have brought USB back, so I don't think it's this failure. But
that kernel carries this PR, so I'd like to know rather than think:

  • the rerun's count (modules installed, ramoops on), with uname -r, the UFS
    grep after each wake and one Power-key wake;
  • if a sleep fails to wake again, the console, and then the same series on
    the same kernel without the two UFS commits (the tip plus the PMIC commits
    only). If it fails there too, it isn't this PR's, and the clean wakes stand
    as its evidence. If it only fails with them, we look at the sleep pin
    state.

Please keep the first run of each of those kernels attended, as the sleep
rule asks. Once the run is in and the no-wake is either gone or shown not to
be these commits', this merges.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

Run 4, with Wi-Fi, as you asked: 36 of 36 again. The same sources (7.3.0-rc6-omamobile-push1-g6c9420bfa77f, build #2: omamobile at abcd6aac51cf, this PR's two commits, the PMIC interrupt now in #236), built with omarchy-mobile#274's CONFIG_PHY_ZUMA_PCIE=m on top of the shared config, so the 9a's PCIe link and the BCM4383 came up from the in-tree drivers (13120000.pcie Gen2 x1, wlan0 up; GPU, display, haptics, torch, gauge, Bluetooth and NFC too). Wi-Fi stayed loaded across the sleeps (the palmtop's keep flag), and brcmfmac probed the chip again after each wake, as it does on the owner's kernel.

36 systemctl suspend sleeps of 60 s (59.1 to 60.5 s by dmesg), each woken by the RTC alarm (pm_wakeup_irq 52), the UFS grep empty after each and over the whole boot, no oops, suspend_stats 36 and 0, and wlan0 passed traffic afterwards. That makes 72 of 72 on the published tree with this PR across runs 3 and 4. The Power-key wake on this kernel comes when the owner is back this evening; I'll add it here.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

The Power-key wake, attended (2026-11-09, 19:23 MST): one systemctl suspend with a 20-minute RTC alarm as the net, the owner pressed Power once after 75 s. pm_wakeup_irq 57, the gpio-keys Power line (the RTC wakes read 52), PM: suspend exit 74.3 s after the entry, the lock screen lit on that press, no UFS line, suspend_stats 1 and 0.

One difference from runs 3 and 4 to name: this boot was 7.3.0-rc6-omamobile-push1-g773b4102f5ec, the same tree (the tip, this PR, #236) plus one device tree commit, the 9a's GNSS nodes (#237), with their modules blacklisted and the modem off, so nothing of it was bound. I used that boot because the palmtop was on it for the modem run; if you'd rather have the key wake on g6c9420bfa77f exactly, it's one push and one press.

That completes what you listed for the re-run: 36 of 36 with modules and ramoops, 36 of 36 with Wi-Fi up, the UFS grep empty after every wake, and the Power-key wake. The no-wake of the first run hasn't come back in 73 sleeps on this tree, so I have no console for it; if it does, the comparison run without these two commits is the next step, as you said.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

One thing you should know before merging, though it isn't against this PR: an overnight series on the tip (2f890a5) with this, #236 and #237 on top woke 212 times and hung at the 213th sleep. The console survived, and it is a different failure from the one this PR fixes: storage never came into it, the kernel stayed alive, and CPU 7 stopped answering just after the device suspend (an RCU stall on it every 63 s until the reset). The lines and what I'm doing next are on omarchy-mobile#409. None of the 212 wakes, nor the Power-key wake before them, logged a UFS line.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham. Run 4 with Wi-Fi up, the Power-key wake, and then 212
more wakes overnight with no UFS line in any of them: on the question this PR
answers, dead storage after a wake, that is 307 wakes on the published tree
without one. And thank you for telling me about the 213th before I merged
rather than after. The console you saved is the first real look at this
no-wake.

I re-ran the checks on this head against today's omamobile
(2f890a53389e): it merges cleanly, both commits are signed off, checkpatch
is quiet, the touched file builds with W=1 and the device trees show no new
warnings. Nothing changes on the Pixel 7 Pro.

Last time I said this merges once the no-wake is gone or shown not to be
these commits'. It came back, on a kernel that carries them, so I'd like one
more piece of evidence before I merge, and I think it's cheap. Here is how I
read your console:

  • idle=8274/1/0x4000000000000000 says CPU 7 was running a task: not idle,
    and not inside an interrupt handler. With the clock still advancing and the
    other cores in default_idle_call, the hang came before timekeeping froze,
    so during or just after the device suspend callbacks.
  • This PR's two calls write gph5's registers under the pin bank's spinlock
    with interrupts off, in the UFS device's suspend and resume. A bus access
    that never returns there would look like what you saw. I don't think it's
    that: the bank's bus clock is critical on the published tree
    (CLK_GOUT_GPIO_HSI2UFS), and your 600 sleeps on the integration tree had
    the same calls. But I can't rule it out from the source.

Two things would settle it without waiting for a lucky stack:

  • echo 1 > /sys/power/pm_print_times and
    echo 1 > /sys/power/pm_debug_messages before the series. The console then
    names each device's suspend callback as it starts and returns, so the next
    hang's last lines say whether it stopped inside 13200000.ufs, in another
    device, or after all of them. That needs no new kernel.
  • kernel.panic_on_rcu_stall=1 is a sysctl, so you can have the panic, the
    reboot through panic=10 and a clean pstore console on the kernel you
    already ran, without pseudo-NMI. If you do build pseudo-NMI, please check
    the boot log says GICv3: Pseudo-NMIs enabled; I don't know whether
    zumapro's firmware allows it, and I haven't seen a single core go quiet
    like this on the 7 Pro.

Please keep the first run of any new build attended, as the sleep rule asks;
that includes the pseudo-NMI build. And if CPU 7 answers no NMI either, then
it's below the kernel, and the series without these two commits is the next
step, as you wrote. Bear in mind that without them the dead-storage failure
will probably end that series first.

As soon as one console places that core outside the UFS suspend and the pin
controller, I'll merge this as it is. Nothing to change in the commits.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

Thank you, that is a fair bar and a cheap one. Your reading of the idle= fields is better than mine: a task running on CPU 7 with interrupts off, or the core below the kernel, and before timekeeping froze.

Tonight's series runs with pm_print_times and pm_debug_messages on and kernel.panic_on_rcu_stall=1, so the next hang names the last callback that started and didn't return, and the palmtop reboots by itself with a clean console. The build is the same sources with the function tracer added (no pseudo-NMI on the command line), so the panic also prints CPU 7's last calls from its own buffer. It's a new build, so its first run is attended, this evening, before it goes overnight.

If the console puts that core inside 13200000.ufs or the pin controller, I'll say so here first.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham. Nothing has changed in the commits, so only where
this stands.

I re-ran the checks on this head against today's omamobile
(df3a12b9028f), which now has #236 and #237: it merges cleanly, both
commits are signed off, checkpatch is quiet, ufs-exynos.c builds with W=1
and the device trees show no new warnings. So the published tip plus these
two commits is now exactly the tree your 212 wakes ran on. I also looked at
the other boards again, since the Pixel 6 family got its display today:
their UFS node has only a "default" pin state, so both new calls do nothing
there, and the Pixel 7 Pro's node has no pin states at all. Only zumapro
changes.

This waits for tonight's console and nothing else. Two things so that it
can't wait forever:

  • If the hang comes back, the line I'm after is the PM debug line for
    13200000.ufs starting and not returning, or its absence, and CPU 7's
    last calls from the tracer. A stop inside the UFS suspend isn't yet a
    stop in these two calls (the PHY power-off and the host's own suspend are
    in there too), so please post the tracer's tail for that core either way.
  • If it doesn't come back: two more series with the PM switches on, about
    400 wakes together, each new build's first run attended, and I'll merge
    on those plus the 307. The fix removes a failure that cost the root
    filesystem about once in a hundred wakes, and the no-wake was there
    before it.

Good luck tonight.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

The no-wake came back tonight, on the 94th sleep of the boot, and the palmtop did not reboot by itself, so there is no console yet.

What ran: the published tip plus these two commits with the function tracer built in (7.3.0-rc6-omamobile-push1-dbg-gff4fdf58035e, no pseudo-NMI on the command line), pm_print_times and pm_debug_messages on, kernel.panic_on_rcu_stall=1, panic=10, the tracer on cores 4 to 7. The owner was at the palmtop for the first run. 93 sleeps woke (one test sleep, a series of 90, two of the next series); the next one left the USB bus at about 20:45 MST and has not come back in the two hours since. On 11-10 the rest of the kernel kept running and reported the stall; tonight either no stall was reported or the panic didn't get as far as the reboot. The hardware watchdog is stopped across a sleep, so nothing else resets it.

The owner resets it by the buttons in the morning. The ramoops console survived that reset on 11-10, so I expect the PM debug lines of the last sleep to be there, and I'll post the lines around 13200000.ufs and whatever the tracer printed for the stuck core, either way, as you asked.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

A correction to my comment above: there was no no-wake last night. I read a silent USB link as a hang, and it was a sleep with no alarm.

What the journal of that boot shows (palmtop clock, MST):

20:44:49.87 kernel: PM: suspend entry (s2idle)                 the series' own sleep, alarm in 35 s
20:45:24.19 kernel: PM: Triggering wakeup from IRQ 53
20:45:24.42 kernel: PM: suspend exit
20:45:31.73 omamobile-sleepd: sleeping (dark resume, 5 s grace)
20:45:31.92 systemd-logind: suspend requested from client PID ... ('systemctl') (unit user@1000.service)...
20:45:33.65 kernel: PM: suspend entry (s2idle)                 no alarm armed
06:18:55.14 kernel: PM: Triggering wakeup from IRQ 57          the owner's Power key
06:18:55.45 kernel: PM: suspend exit

The series' third sleep woke on its alarm. Seven seconds later the shell's auto-sleep, which is on on this palmtop, took the dark resume back to sleep, as it is meant to: the screen was off behind the lock, no SSH session was open at that moment, and the series had released its keep-awake inhibitor at its first cycle. Nothing had armed an alarm for that sleep, so it lasted 9 hours 33 minutes, until the owner pressed the Power key in the morning, and it woke at once with no UFS line. In the first run the lock wasn't secure yet (the owner was at the palmtop), and in the two cycles before this one an SSH poll happened to be open ("not sleeping: an SSH session is open").

So on this build (7.3.0-rc6-omamobile-push1-dbg-gff4fdf58035e, the function tracer, pm_print_times, pm_debug_messages, kernel.panic_on_rcu_stall=1) that boot entered 96 sleeps: 94 woke on the RTC alarm (IRQ 53; the first was a single test sleep before the PM switches were on, the other 93 had them), one of 9.5 hours on the Power key, and the 96th, which the series started seven seconds after that wake, was cut by the Power + Volume Up reset I had asked the owner for the night before. No stall, no panic, no UFS error line in the boot's kernel log.

This is the trap of 11-10's 27-minute first sleep again, and it's mine: the series left the palmtop to the auto-sleep between its cycles. The series script now holds its own block inhibitor for the whole run and sleeps past it (systemctl suspend -i; it refuses to start with the modem up), so only its own sleeps, each with an alarm, can happen. I'm sorry for the false alarm.

For your count: 93 alarm wakes with the PM switches on so far, and the long one. The next series runs tonight toward the 400; its script is changed, so its first run is attended.

@derekross

Copy link
Copy Markdown
Owner

Thank you for the correction, and for making it as clearly as the first report: the journal lines with what asked for each sleep leave no doubt that the long one was the auto-sleep with no alarm, not a hang. I'd much rather read "I was wrong last night" at breakfast than find it out in a week. The fix for the series is on omarchy-mobile#659, and I've reviewed it there.

Where this stands: the two commits are unchanged, and I re-ran the checks on this head against today's omamobile (fc7086a68fca). It merges cleanly, both commits are signed off, checkpatch is quiet, ufs-exynos.c builds with W=1 and the device trees show no new warnings. I looked at the other boards again after this week's additions: none of them has a "sleep" pin state on its UFS node, so only zumapro changes, and nothing changes on the Pixel 7 Pro.

So the count is what's left, as I wrote last time. The hang hasn't come back, so it's the second branch: about 400 alarm wakes with the PM switches on, and I'll merge on those plus the 307. You have 93. Three requests for the rest:

  • The series script changed (#659), so its first run is attended, as you already plan. I've asked for three small fixes there; please run with those in.
  • Please use the 60 s alarm the earlier series used. Ninety of the 93 were 10 s alarms, and I'd like the wakes I add together to be the same kind of sleep.
  • Keep the UFS grep after each wake and post the totals here: wakes, suspend_stats success and fail, UFS lines, and the kernel release.

If a hang does come back, the PM debug lines around 13200000.ufs and the tracer's tail for the stuck core, as before. And the 9.5-hour sleep is worth keeping in the record for this PR too: it woke at the first key press, with no UFS line.

@W4HO-ham

Copy link
Copy Markdown
Contributor Author

The count is in: 406 alarm wakes with the PM switches on, no hang.

Today's series, with the fixes you asked for on omarchy-mobile#659 in the script and the 60 s alarm (palmtop clock, MST, 2026-11-11):

Sleeps Alarm Woke on the alarm suspend_stats success / fail UFS lines Stall, panic, oops
first run of the script, 08:38 to 08:42 3 60 s 3 3 / 0 0 0
the series, 08:43 to 16:20 310 60 s 310 313 / 0 (with the three above) 0 0

Kernel 7.3.0-rc6-omamobile-push1-dbg-gff4fdf58035e (the published tree plus these two commits, with the function tracer), one boot, pm_print_times, pm_debug_messages and kernel.panic_on_rcu_stall=1 on, the tracer running. Each sleep lasted 58.8 to 60.0 s by the kernel log, and every wake was PM: Triggering wakeup from IRQ 53, the PMIC's line. The UFS grep ran after each wake and stopped nothing. The boot's kernel journal agrees with the series' log: 313 PM: suspend entry, 313 PM: suspend exit, 313 wakes from IRQ 53 and none from anything else, and no link startup failed, Host init failed, I/O error, ufshcd_err_handler, RCU stall or panic line. The auto-sleep asked for no sleep in that boot; the series' inhibitor held it off.

With last night's 93 (90 of them 10 s alarms, as you noted) that is 406 with the switches on, 313 of them 60 s sleeps like the earlier series'. Not counted: 90 more this morning with a 35 s alarm on the script's first form, all woken by the alarm with no UFS line (they are in the note on #659).

Who was there: nobody at the palmtop for the first cycles at 08:38; the owner saw it cycling at several points through the day and watched a few cycles at about 11:25. #659 has that as it was.

The 9.5-hour sleep that woke at the first key press with no UFS line is in the SLEEP-PLAN note for this PR's record, as you asked.

@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham. That is the count I asked for, and a clear one: 313 sleeps of 60 s on one boot with the PM switches on, every one woken by the alarm, suspend_stats 313 and 0, and the kernel journal agreeing with the series' log line for line. With the 93 of the evening before that makes 406, and with the 307 before those, more than 700 wakes on the published tree with these two commits and not one UFS line. Before the fix this failure cost the root filesystem about once in a hundred wakes.

I re-ran the checks on this head against today's omamobile (fc7086a68fca): it merges cleanly, both commits are signed off, checkpatch is quiet, ufs-exynos.c builds with W=1, the device trees show no new warnings, and zumapro-tegu.dtb validates against the schemas with nothing reported, before and after. No other board's UFS node has a "sleep" pin state, and the Pixel 7 Pro's has no pin states at all, so only zumapro changes.

So this merges as it is, both commits.

Two things for the record, neither against the commits:

  • The plan on omarchy-mobile#659 notes five cycles of the 310 where the RTC difference didn't fit (three about 20 s short, one 7 s and one about 40 s long) while the kernel's clock had them at 60 s. It isn't this PR's failure, and with no idle states on the published tree the kernel clock does run through the sleep, so I take the 58.8 to 60.0 s as real. But an RTC that reads wrong after a wake deserves its own line in the plan, next to the s5m-rtc … -62 you saw on 11-09.
  • You wrote here that the changed script's first run would be attended, and you've said plainly that it wasn't. Thank you for saying so rather than leaving it out. I'll answer that on #659, where the note is.

The no-wake with CPU 7 silent is still open, and it's the next thing I'd like to understand on this phone; the console from 11-10 and the tracing build are the right tools for it.

@derekross
derekross merged commit 8aaf93e into derekross:omamobile Nov 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants