From 9ee19f9cd6ce29dcd6d15468b2e48eb7cc31c542 Mon Sep 17 00:00:00 2001 From: W4HO <87742609+W4HO-ham@users.noreply.github.com> Date: Sat, 14 Nov 2026 22:52:59 -0644 Subject: [PATCH 1/3] device/tegu SLEEP-PLAN, bringup, AGENTS: the no-wake caught twice with a console, its cause, and #247 The follow-up #659 left for later. The no-wake on the published kernel came back on 2026-11-12 with the heavy debug switches off, twice, and the console with the device callback events on stops at the start of the noirq resume of an HSI0 USI on CPU 7. A bring-up module showed the cause awake on 11-13: a register access to a USI in the first microseconds after its clock gates are opened never returns. linux-omarchy-mobile#247 keeps the six gates on; on the published tip with only that change 300 of 300 sleeps woke, and touch and GNSS work. - SLEEP-PLAN: the status, the five runs of 11-12 and 11-13 in the run table, "What is left" item 1 rewritten around the cause and the fix, and the dated notes of both catches, the awake test and the fix's runs. - sleep-series.sh: TRACE=2 (the events without the function tracer), PMPRINT=0 (no callback names on the console) and the device PM callback events, the form that caught it. - bringup/hsi0-usi: the two test modules, the clock hold and the stress test, with a README and a header that says the stress test hangs the SoC by design. - AGENTS.md: the 9a sleep entry's open no-wake brought up to the cause and #247. --- AGENTS.md | 2 +- device/tegu/SLEEP-PLAN.md | 232 ++++++++++++++++-- device/tegu/bringup/hsi0-usi/Makefile | 2 + device/tegu/bringup/hsi0-usi/README.md | 43 ++++ .../bringup/hsi0-usi/tegu-hsi0-usi-clk-hold.c | 74 ++++++ .../bringup/hsi0-usi/tegu-hsi0-usi-stress.c | 135 ++++++++++ device/tegu/bringup/sleep-series.sh | 19 +- 7 files changed, 474 insertions(+), 33 deletions(-) create mode 100644 device/tegu/bringup/hsi0-usi/Makefile create mode 100644 device/tegu/bringup/hsi0-usi/README.md create mode 100644 device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-clk-hold.c create mode 100644 device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c diff --git a/AGENTS.md b/AGENTS.md index 25acda816..154bb5763 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -189,7 +189,7 @@ Each entry gives the gotcha, then the reason or the fix. Details are in the link ### Suspend and modem - **On the 9a, NFC must not poll through s2idle.** A suspend with neard polling reset the palmtop (2026-10-27); with polling stopped, 35 s2idle cycles with RTC wakes that day all woke (`7.3.0-rc6-omamobile-tegu-s2mpg15-m1`). The cause isn't known: nxp-nci has no PM callbacks, but neither does the 7 Pro's st-nci, so the controller's interrupt or the I2C bus during the freeze are the next suspects. A sleep hook on the palmtop, `omamobile-tegu-nfc` (not in the repository yet; it comes with tegu's device package), stops neard's poll loop before a systemd sleep and restarts it after; a sleep written straight to `/sys/power/state` (`rtcwake -m mem`) skips the hook, so stop polling by hand first. Without [linux-omarchy-mobile#131](https://github.com/derekross/linux-omarchy-mobile/pull/131) the ST54L also doesn't answer on I2C after a wake (`Error -6 on I2C send`, also after the driver's retry), or after about 2-3 s of idle at any time, until neard powers it off and on ([#132](https://github.com/derekross/omarchy-mobile/issues/132)): nxp-nci never turned its NFC mode on, as ST's HAL does after init. With #131 (the results below on `core.c` blob `77fc35b61da5`, #131's first version; the merged version, `c96704ddd480`, re-run for six power-ons and polling at 3 s and 30 s; the release, `7.3.0-rc6-omamobile-tegu-s2mpg15-m1-g991042e2eff3-dirty`, is the same as without the fix), NFC polls after 3 s to 3 minutes of idle and polled again after two RTC-woken s2idle cycles (2026-10-29): one with a hook that stops polling and powers the adapter off, and one without that hook, for which it isn't recorded whether polling was stopped, so whether NFC comes back after a wake with the stop-poll hook alone rests on that one cycle and needs a re-run with the hook's log. No power-off hook is needed for NFC; the stop-poll rule above stands on its 35 of 35. [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" -- **On the 9a, a new sleep test runs attended first, and the direct freeze path is still not used.** Unattended sleep series are allowed since 2026-11-05 (#409) on the contributor's conditions and three of the maintainer's. The contributor's: on the charger with the cable in; the modem off for the boot; the script stops itself on any UFS error line or oops; a wake with dead storage (SSH still answers) gets `echo b > /proc/sysrq-trigger` remotely, and only a silent hang waits for the owner's buttons; the owner says yes before each run. (Their "slot B marked" condition is not part of it: marking B on the 9a stays on hold, see the boot chain notes above, so a series that resets the palmtop spends ABL tries and stays short.) The maintainer's: nothing new (a kernel, a module or a script) runs unattended before one attended run of it; the automatic sysrq recovery is capped per night, and the next boot's ext4 recovery is checked and recorded; and the recovery script is in the tree (`device/tegu/bringup/`, where the series scripts already are) before it runs unattended. The plan says which rule each step runs under, and the counts below are the early ones; [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) keeps the current tally. Four sleeps have never woken (not from the RTC alarm, not from the power key; Power + Volume Up to reset, nothing in pstore): two direct `rtcwake -m freeze` runs (2026-10-28, 2026-10-29), one 30-minute `systemctl suspend` with `tegu-mif-sleep` loaded (2026-10-30, cable out) and the sixth 30 s `systemctl suspend` cycle on 2026-10-30 (the daily boot, no module; it left the USB bus); 35 short and seven longer systemd sleeps woke (two 30-minute and four 240 s ones on 2026-10-29, a cable-in 30-minute one with the module on 2026-10-30; the afternoon's cable-out run with the module makes eight). No single factor explains it, so keep `power-ab` on `--suspend systemd` (the sleep hooks run that way), and don't load `tegu-mif-sleep` at boot until the hang is understood. The shell's auto-sleep (the `omamobile-sleep` toggle, systemd path, no module) woke every time while it was on, including seven sleeps of 11 s to 22 minutes in a day of use on 2026-11-01 (lengths from the RTC); the package ships it off, and the owner turned it back on on the palmtop that day, so keeping it on is the owner's call. Two causes are known. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224) re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. Open: on the published kernel, which has no idle states, a sleep still fails to wake now and then: one in 22 on 2026-11-09 (a boot without modules, no console), none in two re-runs of 36 with modules loaded, and one in 213 overnight on 2026-11-10 on the newest tip with the fixes. That one left a console: the kernel stayed alive, and CPU 7 answered nothing from just after the device suspend until the owner's reset (an RCU stall on CPU 7 every 63 s; neither the alarm nor the Power key got through). So a palmtop that stays silent after a sleep may be a running kernel with one core locked up, not a SoC that stayed down; what that core was doing isn't known. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; on a kernel from before it `rtcwake` fails and a sleep ends only on a key. Check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. The rule above stands. A sleep series has to keep the shell's auto-sleep out: with it on, `omamobile-sleepd` puts a wake that leaves the screen off behind a secure lock back to sleep after 5 s, with no alarm, and from the laptop that sleep looks like a no-wake (2026-11-10, twice; the second lasted nine and a half hours and was reported as a hang). `device/tegu/bringup/sleep-series.sh` holds a block inhibitor of its own for that; before calling a silent palmtop hung, ask for one short press of the Power key and read the journal. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 +- **On the 9a, a new sleep test runs attended first, and the direct freeze path is still not used.** Unattended sleep series are allowed since 2026-11-05 (#409) on the contributor's conditions and three of the maintainer's. The contributor's: on the charger with the cable in; the modem off for the boot; the script stops itself on any UFS error line or oops; a wake with dead storage (SSH still answers) gets `echo b > /proc/sysrq-trigger` remotely, and only a silent hang waits for the owner's buttons; the owner says yes before each run. (Their "slot B marked" condition is not part of it: marking B on the 9a stays on hold, see the boot chain notes above, so a series that resets the palmtop spends ABL tries and stays short.) The maintainer's: nothing new (a kernel, a module or a script) runs unattended before one attended run of it; the automatic sysrq recovery is capped per night, and the next boot's ext4 recovery is checked and recorded; and the recovery script is in the tree (`device/tegu/bringup/`, where the series scripts already are) before it runs unattended. The plan says which rule each step runs under, and the counts below are the early ones; [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) keeps the current tally. Four sleeps have never woken (not from the RTC alarm, not from the power key; Power + Volume Up to reset, nothing in pstore): two direct `rtcwake -m freeze` runs (2026-10-28, 2026-10-29), one 30-minute `systemctl suspend` with `tegu-mif-sleep` loaded (2026-10-30, cable out) and the sixth 30 s `systemctl suspend` cycle on 2026-10-30 (the daily boot, no module; it left the USB bus); 35 short and seven longer systemd sleeps woke (two 30-minute and four 240 s ones on 2026-10-29, a cable-in 30-minute one with the module on 2026-10-30; the afternoon's cable-out run with the module makes eight). No single factor explains it, so keep `power-ab` on `--suspend systemd` (the sleep hooks run that way), and don't load `tegu-mif-sleep` at boot until the hang is understood. The shell's auto-sleep (the `omamobile-sleep` toggle, systemd path, no module) woke every time while it was on, including seven sleeps of 11 s to 22 minutes in a day of use on 2026-11-01 (lengths from the RTC); the package ships it off, and the owner turned it back on on the palmtop that day, so keeping it on is the owner's call. Two causes are known. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224) re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. A third cause, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (2026-11-09 to 11-12). The consoles show the kernel alive and CPU 7, which runs the suspend, taking no tick and answering no IPI from the noirq resume of devices on, so a palmtop that stays silent after a sleep may be a running kernel with one core locked up, not a SoC that stayed down; neither the alarm nor the Power key gets through. The core is in a register access to an HSI0 USI (the touchscreen's or the GNSS SPI's) in the first microseconds after its clock gates are opened, which never returns: shown awake with a test module that hangs the SoC on purpose (`device/tegu/bringup/hsi0-usi/`, attended only). [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps those six gates on (300 of 300 on the published tip with it alone, 2026-11-13); until a kernel the palmtop runs carries it, the no-wake stays possible, about one in 300. The PM debug switches of `sleep-series.sh` hide it (701 sleeps with them on and none hung): to catch it use `PMDEBUG=1 TRACE=2 PMPRINT=0`. The stall panic writes the console but its own restart hangs, so the buttons are still needed, and pstore may reject the console's header; the text can be read raw from the ramoops region on the next boot ([tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 item 1). The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; on a kernel from before it `rtcwake` fails and a sleep ends only on a key. Check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. The rule above stands. A sleep series has to keep the shell's auto-sleep out: with it on, `omamobile-sleepd` puts a wake that leaves the screen off behind a secure lock back to sleep after 5 s, with no alarm, and from the laptop that sleep looks like a no-wake (2026-11-10, twice; the second lasted nine and a half hours and was reported as a hang). `device/tegu/bringup/sleep-series.sh` holds a block inhibitor of its own for that; before calling a silent palmtop hung, ask for one short press of the Power key and read the journal. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 - **On the 9a, a sleep can end badly in two ways: UFS fails to come back and the root filesystem goes read-only, or the SoC hangs and leaves the USB bus.** The first seen once on 2026-10-30 (`link startup failed 1`, `Host init failed 1` x5, `ufshcd_wl_resume failed`, then `I/O error, dev sda` and ext4 `emergency_ro`; `spm_lvl` 5). The palmtop woke on its alarm but nothing on disk could be read: `systemctl suspend`/`reboot` did nothing, the power key could not light the screen, SSH kept answering from memory. A dark palmtop that ignores the power key may therefore be awake with dead storage, not asleep. After every s2idle, `dmesg | grep -E "link startup failed|Host init failed|I/O error"` first; if it matches, `echo b > /proc/sysrq-trigger` (binaries won't load), then check the ext4 recovery on the next boot. The dead storage is understood: gph5-0 and gph5-1 (the UFS reference clock and reset, `13060000.pinctrl`) read `CON 0x0` after such a wake, so the device gets no clock and no reset edge; linux-omarchy-mobile#224 re-applies the UFS pin state in `exynos_ufs_resume()`. On a kernel without it the check above stays. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 - **UFS must be at `spm_lvl` 5 in system suspend.** At the core default 3, UFS failed its first hibern8 exit on resume and needed a host reset (it recovered, no I/O errors); at 5 the first real s2idle freeze resumed clean. gs201 defaults to 5 in the kernel (S3a, `d0c08c464ef8` on `omamobile`), and gs101 too since linux-omarchy-mobile#188 (2026-11-04), so a suspend test only checks that `spm_lvl` reads 5; on a kernel without those commits, `echo 5` first. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) S1-S3 - **Only s2idle, never `deep`.** The firmware advertises PSCI SYSTEM_SUSPEND, so mainline offers `deep`, but Linux has no gs201 SoC sleep sequence (SYS_SLEEP), and the phone wouldn't wake. `mem_sleep_default=s2idle` in the bootargs and device-cheetah's `sleep.conf` drop-in keep it on s2idle. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) §1, §5 diff --git a/device/tegu/SLEEP-PLAN.md b/device/tegu/SLEEP-PLAN.md index b03f250ad..19416c1a2 100644 --- a/device/tegu/SLEEP-PLAN.md +++ b/device/tegu/SLEEP-PLAN.md @@ -1,7 +1,7 @@ # Pixel 9a sleep: why s2idle drains so much -Status (2026-11-13): Two causes are known and fixed, and a third failure is open. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224), merged on 2026-11-12 (00:41Z), re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. Open: on the published kernel, which has no idle states, a sleep has failed to wake twice: one in 22 on 2026-11-09 (a boot without modules, no console) and one in 213 overnight on 2026-11-10 on the newest tip with the fixes. That one left a console: the kernel stayed alive, and CPU 7 answered nothing from just after the device suspend until the owner's reset (an RCU stall on CPU 7 every 63 s; neither the alarm nor the Power key got through); what that core was doing wasn't known then. The tally on that tree, with #224's two commits: 1,100 sleeps woken and those two not. With the PM debug switches on it didn't come back: 701 sleeps from 11-10 to 11-12 with the PM debug switches on (`pm_print_times`, `pm_debug_messages`, a stall made a panic, the function tracer), 313 and 295 of them 60 s sleeps on one boot each, all woke on the alarm with no UFS line. That is not shown to be a fix: at the earlier rate four or five no-wakes were to be expected in 701 and none would be a one-in-ninety chance, but two no-wakes fix that rate poorly (they fit anything from about one in 1,300 sleeps to one in 40), and the switches may change the timing. With the function tracer and `pm_print_times` off it came back on 2026-11-12, twice; the console places CPU 7 in the noirq resume of an HSI0 USI, and [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) proposes keeping those clocks on; the record follows. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; a kernel from before it has no alarm (`rtcwake` fails and a sleep ends only on a key), so check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. Evidence: 300 of 300 RTC-woken 60 s sleeps twice (#96 WFI-only, #98 C2 at runtime, 2026-11-07/08), 15 of 15 wakes on the everyday path, the shell's auto-sleep behind the lock, twelve of them half-hour sleeps (#99, 2026-11-08), 12 of 12 half-hour auto-sleeps on #100 (11-09), and on the published tree with #224 and the PMIC interrupt 1,100 wakes and two no-wakes (11-09 to 11-12). The attended-first rule (S1, AGENTS.md) stands; the overnight run of 11-10 and the runs of 11-11 did not keep it (the run table and its header say how each ran). +Status (2026-11-15): Three causes are known: two are fixed, one in the kernel and one by a sleep hook, and the third has a kernel fix under review. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224), merged on 2026-11-12 (00:41Z), re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. The third, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (the 22nd of a run on 2026-11-09, with no console; the 213th of a night on 11-10; the 70th and the 41st of two boots on 11-12). The three consoles agree: CPU 7, which runs the suspend, takes no tick and answers no IPI from the noirq resume of devices on, and the console with the device callback events on stops at the start of the resume of an HSI0 USI (the touchscreen's, 111d00c0). The cause was shown awake on 11-13: a register access to one of those USIs in the first microseconds after its clock gates are opened never returns, and a test module that does it hangs the SoC within 1,000 rounds. [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps the six gates on; on the published tip with only that change, 300 of 300 sleeps woke (11-13), and touch and GNSS work. The PM debug switches hid it: 701 sleeps with them on and none hung (11-10 to 11-12), most likely through the microseconds the function tracer puts between the gate's write and the register read. The record is in "What is left" item 1 and the notes of 11-12 and 11-13. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; a kernel from before it has no alarm (`rtcwake` fails and a sleep ends only on a key), so check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. Evidence: 300 of 300 RTC-woken 60 s sleeps twice (#96 WFI-only, #98 C2 at runtime, 2026-11-07/08), 15 of 15 wakes on the everyday path, the shell's auto-sleep behind the lock, twelve of them half-hour sleeps (#99, 2026-11-08), 12 of 12 half-hour auto-sleeps on #100 (11-09), on the published tree with #224 and the PMIC interrupt 1,209 wakes and four no-wakes (11-09 to 11-12), and with #247 added 310 of 310 (11-13). The attended-first rule (S1, AGENTS.md) stands; the overnight run of 11-10, the runs of 11-11 and two runs of 11-12 did not keep it (the run table and its header say how each ran). S1 is the condensed account: the two causes, the fix, a run table, what is left and what was ruled out. The dated working notes that led there follow it, oldest first. @@ -104,6 +104,11 @@ the 310 sleeps that followed): | the same build, 11-11, `sleep-series.sh` holding its own sleep inhibitor (first form, before its review) | the inhibitor, and whether the auto-sleep stays out | WFI | **90 of 90** with a 35 s alarm (33.8 to 34.9 s, `pm_wakeup_irq` 53, no UFS line), stopped by hand for the review's changes | no (the owner out) | | the same build, 11-11, the script as reviewed (`PMDEBUG=1 TRACE=1`, 60 s alarm) | the count with the PM switches on | WFI | **313 of 313** (3 in the script's first run, then 310 from 08:43 to 16:20; 58.8 to 60.0 s, `pm_wakeup_irq` 53 each, no UFS line, no stall, `suspend_stats` 313 and 0) | no: nobody at the palmtop for the first cycles; the owner saw it cycling through the day and watched a few cycles at about 11:25 | | the same build and script, the night of 11-11 to 11-12 (`PMDEBUG=1 TRACE=1`, 60 s alarm) | the count again, overnight | WFI | **295 of 295** (21:35 to 04:50; 58.7 to 60.0 s, `pm_wakeup_irq` 53 each, no UFS line, no stall, `suspend_stats` 295 and 0) | the owner's yes for the night, given before he went to sleep; he slept through it. The attended runs it rests on: the build's on the evening of 11-10, and the script's, in this form, the cycles the owner watched at about 11:25 on 11-11 (the screen coming on at the lock screen, going dark for about a minute and coming on again), in a run that had begun with nobody there | +| the same sources with the S2MPG14 RTC wait of kernel #245 (`push1-dbg-gc63fa9271bb8`, build #6), 11-12, the stall panic and the trace events on, the function tracer and `pm_print_times` off (`PMDEBUG=1 TRACE=2 PMPRINT=0`) | whether the no-wake comes back without the heavy switches, and the RTC wait's first sleeps | WFI | **10 of 10** attended (07:38 to 07:52), then 59 woke and the 60th never did (entered 09:18:40): CPU 7 silent from the start of the noirq resume, the stall a panic, the panic's reboot hung, the console read raw from the ramoops region after the owner's reset. 69 woke on the boot, each 59.3 to 60.3 s, all 69 RTC differences 62 or 63 s | the ten yes, the owner at the palmtop; the 300 with his yes, he was in and out | +| the same build and switches with the device callback events on (`power:device_pm_callback_start` and `_end`), 11-12 | which callback it stops in | WFI | **10 of 10** attended (09:54 to 10:09), then 30 woke and the 31st never did (entered 10:53:46): CPU 7's last event is the start of the noirq resume of the USI at 111d00c0, the touchscreen's, with no end. 40 woke on the boot | the ten yes, the owner at the palmtop; the 300 with his yes, he was out | +| the same build and switches, the evening of 11-12, with a bring-up module holding `pclk` and `ipclk` of the USIs at 111d00c0 and 111e00c0 on | whether the no-wake goes when those clock gates are never switched | WFI | **300 of 300** (16:20 to 23:40; 59.2 to 60.3 s, `pm_wakeup_irq` 53 each, no UFS line, no stall, `suspend_stats` 300 and 0, every RTC difference 62 s) | no: nobody at the palmtop, and the module had had no attended run; on the owner's standing instruction for his own palmtop, so outside the attended-first rule | +| the same sources with the PCLK gates of USI2 and USI3 critical in the clock driver (`push1-dbg-g9af120756764`, build #8), the night of 11-12 to 11-13, the same switches, no module | whether the bus clocks alone are enough | WFI | 226 woke and the 227th never did (23:44 to 05:15): USI2's noirq resume returns and the suspend task's last event is the start of USI3's, at 111e00c0. 59.0 to 60.3 s, `pm_wakeup_irq` 53 each, all 226 RTC differences 62 or 63 s | no: nobody at the palmtop, and the kernel had had no attended run; on the owner's standing instruction, outside the attended-first rule | +| the published tip `2db2bc35b` plus only the six USI2 and USI3 gates critical (kernel #247; `push1-gd74c5bd9dd0e`, build #11), 11-13, no debug switches | the fix on the tree as published | WFI | **10 of 10** attended (06:54 to 07:07; 59.3 to 60.3 s, `pm_wakeup_irq` 53 each, no UFS line), then **300 of 300** with his yes (07:08 to 14:27; 59.2 to 60.3 s, `pm_wakeup_irq` 53 each, no UFS line, no oops, `suspend_stats` 310 and 0) | the ten yes, the owner watching the screen; the 300 with his yes | Before #96, one wake in 15 to 215 lost the storage; before #98, C2 in the sleep lost the palmtop after 6 to 135 cycles. @@ -120,32 +125,101 @@ missing pull, NFC polling (stopped by the hook throughout), the modem **What is left.** -1. The no-wake on the published kernel. On that tree 1,100 sleeps have woken - (21, 36, 36, two ended by the Power key, 212, the 95 of the evening of - 11-10, 90 and 313 on 11-11, and 295 in the night after) and two have not: the 22nd of 11-09's first run, with no - console, and the 213th of 11-10's, whose console shows CPU 7 not - answering while the rest of the kernel runs (failure 3 above). With the - switches below on it didn't come back. What was set up for it, in place of the pseudo-NMI build - planned here before (the maintainer's pointers on kernel #224): - `pm_print_times` and `pm_debug_messages`, so the console names each - device's suspend callback as it starts and returns; - `kernel.panic_on_rcu_stall=1`, so the stall becomes a panic, a reboot - through `panic=10` and a console in pstore with no buttons; and, on a - build with the function tracer, each core's last calls in the panic's - dump (`PMDEBUG=1` and `TRACE=1` in `sleep-series.sh`). With them on, - 701 sleeps have woken on the alarm and none has hung: 93 on the evening - of 11-10, 313 of 60 s on 11-11 and 295 of 60 s in the night after - (the last notes below). The first 406 are the count the maintainer - asked for, and #224 merged on it. It doesn't show the no-wake gone: - without the switches that tree had two in about 310 sleeps, so four - or five were to be expected in 701 and none would happen by chance - about one time in ninety; but two no-wakes fix that rate poorly - (anything from about one in 1,300 sleeps to one in 40 fits them), and - at the low end 701 clean sleeps are an even chance. And the switches - may change the timing. With the function tracer and `pm_print_times` - off it came back on 2026-11-12, twice; the console places CPU 7 in the - noirq resume of an HSI0 USI, and [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) - proposes keeping those clocks on; the record follows. +1. The no-wake on the published kernel. On that tree, with nothing + holding the HSI0 USI clocks, 1,209 sleeps have woken (21, 36, 36, two + ended by the Power key, 212, the 95 of the evening of 11-10, 90 and + 313 on 11-11, 295 in the night after, and 69 and 40 on 11-12) and four + have not: the 22nd of 11-09's first run, with no console; the 213th of + 11-10's; and the 70th and the 41st of 11-12's two boots. The last + three consoles show the same thing, CPU 7 taking no tick and answering + nothing while the other seven run (failure 3 above), and 11-12's two + say where: the alarm arrives on time, the suspend task + (`systemd-sleep`, on CPU 7) leaves s2idle, kicks the other CPUs out of + idle, logs `machine_suspend` end and `dpm_resume_noirq` begin, and in + the second, which had the device callback events on, gets through the + bus and the first three USIs and starts the callback of the fourth, + 111d00c0; that start is CPU 7's last event (the notes of 11-12 + below). So it is a hang in the noirq resume of one device, on the CPU + that runs the suspend. On this tree that phase is 17 callbacks: `simple_pm_bus_resume` (soc), then + `exynos_usi_resume_noirq` for the USIs at 108b00c0, 108d00c0, 10c900c0, + 111d00c0, 111e00c0 and 111f00c0, `exynos_cpupm_resume_noirq` + (exynos-pmu), the USI at 155d00c0, `exynos5_i2c_resume_noirq` for + 108b0000, 108d0000, 10c90000 and 111f0000, `s3c24xx_serial_resume_noirq` + (155d0000) and, in workers, `pci_pm_resume_noirq` for the Wi-Fi's port + and device. All but the PCI ones run in the suspend task, in that + order. + + What 111d00c0 is, and the reading. `usi2_hsi0`, the USI of the + touchscreen's SPI, the first of three in the HSI0 block. Its noirq + resume writes one sysreg field, turns `pclk` and `ipclk` on, reads and + writes `USI_CON`, waits 1 us and turns them off. The three USIs + before it are in PERIC0 and PERIC1, whose USI gates `clk-zumapro.c` + registers read-only, so they switch nothing; HSI0's USI2 and USI3 + gates are writable and in manual mode, and nothing holds them while + the SPI controllers are idle. So it is the first callback of the phase + that switches a manual-mode bus clock gate on and touches the block at + once, which is what kernel #172 took away for the PERIC0 GPIO gate, + where the SoC hung with nothing in the log. The reading: the register + access hangs the bus when it meets the gate's switch, and CPU 7 sits + in it. + + The test of it. With a bring-up module holding `pclk` and `ipclk` of + USI2 and USI3 on, the same build and switches gave 300 of 300 on the + evening of 11-12, where the two boots before had stopped at the 70th + and the 41st sleep; at one hang in about 55 sleeps that would come up + by chance about once in 240 tries. With only the two + PCLK gates critical in the clock driver and no module, 226 sleeps + woke and the 227th hung one callback further on: USI2's resume + returned, and the start of USI3's (111e00c0) is the last event. Its + bus clock was on, so what its resume switched was the function clock. + + The cause, shown awake (11-13). A bring-up module + (`tegu-hsi0-usi-stress`) repeats the driver's sequence on USI3 from a + thread on CPU 7: clocks on, `USI_CON` read and written back, 1 us, + clocks off. On the PCLK-only kernel, so with the function and source + gates switched: with no wait it hung the SoC within its first 1,000 + rounds, twice of twice, until the cluster watchdog reset the palmtop + (`APC Watchdog Early`); with 10 us between the clocks and the access, + 2,000,000 rounds passed; with the clocks switched and no register + touched, 2,000,000 passed. So a register access to a USI in the first + microseconds after its gates are opened never returns, whichever of + its clocks was closed, and that is what the noirq resume does. It is + not the wake path as such: the SPI controllers' runtime resume opens + the same gates and writes its registers right after, so the same + thing could happen awake. + + The fix is [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247): the six gates (PCLK, IP and + SOURCE of USI2 and USI3) critical, so nothing closes them. On the + published tip with only that change the module ran 2,000,000 rounds + on each USI with the gates never closed, and 300 of 300 sleeps of 60 s + woke on that build after its ten attended ones (11-13), where the same + series without it had stopped at the 70th, the 41st and, with the bus + clocks alone, the 227th. Touch and a GNSS session work on it (#247's + thread). + + How it was caught, and what the counts say. The maintainer's switches + (kernel #224): `pm_print_times` and `pm_debug_messages` name each + callback on the console; `kernel.panic_on_rcu_stall=1` makes the stall + a panic; the tracing build dumps each core's last events in the panic + (`PMDEBUG=1` and `TRACE=1` in `sleep-series.sh`). With all of them on, + 701 sleeps woke and none hung (93 on the evening of 11-10, 313 on + 11-11, 295 in the night after; the first 406 are the count #224 merged + on). With `pm_print_times` and the function tracer off and the stall + panic and the trace events kept (`TRACE=2 PMPRINT=0`), it came back at + the 70th sleep of one boot and the 41st of the next. Together with + two in about 310 before the switches, that says the heavy switches + hide it, most likely by the microseconds the function tracer puts + between the gate's write and the register read. + + What the stall panic does and doesn't do here: it gets the console and + the trace written, and then its own restart hangs ("Rebooting in 10 + seconds.." is the last line), so the palmtop still waits for the + buttons, and the reset's bit errors make pstore reject the console's + header. The text stays in the reserved region and can be read raw on + the next boot before the new console overwrites it (the note below). + + Next: #247's merge (its checks are done: the 300 below, touch and GNSS + on that build), then the same count on a release kernel that carries it. 2. The sleep hook ([omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591)): merged on 2026-11-10 as `omamobile-device-tegu` 0.2.11; its series is the 40 of 40 in the notes below. @@ -4950,3 +5024,107 @@ The long sleep of 11-10, nine and a half hours on this tree, ended at the first **The night of 2026-11-11 to 11-12: 295 of 295 (21:35 to 04:50 MST).** The tracing build again (`7.3.0-rc6-omamobile-push1-dbg-gff4fdf58035e`, build #5, a fresh boot), the script in the form that ran the 313 (`N=295 S=60 PMDEBUG=1 TRACE=1`), the modem boot unit masked for the boot, on the charger, Wi-Fi kept. The owner said yes to the night before he went to sleep, and slept through it. Neither part was new, and each had had its attended run: the build on the evening of 11-10, the owner at the palmtop for the first run; the script in this form at about 11:25 on 11-11, when the owner watched a few of its cycles (the screen coming on at the lock screen, going dark for about a minute and coming on again), in the run of 313 that had begun with nobody there. Every sleep lasted 58.7 to 60.0 s by the kernel log and was woken by the alarm (`pm_wakeup_irq` 53), with no UFS line, no ACPM timeout and no oops. The boot's kernel journal agrees with the log: 295 `PM: suspend entry`, 295 `PM: suspend exit`, 295 `PM: Triggering wakeup from IRQ 53` and none from another interrupt, no `link startup failed`, `Host init failed`, `I/O error` or `ufshcd_err_handler` line, no RCU stall, no panic; `suspend_stats` 295 and 0. Four cycles logged an RTC difference that doesn't fit (cycle 48 21 s short, 91 6 s long, 148 and 181 39 s long), the old reads of "What is left" item 6. At the end the "restored:" lines read as before (the switches off, the tracer `nop`, the watchdog back at 10 s). The daily kernel went back at 04:53 and the modem was up and registered at 04:54. With it the count with the PM switches on is 701 sleeps woken by the alarm and none hung. Item 1 has what that does and doesn't say. + +**The no-wake caught with a console: the 70th sleep of a boot stops at the start of the noirq resume (2026-11-12, 07:38 to 09:31 MST).** The build with the S2MPG14 RTC wait (`7.3.0-rc6-omamobile-push1-dbg-gc63fa9271bb8`, build #6: the tracing build's sources plus kernel #245's change), the modem boot unit masked, on the charger, Wi-Fi kept. `sleep-series.sh` with `PMDEBUG=1 TRACE=2 PMPRINT=0`: the stall panic and the trace events (scheduling, interrupts, IPIs, idle, timers, the suspend phases) on, the function tracer and `pm_print_times` off. + +- **Attended first.** Ten sleeps of 60 s from 07:38 to 07:52 with the owner at the palmtop watching the screen cycle: 10 of 10 woken by the alarm, 59.3 to 60.3 s each, no UFS line, the "restored:" lines as before. +- **Then 300 with his yes, from 07:53.** 59 woke. The 60th was entered at 09:18:40 (`PM: suspend entry` is the journal's last line) and the palmtop never came back on USB. The owner pressed Power once at about 09:29, which did nothing, and reset it with Power + Volume Up. So 69 sleeps woke on that boot and the 70th hung. All 69 RTC differences were 62 or 63 s ("What is left" item 6). +- **The console was there, but not in pstore.** The next boot logged `ramoops: found existing invalid buffer, size 2029723, start 2029979` and `/sys/fs/pstore` was empty: one flipped bit in the console's header (the start is 256 above the size). The text itself was still in the reserved region. `bringup/ramoops-raw-read.py` copied the region half a minute into the new boot, when the new console had overwritten only its first 86 KB; the old console's last 1.9 MB were intact apart from the reset's bit errors, which are all through it, as on 11-10. What follows is read through that damage (`bringup/console-fuzzy-find.py` finds the lines; the trace lines were read by eye), so the lines are reconstructions. +- **The kernel's own account** (kernel time, which runs through the sleep on this tree, see below): + + ``` + [ 6287.9 ] PM: suspend entry (s2idle) + [ 6288.5 ] samsung-pinctrl ...: wake enabled for irq ... the suspend's last lines + [ 6412.8 ] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: + [ 6412.8 ] rcu: 7-... (detected by 5, t=5252 jiffies, g=80105) + [ 6412.8 ] Sending NMI from CPU 5 to CPUs 7: + [ 6423.05] ... these CPUs still have not responded to the NMI + [ 6423.05] rcu: Possible timer handling issue on cpu=7 timer-softirq=37780 + [ 6423.06] Kernel panic - not syncing: RCU Stall from the tick on an idle CPU + [ 6423.06] SMP: stopping secondary CPUs + [ 6424.15] SMP: failed to stop secondary CPUs + [ 6424.6 ] Dumping ftrace buffer: about 5,700 lines + [ 6424.79] Rebooting in 10 seconds.. + ``` + + And no reboot: the palmtop stayed dark for the ten minutes until the buttons. The "NMI" is an ordinary IPI on this build (pseudo-NMI is compiled in and off). +- **The trace places it.** Without idle states `cpuidle_idle_call()` takes the plain idle path in s2idle, the tick isn't frozen, and the trace clock runs through the sleep. The sleep begins at 6288.6 and the wake comes at 6347.148, 58.6 s later: the alarm, on time. At the wake the suspend task, `systemd-sleep` (5643), is on CPU 7 and does this, all within 150 us: + + ``` + ipi_send_cpu: cpu=4 callsite=wake_up_if_idle and the other idle CPUs: s2idle's kick out of idle + suspend_resume: machine_suspend[1] end + suspend_resume: dpm_resume_noirq[16] begin + ipi_send_cpu: ... callsite=ttwu_queue_wakelist a worker woken, for the callbacks that run in workers + ``` + + That is the last event from CPU 7 that can be read. The other seven CPUs go on idling and taking their ticks for the next 76 s, CPUs 0 and 1 up to the panic. No resume line reached the console; in a good cycle on this boot the first one is the watchdog's, from its resume. +- **What it says.** The alarm woke the palmtop. The suspend task left s2idle on CPU 7, began the noirq resume of devices, and CPU 7 never took another tick or answered an IPI: a CPU with its interrupts masked, or below the kernel, in the noirq resume phase. "What is left" item 1 lists the phase's 17 callbacks, from a good cycle of the night before, when `pm_print_times` was on. +- **What it doesn't say.** Which callback, or whether it is one: the device callback events weren't on. About a third of the dump's lines are too damaged to parse, so "last" means the last readable, and a `dpm_resume_noirq` end could have been lost rather than missing. Why the panic's restart hung. Whether the 22nd sleep of 11-09 was this. +- **The raw copy** holds the bootloader's log too and stays off the repository. + +**The second catch names the callback: the noirq resume of the touchscreen's USI, 111d00c0 (2026-11-12, 09:54 to 10:54 MST, read at 16:15).** The same build and switches, with `power:device_pm_callback_start` and `_end` among the trace events (the script as it is in the tree now). + +- **Attended first.** Ten sleeps from 09:54 to 10:09 with the owner watching: 10 of 10. +- **Then 300 with his yes, from 10:09:58.** 30 woke, all 30 RTC differences 62 s. The 31st, the 41st of the boot, was entered just after 10:53:46 and never came back. The owner was out; at about 16:13 he pressed Power once (nothing) and reset the palmtop. +- **The console** was read raw from the region again (pstore empty, `found existing invalid buffer`). The damage is heavier than in the morning: the panic's lines can't be found by pattern, only `Rebooting in 10 seconds..` at 3825.55 and the trace dump before it. The events below are read by eye. +- **The trace.** The sleep is entered at 3734.58 and the wake comes at 3793.13, 58.6 s later. The suspend task (`systemd-sleep`, 34805) is on CPU 7 again, and within a few milliseconds does this: + + ``` + ipi_send_cpu ... callsite=wake_up_if_idle the idle CPUs, one each + suspend_resume: machine_suspend[3] end + suspend_resume: dpm_resume_noirq[16] begin + ipi_send_cpu ... callsite=ttwu_queue_wakelist + device_pm_callback_start: simple-pm-bus soc, parent: platform, noirq driver [resume] + device_pm_callback_end: simple-pm-bus soc + device_pm_callback_start: exynos-usi 108b00c0.usi, parent: soc, noirq driver [resume] + device_pm_callback_end: exynos-usi 108b00c0.usi, err=0 + device_pm_callback_start: exynos-usi 108d00c0.usi, parent: soc, noirq driver [resume] + device_pm_c... the rest is lost; by its place, 108d00c0's end + ipi_send_cpu: cpu=1 callsite=ttwu_queue_wakelist + device_pm_callback_start: exynos-usi 10c900c0.usi, parent: soc, noirq driver [resume] + device_pm_callback_end: exynos-usi 10c900c0.usi, err=0 + device_pm_callback_start: exynos-usi 111d00c0.usi, parent: soc, noirq driver [resume] + ``` + + That start is CPU 7's last event. It has no end, and CPU 7 has nothing more in the 32 s of trace that follow, while the other seven CPUs idle, tick and run workers up to the dump. In the last line two digits of the address are damaged (`?00d00c0`); it is taken as 111d00c0 because 108b00c0, 108d00c0 and 10c900c0 come just before it and 111d00c0 is next in the phase's order ("What is left" item 1). +- **What 111d00c0 is.** `usi2_hsi0`, the USI of the touchscreen's SPI, the first of three in the HSI0 block. Its noirq resume (`exynos_usi_configure()`) writes one sysreg field, turns `pclk` and `ipclk` on, reads and writes `USI_CON`, waits 1 us and turns the clocks off. The three USIs before it are in PERIC0 and PERIC1, whose USI gates `clk-zumapro.c` registers read-only, so their callbacks switch nothing; HSI0's USI2 and USI3 gates are writable and in manual mode, and at idle nothing holds them (`clk_enable_count` 0 for the four on the next boot). So it is the first callback of the phase that switches a manual-mode bus clock gate on and reads the block at once: what kernel #172 removed for the PERIC0 GPIO gate, where the SoC hung with nothing in the log. +- **The reading, to be tested:** the register access hangs the bus when it meets the gate's switch, and CPU 7 sits in that access. It fits the tick and the IPIs going unanswered, and it fits the 701 clean sleeps with the function tracer on, which puts microseconds between the gate write and the read. + +**With those clocks held on, 300 of 300 (2026-11-12, 16:20 to 23:40 MST).** The same build (`push1-dbg-gc63fa9271bb8`, build #6), the same switches (`N=300 S=60 PMDEBUG=1 TRACE=2 PMPRINT=0`), a fresh boot after the reset, the modem off, and one thing added: a bring-up module, loaded by hand before the run, that takes a reference on `pclk` and `ipclk` of the USIs at 111d00c0 and 111e00c0 (`of_clk_get_by_name()` and `clk_prepare_enable()` on each, released at unload), so the clock framework never switches those gates. `clk_enable_count` read 0 for the four gates before it loaded and 1 after, and still 1 at the end of the run. + +- All 300 woke on the alarm: 59.2 to 60.3 s by the kernel log, `pm_wakeup_irq` 53 each, no UFS line, no stall, `suspend_stats` 300 and 0, every RTC difference 62 s. +- The two runs without the module, on this build with these switches, stopped at the 70th and the 41st sleep of their boots. If a sleep hung once in about 55, as those two say, 300 clean sleeps in a row would come up about once in 240 tries. Two events are a thin base for a rate, so this is strong support for the reading, not proof of it. +- It ran with nobody at the palmtop and with a module that had not had an attended run first, on the owner's standing instruction for his own palmtop; said on #659 before it started. It doesn't count as a run under the attended-first rule. +- It doesn't tell `pclk` from `ipclk`, or USI2 from USI3: all four were held. + + +**With only the bus clocks kept on, the 227th sleep hangs one callback further on (the night of 2026-11-12 to 11-13, 23:44 to 05:15 MST).** The same sources with one change in `clk-zumapro.c`: the PCLK gates of USI2 and USI3 `CLK_IS_CRITICAL` (`7.3.0-rc6-omamobile-push1-dbg-g9af120756764`, build #8), no module, the same switches (`N=300 S=60 PMDEBUG=1 TRACE=2 PMPRINT=0`), the modem off. `clk_enable_count` read 1 for the two PCLK gates and 0 for the IP and SOURCE gates, which the resume still switched. + +- 226 sleeps woke, each 59.0 to 60.3 s and on `pm_wakeup_irq` 53, all 226 RTC differences 62 or 63 s. The 227th was entered after 05:15:22 and never came back; the owner reset the palmtop at 06:39. +- The console was read raw from the region again. The trace: the wake comes on time, the suspend task on CPU 7 logs `dpm_resume_noirq[16] begin` and runs the bus, 108b00c0, 108d00c0 and 10c900c0 as before, then + + ``` + device_pm_callback_start: exynos-usi 111d00c0.usi, parent: soc, noirq driver [resume] + device_pm_callback_end: exynos-usi 111d00c0.usi, err=0 + device_pm_callback_start: exynos-usi 111e00c0.usi, parent: soc, noirq driver [resume] + ``` + + and nothing more from that task to the end of the dump. The lines are read through the reset's bit errors (the last address reads `?11e00b8`; it is taken as 111e00c0 by its place). +- So USI2 got through with its bus clock held, and USI3 hung with its bus clock held too. What USI3's resume still switched was its function clock and that clock's source gate. +- It ran with nobody at the palmtop, on a kernel that had had no attended run, on the owner's standing instruction; it doesn't count under the attended-first rule. + +**The cause, awake and in under a second: a register access right after the gate opens (2026-11-13, 06:43 to 06:50 MST).** A bring-up module, `tegu-hsi0-usi-stress`, takes one USI's `pclk` and `ipclk`, maps its registers and, from a thread bound to CPU 7, repeats what `exynos_usi_enable()` does: `clk_prepare_enable()` on both, `USI_CON` read and written back with the reset bit clear, 1 us, both clocks off. It counts the rounds after which the function gate reads off. On the PCLK-only kernel, on USI3 (nothing else uses it with the modem off): + +| variant | result | +| --- | --- | +| as the driver does it, no wait | the start line and nothing after: no report at 1,000 rounds, the palmtop silent on USB, and within about 40 s back from a reset by the cluster watchdog (the bootloader's `APC Watchdog Early`, `CLUSTER0_NONCPU_WDTRESET`). Twice of twice | +| the clocks switched, no register touched | 2,000,000 rounds, the gate off after each, no hang | +| 10 us between the clocks and the access | 2,000,000 rounds, the gate off after each, no hang | + +- So the hang is the access, not the switch, and it is a matter of microseconds: a USI register read in the first moments after its gates are opened never returns. That is why the function tracer hid it for 701 sleeps, and why it showed about once in 55 resumes and not every time: the resume does it once per USI, the module a thousand times in a few milliseconds. +- Awake the watchdog ends it, so the test needs no buttons. In a sleep nothing does, and the palmtop waits. +- **The fix** is [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247): PCLK, IP and SOURCE of USI2 and USI3 critical. On the published tip `2db2bc35b` with only that change (`7.3.0-rc6-omamobile-push1-gd74c5bd9dd0e`, build #11, defconfig and main's `omamobile.config`, no debug switches) the six gates read `clk_enable_count` 1 or 2 and `CLK_IS_CRITICAL`, and the module with no wait ran 2,000,000 rounds on USI3 and 2,000,000 on USI2, the function gate off after none of them. +- **Sleeps on that build.** Ten attended, 06:54 to 07:07, the owner watching the screen go dark and come back: 10 of 10, 59.3 to 60.3 s, `pm_wakeup_irq` 53 each, no UFS line. Two RTC differences were off (41 s and 67 s; the others 62 s): this build is the published tip, which doesn't have kernel #245's RTC wait. +- **Then 300 of 300 with his yes**, 07:08 to 14:27 MST: 59.2 to 60.3 s by the kernel log, each woken by the alarm, no UFS line, link failure or oops; `suspend_stats` 310 and 0 for the boot. The script's RTC difference read 62 or 63 s in 224 cycles and 8 to 108 s in 76, the reading #245 fixed; the kernel log's times are the count. +- **Touch and GNSS on that build**, with the owner at the palmtop, for #247's review: touch in the session on the boot the 300 ran on (the touch interrupt count 62 to 292 while he unlocked and opened apps, no SPI or `synaptics-tcm` error line); after a clean reboot with the modem up, one GNSS session of 150 s with 104 of its 109 GGA sentences fixed, and no `-110`, `I/O Error` or `failed to transfer` line in the boot. +- Not done: the module on the published tip as it is, without the change (the hangs above are on the tracing build of the same sources with the PCLK gates held). diff --git a/device/tegu/bringup/hsi0-usi/Makefile b/device/tegu/bringup/hsi0-usi/Makefile new file mode 100644 index 000000000..5af9d2ab9 --- /dev/null +++ b/device/tegu/bringup/hsi0-usi/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-m += tegu-hsi0-usi-clk-hold.o tegu-hsi0-usi-stress.o diff --git a/device/tegu/bringup/hsi0-usi/README.md b/device/tegu/bringup/hsi0-usi/README.md new file mode 100644 index 000000000..e8c47effd --- /dev/null +++ b/device/tegu/bringup/hsi0-usi/README.md @@ -0,0 +1,43 @@ + +# HSI0 USI bring-up modules (Pixel 9a) + +The evidence for the sleep that never woke ([SLEEP-PLAN](../../SLEEP-PLAN.md) +S1, "What is left" item 1, and the notes of 2026-11-12 and 11-13): a register +access to a USI in the first microseconds after its clock gates are opened +never returns, and the noirq resume does exactly that. The fix is +[linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247), +which keeps the six gates of USI2 and USI3 on. Built out of tree against the +kernel under test: + + make -C O= ARCH=arm64 LLVM=1 M=$PWD modules + +Neither runs on a palmtop before an attended run with the owner's yes +(AGENTS.md, the 9a sleep entry). The first runs of both, on 2026-11-12 and +11-13, did not have one. + +- `tegu-hsi0-usi-clk-hold.c` takes a reference on `pclk` and `ipclk` of the + USIs at 111d00c0 and 111e00c0 (HSI0's USI2 and USI3) while loaded, so the + clock framework never closes their gates; `ip=0` holds `pclk` alone. It + reads and writes no register of its own. With it loaded, 300 of 300 sleeps + woke on the build that had hung at the 70th and the 41st + (`push1-dbg-gc63fa9271bb8`, 2026-11-12). With #247 in the kernel it changes + nothing. +- `tegu-hsi0-usi-stress.c` **hangs the SoC by design.** It repeats + `exynos_usi_enable()`'s sequence on one USI from a thread bound to a CPU: + both clocks on, `USI_CON` read and written back with the reset bit clear + (what the driver writes at every resume), 1 us, both clocks off. On the + build with only the PCLK gates critical (`push1-dbg-g9af120756764`), + `usi=3` with no wait hung within the first 1,000 rounds, twice of twice, + and the cluster watchdog reset the palmtop about 40 s later (systemd's + runtime watchdog was feeding it; without a fed watchdog it waits for the + buttons). `access=0` (gates switched, no register touched) and + `delay_us=10` (10 us between the clocks and the access) each ran + 2,000,000 rounds with no hang. On a kernel with #247 the gates never close + and 2,000,000 rounds passed on each USI. + + Run it only with the modem off (nothing else uses USI3 then), the + filesystems synced, a watchdog fed from userspace, and never with `usi=2` + while the touchscreen is in use: + + insmod tegu-hsi0-usi-stress.ko loops=2000000 report=1000 usi=3 cpu=7 \ + [access=0] [delay_us=10] diff --git a/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-clk-hold.c b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-clk-hold.c new file mode 100644 index 000000000..e6baf456e --- /dev/null +++ b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-clk-hold.c @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Bring-up only (SLEEP-PLAN item 1): hold the bus and function clocks of + * the HSI0 block's USI2 and USI3 on while loaded, so that neither the USI + * driver's resume nor the SPI controllers' runtime PM gates them. They + * are manual-mode gates, as the PERIC0 GPIO bus clock was. + * ip=0 holds the bus clocks (pclk) alone. + */ +#include +#include +#include +#include + +static bool ip = true; +module_param(ip, bool, 0444); +MODULE_PARM_DESC(ip, "Also hold the function clocks (ipclk), default on"); + +static const char * const nodes[] = { "/soc/usi@111d00c0", "/soc/usi@111e00c0" }; +static const char * const names[] = { "pclk", "ipclk" }; +static struct clk *held[4]; +static int nheld; + +static void hold_release(void) +{ + while (nheld) { + nheld--; + clk_disable_unprepare(held[nheld]); + clk_put(held[nheld]); + } +} + +static int __init hold_init(void) +{ + int i, j, ret; + + for (i = 0; i < ARRAY_SIZE(nodes); i++) { + struct device_node *np = of_find_node_by_path(nodes[i]); + + if (!np) { + hold_release(); + return -ENODEV; + } + for (j = 0; j < (ip ? 2 : 1); j++) { + struct clk *c = of_clk_get_by_name(np, names[j]); + + if (IS_ERR(c)) { + of_node_put(np); + hold_release(); + return PTR_ERR(c); + } + ret = clk_prepare_enable(c); + if (ret) { + clk_put(c); + of_node_put(np); + hold_release(); + return ret; + } + held[nheld++] = c; + pr_info("tegu-hsi0-usi-clk-hold: %s held on\n", __clk_get_name(c)); + } + of_node_put(np); + } + return 0; +} + +static void __exit hold_exit(void) +{ + hold_release(); +} + +module_init(hold_init); +module_exit(hold_exit); +MODULE_DESCRIPTION("Bring-up: hold the HSI0 USI2 and USI3 clocks on"); +MODULE_LICENSE("GPL"); diff --git a/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c new file mode 100644 index 000000000..4dd9fb367 --- /dev/null +++ b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * THIS MODULE HANGS THE SOC BY DESIGN on a kernel whose HSI0 USI clock + * gates are switched (without linux-omarchy-mobile#247): that is what it is + * for. Run it only as README describes, attended, with the modem off. + * + * Bring-up only (SLEEP-PLAN item 1): do what exynos_usi_enable() does in the + * noirq resume, over and over, awake: turn pclk and ipclk of one HSI0 USI + * on, read USI_CON, write it back with the reset bit clear (as the driver + * does at every resume), wait 1 us and turn the clocks off. To see whether + * the switch of a manual-mode gate followed at once by a register access + * hangs the SoC outside the wake path too. A hang ends in a reset by the + * cluster watchdog while systemd feeds it, and otherwise needs the buttons. + * insmod tegu-hsi0-usi-stress.ko loops=200000 usi=2 cpu=7 + * access=0 switches the clocks and touches no register; delay_us=N waits + * N us between turning the clocks on and the first access. + */ +#include +#include +#include +#include +#include +#include +#include +#include + +#define USI_CON 0x04 +#define USI_CON_RESET BIT(0) + +static unsigned long loops = 200000; +module_param(loops, ulong, 0444); +static unsigned long report = 10000; +module_param(report, ulong, 0444); +static int usi = 2; +module_param(usi, int, 0444); +static int cpu = 7; +module_param(cpu, int, 0444); +static bool access = true; +module_param(access, bool, 0444); +static unsigned int delay_us; +module_param(delay_us, uint, 0444); + +static struct clk *pclk, *ipclk; +static void __iomem *regs; +static struct task_struct *task; + +static int stress_fn(void *unused) +{ + unsigned long i, off = 0; + u32 val = 0; + + pr_info("tegu-hsi0-usi-stress: usi%d, %lu loops on cpu %d, access %d, delay %u us, ipclk %s, pclk %s\n", + usi, loops, raw_smp_processor_id(), access, delay_us, + __clk_is_enabled(ipclk) ? "on" : "off", + __clk_is_enabled(pclk) ? "on" : "off"); + for (i = 1; i <= loops && !kthread_should_stop(); i++) { + if (clk_prepare_enable(pclk)) + break; + if (clk_prepare_enable(ipclk)) { + clk_disable_unprepare(pclk); + break; + } + if (delay_us) + udelay(delay_us); + if (access) { + val = readl(regs + USI_CON); + val &= ~USI_CON_RESET; + writel(val, regs + USI_CON); + } + udelay(1); + clk_disable_unprepare(ipclk); + clk_disable_unprepare(pclk); + if (!__clk_is_enabled(ipclk)) + off++; + if (!(i % report)) { + pr_info("tegu-hsi0-usi-stress: %lu loops, ipclk gate off after %lu of them, USI_CON 0x%x\n", + i, off, val); + cond_resched(); + } + } + pr_info("tegu-hsi0-usi-stress: done after %lu loops, ipclk gate off after %lu of them\n", + i - 1, off); + while (!kthread_should_stop()) + schedule_timeout_interruptible(HZ); + return 0; +} + +static int __init stress_init(void) +{ + static const char * const nodes[] = { "/soc/usi@111d00c0", "/soc/usi@111e00c0" }; + struct device_node *np; + int ret = -ENODEV; + + if (usi != 2 && usi != 3) + return -EINVAL; + np = of_find_node_by_path(nodes[usi - 2]); + if (!np) + return -ENODEV; + pclk = of_clk_get_by_name(np, "pclk"); + ipclk = of_clk_get_by_name(np, "ipclk"); + regs = of_iomap(np, 0); + of_node_put(np); + if (IS_ERR(pclk) || IS_ERR(ipclk) || !regs) + goto err; + task = kthread_create(stress_fn, NULL, "usi-stress"); + if (IS_ERR(task)) { + ret = PTR_ERR(task); + goto err; + } + if (cpu >= 0) + kthread_bind(task, cpu); + wake_up_process(task); + return 0; +err: + if (regs) + iounmap(regs); + if (!IS_ERR_OR_NULL(ipclk)) + clk_put(ipclk); + if (!IS_ERR_OR_NULL(pclk)) + clk_put(pclk); + return ret; +} + +static void __exit stress_exit(void) +{ + kthread_stop(task); + iounmap(regs); + clk_put(ipclk); + clk_put(pclk); +} + +module_init(stress_init); +module_exit(stress_exit); +MODULE_DESCRIPTION("Bring-up: switch an HSI0 USI's clocks around a register access, awake"); +MODULE_LICENSE("GPL"); diff --git a/device/tegu/bringup/sleep-series.sh b/device/tegu/bringup/sleep-series.sh index d7a61c2d1..7be9a526d 100644 --- a/device/tegu/bringup/sleep-series.sh +++ b/device/tegu/bringup/sleep-series.sh @@ -8,7 +8,7 @@ # errors, and the link-wait lines (a slow wake rather than a dead one). N=${N:-300}; S=${S:-60}; W=${W:-20}; L=${L:-/var/log/omamobile/sleep-linkwait-20261107.log}; U=/sys/devices/platform/soc/13200000.ufs mkdir -p /var/log/omamobile -DBG=0; { [ "${PMDEBUG:-0}" = 1 ] || [ "${TRACE:-0}" = 1 ]; } && DBG=1 +DBG=0; { [ "${PMDEBUG:-0}" = 1 ] || [ "${TRACE:-0}" = 1 ] || [ "${TRACE:-0}" = 2 ]; } && DBG=1 # Refusals come first, before anything is changed. modem_up() { systemctl is-active -q omamobile-modem omamobile-tegu-modem-boot 2>/dev/null || [ -d /sys/module/cpif ]; } modem_up && { echo "the modem is up: stop" >> $L; exit 1; } @@ -37,14 +37,14 @@ st1() { for f in /sys/devices/system/cpu/cpu*/cpuidle/state1/disable; do [ -e $f # always-on is cut only by the system suspend, not by every idle period. rpm_before=$(cat $U/rpm_lvl) # What the debug switches change is read here, before anything is set, and restore() puts all of it back. -T=/sys/kernel/tracing; WD=org.freedesktop.systemd1; EV="sched/sched_switch irq/irq_handler_entry irq/irq_handler_exit ipi power/cpu_idle power/suspend_resume timer/hrtimer_expire_entry" +T=/sys/kernel/tracing; WD=org.freedesktop.systemd1; EV="sched/sched_switch irq/irq_handler_entry irq/irq_handler_exit ipi power/cpu_idle power/suspend_resume power/device_pm_callback_start power/device_pm_callback_end timer/hrtimer_expire_entry" wd() { busctl "$1"-property $WD /org/freedesktop/systemd1 $WD.Manager RuntimeWatchdogUSec ${2:+t $2}; } if [ $DBG = 1 ]; then ppt_before=$(cat /sys/power/pm_print_times); pdm_before=$(cat /sys/power/pm_debug_messages) kmsg_before=$(cat /proc/sys/kernel/printk_devkmsg) stall_before=$(cat /proc/sys/kernel/panic_on_rcu_stall); wd_before=$(wd get | awk '{print $2}') fi -if [ "${TRACE:-0}" = 1 ] && [ -d $T ]; then +if { [ "${TRACE:-0}" = 1 ] || [ "${TRACE:-0}" = 2 ]; } && [ -d $T ]; then mountpoint -q $T || mount -t tracefs nodev $T dump_before=$(cat /proc/sys/kernel/ftrace_dump_on_oops); tr_before=$(cat $T/current_tracer); on_before=$(cat $T/tracing_on) # Before tracing is first used the size reads "7 (expanded: 1408)": 7 KB now, 1408 on first use. Writing 7 @@ -83,6 +83,10 @@ echo N > /sys/module/printk/parameters/console_suspend # tracing in small per-CPU buffers that the panic dumps to the console. The dump has to fit the 2 MB ramoops # console together with the sleep's own callback lines: it prints oldest first, and 24 KB per CPU is about 900 # calls a core and under 1 MB of text for eight (measure it with an awake sysrq-c before a series). +# TRACE=2 records the events alone, without the function tracer, and PMPRINT=0 (with PMDEBUG=1) leaves +# pm_print_times and pm_debug_messages off: with both heavy switches on the no-wake didn't come back in 701 +# sleeps, and with them off (PMDEBUG=1 TRACE=2 PMPRINT=0) it came back at the 70th and the 41st (2026-11-12). +# The events include the device PM callbacks' start and end, which name the callback a core stops in. # Either one first fills the kernel's log ring: three test crashes on the tracing build (sysrq-c, awake) lost # everything after record 4096 since boot, the trace dump's tail and the "Rebooting" line with it, when the # ring's first wrap fell inside the panic; with the ring filled beforehand the dump was whole. @@ -91,12 +95,17 @@ echo N > /sys/module/printk/parameters/console_suspend # the run: still a net under an awake hang. WDSEC=0 turns it off. if [ $DBG = 1 ]; then echo on > /proc/sys/kernel/printk_devkmsg 2>/dev/null; for i in $(seq 1 5000); do echo "ring fill $i" > /dev/kmsg; done - echo 1 > /sys/power/pm_print_times; echo 1 > /sys/power/pm_debug_messages + # PMPRINT=0 leaves the callback names off: about 64 KB of console text a cycle changes the timing of a sleep, + # and the no-wake didn't come back in 701 sleeps with them on. + if [ "${PMPRINT:-1}" = 1 ]; then echo 1 > /sys/power/pm_print_times; echo 1 > /sys/power/pm_debug_messages; fi echo 1 > /proc/sys/kernel/panic_on_rcu_stall wd set $(( ${WDSEC:-60} * 1000000 )) fi if [ -n "$tr_before" ]; then - echo 0 > $T/tracing_on; echo ${TRACE_KB:-24} > $T/buffer_size_kb; echo function > $T/current_tracer + # TRACE=2 records the events only (interrupts, timers, scheduling, idle), without the function tracer: + # far less work per call, and the dump still says what woke a core last and what ran on it. + echo 0 > $T/tracing_on; echo ${TRACE_KB:-24} > $T/buffer_size_kb + if [ "${TRACE:-0}" = 2 ]; then echo nop > $T/current_tracer; else echo function > $T/current_tracer; fi for e in $EV; do echo 1 > $T/events/$e/enable; done echo 1 > $T/tracing_on; echo 1 > /proc/sys/kernel/ftrace_dump_on_oops fi From 3ba8c01a1c3a9a95c741b8d2bd34dd16faf1ed47 Mon Sep 17 00:00:00 2001 From: W4HO <87742609+W4HO-ham@users.noreply.github.com> Date: Sun, 15 Nov 2026 12:21:15 -0644 Subject: [PATCH 2/3] device/tegu SLEEP-PLAN, hsi0-usi: the stress module refuses anything but a 9a and needs usi=; status words that hold after #247 The stress module now refuses to load unless the machine is google,zumapro-tegu, and usi= has no default, so a bare insmod never switches the touchscreen's USI and no other board runs it by luck of a node path. The status line and item 1's next step no longer depend on #247 being under review. --- device/tegu/SLEEP-PLAN.md | 6 +++--- device/tegu/bringup/hsi0-usi/README.md | 9 ++++++--- .../tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c | 15 ++++++++++++--- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/device/tegu/SLEEP-PLAN.md b/device/tegu/SLEEP-PLAN.md index 19416c1a2..22e86cb23 100644 --- a/device/tegu/SLEEP-PLAN.md +++ b/device/tegu/SLEEP-PLAN.md @@ -1,7 +1,7 @@ # Pixel 9a sleep: why s2idle drains so much -Status (2026-11-15): Three causes are known: two are fixed, one in the kernel and one by a sleep hook, and the third has a kernel fix under review. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224), merged on 2026-11-12 (00:41Z), re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. The third, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (the 22nd of a run on 2026-11-09, with no console; the 213th of a night on 11-10; the 70th and the 41st of two boots on 11-12). The three consoles agree: CPU 7, which runs the suspend, takes no tick and answers no IPI from the noirq resume of devices on, and the console with the device callback events on stops at the start of the resume of an HSI0 USI (the touchscreen's, 111d00c0). The cause was shown awake on 11-13: a register access to one of those USIs in the first microseconds after its clock gates are opened never returns, and a test module that does it hangs the SoC within 1,000 rounds. [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps the six gates on; on the published tip with only that change, 300 of 300 sleeps woke (11-13), and touch and GNSS work. The PM debug switches hid it: 701 sleeps with them on and none hung (11-10 to 11-12), most likely through the microseconds the function tracer puts between the gate's write and the register read. The record is in "What is left" item 1 and the notes of 11-12 and 11-13. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; a kernel from before it has no alarm (`rtcwake` fails and a sleep ends only on a key), so check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. Evidence: 300 of 300 RTC-woken 60 s sleeps twice (#96 WFI-only, #98 C2 at runtime, 2026-11-07/08), 15 of 15 wakes on the everyday path, the shell's auto-sleep behind the lock, twelve of them half-hour sleeps (#99, 2026-11-08), 12 of 12 half-hour auto-sleeps on #100 (11-09), on the published tree with #224 and the PMIC interrupt 1,209 wakes and four no-wakes (11-09 to 11-12), and with #247 added 310 of 310 (11-13). The attended-first rule (S1, AGENTS.md) stands; the overnight run of 11-10, the runs of 11-11 and two runs of 11-12 did not keep it (the run table and its header say how each ran). +Status (2026-11-15): Three causes are known: two are fixed, one in the kernel and one by a sleep hook, and the third has a kernel fix, [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247), not yet in a release kernel. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224), merged on 2026-11-12 (00:41Z), re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. The third, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (the 22nd of a run on 2026-11-09, with no console; the 213th of a night on 11-10; the 70th and the 41st of two boots on 11-12). The three consoles agree: CPU 7, which runs the suspend, takes no tick and answers no IPI from the noirq resume of devices on, and the console with the device callback events on stops at the start of the resume of an HSI0 USI (the touchscreen's, 111d00c0). The cause was shown awake on 11-13: a register access to one of those USIs in the first microseconds after its clock gates are opened never returns, and a test module that does it hangs the SoC within 1,000 rounds. [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps the six gates on; on the published tip with only that change, 300 of 300 sleeps woke (11-13), and touch and GNSS work. The PM debug switches hid it: 701 sleeps with them on and none hung (11-10 to 11-12), most likely through the microseconds the function tracer puts between the gate's write and the register read. The record is in "What is left" item 1 and the notes of 11-12 and 11-13. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; a kernel from before it has no alarm (`rtcwake` fails and a sleep ends only on a key), so check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. Evidence: 300 of 300 RTC-woken 60 s sleeps twice (#96 WFI-only, #98 C2 at runtime, 2026-11-07/08), 15 of 15 wakes on the everyday path, the shell's auto-sleep behind the lock, twelve of them half-hour sleeps (#99, 2026-11-08), 12 of 12 half-hour auto-sleeps on #100 (11-09), on the published tree with #224 and the PMIC interrupt 1,209 wakes and four no-wakes (11-09 to 11-12), and with #247 added 310 of 310 (11-13). The attended-first rule (S1, AGENTS.md) stands; the overnight run of 11-10, the runs of 11-11 and two runs of 11-12 did not keep it (the run table and its header say how each ran). S1 is the condensed account: the two causes, the fix, a run table, what is left and what was ruled out. The dated working notes that led there follow it, oldest first. @@ -218,8 +218,8 @@ missing pull, NFC polling (stopped by the hook throughout), the modem header. The text stays in the reserved region and can be read raw on the next boot before the new console overwrites it (the note below). - Next: #247's merge (its checks are done: the 300 below, touch and GNSS - on that build), then the same count on a release kernel that carries it. + Next: the same count on a release kernel that carries #247 (its checks + on the test build are done: the 300 below, touch and GNSS). 2. The sleep hook ([omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591)): merged on 2026-11-10 as `omamobile-device-tegu` 0.2.11; its series is the 40 of 40 in the notes below. diff --git a/device/tegu/bringup/hsi0-usi/README.md b/device/tegu/bringup/hsi0-usi/README.md index e8c47effd..e75ba55c9 100644 --- a/device/tegu/bringup/hsi0-usi/README.md +++ b/device/tegu/bringup/hsi0-usi/README.md @@ -35,9 +35,12 @@ Neither runs on a palmtop before an attended run with the owner's yes 2,000,000 rounds with no hang. On a kernel with #247 the gates never close and 2,000,000 rounds passed on each USI. - Run it only with the modem off (nothing else uses USI3 then), the - filesystems synced, a watchdog fed from userspace, and never with `usi=2` - while the touchscreen is in use: + It refuses to load on anything but a Pixel 9a (`google,zumapro-tegu`), + and `usi=` has no default: without `usi=2` or `usi=3` it refuses, so the + touchscreen's USI is never switched by accident. Run it only with the + modem off (nothing else uses USI3 then), the filesystems synced, a + watchdog fed from userspace, and never with `usi=2` while the touchscreen + is in use: insmod tegu-hsi0-usi-stress.ko loops=2000000 report=1000 usi=3 cpu=7 \ [access=0] [delay_us=10] diff --git a/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c index 4dd9fb367..a34b87174 100644 --- a/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c +++ b/device/tegu/bringup/hsi0-usi/tegu-hsi0-usi-stress.c @@ -11,7 +11,9 @@ * the switch of a manual-mode gate followed at once by a register access * hangs the SoC outside the wake path too. A hang ends in a reset by the * cluster watchdog while systemd feeds it, and otherwise needs the buttons. - * insmod tegu-hsi0-usi-stress.ko loops=200000 usi=2 cpu=7 + * insmod tegu-hsi0-usi-stress.ko loops=200000 usi=3 cpu=7 + * It refuses anything but a Pixel 9a (google,zumapro-tegu), and usi= (2 or + * 3) has no default: USI2 is the touchscreen's, so the choice is explicit. * access=0 switches the clocks and touches no register; delay_us=N waits * N us between turning the clocks on and the first access. */ @@ -31,8 +33,9 @@ static unsigned long loops = 200000; module_param(loops, ulong, 0444); static unsigned long report = 10000; module_param(report, ulong, 0444); -static int usi = 2; +static int usi; module_param(usi, int, 0444); +MODULE_PARM_DESC(usi, "the HSI0 USI to switch: 2 (the touchscreen's) or 3; required"); static int cpu = 7; module_param(cpu, int, 0444); static bool access = true; @@ -91,8 +94,14 @@ static int __init stress_init(void) struct device_node *np; int ret = -ENODEV; - if (usi != 2 && usi != 3) + if (!of_machine_is_compatible("google,zumapro-tegu")) { + pr_err("tegu-hsi0-usi-stress: not a Pixel 9a (google,zumapro-tegu): refusing\n"); + return -ENODEV; + } + if (usi != 2 && usi != 3) { + pr_err("tegu-hsi0-usi-stress: usi=2 or usi=3 is required\n"); return -EINVAL; + } np = of_find_node_by_path(nodes[usi - 2]); if (!np) return -ENODEV; From 8254e0df277720e91f5f15a6a6ae2cd61bdf3e13 Mon Sep 17 00:00:00 2001 From: W4HO <87742609+W4HO-ham@users.noreply.github.com> Date: Sun, 15 Nov 2026 13:40:23 -0644 Subject: [PATCH 3/3] AGENTS: resolve the merge with main properly The previous commit kept the merge's conflict markers in the 9a sleep entry by mistake. This keeps main's #809 note at the head of the entry and this branch's sentence on the no-wake's cause and #247. --- AGENTS.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5d53ab9ed..44f5549f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -189,11 +189,7 @@ Each entry gives the gotcha, then the reason or the fix. Details are in the link ### Suspend and modem - **On the 9a, NFC must not poll through s2idle.** A suspend with neard polling reset the palmtop (2026-10-27); with polling stopped, 35 s2idle cycles with RTC wakes that day all woke (`7.3.0-rc6-omamobile-tegu-s2mpg15-m1`). The cause isn't known: nxp-nci has no PM callbacks, but neither does the 7 Pro's st-nci, so the controller's interrupt or the I2C bus during the freeze are the next suspects. A sleep hook on the palmtop, `omamobile-tegu-nfc` (not in the repository yet; it comes with tegu's device package), stops neard's poll loop before a systemd sleep and restarts it after; a sleep written straight to `/sys/power/state` (`rtcwake -m mem`) skips the hook, so stop polling by hand first. Without [linux-omarchy-mobile#131](https://github.com/derekross/linux-omarchy-mobile/pull/131) the ST54L also doesn't answer on I2C after a wake (`Error -6 on I2C send`, also after the driver's retry), or after about 2-3 s of idle at any time, until neard powers it off and on ([#132](https://github.com/derekross/omarchy-mobile/issues/132)): nxp-nci never turned its NFC mode on, as ST's HAL does after init. With #131 (the results below on `core.c` blob `77fc35b61da5`, #131's first version; the merged version, `c96704ddd480`, re-run for six power-ons and polling at 3 s and 30 s; the release, `7.3.0-rc6-omamobile-tegu-s2mpg15-m1-g991042e2eff3-dirty`, is the same as without the fix), NFC polls after 3 s to 3 minutes of idle and polled again after two RTC-woken s2idle cycles (2026-10-29): one with a hook that stops polling and powers the adapter off, and one without that hook, for which it isn't recorded whether polling was stopped, so whether NFC comes back after a wake with the stop-poll hook alone rests on that one cycle and needs a re-run with the hook's log. No power-off hook is needed for NFC; the stop-poll rule above stands on its 35 of 35. [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" -<<<<<<< HEAD -- **On the 9a, a new sleep test runs attended first, and the direct freeze path is still not used.** Unattended sleep series are allowed since 2026-11-05 (#409) on the contributor's conditions and three of the maintainer's. The contributor's: on the charger with the cable in; the modem off for the boot; the script stops itself on any UFS error line or oops; a wake with dead storage (SSH still answers) gets `echo b > /proc/sysrq-trigger` remotely, and only a silent hang waits for the owner's buttons; the owner says yes before each run. (Their "slot B marked" condition is not part of it: marking B on the 9a stays on hold, see the boot chain notes above, so a series that resets the palmtop spends ABL tries and stays short.) The maintainer's: nothing new (a kernel, a module or a script) runs unattended before one attended run of it; the automatic sysrq recovery is capped per night, and the next boot's ext4 recovery is checked and recorded; and the recovery script is in the tree (`device/tegu/bringup/`, where the series scripts already are) before it runs unattended. The plan says which rule each step runs under, and the counts below are the early ones; [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) keeps the current tally. Four sleeps have never woken (not from the RTC alarm, not from the power key; Power + Volume Up to reset, nothing in pstore): two direct `rtcwake -m freeze` runs (2026-10-28, 2026-10-29), one 30-minute `systemctl suspend` with `tegu-mif-sleep` loaded (2026-10-30, cable out) and the sixth 30 s `systemctl suspend` cycle on 2026-10-30 (the daily boot, no module; it left the USB bus); 35 short and seven longer systemd sleeps woke (two 30-minute and four 240 s ones on 2026-10-29, a cable-in 30-minute one with the module on 2026-10-30; the afternoon's cable-out run with the module makes eight). No single factor explains it, so keep `power-ab` on `--suspend systemd` (the sleep hooks run that way), and don't load `tegu-mif-sleep` at boot until the hang is understood. The shell's auto-sleep (the `omamobile-sleep` toggle, systemd path, no module) woke every time while it was on, including seven sleeps of 11 s to 22 minutes in a day of use on 2026-11-01 (lengths from the RTC); the package ships it off, and the owner turned it back on on the palmtop that day, so keeping it on is the owner's call. Two causes are known. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224) re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. A third cause, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (2026-11-09 to 11-12). The consoles show the kernel alive and CPU 7, which runs the suspend, taking no tick and answering no IPI from the noirq resume of devices on, so a palmtop that stays silent after a sleep may be a running kernel with one core locked up, not a SoC that stayed down; neither the alarm nor the Power key gets through. The core is in a register access to an HSI0 USI (the touchscreen's or the GNSS SPI's) in the first microseconds after its clock gates are opened, which never returns: shown awake with a test module that hangs the SoC on purpose (`device/tegu/bringup/hsi0-usi/`, attended only). [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps those six gates on (300 of 300 on the published tip with it alone, 2026-11-13); until a kernel the palmtop runs carries it, the no-wake stays possible, about one in 300. The PM debug switches of `sleep-series.sh` hide it (701 sleeps with them on and none hung): to catch it use `PMDEBUG=1 TRACE=2 PMPRINT=0`. The stall panic writes the console but its own restart hangs, so the buttons are still needed, and pstore may reject the console's header; the text can be read raw from the ramoops region on the next boot ([tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 item 1). The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; on a kernel from before it `rtcwake` fails and a sleep ends only on a key. Check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. The rule above stands. A sleep series has to keep the shell's auto-sleep out: with it on, `omamobile-sleepd` puts a wake that leaves the screen off behind a secure lock back to sleep after 5 s, with no alarm, and from the laptop that sleep looks like a no-wake (2026-11-10, twice; the second lasted nine and a half hours and was reported as a hang). `device/tegu/bringup/sleep-series.sh` holds a block inhibitor of its own for that; before calling a silent palmtop hung, ask for one short press of the Power key and read the journal. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 -======= -- **On the 9a, a new sleep test runs attended first, and the direct freeze path is still not used** (a modem-up series on W4HO-ham's own 9a runs instead under [#809](https://github.com/derekross/omarchy-mobile/issues/809)'s terms, in the modem-up entry below). Unattended sleep series are allowed since 2026-11-05 (#409) on the contributor's conditions and three of the maintainer's. The contributor's: on the charger with the cable in; the modem off for the boot; the script stops itself on any UFS error line or oops; a wake with dead storage (SSH still answers) gets `echo b > /proc/sysrq-trigger` remotely, and only a silent hang waits for the owner's buttons; the owner says yes before each run. (Their "slot B marked" condition is not part of it: marking B on the 9a stays on hold, see the boot chain notes above, so a series that resets the palmtop spends ABL tries and stays short.) The maintainer's: nothing new (a kernel, a module or a script) runs unattended before one attended run of it; the automatic sysrq recovery is capped per night, and the next boot's ext4 recovery is checked and recorded; and the recovery script is in the tree (`device/tegu/bringup/`, where the series scripts already are) before it runs unattended. The plan says which rule each step runs under, and the counts below are the early ones; [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) keeps the current tally. Four sleeps have never woken (not from the RTC alarm, not from the power key; Power + Volume Up to reset, nothing in pstore): two direct `rtcwake -m freeze` runs (2026-10-28, 2026-10-29), one 30-minute `systemctl suspend` with `tegu-mif-sleep` loaded (2026-10-30, cable out) and the sixth 30 s `systemctl suspend` cycle on 2026-10-30 (the daily boot, no module; it left the USB bus); 35 short and seven longer systemd sleeps woke (two 30-minute and four 240 s ones on 2026-10-29, a cable-in 30-minute one with the module on 2026-10-30; the afternoon's cable-out run with the module makes eight). No single factor explains it, so keep `power-ab` on `--suspend systemd` (the sleep hooks run that way), and don't load `tegu-mif-sleep` at boot until the hang is understood. The shell's auto-sleep (the `omamobile-sleep` toggle, systemd path, no module) woke every time while it was on, including seven sleeps of 11 s to 22 minutes in a day of use on 2026-11-01 (lengths from the RTC); the package ships it off, and the owner turned it back on on the palmtop that day, so keeping it on is the owner's call. Two causes are known. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224) re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. Open: on the published kernel, which has no idle states, a sleep still fails to wake now and then: one in 22 on 2026-11-09 (a boot without modules, no console), none in two re-runs of 36 with modules loaded, and one in 213 overnight on 2026-11-10 on the newest tip with the fixes. That one left a console: the kernel stayed alive, and CPU 7 answered nothing from just after the device suspend until the owner's reset (an RCU stall on CPU 7 every 63 s; neither the alarm nor the Power key got through). So a palmtop that stays silent after a sleep may be a running kernel with one core locked up, not a SoC that stayed down; what that core was doing isn't known. The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; on a kernel from before it `rtcwake` fails and a sleep ends only on a key. Check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. The rule above stands. A sleep series has to keep the shell's auto-sleep out: with it on, `omamobile-sleepd` puts a wake that leaves the screen off behind a secure lock back to sleep after 5 s, with no alarm, and from the laptop that sleep looks like a no-wake (2026-11-10, twice; the second lasted nine and a half hours and was reported as a hang). `device/tegu/bringup/sleep-series.sh` holds a block inhibitor of its own for that; before calling a silent palmtop hung, ask for one short press of the Power key and read the journal. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 ->>>>>>> origin/main +- **On the 9a, a new sleep test runs attended first, and the direct freeze path is still not used** (a modem-up series on W4HO-ham's own 9a runs instead under [#809](https://github.com/derekross/omarchy-mobile/issues/809)'s terms, in the modem-up entry below). Unattended sleep series are allowed since 2026-11-05 (#409) on the contributor's conditions and three of the maintainer's. The contributor's: on the charger with the cable in; the modem off for the boot; the script stops itself on any UFS error line or oops; a wake with dead storage (SSH still answers) gets `echo b > /proc/sysrq-trigger` remotely, and only a silent hang waits for the owner's buttons; the owner says yes before each run. (Their "slot B marked" condition is not part of it: marking B on the 9a stays on hold, see the boot chain notes above, so a series that resets the palmtop spends ABL tries and stays short.) The maintainer's: nothing new (a kernel, a module or a script) runs unattended before one attended run of it; the automatic sysrq recovery is capped per night, and the next boot's ext4 recovery is checked and recorded; and the recovery script is in the tree (`device/tegu/bringup/`, where the series scripts already are) before it runs unattended. The plan says which rule each step runs under, and the counts below are the early ones; [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) keeps the current tally. Four sleeps have never woken (not from the RTC alarm, not from the power key; Power + Volume Up to reset, nothing in pstore): two direct `rtcwake -m freeze` runs (2026-10-28, 2026-10-29), one 30-minute `systemctl suspend` with `tegu-mif-sleep` loaded (2026-10-30, cable out) and the sixth 30 s `systemctl suspend` cycle on 2026-10-30 (the daily boot, no module; it left the USB bus); 35 short and seven longer systemd sleeps woke (two 30-minute and four 240 s ones on 2026-10-29, a cable-in 30-minute one with the module on 2026-10-30; the afternoon's cable-out run with the module makes eight). No single factor explains it, so keep `power-ab` on `--suspend systemd` (the sleep hooks run that way), and don't load `tegu-mif-sleep` at boot until the hang is understood. The shell's auto-sleep (the `omamobile-sleep` toggle, systemd path, no module) woke every time while it was on, including seven sleeps of 11 s to 22 minutes in a day of use on 2026-11-01 (lengths from the RTC); the package ships it off, and the owner turned it back on on the palmtop that day, so keeping it on is the owner's call. Two causes are known. Dead storage after a wake: the UFS pads came back as plain inputs; [linux-omarchy-mobile#224](https://github.com/derekross/linux-omarchy-mobile/pull/224) re-applies the pin state at resume. A silent no-wake with C2 entered through s2idle's own path: with C2 disabled for the sleep (#96) or left to the ordinary idle path (#98) the RTC alarm woke the palmtop 300 of 300 times each; [omarchy-mobile#591](https://github.com/derekross/omarchy-mobile/pull/591) is the sleep hook for it. A third cause, on the published kernel, which has no idle states: a sleep failed to wake four times in 1,213 (2026-11-09 to 11-12). The consoles show the kernel alive and CPU 7, which runs the suspend, taking no tick and answering no IPI from the noirq resume of devices on, so a palmtop that stays silent after a sleep may be a running kernel with one core locked up, not a SoC that stayed down; neither the alarm nor the Power key gets through. The core is in a register access to an HSI0 USI (the touchscreen's or the GNSS SPI's) in the first microseconds after its clock gates are opened, which never returns: shown awake with a test module that hangs the SoC on purpose (`device/tegu/bringup/hsi0-usi/`, attended only). [linux-omarchy-mobile#247](https://github.com/derekross/linux-omarchy-mobile/pull/247) keeps those six gates on (300 of 300 on the published tip with it alone, 2026-11-13); until a kernel the palmtop runs carries it, the no-wake stays possible, about one in 300. The PM debug switches of `sleep-series.sh` hide it (701 sleeps with them on and none hung): to catch it use `PMDEBUG=1 TRACE=2 PMPRINT=0`. The stall panic writes the console but its own restart hangs, so the buttons are still needed, and pstore may reject the console's header; the text can be read raw from the ramoops region on the next boot ([tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 item 1). The main PMIC's interrupt, and with it the RTC alarm, came to the published `zumapro-tegu.dts` with [linux-omarchy-mobile#236](https://github.com/derekross/linux-omarchy-mobile/pull/236), merged 2026-11-10; on a kernel from before it `rtcwake` fails and a sleep ends only on a key. Check that `/sys/class/rtc/rtc0/wakealarm` exists before any RTC-woken sleep. The rule above stands. A sleep series has to keep the shell's auto-sleep out: with it on, `omamobile-sleepd` puts a wake that leaves the screen off behind a secure lock back to sleep after 5 s, with no alarm, and from the laptop that sleep looks like a no-wake (2026-11-10, twice; the second lasted nine and a half hours and was reported as a hang). `device/tegu/bringup/sleep-series.sh` holds a block inhibitor of its own for that; before calling a silent palmtop hung, ask for one short press of the Power key and read the journal. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 - **On the 9a, a sleep can end badly in two ways: UFS fails to come back and the root filesystem goes read-only, or the SoC hangs and leaves the USB bus.** The first seen once on 2026-10-30 (`link startup failed 1`, `Host init failed 1` x5, `ufshcd_wl_resume failed`, then `I/O error, dev sda` and ext4 `emergency_ro`; `spm_lvl` 5). The palmtop woke on its alarm but nothing on disk could be read: `systemctl suspend`/`reboot` did nothing, the power key could not light the screen, SSH kept answering from memory. A dark palmtop that ignores the power key may therefore be awake with dead storage, not asleep. After every s2idle, `dmesg | grep -E "link startup failed|Host init failed|I/O error"` first; if it matches, `echo b > /proc/sysrq-trigger` (binaries won't load), then check the ext4 recovery on the next boot. The dead storage is understood: gph5-0 and gph5-1 (the UFS reference clock and reset, `13060000.pinctrl`) read `CON 0x0` after such a wake, so the device gets no clock and no reset edge; linux-omarchy-mobile#224 re-applies the UFS pin state in `exynos_ufs_resume()`. On a kernel without it the check above stays. [tegu SLEEP-PLAN](device/tegu/SLEEP-PLAN.md) S1 - **On the 9a the kernel's clock runs through a sleep, so no service watchdog survives a long one.** s2idle stops CLOCK_MONOTONIC only when a cpuidle state is entered through its s2idle callback (`tick_freeze()`); the 9a's published device tree has no idle states, and where a kernel has them the cpuidle sleep hook keeps the cores in WFI for the sleep. systemd's service watchdogs count on that clock, so after a sleep longer than a service's `WatchdogSec` (3 minutes for journald, logind, udevd, homed, userdbd) systemd aborted it at the thaw: every sleep of 3 min 41 s or more cost at least one service (2026-11-14), and with systemd-logind gone the lock screen took no swipe and no call could be answered until a reboot ([#765](https://github.com/derekross/omarchy-mobile/issues/765)). `omamobile-device-tegu` 0.2.21 turns the service watchdogs off (`service.d/10-omamobile-tegu-sleep-watchdog.conf`; it must be there at boot, a running service keeps the watchdog it started with). On a palmtop without it, reboot after any sleep test longer than three minutes, and check `journalctl -b | grep 'Watchdog timeout'` before blaming a dead lock screen on anything else. For the same reason dmesg and journal time deltas are not a sleep's length there: read the RTC. - **UFS must be at `spm_lvl` 5 in system suspend.** At the core default 3, UFS failed its first hibern8 exit on resume and needed a host reset (it recovered, no I/O errors); at 5 the first real s2idle freeze resumed clean. gs201 defaults to 5 in the kernel (S3a, `d0c08c464ef8` on `omamobile`), and gs101 too since linux-omarchy-mobile#188 (2026-11-04), so a suspend test only checks that `spm_lvl` reads 5; on a kernel without those commits, `echo 5` first. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) S1-S3