diff --git a/AGENTS.md b/AGENTS.md index 5736ccc2e..864a98545 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -193,7 +193,7 @@ Each entry gives the gotcha, then the reason or the fix. Details are in the link - **Some unattended RTC cycles wake early on the PMIC interrupt** (2 of 20 in S4, 4 of 31 in S5c, at 13-16 s instead of 30 s). Which PMIC event it is hasn't been read yet; count it, don't treat it as a failed cycle. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) S4, S5c - **Volume Up is the one key without an internal pull** (`PUD` 0x0, as on stock), and the first freeze woke 3 s in from it with nobody touching it. It didn't recur in the 51 RTC cycles of S4 and S5c, and it isn't proven to be a false edge; if spurious wakes recur, check gpa8-5's power-down pull before blaming the wake path. - **UFS ran its error handler around modem power changes** (PA LINERESET indications, recovered without a reset). Stock's DT shows no shared rail, clock or pin. M2.5 narrowed it down: power-on and a running link are clean, but every modem power-off is followed by one recovered UFS error-handler pass (a PA LINERESET, no I/O errors or resets). Stock powers the modem off only at shutdown, so the modem stays on once it is up (M2.6 is open), and a reboot with the modem up costs one such pass. Keep modem tests attended, don't power-cycle the modem, and check `dmesg | grep ufshcd_err_handler` after each step. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M2, M2.5, M2.6 -- **Suspend with the modem on has never run on the 7 Pro, and is unqualified on the 9a**: its one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part is unknown. CPIF's PM paths are wake-armed in code, but M6 is design only and no SIM test has happened (M4.2). Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" +- **Suspend with the modem on has never run on the 7 Pro, and is unqualified on the 9a**: its one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part was unknown. With the sleep fixes (2026-11-09, both link drivers) it neither hung nor reset: CPIF refuses most sleeps (`suspend_cp: abort suspend`, -16, while the CP holds `cp2ap_wakeup`), and the three that were entered woke on the modem within seconds. But one sleep attempt, entered or refused, leaves CPIF unable to reach the CP (`Can't set D0 state`, its command register reading 0xffff) and the phone service timing out until a reboot, while `omamobile-sit status` goes on showing the last state read. So with the modem up nothing passes or goes round its sleep inhibitor (`systemctl suspend -i`, `rtcwake -m`, a write to `/sys/power/state`) except an attended test on the charger that ends in a reboot; on the 7 Pro that test is the maintainer's. CPIF's PM paths are wake-armed in code, but M6 is design only. Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" - **Never `systemctl restart omamobile-modem` (or stop and start it) to recover a crashed modem.** An in-place restart left raven's CP in BOOTING, and a bring-up that never reaches ONLINE sets a latch that makes the next boot skip the modem. Reboot instead. If a boot skipped it, remove the latch (`rm /var/lib/omamobile/modem-bringup-started`) and `systemctl start omamobile-modem`; `/usr/lib/omamobile-vendor/omamobile-modem-start --retry` also clears it, but run by hand it brings the modem up outside the unit (a stray `rfsd` in your shell's session). From `omamobile-modem` 0.1.13 there is a second marker, `/var/lib/omamobile/modem-up`, written at ONLINE and removed by the unit's clean stop: a boot that hung or reset with the modem up leaves it. The next boot logs "ended without a clean stop" and, from 0.1.19, starts the modem all the same (0.1.13 to 0.1.18 skipped it for that boot); a second unclean end in a row refuses "(held)" until the markers are removed the same way (cheetah [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6). On the 7 Pro the stack starts at boot by default since device-cheetah 0.1.43 ([MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6b); to keep it off from the next boot, `systemctl mask omamobile-modem-boot.target` (`unmask` turns it back on; `disable` doesn't remove device-cheetah's vendor wants link); it also holds a sleep inhibitor until the reboot. [raven MODEM-PLAN](device/raven/MODEM-PLAN.md) D4. The same on the 9a: after the modem profile has stopped, starting it again in the same boot reset the palmtop silently, a watchdog reset with nothing in pstore (the one try was in the rfsd stage). [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) G3. **From `omamobile-modem` 0.1.10, `omamobile-vendor-chroot modem check` refuses once the profile has started this boot** ("the modem profile already started this boot"), so it is no longer a post-install health check while the modem is up: it is read-only and safe to run, but a refusal means the profile ran, not that anything is wrong. After a reboot it says "preconditions met" again. - **Linux never writes the modem's NV partitions.** The modem chroot profile (cbd, rfsd) refuses to start unless the NV partitions read `ro=1` and none is mounted, and works on checksum-verified RAM copies only. What the modem writes is lost at stop, except its writes to `efs/nv_normal.bin` (expected to hold the carrier profile, network and IMS data; the runbook confirms it) when the NV overlay is on (section 1): those are captured every 30 minutes and at the unit's stop, after the CP is off, and laid over the verified copy at the next start. `reboot-to` (and `bin/omamobile push`) skips that stop, so stop the modem first (section 2) or run `omamobile-modem nv-overlay capture`. Don't give the chroot a real block device. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) "NV overlay" diff --git a/device/tegu/MODEM-PLAN.md b/device/tegu/MODEM-PLAN.md index b9a9af3f9..d29c2b151 100644 --- a/device/tegu/MODEM-PLAN.md +++ b/device/tegu/MODEM-PLAN.md @@ -1612,6 +1612,95 @@ the interrupt. So no power-off hook is needed for NFC to come back after a wake; the stop-poll hook stays, for the reset of 10-27, which this doesn't explain. +### Suspend with the modem on, with the sleep fixes: refused or short, and it costs the phone service (2026-11-09, night) + +The first tries since 10-27, now with the UFS pin fix, C2 kept out of the +sleep and NFC polling stopped by its hook. On the charger, cable in, the CP +ONLINE and registered (home LTE, IMS voice and SMS), the modem's block +inhibitor passed on purpose with `systemctl suspend -i`, an RTC alarm armed +each time (`device/tegu/bringup/modem-sleep-test.sh`). Two kernels: +`7.3.0-rc6-omamobile-tegu-s2mpg15-m1-g991042e2eff3-dirty` build #100 with +the bring-up modules (four boots, 22 sleep requests), and +`7.3.0-rc6-omamobile-push1-g773b4102f5ec` with the in-tree `pcie_gs201_cp` +and CPIF (one boot, 3 requests). That second kernel is not the published +one: it is the published tip of that day (`abcd6aac51cf`) plus kernel #224, +#236 and #237 and #274's PHY config line, as in section P below, and kernel +#224, one of the sleep fixes this test rests on, was not merged. "The +in-tree build" below means it. + +How it ran: five boots between 22:32 and 23:15 MST (the modem up at 22:32, +22:38, 22:44 and 22:52 on #100, and at 23:08 on the in-tree build), all on +the charger with the cable in, and all of them unattended. The owner had +gone to bed; what he had given was a standing word to keep working through +the night, not a yes for each run, and nobody was at the palmtop for any of +the five. So this was outside the 9a's sleep rule in two ways: the tests +were not attended, and they slept with the modem up. Each test boot ended +in `systemctl reboot`, and each of those stops was confirmed. + +- **Nothing hung.** No reset, no silent no-wake, no UFS line, and the CP's + state read ONLINE throughout. +- **Most sleeps are refused by CPIF.** 22 of the 25 requests failed with + `cpif: suspend_cp: abort suspend` and `PM: failed to suspend noirq: error + -16`: `suspend_cp()` returns -EBUSY while the CP holds `cp2ap_wakeup` + high, and the CP raised it during the suspend sequence itself + (`ap_wakeup_handler: cp2ap_wakeup work pending. gpio_val : 1` right after + "Suspend prepare"). +- **Three sleeps were entered, and the modem woke each within seconds.** + Two on #100 (about 6 s and 4 s, `pm_wakeup_irq` the modem's PCIe + interrupt and `cp2ap_wakeup`), one on the in-tree build (about 2 s, + `cp2ap_wakeup`). So the CP's wake line does wake the palmtop from s2idle. +- **One sleep attempt, entered or refused, costs the phone service until a + reboot.** In all five boots omamobile-sitd's requests to the CP began to + time out within 95 s of the first attempt ("poll failed: TimeoutError") + and never came back, on both kernels; `omamobile-sit status` went on + showing the last state it had read, and the screen-state request + answered "Invalid or unavailable phone request". The kernel log of the + first attempt, the same on both link drivers: `pcieport ...: Unable to + change power state from D0 to D3hot, device inaccessible`, then on the + way back `Unable to change power state from D3cold to D0, device + inaccessible` and `Data Link Layer Link Active not set in 100 msec`; the + link driver brings the link up again (Gen3 x2), and CPIF then logs + `s51xx_pcie_restore_state: Can't set D0 state!!!!`, + `s51xx_pcie_send_doorbell_int: Can't send Interrupt(not setted bme_en, + 0xffff)`, `cmd reg = 0xffff`, `BME is not set`, and "link-down recovery + requested"; on #100 the doorbell failed about 3000 times in the next 40 s. + Our reading, not yet tested: the PCI core's system-sleep callbacks put + the CP's root port and endpoint through D3 while CPIF, which powers this + link up and down itself and saves and restores the endpoint's state + itself, has the PHY down, and afterwards the endpoint's configuration + space reads all ones to CPIF. Every `systemctl reboot` after these boots + still logged the confirmed stop. +- **Why the CP is awake so much: nobody tells it the screen is off.** Idle + and registered, the link came up 49 times a minute (98 `cp2ap_wakeup` + edges, SIT indication 0906 about 51 times a minute; 25 link-ups in 30 s + on the in-tree build too). omamobile-sitd has the request for it + (`{"action": "screen-state", "on": false}`: SIT 0902 with 0, and the + indication filter 0928 with 0), and nothing in the tree sends it. Sent by + hand it took the idle link to 3 or 4 link-ups a minute (6 to 8 edges, no + 0906), and back to 49 with it on again. With it off the five sleeps of + that boot were still refused, so the CP's wake at the moment of the + sleep has another source, not found yet (the Wi-Fi interface going down + for the sleep moves the default route to `rmnet0`, and sitd polls every + 30 s). +- **A refused sleep can cost Wi-Fi too.** systemd tries its sleep states in + turn (`mem`, which is s2idle here, then `freeze`), so CPIF's refusal was + followed 0.5 s later by a second attempt, in which `brcmfmac` failed to + suspend (`error -5`) or lost its interface; the Wi-Fi hook then logs + "Wi-Fi lost across sleep; unloading it around sleeps until reboot". With + one state only (a runtime `SuspendState=freeze` drop-in for the test) + Wi-Fi was still lost at the second refused sleep on #100, and stayed up + through one entered and two refused sleeps on the in-tree build. + +So the inhibitor `omamobile-tegu-modem-up` takes is not only about a hang: +until the link survives a sleep, a suspend with the modem up cuts the phone +service off, silently, until the next reboot. What M6's sleep part needs, +in order: the CP's PCI devices kept out of the PCI core's system sleep (or +their state restored after it), which is kernel work on the shared link +driver and so the 7 Pro's too; the shell telling sitd the screen state, so +the CP is idle when a sleep starts; one sleep state in the device's +`sleep.conf`; then the qualification itself (RTC wakes, a call and a text +waking the palmtop). + ### M2: the link only (modem link driver, `allow_hw=1`; risk: medium) One step per run, the CPIF overlay not applied, the link overlay applied diff --git a/device/tegu/bringup/modem-sleep-test.sh b/device/tegu/bringup/modem-sleep-test.sh new file mode 100755 index 000000000..28d637a55 --- /dev/null +++ b/device/tegu/bringup/modem-sleep-test.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT +# On the palmtop: N systemd sleeps of S seconds with the modem up (MODEM-PLAN, "Suspend +# with the modem on"), an RTC alarm as the wake. A test only, and it ends in a reboot: +# the modem's own block inhibitor is passed with "systemctl suspend -i", and one sleep +# attempt, entered or refused, has left the phone service unable to reach the CP until +# the next reboot (2026-11-09). The sleep hooks run (NFC polling stopped, C2 out of the +# sleep, Wi-Fi kept). Records for each request how long it lasted, what woke it, and the +# CP's and the phone service's state after; "omamobile-sit status" shows the last state +# read, so check sitd's journal for "poll failed" as well. Stops when the CP isn't ONLINE, +# sitd isn't active or a UFS line appears. Attended, on the charger (AGENTS.md, "Suspend +# with the modem on"). Run detached: +# systemd-run --unit=x-modem-sleep /root/modem-sleep-test.sh [N] [S] +# and when it is done, the last step is always "systemctl reboot", never reboot-to, so the +# modem stops cleanly and the phone service comes back with the next boot. +N=${1:-1}; S=${2:-60}; L=/var/log/omamobile/modem-sleep-test.log +U='link startup failed|Host init failed|I/O error, dev|ufshcd_err_handler' +CP=/usr/lib/omamobile-vendor/omamobile-modem-cp +st() { echo "cp $($CP state 2>&1), sitd $(systemctl is-active omamobile-sitd), $(omamobile-sit status 2>/dev/null | sed -n 's/^network: //p'), $(omamobile-sit status 2>/dev/null | sed -n 's/^IMS voice: //p' | cut -c1-60)"; } +echo "$(date -Is) $(uname -r) $(uname -v | cut -d' ' -f1): $N sleeps of $S s with the modem up; before: $(st); state1 disable $(cat /sys/devices/system/cpu/cpu*/cpuidle/state1/disable 2>/dev/null | tr -d '\n'), spm_lvl $(cat /sys/devices/platform/soc/13200000.ufs/spm_lvl 2>/dev/null)" >> $L +[[ $($CP state 2>&1) == ONLINE ]] || { echo "the CP is not ONLINE: stop" >> $L; exit 1; } +for c in $(seq 1 "$N"); do + s0=$(cat /sys/power/suspend_stats/success); f0=$(cat /sys/power/suspend_stats/fail); u0=$(journalctl -b -k --no-pager | grep -cE "$U") + rtcwake -m no -s "$S" >/dev/null || { echo "cycle $c: rtcwake could not arm the alarm: stop" >> $L; exit 1; } + r0=$(cat /sys/class/rtc/rtc0/since_epoch); k0=$(awk '{print $1}' /proc/uptime) + systemctl suspend -i || { echo "cycle $c: systemctl suspend -i refused: stop" >> $L; exit 1; } + for _ in $(seq 1 $((S * 2 + 60))); do [[ $(cat /sys/power/suspend_stats/success) != "$s0" || $(cat /sys/power/suspend_stats/fail) != "$f0" ]] && break; sleep 0.5; done + r1=$(cat /sys/class/rtc/rtc0/since_epoch); sleep 6 + irq=$(cat /sys/power/pm_wakeup_irq 2>/dev/null) + line="cycle $c $(date +%T): rtc $((r1 - r0)) s, suspend_stats $(cat /sys/power/suspend_stats/success)/$(cat /sys/power/suspend_stats/fail) (was $s0/$f0), wake irq $irq ($(grep -E "^ *$irq:" /proc/interrupts | awk '{print $(NF-1), $NF}')), new ufs lines $(($(journalctl -b -k --no-pager | grep -cE "$U") - u0)), after: $(st)" + echo "$line" >> $L + journalctl -b -k --no-pager --since "-$((S + 30)) s" | grep -E "PM: suspend (entry|exit)|PM: .*(abort|fail|wakeup)|cpif:.*(suspend|resume|ERR|crash)|pcie.*(suspend|resume|fail)" | tail -8 | cut -c40-200 | sed 's/^/ /' >> $L + rtcwake -m disable >/dev/null 2>&1 + echo " wlan0 $(cat /sys/class/net/wlan0/operstate 2>/dev/null || echo gone), wifi hook: $(journalctl -b --no-pager --since "-$((S + 40)) s" | grep -o "Wi-Fi lost across sleep[^;]*" | tail -1)" >> $L + case "$line" in *"new ufs lines 0, after: cp ONLINE, sitd active"*) ;; *) echo "stop: the modem or storage is not as before" >> $L; exit 2 ;; esac + sleep 20 +done +echo "done: $N sleeps" >> $L