Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ Each entry gives the gotcha, then the reason or the fix. Details are in the link
- **Some unattended RTC cycles wake early on the PMIC interrupt** (2 of 20 in S4, 4 of 31 in S5c, at 13-16 s instead of 30 s). Which PMIC event it is hasn't been read yet; count it, don't treat it as a failed cycle. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) S4, S5c
- **Volume Up is the one key without an internal pull** (`PUD` 0x0, as on stock), and the first freeze woke 3 s in from it with nobody touching it. It didn't recur in the 51 RTC cycles of S4 and S5c, and it isn't proven to be a false edge; if spurious wakes recur, check gpa8-5's power-down pull before blaming the wake path.
- **UFS ran its error handler around modem power changes** (PA LINERESET indications, recovered without a reset). Stock's DT shows no shared rail, clock or pin. M2.5 narrowed it down: power-on and a running link are clean, but every modem power-off is followed by one recovered UFS error-handler pass (a PA LINERESET, no I/O errors or resets). Stock powers the modem off only at shutdown, so the modem stays on once it is up (M2.6 is open), and a reboot with the modem up costs one such pass. Keep modem tests attended, don't power-cycle the modem, and check `dmesg | grep ufshcd_err_handler` after each step. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M2, M2.5, M2.6
- **Suspend with the modem on has never run on the 7 Pro, and is unqualified on the 9a**: its one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part is unknown. CPIF's PM paths are wake-armed in code, but M6 is design only and no SIM test has happened (M4.2). Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on"
- **Suspend with the modem on has never run on the 7 Pro, and is unqualified on the 9a**: its one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part was unknown. With the sleep fixes (2026-11-09, both link drivers) it neither hung nor reset: CPIF refuses most sleeps (`suspend_cp: abort suspend`, -16, while the CP holds `cp2ap_wakeup`), and the three that were entered woke on the modem within seconds. But one sleep attempt, entered or refused, leaves CPIF unable to reach the CP (`Can't set D0 state`, its command register reading 0xffff) and the phone service timing out until a reboot, while `omamobile-sit status` goes on showing the last state read. So with the modem up nothing passes or goes round its sleep inhibitor (`systemctl suspend -i`, `rtcwake -m`, a write to `/sys/power/state`) except an attended test on the charger that ends in a reboot; on the 7 Pro that test is the maintainer's. CPIF's PM paths are wake-armed in code, but M6 is design only. Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on"
- **Never `systemctl restart omamobile-modem` (or stop and start it) to recover a crashed modem.** An in-place restart left raven's CP in BOOTING, and a bring-up that never reaches ONLINE sets a latch that makes the next boot skip the modem. Reboot instead. If a boot skipped it, remove the latch (`rm /var/lib/omamobile/modem-bringup-started`) and `systemctl start omamobile-modem`; `/usr/lib/omamobile-vendor/omamobile-modem-start --retry` also clears it, but run by hand it brings the modem up outside the unit (a stray `rfsd` in your shell's session). From `omamobile-modem` 0.1.13 there is a second marker, `/var/lib/omamobile/modem-up`, written at ONLINE and removed by the unit's clean stop: a boot that hung or reset with the modem up leaves it. The next boot logs "ended without a clean stop" and, from 0.1.19, starts the modem all the same (0.1.13 to 0.1.18 skipped it for that boot); a second unclean end in a row refuses "(held)" until the markers are removed the same way (cheetah [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6). On the 7 Pro the stack starts at boot by default since device-cheetah 0.1.43 ([MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6b); to keep it off from the next boot, `systemctl mask omamobile-modem-boot.target` (`unmask` turns it back on; `disable` doesn't remove device-cheetah's vendor wants link); it also holds a sleep inhibitor until the reboot. [raven MODEM-PLAN](device/raven/MODEM-PLAN.md) D4. The same on the 9a: after the modem profile has stopped, starting it again in the same boot reset the palmtop silently, a watchdog reset with nothing in pstore (the one try was in the rfsd stage). [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) G3. **From `omamobile-modem` 0.1.10, `omamobile-vendor-chroot modem check` refuses once the profile has started this boot** ("the modem profile already started this boot"), so it is no longer a post-install health check while the modem is up: it is read-only and safe to run, but a refusal means the profile ran, not that anything is wrong. After a reboot it says "preconditions met" again.
- **Linux never writes the modem's NV partitions.** The modem chroot profile (cbd, rfsd) refuses to start unless the NV partitions read `ro=1` and none is mounted, and works on checksum-verified RAM copies only. What the modem writes is lost at stop, except its writes to `efs/nv_normal.bin` (expected to hold the carrier profile, network and IMS data; the runbook confirms it) when the NV overlay is on (section 1): those are captured every 30 minutes and at the unit's stop, after the CP is off, and laid over the verified copy at the next start. `reboot-to` (and `bin/omamobile push`) skips that stop, so stop the modem first (section 2) or run `omamobile-modem nv-overlay capture`. Don't give the chroot a real block device. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) "NV overlay"

Expand Down
89 changes: 89 additions & 0 deletions device/tegu/MODEM-PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1612,6 +1612,95 @@ the interrupt. So no power-off hook is needed for NFC to come back after a
wake; the stop-poll hook stays, for the reset of 10-27, which this doesn't
explain.

### Suspend with the modem on, with the sleep fixes: refused or short, and it costs the phone service (2026-11-09, night)

The first tries since 10-27, now with the UFS pin fix, C2 kept out of the
sleep and NFC polling stopped by its hook. On the charger, cable in, the CP
ONLINE and registered (home LTE, IMS voice and SMS), the modem's block
inhibitor passed on purpose with `systemctl suspend -i`, an RTC alarm armed
each time (`device/tegu/bringup/modem-sleep-test.sh`). Two kernels:
`7.3.0-rc6-omamobile-tegu-s2mpg15-m1-g991042e2eff3-dirty` build #100 with
the bring-up modules (four boots, 22 sleep requests), and
`7.3.0-rc6-omamobile-push1-g773b4102f5ec` with the in-tree `pcie_gs201_cp`
and CPIF (one boot, 3 requests). That second kernel is not the published
one: it is the published tip of that day (`abcd6aac51cf`) plus kernel #224,
#236 and #237 and #274's PHY config line, as in section P below, and kernel
#224, one of the sleep fixes this test rests on, was not merged. "The
in-tree build" below means it.

How it ran: five boots between 22:32 and 23:15 MST (the modem up at 22:32,
22:38, 22:44 and 22:52 on #100, and at 23:08 on the in-tree build), all on
the charger with the cable in, and all of them unattended. The owner had
gone to bed; what he had given was a standing word to keep working through
the night, not a yes for each run, and nobody was at the palmtop for any of
the five. So this was outside the 9a's sleep rule in two ways: the tests
were not attended, and they slept with the modem up. Each test boot ended
in `systemctl reboot`, and each of those stops was confirmed.

- **Nothing hung.** No reset, no silent no-wake, no UFS line, and the CP's
state read ONLINE throughout.
- **Most sleeps are refused by CPIF.** 22 of the 25 requests failed with
`cpif: suspend_cp: abort suspend` and `PM: failed to suspend noirq: error
-16`: `suspend_cp()` returns -EBUSY while the CP holds `cp2ap_wakeup`
high, and the CP raised it during the suspend sequence itself
(`ap_wakeup_handler: cp2ap_wakeup work pending. gpio_val : 1` right after
"Suspend prepare").
- **Three sleeps were entered, and the modem woke each within seconds.**
Two on #100 (about 6 s and 4 s, `pm_wakeup_irq` the modem's PCIe
interrupt and `cp2ap_wakeup`), one on the in-tree build (about 2 s,
`cp2ap_wakeup`). So the CP's wake line does wake the palmtop from s2idle.
- **One sleep attempt, entered or refused, costs the phone service until a
reboot.** In all five boots omamobile-sitd's requests to the CP began to
time out within 95 s of the first attempt ("poll failed: TimeoutError")
and never came back, on both kernels; `omamobile-sit status` went on
showing the last state it had read, and the screen-state request
answered "Invalid or unavailable phone request". The kernel log of the
first attempt, the same on both link drivers: `pcieport ...: Unable to
change power state from D0 to D3hot, device inaccessible`, then on the
way back `Unable to change power state from D3cold to D0, device
inaccessible` and `Data Link Layer Link Active not set in 100 msec`; the
link driver brings the link up again (Gen3 x2), and CPIF then logs
`s51xx_pcie_restore_state: Can't set D0 state!!!!`,
`s51xx_pcie_send_doorbell_int: Can't send Interrupt(not setted bme_en,
0xffff)`, `cmd reg = 0xffff`, `BME is not set`, and "link-down recovery
requested"; on #100 the doorbell failed about 3000 times in the next 40 s.
Our reading, not yet tested: the PCI core's system-sleep callbacks put
the CP's root port and endpoint through D3 while CPIF, which powers this
link up and down itself and saves and restores the endpoint's state
itself, has the PHY down, and afterwards the endpoint's configuration
space reads all ones to CPIF. Every `systemctl reboot` after these boots
still logged the confirmed stop.
- **Why the CP is awake so much: nobody tells it the screen is off.** Idle
and registered, the link came up 49 times a minute (98 `cp2ap_wakeup`
edges, SIT indication 0906 about 51 times a minute; 25 link-ups in 30 s
on the in-tree build too). omamobile-sitd has the request for it
(`{"action": "screen-state", "on": false}`: SIT 0902 with 0, and the
indication filter 0928 with 0), and nothing in the tree sends it. Sent by
hand it took the idle link to 3 or 4 link-ups a minute (6 to 8 edges, no
0906), and back to 49 with it on again. With it off the five sleeps of
that boot were still refused, so the CP's wake at the moment of the
sleep has another source, not found yet (the Wi-Fi interface going down
for the sleep moves the default route to `rmnet0`, and sitd polls every
30 s).
- **A refused sleep can cost Wi-Fi too.** systemd tries its sleep states in
turn (`mem`, which is s2idle here, then `freeze`), so CPIF's refusal was
followed 0.5 s later by a second attempt, in which `brcmfmac` failed to
suspend (`error -5`) or lost its interface; the Wi-Fi hook then logs
"Wi-Fi lost across sleep; unloading it around sleeps until reboot". With
one state only (a runtime `SuspendState=freeze` drop-in for the test)
Wi-Fi was still lost at the second refused sleep on #100, and stayed up
through one entered and two refused sleeps on the in-tree build.

So the inhibitor `omamobile-tegu-modem-up` takes is not only about a hang:
until the link survives a sleep, a suspend with the modem up cuts the phone
service off, silently, until the next reboot. What M6's sleep part needs,
in order: the CP's PCI devices kept out of the PCI core's system sleep (or
their state restored after it), which is kernel work on the shared link
driver and so the 7 Pro's too; the shell telling sitd the screen state, so
the CP is idle when a sleep starts; one sleep state in the device's
`sleep.conf`; then the qualification itself (RTC wakes, a call and a text
waking the palmtop).

### M2: the link only (modem link driver, `allow_hw=1`; risk: medium)

One step per run, the CPIF overlay not applied, the link overlay applied
Expand Down
38 changes: 38 additions & 0 deletions device/tegu/bringup/modem-sleep-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/bin/bash
# SPDX-License-Identifier: MIT
# On the palmtop: N systemd sleeps of S seconds with the modem up (MODEM-PLAN, "Suspend
# with the modem on"), an RTC alarm as the wake. A test only, and it ends in a reboot:
# the modem's own block inhibitor is passed with "systemctl suspend -i", and one sleep
# attempt, entered or refused, has left the phone service unable to reach the CP until
# the next reboot (2026-11-09). The sleep hooks run (NFC polling stopped, C2 out of the
# sleep, Wi-Fi kept). Records for each request how long it lasted, what woke it, and the
# CP's and the phone service's state after; "omamobile-sit status" shows the last state
# read, so check sitd's journal for "poll failed" as well. Stops when the CP isn't ONLINE,
# sitd isn't active or a UFS line appears. Attended, on the charger (AGENTS.md, "Suspend
# with the modem on"). Run detached:
# systemd-run --unit=x-modem-sleep /root/modem-sleep-test.sh [N] [S]
# and when it is done, the last step is always "systemctl reboot", never reboot-to, so the
# modem stops cleanly and the phone service comes back with the next boot.
N=${1:-1}; S=${2:-60}; L=/var/log/omamobile/modem-sleep-test.log
U='link startup failed|Host init failed|I/O error, dev|ufshcd_err_handler'
CP=/usr/lib/omamobile-vendor/omamobile-modem-cp
st() { echo "cp $($CP state 2>&1), sitd $(systemctl is-active omamobile-sitd), $(omamobile-sit status 2>/dev/null | sed -n 's/^network: //p'), $(omamobile-sit status 2>/dev/null | sed -n 's/^IMS voice: //p' | cut -c1-60)"; }
echo "$(date -Is) $(uname -r) $(uname -v | cut -d' ' -f1): $N sleeps of $S s with the modem up; before: $(st); state1 disable $(cat /sys/devices/system/cpu/cpu*/cpuidle/state1/disable 2>/dev/null | tr -d '\n'), spm_lvl $(cat /sys/devices/platform/soc/13200000.ufs/spm_lvl 2>/dev/null)" >> $L
[[ $($CP state 2>&1) == ONLINE ]] || { echo "the CP is not ONLINE: stop" >> $L; exit 1; }
for c in $(seq 1 "$N"); do
s0=$(cat /sys/power/suspend_stats/success); f0=$(cat /sys/power/suspend_stats/fail); u0=$(journalctl -b -k --no-pager | grep -cE "$U")
rtcwake -m no -s "$S" >/dev/null || { echo "cycle $c: rtcwake could not arm the alarm: stop" >> $L; exit 1; }
r0=$(cat /sys/class/rtc/rtc0/since_epoch); k0=$(awk '{print $1}' /proc/uptime)
systemctl suspend -i || { echo "cycle $c: systemctl suspend -i refused: stop" >> $L; exit 1; }
for _ in $(seq 1 $((S * 2 + 60))); do [[ $(cat /sys/power/suspend_stats/success) != "$s0" || $(cat /sys/power/suspend_stats/fail) != "$f0" ]] && break; sleep 0.5; done
r1=$(cat /sys/class/rtc/rtc0/since_epoch); sleep 6
irq=$(cat /sys/power/pm_wakeup_irq 2>/dev/null)
line="cycle $c $(date +%T): rtc $((r1 - r0)) s, suspend_stats $(cat /sys/power/suspend_stats/success)/$(cat /sys/power/suspend_stats/fail) (was $s0/$f0), wake irq $irq ($(grep -E "^ *$irq:" /proc/interrupts | awk '{print $(NF-1), $NF}')), new ufs lines $(($(journalctl -b -k --no-pager | grep -cE "$U") - u0)), after: $(st)"
echo "$line" >> $L
journalctl -b -k --no-pager --since "-$((S + 30)) s" | grep -E "PM: suspend (entry|exit)|PM: .*(abort|fail|wakeup)|cpif:.*(suspend|resume|ERR|crash)|pcie.*(suspend|resume|fail)" | tail -8 | cut -c40-200 | sed 's/^/ /' >> $L
rtcwake -m disable >/dev/null 2>&1
echo " wlan0 $(cat /sys/class/net/wlan0/operstate 2>/dev/null || echo gone), wifi hook: $(journalctl -b --no-pager --since "-$((S + 40)) s" | grep -o "Wi-Fi lost across sleep[^;]*" | tail -1)" >> $L
case "$line" in *"new ufs lines 0, after: cp ONLINE, sitd active"*) ;; *) echo "stop: the modem or storage is not as before" >> $L; exit 2 ;; esac
sleep 20
done
echo "done: $N sleeps" >> $L