diff --git a/AGENTS.md b/AGENTS.md index a3387f4c1..ed4818a19 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -197,7 +197,7 @@ Each entry gives the gotcha, then the reason or the fix. Details are in the link - **Some unattended RTC cycles wake early on the PMIC interrupt** (2 of 20 in S4, 4 of 31 in S5c, at 13-16 s instead of 30 s). Which PMIC event it is hasn't been read yet; count it, don't treat it as a failed cycle. [SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) S4, S5c - **Volume Up is the one key without an internal pull** (`PUD` 0x0, as on stock), and the first freeze woke 3 s in from it with nobody touching it. It didn't recur in the 51 RTC cycles of S4 and S5c, and it isn't proven to be a false edge; if spurious wakes recur, check gpa8-5's power-down pull before blaming the wake path. - **UFS ran its error handler around modem power changes** (PA LINERESET indications, recovered without a reset). Stock's DT shows no shared rail, clock or pin. M2.5 narrowed it down: power-on and a running link are clean, but every modem power-off is followed by one recovered UFS error-handler pass (a PA LINERESET, no I/O errors or resets). Stock powers the modem off only at shutdown, so the modem stays on once it is up (M2.6 is open), and a reboot with the modem up costs one such pass. Keep modem tests attended, don't power-cycle the modem, and check `dmesg | grep ufshcd_err_handler` after each step. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M2, M2.5, M2.6 -- **Suspend with the modem up is on by default on the 7 Pro, and unqualified on the 9a.** On the Pixel 7 Pro with a kernel carrying linux-omarchy-mobile#232 (kernel-2026.11.15 and later), `omamobile-modem-sit` 0.3.42 and device-cheetah 0.1.60, the modem's sleep inhibitor is skipped and the phone sleeps with the modem up ([SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) M6-D, 2026-11-15: 85 wakes in 45 minutes with data back after every drop). The network drops the internet bearer every few minutes while the phone sleeps; sitd sets it up again (#817), and without that data stays down until a reboot, which is why device-cheetah 0.1.60 conflicts with an older `omamobile-modem-sit`. On an older kernel `omamobile-modem-sleep-policy` keeps the inhibitor, and the owner opts out with `/etc/omamobile/modem-suspend-disable` (from the next boot). On every other phone the rest of this entry stands. The 9a's one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part was unknown. With the sleep fixes (2026-11-09, both link drivers) it neither hung nor reset: CPIF refuses most sleeps (`suspend_cp: abort suspend`, -16, while the CP holds `cp2ap_wakeup`), and the three that were entered woke on the modem within seconds. But one sleep attempt, entered or refused, leaves CPIF unable to reach the CP (`Can't set D0 state`, its command register reading 0xffff) and the phone service timing out until a reboot, while `omamobile-sit status` goes on showing the last state read. So wherever the sleep policy keeps the inhibitor (no device declaration, a kernel without #232, the owner's opt-out), with the modem up nothing passes or goes round it (`systemctl suspend -i`, `rtcwake -m`, a write to `/sys/power/state`) except an attended test on the charger that ends in a reboot. On a 7 Pro that sleeps with the modem up, a sleep goes only through logind (auto-sleep or `systemctl suspend`), never `rtcwake -m` or a write to `/sys/power/state`: those skip the sleep hooks M6 verified (the modem told the screen is off, sitd's PrepareForSleep), and a new modem-up sleep test there (a new kernel, a change to CPIF or sitd's data path) is still the maintainer's. One exception, on W4HO-ham's own 9a only ([#809](https://github.com/derekross/omarchy-mobile/issues/809), agreed 2026-11-15): W4HO-ham's modem-up sleep series may run unattended, a new kernel's first among them, on these terms: on the charger (#758's `modem-sleep-series.sh` refuses and stops off it); every boot that had a sleep with the modem up ends in `systemctl reboot`, never `reboot-to`; a new kernel's first run is at most 10 sleeps and its result is posted before a longer one; the series stops itself on the CP not ONLINE, a UFS error line or an oops, and a hang is left to the cluster watchdog or the owner; the NV partitions stay read-only and are checked against the backup after any boot that hung or reset; the record says which rule each run ran under. CPIF's PM paths are wake-armed in code. A modem-up sleep record checks mobile data after every wake with no other route ahead of it, not once at the end: the 7 Pro's bearer drops showed only so. Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" +- **Suspend with the modem up is on by default on the 7 Pro, and unqualified on the 9a.** On the Pixel 7 Pro with a kernel carrying linux-omarchy-mobile#232 (kernel-2026.11.15 and later), `omamobile-modem-sit` 0.3.42 and device-cheetah 0.1.60, the modem's sleep inhibitor is skipped and the phone sleeps with the modem up ([SUSPEND-PLAN](device/cheetah/SUSPEND-PLAN.md) M6-D, 2026-11-15: 85 wakes in 45 minutes with data back after every drop). The network drops the internet bearer every few minutes while the phone sleeps; sitd sets it up again (#817), and without that data stays down until a reboot, which is why device-cheetah 0.1.60 conflicts with an older `omamobile-modem-sit`. On an older kernel `omamobile-modem-sleep-policy` keeps the inhibitor, and the owner opts out with `/etc/omamobile/modem-suspend-disable` (from the next boot). On every other phone the rest of this entry stands. The 9a's one try (2026-10-27) also had NFC polling, which alone resets the 9a in s2idle, so the modem's part was unknown. With the sleep fixes (2026-11-09, both link drivers) it neither hung nor reset: CPIF refuses most sleeps (`suspend_cp: abort suspend`, -16, while the CP holds `cp2ap_wakeup`), and the three that were entered woke on the modem within seconds. But one sleep attempt, entered or refused, leaves CPIF unable to reach the CP (`Can't set D0 state`, its command register reading 0xffff) and the phone service timing out until a reboot, while `omamobile-sit status` goes on showing the last state read. So wherever the sleep policy keeps the inhibitor (no device declaration, a kernel without #232, the owner's opt-out), with the modem up nothing passes or goes round it (`systemctl suspend -i`, `rtcwake -m`, a write to `/sys/power/state`) except an attended test on the charger that ends in a reboot. On a 7 Pro that sleeps with the modem up, a sleep goes only through logind (auto-sleep or `systemctl suspend`), never `rtcwake -m` or a write to `/sys/power/state`: those skip the sleep hooks M6 verified (the modem told the screen is off, sitd's PrepareForSleep), and a new modem-up sleep test there (a new kernel, a change to CPIF or sitd's data path) is still the maintainer's. On a 9a test kernel with the sleep guard turned on for zumapro ([linux-omarchy-mobile#258](https://github.com/derekross/linux-omarchy-mobile/pull/258)), 299 of 300 sleeps with the modem registered were entered and woke with the phone service kept (2026-11-13; Wi-Fi connected, mobile data checked at the end), and a refusal cost nothing; that kernel also hung once at a link power-up while awake, so on the 9a the rule is unchanged until a release carries the guard and the 9a's own declaration is decided. One exception, on W4HO-ham's own 9a only ([#809](https://github.com/derekross/omarchy-mobile/issues/809), agreed 2026-11-15): W4HO-ham's modem-up sleep series may run unattended, a new kernel's first among them, on these terms: on the charger (#758's `modem-sleep-series.sh` refuses and stops off it); every boot that had a sleep with the modem up ends in `systemctl reboot`, never `reboot-to`; a new kernel's first run is at most 10 sleeps and its result is posted before a longer one; the series stops itself on the CP not ONLINE, a UFS error line or an oops, and a hang is left to the cluster watchdog or the owner; the NV partitions stay read-only and are checked against the backup after any boot that hung or reset; the record says which rule each run ran under. CPIF's PM paths are wake-armed in code. A modem-up sleep record checks mobile data after every wake with no other route ahead of it, not once at the end: the 7 Pro's bearer drops showed only so. Keep modem tests on the charger, so upower's critical action doesn't suspend the phone mid-test. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) 6.3, [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) "Suspend with the modem on" - **Never `systemctl restart omamobile-modem` (or stop and start it) to recover a crashed modem.** An in-place restart left raven's CP in BOOTING, and a bring-up that never reaches ONLINE sets a latch that makes the next boot skip the modem. Reboot instead. If a boot skipped it, remove the latch (`rm /var/lib/omamobile/modem-bringup-started`) and `systemctl start omamobile-modem`; `/usr/lib/omamobile-vendor/omamobile-modem-start --retry` also clears it, but run by hand it brings the modem up outside the unit (a stray `rfsd` in your shell's session). From `omamobile-modem` 0.1.13 there is a second marker, `/var/lib/omamobile/modem-up`, written at ONLINE and removed by the unit's clean stop: a boot that hung or reset with the modem up leaves it. The next boot logs "ended without a clean stop" and, from 0.1.19, starts the modem all the same (0.1.13 to 0.1.18 skipped it for that boot); a second unclean end in a row refuses "(held)" until the markers are removed the same way (cheetah [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6). On the 7 Pro the stack starts at boot by default since device-cheetah 0.1.43 ([MODEM-PLAN](device/cheetah/MODEM-PLAN.md) M4.6b); to keep it off from the next boot, `systemctl mask omamobile-modem-boot.target` (`unmask` turns it back on; `disable` doesn't remove device-cheetah's vendor wants link); it also holds a sleep inhibitor until the reboot. [raven MODEM-PLAN](device/raven/MODEM-PLAN.md) D4. The same on the 9a: after the modem profile has stopped, starting it again in the same boot reset the palmtop silently, a watchdog reset with nothing in pstore (the one try was in the rfsd stage). [tegu MODEM-PLAN](device/tegu/MODEM-PLAN.md) G3. **From `omamobile-modem` 0.1.10, `omamobile-vendor-chroot modem check` refuses once the profile has started this boot** ("the modem profile already started this boot"), so it is no longer a post-install health check while the modem is up: it is read-only and safe to run, but a refusal means the profile ran, not that anything is wrong. After a reboot it says "preconditions met" again. - **Linux never writes the modem's NV partitions.** The modem chroot profile (cbd, rfsd) refuses to start unless the NV partitions read `ro=1` and none is mounted, and works on checksum-verified RAM copies only. What the modem writes is lost at stop, except its writes to `efs/nv_normal.bin` (expected to hold the carrier profile, network and IMS data; the runbook confirms it) when the NV overlay is on (section 1): those are captured every 30 minutes and at the unit's stop, after the CP is off, and laid over the verified copy at the next start. `reboot-to` (and `bin/omamobile push`) skips that stop, so stop the modem first (section 2) or run `omamobile-modem nv-overlay capture`. Don't give the chroot a real block device. [MODEM-PLAN](device/cheetah/MODEM-PLAN.md) "NV overlay" diff --git a/device/tegu/MODEM-PLAN.md b/device/tegu/MODEM-PLAN.md index dd96bfe5f..de0f61e70 100644 --- a/device/tegu/MODEM-PLAN.md +++ b/device/tegu/MODEM-PLAN.md @@ -1714,6 +1714,198 @@ the test), on `7.3.0-rc6-omamobile-push1-dbg-gff4fdf58035e`: error lines. It doesn't settle the Wi-Fi loss above, which also happened with one state. +### Suspend with the modem on, with the sleep guard: it sleeps and keeps the phone service (2026-11-13 and the night after) + +A test kernel only; nothing here is in a published kernel or a package yet, +so the modem's inhibitor and the rule in AGENTS.md stand as they are. + +The kernel: `7.3.0-rc6-omamobile-push1-gc50fd8fdf42a`, which is the +published `omamobile` at `2db2bc35beec` plus kernel #247 (the HSI0 USI +gates), linux-omarchy-mobile#232's four commits (`f4e5e4846c39`: the link +driver refuses a system sleep while the modem's link is up, keeps the PCI +core's sleep handling off the modem's root port and leaves the endpoint +out of system sleep), the maintainer's two patches from #232's review +(6.1: the refusal moved to a PM notifier before the freeze, the root port +a syscore device; 6.2: a ring trace) and one line of ours: all of it is +behind `sleep_guard`, which only gs201 sets, so `.sleep_guard = true` in +`zumapro_pcie_cp_data`. Userspace was main's with #697's phone packages. +The 9a has nothing that tells the modem the screen state, so each test +sent "screen off" through sitd's socket before a request and "screen on" +after, waited for the link to be down for 5 s, armed an RTC alarm and +passed the modem's inhibitor with `systemctl suspend -i`. On the charger, +cable in, home LTE with IMS voice and SMS, and every boot ended in +`systemctl reboot`. + +**These runs were outside the rule as AGENTS.md has it.** With the modem +up, every request here went past its sleep inhibitor (`systemctl suspend +-i`) with nobody at the palmtop, on a kernel that was new that day and had +not had an attended run first. Their boots began, by the palmtop's journal +(MST): on 2026-11-13 the five comparison boots on the daily kernel from +15:10 to 16:04, the refusals and the three 45 s sleeps at 16:36, the twenty +at 17:02, the boot that hung at 17:51 and the night at 19:28; on 11-14 the +boots with #556's package and the calls at 04:09, 05:37, 06:29 and 06:56, +and the boot from 10:31 on 11-14 to 07:23 on 11-15 on +`push1-g716f6d8ef355` (#232, #258's line and #247), with the modem up all +boot, the shell's auto-sleep doing the sleeping and the cable out for +10.8 h overnight. That night 37 of 37 sleep requests woke (24 on +`cp2ap_wakeup`, 11 on the alarm, two by the owner in the morning), with no +hang, UFS line or oops, and in the morning the CP was ONLINE and +registered with IMS, but a ping over `rmnet0` got 0 of 5 back with the +bearer reported up. `rmnet0` had carried almost nothing that boot and sitd +was 0.3.37, without #817's reconnect, so the night can't say whether the +internet bearer was dropped and not set up again, as on the 7 Pro (#817). +Beside it, the Wi-Fi night below ended with `rmnet0` answering 5 of 5. +From 2026-11-15 the owner's palmtop runs these under the maintainer's +decision in [#809](https://github.com/derekross/omarchy-mobile/issues/809), +whose terms, as AGENTS.md has them, are: W4HO-ham's modem-up sleep series +may run unattended, a new kernel's first among them, on the charger +(`modem-sleep-series.sh` refuses and stops off it); every boot that had a +sleep with the modem up ends in `systemctl reboot`, never `reboot-to`; a +new kernel's first run is at most 10 sleeps and its result is posted +before a longer one; the series stops itself on the CP not ONLINE, a UFS +error line or an oops, and a hang is left to the cluster watchdog or the +owner; the NV partitions stay read-only and are checked against the +backup after any boot that hung or reset; the record says which rule each +run ran under. Every other 9a keeps the rule as AGENTS.md has it. +The series script is `device/tegu/bringup/modem-sleep-series.sh` (the copy +that ran logged to another file and lacked the check that `rtcwake` armed +the alarm; otherwise the same). + +- **The same day without the guard, for comparison** (the daily kernel #100 + with its out-of-tree link driver, one request per boot, five boots). As + they were: in one boot the link was down at the request and the modem + was lost as on 11-09 (`pcieport … D0 to D3hot, device inaccessible`, + `Data Link Layer Link Active not set in 100 msec`, `Can't set D0 state`, + 33 failed polls in four minutes); in the other the endpoint was in D0 at + the request and nothing was lost. With 0 written to the endpoint's + `d3cold_allowed` first, three boots, the link down at each request: none + lost the modem. That fits #232's reading (the root port marked D3cold, + and the resume waiting for a link that isn't coming). Both boots that + entered a sleep then failed to shut down: `systemctl reboot` logged + nothing more, the palmtop came back 57 s later and the modem's stop was + not recorded, which the modem's guard counted as unclean ends. The NV + partitions read SAME against the backup afterwards. +- **A refusal no longer costs the phone service.** 14 requests in one boot + were refused by the notifier (`sleep refused before the freeze: link up, + PHY out of isolation`), twelve of them under `ping -i 1 -I rmnet0`: 273 + of 273 and 200 of 200 replies, no failed poll in sitd. +- **It sleeps.** With the cellular data service stopped: three requests of + 45 s in that boot, all entered (45.2 s and 44.6 s to the alarm, one of + 15.1 s ended by `cp2ap_wakeup`), then twenty of 60 s in the next boot, + 19 entered (17 to the alarm, one ended at 27.7 s by `cp2ap_wakeup`, one + at 27.8 s by the PMIC's interrupt) and one refused. After each: the CP + ONLINE, sitd's polls answered, `ping -I rmnet0` 5 of 5. No `Unable to + change power state`, `Link Active`, `BME` or UFS error-handler line, the + root port in D0 and the endpoint in D3hot at the end, and both + `systemctl reboot`s stopped the modem cleanly (`CP OFFLINE`, the marker + removed), back in 25 s. Mobile data was checked after every sleep here, + but with the data service stopped, so not in the shape that lost it on + the 7 Pro (below). +- **With Wi-Fi connected and the data service as on main, every request is + refused**: those 14 were all there were. At logind's sleep announcement NetworkManager takes + `wlan0` down, the bearer's metric-700 default route is the only one + left, whatever was talking over Wi-Fi goes out over `rmnet0` in the same + second, and the link is up when the notifier looks. The fix, written + as omarchy-mobile#750 and now part of #556, has `omamobile-sit-data` + remove its route at the announcement and put it back once Wi-Fi's is + back, or after 10 s without one. Put + back at once after the wake it was worse than useless: 18 of 20 entered + and none reached the alarm, the answers to the few seconds of traffic + ending each next sleep early. +- **The night: 300 requests of 60 s in one boot, 299 entered and woke, 1 + refused** (7 h 40 min, Wi-Fi connected, the data service running with + #750's change copied over the packaged file for the test and put back + after). `suspend_stats` success 299, fail 1. 239 ended on the PMIC's + interrupt (the RTC alarm) and 60 on `cp2ap_wakeup`. After every one of + the 300: the CP ONLINE, sitd active, no failed poll, Wi-Fi up, no UFS + line; the default routes read Wi-Fi first with the bearer behind before + every request, and the service logged the route back behind Wi-Fi 300 + times. At the end: registered with IMS voice, `ping -I rmnet0` 5 of 5, + no bad line in the kernel log of the boot (the list above, plus oops, + warning and RCU stall). Entry took 0.52 to 0.68 s to the noirq stage and + resume 0.72 to 0.87 s by the kernel's stamps. The reboot after it + stopped the modem cleanly. **Where data was checked:** once, at the end. + At every wake Wi-Fi was the first route and the bearer's was behind it, + so the night doesn't show whether data over the bearer answered just + after each wake, which is what failed on the 7 Pro. The boot's kernel + log, read again on 11-15, has 1,205 `Reserve doorbell interrupt` lines, + 1,142 of them `PCI not powered on`, the line the 7 Pro logged while its + data was down; here data answered at the end. The daily kernel logs the + same line awake with no sleep at all (23 to 97 in four boots of 7 to 44 + minutes on 11-13 and 11-14), so on the 9a the line alone doesn't mark + lost data. +- **One hang, awake, on this build, not understood.** The evening before + the night run, a boot that had run 52 requests (49 entered, 3 refused) + stopped 20.4 s after a sleep had ended normally: the console's last + lines are the `cp2ap_wakeup` edge that starts every link power-up, and + the `zuma-pcie-phy … external PLL lock` line that follows about 12 ms + later in every other cycle never came. The watchdog reset the palmtop + about two minutes later (`APC Watchdog Early`, `RST_STAT: 0x1 - + CLUSTER0_NONCPU_WDTRESET`); nobody touched it. That was the boot's + 1,549th link power-up; the night's boot did 6,311 without one. It is on + the in-tree link driver's power-up path (`pcie-gs201-cp` with + `phy-zuma-pcie`), which the daily kernel doesn't use (it has the + out-of-tree copy), and whether it happens without #232 isn't known. + Reported on #232. The next boot started the modem after the one unclean + end the guard allows. +- **The alarm is not always what ends a 60 s sleep, and not always at + 60 s.** In the night's 239 alarm wakes, by the RTC from the arming: 223 + at 58 to 64 s, 12 at 28 to 36 s, 3 at 89 to 90 s and one at 121 s; five + more sleeps ran 83 to 93 s and were ended by the modem. The PMIC's + interrupt counted 522 in the boot, twice `rtc-alarm0`'s 261, so every + PMIC wake was an RTC alarm: the RTC raised one about 30 s early or 30 to + 60 s late now and then. The 300 modem-off sleeps on this kernel's base + earlier on 11-13 were all 59.2 to 60.3 s. Not traced; a second RTC timer beside + `rtcwake`'s is our guess (the kernel's alarmtimer programs the RTC at + suspend for the next userspace alarm), and the `rtc` and `alarmtimer` + trace events over a run would show it. The palmtop woke every time. + +- **With omarchy-mobile#556's package nothing is done by hand** (2026-11-14, + its head `6eddef24` merged with #697 for the build, `[control] + follow_screen = yes`, the same kernel): 30 requests of 60 s, 30 entered, + 24 ended by the alarm and 6 by the modem. sitd told the modem "screen + off" once, 16 s after its start with the panel dark, and the sleep hook + logged "screen off before sleep told" 30 times; the link was already + idle at 26 of the 30 requests, and the whole 46-minute boot had 97 link + power-ups, where the modem on its "screen on" reports does about 49 a + minute. The route came back behind Wi-Fi 30 of 30 times. After each + sleep the CP was ONLINE with no failed poll, and `ping -I rmnet0` got 5 + of 5 at the end, with Wi-Fi the first route; data wasn't checked after + each wake. + +- **A call and a text wake the sleeping palmtop** (2026-11-14, the same + build and package; the owner called and texted from another phone, the + palmtop untouched and no call answered). Four calls arrived in sleeps + of 82 s to 6 min 17 s: each woke it on `cp2ap_wakeup`, and sitd logged + "Incoming call ringing" 0.1 to 0.8 s after the kernel's suspend exit. + Two texts arrived 8 s and 15 s into a sleep and were stored 0.06 and + 0.08 s after the exit. Six sleep requests that morning were called off + (three refused with the link up, three ended in noirq by CPIF with the + CP's wake line pending), and calls and texts kept arriving after each. +- **One of those calls left a stale `ringing-in` in sitd**, which is + omarchy-mobile#729 (its fix for a declined call is #754): the modem + refused the hang-up with error 2 and sitd stopped itself 12 s later. + Until then, with `follow_screen` on, sitd never told the modem "screen + off" for the call it still held, the modem stayed at 48 link power-ups + a minute, and three sleep requests in a row were called off. Not seen + again in three more calls into a sleep. + +What this leaves before the modem's inhibitor can go on the 9a: a release +kernel with the zumapro guard (#232 is in kernel-2026.11.15, #258 merged on +2026-11-15 and is in no release yet); the hang above understood, or at +least counted on a build without #232; `follow_screen` turned on for the +9a (#556 is in main since #791); a stale call ended without the owner's +help (#729), since it keeps the palmtop awake; and what no run here +tested: **mobile data in the shape that lost it on the 7 Pro** (#799): +Wi-Fi off for the sleeps so the bearer is the only route, RTC wakes, and +data over `rmnet0` (a ping and a DNS lookup) a few seconds after every +wake, with sitd's data-call lines from that sleep next to it, so a drop +and its reconnect (#817) show up wake by wake. `modem-sleep-series.sh` +with `WIFI=off` does all of that, and stops itself as #809 asks: off the +charger, on the CP not ONLINE, and on a UFS error line or an oops in +`dmesg`. +Then a call answered after it woke the palmtop, and a sleep of hours. + ### M2: the link only (modem link driver, `allow_hw=1`; risk: medium) One step per run, the CPIF overlay not applied, the link overlay applied @@ -1903,7 +2095,7 @@ Preconditions, as SLEEP-PLAN's: | The bridge for apps | `omamobile-mmd` not installed | not part of this test (T2 follows a passing run) | | The link watcher | `omamobile-modem` 0.1.9 () keeps one open file on the boot device, which retires the old open-and-close watcher that woke the CP link every half second; 0.1.10 () adds the once-per-boot refusal | with the CP ONLINE the link idles and cycles on its own; on 2026-10-29 the palmtop hung eight times in a morning with the modem up for hours, the early ones tied to the watcher's open and close of the boot device (the console logs end on CPIF's `cp2ap_wakeup` wake toggles; `logs/tegu-review-20261029/hangs/`). The run uses 0.1.10 (SIM-1). Keep the session short (under an hour) and attended; `cpif-keep-link.ko` (G3-L1 step 1, untested) is the candidate fix, not part of this run | | Call audio | the AoC card is `google,tegu-aoc-snd-card` with its EP PCMs, but `omamobile-speaker-protection` is inactive and the amplifiers are TAS2572, which `voice_audio.py`'s earpiece checks (CS35L41 controls) don't know | **no audio routing exists on the 9a: calls are not part of this test.** `[audio] card = none` stays; a signalling-only call (cheetah's S10) is a later, separate step | -| Suspend | unqualified with the modem on (section "Suspend with the modem on"); the start script's inhibitor blocks `systemctl suspend` for the boot | charger in for the whole test | +| Suspend | unqualified with the modem on (section "Suspend with the modem on"); the start script's inhibitor blocks `systemctl suspend` for the boot. On a test kernel with linux-omarchy-mobile#232 and the sleep guard on for zumapro, 299 of 300 sleeps were entered and woke with the phone service kept (2026-11-13; Wi-Fi connected, mobile data checked at the end) | charger in for the whole test | | Identifiers | the ICCID (printed on the card), IMSI, IMEI, phone number, the carrier's name, its APN, IP addresses and message text never go into the repository, a log excerpt or a screenshot | sitd logs states and codes only; before quoting a log: `grep -nE '[0-9]{7,}|\+[0-9]|[0-9a-f]{2}(:[0-9a-f]{2}){5}|@' logs/` must show nothing you would quote | **What to expect, honestly:** SIM detection likely (GET_SIM_STATUS has diff --git a/device/tegu/bringup/modem-sleep-series.sh b/device/tegu/bringup/modem-sleep-series.sh new file mode 100755 index 000000000..bb8039726 --- /dev/null +++ b/device/tegu/bringup/modem-sleep-series.sh @@ -0,0 +1,134 @@ +#!/bin/bash +# SPDX-License-Identifier: MIT +# On the palmtop: N systemd sleeps of S seconds with the modem up, for a kernel whose +# link driver has the sleep guard on for zumapro (linux-omarchy-mobile#232; MODEM-PLAN, +# "Suspend with the modem on, with the sleep guard"). On a kernel without it one sleep +# costs the phone service until a reboot: use modem-sleep-test.sh there, attended. +# Before each request the modem is told "screen off" through sitd's socket and the +# script waits until the link driver's last line is "link down" and 5 s old; after it, +# "screen on". The modem's block inhibitor is passed with "systemctl suspend -i", an RTC +# alarm is the wake. The cellular data service is started if it isn't running, and the +# default routes are logged before each request: with Wi-Fi connected and the service +# as on main, the bearer's route is the only one left at the freeze and every request +# is refused (fixed by omarchy-mobile#556). 5 s after each wake: a ping and a DNS lookup over the +# bearer's interface (rmnet0), so mobile data is checked after every sleep, not only at the +# end, with sitd's data-call lines from that sleep ("Data-call list changed", "Internet +# bearer set up again", #817) beside it; 20 s after: the CP's state, sitd, its failed polls +# and Wi-Fi. WIFI=off turns the Wi-Fi radio off for the run (back on when the script ends, +# however it ends), so mobile data is the only route at each wake, the shape that lost data +# on the 7 Pro (omarchy-mobile#799, SUSPEND-PLAN M6-D). Stops itself, as omarchy-mobile#809 +# asks: off the charger (and refuses to start there), on the CP not ONLINE, and on a new UFS +# error line or an oops, both read from dmesg, which is in memory, so the check still works +# if storage is what failed. The end counts the boot's "Reserve doorbell interrupt" and +# "PCI not powered on" lines. +# Before each request and with each data check, one line of CPIF's own state, as the 7 Pro's +# M6-D recorder reads it (device/cheetah/bringup/m6-data-capture.sh): the FMT and raw queues' +# heads and tails, the wake lines, pcie_powered_on and reserve_doorbell, each pktproc queue's +# fore/rear and pass, and the mif_cp2ap_msg count, so a failed wake says whether uplink, +# downlink or the control channel stopped. +# On W4HO-ham's palmtop under #809's terms (AGENTS.md, the 9a's modem-up entry), everywhere +# else attended; on the charger either way, and the boot ends in a reboot. N defaults to 10, +# the most a new kernel's first run may have: +# [WIFI=off] systemd-run --unit=x-modem-sleep-series /root/modem-sleep-series.sh [N] [S] [append] +# The last step afterwards is "systemctl reboot", never reboot-to. +L=/var/log/omamobile/modem-sleep-series.log; CP=/usr/lib/omamobile-vendor/omamobile-modem-cp +N=${1:-10}; S=${2:-60}; H=1.1.1.1; Q=example.com; BAT=/sys/class/power_supply/max77779-battery +U='link startup failed|Host init failed|I/O error, dev|ufshcd_err_handler' +# An oops, a BUG or a kernel panic path in dmesg +O='Internal error:|Unable to handle kernel|kernel BUG at|BUG: |Oops' +bad() { dmesg | grep -cE "$1"; } +say() { echo "$(date +%T) $*" >> $L; sync "$L" 2>/dev/null; } +screen() { python3 - "$1" <<'PY' 2>&1 | tail -1 +import json, socket, sys +s = socket.socket(socket.AF_UNIX); s.settimeout(15); s.connect('/run/omamobile-sit/socket') +s.sendall(json.dumps({'action': 'screen-state', 'on': sys.argv[1] == 'on'}).encode() + b'\n') +print(s.recv(4096).decode().strip()[:120]) +PY +} +st() { echo "cp $($CP state 2>&1 | tail -1), sitd $(systemctl is-active omamobile-sitd), poll failures $(journalctl -b -u omamobile-sitd --no-pager | grep -c 'poll failed'), wlan0 $(cat /sys/class/net/wlan0/operstate 2>/dev/null || echo gone)"; } +idle() { # wait until the link has been down for 5 s; prints how long it took + local t0=$SECONDS last + while ((SECONDS - t0 < 60)); do + last=$(journalctl -b -k --no-pager --since "-5 s" | grep -cE "gs201-pcie-cp .*(link up|link down)") + [[ $last == 0 ]] && journalctl -b -k --no-pager -n 400 | grep -E "gs201-pcie-cp .*(link up|link down)" | tail -1 | grep -c "link down" >/dev/null && { echo "idle after $((SECONDS - t0)) s"; return 0; } + sleep 1 + done + echo "not idle in 60 s"; return 1 +} +# On the charger: the battery charging, or full with the cable in (the charger then holds it +# and the current is about 0; a series stopped at sleep 47 on 2026-11-15 when the pack filled). +# Off the cable the status is Discharging. +charging() { case $(cat $BAT/status 2>/dev/null) in Charging | Full) return 0 ;; esac + (( $(cat $BAT/current_avg 2>/dev/null || echo 0) > 0 )); } +data() { # a ping and a DNS lookup over the bearer's interface + [[ -n $IF ]] || { echo "no bearer interface"; return; } + echo "ping $(ping -c 3 -W 3 -I "$IF" "$H" 2>&1 | grep -o '[0-9]* received'), dns $(dns)" +} +cpif() { # one line of CPIF's state (read-only sysfs and /proc/interrupts) + local c=/sys/devices/platform/cpif + [[ -r $c/legacy_queues ]] || { echo "no cpif state"; return; } + echo "fmt $(grep -oE '^FMT: tx head:[0-9]+ tail:[0-9]+|rx head:[0-9]+ tail:[0-9]+' $c/legacy_queues | head -2 | grep -oE '[0-9]+' | paste -sd/)" \ + "raw $(grep '^NORM_RAW' $c/legacy_queues | grep -oE '(head|tail):[0-9]+' | cut -d: -f2 | paste -sd/)" \ + "$(grep -oE '(cp2ap_wakeup|ap2cp_wakeup|pcie_powered_on|reserve_doorbell):[0-9]+' $c/legacy_queues | paste -sd' ')" \ + "dl $(awk '/fore\/rear\/done/ {f=$NF} /^ *pass:/ {printf "%s,%s ", f, $1}' $c/pktproc/status | sed 's/pass://g; s|fore/rear/done:||g')" \ + "ul $(awk '/fore\/rear:/ {f=$NF} /^ *pass:/ {printf "%s,%s ", f, $1}' $c/pktproc_ul/status | sed 's/pass://g; s|fore/rear:||g')" \ + "msg $(awk '/mif_cp2ap_msg/ {s=0; for (i = 2; i <= 9; i++) s += $i; print s}' /proc/interrupts)" +} +dns() { # one A query for $Q to the bearer's first DNS server, sent out of $IF itself: with Wi-Fi + # connected the bearer's link is not resolved's default route, so resolvectl won't use it + local srv; srv=$(resolvectl dns "$IF" 2>/dev/null | awk '{print $4}') + [[ -n $srv ]] || { echo "no server"; return; } + python3 - "$IF" "$srv" "$Q" <<'PY' 2>&1 | tail -1 +import socket, struct, sys +dev, srv, name = sys.argv[1:4] +fam = socket.AF_INET6 if ':' in srv else socket.AF_INET +q = struct.pack('>HHHHHH', 0x5a5a, 0x0100, 1, 0, 0, 0) +q += b''.join(bytes([len(x)]) + x.encode() for x in name.split('.')) + b'\0' + struct.pack('>HH', 1, 1) +s = socket.socket(fam, socket.SOCK_DGRAM); s.settimeout(4) +s.setsockopt(socket.SOL_SOCKET, socket.SO_BINDTODEVICE, dev.encode()) +try: + s.sendto(q, (srv, 53)); r = s.recv(512) + ok = r[:2] == q[:2] and r[3] & 0x0f == 0 and struct.unpack('>H', r[6:8])[0] > 0 + print('ok' if ok else 'bad answer') +except OSError as e: + print('fail (%s)' % (e.strerror or 'timeout')) +PY +} +[[ -e /sys/class/rtc/rtc0/wakealarm ]] || { echo "no RTC wake alarm on this kernel" >&2; exit 1; } +charging || { echo "not charging: run this on the charger" >&2; exit 1; } +[[ ${3:-} == append ]] || : > $L +systemctl is-active -q omamobile-sit-data || { systemctl start omamobile-sit-data; sleep 8; } +[[ ${WIFI:-} == off ]] && { trap 'nmcli radio wifi on' EXIT; nmcli radio wifi off; sleep 10; } +IF=$(cat /run/omamobile-sit/data-interface /run/omamobile-sit/data-interface-kept 2>/dev/null | head -1) +say "$(uname -r) $(uname -v | cut -d' ' -f1); the data service: $(systemctl is-active omamobile-sit-data), device ${IF:-none}, Wi-Fi radio $(nmcli radio wifi), default via $(ip route show default | awk '{print $5}' | paste -sd,); $(st); data: $(data)" +sleep 10 +for n in $(seq 1 "$N"); do + s0=$(cat /sys/power/suspend_stats/success); f0=$(cat /sys/power/suspend_stats/fail); u0=$(bad "$U"); o0=$(bad "$O"); t0=$(date +%s) + charging || { say "sleep $n: not charging any more: stop"; break; } + [[ ${WIFI:-} == off ]] || for _ in $(seq 1 30); do ip route show default | grep -c wlan0 >/dev/null && break; sleep 1; done + say "sleep $n: default via $(ip route show default | awk '{print $5}' | paste -sd,); screen off -> $(screen off); link: $(idle); link-ups in the last 20 s: $(journalctl -b -k --no-pager --since '-20 s' | grep -c 'link up: Gen3')" + say "sleep $n, before: cpif $(cpif)" + rtcwake -m no -s "$S" >/dev/null 2>&1 || { say "sleep $n: rtcwake could not arm the alarm: stop"; break; } + r0=$(cat /sys/class/rtc/rtc0/since_epoch) + systemctl suspend -i + for _ in $(seq 1 300); do [[ $(cat /sys/power/suspend_stats/success) != "$s0" || $(cat /sys/power/suspend_stats/fail) != "$f0" ]] && break; sleep 0.5; done + r1=$(cat /sys/class/rtc/rtc0/since_epoch); sleep 3; rtcwake -m disable >/dev/null 2>&1 + if [[ $(cat /sys/power/suspend_stats/success) != "$s0" ]]; then + irq=$(cat /sys/power/pm_wakeup_irq 2>/dev/null) + res="entered, $((r1 - r0)) s, wake irq $irq ($(grep -E "^ *$irq:" /proc/interrupts | awk '{print $NF}'))" + elif [[ $(cat /sys/power/suspend_stats/fail) != "$f0" ]]; then res="refused (fail $(cat /sys/power/suspend_stats/fail))" + else res="no change in suspend_stats after 150 s"; fi + say "sleep $n: $res; screen on -> $(screen on)" + sleep 5 + say "sleep $n, 5 s after: default via $(ip route show default | awk '{print $5}' | paste -sd,); cpif $(cpif); data: $(data); cpif after the data check: $(cpif); sitd's data-call lines this sleep: $(journalctl -u omamobile-sitd --since "@$t0" --no-pager -o cat | grep -E 'Data-call list changed|Internet bearer' | cut -c1-160 | paste -sd'|' | sed 's/^$/none/')" + sleep 15 + say "sleep $n, 20 s after: $(st), new ufs lines $(( $(bad "$U") - u0 )), new oops lines $(( $(bad "$O") - o0 )), sitd's last line: $(journalctl -b -u omamobile-sitd --no-pager | tail -1 | cut -c40-120)" + (( $(bad "$U") > u0 )) && { say "a UFS error line in dmesg: stop"; dmesg | grep -E "$U" | tail -3 >> $L; break; } + (( $(bad "$O") > o0 )) && { say "an oops in dmesg: stop"; dmesg | grep -E "$O" | tail -3 >> $L; break; } + [[ $($CP state 2>&1 | tail -1) == ONLINE ]] || { say "the CP is not ONLINE: stop"; break; } +done +[[ -n $IF ]] && say "ping over $IF: $(ping -c 5 -W 3 -I "$IF" "$H" 2>&1 | grep -o '[0-9]* received')" +say "end: $(st); in the boot's kernel log: Reserve doorbell interrupt $(journalctl -b -k --no-pager | grep -c 'Reserve doorbell interrupt'), PCI not powered on $(journalctl -b -k --no-pager | grep -c 'PCI not powered on')" +say "kernel lines of the boot:" +journalctl -b -k --no-pager | grep -E "sleep refused|Unable to change power state|Link Active not set|Can.t set D0|BME is not set|link-down recovery|suspend_cp: abort|failed to suspend" | sed -E 's/^.{16}[^ ]+ kernel: //' | cut -c1-110 | sort | uniq -c | sort -rn | head -20 >> $L +say "done"