Skip to content

raven: BCM4389 Bluetooth start-up after Omarchy kernels with the CMU_APM override #77

Open

Description

@robotosaith

Split out of #41, as suggested in review, so the follow-up tests don't get lost. Details are in raven's BLUETOOTH-PLAN (section B3) on #41's branch.

What happened. On 2026-10-26, GrapheneOS on raven (Pixel 6 Pro) reported a Bluetooth stack crash at boot, and the BCM4389 only came up after a few presses of the Bluetooth tile (each a power cycle and firmware download). Before the Omarchy work, GrapheneOS booted Bluetooth cleanly. Omarchy had also seen the chip go silent after a forced restart (B3, 2026-10-21): the baud rate change and HCI_Reset timed out, and no reset we tried brought it back, including dropping BT_REG_ON and WL_REG_ON.

Best explanation so far. State carried across a warm reboot from Omarchy kernels that set the old assigned-clock-parents override, which wrote a reserved bit in the always-on CMU_APM (found in review of linux-omarchy-mobile #6; v2 drops it). The chip's supplies come straight from the battery (vbat and vddio are dummy regulators), so nothing power-cycles it across a warm reset.

Evidence since:

  • 2026-10-26: the first GrapheneOS boot after a kernel with the CMU_APM fix (gb3d6253b0d61, warm path through fastboot): Bluetooth came up cleanly.
  • 2026-10-27: GrapheneOS powered off from its menu, then powered on (a cold boot): clean again, no crash, no retries.

Two clean boots, not proof.

Open:

  1. A GrapheneOS boot straight after an Omarchy kernel with the override, to see whether the crash comes back (only if we're willing to reproduce it).
  2. Bluetooth at the first Omarchy boot after GrapheneOS, warm and cold.
  3. On Omarchy, do what Android does: a boot-time retry that power-cycles the controller a few times, logging each attempt. If it helps, move it into hci_bcm's setup for upstream.

Activity

  1. robotosaith commented on Oct 27, 2026

    @robotosaith
    ContributorAuthor

    New data point, on Omarchy this time (2026-10-27, kernel gb3d6253b0d61, which has the CMU_APM fix from derekross/linux-omarchy-mobile#6 v2):

    • Bluetooth came up on three boots of this kernel during the day, then a plain warm reboot (bin/omamobile push) found the chip silent again: 0xfc18 timed out, and the HCI_Reset after it too, on both probes. So the override isn't the whole story; a warm reset can leave the chip in this state on the fixed kernel too.
    • A real cold boot cleared it: systemctl poweroff with the cable out (so the bootloader can't restart it), about 30 s off, then Power. The first 0xfc18 still timed out, but the driver's retry after a reset answered, the patch loaded and the controller powered up. That's the first thing found on Omarchy that brings it back; warm reboots, forced restarts and dropping BT_REG_ON/WL_REG_ON never did.

    So it looks like state that only a power cut clears, which points at the power-up sequence. Next on our side: a longer BT_REG_ON hold before power-up (hci_bcm holds it 30 ms for the BCM4389; stock's HAL uses 100 ms) and a boot-time retry that power-cycles the controller as Android does. Details in raven's BLUETOOTH-PLAN (B3) on #41's branch.

  2. derekross commented on Oct 27, 2026

    @derekross
    Owner

    Thanks for splitting this out and for the new data point. The cold-boot result is the most useful fact so far, and I'd like to make sure the next steps go after what it actually tells us. I read hci_bcm with #6 in, Google's nitrous.c (google-modules/bluetooth/broadcom, the raviole android14 branch) and your BLUETOOTH-PLAN tables again, and I don't think either of the two proposed steps is where the answer is:

    A longer BT_REG_ON hold. hci_bcm's 30 ms is Google's own number: nitrous_rfkill_set_power() does REG_ON low, msleep(30), REG_ON high, DEV_WAKE high, then a 10-20 ms wait. The HAL's BigHammerBtRegOnDelay = 100 is its retry step, not the kernel's power-on hold. More to the point, your plan has already held the line low far longer than 100 ms without effect: about a second on every rebind omamobile-bt-addr does, 4 s by hand on 10-21, and both REG_ONs for 2-3 s with Wi-Fi unloaded on 10-16. And B1 on 10-20 sent the chip into nine warm reboots with REG_ON already low, and all nine stayed silent. So the silent state survives the BT core being held in reset for seconds. A longer hold at power-on can't be the fix.

    A boot-time retry that power-cycles the controller. The package already does this: omamobile-bt-addr rebinds hci_uart_bcm up to three times a boot, and that's a full power-down and power-up through bcm_close()/bcm_open(). It helped in the "first command lost" case on 10-16 and it was running on the silent boots that didn't come back. Android's "big hammer" is the same thing with more tries. I don't think more tries change the outcome, and I wouldn't take a retry loop into hci_bcm without understanding what it's retrying against.

    What the data says instead: on a silent boot the APM UART read sane with CTS asserted and tx:28 rx:0, so the host transmits and the chip answers nothing; every REG_ON-based reset fails; only a PMIC reset or a real power-off clears it. The things a power-off touches that nothing else does are outside our device tree: the chip's VDDIO rail and its 32 kHz LPO, both of which our DT doesn't describe (vbat and vddio are dummy regulators, and the node has no clock), so Linux never drops or re-enables them. Your own 10-16 note already pointed there. Two things I'd do next, both read-only:

    1. Find which PMIC output feeds the BCM4389's VDDIO and which provides its 32 kHz clock on raven, from Google's raviole device tree and the S2MPG10 register header, and read that output's enable through the kernel's PMIC path on a good boot and on a silent one. If the LPO buffer or the rail is off on the silent boot, we have the cause and a one-line fix; if it's on in both, we can rule the PMIC out.
    2. Before each reboot, log the controller's state: power/runtime_status of the serdev device (whether hci_bcm had runtime-suspended it, which drops BT_DEV_WAKE and puts the chip into its low-power mode), whether Bluetooth was in use, and the level of BT_HOST_WAKE (gpa6-6). Three good boots and one silent one on the same kernel means the difference is in what the chip was doing when the reset hit, and that's cheap to correlate.

    If those show nothing, the next thing I'd try is one change per series, each started from a forced restart: the tools/bt-uart-test hci run on a silent boot with the device-wake line both ways (your plan's step 2), to see whether the chip answers a raw HCI_Reset at 115200 when nothing else has touched it.

    Please keep the results in BLUETOOTH-PLAN B3 as you have been; it's a good record.

  3. robotosaith commented on Nov 3, 2026

    @robotosaith
    ContributorAuthor

    Seen again on 2026-11-03 (kernel ge1e6adb69bbf, which has the BT_REG_ON reset pulse): after 28 minutes up with A2DP audio to a speaker, a clean systemctl reboot left the BCM4389 silent on the next boot (command 0xfc18 tx timeout, BCM: Reset failed (-110), Failed to set baudrate, retried twice). Four reboot-to restarts earlier that day (no shutdown) all brought Bluetooth up. A forced restart (Power about 30 s) recovered it, and the paired speaker reconnected on its own. The serdev hci_bcm driver has no shutdown hook, so the kernel doesn't touch BT_REG_ON at reboot; bluetoothd only unregistered its media endpoints at stop. Next: clean reboots with Bluetooth idle versus right after audio, to find the trigger.

  4. robotosaith commented on Nov 3, 2026

    @robotosaith
    ContributorAuthor

    A correction and a result.

    Correction: my last comment pointed at "Wi-Fi and Bluetooth both busy before the shutdown". That was one failure in seven reboots, and the same silent chip came after reboots with no Bluetooth audio at all (B1, B3, the 10-27 push), at about one boot in five, so audio isn't the cause. #395's BLUETOOTH-PLAN entry now says so.

    Result: a test kernel (the daily one plus a single change, hci_bcm's BCM4389 power_on_delay_ms from 30 to 100 ms, as stock's driver holds BT_REG_ON low before power-up; 7.3.0-rc6-omamobile-push1-gcbf4d42277a6), 2026-11-03:

    30 ms (daily kernel) 100 ms (test kernel)
    clean systemctl reboot 1 silent in 7 that day, about 1 in 5 since B1 0 in 15
    first boot of the new kernel (the push's reboot-to from the 30 ms kernel) silent (0xfc18 timeouts, BCM: Reset failed (-110))
    recovery from a silent chip a forced restart only (B1, B3, that morning) the next clean reboot, on its own

    At the old rate, 15 clean reboots in a row would happen by chance about one time in fifteen, so it's strong evidence rather than proof, and the one failure came straight out of a 30 ms kernel's run. The longer hold is now on the raven phone daily; if it holds up over the next days, it becomes a one-line hci_bcm patch for the BCM4389.

  5. derekross commented on Nov 3, 2026

    @derekross
    Owner

    Thank you, this is a real result, and it means part of what I wrote here on 10-27 was wrong. I said a longer hold at power-on couldn't be the fix, because the plan had already held BT_REG_ON low for seconds. Those holds were with the driver unbound (the UART closed). power_on_delay_ms holds the line low with the UART already open, right before raising it (hci_bcm's own comment says so), and that sequence the earlier tests never varied. 0 of 15 against about 1 in 5 is strong, as you say. The recovery is interesting too: earlier, no warm path ever brought the chip back, and on the 100 ms kernel the next clean reboot did.

    One correction for the plan and the eventual commit message: Google's driver doesn't hold it for 100 ms. nitrous_rfkill_set_power() (google-modules/bluetooth/broadcom, nitrous.c) does REG_ON low, msleep(30), high. The 100 ms is the HAL's BigHammerBtRegOnDelay, the wait between its power-off and power-on. So stock, through the HAL, ends up with the line low for well over 100 ms before power-up, which fits your result. It just isn't the driver's number.

    Before this becomes a patch:

    • Could you push the test commit (gcbf4d42277a6 isn't on GitHub yet) to a branch on your fork, so the run is tied to code we can read?
    • bcm4389_device_data is shared with the Pixel 7 Pro (gs201-cheetah.dts uses brcm,bcm4389-bt too), so changing 30 to 100 there changes cheetah's Bluetooth power-up. Under CONTRIBUTING §6.2 that needs either something only raven's board sets, or a reason that holds on the 7 Pro as well. I haven't seen the silent chip on cheetah. Let's decide which when you open the kernel PR.
    • Please keep counting over the next days, clean systemctl reboot and reboot-to both, and put the running tally in BLUETOOTH-PLAN (B5 or the end of B4).
  6. robotosaith commented on Nov 4, 2026

    @robotosaith
    ContributorAuthor

    Thanks, and for correcting your 10-27 reasoning so clearly: holding the line with the UART open is the one sequence the earlier tests never varied.

    • The test commit is on omamobile-gs101-bt-hold100-test (one NOT FOR MERGE commit on the daily kernel; uname -r 7.3.0-rc6-omamobile-push1-gcbf4d42277a6). Its message still says "as stock's driver does"; the plan now says 100 ms is the HAL's BigHammerBtRegOnDelay and that nitrous_rfkill_set_power() holds REG_ON low 30 ms, and the real commit will say the same.
    • A first count against it: the next clean systemctl reboot on the 100 ms kernel, this morning after 14 h up (overnight, no sleep, Bluetooth in use), left the chip silent, and the reboot-to after it carried that into a 30 ms kernel. So it's 1 silent in 16 clean reboots on 100 ms, and the 15 good ones each came after about 2 minutes up. That's still better than the old 1 in 5, but it isn't a fix, and the uptime may matter. The tally is at the end of B4 in BLUETOOTH-PLAN (device/raven, DEVICES: Bluetooth audio and clean reboots, suspend to testing #395, 4c9848e), boot by boot from the journal, and I'll keep it going there with systemctl reboot and reboot-to counted separately.
    • On cheetah: agreed, we'll decide that with the kernel PR. With a failure on 100 ms, I'd hold the patch until the count says more.
  7. derekross commented on Nov 5, 2026

    @derekross
    Owner

    Thank you for pushing the branch and for posting the 14-hour failure straight away. Recording a result against the idea that looked so good is what keeps this issue useful.

    With one silent boot in 16, I don't think we can tell the 100 ms kernel from the old rate yet: at one in five, one or fewer in 16 happens about one time in seven. Your uptime column suggests something else to control, though. All 15 good boots came after about two minutes up, and both failures you've logged on clean reboots (28 minutes on 11-03, 14 hours this morning) came after much longer. So far each run differs from the others in two things, the hold and the uptime. Could the next ones change only one? For example, a few clean reboots on each kernel after an hour or more up, with Bluetooth idle and nothing else changed. If the silent chip follows uptime on both kernels, the hold isn't the fix. If it only follows uptime on 30 ms, it probably is.

    One read-only thing that might help on a long-uptime boot, just before the reboot: the controller's runtime PM state (cat /sys/bus/serial/devices/serial0-0/power/runtime_status, or wherever hci_bcm's serdev device sits on raven) and whether the HCI device is up. If the chip is in its low-power state when the reset comes, that would fit a failure that needs time to appear.

    On cheetah, agreed: nothing to decide until the count says the hold helps. When it does, it would need either something only raven's board sets, or a reason that holds on the 7 Pro too, plus a run on mine before merge.

  8. robotosaith commented on Nov 10, 2026

    @robotosaith
    ContributorAuthor

    A new silent spell on raven, and the two checks from your 10-27 comment. Kernel 7.3.0-rc6-omamobile-push1-ge1e6adb69bbf (30 ms hold).

    GrapheneOS ran on the phone from 11-04 to today. Since Omarchy came back, the chip has been silent on every boot: two clean systemctl reboots, a forced restart (Power + Volume Up), and a real cold boot (systemctl poweroff, cable out, more than 60 s off, Power). Each boot shows the usual 0xfc18 tx timeout and BCM: Reset failed (-110), and the three omamobile-bt-addr rebinds don't help. Ten more rebinds by hand, each with a 3 s unbind, didn't either. So this time even a power-off didn't clear it, unlike the 10-27 cold boot.

    Raw HCI on a silent boot (one boot with the bluetooth child set disabled, nothing else in the DTB changed; tools/bt-uart-test):

    • loop --tty /dev/ttySAC1: PASS at 115200 and 3000000, 256 bytes back intact each time. The APM UART, its clock (196.6 MHz) and its IRQ work.
    • hci --tty /dev/ttySAC1 --wake gpp14:3: HCI_Reset got no answer (0 bytes), with --on-ms at the default, at 500 and at 2000, and at 115200 and 3000000. The UART counters read rx:512, all of it from the loopback. The chip sent nothing at all.
    • With brcmfmac unloaded, regulator-wlan (WL_REG_ON) stayed out hi, so I couldn't get both REG_ONs low that way. That test is still open.

    The PMIC's 32 kHz outputs (clk debugfs, read through s2mpg10-clk, so the hardware column comes from RTCBUF): rtc32k_ap, peri32k1 and peri32k2 all read enabled, at 32768 Hz. If one of them is the BCM4389's LPO, the LPO is running on a silent boot. I haven't confirmed which output feeds the chip, and VDDIO's rail is still unknown, since Google's raviole tree isn't on hand here. I don't have a good boot to compare against yet either.

    So this spell isn't tied to uptime (it started at the first boot after GrapheneOS), and a power-off doesn't end it. The one thing that changed before it is GrapheneOS running for six days with its own Bluetooth stack.

  9. robotosaith commented on Nov 10, 2026

    @robotosaith
    ContributorAuthor

    A follow-up on the VDDIO/LPO question, from Google's raviole tree (kernel/google-modules/raviole-device, branch android14-gs-pixel-6.1, gs101-common-bluetooth.dtsi and gs101-raven-bluetooth.dtsi). Stock's Bluetooth node is btbcm under &odm, compatible = "goog,nitrous", and it has no *-supply and no clocks. It only has shutdown-gpios (gpp16-2), device-wakeup-gpios (gpp14-3), host-wakeup-gpios (gpa6-6), raven's timesync-gpios (gpp11-1), and their pinctrl. nitrous.c reads only those GPIOs, wake-polarity and the default pinctrl state: no devm_clk_get, no devm_regulator_get. The wlan node under &pcie_1 has no supplies or clocks either, and gs101-pmic.dtsi has no 32 kHz node.

    So stock never switches the BCM4389's VDDIO or LPO either. Both must be rails that are always on (or set by the bootloader). That makes "rails Linux never touches" not a difference between stock and us. What stock does configure that we may not is pin state: bt_reg_on and bt_dev_wake at 2.5 mA with pin-con-pdn PREV, bt_host_wake as function F (EINT) with a pull-down, and bt_timesync (gpp11-1, function F, pull-down), which our DT leaves alone. I'll compare those pins' CON/PUD/DRV on raven against these values next, reading one controller at a time.

  10. robotosaith commented on Nov 10, 2026

    @robotosaith
    ContributorAuthor

    The pins on the same silent boot, read one controller at a time from /sys/kernel/debug/pinctrl/<controller>/pins (10840000 for gpp11/14/16, 174e0000 for gpa6):

    Pin Raven now Stock
    gpp16-2 (REG_ON) CON 0x1 (out), DAT 1, PUD 0, DRV 0 (2.5 mA), CON_PDN 0x2, PUD_PDN 0x1 out, PULL_NONE, 2.5 mA, PDN_PREV (3), PUD_PDN NONE
    gpp14-3 (DEV_WAKE) CON 0x1, DAT 1, PUD 0, DRV 0, CON_PDN 0x2, PUD_PDN 0x1 out, PULL_NONE, 2.5 mA, PDN_PREV, PUD_PDN DOWN
    gpa6-6 (HOST_WAKE) CON 0xf (EINT), PUD 0x1, DRV 0 func F, PULL_DOWN, 2.5 mA
    gpp11-1 (timesync) CON 0x0 (in), PUD 0x1, DRV 0, CON_PDN 0x2 func F, PULL_DOWN, PDN_PREV

    In normal operation everything matches stock. The differences are CON_PDN (input here, PREV on stock), which only applies in SoC power-down, REG_ON's PUD_PDN, and timesync as a plain input instead of EINT. None of these can explain a chip that doesn't answer HCI_Reset at boot. So with the rails, the LPO outputs and the pins as on stock, and the UART passing loopback, what's left is the chip's own state.

  11. robotosaith commented on Nov 10, 2026

    @robotosaith
    ContributorAuthor

    More on the same silent spell: the start sequence, done by hand on a boot with the bluetooth child disabled. I used a local copy of tools/bt-uart-test hci with a few variants added. Each one power-cycles BT_REG_ON first.

    Variant Result
    the tool's order (DEV_WAKE high, then REG_ON high), --on-ms 200 to 2000 0 bytes
    nitrous order (REG_ON low 30 ms, high, 30 ms, then DEV_WAKE high) 0 bytes
    DEV_WAKE left low 0 bytes
    tty opened only after REG_ON high 0 bytes
    no RTS/CTS 0 bytes
    RTS deasserted (pin high) through power-on, as stock's rts pinctrl state can leave it, then RTS on and CRTSCTS 0 bytes
    the same with REG_ON low for 2 s first 0 bytes
    pcie_gs201 unloaded too, so wlan-reg-on reads out lo (gpp14-2 DAT 0): the whole BCM4389 off, then BT alone 0 bytes
    a baud scan, 9600 to 4000000 (13 rates), logging anything after power-on and after HCI_Reset 0 bytes at every rate, not even line noise

    On the boot with Wi-Fi blocked entirely (brcmfmac never loaded, WL_REG_ON high from PCIe), hci_bcm's own start failed the same way. Stock's UART pins have no pulls either (uart16_bus, PULL_NONE; stock adds rts and tx_dat GPIO states through samsung,rts-gpio-control). With gpa3-0 and gpa3-3 floating, they read 0 whether REG_ON is high or low, so the pins can't tell us whether the chip drives them.

    So everything on our side matches stock and the chip still sends nothing at any rate: the rails, the 32 kHz outputs, the BT pins, the UART pins, the power-up order and the WLAN half. The owner adds that on GrapheneOS, Bluetooth has needed a few presses of the tile for a few weeks now, each one a full power cycle, where it used to come up the first time. The chip's own start-up seems to have got worse, whatever the OS. I'm out of host-side differences to try. If you know anything the HAL does before or between its big-hammer tries that isn't in bt_vendor.conf, that's the remaining place to look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions