Skip to content

firmware: samsung: acpm: release a timed-out request's sequence slot - #152

Draft
W4HO-ham wants to merge 1 commit into
derekross:omamobilefrom
W4HO-ham:acpm-seqnum-timeout
Draft

W4HO-ham wants to merge 1 commit into
derekross:omamobilefrom
W4HO-ham:acpm-seqnum-timeout

Conversation

@W4HO-ham

@W4HO-ham W4HO-ham commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Draft because it is untested on hardware: the palmtop is hung awaiting a forced restart, so I could not run a single ACPM timeout against it. The bug itself is in the code, not inferred from a log.

The leak. A request claims one of the channel's 63 sequence slots in acpm_prepare_xfer() and gives it back in acpm_dequeue_by_polling(), on the path where the answer arrives. The timeout path prints and returns -ETIME with the slot still claimed, and nothing else releases it. After 63 timeouts the channel answers everything with -EBUSY and "ACPM sequence number pool exhausted", until a reboot.

The evidence, stated correctly. It comes from a Pixel 9a console of 2026-10-30 18:31, a boot that ended in a watchdog reset during a GNSS session, not from the UFS resume failure earlier that day (an earlier version of this description said the latter, which was my mistake; the UFS console has no ACPM timeout in it at all). In that boot, 10 s after an AoC DSP fault and its ALSA command timeouts, the thermal channel stopped answering:

[  158.562159] exynos-acpm-protocol firmware:power-management: Timeout! ch:9 s:46 bitmap:600000000000.
[  158.562170] thermal thermal_zone2: Temperature check failed (-62)
[  158.562175] exynos-acpm-protocol firmware:power-management: Timeout! ch:9 s:47 bitmap:600000000000.
[  158.562180] thermal thermal_zone1: Temperature check failed (-62)

Two slots are held at once in that bitmap while allocations are serialised under tx_lock, so one had already leaked when the second timeout printed. Thermal polls about once a second, so a firmware that has stopped answering burns the pool in about a minute, and the same code serves the PMIC and DVFS channels. I am not claiming the leak caused that reset; it is a defect found while reading those paths, and it is worth fixing on its own.

Why not simply clear the bit on timeout. The firmware may still answer, and its answer carries that sequence number. A slot handed to the next request would then take the old answer as the new request's reply. So the slot is marked abandoned and acpm_get_rx() releases it when the late answer arrives, dropping the answer. A firmware that never answers at all still exhausts the pool, which is then an honest report of a dead channel.

Builds with W=1 for arm64, no new warnings; checkpatch --strict clean. What it needs before leaving draft: a run on hardware where a channel times out, showing the bitmap returning to zero afterwards and no "pool exhausted".

@W4HO-ham
W4HO-ham force-pushed the acpm-seqnum-timeout branch from d020254 to 0b46b0d Compare October 31, 2026 04:41
@derekross

Copy link
Copy Markdown
Owner

Thanks, this is a real leak and a sensible design. Three things:

  1. The commit needs your Signed-off-by.
  2. In the evidence, tx_lock is released before the wait, so the two bits in that bitmap (0x600000000000, s:46 and s:47) are the two requests that timed out together, not a leaked slot. Please reword that part.
  3. Please set the abandon flag under rx_lock, after checking completed. Otherwise an answer drained by another thread just before the timeout still leaks the slot.

Once the 9a run is in, I'll boot it on the 7 Pro.

@W4HO-ham
W4HO-ham force-pushed the acpm-seqnum-timeout branch from 4f58d4a to e30c106 Compare October 31, 2026 15:58
@W4HO-ham

Copy link
Copy Markdown
Contributor Author

Points 2 and 3 are in, pushed as one amended commit (still a draft until the sign-off, which is the owner's and is queued for him).

  1. The evidence paragraph now says what the bitmap shows: the two bits are the two requests that timed out together, since tx_lock is released before the wait, not a leaked slot; what the log demonstrates is the rate at which a stalled channel burns the pool.
  2. The abandon decision is now taken under rx_lock after one more look at completed. If another thread's drain saved the answer just before the deadline, the request is treated as a late success: the saved payload is copied and the slot released, instead of marking a slot abandoned that no further answer will ever clear.

Builds for arm64 with W=1 (compile-tested on the 9a's tree), checkpatch clean apart from the sign-off. The 9a run against a timing-out channel is still owed.

@W4HO-ham
W4HO-ham force-pushed the acpm-seqnum-timeout branch from e30c106 to 183ae76 Compare October 31, 2026 17:50
@derekross

Copy link
Copy Markdown
Owner

Thanks, the rx_lock re-check reads right, and the new evidence paragraph is accurate. One thing left on the branch: it still has the original unsigned commit underneath the signed one, and kernel PRs land commit by commit. Please squash them into a single commit with the sign-off. Once the 9a run against a stalled channel is in, I'll boot it on my 7 Pro.

@W4HO-ham

W4HO-ham commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Squashed into one signed commit and rebased onto today's omamobile. checkpatch --strict and the W=1 arm64 build of exynos-acpm.o are clean.

The 9a run against a stalled channel is still to come. Today's palmtop console (without this patch) shows a stall on the DVFS channel: Timeout! ch:0 s:41 bitmap:10000000000, then s:42 to s:54 every 100 ms with the bitmap filling one bit per timeout, then RX front can not keep up with TX front. That's a different stop than the pool running out, but it's the case this patch is about. That stall came from a brcmfmac oops (kernel #191), so I can reproduce it, and I'll run it with this patch and post the result here.

@W4HO-ham
W4HO-ham force-pushed the acpm-seqnum-timeout branch from 183ae76 to 413fa4a Compare November 3, 2026 17:51
@derekross

Copy link
Copy Markdown
Owner

Thank you, @W4HO-ham, for the squash and the rebase. All three points from my review are in, and the squashed patch is line for line what I read on the 31st. I went through the timeout path once more. completed is only written under rx_lock, so the re-check can't race. A late answer for an abandoned slot never touches the caller's buffer. The slot can't be handed out again until that answer has cleared it. kernel-pr-check --build --merge on today's omamobile passes and CI is green. Mainline has the same leak, so this is worth sending upstream once it has run.

About the 9a run. Today's DVFS stall, with the bitmap filling one bit per timeout and then "RX front can not keep up with TX front", looks like firmware that has stopped answering. On such a channel the patch changes nothing you can see, because the pool still fills, as the message says. The run shows the fix only if the firmware answers late at least once: a Timeout!, then later requests on that channel with the abandoned bit cleared and no "pool exhausted". If the channel just dies, please record it as not exercised rather than as a pass. Since the trigger you have in mind is the brcmfmac oops that reset the palmtop last time, please run it attended and read pstore afterwards.

One small thing: this file's history, here and upstream, uses the prefix firmware: samsung: acpm:. Could you retitle the commit to match?

Once the 9a run is in, I'll boot it on my 7 Pro (DVFS, thermal and PMIC reads, smoke test) before merging.

A request claims one of the channel's 63 sequence slots in
acpm_prepare_xfer() and gives it back in acpm_dequeue_by_polling(), on the
path where the answer arrives. The timeout path prints and returns -ETIME
with the slot still claimed, and nothing ever releases it, so every
timeout costs the channel one slot for good. After 63 of them the channel
answers every request with -EBUSY and "ACPM sequence number pool
exhausted", and only a reboot brings it back.

This is reachable: on a Pixel 9a whose thermal channel stopped answering,
the console shows repeated "Timeout! ch:9 s:46 bitmap:600000000000" and
"s:47" with the same bitmap. Those two bits are the two requests that
timed out together (tx_lock is released before the wait, so two can be
outstanding), not a leaked slot; what the log shows is the rate: thermal
polls about once a second, so against a stalled firmware the pool is gone
in about a minute. The same code serves the PMIC and DVFS channels.

The slot cannot simply be released on timeout: the firmware may still
answer, and the answer carries that sequence number, so a slot handed to
the next request would deliver the old answer as the new request's. Mark
the slot abandoned instead, and let acpm_get_rx() release it when the late
answer arrives, dropping that answer. Take that decision under rx_lock
after one more look at completed: another thread draining the queue may
have saved this answer just before the deadline, and abandoning the slot
after that would leak it, since no further answer will come; the request
has then succeeded, late, and is treated as such. A firmware that never
answers at all still exhausts the pool, which is then an honest report of
a dead channel rather than the result of bookkeeping.

Signed-off-by: W4HO <87742609+W4HO-ham@users.noreply.github.com>
@W4HO-ham
W4HO-ham force-pushed the acpm-seqnum-timeout branch from 413fa4a to dcfa3ff Compare November 3, 2026 21:16
@W4HO-ham W4HO-ham changed the title firmware: exynos-acpm: release a timed-out request's sequence slot firmware: samsung: acpm: release a timed-out request's sequence slot Nov 3, 2026
@W4HO-ham

W4HO-ham commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Retitled to firmware: samsung: acpm: (dcfa3ff), no other change. Understood on the run: it has to be attended and pstore read afterwards, and it only counts if the channel answers late at least once (a Timeout!, then later requests on that channel with the abandoned bit cleared and no "pool exhausted"). A channel that just dies goes on record as not exercised.

@derekross

Copy link
Copy Markdown
Owner

Thanks, @W4HO-ham, for the retitle. It now matches the file's history, and
the tree is identical to what I read, so nothing else needs checking. We
agree on the run: a late answer that frees the slot is what counts, and a
dead channel goes down as not exercised. I noticed your smoke kernel for
#187 also carried this patch, but there was no ACPM timeout in that boot, so
it doesn't count as the run. That's fine, no hurry. Once the 9a run is in,
I'll boot it on my 7 Pro before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants