Skip to content

fix(core): release the radio lock while waiting for ACKs - #291

Merged
agessaman merged 1 commit into
devfrom
fix/radio-lock-ack-wait
Sep 19, 2026
Merged

agessaman merged 1 commit into
devfrom
fix/radio-lock-ack-wait

Conversation

@agessaman

Copy link
Copy Markdown
Owner

What this changes

The radio command lock now covers one frame and the radio's immediate reply, instead of a whole command call. A DM waiting for its ACK no longer holds up every other radio command.

  • modules/core.py: _SerializedCommands is gone. _serialize_command_frames() wraps CommandHandler.send() on the handler instance, which is where every meshcore command writes its frame and waits for the radio's reply (OK, ERROR, MSG_SENT). There's still at most one in-flight companion frame, paced by command_min_interval_ms as before. Library methods call self.send, so composite commands and meshcore_cli.next_cmd are covered without a proxy.
  • New MeshCoreBot.radio_session() holds the radio across a short sequence of frames that must not interleave. Re-entering from the same task is a no-op.
  • modules/command_manager.py: region-scoped channel sends run set scope → send → restore inside a session.
  • neighbors_discovery.py and docs/packet-capture.md no longer list "stalls bot replies" as a reason scope collection is off by default. The other reason (it rewrites device contact paths) still stands, so the default is unchanged.

Why

I found this while chasing #290. The old proxy wrapped each coroutine on meshcore.commands, so a composite command held the lock for everything it awaited. send_msg_with_retry kept the radio for its whole retry loop, ACK waits included—up to ~36 s with three attempts when nothing comes back. During that time, channel replies, other DMs, scheduled sends, and automatic message fetching all waited. In one live run, a webhook DM sat behind a keyword reply's retry loop for 5–8 s before its first frame went out. req_regions_sync did the same thing for each neighbor scope reply.

The per-call lock also never kept a scoped channel send together. set_flood_scope, send_chan_msg, and the restore were three separate calls, so a DM retry loop queued between them ran its flood attempts under the channel's region scope. With per-frame locking, stray frames would land in that gap more often, so the sequence now holds the radio explicitly.

Testing

  • Rewrote tests/unit/test_command_serializer.py. The new tests cover:
    • Frames are serialized and paced as before.
    • A reply wait doesn't hold the lock.
    • While meshcore's real send_msg_with_retry waits for an ACK, another command goes out immediately. Against the old proxy, that command waited out the full 1.1 s ACK window.
    • A radio session keeps set → send → restore together while another task tries to send, and re-entering a session doesn't deadlock.
    • Installing the serializer twice doesn't double-wrap.
  • The serializer tests pass against meshcore 2.3.8 (the release pinned in CI) and 2.3.11.
  • make test: 4618 passed, 11 skipped. make lint and the CI strict mypy step are clean.
  • Live, over TCP to the companion interface of an openhop repeater (openhop_repeater 043128e, openhop_core 1.1.1, SF7) with meshcore 2.3.11, I sent two webhook DMs to a companion 3–4 hops out, 3 s apart. The second DM's frame went out while the first was still waiting for its ACK. Both were delivered, and each matched its own ACK code. The meshcore install there also carries an unmerged send_msg_with_retry fix of mine, which doesn't touch the lock.
  • Not tested on air: a region-scoped channel send. The unit test covers the sequence.

Checklist

  • Branched from dev and targeting dev
  • make test passes
  • make lint passes (ruff + mypy)
  • Frontend lint passes if templates changed (npm run lint:frontend)
  • Tests added or updated for behavior changes
  • CHANGELOG.md updated under ## [Unreleased] if user-visible
  • Config changes are reflected in config.ini.example (and the minimal/quickstart templates where relevant) — CI validates these with validate_config.py --strict
  • New docs pages are added to nav: in mkdocs.yml
  • Any new command justifies its airtime and defaults conservatively

Radio commands were serialized per method call, so a composite command held the lock for everything it awaited. A DM's send_msg_with_retry kept every other command waiting through all of its ACK timeouts (up to ~36 s with three attempts), stalling channel replies, other DMs, scheduled sends, and automatic message fetching. req_regions_sync did the same for each neighbor scope reply.

Serialize at the frame instead: wrap CommandHandler.send() on the handler instance, which is where every meshcore command writes its frame and waits for the radio's immediate reply. There is still at most one in-flight companion frame, paced as before, and library methods that call self.send are covered without a proxy. Waits for ACKs and remote responses now run outside the lock.

Add MeshCoreBot.radio_session() for short sequences that must not interleave, and use it for region-scoped channel sends so set scope, send, and restore stay together. The per-call lock never guaranteed that: a DM retry loop queued between set_flood_scope and send_chan_msg sent its flood attempts under the channel's scope.
@agessaman
agessaman merged commit 9939e22 into dev Sep 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant