Skip to content

Pace configuration-packet sends to avoid device UDP buffer overrun#191

Merged
cboulay merged 1 commit into
masterfrom
bugfix/cerelink-config-send-pacing
Jul 7, 2026
Merged

Pace configuration-packet sends to avoid device UDP buffer overrun#191
cboulay merged 1 commit into
masterfrom
bugfix/cerelink-config-send-pacing

Conversation

@cboulay

@cboulay cboulay commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

Configuring a device via Orion (Observatory) failed on a Gemini Hub running firmware 7.7.0 with repeated Response timeout on the runlevel sync barrier, and channel configuration never completed:

[CereLink] Clear-config sync failed: Response timeout   (×5)
[CereLink] Timed out waiting to confirm cleared channel configuration
[CereLink] Channel configuration failed

Root cause (verified on real Hub1 hardware)

Not a protocol issue — the Hub negotiates protocol 4.1. Orion's clear-config phase sends one setChannelConfig() per channel (272 channels × 2 = ~544 UDP datagrams) back-to-back with no pacing, then calls sync(). That burst overruns the device's small UDP receive buffer (~8 CHANINFO packets — the exact limit DeviceSession::sendPackets() already documents), dropping packets including the runlevel sync barrier, so sync() times out.

Newer firmware drains fast enough to survive the burst, which is why this looked like a 7.7.0 backward-compat regression.

Reproduced and fixed on hardware: unpaced → times out every retry; with pacing → clear-config + sync succeeds on the first attempt (3/3 runs).

Fix

DeviceSession::sendPacket() enforces a 200 µs minimum gap between consecutive configuration-channel sends (chid & cbPKTCHAN_CONFIGURATION). Data/streaming packets and isolated config sends (gap already elapsed) incur no delay. This extends the pacing that sendPackets() already does for its own batches to the per-packet path that callers use — so it protects every caller with no change required in Orion.

Testing

  • New unit test ConfigSends_ArePaced_DataSends_AreNot — uses sleep_for's guaranteed-minimum semantics, so the timing assertion cannot be flaky.
  • New examples/ClearConfigRepro/clear_config_repro.cpp reproduces the failure against real hardware (clear_config_repro HUB1, env PACE=<n> to compare).
  • Full build + cbdev_device_tests / cbdev_tests / cbsdk_tests pass. (ConnectionParams_Predefined_NPlay fails on master too — pre-existing, unrelated.)

🤖 Generated with Claude Code

Bulk per-channel setChannelConfig() bursts (e.g. Orion clearing LNC and
spike processing on every channel — hundreds of datagrams back-to-back)
overran the device's small UDP receive buffer (~8 CHANINFO packets),
dropping packets including a following runlevel sync barrier, so sync()
returned "Response timeout" and channel configuration failed. Older
firmware (observed on a Gemini Hub, fw 7.7.0) is most affected; newer
firmware drains fast enough to mask it, so it looked like a backward-compat
regression.

DeviceSession::sendPacket() now enforces a 200us minimum gap between
consecutive configuration-channel sends (chid & cbPKTCHAN_CONFIGURATION).
Data/streaming packets and isolated config sends (where the gap has already
elapsed) are unaffected. sendPackets() already documented and paced its own
batches; this extends that protection to the per-packet path callers use.

Adds ConfigSends_ArePaced_DataSends_AreNot regression test (uses
sleep_for's guaranteed-minimum semantics, so it can't be flaky) and a
ClearConfigRepro example that reproduces the failure against real hardware.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cboulay cboulay merged commit 4492647 into master Jul 7, 2026
16 checks passed
@cboulay cboulay deleted the bugfix/cerelink-config-send-pacing branch July 7, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant