Add DirectApi::inject_rtp to feed externally reconstructed RTP packets - #1026
Add DirectApi::inject_rtp to feed externally reconstructed RTP packets#1026hronro wants to merge 2 commits into
DirectApi::inject_rtp to feed externally reconstructed RTP packets#1026Conversation
Repair schemes str0m does not implement — FlexFEC (flexfec-03/RFC 8627) and RED — reconstruct whole RTP packets the receiver never saw on the wire. There was no way to hand one back: `handle_input` requires an SRTP protected packet, so the only route was to give up sample mode and depacketize in the application. Split the receive path into `handle_rtp_inner`, taking the plaintext payload as an `Option`. `None` behaves exactly as before and decrypts via SRTP; `Some` skips decryption and needs no SRTP context. Everything after that point — sequence number extension, replay rejection, TWCC and NACK bookkeeping, RTX unwrapping, depacketization — is shared, so an injected packet is indistinguishable from a received one downstream, and in particular counts as received rather than being NACKed again. The public entry point is on `DirectApi` rather than `Rtc`, since declaring the receive stream up front already requires it.
|
As I understand it, the fundamental request isn’t about RED or FlexFEC. It’s that str0m has no incoming plaintext-RTP endpoint corresponding to its normal post-SRTP receive path. Is that accurate? If so, I think the PR should motivate and document that general API directly. RED/FEC seems a bit beside the point. It's especially strange given that this probably will land any day #982 and that I imagine we might implement FEC as well. |
|
Probably be nicer to just implement flex_fec in str0m or have it injectable in some way for reconstruction. |
|
Yes I agree. I'll rewrite the PR to motivate that API directly. On RED: I'll drop it as an example. By reading #982 I also found something I did wrong. As written,
The first two are the ones that matter, and the error isn't marginal by construction: every packet a repair scheme reconstructs is one the network actually dropped, so the distortion is exactly the recovery rate. That's also the signal the sender uses to decide how much repair to send, so hiding it suppresses the thing that produced the repair packets in the first place. I don't think the conclusion is "always bypass the accounting", though, because the two legitimate uses of a plaintext ingress want opposite answers:
So if you'd take it, I'd rather the API say which: rtc.direct_api().inject_rtp(now, &packet, Injected::Recovered);
rtc.direct_api().inject_rtp(now, &packet, Injected::Received);
On str0m implementing FEC itself: if it does, most of my own need goes away. Two things would still sit outside it. libwebrtc's FlexFEC is draft-03, which isn't wire-compatible with the published RFC 8627, so whichever variant str0m implements the other one still exists in the wild. And the replay, relay and bridge cases above have nothing to do with FEC at all. But if you'd rather str0m not carry a general plaintext ingress, say so and I'll close this and keep it in a fork — no hard feelings. Practically: #982 rewrites the part of |
The first version of this registered every injected packet as received, and the commit message sold that as the feature. It is wrong for the case that motivated the API. A reconstructed packet is, by definition, one the network dropped. Registering it updates the TWCC receive register and the register that feeds receiver-report loss -- both reported to the sender, and both what it uses to size its bitrate and its repair overhead. So registering a recovered packet hides exactly as much loss as the repair scheme fixed, which suppresses the signal that produced the repair data in the first place. The error is not marginal by construction: it equals the recovery rate. `inject_rtp` now takes `Injected`, because the two honest reasons to hand str0m plaintext RTP want opposite accounting: * `Received` -- the packet genuinely arrived by a route str0m did not decrypt: a replayed capture, a relay that decrypted once and fans out, a bridge from another transport. Counted exactly like one off the wire, or the statistics understate what arrived. * `Recovered` -- reconstructed, and so deliberately invisible to receive accounting: no TWCC, no register update, no byte or packet counters. It still reaches the depacketizer, which is the entire point. `Recovered` also leaves `max_rx_seq_lookup` alone, since that exists to keep SRTP's rollover counter continuous and a packet that was never decrypted has no bearing on it. `StreamRx::media_time_for` supplies the extended timestamp `update_register` would otherwise have produced. The cost of not registering is that a retransmission of a recovered packet may still be requested and arrive. That is harmless, and was checked rather than assumed: `DepacketizingBuffer::push_entry` drops an exactly-matching sequence number, and drops anything at or before the last emitted.
|
I've pushed this as a second commit: |
Problem
Repair schemes str0m doesn't implement — FlexFEC (
flexfec-03, RFC 8627) and RED — reconstruct whole RTP packets the receiver never saw on the wire. There is currently no way to hand one back.Rtc::handle_inputis the only entry point, and it expects an SRTP-protected packet, so a plaintext packet built by the application can't go in. That leaves only one option: turn onrtp_modeand reimplement depacketization and reordering in the application, replacingpacket::{h264, h265, av1}andDepacketizingBuffer. For H.264/H.265/AV1 that's several thousand lines of code str0m already has and already tests.Change
handle_rtpbecomes a thin wrapper over a newhandle_rtp_inner, which takes the plaintext payload as anOption<&[u8]>:None— the existing path, unchanged. Decrypts via SRTP and rejects packets arriving before anSrtpContextexists.Some— skips decryption and needs no SRTP context at all.Everything after that point is shared: sequence-number extension, replay rejection, TWCC and NACK bookkeeping, RTX unwrapping, and depacketization. So an injected packet is indistinguishable from a received one downstream, and in particular it's registered as received, which means it stops being NACKed. That's the property that makes this useful rather than just convenient — a repair mechanism outside str0m still cooperates with the one inside it.
The public entry point is:
It's on
DirectApirather thanRtcbecause declaring the receive stream up front already requiresDirectApi::expect_stream_rx, so any caller is there anyway. Unknown SSRCs and malformed input are dropped with atrace!, matching how the receive path already treats unexpected packets.Compatibility
Additive. The
Nonearm ofhandle_rtp_inneris the old code path verbatim, so nothing changes for existing users — no behaviour change, no API break, no new dependency, and no cost when the feature isn't used.