From 4711f288632a283a56e265ea6d4e59f7aa592227 Mon Sep 17 00:00:00 2001 From: David Canar Date: Sun, 16 Aug 2026 13:22:36 -0600 Subject: [PATCH] Default data QPs to infinite RNR retries A SEND that arrives before the peer has posted its receive gets an RNR NAK. With the verbs defaults (rnr_retry = 0, retry_cnt = 0) the module completes that SEND immediately with RNR_RETRY_EXC_ERR and then marks the QP in error, taking the whole connection down on a single scheduler-induced race between the sender's post_send and the receiver's post_recv. Default data QPs to infinite RNR retries (7, per the IB spec) and a retry count instead: the SEND then waits for the peer's advertised recv credits (tbv_send_rnr_waits_for_recv_credit). Users can still override both through modify_qp (IB_QP_RNR_RETRY / IB_QP_RETRY_CNT). Validated with ds4's two-machine tensor parallelism over usb4_rdma0 (concurrent sessions, prefill row swaps, long decodes) and with the uc_oneway bidi bench: zero QP errors where the previous default killed the QP on the first post_recv/post_send race. --- kernel/ibdev.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/ibdev.c b/kernel/ibdev.c index cb96971..2702c25 100644 --- a/kernel/ibdev.c +++ b/kernel/ibdev.c @@ -2502,6 +2502,23 @@ static int tbv_create_qp(struct ib_qp *qp, struct ib_qp_init_attr *init_attr, init_attr->qp_type != IB_QPT_UC && init_attr->qp_type != IB_QPT_GSI) return -EOPNOTSUPP; + /* + * The transport cannot retransmit on its own: a SEND that arrives + * before the peer has posted its receive gets an RNR NAK. With the + * verbs defaults (rnr_retry = 0, retry_cnt = 0) that SEND completes + * immediately with RNR_RETRY_EXC_ERR and marks the QP in error, + * taking the whole connection down on a single scheduler-induced + * race between the sender's post_send and the receiver's + * post_recv. Default data QPs to infinite RNR retries and a retry + * count instead: the SEND then waits for the peer's advertised + * recv credits (see tbv_send_rnr_waits_for_recv_credit). Users can + * still override both through modify_qp (IB_QP_RNR_RETRY / + * IB_QP_RETRY_CNT). + */ + if (!gsi) { + tqp->attr.rnr_retry = 7; /* infinite, per the IB spec */ + tqp->attr.retry_cnt = TBV_SEND_MAX_RETRIES; + } tqp->backend = tbv_ibdev_backend(qp->device); if (tbv_backend_is_apple(tqp->backend) && init_attr->qp_type != IB_QPT_UC)