From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.176 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794611345; b=NT2e5HJMwyz+AaL9jXI6n0MPpMnZSF1QdR1x5cigT8/4u2L2kO/O9a7FGuI/0oQQtX6b qAENUagN2BSlkOI2KvOZnWcE8J6vE2/PdKUWqYSHqFBLUX/VCyxqjsNV/PKcdtchLMMDD y8Q2FgvUazGYy0IEolWZZ7iq5SJfkreUSuB/lXIeHdY2OeOI9pr3Kp93oentbPym2EyOr E078ySjTpgYrnG1a38BkVgJJQCh3jaIhjdaBXhA/Ko5vQBemWaLt1RCGWT7tJheMdImaj eHsglcQ3VhIXqrdbHrmMmOcpRV2PpFjKLRUH2S1dUjGoVKgdlmMvjKvPkvwtiAWbmKw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794611345; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Gm-Gg:X-Received:Received: From:To:Cc:Subject:Date:Message-ID:X-Mailer:MIME-Version: Content-Transfer-Encoding; bh=5DlzMUMtEfBl4sH+z/6/msTIw6TRSjv0D6qDnAX+XMg=; b=dYzZ2HTg/q4hd/aNLAAFkWUzPfefuUvXDRgsDzWQmG/6h30F5s2pwqBcNQDsjMn0yDhU gWH+MRJgrP3zDGfTEKvYqwOGKYqUszjzklXpf7HzBHORH5HTVSYxGoAQt2FEtWMC8weg2 UiVKQ5eWMcH0d9amrNoacludAAda8jDgNmOP9S/TJ7niSbWX6L+3upvBKCKoeH/zH/3xx xRbB9HihM2zWyJoUvuStCx53twhGDIzU5ZGMkiwPxb4a1Tk2GNRci93lunXgp2l6muZ4Y dCqRwlMQ17rTd6TSyaiEVkl8W6Z5Ku5KkCxMafOJ7utjgduGBoICBTB/BAhVhCXbsVQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=purestorage.com; spf=pass smtp.mailfrom=purestorage.com; dkim=pass header.d=purestorage.com header.i=@purestorage.com header.a=rsa-sha256 header.s=google2022 header.b=XDxqczw+; arc=none smtp.remote-ip=74.125.82.176 Received: from mail-dy1-f176.google.com (mail-dy1-f176.google.com [74.125.82.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7933D606A329 for ; Fri, 13 Nov 2026 23:09:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 7933D606A329 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=purestorage.com header.i=@purestorage.com header.a=rsa-sha256 header.s=google2022 header.b=XDxqczw+ Received: by mail-dy1-f176.google.com with SMTP id 5a478bee46e88-33fb4680717so4400727eec.1 for ; Fri, 13 Nov 2026 15:09:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1794611343; x=1795216143; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=5DlzMUMtEfBl4sH+z/6/msTIw6TRSjv0D6qDnAX+XMg=; b=XDxqczw+fhNCa5AT80FKOoJM6Ou3zh6RfXXx+DM/pNL8ZMxraPW2kO3doqaHkVmNF3 uDe28tAUfDd1xKaE8lrESwZxTLQrTEO587sFSdjD9nVv5CYdVlXFckIfZILP2Mc00zI+ fyJjy/CKt7RywPBU8e1v5SiBILoWKHGLoIKFn3m3jkXkYJ59L9YBF7KguyzMRkTFmmBg 2r1310P1aFqsutcsuq/r+FsKbWN/nxE0fyCJxGYZ6iJDv2fmyQHlhhQdfctS6CXF0wgF FYfN07y5mx0Obeji+cDo3Cqr7H79D5Zsw0nyKHmP4Xav54MRv+Xcdwmy64eWcP2BJVjt NLyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794611343; x=1795216143; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5DlzMUMtEfBl4sH+z/6/msTIw6TRSjv0D6qDnAX+XMg=; b=E6NYFEqpy1XmGFyJP9MABquWM+t0J4yvd92pw05bHPCt47af14QfJIK536Tugy86fG GVIietznBZxpYmHkYXriEaOM6MvSrM9onMFSxP1jJdcGQ+XT+vcg4PGFRZvngLIb9NtY 99+Ud+UahFDrNqXvEuAdr7P/NARlrIW96LYN7fIHWhwCnzQSI12Pv2ez3SF4vZBjaBaa ++krvLbVe9ww4olbNnkxwQtFLpVTESh5JUnZO2+lSod66ZHzC4qlB7gntPq81+JVRVCQ jwuAvZzCtNyYGeSxm8qyjH/IKRM7wCPbUTfRby95ZKwGkCZyCwPgV3ks+aKIm894MQZe 9kKA== X-Forwarded-Encrypted: i=1; AL+hErF1lLBJZ9gQ/IHJY+JTBOfYncj75vV5IE/5LVW2I36Il6HyfRxI44iGAUI/DybGLAugiddPZEwR+5D5I7k=@vger.kernel.org X-Gm-Message-State: APuU4Z+TY089W63IeA03+P/5l2OZjk2/VQ4flU3HEmg9FFqPvUThZdER x+Jl2Z/j+VGtXzcvU6zdJac6MfaNVlMhXI34dyHyJ3ONh7cD0Vz0q6GkLMLN/hyjla0= X-Gm-Gg: AYBFou2kzR/RxE8Z8qwF8xIYx5bw/aeRWIrb0sLT/vIvXLDjBr7t+Crp1SQPKmYwYjb /3wzStfOXcCm7rcrqun2e/J89peISEF3Gmr/PcCSnowsFgi6cpanWkK0YFcR6aHFVzKxp4Xbyee fgNe79E4Cwqw4cWOYnrdrXmR/jWfask6MC36aRQv5gkEXqyVzJ+WFCExW+CqZx78V2Y0fnx/mMw kpa6pz51QwijNAlNg2q9IGgLe/bO3Qp7kzx0vZjTn4Dwbl8uRaZkv7PUt9G1rwwRavXLmw7SlNK BhZuUulYzIMPzeITjlk1wMriP5/y+wrwMT0J9R9uLSdkoaef/hRGlq54fwh5o4Leq/J6eJJSkvr DCvJOv4qBb/wRbAyIdTykKN4laBarh3EYtOm71kA3WdG5uZgtadyN2ja2L9dEfKrgLMI/I15vDb 3zuA/wcLO6izDQUM3FaFKu6yUcAyCBHtQ4ZUatPACUIsMTX/BBTzt/4NW/XS2UtT9wOgaFpi0pQ 3k3yOC19gwXDY0kOO2I6Ev4g1bpV4OcdwGkzUtLTxSb0r4TeT3OvLpk2SYdqWUMbuj47PiKI6vb min+5IfMk6fPpWh3uxIjYiJXt+VaOewsKRr5WGbUsCcTAvW8IXRgoNF+0b0UdhkH9GlByK1F6w= = X-Received: by 2002:a05:7301:db16:b0:376:9e80:2bf5 with SMTP id 5a478bee46e88-379680cf4b9mr3687174eec.38.1794611343074; Fri, 13 Nov 2026 15:09:03 -0800 (PST) Received: from apollo.purestorage.com ([208.88.152.253]) by smtp.googlemail.com with ESMTPSA id 5a478bee46e88-3795378d73dsm10737887eec.2.2026.11.13.15.09.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Nov 2026 15:09:02 -0800 (PST) From: Mohamed Khalfella To: Zhu Yanjun Cc: Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mohamed Khalfella Subject: [PATCH] RDMA/rxe: Do not re-execute completed local ops after a retry Date: Fri, 13 Nov 2026 15:08:31 -0800 Message-ID: <20261113230831.2682071-1-mkhalfella@purestorage.com> X-Mailer: git-send-email 2.56.0 X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Local operations (IB_WR_REG_MR, IB_WR_LOCAL_INV, IB_WR_BIND_MW) are executed inline by rxe_requester() through rxe_do_local_ops(), which marks the wqe wqe_state_done and advances qp->req.wqe_index past it. Unlike wire operations, nothing is posted to the network and no ack is expected. The completer only consumes wqes from the head of the send queue, in order. A done local op wqe therefore stays in the queue until every older wqe has completed, so it can sit behind a SEND or RDMA WRITE that is still waiting for an ack. With nvme-rdma, which posts REG_MR + SEND for every I/O, the queue regularly looks like: SEND(A) pending, REG_MR(B) done, SEND(B) pending, REG_MR(C) done ... When a retransmit timeout fires or a NAK arrives, req_retry() rewinds qp->req.wqe_index to the consumer so that every unacked wqe is resent. Its loop skips wqe_state_done entries when resetting DMA state, but rxe_requester() itself has no such check. After the rewind it walks forward through the queue, reaches REG_MR(B), sees WR_LOCAL_OP_MASK and calls rxe_do_local_ops() on it a second time. rxe_reg_fast_mr() rejects the repeated registration because the MR is already RXE_MR_STATE_VALID, not FREE. The wqe is given IB_WC_LOC_QP_OP_ERR and the QP is moved to the error state. The completer then flushes the entire send queue with IB_WC_WR_FLUSH_ERR, including the wqe that failed, so the consumer never sees the real error status. nvme-rdma logs only "SEND ... failed with status WR flushed (5)" and tears down the controller. Any packet loss that triggers a retry while a later request is queued is enough to hit this. On a soft-RoCE setup with a few hundred lost frames per run every controller reconnect traced back to the same rxe_dbg_mr() line: rxe1: mr#776 rxe_reg_fast_mr: mr->lkey = 0x3084c not free Skip local op wqes that are already done when the requester reaches them again after a rewind. Advance wqe_index past them exactly as a successful execution would and continue with the next wqe. The MR keeps its registered state and the completer still reports the wqe in order with its original status. Only local ops can be in wqe_state_done ahead of the consumer on an RC QP, since wire operations stay pending until acked, so the check cannot match anything else. Fixes: 001345339f4c ("RDMA/rxe: Separate HW and SW l/rkeys") Cc: stable@vger.kernel.org Signed-off-by: Mohamed Khalfella --- drivers/infiniband/sw/rxe/rxe_req.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 4f353abd76f5..fdb55366d668 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -723,6 +723,20 @@ int rxe_requester(struct rxe_qp *qp) goto err; } + /* + * A retry rewinds wqe_index to the send queue consumer. Local + * operations that were already executed may still sit in the + * queue behind an unacked wqe waiting for in-order completion. + * Re-executing them fails (e.g. REG_MR on an MR that is no + * longer free) and would move the QP to the error state, so + * skip them instead. + */ + if ((wqe->mask & WR_LOCAL_OP_MASK) && + wqe->state == wqe_state_done) { + qp->req.wqe_index = queue_next_index(q, qp->req.wqe_index); + goto done; + } + if (rxe_wqe_is_fenced(qp, wqe)) { qp->req.wait_fence = 1; goto exit; -- 2.56.0