From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49DF9442B2F for ; Wed, 22 Jul 2026 18:59:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746768; cv=none; b=rHxJbKINC7HU+W0UuRndGePsApyg6IamPmX2wVGm4K83QSTPH5uBSvsLiOGvMQimO7BLikGJlG9ybuvq3QtTIvKSlvjNYQqx0SN/THFiv4IUjAKTcY/1GixCfcwGXW5pfEN1tEZ3uyvVZ2caJtrt79xZQxQgsb++fGH3o+Km8yg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746768; c=relaxed/simple; bh=PFyLWHE9pTJ4175QPKDcNHyfOtGQRFYQNlkqnpPL1lc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PkAiM3M28JEd944m3uQVQUihydMiy9Qz8dTlp5Qurr/Plq4oxbZbVwa66R0YQyJf/Xb6nf+xDO/6378EgwudXMjQ71WZ0G/KYnqPBWhMFyO2chQ50BtzKnaE1Ms8R0VqxNLoYINj7+gWB+ROQ3ZeOW3Ezqlhpdvj56LynZ9dbVI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=gFkA1PYc; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="gFkA1PYc" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-495437bb891so54357465e9.1 for ; Wed, 22 Jul 2026 11:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746759; x=1785351559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZyX4ma10pC/frHu+Y/NUZniwvhp9EvBkdSwrv0doV3k=; b=gFkA1PYcdgJBj7Vk0LpWIXQdy5xoIr7JDJ4V6i5w+Cvel3qWwN8RjqopjcOvvtLeMM uBqArxT30CfTjh0676/kjNn2ppFN731beeq9rT71vPCEeMveVlDaadsU54BS0g55Vd9i rAYDk044Y5AnWmYPGvtkJtk3GSJ8Yr8SRmpRQK1DkmZEcGTlkXrF5dLRlooBfwJI5eLH 6fsDprOXAb51C2TyF8cYKU5NK5+jJi4SjQq5AYe0ragqMg7YykVIzzSw9P39yIym5uVr 1aK+008mvyyLPI8ftiReDk5SFbPVFZ2UnJ09yY1ZYjvjB1rVitvOviOPkHiza7yzEJeV WKRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746759; x=1785351559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZyX4ma10pC/frHu+Y/NUZniwvhp9EvBkdSwrv0doV3k=; b=HeM4ezo0GQR+zCTf1D3l+0OvqbKTsMYBckLHWS/M9zfZetm+ab+lOOiDaNdl8bK6Gi xiwdvkcEPP5+u0koQmOIfBUkrbFnOFXzaHItl8eOKZcLiyU7ylx1ubPydGNioOMsJhl7 Qr/A7EV0N5my2jPaHnhLH1XgGcV5zb78xBg5dCmgHNCnh1IwzXYmolNWvZJH+nL1T8Tf MqyA39OrD/I3tibRIiP0TVdtx4ngYUO5A2Pc2veNwW6TES9cRnxdqc95JAABfyKL3MNv 6dWWUS83qrUH1Q6lTiSLreKbRHvIHEjbqTAVj8UBM+4GYAe+KlAiJ20hniOLykRANc8W aFpA== X-Forwarded-Encrypted: i=1; AHgh+RrWgQv0u791S7K4CyTf8Gfsz2/nTFewXWlbbamUSRoqvBXoOG+nBg4s7FenfFtty3tDtC7GrCtcq5qGY1k=@vger.kernel.org X-Gm-Message-State: AOJu0YyEpkc7yWujOas+tsE3y1XyLwBRIE1QZ4fVJD4jgi/Kzj8GbUOC x1RRuqN6d+cn/4U4h7XhNFymvCrdA6/W4HnEbL03FoRiDKzve/pamKnvzvxdUs4lPg== X-Gm-Gg: AR+sD10uTtl16xHTERkAiJsNCnSYwYPQJdgNuggIk46mmYfHRJ+W1nCyuXqKMmxavlR c+UBG9tcWjIl82hI7XvaVeNpKn8bCpqZDC7JLUzxx9q/MyJHD7mhX8aNnX9sON8wnc0rXaj5luS vB9259irY0yUrK8wQ0eytWvv3VezCAeWWuY3P5aqtED1ZvrX6HHRsEBh3+gkwYSYOP7YOHkdyyA 2XJ6IOCSXsJ6J8cRZgM+J304Vn7LED8PNtquHRS8rWUEduGg7jGTfX/CUNCJm96SL/z8tcE7woo ClidGKlJBvQPBwFx3sBdnlpuHpGMomtx/DJry+kN2evDHqDbQ3u7sNI9h/x5BRnuMsMZhTg6TgH qK+/P+MbuO+qMXIAyUFKJGJnE3Aml1m5XkcEc8pup2QabAeTJvAowAluKDJxbnH2RTGka4xNk2v KRRPhdl7Y/EKPpP1qyROkHozbfbiDiwp3zfvCM7UZ8b63ZWFE= X-Received: by 2002:a05:600c:6610:b0:492:6f5c:fd8c with SMTP id 5b1f17b1804b1-4956a511b49mr53704175e9.15.1784746758682; Wed, 22 Jul 2026 11:59:18 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:18 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 9/9] nvme-rdma: return BLK_STS_P2PDMA for unsupported P2P transfers Date: Wed, 22 Jul 2026 20:58:41 +0200 Message-ID: <20260722185841.449934-10-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit When the PCIe topology cannot route a P2P transfer between two devices, the DMA layer fails the mapping with -EREMOTEIO, but ib_dma_map_sg() returns 0 and the error is lost. nvme-rdma then fails the I/O with a retryable status, so multipath requeues it forever and a single path wastes its whole retry budget on an I/O that can never succeed. Map the data and metadata scatterlists with ib_dma_map_sgtable_attrs(), which preserves the error, and translate -EREMOTEIO to the new BLK_STS_P2PDMA status so the request fails fast instead of being retried. While at it, call nvme_start_request() only after mapping succeeds and ratelimit the map-failure message. Fixes: 23528aa3320a ("nvme: enable PCI P2PDMA support for RDMA transport") Cc: stable@vger.kernel.org # v7.1 Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/nvme/host/rdma.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 6909e3542794..2f50509a7a61 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1469,6 +1469,7 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, int *count, int *pi_count) { struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); + struct sg_table sgt; int ret; req->data_sgl.sg_table.sgl = (struct scatterlist *)(req + 1); @@ -1480,12 +1481,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, req->data_sgl.nents = blk_rq_map_sg(rq, req->data_sgl.sg_table.sgl); - *count = ib_dma_map_sg(ibdev, req->data_sgl.sg_table.sgl, - req->data_sgl.nents, rq_dma_dir(rq)); - if (unlikely(*count <= 0)) { - ret = -EIO; + sgt = (struct sg_table) { + .sgl = req->data_sgl.sg_table.sgl, + .orig_nents = req->data_sgl.nents, + }; + ret = ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_table; - } + *count = sgt.nents; if (blk_integrity_rq(rq)) { req->metadata_sgl->sg_table.sgl = @@ -1501,14 +1504,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *ibdev, struct request *rq, req->metadata_sgl->nents = blk_rq_map_integrity_sg(rq, req->metadata_sgl->sg_table.sgl); - *pi_count = ib_dma_map_sg(ibdev, - req->metadata_sgl->sg_table.sgl, - req->metadata_sgl->nents, - rq_dma_dir(rq)); - if (unlikely(*pi_count <= 0)) { - ret = -EIO; + sgt = (struct sg_table) { + .sgl = req->metadata_sgl->sg_table.sgl, + .orig_nents = req->metadata_sgl->nents, + }; + ret = ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_pi_table; - } + *pi_count = sgt.nents; } return 0; @@ -2026,8 +2029,6 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) goto unmap_qe; - nvme_start_request(rq); - if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || @@ -2039,11 +2040,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, err = nvme_rdma_map_data(queue, rq, c); if (unlikely(err < 0)) { - dev_err(queue->ctrl->ctrl.device, - "Failed to map data (%d)\n", err); + dev_err_ratelimited(queue->ctrl->ctrl.device, + "Failed to map data (%d)\n", err); goto err; } + nvme_start_request(rq); + sqe->cqe.done = nvme_rdma_send_done; ib_dma_sync_single_for_device(dev, sqe->dma, @@ -2063,6 +2066,9 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_host_path_error(rq); else if (err == -ENOMEM || err == -EAGAIN) ret = BLK_STS_RESOURCE; + /* Peer memory unreachable from this device: don't retry. */ + else if (err == -EREMOTEIO) + ret = BLK_STS_P2PDMA; else ret = BLK_STS_IOERR; nvme_cleanup_cmd(rq); -- 2.52.0