From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03F4A43304B for ; Fri, 24 Jul 2026 11:11:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784891493; cv=none; b=q1HEC0SXzzgB4Uz9a1D9ZMSbMwPlxK0OgBCzBrK8PdM5c9GCYU0Do/odKg3EHB8Y8lEvM75AIKPEy/NYoDJ0Zs+McVAkBItF9H5g8arIpnKuXZrHG7XrEQoKB/pTVBZhkg3KUeqoT5l0K71TzAOe9KFxXox9RSJ4YhXNGsQoKP8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784891493; c=relaxed/simple; bh=Hl9wyVfReowmH9lQXOTj9hunvFlCPgM9Uuy4AsqCz3I=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=BCCZFKzzBUqM6hI8yUob8l9TBLiCBkVauj565UWoeChX9fyp1ECygsrXPo+6sHlrVJ3bhI6GVBJucsw6IzZEcHKoHl6xaH5o/Xqe0/LuMM8OPiUqUsNngFf06kSdp+zezCNQvmK553+qOwICE3V9H0OxkAyuyq7M26EQ34miPEQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=u5PmhNDy; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="u5PmhNDy" Received: by smtp.kernel.org (Postfix) with ESMTPS id DA75AC4FF61; Fri, 24 Jul 2026 11:11:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1784891492; bh=Hl9wyVfReowmH9lQXOTj9hunvFlCPgM9Uuy4AsqCz3I=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=u5PmhNDy62FU/PIUuw/jJ1O/ciA0R62JR06BjM31bi1UKrv1AuDEXUvggnMwTenUj vRQ9wNcbxuy3ewT4eux4w78z6T4FoUiCienbXubjq+yR7LTccO/6DmBsBDBLPQ1DGV 9J920IQmy+J9TFI7OkrfdRZalDiFGGWri+su+vk/2yU6jrGwE/brUa3LE78FIz1Rlt 0wnRJD2Oil62WuIjGRp7tqduiBdohc9tUSAc6wf+nmixyHM1ddXEPRdhiLwxAvZ7cE y7miuoVMNin8UkANKD+7drEK/pZhLDUHMJHtspztnF2MXMd6oP0xqoVJGrkrDiXbn0 MJqvQKacuR6eg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4C06C53215; Fri, 24 Jul 2026 11:11:32 +0000 (UTC) From: Joel Granados Date: Fri, 24 Jul 2026 13:11:05 +0200 Subject: [PATCH RFC v2 11/14] nvme: Implement CDQ traversal on the read path Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260724-jag-cdq-lkml-v2-11-659a7449e849@kernel.org> References: <20260724-jag-cdq-lkml-v2-0-659a7449e849@kernel.org> In-Reply-To: <20260724-jag-cdq-lkml-v2-0-659a7449e849@kernel.org> To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , Jason Gunthorpe Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Joel Granados X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=openpgp-sha256; l=4810; i=joel.granados@kernel.org; h=from:subject:message-id; bh=Hl9wyVfReowmH9lQXOTj9hunvFlCPgM9Uuy4AsqCz3I=; b=owJ4nAHtARL+kA0DAAoBupfNUreWQU8ByyZiAGpjSF+n2iXlrweqzc/nxEwB/z4f0UK8MBU+j NCK/boeFLJJjYkBswQAAQoAHRYhBK5HCVcl5jElzssnkLqXzVK3lkFPBQJqY0hfAAoJELqXzVK3 lkFPUhsMAIGYljanPcOwQ6j/9z4kH80xLUK3aACtckcxSC/qwqXnuS1jRQ/wfkFNlGm7mHLaLmG soKtFi6Cqblp/e8GuLUJqKYK5APRpeoSw8jyNNVEGP/9W+ddXMDjeS29pAXKzw3+OLrzk6E70hd 4BG+b5iBTlhF/R5MpKCguGYIHIy7JgGH45natCdyWbEItJcYCsTijxW1vYXNzs0Qtn5T+9d1n8E P8iGo60G2+RoGKfqGCMQ4cAdbRQgWwYmrzOs7w9rQEHgqcMqeu6cUtb8GWCXarUnKZpVqontwfW yB2C6wgfTLS01ethQcQUpleFB/9btqTf9fBJMMyK6MF9A9mSjLQ/GJAq+/nqGK1Gc6lT0K+R8MV av9LZLbMOkwXtnOqt7PJCSPNc+JSkH6EmBqB2Oxqh/GF72ohYT2WDyAbCQ92d6o6JYfM1x9yxEs kRTx2fNi3KWNnH7kkLnqw2+QKGyYA2M0eMgy+P+O2KTTnc2sxgMn3aRTNXZH4jevDHI9I6bkAq7 HY= X-Developer-Key: i=joel.granados@kernel.org; a=openpgp; fpr=F1F8E46D30F0F6C4A45FF4465895FAAC338C6E77 X-Endpoint-Received: by B4 Relay for joel.granados@kernel.org/default with auth_id=239 Replace the initial -EOPNOTSUPP stub with a call to nvme_traversecopy_cdq which copies new controller entries to the user space buffer, forwards and wraps host_head around when it reaches the end of the queue. Signed-off-by: Joel Granados --- drivers/nvme/host/cdq.c | 73 +++++++++++++++++++++++++++++++++++++++++++++++-- drivers/nvme/host/cdq.h | 7 +++++ 2 files changed, 78 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/cdq.c b/drivers/nvme/host/cdq.c index 7fe08afa0d80db5f3d60993aba05624b5c59b309..36107cf2fe11ff212d5b02dd5513fe02e3ae8f81 100644 --- a/drivers/nvme/host/cdq.c +++ b/drivers/nvme/host/cdq.c @@ -5,6 +5,7 @@ #include #include +#include #include "nvme.h" #include "cdq.h" @@ -169,6 +170,75 @@ static inline void nvme_release_cdq_backing(struct cdq_nvme_queue *cdq) nvme_free_cdqmem_chunks(cdq); } + +static inline void *nvme_get_cdq_entryvaddr(struct cdq_nvme_queue *cdq, + unsigned int entry_idx) +{ + return cdq->chunks[entry_idx / NVME_CDQ_MQ_ENTRY_PER_CHUNK].vaddr + + (entry_idx % NVME_CDQ_MQ_ENTRY_PER_CHUNK) * NVME_CDQ_MQ_ENTRY_NRBYTES; +} + +/* Advance cdq->host_head at most nrbytes and return actual advanced bytes */ +static size_t nvme_advance_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte) +{ + size_t target_nbyte = 0; + void *entry; + u8 phase_bit; + u32 cdq_nrentry = cdq->size_nbyte / NVME_CDQ_MQ_ENTRY_NRBYTES; + + while (target_nbyte < max_nrbyte) { + entry = nvme_get_cdq_entryvaddr(cdq, cdq->host_head); + phase_bit = (*(u8 *)(entry + NVME_CDQ_MQ_PHASE_OFFSET) & NVME_CDQ_MQ_PHASE_MASK); + + if (phase_bit == cdq->phase_bit) + break; + + cdq->host_head = (cdq->host_head + 1) % cdq_nrentry; + target_nbyte += NVME_CDQ_MQ_ENTRY_NRBYTES; + if (unlikely(cdq->host_head == 0)) + cdq->phase_bit = ~cdq->phase_bit & NVME_CDQ_MQ_PHASE_MASK; + } + + return target_nbyte; +} + +static ssize_t nvme_traversecopy_cdq(struct cdq_nvme_queue *cdq, size_t max_nrbyte, + void *priv_data) +{ + char __user *to_buf = priv_data; + void *from_buf; + u32 init_host_head = cdq->host_head; + u8 init_phase_bit = cdq->phase_bit; + + size_t target_nbyte = nvme_advance_cdq(cdq, max_nrbyte); + size_t copied_nbyte = 0, chunks_idx, entry_idx, tx_nbytes; + + if (target_nbyte == 0) + goto out; + + for (chunks_idx = init_host_head / NVME_CDQ_MQ_ENTRY_PER_CHUNK, + entry_idx = init_host_head % NVME_CDQ_MQ_ENTRY_PER_CHUNK; + copied_nbyte < target_nbyte; + chunks_idx = (chunks_idx + 1) % cdq->nr_chunks, entry_idx = 0) { + from_buf = cdq->chunks[chunks_idx].vaddr + + (entry_idx * NVME_CDQ_MQ_ENTRY_NRBYTES); + tx_nbytes = min(target_nbyte - copied_nbyte, + NVME_CDQ_CHUNK_SIZE - (entry_idx * NVME_CDQ_MQ_ENTRY_NRBYTES)); + if (copy_to_user(to_buf, from_buf, tx_nbytes)) + goto err_out; + copied_nbyte += tx_nbytes; + to_buf += tx_nbytes; + } + +out: + return copied_nbyte; + +err_out: + cdq->host_head = init_host_head; + cdq->phase_bit = init_phase_bit; + return -EFAULT; +} + static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf, size_t size_nbyte, loff_t *ppos) { @@ -187,8 +257,7 @@ static ssize_t nvme_cdq_fops_read(struct file *filep, char __user *buf, if (!READ_ONCE(cdq->valid_mem)) return -EINVAL; - /* CDQ traversal not implemented yet. */ - return -EOPNOTSUPP; + return nvme_traversecopy_cdq(cdq, nbytes, buf); } /* File reference already dropped by the close path, so don't fput() */ diff --git a/drivers/nvme/host/cdq.h b/drivers/nvme/host/cdq.h index a8c714ab5fada4325463013c2ffb23a117cd8423..96f208264c6144d26aa4ffc34854bb16d3f19af9 100644 --- a/drivers/nvme/host/cdq.h +++ b/drivers/nvme/host/cdq.h @@ -9,6 +9,8 @@ #include "nvme.h" #define NVME_CDQ_MQ_ENTRY_NRBYTES 32 +#define NVME_CDQ_MQ_PHASE_MASK 0x1 +#define NVME_CDQ_MQ_PHASE_OFFSET (NVME_CDQ_MQ_ENTRY_NRBYTES - 1) /* * The CDQ backing is a set of coherent DMA chunks expressed in @@ -17,6 +19,7 @@ #define NVME_CDQ_CHUNK_ORDER 2 #define NVME_CDQ_CHUNK_SIZE (PAGE_SIZE << NVME_CDQ_CHUNK_ORDER) #define NVME_CDQ_PAGES_PER_CHUNK (NVME_CDQ_CHUNK_SIZE / NVME_CTRL_PAGE_SIZE) +#define NVME_CDQ_MQ_ENTRY_PER_CHUNK (NVME_CDQ_CHUNK_SIZE / NVME_CDQ_MQ_ENTRY_NRBYTES) /* Max PRP List pages we are willing to chain to describe a discontiguous CDQ. */ #define MAX_NR_CDQ_PRPS 20 @@ -45,6 +48,10 @@ struct cdq_nvme_queue { /* True if mem for chunks and prps is valid */ bool valid_mem; + /* How far the CDQ was consumed by the host */ + u32 host_head; + u8 phase_bit; + /* Manage refs for read FD and controller xarray */ struct kref ref; }; -- 2.50.1