* [RFC PATCH 0/2] nvme-tcp: remove the direct send path
@ 2026-11-22 1:18 Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 1/2] " Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 2/2] nvme-tcp: remove send_mutex Seiji Nishikawa
0 siblings, 2 replies; 3+ messages in thread
From: Seiji Nishikawa @ 2026-11-22 1:18 UTC (permalink / raw)
To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
Cc: Hannes Reinecke, Chris Leech, Seiji Nishikawa, linux-nvme, linux-kernel
Hannes suggested removing the direct send path in
nvme_tcp_queue_request() while reviewing commit 5db358f3a7f9 ("nvme-tcp:
fix io_work priority inversion when send_mutex is held"). This series
does that, on top of nvme-7.4.
Patch 1 removes the direct send path, so io_work is the only sender.
Patch 2 removes send_mutex, which has nothing left to protect after
patch 1.
The direct send path was added to cut latency when the queue is idle,
so I measured it on two machines with 1Gb NICs. The target is nvmet-tcp
backed by null_blk (irqmode=0). The host runs fio like this, 60 seconds
per run and 3 runs per case:
fio --name=qd1 --filename=/dev/nvme0n1 --direct=1 --ioengine=libaio \
--bs=4k --rw=randread --iodepth=1 --numjobs=1 --cpus_allowed=2 \
--time_based --runtime=60 --ramp_time=5
Average of the 3 runs at queue depth 1, latency in usec, "patched" is
this series:
IOPS slat clat lat clat p99
randread base 4153 7.1 232.3 239.3 277
patched 4154 4.7 234.7 239.4 280
randwrite base 2472 24.5 375.6 400.1 553
patched 2460 14.3 387.6 402.0 570
The send time moves from slat to clat. With the direct send path,
io_submit() itself sends the command on the socket, so that time is
counted in slat. Without it, io_submit() only queues the request and
returns, and io_work sends it, so the same time is counted in clat. lat
is the sum of the two, and it changes by 0.1 usec for randread and 1.9
usec (+0.5%) for randwrite.
At queue depth 32 with 4 jobs, both kernels reach the 1Gb link limit
(28.2k IOPS), so there is no difference there.
On the same two machines, I also ran 5 controller resets during a 120
second randrw fio run on the patched kernel, and fio finished with no
errors. Separately, blktests nvme over tcp on a VM with an nvmet tcp
loopback target gives the same results with and without this series.
On a 1Gb network one I/O takes about 240 us, so a difference of a few
microseconds is easy to miss. Numbers from faster networks would show
whether it matters there. I am posting this as RFC so the performance
side can be discussed on the list.
Seiji Nishikawa (2):
nvme-tcp: remove the direct send path
nvme-tcp: remove send_mutex
drivers/nvme/host/tcp.c | 60 +++++++----------------------------------
1 file changed, 9 insertions(+), 51 deletions(-)
base-commit: 68099cf27253a8c89d703a282d3d811446d9453b
--
2.56.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [RFC PATCH 1/2] nvme-tcp: remove the direct send path
2026-11-22 1:18 [RFC PATCH 0/2] nvme-tcp: remove the direct send path Seiji Nishikawa
@ 2026-11-22 1:18 ` Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 2/2] nvme-tcp: remove send_mutex Seiji Nishikawa
1 sibling, 0 replies; 3+ messages in thread
From: Seiji Nishikawa @ 2026-11-22 1:18 UTC (permalink / raw)
To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
Cc: Hannes Reinecke, Chris Leech, Seiji Nishikawa, linux-nvme, linux-kernel
nvme_tcp_queue_request() sends the request directly from the submitting
context when it runs on the io_cpu of the queue, the queue is empty and
it can take send_mutex. Commit db5ad6b7f8cd ("nvme-tcp: try to send
request in queue_rq context") added this to skip the workqueue hop when
the queue is idle.
Since then this path has needed a series of fixes for races and
priority inversions between it and io_work. These are
commit 5c11f7d9f843 ("nvme-tcp: Fix possible race of io_work and direct
send"), commit a0fdd1418007 ("nvme-tcp: rerun io_work if req_list is
not empty"), commit 70f437fb4395 ("nvme-tcp: fix io_work priority
inversion"), commit 08acb54b063a ("nvme-tcp: defer TLS inline send to
io_work") and commit 5db358f3a7f9 ("nvme-tcp: fix io_work priority
inversion when send_mutex is held").
All of them come from having 2 contexts that send on the same queue and
coordinate through send_mutex, while io_work runs on a high priority
workqueue and the submitting task does not.
Remove the direct send path and always send from io_work. TLS queues
already work this way since 08acb54b063a. With a single sender, io_work
has to be queued whenever the last request of a batch is added, so the
has_pending check in nvme_tcp_queue_request() goes away too, and
nvme_tcp_send_all() has no caller left.
Suggested-by: Hannes Reinecke <hare@suse.de>
Assisted-by: LLM
Signed-off-by: Seiji Nishikawa <snishika@redhat.com>
---
drivers/nvme/host/tcp.c | 33 ++-------------------------------
1 file changed, 2 insertions(+), 31 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 62a594b8ab9b..2b832ac3068a 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -213,7 +213,6 @@ struct nvme_tcp_ctrl {
static struct workqueue_struct *nvme_tcp_wq;
static const struct blk_mq_ops nvme_tcp_mq_ops;
static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
-static int nvme_tcp_try_send(struct nvme_tcp_queue *queue);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/* lockdep can detect a circular dependency of the form
@@ -420,16 +419,6 @@ static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,
}
}
-static inline void nvme_tcp_send_all(struct nvme_tcp_queue *queue)
-{
- int ret;
-
- /* drain the send queue as much as we can... */
- do {
- ret = nvme_tcp_try_send(queue);
- } while (ret > 0);
-}
-
static inline bool nvme_tcp_queue_has_pending(struct nvme_tcp_queue *queue)
{
return !list_empty(&queue->send_list) ||
@@ -446,28 +435,10 @@ static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,
bool last)
{
struct nvme_tcp_queue *queue = req->queue;
- bool empty;
- empty = llist_add(&req->lentry, &queue->req_list) &&
- list_empty(&queue->send_list) && !queue->request;
-
- /*
- * if we're the first on the send_list and we can try to send
- * directly, otherwise queue io_work. Also, only do that if we
- * are on the same cpu, so we don't introduce contention.
- *
- * TLS kTLS send takes ctx->tx_lock while blk_mq holds set->srcu.
- * lockdep reports circular locking via elevator_lock. Defer TLS
- * sends to the io workqueue instead of inline from this path.
- */
- if (queue->io_cpu == raw_smp_processor_id() &&
- !nvme_tcp_queue_tls(queue) &&
- empty && mutex_trylock(&queue->send_mutex)) {
- nvme_tcp_send_all(queue);
- mutex_unlock(&queue->send_mutex);
- }
+ llist_add(&req->lentry, &queue->req_list);
- if (last && nvme_tcp_queue_has_pending(queue))
+ if (last)
queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
}
--
2.56.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [RFC PATCH 2/2] nvme-tcp: remove send_mutex
2026-11-22 1:18 [RFC PATCH 0/2] nvme-tcp: remove the direct send path Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 1/2] " Seiji Nishikawa
@ 2026-11-22 1:18 ` Seiji Nishikawa
1 sibling, 0 replies; 3+ messages in thread
From: Seiji Nishikawa @ 2026-11-22 1:18 UTC (permalink / raw)
To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
Cc: Hannes Reinecke, Chris Leech, Seiji Nishikawa, linux-nvme, linux-kernel
With the direct send path gone, io_work is the only context that sends
on a queue. A work item never runs concurrently with itself, so
send_mutex does not protect anything anymore.
Remove it, together with the mutex_trylock() in nvme_tcp_io_work() and
the got_send_mutex check, which is now always true.
Assisted-by: LLM
Signed-off-by: Seiji Nishikawa <snishika@redhat.com>
---
drivers/nvme/host/tcp.c | 27 +++++++--------------------
1 file changed, 7 insertions(+), 20 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 2b832ac3068a..b116268fe2f5 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -142,7 +142,6 @@ struct nvme_tcp_queue {
int io_cpu;
struct mutex queue_lock;
- struct mutex send_mutex;
struct mutex pf_cache_lock;
struct llist_head req_list;
struct list_head send_list;
@@ -1482,18 +1481,13 @@ static void nvme_tcp_io_work(struct work_struct *w)
do {
bool pending = false;
- bool got_send_mutex = false;
int result;
- if (mutex_trylock(&queue->send_mutex)) {
- got_send_mutex = true;
- result = nvme_tcp_try_send(queue);
- mutex_unlock(&queue->send_mutex);
- if (result > 0)
- pending = true;
- else if (unlikely(result < 0))
- break;
- }
+ result = nvme_tcp_try_send(queue);
+ if (result > 0)
+ pending = true;
+ else if (unlikely(result < 0))
+ break;
result = nvme_tcp_try_recv(queue);
if (result > 0)
@@ -1501,13 +1495,8 @@ static void nvme_tcp_io_work(struct work_struct *w)
else if (unlikely(result < 0))
return;
- /*
- * did we get some space after spending time in recv?
- * Only check this if we could take send_mutex. Otherwise
- * the task holding it sends the requests and queues io_work
- * again after it releases the mutex.
- */
- if (got_send_mutex && nvme_tcp_queue_has_pending(queue) &&
+ /* did we get some space after spending time in recv? */
+ if (nvme_tcp_queue_has_pending(queue) &&
sk_stream_is_writeable(queue->sock->sk))
pending = true;
@@ -2087,7 +2076,6 @@ static void nvme_tcp_init_queue(struct nvme_tcp_ctrl *ctrl, int qid)
queue->ctrl = ctrl;
mutex_init(&queue->queue_lock);
- mutex_init(&queue->send_mutex);
mutex_init(&queue->pf_cache_lock);
INIT_WORK(&queue->io_work, nvme_tcp_io_work);
queue->hdr_digest = ctrl->ctrl.opts->hdr_digest;
@@ -2099,7 +2087,6 @@ static void nvme_tcp_exit_queue(struct nvme_tcp_ctrl *ctrl, int qid)
struct nvme_tcp_queue *queue = &ctrl->queues[qid];
mutex_destroy(&queue->queue_lock);
- mutex_destroy(&queue->send_mutex);
mutex_destroy(&queue->pf_cache_lock);
}
--
2.56.1
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-22 1:19 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-22 1:18 [RFC PATCH 0/2] nvme-tcp: remove the direct send path Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 1/2] " Seiji Nishikawa
2026-11-22 1:18 ` [RFC PATCH 2/2] nvme-tcp: remove send_mutex Seiji Nishikawa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®