* [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP @ 2024-07-17 7:52 Ping Gan 2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan 2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan 0 siblings, 2 replies; 6+ messages in thread From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw) To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan hen running nvmf on SMP platform, current nvme target's RDMA and TCP use bounded workqueue to handle IO, but when there is other high workload on the system(eg: kubernetes), the competition between the bounded kworker and other workload is very radical. To decrease the resource race of OS among them, this patchset will enable unbounded workqueue for nvmet-rdma and nvmet-tcp; besides that, it can also get some performance improvement. And this patchset bases on previous discussion from below session. https://lore.kernel.org/lkml/20240717005318.109027-1-jacky_gam_2001@163.com/ Ping Gan (2): nvmet-tcp: add unbound_wq support for nvmet-tcp nvmet-rdma: add unbound_wq support for nvmet-rdma drivers/nvme/target/rdma.c | 10 +++++++++- drivers/nvme/target/tcp.c | 12 ++++++++++-- 2 files changed, 19 insertions(+), 3 deletions(-) -- 2.26.2 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp 2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan @ 2024-07-17 7:52 ` Ping Gan 2024-07-17 8:20 ` Hannes Reinecke 2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan 1 sibling, 1 reply; 6+ messages in thread From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw) To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan To define a module parameter use_unbound_wq to enable unbound workqueue to handle TCP's IO. Signed-off-by: jackygam2001 <jacky_gam_2001@163.com> --- drivers/nvme/target/tcp.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 5bff0d5464d1..f71d56843e1a 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -73,6 +73,10 @@ device_param_cb(idle_poll_period_usecs, &set_param_ops, MODULE_PARM_DESC(idle_poll_period_usecs, "nvmet tcp io_work poll till idle time period in usecs: Default 0"); +static bool use_unbound_wq; +module_param(use_unbound_wq, bool, 0444); +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false"); + #ifdef CONFIG_NVME_TARGET_TCP_TLS /* * TLS handshake timeout @@ -2196,9 +2200,13 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = { static int __init nvmet_tcp_init(void) { int ret; + unsigned int flags; + + flags = WQ_MEM_RECLAIM | WQ_HIGHPRI; + if (use_unbound_wq) + flags |= (WQ_UNBOUND | WQ_SYSFS); - nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", - WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); + nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", flags, 0); if (!nvmet_tcp_wq) return -ENOMEM; -- 2.26.2 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp 2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan @ 2024-07-17 8:20 ` Hannes Reinecke 2024-07-17 8:52 ` Ping Gan 0 siblings, 1 reply; 6+ messages in thread From: Hannes Reinecke @ 2024-07-17 8:20 UTC (permalink / raw) To: Ping Gan, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan On 7/17/24 09:52, Ping Gan wrote: > To define a module parameter use_unbound_wq to enable unbound > workqueue to handle TCP's IO. > > Signed-off-by: jackygam2001 <jacky_gam_2001@163.com> Please use your real name 'Ping Gan' as the author name, not the mail alias. > --- > drivers/nvme/target/tcp.c | 12 ++++++++++-- > 1 file changed, 10 insertions(+), 2 deletions(-) > > diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c > index 5bff0d5464d1..f71d56843e1a 100644 > --- a/drivers/nvme/target/tcp.c > +++ b/drivers/nvme/target/tcp.c > @@ -73,6 +73,10 @@ device_param_cb(idle_poll_period_usecs, &set_param_ops, > MODULE_PARM_DESC(idle_poll_period_usecs, > "nvmet tcp io_work poll till idle time period in usecs: Default 0"); > > +static bool use_unbound_wq; > +module_param(use_unbound_wq, bool, 0444); > +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false"); > + > #ifdef CONFIG_NVME_TARGET_TCP_TLS > /* > * TLS handshake timeout > @@ -2196,9 +2200,13 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = { > static int __init nvmet_tcp_init(void) > { > int ret; > + unsigned int flags; > + > + flags = WQ_MEM_RECLAIM | WQ_HIGHPRI; > + if (use_unbound_wq) > + flags |= (WQ_UNBOUND | WQ_SYSFS); > > - nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", > - WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); > + nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", flags, 0); > if (!nvmet_tcp_wq) > return -ENOMEM; > Otherwise looks good. Reviewed-by: Hannes Reinecke <hare@suse.de> Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp 2024-07-17 8:20 ` Hannes Reinecke @ 2024-07-17 8:52 ` Ping Gan 0 siblings, 0 replies; 6+ messages in thread From: Ping Gan @ 2024-07-17 8:52 UTC (permalink / raw) To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan > On 7/17/24 09:52, Ping Gan wrote: >> To define a module parameter use_unbound_wq to enable unbound >> workqueue to handle TCP's IO. >> >> Signed-off-by: jackygam2001 <jacky_gam_2001@163.com> > > Please use your real name 'Ping Gan' as the author name, not the mail > alias. Sorry, forgot to change, will do in next version. ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma 2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan 2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan @ 2024-07-17 7:52 ` Ping Gan 2024-07-17 8:21 ` Hannes Reinecke 1 sibling, 1 reply; 6+ messages in thread From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw) To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan To define a module parameter use_unbound_wq to enable unbound workqueue to handle RDMA's IO of CQ. Signed-off-by: jackygam2001 <jacky_gam_2001@163.com> --- drivers/nvme/target/rdma.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 1eff8ca6a5f1..bfd7106316bc 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -155,6 +155,10 @@ static int nvmet_rdma_srq_size = 1024; module_param_cb(srq_size, &srq_size_ops, &nvmet_rdma_srq_size, 0644); MODULE_PARM_DESC(srq_size, "set Shared Receive Queue (SRQ) size, should >= 256 (default: 1024)"); +static bool use_unbound_wq; +module_param(use_unbound_wq, bool, 0444); +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false"); + static DEFINE_IDA(nvmet_rdma_queue_ida); static LIST_HEAD(nvmet_rdma_queue_list); static DEFINE_MUTEX(nvmet_rdma_queue_mutex); @@ -1259,7 +1263,11 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue) */ nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size; - queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, + if (use_unbound_wq) + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, + queue->comp_vector, IB_POLL_UNBOUND_WORKQUEUE); + else + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, queue->comp_vector, IB_POLL_WORKQUEUE); if (IS_ERR(queue->cq)) { ret = PTR_ERR(queue->cq); -- 2.26.2 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma 2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan @ 2024-07-17 8:21 ` Hannes Reinecke 0 siblings, 0 replies; 6+ messages in thread From: Hannes Reinecke @ 2024-07-17 8:21 UTC (permalink / raw) To: Ping Gan, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan On 7/17/24 09:52, Ping Gan wrote: > To define a module parameter use_unbound_wq to enable unbound > workqueue to handle RDMA's IO of CQ. > > Signed-off-by: jackygam2001 <jacky_gam_2001@163.com> Similar here, please use your real name as the author name. > --- > drivers/nvme/target/rdma.c | 10 +++++++++- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c > index 1eff8ca6a5f1..bfd7106316bc 100644 > --- a/drivers/nvme/target/rdma.c > +++ b/drivers/nvme/target/rdma.c > @@ -155,6 +155,10 @@ static int nvmet_rdma_srq_size = 1024; > module_param_cb(srq_size, &srq_size_ops, &nvmet_rdma_srq_size, 0644); > MODULE_PARM_DESC(srq_size, "set Shared Receive Queue (SRQ) size, should >= 256 (default: 1024)"); > > +static bool use_unbound_wq; > +module_param(use_unbound_wq, bool, 0444); > +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false"); > + > static DEFINE_IDA(nvmet_rdma_queue_ida); > static LIST_HEAD(nvmet_rdma_queue_list); > static DEFINE_MUTEX(nvmet_rdma_queue_mutex); > @@ -1259,7 +1263,11 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue) > */ > nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size; > > - queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, > + if (use_unbound_wq) > + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, > + queue->comp_vector, IB_POLL_UNBOUND_WORKQUEUE); > + else > + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1, > queue->comp_vector, IB_POLL_WORKQUEUE); > if (IS_ERR(queue->cq)) { > ret = PTR_ERR(queue->cq); Otherwise: Reviewed-by: Hannes Reinecke <hare@suse.de> Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-07-17 8:53 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan 2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan 2024-07-17 8:20 ` Hannes Reinecke 2024-07-17 8:52 ` Ping Gan 2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan 2024-07-17 8:21 ` Hannes Reinecke
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®