* [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP
@ 2024-07-17 7:52 Ping Gan
2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan
2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan
0 siblings, 2 replies; 6+ messages in thread
From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw)
To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan
hen running nvmf on SMP platform, current nvme target's RDMA and
TCP use bounded workqueue to handle IO, but when there is other high
workload on the system(eg: kubernetes), the competition between the
bounded kworker and other workload is very radical. To decrease the
resource race of OS among them, this patchset will enable unbounded
workqueue for nvmet-rdma and nvmet-tcp; besides that, it can also
get some performance improvement. And this patchset bases on previous
discussion from below session.
https://lore.kernel.org/lkml/20240717005318.109027-1-jacky_gam_2001@163.com/
Ping Gan (2):
nvmet-tcp: add unbound_wq support for nvmet-tcp
nvmet-rdma: add unbound_wq support for nvmet-rdma
drivers/nvme/target/rdma.c | 10 +++++++++-
drivers/nvme/target/tcp.c | 12 ++++++++++--
2 files changed, 19 insertions(+), 3 deletions(-)
--
2.26.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp
2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan
@ 2024-07-17 7:52 ` Ping Gan
2024-07-17 8:20 ` Hannes Reinecke
2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan
1 sibling, 1 reply; 6+ messages in thread
From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw)
To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan
To define a module parameter use_unbound_wq to enable unbound
workqueue to handle TCP's IO.
Signed-off-by: jackygam2001 <jacky_gam_2001@163.com>
---
drivers/nvme/target/tcp.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 5bff0d5464d1..f71d56843e1a 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -73,6 +73,10 @@ device_param_cb(idle_poll_period_usecs, &set_param_ops,
MODULE_PARM_DESC(idle_poll_period_usecs,
"nvmet tcp io_work poll till idle time period in usecs: Default 0");
+static bool use_unbound_wq;
+module_param(use_unbound_wq, bool, 0444);
+MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false");
+
#ifdef CONFIG_NVME_TARGET_TCP_TLS
/*
* TLS handshake timeout
@@ -2196,9 +2200,13 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = {
static int __init nvmet_tcp_init(void)
{
int ret;
+ unsigned int flags;
+
+ flags = WQ_MEM_RECLAIM | WQ_HIGHPRI;
+ if (use_unbound_wq)
+ flags |= (WQ_UNBOUND | WQ_SYSFS);
- nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq",
- WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
+ nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", flags, 0);
if (!nvmet_tcp_wq)
return -ENOMEM;
--
2.26.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma
2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan
2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan
@ 2024-07-17 7:52 ` Ping Gan
2024-07-17 8:21 ` Hannes Reinecke
1 sibling, 1 reply; 6+ messages in thread
From: Ping Gan @ 2024-07-17 7:52 UTC (permalink / raw)
To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan, Ping Gan
To define a module parameter use_unbound_wq to enable unbound
workqueue to handle RDMA's IO of CQ.
Signed-off-by: jackygam2001 <jacky_gam_2001@163.com>
---
drivers/nvme/target/rdma.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 1eff8ca6a5f1..bfd7106316bc 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -155,6 +155,10 @@ static int nvmet_rdma_srq_size = 1024;
module_param_cb(srq_size, &srq_size_ops, &nvmet_rdma_srq_size, 0644);
MODULE_PARM_DESC(srq_size, "set Shared Receive Queue (SRQ) size, should >= 256 (default: 1024)");
+static bool use_unbound_wq;
+module_param(use_unbound_wq, bool, 0444);
+MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false");
+
static DEFINE_IDA(nvmet_rdma_queue_ida);
static LIST_HEAD(nvmet_rdma_queue_list);
static DEFINE_MUTEX(nvmet_rdma_queue_mutex);
@@ -1259,7 +1263,11 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
*/
nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size;
- queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
+ if (use_unbound_wq)
+ queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
+ queue->comp_vector, IB_POLL_UNBOUND_WORKQUEUE);
+ else
+ queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
queue->comp_vector, IB_POLL_WORKQUEUE);
if (IS_ERR(queue->cq)) {
ret = PTR_ERR(queue->cq);
--
2.26.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp
2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan
@ 2024-07-17 8:20 ` Hannes Reinecke
2024-07-17 8:52 ` Ping Gan
0 siblings, 1 reply; 6+ messages in thread
From: Hannes Reinecke @ 2024-07-17 8:20 UTC (permalink / raw)
To: Ping Gan, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan
On 7/17/24 09:52, Ping Gan wrote:
> To define a module parameter use_unbound_wq to enable unbound
> workqueue to handle TCP's IO.
>
> Signed-off-by: jackygam2001 <jacky_gam_2001@163.com>
Please use your real name 'Ping Gan' as the author name, not the mail alias.
> ---
> drivers/nvme/target/tcp.c | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index 5bff0d5464d1..f71d56843e1a 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -73,6 +73,10 @@ device_param_cb(idle_poll_period_usecs, &set_param_ops,
> MODULE_PARM_DESC(idle_poll_period_usecs,
> "nvmet tcp io_work poll till idle time period in usecs: Default 0");
>
> +static bool use_unbound_wq;
> +module_param(use_unbound_wq, bool, 0444);
> +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false");
> +
> #ifdef CONFIG_NVME_TARGET_TCP_TLS
> /*
> * TLS handshake timeout
> @@ -2196,9 +2200,13 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = {
> static int __init nvmet_tcp_init(void)
> {
> int ret;
> + unsigned int flags;
> +
> + flags = WQ_MEM_RECLAIM | WQ_HIGHPRI;
> + if (use_unbound_wq)
> + flags |= (WQ_UNBOUND | WQ_SYSFS);
>
> - nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq",
> - WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
> + nvmet_tcp_wq = alloc_workqueue("nvmet_tcp_wq", flags, 0);
> if (!nvmet_tcp_wq)
> return -ENOMEM;
>
Otherwise looks good.
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma
2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan
@ 2024-07-17 8:21 ` Hannes Reinecke
0 siblings, 0 replies; 6+ messages in thread
From: Hannes Reinecke @ 2024-07-17 8:21 UTC (permalink / raw)
To: Ping Gan, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan
On 7/17/24 09:52, Ping Gan wrote:
> To define a module parameter use_unbound_wq to enable unbound
> workqueue to handle RDMA's IO of CQ.
>
> Signed-off-by: jackygam2001 <jacky_gam_2001@163.com>
Similar here, please use your real name as the author name.
> ---
> drivers/nvme/target/rdma.c | 10 +++++++++-
> 1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 1eff8ca6a5f1..bfd7106316bc 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -155,6 +155,10 @@ static int nvmet_rdma_srq_size = 1024;
> module_param_cb(srq_size, &srq_size_ops, &nvmet_rdma_srq_size, 0644);
> MODULE_PARM_DESC(srq_size, "set Shared Receive Queue (SRQ) size, should >= 256 (default: 1024)");
>
> +static bool use_unbound_wq;
> +module_param(use_unbound_wq, bool, 0444);
> +MODULE_PARM_DESC(use_unbound_wq, "use unbound workqueue to handle IO request: Default false");
> +
> static DEFINE_IDA(nvmet_rdma_queue_ida);
> static LIST_HEAD(nvmet_rdma_queue_list);
> static DEFINE_MUTEX(nvmet_rdma_queue_mutex);
> @@ -1259,7 +1263,11 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
> */
> nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size;
>
> - queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
> + if (use_unbound_wq)
> + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
> + queue->comp_vector, IB_POLL_UNBOUND_WORKQUEUE);
> + else
> + queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
> queue->comp_vector, IB_POLL_WORKQUEUE);
> if (IS_ERR(queue->cq)) {
> ret = PTR_ERR(queue->cq);
Otherwise:
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp
2024-07-17 8:20 ` Hannes Reinecke
@ 2024-07-17 8:52 ` Ping Gan
0 siblings, 0 replies; 6+ messages in thread
From: Ping Gan @ 2024-07-17 8:52 UTC (permalink / raw)
To: hare, sagi, hch, kch, linux-nvme, linux-kernel; +Cc: ping.gan
> On 7/17/24 09:52, Ping Gan wrote:
>> To define a module parameter use_unbound_wq to enable unbound
>> workqueue to handle TCP's IO.
>>
>> Signed-off-by: jackygam2001 <jacky_gam_2001@163.com>
>
> Please use your real name 'Ping Gan' as the author name, not the mail
> alias.
Sorry, forgot to change, will do in next version.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-07-17 8:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-17 7:52 [PATCH 0/2] nvmet: support unbound_wq for RDMA and TCP Ping Gan
2024-07-17 7:52 ` [PATCH 1/2] nvmet-tcp: add unbound_wq support for nvmet-tcp Ping Gan
2024-07-17 8:20 ` Hannes Reinecke
2024-07-17 8:52 ` Ping Gan
2024-07-17 7:52 ` [PATCH 2/2] nvmet-rdma: add unbound_wq support for nvmet-rdma Ping Gan
2024-07-17 8:21 ` Hannes Reinecke
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®