Re: [PATCH] nvme-rdma: parallelize I/O queue allocation and startup

[Keith Busch <kbusch@kernel.org>]

On Thu, May 28, 2026 at 06:13:54PM -0600, Surabhi Gogte wrote:
> Refactor nvme rdma I/O queue setup to use parallel work queues,
> combining allocation and startup into a single parallel operation per
> queue. This reduces connection and reconnection setup time when there
> are delays in establishing connections, which is especially important
> for high-core-count hosts.

Seems like a good idea.

> +struct workqueue_struct *nvme_setup_wq;
> +EXPORT_SYMBOL_GPL(nvme_setup_wq);

RDMA is the only one using it, so I don't think we want burden the
common core with this. But I also think a new workqueue is overkill.
Just use the async APIs. Suggestions below:

> +static void nvme_rdma_setup_queue_work(struct work_struct *work)
>  {
> -	int i, ret = 0;
> +	struct nvme_rdma_queue *queue =
> +		container_of(work, struct nvme_rdma_queue, setup_work);

static void nvme_rdma_setup_queue_async(void *data, async_cookie_t cookie)
{
	struct nvme_rdma_queue *queue = data;
	...
}

> +static int nvme_rdma_setup_io_queues(struct nvme_rdma_ctrl *ctrl, int first,
> +				     int last, size_t queue_size)
>  {

...

> +	for (i = 0; i < nr_queues; i++) {
> +		struct nvme_rdma_queue *queue = &ctrl->queues[first + i];
> 
> -	nvmf_set_io_queues(opts, nr_io_queues, ctrl->io_queues);
> -	for (i = 1; i < ctrl->ctrl.queue_count; i++) {
> -		ret = nvme_rdma_alloc_queue(ctrl, i,
> -				ctrl->ctrl.sqsize + 1);
> -		if (ret)
> -			goto out_free_queues;
> +		queue->ctrl = ctrl;
> +		queue->queue_idx = first + i;
> +		queue->queue_size = queue_size;
> +		INIT_WORK(&queue->setup_work, nvme_rdma_setup_queue_work);
> +		queue_work(nvme_setup_wq, &queue->setup_work);
>  	}

>  
> -	return 0;
> +	for (i = 0; i < nr_queues; i++)
> +		flush_work(&ctrl->queues[first + i].setup_work);

And this part can be:

	ASYNC_DOMAIN_EXCLUSIVE(domain);
	...
	for (i = 0; i < nr_queues; i++) {
		...
		async_schedule_domain(nvme_rdma_setup_queue_async, queue,
			&domain);
        }

        async_synchronize_full_domain(&domain);

It's simpler this way.