From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A0FC326D4B for ; Mon, 17 Nov 2025 09:23:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=203.254.224.25 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763371420; cv=none; b=sYlUJ7VhYqfosmFAZUM8LqkliQGFMttBaq1UdUqE0geIjqFSsQoT8GCLoUAM1uW0KuDKfHzsieI42FwcbjAcwqvMqy/q0HRKaNyiHwTv071lOeJxOf2qtkAlbII9oBrrmoghbIoE80Z27+ArRZMzScCfzv8adzkwy8g1qy28iFQ= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763371420; c=relaxed/simple; bh=5Le79+xZo0pnPqRtJoLbHkAQdzsDca6tpNt/0CwyC9M=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Content-Type: References; b=YN3J05xmDSoQwnZtUECHDBy7jUJ2QU3jNJVXWOqqI+Ds/1pV1Fsq2ghJy1NsndIbFNOo7ijyMf8mJrVCXh+uRoPyD4Oshy0XV3iUrH1K1f+8LaGC9G+mA46hnXkGSCHTibZSUIq1Bz2KzWdMjMRGrKixdm8BeaiMTMSAdvyVtXk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com; spf=pass smtp.mailfrom=samsung.com; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b=PvRo0BOK; arc=none smtp.client-ip=203.254.224.25 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=samsung.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=samsung.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=samsung.com header.i=@samsung.com header.b="PvRo0BOK" Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20251117092333epoutp02d9d62ed83ad3a0e4086dbf7d9cfe5eb9~4wPUdq8qv0888808888epoutp02U for ; Mon, 17 Nov 2025 09:23:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20251117092333epoutp02d9d62ed83ad3a0e4086dbf7d9cfe5eb9~4wPUdq8qv0888808888epoutp02U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1763371413; bh=0izvina2OzYAXBqg81C+fhf6rBfcPmphJHfa/0+bf+o=; h=From:To:Cc:Subject:Date:References:From; b=PvRo0BOKpu8hQOll56tqkoCsuKa6X0jfD8A+tUJqG0pLWgW5yxxauT3xhAHxJr/Zw vTJeDaXavhHDegg+LEhUtNHq/HhrIELuwk39pRa7LBYTG+Do/9J5yoyAtTCTwHxgQK N219RaE9m65FzCJQraQwJVpaBDiMj2iPb3JH44P8= Received: from epsnrtp02.localdomain (unknown [182.195.42.154]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPS id 20251117092332epcas5p2da3b920c56548b8c08c218a2181e1cd4~4wPUEt-bR0872708727epcas5p2o; Mon, 17 Nov 2025 09:23:32 +0000 (GMT) Received: from epcas5p1.samsung.com (unknown [182.195.38.91]) by epsnrtp02.localdomain (Postfix) with ESMTP id 4d92Ng4vRkz2SSKX; Mon, 17 Nov 2025 09:23:31 +0000 (GMT) Received: from epsmtip2.samsung.com (unknown [182.195.34.31]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20251117085321epcas5p3de0647f01a818db024bea32870f223f4~4v089sQ8o2847328473epcas5p3a; Mon, 17 Nov 2025 08:53:21 +0000 (GMT) Received: from node122.. (unknown [109.105.118.122]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20251117085320epsmtip2650ae9842d13f4a70f4eb8a284f4787c~4v08SduAF1001810018epsmtip2c; Mon, 17 Nov 2025 08:53:20 +0000 (GMT) From: Xue He To: axboe@kernel.dk, ming.lei@redhat.com Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai@fnnas.com, hexue Subject: [PATCH v6] block: plug attempts to batch allocate tags multiple times Date: Mon, 17 Nov 2025 08:48:45 +0000 Message-Id: <20251117084845.254680-1-xue01.he@samsung.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CMS-MailID: 20251117085321epcas5p3de0647f01a818db024bea32870f223f4 X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P cpgsPolicy: CPGSC10-505,Y X-CFilter-Loop: Reflected X-CMS-RootMailID: 20251117085321epcas5p3de0647f01a818db024bea32870f223f4 References: This patch aims to enable batch allocation of sufficient tags after batch IO submission with plug mechanism, thereby avoiding the need for frequent individual requests when the initial allocation is insufficient. ----------------------------------------------------------- HW: 16 CPUs/16 poll queues Disk: Samsung PM9A3 Gen4 3.84T CMD: [global] ioengine=io_uring group_reporting=1 time_based=1 runtime=1m refill_buffers=1 norandommap=1 randrepeat=0 fixedbufs=1 registerfiles=1 rw=randread iodepth=128 iodepth_batch_submit=32 iodepth_batch_complete_min=32 iodepth_batch_complete_max=128 iodepth_low=32 bs=4k numjobs=1 direct=1 hipri=1 [job1] filename=/dev/nvme0n1 name=batch_test ------------------------------------------------------------ Perf: base code: __blk_mq_alloc_requests() 1.47% patch: __blk_mq_alloc_requests() 0.78% ------------------------------------------------------------ --- changes since v1: - Modify multiple batch registrations into a single loop to achieve the batch quantity changes since v2: - Modify the call location of remainder handling - Refactoring sbitmap cleanup time changes since v3: - Add handle operation in loop - Add helper sbitmap_find_bits_in_word changes since v4: - Split blk-mq.c changes from sbitmap changes since v5: - Add workload with perf - Modify over-counting bug Signed-off-by: hexue --- block/blk-mq.c | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index d626d32f6e57..9e6fca1b5fb7 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -467,26 +467,31 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data) unsigned long tag_mask; int i, nr = 0; - tag_mask = blk_mq_get_tags(data, data->nr_tags, &tag_offset); - if (unlikely(!tag_mask)) - return NULL; + do { + tag_mask = blk_mq_get_tags(data, data->nr_tags, &tag_offset); + if (unlikely(!tag_mask)) { + if (nr == 0) + return NULL; + break; + } + tags = blk_mq_tags_from_data(data); + for (i = 0; tag_mask; i++) { + if (!(tag_mask & (1UL << i))) + continue; + tag = tag_offset + i; + prefetch(tags->static_rqs[tag]); + tag_mask &= ~(1UL << i); + rq = blk_mq_rq_ctx_init(data, tags, tag); + rq_list_add_head(data->cached_rqs, rq); + data->nr_tags--; + nr++; + } + } while (data->nr_tags); - tags = blk_mq_tags_from_data(data); - for (i = 0; tag_mask; i++) { - if (!(tag_mask & (1UL << i))) - continue; - tag = tag_offset + i; - prefetch(tags->static_rqs[tag]); - tag_mask &= ~(1UL << i); - rq = blk_mq_rq_ctx_init(data, tags, tag); - rq_list_add_head(data->cached_rqs, rq); - nr++; - } - if (!(data->rq_flags & RQF_SCHED_TAGS)) - blk_mq_add_active_requests(data->hctx, nr); /* caller already holds a reference, add for remainder */ percpu_ref_get_many(&data->q->q_usage_counter, nr - 1); - data->nr_tags -= nr; + if (!(data->rq_flags & RQF_SCHED_TAGS)) + blk_mq_add_active_requests(data->hctx, nr); return rq_list_pop(data->cached_rqs); } -- 2.34.1