From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dy1-f178.google.com (mail-dy1-f178.google.com [74.125.82.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F165238AC88 for ; Mon, 8 Jun 2026 06:30:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780900210; cv=none; b=lBMTyhieW5XnrvizUYTCrkr/Fq6MYR6E0fvpGnrTMRHDLQzfaf2iBpIDrgUeOXN4/aGFqWKK0z2YTE8O2HDtPeSmXCKCk2Sm8uo01lp2/rtAeTrrZXBY02tQwd+4VaRheI7CsrKZRxBK1M7fkZ05+VHcVx5f1Qje5lADFc54ndY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780900210; c=relaxed/simple; bh=EC6zXd58d8z49QA+EzUwoN6+QMz3OHbzht5PlmcPBcQ=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=Fwn0q7zVkCKq3VLWFQA3AcqMEjr9IbsWTZbQATuG+pbAgNsPUAHDlVrO2MraR8UBmBpaNwSgM8Zy9mhcjU5Vdp+afO8tU6z+SR+VXLg01lBYyGI/WuJmegjQJ8lEE0oF9p86gygcT/TgWovru/TCkpV025iXlIrmqP1AL3ZzewQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FI3KbJ0c; arc=none smtp.client-ip=74.125.82.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FI3KbJ0c" Received: by mail-dy1-f178.google.com with SMTP id 5a478bee46e88-3075ce9c05aso7127323eec.1 for ; Sun, 07 Jun 2026 23:30:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1780900206; x=1781505006; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=qDYJ9YnYahbdpvzrWk3FO5n4lndwP1vG0GrraJw28yE=; b=FI3KbJ0ckQM4hXhUKOJ1N+luv0Mpej3sHPCB48/SDAkUumrC3Htxt0BxptpUYLc22U OokxLp10u9FU/VSxprnj+FnJ/hs+tFk6CO8/NXX67hMMzu5iTXnQ50gcYk+NhpvUpgEb 1xwtmeHYpksUuuaPLNdOUBlMQD/DCi/Ov5a1BTW24G3DLnCxY75vqzG03pu8pV5U5pdZ ga+uD084camyL6J6DMwY/PT58dH0XPnYmSUwZaGkWqTrQf3mhhzCQk1ON7ouWw+BIOgy 2XDapsZf/Y0+MgMcffzmKjold1YJ6ZKYYNspXKDgGM9X/R1IhqZEmh/5n4rkw/O8SXjJ nUDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780900206; x=1781505006; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qDYJ9YnYahbdpvzrWk3FO5n4lndwP1vG0GrraJw28yE=; b=W0b42B5ZFitvVmX3GkGPRvJw0mUymuvuxIp2vcxUILJACSiiGZ/f5Wk6GqeLhsTJ4X yb441ZJZgaxf/+6uJUIDxUvPCrJPhZUntPNfPEixq0yQv1oOiN1JAssB1Wq/Zsh45zRP 74MgYWuyCWCm+mxJuGdPJ6r2a5PAKu+kBgMUwIKnbzL5IHNUklk4pz7xtwIpfhzakbKA XCjIpZBBk/tLCtl0ThSV7hojJRoOwc/2OcH08MwTkr068YY2xtiOcZ/fOUMYxICYjL4e kLotWA9rkzuCp4aiYKnoOTBqu7aT3b+alFNdeIrPFXWr/1+G6+N2PlCnSKacaZ6etmy3 g2ww== X-Gm-Message-State: AOJu0YxlMb8niShFcdQ8QjZ8NuvDamJryXa74b3nJmzpMmq8kPH3sUpA 4uaj7O5PSMmM6w2AqE2bTaDmK+bM+IoTXV6zgOkBdITzJb6H3T6RNcC6ffPVhg== X-Gm-Gg: Acq92OEcPftSz8c0h893EVmQCw5DK38u2mWtgWF8ixDodvCBAhhqhkn1WKdTR95Y6DD luUxzQIOGKTPGmGbBRH9JyV7XvAp8pffdl+qh4C0mOpTIa0RkCFatoCyVSkuLkPDdw5HAq7OxMC rrkZjeUJ8En6JFVpa1aIjUvJ6aXeRMIls7VjStyqqnJY64tUWW/7VEsyFs7IHFvHtMDFF1PpQ2h a3nhae+pvPF/47Dv5XqFgn6tIJ5DJ4CV0yz/hu++cHUBPL5sUXAjhqxlYyUxtjrO4qzR8QbDA5N ICHPiltsXUqLP09qiHcgC2Wov1YkAjpaysnpEhTuRho1bTBdvPyzV+esS7Se2UlISZuy0aGBzZ6 AKUE3+y4ciy/FGZ4gt/NMBN+rbDfE8CpLyzd91LCFY8a6MTJkciWs81LfoDMlXWojGdLoUQL6HH BY3AYgYqen7iawnl8oBqAQ9q4Z9C7VlP5h0iB5fAOzmBp4laaLKvu65lZgF6iR5JvERL+UZ6Hf8 YNE7bzfxhnSLNzUErEGYADzERN6agUfeKtZ X-Received: by 2002:a05:7300:4307:b0:2ce:3aa1:d39b with SMTP id 5a478bee46e88-3077b1e1921mr7540153eec.20.1780900205592; Sun, 07 Jun 2026 23:30:05 -0700 (PDT) Received: from 5163NRD-SPRABHU.ssi.samsung.com (c-73-222-128-44.hsd1.ca.comcast.net. [73.222.128.44]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3074dea8708sm16871033eec.15.2026.06.07.23.30.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Jun 2026 23:30:05 -0700 (PDT) From: sw.prabhu6@gmail.com To: axboe@kernel.dk, io-uring@vger.kernel.org Cc: linux-kernel@vger.kernel.org, dave@stgolabs.net, dongjoo.seo1@samsung.com, Swarna Prabhu , Swarna Prabhu Subject: [RFC v1] io_uring/rsrc: add fast path huge page handling in buffer registration Date: Sun, 7 Jun 2026 23:29:37 -0700 Message-Id: <20260608062937.804758-1-sw.prabhu6@gmail.com> X-Mailer: git-send-email 2.39.5 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Swarna Prabhu io_uring sqe buffer registration path returns pinned user pages in 4k granularity. If the first pinned page is in a hugetlb folio and pages[nr_pages - 1] is also in the same folio then store a single page entry and report *npages = 1 while dropping nr_pages - 1 of the pin references it took earlier. io_uring has support to identify and coalesce multi-hugepage-backed fixed buffers from the function 'io_check_coalesce_buffer()'. However we need to iterate over the entire page array and this patch bypasses the additional checks for this case. The fast path reduces the overall sqe buffer registration time that are backed by huge pages. Measured with fio on bare metal backed by 1024 boot-allocated 2MB hugetlb pages and setting the cpu cores to governor for max performance. (hugepages=1024,hugepage_size=2M): fio --ioengine=io_uring --rw=randwrite --bs=1M --size=2G --iodepth=256 --direct=1 --numjobs=5 --fixedbufs=1 --registerfiles=1 --iomem=mmaphuge --hugepage-size=2M. Avg across 3 runs: Metric Upstream(7.1-rc1) Patched Delta Reg time(io_sqe_buffer_register): 3797ns 2970ns -21.8% Total reg for workload: 14.35ms 11.34ms -21.9% fio write bandwidth: 1416MiB/s 1416MiB/s No regression Signed-off-by: Swarna Prabhu --- io_uring/memmap.c | 66 +++++++++++++++++++++++++++++++++++++++++++++-- io_uring/memmap.h | 3 +++ io_uring/rsrc.c | 9 +++++-- 3 files changed, 74 insertions(+), 4 deletions(-) diff --git a/io_uring/memmap.c b/io_uring/memmap.c index 4f9b439319c4..957e67d2d8e8 100644 --- a/io_uring/memmap.c +++ b/io_uring/memmap.c @@ -37,11 +37,11 @@ static bool io_mem_alloc_compound(struct page **pages, int nr_pages, return true; } -struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) +struct page **io_pin_pages_alloc(unsigned long uaddr, unsigned long len, + unsigned long *nr_pages_out) { unsigned long start, end, nr_pages; struct page **pages; - int ret; if (check_add_overflow(uaddr, len, &end)) return ERR_PTR(-EOVERFLOW); @@ -60,6 +60,20 @@ struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) if (!pages) return ERR_PTR(-ENOMEM); + *nr_pages_out = nr_pages; + return pages; +} + +struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) +{ + unsigned long nr_pages; + struct page **pages; + int ret; + + pages = io_pin_pages_alloc(uaddr, len, &nr_pages); + if (IS_ERR(pages)) + return pages; + ret = pin_user_pages_fast(uaddr, nr_pages, FOLL_WRITE | FOLL_LONGTERM, pages); /* success, mapped all pages */ @@ -79,6 +93,54 @@ struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) return ERR_PTR(ret); } +struct page **io_pin_pages_fast_path(unsigned long uaddr, unsigned long len, int *npages) +{ + unsigned long nr_pages; + struct page **pages; + int ret; + + pages = io_pin_pages_alloc(uaddr, len, &nr_pages); + if (IS_ERR(pages)) + return pages; + + ret = pin_user_pages_fast(uaddr, nr_pages, FOLL_WRITE | FOLL_LONGTERM, + pages); + /* success, mapped all pages */ + if (ret == nr_pages) { + struct folio *folio = page_folio(pages[0]); + + if (nr_pages > 1 && folio_test_hugetlb(folio) && + page_folio(pages[nr_pages - 1]) == folio) { + struct page **huge_pages; + + huge_pages = kvmalloc_objs(struct page *, 1, GFP_KERNEL_ACCOUNT); + if (!huge_pages) { + *npages = nr_pages; + return pages; + } + unpin_user_folio(folio, nr_pages - 1); + + huge_pages[0] = pages[0]; + kvfree(pages); + pages = huge_pages; + *npages = 1; + } else { + *npages = nr_pages; + } + return pages; + } + + /* partial map, or didn't map anything */ + if (ret >= 0) { + /* if we did partial map, release any pages we did get */ + if (ret) + unpin_user_pages(pages, ret); + ret = -EFAULT; + } + kvfree(pages); + return ERR_PTR(ret); +} + enum { /* memory was vmap'ed for the kernel, freeing the region vunmap's it */ IO_REGION_F_VMAP = 1, diff --git a/io_uring/memmap.h b/io_uring/memmap.h index f4cfbb6b9a1f..cc41af3fae61 100644 --- a/io_uring/memmap.h +++ b/io_uring/memmap.h @@ -7,7 +7,10 @@ #define IORING_OFF_ZCRX_SHIFT 16 +struct page **io_pin_pages_alloc(unsigned long uaddr, unsigned long len, + unsigned long *nr_pages_out); struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages); +struct page **io_pin_pages_fast_path(unsigned long uaddr, unsigned long len, int *npages); #ifndef CONFIG_MMU unsigned int io_uring_nommu_mmap_capabilities(struct file *file); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 650303626be6..e117b10bef0b 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -771,7 +771,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_rsrc_node *node; unsigned long off; size_t size; - int ret, nr_pages, i; + int ret, nr_pages, i, orig_nr_pages; struct io_imu_folio_data data; bool coalesced = false; @@ -792,7 +792,10 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, return ERR_PTR(-ENOMEM); ret = -ENOMEM; - pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, + orig_nr_pages = ((unsigned long)iov->iov_base + iov->iov_len + + PAGE_SIZE - 1) >> PAGE_SHIFT; + orig_nr_pages -= (unsigned long)iov->iov_base >> PAGE_SHIFT; + pages = io_pin_pages_fast_path((unsigned long) iov->iov_base, iov->iov_len, &nr_pages); if (IS_ERR(pages)) { ret = PTR_ERR(pages); @@ -826,6 +829,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, imu->dir = IO_IMU_DEST | IO_IMU_SOURCE; if (coalesced) imu->folio_shift = data.folio_shift; + else if (nr_pages == 1 && orig_nr_pages > 1) + imu->folio_shift = folio_shift(page_folio(pages[0])); refcount_set(&imu->refs, 1); off = (unsigned long)iov->iov_base & ~PAGE_MASK; -- 2.39.5