From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E28F7486412 for ; Wed, 7 Oct 2026 10:47:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791370111; cv=none; b=kUTrh4iif/m8oNl0eJRRpDd29gAK29NsUnOPZ4rrLShbFg/G641d2dE8BgPvhC23LDWQWZuWzG2MmubgrNY3NnaUvmDhDE+LuRSdzGTXOSxQo8MsQgXfe4pWvG2JB5D7BrwcGH25q8/GUoRlIs/erErWGkdGnLwGZLuRajkEP2E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791370111; c=relaxed/simple; bh=VSMCXiryW9bXd0+13NiOnq8uL8z+On7sjzmHDe6GPp0=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=W9qR0wN4cB+WvqIWQgOhVlT7FUCBf1eK9nsx7cyejKXdfWqThAVIPvmHqX+fJU3VrgwQ3gSuloAx+L/DxGB0lnlj9mYZCptlpnC2PK+yGSsWfv1CHo3FO8DoH6rcbZvRKFfeq/yLBQSSp2Z4f+ercC6jevdmJFRzmSJC4SWFPxM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=ZFU0+li8; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="ZFU0+li8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=MIME-Version:Content-Transfer-Encoding:Content-Type:References: In-Reply-To:Date:Cc:To:From:Subject:Message-ID:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID; bh=eHWw+UmZtYdvqxe4hUGh5SBBbAD7g60VuTJC0Vf5JM8=; b=ZFU0+l i8Rtyg1iPkyPedc1z1baeYcAfaxOHKohenBGWw0OQFG6eHvnzFmHoaZyFuyjsXY/GQZiXjjZdSbcx 1bz56Xnn17VBuRB0BJHM5TH6mcFi3z6NMXdrdaAdJmEj0ttHXIB1FbmYgHGrg7qis12GCiUDVm0XY 78yJ7Gwmk0VMss/4DgyCrQ2Fjv+6XFwEFylNsNiYp+3GAvwAEnSc0a/DgYvMVpVMmNgrr8CJxfIGB CMxHfYAQvEU5vukKRlpLFsXK0PmkYxiiGTlDn3SaF71h8QpRXe5MF5Vr5rv507DlBuAh6b4O7g6To MQhODRmKczxDx8USwf0rmuPshdZA==; Received: from [2601:18c:8100:a0e0:2541:b86e:2586:d219] by shelob.surriel.com with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.99.5) (envelope-from ) id 1xEPAz-0000000B6vR-0oEP; Wed, 07 Oct 2026 10:47:09 +0000 Message-ID: <7dd4ec994502233cdc09fe5c4369adc89168d738.camel@surriel.com> Subject: Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range() From: Rik van Riel To: Yuan-Hao Hsu , Andrew Morton , David Hildenbrand Cc: Jason Gunthorpe , John Hubbard , Peter Xu , Lorenzo Stoakes , liam@infradead.org, Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Aristeu Rozanski , Ryan Roberts , Dev Jain , linux-mm@kvack.org, linux-kernel@vger.kernel.org Date: Wed, 07 Oct 2026 06:47:06 -0400 In-Reply-To: <20260917083747.786-1-aa9736195201@gmail.com> References: <20260917083747.786-1-aa9736195201@gmail.com> Autocrypt: addr=riel@surriel.com; prefer-encrypt=mutual; keydata=mQENBFIt3aUBCADCK0LicyCYyMa0E1lodCDUBf6G+6C5UXKG1jEYwQu49cc/gUBTTk33A eo2hjn4JinVaPF3zfZprnKMEGGv4dHvEOCPWiNhlz5RtqH3SKJllq2dpeMS9RqbMvDA36rlJIIo47 Z/nl6IA8MDhSqyqdnTY8z7LnQHqq16jAqwo7Ll9qALXz4yG1ZdSCmo80VPetBZZPw7WMjo+1hByv/ lvdFnLfiQ52tayuuC1r9x2qZ/SYWd2M4p/f5CLmvG9UcnkbYFsKWz8bwOBWKg1PQcaYHLx06sHGdY dIDaeVvkIfMFwAprSo5EFU+aes2VB2ZjugOTbkkW2aPSWTRsBhPHhV6dABEBAAG0HlJpayB2YW4gU mllbCA8cmllbEByZWRoYXQuY29tPokBHwQwAQIACQUCW5LcVgIdIAAKCRDOed6ShMTeg05SB/986o gEgdq4byrtaBQKFg5LWfd8e+h+QzLOg/T8mSS3dJzFXe5JBOfvYg7Bj47xXi9I5sM+I9Lu9+1XVb/ r2rGJrU1DwA09TnmyFtK76bgMF0sBEh1ECILYNQTEIemzNFwOWLZZlEhZFRJsZyX+mtEp/WQIygHV WjwuP69VJw+fPQvLOGn4j8W9QXuvhha7u1QJ7mYx4dLGHrZlHdwDsqpvWsW+3rsIqs1BBe5/Itz9o 6y9gLNtQzwmSDioV8KhF85VmYInslhv5tUtMEppfdTLyX4SUKh8ftNIVmH9mXyRCZclSoa6IMd635 Jq1Pj2/Lp64tOzSvN5Y9zaiCc5FucXtB9SaWsgdmFuIFJpZWwgPHJpZWxAc3VycmllbC5jb20+iQE +BBMBAgAoBQJSLd2lAhsjBQkSzAMABgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRDOed6ShMTe g4PpB/0ZivKYFt0LaB22ssWUrBoeNWCP1NY/lkq2QbPhR3agLB7ZXI97PF2z/5QD9Fuy/FD/jddPx KRTvFCtHcEzTOcFjBmf52uqgt3U40H9GM++0IM0yHusd9EzlaWsbp09vsAV2DwdqS69x9RPbvE/Ne fO5subhocH76okcF/aQiQ+oj2j6LJZGBJBVigOHg+4zyzdDgKM+jp0bvDI51KQ4XfxV593OhvkS3z 3FPx0CE7l62WhWrieHyBblqvkTYgJ6dq4bsYpqxxGJOkQ47WpEUx6onH+rImWmPJbSYGhwBzTo0Mm G1Nb1qGPG+mTrSmJjDRxrwf1zjmYqQreWVSFEt26tBpSaWsgdmFuIFJpZWwgPHJpZWxAZmIuY29tP okBPgQTAQIAKAUCW5LbiAIbIwUJEswDAAYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQznneko TE3oOUEQgAsrGxjTC1bGtZyuvyQPcXclap11Ogib6rQywGYu6/Mnkbd6hbyY3wpdyQii/cas2S44N cQj8HkGv91JLVE24/Wt0gITPCH3rLVJJDGQxprHTVDs1t1RAbsbp0XTksZPCNWDGYIBo2aHDwErhI omYQ0Xluo1WBtH/UmHgirHvclsou1Ks9jyTxiPyUKRfae7GNOFiX99+ZlB27P3t8CjtSO831Ij0Ip QrfooZ21YVlUKw0Wy6Ll8EyefyrEYSh8KTm8dQj4O7xxvdg865TLeLpho5PwDRF+/mR3qi8CdGbkE c4pYZQO8UDXUN4S+pe0aTeTqlYw8rRHWF9TnvtpcNzZw== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.60.2 (3.60.2-1.fc44) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Thu, 2026-09-17 at 16:37 +0800, Yuan-Hao Hsu wrote: > GUP-fast grabs a PTE-mapped large folio one page at a time.=C2=A0 Every > PTE > costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount > atomic, a full barrier and a node stat update for FOLL_PIN), a > gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB > mTHP > pays sixteen of each and a PTE-mapped 2 MB THP pays 512.=C2=A0 The PMD an= d > PUD leaf paths already take the whole range with one > try_grab_folio_fast() call, and the slow path is getting the same > treatment for PTEs in Rik's follow_page_mask() series. >=20 > fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O > buffer in 64 kB mTHP, median of 15 five-second runs: >=20 > =C2=A0 bs=3D1M psync=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 126.2/125.6 GB/s=C2=A0 ->=C2= =A0 220.6 GB/s > =C2=A0 bs=3D1M io_uring, iodepth 16=C2=A0=C2=A0 71.2/71.6 GB/s=C2=A0=C2= =A0 ->=C2=A0 112.1 GB/s > =C2=A0 bs=3D64k io_uring, iodepth 16=C2=A0 50.2/50.7 GB/s=C2=A0=C2=A0 ->= =C2=A0=C2=A0 60.1 GB/s >=20 That's huge! Nice. >=20 > +static unsigned int gup_fast_pte_batch(struct folio *folio, > + struct page *page, pmd_t pmd, pmd_t *pmdp, pte_t > *ptep, > + pte_t pte, unsigned int flags, unsigned int max_nr, > + struct page **pages) > +{ > + unsigned int nr, i; > + > + if (max_nr =3D=3D 1) > + return 1; > + > + /* > + * The reference on @folio keeps folio_nr_pages() stable.=C2=A0 > The PTEs > + * are read without the PTL, hence FPB_LOCKLESS. > + */ > + nr =3D folio_pte_batch_flags(folio, NULL, ptep, &pte, max_nr, > + =C2=A0=C2=A0 FPB_RESPECT_WRITE | > FPB_LOCKLESS); >=20 ... > + if (try_grab_folio(folio, nr - 1, flags)) > + return 1; >=20 Since this is lockless, the PTEs could change under us between when we walk them in folio_pte_batch_flags() and the batched try_grab_folio(). Is that race condition safe, or do we need to re-check afterward that: 1) the ptes still point at the same folio, and/or 2) the folio still spans the entire range, and was not split before we grabbed the refcount? The equivalent changes to GUP do not need that check, because that all happens under the pte_lock: https://lore.kernel.org/all/20260811025157.1632867-1-riel@surriel.com/ --=20 All Rights Reversed.