From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.198 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794117623; b=RIiOuuVOtjUZA90zMHl2HWzqwBXiwBQtCUPj0NDSM8VzI0PQ1LBX18dHjY/XF+ndRmDW IUsOGZHKUkXGPCwi1K5w6pQxltgWt1ajybj77xFdcbJDUbIuJAyQpYgwnVYX2QdpWHhm/ /uu/Hm5cJvPRXlmlrFHps4eCulQnxyleMbVshHrVN/8iSvfJ8lzFXOMdCt8ro2B9MaDZg M++yj6seV3NUDAIwWfqmzwtRa11TtclXsjbwmanN3z5WJe4tLaoyglZi3270QXV7ZhQeA bRRqmcnaazpbovS5cvHqFw7jrS7d672WwnA+oRHY5wg5GPGufkdo8MYMCyhk+t3Riww== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794117623; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=ls8Vsqz37bk9Lc/TUQk6BA8al47BcGIFX0n6D6YIHe4=; b=3o3beih7jICGkLdfqxly4W2tVLfMUx89cGWMj+tQXIIPysZ9XGVJ7gSEdmeyWWRbrSpO GLTC+e4R2zP/0REFEBQZkmxi90k+typLzO5YgmZ85T0EXvxSn1GOlEi+1qt9t5xR8CWOI RmnIdq4IZQnbuAm+IbFPGQhMMSiNHOh7X3efsYP0Q3dlHaUZAjWoOmQ5nZWpQeMoUp8ma Al2luarFQuOXzeiOE/jvn7JxLeBTlwy5MZj6ISEUEm8WCML0eUjHiB4MVp/hQCqm54yNf aGttXrz8p5WtYGLVzUC9EbJapyyhAP7SqD2S8U1H5teLNWwPdhhPJp0RzhKnIitZA9g== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=YpdVdLqn; arc=none smtp.remote-ip=74.125.82.198 Received: from mail-dy1-f198.google.com (mail-dy1-f198.google.com [74.125.82.198]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 528F9606EE82 for ; Sun, 8 Nov 2026 06:00:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 528F9606EE82 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=YpdVdLqn Received: by mail-dy1-f198.google.com with SMTP id 5a478bee46e88-33c35f5ca6cso651940eec.1 for ; Sat, 07 Nov 2026 22:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1794117622; x=1794722422; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ls8Vsqz37bk9Lc/TUQk6BA8al47BcGIFX0n6D6YIHe4=; b=YpdVdLqnbIFVZl1j3c/pO8uJGsznuNzYMG+4HBi6dyoxJhJV9D0u/I14u/Q4/SNfK1 a3quyC7hVQkEKMy7b6LdLxTu9On3TL8sr60HumlFrDPh5qbVRBsXihEO4qRq3w0Vmm0H rnJUIPMYeUTjfaTvNQAmObrSCvl3XilojYbmXBhbOit73pDdTZXZfJ/lJEyEew3aLqC0 xWRQNcZyhz5fKknGQckegaWciy/sNTCBxlHyqkO3om8Gon0ufW7n1nRztqC9CbQqkhSR t1X3m5N+FHhd8U8iFZuJweStgNR95R8b/AWdTBmmSMd+20PIDIHoDngYJzzspPp7rz5f /DMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794117622; x=1794722422; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ls8Vsqz37bk9Lc/TUQk6BA8al47BcGIFX0n6D6YIHe4=; b=u4A7fEb0XgNAVgNj6KeJml193KEVpyxm+gTI8BV67gLkROX1ZKBdCsOJrpPkCyRdOP dBAQNkr4epEOe7JRwy2g6JW1dr3W2IIFNjqT5gxTeaIAXeoZgwNfRF/uNV9+wCJB/pxM RXxH9tf1lH02ORTD7G/ZST0AvdWHzRRxBBhRkcQ/TcXB5xTkBrvxZW+gB3xqu1K/A7TM zj+RlY5wiCBlkAZODbWCxxwJkqCxLmbB6yjsf+XOLHMilhuSsTKUX2AJK8NpCF7aBiNL G8wOlL7HzI5C5r6hFpsgwpX+0oliOZITA/S5k9V26OlWW0/DxiJc+KYh1zFsd/f6TYww 7xWg== X-Forwarded-Encrypted: i=1; AKwUvByWfv7luXSxPOeiN32bBKT5cSY0aidLoRybGrFi/hWjRf7+ePk72idBM+/S2NUI389sZQYNs7cg9QA7NN4=@vger.kernel.org X-Gm-Message-State: APuU4Z/6ewamiu7Dk+gYqshxMb5a3lGHNJAwMtgvRM/qy/8RKimRyWhi 0rgIZYi5CeObHnjDChQhM8kQZjLGMy+sEbkwdb6YLsFUuSDhUxbV3TTWC/6tGEZLt+F+jLn7wCz NfDe3bzKXVTlq4EVhzEQrOC/3kw== X-Received: from dycmb13-n2.prod.google.com ([2002:a05:693c:20cd:20b0:36a:17c6:6824]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:693c:6390:20b0:36c:9bc6:3ec with SMTP id 5a478bee46e88-37252f3042emr1787652eec.3.1794117622089; Sat, 07 Nov 2026 22:00:22 -0800 (PST) Date: Sun, 8 Nov 2026 06:00:14 +0000 In-Reply-To: <20261108060016.2131509-1-almasrymina@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261108060016.2131509-1-almasrymina@google.com> X-Mailer: git-send-email 2.56.0.455.g6c5a503b38-goog Message-ID: <20261108060016.2131509-4-almasrymina@google.com> Subject: [PATCH net-next v2 3/5] tcp: detach page_pool pages before mapping in tcp_zerocopy_receive() From: Mina Almasry To: netdev@vger.kernel.org Cc: Mina Almasry , Jesper Dangaard Brouer , Ilias Apalodimas , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Neal Cardwell , Kuniyuki Iwashima , Pavel Begunkov , Bobby Eshleman , Stanislav Fomichev , linux-kernel@vger.kernel.org, Steffen Klassert , Herbert Xu , Willem de Bruijn , John Fastabend , Sabrina Dubroca , Andrew Lunn , "Michael S. Tsirkin" , Jason Wang , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Xuan Zhuo , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Alexei Starovoitov , Daniel Borkmann , Johannes Berg , Byungchul Park , Zi Yan , virtualization@lists.linux.dev, linux-mm@kvack.org, bpf@vger.kernel.org, linux-wireless@vger.kernel.org, Jonathan Corbet , Shuah Khan , Randy Dunlap , Lorenzo Bianconi , Wen Gu , Philo Lu , Justin Chen , Florian Fainelli , Vikas Gupta , Michael Chan , Pavan Chebbi , Doug Berger , Nicolai Buchwitz , Broadcom internal kernel review list , Satish Kharat , Wei Fang , Frank Li , Shenwei Wang , Joshua Washington , Harshitha Ramamurthy , Jijie Shao , Jian Shen , Fan Gong , Tony Nguyen , Przemek Kitszel , Marcin Wojtas , Russell King , Sunil Goutham , Geetha sowjanya , Ratheesh Kannoth , Subbaraya Sundeep , Bharat Bhushan , Felix Fietkau , Matthias Brugger , AngeloGioacchino Del Regno , Tariq Toukan , Saeed Mahameed , Mark Bloch , Leon Romanovsky , Ido Schimmel , Petr Machata , Horatiu Vultur , UNGLinuxDriver@microchip.com, Daniel Machon , Steen Hegelund , Haiyang Zhang , Wei Liu , Dexuan Cui , Yibo Dong , Brett Creeley , Justin Lai , Larry Chiu , "=?UTF-8?q?Niklas=20S=C3=B6derlund?=" , Paul Barker , Masahisa Kojima , Maxime Chevallier , Maxime Coquelin , Alexandre Torgue , FUJITA Tomonori , Siddharth Vadapalli , Roger Quadros , MD Danish Anwar , Jiawen Wu , Mengyuan Lou , Ronak Doshi , Ryder Lee , Shayne Chen , Sean Wang , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Jordan Rhee , Ankit Garg , Tim Hostetler , Eric Joyner , Prabu Thayalan , "Nikhil P. Rao" , Chintan Vankar , Nora Schiffer , Alexander Sverdlin , Kevin Hao , Karl Mehltretter , Aleksandr Loktionov , Meghana Malladi , Jacob Keller , StanleyYP Wang , linux-doc@vger.kernel.org, imx@lists.linux.dev, linux-rdma@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-omap@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Once tcp_zerocopy_receive() maps an order-0 RX fragment page into a userspace VMA via vm_insert_pages(), vm_insert_pages() increments page->_refcount, which prevents the page from being recycled back into its page_pool when the RX SKB is freed. However, while the RX SKB still holds its fragment reference, the page remains marked as a page_pool page (page_pool_page_is_pp(page) == true) inside the userspace VMA, so subsequent GUP callers such as sendmsg(MSG_ZEROCOPY) or vmsplice() can encounter a page_pool page while holding only a raw struct page reference. Establish the invariant that pp_ref_count is only incremented by callers that already hold an active pp_ref_count (>= 1) on that netmem, and a raw page_ref_count is never converted into a pp_ref_count: - In can_map_frag(), require !skb_cloned(skb), page_pool_ref_count == 1, and page_ref_count == 1 for page_pool pages so the fragment being mapped holds the sole reference to the page; cloned SKBs or page_pool pages with additional references (including drivers holding an undrained fragment bias) fall back to copying. - In tcp_zerocopy_receive(), convert the fragment's sole pp_ref_count into a regular page reference and detach the page from its page_pool via get_page(page) + napi_pp_put_page(page_to_netmem(page)) before queueing the page for vm_insert_pages(). If vm_insert_pages() subsequently fails, the detached page remains referenced by the RX SKB via its regular page reference and is freed back to the page allocator when the SKB is consumed. Signed-off-by: Mina Almasry --- include/net/page_pool/helpers.h | 5 ++++ net/core/datagram.c | 2 ++ net/ipv4/tcp.c | 44 ++++++++++++++++++++++++++++----- 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 28635ce9454e8..c814ce95282eb 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -242,6 +242,11 @@ page_pool_get_dma_dir(const struct page_pool *pool) return pool->p.dma_dir; } +static inline long page_pool_ref_count(netmem_ref netmem) +{ + return atomic_long_read(netmem_get_pp_ref_count_ref(netmem)); +} + static inline void page_pool_fragment_netmem(netmem_ref netmem, long nr) { atomic_long_set(netmem_get_pp_ref_count_ref(netmem), nr); diff --git a/net/core/datagram.c b/net/core/datagram.c index ecc30a6f83023..808eda4f7efb5 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -34,6 +34,7 @@ * */ +#include #include #include #include @@ -702,6 +703,7 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, } return 0; } +EXPORT_SYMBOL_IF_KUNIT(zerocopy_fill_skb_from_iter); static int zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index b9de7c302ba83..a984e7e93e4c9 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -286,6 +286,7 @@ #include #include #include +#include #include #include @@ -1921,7 +1922,7 @@ static skb_frag_t *skb_advance_to_frag(struct sk_buff *skb, u32 offset_skb, return frag; } -static bool can_map_frag(const skb_frag_t *frag) +static bool can_map_frag(const struct sk_buff *skb, const skb_frag_t *frag) { struct page *page; @@ -1935,18 +1936,32 @@ static bool can_map_frag(const skb_frag_t *frag) if (PageCompound(page) || page->mapping) return false; + /* Before mapping a page_pool page into userspace, + * tcp_zerocopy_receive() converts this fragment's pp_ref_count into a + * regular page reference and detaches the page from its page_pool via + * napi_pp_put_page(). Require this skb_frag_t to hold the sole + * reference to the page so no other fragment or SKB clone expects + * pp_ref_count on it after detachment. + */ + if (page_pool_page_is_pp(page) && + (skb_cloned(skb) || + page_pool_ref_count(page_to_netmem(page)) != 1 || + page_ref_count(page) != 1)) + return false; + return true; } -static int find_next_mappable_frag(const skb_frag_t *frag, +static int find_next_mappable_frag(const struct sk_buff *skb, + const skb_frag_t *frag, int remaining_in_skb) { int offset = 0; - if (likely(can_map_frag(frag))) + if (likely(can_map_frag(skb, frag))) return 0; - while (offset < remaining_in_skb && !can_map_frag(frag)) { + while (offset < remaining_in_skb && !can_map_frag(skb, frag)) { offset += skb_frag_size(frag); ++frag; } @@ -1986,7 +2001,8 @@ static void tcp_zerocopy_set_hint_for_skb(struct sock *sk, * mappable_offset: Bytes till next mappable frag, *not* counting bytes * in partial_frag_remainder. */ - mappable_offset = find_next_mappable_frag(frag, zc->recv_skip_hint); + mappable_offset = find_next_mappable_frag(skb, frag, + zc->recv_skip_hint); zc->recv_skip_hint = mappable_offset + partial_frag_remainder; } @@ -2290,7 +2306,7 @@ static int tcp_zerocopy_receive(struct sock *sk, break; } - mappable_offset = find_next_mappable_frag(frags, + mappable_offset = find_next_mappable_frag(skb, frags, zc->recv_skip_hint); if (mappable_offset) { zc->recv_skip_hint = mappable_offset; @@ -2300,6 +2316,22 @@ static int tcp_zerocopy_receive(struct sock *sk, if (WARN_ON_ONCE(!page)) break; + /* Convert this fragment's sole pp_ref_count into a regular + * page reference and detach the page from its page_pool before + * mapping it into userspace. Once vm_insert_pages() elevates + * page_ref_count the page cannot be recycled into the + * page_pool anyway (and if vm_insert_pages() fails, the + * detached page is freed to the page allocator when the skb is + * consumed); detaching it up front ensures that a page_pool + * page is never mapped into a userspace VMA (where GUP callers + * such as MSG_ZEROCOPY or vmsplice() would otherwise observe a + * page_pool page while holding only a raw page_ref_count). + */ + if (page_pool_page_is_pp(page)) { + get_page(page); + napi_pp_put_page(page_to_netmem(page)); + } + prefetchw(page); pages[pages_to_map++] = page; length += PAGE_SIZE; -- 2.56.0.455.g6c5a503b38-goog