From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.200 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793740472; b=iov3M/MNIEOVfTP9jLc6slp/FJKlii6sQZ5pBCBkqkvGUfdap4C7hoHgqckKMp9EXU3Y SSa+WBqrIKGwv4uS6pN+NPDAqMAavKbrVKu8sPzaRlBu9zw8ryHbJZ1/S4KQ/euEEQt8n zL5jqaVS2oADYfMA23wXSz4KEZf7+G13Lk3fEZDZ//gcdcsVz4AkKqVpsDeYatnVhMQyk j4qtZJ/mXs/KBd+BzpW4rDLFg8NP1FTgTSS2TmG3+a9E/EQkLdK+Otw1f/bypbNooiugR VtxeyNfPg+FklA0b11FovRAfjFgoofMcY4SFm7qJYwGH2sUNfucab1EQIINM7lWxC6w== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793740472; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=CmNaC/1fJXFtVCoGgULUJOrf6F4b0TSeiMbURfAjOmOg8llhdB76ZM766zPa+GJQ2bLN vEmTxlQ47eibxfV0QbqvEdaD2fBaHEVYflEL/5GFgV9RiPN90IDzwwz7XwUjxAJfa82wX s45jMFsyZlmGk1aIWBTp/tcsa22eY+rsXbw3sB/L8VWnWINxDC3IKE5rljv2mdvriTuck KY6Id/QTXQ3fM4bZyhIL1vlpVLBU1lXgLgEsbffSxYWGZWmvZjI2yjSAeBOyLhYSvZsj2 yI4ps3LXKaczFXnQ3kCM/7gTxkoSAVbYiTBi0xpVWh9izldiuGrL71feOZc0zpCHNNQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=J0GTklSJ; arc=none smtp.remote-ip=74.125.82.200 Received: from mail-dy1-f200.google.com (mail-dy1-f200.google.com [74.125.82.200]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B01B606EE8B for ; Tue, 3 Nov 2026 21:14:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 8B01B606EE8B Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=J0GTklSJ Received: by mail-dy1-f200.google.com with SMTP id 5a478bee46e88-30bcb065bfdso8206742eec.0 for ; Tue, 03 Nov 2026 13:14:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1793740471; x=1794345271; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=J0GTklSJA0yPK2BwCTAjrh9eqi5L3NgNqWDz3qWwHvFAKjuw260KypyEmWMZHb8zpQ 4g53u+aqrx7AhHkDvpATs7i2/vmEWInSfCx9tDXP8qTyqK6I+V4keW/5NkamP7Vf+maW y27HLk7d4vqhxNompL8ALUuNpOyDgHI8EGkgu6WK/R/fin39qMbZc6ODbIqS8utfqPoQ cCBJMVtE0c/FYOuCjqhHjAyzFs1P51xbWZYILfE9CzZkglJU8u1JpAaov5qOJloEcGvi Wc+RgAJeTXEot7KayINFeiN6WVP2QWc2ouNKaShXUpts1teJX3yrpS8WDvF8BkeVJz9N gudg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793740471; x=1794345271; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=1+gwyLU6DAbWCtUmorewmD/99+SNFZ747gFZ1fjSCG1FENl+daKETHq9phmTrg5XJD +2UtSbIjFzEnDnQQwyG8yeHV6gWUFtxRS3W6X6swAUnb9sA2h/Gw8w91gUAcfl9izv7C fV1HWmCKfpCZIt678mgbDowBS8hQND7OGmeeup1zwpeBvtKlwHSMyjYJzNKGE2sdsEos tqCxJDTiJuBjOCgXSbtzraJkxfGw/tWlxvlFppMclNwJCoOB73A8f+E2UFHv+jDT5ZdV EzRcH+Lp/hPz6ZDySclz3QjotaW2w/b17CVeKe3AIH/ByMkQgbPDlPrkdwl5LYZ7UdtZ SnyQ== X-Forwarded-Encrypted: i=1; AKwUvBxwKRQNw81nH0VZdFpfIWacserUo9pH3JGRf/FGRJLLEVnR9B3DWlopZyNcIny7YHvJfRSFzKTzjvKzdXY=@vger.kernel.org X-Gm-Message-State: APuU4Z/dFN54th4cu1NQvOtaMav3wgXAy1hatyDQAFlAEOEkQWutIAZt d+MD5v65d6GJA2SK7fZcPcvAv5+i7306idVetb63nSnTNeED0IQOkImPXxId6qbqNeiqjqX84QX YKYP56AWMmHTh4yd6shajod7Vog== X-Received: from dyvj18-n1.prod.google.com ([2002:a05:693c:87d2:10b0:368:2c6d:de10]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:693c:8959:20b0:369:f5f6:36a7 with SMTP id 5a478bee46e88-36a423d1cc5mr22095354eec.16.1793740470274; Tue, 03 Nov 2026 13:14:30 -0800 (PST) Date: Tue, 3 Nov 2026 21:14:22 +0000 In-Reply-To: <20261103211424.127627-1-almasrymina@google.com> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261103211424.127627-1-almasrymina@google.com> X-Mailer: git-send-email 2.56.0.385.gac671c819a-goog Message-ID: <20261103211424.127627-4-almasrymina@google.com> Subject: [PATCH net-next v1 3/5] net: datagram: refcount page_pool frags in zerocopy_fill_skb_from_iter() From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, bpf@vger.kernel.org, linux-wireless@vger.kernel.org, linux-doc@vger.kernel.org, bcm-kernel-feedback-list@broadcom.com, imx@lists.linux.dev, linux-rdma@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-omap@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert , Herbert Xu , Willem de Bruijn , Stanislav Fomichev , Pavel Begunkov , Bobby Eshleman , Andrew Lunn , "Michael S. Tsirkin" , Jason Wang , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Xuan Zhuo , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jesper Dangaard Brouer , Ilias Apalodimas , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sabrina Dubroca , Johannes Berg , Byungchul Park , Jonathan Corbet , Shuah Khan , Randy Dunlap , Lorenzo Bianconi , Wen Gu , Philo Lu , Justin Chen , Florian Fainelli , Vikas Gupta , Michael Chan , Pavan Chebbi , Doug Berger , Nicolai Buchwitz , Satish Kharat , Wei Fang , Frank Li , Shenwei Wang , Joshua Washington , Harshitha Ramamurthy , Jijie Shao , Jian Shen , Fan Gong , Tony Nguyen , Przemek Kitszel , Marcin Wojtas , Russell King , Sunil Goutham , Geetha sowjanya , Ratheesh Kannoth , Subbaraya Sundeep , Bharat Bhushan , Felix Fietkau , Matthias Brugger , AngeloGioacchino Del Regno , Tariq Toukan , Saeed Mahameed , Mark Bloch , Leon Romanovsky , Ido Schimmel , Petr Machata , Horatiu Vultur , UNGLinuxDriver@microchip.com, Daniel Machon , Steen Hegelund , Haiyang Zhang , Wei Liu , Dexuan Cui , Yibo Dong , Brett Creeley , Justin Lai , Larry Chiu , "=?UTF-8?q?Niklas=20S=C3=B6derlund?=" , Paul Barker , Masahisa Kojima , Maxime Chevallier , Maxime Coquelin , Alexandre Torgue , FUJITA Tomonori , Siddharth Vadapalli , Roger Quadros , MD Danish Anwar , Jiawen Wu , Mengyuan Lou , Ronak Doshi , Ryder Lee , Shayne Chen , Sean Wang , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Jordan Rhee , Ankit Garg , Tim Hostetler , Erni Sri Satya Vennela , Eric Joyner , Mohammad Heib , Prabu Thayalan , Chintan Vankar , Nora Schiffer , Alexander Sverdlin , Kevin Hao , Karl Mehltretter , Aleksandr Loktionov , Meghana Malladi , Jacob Keller , StanleyYP Wang Content-Type: text/plain; charset="UTF-8" When an application maps order-0 page_pool RX fragments into user space via TCP_ZEROCOPY_RECEIVE and forwards that mapped address into TX sendmsg(MSG_ZEROCOPY), zerocopy_fill_skb_from_iter() pins the pages via iov_iter_get_pages2() (which increments page->_refcount) while skb_release_data() releases fragments via netmem_dec_ref(). Acquire each fragment reference via netmem_inc_ref() when attaching a fragment in zerocopy_fill_skb_from_iter() and release the corresponding GUP page->_refcount reference via the existing batched page_ref_sub() path so the TX SKB fragment reference pairs with netmem_dec_ref() in skb_release_data() once netmem_inc_ref() and netmem_dec_ref() handle page_pool netmems. Signed-off-by: Mina Almasry --- net/core/datagram.c | 5 ++ net/core/net_test.c | 146 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+) diff --git a/net/core/datagram.c b/net/core/datagram.c index ecc30a6f83023..f2d6d1d7be2b6 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -34,6 +34,7 @@ * */ +#include #include #include #include @@ -696,12 +697,16 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, refs = 0; } skb_fill_page_desc_noacc(skb, frag++, head, start, size); + netmem_inc_ref(page_to_netmem(head)); + last_head = head; + refs++; } if (refs) page_ref_sub(last_head, refs); } return 0; } +EXPORT_SYMBOL_IF_KUNIT(zerocopy_fill_skb_from_iter); static int zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, diff --git a/net/core/net_test.c b/net/core/net_test.c index e20059a00003c..73da75e6ca2c8 100644 --- a/net/core/net_test.c +++ b/net/core/net_test.c @@ -515,7 +515,9 @@ static void ip_tunnel_flags_test_run(struct kunit *test) /* Netmem & SKB fragment refcounting */ +#include #include +#include #include #include #include @@ -841,6 +843,149 @@ static void netmem_ref_test_pp_page(struct kunit *test) page_pool_put_full_page(pool, page, false); page_pool_destroy(pool); } + +int zerocopy_fill_skb_from_iter(struct sk_buff *skb, + struct iov_iter *from, size_t length); + +static void netmem_ref_test_zerocopy_fill_skb_from_iter(struct kunit *test) +{ + struct page_pool_params pp_params = { + .order = 0, + .pool_size = 4, + .nid = NUMA_NO_NODE, + }; + struct page *pp_page, *comp_page, *reg_page; + struct page_pool *pool, *comp_pool; + int base_ref, base_comp_ref, base_reg_ref; + long base_pp_ref, base_comp_pp_ref; + struct folio_queue folioq; + struct bio_vec bvec[3]; + struct iov_iter iter; + struct sk_buff *skb; + + pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool); + pp_page = page_pool_dev_alloc_pages(pool); + KUNIT_ASSERT_NOT_NULL(test, pp_page); + base_ref = page_ref_count(pp_page); + base_pp_ref = atomic_long_read(&pp_page->pp_ref_count); + + /* 1. Single order-0 page_pool page (simulating TCP_ZEROCOPY_RECEIVE + * page forwarded into MSG_ZEROCOPY TX via zerocopy_fill_skb_from_iter). + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 128, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 1, 128); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 128), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 2. Contiguous segments on the same page_pool page coalesce into a + * single skb_frag_t while a non-contiguous segment creates a second + * skb_frag_t; all 3 GUP _refcounts are dropped. + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 64, 0); + bvec_set_page(&bvec[1], pp_page, 64, 64); + bvec_set_page(&bvec[2], pp_page, 64, 256); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 3, 192); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 192), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), 128U); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[1]), 64U); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 2); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 3. Order-1 compound page_pool page spanning 2 subpages in a single + * iov_iter_get_pages2() batch coalesces into 1 skb_frag_t and drops + * both GUP _refcounts via batched page_ref_sub, alongside a + * non-page_pool page in the same skb. + */ + pp_params.order = 1; + comp_pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, comp_pool); + comp_page = page_pool_dev_alloc_pages(comp_pool); + KUNIT_ASSERT_NOT_NULL(test, comp_page); + base_comp_ref = page_ref_count(comp_page); + base_comp_pp_ref = atomic_long_read(&comp_page->pp_ref_count); + + reg_page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, reg_page); + base_reg_ref = page_ref_count(reg_page); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], comp_page, 2 * PAGE_SIZE, 0); + bvec_set_page(&bvec[1], reg_page, 64, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 2, 2 * PAGE_SIZE + 64); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 2 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), + (unsigned int)(2 * PAGE_SIZE)); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + + /* 4. Single-batch mixed folios via ITER_FOLIOQ: order-1 compound + * page_pool folio (2 subpages) + non-page_pool folio + order-0 + * page_pool folio returned in a single iov_iter_get_pages2() call. + */ + folioq_init(&folioq, 0); + folioq_append(&folioq, page_folio(comp_page)); + folioq_append(&folioq, page_folio(reg_page)); + folioq_append(&folioq, page_folio(pp_page)); + iov_iter_folio_queue(&iter, ITER_SOURCE, &folioq, 0, 0, + 3 * PAGE_SIZE + 64); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 3 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 3); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + __free_page(reg_page); + page_pool_put_full_page(comp_pool, comp_page, false); + page_pool_destroy(comp_pool); + page_pool_put_full_page(pool, pp_page, false); + page_pool_destroy(pool); +} #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) @@ -1738,6 +1883,7 @@ static struct kunit_case net_test_cases[] = { KUNIT_CASE(netmem_ref_test_page), #ifdef CONFIG_PAGE_POOL KUNIT_CASE(netmem_ref_test_pp_page), + KUNIT_CASE(netmem_ref_test_zerocopy_fill_skb_from_iter), #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) KUNIT_CASE(netmem_ref_test_net_iov), -- 2.56.0.385.gac671c819a-goog