mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Yunsheng Lin <linyunsheng@huawei.com>
To: <davem@davemloft.net>, <kuba@kernel.org>, <pabeni@redhat.com>
Cc: <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
	Yunsheng Lin <linyunsheng@huawei.com>,
	Lorenzo Bianconi <lorenzo@kernel.org>,
	Alexander Duyck <alexander.duyck@gmail.com>,
	Jesper Dangaard Brouer <hawk@kernel.org>,
	Ilias Apalodimas <ilias.apalodimas@linaro.org>,
	Eric Dumazet <edumazet@google.com>
Subject: [PATCH net-next v4 2/5] page_pool: unify frag_count handling in page_pool_is_last_frag()
Date: Mon, 12 Jun 2023 21:02:53 +0800	[thread overview]
Message-ID: <20230612130256.4572-3-linyunsheng@huawei.com> (raw)
In-Reply-To: <20230612130256.4572-1-linyunsheng@huawei.com>

Currently when page_pool_create() is called with
PP_FLAG_PAGE_FRAG flag, page_pool_alloc_pages() is only
allowed to be called under the below constraints:
1. page_pool_fragment_page() need to be called to setup
   page->pp_frag_count immediately.
2. page_pool_defrag_page() often need to be called to drain
   the page->pp_frag_count when there is no more user will
   be holding on to that page.

Those constraints exist in order to support a page to be
split into multi frags.

And those constraints have some overhead because of the
cache line dirtying/bouncing and atomic update.

Those constraints are unavoidable for case when we need a
page to be split into more than one frag, but there is also
case that we want to avoid the above constraints and their
overhead when a page can't be split as it can only hold a big
frag as requested by user, depending on different use cases:
use case 1: allocate page without page splitting.
use case 2: allocate page with page splitting.
use case 3: allocate page with or without page splitting
            depending on the frag size.

Currently page pool only provide page_pool_alloc_pages() and
page_pool_alloc_frag() API to enable the 1 & 2 separately,
so we can not use a combination of 1 & 2 to enable 3, it is
not possible yet because of the per page_pool flag
PP_FLAG_PAGE_FRAG.

So in order to allow allocating unsplit page without the
overhead of split page while still allow allocating split
page we need to remove the per page_pool flag in
page_pool_is_last_frag(), as best as I can think of, it seems
there are two methods as below:
1. Add per page flag/bit to indicate a page is split or
   not, which means we might need to update that flag/bit
   everytime the page is recycled, dirtying the cache line
   of 'struct page' for use case 1.
2. Unify the page->pp_frag_count handling for both split and
   unsplit page by assuming all pages in the page pool is split
   into a big frag initially.

As page pool already supports use case 1 without dirtying the
cache line of 'struct page' whenever a page is recyclable, we
need to support the above use case 3 with minimal overhead,
especially not adding any noticeable overhead for use case 1,
and we are already doing an optimization by not updating
pp_frag_count in page_pool_defrag_page() for the last frag
user, this patch chooses to unify the pp_frag_count handling
to support the above use case 3.

Signed-off-by: Yunsheng Lin <linyunsheng@huawei.com>
CC: Lorenzo Bianconi <lorenzo@kernel.org>
CC: Alexander Duyck <alexander.duyck@gmail.com>
---
 include/net/page_pool.h | 49 ++++++++++++++++++++++++++++++-----------
 net/core/page_pool.c    |  8 +++++++
 2 files changed, 44 insertions(+), 13 deletions(-)

diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index 5c7f7501f300..0b8cd2acc1d7 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -324,7 +324,8 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page,
  */
 static inline void page_pool_fragment_page(struct page *page, long nr)
 {
-	atomic_long_set(&page->pp_frag_count, nr);
+	if (!PAGE_POOL_DMA_USE_PP_FRAG_COUNT)
+		atomic_long_set(&page->pp_frag_count, nr);
 }
 
 static inline long page_pool_defrag_page(struct page *page, long nr)
@@ -332,19 +333,43 @@ static inline long page_pool_defrag_page(struct page *page, long nr)
 	long ret;
 
 	/* If nr == pp_frag_count then we have cleared all remaining
-	 * references to the page. No need to actually overwrite it, instead
-	 * we can leave this to be overwritten by the calling function.
+	 * references to the page:
+	 * 1. 'n == 1': no need to actually overwrite it.
+	 * 2. 'n != 1': overwrite it with one, which is the rare case
+	 *              for frag draining.
 	 *
-	 * The main advantage to doing this is that an atomic_read is
-	 * generally a much cheaper operation than an atomic update,
-	 * especially when dealing with a page that may be partitioned
-	 * into only 2 or 3 pieces.
+	 * The main advantage to doing this is that not only we avoid a
+	 * atomic update, as an atomic_read is generally a much cheaper
+	 * operation than an atomic update, especially when dealing with
+	 * a page that may be partitioned into only 2 or 3 pieces; but
+	 * also unify the frag and non-frag handling by ensuring all
+	 * pages have been split into one big frag initially, and only
+	 * overwrite it when the page is split into more than one frag.
 	 */
-	if (atomic_long_read(&page->pp_frag_count) == nr)
+	if (atomic_long_read(&page->pp_frag_count) == nr) {
+		/* As we have ensured nr is always one for constant case
+		 * using the BUILD_BUG_ON(), only need to handle the
+		 * non-constant case here for frag count draining, which
+		 * is a rare case.
+		 */
+		BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1);
+		if (!__builtin_constant_p(nr))
+			atomic_long_set(&page->pp_frag_count, 1);
+
 		return 0;
+	}
 
 	ret = atomic_long_sub_return(nr, &page->pp_frag_count);
 	WARN_ON(ret < 0);
+
+	/* We are the last user here too, reset frag count back to 1 to
+	 * ensure all pages have been split into one big frag initially,
+	 * this should be the rare case when the last two frag users call
+	 * page_pool_defrag_page() currently.
+	 */
+	if (unlikely(!ret))
+		atomic_long_set(&page->pp_frag_count, 1);
+
 	return ret;
 }
 
@@ -353,12 +378,10 @@ static inline bool page_pool_is_last_frag(struct page_pool *pool,
 {
 	/* We assume we are the last frag user that is still holding
 	 * on to the page if:
-	 * 1. Fragments aren't enabled.
-	 * 2. We are running in 32-bit arch with 64-bit DMA.
-	 * 3. page_pool_defrag_page() indicate we are the last user.
+	 * 1. We are running in 32-bit arch with 64-bit DMA.
+	 * 2. page_pool_defrag_page() indicate we are the last user.
 	 */
-	return !(pool->p.flags & PP_FLAG_PAGE_FRAG) ||
-	       PAGE_POOL_DMA_USE_PP_FRAG_COUNT ||
+	return PAGE_POOL_DMA_USE_PP_FRAG_COUNT ||
 	       (page_pool_defrag_page(page, 1) == 0);
 }
 
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 9c4118c62997..69e3c5175236 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -352,6 +352,14 @@ static void page_pool_set_pp_info(struct page_pool *pool,
 {
 	page->pp = pool;
 	page->pp_magic |= PP_SIGNATURE;
+
+	/* Ensuring all pages have been split into one big frag initially:
+	 * page_pool_set_pp_info() is only called once for every page when it
+	 * is allocated from the page allocator and page_pool_fragment_page()
+	 * is dirtying the same cache line as the page->pp_magic above, so
+	 * the overhead is negligible.
+	 */
+	page_pool_fragment_page(page, 1);
 	if (pool->p.init_callback)
 		pool->p.init_callback(page, pool->p.init_arg);
 }
-- 
2.33.0


  parent reply	other threads:[~2023-06-12 13:05 UTC|newest]

Thread overview: 92+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-06-12 13:02 [PATCH net-next v4 0/5] introduce page_pool_alloc() API Yunsheng Lin
2023-06-12 13:02 ` [PATCH net-next v4 1/5] page_pool: frag API support for 32-bit arch with 64-bit DMA Yunsheng Lin
2023-06-13 13:30   ` Alexander Lobakin
2023-06-14  3:36     ` Yunsheng Lin
2023-06-14  3:55       ` Jakub Kicinski
2023-06-14 10:52       ` Alexander Lobakin
2023-06-14 12:15         ` Yunsheng Lin
2023-06-14 12:42           ` Alexander Lobakin
2023-06-14  4:09   ` Jakub Kicinski
2023-06-14 11:42     ` Yunsheng Lin
2023-06-14 17:07       ` Jakub Kicinski
2023-06-15  7:29         ` Yunsheng Lin
2023-06-12 13:02 ` Yunsheng Lin [this message]
2023-06-14  4:33   ` [PATCH net-next v4 2/5] page_pool: unify frag_count handling in page_pool_is_last_frag() Jakub Kicinski
2023-06-14 11:55     ` Yunsheng Lin
2023-06-12 13:02 ` [PATCH net-next v4 3/5] page_pool: introduce page_pool_alloc() API Yunsheng Lin
2023-06-13 13:08   ` Alexander Lobakin
2023-06-13 13:11     ` Alexander Lobakin
2023-06-14  3:17       ` Yunsheng Lin
2023-06-12 13:02 ` [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag Yunsheng Lin
2023-06-14 17:19   ` Jakub Kicinski
2023-06-15  7:17     ` Yunsheng Lin
2023-06-15 16:51       ` Jakub Kicinski
2023-06-15 18:26         ` Alexander Duyck
2023-06-16 12:20           ` Yunsheng Lin
2023-06-16 15:01             ` Alexander Duyck
2023-06-16 18:59               ` Jesper Dangaard Brouer
2023-06-16 19:21                 ` Jakub Kicinski
2023-06-16 20:42                   ` Memory providers multiplexing (Was: [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag) Jesper Dangaard Brouer
2023-06-19 18:07                     ` Jakub Kicinski
2023-06-20 15:12                       ` Jesper Dangaard Brouer
2023-06-20 15:39                         ` Jakub Kicinski
2023-06-30  2:27                       ` Mina Almasry
2023-07-03  4:20                         ` David Ahern
2023-07-03  6:22                           ` Mina Almasry
2023-07-03 14:45                             ` David Ahern
2023-07-03 17:13                               ` Eric Dumazet
2023-07-03 17:23                                 ` David Ahern
2023-07-06  1:19                                   ` Mina Almasry
2023-07-03 17:15                               ` Eric Dumazet
2023-07-03 17:25                                 ` David Ahern
2023-07-03 21:43                             ` Jason Gunthorpe
2023-07-06  1:17                               ` Mina Almasry
2023-07-10 17:44                                 ` Jason Gunthorpe
2023-07-10 23:02                                   ` Mina Almasry
2023-07-10 23:49                                     ` Jason Gunthorpe
2023-07-11  0:45                                       ` Mina Almasry
2023-07-11 13:11                                         ` Jason Gunthorpe
2023-07-11 17:24                                           ` Mina Almasry
2023-07-11  4:27                                       ` Christoph Hellwig
2023-07-11  4:59                                         ` Jakub Kicinski
2023-07-11  5:04                                           ` Christoph Hellwig
2023-07-11 12:05                                             ` Jason Gunthorpe
2023-07-11 16:00                                               ` Jakub Kicinski
2023-07-11 16:20                                                 ` David Ahern
2023-07-11 16:32                                                   ` Jakub Kicinski
2023-07-11 17:06                                                     ` Mina Almasry
2023-07-11 20:39                                                       ` Jakub Kicinski
2023-07-11 21:39                                                         ` David Ahern
2023-07-12  3:42                                                           ` Mina Almasry
2023-07-12  7:55                                                             ` Christian König
2023-07-12 13:03                                                               ` Jason Gunthorpe
2023-07-12 13:35                                                                 ` Christian König
2023-07-12 22:41                                                                   ` Mina Almasry
2023-07-12 13:01                                                             ` Jason Gunthorpe
2023-07-12 20:16                                                               ` Mina Almasry
2023-07-12 23:57                                                                 ` Jason Gunthorpe
2023-07-13  7:56                                                                 ` Christian König
2023-07-14 14:55                                                                   ` Mina Almasry
2023-07-14 15:18                                                                     ` David Ahern
2023-07-17  2:05                                                                       ` Mina Almasry
2023-07-17  3:08                                                                         ` David Ahern
2023-07-14 15:55                                                                     ` Jason Gunthorpe
2023-07-17  1:53                                                                       ` Mina Almasry
2023-07-24 14:56                                                                         ` Jesper Dangaard Brouer
2023-07-24 16:28                                                                           ` Jason Gunthorpe
2023-07-25  4:04                                                                           ` Mina Almasry
2023-07-26 17:36                                                                             ` Jesper Dangaard Brouer
2023-07-11 16:42                                                 ` Jason Gunthorpe
2023-07-11 17:06                                                   ` Jakub Kicinski
2023-07-11 18:52                                                     ` Jason Gunthorpe
2023-07-11 20:34                                                       ` Jakub Kicinski
2023-07-11 23:56                                                         ` Jason Gunthorpe
2023-07-11  6:52                                       ` Dan Williams
2023-07-06 16:50                         ` Jakub Kicinski
2023-06-17 12:19               ` [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag Yunsheng Lin
2023-06-15 13:59     ` Alexander Lobakin
2023-06-12 13:02 ` [PATCH net-next v4 5/5] page_pool: update document about frag API Yunsheng Lin
2023-06-14  4:40   ` Jakub Kicinski
2023-06-14 12:04     ` Yunsheng Lin
2023-06-14 16:56       ` Jakub Kicinski
2023-06-15  6:49         ` Yunsheng Lin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230612130256.4572-3-linyunsheng@huawei.com \
    --to=linyunsheng@huawei.com \
    --cc=alexander.duyck@gmail.com \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=hawk@kernel.org \
    --cc=ilias.apalodimas@linaro.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lorenzo@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®