From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FACFC001B1 for ; Mon, 3 Jul 2023 21:43:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231183AbjGCVnW (ORCPT ); Mon, 3 Jul 2023 17:43:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229517AbjGCVnT (ORCPT ); Mon, 3 Jul 2023 17:43:19 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68A5BE49 for ; Mon, 3 Jul 2023 14:43:18 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1b8054180acso39588755ad.1 for ; Mon, 03 Jul 2023 14:43:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1688420598; x=1691012598; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=zb/tmr/noeY2rNeZVS/rd5RWOm9XHi+842eE/ic0+a0=; b=AiAmewfofTMCDjim2n5Tw+1ZtqlOIk0bXi6iRE5v5TlHNIye7/DkwQc6W3qq8LjOIp Xqa4BLwykqMBs3rTWLQCqB46cY7yk3e1UlXmD3dachyhxKT3HwGyQ5ZQUBoEeRTg6FWt V3FoZeb/Pfcslwv6G9ml4ZN+Vhlb3NLpYwVIhyIi4T14lqPagW7Xqsj1Vy9HAv94g+UV eCE+uAt/Ww3sa8lilIbrUy2CO/y3njkzPQf1M1G2yc4qYYfB6dr/PTXL5EfFbJB+6aSL RGoqXV5w3f2vXm8fzuxp5DSKfDMzQA8c3OkoqHlFbZHcWSFOF/jGyPHFIwQaqg3O6Ipn SNTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688420598; x=1691012598; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zb/tmr/noeY2rNeZVS/rd5RWOm9XHi+842eE/ic0+a0=; b=e2AzJ39aygkwS7P+K/nxYF4waohTfbBcWigHWCs/t/Iry637I4yF5bzDHrArgZkSIV RLktEp29Rtt8s7SZWWcFcm6cunUlAyVJAX3nbtp0DjsaZe2RoeX5AAGCCK8hTgrSKoSB DhAsw6VETK/Agu3ej7EenjAn4ZeFUTYxINHlsObdIAE4c0rZmrcImNPBatDOOIgzPvoD hMJADm1l8f9S9ZpavBwJSE4glTP2me0/9Bio7IbJnqWOBzpkVwItkmxycTeLKfwPx3yT SC2fmmmJO5iCW3WoxHrbLjbl8YvHGRFVRZ8Ymb4AoS/UCSDCtvZ6+27n2Hi1NCnRSfDh A8Gg== X-Gm-Message-State: ABy/qLYu79PqR/e31aST46LWH7ktN3rNNWQN+y2M8kyYOgM0i+r9XTFv +1B3UiFvPYh8mxqu99a1f15g7Q== X-Google-Smtp-Source: APBJJlFY5Wk4ycN3RNcdbeJSOzZJDY4GA9boj1G/G7OddE78CxDwcF9OafgCPpqmAI1xuqydRL+o/g== X-Received: by 2002:a17:902:c1cd:b0:1b8:400a:48f2 with SMTP id c13-20020a170902c1cd00b001b8400a48f2mr12348909plc.62.1688420597872; Mon, 03 Jul 2023 14:43:17 -0700 (PDT) Received: from ziepe.ca (ip-216-194-73-131.syban.net. [216.194.73.131]) by smtp.gmail.com with ESMTPSA id h23-20020a17090aea9700b0025dc5749b4csm14888942pjz.21.2023.07.03.14.43.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jul 2023 14:43:17 -0700 (PDT) Received: from jgg by jggl with local (Exim 4.95) (envelope-from ) id 1qGRKG-000BU8-Jc; Mon, 03 Jul 2023 18:43:16 -0300 Date: Mon, 3 Jul 2023 18:43:16 -0300 From: Jason Gunthorpe To: Mina Almasry Cc: David Ahern , Jakub Kicinski , Jesper Dangaard Brouer , brouer@redhat.com, Alexander Duyck , Yunsheng Lin , davem@davemloft.net, pabeni@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Lorenzo Bianconi , Yisen Zhuang , Salil Mehta , Eric Dumazet , Sunil Goutham , Geetha sowjanya , Subbaraya Sundeep , hariprasad , Saeed Mahameed , Leon Romanovsky , Felix Fietkau , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Jesper Dangaard Brouer , Ilias Apalodimas , linux-rdma@vger.kernel.org, linux-wireless@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, Jonathan Lemon Subject: Re: Memory providers multiplexing (Was: [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag) Message-ID: References: <908b8b17-f942-f909-61e6-276df52a5ad5@huawei.com> <72ccf224-7b45-76c5-5ca9-83e25112c9c6@redhat.com> <20230616122140.6e889357@kernel.org> <20230619110705.106ec599@kernel.org> <5e0ac5bb-2cfa-3b58-9503-1e161f3c9bd5@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Jul 02, 2023 at 11:22:33PM -0700, Mina Almasry wrote: > On Sun, Jul 2, 2023 at 9:20 PM David Ahern wrote: > > > > On 6/29/23 8:27 PM, Mina Almasry wrote: > > > > > > Hello Jakub, I'm looking into device memory (peer-to-peer) networking > > > actually, and I plan to pursue using the page pool as a front end. > > > > > > Quick description of what I have so far: > > > current implementation uses device memory with struct pages; I am > > > putting all those pages in a gen_pool, and we have written an > > > allocator that allocates pages from the gen_pool. In the driver, we > > > use this allocator instead of alloc_page() (the driver in question is > > > gve which currently doesn't use the page pool). When the driver is > > > done with the p2p page, it simply decrements the refcount on it and > > > the page is freed back to the gen_pool. > > Quick update here, I was able to get my implementation working with > the page pool as a front end with the memory provider API Jakub wrote > here: > https://github.com/kuba-moo/linux/tree/pp-providers > > The main complication indeed was the fact that my device memory pages > are ZONE_DEVICE pages, which are incompatible with the page_pool due > to the union in struct page. I thought of a couple of approaches to > resolve that. > > 1. Make my device memory pages non-ZONE_DEVICE pages. Hard no on this from a mm perspective.. We need P2P memory to be properly tagged and have the expected struct pages to be DMA mappable and otherwise, you totally break everything if you try to do this.. > 2. Convert the pages from ZONE_DEVICE pages to page_pool pages and > vice versa as they're being inserted and removed from the page pool. This is kind of scary, it is very, very, fragile to rework the pages like this. Eg what happens when the owning device unplugs and needs to revoke these pages? I think it would likely crash.. I think it also technically breaks the DMA API as we may need to look into the pgmap to do cache ops on some architectures. I suggest you try to work with 8k folios and then the tail page's struct page is empty enough to store the information you need.. Or allocate per page memory and do a memdesc like thing.. Though overall, you won't find devices creating struct pages for their P2P memory today, so I'm not sure what the purpose is. Jonathan already got highly slammed for proposing code to the kernel that was unusable. Please don't repeat that. Other than a special NVMe use case the interface for P2P is DMABUF right now and it is not struct page backed. Even if we did get to struct pages for device memory, it is highly likely cases you are interested in will be using larger than 4k folios, so page pool would need to cope with this nicely as well. Jason