From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71E5A8462 for ; Wed, 29 Jan 2025 13:39:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738157946; cv=none; b=mqK+UGh59hzqI65qfiH+EGr2+5pxYWI/UYQYhUaBPEWTitr8Jin2oHJ1NEg4MKiCZZV82GYNcGXnQ30eBn9KOTSyl0iyk5ugOn90F2REowfD6kQhEfEH2oeT7ahQWfQWeNOji2GDLBVleBHkQBgHU7KshBMO5VEqavylkSFy9uA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738157946; c=relaxed/simple; bh=RxH/qA/BgpYkjBVmBtOP+Wt33+Id7p5IBZ1sJh+PVe8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=MkVYKm47AvBuh7Wn4Uw0iy4RsnZfAAvB79pYaJxf8+6+QchZWwBKPMSjn4R7QhaNVhg85kSuoXzkNIU9yhF6rn3yxuKVocw+G53+HfaoWCh87W3L1jI6WRXee/EGqh0rdwAG0RIdcLXbweEdw2cfIuZ5bU8mAL7A9gmrN1kH0W4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch; spf=none smtp.mailfrom=ffwll.ch; dkim=pass (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b=LRbXARLx; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ffwll.ch Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ffwll.ch header.i=@ffwll.ch header.b="LRbXARLx" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-436202dd730so48733325e9.2 for ; Wed, 29 Jan 2025 05:39:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; t=1738157942; x=1738762742; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=ekPyuSS82CPNI1Lh/E6IsIbzoEEGT2LSw6fRdzUam+8=; b=LRbXARLxcVlSVLOIW9nPwxiGleXpTsHVUK1xrS0UwYak72R+r5JA7lmqK8UxnnQHw+ hjfkoSJN3yj16qLBqvfCG4JqRqP92QoaOZ3ZY+1vjEGDAPpILcv+DlIlstdU29B49j00 9/MvPxpeF0nk8EcLadAsFREGfVtX/6b+FsBDM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738157942; x=1738762742; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ekPyuSS82CPNI1Lh/E6IsIbzoEEGT2LSw6fRdzUam+8=; b=B3b/jAr6FpzPHesvJ2ZaZvcRnsy01QbSQi28sLfJb0RhJNluHD01N7P0O/98oBa8S2 EmK1di7B0yy2cGVDUul6EON/te7EfiwY+Ic7mIoQ4Ljq4Mt924FlxDfkvW+i8/TK/twa 9nLk7qsqRcmI3wQ+PCmfWu8/yXDxM8SC27r8yHLkkTJj2qOQeMHyri/FLqqYYE7X01n6 T50lA9YZDiOqFqW3oaP3dNHl+9ftdqRPgA+1oA/CHu+LRS/Gus/Cig8oy0Xk0lPCTPoD hoehMyBBxxQg7yqNGrEZOKI9jPkEv0a1zJAQPwkcKG3yJqPYmaChYTi1vo430JGkHQSZ S+og== X-Forwarded-Encrypted: i=1; AJvYcCWD3IebXUWkaZJVY5czqz9dlgovJQiaUc+JV1lv7BVYXTQADP2cQ+V+hqZ+iJyAB3nZGCNLByKTb4fVzrE=@vger.kernel.org X-Gm-Message-State: AOJu0YwB7kq7V3S893A/DG0FfJU23p4/gLgn/P8riTiLYVNdwYi+80q3 OQeaT5tXbF1ovXRhRvskODg2xWUQIgyAmYiCjEpY9CmyUumN6DT6euZ0jnzStI8= X-Gm-Gg: ASbGncvv+VsSTiR7b+ttIUTKQPRGExIqkexqZhe4kYBJhF9QiXjl3pPdEE8jMeWf7PE 4uP5zGK+DUqApPZJ6I+ycI+QbCJ/WmWAF7YtSLix+RlGaJaFIMRoV+O8ox7eMQkcEJyshyBGaGI SfY6hjPpYlsotdFTwemjgbQ2A89dX48BiggtbrXg3/NmWNAkg1AM3eFieWIz792pN5+S2ZSkxwq 23Ax8NwUjeQgmin2yw5YipaWpCvBuldMI/g8hebDfUHJlI/TMD4xpmOMPDFBHXoNo3aAS1wi+3a XdXECOr+tlRwaejPkpdh/rEGYms= X-Google-Smtp-Source: AGHT+IFp40yYl/FrcEC18d1HA1Wbr1YBHRju+Twtn7vgv1BIenNhcb0w0RrYUn5K84MH3BCfQBYGmw== X-Received: by 2002:a05:600c:5486:b0:433:c76d:d57e with SMTP id 5b1f17b1804b1-438dc3a40d3mr30211795e9.5.1738157942387; Wed, 29 Jan 2025 05:39:02 -0800 (PST) Received: from phenom.ffwll.local ([2a02:168:57f4:0:5485:d4b2:c087:b497]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438dcc263f0sm23501465e9.9.2025.01.29.05.39.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jan 2025 05:39:01 -0800 (PST) Date: Wed, 29 Jan 2025 14:38:58 +0100 From: Simona Vetter To: Jason Gunthorpe Cc: Thomas =?iso-8859-1?Q?Hellstr=F6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org Subject: Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Message-ID: Mail-Followup-To: Jason Gunthorpe , Thomas =?iso-8859-1?Q?Hellstr=F6m?= , Yonatan Maman , kherbst@redhat.com, lyude@redhat.com, dakr@redhat.com, airlied@gmail.com, simona@ffwll.ch, leon@kernel.org, jglisse@redhat.com, akpm@linux-foundation.org, GalShalom@nvidia.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-mm@kvack.org, linux-tegra@vger.kernel.org References: <20241201103659.420677-1-ymaman@nvidia.com> <20241201103659.420677-2-ymaman@nvidia.com> <7282ac68c47886caa2bc2a2813d41a04adf938e1.camel@linux.intel.com> <20250128132034.GA1524382@ziepe.ca> <20250128151610.GC1524382@ziepe.ca> <20250128172123.GD1524382@ziepe.ca> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20250128172123.GD1524382@ziepe.ca> X-Operating-System: Linux phenom 6.12.11-amd64 On Tue, Jan 28, 2025 at 01:21:23PM -0400, Jason Gunthorpe wrote: > On Tue, Jan 28, 2025 at 05:32:23PM +0100, Thomas Hellström wrote: > > > This series supports three case: > > > > > >  1) pgmap->owner == range->dev_private_owner > > >     This is "driver private fast interconnect" in this case HMM > > > should > > >     immediately return the page. The calling driver understands the > > >     private parts of the pgmap and computes the private interconnect > > >     address. > > > > > >     This requires organizing your driver so that all private > > >     interconnect has the same pgmap->owner. > > > > Yes, although that makes this map static, since pgmap->owner has to be > > set at pgmap creation time. and we were during initial discussions > > looking at something dynamic here. However I think we can probably do > > with a per-driver owner for now and get back if that's not sufficient. > > The pgmap->owner doesn't *have* to fixed, certainly during early boot before > you hand out any page references it can be changed. I wouldn't be > surprised if this is useful to some requirements to build up the > private interconnect topology? The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on pgmap->owner to figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet. You can fake that by doing these checks after hmm_range_fault returned, and if you get a bunch of unsuitable pages, toss it back to hmm_range_fault asking for an unconditional migration to system memory for those. But that's kinda not great and I think goes at least against the spirit of how you want to handle pci p2p in step 2 below? Cheers, Sima > > >  2) The page is DEVICE_PRIVATE and get_dma_pfn_for_device() exists. > > >     The exporting driver has the option to return a P2P struct page > > >     that can be used for PCI P2P without any migration. In a PCI GPU > > >     context this means the GPU has mapped its local memory to a PCI > > >     address. The assumption is that P2P always works and so this > > >     address can be DMA'd from. > > > > So do I understand it correctly, that the driver then needs to set up > > one device_private struct page and one pcie_p2p struct page for each > > page of device memory participating in this way? > > Yes, for now. I hope to remove the p2p page eventually. > > > > If you are just talking about your private multi-path, then that is > > > already handled.. > > > > No, the issue I'm having with this is really why would > > hmm_range_fault() need the new pfn when it could easily be obtained > > from the device-private pfn by the hmm_range_fault() caller? > > That isn't the API of HMM, the caller uses hmm to get PFNs it can use. > > Deliberately returning PFNs the caller cannot use is nonsensical to > it's purpose :) > > > So anyway what we'll do is to try to use an interconnect-common owner > > for now and revisit the problem if that's not sufficient so we can come > > up with an acceptable solution. > > That is the intention for sure. The idea was that the drivers under > the private pages would somehow generate unique owners for shared > private interconnect segments. > > I wouldn't say this is the end all of the idea, if there are better > ways to handle accepting private pages they can certainly be > explored.. > > Jason -- Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch