From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD7F6C0032E for ; Wed, 25 Oct 2023 14:30:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344948AbjJYOax (ORCPT ); Wed, 25 Oct 2023 10:30:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344931AbjJYOau (ORCPT ); Wed, 25 Oct 2023 10:30:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 850F8A3 for ; Wed, 25 Oct 2023 07:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698244202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=clFd29QeDWnwn0GXLK2EBVzyNpuURUgqAcJDTvU+22I=; b=jVVWZeWiHbSUOZqj7p3gUjEZJqgfqteYt20/mfws7/FobrthcbILFzIqin6tGy9NiOrUgk w1HA1DiT8wY1ZUacfWe7uwOBo3er8vuZvLowOEqBZ5ZJyTvDBVk2/GbCWPSAj7FW7y6BPP dFmLs6OYEsqNMBfdWNhBn5ARoa3nfPE= Received: from mail-il1-f200.google.com (mail-il1-f200.google.com [209.85.166.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-680-SHphp9m7Oby7z97zx8zS1A-1; Wed, 25 Oct 2023 10:29:50 -0400 X-MC-Unique: SHphp9m7Oby7z97zx8zS1A-1 Received: by mail-il1-f200.google.com with SMTP id e9e14a558f8ab-3574a84ef27so61086195ab.3 for ; Wed, 25 Oct 2023 07:29:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698244190; x=1698848990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=clFd29QeDWnwn0GXLK2EBVzyNpuURUgqAcJDTvU+22I=; b=RFXXPQ9RWDIMjSUy0wdEkCwWWe/2olOJf3jBdjsIiQB2J5SwmCAfA/3X1x7jK/WPeu jPGia9DvyZFEZuGQAxYNtQwJUkT70Vms1IXNPDeBslk258CN1dD30/x5UVEHalrC3KHY jDXrv1GVrs7TN8Mwp7hEP+qoEGdxWEm0FG6PpObwyr7nWyoAzkus+eZoniuc+M0NqhJ7 VG+t5A7iW5k99A7QYWG9/wISywI99JQScgw0/g/jE1aN9sn0Wth5wrhmZDP5vuu/3kKY CkgtzFEIkEaBAGQ99nQZnl+2XQPLXyjcmDkKYA2puiAx+s058ABKilQMVZMiaw/JnuGM SrTQ== X-Gm-Message-State: AOJu0Yx8AylR3fBlXSDwYVmYzd1nTrh7JX6z+W/0DlJd5j2fHUIAVETf V+OFzNOtOTAFVIlf8V19LQtXXVXV2yQr/E01G6hbo7AyORqs1IM07+i1q7IjKEfBdALfRu2qgk9 VYQ5lHYsL0fMya3JqXcjhkVvS X-Received: by 2002:a92:dc02:0:b0:357:fa1b:475 with SMTP id t2-20020a92dc02000000b00357fa1b0475mr1918076iln.10.1698244189929; Wed, 25 Oct 2023 07:29:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF6/mm7xg+tqsYYT06Tao2qBsLe5d3EVw1veyQKDmmT2B2lvS3D7AxwCL48WgNlBFEzd/DMzw== X-Received: by 2002:a92:dc02:0:b0:357:fa1b:475 with SMTP id t2-20020a92dc02000000b00357fa1b0475mr1918051iln.10.1698244189642; Wed, 25 Oct 2023 07:29:49 -0700 (PDT) Received: from redhat.com ([38.15.60.12]) by smtp.gmail.com with ESMTPSA id m18-20020a92c532000000b0035298bd42a8sm3798992ili.20.2023.10.25.07.29.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:29:49 -0700 (PDT) Date: Wed, 25 Oct 2023 08:29:47 -0600 From: Alex Williamson To: "Tian, Kevin" Cc: Ankit Agrawal , Jason Gunthorpe , Yishai Hadas , "shameerali.kolothum.thodi@huawei.com" , Aniket Agashe , Neo Jia , Kirti Wankhede , "Tarun Gupta (SW-GPU)" , Vikram Sethi , "Currid, Andy" , Alistair Popple , John Hubbard , Dan Williams , "Anuj Aggarwal (SW-GPU)" , "kvm@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper Message-ID: <20231025082947.6094361d.alex.williamson@redhat.com> In-Reply-To: References: <20231015163047.20391-1-ankita@nvidia.com> <20231017165437.69a84f0c.alex.williamson@redhat.com> <20231023084312.15b8e37e.alex.williamson@redhat.com> <20231024082854.0b767d74.alex.williamson@redhat.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.35; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 25 Oct 2023 08:28:44 +0000 "Tian, Kevin" wrote: > > From: Alex Williamson > > Sent: Tuesday, October 24, 2023 10:29 PM > >=20 > > On Tue, 24 Oct 2023 14:03:25 +0000 > > Ankit Agrawal wrote: > > =20 > > > >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is e= mulated > > > >> > as an IO Port BAR, it occurs to me that there's no config space > > > >> > emulation of BAR2 (or BAR3) here.=C2=A0 Doesn't this mean that Q= EMU =20 > > registers =20 > > > >> > the BAR as 32-bit, non-prefetchable?=C2=A0 ie. VFIOBAR.type & .m= em64 are > > > >> > wrong? =20 > > > >> > > > >> Maybe I didn't understand the question, but the PCI config space = =20 > > read/write =20 > > > >> would still be handled by vfio_pci_core_read/write() which returns= the > > > >> appropriate flags. I have checked that the device BARs are 64b and > > > >> prefetchable in the VM. =20 > > > > > > > > vfio_pci_core_read/write() accesses the physical device, which does= n't > > > > implement BAR2.=C2=A0 Why would an unimplemented BAR2 on the physic= al =20 > > device =20 > > > > report 64-bit, prefetchable? > > > > > > > > QEMU records VFIOBAR.type and .mem64 from reading the BAR register = =20 > > in =20 > > > > vfio_bar_prepare() and passes this type to pci_register_bar() in > > > > vfio_bar_register().=C2=A0 Without an implementation of a config sp= ace read > > > > op in the variant driver and with no physical implementation of BAR= 2 on > > > > the device, I don't see how we get correct values in these fields. = =20 > > > > > > I think I see the cause of confusion. There are real PCIe compliant B= ARs > > > present on the device, just that it isn't being used once the C2C > > > interconnect is active. The BARs are 64b prefetchable. Here it the ls= pci > > > snippet of the device on the host. > > > # lspci -v -s 9:1:0.0 > > > 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1) > > > Subsystem: NVIDIA Corporation Device 16eb > > > Physical Slot: 0-5 > > > Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node= 0, =20 > > IOMMU group 19 =20 > > > Memory at 661002000000 (64-bit, prefetchable) [size=3D16M] > > > Memory at 662000000000 (64-bit, prefetchable) [size=3D128G] > > > Memory at 661000000000 (64-bit, prefetchable) [size=3D32M] > > > > > > I suppose this answers the BAR sizing question as well? =20 > >=20 > > Does this BAR2 size match the size we're reporting for the region? Now > > I'm confused why we need to intercept the BAR2 region info if there's > > physically a real BAR behind it. Thanks, > > =20 >=20 > same confusion. >=20 > probably vfio-pci-core can include a helper for cfg space emulation > on emulated BARs to be used by all variant drivers in that category? >=20 > btw intel vgpu also includes an emulation of BAR sizing. same for > future SIOV devices. so there sounds like a general requirement but > of course sharing it between vfio-pci and mdev/siov would be more > difficult. Yes, I see a need for this in the virtio-vfio-pci driver as well. It would simplify config emulation a lot if the variant driver could manipulate the perm_bits.virt and .write bit arrays and simply update vconfig for things like device-id and revision. Thanks, Alex