From: Mitchell Augustin <mitchell.augustin@canonical.com>
To: bhelgaas@google.com, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org
Cc: mitchell.augustin@canonical.com, alex.williamson@redhat.com
Subject: [PATCH 0/1] PCI: Add decode disable/enable to device level and separate BAR info logging into separate function
Date: Thu, 5 Dec 2024 18:03:50 -0600 [thread overview]
Message-ID: <20241206000351.884656-1-mitchell.augustin@canonical.com> (raw)
In a virtualized environment with PCI devices passed-through,
disabling and enabling decoding is not trivial, and in some
configurations can take up to 2 seconds of wall clock time to
complete. Since this operation could be done once around each
device for devices with multiple BARs (reducing redundancy),
add an additional decode disable/enable mask at the device
level to prevent redundant disable/enables from occurring
during each BAR sizing operation, when pci_read_bases() is
the originator.
Since __pci_read_base() can also be called independently,
keep the current disable/enable mask in that function as-is.
Since printk cannot be used while decoding is disabled,
move the debug prints in __pci_read_base() to
a separate function, __pci_print_bar_status().
To enable this, add pointers to the signature for
__pci_read_base() through which the caller can access
necessary data from __pci_read_base() and pass it to
__pci_print_bar_status().
This has been tested on an SR670v2 host and guest VM,
a DGX H100 host and guest VM, and a DGX A100 host and guest
VM. I confirmed that BAR info logged to dmesg was consistent
on each between unmodified 6.12.1 and with this patch, that
BAR mappings in /proc/iomem were consistent between
versions, and that lspci -vv results were consistent
between versions. On the A100/H100, I also confirmed that the
Nvidia driver loads as expected with the patch, and that
VM boot time with cold-plugged, passed-through GPUs is about
2x faster. No regressions were observed.
Originally developed/tested against upstream 6.12.1, but
I also confirmed that this applies cleanly to the pci tree
@ v6.13-rc1 and that it results in the same boot speed
decrease there when tested in an H100 guest VM.
Link: https://lore.kernel.org/all/CAHTA-uYp07FgM6T1OZQKqAdSA5JrZo0ReNEyZgQZub4mDRrV5w@mail.gmail.com/
Reported-by: "Mitchell Augustin" <mitchell.augustin@canonical.com>
Closes: https://lore.kernel.org/all/CAHTA-uYp07FgM6T1OZQKqAdSA5JrZo0ReNEyZgQZub4mDRrV5w@mail.gmail.com/
Mitchell Augustin (1):
PCI: Add decode disable/enable to device level and separate BAR info
logging into separate function
drivers/pci/iov.c | 16 ++++-
drivers/pci/pci.h | 7 ++-
drivers/pci/probe.c | 149 +++++++++++++++++++++++++++++++++++++++++---
3 files changed, 158 insertions(+), 14 deletions(-)
---
base-commit: d390303b28dabbb91b2d32016a4f72da478733b9
--
2.43.0
next reply other threads:[~2024-12-06 0:04 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-12-06 0:03 Mitchell Augustin [this message]
2024-12-06 0:03 ` [PATCH 1/1] " Mitchell Augustin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241206000351.884656-1-mitchell.augustin@canonical.com \
--to=mitchell.augustin@canonical.com \
--cc=alex.williamson@redhat.com \
--cc=bhelgaas@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®