Re: [PATCH] MMIO should have more priority then IO

[Nadav Amit <namit@vmware.com>]

On Jul 8, 2022, at 5:56 AM, Matthew Wilcox <willy@infradead.org> wrote:

> And looking at the results above, it's not so much the PIO vs MMIO
> that makes a difference, it's the virtualisation. A mmio access goes
> from 269ns to 85us. Rather than messing around with preferring MMIO
> over PIO for config space, having an "enlightenment" to do config
> space accesses would be a more profitable path.

I am unfamiliar with the motivation for this patch, but I just wanted to
briefly regard the advice about enlightments.

“enlightenment”, AFAIK, is Microsoft’s term for "para-virtualization", so
let’s regard the generic term. I think that you consider the bare-metal
results as the possible results from a paravirtual machine, which is mostly
wrong. Para-virtualization usually still requires a VM-exit and for the most
part the hypervisor/host runs similar code for MMIO/hypercall (conceptually;
the code of paravirtual and fully-virtual devices is often different, but
IIUC, this is not what Ajay measured).

Para-virtualization could have *perhaps* helped to reduce the number of
PIO/MMIO and improve performance this way. If, for instance, all the
PIO/MMIO are done during initialization, a paravirtual interface can be use
to batch them together, and that would help. But it is more complicated to
get a performance benefit from paravirtualization if the PIO/MMIO accesses
are “spread”, for instance, done after each interrupt.

Para-virtauilzation and full-virtualization both have pros and cons.
Para-virtualization is many times more efficient, but requires the VM to
have dedicated device drivers for the matter. Try to run a less-common OS
than Linux and it would not work since the OS would not have drivers for the
paras-virtual devices. And even if you add support today for a para-virtual
devices, there are many deployed OSes that do not have such support, and you
would not be able to run them in a VM.

Regardless to virtualization, Ajay’s results show PIO is slower on
bare-metal, and according to his numbers by 165ns, which is significant.
Emulating PIO in hypervisors on x86 is inherently more complex than MMIO, so
the results he got would most likely happen on all hypervisors.

tl;dr: Let’s keep this discussion focused and put paravirtualization aside.
It is not a solution for all the problems in the world.