On Sat, Dec 03, 2011 at 03:44:36PM +1030, Rusty Russell wrote: > On Sat, 03 Dec 2011 10:09:44 +1100, Benjamin Herrenschmidt wrote: > > On Tue, 2011-11-29 at 14:31 +0200, Ohad Ben-Cohen wrote: > > > A trivial, albeit sub-optimal, solution would be to simply revert > > > commit d57ed95 "virtio: use smp_XX barriers on SMP". Obviously, though, > > > that's going to have a negative impact on performance of SMP-based > > > virtualization use cases. > > > > Have you measured the impact of using normal barriers (non-SMP ones) > > like we use on normal HW drivers unconditionally ? > > > > IE. If the difference is small enough I'd say just go for it and avoid > > the bloat. > > Yep. Plan is: > 1) Measure the difference. > 2) Difference unmeassurable? Use normal barriers (ie. revert d57ed95). > 3) Difference small? Revert d57ed95 for 3.2, revisit for 3.3. > 4) Difference large? Runtime switch based on "if you're PCI" for 3.2, > revisit for 3.3. > > Cheers, > Rusty. Forwarding some results by Amos, who run multiple netperf streams in parallel, from an external box to the guest. TCP_STREAM results were noisy. This could be due to buffering done by TCP, where packet size varies even as message size is constant. TCP_RR results were consistent. In this benchmark, after switching to mandatory barriers, CPU utilization increased by up to 35% while throughput went down by up to 14%. the normalized throughput/cpu regressed consistently, between 7 and 35% The "fix" applied was simply this: diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 3198f2e..fdccb77 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -23,7 +23,7 @@ /* virtio guest is communicating with a virtual "device" that actually runs on * a host processor. Memory barriers are used to control SMP effects. */ -#ifdef CONFIG_SMP +#if 0 /* Where possible, use SMP barriers which are more lightweight than mandatory * barriers, because mandatory barriers control MMIO effects on accesses * through relaxed memory I/O windows (which virtio does not use). */ -- MST