Re: Netperf UDP_STREAM regression due to not sending IPIs in ttwu_queue()

[Mike Galbraith <efault@gmx.de>]

On Tue, 2012-10-02 at 14:14 +0100, Mel Gorman wrote: 
> On Tue, Oct 02, 2012 at 11:31:22AM +0200, Mike Galbraith wrote:
> > On Tue, 2012-10-02 at 09:45 +0100, Mel Gorman wrote: 
> > > On Tue, Oct 02, 2012 at 09:49:36AM +0200, Mike Galbraith wrote:
> > 
> > > > Hm, 518cd623 fixed up the troubles I saw.  How exactly are you running
> > > > this?
> > > > 
> > > 
> > > You saw problems with TCP_RR where as this is UDP_STREAM.
> > 
> > Yeah, but I wanted to stare at UDP_STREAM as you run it to see if it
> > would tell me anything about why those numbers happen.
> > 
> > > I'm running this through MMTests with a version of the
> > > configs/config-global-dhp__network-performance file that only runs
> > > netperf-udp. Ultimately it runs netperf for a size something like
> > > this
> > > 
> > > SIZE=64
> > > taskset -c 0 netserver
> > > taskset -c 1 netperf -t UDP_STREAM -i 50,6 -I 99,1 -l 20 -H 127.0.0.1 -- -P 15895 -s 32768 -S 32768 -m $SIZE -M $SIZE
> > 
> 
> lock_stat points at the runqueue lock which makes sense as without the
> IPI the rq->lock has to be taken

Perf top says we're spinning in ttwu() with NO_TTWU_QUEUE.

nohz=off idle=halt, netperf -l bignum vs -i 50,6 -I 99,1 -l 20, watching
with taskset -c 3 perf top -C 1 -U.

Switch rate rises by nearly 200k/s with NO_TTWU_QUEUE, ttwu() climbs to
#1 spot.  Annotate shows while (p->on_cpu) cpu_relax() eating ~50% of
all ttwu() cycles.  Turn TTWU_QUEUE back on, spin evaporates, switch
rate drops, throughput climbs.  Nifty.

-Mike