Re: PREEMPT_RT and I-PIPE: the numbers, part 4

[Ingo Molnar <mingo@elte.hu>]

* Kristian Benoit <kbenoit@opersys.com> wrote:

> The numbers for PREEMPT_RT, however, have dramatically improved. All 
> the 50%+ overhead we saw earlier has now gone away completely. The 
> improvement is in fact nothing short of amazing. We were actually so 
> surprised that we went around looking for any mistakes we may have 
> done in our testing. We haven't found any though. So unless someone 
> comes out with another set of numbers showing differently, we think 
> that a warm round of applause should go to the PREEMPT_RT folks. If 
> nothing else, it gives us satisfaction to know that these test rounds 
> have helped make things better.

yeah, they definitely have helped, and thanks for this round of testing 
too! I'll explain the recent changes to PREEMPT_RT that resulted in 
these speedups in another mail.

Looking at your numbers i realized that the area where PREEMPT_RT is 
still somewhat behind (the flood ping +~10% overhead), you might be 
using an invalid test methodology:

> ping           = on host: "sudo ping -f $TARGET_IP_ADDR"

i've done a couple of ping -f flood tests between various testboxes 
myself, and one thing i found was that it's close to impossible to 
create a stable, comparable packets per second workload! The pps rate 
heavily fluctuated even within the same testrun. Another phenomenon i 
noticed is that the PREEMPT_RT kernel has a tendency to handle _more_ 
ping packets per second, while the vanilla (and thus i suspect the 
i-pipe) kernel throws away more packets.

Thus lmbench under PREEMPT_RT may perform 'slower', but in fact it was 
just an unbalanced and thus unfair test. Once i created a stable packet 
rate, PREEMPT_RT's IRQ overhead became acceptable.

(if your goal was to check how heavily external interrupts can influence 
a PREEMPT_RT box, you should chrt the network IRQ thread to SCHED_OTHER 
and renice it and softirq-net-rx and softirq-net-tx to nice +19.)

this phenomenon could be a speciality of my network setup, but still, 
could you please verify the comparability of the ping -f workloads on 
the vanilla and the PREEMPT_RT kernels? In particular, the interrupt 
rate should be constant and comparable - but it might be better to look 
at both the received and transmitted packets per second. (Since things 
like iptraf are quite expensive when flood pinging is going on, the best 
way i found to measure the packet rate was to process netstat -s output 
via a simple script.)

	Ingo