Re: [Lse-tech] Re: (RFC): SKB Initialization

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Mala Anand]

>On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>>
>> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
>> >> The patch reduces the number of cylces by 25%
>>
>> >The data you are reporting is flawed: where are the average cycle
>> >times spent in __kfree_skb with the patch?
>>
>> I measured the cycles for only the initialization code in alloc_skb
>> and __kfree_skb. Since the init code is removed from __kfree_skb,
>> no cycles are spent there.

>Then the testing technique is flawed.  You should include all of the
>operations included in an alloc_skb/kfree_skb pair in order to see
>the overall effect of the change, otherwise your change could have a
>net negative effect which would not be noticed.

Cycles for the whole routines alloc_skb and __kfree_skb are as follows:

Baseline 2.5.25
----------------
       alloc/free average cycles
       -------------------------
Runs:      1st              2nd          3rd

CPU0:    337/1163       336/1132      304/1100
CPU1:    318/1164       309/1153      311/1127


2.5.25+skbinit patch
--------------------

       alloc/free average cycles
       -------------------------
Runs:      1st          2nd            3rd

CPU0:   447/1015       580/846        402/905
CPU1:   419/1003       383/915        547/856

The above figures indicate that the cycles spent in alloc_skb and
__kfree_skb have gained 5% in the patch case.  However if you
take the absolute cycles and average them for the three runs it
comes around 145 cycles saving that is close to what I posted earlier
by measuring just the changed code. As the scope of the code measured
widens the percentage improvement comes down.

So the first two scopes, 1. measuring the cycles spent in changed code
2. measuring the cycles spent in alloc_skb and __kfree_skb, results
are consistent.

The third scope would be measuring this patch in a workload environment.
We measured it in a web serving workload and found that we get 0.7%
improvement.

I would like to stress again that this patch helps only when the
allocations
and frees occur on two different CPUs.  I measured it in a UNI system and
did not see any impact.

Regards,
    Mala


   Mala Anand
   IBM Linux Technology Center - Kernel Performance
   E-mail:manand@us.ibm.com
   http://www-124.ibm.com/developerworks/opensource/linuxperf
   http://www-124.ibm.com/developerworks/projects/linuxperf
   Phone:838-8088; Tie-line:678-8088




                                                                                                                                               
                      Benjamin LaHaise                                                                                                         
                      <bcrl@redhat.com>                To:       Mala Anand/Austin/IBM@IBMUS                                                   
                      Sent by:                         cc:       alan@lxorguk.ukuu.org.uk, Bill Hartner/Austin/IBM@IBMUS, davem@redhat.com,    
                      lse-tech-admin@lists.sour         linux-kernel@vger.kernel.org, lse-tech@lists.sourceforge.net                           
                      ceforge.net                      Subject:  [Lse-tech] Re: (RFC): SKB Initialization                                      
                                                                                                                                               
                                                                                                                                               
                      08/21/02 01:16 PM                                                                                                        
                                                                                                                                               
                                                                                                                                               



On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>
> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
> >> The patch reduces the numer of cylces by 25%
>
> >The data you are reporting is flawed: where are the average cycle
> >times spent in __kfree_skb with the patch?
>
> I measured the cycles for only the initialization code in alloc_skb
> and __kfree_skb. Since the init code is removed from __kfree_skb,
> no cycles are spent there.

Then the testing technique is flawed.  You should include all of the
operations included in an alloc_skb/kfree_skb pair in order to see
the overall effect of the change, otherwise your change could have a
net negative effect which would not be noticed.

                         -ben
--
"You will be reincarnated as a toad; and you will be much happier."


-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone?  Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
Lse-tech mailing list
Lse-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lse-tech

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Benjamin LaHaise]

On Thu, Aug 22, 2002 at 12:22:34PM -0500, Mala Anand wrote:
> I would like to stress again that this patch helps only when the
> allocations
> and frees occur on two different CPUs.  I measured it in a UNI system and
> did not see any impact.

Thanks, that looks a lot more complete.  We discussed this on irc a bit, and 
Andi Kleen pointed out that several years of hacking on skbs has probably 
changed the layout significantly from the original intention of keeping all 
the initializations to a cacheline or two.  I also pointed out that it might 
be worth looking at cache misses and perhaps adding a prefetch instruction 
or two, especially during allocation when an skb will be used immediately.  
Another point is to check the order of writes that gcc is generating to the 
skb: if the writes are sequential, the cpu can combine them and make use of 
the internal 64 bit bus to the cache.  In combination with write buffers in 
the cpu, that makes the writes in __kfree_skb almost free, but if the cache 
lines are spread out or cold, that would explain the degredation you're 
seeing.  Cheers,

		-ben

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Dave Hansen]

Mala Anand wrote:
> The third scope would be measuring this patch in a workload environment.
> We measured it in a web serving workload and found that we get 0.7%
> improvement.

First of all, the patch doesn't apply at all against the current 
bitkeeper tree.  I can post the exact one I used if you like.

I tried this under our Specweb99 setup.  Here's a snippet of 
readprofile with, then without the patch:

   8788 __kfree_skb
   8970 mod_timer
   9095 file_read_actor
  10778 alloc_skb
  10905 skb_clone
  11368 e1000_clean_tx_irq
  13595 e1000_intr
  18367 csum_partial_copy_generic
  27848 e1000_xmit_frame
225838 poll_idle
623160 total                                      0.4107

alloc:free ratio: 1.226
(__kfree_skb+alloc_skb)/total = 3.14%

   4535 alloc_skb
   4559 do_tcp_sendpages
   4596 e1000_clean_rx_irq
   4847 dev_queue_xmit
   5020 tcp_clean_rtx_queue
   5155 batch_entropy_store
   5165 kmalloc
   5309 tcp_transmit_skb
   6060 do_schedule
   6138 qdisc_restart
   6235 tcp_v4_rcv
   6393 kfree
   6787 do_gettimeofday
   7089 __d_lookup
   7810 ip_queue_xmit
   8303 skb_clone
   8858 file_read_actor
   8885 mod_timer
   9375 .text.lock.namei
  10267 .text.lock.dec_and_lock
  10936 e1000_clean_tx_irq
  13001 __kfree_skb
  13322 skb_release_data
  13562 e1000_intr
  18099 csum_partial_copy_generic
  27447 e1000_xmit_frame
225023 poll_idle
628695 total                                      0.4143

alloc:free ratio: 0.348
(__kfree_skb+alloc_skb)/total = 2.79%

You can see the entire readprofile here:
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
No, I don't know why I have so much idle time.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [Ibm-specweb99] Re: [Lse-tech] Re: (RFC): SKB Initialization

[Nivedita Singhvi]

Dave, 

Just FYI, the profile of the second link (*mala*) is the
one youre quoting first in this msg, and the profile of
the first link (presumably prior to mala's patch) is the
one you've quoted second in the mail. Hopefully, the links
are just misnamed, and the profiles listed before/after are
the right ones here in the mail. :).

It would be useful to know how consistent these profiles are,
and the variance youre seeing with these runs, before reaching any 
conclusions..

For instance, skb_release_data(), which wasnt altered,
increased from 7259 to 13,322, which is on par with the
kind of gain expected by the patch in the other functions.
So is this just normal variance, or a result of the patch?

Looking at most of the Specweb profiles and networking
in general, and because so much here depends on which cpu
code gets run on, and cache behaviour, I'd say youre going
to get a lot of variance...

thanks,
Nivedita

Quoting Dave Hansen <haveblue@us.ibm.com>:

[snip] 

> First of all, the patch doesn't apply at all against the current 
> bitkeeper tree.  I can post the exact one I used if you like.
> 
> I tried this under our Specweb99 setup.  Here's a snippet of 
> readprofile with, then without the patch:
> 
>    8788 __kfree_skb
>    8970 mod_timer
>    9095 file_read_actor
>   10778 alloc_skb
>   10905 skb_clone
>   11368 e1000_clean_tx_irq
>   13595 e1000_intr
>   18367 csum_partial_copy_generic
>   27848 e1000_xmit_frame
> 225838 poll_idle
> 623160 total                                      0.4107
> 
> alloc:free ratio: 1.226
> (__kfree_skb+alloc_skb)/total = 3.14%
> 
>    4535 alloc_skb
>    4559 do_tcp_sendpages
>    4596 e1000_clean_rx_irq
>    4847 dev_queue_xmit
>    5020 tcp_clean_rtx_queue
>    5155 batch_entropy_store
>    5165 kmalloc
>    5309 tcp_transmit_skb
>    6060 do_schedule
>    6138 qdisc_restart
>    6235 tcp_v4_rcv
>    6393 kfree
>    6787 do_gettimeofday
>    7089 __d_lookup
>    7810 ip_queue_xmit
>    8303 skb_clone
>    8858 file_read_actor
>    8885 mod_timer
>    9375 .text.lock.namei
>   10267 .text.lock.dec_and_lock
>   10936 e1000_clean_tx_irq
>   13001 __kfree_skb
>   13322 skb_release_data
>   13562 e1000_intr
>   18099 csum_partial_copy_generic
>   27447 e1000_xmit_frame
> 225023 poll_idle
> 628695 total                                      0.4143
> 
> alloc:free ratio: 0.348
> (__kfree_skb+alloc_skb)/total = 2.79%
> 
> You can see the entire readprofile here:
> 
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap
-08-22-2002-11.20.17/
> 
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap
-mala-08-22-2002-11.44.25/
> No, I don't know why I have so much idle time.
> 
> -- 
> Dave Hansen
> haveblue@us.ibm.com
> 
> _______________________________________________
> ibm-specweb99 mailing list
> ibm-specweb99@linux.ibm.com
> http://ltc.linux.ibm.com/mailman/listinfo/ibm-specweb99
> 
> 

Re: [Lse-tech] Re: (RFC): SKB Initialization

[William Lee Irwin III]

On Thu, Aug 22, 2002 at 12:02:27PM -0700, Dave Hansen wrote:
> You can see the entire readprofile here:
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
> No, I don't know why I have so much idle time.

Hmm, I found that tiobench was spending a lot of time idle due to
__wait_on_inode() and get_request_wait(). I bumped up the size of
the inode wait table to 1024 and the request queue size to 16384
and saw that most of them then spent their time stuck on ->i_sem
during the initial open of the file they were going to pound on
for the duration of the run.

I determined this by just ^C'ing with kgdb and backtracing various
"stuck" processes. I think various profiling patches might be able to
give you an idea of what people are going to sleep on too.


Cheers,
Bill

Re: [Lse-tech] Re: (RFC): SKB Initialization

[Bill Hartner]

Mala Anand wrote:
> 
> Baseline 2.5.25
> ----------------
>        alloc/free average cycles
>        -------------------------
> Runs:      1st              2nd          3rd
> 
> CPU0:    337/1163       336/1132      304/1100
> CPU1:    318/1164       309/1153      311/1127
> 
> 2.5.25+skbinit patch
> --------------------
> 
>        alloc/free average cycles
>        -------------------------
> Runs:      1st          2nd            3rd
> 
> CPU0:   447/1015       580/846        402/905
> CPU1:   419/1003       383/915        547/856
> 
> The above figures indicate that the cycles spent in alloc_skb and
> __kfree_skb have gained 5% in the patch case.  However if you
> take the absolute cycles and average them for the three runs it
> comes around 145 cycles saving that is close to what I posted earlier
> by measuring just the changed code. As the scope of the code measured
> widens the percentage improvement comes down.

Measuring just the initialization code yielded a reduction of 156 cycles.
Measuring alloc_skb and __kfree_skb yielded a reduction of 145 cycles.
This was on a 2 CPU system.

The worst case scenario would be allocating the skb header on one
CPU then freeing it on another CPU.  The best case would be doing 
all of the allocs and frees on one CPU.

You can use process/irq affinity to create both of these cases.
Can you measure these ?

Bill