* Re: [Lse-tech] Re: (RFC): SKB Initialization
@ 2002-08-22 17:22 Mala Anand
2002-08-22 18:32 ` Benjamin LaHaise
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Mala Anand @ 2002-08-22 17:22 UTC (permalink / raw)
To: Benjamin LaHaise
Cc: alan, Bill Hartner, davem, linux-kernel, lse-tech, lse-tech-admin
>On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>>
>> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
>> >> The patch reduces the number of cylces by 25%
>>
>> >The data you are reporting is flawed: where are the average cycle
>> >times spent in __kfree_skb with the patch?
>>
>> I measured the cycles for only the initialization code in alloc_skb
>> and __kfree_skb. Since the init code is removed from __kfree_skb,
>> no cycles are spent there.
>Then the testing technique is flawed. You should include all of the
>operations included in an alloc_skb/kfree_skb pair in order to see
>the overall effect of the change, otherwise your change could have a
>net negative effect which would not be noticed.
Cycles for the whole routines alloc_skb and __kfree_skb are as follows:
Baseline 2.5.25
----------------
alloc/free average cycles
-------------------------
Runs: 1st 2nd 3rd
CPU0: 337/1163 336/1132 304/1100
CPU1: 318/1164 309/1153 311/1127
2.5.25+skbinit patch
--------------------
alloc/free average cycles
-------------------------
Runs: 1st 2nd 3rd
CPU0: 447/1015 580/846 402/905
CPU1: 419/1003 383/915 547/856
The above figures indicate that the cycles spent in alloc_skb and
__kfree_skb have gained 5% in the patch case. However if you
take the absolute cycles and average them for the three runs it
comes around 145 cycles saving that is close to what I posted earlier
by measuring just the changed code. As the scope of the code measured
widens the percentage improvement comes down.
So the first two scopes, 1. measuring the cycles spent in changed code
2. measuring the cycles spent in alloc_skb and __kfree_skb, results
are consistent.
The third scope would be measuring this patch in a workload environment.
We measured it in a web serving workload and found that we get 0.7%
improvement.
I would like to stress again that this patch helps only when the
allocations
and frees occur on two different CPUs. I measured it in a UNI system and
did not see any impact.
Regards,
Mala
Mala Anand
IBM Linux Technology Center - Kernel Performance
E-mail:manand@us.ibm.com
http://www-124.ibm.com/developerworks/opensource/linuxperf
http://www-124.ibm.com/developerworks/projects/linuxperf
Phone:838-8088; Tie-line:678-8088
Benjamin LaHaise
<bcrl@redhat.com> To: Mala Anand/Austin/IBM@IBMUS
Sent by: cc: alan@lxorguk.ukuu.org.uk, Bill Hartner/Austin/IBM@IBMUS, davem@redhat.com,
lse-tech-admin@lists.sour linux-kernel@vger.kernel.org, lse-tech@lists.sourceforge.net
ceforge.net Subject: [Lse-tech] Re: (RFC): SKB Initialization
08/21/02 01:16 PM
On Wed, Aug 21, 2002 at 01:07:09PM -0500, Mala Anand wrote:
>
> >On Wed, Aug 21, 2002 at 11:59:44AM -0500, Mala Anand wrote:
> >> The patch reduces the numer of cylces by 25%
>
> >The data you are reporting is flawed: where are the average cycle
> >times spent in __kfree_skb with the patch?
>
> I measured the cycles for only the initialization code in alloc_skb
> and __kfree_skb. Since the init code is removed from __kfree_skb,
> no cycles are spent there.
Then the testing technique is flawed. You should include all of the
operations included in an alloc_skb/kfree_skb pair in order to see
the overall effect of the change, otherwise your change could have a
net negative effect which would not be noticed.
-ben
--
"You will be reincarnated as a toad; and you will be much happier."
-------------------------------------------------------
This sf.net email is sponsored by: OSDN - Tired of that same old
cell phone? Get a new here for FREE!
https://www.inphonic.com/r.asp?r=sourceforge1&refcode1=vs3390
_______________________________________________
Lse-tech mailing list
Lse-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lse-tech
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Lse-tech] Re: (RFC): SKB Initialization
2002-08-22 17:22 [Lse-tech] Re: (RFC): SKB Initialization Mala Anand
@ 2002-08-22 18:32 ` Benjamin LaHaise
2002-08-22 19:02 ` Dave Hansen
2002-08-23 19:09 ` Bill Hartner
2 siblings, 0 replies; 6+ messages in thread
From: Benjamin LaHaise @ 2002-08-22 18:32 UTC (permalink / raw)
To: Mala Anand
Cc: alan, Bill Hartner, davem, linux-kernel, lse-tech, lse-tech-admin, ak
On Thu, Aug 22, 2002 at 12:22:34PM -0500, Mala Anand wrote:
> I would like to stress again that this patch helps only when the
> allocations
> and frees occur on two different CPUs. I measured it in a UNI system and
> did not see any impact.
Thanks, that looks a lot more complete. We discussed this on irc a bit, and
Andi Kleen pointed out that several years of hacking on skbs has probably
changed the layout significantly from the original intention of keeping all
the initializations to a cacheline or two. I also pointed out that it might
be worth looking at cache misses and perhaps adding a prefetch instruction
or two, especially during allocation when an skb will be used immediately.
Another point is to check the order of writes that gcc is generating to the
skb: if the writes are sequential, the cpu can combine them and make use of
the internal 64 bit bus to the cache. In combination with write buffers in
the cpu, that makes the writes in __kfree_skb almost free, but if the cache
lines are spread out or cold, that would explain the degredation you're
seeing. Cheers,
-ben
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Lse-tech] Re: (RFC): SKB Initialization
2002-08-22 17:22 [Lse-tech] Re: (RFC): SKB Initialization Mala Anand
2002-08-22 18:32 ` Benjamin LaHaise
@ 2002-08-22 19:02 ` Dave Hansen
2002-08-22 20:58 ` [Ibm-specweb99] " Nivedita Singhvi
2002-08-22 22:05 ` William Lee Irwin III
2002-08-23 19:09 ` Bill Hartner
2 siblings, 2 replies; 6+ messages in thread
From: Dave Hansen @ 2002-08-22 19:02 UTC (permalink / raw)
To: Mala Anand
Cc: Benjamin LaHaise, alan, Bill Hartner, davem, linux-kernel,
lse-tech, lse-tech-admin
Mala Anand wrote:
> The third scope would be measuring this patch in a workload environment.
> We measured it in a web serving workload and found that we get 0.7%
> improvement.
First of all, the patch doesn't apply at all against the current
bitkeeper tree. I can post the exact one I used if you like.
I tried this under our Specweb99 setup. Here's a snippet of
readprofile with, then without the patch:
8788 __kfree_skb
8970 mod_timer
9095 file_read_actor
10778 alloc_skb
10905 skb_clone
11368 e1000_clean_tx_irq
13595 e1000_intr
18367 csum_partial_copy_generic
27848 e1000_xmit_frame
225838 poll_idle
623160 total 0.4107
alloc:free ratio: 1.226
(__kfree_skb+alloc_skb)/total = 3.14%
4535 alloc_skb
4559 do_tcp_sendpages
4596 e1000_clean_rx_irq
4847 dev_queue_xmit
5020 tcp_clean_rtx_queue
5155 batch_entropy_store
5165 kmalloc
5309 tcp_transmit_skb
6060 do_schedule
6138 qdisc_restart
6235 tcp_v4_rcv
6393 kfree
6787 do_gettimeofday
7089 __d_lookup
7810 ip_queue_xmit
8303 skb_clone
8858 file_read_actor
8885 mod_timer
9375 .text.lock.namei
10267 .text.lock.dec_and_lock
10936 e1000_clean_tx_irq
13001 __kfree_skb
13322 skb_release_data
13562 e1000_intr
18099 csum_partial_copy_generic
27447 e1000_xmit_frame
225023 poll_idle
628695 total 0.4143
alloc:free ratio: 0.348
(__kfree_skb+alloc_skb)/total = 2.79%
You can see the entire readprofile here:
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
No, I don't know why I have so much idle time.
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Ibm-specweb99] Re: [Lse-tech] Re: (RFC): SKB Initialization
2002-08-22 19:02 ` Dave Hansen
@ 2002-08-22 20:58 ` Nivedita Singhvi
2002-08-22 22:05 ` William Lee Irwin III
1 sibling, 0 replies; 6+ messages in thread
From: Nivedita Singhvi @ 2002-08-22 20:58 UTC (permalink / raw)
To: ibm-specweb99, Dave Hansen
Cc: Mala Anand, Benjamin LaHaise, alan, Bill Hartner, davem,
linux-kernel, lse-tech, lse-tech-admin
Dave,
Just FYI, the profile of the second link (*mala*) is the
one youre quoting first in this msg, and the profile of
the first link (presumably prior to mala's patch) is the
one you've quoted second in the mail. Hopefully, the links
are just misnamed, and the profiles listed before/after are
the right ones here in the mail. :).
It would be useful to know how consistent these profiles are,
and the variance youre seeing with these runs, before reaching any
conclusions..
For instance, skb_release_data(), which wasnt altered,
increased from 7259 to 13,322, which is on par with the
kind of gain expected by the patch in the other functions.
So is this just normal variance, or a result of the patch?
Looking at most of the Specweb profiles and networking
in general, and because so much here depends on which cpu
code gets run on, and cache behaviour, I'd say youre going
to get a lot of variance...
thanks,
Nivedita
Quoting Dave Hansen <haveblue@us.ibm.com>:
[snip]
> First of all, the patch doesn't apply at all against the current
> bitkeeper tree. I can post the exact one I used if you like.
>
> I tried this under our Specweb99 setup. Here's a snippet of
> readprofile with, then without the patch:
>
> 8788 __kfree_skb
> 8970 mod_timer
> 9095 file_read_actor
> 10778 alloc_skb
> 10905 skb_clone
> 11368 e1000_clean_tx_irq
> 13595 e1000_intr
> 18367 csum_partial_copy_generic
> 27848 e1000_xmit_frame
> 225838 poll_idle
> 623160 total 0.4107
>
> alloc:free ratio: 1.226
> (__kfree_skb+alloc_skb)/total = 3.14%
>
> 4535 alloc_skb
> 4559 do_tcp_sendpages
> 4596 e1000_clean_rx_irq
> 4847 dev_queue_xmit
> 5020 tcp_clean_rtx_queue
> 5155 batch_entropy_store
> 5165 kmalloc
> 5309 tcp_transmit_skb
> 6060 do_schedule
> 6138 qdisc_restart
> 6235 tcp_v4_rcv
> 6393 kfree
> 6787 do_gettimeofday
> 7089 __d_lookup
> 7810 ip_queue_xmit
> 8303 skb_clone
> 8858 file_read_actor
> 8885 mod_timer
> 9375 .text.lock.namei
> 10267 .text.lock.dec_and_lock
> 10936 e1000_clean_tx_irq
> 13001 __kfree_skb
> 13322 skb_release_data
> 13562 e1000_intr
> 18099 csum_partial_copy_generic
> 27447 e1000_xmit_frame
> 225023 poll_idle
> 628695 total 0.4143
>
> alloc:free ratio: 0.348
> (__kfree_skb+alloc_skb)/total = 2.79%
>
> You can see the entire readprofile here:
>
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap
-08-22-2002-11.20.17/
>
http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap
-mala-08-22-2002-11.44.25/
> No, I don't know why I have so much idle time.
>
> --
> Dave Hansen
> haveblue@us.ibm.com
>
> _______________________________________________
> ibm-specweb99 mailing list
> ibm-specweb99@linux.ibm.com
> http://ltc.linux.ibm.com/mailman/listinfo/ibm-specweb99
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Lse-tech] Re: (RFC): SKB Initialization
2002-08-22 19:02 ` Dave Hansen
2002-08-22 20:58 ` [Ibm-specweb99] " Nivedita Singhvi
@ 2002-08-22 22:05 ` William Lee Irwin III
1 sibling, 0 replies; 6+ messages in thread
From: William Lee Irwin III @ 2002-08-22 22:05 UTC (permalink / raw)
To: Dave Hansen
Cc: Mala Anand, Benjamin LaHaise, alan, Bill Hartner, davem,
linux-kernel, lse-tech, lse-tech-admin
On Thu, Aug 22, 2002 at 12:02:27PM -0700, Dave Hansen wrote:
> You can see the entire readprofile here:
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-08-22-2002-11.20.17/
> http://www.sr71.net/~specweb99/run-specweb-100sec-2400-2.5.31-bk+4-kmap-mala-08-22-2002-11.44.25/
> No, I don't know why I have so much idle time.
Hmm, I found that tiobench was spending a lot of time idle due to
__wait_on_inode() and get_request_wait(). I bumped up the size of
the inode wait table to 1024 and the request queue size to 16384
and saw that most of them then spent their time stuck on ->i_sem
during the initial open of the file they were going to pound on
for the duration of the run.
I determined this by just ^C'ing with kgdb and backtracing various
"stuck" processes. I think various profiling patches might be able to
give you an idea of what people are going to sleep on too.
Cheers,
Bill
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Lse-tech] Re: (RFC): SKB Initialization
2002-08-22 17:22 [Lse-tech] Re: (RFC): SKB Initialization Mala Anand
2002-08-22 18:32 ` Benjamin LaHaise
2002-08-22 19:02 ` Dave Hansen
@ 2002-08-23 19:09 ` Bill Hartner
2 siblings, 0 replies; 6+ messages in thread
From: Bill Hartner @ 2002-08-23 19:09 UTC (permalink / raw)
To: Mala Anand
Cc: Benjamin LaHaise, alan, Bill Hartner, davem, linux-kernel,
lse-tech, lse-tech-admin
Mala Anand wrote:
>
> Baseline 2.5.25
> ----------------
> alloc/free average cycles
> -------------------------
> Runs: 1st 2nd 3rd
>
> CPU0: 337/1163 336/1132 304/1100
> CPU1: 318/1164 309/1153 311/1127
>
> 2.5.25+skbinit patch
> --------------------
>
> alloc/free average cycles
> -------------------------
> Runs: 1st 2nd 3rd
>
> CPU0: 447/1015 580/846 402/905
> CPU1: 419/1003 383/915 547/856
>
> The above figures indicate that the cycles spent in alloc_skb and
> __kfree_skb have gained 5% in the patch case. However if you
> take the absolute cycles and average them for the three runs it
> comes around 145 cycles saving that is close to what I posted earlier
> by measuring just the changed code. As the scope of the code measured
> widens the percentage improvement comes down.
Measuring just the initialization code yielded a reduction of 156 cycles.
Measuring alloc_skb and __kfree_skb yielded a reduction of 145 cycles.
This was on a 2 CPU system.
The worst case scenario would be allocating the skb header on one
CPU then freeing it on another CPU. The best case would be doing
all of the allocs and frees on one CPU.
You can use process/irq affinity to create both of these cases.
Can you measure these ?
Bill
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2002-08-23 19:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2002-08-22 17:22 [Lse-tech] Re: (RFC): SKB Initialization Mala Anand
2002-08-22 18:32 ` Benjamin LaHaise
2002-08-22 19:02 ` Dave Hansen
2002-08-22 20:58 ` [Ibm-specweb99] " Nivedita Singhvi
2002-08-22 22:05 ` William Lee Irwin III
2002-08-23 19:09 ` Bill Hartner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®