Network throughput is reducedof in total bandwidth in Mips.

Network throughput is reducedof in total bandwidth in Mips.

[Sadasivan Shaiju]

HI,

When  using iperf   with  small  buffers  there  is a performance
degradation   for  3.10  kernel   from  2.6.32  kernel .   Using
profiling  tools   I am getting  the  following  data.

2.6.32   kerne  data
........
samples  %        symbol name
160321   39.1122  r4k_wait
43731    10.6687  __do_softirq
27547     6.7204  cvm_oct_napi_poll_68
16569     4.0422  _raw_spin_unlock_irqrestore
12537     3.0586  cvm_oct_xmit
7362      1.7960  kfree
5916      1.4433  tcp_ack
5480      1.3369  net_rx_action
5353      1.3059  __copy_user_common
5097      1.2435  put_page
4447      1.0849  __kmalloc
4361      1.0639  kmem_cache_alloc
4359      1.0634  skb_segment
4309      1.0512  tcp_v4_rcv
4120      1.0051  __inet_lookup_established
4016      0.9798  ip_route_input
4010      0.9783  napi_complete
3766      0.9188  tcp_write_xmit
3297      0.8043  free_hot_cold_page
3189      0.7780  kmem_cache_free

3.10  kernel  data
.......
samples  %        symbol name
186       8.6431  __do_softirq
127       5.9015  cvm_oct_napi_poll_68
80        3.7175  tcp_sendmsg
76        3.5316  tcp_write_xmit
72        3.3457  cvm_oct_xmit_lockless
71        3.2993  _raw_spin_lock_bh
48        2.2305  tcp_release_cb
47        2.1840  release_sock
44        2.0446  add_preempt_count
44        2.0446  tcp_transmit_skb
34        1.5799  skb_split
31        1.4405  sub_preempt_count
27        1.2546  dev_hard_start_xmit
25        1.1617  __srcu_read_lock
25        1.1617  skb_release_data
24        1.1152  tcp_ack
23        1.0688  __bzero
23        1.0688  _raw_spin_unlock_irqrestore
23        1.0688  ip_finish_output
22        1.0223  __copy_skb_header
21        0.9758  __kfree_skb
21        0.9758  ip_queue_xmit

Looks  like  mainly the  delay  is  causing   by  tcp_sendmsg()  and
tcp_write_xmit()   from  the  stack .   However   if  the  buffer  is
increased   to  default  (128k)  there  is  no  performance
Degradation .  This is  happening  only    for  small  buffers.

When  iperf is    used   as  below    the  bandwidth  on  3.10 kernel  is
only  around   600Mbps    compared   to  900Mbps   in  2.6.32  kernel.

./iperf3 -c 10.162.103.131 -l 1500

When  iperf  is  ran  with  default  buffers  as  follows  the  bandwidth
is same  in 3.10  kernel  and 2.6.32  kernel.

./iperf3 -c 10.162.103.131

Please  let me know  if  there is any fixes  available for  this .

Re: Network throughput is reducedof in total bandwidth in Mips.

[Eric Dumazet]

On Mon, 2014-11-24 at 14:31 -0800, Sadasivan Shaiju wrote:
> HI,
> 
> When  using iperf   with  small  buffers  there  is a performance
> degradation   for  3.10  kernel   from  2.6.32  kernel .   Using
> profiling  tools   I am getting  the  following  data.
> 
> 2.6.32   kerne  data
> ........
> samples  %        symbol name
> 160321   39.1122  r4k_wait
> 43731    10.6687  __do_softirq
> 27547     6.7204  cvm_oct_napi_poll_68
> 16569     4.0422  _raw_spin_unlock_irqrestore
> 12537     3.0586  cvm_oct_xmit
> 7362      1.7960  kfree
> 5916      1.4433  tcp_ack
> 5480      1.3369  net_rx_action
> 5353      1.3059  __copy_user_common
> 5097      1.2435  put_page
> 4447      1.0849  __kmalloc
> 4361      1.0639  kmem_cache_alloc
> 4359      1.0634  skb_segment
> 4309      1.0512  tcp_v4_rcv
> 4120      1.0051  __inet_lookup_established
> 4016      0.9798  ip_route_input
> 4010      0.9783  napi_complete
> 3766      0.9188  tcp_write_xmit
> 3297      0.8043  free_hot_cold_page
> 3189      0.7780  kmem_cache_free
> 
> 3.10  kernel  data
> .......
> samples  %        symbol name
> 186       8.6431  __do_softirq
> 127       5.9015  cvm_oct_napi_poll_68
> 80        3.7175  tcp_sendmsg
> 76        3.5316  tcp_write_xmit
> 72        3.3457  cvm_oct_xmit_lockless
> 71        3.2993  _raw_spin_lock_bh
> 48        2.2305  tcp_release_cb
> 47        2.1840  release_sock
> 44        2.0446  add_preempt_count
> 44        2.0446  tcp_transmit_skb
> 34        1.5799  skb_split
> 31        1.4405  sub_preempt_count
> 27        1.2546  dev_hard_start_xmit
> 25        1.1617  __srcu_read_lock
> 25        1.1617  skb_release_data
> 24        1.1152  tcp_ack
> 23        1.0688  __bzero
> 23        1.0688  _raw_spin_unlock_irqrestore
> 23        1.0688  ip_finish_output
> 22        1.0223  __copy_skb_header
> 21        0.9758  __kfree_skb
> 21        0.9758  ip_queue_xmit
> 
> Looks  like  mainly the  delay  is  causing   by  tcp_sendmsg()  and
> tcp_write_xmit()   from  the  stack .   However   if  the  buffer  is
> increased   to  default  (128k)  there  is  no  performance
> Degradation .  This is  happening  only    for  small  buffers.
> 
> When  iperf is    used   as  below    the  bandwidth  on  3.10 kernel  is
> only  around   600Mbps    compared   to  900Mbps   in  2.6.32  kernel.
> 
> ./iperf3 -c 10.162.103.131 -l 1500
> 
> When  iperf  is  ran  with  default  buffers  as  follows  the  bandwidth
> is same  in 3.10  kernel  and 2.6.32  kernel.
> 
> ./iperf3 -c 10.162.103.131
> 
> Please  let me know  if  there is any fixes  available for  this .
> --

CC netdev, where you can actually find some help ;)

Quite frankly, there are probably some reasons Octeon ethernet driver is
still in staging.

1) This REUSE_SKBUFFS_WITHOUT_FREE kludge is quite horrible... Dont know
if it was used in 2.6.32 and still used in 3.10 on your config.

2) TX completions seem to be delayed by undefined amount of time.
   TCP Small Queues was added in 3.6 so this driver problem could
explain a lower throughput.

You could try adding skb_orphan() into this driver...

diff --git a/drivers/staging/octeon/ethernet-tx.c b/drivers/staging/octeon/ethernet-tx.c
index 4e54d854021955868e2c602ac829b9b217960efd..85f42f32419aabcda28f1b2aa44e396a462b3fb2 100644
--- a/drivers/staging/octeon/ethernet-tx.c
+++ b/drivers/staging/octeon/ethernet-tx.c
@@ -175,6 +175,7 @@ int cvm_oct_xmit(struct sk_buff *skb, struct net_device *dev)
 	 */
 	prefetch(priv);
 
+	skb_orphan(skb);
 	/*
 	 * The check on CVMX_PKO_QUEUES_PER_PORT_* is designed to
 	 * completely remove "qos" in the event neither interface