Re: [PATCH 1/2]percpu: introduce read mostly percpu API

[Eric Dumazet <eric.dumazet@gmail.com>]

Le mercredi 20 octobre 2010 à 11:07 +0800, Shaohua Li a écrit :
> Add a new readmostly percpu section and api, next patch will use it.
> 
> Signed-off-by: Shaohua Li <shaohua.li@intel.com>
> ---


Could you precisely describe why grouping together read mostly percpu
variables is a win ? Especially when you add in your next patch a single
variable ?

>  include/asm-generic/vmlinux.lds.h |    4 ++++
>  include/linux/percpu-defs.h       |    9 +++++++++
>  2 files changed, 13 insertions(+)
> 
> Index: linux/include/asm-generic/vmlinux.lds.h
> ===================================================================
> --- linux.orig/include/asm-generic/vmlinux.lds.h	2010-10-20 09:32:52.000000000 +0800
> +++ linux/include/asm-generic/vmlinux.lds.h	2010-10-20 10:03:38.000000000 +0800
> @@ -677,6 +677,8 @@
>  				- LOAD_OFFSET) {			\
>  		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
>  		*(.data..percpu..first)					\
> +		. = ALIGN(PAGE_SIZE);					\
> +		*(.data..percpu..readmostly)				\
>  		*(.data..percpu..page_aligned)				\
>  		*(.data..percpu)					\
>  		*(.data..percpu..shared_aligned)			\

So percpu..page_aligned is not any more aligned to a PAGE ? or we have a
big hole before it ? Hmm....

Maybe you should put first data..percpu..page_aligned, then align to one
cache line (L1_CACHE_BYTES), then data..percpu..readmostly, so that hole
is small.

We should take care of not introducing too much holes in percpu zone.
Maybe using a new subzone ".data..percpu..small_objects" to put small
objects in it.

nm -v vmlinux | grep -10 sockets_in_use   # select one random part 

0000000000011e0c d cpu_min_freq
0000000000011e10 d cpu_cur_freq
0000000000011e14 d cpu_set_freq
0000000000011e18 d cpu_is_managed
0000000000011e20 d od_cpu_dbs_info
0000000000011fa0 d cs_cpu_dbs_info
00000000000120f0 d cpufreq_show_table
0000000000012100 D cpuidle_devices
0000000000012120 d ladder_devices
0000000000012200 d menu_devices       
00000000000122c0 d sockets_in_use     # object_size = 4 , hole = 28bytes
00000000000122e0 d prot_inuse         
00000000000123e0 D nf_conntrack_untracked
0000000000012560 d rt_cache_stat
00000000000125a0 d ipv4_cookie_scratch
0000000000012740 D init_tss
0000000000014a00 D irq_stat
0000000000014a40 D cpu_info
0000000000014b00 d hv_clock
0000000000014b40 D cpu_tlbstate
0000000000014b80 d runqueues

We can see many holes because of 2^5 alignments of
individual .o  .data..percpu sections.

find . -name "*.o"|xargs objdump -h|grep percpu

Linker promotes a section alignment from natural alignment to 2^5 as
soon as the size reaches 2^5

For example in net/ipv4/route.o, we have a per_cpu structure
(rt_cache_stat), that is an array of 16 integers. The natural alignement
should be 4 (alignof(int)), but we get :

# objdump -h net/ipv4/route.o|grep percpu
 19 .data..percpu 00000040  0000000000000000  0000000000000000  00007a80  2**5

For a section replicated N times, this really is a concern.

Thanks !