2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]


Hi all,

As the subject says, is this a know issue? I have been tracking the growth at
one second intervals and it looks like this:

512 (67716608) Slab:             606744 kB
512 (67717120) Slab:             606752 kB
512 (67717632) Slab:             606756 kB
0 (67718144) Slab:             606756 kB
512 (67718144) Slab:             606760 kB
1024 (67718656) Slab:             606768 kB
512 (67719680) Slab:             606772 kB
512 (67720192) Slab:             606776 kB
0 (67720704) Slab:             606776 kB
512 (67720704) Slab:             606780 kB

First fiels is the diff in numobj, in parentheses is actual numobj and last is
the total slab size from /proc/meminfo.

kmalloc-8         68044800 68044800      8  512    1 : tunables    0    0    0
: slabdata 132900 132900      0

dm-io slab also seems large, don't know... :
dm_io              75924 147288     40  102    1 : tunables    0    0    0 :
slabdata   1444   1444      0

Some other info about the system/kernel is attached.

Tvrtko



Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

[-- Attachment #2: lsmod --]
[-- Type: text/plain, Size: 3728 bytes --]

Module                  Size  Used by
bridge                 49385  0      
stp                     1903  1 bridge
llc                     4749  2 bridge,stp
tun                    12712  4           
nls_utf8                1325  1           
cifs                  249220  2           
power_meter             8721  0           
dell_rbu                5994  0           
nfsd                  103640  7           
exportfs                3808  1 nfsd      
snd_pcm_oss            35939  0           
snd_mixer_oss          14873  1 snd_pcm_oss
snd_seq                51077  0            
snd_seq_device          5913  1 snd_seq    
edd                     8495  0            
nfs                   162373  3            
lockd                  70096  2 nfsd,nfs   
nfs_acl                 2517  2 nfsd,nfs   
sunrpc                206266  27 nfsd,nfs,lockd,nfs_acl
radeon                704591  2                        
ttm                    52379  1 radeon                 
drm_kms_helper         27568  1 radeon                 
drm                   178090  5 radeon,ttm,drm_kms_helper
i2c_algo_bit            5267  1 radeon                   
af_packet              18575  0                          
cpufreq_conservative    10990  0                         
cpufreq_userspace       2240  0                          
cpufreq_powersave       1002  0                          
acpi_cpufreq            6447  0                          
fuse                   61697  1                          
loop                   15435  0                          
snd_hda_codec_realtek   281691  1                        
snd_hda_intel          23948  2                          
snd_hda_codec          70603  2 snd_hda_codec_realtek,snd_hda_intel
dcdbas                  6609  0                                    
snd_hwdep               6250  1 snd_hda_codec                      
snd_pcm                71241  3 snd_pcm_oss,snd_hda_intel,snd_hda_codec
iTCO_wdt                9926  0                                        
snd_timer              19944  2 snd_seq,snd_pcm                        
kvm_intel              42483  3                                        
tg3                   120566  0                                        
iTCO_vendor_support     2563  1 iTCO_wdt                               
broadcom                6767  0
snd                    60878  14 snd_pcm_oss,snd_mixer_oss,snd_seq,snd_seq_device,snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_pcm,snd_timer
kvm                   179805  1 kvm_intel
button                  5413  0
snd_page_alloc          7637  2 snd_hda_intel,snd_pcm
sr_mod                 12762  0
libphy                 20434  2 tg3,broadcom
intel_agp              27929  0
pcspkr                  1870  0
sg                     25183  0
cdrom                  36645  1 sr_mod
ext4                  244065  9
jbd2                   58645  1 ext4
crc16                   1471  1 ext4
linear                  4294  0
raid456                50446  0
async_raid6_recov       5589  1 raid456
async_pq                3529  2 raid456,async_raid6_recov
raid6_pq               79292  2 async_raid6_recov,async_pq
async_xor               2658  3 raid456,async_raid6_recov,async_pq
xor                     4672  1 async_xor
async_memcpy            1490  2 raid456,async_raid6_recov
async_tx                2137  5 raid456,async_raid6_recov,async_pq,async_xor,async_memcpy
raid1                  21608  0
raid0                   6118  1
dm_snapshot            28785  0
dm_mod                 64837  25 dm_snapshot
fan                     3519  0
processor              31361  1 acpi_cpufreq
thermal                13003  0
thermal_sys            15131  3 fan,processor,thermal

[-- Attachment #3: dmesg.gz --]
[-- Type: application/x-gzip, Size: 17684 bytes --]

[-- Attachment #4: config.gz --]
[-- Type: application/x-gzip, Size: 14670 bytes --]

[-- Attachment #5: slabinfo --]
[-- Type: application/x-zerosize, Size: 9535 bytes --]

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Pekka Enberg]

Hi Tvrtko,

On Wed, May 12, 2010 at 1:21 PM, Tvrtko Ursulin
<tvrtko.ursulin@sophos.com> wrote:
> As the subject says, is this a know issue? I have been tracking the growth at
> one second intervals and it looks like this:
>
> 512 (67716608) Slab:             606744 kB
> 512 (67717120) Slab:             606752 kB
> 512 (67717632) Slab:             606756 kB
> 0 (67718144) Slab:             606756 kB
> 512 (67718144) Slab:             606760 kB
> 1024 (67718656) Slab:             606768 kB
> 512 (67719680) Slab:             606772 kB
> 512 (67720192) Slab:             606776 kB
> 0 (67720704) Slab:             606776 kB
> 512 (67720704) Slab:             606780 kB
>
> First fiels is the diff in numobj, in parentheses is actual numobj and last is
> the total slab size from /proc/meminfo.
>
> kmalloc-8         68044800 68044800      8  512    1 : tunables    0    0    0
> : slabdata 132900 132900      0
>
> dm-io slab also seems large, don't know... :
> dm_io              75924 147288     40  102    1 : tunables    0    0    0 :
> slabdata   1444   1444      0
>
> Some other info about the system/kernel is attached.

[snip]

You might want to try out the built-in kernel memory leak detector.
See Documentation/kmemleak.txt for details how to enable it and use
it.

                                   Pekka

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

On Wednesday 12 May 2010 14:00:51 Pekka Enberg wrote:
> [snip]
>
> You might want to try out the built-in kernel memory leak detector.
> See Documentation/kmemleak.txt for details how to enable it and use
> it.

Thanks for the suggestion, I will have to try it out once but not this time :)
because 2.6.34-rc7 seems to be fine, or in other words not leaking.

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Catalin Marinas]

Tvrtko Ursulin <tvrtko.ursulin@sophos.com> wrote:
> On Wednesday 12 May 2010 14:00:51 Pekka Enberg wrote:
>> [snip]
>>
>> You might want to try out the built-in kernel memory leak detector.
>> See Documentation/kmemleak.txt for details how to enable it and use
>> it.
>
> Thanks for the suggestion, I will have to try it out once but not this time :)
> because 2.6.34-rc7 seems to be fine, or in other words not leaking.

You could just enable CONFIG_DEBUG_KMEMLEAK on 2.6.33, mount the debugfs
and do a 'cat /sys/kernel/debug/kmemleak' after a half-hour or so.

-- 
Catalin

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

On Thursday 13 May 2010 13:29:49 Catalin Marinas wrote:
> Tvrtko Ursulin <tvrtko.ursulin@sophos.com> wrote:
> > On Wednesday 12 May 2010 14:00:51 Pekka Enberg wrote:
> >> [snip]
> >>
> >> You might want to try out the built-in kernel memory leak detector.
> >> See Documentation/kmemleak.txt for details how to enable it and use
> >> it.
> >
> > Thanks for the suggestion, I will have to try it out once but not this
> > time :) because 2.6.34-rc7 seems to be fine, or in other words not
> > leaking.
>
> You could just enable CONFIG_DEBUG_KMEMLEAK on 2.6.33, mount the debugfs
> and do a 'cat /sys/kernel/debug/kmemleak' after a half-hour or so.

I could do it if radeon/drm guys would be interested in those results? (Given
how 2.6.34-rc7 is not leaking.)

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[David Rientjes]

On Thu, 13 May 2010, Tvrtko Ursulin wrote:

> > You could just enable CONFIG_DEBUG_KMEMLEAK on 2.6.33, mount the debugfs
> > and do a 'cat /sys/kernel/debug/kmemleak' after a half-hour or so.
> 
> I could do it if radeon/drm guys would be interested in those results? (Given
> how 2.6.34-rc7 is not leaking.)
> 

Did you get a chance to find out if kmemleak was able to identify the 
source of this problem?

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

On Wednesday 26 May 2010 10:21:36 David Rientjes wrote:
> On Thu, 13 May 2010, Tvrtko Ursulin wrote:
> > > You could just enable CONFIG_DEBUG_KMEMLEAK on 2.6.33, mount the
> > > debugfs and do a 'cat /sys/kernel/debug/kmemleak' after a half-hour or
> > > so.
> >
> > I could do it if radeon/drm guys would be interested in those results?
> > (Given how 2.6.34-rc7 is not leaking.)
>
> Did you get a chance to find out if kmemleak was able to identify the
> source of this problem?

Well no, I asked DRM/radeon guys to explicitly say whether it would be useful,
otherwise it is unfortunately sufficiently low on my priority list that it
will probably never happen.

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

On Wednesday 12 May 2010 11:21:48 Tvrtko Ursulin wrote:
> Hi all,
>
> As the subject says, is this a know issue? I have been tracking the growth
>  at one second intervals and it looks like this:
>
> 512 (67716608) Slab:             606744 kB
> 512 (67717120) Slab:             606752 kB
> 512 (67717632) Slab:             606756 kB
> 0 (67718144) Slab:             606756 kB
> 512 (67718144) Slab:             606760 kB
> 1024 (67718656) Slab:             606768 kB
> 512 (67719680) Slab:             606772 kB
> 512 (67720192) Slab:             606776 kB
> 0 (67720704) Slab:             606776 kB
> 512 (67720704) Slab:             606780 kB
>
> First fiels is the diff in numobj, in parentheses is actual numobj and last
>  is the total slab size from /proc/meminfo.
>
> kmalloc-8         68044800 68044800      8  512    1 : tunables    0    0
>   0
>
> : slabdata 132900 132900      0
>
> dm-io slab also seems large, don't know... :
> dm_io              75924 147288     40  102    1 : tunables    0    0    0
>  : slabdata   1444   1444      0
>
> Some other info about the system/kernel is attached.

I think it is DRM (radeon) related, leak stopped when I closed all X programs.
I am compiling 2.6.33.3 right now and will soon reboot into it.

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

On Wednesday 12 May 2010 11:38:05 Tvrtko Ursulin wrote:

[snip]
> I think it is DRM (radeon) related, leak stopped when I closed all X
>  programs. I am compiling 2.6.33.3 right now and will soon reboot into it.

Leak is still present in 2.6.33.3 - the more GUI activity the more it leaks.
So the ~512 objects per second was with xosview running, but moving windows or
browsing through menus can leak a lot more (per second).

Is version of userspace drm library interesting? It is 2.4.20
(libdrm-2.4.20-12.1.x86_64).

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Jerome Glisse]

On Wed, May 12, 2010 at 11:50:41AM +0100, Tvrtko Ursulin wrote:
> On Wednesday 12 May 2010 11:38:05 Tvrtko Ursulin wrote:
> 
> [snip]
> > I think it is DRM (radeon) related, leak stopped when I closed all X
> >  programs. I am compiling 2.6.33.3 right now and will soon reboot into it.
> 
> Leak is still present in 2.6.33.3 - the more GUI activity the more it leaks.
> So the ~512 objects per second was with xosview running, but moving windows or
> browsing through menus can leak a lot more (per second).
> 
> Is version of userspace drm library interesting? It is 2.4.20
> (libdrm-2.4.20-12.1.x86_64).
> 
> Tvrtko
> 

Do the number of object stay constant over longuer period of time
(30min, 1hour) ? If there was a leak i think we would have notice
it by now, some of my desktop are running 24h for several days with
radeon kms.

Cheers,
Jerome

Re: 2.6.33.2 kmalloc-8 slab leaks ~512 objects per second

[Tvrtko Ursulin]

[Snipped localdomain guy from Cc]

On Wednesday 12 May 2010 14:29:16 Jerome Glisse wrote:
> On Wed, May 12, 2010 at 11:50:41AM +0100, Tvrtko Ursulin wrote:
> > On Wednesday 12 May 2010 11:38:05 Tvrtko Ursulin wrote:
> >
> > [snip]
> >
> > > I think it is DRM (radeon) related, leak stopped when I closed all X
> > >  programs. I am compiling 2.6.33.3 right now and will soon reboot into
> > > it.
> >
> > Leak is still present in 2.6.33.3 - the more GUI activity the more it
> > leaks. So the ~512 objects per second was with xosview running, but
> > moving windows or browsing through menus can leak a lot more (per
> > second).
> >
> > Is version of userspace drm library interesting? It is 2.4.20
> > (libdrm-2.4.20-12.1.x86_64).
> >
> > Tvrtko
>
> Do the number of object stay constant over longuer period of time
> (30min, 1hour) ? If there was a leak i think we would have notice
> it by now, some of my desktop are running 24h for several days with
> radeon kms.

No, it grew constantly in presence of compositing activity under X by 512 8
byte objects per second or more, possibly multiples of 512 or very close.

If you haven't seen my original  post, kmalloc-8 cache after some days of
uptime looked like this:

kmalloc-8         68044800 68044800      8  512    1 : tunables    0    0    0
: slabdata 132900 132900      0

I am now running 2.6.34-rc7 which seems not to leak.

Tvrtko

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.