Re: rcu_sched_state detected stall on CPU issue

mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed

* Re: rcu_sched_state detected stall on CPU issue
       [not found] <20110316101820.GA17540@linux.vnet.ibm.com>
@ 2011-03-17  0:18 ` KOSAKI Motohiro
  2011-03-17  2:25   ` Paul E. McKenney
  0 siblings, 1 reply; 5+ messages in thread
From: KOSAKI Motohiro @ 2011-03-17  0:18 UTC (permalink / raw)
  To: paulmck; +Cc: kosaki.motohiro, mel, akpm, -c, linux-kernel

Hi Paul,

> Hello!
> 
> It looks like Preeti found a way to loop in kswapd with preemption
> disabled, please see attached dmesg.   This is using 2.6.35-45.fc14.
> 
> Any thoughts or recent patches that might help?

We fixed a kswapd infinite loop issue when memory is fragmented at 2.6.38.
Could you please try following commits? (all are mm/vmscan.c change)

commit dc83edd941f412e938841b4989be24aa288a1aa6
commit 355b09c47a0cbb73b3e65a57c03f157f2e7ddb0b
commit 4d40502ea580c35414a1466d86f96484910ebaec
commit 0abdee2bd4118366c62349a304f81537be69af33
commit 1741c87757448cedd03224f01586504f9256415d
commit 9950474883e027e6e728cbcff25f7f2bf0c96530


And, your attached file and your quoted mail don't have /proc/zoneinfo
information. then I have no way to confirn his environment is heavy
fragmented or not. If possible, could you please get /proc/zoneinfo
info?





^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: rcu_sched_state detected stall on CPU issue
  2011-03-17  0:18 ` rcu_sched_state detected stall on CPU issue KOSAKI Motohiro
@ 2011-03-17  2:25   ` Paul E. McKenney
  2011-03-17 14:11     ` Preeti Khurana
  0 siblings, 1 reply; 5+ messages in thread
From: Paul E. McKenney @ 2011-03-17  2:25 UTC (permalink / raw)
  To: KOSAKI Motohiro; +Cc: mel, akpm, linux-kernel, Preeti.Khurana

On Thu, Mar 17, 2011 at 09:18:19AM +0900, KOSAKI Motohiro wrote:
> Hi Paul,
> 
> > Hello!
> > 
> > It looks like Preeti found a way to loop in kswapd with preemption
> > disabled, please see attached dmesg.   This is using 2.6.35-45.fc14.
> > 
> > Any thoughts or recent patches that might help?
> 
> We fixed a kswapd infinite loop issue when memory is fragmented at 2.6.38.
> Could you please try following commits? (all are mm/vmscan.c change)
> 
> commit dc83edd941f412e938841b4989be24aa288a1aa6
> commit 355b09c47a0cbb73b3e65a57c03f157f2e7ddb0b
> commit 4d40502ea580c35414a1466d86f96484910ebaec
> commit 0abdee2bd4118366c62349a304f81537be69af33
> commit 1741c87757448cedd03224f01586504f9256415d
> commit 9950474883e027e6e728cbcff25f7f2bf0c96530

Thank you, Kosaki!

> And, your attached file and your quoted mail don't have /proc/zoneinfo
> information. then I have no way to confirn his environment is heavy
> fragmented or not. If possible, could you please get /proc/zoneinfo
> info?

Preeti, could you please send this info and try out the above patches?

							Thanx, Paul

^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: rcu_sched_state detected stall on CPU issue
  2011-03-17  2:25   ` Paul E. McKenney
@ 2011-03-17 14:11     ` Preeti Khurana
  2011-03-22  4:44       ` Preeti Khurana
  0 siblings, 1 reply; 5+ messages in thread
From: Preeti Khurana @ 2011-03-17 14:11 UTC (permalink / raw)
  To: paulmck, KOSAKI Motohiro; +Cc: mel, akpm, linux-kernel



> > > Any thoughts or recent patches that might help?
> >
> > We fixed a kswapd infinite loop issue when memory is fragmented at
> 2.6.38.
> > Could you please try following commits? (all are mm/vmscan.c change)
> >
> > commit dc83edd941f412e938841b4989be24aa288a1aa6
> > commit 355b09c47a0cbb73b3e65a57c03f157f2e7ddb0b
> > commit 4d40502ea580c35414a1466d86f96484910ebaec
> > commit 0abdee2bd4118366c62349a304f81537be69af33
> > commit 1741c87757448cedd03224f01586504f9256415d
> > commit 9950474883e027e6e728cbcff25f7f2bf0c96530
> 
> Thank you, Kosaki!

    I will try patching with the commits & test the same, though patches don't cleanly fit because they are on top of the changes which are not there in 2.6.35  ( have come in 2.6.37), so it's a lttle challenge.
Meanwhile, I  am trying out the changes specified in thread https://lkml.org/lkml/2011/1/6/131 because I think that also might issue.

> 
> > And, your attached file and your quoted mail don't have /proc/zoneinfo
> > information. then I have no way to confirn his environment is heavy
> > fragmented or not. If possible, could you please get /proc/zoneinfo
> > info?
> 
> Preeti, could you please send this info and try out the above patches?

  Output of /proc/zoneinfo at the time of stall :


Node 0, zone   Normal
  pages free     7484
        min      4982
        low      6227
        high     7473
        scanned  0
        spanned  12582912
        present  12410880
    nr_free_pages 7422
    nr_inactive_anon 42096
    nr_active_anon 882918
    nr_inactive_file 5635172
    nr_active_file 5635111
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 924987
    nr_mapped    1208
    nr_file_pages 11270146
    nr_dirty     49494
    nr_writeback 0
    nr_slab_reclaimable 169648
    nr_slab_unreclaimable 4202
    nr_page_table_pages 2271
    nr_kernel_stack 295
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 291
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     27
    numa_hit     225959693
    numa_miss    20838462
    numa_foreign 69636123
    numa_interleave 27282
    numa_local   225959685
    numa_other   20838470
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 172
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 1
              count: 165
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 2
              count: 60
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 3
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 4
              count: 63
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 5
              count: 94
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 6
              count: 63
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 7
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 8
              count: 55
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 9
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 10
              count: 41
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 11
              count: 112
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 12
              count: 75
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 13
              count: 18
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 14
              count: 171
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 15
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         12845056
  inactive_ratio:    21
Node 1, zone      DMA
  pages free     3979
        min      1
        low      1
        high     1
        scanned  0
        spanned  4095
        present  3943
    nr_free_pages 3979
    nr_inactive_anon 0
    nr_active_anon 0
    nr_inactive_file 0
    nr_active_file 0
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 0
    nr_mapped    0
    nr_file_pages 0
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 0
    nr_slab_unreclaimable 0
    nr_page_table_pages 0
    nr_kernel_stack 0
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    numa_hit     0
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   0
    numa_other   0
        protection: (0, 2987, 48437, 48437)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 3
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 4
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 5
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 6
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 7
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 8
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 9
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 10
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 11
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 12
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 13
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 14
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
    cpu: 15
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 10
  all_unreclaimable: 1
  prev_priority:     12
  start_pfn:         1
  inactive_ratio:    1
Node 1, zone    DMA32
  pages free     45968
        min      307
        low      383
        high     460
        scanned  0
        spanned  1044480
        present  764849
    nr_free_pages 45968
    nr_inactive_anon 9708
    nr_active_anon 38161
    nr_inactive_file 291211
    nr_active_file 291169
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 47869
    nr_mapped    50
    nr_file_pages 582385
    nr_dirty     1897
    nr_writeback 0
    nr_slab_reclaimable 81872
    nr_slab_unreclaimable 115
    nr_page_table_pages 98
    nr_kernel_stack 2
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 116
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    numa_hit     6109982
    numa_miss    9113118
    numa_foreign 0
    numa_interleave 0
    numa_local   6109665
    numa_other   9113435
        protection: (0, 0, 45450, 45450)
  pagesets
    cpu: 0
              count: 65
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 1
              count: 181
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 2
              count: 46
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 3
              count: 27
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 4
              count: 78
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 5
              count: 33
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 6
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 7
              count: 57
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 8
              count: 90
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 9
              count: 166
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 10
              count: 65
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 11
              count: 25
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 12
              count: 29
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 13
              count: 29
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 14
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 60
    cpu: 15
              count: 77
              high:  186
              batch: 31
  vm stats threshold: 60
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         4096
  inactive_ratio:    4
Node 1, zone   Normal
  pages free     6028
        min      4670
        low      5837
        high     7005
        scanned  0
        spanned  11796480
        present  11635200
    nr_free_pages 6028
    nr_inactive_anon 27762
    nr_active_anon 554870
    nr_inactive_file 5417543
    nr_active_file 5416505
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 582591
    nr_mapped    2512
    nr_file_pages 10834147
    nr_dirty     24700
    nr_writeback 0
    nr_slab_reclaimable 169345
    nr_slab_unreclaimable 1736
    nr_page_table_pages 1640
    nr_kernel_stack 32
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 143
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     45
    numa_hit     154438673
    numa_miss    60523005
    numa_foreign 20838462
    numa_interleave 27227
    numa_local   154411416
    numa_other   60550262
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 149
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 1
              count: 154
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 2
              count: 43
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 3
              count: 175
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 4
              count: 46
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 5
              count: 57
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 6
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 7
              count: 104
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 8
              count: 92
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 9
              count: 21
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 10
              count: 147
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 11
              count: 168
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 12
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
             count: 92
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 9
              count: 21
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 10
              count: 147
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 11
              count: 168
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 12
              count: 0
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 13
              count: 46
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 14
              count: 146
              high:  186
              batch: 31
  vm stats threshold: 100
    cpu: 15
              count: 180
              high:  186
              batch: 31
  vm stats threshold: 100
  all_unreclaimable: 0
  prev_priority:     12
  start_pfn:         1048576
  inactive_ratio:    20


^ permalink raw reply	[flat|nested] 5+ messages in thread

* RE: rcu_sched_state detected stall on CPU issue
  2011-03-17 14:11     ` Preeti Khurana
@ 2011-03-22  4:44       ` Preeti Khurana
  2011-03-22  6:32         ` Paul E. McKenney
  0 siblings, 1 reply; 5+ messages in thread
From: Preeti Khurana @ 2011-03-22  4:44 UTC (permalink / raw)
  To: Preeti Khurana, paulmck, KOSAKI Motohiro; +Cc: mel, akpm, linux-kernel



>     I will try patching with the commits & test the same, though patches don't
> cleanly fit because they are on top of the changes which are not there in
> 2.6.35  ( have come in 2.6.37), so it's a lttle challenge.
> Meanwhile, I  am trying out the changes specified in thread
> https://lkml.org/lkml/2011/1/6/131 because I think that also might issue.


	The patch at  https://lkml.org/lkml/2011/1/6/131 (commit 88f5acf88ae6a9778f6d25d0d5d7ec2d57764a97) has solved the issue.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: rcu_sched_state detected stall on CPU issue
  2011-03-22  4:44       ` Preeti Khurana
@ 2011-03-22  6:32         ` Paul E. McKenney
  0 siblings, 0 replies; 5+ messages in thread
From: Paul E. McKenney @ 2011-03-22  6:32 UTC (permalink / raw)
  To: Preeti Khurana; +Cc: KOSAKI Motohiro, mel, akpm, linux-kernel

On Tue, Mar 22, 2011 at 04:44:46AM +0000, Preeti Khurana wrote:
> 
> 
> >     I will try patching with the commits & test the same, though patches don't
> > cleanly fit because they are on top of the changes which are not there in
> > 2.6.35  ( have come in 2.6.37), so it's a lttle challenge.
> > Meanwhile, I  am trying out the changes specified in thread
> > https://lkml.org/lkml/2011/1/6/131 because I think that also might issue.
> 
> 
> 	The patch at  https://lkml.org/lkml/2011/1/6/131 (commit 88f5acf88ae6a9778f6d25d0d5d7ec2d57764a97) has solved the issue.

Glad it worked for you!

							Thanx, Paul

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2011-03-22  6:32 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20110316101820.GA17540@linux.vnet.ibm.com>
2011-03-17  0:18 ` rcu_sched_state detected stall on CPU issue KOSAKI Motohiro
2011-03-17  2:25   ` Paul E. McKenney
2011-03-17 14:11     ` Preeti Khurana
2011-03-22  4:44       ` Preeti Khurana
2011-03-22  6:32         ` Paul E. McKenney

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

Powered by JetHome