Oops with 2.6.15.3 on amd64

Oops with 2.6.15.3 on amd64

[Steinar H. Gunderson]

Hi,

My dual-core AMD64 machine (Athlon 64 X2 3800+) has been unstable (hanging
completely from the to time) since I ever got it, with various kernels (2.6.8
up to 2.6.17-rc1), even more so after the I/O load increased on it (earlier
it hung about once a month; now it's more once a day). This time, however, I
had a minicom logging everything on the serial console, so I have a
traceback:

[117109.540473] Unable to handle kernel paging request at ffff8000027a7c68 RIP:                        
[117109.545881] <ffffffff8015abe8>{isolate_lru_pages+77}                        
[117109.554498] PGD 0                                   
[117109.556897] Oops: 0002 [1] SMP 
[117109.560610] CPU 0              
[117109.563007] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_disk ide_cd cdrom ipv6 joydev evdev i2c_nforce2 snd_ac97_bus analog psmouse floppy parport_pc parport i2c_core soundcore gameport pcspkr serio_raw rtc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod ide_generic sd_mod sata_nv libata scsi_mod ehci_hcd generic forcedeth amd74xx ide_core ohci_hcd thermal processor fan unix
[117109.605544] Pid: 192, comm: kswapd0 Not tainted 2.6.15.3 #1 
[117109.612045] RIP: 0010:[<ffffffff8015abe8>] <ffffffff8015abe8>{isolate_lru_pages+77}
[117109.620752] RSP: 0018:ffff81007e729c58  EFLAGS: 00010086                           
[117109.627149] RAX: ffff8000027a7c68 RBX: 0000000000000020 RCX: 0000000000000018
[117109.635469] RDX: ffff81000000c618 RSI: ffff81000000c618 RDI: ffff81007e729e04
[117109.643796] RBP: ffff81000000c380 R08: ffff810001761ce0 R09: ffff81007e729de8
[117109.652086] R10: 0000000000000018 R11: 0000000000000019 R12: 0000000000000000
[117109.660388] R13: 000000000000000c R14: ffff81007e729e78 R15: 0000000000000000
[117109.668722] FS:  0000000000000000(0000) GS:ffffffff803a8800(0000) knlGS:0000000055844e60
[117109.678123] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b                           
[117109.684840] CR2: ffff8000027a7c68 CR3: 000000001791d000 CR4: 00000000000006e0
[117109.693134] Process kswapd0 (pid: 192, threadinfo ffff81007e728000, task ffff81007e724f60)
[117109.702739] Stack: ffff81000000c380 ffffffff8015b288 ffff810025a08000 0000000000000010    
[117109.711844]        0000000000000000 0000000000000041 0000002000000000 0000000000000020 
[117109.721158]        0000000000000000 0000000000000001                                   
[117109.727087] Call Trace:<ffffffff8015b288>{shrink_zone+1474} <ffffffff801d7c3d>{find_next_bit+90}
[117109.737326]        <ffffffff8810406f>{:mbcache:mb_cache_shrink_fn+41} <ffffffff801d63ba>{__up_read+19}
[117109.748145]        <ffffffff8015aa42>{shrink_slab+314} <ffffffff8015bcd0>{balance_pgdat+535}          
[117109.757953]        <ffffffff8029f279>{schedule+1} <ffffffff8015bf36>{kswapd+263}            
[117109.766566]        <ffffffff801420ac>{autoremove_wake_function+0} <ffffffff8010e66e>{child_rip+8}
[117109.776910]        <ffffffff8015be2f>{kswapd+0} <ffffffff8010e666>{child_rip+0}                  
[117109.785461]                                                                    
[117109.788162]        
[117109.788163] Code: 48 89 10 49 c7 40 08 00 02 20 00 49 c7 00 00 01 10 00 f0 41 
[117109.798435] RIP <ffffffff8015abe8>{isolate_lru_pages+77} RSP <ffff81007e729c58>
[117109.806956] CR2: ffff8000027a7c68                                              
[117109.810858]  NMI Watchdog detected LOCKUP on CPU 1
[117115.843485] CPU 1                                 
[117115.845901] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_disk ide_cd cdrom ipv6 joydev evdev i2c_nforce2 snd_ac97_bus analog psmouse floppy parport_pc parport i2c_core soundcore gameport pcspkr serio_raw rtc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod ide_generic sd_mod sata_nv libata scsi_mod ehci_hcd generic forcedeth amd74xx ide_core ohci_hcd thermal processor fan unix
[117115.888369] Pid: 18498, comm: proftpd Not tainted 2.6.15.3 #1
[117115.895066] RIP: 0010:[<ffffffff802a08e7>] <ffffffff802a08e7>{.text.lock.spinlock+46}
[117115.903984] RSP: 0018:ffff81004c72fb40  EFLAGS: 00200082                             
[117115.910397] RAX: 0000000000000001 RBX: ffff81000000c380 RCX: 0000000000001000
[117115.918699] RDX: ffff81004c72fc10 RSI: 0000000000000000 RDI: ffff81000000c600
[117115.926992] RBP: ffff810001b83b60 R08: ffff81005344f940 R09: ffff81005344f940
[117115.935300] R10: ffffffff801d06de R11: ffffffff801d06de R12: ffff81004c72fb88
[117115.943609] R13: 0000000000000000 R14: 0000000000000020 R15: 0000000000000020
[117115.951936] FS:  0000000000000000(0000) GS:ffffffff803a8880(0063) knlGS:000000005589dea0
[117115.961327] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b                           
[117115.968009] CR2: 000000005556f000 CR3: 0000000047c8b000 CR4: 00000000000006e0
[117115.976312] Process proftpd (pid: 18498, threadinfo ffff81004c72e000, task ffff81005169f790)
[117115.986117] Stack: ffffffff8015a02a ffff81005344f940 ffff810001b83c08 ffff81004c72fcf8      
[117115.995231]        0000000000000020 ffff81005344f940 ffffffff801910f3 ffffffff8811fa86 
[117116.004543]        ffff810060f761f0 0000000000000004                                   
[117116.010465] Call Trace:<ffffffff8015a02a>{__pagevec_lru_add+76} <ffffffff801910f3>{mpage_readpages+246}
[117116.021389]        <ffffffff8811fa86>{:ext3:ext3_get_block+0} <ffffffff8015614c>{__do_page_cache_readahead+318}
[117116.033151]        <ffffffff8024295c>{lock_sock+158} <ffffffff80242c9b>{release_sock+19}                       
[117116.042579]        <ffffffff80271bd7>{tcp_sendpage+1565} <ffffffff8015646b>{blockable_page_cache_readahead+83}
[117116.054194]        <ffffffff8015654b>{make_ahead_window+129} <ffffffff801566f4>{page_cache_readahead+401}     
[117116.065298]        <ffffffff80150fae>{do_generic_mapping_read+294} <ffffffff8014f816>{file_send_actor+0} 
[117116.076326]        <ffffffff8015127e>{generic_file_sendfile+71} <ffffffff80170642>{do_sendfile+458}     
[117116.086860]        <ffffffff8018088c>{sys_fcntl+718} <ffffffff8017072e>{sys_sendfile64+75}         
[117116.096459]        <ffffffff8011d14b>{cstar_do_call+27}                                   
[117116.102883]                                             
[117116.102884] Code: 83 3f 00 7e f9 e9 c1 fe ff ff f3 90 83 3f 00 7e f9 e9 d2 fe 
[117116.113139] console shuts up ...                                              
[117116.116932]  <0>Kernel panic - not syncing: Aiee, killing interrupt handler!
[117116.125356]  NMI Watchdog detected LOCKUP on CPU 0                          
[117121.478568] CPU 0                                 
[117121.480969] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_disk ide_cd cdrom ipv6 joydev evdev i2c_nforce2 snd_ac97_bus analog psmouse floppy parport_pc parport i2c_core soundcore gameport pcspkr serio_raw rtc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod ide_generic sd_mod sata_nv libata scsi_mod ehci_hcd generic forcedeth amd74xx ide_core ohci_hcd thermal processor fan unix
[117121.523462] Pid: 7245, comm: syslogd Not tainted 2.6.15.3 #1
[117121.530074] RIP: 0010:[<ffffffff802a08e7>] <ffffffff802a08e7>{.text.lock.spinlock+46}
[117121.538983] RSP: 0018:ffff81007c903ab0  EFLAGS: 00000082                             
[117121.545395] RAX: 0000000000000001 RBX: ffff810002745620 RCX: 0000000000000000
[117121.553689] RDX: 0000000000000000 RSI: ffff81007c903b28 RDI: ffff81000000c600
[117121.561972] RBP: ffff81000000c380 R08: 0000000000000004 R09: 0000000000000008
[117121.570283] R10: 00000000fffffff9 R11: 0000000000000000 R12: ffff810002c14fa0
[117121.578583] R13: ffff810002c14f60 R14: 0000000000000000 R15: ffff810018018138
[117121.586893] FS:  0000000000000000(0000) GS:ffffffff803a8800(0063) knlGS:00000000556b36c0
[117121.596301] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b                           
[117121.603000] CR2: ffff8000027a7c68 CR3: 000000007d881000 CR4: 00000000000006e0
[117121.611311] Process syslogd (pid: 7245, threadinfo ffff81007c902000, task ffff81007d4c8380)
[117121.621029] Stack: ffffffff8015a184 0000000000000000 ffff810002745620 0000000000000008     
[117121.630152]        ffffffff8015a22b ffff8100251a1298 ffffffff801738cf ffff8100251a1298 
[117121.639431]        ffff81005e7106b8 ffff810029ccba80                                   
[117121.645316] Call Trace:<ffffffff8015a184>{activate_page+34} <ffffffff8015a22b>{mark_page_accessed+33}
[117121.656034]        <ffffffff801738cf>{__find_get_block+373} <ffffffff801738fa>{__getblk+25}          
[117121.665745]        <ffffffff8811d68b>{:ext3:__ext3_get_inode_loc+549} <ffffffff8811d81d>{:ext3:ext3_reserve_inode_write+33}
[117121.678660]        <ffffffff8811dd18>{:ext3:ext3_mark_inode_dirty+22} <ffffffff8811e15e>{:ext3:ext3_dirty_inode+99}        
[117121.690804]        <ffffffff8018f9fc>{__mark_inode_dirty+44} <ffffffff80151da8>{__generic_file_aio_write_nolock+678}
[117121.703008]        <ffffffff80151ee1>{__generic_file_write_nolock+149}                                              
[117121.710632]        <ffffffff80181530>{poll_freewait+73} <ffffffff801420ac>{autoremove_wake_function+0}
[117121.721437]        <ffffffff80151f38>{generic_file_writev+62} <ffffffff801960dc>{compat_do_readv_writev+374}
[117121.732860]        <ffffffff8016fa8b>{do_sync_write+0} <ffffffff80196254>{compat_sys_writev+89}             
[117121.742971]        <ffffffff8011d14b>{cstar_do_call+27}                                        
[117121.749368]                                             
[117121.749369] Code: 83 3f 00 7e f9 e9 c1 fe ff ff f3 90 83 3f 00 7e f9 e9 d2 fe 
[117121.759660] console shuts up ...                                              

I guess the two last follow from the first one, but I guess having all the
information there still would be in order. :-) Please Cc me on any replies;
I'm not subscribed to the list.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Steinar H. Gunderson]

On Sun, Apr 23, 2006 at 12:12:32AM +0200, Steinar H. Gunderson wrote:
> My dual-core AMD64 machine (Athlon 64 X2 3800+) has been unstable (hanging
> completely from the to time) since I ever got it, with various kernels (2.6.8
> up to 2.6.17-rc1), even more so after the I/O load increased on it (earlier
> it hung about once a month; now it's more once a day). This time, however, I
> had a minicom logging everything on the serial console, so I have a
> traceback:

I reproduced this with 2.6.17-rc2 on the same machine:

[261604.531829] Unable to handle kernel paging request at ffff8000020369d8 RIP: 
[261604.536538] <ffffffff802509e6>{isolate_lru_pages+74}
[261604.544074] PGD 0 
[261604.546196] Oops: 0002 [1] SMP 
[261604.549475] CPU 0 
[261604.551606] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_generic ide_disk ide_cd cdrom ipv6 i810_audio ac97_codec psmouse serio_raw analog snd_intel8x0 snd_ac97_codec snd_ac97_bus floppy snd_pcm snd_timer snd i2c_nforce2 gameport parport_pc pcspkr rtc soundcore i2c_core parport snd_page_alloc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod sd_mod sata_nv libata scsi_mod forcedeth generic amd74xx ehci_hcd ide_core ohci_hcd thermal processor fan unix
[261604.595778] Pid: 194, comm: kswapd0 Not tainted 2.6.17-rc2 #1
[261604.601603] RIP: 0010:[<ffffffff802509e6>] <ffffffff802509e6>{isolate_lru_pages+74}
[261604.609194] RSP: 0000:ffff81007fe3fc90  EFLAGS: 00210002
[261604.614839] RAX: ffff8000020369d8 RBX: 0000000000000020 RCX: ffff81007fe3fe20
[261604.622128] RDX: ffff81000000c5c8 RSI: ffff81000000c5c8 RDI: ffff81007fe3fcf8
[261604.629406] RBP: 0000000000000002 R08: ffff810000dabba0 R09: ffff810000dabb78
[261604.636625] R10: 0000000000000012 R11: 0000000000000012 R12: 0000000000000000
[261604.643904] R13: ffff81007fe3fe98 R14: 0000000000000001 R15: 0000000000000000
[261604.651167] FS:  0000000000000000(0000) GS:ffffffff804be000(0000) knlGS:00000000ed5e8bb0
[261604.659345] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[261604.665188] CR2: ffff8000020369d8 CR3: 000000000dc2b000 CR4: 00000000000006e0
[261604.672416] Process kswapd0 (pid: 194, threadinfo ffff81007fe3e000, task ffff81007fe1a8a0)
[261604.680768] Stack: 0000000000000020 ffff81000000c340 ffffffff80250f25 000000000000000f 
[261604.688747]        000000000000000c 0000000000000064 0000000000000005 0000000000000000 
[261604.696909]        0000000000000020 0000000000000020 
[261604.702111] Call Trace: <ffffffff80250f25>{shrink_zone+402} <ffffffff88112114>{:mbcache:mb_cache_shrink_fn+39}
[261604.712303]        <ffffffff80250945>{shrink_slab+317} <ffffffff80251bc2>{balance_pgdat+536}
[261604.720911]        <ffffffff80251e11>{kswapd+263} <ffffffff80238998>{autoremove_wake_function+0}
[261604.729848]        <ffffffff8020a3d6>{child_rip+8} <ffffffff80214ee8>{flat_send_IPI_mask+0}
[261604.738357]        <ffffffff80251d0a>{kswapd+0} <ffffffff8020a3ce>{child_rip+0}
[261604.746032] 
[261604.746033] Code: 48 89 10 41 8b 50 e0 49 c7 00 00 01 10 00 49 c7 40 08 00 02 
[261604.755292] RIP <ffffffff802509e6>{isolate_lru_pages+74} RSP <ffff81007fe3fc90>
[261604.762729] CR2: ffff8000020369d8
[261604.766157]  NMI Watchdog detected LOCKUP on CPU 1
[261609.865491] CPU 1 
[261609.867613] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_generic ide_disk ide_cd cdrom ipv6 i810_audio ac97_codec psmouse serio_raw analog snd_intel8x0 snd_ac97_codec snd_ac97_bus floppy snd_pcm snd_timer snd i2c_nforce2 gameport parport_pc pcspkr rtc soundcore i2c_core parport snd_page_alloc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod sd_mod sata_nv libata scsi_mod forcedeth generic amd74xx ehci_hcd ide_core ohci_hcd thermal processor fan unix
[261609.911819] Pid: 32429, comm: apache2 Not tainted 2.6.17-rc2 #1
[261609.917827] RIP: 0010:[<ffffffff8039fc92>] <ffffffff8039fc92>{.text.lock.spinlock+49}
[261609.925660] RSP: 0000:ffff8100168e9c30  EFLAGS: 00200082
[261609.931243] RAX: 0000000000000001 RBX: ffff810001cccba0 RCX: ffff810002117040
[261609.938532] RDX: ffff810000635710 RSI: 0000000000000000 RDI: ffff81000000c5c0
[261609.945742] RBP: ffff81000000c340 R08: 0000000000000000 R09: 0000000000000000
[261609.952970] R10: 0000000000000000 R11: 0000000000000000 R12: ffff810002117040
[261609.960249] R13: 0000000000000000 R14: ffff81007aa0e608 R15: 0000000000000000
[261609.967539] FS:  0000000000000000(0000) GS:ffff81000245b940(0063) knlGS:00000000f65fabb0
[261609.975725] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[261609.981550] CR2: 00000000e9052004 CR3: 0000000049f38000 CR4: 00000000000006e0
[261609.988763] Process apache2 (pid: 32429, threadinfo ffff8100168e8000, task ffff81007fa609e0)
[261609.997287] Stack: ffffffff8024ff25 0000000000000000 0000000000000000 ffff810000635710 
[261610.005273]        00000000000200d2 ffff81007f895528 ffffffff8024928b ffff81007aa0e608 
[261610.013437]        0000000000000000 ffff810000635710 
[261610.018620] Call Trace: <ffffffff8024ff25>{__pagevec_lru_add+76}
[261610.024780]        <ffffffff8024928b>{add_to_page_cache_lru+25} <ffffffff80263ccd>{shmem_getpage+1119}
[261610.034241]        <ffffffff80264744>{shmem_nopage+80} <ffffffff8025602c>{__handle_mm_fault+872}
[261610.043202]        <ffffffff803a1994>{do_page_fault+1087} <ffffffff8039fea5>{__kprobes_text_start+253}
[261610.052668]        <ffffffff8039e248>{thread_return+0} <ffffffff8039dc77>{schedule+334}
[261610.060848]        <ffffffff8020a21d>{error_exit+0}
[261610.066065] 
[261610.066066] Code: 7e f9 e9 c1 fe ff ff f3 90 83 3f 00 7e f9 e9 d2 fe ff ff e8 
[261610.075325] console shuts up ...
[261610.078632]  <0>Kernel panic - not syncing: Aiee, killing interrupt handler!
[261610.085970]  NMI Watchdog detected LOCKUP on CPU 0
[261615.835545] CPU 0 
[261615.837677] Modules linked in: af_packet w83627hf hwmon_vid i2c_isa ide_generic ide_disk ide_cd cdrom ipv6 i810_audio ac97_codec psmouse serio_raw analog snd_intel8x0 snd_ac97_codec snd_ac97_bus floppy snd_pcm snd_timer snd i2c_nforce2 gameport parport_pc pcspkr rtc soundcore i2c_core parport snd_page_alloc ext3 jbd mbcache raid6 raid5 xor raid10 raid1 raid0 linear md_mod dm_mod sd_mod sata_nv libata scsi_mod forcedeth generic amd74xx ehci_hcd ide_core ohci_hcd thermal processor fan unix
[261615.882039] Pid: 32474, comm: apache2 Not tainted 2.6.17-rc2 #1
[261615.888047] RIP: 0010:[<ffffffff8039fc8d>] <ffffffff8039fc8d>{.text.lock.spinlock+44}
[261615.895846] RSP: 0000:ffff81000edfdc30  EFLAGS: 00200082
[261615.901482] RAX: 0000000000000001 RBX: ffff810000afb648 RCX: ffff81000210f040
[261615.908701] RDX: ffff810000a06760 RSI: 0000000000000000 RDI: ffff81000000c5c0
[261615.915947] RBP: ffff81000000c340 R08: 0000000000000000 R09: 0000000000000000
[261615.923182] R10: 0000000000000000 R11: 0000000000000000 R12: ffff81000210f040
[261615.930402] R13: 0000000000000000 R14: ffff81005ede6ac0 R15: 0000000000000000
[261615.937630] FS:  0000000000000000(0000) GS:ffffffff804be000(0063) knlGS:00000000ed5e8bb0
[261615.945868] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[261615.951702] CR2: 00000000e3784004 CR3: 000000000dc2b000 CR4: 00000000000006e0
[261615.958932] Process apache2 (pid: 32474, threadinfo ffff81000edfc000, task ffff81007fa0b5a0)
[261615.967472] Stack: ffffffff8024ff25 0000000000000000 0000000000000000 ffff810000a06760 
[261615.975496]        00000000000200d2 ffff81007e16c3d0 ffffffff8024928b ffff81005ede6ac0 
[261615.983666]        0000000000000000 ffff810000a06760 
[261615.988859] Call Trace: <ffffffff8024ff25>{__pagevec_lru_add+76}
[261615.994999]        <ffffffff8024928b>{add_to_page_cache_lru+25} <ffffffff80263ccd>{shmem_getpage+1119}
[261616.004495]        <ffffffff8024a9cb>{filemap_nopage+389} <ffffffff80264744>{shmem_nopage+80}
[261616.013219]        <ffffffff8025602c>{__handle_mm_fault+872} <ffffffff8039f5fe>{__down_read+130}
[261616.022205]        <ffffffff803a1994>{do_page_fault+1087} <ffffffff8039fea5>{__kprobes_text_start+253}
[261616.031726]        <ffffffff803a2280>{kprobe_flush_task+30} <ffffffff8039e2be>{thread_return+118}
[261616.040826]        <ffffffff8020a21d>{error_exit+0}
[261616.046039] 
[261616.046040] Code: f3 90 83 3f 00 7e f9 e9 c1 fe ff ff f3 90 83 3f 00 7e f9 e9 
[261616.055293] console shuts up ...
[261616.058609]  

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[R. J. Wysocki]

On Wednesday 26 April 2006 17:15, Steinar H. Gunderson wrote:
> On Sun, Apr 23, 2006 at 12:12:32AM +0200, Steinar H. Gunderson wrote:
> > My dual-core AMD64 machine (Athlon 64 X2 3800+) has been unstable (hanging
> > completely from the to time) since I ever got it, with various kernels (2.6.8
> > up to 2.6.17-rc1), even more so after the I/O load increased on it (earlier
> > it hung about once a month; now it's more once a day). This time, however, I
> > had a minicom logging everything on the serial console, so I have a
> > traceback:
> 
> I reproduced this with 2.6.17-rc2 on the same machine:
> 
> [261604.531829] Unable to handle kernel paging request at ffff8000020369d8 RIP: 
> [261604.536538] <ffffffff802509e6>{isolate_lru_pages+74}

If your kernel is compiled with the debug info, could you please do
"gdb vmlinux"  in the kernel sorces directory and then (in gdb)
"l *(isolate_lru_pages+74)" to see which source line it corresponds to?

Greetings,
Rafael

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Steinar H. Gunderson]

On Wed, Apr 26, 2006 at 05:40:46PM +0200, R. J. Wysocki wrote:
>> I reproduced this with 2.6.17-rc2 on the same machine:
>> 
>> [261604.531829] Unable to handle kernel paging request at ffff8000020369d8 RIP: 
>> [261604.536538] <ffffffff802509e6>{isolate_lru_pages+74}
> If your kernel is compiled with the debug info, could you please do
> "gdb vmlinux"  in the kernel sorces directory and then (in gdb)
> "l *(isolate_lru_pages+74)" to see which source line it corresponds to?

It isn't, but I hadn't changed the sources, .config or build environment
since I built it, so I did a straight recompile with CONFIG_DEBUG_INFO set,
and got:

  (gdb) l *(isolate_lru_pages+74)
  0xffffffff80250c2a is in isolate_lru_pages (list.h:154).

(I had to run gdb on a machine with 64-bit userspace, but I guess just
copying the vmlinux file should suffice.)

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Steinar H. Gunderson]

On Wed, Apr 26, 2006 at 06:12:14PM +0200, Steinar H. Gunderson wrote:
> It isn't, but I hadn't changed the sources, .config or build environment
> since I built it, so I did a straight recompile with CONFIG_DEBUG_INFO set,
> and got:
> 
>   (gdb) l *(isolate_lru_pages+74)
>   0xffffffff80250c2a is in isolate_lru_pages (list.h:154).
> 
> (I had to run gdb on a machine with 64-bit userspace, but I guess just
> copying the vmlinux file should suffice.)

FWIW, I kept on looking at this, and I believe the kernel's symbol table is
borked somehow. The code doesn't match with the disassembly at all, and the
address of isolate_lru_pages in my disassembly doesn't really match what's
in /proc/kallsyms. OTOH, if I disassemble, I can find the exact code fragment
from the "Code" dump at:

  ffffffff8025096a <invalidate_mapping_pages>:
  
  <...>

  static inline int page_mapped(struct page *page)
  {
          return atomic_read(&(page)->_mapcount) >= 0;
  ffffffff802509cf:       75 0a                   jne    ffffffff802509db <invalidate_mapping_pages+0x71>
  ffffffff802509d1:       0f 0b                   ud2a   
  ffffffff802509d3:       68 9d 0f 3c 80          pushq  $0xffffffff803c0f9d
  ffffffff802509d8:       c2 4f 02                retq   $0x24f
  ffffffff802509db:       49 8b 10                mov    (%r8),%rdx
  ffffffff802509de:       49 8b 40 08             mov    0x8(%r8),%rax
  ffffffff802509e2:       48 89 42 08             mov    %rax,0x8(%rdx)
  ffffffff802509e6:       48 89 10                mov    %rdx,(%rax)                      <-- crash is here
  ffffffff802509e9:       41 8b 50 e0             mov    0xffffffffffffffe0(%r8),%edx
  ffffffff802509ed:       49 c7 00 00 01 10 00    movq   $0x100100,(%r8)
  ffffffff802509f4:       49 c7 40 08 00 02 20    movq   $0x200200,0x8(%r8)
  ffffffff802509fb:       00 

which matches the _address_ from the kernel, but not the _symbol_ it spits
out. Something's odd here :-)

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Rafael J. Wysocki]

[sorry, I sent the previous message with a wrong address by mistake]

On Wednesday 26 April 2006 18:12, Steinar H. Gunderson wrote:
> On Wed, Apr 26, 2006 at 05:40:46PM +0200, R. J. Wysocki wrote:
> >> I reproduced this with 2.6.17-rc2 on the same machine:
> >> 
> >> [261604.531829] Unable to handle kernel paging request at ffff8000020369d8 RIP: 
> >> [261604.536538] <ffffffff802509e6>{isolate_lru_pages+74}
> > If your kernel is compiled with the debug info, could you please do
> > "gdb vmlinux"  in the kernel sorces directory and then (in gdb)
> > "l *(isolate_lru_pages+74)" to see which source line it corresponds to?
> 
> It isn't, but I hadn't changed the sources, .config or build environment
> since I built it, so I did a straight recompile with CONFIG_DEBUG_INFO set,
> and got:
> 
>   (gdb) l *(isolate_lru_pages+74)
>   0xffffffff80250c2a is in isolate_lru_pages (list.h:154).
> 
> (I had to run gdb on a machine with 64-bit userspace, but I guess just
> copying the vmlinux file should suffice.)

This kind of agrees with my result ie. list_del() in isolate_lru_pages():

0xffffffff80265b7a is in isolate_lru_pages (list.h:160).
155      * Note: list_empty on entry does not return true after this, the entry is
156      * in an undefined state.
157      */
158     static inline void list_del(struct list_head *entry)
159     {
160             BUG_ON(entry->prev->next != entry);
161             BUG_ON(entry->next->prev != entry);
162             __list_del(entry->prev, entry->next);
163             entry->next = LIST_POISON1;
164             entry->prev = LIST_POISON2;

It looks like we have passed an empty list or a NULL to list_del().  Strange.

Greetings,
Rafael

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Steinar H. Gunderson]

On Wed, Apr 26, 2006 at 10:59:39PM +0200, Rafael J. Wysocki wrote:
> This kind of agrees with my result ie. list_del() in isolate_lru_pages():

Well, look at the followup -- I'm not convinced the address lookup is right.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: kswapd oops reproduced with 2.6.17-rc2 (was Oops with 2.6.15.3 on amd64)

[Rafael J. Wysocki]

On Wednesday 26 April 2006 23:01, Steinar H. Gunderson wrote:
> On Wed, Apr 26, 2006 at 10:59:39PM +0200, Rafael J. Wysocki wrote:
> > This kind of agrees with my result ie. list_del() in isolate_lru_pages():
> 
> Well, look at the followup -- I'm not convinced the address lookup is right.

Beats me. :-(

Rafael