From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932295AbWGYJlF (ORCPT ); Tue, 25 Jul 2006 05:41:05 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S932303AbWGYJlE (ORCPT ); Tue, 25 Jul 2006 05:41:04 -0400 Received: from liaag1ae.mx.compuserve.com ([149.174.40.31]:65457 "EHLO liaag1ae.mx.compuserve.com") by vger.kernel.org with ESMTP id S932287AbWGYJlB (ORCPT ); Tue, 25 Jul 2006 05:41:01 -0400 Date: Tue, 25 Jul 2006 05:36:56 -0400 From: Chuck Ebbert <76306.1226@compuserve.com> Subject: Re: Unable to handle kernel paging request, another 2.6.16.25 server reboots To: Jim Klimov Cc: linux-kernel , linux-netdev Message-ID: <200607250538_MC3-1-C5FF-4726@compuserve.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org In-Reply-To: <254599816.20060724120148@2ka.mipt.ru> On Mon, 24 Jul 2006 12:01:48 +0400, Jim Klimov wrote: > I recently wrote about problems with a fileserver rebooting > frequently. Another similar server got under NFS load today > and rebooted at least twice in the past few hours. > > This server has a similar motherboard (Supermicro X5DP8-G2), > dual Xeons@533, two older 3Ware controllers (7506+8506) and > a reiserfs v3 archive. > > The server reported last week has two 3Ware 9550 controllers, > ext3 archives and primarily a Samba usage. I decoded your oops. It's in netfilter: Unable to handle kernel paging request at virtual address f9445d43 printing eip: c0392bba *pde = 32e59067 Oops: 0000 [#1] SMP Modules linked in: w83781d hwmon_vid hwmon i2c_isa i2c_core w83627hf_wdt CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010282 (2.6.16.25 #3) EIP is at ipt_do_table+0xae/0x385 eax: 00000003 ebx: 00000000 ecx: cbf4b8d8 edx: f944a2c8 esi: e2262940 edi: f9445cf0 ebp: 80000000 esp: f700fac8 ds: 007b es: 007b ss: 0068 Process nfsd (pid: 10685, threadinfo=f700e000 task=f36a7ab0) Stack: f9446b10 00000282 c33e2180 f36cda80 00000000 c047deec f944a2c8 f9418000 f7788800 c0530cd4 00000000 cbf4b8d8 00000000 00000003 f700fba0 00000000 f700fba0 00000003 c052f0d8 80000000 c03947e7 f7788800 c047dec0 00000000 Call Trace: [] ipt_local_out_hook+0x72/0x77 [] nf_iterate+0x69/0x83 [] dst_output+0x0/0x7 [] dst_output+0x0/0x7 [] nf_hook_slow+0x5d/0xea [] dst_output+0x0/0x7 [] ip_queue_xmit+0x3d4/0x4f5 [] dst_output+0x0/0x7 [] __rcu_process_callbacks+0x7d/0xc5 [] activate_task+0x99/0xa5 [] try_to_wake_up+0x29c/0x33b [] tcp_v4_send_check+0x4a/0xdc [] tcp_transmit_skb+0x2e6/0x45a [] tcp_push_one+0x97/0x104 [] tcp_sendmsg+0x36b/0xb4d [] nf_iterate+0x69/0x83 [] tcp_v4_rcv+0x4e6/0x81f [] inet_sendmsg+0x47/0x5f [] sock_sendmsg+0xc9/0xe3 [] ip_rcv+0x2bc/0x56f [] netif_receive_skb+0x227/0x2d7 [] __kfree_skb+0x3a/0xc3 [] autoremove_wake_function+0x0/0x43 [] update_wall_time_one_tick+0x6/0x7e [] update_wall_time+0x8/0x35 [] timer_interrupt+0x5b/0x86 [] handle_IRQ_event+0x26/0x59 [] kernel_sendmsg+0x2e/0x3c [] sock_no_sendpage+0x80/0x9f [] tcp_sendpage+0x49/0x85 [] svc_sendto+0x134/0x250 [] net_rx_action+0x88/0x15f [] do_IRQ+0x1e/0x24 [] common_interrupt+0x1a/0x20 [] svc_tcp_sendto+0x4d/0x99 [] _atomic_dec_and_lock+0x33/0x4c [] svc_send+0xaa/0xed [] fh_put+0x133/0x17d [] svcauth_unix_release+0x43/0x45 [] nfs3svc_release_fhandle+0x0/0xe [] svc_process+0x1b1/0x619 [] default_wake_function+0x0/0xc [] nfsd+0x178/0x301 [] nfsd+0x0/0x301 [] kernel_thread_helper+0x5/0xb 6: 8b 40 10 mov 0x10(%eax),%eax 9: 8b 44 86 34 mov 0x34(%esi,%eax,4),%eax d: 89 44 24 1c mov %eax,0x1c(%esp) 11: 89 c7 mov %eax,%edi 13: 8b 44 24 34 mov 0x34(%esp),%eax 17: 8b 54 24 1c mov 0x1c(%esp),%edx 1b: 03 7c 86 0c add 0xc(%esi,%eax,4),%edi 1f: 03 54 86 20 add 0x20(%esi,%eax,4),%edx 23: 89 5c 24 10 mov %ebx,0x10(%esp) 27: 89 54 24 18 mov %edx,0x18(%esp) 0: 0f b6 5f 53 movzbl 0x53(%edi),%ebx <===== 4: 89 d8 mov %ebx,%eax 6: 24 08 and $0x8,%al 8: 84 c0 test %al,%al a: 0f 84 b4 02 00 00 je 2c4 <_EIP+0x2c4> 10: 8b 47 08 mov 0x8(%edi),%eax This is in net/ipv4/netfiler/ip_tables.c::ipt_do_table(): table_base = (void *)private->entries[smp_processor_id()]; e = get_entry(table_base, private->hook_entry[hook]); /* For return from builtin chain */ back = get_entry(table_base, private->underflow[hook]); do { IP_NF_ASSERT(e); IP_NF_ASSERT(back); ===> if (ip_packet_match(ip, indev, outdev, &e->ip, offset)) { 'e' is an invalid pointer. (ip_packet_match() was inlined.) hook == 3 The call trace seems to show that svc_tcp_sendto() was interrupted by an IRQ for an incoming packet, or maybe the timer interrupt? Can you build with CONFIG_FRAME_POINTERS and see if you can get a cleaner trace? -- Chuck