From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753937Ab1LUXND (ORCPT ); Wed, 21 Dec 2011 18:13:03 -0500 Received: from yuna.grokhost.net ([87.117.228.63]:50134 "EHLO yuna.grokhost.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751933Ab1LUXM6 convert rfc822-to-8bit (ORCPT ); Wed, 21 Dec 2011 18:12:58 -0500 Subject: Re: BUG: unable to handle kernel NULL pointer dereference in ipv6_select_ident Mime-Version: 1.0 (Apple Message framework v1254) Content-Type: text/plain; charset=iso-8859-1 From: Chris Boot In-Reply-To: <4EF2568C.6040006@bootc.net> Date: Wed, 21 Dec 2011 23:12:53 +0000 Cc: lkml , netdev Content-Transfer-Encoding: 8BIT Message-Id: References: <4EF200BB.7000209@bootc.net> <1324484956.2301.24.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> <4EF2117F.6000803@bootc.net> <1324488984.2301.45.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> <1324490401.2301.46.camel@edumazet-HP-Compaq-6005-Pro-SFF-PC> <4EF23BF2.4000601@bootc.net> <1324499332.2621.7.camel@edumazet-laptop> <1324500775.2621.9.camel@edumazet-laptop> <4EF2568C.6040006@bootc.net> To: Eric Dumazet X-Mailer: Apple Mail (2.1254) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21 Dec 2011, at 21:58, Chris Boot wrote: > On 21/12/2011 20:52, Eric Dumazet wrote: >> Le mercredi 21 décembre 2011 à 21:28 +0100, Eric Dumazet a écrit : >>> Le mercredi 21 décembre 2011 à 20:05 +0000, Chris Boot a écrit : >>>> On 21/12/2011 18:00, Eric Dumazet wrote: >>>>> Le mercredi 21 décembre 2011 à 18:36 +0100, Eric Dumazet a écrit : >>>>> >>>>>> Good point, thats a different problem then, since 3.1 is not supposed to >>>>>> have this bug. >>>>>> >>>>>> It seems rt->rt6i_peer points to invalid memory in your crash. >>>>>> >>>>>> (RBX=00000000000001f4) >>>>>> >>>>>> 8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax p->refcnt >>>>>> 1f4+a4 -> CR2=0000000000000298 >>>>>> >>>>> It would help if you can confirm latest linux tree can reproduce the >>>>> bug. >>>> Hi Eric, >>>> >>>> I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian >>>> 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as >>>> with the Debian kernel. Unfortunately I wasn't able to get an entire >>>> trace, for some reason it didn't appear to be printed to the serial port >>>> and hung after the (long) list of loaded kernel modules. The crash >>>> happens at the same offset: >>>> >>> Thanks ! >>> >>> Oh well, br_netfilter fake_rtable strikes again. >>> >>> I'll cook a patch in a couple of minutes... >>> >> Could you try following patch ? >> >> [snip] > > Eric, > > It looks good! The rsync that caused the crash real quick hasn't done it at all with the patch applied. I'll keep testing it of course, but I think that's done it. No, sorry, false hope. The following does look rather different however: [ 1944.056075] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1944.063987] IP: [< (null)>] (null) [ 1944.069156] PGD 412c75067 PUD 413722067 PMD 0 [ 1944.073783] Oops: 0010 [#1] SMP [ 1944.077201] CPU 0 [ 1944.079136] Modules linked in: sha1_ssse3 sha1_generic hmac sha256_generic dlm configfs ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace microcode xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_hfsc sch_htb sch_ingress sch_sfq xt_realm xt_addrtype xt_connlimit ip_set_hash_ip iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ip6_queue xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_conntrack_proto_udplite nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp ts_kmp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_amanda nf_conntrack_h323 nf_conntrack_ftp xt_time xt_TCPMSS xt_TPROXY xt_sctp xt_policy nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit ip6t_LOG ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp ip6table_mangle xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables ip6table_filter ip6_tables x_tables bridge stp bonding w83627ehf hwmon_vid coretemp crc32c_intel aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf ipmi_si ipmi_msghandler vhost_net macvtap macvlan tun drbd lru_cache cn loop kvm_intel kvm snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core processor psmouse iTCO_wdt thermal_sys evdev button pcspkr joydev serio_raw iTCO_vendor_support ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod usb_storage usbhid uas hid sd_mod crc_t10dif ahci libahci libata ehci_hcd igb scsi_mod dca usbcore e1000e usb_common [last unloaded: scsi_wait_scan] [ 1944.270355] [ 1944.271898] Pid: 6833, comm: vhost-6831 Not tainted 3.2.0-rc6+ #2 Supermicro X9SCL/X9SCM/X9SCL/X9SCM [ 1944.281431] RIP: 0010:[<0000000000000000>] [< (null)>] (null) [ 1944.288974] RSP: 0018:ffff88043fc03960 EFLAGS: 00010206 [ 1944.294422] RAX: ffffffffa0394940 RBX: ffff8804249ab580 RCX: 0000000000000000 [ 1944.301690] RDX: ffff880425f36f70 RSI: ffffffffa03897ac RDI: ffff880425f36f70 [ 1944.308975] RBP: ffff8803fb9a904e R08: ffff880425f36000 R09: ffff88043fc03970 [ 1944.316295] R10: ffff8804249ab580 R11: ffff8804249ab580 R12: 0000000000000014 [ 1944.323669] R13: ffff880427373200 R14: ffff880425f36000 R15: ffff88042315c000 [ 1944.330984] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 [ 1944.339247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1944.345153] CR2: 0000000000000000 CR3: 0000000412e64000 CR4: 00000000000426e0 [ 1944.352471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1944.359696] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1944.366948] Process vhost-6831 (pid: 6833, threadinfo ffff8804134b2000, task ffff880415e93080) [ 1944.375836] Stack: [ 1944.377893] ffffffff812a62e1 ffff880425f36f70 ffffffff8168f810 ffff880400000014 [ 1944.385566] ffffffffa038e82d ffff8804249ab580 ffffffffa03897ac ffff880427373200 [ 1944.393283] 0000000000000008 ffff880425f36000 ffff8804249ab580 ffff880425f36000 [ 1944.400926] Call Trace: [ 1944.403442] [ 1944.405668] [] ? ip_fragment+0x9d/0x62a [ 1944.411264] [] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.418403] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.424500] [] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.431661] [] ? nf_iterate+0x41/0x77 [ 1944.437075] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.443168] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.449299] [] ? nf_hook_slow+0x68/0x101 [ 1944.455055] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.461162] [] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.468782] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.474866] [] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.481904] [] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1944.489657] [] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1944.496793] [] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1944.503956] [] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1944.510928] [] ? nf_iterate+0x41/0x77 [ 1944.516355] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.522803] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.529241] [] ? nf_hook_slow+0x68/0x101 [ 1944.534941] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.541337] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.548501] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.554921] [] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.561963] [] ? br_forward+0x16/0x5a [bridge] [ 1944.568219] [] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1944.575725] [] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1944.583459] [] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1944.589926] [] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1944.597036] [] ? nf_iterate+0x41/0x77 [ 1944.602482] [] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1944.609613] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.616701] [] ? nf_hook_slow+0x68/0x101 [ 1944.622438] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.629566] [] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1944.636506] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.643533] [] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1944.650579] [] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1944.657440] [] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1944.664924] [] ? __netif_receive_skb+0x324/0x41f [ 1944.671318] [] ? process_backlog+0x6c/0x123 [ 1944.677277] [] ? blk_done_softirq+0x65/0x74 [ 1944.683235] [] ? net_rx_action+0xa1/0x1af [ 1944.688995] [] ? test_tsk_need_resched+0xa/0x13 [ 1944.695229] [] ? __do_softirq+0xb9/0x177 [ 1944.700887] [] ? call_softirq+0x1c/0x30 [ 1944.706513] [ 1944.708704] [] ? do_softirq+0x3c/0x7b [ 1944.714116] [] ? netif_rx_ni+0x1e/0x27 [ 1944.719617] [] ? tun_get_user+0x39a/0x3c2 [tun] [ 1944.725927] [] ? tun_sendmsg+0x1d/0x1f [tun] [ 1944.731950] [] ? handle_tx+0x340/0x3de [vhost_net] [ 1944.738506] [] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1944.745315] [] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1944.753141] [] ? kthread+0x76/0x7e [ 1944.758332] [] ? kernel_thread_helper+0x4/0x10 [ 1944.764533] [] ? kthread_worker_fn+0x139/0x139 [ 1944.770759] [] ? gs_change+0x13/0x13 [ 1944.776109] Code: Bad RIP value. [ 1944.779528] RIP [< (null)>] (null) [ 1944.784777] RSP [ 1944.788327] CR2: 0000000000000000 [ 1944.860298] ---[ end trace 7b7c89d77e0af47a ]--- [ 1944.864943] Kernel panic - not syncing: Fatal exception in interrupt [ 1944.871420] Pid: 6833, comm: vhost-6831 Tainted: G D 3.2.0-rc6+ #2 [ 1944.878608] Call Trace: [ 1944.881070] [] ? panic+0x95/0x1a5 [ 1944.886655] [] ? oops_end+0xa9/0xb6 [ 1944.891815] [] ? no_context+0x1ff/0x20e [ 1944.897343] [] ? do_page_fault+0x1a8/0x337 [ 1944.903091] [] ? sch_direct_xmit+0x85/0x12f [ 1944.908942] [] ? page_fault+0x25/0x30 [ 1944.914291] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.920282] [] ? ip_fragment+0x9d/0x62a [ 1944.925826] [] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.932948] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.938939] [] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.945967] [] ? nf_iterate+0x41/0x77 [ 1944.951296] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.957272] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.963246] [] ? nf_hook_slow+0x68/0x101 [ 1944.968857] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.974810] [] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.982277] [] ? br_flood+0xfa/0xfa [bridge] [ 1944.988264] [] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.995213] [] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1945.002868] [] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1945.009908] [] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1945.016924] [] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1945.023815] [] ? nf_iterate+0x41/0x77 [ 1945.029154] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.035466] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.041807] [] ? nf_hook_slow+0x68/0x101 [ 1945.047397] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.053711] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.060641] [] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.066982] [] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1945.073908] [] ? br_forward+0x16/0x5a [bridge] [ 1945.080083] [] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1945.087497] [] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1945.095719] [] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1945.102187] [] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1945.109095] [] ? nf_iterate+0x41/0x77 [ 1945.114455] [] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1945.121420] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.128399] [] ? nf_hook_slow+0x68/0x101 [ 1945.134054] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.141043] [] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1945.147903] [] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.154875] [] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1945.161830] [] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1945.168580] [] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1945.175945] [] ? __netif_receive_skb+0x324/0x41f [ 1945.182243] [] ? process_backlog+0x6c/0x123 [ 1945.188118] [] ? blk_done_softirq+0x65/0x74 [ 1945.193958] [] ? net_rx_action+0xa1/0x1af [ 1945.199638] [] ? test_tsk_need_resched+0xa/0x13 [ 1945.205887] [] ? __do_softirq+0xb9/0x177 [ 1945.211498] [] ? call_softirq+0x1c/0x30 [ 1945.217018] [] ? do_softirq+0x3c/0x7b [ 1945.223149] [] ? netif_rx_ni+0x1e/0x27 [ 1945.228602] [] ? tun_get_user+0x39a/0x3c2 [tun] [ 1945.234841] [] ? tun_sendmsg+0x1d/0x1f [tun] [ 1945.240805] [] ? handle_tx+0x340/0x3de [vhost_net] [ 1945.247269] [] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1945.254020] [] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1945.261773] [] ? kthread+0x76/0x7e [ 1945.266844] [] ? kernel_thread_helper+0x4/0x10 [ 1945.272963] [] ? kthread_worker_fn+0x139/0x139 [ 1945.279100] [] ? gs_change+0x13/0x13 [ 1945.284417] Rebooting in 120 seconds.. (gdb) list *ip_fragment+0x9d 0xffffffff812a62e1 is in ip_fragment (include/net/dst.h:209). 204 return dst_metric(dst, RTAX_FEATURES) & feature; 205 } 206 207 static inline u32 dst_mtu(const struct dst_entry *dst) 208 { 209 return dst->ops->mtu(dst); 210 } 211 212 /* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */ 213 static inline unsigned long dst_metric_rtt(const struct dst_entry *dst, int metric) (gdb) list *br_flood+0xfa 0x17ac is in br_dev_queue_push_xmit (net/bridge/br_forward.c:43). 38 { 39 return skb->len - (skb->protocol == htons(ETH_P_8021Q) ? VLAN_HLEN : 0); 40 } 41 42 int br_dev_queue_push_xmit(struct sk_buff *skb) 43 { 44 /* ip_fragment doesn't copy the MAC header */ 45 if (nf_bridge_maybe_copy_header(skb) || 46 (packet_length(skb) > skb->dev->mtu && !skb_is_gso(skb))) { 47 kfree_skb(skb); Thanks, Chris -- Chris Boot bootc@bootc.net