From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933271AbcBPTOq (ORCPT ); Tue, 16 Feb 2016 14:14:46 -0500 Received: from smtp2.megiteam.pl ([213.189.52.193]:54421 "EHLO smtp.megiteam.pl" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751730AbcBPTOo (ORCPT ); Tue, 16 Feb 2016 14:14:44 -0500 X-Greylist: delayed 1872 seconds by postgrey-1.27 at vger.kernel.org; Tue, 16 Feb 2016 14:14:43 EST To: linux-kernel@vger.kernel.org From: Grzegorz Nosek Subject: Panic in nf_ct_seq_adjust (be2net, synproxy, heavy load) Message-ID: <56C36DD6.6000705@localdomain.pl> Date: Tue, 16 Feb 2016 19:43:34 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, I have finally managed to capture a stack trace from an intermittent crash we've been observing under heavy load. There are two identical machines w/dual-port Emulex 10 Gbps NICs running in an active-passive firewall cluster (though more like passive-passive in reality). As far as I know, at the time of the reboot there was a pretty large DoS attack (6 Gbps) on the same network segment, not directed at the firewalls themselves. The firewalls were being tested with a ~80Mbps stream of small HTTP requests forwarded with SYNPROXY, conntrackd and some traffic shaping enabled. Unfortunately, this trace comes from Debian distro kernel (3.16), as I haven't yet upgraded this machine and I don't have serial console on the other one at the moment. The other node is running 4.4.1 and rebooted at the same time +/- seconds, /proc/uptime says it's up 15 seconds longer. netconsole stays silent during reboots. > [436190.637310] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 > [436190.674698] IP: [] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436190.709683] PGD 0 > [436190.719785] Oops: 0000 [#1] SMP > [436190.735431] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrac > k_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_inte > l kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter > pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor > nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct1 > 0dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_common be2net scsi_mod therm > al vxlan thermal_sys > [436191.160517] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2 > [436191.208587] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013 > [436191.239665] task: ffff88020af6c110 ti: ffff88020af78000 task.ti: ffff88020af78000 > [436191.274842] RIP: 0010:[] [] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436191.321186] RSP: 0018:ffff88020ba43c40 EFLAGS: 00010246 > [436191.346218] RAX: 0000000000030003 RBX: ffff880202931300 RCX: 0000000000000003 > [436191.381252] RDX: 00000000ff93df09 RSI: 0000000000000028 RDI: ffff8802064bdb0c > [436191.414649] RBP: ffff8802064bdb08 R08: ffffffff81459ea0 R09: 0000000000000020 > [436191.449511] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028 > [436191.482769] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8801e21b3062 > [436191.516924] FS: 0000000000000000(0000) GS:ffff88020ba40000(0000) knlGS:0000000000000000 > [436191.554343] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [436191.581010] CR2: 0000000000000004 CR3: 0000000001813000 CR4: 00000000000007e0 > [436191.613999] Stack: > [436191.624137] ffffffff00000014 ffffc9000d8c1800 ffff8802064bdb0c ffffffff00000000 > [436191.660568] ffff8801f92b0af0 0000000000000000 ffff880202931300 ffff8802064bdb08 > [436191.696262] ffff880203600000 0000000000000000 ffff880202931300 ffffffff818e95a0 > [436191.730498] Call Trace: > [436191.742505] > [436191.752229] [] ? ipv4_confirm+0xa3/0xf0 [nf_conntrack_ipv4] > [436191.786089] [] ? ip_fragment+0x880/0x880 > [436191.813367] [] ? nf_iterate+0x65/0xa0 > [436191.838356] [] ? ip_fragment+0x880/0x880 > [436191.864832] [] ? nf_hook_slow+0x76/0x130 > [436191.891480] [] ? ip_fragment+0x880/0x880 > [436191.917857] [] ? ip_output+0x82/0x90 > [436191.943454] [] ? __netif_receive_skb_core+0x543/0x750 > [436191.974406] [] ? netif_receive_skb_internal+0x1f/0x80 > [436192.006268] [] ? napi_gro_frags+0x1ba/0x2c0 > [436192.032907] [] ? be_process_rx+0x2c4/0x780 [be2net] > [436192.064273] [] ? be_poll+0x1f2/0x400 [be2net] > [436192.091625] [] ? net_rx_action+0x140/0x240 > [436192.119082] [] ? __do_softirq+0xf1/0x290 > [436192.144594] [] ? irq_exit+0x95/0xa0 > [436192.168465] [] ? do_IRQ+0x52/0xe0 > [436192.192353] [] ? common_interrupt+0x6d/0x6d > [436192.219491] > [436192.228773] [] ? __hrtimer_start_range_ns+0x1cd/0x390 > [436192.261747] [] ? cpuidle_enter_state+0x4f/0xc0 > [436192.290427] [] ? cpuidle_enter_state+0x48/0xc0 > [436192.319838] [] ? cpu_startup_entry+0x2f8/0x400 > [436192.348103] [] ? start_secondary+0x20f/0x2d0 > [436192.376655] Code: 00 00 00 88 44 24 08 83 e0 01 48 8d 14 00 48 01 d0 4d 8d 74 85 00 48 8d 45 04 48 89 c7 48 89 44 24 10 e8 a1 88 3c e1 41 8b 57 04 <41> 8b 76 04 45 8b 46 08 41 8b 7f 08 89 d1 0f c9 41 39 0e 0f 88 > [436192.464643] RIP [] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.499548] RSP > [436192.515631] CR2: 0000000000000004 > [436192.531367] ---[ end trace 84529a434dcea659 ]--- > [436192.531372] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 > [436192.531380] IP: [] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.531381] PGD 0 > [436192.531382] Oops: 0000 [#2] SMP > [436192.531402] Modules linked in: ipt_SYNPROXY nf_synproxy_core binfmt_misc xt_CLASSIFY xt_mac iptable_filter macvlan ip_set_hash_net cls_u32 sch_red sch_htb nf_conntrack_netlink xt_set ip_set nfnetlink iptable_mangle xt_tcpudp xt_CT xt_conntrack iptable_raw ip_tables arptable_filter arp_tables x_tables intel_powerclamp coretemp kvm_intel kvm crc32_pclmul aesni_intel aes_x86_64 joydev lrw gf128mul glue_helper ttm drm_kms_helper hpwdt drm i2c_algo_bit i2c_core hpilo psmouse evdev iTCO_wdt acpi_power_meter pcspkr serio_raw iTCO_vendor_support lpc_ich ablk_helper cryptd shpchp i7core_edac nf_conntrack_ftp mfd_core button edac_core nf_conntrack_ipv4 nf_defrag_ipv4 processor nf_conntrack ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler autofs4 ext4 crc16 mbcache jbd2 dm_mod hid_generic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common crc32c_intel be2iscsi ehci_pci uhci_hcd iscsi_boot_sysfs libiscsi ehci_hcd usbcore scsi_transport_iscsi hpsa usb_commo n be2net scsi_mod thermal vxlan thermal_sys > [436192.531411] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u2 > [436192.531411] Hardware name: HP ProLiant BL460c G7, BIOS I27 07/02/2013 > [436192.531412] task: ffff88020af6d3b0 ti: ffff88020af70000 task.ti: ffff88020af70000 > [436192.531416] RIP: 0010:[] [] nf_ct_seq_adjust+0xa3/0x3a0 [nf_conntrack] > [436192.531416] RSP: 0018:ffff88010bc23ca0 EFLAGS: 00010246 > [436192.531417] RAX: 0000000000030003 RBX: ffff8800371f0800 RCX: 0000000000000003 > [436192.531418] RDX: 000000001a710bff RSI: 0000000000000028 RDI: ffff8801f913ac44 > [436192.531418] RBP: ffff8801f913ac40 R08: ffffffff81459ea0 R09: 0000000000000020 > [436192.531419] R10: ffff880203600000 R11: ffff8800e7b38000 R12: 0000000000000028 > [436192.531419] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88010b84b862 > [436192.531420] FS: 0000000000000000(0000) GS:ffff88010bc20000(0000) knlGS:000000 The kernel log ends here, even though it looks like we could be expecting another identical or similar trace on another CPU (#2 vs #4 earlier). The issue repeats intermittently but I cannot reliably reproduce it (except that DoS attacks tend to bring the whole thing down). I don't know the nature of the attack, but the machines have easily survived 9 Gbps of iperf tests. Any help is greatly appreciated while I dig into the code on my own. Best regards, Grzegorz Nosek