FYI, we noticed the following commit: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 5413d1babe8f10de13d72496c12b862eef8ba613 ("net: do not block BH while processing socket backlog") on test machine: vm-lkp-wsx03-1G: 1 threads qemu-system-x86_64 -enable-kvm -cpu host with 1G memory caused below changes: [ 35.296699] mount.nfs (3510) used greatest stack depth: 11536 bytes left [ 42.600792] [ 42.601632] ====================================================== [ 42.603227] [ INFO: SOFTIRQ-safe -> SOFTIRQ-unsafe lock order detected ] [ 42.606492] 4.6.0-rc5-01190-g5413d1b #1 Not tainted [ 42.607747] ------------------------------------------------------ [ 42.609227] wget/3661 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire: [ 42.610880] (&(&tcp_hashinfo.bhash[i].lock)->rlock){+.+...}, at: [] __inet_twsk_hashdance+0x52/0x11d [ 42.613890] [ 42.613890] and this task is already holding: [ 42.615802] (slock-AF_INET){+.-...}, at: [] tcp_close+0x134/0x3ba [ 42.618697] which would create a new lock dependency: [ 42.620137] (slock-AF_INET){+.-...} -> (&(&tcp_hashinfo.bhash[i].lock)->rlock){+.+...} [ 42.623119] [ 42.623119] but this new dependency connects a SOFTIRQ-irq-safe lock: [ 42.625296] (slock-AF_INET){+.-...} ... which became SOFTIRQ-irq-safe at: [ 42.627562] [] __lock_acquire+0x970/0x192a [ 42.629139] [] lock_acquire+0x188/0x223 [ 42.630562] [] _raw_spin_lock+0x34/0x6a [ 42.632030] [] udp_queue_rcv_skb+0x204/0x4e8 [ 42.633558] [] __udp4_lib_rcv+0x872/0x9ce [ 42.634980] [] udp_rcv+0x1a/0x1c [ 42.636301] [] ip_local_deliver_finish+0x1aa/0x329 [ 42.637836] [] ip_local_deliver+0xb8/0xbf [ 42.639277] [] ip_rcv_finish+0x496/0x4ca [ 42.640702] [] ip_rcv+0x2d7/0x333 [ 42.642039] [] __netif_receive_skb_core+0x6a9/0x811 [ 42.643647] [] __netif_receive_skb+0x1d/0x5f [ 42.645141] [] netif_receive_skb_internal+0xc6/0x187 [ 42.646697] [] napi_gro_receive+0x15c/0x224 [ 42.648221] [] e1000_clean_rx_irq+0x374/0x3f5 [ 42.649722] [] e1000_clean+0x521/0x6b5 [ 42.651121] [] net_rx_action+0x12f/0x422 [ 42.652545] [] __do_softirq+0x207/0x4d0 [ 42.654067] [] run_ksoftirqd+0x25/0x62 [ 42.655477] [] smpboot_thread_fn+0x1cc/0x1e3 [ 42.656953] [] kthread+0xfb/0x103 [ 42.658296] [] ret_from_fork+0x22/0x50 [ 42.659725] [ 42.659725] to a SOFTIRQ-irq-unsafe lock: [ 42.661558] (&(&tcp_hashinfo.bhash[i].lock)->rlock){+.+...} ... which became SOFTIRQ-irq-unsafe at: [ 42.664127] ... [] __lock_acquire+0x9e2/0x192a [ 42.665706] [] lock_acquire+0x188/0x223 [ 42.667116] [] _raw_spin_lock+0x34/0x6a [ 42.668668] [] __inet_twsk_hashdance+0x52/0x11d [ 42.670170] [] tcp_time_wait+0x1da/0x210 [ 42.671770] [] tcp_fin+0x70/0x370 [ 42.673109] [] tcp_data_queue+0x2e7/0xab9 [ 42.674552] [] tcp_rcv_state_process+0xae1/0xc22 [ 42.676071] [] tcp_v4_do_rcv+0x106/0x14f [ 42.677501] [] release_sock+0xd2/0x1bb [ 42.678915] [] tcp_close+0x11b/0x3ba [ 42.680287] [] inet_release+0x52/0x59 [ 42.681761] [] sock_release+0x1f/0x78 [ 42.683215] [] sock_close+0x12/0x16 [ 42.684679] [] __fput+0x10d/0x1c6 [ 42.686024] [] ____fput+0xe/0x10 [ 42.687342] [] task_work_run+0x67/0x90 [ 42.688760] [] do_exit+0x556/0xb30 [ 42.690108] [] do_group_exit+0x4a/0xb5 [ 42.691505] [] __wake_up_parent+0x0/0x28 [ 42.692926] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.694459] [ 42.694459] other info that might help us debug this: [ 42.694459] [ 42.697180] Possible interrupt unsafe locking scenario: [ 42.697180] [ 42.699260] CPU0 CPU1 [ 42.700448] ---- ---- [ 42.701627] lock(&(&tcp_hashinfo.bhash[i].lock)->rlock); [ 42.703104] local_irq_disable(); [ 42.704490] lock(slock-AF_INET); [ 42.706017] lock(&(&tcp_hashinfo.bhash[i].lock)->rlock); [ 42.708427] [ 42.709351] lock(slock-AF_INET); [ 42.710586] [ 42.710586] *** DEADLOCK *** [ 42.710586] [ 42.712967] 1 lock held by wget/3661: [ 42.714042] #0: (slock-AF_INET){+.-...}, at: [] tcp_close+0x134/0x3ba [ 42.716659] the dependencies between SOFTIRQ-irq-safe lock and the holding lock: [ 42.718884] -> (slock-AF_INET){+.-...} ops: 163 { [ 42.720519] HARDIRQ-ON-W at: [ 42.721609] [] __lock_acquire+0x9c0/0x192a [ 42.723853] [] lock_acquire+0x188/0x223 [ 42.725475] [] _raw_spin_lock_bh+0x3c/0x72 [ 42.727632] [] lock_sock_nested+0x3b/0x9a [ 42.729803] [] sock_setsockopt+0x21e/0x8f9 [ 42.731979] [] SyS_setsockopt+0x63/0x9e [ 42.733607] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.735936] IN-SOFTIRQ-W at: [ 42.737020] [] __lock_acquire+0x970/0x192a [ 42.739196] [] lock_acquire+0x188/0x223 [ 42.740869] [] _raw_spin_lock+0x34/0x6a [ 42.742503] [] udp_queue_rcv_skb+0x204/0x4e8 [ 42.744687] [] __udp4_lib_rcv+0x872/0x9ce [ 42.746838] [] udp_rcv+0x1a/0x1c [ 42.748380] [] ip_local_deliver_finish+0x1aa/0x329 [ 42.750655] [] ip_local_deliver+0xb8/0xbf [ 42.752827] [] ip_rcv_finish+0x496/0x4ca [ 42.754986] [] ip_rcv+0x2d7/0x333 [ 42.756547] [] __netif_receive_skb_core+0x6a9/0x811 [ 42.758830] [] __netif_receive_skb+0x1d/0x5f [ 42.760997] [] netif_receive_skb_internal+0xc6/0x187 [ 42.763252] [] napi_gro_receive+0x15c/0x224 [ 42.765407] [] e1000_clean_rx_irq+0x374/0x3f5 [ 42.767657] [] e1000_clean+0x521/0x6b5 [ 42.769266] [] net_rx_action+0x12f/0x422 [ 42.771401] [] __do_softirq+0x207/0x4d0 [ 42.773017] [] run_ksoftirqd+0x25/0x62 [ 42.774619] [] smpboot_thread_fn+0x1cc/0x1e3 [ 42.776855] [] kthread+0xfb/0x103 [ 42.778408] [] ret_from_fork+0x22/0x50 [ 42.780013] INITIAL USE at: [ 42.781094] [] __lock_acquire+0xa28/0x192a [ 42.783228] [] lock_acquire+0x188/0x223 [ 42.784825] [] _raw_spin_lock_bh+0x3c/0x72 [ 42.786948] [] lock_sock_nested+0x3b/0x9a [ 42.789093] [] sock_setsockopt+0x21e/0x8f9 [ 42.791251] [] SyS_setsockopt+0x63/0x9e [ 42.792853] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.795088] } [ 42.795898] ... key at: [] af_family_slock_keys+0x10/0x160 [ 42.798089] ... acquired at: [ 42.799074] [] check_irq_usage+0x55/0xaa [ 42.800522] [] __lock_acquire+0x130d/0x192a [ 42.801982] [] lock_acquire+0x188/0x223 [ 42.803414] [] _raw_spin_lock+0x34/0x6a [ 42.804826] [] __inet_twsk_hashdance+0x52/0x11d [ 42.806340] [] tcp_time_wait+0x1da/0x210 [ 42.807780] [] tcp_close+0x245/0x3ba [ 42.809185] [] inet_release+0x52/0x59 [ 42.810584] [] sock_release+0x1f/0x78 [ 42.811987] [] sock_close+0x12/0x16 [ 42.813362] [] __fput+0x10d/0x1c6 [ 42.814790] [] ____fput+0xe/0x10 [ 42.816198] [] task_work_run+0x67/0x90 [ 42.817611] [] do_exit+0x556/0xb30 [ 42.818973] [] do_group_exit+0x4a/0xb5 [ 42.820380] [] __wake_up_parent+0x0/0x28 [ 42.821797] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.823327] [ 42.824105] the dependencies between the lock to be acquired and SOFTIRQ-irq-unsafe lock: [ 42.839607] -> (&(&tcp_hashinfo.bhash[i].lock)->rlock){+.+...} ops: 19 { [ 42.841688] HARDIRQ-ON-W at: [ 42.842789] [] __lock_acquire+0x9c0/0x192a [ 42.844954] [] lock_acquire+0x188/0x223 [ 42.846581] [] _raw_spin_lock_bh+0x3c/0x72 [ 42.848754] [] inet_csk_get_port+0x78/0x435 [ 42.850925] [] inet_bind+0x148/0x1c6 [ 42.852524] [] SyS_bind+0x6a/0x97 [ 42.854078] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.856336] SOFTIRQ-ON-W at: [ 42.857425] [] __lock_acquire+0x9e2/0x192a [ 42.859623] [] lock_acquire+0x188/0x223 [ 42.861257] [] _raw_spin_lock+0x34/0x6a [ 42.862881] [] __inet_twsk_hashdance+0x52/0x11d [ 42.865114] [] tcp_time_wait+0x1da/0x210 [ 42.867351] [] tcp_fin+0x70/0x370 [ 42.869032] [] tcp_data_queue+0x2e7/0xab9 [ 42.871183] [] tcp_rcv_state_process+0xae1/0xc22 [ 42.873565] [] tcp_v4_do_rcv+0x106/0x14f [ 42.875801] [] release_sock+0xd2/0x1bb [ 42.877408] [] tcp_close+0x11b/0x3ba [ 42.878999] [] inet_release+0x52/0x59 [ 42.880606] [] sock_release+0x1f/0x78 [ 42.882205] [] sock_close+0x12/0x16 [ 42.883876] [] __fput+0x10d/0x1c6 [ 42.885520] [] ____fput+0xe/0x10 [ 42.887060] [] task_work_run+0x67/0x90 [ 42.889781] [] do_exit+0x556/0xb30 [ 42.891522] [] do_group_exit+0x4a/0xb5 [ 42.893122] [] __wake_up_parent+0x0/0x28 [ 42.895331] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.897630] INITIAL USE at: [ 42.898709] [] __lock_acquire+0xa28/0x192a [ 42.900929] [] lock_acquire+0x188/0x223 [ 42.902664] [] _raw_spin_lock_bh+0x3c/0x72 [ 42.904895] [] inet_csk_get_port+0x78/0x435 [ 42.907129] [] inet_bind+0x148/0x1c6 [ 42.908721] [] SyS_bind+0x6a/0x97 [ 42.910250] [] entry_SYSCALL_64_fastpath+0x1f/0xbd [ 42.912481] } [ 42.913278] ... key at: [] __key.62185+0x0/0x8 [ 42.914811] ... acquired at: [ 42.915781] [] check_irq_usage+0x55/0xaa [ 42.917204] [] __lock_acquire+0x130d/0x192a [ 42.918684] [] lock_acquire+0x188/0x223 [ 42.920100] [] _raw_spin_lock+0x34/0x6a [ 42.921529] [] __inet_twsk_hashdance+0x52/0x11d [ 42.923031] [] tcp_time_wait+0x1da/0x210 [ 42.924463] [] tcp_close+0x245/0x3ba [ 42.925839] [] inet_release+0x52/0x59 [ 42.927229] [] sock_release+0x1f/0x78 [ 42.928642] [] sock_close+0x12/0x16 [ 42.930009] [] __fput+0x10d/0x1c6 [ 42.931477] [] ____fput+0xe/0x10 FYI, raw QEMU command line is: qemu-system-x86_64 -enable-kvm -cpu host -kernel /pkg/linux/x86_64-nfsroot/gcc-5/5413d1babe8f10de13d72496c12b862eef8ba613/vmlinuz-4.6.0-rc5-01190-g5413d1b -append 'root=/dev/ram0 user=lkp job=/lkp/scheduled/vm-lkp-wsx03-1G-23/bisect_boot-1-debian-x86_64-2015-02-07.cgz-x86_64-nfsroot-5413d1babe8f10de13d72496c12b862eef8ba613-20160504-63843-5pexbh-1.yaml ARCH=x86_64 kconfig=x86_64-nfsroot branch=internal-eywa/master commit=5413d1babe8f10de13d72496c12b862eef8ba613 BOOT_IMAGE=/pkg/linux/x86_64-nfsroot/gcc-5/5413d1babe8f10de13d72496c12b862eef8ba613/vmlinuz-4.6.0-rc5-01190-g5413d1b max_uptime=600 RESULT_ROOT=/result/boot/1/vm-lkp-wsx03-1G/debian-x86_64-2015-02-07.cgz/x86_64-nfsroot/gcc-5/5413d1babe8f10de13d72496c12b862eef8ba613/1 LKP_SERVER=inn earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal rw ip=::::vm-lkp-wsx03-1G-23::dhcp' -initrd /fs/sdc1/initrd-vm-lkp-wsx03-1G-23 -m 1024 -smp 1 -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::23622-:22 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -drive file=/fs/sdc1/disk0-vm-lkp-wsx03-1G-23,media=disk,if=virtio -drive file=/fs/sdc1/disk1-vm-lkp-wsx03-1G-23,media=disk,if=virtio -pidfile /dev/shm/kboot/pid-vm-lkp-wsx03-1G-23 -serial file:/dev/shm/kboot/serial-vm-lkp-wsx03-1G-23 -daemonize -display none -monitor null Thanks, Xiaolong