Hi Frederic, FYI, here is another bisect result. git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit b30f0e3ffedfa52b1d67a302ae5860c49998e5e2 Author: Frederic Weisbecker AuthorDate: Tue May 12 16:41:49 2015 +0200 Commit: Ingo Molnar CommitDate: Tue May 19 08:39:12 2015 +0200 sched/preempt: Optimize preemption operations on __schedule() callers __schedule() disables preemption and some of its callers (the preempt_schedule*() family) also set PREEMPT_ACTIVE. So we have two preempt_count() modifications that could be performed at once. Lets remove the preemption disablement from __schedule() and pull this responsibility to its callers in order to optimize preempt_count() operations in a single place. Suggested-by: Linus Torvalds Signed-off-by: Frederic Weisbecker Signed-off-by: Peter Zijlstra (Intel) Cc: Peter Zijlstra Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com Signed-off-by: Ingo Molnar +-----------------------------------------------------------------------------+------------+------------+---------------+ | | 90b62b5129 | b30f0e3ffe | next-20150529 | +-----------------------------------------------------------------------------+------------+------------+---------------+ | boot_successes | 123 | 0 | 0 | | boot_failures | 0 | 41 | 14 | | BUG:scheduling_while_atomic | 0 | 1 | 2 | | BUG:unable_to_handle_kernel | 0 | 38 | 9 | | Oops | 0 | 28 | 9 | | RIP:task_curr | 0 | 19 | | | Kernel_panic-not_syncing:Fatal_exception | 0 | 26 | 10 | | BUG:kernel_boot_hang | 0 | 1 | 1 | | RIP:put_prev_task_fair | 0 | 1 | | | WARNING:at_kernel/sched/core.c:#__might_sleep() | 0 | 9 | | | RIP:ftrace_push_return_trace | 0 | 3 | 3 | | WARNING:at_kernel/trace/trace_functions_graph.c:#ftrace_return_to_handler() | 0 | 2 | 2 | | RIP:print_ftrace_graph_addr | 0 | 3 | 1 | | RIP:update_curr | 0 | 3 | 3 | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 0 | 3 | | | kernel_BUG_at_arch/x86/kernel/traps.c | 0 | 1 | | | invalid_opcode | 0 | 1 | | | RIP:fixup_bad_iret | 0 | 1 | | | general_protection_fault | 0 | 2 | 1 | | WARNING:CPU:#PID:#at_kernel/trace/trace_functions_graph.c:##() | 0 | 1 | | | Kernel_panic-not_syncing:stack-protector:Kernel_stack_is_corrupted_in | 0 | 1 | | | WARNING:at_fs/proc/generic.c:#__proc_create() | 0 | 1 | | | RIP:parameqn | 0 | 1 | | | backtrace:proc_create_data | 0 | 1 | | | backtrace:netlink_net_init | 0 | 1 | | | backtrace:ops_init | 0 | 1 | | | backtrace:register_pernet_subsys | 0 | 1 | | | backtrace:netlink_proto_init | 0 | 1 | | | backtrace:kernel_init_freeable | 0 | 1 | | | backtrace:parse_args | 0 | 1 | | | RIP:vsnprintf | 0 | 3 | | | WARNING:at_include/linux/uaccess.h:#is_prefetch() | 0 | 0 | 2 | | WARNING:at_include/linux/uaccess.h:#return_to_handler() | 0 | 0 | 2 | | WARNING:at_include/linux/uaccess.h:#bad_address() | 0 | 0 | 2 | | RIP:strnlen | 0 | 0 | 2 | | WARNING:at_include/linux/uaccess.h:#show_regs() | 0 | 0 | 2 | | RIP:rb_erase | 0 | 0 | 1 | +-----------------------------------------------------------------------------+------------+------------+---------------+ [ 0.755065] Testing tracer function_graph: [ 0.755065] Testing tracer function_graph: [ 0.801268] BUG: scheduling while atomic: swapper/0/0/0x00200002 [ 0.801268] BUG: scheduling while atomic: swapper/0/0/0x00200002 [ 0.820350] no locks held by swapper/0/0. [ 0.820350] no locks held by swapper/0/0. [ 0.823335] Preemption disabled at: [ 0.823335] Preemption disabled at:[] return_to_handler+0x0/0x28 [] return_to_handler+0x0/0x28 [ 0.828183] [ 0.828183] [ 0.874182] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc4-00025-gb30f0e3 #1 [ 0.874182] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc4-00025-gb30f0e3 #1 [ 0.875191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 0.875191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 0.876169] ffffffff841a9e08 [ 0.876169] ffffffff841a9e08 ffffffff841a9e08 ffffffff841a9e08 ffffffff8300ca98 ffffffff8300ca98 ffffffff84221340 ffffffff84221340 [ 0.880165] 0000000000000000 [ 0.880165] 0000000000000000 ffffffff841a9e38 ffffffff841a9e38 ffffffff81171047 ffffffff81171047 0000000000000000 0000000000000000 [ 0.960323] ffff880010fd5b80 [ 0.960323] ffff880010fd5b80 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffffffff841a9eb8 ffffffff841a9eb8 [ 1.056361] Call Trace: [ 1.056361] Call Trace: [ 1.059166] [ 1.059166] [ 1.166614] BUG: unable to handle kernel [ 1.166614] BUG: unable to handle kernel paging requestpaging request at 00000003a6e9c380 at 00000003a6e9c380 [ 1.167000] IP: [ 1.167000] IP: [] task_curr+0x1b/0x40 [] task_curr+0x1b/0x40 [ 1.167000] PGD 0 [ 1.167000] PGD 0 [ 1.167000] Thread overran stack, or stack corrupted [ 1.167000] Thread overran stack, or stack corrupted [ 1.167000] Oops: 0000 [#1] [ 1.167000] Oops: 0000 [#1] PREEMPT PREEMPT SMP SMP DEBUG_PAGEALLOC DEBUG_PAGEALLOC [ 1.167000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.1.0-rc4-00025-gb30f0e3 #1 [ 1.167000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 4.1.0-rc4-00025-gb30f0e3 #1 [ 1.167000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 1.167000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014 [ 1.167000] task: ffffffff84221340 ti: ffffffff84200000 task.ti: ffffffff84200000 [ 1.167000] task: ffffffff84221340 ti: ffffffff84200000 task.ti: ffffffff84200000 [ 1.167000] RIP: 0010:[] [ 1.167000] RIP: 0010:[] [] task_curr+0x1b/0x40 [] task_curr+0x1b/0x40 [ 1.167000] RSP: 0000:ffffffff841a9d50 EFLAGS: 00010002 [ 1.167000] RSP: 0000:ffffffff841a9d50 EFLAGS: 00010002 [ 1.167000] RAX: 00000000001d5b80 RBX: ffffffff84221340 RCX: 0000000000000002 [ 1.167000] RAX: 00000000001d5b80 RBX: ffffffff84221340 RCX: 0000000000000002 [ 1.167000] RDX: 00000000842000d8 RSI: ffffffff84221340 RDI: ffffffff84221340 [ 1.167000] RDX: 00000000842000d8 RSI: ffffffff84221340 RDI: ffffffff84221340 [ 1.167000] RBP: ffffffff841a9da8 R08: ffffffff84221db0 R09: 0000000000000001 [ 1.167000] RBP: ffffffff841a9da8 R08: ffffffff84221db0 R09: 0000000000000001 [ 1.167000] R10: 0000000000000050 R11: 0000000000000003 R12: 0000000000000004 [ 1.167000] R10: 0000000000000050 R11: 0000000000000003 R12: 0000000000000004 [ 1.167000] R13: ffffffff84223440 R14: 0000000000000005 R15: 0000000000000000 [ 1.167000] R13: ffffffff84223440 R14: 0000000000000005 R15: 0000000000000000 [ 1.167000] FS: 0000000000000000(0000) GS:ffff880010e00000(0000) knlGS:0000000000000000 [ 1.167000] FS: 0000000000000000(0000) GS:ffff880010e00000(0000) knlGS:0000000000000000 [ 1.167000] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1.167000] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1.167000] CR2: 00000003a6e9c380 CR3: 000000000421a000 CR4: 00000000000006b4 [ 1.167000] CR2: 00000003a6e9c380 CR3: 000000000421a000 CR4: 00000000000006b4 [ 1.167000] Stack: [ 1.167000] Stack: [ 1.167000] ffffffff8112f01b [ 1.167000] ffffffff8112f01b ffffffff8112ed05 ffffffff8112ed05 0000000000000000 0000000000000000 ffffffff841a9da8 ffffffff841a9da8 [ 1.167000] ffffffff816070b3 [ 1.167000] ffffffff816070b3 ffffffff86454b20 ffffffff86454b20 0000000000000000 0000000000000000 0000000000000005 0000000000000005 [ 1.167000] ffffffff841a9e58 [ 1.167000] ffffffff841a9e58 ffffffff84221340 ffffffff84221340 0000000000000002 0000000000000002 ffffffff841a9e08 ffffffff841a9e08 [ 1.167000] Call Trace: [ 1.167000] Call Trace: [ 1.167000] [ 1.167000] [ 1.167000] Code: [ 1.167000] Code: e8 e8 eb eb fd fd ff ff ff ff 48 48 83 83 05 05 03 03 28 28 2b 2b 05 05 01 01 5d 5d c3 c3 90 90 48 48 8b 8b 57 57 08 08 55 55 48 48 c7 c7 c0 c0 80 80 5b 5b 1d 1d 00 00 48 48 83 83 05 05 24 24 29 29 2b 2b 05 05 01 01 8b 8b 52 52 10 10 48 48 89 89 e5 e5 5d 5d <48> <48> 8b 8b 14 14 d5 d5 c0 c0 bc bc e9 e9 85 85 48 48 39 39 bc bc 10 10 e0 e0 08 08 00 00 00 00 0f 0f 94 94 c0 c0 0f 0f b6 b6 [ 1.167000] RIP [ 1.167000] RIP [] task_curr+0x1b/0x40 [] task_curr+0x1b/0x40 [ 1.167000] RSP [ 1.167000] RSP [ 1.167000] CR2: 00000003a6e9c380 [ 1.167000] CR2: 00000003a6e9c380 [ 1.167000] ---[ end trace 6e322970baec994b ]--- [ 1.167000] ---[ end trace 6e322970baec994b ]--- git bisect start 7732a9817fb01002bde7615066e86c156fb5a31b ba155e2d21f6bf05de86a78dbe5bfd8757604a65 -- git bisect good e7900c6f171ee82868cf0dcf398cf3b8f44114f1 # 12:47 25+ 2 Merge remote-tracking branch 'thermal-soc/next' git bisect bad c566f03d886fe9b6979b0a4e9a49aa61bfa3dd0c # 12:53 4- 22 Merge remote-tracking branch 'clockevents/clockevents/next' git bisect good 02fe2b99dfd80f254c8c491a9c00b5fc42a2eae7 # 13:02 25+ 1 Merge remote-tracking branch 'device-mapper/for-next' git bisect good b89ac4e6de3d2234180cccbf25e23979d1673c9f # 13:12 25+ 2 Merge remote-tracking branch 'security/next' git bisect good cd8133cd99ac0863dbbd2920ceba899218844fbd # 13:21 25+ 4 Merge remote-tracking branch 'trivial/for-next' git bisect good e0aca8d773e99485aefe3115396f92fd4fbd238b # 13:29 25+ 0 Merge remote-tracking branch 'mailbox/mailbox-for-next' git bisect good 342b07b3e19807e536755c7394bc9eb0ade5341c # 13:38 25+ 5 Merge remote-tracking branch 'spi/for-next' git bisect bad b114f50dde045c71fa1833ddabe2b70c52576f90 # 13:44 5- 22 Merge remote-tracking branch 'tip/auto-latest' git bisect bad c41f3eb84e58355b9209c961328067e2c23a1aeb # 13:50 11- 25 Merge branch 'timers/nohz' git bisect good 17186ccda374ae02ef231cbbc8f1825e7c19ddbd # 13:58 27+ 0 perf/x86/intel: Make WARN()ings consistent git bisect bad b7794610cdd2533e303de1027c1a4d9576875e51 # 14:12 17- 25 manual merge of sched/core git bisect bad 06931e62246844c73fba24d7aeb4a5dc897a2739 # 14:19 3- 40 sched/topology: Rename topology_thread_cpumask() to topology_sibling_cpumask() git bisect bad 8bcbde5480f9777f8b74d71493722c663e22c21b # 14:26 23- 41 sched/preempt, mm/fault: Count pagefault_disable() levels in pagefault_disabled git bisect good 7110744516276e906f9197e2857d026eb2343393 # 02:10 41+ 4 sched, timer: Use the atomic task_cputime in thread_group_cputimer git bisect good a22ae718067c233af790b8690b3d8f6190859ead # 02:28 41+ 10 Merge tag 'v4.1-rc4' into sched/core, before applying new patches git bisect good 90b62b5129d5cb50f62f40e684de7a1961e57197 # 02:36 41+ 0 sched/preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET git bisect bad e017cf21ae82e0b36f026b22083a8ae67926f465 # 02:43 0- 34 sched/preempt: Fix out of date comment git bisect bad b30f0e3ffedfa52b1d67a302ae5860c49998e5e2 # 02:53 2- 40 sched/preempt: Optimize preemption operations on __schedule() callers # first bad commit: [b30f0e3ffedfa52b1d67a302ae5860c49998e5e2] sched/preempt: Optimize preemption operations on __schedule() callers git bisect good 90b62b5129d5cb50f62f40e684de7a1961e57197 # 02:57 123+ 0 sched/preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET # extra tests with DEBUG_INFO git bisect bad b30f0e3ffedfa52b1d67a302ae5860c49998e5e2 # 03:05 7- 109 sched/preempt: Optimize preemption operations on __schedule() callers # extra tests on HEAD of next/master git bisect bad 7732a9817fb01002bde7615066e86c156fb5a31b # 03:15 0- 14 Add linux-next specific files for 20150529 # extra tests on tree/branch next/master # extra tests with first bad commit reverted # extra tests on tree/branch linus/master git bisect good c65b99f046843d2455aa231747b5a07a999a9f3d # 04:38 123+ 21 Linux 4.1-rc6 # extra tests on tree/branch next/master This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 kvm=( qemu-system-x86_64 -enable-kvm -cpu kvm64 -kernel $kernel -m 300 -smp 2 -device e1000,netdev=net0 -netdev user,id=net0 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang