Paul, Here is another dmesg for the same first bad commit commit 1f9c26968e1a501f3a839603c9633f21abca65b3 Commit: Paul E. McKenney CommitDate: Wed Mar 5 07:49:28 2014 -0800 torture: Increase stutter-end intensity +---------------------------------------------------------+------------+------------+------------+ | | 4c79d7d899 | 1f9c26968e | d478a960ed | +---------------------------------------------------------+------------+------------+------------+ | boot_successes | 203 | 4 | 2 | | boot_failures | 3 | 22 | 17 | | WARNING:CPU:PID:at_arch/x86/kernel/cpu/amd.c:init_amd() | 3 | 2 | | | backtrace:init_amd | 3 | 2 | | | backtrace:warn_slowpath_fmt | 3 | 2 | | | INFO:rcu_sched_self-detected_stall_on_CPU | 0 | 20 | 14 | | EIP_is_at_native_apic_mem_write | 0 | 18 | 14 | | EIP_is_at_native_safe_halt | 0 | 19 | 14 | | INFO:task_blocked_for_more_than_seconds | 0 | 19 | 14 | | EIP_is_at_stutter_wait | 0 | 20 | 14 | | Kernel_panic-not_syncing:hung_task:blocked_tasks | 0 | 19 | 14 | | backtrace:lock_torture_writer | 0 | 20 | 14 | | backtrace:cpu_startup_entry | 0 | 6 | 3 | | backtrace:SYSC_umount | 0 | 14 | 9 | | backtrace:SyS_oldumount | 0 | 13 | 6 | | backtrace:watchdog | 0 | 19 | 14 | | INFO:rcu_sched_detected_stalls_on_CPUs/tasks | 0 | 3 | 1 | | backtrace:lru_add_drain_all | 0 | 5 | 5 | | backtrace:SyS_mlockall | 0 | 3 | 4 | | backtrace:SyS_umount | 0 | 1 | 3 | | BUG:kernel_boot_hang | 0 | 1 | 2 | | BUG:kernel_test_hang | 0 | 2 | 1 | | EIP_is_at__default_send_IPI_dest_field | 0 | 2 | | | EIP_is_at_slob_page_alloc | 0 | 1 | | | INFO:lockdep_is_turned_off | 0 | 2 | | | backtrace:iterate_dir | 0 | 1 | | | backtrace:SyS_getdents64 | 0 | 1 | | | EIP_is_at_slob_alloc | 0 | 1 | | | EIP_is_at_page_fault | 0 | 1 | | | backtrace:SYSC_socketcall | 0 | 1 | | | backtrace:SyS_socketcall | 0 | 1 | | | backtrace:SyS_mlock | 0 | 2 | 1 | +---------------------------------------------------------+------------+------------+------------+ [ 23.969075] init: tty3 main process ended, respawning [ 23.985535] init: tty6 main process (264) terminated with status 1 [ 23.986111] init: tty6 main process ended, respawning [ 33.470004] INFO: rcu_sched self-detected stall on CPU [ 33.470004] [ 33.470004] [ 33.470004] sending NMI to all CPUs: [ 33.470004] NMI backtrace for cpu 0 [ 33.470004] CPU: 0 PID: 45 Comm: lock_torture_wr Not tainted 3.14.0-rc5-03765-gd478a96 #1 [ 33.470004] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 33.470004] task: 8d3e8000 ti: 8d3ea000 task.ti: 8d3ea000 [ 33.470004] EIP: 0060:[<81014de9>] EFLAGS: 00010046 CPU: 0 [ 33.470004] EIP is at native_apic_mem_write+0x12/0x14 [ 33.470004] EAX: ffff8300 EBX: 00000002 ECX: fffff000 EDX: 00000c00 [ 33.470004] ESI: 00000c00 EDI: 03000000 EBP: 8d3ebda4 ESP: 8d3ebda4 [ 33.470004] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 33.470004] CR0: 8005003b CR2: 00000000 CR3: 01abb000 CR4: 000006b0 [ 33.470004] Stack: [ 33.470004] 8d3ebdb8 81014e68 00000006 00000003 00000002 8d3ebdcc 8101509b 00002710 [ 33.470004] 92255a54 00006e36 8d3ebdd4 8101510c 8d3ebde4 81015241 8176c437 81858040 [ 33.470004] 8d3ebe20 8105f0d6 81775e21 00000834 ffffff45 ffffff44 00006e36 81046824 [ 33.470004] Call Trace: [ 33.470004] [<81014e68>] __default_send_IPI_dest_field+0x4c/0x51 [ 33.470004] [<8101509b>] default_send_IPI_mask_logical+0x47/0x53 [ 33.470004] [<8101510c>] default_send_IPI_all+0x21/0x32 [ 33.470004] [<81015241>] arch_trigger_all_cpu_backtrace+0x38/0x60 [ 33.470004] [<8105f0d6>] rcu_check_callbacks+0x1a2/0x444 [ 33.470004] [<81046824>] ? account_process_tick+0xfe/0x154 [ 33.470004] [<8102f6a9>] update_process_times+0x2d/0x50 [ 33.470004] [<81067398>] tick_sched_handle+0x30/0x32 [ 33.470004] [<81067923>] tick_sched_timer+0x2e/0x53 [ 33.470004] [<8103d6a5>] __run_hrtimer.isra.16+0x57/0xab [ 33.470004] [<810678f5>] ? tick_sched_do_timer+0x28/0x28 [ 33.470004] [<8103e02a>] hrtimer_interrupt+0xf0/0x1ec [ 33.470004] [<81055459>] ? torture_lock_busted_write_unlock+0x5/0x5 [ 33.470004] [<810140ab>] local_apic_timer_interrupt+0x41/0x46 [ 33.470004] [<810142ae>] smp_apic_timer_interrupt+0x20/0x2f [ 33.470004] [<8154486d>] apic_timer_interrupt+0x2d/0x34 [ 33.470004] [<81055459>] ? torture_lock_busted_write_unlock+0x5/0x5 [ 33.470004] [<810200d8>] ? aes_dec_blk+0x4c0/0xb6a [ 33.470004] [<81078ce4>] ? stutter_wait+0x1d/0x5a [ 33.470004] [<8105550e>] lock_torture_writer+0xb5/0xd9 [ 33.470004] [<8103b78c>] kthread+0x8e/0x93 [ 33.470004] [<81544c37>] ret_from_kernel_thread+0x1b/0x28 [ 33.470004] [<8103b6fe>] ? __kthread_parkme+0x50/0x50 [ 33.470004] Code: 00 5d c3 55 89 e5 50 9d 8d 74 26 00 5d c3 55 89 e5 fa 90 8d 74 26 00 5d c3 8b 0d 08 38 85 81 55 89 e5 8d 84 08 00 90 ff ff 89 10 <5d> c3 55 8d 48 01 89 e5 53 89 d3 89 d8 ba 08 00 00 00 e8 e8 ee [ 33.480082] NMI backtrace for cpu 1 [ 33.480082] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.14.0-rc5-03765-gd478a96 #1 [ 33.480082] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 33.480082] task: 80056000 ti: 80058000 task.ti: 80058000 [ 33.480082] EIP: 0060:[<8101973a>] EFLAGS: 00200286 CPU: 1 [ 33.480082] EIP is at native_safe_halt+0x5/0x7 [ 33.480082] EAX: ffffffed EBX: 80058000 ECX: 00200046 EDX: 00200046 [ 33.480082] ESI: 00000000 EDI: 80059fbc EBP: 80059f78 ESP: 80059f78 [ 33.480082] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 33.480082] CR0: 8005003b CR2: 775c3680 CR3: 13323000 CR4: 000006b0 [ 33.480082] DR0: 74238000 DR1: 74238000 DR2: 00000000 DR3: 00000000 [ 33.480082] DR6: ffff0ff0 DR7: 00000600 [ 33.480082] Stack: [ 33.480082] 80059f80 810078bc 80059f88 81007dd0 80059f94 8104df1c 01000800 80059fa8 [ 33.480082] 81012c77 00000001 00000000 80059fbc 00000000 00000000 00200002 00000000 [ 33.480082] 00000000 00000000 00000000 00000000 00000000 00000000 0000007b 0000007b [ 33.480082] Call Trace: [ 33.480082] [<810078bc>] default_idle+0x8/0xb [ 33.480082] [<81007dd0>] arch_cpu_idle+0x9/0xb [ 33.480082] [<8104df1c>] cpu_startup_entry+0xb2/0x134 [ 33.480082] [<81012c77>] start_secondary+0x202/0x20b [ 33.480082] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 89 c7 56 89 ce 53 53 89 55 git bisect start d478a960edf1ea61ca31a07a48a8771f043dba78 0414855fdc4a40da05221fc6062cccbc0c30f169 -- git bisect good 6c2191ad9b6225860eef70a77d300c3d5ad39182 # 21:36 22+ 0 Merge 'digsig/for-mimi' into devel-hourly-2014030618 git bisect bad 8d06d39b2d95c1c3f10ce3d1969e1e381935bbaf # 21:43 2- 3 Merge 'rcu/rcu/dev' into devel-hourly-2014030618 git bisect good 6845af00920e6b0d6e252c44784aaca3b0d2f42f # 21:50 26+ 0 Merge 'drm-intel/drm-intel-nightly' into devel-hourly-2014030618 git bisect good e5a16a9dbec8844bbd4b35ba52fef5c5f22e1e6a # 22:05 26+ 5 Merge 'ath6kl/ath-next-test' into devel-hourly-2014030618 git bisect good 5c709b0d8abc5dac4d7323fe6e3fdc8e6c4d0e59 # 22:10 26+ 5 Merge 'tip/x86/efi' into devel-hourly-2014030618 git bisect good fbe938e4f6c837ca17c284c001b43bb21d5ca389 # 22:12 26+ 0 Merge 'arm-soc/next/drivers' into devel-hourly-2014030618 git bisect good b46f358ae5e3558a9072f0354128d5749a584d01 # 22:18 26+ 2 rcutorture: Place kvm-test-1-run.sh output into res directory git bisect good f5604f67fe8cbd6f2088b20b9463f721aa613d4b # 22:25 26+ 0 Merge branch 'torture.2014.02.23a' into HEAD git bisect good dd35b81c341c685bafd15bc9a6bfa79584e07b07 # 22:30 26+ 7 torture: Rename RCU_BUILDONLY to TORTURE_BUILDONLY git bisect good 8864f0996e8b7f07a536aa07553c4461cd5f4211 # 22:33 26+ 0 torture: Make "--dryrun script" output self-sufficient git bisect good b2d777a6d9d64929b9701f1905fdeda8fc3cb16d # 22:37 26+ 0 rcutorture: Print negatives for SRCU counter wraparound git bisect good 4c79d7d89902a2e9c96b3db5115af52b3cf94da9 # 22:43 26+ 3 torture: Report diagnostics from qemu git bisect bad 1f9c26968e1a501f3a839603c9633f21abca65b3 # 22:49 4- 1 torture: Increase stutter-end intensity # first bad commit: [1f9c26968e1a501f3a839603c9633f21abca65b3] torture: Increase stutter-end intensity git bisect good 4c79d7d89902a2e9c96b3db5115af52b3cf94da9 # 02:18 180+ 3 torture: Report diagnostics from qemu git bisect bad d478a960edf1ea61ca31a07a48a8771f043dba78 # 02:18 0- 17 0day head guard for 'devel-hourly-2014030618' git bisect good 1fa4881b27aa785f667a3cf833a7b688b6d7066e # 02:31 180+ 0 Revert "torture: Increase stutter-end intensity" git bisect good c3bebc71c4bcdafa24b506adf0c1de3c1f77e2e0 # 05:41 180+ 2 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net Thanks, Fengguang