Greetings, I got the below dmesg and the first bad commit is commit b70aa55ec9532d5f5d3e367e4cad8997b1135f86 Author: Jens Axboe Date: Fri May 24 20:22:33 2013 +0200 virtio_blk: blk-mq support Pretend we have 4 issue queues. Signed-off-by: Jens Axboe [ 277.857736] blk-mq: CPU -> queue map [ 277.871142] CPU 0 -> Queue 0 [ 277.934080] BUG: unable to handle kernel paging request at ff7f5784 [ 277.934080] IP: [<812fe4c7>] blk_mq_init_queue+0x5fa/0x6f7 [ 277.934080] *pde = 06814067 *pte = 00000000 [ 277.934080] Oops: 0002 [#1] SMP DEBUG_PAGEALLOC [ 277.934080] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-rc3-00011-g5754ab5 #98 [ 277.934080] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 277.934080] task: 8d448000 ti: 8d444000 task.ti: 8d444000 [ 277.934080] EIP: 0060:[<812fe4c7>] EFLAGS: 00000203 CPU: 0 [ 277.934080] EIP is at blk_mq_init_queue+0x5fa/0x6f7 [ 277.934080] EAX: 86906000 EBX: ff7f5740 ECX: 868f9088 EDX: 00000001 [ 277.934080] ESI: 868c5550 EDI: 00000002 EBP: 8d445e3c ESP: 8d445df0 [ 277.934080] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 277.934080] CR0: 8005003b CR2: ff7f5784 CR3: 02941000 CR4: 00000690 [ 277.934080] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 277.934080] DR6: 00000000 DR7: 00000000 [ 277.934080] Stack: [ 277.934080] 0000000a 86a17980 00000004 868eb780 00000000 00000000 869066bc f3c6a740 [ 277.934080] ffffffff 00000d80 868af420 0000000a 8e5c8280 00000004 82772fec 86a18700 [ 277.934080] 868eb780 868ee400 00000c54 8d445e7c 81424cc2 86e6e000 00000000 86e6e000 [ 277.934080] Call Trace: [ 277.934080] [<81424cc2>] virtblk_probe+0x165/0x579 [ 277.934080] [<813979f6>] ? vring_transport_features+0x12/0x22 [ 277.934080] [<813966da>] virtio_dev_probe+0xae/0xe3 [ 277.934080] [<813f8059>] driver_probe_device+0xf2/0x26e [ 277.934080] [<813f8223>] __driver_attach+0x4e/0x6a [ 277.934080] [<813f8223>] ? __driver_attach+0x4e/0x6a [ 277.934080] [<813f6a02>] bus_for_each_dev+0x3e/0x68 [ 277.934080] [<813f7dc6>] driver_attach+0x14/0x16 [ 277.934080] [<813f81d5>] ? driver_probe_device+0x26e/0x26e [ 277.934080] [<813f7098>] bus_add_driver+0xd9/0x1da [ 277.934080] [<8130f114>] ? kset_find_obj+0x5d/0x65 [ 277.934080] [<813f8519>] driver_register+0x74/0xef [ 277.934080] [<813f8519>] ? driver_register+0x74/0xef [ 277.934080] [<81f88190>] ? mutex_unlock+0x8/0xa [ 277.934080] [<8139676d>] register_virtio_driver+0x1d/0x1f [ 277.934080] [<828bb2d1>] init+0x49/0x76 [ 277.934080] [<828bb288>] ? mm_init+0x151/0x151 [ 277.934080] [<8288e9d4>] do_one_initcall+0x95/0x126 [ 277.934080] [<828a85d6>] ? init_nls_iso8859_4+0xf/0xf [ 277.934080] [<8288eb82>] kernel_init_freeable+0x11d/0x1a2 [ 277.934080] [<8288eb82>] ? kernel_init_freeable+0x11d/0x1a2 [ 277.934080] [<81f67d2c>] kernel_init+0x8/0xb4 [ 277.934080] [<81f8ac17>] ret_from_kernel_thread+0x1b/0x28 [ 277.934080] [<81f67d24>] ? rest_init+0x118/0x118 [ 277.934080] Code: ff 8b 86 c0 00 00 00 8b 3c 98 3b 9e c4 00 00 00 72 b6 e9 e4 00 00 00 8b 8e b0 00 00 00 89 fa 89 f0 47 ff 51 04 8b 90 a8 00 00 00 <89> 53 44 8b 90 a8 00 00 00 8b 88 ac 00 00 00 89 1c 91 42 89 90 [ 277.934080] EIP: [<812fe4c7>] blk_mq_init_queue+0x5fa/0x6f7 SS:ESP 0068:8d445df0 [ 277.934080] CR2: 00000000ff7f5784 [ 277.934080] ---[ end trace c7ec3499622b2c6b ]--- git bisect start 5754ab54c1b28888e9700550585c5729fccbd15f e4aa937ec75df0eea0bee03bffa3303ad36c986b -- git bisect good dc2978537d4353ef173e99b28287effeeb019dba # 662 2013-06-03 15:20:31 null_blk: multi queue aware block test driver git bisect bad a9e8919f71fcbd545812f6bcdc53eaf7747a5281 # 654 2013-06-03 15:21:05 direct-io: only inc/dec inode->i_dio_count for file systems git bisect bad b70aa55ec9532d5f5d3e367e4cad8997b1135f86 # 676 2013-06-03 15:21:41 virtio_blk: blk-mq support git bisect good 00060f37d61cce84b05ab265a1d3532fb83b1002 # 3132 2013-06-03 15:22:15 mtip32xx: basic mq support git bisect good 00060f37d61cce84b05ab265a1d3532fb83b1002 # 3134 2013-06-03 15:23:02 mtip32xx: basic mq support git bisect bad 5754ab54c1b28888e9700550585c5729fccbd15f # 522 2013-06-03 15:23:46 blk-mq: timeout fixes git bisect good 202b60e0678542e0a26afe501e83d436904e9ba0 # 1431 2013-06-03 18:14:49 Revert "virtio_blk: blk-mq support" git bisect good 6318c0ec14d5f7aea22987223e1a99dbda7b1fd0 # 1431 2013-06-03 18:55:32 Add linux-next specific files for 20130603 Thanks, Fengguang