Hi Hong Zhiguo, The WARNINGs introduced by this commit are still in linux-next 20140307. commit 2c575026fae6e63771bd2a4c1d407214a8096a89 Author: Hong Zhiguo AuthorDate: Wed Nov 20 10:35:05 2013 -0700 Commit: Jens Axboe CommitDate: Wed Nov 20 15:33:04 2013 -0700 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. Signed-off-by: Hong Zhiguo Acked-by: Tejun Heo Cc: stable@kernel.org Signed-off-by: Jens Axboe +------------------------------------------------------------+------------+---------------+ | | 2c575026fa | next-20140306 | +------------------------------------------------------------+------------+---------------+ | boot_successes | 0 | 0 | | boot_failures | 20 | 19 | | WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable() | 20 | | | inconsistent_IN-SOFTIRQ-W-SOFTIRQ-ON-W_usage | 20 | | | backtrace:vfs_read | 20 | 18 | | backtrace:SyS_read | 20 | 18 | | backtrace:smpboot_thread_fn | 14 | | | page_allocation_failure:order:,mode | 1 | | | backtrace:vmalloc | 1 | | | backtrace:SyS_init_module | 1 | | | WARNING:CPU:PID:at_kernel/softirq.c:__local_bh_enable_ip() | 0 | 16 | | inconsistent_IN-HARDIRQ-W-HARDIRQ-ON-W_usage | 0 | 19 | | Out_of_memory:Kill_process | 0 | 1 | | backtrace:do_mount | 0 | 1 | | backtrace:SyS_mount | 0 | 1 | | BUG:kernel_boot_hang | 0 | 1 | +------------------------------------------------------------+------------+---------------+ [ 8.215262] ------------[ cut here ]------------ [ 8.215318] WARNING: CPU: 0 PID: 287 at /kernel-tests/linux4/kernel/softirq.c:156 local_bh_enable+0xbf/0x20e() [ 8.215331] Modules linked in: [ 8.215338] CPU: 0 PID: 287 Comm: blkid Not tainted 3.12.0-10878-g2c57502 #109 [ 8.215340] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 8.215355] 00000000 c53a5bd0 c152b622 c53a5be8 c0259135 c025fdb2 c068e196 00000000 [ 8.215364] c56dac00 c53a5bf8 c02591a1 00000009 00000000 c53a5c04 c025fdb2 00000000 [ 8.215372] c53a5cac c068e196 00000001 00000000 c56d6000 c57ba840 c56dac00 c068002f [ 8.215374] Call Trace: [ 8.215403] [] dump_stack+0x40/0x5e [ 8.215408] [] warn_slowpath_common+0xa9/0xea [ 8.215413] [] ? local_bh_enable+0xbf/0x20e [ 8.215435] [] ? __cfq_set_active_queue+0x110/0x27d [ 8.215440] [] warn_slowpath_null+0x2b/0x3d [ 8.215444] [] local_bh_enable+0xbf/0x20e [ 8.215449] [] __cfq_set_active_queue+0x110/0x27d [ 8.215454] [] ? add_disk+0x12c/0x781 [ 8.215459] [] ? cfq_resort_rr_list+0x49/0x5b [ 8.215463] [] ? cfq_add_cfqq_rr+0xde/0xf3 [ 8.215468] [] ? cfq_update_idle_window.isra.78+0xf2/0x1f4 [ 8.215480] [] ? sched_clock_cpu+0x1c4/0x1f9 [ 8.215487] [] ? trace_hardirqs_off+0x27/0x37 [ 8.215492] [] ? local_clock+0xff/0x176 [ 8.215497] [] cfq_dispatch_requests+0xb92/0x11b0 [ 8.215504] [] ? kvm_clock_read+0x30/0x55 [ 8.215515] [] ? sched_clock+0x3e/0x50 [ 8.215520] [] ? cfq_insert_request+0x3c9/0x869 [ 8.215525] [] ? trace_hardirqs_off+0x27/0x37 [ 8.215530] [] ? __lock_acquire+0x15a4/0x16ca [ 8.215538] [] ? list_sort+0x1fb/0x253 [ 8.215547] [] blk_peek_request+0x9e/0x31a [ 8.215566] [] scsi_request_fn+0x62/0x89e [ 8.215571] [] ? blk_flush_plug_list+0x2a4/0x447 [ 8.215577] [] __blk_run_queue+0x55/0x6c [ 8.215582] [] queue_unplugged.isra.29+0x40/0x6a [ 8.215586] [] blk_flush_plug_list+0x366/0x447 [ 8.215591] [] blk_finish_plug+0x1b/0x51 [ 8.215607] [] __do_page_cache_readahead+0x314/0x364 [ 8.215614] [] force_page_cache_readahead+0x9e/0xea [ 8.215620] [] page_cache_sync_readahead+0x5d/0x91 [ 8.215624] [] generic_file_aio_read+0x393/0xb7e [ 8.215638] [] blkdev_aio_read+0x95/0xb1 [ 8.215646] [] do_sync_read+0x5a/0xab [ 8.215651] [] ? do_sync_write+0xab/0xab [ 8.215655] [] vfs_read+0xd3/0x1ee [ 8.215659] [] SyS_read+0x73/0xe8 [ 8.215668] [] sysenter_do_call+0x12/0x3c [ 8.215672] ---[ end trace 6f72697fe2d05017 ]--- [ 8.215727] [ 8.215729] ================================= [ 8.215730] [ INFO: inconsistent lock state ] [ 8.215733] 3.12.0-10878-g2c57502 #109 Tainted: G W [ 8.215735] --------------------------------- [ 8.215737] inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage. [ 8.215740] blkid/287 [HC0[0]:SC0[0]:HE1:SE1] takes: [ 8.215749] (&(&q->__queue_lock)->rlock){+.?...}, at: [] blk_flush_plug_list+0x2a4/0x447 [ 8.215750] {IN-SOFTIRQ-W} state was registered at: [ 8.215756] [] __lock_acquire+0xeab/0x16ca [ 8.215760] [] lock_acquire+0xe6/0x147 [ 8.215767] [] _raw_spin_lock+0x4e/0xee [ 8.215771] [] scsi_device_unbusy+0xee/0x134 [ 8.215781] [] scsi_finish_command+0x2b/0x107 [ 8.215785] [] scsi_softirq_done+0x1b5/0x1cb [ 8.215791] [] blk_done_softirq+0x10e/0x133 [ 8.215796] [] __do_softirq+0x114/0x318 [ 8.215798] irq event stamp: 2152 [ 8.215803] hardirqs last enabled at (2149): [] _raw_spin_unlock_irq+0x5a/0xba [ 8.215808] hardirqs last disabled at (2150): [] blk_flush_plug_list+0x1ff/0x447 [ 8.215813] softirqs last enabled at (2152): [] __cfq_set_active_queue+0x110/0x27d [ 8.215818] softirqs last disabled at (2151): [] __cfq_set_active_queue+0xd9/0x27d [ 8.215820] [ 8.215820] other info that might help us debug this: [ 8.215824] Possible unsafe locking scenario: [ 8.215824] [ 8.215825] CPU0 [ 8.215827] ---- [ 8.215830] lock(&(&q->__queue_lock)->rlock); [ 8.215831] [ 8.215833] lock(&(&q->__queue_lock)->rlock); [ 8.215834] [ 8.215834] *** DEADLOCK *** [ 8.215834] [ 8.215837] 1 lock held by blkid/287: [ 8.215844] #0: (&(&q->__queue_lock)->rlock){+.?...}, at: [] blk_flush_plug_list+0x2a4/0x447 [ 8.215846] [ 8.215846] stack backtrace: [ 8.215849] CPU: 0 PID: 287 Comm: blkid Tainted: G W 3.12.0-10878-g2c57502 #109 [ 8.215851] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 8.215859] 00000000 c53a5b78 c152b622 c53a5b98 c15263b3 c1a4e107 c1a4dedf c1a4ed62 [ 8.215867] 00000006 c535a8b0 00000040 c53a5bc0 c02a5d9e 00000006 00000000 c02a2e66 [ 8.215875] 00000004 c535a530 c535a8b0 c535a530 00000000 c53a5be0 c02a92e0 e2d05017 [ 8.215876] Call Trace: [ 8.215881] [] dump_stack+0x40/0x5e [ 8.215891] [] print_usage_bug+0x493/0x4ad [ 8.215896] [] mark_lock+0x575/0xa2e [ 8.215901] [] ? print_irq_inversion_bug+0x374/0x374 [ 8.215905] [] mark_held_locks+0x163/0x199 [ 8.215910] [] ? local_bh_enable+0x1c2/0x20e [ 8.215915] [] trace_hardirqs_on_caller+0x36b/0x3b1 [ 8.215920] [] ? __cfq_set_active_queue+0x110/0x27d [ 8.215924] [] trace_hardirqs_on+0x27/0x37 [ 8.215928] [] local_bh_enable+0x1c2/0x20e [ 8.215932] [] __cfq_set_active_queue+0x110/0x27d [ 8.215937] [] ? add_disk+0x12c/0x781 [ 8.215941] [] ? cfq_resort_rr_list+0x49/0x5b [ 8.215945] [] ? cfq_add_cfqq_rr+0xde/0xf3 [ 8.215949] [] ? cfq_update_idle_window.isra.78+0xf2/0x1f4 [ 8.215953] [] ? sched_clock_cpu+0x1c4/0x1f9 [ 8.215958] [] ? trace_hardirqs_off+0x27/0x37 [ 8.215962] [] ? local_clock+0xff/0x176 [ 8.215966] [] cfq_dispatch_requests+0xb92/0x11b0 [ 8.215970] [] ? kvm_clock_read+0x30/0x55 [ 8.215975] [] ? sched_clock+0x3e/0x50 [ 8.215979] [] ? cfq_insert_request+0x3c9/0x869 [ 8.215984] [] ? trace_hardirqs_off+0x27/0x37 [ 8.215988] [] ? __lock_acquire+0x15a4/0x16ca [ 8.215992] [] ? list_sort+0x1fb/0x253 [ 8.215997] [] blk_peek_request+0x9e/0x31a [ 8.216001] [] scsi_request_fn+0x62/0x89e [ 8.216006] [] ? blk_flush_plug_list+0x2a4/0x447 [ 8.216010] [] __blk_run_queue+0x55/0x6c [ 8.216015] [] queue_unplugged.isra.29+0x40/0x6a [ 8.216019] [] blk_flush_plug_list+0x366/0x447 [ 8.216024] [] blk_finish_plug+0x1b/0x51 [ 8.216029] [] __do_page_cache_readahead+0x314/0x364 [ 8.216035] [] force_page_cache_readahead+0x9e/0xea [ 8.216040] [] page_cache_sync_readahead+0x5d/0x91 [ 8.216044] [] generic_file_aio_read+0x393/0xb7e git bisect start v3.13 v3.12 -- git bisect good 3bad8bb5cd3048a67df43ac6b1e2f191f19d9ff0 # 05:30 20+ 0 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6 git bisect bad 754ac45745711a7ce0067087e5b55a9c277952ad # 05:52 0- 4 Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input git bisect good d644c026e6b21959af1f86b15206ae26c64c2e51 # 06:06 20+ 0 Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 git bisect bad 002acf1fc16cf60e60345bd68e03734628505b83 # 06:19 0- 2 Merge tag 'pm-3.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect good da1965d353c7266fce4fe6c658f2096c88a8bebc # 06:31 20+ 0 Merge tag 'pinctrl-v3.13-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl git bisect good ef1e4e32d595d3e6c9a6d3d2956f087d5886c5e5 # 06:45 20+ 0 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs git bisect bad 5ee540613db504a10e15fafaf4c08cac96aa1823 # 06:58 0- 3 Merge branch 'for-linus' of git://git.kernel.dk/linux-block git bisect good 29be6345bbaec8502a70c4e2204d5818b48c4e8f # 07:06 20+ 0 Merge tag 'nfs-for-3.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs git bisect bad e345d767f6530ec9cb0aabab7ea248072a9c6975 # 07:26 0- 5 Merge branch 'stable/for-jens-3.13-take-two' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-linus git bisect bad c170bbb45febc03ac4d34ba2b8bb55e06104b7e7 # 07:35 0- 3 block: submit_bio_wait() conversions git bisect bad 2c575026fae6e63771bd2a4c1d407214a8096a89 # 07:43 0- 4 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. # first bad commit: [2c575026fae6e63771bd2a4c1d407214a8096a89] Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. git bisect good 82023bb7f75b0052f40d3e74169d191c3e4e6286 # 08:02 60+ 0 Merge tag 'pm+acpi-2-3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad 0ffb2fe7b9c30082876fa3a17da018bf0632cf03 # 08:02 0- 19 Add linux-next specific files for 20140306 git bisect good 538959a390127125581a4fa2f2f7e035b82fa421 # 08:25 60+ 37 Revert "Update of blkg_stat and blkg_rwstat may happen in bh context." git bisect bad b053940df41808f0f27568eb36820d10a8a987f8 # 08:34 0- 2 ARC: Use correct PTAG register for icache flush git bisect bad 1b0a7e3263168a06d3858798e48c5a21d1c78d3c # 08:44 1- 16 Add linux-next specific files for 20140307 Thanks, Fengguang