Processes hanging under heavy write loads

Processes hanging under heavy write loads

[Simon Kirby]

Hi all,

On an storage head box running 2.6.30, it's easy to see even sshd hang
when allocating memory to send a packet (eg: while watching "top"),
sometimes for several seconds.  The hung process detector, with the
timeout lowered a bit, spits out a backtrace such as:

INFO: task sshd:31015 blocked for more than 4 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sshd          D ffffffff8087b144     0 31015   3378
 ffff8801c5afd918 0000000000000086 0000000000000000 ffff880100b3e070
 ffff880100b3ddc0 ffff880183757080 ffff880100b3e070 ffffe2000dd81780
 ffff8801c5afd8f8 ffffffff8028c235 ffffe2000e578bd0 ffffffffffffffff
Call Trace:
 [<ffffffff8028c235>] ? determine_dirtyable_memory+0x15/0x30
 [<ffffffff806cf451>] __mutex_lock_slowpath+0xd1/0x150
 [<ffffffff806cf2be>] mutex_lock+0x1e/0x40
 [<ffffffff802c77dd>] shrink_icache_memory+0x7d/0x2b0
 [<ffffffff80291445>] shrink_slab+0x125/0x180
 [<ffffffff8029170a>] try_to_free_pages+0x26a/0x3e0
 [<ffffffff8028f5a0>] ? isolate_pages_global+0x0/0x290
 [<ffffffff8028af0f>] __alloc_pages_internal+0x19f/0x440
 [<ffffffff802c3a90>] ? pollwake+0x0/0x60
 [<ffffffff802ae061>] __slab_alloc+0x151/0x570
 [<ffffffff80617006>] ? __alloc_skb+0x46/0x170
 [<ffffffff802ae5b9>] kmem_cache_alloc+0xb9/0x110
 [<ffffffff80617006>] __alloc_skb+0x46/0x170
 [<ffffffff8064b041>] sk_stream_alloc_skb+0x41/0x110
 [<ffffffff8064c550>] tcp_sendmsg+0x2f0/0xad0
 [<ffffffff8060e920>] sock_aio_write+0xf0/0x100
 [<ffffffff802b3b61>] do_sync_write+0xf1/0x130
 [<ffffffff80256660>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff802453e2>] ? current_fs_time+0x22/0x30
 [<ffffffff80494028>] ? tty_ldisc_deref+0x58/0x70
 [<ffffffff802b4455>] vfs_write+0x175/0x180
 [<ffffffff802b4a30>] sys_write+0x50/0x90
 [<ffffffff8020be02>] system_call_fastpath+0x16/0x1b

...This mutex appears to be iprune_mutex, called from prune_icache in
fs/inode.c.  I watched this for a while, and all of the backtraces seem
to be the same.

Would it be a reasonable idea to convert this to a mutex_trylock since a
holder of it is trying to do the same work anyway?  I'm not sure what is
taking so long during heavy write sessions, but it has to be either
invalidate_inodes() or prune_icache().

The current behaviour is horrible to work with when non-guilty processes,
such as sshd, happen to get stuck on it...

Simon-