Can LVM block I/O and hang a system?

Can LVM block I/O and hang a system?

[Maurice Volaski]

A working system begins hanging and it seems to be stuck on I/O 
processes that use ext3 partitions that are running on top of LVM. 
The system is AMD 64-bit running Gentoo. Kernel is Gentoo 2.6.22-r3 
and LVM lvm2-2.02.27. Here is the disk setup:

Boot disk, attached to motherboard via SATA
1) some partitions accessed via ext3 -> hardware partition.
2) some partitions accessed via ext3 -> drbd, which is version 8.0.5, 
-> hardware partition.

External SATA-SCSI RAID, attached to via an LSI Logic card,
3) one partition accessed via ext3 -> drbd -> hardware partition.
4) some partitions accessed via ext3 -> LVM -> drbd -> hardware partition.

On repeated reboots, #1) boots fine, and I can fsck #2) no problem. I 
can also fsck #3, but the fsck processes on #4, which all are trying 
to recover the journals, just seem to not do anything. There is no 
evidence of I/O and there are no errors reported anywhere. The frozen 
fsck processes cannot even be killed and the system ignores the 
shutdown command.

That the hanging fsck processes are all occurring on just the LVM 
partitions seems to imply that LVM is responsible.

drbd had been unattached to its peer during this time, and when I 
reattached it, it had no trouble syncing to the peer. That system, 
which should basically be identical, however, has no trouble running 
running fsck everywhere. I'm not sure, though, if that lets LVM off 
the hook.
-- 

Maurice Volaski, mvolaski@aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

Re: Can LVM block I/O and hang a system?

[Andrew Morton]

On Wed, 12 Sep 2007 03:09:14 -0400 Maurice Volaski <mvolaski@aecom.yu.edu> wrote:

> A working system begins hanging and it seems to be stuck on I/O 
> processes that use ext3 partitions that are running on top of LVM. 
> The system is AMD 64-bit running Gentoo. Kernel is Gentoo 2.6.22-r3 
> and LVM lvm2-2.02.27. Here is the disk setup:
> 
> Boot disk, attached to motherboard via SATA
> 1) some partitions accessed via ext3 -> hardware partition.
> 2) some partitions accessed via ext3 -> drbd, which is version 8.0.5, 
> -> hardware partition.
> 
> External SATA-SCSI RAID, attached to via an LSI Logic card,
> 3) one partition accessed via ext3 -> drbd -> hardware partition.
> 4) some partitions accessed via ext3 -> LVM -> drbd -> hardware partition.
> 
> On repeated reboots, #1) boots fine, and I can fsck #2) no problem. I 
> can also fsck #3, but the fsck processes on #4, which all are trying 
> to recover the journals, just seem to not do anything. There is no 
> evidence of I/O and there are no errors reported anywhere. The frozen 
> fsck processes cannot even be killed and the system ignores the 
> shutdown command.
> 
> That the hanging fsck processes are all occurring on just the LVM 
> partitions seems to imply that LVM is responsible.
> 
> drbd had been unattached to its peer during this time, and when I 
> reattached it, it had no trouble syncing to the peer. That system, 
> which should basically be identical, however, has no trouble running 
> running fsck everywhere. I'm not sure, though, if that lets LVM off 
> the hook.

Next time it hangs, please do

dmesg -c
echo t > /proc/sysrq-trigger
dmesg -s 1000000 > foo

and send foo.