Hyper-V stalls on device errors

Hyper-V stalls on device errors

[Sitsofe Wheeler]

-- 
Sitsofe | http://sucs.org/~sits/

Re: Hyper-V stalls on device errors

[Sitsofe Wheeler]

Apologies for the previous empty mail.

While testing a Windows 2012 host with a Fedora 18 guest running a 3.9
kernel I've found that Hyper-v will stall all access to
(para)virtualised disk devices when an underlying disk device returns an
error. Every ten seconds a tiny bit of I/O goes through before being
stalled again and it plays havoc with asynchronous I/O to disk devices
too.

To produce this I created a device mapper device with a single error in
it by using

dd if=/dev/zero of=/tmp/fakeblock0 bs=100M count=1
losetup --find --show /tmp/fakeblock0
# Assuming losetup uses /dev/loop0
cat << EOF | dmsetup create oneerror
0 13443 linear /dev/loop0 0
13443 1 error
13444 191356 linear /dev/loop0 0
EOF

After installing scsi-target-utils the /dev/mapper/oneerror device was
then turned into a iSCSI target by adding

<target iqn.2013-04.com.stormagic:oneerror>
     backing-store /dev/mapper/oneerror
     write-cache off
</target>

to /etc/tgt/targets.conf . The iSCSI target service was started with
systemctl start tgtd.service (watch out for
https://bugzilla.redhat.com/show_bug.cgi?id=848942 and you may need to
disable the firewall by using systemctl stop firewalld.service ).

The Windows 2012 iSCSI initiator was used to add the target to the
machine with the hypervisor (the usual discovery should work to the
Linux box serving the SCSI target). Once done, this disk was then added
to the Linux guest's Hyper-V settings via the SCSI controller. A spare
IDE controller disk was also added.

In the Linux guest a badblock run was started on the spare IDE disk
block device so that I/O was visible. A 
dd if=/dev/zero of=/dev/sdc oflag=direct
(where /dev/sdc is the erroring block device that was added earlier) was
then done to trigger the access of the bad sector.

The following appeared in dmesg:

[  160.718836] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  170.991312] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  181.039597] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  191.081242] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  201.116790] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  211.127741] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
[  221.140338] sd 3:0:0:2: [sdc] Unhandled error code
[  221.140346] sd 3:0:0:2: [sdc]  
[  221.140349] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  221.140352] sd 3:0:0:2: [sdc] CDB: 
[  221.140354] Write(10): 2a 00 00 00 34 00 00 01 00 00
[  221.140366] end_request: critical target error, dev sdc, sector 13312

A Fedora 18 guest on VMWare ESXi returned the error in under a second
and only had the following in dmesg:

[  293.917383] sd 2:0:1:0: [sdb] Unhandled sense code
[  293.917391] sd 2:0:1:0: [sdb]
[  293.917394] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  293.917408] sd 2:0:1:0: [sdb]
[  293.917414] Sense Key : Medium Error [current]
[  293.917418] sd 2:0:1:0: [sdb]
[  293.917421] Add. Sense: Unrecovered read error
[  293.917424] sd 2:0:1:0: [sdb] CDB:
[  293.917428] Write(10): 2a 00 00 00 34 00 00 04 00 00
[  293.917436] end_request: critical target error, dev sdb, sector 13312

The stalls do not occur when the bad block device is created directly in
the Linux guest.  From the previous log messages it looks like Hyper-V
is trying for up to a minute before returning an error and the I/O
stalls to separate (but virtualised) devices on different buses looks
like an unintended side effect...

-- 
Sitsofe | http://sucs.org/~sits/

RE: Hyper-V stalls on device errors

[KY Srinivasan]

Thanks Sitsofe; we will look into this.

Regards,

K. Y

> -----Original Message-----
> From: Sitsofe Wheeler [mailto:sitsofe@yahoo.com]
> Sent: Tuesday, April 30, 2013 12:12 PM
> To: KY Srinivasan; Haiyang Zhang
> Cc: devel@linuxdriverproject.org; James E.J. Bottomley; linux-
> kernel@vger.kernel.org
> Subject: Re: Hyper-V stalls on device errors
> 
> Apologies for the previous empty mail.
> 
> While testing a Windows 2012 host with a Fedora 18 guest running a 3.9
> kernel I've found that Hyper-v will stall all access to
> (para)virtualised disk devices when an underlying disk device returns an
> error. Every ten seconds a tiny bit of I/O goes through before being
> stalled again and it plays havoc with asynchronous I/O to disk devices
> too.
> 
> To produce this I created a device mapper device with a single error in
> it by using
> 
> dd if=/dev/zero of=/tmp/fakeblock0 bs=100M count=1
> losetup --find --show /tmp/fakeblock0
> # Assuming losetup uses /dev/loop0
> cat << EOF | dmsetup create oneerror
> 0 13443 linear /dev/loop0 0
> 13443 1 error
> 13444 191356 linear /dev/loop0 0
> EOF
> 
> After installing scsi-target-utils the /dev/mapper/oneerror device was
> then turned into a iSCSI target by adding
> 
> <target iqn.2013-04.com.stormagic:oneerror>
>      backing-store /dev/mapper/oneerror
>      write-cache off
> </target>
> 
> to /etc/tgt/targets.conf . The iSCSI target service was started with
> systemctl start tgtd.service (watch out for
> https://bugzilla.redhat.com/show_bug.cgi?id=848942 and you may need to
> disable the firewall by using systemctl stop firewalld.service ).
> 
> The Windows 2012 iSCSI initiator was used to add the target to the
> machine with the hypervisor (the usual discovery should work to the
> Linux box serving the SCSI target). Once done, this disk was then added
> to the Linux guest's Hyper-V settings via the SCSI controller. A spare
> IDE controller disk was also added.
> 
> In the Linux guest a badblock run was started on the spare IDE disk
> block device so that I/O was visible. A
> dd if=/dev/zero of=/dev/sdc oflag=direct
> (where /dev/sdc is the erroring block device that was added earlier) was
> then done to trigger the access of the bad sector.
> 
> The following appeared in dmesg:
> 
> [  160.718836] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  170.991312] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  181.039597] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  191.081242] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  201.116790] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  211.127741] hv_storvsc vmbus_0_12: cmd 0x2a scsi status 0x2 srb status 0x4
> [  221.140338] sd 3:0:0:2: [sdc] Unhandled error code
> [  221.140346] sd 3:0:0:2: [sdc]
> [  221.140349] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
> [  221.140352] sd 3:0:0:2: [sdc] CDB:
> [  221.140354] Write(10): 2a 00 00 00 34 00 00 01 00 00
> [  221.140366] end_request: critical target error, dev sdc, sector 13312
> 
> A Fedora 18 guest on VMWare ESXi returned the error in under a second
> and only had the following in dmesg:
> 
> [  293.917383] sd 2:0:1:0: [sdb] Unhandled sense code
> [  293.917391] sd 2:0:1:0: [sdb]
> [  293.917394] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [  293.917408] sd 2:0:1:0: [sdb]
> [  293.917414] Sense Key : Medium Error [current]
> [  293.917418] sd 2:0:1:0: [sdb]
> [  293.917421] Add. Sense: Unrecovered read error
> [  293.917424] sd 2:0:1:0: [sdb] CDB:
> [  293.917428] Write(10): 2a 00 00 00 34 00 00 04 00 00
> [  293.917436] end_request: critical target error, dev sdb, sector 13312
> 
> The stalls do not occur when the bad block device is created directly in
> the Linux guest.  From the previous log messages it looks like Hyper-V
> is trying for up to a minute before returning an error and the I/O
> stalls to separate (but virtualised) devices on different buses looks
> like an unintended side effect...
> 
> --
> Sitsofe | http://sucs.org/~sits/
>