[PATCH -next] block: update io_ticks when io hang

[PATCH -next] block: update io_ticks when io hang

[Zhang Wensheng]

When the inflight IOs are slow and no new IOs are issued, we expect
iostat could manifest the IO hang problem. However after
commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
precise counting"), io_tick and time_in_queue will not be updated until
the end of IO, and the avgqu-sz and %util columns of iostat will be zero.

Because it has using stat.nsecs accumulation to express time_in_queue
which is not suitable to change, and may %util will express the status
better when io hang occur. To fix io_ticks, we use update_io_ticks and
inflight to update io_ticks when diskstats_show and part_stat_show
been called.

Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to less precise counting")
Signed-off-by: Zhang Wensheng <zhangwensheng5@huawei.com>
---
 block/genhd.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/block/genhd.c b/block/genhd.c
index 626c8406f21a..df0656ffb4ad 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
 	struct disk_stats stat;
 	unsigned int inflight;
 
-	part_stat_read_all(bdev, &stat);
 	if (queue_is_mq(q))
 		inflight = blk_mq_in_flight(q, bdev);
 	else
 		inflight = part_in_flight(bdev);
 
+	if (inflight)
+		update_io_ticks(bdev, jiffies, true);
+	part_stat_read_all(bdev, &stat);
 	return sprintf(buf,
 		"%8lu %8lu %8llu %8u "
 		"%8lu %8lu %8llu %8u "
@@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file *seqf, void *v)
 	xa_for_each(&gp->part_tbl, idx, hd) {
 		if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
 			continue;
-		part_stat_read_all(hd, &stat);
 		if (queue_is_mq(gp->queue))
 			inflight = blk_mq_in_flight(gp->queue, hd);
 		else
 			inflight = part_in_flight(hd);
 
+		if (inflight)
+			update_io_ticks(hd, jiffies, true);
+		part_stat_read_all(hd, &stat);
 		seq_printf(seqf, "%4d %7d %pg "
 			   "%lu %lu %lu %u "
 			   "%lu %lu %lu %u "
-- 
2.31.1

Re: [PATCH -next] block: update io_ticks when io hang

[zhangwensheng (E)]

friendly ping...

在 2022/1/28 14:22, zhangwensheng (E) 写道:
> friendly ping...
>
> 在 2022/1/25 17:19, Zhang Wensheng 写道:
>> When the inflight IOs are slow and no new IOs are issued, we expect
>> iostat could manifest the IO hang problem. However after
>> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
>> precise counting"), io_tick and time_in_queue will not be updated until
>> the end of IO, and the avgqu-sz and %util columns of iostat will be 
>> zero.
>>
>> Because it has using stat.nsecs accumulation to express time_in_queue
>> which is not suitable to change, and may %util will express the status
>> better when io hang occur. To fix io_ticks, we use update_io_ticks and
>> inflight to update io_ticks when diskstats_show and part_stat_show
>> been called.
>>
>> Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to 
>> less precise counting")
>> Signed-off-by: Zhang Wensheng <zhangwensheng5@huawei.com>
>> ---
>>   block/genhd.c | 8 ++++++--
>>   1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/block/genhd.c b/block/genhd.c
>> index 626c8406f21a..df0656ffb4ad 100644
>> --- a/block/genhd.c
>> +++ b/block/genhd.c
>> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
>>       struct disk_stats stat;
>>       unsigned int inflight;
>>   -    part_stat_read_all(bdev, &stat);
>>       if (queue_is_mq(q))
>>           inflight = blk_mq_in_flight(q, bdev);
>>       else
>>           inflight = part_in_flight(bdev);
>>   +    if (inflight)
>> +        update_io_ticks(bdev, jiffies, true);
>> +    part_stat_read_all(bdev, &stat);
>>       return sprintf(buf,
>>           "%8lu %8lu %8llu %8u "
>>           "%8lu %8lu %8llu %8u "
>> @@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file 
>> *seqf, void *v)
>>       xa_for_each(&gp->part_tbl, idx, hd) {
>>           if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
>>               continue;
>> -        part_stat_read_all(hd, &stat);
>>           if (queue_is_mq(gp->queue))
>>               inflight = blk_mq_in_flight(gp->queue, hd);
>>           else
>>               inflight = part_in_flight(hd);
>>   +        if (inflight)
>> +            update_io_ticks(hd, jiffies, true);
>> +        part_stat_read_all(hd, &stat);
>>           seq_printf(seqf, "%4d %7d %pg "
>>                  "%lu %lu %lu %u "
>>                  "%lu %lu %lu %u "

Re: [PATCH -next] block: update io_ticks when io hang

[zhangwensheng (E)]

friendly ping...

在 2022/2/8 10:12, zhangwensheng (E) 写道:
> friendly ping...
>
> 在 2022/1/28 14:22, zhangwensheng (E) 写道:
>> friendly ping...
>>
>> 在 2022/1/25 17:19, Zhang Wensheng 写道:
>>> When the inflight IOs are slow and no new IOs are issued, we expect
>>> iostat could manifest the IO hang problem. However after
>>> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
>>> precise counting"), io_tick and time_in_queue will not be updated until
>>> the end of IO, and the avgqu-sz and %util columns of iostat will be 
>>> zero.
>>>
>>> Because it has using stat.nsecs accumulation to express time_in_queue
>>> which is not suitable to change, and may %util will express the status
>>> better when io hang occur. To fix io_ticks, we use update_io_ticks and
>>> inflight to update io_ticks when diskstats_show and part_stat_show
>>> been called.
>>>
>>> Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to 
>>> less precise counting")
>>> Signed-off-by: Zhang Wensheng <zhangwensheng5@huawei.com>
>>> ---
>>>   block/genhd.c | 8 ++++++--
>>>   1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/block/genhd.c b/block/genhd.c
>>> index 626c8406f21a..df0656ffb4ad 100644
>>> --- a/block/genhd.c
>>> +++ b/block/genhd.c
>>> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
>>>       struct disk_stats stat;
>>>       unsigned int inflight;
>>>   -    part_stat_read_all(bdev, &stat);
>>>       if (queue_is_mq(q))
>>>           inflight = blk_mq_in_flight(q, bdev);
>>>       else
>>>           inflight = part_in_flight(bdev);
>>>   +    if (inflight)
>>> +        update_io_ticks(bdev, jiffies, true);
>>> +    part_stat_read_all(bdev, &stat);
>>>       return sprintf(buf,
>>>           "%8lu %8lu %8llu %8u "
>>>           "%8lu %8lu %8llu %8u "
>>> @@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file 
>>> *seqf, void *v)
>>>       xa_for_each(&gp->part_tbl, idx, hd) {
>>>           if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
>>>               continue;
>>> -        part_stat_read_all(hd, &stat);
>>>           if (queue_is_mq(gp->queue))
>>>               inflight = blk_mq_in_flight(gp->queue, hd);
>>>           else
>>>               inflight = part_in_flight(hd);
>>>   +        if (inflight)
>>> +            update_io_ticks(hd, jiffies, true);
>>> +        part_stat_read_all(hd, &stat);
>>>           seq_printf(seqf, "%4d %7d %pg "
>>>                  "%lu %lu %lu %u "
>>>                  "%lu %lu %lu %u "

Re: [PATCH -next] block: update io_ticks when io hang

[Jens Axboe]

On Tue, 25 Jan 2022 17:19:38 +0800, Zhang Wensheng wrote:
> When the inflight IOs are slow and no new IOs are issued, we expect
> iostat could manifest the IO hang problem. However after
> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
> precise counting"), io_tick and time_in_queue will not be updated until
> the end of IO, and the avgqu-sz and %util columns of iostat will be zero.
> 
> Because it has using stat.nsecs accumulation to express time_in_queue
> which is not suitable to change, and may %util will express the status
> better when io hang occur. To fix io_ticks, we use update_io_ticks and
> inflight to update io_ticks when diskstats_show and part_stat_show
> been called.
> 
> [...]

Applied, thanks!

[1/1] block: update io_ticks when io hang
      commit: 62847731488f59971413484005a7fb2772cb9249

Best regards,
-- 
Jens Axboe

Re: [PATCH -next] block: update io_ticks when io hang

[Jens Axboe]

On 2/16/22 8:07 PM, zhangwensheng (E) wrote:
> hi, Jens:
> 
> I am sorry that there are some problems with the patch below:
> 
> 
> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
>       struct disk_stats stat;
>       unsigned int inflight;
>   -    part_stat_read_all(bdev, &stat);
>       if (queue_is_mq(q))
>           inflight = blk_mq_in_flight(q, bdev);
>       else
>           inflight = part_in_flight(bdev);
>   +    if (inflight)
> +        update_io_ticks(bdev, jiffies, true);
> +    part_stat_read_all(bdev, &stat);
>       return sprintf(buf,
>           "%8lu %8lu %8llu %8u "
>           "%8lu %8lu %8llu %8u "
> 
> It shoule use part_stat_lock() & part_stat_unlock() to protect update_io_ticks(),
> 
> because it operates on per-cpu variables.  And I will fix it in V2.

I'll drop it, please resend a fixed and tested one.

-- 
Jens Axboe