RE: Random high CPU utilization in blk-mq with the none scheduler

[Dexuan Cui <decui@microsoft.com>]

> From: Ming Lei <ming.lei@redhat.com>
> Sent: Sunday, December 12, 2021 11:38 PM

Ming, thanks so much for the detailed analysis!

> From the log:
> 
> 1) dm-mpath:
> - queue depth: 2048
> - busy: 848, and 62 of them are in sw queue, so run queue is often
>   caused
> - nr_hw_queues: 1
> - dm-2 is in use, and dm-1/dm-3 is idle
> - dm-2's dispatch busy is 8, that should be the reason why excessive CPU
> usage is observed when flushing plug list without commit dc5fc361d891 in
> which hctx->dispatch_busy is just bypassed
> 
> 2) iscsi
> - dispatch_busy is 0
> - nr_hw_queues: 1
> - queue depth: 113
> - busy=~33, active_queues is 3, so each LUN/iscsi host is saturated
> - 23 active LUNs, 23 * 33 = 759 in-flight commands
> 
> The high CPU utilization may be caused by:
> 
> 1) big queue depth of dm mpath, the situation may be improved much if it
> is reduced to 1024 or 800. The max allowed inflight commands from iscsi
> hosts can be figured out, if dm's queue depth is much more than this number,
> the extra commands need to dispatch, and run queue can be scheduled
> immediately, so high CPU utilization is caused.

I think you're correct:
with dm_mod.dm_mq_queue_depth=256, the max CPU utilization is 8%.
with dm_mod.dm_mq_queue_depth=400, the max CPU utilization is 12%. 
with dm_mod.dm_mq_queue_depth=800, the max CPU utilization is 88%.

The performance with queue_depth=800 is poor.
The performance with queue_depth=400 is good.
The performance with queue_depth=256 is also good, and there is only a 
small drop comared with the 400 case.

> 2) single hw queue, so contention should be big, which should be avoided
> in big machine, nvme-tcp might be better than iscsi here
> 
> 3) iscsi io latency is a bit big
> 
> Even CPU utilization is reduced by commit dc5fc361d891, io performance
> can't be good too with v5.16-rc, I guess.
> 
> Thanks,
> Ming

Actually the I/O performance of v5.16-rc4 (commit dc5fc361d891 is included)
is good -- it's about the same as the case where v5.16-rc4 + reverting
dc5fc361d891 + dm_mod.dm_mq_queue_depth=400 (or 256).

Thanks,
Dexuan