Re: request baset device mapper in Linux

[Lukas Hejtmanek <xhejtman@ics.muni.cz>]

On Thu, Jul 21, 2011 at 08:11:32PM +0900, Kiyoshi Ueda wrote:
> I don't understand why you are saying request-based device-mapper makes
> serious troubles.
> BIO merging is done in the block layer.  Don't you see the same thing
> if you use sd devices (/dev/sdX)?

no, if I use sd devices directly, it does not overload ksoftirqd and this is
obvious. The problem with overloading ksoftirqd has roots in request based
stuff in dm layer, in particular in dm_softirq_done() call. 

> Also, what do you see with the latest upstream kernel (say, 3.0-rc7)?

I didn't check it with this kernel. I don't see many changes in request based
stuff in dm.

> If you see your problem only with request-based device-mapper, please
> elaborate about below:
>     - end_clone_bio() has someting like quadratic complexity.
>         * What do you mean the "quadratic complexity"?

end_clone_bio calls blk_update_request which calls __blk_recalc_rq_segments
which has code:
for_each_bio(bio) {
        bio_for_each_segment(bv, bio, i) {

in total, it seems that *whole* bio list is traversed again and again as some parts
are done and some not which leads to comlexity O(n^2) with respect to number
of bio and segments. But this is just wild guess. The real problem could be
elsewhere.

However, oprofile or sysprof show that ksoftirq spends most time in
__blk_recalc_rq_segments().

>     - each request contains more then 100 bios which makes serious
>       troubles for ksoftirqd call backs.
>         * What do you mean the "serious troubles for ksoftirqd call backs"?

serious troubles means that ksoftirqd eats 100% CPU and slows down I/O
significantly (from 2.8GB/s to 500MB/s).

-- 
Lukáš Hejtmánek