Re: *Really* bad I/O latency with md raid5+dm-crypt+lvm

[Christian Pernegger <pernegger@gmail.com>]

>> [Please keep me CCed as I'm not subscribed to LKML]
>>
>> Summary: I was hoping to use a layered storage setup, namely lvm on
>> dm-crypt on md raid5 for a new box I'm setting up, but that isn't
>> looking so good since a single heavyish writer will monopolise any and
>> all I/O on the "device". F. ex. while cp'ing a few GB of data from an
>> external disk to the array it takes ~10sec to run ls and ~2min to
>> start aptitude. Clueless attempts at a diagnosis below.

> Also note that dm-crypt is not "SMP-ready", so whatever hardware you have,
> it will only use once CPU - this may seriously limit the performance,
> depending on your usage and hardware.

The crypto performance itself is fine. Yes, it limits throughput to a
little over 100MiB/s but so what, that's plenty. Multi-core support
will come in time, I can wait. What I can't live with is a single
streaming write singlehandedly starving all reads. Linux has never
been great at this and it has been getting worse since ~2.6.18 but it
was never more than a nuisance (say <1sec delay).

It's as if the I/O scheduler weren't there.

> [latencytop? regular top?]

Actually I hadn't heard of latencytop but it looks nifty. Will have to
compile a custom kernel for it, though, since Debian kernels don't
have CONFIG_LATENCYTOP set.

Regular top has kcryptd (67%), mv (36%), md1_raid5 (36%), pdflush
(7%), kjournald (5%) at the top. Seems a bit much for md, doesn't it?
This is while mv'ing in some data from an sdditional SATA disk, lateny
isn't *too* bad, ~3s for an ls. According to iostat mv is writing to
the array at 50-60 MB/s. The fun part: it's using ~15000tps averaging
out to 4k per transaction as observed via btrace.

That can't be normal, can it?

Thanks,

C.