Fwd: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

Fwd: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

[dave b]

I am forwarding this to the linux kernel mailing list to see if anyone
is actually interested with this bug or not. I suspect that the patch
to allow DM-CRYPT to use multiple cpus(Scale to multiple CPUs) will
remove most of the problem but *not* all of it. As I previously was
triggering the bug on a single core system.

See https://bugzilla.kernel.org/show_bug.cgi?id=18302 for the original thread.
I filed this as bug  18302 at
https://bugzilla.kernel.org/show_bug.cgi?id=18302 .



---------- Forwarded message ----------
From: dave b <db.pub.mail@gmail.com>
Date: 7 October 2010 04:58
Subject: Re: [dm-crypt] [BUG] bad performance and system stalls when
using dm-crypt
To: Milan Broz <mbroz@redhat.com>
Cc: dm-crypt@saout.de


On 12 September 2010 21:45, dave b <db.pub.mail@gmail.com> wrote:
> On 12 September 2010 19:07, Milan Broz <mbroz@redhat.com> wrote:
>> On 09/12/2010 10:36 AM, dave b wrote:
>>>
>>> Should I forward my 'bug' to the linux kernel maliing list ?
>>
>> Better report it to kernel bugzilla, it is better for tracking.
>> (Also see https://bugzilla.kernel.org/show_bug.cgi?id=17892 )
>
> Will do :)
>

Ok no one has responded to that bug ... :/
On a system lock up I ssh into the box and saw wait time was 83% in top.
I also saw that kcrypd was using 23% of cpu time - and that only one
core was in use.
1012 root      20   0     0    0    0 S   11  0.0  30:15.46 kcryptd

The system was stalled for 20minutes ...

load average: 19.16, 15.74, 11.43

--
Let me take you a button-hole lower.            -- William Shakespeare, "Love's
Labour's Lost"

Re: Fwd: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

[Milan Broz]

On 10/24/2010 05:32 PM, dave b wrote:
> I am forwarding this to the linux kernel mailing list to see if anyone
> is actually interested with this bug or not. I suspect that the patch
> to allow DM-CRYPT to use multiple cpus(Scale to multiple CPUs) will
> remove most of the problem but *not* all of it. As I previously was
> triggering the bug on a single core system.

Hi,
sorry for not updating the bug. We know about this.

Fix was expected to be based on top of the Andi's dm-crypt per-cpu
patch but unfortunately I found serious problems there
(see this thread http://lkml.org/lkml/2010/10/20/215 )

So there are now several known situations when dm-crypt doesn't
perform as expected (another problem just appeared when using CFQ,
(because dm(-crypt) lost the issuing process reference)
see http://lkml.org/lkml/2010/10/24/59).

Milan

Re: Fwd: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

[dave b]

On 25 October 2010 03:31, Milan Broz <mbroz@redhat.com> wrote:
> On 10/24/2010 05:32 PM, dave b wrote:
>> I am forwarding this to the linux kernel mailing list to see if anyone
>> is actually interested with this bug or not. I suspect that the patch
>> to allow DM-CRYPT to use multiple cpus(Scale to multiple CPUs) will
>> remove most of the problem but *not* all of it. As I previously was
>> triggering the bug on a single core system.
>
> Hi,
> sorry for not updating the bug. We know about this.
>
> Fix was expected to be based on top of the Andi's dm-crypt per-cpu
> patch but unfortunately I found serious problems there
> (see this thread http://lkml.org/lkml/2010/10/20/215 )
>
> So there are now several known situations when dm-crypt doesn't
> perform as expected (another problem just appeared when using CFQ,
> (because dm(-crypt) lost the issuing process reference)
> see http://lkml.org/lkml/2010/10/24/59).

Thank for you replying  ^ ^
I was aware with the problems and I am following Andi's dm-crypt
per-cpu patch (I haven't applied or tested it). However, I didn't know
about http://lkml.org/lkml/2010/10/24/59 -->
So this may well be the root of the problem!
Also, won't http://lkml.org/lkml/2010/10/24/59  need to be altered
(again) to work with Andi's patch ?


--
There's small choice in rotten apples.		-- William Shakespeare, "The
Taming of the Shrew"

Re: Fwd: [dm-crypt] [BUG] bad performance and system stalls when using dm-crypt

[Milan Broz]

On 10/24/2010 06:52 PM, dave b wrote:
>> So there are now several known situations when dm-crypt doesn't
>> perform as expected (another problem just appeared when using CFQ,
>> (because dm(-crypt) lost the issuing process reference)
>> see http://lkml.org/lkml/2010/10/24/59).
> 
> Thank for you replying  ^ ^
> I was aware with the problems and I am following Andi's dm-crypt
> per-cpu patch (I haven't applied or tested it). However, I didn't know
> about http://lkml.org/lkml/2010/10/24/59 -->
> So this may well be the root of the problem!
> Also, won't http://lkml.org/lkml/2010/10/24/59  need to be altered
> (again) to work with Andi's patch ?

CFQ related problem should be solved - probably device-mapper core
have to provide some help. (It is not only about dm-crypt,
lot of situations when IO is submitted from different internal process.
The proposed patch is not the proper and complete way how to solve it.)

For the per-cpu patch - I am waiting for replay, apparently some
additional work there is needed.
But I would like to solve it ASAP.

Milan