dm-crypt and huge performance penalty

dm-crypt and huge performance penalty

[Dmitry Nezhevenko]

Hi,

I've just purchased three HDD drives and tried to setup mdadm RAID5 +
dm-crypt.

What I've found is that dm-crypt reduces storage I/O performance very much
while CPU is not bottleneck.

Firstly I've tried to setup raid5 and then dm-crypt on top of it. After
discovering issue I've tried just raid5 and confirmed that mdadm itself
works as expected. 

And the last idea is to setup three dm-crypt partitions and then raid5 on
top of independently encrypted drives.

So here are my results:

           |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |


Also here are some results using both dd and bonnie as benchmark:
http://pastebin.com/hwwkRRTd

I'm using 3 x WD1002FAEX-00Z3A0 HDD. SATA controller is just embedded to
m/b:

00:1f.2 IDE interface: Intel Corporation 5 Series/3400 Series Chipset 4
port SATA IDE Controller (rev 06)
00:1f.5 IDE interface: Intel Corporation 5 Series/3400 Series Chipset 2
port SATA IDE Controller (rev 06)
 
dd tests were done using simple dd:
if=/dev/resulting-device of=/dev/zero bs=1m count=512

For Bonnie I've used 50GB reiserfs3 partition on top of resulting device.

Are these numbers expected? They looks really strange for me.

-- 
WBR, Dmitry

Re: dm-crypt and huge performance penalty

[Ryan Castellucci]

On Mon, Sep 20, 2010 at 3:37 PM, Dmitry Nezhevenko <dion@inhex.net> wrote:
> Firstly I've tried to setup raid5 and then dm-crypt on top of it. After
> discovering issue I've tried just raid5 and confirmed that mdadm itself
> works as expected.
>
> And the last idea is to setup three dm-crypt partitions and then raid5 on
> top of independently encrypted drives.
>
> So here are my results:
>
>           |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
> Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
> Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |

dm-crypt will only use one core per device, so this is expected
behavior (and I believe an issue that is being worked on). From these
numbers it looks like you have a dual core or dual cpu system?

-- 
Ryan Castellucci http://ryanc.org/

Re: dm-crypt and huge performance penalty

[Dmitry Nezhevenko]

On Wed, Sep 22, 2010 at 04:52:17PM -0700, Ryan Castellucci wrote:
> On Mon, Sep 20, 2010 at 3:37 PM, Dmitry Nezhevenko <dion@inhex.net> wrote:
> > Firstly I've tried to setup raid5 and then dm-crypt on top of it. After
> > discovering issue I've tried just raid5 and confirmed that mdadm itself
> > works as expected.
> >
> > And the last idea is to setup three dm-crypt partitions and then raid5 on
> > top of independently encrypted drives.
> >
> > So here are my results:
> >
> >           |  mdadm   | mdadm+dm-crypt | 3 x dm-crypt + mdadm |
> > Seq. read  | 168 Mb/s | 57 Mb/s        | 119 Mb/s             |
> > Seq. write | 80 Mb/s  | 36 Mb/s        | 64.4 Mb/s            |
> 
> dm-crypt will only use one core per device, so this is expected
> behavior (and I believe an issue that is being worked on). From these
> numbers it looks like you have a dual core or dual cpu system?
> 

Hi, Yes, The machine is home NAS with inexpensive Intel Core i3 CPU that
has two real cores and HT.
 
-- 
WBR, Dmitry