mke2fs -j goes nuts on 3Ware 8506-4LP

mke2fs -j goes nuts on 3Ware 8506-4LP

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I have a 3Ware 8506-4LP controller with 4 250GB Maxtor SATA drives, in a 
raid-5 configuration (64K blocks)
System is:
Dual Opteron 246 (2GHz)
2GB RAM
Tyan S2875 motherboard

Kernel: 2.6.8-rc2 (pre-empt is ON)
Rest of OS: Mandrake 10.0 AMD64 edition.

When I execute a mke2fs -j /dev/sda7  to format a 600GB partition on the raid 
as ext3, the system slows to a crawl.

Basically, the inode creation goes fine for the first 1000 nodes or so, then 
starts slowing to about 1 node per second.

According to 'top' the systems 'buffers' value is almost all my ram and both 
cpus are at almost 100% in I/O wait.

The system load will start to climb and it eventually hit ~15 before I got 
bored and rebotoed.

X is totally unresponsive although 'top' does refresh occasionally in a 
konsole.


I do understand that the 3Ware 8506 cards arent exactly stellar at raid-5 (we 
see ~22MB/sec write and upto 80MB/sec read if we're lucky) but surely a 
simple format of a filesystem shouldn't tie up the system like this?

Would a different card (3Ware 9500 series or LSI Logic MegaRAID) help here?

Cheers,

Mark.

P.S. As I write this, I'm doing an sa-learn (spamassassin) over 14k messages 
and the system is a little sluggish even though gkrellm is only showing 
~1MB/sec on the 3Ware, so I'm guessing these are just pants cards in general.
Periodically runnung 'sync' helps for a while but it fades again after that.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBB3bgBn4EFUVUIO0RAsi1AKDdSQCoGU3XZgQfo7h6yVoYgJxfewCgjdW/
lQyWsb7xGk1ZCntcDAp8C20=
=hJon
-----END PGP SIGNATURE-----

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Andrew Morton]

Mark Watts <m.watts@eris.qinetiq.com> wrote:
>
> I have a 3Ware 8506-4LP controller with 4 250GB Maxtor SATA drives, in a 
>  raid-5 configuration (64K blocks)
>  System is:
>  Dual Opteron 246 (2GHz)
>  2GB RAM
>  Tyan S2875 motherboard
> 
>  Kernel: 2.6.8-rc2 (pre-empt is ON)
>  Rest of OS: Mandrake 10.0 AMD64 edition.
> 
>  When I execute a mke2fs -j /dev/sda7  to format a 600GB partition on the raid 
>  as ext3, the system slows to a crawl.

It's conceivably a memory reclaim problem.  Please try booting with the
boot command line option `mem=768M', then see if it goes any better.

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Jason L Tibbitts III]

>>>>> "MW" == Mark Watts <m.watts@eris.qinetiq.com> writes:

MW> I have a 3Ware 8506-4LP controller with 4 250GB Maxtor SATA
MW> drives, in a raid-5 configuration (64K blocks) System is: Dual
MW> Opteron 246 (2GHz) 2GB RAM Tyan S2875 motherboard

I have a distantly similar configuration: 3ware 8506-4LP controller, 4
250GB Western Digital SATA drives, RAID5 (with one hot spare, so the
array is only 500GB), running dual 2.8GHz Xeon CPUs on a Supermicro
X5DPEG2 motherboard with 4GB of RAM.  The OS is Fedora Core 2 fully
updated; kernel is 2.6.6-1.435.2.3smp (which is basically 2.6.7rc3
plus security fixes).

IO is just glacial.  Things go well for a time but then everything
slows to a crawl.  The machine has very little load: a mostly inactive
NFS export, mysql running a few queries per second on a very small
database, that kind of thing.  Certainly the entire working set fits
easily into available RAM.

Yet on occasion it can take upwards of a minute just to ssh in.  This
machine took over half of the load from a poor 2GHz Athlon machine
with 1GB of RAM running Red Hat 7.2 and it feels far slower.

It took three seconds to run this:

> cat /proc/meminfo
MemTotal:      4155116 kB
MemFree:        211256 kB
Buffers:       3643760 kB
Cached:         129880 kB
SwapCached:          0 kB
Active:         284676 kB
Inactive:      3527304 kB
HighTotal:      163264 kB
HighFree:         3648 kB
LowTotal:      3991852 kB
LowFree:        207608 kB
SwapTotal:     8385920 kB
SwapFree:      8385920 kB
Dirty:             272 kB
Writeback:           0 kB
Mapped:          53464 kB
Slab:           111260 kB
Committed_AS:   263768 kB
PageTables:       4728 kB
VmallocTotal:   106488 kB
VmallocUsed:      6152 kB
VmallocChunk:    99996 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

I actually have several of these machines and can spare one for
testing.  If anyone has any suggestions, I'm happy to try them out.

MW> P.S. As I write this, I'm doing an sa-learn (spamassassin) over
MW> 14k messages and the system is a little sluggish even though
MW> gkrellm is only showing ~1MB/sec on the 3Ware, so I'm guessing
MW> these are just pants cards in general.

I have used 3ware cards extensively for many years now and have never
known them to be slow like this.  This configuration should easily be
able to saturate the drives (or the bus, whichever comes first).

 - J<

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> >>>>> "MW" == Mark Watts <m.watts@eris.qinetiq.com> writes:
>
> MW> I have a 3Ware 8506-4LP controller with 4 250GB Maxtor SATA
> MW> drives, in a raid-5 configuration (64K blocks) System is: Dual
> MW> Opteron 246 (2GHz) 2GB RAM Tyan S2875 motherboard
>
> I have a distantly similar configuration: 3ware 8506-4LP controller, 4
> 250GB Western Digital SATA drives, RAID5 (with one hot spare, so the
> array is only 500GB), running dual 2.8GHz Xeon CPUs on a Supermicro
> X5DPEG2 motherboard with 4GB of RAM.  The OS is Fedora Core 2 fully
> updated; kernel is 2.6.6-1.435.2.3smp (which is basically 2.6.7rc3
> plus security fixes).
>
> IO is just glacial.  Things go well for a time but then everything
> slows to a crawl.  The machine has very little load: a mostly inactive
> NFS export, mysql running a few queries per second on a very small
> database, that kind of thing.  Certainly the entire working set fits
> easily into available RAM.
>
> Yet on occasion it can take upwards of a minute just to ssh in.  This
> machine took over half of the load from a poor 2GHz Athlon machine
> with 1GB of RAM running Red Hat 7.2 and it feels far slower.
>
> It took three seconds to run this:
> > cat /proc/meminfo
>
<snip>
> I actually have several of these machines and can spare one for
> testing.  If anyone has any suggestions, I'm happy to try them out.
>
> MW> P.S. As I write this, I'm doing an sa-learn (spamassassin) over
> MW> 14k messages and the system is a little sluggish even though
> MW> gkrellm is only showing ~1MB/sec on the 3Ware, so I'm guessing
> MW> these are just pants cards in general.
>
> I have used 3ware cards extensively for many years now and have never
> known them to be slow like this.  This configuration should easily be
> able to saturate the drives (or the bus, whichever comes first).
>
>  - J<


I've since disabled CONFIG_PREEMPT in an attempt to see if ti fixes a full 
system freeze I've been seeing ove the last few days (3 freezes in 24 hrs but 
nothing on a serial console).
It seems to be behaving from that regard.

Copying a couple of ISO images (~650MB each) from a USB2 drive (LaCie 1TB 
disk) to the Raid has practically tied up the system again.
X is so unresponsive that I've managed to type the last paragraph before it 
hit the screen...

Once the copy has finished and caches have been synced to disk, the system 
becomes more lively again, but my P3/1.2Ghz/256MB laptop with an old 20GB 
disk is able to remain fully responsive when I copy the same set of isos over 
the network.

I have an LSI Logic Megaraid SATA raid card on order from my supplier so I'll 
be sticking it into another identical system to run some more tests, although 
I suspect it will confirm that its the 3Ware rather than the system thats the 
bottleneck.

Out of interest Jason, do you have the 3Ware plugged into a 64bit or 32bit pci 
slot?

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBCRKSBn4EFUVUIO0RApm2AJ0cSwI7nqUeVqpKzcGS76cMbPwjtACfcsB3
WkyNVT6hQq0mnLMoqoKgoa0=
=lWPW
-----END PGP SIGNATURE-----

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Jason L Tibbitts III]

>>>>> "MW" == Mark Watts <m.watts@eris.qinetiq.com> writes:

MW> Out of interest Jason, do you have the 3Ware plugged into a 64bit
MW> or 32bit pci slot?

This board has only 64-bit slots; the card is actually in a 133MHz
PCI-X slot, but of course it only runs at 66MHz.

 - J<

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> >>>>> "MW" == Mark Watts <m.watts@eris.qinetiq.com> writes:
>
> MW> Out of interest Jason, do you have the 3Ware plugged into a 64bit
> MW> or 32bit pci slot?
>
> This board has only 64-bit slots; the card is actually in a 133MHz
> PCI-X slot, but of course it only runs at 66MHz.
>
>  - J<

Ok - it seems it doesn't matter what kind of slot its in then - mine is in a 
32bit/33MHz slot.

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBCRWqBn4EFUVUIO0RAlO+AJ4iHMkqQclb5uLziLRhnd2l+W+N6wCeI2kn
1p7ZB3z/g/8xXUprOzu575o=
=cwho
-----END PGP SIGNATURE-----

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Jason L Tibbitts III]

>>>>> "MW" == Mark Watts <m.watts@eris.qinetiq.com> writes:

MW> Ok - it seems it doesn't matter what kind of slot its in then -
MW> mine is in a 32bit/33MHz slot.

It's not hard to saturate a bus at that speed, but you don't seem to
be getting to that point.

My experience with 3ware cards is that they will happily saturate any
bus you plug them into as long as the drives are fast enough.  On a
machine with a 3w7506-8 card on each of two buses running software
RAID striping across them, both cards will saturate their respective
buses.  (I have a few machines like this.)  I really doubt that the
cards are the root of the issue here.

 - J<

Re: mke2fs -j goes nuts on 3Ware 8506-4LP

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Mark Watts <m.watts@eris.qinetiq.com> wrote:
> > I have a 3Ware 8506-4LP controller with 4 250GB Maxtor SATA drives, in a
> >  raid-5 configuration (64K blocks)
> >  System is:
> >  Dual Opteron 246 (2GHz)
> >  2GB RAM
> >  Tyan S2875 motherboard
> >
> >  Kernel: 2.6.8-rc2 (pre-empt is ON)
> >  Rest of OS: Mandrake 10.0 AMD64 edition.
> >
> >  When I execute a mke2fs -j /dev/sda7  to format a 600GB partition on the
> > raid as ext3, the system slows to a crawl.
>
> It's conceivably a memory reclaim problem.  Please try booting with the
> boot command line option `mem=768M', then see if it goes any better.

No change at all, apart from the slowdown happening quicker.

I reverted to 2.6.8rc1 without pre-empt and that didnt help either.

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBCgUbBn4EFUVUIO0RAkHSAJ0YogN0UuxNY7680W8foDM993mJRQCeMVAY
kQeNvt6WHSiFWJ6J0f97g5g=
=0UVo
-----END PGP SIGNATURE-----