3Ware Hard Bus Hang 2.4.18 > 220 MB/S

3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Jeff V. Merkey]

Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
(32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
seeing about 216-224 MB/S total throughput on writes to local 
arrays on 2.4.18.  

The system is also running an Intel Gigabit Ethernet Card at 
116-122 MB/S with full network traffic and writing this traffic to 
the 3Ware arrays.  All this hardware is running on a Serverworks 
HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
processors.

After running for about 3 hours, the system will hard hang and die.  
Using debugging tools, I have isolated to the hang to the 3Ware 
adapters.  If I remove all but a single 3Ware adapter, the system will
run reliably for days at these data rates.  The moment I add more 
than one 3Ware 7810 adapter, the system will lock up.  Recent testing
reveals that the hang is in the 3Ware card itself (all the LEDs go 
on at once and stay on).  Attempts by the system to reset the adapter
fail until the system is power cycled.  

3Ware dfriver version is .16 from the 2.4.18 tree.  Firmware is the 48 bit
LBA version.  

Please advise.  

Jeff

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Jeff V. Merkey]

On Wed, Feb 27, 2002 at 10:25:45AM -0700, Jeff V. Merkey wrote:
> 


More info.  I put in some trace code to determine how many io buffer
heads were being fed to each adapter.  When the number of bh's reaches
numbers above 4244+- buffer heads outstanding at one time, then I see the 
cards lockup.  

Jeff

> 
> 
> Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> seeing about 216-224 MB/S total throughput on writes to local 
> arrays on 2.4.18.  
> 
> The system is also running an Intel Gigabit Ethernet Card at 
> 116-122 MB/S with full network traffic and writing this traffic to 
> the 3Ware arrays.  All this hardware is running on a Serverworks 
> HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> processors.
> 
> After running for about 3 hours, the system will hard hang and die.  
> Using debugging tools, I have isolated to the hang to the 3Ware 
> adapters.  If I remove all but a single 3Ware adapter, the system will
> run reliably for days at these data rates.  The moment I add more 
> than one 3Ware 7810 adapter, the system will lock up.  Recent testing
> reveals that the hang is in the 3Ware card itself (all the LEDs go 
> on at once and stay on).  Attempts by the system to reset the adapter
> fail until the system is power cycled.  
> 
> 3Ware dfriver version is .16 from the 2.4.18 tree.  Firmware is the 48 bit
> LBA version.  
> 
> Please advise.  
> 
> Jeff

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Andreas Dilger]

On Feb 27, 2002  10:25 -0700, Jeff V. Merkey wrote:
> Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> seeing about 216-224 MB/S total throughput on writes to local 
> arrays on 2.4.18.  

Have you done any kind of variations on this configuration to see when
or where the maximum throughput happens?  Daniel and I were speculating
about where the 3ware limits are.  Specs say 100MB/s per adapter (for
both 6000 and 7000 series), you would probably hit max bandwidth with
2 adapters.  The drives themselves are not a limiting factor, unless
you are down to striping across only 2 drives instead of all 8.  I take
it you are using the hardware RAID instead of software MD RAID?

> The system is also running an Intel Gigabit Ethernet Card at 
> 116-122 MB/S with full network traffic and writing this traffic to 
> the 3Ware arrays.  All this hardware is running on a Serverworks 
> HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> processors.

Does this board have multiple PCI busses?  Is the GigE card on a
different bus than the 3ware cards?

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Jeff V. Merkey]

On Wed, Feb 27, 2002 at 10:48:25AM -0700, Andreas Dilger wrote:
> On Feb 27, 2002  10:25 -0700, Jeff V. Merkey wrote:
> > Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> > for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> > (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> > seeing about 216-224 MB/S total throughput on writes to local 
> > arrays on 2.4.18.  
> 
> Have you done any kind of variations on this configuration to see when
> or where the maximum throughput happens?  Daniel and I were speculating
> about where the 3ware limits are.  Specs say 100MB/s per adapter (for
> both 6000 and 7000 series), you would probably hit max bandwidth with
> 2 adapters.  The drives themselves are not a limiting factor, unless
> you are down to striping across only 2 drives instead of all 8.  I take
> it you are using the hardware RAID instead of software MD RAID?

Single adapter can push 130 MB/S with 8 drives striped at RAID 0 
(hardware RAID not software).  With 2 adapters with 8 drives, the 
max rate of 216 MB/S is achieved.  Adding additional adapters 
on the same PCI bus hits the wall at 238+- MB/S.  I have not tried 
putting the two other 3Ware's on the 66 Mhz bus, but in theory, this
would result in 350 MB/S total throughput from what I am seeing
with the Gigabit ethernet adapter.

> 
> > The system is also running an Intel Gigabit Ethernet Card at 
> > 116-122 MB/S with full network traffic and writing this traffic to 
> > the 3Ware arrays.  All this hardware is running on a Serverworks 
> > HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> > processors.
> 
> Does this board have multiple PCI busses?  Is the GigE card on a
> different bus than the 3ware cards?

The Gigabit adapter is running on the 66 Mhz bus.  When combined with 
SCI I am able to push 238 MB/S through the SCI to the 3Ware adapters
and out to disk.  At present, I have modified the SCI drivers and I am 
DMA'ing directly into Linus buffer cache from the SCI disk data.  

Very very fast.  The 3Ware adapters are barfing when the BH count 
gets too high.  I have no idea why.

Jeff


> 
> Cheers, Andreas
> --
> Andreas Dilger
> http://sourceforge.net/projects/ext2resize/
> http://www-mddsp.enel.ucalgary.ca/People/adilger/

3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Harald van Pee]

Hi,

because I am planning to buy a very similar system, I would like to know:
- is this problem solved in the meantime?
- does it also occure with older kernels (2.4.16, 2.2.16) or older firmware 
(and smaler disks)?
- is it only a problem of high transfer rates and does not occure on lower ~ 
150 MB/sec or 30 MB/sec?

Thanks for your help
Harald

On Wed, Feb 27, 2002 at 10:25:45AM -0700, Jeff V. Merkey wrote:
>
More info.  I put in some trace code to determine how many io buffer
heads were being fed to each adapter.  When the number of bh's reaches
numbers above 4244+- buffer heads outstanding at one time, then I see the
cards lockup.
Jeff
>
>
> Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> seeing about 216-224 MB/S total throughput on writes to local
> arrays on 2.4.18. 
>
> The system is also running an Intel Gigabit Ethernet Card at
> 116-122 MB/S with full network traffic and writing this traffic to
> the 3Ware arrays.  All this hardware is running on a Serverworks
> HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> processors.
>
> After running for about 3 hours, the system will hard hang and die. 
> Using debugging tools, I have isolated to the hang to the 3Ware
> adapters.  If I remove all but a single 3Ware adapter, the system will
> run reliably for days at these data rates.  The moment I add more
> than one 3Ware 7810 adapter, the system will lock up.  Recent testing
> reveals that the hang is in the 3Ware card itself (all the LEDs go
> on at once and stay on).  Attempts by the system to reset the adapter
> fail until the system is power cycled. 
>
> 3Ware dfriver version is .16 from the 2.4.18 tree.  Firmware is the 48 bit
> LBA version. 
>
> Please advise. 
>
> Jeff

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Jeff V. Merkey]

Problem seems specific to the SuperMicro motherboard + 3Ware.

Jeff

On Mon, Mar 04, 2002 at 06:06:20PM +0100, Harald van Pee wrote:
> Hi,
> 
> because I am planning to buy a very similar system, I would like to know:
> - is this problem solved in the meantime?
> - does it also occure with older kernels (2.4.16, 2.2.16) or older firmware 
> (and smaler disks)?
> - is it only a problem of high transfer rates and does not occure on lower ~ 
> 150 MB/sec or 30 MB/sec?
> 
> Thanks for your help
> Harald
> 
> On Wed, Feb 27, 2002 at 10:25:45AM -0700, Jeff V. Merkey wrote:
> >
> More info.  I put in some trace code to determine how many io buffer
> heads were being fed to each adapter.  When the number of bh's reaches
> numbers above 4244+- buffer heads outstanding at one time, then I see the
> cards lockup.
> Jeff
> >
> >
> > Running 4 3Ware 7810 Adapters with the updated 48 bit LBA firmware
> > for the 78110, and attached to 8 Maxtor 160 GB hard disks on each card
> > (32 drives total) striping Raid 0m across 5.6 terabytes of disk, I am
> > seeing about 216-224 MB/S total throughput on writes to local
> > arrays on 2.4.18. 
> >
> > The system is also running an Intel Gigabit Ethernet Card at
> > 116-122 MB/S with full network traffic and writing this traffic to
> > the 3Ware arrays.  All this hardware is running on a Serverworks
> > HE chipset with a SuperMicro motherboard and dual 933 Mhz PIII
> > processors.
> >
> > After running for about 3 hours, the system will hard hang and die. 
> > Using debugging tools, I have isolated to the hang to the 3Ware
> > adapters.  If I remove all but a single 3Ware adapter, the system will
> > run reliably for days at these data rates.  The moment I add more
> > than one 3Ware 7810 adapter, the system will lock up.  Recent testing
> > reveals that the hang is in the 3Ware card itself (all the LEDs go
> > on at once and stay on).  Attempts by the system to reset the adapter
> > fail until the system is power cycled. 
> >
> > 3Ware dfriver version is .16 from the 2.4.18 tree.  Firmware is the 48 bit
> > LBA version. 
> >
> > Please advise. 
> >
> > Jeff
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Harald van Pee]

On Monday 04 March 2002 18:38, Jeff V. Merkey wrote:
> Problem seems specific to the SuperMicro motherboard + 3Ware.
>
Because this is the same configuration which I will buy
(the very similar means I need only three but 4 3ware cards, and I have the 
option to buy a netgear card)
this doesn't help me.

Such systems are used at DESY in Hamburg 
with no problems 
(www.desy.de/unix/linux/delfi/delfi.html)
and therefore I want to know if the rate, firmware or the kernel or driver 
version makes the difference.

Regards
Harald

Re: 3Ware Hard Bus Hang 2.4.18 > 220 MB/S

[Jeff V. Merkey]

#Ware is fixing this problem, so I expect a fix soon.  You won't
see the problem unless you are using multiple cards, and running 
them at very high data rates.

Jeff


On Mon, Mar 04, 2002 at 06:36:33PM +0100, Harald van Pee wrote:
> On Monday 04 March 2002 18:38, Jeff V. Merkey wrote:
> > Problem seems specific to the SuperMicro motherboard + 3Ware.
> >
> Because this is the same configuration which I will buy
> (the very similar means I need only three but 4 3ware cards, and I have the 
> option to buy a netgear card)
> this doesn't help me.
> 
> Such systems are used at DESY in Hamburg 
> with no problems 
> (www.desy.de/unix/linux/delfi/delfi.html)
> and therefore I want to know if the rate, firmware or the kernel or driver 
> version makes the difference.
> 
> Regards
> Harald