RAID resync speed

RAID resync speed

[Eyal Lebedinsky]

I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
added a fourth disk it goes at only 20+MB/s. This is on an idle machine.

Individually, each disk measures 60+MB/s with hdparm.

kernel: 2.6.13 on ia32
Controller: Promise SATAII150 TX4
Disks: WD 320GB SATA

Q: Is this the way the raid code works? The way the disk-io is managed? Or
could it be due to the SATA controller?

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au) <http://samba.org/eyal/>
	attach .zip as .dat

Re: RAID resync speed

[Nuno Silva]

Hi,

Eyal Lebedinsky wrote:
> I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
> added a fourth disk it goes at only 20+MB/s. This is on an idle machine.

3*40=120

4*20=80


> Individually, each disk measures 60+MB/s with hdparm.


And concurrent hdparms? Or some dd's concurrently?


> kernel: 2.6.13 on ia32
> Controller: Promise SATAII150 TX4
> Disks: WD 320GB SATA
> 
> Q: Is this the way the raid code works? The way the disk-io is managed? Or
> could it be due to the SATA controller?

You can isolate the performance drop with some dd's. Maybe this card is 
in a pci32/33mhz and you're hitting the pci bus' limits? (120~130MB/sec).

Regards,
Nuno Silva

Re: RAID resync speed

[Eyal Lebedinsky]

Nuno Silva wrote:
>
> Hi,
>
> Eyal Lebedinsky wrote:
>
>> I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
>> added a fourth disk it goes at only 20+MB/s. This is on an idle machine.
>
> 3*40=120
>
> 4*20=80

What does this mean? The raid is syncing at 20MB/s, not each disk, so I do
not see what the multiplication is about.

>> Individually, each disk measures 60+MB/s with hdparm.
>
> And concurrent hdparms? Or some dd's concurrently?

I do not see this as relevant, but four concurrent hdparms (each to a
different disk) give about 30MB/s per disk. I expect the controller
to talk to the four disks at their full speed so concurrency should
not be the issue.

>> kernel: 2.6.13 on ia32
>> Controller: Promise SATAII150 TX4
>> Disks: WD 320GB SATA
>>
>> Q: Is this the way the raid code works? The way the disk-io is
>> managed? Or
>> could it be due to the SATA controller?
>
> You can isolate the performance drop with some dd's. Maybe this card is
> in a pci32/33mhz and you're hitting the pci bus' limits? (120~130MB/sec).

'hdparm -T' gives about 1250 MB/sec so this is not the limiting
factor.

> Regards,
> Nuno Silva

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au) <http://samba.org/eyal/>
	attach .zip as .dat

Re: RAID resync speed

[Nuno Silva]

Eyal Lebedinsky wrote:
> Nuno Silva wrote:
>>Hi,
>>Eyal Lebedinsky wrote:
>>>I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
>>>added a fourth disk it goes at only 20+MB/s. This is on an idle machine.
>>3*40=120
>>4*20=80
> What does this mean? The raid is syncing at 20MB/s, not each disk, so I do
> not see what the multiplication is about.


Yes, you're correct :-)


>>>Individually, each disk measures 60+MB/s with hdparm.
>>And concurrent hdparms? Or some dd's concurrently?
> 
> I do not see this as relevant, but four concurrent hdparms (each to a
> different disk) give about 30MB/s per disk. I expect the controller
> to talk to the four disks at their full speed so concurrency should
> not be the issue.


I guess you're using linux's software raid?
If so, you're hitting the 120MB/sec (and I *think* this time I'm 
correct! :-)

If this is a PCI32/33mhz slot you're not going to be able to get more 
juice. (I bet that 3 concurrent dd's gets you 40MB each).

Anyway, this may be offtopic because the problem (only 20MB/sec for the 
raid with 4 disks) should be something else... Sorry for the noise.

>>>kernel: 2.6.13 on ia32
>>>Controller: Promise SATAII150 TX4
>>>Disks: WD 320GB SATA
>>>
>>>Q: Is this the way the raid code works? The way the disk-io is
>>>managed? Or
>>>could it be due to the SATA controller?
>>
>>You can isolate the performance drop with some dd's. Maybe this card is
>>in a pci32/33mhz and you're hitting the pci bus' limits? (120~130MB/sec).
> 
> 
> 'hdparm -T' gives about 1250 MB/sec so this is not the limiting
> factor.


Mine outputs some fabulous values too... I'm not sure I trust them ;)

# hdparm -T /dev/sda

/dev/sda:
  Timing cached reads:   3536 MB in  2.00 seconds = 1767.38 MB/sec

Regards,
Nuno Silva

Re: RAID resync speed

[Joel Jaeggli]

On Sat, 10 Sep 2005, Nuno Silva wrote:

> Eyal Lebedinsky wrote:
>> Nuno Silva wrote:
>>> Hi,
>>> Eyal Lebedinsky wrote:
>>>> I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
>>>> added a fourth disk it goes at only 20+MB/s. This is on an idle machine.
>>> 3*40=120
>>> 4*20=80
>> What does this mean? The raid is syncing at 20MB/s, not each disk, so I do
>> not see what the multiplication is about.
>
>
> Yes, you're correct :-)

The raid resync speed min and max are governed by the linux raid 
subsystem, I do not remember how to tweak them off hand and it's isn't in 
the software raid howto.

>
>>>> Individually, each disk measures 60+MB/s with hdparm.
>>> And concurrent hdparms? Or some dd's concurrently?
>> 
>> I do not see this as relevant, but four concurrent hdparms (each to a
>> different disk) give about 30MB/s per disk. I expect the controller
>> to talk to the four disks at their full speed so concurrency should
>> not be the issue.
>
>
> I guess you're using linux's software raid?
> If so, you're hitting the 120MB/sec (and I *think* this time I'm correct! :-)
>
> If this is a PCI32/33mhz slot you're not going to be able to get more juice. 
> (I bet that 3 concurrent dd's gets you 40MB each).
>
> Anyway, this may be offtopic because the problem (only 20MB/sec for the raid 
> with 4 disks) should be something else... Sorry for the noise.

Generally the maximum throughput for a pci controller is a bit less than 
the theoretical limit. some if not most motherboard implementations these 
days have the south brdiges sata ports attached to something other than 
the pci bus... This will get even more important in the context of 
sataII. pci-x or pci express sata controllers help quite a bit as 
well for machines that need more ports. Even some of the lower-end promise 
cards will run in a 66mhz slot these days though it's sort of wasteful to 
toss a 32bit card in a pci-x slot.

>>>> kernel: 2.6.13 on ia32
>>>> Controller: Promise SATAII150 TX4
>>>> Disks: WD 320GB SATA
>>>> 
>>>> Q: Is this the way the raid code works? The way the disk-io is
>>>> managed? Or
>>>> could it be due to the SATA controller?
>>> 
>>> You can isolate the performance drop with some dd's. Maybe this card is
>>> in a pci32/33mhz and you're hitting the pci bus' limits? (120~130MB/sec).
>> 
>> 
>> 'hdparm -T' gives about 1250 MB/sec so this is not the limiting
>> factor.
>
>
> Mine outputs some fabulous values too... I'm not sure I trust them ;)
>
> # hdparm -T /dev/sda
>
> /dev/sda:
> Timing cached reads:   3536 MB in  2.00 seconds = 1767.38 MB/sec

Those are measuring the performance of the linux buffer cache without disk 
access. really this a benchmark of the processor and memory of the system 
not the disk. -t meuse speed through the buffer without caching, which 
might give you a ballpark indication of disk speed. Really you need to use 
something like iozone to get a sense of how fast a disk is.

on the laptop I'm sending the mail on...

[root@podocarpus-totara joelja]# hdparm -T /dev/sda

/dev/sda:
  Timing cached reads:   2480 MB in  2.00 seconds = 1240.19 MB/sec

[root@podocarpus-totara joelja]# hdparm -t /dev/sda

/dev/sda:
  Timing buffered disk reads:   90 MB in  3.04 seconds =  29.60 MB/sec


30MB/s is pretty fast for a laptop, but it is a 100GB 5400rpm drive so 
it's probablly a little faster than what I'm used to.

> Regards,
> Nuno Silva
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
--------------------------------------------------------------------------
Joel Jaeggli  	       Unix Consulting 	       joelja@darkwing.uoregon.edu
GPG Key Fingerprint:     5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2

Re: RAID resync speed

[Eyal Lebedinsky]

Nuno Silva wrote:
> Eyal Lebedinsky wrote:
> 
>> Nuno Silva wrote:
>>
>>> Hi,
>>> Eyal Lebedinsky wrote:
>>>
>>>> I noticed that my 3-disk RAID was syncing at about 40MB/s, now that I
>>>> added a fourth disk it goes at only 20+MB/s. This is on an idle
>>>> machine.
>>>
>>> 3*40=120
>>> 4*20=80
>>
>> What does this mean? The raid is syncing at 20MB/s, not each disk, so
>> I do
>> not see what the multiplication is about.
> 
> Yes, you're correct :-)

Actually, I took another look at this matter and I now think that you
had the correct approach.

The rebuild speed is the speed at which the new disk is being built, not
the total rebuild i/o. This means that it does not contain the read
operations. So the PCI limit is a limiting factor. On a 32-bit 33MHz PCI
controller (132MB/s theoretical bandwidth) a 2->3 rebuild cannot be
faster 44MB/s and a 3->4 is limited to 33MB/s.

I think this is true.

The same limit will also apply to any raid i/o as we read/write to all
the disks for any data.

To use 5 60MB/s disks I will need 300MB/s bandwidth which a 64-bit 66MHz
PCI can deliver. A 32-bit/66MHz will come close - what can PCIe do?.
A proper RAID card will alleviate the PCI limitation as it will have
dedicated channels for each disk (well, a good controller should) with
full bandwidth and the PCI will only need to go at the one-disk speed
(for raid-5).

On-board SATA controllers will have better bandwidth if they sit on a
better than PCI bus (or on more than one PCI bus).

-- 
Eyal Lebedinsky (eyal@eyal.emu.id.au) <http://samba.org/eyal/>
	attach .zip as .dat

RE: RAID resync speed

[Roger Heflin]

 

> 
> Actually, I took another look at this matter and I now think 
> that you had the correct approach.
> 
> The rebuild speed is the speed at which the new disk is being 
> built, not the total rebuild i/o. This means that it does not 
> contain the read operations. So the PCI limit is a limiting 
> factor. On a 32-bit 33MHz PCI controller (132MB/s theoretical 
> bandwidth) a 2->3 rebuild cannot be faster 44MB/s and a 3->4 
> is limited to 33MB/s.
> 
> I think this is true.
> 
> The same limit will also apply to any raid i/o as we 
> read/write to all the disks for any data.
> 
> To use 5 60MB/s disks I will need 300MB/s bandwidth which a 
> 64-bit 66MHz PCI can deliver. A 32-bit/66MHz will come close 
> - what can PCIe do?.
> A proper RAID card will alleviate the PCI limitation as it 
> will have dedicated channels for each disk (well, a good 
> controller should) with full bandwidth and the PCI will only 
> need to go at the one-disk speed (for raid-5).

You will need to carefully check the real raid controllers 
as some have separate channels some do not, it is all very 
much a mess, and you need to be very careful in checking
them and very careful in testing them if you need
real speed.

> 
> On-board SATA controllers will have better bandwidth if they 
> sit on a better than PCI bus (or on more than one PCI bus).

The all of on-board SATA controllers I have used have lots 
of shared hardware and are very very bad if you use more
than 1 disk per set of shared hardware, and  build in does
not mean that they will have a better PCI connection than
a card, it all depends on the sata chipset that they used,
there is stuff build into motherboards with pcix slots that
have the ide, sata, and network subsystems connected to 
33mhz buses.

Most of the stuff I have used is high end dual and quad
cpu motherboards, and the build in stuff does have lots
of corners cut, I would expect the desktop class stuff
to be as bad or worse.

                    Roger