* RE: PCI bandwidth measurement methods
2004-06-15 20:56 PCI bandwidth measurement methods Arthur Perry
@ 2004-06-15 21:58 ` Fab Tillier
0 siblings, 0 replies; 2+ messages in thread
From: Fab Tillier @ 2004-06-15 21:58 UTC (permalink / raw)
To: 'Arthur Perry', linux-kernel
> From: Arthur Perry [mailto:kernel@linuxfarms.com]
> Sent: Tuesday, June 15, 2004 1:56 PM
>
> Hello,
>
> I have a question about methods of measuring PCI-X bandwidth.
> I was wondering if anybody has any ideas.
> What I am looking for is an "industry standard" method, if one exists.
> At this point, such software does not exist in SPEC.
> Different HBA cards such as Myricom and Mellanox (Myrinet and Infiniband
> hosts) have utilitites to test your PCI bandwidth..
> But what do you do when they come up with completely different numbers?
There are many variables that will determine peak bandwidth for an I/O
adapter. The size of the I/O request as well as the number of I/O requests
outstanding affect bandwidth. The PCI commands used by the adapters to
transfer the memory (burst length, etc) will change the amount of overhead
for the different devices too.
Unless mistaken, Myrinet PCI-X adapters operate at 2.5Gb full duplex per
port, while the Mellanox 4x InfiniBand HCA operates at 10Gb full duplex per
port. From the HW characteristics, you should be able to get 4x the
bandwidth out of the IB HCA (assuming linear scale-up and comparable HW
operation), so that might explain your bandwidth discrepancy between the
different HW.
Lastly, a PCI analyzer can be useful in seeing exactly what commands are
used and how large a burst the different HW adapters are capable of, but
that moves you out of pure SW benchmarking.
>
> What I need is an unbiased, objective PCI-X bandwidth test.
> It's all about performance in Linux actually... So I can't say "completely
> objective"..
You don't need a single test. What you need is to control what each test
does. Same size I/O, same number of outstanding requests, etc. If you have
that, you should be able to compare.
>
> I can write software that will do this, possibly by performing PCI
> writes to scratch memory area on several different HBAs..
PCI writes to HBA attached memory may not use the same PCI commands as I/O
DMA, and thus might lead to totally different results.
> Or maybe adding a counter into the Linux PCI device driver to verify the
> bandwidth claims that Mellanox and Myricom's tools report.
> But before I re-invent the wheel, I was wondering if anybody knew of
> anything that is already out there.
Things like MPI benchmarks should do the trick, since only the underlying HW
provider changes. I'm assuming you can get MPI on top of Myrinet.
> If I were to write something, a whole software validation process would
> have to be performed on top of the actual work.
> If anybody knows of something that already exists and has been in use, it
> would be ideal.
Give MPI benchmarks a try.
Good luck,
- Fab
^ permalink raw reply [flat|nested] 2+ messages in thread