Re: 2.6.28.9: EXT3/NFS inodes corruption

[Daniel J Blueman <daniel.blueman@gmail.com>]

Hi Sylvain,

On Tue, Jul 28, 2009 at 5:45 PM, Sylvain Rochet<gradator@gradator.net> wrote:
> On Tue, Jul 28, 2009 at 09:40:42AM -0700, Daniel J Blueman wrote:
>> On Apr 20, 5:30 pm, Sylvain Rochet <grada...@gradator.net> wrote:
>> > Hi,
>> >
>> > We(TuxFamily) are having some inodes corruptions on a NFS server.
>> >
>> > So, let's start with the facts.
>> >
>> > ==== NFS Server
>> >
>> > Linux bazooka 2.6.28.9 #1 SMP Mon Mar 30 12:58:22 CEST 2009 x86_64 GNU/Linux
>> [snip]
>>
>> Can you do a 'lspci -v' on the server please?
>
> Of course yes.
>
> I attached the 'lspci -v' of the previous and the current storage
> server.

The reason I ask, I was chasing data corruption across the PCIe bus
with some high-performance Quadrics interconnect adapters a while ago.
The reproducer involved multiple outstanding main memory read requests
to related addresses and a small block of data would be returned from
the wrong offset.

In the end, I found the nVidia CK804 (also MCP55) HT->PCIe bridge was
at fault and later found disk corruption when doing heavy rsyncs to
network. This was never publicly acknowledged, but I guess it
illustrates the need for some micro-tests to verify data-soundness
under duress; it took a day (and petabytes of data) of the production
I/O workload to get this data corruption, and 3 seconds with the right
reproducer, (still non-trivial to catch on a PCIe protocol analyser).

Sometime I'll develop a stress-test driver for a common SATA or
network controller to drive it's DMA engine with I/O patterns to and
from main memory, checking the data integrity every few seconds; this
could be generalised with OpenGL nicely for graphics cards on
workstations I imagine.

Daniel
-- 
Daniel J Blueman