fsx for Linux showing up reiserfs problem?

fsx for Linux showing up reiserfs problem?

[Dave Jones]

Hi folks,
 After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
I took a look at fsx and played with it a little under Linux..

The changes to make it work are trivial, and are at
http://www.codemonkey.org.uk/cruft/fsx-linux.c
(non-existant include & expected mmap() behaviour differences)

I've done a few tests on local filesystems, and so far Ext2 & Ext3
seem to be holding up..

Reiserfs however dies very early into the test..

  truncating to largest ever: 0x3f15f
  READ BAD DATA: offset = 0x1d3d4, size = 0x962f
  OFFSET  GOOD    BAD     RANGE
  0x1d3d4 0x177d  0x0000  0x  563
  operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops

Options used were ./fsx -c1234 /mnt/test/testfile
(Although it seems to crash with any -c option)

Looks like an interesting tool, and probably something that should
be added to testsuites like Cerberus.

regards,
Dave.

-- 
| Dave Jones.                    http://www.codemonkey.org.uk
| SuSE Labs .

Re: fsx for Linux showing up reiserfs problem?

[Martin Josefsson]

On Sat, 15 Dec 2001, Dave Jones wrote:


> I've done a few tests on local filesystems, and so far Ext2 & Ext3
> seem to be holding up..

I've tested ext3 on 2.4.17-pre8 and it works fine.

> Reiserfs however dies very early into the test..
> 
>   truncating to largest ever: 0x3f15f
>   READ BAD DATA: offset = 0x1d3d4, size = 0x962f
>   OFFSET  GOOD    BAD     RANGE
>   0x1d3d4 0x177d  0x0000  0x  563
>   operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops

I tested with reiserfs on 2.4.9-ac14 and 2.4.15-pre4-xfs and it fails in
the same way on both.

I tested on a xfs partition on the 2.4.15-pre4-xfs kernel and didn't get
any errors either.

/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

More fun with fsx.

[Dave Jones]

On Sat, 15 Dec 2001, Dave Jones wrote:

> The changes to make it work are trivial, and are at
> http://www.codemonkey.org.uk/cruft/fsx-linux.c
> (non-existant include & expected mmap() behaviour differences)

Whilst waiting for the local fs tests to finish/progress I did
a quick test with nfs. It dies after a while with an io error.

You can skip over the boring part and go straight to the interesting
failure case by using the options -c1234 -b 48870
Should happen quite quickly after that.

Full log of the failcase is at
http://www.codemonkey.org.uk/cruft/nfs-fsx.txt


regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: [reiserfs-dev] fsx for Linux showing up reiserfs problem?

[Hans Reiser]

Dave Jones wrote:

>Hi folks,
> After reading the article at http://www.kerneltrap.com/article.php?sid=415&mode=thread&order=0
>on the FreeBSD guys finding a bunch of NFS bugs with a stress tool,
>I took a look at fsx and played with it a little under Linux..
>
>The changes to make it work are trivial, and are at
>http://www.codemonkey.org.uk/cruft/fsx-linux.c
>(non-existant include & expected mmap() behaviour differences)
>
>I've done a few tests on local filesystems, and so far Ext2 & Ext3
>seem to be holding up..
>
>Reiserfs however dies very early into the test..
>
>  truncating to largest ever: 0x3f15f
>  READ BAD DATA: offset = 0x1d3d4, size = 0x962f
>  OFFSET  GOOD    BAD     RANGE
>  0x1d3d4 0x177d  0x0000  0x  563
>  operation# (mod 256) for the bad data unknown, check HOLE and EXTEND ops
>
>Options used were ./fsx -c1234 /mnt/test/testfile
>(Although it seems to crash with any -c option)
>
>Looks like an interesting tool, and probably something that should
>be added to testsuites like Cerberus.
>
>regards,
>Dave.
>
Thanks Dave, Elena and Nikita and Green, take a look at this.

Hans

More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Sat, 15 Dec 2001, Dave Jones wrote:
    >> The changes to make it work are trivial, and are at
    >> http://www.codemonkey.org.uk/cruft/fsx-linux.c (non-existant
    >> include & expected mmap() behaviour differences)

     > Whilst waiting for the local fs tests to finish/progress I did
     > a quick test with nfs. It dies after a while with an io error.

     > You can skip over the boring part and go straight to the
     > interesting failure case by using the options -c1234 -b 48870
     > Should happen quite quickly after that.

     > Full log of the failcase is at
     > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt

Which kernel and mount options?

Cheers,
   Trond

Re: More fun with fsx.

[Dave Jones]

On Sun, 16 Dec 2001, Trond Myklebust wrote:

>      > Full log of the failcase is at
>      > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
> Which kernel and mount options?

client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special.
I'll try to reproduce with a newer kernel on the client.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Dave Jones]

On Sun, 16 Dec 2001, Dave Jones wrote:

> > > Full log of the failcase is at
> > > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
> > Which kernel and mount options?
> client 2.4.13-ac5, server 2.4.17rc1, default options, nothing special.
> I'll try to reproduce with a newer kernel on the client.

Yup, it's still there with 2.4.17-rc1 as the client too.
Exact same failure.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > Yup, it's still there with 2.4.17-rc1 as the client too.  Exact
     > same failure.

I get a size error when I don't apply the patch that fixes the
attribute update race. Can you try with that patch applied? It
can be found on

  http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

Cheers,
  Trond

Re: More fun with fsx.

[Dave Jones]

On Sun, 16 Dec 2001, Trond Myklebust wrote:

> I get a size error when I don't apply the patch that fixes the
> attribute update race. Can you try with that patch applied? It
> can be found on
>   http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

No change with this applied on client side. Should I bother
recompiling server side with this ?

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Sun, 16 Dec 2001, Trond Myklebust wrote:
    >> I get a size error when I don't apply the patch that fixes the
    >> attribute update race. Can you try with that patch applied? It
    >> can be found on
    >> http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

     > No change with this applied on client side. Should I bother
     > recompiling server side with this ?

Nope. It is client-only...

Cheers,
  Trond

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Sun, 16 Dec 2001, Dave Jones wrote:
    >> > > Full log of the failcase is at
    >> > > http://www.codemonkey.org.uk/cruft/nfs-fsx.txt
    >> > Which kernel and mount options?
    >> client 2.4.13-ac5, server 2.4.17rc1, default options, nothing
    >> special.  I'll try to reproduce with a newer kernel on the
    >> client.

     > Yup, it's still there with 2.4.17-rc1 as the client too.  Exact
     > same failure.

I found the bug. It's a pretty ugly race...

The problem is that although the inode->i_sem semaphore protects you
against races with ordinary writes, it does *not* protect you against
the memory management routines (such as page_launder()) from calling
writepage() on dirty pages.
For most local filesystems, this is not a problem, since they can call
vmtruncate() *before* they do their thing on the disk itself.  For
non-local filesystems, you usually can't do this, since you cannot
guarantee that the RPC call will succeed on the server.

The only solution I can see, is to force a synchronous write of those
dirty pages to the server prior to calling setattr. I've updated the
patch on

   http://www.fys.uio.no/~trondmy/src/2.4.17/linux-2.4.17-fattr.dif

to do this. Fixes the race for me...

Cheers,
   Trond

Re: More fun with fsx.

[Dave Jones]

On Sun, 16 Dec 2001, Trond Myklebust wrote:

> I found the bug. It's a pretty ugly race...

Well, you found _a_ bug perhaps, but not this one..
Still repeatedly fails in exactly the same part with
your second patch applied instead.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Sun, 16 Dec 2001, Trond Myklebust wrote:
    >> I found the bug. It's a pretty ugly race...

     > Well, you found _a_ bug perhaps, but not this one..  Still
     > repeatedly fails in exactly the same part with your second
     > patch applied instead.

In that case, I'll need a tcpdump of the point at which the error
occurs.

I'm unable to produce any problem whatsoever with the new patch
applied. Certainly not an EIO: that can normally only occur if you are
using soft mounts (which I assume is not the case?) or if the server
is screwed up...

Cheers,
  Trond

Re: More fun with fsx.

[Dave Jones]

On Sun, 16 Dec 2001, Trond Myklebust wrote:

> In that case, I'll need a tcpdump of the point at which the error
> occurs.

Sure no problem... any particular preferred options ?
Want client and server, or just client ?

> I'm unable to produce any problem whatsoever with the new patch
> applied. Certainly not an EIO: that can normally only occur if you are
> using soft mounts (which I assume is not the case?) or if the server
> is screwed up...

This is 'defaults' so, I would guess that is the case looking at the
man page. but, I just tested and its repeatable with hard and soft.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Sun, 16 Dec 2001, Trond Myklebust wrote:
    >> In that case, I'll need a tcpdump of the point at which the
    >> error occurs.

     > Sure no problem... any particular preferred options ?  Want
     > client and server, or just client ?

Client should suffice. If you could start something like

tcpdump -s 256 -w tcpdump.out

close to the point at which the error occurs, and send me the
resulting file, then that should be OK. I'm mainly out to check
whether or not the server is the one returning the EIO error, or
possibly if it is returning bad post-op attributes. Those are the only
remaining possibilities if hard mounts are being used.

BTW: could you also tell me a bit about the server? Is this an ext[23]
partition and knfsd? I'm still a bit wary of ReiserFS...

Cheers,
   Trond

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > On Mon, 17 Dec 2001, Trond Myklebust wrote:
    >> tcpdump -s 256 -w tcpdump.out

     > also added "host equinox" (this is on a hub, and wanted to cut
     > out the noise), let me know if you want the unfiltered version.

No. I think I've got it. You are running NFSv2 (I assumed v3) and I'll
bet that if you look in your syslog, you'll see the error message

NFS: Odd RPC header size in read reply: 

Am I right?

This is a bug that was fixed ages ago in the NFSv3 code. I've updated
the 'fattr' patch yet again...

Cheers,
   Trond

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Trond Myklebust wrote:

> No. I think I've got it. You are running NFSv2 (I assumed v3) and I'll
> bet that if you look in your syslog, you'll see the error message
> NFS: Odd RPC header size in read reply:
> Am I right?

The only stuff that looks related is...

call_verify: server accept status: 2
call_verify: server accept status: 2
call_verify: server accept status: 2
RPC: garbage, exit EIO
nfs_get_root: getattr error = 5


> This is a bug that was fixed ages ago in the NFSv3 code. I've updated
> the 'fattr' patch yet again...

Ok, I'll give it a try soon..

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > The only stuff that looks related is...

     > call_verify: server accept status: 2 call_verify: server accept
     > status: 2 call_verify: server accept status: 2 RPC: garbage,
     > exit EIO nfs_get_root: getattr error = 5

Are the timestamps correct? If the server is replying with a 'garbage'
message, then certainly that will result in an EIO. However I didn't
see anything like that in your tcpdump, and the juxtaposed
'nfs_get_root' error rather points to that as being a mount error.

Cheers,
  Trond

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Trond Myklebust wrote:

> Are the timestamps correct? If the server is replying with a 'garbage'
> message, then certainly that will result in an EIO. However I didn't
> see anything like that in your tcpdump, and the juxtaposed
> 'nfs_get_root' error rather points to that as being a mount error.

Timestamps match on client and server, and are the correct times.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Trond Myklebust wrote:

> No. I think I've got it. You are running NFSv2 (I assumed v3) and I'll
> bet that if you look in your syslog, you'll see the error message
> This is a bug that was fixed ages ago in the NFSv3 code. I've updated
> the 'fattr' patch yet again...

And it fixes the problem, good work!
But.. (Here comes the sting..)

The test progresses just a little further and hits another bug..
http://www.codemonkey.org.uk/cruft/nfs-fsx2.txt

want tcpdump again?

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > And it fixes the problem, good work!  But.. (Here comes the
     > sting..)

     > The test progresses just a little further and hits another
     > bug..  http://www.codemonkey.org.uk/cruft/nfs-fsx2.txt

     > want tcpdump again?

Nah. I hit this one myself just half an hour after 1 fired off my last
mail.

'fattr' patch updated yet again...

The problem here was that the writer was creating a hole in the file
using the combination (llseek + write). Of course, the write was
cached, and so the server didn't know about the hole, and since the
subsequent read was to the hole, we didn't flush out any further
writes that might give the server a clue.
Result: server replies to our read request with a read of length
zero. Client then fails to zero out and mark those pages as uptodate
(bug!!!), and so 'generic_file_read' decides to return EIO.

Fixing the above has allowed me to progress a bit more. I've had fsx
running overnight (what little there has been). The good and bad news
is that it's still running.
However, 2 races + 4 bugs observed is already pretty much a record for
a single program. Kudos to the NeXT developers...

Cheers,
  Trond

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Trond Myklebust wrote:

> Nah. I hit this one myself just half an hour after 1 fired off my last
> mail.
> 'fattr' patch updated yet again...

Seems to do the trick. fsx has been running for about half hour so far
with no problems.

> However, 2 races + 4 bugs observed is already pretty much a record for
> a single program. Kudos to the NeXT developers...

Indeed. It's a really neat tool, we could use more like it.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Dave Jones wrote:

> > Nah. I hit this one myself just half an hour after 1 fired off my last
> > mail.
> > 'fattr' patch updated yet again...
> Seems to do the trick. fsx has been running for about half hour so far
> with no problems.

Here's an interesting one.. Run two instances of fsx together
using the same test file..

./fsx -c666 /mnt/nfs/noodles/test
./fsx -c667 /mnt/nfs/noodles/test

I typoed the 2nd line (and meant to do 'test2' concurrently)
Some kind of file locking problem?

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Trond Myklebust]

>>>>> " " == Dave Jones <davej@suse.de> writes:

     > Here's an interesting one.. Run two instances of fsx together
     > using the same test file..

     > ./fsx -c666 /mnt/nfs/noodles/test ./fsx -c667
     > /mnt/nfs/noodles/test

     > I typoed the 2nd line (and meant to do 'test2' concurrently)
     > Some kind of file locking problem?

Is that supposed to work? That gives 2 programs messing with the same
file 'test', but since the variable 'good_buf' is not shared, I'd
expect failure.

Cheers,
   Trond

Re: More fun with fsx.

[Dave Jones]

On Mon, 17 Dec 2001, Trond Myklebust wrote:

> > Some kind of file locking problem?
> Is that supposed to work? That gives 2 programs messing with the same
> file 'test', but since the variable 'good_buf' is not shared, I'd
> expect failure.

*nod*, this is repeatable on all fs's, so it looks like a shortcoming
of fsx rather than a fs failure.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[David S. Miller]

   From: Dave Jones <davej@suse.de>
   Date: Mon, 17 Dec 2001 15:30:32 +0100 (CET)

   On Mon, 17 Dec 2001, Trond Myklebust wrote:

   > However, 2 races + 4 bugs observed is already pretty much a record for
   > a single program. Kudos to the NeXT developers...
   
   Indeed. It's a really neat tool, we could use more like it.

It also is good at finding cache flushing bugs on platforms
where that is an issue :-)

BTW, here is a portability fix for fsx-linux.c :-)

--- fsx-linux.c.~1~	Mon Dec 17 16:24:11 2001
+++ fsx-linux.c	Mon Dec 17 16:28:19 2001
@@ -67,7 +67,7 @@
 #define OP_SKIPPED	7
 
 #ifndef PAGE_SIZE
-#define PAGE_SIZE       4096
+#define PAGE_SIZE       getpagesize()
 #endif
 #define PAGE_MASK       (PAGE_SIZE - 1)
 

Re: More fun with fsx.

[Dave Jones]

On Tue, 18 Dec 2001, David S. Miller wrote:

> BTW, here is a portability fix for fsx-linux.c :-)

Cool, danke.
I'll wrap it in an ifdef and push it to the Apple folks
when I send my next changes.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: More fun with fsx.

[Nathan Straz]

On Tue, Dec 18, 2001 at 09:36:10PM +0100, Dave Jones wrote:
> On Tue, 18 Dec 2001, David S. Miller wrote:
> 
> > BTW, here is a portability fix for fsx-linux.c :-)
> 
> Cool, danke.
> I'll wrap it in an ifdef and push it to the Apple folks
> when I send my next changes.

fsx had been added to the Linux Test Project with permission by Apple.
I've integrated the above change.  Any other changes can be submitted to
the LTP list or as a patch on the LTP project page.  fsx is filed in
ltp/testcases/kernel/fs/fsx-linux.

Thank to everyone how dug this up and polished it off.  
-- 
Nate Straz                                              nstraz@sgi.com
sgi, inc                                           http://www.sgi.com/
Linux Test Project                                  http://ltp.sf.net/

Re: More fun with fsx.

[Dave Jones]

On Thu, 20 Dec 2001, Nathan Straz wrote:

> fsx had been added to the Linux Test Project with permission by Apple.
> I've integrated the above change.  Any other changes can be submitted to
> the LTP list or as a patch on the LTP project page.  fsx is filed in
> ltp/testcases/kernel/fs/fsx-linux.

Great, thanks.. I pushed this down my TODO and hadn't got around to it
yet.. Can you make sure conrad@apple.com gets any worthwehile changes the
ltp makes to it too ?

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs