[BUG] 2.4.21: NFS copy produces I/O errors

[BUG] 2.4.21: NFS copy produces I/O errors

[Ole Marggraf]

Hello all.

As it seems (to me), there is some serious problem in the NFS code of
2.4.21 (and also of 2.4.20), causing I/O errors quite immediately.

Symptoms are as follows:

    Setup:

    Machine A, running 2.4.21 with NFSv3 client/server
    Machine B, running 2.4.19 with NFSv3 client/server
    (CONFIG_NFS_FS=y
     CONFIG_NFS_V3=y
     CONFIG_NFSD=y
     CONFIG_NFSD_V3=y
    )

    Both machines run the Kernel Automounter v4 (CONFIG_AUTOFS4_FS=y)

    (this should not matter: both machines are running debian testing,
     both have 3c905B ethernet cards)

We copy a file of ca. 100 MB size.

- logged in on machine A:
    cp /A_dir/file /B_dir: I/O error (after 1056768 bytes are written)

- logged in on machine B:
    cp /A_dir/file /B_dir: ok

No error messages are written to the logfiles (client and server), and the
same problem also occurs if we run 2.4.20 instead of 2.4.21.

If _both_ machines run 2.4.21, we get the same problem, i.e. pulling a
file via NFS to a local disk works, pushing to a remote disk fails.


Since we do not get any error messages in the logs, only the plain
"Input/output error" from cp, tracking down the problem is, from our side,
a bit difficult. The size of the file written until the error occurs could
help, it seems to be constant also for different files.

I am curious that this problem never had been observed before, since it
also seems to exist in 2.4.20. At least I did not find any previous
description in the archives.

If I can be of any further help, providing more information or do some
testing, just contact me. If you have any solution, please also give me a
hint ;-) This problem is a bit annoying...


Best regards,

Ole


-- 
+------------------------------------------------------------------------------+
 Ole Marggraf                     email: marggraf@gmx.net
 Sternwarte, Universitaet Bonn           marggraf@astro.uni-bonn.de
 Auf dem Huegel 71
 D-53121 Bonn, Germany            WWW:   http://www.astro.uni-bonn.de/~marggraf
+------------------------------------------------------------------------------+

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Trond Myklebust]

>>>>> " " == Ole Marggraf <marggraf@astro.uni-bonn.de> writes:

     > Hello all.  As it seems (to me), there is some serious problem
     > in the NFS code of
     > 2.4.21 (and also of 2.4.20), causing I/O errors quite
     >        immediately.


This is a FAQ, has been discussed to death several times on this list,
and is all over the place in the archives.... 'man 5 nfs' and read
carefully the section on soft mounts.


Trond (I'm really sick of this question) Myklebust

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Ole Marggraf]

Hello Trond, Andy.

Thank you, both, for the prompt reply.


On Mon, 16 Jun 2003, Andrew Ryan wrote:

> Trond, will tell you to always use hard mounts.  Though I believe it is an
> excuse to not fix what it really broken since other UNIXes seem to not have
> problems with soft mounts, it is a Linux problem, IMHO.

Actually, I just played around with the timeouts... It seems to work (only
one try, which is quite bad statistics...) also for a soft mount (hard
mount works anyway) when I raise "retrans" to 20. At retrans=15 I still
get a timeout (but much later, after about 30MB transmitted). This is for
an internal connection via only one switch, and with no significant
network load at the moment.

Which makes me wonder, since our system did not have any such problems for
years with the default retrans=3 (and soft mounts), until we ran into
2.4.20. That's why I actually did not expect that there was a need for
finetuning these parameters, and not by this amount... . 2.4.19 was
running fine, so something has to have changed on the way to 2.4.20.

> I don't know is this is the "right" way to fix this, but this patch has
> worked for me.  Kernel versions should not matter, as long as you add the
> !clnt->cl_softrtry to the if statement shown below things should work.
>
> But if you don't want to try this :

I will test the patch tomorrow, yes, thank you. I'll keep you informed.


Best regards,

Ole

-- 
+------------------------------------------------------------------------------+
 Ole Marggraf                     email: marggraf@gmx.net
 Sternwarte, Universitaet Bonn           marggraf@astro.uni-bonn.de
 Auf dem Huegel 71
 D-53121 Bonn, Germany            WWW:   http://www.astro.uni-bonn.de/~marggraf
+------------------------------------------------------------------------------+

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Felipe Alfaro Solana]

On Mon, 2003-06-16 at 20:51, Ole Marggraf wrote:
> Hello all.
> 
> As it seems (to me), there is some serious problem in the NFS code of
> 2.4.21 (and also of 2.4.20), causing I/O errors quite immediately.

By the way, are you using the "soft" option to mount the NFS volumes?

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Ole Marggraf]

On Mon, 16 Jun 2003, Felipe Alfaro Solana wrote:

> On Mon, 2003-06-16 at 20:51, Ole Marggraf wrote:
> > Hello all.
> >
> > As it seems (to me), there is some serious problem in the NFS code of
> > 2.4.21 (and also of 2.4.20), causing I/O errors quite immediately.
>
> By the way, are you using the "soft" option to mount the NFS volumes?
>

Hello.

Yes, forgot to mention... But its quite clear from the error message.

Mount options are rw,soft,bw,rsize=8192,wsize=8192.

And, as I wrote somewhere else, increasing retrans to 20 helps, which is
quite far from the default (default retrans=3 did work under 2.4.19).

Best regards,

Ole

-- 
+------------------------------------------------------------------------------+
 Ole Marggraf                     email: marggraf@gmx.net
 Sternwarte, Universitaet Bonn           marggraf@astro.uni-bonn.de
 Auf dem Huegel 71
 D-53121 Bonn, Germany            WWW:   http://www.astro.uni-bonn.de/~marggraf
+------------------------------------------------------------------------------+

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Ole Marggraf]

> > By the way, are you using the "soft" option to mount the NFS volumes?
> >
>
> Yes, forgot to mention... But its quite clear from the error message.
>
> Mount options are rw,soft,bw,rsize=8192,wsize=8192.

Ok, correction. (I should go home and get some sleep...)
rsize=8192,wsize=8192 were not set in the original testing setup, I had
the defaults there (1024 both). The amount of bytes written till the I/O
error gets up does scale with that option.

Ole

-- 
+------------------------------------------------------------------------------+
 Ole Marggraf                     email: marggraf@gmx.net
 Sternwarte, Universitaet Bonn           marggraf@astro.uni-bonn.de
 Auf dem Huegel 71
 D-53121 Bonn, Germany            WWW:   http://www.astro.uni-bonn.de/~marggraf
+------------------------------------------------------------------------------+

Re: [BUG] 2.4.21: NFS copy produces I/O errors

[Felipe Alfaro Solana]

On Tue, 2003-06-17 at 00:04, Ole Marggraf wrote:
> > > By the way, are you using the "soft" option to mount the NFS volumes
> >
> > Yes, forgot to mention... But its quite clear from the error message.
> >
> > Mount options are rw,soft,bw,rsize=8192,wsize=8192.
> 
> Ok, correction. (I should go home and get some sleep...)
> rsize=8192,wsize=8192 were not set in the original testing setup, I had
> the defaults there (1024 both). The amount of bytes written till the I/O
> error gets up does scale with that option.

Please, don't use the "soft" option. Instead, use "hard" as there are
problems with the former. I had the same problems with I/O errors. Trond
told me to stick with "hard" at the moment.