[NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[Hans-Peter Jansen]

Hi Trond et al.,

after applying $subject and resolving the rej with 2.4.18-pre1, 
I found the following strangeness in my SuSE 7.3 based NFS diskless setup:

Trying to start openuniverse for the first time (ever), it complaint about
a missing config file. SuSE installed it's data at /usr/share/openuniverse 
with the config file conf/ou.conf symlinked to ../../../../etc/ou.conf (=> 
/etc/ou.conf), which was missing within my diskless setup.

The problem is, ls on the client side complains about an I/O error, when 
listing the conf/ dir.

After removing this symlink (within the server), ls is OK within
the client. Trying to copy servers /etc/ou.conf file to
/usr/share/openuniverse within the client, cp complains about to many levels 
of symlinks?!? (/usr is shared)

May be I missed/confused some patches. I could send you a plain
copy of the interresting modules, if you like. 

Somehow, the dir entry on the client survived the rm from the server.

Compiling/KDE 2.2.2 Desktop is working fine so far.

Any ideas?

Cheers,
Hans-Peter

[NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[Trond Myklebust]

>>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:

     > The problem is, ls on the client side complains about an I/O
     > error, when listing the conf/ dir.

What server is this?

     > After removing this symlink (within the server), ls is OK
     > within the client. Trying to copy servers /etc/ou.conf file to
     > /usr/share/openuniverse within the client, cp complains about
     > to many levels of symlinks?!? (/usr is shared)

That can happen, yes. The symlink is still in the dcache, and so the
VFS thinks that we want to open whatever it is that the symlink is
pointing to (not the symlink itself). For this reason, less strict
checking is performed, and so the client does not immediately see the
change.

If, however, you had first done 'ls -l' or something that tries to
read the symlink itself, more strict revalidation checks are
performed, and the stale dentry would have been detected.

I can tighten the checks on this sort of thing a bit, but if so, it
needs to be done carefully. It is important to make sure that
operations like
   'ls /usr/lib/*'
(in which you want the system to repeatedly look up the same path) are
efficient by caching the '/usr/lib' bit even if that /usr/lib is a
symlink.
Of course every time we do open("/usr/lib/libc.so"), we *do* want to
make sure that we perform strict checks when we do the lookup of the
last element of the path (on the actual file "libc.so").

Cheers,
  Trond

Re: [NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[Hans-Peter Jansen]

On Friday, 11. January 2002 16:07, Trond Myklebust wrote:
> >>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:
>      > The problem is, ls on the client side complains about an I/O
>      > error, when listing the conf/ dir.
>
> What server is this?

Oups, sorry: 
Linux shrek 2.4.13-ac7 #6 SMP Son Dez 2 20:02:04 CET 2001 i686 unknown
(on patches, IIRC)
>
>      > After removing this symlink (within the server), ls is OK
>      > within the client. Trying to copy servers /etc/ou.conf file to
>      > /usr/share/openuniverse within the client, cp complains about

Oups, correction: /usr/share/openuniverse/conf

>      > to many levels of symlinks?!? (/usr is shared)
>
> That can happen, yes. The symlink is still in the dcache, and so the
> VFS thinks that we want to open whatever it is that the symlink is
> pointing to (not the symlink itself). For this reason, less strict
> checking is performed, and so the client does not immediately see the
> change.

Shouldn't the client invalidate the dcache entry some time?
Now, 5 hours later, the dcache entry isn't invalidated, yet.

I think, this is a thinko. I'm not familiar with nfs internals,
but have done stuff like this before (anybody remember BioNet?).

Please try this:

shrek is the server, elfe the client
/raid is shared (reiserfs, no raid), /tmp isn't

shrek:/raid/tmp# touch /tmp/huhu
shrek:/raid/tmp# ln -s /tmp/huhu
shrek:/raid/tmp# l
total 1
drwxr-xr-x    2 root     root           55 Jan 11 20:07 ./
drwxr-xr-x    9 root     root          218 Jan 11 20:06 ../
lrwxrwxrwx    1 root     root            9 Jan 11 20:07 huhu -> /tmp/huhu

elfe:/raid/tmp# echo "huhu" > /tmp/huhu
elfe:/raid/tmp# l
insgesamt 1
drwxr-xr-x    2 root     root           55 11. Jan 20:07 ./
drwxr-xr-x    9 root     root          218 11. Jan 20:06 ../
lrwxrwxrwx    1 root     root            9 11. Jan 20:07 huhu -> /tmp/huhu
elfe:/raid/tmp# cat huhu 
huhu

shrek:/raid/tmp# rm huhu

elfe:/raid/tmp# l
insgesamt 1
drwxr-xr-x    2 root     root           35 11. Jan 20:07 ./
drwxr-xr-x    9 root     root          218 11. Jan 20:06 ../
elfe:/raid/tmp# touch huhu
elfe:/raid/tmp# l
ls: huhu: Eingabe-/Ausgabefehler
insgesamt 1
drwxr-xr-x    2 root     root           55 11. Jan 20:08 ./
drwxr-xr-x    9 root     root          218 11. Jan 20:06 ../

Pretty simple. Can you reproduce this?

> If, however, you had first done 'ls -l' or something that tries to
> read the symlink itself, more strict revalidation checks are
> performed, and the stale dentry would have been detected.

Not sure.

> I can tighten the checks on this sort of thing a bit, but if so, it
> needs to be done carefully. It is important to make sure that
> operations like
>    'ls /usr/lib/*'
> (in which you want the system to repeatedly look up the same path) are
> efficient by caching the '/usr/lib' bit even if that /usr/lib is a
> symlink.
> Of course every time we do open("/usr/lib/libc.so"), we *do* want to
> make sure that we perform strict checks when we do the lookup of the
> last element of the path (on the actual file "libc.so").
>
> Cheers,
>   Trond

Thank for your patience,
Hans-Peter

Re: [NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[Trond Myklebust]

>>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:

     > Are you sure that you didn't mess up the fixups with
     > 2.4.18-pre1? That might explain things, since you would be
     > messing with nfs_refresh_inode. What you need to do against
     > 2.4.18-pre1 is first to revert the patch
     > linux-2.4.17-fattr.dif. After that you should be able to apply
     > linux-2.4.17-NFS_ALL.dif directly without any rejections...

Please also note that the version of linux-2.4.17-NFS_ALL.dif that
appeared on my site dated prior to 20th December had a bug within
nfs_refresh_inode that cause corruption of file attributes.
Is it possible that you might be using this buggy version?

Cheers,
  Trond

[NFS] some strangeness (at least) with linux-2.4.18-NFS_ALL patch

[Hans-Peter Jansen]

Hey, Howdy, Hey Trond, [forgive me, watched Toy Story 2 dvd today :)]

me again ;-)

On Saturday, 12. January 2002 03:24, Trond Myklebust wrote:
> >>>>> " " == Trond Myklebust <trond.myklebust@fys.uio.no> writes:
>      > Are you sure that you didn't mess up the fixups with
>      > 2.4.18-pre1? That might explain things, since you would be
>      > messing with nfs_refresh_inode. What you need to do against
>      > 2.4.18-pre1 is first to revert the patch
>      > linux-2.4.17-fattr.dif. After that you should be able to apply
>      > linux-2.4.17-NFS_ALL.dif directly without any rejections...

I've messed it up, mea culpa. Sorry!

I redid my kernel today, using your 2.4.18-NFS_ALL from last night ;-),
based on 2.4.18-pre3, but kept out Andrea's 00_lowlatency-fixes-4 for 
now:
imon-0.0.2-2.4.12-hp
linux-2.4.18-NFS_ALL.dif
00_nanosleep-5.dif
pnpbios.patch_latest
vmscan.patch.2.4.17.dif
ide.2.4.16.12102001.patch.bz2
bttv-0.7.88-2.4.17
btaudio-2.4.17
and my symlink problem disappeared. I thought, I could reproduce a
longer standing problem, which bites me from time to time.

Here we go:

When it came to the usual lm_sensors 2.6.2 modules install within 
the client, this happened:

[...]
gcc -shared -Wl,-soname,libsensors.so.1 -o lib/libsensors.so.1.2.0 
lib/data.lo lib/general.lo lib/error.lo lib/chips.lo lib/proc.lo 
lib/access.lo lib/init.lo lib/conf-parse.lo lib/conf-lex.lo -lc
make: stat:lib/libsensors.so.1: Eingabe-/Ausgabefehler
rm -f lib/libsensors.so.1
rm: Entfernen von »lib/libsensors.so.1« nicht möglich: Eingabe-/Ausgabefehler
make: *** [lib/libsensors.so.1] Error 1
hp@elfe:~/Downloads/linux/lm_sensors/lm_sensors-2.6.2> l lib/libsensors.*   
ls: lib/libsensors.so.1: Eingabe-/Ausgabefehler
-rw-rw-r--    1 hp       lisa         6559  8. Feb 1999  lib/libsensors.3
-rw-rw-r--    1 hp       lisa        87100 12. Jan 16:58 lib/libsensors.a
lrwxrwxrwx    1 hp       lisa           19 12. Jan 16:58 lib/libsensors.so -> 
libsensors.so.1.2.0*
-rwxrwxr-x    1 root     root        83142 12. Jan 16:59 
lib/libsensors.so.1.2.0*

server's view:
hp@shrek:~/Downloads/linux/lm_sensors/lm_sensors-2.6.2/lib> l libsensors.*
-rw-rw-r--    1 hp       lisa         6559 Feb  8  1999 libsensors.3
-rw-rw-r--    1 hp       lisa        87100 Jan 12 16:58 libsensors.a
lrwxrwxrwx    1 hp       lisa           19 Jan 12 16:58 libsensors.so -> 
libsensors.so.1.2.0*
lrwxrwxrwx    1 hp       lisa           19 Jan 12 16:58 libsensors.so.1 -> 
libsensors.so.1.2.0*
-rwxrwxr-x    1 root     root        83142 Jan 12 16:59 libsensors.so.1.2.0*

Somehow, gcc managed it to create an invalid link in the above sequence.
Really bad is, you cannot get around this within the client. After 
rm'ing lib/libsensors.so.1 on the server, make install succeeds on the client.

[...]
rm -f lib/libsensors.so.1
ln -sfn libsensors.so.1.2.0 lib/libsensors.so.1
mkdir -p /usr/lib /usr/include/sensors /usr/man/man3 /usr/man/man5
install -o root -g root -m 644 lib/libsensors.a lib/libsensors.so.1.2.0 
lib/libsensors.so.1 lib/libsensors.so /usr/lib
ln -sfn libsensors.so.1.2.0 /usr/lib/libsensors.so.1
ln -sfn libsensors.so.1 /usr/lib/libsensors.so
[...]

I can reproduce this now. Do you?

Hope, I don't bother you above the critical mass, because I don't want
to trigger the third world war, in case you get out of control and explode
unconditionally. Warn me in time plz :)

Happy NFSing'ly yours,
Hans-Peter

[NFS] some strangeness (at least) with linux-2.4.18-NFS_ALL patch

[Trond Myklebust]

>>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:

     > Somehow, gcc managed it to create an invalid link in the above
     > sequence.  Really bad is, you cannot get around this within the
     > client. After rm'ing lib/libsensors.so.1 on the server, make
     > install succeeds on the client.

<snip>

     > I can reproduce this now. Do you?

Nope.

gcc -shared -Wl,-soname,libsensors.so.1 -o lib/libsensors.so.1.2.0
lib/data.lo lib/general.lo lib/error.lo lib/chips.lo lib/proc.lo
lib/access.lo lib/init.lo lib/conf-parse.lo lib/conf-lex.lo -lc
rm -f lib/libsensors.so.1
ln -sfn libsensors.so.1.2.0 lib/libsensors.so.1
rm -f lib/libsensors.so
ln -sfn libsensors.so.1.2.0 lib/libsensors.so

Could you try just plain 2.4.18-pre3 + 2.4.18-NFS_ALL? That is what I
am running (on both client + server).

Cheers,
   Trond

Re: [NFS] some strangeness (at least) with linux-2.4.18-NFS_ALL patch

[Hans-Peter Jansen]

On Saturday, 12. January 2002 19:47, Trond Myklebust wrote:
> >>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:
>      > Somehow, gcc managed it to create an invalid link in the above
>      > sequence.  Really bad is, you cannot get around this within the
>      > client. After rm'ing lib/libsensors.so.1 on the server, make
>      > install succeeds on the client.
>
> <snip>
>
>      > I can reproduce this now. Do you?
>
> Nope.
>
> gcc -shared -Wl,-soname,libsensors.so.1 -o lib/libsensors.so.1.2.0
> lib/data.lo lib/general.lo lib/error.lo lib/chips.lo lib/proc.lo
> lib/access.lo lib/init.lo lib/conf-parse.lo lib/conf-lex.lo -lc
> rm -f lib/libsensors.so.1
> ln -sfn libsensors.so.1.2.0 lib/libsensors.so.1
> rm -f lib/libsensors.so
> ln -sfn libsensors.so.1.2.0 lib/libsensors.so
>
> Could you try just plain 2.4.18-pre3 + 2.4.18-NFS_ALL? That is what I
> am running (on both client + server).

Ok, done. Sad news, problem persist:
On compile:
gcc -shared -Wl,-soname,libsensors.so.1 -o lib/libsensors.so.1.2.0 
lib/data.lo lib/general.lo lib/error.lo lib/chips.lo lib/proc.lo 
lib/access.lo lib/init.lo lib/conf-parse.lo lib/conf-lex.lo -lc
rm -f lib/libsensors.so.1
ln -sfn libsensors.so.1.2.0 lib/libsensors.so.1
make: stat:lib/libsensors.so.1: Eingabe-/Ausgabefehler
rm -f lib/libsensors.so 
ln -sfn libsensors.so.1.2.0 lib/libsensors.so

On install:
make: stat:lib/libsensors.so.1: Eingabe-/Ausgabefehler
rm -f lib/libsensors.so.1
rm: Entfernen von »lib/libsensors.so.1« nicht möglich: Eingabe-/Ausgabefehler
make: *** [lib/libsensors.so.1] Error 1

Some system parameter:
server:
Dual Pentium 3/500, Asus P2B DS, 512 MB 
Linux version 2.4.18-pre3 (hp@shrek) (gcc version 2.95.3 20010315 (SuSE)) #2 
SMP Sam Jan 12 21:00:36 CET 2002
knfsd, reiserfs partition

client:
Athlon 1.2, Asus Via KT133, 768 MB
Linux version 2.4.18-pre3 (hp@elfe) (gcc version 2.95.3 20010315 (SuSE)) #3
Sam Jan 12 21:26:40 CET 2002
mount opt:
rw,nodev,v3,rsize=4096,wsize=4096,soft,intr,udp,lock,addr=shrek

Any more ideas? What's wrong with my setup?

Cheers,
Hans-Peter

Re: [NFS] some strangeness (at least) with linux-2.4.18-NFS_ALL patch

[Trond Myklebust]

>>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:

     > client: Athlon 1.2, Asus Via KT133, 768 MB Linux version
     > 2.4.18-pre3 (hp@elfe) (gcc version 2.95.3 20010315 (SuSE)) #3
     > Sam Jan 12 21:26:40 CET 2002 mount opt:
     > rw,nodev,v3,rsize=4096,wsize=4096,soft,intr,udp,lock,addr=shrek

     > Any more ideas? What's wrong with my setup?

10 to 1 it's the 'soft' mount option.

Cheers,
   Trond

Re: [NFS] some strangeness (at least) with linux-2.4.17-NFS_ALL patch

[Trond Myklebust]

>>>>> " " == Hans-Peter Jansen <hpj@urpla.net> writes:
     >> What server is this?
     > Oups, sorry:
     > Linux shrek 2.4.13-ac7 #6 SMP Son Dez 2 20:02:04 CET 2001 i686 unknown
     > (on patches, IIRC)

And the knfsd server daemons (not the user-space 'unfsd')?

     > Shouldn't the client invalidate the dcache entry some time?
     > Now, 5 hours later, the dcache entry isn't invalidated, yet.

5 hours??? Unless you have set some crazy value for acdirmax, then it
should indeed have been revalidated under normal circumstances.

The default is for all attribute/dcache entries to time out at least
once every 60 seconds. That is what happens on my test setup at least.

     > I think, this is a thinko.

It shouldn't be. There might be a bug lingering somewhere, but
revalidation *is* normally supposed to occur.

Are you sure that you didn't mess up the fixups with 2.4.18-pre1? That
might explain things, since you would be messing with
nfs_refresh_inode. What you need to do against 2.4.18-pre1 is first to
revert the patch linux-2.4.17-fattr.dif. After that you should be able
to apply linux-2.4.17-NFS_ALL.dif directly without any rejections...

     > Please try this:

     > shrek is the server, elfe the client /raid is shared (reiserfs,
     > no raid), /tmp isn't

<snip>

     > Eingabe-/Ausgabefehler insgesamt 1 drwxr-xr-x 2 root root 55
     > 11. Jan 20:08 ./ drwxr-xr-x 9 root root 218 11. Jan 20:06 ../

     > Pretty simple. Can you reproduce this?

Nope. With /mnt/trondmy == NFS share, /tmp == local:

[trondmy@fyspc-epf03 gnurr]$ pwd
/mnt/trondmy/gnurr
[trondmy@fyspc-epf03 gnurr]$ touch /tmp/huhu
[trondmy@fyspc-epf03 gnurr]$ ln -s /tmp/huhu
[trondmy@fyspc-epf03 gnurr]$ ls -al
totalt 8
drwxr-xr-x    2 trondmy  trondmy      4096 jan 12 02:43 .
drwxr-xr-x   41 trondmy  trondmy      4096 jan 12 02:42 ..
lrwxrwxrwx    1 trondmy  trondmy         9 jan 12 02:43 huhu ->
/tmp/huhu
[trondmy@fyspc-epf03 gnurr]$ echo "huhu" > /tmp/huhu
[trondmy@fyspc-epf03 gnurr]$ ls -al
totalt 8
drwxr-xr-x    2 trondmy  trondmy      4096 jan 12 02:43 .
drwxr-xr-x   41 trondmy  trondmy      4096 jan 12 02:42 ..
lrwxrwxrwx    1 trondmy  trondmy         9 jan 12 02:43 huhu ->
/tmp/huhu
[trondmy@fyspc-epf03 gnurr]$ cat huhu
huhu
[trondmy@fyspc-epf03 gnurr]$ rm huhu
[trondmy@fyspc-epf03 gnurr]$ ls -al
totalt 8
drwxr-xr-x    2 trondmy  trondmy      4096 jan 12 02:44 .
drwxr-xr-x   41 trondmy  trondmy      4096 jan 12 02:42 ..
[trondmy@fyspc-epf03 gnurr]$ touch huhu
[trondmy@fyspc-epf03 gnurr]$ ls -al
totalt 8
drwxr-xr-x    2 trondmy  trondmy      4096 jan 12 02:44 .
drwxr-xr-x   41 trondmy  trondmy      4096 jan 12 02:42 ..
-rw-r--r--    1 trondmy  trondmy         0 jan 12 02:44 huhu

Cheers,
  Trond