Re: Bugs in knfsd -- Problem re-exporting an NFS share

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Andrzej Krzysztofowicz]

> On Friday December 29, Frank.Olsen@stonesoft.com wrote:
> > Hi -- could you please CC me if you reply to this mail.
> > 
> > A:     /exports/A                                 - Redhat 7.0
> > B1/B2: mount /exports/A on /export/A from A       - Redhat 6.2
> > C:     mount /exports/A on /mnt/A from B1 or B2   - Redhat 6.2
> > 
> > I use knfsd/nfs-utils on each machine.
> > 
> > bash# ls /mnt/A
> > /mnt/A/A.txt: No such file or directory
> 
> This is not a supported configuration.  You cannot export NFS mounted
> filesystems with NFS. The protocol does not cope, and it
> implementation doesn't even try.
> NFS is for export local filesystems only.

As I understand problem is somewhere else.
If this is intentionally unsupported configuration - OK. So why the error
appears ? The directory should be empty then.

If the configuration is unsupported at the moment and the  A.txt file is
located on A, some code that attempts to read re-exported files/directories
should be turned off (eg. #if 0).

If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug.
Sucgh a file should be hidden at the act of mounting. For both local and
remote access.

Neil, could you tell us where the A.txt file is *really* located ?

Regards 
   Andrzej

BTW. AFAIR, I observed similar behaviour (files are visible but
     inaccessible) while mounting a local filesystem at a busy directory
     (eg.: mount /dev/fd0 .;ls -l) even in 2.2...

-- 
=======================================================================
  Andrzej M. Krzysztofowicz               ankry@mif.pg.gda.pl
  phone (48)(58) 347 14 61
Faculty of Applied Phys. & Math.,   Technical University of Gdansk
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Neil Brown]

On Saturday December 30, ankry@green.mif.pg.gda.pl wrote:
> 
> > On Friday December 29, Frank.Olsen@stonesoft.com wrote:
> > > Hi -- could you please CC me if you reply to this mail.
> > > 
> > > A:     /exports/A                                 - Redhat 7.0
> > > B1/B2: mount /exports/A on /export/A from A       - Redhat 6.2
> > > C:     mount /exports/A on /mnt/A from B1 or B2   - Redhat 6.2
> > > 
> > > I use knfsd/nfs-utils on each machine.
> > > 
> > > bash# ls /mnt/A
> > > /mnt/A/A.txt: No such file or directory
> > 
> > This is not a supported configuration.  You cannot export NFS mounted
> > filesystems with NFS. The protocol does not cope, and it
> > implementation doesn't even try.
> > NFS is for export local filesystems only.
> 
> As I understand problem is somewhere else.
> If this is intentionally unsupported configuration - OK. So why the error
> appears ? The directory should be empty then.
> 
> If the configuration is unsupported at the moment and the  A.txt file is
> located on A, some code that attempts to read re-exported files/directories
> should be turned off (eg. #if 0).
> 
> If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug.
> Sucgh a file should be hidden at the act of mounting. For both local and
> remote access.
> 
> Neil, could you tell us where the A.txt file is *really* located ?

Well.... its a long story.

The interface between knfsd and filesystems is not (currently) a clean
one.  The current interface to filesystems is dictated by the VFS
layer, and it works primarily in terms of names.
knfsd wants to work in terms of inode numbers (or something similar)
and there is no such interface.

knfsd does manage to do its job, but only by abusing part of the VFS
interface - the "iget" routine which calls s_op->read_inode.
iget is meant to be an private interface, only ever called by the
filesystem itself (typically in the lookup operation) but, for ext2 at
least, it works for knfsd, so knfsd uses it.

Basically, knfsd assumes that is a filesystem has "read_inode"
defined, then it can safely use iget.

This works for ext2, but not for nfs.

The nfs filesystem does define read_inode, but uses it quite
differently to ext2.

The net result is that when you try to map an inode number to an inode
using iget with ext2 it always works, but if you do it with nfs it only
works if the inode is already in the cache.

So, when you:
   ls /mnt/A
from C, knfsd on  B1 (or B2) tries to find the directory  'A' and
succeeds beause it is in cache.  It does a readdir on this directory
and gets a list of files including "A.txt".
But when it tries to access A.txt, iget doesn't work for it, and it
cannot find it.

So A.txt is really on A, not on B1 or B2.  But when you look at A.txt
from C through B1 or B2 you only get a blurred image.  You can see
that it is there, but not what it is.

Does that help?

NeilBrown


> 
> Regards 
>    Andrzej
> 
> BTW. AFAIR, I observed similar behaviour (files are visible but
>      inaccessible) while mounting a local filesystem at a busy directory
>      (eg.: mount /dev/fd0 .;ls -l) even in 2.2...
> 
> -- 
> =======================================================================
>   Andrzej M. Krzysztofowicz               ankry@mif.pg.gda.pl
>   phone (48)(58) 347 14 61
> Faculty of Applied Phys. & Math.,   Technical University of Gdansk
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Jorge L. deLyra]

> This is not a supported configuration.  You cannot export NFS mounted
> filesystems with NFS. The protocol does not cope, and it
> implementation doesn't even try.
> NFS is for export local filesystems only.

Just want to point out that this unsupported feature, which exists in the
old user-space NFS server, is very useful under some circumstances.

Case in point: you have a Beowulf-style parallel cluster on a private
network, physically connected to the Internet only through the front-end,
but with no direct routing to it, which saves addresses and security
concerns (just imagine you have 1000 nodes); you want to import on the
front-end various homes of users scattered around your lan, and then have
the front-end re-export them to the cluster on the private network.

Since the private network does not talk directly to the Internet, there is
no danger of loopback mounts locking up. This scheme makes the use of the
cluster, usually a resource shared among various groups and institutions,
very convenient for all users: their own user configs and environment are
available when they use the cluster, logging in through the front-end, and
their data goes out transparently to their home systems. Very nice!

It would be nice if a way was found to implement this feature on knfsd.

						Best regards,

----------------------------------------------------------------
        Jorge L. deLyra,  Associate Professor of Physics
            The University of Sao Paulo,  IFUSP-DFMA
       For more information: finger delyra@latt.if.usp.br
----------------------------------------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Andi Kleen]

On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote:
> It would be nice if a way was found to implement this feature on knfsd.

There is: just run unfsd too, bound to an own IP address to not conflict. 

More efficient would probably be a proxy though that just forwards packets.
I see no reason in core NFS why multiple clients could not share a single 
multiplexed mount. It could need some lockd support though. 

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Jorge L. deLyra]

> Date: Wed, 3 Jan 2001 11:52:36 +0100
> From: Andi Kleen <ak@suse.de>
> To: "Jorge L. deLyra" <delyra@latt.if.usp.br>
> Cc: Neil Brown <neilb@cse.unsw.edu.au>,
>     Andrzej Krzysztofowicz <ankry@green.mif.pg.gda.pl>,
>     Frank.Olsen@stonesoft.com, kernel list <linux-kernel@vger.kernel.org>
> Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share
> 
> On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote:
> > It would be nice if a way was found to implement this feature on knfsd.
> 
> There is: just run unfsd too, bound to an own IP address to not conflict. 

Hum, interesting, I didn't know you could run both servers at the same
time. The problem is that the old server is not being developed any more,
as dully advised on the 2.2.18 kernel help messages. In fact, we have been
having some problems with it for some time, I/O errors and mounts hanging,
mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases
around here, until we migrated to knfsd: gave us sure-fire I/O errors and
hangs. So this might be a stopgap measure for some time, but no permanent
solution. In any case, knfsd seems so much more solid, we want to use it!

> More efficient would probably be a proxy though that just forwards packets.
> I see no reason in core NFS why multiple clients could not share a single 
> multiplexed mount. It could need some lockd support though. 

I'm afraid you lost me here... If this would have to be some kind of new
daemon or kernel module that filters out NFS packets and forwards them,
wouldn't this be equivalent to making knfsd do the same thing? Humm, on
second thoughts, maybe not, because this beast would not have to interpret
the mount as a filesystem on the front end. Am I following you correctly?

----------------------------------------------------------------
        Jorge L. deLyra,  Associate Professor of Physics
            The University of Sao Paulo,  IFUSP-DFMA
       For more information: finger delyra@latt.if.usp.br
----------------------------------------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Andi Kleen]

On Wed, Jan 03, 2001 at 09:14:21AM -0200, Jorge L. deLyra wrote:
> > Date: Wed, 3 Jan 2001 11:52:36 +0100
> > From: Andi Kleen <ak@suse.de>
> > To: "Jorge L. deLyra" <delyra@latt.if.usp.br>
> > Cc: Neil Brown <neilb@cse.unsw.edu.au>,
> >     Andrzej Krzysztofowicz <ankry@green.mif.pg.gda.pl>,
> >     Frank.Olsen@stonesoft.com, kernel list <linux-kernel@vger.kernel.org>
> > Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share
> > 
> > On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote:
> > > It would be nice if a way was found to implement this feature on knfsd.
> > 
> > There is: just run unfsd too, bound to an own IP address to not conflict. 
> 
> Hum, interesting, I didn't know you could run both servers at the same
> time. The problem is that the old server is not being developed any more,
> as dully advised on the 2.2.18 kernel help messages. In fact, we have been
> having some problems with it for some time, I/O errors and mounts hanging,
> mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases
> around here, until we migrated to knfsd: gave us sure-fire I/O errors and
> hangs. So this might be a stopgap measure for some time, but no permanent
> solution. In any case, knfsd seems so much more solid, we want to use it!

That's a bit surprising that you have so many problems with unfsd, I know
lots of installations where it is (still) used successfully in its limits.
I did recently some mainteance work on it to fix it for reiserfs.

> 
> > More efficient would probably be a proxy though that just forwards packets.
> > I see no reason in core NFS why multiple clients could not share a single 
> > multiplexed mount. It could need some lockd support though. 
> 
> I'm afraid you lost me here... If this would have to be some kind of new
> daemon or kernel module that filters out NFS packets and forwards them,
> wouldn't this be equivalent to making knfsd do the same thing? Humm, on
> second thoughts, maybe not, because this beast would not have to interpret
> the mount as a filesystem on the front end. Am I following you correctly?

(when you ignore locking and NFS over TCP ATM) All you need is to forward
UDP packets. This can either be done by a normal user space daemon that
uses portmap and just sends the packets out again, or alternatively by using
UDP masquerading and appropiate routes on the internal boxes.
Only forwarding packets is very different from full reexport support in knfsd
and much simpler.


-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Jorge L. deLyra]

> That's a bit surprising that you have so many problems with unfsd, I know
> lots of installations where it is (still) used successfully in its limits.
> I did recently some mainteance work on it to fix it for reiserfs.

Well, let me qualify that better. The problems were not really bad before
2.2.18. There where none in 2.0, by the way, as far as I remember. They
first showed up when we switched to 2.2, and were more frequent at the
beginning. When we got to 2.2.17 we had pretty much forgotten about them.
We have NFS exports on many machines around here and it only happened in
some, so it is difficult to pinpoint the problem. We were never quite sure
whether the problem was on the server or client, sometimes it looked one
way, sometimes another. Once there was a certain directory such that an ls
-lR on it was sure-fire to give an I/O error message and hang the mount.
Moving the contents to another directory and deleting the first one solved
it, don't ask me why. Very often a way to produce the problem was to sit
at the mount point and do a find for something. When a hang was produced,
you had to kill autofs, unmount by hand, maybe killing a few programs in
order not to get "busy" messages. Then just mount it again and it's OK.
Note that neither killing and restarting the server nor rebooting the
client seemed necessary. You just had to unmount and remount.

> (when you ignore locking and NFS over TCP ATM) All you need is to forward
> UDP packets. This can either be done by a normal user space daemon that
> uses portmap and just sends the packets out again, or alternatively by using
> UDP masquerading and appropiate routes on the internal boxes.
> Only forwarding packets is very different from full reexport support in knfsd
> and much simpler.

Well, I hope some solution is found, since this is an important feature.
It would be nicer in knfsd, I think, but if that proves unpractical for
some reason, your packet filter/forwarder might just be the answer.

							Best luck,

----------------------------------------------------------------
        Jorge L. deLyra,  Associate Professor of Physics
            The University of Sao Paulo,  IFUSP-DFMA
       For more information: finger delyra@latt.if.usp.br
----------------------------------------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Andi Kleen]

On Wed, Jan 03, 2001 at 10:49:54AM -0200, Jorge L. deLyra wrote:
> > That's a bit surprising that you have so many problems with unfsd, I know
> > lots of installations where it is (still) used successfully in its limits.
> > I did recently some mainteance work on it to fix it for reiserfs.
> 
> Well, let me qualify that better. The problems were not really bad before
> 2.2.18. There where none in 2.0, by the way, as far as I remember. They
> first showed up when we switched to 2.2, and were more frequent at the
> beginning. When we got to 2.2.17 we had pretty much forgotten about them.
> We have NFS exports on many machines around here and it only happened in
> some, so it is difficult to pinpoint the problem. We were never quite sure
> whether the problem was on the server or client, sometimes it looked one
> way, sometimes another. Once there was a certain directory such that an ls
> -lR on it was sure-fire to give an I/O error message and hang the mount.
> Moving the contents to another directory and deleting the first one solved
> it, don't ask me why. Very often a way to produce the problem was to sit
> at the mount point and do a find for something. When a hang was produced,
> you had to kill autofs, unmount by hand, maybe killing a few programs in
> order not to get "busy" messages. Then just mount it again and it's OK.
> Note that neither killing and restarting the server nor rebooting the
> client seemed necessary. You just had to unmount and remount.

You probably had a inode namespace collision. unfsd encodes the dev_t in the 
upper in ext2 unused bits of the inode to be able to export multiple file systems in 
a single export (knfsd dropped that broken feature) That sometimes fails, yielding
duplicate inode numbers. When the collision happens for objects with different
types (e.g. directory and file) then the client detects it immediately when 
the other object is still in its inode cache and marks the inode bad, giving
these EIOs. Newer unfsd has various tuning options to avoid these collisions,
but it does not always work. It depends on the actual device numbers and inode
used so could suddenly appear on a system update.

> 
> > (when you ignore locking and NFS over TCP ATM) All you need is to forward
> > UDP packets. This can either be done by a normal user space daemon that
> > uses portmap and just sends the packets out again, or alternatively by using
> > UDP masquerading and appropiate routes on the internal boxes.
> > Only forwarding packets is very different from full reexport support in knfsd
> > and much simpler.
> 
> Well, I hope some solution is found, since this is an important feature.
> It would be nicer in knfsd, I think, but if that proves unpractical for
> some reason, your packet filter/forwarder might just be the answer.

It is impractical in knfsd for NFS for most other network file systems (especially
those that have "artificial" inode numbers like smbfs). knfsd over FAT or amiga ffs
is probably a problem too. 

Your problem case really sounds like you just need a proxy.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Jorge L. deLyra]

> You probably had a inode namespace collision. unfsd encodes the dev_t in
> the upper in ext2 unused bits of the inode to be able to export multiple
> file systems in a single export (knfsd dropped that broken feature)

Well, not completely. There is now this option "nohide" on /etc/exports
which does this. Another useful thing we use a lot here. With knfsd you
still have to authorize (export) separately each local mount point (unfsd
used to automatically authorize all the underlying tree), but with this
option you can mount just once and go down into directories which are on
different filesystems on the server. A very useful feature, specially
while autofs is not able to do cascaded mounts. We use this for some
sysadmin tasks on clients, which export their root to their server.

I recognize the need for a robust NFS structure on Linux and understand
that some restrictions on features might be needed for that. But all these
features are quite nice and useful, so I hope ways are found to implement
them in robust ways.
							Regards,

----------------------------------------------------------------
        Jorge L. deLyra,  Associate Professor of Physics
            The University of Sao Paulo,  IFUSP-DFMA
       For more information: finger delyra@latt.if.usp.br
----------------------------------------------------------------

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Jorge L. deLyra]

> From cks@utcc.utoronto.ca Sat Jan  6 16:36:57 2001
> Date: Sat, 6 Jan 2001 00:52:21 -0500
> From: Chris Siebenmann <cks@utcc.utoronto.ca>
> To: "Jorge L. deLyra" <delyra@latt.if.usp.br>
> Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share
> 
> You write:
> [about re-exporting NFS mounts via NFS:]
> | Well, I hope some solution is found, since this is an important feature.
> | It would be nicer in knfsd, I think, but if that proves unpractical for
> | some reason, your packet filter/forwarder might just be the answer.
> 
>  This is essentially impossible to do reliably; one way or another you
> have to make tradeoffs. The basic problem is that of NFS filehandles.
> NFS filehandles are fixed-size persisent opaque objects that uniquely
> identify a given file to the server. Now, imagine that you are
> attempting to re-export NFS partitions (from multiple systems) as well
> as export some of your own local partitions. How do you come up with the
> NFS filehandles that you're going to give clients, in general, and keep
> the result persistent?
> 
>  You have to somehow add information (what server/filesystem the object
> is on/for) to an existing, opaque block of information of the same size,
> all of which may be significant, and that you can't make any assumptions
> about the format of. There's no place for the information to go, unless
> you create a potentially unboundedly large mapping table and store it
> locally, or unless your code is only really going to work with known
> sources of filehandles so you know how they're put together and thus
> where you can stuff some extra information in.
> 
> (unfsd's NFS filehandles are derived from file name and file location,
> if I remember correctly, and can erroneously become invalid if the file
> is moved around in the wrong way. That's the other way out, but it is
> not 100% reliable, and if what someone wants to do with the server hits
> this case hard you are truly out of luck.)

OK, I think I got your point, we seem to be dealing with a limitation of
the NFS protocol. However, Andi Kleen has this idea of a packet filtering
and relay daemon. Since the packets of the private network cannot appear
on the Internet, I guess there would have to be some kind of tunneling
involved. Well, trying to elaborate on that, couldn't you solve the
file-handling problem by building a local cache on disk in the front-end?
That would reduce the re-exportation problem to the problem of exporting a
local ext2 filesystem. I imagine something like this: let's say that

	latt.int is one home server on the Internet
	dfma.int is another home server on the Internet
	pmcs.int is the front-end, which is on both networks
	pmcs.prv is the name of the front end in the private network
	node.prv is any compute node in the private network

latt.int:/home <---> mounted as /latt/home on pmcs.int
dfma.int:/home <---> mounted as /dfma/home on pmcs.int
                                     ^
                                     |
   re-export daemon: files requested | get cached into
                                     |
                                     V
                                   /rexp on pmcs
                (say, with paths /rexp/latt/home/..., etc)
                                     ^
                                     |
                      pmcs.prv:/rexp +---> mounted as /rexp on node.prv

The re-export daemon on pmcs would have to keep in /rexp the files which
are required via the mount of /rexp by the nodes, copying them from the
appropriate Internet mount, and copying back into these mounts new files
which show up in /rexp/. Would something like this be feasible?

							Cheers,

----------------------------------------------------------------
        Jorge L. deLyra,  Associate Professor of Physics
            The University of Sao Paulo,  IFUSP-DFMA
       For more information: finger delyra@latt.if.usp.br
----------------------------------------------------------------


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Bugs in knfsd -- Problem re-exporting an NFS share

[Frank.Olsen]

Hi -- could you please CC me if you reply to this mail.

My problem is that I get an error when setting up the following
configuration:

A:     /exports/A                                 - Redhat 7.0
B1/B2: mount /exports/A on /export/A from A       - Redhat 6.2
C:     mount /exports/A on /mnt/A from B1 or B2   - Redhat 6.2

I use knfsd/nfs-utils on each machine.

bash# ls /mnt/A
/mnt/A/A.txt: No such file or directory


I searched for a while on deja.com, and there seemed to be some indications
that knfsd was bugged and that using the user-mode code would work.
However, no one replied specifically to my message, so I'm still not sure.

BTW, what I tried to do was to set up a HA configuration of machines B1/B2
using A as a "shared disk".
This is just to try out the HA software without buying more hardware.

Thanks in advance for any help!

Best regards,
Frank Olsen

PS Happy new year!


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Bugs in knfsd -- Problem re-exporting an NFS share

[Neil Brown]

On Friday December 29, Frank.Olsen@stonesoft.com wrote:
> Hi -- could you please CC me if you reply to this mail.
> 
> My problem is that I get an error when setting up the following
> configuration:
> 
> A:     /exports/A                                 - Redhat 7.0
> B1/B2: mount /exports/A on /export/A from A       - Redhat 6.2
> C:     mount /exports/A on /mnt/A from B1 or B2   - Redhat 6.2
> 
> I use knfsd/nfs-utils on each machine.
> 
> bash# ls /mnt/A
> /mnt/A/A.txt: No such file or directory
> 

This is not a supported configuration.  You cannot export NFS mounted
filesystems with NFS. The protocol does not cope, and it
implementation doesn't even try.
NFS is for export local filesystems only.

> 
> I searched for a while on deja.com, and there seemed to be some indications
> that knfsd was bugged and that using the user-mode code would work.
> However, no one replied specifically to my message, so I'm still not sure.
> 
> BTW, what I tried to do was to set up a HA configuration of machines B1/B2
> using A as a "shared disk".
> This is just to try out the HA software without buying more
> hardware.

Try "nbd" the network block device.  That should be able to give a
more realistic imitation of a share disk.

NeilBrown


> 
> Thanks in advance for any help!
> 
> Best regards,
> Frank Olsen
> 
> PS Happy new year!
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/