NFS bug?

NFS bug?

[Robert Merrill]

we have an SMP login server we just recently switched to debian
testing from FreeBSD and it's giving us a little trouble.

it mounts its /home on a seperate machine, which is still running BSD,
over a NIC-to-NIC 1000BASE-T link.

We've found the following bug exists in 2.6.15 and .16: If a directory
under /home is readable but not executable, a call to getdents64() on
it will kill the process with an invalid operand error in
__copy_from_user_ll

has this been fixed already, and is there a patch which is readily applicable?

we're not using the latest kernel, unfortunately, because it has lockd problems.

Re: NFS bug?

[Trond Myklebust]

On Thu, 2006-04-20 at 09:37 -0700, Robert Merrill wrote:
> we have an SMP login server we just recently switched to debian
> testing from FreeBSD and it's giving us a little trouble.
> 
> it mounts its /home on a seperate machine, which is still running BSD,
> over a NIC-to-NIC 1000BASE-T link.
> 
> We've found the following bug exists in 2.6.15 and .16: If a directory
> under /home is readable but not executable, a call to getdents64() on
> it will kill the process with an invalid operand error in
> __copy_from_user_ll

> has this been fixed already, and is there a patch which is readily applicable?

No idea. Can you supply us with a strace of the problem?

> we're not using the latest kernel, unfortunately, because it has lockd problems.

Care to elaborate?

Cheers,
  Trond

Re: NFS bug?

[Robert Merrill]

On 4/20/06, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> Given that you are reporting an error with copy_from_user, then it is
> _definitely_ of interest to figure out what your glibc is telling the
> kernel to copy.

execve("./a.out", ["./a.out", "foo"], [/* 16 vars */]) = 0
uname({sys="Linux", node="soda", ...})  = 0
brk(0)                                  = 0x804a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
old_mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7f5f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=20397, ...}) = 0
old_mmap(NULL, 20397, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f5a000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260O\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1270928, ...}) = 0
old_mmap(NULL, 1276892, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7e22000
old_mmap(0xb7f50000, 32768, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12e000) = 0xb7f50000
old_mmap(0xb7f58000, 7132, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7f58000
close(3)                                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb7e21000
mprotect(0xb7f50000, 20480, PROT_READ)  = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e218e0,
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0xb7f5a000, 20397)               = 0
open("foo", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0666, st_size=512, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
brk(0)                                  = 0x804a000
brk(0x806f000)                          = 0x806f000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 21), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7f5e000
write(1, "0\n", 20
)                      = 2
getdents64(3,  <unfinished ...>
+++ killed by SIGSEGV +++

Re: NFS bug?

[Trond Myklebust]

On Thu, 2006-04-20 at 11:02 -0700, Robert Merrill wrote:
> On 4/20/06, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > Given that you are reporting an error with copy_from_user, then it is
> > _definitely_ of interest to figure out what your glibc is telling the
> > kernel to copy.

Oh... and could you also send us the Oops/stack trace from the BUG_ON()?

Cheers,
  Trond

Re: NFS bug?

[Robert Merrill]

> Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
>
 ------------[ cut here ]------------
kernel BUG at arch/i386/lib/usercopy.c:582!
invalid operand: 0000 [#49]
SMP
Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl
sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev
uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy
usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
CPU:    2
EIP:    0060:[<c01ff157>]    Not tainted VLI
EFLAGS: 00010282   (2.6.15.7-soda0)
EIP is at __copy_from_user_ll+0x12/0xe2
eax: 00000000   ebx: 00000003   ecx: fffffffb   edx: fffffffb
esi: 0804a024   edi: 00000000   ebp: 00000000   esp: f6964f84
ds: 007b   es: 007b   ss: 0068
Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
       f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
       00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
Call Trace:
Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89
c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f>
0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Re: NFS bug?

[Trond Myklebust]

On Thu, 2006-04-20 at 11:14 -0700, Robert Merrill wrote:
> > Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
> >
>  ------------[ cut here ]------------
> kernel BUG at arch/i386/lib/usercopy.c:582!
> invalid operand: 0000 [#49]
> SMP
> Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
> i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl
> sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev
> uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy
> usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
> CPU:    2
> EIP:    0060:[<c01ff157>]    Not tainted VLI
> EFLAGS: 00010282   (2.6.15.7-soda0)
> EIP is at __copy_from_user_ll+0x12/0xe2
> eax: 00000000   ebx: 00000003   ecx: fffffffb   edx: fffffffb
> esi: 0804a024   edi: 00000000   ebp: 00000000   esp: f6964f84
> ds: 007b   es: 007b   ss: 0068
> Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
> Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
>        f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
>        00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
> Call Trace:
> Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89
> c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f>
> 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0

Was there no stack trace in that Oops? AFAICS, getdents64() isn't
supposed to be calling __copy_from_user_ll() at all, so you appear to
have something very weird going here.

Cheers,
  Trond

Re: NFS bug?

[Robert Merrill]

It's not an oops. It's a BUG_ON.

That was the entire message.

On 4/20/06, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> On Thu, 2006-04-20 at 11:14 -0700, Robert Merrill wrote:
> > > Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
> > >
> >  ------------[ cut here ]------------
> > kernel BUG at arch/i386/lib/usercopy.c:582!
> > invalid operand: 0000 [#49]
> > SMP
> > Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
> > i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl
> > sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev
> > uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy
> > usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
> > CPU:    2
> > EIP:    0060:[<c01ff157>]    Not tainted VLI
> > EFLAGS: 00010282   (2.6.15.7-soda0)
> > EIP is at __copy_from_user_ll+0x12/0xe2
> > eax: 00000000   ebx: 00000003   ecx: fffffffb   edx: fffffffb
> > esi: 0804a024   edi: 00000000   ebp: 00000000   esp: f6964f84
> > ds: 007b   es: 007b   ss: 0068
> > Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
> > Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb
> 000000d0
> >        f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003
> 0804a024
> >        00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b
> 000000dc
> > Call Trace:
> > Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89
> > c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f>
> > 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
>
> Was there no stack trace in that Oops? AFAICS, getdents64() isn't
> supposed to be calling __copy_from_user_ll() at all, so you appear to
> have something very weird going here.
>
> Cheers,
>   Trond
>
>

Re: NFS bug?

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Thu, 2006-04-20 at 11:14 -0700, Robert Merrill wrote:
>  > > Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
>  > >
>  >  ------------[ cut here ]------------
>  > kernel BUG at arch/i386/lib/usercopy.c:582!
>  > invalid operand: 0000 [#49]
>  > SMP
>  > Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
>  > i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl
>  > sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev
>  > uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy
>  > usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
>  > CPU:    2
>  > EIP:    0060:[<c01ff157>]    Not tainted VLI
>  > EFLAGS: 00010282   (2.6.15.7-soda0)
>  > EIP is at __copy_from_user_ll+0x12/0xe2
>  > eax: 00000000   ebx: 00000003   ecx: fffffffb   edx: fffffffb
>  > esi: 0804a024   edi: 00000000   ebp: 00000000   esp: f6964f84
>  > ds: 007b   es: 007b   ss: 0068
>  > Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
>  > Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
>  >        f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
>  >        00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
>  > Call Trace:
>  > Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89
>  > c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f>
>  > 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
> 
>  Was there no stack trace in that Oops? AFAICS, getdents64() isn't
>  supposed to be calling __copy_from_user_ll() at all, so you appear to
>  have something very weird going here.

I'd be guessing that filldir64() was passed a negative namlen.

Perhaps Robert could test this:

--- devel/fs/readdir.c~a	2006-04-21 00:54:33.000000000 -0700
+++ devel-akpm/fs/readdir.c	2006-04-21 00:54:58.000000000 -0700
@@ -231,6 +231,10 @@ static int filldir64(void * __buf, const
 	buf->error = -EINVAL;	/* only used if we fail.. */
 	if (reclen > buf->count)
 		return -EINVAL;
+	if (namlen < 0) {
+		dump_stack();
+		return -EINVAL;
+	}
 	dirent = buf->previous;
 	if (dirent) {
 		if (__put_user(offset, &dirent->d_off))
_

Re: NFS bug?

[Trond Myklebust]

On Fri, 2006-04-21 at 00:55 -0700, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > On Thu, 2006-04-20 at 11:14 -0700, Robert Merrill wrote:
> >  > > Oh... and could you also send us the Oops/stack trace from the BUG_ON()?
> >  > >
> >  >  ------------[ cut here ]------------
> >  > kernel BUG at arch/i386/lib/usercopy.c:582!
> >  > invalid operand: 0000 [#49]
> >  > SMP
> >  > Modules linked in: w83627hf eeprom lm85 w83781d hwmon_vid i2c_isa
> >  > i2c_dev thermal fan button processor ac battery nfs lockd nfs_acl
> >  > sunrpc ipv6 quota_v1 ide_cd cdrom generic joydev piix psmouse evdev
> >  > uhci_hcd ehci_hcd parport_pc parport e1000 rtc serio_raw floppy
> >  > usbcore i2c_i801 ide_core i2c_core mousedev pcspkr shpchp pci_hotplug
> >  > CPU:    2
> >  > EIP:    0060:[<c01ff157>]    Not tainted VLI
> >  > EFLAGS: 00010282   (2.6.15.7-soda0)
> >  > EIP is at __copy_from_user_ll+0x12/0xe2
> >  > eax: 00000000   ebx: 00000003   ecx: fffffffb   edx: fffffffb
> >  > esi: 0804a024   edi: 00000000   ebp: 00000000   esp: f6964f84
> >  > ds: 007b   es: 007b   ss: 0068
> >  > Process a.out (pid: 6994, threadinfo=f6964000 task=f70e7030)
> >  > Stack: fffffffb b7f55ff4 f893c2a0 00000000 0804a024 fffffffb fffffffb 000000d0
> >  >        f70e7030 00000003 0804a024 b7f55ff4 f6964000 f893dc1d 00000003 0804a024
> >  >        00004000 0804a024 b7f55ff4 bf973d50 ffffffda 0000007b c010007b 000000dc
> >  > Call Trace:
> >  > Code: 07 29 c8 f3 a4 89 c1 c1 e9 02 83 e0 03 90 f3 a5 89 c1 f3 a4 89
> >  > c8 5e 5f c3 57 56 8b 7c 24 0c 8b 74 24 10 8b 4c 24 14 85 c9 79 08 <0f>
> >  > 0b 46 02 63 92 2f c0 83 f9 3f 0f 86 99 00 00 00 89 f8 31 f0
> > 
> >  Was there no stack trace in that Oops? AFAICS, getdents64() isn't
> >  supposed to be calling __copy_from_user_ll() at all, so you appear to
> >  have something very weird going here.
> 
> I'd be guessing that filldir64() was passed a negative namlen.

Why would that trigger a bug in __copy_from_user_ll()? I could see it
triggering errors in copy_to_user(), but not copy_from_*...

Cheers,
  Trond

Re: NFS bug?

[Joshua Hudson]

> > >  Was there no stack trace in that Oops? AFAICS, getdents64() isn't
> > >  supposed to be calling __copy_from_user_ll() at all, so you appear to
> > >  have something very weird going here.
> >
> > I'd be guessing that filldir64() was passed a negative namlen.
>
> Why would that trigger a bug in __copy_from_user_ll()? I could see it
> triggering errors in copy_to_user(), but not copy_from_*...
>
> Cheers,
>   Trond

I've made that mistake before. Passing copy_from_user a negative
length -> crash.

Re: NFS bug?

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> > I'd be guessing that filldir64() was passed a negative namlen.
> 
>  Why would that trigger a bug in __copy_from_user_ll()? I could see it
>  triggering errors in copy_to_user(), but not copy_from_*...

Ah.  No, I cannot see why getdents wold run copy_from_user().

I wonder why that stack trace didn't come out.  Perhaps running `dmesg -n
7' prior to triggerng the crash will help.  (It shouldn't, but we might
have broken it).

Re: NFS bug?

[Trond Myklebust]

On Fri, 2006-04-21 at 11:31 -0700, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > > I'd be guessing that filldir64() was passed a negative namlen.
> > 
> >  Why would that trigger a bug in __copy_from_user_ll()? I could see it
> >  triggering errors in copy_to_user(), but not copy_from_*...
> 
> Ah.  No, I cannot see why getdents wold run copy_from_user().
> 
> I wonder why that stack trace didn't come out.  Perhaps running `dmesg -n
> 7' prior to triggerng the crash will help.  (It shouldn't, but we might
> have broken it).
> 

Also, please check that the kernel was compiled with
CONFIG_FRAME_POINTER and CONFIG_KALLSYMS.

Cheers,
  Trond

Re: NFS bug?

[Robert Merrill]

> Also, please check that the kernel was compiled with
> CONFIG_FRAME_POINTER and CONFIG_KALLSYMS.

Don't have the former