Re: nfs_update_inode: inode X mode changed, Y to Z

[Yaroslav Halchenko <yoh@psychology.rutgers.edu>]

Dear Kernel People,

Problem persists and it seems that I am not alone who hits this problem:
http://lkml.org/lkml/2007/3/1/53

in my case it happens if there is a lot (>50) nfs clients which in a
fast pace create a small file, close it and then open it for reading.
Some times it fails saying that the file is a directory, some times it
just fails to open it for writing, etc

I would really appreciate any comments and/or ideas on how to
troubleshoot/eliminate this issue.

Cheers
Yarik


On Fri, 23 Feb 2007, Yaroslav Halchenko wrote:

> Dear Kernel People,

> I am running a tiny cluster (27 nodes). Setup is as follows:

> * NFS server:
>    kernel: vanilla 2.6.18.2 + areca drivers
>    mounted partition is XFS
>    nfs-kernel-server: 1.0.10-1~bpo.1 (from backports.org)

> * clients:
>    kernel: 2.6.17-2-amd64 (Debian stock from etch installed few months ago)

> Today under heavy load which consisted of the nodes manipulating tons of
> small files, I started to mention that software reported weird errors
> like:

> EOFError: EOF read where object expected
> mv: cannot stat <FILENAME> : No such file or directory
> etc

> Looking at the client's dmesg I found nasty:

> node17.ravana.rutgers.edu: Feb 23 13:32:06 node17 kernel: nfs_update_inode: inode 680223263 mode changed, 0042755 to 0100644
> node17.ravana.rutgers.edu: Feb 23 13:41:19 node17 kernel: nfs_update_inode: inode 681427742 mode changed, 0042755 to 0100644
> node22.ravana.rutgers.edu: Feb 22 22:58:15 node22 kernel: nfs_update_inode: inode 677306152 mode changed, 0042755 to 0100644
> node22.ravana.rutgers.edu: Feb 23 13:48:33 node22 kernel: nfs_update_inode: inode 681695507 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:31:01 node12 kernel: nfs_update_inode: inode 680150798 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:34:57 node12 kernel: nfs_update_inode: inode 680418141 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:37:01 node12 kernel: nfs_update_inode: inode 680637478 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:39:25 node12 kernel: nfs_update_inode: inode 681034087 mode changed, 0100644 to 0042755
> node12.ravana.rutgers.edu: Feb 23 13:40:06 node12 kernel: nfs_update_inode: inode 681225056 mode changed, 0042755 to 0100644
> node12.ravana.rutgers.edu: Feb 23 13:43:26 node12 kernel: nfs_update_inode: inode 681474682 mode changed, 0100644 to 0042755
> ......

> server logs didn't show any abnormal things... where should I look for the
> source of the problem? googling up seems to be of no interesting result...
> is there way to eliminate cause may be by tuning some performance
> parameters (ie sacrificing performance for stability)?

> Thanks everyone in advance for hints
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik