Re: udevd is killing file write performance.

[Andrew Morton <akpm@osdl.org>]

Robin Holt <holt@sgi.com> wrote:
>
> 
> I have a customer application which sees a significant performance
> degradation when run with udevd running.  This appears to be due to:
> 
> 
> void inotify_dentry_parent_queue_event(struct dentry *dentry, u32 mask,
>                                        u32 cookie, const char *name)
> {
> ...
>         if (!atomic_read (&inotify_watches))
>                 return;
> 
>         spin_lock(&dentry->d_lock);
> 
> The particular application is a 512 thread application.  When run with
> udevd turned off the application finishes in about 6 seconds.  With it
> turned on, the appliction takes minutes (I think we timed it to around
> 22 minutes, but we have not been patient enough to wait it through to
> the end in some time).  This is being run on a 512cpu system, but there
> is a noticable performance hit on smaller systems as well.
> 
> As far as I can tell, the application has multiple threads writing
> different portions of the same file, but the customer is still working
> on providing a simplified test case.

Are you able to work out whether inotify_inode_watched(inode) is returning
true?

Probably it isn't, and everyone is hammering dentry->d_lock.  You'll
probably find that if all those processes were accessing the shared file
via separate filenames (all hardlinked to the same file), things would
improve.

I get a screwed feeling about this.  We have to take d_lock so we can get
at the parent dentry.  Otherwise we have obscure races with rename and
unlink.

> I know _VERY_ little about filesystems.  udevd appears to be looking
> at /etc/udev/rules.d.  This bumps inotify_watches to 1.  The file
> being written is on an xfs filesystem mounted at a different mountpoint.
> Could the inotify flag be moved from a global to a sb (or something
> finer) point and therefore avoid taking the dentry->d_lock when there
> is no possibility of a watch event being queued.

umm, yes, that's a bit of a palliative, but we could probably move
inotify_watches into dentry->d_inode->i_sb.