mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Trond Myklebust <trond.myklebust@fys.uio.no>
To: Pavel Machek <pavel@ucw.cz>
Cc: Lukas Hejtmanek <xhejtman@ics.muni.cz>,
	linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-fsdevel@vger.kernel.org, salvet@ics.muni.cz
Subject: Re: Deadlock in NFSv4 in all kernels
Date: Tue, 25 May 2010 08:28:40 -0400	[thread overview]
Message-ID: <1274790520.2949.20.camel@heimdal.trondhjem.org> (raw)
In-Reply-To: <20100524212422.GA1286@ucw.cz>

On Mon, 2010-05-24 at 23:24 +0200, Pavel Machek wrote: 
> Hi!
> 
> > I encountered the following problem. We use short expiration time for
> > kerberos contexts created by rpc.gssd (some patches were included in mainline
> > nfs-utils). In particular, we use 120secs expiration time.
> > 
> > Now, I run application that eats 80% of available RAM. Then I run 10 parallel
> > dd processes that write data into NFS4 volume with sec=krb5.
> > 
> > As soon as the kerberos context expires (i.e., up to 120 secs), the whole
> > system gets stuck in do_page_fault and succesive functions. It is because
> > there is no free memory in kernel, all free memory is used as cache for NFS4
> > (due to dd traffic), kernel ask NFS to write back its pages but NFS cannot do
> > anything as it is missing valid context. NFS contacts rpc.gssd to provide
> > a renewed context, the rpc.gssd does not provide the context as it needs some memory
> > to scan /tmp for a ticket. I.e., it deadlocks.
> > 
> > Longer context expiration time is no real solution as it only makes the
> > deadlock less often. 
> > 
> > Any ideas what can be done here? (Please cc me.) We could preallocate some
> > memory in rpc.gssd and use mlockall but not sure whether this proctects also
> > kernel malloc for things related to rpc.gssd and context creation (new file
> > descriptors and so on). 
> > 
> > This is seen in 2.6.32 kernel but most probably this is related to all kernel
> > versions.
> 
> Seems like pretty fundamental problem in nfs :-(. Limiting writeback
> caches for nfs, so that system has enough memory to perform rpc calls
> with the rest might do the trick, but...
> 

It's the same problem that you have for any file or storage system that
has initiators in userland. On the storage side, iSCSI in particular has
the same problem. On the filesystem side, CIFS, AFS, coda, .... do too.
The clustered filesystems can deadlock if the node that is running the
DLM runs out of memory...

A few years ago there were several people proposing various solutions
for allowing these daemons to run in a protected memory environment to
avoid deadlocks, but those efforts have since petered out. Perhaps it is
time to review the problem?

Cheers
  Trond


  reply	other threads:[~2010-05-25 12:28 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-05-07 15:39 Lukas Hejtmanek
2010-05-24 21:24 ` Pavel Machek
2010-05-25 12:28   ` Trond Myklebust [this message]
2010-05-25 12:58     ` Lukas Hejtmanek
2010-05-25 13:39       ` Trond Myklebust
2010-05-25 14:07         ` Zdenek Salvet
2010-05-25 17:10     ` Sunil Mushran
2010-05-25 13:45 ` William A. (Andy) Adamson
2010-05-25 14:02   ` Lukas Hejtmanek
2010-05-25 14:10     ` William A. (Andy) Adamson
2010-05-25 14:29       ` Trond Myklebust
2010-05-25 14:04   ` Trond Myklebust

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1274790520.2949.20.camel@heimdal.trondhjem.org \
    --to=trond.myklebust@fys.uio.no \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=pavel@ucw.cz \
    --cc=salvet@ics.muni.cz \
    --cc=xhejtman@ics.muni.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®