From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8258E2853F2 for ; Fri, 10 Apr 2026 14:30:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775831443; cv=none; b=W82Q0IOlOV5WXObB1WfKxSY7QABqnIlbbwqTaA5fj5Ft3wKkrtIhZ3KqhbLUOR9BJADB7QngOK4eeYEAe/GuNBfZWUI2BVxqGfdRQHXp8mF5BVV5y7qqATl2Wmq6mij9M9Zo9nMkqOutivrs/5CrLjdgdOTM3Q21nfd7r0Tsjaw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775831443; c=relaxed/simple; bh=TQeeegfY0L21N3lRHnbJ5Dm4HiBHSorSFFIp2y8ypl4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=q26mTVF9xlee61UlyUfTdjYVEO0XWplBPnEOBbu9byPL0yuS6zkmaY5q6scel4M5LPmnamzeRhLITiuz41Q8T3wfjH5YTPHR/kxny1JmvThpq9qV6eeJ0b6bivk/xN6rL7LtDT6R/x1CWFqbK4Abs2i0qBlf9+Gm39PPyWNmLKo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hammerspace.com; spf=pass smtp.mailfrom=hammerspace.com; dkim=pass (2048-bit key) header.d=hammerspace.com header.i=@hammerspace.com header.b=REGQiAUN; arc=none smtp.client-ip=209.85.210.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hammerspace.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hammerspace.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hammerspace.com header.i=@hammerspace.com header.b="REGQiAUN" Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-7dbdcb85067so1760106a34.1 for ; Fri, 10 Apr 2026 07:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hammerspace.com; s=google; t=1775831440; x=1776436240; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mDf17iELweSQBEB0PxUVBr9/blquXg+klPEZm7g/By4=; b=REGQiAUNmCnvfYER6ocZ3QFDECMvr9DYAf9IAZHez4pCKVqKV/fHPxJ3CYfUZAVYaB z2YsdFk4byWS/ytmrKfihrk7wbl6TUkq7cCwLz/8ewvajN+R+GbYBSjOWsRm82iw2FZc AkDXArQ4yKvW3VbdTYP5o6hZJeiiJEfir1ZEkAoSP/rS5grn33ccwpaISHS6RYmGMowH O2LGH+804giKjuwmavH+hK01ntIGfXjpB165YTE7ucRqYc1ax0v6pIIk8hMaGLpQnB/E dNw2hw6kKTsj5hjNuE1VlsqPxoMskUiDxe6+WO5N20SgA2Sg/o4lpl1Wsqdgfz9Mgi1g Io6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775831440; x=1776436240; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=mDf17iELweSQBEB0PxUVBr9/blquXg+klPEZm7g/By4=; b=MOmDvv3ZVqgvdZdMzYb7fvyxDDJJaQqBOcm+Ei3y8tnzorkJIZ9nvLEu7no2J0R6NJ /IDShQ62KP0NTgzQXuUrQwZqDtGApMbyLzgTTOvY87jBYw+wLw+YBhdtjV/V8bOXcsYp bpEGR2tn+gvdcn9g2/aoMQsHm4W5UgtGmGNcFP3KncfRUF5AZ96nM7PMkiQZLiyHEDW8 mrpikwPgmwfbcaJiZg8uWbPX6Ajog20pV2uU807U5AehHxxVIoc5UVgt8MUqL61/OCjq J52ia26Ba6dgW5s9Ha7/7Mu3RoA3kyCYi1r9CAzO91k+mnVpdbo7vDYrmOK/TMpfNmUG ZX5A== X-Forwarded-Encrypted: i=1; AJvYcCWguwPfQaUE/eMvTBlAoEW9ozlO76d25Ro0jQlOfMsWaqFEnVpaz6eaia9UDNR852MTMe5MxoMjQsWbhh8=@vger.kernel.org X-Gm-Message-State: AOJu0YzjT1lUYcLwAvarlFA6TaoKIMQDCMnVEGUxwx1Qmt3qjvvqhFGN 35Q74RilXqtWyz5+Fr0kOq7FqblpbBI6H6Cn0dAdNefYiSwOOfQezClFScpE141Eb3w= X-Gm-Gg: AeBDietf/2CELHRnx+DBlL0sWsYSiemb7eSFGx+FcQA41Y0e4MyZcAd+AfpaAucgc9v DuX+hn8CsgkBPotqzQXdFj+nlJmnXl8ra2+CYrBJrrSEWhnwuJWImMlRmtD2xa1g7YP4x/TBZrc a0LOUnUYuYzIkVTa5lK9VHqCz2nzKru0R2S95PCucutIncOs2Ourwu0cWcDkC914FtohzDwA5jX FqHogJmOh4sEbAlhU27VA2QO8Uls/P+Fzb1Klk6c/8dClITRXHb/j81CwKctwE389bZVErH7bMa ZURUcon0KNogbd8dfE3ADdkIFdMMabce6g+j2Yv8xgjjb9ifKWxrXVx8jCVaGEnzBYn+oM2SQgB BK8rL26UGf+r7JxSYkSt+TnAOm9Jjb9ziNrSdIa6CMMcBminYX2+j8T5TplqZUP+1Tri/IHKQ27 FbfVxakaMkJiS8HoTkt0Nb/fxQ/ZgeR28PJ2qw1e8oJHo= X-Received: by 2002:a05:6820:750d:b0:67c:27a7:8c4b with SMTP id 006d021491bc7-68be8bde8f8mr1122751eaf.54.1775831440290; Fri, 10 Apr 2026 07:30:40 -0700 (PDT) Received: from [192.168.254.51] ([66.97.168.37]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-68bcacc9925sm1525379eaf.9.2026.04.10.07.30.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 07:30:39 -0700 (PDT) From: Benjamin Coddington X-Google-Original-From: Benjamin Coddington To: Ben Roberts Cc: Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] pNFS: deadlock in pnfs_send_layoutreturn Date: Fri, 10 Apr 2026 10:30:38 -0400 X-Mailer: MailMate (2.0r6272) Message-ID: In-Reply-To: <20260408122534.537816-1-ben.roberts@gsacapital.com> References: <20260408122534.537816-1-ben.roberts@gsacapital.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Hi Ben, Did you reproduce and diagnose this problem on a recent upstream kernel version? If not, you probably want to report the issue to your Red Hat support channel directly instead of sending fixes upstream because the maintainers will want to see patches and fixes against the upstream code.= The code in 5.14.0-611.9.1.el9 may have different behaviors. Ben On 8 Apr 2026, at 8:25, Ben Roberts wrote: > On a HPC cluster running 5.14.0-611.9.1.el9.x86_64, regular deadlocks w= ere > seen within pnfs_send_layoutreturn leading to userspace processes stuck= in > uninterruptible sleep, ultimately requiring reboots to clear. This was > occurring frequently, sometimes multiple times per day on specific host= s > with heavy load. Claude code was tasked with hunting down any potentia= l > deadlocks within pnfs_send_layoutreturn, and identified the following > condition. This patch has been running in production on top of the EL9 > kernel for over three months without any reoccurrence of the deadlock. > > The pnfs_send_layoutreturn() function can deadlock when memory > allocation fails. The issue occurs in the error path where > pnfs_put_layout_hdr() is called, which may trigger > pnfs_layoutreturn_before_put_layout_hdr(), potentially causing > a recursive call back to pnfs_send_layoutreturn(). > > Call chain that triggers the deadlock: > 1. pnfs_send_layoutreturn() - kzalloc() fails > 2. Error path calls pnfs_put_layout_hdr(lo) > 3. pnfs_put_layout_hdr() calls pnfs_layoutreturn_before_put_layout_hdr(= ) > 4. If NFS_LAYOUT_RETURN_REQUESTED is still set, attempts another > layoutreturn, creating recursion/deadlock > > The fix ensures that NFS_LAYOUT_RETURN_REQUESTED is cleared in the > allocation failure path before calling pnfs_put_layout_hdr(). This > prevents pnfs_layoutreturn_before_put_layout_hdr() from attempting > another layout return, breaking the recursion cycle. > > v2 fixes a syntax error introduced while composing the original mail. > > Signed-off-by: Ben Roberts > Assisted-by: Claude:claude-sonnet-4-5 > --- > fs/nfs/pnfs.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c > index bc13d1e69449..47bda53b2b3a 100644 > --- a/fs/nfs/pnfs.c > +++ b/fs/nfs/pnfs.c > @@ -1361,6 +1361,7 @@ pnfs_send_layoutreturn(struct pnfs_layout_hdr *lo= , > if (unlikely(lrp =3D=3D NULL)) { > status =3D -ENOMEM; > spin_lock(&ino->i_lock); > + pnfs_clear_layoutreturn_info(lo); > pnfs_clear_layoutreturn_waitbit(lo); > spin_unlock(&ino->i_lock); > put_cred(cred); > -- > 2.43.0 > > For details of how GSA uses your personal information, please see our P= rivacy Notice here: https://www.gsacapital.com/privacy-notice > > This email and any files transmitted with it contain confidential and p= roprietary information and is solely for the use of the intended recipien= t. > If you are not the intended recipient please return the email to the se= nder and delete it from your computer and you must not use, disclose, dis= tribute, copy, print or rely on this email or its contents. > This communication is for informational purposes only. > It is not intended as an offer or solicitation for the purchase or sale= of any financial instrument or as an official confirmation of any transa= ction. > Any comments or statements made herein do not necessarily reflect those= of GSA Capital. > GSA Capital Partners LLP is authorised and regulated by the Financial C= onduct Authority and is registered in England and Wales at Stratton House= , 5 Stratton Street, London W1J 8LA, number OC309261. > GSA Capital Services Limited is registered in England and Wales at the = same address, number 5320529.