[PATCH] fuse: add hang check in request_wait_answer()

[PATCH] fuse: add hang check in request_wait_answer()

[Zhang Tianci]

If the FUSEDaemon fails to respond to FUSE requests
due to certain reasons (e.g., deadlock), the kernel
can detect this situation and issue an alert via logging.
Based on monitoring of such alerts in the kernel logs,
we can configure hang event alerts for large-scale deployed
FUSEDaemon clusters.

Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
Signed-off-by: Zhang Tianci <zhangtianci.1997@bytedance.com>
---
 fs/fuse/dev.c | 46 ++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 8 deletions(-)

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 6d59cbc877c6a..7b3d4160647a2 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -30,6 +30,9 @@
 MODULE_ALIAS_MISCDEV(FUSE_MINOR);
 MODULE_ALIAS("devname:fuse");
 
+static unsigned long hang_complain_secs = 60;
+module_param(hang_complain_secs, ulong, 0644);
+
 static struct kmem_cache *fuse_req_cachep;
 
 const unsigned long fuse_timeout_timer_freq =
@@ -545,14 +548,24 @@ static void request_wait_answer(struct fuse_req *req)
 {
 	struct fuse_conn *fc = req->fm->fc;
 	struct fuse_iqueue *fiq = &fc->iq;
+	unsigned int hang_check_time = 0;
 	int err;
 
 	if (!fc->no_interrupt) {
-		/* Any signal may interrupt this */
-		err = wait_event_interruptible(req->waitq,
-					test_bit(FR_FINISHED, &req->flags));
-		if (!err)
-			return;
+		while (true) {
+			/* Any signal may interrupt this */
+			err = wait_event_interruptible_timeout(
+				req->waitq, test_bit(FR_FINISHED, &req->flags),
+				READ_ONCE(hang_complain_secs) * HZ);
+			if (err > 0)
+				goto out;
+			if (err == -ERESTARTSYS)
+				break;
+			if (hang_check_time++ == 0) {
+				pr_debug("fuse conn %u req %llu (opcode %u) may hang.\n",
+					fc->dev, req->in.h.unique, req->args->opcode);
+			}
+		}
 
 		set_bit(FR_INTERRUPTED, &req->flags);
 		/* matches barrier in fuse_dev_do_read() */
@@ -568,21 +581,38 @@ static void request_wait_answer(struct fuse_req *req)
 		err = wait_event_killable(req->waitq,
 					test_bit(FR_FINISHED, &req->flags));
 		if (!err)
-			return;
+			goto out;
 
 		if (test_bit(FR_URING, &req->flags))
 			removed = fuse_uring_remove_pending_req(req);
 		else
 			removed = fuse_remove_pending_req(req, &fiq->lock);
 		if (removed)
-			return;
+			goto out;
 	}
 
 	/*
 	 * Either request is already in userspace, or it was forced.
 	 * Wait it out.
 	 */
-	wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags));
+	while (true) {
+		err = wait_event_timeout(req->waitq, test_bit(FR_FINISHED, &req->flags),
+				 READ_ONCE(hang_complain_secs) * HZ);
+		if (err > 0)
+			goto out;
+		if (err == -ERESTARTSYS)
+			break;
+		if (hang_check_time++ == 0) {
+			pr_debug("fuse conn %u req %llu (opcode %u) may hang.\n",
+				fc->dev, req->in.h.unique, req->args->opcode);
+		}
+	}
+out:
+	if (hang_check_time) {
+		pr_debug("fuse conn %u req %llu (opcode %u) recovery after %lu seconds\n",
+			fc->dev, req->in.h.unique, req->args->opcode,
+			hang_check_time * READ_ONCE(hang_complain_secs));
+	}
 }
 
 static void __fuse_request_send(struct fuse_req *req)
-- 
2.39.5

Re: [PATCH] fuse: add hang check in request_wait_answer()

[Joanne Koong]

On Mon, Dec 22, 2025 at 10:22 PM Zhang Tianci
<zhangtianci.1997@bytedance.com> wrote:
>
> If the FUSEDaemon fails to respond to FUSE requests
> due to certain reasons (e.g., deadlock), the kernel
> can detect this situation and issue an alert via logging.
> Based on monitoring of such alerts in the kernel logs,
> we can configure hang event alerts for large-scale deployed
> FUSEDaemon clusters.

Hi Zhang,

Does setting a timeout on fuse requests suffice for what you need?
That will detect deadlocks and abort the connection if so.

Thanks,
Joanne
>
> Signed-off-by: Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
> Signed-off-by: Zhang Tianci <zhangtianci.1997@bytedance.com>
> ---
>  fs/fuse/dev.c | 46 ++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 38 insertions(+), 8 deletions(-)
>

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Zhang Tianci]

Hi Joanne,

Thanks your reply,

> Does setting a timeout on fuse requests suffice for what you need?
> That will detect deadlocks and abort the connection if so.
>
The timeout mechanism does not quite meet our expectations,
because the hang state of the FUSEDaemon may recover in some ways.
We only expect an alert for the hang event, after which we can manually
handle the FUSEDaemon and attempt to recover it or abort the connection.

Thanks,
Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Joanne Koong]

On Tue, Dec 23, 2025 at 7:04 PM Zhang Tianci
<zhangtianci.1997@bytedance.com> wrote:
>
> Hi Joanne,
>
> Thanks your reply,
>
> > Does setting a timeout on fuse requests suffice for what you need?
> > That will detect deadlocks and abort the connection if so.
> >
> The timeout mechanism does not quite meet our expectations,
> because the hang state of the FUSEDaemon may recover in some ways.
> We only expect an alert for the hang event, after which we can manually
> handle the FUSEDaemon and attempt to recover it or abort the connection.
>

Hi Tianci,

That makes sense. In that case, I think it's cleaner to detect and
print the corresponding debug statements for this through libfuse
instead of the kernel.

Thanks,
Joanne

> Thanks,
> Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Zhang Tianci]

Hi Joanne, I apologize for the delayed response. Happy New Year!

> That makes sense. In that case, I think it's cleaner to detect and

> print the corresponding debug statements for this through libfuse

> instead of the kernel.

Yes, you're absolutely right. From the perspective of FUSEDaemon maintainers,
it is appropriate to place such checks in libfuse.

However, from the viewpoint of personnel responsible for maintaining
cluster kernel stability, they are more concerned with whether the
kernel itself is affected.

Additionally, if FUSEDaemon enters a state where it neither responds to kernel
FUSE requests nor can exit during the process exit phase due to certain bugs
(such as when FUSEDaemon incorrectly depends on the mount point it
provides and thus enters a deadlock state), the alerts in libfuse will
become ineffective.

Therefore, I think there is no need for an either-or choice between kernel
alerts and FUSEDaemon alerts.

Thanks,
Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Joanne Koong]

On Sat, Jan 3, 2026 at 11:22 PM Zhang Tianci
<zhangtianci.1997@bytedance.com> wrote:
>
> Hi Joanne, I apologize for the delayed response. Happy New Year!
Hi Tianci,

No worries at all, happy 2026!
>
> > That makes sense. In that case, I think it's cleaner to detect and
>
> > print the corresponding debug statements for this through libfuse
>
> > instead of the kernel.
>
> Yes, you're absolutely right. From the perspective of FUSEDaemon maintainers,
> it is appropriate to place such checks in libfuse.
>
> However, from the viewpoint of personnel responsible for maintaining
> cluster kernel stability, they are more concerned with whether the
> kernel itself is affected.
>
> Additionally, if FUSEDaemon enters a state where it neither responds to kernel
> FUSE requests nor can exit during the process exit phase due to certain bugs
> (such as when FUSEDaemon incorrectly depends on the mount point it
> provides and thus enters a deadlock state), the alerts in libfuse will
> become ineffective.

imo it's possible to check whether the kernel itself is affected just
purely through libfuse changes to fuse_lowlevel.c where the request
communication with the kernel happens. The number of requests ready by
the kernel is exposed to userspace through sysfs, so if the daemon is
deadlocked or cannot read fuse requests, that scenario is detectable
by userspace.

Thanks,
Joanne

>
> Therefore, I think there is no need for an either-or choice between kernel
> alerts and FUSEDaemon alerts.
>
> Thanks,
> Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Zhang Tianci]

Hi Joanne，

> imo it's possible to check whether the kernel itself is affected just
> purely through libfuse changes to fuse_lowlevel.c where the request
> communication with the kernel happens. The number of requests ready by
> the kernel is exposed to userspace through sysfs, so if the daemon is
> deadlocked or cannot read fuse requests, that scenario is detectable
> by userspace.

Yes, checking in libfuse/fuse_lowlevel.c is feasible, but it depends on
the running state of FUSEDaemon(if FUSEDaemon is in a process exit state,
this check cannot be performed), I think we do need this approach,
but it cannot fully cover all scenarios. Therefore, I believe it
should coexist with this patch.

The content of the /sys/fs/fuse/connections/${devid}/waiting interface
is inaccurate;
it cannot distinguish between normal waiting and requests that have been hanging
for a period of time.

Thanks,
Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Joanne Koong]

On Tue, Jan 6, 2026 at 6:43 PM Zhang Tianci
<zhangtianci.1997@bytedance.com> wrote:
>
> Hi Joanne，
>
> > imo it's possible to check whether the kernel itself is affected just
> > purely through libfuse changes to fuse_lowlevel.c where the request
> > communication with the kernel happens. The number of requests ready by
> > the kernel is exposed to userspace through sysfs, so if the daemon is
> > deadlocked or cannot read fuse requests, that scenario is detectable
> > by userspace.

Hi Tianci,

>
> Yes, checking in libfuse/fuse_lowlevel.c is feasible, but it depends on
> the running state of FUSEDaemon(if FUSEDaemon is in a process exit state,
> this check cannot be performed), I think we do need this approach,
> but it cannot fully cover all scenarios. Therefore, I believe it
> should coexist with this patch.
>
> The content of the /sys/fs/fuse/connections/${devid}/waiting interface
> is inaccurate;
> it cannot distinguish between normal waiting and requests that have been hanging
> for a period of time.

I think if the fusedaemon is in a process exit state (by "process exit
state", I think you're talking about the state where
fuse_session_exit() has been called but the daemon is stuck/hanging
before actual process exit?), this can still be detected in libfuse.
For example one idea could be libfuse spinning up a watchdog monitor
thread that has logic checking if the session's mt_exited has been set
with no progress on /sys/fs/fuse/.../waiting requests being fulfilled.

Thanks,
Joanne

>
> Thanks,
> Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Zhang Tianci]

Hi Joanne,

> I think if the fusedaemon is in a process exit state (by "process exit
> state", I think you're talking about the state where
> fuse_session_exit() has been called but the daemon is stuck/hanging
> before actual process exit?), this can still be detected in libfuse.
> For example one idea could be libfuse spinning up a watchdog monitor
> thread that has logic checking if the session's mt_exited has been set
> with no progress on /sys/fs/fuse/.../waiting requests being fulfilled.

The process exit state I referred to is a more severe scenario:
the FUSEDaemon may be killed abruptly due to bugs or OOM.
In such an unexpected exit, no userspace threads can run normally.
However, some threads may remain stuck in the kernel and fail to exit properly.

We have encountered at least two such cases:

1. The mount system call of the FUSEDaemon is blocked by other threads
    and cannot acquire the super_block_list lock.(Our FUSEDaemon supports
    multiple mount points, so this mount operation will affect the
other mount points
    within the FUSEDaemon process.)
2. The jbd2 subsystem of the ext4 filesystem, which the FUSEDaemon
     logging system depends on, triggers a logical deadlock caused by
     priority inversion.

In these instances, a userspace watchdog would be ineffective.

Thanks,
Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Joanne Koong]

On Wed, Jan 7, 2026 at 6:25 PM Zhang Tianci
<zhangtianci.1997@bytedance.com> wrote:
>
> Hi Joanne,
>
> > I think if the fusedaemon is in a process exit state (by "process exit
> > state", I think you're talking about the state where
> > fuse_session_exit() has been called but the daemon is stuck/hanging
> > before actual process exit?), this can still be detected in libfuse.
> > For example one idea could be libfuse spinning up a watchdog monitor
> > thread that has logic checking if the session's mt_exited has been set
> > with no progress on /sys/fs/fuse/.../waiting requests being fulfilled.
>
> The process exit state I referred to is a more severe scenario:
> the FUSEDaemon may be killed abruptly due to bugs or OOM.
> In such an unexpected exit, no userspace threads can run normally.
> However, some threads may remain stuck in the kernel and fail to exit properly.

Hmm, doesn't the CONFIG_DETECT_HUNG_TASK config already detect this?
The summary of it [1] says:

"Say Y here to enable the kernel to detect "hung tasks",
which are bugs that cause the task to be stuck in
uninterruptible "D" state indefinitely.

When a hung task is detected, the kernel will print the
current stack trace (which you should report), but the
task will stay in uninterruptible state. If lockdep is
enabled then all held locks will also be reported. This
feature has negligible overhead."

[1] https://www.kernelconfig.io/config_detect_hung_task

Another idea maybe is having some sort of system script that runs post
daemon process exit that checks if there's still any lingering d-state
children threads hanging around.

Thanks,
Joanne

>
> We have encountered at least two such cases:
>
> 1. The mount system call of the FUSEDaemon is blocked by other threads
>     and cannot acquire the super_block_list lock.(Our FUSEDaemon supports
>     multiple mount points, so this mount operation will affect the
> other mount points
>     within the FUSEDaemon process.)
> 2. The jbd2 subsystem of the ext4 filesystem, which the FUSEDaemon
>      logging system depends on, triggers a logical deadlock caused by
>      priority inversion.
>
> In these instances, a userspace watchdog would be ineffective.
>
> Thanks,
> Tianci

Re: [External] Re: [PATCH] fuse: add hang check in request_wait_answer()

[Bernd Schubert]

Hi Joanne,

On 1/8/26 01:00, Joanne Koong wrote:
> On Tue, Jan 6, 2026 at 6:43 PM Zhang Tianci
> <zhangtianci.1997@bytedance.com> wrote:
>>
>> Hi Joanne，
>>
>>> imo it's possible to check whether the kernel itself is affected just
>>> purely through libfuse changes to fuse_lowlevel.c where the request
>>> communication with the kernel happens. The number of requests ready by
>>> the kernel is exposed to userspace through sysfs, so if the daemon is
>>> deadlocked or cannot read fuse requests, that scenario is detectable
>>> by userspace.
> 
> Hi Tianci,
> 
>>
>> Yes, checking in libfuse/fuse_lowlevel.c is feasible, but it depends on
>> the running state of FUSEDaemon(if FUSEDaemon is in a process exit state,
>> this check cannot be performed), I think we do need this approach,
>> but it cannot fully cover all scenarios. Therefore, I believe it
>> should coexist with this patch.
>>
>> The content of the /sys/fs/fuse/connections/${devid}/waiting interface
>> is inaccurate;
>> it cannot distinguish between normal waiting and requests that have been hanging
>> for a period of time.
> 
> I think if the fusedaemon is in a process exit state (by "process exit
> state", I think you're talking about the state where
> fuse_session_exit() has been called but the daemon is stuck/hanging
> before actual process exit?), this can still be detected in libfuse.
> For example one idea could be libfuse spinning up a watchdog monitor
> thread that has logic checking if the session's mt_exited has been set
> with no progress on /sys/fs/fuse/.../waiting requests being fulfilled.

I added one kernel connection watchdog last week

https://github.com/libfuse/libfuse/commit/6278995cca991978abd25ebb2c20ebd3fc9e8a13

Thanks,
Bernd