* [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
@ 2026-11-11 21:02 Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Mani Tofigh via B4 Relay @ 2026-11-11 21:02 UTC (permalink / raw)
To: Christian Brauner, Alexander Viro
Cc: Jan Kara, Oleg Nesterov, linux-fsdevel, linux-kernel, tz2294,
stable, Mani Tofigh
From: Mani Tofigh <mt4022@columbia.edu>
A task blocked in read() on a pipe can sleep while the pipe holds data,
when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
reader, so a write into an empty pipe wakes only the watcher. The
watcher reads without sleeping in anon_pipe_read() and doesn't pass the
wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
wake_up(rd_wait) for EPOLLET consumers") the next writes into the
non-empty pipe don't wake anyone anymore:
Task E Task A Task C
epoll_wait(ep, -1)
/* EPOLLEXCLUSIVE */
read(p[0])
write(p[1], "1")
/* wakes E only */
write(p[1], "2")
/* no wakeup */
read(p[0], 1 byte)
exit()
/* still asleep, "2" is in the pipe */
A then sleeps until the next write into an empty pipe, if ever. Before
that commit any pipe_poll() enabled the per-write wakeups. Those
reached A because an EPOLLEXCLUSIVE entry only takes a wakeup while a
task sleeps in epoll_wait().
So EPOLLEXCLUSIVE consumers depend on the per-write wakeups too, for a
different reason than EPOLLET. Enable them for EPOLLEXCLUSIVE as well
and say why in the comment. With the fix, A reads the second byte.
This bug was identified and confirmed with the help of a deterministic
hypervisor currently under development. The C reproducer below however
hits it within about 2s.
Fixes: f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers")
Cc: stable@vger.kernel.org
Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
---
Reproducer: https://github.com/manitofigh/files/blob/main/linux-repros/pipeexcl.c
There are a couple of older gaps like this (a single write, and the
write side with EPOLLOUT|EPOLLEXCLUSIVE). I left both alone here though.
---
fs/pipe.c | 6 +++++-
include/linux/pipe_fs_i.h | 2 +-
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/pipe.c b/fs/pipe.c
index 4c1122597a3c..8ad792d300c9 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
* and the ->pseudo_edgetrigger check in anon_pipe_write().
* Currently io_uring sets EPOLLET for multishot polls, so it gets the
* same behaviour.
+ *
+ * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
+ * ->rd_wait that can take the wakeup meant for a blocked reader, and
+ * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
*/
if ((filp->f_mode & FMODE_READ) &&
- wait && (wait->_key & EPOLLET) &&
+ wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
index 95f6236c5c5b..6b2e14ec6ef8 100644
--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -90,7 +90,7 @@ struct anon_pipe_prealloc {
* @files: number of struct file referring this pipe (protected by ->i_lock)
* @r_counter: reader counter
* @w_counter: writer counter
- * @pseudo_edgetrigger: has an EPOLLET consumer, enable per-write wakeups
+ * @pseudo_edgetrigger: enable per-write wakeups for EPOLLET/EPOLLEXCLUSIVE
* @fasync_readers: reader side fasync
* @fasync_writers: writer side fasync
* @bufs: the circular array of pipe buffers
---
base-commit: b0993210e420f2f1d61e6719fbcd3966b96ff9ed
change-id: 20261111-pipe-epollexclusive-wakeup-981528041b4d
Best regards,
--
Mani Tofigh <mt4022@columbia.edu>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
@ 2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 10:26 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
2 siblings, 1 reply; 5+ messages in thread
From: Oleg Nesterov @ 2026-11-11 23:34 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> From: Mani Tofigh <mt4022@columbia.edu>
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher.
I am already sleeping, but at first glance this is already wrong?
I mean, a WQ_FLAG_EXCLUSIVE waiter should not sit ahead of non-exclusive
waiter, no?
Oleg.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 23:34 ` Oleg Nesterov
@ 2026-11-12 10:26 ` Oleg Nesterov
0 siblings, 0 replies; 5+ messages in thread
From: Oleg Nesterov @ 2026-11-12 10:26 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
On 11/12, Oleg Nesterov wrote:
>
> On 11/11, Mani Tofigh via B4 Relay wrote:
> >
> > From: Mani Tofigh <mt4022@columbia.edu>
> >
> > A task blocked in read() on a pipe can sleep while the pipe holds data,
> > when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> > it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> > reader, so a write into an empty pipe wakes only the watcher.
>
> I am already sleeping, but at first glance this is already wrong?
>
> I mean, a WQ_FLAG_EXCLUSIVE waiter should not sit ahead of non-exclusive
> waiter, no?
Ah, I forgot that anon_pipe_read() uses wait_event_exclusive(), and it
assumes that this wait-event is the only reason to set wake_next_reader...
I seem to understand, let me think about it.
Oleg.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
@ 2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
2 siblings, 0 replies; 5+ messages in thread
From: Oleg Nesterov @ 2026-11-12 12:56 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay
Cc: Christian Brauner, Alexander Viro, Jan Kara, linux-fsdevel,
linux-kernel, tz2294, stable, Mani Tofigh
This probably needs more discussion... I will write another email.
But so far I think we don't really care.
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher. The
> watcher reads without sleeping in anon_pipe_read() and doesn't pass the
> wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
> wake_up(rd_wait) for EPOLLET consumers") the next writes into the
> non-empty pipe don't wake anyone anymore:
>
> Task E Task A Task C
> epoll_wait(ep, -1)
> /* EPOLLEXCLUSIVE */
> read(p[0])
> write(p[1], "1")
> /* wakes E only */
> write(p[1], "2")
> /* no wakeup */
> read(p[0], 1 byte)
> exit()
> /* still asleep, "2" is in the pipe */
Yes, in this case f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait)
for EPOLLET consumers") makes a difference.
However, this trivial test-case
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#include <assert.h>
#include <stdio.h>
int main(void)
{
int pfd[2], efd;
struct epoll_event evt = { .events = EPOLLIN | EPOLLEXCLUSIVE };
char c;
pipe(pfd);
efd = epoll_create1(0);
epoll_ctl(efd, EPOLL_CTL_ADD, pfd[0], &evt);
if (!fork()) {
epoll_wait(efd, &evt, 1, -1);
read(pfd[0], &c, 1);
printf("1st: %c\n", c);
return 0;
}
sleep(1);
if (!fork()) {
read(pfd[0], &c, 1);
printf("2nd: %c\n", c);
return 0;
}
sleep(1);
write(pfd[1], "12", 2);
assert(wait(NULL) > 0);
assert(wait(NULL) > 0);
return 0;
}
hangs even with this patch
+++ b/fs/pipe.c
@@ -749,6 +749,7 @@ pipe_poll(struct file *filp, poll_table *wait)
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
+WRITE_ONCE(pipe->pseudo_edgetrigger, true);
/*
* Reading pipe state only -- no need for acquiring the semaphore.
*
which reverts f1cebdec9739.
The 2nd child sleeps in read(pfd[0]) forever even if the pipe is not empty.
So I am not sure we want to enforce another oddity in pipe_poll/read ...
Oleg.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
@ 2026-11-17 13:27 ` Oleg Nesterov
2 siblings, 0 replies; 5+ messages in thread
From: Oleg Nesterov @ 2026-11-17 13:27 UTC (permalink / raw)
To: Mani Tofigh via B4 Relay, Breno Leitao, Christian Brauner,
Linus Torvalds, Mateusz Guzik
Cc: Alexander Viro, Jan Kara, linux-fsdevel, linux-kernel, tz2294,
stable, Mani Tofigh
Mani reports the case when f1cebdec9739 ("pipe: only enable the extra
wake_up(rd_wait) for EPOLLET consumers") makes a difference, see his
email below.
But lets forget about that commit for the moment. With or without that
commit this trivial test-case hangs,
#include <unistd.h>
#include <sys/epoll.h>
#include <sys/wait.h>
#include <assert.h>
#include <stdio.h>
int main(void)
{
int pfd[2], efd;
struct epoll_event evt = { .events = EPOLLIN | EPOLLEXCLUSIVE };
char c;
pipe(pfd);
efd = epoll_create1(0);
epoll_ctl(efd, EPOLL_CTL_ADD, pfd[0], &evt);
if (!fork()) {
epoll_wait(efd, &evt, 1, -1);
read(pfd[0], &c, 1);
printf("1st: %c\n", c);
return 0;
}
sleep(1);
if (!fork()) {
read(pfd[0], &c, 1);
printf("2nd: %c\n", c);
return 0;
}
sleep(1);
write(pfd[1], "12", 2);
assert(wait(NULL) > 0);
assert(wait(NULL) > 0);
return 0;
}
the 2nd non-epoll child sleeps in read(pfd[0]) even if pipe is not empty.
anon_pipe_read() assumes that all exclusive waiters on pipe->rd_wait come
from anon_pipe_read()->wait_event_interruptible_exclusive(), and the woken
reader will set wake_next_reader = true.
However, EPOLLEXCLUSIVE breaks this logic. ep_ptable_queue_proc() does
add_wait_queue_exclusive() if EPOLLEXCLUSIVE is set. So write(pfd[1]) above
wakes up the 1st "epoll" child, which then reads one byte from the pipe without
sleeping, so it doesn't set wake_next_reader and thus it doesn't wakeup the
2nd waiter.
Again, this happens with or without f1cebdec9739. But that commit makes a
difference: the 2nd child will continue to sleep even after another write
to this pipe. So Mani suggests the trivial change which should restore the
previous behaviour
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
* and the ->pseudo_edgetrigger check in anon_pipe_write().
* Currently io_uring sets EPOLLET for multishot polls, so it gets the
* same behaviour.
+ *
+ * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
+ * ->rd_wait that can take the wakeup meant for a blocked reader, and
+ * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
*/
if ((filp->f_mode & FMODE_READ) &&
- wait && (wait->_key & EPOLLET) &&
+ wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
WRITE_ONCE(pipe->pseudo_edgetrigger, true);
But I can't decide if we actually want to bless another historical
"nonsensical" wakeup until have a user application that actually cares?
Again, the test-case above will hang with or without the patch above, with
or without f1cebdec9739. So do we really care? If yes, then perhaps we should
also change anon_pipe_read(), something like
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -468,6 +468,8 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
}
if (pipe_is_empty(pipe))
wake_next_reader = false;
+ else if (pipe->pseudo_edgetrigger) // for EPOLLEXCLUSIVE
+ wake_next_reader = true;
/* Consumed buffers may have refilled the pool; trim it and unlock. */
anon_pipe_trim_and_unlock(pipe);
?
(of course, this won't help if the 1st child doesn't do read() after epoll_wait()).
What do you think?
Oleg.
On 11/11, Mani Tofigh via B4 Relay wrote:
>
> From: Mani Tofigh <mt4022@columbia.edu>
>
> A task blocked in read() on a pipe can sleep while the pipe holds data,
> when an epoll(EPOLLEXCLUSIVE) watcher of the read end was added before
> it. The watcher's entry on ->rd_wait is exclusive and sits ahead of the
> reader, so a write into an empty pipe wakes only the watcher. The
> watcher reads without sleeping in anon_pipe_read() and doesn't pass the
> wakeup on. Since commit f1cebdec9739 ("pipe: only enable the extra
> wake_up(rd_wait) for EPOLLET consumers") the next writes into the
> non-empty pipe don't wake anyone anymore:
>
> Task E Task A Task C
> epoll_wait(ep, -1)
> /* EPOLLEXCLUSIVE */
> read(p[0])
> write(p[1], "1")
> /* wakes E only */
> write(p[1], "2")
> /* no wakeup */
> read(p[0], 1 byte)
> exit()
> /* still asleep, "2" is in the pipe */
>
> A then sleeps until the next write into an empty pipe, if ever. Before
> that commit any pipe_poll() enabled the per-write wakeups. Those
> reached A because an EPOLLEXCLUSIVE entry only takes a wakeup while a
> task sleeps in epoll_wait().
>
> So EPOLLEXCLUSIVE consumers depend on the per-write wakeups too, for a
> different reason than EPOLLET. Enable them for EPOLLEXCLUSIVE as well
> and say why in the comment. With the fix, A reads the second byte.
>
> This bug was identified and confirmed with the help of a deterministic
> hypervisor currently under development. The C reproducer below however
> hits it within about 2s.
>
> Fixes: f1cebdec9739 ("pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers")
> Cc: stable@vger.kernel.org
> Signed-off-by: Mani Tofigh <mt4022@columbia.edu>
> ---
> Reproducer: https://github.com/manitofigh/files/blob/main/linux-repros/pipeexcl.c
>
> There are a couple of older gaps like this (a single write, and the
> write side with EPOLLOUT|EPOLLEXCLUSIVE). I left both alone here though.
> ---
> fs/pipe.c | 6 +++++-
> include/linux/pipe_fs_i.h | 2 +-
> 2 files changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/fs/pipe.c b/fs/pipe.c
> index 4c1122597a3c..8ad792d300c9 100644
> --- a/fs/pipe.c
> +++ b/fs/pipe.c
> @@ -700,9 +700,13 @@ pipe_poll(struct file *filp, poll_table *wait)
> * and the ->pseudo_edgetrigger check in anon_pipe_write().
> * Currently io_uring sets EPOLLET for multishot polls, so it gets the
> * same behaviour.
> + *
> + * EPOLLEXCLUSIVE needs them too. Its entry is an exclusive waiter on
> + * ->rd_wait that can take the wakeup meant for a blocked reader, and
> + * unlike a woken reader in anon_pipe_read() it doesn't pass it on.
> */
> if ((filp->f_mode & FMODE_READ) &&
> - wait && (wait->_key & EPOLLET) &&
> + wait && (wait->_key & (EPOLLET | EPOLLEXCLUSIVE)) &&
> unlikely(!READ_ONCE(pipe->pseudo_edgetrigger)))
> WRITE_ONCE(pipe->pseudo_edgetrigger, true);
>
> diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h
> index 95f6236c5c5b..6b2e14ec6ef8 100644
> --- a/include/linux/pipe_fs_i.h
> +++ b/include/linux/pipe_fs_i.h
> @@ -90,7 +90,7 @@ struct anon_pipe_prealloc {
> * @files: number of struct file referring this pipe (protected by ->i_lock)
> * @r_counter: reader counter
> * @w_counter: writer counter
> - * @pseudo_edgetrigger: has an EPOLLET consumer, enable per-write wakeups
> + * @pseudo_edgetrigger: enable per-write wakeups for EPOLLET/EPOLLEXCLUSIVE
> * @fasync_readers: reader side fasync
> * @fasync_writers: writer side fasync
> * @bufs: the circular array of pipe buffers
>
> ---
> base-commit: b0993210e420f2f1d61e6719fbcd3966b96ff9ed
> change-id: 20261111-pipe-epollexclusive-wakeup-981528041b4d
>
> Best regards,
> --
> Mani Tofigh <mt4022@columbia.edu>
>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-17 13:27 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-11 21:02 [PATCH] pipe: keep per-write wakeups for EPOLLEXCLUSIVE consumers Mani Tofigh via B4 Relay
2026-11-11 23:34 ` Oleg Nesterov
2026-11-12 10:26 ` Oleg Nesterov
2026-11-12 12:56 ` Oleg Nesterov
2026-11-17 13:27 ` Oleg Nesterov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®