[PATCH] eventpoll: check the event mask before taking ep->lock

[PATCH] eventpoll: check the event mask before taking ep->lock

[Jiayuan Chen]

ep_poll_callback() takes ep->lock and only then checks whether the
event matches epi->event.events. But epi->event.events is not protected
by ep->lock, ep_modify() and the EPOLLONESHOT disarm update it under
ep->mtx.

So a wakeup for an event nobody asked for takes and releases ep->lock
for nothing. E.g. a UDP socket registered with EPOLLIN only gets an
EPOLLOUT wakeup for every freed tx skb.

Check the mask first and return early if the event is not wanted. The
comment in ep_modify() counts on ep_poll_callback() taking ep->lock, so
do smp_mb() and read the mask again before dropping the event. Wanted
events take the lock as before.

Benchmark, based on stress-epoll.c [1]: N threads share one epoll, each
thread registers its own eventfd with EPOLLPRI, which eventfd never
raises, and loops on write() + read(). Both call ep_poll_callback(),
with EPOLLIN and EPOLLOUT. 12 vCPU VM, ops/ms:

  threads   before    after
        1      396      402
        4     1514     1604
       12     1264     4817

The original stress-epoll.c hits this path too: the poller read()s the
eventfds, and eventfd_read() wakes with EPOLLOUT while they are
registered with EPOLLIN. 32 vCPU VM, events/ms, average of 6 runs:

  threads   before    after
        8     1293     1315
       16     1407     1422
       32     1304     1431

Real case, udp rr over veth with 1000 connected sockets registered with
EPOLLIN: _raw_spin_lock_irqsave drops by ~1%.

[1] https://github.com/rouming/test-tools/blob/master/stress-epoll.c

Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---
 fs/eventpoll.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index e0c4bf88a838..29e034e3a295 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1495,8 +1495,22 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
 	struct eventpoll *ep = epi->ep;
 	__poll_t pollflags = key_to_poll(key);
 	unsigned long flags;
+	__poll_t events;
 	int ewake = 0;
 
+	/*
+	 * epi->event.events is updated under ep->mtx, not ep->lock, so an
+	 * event nobody asked for can be dropped without taking ep->lock.
+	 */
+	events = READ_ONCE(epi->event.events);
+	if (pollflags && !(pollflags & (events | POLLFREE))) {
+		/* Pairs with smp_mb() in ep_modify(). */
+		smp_mb();
+		events = READ_ONCE(epi->event.events);
+		if (!(pollflags & events))
+			return !(events & EPOLLEXCLUSIVE);
+	}
+
 	spin_lock_irqsave(&ep->lock, flags);
 
 	ep_set_busy_poll_napi_id(epi);
-- 
2.43.0