From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A6F2CCA47F for ; Sat, 16 Jul 2022 01:27:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231187AbiGPB1k (ORCPT ); Fri, 15 Jul 2022 21:27:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59064 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230072AbiGPB1g (ORCPT ); Fri, 15 Jul 2022 21:27:36 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD7028AEDB for ; Fri, 15 Jul 2022 18:27:34 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-31dfe25bd47so18406817b3.18 for ; Fri, 15 Jul 2022 18:27:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=oNUmtpieKLTpKC9Phy1E+O2Qwr+LcgpjVlKPKz6ZsMY=; b=o3YYx9oRz4vs+i20GvTWtA2fo2ETFMeVSiAqrPAjMumbgk6QsUfn/L2857Y1U4kr91 vklLL5DSuzEDM920bdCqKAMwVm4TZrdIbmt15k1jgJIHxfR2M3E7zYEP8JAC7UfjbOPP BUfsSW6TIpP5Zd+0767BBU6jYGQIuKTH6NuHwbuI8Ok0AGwudcmQcNEJKE0uPakdLJha BSQHl/QlbT7Tx2Gt4EzHMIkcAgxAelFpUjH17w/t9BXELeqSi0tBgNQP9F0wr/k6HFmy H0kpyAk/8g5j0lUBzM1Q5DdfBFyv+9bdMt4D3AGxguc5NSFR9q3njhxZLt3co38DKu8I 5FYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=oNUmtpieKLTpKC9Phy1E+O2Qwr+LcgpjVlKPKz6ZsMY=; b=zSr3Wk7PXtElL0ZxU1mWwygqR0ZTpXj30wGchg/0FRRn593Y/0b28lvHXj3fTatP+T OYvyaQl13vaxLvzEYdyKRgt4QjZM2hO5UzCe0sQCgqcGCDMXBPNqX0YJvD9YT/VPARd6 1C2jeEeri+kxeWdd/5Kn2GA+GOU03vOnoGVjpphLDnEk4uAsL4aIvDaN7q8Pcv92m4OJ oa+SXhx/M0uTOY/xH+Ok37IH/rC12qxn7NiI9Tm036YoEeKyEPkLIzZgiV2VWjzDJxpo hIFvHQXKm6WvNWR0u+ioEkzE/c19ESPMhWnbf5bfkxZEs5diOz45zoOkEgIAjUdPSV/y Lbvw== X-Gm-Message-State: AJIora/n8r4Y3XOGncOkHY8ux3stfbJXWrSSiF93ohQwhpOvr1kn1e4Y I3vceCTr2/Jt9ZAu+akp7S8L6pA7kHxaEg== X-Google-Smtp-Source: AGRyM1tgce6Og74FBQ8kWDBrQFEYIJs1vYjRR78miz5377H8QoZuC/el3NaKqVJGVtRTP4sANPPWTE2wuKnhCw== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:28b]) (user=shakeelb job=sendgmr) by 2002:a25:e74d:0:b0:66e:5c8e:609 with SMTP id e74-20020a25e74d000000b0066e5c8e0609mr16394088ybh.585.1657934854024; Fri, 15 Jul 2022 18:27:34 -0700 (PDT) Date: Sat, 16 Jul 2022 01:27:31 +0000 In-Reply-To: Message-Id: <20220716012731.2zz7hpg3qbhwgeqd@google.com> Mime-Version: 1.0 References: <20220629165542.da7fc8a2a5dbd53cf99572aa@linux-foundation.org> <20220629192435.df27c0dbb07ef72165e1de5e@linux-foundation.org> Subject: Re: [RESEND RFC PATCH] epoll: autoremove wakers even more aggressively From: Shakeel Butt To: Andrew Morton Cc: Benjamin Segall , Alexander Viro , linux-fsdevel , LKML , Linus Torvalds , Eric Dumazet , Roman Penyaev , Jason Baron , Khazhismel Kumykov , Heiher Content-Type: text/plain; charset="us-ascii" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jun 30, 2022 at 07:59:05AM -0700, Shakeel Butt wrote: > On Wed, Jun 29, 2022 at 7:24 PM Andrew Morton wrote: > > > > On Wed, 29 Jun 2022 18:12:46 -0700 Shakeel Butt wrote: > > > > > On Wed, Jun 29, 2022 at 4:55 PM Andrew Morton wrote: > > > > > > > > On Wed, 15 Jun 2022 14:24:23 -0700 Benjamin Segall wrote: > > > > > > > > > If a process is killed or otherwise exits while having active network > > > > > connections and many threads waiting on epoll_wait, the threads will all > > > > > be woken immediately, but not removed from ep->wq. Then when network > > > > > traffic scans ep->wq in wake_up, every wakeup attempt will fail, and > > > > > will not remove the entries from the list. > > > > > > > > > > This means that the cost of the wakeup attempt is far higher than usual, > > > > > does not decrease, and this also competes with the dying threads trying > > > > > to actually make progress and remove themselves from the wq. > > > > > > > > > > Handle this by removing visited epoll wq entries unconditionally, rather > > > > > than only when the wakeup succeeds - the structure of ep_poll means that > > > > > the only potential loss is the timed_out->eavail heuristic, which now > > > > > can race and result in a redundant ep_send_events attempt. (But only > > > > > when incoming data and a timeout actually race, not on every timeout) > > > > > > > > > > > > > Thanks. I added people from 412895f03cbf96 ("epoll: atomically remove > > > > wait entry on wake up") to cc. Hopefully someone there can help review > > > > and maybe test this. > > > > > > > > > > > > > > Thanks Andrew. Just wanted to add that we are seeing this issue in > > > production with real workloads and it has caused hard lockups. > > > Particularly network heavy workloads with a lot of threads in > > > epoll_wait() can easily trigger this issue if they get killed > > > (oom-killed in our case). > > > > Hard lockups are undesirable. Is a cc:stable justified here? > > Not for now as I don't know if we can blame a patch which might be the > source of this behavior. I am able to repro the epoll hard lockup on next-20220715 with Ben's patch reverted. The repro is a simple TCP server and tens of clients communicating over loopback. Though to cause the hard lockup I have to create a couple thousand threads in epoll_wait() in server and also reduce the kernel.watchdog_thresh. With Ben's patch the repro does not cause the hard lockup even with kernel.watchdog.thresh=1. Please add: Tested-by: Shakeel Butt