From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 884DC2FFF95 for ; Wed, 10 Dec 2025 05:45:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765345538; cv=none; b=PKInNzLAnNWMnesO+NHPv58TbIEAw7z3NHdwpfSUcBxA8mVfWPz3tIcuIafN7BojKvrb79hLpBmzZPyu+hTf27IufYazRMt9AdOSb6rTaf6Rqtp0Yl2VM7DaIpF2RoMRSZwk3DdbhqAM5RHO7VrW51jyixAFAb31fUsOROSOuKA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765345538; c=relaxed/simple; bh=w6iiCVQLpFikPUe4uciKj0mkXr2KE4ORLhI9r5rmOzE=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=ImUmbc9FtEiwaAu0P+lCkFZ/QbWT2l+wEL7r/KcZl+2czbo9rDk1HWpQvVLapfVlMO8PlRFVFzpyOV2jV6A8Iv6R7r2wPNDpmvS274BF363BB513CUj4Z75dTBfWCdcbwhSInVbRFJ2GzaOKyKF90HOrewRuMPa/tlMwki/uLtc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=aMpzgr5J; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=uNyuxbrF; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aMpzgr5J"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="uNyuxbrF" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1765345535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U5u9zjHxp5nOkcmQAgH5/2pjjFKz+7xciix+1rrgeQw=; b=aMpzgr5JEKi+SSFpRO+X3Nb39rOBFKDCPtX8Tvez79DzxXSE7FIsam/z9B5/p8aKUNnNVJ LLG/Jj3l60DCKnyRhW2V77IXD/WdpmEc9G7LYjI6lTEZFEQ4LNG3iZmbs0kS8hAhTrO4B7 iPe4v75wBvNoyFpUDNBAxhRzTFpNnls= Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-jk2RFXctOYiPxjcL5P6F-w-1; Wed, 10 Dec 2025 00:45:33 -0500 X-MC-Unique: jk2RFXctOYiPxjcL5P6F-w-1 X-Mimecast-MFC-AGG-ID: jk2RFXctOYiPxjcL5P6F-w_1765345533 Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-343daf0f38aso7987670a91.3 for ; Tue, 09 Dec 2025 21:45:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1765345533; x=1765950333; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=U5u9zjHxp5nOkcmQAgH5/2pjjFKz+7xciix+1rrgeQw=; b=uNyuxbrFUwznq8pXheaKyaiB4Kcldfm3hLR4OEcsN19qwml4AQuRsc8fJfe2qhX9GA Qa80B1L7G8eheVQePKNTWZgEtMoGc2WOw8O/Mzcg9zdeg90253SJjo7r1FL/YNxpb2ll 8pOwppG62ajqpOgANgjsW3I0+tIVTfa8DSuc4AxyhfgnL683xDF6hMFLILmtOsUj40YV Nm0EP0ZWR2FaZ83f/yPb3hlVxYcj5zVy52NxNGvVsojAZu8W1gumrm+Jrr/9Jhqhl2IP kqe5DI+QuFtaOJcatG5SB1JaH9cW080S6xVnBTeIdb7x7y/Vxcc+8oLOQlKCs8Ix4hF/ CHUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765345533; x=1765950333; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=U5u9zjHxp5nOkcmQAgH5/2pjjFKz+7xciix+1rrgeQw=; b=H/hs2MuX7DahHHqv5QUtK/Q3RhsJ7tAVLCe4JdpHs1JovC/RSVH0agyaYtrNNrrJLl EkHOcH5f4bYQ/PA/uHCiv4VhDXGptNqqcCb1WVmvDvJvcLmZjxatt/qoBajkAmCPQBIe +LXsEuPAsZlhgUFmKqI2Q6mLwWhvQadQTWf12nYVLQNO3q/erK9JSwpWP5CikFyfUN0X t7g+g0Ge0YtC++q26fr8Mt9v2ebLaSJ0jImTev1d5qc9kCPZJ1KfLBgvdxE/86fW/rlo xMTkZy2Fna7w2IrqUyK+RybXVGQxqQv9rd6WZB3CFMxw4hqTBnLQEqkx1avKoyjLpzoL 1L5A== X-Forwarded-Encrypted: i=1; AJvYcCWBlc2eNMQMVjxHxTltnIoXi4Lkmt5DrRThhK/gXrAiZsmOPeFq8oU/BRTubluDC78zkfedsN3friA/K0w=@vger.kernel.org X-Gm-Message-State: AOJu0YwzQww1Tdq3KSvBrA7+XlGlulNqfEzECZSNTeVZLVfJ1iH+n3AC swQgNP3Z0OHzJQCm7iuRDValUluDrCyMRo9+lLP/zzdXMmS/1w/fAFMIMkWlm9lN19TPHKJ/UGg IkWVBD8ALx5P8QVxeBDZ8RDdlX7IkHA0aipy1fABhot95q+cT+75bSAKPWbNj/mRe4tR0JX150g stBtxZLZ5UPvTxBDl/xoyNPgPaZiKlIsRWYbTCI+M5 X-Gm-Gg: AY/fxX5V3OHNsQj0rX5PytlBRj5bnhqJ9BLoN/1hSefbP1mUvmarLpXvYttmoXzebvR b/hq6rbl4gZtgyzT7MA4tmr167GXX0U8/E/+LcmH4KnIFPttXlpIs9i9gEGEo+9DGtB7oacrHFG i3l75QUXOhUGNMV/mpHN/zd38/e04/Fls/7M/GynB4Ln/Vv/Hn5uD741u7AoHGwWq4Bk4= X-Received: by 2002:a17:90b:3847:b0:340:a5b2:c305 with SMTP id 98e67ed59e1d1-34a72808f67mr1088292a91.2.1765345532539; Tue, 09 Dec 2025 21:45:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IGh6+dEVRAMV2C5vYHDGZU4Qf85RNHec+uPwoESl74tc15tqIpJSGl+Ri/dVxp9XR0wrBpc75hUH8L0OPBAOfQ= X-Received: by 2002:a17:90b:3847:b0:340:a5b2:c305 with SMTP id 98e67ed59e1d1-34a72808f67mr1088273a91.2.1765345532051; Tue, 09 Dec 2025 21:45:32 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20251208153419.18196-1-minhquangbui99@gmail.com> <66d9f44c-295e-4b62-86ae-a0aff5f062bb@gmail.com> In-Reply-To: <66d9f44c-295e-4b62-86ae-a0aff5f062bb@gmail.com> From: Jason Wang Date: Wed, 10 Dec 2025 13:45:19 +0800 X-Gm-Features: AQt7F2rVVyUHdSbcdKdzi21zc9e3z98C7ZeLVe5qQ4lJ-Awfq6TRhCbVeKZ8w04 Message-ID: Subject: Re: [PATCH net] virtio-net: enable all napis before scheduling refill work To: Bui Quang Minh Cc: netdev@vger.kernel.org, "Michael S. Tsirkin" , Xuan Zhuo , =?UTF-8?Q?Eugenio_P=C3=A9rez?= , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Stanislav Fomichev , virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Dec 9, 2025 at 11:23=E2=80=AFPM Bui Quang Minh wrote: > > On 12/9/25 11:30, Jason Wang wrote: > > On Mon, Dec 8, 2025 at 11:35=E2=80=AFPM Bui Quang Minh wrote: > >> Calling napi_disable() on an already disabled napi can cause the > >> deadlock. In commit 4bc12818b363 ("virtio-net: disable delayed refill > >> when pausing rx"), to avoid the deadlock, when pausing the RX in > >> virtnet_rx_pause[_all](), we disable and cancel the delayed refill wor= k. > >> However, in the virtnet_rx_resume_all(), we enable the delayed refill > >> work too early before enabling all the receive queue napis. > >> > >> The deadlock can be reproduced by running > >> selftests/drivers/net/hw/xsk_reconfig.py with multiqueue virtio-net > >> device and inserting a cond_resched() inside the for loop in > >> virtnet_rx_resume_all() to increase the success rate. Because the work= er > >> processing the delayed refilled work runs on the same CPU as > >> virtnet_rx_resume_all(), a reschedule is needed to cause the deadlock. > >> In real scenario, the contention on netdev_lock can cause the > >> reschedule. > >> > >> This fixes the deadlock by ensuring all receive queue's napis are > >> enabled before we enable the delayed refill work in > >> virtnet_rx_resume_all() and virtnet_open(). > >> > >> Fixes: 4bc12818b363 ("virtio-net: disable delayed refill when pausing = rx") > >> Reported-by: Paolo Abeni > >> Closes: https://netdev-ctrl.bots.linux.dev/logs/vmksft/drv-hw-dbg/resu= lts/400961/3-xdp-py/stderr > >> Signed-off-by: Bui Quang Minh > >> --- > >> drivers/net/virtio_net.c | 59 +++++++++++++++++++-------------------= -- > >> 1 file changed, 28 insertions(+), 31 deletions(-) > >> > >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > >> index 8e04adb57f52..f2b1ea65767d 100644 > >> --- a/drivers/net/virtio_net.c > >> +++ b/drivers/net/virtio_net.c > >> @@ -2858,6 +2858,20 @@ static bool try_fill_recv(struct virtnet_info *= vi, struct receive_queue *rq, > >> return err !=3D -ENOMEM; > >> } > >> > >> +static void virtnet_rx_refill_all(struct virtnet_info *vi) > >> +{ > >> + bool schedule_refill =3D false; > >> + int i; > >> + > >> + enable_delayed_refill(vi); > > This seems to be still racy? > > > > For example, in virtnet_open() we had: > > > > static int virtnet_open(struct net_device *dev) > > { > > struct virtnet_info *vi =3D netdev_priv(dev); > > int i, err; > > > > for (i =3D 0; i < vi->max_queue_pairs; i++) { > > err =3D virtnet_enable_queue_pair(vi, i); > > if (err < 0) > > goto err_enable_qp; > > } > > > > virtnet_rx_refill_all(vi); > > > > So NAPI and refill work is enabled in this case, so the refill work > > could be scheduled and run at the same time? > > Yes, that's what we expect. We must ensure that refill work is scheduled > only when all NAPIs are enabled. The deadlock happens when refill work > is scheduled but there are still disabled RX NAPIs. Just to make sure we are on the same page, I meant, after refill work is enabled, rq0 is NAPI is enabled, in this case the refill work could be triggered by the rq0's NAPI so we may end up in the refill work that it tries to disable rq1's NAPI while holding the netdev lock. Thanks > > Thanks, > Quang Minh. >