From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF8B31E2834 for ; Fri, 22 May 2026 06:46:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779432365; cv=none; b=uX37pDUVqk2icFBh/YEiRPG9zGwDhm01iJiS72pmxGSTGJ4/n4XMWUO0MPfvqSY279guwNs3dCTsXQ/cOuAYQoeDhwgJcMclIzPa3aNVwxoFsy8zg+lftRT4qRZnu1y8UbXlyQYB/wh5d0JMHBA3uc8KfeDZLyatX77yLQIxlZI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779432365; c=relaxed/simple; bh=OuypiTPvo6MlH6Aw9Q/Fs2RcYOmyy1Gt284dvTUNXFg=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=j/37IRlGr+03A6dVyma818YknRznjs7X3sRWn2tGSNJdipfsnf28tgKCxpCpWWVyt7jdwazueMtj2SvTJqrXfjznqgcFghaAT1p0JMO0Rt6ZLObUXA4lavyCBWR8mrkjIKTTUMT46vhSuNF/yMGWX53rhvtWVzZV0NLQP9ugZto= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TczTQqNi; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=nRL2YKnR; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TczTQqNi"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="nRL2YKnR" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1779432362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=OuypiTPvo6MlH6Aw9Q/Fs2RcYOmyy1Gt284dvTUNXFg=; b=TczTQqNi1LRfmo6XW11RRRnaN+MuzyHvc4JKxLqxjUZfnA7WmO+5N/7lHPdGDNFrMiJUlt PK2WLTiwu39kydzUC/99E+1petZt4nEgnQC2xak7FG58zW6VO6ohZIqEy0uy0TE08KSBQS Af3+YE9nTfTVQGxGbVZoG3h+agOXmag= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-377--ne_lKehNlOTeLGmft8ZWQ-1; Fri, 22 May 2026 02:46:01 -0400 X-MC-Unique: -ne_lKehNlOTeLGmft8ZWQ-1 X-Mimecast-MFC-AGG-ID: -ne_lKehNlOTeLGmft8ZWQ_1779432360 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-49045243094so4869685e9.2 for ; Thu, 21 May 2026 23:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1779432360; x=1780037160; darn=vger.kernel.org; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id:from:to :cc:subject:date:message-id:reply-to; bh=OuypiTPvo6MlH6Aw9Q/Fs2RcYOmyy1Gt284dvTUNXFg=; b=nRL2YKnREMk8x5yPVl6OJz4guKwoPo+WB9bMcWTGJGJEayypZojx1CN8W0qJf4tHHr Ep2cfF4x2G1anmJ9muFktEqO2/zSLZ0yhkzeHdi2zqgBJRih/25uDSpsztB9NXKDMh49 tDvi2lDWWkuLytiJVuVwOa/PieKOvV1zDy6wJOGyVapw0Ve0YAqNI/dapoHOBE+jcS3+ aRbDw24cOqSxlDkzDJWap38CrLyUksFO3O6dxjgYhCNtFoGEHHWe4IZZlOXu3+xzJliA 5m1J7DyYF+tSpAK12vs+UrFoFNt62/OxPa0rL3fuyyefRlWpiSN+RaZjV+S/CCk2kfkt J7JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779432360; x=1780037160; h=mime-version:user-agent:content-transfer-encoding:autocrypt :references:in-reply-to:date:cc:to:from:subject:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OuypiTPvo6MlH6Aw9Q/Fs2RcYOmyy1Gt284dvTUNXFg=; b=hLVuVbVpwn0ZqTR3lOWVACGPPTcNB/b5vk6FXGk04jrY8kIXCyd54Z9uKw9raPTpAb 1iz8Om9y1JmkW9pAWxELnSLRoueAR37yLBdju2DuL/w8tQ1WfUpAWJOi8MR4f9bT3I/Z rai9u0HEdrDZpdDKQeG2h/whQT6dimodWbY4c2OgAoQn7+jeXylpbv9uQS9yf9O17FeE rauQG7Lt0OkkahMfg7vEmFe+/C+J5f5Nlmg+tBx+1iBSzHzH7z9BgrwtTjhbsg6BmZsv rhRZIUAplQSFhr2mgZeOU1lXr1QkKSuW+Ef1yJBsCyTqrFDmj1GDdqWdxXspMEdd+GV6 w73Q== X-Gm-Message-State: AOJu0YyZPO98KQYzJglnJf5W3YkKOz6ZyIJ3FBuCoOtmhweP7UpvXIhi POy7lA0gEhKVHrv1fZfJENKjDcCmdG77kryUS75qJLBKuHd8tNWHWJJW7Wb8N+KZ/SLAaqQJJEq 5v0ILN9mz8fLHZbKbeGKYC8UO3J3NfuGIoFWmBbJcTQNikr3O6RccYmrFzEjQoaG937uYWO+hma zF X-Gm-Gg: Acq92OHxgDd624MiFAg28vcerc4IImLoZ3aIT2qbXIl8T1XXtfxVCgXDNg4RNpiERRZ zl0XYy5+mBY7PrakgRfstdl5KE7i/h1sE8oC0pzBAa5oZ9Vb+A/TijLnorBG7K3W8deUWMQ9EWe d2BiBohve/4HzAKwa4r/nV1h6VmnEB/91FrJXP+qmEeuCHCPmt2rzt6XfdSymuxZ5Um3iZKr/4b +ce680q3cHjHA3chUvTAKIii7humrE1TYnS+fa1YXRW5cta1q84r04HPwJJlg9p48ddiZUy1tOI lm/hLmE/QqhWYG2v7TMhCXE0axLxro/EICggRbzkv7YO2KV3Yq0AQ1ABdT4Zqchtex8kk+L6yKS rETj0b8mrCPAU3Vzcdw7JtxNSTxcIjFEH1zJE4MsXhN3oc8AJXy6sU0T0iiSzj9L30WPvtpw8k7 0K6mIrj1DD0xHgNtI= X-Received: by 2002:a05:600c:1c0b:b0:488:b187:3c with SMTP id 5b1f17b1804b1-490426aa7acmr29440625e9.14.1779432359927; Thu, 21 May 2026 23:45:59 -0700 (PDT) X-Received: by 2002:a05:600c:1c0b:b0:488:b187:3c with SMTP id 5b1f17b1804b1-490426aa7acmr29440265e9.14.1779432359533; Thu, 21 May 2026 23:45:59 -0700 (PDT) Received: from gmonaco-thinkpadt14gen3.rmtit.csb (212-8-243-115.hosted-by-worldstream.net. [212.8.243.115]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490454ea134sm25438455e9.8.2026.05.21.23.45.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 23:45:59 -0700 (PDT) Message-ID: Subject: Re: [PATCH] Re: Re: [RFC PATCH v2 04/10] rv/da: add pre-allocated storage pool for per-object monitors From: Gabriele Monaco To: Wen Yang Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, rostedt@goodmis.org Date: Fri, 22 May 2026 08:45:57 +0200 In-Reply-To: <7917085b-fef1-4017-b289-eefe84332ff5@linux.dev> References: <668f83581c58644a84cab5e6736864a439bb8e28.camel@redhat.com> <20260515083002.106512-1-gmonaco@redhat.com> <6d2f3490-5e30-4966-a3cd-372a34e10ba2@linux.dev> <5183dc18d63b617ab0f19290e8a790fa6898f372.camel@redhat.com> <7917085b-fef1-4017-b289-eefe84332ff5@linux.dev> Autocrypt: addr=gmonaco@redhat.com; prefer-encrypt=mutual; keydata=mDMEZuK5YxYJKwYBBAHaRw8BAQdAmJ3dM9Sz6/Hodu33Qrf8QH2bNeNbOikqYtxWFLVm0 1a0JEdhYnJpZWxlIE1vbmFjbyA8Z21vbmFjb0BrZXJuZWwub3JnPoiZBBMWCgBBFiEEysoR+AuB3R Zwp6j270psSVh4TfIFAmjKX2MCGwMFCQWjmoAFCwkIBwICIgIGFQoJCAsCBBYCAwECHgcCF4AACgk Q70psSVh4TfIQuAD+JulczTN6l7oJjyroySU55Fbjdvo52xiYYlMjPG7dCTsBAMFI7dSL5zg98I+8 cXY1J7kyNsY6/dcipqBM4RMaxXsOtCRHYWJyaWVsZSBNb25hY28gPGdtb25hY29AcmVkaGF0LmNvb T6InAQTFgoARAIbAwUJBaOagAULCQgHAgIiAgYVCgkICwIEFgIDAQIeBwIXgBYhBMrKEfgLgd0WcK eo9u9KbElYeE3yBQJoymCyAhkBAAoJEO9KbElYeE3yjX4BAJ/ETNnlHn8OjZPT77xGmal9kbT1bC1 7DfrYVISWV2Y1AP9HdAMhWNAvtCtN2S1beYjNybuK6IzWYcFfeOV+OBWRDQ== Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.60.1 (3.60.1-1.fc44) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Hi Wen, On Fri, 2026-05-22 at 01:40 +0800, Wen Yang wrote: > Hi Gabriele,=20 >=20 > No specific reason for REL_SOFT =E2=80=94 not intentional, reverting to= =20 > REL_HARD. > =C2=A0 > Reproduced the stall on the same config (PREEMPT_RT +=20 > PROVE_LOCKING/PROVE_RCU).=20 >=20 > Root cause is a cleanup ordering bug in uprobe_detail_waiting.tc,=20 > unrelated to REL_SOFT/REL_HARD: > =C2=A0 >=20 > =C2=A0=C2=A0 # original cleanup =E2=80=94 wrong order=20 >=20 > =C2=A0=C2=A0 echo "-${UPROBE_TARGET}:${start_offset}" > "$TLOB_MONITOR"= =C2=A0 # (A)=20 >=20 > =C2=A0=C2=A0 kill "$hog_pid"=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= # (B)=20 >=20 > =C2=A0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (A)= =20 > triggers synchronize_srcu() in the kernel. But tlob_target is stuck=20 > mid-uprobe_notify_resume holding an SRCU read lock, preempted by the=20 > FIFO-99 hog -> so the reader never finishes and (B) is never reached.=20 > rcuc/0 (a kthread on PREEMPT_RT) is also starved by the hog -> RCU stall. great you found the issue and solution. Wonder why lockdep wasn't more informative, but probably the issue was so frequent to hog that too. > =C2=A0=C2=A0 Fix: kill the hog first:=20 > =C2=A0 >=20 > =C2=A0=C2=A0 kill "$hog_pid"; wait "$hog_pid"=20 >=20 > =C2=A0=C2=A0 echo "-${UPROBE_TARGET}:${start_offset}" > "$TLOB_MONITOR"= =20 >=20 >=20 > On the PREEMPT_RT it is more reliably triggered there because rcuc/0 > runs as a preemptible kthread rather than in softirq, making it easier > for the hog to monopolise the CPU long enough to hit the stall. >=20 > Thank you for the thorough review and valuable suggestions. We are=20 > working through all of them and running the full test suite. > We expect to post v3 within the next two days. Alright, sounds good. Thanks, Gabriele