From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34FAC38F233 for ; Tue, 13 Jan 2026 12:18:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.47 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768306741; cv=none; b=EyxLZylrMliysXe1omVUDZ3asmKu5YW8PYE4kaNZ5a78sZgavGxspyHr/Oi9rZRwrObVn7uuBD2Rk5VLC6ydHZ434I1XN+/22J976nQmHnFt+7bMb9zr0tk2YvhRuO6jBsL6P440+zXbCs3bUdWK9mMgcjSXlbirgCD3NiE4Cek= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768306741; c=relaxed/simple; bh=ncgyEwPDNVPYcjMVwZ7h1A0seyYkP8iBA3xNasKKHGw=; h=From:Date:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CC/r1WgNoqtYSWRoEYSMo+tas78GH9SAudPrz6g2VmcLHXt8M9cNbThx8J4pLlcWixD7fQiXAM/KD0rp1oZyF7BXfi7urgYcepYzjs7/Fogdlc3y6pSGU524W+oeVaJoIB9XH81Hn2RRBLE7iUYOD+egZ/zEoBaP41PQdMhLuN4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Lv9bCBfh; arc=none smtp.client-ip=209.85.167.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Lv9bCBfh" Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-59b855a46d2so2666979e87.2 for ; Tue, 13 Jan 2026 04:18:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1768306737; x=1768911537; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from:from:to :cc:subject:date:message-id:reply-to; bh=tI9yXGO6dzXhzNiYW61aS5a3OGj+G+TCRZHIgPPDUqY=; b=Lv9bCBfhQACMPEfs+kPZTn9I8cUG2T33k17kwqyLnkw6D6oEuMkajGB4llutfHI/rs JNMM3KyMsXqMmAToZ4d0HrOIyF0bAgHM2qTx3UJccfSMxq8WFgwKidp6oiXxcGBWvfws loFeSiLJdGVh/FrfoI8pQJCEkuT+Sjq2O+550MQQ8cWODx599xCosLrEg7fD0On3yj9Z laMlM9EoC1dZHgrQXAhCgcu/aBOH94/t6yA+dLdyYaDyHRoSatkKC73jawC8NYqBiAUs aODvPhxbTySJ3/RbBQN2cfER7YjGvDhKc5DZK6f6PpNLR5VuRZcqIINv2UXZp1RL7hhm OfcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1768306737; x=1768911537; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tI9yXGO6dzXhzNiYW61aS5a3OGj+G+TCRZHIgPPDUqY=; b=j+dXwxKe4MxQ5l+NfC/z2opJ+NtHvnGMzXMitUsayMTYmxUkiAxFdm1TAYgkJ/4a2J 6ON4oi+S394DLJ5hDmYtLJU6tzRQeMKqIlzyYE6zMX0kArrufd4KLYCAS+PByoeAAOOl dDki3KS1Ar7918Zl/Tn/5Ocxee45ovbcaLX9ZhxQNXCirXxPIf0kac9Nqc9yssQCPM2t Y9Wc/7oTCZckZSqtRHSDaLpahk4C9hC+fIaVJCBtDKre2+83gfPdbmWvexmPpHbbQMxc Q4SeA6WCkjczI6C1M9qf7Xmv5p7YsbCMYYif4R4kM4vkdSnL+VVr4esBe7Gag7ilFynt NRog== X-Forwarded-Encrypted: i=1; AJvYcCX/f2n3i95Tc42UFtqQWXcoF9egXvaQ+mtIZHA9aMGfKjfosqU3LgEO5Q/NTPs+KxSJUS4kvgZtGiofprw=@vger.kernel.org X-Gm-Message-State: AOJu0YzaF5Vdy/9ZKGdFz4AHnvwg9xPtDQCSEBFhOoH1FCsBjntfVARA k9pzojt9oZcOJIs1mSbtKmFdCV7V2jLFMyRc3O3CrvKekDqn0M6BVmcO X-Gm-Gg: AY/fxX7Puhe3YvyQOQW5nqmCqKxWu0PZIQTCYQl7czTHAhXDqg4lKCkJdr2ZcbJ63e/ NgnHacpkgH4WpcRTfVkHHKJoRhpLKpJEBW5Dkre3jWsk8j8Sl0ceDrzDYqI6d2rDqZ0QjXVpofL Bg4HWadKfD5jBGDC1wGRDERDv1AFSPUBqDYIEy+nfn6kFPmh17Kq0dGzJDiG/KxKGSGOz3rv7qE z/lLCzXDYB1xV38E2hwcS6LTWo079MU6QFvQMtj9Q8+tiYSvHGFwqSUvLS7y7bUhzpV7yoH7V2o 2n5UVgh1NbfsbUL3Iyk32ZHlgTAyUHVkQAhq9EgS0CHzsatAmQTvMRRQQd1e/fXSfIdo6pYzu/E G1fhnsUZa6ux96TUa2NNhV9lZjR/CE4bdgJ7itt4MGUlxdRX6o84Nvf9BDpPAglSn+Z7ORWl5N+ 7G55XOQbvevidHnZ1asZhDh8XegBpl98yzwf5r0w== X-Google-Smtp-Source: AGHT+IGVhARLqlfB4wu7e386cdEoE6kn/ckGH/qN3S6DPtqKvVRIt96GkWzgHeqUvfF5MEsai9hT6g== X-Received: by 2002:a05:6512:3c9f:b0:59b:7947:f420 with SMTP id 2adb3069b0e04-59b7947f58cmr5490713e87.18.1768306736568; Tue, 13 Jan 2026 04:18:56 -0800 (PST) Received: from pc636 (host-95-203-18-139.mobileonline.telia.com. [95.203.18.139]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-59b731aa442sm4524767e87.95.2026.01.13.04.18.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jan 2026 04:18:55 -0800 (PST) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Tue, 13 Jan 2026 13:18:53 +0100 To: Joel Fernandes Cc: Uladzislau Rezki , Shrikanth Hegde , Vishal Chourasia , "rcu@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "paulmck@kernel.org" , "frederic@kernel.org" , "neeraj.upadhyay@kernel.org" , "josh@joshtriplett.org" , "boqun.feng@gmail.com" , "rostedt@goodmis.org" , "tglx@linutronix.de" , "peterz@infradead.org" , "srikar@linux.ibm.com" Subject: Re: [PATCH] cpuhp: Expedite synchronize_rcu during CPU hotplug operations Message-ID: References: <20260112094332.66006-2-vishalc@linux.ibm.com> <5a2b00f2-5e73-4c89-89b5-1a69cb8a7fa2@linux.ibm.com> <91138C31-EF47-4CA6-BD9F-A41981F543EE@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Mon, Jan 12, 2026 at 05:36:24PM +0000, Joel Fernandes wrote: > > > > On Jan 12, 2026, at 12:09 PM, Uladzislau Rezki wrote: > > > > On Mon, Jan 12, 2026 at 04:09:49PM +0000, Joel Fernandes wrote: > >> > >> > >>>> On Jan 12, 2026, at 7:57 AM, Uladzislau Rezki wrote: > >>> > >>> Hello, Shrikanth! > >>> > >>>> > >>>>> On 1/12/26 3:38 PM, Uladzislau Rezki wrote: > >>>>> On Mon, Jan 12, 2026 at 03:13:33PM +0530, Vishal Chourasia wrote: > >>>>>> Bulk CPU hotplug operations—such as switching SMT modes across all > >>>>>> cores—require hotplugging multiple CPUs in rapid succession. On large > >>>>>> systems, this process takes significant time, increasing as the number > >>>>>> of CPUs grows, leading to substantial delays on high-core-count > >>>>>> machines. Analysis [1] reveals that the majority of this time is spent > >>>>>> waiting for synchronize_rcu(). > >>>>>> > >>>>>> Expedite synchronize_rcu() during the hotplug path to accelerate the > >>>>>> operation. Since CPU hotplug is a user-initiated administrative task, > >>>>>> it should complete as quickly as possible. > >>>>>> > >>>>>> Performance data on a PPC64 system with 400 CPUs: > >>>>>> > >>>>>> + ppc64_cpu --smt=1 (SMT8 to SMT1) > >>>>>> Before: real 1m14.792s > >>>>>> After: real 0m03.205s # ~23x improvement > >>>>>> > >>>>>> + ppc64_cpu --smt=8 (SMT1 to SMT8) > >>>>>> Before: real 2m27.695s > >>>>>> After: real 0m02.510s # ~58x improvement > >>>>>> > >>>>>> Above numbers were collected on Linux 6.19.0-rc4-00310-g755bc1335e3b > >>>>>> > >>>>>> [1] https://lore.kernel.org/all/5f2ab8a44d685701fe36cdaa8042a1aef215d10d.camel@linux.vnet.ibm.com > >>>>>> > >>>>> Also you can try: echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp > >>>>> to speedup regular synchronize_rcu() call. But i am not saying that it would beat > >>>>> your "expedited switch" improvement. > >>>>> > >>>> > >>>> Hi Uladzislau. > >>>> > >>>> Had a discussion on this at LPC, having in kernel solution is likely > >>>> better than having it in userspace. > >>>> > >>>> - Having it in kernel would make it work across all archs. Why should > >>>> any user wait when one initiates the hotplug. > >>>> > >>>> - userspace tools are spread across such as chcpu, ppc64_cpu etc. > >>>> though internally most do "0/1 > /sys/devices/system/cpu/cpuN/online". > >>>> We will have to repeat the same in each tool. > >>>> > >>>> - There is already /sys/kernel/rcu_expedited which is better if at all > >>>> we need to fallback to userspace. > >>>> > >>> Sounds good to me. I agree it is better to bypass parameters. > >> > >> Another way to make it in-kernel would be to make the RCU normal wake from GP optimization enabled for > 16 CPUs by default. > >> > >> I was considering this, but I did not bring it up because I did not know that there are large systems that might benefit from it until now. > >> > > IMO, we can increase that threshold. 512/1024 is not a problem at all. > > But as Paul mentioned, we should consider scalability enhancement. From > > the other hand it is also probably worth to get into the state when we > > really see them :) > > Instead of pegging to number of CPUs, perhaps the optimization should be dynamic? That is, default to it unless synchronize_rcu load is high, default to the sr_normal wake-up optimization. Of course carefully considering all corner cases, adequate testing and all that ;-) > Honestly i do not see use cases when we are not up to speed to process all callbacks in time keeping in mind that it is blocking context call. How many of them should be in flight(blocked contexts) to make it starve... :) According to my last evaluation it was ~64K. Note i do not say that it should not be scaled. -- Uladzislau Rezki