From: Samir M <samir@linux.ibm.com>
To: Shrikanth Hegde <sshegde@linux.ibm.com>,
Joel Fernandes <joelagnelf@nvidia.com>
Cc: Uladzislau Rezki <urezki@gmail.com>,
Paul E McKenney <paulmck@kernel.org>,
Vishal Chourasia <vishalc@linux.ibm.com>,
Neeraj upadhyay <Neeraj.Upadhyay@amd.com>,
RCU <rcu@vger.kernel.org>, LKML <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <frederic@kernel.org>
Subject: Re: [PATCH] rcu: Latch normal synchronize_rcu() path on flood
Date: Sun, 18 Jan 2026 16:54:46 +0530 [thread overview]
Message-ID: <9c544560-1448-42a4-b16d-000f06733d0b@linux.ibm.com> (raw)
In-Reply-To: <00e91ebc-0783-4519-9727-53dd3a625298@linux.ibm.com>
On 17/01/26 5:48 pm, Shrikanth Hegde wrote:
>
>
> On 1/17/26 2:18 PM, Joel Fernandes wrote:
>>
>>
>>> On Jan 17, 2026, at 1:17 AM, Samir M <samir@linux.ibm.com> wrote:
>>>
>>>
>>>> On 15/01/26 12:04 am, Uladzislau Rezki (Sony) wrote:
>>>> Currently, rcu_normal_wake_from_gp is only enabled by default
>>>> on small systems(<= 16 CPUs) or when a user explicitly set it
>>>> enabled.
>>>>
>>>> This patch introduces an adaptive latching mechanism:
>>>> * Tracks the number of in-flight synchronize_rcu() requests
>>>> using a new atomic_t counter(rcu_sr_normal_count);
>>>>
>>>> * If the count exceeds RCU_SR_NORMAL_LATCH_THR(64), it sets
>>>> the rcu_sr_normal_latched, reverting new requests onto the
>>>> scaled wait_rcu_gp() path;
>>>>
>>>> * The latch is cleared only when the pending requests are fully
>>>> drained(nr == 0);
>>>>
>>>> * Enables rcu_normal_wake_from_gp by default for all systems,
>>>> relying on this dynamic throttling instead of static CPU
>>>> limits.
>>>>
>>>> Suggested-by: Joel Fernandes <joelagnelf@nvidia.com>
>>>> Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
>>>> ---
>>>> kernel/rcu/tree.c | 37 ++++++++++++++++++++++++++-----------
>>>> 1 file changed, 26 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
>>>> index 293bbd9ac3f4..c42d480d6e0b 100644
>>>> --- a/kernel/rcu/tree.c
>>>> +++ b/kernel/rcu/tree.c
>>>> @@ -1631,17 +1631,21 @@ static void rcu_sr_put_wait_head(struct
>>>> llist_node *node)
>>>> atomic_set_release(&sr_wn->inuse, 0);
>>>> }
>>>> -/* Enable rcu_normal_wake_from_gp automatically on small
>>>> systems. */
>>>> -#define WAKE_FROM_GP_CPU_THRESHOLD 16
>>>> -
>>>> -static int rcu_normal_wake_from_gp = -1;
>>>> +static int rcu_normal_wake_from_gp = 1;
>>>> module_param(rcu_normal_wake_from_gp, int, 0644);
>>>> static struct workqueue_struct *sync_wq;
>>>> +#define RCU_SR_NORMAL_LATCH_THR 64
>>>> +
>>>> +/* Number of in-flight synchronize_rcu() calls queued on srs_next. */
>>>> +static atomic_long_t rcu_sr_normal_count;
>>>> +static atomic_t rcu_sr_normal_latched;
>>>> +
>>>> static void rcu_sr_normal_complete(struct llist_node *node)
>>>> {
>>>> struct rcu_synchronize *rs = container_of(
>>>> (struct rcu_head *) node, struct rcu_synchronize, head);
>>>> + long nr;
>>>> WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) &&
>>>> !poll_state_synchronize_rcu_full(&rs->oldstate),
>>>> @@ -1649,6 +1653,15 @@ static void rcu_sr_normal_complete(struct
>>>> llist_node *node)
>>>> /* Finally. */
>>>> complete(&rs->completion);
>>>> + nr = atomic_long_dec_return(&rcu_sr_normal_count);
>>>> + WARN_ON_ONCE(nr < 0);
>>>> +
>>>> + /*
>>>> + * Unlatch: switch back to normal path when fully
>>>> + * drained and if it has been latched.
>>>> + */
>>>> + if (nr == 0)
>>>> + (void)atomic_cmpxchg(&rcu_sr_normal_latched, 1, 0);
>>>> }
>>>> static void rcu_sr_normal_gp_cleanup_work(struct work_struct
>>>> *work)
>>>> @@ -1794,7 +1807,14 @@ static bool rcu_sr_normal_gp_init(void)
>>>> static void rcu_sr_normal_add_req(struct rcu_synchronize *rs)
>>>> {
>>>> + long nr;
>>>> +
>>>> llist_add((struct llist_node *) &rs->head, &rcu_state.srs_next);
>>>> + nr = atomic_long_inc_return(&rcu_sr_normal_count);
>>>> +
>>>> + /* Latch: only when flooded and if unlatched. */
>>>> + if (nr >= RCU_SR_NORMAL_LATCH_THR)
>>>> + (void)atomic_cmpxchg(&rcu_sr_normal_latched, 0, 1);
>>>> }
>>>> /*
>>>> @@ -3268,7 +3288,8 @@ static void synchronize_rcu_normal(void)
>>>> trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request"));
>>>> - if (READ_ONCE(rcu_normal_wake_from_gp) < 1) {
>>>> + if (READ_ONCE(rcu_normal_wake_from_gp) < 1 ||
>>>> + atomic_read(&rcu_sr_normal_latched)) {
>>>> wait_rcu_gp(call_rcu_hurry);
>>>> goto trace_complete_out;
>>>> }
>>>> @@ -4892,12 +4913,6 @@ void __init rcu_init(void)
>>>> sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM |
>>>> WQ_UNBOUND, 0);
>>>> WARN_ON(!sync_wq);
>>>> - /* Respect if explicitly disabled via a boot parameter. */
>>>> - if (rcu_normal_wake_from_gp < 0) {
>>>> - if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
>>>> - rcu_normal_wake_from_gp = 1;
>>>> - }
>>>> -
>>>> /* Fill in default value for rcutree.qovld boot parameter. */
>>>> /* -After- the rcu_node ->lock fields are initialized! */
>>>> if (qovld < 0)
>>>
>>>
>>> Hi Uladzislau,
>>>
>>> I verified this patch using the configuration described below.
>>> Configuration:
>>> • Kernel version: 6.19.0-rc5
>>> • Number of CPUs: 2048
>>>
>>> Using this setup, I evaluated the patch with both SMT enabled and
>>> SMT disabled. The results indicate that when SMT is enabled, the
>>> system time is noticeably higher. In contrast, with SMT disabled, no
>>> significant increase in system time is observed.
>>>
>>> SMT=ON -> sys 31m22.922s
>>> SMT=OFF -> sys 0m0.046s
>>>
>>>
>>> SMT Mode | Without Patch | With Patch | % Improvement |
>>> ------------------------------------------------------------------
>>> SMT=off | 30m 53.194s | 26m 24.009s | +14.53% |
>>> SMT=on | 49m 5.920s | 47m 5.513s | +4.09%
>>
>> So it takes you 47 minutes to offline CPUs and you are Ok with that?
>>
>> - Joel
>>
>
>
> This is certainly quite long. IMO not worth the added complexity
> of atomic inc/dec reads happening(even though till 64 CPUs)
>
> Samir,
> can you post number's with vishal's patch of using rcu expedited?
>
>
>>
>>
>>> |
>>>
>>>
>>> Please add below tag: Tested-by: Samir M <samir@linux.ibm.com>
>>>
>>> Regards,
>>> Samir
>>>
>>>
>
Shrikanth,
I verified Vishal’s patch as described below, using the following
configuration.
Configuration:
• Kernel version: 6.19.0-rc5
• Number of CPUs: 2048
Vishal's patch link:
https://lore.kernel.org/all/20260112094332.66006-2-vishalc@linux.ibm.com/
Verification of Vishal’s patch shows a significant improvement in the
*SMT=off* case and a measurable improvement in the *SMT=on* case.
SMT Mode | Without Patch | With Patch | % Improvement |
------------------------------------------------------------------
SMT=off | 30m 53.194s | 6m 4.250s | +80.40% |
SMT=on | 49m 5.920s | 36m 50.386s | +25.01% |
Regards,
Samir
next prev parent reply other threads:[~2026-01-18 11:25 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-01-14 18:34 Uladzislau Rezki (Sony)
2026-01-15 17:23 ` Shrikanth Hegde
2026-01-15 21:39 ` Uladzislau Rezki
2026-01-17 6:17 ` Samir M
2026-01-17 8:48 ` Joel Fernandes
2026-01-17 12:18 ` Shrikanth Hegde
2026-01-18 11:24 ` Samir M [this message]
2026-01-25 14:22 ` Uladzislau Rezki
2026-01-29 19:33 ` Samir M
2026-02-02 11:25 ` Uladzislau Rezki
2026-01-19 11:24 ` Uladzislau Rezki
2026-01-17 15:16 ` Markus Elfring
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9c544560-1448-42a4-b16d-000f06733d0b@linux.ibm.com \
--to=samir@linux.ibm.com \
--cc=Neeraj.Upadhyay@amd.com \
--cc=frederic@kernel.org \
--cc=joelagnelf@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=rcu@vger.kernel.org \
--cc=sshegde@linux.ibm.com \
--cc=urezki@gmail.com \
--cc=vishalc@linux.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®