Re: [PATCH] rcu: Enable runtime reset of the RCU stall panic count

[Joel Fernandes <joelagnelf@nvidia.com>]

> On Oct 6, 2026, at 7:55 AM, Cunlong Li <shenxiaogll@gmail.com> wrote:
> 
> Hi Joel, thanks for the review!
> 
>> On Mon, Oct 05, 2026 at 12:56:30PM +0000, Joel Fernandes wrote:
>> 
>> 
>>>> On Oct 3, 2026, at 9:14 PM, Cunlong Li <shenxiaogll@gmail.com> wrote:
>>> 
>>> The kernel.panic_on_rcu_stall and kernel.max_rcu_stall_to_panic sysctls
>>> count RCU CPU stalls since system boot, and invoke panic() once
>>> max_rcu_stall_to_panic stalls have elapsed.  This count is never reset,
>>> so stalls caused by transient incidents keep consuming the budget of a
>>> long-running system, and a later unrelated stall can
>> 
>> Please provide examples of specific transient events you ran into, and recovered from?
>> 
>>> immediately
>>> trigger panic() instead of allowing the intended fresh window of
>>> stalls.
>>> 
>>> This commit therefore introduces the kernel.rcu_stall_panic_count
>>> sysctl.  Reading this file reports the number of stalls counted so far,
>>> and writing 0 to it resets the count.  This allows the cumulative stall
>>> history to be cleared after an incident has been resolved, and also
>>> allows userspace to clear the count periodically, so that panic() is
>>> only triggered by a burst of stalls occurring within a short period.
>> 
>> Could you provide details of such an incident that got resolved without requiring a reboot?
> 
> The motivation comes from our customer systems running codex on Ubuntu
> 24.04, with a memory limit of ~8G and zram enabled for swap.  When the
> agent's memory footprint grows, the box spends long stretches in direct
> reclaim (zram makes reclaim CPU-heavy), and we get a continuous stream
> of RCU stall warnings.  In the incidents we have seen so far, the
> warnings kept coming and the systems stayed hung, so we would like to
> enable panic_on_rcu_stall so that the box reboots and the service
> recovers automatically.  However, we do not actually know whether some
> of those stalls were transient, i.e. whether the systems would have
> recovered on their own once the memory pressure subsided.  If they
> were, a panic would kill an otherwise recoverable system, which is
> what we would like to avoid.
> 
> We have not yet been able to reproduce this in the lab.  But transient,
> self-recovering stalls are not a hypothetical: the commit that
> introduced max_rcu_stall_to_panic, dfe564045c65 ("rcu: Panic after
> fixed number of stalls"), states the premise outright:
> 
>    Some stalls are transient, so that system fully recovers.  This
>    commit therefore allows users to configure the number of stalls
>    that must happen in order to trigger kernel panic.
> 
> As described in the commit message, since the counter is never reset,
> stalls caused by transient incidents keep consuming the budget of
> max_rcu_stall_to_panic, and a later unrelated stall can immediately
> trigger panic() instead of allowing the intended fresh window of
> stalls.  This is what motivated the patch.

Ok, add all this to commit message. Use case justification is a must as we weigh in additional complexity.

> 
>> 
>> 
>>> 
>>> Signed-off-by: Cunlong Li <shenxiaogll@gmail.com>
>> 
>> If the commit is AI assisted, please add an assisted tag.
> 
> Yes, it is AI-assisted, and I will state that in the v2 commit
> message.

OK.

Thanks,

Joel 

> 
> Thanks,
> Cunlong
> 
>> 
>> Thanks,
>> 
>> Joel
>> 
>>> ---
>>> Documentation/admin-guide/sysctl/kernel.rst | 15 ++++++++++++++-
>>> kernel/rcu/tree_stall.h                     | 28 ++++++++++++++++++++++++++--
>>> 2 files changed, 40 insertions(+), 3 deletions(-)
>>> 
>>> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
>>> index ffea61d448eb..64fe2985e358 100644
>>> --- a/Documentation/admin-guide/sysctl/kernel.rst
>>> +++ b/Documentation/admin-guide/sysctl/kernel.rst
>>> @@ -959,7 +959,20 @@ max_rcu_stall_to_panic
>>> When ``panic_on_rcu_stall`` is set to 1, this value determines the
>>> number of times that RCU can stall before panic() is called.
>>> 
>>> -When ``panic_on_rcu_stall`` is set to 0, this value is has no effect.
>>> +When ``panic_on_rcu_stall`` is set to 0, this value has no effect.
>>> +
>>> +rcu_stall_panic_count
>>> +=====================
>>> +
>>> +Indicates the number of RCU CPU stalls that have been counted since
>>> +system boot or since the counter was reset. When ``panic_on_rcu_stall``
>>> +is set to 1, this count is compared against ``max_rcu_stall_to_panic``
>>> +to decide whether panic() should be called.
>>> +
>>> +Writing 0 to this file resets the counter to zero, which restarts the
>>> +``max_rcu_stall_to_panic`` window of stalls. This allows system
>>> +administrators to clear the cumulative stall count after an incident
>>> +has been resolved, without requiring a system restart.
>>> 
>>> perf_cpu_time_max_percent
>>> =========================
>>> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
>>> index 091e7850ab6e..a80f03e1c7ea 100644
>>> --- a/kernel/rcu/tree_stall.h
>>> +++ b/kernel/rcu/tree_stall.h
>>> @@ -19,6 +19,20 @@
>>> /* panic() on RCU Stall sysctl. */
>>> static int sysctl_panic_on_rcu_stall __read_mostly;
>>> static int sysctl_max_rcu_stall_to_panic __read_mostly;
>>> +static unsigned long sysctl_rcu_stall_panic_count;
>>> +
>>> +/* Reset the RCU stall panic count when written to. */
>>> +static int proc_do_rcu_stall_panic_count(const struct ctl_table *table, int write,
>>> +                     void *buffer, size_t *lenp, loff_t *ppos)
>>> +{
>>> +    if (!write)
>>> +        return proc_doulongvec_minmax(table, write, buffer, lenp, ppos);
>>> +
>>> +    WRITE_ONCE(sysctl_rcu_stall_panic_count, 0);
>>> +    *ppos += *lenp;
>>> +
>>> +    return 0;
>>> +}
>>> 
>>> static const struct ctl_table rcu_stall_sysctl_table[] = {
>>>   {
>>> @@ -39,6 +53,13 @@ static const struct ctl_table rcu_stall_sysctl_table[] = {
>>>       .extra1        = SYSCTL_ONE,
>>>       .extra2        = SYSCTL_INT_MAX,
>>>   },
>>> +    {
>>> +        .procname    = "rcu_stall_panic_count",
>>> +        .data        = &sysctl_rcu_stall_panic_count,
>>> +        .maxlen        = sizeof(sysctl_rcu_stall_panic_count),
>>> +        .mode        = 0644,
>>> +        .proc_handler    = proc_do_rcu_stall_panic_count,
>>> +    },
>>> };
>>> 
>>> static int __init init_rcu_stall_sysctl(void)
>>> @@ -161,7 +182,7 @@ early_initcall(check_cpu_stall_init);
>>> /* If so specified via sysctl, panic, yielding cleaner stall-warning output. */
>>> static void panic_on_rcu_stall(const struct cpumask *stalled_mask)
>>> {
>>> -    static int cpu_stall;
>>> +    unsigned long count;
>>> 
>>>   /*
>>>    * Attempt to kick out the BPF scheduler if it's installed and defer
>>> @@ -170,7 +191,10 @@ static void panic_on_rcu_stall(const struct cpumask *stalled_mask)
>>>   if (scx_rcu_cpu_stall(stalled_mask))
>>>       return;
>>> 
>>> -    if (++cpu_stall < sysctl_max_rcu_stall_to_panic)
>>> +    /* A lost RMW update only delays the panic by one stall. */
>>> +    count = READ_ONCE(sysctl_rcu_stall_panic_count) + 1;
>>> +    WRITE_ONCE(sysctl_rcu_stall_panic_count, count);
>>> +    if (count < (unsigned long)READ_ONCE(sysctl_max_rcu_stall_to_panic))
>>>       return;
>>> 
>>>   if (sysctl_panic_on_rcu_stall)
>>> 
>>> ---
>>> base-commit: ce1e0223d8ad4211275c82a17ed6d43ab81e13d9
>>> change-id: 20261003-rcu-375496d7704e
>>> 
>>> Best regards,
>>> --
>>> Cunlong Li <shenxiaogll@gmail.com>
>>>