From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 458B61B3925 for ; Thu, 2 Apr 2026 12:40:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775133646; cv=none; b=I2uV2PALKij8fnULZoTCto/WdVB8cxmFzLIK1R/iSKG6S+GxO68Jn/oUwNc5XXsnnt6YmroDTyGKdbMsI0R74TpnLfy+PB08Z3cTqanGnhF1fhUvj3oDZJX8aTRmXDy0Sy3+KEQa75x0i5tS2iz2YJA0kqdsYDccGYwfq5QQaMc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775133646; c=relaxed/simple; bh=XYeq/PNjyN9HI+842/AHDGKzcK+nvNQ4ceQ2oKePnds=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=VZDygQnNCSgh40r2d7ANCdfpwjJWDR9Ex6sF8B9a5f82HBPLUwtPTCDcZnn04Z8HN3pO/bJU4rg9tlgFenWc2gH6zFwIra4+tvW+Fx1LR/NL+IdSBSRy5WvOGCKsjlg7of2X1RoXsSXn8FLHfg/aa7CjxcOJF8yMxkjfDBaVzaY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Keh6EsuO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Keh6EsuO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0B01CC19423; Thu, 2 Apr 2026 12:40:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1775133646; bh=XYeq/PNjyN9HI+842/AHDGKzcK+nvNQ4ceQ2oKePnds=; h=Date:Subject:From:To:Cc:References:In-Reply-To:From; b=Keh6EsuOB14aDnPKmGZxn8GvBhSg0AkRYDDA/mzz6BEuCd30UWuNQ1etMqXWdkYU/ vBLJfLWNTtSw5UYBWCHZ44KE4IlRlSLyVw1vRd0vO1aC04VI83RC0DjCZ/MGVBdv46 8NKcYAZ7RusqRFBvY6WaMW68MegqH4HvuyY0/pf2xuwory48HklD9nY5Lfm4VrAoJi sY5AU3xQQ0zD5Ug55D07OrKz4mFufyw57QS/06bPwuQJwprxF5olIrj2VpVMcBX+WH GPp8tAQM/Fg4aYfeYYgSsT0gDPTVBccCmt2nb5pfeSI2wl9elDz2BPdIXnTyVvHVmi l3Vh3P6Qo8RKA== Message-ID: Date: Thu, 2 Apr 2026 14:40:41 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm/vmstat: spread vmstat_update requeue across the stat interval From: "Vlastimil Babka (SUSE)" To: Breno Leitao , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kas@kernel.org, shakeel.butt@linux.dev, usama.arif@linux.dev, kernel-team@meta.com References: <20260401-vmstat-v1-1-b68ce4a35055@debian.org> Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 4/1/26 7:46 PM, Vlastimil Babka (SUSE) wrote: > On 4/1/26 15:57, Breno Leitao wrote: >> vmstat_update uses round_jiffies_relative() when re-queuing itself, >> which aligns all CPUs' timers to the same second boundary. When many >> CPUs have pending PCP pages to drain, they all call decay_pcp_high() -> >> free_pcppages_bulk() simultaneously, serializing on zone->lock and >> hitting contention. >> >> Introduce vmstat_spread_delay() which distributes each CPU's >> vmstat_update evenly across the stat interval instead of aligning them. >> >> This does not increase the number of timer interrupts — each CPU still >> fires once per interval. The timers are simply staggered rather than >> aligned. Additionally, vmstat_work is DEFERRABLE_WORK, so it does not >> wake idle CPUs regardless of scheduling; the spread only affects CPUs >> that are already active >> >> `perf lock contention` shows 7.5x reduction in zone->lock contention >> (872 -> 117 contentions, 199ms -> 81ms total wait) on a 72-CPU aarch64 >> system under memory pressure. >> >> Tested on a 72-CPU aarch64 system using stress-ng --vm to generate >> memory allocation bursts. Lock contention was measured with: >> >> perf lock contention -a -b -S free_pcppages_bulk >> >> Results with KASAN enabled: >> >> free_pcppages_bulk contention (KASAN): >> +--------------+----------+----------+ >> | Metric | No fix | With fix | >> +--------------+----------+----------+ >> | Contentions | 872 | 117 | >> | Total wait | 199.43ms | 80.76ms | >> | Max wait | 4.19ms | 35.76ms | >> +--------------+----------+----------+ >> >> Results without KASAN: >> >> free_pcppages_bulk contention (no KASAN): >> +--------------+----------+----------+ >> | Metric | No fix | With fix | >> +--------------+----------+----------+ >> | Contentions | 240 | 133 | >> | Total wait | 34.01ms | 24.61ms | >> | Max wait | 965us | 1.35ms | >> +--------------+----------+----------+ >> >> Signed-off-by: Breno Leitao > > Cool! > > I noticed __round_jiffies_relative() exists and the description looks like > it's meant for exactly this use case? On closer look, using round_jiffies_relative() as before your patch means it's calling __round_jiffies_relative(j, raw_smp_processor_id()) so that's already doing this spread internally. You're also relying smp_processor_id() so it's not about using a different cpu id. But your patch has better results, why? I still think it's not doing what it intends - I think it makes every cpu have different interval length (up to twice the original length), not skew. Is it that, or that the 3 jiffies skew per cpu used in round_jiffies_common() is insufficient? Or it a bug in its skew implementation? Ideally once that's clear, the findings could be used to improve round_jiffies_common() and hopefully there's nothing here that's vmstat specific. Thanks, Vlastimil >> --- >> mm/vmstat.c | 25 ++++++++++++++++++++++++- >> 1 file changed, 24 insertions(+), 1 deletion(-) >> >> diff --git a/mm/vmstat.c b/mm/vmstat.c >> index 2370c6fb1fcd..2e94bd765606 100644 >> --- a/mm/vmstat.c >> +++ b/mm/vmstat.c >> @@ -2032,6 +2032,29 @@ static int vmstat_refresh(const struct ctl_table *table, int write, >> } >> #endif /* CONFIG_PROC_FS */ >> >> +/* >> + * Return a per-cpu delay that spreads vmstat_update work across the stat >> + * interval. Without this, round_jiffies_relative() aligns every CPU's >> + * timer to the same second boundary, causing a thundering-herd on >> + * zone->lock when multiple CPUs drain PCP pages simultaneously via >> + * decay_pcp_high() -> free_pcppages_bulk(). >> + */ >> +static unsigned long vmstat_spread_delay(void) >> +{ >> + unsigned long interval = sysctl_stat_interval; >> + unsigned int nr_cpus = num_online_cpus(); >> + >> + if (nr_cpus <= 1) >> + return round_jiffies_relative(interval); >> + >> + /* >> + * Spread per-cpu vmstat work evenly across the interval. Don't >> + * use round_jiffies_relative() here -- it would snap every CPU >> + * back to the same second boundary, defeating the spread. >> + */ >> + return interval + (interval * (smp_processor_id() % nr_cpus)) / nr_cpus; > > Hm doesn't this mean that lower id cpus will consistently fire in shorter > intervals and higher id in longer intervals? What we want is same interval > but differently offset, no? > >> +} >> + >> static void vmstat_update(struct work_struct *w) >> { >> if (refresh_cpu_vm_stats(true)) { >> @@ -2042,7 +2065,7 @@ static void vmstat_update(struct work_struct *w) >> */ >> queue_delayed_work_on(smp_processor_id(), mm_percpu_wq, >> this_cpu_ptr(&vmstat_work), >> - round_jiffies_relative(sysctl_stat_interval)); >> + vmstat_spread_delay()); >> } >> } >> >> >> --- >> base-commit: cf7c3c02fdd0dfccf4d6611714273dcb538af2cb >> change-id: 20260401-vmstat-048e0feaf344 >> >> Best regards, >> -- >> Breno Leitao >> >