[PATCH v3 0/2] x86/mce: Rate-limit messages from corrected error floods

[PATCH v3 0/2] x86/mce: Rate-limit messages from corrected error floods

[Breno Leitao]

A machine check bank that reports corrected errors at a high rate
makes the MCE code print two messages without any bound: "Machine
check events logged" for every error mce_log() queues, and "CMCI storm
detected" / "CMCI storm subsided" every time the bank enters or leaves
storm mode.

One host printed about 100 million "Machine check events logged" lines
an hour. On another, the storm notices were 82% of everything it wrote
to the kernel log.

Patch 1, from Rik, restores the two-per-minute limit that "Machine
check events logged" had before it moved into mce_log(). It is a fix
and is tagged for stable.

Patch 2 puts both storm notices behind one ratelimit, with the default
interval and burst.

Only the messages are limited. The errors are still logged and storm
handling is unchanged.

Both patches were posted separately before. Borislav asked to merge
them into one series [1].

[1] https://lore.kernel.org/r/20261112182125.GDavYDpdJpmpU2Piyu@fat_crate.local

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v3:
- Add Rik's "x86/mce: Restore rate limit on MCE logged message" as
  patch 1 (Borislav). It carries the Reviewed-by tags from its
  posting; only its first changelog sentence changed (typo, commit
  reference style).
- Link to Rik's posting: https://patch.msgid.link/20261108143751.3af49e48@fangorn
- Link to v2: https://patch.msgid.link/20261112-mci_ratelimit-v2-1-a9e67055526b@debian.org

Changes in v2:
- rebased on linux-next
- Link to v1: https://patch.msgid.link/20260827-mci_ratelimit-v1-1-ac43df769b36@debian.org

---
Breno Leitao (1):
      x86/mce: Rate-limit the CMCI storm transition notices

Rik van Riel (1):
      x86/mce: Restore rate limit on MCE logged message

 arch/x86/kernel/cpu/mce/core.c      |  6 +++++-
 arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++--
 2 files changed, 19 insertions(+), 3 deletions(-)
---
base-commit: aaa2f954c21b205a0bcd0961e5b2be4e82e58455
change-id: 20260827-mci_ratelimit-064421e2d8cd

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH v3 1/2] x86/mce: Restore rate limit on MCE logged message

[Breno Leitao]

From: Rik van Riel <riel@surriel.com>

Commit ea324444ece9f ("x86/mce: Restore MCA polling interval halving")
moved the "Machine check events logged" print from mce_notify_irq()
into mce_log() and dropped its two-per-minute rate limit as a side
effect.

A particularly unhappy system can spew out around 100 million
of those "Machine check events logged" messages every hour,
without that rate limit. A large fleet may have several unhappy
systems, leaving the netcons infrastructure somewhat flustered.

Reinstate the rate limit we had before.

Fixes: ea324444ece9f ("x86/mce: Restore MCA polling interval halving")
Cc: stable@vger.kernel.org
Signed-off-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Tony Luck <tony.luck@intel.com>
Reviewed-by: Qiuxu Zhuo <qzhuo@nvidia.com>
Reviewed-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 arch/x86/kernel/cpu/mce/core.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 50b1614968be6a..06f858faf2b1cd 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -150,8 +150,12 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);
 
 void mce_log(struct mce_hw_err *err)
 {
+	/* Not more than two messages every minute */
+	static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2);
+
 	if (mce_gen_pool_add(err)) {
-		pr_info(HW_ERR "Machine check events logged\n");
+		if (__ratelimit(&ratelimit))
+			pr_info(HW_ERR "Machine check events logged\n");
 		irq_work_queue(&mce_irq_work);
 	}
 }

-- 
2.53.1-Meta

[PATCH v3 2/2] x86/mce: Rate-limit the CMCI storm transition notices

[Breno Leitao]

mce_track_storm() prints a line every time a bank enters storm mode and
another every time it leaves. Neither is bounded, so the log volume
follows whatever rate the bank flaps at.

One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
"CMCI storm subsided" lines for a single bank over four days, which is
a bit excessive for all monitoring purposes.

Put both transitions behind one ratelimit.

Acked-by: Tony Luck <tony.luck@intel.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
index 6c370d5af5bd23..8591715ce24306 100644
--- a/arch/x86/kernel/cpu/mce/threshold.c
+++ b/arch/x86/kernel/cpu/mce/threshold.c
@@ -4,6 +4,7 @@
  */
 #include <linux/interrupt.h>
 #include <linux/kernel.h>
+#include <linux/ratelimit.h>
 
 #include <asm/irq_vectors.h>
 #include <asm/traps.h>
@@ -111,6 +112,13 @@ void cmci_storm_end(unsigned int bank)
 		mce_timer_kick(false);
 }
 
+/*
+ * Shared by both transitions so that a bank flapping between them cannot
+ * outrun the console.
+ */
+static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
 void mce_track_storm(struct mce *mce)
 {
 	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
@@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
 	if (storm->banks[mce->bank].in_storm_mode) {
 		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, false);
 		cmci_storm_end(mce->bank);
 	} else {
 		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, true);
 		cmci_storm_begin(mce->bank);
 	}

-- 
2.53.1-Meta

RE: [PATCH v3 2/2] x86/mce: Rate-limit the CMCI storm transition notices

[Qiuxu (Leo) Zhuo]

> From: Breno Leitao <leitao@debian.org>
> Sent: Friday, November 13, 2026 8:49 PM
> To: Tony Luck <tony.luck@intel.com>; Borislav Petkov <bp@alien8.de>; Thomas
> Gleixner <tglx@kernel.org>; Ingo Molnar <mingo@redhat.com>; Dave Hansen
> <dave.hansen@linux.intel.com>; x86@kernel.org; H. Peter Anvin
> <hpa@zytor.com>; Qiuxu (Leo) Zhuo <qzhuo@nvidia.com>; Qiuxu Zhuo
> <qiuxu.zhuo@intel.com>
> Cc: linux-edac@vger.kernel.org; linux-kernel@vger.kernel.org; Rik van Riel
> <riel@surriel.com>; Breno Leitao <leitao@debian.org>; kernel-team@meta.com
> Subject: [PATCH v3 2/2] x86/mce: Rate-limit the CMCI storm transition notices
> 
> 
> mce_track_storm() prints a line every time a bank enters storm mode and another
> every time it leaves. Neither is bounded, so the log volume follows whatever rate
> the bank flaps at.
> 
> One host in our fleet logged 9970726 "CMCI storm detected" and 9970734 "CMCI
> storm subsided" lines for a single bank over four days, which is a bit excessive for all
> monitoring purposes.
> 
> Put both transitions behind one ratelimit.
> 
> Acked-by: Tony Luck <tony.luck@intel.com>
> Signed-off-by: Breno Leitao <leitao@debian.org>

LGTM. Thanks.

Reviewed-by: Qiuxu Zhuo <qzhuo@nvidia.com>

Re: [PATCH v3 1/2] x86/mce: Restore rate limit on MCE logged message

[Sohil Mehta]

On 11/13/2026 4:49 AM, Breno Leitao wrote:
> From: Rik van Riel <riel@surriel.com>
> 
> Commit ea324444ece9f ("x86/mce: Restore MCA polling interval halving")
> moved the "Machine check events logged" print from mce_notify_irq()
> into mce_log() and dropped its two-per-minute rate limit as a side
> effect.
> 
> A particularly unhappy system can spew out around 100 million
> of those "Machine check events logged" messages every hour,
> without that rate limit. A large fleet may have several unhappy
> systems, leaving the netcons infrastructure somewhat flustered.
> 
> Reinstate the rate limit we had before.
> 

Nit: How about avoiding the 'we' and saying:

  Reinstate the two-per-minute rate limit.

> Fixes: ea324444ece9f ("x86/mce: Restore MCA polling interval halving")
> Cc: stable@vger.kernel.org
> Signed-off-by: Rik van Riel <riel@surriel.com>
> Reviewed-by: Tony Luck <tony.luck@intel.com>
> Reviewed-by: Qiuxu Zhuo <qzhuo@nvidia.com>
> Reviewed-by: Breno Leitao <leitao@debian.org>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---

Reviewed-by: Sohil Mehta <sohil.mehta@intel.com>

>  arch/x86/kernel/cpu/mce/core.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
> index 50b1614968be6a..06f858faf2b1cd 100644
> --- a/arch/x86/kernel/cpu/mce/core.c
> +++ b/arch/x86/kernel/cpu/mce/core.c
> @@ -150,8 +150,12 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);
>  
>  void mce_log(struct mce_hw_err *err)
>  {
> +	/* Not more than two messages every minute */
> +	static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2);
> +
>  	if (mce_gen_pool_add(err)) {
> -		pr_info(HW_ERR "Machine check events logged\n");
> +		if (__ratelimit(&ratelimit))
> +			pr_info(HW_ERR "Machine check events logged\n");
>  		irq_work_queue(&mce_irq_work);
>  	}
>  }
> 

Re: [PATCH v3 2/2] x86/mce: Rate-limit the CMCI storm transition notices

[Sohil Mehta]

On 11/13/2026 4:49 AM, Breno Leitao wrote:
> mce_track_storm() prints a line every time a bank enters storm mode and
> another every time it leaves. Neither is bounded, so the log volume
> follows whatever rate the bank flaps at.
> 
> One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
> "CMCI storm subsided" lines for a single bank over four days, which is
> a bit excessive for all monitoring purposes.
> 
> Put both transitions behind one ratelimit.
> 
> Acked-by: Tony Luck <tony.luck@intel.com>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---

Reviewed-by: Sohil Mehta <sohil.mehta@intel.com>

A minor nit below.

>  arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++--
>  1 file changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
> index 6c370d5af5bd23..8591715ce24306 100644
> --- a/arch/x86/kernel/cpu/mce/threshold.c
> +++ b/arch/x86/kernel/cpu/mce/threshold.c
> @@ -4,6 +4,7 @@
>   */
>  #include <linux/interrupt.h>
>  #include <linux/kernel.h>
> +#include <linux/ratelimit.h>
>  
>  #include <asm/irq_vectors.h>
>  #include <asm/traps.h>
> @@ -111,6 +112,13 @@ void cmci_storm_end(unsigned int bank)
>  		mce_timer_kick(false);
>  }
>  
> +/*
> + * Shared by both transitions so that a bank flapping between them cannot
> + * outrun the console.
> + */
> +static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL,
> +			      DEFAULT_RATELIMIT_BURST);
> +

Should this be defined as a function local similar to patch 1?

>  void mce_track_storm(struct mce *mce)
>  {
>  	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
> @@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
>  	if (storm->banks[mce->bank].in_storm_mode) {
>  		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
>  			return;
> -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
> +		if (__ratelimit(&storm_rs))
> +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
> +					smp_processor_id(), mce->bank);
>  		mce_handle_storm(mce->bank, false);
>  		cmci_storm_end(mce->bank);
>  	} else {
>  		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
>  			return;
> -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
> +		if (__ratelimit(&storm_rs))
> +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
> +					smp_processor_id(), mce->bank);
>  		mce_handle_storm(mce->bank, true);
>  		cmci_storm_begin(mce->bank);
>  	}
>