[PATCH] x86/mce: Restore rate limit on MCE logged message

[PATCH] x86/mce: Restore rate limit on MCE logged message

[Rik van Riel]

FCommit ea324444ece9f moved the "Machine check events logged"
print from mce_notify_irq() into mce_log() and dropped its
two-per-minute rate limit as a side effect.

A particularly unhappy system can spew out around 100 million
of those "Machine check events logged" messages every hour,
without that rate limit. A large fleet may have several unhappy
systems, leaving the netcons infrastructure somewhat flustered.

Reinstate the rate limit we had before.

Fixes: ea324444ece9f ("x86/mce: Restore MCA polling interval halving")
Cc: stable@vger.kernel.org
Signed-off-by: Rik van Riel <riel@surriel.com>
---
 arch/x86/kernel/cpu/mce/core.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index 50b1614968be..06f858faf2b1 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -150,8 +150,12 @@ EXPORT_PER_CPU_SYMBOL_GPL(injectm);
 
 void mce_log(struct mce_hw_err *err)
 {
+	/* Not more than two messages every minute */
+	static DEFINE_RATELIMIT_STATE(ratelimit, 60*HZ, 2);
+
 	if (mce_gen_pool_add(err)) {
-		pr_info(HW_ERR "Machine check events logged\n");
+		if (__ratelimit(&ratelimit))
+			pr_info(HW_ERR "Machine check events logged\n");
 		irq_work_queue(&mce_irq_work);
 	}
 }
-- 
2.56.0

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Borislav Petkov]

On Sun, Nov 08, 2026 at 02:37:51PM -0500, Rik van Riel wrote:
> FCommit ea324444ece9f moved the "Machine check events logged"
> print from mce_notify_irq() into mce_log() and dropped its
> two-per-minute rate limit as a side effect.
> 
> A particularly unhappy system can spew out around 100 million
> of those "Machine check events logged" messages every hour,
> without that rate limit. A large fleet may have several unhappy
> systems, leaving the netcons infrastructure somewhat flustered.

I need more info here. How are you running those systems?

Usually, correctable errors are consumed by the CEC and you don't see
anything. So you must be doing something else.

.config and dmesg pls.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Luck, Tony]

> Usually, correctable errors are consumed by the CEC and you don't see
> anything. So you must be doing something else.

mce_log() is called from the interrupt.  CEC is called later because it registered
with mce_register_decode_chain() to be told about errors that were added to the
queue by mce_gen_pool_add().

So the message goes out to the console before CEC has a chance to squash
reporting of corrected errors.

Change looks good to me:

Reviewed-by: Tony Luck <tony.luck@intel.com>

-Tony

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Borislav Petkov]

On Mon, Nov 09, 2026 at 05:54:34PM +0000, Luck, Tony wrote:
> mce_log() is called from the interrupt.

Aha, through machine_check_poll(). Let's hear from Rik first tho.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Rik van Riel]

On Mon, 2026-11-09 at 11:52 -0800, Borislav Petkov wrote:
> On Mon, Nov 09, 2026 at 05:54:34PM +0000, Luck, Tony wrote:
> > mce_log() is called from the interrupt.
> 
> Aha, through machine_check_poll(). Let's hear from Rik first tho.

Not only the poll path, but also the #MC exception
handler.

exc_machine_check
  --> exc_machine_check_{kernel,user}
      --> do_machine_check
          --> mce_log
              --> pr_info("Machine check events logged")

As for the dmesg, it is almost entirely filled
with "Machine check events logged" errors, at
a rate of about 100 million an hour.

There are a dozen or so "mce: MCE records pool full!"
lines, and then two of these:
"mce_gen_pool_add: <bignum> callbacks suppressed"

I suspect there isn't anything special about the
kernel here, we are just looking at some unusually
broken hardware, which will be present in a fleet
this size.

-- 
All Rights Reversed.

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Borislav Petkov]

On Mon, Nov 09, 2026 at 03:12:43PM -0500, Rik van Riel wrote:
> There are a dozen or so "mce: MCE records pool full!"
> lines, and then two of these:
> "mce_gen_pool_add: <bignum> callbacks suppressed"

Sounds to me like you getting errors logged and there's no userspace consumer
at all.

What is your fleet policy about collecting and logging hw errors?

Do you even care about them?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Rik van Riel]

On Mon, 2026-11-09 at 13:23 -0800, Borislav Petkov wrote:
> On Mon, Nov 09, 2026 at 03:12:43PM -0500, Rik van Riel wrote:
> > There are a dozen or so "mce: MCE records pool full!"
> > lines, and then two of these:
> > "mce_gen_pool_add: <bignum> callbacks suppressed"
> 
> Sounds to me like you getting errors logged and there's no userspace
> consumer
> at all.
> 
> What is your fleet policy about collecting and logging hw errors?
> 
> Do you even care about them?

We definitely care about them. Errors are used
to figure out which hosts need to be repaired.

I do not know how exactly memory errors are getting
collected. There are some tools that watch netcons,
some that collect data on the host, and some that
grab data off the BMC. I have no idea which of
those paths the memory error collection uses.

-- 
All Rights Reversed.

Re: [PATCH] x86/mce: Restore rate limit on MCE logged message

[Borislav Petkov]

On Mon, Nov 09, 2026 at 04:29:32PM -0500, Rik van Riel wrote:
> I do not know how exactly memory errors are getting
> collected. There are some tools that watch netcons,
> some that collect data on the host, and some that
> grab data off the BMC. I have no idea which of
> those paths the memory error collection uses.

It probably doesn't matter for the problem at hand - I guess we should
ratelimit that printk after all...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette