Re: [PATCH] ghes: Track number of recovered hardware errors

[Shuai Xue <xueshuai@linux.alibaba.com>]

在 2025/7/15 23:09, Borislav Petkov 写道:
> On Tue, Jul 15, 2025 at 09:46:03PM +0800, Shuai Xue wrote:
>> For the purpose of counting, how about using the cmdline of rasdaemon?
> 
> That would mean you have to run rasdaemon on those machines before they
> explode and then carve out the rasdaemon db from the coredump (this is
> post-mortem analysis).

Rasdaemon is a userspace tool that will collect all hardware error 
events reported by the Linux Kernel from several sources (EDAC, MCE, 
PCI, ...) into one common framework. And it has been a standard tools
in Alibaba. As far as I know, twitter also use Rasdaemon in its production.

> 
> I would love for rasdaemon to log over the network and then other tools can
> query those centralized logs but that has its own challenges...
> 

I also prefer collecting rasdaemon data in a centralized data center, as 
this is more beneficial for using big data analytics to analyze and 
predict errors. At the same time, the centralized side also uses 
rasdaemon logs as one of the references for machine operations and 
maintenance.

As for rasdaemon itself, it is just a single-node event collector and 
database, although it does also print logs. In practice, we use SLS [1] 
to collect rasdaemon text logs from individual nodes and parse them on 
the central side.

Thanks.
Shuai

[1]https://www.alibabacloud.com/help/en/sls/getting-started
[2]https://blog.x.com/engineering/en_us/topics/infrastructure/2023/how-twitter-uses-rasdaemon-for-hardware-reliability