From: Yazen Ghannam <yazen.ghannam@amd.com>
To: Borislav Petkov <bp@alien8.de>, Tony Luck <tony.luck@intel.com>
Cc: yazen.ghannam@amd.com, Smita.KoralahalliChannabasappa@amd.com,
dave.hansen@linux.intel.com, x86@kernel.org,
linux-edac@vger.kernel.org, linux-kernel@vger.kernel.org,
patches@lists.linux.dev
Subject: Re: [PATCH v4 2/5] x86/mce: Add per-bank CMCI storm mitigation
Date: Tue, 11 Apr 2023 10:06:45 -0400 [thread overview]
Message-ID: <6b6e53c5-8562-8ea0-befa-88c8b38663d6@amd.com> (raw)
In-Reply-To: <20230411123234.GGZDVTYppLWRN5t0jX@fat_crate.local>
On 4/11/23 08:32, Borislav Petkov wrote:
> On Mon, Apr 03, 2023 at 02:07:13PM -0700, Tony Luck wrote:
>> Add a hook into machine_check_poll() to keep track of per-CPU, per-bank
>> corrected error logs.
>>
>> Maintain a bitmap history for each bank showing whether the bank
>> logged an corrected error or not each time it is polled.
>>
>> In normal operation the interval between polls of this banks
>> determines how far to shift the history. The 64 bit width corresponds
>> to about one second.
>>
>> When a storm is observed the Rate of interrupts is reduced by setting
>> a large threshold value for this bank in IA32_MCi_CTL2. This bank is
>> added to the bitmap of banks for this CPU to poll. The polling rate
>> is increased to once per second.
>> During a storm each bit in the history indicates the status of the
>> bank each time it is polled. Thus the history covers just over a minute.
>>
>> Declare a storm for that bank if the number of corrected interrupts
>> seen in that history is above some threshold (5 in this RFC code for
>> ease of testing, likely move to 15 for compatibility with previous
>> storm detection).
>>
>> A storm on a bank ends if enough consecutive polls of the bank show
>> no corrected errors (currently 30, may also change). That resets the
>> threshold in IA32_MCi_CTL2 back to 1, removes the bank from the bitmap
>> for polling, and changes the polling rate back to the default.
>>
>> If a CPU with banks in storm mode is taken offline, the new CPU
>> that inherits ownership of those banks takes over management of
>> storm(s) in the inherited bank(s).
>>
>> Signed-off-by: Tony Luck <tony.luck@intel.com>
>> Reviewed-by: Yazen Ghannam <yazen.ghannam@amd.com>
>> Tested-by: Yazen Ghannam <yazen.ghannam@amd.com>
>> ---
>> arch/x86/kernel/cpu/mce/internal.h | 4 +-
>> arch/x86/kernel/cpu/mce/core.c | 26 ++++--
>> arch/x86/kernel/cpu/mce/intel.c | 139 ++++++++++++++++++++++++++++-
>> 3 files changed, 158 insertions(+), 11 deletions(-)
>
> ld: vmlinux.o: in function `machine_check_poll':
> /home/boris/kernel/2nd/linux/arch/x86/kernel/cpu/mce/core.c:683: undefined reference to `track_cmci_storm'
> make[1]: *** [scripts/Makefile.vmlinux:35: vmlinux] Error 1
> make: *** [Makefile:1249: vmlinux] Error 2
> make: *** Waiting for unfinished jobs....
>
Ah, this is with CONFIG_MCE_INTEL=n and everything =y. Is there an automated
way to test every config, not just random, combination in a subsystem?
I'll try to add something like this to my flow. It seems allnoconfig,
defconfig, etc. aren't enough. And it's too easy to overlook during code review.
Thanks,
Yazen
next prev parent reply other threads:[~2023-04-11 14:07 UTC|newest]
Thread overview: 104+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-06 6:35 [RFC PATCH 0/5] Handle corrected machine check interrupt storms Smita Koralahalli
2022-04-06 6:35 ` [PATCH 1/5] x86/mce: Remove old CMCI storm mitigation code Smita Koralahalli
2022-04-06 6:35 ` [PATCH 2/5] x86/mce: Add per-bank CMCI storm mitigation Smita Koralahalli
2022-04-06 6:35 ` [RFC PATCH 3/5] x86/mce: Introduce a function pointer mce_handle_storm Smita Koralahalli
2022-04-06 22:38 ` Luck, Tony
2022-04-06 6:35 ` [RFC PATCH 4/5] x86/mce: Move storm handling to core Smita Koralahalli
2022-06-21 5:08 ` Luck, Tony
2022-06-27 17:36 ` [PATCH v2 0/5] Handle corrected machine check interrupt storms Tony Luck
2022-06-27 17:36 ` [PATCH v2 1/5] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2022-06-27 17:36 ` [PATCH v2 2/5] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2022-06-27 17:36 ` [PATCH v2 3/5] x86/mce: Introduce mce_handle_storm() to deal with begin/end of storms Tony Luck
2022-06-27 17:36 ` [PATCH v2 4/5] x86/mce: Move storm handling to core Tony Luck
2022-06-27 17:36 ` [PATCH v2 5/5] x86/mce: Handle AMD threshold interrupt storms Tony Luck
2023-03-17 14:50 ` [PATCH v2 0/5] Handle corrected machine check " Yazen Ghannam
2023-03-17 17:20 ` [PATCH v3 " Tony Luck
2023-03-17 17:20 ` [PATCH v3 1/5] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-03-17 17:20 ` [PATCH v3 2/5] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-03-17 17:20 ` [PATCH v3 3/5] x86/mce: Introduce mce_handle_storm() to deal with begin/end of storms Tony Luck
2023-03-23 15:22 ` Yazen Ghannam
2023-03-23 18:00 ` Tony Luck
2023-03-17 17:20 ` [PATCH v3 4/5] x86/mce: Move storm handling to core Tony Luck
2023-03-23 15:27 ` Yazen Ghannam
2023-03-23 18:10 ` Luck, Tony
2023-03-23 20:26 ` Luck, Tony
2023-03-24 20:44 ` Yazen Ghannam
2023-03-29 15:26 ` Yazen Ghannam
2023-04-03 19:03 ` Luck, Tony
2023-04-03 21:07 ` [PATCH v4 0/5] Handle corrected machine check interrupt storms Tony Luck
2023-04-03 21:07 ` [PATCH v4 1/5] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-04-03 21:07 ` [PATCH v4 2/5] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-04-11 12:32 ` Borislav Petkov
2023-04-11 14:06 ` Yazen Ghannam [this message]
2023-04-11 16:06 ` Luck, Tony
2023-04-11 17:17 ` Borislav Petkov
2023-04-03 21:07 ` [PATCH v4 3/5] x86/mce: Introduce mce_handle_storm() to deal with begin/end of storms Tony Luck
2023-04-03 21:07 ` [PATCH v4 4/5] x86/mce: Move storm handling to core Tony Luck
2023-04-03 21:07 ` [PATCH v4 5/5] x86/mce: Handle AMD threshold interrupt storms Tony Luck
2023-04-11 17:38 ` [PATCH v5 0/5] Handle corrected machine check " Tony Luck
2023-04-11 17:38 ` [PATCH v5 1/5] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-04-11 17:38 ` [PATCH v5 2/5] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-06-13 17:45 ` Borislav Petkov
2023-06-16 18:15 ` Tony Luck
2023-04-11 17:38 ` [PATCH v5 3/5] x86/mce: Introduce mce_handle_storm() to deal with begin/end of storms Tony Luck
2023-04-11 17:38 ` [PATCH v5 4/5] x86/mce: Move storm handling to core Tony Luck
2023-04-11 17:38 ` [PATCH v5 5/5] x86/mce: Handle AMD threshold interrupt storms Tony Luck
2023-06-16 18:27 ` [PATCH v6 0/4] Handle corrected machine check " Tony Luck
2023-06-16 18:27 ` [PATCH v6 1/4] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-06-16 18:27 ` [PATCH v6 2/4] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-06-23 12:09 ` Borislav Petkov
2023-06-23 15:40 ` Luck, Tony
2023-07-17 8:58 ` Borislav Petkov
2023-06-16 18:27 ` [PATCH v6 3/4] x86/mce: Handle AMD threshold interrupt storms Tony Luck
2023-06-23 14:45 ` Borislav Petkov
2023-06-23 15:54 ` Yazen Ghannam
2023-06-16 18:27 ` [PATCH v6 4/4] x86/mce: Handle Intel " Tony Luck
2023-07-18 21:08 ` [PATCH v7 0/3] Handle corrected machine check " Tony Luck
2023-07-18 21:08 ` [PATCH v7 1/3] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-07-18 21:08 ` [PATCH v7 2/3] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-09-19 17:44 ` Yazen Ghannam
2023-09-20 15:56 ` Yazen Ghannam
2023-09-20 16:09 ` Luck, Tony
2023-07-18 21:08 ` [PATCH v7 3/3] x86/mce: Handle Intel threshold interrupt storms Tony Luck
2023-09-19 17:59 ` Yazen Ghannam
2023-09-29 18:16 ` [PATCH v8 0/3] Handle corrected machine check " Tony Luck
2023-09-29 18:16 ` [PATCH v8 1/3] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-09-29 18:16 ` [PATCH v8 2/3] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-09-29 18:16 ` [PATCH v8 3/3] x86/mce: Handle Intel threshold interrupt storms Tony Luck
2023-10-02 17:57 ` [PATCH v8 0/3] Handle corrected machine check " Luck, Tony
2023-10-04 18:36 ` [PATCH v9 " Tony Luck
2023-10-04 18:36 ` [PATCH v9 1/3] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-10-04 18:36 ` [PATCH v9 2/3] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-10-11 9:11 ` kernel test robot
2023-10-11 15:16 ` Luck, Tony
2023-10-11 15:42 ` Feng Tang
2023-10-11 17:23 ` Luck, Tony
2023-10-12 5:36 ` Feng Tang
2023-10-12 5:56 ` Feng Tang
2023-10-12 2:35 ` Philip Li
2023-10-19 15:12 ` Borislav Petkov
2023-10-23 18:14 ` Tony Luck
2023-11-14 19:23 ` Borislav Petkov
2023-11-14 22:04 ` Tony Luck
2023-11-21 11:54 ` Borislav Petkov
2023-11-27 19:50 ` Tony Luck
2023-11-27 20:14 ` Tony Luck
2023-11-28 0:42 ` Tony Luck
2023-11-28 15:32 ` Yazen Ghannam
2023-12-14 16:58 ` Borislav Petkov
2023-12-14 18:03 ` Luck, Tony
2023-10-04 18:36 ` [PATCH v9 3/3] x86/mce: Handle Intel threshold interrupt storms Tony Luck
2023-11-15 19:54 ` [PATCH v10 0/3] Handle corrected machine check " Tony Luck
2023-11-15 19:54 ` [PATCH v10 1/3] x86/mce: Remove old CMCI storm mitigation code Tony Luck
2023-12-15 16:10 ` [tip: ras/core] " tip-bot2 for Tony Luck
2023-11-15 19:54 ` [PATCH v10 2/3] x86/mce: Add per-bank CMCI storm mitigation Tony Luck
2023-12-15 16:10 ` [tip: ras/core] " tip-bot2 for Tony Luck
2023-11-15 19:54 ` [PATCH v10 3/3] x86/mce: Handle Intel threshold interrupt storms Tony Luck
2023-12-15 16:10 ` [tip: ras/core] " tip-bot2 for Tony Luck
2023-12-15 17:21 ` Luck, Tony
2023-12-15 17:37 ` Borislav Petkov
2023-03-17 17:20 ` [PATCH v3 5/5] x86/mce: Handle AMD " Tony Luck
2022-04-06 6:35 ` [RFC PATCH " Smita Koralahalli
2022-04-06 22:44 ` Luck, Tony
2022-04-08 7:48 ` Koralahalli Channabasappa, Smita
2022-04-08 19:29 ` Luck, Tony
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6b6e53c5-8562-8ea0-befa-88c8b38663d6@amd.com \
--to=yazen.ghannam@amd.com \
--cc=Smita.KoralahalliChannabasappa@amd.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=linux-edac@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=patches@lists.linux.dev \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®