mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ben Horgan <ben.horgan@arm.com>
To: Fenghua Yu <fenghuay@nvidia.com>
Cc: amitsinght@marvell.com, baisheng.gao@unisoc.com,
	baolin.wang@linux.alibaba.com, carl@os.amperecomputing.com,
	dave.martin@arm.com, david@kernel.org, dfustini@baylibre.com,
	gshan@redhat.com, james.morse@arm.com,
	jonathan.cameron@huawei.com, kobak@nvidia.com,
	lcherian@marvell.com, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, peternewman@google.com,
	punit.agrawal@oss.qualcomm.com, quic_jiles@quicinc.com,
	reinette.chatre@intel.com, rohit.mathew@arm.com,
	scott@os.amperecomputing.com, sdonthineni@nvidia.com,
	tan.shaopeng@fujitsu.com, xhao@linux.alibaba.com,
	catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net,
	maz@kernel.org, oupton@kernel.org, joey.gouly@arm.com,
	suzuki.poulose@arm.com, kvmarm@lists.linux.dev,
	zengheng4@huawei.com, linux-doc@vger.kernel.org,
	Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
Subject: Re: [PATCH v5 38/41] arm_mpam: Add workaround for T241-MPAM-4
Date: Mon, 2 Mar 2026 17:11:05 +0000	[thread overview]
Message-ID: <3e26371d-21b5-435e-9f31-7187466c57a2@arm.com> (raw)
In-Reply-To: <7e496cb6-ab31-4a89-9630-0fa2e6b3607b@nvidia.com>

Hi Fenghua,

On 3/1/26 17:28, Fenghua Yu wrote:
> Hi, Ben,
> 
> On 2/24/26 09:57, Ben Horgan wrote:
>> From: Shanker Donthineni <sdonthineni@nvidia.com>
>>
>> In the T241 implementation of memory-bandwidth partitioning, in the
>> absence
>> of contention for bandwidth, the minimum bandwidth setting can affect the
>> amount of achieved bandwidth. Specifically, the achieved bandwidth in the
>> absence of contention can settle to any value between the values of
>> MPAMCFG_MBW_MIN and MPAMCFG_MBW_MAX.  Also, if MPAMCFG_MBW_MIN is set
>> zero (below 0.78125%), once a core enters a throttled state, it will
>> never
>> leave that state.
>>
>> The first issue is not a concern if the MPAM software allows to program
>> MPAMCFG_MBW_MIN through the sysfs interface. This patch ensures program
>> MBW_MIN=1 (0.78125%) whenever MPAMCFG_MBW_MIN=0 is programmed.
>>
>> In the scenario where the resctrl doesn't support the MBW_MIN
>> interface via
>> sysfs, to achieve bandwidth closer to MBW_MAX in the absence of
>> contention,
>> software should configure a relatively narrow gap between MBW_MIN and
>> MBW_MAX. The recommendation is to use a 5% gap to mitigate the problem.
>>
>> Clear the feature MBW_MIN feature from the class to ensure we don't
>> accidentally change behaviour when resctrl adds support for a MBW_MIN
>> interface.
>>
>> Tested-by: Gavin Shan <gshan@redhat.com>
>> Tested-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
>> Reviewed-by: Shaopeng Tan <tan.shaopeng@jp.fujitsu.com>
>> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
>> Signed-off-by: James Morse <james.morse@arm.com>
>> Signed-off-by: Ben Horgan <ben.horgan@arm.com>
> 
> Reviewed-by: Fenghua Yu <fenghuay@nvidia.com>
> 
> This patch itself is good.
> 
> Please check the following comments.
> 
>> ---
>> [ morse: Added as second quirk, adapted to use the new intermediate
>> values
>> in mpam_extend_config() ]
>>
>> Changes since rfc:
>> MPAM_IIDR_NVIDIA_T421 -> MPAM_IIDR_NVIDIA_T241
>> Handling when reset_mbw_min is set
>>
>> Changes since v3:
>> Move the 5% gap policy back here
>> Clear mbw_min feature in class
>> ---
>>   Documentation/arch/arm64/silicon-errata.rst |  2 +
>>   drivers/resctrl/mpam_devices.c              | 50 +++++++++++++++++++--
>>   drivers/resctrl/mpam_internal.h             |  1 +
>>   3 files changed, 50 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/arch/arm64/silicon-errata.rst b/
>> Documentation/arch/arm64/silicon-errata.rst
>> index a65620f98e3a..a4b246655e37 100644
>> --- a/Documentation/arch/arm64/silicon-errata.rst
>> +++ b/Documentation/arch/arm64/silicon-errata.rst
>> @@ -249,6 +249,8 @@ stable kernels.
>>   +----------------+-----------------+-----------------
>> +-----------------------------+
>>   | NVIDIA         | T241 MPAM       | T241-MPAM-1     | N/
>> A                         |
>>   +----------------+-----------------+-----------------
>> +-----------------------------+
>> +| NVIDIA         | T241 MPAM       | T241-MPAM-4     | N/
>> A                         |
>> ++----------------+-----------------+-----------------
>> +-----------------------------+
>>   +----------------+-----------------+-----------------
>> +-----------------------------+
>>   | Freescale/NXP  | LS2080A/LS1043A | A-008585        |
>> FSL_ERRATUM_A008585         |
>>   +----------------+-----------------+-----------------
>> +-----------------------------+
>> diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/
>> mpam_devices.c
>> index 08cb080592d9..8f44e9dee207 100644
>> --- a/drivers/resctrl/mpam_devices.c
>> +++ b/drivers/resctrl/mpam_devices.c
>> @@ -679,6 +679,12 @@ static const struct mpam_quirk mpam_quirks[] = {
>>       .iidr_mask  = MPAM_IIDR_MATCH_ONE,
>>       .workaround = T241_SCRUB_SHADOW_REGS,
>>       },
>> +    {
>> +    /* NVIDIA t241 erratum T241-MPAM-4 */
>> +    .iidr       = MPAM_IIDR_NVIDIA_T241,
>> +    .iidr_mask  = MPAM_IIDR_MATCH_ONE,
>> +    .workaround = T241_FORCE_MBW_MIN_TO_ONE,
>> +    },
>>       { NULL } /* Sentinel */
>>   };
>>   @@ -1464,6 +1470,31 @@ static void
>> mpam_quirk_post_config_change(struct mpam_msc_ris *ris, u16 partid,
>>           mpam_apply_t241_erratum(ris, partid);
>>   }
>>   +static u16 mpam_wa_t241_force_mbw_min_to_one(struct mpam_props *props)
>> +{
>> +    u16 max_hw_value, min_hw_granule, res0_bits;
>> +
>> +    res0_bits = 16 - props->bwa_wd;
>> +    max_hw_value = ((1 << props->bwa_wd) - 1) << res0_bits;
>> +    min_hw_granule = ~max_hw_value;
>> +
>> +    return min_hw_granule + 1;
>> +}
>> +
>> +static u16 mpam_wa_t241_calc_min_from_max(struct mpam_config *cfg)
>> +{
>> +    u16 val = 0;
>> +
>> +    if (mpam_has_feature(mpam_feat_mbw_max, cfg)) {
> 
> But the problem is mpam_feat_mbw_max feature is NOT set in cfg.
> 
>> +        u16 delta = ((5 * MPAMCFG_MBW_MAX_MAX) / 100) - 1;
>> +
>> +        if (cfg->mbw_max > delta)
>> +            val = cfg->mbw_max - delta;
>> +    }
>> +
>> +    return val;
> 
> So 0 is always returned.
> 
> The workaround will set mbw_min as 1% which is too small and will cause
> performance degradation, e.g. about 20% degradation on some benchmarks.
> 
> This patch itself doesn't have any issue.
> 
> The issue is the mbw_max feature bit in cfg is not set.

This is intended behaviour as the reset is done independently
from the value set in the config. The value is there so that 
resctrl can display the expected values.

> This is a legacy issue, not introduced by this patch set.
> > Here is a fix patch for the issue:
> https://lore.kernel.org/lkml/20260301171829.1357886-1-
> fenghuay@nvidia.com/T/#u

I've commented on that patch. I think it's best to fix it in the context
of the erratum.

Does the below solve your performance problems?

diff --git a/drivers/resctrl/mpam_devices.c b/drivers/resctrl/mpam_devices.c
index 236f78ab9163..60d3d3e2193f 100644
--- a/drivers/resctrl/mpam_devices.c
+++ b/drivers/resctrl/mpam_devices.c
@@ -1515,16 +1515,20 @@ static u16 mpam_wa_t241_force_mbw_min_to_one(struct mpam_props *props)
        return min_hw_granule + 1;
 }
 
-static u16 mpam_wa_t241_calc_min_from_max(struct mpam_config *cfg)
+static u16 mpam_wa_t241_calc_min_from_max(struct mpam_props *props,
+                                         struct mpam_config *cfg)
 {
        u16 val = 0;
+       u16 max;
+       u16 delta = ((5 * MPAMCFG_MBW_MAX_MAX) / 100) - 1;
 
-       if (mpam_has_feature(mpam_feat_mbw_max, cfg)) {
-               u16 delta = ((5 * MPAMCFG_MBW_MAX_MAX) / 100) - 1;
+       if (mpam_has_feature(mpam_feat_mbw_max, cfg))
+               max = cfg->mbw_max;
+       else
+               max = GENMASK(15, 16 - cprops->bwa_wd);
 
-               if (cfg->mbw_max > delta)
-                       val = cfg->mbw_max - delta;
-       }
+       if (max > delta)
+               val = max - delta;
 
        return val;
 }
@@ -1577,9 +1581,8 @@ static void mpam_reprogram_ris_partid(struct mpam_msc_ris *ris, u16 partid,
                if (mpam_has_quirk(T241_FORCE_MBW_MIN_TO_ONE, msc)) {
                        u16 min = mpam_wa_t241_force_mbw_min_to_one(rprops);
 
-                       val = mpam_wa_t241_calc_min_from_max(cfg);
-                       if (val < min)
-                               val = min;
+                       val = mpam_wa_t241_calc_min_from_max(rprops, cfg);
+                       val = max(val, min);
                }
 
                mpam_write_partsel_reg(msc, MBW_MIN, val);

> 
> If the fix patch is good, could you please add it into the next version
> of this series?
> 
> Thanks.
> 
> -Fenghua

Thanks,

Ben


  reply	other threads:[~2026-03-02 17:11 UTC|newest]

Thread overview: 73+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-02-24 17:56 [PATCH v5 00/41] arm_mpam: Add KVM/arm64 and resctrl glue code Ben Horgan
2026-02-24 17:56 ` [PATCH v5 01/41] arm64/sysreg: Add MPAMSM_EL1 register Ben Horgan
2026-02-24 17:56 ` [PATCH v5 02/41] KVM: arm64: Preserve host MPAM configuration when changing traps Ben Horgan
2026-03-02 17:52   ` Marc Zyngier
2026-02-24 17:56 ` [PATCH v5 03/41] KVM: arm64: Make MPAMSM_EL1 accesses UNDEF Ben Horgan
2026-03-02 17:54   ` Marc Zyngier
2026-02-24 17:56 ` [PATCH v5 04/41] arm64: mpam: Context switch the MPAM registers Ben Horgan
2026-02-24 17:56 ` [PATCH v5 05/41] arm64: mpam: Re-initialise MPAM regs when CPU comes online Ben Horgan
2026-02-24 17:56 ` [PATCH v5 06/41] arm64: mpam: Drop the CONFIG_EXPERT restriction Ben Horgan
2026-03-09  6:42   ` Gavin Shan
2026-02-24 17:56 ` [PATCH v5 07/41] arm64: mpam: Advertise the CPUs MPAM limits to the driver Ben Horgan
2026-03-09  6:43   ` Gavin Shan
2026-02-24 17:56 ` [PATCH v5 08/41] arm64: mpam: Add cpu_pm notifier to restore MPAM sysregs Ben Horgan
2026-02-24 17:56 ` [PATCH v5 09/41] arm64: mpam: Initialise and context switch the MPAMSM_EL1 register Ben Horgan
2026-02-24 17:56 ` [PATCH v5 10/41] arm64: mpam: Add helpers to change a task or cpu's MPAM PARTID/PMG values Ben Horgan
2026-03-09  6:44   ` Gavin Shan
2026-02-24 17:56 ` [PATCH v5 11/41] KVM: arm64: Force guest EL1 to use user-space's partid configuration Ben Horgan
2026-03-02 17:58   ` Marc Zyngier
2026-03-09  6:45   ` Gavin Shan
2026-02-24 17:56 ` [PATCH v5 12/41] KVM: arm64: Use kernel-space partid configuration for hypercalls Ben Horgan
2026-03-02 18:15   ` Marc Zyngier
2026-03-03 16:33     ` Ben Horgan
2026-03-13  9:43       ` Ben Horgan
2026-02-24 17:56 ` [PATCH v5 13/41] arm_mpam: resctrl: Add boilerplate cpuhp and domain allocation Ben Horgan
2026-03-10  6:17   ` Gavin Shan
2026-03-10 10:34     ` Ben Horgan
2026-02-24 17:56 ` [PATCH v5 14/41] arm_mpam: resctrl: Pick the caches we will use as resctrl resources Ben Horgan
2026-02-24 17:56 ` [PATCH v5 15/41] arm_mpam: resctrl: Implement resctrl_arch_reset_all_ctrls() Ben Horgan
2026-02-25 11:03   ` Jonathan Cameron
2026-02-24 17:56 ` [PATCH v5 16/41] arm_mpam: resctrl: Add resctrl_arch_get_config() Ben Horgan
2026-02-24 17:56 ` [PATCH v5 17/41] arm_mpam: resctrl: Implement helpers to update configuration Ben Horgan
2026-02-24 17:56 ` [PATCH v5 18/41] arm_mpam: resctrl: Add plumbing against arm64 task and cpu hooks Ben Horgan
2026-02-24 17:56 ` [PATCH v5 19/41] arm_mpam: resctrl: Add CDP emulation Ben Horgan
2026-02-25  6:25   ` Zeng Heng
2026-02-24 17:56 ` [PATCH v5 20/41] arm_mpam: resctrl: Convert to/from MPAMs fixed-point formats Ben Horgan
2026-02-24 17:57 ` [PATCH v5 21/41] arm_mpam: resctrl: Add kunit test for control format conversions Ben Horgan
2026-02-24 17:57 ` [PATCH v5 22/41] arm_mpam: resctrl: Add rmid index helpers Ben Horgan
2026-02-24 17:57 ` [PATCH v5 23/41] arm_mpam: resctrl: Add kunit test for rmid idx conversions Ben Horgan
2026-02-24 17:57 ` [PATCH v5 24/41] arm_mpam: resctrl: Wait for cacheinfo to be ready Ben Horgan
2026-02-24 17:57 ` [PATCH v5 25/41] arm_mpam: resctrl: Add support for 'MB' resource Ben Horgan
2026-02-24 17:57 ` [PATCH v5 26/41] arm_mpam: resctrl: Add monitor initialisation and domain boilerplate Ben Horgan
2026-02-25 11:14   ` Jonathan Cameron
2026-02-26  3:47   ` Zeng Heng
2026-02-26 10:26     ` Ben Horgan
2026-02-27  3:01       ` Zeng Heng
2026-02-24 17:57 ` [PATCH v5 27/41] arm_mpam: resctrl: Add support for csu counters Ben Horgan
2026-02-24 17:57 ` [PATCH v5 28/41] arm_mpam: resctrl: Pick classes for use as mbm counters Ben Horgan
2026-02-24 17:57 ` [PATCH v5 29/41] arm_mpam: resctrl: Pre-allocate free running monitors Ben Horgan
2026-02-24 17:57 ` [PATCH v5 30/41] arm_mpam: resctrl: Allow resctrl to allocate monitors Ben Horgan
2026-02-24 17:57 ` [PATCH v5 31/41] arm_mpam: resctrl: Add resctrl_arch_rmid_read() and resctrl_arch_reset_rmid() Ben Horgan
2026-03-07  9:29   ` Zeng Heng
2026-03-09 16:30     ` Ben Horgan
2026-03-10  3:23       ` Zeng Heng
2026-02-24 17:57 ` [PATCH v5 32/41] arm_mpam: resctrl: Update the rmid reallocation limit Ben Horgan
2026-02-24 17:57 ` [PATCH v5 33/41] arm_mpam: resctrl: Add empty definitions for assorted resctrl functions Ben Horgan
2026-02-24 17:57 ` [PATCH v5 34/41] arm64: mpam: Select ARCH_HAS_CPU_RESCTRL Ben Horgan
2026-02-24 17:57 ` [PATCH v5 35/41] arm_mpam: resctrl: Call resctrl_init() on platforms that can support resctrl Ben Horgan
2026-02-24 17:57 ` [PATCH v5 36/41] arm_mpam: Add quirk framework Ben Horgan
2026-02-24 17:57 ` [PATCH v5 37/41] arm_mpam: Add workaround for T241-MPAM-1 Ben Horgan
2026-02-24 17:57 ` [PATCH v5 38/41] arm_mpam: Add workaround for T241-MPAM-4 Ben Horgan
2026-03-01 17:28   ` Fenghua Yu
2026-03-02 17:11     ` Ben Horgan [this message]
2026-03-09 17:39       ` Fenghua Yu
2026-03-10 11:26         ` Ben Horgan
2026-02-24 17:57 ` [PATCH v5 39/41] arm_mpam: Add workaround for T241-MPAM-6 Ben Horgan
2026-02-24 17:57 ` [PATCH v5 40/41] arm_mpam: Quirk CMN-650's CSU NRDY behaviour Ben Horgan
2026-02-24 17:57 ` [PATCH v5 41/41] arm64: mpam: Add initial MPAM documentation Ben Horgan
2026-02-25 11:01   ` Jonathan Cameron
2026-02-25 21:10 ` [PATCH v5 00/41] arm_mpam: Add KVM/arm64 and resctrl glue code Ben Horgan
2026-02-27 17:04   ` Catalin Marinas
2026-02-26  7:34 ` Zeng Heng
2026-03-03 20:18 ` Punit Agrawal
2026-03-04  9:42   ` Ben Horgan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3e26371d-21b5-435e-9f31-7187466c57a2@arm.com \
    --to=ben.horgan@arm.com \
    --cc=amitsinght@marvell.com \
    --cc=baisheng.gao@unisoc.com \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=carl@os.amperecomputing.com \
    --cc=catalin.marinas@arm.com \
    --cc=corbet@lwn.net \
    --cc=dave.martin@arm.com \
    --cc=david@kernel.org \
    --cc=dfustini@baylibre.com \
    --cc=fenghuay@nvidia.com \
    --cc=gshan@redhat.com \
    --cc=james.morse@arm.com \
    --cc=joey.gouly@arm.com \
    --cc=jonathan.cameron@huawei.com \
    --cc=kobak@nvidia.com \
    --cc=kvmarm@lists.linux.dev \
    --cc=lcherian@marvell.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=maz@kernel.org \
    --cc=oupton@kernel.org \
    --cc=peternewman@google.com \
    --cc=punit.agrawal@oss.qualcomm.com \
    --cc=quic_jiles@quicinc.com \
    --cc=reinette.chatre@intel.com \
    --cc=rohit.mathew@arm.com \
    --cc=scott@os.amperecomputing.com \
    --cc=sdonthineni@nvidia.com \
    --cc=suzuki.poulose@arm.com \
    --cc=tan.shaopeng@fujitsu.com \
    --cc=tan.shaopeng@jp.fujitsu.com \
    --cc=will@kernel.org \
    --cc=xhao@linux.alibaba.com \
    --cc=zengheng4@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®