mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ben Horgan <ben.horgan@arm.com>
To: Jinjie Ruan <ruanjinjie@huawei.com>,
	tony.luck@intel.com, reinette.chatre@intel.com,
	Dave.Martin@arm.com, james.morse@arm.com, babu.moger@amd.com,
	corbet@lwn.net, skhan@linuxfoundation.org, rdunlap@infradead.org,
	bp@alien8.de, tan.shaopeng@fujitsu.com, gshan@redhat.com,
	jic23@kernel.org, fenghuay@nvidia.com, zengheng4@huawei.com,
	x86@kernel.org, linux-kernel@vger.kernel.org,
	linux-doc@vger.kernel.org
Subject: Re: [PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes
Date: Thu, 19 Nov 2026 16:49:27 +0000	[thread overview]
Message-ID: <6dd79d09-576b-4086-ba5c-7f8663292a6e@arm.com> (raw)
In-Reply-To: <20261119094858.813874-6-ruanjinjie@huawei.com>

Hi Jinjie,

On 19/11/2026 09:48, Jinjie Ruan wrote:
> On MPAM the counter of an RMID is selected by the PARTIDs of the CLOSID
> that owned it. Without CDP, CLOSID c is PARTID c; with CDP it is
> PARTIDs 2c and 2c+1. An RMID that stays in limbo across an unmount and
> a mount that changes the CDP state is read through the PARTIDs of the
> new state, which are not the PARTIDs its cache lines are tagged with.
> 
> With 5 PARTIDs:
> 
>   mount                  CDP off, CLOSID c -> PARTID c
>   mkdir g1..g4           CLOSIDs 1..4
>   umount                 RMIDs enter limbo with CLOSIDs 1..4
>   mount -o cdp           CDP on, CLOSID c -> PARTIDs 2c, 2c+1
>   limbo handler          reads CLOSID 2 as PARTID 5 > PARTID_MAX 4
>                          MSC error interrupt, mpam_disable()
> 
> The other direction is worse because nothing reports it. If resctrl is
> mounted with CDP and mounted again without it, CLOSID c is read as
> PARTID c, which is in range, so the limbo handler silently tests the
> occupancy of the PARTID of a different CLOSID and frees or keeps the
> RMIDs based on it.
> 
> Where the RMID depends on the CLOSID, as on MPAM, release the RMIDs in
> limbo without reading them when a mount changes the CDP state, instead
> of restarting the limbo handler. Cache lines that still carry those
> RMIDs are accounted to the next user of each RMID. Where the RMID does
> not depend on the CLOSID, as on x86, the CDP state does not change how
> an RMID is read and the limbo handler is restarted as before.
> 
> Compare the CDP state of every resource, not just the L3: on MPAM, CDP
> enabled on any resource gives every CLOSID two PARTIDs in all MSCs, so
> mounting with -o cdpl2 alone also changes how the L3 monitors read a
> CLOSID, and cache levels beyond the L3 would too.
> 
> Reproduced on QEMU with the sequence above:
> 
>   mpam:__mpam_irq_handler: error irq from msc:1 'MSMONCFG_ID_RANGE', partid:5, pmg: 0, ris: 0
>   mpam:mpam_disable: MPAM disabled due to hardware error interrupt
> 
> With the change the sequence completes without an error.
> 
> Fixes: 6789fb99282c ("arm_mpam: resctrl: Add CDP emulation")
> Suggested-by: Ben Horgan <ben.horgan@arm.com>
> Suggested-by: Reinette Chatre <reinette.chatre@intel.com>
> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: <stable@vger.kernel.org>
> ---
> v6:
> - Drop resctrl_arch_mon_cdp_enabled() and compare the CDP state of
>   every resouce's cdp state, with a comment on why they are
>   checked (Reinette, Ben).
> - Restrict the release to the systems where the RMID depends on the
>   CLOSID, in the changelog and the comments, and keep the hardware IDs
>   out of the file system comments (Reinette).
> - Reword the limbo_cdp_enabled comment (Reinette).
> - Drop the stable version range, the Fixes tag already gives it.
> v5:
> - Split the change of the limbo handler lifetime into the previous
>   patch. The limbo handler now stops while resctrl is unmounted on every
>   architecture, and only the release of the RMIDs depends on
>   CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Drop the cqm_work_cpu changes. resctrl_limbo_resume() cancels the
>   worker before it queues it again or releases the RMIDs.
> - Move the rdt_disable_ctx() reordering to a separate patch.
> - Add resctrl_arch_mon_cdp_enabled() instead of comparing the CDP state
>   of L2 in the file system.
> - Rename limbo_cdp to limbo_cdp_enabled and document that it is only set
>   with CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Check QOS_L3_OCCUP_EVENT_ID first and drop the rmid_busy_llc check.
> - Use lockdep_assert_cpus_held() at the start of resctrl_limbo_resume().
> - Describe the reverse sequence, from CDP on to CDP off.
> - Link to v4: https://lore.kernel.org/all/20261105070934.2258613-4-ruanjinjie@huawei.com/
> v4:
> - Use resctrl_mounted instead of a separate limbo_paused flag.
> - Record the CDP state once when resctrl is mounted.
> - Add resctrl_limbo_resume(), called once resctrl_mounted is set.
> - Previous Link: https://lore.kernel.org/all/20261031094616.1691026-1-ruanjinjie@huawei.com/
> ---
>  fs/resctrl/monitor.c | 35 +++++++++++++++++++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c
> index ede1708b920e..2c8c60138768 100644
> --- a/fs/resctrl/monitor.c
> +++ b/fs/resctrl/monitor.c
> @@ -70,6 +70,13 @@ static u32 *closid_num_dirty_rmid;
>   */
>  static unsigned int rmid_limbo_count;
>  
> +/*
> + * @limbo_cdp_enabled - True if the RMIDs in limbo were placed there while CDP
> + *     was enabled. Only used when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
> + *     defined.
> + */
> +static bool limbo_cdp_enabled;
> +
>  /*
>   * @rmid_entry - The entry in the limbo and free lists.
>   */
> @@ -1044,12 +1051,15 @@ int setup_rmid_lru_list(void)
>   * cqm_resume_limbo_handler() - Restart the limbo handler after a mount.
>   *
>   * The limbo handler stops while resctrl is unmounted. Restart it on every
> - * domain that still has RMIDs in limbo.
> + * domain that still has RMIDs in limbo. Where the RMID depends on the CLOSID
> + * and the mount changed the RMID index a CLOSID is monitored with, the RMIDs
> + * in limbo can no longer be read: release them instead.
>   */
>  void cqm_resume_limbo_handler(void)
>  {
>  	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
>  	struct rdt_l3_mon_domain *d;
> +	bool release = false;
>  
>  	lockdep_assert_cpus_held();
>  	lockdep_assert_held(&rdtgroup_mutex);
> @@ -1057,6 +1067,24 @@ void cqm_resume_limbo_handler(void)
>  	if (!r->mon_capable || !resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
>  		return;
>  
> +	/*
> +	 * Where the RMID depends on the CLOSID, the RMIDs in limbo can only
> +	 * be read with the CDP state they were freed under. Check every
> +	 * resource because CDP enabled on any resource changes how every
> +	 * CLOSID is monitored, so CDP on another resource alone also changes
> +	 * how the L3 monitors are read.
> +	 */
> +	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> +		struct rdt_resource *res;
> +		bool cdp_enabled = false;
> +
> +		for_each_rdt_resource(res)
> +			cdp_enabled |= resctrl_arch_get_cdp_enabled(res->rid);
> +
> +		release = cdp_enabled != limbo_cdp_enabled;
> +		limbo_cdp_enabled = cdp_enabled;
> +	}
> +
>  	list_for_each_entry_rcu(d, &r->mon_domains, hdr.list,
>  				lockdep_is_cpus_held()) {
>  		if (!has_busy_rmid(d))
> @@ -1067,7 +1095,10 @@ void cqm_resume_limbo_handler(void)
>  		 * it is queued again on a CPU picked now.
>  		 */
>  		cancel_delayed_work(&d->cqm_limbo);
> -		cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
> +		if (release)
> +			release_limbo_rmids(d);

Rather than release all the rmids don't we want to check them all to see which are busy?

Thanks,

Ben

> +		else
> +			cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
>  	}
>  }
>  


  reply	other threads:[~2026-11-19 16:49 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  9:48 [PATCH v6 0/6] fs/resctrl: Fix some resctl bugs related to mpam Jinjie Ruan
2026-11-19  9:48 ` [PATCH v6 1/6] fs/resctrl: Report a temporary RMID shortage per CLOSID Jinjie Ruan
2026-11-19  9:48 ` [PATCH v6 2/6] fs/resctrl: Do not allocate a monitor context when releasing limbo RMIDs Jinjie Ruan
2026-11-19  9:48 ` [PATCH v6 3/6] fs/resctrl: Clear the CDP state after the cleanup Jinjie Ruan
2026-11-19 11:21   ` Ben Horgan
2026-11-19 12:15     ` Jinjie Ruan
2026-11-19  9:48 ` [PATCH v6 4/6] fs/resctrl: Stop the limbo handler while resctrl is unmounted Jinjie Ruan
2026-11-19 12:34   ` Ben Horgan
2026-11-19  9:48 ` [PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes Jinjie Ruan
2026-11-19 16:49   ` Ben Horgan [this message]
2026-11-19  9:48 ` [PATCH v6 6/6] fs/resctrl: Disallow monitor group move when the RMID index changes Jinjie Ruan
2026-11-19 16:57   ` Ben Horgan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6dd79d09-576b-4086-ba5c-7f8663292a6e@arm.com \
    --to=ben.horgan@arm.com \
    --cc=Dave.Martin@arm.com \
    --cc=babu.moger@amd.com \
    --cc=bp@alien8.de \
    --cc=corbet@lwn.net \
    --cc=fenghuay@nvidia.com \
    --cc=gshan@redhat.com \
    --cc=james.morse@arm.com \
    --cc=jic23@kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=rdunlap@infradead.org \
    --cc=reinette.chatre@intel.com \
    --cc=ruanjinjie@huawei.com \
    --cc=skhan@linuxfoundation.org \
    --cc=tan.shaopeng@fujitsu.com \
    --cc=tony.luck@intel.com \
    --cc=x86@kernel.org \
    --cc=zengheng4@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®