From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 035515F863 for ; Mon, 22 Jan 2024 18:06:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705946790; cv=none; b=ZaMM3basFFaW2OJ7Fz9iQOwd9fUygy0FbjJPQZee8nlxF0L1y1AV6QCuNKkvmjXzSAn7g5NJv+VSRMsQqJq5TFmQ4YNChJSg6YPqAol8VPtfmpnQLDqObW4ELCoZ6J6NOjhVyoC7RpGu/ipjou2feEN0NDflY7EnDYdIUHyiJ54= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705946790; c=relaxed/simple; bh=cQLkUPBBLrRH/cRaXmUrTHiSWdeMuSt7rHG+1NHts0Q=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=gCEh/Pb1h0bKLLkjY4RHYu157nKii54a5iVRKPCS+y2Gryt3QgyLWpEUMDoqBsnLqyUUafq0/BzxLNgdfM4NAe1UubjN9rTg7aMxdWzOoyX9YS3BWVlpHvfDjEaG3S0RNKqLxp1OdDfojnIWCwQbR+uBWrpMxMaGsuuZmXHYs0I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4012D1476; Mon, 22 Jan 2024 10:07:13 -0800 (PST) Received: from [10.1.197.60] (eglon.cambridge.arm.com [10.1.197.60]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7C6963F5A1; Mon, 22 Jan 2024 10:06:24 -0800 (PST) Message-ID: Date: Mon, 22 Jan 2024 18:06:23 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v8 14/24] x86/resctrl: Allow resctrl_arch_rmid_read() to sleep Content-Language: en-GB To: babu.moger@amd.com, x86@kernel.org, linux-kernel@vger.kernel.org Cc: Fenghua Yu , Reinette Chatre , Thomas Gleixner , Ingo Molnar , Borislav Petkov , H Peter Anvin , shameerali.kolothum.thodi@huawei.com, D Scott Phillips OS , carl@os.amperecomputing.com, lcherian@marvell.com, bobo.shaobowang@huawei.com, tan.shaopeng@fujitsu.com, baolin.wang@linux.alibaba.com, Jamie Iles , Xin Hao , peternewman@google.com, dfustini@baylibre.com, amitsinght@marvell.com References: <20231215174343.13872-1-james.morse@arm.com> <20231215174343.13872-15-james.morse@arm.com> <290d6066-236d-4681-8dc5-8a380c878aeb@amd.com> From: James Morse In-Reply-To: <290d6066-236d-4681-8dc5-8a380c878aeb@amd.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi Babu, On 03/01/2024 19:43, Moger, Babu wrote: > On 12/15/23 11:43, James Morse wrote: >> MPAM's cache occupancy counters can take a little while to settle once >> the monitor has been configured. The maximum settling time is described >> to the driver via a firmware table. The value could be large enough >> that it makes sense to sleep. To avoid exposing this to resctrl, it >> should be hidden behind MPAM's resctrl_arch_rmid_read(). >> >> resctrl_arch_rmid_read() may be called via IPI meaning it is unable >> to sleep. In this case resctrl_arch_rmid_read() should return an error >> if it needs to sleep. This will only affect MPAM platforms where >> the cache occupancy counter isn't available immediately, nohz_full is >> in use, and there are no housekeeping CPUs in the necessary domain. >> >> There are three callers of resctrl_arch_rmid_read(): >> __mon_event_count() and __check_limbo() are both called from a >> non-migrateable context. mon_event_read() invokes __mon_event_count() >> using smp_call_on_cpu(), which adds work to the target CPUs workqueue. >> rdtgroup_mutex() is held, meaning this cannot race with the resctrl >> cpuhp callback. __check_limbo() is invoked via schedule_delayed_work_on() >> also adds work to a per-cpu workqueue. >> >> The remaining call is add_rmid_to_limbo() which is called in response >> to a user-space syscall that frees an RMID. This opportunistically >> reads the LLC occupancy counter on the current domain to see if the >> RMID is over the dirty threshold. This has to disable preemption to >> avoid reading the wrong domain's value. Disabling pre-emption here >> prevents resctrl_arch_rmid_read() from sleeping. >> >> add_rmid_to_limbo() walks each domain, but only reads the counter >> on one domain. If the system has more than one domain, the RMID will >> always be added to the limbo list. If the RMIDs usage was not over the >> threshold, it will be removed from the list when __check_limbo() runs. >> Make this the default behaviour. Free RMIDs are always added to the >> limbo list for each domain. >> >> The user visible effect of this is that a clean RMID is not available >> for re-allocation immediately after 'rmdir()' completes, this behaviour >> was never portable as it never happened on a machine with multiple >> domains. >> >> Removing this path allows resctrl_arch_rmid_read() to sleep if its called >> with interrupts unmasked. Document this is the expected behaviour, and >> add a might_sleep() annotation to catch changes that won't work on arm64. > Reviewed-by: Babu Moger Thanks! James