From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A20DC2BCF5 for ; Wed, 3 Dec 2025 02:20:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.168.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764728457; cv=none; b=B/iFg4b2VmuWLsqvp4ZO/VRfOVCQBRMTn2oO0NRicd5mZTPRO5WjyW+QVuTYtIBFQDoets8GGmla6U8ZD23k3xph7M4iGWVjodC8C/bL3VXUCvxzykz70RkLEjiTuTXxWlH2kbQFOK9EFheqktf9doS5AK633+wea2U86iZ656s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764728457; c=relaxed/simple; bh=Yq4E4oHs86aEtNdAgCEdLITL6+HOrW27JPr7Z6YrTBs=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=GMe1MReYWtb8i3saIPWqxph0yy+SMNTYzBF0yIdNJJTNbHbuh1w7WAUW8Wx5oVjTTo/wG5KK4+ShkMbhRU+Eu76HD0E7ty/Qs4EAxbywB22Gj1cyy/V0EaARTDEuU9Ee8OC9ltK3VXdnwWNEEnqf5WfEvUhPeIV/59dHtAlRDSc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com; spf=pass smtp.mailfrom=oss.qualcomm.com; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b=G/RXfCew; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b=IOlKM79F; arc=none smtp.client-ip=205.220.168.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b="G/RXfCew"; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="IOlKM79F" Received: from pps.filterd (m0279867.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 5B2GXOgx3714769 for ; Wed, 3 Dec 2025 02:20:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= kDByvDOazZtMwBsVROQkN35F4yNKa1o3XDTu60lntnE=; b=G/RXfCew71h8wajt 1ulzP/LWPjXjFbgAK7aAN7gwfbQwOebosMLoJWcKg7IdyoRqRvdqyIBFbPUw9Gqp bUhWvzd1kqYJdgIZTHFGcMy8FcHvNuh9n6kJNST30QGOP6w8Gxo696Tc6AGfFrQp hqhHABuJTSGnN0kjMb9kK8Jv+7MbnVJQsFIGU8hTwTsrm40CJXKfCpMeBeyCMEaN 55jPrj6Sq54T6rdX8FChivEdkpHSHugtSjRtyjoOGGnGxlbjhJo5Omb2rHMxNn7B PMyKoomZ25nSjj/hW4xqpqGw83TfZUaPIVyJz3BfgmmNPwHmj0EgWLnnyRmCWsn4 JbP/Kw== Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4at3r21g21-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Wed, 03 Dec 2025 02:20:53 +0000 (GMT) Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-340bc4ef67fso6970721a91.3 for ; Tue, 02 Dec 2025 18:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1764728453; x=1765333253; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=kDByvDOazZtMwBsVROQkN35F4yNKa1o3XDTu60lntnE=; b=IOlKM79FkjUQiKvrjH01YN84oxwL8I+HW0ccy/z3k8uaYYGYSmQpavNpljqzXWBV6m Ed2bBKgSEtfVu9cfipbWVSKXNNGH3ucUxOYftR1RWhCjOHkVcRImeAplYqF3oAg1zKwb OxPuUbmOur1YhJtPTckNk7jKj8SBe5UaoJ0bZQB89Pi2+6ftgwxhOHkuFkX9p03lqxE7 +/462iH0xAPr8EpcX+/ch9Fb9RNJ2h86WL+V1c+ztTpnnVKAFvLVb7TadSxjvwtjldNq JOW615D7HBbGn9JDlzC+bXQDkDHqwT+TVXwZ2rrK8+GcnIPz4XIQA75X3eeG5W6SKbm6 k5fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764728453; x=1765333253; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kDByvDOazZtMwBsVROQkN35F4yNKa1o3XDTu60lntnE=; b=BRk5O+h8aizp9C/2K+xHIkOp0COcbiKhS+XghQvZwH3iAR/cQGY7rvve5xwBRsA3Qz Ze28NHzGw86olvbFV+l+/79k53ElnS8/DNk7Y/fdDdlJcovs6BojPw+ylEXsUAOdBx5b V/D4quvf+fIW/08SiXXaa6ADwnzeHeqpuGmOEcxUkokiJiIGM4oGhQuDYumVZPVIRDQl kznOwVILuRfH+szPXX4PeY6jQLZIoh/8rUkcp0RLFaWHS/1killeZ+1xE2rgwCAxDCa+ Z0beb+Yxi5YHIZyuPCIdc6AjNzn890gekvkN7eYrLHOQR4mQCAez/c6cGZco18dw88x2 R5uw== X-Forwarded-Encrypted: i=1; AJvYcCX3xX1qgKVGdCPWYbI88ybeMfJAFHbmy1D9LvwXny1Omji1KDqvMigZ85hF0em21sl8ZTM+hhtpJ4GKH2k=@vger.kernel.org X-Gm-Message-State: AOJu0YybcPJBVM3ZXSQPj3cF9t92i0Gft+rk2EVcqMvfUbNfgJpTXXq5 LT8jyysnPpmUAJTX04nD9yq5n+HjRTO4v5QkO13DxqsuFUPVwedOy9ul5MK51oh6cmOzwy71tgk nk1ZGjgr0l0NDEGGj6KNjssf0ObrlozxTn4pi/N12Or67PERoiwqBzNQTTQ6r6Ze7oyI= X-Gm-Gg: ASbGncv8BvrA0NJxN33+Une5pKxjGRWMLSnXSgjiBBQ9B9SQjt6HsLxxaxtXhKyceuG jjm8zBq+vezPTx6q5HjFX+APYbky5HFztCwZ/X4hJabd2aZKdH/Vjb60iNE/+FyPuZmWSkAKPse D7KM0BLezJr7fmBrI98LUBtGCeUCNQZBL0p09lwiwUrxmEeTYYkPnPd+mwDMlbWvENBlU8r2E20 szxNV52n1feYdYd+3hqWg/prbD80TNy7AXSFgBPhzxCIOslEALwEM9Yw8eZtkpwBj7nBY6NehpJ R2PjAMuZ0bIgFJRUYNwpG0aYOAhcroxKdWE0lVRfr0gFLoTbowcLQrTgOvdU58Bhuuw3I827Hzg ECkF8pc0E2yaNjnWqg9upi2ba+9P+B6XwVUzBBL7tX9LHPApG3olNABRPUK27RrNu4l/jbJd0BA == X-Received: by 2002:a05:6a20:a124:b0:35e:d336:2818 with SMTP id adf61e73a8af0-363f5d4cf1amr1114136637.17.1764728452700; Tue, 02 Dec 2025 18:20:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IE/I4e3KRedmyngWONDAGx91EACDbwgLzxFY7mVQclJGntTqzy/bUaeZgR9UTaanW00g586ng== X-Received: by 2002:a05:6a20:a124:b0:35e:d336:2818 with SMTP id adf61e73a8af0-363f5d4cf1amr1114115637.17.1764728452160; Tue, 02 Dec 2025 18:20:52 -0800 (PST) Received: from [10.133.33.185] (tpe-colo-wan-fw-bordernet.qualcomm.com. [103.229.16.4]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-be5093b5b79sm16411183a12.25.2025.12.02.18.20.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 02 Dec 2025 18:20:51 -0800 (PST) Message-ID: <560a7dab-1f3e-4bda-bec3-3ed1723501cb@oss.qualcomm.com> Date: Wed, 3 Dec 2025 10:20:41 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] interconnect: Use rt_mutex for icc_bw_lock To: Georgi Djakov , Mike Tipton Cc: linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, quic_aiquny@quicinc.com, quic_okukatla@quicinc.com References: <20250506145159.1951159-1-quic_mdtipton@quicinc.com> Content-Language: en-US From: Yin Li In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Proofpoint-ORIG-GUID: 8lcAHYgfvqfT0DL0A1QLnEQvkp583cfM X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUxMjAzMDAxNiBTYWx0ZWRfX86XpKF8nM338 JO9DSnHAcib5tAlhPf27oQrAJEc89i2ujstsY6P8otWqVnB1d7HhNY4NxUffI9UFKr9VWm9VnMt NLP0k9WzKBI3sQmbmRUlQ6OOLdyg5aQUM7MmX0rE0Wu+PKI6IGrJXRuh2RKcA1+2bokrGtncsNX hUNxFTebYGr1PKNO9SOlIe/CkTXaiMQTL+MbR/sBYdzBpzZmBisSzw/UhhEroWOsuO8RyMILBzt +rC+XpBra5f+aQ0UahjWnwKCt+4+vxgb0pdwyizAKrTHJxjLAv5/RqoxraeqTemKb4nAooGlx5z HsAxvUKrbLQ+S1OXAj0/j3F+zs/JKNq67j/e6kgP2ad1mjwqGMPxdaxnln4vIUwbdQC2zNTcwWg 94Dg1sFHL7TXIwKlm9uAaYU03SxQ8g== X-Proofpoint-GUID: 8lcAHYgfvqfT0DL0A1QLnEQvkp583cfM X-Authority-Analysis: v=2.4 cv=c+WmgB9l c=1 sm=1 tr=0 ts=692f9e85 cx=c_pps a=UNFcQwm+pnOIJct1K4W+Mw==:117 a=nuhDOHQX5FNHPW3J6Bj6AA==:17 a=IkcTkHD0fZMA:10 a=wP3pNCr1ah4A:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=VwQbUJbxAAAA:8 a=COk6AnOGAAAA:8 a=1XWaLZrsAAAA:8 a=dDA9lvMg8NtrYOsEGZYA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=uKXjsCUrEbL0IQVhDsJ9:22 a=TjNXssC_j7lpFel5tvFf:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1121,Hydra:6.1.9,FMLib:17.12.100.49 definitions=2025-12-01_01,2025-11-27_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 adultscore=0 phishscore=0 bulkscore=0 impostorscore=0 malwarescore=0 suspectscore=0 spamscore=0 clxscore=1015 priorityscore=1501 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2510240001 definitions=main-2512030016 On 5/16/2025 11:50 PM, Georgi Djakov wrote: > Hi Mike, > > On 6.05.25 17:51, Mike Tipton wrote: >> The icc_set_bw() function is often used in latency sensitive paths to >> scale BW on a per-frame basis by high priority clients such as GPU and >> display. However, there are many low priority clients of icc_set_bw() as >> well. This can lead to priority inversion and unacceptable delays for >> the high priority clients. Which in the case of GPU and display can >> result in frame drops and visual glitches. > > Ok, so the issue we see is caused by lock contention, as we have many > clients and some of them try to do very aggressive scaling. > >> To prevent this priority inversion, switch to using rt_mutex for >> icc_bw_lock. This isn't needed for icc_lock since that's not used in the >> critical, latency-sensitive voting paths. > > If the issue does not occur anymore with this patch, then this is a good > sign, but we still need to get some numbers and put them in the commit > message. The RT mutexes add some overhead and complexity that could > increase latency for both uncontended and contended paths. I am curious > if there is any regression for the non-priority scenarios. Also if there > are many threads, the mutex cost itself could become a bottleneck. Hi Georgi, We constructed a priority inversion test scenario, which included multiple real-time threads with different priorities and CFS threads with different nice values ​​competing for a mutex, to verify the overhead of the RT thread acquiring the lock mutex. The maximum, minimum, and average of overhead were determined through 100 iterations of testing. Then replace the mutex with an rt-mutex and perform the same test, obtaining the overhead's max, min, and average through 100 loops. Calculate the change in average. Finally we can draw the conclusion: 1) In a scenario where the overhead of threads competing for a mutex is set to 5ms, using a mutex will result in an average overhead of 4127687ns for the tested rt threads to acquire the mutex. 2) After replacing the mutex with rt-mutex, the latency can be reduced to 2010555ns, which greatly improves the mutex overhead brought by priority inversion and reduces latency by about 50%. 3) Furthermore, to align with the user's given overhead of 40ms, the test case was modified to have a competing mutex thread overhead of 40ms, and the experiment was repeated, yielding similar results. After testing, the overhead of a single rt-mutex is approximately 937ns, and the overhead of a single mutex is approximately 520ns. The overhead of a single rt-mutex does indeed lead to more latency. However, in scenarios where multiple clients frequently access the interconnect API, the latency of using mutexes far outweighs the overhead added by rt-mutexes themselves. Compared to the performance improvement of rt-mutex in a thread-contention environment, the latency itself is perfectly acceptable. >> >> Signed-off-by: Mike Tipton >> --- >> >> Since the original patch was posted a couple years ago, we've continued >> to hit this for display and now for GPU as well. How frequently depends >> heavily on the specific chip, product, and use case. Different >> configurations hit it easier than others. But for both cases it results >> in obvious visual glitches. >> >> The paths being voted for (primarily DDR) are fundamentally shared >> between clients of all types and priority levels. We can't control their >> priorities, so aside from having those priorities inherited we're always >> subject to these sorts of inversions. >> >> The motivation isn't really for general performance improvement, but >> instead to fix the rare cases of visual glitches and artifacts. >> >> A similar patch was posted last year [1] to address similar problems. >> >> [1] https://lore.kernel.org/all/20240220074300.10805-1- >> wangrumeng@xiaomi.corp-partner.google.com/ >> >> Changes in v2: >> - Rebase onto linux-next. >> - Select RT_MUTEXES in Kconfig. >> - Only use rt_mutex for icc_bw_lock since now there are separate locks >>    and icc_lock isn't in the critical path. >> - Reword commit text. >> - Link to v1: https://lore.kernel.org/all/20220906191423.30109-1- >> quic_mdtipton@quicinc.com/ >> >>   drivers/interconnect/Kconfig |  1 + >>   drivers/interconnect/core.c  | 23 ++++++++++++----------- >>   2 files changed, 13 insertions(+), 11 deletions(-) >> >> diff --git a/drivers/interconnect/Kconfig b/drivers/interconnect/Kconfig >> index f2e49bd97d31..f6fd5f2d7d40 100644 >> --- a/drivers/interconnect/Kconfig >> +++ b/drivers/interconnect/Kconfig >> @@ -1,6 +1,7 @@ >>   # SPDX-License-Identifier: GPL-2.0-only >>   menuconfig INTERCONNECT >>       bool "On-Chip Interconnect management support" >> +    select RT_MUTEXES > > This pulls in unconditionally all the RT-mutex stuff, which some people > might not want (although today it's also selected by the I2C subsystem > for example). I am wondering if we should make it configurable with the > normal mutex being the default or just follow the i2c example... but > maybe we can decide this when we have some numbers. > Making locks configurable is not a common practice. We do not intend to make changes in this patch. -- Thx and BRs, Yin