From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B0D0286D7E for ; Wed, 10 Dec 2025 15:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765382297; cv=none; b=cbyDVK2DKVPMcA8MOlI4sUBeDCW7P6Lt7EBlTorIG1V5erBw5Sh7CAt5v1gBpGzwr2I9euM3l01hv55QvMwpXKiBTE7z14TXTXh09YQ4craaKQ7GwNcjufv2hde13ERkX46xNhbxwfsHwxa2ZUviyA+u2u91uv0xSnfmJ19syo8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765382297; c=relaxed/simple; bh=hVJ4rBMGHS+mFzZ9rkThfG6sQO5hkQ0MJGnY0xf1jVM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZhUnoCK+f4OPiEqa4but4TAKy6s0xyvo/8Ku9HqQeUSwm65wmtsYxv8ZBke6xOe4wKNFAz32LtUIsQQEQVX9t7s7G85yHnOM2Aitgm2mQ+RR4XKSpLr4oqSK/gq/1+3+5IqclUYyaACKgWCtc4WlOfRjPZf9r7haKD2PtwewPy8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=LoMvf2CO; arc=none smtp.client-ip=209.85.219.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="LoMvf2CO" Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-88859a63001so13687066d6.0 for ; Wed, 10 Dec 2025 07:58:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1765382293; x=1765987093; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TvlpYkHXmfRzH67FR0h7OL1xiFOKV1y+D5XDurjoZt0=; b=LoMvf2COszqdE2m5+dmBNDyWJyPfT/OAi73FPvu05BQ3rAMQLFW6o2Wd3RulAV1RhC o5O5RD4TcexFCcn5xAY9mvib41ud+Nha+Otwl9pfs1LQep2B6QiXxULHicaGo35GIio9 09SokdQgPZIvyopXODbFmwF4/lWlTuo2DLt1MVmEOHmJsS4MSnAuYHf+2+z0hWzUl9m4 g2UmYOkW7IQBA6AnIOHq5F5TNtJYFuJ+rvFcx/PyHIsDE8RUjHfVZURh0jLxdxF/AKJD 0KDb+jeJ/e3J8xD23GMZUQQEkiLe34Z5apr8AORUQM+EvxH0ebzBZFK3YApTncdkwIwE nfJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765382293; x=1765987093; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TvlpYkHXmfRzH67FR0h7OL1xiFOKV1y+D5XDurjoZt0=; b=EZIq1v3P2NmxyhBm/0skv+0clZ0nZE56pmK69pJuDh1xeB3xaCaBFKoqJBAAhblS+7 ZxxovWY5bYwABkbjd7QwOo0BBx4yP+LoPXuUgFR3IqAJ+txfUPZ9xhW3SlT5mr9NYljr XasRNNzglJBgtb1OHJNeM8EqyinWMNMnwZq0BFq4TYCTEoBSoJESgQ3QSqXzNCst3aa+ ctB+0IcJiZtpkf5Jy2apN92RRERzU61yXloQ/ZlTSuCaz5brd0b3O0ZuyDzelKJ2WK9U b+eu7EcB2udNVMF1TI6dGC25Kq7HHbWReyI+80ye2TZ7ZKP4dGYzIy6Q3NRh5LjIaulP fQsw== X-Forwarded-Encrypted: i=1; AJvYcCXmhMX+HB0WCkVMFqGPzLFNirJ123nDCWR9uLeRUe7yoz677QS14jRyA5BHfZ/ggUkBWfl081XI4vRmM1U=@vger.kernel.org X-Gm-Message-State: AOJu0YxjjV71BKsvzfO24qJYCuSpJT8VeNunuhGVg4mQW751EtiEEqxi 7Nu2JBI/ARcVDQQskOU1Y3BrKrRrl6zOoLAh3LuAJ8y9D0qdjQ10MBMFUN3PbQoU6Vs= X-Gm-Gg: AY/fxX6+vUP6K4XDaC/f4pRA1w2MxigyXYR4VlxXwlGjcGYrU13QUjNf2iOCTiXE3P3 xKS0E95b4CNKsh6Rg/T/Xv/jBGhCcVM5lyE9SUV4dkUUMLOItD1O0EqsIfzmASftaUmUd4uy683 rHAnUBlFCH8aMxSDjpmiwUo+tXWoarjCpz5yqo/WaXt2F3wHZYdb3K+kgN8jz0r/ccT2P0FeiM9 nV6GMPvpcvnMFnPRZ1T9Q6HFcOvnPlrUFKZQpNUYakph323ARjhK4oZWQ7h0OLe8giu2q9IXHvU iacMG9lVp/GVtjN3nm1QTJOxRmoJK7OOBaYavcRHhZk/JHxH5rV9YaJmZ9VRxgL+FHgFg3Iwv32 nEWVQyjn62/xlPOjptIBkznFcQWefi0byE/7rQISAG6Zdgbc2at0movEY6i9m8asxCYEon2mUrM 2mzFG4AAzoNQ== X-Google-Smtp-Source: AGHT+IHPp6u1dKCE1eLfdy2yx3StYaQ/YgmmDKg/CTEOtV8FkL4YG52ZQcWJwyJ4F/NS2/vjb/Dprw== X-Received: by 2002:a05:6214:1309:b0:880:477f:88eb with SMTP id 6a1803df08f44-88863ad4edfmr44599636d6.66.1765382292890; Wed, 10 Dec 2025 07:58:12 -0800 (PST) Received: from localhost ([2603:7000:c01:2716:929a:4aff:fe16:c778]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8886ec28de0sm358876d6.4.2025.12.10.07.58.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Dec 2025 07:58:12 -0800 (PST) From: Johannes Weiner To: Peter Zijlstra , Suren Baghdasaryan , Ingo Molnar Cc: Chengming Zhou , Dietmar Eggemann , John Stultz , linux-kernel@vger.kernel.org Subject: [PATCH 2/2] sched: psi: use rq_clock() during task state changes Date: Wed, 10 Dec 2025 10:58:05 -0500 Message-ID: <20251210155805.752523-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20251210155805.752523-1-hannes@cmpxchg.org> References: <20251210155805.752523-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit In the hottest psi paths, the scheduler already caches the cpu_clock() call for the event in rq->clock. Now that the clocks between state changes and pressure aggregation don't need to be synchronized inside the seqcount section anymore, use the cheaper rq_clock(). Add update_rq_clock() calls to the few places where psi is entered without the rq already locked. schbench -n 0 (no think ops): Before: average rps: 204408.50 After: average rps: 204755.90 2.67% -0.54% [kernel.kallsyms] [k] sched_clock_noinstr Signed-off-by: Johannes Weiner --- kernel/sched/core.c | 3 ++- kernel/sched/psi.c | 12 +++++++----- kernel/sched/stats.h | 1 + 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 41ba0be16911..cc66415b85a1 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5519,9 +5519,10 @@ void sched_tick(void) rq_lock(rq, &rf); donor = rq->donor; + update_rq_clock(rq); + psi_account_irqtime(rq, donor, NULL); - update_rq_clock(rq); hw_pressure = arch_scale_hw_pressure(cpu_of(rq)); update_hw_load_avg(rq_clock_task(rq), rq, hw_pressure); diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c index 4b7bf8eb46c2..4a0a83f1d1dc 100644 --- a/kernel/sched/psi.c +++ b/kernel/sched/psi.c @@ -932,7 +932,7 @@ void psi_task_change(struct task_struct *task, int clear, int set) return; cpu = task_cpu(task); - now = cpu_clock(cpu); + now = rq_clock(cpu_rq(cpu)); psi_flags_change(task, clear, set); @@ -947,7 +947,7 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next, { struct psi_group *common = NULL; int cpu = task_cpu(prev); - u64 now = cpu_clock(cpu); + u64 now = rq_clock(cpu_rq(cpu)); psi_write_begin(cpu); @@ -1026,9 +1026,8 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st { int cpu = task_cpu(curr); struct psi_group_cpu *groupc; + u64 irq, now; s64 delta; - u64 irq; - u64 now; if (static_branch_likely(&psi_disabled) || !irqtime_enabled()) return; @@ -1046,7 +1045,7 @@ void psi_account_irqtime(struct rq *rq, struct task_struct *curr, struct task_st return; rq->psi_irq_time = irq; - now = cpu_clock(cpu); + now = rq_clock(rq); psi_write_begin(cpu); for_each_group(group, task_psi_group(curr)) { @@ -1089,6 +1088,7 @@ void psi_memstall_enter(unsigned long *flags) * race with CPU migration. */ rq = this_rq_lock_irq(&rf); + update_rq_clock(rq); current->in_memstall = 1; psi_task_change(current, 0, TSK_MEMSTALL | TSK_MEMSTALL_RUNNING); @@ -1119,6 +1119,7 @@ void psi_memstall_leave(unsigned long *flags) * race with CPU migration. */ rq = this_rq_lock_irq(&rf); + update_rq_clock(rq); current->in_memstall = 0; psi_task_change(current, TSK_MEMSTALL | TSK_MEMSTALL_RUNNING, 0); @@ -1187,6 +1188,7 @@ void cgroup_move_task(struct task_struct *task, struct css_set *to) } rq = task_rq_lock(task, &rf); + update_rq_clock(rq); /* * We may race with schedule() dropping the rq lock between diff --git a/kernel/sched/stats.h b/kernel/sched/stats.h index c903f1a42891..cb67a81e92b6 100644 --- a/kernel/sched/stats.h +++ b/kernel/sched/stats.h @@ -210,6 +210,7 @@ static inline void psi_ttwu_dequeue(struct task_struct *p) struct rq *rq; rq = __task_rq_lock(p, &rf); + update_rq_clock(rq); psi_task_change(p, p->psi_flags, 0); __task_rq_unlock(rq, p, &rf); } -- 2.52.0