[RFC 0/3] powerpc64: Preempt count optimizations

[RFC 0/3] powerpc64: Preempt count optimizations

[Shrikanth Hegde]

This is an RFC and initial implementation of the idea which was discussed
in few threads. [1][2][3]

It is lightly tested, but given changes are involving the core api,
it definitely needs more testing. Also I need to investigate the
patches in more detail and likely changelogs needs update too.

Please go ahead and try to see if works/breaks for you.

Ideas in brief:

1. Move preempt count to PACA.
   preempt count in PACA is better as it make it direct access from r13 instead of
   obtaining from thread info. 

2. Fold need_resched into preempt count.
   Also it was noted that adopting similar style of folding need_resched
   into preempt count as ARM64 is better as it helps load-store archs and
   helps to avoid accessing thread info in fast paths.

3. Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS.
   While there enable the HAS_SEPARATE_PREEMPT_RESCHED_BITS which is a
   recent trick to fold the NMI bits into preempt count(29-32bits)
   instead of using separate per_cpu count.

[1] https://lore.kernel.org/all/e8efdade-161e-4efe-8bd3-abb12ad45dee@linux.ibm.com/
[2] https://lore.kernel.org/all/1eab63fc-b70d-41ef-b720-8ae77d8e56d8@linux.ibm.com/#t
[3] https://lore.kernel.org/all/20260805094219.GM687043@noisy.programming.kicks-ass.net/

Below are hackbench results.

Test                       	  Baseline    Patched   Gain
=============================================================
Hackbench process 10 groups         1.000x    0.991x   +0.88%
Hackbench process 20 groups         1.000x    0.972x   +2.81%
Hackbench process 30 groups         1.000x    0.976x   +2.37%
Hackbench process 40 groups         1.000x    0.979x   +2.11%
Hackbench process 50 groups         1.000x    0.980x   +2.05%
Hackbench process 60 groups         1.000x    0.982x   +1.79%
Hackbench thread 10  groups         1.000x    0.978x   +2.23%
Hackbench thread 20  groups         1.000x    0.971x   +2.93%
Hackbench process pipe 10 groups    1.000x    0.957x   +4.35%
Hackbench process pipe 20 groups    1.000x    0.977x   +2.32%
Hackbench process pipe 30 groups    1.000x    0.981x   +1.95%
Hackbench process pipe 40 groups    1.000x    0.981x   +1.93%
Hackbench process pipe 50 groups    1.000x    0.975x   +2.54%
Hackbench process pipe 60 groups    1.000x    1.024x   -2.37%
Hackbench thread pipe 10 groups     1.000x    0.970x   +3.04%
Hackbench thread pipe 20 groups     1.000x    0.981x   +1.89%

Things tested(no warnings/crash):
- Build and boot.
- hackbench run.
- daytrader run.
- kexec.

Shrikanth Hegde (3):
  powerpc64: move preempt count to paca
  powerpc64/preempt: Add need_resched bit inside preempt_count
  powerpc64/Kconfig: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

 arch/powerpc/Kconfig                   |   1 +
 arch/powerpc/include/asm/paca.h        |  12 +++
 arch/powerpc/include/asm/preempt.h     | 113 +++++++++++++++++++++++++
 arch/powerpc/include/asm/thread_info.h |  14 ++-
 arch/powerpc/kernel/paca.c             |   1 +
 arch/powerpc/kexec/core_64.c           |   2 +-
 6 files changed, 139 insertions(+), 4 deletions(-)
 create mode 100644 arch/powerpc/include/asm/preempt.h

-- 
2.52.0

[RFC 1/3] powerpc64: move preempt count to paca

[Shrikanth Hegde]

PowerPC uses generic implementation of preempt as of now.
That means preempt count in embedded in thread_info.

preempt_count = current_thread_info->preempt_count.

Current is first obtained from PACA. So this involves multiple
cachelines/pointers access. But preempt count is a per cpu storage
attribute indicating whether preemption can happen on
this CPU. per cpu storage has been used in other archs such
x86 already.

Powerpc64 architecture using PACA instead seems to be a more natural
fit. r13 is local paca pointer and it is always cacheline hot.

So getting preempt_count from paca means avoiding two deferrences.
It is worth doing that since preempt count is very high frequency
operation involved in various kernel primitives.

Given that after 7.0 powerpc has only preemptible kernels, makes it
even more relevant.

No functional change is intended.

Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
---
 arch/powerpc/include/asm/paca.h        |   1 +
 arch/powerpc/include/asm/preempt.h     | 103 +++++++++++++++++++++++++
 arch/powerpc/include/asm/thread_info.h |  14 +++-
 arch/powerpc/kernel/paca.c             |   1 +
 arch/powerpc/kexec/core_64.c           |   2 +-
 5 files changed, 117 insertions(+), 4 deletions(-)
 create mode 100644 arch/powerpc/include/asm/preempt.h

diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index 1d58da946739..0aa729c2faf5 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -164,6 +164,7 @@ struct paca_struct {
 	u64 saved_r1;			/* r1 save for RTAS calls or PM or EE=0 */
 	u64 saved_msr;			/* MSR saved here by enter_rtas */
 	u64 exit_save_r1;		/* Syscall/interrupt R1 save */
+	int preempt_count;
 #ifdef CONFIG_PPC_BOOK3E_64
 	u16 trap_save;			/* Used when bad stack is encountered */
 #endif
diff --git a/arch/powerpc/include/asm/preempt.h b/arch/powerpc/include/asm/preempt.h
new file mode 100644
index 000000000000..e2785d5a13e0
--- /dev/null
+++ b/arch/powerpc/include/asm/preempt.h
@@ -0,0 +1,103 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * PPC64 stores preempt count in PACA.
+ * PPC32 stores preempt count in thread_info.
+ */
+
+#ifndef __ASM_POWERPC_PREEMPT_H
+#define __ASM_POWERPC_PREEMPT_H
+
+#ifdef CONFIG_PPC64
+#include <linux/thread_info.h>
+#include <asm/paca.h>
+
+#define PREEMPT_ENABLED	(0)
+
+static __always_inline int preempt_count(void)
+{
+	return READ_ONCE(local_paca->preempt_count);
+}
+
+static __always_inline void preempt_count_set(int pc)
+{
+	WRITE_ONCE(local_paca->preempt_count, pc);
+}
+
+#define init_task_preempt_count(p) do { } while (0)
+#define init_idle_preempt_count(p, cpu) do { } while (0)
+
+static inline void set_preempt_need_resched(void)
+{
+}
+
+static inline void clear_preempt_need_resched(void)
+{
+}
+
+static inline bool test_preempt_need_resched(void)
+{
+	return false;
+}
+
+static __always_inline void __preempt_count_add(int val)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	pc += val;
+	WRITE_ONCE(local_paca->preempt_count, pc);
+}
+
+static __always_inline void __preempt_count_sub(int val)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	pc -= val;
+	WRITE_ONCE(local_paca->preempt_count, pc);
+}
+
+static inline int __preempt_count_add_return(int val)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	pc += val;
+	WRITE_ONCE(local_paca->preempt_count, pc);
+	return pc;
+}
+
+static inline int __preempt_count_sub_return(int val)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	pc -= val;
+	WRITE_ONCE(local_paca->preempt_count, pc);
+	return pc;
+}
+
+static inline bool __preempt_count_dec_and_test(void)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	WRITE_ONCE(local_paca->preempt_count, --pc);
+	return !pc && tif_need_resched();
+}
+
+static inline bool should_resched(int preempt_offset)
+{
+	int pc = READ_ONCE(local_paca->preempt_count);
+
+	return unlikely(pc == preempt_offset && tif_need_resched());
+}
+
+#ifdef CONFIG_PREEMPTION
+void preempt_schedule(void);
+void preempt_schedule_notrace(void);
+
+#define __preempt_schedule()		preempt_schedule()
+#define __preempt_schedule_notrace()	preempt_schedule_notrace()
+#endif /* CONFIG_PREEMPTION */
+
+#else /* !CONFIG_PPC64 */
+#include <asm-generic/preempt.h>
+#endif
+
+#endif
diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h
index 1e069a2e7ce8..89fd63442f7a 100644
--- a/arch/powerpc/include/asm/thread_info.h
+++ b/arch/powerpc/include/asm/thread_info.h
@@ -52,8 +52,9 @@
  * low level task data.
  */
 struct thread_info {
-	int		preempt_count;		/* 0 => preemptable,
-						   <0 => BUG */
+#ifndef CONFIG_PPC64
+	int		preempt_count;
+#endif
 #ifdef CONFIG_SMP
 	unsigned int	cpu;
 #endif
@@ -74,12 +75,19 @@ struct thread_info {
 	unsigned long	flags ____cacheline_aligned_in_smp;
 };
 
+#ifdef CONFIG_PPC64
+#define INIT_THREAD_INFO_PREEMPT_COUNT
+#else
+#define INIT_THREAD_INFO_PREEMPT_COUNT \
+	.preempt_count = INIT_PREEMPT_COUNT,
+#endif
+
 /*
  * macros/functions for gaining access to the thread information structure
  */
 #define INIT_THREAD_INFO(tsk)			\
 {						\
-	.preempt_count = INIT_PREEMPT_COUNT,	\
+	INIT_THREAD_INFO_PREEMPT_COUNT		\
 	.flags =	0,			\
 }
 
diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c
index 7502066c3c53..f19d057c37b3 100644
--- a/arch/powerpc/kernel/paca.c
+++ b/arch/powerpc/kernel/paca.c
@@ -200,6 +200,7 @@ void __init initialise_paca(struct paca_struct *new_paca, int cpu)
 	new_paca->hw_cpu_id = 0xffff;
 	new_paca->kexec_state = KEXEC_STATE_NONE;
 	new_paca->__current = &init_task;
+	new_paca->preempt_count = PREEMPT_DISABLED;
 	new_paca->data_offset = 0xfeeeeeeeeeeeeeeeULL;
 #ifdef CONFIG_PPC_64S_HASH_MMU
 	new_paca->slb_shadow_ptr = NULL;
diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
index 58c13a59b93b..f3954695e0b5 100644
--- a/arch/powerpc/kexec/core_64.c
+++ b/arch/powerpc/kexec/core_64.c
@@ -352,7 +352,7 @@ void default_machine_kexec(struct kimage *image)
 	 * XXX: the task struct will likely be invalid once we do the copy!
 	 */
 	current_thread_info()->flags = 0;
-	current_thread_info()->preempt_count = HARDIRQ_OFFSET;
+	WRITE_ONCE(local_paca->preempt_count, HARDIRQ_OFFSET);
 
 	/* We need a static PACA, too; copy this CPU's PACA over and switch to
 	 * it. Also poison per_cpu_offset and NULL lppaca to catch anyone using
-- 
2.52.0

[RFC 2/3] powerpc64/preempt: Add need_resched bit inside preempt_count

[Shrikanth Hegde]

Current implementation doesn't make use of the PREEMPT_NEED_RESCHED flag,
since this can interact badly with load/store architectures which
rely on the preempt_count word being unchanged across an interrupt.

On PPC64 which is 64 bit and the preempt count is only 32 bits wide,
one can simply pack it next to the resched flag and load the whole thing
in one go, so that a dec-and-test operation doesn't need to check tif.

[Adopted from arm64 implementation, including changelog bits]
Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
---
 arch/powerpc/include/asm/paca.h    | 13 +++++++-
 arch/powerpc/include/asm/preempt.h | 52 ++++++++++++++++++------------
 arch/powerpc/kexec/core_64.c       |  2 +-
 3 files changed, 44 insertions(+), 23 deletions(-)

diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h
index 0aa729c2faf5..a73e214bd6f3 100644
--- a/arch/powerpc/include/asm/paca.h
+++ b/arch/powerpc/include/asm/paca.h
@@ -164,7 +164,18 @@ struct paca_struct {
 	u64 saved_r1;			/* r1 save for RTAS calls or PM or EE=0 */
 	u64 saved_msr;			/* MSR saved here by enter_rtas */
 	u64 exit_save_r1;		/* Syscall/interrupt R1 save */
-	int preempt_count;
+	union {
+		u64		preempt_count;
+		struct {
+#ifdef __BIG_ENDIAN__
+			u32	need_resched;
+			u32	count;
+#else
+			u32	count;
+			u32	need_resched;
+#endif
+		} preempt;
+	};
 #ifdef CONFIG_PPC_BOOK3E_64
 	u16 trap_save;			/* Used when bad stack is encountered */
 #endif
diff --git a/arch/powerpc/include/asm/preempt.h b/arch/powerpc/include/asm/preempt.h
index e2785d5a13e0..d17c8acac7bd 100644
--- a/arch/powerpc/include/asm/preempt.h
+++ b/arch/powerpc/include/asm/preempt.h
@@ -8,19 +8,20 @@
 #define __ASM_POWERPC_PREEMPT_H
 
 #ifdef CONFIG_PPC64
-#include <linux/thread_info.h>
 #include <asm/paca.h>
 
-#define PREEMPT_ENABLED	(0)
+#define PREEMPT_NEED_RESCHED	BIT(32)
+#define PREEMPT_ENABLED	(PREEMPT_NEED_RESCHED)
 
 static __always_inline int preempt_count(void)
 {
-	return READ_ONCE(local_paca->preempt_count);
+	return READ_ONCE(local_paca->preempt.count);
 }
 
-static __always_inline void preempt_count_set(int pc)
+static __always_inline void preempt_count_set(u64 pc)
 {
-	WRITE_ONCE(local_paca->preempt_count, pc);
+	/* Preserve existing value of PREEMPT_NEED_RESCHED */
+	WRITE_ONCE(local_paca->preempt.count, pc);
 }
 
 #define init_task_preempt_count(p) do { } while (0)
@@ -28,64 +29,73 @@ static __always_inline void preempt_count_set(int pc)
 
 static inline void set_preempt_need_resched(void)
 {
+	WRITE_ONCE(local_paca->preempt.need_resched, 0);
 }
 
 static inline void clear_preempt_need_resched(void)
 {
+	WRITE_ONCE(local_paca->preempt.need_resched, 1);
 }
 
 static inline bool test_preempt_need_resched(void)
 {
-	return false;
+	return !READ_ONCE(local_paca->preempt.need_resched);
 }
 
 static __always_inline void __preempt_count_add(int val)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
+	u32 pc = READ_ONCE(local_paca->preempt.count);
 
 	pc += val;
-	WRITE_ONCE(local_paca->preempt_count, pc);
+	WRITE_ONCE(local_paca->preempt.count, pc);
 }
 
 static __always_inline void __preempt_count_sub(int val)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
+	u32 pc = READ_ONCE(local_paca->preempt.count);
 
 	pc -= val;
-	WRITE_ONCE(local_paca->preempt_count, pc);
+	WRITE_ONCE(local_paca->preempt.count, pc);
 }
 
 static inline int __preempt_count_add_return(int val)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
+	u32 pc = READ_ONCE(local_paca->preempt.count);
 
 	pc += val;
-	WRITE_ONCE(local_paca->preempt_count, pc);
+	WRITE_ONCE(local_paca->preempt.count, pc);
 	return pc;
 }
 
 static inline int __preempt_count_sub_return(int val)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
+	u32 pc = READ_ONCE(local_paca->preempt.count);
 
 	pc -= val;
-	WRITE_ONCE(local_paca->preempt_count, pc);
+	WRITE_ONCE(local_paca->preempt.count, pc);
 	return pc;
 }
 
 static inline bool __preempt_count_dec_and_test(void)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
-
-	WRITE_ONCE(local_paca->preempt_count, --pc);
-	return !pc && tif_need_resched();
+	u64 pc = READ_ONCE(local_paca->preempt_count);
+
+	/* Update only the count field, leaving need_resched unchanged */
+	WRITE_ONCE(local_paca->preempt.count, --pc);
+	/*
+	 * If we wrote back all zeroes, then we're preemptible and in
+	 * need of a reschedule. Otherwise, we need to reload the
+	 * preempt_count in case the need_resched flag was cleared by an
+	 * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE
+	 * pair.
+	 */
+	return !pc || !READ_ONCE(local_paca->preempt_count);
 }
 
 static inline bool should_resched(int preempt_offset)
 {
-	int pc = READ_ONCE(local_paca->preempt_count);
-
-	return unlikely(pc == preempt_offset && tif_need_resched());
+	u64 pc = READ_ONCE(local_paca->preempt_count);
+	return unlikely(pc == preempt_offset);
 }
 
 #ifdef CONFIG_PREEMPTION
diff --git a/arch/powerpc/kexec/core_64.c b/arch/powerpc/kexec/core_64.c
index f3954695e0b5..0f17c75cee28 100644
--- a/arch/powerpc/kexec/core_64.c
+++ b/arch/powerpc/kexec/core_64.c
@@ -352,7 +352,7 @@ void default_machine_kexec(struct kimage *image)
 	 * XXX: the task struct will likely be invalid once we do the copy!
 	 */
 	current_thread_info()->flags = 0;
-	WRITE_ONCE(local_paca->preempt_count, HARDIRQ_OFFSET);
+	WRITE_ONCE(local_paca->preempt.count, HARDIRQ_OFFSET);
 
 	/* We need a static PACA, too; copy this CPU's PACA over and switch to
 	 * it. Also poison per_cpu_offset and NULL lppaca to catch anyone using
-- 
2.52.0

[RFC 3/3] powerpc64/Kconfig: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

[Shrikanth Hegde]

Now there is separate need resched bit, enable the generic config
of HAS_SEPARATE_PREEMPT_RESCHED_BITS. This would make use of higher bits
of preempt count(29-32bits) for NMI bits.

Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com>
---
 arch/powerpc/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 143381d267ce..001bb2f3eec1 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -215,6 +215,7 @@ config PPC
 	select GENERIC_PCI_IOMAP		if PCI
 	select GENERIC_SMP_IDLE_THREAD
 	select HAS_IOPORT			if PCI
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS	if PPC64
 	select HAVE_ARCH_AUDITSYSCALL
 	select HAVE_ARCH_HUGE_VMALLOC		if HAVE_ARCH_HUGE_VMAP
 	select HAVE_ARCH_HUGE_VMAP		if PPC_RADIX_MMU || PPC_8xx
-- 
2.52.0