[PATCH v2 0/7] ARM: Broadcom Brahma-B15 readahead cache support

[PATCH v2 0/7] ARM: Broadcom Brahma-B15 readahead cache support

[Florian Fainelli]

Hi all,

This patch series adds support for the Broadcom Brahma-B15 readahead cache.
I submitted that patch series a couple of years ago, and then slept on it so
here is another stab at it.

Note that we did not implement this cache as a version of an outer cache
for several reasons:

- we initially thought we needed to intercept flush_icache_all and
  flush_kern_cache_louis but upon further inspection we convinced ourselves
  this is no longer needed, still, flush_cache_all() needs special handling
  here and needs to be wrapped around

- the outer cache does not allow differentiating a DMA transfer direction
  this is a readahead cache, so it does not participate in writes, flushing
  it during reads *and* writes kills the performance completely

- finally, most operations that outer_cache cares about are on MVA, which
  is transparent to the readahead cache here

Changes in v2:

- clarify that the read-ahead caches does invalidates on writes (IOW) based
  on Russell's feedback

Florian Fainelli (7):
  ARM: v7: allow setting different cache functions
  ARM: Add Broadcom Brahma-B15 readahead cache support
  ARM: Hook B15 readahead cache functions based on processor
  ARM: B15: Add CPU hotplug awareness
  ARM: B15: Add suspend/resume hooks
  ARM: B15: Register reboot notifier for KEXEC
  MAINTAINERS: Update brcmstb entries to cover B15 code

 MAINTAINERS                                   |   2 +
 arch/arm/include/asm/glue-cache.h             |   4 +
 arch/arm/include/asm/hardware/cache-b15-rac.h |  10 +
 arch/arm/mm/Kconfig                           |   8 +
 arch/arm/mm/Makefile                          |   1 +
 arch/arm/mm/cache-b15-rac.c                   | 360 ++++++++++++++++++++++++++
 arch/arm/mm/cache-v7.S                        |  21 ++
 arch/arm/mm/proc-v7.S                         |   6 +-
 include/linux/cpuhotplug.h                    |   2 +
 9 files changed, 411 insertions(+), 3 deletions(-)
 create mode 100644 arch/arm/include/asm/hardware/cache-b15-rac.h
 create mode 100644 arch/arm/mm/cache-b15-rac.c

-- 
2.9.3

[PATCH v2 1/7] ARM: v7: allow setting different cache functions

[Florian Fainelli]

In preparation for adding support for the Broadcom Brahma-B15 read-ahead
cache which requires a different set of cache functions, allow the
__v7_proc macro to override the cache_fns settings, and default to
v7_cache_fns unless specified otherwise.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/mm/proc-v7.S | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index 01d64c0b2563..812ac861cd23 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -567,7 +567,7 @@ __v7_setup_stack:
 	/*
 	 * Standard v7 proc info content
 	 */
-.macro __v7_proc name, initfunc, mm_mmuflags = 0, io_mmuflags = 0, hwcaps = 0, proc_fns = v7_processor_functions
+.macro __v7_proc name, initfunc, mm_mmuflags = 0, io_mmuflags = 0, hwcaps = 0, proc_fns = v7_processor_functions, cache_fns = v7_cache_fns
 	ALT_SMP(.long	PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_SECT_AP_READ | \
 			PMD_SECT_AF | PMD_FLAGS_SMP | \mm_mmuflags)
 	ALT_UP(.long	PMD_TYPE_SECT | PMD_SECT_AP_WRITE | PMD_SECT_AP_READ | \
@@ -583,7 +583,7 @@ __v7_setup_stack:
 	.long	\proc_fns
 	.long	v7wbi_tlb_fns
 	.long	v6_user_fns
-	.long	v7_cache_fns
+	.long	\cache_fns
 .endm
 
 #ifndef CONFIG_ARM_LPAE
-- 
2.9.3

[PATCH v2 2/7] ARM: Add Broadcom Brahma-B15 readahead cache support

[Florian Fainelli]

This patch adds support for the Broadcom Brahma-B15 CPU readahead cache
controller. This cache controller sits between the L2 and the memory bus
and its purpose is to provide a friendler burst size towards the DDR
interface than the native cache line size.

The readahead cache is mostly transparent, except for
flush_kern_cache_all, which is precisely what we are overriding here.

The readahead cache only intercepts reads, and does invalidate on
writes (IOW), as such, some data can remain stale in any of its buffers, such
that we need to flush it, which is an operation that needs to happen in
a particular order:

- disable the readahead cache
- flush it
- call the appropriate cache-v7.S function
- re-enable

This patch tries to minimize the impact to the cache-v7.S file by only
providing a stub in case CONFIG_CACHE_B15_RAC is enabled (default for
ARCH_BRCMSTB since it is the current user).

Signed-off-by: Alamy Liu <alamyliu@broadcom.com>
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/include/asm/glue-cache.h             |   4 +
 arch/arm/include/asm/hardware/cache-b15-rac.h |  10 ++
 arch/arm/mm/Kconfig                           |   8 ++
 arch/arm/mm/Makefile                          |   1 +
 arch/arm/mm/cache-b15-rac.c                   | 177 ++++++++++++++++++++++++++
 5 files changed, 200 insertions(+)
 create mode 100644 arch/arm/include/asm/hardware/cache-b15-rac.h
 create mode 100644 arch/arm/mm/cache-b15-rac.c

diff --git a/arch/arm/include/asm/glue-cache.h b/arch/arm/include/asm/glue-cache.h
index 01c3d92624e5..8d1f498e5dd8 100644
--- a/arch/arm/include/asm/glue-cache.h
+++ b/arch/arm/include/asm/glue-cache.h
@@ -117,6 +117,10 @@
 # endif
 #endif
 
+#if defined(CONFIG_CACHE_B15_RAC)
+# define MULTI_CACHE 1
+#endif
+
 #if defined(CONFIG_CPU_V7M)
 #  define MULTI_CACHE 1
 #endif
diff --git a/arch/arm/include/asm/hardware/cache-b15-rac.h b/arch/arm/include/asm/hardware/cache-b15-rac.h
new file mode 100644
index 000000000000..3d43ec06fd35
--- /dev/null
+++ b/arch/arm/include/asm/hardware/cache-b15-rac.h
@@ -0,0 +1,10 @@
+#ifndef __ASM_ARM_HARDWARE_CACHE_B15_RAC_H
+#define __ASM_ARM_HARDWARE_CACHE_B15_RAC_H
+
+#ifndef __ASSEMBLY__
+
+void b15_flush_kern_cache_all(void);
+
+#endif
+
+#endif
diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index c6c4c9c8824b..cf7db07c98f4 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -904,6 +904,14 @@ config OUTER_CACHE_SYNC
 	  The outer cache has a outer_cache_fns.sync function pointer
 	  that can be used to drain the write buffer of the outer cache.
 
+config CACHE_B15_RAC
+	bool "Enable the Broadcom Brahma-B15 read-ahead cache controller"
+	depends on ARCH_BRCMSTB
+	default y
+	help
+	  This option enables the Broadcom Brahma-B15 read-ahead cache
+	  controller. If disabled, the read-ahead cache remains off.
+
 config CACHE_FEROCEON_L2
 	bool "Enable the Feroceon L2 cache controller"
 	depends on ARCH_MV78XX0 || ARCH_MVEBU
diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile
index b3dea80715b4..0d7937e884f8 100644
--- a/arch/arm/mm/Makefile
+++ b/arch/arm/mm/Makefile
@@ -102,6 +102,7 @@ AFLAGS_proc-v6.o	:=-Wa,-march=armv6
 AFLAGS_proc-v7.o	:=-Wa,-march=armv7-a
 
 obj-$(CONFIG_OUTER_CACHE)	+= l2c-common.o
+obj-$(CONFIG_CACHE_B15_RAC)	+= cache-b15-rac.o
 obj-$(CONFIG_CACHE_FEROCEON_L2)	+= cache-feroceon-l2.o
 obj-$(CONFIG_CACHE_L2X0)	+= cache-l2x0.o l2c-l2x0-resume.o
 obj-$(CONFIG_CACHE_L2X0_PMU)	+= cache-l2x0-pmu.o
diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c
new file mode 100644
index 000000000000..679d44f003fd
--- /dev/null
+++ b/arch/arm/mm/cache-b15-rac.c
@@ -0,0 +1,177 @@
+/*
+ * Broadcom Brahma-B15 CPU read-ahead cache management functions
+ *
+ * Copyright (C) 2015-2016 Broadcom
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/err.h>
+#include <linux/spinlock.h>
+#include <linux/io.h>
+#include <linux/bitops.h>
+#include <linux/of_address.h>
+
+#include <asm/cacheflush.h>
+#include <asm/hardware/cache-b15-rac.h>
+
+extern void v7_flush_kern_cache_all(void);
+
+/* RAC register offsets, relative to the HIF_CPU_BIUCTRL register base */
+#define RAC_CONFIG0_REG			(0x78)
+#define  RACENPREF_MASK			(0x3)
+#define  RACPREFINST_SHIFT		(0)
+#define  RACENINST_SHIFT		(2)
+#define  RACPREFDATA_SHIFT		(4)
+#define  RACENDATA_SHIFT		(6)
+#define  RAC_CPU_SHIFT			(8)
+#define  RACCFG_MASK			(0xff)
+#define RAC_CONFIG1_REG			(0x7c)
+#define RAC_FLUSH_REG			(0x80)
+#define  FLUSH_RAC			(1 << 0)
+
+/* Bitmask to enable instruction and data prefetching with a 256-bytes stride */
+#define RAC_DATA_INST_EN_MASK		(1 << RACPREFINST_SHIFT | \
+					 RACENPREF_MASK << RACENINST_SHIFT | \
+					 1 << RACPREFDATA_SHIFT | \
+					 RACENPREF_MASK << RACENDATA_SHIFT)
+
+#define RAC_ENABLED			0
+
+static void __iomem *b15_rac_base;
+static DEFINE_SPINLOCK(rac_lock);
+
+/* Initialization flag to avoid checking for b15_rac_base, and to prevent
+ * multi-platform kernels from crashing here as well.
+ */
+static unsigned long b15_rac_flags;
+
+static inline u32 __b15_rac_disable(void)
+{
+	u32 val = __raw_readl(b15_rac_base + RAC_CONFIG0_REG);
+	__raw_writel(0, b15_rac_base + RAC_CONFIG0_REG);
+	dmb();
+	return val;
+}
+
+static inline void __b15_rac_flush(void)
+{
+	u32 reg;
+
+	__raw_writel(FLUSH_RAC, b15_rac_base + RAC_FLUSH_REG);
+	do {
+		/* This dmb() is required to force the Bus Interface Unit
+		 * to clean oustanding writes, and forces an idle cycle
+		 * to be inserted.
+		 */
+		dmb();
+		reg = __raw_readl(b15_rac_base + RAC_FLUSH_REG);
+	} while (reg & FLUSH_RAC);
+}
+
+static inline u32 b15_rac_disable_and_flush(void)
+{
+	u32 reg;
+
+	reg = __b15_rac_disable();
+	__b15_rac_flush();
+	return reg;
+}
+
+static inline void __b15_rac_enable(u32 val)
+{
+	__raw_writel(val, b15_rac_base + RAC_CONFIG0_REG);
+	/* dsb() is required here to be consistent with __flush_icache_all() */
+	dsb();
+}
+
+#define BUILD_RAC_CACHE_OP(name, bar)				\
+void b15_flush_##name(void)					\
+{								\
+	unsigned int do_flush;					\
+	u32 val = 0;						\
+								\
+	spin_lock(&rac_lock);					\
+	do_flush = test_bit(RAC_ENABLED, &b15_rac_flags);	\
+	if (do_flush)						\
+		val = b15_rac_disable_and_flush();		\
+	v7_flush_##name();					\
+	if (!do_flush)						\
+		bar;						\
+	else							\
+		__b15_rac_enable(val);				\
+	spin_unlock(&rac_lock);					\
+}
+
+#define nobarrier
+
+/* The readahead cache present in the Brahma-B15 CPU is a special piece of
+ * hardware after the integrated L2 cache of the B15 CPU complex whose purpose
+ * is to prefetch instruction and/or data with a line size of either 64 bytes
+ * or 256 bytes. The rationale is that the data-bus of the CPU interface is
+ * optimized for 256-bytes transactions, and enabling the readahead cache
+ * provides a significant performance boost we want it enabled (typically
+ * twice the performance for a memcpy benchmark application).
+ *
+ * The readahead cache is transparent for Modified Virtual Addresses
+ * cache maintenance operations: ICIMVAU, DCIMVAC, DCCMVAC, DCCMVAU and
+ * DCCIMVAC.
+ *
+ * It is however not transparent for the following cache maintenance
+ * operations: DCISW, DCCSW, DCCISW, ICIALLUIS and ICIALLU which is precisely
+ * what we are patching here with our BUILD_RAC_CACHE_OP here.
+ */
+BUILD_RAC_CACHE_OP(kern_cache_all, nobarrier);
+
+static void b15_rac_enable(void)
+{
+	unsigned int cpu;
+	u32 enable = 0;
+
+	for_each_possible_cpu(cpu)
+		enable |= (RAC_DATA_INST_EN_MASK << (cpu * RAC_CPU_SHIFT));
+
+	b15_rac_disable_and_flush();
+	__b15_rac_enable(enable);
+}
+
+static int __init b15_rac_init(void)
+{
+	struct device_node *dn;
+	int ret = 0, cpu;
+	u32 reg, en_mask = 0;
+
+	dn = of_find_compatible_node(NULL, NULL, "brcm,brcmstb-cpu-biu-ctrl");
+	if (!dn)
+		return -ENODEV;
+
+	if (WARN(num_possible_cpus() > 4, "RAC only supports 4 CPUs\n"))
+		goto out;
+
+	b15_rac_base = of_iomap(dn, 0);
+	if (!b15_rac_base) {
+		pr_err("failed to remap BIU control base\n");
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	spin_lock(&rac_lock);
+	reg = __raw_readl(b15_rac_base + RAC_CONFIG0_REG);
+	for_each_possible_cpu(cpu)
+		en_mask |= ((1 << RACPREFDATA_SHIFT) << (cpu * RAC_CPU_SHIFT));
+	WARN(reg & en_mask, "Read-ahead cache not previously disabled\n");
+
+	b15_rac_enable();
+	set_bit(RAC_ENABLED, &b15_rac_flags);
+	spin_unlock(&rac_lock);
+
+	pr_info("Broadcom Brahma-B15 readahead cache at: 0x%p\n",
+		b15_rac_base + RAC_CONFIG0_REG);
+
+out:
+	of_node_put(dn);
+	return ret;
+}
+arch_initcall(b15_rac_init);
-- 
2.9.3

[PATCH v2 3/7] ARM: Hook B15 readahead cache functions based on processor

[Florian Fainelli]

If we detect that we are running on a Broadcom Brahma-B15 CPU, and
CONFIG_CACHE_B15_RAC is enabled, make sure that we pick-up the
b15_cache_fns function operations.

If CONFIG_CACHE_B15_RAC is enabled, but we are not running on a Broadcom
Brahma-B15 CPU, we will fallback to calling into the regular
v7_cache_fns with no cost. If CONFIG_CACHE_B15_RAC is disabled, there is
no cost and we just use the regular v7_cache_fns.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/mm/cache-v7.S | 21 +++++++++++++++++++++
 arch/arm/mm/proc-v7.S  |  2 +-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/arch/arm/mm/cache-v7.S b/arch/arm/mm/cache-v7.S
index de78109d002d..215df435bfb9 100644
--- a/arch/arm/mm/cache-v7.S
+++ b/arch/arm/mm/cache-v7.S
@@ -15,6 +15,7 @@
 #include <asm/assembler.h>
 #include <asm/errno.h>
 #include <asm/unwind.h>
+#include <asm/hardware/cache-b15-rac.h>
 
 #include "proc-macros.S"
 
@@ -446,3 +447,23 @@ ENDPROC(v7_dma_unmap_area)
 
 	@ define struct cpu_cache_fns (see <asm/cacheflush.h> and proc-macros.S)
 	define_cache_functions v7
+
+	/* The Broadcom Brahma-B15 read-ahead cache requires some modifications
+	 * to the v7_cache_fns, we only override the ones we need
+	 */
+#ifndef CONFIG_CACHE_B15_RAC
+	globl_equ	b15_flush_kern_cache_all,	v7_flush_kern_cache_all
+#endif
+	globl_equ	b15_flush_icache_all,		v7_flush_icache_all
+	globl_equ	b15_flush_kern_cache_louis,	v7_flush_kern_cache_louis
+	globl_equ	b15_flush_user_cache_all,	v7_flush_user_cache_all
+	globl_equ	b15_flush_user_cache_range,	v7_flush_user_cache_range
+	globl_equ	b15_coherent_kern_range,	v7_coherent_kern_range
+	globl_equ	b15_coherent_user_range,	v7_coherent_user_range
+	globl_equ	b15_flush_kern_dcache_area,	v7_flush_kern_dcache_area
+
+	globl_equ	b15_dma_map_area,		v7_dma_map_area
+	globl_equ	b15_dma_unmap_area,		v7_dma_unmap_area
+	globl_equ	b15_dma_flush_range,		v7_dma_flush_range
+
+	define_cache_functions b15
diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index 812ac861cd23..d55d493f9a1e 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -678,7 +678,7 @@ __v7_ca15mp_proc_info:
 __v7_b15mp_proc_info:
 	.long	0x420f00f0
 	.long	0xff0ffff0
-	__v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup
+	__v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup, cache_fns = b15_cache_fns
 	.size	__v7_b15mp_proc_info, . - __v7_b15mp_proc_info
 
 	/*
-- 
2.9.3

[PATCH v2 4/7] ARM: B15: Add CPU hotplug awareness

[Florian Fainelli]

The Broadcom Brahma-B15 readahead cache needs to be disabled,
respectively re-enable during a CPU hotplug. In case we were not to do,
CPU hotplug would occasionally fail with random crashes when a given CPU
exits the coherency domain while the RAC is still enabled, as it would
get stale data from the RAC.

In order to avoid adding any specific B15 readahead-cache awareness to
arch/arm/mach-bcm/hotplug-brcmstb.c we use a CPU hotplug state machine
which allows us to catch CPU hotplug events and disable/flush enable the
RAC accordingly.

Signed-off-by: Alamy Liu <alamyliu@broadcom.com>
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/mm/cache-b15-rac.c | 91 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/cpuhotplug.h  |  2 +
 2 files changed, 93 insertions(+)

diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c
index 679d44f003fd..d85b63211759 100644
--- a/arch/arm/mm/cache-b15-rac.c
+++ b/arch/arm/mm/cache-b15-rac.c
@@ -13,6 +13,8 @@
 #include <linux/io.h>
 #include <linux/bitops.h>
 #include <linux/of_address.h>
+#include <linux/notifier.h>
+#include <linux/cpu.h>
 
 #include <asm/cacheflush.h>
 #include <asm/hardware/cache-b15-rac.h>
@@ -42,6 +44,7 @@ extern void v7_flush_kern_cache_all(void);
 
 static void __iomem *b15_rac_base;
 static DEFINE_SPINLOCK(rac_lock);
+static u32 rac_config0_reg;
 
 /* Initialization flag to avoid checking for b15_rac_base, and to prevent
  * multi-platform kernels from crashing here as well.
@@ -137,6 +140,74 @@ static void b15_rac_enable(void)
 	__b15_rac_enable(enable);
 }
 
+#ifdef CONFIG_HOTPLUG_CPU
+/* The CPU hotplug case is the most interesting one, we basically need to make
+ * sure that the RAC is disabled for the entire system prior to having a CPU
+ * die, in particular prior to this dying CPU having exited the coherency
+ * domain.
+ *
+ * Once this CPU is marked dead, we can safely re-enable the RAC for the
+ * remaining CPUs in the system which are still online.
+ *
+ * Offlining a CPU is the problematic case, onlining a CPU is not much of an
+ * issue since the CPU and its cache-level hierarchy will start filling with
+ * the RAC disabled, so L1 and L2 only.
+ *
+ * In this function, we should NOT have to verify any unsafe setting/condition
+ * b15_rac_base:
+ *
+ *   It is protected by the RAC_ENABLED flag which is cleared by default, and
+ *   being cleared when initial procedure is done. b15_rac_base had been set at
+ *   that time.
+ *
+ * RAC_ENABLED:
+ *   There is a small timing windows, in b15_rac_init(), between
+ *      cpuhp_setup_state_*()
+ *      ...
+ *      set RAC_ENABLED
+ *   However, there is no hotplug activity based on the Linux booting procedure.
+ *
+ * Since we have to disable RAC for all cores, we keep RAC on as long as as
+ * possible (disable it as late as possible) to gain the cache benefit.
+ *
+ * Thus, dying/dead states are chosen here
+ *
+ * We are choosing not do disable the RAC on a per-CPU basis, here, if we did
+ * we would want to consider disabling it as early as possible to benefit the
+ * other active CPUs.
+ */
+
+/* Running on the dying CPU */
+static int b15_rac_dying_cpu(unsigned int cpu)
+{
+	spin_lock(&rac_lock);
+
+	/* Indicate that we are starting a hotplug procedure */
+	__clear_bit(RAC_ENABLED, &b15_rac_flags);
+
+	/* Disable the readahead cache and save its value to a global */
+	rac_config0_reg = b15_rac_disable_and_flush();
+
+	spin_unlock(&rac_lock);
+
+	return 0;
+}
+
+/* Running on a non-dying CPU */
+static int b15_rac_dead_cpu(unsigned int cpu)
+{
+	spin_lock(&rac_lock);
+
+	/* And enable it */
+	__b15_rac_enable(rac_config0_reg);
+	__set_bit(RAC_ENABLED, &b15_rac_flags);
+
+	spin_unlock(&rac_lock);
+
+	return 0;
+}
+#endif /* CONFIG_HOTPLUG_CPU */
+
 static int __init b15_rac_init(void)
 {
 	struct device_node *dn;
@@ -157,6 +228,20 @@ static int __init b15_rac_init(void)
 		goto out;
 	}
 
+#ifdef CONFIG_HOTPLUG_CPU
+	ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CACHE_B15_RAC_DEAD,
+					"arm/cache-b15-rac:dead",
+					NULL, b15_rac_dead_cpu);
+	if (ret)
+		goto out_unmap;
+
+	ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CACHE_B15_RAC_DYING,
+					"arm/cache-b15-rac:dying",
+					NULL, b15_rac_dying_cpu);
+	if (ret)
+		goto out_cpu_dead;
+#endif
+
 	spin_lock(&rac_lock);
 	reg = __raw_readl(b15_rac_base + RAC_CONFIG0_REG);
 	for_each_possible_cpu(cpu)
@@ -170,6 +255,12 @@ static int __init b15_rac_init(void)
 	pr_info("Broadcom Brahma-B15 readahead cache at: 0x%p\n",
 		b15_rac_base + RAC_CONFIG0_REG);
 
+	goto out;
+
+out_cpu_dead:
+	cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CACHE_B15_RAC_DYING);
+out_unmap:
+	iounmap(b15_rac_base);
 out:
 	of_node_put(dn);
 	return ret;
diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h
index 0f2a80377520..69d504fb23f2 100644
--- a/include/linux/cpuhotplug.h
+++ b/include/linux/cpuhotplug.h
@@ -41,6 +41,7 @@ enum cpuhp_state {
 	CPUHP_LUSTRE_CFS_DEAD,
 	CPUHP_SCSI_BNX2FC_DEAD,
 	CPUHP_SCSI_BNX2I_DEAD,
+	CPUHP_AP_ARM_CACHE_B15_RAC_DEAD,
 	CPUHP_WORKQUEUE_PREP,
 	CPUHP_POWER_NUMA_PREPARE,
 	CPUHP_HRTIMERS_PREPARE,
@@ -119,6 +120,7 @@ enum cpuhp_state {
 	CPUHP_AP_ARM64_ISNDEP_STARTING,
 	CPUHP_AP_SMPCFD_DYING,
 	CPUHP_AP_X86_TBOOT_DYING,
+	CPUHP_AP_ARM_CACHE_B15_RAC_DYING,
 	CPUHP_AP_ONLINE,
 	CPUHP_TEARDOWN_CPU,
 	CPUHP_AP_ONLINE_IDLE,
-- 
2.9.3

[PATCH v2 5/7] ARM: B15: Add suspend/resume hooks

[Florian Fainelli]

The Broadcom Brahma-B15 CPU readahead cache registers will be restored
to their Power-on-Reset values after a S3 suspend/resume cycles, so we
want to restore what we had enabled before.

Another thing we want to take care of is disabling the read-ahead cache
prior to suspending to avoid any sort of side effect with the spinlock
we need to grab to serialize register accesses.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/mm/cache-b15-rac.c | 48 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c
index d85b63211759..9ee1d89cced0 100644
--- a/arch/arm/mm/cache-b15-rac.c
+++ b/arch/arm/mm/cache-b15-rac.c
@@ -15,6 +15,7 @@
 #include <linux/of_address.h>
 #include <linux/notifier.h>
 #include <linux/cpu.h>
+#include <linux/syscore_ops.h>
 
 #include <asm/cacheflush.h>
 #include <asm/hardware/cache-b15-rac.h>
@@ -41,6 +42,10 @@ extern void v7_flush_kern_cache_all(void);
 					 RACENPREF_MASK << RACENDATA_SHIFT)
 
 #define RAC_ENABLED			0
+/* Special state where we want to bypass the spinlock and call directly
+ * into the v7 cache maintenance operations during suspend/resume
+ */
+#define RAC_SUSPENDED			1
 
 static void __iomem *b15_rac_base;
 static DEFINE_SPINLOCK(rac_lock);
@@ -96,6 +101,12 @@ void b15_flush_##name(void)					\
 	unsigned int do_flush;					\
 	u32 val = 0;						\
 								\
+	if (test_bit(RAC_SUSPENDED, &b15_rac_flags)) {		\
+		v7_flush_##name();				\
+		bar;						\
+		return;						\
+	}							\
+								\
 	spin_lock(&rac_lock);					\
 	do_flush = test_bit(RAC_ENABLED, &b15_rac_flags);	\
 	if (do_flush)						\
@@ -208,6 +219,39 @@ static int b15_rac_dead_cpu(unsigned int cpu)
 }
 #endif /* CONFIG_HOTPLUG_CPU */
 
+#ifdef CONFIG_PM_SLEEP
+static int b15_rac_suspend(void)
+{
+	/* Suspend the read-ahead cache oeprations, forcing our cache
+	 * implementation to fallback to the regular ARMv7 calls.
+	 *
+	 * We are guaranteed to be running on the boot CPU at this point and
+	 * with every other CPU quiesced, so setting RAC_SUSPENDED is not racy
+	 * here.
+	 */
+	rac_config0_reg = b15_rac_disable_and_flush();
+	set_bit(RAC_SUSPENDED, &b15_rac_flags);
+
+	return 0;
+}
+
+static void b15_rac_resume(void)
+{
+	/* Coming out of a S3 suspend/resume cycle, the read-ahead cache
+	 * register RAC_CONFIG0_REG will be restored to its default value, make
+	 * sure we re-enable it and set the enable flag, we are also guaranteed
+	 * to run on the boot CPU, so not racy again.
+	 */
+	__b15_rac_enable(rac_config0_reg);
+	clear_bit(RAC_SUSPENDED, &b15_rac_flags);
+}
+
+static struct syscore_ops b15_rac_syscore_ops = {
+	.suspend	= b15_rac_suspend,
+	.resume		= b15_rac_resume,
+};
+#endif
+
 static int __init b15_rac_init(void)
 {
 	struct device_node *dn;
@@ -242,6 +286,10 @@ static int __init b15_rac_init(void)
 		goto out_cpu_dead;
 #endif
 
+#ifdef CONFIG_PM_SLEEP
+	register_syscore_ops(&b15_rac_syscore_ops);
+#endif
+
 	spin_lock(&rac_lock);
 	reg = __raw_readl(b15_rac_base + RAC_CONFIG0_REG);
 	for_each_possible_cpu(cpu)
-- 
2.9.3

[PATCH v2 6/7] ARM: B15: Register reboot notifier for KEXEC

[Florian Fainelli]

During kexec, we will go through kernel_kexec() -> syscore_suspend() if
CONFIG_KEXEC_JUMP is set, if not, down the road we end-up calling
kernel_restart_prepare() which invokes reboot notifiers with
SYS_RESTART.

We register a reboot notifier to make sure that the B15 read-ahead cache
is disabled, since it is another level of instruction and data cache,
and we want to avoid any potential side effects with booting a new
kernel with such a cache still turned on.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 arch/arm/mm/cache-b15-rac.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/arch/arm/mm/cache-b15-rac.c b/arch/arm/mm/cache-b15-rac.c
index 9ee1d89cced0..f76988790011 100644
--- a/arch/arm/mm/cache-b15-rac.c
+++ b/arch/arm/mm/cache-b15-rac.c
@@ -16,6 +16,7 @@
 #include <linux/notifier.h>
 #include <linux/cpu.h>
 #include <linux/syscore_ops.h>
+#include <linux/reboot.h>
 
 #include <asm/cacheflush.h>
 #include <asm/hardware/cache-b15-rac.h>
@@ -151,6 +152,29 @@ static void b15_rac_enable(void)
 	__b15_rac_enable(enable);
 }
 
+static int b15_rac_reboot_notifier(struct notifier_block *nb,
+				   unsigned long action,
+				   void *data)
+{
+	/* During kexec, we are not yet migrated on the boot CPU, so we need to
+	 * make sure we are SMP safe here. Once the RAC is disabled, flag it as
+	 * suspended such that the hotplug notifier returns early.
+	 */
+	if (action == SYS_RESTART) {
+		spin_lock(&rac_lock);
+		b15_rac_disable_and_flush();
+		clear_bit(RAC_ENABLED, &b15_rac_flags);
+		set_bit(RAC_SUSPENDED, &b15_rac_flags);
+		spin_unlock(&rac_lock);
+	}
+
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block b15_rac_reboot_nb = {
+	.notifier_call	= b15_rac_reboot_notifier,
+};
+
 #ifdef CONFIG_HOTPLUG_CPU
 /* The CPU hotplug case is the most interesting one, we basically need to make
  * sure that the RAC is disabled for the entire system prior to having a CPU
@@ -191,6 +215,12 @@ static void b15_rac_enable(void)
 /* Running on the dying CPU */
 static int b15_rac_dying_cpu(unsigned int cpu)
 {
+	/* During kexec/reboot, the RAC is disabled via the reboot notifier
+	 * return early here.
+	 */
+	if (test_bit(RAC_SUSPENDED, &b15_rac_flags))
+		return 0;
+
 	spin_lock(&rac_lock);
 
 	/* Indicate that we are starting a hotplug procedure */
@@ -207,6 +237,12 @@ static int b15_rac_dying_cpu(unsigned int cpu)
 /* Running on a non-dying CPU */
 static int b15_rac_dead_cpu(unsigned int cpu)
 {
+	/* During kexec/reboot, the RAC is disabled via the reboot notifier
+	 * return early here.
+	 */
+	if (test_bit(RAC_SUSPENDED, &b15_rac_flags))
+		return 0;
+
 	spin_lock(&rac_lock);
 
 	/* And enable it */
@@ -272,6 +308,13 @@ static int __init b15_rac_init(void)
 		goto out;
 	}
 
+	ret = register_reboot_notifier(&b15_rac_reboot_nb);
+	if (ret) {
+		pr_err("failed to register reboot notifier\n");
+		iounmap(b15_rac_base);
+		goto out;
+	}
+
 #ifdef CONFIG_HOTPLUG_CPU
 	ret = cpuhp_setup_state_nocalls(CPUHP_AP_ARM_CACHE_B15_RAC_DEAD,
 					"arm/cache-b15-rac:dead",
@@ -308,6 +351,7 @@ static int __init b15_rac_init(void)
 out_cpu_dead:
 	cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CACHE_B15_RAC_DYING);
 out_unmap:
+	unregister_reboot_notifier(&b15_rac_reboot_nb);
 	iounmap(b15_rac_base);
 out:
 	of_node_put(dn);
-- 
2.9.3

[PATCH v2 7/7] MAINTAINERS: Update brcmstb entries to cover B15 code

[Florian Fainelli]

Update the brcmstb entry to cover the Broadcom Brahma-B15 processor
read-ahead cache support code.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
---
 MAINTAINERS | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 09b5ab6a8a5c..1e2035ef7c1c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2748,6 +2748,8 @@ S:	Maintained
 F:	arch/arm/mach-bcm/*brcmstb*
 F:	arch/arm/boot/dts/bcm7*.dts*
 F:	drivers/bus/brcmstb_gisb.c
+F:	arch/arm/mm/cache-b15-rac.c
+F:	arch/arm/include/asm/hardware/cache-b15-rac.h
 N:	brcmstb
 
 BROADCOM BMIPS MIPS ARCHITECTURE
-- 
2.9.3

Re: [PATCH v2 0/7] ARM: Broadcom Brahma-B15 readahead cache support

[Florian Fainelli]

On 06/23/2017 10:52 AM, Florian Fainelli wrote:
> Hi all,
> 
> This patch series adds support for the Broadcom Brahma-B15 readahead cache.
> I submitted that patch series a couple of years ago, and then slept on it so
> here is another stab at it.
> 
> Note that we did not implement this cache as a version of an outer cache
> for several reasons:
> 
> - we initially thought we needed to intercept flush_icache_all and
>   flush_kern_cache_louis but upon further inspection we convinced ourselves
>   this is no longer needed, still, flush_cache_all() needs special handling
>   here and needs to be wrapped around
> 
> - the outer cache does not allow differentiating a DMA transfer direction
>   this is a readahead cache, so it does not participate in writes, flushing
>   it during reads *and* writes kills the performance completely
> 
> - finally, most operations that outer_cache cares about are on MVA, which
>   is transparent to the readahead cache here
> 
> Changes in v2:
> 
> - clarify that the read-ahead caches does invalidates on writes (IOW) based
>   on Russell's feedback

Any comments on this? Thank you.

> 
> Florian Fainelli (7):
>   ARM: v7: allow setting different cache functions
>   ARM: Add Broadcom Brahma-B15 readahead cache support
>   ARM: Hook B15 readahead cache functions based on processor
>   ARM: B15: Add CPU hotplug awareness
>   ARM: B15: Add suspend/resume hooks
>   ARM: B15: Register reboot notifier for KEXEC
>   MAINTAINERS: Update brcmstb entries to cover B15 code
> 
>  MAINTAINERS                                   |   2 +
>  arch/arm/include/asm/glue-cache.h             |   4 +
>  arch/arm/include/asm/hardware/cache-b15-rac.h |  10 +
>  arch/arm/mm/Kconfig                           |   8 +
>  arch/arm/mm/Makefile                          |   1 +
>  arch/arm/mm/cache-b15-rac.c                   | 360 ++++++++++++++++++++++++++
>  arch/arm/mm/cache-v7.S                        |  21 ++
>  arch/arm/mm/proc-v7.S                         |   6 +-
>  include/linux/cpuhotplug.h                    |   2 +
>  9 files changed, 411 insertions(+), 3 deletions(-)
>  create mode 100644 arch/arm/include/asm/hardware/cache-b15-rac.h
>  create mode 100644 arch/arm/mm/cache-b15-rac.c
> 


-- 
Florian

Re: [PATCH v2 0/7] ARM: Broadcom Brahma-B15 readahead cache support

[Florian Fainelli]

On 07/14/2017 03:11 PM, Florian Fainelli wrote:
> On 06/23/2017 10:52 AM, Florian Fainelli wrote:
>> Hi all,
>>
>> This patch series adds support for the Broadcom Brahma-B15 readahead cache.
>> I submitted that patch series a couple of years ago, and then slept on it so
>> here is another stab at it.
>>
>> Note that we did not implement this cache as a version of an outer cache
>> for several reasons:
>>
>> - we initially thought we needed to intercept flush_icache_all and
>>   flush_kern_cache_louis but upon further inspection we convinced ourselves
>>   this is no longer needed, still, flush_cache_all() needs special handling
>>   here and needs to be wrapped around
>>
>> - the outer cache does not allow differentiating a DMA transfer direction
>>   this is a readahead cache, so it does not participate in writes, flushing
>>   it during reads *and* writes kills the performance completely
>>
>> - finally, most operations that outer_cache cares about are on MVA, which
>>   is transparent to the readahead cache here
>>
>> Changes in v2:
>>
>> - clarify that the read-ahead caches does invalidates on writes (IOW) based
>>   on Russell's feedback
> 
> Any comments on this? Thank you.

Should I just go a ahead and queue these changes in Russell's patch tracker?

> 
>>
>> Florian Fainelli (7):
>>   ARM: v7: allow setting different cache functions
>>   ARM: Add Broadcom Brahma-B15 readahead cache support
>>   ARM: Hook B15 readahead cache functions based on processor
>>   ARM: B15: Add CPU hotplug awareness
>>   ARM: B15: Add suspend/resume hooks
>>   ARM: B15: Register reboot notifier for KEXEC
>>   MAINTAINERS: Update brcmstb entries to cover B15 code
>>
>>  MAINTAINERS                                   |   2 +
>>  arch/arm/include/asm/glue-cache.h             |   4 +
>>  arch/arm/include/asm/hardware/cache-b15-rac.h |  10 +
>>  arch/arm/mm/Kconfig                           |   8 +
>>  arch/arm/mm/Makefile                          |   1 +
>>  arch/arm/mm/cache-b15-rac.c                   | 360 ++++++++++++++++++++++++++
>>  arch/arm/mm/cache-v7.S                        |  21 ++
>>  arch/arm/mm/proc-v7.S                         |   6 +-
>>  include/linux/cpuhotplug.h                    |   2 +
>>  9 files changed, 411 insertions(+), 3 deletions(-)
>>  create mode 100644 arch/arm/include/asm/hardware/cache-b15-rac.h
>>  create mode 100644 arch/arm/mm/cache-b15-rac.c
>>
> 
> 

-- 
Florian

Re: [PATCH v2 0/7] ARM: Broadcom Brahma-B15 readahead cache support

[Florian Fainelli]

On 08/14/2017 09:23 AM, Florian Fainelli wrote:
> 
> 
> On 07/14/2017 03:11 PM, Florian Fainelli wrote:
>> On 06/23/2017 10:52 AM, Florian Fainelli wrote:
>>> Hi all,
>>>
>>> This patch series adds support for the Broadcom Brahma-B15 readahead cache.
>>> I submitted that patch series a couple of years ago, and then slept on it so
>>> here is another stab at it.
>>>
>>> Note that we did not implement this cache as a version of an outer cache
>>> for several reasons:
>>>
>>> - we initially thought we needed to intercept flush_icache_all and
>>>   flush_kern_cache_louis but upon further inspection we convinced ourselves
>>>   this is no longer needed, still, flush_cache_all() needs special handling
>>>   here and needs to be wrapped around
>>>
>>> - the outer cache does not allow differentiating a DMA transfer direction
>>>   this is a readahead cache, so it does not participate in writes, flushing
>>>   it during reads *and* writes kills the performance completely
>>>
>>> - finally, most operations that outer_cache cares about are on MVA, which
>>>   is transparent to the readahead cache here
>>>
>>> Changes in v2:
>>>
>>> - clarify that the read-ahead caches does invalidates on writes (IOW) based
>>>   on Russell's feedback
>>
>> Any comments on this? Thank you.
> 
> Should I just go a ahead and queue these changes in Russell's patch tracker?

Per IRC conversation with Russell, queued into his patch tracking system as:

http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8724/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8725/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8729/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8726/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8730/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8728/1
http://www.armlinux.org.uk/developer/patches/viewpatch.php?id=8727/1
-- 
Florian