[PATCH 0/7] MIPS: arch_numa enablement

[PATCH 0/7] MIPS: arch_numa enablement

[Jiaxun Yang]

Hi all,

This series enabled arch_numa for generic MIPS machine, and ported
loongson64 machine to arch_numa infra. SGI IP27 is left untouched
with legacy NUMA implementation as I don't have access to hardware.

Tested on Loongson64 NUMA system and numa_emu on Boston.

Please review.

Thanks

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
Jiaxun Yang (7):
      arch_numa: Provide platform numa init hook
      MIPS: pci: Unify pcibus_to_node implementation
      MIPS: Prepare NUMA headers for arch_numa
      MIPS: mm: init: Prepare for arch_numa
      MIPS: smp: Process NUMA information
      MIPS: generic: Make NUMA available
      MIPS: Loongson64: Migrate to arch_numa

 arch/mips/Kconfig                                |   4 +
 arch/mips/include/asm/mach-generic/mmzone.h      |  11 ++
 arch/mips/include/asm/mach-generic/numa.h        |  11 ++
 arch/mips/include/asm/mach-generic/topology.h    |   7 +
 arch/mips/include/asm/mach-ip27/numa.h           |  22 ++++
 arch/mips/include/asm/mach-ip27/topology.h       |  12 +-
 arch/mips/include/asm/mach-loongson64/mmzone.h   |   2 -
 arch/mips/include/asm/mach-loongson64/topology.h |  25 ----
 arch/mips/include/asm/mmzone.h                   |   5 +-
 arch/mips/include/asm/numa.h                     |  12 ++
 arch/mips/include/asm/pci.h                      |  12 ++
 arch/mips/include/asm/smp-ops.h                  |   7 +-
 arch/mips/kernel/setup.c                         |   3 +-
 arch/mips/kernel/smp-cps.c                       |   6 +
 arch/mips/kernel/smp.c                           |  26 ++++
 arch/mips/loongson64/init.c                      |  27 ++--
 arch/mips/loongson64/numa.c                      | 158 ++---------------------
 arch/mips/loongson64/smp.c                       |   2 +
 arch/mips/mm/init.c                              |  14 +-
 arch/mips/pci/pci-ip27.c                         |  10 --
 arch/mips/pci/pci-xtalk-bridge.c                 |   1 +
 drivers/base/arch_numa.c                         |   7 +
 include/asm-generic/numa.h                       |   1 +
 23 files changed, 155 insertions(+), 230 deletions(-)
---
base-commit: eec5d86d5bac6b3e972eb9c1898af3c08303c52d
change-id: 20240807-mips-numa-6ccdd482141b

Best regards,
-- 
Jiaxun Yang <jiaxun.yang@flygoat.com>

[PATCH 1/7] arch_numa: Provide platform numa init hook

[Jiaxun Yang]

For some pre-devicetree systems, NUMA information may come from
platform specific way.

Provide platform numa init hook to allow platform code kick in
as last resort method to supply NUMA configuration.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 drivers/base/arch_numa.c   | 7 +++++++
 include/asm-generic/numa.h | 1 +
 2 files changed, 8 insertions(+)

diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 8d49893c0e94..d464baeee69a 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -305,6 +305,11 @@ static int __init arch_acpi_numa_init(void)
 }
 #endif
 
+int __init __weak arch_platform_numa_init(void)
+{
+	return -EOPNOTSUPP;
+}
+
 /**
  * arch_numa_init() - Initialize NUMA
  *
@@ -318,6 +323,8 @@ void __init arch_numa_init(void)
 			return;
 		if (acpi_disabled && !numa_init(of_numa_init))
 			return;
+		if (!numa_init(arch_platform_numa_init))
+			return;
 	}
 
 	numa_init(dummy_numa_init);
diff --git a/include/asm-generic/numa.h b/include/asm-generic/numa.h
index c2b046d1fd82..53a8210fde00 100644
--- a/include/asm-generic/numa.h
+++ b/include/asm-generic/numa.h
@@ -31,6 +31,7 @@ static inline const struct cpumask *cpumask_of_node(int node)
 #endif
 
 void __init arch_numa_init(void);
+int __init arch_platform_numa_init(void);
 int __init numa_add_memblk(int nodeid, u64 start, u64 end);
 void __init early_map_cpu_to_node(unsigned int cpu, int nid);
 int __init early_cpu_to_node(int cpu);

-- 
2.46.0

[PATCH 2/7] MIPS: pci: Unify pcibus_to_node implementation

[Jiaxun Yang]

Nowadays PCI Bus NUMA node information is always stored in struct
device of host bridge.

Unify the implementation of pcibus_to_node to simplify code.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/include/asm/mach-ip27/topology.h       |  4 ----
 arch/mips/include/asm/mach-loongson64/topology.h |  5 -----
 arch/mips/include/asm/pci.h                      | 12 ++++++++++++
 arch/mips/loongson64/numa.c                      |  7 -------
 arch/mips/pci/pci-ip27.c                         | 10 ----------
 arch/mips/pci/pci-xtalk-bridge.c                 |  1 +
 6 files changed, 13 insertions(+), 26 deletions(-)

diff --git a/arch/mips/include/asm/mach-ip27/topology.h b/arch/mips/include/asm/mach-ip27/topology.h
index d66cc53feab8..0ad74e2e0d85 100644
--- a/arch/mips/include/asm/mach-ip27/topology.h
+++ b/arch/mips/include/asm/mach-ip27/topology.h
@@ -17,10 +17,6 @@ extern struct cpuinfo_ip27 sn_cpu_info[NR_CPUS];
 #define cpumask_of_node(node)	((node) == -1 ?				\
 				 cpu_all_mask :				\
 				 &hub_data(node)->h_cpus)
-struct pci_bus;
-extern int pcibus_to_node(struct pci_bus *);
-
-#define cpumask_of_pcibus(bus)	(cpumask_of_node(pcibus_to_node(bus)))
 
 extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
 
diff --git a/arch/mips/include/asm/mach-loongson64/topology.h b/arch/mips/include/asm/mach-loongson64/topology.h
index 3414a1fd1783..c60ccdbe5a94 100644
--- a/arch/mips/include/asm/mach-loongson64/topology.h
+++ b/arch/mips/include/asm/mach-loongson64/topology.h
@@ -9,11 +9,6 @@
 extern cpumask_t __node_cpumask[];
 #define cpumask_of_node(node)	(&__node_cpumask[node])
 
-struct pci_bus;
-extern int pcibus_to_node(struct pci_bus *);
-
-#define cpumask_of_pcibus(bus)	(cpu_online_mask)
-
 extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
 
 #define node_distance(from, to)	(__node_distances[(from)][(to)])
diff --git a/arch/mips/include/asm/pci.h b/arch/mips/include/asm/pci.h
index d993df6302dc..09323348d362 100644
--- a/arch/mips/include/asm/pci.h
+++ b/arch/mips/include/asm/pci.h
@@ -134,6 +134,18 @@ static inline int pci_proc_domain(struct pci_bus *bus)
 }
 #endif /* CONFIG_PCI_DOMAINS */
 
+#ifdef CONFIG_NUMA
+static inline int pcibus_to_node(struct pci_bus *bus)
+{
+	return dev_to_node(&bus->dev);
+}
+#ifndef cpumask_of_pcibus
+#define cpumask_of_pcibus(bus)	(pcibus_to_node(bus) == -1 ?		\
+				 cpu_all_mask :				\
+				 cpumask_of_node(pcibus_to_node(bus)))
+#endif
+#endif
+
 #endif /* __KERNEL__ */
 
 /* Do platform specific device initialization at pci_enable_device() time */
diff --git a/arch/mips/loongson64/numa.c b/arch/mips/loongson64/numa.c
index 8388400d052f..d49180562c9f 100644
--- a/arch/mips/loongson64/numa.c
+++ b/arch/mips/loongson64/numa.c
@@ -171,13 +171,6 @@ void __init mem_init(void)
 	setup_zero_pages();	/* This comes from node 0 */
 }
 
-/* All PCI device belongs to logical Node-0 */
-int pcibus_to_node(struct pci_bus *bus)
-{
-	return 0;
-}
-EXPORT_SYMBOL(pcibus_to_node);
-
 void __init prom_init_numa_memory(void)
 {
 	pr_info("CP0_Config3: CP0 16.3 (0x%x)\n", read_c0_config3());
diff --git a/arch/mips/pci/pci-ip27.c b/arch/mips/pci/pci-ip27.c
index 973faea61cad..d919c70e02b1 100644
--- a/arch/mips/pci/pci-ip27.c
+++ b/arch/mips/pci/pci-ip27.c
@@ -17,16 +17,6 @@
 #include <asm/sn/ioc3.h>
 #include <asm/pci/bridge.h>
 
-#ifdef CONFIG_NUMA
-int pcibus_to_node(struct pci_bus *bus)
-{
-	struct bridge_controller *bc = BRIDGE_CONTROLLER(bus);
-
-	return bc->nasid;
-}
-EXPORT_SYMBOL(pcibus_to_node);
-#endif /* CONFIG_NUMA */
-
 static void ip29_fixup_phy(struct pci_dev *dev)
 {
 	int nasid = pcibus_to_node(dev->bus);
diff --git a/arch/mips/pci/pci-xtalk-bridge.c b/arch/mips/pci/pci-xtalk-bridge.c
index 45ddbaa6c123..45b2b390e553 100644
--- a/arch/mips/pci/pci-xtalk-bridge.c
+++ b/arch/mips/pci/pci-xtalk-bridge.c
@@ -633,6 +633,7 @@ static int bridge_probe(struct platform_device *pdev)
 		return -ENOMEM;
 	}
 
+	set_dev_node(dev, bd->nasid);
 	pci_set_flags(PCI_PROBE_ONLY);
 
 	host = devm_pci_alloc_host_bridge(dev, sizeof(*bc));

-- 
2.46.0

[PATCH 3/7] MIPS: Prepare NUMA headers for arch_numa

[Jiaxun Yang]

Massage headers so that we can include asm-generic/numa.h
for generic platform.

Provide dummy arch_numa like functions in Loongson64/IP27 platform
to fill the gap between them and generic platform.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/include/asm/mach-generic/mmzone.h      | 11 +++++++++++
 arch/mips/include/asm/mach-generic/numa.h        | 11 +++++++++++
 arch/mips/include/asm/mach-generic/topology.h    |  7 +++++++
 arch/mips/include/asm/mach-ip27/numa.h           | 22 ++++++++++++++++++++++
 arch/mips/include/asm/mach-ip27/topology.h       |  8 +++-----
 arch/mips/include/asm/mach-loongson64/numa.h     | 22 ++++++++++++++++++++++
 arch/mips/include/asm/mach-loongson64/topology.h | 10 ++++------
 arch/mips/include/asm/mmzone.h                   |  5 +----
 arch/mips/include/asm/numa.h                     | 12 ++++++++++++
 arch/mips/kernel/setup.c                         |  1 +
 10 files changed, 94 insertions(+), 15 deletions(-)

diff --git a/arch/mips/include/asm/mach-generic/mmzone.h b/arch/mips/include/asm/mach-generic/mmzone.h
new file mode 100644
index 000000000000..d4374b71a900
--- /dev/null
+++ b/arch/mips/include/asm/mach-generic/mmzone.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
+ */
+
+#ifndef __ASM_MACH_GENERIC_MMZONE_H
+#define __ASM_MACH_GENERIC_MMZONE_H
+
+/* Intentionally empty file ...	 */
+
+#endif /* __ASM_MACH_GENERIC_MMZONE_H */
diff --git a/arch/mips/include/asm/mach-generic/numa.h b/arch/mips/include/asm/mach-generic/numa.h
new file mode 100644
index 000000000000..311a5871807c
--- /dev/null
+++ b/arch/mips/include/asm/mach-generic/numa.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
+ */
+
+#ifndef __ASM_MACH_GENERIC_NUMA_H
+#define __ASM_MACH_GENERIC_NUMA_H
+
+#include <asm-generic/numa.h>
+
+#endif	/* __ASM_NUMA_H */
diff --git a/arch/mips/include/asm/mach-generic/topology.h b/arch/mips/include/asm/mach-generic/topology.h
index 5428f333a02c..fa82d18ba9e7 100644
--- a/arch/mips/include/asm/mach-generic/topology.h
+++ b/arch/mips/include/asm/mach-generic/topology.h
@@ -1 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef __ASM_MACH_GENERIC_TOPOLOGY_H
+#define __ASM_MACH_GENERIC_TOPOLOGY_H
+
+#include <asm/numa.h>
 #include <asm-generic/topology.h>
+
+#endif
diff --git a/arch/mips/include/asm/mach-ip27/numa.h b/arch/mips/include/asm/mach-ip27/numa.h
new file mode 100644
index 000000000000..e2adadf8a589
--- /dev/null
+++ b/arch/mips/include/asm/mach-ip27/numa.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
+ */
+
+#ifndef __ASM_MACH_NUMA_H
+#define __ASM_MACH_NUMA_H
+
+#ifdef CONFIG_NUMA
+extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
+
+#define node_distance(from, to) (__node_distances[(from)][(to)])
+#endif
+
+/* Hanlded in platform code */
+static inline void numa_store_cpu_info(unsigned int cpu) { }
+static inline void numa_add_cpu(unsigned int cpu) { }
+static inline void numa_remove_cpu(unsigned int cpu) { }
+static inline void arch_numa_init(void) { }
+static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
+
+#endif	/* __ASM_NUMA_H */
diff --git a/arch/mips/include/asm/mach-ip27/topology.h b/arch/mips/include/asm/mach-ip27/topology.h
index 0ad74e2e0d85..33d5bf925431 100644
--- a/arch/mips/include/asm/mach-ip27/topology.h
+++ b/arch/mips/include/asm/mach-ip27/topology.h
@@ -4,6 +4,7 @@
 
 #include <asm/sn/types.h>
 #include <asm/mmzone.h>
+#include <asm/numa.h>
 
 struct cpuinfo_ip27 {
 	nasid_t		p_nasid;	/* my node ID in numa-as-id-space */
@@ -13,15 +14,12 @@ struct cpuinfo_ip27 {
 
 extern struct cpuinfo_ip27 sn_cpu_info[NR_CPUS];
 
-#define cpu_to_node(cpu)	(cputonasid(cpu))
+#define early_cpu_to_node(cpu)	(cputonasid(cpu))
+#define cpu_to_node(cpu)  early_cpu_to_node(cpu)
 #define cpumask_of_node(node)	((node) == -1 ?				\
 				 cpu_all_mask :				\
 				 &hub_data(node)->h_cpus)
 
-extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
-
-#define node_distance(from, to) (__node_distances[(from)][(to)])
-
 #include <asm-generic/topology.h>
 
 #endif /* _ASM_MACH_TOPOLOGY_H */
diff --git a/arch/mips/include/asm/mach-loongson64/numa.h b/arch/mips/include/asm/mach-loongson64/numa.h
new file mode 100644
index 000000000000..2fac370d981f
--- /dev/null
+++ b/arch/mips/include/asm/mach-loongson64/numa.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
+ */
+
+#ifndef __ASM_MACH_LOONGSON64_NUMA_H
+#define __ASM_MACH_LOONGSON64_NUMA_H
+
+#ifdef CONFIG_NUMA
+extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
+
+#define node_distance(from, to)	(__node_distances[(from)][(to)])
+#endif
+
+/* Hanlded in platform code */
+static inline void numa_store_cpu_info(unsigned int cpu) { }
+static inline void numa_add_cpu(unsigned int cpu) { }
+static inline void numa_remove_cpu(unsigned int cpu) { }
+static inline void arch_numa_init(void) { }
+static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
+
+#endif	/* __ASM_NUMA_H */
diff --git a/arch/mips/include/asm/mach-loongson64/topology.h b/arch/mips/include/asm/mach-loongson64/topology.h
index c60ccdbe5a94..ebb086821401 100644
--- a/arch/mips/include/asm/mach-loongson64/topology.h
+++ b/arch/mips/include/asm/mach-loongson64/topology.h
@@ -2,17 +2,15 @@
 #ifndef _ASM_MACH_TOPOLOGY_H
 #define _ASM_MACH_TOPOLOGY_H
 
-#ifdef CONFIG_NUMA
+#include <asm/numa.h>
 
-#define cpu_to_node(cpu)	(cpu_logical_map(cpu) >> 2)
+#ifdef CONFIG_NUMA
+#define early_cpu_to_node(cpu)	(cpu_logical_map(cpu) >> 2)
+#define cpu_to_node(cpu)  early_cpu_to_node(cpu)
 
 extern cpumask_t __node_cpumask[];
 #define cpumask_of_node(node)	(&__node_cpumask[node])
 
-extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
-
-#define node_distance(from, to)	(__node_distances[(from)][(to)])
-
 #endif
 
 #include <asm-generic/topology.h>
diff --git a/arch/mips/include/asm/mmzone.h b/arch/mips/include/asm/mmzone.h
index 14226ea42036..f472c8ec3a63 100644
--- a/arch/mips/include/asm/mmzone.h
+++ b/arch/mips/include/asm/mmzone.h
@@ -7,10 +7,7 @@
 #define _ASM_MMZONE_H_
 
 #include <asm/page.h>
-
-#ifdef CONFIG_NUMA
-# include <mmzone.h>
-#endif
+#include <mmzone.h>
 
 #ifndef pa_to_nid
 #define pa_to_nid(addr) 0
diff --git a/arch/mips/include/asm/numa.h b/arch/mips/include/asm/numa.h
new file mode 100644
index 000000000000..ff1eb9ef51e6
--- /dev/null
+++ b/arch/mips/include/asm/numa.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
+ */
+
+#ifndef __ASM_NUMA_H
+#define __ASM_NUMA_H
+
+#include <asm/topology.h>
+#include <numa.h>
+
+#endif /* __ASM_NUMA_H */
diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index 12a1a4ffb602..655391c04477 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -664,6 +664,7 @@ static void __init arch_mem_init(char **cmdline_p)
 
 	mips_parse_crashkernel();
 	device_tree_init();
+	arch_numa_init();
 
 	/*
 	 * In order to reduce the possibility of kernel panic when failed to

-- 
2.46.0

[PATCH 4/7] MIPS: mm: init: Prepare for arch_numa

[Jiaxun Yang]

Opt-in various mm initailise routines for NUMA platforms
with arch_numa enabled.

Loongson64 & IP27 have their own implementations but arch_numa
is fine with the generic implementation.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/mm/init.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/mips/mm/init.c b/arch/mips/mm/init.c
index 4583d1a2a73e..4d7d5e817939 100644
--- a/arch/mips/mm/init.c
+++ b/arch/mips/mm/init.c
@@ -401,7 +401,7 @@ void maar_init(void)
 	}
 }
 
-#ifndef CONFIG_NUMA
+#if !defined(CONFIG_NUMA) || defined(CONFIG_GENERIC_ARCH_NUMA)
 void __init paging_init(void)
 {
 	unsigned long max_zone_pfns[MAX_NR_ZONES];
@@ -424,14 +424,14 @@ void __init paging_init(void)
 		       (highend_pfn - max_low_pfn) << (PAGE_SHIFT - 10));
 		max_zone_pfns[ZONE_HIGHMEM] = max_low_pfn;
 
-		max_mapnr = max_low_pfn;
+		set_max_mapnr(max_low_pfn);
 	} else if (highend_pfn) {
-		max_mapnr = highend_pfn;
+		set_max_mapnr(highend_pfn);
 	} else {
-		max_mapnr = max_low_pfn;
+		set_max_mapnr(max_low_pfn);
 	}
 #else
-	max_mapnr = max_low_pfn;
+	set_max_mapnr(max_low_pfn);
 #endif
 	high_memory = (void *) __va(max_low_pfn << PAGE_SHIFT);
 
@@ -482,7 +482,7 @@ void __init mem_init(void)
 				0x80000000 - 4, KCORE_TEXT);
 #endif
 }
-#endif /* !CONFIG_NUMA */
+#endif
 
 void free_init_pages(const char *what, unsigned long begin, unsigned long end)
 {
@@ -519,7 +519,7 @@ void __ref free_initmem(void)
 		free_initmem_default(POISON_FREE_INITMEM);
 }
 
-#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
+#if defined(CONFIG_NUMA) && !defined(CONFIG_GENERIC_ARCH_NUMA)
 unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
 EXPORT_SYMBOL(__per_cpu_offset);
 

-- 
2.46.0

[PATCH 5/7] MIPS: smp: Process NUMA information

[Jiaxun Yang]

Performing numa_add_cpu and numa_remove_cpu as necessary, store
NUMA map from devicetree and fix up if platform SMP code didn't
do mapping properly.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/include/asm/smp-ops.h |  7 +------
 arch/mips/kernel/smp-cps.c      |  6 ++++++
 arch/mips/kernel/smp.c          | 26 ++++++++++++++++++++++++++
 3 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/smp-ops.h b/arch/mips/include/asm/smp-ops.h
index 1617b207723f..4d984f0088cb 100644
--- a/arch/mips/include/asm/smp-ops.h
+++ b/arch/mips/include/asm/smp-ops.h
@@ -40,12 +40,7 @@ struct plat_smp_ops {
 
 extern void register_smp_ops(const struct plat_smp_ops *ops);
 
-static inline void plat_smp_setup(void)
-{
-	extern const struct plat_smp_ops *mp_ops;	/* private */
-
-	mp_ops->smp_setup();
-}
+extern void __init plat_smp_setup(void);
 
 extern void mips_smp_send_ipi_single(int cpu, unsigned int action);
 extern void mips_smp_send_ipi_mask(const struct cpumask *mask,
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 395622c37325..be9f13c0b058 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -8,6 +8,7 @@
 #include <linux/delay.h>
 #include <linux/io.h>
 #include <linux/memblock.h>
+#include <linux/of.h>
 #include <linux/sched/task_stack.h>
 #include <linux/sched/hotplug.h>
 #include <linux/slab.h>
@@ -179,10 +180,14 @@ static void __init cps_smp_setup(void)
 
 	/* Indicate present CPUs (CPU being synonymous with VPE) */
 	for (v = 0; v < min_t(unsigned, nvpes, NR_CPUS); v++) {
+		struct device_node *np = of_get_cpu_node(v, NULL);
+
 		set_cpu_possible(v, cpu_cluster(&cpu_data[v]) == 0);
 		set_cpu_present(v, cpu_cluster(&cpu_data[v]) == 0);
 		__cpu_number_map[v] = v;
 		__cpu_logical_map[v] = v;
+		if (np)
+			early_map_cpu_to_node(v, of_node_to_nid(np));
 	}
 
 	/* Set a coherent default CCA (CWB) */
@@ -553,6 +558,7 @@ static int cps_cpu_disable(void)
 	atomic_sub(1 << cpu_vpe_id(&current_cpu_data), &core_cfg->vpe_mask);
 	smp_mb__after_atomic();
 	set_cpu_online(cpu, false);
+	numa_remove_cpu(cpu);
 	calculate_cpu_foreign_map();
 	irq_migrate_all_off_this_cpu();
 
diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 0362fc5df7b0..17dd97ad3cbb 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -153,6 +153,11 @@ void calculate_cpu_foreign_map(void)
 			       &temp_foreign_map, &cpu_sibling_map[i]);
 }
 
+bool arch_match_cpu_phys_id(int cpu, u64 phys_id)
+{
+	return phys_id == cpu_logical_map(cpu);
+}
+
 const struct plat_smp_ops *mp_ops;
 EXPORT_SYMBOL(mp_ops);
 
@@ -164,6 +169,19 @@ void register_smp_ops(const struct plat_smp_ops *ops)
 	mp_ops = ops;
 }
 
+void __init plat_smp_setup(void)
+{
+	unsigned int cpu;
+
+	mp_ops->smp_setup();
+
+	for_each_possible_cpu(cpu) {
+		/* Workaround platform SMP code not handling NUMA map */
+		if (early_cpu_to_node(cpu) == NUMA_NO_NODE)
+			early_map_cpu_to_node(cpu, 0);
+	}
+}
+
 #ifdef CONFIG_GENERIC_IRQ_IPI
 void mips_smp_send_ipi_single(int cpu, unsigned int action)
 {
@@ -372,6 +390,7 @@ asmlinkage void start_secondary(void)
 
 	set_cpu_sibling_map(cpu);
 	set_cpu_core_map(cpu);
+	numa_add_cpu(cpu);
 
 	cpumask_set_cpu(cpu, &cpu_coherent_mask);
 	notify_cpu_starting(cpu);
@@ -426,6 +445,8 @@ void __init smp_cpus_done(unsigned int max_cpus)
 /* called from main before smp_init() */
 void __init smp_prepare_cpus(unsigned int max_cpus)
 {
+	unsigned int cpu;
+
 	init_new_context(current, &init_mm);
 	current_thread_info()->cpu = 0;
 	mp_ops->prepare_cpus(max_cpus);
@@ -436,6 +457,11 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 	init_cpu_present(cpu_possible_mask);
 #endif
 	cpumask_copy(&cpu_coherent_mask, cpu_possible_mask);
+
+	for_each_possible_cpu(cpu)
+		numa_store_cpu_info(cpu);
+
+	numa_add_cpu(0);
 }
 
 /* preload SMP state for boot cpu */

-- 
2.46.0

[PATCH 6/7] MIPS: generic: Make NUMA available

[Jiaxun Yang]

Enable arch_numa and dependent options for generic and future NUMA
platforms.

Enable SPARSEMEM for 64 bit generic kernel.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 43da6d596e2b..9284a06db2b4 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -140,6 +140,7 @@ choice
 
 config MIPS_GENERIC_KERNEL
 	bool "Generic board-agnostic MIPS kernel"
+	select ARCH_SPARSEMEM_ENABLE if 64BIT
 	select MIPS_GENERIC
 	select BOOT_RAW
 	select BUILTIN_DTB
@@ -178,6 +179,7 @@ config MIPS_GENERIC_KERNEL
 	select SYS_SUPPORTS_MIPS16
 	select SYS_SUPPORTS_MIPS_CPS
 	select SYS_SUPPORTS_MULTITHREADING
+	select SYS_SUPPORTS_NUMA
 	select SYS_SUPPORTS_RELOCATABLE
 	select SYS_SUPPORTS_SMARTMIPS
 	select SYS_SUPPORTS_ZBOOT
@@ -2599,8 +2601,10 @@ config NUMA
 	bool "NUMA Support"
 	depends on SYS_SUPPORTS_NUMA
 	select SMP
+	select GENERIC_ARCH_NUMA if !SGI_IP27 && !MACH_LOONGSON64
 	select HAVE_SETUP_PER_CPU_AREA
 	select NEED_PER_CPU_EMBED_FIRST_CHUNK
+	select USE_PERCPU_NUMA_NODE_ID if GENERIC_ARCH_NUMA
 	help
 	  Say Y to compile the kernel to support NUMA (Non-Uniform Memory
 	  Access).  This option improves performance on systems with more

-- 
2.46.0

[PATCH 7/7] MIPS: Loongson64: Migrate to arch_numa

[Jiaxun Yang]

Migrate to generic arch_numa infra to reduce maintenance burden.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
 arch/mips/Kconfig                                |   2 +-
 arch/mips/include/asm/mach-loongson64/mmzone.h   |   2 -
 arch/mips/include/asm/mach-loongson64/numa.h     |  22 ----
 arch/mips/include/asm/mach-loongson64/topology.h |  18 ---
 arch/mips/kernel/setup.c                         |   2 +-
 arch/mips/loongson64/init.c                      |  27 ++--
 arch/mips/loongson64/numa.c                      | 153 ++---------------------
 arch/mips/loongson64/smp.c                       |   2 +
 8 files changed, 24 insertions(+), 204 deletions(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 9284a06db2b4..1edf36f89171 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -2601,7 +2601,7 @@ config NUMA
 	bool "NUMA Support"
 	depends on SYS_SUPPORTS_NUMA
 	select SMP
-	select GENERIC_ARCH_NUMA if !SGI_IP27 && !MACH_LOONGSON64
+	select GENERIC_ARCH_NUMA if !SGI_IP27
 	select HAVE_SETUP_PER_CPU_AREA
 	select NEED_PER_CPU_EMBED_FIRST_CHUNK
 	select USE_PERCPU_NUMA_NODE_ID if GENERIC_ARCH_NUMA
diff --git a/arch/mips/include/asm/mach-loongson64/mmzone.h b/arch/mips/include/asm/mach-loongson64/mmzone.h
index 8fb70fd3c9c4..f5946e69ad55 100644
--- a/arch/mips/include/asm/mach-loongson64/mmzone.h
+++ b/arch/mips/include/asm/mach-loongson64/mmzone.h
@@ -14,6 +14,4 @@
 #define pa_to_nid(addr)  (((addr) & 0xf00000000000) >> NODE_ADDRSPACE_SHIFT)
 #define nid_to_addrbase(nid) ((unsigned long)(nid) << NODE_ADDRSPACE_SHIFT)
 
-extern void __init prom_init_numa_memory(void);
-
 #endif /* _ASM_MACH_MMZONE_H */
diff --git a/arch/mips/include/asm/mach-loongson64/numa.h b/arch/mips/include/asm/mach-loongson64/numa.h
deleted file mode 100644
index 2fac370d981f..000000000000
--- a/arch/mips/include/asm/mach-loongson64/numa.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Copyright (C) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
- */
-
-#ifndef __ASM_MACH_LOONGSON64_NUMA_H
-#define __ASM_MACH_LOONGSON64_NUMA_H
-
-#ifdef CONFIG_NUMA
-extern unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
-
-#define node_distance(from, to)	(__node_distances[(from)][(to)])
-#endif
-
-/* Hanlded in platform code */
-static inline void numa_store_cpu_info(unsigned int cpu) { }
-static inline void numa_add_cpu(unsigned int cpu) { }
-static inline void numa_remove_cpu(unsigned int cpu) { }
-static inline void arch_numa_init(void) { }
-static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
-
-#endif	/* __ASM_NUMA_H */
diff --git a/arch/mips/include/asm/mach-loongson64/topology.h b/arch/mips/include/asm/mach-loongson64/topology.h
deleted file mode 100644
index ebb086821401..000000000000
--- a/arch/mips/include/asm/mach-loongson64/topology.h
+++ /dev/null
@@ -1,18 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_MACH_TOPOLOGY_H
-#define _ASM_MACH_TOPOLOGY_H
-
-#include <asm/numa.h>
-
-#ifdef CONFIG_NUMA
-#define early_cpu_to_node(cpu)	(cpu_logical_map(cpu) >> 2)
-#define cpu_to_node(cpu)  early_cpu_to_node(cpu)
-
-extern cpumask_t __node_cpumask[];
-#define cpumask_of_node(node)	(&__node_cpumask[node])
-
-#endif
-
-#include <asm-generic/topology.h>
-
-#endif /* _ASM_MACH_TOPOLOGY_H */
diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index 655391c04477..242945d240c5 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -252,7 +252,7 @@ static unsigned long __init init_initrd(void)
  * Initialize the bootmem allocator. It also setup initrd related data
  * if needed.
  */
-#if defined(CONFIG_SGI_IP27) || (defined(CONFIG_CPU_LOONGSON64) && defined(CONFIG_NUMA))
+#if defined(CONFIG_SGI_IP27)
 
 static void __init bootmem_init(void)
 {
diff --git a/arch/mips/loongson64/init.c b/arch/mips/loongson64/init.c
index a35dd7311795..dad78bcce93f 100644
--- a/arch/mips/loongson64/init.c
+++ b/arch/mips/loongson64/init.c
@@ -46,9 +46,9 @@ void virtual_early_config(void)
 	node_id_offset = 44;
 }
 
-void __init szmem(unsigned int node)
+static void __init prom_init_memory(void)
 {
-	u32 i, mem_type;
+	u32 i, mem_type, max_node_id = 0;
 	phys_addr_t node_id, mem_start, mem_size;
 
 	/* Otherwise come from DTB */
@@ -58,8 +58,8 @@ void __init szmem(unsigned int node)
 	/* Parse memory information and activate */
 	for (i = 0; i < loongson_memmap->nr_map; i++) {
 		node_id = loongson_memmap->map[i].node_id;
-		if (node_id != node)
-			continue;
+		if (node_id > max_node_id)
+			max_node_id = node_id;
 
 		mem_type = loongson_memmap->map[i].mem_type;
 		mem_size = loongson_memmap->map[i].mem_size;
@@ -78,7 +78,7 @@ void __init szmem(unsigned int node)
 		case UMA_VIDEO_RAM:
 			pr_info("Node %d, mem_type:%d\t[%pa], %pa bytes usable\n",
 				(u32)node_id, mem_type, &mem_start, &mem_size);
-			memblock_add_node(mem_start, mem_size, node,
+			memblock_add_node(mem_start, mem_size, node_id,
 					  MEMBLOCK_NONE);
 			break;
 		case SYSTEM_RAM_RESERVED:
@@ -104,16 +104,11 @@ void __init szmem(unsigned int node)
 		memblock_reserve(virt_to_phys((void *)loongson_sysconf.vgabios_addr),
 				SZ_256K);
 	/* set nid for reserved memory */
-	memblock_set_node((u64)node << 44, (u64)(node + 1) << 44,
-			&memblock.reserved, node);
-}
-
-#ifndef CONFIG_NUMA
-static void __init prom_init_memory(void)
-{
-	szmem(0);
+	for (i = 0; i <= max_node_id; i++) {
+		memblock_set_node(nid_to_addrbase(i), nid_to_addrbase(i + 1),
+				  &memblock.reserved, i);
+	}
 }
-#endif
 
 void __init prom_init(void)
 {
@@ -133,11 +128,7 @@ void __init prom_init(void)
 	if (loongson_sysconf.early_config)
 		loongson_sysconf.early_config();
 
-#ifdef CONFIG_NUMA
-	prom_init_numa_memory();
-#else
 	prom_init_memory();
-#endif
 
 	/* Hardcode to CPU UART 0 */
 	if ((read_c0_prid() & PRID_IMP_MASK) == PRID_IMP_LOONGSON_64R)
diff --git a/arch/mips/loongson64/numa.c b/arch/mips/loongson64/numa.c
index d49180562c9f..9431e6e0a082 100644
--- a/arch/mips/loongson64/numa.c
+++ b/arch/mips/loongson64/numa.c
@@ -9,45 +9,10 @@
 #include <linux/init.h>
 #include <linux/kernel.h>
 #include <linux/mm.h>
-#include <linux/mmzone.h>
-#include <linux/export.h>
-#include <linux/nodemask.h>
-#include <linux/swap.h>
-#include <linux/memblock.h>
-#include <linux/pfn.h>
-#include <linux/highmem.h>
-#include <asm/page.h>
-#include <asm/pgalloc.h>
-#include <asm/sections.h>
-#include <linux/irq.h>
-#include <asm/bootinfo.h>
-#include <asm/mc146818-time.h>
-#include <asm/time.h>
-#include <asm/wbflush.h>
+#include <linux/numa_memblks.h>
 #include <boot_param.h>
 #include <loongson.h>
 
-unsigned char __node_distances[MAX_NUMNODES][MAX_NUMNODES];
-EXPORT_SYMBOL(__node_distances);
-
-cpumask_t __node_cpumask[MAX_NUMNODES];
-EXPORT_SYMBOL(__node_cpumask);
-
-static void cpu_node_probe(void)
-{
-	int i;
-
-	nodes_clear(node_possible_map);
-	nodes_clear(node_online_map);
-	for (i = 0; i < loongson_sysconf.nr_nodes; i++) {
-		node_set_state(num_online_nodes(), N_POSSIBLE);
-		node_set_online(num_online_nodes());
-	}
-
-	pr_info("NUMA: Discovered %d cpus on %d nodes\n",
-		loongson_sysconf.nr_cpus, num_online_nodes());
-}
-
 static int __init compute_node_distance(int row, int col)
 {
 	int package_row = row * loongson_sysconf.cores_per_node /
@@ -63,117 +28,21 @@ static int __init compute_node_distance(int row, int col)
 		return 100;
 }
 
-static void __init init_topology_matrix(void)
-{
-	int row, col;
-
-	for (row = 0; row < MAX_NUMNODES; row++)
-		for (col = 0; col < MAX_NUMNODES; col++)
-			__node_distances[row][col] = -1;
-
-	for_each_online_node(row) {
-		for_each_online_node(col) {
-			__node_distances[row][col] =
-				compute_node_distance(row, col);
-		}
-	}
-}
-
-static void __init node_mem_init(unsigned int node)
-{
-	unsigned long node_addrspace_offset;
-	unsigned long start_pfn, end_pfn;
-
-	node_addrspace_offset = nid_to_addrbase(node);
-	pr_info("Node%d's addrspace_offset is 0x%lx\n",
-			node, node_addrspace_offset);
-
-	get_pfn_range_for_nid(node, &start_pfn, &end_pfn);
-	pr_info("Node%d: start_pfn=0x%lx, end_pfn=0x%lx\n",
-		node, start_pfn, end_pfn);
-
-	alloc_node_data(node);
-
-	NODE_DATA(node)->node_start_pfn = start_pfn;
-	NODE_DATA(node)->node_spanned_pages = end_pfn - start_pfn;
-
-	if (node == 0) {
-		/* kernel start address */
-		unsigned long kernel_start_pfn = PFN_DOWN(__pa_symbol(&_text));
-
-		/* kernel end address */
-		unsigned long kernel_end_pfn = PFN_UP(__pa_symbol(&_end));
-
-		/* used by finalize_initrd() */
-		max_low_pfn = end_pfn;
-
-		/* Reserve the kernel text/data/bss */
-		memblock_reserve(kernel_start_pfn << PAGE_SHIFT,
-				 ((kernel_end_pfn - kernel_start_pfn) << PAGE_SHIFT));
-
-		/* Reserve 0xfe000000~0xffffffff for RS780E integrated GPU */
-		if (node_end_pfn(0) >= (0xffffffff >> PAGE_SHIFT))
-			memblock_reserve((node_addrspace_offset | 0xfe000000),
-					 32 << 20);
-
-		/* Reserve pfn range 0~node[0]->node_start_pfn */
-		memblock_reserve(0, PAGE_SIZE * start_pfn);
-		/* set nid for reserved memory on node 0 */
-		memblock_set_node(0, 1ULL << 44, &memblock.reserved, 0);
-	}
-}
-
-static __init void prom_meminit(void)
+int __init arch_platform_numa_init(void)
 {
-	unsigned int node, cpu, active_cpu = 0;
+	int nr_nodes, nid, row, col;
 
-	cpu_node_probe();
-	init_topology_matrix();
+	nr_nodes = loongson_sysconf.nr_nodes;
 
-	for (node = 0; node < loongson_sysconf.nr_nodes; node++) {
-		if (node_online(node)) {
-			szmem(node);
-			node_mem_init(node);
-			cpumask_clear(&__node_cpumask[node]);
-		}
+	for (nid = 0; nid < nr_nodes; nid++) {
+		node_set(nid, numa_nodes_parsed);
+		numa_add_memblk(nid, nid_to_addrbase(nid), nid_to_addrbase(nid + 1));
 	}
-	max_low_pfn = PHYS_PFN(memblock_end_of_DRAM());
-
-	for (cpu = 0; cpu < loongson_sysconf.nr_cpus; cpu++) {
-		node = cpu / loongson_sysconf.cores_per_node;
-		if (node >= num_online_nodes())
-			node = 0;
-
-		if (loongson_sysconf.reserved_cpus_mask & (1<<cpu))
-			continue;
 
-		cpumask_set_cpu(active_cpu, &__node_cpumask[node]);
-		pr_info("NUMA: set cpumask cpu %d on node %d\n", active_cpu, node);
-
-		active_cpu++;
+	for (row = 0; row < nr_nodes; row++) {
+		for (col = 0; col < nr_nodes; col++)
+			numa_set_distance(row, col, compute_node_distance(row, col));
 	}
-}
-
-void __init paging_init(void)
-{
-	unsigned long zones_size[MAX_NR_ZONES] = {0, };
-
-	pagetable_init();
-	zones_size[ZONE_DMA32] = MAX_DMA32_PFN;
-	zones_size[ZONE_NORMAL] = max_low_pfn;
-	free_area_init(zones_size);
-}
 
-void __init mem_init(void)
-{
-	high_memory = (void *) __va(get_num_physpages() << PAGE_SHIFT);
-	memblock_free_all();
-	setup_zero_pages();	/* This comes from node 0 */
-}
-
-void __init prom_init_numa_memory(void)
-{
-	pr_info("CP0_Config3: CP0 16.3 (0x%x)\n", read_c0_config3());
-	pr_info("CP0_PageGrain: CP0 5.1 (0x%x)\n", read_c0_pagegrain());
-	prom_meminit();
+	return 0;
 }
diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
index 147acd972a07..417063b8145b 100644
--- a/arch/mips/loongson64/smp.c
+++ b/arch/mips/loongson64/smp.c
@@ -464,6 +464,7 @@ static void __init loongson3_smp_setup(void)
 			set_cpu_possible(num, true);
 			/* Loongson processors are always grouped by 4 */
 			cpu_set_cluster(&cpu_data[num], i / 4);
+			early_map_cpu_to_node(num, i / loongson_sysconf.cores_per_node);
 			num++;
 		}
 		i++;
@@ -517,6 +518,7 @@ static int loongson3_cpu_disable(void)
 	unsigned int cpu = smp_processor_id();
 
 	set_cpu_online(cpu, false);
+	numa_remove_cpu(cpu);
 	calculate_cpu_foreign_map();
 	local_irq_save(flags);
 	clear_c0_status(ST0_IM);

-- 
2.46.0

Re: [PATCH 1/7] arch_numa: Provide platform numa init hook

[Arnd Bergmann]

On Fri, Aug 9, 2024, at 21:25, Jiaxun Yang wrote:
> For some pre-devicetree systems, NUMA information may come from
> platform specific way.
>
> Provide platform numa init hook to allow platform code kick in
> as last resort method to supply NUMA configuration.
>
> Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>

Can you do this with a Kconfig symbol in the header instead
of a __weak symbol?

      Arnd

Re: [PATCH 1/7] arch_numa: Provide platform numa init hook

[Jiaxun Yang]

在2024年8月9日八月 下午8:41，Arnd Bergmann写道：
> On Fri, Aug 9, 2024, at 21:25, Jiaxun Yang wrote:
>> For some pre-devicetree systems, NUMA information may come from
>> platform specific way.
>>
>> Provide platform numa init hook to allow platform code kick in
>> as last resort method to supply NUMA configuration.
>>
>> Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
>
> Can you do this with a Kconfig symbol in the header instead
> of a __weak symbol?

Hi Arnd,

Sure, is this some kind of subsystem policy or general recommendation
applies to the whole tree?

Thanks
- Jiaxun

>
>       Arnd

-- 
- Jiaxun

Re: [PATCH 1/7] arch_numa: Provide platform numa init hook

[Arnd Bergmann]

On Fri, Aug 9, 2024, at 21:56, Jiaxun Yang wrote:
> 在2024年8月9日八月 下午8:41，Arnd Bergmann写道：
>> On Fri, Aug 9, 2024, at 21:25, Jiaxun Yang wrote:
>>> For some pre-devicetree systems, NUMA information may come from
>>> platform specific way.
>>>
>>> Provide platform numa init hook to allow platform code kick in
>>> as last resort method to supply NUMA configuration.
>>>
>>> Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
>>
>> Can you do this with a Kconfig symbol in the header instead
>> of a __weak symbol?
>
> Sure, is this some kind of subsystem policy or general recommendation
> applies to the whole tree?

I don't think it's a general policy, possibly it's just me, but I've
had to debug too many issues that could have been avoided by not
__weak symbols, so I try to not have them in code I'm responsible
for like the asm-generic headers.

The main places that use __weak symbols are arch/mips and
drivers/pci, but there are also a number of them in mm/
and kernel/.

     Arnd