[PATCH RFC net-next 0/3] dpaa2: avoid scheduling dpio interrupts on isolated CPUs

[PATCH RFC net-next 0/3] dpaa2: avoid scheduling dpio interrupts on isolated CPUs

[Josua Mayer]

Sent as RFC due to very superficial testing, and confusion wrt. which
list/tree to target. Suggest net-next. Last patch is logical first, but
must be applied last.

Each DPAA2 software portal (DPIO) is bound to one CPU. The dpaa2-eth and
caam qi2 drivers allocate channels and queues for each online CPU.
Notifications, and NAPI polling are processed on each DPIO's affine CPU.
This balances I/O load nicely across all available CPU cores while
keeping data on the hot path.

When isolcpus=managed_irq is used to reserve CPUs for latency-sensitive
applications, those CPUs are expected to be free of routine device
interrupts. However, DPIO interrupts currently continue to arrive on
isolated CPUs, interrupting any application running there.

This series assigns DPIOs only to CPUs in the HK_TYPE_MANAGED_IRQ
housekeeping mask, and changes dpaa2-eth and caam qi2 to set up their
per-CPU channels and queue pairs only on those CPUs. Without the driver
changes, both would defer probe indefinitely, waiting for a DPIO on an
isolated CPU.

Isolated CPUs can still transmit frames and enqueue crypto requests;
the resulting ingress, Tx confirmation and response processing runs on
housekeeping CPUs. Systems without isolcpus=managed_irq are unaffected.

Note: This series is submitted in reverse, to ease bisecting.
Otherwise caam/dpaa2-eth would defer probe iondefinitely when booted
with one or more isolated CPUs.

Signed-off-by: Josua Mayer <josua@solid-run.com>
---
Josua Mayer (3):
      dpaa2-eth: skip isolated CPUs when allocating channels
      crypto: caam/qi2 - skip isolated CPUs when setting up queue pairs
      soc: fsl: dpio: skip isolated CPUs when assigning portals

 drivers/crypto/caam/caamalg_qi2.c                | 26 +++++++++++++++---------
 drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 10 +++++++--
 drivers/soc/fsl/dpio/dpio-driver.c               |  7 +++++--
 3 files changed, 29 insertions(+), 14 deletions(-)
---
base-commit: 47a1446725732cd3996edf607e8739334bbf4d78
change-id: 20260930-dpaa2-isolcpus-ecb715f2004d

Best regards,
-- 
Josua Mayer <josua@solid-run.com>

[PATCH RFC net-next 1/3] dpaa2-eth: skip isolated CPUs when allocating channels

[Josua Mayer]

The driver allocates one channel per online CPU and registers it with
that CPU's affine DPIO.

When isolcpus=managed_irq is used to reserve CPUs for latency-sensitive
applications, those CPUs are expected to be free of routine device
interrupts.

The DPIO driver assigns affine CPUs only from the HK_TYPE_MANAGED_IRQ
housekeeping mask, leaving isolated CPUs without an affine DPIO. This
defers dpaa2-eth probe indefinitely once channel setup reaches an
isolated CPU.

Restrict channel setup to the online CPUs in the HK_TYPE_MANAGED_IRQ
housekeeping mask, so that isolated CPUs are skipped and don't defer
probe.

Signed-off-by: Josua Mayer <josua@solid-run.com>
---
 drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
index 6f1046c9cc515..850834388790a 100644
--- a/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
+++ b/drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c
@@ -15,6 +15,7 @@
 #include <linux/bpf_trace.h>
 #include <linux/fsl/ptp_qoriq.h>
 #include <linux/ptp_classify.h>
+#include <linux/sched/isolation.h>
 #include <net/pkt_cls.h>
 #include <net/sock.h>
 #include <net/tso.h>
@@ -3173,6 +3174,7 @@ static int dpaa2_eth_setup_dpio(struct dpaa2_eth_priv *priv)
 	struct dpaa2_eth_channel *channel;
 	struct dpcon_notification_cfg dpcon_notif_cfg;
 	struct device *dev = priv->net_dev->dev.parent;
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	int i, err;
 
 	/* We want the ability to spread ingress traffic (RX, TX conf) to as
@@ -3182,10 +3184,11 @@ static int dpaa2_eth_setup_dpio(struct dpaa2_eth_priv *priv)
 	 * Allocate one channel per core and register it to the core's
 	 * affine DPIO. If not enough channels are available for all cores
 	 * or if some cores don't have an affine DPIO, there will be no
-	 * ingress frame processing on those cores.
+	 * ingress frame processing on those cores. Isolated cores
+	 * (isolcpus=managed_irq) have no affine DPIO.
 	 */
 	cpumask_clear(&priv->dpio_cpumask);
-	for_each_online_cpu(i) {
+	for_each_cpu_and(i, cpu_online_mask, hk_mask) {
 		/* Try to allocate a channel */
 		channel = dpaa2_eth_alloc_channel(priv);
 		if (IS_ERR_OR_NULL(channel)) {
@@ -3245,6 +3248,9 @@ static int dpaa2_eth_setup_dpio(struct dpaa2_eth_priv *priv)
 			break;
 	}
 
+	if (cpumask_empty(&priv->dpio_cpumask))
+		goto err_alloc_ch;
+
 	return 0;
 
 err_set_cdan:

-- 
2.51.0

[PATCH RFC net-next 2/3] crypto: caam/qi2 - skip isolated CPUs when setting up queue pairs

[Josua Mayer]

The driver looks up the affine DPIO and sets up one Rx/Tx queue pair for
each online CPU, up to the number of queues. Response notifications and
NAPI polling are handled on the DPIO object's assigned CPU. All online
may enqueue requests.

When isolcpus=managed_irq is used to reserve CPUs for latency-sensitive
applications, those CPUs are expected to be free of routine device
interrupts.

The DPIO driver assigns affine CPUs only from the HK_TYPE_MANAGED_IRQ
housekeeping mask, leaving isolated CPUs without an affine DPIO. This
defers caam qi2 probe indefinitely once DPIO setup reaches an isolated
CPU.

Restrict queue pairs, notifications and NAPI polling to the online CPUs
in the HK_TYPE_MANAGED_IRQ housekeeping mask, so that isolated CPUs are
skipped and don't defer probe. Isolated CPUs may still enqueue requests,
with responses handled on housekeeping CPUs.

As the dequeuing CPUs are no longer necessarily CPUs 0 to num_pairs - 1,
enable and disable NAPI based on the bookkeeping in clean_mask instead.

Signed-off-by: Josua Mayer <josua@solid-run.com>
---
 drivers/crypto/caam/caamalg_qi2.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/drivers/crypto/caam/caamalg_qi2.c b/drivers/crypto/caam/caamalg_qi2.c
index 6b47bcc16a506..e9faaee009b74 100644
--- a/drivers/crypto/caam/caamalg_qi2.c
+++ b/drivers/crypto/caam/caamalg_qi2.c
@@ -25,6 +25,7 @@
 #include <soc/fsl/dpaa2-fd.h>
 #include <crypto/xts.h>
 #include <linux/unaligned.h>
+#include <linux/sched/isolation.h>
 
 #define CAAM_CRA_PRIORITY	2000
 
@@ -4683,9 +4684,10 @@ static int __cold dpaa2_dpseci_dpio_setup(struct dpaa2_caam_priv *priv)
 	struct device *dev = priv->dev;
 	struct dpaa2_io_notification_ctx *nctx;
 	struct dpaa2_caam_priv_per_cpu *ppriv;
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	int err, i = 0, cpu;
 
-	for_each_online_cpu(cpu) {
+	for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
 		ppriv->priv = priv;
 		nctx = &ppriv->nctx;
@@ -4725,14 +4727,14 @@ static int __cold dpaa2_dpseci_dpio_setup(struct dpaa2_caam_priv *priv)
 	return 0;
 
 err:
-	for_each_online_cpu(cpu) {
+	for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
 		if (!ppriv->nctx.cb)
 			break;
 		dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx, dev);
 	}
 
-	for_each_online_cpu(cpu) {
+	for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
 		if (!ppriv->store)
 			break;
@@ -4745,9 +4747,10 @@ static int __cold dpaa2_dpseci_dpio_setup(struct dpaa2_caam_priv *priv)
 static void __cold dpaa2_dpseci_dpio_free(struct dpaa2_caam_priv *priv)
 {
 	struct dpaa2_caam_priv_per_cpu *ppriv;
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	int i = 0, cpu;
 
-	for_each_online_cpu(cpu) {
+	for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
 		dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx,
 					    priv->dev);
@@ -4764,10 +4767,11 @@ static int dpaa2_dpseci_bind(struct dpaa2_caam_priv *priv)
 	struct device *dev = priv->dev;
 	struct fsl_mc_device *ls_dev = to_fsl_mc_device(dev);
 	struct dpaa2_caam_priv_per_cpu *ppriv;
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
 	int err = 0, i = 0, cpu;
 
 	/* Configure Rx queues */
-	for_each_online_cpu(cpu) {
+	for_each_cpu_and(cpu, cpu_online_mask, hk_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, cpu);
 
 		rx_queue_cfg.options = DPSECI_QUEUE_OPT_DEST |
@@ -5012,6 +5016,8 @@ static int __cold dpaa2_dpseci_setup(struct fsl_mc_device *ls_dev)
 	struct device *dev = &ls_dev->dev;
 	struct dpaa2_caam_priv *priv;
 	struct dpaa2_caam_priv_per_cpu *ppriv;
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
+	unsigned int num_online_hk_cpus = cpumask_weight_and(cpu_online_mask, hk_mask);
 	int err, cpu;
 	u8 i;
 
@@ -5070,10 +5076,10 @@ static int __cold dpaa2_dpseci_setup(struct fsl_mc_device *ls_dev)
 
 	priv->num_pairs = min(priv->dpseci_attr.num_rx_queues,
 			      priv->dpseci_attr.num_tx_queues);
-	if (priv->num_pairs > num_online_cpus()) {
+	if (priv->num_pairs > num_online_hk_cpus) {
 		dev_warn(dev, "%d queues won't be used\n",
-			 priv->num_pairs - num_online_cpus());
-		priv->num_pairs = num_online_cpus();
+			 priv->num_pairs - num_online_hk_cpus);
+		priv->num_pairs = num_online_hk_cpus;
 	}
 
 	for (i = 0; i < priv->dpseci_attr.num_rx_queues; i++) {
@@ -5151,7 +5157,7 @@ static int dpaa2_dpseci_enable(struct dpaa2_caam_priv *priv)
 	struct dpaa2_caam_priv_per_cpu *ppriv;
 	int i;
 
-	for (i = 0; i < priv->num_pairs; i++) {
+	for_each_cpu(i, priv->clean_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, i);
 		napi_enable(&ppriv->napi);
 	}
@@ -5180,7 +5186,7 @@ static int __cold dpaa2_dpseci_disable(struct dpaa2_caam_priv *priv)
 
 	dev_dbg(dev, "disable: %s\n", str_false_true(enabled));
 
-	for (i = 0; i < priv->num_pairs; i++) {
+	for_each_cpu(i, priv->clean_mask) {
 		ppriv = per_cpu_ptr(priv->ppriv, i);
 		napi_disable(&ppriv->napi);
 		netif_napi_del(&ppriv->napi);

-- 
2.51.0

[PATCH RFC net-next 3/3] soc: fsl: dpio: skip isolated CPUs when assigning portals

[Josua Mayer]

During probe, each DPIO object is assigned to exactly one CPU, and its
interrupt affinity is set accordingly. CPUs are picked from all online
CPUs, regardless of CPU isolation.

The dpaa2-eth and caam qi2 drivers use one DPIO per CPU and follow its
affinity: notifications from each DPIO and the matching NAPI polling are
handled on the DPIO object's assigned CPU.

When isolcpus=managed_irq is used to reserve CPUs for latency-sensitive
applications, those CPUs are expected to be free of routine device
interrupts. However, DPIO interrupts currently continue to arrive on
isolated CPUs.

Restrict the CPUs available to DPIO objects to the online CPUs in the
HK_TYPE_MANAGED_IRQ housekeeping mask, so that isolated CPUs are skipped
and left to the applications pinned to them.

Signed-off-by: Josua Mayer <josua@solid-run.com>
---
 drivers/soc/fsl/dpio/dpio-driver.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/soc/fsl/dpio/dpio-driver.c b/drivers/soc/fsl/dpio/dpio-driver.c
index d5198ffc44550..242db78affb95 100644
--- a/drivers/soc/fsl/dpio/dpio-driver.c
+++ b/drivers/soc/fsl/dpio/dpio-driver.c
@@ -13,6 +13,7 @@
 #include <linux/dma-mapping.h>
 #include <linux/delay.h>
 #include <linux/io.h>
+#include <linux/sched/isolation.h>
 #include <linux/sys_soc.h>
 
 #include <linux/fsl/mc.h>
@@ -173,7 +174,7 @@ static int dpaa2_dpio_probe(struct fsl_mc_device *dpio_dev)
 	/* get the cpu to use for the affinity hint */
 	possible_next_cpu = cpumask_first(cpus_unused_mask);
 	if (possible_next_cpu >= nr_cpu_ids) {
-		dev_err(dev, "probe failed. Number of DPIOs exceeds NR_CPUS.\n");
+		dev_err(dev, "No unused managed_irq cpu left for dpio\n");
 		err = -ERANGE;
 		goto err_allocate_irqs;
 	}
@@ -317,9 +318,11 @@ static struct fsl_mc_driver dpaa2_dpio_driver = {
 
 static int dpio_driver_init(void)
 {
+	const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
+
 	if (!zalloc_cpumask_var(&cpus_unused_mask, GFP_KERNEL))
 		return -ENOMEM;
-	cpumask_copy(cpus_unused_mask, cpu_online_mask);
+	cpumask_and(cpus_unused_mask, cpu_online_mask, hk_mask);
 
 	return fsl_mc_driver_register(&dpaa2_dpio_driver);
 }

-- 
2.51.0