mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: "Jinhui Guo" <guojinhui.liam@bytedance.com>
To: <dakr@kernel.org>
Cc: <alexander.h.duyck@linux.intel.com>, <alexanderduyck@fb.com>,
	 <bhelgaas@google.com>, <bvanassche@acm.org>,
	<dan.j.williams@intel.com>,  <gregkh@linuxfoundation.org>,
	<guojinhui.liam@bytedance.com>,  <helgaas@kernel.org>,
	<linux-kernel@vger.kernel.org>,  <linux-pci@vger.kernel.org>,
	<rafael@kernel.org>, <tj@kernel.org>
Subject: Re: [PATCH 2/3] driver core: Add NUMA-node awareness to the synchronous probe path
Date: Thu,  8 Jan 2026 16:28:15 +0800	[thread overview]
Message-ID: <20260108082815.1876-1-guojinhui.liam@bytedance.com> (raw)
In-Reply-To: <DFIKE4Z23Q0O.2ZC7FR40XO8SR@kernel.org>

On Wed Jan 07, 2026 at 19:22:15 +0100, Danilo Krummrich wrote:
> On Wed Jan 7, 2026 at 6:55 PM CET, Jinhui Guo wrote:
> > + * __exec_on_numa_node - Execute a function on a specific NUMA node synchronously
> > + * @node: Target NUMA node ID
> > + * @func: The wrapper function to execute
> > + * @arg1: First argument (void *)
> > + * @arg2: Second argument (void *)
> > + *
> > + * Returns the result of the function execution, or -ENODEV if initialization fails.
> > + * If the node is invalid or offline, it falls back to local execution.
> > + */
> > +static int __exec_on_numa_node(int node, numa_func_t func, void *arg1, void *arg2)
> > +{
> > +	struct numa_work_ctx ctx;
> > +
> > +	/* Fallback to local execution if the node is invalid or offline */
> > +	if (node < 0 || node >= MAX_NUMNODES || !node_online(node))
> > +		return func(arg1, arg2);
> 
> Just a quick drive-by comment (I’ll go through it more thoroughly later).
> 
> What about the case where we are already on the requested node?
> 
> Also, we should probably set the corresponding CPU affinity for the time we are
> executing func() to prevent migration.

Hi Danilo,

Thank you for your time and helpful comments.

Relying on queue_work_node() for node affinity is safer, even if the thread
is already on the target CPU.

Checking the current CPU and then setting affinity ourselves would require
handling CPU-hotplug and isolated CPUs—corner cases that become complex
quickly.

The PCI driver tried this years ago and ran into numerous problems; delegating
the decision to queue_work_node() avoids repeating that history.

- Commit d42c69972b85 ("[PATCH] PCI: Run PCI driver initialization on local node")
  first added NUMA awareness with set_cpus_allowed_ptr().
- Commit 1ddd45f8d76f ("PCI: Use cpu_hotplug_disable() instead of get_online_cpus()")
  handled CPU-hotplug.
- Commits 69a18b18699b ("PCI: Restrict probe functions to housekeeping CPUs") and
  9d42ea0d6984 ("pci: Decouple HK_FLAG_WQ and HK_FLAG_DOMAIN cpumask fetch") dealt
  with isolated CPUs.

I considered setting CPU affinity, but the performance gain is minimal:

1. Driver probing happens mainly at boot, when load is light, so queuing a worker
   incurs little delay.
2. With many devices they are usually spread across nodes, so workers are not
   stalled long within any NUMA node.
3. Even after pinning, tasks can still be migrated by load balancing within the
   NUMA node, so the reduction in context switches versus using queue_work_node()
   alone is negligible.

Test data [1] shows that queue_work_node() has negligible impact on synchronous probe time.

[1] https://lore.kernel.org/all/20260107175548.1792-1-guojinhui.liam@bytedance.com/

If you have any other concerns, please let me know.

Best Regards,
Jinhui

  reply	other threads:[~2026-01-08  8:28 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-01-07 17:55 [PATCH 0/3] Add NUMA-node-aware synchronous probing to driver core Jinhui Guo
2026-01-07 17:55 ` [PATCH 1/3] driver core: Introduce helper function __device_attach_driver_scan() Jinhui Guo
2026-01-17 13:36   ` Danilo Krummrich
2026-01-07 17:55 ` [PATCH 2/3] driver core: Add NUMA-node awareness to the synchronous probe path Jinhui Guo
2026-01-07 18:22   ` Danilo Krummrich
2026-01-08  8:28     ` Jinhui Guo [this message]
2026-01-17 14:03   ` Danilo Krummrich
2026-01-20 17:23     ` Jinhui Guo
2026-01-07 17:55 ` [PATCH 3/3] PCI: Clean up NUMA-node awareness in pci_bus_type probe Jinhui Guo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260108082815.1876-1-guojinhui.liam@bytedance.com \
    --to=guojinhui.liam@bytedance.com \
    --cc=alexander.h.duyck@linux.intel.com \
    --cc=alexanderduyck@fb.com \
    --cc=bhelgaas@google.com \
    --cc=bvanassche@acm.org \
    --cc=dakr@kernel.org \
    --cc=dan.j.williams@intel.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=helgaas@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=rafael@kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®