Re: [PATCH v7 05/10] scsi: Use block layer helpers to constrain queue affinity

[Hannes Reinecke <hare@suse.de>]

On 7/4/25 11:37, Daniel Wagner wrote:
> On Thu, Jul 03, 2025 at 08:43:01AM +0200, Hannes Reinecke wrote:
>> All of these drivers are not aware of CPU hotplug, and as such
>> will not be notified when the number of CPUs changes.
> 
> Ah, this explains this part.
> 
>> But you use 'blk_mq_online_queue_affinity()' for all of these
>> drivers.
> 
> All these drivers are also using blk_mq_num_online_queue. When I only
> used cpu_possible_mask the resulting mapping was not usable.
> 
Yeah, I'd imagine so. Quite some drivers have 'interesting' ideas how
the firmware interface should look like.

But it also means that there is a very high likelyhood that these
drivers become inoperable under CPU hotplug.
Is there a way of disabling CPU hotplug when these drivers are in use?

>> Wouldn't 'blk_mq_possible_queue_affinit()' a better choice here
>> to insulate against CPU hotplug effects?
> 
> With this mask the queues will be distributed to all possible CPUs and
> some of the hardware queues could be assigned to offline CPUs. I think
> this would work but the question is, is this okay to leave some of the
> perfomance on the road?
> 
It really shouldn't be an issue when the cpus are distributed 
'correctly' :-)
We have several possibilities:
-> #hwq > num_possible_cpus: easy, 1:1 mapping, no problem
-> num_online_cpu < #hwq < num_possible_cpus: Not as easy, but if we
    ensure that each online cpu is mapped to a different hwq we don't
    have a performance impact.
-> #hwq < num_online_cpu: If we ensure that a) the number of online cpus
    per hwq is (roughly) identical we won't have a performance impact.
    As a bonus we should strive to have the number of offline cpus
    distributed equally on each hwq.

Of course, that doesn't take into accound NUMA locality; with NUMA 
locality you would need to ensure to have at least one CPU per NUMA node
mapped to each hwq. Which actually would impose a lower limit on the
number (and granularity!) of hwqs (namely the number of NUMA nodes), but 
that's fair, I guess.

But this really can be delegated to later patches; initially we really
should identify which drivers might have issues with CPU hotplug,
and at the very least issue a warning for these drivers.

> I am not against this, just saying it would change the existing
> behavior.
> 

Oh, sure. No-one (except lpfc on Power) is testing CPU hotplug actively.

>> Also some drivers which are using irq affinity (eg aacraid, lpfc) are
>> missing from these conversions. Why?
> 
> I was not aware of aacraid. I started to work on lpfc and well let's put
> it this way, it's complicated. lpfc needs a lot of work to make it
> isolcpus aware.

Yeah, I know. Sorry ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich