Re: [PATCH] workqueue: Ensure that cpumask set for pools created after boot

[Michael Bringmann <mwb@linux.vnet.ibm.com>]

On 06/12/2017 11:14 AM, Tejun Heo wrote:
> Hello,
> 
> On Mon, Jun 12, 2017 at 09:47:31AM -0500, Michael Bringmann wrote:
>>> I'm not sure because it doesn't make any logical sense and it's not
>>> right in terms of correctness.  The above would be able to enable CPUs
>>> which are explicitly excluded from a workqueue.  The only fallback
>>> which makes sense is falling back to the default pwq.
>>
>> What would that look like?  Are you sure that would always be valid?
>> In a system that is hot-adding and hot-removing CPUs?
> 
> The reason why we're ending up with empty masks is because
> wq_calc_node_cpumask() is assuming that the possible node cpumask is
> always a superset of online (as it should).  We can trigger a fat
> warning there if that isn't so and just return false from that
> function.

What would that look like?  I should be able to test it on top of the
other changes / corrections.

>>> The only way offlining can lead to this failure is when wq numa
>>> possible cpu mask is a proper subset of the matching online mask.  Can
>>> you please print out the numa online cpu and wq_numa_possible_cpumask
>>> masks and verify that online stays within the possible for each node?
>>> If not, the ppc arch init code needs to be updated so that cpu <->
>>> node binding is establish for all possible cpus on boot.  Note that
>>> this isn't a requirement coming solely from wq.  All node affine (thus
>>> percpu) allocations depend on that.
>>
>> The ppc arch init code already records all nodes used by the CPUs visible in
>> the device-tree at boot time into the possible and online node bindings.  The
>> problem here occurs when we hot-add new CPUs to the powerpc system -- they may
>> require nodes that are mentioned by the VPHN hcall, but which were not used
>> at boot time.
> 
> We need all the possible (so, for cpus which aren't online yet too)
> CPU -> node mappings to be established on boot.  This isn't just a
> requirement from workqueue.  We don't have any synchronization
> regarding cpu <-> numa mapping in memory allocation paths either.
> 
>> I will run a test that dumps these masks later this week to try to provide
>> the information that you are interested in.
>>
>> Right now we are having a discussion on another thread as to how to properly
>> set the possible node mask at boot given that there is no mechanism to hot-add
>> nodes to the system.  The latest idea appears to be adding another property
>> or two to define the maximum number of nodes that should be added to the
>> possible / online node masks to allow for dynamic growth after boot.
> 
> I have no idea about the specifics of ppc but at least the code base
> we have currently expect all possible cpus and nodes and their
> mappings to be established on boot.

Hopefully, the new properties will fix the holes in the current implementation
with regard to hot-add.

> 
> Thanks.
> 

-- 
Michael W. Bringmann
Linux Technology Center
IBM Corporation
Tie-Line  363-5196
External: (512) 286-5196
Cell:       (512) 466-0650
mwb@linux.vnet.ibm.com