From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751581AbdFFOeP (ORCPT ); Tue, 6 Jun 2017 10:34:15 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:57247 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751440AbdFFOeL (ORCPT ); Tue, 6 Jun 2017 10:34:11 -0400 Subject: Re: [PATCH v2] workqueue: Fix edge cases for calc of pool's cpumask To: Tejun Heo Cc: Lai Jiangshan , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Michael Bringmann from Kernel Team , Nathan Fontenot References: <735de0b9-13bb-e245-397c-2d12ca03f833@linux.vnet.ibm.com> <20170606142009.GA18318@htj.duckdns.org> From: Michael Bringmann Organization: IBM Linux Technology Center Date: Tue, 6 Jun 2017 09:34:05 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.0 MIME-Version: 1.0 In-Reply-To: <20170606142009.GA18318@htj.duckdns.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 x-cbid: 17060614-0056-0000-0000-0000037F07A3 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007183; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000212; SDB=6.00870974; UDB=6.00433149; IPR=6.00650967; BA=6.00005402; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015719; XFM=3.00000015; UTC=2017-06-06 14:34:08 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17060614-0057-0000-0000-000007B50DCB Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-06-06_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1706060245 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 06/06/2017 09:20 AM, Tejun Heo wrote: > Hello, Michael. > > It would have been better to continue debugging in the prev thread. > This still seems incorrect for the same reason as before. > > On Tue, Jun 06, 2017 at 09:09:40AM -0500, Michael Bringmann wrote: >> On NUMA systems with dynamic processors, the content of the cpumask >> may change over time. As new processors are added via DLPAR operations, >> workqueues are created for them. Depending upon the order in which CPUs >> are added/removed, we may run into problems with the content of the >> cpumask used by the workqueues. This patch deals with situations where >> the online cpumask for a node is a proper superset of possible cpumask >> for the node. It also deals with edge cases where the order in which >> CPUs are removed/added from the online cpumask may leave the set for a >> node empty, and require execution by CPUs on another node. >> >> In these and other cases, the patch attempts to ensure that a valid, >> usable cpumask is used to set up newly created pools for workqueues. >> >> Signed-off-by: Tejun Heo & Michael Bringmann > > Heh, you can't add sob's for other people. For partial attributions, > you can just note in the description. Sorry for the error. > >> diff --git a/kernel/workqueue.c b/kernel/workqueue.c >> index c74bf39..460de61 100644 >> --- a/kernel/workqueue.c >> +++ b/kernel/workqueue.c >> @@ -3366,6 +3366,9 @@ static struct worker_pool *get_unbound_pool(const struct workqueue_attrs *attrs) >> copy_workqueue_attrs(pool->attrs, attrs); >> pool->node = target_node; >> >> + if (!cpumask_weight(pool->attrs->cpumask)) >> + cpumask_copy(pool->attrs->cpumask, cpumask_of(smp_processor_id())); > > So, this is still wrong. It only catches if something has gone wrong before. The alternative in this case would be, BUG(!cpumask_weight(pool->attrs->cpumask)); > >> /* >> * no_numa isn't a worker_pool attribute, always clear it. See >> * 'struct workqueue_attrs' comments for detail. >> @@ -3559,13 +3562,13 @@ static struct pool_workqueue *alloc_unbound_pwq(struct workqueue_struct *wq, >> * stable. >> * >> * Return: %true if the resulting @cpumask is different from @attrs->cpumask, >> - * %false if equal. >> + * %false if equal. On %false return, the content of @cpumask is undefined. >> */ >> static bool wq_calc_node_cpumask(const struct workqueue_attrs *attrs, int node, >> int cpu_going_down, cpumask_t *cpumask) >> { >> if (!wq_numa_enabled || attrs->no_numa) >> - goto use_dfl; >> + return false; >> >> /* does @node have any online CPUs @attrs wants? */ >> cpumask_and(cpumask, cpumask_of_node(node), attrs->cpumask); >> @@ -3573,15 +3576,13 @@ static bool wq_calc_node_cpumask(const struct workqueue_attrs *attrs, int node, >> cpumask_clear_cpu(cpu_going_down, cpumask); >> >> if (cpumask_empty(cpumask)) >> - goto use_dfl; >> + return false; >> >> /* yeap, return possible CPUs in @node that @attrs wants */ >> cpumask_and(cpumask, attrs->cpumask, wq_numa_possible_cpumask[node]); >> - return !cpumask_equal(cpumask, attrs->cpumask); >> >> -use_dfl: >> - cpumask_copy(cpumask, attrs->cpumask); >> - return false; >> + return !cpumask_empty(cpumask) && >> + !cpumask_equal(cpumask, attrs->cpumask); > > And this part doesn't really change that. > > CPUs going offline or online shouldn't change their relation to > wq_numa_possible_cpumask. I wonder whether the arch code is changing > CPU id <-> NUMA node mapping on CPU on/offlining. x86 used to do that > too and got recently modified. Can you see whether that's the case? The but that I see does not appear to be related to changing of CPU/Node mapping -- they are not changing their place when going offline/online. Rather new CPUs are being hot-added to the system (i.e. they were not present at boot), and the node to which they are being added had no CPUs at boot. > > Thanks. > Thanks. -- Michael W. Bringmann Linux Technology Center IBM Corporation Tie-Line 363-5196 External: (512) 286-5196 Cell: (512) 466-0650 mwb@linux.vnet.ibm.com