From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751529AbdG0UPw (ORCPT ); Thu, 27 Jul 2017 16:15:52 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:44141 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751125AbdG0UPv (ORCPT ); Thu, 27 Jul 2017 16:15:51 -0400 Subject: Re: [PATCH v6] workqueue: Fix edge cases for calc of pool's cpumask To: Tejun Heo Cc: Lai Jiangshan , linux-kernel@vger.kernel.org, nfont@linux.vnet.ibm.com References: <5c421712-fb38-3020-d3fc-b9bdea792cd3@linux.vnet.ibm.com> <20170727183148.GH742618@devbig577.frc2.facebook.com> <20170727192410.GI742618@devbig577.frc2.facebook.com> From: Michael Bringmann Organization: IBM Linux Technology Center Date: Thu, 27 Jul 2017 15:15:47 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.0 MIME-Version: 1.0 In-Reply-To: <20170727192410.GI742618@devbig577.frc2.facebook.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 17072720-0020-0000-0000-00000C73DB2C X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007437; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000214; SDB=6.00893782; UDB=6.00446868; IPR=6.00673943; BA=6.00005495; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00016413; XFM=3.00000015; UTC=2017-07-27 20:15:50 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17072720-0021-0000-0000-00005D735B2C Message-Id: <7c21c88d-6ce3-ae1a-d257-3042efa1bddf@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-07-27_11:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1706020000 definitions=main-1707270314 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Revised text: There is an underlying assumption in many layers / modules of the Linux system that CPU <-> node mapping is static. This is despite the presence of features like NUMA and 'hotplug' that support the dynamic addition/ removal of fundamental system resources like CPUs and memory. PowerPC systems, however, do provide extensive features for the dynamic change of resources available to a system. Currently, there is little or no synchronization protection around the updating of the CPU <-> node mapping, and the export/update of this information for other layers / modules. In systems which can change this mapping during 'hotplug', like PowerPC, the information is changing underneath all layers that might reference it. This patch attempts to ensure that a valid, usable cpumask attribute is used by the workqueue infrastructure when setting up new resource pools. It prevents a crash that has been observed when an 'empty' cpumask is passed along to the worker/task scheduling code. It is intended as an intermediate fix until a more fundamental review and correction of the issue can be done. On 07/27/2017 02:24 PM, Tejun Heo wrote: > Hello, Michael. > > On Thu, Jul 27, 2017 at 02:07:53PM -0500, Michael Bringmann wrote: >> The problem lies with the ordering of events with respect to the order in >> which we add (or remove) CPUs to NUMA systems, and make use of that knowledge. > > Isn't the root cause that the upper layers including workqueue expect > cpu <-> node mapping to be static but powerpc doesn't follow that? I > don't get why ordering matters here. > >> The CPUs present are assigned to nodes, and workqueues and their infrastructure >> are created to use the CPUs in a node. Workqueues are created at boot time >> and updated or created as CPUs are added or removed. However, there is little >> or no synchronization or ordering of these events, and the data structures > > What I meant was that there's no synchronization construct protection > cpu <-> node mapping. If arch code changes it during hot plug, it's > changing it underneath anybody who might be using that association. > >> mapping CPUs to nodes may not be updated before the workqueue infrastructure >> is built for a node. Thus we have the possibility of an invalid CPU mask >> attribute being attached to a newly created workqueue before the CPUs have >> been properly registered and published to a node. >> >> This patch attempts to provide a partial ordering of events within workqueue >> by delaying the use of newly calculated CPU masks as the value for a workqueue >> attribute until they have valid content. Instead the workqueue code must delay >> creating new workqueues until this function succeeds, or it can use a previously >> calculated cpumask attribute that is known to be valid. >> >> This patch attempts to ensure that a valid, usable cpumask is used to set up >> newly created pools for workqueues. This patch provides a fix for NUMA systems >> which can add/subtract processors dynamically. The patch is expected to be an >> intermediate one while developers find and correct any underlying issues. > > And what this patch does is adding a bandaid so that we at least don't > crash immediately when this condition triggers until the arch code can > be fixed properly. > > Thanks. > -- Michael W. Bringmann Linux Technology Center IBM Corporation Tie-Line 363-5196 External: (512) 286-5196 Cell: (512) 466-0650 mwb@linux.vnet.ibm.com