From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1036252AbdEXXzP (ORCPT ); Wed, 24 May 2017 19:55:15 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:49096 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1032837AbdEXXzO (ORCPT ); Wed, 24 May 2017 19:55:14 -0400 Subject: Re: [Patch 2/2]: powerpc/hotplug/mm: Fix hot-add memory node assoc To: Michael Ellerman , Reza Arbab References: <3bb44d92-b2ff-e197-4bdf-ec6d588d6dab@linux.vnet.ibm.com> <20170523155251.bqwc5mc4jpgzkqlm@arbab-laptop.localdomain> <1c1d70e3-4e45-b035-0e75-1b0f531c111b@linux.vnet.ibm.com> <20170523214922.bns675oqzqj4pkhc@arbab-laptop.localdomain> <87poeya4dt.fsf@concordia.ellerman.id.au> Cc: linux-kernel@vger.kernel.org, Paul Mackerras , "Aneesh Kumar K.V" , Bharata B Rao , Shailendra Singh , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org, Sebastian Andrzej Siewior From: Michael Bringmann Organization: IBM Linux Technology Center Date: Wed, 24 May 2017 18:55:08 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <87poeya4dt.fsf@concordia.ellerman.id.au> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 x-cbid: 17052423-0052-0000-0000-00000209D433 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007112; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000212; SDB=6.00865179; UDB=6.00429569; IPR=6.00644959; BA=6.00005372; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015572; XFM=3.00000015; UTC=2017-05-24 23:55:12 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17052423-0053-0000-0000-000050B33647 Message-Id: <8e2417d8-d108-2949-40f2-997d53a3f367@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-24_17:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705240157 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/24/2017 06:19 AM, Michael Ellerman wrote: > Michael Bringmann writes: > >> On 05/23/2017 04:49 PM, Reza Arbab wrote: >>> On Tue, May 23, 2017 at 03:05:08PM -0500, Michael Bringmann wrote: >>>> On 05/23/2017 10:52 AM, Reza Arbab wrote: >>>>> On Tue, May 23, 2017 at 10:15:44AM -0500, Michael Bringmann wrote: >>>>>> +static void setup_nodes(void) >>>>>> +{ >>>>>> + int i, l = 32 /* MAX_NUMNODES */; >>>>>> + >>>>>> + for (i = 0; i < l; i++) { >>>>>> + if (!node_possible(i)) { >>>>>> + setup_node_data(i, 0, 0); >>>>>> + node_set(i, node_possible_map); >>>>>> + } >>>>>> + } >>>>>> +} >>>>> >>>>> This seems to be a workaround for 3af229f2071f ("powerpc/numa: Reset node_possible_map to only node_online_map"). >>>> >>>> They may be related, but that commit is not a replacement. The above patch ensures that >>>> there are enough of the nodes initialized at startup to allow for memory hot-add into a >>>> node that was not used at boot. (See 'setup_node_data' function in 'numa.c'.) That and >>>> recording that the node was initialized. >>> >>> Is it really necessary to preinitialize these empty nodes using setup_node_data()? When you do memory hotadd into a node that was not used at boot, the node data already gets set up by >>> >>> add_memory >>> add_memory_resource >>> hotadd_new_pgdat >>> arch_alloc_nodedata <-- allocs the pg_data_t >>> ... >>> free_area_init_node <-- sets NODE_DATA(nid)->node_id, etc. >>> >>> Removing setup_node_data() from that loop leaves only the call to node_set(). If 3af229f2071f (which reduces node_possible_map) was reverted, you wouldn't need to do that either. >> >> With or without 3af229f2071f, we would still need to add something, somewhere to add new >> bits to the 'node_possible_map'. That is not being done. > > You mustn't add bits to the possible map after boot. > > That's its purpose, to tell you what nodes could ever *possibly* exist. The problem that I have been encountering is that the 'possible map' did *not* show all of the possible nodes. Rather, it showed only the nodes that were assigned memory at boot-up. If more memory were hot-added to the kernel, it could be assigned into one of the nodes that were skipped at boot. However, nothing was updating the 'node_possible_map' correctly in the kernel memory code. Reza pointed out a code change in commit 3af229f2071f that has not made it into the 4.12 checkout i.e. removing the instruction that reduces the node_possible_map. This may well be a suitable replacement for the code that I have here, and I will test it here next. > > cheers > > Later. -- Michael W. Bringmann Linux Technology Center IBM Corporation Tie-Line 363-5196 External: (512) 286-5196 Cell: (512) 466-0650 mwb@linux.vnet.ibm.com