From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751643AbbJLKft (ORCPT ); Mon, 12 Oct 2015 06:35:49 -0400 Received: from mail-db3on0142.outbound.protection.outlook.com ([157.55.234.142]:40673 "EHLO emea01-db3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751041AbbJLKfr (ORCPT ); Mon, 12 Oct 2015 06:35:47 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=daniel@numascale.com; Date: Mon, 12 Oct 2015 18:21:01 +0800 From: Daniel J Blueman Subject: Re: [PATCH 3/3] x86/apic: Use smaller array for __apicid_to_node[] mapping To: Jiang Liu , Thomas Gleixner , Denys Vlasenko , Ingo Molnar CC: Len Brown , , Message-ID: <1444645261.25258.1@outlook-emeawest.office365.com> In-Reply-To: <5617DEC4.5060705@linux.intel.com> References: <1443813145-29102-1-git-send-email-dvlasenk@redhat.com> <1443813145-29102-3-git-send-email-dvlasenk@redhat.com> <5617DEC4.5060705@linux.intel.com> X-Mailer: geary/0.10.0 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8"; format=flowed X-Originating-IP: [115.42.131.38] X-ClientProxiedBy: AM3PR03CA044.eurprd03.prod.outlook.com (10.141.191.172) To AM4PR07MB1298.eurprd07.prod.outlook.com (25.164.81.156) X-Microsoft-Exchange-Diagnostics: 1;AM4PR07MB1298;2:ZhN01+Mc7CGrMmYH9wOc7rvke4frfT2Dq162aQfGDKGFZjwf2nw0YFT0UIeIcXWbeiyih6viBceuA43xAIHgOYlYjQ6DXT8s/9tQZi20WsST8A21omZJuGDAcheoYowZzXgWF2M70dxioJWJhYpcM8IuQP1OSwCDgvLr1rWbY2k=;3:nSnoK7iWRnBVjBqooeFmg0x9VTvwK2F34WGhR48fF1kP2J+4hh4bIi2qD+feG7tEHpkV3A8J/UpZJQlb92LKy8I93naMdeHGihEWuznRjy5kR60E2QL+2x7Qx55Cm+vdrUxPJQ5mo34O0Yp/cPx2ZQ==;25:m4FNOt36jz5XXGZg1t1xntUZ08Iv13y82MmcCBhl/02FD9D8TCwDsjpvQ8p6wk7SSnt4FRl1C9CIWjsxsYgF7hKEFEIZvThyFGIbQwT3XYM8c1tyU/3d/8ylosflMzzXYBm7BcGUpXr/1PBI1bJ7DlRZvqrmnh28oq2S8r6t6iNYpiFQ44IWQz6mochm8g+7BBp3iXyZsPPmev9y8xmk2UCKNaPxWLhLC5nB2lz12gavMAFZlsjZnD260v34apEWR2Ssna0gPdlSmT1EqmZO2w== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM4PR07MB1298; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(520078)(3002001);SRVR:AM4PR07MB1298;BCL:0;PCL:0;RULEID:;SRVR:AM4PR07MB1298; X-Microsoft-Exchange-Diagnostics: 1;AM4PR07MB1298;4:q5La5v+4/J1TC+OoG69Xah3A4vwty8kGcFhMs5RxXcRFM19xAlk6jXoFGS4MSUrH3InkdBc5iApUyukQTsr75nlKdDJ0JrJQWlXw340I3I1BWjnI4oppXPesIxPxvFPjWfNzYE+e3JQV7pEgvyOtYQ86war4B9upfCf0HScEKow6ANO36O7x9tKnK07mkFMLg9qESzyEQsVOq2dICcAiFeAZUJ8i+7sSXs54Tl7bYb2v36r/awiauAfrIgHGMR8UCUKQGWmDKN/m6t9IT7JNAopGiYYBLK2KReln31yhjE/G41SravgdjzUBoG7tiV4g9F9i28IQWj1F/m3PKYO1fz8gh4xy6vjCXAzfQd0cbVM= X-Forefront-PRVS: 0727122FC6 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(979002)(6009001)(6049001)(199003)(243025005)(189002)(377454003)(479174004)(24454002)(76176999)(42186005)(5008740100001)(23676002)(64706001)(88526003)(2950100001)(5001960100002)(50986999)(101416001)(86362001)(189998001)(77096005)(47776003)(40100003)(81156007)(5004730100002)(97736004)(50226001)(92566002)(46102003)(87976001)(15975445007)(5007970100001)(66066001)(19580395003)(105586002)(117156001)(5001770100001)(106356001)(50466002)(122386002)(19580405001)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:AM4PR07MB1298;H:[192.168.20.240];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtBTTRQUjA3TUIxMjk4OzIzOjhTVml4bnVYSGYva3A5bzZFck5PL2k5Vi9j?= =?utf-8?B?WHoxd2ZPRWI2UGxlNW9vZk5oS1Voem9lcDN5MHFYNEloZTk1bnN6TGJ3bHA3?= =?utf-8?B?aFVZRS9UQXR3M2o0Q0JCRC9FcTdabGpwNjRqLzhZblVBSzJyWElWbDlrcmYx?= =?utf-8?B?NFhGTmZlTE52cHBvNGdYcmFvTVFuWVpCNnRlbkpGWjlyVUR0eDRVOXBHSjZT?= =?utf-8?B?STlxSis3NGtMT29ucFk0cUZjZmx4SDdRTGkxNzJoTW5ncEt3WC9PTXJyY1VE?= =?utf-8?B?UUxrcm9XajI5KzJ5MEFPTytMdENldW5KMnhlNisrK0gya3RhVTdncHN4Rkgr?= =?utf-8?B?aVJyRkU0cE9oQnhScVkrVkFaZ28xaVNLOGI3V3pFM011S3dSQTlqY3JlQ3ZC?= =?utf-8?B?bkJNZEFhY2JGby9HR3NHV2s5N29HVk0wWWs3cS9VQnBQOTBOTWQ4UFl1amJN?= =?utf-8?B?NkVXMmoyKzJBUWFoK0xWNWZ2a1ZXeGt4L1JIYTB2V1RkNDB1V3J2cjN4M2Yr?= =?utf-8?B?VWIyWFRHUWNwTUtkVlhMSy9tNGpxWkVReHp6cnR6cXNwTTZjNEJQbTdWQStG?= =?utf-8?B?YUtva2tFWTZXRHcxM1FJbldnaG9oZVo5ZU1TUy9heThCenZtWDNqYkoxanZs?= =?utf-8?B?cDBiTGpNUTZEaVNjUDdoWjA1ZkFtSDRJb1ZpSnNDWHBXNGN3SHJBbDZOZzhI?= =?utf-8?B?SWhLcmxJNUV3YU1SU1dnMVNGTWFjYnV4enNSczc1eGozTzdKVGFlWGJZci90?= =?utf-8?B?ZXJBQjhlMlhSY2RlZnhPTmNKMDJvZGJVaEd4ZUQwMjRzSUEydytpTnFaajRH?= =?utf-8?B?TVpua3Zaemh0U2kyMUZQdUl0cFVsNFFXWkF6b2pFM25MamRRRFhmckFhcjVN?= =?utf-8?B?UWVrSFZvZkcwL0NoODRxOWlSTU9iQzhwZGZZVVd2b051SDZhQ21kdjdNVGpr?= =?utf-8?B?RnJKYXhKRXhFQU5rRC9LbmROK1BUUURjQWZTU3l1Z1FoaFJndGp0bmU5WGxC?= =?utf-8?B?WDE4Y0VXb2YrODFEaTF2M1ZzVm4wbjdkMlc4LzROT1dKMmxqUVJ5R29NWVBp?= =?utf-8?B?MDM5OXRTVTZHWHpIbGVVZzJWcUFQTTdLZmw4UjlMc2lmMk1sbU1rb0dmUHND?= =?utf-8?B?T2ZyckNRNDQ0aUwyU1hFdUxiU2M0V1pVQUpHVlUrdUx6OWZoQmpjendDQ2FE?= =?utf-8?B?dzQ4Z0c2Q3k2emdNTzZrRmJhWlVGTHJBaTAvNVBYSVdma3hPWHVJWVl0M1NT?= =?utf-8?B?VUhQOUtWRno0NFBuUCtvUEhhbUdJRVhlNUpiR2pEbVd4M25nbndIaGo3b1ox?= =?utf-8?B?NEt4WTIrRGFHRUpCMXpLVng4dVh2OUFPdWsxWkZ2MEJydnlCZmtmQnJyNE9m?= =?utf-8?B?V3A2bzQwUVJhUklTd0dYN0tnYTFWTFcxa1l0bWdCVkJvbGhkeWU5eHRsVUNu?= =?utf-8?B?aEIrdW40bk5NUUhxVXFLbG8vb1J4YTA0aFlaU21nMmxHV2JJcmxJRjZGWnl3?= =?utf-8?B?amRXb0JKM0RLT2RYSG15SExGeVN3WlI1eXNEMEFJVXpvZ0xLWlR0SERaZEtV?= =?utf-8?B?cnZ2V05aWHVzb2xtc3RUNkVrcThDRTMreTlGV0cwSUZuZmJGamRnZUV4UmMy?= =?utf-8?B?dmFxVDE4Nk1LWlFMS3d2dnV1V0c4cUc1bGNnYjZlYkFiSmlnVVVSU1lBPT0=?= X-Microsoft-Exchange-Diagnostics: 1;AM4PR07MB1298;5:xBIAwCy2Zuv6cScA5hOgv63OezNbpUhtMl5BzDEN1hXdvl19McCCkCLiQIBcARBamQPQnKPTrCJ0/pV6GR9rll7JorMdrT63moYUfUOEQFf2pBKCh3TE9EcNHC0obfN4g3N8N4YCeWr70SFCDDD1kg==;24:BO1n0ugscGmpW+ss9sLsYnPRpUxppJNSCusisLp4SaMTnkXt3SzcrsQTwr06hIqEGKl5vy6u+Ex0DcZWIHF1vbZmNtwqUaDT1G5ciLCK4T8=;20:mGlPSIinffr67Gy5P3g9ZE052/6t00WFlAyjh1DHGhJKDn2D6pkhEPsPWWl47zxETfdiq5MiBhtzf9KZi6bYfQ== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: numascale.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2015 10:21:06.9846 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR07MB1298 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 9, 2015 at 11:35 PM, Jiang Liu wrote: > On 2015/10/3 3:12, Denys Vlasenko wrote: >> From: Daniel J Blueman >> >> The Intel x2APIC spec states the upper 16-bits of APIC ID is the >> cluster ID [1, p2-12], intended for future distributed systems. >> Beyond >> the legacy 8-bit APIC ID, Numascale NumaConnect uses 4-bits for the >> position of a server on each axis of a multi-dimension torus; SGI >> NUMAlink also structures the APIC ID space. >> >> Instead, define an array based on NR_CPUs to achieve a 1:1 mapping >> and >> perform linear search; this addresses the binary bloat and the >> present >> artificial APIC ID limits. With CONFIG_NR_CPUS=256: >> >> $ size vmlinux vmlinux-patched >> text data bss dec hex filename >> 18232877 1849656 2281472 22364005 1553f65 vmlinux >> 18233034 1786168 2281472 22300674 1544802 vmlinux-patched >> >> That is, ~64 kbytes less data. >> >> Works peachy on a 256-core system with a 20-bit APIC ID space, and >> on a >> 48-core legacy 8-bit APIC ID system. If we care, I can make >> numa_cpu_node O(1) lookup for typical cases. >> >> Signed-off-by: Daniel J Blueman >> CC: Ingo Molnar >> CC: Daniel J Blueman >> CC: Jiang Liu >> CC: Thomas Gleixner >> CC: Len Brown >> CC: x86@kernel.org >> CC: linux-kernel@vger.kernel.org >> >> [1] >> >> http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf >> --- >> >> I added forgotten change in arch/x86/mm/numa_emulation.c (Denys) >> >> arch/x86/include/asm/numa.h | 13 +++++++------ >> arch/x86/kernel/cpu/amd.c | 8 ++++---- >> arch/x86/mm/numa.c | 31 +++++++++++++++++++++++-------- >> arch/x86/mm/numa_emulation.c | 6 +++--- >> 4 files changed, 37 insertions(+), 21 deletions(-) >> >> diff --git a/arch/x86/include/asm/numa.h >> b/arch/x86/include/asm/numa.h >> index c2ecfd0..33becb8 100644 >> --- a/arch/x86/include/asm/numa.h >> +++ b/arch/x86/include/asm/numa.h >> @@ -17,6 +17,11 @@ >> */ >> #define NODE_MIN_SIZE (4*1024*1024) >> >> +struct apicid_to_node { >> + int apicid; >> + s16 node; >> +}; >> + >> extern int numa_off; >> >> /* >> @@ -27,17 +32,13 @@ extern int numa_off; >> * should be accessed by the accessors - set_apicid_to_node() and >> * numa_cpu_node(). >> */ >> -extern s16 __apicid_to_node[MAX_LOCAL_APICID]; >> +extern struct apicid_to_node __apicid_to_node[NR_CPUS]; > Hi Denys and Daniel, > I still have some concerns about limiting the array to NR_CPUS. > __apicid_to_node are populated according to the order that CPUs are > listed in ACPI SRAT table. And CPU IDs are allocated according to the > order that CPUs are listed in ACPI MADT(APIC) order. So it may cause > trouble if: > 1) system has more than NR_CPUS CPUs > 2) CPUs are listed in different order in SRAT and MADT tables. Another approach which may be suitable without changing SRAT parsing to be after the memory allocator is up, is to exploit the associativity of the bottom APIC ID bits. We'd have a searchable static array based on NUMA_SHIFT and use the bit-shift encoded in the MSRs. That said, this may run into the issue Jiang cited albeit with CONFIG_NUMA_SHIFT. Perhaps the constraints or risk of restructuring SRAT parsing aren't worth the payoff? Finally, the only alternative is as the current mapping is initialised in numa_init, we can drop the static initialisation and move the 64KB to the BSS to avoid bloating the binary image, but this may not achieve the initial goal of runtime footprint reduction. Daniel