From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751585AbdJWQro (ORCPT ); Mon, 23 Oct 2017 12:47:44 -0400 Received: from mout.gmx.net ([212.227.15.19]:50996 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751429AbdJWQrm (ORCPT ); Mon, 23 Oct 2017 12:47:42 -0400 Subject: Re: PROBLEM: Remapping hugepages mappings causes kernel to return EINVAL To: Michal Hocko Cc: Mike Kravetz , linux-mm@kvack.org, linux-kernel , Andrea Arcangeli , "Kirill A. Shutemov" , Vlastimil Babka References: <93684e4b-9e60-ef3a-ba62-5719fdf7cff9@gmx.de> <6b639da5-ad9a-158c-ad4a-7a4e44bd98fc@gmx.de> <5fb8955d-23af-ec85-a19f-3a5b26cc04d1@oracle.com> <20171023114210.j7ip75ewoy2tiqs4@dhcp22.suse.cz> <20171023124122.tjmrbcwo2btzk3li@dhcp22.suse.cz> <20171023161316.ajrxgd2jzo3u52eu@dhcp22.suse.cz> From: "C.Wehrmeyer" Message-ID: <93ffc1c8-3401-2bea-732a-17d373d2f24c@gmx.de> Date: Mon, 23 Oct 2017 18:46:59 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: <20171023161316.ajrxgd2jzo3u52eu@dhcp22.suse.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:Zaf05CZU3+17uHapjPONcSPrCvSauKcVScxHbuD2NP7DcnfPBWW 3uXdDZN0HqKEPogNOurf+k90kfWrKCOThsE/9c8Gvvs1Vz7olAHEfLgRnZJ2NZL0TwJEP28 M/QpwOkiukGbW1Le2JWiXVjHREBF2SGoJeXabUc4UQOG1rTIW6O6TPycsRB8m7NPRw1M3bN Q2oxk0JDmJAN0mhhOYxZw== X-UI-Out-Filterresults: notjunk:1;V01:K0:lAZmyPGne2k=:HHCZMWYCANJPqq2cdLyay0 Ukjf4CoPAqqtLD72Sy3nuaVelTqx57GB409xyHA3enJMx+PAJ7QpCsNrmUE3dRdxCFFxQgEXn teh4yTR7nQHhAKUpw4XojIMAVyXlR0NjSMbV4aDvFvNP/2ITm+CQ+7Rfy9HiV2mSrWw179yL+ 0Xf0l/99SOE7nYwtU8vSKkqelPbnMrZhxYyNuR9MihZgGBkQ5ht1/qXD+Bnn+PoTdI1mKyIzT uMFbuVbECDpAIYhRqr49zq5/rVC4uBSK8IHtvLNqTDzDG5KfgJgNyvuvsX9uRDaIBFeWZ4DYa k+hmDM69xA6I0l2NI0j8w9R3ANxUh5RHoBCNVDSONmhGDvftMZ5uHM3c1VnaCluI0VU1crarx UvIObEG2ENvs4aEanBtf0lBg9aF60eKz0eg/zt34/rVuU7xp7rmdfb2A/DPDC/4FCo9rcRmUr 7skutqhncG7OThVZZb0S+Y3L0+wgET7OVlbAdqgpXeMUDuX/PsveUP/iSFF4rYPFg7wEu1YOf 3JObJxQgEZ1i8/mhXpYAPEc4AthzU2wa5Q/sJfh7OJ3148rYpqFH+HS7YIxBpw8HwMt3nwbBk EKC7kgST7tXv1sz6OcdtpcpB7wR9cW9ZOQmQqkbwXXw8OD9+fHsnIvOPB34WyhTLaetx0vnt5 w5mMuJSK971dYLV62rBB/xRYlw6lNOsRMfeafFrCV23kIwZm8XWqRztX3arpXE7Em0Okh1ua5 xPK7u8yEZseD10ad3II31KmwlIp+qCQXvTzCi5dSQ16C67X1v+BUCyYzzgVXI2Q1jVoeKojyl 3jL+4b520iJ5w6lg5G/l+X9B2CKXDqGk/0LzWGvyoXxITNCWD0= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 23-10-17 18:13, Michal Hocko wrote: > On Mon 23-10-17 16:00:13, C.Wehrmeyer wrote: >> And just to be very sure I've added: >> >> if (madvise(buf1,ALLOC_SIZE_1,MADV_HUGEPAGE)) { >> errno_tmp = errno; >> fprintf(stderr,"madvise: %u\n",errno_tmp); >> goto out; >> } >> >> /*Make sure the mapping is actually used*/ >> memset(buf1,'!',ALLOC_SIZE_1); > > Is the buffer aligned to 2MB? When I omit MAP_HUGETLB for the flags that mmap receives - no. #define ALLOC_SIZE_1 (2 * 1024 * 1024) [...] buf1 = mmap ( NULL, ALLOC_SIZE_1, prot, /*PROT_READ | PROT_WRITE*/ flags /*MAP_PRIVATE | MAP_ANONYMOUS*/, -1, 0 ); In such a case buf1 usually contains addresses which are aligned to 4 KiBs, such as 0x7f07d76e9000. 2-MiB-aligned addresses, such as 0x7f89f5e00000, are only produced with MAP_HUGETLB - which, if I understood the documentation correctly, is not the point of THPs as they are supposed to be transparent. I'm not exactly sure how I'm supposed to force mmap to give me any other kind of address, if that is going to be your suggestion - unless I'd read the mapping configuration for the current process and find myself a spot where I can tell mmap to create a mapping for me using MAP_FIXED. But that wouldn't be transparent, either. >> /*Give me time for monitoring*/ >> sleep(2000); >> >> right after the mmap call. I've also made sure that nothing is being >> optimised away by the compiler. With a 2MiB mapping being requested this >> should be a good opportunity for the kernel, and yet when I try to figure >> out how many THPs my processes uses: >> >> $ cat /proc/21986/smaps | grep 'AnonHuge' >> >> I just end up with lots of: >> >> AnonHugePages: 0 kB >> >> And cat /proc/meminfo | grep 'Huge' doesn't change significantly as well. Am >> I just doing something wrong here, or shouldn't I trust the THP mechanisms >> to actually allocate hugepages for me? > > If the mapping is aligned properly then the rest is up to system and > availability of large physically contiguous memory blocks. I have about 5 GiBs of free memory right now, and while I can not guarantee that memory fragmentation prevents the kernel from using THP, manually reserving 256 2-MiB pages through nr_hugepages and then freeing them works just fine. Yes, after allocating them I checked if nr_hugepages actually was 256. And yet, after immediately running my program, there would be no change any of the AnonHugePages elements that smaps exports. Also (while omitting MAP_HUGETLB) buf1 remains to be aligned to 4 KiB.