From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751493AbaHLFBo (ORCPT ); Tue, 12 Aug 2014 01:01:44 -0400 Received: from mtaout24.012.net.il ([80.179.55.180]:41944 "EHLO mtaout24.012.net.il" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750765AbaHLFBn (ORCPT ); Tue, 12 Aug 2014 01:01:43 -0400 Date: Tue, 12 Aug 2014 08:01:41 +0300 From: Oren Twaig Subject: x86: vmalloc and THP X-012-Sender: ira_oren@012.net.il To: "linux-kernel@vger.kernel.org" , linux-mm@kvack.org Cc: "Shai Fultheim (Shai@ScaleMP.com)" Message-id: <53E99FB5.1020506@scalemp.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit X-Antivirus: avast! (VPS 140811-1, 08/11/2014), Outbound message X-Antivirus-Status: Clean User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, Does memory allocated using vmalloc() will be mapped using huge pages either directly or later by THP ? If not, is there any fast way to change this behavior ? Maybe by changing the granularity/alignment of such allocations to allow such mapping ? Thanks, Oren Twaig. --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752658AbaHLGIH (ORCPT ); Tue, 12 Aug 2014 02:08:07 -0400 Received: from mta-out1.inet.fi ([62.71.2.193]:35647 "EHLO jenni2.inet.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751444AbaHLGIF (ORCPT ); Tue, 12 Aug 2014 02:08:05 -0400 Date: Tue, 12 Aug 2014 09:07:45 +0300 From: "Kirill A. Shutemov" To: Oren Twaig Cc: "linux-kernel@vger.kernel.org" , linux-mm@kvack.org, "Shai Fultheim (Shai@ScaleMP.com)" Subject: Re: x86: vmalloc and THP Message-ID: <20140812060745.GA7987@node.dhcp.inet.fi> References: <53E99F86.5020100@scalemp.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <53E99F86.5020100@scalemp.com> User-Agent: Mutt/1.5.22.1 (2013-10-16) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote: > plain/text, please. >Hello, > >Does memory allocated using vmalloc() will be mapped using huge >pages either directly or later by THP ? No. It's neither aligned properly, nor physically contiguous. >If not, is there any fast way to change this behavior ? Maybe by >changing the granularity/alignment of such allocations to allow such >mapping ? What's the point to use vmalloc() in this case? -- Kirill A. Shutemov From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752377AbaHLM2y (ORCPT ); Tue, 12 Aug 2014 08:28:54 -0400 Received: from mail-pa0-f54.google.com ([209.85.220.54]:63997 "EHLO mail-pa0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751639AbaHLM2x (ORCPT ); Tue, 12 Aug 2014 08:28:53 -0400 Message-ID: <1407846532.10122.66.camel@edumazet-glaptop2.roam.corp.google.com> Subject: Re: x86: vmalloc and THP From: Eric Dumazet To: "Kirill A. Shutemov" Cc: Oren Twaig , "linux-kernel@vger.kernel.org" , linux-mm@kvack.org, "Shai Fultheim (Shai@ScaleMP.com)" Date: Tue, 12 Aug 2014 05:28:52 -0700 In-Reply-To: <20140812060745.GA7987@node.dhcp.inet.fi> References: <53E99F86.5020100@scalemp.com> <20140812060745.GA7987@node.dhcp.inet.fi> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3-0ubuntu6 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote: > On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote: > >Does memory allocated using vmalloc() will be mapped using huge > >pages either directly or later by THP ? > > No. It's neither aligned properly, nor physically contiguous. > > >If not, is there any fast way to change this behavior ? Maybe by > >changing the granularity/alignment of such allocations to allow such > >mapping ? > > What's the point to use vmalloc() in this case? Look at various large hashes we have in the system, all using vmalloc() : [ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) [ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes) I would imagine a performance difference if we were using hugepages. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753199AbaHLPBz (ORCPT ); Tue, 12 Aug 2014 11:01:55 -0400 Received: from mta-out1.inet.fi ([62.71.2.193]:42880 "EHLO jenni1.inet.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752724AbaHLPBy (ORCPT ); Tue, 12 Aug 2014 11:01:54 -0400 Date: Tue, 12 Aug 2014 18:01:31 +0300 From: "Kirill A. Shutemov" To: Eric Dumazet Cc: Oren Twaig , "linux-kernel@vger.kernel.org" , linux-mm@kvack.org, "Shai Fultheim (Shai@ScaleMP.com)" Subject: Re: x86: vmalloc and THP Message-ID: <20140812150131.GA12187@node.dhcp.inet.fi> References: <53E99F86.5020100@scalemp.com> <20140812060745.GA7987@node.dhcp.inet.fi> <1407846532.10122.66.camel@edumazet-glaptop2.roam.corp.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1407846532.10122.66.camel@edumazet-glaptop2.roam.corp.google.com> User-Agent: Mutt/1.5.22.1 (2013-10-16) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 12, 2014 at 05:28:52AM -0700, Eric Dumazet wrote: > On Tue, 2014-08-12 at 09:07 +0300, Kirill A. Shutemov wrote: > > On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote: > > >If not, is there any fast way to change this behavior ? Maybe by > > >changing the granularity/alignment of such allocations to allow such > > >mapping ? > > > > What's the point to use vmalloc() in this case? > > Look at various large hashes we have in the system, all using > vmalloc() : > > [ 0.006856] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) > [ 0.033130] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) > [ 1.197621] TCP established hash table entries: 524288 (order: 11, 8388608 bytes) I see lower-order allocation in upstream code. Is it some distribution tweak? > I would imagine a performance difference if we were using hugepages. Okay, it's *probably* a valid point. The hash tables are only allocated with vmalloc() on NUMA system, if hashdist=1 (default on NUMA). It does it to distribute memory between nodes. vmalloc() in NUMA_NO_NODE case will allocate all memory with 0-order page allocations: no physical contiguous memory for hugepage mappings. I guess we could teach vmalloc() to interleave between nodes on PMD_SIZE chunks rather then on PAGE_SIZE if caller asks for big memory allocations. Although, I'm not sure it it would fit all vmalloc() users. We also would need to allocate PMD_SIZE-aligned virtual address range to be able to mapped allocated memory with pmds. It's *potentially* interesting research project. Any volunteers? -- Kirill A. Shutemov From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753963AbaHLQVD (ORCPT ); Tue, 12 Aug 2014 12:21:03 -0400 Received: from mtaout22.012.net.il ([80.179.55.172]:52915 "EHLO mtaout22.012.net.il" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753300AbaHLQVA (ORCPT ); Tue, 12 Aug 2014 12:21:00 -0400 Date: Tue, 12 Aug 2014 19:20:52 +0300 From: Oren Twaig Subject: Re: x86: vmalloc and THP In-reply-to: <20140812060745.GA7987@node.dhcp.inet.fi> X-012-Sender: ira_oren@012.net.il To: "Kirill A. Shutemov" Cc: "linux-kernel@vger.kernel.org" , linux-mm@kvack.org, "Shai Fultheim (Shai@ScaleMP.com)" Message-id: <53EA3EE4.6090100@scalemp.com> MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7bit X-Antivirus: avast! (VPS 140812-0, 08/12/2014), Outbound message X-Antivirus-Status: Clean References: <53E99F86.5020100@scalemp.com> <20140812060745.GA7987@node.dhcp.inet.fi> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Kirill, I saw the thread has developed nicely :), still - wanted to answer your question below. On 8/12/2014 9:07 AM, Kirill A. Shutemov wrote: > On Tue, Aug 12, 2014 at 08:00:54AM +0300, Oren Twaig wrote: >> > plain/text, please. Yes - noticed the html, sent again in plain text. >> If not, is there any fast way to change this behavior ? Maybe by >> changing the granularity/alignment of such allocations to allow such >> mapping ? > What's the point to use vmalloc() in this case? I've noticed that some lock/s are using linear addresses which are located at 0xffffc901922b4500 and from what I understand from mm.txt (kernel 3.0.101): *ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space *So I'm not sure who/how/why this lock got allocated there, but obviously it is using that linear set. No ? > --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754288AbaHLVlP (ORCPT ); Tue, 12 Aug 2014 17:41:15 -0400 Received: from mta-out1.inet.fi ([62.71.2.193]:48525 "EHLO jenni1.inet.fi" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751415AbaHLVlO (ORCPT ); Tue, 12 Aug 2014 17:41:14 -0400 Date: Wed, 13 Aug 2014 00:40:51 +0300 From: "Kirill A. Shutemov" To: Oren Twaig Cc: "linux-kernel@vger.kernel.org" , linux-mm@kvack.org, "Shai Fultheim (Shai@ScaleMP.com)" Subject: Re: x86: vmalloc and THP Message-ID: <20140812214051.GA17497@node.dhcp.inet.fi> References: <53E99F86.5020100@scalemp.com> <20140812060745.GA7987@node.dhcp.inet.fi> <53EA3EE4.6090100@scalemp.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <53EA3EE4.6090100@scalemp.com> User-Agent: Mutt/1.5.22.1 (2013-10-16) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 12, 2014 at 07:20:52PM +0300, Oren Twaig wrote: > >What's the point to use vmalloc() in this case? > I've noticed that some lock/s are using linear addresses which are > located at 0xffffc901922b4500 and from what I understand > from mm.txt (kernel 3.0.101): > *ffffc90000000000 - ffffe8ffffffffff (=45 bits) vmalloc/ioremap space > > *So I'm not sure who/how/why this lock got allocated there, but obviously > it is using that linear set. No ? It would be nice to know what lock it was, but nothing is inherently wrong with lock in vmalloc space. -- Kirill A. Shutemov