From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AE7371F9A81 for ; Fri, 3 Jan 2025 10:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735899023; cv=none; b=ObT638/Ed50iPizI+rjapm4wWcNewE3obsQa41egUjbukGev7fYy2+1nJyOIEsW3DSjtyy9HzxsXYQEHqwpa1QigF5vyaZRdJhJPAnKspHyqlo8VR2R4+nSedRrMGUrDpAf4hrLrFQ4RA1M6XSEsK2Yv4fs3RSX7T5kXDTKLhJs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735899023; c=relaxed/simple; bh=0r/5pWgUjy5Xjpi/5W48FUJgaSl3fKVQysS5loIS2wI=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=uAy58er0Q+/9qErrUCWe/vGKASM2ahKnj1B5tCf080Abag8LYy/uNhEsEuXfPQisWneI4vwUAJ8ilyKujWQcbl6Uxl5iGjXDX8fVJiqGeQYPkmvG55SkqZvimhw74c1oTKji+b9AIZGHDVrd1GlZ74ctxAyOwudpXIapbbWRTBk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 391CA1480; Fri, 3 Jan 2025 02:10:49 -0800 (PST) Received: from [10.162.41.37] (K4MQJ0H1H2.blr.arm.com [10.162.41.37]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0F2E03F6A8; Fri, 3 Jan 2025 02:10:10 -0800 (PST) Message-ID: <4751a528-67ac-4fb2-a0d4-441d3efd8d08@arm.com> Date: Fri, 3 Jan 2025 15:40:07 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH 12/12] selftests/mm: khugepaged: Enlighten for mTHP collapse To: Ryan Roberts , Zi Yan , akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20241216165105.56185-1-dev.jain@arm.com> <20241216165105.56185-13-dev.jain@arm.com> <7098654a-776d-413b-8aca-28f811620df7@arm.com> <07b40938-029b-42ef-a273-46b504e67eda@arm.com> <350a4f5b-c4f1-43cc-ad93-ba02c454f19c@arm.com> Content-Language: en-US From: Dev Jain In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 02/01/25 5:13 pm, Ryan Roberts wrote: > On 30/12/2024 16:36, Zi Yan wrote: >> On Mon Dec 30, 2024 at 2:09 AM EST, Dev Jain wrote: >>> On 20/12/24 4:35 pm, Ryan Roberts wrote: >>>> On 18/12/2024 09:50, Dev Jain wrote: >>>>> On 18/12/24 2:33 pm, Ryan Roberts wrote: >>>>>> On 16/12/2024 16:51, Dev Jain wrote: >>>>>>> One of the testcases triggers a CoW on the 255th page (0-indexing) with >>>>>>> max_ptes_shared = 256. This leads to 0-254 pages (255 in number) being unshared, >>>>>>> and 257 pages shared, exceeding the constraint. Suppose we run the test as >>>>>>> ./khugepaged -s 2. Therefore, khugepaged starts collapsing the range to order-2 >>>>>>> folios, since PMD-collapse will fail due to the constraint. >>>>>>> When the scan reaches 254-257 PTE range, because at least one PTE in this range >>>>>>> is writable, with other 3 being read-only, khugepaged collapses this into an >>>>>>> order-2 mTHP, resulting in 3 extra PTEs getting unshared. After this, we >>>>>>> encounter >>>>>>> a 4-sized chunk of read-only PTEs, and mTHP collapse stops according to the >>>>>>> scaled >>>>>>> constraint, but the number of shared PTEs have now come under the constraint for >>>>>>> PMD-sized THPs. Therefore, the next scan of khugepaged will be able to collapse >>>>>>> this range into a PMD-mapped hugepage, leading to failure of this subtest. Fix >>>>>>> this by reducing the CoW range. >>>>>> Is this description essentially saying that it's now possible to creep towards >>>>>> collapsing to a full PMD-size block over successive scans due to rounding errors >>>>>> in the scaling? Or is this just trying an edge case and the problem doesn't >>>>>> generalize? >>>>> For this case, max_ptes_shared for order-2 is 256 >> (9 - 2) = 2, without >>>>> rounding errors. We cannot >>>>> really get a rounding problem because we are rounding down, essentially either >>>>> keep the restriction >>>>> same, or making it stricter, as we go down the orders. >>>>> >>>>> But thinking again, this behaviour may generalize: essentially, let us say that >>>>> the distribution >>>>> of none ptes vs filled ptes is very skewed for the PMD case. In a local region, >>>>> this distribution >>>>> may not be skewed, and then an mTHP collapse will occur, making this entire >>>>> region uniform. Over time this >>>>> may keep happening and then the region will become globally uniform to come >>>>> under the PMD-constraint >>>>> on max_ptes_none, and eventually PMD-collapse will occur. Which may beg the >>>>> question whether we want >>>>> to detach khugepaged orders from mTHP sysfs settings. >>>> We want to avoid new user controls at all costs, I think. >>>> >>>> I think an example of the problem you are describing is: Let's say we start off >>>> with all mTHP orders enabled and max_ptes_none is 50% (so 256). We have a 2M VMA >>>> aligned over a single PMD. The first 2 4K pages of the VMA are allocated. >>>> >>>> khugepaged will scan this VMA and decide to collapse the first 4 PTEs to a >>>> single order-2 (16K) folio; that's allowed because 50% of the PTEs were none. >>>> But now on the next scan, 50% of the first 8 PTEs are none so it will collapse >>>> to 32K. Then on the next scan it will collapse to 64K, and so on all the way to >>>> 2M. So by faulting in 2 pages originally we have now collapsed to 2M dispite the >>>> control trying to prevent it, and we have done it in a very inefficient way. >>>> >>>> If max_ptes_none was 75% you would only need every other order enabled (I think?). >>>> >>>> In practice perhaps it's not a problem because you are only likely to have 1 or >>>> 2 mTHP sizes enabled. But I wonder if we need to think about how to protect from >>>> this "creep"? >>>> >>>> Perhaps only consider a large folio for collapse into a larger folio if it >>>> wasn't originally collapsed by khugepaged in the first place? That would need a >>>> folio flag... and I suspect that will cause other edge case issues if we think >>>> about it for 5 mins... >>>> >>>> Another way of thinking about it is; if all the same mTHP orders were enabled at >>>> fault time (and the allocation suceeded) we would have allocated the largest >>>> order anyway, so the end states are the same. But the large number of >>>> incremental collapses that khugepaged will perform feel like a problem. >>>> >>>> I'm not quite sure what the answer is. >>> Can't really think of anything else apart from decoupling khugepaged sysfs from mTHP sysfs... >> One (not so effective) workaround is to add a VMA flag to make >> khugepaged to skip scanning a VMA that khugepaged has collapsed before >> and reset the flag in a future page fault. This would prevent khugepaged >> from doing this "creep" collapse behavior until the page tables covered >> by the VMA is changed. This is not perfect, since the page faults might >> not change the aforementioned region and later khugepaged can still >> perform the "creep". >> > I guess what you really want is a bitmap with a bit per page in the VMA to tell > you whether it exists due to fault or collapse. But... yuk... > > It looks like Ubuntu at least is not modifying the default max_ptes_none, which > is 511. So for general purpose distros I guess we won't see this issue in > practice because we will always collapse to the largest enabled size. Will still see this problem for max_ptes_shared = 256... We can set this to 255, so that the fraction progress as follows: 255/512 -> 127/256 -> 63/128 -> 31/64 -> 15/32 -> 7/16 -> 3/8 -> 1/4 -> 0/2 This is the best possible fractional decrease we can get since we always end up on an odd number and lose a 1 due to rounding. The only real solution seems to be to track whether the PTE/page we have is original or collapsed. > > So with that in mind, perhaps Zi's suggested single VM flag idea will be good > enough? > > Thanks, > Ryan > >