From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA2F21B0408 for ; Wed, 18 Dec 2024 14:43:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734532994; cv=none; b=XfB6ARFuDOkew0oBFexCo+QKMt0AgVeOjx8rQ+x00TlRRc3MO+1Y0aLERspvODUfhFMY0+VEhKrowCnRMR9mPTvCcUUKukBUq2bLhQ2sFXpGpKMjwOBgm1Gt7zV1xuST3p7lJ24Z/KPqZ3ayzOArSKaWBMaXObreqra8S+s7nfs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734532994; c=relaxed/simple; bh=Zv+HrWlcLPmYcX+iJ+MjF4BC3+mamL0HZnZRtCdjtFM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=q9ppEfAyItHU+HBtIl8Ps2OGw/yvA7EcATkJg1VnFC4xsvj+YsFxuuOLiOiSELuvFPAXZ1vUCfDl5k3Ilg9eSB+YvjTdPXMWDGj9AjCH/IaoO5byKKijs+QCG9gJC1UQntv4Zhvji4uP7aI+thTVfO+tZ1t3ApRLrfMkue3adlk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=a1XB+n/p; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="a1XB+n/p" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-216395e151bso6153425ad.0 for ; Wed, 18 Dec 2024 06:43:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734532991; x=1735137791; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=R6GLWJQvuq4J3hubDyBKKbnWzxC36CZIoGbCxs2ghUQ=; b=a1XB+n/pzH/nLRQIC3yzgajal/5pnByOWObB7TwHP3MoUFAEMeIpMnbXiOf/26BYm7 Vk5cVyliLjN+nxznevzCqvu9kCLfUD92D4PnrBAQj2NXJL1Lj6cLPCQStkEIe1RBzu/4 IC1qw3YcRWQYrUOF22UtYG45kqRMtK6u8ly6QQn9ZsnIGsgv5L6kaDNcEjEcsKTzR18a FESoVvJEonmUSL6Sv370E3K3DpJQQeUozaCfsTMRlpqcCba0xLTVF9lkMPAAySNb4I1G 3PWylq1c9qx44wN+Vo3qOtvN3W7xx9ubVGzq8/7vNwSkbDtca7QjyfgG4rlOfVj43kvA aAjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734532991; x=1735137791; h=content-transfer-encoding:in-reply-to:content-language:from :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R6GLWJQvuq4J3hubDyBKKbnWzxC36CZIoGbCxs2ghUQ=; b=mJHQqrDhrUmnWon+edSUMEqNj2LeMavY8/GVfivKKn97oHW0BhQsIQx/RaPgRJtgrc eZhDpkiPIMH4W7SwmrMl4j0RtdXLcKou767iYiTu4eC17q9iNpXKbAY+37yBxJZt0q49 1L6cISr+ckMAg9TPrdQAWLLaPu2EIV5+3aQOQocEBSKsWwKyQ4xbtbJK2TnGrICv0paX Ra1i6klFr33Yeo9qLCPbmTnEX3Q+tf+thzAufvsARQhr9KacRH8tigVgvE5FA02OXzJe cATjKWo4gloz9FMncQWHD2KpVmrLyNw72Oi8UHQuNRLFDMJaNitO7fwkGWZA1NfIAuzL LSVw== X-Forwarded-Encrypted: i=1; AJvYcCVqP9c1S3TdZrJdJLIKsLxEuZ3vA1QdV8y9QU7yeFrNGZqROYFGn+7WURvjH+mKMQGt/af2pwEQ7rdhYV0=@vger.kernel.org X-Gm-Message-State: AOJu0YwcTgraXpDjFzDl7Ygf08TAKh2EidWKTHhZmFaDVAuGS88s/5GI /tqfCnUtpNkyRHiEM4SeAbdrmjvo9WskDr7JTJ/YS9y5wv4Etza1pzfN8kzHqbs= X-Gm-Gg: ASbGncuf10EFhMrpGYmaTj+uioRuMdRSkRnMeDfm1+1GMxW7JCNHpfSi9/Vsjcfj870 KI2MMfcGFQagIw+q5myQvJFqu0hcT5OZid0wtnGbfyiWtqqqbKrvLQOtpB6oVvTzil85fhlufD9 ZNmG90yPlAQKXtO+avKK3od9CzHL9pKZtjQ4ueIqdkTNPzxrXoSkK8Mmb8QYTPhBodQz2hjEi2a WYK9uhmItfF+ByF+mkjy59IYdp3Cshs8yPMWwLhX2biGak76ab6vlurPCnU1nx/Fa19RQw93EnJ 1OjYctRHTDPtZPXLWXJT2Jswo4VAAnWXXF92F2qgCdemq21oBxtt X-Google-Smtp-Source: AGHT+IGpAfSUXA7trx1t771l9SXHKyNia4RKYigVXyNmSAcnSlXmn7j7tBria63/jx4Dvw4h7RaVyQ== X-Received: by 2002:a17:903:1443:b0:216:6a4a:9a47 with SMTP id d9443c01a7336-218c9262eeamr87431315ad.21.1734532990760; Wed, 18 Dec 2024 06:43:10 -0800 (PST) Received: from ?IPV6:2409:8a28:f4f:a9a4:df6:1df4:e2c5:1e97? ([2409:8a28:f4f:a9a4:df6:1df4:e2c5:1e97]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1e5c4dfsm76686595ad.177.2024.12.18.06.43.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 18 Dec 2024 06:43:10 -0800 (PST) Message-ID: Date: Wed, 18 Dec 2024 22:42:58 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 02/12] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() To: Klara Modin Cc: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: From: Qi Zheng Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2024/12/18 22:26, Klara Modin wrote: > Hi, > > On 2024-12-14 10:02, Qi Zheng wrote: >> Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a >> wrapper for get_zeroed_page() and p4d_free() as a wrapper for >> free_page(). >> >> For these architectures, provide a generic implementation in >> asm-generic/pgalloc.h and convert them to use it. And like other levels >> of page tables, add statistics for P4D level page table. >> >> For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its >> own logic, so skip it. >> >> Signed-off-by: Qi Zheng >> --- >>   arch/arm64/include/asm/pgalloc.h | 15 ++++----- >>   arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- >>   arch/x86/include/asm/pgalloc.h   | 16 ++++------ >>   arch/x86/mm/pgtable.c            |  3 ++ >>   include/asm-generic/pgalloc.h    | 55 ++++++++++++++++++++++++++++++++ >>   include/linux/mm.h               | 16 ++++++++++ >>   6 files changed, 98 insertions(+), 32 deletions(-) >> >> diff --git a/arch/arm64/include/asm/pgalloc.h >> b/arch/arm64/include/asm/pgalloc.h >> index e75422864d1bd..679c530549327 100644 >> --- a/arch/arm64/include/asm/pgalloc.h >> +++ b/arch/arm64/include/asm/pgalloc.h >> @@ -15,6 +15,8 @@ >>   #define __HAVE_ARCH_PGD_FREE >>   #define __HAVE_ARCH_PUD_FREE >> +#define __HAVE_ARCH_P4D_ALLOC_ONE >> +#define __HAVE_ARCH_P4D_FREE >>   #include >>   #define PGD_SIZE    (PTRS_PER_PGD * sizeof(pgd_t)) >> @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct >> *mm, pgd_t *pgdp, p4d_t *p4dp) >>   static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned >> long addr) >>   { >> -    gfp_t gfp = GFP_PGTABLE_USER; >> +    if (!pgtable_l5_enabled()) >> +        return NULL; >> -    if (mm == &init_mm) >> -        gfp = GFP_PGTABLE_KERNEL; >> -    return (p4d_t *)get_zeroed_page(gfp); >> +    return __p4d_alloc_one(mm, addr); >>   } >>   static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) >>   { >> -    if (!pgtable_l5_enabled()) >> -        return; >> -    BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); >> -    free_page((unsigned long)p4d); >> +    if (pgtable_l5_enabled()) >> +        __p4d_free(mm, p4d); >>   } >>   #define __p4d_free_tlb(tlb, p4d, addr)  p4d_free((tlb)->mm, p4d) >> diff --git a/arch/riscv/include/asm/pgalloc.h >> b/arch/riscv/include/asm/pgalloc.h >> index f52264304f772..bb6e1c5f1fb19 100644 >> --- a/arch/riscv/include/asm/pgalloc.h >> +++ b/arch/riscv/include/asm/pgalloc.h >> @@ -14,6 +14,8 @@ >>   #ifdef CONFIG_MMU >>   #define __HAVE_ARCH_PUD_ALLOC_ONE >>   #define __HAVE_ARCH_PUD_FREE >> +#define __HAVE_ARCH_P4D_ALLOC_ONE >> +#define __HAVE_ARCH_P4D_FREE >>   #include >>   static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, >> void *pt) >> @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct >> mmu_gather *tlb, pud_t *pud, >>   #define p4d_alloc_one p4d_alloc_one >>   static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned >> long addr) >>   { >> -    if (pgtable_l5_enabled) { >> -        gfp_t gfp = GFP_PGTABLE_USER; >> - >> -        if (mm == &init_mm) >> -            gfp = GFP_PGTABLE_KERNEL; >> -        return (p4d_t *)get_zeroed_page(gfp); >> -    } >> +    if (!pgtable_l5_enabled) >> +        return NULL; >> -    return NULL; >> -} >> - >> -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) >> -{ >> -    BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); >> -    free_page((unsigned long)p4d); >> +    return __p4d_alloc_one(mm, addr); >>   } >>   #define p4d_free p4d_free >> @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, >> p4d_t *p4d) >>   static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, >>                     unsigned long addr) >>   { >> -    if (pgtable_l5_enabled) >> +    if (pgtable_l5_enabled) { >> +        struct ptdesc *ptdesc = virt_to_ptdesc(p4d); >> + >> +        pagetable_p4d_dtor(ptdesc); >>           riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); >> +    } >>   } >>   #endif /* __PAGETABLE_PMD_FOLDED */ >> diff --git a/arch/x86/include/asm/pgalloc.h >> b/arch/x86/include/asm/pgalloc.h >> index dcd836b59bebd..d9bc6cae77c9e 100644 >> --- a/arch/x86/include/asm/pgalloc.h >> +++ b/arch/x86/include/asm/pgalloc.h >> @@ -8,6 +8,8 @@ >>   #define __HAVE_ARCH_PTE_ALLOC_ONE >>   #define __HAVE_ARCH_PGD_FREE >> +#define __HAVE_ARCH_P4D_ALLOC_ONE >> +#define __HAVE_ARCH_P4D_FREE >>   #include >>   static inline int  __paravirt_pgd_alloc(struct mm_struct *mm) { >> return 0; } >> @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct >> mm_struct *mm, pgd_t *pgd, p4d_t *p4 >>   static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned >> long addr) >>   { >> -    gfp_t gfp = GFP_KERNEL_ACCOUNT; >> +    if (!pgtable_l5_enabled()) >> +        return NULL; >> -    if (mm == &init_mm) >> -        gfp &= ~__GFP_ACCOUNT; >> -    return (p4d_t *)get_zeroed_page(gfp); >> +    return __p4d_alloc_one(mm, addr); >>   } >>   static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) >>   { >> -    if (!pgtable_l5_enabled()) >> -        return; >> - >> -    BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); >> -    free_page((unsigned long)p4d); >> +    if (pgtable_l5_enabled()) >> +        return __p4d_free(mm, p4d); >>   } >>   extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); >> diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c >> index 69a357b15974a..3d6e84da45b24 100644 >> --- a/arch/x86/mm/pgtable.c >> +++ b/arch/x86/mm/pgtable.c >> @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t >> *pud) >>   #if CONFIG_PGTABLE_LEVELS > 4 >>   void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) >>   { >> +    struct ptdesc *ptdesc = virt_to_ptdesc(p4d); >> + >> +    pagetable_p4d_dtor(ptdesc); >>       paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); >>       paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); >>   } >> diff --git a/include/asm-generic/pgalloc.h >> b/include/asm-generic/pgalloc.h >> index 7c48f5fbf8aa7..dbf61819b3581 100644 >> --- a/include/asm-generic/pgalloc.h >> +++ b/include/asm-generic/pgalloc.h >> @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, >> pud_t *pud) >>   #endif /* CONFIG_PGTABLE_LEVELS > 3 */ >> +#if CONFIG_PGTABLE_LEVELS > 4 >> + >> +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, >> unsigned long addr) >> +{ >> +    gfp_t gfp = GFP_PGTABLE_USER; >> +    struct ptdesc *ptdesc; >> + >> +    if (mm == &init_mm) >> +        gfp = GFP_PGTABLE_KERNEL; >> +    gfp &= ~__GFP_HIGHMEM; >> + >> +    ptdesc = pagetable_alloc_noprof(gfp, 0); >> +    if (!ptdesc) >> +        return NULL; >> + >> +    pagetable_p4d_ctor(ptdesc); >> +    return ptdesc_address(ptdesc); >> +} >> +#define __p4d_alloc_one(...) >> alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) >> + >> +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE >> +/** >> + * p4d_alloc_one - allocate memory for a P4D-level page table >> + * @mm: the mm_struct of the current context >> + * >> + * Allocate memory for a page table using %GFP_PGTABLE_USER for user >> context >> + * and %GFP_PGTABLE_KERNEL for kernel context. >> + * >> + * Return: pointer to the allocated memory or %NULL on error >> + */ >> +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, >> unsigned long addr) >> +{ >> +    return __p4d_alloc_one_noprof(mm, addr); >> +} >> +#define p4d_alloc_one(...) >> alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) >> +#endif >> + >> +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) >> +{ >> +    struct ptdesc *ptdesc = virt_to_ptdesc(p4d); >> + >> +    BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); >> +    pagetable_p4d_dtor(ptdesc); >> +    pagetable_free(ptdesc); >> +} >> + > > >> +#ifndef __HAVE_ARCH_P4D_FREE >> +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) > > Should this perhaps be p4d_t *p4d rather than pud_t *p4d? Yes, my bad. Will fix it. Thanks! > > Otherwise I get this build error: > > In file included from > /home/klara/git/linux/arch/riscv/include/asm/kfence.h:8, >                  from /home/klara/git/linux/mm/kfence/core.c:34: > /home/klara/git/linux/include/asm-generic/pgalloc.h: In function > ‘p4d_free’: > /home/klara/git/linux/include/asm-generic/pgalloc.h:252:24: error: > passing argument 2 of ‘__p4d_free’ from incompatible pointer type > [-Wincompatible-pointer-types] >   252 |         __p4d_free(mm, p4d); >       |                        ^~~ >       |                        | >       |                        pud_t * > /home/klara/git/linux/include/asm-generic/pgalloc.h:244:60: note: > expected ‘p4d_t *’ but argument is of type ‘pud_t *’ >   244 | static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) >       |                                                     ~~~~~~~^~~ > In file included from > /home/klara/git/linux/arch/riscv/include/asm/kfence.h:8, >                  from /home/klara/git/linux/mm/kfence/report.c:22: > /home/klara/git/linux/include/asm-generic/pgalloc.h: In function > ‘p4d_free’: > /home/klara/git/linux/include/asm-generic/pgalloc.h:252:24: error: > passing argument 2 of ‘__p4d_free’ from incompatible pointer type > [-Wincompatible-pointer-types] >   252 |         __p4d_free(mm, p4d); >       |                        ^~~ >       |                        | >       |                        pud_t * > /home/klara/git/linux/include/asm-generic/pgalloc.h:244:60: note: > expected ‘p4d_t *’ but argument is of type ‘pud_t *’ >   244 | static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) >       |                                                     ~~~~~~~^~~ > > Regards, > Klara Modin > >> +{ >> +    __p4d_free(mm, p4d); >> +} >> +#endif >> + >> +#endif >> + >>   #ifndef __HAVE_ARCH_PGD_FREE >>   static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) >>   { >> diff --git a/include/linux/mm.h b/include/linux/mm.h >> index 5e73e53c34e9e..807a12ed8ec96 100644 >> --- a/include/linux/mm.h >> +++ b/include/linux/mm.h >> @@ -3237,6 +3237,22 @@ static inline void pagetable_pud_dtor(struct >> ptdesc *ptdesc) >>       lruvec_stat_sub_folio(folio, NR_PAGETABLE); >>   } >> +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) >> +{ >> +    struct folio *folio = ptdesc_folio(ptdesc); >> + >> +    __folio_set_pgtable(folio); >> +    lruvec_stat_add_folio(folio, NR_PAGETABLE); >> +} >> + >> +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) >> +{ >> +    struct folio *folio = ptdesc_folio(ptdesc); >> + >> +    __folio_clear_pgtable(folio); >> +    lruvec_stat_sub_folio(folio, NR_PAGETABLE); >> +} >> + >>   extern void __init pagecache_init(void); >>   extern void free_initmem(void); >