[PATCH -next 0/2] riscv/mm: Enable THP migration

[PATCH -next 0/2] riscv/mm: Enable THP migration

[Nanyong Sun]

This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
like the behavior of pmd_present() and pmd_trans_huge() when in
THP splitting or migration.

This feature can reduce the time of THP migration by not splits THP
before migration and can guarantee the pages after migration are still
contiguous.[1]

I have tested the below test case on qemu based on riscv after
enabling this feature, the throughput of THP migration gains 13x
performance improvement:
https://github.com/x-y-z/thp-migration-bench

I also have tested and passed the test cases under
tools/testing/selftests/vm.

[1]: https://lwn.net/Articles/723764/

Nanyong Sun (2):
  riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
  riscv/mm: Enable THP migration

 arch/riscv/Kconfig                    |  1 +
 arch/riscv/include/asm/pgtable-bits.h |  2 +-
 arch/riscv/include/asm/pgtable.h      | 16 +++++++++++-----
 3 files changed, 13 insertions(+), 6 deletions(-)

-- 
2.25.1

[PATCH -next 1/2] riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics

[Nanyong Sun]

This is a preparation for enabling THP migration.
As the commit b65399f6111b("arm64/mm: Change THP helpers
to comply with generic MM semantics") mentioned, pmd_present()
and pmd_trans_huge() are expected to behave in the following
manner:
-------------------------------------------------------------------------
|	PMD states	|	pmd_present	|	pmd_trans_huge	|
-------------------------------------------------------------------------
|	Mapped		|	Yes		|	Yes		|
-------------------------------------------------------------------------
|	Splitting	|	Yes		|	Yes		|
-------------------------------------------------------------------------
|	Migration/Swap	|	No		|	No		|
-------------------------------------------------------------------------

At present the PROT_NONE bit reuses the READ bit could not comply with
above semantics with two problems:
1. When splitting a PMD THP, PMD is first invalidated with
pmdp_invalidate()->pmd_mkinvalid(), which clears the PRESENT bit
and PROT_NONE bit/READ bit, if the PMD is read-only, then the PAGE_LEAF
property is also cleared, which results in pmd_present() return false.
2. When migrating, the swap entry only clear the PRESENT bit
and PROT_NONE bit/READ bit, the W/X bit may be set, so _PAGE_LEAF may be
true which results in pmd_present() return true.

Solution:
Adjust PROT_NONE bit from READ to GLOBAL bit can satisfy the above rules:
1. GLOBAL bit has no other meanings, not like the R/W/X bit, which is
also relative with _PAGE_LEAF property.
2. GLOBAL bit is at bit 5, making swap entry start from bit 6, bit 0-5
are zero, which means the PRESENT, PROT_NONE, and PAGE_LEAF are
all false, then the pmd_present() and pmd_trans_huge() return false when
in migration/swap.

Signed-off-by: Nanyong Sun <sunnanyong@huawei.com>
---
 arch/riscv/include/asm/pgtable-bits.h |  2 +-
 arch/riscv/include/asm/pgtable.h      | 11 ++++++-----
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h
index 2ee413912926..a6b0c89824c2 100644
--- a/arch/riscv/include/asm/pgtable-bits.h
+++ b/arch/riscv/include/asm/pgtable-bits.h
@@ -31,7 +31,7 @@
  * _PAGE_PROT_NONE is set on not-present pages (and ignored by the hardware) to
  * distinguish them from swapped out pages
  */
-#define _PAGE_PROT_NONE _PAGE_READ
+#define _PAGE_PROT_NONE _PAGE_GLOBAL
 
 #define _PAGE_PFN_SHIFT 10
 
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index bf204e7c1f74..8bb2010f9a25 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -119,7 +119,7 @@
 /* Page protection bits */
 #define _PAGE_BASE	(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_USER)
 
-#define PAGE_NONE		__pgprot(_PAGE_PROT_NONE)
+#define PAGE_NONE		__pgprot(_PAGE_PROT_NONE | _PAGE_READ)
 #define PAGE_READ		__pgprot(_PAGE_BASE | _PAGE_READ)
 #define PAGE_WRITE		__pgprot(_PAGE_BASE | _PAGE_READ | _PAGE_WRITE)
 #define PAGE_EXEC		__pgprot(_PAGE_BASE | _PAGE_EXEC)
@@ -628,11 +628,12 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
  *
  * Format of swap PTE:
  *	bit            0:	_PAGE_PRESENT (zero)
- *	bit            1:	_PAGE_PROT_NONE (zero)
- *	bits      2 to 6:	swap type
- *	bits 7 to XLEN-1:	swap offset
+ *	bit       1 to 3:       _PAGE_LEAF (zero)
+ *	bit            5:	_PAGE_PROT_NONE (zero)
+ *	bits      6 to 10:	swap type
+ *	bits 10 to XLEN-1:	swap offset
  */
-#define __SWP_TYPE_SHIFT	2
+#define __SWP_TYPE_SHIFT	6
 #define __SWP_TYPE_BITS		5
 #define __SWP_TYPE_MASK		((1UL << __SWP_TYPE_BITS) - 1)
 #define __SWP_OFFSET_SHIFT	(__SWP_TYPE_BITS + __SWP_TYPE_SHIFT)
-- 
2.25.1

[PATCH -next 2/2] riscv/mm: Enable THP migration

[Nanyong Sun]

Add two THP helpers required to create PMD migration swap entries,
and enable THP migration via ARCH_ENABLE_THP_MIGRATION. This can
reduce time of THP migration without splitting and guarantee the
migrated pages are still contiguous.

Signed-off-by: Nanyong Sun <sunnanyong@huawei.com>
---
 arch/riscv/Kconfig               | 1 +
 arch/riscv/include/asm/pgtable.h | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 821252b65f89..7cad1a8e71bf 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -75,6 +75,7 @@ config RISCV
 	select HAVE_ARCH_SECCOMP_FILTER
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE if 64BIT && MMU
+	select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
 	select HAVE_ARCH_THREAD_STRUCT_WHITELIST
 	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
 	select HAVE_ASM_MODVERSIONS
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 8bb2010f9a25..b9bf51e31adc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -649,6 +649,11 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 #define __pte_to_swp_entry(pte)	((swp_entry_t) { pte_val(pte) })
 #define __swp_entry_to_pte(x)	((pte_t) { (x).val })
 
+#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
+#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
+#define __swp_entry_to_pmd(swp) __pmd((swp).val)
+#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
+
 /*
  * In the RV64 Linux scheme, we give the user half of the virtual-address space
  * and give the kernel the other (upper) half.
-- 
2.25.1

Re: [PATCH -next 0/2] riscv/mm: Enable THP migration

[Nanyong Sun]

Hi Palmer,

This patchset has been sent for one month ago, I was wondering if you could

have a review and give a feedback please, thanks.

On 2021/11/23 22:06, Nanyong Sun wrote:
> This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
> But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
> like the behavior of pmd_present() and pmd_trans_huge() when in
> THP splitting or migration.
>
> This feature can reduce the time of THP migration by not splits THP
> before migration and can guarantee the pages after migration are still
> contiguous.[1]
>
> I have tested the below test case on qemu based on riscv after
> enabling this feature, the throughput of THP migration gains 13x
> performance improvement:
> https://github.com/x-y-z/thp-migration-bench
>
> I also have tested and passed the test cases under
> tools/testing/selftests/vm.
>
> [1]: https://lwn.net/Articles/723764/
>
> Nanyong Sun (2):
>    riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
>    riscv/mm: Enable THP migration
>
>   arch/riscv/Kconfig                    |  1 +
>   arch/riscv/include/asm/pgtable-bits.h |  2 +-
>   arch/riscv/include/asm/pgtable.h      | 16 +++++++++++-----
>   3 files changed, 13 insertions(+), 6 deletions(-)
>

Re: [PATCH -next 0/2] riscv/mm: Enable THP migration

[Palmer Dabbelt]

On Wed, 29 Dec 2021 22:45:57 PST (-0800), sunnanyong@huawei.com wrote:
> Hi Palmer,
>
> This patchset has been sent for one month ago, I was wondering if you could
>
> have a review and give a feedback please, thanks.

Sorry for being slow, the queue got long over the holidays and I'm still 
trying to dig out.  I was kind of worried about losing some bits, but I 
don't think it's actually that big of a deal -- it's not like there's 
really any user ABI here, aside from being able to use less swap.  If 
someone wants to fix that I'm all ears, but I don't think it's worth 
waiting.

This is on for-next.

Thanks!

> On 2021/11/23 22:06, Nanyong Sun wrote:
>> This series enables THP migration on riscv via ARCH_ENABLE_THP_MIGRATION.
>> But first this adjusts PAGE_PROT_NONE to satisfy generic memory semantics
>> like the behavior of pmd_present() and pmd_trans_huge() when in
>> THP splitting or migration.
>>
>> This feature can reduce the time of THP migration by not splits THP
>> before migration and can guarantee the pages after migration are still
>> contiguous.[1]
>>
>> I have tested the below test case on qemu based on riscv after
>> enabling this feature, the throughput of THP migration gains 13x
>> performance improvement:
>> https://github.com/x-y-z/thp-migration-bench
>>
>> I also have tested and passed the test cases under
>> tools/testing/selftests/vm.
>>
>> [1]: https://lwn.net/Articles/723764/
>>
>> Nanyong Sun (2):
>>    riscv/mm: Adjust PAGE_PROT_NONE to comply with THP semantics
>>    riscv/mm: Enable THP migration
>>
>>   arch/riscv/Kconfig                    |  1 +
>>   arch/riscv/include/asm/pgtable-bits.h |  2 +-
>>   arch/riscv/include/asm/pgtable.h      | 16 +++++++++++-----
>>   3 files changed, 13 insertions(+), 6 deletions(-)
>>