From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6FD4288C3D for ; Tue, 19 May 2026 03:21:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779160871; cv=none; b=Oid1uT2FRLn/SXCHMZstHNLW2iRCeMJCyDL2pwsyA7X+VuilB8tjOF/5vkjcBjQZxtIc7IGR7kZo/Mey2DX7s/ECFje04fLsCnO5gduMRfqQHM5hgUB9zdzdUvmGbDoEb/fcq9QaQe94ROr/ASUnpvyRNU5gqy3U1Y979GRGvP8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779160871; c=relaxed/simple; bh=qcH8PMPAyJiTkepaghd+65iBhrd37pavq9xijlY+swk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UyR1OJ/e6k937Sj1hWLJ/0Sjr3QpO+dJs12ILY8hTRpt1qFaIxgy+1k7M4n8nLIWumFOoFgzD+v0rbAlGdALonrHI/Eiq8/iq6s3gFd3kqB8Ry2GZV98M8VLqqJCQ4rceKw7+XSRl0HFNz7SdQAqyziCainFXNog6uNQBgkYFME= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EH2dSuPd; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EH2dSuPd" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-368f25ff4c4so1433169a91.2 for ; Mon, 18 May 2026 20:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1779160869; x=1779765669; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4fHtLWbjgxAw/RuryI1Zn991wS9DJVk1T6sfJuVej8=; b=EH2dSuPdSOjB9dEI444/xp8zxzGSkdJx3huba+y6UjpG/jh1UIr5DgV0JeQMjy2cLY CDcmW5Dq1bxOM9BRI2Jq+iX1x00l/0WugxmvRtOkWVhDEGglCgCNAOdheQMLOpynlAlQ 5IneR+HpD64SDgPxhT0/NVHYUOkF1a2DiWQcw9m1bjluKrTFieYS6S0S8IrEQSloexza UGiIhGuLJBlqQUuhGdiu/e3s2dhTCGtRMl4U8/p7z5s3typ81LXitLrsRRqZ3yxeWODH Bbr1KLuAf0LmJdDlCwfuee4OWPo0IXo8MghNNXo0pAKsRmPH9/dEIptJdw2Ac7kltWmg MyoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779160869; x=1779765669; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=h4fHtLWbjgxAw/RuryI1Zn991wS9DJVk1T6sfJuVej8=; b=IELItO0RaJQd6TKZteBqi2TYgUGw1f9QrmclqXR6s2psmV9hFkZzAfQv6LRmXD7C6q 1e+5mvjdNK6C60kHdlDreEaCsrCK+gmqqjWYVCXGerAhbslCAW4c/Ug+3eujTbazLShl NQJLq1ISXPXSb0kM6FUX0GSFJCOqzg1NWo3w59pgpxzYk5Lc89yK2hKLn7K/DPmu03NS yy1cxhA9Ai5YCroLTYpzxL1s8Kxa3Hy7AqaFSHl0dMgmnkcaNu9G6kXcvuxlSiFEiqZQ hLF9FiRfTmDOd6LpW00mNWJZieAP9GWB6LTVkvRMAKKgrOpE2nq2NnzNus/cHCUDnmbn j9SA== X-Forwarded-Encrypted: i=1; AFNElJ+nFBo5Y7cfyo6Bi5yXwUvwCh8HnBeCytuHBs1f5FRD1Mym5noK1n4wDwvzjapoDidz6URq5Q/E2gszBUU=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9Sztg3Y4AytjRi8HFncwyHa58Dx1P4ZvYtjC2uUOJRov+2V5r fXyU6QI5Y3UOHEZIXeyC7nxU7g7cOtvjMFpwQ9UE07C8WT99RciJ0U1xSZ05KqFPovXrIJMTnxB ir9Rob6sNRw== X-Gm-Gg: Acq92OHWLaQK9eg0FqXfEvpFYK2oMPbXmU2/nSqW6rmk6gOepxENUsePpvw5YkQB7R1 XI5TbUZf4maELpNwMnMxNICV/dyavKYsqyNGI1CMTtWsdtVBQ1v30eHVawV+sqVA3jwKRlUKiq/ aTptJT9mapZ6+J6jM83GBby4n/CK/zXTb8JYXVa3GyUDdOSFKLSo2OAUW6XFPUe8oJl4N0cKyWG pQ1HRNyD93I2C0i3v7z36UXhSTayX4Qagc0qHU9wjWaU99Kl9zXurcIbsr3/kHfuHakck4F9IJj shk47kWg6pfSJ6QOmdFcg58DK+qgJtjfs5Im1CQ7Saw2sG0++BEVITTxKpZuXwqlPfGbmpEKQEY Cr5k5Xy/2sjtYz4ZLRy5xbqf7Wi7+dZxaZEfgkv+mNpjHrjD/50NmpBYyBC9mBPBtLiC2Gv7H6f WgYts1CH6tLDq3dbb6AAi60pGLT/9delz98GP6v8qKMIb5XsNO X-Received: by 2002:a17:903:2a8f:b0:2b2:57f3:8d07 with SMTP id d9443c01a7336-2bd7e8ae696mr183879905ad.7.1779160868998; Mon, 18 May 2026 20:21:08 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([2001:218:2001:0:1000:0:1:7]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2bd5c16ac4asm156453495ad.33.2026.05.18.20.21.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 18 May 2026 20:21:08 -0700 (PDT) From: Yunhui Cui To: pjw@kernel.org, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, akpm@linux-foundation.org, pasha.tatashin@soleen.com, andrew+kernel@donnellan.id.au, rmclure@linux.ibm.com, debug@rivosinc.com, baolin.wang@linux.alibaba.com, zhangchunyan@iscas.ac.cn, cuiyunhui@bytedance.com, apopple@nvidia.com, namcao@linutronix.de, wangruikang@iscas.ac.cn, apatel@ventanamicro.com, liu.xuemei1@zte.com.cn, ajones@ventanamicro.com, cleger@rivosinc.com, charlie@rivosinc.com, hui.wang@canonical.com, guodong@riscstar.com, pincheng.plct@isrc.iscas.ac.cn, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Qingwei Hu Subject: [PATCH 2/2] riscv: merge hardware A/D updates in PTE accessors Date: Tue, 19 May 2026 11:19:27 +0800 Message-Id: <20260519031927.70683-2-cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260519031927.70683-1-cuiyunhui@bytedance.com> References: <20260519031927.70683-1-cuiyunhui@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Use cmpxchg-based merges for live RISC-V PTE permission updates so software changes do not lose concurrently hardware-updated accessed and dirty state. Cover ptep_set_access_flags(), ptep_test_and_clear_young(), and ptep_set_wrprotect(), and extend the same wrprotect handling to the PUD leaf helper used by huge mappings. Keep the existing Svvptc flush behaviour, but only flush when the merged PTE value actually changed. Signed-off-by: Yunhui Cui Reviewed-by: Qingwei Hu --- arch/riscv/include/asm/pgtable.h | 19 +++++++-- arch/riscv/mm/pgtable.c | 68 ++++++++++++++++++++++++++------ 2 files changed, 73 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 20663a466cf6c..984c37ca8aef7 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -668,15 +668,21 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pte_t read_pte = READ_ONCE(*ptep); + pte_t old_pte; + pte_t pte; /* * ptep_set_wrprotect can be called for shadow stack ranges too. * shadow stack memory is XWR = 010 and thus clearing _PAGE_WRITE will lead to * encoding 000b which is wrong encoding with V = 1. This should lead to page fault * but we dont want this wrong configuration to be set in page tables. */ - atomic_long_set((atomic_long_t *)ptep, - ((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ)); + pte = READ_ONCE(*ptep); + do { + old_pte = pte; + pte = pte_wrprotect(pte); + pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), + pte_val(pte)); + } while (pte_val(pte) != pte_val(old_pte)); } #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH @@ -1030,6 +1036,13 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect(struct mm_struct *mm, + unsigned long address, pud_t *pudp) +{ + ptep_set_wrprotect(mm, address, (pte_t *)pudp); +} + #define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 9c4427d0b1874..b77e82362442e 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,23 +5,55 @@ #include #include +#define RISCV_PTE_ACCESS_FLAG_MASK (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC | \ + _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SOFT_DIRTY) + +static inline unsigned long riscv_pte_access_flags(unsigned long cur, + unsigned long entry) +{ + unsigned long pteval; + unsigned long hw_flags; + + hw_flags = _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_SOFT_DIRTY; + pteval = cur & ~RISCV_PTE_ACCESS_FLAG_MASK; + pteval |= entry & (RISCV_PTE_ACCESS_FLAG_MASK & ~hw_flags); + pteval |= (cur | entry) & hw_flags; + + return pteval; +} + int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { + unsigned long old_pteval; + unsigned long new_pteval; + unsigned long prev_pteval; + bool changed; + + old_pteval = pte_val(ptep_get(ptep)); + do { + new_pteval = riscv_pte_access_flags(old_pteval, pte_val(entry)); + if (new_pteval == old_pteval) + break; + + prev_pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, + new_pteval); + if (prev_pteval == old_pteval) + break; + + old_pteval = prev_pteval; + } while (1); + + changed = old_pteval != new_pteval; if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) { - if (!pte_same(ptep_get(ptep), entry)) { - __set_pte_at(vma->vm_mm, ptep, entry); - /* Here only not svadu is impacted */ + if (changed) flush_tlb_page(vma, address); - return true; - } - return false; + return changed; } - if (!pte_same(ptep_get(ptep), entry)) - __set_pte_at(vma->vm_mm, ptep, entry); /* * update_mmu_cache will unconditionally execute, handling both * the case that the PTE changed and the spurious fault case. @@ -32,9 +64,23 @@ int ptep_set_access_flags(struct vm_area_struct *vma, bool ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) - return false; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); + unsigned long old_pteval; + unsigned long new_pteval; + unsigned long prev_pteval; + + old_pteval = pte_val(ptep_get(ptep)); + do { + if (!(old_pteval & _PAGE_ACCESSED)) + return false; + + new_pteval = pte_val(pte_mkold(__pte(old_pteval))); + prev_pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, + new_pteval); + if (prev_pteval == old_pteval) + return true; + + old_pteval = prev_pteval; + } while (1); } EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); -- 2.39.5