From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE4F838837F for ; Thu, 18 Jun 2026 06:45:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781765117; cv=none; b=i6BZ2VjxJyeI9Kxj1U+J+gPy2Unp5u4mSDv7EqkOEeqz7GlFpZw3WgfmN4wKhnHsi5p3Fcysj7ZnqPHBZG8BhdfOkIMF/tMXvaqkNOFuVFOrMTgMKLdxUkMScekOOyWqiZ7gewU4LSXhtvGN5Wz0qo6/VKC2TcQkLiiX6sg0p0c= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781765117; c=relaxed/simple; bh=Nlerjuv8zYT6mH29QAsKXwKO76Dvq+OY7opxSWH52kI=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tWOjRkjzB78f8eyYW7kjtrH4jD1V4Onx4/2dyKOpckbwYsZ7D4YQHLcHhpicAUxQp5L9bMptJbTvWGgAxTJQxufILD+x/nGs0a6CXB6wwEHwwOS6bGQSYYNdHdFMQdvBsJfCrjoiFlI8RFiUZDRgVIWSU+svCetbLKVeY8w3kE8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NnUj05gc; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NnUj05gc" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2c0aa420401so3762355ad.3 for ; Wed, 17 Jun 2026 23:45:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1781765115; x=1782369915; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=3mY4IoCKia/XuJ/oNtOCFVHCKeBb5u4W4dO0pcbEY+A=; b=NnUj05gc2K1MPy71HkJlrj/V25xUvqUrAtf6G7Rb4VDwUQwBCcwFUb8jH79C23ale6 /UDlU3H8SCrvdJvK2Mfp0i6eFNebPJduXzaMzTErgrHLAPHpOY9zQ2xVonfkAoGw8alc r7axE/7fyGXlt8a1KkdZfjx24G7gxwU8CRld4uZsf4jD/d2XeOdgAo74FkKspHyQ46Bz Og9N+WLY09q+oWENPNZkuAkuFmXyqRvxtJkgGBKLoyNCtf4lao5p/NIiOo/s+k4VLH2F SDFo6D2jGenl41LdZalUxbTwb5WZh3C/1eO1jmw12bBfF4WQHCvV4mN5HAhxdi/9PXDT QPHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781765115; x=1782369915; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=3mY4IoCKia/XuJ/oNtOCFVHCKeBb5u4W4dO0pcbEY+A=; b=XOfGmOq0cCe3Uc9fb2IvsDAJCl5fNBpCZabPQb9FDMDr7uh+JuC2V5WZ2FgX9b3FZa tsYViaEEVHDcqm513nk6KlNhDcP9J0FFuAUYdKQgx2vrFmQmsn2a/SISQeYnr5oe591Q 99u5cSKIySWZfuO+9+BUSefHtDGzGtASZ6WU23bvi0W09tYwHPv3E0gRYXQB3v+uHIi1 ZT0U/sflKDA3rfNV1acVPzGysmmVZa3XDGaT+QyeBnvoZpbvkFz91lUGQu+Ok7WNvQPp adkj7pgN6zvwhAhqu6dKlQ+CbGSWUujR6EipkinCtmlT+pOybm97t7Kg3ijRL/HtTqUd gobw== X-Forwarded-Encrypted: i=1; AFNElJ+cZ6jStilYBYx+JYeTUHeQc1iJlIIiwwB5NCbpLkmpDviy0TuBVNoK9HkS8Ya1W1ijh/zUR8ACb7FdNrw=@vger.kernel.org X-Gm-Message-State: AOJu0Yyc6OcbVOkTO5Yvkfc9ucjv/FjRbG60kddRkQuug8KfwzNjryVm 5QajWcC6cKCYZU+FuyeOAK/8SO9jOTesamYiWdoGVP1qejMn3MtS/BtfRNh44/Na2xE= X-Gm-Gg: AfdE7clEFIQmERPZkP2OjKfyAVSLQ40WrmdyodZ2HFtpp9ZIRU6SY5mS7BaOChyvaPn bKIXTTIy/9vLeJdyOy2aN6pTGrFRwxJOH7NL0XExjuwg4fqavl2tHLyuzLW3NP7BnIVMK6Iy2aG r+s2ILB/0ZrFURr2SCff6Ca+YR4SLn6wtlumwqEba916XgllZ2KxBZgiqeV/LV5P7WrY0LT53dv ImPH981ciow2G/RMN7lUrIInYkiheTsdEAQcEBDnyJp1Bcafb7IsNjjReK+boFUfFzjCWLAlkqe S2+5yaGptbGi4As2nh3AO/gXeMr5mFAIKFTc31drudE6KpcdasfEay9VXBaMEhk3H90D7pkZ77d UNpIBFEZD7W7Bxc2YMA6oNbPO5bBQcDm/852LBecjvvVJ5JVyTZoxqO2NRAgAgawm2SqbjzGRqJ N5gwM+xfN33kXPxRxQ0LgEQxcjTTMvXM1QlWQPPCS6CcDF2Q== X-Received: by 2002:a17:903:1b6f:b0:2bf:77b2:8b2d with SMTP id d9443c01a7336-2c6bc25ca62mr70159885ad.30.1781765114873; Wed, 17 Jun 2026 23:45:14 -0700 (PDT) Received: from L6YN4KR4K9.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c4328a4c1fsm185275985ad.53.2026.06.17.23.45.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 17 Jun 2026 23:45:14 -0700 (PDT) From: Yunhui Cui To: akpm@linux-foundation.org, alex@ghiti.fr, andrew+kernel@donnellan.id.au, aou@eecs.berkeley.edu, apatel@ventanamicro.com, apopple@nvidia.com, atishp@rivosinc.com, baolin.wang@linux.alibaba.com, cleger@rivosinc.com, conor+dt@kernel.org, cuiyunhui@bytedance.com, debug@rivosinc.com, devicetree@vger.kernel.org, guodong@riscstar.com, hui.wang@canonical.com, krzk+dt@kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, liu.xuemei1@zte.com.cn, namcao@linutronix.de, nick.hu@sifive.com, palmer@dabbelt.com, pincheng.plct@isrc.iscas.ac.cn, pjw@kernel.org, qingwei.hu@bytedance.com, ritesh.list@gmail.com, rmclure@linux.ibm.com, robh@kernel.org, wangruikang@iscas.ac.cn, zhangchunyan@iscas.ac.cn, zong.li@sifive.com Subject: [PATCH v4 3/3] riscv: preserve A/D and soft-dirty state across PTE updates Date: Thu, 18 Jun 2026 14:44:06 +0800 Message-Id: <20260618064406.14508-4-cuiyunhui@bytedance.com> X-Mailer: git-send-email 2.39.2 (Apple Git-143) In-Reply-To: <20260618064406.14508-1-cuiyunhui@bytedance.com> References: <20260618064406.14508-1-cuiyunhui@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Use cmpxchg-based PTE updates so software permission changes do not lose concurrent A/D updates from hardware. Preserve soft-dirty state as well, since RISC-V marks PTEs dirty and soft-dirty together. Signed-off-by: Yunhui Cui Reviewed-by: Qingwei Hu --- arch/riscv/include/asm/pgtable.h | 27 +++++++++---- arch/riscv/mm/pgtable.c | 68 ++++++++++++++++++++++++++------ 2 files changed, 77 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 5d5756bda82e3..02286b48dc471 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -678,15 +678,21 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pte_t read_pte = READ_ONCE(*ptep); + pte_t old_pte; + pte_t pte; /* * ptep_set_wrprotect can be called for shadow stack ranges too. * shadow stack memory is XWR = 010 and thus clearing _PAGE_WRITE will lead to * encoding 000b which is wrong encoding with V = 1. This should lead to page fault * but we dont want this wrong configuration to be set in page tables. */ - atomic_long_set((atomic_long_t *)ptep, - ((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ)); + pte = READ_ONCE(*ptep); + do { + old_pte = pte; + pte = pte_wrprotect(pte); + pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep), pte_val(old_pte), + pte_val(pte)); + } while (pte_val(pte) != pte_val(old_pte)); } #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH @@ -742,14 +748,14 @@ static inline pgprot_t pgprot_writecombine(pgprot_t _prot) #define pgprot_dmacoherent pgprot_writecombine /* - * Both Svade and Svadu control the hardware behavior when the PTE A/D bits need to be set. By - * default the M-mode firmware enables the hardware updating scheme when only Svadu is present in - * DT. + * Both Svade and Svadu control the hardware behavior when the PTE A/D bits + * need to be set. The core MM code only cares whether hardware updating of + * the accessed/dirty state is currently active. */ #define arch_has_hw_pte_young arch_has_hw_pte_young static inline bool arch_has_hw_pte_young(void) { - return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVADU); + return riscv_has_hw_pte_ad_updating(); } /* @@ -1040,6 +1046,13 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect(struct mm_struct *mm, + unsigned long address, pud_t *pudp) +{ + ptep_set_wrprotect(mm, address, (pte_t *)pudp); +} + #define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 9c4427d0b1874..98eed19ea70de 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,23 +5,55 @@ #include #include +#define RISCV_PTE_ACCESS_FLAG_MASK (_PAGE_READ | _PAGE_WRITE | _PAGE_EXEC | \ + _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SOFT_DIRTY) + +static inline unsigned long riscv_pte_access_flags(unsigned long cur, + unsigned long entry) +{ + unsigned long pteval; + unsigned long preserved_flags; + + preserved_flags = _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_SOFT_DIRTY; + pteval = cur & ~RISCV_PTE_ACCESS_FLAG_MASK; + pteval |= entry & (RISCV_PTE_ACCESS_FLAG_MASK & ~preserved_flags); + pteval |= (cur | entry) & preserved_flags; + + return pteval; +} + int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { + unsigned long old_pteval; + unsigned long new_pteval; + unsigned long prev_pteval; + bool changed; + + old_pteval = pte_val(ptep_get(ptep)); + do { + new_pteval = riscv_pte_access_flags(old_pteval, pte_val(entry)); + if (new_pteval == old_pteval) + break; + + prev_pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, + new_pteval); + if (prev_pteval == old_pteval) + break; + + old_pteval = prev_pteval; + } while (1); + + changed = old_pteval != new_pteval; if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) { - if (!pte_same(ptep_get(ptep), entry)) { - __set_pte_at(vma->vm_mm, ptep, entry); - /* Here only not svadu is impacted */ + if (changed) flush_tlb_page(vma, address); - return true; - } - return false; + return changed; } - if (!pte_same(ptep_get(ptep), entry)) - __set_pte_at(vma->vm_mm, ptep, entry); /* * update_mmu_cache will unconditionally execute, handling both * the case that the PTE changed and the spurious fault case. @@ -32,9 +64,23 @@ int ptep_set_access_flags(struct vm_area_struct *vma, bool ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) - return false; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); + unsigned long old_pteval; + unsigned long new_pteval; + unsigned long prev_pteval; + + old_pteval = pte_val(ptep_get(ptep)); + do { + if (!(old_pteval & _PAGE_ACCESSED)) + return false; + + new_pteval = pte_val(pte_mkold(__pte(old_pteval))); + prev_pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, + new_pteval); + if (prev_pteval == old_pteval) + return true; + + old_pteval = prev_pteval; + } while (1); } EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); -- 2.39.5