* [PATCH v5] arm64: errata: work around NVIDIA Olympus device store/load ordering
@ 2026-07-15 20:48 Shanker Donthineni
2026-07-16 9:15 ` Vladimir Murzin
0 siblings, 1 reply; 2+ messages in thread
From: Shanker Donthineni @ 2026-07-15 20:48 UTC (permalink / raw)
To: Catalin Marinas, Will Deacon, Vladimir Murzin
Cc: Jason Gunthorpe, linux-arm-kernel, Mark Rutland, linux-kernel,
linux-doc, Shanker Donthineni, Vikram Sethi, Jason Sequeira
On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
observed by a peripheral before an older, non-overlapping Device-nGnR*
store to the same peripheral. This breaks the program-order guarantee
that software expects for Device-nGnR* accesses and can leave a
peripheral in an incorrect state.
The erratum can occur only when all of the following apply:
- A PE executes a Device-nGnR* store followed by a younger
Device-nGnR* load.
- The store is not a store-release.
- The accesses target the same peripheral and do not overlap in bytes.
- There is at most one intervening Device-nGnR* store in program
order, and there are no intervening Device-nGnR* loads.
- There is no DSB or full DMB between the store and the load.
- Specific microarchitectural and timing conditions occur.
Insert a DMB OSH immediately before each raw MMIO load on affected CPUs.
As a full barrier, DMB OSH orders the older Device store before the
younger Device load and prevents the erroneous observation.
Add the barrier directly to the __raw_read*() helpers, independently of
the existing device-load-acquire alternative. On affected CPUs this adds
one DMB OSH per raw MMIO load, including each load used by
memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.
Co-developed-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Vikram Sethi <vsethi@nvidia.com>
Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
Link: https://lore.kernel.org/all/akPQ8F3OgER621UP@willie-the-truck/
---
Changes since v4:
- Reworked the workaround following Will Deacon's review: leave the raw
MMIO write helpers unchanged and insert a DMB OSH before raw MMIO loads.
- Use DMB OSH after hardware confirmation that it fixes T410-OLY-1027.
- Dropped the separate memcpy_fromio() optimization patch because the
benchmark showed no noticeable benefit over the per-load workaround.
- Updated the cpucap, Kconfig help text, and commit messages for the
load-side workaround.
Changes since v3:
- Split the workaround into two patches: the erratum fix (1/2) and the
arm64 memset_io()/memcpy_toio() block writers (2/2).
- Reworked the raw MMIO write helpers to use a direct base-register
str*/stlr* alternative sequence instead of a per-write static branch.
- Covered the write-combining __iowrite{32,64}_copy() path by patching
dgh() to dmb osh on affected CPUs, keeping the contiguous STR groups
and the ordering barrier outside the copy loop; the single-element
case now uses a plain str* as well.
- Added arm64 memset_io()/memcpy_toio() so the byte/word block writers
take one trailing dmb osh instead of a per-store store-release.
- Updated the commit messages to describe the offset-addressing
trade-off.
Changes since v2:
- Reworked the raw MMIO write helpers so unaffected CPUs keep the
existing offset-addressed STR sequence, while affected CPUs use the
base-register STLR path.
- Updated the commit message to match the code changes.
- Rebased on top of the arm64 for-next/errata branch:
https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata
Changes since v1:
- Updated the commit message based on feedback from Vladimir Murzin.
---
Documentation/arch/arm64/silicon-errata.rst | 2 ++
arch/arm64/Kconfig | 22 +++++++++++++++++++++
arch/arm64/include/asm/io.h | 16 ++++++++++++----
arch/arm64/kernel/cpu_errata.c | 8 ++++++++
arch/arm64/tools/cpucaps | 1 +
5 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index ad04d1cdc0f0..c4137f89acef 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -298,6 +298,8 @@ stable kernels.
+----------------+-----------------+-----------------+-----------------------------+
| NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM |
+----------------+-----------------+-----------------+-----------------------------+
+| NVIDIA | Olympus core | T410-OLY-1027 | NVIDIA_OLYMPUS_1027_ERRATUM |
++----------------+-----------------+-----------------+-----------------------------+
| NVIDIA | Olympus core | T410-OLY-1029 | ARM64_ERRATUM_4118414 |
+----------------+-----------------+-----------------+-----------------------------+
| NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 10c69474f276..73e3e68db161 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1363,6 +1363,28 @@ config NVIDIA_CARMEL_CNP_ERRATUM
If unsure, say Y.
+config NVIDIA_OLYMPUS_1027_ERRATUM
+ bool "NVIDIA Olympus: device store/load ordering erratum"
+ default y
+ help
+ This option adds an alternative code sequence to work around an
+ NVIDIA Olympus core erratum where a Device-nGnR* store can be
+ observed by a peripheral after a younger Device-nGnR* load to the
+ same peripheral. This breaks the program order that drivers rely
+ on for MMIO and can leave a device in an incorrect state.
+
+ The workaround inserts a DMB OSH immediately before raw MMIO loads.
+ The erratum cannot occur when a DMB that orders loads appears
+ between the store and load, preventing the younger load from being
+ observed before the older store.
+
+ The alternatives framework patches in DMB OSH only when an affected
+ CPU is detected. Other CPUs execute a NOP in its place. Disabling
+ this option leaves the original MMIO read instruction stream
+ unchanged.
+
+ If unsure, say Y.
+
config ROCKCHIP_ERRATUM_3568002
bool "Rockchip 3568002: GIC600 can not access physical addresses higher than 4GB"
default y
diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 8cbd1e96fd50..6d6d54c1b74c 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -54,7 +54,9 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr)
static __always_inline u8 __raw_readb(const volatile void __iomem *addr)
{
u8 val;
- asm volatile(ALTERNATIVE("ldrb %w0, [%1]",
+ asm volatile(ALTERNATIVE("nop", "dmb osh",
+ ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+ ALTERNATIVE("ldrb %w0, [%1]",
"ldarb %w0, [%1]",
ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
: "=r" (val) : "r" (addr));
@@ -66,7 +68,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
{
u16 val;
- asm volatile(ALTERNATIVE("ldrh %w0, [%1]",
+ asm volatile(ALTERNATIVE("nop", "dmb osh",
+ ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+ ALTERNATIVE("ldrh %w0, [%1]",
"ldarh %w0, [%1]",
ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
: "=r" (val) : "r" (addr));
@@ -77,7 +81,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
{
u32 val;
- asm volatile(ALTERNATIVE("ldr %w0, [%1]",
+ asm volatile(ALTERNATIVE("nop", "dmb osh",
+ ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+ ALTERNATIVE("ldr %w0, [%1]",
"ldar %w0, [%1]",
ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
: "=r" (val) : "r" (addr));
@@ -88,7 +94,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
{
u64 val;
- asm volatile(ALTERNATIVE("ldr %0, [%1]",
+ asm volatile(ALTERNATIVE("nop", "dmb osh",
+ ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+ ALTERNATIVE("ldr %0, [%1]",
"ldar %0, [%1]",
ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
: "=r" (val) : "r" (addr));
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 4b0d5d932897..740ba3d51e30 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -839,6 +839,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL),
},
#endif
+#ifdef CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM
+ {
+ /* NVIDIA Olympus core */
+ .desc = "NVIDIA Olympus device store/load ordering erratum",
+ .capability = ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
+ ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS),
+ },
+#endif
#ifdef CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE
{
/*
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 811c2479e82d..8d919b6699f0 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -121,6 +121,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM
WORKAROUND_CLEAN_CACHE
WORKAROUND_DEVICE_LOAD_ACQUIRE
WORKAROUND_NVIDIA_CARMEL_CNP
+WORKAROUND_NVIDIA_OLYMPUS_1027
WORKAROUND_PMUV3_IMPDEF_TRAPS
WORKAROUND_QCOM_FALKOR_E1003
WORKAROUND_QCOM_ORYON_CNTVOFF
--
2.54.0.windows.1
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [PATCH v5] arm64: errata: work around NVIDIA Olympus device store/load ordering
2026-07-15 20:48 [PATCH v5] arm64: errata: work around NVIDIA Olympus device store/load ordering Shanker Donthineni
@ 2026-07-16 9:15 ` Vladimir Murzin
0 siblings, 0 replies; 2+ messages in thread
From: Vladimir Murzin @ 2026-07-16 9:15 UTC (permalink / raw)
To: Shanker Donthineni, Catalin Marinas, Will Deacon
Cc: Jason Gunthorpe, linux-arm-kernel, Mark Rutland, linux-kernel,
linux-doc, Vikram Sethi, Jason Sequeira
On 7/15/26 21:48, Shanker Donthineni wrote:
> On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
> observed by a peripheral before an older, non-overlapping Device-nGnR*
> store to the same peripheral. This breaks the program-order guarantee
> that software expects for Device-nGnR* accesses and can leave a
> peripheral in an incorrect state.
>
> The erratum can occur only when all of the following apply:
>
> - A PE executes a Device-nGnR* store followed by a younger
> Device-nGnR* load.
> - The store is not a store-release.
> - The accesses target the same peripheral and do not overlap in bytes.
> - There is at most one intervening Device-nGnR* store in program
> order, and there are no intervening Device-nGnR* loads.
> - There is no DSB or full DMB between the store and the load.
> - Specific microarchitectural and timing conditions occur.
>
> Insert a DMB OSH immediately before each raw MMIO load on affected CPUs.
> As a full barrier, DMB OSH orders the older Device store before the
> younger Device load and prevents the erroneous observation.
>
> Add the barrier directly to the __raw_read*() helpers, independently of
> the existing device-load-acquire alternative. On affected CPUs this adds
> one DMB OSH per raw MMIO load, including each load used by
> memcpy_fromio(). On unaffected CPUs the alternative remains a NOP.
>
> Co-developed-by: Vikram Sethi <vsethi@nvidia.com>
> Signed-off-by: Vikram Sethi <vsethi@nvidia.com>
> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
> Link: https://lore.kernel.org/all/akPQ8F3OgER621UP@willie-the-truck/
> ---
> Changes since v4:
> - Reworked the workaround following Will Deacon's review: leave the raw
> MMIO write helpers unchanged and insert a DMB OSH before raw MMIO loads.
> - Use DMB OSH after hardware confirmation that it fixes T410-OLY-1027.
> - Dropped the separate memcpy_fromio() optimization patch because the
> benchmark showed no noticeable benefit over the per-load workaround.
> - Updated the cpucap, Kconfig help text, and commit messages for the
> load-side workaround.
>
> Changes since v3:
> - Split the workaround into two patches: the erratum fix (1/2) and the
> arm64 memset_io()/memcpy_toio() block writers (2/2).
> - Reworked the raw MMIO write helpers to use a direct base-register
> str*/stlr* alternative sequence instead of a per-write static branch.
> - Covered the write-combining __iowrite{32,64}_copy() path by patching
> dgh() to dmb osh on affected CPUs, keeping the contiguous STR groups
> and the ordering barrier outside the copy loop; the single-element
> case now uses a plain str* as well.
> - Added arm64 memset_io()/memcpy_toio() so the byte/word block writers
> take one trailing dmb osh instead of a per-store store-release.
> - Updated the commit messages to describe the offset-addressing
> trade-off.
>
> Changes since v2:
> - Reworked the raw MMIO write helpers so unaffected CPUs keep the
> existing offset-addressed STR sequence, while affected CPUs use the
> base-register STLR path.
> - Updated the commit message to match the code changes.
> - Rebased on top of the arm64 for-next/errata branch:
> https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/log/?h=for-next/errata
>
> Changes since v1:
> - Updated the commit message based on feedback from Vladimir Murzin.
>
> ---
> Documentation/arch/arm64/silicon-errata.rst | 2 ++
> arch/arm64/Kconfig | 22 +++++++++++++++++++++
> arch/arm64/include/asm/io.h | 16 ++++++++++++----
> arch/arm64/kernel/cpu_errata.c | 8 ++++++++
> arch/arm64/tools/cpucaps | 1 +
> 5 files changed, 45 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
> index ad04d1cdc0f0..c4137f89acef 100644
> --- a/Documentation/arch/arm64/silicon-errata.rst
> +++ b/Documentation/arch/arm64/silicon-errata.rst
> @@ -298,6 +298,8 @@ stable kernels.
> +----------------+-----------------+-----------------+-----------------------------+
> | NVIDIA | Carmel Core | N/A | NVIDIA_CARMEL_CNP_ERRATUM |
> +----------------+-----------------+-----------------+-----------------------------+
> +| NVIDIA | Olympus core | T410-OLY-1027 | NVIDIA_OLYMPUS_1027_ERRATUM |
> ++----------------+-----------------+-----------------+-----------------------------+
> | NVIDIA | Olympus core | T410-OLY-1029 | ARM64_ERRATUM_4118414 |
> +----------------+-----------------+-----------------+-----------------------------+
> | NVIDIA | T241 GICv3/4.x | T241-FABRIC-4 | N/A |
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 10c69474f276..73e3e68db161 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -1363,6 +1363,28 @@ config NVIDIA_CARMEL_CNP_ERRATUM
>
> If unsure, say Y.
>
> +config NVIDIA_OLYMPUS_1027_ERRATUM
> + bool "NVIDIA Olympus: device store/load ordering erratum"
> + default y
> + help
> + This option adds an alternative code sequence to work around an
> + NVIDIA Olympus core erratum where a Device-nGnR* store can be
> + observed by a peripheral after a younger Device-nGnR* load to the
> + same peripheral. This breaks the program order that drivers rely
> + on for MMIO and can leave a device in an incorrect state.
> +
> + The workaround inserts a DMB OSH immediately before raw MMIO loads.
> + The erratum cannot occur when a DMB that orders loads appears
> + between the store and load, preventing the younger load from being
> + observed before the older store.
> +
> + The alternatives framework patches in DMB OSH only when an affected
> + CPU is detected. Other CPUs execute a NOP in its place. Disabling
> + this option leaves the original MMIO read instruction stream
> + unchanged.
> +
> + If unsure, say Y.
> +
> config ROCKCHIP_ERRATUM_3568002
> bool "Rockchip 3568002: GIC600 can not access physical addresses higher than 4GB"
> default y
> diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
> index 8cbd1e96fd50..6d6d54c1b74c 100644
> --- a/arch/arm64/include/asm/io.h
> +++ b/arch/arm64/include/asm/io.h
> @@ -54,7 +54,9 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr)
> static __always_inline u8 __raw_readb(const volatile void __iomem *addr)
> {
> u8 val;
> - asm volatile(ALTERNATIVE("ldrb %w0, [%1]",
> + asm volatile(ALTERNATIVE("nop", "dmb osh",
> + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
> + ALTERNATIVE("ldrb %w0, [%1]",
> "ldarb %w0, [%1]",
> ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
> : "=r" (val) : "r" (addr));
> @@ -66,7 +68,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
> {
> u16 val;
>
> - asm volatile(ALTERNATIVE("ldrh %w0, [%1]",
> + asm volatile(ALTERNATIVE("nop", "dmb osh",
> + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
> + ALTERNATIVE("ldrh %w0, [%1]",
> "ldarh %w0, [%1]",
> ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
> : "=r" (val) : "r" (addr));
> @@ -77,7 +81,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
> static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
> {
> u32 val;
> - asm volatile(ALTERNATIVE("ldr %w0, [%1]",
> + asm volatile(ALTERNATIVE("nop", "dmb osh",
> + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
> + ALTERNATIVE("ldr %w0, [%1]",
> "ldar %w0, [%1]",
> ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
> : "=r" (val) : "r" (addr));
> @@ -88,7 +94,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
> static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
> {
> u64 val;
> - asm volatile(ALTERNATIVE("ldr %0, [%1]",
> + asm volatile(ALTERNATIVE("nop", "dmb osh",
> + ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
> + ALTERNATIVE("ldr %0, [%1]",
> "ldar %0, [%1]",
> ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
> : "=r" (val) : "r" (addr));
> diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
> index 4b0d5d932897..740ba3d51e30 100644
> --- a/arch/arm64/kernel/cpu_errata.c
> +++ b/arch/arm64/kernel/cpu_errata.c
> @@ -839,6 +839,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
> ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL),
> },
> #endif
> +#ifdef CONFIG_NVIDIA_OLYMPUS_1027_ERRATUM
> + {
> + /* NVIDIA Olympus core */
> + .desc = "NVIDIA Olympus device store/load ordering erratum",
> + .capability = ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
> + ERRATA_MIDR_ALL_VERSIONS(MIDR_NVIDIA_OLYMPUS),
> + },
> +#endif
> #ifdef CONFIG_ARM64_WORKAROUND_TRBE_OVERWRITE_FILL_MODE
> {
> /*
> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index 811c2479e82d..8d919b6699f0 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -121,6 +121,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM
> WORKAROUND_CLEAN_CACHE
> WORKAROUND_DEVICE_LOAD_ACQUIRE
> WORKAROUND_NVIDIA_CARMEL_CNP
Nitpick: this ^ line has changed recently, so patch might not apply
> +WORKAROUND_NVIDIA_OLYMPUS_1027
> WORKAROUND_PMUV3_IMPDEF_TRAPS
> WORKAROUND_QCOM_FALKOR_E1003
> WORKAROUND_QCOM_ORYON_CNTVOFF
> -- 2.54.0.windows.1
>
FWIW,
Reviewed-by: Vladimir Murzin <vladimir.murzin@arm.com>
Thanks
Vladimir
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-07-16 9:15 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-07-15 20:48 [PATCH v5] arm64: errata: work around NVIDIA Olympus device store/load ordering Shanker Donthineni
2026-07-16 9:15 ` Vladimir Murzin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox