mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
@ 2026-11-09  3:08 Miroslav Cimerman
  2026-11-12 12:21 ` Miroslav Cimerman
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Miroslav Cimerman @ 2026-11-09  3:08 UTC (permalink / raw)
  To: Song Liu, Yu Kuai, Andrew Morton
  Cc: Li Nan, Xiao Ni, Ard Biesheuvel, Eric Biggers, linux-raid, linux-kernel

xor_syndrome() updates P and Q in place for a read-modify-write. md
calls it twice on the same P/Q page per write, to remove the old data
and to add the new data. The x1 and x2 unrolled versions write P and Q
with normal stores since commit a582564b24be ("md/raid6 algorithms:
xor_syndrome() for SSE2"), which argued that non-temporal stores of
less than a cache line are slow, but kept non-temporal stores in the
x4 version, as it writes whole cache lines. The AVX2 and AVX-512
versions followed the same pattern.

The non-temporal stores evict P and Q from the cache and the sfence
waits until they are written to memory, just before the second call
reads the same P and Q again, now from memory.

Use normal aligned stores in the x4 unrolled versions too, like the x1
and x2 versions and the other architectures.

md RAID-6, 4 KiB random writes, md thread on one core, Xeon E5-2699 v3:

  8+2 on null_blk (dm-raid):     85.6k -> 98.1k IOPS (+15%)
  6+2 on 8 Intel Optane M10:     57.3k -> 62.1k IOPS (+8%)

Change in time per write in md's pattern (both calls and the 4 KiB copy
between them), kernel code built in userspace, one core, geometric
mean over 2-24 data disks, data in cache / DRAM:

  avx2x4 (Xeon E5-2699 v3):  -66% / -38%
  avx512x4 (Xeon Gold 6130): -78% / -48%
  sse2x4 (Core i5-2520M):    -37% / -19%

Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
Notes:
- The IOPS were measured on 2651f6deb332 (between 7.4-rc1 and -rc2) with
  and without this patch, with avx2x4 forced by a test-only change in
  raid6_choose_gen().
- The raid6 benchmark picks the implementation by gen_syndrome() speed
  alone. On Haswell-EP, AVX2 x2 and x4 differ only by noise, so either
  can be picked. With x2 picked, xor_syndrome() already uses normal
  stores, so this patch makes no difference in that case.
- The sfence at the end of the xor_syndrome() functions (x1, x2 versions
  since 2014, x4 with this patch) is no longer needed, as it no longer
  orders any non-temporal stores. Removing it made no measurable
  difference. I am happy to send a follow-up patch that removes it from
  all xor_syndrome() variants.
- Tested: raid6 KUnit on SSE2 and AVX2, AVX-512 was checked in userspace
  against a scalar reference on a Xeon Gold 6130.

 lib/raid/raid6/x86/avx2.c   | 16 ++++++++--------
 lib/raid/raid6/x86/avx512.c | 16 ++++++++--------
 lib/raid/raid6/x86/sse2.c   | 16 ++++++++--------
 3 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index 3cc2fe7ac42c..fbc1328574e1 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -435,18 +435,18 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
 			asm volatile("vpxor %ymm13,%ymm12,%ymm12");
 			asm volatile("vpxor %ymm15,%ymm14,%ymm14");
 		}
-		asm volatile("vmovntdq %%ymm2,%0" : "=m" (p[d]));
-		asm volatile("vmovntdq %%ymm3,%0" : "=m" (p[d+32]));
-		asm volatile("vmovntdq %%ymm10,%0" : "=m" (p[d+64]));
-		asm volatile("vmovntdq %%ymm11,%0" : "=m" (p[d+96]));
+		asm volatile("vmovdqa %%ymm2,%0" : "=m" (p[d]));
+		asm volatile("vmovdqa %%ymm3,%0" : "=m" (p[d+32]));
+		asm volatile("vmovdqa %%ymm10,%0" : "=m" (p[d+64]));
+		asm volatile("vmovdqa %%ymm11,%0" : "=m" (p[d+96]));
 		asm volatile("vpxor %0,%%ymm4,%%ymm4" : : "m" (q[d]));
 		asm volatile("vpxor %0,%%ymm6,%%ymm6" : : "m" (q[d+32]));
 		asm volatile("vpxor %0,%%ymm12,%%ymm12" : : "m" (q[d+64]));
 		asm volatile("vpxor %0,%%ymm14,%%ymm14" : : "m" (q[d+96]));
-		asm volatile("vmovntdq %%ymm4,%0" : "=m" (q[d]));
-		asm volatile("vmovntdq %%ymm6,%0" : "=m" (q[d+32]));
-		asm volatile("vmovntdq %%ymm12,%0" : "=m" (q[d+64]));
-		asm volatile("vmovntdq %%ymm14,%0" : "=m" (q[d+96]));
+		asm volatile("vmovdqa %%ymm4,%0" : "=m" (q[d]));
+		asm volatile("vmovdqa %%ymm6,%0" : "=m" (q[d+32]));
+		asm volatile("vmovdqa %%ymm12,%0" : "=m" (q[d+64]));
+		asm volatile("vmovdqa %%ymm14,%0" : "=m" (q[d+96]));
 	}
 	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 772bfc4af6df..520094efbb71 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -516,18 +516,18 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
 				     :
 				     : );
 		}
-		asm volatile("vmovntdq %%zmm2,%0\n\t"
-			     "vmovntdq %%zmm3,%1\n\t"
-			     "vmovntdq %%zmm10,%2\n\t"
-			     "vmovntdq %%zmm11,%3\n\t"
+		asm volatile("vmovdqa64 %%zmm2,%0\n\t"
+			     "vmovdqa64 %%zmm3,%1\n\t"
+			     "vmovdqa64 %%zmm10,%2\n\t"
+			     "vmovdqa64 %%zmm11,%3\n\t"
 			     "vpxorq %4,%%zmm4,%%zmm4\n\t"
 			     "vpxorq %5,%%zmm6,%%zmm6\n\t"
 			     "vpxorq %6,%%zmm12,%%zmm12\n\t"
 			     "vpxorq %7,%%zmm14,%%zmm14\n\t"
-			     "vmovntdq %%zmm4,%4\n\t"
-			     "vmovntdq %%zmm6,%5\n\t"
-			     "vmovntdq %%zmm12,%6\n\t"
-			     "vmovntdq %%zmm14,%7"
+			     "vmovdqa64 %%zmm4,%4\n\t"
+			     "vmovdqa64 %%zmm6,%5\n\t"
+			     "vmovdqa64 %%zmm12,%6\n\t"
+			     "vmovdqa64 %%zmm14,%7"
 			     :
 			     : "m" (p[d]),  "m" (p[d+64]), "m" (p[d+128]),
 			       "m" (p[d+192]), "m" (q[d]),  "m" (q[d+64]),
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index 43b09ce58270..da8374e506b6 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -432,18 +432,18 @@ static void raid6_sse24_xor_syndrome(int disks, int start, int stop,
 			asm volatile("pxor %xmm13,%xmm12");
 			asm volatile("pxor %xmm15,%xmm14");
 		}
-		asm volatile("movntdq %%xmm2,%0" : "=m" (p[d]));
-		asm volatile("movntdq %%xmm3,%0" : "=m" (p[d+16]));
-		asm volatile("movntdq %%xmm10,%0" : "=m" (p[d+32]));
-		asm volatile("movntdq %%xmm11,%0" : "=m" (p[d+48]));
+		asm volatile("movdqa %%xmm2,%0" : "=m" (p[d]));
+		asm volatile("movdqa %%xmm3,%0" : "=m" (p[d+16]));
+		asm volatile("movdqa %%xmm10,%0" : "=m" (p[d+32]));
+		asm volatile("movdqa %%xmm11,%0" : "=m" (p[d+48]));
 		asm volatile("pxor %0,%%xmm4" : : "m" (q[d]));
 		asm volatile("pxor %0,%%xmm6" : : "m" (q[d+16]));
 		asm volatile("pxor %0,%%xmm12" : : "m" (q[d+32]));
 		asm volatile("pxor %0,%%xmm14" : : "m" (q[d+48]));
-		asm volatile("movntdq %%xmm4,%0" : "=m" (q[d]));
-		asm volatile("movntdq %%xmm6,%0" : "=m" (q[d+16]));
-		asm volatile("movntdq %%xmm12,%0" : "=m" (q[d+32]));
-		asm volatile("movntdq %%xmm14,%0" : "=m" (q[d+48]));
+		asm volatile("movdqa %%xmm4,%0" : "=m" (q[d]));
+		asm volatile("movdqa %%xmm6,%0" : "=m" (q[d+16]));
+		asm volatile("movdqa %%xmm12,%0" : "=m" (q[d+32]));
+		asm volatile("movdqa %%xmm14,%0" : "=m" (q[d+48]));
 	}
 	asm volatile("sfence" : : : "memory");
 	kernel_fpu_end();

base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-09  3:08 [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions Miroslav Cimerman
@ 2026-11-12 12:21 ` Miroslav Cimerman
  2026-11-12 12:33   ` Anton Gavriliuk
  2026-11-16  6:11 ` Christoph Hellwig
  2026-11-16 21:04 ` Eric Biggers
  2 siblings, 1 reply; 7+ messages in thread
From: Miroslav Cimerman @ 2026-11-12 12:21 UTC (permalink / raw)
  To: song, yukuai, akpm
  Cc: ardb, ebiggers, linux-kernel, linux-raid, magiclinan, xiao, antosha20xx

Anton Gavriliuk tested the patch on Zen 5 (thread "xor_gen() md-raid6
performance improvement"):

HPE DL345 Gen11, AMD EPYC 9455, 6x PCIe 5.0 NVMe (CM7-V 3.2 TB),
Fedora 45 config, v7.4-rc2 with and without the patch, both kernels
select avx512x4. RAID-6 4+2, chunk 512K, no bitmap, rmw_level=2,
fio 4k randwrite, O_DIRECT, io_uring, 60 s, 3 runs per kernel:

  md defaults, 1 job x QD64:
    without  127k / 122k / 125k IOPS  (mean 124.7k)
    with     132k / 139k / 137k IOPS  (mean 136.0k)    +9.1%
    (a repeat run a day later: 127k without, 135k with)

  group_thread_cnt=8, stripe_cache_size=32768, 16 jobs x QD64:
    without  467k / 468k / 470k IOPS  (mean 468.3k)
    with     473k / 471k / 471k IOPS  (mean 471.7k)    +0.7%

With md's defaults one md thread does the stripe work for all writes
and is busy most of the time (about 80% in top), so a cheaper
xor_syndrome() shows up in IOPS. With 8 workers per node the limit is
lock contention in md instead (on my machine most of the time goes to
spinning on device_lock), so the saving hardly shows.

Miroslav

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-12 12:21 ` Miroslav Cimerman
@ 2026-11-12 12:33   ` Anton Gavriliuk
  0 siblings, 0 replies; 7+ messages in thread
From: Anton Gavriliuk @ 2026-11-12 12:33 UTC (permalink / raw)
  To: Miroslav Cimerman
  Cc: song, yukuai, akpm, ardb, ebiggers, linux-kernel, linux-raid,
	magiclinan, xiao

Tested-by: Anton Gavriliuk <antosha20xx@gmail.com>

чт, 12 нояб. 2026 г. в 14:23, Miroslav Cimerman <mc@raidlab.org>:
>
> Anton Gavriliuk tested the patch on Zen 5 (thread "xor_gen() md-raid6
> performance improvement"):
>
> HPE DL345 Gen11, AMD EPYC 9455, 6x PCIe 5.0 NVMe (CM7-V 3.2 TB),
> Fedora 45 config, v7.4-rc2 with and without the patch, both kernels
> select avx512x4. RAID-6 4+2, chunk 512K, no bitmap, rmw_level=2,
> fio 4k randwrite, O_DIRECT, io_uring, 60 s, 3 runs per kernel:
>
>   md defaults, 1 job x QD64:
>     without  127k / 122k / 125k IOPS  (mean 124.7k)
>     with     132k / 139k / 137k IOPS  (mean 136.0k)    +9.1%
>     (a repeat run a day later: 127k without, 135k with)
>
>   group_thread_cnt=8, stripe_cache_size=32768, 16 jobs x QD64:
>     without  467k / 468k / 470k IOPS  (mean 468.3k)
>     with     473k / 471k / 471k IOPS  (mean 471.7k)    +0.7%
>
> With md's defaults one md thread does the stripe work for all writes
> and is busy most of the time (about 80% in top), so a cheaper
> xor_syndrome() shows up in IOPS. With 8 workers per node the limit is
> lock contention in md instead (on my machine most of the time goes to
> spinning on device_lock), so the saving hardly shows.
>
> Miroslav

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-09  3:08 [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions Miroslav Cimerman
  2026-11-12 12:21 ` Miroslav Cimerman
@ 2026-11-16  6:11 ` Christoph Hellwig
  2026-11-16  6:39   ` AW: " Markus Stockhausen
  2026-11-16 21:04 ` Eric Biggers
  2 siblings, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2026-11-16  6:11 UTC (permalink / raw)
  To: Miroslav Cimerman
  Cc: Song Liu, Yu Kuai, Andrew Morton, Li Nan, Xiao Ni,
	Ard Biesheuvel, Eric Biggers, linux-raid, linux-kernel,
	Markus Stockhausen

On Mon, Nov 09, 2026 at 03:08:05AM +0000, Miroslav Cimerman wrote:
> xor_syndrome() updates P and Q in place for a read-modify-write. md
> calls it twice on the same P/Q page per write, to remove the old data
> and to add the new data. The x1 and x2 unrolled versions write P and Q
> with normal stores since commit a582564b24be ("md/raid6 algorithms:
> xor_syndrome() for SSE2"), which argued that non-temporal stores of
> less than a cache line are slow, but kept non-temporal stores in the
> x4 version, as it writes whole cache lines. The AVX2 and AVX-512
> versions followed the same pattern.

Adding Markus as the author of that commit.

> The non-temporal stores evict P and Q from the cache and the sfence
> waits until they are written to memory, just before the second call
> reads the same P and Q again, now from memory.
> 
> Use normal aligned stores in the x4 unrolled versions too, like the x1
> and x2 versions and the other architectures.
> 
> md RAID-6, 4 KiB random writes, md thread on one core, Xeon E5-2699 v3:
> 
>   8+2 on null_blk (dm-raid):     85.6k -> 98.1k IOPS (+15%)
>   6+2 on 8 Intel Optane M10:     57.3k -> 62.1k IOPS (+8%)

That's pretty nice numbers.

Note that for the new xraid code we'd actually use the XOR code in a
single pass, so non-temporal stores might make some sense.  But
I can't good heatedly argue that mainline should care for this right
now.

> - The sfence at the end of the xor_syndrome() functions (x1, x2 versions
>   since 2014, x4 with this patch) is no longer needed, as it no longer
>   orders any non-temporal stores. Removing it made no measurable
>   difference. I am happy to send a follow-up patch that removes it from
>   all xor_syndrome() variants.

This probably should go away.

Reviewed-by: Christoph Hellwig <hch@lst.de>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* AW: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-16  6:11 ` Christoph Hellwig
@ 2026-11-16  6:39   ` Markus Stockhausen
  0 siblings, 0 replies; 7+ messages in thread
From: Markus Stockhausen @ 2026-11-16  6:39 UTC (permalink / raw)
  To: Christoph Hellwig, Miroslav Cimerman
  Cc: Song Liu, Yu Kuai, Andrew Morton, Li Nan, Xiao Ni,
	Ard Biesheuvel, Eric Biggers, linux-raid, linux-kernel

> Von: Christoph Hellwig <hch@infradead.org> 
> Gesendet: Montag, 16. November 2026 07:11
> An: Miroslav Cimerman <mc@raidlab.org>
> <markus.stockhausen@collogia.de>
> Betreff: Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
> 
> On Mon, Nov 09, 2026 at 03:08:05AM +0000, Miroslav Cimerman wrote:
> > xor_syndrome() updates P and Q in place for a read-modify-write. md
> > calls it twice on the same P/Q page per write, to remove the old data
> > and to add the new data. The x1 and x2 unrolled versions write P and Q
> > with normal stores since commit a582564b24be ("md/raid6 algorithms:
> > xor_syndrome() for SSE2"), which argued that non-temporal stores of
> > less than a cache line are slow, but kept non-temporal stores in the
> > x4 version, as it writes whole cache lines. The AVX2 and AVX-512
> > versions followed the same pattern.
>
> Adding Markus as the author of that commit.

Thanks for the update after a long time. Happy to see that this is still in
place. I had only 2008 hardware at the time of the initial commit. So
all observations might be wrong with newer hardware.

Markus

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-09  3:08 [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions Miroslav Cimerman
  2026-11-12 12:21 ` Miroslav Cimerman
  2026-11-16  6:11 ` Christoph Hellwig
@ 2026-11-16 21:04 ` Eric Biggers
  2026-11-16 22:14   ` David Laight
  2 siblings, 1 reply; 7+ messages in thread
From: Eric Biggers @ 2026-11-16 21:04 UTC (permalink / raw)
  To: Miroslav Cimerman
  Cc: Song Liu, Yu Kuai, Andrew Morton, Li Nan, Xiao Ni,
	Ard Biesheuvel, linux-raid, linux-kernel

On Mon, Nov 09, 2026 at 03:08:05AM +0000, Miroslav Cimerman wrote:
> - The sfence at the end of the xor_syndrome() functions (x1, x2 versions
>   since 2014, x4 with this patch) is no longer needed, as it no longer
>   orders any non-temporal stores. Removing it made no measurable
>   difference. I am happy to send a follow-up patch that removes it from
>   all xor_syndrome() variants.

Removing the 'asm volatile("sfence" : : : "memory");' too sounds good,
but be careful: a lot of the inline asm statements in
lib/raid/raid6/x86/avx512.c incorrectly list the memory operands as
inputs only.  So, technically they rely on that "memory" clobber.  The
constraints should be fixed accordingly.

Also, ideally the 'prefetchnta' on q in the x4 xor_syndrome() functions
would become 'prefetcht0', to match the temporal stores.

If the scope of this patch is just replacing the store instructions
though, it looks fine.

Reviewed-by: Eric Biggers <ebiggers@kernel.org>

- Eric

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions
  2026-11-16 21:04 ` Eric Biggers
@ 2026-11-16 22:14   ` David Laight
  0 siblings, 0 replies; 7+ messages in thread
From: David Laight @ 2026-11-16 22:14 UTC (permalink / raw)
  To: Eric Biggers
  Cc: Miroslav Cimerman, Song Liu, Yu Kuai, Andrew Morton, Li Nan,
	Xiao Ni, Ard Biesheuvel, linux-raid, linux-kernel

On Mon, 16 Nov 2026 21:04:40 +0000
Eric Biggers <ebiggers@kernel.org> wrote:

> On Mon, Nov 09, 2026 at 03:08:05AM +0000, Miroslav Cimerman wrote:
> > - The sfence at the end of the xor_syndrome() functions (x1, x2 versions
> >   since 2014, x4 with this patch) is no longer needed, as it no longer
> >   orders any non-temporal stores. Removing it made no measurable
> >   difference. I am happy to send a follow-up patch that removes it from
> >   all xor_syndrome() variants.  
> 
> Removing the 'asm volatile("sfence" : : : "memory");' too sounds good,
> but be careful: a lot of the inline asm statements in
> lib/raid/raid6/x86/avx512.c incorrectly list the memory operands as
> inputs only.  So, technically they rely on that "memory" clobber.  The
> constraints should be fixed accordingly.

Or, at least, put in a barrier() and a comment.

David

> Also, ideally the 'prefetchnta' on q in the x4 xor_syndrome() functions
> would become 'prefetcht0', to match the temporal stores.
> 
> If the scope of this patch is just replacing the store instructions
> though, it looks fine.
> 
> Reviewed-by: Eric Biggers <ebiggers@kernel.org>
> 
> - Eric
> 


^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-16 22:14 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-09  3:08 [PATCH] raid6: use normal stores in the x86 x4 xor_syndrome functions Miroslav Cimerman
2026-11-12 12:21 ` Miroslav Cimerman
2026-11-12 12:33   ` Anton Gavriliuk
2026-11-16  6:11 ` Christoph Hellwig
2026-11-16  6:39   ` AW: " Markus Stockhausen
2026-11-16 21:04 ` Eric Biggers
2026-11-16 22:14   ` David Laight

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®