[PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Rosen Penev]

__csum_partial() walks the buffer linearly and reads each cache line
exactly once. On cores without a hardware prefetcher, such as the 464FP
in the APM82181, every new line is therefore a demand miss that stalls
the loop. Packets received by non-coherent DMA are never in the cache,
so this is the common case for GRO checksum validation: when routing
TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
of all cycles and is the top entry in the profile.

Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
dcbt is only a hint and never faults, so prefetching past the end of
the buffer is harmless. Lines that are already cached, for example when
checksumming data the CPU just wrote, are left untouched.

Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
was toggled at runtime, in the same boot:

              LAN->WAN               WAN->LAN
              Mbit/s  csum cycles    Mbit/s  csum cycles
  before      495     30.5%          439     27.2%
  after       599     15.3%          520     13.7%

That is 58% less time spent in __csum_partial() per byte.

The prefetch distance was chosen with an earlier, slower driver
configuration (median of 3, Mbit/s):

  distance    LAN->WAN   WAN->LAN
  none        370        341
  64 bytes    430        381
  96 bytes    438        397
  128 bytes   438        385
  160 bytes   434        386
  256 bytes   423        383

Prefetching the first lines before entering the loop made no
measurable difference and is not done.

Assisted-by: LLM
Signed-off-by: Rosen Penev <rosenp@gmail.com>
---
 v2: reword commit message and retest numbers.
 arch/powerpc/lib/checksum_32.S | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
index cd00b9bdd772..5a14d45c7b9d 100644
--- a/arch/powerpc/lib/checksum_32.S
+++ b/arch/powerpc/lib/checksum_32.S
@@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
 	bdnz	2b
 21:	srwi.	r6,r4,4		/* # blocks of 4 words to do */
 	beq	3f
+	li	r9,4*L1_CACHE_BYTES	/* prefetch distance */
 	lwz	r0,4(r3)
 	mtctr	r6
 	lwz	r6,8(r3)
@@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
 	lwzu	r8,16(r3)
 	adde	r5,r5,r7
 	bdz	23f
-22:	lwz	r0,4(r3)
+22:	dcbt	r3,r9
+	lwz	r0,4(r3)
 	adde	r5,r5,r8
 	lwz	r6,8(r3)
 	adde	r5,r5,r0
-- 
2.56.0

Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Christophe Leroy (CS GROUP)]

Le 12/10/2026 à 20:00, Rosen Penev a écrit :
> __csum_partial() walks the buffer linearly and reads each cache line
> exactly once. On cores without a hardware prefetcher, such as the 464FP
> in the APM82181, every new line is therefore a demand miss that stalls
> the loop. Packets received by non-coherent DMA are never in the cache,
> so this is the common case for GRO checksum validation: when routing
> TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
> of all cycles and is the top entry in the profile.
> 
> Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
> dcbt is only a hint and never faults, so prefetching past the end of
> the buffer is harmless. Lines that are already cached, for example when
> checksumming data the CPU just wrote, are left untouched.
> 
> Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
> routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
> was toggled at runtime, in the same boot:
> 
>                LAN->WAN               WAN->LAN
>                Mbit/s  csum cycles    Mbit/s  csum cycles
>    before      495     30.5%          439     27.2%
>    after       599     15.3%          520     13.7%
> 
> That is 58% less time spent in __csum_partial() per byte.
> 
> The prefetch distance was chosen with an earlier, slower driver
> configuration (median of 3, Mbit/s):
> 
>    distance    LAN->WAN   WAN->LAN
>    none        370        341
>    64 bytes    430        381
>    96 bytes    438        397
>    128 bytes   438        385
>    160 bytes   434        386
>    256 bytes   423        383
> 
> Prefetching the first lines before entering the loop made no
> measurable difference and is not done.
> 
> Assisted-by: LLM
> Signed-off-by: Rosen Penev <rosenp@gmail.com>

csum_partial_copy_generic() does the same, so your patch makes sense.

Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.

Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>

> ---
>   v2: reword commit message and retest numbers.
>   arch/powerpc/lib/checksum_32.S | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
> index cd00b9bdd772..5a14d45c7b9d 100644
> --- a/arch/powerpc/lib/checksum_32.S
> +++ b/arch/powerpc/lib/checksum_32.S
> @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
>   	bdnz	2b
>   21:	srwi.	r6,r4,4		/* # blocks of 4 words to do */
>   	beq	3f
> +	li	r9,4*L1_CACHE_BYTES	/* prefetch distance */
>   	lwz	r0,4(r3)
>   	mtctr	r6
>   	lwz	r6,8(r3)
> @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
>   	lwzu	r8,16(r3)
>   	adde	r5,r5,r7
>   	bdz	23f
> -22:	lwz	r0,4(r3)
> +22:	dcbt	r3,r9
> +	lwz	r0,4(r3)
>   	adde	r5,r5,r8
>   	lwz	r6,8(r3)
>   	adde	r5,r5,r0

Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Rosen Penev]

On Mon, Oct 12, 2026 at 11:18 AM Christophe Leroy (CS GROUP)
<chleroy@kernel.org> wrote:
>
>
>
> Le 12/10/2026 à 20:00, Rosen Penev a écrit :
> > __csum_partial() walks the buffer linearly and reads each cache line
> > exactly once. On cores without a hardware prefetcher, such as the 464FP
> > in the APM82181, every new line is therefore a demand miss that stalls
> > the loop. Packets received by non-coherent DMA are never in the cache,
> > so this is the common case for GRO checksum validation: when routing
> > TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
> > of all cycles and is the top entry in the profile.
> >
> > Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
> > dcbt is only a hint and never faults, so prefetching past the end of
> > the buffer is harmless. Lines that are already cached, for example when
> > checksumming data the CPU just wrote, are left untouched.
> >
> > Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
> > routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
> > was toggled at runtime, in the same boot:
> >
> >                LAN->WAN               WAN->LAN
> >                Mbit/s  csum cycles    Mbit/s  csum cycles
> >    before      495     30.5%          439     27.2%
> >    after       599     15.3%          520     13.7%
> >
> > That is 58% less time spent in __csum_partial() per byte.
> >
> > The prefetch distance was chosen with an earlier, slower driver
> > configuration (median of 3, Mbit/s):
> >
> >    distance    LAN->WAN   WAN->LAN
> >    none        370        341
> >    64 bytes    430        381
> >    96 bytes    438        397
> >    128 bytes   438        385
> >    160 bytes   434        386
> >    256 bytes   423        383
> >
> > Prefetching the first lines before entering the loop made no
> > measurable difference and is not done.
> >
> > Assisted-by: LLM
> > Signed-off-by: Rosen Penev <rosenp@gmail.com>
>
> csum_partial_copy_generic() does the same, so your patch makes sense.
>
> Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.
li      r9,MAX_COPY_PREFETCH*L1_CACHE_BYTES /* prefetch distance */

LLM comment:

What changes per platform: MAX_COPY_PREFETCH is 4 on every 32-bit
config except 8xx, where it is 1. So 44x, 47x and e500mc behave
exactly as before; on the 44x (canyonlands) build the instruction is
still li r9, 128. The one difference is 8xx, which now prefetches 16
bytes ahead instead of 64. That fits how the copy routine already
treats 8xx, but I didn't benchmark it.

My comment:
I don't have an 8xx device so I don't know if this works properly there or not.
>
> Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
I'll keep this in v3.
>
> > ---
> >   v2: reword commit message and retest numbers.
> >   arch/powerpc/lib/checksum_32.S | 4 +++-
> >   1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
> > index cd00b9bdd772..5a14d45c7b9d 100644
> > --- a/arch/powerpc/lib/checksum_32.S
> > +++ b/arch/powerpc/lib/checksum_32.S
> > @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
> >       bdnz    2b
> >   21: srwi.   r6,r4,4         /* # blocks of 4 words to do */
> >       beq     3f
> > +     li      r9,4*L1_CACHE_BYTES     /* prefetch distance */
> >       lwz     r0,4(r3)
> >       mtctr   r6
> >       lwz     r6,8(r3)
> > @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
> >       lwzu    r8,16(r3)
> >       adde    r5,r5,r7
> >       bdz     23f
> > -22:  lwz     r0,4(r3)
> > +22:  dcbt    r3,r9
> > +     lwz     r0,4(r3)
> >       adde    r5,r5,r8
> >       lwz     r6,8(r3)
> >       adde    r5,r5,r0
>

Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Christophe Leroy (CS GROUP)]

Le 12/10/2026 à 20:28, Rosen Penev a écrit :
> On Mon, Oct 12, 2026 at 11:18 AM Christophe Leroy (CS GROUP)
> <chleroy@kernel.org> wrote:
>>
>>
>>
>> Le 12/10/2026 à 20:00, Rosen Penev a écrit :
>>> __csum_partial() walks the buffer linearly and reads each cache line
>>> exactly once. On cores without a hardware prefetcher, such as the 464FP
>>> in the APM82181, every new line is therefore a demand miss that stalls
>>> the loop. Packets received by non-coherent DMA are never in the cache,
>>> so this is the common case for GRO checksum validation: when routing
>>> TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
>>> of all cycles and is the top entry in the profile.
>>>
>>> Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
>>> dcbt is only a hint and never faults, so prefetching past the end of
>>> the buffer is harmless. Lines that are already cached, for example when
>>> checksumming data the CPU just wrote, are left untouched.
>>>
>>> Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
>>> routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
>>> was toggled at runtime, in the same boot:
>>>
>>>                 LAN->WAN               WAN->LAN
>>>                 Mbit/s  csum cycles    Mbit/s  csum cycles
>>>     before      495     30.5%          439     27.2%
>>>     after       599     15.3%          520     13.7%
>>>
>>> That is 58% less time spent in __csum_partial() per byte.
>>>
>>> The prefetch distance was chosen with an earlier, slower driver
>>> configuration (median of 3, Mbit/s):
>>>
>>>     distance    LAN->WAN   WAN->LAN
>>>     none        370        341
>>>     64 bytes    430        381
>>>     96 bytes    438        397
>>>     128 bytes   438        385
>>>     160 bytes   434        386
>>>     256 bytes   423        383
>>>
>>> Prefetching the first lines before entering the loop made no
>>> measurable difference and is not done.
>>>
>>> Assisted-by: LLM
>>> Signed-off-by: Rosen Penev <rosenp@gmail.com>
>>
>> csum_partial_copy_generic() does the same, so your patch makes sense.
>>
>> Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.
> li      r9,MAX_COPY_PREFETCH*L1_CACHE_BYTES /* prefetch distance */
> 
> LLM comment:
> 
> What changes per platform: MAX_COPY_PREFETCH is 4 on every 32-bit
> config except 8xx, where it is 1. So 44x, 47x and e500mc behave
> exactly as before; on the 44x (canyonlands) build the instruction is
> still li r9, 128. The one difference is 8xx, which now prefetches 16
> bytes ahead instead of 64. That fits how the copy routine already
> treats 8xx, but I didn't benchmark it.
> 
> My comment:
> I don't have an 8xx device so I don't know if this works properly there or not.

I have an 885 and might do the test one day, but I won't have time 
during the coming weeks. Nevertheless I have the feeling it makes sense 
to implement something based on MAX_COPY_PREFETCH to resemble 
csum_partial_copy_generic()

>>
>> Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
> I'll keep this in v3.
>>
>>> ---
>>>    v2: reword commit message and retest numbers.
>>>    arch/powerpc/lib/checksum_32.S | 4 +++-
>>>    1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
>>> index cd00b9bdd772..5a14d45c7b9d 100644
>>> --- a/arch/powerpc/lib/checksum_32.S
>>> +++ b/arch/powerpc/lib/checksum_32.S
>>> @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
>>>        bdnz    2b
>>>    21: srwi.   r6,r4,4         /* # blocks of 4 words to do */
>>>        beq     3f
>>> +     li      r9,4*L1_CACHE_BYTES     /* prefetch distance */
>>>        lwz     r0,4(r3)
>>>        mtctr   r6
>>>        lwz     r6,8(r3)
>>> @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
>>>        lwzu    r8,16(r3)
>>>        adde    r5,r5,r7
>>>        bdz     23f
>>> -22:  lwz     r0,4(r3)
>>> +22:  dcbt    r3,r9
>>> +     lwz     r0,4(r3)
>>>        adde    r5,r5,r8
>>>        lwz     r6,8(r3)
>>>        adde    r5,r5,r0
>>

Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

[Rosen Penev]

On Mon, Oct 12, 2026 at 11:38 AM Christophe Leroy (CS GROUP)
<chleroy@kernel.org> wrote:
>
>
>
> Le 12/10/2026 à 20:28, Rosen Penev a écrit :
> > On Mon, Oct 12, 2026 at 11:18 AM Christophe Leroy (CS GROUP)
> > <chleroy@kernel.org> wrote:
> >>
> >>
> >>
> >> Le 12/10/2026 à 20:00, Rosen Penev a écrit :
> >>> __csum_partial() walks the buffer linearly and reads each cache line
> >>> exactly once. On cores without a hardware prefetcher, such as the 464FP
> >>> in the APM82181, every new line is therefore a demand miss that stalls
> >>> the loop. Packets received by non-coherent DMA are never in the cache,
> >>> so this is the common case for GRO checksum validation: when routing
> >>> TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
> >>> of all cycles and is the top entry in the profile.
> >>>
> >>> Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
> >>> dcbt is only a hint and never faults, so prefetching past the end of
> >>> the buffer is harmless. Lines that are already cached, for example when
> >>> checksumming data the CPU just wrote, are left untouched.
> >>>
> >>> Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
> >>> routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
> >>> was toggled at runtime, in the same boot:
> >>>
> >>>                 LAN->WAN               WAN->LAN
> >>>                 Mbit/s  csum cycles    Mbit/s  csum cycles
> >>>     before      495     30.5%          439     27.2%
> >>>     after       599     15.3%          520     13.7%
> >>>
> >>> That is 58% less time spent in __csum_partial() per byte.
> >>>
> >>> The prefetch distance was chosen with an earlier, slower driver
> >>> configuration (median of 3, Mbit/s):
> >>>
> >>>     distance    LAN->WAN   WAN->LAN
> >>>     none        370        341
> >>>     64 bytes    430        381
> >>>     96 bytes    438        397
> >>>     128 bytes   438        385
> >>>     160 bytes   434        386
> >>>     256 bytes   423        383
> >>>
> >>> Prefetching the first lines before entering the loop made no
> >>> measurable difference and is not done.
> >>>
> >>> Assisted-by: LLM
> >>> Signed-off-by: Rosen Penev <rosenp@gmail.com>
> >>
> >> csum_partial_copy_generic() does the same, so your patch makes sense.
> >>
> >> Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.
> > li      r9,MAX_COPY_PREFETCH*L1_CACHE_BYTES /* prefetch distance */
> >
> > LLM comment:
> >
> > What changes per platform: MAX_COPY_PREFETCH is 4 on every 32-bit
> > config except 8xx, where it is 1. So 44x, 47x and e500mc behave
> > exactly as before; on the 44x (canyonlands) build the instruction is
> > still li r9, 128. The one difference is 8xx, which now prefetches 16
> > bytes ahead instead of 64. That fits how the copy routine already
> > treats 8xx, but I didn't benchmark it.
> >
> > My comment:
> > I don't have an 8xx device so I don't know if this works properly there or not.
>
> I have an 885 and might do the test one day, but I won't have time
> during the coming weeks. Nevertheless I have the feeling it makes sense
> to implement something based on MAX_COPY_PREFETCH to resemble
> csum_partial_copy_generic()
I got a user to test this setup with MAX_COPY_PREFETCH

Using Sophos RED 15w, right after sysupgrade -n, got an iperf3 -c
result of ~633Mb/s with the suggested change, vs. ~583Mb/s with the
latest snapshot build from the web site, ie. without this PR.
>
> >>
> >> Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
> > I'll keep this in v3.
> >>
> >>> ---
> >>>    v2: reword commit message and retest numbers.
> >>>    arch/powerpc/lib/checksum_32.S | 4 +++-
> >>>    1 file changed, 3 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
> >>> index cd00b9bdd772..5a14d45c7b9d 100644
> >>> --- a/arch/powerpc/lib/checksum_32.S
> >>> +++ b/arch/powerpc/lib/checksum_32.S
> >>> @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
> >>>        bdnz    2b
> >>>    21: srwi.   r6,r4,4         /* # blocks of 4 words to do */
> >>>        beq     3f
> >>> +     li      r9,4*L1_CACHE_BYTES     /* prefetch distance */
> >>>        lwz     r0,4(r3)
> >>>        mtctr   r6
> >>>        lwz     r6,8(r3)
> >>> @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
> >>>        lwzu    r8,16(r3)
> >>>        adde    r5,r5,r7
> >>>        bdz     23f
> >>> -22:  lwz     r0,4(r3)
> >>> +22:  dcbt    r3,r9
> >>> +     lwz     r0,4(r3)
> >>>        adde    r5,r5,r8
> >>>        lwz     r6,8(r3)
> >>>        adde    r5,r5,r0
> >>
>