Re: [PATCH v5 1/2] selftests/resctrl: Implement cl_flush() and sb() for aarch64

[Richard Cheng <icheng@nvidia.com>]

On Wed, Oct 21, 2026 at 10:55:35AM +0800, Ben Horgan wrote:
> Hi Richard,
> 
> On 20/10/2026 17:30, Ben Horgan wrote:
> > Hi Richard,
> > 
> > Thanks for having a look at the arm64 side of the resctrl selftests. That hasn't had much love but
> > it would be good to have them working properly.
> > 
> > On 12/10/2026 08:47, Richard Cheng wrote:
> >> cl_flush() and sb() compile to empty functions on aarch64, making
> >> mem_flush() a silent no-op and leaving CMT/CAT tests operating on
> >> unflushed state.
> >>
> >> Add "dsb sy" for sb(), ARM requires a DSB whose access type covers both
> >> loads and stores.
> > 
> > I guess this is referring to the synchronization required for the cache maintenance instruction but
> > that's not what you are using in this patch.
> > 
> >>
> >> For cl_flush(), "dc civac" only reaches the Point of Coherency, on
> >> ARM MPAM systems the SLC lies past the PoC and may be a NOP on coherent
> >> platforms. Instead, dirty each cacheline with a store so that
> >> mem_flush's full-buffer sweep evicts lines from the SLC via LRU
> >> pressure.
> >>
> >> Signed-off-by: Richard Cheng <icheng@nvidia.com>
> >> ---
> >>  tools/testing/selftests/resctrl/fill_buf.c | 11 +++++++++++
> >>  1 file changed, 11 insertions(+)
> >>
> >> diff --git a/tools/testing/selftests/resctrl/fill_buf.c b/tools/testing/selftests/resctrl/fill_buf.c
> >> index b9fa7968cd6e..eae0262fe265 100644
> >> --- a/tools/testing/selftests/resctrl/fill_buf.c
> >> +++ b/tools/testing/selftests/resctrl/fill_buf.c
> >> @@ -27,6 +27,9 @@ static void sb(void)
> >>  #if defined(__i386) || defined(__x86_64)
> >>  	asm volatile("sfence\n\t"
> >>  		     : : : "memory");
> >> +#elif defined(__aarch64__)
> >> +	asm volatile("dsb sy\n\t"
> >> +		     : : : "memory");
> >>  #endif
> >>  }
> >>  
> >> @@ -35,6 +38,14 @@ static void cl_flush(void *p)
> >>  #if defined(__i386) || defined(__x86_64)
> >>  	asm volatile("clflush (%0)\n\t"
> >>  		     : : "r"(p) : "memory");
> >> +#elif defined(__aarch64__)
> >> +	/*
> >> +	 * Dirty the cache line with a store. As mem_flush() sweeps
> >> +	 * the full test buffer (sized larger than the SLC), cache pressure
> >> +	 * evicts lines from the SLC via LRU replacement.
> >> +	 */
> >> +	asm volatile("strb wzr, [%0]\n\t"
> >> +		     : : "r"(p) : "memory");
> >>  #endif
> 
> One more thing that occurred to me is that if this is intended to evict all existing data in the
> cache then it would require that the cache portions bitmap for the control group (PARTID) in use is
> set to all 1s. E.g. 'L3:0=0xffff'. Otherwise, you may not affect the whole cache. This likely means
> that the cl_flush() function isn't the right abstraction for MPAM and the test needs a bit of
> restructuring to get this to fit in elegantly. Similarly, to the limbo handler in resctrl you could
> then check the llc_occupancy() to check it's under a "low enough" threshold.
> 
> Thanks,
> 
> Ben
>

Hi Ben,

Thanks for the feedback. I checked the code and ran some tests.
You're right, my implementation writes into the benchmark's own buffer using
the same PARRID/PMG, and the buffer isn't always larger than the SLC.

I can't claim that it produces an eviction sweep 100%.

I think what are want to do is to check whether reducing a group's cache
allocation causes more cache misses, thus the data left in cache from the
previous allocation must not affect the next measurement. That is also
why I want to reuse the concept of Intel's test with CLFLUSH before
warming the buffer under the new allocation.

I'm thinking maybe the assumption can't be achieve on ARM, I ran tests to
measure occupancy before and after each preparation method.

I tried this patch's implement, and "DC CIVAC" since I think it can affectively
invalidate cacheline without throwing old data away, which should left
occupancy almost unchanged.
And one more method with a separate large buffer in another group.

The result turns out like the following

Preparation                     Before (MiB)   After (MiB)
DC CIVAC + DSB SY                     51.19         50.47
zero-byte stores + DSB SY          53.12         56.43
Separate-buffer pressure             53.25          7.14


The current method increased the benchmark group's occupancy instead of
reducing it. "DC CIVAC" left occupancy almost unchanged.
The separate buffer method reduced occupancy very much, but still left data
behind.

It makes me think we should reconsider the test method. I think the goal is
to check whether the MPAM allocation limit works, clearing the cache is only
how the Intel RDT test prepares for that check.

How about we set a group's mask before its worker initialize the buffer and
then run sustained cache traffic and measure occupancy after it settles?
Maybe compare full, half and quarter masks to see whether the allocation limits take effect.

Does that sound like a useful direction for MPAM test for you ?

Best regards,
Richard Cheng.


 
> > 
> > Isn't this cl_flush() used without changing the resctrl control and monitor group (PARTID/PMG)?
> > Hence, I expect you are ending up making the cache full of data associated with the identifiers that
> > the benchmark runs with. Is that the case?
> > 
> > I'm also unclear how this the cat_test will detect any performance differences on arm/MPAM as the
> > perf events would need to specific to the system level cache implementation.
> > 
> > Thanks,
> > 
> > Ben
> > 
> > 
> > 
> >>  }
> >>  
> > 
>