[linus:master] [fs/pipe] f7f4665dc5: lmbench3.PIPE.bandwidth.MB/sec 18.5% regression

[linus:master] [fs/pipe] f7f4665dc5: lmbench3.PIPE.bandwidth.MB/sec 18.5% regression

[kernel test robot]

Hello,

kernel test robot noticed a 18.5% regression of lmbench3.PIPE.bandwidth.MB/sec on:


commit: f7f4665dc520fd8bbc1db20e59945773e03744a0 ("fs/pipe: unify the page pools into a single per-pipe pool") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

testcase: lmbench3
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 224 threads 2 sockets Intel(R) Xeon(R) (Sapphire Rapids) with 128G memory
parameters:

	test_memory_size: 50%
	nr_threads: 20%
	mode: development
	test: PIPE
	cpufreq_governor: performance

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <yi1.lai@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202611041404.befe64cd-lkp@intel.com

Details are as below:
-------------------------------------------------------------------------------------------------->

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20261104/202611041404.befe64cd-lkp@intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_threads/rootfs/tbox_group/test/test_memory_size/testcase:
  gcc-14/performance/x86_64-rhel-9.4/development/20%/debian-13-x86_64-20250902.cgz/lkp-spr-2sp2/PIPE/50%/lmbench3

commit: 
  c610d2d078 ("fs: annotate inode timestamp accessors")
  f7f4665dc5 ("fs/pipe: unify the page pools into a single per-pipe pool")

c610d2d0787961cd f7f4665dc520fd8bbc1db20e599
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     29142 ±  2%     -18.5%      23737        lmbench3.PIPE.bandwidth.MB/sec
    191.80 ±  8%     +54.1%     295.65 ±  5%  lmbench3.time.elapsed_time
    191.80 ±  8%     +54.1%     295.65 ±  5%  lmbench3.time.elapsed_time.max
     23261 ± 16%     +19.3%      27743 ±  5%  lmbench3.time.involuntary_context_switches
      9853 ± 16%     +27.9%      12605 ±  5%  lmbench3.time.system_time
     51.80 ±  2%     +11.6%      57.80 ±  3%  lmbench3.time.user_time
 2.928e+08 ±  3%     +26.6%  3.706e+08 ±  4%  lmbench3.time.voluntary_context_switches


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [linus:master] [fs/pipe] f7f4665dc5: lmbench3.PIPE.bandwidth.MB/sec 18.5% regression

[Christian Brauner]

On Thu, Nov 05, 2026 at 08:38:45AM +0800, kernel test robot wrote:
> 
> 
> Hello,
> 
> kernel test robot noticed a 18.5% regression of lmbench3.PIPE.bandwidth.MB/sec on:
> 
> 
> commit: f7f4665dc520fd8bbc1db20e59945773e03744a0 ("fs/pipe: unify the page pools into a single per-pipe pool") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

@Breno can you reproduce this?
It's something we should at least investigate.

> 
> testcase: lmbench3
> config: x86_64-rhel-9.4
> compiler: gcc-14
> test machine: 224 threads 2 sockets Intel(R) Xeon(R) (Sapphire Rapids) with 128G memory
> parameters:
> 
> 	test_memory_size: 50%
> 	nr_threads: 20%
> 	mode: development
> 	test: PIPE
> 	cpufreq_governor: performance
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <yi1.lai@intel.com>
> | Closes: https://lore.kernel.org/oe-lkp/202611041404.befe64cd-lkp@intel.com
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20261104/202611041404.befe64cd-lkp@intel.com
> 
> =========================================================================================
> compiler/cpufreq_governor/kconfig/mode/nr_threads/rootfs/tbox_group/test/test_memory_size/testcase:
>   gcc-14/performance/x86_64-rhel-9.4/development/20%/debian-13-x86_64-20250902.cgz/lkp-spr-2sp2/PIPE/50%/lmbench3
> 
> commit: 
>   c610d2d078 ("fs: annotate inode timestamp accessors")
>   f7f4665dc5 ("fs/pipe: unify the page pools into a single per-pipe pool")
> 
> c610d2d0787961cd f7f4665dc520fd8bbc1db20e599
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>      29142 ±  2%     -18.5%      23737        lmbench3.PIPE.bandwidth.MB/sec
>     191.80 ±  8%     +54.1%     295.65 ±  5%  lmbench3.time.elapsed_time
>     191.80 ±  8%     +54.1%     295.65 ±  5%  lmbench3.time.elapsed_time.max
>      23261 ± 16%     +19.3%      27743 ±  5%  lmbench3.time.involuntary_context_switches
>       9853 ± 16%     +27.9%      12605 ±  5%  lmbench3.time.system_time
>      51.80 ±  2%     +11.6%      57.80 ±  3%  lmbench3.time.user_time
>  2.928e+08 ±  3%     +26.6%  3.706e+08 ±  4%  lmbench3.time.voluntary_context_switches
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> 
> -- 
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
> 
> 
> 

-- 

Re: [linus:master] [fs/pipe] f7f4665dc5: lmbench3.PIPE.bandwidth.MB/sec 18.5% regression

[Breno Leitao]

On Thu, Nov 05, 2026 at 04:04:58PM +0100, Christian Brauner wrote:
> On Thu, Nov 05, 2026 at 08:38:45AM +0800, kernel test robot wrote:
> > kernel test robot noticed a 18.5% regression of lmbench3.PIPE.bandwidth.MB/sec on:
> > 
> > commit: f7f4665dc520fd8bbc1db20e59945773e03744a0 ("fs/pipe: unify the page pools into a single per-pipe pool") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


> @Breno can you reproduce this?
> It's something we should at least investigate.

Ack, I will check it , but not today unfortunately.

I will come back to it tomorrow, and until end of week I will report my
findings.

Re: [linus:master] [fs/pipe] f7f4665dc5: lmbench3.PIPE.bandwidth.MB/sec 18.5% regression

[Mateusz Guzik]

On Thu, Nov 5, 2026 at 4:28 PM Breno Leitao <leitao@debian.org> wrote:
>
> On Thu, Nov 05, 2026 at 04:04:58PM +0100, Christian Brauner wrote:
> > On Thu, Nov 05, 2026 at 08:38:45AM +0800, kernel test robot wrote:
> > > kernel test robot noticed a 18.5% regression of lmbench3.PIPE.bandwidth.MB/sec on:
> > >
> > > commit: f7f4665dc520fd8bbc1db20e59945773e03744a0 ("fs/pipe: unify the page pools into a single per-pipe pool") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
>
> > @Breno can you reproduce this?
> > It's something we should at least investigate.
>
> Ack, I will check it , but not today unfortunately.
>
> I will come back to it tomorrow, and until end of week I will report my
> findings.

I figured it out.

The test at hand has multiple readers and writers using the same pipe,
with 64KB i/os.

Prior to the patch each writer had its own private pool to draw from,
guaranteeing no allocation will happen under the mutex.

With the patch the reader exit also drains the pool.

As a result tons of allocation ends up happening under the lock again:

-   94.88%    94.88%  bw_pipe          [kernel.kallsyms]     [k]
__alloc_frozen_pages_noprof
     94.88% __internal_syscall_cancel
        entry_SYSCALL_64_after_hwframe
        do_syscall_64
        ksys_write
        vfs_write
      - anon_pipe_write
         - 47.46% alloc_pages_noprof
              alloc_pages_mpol
              __alloc_frozen_pages_noprof
         - 47.42% anon_pipe_prefill_and_lock
              alloc_pages_noprof
              alloc_pages_mpol
              __alloc_frozen_pages_noprof

I don't think multiple high-volume writers vs multiple high
volume-readers is a realistic scenario and I would not bother fixing
it.

However, should someone want to address it, I think the patch should
be reverted and the resulting state tweaked. The patch at hand came
with some win for throughput in one reader vs one writer scenario,
which I'm pretty sure stems from postponing page deallocation until
after the mutex is released -- the kernel prior to any of this would
still put_page on the reader side under the lock. This bit would have
to get tweaked to recover the loss in that scenario (not to be
confused with the loss reported here).

tl;dr i think this is a EIRRELEVANT kind of a report