pgbench buffer readwrite mode showing around 24% perf regression

pgbench buffer readwrite mode showing around 24% perf regression

[samasth.norway.ananda]

Hi,


We recently discovered a performance regression while running pgbench 
buffer readwrite metric over the 6.12 kernel.
After bisecting we were able to narrow it down to the commit
9a42891c35d50a8472b42c61256867b4dfcc1941 (“block: fix lost bio for plug 
enabled bio based device”)

The postgresql db used for this benchmark is stored on an xfs filesystem 
on top of a stripe across 6 disks.
lsblk output -

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda            8:0    0 557.9G  0 disk
└─sda1         8:1    0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdb            8:16   0 557.9G  0 disk
└─sdb1         8:17   0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdc            8:32   0 557.9G  0 disk
└─sdc1         8:33   0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdd            8:48   0 557.9G  0 disk
sde            8:64   0 557.9G  0 disk
└─sde1         8:65   0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdf            8:80   0 557.9G  0 disk
└─sdf1         8:81   0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdg            8:96   0 557.9G  0 disk
└─sdg1         8:97   0 557.9G  0 part
   └─tank-lvm 252:0    0   3.3T  0 lvm  /data
sdh            8:112  0 557.9G  0 disk
├─sdh1         8:113  0     1M  0 part
├─sdh2         8:114  0   9.8G  0 part /boot
├─sdh3         8:115  0    32G  0 part /
├─sdh4         8:116  0     1K  0 part
├─sdh5         8:117  0    16G  0 part [SWAP]
└─sdh6         8:118  0 500.1G  0 part /export/bench
0       /var/crash


The regression was only observed on some architectures.
I ran perf on the pqbench operation and found the following report.

Base kernel based on 6.12 -

# Overhead  Command          Shared Object             Symbol 

# ........  ...............  ........................ 
.........................................
#
      8.34%  pgbench          [kernel.kallsyms]         [k] 
update_sg_lb_stats.isra.0
      3.34%  pgbench          [kernel.kallsyms]         [k] 
entry_SYSRETQ_unsafe_stack
      2.57%  pgbench          [kernel.kallsyms]         [k] 
sched_balance_find_src_rq
      2.44%  pgbench          [kernel.kallsyms]         [k] 
_find_next_and_bit
      2.08%  pgbench          [kernel.kallsyms]         [k] idle_cpu
      1.99%  pgbench          [kernel.kallsyms]         [k] 
__raw_spin_lock_irqsave
      1.56%  pgbench          [kernel.kallsyms]         [k] _raw_spin_lock
      1.31%  pgbench          [kernel.kallsyms]         [k] 
native_queued_spin_lock_slowpath
      1.31%  pgbench          libpq.so.private16-5.16   [.] 
printfPQExpBuffer
      1.13%  pgbench          [kernel.kallsyms]         [k] 
cpu_util.constprop.0
      1.11%  pgbench          [kernel.kallsyms]         [k] 
syscall_return_via_sysret
      1.10%  pgbench          [kernel.kallsyms]         [k] 
rep_movs_alternative
      1.08%  pgbench          [kernel.kallsyms]         [k] unix_poll
      ----------------------cut here----------------------------

Base kernel based on 6.12 – with reverting the bisected commit -

  Overhead  Command          Shared Object            Symbol 

# ........  ...............  ....................... 
.........................................
#
      9.19%  pgbench          [kernel.kallsyms]        [k] 
update_sg_lb_stats.isra.0
      3.44%  pgbench          [kernel.kallsyms]        [k] 
entry_SYSRETQ_unsafe_stack
      3.39%  pgbench          [kernel.kallsyms]        [k] 
_find_next_and_bit
      1.85%  pgbench          [kernel.kallsyms]        [k] 
__raw_spin_lock_irqsave
      1.83%  pgbench          [kernel.kallsyms]        [k] idle_cpu
      1.79%  pgbench          [kernel.kallsyms]        [k] 
sched_balance_find_src_rq
      1.61%  pgbench          [kernel.kallsyms]        [k] _raw_spin_lock
      1.21%  pgbench          [kernel.kallsyms]        [k] entry_SYSCALL_64
      1.16%  pgbench          [kernel.kallsyms]        [k] unix_poll
      1.16%  pgbench          [kernel.kallsyms]        [k] 
syscall_return_via_sysret
      1.12%  pgbench          [kernel.kallsyms]        [k] 
cpu_util.constprop.0
      1.09%  pgbench          [kernel.kallsyms]        [k] 
native_queued_spin_lock_slowpath
      ----------------------cut here----------------------------

Please let me know if you need more information.

Thanks,
Samasth.