Re: [PATCH] x86: Drop KGDB support

[David Laight <david.laight.linux@gmail.com>]

On Fri, 30 Oct 2026 15:56:48 -0500
Steve Wahl <steve.wahl@hpe.com> wrote:

> On Fri, Oct 30, 2026 at 04:28:41PM +0000, David Laight wrote:
> > On Thu, 29 Oct 2026 10:32:58 -0500
> > Steve Wahl <steve.wahl@hpe.com> wrote:
> >   
> > > On Thu, Oct 29, 2026 at 10:51:21AM +0000, David Laight wrote:  
> > > > On Wed, 28 Oct 2026 14:58:12 -0500
> > > > Steve Wahl <steve.wahl@hpe.com> wrote:
> > > >     
> > > > > On Wed, Oct 28, 2026 at 08:10:54AM -0700, Dave Hansen wrote:    
> > > > ...    
> > > > > 
> > > > > For an example, I recently used this method to discover that at
> > > > > least one version of Intel's MPI library likes to read up
> > > > > /proc/kallsyms from each of the processes it starts, and somewhere
> > > > > north of 1000's of processors the atomic increment on opening
> > > > > kallsyms, implemented with a compare-exchange, becomes a hot lock
> > > > > and initialization that takes little time on a smaller number of
> > > > > processors, suddenly balloons to half an hour.    
> > > > 
> > > > I'm guessing that was pre-ARMv8.1 ??    
> > > 
> > > x86, not arm.  And it was on a distro kernel, I didn't check the
> > > current community kernel because it was a situation where it was
> > > enough to understand what's going on in the distro kernel, and avoid
> > > the behavior in the application.   
> > 
> > x86 shouldn't have an exponential issue with cas.
> > Something else must have been going on.  
> 
> Well, the large majority of some 2000+ cpus were all stuck in
> use_pde() in inode.c, which is
> 
> 	return likely(atomic_inc_unless_negative(&pde->in_use));

There will be one of those per read system call.
If the code is using a sane buffer length that isn't really that many at all
(unless you are running an 'allyesconfig' kernel).
Provided the library isn't doing something really silly like 1 byte reads!

It is possible that /proc/kallsyms could be marked PROC_ENTRY_PERMANENT
and so avoid that atomic operation.

> 
> atomic_inc_unless_negative() has alternates, but I think there's no
> arch version for x86, so comes down to this:
> 
> static __always_inline bool
> raw_atomic_inc_unless_negative(atomic_t *v)
> {
> 	int c = raw_atomic_read(v);
> 
> 	do {
> 		if (unlikely(c < 0))
> 			return false;
> 	} while (!raw_atomic_try_cmpxchg(v, &c, c + 1));
> 
> 	return true;
> }
> 
> If you manage to synchronize thousands of processors among many NUMA
> nodes all trying to do that same thing, how can that not create a
> horrible cache coherency protocol traffic jam?

On x86, in order for the cmpxchg to fail it must have succeeded for
a different cpu (as far as I understand things...).
So yes, you get a lot of cache coherency traffic but progress should
be made - probably with a maximum total rate of increments that you
hit well before 1000 cpu hit the code path.

It might just be that with the large NUMA memory system the required number
of cache line bounces (one for each successful CAS) just takes that length
of time - but that is linear in the number of applications/cpus. 

OTOH an LL/SC loop loses the cache when another cpu does the LL (aka read)
bit. So when heavily contended no cpu manages to do a successful write
and overall absolutely no progress is made.
The more instructions between the LL and SC the worse it gets.

David

> 
> Thanks for your thoughts on this; if our hardware is supposed to be
> doing something to alleviate this and it's not, I / we want to know!
> 
> --> Steve  
> 
> > David
> >   
> > >   
> > > > The LL/SC retry loops for atomics are completely broken for contended
> > > > operations.
> > > > The kernel needs to patch them to use the LSE (single instruction
> > > > atomics) for ARMv8.1 which guarantee that some progress is made.
> > > > A CAS loop can still get starved on a single CPU, but some CPU
> > > > do proceed.
> > > > You really don't want to use LL/LC on a system with hraair CPU.
> > > > 
> > > > One CPU (not arm) that implements LL/SC locks the cache line for a
> > > > few clocks after LL to give a reasonable chance of the associated SC
> > > > actually succeeding before the cache line is 'stolen' by another CPU
> > > > doing an LL.    
> > > 
> > > Thanks,
> > >   
> > > --> Steve    
> > >   
> >   
>