mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Calvin Owens <calvin@wbinvd.org>
To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: linux-kernel@vger.kernel.org, "Lai, Yi" <yi1.lai@linux.intel.com>,
	"Peter Zijlstra (Intel)" <peterz@infradead.org>,
	x86@kernel.org
Subject: Re: [tip: locking/urgent] futex: Allow to resize the private local hash
Date: Wed, 18 Jun 2025 13:56:45 -0700	[thread overview]
Message-ID: <aFMoDcWy-OzE3yoV@mozart.vkv.me> (raw)
In-Reply-To: <20250618170924.Z34OXf1E@linutronix.de>

( Dropping linux-tip-commits from Cc )

On Wednesday 06/18 at 19:09 +0200, Sebastian Andrzej Siewior wrote:
> On 2025-06-18 09:49:18 [-0700], Calvin Owens wrote:
> > Didn't get much out of lockdep unfortunately.
> > 
> > It notices the corruption in the spinlock:
> > 
> >     BUG: spinlock bad magic on CPU#2, cargo/4129172
> >      lock: 0xffff8881410ecdc8, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1
> 
> Yes. Which is what I assumed while I suggested this. But it complains
> about bad magic. It says the magic is 0xdead4ead but this is
> SPINLOCK_MAGIC. I was expecting any value but this one.
> 
> > That was followed by this WARN:
> > 
> >     ------------[ cut here ]------------
> >     rcuref - imbalanced put()
> >     WARNING: CPU: 2 PID: 4129172 at lib/rcuref.c:266 rcuref_put_slowpath+0x55/0x70
> 
> This is "reasonable". If the lock is broken, the remaining memory is
> probably garbage anyway. It complains there that the reference put due
> to invalid counter.
> 
> …
> > The oops after that is from a different task this time, but it just
> > looks like slab corruption:
> > 
> …
> 
> The previous complained an invalid free from within the exec.
> 
> > No lock/rcu splats at all.
> It exploded before that could happen.
> 
> > > If it still explodes without LTO, would you mind trying gcc?
> > 
> > Will do.
> 
> Thank you.
> 
> > Haven't had much luck isolating what triggers it, but if I run two copies
> > of these large build jobs in a loop, it reliably triggers in 6-8 hours.
> > 
> > Just to be clear, I can only trigger this on the one machine. I ran it
> > through memtest86+ yesterday and it passed, FWIW, but I'm a little
> > suspicious of the hardware right now too. I double checked that
> > everything in the BIOS related to power/perf is at factory settings.
> 
> But then it is kind of odd that it happens only with the futex code.

I think the missing ingredient was PREEMPT: the 2nd machine has been
trying for over a day, but I rebuilt its kernel with PREEMPT_FULL this
morning (still llvm), and it just hit a similar oops.

    Oops: general protection fault, probably for non-canonical address 0x74656d2f74696750: 0000 [#1] SMP
    CPU: 10 UID: 1000 PID: 542469 Comm: cargo Not tainted 6.16.0-rc2-00045-g4663747812d1 #1 PREEMPT 
    Hardware name: Gigabyte Technology Co., Ltd. A620I AX/A620I AX, BIOS F3 07/10/2023
    RIP: 0010:futex_hash+0x23/0x90
    Code: 1f 84 00 00 00 00 00 41 57 41 56 53 48 89 fb e8 b3 04 fe ff 48 89 df 31 f6 e8 79 00 00 00 48 8b 78 18 49 89 c6 48 85 ff 74 55 <80> 7f 21 00 75 4f f0 83 07 01 79 49 e8 fc 17 37 00 84 c0 75 40 e8
    RSP: 0018:ffffc9002e46fcd8 EFLAGS: 00010202
    RAX: ffff888a68e25c40 RBX: ffffc9002e46fda0 RCX: 0000000036616534
    RDX: 00000000ffffffff RSI: 0000000910180c00 RDI: 74656d2f7469672f
    RBP: 00000000000000b0 R08: 000000000318dd0d R09: 000000002e117cb0
    R10: 00000000318dd0d0 R11: 000000000000001b R12: 0000000000000000
    R13: 000055e79b431170 R14: ffff888a68e25c40 R15: ffff8881ea0ae900
    FS:  00007f1b6037b580(0000) GS:ffff8898a528b000(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    CR2: 0000555830170098 CR3: 0000000d73e93000 CR4: 0000000000350ef0
    Call Trace:
     <TASK>
     futex_wait_setup+0x7e/0x1d0
     __futex_wait+0x63/0x120
     ? __futex_wake_mark+0x40/0x40
     futex_wait+0x5b/0xd0
     ? hrtimer_dummy_timeout+0x10/0x10
     do_futex+0x86/0x120
     __x64_sys_futex+0x10a/0x180
     do_syscall_64+0x48/0x4f0
     entry_SYSCALL_64_after_hwframe+0x4b/0x53

I also enabled DEBUG_PREEMPT, but that didn't print any additional info.

I'm testing a GCC kernel on both machines now.

Thanks,
Calvin

> Sebastian

  reply	other threads:[~2025-06-18 20:56 UTC|newest]

Thread overview: 109+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-04-16 16:29 [PATCH v12 00/21] futex: Add support task local hash maps, FUTEX2_NUMA and FUTEX2_MPOL Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 01/21] rcuref: Provide rcuref_is_dead() Sebastian Andrzej Siewior
2025-05-05 21:09   ` André Almeida
2025-05-08 10:34   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 02/21] mm: Add vmalloc_huge_node() Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 03/21] futex: Move futex_queue() into futex_wait_setup() Sebastian Andrzej Siewior
2025-05-05 21:43   ` André Almeida
2025-05-16 12:53     ` Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 04/21] futex: Pull futex_hash() out of futex_q_lock() Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 05/21] futex: Create hb scopes Sebastian Andrzej Siewior
2025-05-06 23:45   ` André Almeida
2025-05-16 12:20     ` Sebastian Andrzej Siewior
2025-05-16 13:23     ` Peter Zijlstra
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 06/21] futex: Create futex_hash() get/put class Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 07/21] futex: Create private_hash() " Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 08/21] futex: Acquire a hash reference in futex_wait_multiple_setup() Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 09/21] futex: Decrease the waiter count before the unlock operation Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 10/21] futex: Introduce futex_q_lockptr_lock() Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-05-08 19:06   ` [PATCH v12 10/21] " André Almeida
2025-05-16 12:18     ` Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 11/21] futex: Create helper function to initialize a hash slot Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 12/21] futex: Add basic infrastructure for local task local hash Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 13/21] futex: Allow automatic allocation of process wide futex hash Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 14/21] futex: Allow to resize the private local hash Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-05-08 20:32   ` [PATCH v12 14/21] " André Almeida
2025-05-16 10:49     ` Sebastian Andrzej Siewior
2025-05-16 13:00       ` André Almeida
2025-05-10  8:45   ` [PATCH] futex: Fix futex_mm_init() build failure on older compilers, remove rcu_assign_pointer() Ingo Molnar
2025-05-11  8:11     ` [tip: locking/futex] futex: Relax the rcu_assign_pointer() assignment of mm->futex_phash in futex_mm_init() tip-bot2 for Ingo Molnar
2025-06-01  7:39   ` [PATCH v12 14/21] futex: Allow to resize the private local hash Lai, Yi
2025-06-02 11:00     ` Sebastian Andrzej Siewior
2025-06-02 14:36       ` Lai, Yi
2025-06-02 14:44         ` Sebastian Andrzej Siewior
2025-06-02 15:00           ` Lai, Yi
2025-06-11  9:20       ` [tip: locking/urgent] " tip-bot2 for Sebastian Andrzej Siewior
2025-06-11 14:39       ` tip-bot2 for Sebastian Andrzej Siewior
2025-06-11 14:43         ` Sebastian Andrzej Siewior
2025-06-11 15:11           ` Peter Zijlstra
2025-06-11 15:20             ` Peter Zijlstra
2025-06-11 15:35               ` Sebastian Andrzej Siewior
2025-06-16 17:14         ` Calvin Owens
2025-06-17  7:16           ` Sebastian Andrzej Siewior
2025-06-17  9:23             ` Calvin Owens
2025-06-17  9:50               ` Sebastian Andrzej Siewior
2025-06-17 16:11                 ` Calvin Owens
2025-06-18  2:15                   ` Calvin Owens
2025-06-18 16:47                     ` Sebastian Andrzej Siewior
2025-06-18 16:03                   ` Sebastian Andrzej Siewior
2025-06-18 16:49                     ` Calvin Owens
2025-06-18 17:09                       ` Sebastian Andrzej Siewior
2025-06-18 20:56                         ` Calvin Owens [this message]
2025-06-18 22:47                           ` Calvin Owens
2025-06-19 21:07                             ` Calvin Owens
2025-06-20 10:31                               ` Sebastian Andrzej Siewior
2025-06-20 18:56                                 ` Calvin Owens
2025-06-21  1:02                                   ` Calvin Owens
2025-06-21  7:24                                     ` Calvin Owens
2025-06-21 21:01                                       ` Sebastian Andrzej Siewior
2025-06-22 16:17                                         ` Calvin Owens
2025-04-16 16:29 ` [PATCH v12 15/21] futex: Allow to make the private hash immutable Sebastian Andrzej Siewior
2025-05-02 18:01   ` Peter Zijlstra
2025-05-05  7:14     ` Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 16/21] futex: Implement FUTEX2_NUMA Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 17/21] futex: Implement FUTEX2_MPOL Sebastian Andrzej Siewior
2025-05-02 18:45   ` Peter Zijlstra
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Peter Zijlstra
2025-04-16 16:29 ` [PATCH v12 18/21] tools headers: Synchronize prctl.h ABI header Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 19/21] tools/perf: Allow to select the number of hash buckets Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 20/21] selftests/futex: Add futex_priv_hash Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-05-09 21:22   ` [PATCH v12 20/21] " André Almeida
2025-05-16  7:38     ` Sebastian Andrzej Siewior
2025-05-27 11:28   ` Mark Brown
2025-05-27 12:23     ` Sebastian Andrzej Siewior
2025-05-27 12:35       ` Mark Brown
2025-05-27 12:43         ` Sebastian Andrzej Siewior
2025-05-27 12:59           ` Mark Brown
2025-05-27 13:25             ` Sebastian Andrzej Siewior
2025-05-27 13:40               ` Mark Brown
2025-05-27 13:45                 ` Sebastian Andrzej Siewior
2025-04-16 16:29 ` [PATCH v12 21/21] selftests/futex: Add futex_numa_mpol Sebastian Andrzej Siewior
2025-05-02 19:08   ` Peter Zijlstra
2025-05-05  7:33     ` Sebastian Andrzej Siewior
2025-05-02 19:16   ` Peter Zijlstra
2025-05-05  7:36     ` Sebastian Andrzej Siewior
2025-05-08 10:33   ` [tip: locking/futex] " tip-bot2 for Sebastian Andrzej Siewior
2025-04-16 16:31 ` [PATCH v12 00/21] futex: Add support task local hash maps, FUTEX2_NUMA and FUTEX2_MPOL Sebastian Andrzej Siewior
2025-05-02 19:48   ` Peter Zijlstra
2025-05-03 10:09     ` Peter Zijlstra
2025-05-05  7:30       ` Sebastian Andrzej Siewior
2025-05-06  7:36         ` Peter Zijlstra
2025-05-09 11:41           ` Sebastian Andrzej Siewior

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aFMoDcWy-OzE3yoV@mozart.vkv.me \
    --to=calvin@wbinvd.org \
    --cc=bigeasy@linutronix.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=x86@kernel.org \
    --cc=yi1.lai@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®