From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932184AbbCQMQ1 (ORCPT ); Tue, 17 Mar 2015 08:16:27 -0400 Received: from cam-admin0.cambridge.arm.com ([217.140.96.50]:64078 "EHLO cam-admin0.cambridge.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752520AbbCQMQ0 (ORCPT ); Tue, 17 Mar 2015 08:16:26 -0400 From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: Mark Rutland , Andrew Morton , Catalin Marinas , Christoph Lameter , David Rientjes , Jesper Dangaard Brouer , Joonsoo Kim , Linus Torvalds , Pekka Enberg , Steve Capper Subject: [PATCHv2] mm/slub: fix lockups on PREEMPT && !SMP kernels Date: Tue, 17 Mar 2015 12:15:33 +0000 Message-Id: <1426594533-18221-1-git-send-email-mark.rutland@arm.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <20150317120058.GC23340@leverpostej> References: <20150317120058.GC23340@leverpostej> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit 9aabf810a67cd97e ("mm/slub: optimize alloc/free fastpath by removing preemption on/off") introduced an occasional hang for kernels built with CONFIG_PREEMPT && !CONFIG_SMP. The problem is the following loop the patch introduced to slab_alloc_node and slab_free: do { tid = this_cpu_read(s->cpu_slab->tid); c = raw_cpu_ptr(s->cpu_slab); } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); GCC 4.9 has been observed to hoist the load of c and c->tid above the loop for !SMP kernels (as in this case raw_cpu_ptr(x) is compile-time constant and does not force a reload). On arm64 the generated assembly looks like: ffffffc00016d3c4: f9400404 ldr x4, [x0,#8] ffffffc00016d3c8: f9400401 ldr x1, [x0,#8] ffffffc00016d3cc: eb04003f cmp x1, x4 ffffffc00016d3d0: 54ffffc1 b.ne ffffffc00016d3c8 If the thread is preempted between the load of c->tid (into x1) and tid (into x4), and an allocation or free occurs in another thread (bumping the cpu_slab's tid), the thread will be stuck in the loop until s->cpu_slab->tid wraps, which may be forever in the absence of allocations/frees on the same CPU. This patch changes the loop condition to access c->tid with READ_ONCE. This ensures that the value is reloaded even when the compiler would otherwise assume it could cache the value, and also ensures that the load will not be torn. Signed-off-by: Mark Rutland Cc: Andrew Morton Cc: Catalin Marinas Cc: Christoph Lameter Cc: David Rientjes Cc: Jesper Dangaard Brouer Cc: Joonsoo Kim Cc: Linus Torvalds Cc: Pekka Enberg Cc: Steve Capper --- mm/slub.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) Since v1 [1]: * Do not erroneously remove the loop [1] lkml.kernel.org/r/1426261632-8911-1-git-send-email-mark.rutland@arm.com diff --git a/mm/slub.c b/mm/slub.c index 6832c4e..82c4737 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2449,7 +2449,8 @@ redo: do { tid = this_cpu_read(s->cpu_slab->tid); c = raw_cpu_ptr(s->cpu_slab); - } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); + } while (IS_ENABLED(CONFIG_PREEMPT) && + unlikely(tid != READ_ONCE(c->tid))); /* * Irqless object alloc/free algorithm used here depends on sequence @@ -2718,7 +2719,8 @@ redo: do { tid = this_cpu_read(s->cpu_slab->tid); c = raw_cpu_ptr(s->cpu_slab); - } while (IS_ENABLED(CONFIG_PREEMPT) && unlikely(tid != c->tid)); + } while (IS_ENABLED(CONFIG_PREEMPT) && + unlikely(tid != READ_ONCE(c->tid))); /* Same with comment on barrier() in slab_alloc_node() */ barrier(); -- 1.9.1