From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759425AbcHaKR7 (ORCPT ); Wed, 31 Aug 2016 06:17:59 -0400 Received: from mail-pa0-f51.google.com ([209.85.220.51]:34757 "EHLO mail-pa0-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759329AbcHaKR6 (ORCPT ); Wed, 31 Aug 2016 06:17:58 -0400 Subject: Re: [RFC][PATCH] Fix a race between rwsem and the scheduler To: Peter Zijlstra References: <4050f2ce-1aee-d2aa-39e3-36e995b56252@gmail.com> <20160830121937.GQ10138@twins.programming.kicks-ass.net> <1e2e1a49-db2f-bdae-53b2-0bda225be472@gmail.com> <20160831072835.GB10138@twins.programming.kicks-ass.net> Cc: LKML , Oleg Nesterov , Benjamin Herrenschmidt , Nicholas Piggin , Alexey Kardashevskiy From: Balbir Singh Message-ID: Date: Wed, 31 Aug 2016 20:17:50 +1000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <20160831072835.GB10138@twins.programming.kicks-ass.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 31/08/16 17:28, Peter Zijlstra wrote: > On Wed, Aug 31, 2016 at 01:41:33PM +1000, Balbir Singh wrote: >> On 30/08/16 22:19, Peter Zijlstra wrote: >>> On Tue, Aug 30, 2016 at 06:49:37PM +1000, Balbir Singh wrote: >>>> >>>> >>>> The origin of the issue I've seen seems to be related to >>>> rwsem spin lock stealing. Basically I see the system deadlock'd in the >>>> following state >>> >>> As Nick says (good to see you're back Nick!), this is unrelated to >>> rwsems. >>> >>> This is true for pretty much every blocking wait loop out there, they >>> all do: >>> >>> for (;;) { >>> current->state = UNINTERRUPTIBLE; >>> smp_mb(); >>> if (cond) >>> break; >>> schedule(); >>> } >>> current->state = RUNNING; >>> >>> Which, if the wakeup is spurious, is just the pattern you need. >> >> Yes True! My bad Alexey had seen the same basic pattern, I should have been clearer >> in my commit log. Should I resend the patch? > > Yes please. > Done, just now. I've tried to generalize the issue, but I've kept the example >>> There isn't an MB there. The best I can do is UNLOCK+LOCK, which, thanks >>> to PPC, is _not_ MB. It is however sufficient for this case. >>> >> >> The MB comes from the __switch_to() in schedule(). Ben mentioned it in a >> different thread. > > Right, although even without that, there is sufficient ordering, as the > rq unlock from the wakeup, coupled with the rq lock from the schedule > already form a load-store barrier. > >>> Now, this has been present for a fair while, I suspect ever since we >>> reworked the wakeup path to not use rq->lock twice. Curious you only now >>> hit it. >>> >> >> Yes, I just hit it a a week or two back and I needed to collect data to >> explain why p->on_rq got to 0. Hitting it requires extreme stress -- for me >> I needed a system with large threads and less memory running stress-ng. >> Reproducing the problem takes an unpredictable amount of time. > > What hardware do you see this on, is it shiny new Power8 chips which > have never before seen deep queues or something. Or is it 'regular' old > Power7 like stuff? > I don't think the issue is processor specific, it is probabilistic, but I've not tested on Power7. I am seeing it on a Power8 system Balbir Singh