From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932427AbdJJS7Z (ORCPT ); Tue, 10 Oct 2017 14:59:25 -0400 Received: from mga06.intel.com ([134.134.136.31]:53252 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755836AbdJJS7X (ORCPT ); Tue, 10 Oct 2017 14:59:23 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.43,359,1503385200"; d="scan'208";a="144956459" Subject: Re: [PATCH 1/2] IB/hfi1: Use preempt_{dis,en}able_nort() To: Arnaldo Carvalho de Melo , bigeasy@linutronix.de Cc: linux-rt-users@vger.kernel.org, linux-kernel@vger.kernel.org, Arnaldo Carvalho de Melo , Clark Williams , Dean Luick , Doug Ledford , Kaike Wan , Leon Romanovsky , linux-rdma@vger.kernel.org, Peter Zijlstra , Sebastian Andrzej Siewior , Sebastian Sanchez , Steven Rostedt , Thomas Gleixner References: <20171003154920.31566-1-acme@kernel.org> <20171003154920.31566-2-acme@kernel.org> From: Dennis Dalessandro Message-ID: <1d06a3da-426f-c887-1da7-64b760c53425@intel.com> Date: Tue, 10 Oct 2017 14:59:18 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20171003154920.31566-2-acme@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/3/2017 11:49 AM, Arnaldo Carvalho de Melo wrote: > From: Arnaldo Carvalho de Melo > > sc_buffer_alloc() disables preemption that will be reenabled by either > pio_copy() or seg_pio_copy_end(). But before disabling preemption it > grabs a spin lock that will be dropped after it disables preemption, > which ends up triggering a warning in migrate_disable() later on. > > spin_lock_irqsave(&sc->alloc_lock) > migrate_disable() ++p->migrate_disable -> 2 > preempt_disable() > spin_unlock_irqrestore(&sc->alloc_lock) > migrate_enable() in_atomic(), so just returns, migrate_disable stays at 2 > spin_lock_irqsave(some other lock) -> b00m > > And the WARN_ON code ends up tripping over this over and over in > log_store(). > > Sequence captured via ftrace_dump_on_oops + crash utility 'dmesg' > command. > > [512258.613862] sm-3297 16 .....11 359465349134644: sc_buffer_alloc <-hfi1_verbs_send_pio > [512258.613876] sm-3297 16 .....11 359465349134719: migrate_disable <-sc_buffer_alloc > [512258.613890] sm-3297 16 .....12 359465349134798: rt_spin_lock <-sc_buffer_alloc > [512258.613903] sm-3297 16 ....112 359465349135481: rt_spin_unlock <-sc_buffer_alloc > [512258.613916] sm-3297 16 ....112 359465349135556: migrate_enable <-sc_buffer_alloc > [512258.613935] sm-3297 16 ....112 359465349135788: seg_pio_copy_start <-hfi1_verbs_send_pio > [512258.613954] sm-3297 16 ....112 359465349136273: update_sge <-hfi1_verbs_send_pio > [512258.613981] sm-3297 16 ....112 359465349136373: seg_pio_copy_mid <-hfi1_verbs_send_pio > [512258.613999] sm-3297 16 ....112 359465349136873: update_sge <-hfi1_verbs_send_pio > [512258.614017] sm-3297 16 ....112 359465349136956: seg_pio_copy_mid <-hfi1_verbs_send_pio > [512258.614035] sm-3297 16 ....112 359465349137221: seg_pio_copy_end <-hfi1_verbs_send_pio > [512258.614048] sm-3297 16 .....12 359465349137360: migrate_disable <-hfi1_verbs_send_pio > [512258.614065] sm-3297 16 .....12 359465349137476: warn_slowpath_null <-migrate_disable > [512258.614081] sm-3297 16 .....12 359465349137564: __warn <-warn_slowpath_null > [512258.614088] sm-3297 16 .....12 359465349137958: printk <-__warn > [512258.614096] sm-3297 16 .....12 359465349138055: vprintk_default <-printk > [512258.614104] sm-3297 16 .....12 359465349138144: vprintk_emit <-vprintk_default > [512258.614111] sm-3297 16 d....12 359465349138312: _raw_spin_lock <-vprintk_emit > [512258.614119] sm-3297 16 d...112 359465349138789: log_store <-vprintk_emit > [512258.614127] sm-3297 16 .....12 359465349139068: migrate_disable <-vprintk_emit > > According to a discussion (see Link: below) on the linux-rt-users > mailing list, this locking is done for performance reasons, not for > correctness, so use the _nort() variants to avoid the above problem. > > Suggested-by: Julia Cartwright > Cc: Clark Williams > Cc: Dean Luick > Cc: Dennis Dalessandro > Cc: Doug Ledford > Cc: Kaike Wan > Cc: Leon Romanovsky > Cc: linux-rdma@vger.kernel.org > Cc: Peter Zijlstra > Cc: Sebastian Andrzej Siewior > Cc: Sebastian Sanchez > Cc: Steven Rostedt > Cc: Thomas Gleixner > Link: http://lkml.kernel.org/r/20170926210045.GO29872@jcartwri.amer.corp.natinst.com > Signed-off-by: Arnaldo Carvalho de Melo I assume you are not asking for Doug to pick this up for linux-rdma and that this more of an RFC sort of deal and the intended destination is the -rt tree? Anyway, for this patch: Reviewed-by: Dennis Dalessandro