From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933747AbcBYTyh (ORCPT ); Thu, 25 Feb 2016 14:54:37 -0500 Received: from smtprelay0128.hostedemail.com ([216.40.44.128]:52217 "EHLO smtprelay.hostedemail.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S933646AbcBYTyg (ORCPT ); Thu, 25 Feb 2016 14:54:36 -0500 X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Spam-Summary: 2,0,0,,d41d8cd98f00b204,rostedt@goodmis.org,:::::::::::::::::::::::,RULES_HIT:41:355:379:541:599:800:960:966:973:988:989:1260:1277:1311:1313:1314:1345:1359:1437:1515:1516:1518:1534:1543:1593:1594:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2553:2559:2562:2895:3138:3139:3140:3141:3142:3355:3622:3865:3867:3868:3870:3871:3872:3873:3874:4321:4385:4605:5007:6119:6261:6742:7875:7903:8660:9163:10004:10400:10848:10967:11026:11232:11473:11658:11914:12043:12295:12296:12438:12517:12519:12555:12740:13148:13230:13255:14096:14097:14659:21060:21080:21212:21324:30003:30012:30029:30054:30056:30064:30070:30075:30090:30091,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: balls91_ac14d5025106 X-Filterd-Recvd-Size: 4710 Date: Thu, 25 Feb 2016 14:54:32 -0500 From: Steven Rostedt To: "Shi, Yang" Cc: tj@kernel.org, jack@suse.cz, axboe@fb.com, fengguang.wu@intel.com, tglx@linutronix.de, bigeasy@linutronix.de, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-rt-users@vger.kernel.org, linaro-kernel@lists.linaro.org Subject: Re: [PATCH] writeback: call writeback tracepoints withoud holding list_lock in wb_writeback() Message-ID: <20160225145432.3749e5ec@gandalf.local.home> In-Reply-To: <56CF5848.7050806@linaro.org> References: <1456354043-31420-1-git-send-email-yang.shi@linaro.org> <20160224214042.71c3493b@grimm.local.home> <56CF5848.7050806@linaro.org> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 25 Feb 2016 11:38:48 -0800 "Shi, Yang" wrote: > On 2/24/2016 6:40 PM, Steven Rostedt wrote: > > On Wed, 24 Feb 2016 14:47:23 -0800 > > Yang Shi wrote: > > > >> commit 5634cc2aa9aebc77bc862992e7805469dcf83dac ("writeback: update writeback > >> tracepoints to report cgroup") made writeback tracepoints report cgroup > >> writeback, but it may trigger the below bug on -rt kernel due to the list_lock > >> held for the for loop in wb_writeback(). > > > > list_lock is a sleeping mutex, it's not disabling preemption. Moving it > > doesn't make a difference. > > > >> > >> BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:930 > >> in_atomic(): 1, irqs_disabled(): 0, pid: 625, name: kworker/u16:3 > > > > Something else disabled preemption. And note, nothing in the tracepoint > > should have called a sleeping function. > > Yes, it makes me confused too. It sounds like the preempt_ip address is > not that accurate. Yep, but the change you made doesn't look to be the fix. > > > > > > >> INFO: lockdep is turned off. > >> Preemption disabled at:[] wb_writeback+0xec/0x830 Can you disassemble the vmlinux file to see exactly where that call is. I use gdb to find the right locations. gdb> li *0xffffffc000374a5c gdb> disass 0xffffffc000374a5c > >> > >> CPU: 7 PID: 625 Comm: kworker/u16:3 Not tainted 4.4.1-rt5 #20 > >> Hardware name: Freescale Layerscape 2085a RDB Board (DT) > >> Workqueue: writeback wb_workfn (flush-7:0) > >> Call trace: > >> [] dump_backtrace+0x0/0x200 > >> [] show_stack+0x24/0x30 > >> [] dump_stack+0x88/0xa8 > >> [] ___might_sleep+0x2ec/0x300 > >> [] rt_spin_lock+0x38/0xb8 > >> [] kernfs_path_len+0x30/0x90 > >> [] trace_event_raw_event_writeback_work_class+0xe8/0x2e8 > > > > How accurate is this trace back? Here's the code that is executed in > > this tracepoint: > > > > TP_fast_assign( > > struct device *dev = bdi->dev; > > if (!dev) > > dev = default_backing_dev_info.dev; > > strncpy(__entry->name, dev_name(dev), 32); > > __entry->nr_pages = work->nr_pages; > > __entry->sb_dev = work->sb ? work->sb->s_dev : 0; > > __entry->sync_mode = work->sync_mode; > > __entry->for_kupdate = work->for_kupdate; > > __entry->range_cyclic = work->range_cyclic; > > __entry->for_background = work->for_background; > > __entry->reason = work->reason; > > ), > > > > See anything that would sleep? > > According to the stack backtrace, kernfs_path_len calls slepping lock, > which is called by __trace_wb_cgroup_size(wb) in __dynamic_array(char, > cgroup, __trace_wb_cgroup_size(wb)). > > The below is the definition: > > DECLARE_EVENT_CLASS(writeback_work_class, > TP_PROTO(struct bdi_writeback *wb, struct wb_writeback_work *work), > TP_ARGS(wb, work), > TP_STRUCT__entry( > __array(char, name, 32) > __field(long, nr_pages) > __field(dev_t, sb_dev) > __field(int, sync_mode) > __field(int, for_kupdate) > __field(int, range_cyclic) > __field(int, for_background) > __field(int, reason) > __dynamic_array(char, cgroup, __trace_wb_cgroup_size(wb)) > Ah, thanks for pointing that out. I missed that. -- Steve