From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754692AbbIRWOO (ORCPT ); Fri, 18 Sep 2015 18:14:14 -0400 Received: from mail.linuxfoundation.org ([140.211.169.12]:44106 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752509AbbIRWON (ORCPT ); Fri, 18 Sep 2015 18:14:13 -0400 Date: Fri, 18 Sep 2015 15:14:11 -0700 From: Andrew Morton To: Jan Kara Cc: LKML , pmladek@suse.com, rostedt@goodmis.org, Gavin Hu , KY Srinivasan , Jan Kara Subject: Re: [PATCH 0/4] printk: Softlockup avoidance Message-Id: <20150918151411.a3fa65c3e4f33f9f2ddf1fd8@linux-foundation.org> In-Reply-To: <1439998711-7013-1-git-send-email-jack@suse.com> References: <1439998711-7013-1-git-send-email-jack@suse.com> X-Mailer: Sylpheed 3.4.1 (GTK+ 2.24.23; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 19 Aug 2015 17:38:27 +0200 Jan Kara wrote: > From: Jan Kara > > Hello, > > since lately there were several attempts at dealing with softlockups due > to heavy printk traffic [1] [2] and I've been also privately pinged by > couple of people about the state of the patch set, I've decided to respin > the patch set. > > To remind the original problem: > > Currently, console_unlock() prints messages from kernel printk buffer to > console while the buffer is non-empty. When serial console is attached, > printing is slow and thus other CPUs in the system have plenty of time > to append new messages to the buffer while one CPU is printing. Thus the > CPU can spend unbounded amount of time doing printing in console_unlock(). > This is especially serious when printk() gets called under some critical > spinlock or with interrupts disabled. > > In practice users have observed a CPU can spend tens of seconds printing > in console_unlock() (usually during boot when hundreds of SCSI devices > are discovered) resulting in RCU stalls (CPU doing printing doesn't > reach quiescent state for a long time), softlockup reports (IPIs for the > printing CPU don't get served and thus other CPUs are spinning waiting > for the printing CPU to process IPIs), and eventually a machine death > (as messages from stalls and lockups append to printk buffer faster than > we are able to print). So these machines are unable to boot with serial > console attached. Also during artificial stress testing SATA disk > disappears from the system because its interrupts aren't served for too > long. > > This series addresses the problem in the following way: If CPU has printed > more that printk_offload (defaults to 1000) characters, it wakes up one > of dedicated printk kthreads (we don't use workqueue because that has > deadlock potential if printk was called from workqueue code). Once we find > out kthread is spinning on a lock, we stop printing, drop console_sem, and > let kthread continue printing. Since there are two printing kthreads, they > will pass printing between them and thus no CPU gets hogged by printing. I still hate your patchset ;) But nothing better suggests itself. I have a few review comments - please let's work through that stuff, get a fresh version out and we'll see how it goes. Is this patchset being used in the field? Perhaps in the suse kernel? If so, a mention of that in the changelog would help things along.