From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756934AbcH3JKY (ORCPT ); Tue, 30 Aug 2016 05:10:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:36804 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751629AbcH3JKW (ORCPT ); Tue, 30 Aug 2016 05:10:22 -0400 Date: Tue, 30 Aug 2016 17:10:10 +0800 From: Dave Young To: Pratyush Anand Cc: mingo@kernel.org, alexandre.belloni@free-electrons.com, tglx@linutronix.de, hpa@zytor.com, x86@kernel.org, rtc-linux@googlegroups.com, linux-kernel@vger.kernel.org, prarit@redhat.com, dzickus@redhat.com, a.zummo@towertech.it Subject: Re: [PATCH V3 2/2] rtc/rtc-cmos: Initialize software counters before irq is registered Message-ID: <20160830091010.GA7632@dhcp-128-65.nay.redhat.com> References: <90426708da5934284839e6f23d23e265afcc779f.1470985727.git.panand@redhat.com> <20160830082230.GA7000@dhcp-128-65.nay.redhat.com> <20160830083814.GA7308@dhcp-128-65.nay.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160830083814.GA7308@dhcp-128-65.nay.redhat.com> User-Agent: Mutt/1.7.0 (2016-08-17) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 30 Aug 2016 09:10:21 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 08/30/16 at 04:38pm, Dave Young wrote: > On 08/30/16 at 04:22pm, Dave Young wrote: > > Hi, Pratyush > > > > On 08/16/16 at 08:55am, Pratyush Anand wrote: > > > We have observed on few x86 machines with rtc-cmos device that > > > hpet_rtc_interrupt() is called just after irq registration and before > > > cmos_do_probe() could call hpet_rtc_timer_init(). > > > > > > So, neither hpet_default_delta nor hpet_t1_cmp is initialized by the time > > > interrupt is raised in the given situation, and this results in NMI > > > watchdog LOCKUP. > > > > > > It has only been observed sporadically on kdump secondary kernels. > > > > > > See the call trace: > > > ---<-snip->--- > > > 27.913194] Kernel panic - not syncing: Watchdog detected hard LOCKUP on > > > cpu 0 > > > [ 27.915371] CPU: 0 PID: 1 Comm: swapper/0 Not tainted > > > 3.10.0-342.el7.x86_64 #1 > > > [ 27.917503] Hardware name: HP ProLiant DL160 Gen8, BIOS J03 02/10/2014 > > > [ 27.919455] ffffffff8186a728 0000000059c82488 ffff880034e05af0 > > > ffffffff81637bd4 > > > [ 27.921870] ffff880034e05b70 ffffffff8163144a 0000000000000010 > > > ffff880034e05b80 > > > [ 27.924257] ffff880034e05b20 0000000059c82488 0000000000000000 > > > 0000000000000000 > > > [ 27.926599] Call Trace: > > > [ 27.927352] [] dump_stack+0x19/0x1b > > > [ 27.929080] [] panic+0xd8/0x1e7 > > > [ 27.930588] [] ? restart_watchdog_hrtimer+0x50/0x50 > > > [ 27.932502] [] watchdog_overflow_callback+0xc2/0xd0 > > > [ 27.934427] [] __perf_event_overflow+0xa1/0x250 > > > [ 27.936232] [] perf_event_overflow+0x14/0x20 > > > [ 27.937957] [] intel_pmu_handle_irq+0x1e8/0x470 > > > [ 27.939799] [] perf_event_nmi_handler+0x2b/0x50 > > > [ 27.941649] [] nmi_handle.isra.0+0x69/0xb0 > > > [ 27.943348] [] do_nmi+0x169/0x340 > > > [ 27.944802] [] end_repeat_nmi+0x1e/0x2e > > > [ 27.946424] [] ? hpet_rtc_interrupt+0x85/0x380 > > > [ 27.948197] [] ? hpet_rtc_interrupt+0x85/0x380 > > > [ 27.949992] [] ? hpet_rtc_interrupt+0x85/0x380 > > > [ 27.951816] <> [] ? > > > run_timer_softirq+0x43/0x340 > > > [ 27.954114] [] handle_irq_event_percpu+0x3e/0x1e0 > > > [ 27.955962] [] handle_irq_event+0x3d/0x60 > > > [ 27.957635] [] handle_edge_irq+0x77/0x130 > > > [ 27.959332] [] handle_irq+0xbf/0x150 > > > [ 27.960949] [] do_IRQ+0x4f/0xf0 > > > [ 27.962434] [] common_interrupt+0x6d/0x6d > > > [ 27.964101] [] ? > > > _raw_spin_unlock_irqrestore+0x1b/0x40 > > > [ 27.966308] [] __setup_irq+0x2a7/0x570 > > > [ 28.067859] [] ? hpet_cpuhp_notify+0x140/0x140 > > > [ 28.069709] [] request_threaded_irq+0xcc/0x170 > > > [ 28.071585] [] cmos_do_probe+0x1e6/0x450 > > > [ 28.073240] [] ? cmos_do_probe+0x450/0x450 > > > [ 28.074911] [] cmos_pnp_probe+0xbb/0xc0 > > > [ 28.076533] [] pnp_device_probe+0x65/0xd0 > > > [ 28.078198] [] driver_probe_device+0x87/0x390 > > > [ 28.079971] [] __driver_attach+0x93/0xa0 > > > [ 28.081660] [] ? __device_attach+0x40/0x40 > > > [ 28.083662] [] bus_for_each_dev+0x73/0xc0 > > > [ 28.085370] [] driver_attach+0x1e/0x20 > > > [ 28.086974] [] bus_add_driver+0x200/0x2d0 > > > [ 28.088634] [] ? rtc_sysfs_init+0xe/0xe > > > [ 28.090349] [] driver_register+0x64/0xf0 > > > [ 28.091989] [] pnp_register_driver+0x20/0x30 > > > [ 28.093707] [] cmos_init+0x11/0x71 > > > ---<-snip->--- > > > > > > The previous patch split hpet_rtc_timer_init into > > > hpet_rtc_timer_counter_init() and hpet_rtc_timer_enable(). > > > > > > Therefore, this patch moved hpet_rtc_timer_counter_init() before IRQ > > > registration, so that we can gracefully handle such spurious interrupts. > > > > > > We were able to reproduce the problem in maximum 15 trials of kdump > > > secondary kernel boot on an hp-dl160gen8 machine without this patch. > > > However, more than 35 trials went fine after applying this patch. > > > > > > Signed-off-by: Pratyush Anand > > > [dzickus@redhat.com: edited the patch's summary] > > > Signed-off-by: Don Zickus > > > --- > > > drivers/rtc/rtc-cmos.c | 13 ++++++++++++- > > > 1 file changed, 12 insertions(+), 1 deletion(-) > > > > > > diff --git a/drivers/rtc/rtc-cmos.c b/drivers/rtc/rtc-cmos.c > > > index 43745cac0141..089d987f2638 100644 > > > --- a/drivers/rtc/rtc-cmos.c > > > +++ b/drivers/rtc/rtc-cmos.c > > > @@ -129,6 +129,16 @@ static inline int hpet_rtc_dropped_irq(void) > > > return 0; > > > } > > > > > > +static inline int hpet_rtc_timer_counter_init(void) > > > +{ > > > + return 0; > > > +} > > > + > > > +static inline int hpet_rtc_timer_enable(void) > > > +{ > > > + return 0; > > > +} > > > + > > > > Can these dummy functions go to /usr/include/linux/hpet.h alont with > > the #ifdef etc. > > Hmm, seems CONFIG_HPET_EMULATE_RTC is x86 only, so maybe go to > asm/hpet.h should be better.. Oops, asm/hpet will not work since rtc-cmos is also used in other arches., please ignore the comment. Thanks Dave