kdump hangs up by Sysrq+C trigger in high load.

kdump hangs up by Sysrq+C trigger in high load.

[Yoshihiro Takahashi]

Hi.

kdump hangs up by Sysrq+C trigger once in about 10 times in high load.
After the above occurs, kdump cannot collect vmcores with NMI button.
When waiting_for_crash_ipi does case more than 0, mdelay seem to make 
a stall.

While nmi_shootdown_cpus(), mdelay may not work since interruption is
disable.  (ex. when crashing kernel by NMI interruption, all
interruption may be disabled.).

So I think that it should use other way (TSC) instead of mdelay() in
nmi_shootdown_cpus().

Here is the log.
--
SysRq : Trigger a crashdump
BUG: warning at arch/x86/kernel/crash.c:107/nmi_shootdown_cpus() (Not tainted)


    Signed-off-by: Yoshihiro Takahashi <ytakahashi@miraclelinux.com>
---
 crash.c |   26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 2685538..fc63636 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -89,7 +89,8 @@ static struct notifier_block crash_nmi_nb = {
 
 static void nmi_shootdown_cpus(void)
 {
-	unsigned long msecs;
+	int i, tsc_wait, tsc_count;
+	unsigned long tsc_start, tsc_now, tsc_delay;
 
 	atomic_set(&waiting_for_crash_ipi, num_online_cpus() - 1);
 	/* Would it be better to replace the trap vector here? */
@@ -102,10 +103,25 @@ static void nmi_shootdown_cpus(void)
 
 	smp_send_nmi_allbutself();
 
-	msecs = 1000; /* Wait at most a second for the other cpus to stop */
-	while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
-		mdelay(1);
-		msecs--;
+	tsc_delay = HZ; /* Wait at most a second for the other cpus to stop */
+	while ((atomic_read(&waiting_for_crash_ipi) > 0) && tsc_delay) {
+		tsc_wait = 1000000; /* msec */
+		tsc_count = 16384;
+		while (tsc_wait > 0) {
+			if (tsc_count-- < 0) {
+				break;
+			}
+			tsc_start = get_cycles();
+			for (i = 0; i < 100000; i++) {
+				;
+			}
+			tsc_now = get_cycles();
+			tsc_wait -= tsc_now - tsc_start;
+		}
+		if (tsc_count < 0) {
+			break;
+		}
+		tsc_delay--;
 	}
 
 	/* Leave the nmi callback set */

Re: kdump hangs up by Sysrq+C trigger in high load.

[Ingo Molnar]

* Yoshihiro Takahashi <ytakahashi@miraclelinux.com> wrote:

> Hi.
> 
> kdump hangs up by Sysrq+C trigger once in about 10 times in high load. 
> After the above occurs, kdump cannot collect vmcores with NMI button. 
> When waiting_for_crash_ipi does case more than 0, mdelay seem to make a 
> stall.
> 
> While nmi_shootdown_cpus(), mdelay may not work since interruption is 
> disable.  (ex. when crashing kernel by NMI interruption, all 
> interruption may be disabled.).
> 
> So I think that it should use other way (TSC) instead of mdelay() in 
> nmi_shootdown_cpus().

mdelay uses either the tsc or CPU loops - neither stop while IRQs are 
disabled.

Could you figure out exactly why it hangs in your case?

	Ingo

Re: kdump hangs up by Sysrq+C trigger in high load.

[Yoshihiro Takahashi]

Ingo. Thank you, your reply.

I will serch it again.

On Tue, 16 Dec 2008 12:09:49 +0100
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Yoshihiro Takahashi <ytakahashi@miraclelinux.com> wrote:
> 
> > Hi.
> > 
> > kdump hangs up by Sysrq+C trigger once in about 10 times in high load. 
> > After the above occurs, kdump cannot collect vmcores with NMI button. 
> > When waiting_for_crash_ipi does case more than 0, mdelay seem to make a 
> > stall.
> > 
> > While nmi_shootdown_cpus(), mdelay may not work since interruption is 
> > disable.  (ex. when crashing kernel by NMI interruption, all 
> > interruption may be disabled.).
> > 
> > So I think that it should use other way (TSC) instead of mdelay() in 
> > nmi_shootdown_cpus().
> 
> mdelay uses either the tsc or CPU loops - neither stop while IRQs are 
> disabled.
> 
> Could you figure out exactly why it hangs in your case?
> 
> 	Ingo