Re: Kernel hangs in SMP + VMware environment.

Re: Kernel hangs in SMP + VMware environment.

[Roland]

maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834  ?

you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
does that happen only on a dedicated vmware box, or on different ones?
vmware-tools active? ->stop -> different ?

could you provide some more information about your hardware/vmware 
environment ?
does that happen on esx or on hosted products (workstation, server, 
player..) ?

regards
roland




List:       linux-kernel
Subject:    Kernel hangs in SMP + VMware environment.
From:       Tetsuo Handa <penguin-kernel () I-love ! SAKURA ! ne ! jp>
Date:       2008-05-12 21:41:34
Message-ID: 200805130641.CDG56299.QStFOLVOOJFHMF () I-love ! SAKURA ! ne ! 
jp
[Download message RAW]

I'm experiencing hang up problem with recent kernels in VMware environment.

Here are two examples.

http://I-love.SAKURA.ne.jp/tmp/messages.1 (203kB)
http://I-love.SAKURA.ne.jp/tmp/messages.2 (1.7MB)

The messages.1 is a log when "tar" stopped processing
while extracting a .tar.bz2 file by
"rpmbuild -bb --target i586 --with baseonly kernel.spec".
I got this log in runlevel 3 of Fedora 8.

The messages.2 is a log when the compiler processes (e.g. "cc1")
seem to be hanged up (no compiler messages appear for minutes,
which unlikely happen).
I got this log in runlevel 1 of Fedora 8
by "rpmbuild -bb --target i586 --with baseonly kernel.spec"
after starting rsyslog and stopping anacron.

I experience this problem in many distro (e.g. Fedora, Ubuntu, SuSE),
which use kenel (I think) around 2.6.21 and later.

I experience this problem only in VMware.
I have never experienced this problem in native environment.

I experience this problem only when I assign 2 CPUs to VMware.
I have never experienced this problem with 1 CPU.

May be something scheduler related in SMP + VMware environment.

The a.out process in the log files are http://lkml.org/lkml/2008/5/12/130
What other information should I dump for identifying the location of hang 
up?

Regards.

Re: Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

Hello.

Roland wrote:
> maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834  ?
Thank you for URL.
My bug seems to be timer related.

> you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
I don't know exact version, but I don't experience this problem
in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).

> does that happen only on a dedicated vmware box, or on different ones?
> vmware-tools active? ->stop -> different ?
vmware-tools is not installed for my Fedora 8.

> could you provide some more information about your hardware/vmware 
> environment ?
> does that happen on esx or on hosted products (workstation, server, 
> player..) ?

Hardware: ThinkPad X60 (Intel Core 2 Duo, 2048MB RAM, No swap partition)
VMware host environment: CentOS 5.1 (x86_64)
VMware version: VMware Workstation 6.0.2 (x86_64)
VMware guest environment: many distro using recent kernels (all i386)

I don't have ESX server environment.



Today, I tried to reproduce this problem using 2.6.24.5-85.fc8 kernel and
I got 2 patterns.


http://I-love.SAKURA.ne.jp/tmp/hangup-3.png      (10kB)
http://I-love.SAKURA.ne.jp/tmp/messages-3.txt   (174kB)

hangup-3.png is the screenshot of hang up and messages-3.txt is the sysrq logs.
Funny thing is that "tar" process sleeps for minutes at blk_remove_plug()
(while "tar" finishes within a minute if 1 CPU).


http://I-love.SAKURA.ne.jp/tmp/dmesg-4.txt      (120kB)

dmesg-4.txt is a partial output of "dmesg".
Since rsyslog sometimes cannot save logs to /var/log/messages by some reason,
I tried to directly save from /proc/kmsg using "a.out",
but "a.out" couldn't save logs neither.
Funny thing is that "a.out" process sleeps for minutes at getnstimeofday().
The source code of "a.out" is

  #include <stdio.h>
  #include <unistd.h>
  #include <time.h>
  #include <sys/types.h>
  #include <sys/stat.h>
  #include <fcntl.h>
  #include <string.h>
  
  int main(int argc, char *argv[]) {
  	FILE *fp = fopen("/proc/sys/kernel/sysrq", "w");
  	if (!fp) return 1;
  	fprintf(fp, "1\n");
  	fclose(fp);
  	fp = fopen("/proc/sysrq-trigger", "w");
  	if (!fp) return 1;
  	if (fork() == 0) {
  		int fd_r = open("/proc/kmsg", O_RDONLY);
  		int fd_w = open("/root/messages", O_WRONLY | O_TRUNC | O_CREAT, 0600);
  		char buffer[4096];
  		char timebuf[80];
  		memset(timebuf, 0, sizeof(timebuf));
  		memset(buffer, 0, sizeof(buffer));
  		while (1) {
  			const int len = read(fd_r, buffer, sizeof(buffer));
  			static time_t prev = 0;
  			const time_t now = time(NULL);
  			if (now != prev) {
  				static int counter = 0;
  				prev = now;
  				snprintf(timebuf, sizeof(timebuf) - 1, "****************************** %d %lu ******************************\n", counter++, now);
  				write(fd_w, timebuf, strlen(timebuf));
  			}
  			write(fd_w, buffer, len);
  		}
  		_exit(0);
  	}
  	while (1) {
  		sleep(60);
  		fprintf(fp, "t\n");
  		fflush(fp);
  	}
  	return 0;
  }


Other patterns that hanged up are

(1) # time sync

(2) # top

(3) # dmesg > ~/log

Regards.

Re: Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

Tetsuo Handa wrote:
> I'm saying "All kernels since 2.6.18 might have this problem, and we need to apply
> your patch to all kernels since 2.6.18 if they actually have this problem".

Today, I found that the CentOS 5.2's 2.6.18-92.1.6.el5 kernel has the patch applied.
So, this problem existed in all kernels since 2.6.18, right?

Does this problem happen when used in non-virtualized (i.e. native) environment?

Regards.

RE: Kernel hangs in SMP + VMware environment.

[Alok Kataria]

________________________________________
From: jengelh@sovereign.computergmbh.de [jengelh@sovereign.computergmbh.de] On Behalf Of Jan Engelhardt [jengelh@medozas.de]
Sent: Sunday, May 18, 2008 12:45 PM
To: Alok Kataria
Cc: penguin-kernel@i-love.sakura.ne.jp; devzero@web.de; linux-kernel@vger.kernel.org; Daniel Hecht
Subject: Re: Kernel hangs in SMP + VMware environment.

On Wednesday 2008-05-14 20:30, Alok Kataria wrote:
>On Wed, May 14, 2008 at 4:00 AM, Tetsuo Handa
><penguin-kernel@i-love.sakura.ne.jp> wrote:
>> Hello.
>>
>> Roland wrote:
>>> maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834  ?
>> Thank you for URL.
>> My bug seems to be timer related.

I too noticed it; clocksource=pit is my current workaround.

ANK> What kernel do you see this with ? Did you get a a chance to try the patch
ANK> which I sent earlier on this thread ?


>>> you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
>
>> I don't know exact version, but I don't experience this problem
>> in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).

It does not happen for 2.6.23, and IIRC it did not with 2.6.24
either; 2.6.25-rcish is the first to show this behavior.

ANK> Are you sure you didn't see it with 2.6.23/2.6.24 ?
ANK> i am assuming your test case is also similar to that of Tetsuo.
ANK> Multiple guests running simoultaneously.

Oh well, time to bisect.

ANK> Let me know if you get anything their.


Thanks,
Alok

Re: Kernel hangs in SMP + VMware environment.

[Jan Engelhardt]

On Wednesday 2008-05-14 20:30, Alok Kataria wrote:
>On Wed, May 14, 2008 at 4:00 AM, Tetsuo Handa
><penguin-kernel@i-love.sakura.ne.jp> wrote:
>> Hello.
>>
>> Roland wrote:
>>> maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834  ?
>> Thank you for URL.
>> My bug seems to be timer related.

I too noticed it; clocksource=pit is my current workaround.

>>> you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
>
>> I don't know exact version, but I don't experience this problem
>> in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).

It does not happen for 2.6.23, and IIRC it did not with 2.6.24
either; 2.6.25-rcish is the first to show this behavior.
Oh well, time to bisect.

Re: Kernel hangs in SMP + VMware environment.

[Bart Van Assche]

On Sat, May 17, 2008 at 7:59 AM, Alok kataria <alokkataria1@gmail.com> wrote:
> Sorry, but I am not aware of how the stable tree is maintained. Any
> information regarding that would be helpful.

See also Documentation/stable_kernel_rules.txt.

Bart.

Re: Kernel hangs in SMP + VMware environment.

[Alok kataria]

On Fri, May 16, 2008 at 6:34 PM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> Alok Kataria wrote:
>> > I'm not sure, but this problem might exist in all kernels since 2.6.18 , for
>> > I can find clocksource= parameter in Documentation/kernel-parameters.txt .
>> >
>> Sorry, but I dont understand what you are pointing to, the 2.6.18 kernel does
>> have clocksource parameter.
>> And you are correct this could be a problem with all the kernels which are
>> using clocksource.
>>
>> If you are able to reproduce this with any of the kernels with the patch applied
>> Please let me know.
> I'm not saying "I was able to reproduce this problem after applying your patch".
> I'm saying "All kernels since 2.6.18 might have this problem, and we need to apply
> your patch to all kernels since 2.6.18 if they actually have this problem".
>> >

Thats correct, but I am not sure how the stable folks will pickup these patches.
Thomas, now if we need the patch in any previous  kernels stable tree
(in this instance 2.6.18.x), do we need to backport this patch for
each kernel (which shows this problem)  and send these patches to the
stable tree maintainers or is their some other way  ?
Sorry, but I am not aware of how the stable tree is maintained. Any
information regarding that would be helpful.

Thanks,
Alok

>> > The Ctrl-C didn't work, the Alt-F? didn't work, the Ctrl-Alt-Del didn't work.
>> > I had no time to reproduce it, so I'm not sure it has actually hanged up.
>> >
>
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

Alok Kataria wrote:
> > I'm not sure, but this problem might exist in all kernels since 2.6.18 , for
> > I can find clocksource= parameter in Documentation/kernel-parameters.txt .
> >
> Sorry, but I dont understand what you are pointing to, the 2.6.18 kernel does
> have clocksource parameter.
> And you are correct this could be a problem with all the kernels which are
> using clocksource.
>
> If you are able to reproduce this with any of the kernels with the patch applied
> Please let me know.
I'm not saying "I was able to reproduce this problem after applying your patch".
I'm saying "All kernels since 2.6.18 might have this problem, and we need to apply
your patch to all kernels since 2.6.18 if they actually have this problem".
> > 
> > The Ctrl-C didn't work, the Alt-F? didn't work, the Ctrl-Alt-Del didn't work.
> > I had no time to reproduce it, so I'm not sure it has actually hanged up.
> > 

Thanks.

RE: Kernel hangs in SMP + VMware environment.

[Alok Kataria]

________________________________________
From: Tetsuo Handa [penguin-kernel@I-love.SAKURA.ne.jp]
Sent: Friday, May 16, 2008 5:48 AM
To: Alok Kataria
Cc: devzero@web.de; linux-kernel@vger.kernel.org; Daniel Hecht
Subject: Re: Kernel hangs in SMP + VMware environment.

Hello.

Alok Kataria wrote:
> Actually it would be better to use clocksource=acpi_pm on kernels which don't
> have this patch, as ACPI_PM comes with many more featuers like hrtimers,
> nohz, etc.
>
> On the other hand for kernels which have this patch attached,  it would be
> best to use TSC, for performance.
I see. Thanks.



I'm not sure, but this problem might exist in all kernels since 2.6.18 , for
I can find clocksource= parameter in Documentation/kernel-parameters.txt .

ANK> Sorry, but I dont understand what you are pointing to, the 2.6.18 kernel does
ANK> have clocksource parameter.
ANK> And you are correct this could be a problem with all the kernels which are
ANK> using clocksource.

I didn't experience this problem in earlier kernels (e.g. 2.6.20)
since I was running only 1 VMware guest at a time.

Today I ran 2 VMware guests simultaneously (one with kernel 2.6.22 and
the other with kernel 2.6.18, both are assigned 2 virtual CPUs) and I encountered
may-be-hanged-up with the 2.6.18 one.


ANK> If you are able to reproduce this with any of the kernels with the patch applied
ANK> Please let me know.

Thanks,
Alok

The Ctrl-C didn't work, the Alt-F? didn't work, the Ctrl-Alt-Del didn't work.
I had no time to reproduce it, so I'm not sure it has actually hanged up.

Regards.

Re: Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

Hello.

Alok Kataria wrote:
> Actually it would be better to use clocksource=acpi_pm on kernels which don't
> have this patch, as ACPI_PM comes with many more featuers like hrtimers,
> nohz, etc.
> 
> On the other hand for kernels which have this patch attached,  it would be
> best to use TSC, for performance.
I see. Thanks.



I'm not sure, but this problem might exist in all kernels since 2.6.18 , for
I can find clocksource= parameter in Documentation/kernel-parameters.txt .

I didn't experience this problem in earlier kernels (e.g. 2.6.20)
since I was running only 1 VMware guest at a time.
Today I ran 2 VMware guests simultaneously (one with kernel 2.6.22 and
the other with kernel 2.6.18, both are assigned 2 virtual CPUs) and I encountered
may-be-hanged-up with the 2.6.18 one.
The Ctrl-C didn't work, the Alt-F? didn't work, the Ctrl-Alt-Del didn't work.
I had no time to reproduce it, so I'm not sure it has actually hanged up.

Regards.

RE: Kernel hangs in SMP + VMware environment.

[Alok Kataria]

Hi Tetsuo,

Thanks for testing.

Actually it would be better to use clocksource=acpi_pm on kernels which don't have this patch, as ACPI_PM comes with many more featuers like hrtimers, nohz, etc.

On the other hand for kernels which have this patch attached,  it would be best to use TSC, for performance.

HTH,
Alok.

________________________________________
From: Tetsuo Handa [penguin-kernel@i-love.sakura.ne.jp]
Sent: Wednesday, May 14, 2008 11:11 PM
To: Alok Kataria
Cc: devzero@web.de; linux-kernel@vger.kernel.org; Daniel Hecht
Subject: Re: Kernel hangs in SMP + VMware environment.

Hello.

Alok Kataria wrote:
> Can you try the patch attached with this mail, I made this on top of
> 2.6.24.7 but should fit on any other 2.6.24 based distro kernel.

I tried 2.6.24.5-85.fc8 with your patch, and I can no longer reproduce
this problem. I think the problem has been solved.


Also, workaround till your patch is applied to distro kernels seems to be
to add "clocksource=jiffies" or something to avoid using tsc as clocksource.
Fedora 9's 2.6.25-14.fc9 also hangs since the default clocksource is tsc, but
with "clocksource=jiffies" added, I encounter no hangs so far.



>From today, I can compile kernels using 2 CPUs.
Thank you very much.

Re: Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

Hello.

Alok Kataria wrote:
> Can you try the patch attached with this mail, I made this on top of
> 2.6.24.7 but should fit on any other 2.6.24 based distro kernel.

I tried 2.6.24.5-85.fc8 with your patch, and I can no longer reproduce
this problem. I think the problem has been solved.

Also, workaround till your patch is applied to distro kernels seems to be
to add "clocksource=jiffies" or something to avoid using tsc as clocksource.
Fedora 9's 2.6.25-14.fc9 also hangs since the default clocksource is tsc, but
with "clocksource=jiffies" added, I encounter no hangs so far.

>From today, I can compile kernels using 2 CPUs.
Thank you very much.

Re: Kernel hangs in SMP + VMware environment.

[Alok Kataria]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On Wed, May 14, 2008 at 4:00 AM, Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
> Hello.
>
> Roland wrote:
>> maybe related to http://bugzilla.kernel.org/show_bug.cgi?id=9834  ?
> Thank you for URL.
> My bug seems to be timer related.
>
>> you say "recent" , so this does happen from 2.6.21 to 2.6.26rc2 ?
> I don't know exact version, but I don't experience this problem
> in earlier kernels (e.g. Fedora Core 5 which uses kernel 2.6.20).
>
>> does that happen only on a dedicated vmware box, or on different
ones?
>> vmware-tools active? ->stop -> different ?
> vmware-tools is not installed for my Fedora 8.
>
>> could you provide some more information about your hardware/vmware
>> environment ?
>> does that happen on esx or on hosted products (workstation, server,
>> player..) ?
>
> Hardware: ThinkPad X60 (Intel Core 2 Duo, 2048MB RAM, No swap
partition)
> VMware host environment: CentOS 5.1 (x86_64)
> VMware version: VMware Workstation 6.0.2 (x86_64)
> VMware guest environment: many distro using recent kernels (all i386)

Hi Tetsuo,

Can you try the patch attached with this mail, I made this on top of
2.6.24.7 but should fit on any other 2.6.24 based distro kernel.

If the attached patch still gives you the same problem, please send me
your config file and the boot time dmesg's.

Thanks,
Alok

[-- Attachment #2: prevent-tsc-from-going-back.patch --]
[-- Type: text/x-patch, Size: 5958 bytes --]

This is a backport of the patch submitted by Thomas Gleixner to the
x86 git tree, commit d8bb6f4c1670c8324e4135c61ef07486f7f17379 

Comments from the original post :

We already catch most of the TSC problems by sanity checks, but there
is a subtle bug which has been in the code for ever. This can cause
time jumps in the range of hours.

This was reported in:
     http://lkml.org/lkml/2007/8/23/96
and
     http://lkml.org/lkml/2008/3/31/23

I was able to reproduce the problem with a gettimeofday loop test on a
dual core and a quad core machine which both have sychronized
TSCs. The TSCs seems not to be perfectly in sync though, but the
kernel is not able to detect the slight delta in the sync check. Still
there exists an extremly small window where this delta can be observed
with a real big time jump. So far I was only able to reproduce this
with the vsyscall gettimeofday implementation, but in theory this
might be observable with the syscall based version as well.

CPU 0 updates the clock source variables under xtime/vyscall lock and
CPU1, where the TSC is slighty behind CPU0, is reading the time right
after the seqlock was unlocked.

The clocksource reference data was updated with the TSC from CPU0 and
the value which is read from TSC on CPU1 is less than the reference
data. This results in a huge delta value due to the unsigned
subtraction of the TSC value and the reference value. This algorithm
can not be changed due to the support of wrapping clock sources like
pm timer.

The huge delta is converted to nanoseconds and added to xtime, which
is then observable by the caller. The next gettimeofday call on CPU1
will show the correct time again as now the TSC has advanced above the
reference value.

To prevent this TSC specific wreckage we need to compare the TSC value
against the reference value and return the latter when it is larger
than the actual TSC value.

I pondered to mark the TSC unstable when the readout is smaller than
the reference value, but this would render an otherwise good and fast
clocksource unusable without a real good reason.

Signed-off-by: Alok Kataria <akataria@vmware.com>
CC: Thomas Gleixner <tglx@linutronix.de>

Index: linux-2.6.24.7/arch/x86/kernel/tsc_32.c
===================================================================
--- linux-2.6.24.7.orig/arch/x86/kernel/tsc_32.c	2008-05-14 11:25:20.000000000 -0700
+++ linux-2.6.24.7/arch/x86/kernel/tsc_32.c	2008-05-14 11:26:21.000000000 -0700
@@ -268,14 +268,27 @@
 /* clock source code */
 
 static unsigned long current_tsc_khz = 0;
+static struct clocksource clocksource_tsc;
 
+/*
+ * We compare the TSC to the cycle_last value in the clocksource
+ * structure to avoid a nasty time-warp issue. This can be observed in
+ * a very small window right after one CPU updated cycle_last under
+ * xtime lock and the other CPU reads a TSC value which is smaller
+ * than the cycle_last reference value due to a TSC which is slighty
+ * behind. This delta is nowhere else observable, but in that case it
+ * results in a forward time jump in the range of hours due to the
+ * unsigned delta calculation of the time keeping core code, which is
+ * necessary to support wrapping clocksources like pm timer.
+ */
 static cycle_t read_tsc(void)
 {
 	cycle_t ret;
 
 	rdtscll(ret);
 
-	return ret;
+	return ret >= clocksource_tsc.cycle_last ?
+		ret : clocksource_tsc.cycle_last;
 }
 
 static struct clocksource clocksource_tsc = {
Index: linux-2.6.24.7/arch/x86/kernel/tsc_64.c
===================================================================
--- linux-2.6.24.7.orig/arch/x86/kernel/tsc_64.c	2008-05-14 11:25:20.000000000 -0700
+++ linux-2.6.24.7/arch/x86/kernel/tsc_64.c	2008-05-14 11:26:21.000000000 -0700
@@ -10,6 +10,7 @@
 
 #include <asm/hpet.h>
 #include <asm/timex.h>
+#include <asm/vgtod.h>
 
 static int notsc __initdata = 0;
 
@@ -246,18 +247,34 @@
 
 __setup("notsc", notsc_setup);
 
+static struct clocksource clocksource_tsc;
 
-/* clock source code: */
+/*
+ * We compare the TSC to the cycle_last value in the clocksource
+ * structure to avoid a nasty time-warp. This can be observed in a
+ * very small window right after one CPU updated cycle_last under
+ * xtime/vsyscall_gtod lock and the other CPU reads a TSC value which
+ * is smaller than the cycle_last reference value due to a TSC which
+ * is slighty behind. This delta is nowhere else observable, but in
+ * that case it results in a forward time jump in the range of hours
+ * due to the unsigned delta calculation of the time keeping core
+ * code, which is necessary to support wrapping clocksources like pm
+ * timer.
+ */
 static cycle_t read_tsc(void)
 {
 	cycle_t ret = (cycle_t)get_cycles_sync();
-	return ret;
+
+	return ret >= clocksource_tsc.cycle_last ?
+		ret : clocksource_tsc.cycle_last;
 }
 
 static cycle_t __vsyscall_fn vread_tsc(void)
 {
 	cycle_t ret = (cycle_t)get_cycles_sync();
-	return ret;
+
+	return ret >= __vsyscall_gtod_data.clock.cycle_last ?
+		ret : __vsyscall_gtod_data.clock.cycle_last;
 }
 
 static struct clocksource clocksource_tsc = {
Index: linux-2.6.24.7/kernel/time/timekeeping.c
===================================================================
--- linux-2.6.24.7.orig/kernel/time/timekeeping.c	2008-05-14 11:25:20.000000000 -0700
+++ linux-2.6.24.7/kernel/time/timekeeping.c	2008-05-14 11:26:21.000000000 -0700
@@ -189,6 +189,7 @@
 	if (clock == new)
 		return;
 
+	new->cycle_last = 0;
 	now = clocksource_read(new);
 	nsec =  __get_nsec_offset();
 	timespec_add_ns(&xtime, nsec);
@@ -301,6 +302,7 @@
 	/* Make sure that we have the correct xtime reference */
 	timespec_add_ns(&xtime, timekeeping_suspend_nsecs);
 	/* re-base the last cycle value */
+	clock->cycle_last = 0;
 	clock->cycle_last = clocksource_read(clock);
 	clock->error = 0;
 	timekeeping_suspended = 0;

Kernel hangs in SMP + VMware environment.

[Tetsuo Handa]

I'm experiencing hang up problem with recent kernels in VMware environment.

Here are two examples.

http://I-love.SAKURA.ne.jp/tmp/messages.1 (203kB)
http://I-love.SAKURA.ne.jp/tmp/messages.2 (1.7MB)

The messages.1 is a log when "tar" stopped processing
while extracting a .tar.bz2 file by
"rpmbuild -bb --target i586 --with baseonly kernel.spec".
I got this log in runlevel 3 of Fedora 8.

The messages.2 is a log when the compiler processes (e.g. "cc1")
seem to be hanged up (no compiler messages appear for minutes,
which unlikely happen).
I got this log in runlevel 1 of Fedora 8
by "rpmbuild -bb --target i586 --with baseonly kernel.spec"
after starting rsyslog and stopping anacron.

I experience this problem in many distro (e.g. Fedora, Ubuntu, SuSE),
which use kenel (I think) around 2.6.21 and later.

I experience this problem only in VMware.
I have never experienced this problem in native environment.

I experience this problem only when I assign 2 CPUs to VMware.
I have never experienced this problem with 1 CPU.

May be something scheduler related in SMP + VMware environment.

The a.out process in the log files are http://lkml.org/lkml/2008/5/12/130
What other information should I dump for identifying the location of hang up?

Regards.