Re: High priority tasks break SMP balancer?

[Micah Dowty <micah@vmware.com>]

On Sat, Nov 17, 2007 at 12:26:41AM +0100, Dmitry Adamushko wrote:
> Let's say we change a pattern for the niced task: e.g. run for 100 ms.
> and then sleep for 300 ms. (that's ~25% of cpu load) in the loop. Any
> behavioral changes?

For consistency, I tested this using /dev/rtc. I set the rtc frequency
to 16 Hz, and replaced the main loop of my high (-19) priority thread
with:

   while (1) {
      unsigned long data;

      for (i = 0; i < 3; i++) {
         if (read(rtc, &data, sizeof data) != sizeof data) {
            perror("read");
            return 1;
         }
      }

      fcntl(rtc, F_SETFL, O_NONBLOCK);
      while (read(rtc, &data, sizeof data) < 0);
      fcntl(rtc, F_SETFL, 0);
   }

Now it's busy-looping for 62ms, and sleeping for three consecutive
62.5ms chunks totalling 187.5ms.

The results aren't quite what I was expecting. I have only observed
this so far in test cases where I have a very high wakeup frequency,
so I wasn't expecting this to work. I did, however, still observe the
problem where occasionally I get into a state where one CPU is mostly
idle.

Qualitatively, this feels a bit different. With the higher clock
frequency it seemed like the CPU would easily get "stuck" in this
state where it's mostly idle, and it would stay there for a long
time. With the low wakeup frequency, I'm seeing it toggle between the
busy and mostly-idle states more quickly.

I tried a similar test using usleep() and gettimeofday() rather than
/dev/rtc:

   while (1) {
      usleep(300000);

      gettimeofday(&t1, NULL);
      do {
         gettimeofday(&t2, NULL);
      } while (t2.tv_usec - t1.tv_usec +
               (t2.tv_sec - t1.tv_sec) * 1000000 < 100000);
   }

With this test program, I haven't yet seen a CPU imbalance that lasts
longer than a fraction of a second.

--Micah