Hi all,

On one of machines [SUPERMICRO], after installing fresh kernel
(v4.0-rc5-25-g90a5a89), I noticed that loadavg always greater then 1.

I do a lot of digging and finally have more information on this issue:
CONFIG_NO_HZ_IDLE=y  # loadavg always > 1
CONFIG_HZ_PERIODIC=y # loadavg < 1

After this I tried to disable "nohz" at boot, to determine whether it is
statically added code under #ifdef during compilation or not, so I added
"nohz=off" to cmdline, and it helps!

Also if you enable preemption loadavg is also < 1:
CONFIG_PREEMPT_VOLUNTARY=y

After this I tried to reproduce issue under kvm/virtualbox (on desktop
machine, IOW not SUPERMICRO) - without success:
- For kvm I do (to avoid initrd requirement):
  $ sed -i 's/=m$/=y/' .config
- For virtualbox I tried both with initrd (+cgroups for systemd) and
  without.

In attachments there are config for SUPERMICRO and sched_debug snapshot:
- with CONFIG_NO_HZ_IDLE
- with CONFIG_HZ_PERIODIC
- sched_debug for CONFIG_NO_HZ_IDLE
- sched_debug for CONFIG_HZ_PERIODIC

And a little debugging information inlined:
- $ dmesg | fgrep MHz # CONFIG_NO_HZ_IDLE
  [    0.000000] tsc: Detected 2099.968 MHz processor
  [    1.374280] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
  [    2.793721] tsc: Refined TSC clocksource calibration: 2099.999 MHz
- $ dmesg | fgrep MHz # CONFIG_HZ_PERIODIC
  [    0.000000] tsc: Detected 2100.216 MHz processor <-- vs 2099
  [    1.440760] hpet0: 8 comparators, 64-bit 14.318180 MHz counter
  [    2.803963] tsc: Refined TSC clocksource calibration: 2099.998 MHz
- $ while sleep 30; do uptime; done # CONFIG_NO_HZ_IDLE
  15:49:15 up 10 min,  1 user,  load average: 1.03, 0.88, 0.44
  15:49:45 up 10 min,  1 user,  load average: 0.62, 0.80, 0.43
  15:50:15 up 11 min,  1 user,  load average: 3.28, 1.33, 0.61
  15:50:45 up 11 min,  1 user,  load average: 2.04, 1.22, 0.60
- $ while sleep 30; do uptime; done # CONFIG_HZ_PERIODIC
  15:25:26 up 1 day,  5:30,  2 users,  load average: 0.04, 0.07, 0.11
  15:25:56 up 1 day,  5:31,  2 users,  load average: 0.02, 0.06, 0.10
  15:26:26 up 1 day,  5:31,  2 users,  load average: 0.02, 0.05, 0.10
  15:26:56 up 1 day,  5:32,  2 users,  load average: 0.08, 0.06, 0.10
- /proc/stat looks fine (system=0/user=0/idle=100/wait=0 times)
- By using sysrq-trigger and 'l' option (show stack backtrace for all
  active CPUs), seems that all CPUs in a normal state:
  $ dmesg | fgrep RIP | cut -d' ' -f8 | sort -u # CONFIG_HZ_PERIODIC
  __bitmap_andnot+0x1c/0x70
  intel_idle+0xcc/0x150
  $ dmesg | fgrep RIP | cut -d' ' -f8 | sort -u # CONFIG_NO_HZ_IDLE
  intel_idle+0xcc/0x150
  __x2apic_send_IPI_mask+0x12e/0x150

Also I tried to count number of tracepoints sched/* - and for both
configurations I have pretty much the same numbers.

So if anybody could help me to investigate this - I will be grateful.

Thanks,
Azat.