Re: SMP P4 APIC/interrupt balancing

[Shuji YAMAMURA <yamamura@flab.fujitsu.co.jp>]

At Tue, 16 Apr 2002 16:29:32 -0600 (MDT),
James Bourne <jbourne@MtRoyal.AB.CA> wrote:
> Hi All,
> Ok, we have a Dell PE4600 with dual P4X processors.  It will not balance
> interrupts between the two cpus, CPU0 always receives the bulk
> of the interrupts, although both processors are being utilized
> by processes on the system.  This system is not yet in production, so I
> have some time to try things on it.
> 
> Current kernel is 2.4.18 with the LSE APIC Routing patch
> (http://sourceforge.net/projects/lse).  2.4.18 stock (well ok, -rc4) has
> much the same results.

We observed the similar problem on our i860 based P4 SMP machine.
We examined several tests and got the following fact:

    - Only the upper 4 bits of the task priority is effective on
      Pentium 4.

The LSE APIC Routing patch assigns values from 0x10 to 0x18 to TPR for
setting task priority, that means the upper 4 bits are always '0001'.
Based on the previous obserbation, these values are useless in terms
of proper interrupt distribution.

To get an expected operation, upper 4 bits must be changed.

You may refer the following thread that talks this problem.
  "P4 SMP load balancing"
  http://marc.theaimsgroup.com/?t=100287923700006&r=1&w=2


Appendix.

Our experiments were examined on two different systems.

The following table shows the relations between the TPR value of each
CPU and the interrupt destination.

Pentium4 Machine:
  CPU: Pentium4 Xeon 2.0GHz x 2
  Motherboard: Supermicro P4DCE
  Chipset: Intel860

Pentium3 Machine:
  CPU: Pentium3 500MHz x 2
  Motherboard: ASUS CUR-DLS
  Chipset: Serverworks ServerSet3 LE

Boot Processor: CPU#0
   TPR Value     Interrupt CPU
  CPU#0 CPU#1     Pen3   Pen4
=========================================
    01    02      0or1    0
    02    01      0or1    0
    0f    00      0or1    0
-----------------------------------------
    10    01       1      1
    10    11       0      0
    11    10       1      0
    1f    10       1      0
    11    11      0or1    0
-----------------------------------------
    20    20      0or1    0
    20    10       1      1
    20    21       0      0
    21    20       1      0
-----------------------------------------

----
Shuji YAMAMURA
Grid Computing & Bioinformatics Laboratory, FUJITSU Laboratories, LTD.
E-mail: yamamura@flab.fujitsu.co.jp