PROBLEM: Off-by-one bug on sun4u SMP

PROBLEM: Off-by-one bug on sun4u SMP

[Ryan Davis]

On at least some sun4u machines, the CPU numbering starts at one not 0.
This causes an off-by-one bug as other parts of the code assume
zero-based numbering.

If you set max-cpus in the kernel config to the actual number of CPUs,
the last CPU will be ignored and unused.
This is because the CPU numbering starts at 1 but the code to check
against max-cpus assumes a zero-based numbering.

On my computer: Sun Ultra 60 2x450mhz Ultrasparc.
Building with max-cpus of 2 ignores the second cpu because 2 (one
based cpu number) >= 2 (zero based max cpus)
Rebuilding with a larger max-cpus is a workaround but non optimal.
---
Those who will not reason, are bigots, those who cannot, are fools,
and those who dare not, are slaves.
 -- Lord Byron

Re: PROBLEM: Off-by-one bug on sun4u SMP

[David Miller]

From: Ryan Davis <iconoclasmandheresy@gmail.com>
Date: Thu, 10 Jun 2010 11:05:09 -0700

> On at least some sun4u machines, the CPU numbering starts at one not 0.
> This causes an off-by-one bug as other parts of the code assume
> zero-based numbering.
> 
> If you set max-cpus in the kernel config to the actual number of CPUs,
> the last CPU will be ignored and unused.
> This is because the CPU numbering starts at 1 but the code to check
> against max-cpus assumes a zero-based numbering.
> 
> On my computer: Sun Ultra 60 2x450mhz Ultrasparc.
> Building with max-cpus of 2 ignores the second cpu because 2 (one
> based cpu number) >= 2 (zero based max cpus)
> Rebuilding with a larger max-cpus is a workaround but non optimal.

max-cpus means "one larger than the maximum PHYSICAL cpu number", not
the number of cpus.  That's what this setting means, at least on
sparc64.

Re: PROBLEM: Off-by-one bug on sun4u SMP

[Ryan Davis]

On Thu, Jun 10, 2010 at 5:37 PM, David Miller <davem@davemloft.net> wrote:
> From: Ryan Davis <iconoclasmandheresy@gmail.com>
> Date: Thu, 10 Jun 2010 11:05:09 -0700
>
>> On at least some sun4u machines, the CPU numbering starts at one not 0.
>> This causes an off-by-one bug as other parts of the code assume
>> zero-based numbering.
>>
>> If you set max-cpus in the kernel config to the actual number of CPUs,
>> the last CPU will be ignored and unused.
>> This is because the CPU numbering starts at 1 but the code to check
>> against max-cpus assumes a zero-based numbering.
>>
>> On my computer: Sun Ultra 60 2x450mhz Ultrasparc.
>> Building with max-cpus of 2 ignores the second cpu because 2 (one
>> based cpu number) >= 2 (zero based max cpus)
>> Rebuilding with a larger max-cpus is a workaround but non optimal.
>
> max-cpus means "one larger than the maximum PHYSICAL cpu number", not
> the number of cpus.  That's what this setting means, at least on
> sparc64.
>

OK... so there are no kernel data structures allocated for CPU #0
which is not present? It is my understanding that this is not the case
but I could be wrong. If this is true, then a chunk of kernel memory
is wasted tor the phantom CPU #0.

If this is the desired behavior then the semantics of the setting are
questionable, since a kernel compiled for 2 cpus will only run on 1...

A quick search was unable to find documentation on this quirk. Perhaps
you could point me at some?


---
Those who will not reason, are bigots, those who cannot, are fools,
and those who dare not, are slaves.
  -- Lord Byron

Re: PROBLEM: Off-by-one bug on sun4u SMP

[David Miller]

From: Ryan Davis <iconoclasmandheresy@gmail.com>
Date: Thu, 10 Jun 2010 21:01:28 -0700

> OK... so there are no kernel data structures allocated for CPU #0
> which is not present? It is my understanding that this is not the case
> but I could be wrong. If this is true, then a chunk of kernel memory
> is wasted tor the phantom CPU #0.

There are two cases.

1) There are certain arrays which we have to allocate before we know
   how many physical cpus will be present.  There are therefore
   CONFIG_NR_CPUS of these elements allocated statically in the
   kernel image.

   These cases are extremely few and constantly decreasing over time
   as we find ways to remove these cases.

2) Everything else is only allocated for cpus actually present.

Frankly, I often run with CONFIG_NR_CPUS=2048 or some crazy value like
that and the memory wasted by static data structures is very small.

> A quick search was unable to find documentation on this quirk. Perhaps
> you could point me at some?

Feel free to write a patch which adds that documentation.