Re: [patch] SMP alternatives

[ebiederm@xmission.com (Eric W. Biederman)]

Andi Kleen <ak@suse.de> writes:

>> I think I see the source of the confusion.  Scrubbing is the
>> process of taking data that is correctable and writing it back to
>> memory so that if a second correctable error occurs the net is still
>> corrected.
>
> That's supposed to be done by hardware, no? 
> At least the K8 has a hardware scrubber (although it's not always enabled)

Recent good implementations like the Opteron will do it for you.
Older or cheaper memory controllers will not.

Having an architecturally sane software scrubber as backup for
the hardware implementations is nice, and except in the cases
where someone disables the lock prefix it is takes very little
code on x86.

Even on the Opteron you could theoretically have the case of a brain-dead
external memory controller, although that is not likely.

>> Directed killing of processes is something that must be done
>> inside a synchronous exception (like a machine check) because otherwise
>> it is so racy you don't know who has seen the bad data.  
>
> If you try to do it this way then the code will become such
> a mess if not impossible to write that your changes to merge them
> and get it right are very slim. The only sane way to do all the locking etc. 
> is to hand over the handling to a thread. While that make the window
> of misusing the data wider it's the only sane alternative vs not
> doing it at all.
>
> Also due to the way hardware works with machine checks usually being
> async and not precise works you have that window anyways, so it's 
> not even worse. Also consider multiple CPUs.

First I don't have any code to do this, but I have though about it.
The races are the primary reason I have never pushed for something
like this.  With memory errors coming in as machine checks it is now
possible to do a correct version.

Essentially we are talking something with the complexity of a page
fault.  All that must happen synchronously is the task must
be stopped, and flagged.

As for races every cpu that accesses that data should take
a synchronous exception.  DMA should do something similar but I'm 
not as familiar with that side of the problem.  And because everything
takes an exception multiple cpu races are not a problem.

Of course there are still the memory errors that are so bad that
they don't even cause a machine check.  Those are a real pain
to debug, and fix.

In this latter case assuming the memory error is transient and
not hard using a write-combine memory attribute when you write
to re-initialize the ECC state is the way to go.  But remember
you most do it on the cpu that is part of the memory controller.
Otherwise something in the whole read/modify process will fail
to get the ECC state initialized properly on an Opteron.

Eric