Re: [PATCH 3/4] x86: Only direct map addresses that are marked as E820_RAM

[Jacob Shin <jacob.shin@amd.com>]

On Wed, Aug 22, 2012 at 04:30:49PM -0700, H. Peter Anvin wrote:
> On 08/14/2012 03:39 PM, Jacob Shin wrote:
> > Currently direct mappings are created for [ 0 to max_low_pfn<<PAGE_SHIFT )
> > and [ 4GB to max_pfn<<PAGE_SHIFT ), which may include regions that are not
> > backed by actual DRAM. This is fine for holes under 4GB which are covered
> > by fixed and variable range MTRRs to be UC. However, we run into trouble
> > on higher memory addresses which cannot be covered by MTRRs.
> > 
> > Our system with 1TB of RAM has an e820 that looks like this:
> > 
> >  BIOS-e820: [mem 0x0000000000000000-0x00000000000983ff] usable
> >  BIOS-e820: [mem 0x0000000000098400-0x000000000009ffff] reserved
> >  BIOS-e820: [mem 0x00000000000d0000-0x00000000000fffff] reserved
> >  BIOS-e820: [mem 0x0000000000100000-0x00000000c7ebffff] usable
> >  BIOS-e820: [mem 0x00000000c7ec0000-0x00000000c7ed7fff] ACPI data
> >  BIOS-e820: [mem 0x00000000c7ed8000-0x00000000c7ed9fff] ACPI NVS
> >  BIOS-e820: [mem 0x00000000c7eda000-0x00000000c7ffffff] reserved
> >  BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
> >  BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> >  BIOS-e820: [mem 0x00000000fff00000-0x00000000ffffffff] reserved
> >  BIOS-e820: [mem 0x0000000100000000-0x000000e037ffffff] usable
> >  BIOS-e820: [mem 0x000000e038000000-0x000000fcffffffff] reserved
> >  BIOS-e820: [mem 0x0000010000000000-0x0000011ffeffffff] usable
> > 
> > and so direct mappings are created for huge memory hole between
> > 0x000000e038000000 to 0x0000010000000000. Even though the kernel never
> > generates memory accesses in that region, since the page tables mark
> > them incorrectly as being WB, our (AMD) processor ends up causing a MCE
> > while doing some memory bookkeeping/optimizations around that area.
> > 
> > This patch iterates through e820 and only direct maps ranges that are
> > marked as E820_RAM, and keeps track of those pfn ranges. Depending on
> > the alignment of E820 ranges, this may possibly result in using smaller
> > size (i.e. 4K instead of 2M or 1G) page tables.
> > 
> > Signed-off-by: Jacob Shin <jacob.shin@amd.com>
> 
> I have one concern with this, which is that it leaves in place mapping
> below the initial max_pfn_mapped.  Although that neatly resolves the
> legacy area (0-1 MiB) issues, it really isn't right above the 1 MiB
> point.  Any way I could get you to seek out and unmap any such ranges?
> We have already seen some Dell machines which put memory holes in low
> RAM, and perhaps there are still some machines out there with an I/O
> hole at 15 MiB.

So I believe in V2 of the patchset this was done, however, Dave Young
from redhat reported that it broke their KVM guest with a user supplied
memory map that looked like this:

>> [    0.000000] e820: user-defined physical RAM map:
>> [    0.000000] user: [mem 0x0000000000010000-0x000000000009dbff] usable
>> [    0.000000] user: [mem 0x0000000024000000-0x0000000033f6bfff] usable

And looking into that scenario, the early boot code seems to allocates
space for fixmap right under initial max_pfn_mapped, which is no longer
direct mapped with my patch, and that seems to cause problems for later
APIC code that initializes APIC base address into the fixmap area.

So I guess to address your concern, we need to go back to V2 and try to
resolve the fixmap problem with user supplied memory map that reserves
memory below initial max_pfn_mapped ?

> 
> 	-hpa
> 
>