Re: collision between ZONE_MOVABLE and memblock allocations

[Michal Hocko <mhocko@suse.com>]

On Wed 26-07-23 10:44:21, David Hildenbrand wrote:
> On 20.07.23 00:48, Ross Zwisler wrote:
> > On Wed, Jul 19, 2023 at 08:14:48AM +0200, Michal Hocko wrote:
> > > On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
> > > [...]
> > > > I do think that we need to fix this collision between ZONE_MOVABLE and memmap
> > > > allocations, because this issue essentially makes the movablecore= kernel
> > > > command line parameter useless in many cases, as the ZONE_MOVABLE region it
> > > > creates will often actually be unmovable.
> > > 
> > > movablecore is kinda hack and I would be more inclined to get rid of it
> > > rather than build more into it. Could you be more specific about your
> > > use case?
> > 
> > The problem that I'm trying to solve is that I'd like to be able to get kernel
> > core dumps off machines (chromebooks) so that we can debug crashes.  Because
> > the memory used by the crash kernel ("crashkernel=" kernel command line
> > option) is consumed the entire time the machine is booted, there is a strong
> > motivation to keep the crash kernel as small and as simple as possible.  To
> > this end I'm trying to get away without SSD drivers, not having to worry about
> > encryption on the SSDs, etc.
> 
> Okay, so you intend to keep the crashkernel area as small as possible.
> 
> > 
> > So, the rough plan right now is:
> >  > 1) During boot set aside some memory that won't contain kernel
> allocations.
> > I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.
> > 
> > We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE
> > region (or whatever non-kernel region) will be set aside as PMEM in the crash
> > kernel.  This is done with the memmap=nn[KMG]!ss[KMG] kernel command line
> > parameter passed to the crash kernel.
> > 
> > So, in my sample 4G VM system, I see:
> > 
> >    # lsmem --split ZONES --output-all
> >    RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
> >    0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
> >    0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
> >    0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
> >    0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable
> >    Memory block size:       128M
> >    Total online memory:       4G
> >    Total offline memory:      0B
> > 
> > so I'll pass "memmap=256M!0x130000000" to the crash kernel.
> > 
> > 2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set
> > aside only contains user data, which we don't want to store anyway.
> 
> I raised that in different context already, but such assumptions are not
> 100% future proof IMHO. For example, we might at one point be able to make
> user page tables movable and place them on there.
> 
> But yes, most kernel data structures (which you care about) will probably
> never be movable and never end up on these regions.
> 
> > We make a
> > filesystem in there, and create a kernel crash dump using 'makedumpfile':
> > 
> >    mkfs.ext4 /dev/pmem0
> >    mount /dev/pmem0 /mnt
> >    makedumpfile -c -d 31 /proc/vmcore /mnt/kdump
> > 
> > We then set up the next full kernel boot to also have this same PMEM region,
> > using the same memmap kernel parameter.  We reboot back into a full kernel.
> > 
> > 3) The next full kernel will be a normal boot with a full networking stack,
> > SSD drivers, disk encryption, etc.  We mount up our PMEM filesystem, pull out
> > the kdump and either store it somewhere persistent or upload it somewhere.  We
> > can then unmount the PMEM and reconfigure it back to system ram so that the
> > live system isn't missing memory.
> > 
> >    ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
> >    daxctl reconfigure-device --mode=system-ram dax0.0
> > 
> > This is the flow I'm trying to support, and have mostly working in a VM,
> > except up until now makedumpfile would crash because all the memblock
> > structures it needed were in the PMEM area that I had just wiped out by making
> > a new filesystem. :)
> 
> 
> Thinking out loud (and remembering that some architectures relocate the
> crashkernel during kexec, if I am not wrong), maybe the following would also
> work and make your setup eventually easier:
> 
> 1) Don't reserve a crashkernel area in the traditional way, instead reserve
> that area using CMA. It can be used for MOVABLE allocations.
> 
> 2) Let kexec load the crashkernel+initrd into ordinary memory only
> (consuming as much as you would need there).
> 
> 3) On kexec, relocate the crashkernel+initrd into the CMA area (overwriting
> any movable data in there)
> 
> 4) In makedumpfile, don't dump any memory that falls into the crashkernel
> area. It might already have been overwritten by the second kernel

This is more or less what Jiri is looking into.

-- 
Michal Hocko
SUSE Labs