Re: [PATCH 1/1] liveupdate: kho: calculate per-node scratch sizes before allocation

[George Guo <dongtai.guo@linux.dev>]

Hi Sourabh, Pratyush, 

> But in practice, this problem is only on CONFIG_NUMA=n and I don't think
> in practice KHO or LUO is being used in non-NUMA systems. So while I
> think it is worth fixing, I think we should also have a test where we
> enable CONFIG_NUMA.

Confirmed. My vmtest kernel has CONFIG_NUMA unset, and your analysis
matches the data.

I reran the vmtest with Sourabh's debug prints on both configurations:
same kernel, same QEMU command, and without my this fix patch, so the
default percentage policy is exercised. The only difference is
CONFIG_NUMA.

Without CONFIG_NUMA:

  KHO: Before low and global scratch allocations
  KHO: low size = 330185 KB
  KHO: global size = 322 MB
  KHO: Per node 0 = 672 MB
  KHO: After low and global scratch allocations
  KHO: low size = 330185 KB
  KHO: global size = 322 MB
  KHO: Per node 0 = 672 MB
  KHO: Failed to reserve nid 0 scratch buffer
  KHO: Failed to reserve scratch area, disabling kexec handover

With CONFIG_NUMA=y:

  KHO: Before low and global scratch allocations
  KHO: low size = 330197 KB
  KHO: global size = 322 MB
  KHO: Per node 0 = 96 MB
  KHO: After low and global scratch allocations
  KHO: low size = 330197 KB
  KHO: global size = 322 MB
  KHO: Per node 0 = 96 MB
  KHO: After per node allocation
  KHO: low size = 428501 KB
  KHO: global size = 418 MB
  KHO: Per node 0 = 288 MB

In the run without CONFIG_NUMA, the reserved-kern sum the sizing sees
is 330185 KB, which is the 98.45 MiB baseline plus the 224 MiB lowmem
scratch area: with memblock_get_region_node() hardcoded to return 0,
the NUMA_NO_NODE lowmem area is counted as node 0's kernel reservation.
Node 0 therefore requests 200% of (98.45 MiB + 224 MiB), rounded up to
32 MiB alignment: 672 MiB. That no longer fits next to the other areas
in the 1 GiB guest, and KHO disables itself.

In the run with CONFIG_NUMA=y, the real node ID excludes the
NUMA_NO_NODE regions, so node 0 requests 96 MiB. The allocation
succeeds and is visible in the sums printed afterwards (330197 KB ->
428501 KB), and the KHO selftest passes end to end ("KHO: found kexec
handover data", restore succeeds).

Sourabh, this also answers your question. Your PowerPC system runs
CONFIG_NUMA=y, so the node filter excludes the lowmem and global areas
and your numbers stay flat. Your experiment and mine are the two halves
of the same mechanism.

> I think on NUMA systems the problem is the other way round. The
> calculation for the global scratch also counts per-node allocations.

> So I think the proper fix for scratch sizing is what this patch does
> and then a fixup for the global scratch calculation as well.

Agreed. For v2 I plan to:

  - Compute all scratch sizes (lowmem, global, per-node) before any
    scratch area is allocated, per Mike's comment. The sizes are a
    function of the pre-allocation state, so this seals both feedback
    directions at once.
  - State the !CONFIG_NUMA condition in the commit message. The feedback
    described there is not unconditional, which is what triggered the
    question.
  - Add the LLM attribution Mike asked for.
  - Include the CONFIG_NUMA=y vmtest result as coverage.
  - Look at the global scratch calculation on NUMA systems as a
    follow-up.

Thanks,
George