Correlation CMA size and FORCE_MAX_ZONEORDER

Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi all

Working on a small device with 128MB of memory and using imx_v6_v7
defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
are not respected. The calculation done does not allow the requested
size. I think that this should be somehow documented and described but
I did not
find the documentation. Does it work this way?

With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb

Michael

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[David Hildenbrand]

On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> Hi all

Hi,

> 
> Working on a small device with 128MB of memory and using imx_v6_v7
> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> are not respected. The calculation done does not allow the requested
> size. I think that this should be somehow documented and described but
> I did not
> find the documentation. Does it work this way?
> 
> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb

The underlying constraint is that CMA regions require a certain minimum 
alignment+size. They cannot be arbitrarily in size.

CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream 
kernels to the size of a single pageblock.

In previous kernels, it used to be the size of the largest buddy 
allocation granularity (derived from MAX_ORDER, derived from 
FORCE_MAX_ZONEORDER).

On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer 
apply. On most archs, the minimum alignment+size should be 2 MiB 
(x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.

So far the theory. Are you still running into this limitation on 
upstream kernels?

-- 
Thanks,

David / dhildenb

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi David

On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
>
> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> > Hi all
>
> Hi,
>
> >
> > Working on a small device with 128MB of memory and using imx_v6_v7
> > defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> > are not respected. The calculation done does not allow the requested
> > size. I think that this should be somehow documented and described but
> > I did not
> > find the documentation. Does it work this way?
> >
> > With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> > the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
>
> The underlying constraint is that CMA regions require a certain minimum
> alignment+size. They cannot be arbitrarily in size.
>
> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> kernels to the size of a single pageblock.
>
> In previous kernels, it used to be the size of the largest buddy
> allocation granularity (derived from MAX_ORDER, derived from
> FORCE_MAX_ZONEORDER).
>
> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> apply. On most archs, the minimum alignment+size should be 2 MiB
> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
>
> So far the theory. Are you still running into this limitation on
> upstream kernels?
>

I can run 6-rc2 on my board. I test again but according to it, if I
put 4M as CMA in cma=4M in boot
parameters, the result is 32Mb of CMA. Apart of that seems that
process lime tiny membench can not even start
to mblock memory

Michael

> --
> Thanks,
>
> David / dhildenb
>


-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[David Hildenbrand]

On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> Hi David
> 
> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
>>> Hi all
>>
>> Hi,
>>
>>>
>>> Working on a small device with 128MB of memory and using imx_v6_v7
>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
>>> are not respected. The calculation done does not allow the requested
>>> size. I think that this should be somehow documented and described but
>>> I did not
>>> find the documentation. Does it work this way?
>>>
>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
>>
>> The underlying constraint is that CMA regions require a certain minimum
>> alignment+size. They cannot be arbitrarily in size.
>>
>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
>> kernels to the size of a single pageblock.
>>
>> In previous kernels, it used to be the size of the largest buddy
>> allocation granularity (derived from MAX_ORDER, derived from
>> FORCE_MAX_ZONEORDER).
>>
>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
>> apply. On most archs, the minimum alignment+size should be 2 MiB
>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
>>
>> So far the theory. Are you still running into this limitation on
>> upstream kernels?
>>
> 
> I can run 6-rc2 on my board. I test again but according to it, if I
> put 4M as CMA in cma=4M in boot
> parameters, the result is 32Mb of CMA. Apart of that seems that
> process lime tiny membench can not even start
> to mblock memory
> 

The CMA alignemnt change went into v5.19. If "cma=4M" still gives you > 
4M, can you post /proc/meminfo and the early console output?


-- 
Thanks,

David / dhildenb

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi

On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
>
> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> > Hi David
> >
> > On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
> >>
> >> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>> Hi all
> >>
> >> Hi,
> >>
> >>>
> >>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>> are not respected. The calculation done does not allow the requested
> >>> size. I think that this should be somehow documented and described but
> >>> I did not
> >>> find the documentation. Does it work this way?
> >>>
> >>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>
> >> The underlying constraint is that CMA regions require a certain minimum
> >> alignment+size. They cannot be arbitrarily in size.
> >>
> >> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >> kernels to the size of a single pageblock.
> >>
> >> In previous kernels, it used to be the size of the largest buddy
> >> allocation granularity (derived from MAX_ORDER, derived from
> >> FORCE_MAX_ZONEORDER).
> >>
> >> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >> apply. On most archs, the minimum alignment+size should be 2 MiB
> >> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>
> >> So far the theory. Are you still running into this limitation on
> >> upstream kernels?
> >>
> >
> > I can run 6-rc2 on my board. I test again but according to it, if I
> > put 4M as CMA in cma=4M in boot
> > parameters, the result is 32Mb of CMA. Apart of that seems that
> > process lime tiny membench can not even start
> > to mblock memory
> >
>
> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> 4M, can you post /proc/meminfo and the early console output?
>

cat /proc/cmdline
cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
root=ubi0:root rw ubi.mtd=ro
otfs rootfstype=ubifs rootwait=1
# cat /proc/meminfo
MemTotal:         109560 kB
MemFree:           56084 kB
MemAvailable:      56820 kB
Buffers:               0 kB
Cached:            39680 kB
SwapCached:            0 kB
Active:               44 kB
Inactive:            644 kB
Active(anon):         44 kB
Inactive(anon):      644 kB
Active(file):          0 kB
Inactive(file):        0 kB
Unevictable:       39596 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         109560 kB
LowFree:           56084 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           628 kB
Mapped:             1480 kB
Shmem:                84 kB
KReclaimable:       4268 kB
Slab:               8456 kB
SReclaimable:       4268 kB
SUnreclaim:         4188 kB
KernelStack:         392 kB
PageTables:           88 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:       54780 kB
Committed_AS:       1876 kB
VmallocTotal:     901120 kB
VmallocUsed:        2776 kB
VmallocChunk:          0 kB
Percpu:               72 kB
CmaTotal:          32768 kB
CmaFree:           32484 kB
# uname -a
Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
#

Then here https://pastebin.com/6MUB2VBM dmesg

CONFIG_ARM_MODULE_PLTS=y
CONFIG_FORCE_MAX_ZONEORDER=14
CONFIG_ALIGNMENT_TRAP=y
...
CONFIG_CMA
CONFIG_CMA_AREAS=7
...

CONFIG_CMA_SIZE_MBYTES=8
CONFIG_CMA_SIZE_SEL_MBYTES=y

Michael

>
> --
> Thanks,
>
> David / dhildenb
>
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[David Hildenbrand]

On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> Hi
> 
> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
>>> Hi David
>>>
>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
>>>>
>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
>>>>> Hi all
>>>>
>>>> Hi,
>>>>
>>>>>
>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
>>>>> are not respected. The calculation done does not allow the requested
>>>>> size. I think that this should be somehow documented and described but
>>>>> I did not
>>>>> find the documentation. Does it work this way?
>>>>>
>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
>>>>
>>>> The underlying constraint is that CMA regions require a certain minimum
>>>> alignment+size. They cannot be arbitrarily in size.
>>>>
>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
>>>> kernels to the size of a single pageblock.
>>>>
>>>> In previous kernels, it used to be the size of the largest buddy
>>>> allocation granularity (derived from MAX_ORDER, derived from
>>>> FORCE_MAX_ZONEORDER).
>>>>
>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
>>>>
>>>> So far the theory. Are you still running into this limitation on
>>>> upstream kernels?
>>>>
>>>
>>> I can run 6-rc2 on my board. I test again but according to it, if I
>>> put 4M as CMA in cma=4M in boot
>>> parameters, the result is 32Mb of CMA. Apart of that seems that
>>> process lime tiny membench can not even start
>>> to mblock memory
>>>
>>
>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
>> 4M, can you post /proc/meminfo and the early console output?
>>
> 
> cat /proc/cmdline
> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> root=ubi0:root rw ubi.mtd=ro
> otfs rootfstype=ubifs rootwait=1
> # cat /proc/meminfo
> MemTotal:         109560 kB
> MemFree:           56084 kB
> MemAvailable:      56820 kB
> Buffers:               0 kB
> Cached:            39680 kB
> SwapCached:            0 kB
> Active:               44 kB
> Inactive:            644 kB
> Active(anon):         44 kB
> Inactive(anon):      644 kB
> Active(file):          0 kB
> Inactive(file):        0 kB
> Unevictable:       39596 kB
> Mlocked:               0 kB
> HighTotal:             0 kB
> HighFree:              0 kB
> LowTotal:         109560 kB
> LowFree:           56084 kB
> SwapTotal:             0 kB
> SwapFree:              0 kB
> Dirty:                 0 kB
> Writeback:             0 kB
> AnonPages:           628 kB
> Mapped:             1480 kB
> Shmem:                84 kB
> KReclaimable:       4268 kB
> Slab:               8456 kB
> SReclaimable:       4268 kB
> SUnreclaim:         4188 kB
> KernelStack:         392 kB
> PageTables:           88 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:       54780 kB
> Committed_AS:       1876 kB
> VmallocTotal:     901120 kB
> VmallocUsed:        2776 kB
> VmallocChunk:          0 kB
> Percpu:               72 kB
> CmaTotal:          32768 kB
> CmaFree:           32484 kB
> # uname -a
> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> #
> 
> Then here https://pastebin.com/6MUB2VBM dmesg
> 
> CONFIG_ARM_MODULE_PLTS=y
> CONFIG_FORCE_MAX_ZONEORDER=14
> CONFIG_ALIGNMENT_TRAP=y
> ...
> CONFIG_CMA
> CONFIG_CMA_AREAS=7
> ...
> 
> CONFIG_CMA_SIZE_MBYTES=8
> CONFIG_CMA_SIZE_SEL_MBYTES=y
> 

Thanks!

I assume that in your setup, the pageblock size depends on MAX_ORDER 
and, therefore, FORCE_MAX_ZONEORDER.

This should be the case especially if CONFIG_HUGETLB_PAGE is not defined 
(include/linux/pageblock-flags.h).

In contrast to what I remember, the pageblock size does not seem to 
depend on the THP size (weird) as well.


So, yes, that limitation is still in effect for some kernel configs.

One could make the pageblock size configurable (similar to 
CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller 
pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.

I imagine something reasonable might be to set the pageblock size to 
2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.

-- 
Thanks,

David / dhildenb

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi

On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> > Hi
> >
> > On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
> >>
> >> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> >>> Hi David
> >>>
> >>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>
> >>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>>>> Hi all
> >>>>
> >>>> Hi,
> >>>>
> >>>>>
> >>>>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>>>> are not respected. The calculation done does not allow the requested
> >>>>> size. I think that this should be somehow documented and described but
> >>>>> I did not
> >>>>> find the documentation. Does it work this way?
> >>>>>
> >>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>>>
> >>>> The underlying constraint is that CMA regions require a certain minimum
> >>>> alignment+size. They cannot be arbitrarily in size.
> >>>>
> >>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >>>> kernels to the size of a single pageblock.
> >>>>
> >>>> In previous kernels, it used to be the size of the largest buddy
> >>>> allocation granularity (derived from MAX_ORDER, derived from
> >>>> FORCE_MAX_ZONEORDER).
> >>>>
> >>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >>>> apply. On most archs, the minimum alignment+size should be 2 MiB
> >>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>>>
> >>>> So far the theory. Are you still running into this limitation on
> >>>> upstream kernels?
> >>>>
> >>>
> >>> I can run 6-rc2 on my board. I test again but according to it, if I
> >>> put 4M as CMA in cma=4M in boot
> >>> parameters, the result is 32Mb of CMA. Apart of that seems that
> >>> process lime tiny membench can not even start
> >>> to mblock memory
> >>>
> >>
> >> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> >> 4M, can you post /proc/meminfo and the early console output?
> >>
> >
> > cat /proc/cmdline
> > cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> > root=ubi0:root rw ubi.mtd=ro
> > otfs rootfstype=ubifs rootwait=1
> > # cat /proc/meminfo
> > MemTotal:         109560 kB
> > MemFree:           56084 kB
> > MemAvailable:      56820 kB
> > Buffers:               0 kB
> > Cached:            39680 kB
> > SwapCached:            0 kB
> > Active:               44 kB
> > Inactive:            644 kB
> > Active(anon):         44 kB
> > Inactive(anon):      644 kB
> > Active(file):          0 kB
> > Inactive(file):        0 kB
> > Unevictable:       39596 kB
> > Mlocked:               0 kB
> > HighTotal:             0 kB
> > HighFree:              0 kB
> > LowTotal:         109560 kB
> > LowFree:           56084 kB
> > SwapTotal:             0 kB
> > SwapFree:              0 kB
> > Dirty:                 0 kB
> > Writeback:             0 kB
> > AnonPages:           628 kB
> > Mapped:             1480 kB
> > Shmem:                84 kB
> > KReclaimable:       4268 kB
> > Slab:               8456 kB
> > SReclaimable:       4268 kB
> > SUnreclaim:         4188 kB
> > KernelStack:         392 kB
> > PageTables:           88 kB
> > NFS_Unstable:          0 kB
> > Bounce:                0 kB
> > WritebackTmp:          0 kB
> > CommitLimit:       54780 kB
> > Committed_AS:       1876 kB
> > VmallocTotal:     901120 kB
> > VmallocUsed:        2776 kB
> > VmallocChunk:          0 kB
> > Percpu:               72 kB
> > CmaTotal:          32768 kB
> > CmaFree:           32484 kB
> > # uname -a
> > Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> > #
> >
> > Then here https://pastebin.com/6MUB2VBM dmesg
> >
> > CONFIG_ARM_MODULE_PLTS=y
> > CONFIG_FORCE_MAX_ZONEORDER=14
> > CONFIG_ALIGNMENT_TRAP=y
> > ...
> > CONFIG_CMA
> > CONFIG_CMA_AREAS=7
> > ...
> >
> > CONFIG_CMA_SIZE_MBYTES=8
> > CONFIG_CMA_SIZE_SEL_MBYTES=y
> >
>
> Thanks!
>
> I assume that in your setup, the pageblock size depends on MAX_ORDER
> and, therefore, FORCE_MAX_ZONEORDER.
>
> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
> (include/linux/pageblock-flags.h).
>
> In contrast to what I remember, the pageblock size does not seem to
> depend on the THP size (weird) as well.
>
>
> So, yes, that limitation is still in effect for some kernel configs.
>
> One could make the pageblock size configurable (similar to
> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
>
> I imagine something reasonable might be to set the pageblock size to
> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
>

I don't think making more configuration options makes things clear.
When we enable some configuration
we can force down the configuration. You need to explain clearly how
you envision it. FORCE_MAX_ZONEORDER
for me is the largest allocation that you can get from a zone (ex CMA
one). Any request allocation that is align to the
CMA align and can fit inside a region should be allowed

What am I missing?

Michael

> --
> Thanks,
>
> David / dhildenb
>
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[David Hildenbrand]

On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:
> Hi
> 
> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
>>> Hi
>>>
>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
>>>>
>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
>>>>> Hi David
>>>>>
>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
>>>>>>
>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
>>>>>>> Hi all
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>>
>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
>>>>>>> are not respected. The calculation done does not allow the requested
>>>>>>> size. I think that this should be somehow documented and described but
>>>>>>> I did not
>>>>>>> find the documentation. Does it work this way?
>>>>>>>
>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
>>>>>>
>>>>>> The underlying constraint is that CMA regions require a certain minimum
>>>>>> alignment+size. They cannot be arbitrarily in size.
>>>>>>
>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
>>>>>> kernels to the size of a single pageblock.
>>>>>>
>>>>>> In previous kernels, it used to be the size of the largest buddy
>>>>>> allocation granularity (derived from MAX_ORDER, derived from
>>>>>> FORCE_MAX_ZONEORDER).
>>>>>>
>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
>>>>>>
>>>>>> So far the theory. Are you still running into this limitation on
>>>>>> upstream kernels?
>>>>>>
>>>>>
>>>>> I can run 6-rc2 on my board. I test again but according to it, if I
>>>>> put 4M as CMA in cma=4M in boot
>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that
>>>>> process lime tiny membench can not even start
>>>>> to mblock memory
>>>>>
>>>>
>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
>>>> 4M, can you post /proc/meminfo and the early console output?
>>>>
>>>
>>> cat /proc/cmdline
>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
>>> root=ubi0:root rw ubi.mtd=ro
>>> otfs rootfstype=ubifs rootwait=1
>>> # cat /proc/meminfo
>>> MemTotal:         109560 kB
>>> MemFree:           56084 kB
>>> MemAvailable:      56820 kB
>>> Buffers:               0 kB
>>> Cached:            39680 kB
>>> SwapCached:            0 kB
>>> Active:               44 kB
>>> Inactive:            644 kB
>>> Active(anon):         44 kB
>>> Inactive(anon):      644 kB
>>> Active(file):          0 kB
>>> Inactive(file):        0 kB
>>> Unevictable:       39596 kB
>>> Mlocked:               0 kB
>>> HighTotal:             0 kB
>>> HighFree:              0 kB
>>> LowTotal:         109560 kB
>>> LowFree:           56084 kB
>>> SwapTotal:             0 kB
>>> SwapFree:              0 kB
>>> Dirty:                 0 kB
>>> Writeback:             0 kB
>>> AnonPages:           628 kB
>>> Mapped:             1480 kB
>>> Shmem:                84 kB
>>> KReclaimable:       4268 kB
>>> Slab:               8456 kB
>>> SReclaimable:       4268 kB
>>> SUnreclaim:         4188 kB
>>> KernelStack:         392 kB
>>> PageTables:           88 kB
>>> NFS_Unstable:          0 kB
>>> Bounce:                0 kB
>>> WritebackTmp:          0 kB
>>> CommitLimit:       54780 kB
>>> Committed_AS:       1876 kB
>>> VmallocTotal:     901120 kB
>>> VmallocUsed:        2776 kB
>>> VmallocChunk:          0 kB
>>> Percpu:               72 kB
>>> CmaTotal:          32768 kB
>>> CmaFree:           32484 kB
>>> # uname -a
>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
>>> #
>>>
>>> Then here https://pastebin.com/6MUB2VBM dmesg
>>>
>>> CONFIG_ARM_MODULE_PLTS=y
>>> CONFIG_FORCE_MAX_ZONEORDER=14
>>> CONFIG_ALIGNMENT_TRAP=y
>>> ...
>>> CONFIG_CMA
>>> CONFIG_CMA_AREAS=7
>>> ...
>>>
>>> CONFIG_CMA_SIZE_MBYTES=8
>>> CONFIG_CMA_SIZE_SEL_MBYTES=y
>>>
>>
>> Thanks!
>>
>> I assume that in your setup, the pageblock size depends on MAX_ORDER
>> and, therefore, FORCE_MAX_ZONEORDER.
>>
>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
>> (include/linux/pageblock-flags.h).
>>
>> In contrast to what I remember, the pageblock size does not seem to
>> depend on the THP size (weird) as well.
>>
>>
>> So, yes, that limitation is still in effect for some kernel configs.
>>
>> One could make the pageblock size configurable (similar to
>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
>>
>> I imagine something reasonable might be to set the pageblock size to
>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
>>
> 
> I don't think making more configuration options makes things clear.

Yes, in an ideal case it should be automatic.

> When we enable some configuration
> we can force down the configuration. You need to explain clearly how
> you envision it. FORCE_MAX_ZONEORDER
> for me is the largest allocation that you can get from a zone (ex CMA
> one). Any request allocation that is align to the

FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum 
allocation size of the buddy in general.

> CMA align and can fit inside a region should be allowed
> 
> What am I missing?

I think that the issue is that the CMA alignments nowadays depend on the 
pageblock size. And the pageblock size depends on *some* configurations 
on the maximum allocation size of the buddy.

Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size 
alignment is not trivial.

For example, with CONFIG_HUGETLB_PAGE there might not be such an 
interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.


Let me phrase it this way: is it good enough in you setup to get 32mb vs 
8mb or do you want/need to reduce it without adjusting 
FORCE_MAX_ZONEORDER ?

-- 
Thanks,

David / dhildenb

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi David


On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:
> > Hi
> >
> > On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
> >>
> >> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> >>> Hi
> >>>
> >>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>
> >>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> >>>>> Hi David
> >>>>>
> >>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>>>
> >>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>>>>>> Hi all
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>>
> >>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>>>>>> are not respected. The calculation done does not allow the requested
> >>>>>>> size. I think that this should be somehow documented and described but
> >>>>>>> I did not
> >>>>>>> find the documentation. Does it work this way?
> >>>>>>>
> >>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>>>>>
> >>>>>> The underlying constraint is that CMA regions require a certain minimum
> >>>>>> alignment+size. They cannot be arbitrarily in size.
> >>>>>>
> >>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >>>>>> kernels to the size of a single pageblock.
> >>>>>>
> >>>>>> In previous kernels, it used to be the size of the largest buddy
> >>>>>> allocation granularity (derived from MAX_ORDER, derived from
> >>>>>> FORCE_MAX_ZONEORDER).
> >>>>>>
> >>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
> >>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>>>>>
> >>>>>> So far the theory. Are you still running into this limitation on
> >>>>>> upstream kernels?
> >>>>>>
> >>>>>
> >>>>> I can run 6-rc2 on my board. I test again but according to it, if I
> >>>>> put 4M as CMA in cma=4M in boot
> >>>>> parameters, the result is 32Mb of CMA. Apart of that seems that
> >>>>> process lime tiny membench can not even start
> >>>>> to mblock memory
> >>>>>
> >>>>
> >>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> >>>> 4M, can you post /proc/meminfo and the early console output?
> >>>>
> >>>
> >>> cat /proc/cmdline
> >>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> >>> root=ubi0:root rw ubi.mtd=ro
> >>> otfs rootfstype=ubifs rootwait=1
> >>> # cat /proc/meminfo
> >>> MemTotal:         109560 kB
> >>> MemFree:           56084 kB
> >>> MemAvailable:      56820 kB
> >>> Buffers:               0 kB
> >>> Cached:            39680 kB
> >>> SwapCached:            0 kB
> >>> Active:               44 kB
> >>> Inactive:            644 kB
> >>> Active(anon):         44 kB
> >>> Inactive(anon):      644 kB
> >>> Active(file):          0 kB
> >>> Inactive(file):        0 kB
> >>> Unevictable:       39596 kB
> >>> Mlocked:               0 kB
> >>> HighTotal:             0 kB
> >>> HighFree:              0 kB
> >>> LowTotal:         109560 kB
> >>> LowFree:           56084 kB
> >>> SwapTotal:             0 kB
> >>> SwapFree:              0 kB
> >>> Dirty:                 0 kB
> >>> Writeback:             0 kB
> >>> AnonPages:           628 kB
> >>> Mapped:             1480 kB
> >>> Shmem:                84 kB
> >>> KReclaimable:       4268 kB
> >>> Slab:               8456 kB
> >>> SReclaimable:       4268 kB
> >>> SUnreclaim:         4188 kB
> >>> KernelStack:         392 kB
> >>> PageTables:           88 kB
> >>> NFS_Unstable:          0 kB
> >>> Bounce:                0 kB
> >>> WritebackTmp:          0 kB
> >>> CommitLimit:       54780 kB
> >>> Committed_AS:       1876 kB
> >>> VmallocTotal:     901120 kB
> >>> VmallocUsed:        2776 kB
> >>> VmallocChunk:          0 kB
> >>> Percpu:               72 kB
> >>> CmaTotal:          32768 kB
> >>> CmaFree:           32484 kB
> >>> # uname -a
> >>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> >>> #
> >>>
> >>> Then here https://pastebin.com/6MUB2VBM dmesg
> >>>
> >>> CONFIG_ARM_MODULE_PLTS=y
> >>> CONFIG_FORCE_MAX_ZONEORDER=14
> >>> CONFIG_ALIGNMENT_TRAP=y
> >>> ...
> >>> CONFIG_CMA
> >>> CONFIG_CMA_AREAS=7
> >>> ...
> >>>
> >>> CONFIG_CMA_SIZE_MBYTES=8
> >>> CONFIG_CMA_SIZE_SEL_MBYTES=y
> >>>
> >>
> >> Thanks!
> >>
> >> I assume that in your setup, the pageblock size depends on MAX_ORDER
> >> and, therefore, FORCE_MAX_ZONEORDER.
> >>
> >> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
> >> (include/linux/pageblock-flags.h).
> >>
> >> In contrast to what I remember, the pageblock size does not seem to
> >> depend on the THP size (weird) as well.
> >>
> >>
> >> So, yes, that limitation is still in effect for some kernel configs.
> >>
> >> One could make the pageblock size configurable (similar to
> >> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
> >> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
> >>
> >> I imagine something reasonable might be to set the pageblock size to
> >> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
> >>
> >
> > I don't think making more configuration options makes things clear.
>
> Yes, in an ideal case it should be automatic.
>
> > When we enable some configuration
> > we can force down the configuration. You need to explain clearly how
> > you envision it. FORCE_MAX_ZONEORDER
> > for me is the largest allocation that you can get from a zone (ex CMA
> > one). Any request allocation that is align to the
>
> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum
> allocation size of the buddy in general.
>
> > CMA align and can fit inside a region should be allowed
> >
> > What am I missing?
>
> I think that the issue is that the CMA alignments nowadays depend on the
> pageblock size. And the pageblock size depends on *some* configurations
> on the maximum allocation size of the buddy.
>
> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size
> alignment is not trivial.
>
> For example, with CONFIG_HUGETLB_PAGE there might not be such an
> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.
>
>
> Let me phrase it this way: is it good enough in you setup to get 32mb vs
> 8mb or do you want/need to reduce it without adjusting
> FORCE_MAX_ZONEORDER ?

Wait we have:
- CMA kconfig alignment that in most config we have not considered
natural dma alignment but is put to 1Mb in a lot of embedded
- We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not
effect if ZONEORDER is not reasonable and without
HUGETLB_PAGE
- etc

Changing MAX_ZONEORDER is ok and yes if you have an IOT device that
you know about your CMA allocation, it makes no sense to have
it 32MB for a 128Mb device. What I point out is that I need to figure
it out because in Kconfig there is no mention of it. Should it be
added there?

Michael

>
> --
> Thanks,
>
> David / dhildenb
>

--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[David Hildenbrand]

On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote:
> Hi David
> 
> 
> On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:
>>> Hi
>>>
>>> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
>>>>
>>>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
>>>>> Hi
>>>>>
>>>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
>>>>>>
>>>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
>>>>>>> Hi David
>>>>>>>
>>>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
>>>>>>>>
>>>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
>>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
>>>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
>>>>>>>>> are not respected. The calculation done does not allow the requested
>>>>>>>>> size. I think that this should be somehow documented and described but
>>>>>>>>> I did not
>>>>>>>>> find the documentation. Does it work this way?
>>>>>>>>>
>>>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
>>>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
>>>>>>>>
>>>>>>>> The underlying constraint is that CMA regions require a certain minimum
>>>>>>>> alignment+size. They cannot be arbitrarily in size.
>>>>>>>>
>>>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
>>>>>>>> kernels to the size of a single pageblock.
>>>>>>>>
>>>>>>>> In previous kernels, it used to be the size of the largest buddy
>>>>>>>> allocation granularity (derived from MAX_ORDER, derived from
>>>>>>>> FORCE_MAX_ZONEORDER).
>>>>>>>>
>>>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
>>>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
>>>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
>>>>>>>>
>>>>>>>> So far the theory. Are you still running into this limitation on
>>>>>>>> upstream kernels?
>>>>>>>>
>>>>>>>
>>>>>>> I can run 6-rc2 on my board. I test again but according to it, if I
>>>>>>> put 4M as CMA in cma=4M in boot
>>>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that
>>>>>>> process lime tiny membench can not even start
>>>>>>> to mblock memory
>>>>>>>
>>>>>>
>>>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
>>>>>> 4M, can you post /proc/meminfo and the early console output?
>>>>>>
>>>>>
>>>>> cat /proc/cmdline
>>>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
>>>>> root=ubi0:root rw ubi.mtd=ro
>>>>> otfs rootfstype=ubifs rootwait=1
>>>>> # cat /proc/meminfo
>>>>> MemTotal:         109560 kB
>>>>> MemFree:           56084 kB
>>>>> MemAvailable:      56820 kB
>>>>> Buffers:               0 kB
>>>>> Cached:            39680 kB
>>>>> SwapCached:            0 kB
>>>>> Active:               44 kB
>>>>> Inactive:            644 kB
>>>>> Active(anon):         44 kB
>>>>> Inactive(anon):      644 kB
>>>>> Active(file):          0 kB
>>>>> Inactive(file):        0 kB
>>>>> Unevictable:       39596 kB
>>>>> Mlocked:               0 kB
>>>>> HighTotal:             0 kB
>>>>> HighFree:              0 kB
>>>>> LowTotal:         109560 kB
>>>>> LowFree:           56084 kB
>>>>> SwapTotal:             0 kB
>>>>> SwapFree:              0 kB
>>>>> Dirty:                 0 kB
>>>>> Writeback:             0 kB
>>>>> AnonPages:           628 kB
>>>>> Mapped:             1480 kB
>>>>> Shmem:                84 kB
>>>>> KReclaimable:       4268 kB
>>>>> Slab:               8456 kB
>>>>> SReclaimable:       4268 kB
>>>>> SUnreclaim:         4188 kB
>>>>> KernelStack:         392 kB
>>>>> PageTables:           88 kB
>>>>> NFS_Unstable:          0 kB
>>>>> Bounce:                0 kB
>>>>> WritebackTmp:          0 kB
>>>>> CommitLimit:       54780 kB
>>>>> Committed_AS:       1876 kB
>>>>> VmallocTotal:     901120 kB
>>>>> VmallocUsed:        2776 kB
>>>>> VmallocChunk:          0 kB
>>>>> Percpu:               72 kB
>>>>> CmaTotal:          32768 kB
>>>>> CmaFree:           32484 kB
>>>>> # uname -a
>>>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
>>>>> #
>>>>>
>>>>> Then here https://pastebin.com/6MUB2VBM dmesg
>>>>>
>>>>> CONFIG_ARM_MODULE_PLTS=y
>>>>> CONFIG_FORCE_MAX_ZONEORDER=14
>>>>> CONFIG_ALIGNMENT_TRAP=y
>>>>> ...
>>>>> CONFIG_CMA
>>>>> CONFIG_CMA_AREAS=7
>>>>> ...
>>>>>
>>>>> CONFIG_CMA_SIZE_MBYTES=8
>>>>> CONFIG_CMA_SIZE_SEL_MBYTES=y
>>>>>
>>>>
>>>> Thanks!
>>>>
>>>> I assume that in your setup, the pageblock size depends on MAX_ORDER
>>>> and, therefore, FORCE_MAX_ZONEORDER.
>>>>
>>>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
>>>> (include/linux/pageblock-flags.h).
>>>>
>>>> In contrast to what I remember, the pageblock size does not seem to
>>>> depend on the THP size (weird) as well.
>>>>
>>>>
>>>> So, yes, that limitation is still in effect for some kernel configs.
>>>>
>>>> One could make the pageblock size configurable (similar to
>>>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
>>>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
>>>>
>>>> I imagine something reasonable might be to set the pageblock size to
>>>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
>>>>
>>>
>>> I don't think making more configuration options makes things clear.
>>
>> Yes, in an ideal case it should be automatic.
>>
>>> When we enable some configuration
>>> we can force down the configuration. You need to explain clearly how
>>> you envision it. FORCE_MAX_ZONEORDER
>>> for me is the largest allocation that you can get from a zone (ex CMA
>>> one). Any request allocation that is align to the
>>
>> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum
>> allocation size of the buddy in general.
>>
>>> CMA align and can fit inside a region should be allowed
>>>
>>> What am I missing?
>>
>> I think that the issue is that the CMA alignments nowadays depend on the
>> pageblock size. And the pageblock size depends on *some* configurations
>> on the maximum allocation size of the buddy.
>>
>> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size
>> alignment is not trivial.
>>
>> For example, with CONFIG_HUGETLB_PAGE there might not be such an
>> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.
>>
>>
>> Let me phrase it this way: is it good enough in you setup to get 32mb vs
>> 8mb or do you want/need to reduce it without adjusting
>> FORCE_MAX_ZONEORDER ?
> 
> Wait we have:
> - CMA kconfig alignment that in most config we have not considered
> natural dma alignment but is put to 1Mb in a lot of embedded
> - We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not
> effect if ZONEORDER is not reasonable and without
> HUGETLB_PAGE
> - etc
> 
> Changing MAX_ZONEORDER is ok and yes if you have an IOT device that
> you know about your CMA allocation, it makes no sense to have
> it 32MB for a 128Mb device. What I point out is that I need to figure
> it out because in Kconfig there is no mention of it. Should it be
> added there?

Something like

"Note that in some configurations, the CMA size must be aligned to the 
maximum allocation granularity of the buddy allocator, consequently 
depending on FORCE_MAX_ZONEORDER. The requested CMA size might get 
increased accordingly."

Maybe we'd want some kind of a warning in the kernel as well. If someone 
specifies "cma=2MB" but gets 32MB or more that might be a problem.

Does something like that make sense to you?

-- 
Thanks,

David / dhildenb

Re: Correlation CMA size and FORCE_MAX_ZONEORDER

[Michael Nazzareno Trimarchi]

Hi

On Mon, Sep 19, 2022 at 2:08 PM David Hildenbrand <david@redhat.com> wrote:
>
> On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote:
> > Hi David
> >
> >
> > On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@redhat.com> wrote:
> >>
> >> On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:
> >>> Hi
> >>>
> >>> On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@redhat.com> wrote:
> >>>>
> >>>> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> >>>>> Hi
> >>>>>
> >>>>> On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>>>
> >>>>>> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> >>>>>>> Hi David
> >>>>>>>
> >>>>>>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>>>>>>>> Hi all
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>>>>>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>>>>>>>> are not respected. The calculation done does not allow the requested
> >>>>>>>>> size. I think that this should be somehow documented and described but
> >>>>>>>>> I did not
> >>>>>>>>> find the documentation. Does it work this way?
> >>>>>>>>>
> >>>>>>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>>>>>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>>>>>>>
> >>>>>>>> The underlying constraint is that CMA regions require a certain minimum
> >>>>>>>> alignment+size. They cannot be arbitrarily in size.
> >>>>>>>>
> >>>>>>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >>>>>>>> kernels to the size of a single pageblock.
> >>>>>>>>
> >>>>>>>> In previous kernels, it used to be the size of the largest buddy
> >>>>>>>> allocation granularity (derived from MAX_ORDER, derived from
> >>>>>>>> FORCE_MAX_ZONEORDER).
> >>>>>>>>
> >>>>>>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >>>>>>>> apply. On most archs, the minimum alignment+size should be 2 MiB
> >>>>>>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>>>>>>>
> >>>>>>>> So far the theory. Are you still running into this limitation on
> >>>>>>>> upstream kernels?
> >>>>>>>>
> >>>>>>>
> >>>>>>> I can run 6-rc2 on my board. I test again but according to it, if I
> >>>>>>> put 4M as CMA in cma=4M in boot
> >>>>>>> parameters, the result is 32Mb of CMA. Apart of that seems that
> >>>>>>> process lime tiny membench can not even start
> >>>>>>> to mblock memory
> >>>>>>>
> >>>>>>
> >>>>>> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> >>>>>> 4M, can you post /proc/meminfo and the early console output?
> >>>>>>
> >>>>>
> >>>>> cat /proc/cmdline
> >>>>> cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> >>>>> root=ubi0:root rw ubi.mtd=ro
> >>>>> otfs rootfstype=ubifs rootwait=1
> >>>>> # cat /proc/meminfo
> >>>>> MemTotal:         109560 kB
> >>>>> MemFree:           56084 kB
> >>>>> MemAvailable:      56820 kB
> >>>>> Buffers:               0 kB
> >>>>> Cached:            39680 kB
> >>>>> SwapCached:            0 kB
> >>>>> Active:               44 kB
> >>>>> Inactive:            644 kB
> >>>>> Active(anon):         44 kB
> >>>>> Inactive(anon):      644 kB
> >>>>> Active(file):          0 kB
> >>>>> Inactive(file):        0 kB
> >>>>> Unevictable:       39596 kB
> >>>>> Mlocked:               0 kB
> >>>>> HighTotal:             0 kB
> >>>>> HighFree:              0 kB
> >>>>> LowTotal:         109560 kB
> >>>>> LowFree:           56084 kB
> >>>>> SwapTotal:             0 kB
> >>>>> SwapFree:              0 kB
> >>>>> Dirty:                 0 kB
> >>>>> Writeback:             0 kB
> >>>>> AnonPages:           628 kB
> >>>>> Mapped:             1480 kB
> >>>>> Shmem:                84 kB
> >>>>> KReclaimable:       4268 kB
> >>>>> Slab:               8456 kB
> >>>>> SReclaimable:       4268 kB
> >>>>> SUnreclaim:         4188 kB
> >>>>> KernelStack:         392 kB
> >>>>> PageTables:           88 kB
> >>>>> NFS_Unstable:          0 kB
> >>>>> Bounce:                0 kB
> >>>>> WritebackTmp:          0 kB
> >>>>> CommitLimit:       54780 kB
> >>>>> Committed_AS:       1876 kB
> >>>>> VmallocTotal:     901120 kB
> >>>>> VmallocUsed:        2776 kB
> >>>>> VmallocChunk:          0 kB
> >>>>> Percpu:               72 kB
> >>>>> CmaTotal:          32768 kB
> >>>>> CmaFree:           32484 kB
> >>>>> # uname -a
> >>>>> Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> >>>>> #
> >>>>>
> >>>>> Then here https://pastebin.com/6MUB2VBM dmesg
> >>>>>
> >>>>> CONFIG_ARM_MODULE_PLTS=y
> >>>>> CONFIG_FORCE_MAX_ZONEORDER=14
> >>>>> CONFIG_ALIGNMENT_TRAP=y
> >>>>> ...
> >>>>> CONFIG_CMA
> >>>>> CONFIG_CMA_AREAS=7
> >>>>> ...
> >>>>>
> >>>>> CONFIG_CMA_SIZE_MBYTES=8
> >>>>> CONFIG_CMA_SIZE_SEL_MBYTES=y
> >>>>>
> >>>>
> >>>> Thanks!
> >>>>
> >>>> I assume that in your setup, the pageblock size depends on MAX_ORDER
> >>>> and, therefore, FORCE_MAX_ZONEORDER.
> >>>>
> >>>> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
> >>>> (include/linux/pageblock-flags.h).
> >>>>
> >>>> In contrast to what I remember, the pageblock size does not seem to
> >>>> depend on the THP size (weird) as well.
> >>>>
> >>>>
> >>>> So, yes, that limitation is still in effect for some kernel configs.
> >>>>
> >>>> One could make the pageblock size configurable (similar to
> >>>> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
> >>>> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
> >>>>
> >>>> I imagine something reasonable might be to set the pageblock size to
> >>>> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
> >>>>
> >>>
> >>> I don't think making more configuration options makes things clear.
> >>
> >> Yes, in an ideal case it should be automatic.
> >>
> >>> When we enable some configuration
> >>> we can force down the configuration. You need to explain clearly how
> >>> you envision it. FORCE_MAX_ZONEORDER
> >>> for me is the largest allocation that you can get from a zone (ex CMA
> >>> one). Any request allocation that is align to the
> >>
> >> FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum
> >> allocation size of the buddy in general.
> >>
> >>> CMA align and can fit inside a region should be allowed
> >>>
> >>> What am I missing?
> >>
> >> I think that the issue is that the CMA alignments nowadays depend on the
> >> pageblock size. And the pageblock size depends on *some* configurations
> >> on the maximum allocation size of the buddy.
> >>
> >> Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size
> >> alignment is not trivial.
> >>
> >> For example, with CONFIG_HUGETLB_PAGE there might not be such an
> >> interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.
> >>
> >>
> >> Let me phrase it this way: is it good enough in you setup to get 32mb vs
> >> 8mb or do you want/need to reduce it without adjusting
> >> FORCE_MAX_ZONEORDER ?
> >
> > Wait we have:
> > - CMA kconfig alignment that in most config we have not considered
> > natural dma alignment but is put to 1Mb in a lot of embedded
> > - We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not
> > effect if ZONEORDER is not reasonable and without
> > HUGETLB_PAGE
> > - etc
> >
> > Changing MAX_ZONEORDER is ok and yes if you have an IOT device that
> > you know about your CMA allocation, it makes no sense to have
> > it 32MB for a 128Mb device. What I point out is that I need to figure
> > it out because in Kconfig there is no mention of it. Should it be
> > added there?
>
> Something like
>
> "Note that in some configurations, the CMA size must be aligned to the
> maximum allocation granularity of the buddy allocator, consequently
> depending on FORCE_MAX_ZONEORDER. The requested CMA size might get
> increased accordingly."
>
> Maybe we'd want some kind of a warning in the kernel as well. If someone
> specifies "cma=2MB" but gets 32MB or more that might be a problem.
>
> Does something like that make sense to you?

Yes we need to warn the user and add to Kconfig. Maybe wait for other
to get a feedback

Michael

>
> --
> Thanks,
>
> David / dhildenb
>


-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com