Re: [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 8181 bytes --]

On Thu, Oct 15, 2026 at 11:21:33AM -0700, Douglas Anderson wrote:
> 
> Devicetree overlays allow you to have a "base" compiled devicetree and then
> to apply an "overlay" to it to add or modify nodes. Devicetree overlays have
> at least two purposes:
> 
> 1. Supporting add-on boards (like "hats" on a dev board). In this case the
>    main board is described as a base devicetree and the add-on board is an
>    overlay. In this case there is a clear separation between the base and the
>    add-on board and the base/overlay separation models reality nicely.
> 
> 2. Supporting a "family" of similar boards in a space-efficient way. In
>    upstream Linux, there are 47 distinct sc7180-trogdor boards supported.
>    These board are all highly similar, but we compile (and ship) a separate
>    devicetree for each. Devicetree overlays can make this much more efficient
>    by factoring the common bits to the base devicetree and having each
>    of the 47 boards represented by a small devicetree overlay.
> 
> Up to this point, the usage of overlays in the upstream devicetree repo has
> mostly been for case #1.

I don't think it invalidates the concept you're going for, but this
description is a bit misleading without some historical context.  In
particular this is a bit confused by the fact that there are both
source-level (dts) overlays and binary-level (dtb) overlays.

Source overlays came first, in conjuction with /include/ support in
dtc, and were intended for use case (2) and similar.  A group of
boards with the same SoC was the most common case, but families of
similar boards and families of similar SoCs also used this mechanism.
At the time this was still envisaging that a complete tree was built
at boot time, and didn't change.

This started to drift into use case (1) since without hotplug (we'll
get to that), a board with different hats (or whatever) can be
considered variants of the same base board.

That still required the complete tree be built at compile time (or at
least bootloader / kernel installation time).  That wasn't great for
two reasons, roughly matching your two use cases:

 1. Compile time is obviously no good for hotplugged hats.  Even for
    coldplug hats, it's desirable to be able to complete the tree at
    boot time rather than compile / deploy time.

 2. Bootloaders often want to support a variety of related boards, not
    just a single configuration.  That was already possible by
    incorporating a full dtb for every variant, but it gets pretty
    bulky and bootloaders are often pretty constrained.

dtb overlays were invented to address these, though I don't know that
difference between the two motivations was particularly well
understood at the time.

dtb overlays were pretty much a direct transcription of dts overlay
semantics to dtb and runtime.  The use of "in band" encoding (special
properties) is kind of inelegant, but it worked well enough to become
popular.

This works reasonably well for use case (2).  That makes sense: the
semantics are based on dts overlays which were designed for use case
(2).

But, dtb overlays have always been a nasty hack for case (1), even
though that's become the predominant one.  Overlays can change
anywhere in the tree, whether or not that's physically sensible via
the hat connector (or whatever).  The semantics of applying an overlay
lose information (by design), which means that removing an overlay for
hot unplug is a complete mess.  The way we encode the information
necessary to apply an overlay is kind of messy and inconsistent.

Still, they worked well enough for (1) that they've become widespread.
Hervé Condina is now working on a new scheme (dubbed "addons") which
is actually designed for case (1), and should suit it much better.
It's a fair way off being ready for use, but it's a good development.

> Many Android phones attempt to use overlays for case #2 by describing the SoC
> as the base devicetree and then describing boards in overlays. However,
> this solution doesn't get as much savings as it could since there is still
> lots of duplication between the overlays. Upstream reception towards
> structuring boards this way has also been lukewarm [1].
> 
> This patch series introduces a tool to help make case #2 a reality. The
> idea (suggested by Konrad Dybcio at LPC) is to continue to represent all the
> boards in a family as completely separate device trees in source code.
> Then, after we've done building everything, have a tool to automatically
> factor things to a base devicetree and a set of overlays.
> 
> This tool has quite a few advantages over other solutions:
> 1. No change to the way device trees are stored / arranged upstream.
> 2. No need to deal with some of the awkwardness / limitations of overlays.
> 3. Better savings than just a simple partitioning, like between SoC and
>    boards.

I like this idea.  It separates the case where the impetus is just to
save binary size from the case where there's a logical separation
between the pieces.  It decouples the generation of dtb overlays from
their expression as dts overlays (where you want a logical separation
for human readability).

> I've used this tool to reduce sc7180-trogdor device tree files (which
> are stored lz4 compressed) from 1.71 MB to 213 KB, a 87.5% savings.

Nice.  I suspect this gap would be substantially smaller with a large
window compressor.  The trick here is that the factoring is finding
long distance redundancies which lz4 won't be able to see.

> NOTE: on Chromebooks like sc7180-trogdor, using overlays like this is
> an optimization. On Android phones like Pixel 10, the bootloader logic
> bakes in the need for overlays. Specifically, if we want to be able to
> build a single image that supports multiple boards and have the bootloader
> auto-construct the right devicetree, we need the boards to be represented
> by overlays. This fdtfactor tool can make that a reality.
> 
> The fdtfactor tool can either work directly on a set of compiled devicetree
> files or it can work on FIT (flattened image tree) file. The idea is that
> you'd create the FIT file containing all of the duplicated devicetree
> files and then fdtfactor would "optimize" them.
> 
> I'll freely admit that the patches in this series were pretty heavily vibe
> coded. I'll not pretend that I have a deep understanding of the intracicies
> of how overlays are implemented. That being said:
> * I've spent quite a bit of time reviewing the code from a high-level
>   perspective: confirming that functions are well documented, well structured,
>   and make sense.
> * I believe that what the tool is doing and how the tool is used make sense.
> * I have had quite a number of internal "the AI reviews the AI" rounds to
>   the point of diminishing returns.
> At this point, I'm not embarassed to put my name behind this. While I won't
> promise that there is zero cases of the AI doing something really stupid,
> I'd be willing to bet that the end result is better (and cleaner / better
> structured) than what I would have coded up if I had spend a few weeks doing
> this on my own.

Thanks for that clear and helpful context.  My views on AI are
complicated, leaning negative, but I do think it can be a useful tool
and have no plans to reject patches purely because they're AI
assisted.  A statement like that goes a long way to convincing me that
I'm not about to waste my effort reviewing slop.  c.f. my rather
unfriendly response[2] to a recent very sloppish bug report which
vastly overestimated the impact of the bugs in question.

[2] https://lore.kernel.org/devicetree-compiler/as8RYmIp0pDCplpu@gractus.seuss/

Understand that all dtc maintainership is unpaid volunteer time for
me, so I'm less responsive about reviews than would be ideal.  I am
gradually chiping away at the backlog, though.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]