[PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays

[Douglas Anderson <dianders@chromium.org>]

Devicetree overlays allow you to have a "base" compiled devicetree and then
to apply an "overlay" to it to add or modify nodes. Devicetree overlays have
at least two purposes:

1. Supporting add-on boards (like "hats" on a dev board). In this case the
   main board is described as a base devicetree and the add-on board is an
   overlay. In this case there is a clear separation between the base and the
   add-on board and the base/overlay separation models reality nicely.

2. Supporting a "family" of similar boards in a space-efficient way. In
   upstream Linux, there are 47 distinct sc7180-trogdor boards supported.
   These board are all highly similar, but we compile (and ship) a separate
   devicetree for each. Devicetree overlays can make this much more efficient
   by factoring the common bits to the base devicetree and having each
   of the 47 boards represented by a small devicetree overlay.

Up to this point, the usage of overlays in the upstream devicetree repo has
mostly been for case #1.

Many Android phones attempt to use overlays for case #2 by describing the SoC
as the base devicetree and then describing boards in overlays. However,
this solution doesn't get as much savings as it could since there is still
lots of duplication between the overlays. Upstream reception towards
structuring boards this way has also been lukewarm [1].

This patch series introduces a tool to help make case #2 a reality. The
idea (suggested by Konrad Dybcio at LPC) is to continue to represent all the
boards in a family as completely separate device trees in source code.
Then, after we've done building everything, have a tool to automatically
factor things to a base devicetree and a set of overlays.

This tool has quite a few advantages over other solutions:
1. No change to the way device trees are stored / arranged upstream.
2. No need to deal with some of the awkwardness / limitations of overlays.
3. Better savings than just a simple partitioning, like between SoC and
   boards.

I've used this tool to reduce sc7180-trogdor device tree files (which
are stored lz4 compressed) from 1.71 MB to 213 KB, a 87.5% savings.

NOTE: on Chromebooks like sc7180-trogdor, using overlays like this is
an optimization. On Android phones like Pixel 10, the bootloader logic
bakes in the need for overlays. Specifically, if we want to be able to
build a single image that supports multiple boards and have the bootloader
auto-construct the right devicetree, we need the boards to be represented
by overlays. This fdtfactor tool can make that a reality.

The fdtfactor tool can either work directly on a set of compiled devicetree
files or it can work on FIT (flattened image tree) file. The idea is that
you'd create the FIT file containing all of the duplicated devicetree
files and then fdtfactor would "optimize" them.

I'll freely admit that the patches in this series were pretty heavily vibe
coded. I'll not pretend that I have a deep understanding of the intracicies
of how overlays are implemented. That being said:
* I've spent quite a bit of time reviewing the code from a high-level
  perspective: confirming that functions are well documented, well structured,
  and make sense.
* I believe that what the tool is doing and how the tool is used make sense.
* I have had quite a number of internal "the AI reviews the AI" rounds to
  the point of diminishing returns.
At this point, I'm not embarassed to put my name behind this. While I won't
promise that there is zero cases of the AI doing something really stupid,
I'd be willing to bet that the end result is better (and cleaner / better
structured) than what I would have coded up if I had spend a few weeks doing
this on my own.

[1] https://lore.kernel.org/all/20251114-elite-refined-yak-bf9e64@kuoka/


Douglas Anderson (4):
  tests: Fix Python test environment setup in meson.build
  fdtfactor: Add tool to split DTBs into a base DTB and overlays
  fdtfactor: Support compressed DTB inputs and outputs
  fdtfactor: Add compiled FIT image factoring support

 Makefile                   |    2 +-
 README.md                  |   75 ++
 fdtfactor                  | 2109 ++++++++++++++++++++++++++++++++++++
 meson.build                |    1 +
 tests/Makefile.tests       |    5 +-
 tests/fdtfactor_board1.dts |   69 ++
 tests/fdtfactor_board2.dts |   73 ++
 tests/fdtfactor_board3.dts |   96 ++
 tests/fdtfactor_other1.dts |   69 ++
 tests/fdtfactor_other2.dts |   69 ++
 tests/fdtfactor_test.its   |  109 ++
 tests/meson.build          |    4 +-
 tests/run_tests.sh         |  156 ++-
 tests/testutils.sh         |    1 +
 14 files changed, 2833 insertions(+), 5 deletions(-)
 create mode 100755 fdtfactor
 create mode 100644 tests/fdtfactor_board1.dts
 create mode 100644 tests/fdtfactor_board2.dts
 create mode 100644 tests/fdtfactor_board3.dts
 create mode 100644 tests/fdtfactor_other1.dts
 create mode 100644 tests/fdtfactor_other2.dts
 create mode 100644 tests/fdtfactor_test.its

-- 
2.56.0.176.g3575e65682-goog