* [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays
@ 2026-10-15 18:21 Douglas Anderson
2026-10-15 18:21 ` [PATCH 1/4] tests: Fix Python test environment setup in meson.build Douglas Anderson
` (4 more replies)
0 siblings, 5 replies; 7+ messages in thread
From: Douglas Anderson @ 2026-10-15 18:21 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Chen-Yu Tsai, linux-kernel, Devicetree, Brian Norris,
Tudor Ambarus, André Draszik, Julius Werner,
Saravana Kannan, Peter Griffin, Douglas Anderson
Devicetree overlays allow you to have a "base" compiled devicetree and then
to apply an "overlay" to it to add or modify nodes. Devicetree overlays have
at least two purposes:
1. Supporting add-on boards (like "hats" on a dev board). In this case the
main board is described as a base devicetree and the add-on board is an
overlay. In this case there is a clear separation between the base and the
add-on board and the base/overlay separation models reality nicely.
2. Supporting a "family" of similar boards in a space-efficient way. In
upstream Linux, there are 47 distinct sc7180-trogdor boards supported.
These board are all highly similar, but we compile (and ship) a separate
devicetree for each. Devicetree overlays can make this much more efficient
by factoring the common bits to the base devicetree and having each
of the 47 boards represented by a small devicetree overlay.
Up to this point, the usage of overlays in the upstream devicetree repo has
mostly been for case #1.
Many Android phones attempt to use overlays for case #2 by describing the SoC
as the base devicetree and then describing boards in overlays. However,
this solution doesn't get as much savings as it could since there is still
lots of duplication between the overlays. Upstream reception towards
structuring boards this way has also been lukewarm [1].
This patch series introduces a tool to help make case #2 a reality. The
idea (suggested by Konrad Dybcio at LPC) is to continue to represent all the
boards in a family as completely separate device trees in source code.
Then, after we've done building everything, have a tool to automatically
factor things to a base devicetree and a set of overlays.
This tool has quite a few advantages over other solutions:
1. No change to the way device trees are stored / arranged upstream.
2. No need to deal with some of the awkwardness / limitations of overlays.
3. Better savings than just a simple partitioning, like between SoC and
boards.
I've used this tool to reduce sc7180-trogdor device tree files (which
are stored lz4 compressed) from 1.71 MB to 213 KB, a 87.5% savings.
NOTE: on Chromebooks like sc7180-trogdor, using overlays like this is
an optimization. On Android phones like Pixel 10, the bootloader logic
bakes in the need for overlays. Specifically, if we want to be able to
build a single image that supports multiple boards and have the bootloader
auto-construct the right devicetree, we need the boards to be represented
by overlays. This fdtfactor tool can make that a reality.
The fdtfactor tool can either work directly on a set of compiled devicetree
files or it can work on FIT (flattened image tree) file. The idea is that
you'd create the FIT file containing all of the duplicated devicetree
files and then fdtfactor would "optimize" them.
I'll freely admit that the patches in this series were pretty heavily vibe
coded. I'll not pretend that I have a deep understanding of the intracicies
of how overlays are implemented. That being said:
* I've spent quite a bit of time reviewing the code from a high-level
perspective: confirming that functions are well documented, well structured,
and make sense.
* I believe that what the tool is doing and how the tool is used make sense.
* I have had quite a number of internal "the AI reviews the AI" rounds to
the point of diminishing returns.
At this point, I'm not embarassed to put my name behind this. While I won't
promise that there is zero cases of the AI doing something really stupid,
I'd be willing to bet that the end result is better (and cleaner / better
structured) than what I would have coded up if I had spend a few weeks doing
this on my own.
[1] https://lore.kernel.org/all/20251114-elite-refined-yak-bf9e64@kuoka/
Douglas Anderson (4):
tests: Fix Python test environment setup in meson.build
fdtfactor: Add tool to split DTBs into a base DTB and overlays
fdtfactor: Support compressed DTB inputs and outputs
fdtfactor: Add compiled FIT image factoring support
Makefile | 2 +-
README.md | 75 ++
fdtfactor | 2109 ++++++++++++++++++++++++++++++++++++
meson.build | 1 +
tests/Makefile.tests | 5 +-
tests/fdtfactor_board1.dts | 69 ++
tests/fdtfactor_board2.dts | 73 ++
tests/fdtfactor_board3.dts | 96 ++
tests/fdtfactor_other1.dts | 69 ++
tests/fdtfactor_other2.dts | 69 ++
tests/fdtfactor_test.its | 109 ++
tests/meson.build | 4 +-
tests/run_tests.sh | 156 ++-
tests/testutils.sh | 1 +
14 files changed, 2833 insertions(+), 5 deletions(-)
create mode 100755 fdtfactor
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/4] tests: Fix Python test environment setup in meson.build
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
@ 2026-10-15 18:21 ` Douglas Anderson
2026-10-23 4:51 ` David Gibson
2026-10-15 18:21 ` [PATCH 2/4] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
` (3 subsequent siblings)
4 siblings, 1 reply; 7+ messages in thread
From: Douglas Anderson @ 2026-10-15 18:21 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Chen-Yu Tsai, linux-kernel, Devicetree, Brian Norris,
Tudor Ambarus, André Draszik, Julius Werner,
Saravana Kannan, Peter Griffin, Douglas Anderson
Commit ecb21febfdd3 ("meson: port python bindings to build natively via
meson and meson-python") moved the generated `libfdt.py` and `_libfdt`
extension module into `<build_root>/pylibfdt`, leaving
`<source_root>/pylibfdt` with only the source files.
`tests/meson.build` was still setting `PYTHONPATH` to
`meson.project_source_root() / 'pylibfdt'`, which only went unnoticed
because `tests/pylibfdt_tests.py` manually inserts `../pylibfdt` into
`sys.path`. It also checked `if not py.found()` instead of
`if not pylibfdt_enabled` when setting `NO_PYTHON`.
Update `tests/meson.build` to check `pylibfdt_enabled` and set
`PYTHONPATH` to `meson.project_build_root() / 'pylibfdt'` so Python
tools invoked by `run_tests.sh` can import `libfdt` directly.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
tests/meson.build | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tests/meson.build b/tests/meson.build
index c6ac2d7f8ec4..c25500606059 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -114,12 +114,12 @@ endforeach
run_tests = find_program('run_tests.sh')
env = []
-if not py.found()
+if not pylibfdt_enabled
env += 'NO_PYTHON=1'
else
env += [
'PYTHON=' + py.full_path(),
- 'PYTHONPATH=' + meson.project_source_root() / 'pylibfdt',
+ 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt',
'NO_PYTHON=0',
]
endif
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/4] fdtfactor: Add tool to split DTBs into a base DTB and overlays
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-10-15 18:21 ` [PATCH 1/4] tests: Fix Python test environment setup in meson.build Douglas Anderson
@ 2026-10-15 18:21 ` Douglas Anderson
2026-10-15 18:21 ` [PATCH 3/4] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
` (2 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Douglas Anderson @ 2026-10-15 18:21 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Chen-Yu Tsai, linux-kernel, Devicetree, Brian Norris,
Tudor Ambarus, André Draszik, Julius Werner,
Saravana Kannan, Peter Griffin, Douglas Anderson
When shipping a single firmware or kernel image that supports many
closely related board variants or SKUs, bundling a full compiled .dtb
for every variant results in substantial duplication.
Add `fdtfactor`, a Python tool built on `pylibfdt` that takes a set of
compiled device tree blobs (.dtb) compiled with `-@ -L` (`/__symbols__`
and `/__local_fixups__`) and splits them into a single common
`base.dtb` and per-board `.dtbo` overlays:
- Common nodes present across all input DTBs are placed in `base.dtb`,
with each common property assigned the value shared by at least two
boards that maximizes byte savings across overlays.
- Using `/__local_fixups__`, phandle references inside property values
are normalized to target node paths so common properties referencing
common nodes match even when numeric phandles differ across input
DTBs.
- Any common node that carries a phandle in any input DTB receives its
phandle in `base.dtb` so overlays only emit `phandle` properties on
newly added subnodes, matching the requirements of `libfdt`, Linux's
`drivers/of/overlay.c`, and coreboot's `dt_apply_overlay()`.
On a set of 47 Qualcomm SC7180 Trogdor Chromebook board DTBs,
`fdtfactor -S` reduces total uncompressed size from 7.69 MB to 634 KB
(91.8% savings).
Add 3-board test cases covering default mode and `--strip-local-fixups`
to the `fdtoverlay` test suite.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Makefile | 2 +-
README.md | 58 ++
fdtfactor | 1218 ++++++++++++++++++++++++++++++++++++
meson.build | 1 +
tests/Makefile.tests | 2 +-
tests/fdtfactor_board1.dts | 69 ++
tests/fdtfactor_board2.dts | 73 +++
tests/fdtfactor_board3.dts | 96 +++
tests/run_tests.sh | 61 +-
tests/testutils.sh | 1 +
10 files changed, 1578 insertions(+), 3 deletions(-)
create mode 100755 fdtfactor
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
diff --git a/Makefile b/Makefile
index 9110d17796b0..4df5bbda9ce2 100644
--- a/Makefile
+++ b/Makefile
@@ -165,7 +165,7 @@ BIN += fdtget
BIN += fdtput
BIN += fdtoverlay
-SCRIPTS = dtdiff
+SCRIPTS = dtdiff fdtfactor
all: $(BIN) libfdt
diff --git a/README.md b/README.md
index 8f1a5d2a4d67..e2ff2374b01d 100644
--- a/README.md
+++ b/README.md
@@ -91,6 +91,64 @@ $ make NO_PYTHON=1
More work remains to support all of libfdt, including access to numeric
values.
+## fdtfactor
+
+`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
+compiled device tree blobs (`.dtb`) and splits them into a single common
+`base.dtb` and per-board `.dtbo` overlays so that applying a board's
+`.dtbo` to `base.dtb` recreates the original board device tree while
+reducing total storage size.
+
+Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
+and `/__local_fixups__` (used to identify phandle references inside
+property values) are present.
+
+Basic usage:
+
+```
+$ fdtfactor -b base.dtb -o out_dir board1.dtb board2.dtb board3.dtb
+```
+
+Options:
+
+* `-b`, `--base-dtb`: Output path for the common base `.dtb` (placed in
+ `--outdir` if specified without a directory component).
+* `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for
+ `--base-dtb` when given without a directory component).
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ `base.dtb` and final reconstructed device trees. When `-L` is passed to
+ `dtc` solely so `fdtfactor` can locate phandle cells at build time, the
+ original DTB's `/__local_fixups__` subtree is not needed at runtime by
+ either the overlay loader (`fdt_overlay_apply()`) or the OS kernel;
+ passing `-S` strips it to reduce the size of both `base.dtb` and the
+ `.dtbo` overlays. Omit `-S` if downstream tooling or verification
+ expects applying a board's `.dtbo` to `base.dtb` to reproduce the
+ original `.dtb`'s `/__local_fixups__` subtree verbatim.
+* `--verify`: Verify that applying each generated `.dtbo` to `base.dtb`
+ with `libfdt` recreates the original `.dtb`.
+* `-v`, `--verbose`: Print size savings summary.
+
+### TODO
+
+* **Explore omitting `/__fixups__` and `/__symbols__` from overlays by
+ hardcoding base DT phandles**: Normally a standalone `.dtbo` stores
+ references to base tree nodes as `0xffffffff` with string entries in
+ `/__fixups__` that the overlay loader resolves at runtime against
+ `base.dtb`'s `/__symbols__`. Because `fdtfactor` generates `base.dtb`
+ and all `.dtbo` overlays together, it already knows the exact numeric
+ phandle values assigned to every node in `base.dtb`. Neither `libfdt` nor
+ coreboot modifies property cells that are not listed in `/__fixups__` or
+ `/__local_fixups__`. Writing the base phandle integers directly into
+ overlay properties would allow omitting `/__fixups__` from the `.dtbo`
+ files (and omitting `/__symbols__` when runtime symbol lookup is not
+ needed), further reducing overlay size and speeding up boot-time overlay
+ application.
+* **Multi-level overlays**: Support hierarchical overlay layers (for
+ example, SoC base -> board family intermediate overlay -> SKU leaf
+ overlay) when multiple boards share large sub-trees not present across
+ all boards.
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
diff --git a/fdtfactor b/fdtfactor
new file mode 100755
index 000000000000..377f499bea50
--- /dev/null
+++ b/fdtfactor
@@ -0,0 +1,1218 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""Split compiled device tree blobs (.dtb) into a base .dtb and .dtbo overlays.
+
+Given a set of compiled device tree blobs, fdtfactor extracts all common nodes
+and properties into a single base .dtb file and writes the per-board
+differences into .dtbo overlay files. Applying an overlay to the base .dtb
+using fdtoverlay or libfdt's fdt_overlay_apply() recreates the corresponding
+original device tree.
+"""
+
+import argparse
+from collections import Counter
+import os
+import struct
+import sys
+
+import libfdt
+
+
+def split_dtb_filename(path):
+ """Split a DTB path into its directory and filename stem.
+
+ Example:
+ '/dir/board.dtb' -> ('/dir', 'board')
+
+ Args:
+ path: File path to a .dtb or .dtbo file.
+
+ Returns:
+ Tuple of (directory, stem) where `stem` has any trailing `.dtb` or
+ `.dtbo` suffix removed.
+ """
+ directory, filename = os.path.split(path)
+ stem, sep, ext = filename.rpartition('.')
+ if not stem or not sep or ext not in ('dtb', 'dtbo'):
+ stem = filename
+ return directory, stem
+
+
+class NodeTree:
+ """In-memory tree of nodes and properties for sequential FDT creation."""
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+
+ def get_node(self, path):
+ """Return the descendant NodeTree at `path`, creating missing nodes.
+
+ Looks up a child or deeper descendant `NodeTree` by following the
+ slash-separated node names in `path` starting from `self` (leading and
+ trailing `'/'` characters are ignored, so `'/soc/i2c@4000'` and
+ `'soc/i2c@4000'` behave identically). If any node along `path` does not
+ exist yet in `self.children`, a new empty `NodeTree` is created for it
+ automatically (similar to `mkdir -p`). Passing `''` or `'/'` returns
+ `self`.
+
+ Example:
+ root = NodeTree()
+ # The following automatically creates root.children['soc'] and
+ # root.children['soc'].children['i2c@4000'], and returns the
+ # 'i2c@4000' NodeTree.
+ i2c = root.get_node('/soc/i2c@4000')
+ i2c.props['status'] = b'okay\\x00'
+
+ Args:
+ path: Slash-separated node path starting from `self` (leading `'/'`
+ is optional and ignored, e.g. `'/soc/i2c@4000'` or
+ `'soc/i2c@4000'`).
+
+ Returns:
+ The `NodeTree` instance at `path`.
+ """
+ if path == '/' or not path:
+ return self
+ curr = self
+ for part in path.strip('/').split('/'):
+ if part not in curr.children:
+ curr.children[part] = NodeTree()
+ curr = curr.children[part]
+ return curr
+
+ def _write_to_sw(self, sw, name=''):
+ """Recursively write this NodeTree and its children into an FdtSw.
+
+ Args:
+ sw: A `libfdt.FdtSw` sequential-write instance.
+ name: Node name for this subtree (empty string for the root node).
+ """
+ sw.begin_node(name)
+ for prop_name, val in self.props.items():
+ sw.property(prop_name, bytes(val))
+ for child_name, child_node in self.children.items():
+ child_node._write_to_sw(sw, child_name)
+ sw.end_node()
+
+ def to_fdt_blob(self, mem_rsv=(), boot_cpuid_phys=0, size_hint=65536):
+ """Serialize this root NodeTree into a packed FDT byte string.
+
+ Args:
+ mem_rsv: Iterable of (address, size) tuples for the FDT memory
+ reservation block.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+ size_hint: Initial buffer size hint in bytes for `libfdt.FdtSw`.
+
+ Returns:
+ Serialized FDT blob as `bytes`.
+ """
+ sw = libfdt.FdtSw(size_hint)
+ for addr, size in mem_rsv:
+ sw.add_reservemap_entry(addr, size)
+ sw.finish_reservemap()
+ self._write_to_sw(sw, '')
+ # Unlike other FdtSw methods, as_fdt() does not call check_space()
+ # when fdt_finish() appends the 4-byte FDT_END token. If fewer than
+ # 4 bytes remain in the buffer, resize once and retry.
+ try:
+ fdt = sw.as_fdt()
+ except libfdt.FdtException as exc:
+ if not sw.check_space(exc.err):
+ raise
+ fdt = sw.as_fdt()
+ libfdt.fdt_set_boot_cpuid_phys(fdt._fdt, boot_cpuid_phys)
+ fdt.pack()
+ return bytes(fdt.as_bytearray())
+
+
+def _open_fdt_ro(raw_bytes, filename):
+ """Validate and wrap an uncompressed FDT blob in `libfdt.FdtRo`.
+
+ Args:
+ raw_bytes: Uncompressed FDT binary contents as `bytes`.
+ filename: Display name or file path for error reporting.
+
+ Returns:
+ A validated `libfdt.FdtRo` instance.
+
+ Raises:
+ ValueError: If `raw_bytes` fails FDT header or full-structure checks.
+ """
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+ return fdt
+
+
+def _is_dtbo_overlay(fdt_ro):
+ """Return True if `fdt_ro` is a `.dtbo` overlay rather than a base DTB.
+
+ Args:
+ fdt_ro: A `libfdt.FdtRo` instance wrapping an uncompressed FDT blob.
+
+ Returns:
+ True if the FDT has `/__fixups__` or a top-level fragment node
+ containing an `__overlay__` subnode.
+ """
+ if fdt_ro.subnode_offset(0, '__fixups__', quiet=libfdt.QUIET_NOTFOUND) >= 0:
+ return True
+ soff = fdt_ro.first_subnode(0, quiet=libfdt.QUIET_NOTFOUND)
+ while soff >= 0:
+ if fdt_ro.subnode_offset(
+ soff, '__overlay__', quiet=libfdt.QUIET_NOTFOUND
+ ) >= 0:
+ return True
+ soff = fdt_ro.next_subnode(soff, quiet=libfdt.QUIET_NOTFOUND)
+ return False
+
+
+class ParsedDtb:
+ """Parsed representation of an input .dtb blob."""
+
+ def __init__(self, filename, raw_bytes):
+ """Parse an uncompressed FDT blob into nodes, symbols, and fixups.
+
+ Args:
+ filename: Display name or file path for error reporting.
+ raw_bytes: Uncompressed FDT binary contents as `bytes`.
+ """
+ self.filename = filename
+ self.raw_bytes = raw_bytes
+ fdt = _open_fdt_ro(raw_bytes, filename)
+ if _is_dtbo_overlay(fdt):
+ raise ValueError(
+ f'{filename}: Input is already a DTB overlay (.dtbo)'
+ )
+
+ # Extract FDT header metadata (boot_cpuid_phys and /memreserve/ table)
+ # stored outside the DT node/property tree; see _validate_dtb_headers()
+ # for why these require special handling.
+ self.boot_cpuid_phys = fdt.boot_cpuid_phys()
+ self.mem_rsv = [
+ tuple(fdt.get_mem_rsv(i)) for i in range(fdt.num_mem_rsv())
+ ]
+
+ self.has_symbols = False
+ self.symbols = {}
+ self.has_local_fixups = False
+ self.local_fixups = {}
+
+ self.nodes = {}
+ self.phandles = {}
+ self.has_phandle_prop = {}
+ self.has_linux_phandle_prop = {}
+ self.phandle_to_path = {}
+ self.sym_props = {}
+
+ self._parse_tree(fdt)
+
+ @classmethod
+ def from_file(cls, path):
+ """Read and parse a .dtb file from disk.
+
+ Args:
+ path: Path to the input .dtb file.
+
+ Returns:
+ A new `ParsedDtb` instance.
+ """
+ with open(path, 'rb') as f:
+ raw_bytes = f.read()
+ return cls(path, raw_bytes)
+
+ def _parse_tree(self, fdt):
+ """Walk the FDT and extract nodes, properties, symbols, and fixups.
+
+ Args:
+ fdt: A `libfdt.FdtRo` instance wrapping the input blob.
+ """
+ self._walk_regular(fdt, 0, '/')
+
+ sym_off = fdt.subnode_offset(0, '__symbols__',
+ quiet=libfdt.QUIET_NOTFOUND)
+ if sym_off >= 0:
+ self.has_symbols = True
+ poff = fdt.first_property_offset(sym_off,
+ quiet=libfdt.QUIET_NOTFOUND)
+ while poff >= 0:
+ prop = fdt.get_property_by_offset(poff)
+ if prop.name not in ('phandle', 'linux,phandle'):
+ target_path = prop.as_str() if prop else ''
+ if target_path not in self.nodes:
+ raise ValueError(
+ f'{self.filename}: symbol {prop.name!r} references '
+ f'missing node {target_path!r}'
+ )
+ self.symbols[prop.name] = target_path
+ poff = fdt.next_property_offset(poff,
+ quiet=libfdt.QUIET_NOTFOUND)
+
+ lf_off = fdt.subnode_offset(0, '__local_fixups__',
+ quiet=libfdt.QUIET_NOTFOUND)
+ if lf_off >= 0:
+ self.has_local_fixups = True
+ self._walk_local_fixups(fdt, lf_off, '/')
+
+ def _walk_regular(self, fdt, nodeoffset, path):
+ """Recursively collect regular nodes and their properties.
+
+ Args:
+ fdt: A `libfdt.FdtRo` instance.
+ nodeoffset: Offset of the current node in `fdt`.
+ path: Full device tree path of the current node (e.g. '/soc').
+ """
+ props = {}
+ ph_val = None
+ has_ph = False
+ has_linux_ph = False
+
+ poff = fdt.first_property_offset(nodeoffset,
+ quiet=libfdt.QUIET_NOTFOUND)
+ while poff >= 0:
+ prop = fdt.get_property_by_offset(poff)
+ raw = bytes(prop)
+ if prop.name in ('phandle', 'linux,phandle'):
+ if len(raw) != 4:
+ raise ValueError(
+ f'{self.filename}: {path}: {prop.name} length '
+ f'{len(raw)} != 4'
+ )
+ val = struct.unpack('>I', raw)[0]
+ if ph_val is not None and ph_val != val:
+ raise ValueError(
+ f'{self.filename}: {path}: mismatched phandle '
+ f'({ph_val:#x} vs {val:#x})'
+ )
+ ph_val = val
+ if prop.name == 'phandle':
+ has_ph = True
+ else:
+ has_linux_ph = True
+ else:
+ props[prop.name] = raw
+ poff = fdt.next_property_offset(poff, quiet=libfdt.QUIET_NOTFOUND)
+
+ self.nodes[path] = props
+ self.phandles[path] = ph_val
+ self.has_phandle_prop[path] = has_ph
+ self.has_linux_phandle_prop[path] = has_linux_ph
+ if ph_val is not None:
+ if ph_val in (0, 0xffffffff):
+ raise ValueError(
+ f'{self.filename}: invalid phandle {ph_val:#x} on {path}'
+ )
+ if ph_val in self.phandle_to_path:
+ raise ValueError(
+ f'{self.filename}: duplicate phandle {ph_val:#x} on '
+ f'{self.phandle_to_path[ph_val]} and {path}'
+ )
+ self.phandle_to_path[ph_val] = path
+
+ soff = fdt.first_subnode(nodeoffset, quiet=libfdt.QUIET_NOTFOUND)
+ while soff >= 0:
+ name = fdt.get_name(soff)
+ if path == '/' and name in ('__symbols__', '__local_fixups__'):
+ soff = fdt.next_subnode(soff, quiet=libfdt.QUIET_NOTFOUND)
+ continue
+ child_path = '/' + name if path == '/' else f'{path}/{name}'
+ self._walk_regular(fdt, soff, child_path)
+ soff = fdt.next_subnode(soff, quiet=libfdt.QUIET_NOTFOUND)
+
+ def _walk_local_fixups(self, fdt, nodeoffset, rel_path):
+ """Recursively collect phandle byte offsets from `/__local_fixups__`.
+
+ Args:
+ fdt: A `libfdt.FdtRo` instance.
+ nodeoffset: Offset of the current node inside `/__local_fixups__`.
+ rel_path: Corresponding target node path in the device tree.
+
+ Raises:
+ ValueError: If a fixup property length is not a multiple of 4.
+ """
+ poff = fdt.first_property_offset(nodeoffset,
+ quiet=libfdt.QUIET_NOTFOUND)
+ while poff >= 0:
+ prop = fdt.get_property_by_offset(poff)
+ raw = bytes(prop)
+ if len(raw) % 4 != 0:
+ raise ValueError(
+ f'{self.filename}: malformed __local_fixups__ property '
+ f'{rel_path}:{prop.name} (length {len(raw)})'
+ )
+ count = len(raw) // 4
+ offsets = struct.unpack(f'>{count}I', raw)
+ self.local_fixups[(rel_path, prop.name)] = offsets
+ poff = fdt.next_property_offset(poff, quiet=libfdt.QUIET_NOTFOUND)
+
+ soff = fdt.first_subnode(nodeoffset, quiet=libfdt.QUIET_NOTFOUND)
+ while soff >= 0:
+ name = fdt.get_name(soff)
+ child_path = '/' + name if rel_path == '/' else f'{rel_path}/{name}'
+ self._walk_local_fixups(fdt, soff, child_path)
+ soff = fdt.next_subnode(soff, quiet=libfdt.QUIET_NOTFOUND)
+
+ def build_symbolic_props(self):
+ """Build normalized property values for comparison across DTBs.
+
+ Populates `self.sym_props[path][prop_name]` with a tuple:
+ `(masked_bytes, ((offset, target_node_path), ...))`
+ where phandle cells listed in `self.local_fixups` are zeroed in
+ `masked_bytes` and recorded symbolically by their target node path.
+
+ Raises:
+ ValueError: If a fixup references a missing property, its offset is
+ out of range, or it references an unknown phandle value.
+ """
+ for path, prop_name in self.local_fixups:
+ if path not in self.nodes or prop_name not in self.nodes[path]:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing property '
+ f'{path}:{prop_name}'
+ )
+ for path, props in self.nodes.items():
+ node_sym_props = {}
+ for prop_name, raw in props.items():
+ offsets = self.local_fixups.get((path, prop_name), ())
+ if not offsets:
+ node_sym_props[prop_name] = (raw, ())
+ continue
+ masked = bytearray(raw)
+ targets = []
+ for off in offsets:
+ if off + 4 > len(raw):
+ raise ValueError(
+ f'{self.filename}: fixup offset {off} out of range '
+ f'on {path}:{prop_name} (len {len(raw)})'
+ )
+ ph = struct.unpack_from('>I', raw, off)[0]
+ target_path = self.phandle_to_path.get(ph)
+ if target_path is None:
+ raise ValueError(
+ f'{self.filename}: unknown phandle {ph:#x} at '
+ f'{path}:{prop_name}+{off}'
+ )
+ masked[off:off + 4] = b'\x00\x00\x00\x00'
+ targets.append((off, target_path))
+ node_sym_props[prop_name] = (bytes(masked), tuple(targets))
+ self.sym_props[path] = node_sym_props
+
+
+def select_best_candidate(candidates, byte_len_fn):
+ """Choose the value that maximizes space savings across overlays.
+
+ Only candidates shared by at least two input DTBs (count > 1) are eligible
+ for inclusion in the base DTB; if every candidate value is unique to a
+ single DTB, placing one in the base DTB would yield zero net size savings
+ (adding one copy to the base DTB to save one overlay copy).
+
+ Args:
+ candidates: Iterable of candidate values (in input DTB order).
+ byte_len_fn: Function returning the byte cost of a candidate value in
+ an overlay.
+
+ Returns:
+ The chosen candidate value, or None if no value appears more than once.
+ """
+ counts = Counter(candidates)
+ best_val = None
+ best_key = None
+ for val, count in counts.items():
+ if count <= 1:
+ continue
+ length = byte_len_fn(val)
+ # Maximize total bytes saved across overlays:
+ # (count - 1) * (length + 12)
+ key = ((count - 1) * (length + 12), count, length)
+ if best_key is None or key > best_key:
+ best_key = key
+ best_val = val
+ return best_val
+
+
+def _validate_dtb_headers(dtbs):
+ """Verify that FDT header fields and /memreserve/ entries match across DTBs.
+
+ `boot_cpuid_phys` and `/memreserve/` require special handling because:
+ * Unlike regular DT nodes (including `/reserved-memory`), they are stored
+ in the FDT binary header and memory reservation block rather than the
+ DT structure block.
+ * `fdt_overlay_apply()` only merges the `.dtbo` structure block into the
+ base DTB and ignores the `.dtbo`'s header and `/memreserve/` block.
+ * Therefore, every reconstructed board DTB inherits `boot_cpuid_phys` and
+ `/memreserve/` directly from the base DTB, so all DTBs sharing a base
+ DTB must have identical values.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` objects to validate.
+
+ Raises:
+ ValueError: If `boot_cpuid_phys` or `mem_rsv` differs between any two
+ input DTBs.
+ """
+ for d in dtbs[1:]:
+ if d.boot_cpuid_phys != dtbs[0].boot_cpuid_phys:
+ raise ValueError(
+ f'boot_cpuid_phys mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].boot_cpuid_phys}) and {d.filename} '
+ f'({d.boot_cpuid_phys})'
+ )
+ if d.mem_rsv != dtbs[0].mem_rsv:
+ raise ValueError(
+ f'/memreserve/ mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].mem_rsv}) and {d.filename} ({d.mem_rsv})'
+ )
+
+
+def _find_common_nodes_and_phandles(dtbs):
+ """Identify common node paths and assign sequential base DTB phandles.
+
+ Any common node that carries a phandle in any input DTB is assigned a
+ phandle in the base DTB so overlays never attempt to add a `phandle`
+ property to an already-existing base DTB node (which Linux and
+ coreboot/depthcharge disallow).
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` objects.
+
+ Returns:
+ Tuple of `(base_dtb_nodes, common_node_set, base_dtb_phandles)` where:
+ * `base_dtb_nodes`: Ordered list of node paths common to all `dtbs`.
+ * `common_node_set`: Set of node paths in `base_dtb_nodes`.
+ * `base_dtb_phandles`: Dict mapping node path to assigned integer
+ phandle in the base DTB.
+ """
+ common_node_set = set(dtbs[0].nodes.keys())
+ for d in dtbs[1:]:
+ common_node_set &= set(d.nodes.keys())
+ base_dtb_nodes = [p for p in dtbs[0].nodes.keys() if p in common_node_set]
+
+ base_dtb_phandles = {}
+ next_ph = 1
+ for path in base_dtb_nodes:
+ if any(d.phandles[path] is not None for d in dtbs):
+ base_dtb_phandles[path] = next_ph
+ next_ph += 1
+
+ return base_dtb_nodes, common_node_set, base_dtb_phandles
+
+
+def _find_common_symbols(dtbs, common_node_set, base_dtb_phandles):
+ """Find symbols common to all input DTBs that point to the same base node.
+
+ Only symbols that exist in all DTBs and resolve to the same common base
+ DTB node path are placed in the base DTB's `/__symbols__`, ensuring
+ overlays only add new symbols and never need to overwrite existing ones
+ (which coreboot/depthcharge merges with `upd=0`).
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` objects.
+ common_node_set: Set of node paths present in all input DTBs.
+ base_dtb_phandles: Dict mapping base DTB node paths to their assigned
+ integer phandles.
+
+ Returns:
+ Tuple of `(base_dtb_symbols, path_to_base_dtb_symbol)` where:
+ * `base_dtb_symbols`: Dict mapping symbol label to target node path.
+ * `path_to_base_dtb_symbol`: Dict mapping target node path to its
+ primary symbol label in `base_dtb_symbols`.
+ """
+ base_dtb_symbols = {}
+ path_to_base_dtb_symbol = {}
+
+ common_sym_names = set(dtbs[0].symbols.keys())
+ for d in dtbs[1:]:
+ common_sym_names &= set(d.symbols.keys())
+ for label in dtbs[0].symbols.keys():
+ if label not in common_sym_names:
+ continue
+ target_paths = {d.symbols[label] for d in dtbs}
+ if len(target_paths) == 1:
+ chosen_path = dtbs[0].symbols[label]
+ if chosen_path in common_node_set:
+ base_dtb_symbols[label] = chosen_path
+ if (chosen_path in base_dtb_phandles and
+ chosen_path not in path_to_base_dtb_symbol):
+ path_to_base_dtb_symbol[chosen_path] = label
+
+ return base_dtb_symbols, path_to_base_dtb_symbol
+
+
+def _select_base_dtb_properties(dtbs, base_dtb_nodes, base_dtb_phandles,
+ preserve_local_fixups):
+ """Select the common property values to place on each node in the base DTB.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` objects with `sym_props` populated.
+ base_dtb_nodes: Ordered list of node paths in the base DTB.
+ base_dtb_phandles: Dict mapping base DTB node paths to their assigned
+ integer phandles.
+ preserve_local_fixups: Whether `/__local_fixups__` from the input DTBs
+ must be preserved in the merged output.
+
+ Returns:
+ Tuple of `(base_dtb_props, base_dtb_fixup_offsets)` where:
+ * `base_dtb_props`: Dict mapping `path -> {prop_name: sym_val}` for
+ properties placed in the base DTB.
+ * `base_dtb_fixup_offsets`: Dict mapping `(path, prop_name)` to a tuple
+ of byte offsets for the base DTB's `/__local_fixups__`.
+ """
+ base_dtb_props = {}
+ base_dtb_fixup_offsets = {}
+
+ for path in base_dtb_nodes:
+ common_prop_names = set(dtbs[0].nodes[path].keys())
+ for d in dtbs[1:]:
+ common_prop_names &= set(d.nodes[path].keys())
+
+ node_base_props = {}
+ for prop_name in dtbs[0].nodes[path].keys():
+ if prop_name not in common_prop_names:
+ continue
+
+ all_have_prop_fixups = all(
+ bool(d.sym_props[path][prop_name][1]) for d in dtbs
+ )
+ eligible = []
+ for d in dtbs:
+ sym_val = d.sym_props[path][prop_name]
+ _, targets = sym_val
+ if targets:
+ # Overlays cannot delete a property from /__local_fixups__,
+ # so a phandle-bearing value can only go into base.dtb when
+ # every board also has phandle fixups on this property.
+ if preserve_local_fixups and not all_have_prop_fixups:
+ continue
+ if not all(
+ t_path in base_dtb_phandles for _, t_path in targets
+ ):
+ continue
+ eligible.append(sym_val)
+
+ fixup_entry_cost = len(path) + len(prop_name) + 28
+ chosen_val = select_best_candidate(
+ eligible,
+ lambda sv, fc=fixup_entry_cost: (
+ len(sv[0]) + len(sv[1]) * fc
+ ),
+ )
+ if chosen_val is not None:
+ node_base_props[prop_name] = chosen_val
+ if preserve_local_fixups and chosen_val[1]:
+ offs = tuple(off for off, _ in chosen_val[1])
+ base_dtb_fixup_offsets[(path, prop_name)] = offs
+
+ base_dtb_props[path] = node_base_props
+
+ return base_dtb_props, base_dtb_fixup_offsets
+
+
+def _build_base_dtb(dtbs, base_dtb_nodes, base_dtb_props, base_dtb_phandles,
+ base_dtb_symbols, base_dtb_fixup_offsets):
+ """Construct and serialize the common base DTB blob.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` objects.
+ base_dtb_nodes: Ordered list of node paths in the base DTB.
+ base_dtb_props: Dict mapping `path -> {prop_name: sym_val}`.
+ base_dtb_phandles: Dict mapping base DTB node paths to their assigned
+ integer phandles.
+ base_dtb_symbols: Dict mapping symbol label to target node path for the
+ base DTB's `/__symbols__`.
+ base_dtb_fixup_offsets: Dict mapping `(path, prop_name)` to phandle
+ byte offsets for the base DTB's `/__local_fixups__`.
+
+ Returns:
+ Serialized base DTB blob as `bytes`.
+ """
+ base_dtb_root = NodeTree()
+ for path in base_dtb_nodes:
+ node = base_dtb_root.get_node(path)
+ for prop_name, (masked_bytes, targets) in base_dtb_props[path].items():
+ if not targets:
+ node.props[prop_name] = masked_bytes
+ else:
+ val = bytearray(masked_bytes)
+ for off, t_path in targets:
+ struct.pack_into('>I', val, off, base_dtb_phandles[t_path])
+ node.props[prop_name] = bytes(val)
+
+ if path in base_dtb_phandles:
+ ph_bytes = struct.pack('>I', base_dtb_phandles[path])
+ if any(d.has_phandle_prop[path] for d in dtbs):
+ node.props['phandle'] = ph_bytes
+ if any(d.has_linux_phandle_prop[path] for d in dtbs):
+ node.props['linux,phandle'] = ph_bytes
+
+ if base_dtb_symbols:
+ sym_node = base_dtb_root.get_node('/__symbols__')
+ for label, target_path in base_dtb_symbols.items():
+ sym_node.props[label] = target_path.encode('utf-8') + b'\x00'
+
+ if base_dtb_fixup_offsets:
+ lf_root = base_dtb_root.get_node('/__local_fixups__')
+ for (path, prop_name), offs in base_dtb_fixup_offsets.items():
+ lf_node = lf_root.get_node(path)
+ lf_node.props[prop_name] = struct.pack(f'>{len(offs)}I', *offs)
+
+ return base_dtb_root.to_fdt_blob(
+ mem_rsv=dtbs[0].mem_rsv,
+ boot_cpuid_phys=dtbs[0].boot_cpuid_phys,
+ size_hint=max(len(dtbs[0].raw_bytes), 4096),
+ )
+
+
+def _assign_local_phandles(dtb, base_dtb_phandles):
+ """Assign sequential overlay-local phandles to board-specific nodes.
+
+ Args:
+ dtb: The board's `ParsedDtb`.
+ base_dtb_phandles: Dict mapping base DTB node paths to their phandles.
+
+ Returns:
+ Dict mapping board-specific node path to its overlay-local integer
+ phandle (`1, 2, ...`).
+ """
+ local_phandles = {}
+ next_local_ph = 1
+ for path in dtb.nodes.keys():
+ if dtb.phandles[path] is not None and path not in base_dtb_phandles:
+ local_phandles[path] = next_local_ph
+ next_local_ph += 1
+ return local_phandles
+
+
+def _encode_dtbo_property(path, prop_name, sym_val, base_dtb_phandles,
+ path_to_base_dtb_symbol, local_phandles, ext_fixups,
+ ov_local_fixups):
+ """Encode a property value for `/fragment@0/__overlay__` and record fixups.
+
+ Args:
+ path: Target node path in the device tree (e.g. '/soc/i2c@4000').
+ prop_name: Property name.
+ sym_val: Normalized `(masked_bytes, targets)` tuple for the property.
+ base_dtb_phandles: Dict mapping base DTB node paths to their phandles.
+ path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol
+ names in the base DTB's `/__symbols__`.
+ local_phandles: Dict mapping board-only node paths to overlay-local
+ phandles.
+ ext_fixups: Dict of `label -> [fixup_str, ...]` updated in-place for
+ references to base DTB symbols.
+ ov_local_fixups: Dict of `(path, prop_name) -> [offset, ...]` updated
+ in-place for references to overlay-local phandles.
+
+ Returns:
+ Encoded property value as `bytes`.
+ """
+ masked_bytes, targets = sym_val
+ if not targets:
+ return masked_bytes
+
+ val = bytearray(masked_bytes)
+ for off, t_path in targets:
+ if t_path in base_dtb_phandles:
+ sym_label = path_to_base_dtb_symbol.get(t_path)
+ if sym_label is not None:
+ struct.pack_into('>I', val, off, 0xffffffff)
+ ov_path = (
+ '/fragment@0/__overlay__'
+ if path == '/'
+ else f'/fragment@0/__overlay__{path}'
+ )
+ entry = f'{ov_path}:{prop_name}:{off}'
+ ext_fixups.setdefault(sym_label, []).append(entry)
+ else:
+ # Target is a common base DTB node without a symbol shared by
+ # all input DTBs (e.g. an anonymous &{/path} reference or a
+ # board-specific label); use its assigned base DTB phandle.
+ struct.pack_into('>I', val, off, base_dtb_phandles[t_path])
+ else:
+ struct.pack_into('>I', val, off, local_phandles[t_path])
+ ov_local_fixups.setdefault((path, prop_name), []).append(off)
+ return bytes(val)
+
+
+def _populate_dtbo_overlay_nodes(ov_root, dtb, common_node_set, base_dtb_props,
+ base_dtb_phandles, path_to_base_dtb_symbol,
+ local_phandles):
+ """Populate `/fragment@0/__overlay__` with board-specific nodes and props.
+
+ Args:
+ ov_root: The `NodeTree` for `/fragment@0/__overlay__`.
+ dtb: The board's `ParsedDtb`.
+ common_node_set: Set of node paths present in the base DTB.
+ base_dtb_props: Dict mapping `path -> {prop_name: sym_val}` in the
+ base DTB.
+ base_dtb_phandles: Dict mapping base DTB node paths to their phandles.
+ path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol
+ names in the base DTB's `/__symbols__`.
+ local_phandles: Dict mapping board-only node paths to overlay-local
+ phandles.
+
+ Returns:
+ Tuple of `(ext_fixups, ov_local_fixups)` collected across all overlay
+ properties.
+ """
+ ext_fixups = {}
+ ov_local_fixups = {}
+
+ for path, props in dtb.nodes.items():
+ in_base_dtb = path in common_node_set
+ base_node_props = base_dtb_props.get(path, {})
+
+ diff_props = [
+ (prop_name, dtb.sym_props[path][prop_name])
+ for prop_name in props.keys()
+ if (not in_base_dtb or
+ base_node_props.get(prop_name) !=
+ dtb.sym_props[path][prop_name])
+ ]
+
+ if not in_base_dtb or diff_props:
+ ov_node = ov_root.get_node(path)
+ for prop_name, sym_val in diff_props:
+ ov_node.props[prop_name] = _encode_dtbo_property(
+ path, prop_name, sym_val, base_dtb_phandles,
+ path_to_base_dtb_symbol, local_phandles, ext_fixups,
+ ov_local_fixups,
+ )
+
+ if path in local_phandles:
+ ph_bytes = struct.pack('>I', local_phandles[path])
+ if dtb.has_phandle_prop[path]:
+ ov_node.props['phandle'] = ph_bytes
+ if dtb.has_linux_phandle_prop[path]:
+ ov_node.props['linux,phandle'] = ph_bytes
+
+ return ext_fixups, ov_local_fixups
+
+
+def _populate_dtbo_metadata(dtbo_root, ov_root, dtb, base_dtb_symbols,
+ base_dtb_fixup_offsets, ext_fixups,
+ ov_local_fixups, preserve_local_fixups):
+ """Populate `/__symbols__`, `/__fixups__`, and `/__local_fixups__` on DTBO.
+
+ Args:
+ dtbo_root: Root `NodeTree` of the `.dtbo` overlay.
+ ov_root: The `NodeTree` for `/fragment@0/__overlay__`.
+ dtb: The board's `ParsedDtb`.
+ base_dtb_symbols: Dict of symbols already present in the base DTB.
+ base_dtb_fixup_offsets: Dict of `/__local_fixups__` offsets already
+ present in the base DTB.
+ ext_fixups: Dict of `label -> [fixup_str, ...]` referencing base DTB
+ symbols.
+ ov_local_fixups: Dict of `(path, prop_name) -> [offset, ...]`
+ referencing overlay-local phandles.
+ preserve_local_fixups: Whether to emit differing input
+ `/__local_fixups__` entries into
+ `/fragment@0/__overlay__/__local_fixups__`.
+ """
+ # If preserving input /__local_fixups__ in the merged tree, place any
+ # differing fixup offset lists under
+ # /fragment@0/__overlay__/__local_fixups__
+ if preserve_local_fixups:
+ for (path, prop_name), offs in dtb.local_fixups.items():
+ if base_dtb_fixup_offsets.get((path, prop_name)) != offs:
+ merged_lf_root = ov_root.get_node('/__local_fixups__')
+ merged_lf_node = merged_lf_root.get_node(path)
+ merged_lf_node.props[prop_name] = struct.pack(
+ f'>{len(offs)}I', *offs
+ )
+
+ # Emit overlay root /__symbols__ for new or modified symbols
+ diff_syms = [
+ (label, t_path)
+ for label, t_path in dtb.symbols.items()
+ if base_dtb_symbols.get(label) != t_path
+ ]
+ if diff_syms:
+ sym_node = dtbo_root.get_node('/__symbols__')
+ for label, t_path in diff_syms:
+ # Ensure target node exists in __overlay__ so dtc -I dtb -O dts
+ # on the standalone .dtbo does not warn about a missing node.
+ ov_root.get_node(t_path)
+ ov_target = (
+ '/fragment@0/__overlay__'
+ if t_path == '/'
+ else f'/fragment@0/__overlay__{t_path}'
+ )
+ sym_node.props[label] = ov_target.encode('utf-8') + b'\x00'
+
+ # Emit overlay root /__fixups__ for references to base DTB symbols
+ if ext_fixups:
+ fixups_node = dtbo_root.get_node('/__fixups__')
+ for label, entries in ext_fixups.items():
+ fixups_node.props[label] = (
+ '\x00'.join(entries).encode('utf-8') + b'\x00'
+ )
+
+ # Emit overlay root /__local_fixups__ for overlay-local phandles
+ if ov_local_fixups:
+ ov_lf_root = dtbo_root.get_node(
+ '/__local_fixups__/fragment@0/__overlay__'
+ )
+ for (path, prop_name), offs in ov_local_fixups.items():
+ lf_node = ov_lf_root.get_node(path)
+ lf_node.props[prop_name] = struct.pack(f'>{len(offs)}I', *offs)
+
+
+def _build_board_dtbo(dtb, common_node_set, base_dtb_props, base_dtb_phandles,
+ base_dtb_symbols, path_to_base_dtb_symbol,
+ base_dtb_fixup_offsets, preserve_local_fixups):
+ """Construct and serialize the `.dtbo` overlay blob for a single board.
+
+ Args:
+ dtb: The board's `ParsedDtb`.
+ common_node_set: Set of node paths present in the base DTB.
+ base_dtb_props: Dict mapping `path -> {prop_name: sym_val}` in the
+ base DTB.
+ base_dtb_phandles: Dict mapping base DTB node paths to their phandles.
+ base_dtb_symbols: Dict of symbols present in the base DTB.
+ path_to_base_dtb_symbol: Dict mapping base DTB node paths to symbol
+ names in the base DTB's `/__symbols__`.
+ base_dtb_fixup_offsets: Dict of `/__local_fixups__` offsets present in
+ the base DTB.
+ preserve_local_fixups: Whether to preserve `/__local_fixups__` in the
+ merged output.
+
+ Returns:
+ Serialized `.dtbo` overlay blob as `bytes`.
+ """
+ dtbo_root = NodeTree()
+ frag_node = dtbo_root.get_node('/fragment@0')
+ frag_node.props['target-path'] = b'/\x00'
+ ov_root = frag_node.get_node('__overlay__')
+
+ local_phandles = _assign_local_phandles(dtb, base_dtb_phandles)
+ ext_fixups, ov_local_fixups = _populate_dtbo_overlay_nodes(
+ ov_root, dtb, common_node_set, base_dtb_props, base_dtb_phandles,
+ path_to_base_dtb_symbol, local_phandles,
+ )
+ _populate_dtbo_metadata(
+ dtbo_root, ov_root, dtb, base_dtb_symbols, base_dtb_fixup_offsets,
+ ext_fixups, ov_local_fixups, preserve_local_fixups,
+ )
+
+ return dtbo_root.to_fdt_blob(size_hint=16384)
+
+
+def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
+ """Compute the base DTB and per-board DTBO blobs from parsed DTBs.
+
+ Args:
+ dtbs: List of `ParsedDtb` objects (all compiled with `-@ -L`).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the base DTB
+ and from the merged result.
+
+ Returns:
+ Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is `bytes`
+ and `dtbo_blobs` is a list of `bytes` corresponding to `dtbs`.
+ """
+ missing_meta = [
+ d.filename for d in dtbs if not (d.has_symbols and d.has_local_fixups)
+ ]
+ if missing_meta:
+ raise ValueError(
+ f'input DTB(s) lack /__symbols__ or /__local_fixups__: '
+ f'{", ".join(missing_meta)} (compile input DTBs with dtc -@ -L)'
+ )
+
+ _validate_dtb_headers(dtbs)
+
+ for d in dtbs:
+ d.build_symbolic_props()
+
+ base_dtb_nodes, common_node_set, base_dtb_phandles = (
+ _find_common_nodes_and_phandles(dtbs)
+ )
+ base_dtb_symbols, path_to_base_dtb_symbol = _find_common_symbols(
+ dtbs, common_node_set, base_dtb_phandles
+ )
+ preserve_local_fixups = not strip_local_fixups
+ base_dtb_props, base_dtb_fixup_offsets = _select_base_dtb_properties(
+ dtbs, base_dtb_nodes, base_dtb_phandles, preserve_local_fixups
+ )
+
+ base_dtb_blob = _build_base_dtb(
+ dtbs, base_dtb_nodes, base_dtb_props, base_dtb_phandles,
+ base_dtb_symbols, base_dtb_fixup_offsets,
+ )
+ dtbo_blobs = [
+ _build_board_dtbo(
+ d, common_node_set, base_dtb_props, base_dtb_phandles,
+ base_dtb_symbols, path_to_base_dtb_symbol, base_dtb_fixup_offsets,
+ preserve_local_fixups,
+ )
+ for d in dtbs
+ ]
+ return base_dtb_blob, dtbo_blobs
+
+
+def verify_recreated_dtb(orig_dtb, base_dtb_blob, dtbo_blob,
+ strip_local_fixups):
+ """Apply `dtbo_blob` to `base_dtb_blob` and verify it matches `orig_dtb`.
+
+ Args:
+ orig_dtb: The original `ParsedDtb` instance for the board.
+ base_dtb_blob: Uncompressed base DTB binary as `bytes`.
+ dtbo_blob: Uncompressed board `.dtbo` overlay binary as `bytes`.
+ strip_local_fixups: Whether `/__local_fixups__` was stripped.
+
+ Raises:
+ RuntimeError: If `fdt_overlay_apply()` fails or any node, property,
+ phandle, symbol, or header field in the merged DTB differs from
+ `orig_dtb`.
+ """
+ merged_fdt = libfdt.Fdt(base_dtb_blob)
+ merged_fdt.resize(len(base_dtb_blob) + len(dtbo_blob) * 2 + 65536)
+ fdto = bytearray(dtbo_blob)
+ err = libfdt.fdt_overlay_apply(merged_fdt._fdt, fdto)
+ if err != 0:
+ raise RuntimeError(
+ f'{orig_dtb.filename}: fdt_overlay_apply failed with error {err} '
+ f'({libfdt.strerror(err)})'
+ )
+ merged_fdt.pack()
+ recreated = ParsedDtb(
+ f'{orig_dtb.filename} (recreated)', bytes(merged_fdt.as_bytearray())
+ )
+ if strip_local_fixups:
+ if recreated.has_local_fixups:
+ raise RuntimeError(
+ f'{orig_dtb.filename}: /__local_fixups__ present despite '
+ f'--strip-local-fixups'
+ )
+ recreated.local_fixups = orig_dtb.local_fixups
+ else:
+ if orig_dtb.local_fixups != recreated.local_fixups:
+ raise RuntimeError(
+ f'{orig_dtb.filename}: /__local_fixups__ mismatch'
+ )
+
+ recreated.build_symbolic_props()
+
+ if set(orig_dtb.nodes.keys()) != set(recreated.nodes.keys()):
+ missing = set(orig_dtb.nodes.keys()) - set(recreated.nodes.keys())
+ extra = set(recreated.nodes.keys()) - set(orig_dtb.nodes.keys())
+ raise RuntimeError(
+ f'{orig_dtb.filename}: node mismatch (missing={missing}, '
+ f'extra={extra})'
+ )
+
+ for path in orig_dtb.nodes.keys():
+ if orig_dtb.sym_props[path] != recreated.sym_props[path]:
+ for p in (set(orig_dtb.sym_props[path]) |
+ set(recreated.sym_props[path])):
+ if (orig_dtb.sym_props[path].get(p) !=
+ recreated.sym_props[path].get(p)):
+ raise RuntimeError(
+ f'{orig_dtb.filename}: property mismatch at {path}:{p}'
+ )
+ # Common base nodes may gain a phandle if another board referenced them.
+ if (orig_dtb.has_phandle_prop[path] and
+ not recreated.has_phandle_prop[path]):
+ raise RuntimeError(
+ f'{orig_dtb.filename}: missing phandle at {path}'
+ )
+ if (orig_dtb.has_linux_phandle_prop[path] and
+ not recreated.has_linux_phandle_prop[path]):
+ raise RuntimeError(
+ f'{orig_dtb.filename}: missing linux,phandle at {path}'
+ )
+
+ if orig_dtb.symbols != recreated.symbols:
+ raise RuntimeError(f'{orig_dtb.filename}: /__symbols__ mismatch')
+ if orig_dtb.boot_cpuid_phys != recreated.boot_cpuid_phys:
+ raise RuntimeError(
+ f'{orig_dtb.filename}: boot_cpuid_phys mismatch '
+ f'({orig_dtb.boot_cpuid_phys} != {recreated.boot_cpuid_phys})'
+ )
+ if orig_dtb.mem_rsv != recreated.mem_rsv:
+ raise RuntimeError(
+ f'{orig_dtb.filename}: /memreserve/ mismatch '
+ f'({orig_dtb.mem_rsv} != {recreated.mem_rsv})'
+ )
+
+
+def get_dtbo_output_path(dtb_path, outdir):
+ """Determine the output `.dtbo` file path for an input `.dtb` path.
+
+ Args:
+ dtb_path: Path to the input `.dtb` file.
+ outdir: Output directory if `--outdir` was specified, else None.
+
+ Returns:
+ Output `.dtbo` file path string.
+ """
+ dtb_dir, stem = split_dtb_filename(dtb_path)
+ dtbo_name = f'{stem}.dtbo'
+ target_dir = outdir if outdir is not None else dtb_dir
+ return os.path.join(target_dir, dtbo_name)
+
+
+def parse_args():
+ """Parse command-line arguments.
+
+ Returns:
+ Parsed `argparse.Namespace` object.
+ """
+ parser = argparse.ArgumentParser(
+ description=(
+ 'Split a set of compiled device tree blobs (.dtb) into a single '
+ 'common base .dtb and per-board overlay blobs (.dtbo).'
+ )
+ )
+ parser.add_argument(
+ '-b', '--base-dtb',
+ required=True,
+ metavar='BASE_DTB',
+ help=(
+ 'Output common base .dtb file path (placed in --outdir if '
+ 'specified without a directory component)'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--outdir',
+ default=None,
+ metavar='DIR',
+ help=(
+ 'Output directory for generated .dtbo files and bare --base-dtb '
+ 'filenames (default: alongside each input .dtb)'
+ ),
+ )
+ parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from the generated base .dtb and '
+ 'reconstructed device trees'
+ ),
+ )
+ parser.add_argument(
+ '--verify',
+ action='store_true',
+ help=(
+ 'Verify that applying each generated .dtbo to the base .dtb '
+ 'recreates the original .dtb'
+ ),
+ )
+ parser.add_argument(
+ '-v', '--verbose',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+ parser.add_argument(
+ 'dtbs',
+ nargs='+',
+ metavar='DTB',
+ help='Input .dtb files (at least two required)',
+ )
+ args = parser.parse_args()
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
+ return args
+
+
+def factor_dtb_files(args):
+ """Factor a list of standalone .dtb files into a base .dtb and .dtbo files.
+
+ Args:
+ args: Parsed `argparse.Namespace` object.
+ """
+ input_dtb_paths = args.dtbs
+
+ dtbs = [ParsedDtb.from_file(p) for p in input_dtb_paths]
+
+ base_dtb_path = (
+ os.path.join(args.outdir, args.base_dtb)
+ if args.outdir is not None and not os.path.dirname(args.base_dtb)
+ else args.base_dtb
+ )
+
+ input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
+ base_abs_path = os.path.abspath(base_dtb_path)
+ if base_abs_path in input_abs_paths:
+ raise ValueError(
+ f'base DTB output path {base_dtb_path!r} collides with '
+ f'an input DTB file'
+ )
+
+ dtbo_paths = []
+ seen_paths = {base_abs_path}
+ for dtb_path in input_dtb_paths:
+ out_path = get_dtbo_output_path(dtb_path, args.outdir)
+ abs_out = os.path.abspath(out_path)
+ if abs_out in input_abs_paths:
+ raise ValueError(
+ f'output path {out_path!r} collides with an input DTB file'
+ )
+ if abs_out in seen_paths:
+ raise ValueError(
+ f'output path collision on {out_path!r}; ensure input '
+ f'files have distinct names'
+ )
+ seen_paths.add(abs_out)
+ dtbo_paths.append(out_path)
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ dtbs,
+ strip_local_fixups=args.strip_local_fixups,
+ )
+
+ if args.verify:
+ for d, dtbo_blob in zip(dtbs, dtbo_blobs):
+ verify_recreated_dtb(
+ d, base_dtb_blob, dtbo_blob, args.strip_local_fixups
+ )
+
+ base_dtb_dir = os.path.dirname(base_dtb_path)
+ if base_dtb_dir:
+ os.makedirs(base_dtb_dir, exist_ok=True)
+ with open(base_dtb_path, 'wb') as f:
+ f.write(base_dtb_blob)
+
+ if args.outdir:
+ os.makedirs(args.outdir, exist_ok=True)
+ for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ out_dir = os.path.dirname(out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(out_path, 'wb') as f:
+ f.write(dtbo_blob)
+
+ if args.verbose:
+ orig_uncomp = sum(len(d.raw_bytes) for d in dtbs)
+ base_dtb_uncomp = len(base_dtb_blob)
+ dtbo_uncomp = sum(len(b) for b in dtbo_blobs)
+ new_uncomp = base_dtb_uncomp + dtbo_uncomp
+ saved_uncomp = orig_uncomp - new_uncomp
+ pct_uncomp = (
+ (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
+ )
+
+ print(
+ f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} bytes) -> '
+ f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_uncomp:,} B = {new_uncomp:,} bytes '
+ f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
+ )
+
+
+def main():
+ """Run the fdtfactor command-line tool.
+
+ Returns:
+ Process exit code (0 on success).
+ """
+ args = parse_args()
+ try:
+ factor_dtb_files(args)
+ except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc:
+ sys.exit(f'Error: {exc}')
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/meson.build b/meson.build
index 520cd35f9075..73b4687a2852 100644
--- a/meson.build
+++ b/meson.build
@@ -114,6 +114,7 @@ if get_option('tools') and not wheel_only
install_data(
'dtdiff',
+ 'fdtfactor',
install_dir: get_option('bindir'),
install_mode: 'rwxr-xr-x',
)
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index af5d309cf8a9..9f8a2896d597 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,7 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
TESTS_CLEANDIRS_L = fs
diff --git a/tests/fdtfactor_board1.dts b/tests/fdtfactor_board1.dts
new file mode 100644
index 000000000000..cab6b3a4a3cd
--- /dev/null
+++ b/tests/fdtfactor_board1.dts
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ model = "Test Board 1";
+ compatible = "vendor,board1", "vendor,common";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ intc: interrupt-controller@1000 {
+ compatible = "vendor,intc";
+ reg = <0x1000 0x100>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ clk: clock-controller@2000 {
+ compatible = "vendor,clk";
+ reg = <0x2000 0x100>;
+ #clock-cells = <1>;
+ };
+
+ soc {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+ interrupt-parent = <&intc>;
+
+ uart0: serial@3000 {
+ compatible = "vendor,uart";
+ reg = <0x3000 0x100>;
+ interrupts = <1 0>;
+ clocks = <&clk 1>;
+ status = "okay";
+ };
+
+ i2c0: i2c@4000 {
+ compatible = "vendor,i2c";
+ reg = <0x4000 0x100>;
+ interrupts = <2 0>;
+ clocks = <&clk 2>;
+ status = "okay";
+
+ pmic: pmic@40 {
+ compatible = "vendor,pmic";
+ reg = <0x40>;
+
+ reg_vdd: vdd {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ };
+ };
+ };
+
+ spi0: spi@5000 {
+ compatible = "vendor,spi";
+ reg = <0x5000 0x100>;
+ clocks = <&clk 3>;
+ vdd-supply = <®_vdd>;
+ cs-supply = <&board1_reg>;
+ };
+ };
+
+ board1_reg: regulator-board1 {
+ compatible = "regulator-fixed";
+ regulator-name = "board1-cs";
+ };
+};
diff --git a/tests/fdtfactor_board2.dts b/tests/fdtfactor_board2.dts
new file mode 100644
index 000000000000..8d3322e5ce74
--- /dev/null
+++ b/tests/fdtfactor_board2.dts
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ model = "Test Board 2";
+ compatible = "vendor,board2", "vendor,common";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ /*
+ * Extra phandle-bearing node before intc/clk shifts all subsequent
+ * phandle numbers in board2.dtb relative to board1.dtb.
+ */
+ board2_early_reg: regulator-early {
+ compatible = "regulator-fixed";
+ regulator-name = "board2-early";
+ };
+
+ intc: interrupt-controller@1000 {
+ compatible = "vendor,intc";
+ reg = <0x1000 0x100>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ clk: clock-controller@2000 {
+ compatible = "vendor,clk";
+ reg = <0x2000 0x100>;
+ #clock-cells = <1>;
+ };
+
+ soc {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+ interrupt-parent = <&intc>;
+
+ uart0: serial@3000 {
+ compatible = "vendor,uart";
+ reg = <0x3000 0x100>;
+ interrupts = <1 0>;
+ clocks = <&clk 1>;
+ status = "disabled";
+ };
+
+ i2c0: i2c@4000 {
+ compatible = "vendor,i2c";
+ reg = <0x4000 0x100>;
+ interrupts = <2 0>;
+ clocks = <&clk 2>;
+ status = "okay";
+
+ pmic: pmic@40 {
+ compatible = "vendor,pmic";
+ reg = <0x40>;
+
+ reg_vdd: vdd {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <3300000>;
+ };
+ };
+ };
+
+ mmc0: mmc@6000 {
+ compatible = "vendor,mmc";
+ reg = <0x6000 0x100>;
+ clocks = <&clk 4>;
+ vmmc-supply = <®_vdd>;
+ vqmmc-supply = <&board2_early_reg>;
+ };
+ };
+};
diff --git a/tests/fdtfactor_board3.dts b/tests/fdtfactor_board3.dts
new file mode 100644
index 000000000000..1a0fb1641241
--- /dev/null
+++ b/tests/fdtfactor_board3.dts
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ model = "Test Board 3";
+ compatible = "vendor,board3", "vendor,common";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ /*
+ * Two extra phandle-bearing nodes before intc/clk shift all subsequent
+ * phandle numbers in board3.dtb relative to both board1.dtb and
+ * board2.dtb.
+ */
+ board3_early_reg1: regulator-early1 {
+ compatible = "regulator-fixed";
+ regulator-name = "board3-early1";
+ };
+
+ board3_early_reg2: regulator-early2 {
+ compatible = "regulator-fixed";
+ regulator-name = "board3-early2";
+ vin-supply = <&board3_early_reg1>;
+ };
+
+ intc: interrupt-controller@1000 {
+ compatible = "vendor,intc";
+ reg = <0x1000 0x100>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ clk: clock-controller@2000 {
+ compatible = "vendor,clk";
+ reg = <0x2000 0x100>;
+ #clock-cells = <1>;
+ };
+
+ soc {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+ interrupt-parent = <&intc>;
+
+ uart0: serial@3000 {
+ compatible = "vendor,uart";
+ reg = <0x3000 0x100>;
+ interrupts = <1 0>;
+ clocks = <&clk 1>;
+ status = "okay";
+ };
+
+ i2c0: i2c@4000 {
+ compatible = "vendor,i2c";
+ reg = <0x4000 0x100>;
+ interrupts = <2 0>;
+ clocks = <&clk 2>;
+ status = "okay";
+
+ pmic: pmic@40 {
+ compatible = "vendor,pmic";
+ reg = <0x40>;
+
+ reg_vdd: vdd {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <3300000>;
+ };
+ };
+ };
+
+ spi0: spi@5000 {
+ compatible = "vendor,spi";
+ reg = <0x5000 0x100>;
+ clocks = <&clk 3>;
+ vdd-supply = <®_vdd>;
+ cs-supply = <&board3_early_reg2>;
+ };
+
+ mmc0: mmc@6000 {
+ compatible = "vendor,mmc";
+ reg = <0x6000 0x100>;
+ clocks = <&clk 4>;
+ vmmc-supply = <®_vdd>;
+ vqmmc-supply = <&board3_early_reg1>;
+ };
+
+ usb0: usb@7000 {
+ compatible = "vendor,usb";
+ reg = <0x7000 0x100>;
+ interrupts = <5 0>;
+ clocks = <&clk 5>;
+ vbus-supply = <&board3_early_reg2>;
+ };
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 842b543059bb..92fa417d3457 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -239,6 +239,21 @@ run_fdtoverlay_test() {
base_run_test sh "$SRCDIR/fdtoverlay-runtest.sh" "$expect" "$@"
}
+# Compare an original .dtb and a recreated .dtb by decompiling and sorting both
+# with dtc -I dtb -O dts -qq -f -s (like dtdiff) and normalizing the numeric
+# phandle properties assigned to target nodes. Because both blobs contain
+# /__symbols__ and /__local_fixups__, dtc restores all node labels and converts
+# all phandle references in properties back into symbolic <&label> references.
+check_dtb_recreated () {
+ orig="$1"
+ recreated="$2"
+ $DTC -I dtb -O dts -qq -f -s -o "$orig.raw.test.dts" "$orig" || return 1
+ $DTC -I dtb -O dts -qq -f -s -o "$recreated.raw.test.dts" "$recreated" || return 1
+ sed 's/phandle = <0x[0-9a-f]*>;/phandle = <0xffffffff>;/' "$orig.raw.test.dts" > "$orig.cmp.test.dts" || return 1
+ sed 's/phandle = <0x[0-9a-f]*>;/phandle = <0xffffffff>;/' "$recreated.raw.test.dts" > "$recreated.cmp.test.dts" || return 1
+ diff -u "$orig.cmp.test.dts" "$recreated.cmp.test.dts"
+}
+
BAD_FIXUP_TREES="bad_index \
empty \
empty_index \
@@ -1089,6 +1104,50 @@ fdtoverlay_tests() {
shorten_echo "check phandle to noda d wasn't overwritten: "
run_wrap_test test "$bd" = "$pd"
+
+ if ! $no_python; then
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board1.test.dtb "$SRCDIR/fdtfactor_board1.dts"
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board2.test.dtb "$SRCDIR/fdtfactor_board2.dts"
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_board3.test.dtb "$SRCDIR/fdtfactor_board3.dts"
+
+ # Symbolic phandle deduplication mode (3 boards)
+ run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \
+ -b fdtfactor_base.test.dtb \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.test.dtb
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \
+ -o fdtfactor_board1.recreated.test.dtb fdtfactor_board1.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \
+ -o fdtfactor_board2.recreated.test.dtb fdtfactor_board2.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base.test.dtb \
+ -o fdtfactor_board3.recreated.test.dtb fdtfactor_board3.test.dtbo
+ run_wrap_test check_dtb_recreated fdtfactor_board1.test.dtb fdtfactor_board1.recreated.test.dtb
+ run_wrap_test check_dtb_recreated fdtfactor_board2.test.dtb fdtfactor_board2.recreated.test.dtb
+ run_wrap_test check_dtb_recreated fdtfactor_board3.test.dtb fdtfactor_board3.recreated.test.dtb
+ # Majority value ("okay" in boards 1 & 3) should be in base.dtb
+ run_fdtget_test "okay" fdtfactor_base.test.dtb /soc/serial@3000 status -ts
+
+ # Strip local fixups mode (3 boards)
+ run_wrap_test $PYTHON "$FDTFACTOR" --verify -S \
+ --base-dtb fdtfactor_base_stripped.test.dtb -o . \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.test.dtb
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \
+ -o fdtfactor_board1.stripped.test.dtb fdtfactor_board1.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \
+ -o fdtfactor_board2.stripped.test.dtb fdtfactor_board2.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fdtfactor_base_stripped.test.dtb \
+ -o fdtfactor_board3.stripped.test.dtb fdtfactor_board3.test.dtbo
+ run_test check_path fdtfactor_board1.stripped.test.dtb not-exists "/__local_fixups__"
+ # Without /__local_fixups__, dtc -I dtb -O dts cannot turn phandle cells
+ # inside properties back into <&label> references, so compare stripped
+ # against recreated (after removing /__local_fixups__), which shares
+ # fdtfactor's phandle numbering and was already verified against orig.
+ run_wrap_test $DTPUT -r fdtfactor_board1.recreated.test.dtb /__local_fixups__
+ run_wrap_test $DTPUT -r fdtfactor_board2.recreated.test.dtb /__local_fixups__
+ run_wrap_test $DTPUT -r fdtfactor_board3.recreated.test.dtb /__local_fixups__
+ run_wrap_test check_dtb_recreated fdtfactor_board1.recreated.test.dtb fdtfactor_board1.stripped.test.dtb
+ run_wrap_test check_dtb_recreated fdtfactor_board2.recreated.test.dtb fdtfactor_board2.stripped.test.dtb
+ run_wrap_test check_dtb_recreated fdtfactor_board3.recreated.test.dtb fdtfactor_board3.stripped.test.dtb
+ fi
}
pylibfdt_tests () {
@@ -1139,7 +1198,7 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts
for set in $TESTSETS; do
case $set in
diff --git a/tests/testutils.sh b/tests/testutils.sh
index 6b2f0d14e3e5..7f35ba8e4158 100644
--- a/tests/testutils.sh
+++ b/tests/testutils.sh
@@ -27,6 +27,7 @@ DTGET=${TEST_BINDIR}/fdtget
DTPUT=${TEST_BINDIR}/fdtput
FDTDUMP=${TEST_BINDIR}/fdtdump
FDTOVERLAY=${TEST_BINDIR}/fdtoverlay
+FDTFACTOR=${SRCDIR}/../fdtfactor
verbose_run () {
if [ -z "$QUIET_TEST" ]; then
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/4] fdtfactor: Support compressed DTB inputs and outputs
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-10-15 18:21 ` [PATCH 1/4] tests: Fix Python test environment setup in meson.build Douglas Anderson
2026-10-15 18:21 ` [PATCH 2/4] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
@ 2026-10-15 18:21 ` Douglas Anderson
2026-10-15 18:21 ` [PATCH 4/4] fdtfactor: Add compiled FIT image factoring support Douglas Anderson
2026-10-23 4:49 ` [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
4 siblings, 0 replies; 7+ messages in thread
From: Douglas Anderson @ 2026-10-15 18:21 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Chen-Yu Tsai, linux-kernel, Devicetree, Brian Norris,
Tudor Ambarus, André Draszik, Julius Werner,
Saravana Kannan, Peter Griffin, Douglas Anderson
Firmware and FIT image build pipelines frequently store device tree
blobs compressed with LZ4, LZMA/XZ, or gzip (e.g., `.dtb.lz4`).
Extend `fdtfactor` to detect compressed input blobs by magic bytes or
file extension, transparently decompress them before factoring, and
compress the resulting `.dtbo` and `base.dtb` outputs when appropriate:
- Support `.lz4` (via the `lz4` CLI), `.lzma`/`.xz` (via Python's
`lzma` module), and `.gz` (via Python's `gzip` module).
- Preserve the input compression format on generated `.dtbo` files and
compress `base.dtb` when its output path carries a compression suffix
(e.g., `-b base.dtb.lz4`).
- Report compressed size savings alongside uncompressed savings when
`-v`/`--verbose` is specified.
On the 47 Qualcomm SC7180 Trogdor Chromebook `.dtb.lz4` files,
`fdtfactor -S` reduces total LZ4-compressed size from 1.71 MB to 213 KB
(87.5% savings).
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
README.md | 16 +--
fdtfactor | 267 ++++++++++++++++++++++++++++++++++++++-----
tests/Makefile.tests | 5 +-
tests/run_tests.sh | 34 +++++-
4 files changed, 284 insertions(+), 38 deletions(-)
diff --git a/README.md b/README.md
index e2ff2374b01d..6ace31bfbfdb 100644
--- a/README.md
+++ b/README.md
@@ -94,10 +94,10 @@ values.
## fdtfactor
`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`) and splits them into a single common
-`base.dtb` and per-board `.dtbo` overlays so that applying a board's
-`.dtbo` to `base.dtb` recreates the original board device tree while
-reducing total storage size.
+compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
+`.dtb.lzma`, `.dtb.gz`) and splits them into a single common `base.dtb` and
+per-board `.dtbo` overlays so that applying a board's `.dtbo` to `base.dtb`
+recreates the original board device tree while reducing total storage size.
Input `.dtb` files must be compiled with `dtc -@ -L` so that both
`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
@@ -112,8 +112,10 @@ $ fdtfactor -b base.dtb -o out_dir board1.dtb board2.dtb board3.dtb
Options:
-* `-b`, `--base-dtb`: Output path for the common base `.dtb` (placed in
- `--outdir` if specified without a directory component).
+* `-b`, `--base-dtb`: Output path for the common base `.dtb` (compressed if
+ given a `.lz4`, `.xz`, `.lzma`, or `.gz` suffix, whereas each `.dtbo`
+ preserves its input `.dtb`'s compression; placed in `--outdir` if specified
+ without a directory component).
* `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for
`--base-dtb` when given without a directory component).
* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
@@ -127,7 +129,7 @@ Options:
original `.dtb`'s `/__local_fixups__` subtree verbatim.
* `--verify`: Verify that applying each generated `.dtbo` to `base.dtb`
with `libfdt` recreates the original `.dtb`.
-* `-v`, `--verbose`: Print size savings summary.
+* `-v`, `--verbose`: Print uncompressed and compressed size savings.
### TODO
diff --git a/fdtfactor b/fdtfactor
index 377f499bea50..9f06801f973c 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -11,31 +11,195 @@ original device tree.
import argparse
from collections import Counter
+import gzip
+import lzma
import os
import struct
+import subprocess
import sys
import libfdt
+FDT_MAGIC = b'\xd0\x0d\xfe\xed'
+LZ4_MAGIC = b'\x04\x22\x4d\x18'
+LZ4_LEGACY_MAGIC = b'\x02\x21\x4c\x18'
+GZIP_MAGIC = b'\x1f\x8b'
+XZ_MAGIC = b'\xfd7zXZ\x00'
+# Matches raw LZMA_ALONE headers with default lc=3, lp=0, pb=2 (0x5d) and
+# power-of-two dictionary sizes >= 64 KiB; .lzma extension handles any
+# non-default LZMA headers.
+LZMA_ALONE_MAGIC = b'\x5d\x00\x00'
+
+COMP_TO_EXT = {
+ 'none': '',
+ 'lz4': '.lz4',
+ 'lzma': '.lzma',
+ 'xz': '.xz',
+ 'gzip': '.gz',
+}
+
+EXT_TO_COMP = {
+ '.lz4': 'lz4',
+ '.lzma': 'lzma',
+ '.xz': 'xz',
+ '.gz': 'gzip',
+}
+
+
+def detect_compression(raw_bytes, filename=''):
+ """Detect compression algorithm from magic bytes or filename extension.
+
+ Args:
+ raw_bytes: Input file or blob contents as `bytes`.
+ filename: Optional file path or name for extension fallback.
+
+ Returns:
+ Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'xz'`, or
+ `'gzip'`).
+ """
+ if raw_bytes.startswith(FDT_MAGIC):
+ return 'none'
+ if (raw_bytes.startswith(LZ4_MAGIC) or
+ raw_bytes.startswith(LZ4_LEGACY_MAGIC)):
+ return 'lz4'
+ if raw_bytes.startswith(GZIP_MAGIC):
+ return 'gzip'
+ if raw_bytes.startswith(XZ_MAGIC):
+ return 'xz'
+ if raw_bytes.startswith(LZMA_ALONE_MAGIC):
+ return 'lzma'
+ _, ext = os.path.splitext(filename)
+ if ext in EXT_TO_COMP:
+ return EXT_TO_COMP[ext]
+ return 'none'
+
+
+def decompress_blob(raw_bytes, filename='', comp=None):
+ """Decompress a DTB blob if compressed.
+
+ Args:
+ raw_bytes: Input file or blob contents as `bytes`.
+ filename: Optional file path or name for error reporting and extension
+ detection.
+ comp: Optional explicit compression algorithm name; auto-detected if
+ None or empty.
+
+ Returns:
+ Tuple of `(uncompressed_bytes, compression_type)`.
+
+ Raises:
+ RuntimeError: If `lz4` is missing or fails to decompress `raw_bytes`.
+ ValueError: If the compression type is unsupported.
+ """
+ if not comp:
+ comp = detect_compression(raw_bytes, filename)
+ if comp == 'none':
+ return raw_bytes, 'none'
+ if comp == 'lz4':
+ try:
+ res = subprocess.run(
+ ['lz4', '-d', '-q', '-c'],
+ input=raw_bytes,
+ capture_output=True,
+ check=False,
+ )
+ except FileNotFoundError as exc:
+ raise RuntimeError(
+ 'lz4 command not found; install lz4 to process .lz4 files'
+ ) from exc
+ if res.returncode != 0:
+ raise RuntimeError(
+ f'Failed to decompress {filename} with lz4: '
+ f'{res.stderr.decode("utf-8", errors="replace").strip()}'
+ )
+ return res.stdout, 'lz4'
+ if comp == 'gzip':
+ try:
+ return gzip.decompress(raw_bytes), 'gzip'
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {filename} with gzip: {exc}'
+ ) from exc
+ if comp in ('lzma', 'xz'):
+ try:
+ return lzma.decompress(raw_bytes), comp
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {filename} with {comp}: {exc}'
+ ) from exc
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
+def compress_blob(raw_bytes, comp):
+ """Compress an FDT blob using the specified compression algorithm.
+
+ Args:
+ raw_bytes: Uncompressed FDT blob as `bytes`.
+ comp: Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'xz'`,
+ or `'gzip'`).
+
+ Returns:
+ Compressed (or uncompressed if `comp` is `'none'`) blob as `bytes`.
+
+ Raises:
+ RuntimeError: If `lz4` is missing or fails to compress `raw_bytes`.
+ ValueError: If `comp` is not a supported compression type.
+ """
+ if comp in ('none', '', None):
+ return raw_bytes
+ if comp == 'lz4':
+ try:
+ res = subprocess.run(
+ ['lz4', '-12', '-z', '-q', '-c'],
+ input=raw_bytes,
+ capture_output=True,
+ check=False,
+ )
+ except FileNotFoundError as exc:
+ raise RuntimeError(
+ 'lz4 command not found; install lz4 to process .lz4 files'
+ ) from exc
+ if res.returncode != 0:
+ raise RuntimeError(
+ f'Failed to compress blob with lz4: '
+ f'{res.stderr.decode("utf-8", errors="replace").strip()}'
+ )
+ return res.stdout
+ if comp == 'gzip':
+ return gzip.compress(raw_bytes, mtime=0)
+ if comp == 'lzma':
+ return lzma.compress(raw_bytes, format=lzma.FORMAT_ALONE)
+ if comp == 'xz':
+ return lzma.compress(raw_bytes, format=lzma.FORMAT_XZ)
+ raise ValueError(f'Unsupported compression type: {comp}')
+
+
def split_dtb_filename(path):
- """Split a DTB path into its directory and filename stem.
+ """Split a DTB path into its directory, stem, and compression type.
- Example:
- '/dir/board.dtb' -> ('/dir', 'board')
+ Examples:
+ '/dir/board.dtb' -> ('/dir', 'board', 'none')
+ '/dir/board.dtb.lz4' -> ('/dir', 'board', 'lz4')
Args:
- path: File path to a .dtb or .dtbo file.
+ path: File path to a `.dtb` or `.dtbo` (optionally compressed) file.
Returns:
- Tuple of (directory, stem) where `stem` has any trailing `.dtb` or
- `.dtbo` suffix removed.
+ Tuple of `(directory, stem, compression_type)` where `stem` has any
+ trailing compression and `.dtb`/`.dtbo` suffixes removed.
"""
directory, filename = os.path.split(path)
+ comp = 'none'
+ prefix, sep, ext = filename.rpartition('.')
+ if prefix and sep and f'.{ext}' in EXT_TO_COMP:
+ comp = EXT_TO_COMP[f'.{ext}']
+ filename = prefix
stem, sep, ext = filename.rpartition('.')
if not stem or not sep or ext not in ('dtb', 'dtbo'):
stem = filename
- return directory, stem
+ return directory, stem, comp
class NodeTree:
@@ -172,15 +336,24 @@ def _is_dtbo_overlay(fdt_ro):
class ParsedDtb:
"""Parsed representation of an input .dtb blob."""
- def __init__(self, filename, raw_bytes):
+ def __init__(self, filename, raw_bytes, compressed_size=None,
+ compression='none'):
"""Parse an uncompressed FDT blob into nodes, symbols, and fixups.
Args:
filename: Display name or file path for error reporting.
raw_bytes: Uncompressed FDT binary contents as `bytes`.
+ compressed_size: Original on-disk size in bytes before
+ decompression (defaults to `len(raw_bytes)`).
+ compression: Compression algorithm of the input file (`'none'`,
+ `'lz4'`, `'lzma'`, `'xz'`, or `'gzip'`).
"""
self.filename = filename
self.raw_bytes = raw_bytes
+ self.compressed_size = (
+ compressed_size if compressed_size is not None else len(raw_bytes)
+ )
+ self.compression = compression
fdt = _open_fdt_ro(raw_bytes, filename)
if _is_dtbo_overlay(fdt):
raise ValueError(
@@ -211,17 +384,19 @@ class ParsedDtb:
@classmethod
def from_file(cls, path):
- """Read and parse a .dtb file from disk.
+ """Read and parse a .dtb (or compressed .dtb.lz4/.lzma/.xz/.gz) file.
Args:
- path: Path to the input .dtb file.
+ path: Path to the input .dtb (or compressed .dtb) file.
Returns:
A new `ParsedDtb` instance.
"""
with open(path, 'rb') as f:
- raw_bytes = f.read()
- return cls(path, raw_bytes)
+ file_bytes = f.read()
+ raw_bytes, comp = decompress_blob(file_bytes, path)
+ return cls(path, raw_bytes, compressed_size=len(file_bytes),
+ compression=comp)
def _parse_tree(self, fdt):
"""Walk the FDT and extract nodes, properties, symbols, and fixups.
@@ -1036,18 +1211,20 @@ def verify_recreated_dtb(orig_dtb, base_dtb_blob, dtbo_blob,
)
-def get_dtbo_output_path(dtb_path, outdir):
- """Determine the output `.dtbo` file path for an input `.dtb` path.
+def get_dtbo_output_path(dtb_path, outdir, comp='none'):
+ """Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
Args:
- dtb_path: Path to the input `.dtb` file.
+ dtb_path: Path to the input `.dtb` (or compressed `.dtb`) file.
outdir: Output directory if `--outdir` was specified, else None.
+ comp: Compression algorithm name (`'none'`, `'lz4'`, `'lzma'`, `'xz'`,
+ or `'gzip'`).
Returns:
- Output `.dtbo` file path string.
+ Output `.dtbo[.comp]` file path string.
"""
- dtb_dir, stem = split_dtb_filename(dtb_path)
- dtbo_name = f'{stem}.dtbo'
+ dtb_dir, stem, _ = split_dtb_filename(dtb_path)
+ dtbo_name = f'{stem}.dtbo{COMP_TO_EXT[comp]}'
target_dir = outdir if outdir is not None else dtb_dir
return os.path.join(target_dir, dtbo_name)
@@ -1060,8 +1237,9 @@ def parse_args():
"""
parser = argparse.ArgumentParser(
description=(
- 'Split a set of compiled device tree blobs (.dtb) into a single '
- 'common base .dtb and per-board overlay blobs (.dtbo).'
+ 'Split a set of compiled device tree blobs (.dtb, or compressed '
+ '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and '
+ 'per-board overlay blobs (.dtbo).'
)
)
parser.add_argument(
@@ -1069,8 +1247,8 @@ def parse_args():
required=True,
metavar='BASE_DTB',
help=(
- 'Output common base .dtb file path (placed in --outdir if '
- 'specified without a directory component)'
+ 'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) file path '
+ '(placed in --outdir if specified without a directory component)'
),
)
parser.add_argument(
@@ -1101,13 +1279,16 @@ def parse_args():
parser.add_argument(
'-v', '--verbose',
action='store_true',
- help='Print size statistics summary',
+ help='Print size statistics and compression summary',
)
parser.add_argument(
'dtbs',
nargs='+',
metavar='DTB',
- help='Input .dtb files (at least two required)',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (at '
+ 'least two required)'
+ ),
)
args = parser.parse_args()
if len(args.dtbs) < 2:
@@ -1130,6 +1311,7 @@ def factor_dtb_files(args):
if args.outdir is not None and not os.path.dirname(args.base_dtb)
else args.base_dtb
)
+ _, _, base_dtb_comp = split_dtb_filename(base_dtb_path)
input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
base_abs_path = os.path.abspath(base_dtb_path)
@@ -1141,8 +1323,8 @@ def factor_dtb_files(args):
dtbo_paths = []
seen_paths = {base_abs_path}
- for dtb_path in input_dtb_paths:
- out_path = get_dtbo_output_path(dtb_path, args.outdir)
+ for dtb_path, d in zip(input_dtb_paths, dtbs):
+ out_path = get_dtbo_output_path(dtb_path, args.outdir, comp=d.compression)
abs_out = os.path.abspath(out_path)
if abs_out in input_abs_paths:
raise ValueError(
@@ -1167,20 +1349,24 @@ def factor_dtb_files(args):
d, base_dtb_blob, dtbo_blob, args.strip_local_fixups
)
+ base_dtb_file_bytes = compress_blob(base_dtb_blob, base_dtb_comp)
base_dtb_dir = os.path.dirname(base_dtb_path)
if base_dtb_dir:
os.makedirs(base_dtb_dir, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
- f.write(base_dtb_blob)
+ f.write(base_dtb_file_bytes)
if args.outdir:
os.makedirs(args.outdir, exist_ok=True)
- for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ dtbo_file_sizes = []
+ for out_path, dtbo_blob, d in zip(dtbo_paths, dtbo_blobs, dtbs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
+ out_bytes = compress_blob(dtbo_blob, d.compression)
+ dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
- f.write(dtbo_blob)
+ f.write(out_bytes)
if args.verbose:
orig_uncomp = sum(len(d.raw_bytes) for d in dtbs)
@@ -1198,6 +1384,29 @@ def factor_dtb_files(args):
f'{dtbo_uncomp:,} B = {new_uncomp:,} bytes '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
+ all_comps = {base_dtb_comp, *(d.compression for d in dtbs)}
+ comp_names = sorted(c for c in all_comps if c != 'none')
+ if comp_names:
+ comp_label = '/'.join(sorted(all_comps))
+ hdr = (
+ f'Compressed ({comp_label}):'
+ if len(all_comps) == 1
+ else f'On-disk ({comp_label}): '
+ )
+ orig_comp = sum(d.compressed_size for d in dtbs)
+ base_dtb_csize = len(base_dtb_file_bytes)
+ dtbo_csize = sum(dtbo_file_sizes)
+ new_comp = base_dtb_csize + dtbo_csize
+ saved_comp = orig_comp - new_comp
+ pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
+ print(
+ f'{hdr} {len(dtbs)} files '
+ f'({orig_comp:,} bytes) -> base {base_dtb_csize:,} B + '
+ f'{len(dtbo_blobs)} overlays {dtbo_csize:,} B '
+ f'(avg {dtbo_csize // len(dtbo_blobs):,} B) = '
+ f'{new_comp:,} bytes '
+ f'(saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ )
def main():
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 9f8a2896d597..d0bdc4afefb2 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
+ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
+ tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
TESTS_CLEANDIRS_L = fs
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 92fa417d3457..92edec8b724c 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1147,6 +1147,36 @@ fdtoverlay_tests() {
run_wrap_test check_dtb_recreated fdtfactor_board1.recreated.test.dtb fdtfactor_board1.stripped.test.dtb
run_wrap_test check_dtb_recreated fdtfactor_board2.recreated.test.dtb fdtfactor_board2.stripped.test.dtb
run_wrap_test check_dtb_recreated fdtfactor_board3.recreated.test.dtb fdtfactor_board3.stripped.test.dtb
+
+ # Compressed (.dtb.gz / .dtb.xz / .dtb.lzma / .dtb.lz4) modes (3 boards)
+ for comp_spec in "gz:gzip -9 -k -f:gzip -d -f" \
+ "xz:xz -k -f:xz -d -f" \
+ "lzma:xz --format=lzma -k -f:xz -d -f" \
+ "lz4:lz4 -12 -q -f -m:lz4 -d -q -f --rm"; do
+ ext="${comp_spec%%:*}"
+ rest="${comp_spec#*:}"
+ comp_cmd="${rest%%:*}"
+ decomp_cmd="${rest#*:}"
+ if ! command -v "${comp_cmd%% *}" >/dev/null 2>&1; then
+ continue
+ fi
+ run_wrap_test $comp_cmd \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.test.dtb
+ run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \
+ -b "fdtfactor_base.test.dtb.$ext" \
+ "fdtfactor_board1.test.dtb.$ext" \
+ "fdtfactor_board2.test.dtb.$ext" \
+ "fdtfactor_board3.test.dtb.$ext"
+ cp "fdtfactor_base.test.dtb.$ext" "fdtfactor_base.${ext}_dec.test.dtb.$ext"
+ cp "fdtfactor_board1.test.dtbo.$ext" "fdtfactor_board1.${ext}_dec.test.dtbo.$ext"
+ run_wrap_test $decomp_cmd "fdtfactor_base.${ext}_dec.test.dtb.$ext"
+ run_wrap_test $decomp_cmd "fdtfactor_board1.${ext}_dec.test.dtbo.$ext"
+ run_wrap_test $FDTOVERLAY -i "fdtfactor_base.${ext}_dec.test.dtb" \
+ -o "fdtfactor_board1.${ext}_recreated.test.dtb" \
+ "fdtfactor_board1.${ext}_dec.test.dtbo"
+ run_wrap_test check_dtb_recreated \
+ fdtfactor_board1.test.dtb "fdtfactor_board1.${ext}_recreated.test.dtb"
+ done
fi
}
@@ -1198,7 +1228,9 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+ *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
+ *.test.dtb.lz4 *.test.dtbo.lz4
for set in $TESTSETS; do
case $set in
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 4/4] fdtfactor: Add compiled FIT image factoring support
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (2 preceding siblings ...)
2026-10-15 18:21 ` [PATCH 3/4] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
@ 2026-10-15 18:21 ` Douglas Anderson
2026-10-23 4:49 ` [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
4 siblings, 0 replies; 7+ messages in thread
From: Douglas Anderson @ 2026-10-15 18:21 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Chen-Yu Tsai, linux-kernel, Devicetree, Brian Norris,
Tudor Ambarus, André Draszik, Julius Werner,
Saravana Kannan, Peter Griffin, Douglas Anderson
Bootloaders such as depthcharge on Chromebooks and U-Boot use Flattened
Image Tree (FIT) images to bundle a kernel image with multiple board FDT
blobs (`fdt-1`, `fdt-2`, ...) and per-board configuration nodes.
Add `-f`/`--fit`, `-F`/`--fit-out`, and `--single-base-dtb` options to
`fdtfactor` to factor a compiled FIT image directly:
- Parse non-overlay `type = "flat_dt"` nodes under `/images`,
decompressing their `data` blobs according to their `compression`
property (`none`, `lz4`, `lzma`, `gzip`).
- Partition the `flat_dt` blobs into compatible factor groups (by
compression, architecture, FDT header fields, and node-path overlap)
so a single FIT image containing multiple board/SoC families produces
a separate `fdt-base-1`, `fdt-base-2`, etc. for each family.
- Support `--single-base-dtb` to force all `flat_dt` blobs into a single
base DTB regardless of node-path overlap, failing only if their FDT
headers, architecture, or compression differ.
- Insert each `fdt-base-N` node into `/images`, replace each factored
`fdt-*` node's `data` with its compressed `.dtbo` overlay, recompute
any `hash*` subnode digests, and prepend `fdt-base-N` to each matching
configuration's `fdt` string list while preserving all other
configuration properties (`description`, `compatible`, etc.,
populating `compatible` from the board DTB's root node when omitted)
and any pre-existing overlay entries.
Add uncompressed, LZ4-compressed multi-family, and `--single-base-dtb`
FIT image test cases to the `fdtoverlay` test suite.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
README.md | 15 +
fdtfactor | 726 +++++++++++++++++++++++++++++++++++--
tests/Makefile.tests | 2 +-
tests/fdtfactor_other1.dts | 69 ++++
tests/fdtfactor_other2.dts | 69 ++++
tests/fdtfactor_test.its | 109 ++++++
tests/run_tests.sh | 69 +++-
7 files changed, 1033 insertions(+), 26 deletions(-)
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
diff --git a/README.md b/README.md
index 6ace31bfbfdb..30fc28fa7545 100644
--- a/README.md
+++ b/README.md
@@ -110,6 +110,12 @@ Basic usage:
$ fdtfactor -b base.dtb -o out_dir board1.dtb board2.dtb board3.dtb
```
+Compiled FIT image usage (targeting depthcharge / coreboot / U-Boot):
+
+```
+$ fdtfactor -S -f image.fit -F image.factored.fit
+```
+
Options:
* `-b`, `--base-dtb`: Output path for the common base `.dtb` (compressed if
@@ -118,6 +124,11 @@ Options:
without a directory component).
* `-o`, `--outdir`: Output directory for generated `.dtbo` files (and for
`--base-dtb` when given without a directory component).
+* `-f`, `--fit`: Input compiled FIT image (`.fit` / `.itb`) to factor in
+ place (or write to `-F`), creating one `fdt-base-N` per compatible
+ board/SoC family.
+* `-F`, `--fit-out`: Output path for the factored FIT image (default:
+ overwrite `--fit`).
* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
`base.dtb` and final reconstructed device trees. When `-L` is passed to
`dtc` solely so `fdtfactor` can locate phandle cells at build time, the
@@ -127,6 +138,10 @@ Options:
`.dtbo` overlays. Omit `-S` if downstream tooling or verification
expects applying a board's `.dtbo` to `base.dtb` to reproduce the
original `.dtb`'s `/__local_fixups__` subtree verbatim.
+* `--single-base-dtb`: In FIT mode (`-f`/`--fit`), force all `flat_dt`
+ images to be factored into a single `fdt-base-1` base `.dtb` regardless of
+ node-path overlap, failing only if their FDT headers (`boot_cpuid_phys`,
+ `/memreserve/`), architecture, or compression differ.
* `--verify`: Verify that applying each generated `.dtbo` to `base.dtb`
with `libfdt` recreates the original `.dtb`.
* `-v`, `--verbose`: Print uncompressed and compressed size savings.
diff --git a/fdtfactor b/fdtfactor
index 9f06801f973c..8a0c996581e5 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -2,21 +2,24 @@
# SPDX-License-Identifier: GPL-2.0-or-later
"""Split compiled device tree blobs (.dtb) into a base .dtb and .dtbo overlays.
-Given a set of compiled device tree blobs, fdtfactor extracts all common nodes
-and properties into a single base .dtb file and writes the per-board
-differences into .dtbo overlay files. Applying an overlay to the base .dtb
-using fdtoverlay or libfdt's fdt_overlay_apply() recreates the corresponding
-original device tree.
+Given a set of compiled device tree blobs (or a compiled FIT image),
+fdtfactor extracts all common nodes and properties into a base .dtb and writes
+the per-board differences into .dtbo overlays. Applying an overlay to the base
+.dtb using fdtoverlay, libfdt's fdt_overlay_apply(), or depthcharge's FIT
+loader recreates the corresponding original device tree.
"""
import argparse
from collections import Counter
+import copy
import gzip
+import hashlib
import lzma
import os
import struct
import subprocess
import sys
+import zlib
import libfdt
@@ -27,8 +30,8 @@ LZ4_LEGACY_MAGIC = b'\x02\x21\x4c\x18'
GZIP_MAGIC = b'\x1f\x8b'
XZ_MAGIC = b'\xfd7zXZ\x00'
# Matches raw LZMA_ALONE headers with default lc=3, lp=0, pb=2 (0x5d) and
-# power-of-two dictionary sizes >= 64 KiB; .lzma extension handles any
-# non-default LZMA headers.
+# power-of-two dictionary sizes >= 64 KiB; .lzma extension or FIT compression
+# property handles any non-default LZMA headers.
LZMA_ALONE_MAGIC = b'\x5d\x00\x00'
COMP_TO_EXT = {
@@ -209,6 +212,51 @@ class NodeTree:
self.props = {}
self.children = {}
+ @classmethod
+ def from_fdt(cls, fdt, nodeoffset=0):
+ """Recursively copy a `libfdt.FdtRo` subtree into a `NodeTree`.
+
+ Args:
+ fdt: A `libfdt.FdtRo` instance.
+ nodeoffset: Node offset in `fdt` to copy (defaults to root `0`).
+
+ Returns:
+ A `NodeTree` containing all properties and subnodes of
+ `nodeoffset`.
+ """
+ node = cls()
+ poff = fdt.first_property_offset(nodeoffset, quiet=libfdt.QUIET_NOTFOUND)
+ while poff >= 0:
+ prop = fdt.get_property_by_offset(poff)
+ node.props[prop.name] = bytes(prop)
+ poff = fdt.next_property_offset(poff, quiet=libfdt.QUIET_NOTFOUND)
+
+ soff = fdt.first_subnode(nodeoffset, quiet=libfdt.QUIET_NOTFOUND)
+ while soff >= 0:
+ name = fdt.get_name(soff)
+ node.children[name] = cls.from_fdt(fdt, soff)
+ soff = fdt.next_subnode(soff, quiet=libfdt.QUIET_NOTFOUND)
+ return node
+
+ def copy(self):
+ """Return a deep copy of this `NodeTree`."""
+ return copy.deepcopy(self)
+
+ def update_fit_hashes(self):
+ """Recompute `value` properties on any FIT `hash*` subnodes of `self`."""
+ data_bytes = self.props.get('data')
+ if data_bytes is None:
+ return
+ for sub_name, sub_node in self.children.items():
+ if not sub_name.startswith('hash'):
+ continue
+ algo = _prop_as_str(sub_node.props.get('algo'))
+ if not algo:
+ raise ValueError(
+ f'FIT hash subnode {sub_name!r} lacks an algo property'
+ )
+ sub_node.props['value'] = _compute_fit_hash(algo, data_bytes)
+
def get_node(self, path):
"""Return the descendant NodeTree at `path`, creating missing nodes.
@@ -337,7 +385,7 @@ class ParsedDtb:
"""Parsed representation of an input .dtb blob."""
def __init__(self, filename, raw_bytes, compressed_size=None,
- compression='none'):
+ compression='none', fdt=None):
"""Parse an uncompressed FDT blob into nodes, symbols, and fixups.
Args:
@@ -347,6 +395,7 @@ class ParsedDtb:
decompression (defaults to `len(raw_bytes)`).
compression: Compression algorithm of the input file (`'none'`,
`'lz4'`, `'lzma'`, `'xz'`, or `'gzip'`).
+ fdt: Optional pre-validated `libfdt.FdtRo` instance for `raw_bytes`.
"""
self.filename = filename
self.raw_bytes = raw_bytes
@@ -354,11 +403,12 @@ class ParsedDtb:
compressed_size if compressed_size is not None else len(raw_bytes)
)
self.compression = compression
- fdt = _open_fdt_ro(raw_bytes, filename)
- if _is_dtbo_overlay(fdt):
- raise ValueError(
- f'{filename}: Input is already a DTB overlay (.dtbo)'
- )
+ if fdt is None:
+ fdt = _open_fdt_ro(raw_bytes, filename)
+ if _is_dtbo_overlay(fdt):
+ raise ValueError(
+ f'{filename}: Input is already a DTB overlay (.dtbo)'
+ )
# Extract FDT header metadata (boot_cpuid_phys and /memreserve/ table)
# stored outside the DT node/property tree; see _validate_dtb_headers()
@@ -1211,6 +1261,588 @@ def verify_recreated_dtb(orig_dtb, base_dtb_blob, dtbo_blob,
)
+# ---------------------------------------------------------------------------
+# Compiled FIT image factoring (targeting depthcharge / coreboot / U-Boot)
+# ---------------------------------------------------------------------------
+
+def _prop_as_str(raw_bytes):
+ """Decode a NUL-terminated FDT string property.
+
+ Args:
+ raw_bytes: Raw property bytes (or None).
+
+ Returns:
+ Decoded string with trailing NULs stripped, or `''` if empty/None.
+ """
+ if not raw_bytes:
+ return ''
+ return raw_bytes.rstrip(b'\x00').decode('utf-8', errors='replace')
+
+
+def _prop_as_str_list(raw_bytes):
+ """Decode a NUL-separated FDT stringlist property.
+
+ Args:
+ raw_bytes: Raw property bytes (or None).
+
+ Returns:
+ List of decoded non-empty strings.
+ """
+ if not raw_bytes:
+ return []
+ parts = raw_bytes.rstrip(b'\x00').split(b'\x00')
+ return [p.decode('utf-8', errors='replace') for p in parts if p]
+
+
+def _compute_fit_hash(algo_name, data_bytes):
+ """Compute a binary hash digest for a FIT image `hash*` subnode.
+
+ Args:
+ algo_name: Hash algorithm name (e.g. `'sha1'`, `'sha256'`, `'crc32'`).
+ data_bytes: Image `data` property bytes to hash.
+
+ Returns:
+ Binary digest as `bytes`.
+ """
+ if algo_name == 'crc32':
+ return struct.pack('>I', zlib.crc32(data_bytes) & 0xffffffff)
+ h = hashlib.new(algo_name)
+ h.update(data_bytes)
+ return h.digest()
+
+
+def _is_dtb_filename_desc(desc):
+ """Return True if `desc` looks like a `.dtb` filename."""
+ if not desc or ' ' in desc:
+ return False
+ _, ext = os.path.splitext(desc)
+ if ext in EXT_TO_COMP:
+ desc = desc[:-len(ext)]
+ return desc.endswith('.dtb') and len(os.path.basename(desc)) > 4
+
+
+def _derive_base_dtb_description(entries, base_dtb_node_name, comp):
+ """Derive a `description` string for a factored base FDT image node.
+
+ Extracts the common prefix across `.dtb` filename descriptions in `entries`
+ and trims any partially matched trailing token before appending
+ `'-base.dtb[.comp]'` (falling back to `'<base_dtb_node_name>.dtb[.comp]'`
+ if no common prefix remains):
+
+ * Diverging at a letter (truncate back to the last `-`, `_`, or `.`):
+ - `['sc7180-trogdor-lazor', 'sc7180-trogdor-pompom']`
+ -> `'sc7180-trogdor-base.dtb'`
+ - `['sc7180-lazor', 'sc7180-limozeen']` -> `'sc7180-base.dtb'`
+ - `['alpha', 'apricot']` -> `'fdt-base-1.dtb'` (no prior separator)
+ * Diverging at a digit inside a token (strip trailing digits from prefix):
+ - `['fdtfactor_board1', 'fdtfactor_board2']` -> `'fdtfactor_board-base.dtb'`
+ - `['sc7180-trogdor', 'sc7280-herobrine']` -> `'sc-base.dtb'`
+ * Diverging at or right after a separator (keep full tokens before
+ separator):
+ - `['sc7180-1', 'sc7180-2']` -> `'sc7180-base.dtb'`
+ - `['sc7180-trogdor-r1', 'sc7180-trogdor-r1-lte']`
+ -> `'sc7180-trogdor-r1-base.dtb'`
+
+ Args:
+ entries: List of FIT `flat_dt` entry dicts in the factor group.
+ base_dtb_node_name: Node name for the base FDT (e.g. `'fdt-base-1'`).
+ comp: Compression algorithm name (`'none'`, `'lz4'`, etc.).
+
+ Returns:
+ Description string (e.g. `'sc7180-trogdor-base.dtb.lz4'`).
+ """
+ stems = []
+ for e in entries:
+ desc = _prop_as_str(e['node'].props.get('description'))
+ if _is_dtb_filename_desc(desc):
+ _, stem, _ = split_dtb_filename(desc)
+ if stem:
+ stems.append(stem)
+ if len(stems) >= 2:
+ prefix = os.path.commonprefix(stems)
+ if any(
+ len(s) > len(prefix) and s[len(prefix)].isalpha()
+ for s in stems
+ ):
+ sep_idx = max(prefix.rfind(c) for c in ('-', '_', '.'))
+ prefix = prefix[:sep_idx] if sep_idx >= 0 else ''
+ elif (
+ any(
+ len(s) > len(prefix) and s[len(prefix)].isdigit()
+ for s in stems
+ )
+ and not prefix.endswith(('-', '_', '.'))
+ ):
+ prefix = prefix.rstrip('0123456789')
+ prefix = prefix.rstrip('-_.')
+ if prefix:
+ return f'{prefix}-base.dtb{COMP_TO_EXT[comp]}'
+ return f'{base_dtb_node_name}.dtb{COMP_TO_EXT[comp]}'
+
+
+def group_fit_dtbs(fdt_entries):
+ """Partition FIT `flat_dt` entries into compatible factor groups.
+
+ DTBs are grouped by `(compression, arch, load, entry, boot_cpuid_phys,
+ mem_rsv)` and node-path overlap (>= 50%) so that a single FIT image
+ containing multiple board/SoC families (e.g. `sc7180-trogdor` and
+ `sc7280-herobrine`) produces one base DTB per family.
+
+ Args:
+ fdt_entries: List of FIT `flat_dt` entry dicts.
+
+ Returns:
+ List of entry lists, one per compatible factor group.
+ """
+ groups = []
+ for entry in fdt_entries:
+ dtb = entry['dtb']
+ header_key = (
+ entry['compression'],
+ entry['arch'],
+ entry['node'].props.get('load'),
+ entry['node'].props.get('entry'),
+ dtb.boot_cpuid_phys,
+ tuple(dtb.mem_rsv),
+ )
+ entry_nodes = set(dtb.nodes.keys())
+
+ placed = False
+ for grp in groups:
+ if grp['key'] != header_key:
+ continue
+ ref_nodes = grp['ref_nodes']
+ union_len = len(entry_nodes | ref_nodes)
+ overlap = (
+ len(entry_nodes & ref_nodes) / union_len if union_len else 1.0
+ )
+ if overlap >= 0.5:
+ grp['entries'].append(entry)
+ placed = True
+ break
+
+ if not placed:
+ groups.append({
+ 'key': header_key,
+ 'ref_nodes': entry_nodes,
+ 'entries': [entry],
+ })
+
+ return [grp['entries'] for grp in groups]
+
+
+def _extract_fit_fdt_entries(fit_in_path, images_node):
+ """Extract and parse non-overlay `flat_dt` entries from `/images`.
+
+ Any `/images` nodes that are already `.dtbo` overlays are skipped and left
+ intact in `/images`.
+
+ Args:
+ fit_in_path: Input FIT file path for error messages.
+ images_node: The `NodeTree` for `/images`.
+
+ Returns:
+ List of parsed `flat_dt` entry dicts.
+ """
+ if any(name.startswith('fdt-base-') for name in images_node.children):
+ raise ValueError(
+ f'{fit_in_path}: FIT image already contains fdt-base-* node(s)'
+ )
+
+ fdt_entries = []
+ for img_name, img_node in images_node.children.items():
+ # Check all /images/* nodes (including kernel/ramdisk) because
+ # re-serializing fit_root would drop external data appended past
+ # the FDT totalsize.
+ if ('data-offset' in img_node.props or
+ 'data-position' in img_node.props):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} uses external data '
+ f'(data-offset/data-position), which is not supported'
+ )
+ if _prop_as_str(img_node.props.get('type')) != 'flat_dt':
+ continue
+ raw_data = img_node.props.get('data')
+ if raw_data is None:
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} lacks a data property'
+ )
+ node_label = f'{fit_in_path}:/images/{img_name}'
+ comp_prop = _prop_as_str(img_node.props.get('compression')) or 'none'
+ dtb_bytes, comp = decompress_blob(
+ raw_data, filename=node_label, comp=comp_prop
+ )
+ fdt_ro = _open_fdt_ro(dtb_bytes, node_label)
+ if _is_dtbo_overlay(fdt_ro):
+ continue
+
+ arch_str = _prop_as_str(img_node.props.get('arch'))
+ parsed_dtb = ParsedDtb(
+ node_label,
+ dtb_bytes,
+ compressed_size=len(raw_data),
+ compression=comp,
+ fdt=fdt_ro,
+ )
+ fdt_entries.append({
+ 'name': img_name,
+ 'node': img_node,
+ 'dtb': parsed_dtb,
+ 'compression': comp,
+ 'arch': arch_str,
+ })
+
+ if len(fdt_entries) < 2:
+ raise ValueError(
+ 'FIT image must contain at least two non-overlay flat_dt '
+ 'nodes to factor'
+ )
+
+ return fdt_entries
+
+
+def _partition_fit_fdt_entries(fdt_entries, single_base_dtb):
+ """Partition `fdt_entries` into factorable groups and validate constraints.
+
+ Args:
+ fdt_entries: List of non-overlay FIT `flat_dt` entry dicts.
+ single_base_dtb: Whether `--single-base-dtb` was requested.
+
+ Returns:
+ List of entry lists to factor (each with `len >= 2`).
+ """
+ if single_base_dtb:
+ ref_entry = fdt_entries[0]
+ for entry in fdt_entries[1:]:
+ if entry['compression'] != ref_entry['compression']:
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to use '
+ f'the same compression: {ref_entry["name"]} has '
+ f'{ref_entry["compression"]!r} vs {entry["name"]} '
+ f'{entry["compression"]!r}'
+ )
+ if entry['arch'] != ref_entry['arch']:
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to use '
+ f'the same arch: {ref_entry["name"]} has '
+ f'{ref_entry["arch"]!r} vs {entry["name"]} '
+ f'{entry["arch"]!r}'
+ )
+ for prop in ('load', 'entry'):
+ if (entry['node'].props.get(prop) !=
+ ref_entry['node'].props.get(prop)):
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to '
+ f'use the same {prop} property: {ref_entry["name"]} vs '
+ f'{entry["name"]}'
+ )
+ _validate_dtb_headers([e['dtb'] for e in fdt_entries])
+ return [fdt_entries]
+
+ groups = group_fit_dtbs(fdt_entries)
+ factorable_groups = [g for g in groups if len(g) >= 2]
+ if not factorable_groups:
+ raise ValueError(
+ 'FIT image must contain at least two compatible '
+ 'flat_dt nodes to factor'
+ )
+ for g in groups:
+ if len(g) == 1:
+ print(
+ f'Note: FIT flat_dt node {g[0]["name"]} has no '
+ f'compatible peers and will be left unfactored',
+ file=sys.stderr,
+ )
+ return factorable_groups
+
+
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups, verify):
+ """Factor one compatible group of FIT `flat_dt` entries.
+
+ Updates each `entry['node']` in place with its compressed `.dtbo` overlay
+ and returns the newly created `base_dtb_node` along with group size
+ statistics.
+
+ Args:
+ entries: List of FIT `flat_dt` entry dicts in the group.
+ base_dtb_node_name: Name for the group's base DTB node (e.g.
+ `'fdt-base-1'`).
+ strip_local_fixups: Whether to strip `/__local_fixups__`.
+ verify: Whether to verify each reconstructed DTB.
+
+ Returns:
+ Tuple of `(base_dtb_node, group_stat)`.
+ """
+ group_dtbs = [e['dtb'] for e in entries]
+ comp = entries[0]['compression']
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ group_dtbs,
+ strip_local_fixups=strip_local_fixups,
+ )
+
+ if verify:
+ for d, dtbo_blob in zip(group_dtbs, dtbo_blobs):
+ verify_recreated_dtb(
+ d, base_dtb_blob, dtbo_blob, strip_local_fixups
+ )
+
+ base_dtb_comp_bytes = compress_blob(base_dtb_blob, comp)
+ base_dtb_node = entries[0]['node'].copy()
+ for prop in ('phandle', 'linux,phandle', 'compatible'):
+ base_dtb_node.props.pop(prop, None)
+ base_dtb_node.children = {
+ k: v
+ for k, v in base_dtb_node.children.items()
+ if k.startswith('hash')
+ }
+ for sub_node in base_dtb_node.children.values():
+ sub_node.props.pop('phandle', None)
+ sub_node.props.pop('linux,phandle', None)
+ if 'description' in base_dtb_node.props:
+ base_dtb_desc = _derive_base_dtb_description(
+ entries, base_dtb_node_name, comp
+ )
+ base_dtb_node.props['description'] = (
+ base_dtb_desc.encode('utf-8') + b'\x00'
+ )
+ base_dtb_node.props['data'] = base_dtb_comp_bytes
+ if 'compression' in base_dtb_node.props or comp != 'none':
+ base_dtb_node.props['compression'] = comp.encode('utf-8') + b'\x00'
+ base_dtb_node.update_fit_hashes()
+
+ dtbo_comp_sizes = []
+ for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ img_node = entry['node']
+ dtbo_comp_bytes = compress_blob(dtbo_blob, comp)
+ dtbo_comp_sizes.append(len(dtbo_comp_bytes))
+
+ if 'description' in img_node.props:
+ old_desc = _prop_as_str(img_node.props['description'])
+ if _is_dtb_filename_desc(old_desc):
+ desc_dir, desc_stem, _ = split_dtb_filename(old_desc)
+ new_desc_name = f'{desc_stem}.dtbo{COMP_TO_EXT[comp]}'
+ new_desc = (
+ os.path.join(desc_dir, new_desc_name)
+ if desc_dir
+ else new_desc_name
+ )
+ img_node.props['description'] = (
+ new_desc.encode('utf-8') + b'\x00'
+ )
+
+ for prop in ('load', 'entry'):
+ img_node.props.pop(prop, None)
+ for sub_name in list(img_node.children):
+ if sub_name.startswith('signature'):
+ print(
+ f'Warning: Stripping invalidated signature subnode '
+ f'/images/{entry["name"]}/{sub_name}',
+ file=sys.stderr,
+ )
+ del img_node.children[sub_name]
+
+ img_node.props['data'] = dtbo_comp_bytes
+ img_node.update_fit_hashes()
+
+ group_stat = {
+ 'base_dtb_name': base_dtb_node_name,
+ 'comp': comp,
+ 'dtbs': group_dtbs,
+ 'base_dtb_blob': base_dtb_blob,
+ 'dtbo_blobs': dtbo_blobs,
+ 'base_dtb_comp_size': len(base_dtb_comp_bytes),
+ 'dtbo_comp_sizes': dtbo_comp_sizes,
+ }
+ return base_dtb_node, group_stat
+
+
+def _update_fit_images_and_configs(images_node, configs_node,
+ insert_before_first, fdt_to_base_dtb):
+ """Insert `fdt-base-*` nodes into `/images` and update `/configurations`.
+
+ Args:
+ images_node: The `NodeTree` for `/images`.
+ configs_node: The `NodeTree` for `/configurations`.
+ insert_before_first: Dict mapping the first `flat_dt` node name of each
+ group to `(base_dtb_node_name, base_dtb_node)`.
+ fdt_to_base_dtb: Dict mapping each factored `flat_dt` node name to
+ `(base_dtb_node_name, orig_dtb)`.
+ """
+ new_images_children = {}
+ for img_name, img_node in images_node.children.items():
+ if img_name in insert_before_first:
+ b_name, b_node = insert_before_first[img_name]
+ new_images_children[b_name] = b_node
+ new_images_children[img_name] = img_node
+ images_node.children = new_images_children
+
+ for conf_name, conf_node in configs_node.children.items():
+ fdt_list = _prop_as_str_list(conf_node.props.get('fdt'))
+ if not fdt_list:
+ continue
+ primary_fdt = fdt_list[0]
+ for extra_fdt in fdt_list[1:]:
+ if extra_fdt in fdt_to_base_dtb:
+ raise ValueError(
+ f'/configurations/{conf_name} lists factored flat_dt '
+ f'{extra_fdt!r} after the primary FDT'
+ )
+ if primary_fdt in fdt_to_base_dtb:
+ base_dtb_name, orig_dtb = fdt_to_base_dtb[primary_fdt]
+ new_fdt_list = [base_dtb_name] + fdt_list
+ conf_node.props['fdt'] = (
+ '\x00'.join(new_fdt_list).encode('utf-8') + b'\x00'
+ )
+ # If the configuration omitted 'compatible' (relying on the
+ # bootloader reading root 'compatible' from fdt_list[0]), copy the
+ # board's root 'compatible' property onto the configuration node
+ # now that fdt_list[0] is the shared base DTB.
+ if 'compatible' not in conf_node.props:
+ root_compat = orig_dtb.nodes.get('/', {}).get('compatible')
+ if root_compat is not None:
+ conf_node.props['compatible'] = root_compat
+ for sub_name in list(conf_node.children):
+ if sub_name.startswith(('signature', 'hash')):
+ print(
+ f'Warning: Stripping invalidated {sub_name} subnode '
+ f'/configurations/{conf_name}/{sub_name}',
+ file=sys.stderr,
+ )
+ del conf_node.children[sub_name]
+
+
+def _print_fit_stats(group_stats, orig_fit_size, new_fit_size):
+ """Print per-group and total FIT image size savings.
+
+ Args:
+ group_stats: List of group statistics dicts from `_factor_fit_group()`.
+ orig_fit_size: Original FIT image size in bytes.
+ new_fit_size: Factored FIT image size in bytes.
+ """
+ for st in group_stats:
+ g_dtbs = st['dtbs']
+ orig_uncomp = sum(len(d.raw_bytes) for d in g_dtbs)
+ base_dtb_uncomp = len(st['base_dtb_blob'])
+ dtbo_uncomp = sum(len(b) for b in st['dtbo_blobs'])
+ new_uncomp = base_dtb_uncomp + dtbo_uncomp
+ saved_uncomp = orig_uncomp - new_uncomp
+ pct_uncomp = (
+ (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
+ )
+ print(
+ f'[{st["base_dtb_name"]}] Uncompressed DTBs: {len(g_dtbs)} '
+ f'files ({orig_uncomp:,} B) -> base {base_dtb_uncomp:,} B + '
+ f'{len(st["dtbo_blobs"])} overlays {dtbo_uncomp:,} B = '
+ f'{new_uncomp:,} B (saved {saved_uncomp:,} B / '
+ f'{pct_uncomp:.1f}%)'
+ )
+ if st['comp'] != 'none':
+ orig_comp = sum(d.compressed_size for d in g_dtbs)
+ base_dtb_csize = st['base_dtb_comp_size']
+ dtbo_csize = sum(st['dtbo_comp_sizes'])
+ new_comp = base_dtb_csize + dtbo_csize
+ saved_comp = orig_comp - new_comp
+ pct_comp = (
+ (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
+ )
+ print(
+ f'[{st["base_dtb_name"]}] Compressed ({st["comp"]}): '
+ f'{len(g_dtbs)} files ({orig_comp:,} B) -> base '
+ f'{base_dtb_csize:,} B + {len(st["dtbo_blobs"])} overlays '
+ f'{dtbo_csize:,} B = {new_comp:,} B '
+ f'(saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ )
+ fit_saved = orig_fit_size - new_fit_size
+ fit_pct = (fit_saved / orig_fit_size * 100.0) if orig_fit_size else 0.0
+ print(
+ f'FIT image total: {orig_fit_size:,} bytes -> {new_fit_size:,} bytes '
+ f'(saved {fit_saved:,} B / {fit_pct:.1f}%)'
+ )
+
+
+def factor_fit_image(fit_in_path, fit_out_path, strip_local_fixups=False,
+ single_base_dtb=False, verify=False, verbose=False):
+ """Factor all `flat_dt` images in a compiled FIT image into base + overlays.
+
+ Args:
+ fit_in_path: Input `.fit` / `.itb` file path.
+ fit_out_path: Output `.fit` / `.itb` file path.
+ strip_local_fixups: Whether to strip `/__local_fixups__` from factored
+ DTBs.
+ single_base_dtb: If True, force all `flat_dt` nodes in the FIT image to
+ be factored into a single `fdt-base-1` base DTB, failing only if
+ their FDT headers, architecture, or compression differ.
+ verify: Whether to verify each board's reconstructed DTB using
+ `fdt_overlay_apply()`.
+ verbose: Whether to print size savings statistics.
+
+ Raises:
+ ValueError: If the FIT image lacks `/images` or `/configurations`,
+ a `flat_dt` node lacks a `data` property, or `single_base_dtb` is
+ requested and the `flat_dt` nodes have incompatible headers,
+ architecture, or compression.
+ """
+ with open(fit_in_path, 'rb') as f:
+ orig_fit_bytes = f.read()
+
+ orig_fit_fdt = _open_fdt_ro(orig_fit_bytes, fit_in_path)
+ fit_root = NodeTree.from_fdt(orig_fit_fdt)
+
+ images_node = fit_root.children.get('images')
+ configs_node = fit_root.children.get('configurations')
+ if images_node is None or configs_node is None:
+ raise ValueError(
+ f'{fit_in_path}: FIT image must contain /images and '
+ f'/configurations nodes'
+ )
+
+ fdt_entries = _extract_fit_fdt_entries(fit_in_path, images_node)
+ factorable_groups = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb
+ )
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+
+ for idx, entries in enumerate(factorable_groups, 1):
+ base_dtb_node_name = f'fdt-base-{idx}'
+
+ base_dtb_node, group_stat = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups, verify
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
+ group_stats.append(group_stat)
+
+ _update_fit_images_and_configs(
+ images_node, configs_node, insert_before_first, fdt_to_base_dtb
+ )
+
+ fit_mem_rsv = [
+ tuple(orig_fit_fdt.get_mem_rsv(i))
+ for i in range(orig_fit_fdt.num_mem_rsv())
+ ]
+ new_fit_bytes = fit_root.to_fdt_blob(
+ mem_rsv=fit_mem_rsv,
+ boot_cpuid_phys=orig_fit_fdt.boot_cpuid_phys(),
+ size_hint=max(len(orig_fit_bytes) + 65536, 65536),
+ )
+
+ out_dir = os.path.dirname(fit_out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(fit_out_path, 'wb') as f:
+ f.write(new_fit_bytes)
+
+ if verbose:
+ _print_fit_stats(group_stats, len(orig_fit_bytes), len(new_fit_bytes))
+
+
def get_dtbo_output_path(dtb_path, outdir, comp='none'):
"""Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
@@ -1238,13 +1870,13 @@ def parse_args():
parser = argparse.ArgumentParser(
description=(
'Split a set of compiled device tree blobs (.dtb, or compressed '
- '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and '
- 'per-board overlay blobs (.dtbo).'
+ '.dtb.lz4/.xz/.lzma/.gz) or a compiled FIT image into common '
+ 'base .dtb(s) and per-board overlay blobs (.dtbo).'
)
)
parser.add_argument(
'-b', '--base-dtb',
- required=True,
+ default=None,
metavar='BASE_DTB',
help=(
'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) file path '
@@ -1260,6 +1892,23 @@ def parse_args():
'filenames (default: alongside each input .dtb)'
),
)
+ parser.add_argument(
+ '-f', '--fit',
+ default=None,
+ metavar='FIT_IMAGE',
+ help=(
+ 'Input compiled FIT image (.fit / .itb) to factor (updated '
+ 'in-place unless -F/--fit-out is specified)'
+ ),
+ )
+ parser.add_argument(
+ '-F', '--fit-out',
+ default=None,
+ metavar='FIT_OUT',
+ help=(
+ 'Output path for the factored FIT image (default: overwrite --fit)'
+ ),
+ )
parser.add_argument(
'-S', '--strip-local-fixups',
action='store_true',
@@ -1268,6 +1917,15 @@ def parse_args():
'reconstructed device trees'
),
)
+ parser.add_argument(
+ '--single-base-dtb',
+ action='store_true',
+ help=(
+ 'Force all flat_dt nodes in a FIT image (-f/--fit) to be '
+ 'factored into a single fdt-base-1 base DTB, failing only if '
+ 'headers, arch, or compression differ'
+ ),
+ )
parser.add_argument(
'--verify',
action='store_true',
@@ -1283,16 +1941,29 @@ def parse_args():
)
parser.add_argument(
'dtbs',
- nargs='+',
+ nargs='*',
metavar='DTB',
help=(
- 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (at '
- 'least two required)'
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (when '
+ '--fit is not used)'
),
)
args = parser.parse_args()
- if len(args.dtbs) < 2:
- parser.error('at least two input .dtb files are required')
+ if args.fit_out and not args.fit:
+ parser.error('-F/--fit-out requires -f/--fit')
+ if args.single_base_dtb and not args.fit:
+ parser.error('--single-base-dtb requires -f/--fit')
+ if args.fit:
+ if args.dtbs or args.base_dtb or args.outdir:
+ parser.error(
+ '-f/--fit cannot be combined with -b/--base-dtb, '
+ '-o/--outdir, or positional DTB arguments'
+ )
+ else:
+ if not args.base_dtb:
+ parser.error('-b/--base-dtb is required when -f/--fit is not used')
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
return args
@@ -1417,7 +2088,18 @@ def main():
"""
args = parse_args()
try:
- factor_dtb_files(args)
+ if args.fit:
+ fit_out = args.fit_out if args.fit_out else args.fit
+ factor_fit_image(
+ args.fit,
+ fit_out,
+ strip_local_fixups=args.strip_local_fixups,
+ single_base_dtb=args.single_base_dtb,
+ verify=args.verify,
+ verbose=args.verbose,
+ )
+ else:
+ factor_dtb_files(args)
except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc:
sys.exit(f'Error: {exc}')
return 0
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index d0bdc4afefb2..0ca2d70457f9 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,7 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.dtb *.test.dtbo *.test.dts *.test.its *.test.fit *.test.dt.yaml *.dtsv1 \
*.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
*.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
tmp.* *.bak \
diff --git a/tests/fdtfactor_other1.dts b/tests/fdtfactor_other1.dts
new file mode 100644
index 000000000000..26fca3816d5a
--- /dev/null
+++ b/tests/fdtfactor_other1.dts
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ model = "Other Board 1";
+ compatible = "vendor,other1", "vendor,other";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ intc: interrupt-controller@8000 {
+ compatible = "vendor,other-intc";
+ reg = <0x8000 0x100>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ clk: clock-controller@9000 {
+ compatible = "vendor,other-clk";
+ reg = <0x9000 0x100>;
+ #clock-cells = <1>;
+ };
+
+ soc@10000000 {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+ interrupt-parent = <&intc>;
+
+ uart0: serial@a000 {
+ compatible = "vendor,other-uart";
+ reg = <0xa000 0x100>;
+ interrupts = <1 0>;
+ clocks = <&clk 1>;
+ status = "okay";
+ };
+
+ i2c0: i2c@b000 {
+ compatible = "vendor,other-i2c";
+ reg = <0xb000 0x100>;
+ interrupts = <2 0>;
+ clocks = <&clk 2>;
+ status = "okay";
+
+ pmic: pmic@50 {
+ compatible = "vendor,other-pmic";
+ reg = <0x50>;
+
+ reg_buck1: buck1 {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ };
+ };
+ };
+
+ spi0: spi@c000 {
+ compatible = "vendor,other-spi";
+ reg = <0xc000 0x100>;
+ clocks = <&clk 3>;
+ vdd-supply = <®_buck1>;
+ cs-supply = <&other1_reg>;
+ };
+ };
+
+ other1_reg: regulator-other1 {
+ compatible = "regulator-fixed";
+ regulator-name = "other1-cs";
+ };
+};
diff --git a/tests/fdtfactor_other2.dts b/tests/fdtfactor_other2.dts
new file mode 100644
index 000000000000..50cbc73759d2
--- /dev/null
+++ b/tests/fdtfactor_other2.dts
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ model = "Other Board 2";
+ compatible = "vendor,other2", "vendor,other";
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ other2_early_reg: regulator-other-early {
+ compatible = "regulator-fixed";
+ regulator-name = "other2-early";
+ };
+
+ intc: interrupt-controller@8000 {
+ compatible = "vendor,other-intc";
+ reg = <0x8000 0x100>;
+ interrupt-controller;
+ #interrupt-cells = <2>;
+ };
+
+ clk: clock-controller@9000 {
+ compatible = "vendor,other-clk";
+ reg = <0x9000 0x100>;
+ #clock-cells = <1>;
+ };
+
+ soc@10000000 {
+ compatible = "simple-bus";
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges;
+ interrupt-parent = <&intc>;
+
+ uart0: serial@a000 {
+ compatible = "vendor,other-uart";
+ reg = <0xa000 0x100>;
+ interrupts = <1 0>;
+ clocks = <&clk 1>;
+ status = "disabled";
+ };
+
+ i2c0: i2c@b000 {
+ compatible = "vendor,other-i2c";
+ reg = <0xb000 0x100>;
+ interrupts = <2 0>;
+ clocks = <&clk 2>;
+ status = "okay";
+
+ pmic: pmic@50 {
+ compatible = "vendor,other-pmic";
+ reg = <0x50>;
+
+ reg_buck1: buck1 {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <3300000>;
+ };
+ };
+ };
+
+ mmc0: mmc@d000 {
+ compatible = "vendor,other-mmc";
+ reg = <0xd000 0x100>;
+ clocks = <&clk 4>;
+ vmmc-supply = <®_buck1>;
+ vqmmc-supply = <&other2_early_reg>;
+ };
+ };
+};
diff --git a/tests/fdtfactor_test.its b/tests/fdtfactor_test.its
new file mode 100644
index 000000000000..bf3b9ef8c439
--- /dev/null
+++ b/tests/fdtfactor_test.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test FIT image for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "none";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb";
+ data = /incbin/("fdtfactor_board3.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb";
+ data = /incbin/("overlay_overlay_phandle.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 92edec8b724c..70a2f5ef6d97 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1177,6 +1177,69 @@ fdtoverlay_tests() {
run_wrap_test check_dtb_recreated \
fdtfactor_board1.test.dtb "fdtfactor_board1.${ext}_recreated.test.dtb"
done
+
+ # Compiled FIT image mode (2 families with <50% node-path overlap: 3 boards + 2 boards)
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_other1.test.dtb "$SRCDIR/fdtfactor_other1.dts"
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_other2.test.dtb "$SRCDIR/fdtfactor_other2.dts"
+
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor_in.test.fit "$SRCDIR/fdtfactor_test.its"
+ run_wrap_test $PYTHON "$FDTFACTOR" -v --verify \
+ -f fdtfactor_in.test.fit -F fdtfactor.test.fit
+ run_fdtget_test "fdtfactor_board-base.dtb" fdtfactor.test.fit /images/fdt-base-1 description -ts
+ run_fdtget_test "fdtfactor_other-base.dtb" fdtfactor.test.fit /images/fdt-base-2 description -ts
+ run_fdtget_test "fdtfactor_board1.test.dtbo" fdtfactor.test.fit /images/fdt-1 description -ts
+ run_fdtget_test "fdtfactor_board2.test.dtbo" fdtfactor.test.fit /images/fdt-2 description -ts
+ run_fdtget_test "fdtfactor_board3.test.dtbo" fdtfactor.test.fit /images/fdt-3 description -ts
+ run_fdtget_test "fdtfactor_other1.test.dtbo" fdtfactor.test.fit /images/fdt-4 description -ts
+ run_fdtget_test "fdtfactor_other2.test.dtbo" fdtfactor.test.fit /images/fdt-5 description -ts
+ run_wrap_test test -n "$($DTGET fdtfactor.test.fit /images/fdt-base-1/hash-1 value -tx)"
+ run_wrap_test test -n "$($DTGET fdtfactor.test.fit /images/fdt-1/hash-1 value -tx)"
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor.test.fit /configurations/conf-1 fdt -ts
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor.test.fit /configurations/conf-2 fdt -ts
+ run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor.test.fit /configurations/conf-3 fdt -ts
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor.test.fit /configurations/conf-4 fdt -ts
+ run_fdtget_test "fdt-base-2 fdt-5" fdtfactor.test.fit /configurations/conf-5 fdt -ts
+ run_fdtget_test "overlay_overlay_phandle.test.dtb" fdtfactor.test.fit /images/fdt-ovl-1 description -ts
+ run_fdtget_test "board-meta-1" fdtfactor.test.fit /configurations/conf-1 description -ts
+ run_fdtget_test "vendor,board1 vendor,common" fdtfactor.test.fit /configurations/conf-1 compatible -ts
+ run_fdtget_test "vendor,board2 vendor,common" fdtfactor.test.fit /configurations/conf-2 compatible -ts
+ run_fdtget_test "vendor,other2 vendor,other" fdtfactor.test.fit /configurations/conf-5 compatible -ts
+
+ # --single-base-dtb combines all boards (even with <50% overlap) into 1 base DTB
+ run_wrap_test $PYTHON "$FDTFACTOR" --verify -S --single-base-dtb \
+ -f fdtfactor_in.test.fit -F fdtfactor_single.test.fit
+ run_fdtget_test "kernel-1\nfdt-base-1\nfdt-1\nfdt-2\nfdt-3\nfdt-4\nfdt-5\nfdt-ovl-1" \
+ fdtfactor_single.test.fit -l /images
+ run_fdtget_test "fdtfactor-base.dtb" fdtfactor_single.test.fit /images/fdt-base-1 description -ts
+ run_fdtget_test "fdt-base-1 fdt-5" fdtfactor_single.test.fit /configurations/conf-5 fdt -ts
+
+ # --single-base-dtb fails when flat_dt nodes have mismatched arch/headers
+ cp fdtfactor_in.test.fit fdtfactor_mismatch.test.fit
+ run_wrap_test $DTPUT -ts fdtfactor_mismatch.test.fit /images/fdt-5 arch arm
+ run_wrap_error_test $PYTHON "$FDTFACTOR" --verify --single-base-dtb \
+ -f fdtfactor_mismatch.test.fit -F fdtfactor_single_fail.test.fit
+
+ # Compressed (.dtb.lz4) FIT image mode (fdtfactor_board{1,2,3}.test.dtb.lz4
+ # were already generated by the compressed test loop above)
+ if command -v lz4 >/dev/null 2>&1; then
+ run_wrap_test lz4 -12 -q -f -m \
+ fdtfactor_other1.test.dtb fdtfactor_other2.test.dtb overlay_overlay_phandle.test.dtb
+ sed -e 's/\.test\.dtb/.test.dtb.lz4/g' \
+ -e 's/compression = "none"/compression = "lz4"/g' \
+ -e 's/description = "fdtfactor_other1/description = "soc99-alpha/' \
+ -e 's/description = "fdtfactor_other2/description = "soc99-beta/' \
+ "$SRCDIR/fdtfactor_test.its" > fdtfactor_lz4.test.its
+ run_dtc_test -I dts -O dtb -o fdtfactor_lz4.test.fit fdtfactor_lz4.test.its
+ run_wrap_test $PYTHON "$FDTFACTOR" --verify \
+ -f fdtfactor_lz4.test.fit
+ run_fdtget_test "lz4" fdtfactor_lz4.test.fit /images/fdt-base-1 compression -ts
+ run_fdtget_test "lz4" fdtfactor_lz4.test.fit /images/fdt-base-2 compression -ts
+ run_fdtget_test "fdtfactor_board-base.dtb.lz4" fdtfactor_lz4.test.fit /images/fdt-base-1 description -ts
+ run_fdtget_test "soc99-base.dtb.lz4" fdtfactor_lz4.test.fit /images/fdt-base-2 description -ts
+ run_fdtget_test "fdtfactor_board1.test.dtbo.lz4" fdtfactor_lz4.test.fit /images/fdt-1 description -ts
+ run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor_lz4.test.fit /configurations/conf-3 fdt -ts
+ run_fdtget_test "fdt-base-2 fdt-5" fdtfactor_lz4.test.fit /configurations/conf-5 fdt -ts
+ fi
fi
}
@@ -1228,9 +1291,9 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
- *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
- *.test.dtb.lz4 *.test.dtbo.lz4
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.its *.test.fit \
+ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
+ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4
for set in $TESTSETS; do
case $set in
--
2.56.0.176.g3575e65682-goog
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (3 preceding siblings ...)
2026-10-15 18:21 ` [PATCH 4/4] fdtfactor: Add compiled FIT image factoring support Douglas Anderson
@ 2026-10-23 4:49 ` David Gibson
4 siblings, 0 replies; 7+ messages in thread
From: David Gibson @ 2026-10-23 4:49 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Chen-Yu Tsai, linux-kernel,
Devicetree, Brian Norris, Tudor Ambarus, André Draszik,
Julius Werner, Saravana Kannan, Peter Griffin
[-- Attachment #1: Type: text/plain, Size: 8181 bytes --]
On Thu, Oct 15, 2026 at 11:21:33AM -0700, Douglas Anderson wrote:
>
> Devicetree overlays allow you to have a "base" compiled devicetree and then
> to apply an "overlay" to it to add or modify nodes. Devicetree overlays have
> at least two purposes:
>
> 1. Supporting add-on boards (like "hats" on a dev board). In this case the
> main board is described as a base devicetree and the add-on board is an
> overlay. In this case there is a clear separation between the base and the
> add-on board and the base/overlay separation models reality nicely.
>
> 2. Supporting a "family" of similar boards in a space-efficient way. In
> upstream Linux, there are 47 distinct sc7180-trogdor boards supported.
> These board are all highly similar, but we compile (and ship) a separate
> devicetree for each. Devicetree overlays can make this much more efficient
> by factoring the common bits to the base devicetree and having each
> of the 47 boards represented by a small devicetree overlay.
>
> Up to this point, the usage of overlays in the upstream devicetree repo has
> mostly been for case #1.
I don't think it invalidates the concept you're going for, but this
description is a bit misleading without some historical context. In
particular this is a bit confused by the fact that there are both
source-level (dts) overlays and binary-level (dtb) overlays.
Source overlays came first, in conjuction with /include/ support in
dtc, and were intended for use case (2) and similar. A group of
boards with the same SoC was the most common case, but families of
similar boards and families of similar SoCs also used this mechanism.
At the time this was still envisaging that a complete tree was built
at boot time, and didn't change.
This started to drift into use case (1) since without hotplug (we'll
get to that), a board with different hats (or whatever) can be
considered variants of the same base board.
That still required the complete tree be built at compile time (or at
least bootloader / kernel installation time). That wasn't great for
two reasons, roughly matching your two use cases:
1. Compile time is obviously no good for hotplugged hats. Even for
coldplug hats, it's desirable to be able to complete the tree at
boot time rather than compile / deploy time.
2. Bootloaders often want to support a variety of related boards, not
just a single configuration. That was already possible by
incorporating a full dtb for every variant, but it gets pretty
bulky and bootloaders are often pretty constrained.
dtb overlays were invented to address these, though I don't know that
difference between the two motivations was particularly well
understood at the time.
dtb overlays were pretty much a direct transcription of dts overlay
semantics to dtb and runtime. The use of "in band" encoding (special
properties) is kind of inelegant, but it worked well enough to become
popular.
This works reasonably well for use case (2). That makes sense: the
semantics are based on dts overlays which were designed for use case
(2).
But, dtb overlays have always been a nasty hack for case (1), even
though that's become the predominant one. Overlays can change
anywhere in the tree, whether or not that's physically sensible via
the hat connector (or whatever). The semantics of applying an overlay
lose information (by design), which means that removing an overlay for
hot unplug is a complete mess. The way we encode the information
necessary to apply an overlay is kind of messy and inconsistent.
Still, they worked well enough for (1) that they've become widespread.
Hervé Condina is now working on a new scheme (dubbed "addons") which
is actually designed for case (1), and should suit it much better.
It's a fair way off being ready for use, but it's a good development.
> Many Android phones attempt to use overlays for case #2 by describing the SoC
> as the base devicetree and then describing boards in overlays. However,
> this solution doesn't get as much savings as it could since there is still
> lots of duplication between the overlays. Upstream reception towards
> structuring boards this way has also been lukewarm [1].
>
> This patch series introduces a tool to help make case #2 a reality. The
> idea (suggested by Konrad Dybcio at LPC) is to continue to represent all the
> boards in a family as completely separate device trees in source code.
> Then, after we've done building everything, have a tool to automatically
> factor things to a base devicetree and a set of overlays.
>
> This tool has quite a few advantages over other solutions:
> 1. No change to the way device trees are stored / arranged upstream.
> 2. No need to deal with some of the awkwardness / limitations of overlays.
> 3. Better savings than just a simple partitioning, like between SoC and
> boards.
I like this idea. It separates the case where the impetus is just to
save binary size from the case where there's a logical separation
between the pieces. It decouples the generation of dtb overlays from
their expression as dts overlays (where you want a logical separation
for human readability).
> I've used this tool to reduce sc7180-trogdor device tree files (which
> are stored lz4 compressed) from 1.71 MB to 213 KB, a 87.5% savings.
Nice. I suspect this gap would be substantially smaller with a large
window compressor. The trick here is that the factoring is finding
long distance redundancies which lz4 won't be able to see.
> NOTE: on Chromebooks like sc7180-trogdor, using overlays like this is
> an optimization. On Android phones like Pixel 10, the bootloader logic
> bakes in the need for overlays. Specifically, if we want to be able to
> build a single image that supports multiple boards and have the bootloader
> auto-construct the right devicetree, we need the boards to be represented
> by overlays. This fdtfactor tool can make that a reality.
>
> The fdtfactor tool can either work directly on a set of compiled devicetree
> files or it can work on FIT (flattened image tree) file. The idea is that
> you'd create the FIT file containing all of the duplicated devicetree
> files and then fdtfactor would "optimize" them.
>
> I'll freely admit that the patches in this series were pretty heavily vibe
> coded. I'll not pretend that I have a deep understanding of the intracicies
> of how overlays are implemented. That being said:
> * I've spent quite a bit of time reviewing the code from a high-level
> perspective: confirming that functions are well documented, well structured,
> and make sense.
> * I believe that what the tool is doing and how the tool is used make sense.
> * I have had quite a number of internal "the AI reviews the AI" rounds to
> the point of diminishing returns.
> At this point, I'm not embarassed to put my name behind this. While I won't
> promise that there is zero cases of the AI doing something really stupid,
> I'd be willing to bet that the end result is better (and cleaner / better
> structured) than what I would have coded up if I had spend a few weeks doing
> this on my own.
Thanks for that clear and helpful context. My views on AI are
complicated, leaning negative, but I do think it can be a useful tool
and have no plans to reject patches purely because they're AI
assisted. A statement like that goes a long way to convincing me that
I'm not about to waste my effort reviewing slop. c.f. my rather
unfriendly response[2] to a recent very sloppish bug report which
vastly overestimated the impact of the bugs in question.
[2] https://lore.kernel.org/devicetree-compiler/as8RYmIp0pDCplpu@gractus.seuss/
Understand that all dtc maintainership is unpaid volunteer time for
me, so I'm less responsive about reviews than would be ideal. I am
gradually chiping away at the backlog, though.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 1/4] tests: Fix Python test environment setup in meson.build
2026-10-15 18:21 ` [PATCH 1/4] tests: Fix Python test environment setup in meson.build Douglas Anderson
@ 2026-10-23 4:51 ` David Gibson
0 siblings, 0 replies; 7+ messages in thread
From: David Gibson @ 2026-10-23 4:51 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Chen-Yu Tsai, linux-kernel,
Devicetree, Brian Norris, Tudor Ambarus, André Draszik,
Julius Werner, Saravana Kannan, Peter Griffin
[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]
On Thu, Oct 15, 2026 at 11:21:34AM -0700, Douglas Anderson wrote:
> Commit ecb21febfdd3 ("meson: port python bindings to build natively via
> meson and meson-python") moved the generated `libfdt.py` and `_libfdt`
> extension module into `<build_root>/pylibfdt`, leaving
> `<source_root>/pylibfdt` with only the source files.
>
> `tests/meson.build` was still setting `PYTHONPATH` to
> `meson.project_source_root() / 'pylibfdt'`, which only went unnoticed
> because `tests/pylibfdt_tests.py` manually inserts `../pylibfdt` into
> `sys.path`. It also checked `if not py.found()` instead of
> `if not pylibfdt_enabled` when setting `NO_PYTHON`.
>
> Update `tests/meson.build` to check `pylibfdt_enabled` and set
> `PYTHONPATH` to `meson.project_build_root() / 'pylibfdt'` so Python
> tools invoked by `run_tests.sh` can import `libfdt` directly.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
This looks like a good fix, independent of the rest of the series: I
already applied it.
> ---
>
> tests/meson.build | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/meson.build b/tests/meson.build
> index c6ac2d7f8ec4..c25500606059 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -114,12 +114,12 @@ endforeach
> run_tests = find_program('run_tests.sh')
>
> env = []
> -if not py.found()
> +if not pylibfdt_enabled
> env += 'NO_PYTHON=1'
> else
> env += [
> 'PYTHON=' + py.full_path(),
> - 'PYTHONPATH=' + meson.project_source_root() / 'pylibfdt',
> + 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt',
> 'NO_PYTHON=0',
> ]
> endif
> --
> 2.56.0.176.g3575e65682-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-10-23 5:04 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-15 18:21 [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-10-15 18:21 ` [PATCH 1/4] tests: Fix Python test environment setup in meson.build Douglas Anderson
2026-10-23 4:51 ` David Gibson
2026-10-15 18:21 ` [PATCH 2/4] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-10-15 18:21 ` [PATCH 3/4] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-10-15 18:21 ` [PATCH 4/4] fdtfactor: Add compiled FIT image factoring support Douglas Anderson
2026-10-23 4:49 ` [PATCH 0/4] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®