* [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
@ 2026-11-16 20:52 ` Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:52 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`README.md` currently jumps directly from building `dtc` and `pylibfdt`
to mailing list information without mentioning `dtc` itself or the
ancillary utilities built and installed alongside it.
Add brief overview sections with basic usage examples for `dtc`,
`dtdiff`, `fdtget`, `fdtput`, `fdtdump`, `convert-dtsv0`, and
`fdtoverlay` (along with a pointer to `Documentation/manual.txt` for
full command-line details).
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch adding README.md blurbs for dtc and ancillary tools (David
Gibson)
README.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 89 insertions(+)
diff --git a/README.md b/README.md
index fcfc2e55d804..9d38c7e79fc6 100644
--- a/README.md
+++ b/README.md
@@ -112,6 +112,95 @@ $ meson setup -Dpython=disabled build
More work remains to support all of libfdt, including access to numeric
values.
+## dtc
+
+`dtc` compiles device tree source (`.dts`) into flattened device tree
+blobs (`.dtb`), decompiles `.dtb` blobs or `/proc/device-tree`-style
+filesystem trees (`fs`) back to `.dts`, emits YAML (`.yaml`, when built
+with `libyaml`) or assembly (`.S`), and runs semantic checks on the
+tree:
+
+```
+$ dtc -I dts -O dtb -o board.dtb board.dts
+$ dtc -I dtb -O dts -o board.dts board.dtb
+```
+
+See `Documentation/manual.txt` for full command-line reference
+documentation for `dtc` and the C utility tools below.
+
+## dtdiff
+
+`dtdiff` normalizes and sorts two device trees via `dtc -s -O dts` and
+prints a unified diff between them. Each input may be a `.dts` source
+file, a `.dtb` / `.dtbo` blob, or a `/proc/device-tree`-style directory:
+
+```
+$ dtdiff before.dtb after.dtb
+$ dtdiff board.dts /proc/device-tree
+```
+
+## fdtget
+
+`fdtget` reads property values (`<node> <property>` pairs), lists
+property names (`-p`, `--properties`), or lists subnode names (`-l`,
+`--list`) from a `.dtb` blob. Use `-t`, `--type` (e.g. `-ts` for
+strings, `-ti` / `-tu` / `-tx` for signed, unsigned, or hex integers, or
+`-tr` for raw bytes) to control formatting, and `-d`, `--default` to
+provide a fallback when a node or property is missing:
+
+```
+$ fdtget board.dtb / model /cpus/cpu@0 reg
+$ fdtget -tx board.dtb /soc/serial@1000 reg
+$ fdtget -l board.dtb /soc
+```
+
+## fdtput
+
+`fdtput` modifies a `.dtb` blob in place: writing property values,
+creating nodes (`-c`, `--create`), deleting nodes and their subnodes
+(`-r`, `--remove`), or deleting properties (`-d`, `--delete`). Pass
+`-t`, `--type` (e.g. `-ts` for strings or `-tx` for hex integers;
+default is decimal integers) to control value encoding, and `-p`,
+`--auto-path` to create any missing nodes along the target path:
+
+```
+$ fdtput -p -ts board.dtb /soc/i2c@3000 compatible vendor,i2c
+$ fdtput -tx board.dtb /soc/serial@1000 reg 1000 100
+$ fdtput -d board.dtb /soc/i2c@3000 compatible
+```
+
+## fdtdump
+
+`fdtdump` is a low-level debugging tool that dumps the FDT header,
+`/memreserve/` entries, and raw structure-block contents of a `.dtb`
+blob (`-d`, `--debug` prints tag offsets; `-s`, `--scan` scans a larger
+file for an embedded FDT). For general decompilation of a `.dtb` to
+`.dts`, use `dtc -I dtb -O dts` instead:
+
+```
+$ fdtdump -d board.dtb
+```
+
+## convert-dtsv0
+
+`convert-dtsv0` converts obsolete version 0 `.dts` source files to
+`/dts-v1/;` syntax, writing each output to `<filename>v1`:
+
+```
+$ convert-dtsv0 old-board.dts
+```
+
+## fdtoverlay
+
+`fdtoverlay` applies one or more compiled `.dtbo` overlays (`-v`,
+`--verbose` for progress) onto a base `.dtb` blob (`-i`, `--input`,
+compiled with `dtc -@` when overlays reference symbols) and writes the
+merged blob to `-o`, `--output`:
+
+```
+$ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
+```
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
@ 2026-11-16 20:52 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:52 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`Property` represents a snapshot of a property's name and value read
from a flattened device tree. Making `Property` a subclass of the
immutable `bytes` type instead of mutable `bytearray` prevents callers
from accidentally mutating a `Property` instance in place when sharing
it across data structures (such as `DeviceTree`), makes `Property`
hashable, and allows `Property` instances to be passed directly to
`Fdt.setprop()` without converting to `bytes` first.
Also return `bytes` instead of `bytearray` from the
`fdt_get_property_by_offset()` SWIG typemap and drop the redundant
`bytearray()` conversion in `FdtRo.getprop()`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch making Property a subclass of bytes instead of bytearray
(David Gibson)
pylibfdt/libfdt.i | 17 +++++++++--------
tests/pylibfdt_tests.py | 1 +
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index fa5f499ed96d..8a7dfed85f15 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -492,7 +492,7 @@ class FdtRo(object):
Returns:
Value of property as a Property object (which can be used as a
- bytearray/string), or -ve error number. On failure, returns an
+ bytes/string), or -ve error number. On failure, returns an
integer error
Raises:
@@ -502,7 +502,7 @@ class FdtRo(object):
quiet)
if pdata[0] is None:
return pdata[1]
- return Property(prop_name, bytearray(pdata[0]))
+ return Property(prop_name, pdata[0])
def hasprop(self, nodeoffset, prop_name, quiet=()):
"""Check if a node has a property
@@ -893,7 +893,7 @@ class Fdt(FdtRo):
return check_err(fdt_del_node(self._fdt, nodeoffset), quiet)
-class Property(bytearray):
+class Property(bytes):
"""Holds a device tree property name and value.
This holds a copy of a property taken from the device tree. It does not
@@ -902,11 +902,12 @@ class Property(bytearray):
Properties:
name: Property name
- value: Property value as a bytearray
+ value: Property value as bytes
"""
- def __init__(self, name, value):
- bytearray.__init__(self, value)
- self.name = name
+ def __new__(cls, name, value):
+ obj = super().__new__(cls, value)
+ obj.name = name
+ return obj
def __copy__(self):
return type(self)(self.name, self)
@@ -1275,7 +1276,7 @@ typedef uint32_t fdt32_t;
if ($1) {
resultobj = PyUnicode_FromString(
fdt_string(fdt1, fdt32_to_cpu($1->nameoff)));
- buff = PyByteArray_FromStringAndSize(
+ buff = PyBytes_FromStringAndSize(
(const char *)($1 + 1), fdt32_to_cpu($1->len));
resultobj = SWIG_AppendOutput(resultobj, buff);
}
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index ae46b29a8f5c..78d8976c0cf3 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -191,6 +191,7 @@ class PyLibfdtBasicTests(unittest.TestCase):
root = 0
poffset = self.fdt.first_property_offset(root)
prop = self.fdt.get_property_by_offset(poffset)
+ self.assertIsInstance(prop, bytes)
self.assertEqual(prop.name, 'compatible')
self.assertEqual(prop, b'test_tree1\0')
for copied in (copy.copy(prop), copy.deepcopy(prop)):
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt()
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`fdt_finish()` relocates the string table directly after the structure
block and updates the FDT header `totalsize` to the packed size, but
`FdtSw.as_fdt()` left the backing Python `bytearray` at the
pre-allocated `FdtSw` buffer size (with trailing zero bytes beyond
`totalsize()`), forcing callers that serialize `fdt.as_bytearray()` to
call `fdt.pack()` just to truncate `_fdt`.
Truncate `fdtsw` to `fdt_totalsize(fdtsw)` in `FdtSw.as_fdt()` (matching
`Fdt.pack()`) so the returned `Fdt`'s `as_bytearray()` length matches
its `totalsize()` immediately.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch trimming the returned Fdt bytearray in FdtSw.as_fdt() so
DeviceTree.flatten() does not need to call fdt.pack() (David Gibson)
pylibfdt/libfdt.i | 1 +
tests/pylibfdt_tests.py | 2 ++
2 files changed, 3 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 8a7dfed85f15..68f22e316ece 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -1026,6 +1026,7 @@ class FdtSw(FdtRo):
fdtsw = bytearray(self._fdt)
while self.check_space(fdt_finish(fdtsw)):
fdtsw = bytearray(self._fdt)
+ del fdtsw[fdt_totalsize(fdtsw):]
return Fdt(fdtsw)
def check_space(self, val):
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 78d8976c0cf3..26915a015ccb 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -684,6 +684,7 @@ class PyLibfdtSwTests(unittest.TestCase):
# Make sure we did at least two resizes
self.assertTrue(len(fdt.as_bytearray()) > FdtSw.INC_SIZE * 2)
+ self.assertEqual(fdt.totalsize(), len(fdt.as_bytearray()))
# Check that as_fdt() resizes when fewer than 4 bytes remain for FDT_END
sw_exact = FdtSw(76)
@@ -692,6 +693,7 @@ class PyLibfdtSwTests(unittest.TestCase):
sw_exact.end_node()
fdt_exact = sw_exact.as_fdt()
self.assertEqual(80, fdt_exact.totalsize())
+ self.assertEqual(80, len(fdt_exact.as_bytearray()))
class PyLibfdtRoTests(unittest.TestCase):
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (2 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`libfdt.Property` provides `as_*()` methods (`as_cell`, `as_uint32`,
`as_int32`, `as_uint64`, `as_int64`, `as_list`, `as_uint32_list`,
`as_int32_list`, `as_uint64_list`, `as_int64_list`, `as_str`, and
`as_stringlist`) to decode typed values from a `Property`, but lacks the
corresponding creation methods to construct a `Property` from typed
values.
Add `from_*()` classmethods mirroring each `as_*()` method so callers
(such as `DeviceTree` users) can construct typed `Property` instances
directly without duplicating encoding helpers on `DeviceTree`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch adding Property.from_*() creation classmethods so
DeviceTree does not need its own get_*/set_* wrapper methods (David
Gibson)
pylibfdt/libfdt.i | 56 +++++++++++++++++++++++++++++++++++++++++
tests/pylibfdt_tests.py | 49 ++++++++++++++++++++++++++++++++++++
2 files changed, 105 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 68f22e316ece..ff9210d123cb 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -962,6 +962,62 @@ class Property(bytes):
parts = self[:-1].split(b'\x00')
return list(map(lambda x: x.decode('utf-8'), parts))
+ @classmethod
+ def from_cell(cls, name, fmt, val):
+ return cls(name, struct.pack('>' + fmt, val))
+
+ @classmethod
+ def from_uint32(cls, name, val):
+ return cls.from_cell(name, 'L', val)
+
+ @classmethod
+ def from_int32(cls, name, val):
+ return cls.from_cell(name, 'l', val)
+
+ @classmethod
+ def from_uint64(cls, name, val):
+ return cls.from_cell(name, 'Q', val)
+
+ @classmethod
+ def from_int64(cls, name, val):
+ return cls.from_cell(name, 'q', val)
+
+ @classmethod
+ def from_list(cls, name, fmt, vals):
+ vals = list(vals)
+ return cls(name, struct.pack(f'>{len(vals)}{fmt}', *vals))
+
+ @classmethod
+ def from_uint32_list(cls, name, vals):
+ return cls.from_list(name, 'L', vals)
+
+ @classmethod
+ def from_int32_list(cls, name, vals):
+ return cls.from_list(name, 'l', vals)
+
+ @classmethod
+ def from_uint64_list(cls, name, vals):
+ return cls.from_list(name, 'Q', vals)
+
+ @classmethod
+ def from_int64_list(cls, name, vals):
+ return cls.from_list(name, 'q', vals)
+
+ @classmethod
+ def from_str(cls, name, val):
+ """Unicode is supported by encoding to UTF-8"""
+ if '\x00' in val:
+ raise ValueError('String value contains embedded nul characters')
+ return cls(name, val.encode('utf-8') + b'\x00')
+
+ @classmethod
+ def from_stringlist(cls, name, vals):
+ """Unicode is supported by encoding to UTF-8"""
+ vals = list(vals)
+ if any('\x00' in v for v in vals):
+ raise ValueError('String value contains embedded nul characters')
+ return cls(name, b''.join(v.encode('utf-8') + b'\x00' for v in vals))
+
class FdtSw(FdtRo):
"""Software interface to create a device tree from scratch
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 26915a015ccb..e2f9c6d5afe4 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -414,6 +414,16 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(9223372036854775807,
self.get_prop("prop-uint64").as_uint64())
self.assertEqual(-2, self.get_prop("prop-int64").as_int64())
+ self.assertEqual(self.get_prop("prop-hex32"),
+ libfdt.Property.from_uint32("prop-hex32", 0xdeadbeef))
+ self.assertEqual(self.get_prop("prop-int32"),
+ libfdt.Property.from_int32("prop-int32", -2))
+ self.assertEqual(
+ self.get_prop("prop-uint64"),
+ libfdt.Property.from_uint64("prop-uint64", 9223372036854775807),
+ )
+ self.assertEqual(self.get_prop("prop-int64"),
+ libfdt.Property.from_int64("prop-int64", -2))
def testGetIntListProperties(self):
"""Test that we can access properties as integer lists"""
@@ -425,6 +435,28 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.get_prop("prop-int64-array").as_int64_list())
self.assertEqual([0x100000000, 0x1],
self.get_prop("prop-uint64-array").as_uint64_list())
+ self.assertEqual(
+ self.get_prop("prop-int32-array"),
+ libfdt.Property.from_int32_list("prop-int32-array", [128, -16, -2]),
+ )
+ self.assertEqual(
+ self.get_prop("prop-uint32-array"),
+ libfdt.Property.from_uint32_list(
+ "prop-uint32-array", [0x1, 0x98765432, 0xdeadbeef]
+ ),
+ )
+ self.assertEqual(
+ self.get_prop("prop-int64-array"),
+ libfdt.Property.from_int64_list(
+ "prop-int64-array", [0x100000000, -2]
+ ),
+ )
+ self.assertEqual(
+ self.get_prop("prop-uint64-array"),
+ libfdt.Property.from_uint64_list(
+ "prop-uint64-array", [0x100000000, 0x1]
+ ),
+ )
def testGetStringlistProperties(self):
"""Test that we can access properties as string list"""
@@ -434,6 +466,17 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(["this is a placeholder string", "string2"],
self.fdt.getprop(node, "placeholder").as_stringlist())
self.assertEqual([], libfdt.Property('empty', b'').as_stringlist())
+ self.assertEqual(
+ self.fdt.getprop(node, "compatible"),
+ libfdt.Property.from_stringlist(
+ "compatible", ["subsubnode1", "subsubnode"]
+ ),
+ )
+ self.assertEqual(
+ b'', libfdt.Property.from_stringlist("empty", [])
+ )
+ with self.assertRaises(ValueError):
+ libfdt.Property.from_stringlist("bad", ["embedded\x00nul"])
with self.assertRaises(ValueError) as e:
libfdt.Property('bad', b'no-nul').as_stringlist()
self.assertIn('lacks nul termination', str(e.exception))
@@ -552,6 +595,12 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.fdt.setprop_str(node, prop, TEST_STRING_3)
self.assertEqual(TEST_STRING_3,
self.fdt.getprop(node, prop).as_str())
+ self.assertEqual(
+ self.fdt.getprop(node, prop),
+ libfdt.Property.from_str(prop, TEST_STRING_3),
+ )
+ with self.assertRaises(ValueError):
+ libfdt.Property.from_str(prop, 'embedded\x00nul')
def testHasProp(self):
"""Test that we can check if a node has a property"""
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (3 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`Property.as_list()` currently decodes multi-cell properties via
`list(map(lambda x: x[0], struct.iter_unpack('>' + fmt, self)))`, which
constructs an iterator and `map` wrapper and invokes a Python `lambda`
for every cell.
Compute the element size via `struct.calcsize('>' + fmt)` and unpack all
elements in a single C `struct.unpack()` call instead.
On one stress test (decoding ~301k `/__local_fixups__` cell lists across
438 arm64 Qualcomm DTBs) this reduced `Property.as_list()` time by
~2.1x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up Property.as_list() with a single struct.unpack()
call
pylibfdt/libfdt.i | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index ff9210d123cb..e4a9357d278a 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -931,7 +931,8 @@ class Property(bytes):
return self.as_cell('q')
def as_list(self, fmt):
- return list(map(lambda x: x[0], struct.iter_unpack('>' + fmt, self)))
+ size = struct.calcsize('>' + fmt)
+ return list(struct.unpack(f'>{len(self) // size}{fmt}', self))
def as_uint32_list(self):
return self.as_list('L')
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (4 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Iterating over a node's properties via `FdtRo.properties()` currently
crosses the Python/SWIG boundary `1 + 2 * num_props` times per node
(`first_property_offset()`, `get_property_by_offset()`, and
`next_property_offset()`, along with their Python `check_err()` and
`check_err_null()` wrappers). When walking every property across large
device trees from Python, those per-property SWIG round-trips dominate
runtime.
Add a C helper `fdt_node_properties_stub()` in `pylibfdt/libfdt.i`
backing `FdtRo.properties()` so all `Property` objects for a node are
collected in a single C call per node, interning property name strings
via `PyUnicode_InternFromString()`. Also add an invalid-offset exception
check to `testPropertiesIterator` in `tests/pylibfdt_tests.py`.
On one stress test (walking all ~573k nodes and ~2.06M properties across
438 arm64 Qualcomm DTBs) this reduced `FdtRo.properties()` time by
~1.8x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up FdtRo.properties() with a C helper
pylibfdt/libfdt.i | 64 ++++++++++++++++++++++++++++++++++++++---
tests/pylibfdt_tests.py | 3 ++
2 files changed, 63 insertions(+), 4 deletions(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index e4a9357d278a..333a25f27946 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -24,6 +24,61 @@ static int fdt_property_stub(void *fdt, const char *name, const void *val,
return fdt_property(fdt, name, val, len);
}
+static PyObject *fdt_node_properties_stub(const void *fdt, int nodeoffset,
+ PyObject *prop_cls)
+{
+ PyObject *list = PyList_New(0);
+ int poff;
+
+ if (!list)
+ return NULL;
+
+ poff = fdt_first_property_offset(fdt, nodeoffset);
+ while (poff >= 0) {
+ const char *pname;
+ int len;
+ const void *val = fdt_getprop_by_offset(fdt, poff, &pname, &len);
+ PyObject *name_obj, *val_obj, *prop_obj;
+ int rc;
+
+ if (!val) {
+ Py_DECREF(list);
+ return PyLong_FromLong(len);
+ }
+ name_obj = PyUnicode_InternFromString(pname);
+ if (!name_obj) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ val_obj = PyBytes_FromStringAndSize((const char *)val, len);
+ if (!val_obj) {
+ Py_DECREF(name_obj);
+ Py_DECREF(list);
+ return NULL;
+ }
+ prop_obj = PyObject_CallFunctionObjArgs(prop_cls, name_obj, val_obj,
+ NULL);
+ Py_DECREF(name_obj);
+ Py_DECREF(val_obj);
+ if (!prop_obj) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ rc = PyList_Append(list, prop_obj);
+ Py_DECREF(prop_obj);
+ if (rc < 0) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ poff = fdt_next_property_offset(fdt, poff);
+ }
+ if (poff != -FDT_ERR_NOTFOUND) {
+ Py_DECREF(list);
+ return PyLong_FromLong(poff);
+ }
+ return list;
+}
+
%}
%pythoncode %{
@@ -477,10 +532,9 @@ class FdtRo(object):
Raises:
FdtException on error (e.g. invalid nodeoffset)
"""
- poff = self.first_property_offset(nodeoffset, quiet=QUIET_NOTFOUND)
- while poff >= 0:
- yield self.get_property_by_offset(poff)
- poff = self.next_property_offset(poff, quiet=QUIET_NOTFOUND)
+ return iter(check_err(
+ fdt_node_properties_stub(self._fdt, nodeoffset, Property)
+ ))
def getprop(self, nodeoffset, prop_name, quiet=()):
"""Get a property from a node
@@ -1420,5 +1474,7 @@ int fdt_property_cell(void *fdt, const char *name, uint32_t val);
* function and a struct, which confuses SWIG.
*/
int fdt_property_stub(void *fdt, const char *name, const void *val, int len);
+PyObject *fdt_node_properties_stub(const void *fdt, int nodeoffset,
+ PyObject *prop_cls);
%include <libfdt.h>
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index e2f9c6d5afe4..9c0104bfbabb 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -270,6 +270,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(props[0], b'subnode1\0')
leaf = self.fdt.path_offset('/subnode@1/ss1')
self.assertEqual(list(self.fdt.properties(leaf)), [])
+ with self.assertRaises(FdtException) as e:
+ list(self.fdt.properties(-1))
+ self.assertEqual(e.exception.err, -libfdt.BADOFFSET)
def testFirstNextSubnodeOffsetExceptions(self):
"""Check except handling for first/next subnode functions"""
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (5 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`pylibfdt`'s `Fdt` and `FdtRo` classes operate directly on flattened
device tree blobs, where inserting or deleting nodes and properties
shifts subsequent node offsets. That works well for in-place queries and
localized edits, but makes multi-tree comparison, structural diffing,
and bottom-up tree construction (as needed by the upcoming `fdtfactor`
tool) awkward and error-prone.
Add `pylibfdt/libpydt.py` providing `DeviceTree`, an in-memory live tree
representation where each node holds `props` (a dict mapping property
name to `libfdt.Property`) and `children` (a dict mapping child node
name to child `DeviceTree`), along with `unflatten()` and `flatten()` to
convert between `FdtRo`/`Fdt` blobs and live trees, and helpers for path
lookup, node creation, and pre-order traversal.
Add unit tests in `tests/libpydt_tests.py` covering `DeviceTree`
round-tripping, subtree unflattening, duplicate-name and corrupt-tree
rejection, and navigation helpers.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Start the commit message with the motivation for a live tree and
condense the summary (Simon Glass)
- Rename devicetree.py to libpydt.py (under pylibfdt/) to avoid
colliding with PyPI's python-devicetree package, and move its unit
tests into a separate tests/libpydt_tests.py file (Simon Glass, David
Gibson)
- Unflatten the tree in a single O(n) pass using fdt.next_node() instead
of recursive O(d * n) fdt.subnodes() traversal, raising FdtException
on corrupt/truncated trees, and add a unit test (David Gibson, Simon
Glass)
- Implement DeviceTree.walk() iteratively with an explicit stack instead
of recursive yield from, and populate node.props directly in
unflatten()
- Drop redundant fdt.pack() call from DeviceTree.flatten() (David
Gibson)
- Drop get_*/set_* typed wrapper methods from DeviceTree in favor of
Property.as_*() and Property.from_*() plus DeviceTree.set_prop()
(David Gibson)
- Drop unnecessary copy.deepcopy() and compare unordered node/child sets
instead of ordered lists in round-trip test (David Gibson)
Changes in v3:
- Move NodeTree out of pylibfdt/libfdt.i into a new devicetree.py module
and rename it to DeviceTree (David Gibson)
- Store libfdt.Property objects in self.props instead of raw bytes
(David Gibson)
- Return a packed libfdt.Fdt object from flatten() instead of raw bytes
and use fdt.set_boot_cpuid_phys() (David Gibson, Simon Glass)
- Raise ValueError in unflatten() on duplicate property or sibling
subnode names (Simon Glass)
Changes in v2:
- Split live tree representation out of fdtfactor (David Gibson)
pylibfdt/libpydt.py | 184 +++++++++++++++++++++++++++++++++++++++++
pylibfdt/meson.build | 2 +
tests/libpydt_tests.py | 106 ++++++++++++++++++++++++
tests/meson.build | 2 +-
tests/run_tests.sh | 13 ++-
5 files changed, 302 insertions(+), 5 deletions(-)
create mode 100644 pylibfdt/libpydt.py
create mode 100644 tests/libpydt_tests.py
diff --git a/pylibfdt/libpydt.py b/pylibfdt/libpydt.py
new file mode 100644
index 000000000000..6d8501f432ca
--- /dev/null
+++ b/pylibfdt/libpydt.py
@@ -0,0 +1,184 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+"""libpydt - Non-flattened in-memory device tree representation in Python."""
+
+import libfdt
+
+
+class DeviceTree:
+ """In-memory live tree representation of a devicetree node and its subtree.
+
+ `props` and `children` are public dictionaries intended to be accessed and
+ manipulated directly by callers (e.g. `node.props['status'].as_str()`,
+ `node.props.get('reg')`, `node.children.get('soc')`), with `set_prop()`
+ provided as a convenience for inserting a `libfdt.Property` under its own
+ `prop.name`.
+
+ Attributes:
+ props: Dict mapping property name (str) to `libfdt.Property`.
+ children: Dict mapping child node name (str) to child `DeviceTree`.
+ """
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+
+ @classmethod
+ def unflatten(cls, fdt, offset=0):
+ """Unflatten an FdtRo subtree starting at `offset` into a DeviceTree.
+
+ Args:
+ fdt: An `FdtRo` (or `Fdt`) instance.
+ offset: Starting node offset (default 0 for the root node).
+
+ Returns:
+ A `DeviceTree` representing the node at `offset` and its
+ descendants.
+
+ Raises:
+ ValueError: If a node contains duplicate property names or
+ duplicate sibling subnode names.
+ """
+ stack = []
+ depth = 0
+ while depth >= 0:
+ node = cls()
+ props = node.props
+ for prop in fdt.properties(offset):
+ name = prop.name
+ if name in props:
+ raise ValueError(f'Duplicate property {name!r}')
+ props[name] = prop
+ if depth == 0:
+ stack = [node]
+ else:
+ parent = stack[depth - 1]
+ subname = fdt.get_name(offset)
+ if subname in parent.children:
+ raise ValueError(f'Duplicate subnode {subname!r}')
+ parent.children[subname] = node
+ del stack[depth:]
+ stack.append(node)
+ offset, depth = fdt.next_node(offset, depth)
+ if offset < 0:
+ libfdt.check_err(offset)
+ return stack[0]
+
+ def _flatten_subtree(self, sw, name):
+ """Recursively write this DeviceTree and its children into an FdtSw.
+
+ Args:
+ sw: An `FdtSw` sequential-write instance.
+ name: Node name for this subtree (empty string `''` for root).
+ """
+ with sw.add_node(name):
+ for prop_name, val in self.props.items():
+ sw.property(prop_name, val)
+ for child_name, child_node in self.children.items():
+ child_node._flatten_subtree(sw, child_name)
+
+ def flatten(self, mem_rsv=(), boot_cpuid_phys=0):
+ """Serialize this root DeviceTree into a packed `libfdt.Fdt` object.
+
+ Args:
+ mem_rsv: Iterable of (address, size) tuples for the FDT memory
+ reservation block.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+
+ Returns:
+ Packed `libfdt.Fdt` instance.
+ """
+ sw = libfdt.FdtSw()
+ for addr, size in mem_rsv:
+ sw.add_reservemap_entry(addr, size)
+ sw.finish_reservemap()
+ self._flatten_subtree(sw, '')
+ fdt = sw.as_fdt()
+ fdt.set_boot_cpuid_phys(boot_cpuid_phys)
+ return fdt
+
+ def set_prop(self, prop):
+ """Insert or replace a `libfdt.Property` in `self.props` under `prop.name`.
+
+ Args:
+ prop: `libfdt.Property` instance to store at `self.props[prop.name]`.
+ """
+ self.props[prop.name] = prop
+
+ def get_node(self, path):
+ """Look up a descendant DeviceTree by path without modifying the tree.
+
+ Follows the slash-separated node names in `path` starting from `self`
+ (leading, trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically, and `''` or
+ `'/'` returns `self`). If any component along `path` is not present in
+ `children`, returns `None`.
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` at `path`, or `None` if not found.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ node = node.children.get(part)
+ if node is None:
+ return None
+ return node
+
+ def create_path(self, path):
+ """Return the descendant DeviceTree at `path`, creating missing nodes.
+
+ Looks up a child or deeper descendant `DeviceTree` by following the
+ slash-separated node names in `path` starting from `self` (leading,
+ trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically). If any
+ node along `path` does not exist yet in `children`, a new empty
+ `DeviceTree` is created for it automatically (similar to `mkdir -p`).
+ Passing `''` or `'/'` returns `self`.
+
+ Example:
+ root = DeviceTree()
+ # Automatically creates root.children['soc'] and
+ # root.children['soc'].children['i2c@4000'], and returns the
+ # 'i2c@4000' DeviceTree.
+ i2c = root.create_path('/soc/i2c@4000')
+ i2c.set_prop(libfdt.Property.from_str('status', 'okay'))
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` instance at `path`.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ if part not in node.children:
+ node.children[part] = type(self)()
+ node = node.children[part]
+ return node
+
+ def walk(self, path=''):
+ """Yield `(path, node)` pairs for this subtree in pre-order.
+
+ Args:
+ path: Path prefix for `self` (defaults to `''` for the root node,
+ producing paths like `''`, `'/soc'`, `'/soc/i2c@4000'`).
+
+ Yields:
+ Tuple of `(path, node)` for `self` followed in pre-order by each
+ descendant `DeviceTree`.
+ """
+ stack = [(path, self)]
+ while stack:
+ cur_path, node = stack.pop()
+ yield cur_path, node
+ prefix = cur_path.rstrip('/')
+ for name, child in reversed(node.children.items()):
+ stack.append((f'{prefix}/{name}', child))
diff --git a/pylibfdt/meson.build b/pylibfdt/meson.build
index b13d8021497f..ccf6c1bf78a6 100644
--- a/pylibfdt/meson.build
+++ b/pylibfdt/meson.build
@@ -19,3 +19,5 @@ pylibfdt = py.extension_module(
dependencies: [libfdt_dep, py.dependency()],
install: true,
)
+
+py.install_sources('libpydt.py', pure: false)
diff --git a/tests/libpydt_tests.py b/tests/libpydt_tests.py
new file mode 100644
index 000000000000..0f3d8d68945a
--- /dev/null
+++ b/tests/libpydt_tests.py
@@ -0,0 +1,106 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+# libpydt - Tests for in-memory DeviceTree representation in Python
+#
+
+import sys
+import unittest
+
+sys.path.insert(0, '../pylibfdt')
+import libfdt
+from libfdt import FdtSw, Property
+import libpydt
+
+
+class DeviceTreeTests(unittest.TestCase):
+ """Test class for libpydt.DeviceTree"""
+
+ def setUp(self):
+ with open('test_tree1.dtb', mode='rb') as f:
+ self.fdt = libfdt.FdtRo(f.read())
+
+ def testRoundTrip(self):
+ """Test unflattening and flattening a tree via DeviceTree"""
+ tree = libpydt.DeviceTree.unflatten(self.fdt)
+ self.assertIsInstance(tree.props['compatible'], Property)
+ rsvmap = self.fdt.get_mem_rsv_list()
+ cpuid = self.fdt.boot_cpuid_phys()
+ fdt2 = tree.flatten(rsvmap, cpuid)
+ self.assertIsInstance(fdt2, libfdt.Fdt)
+ self.assertEqual(rsvmap, fdt2.get_mem_rsv_list())
+ self.assertEqual(cpuid, fdt2.boot_cpuid_phys())
+
+ tree2 = libpydt.DeviceTree.unflatten(fdt2)
+ nodes1 = {
+ path: (n.props, set(n.children.keys()))
+ for path, n in tree.walk()
+ }
+ nodes2 = {
+ path: (n.props, set(n.children.keys()))
+ for path, n in tree2.walk()
+ }
+ self.assertEqual(nodes1, nodes2)
+
+ def testUnflattenSubtree(self):
+ """Test unflattening a non-root subtree offset"""
+ suboff = self.fdt.path_offset('/subnode@1')
+ sub = libpydt.DeviceTree.unflatten(self.fdt, suboff)
+ self.assertEqual('subnode1', sub.props['compatible'].as_str())
+ self.assertEqual({'subsubnode', 'ss1'}, set(sub.children.keys()))
+
+ def testDuplicates(self):
+ """Test that DeviceTree.unflatten() rejects duplicate names"""
+ sw_prop = FdtSw()
+ sw_prop.finish_reservemap()
+ with sw_prop.add_node(''):
+ sw_prop.property_u32('dup', 1)
+ sw_prop.property_u32('dup', 2)
+ with self.assertRaises(ValueError):
+ libpydt.DeviceTree.unflatten(sw_prop.as_fdt())
+
+ sw_node = FdtSw()
+ sw_node.finish_reservemap()
+ with sw_node.add_node(''):
+ with sw_node.add_node('dup'):
+ pass
+ with sw_node.add_node('dup'):
+ pass
+ with self.assertRaises(ValueError):
+ libpydt.DeviceTree.unflatten(sw_node.as_fdt())
+
+ def testCorruptTree(self):
+ """Test that DeviceTree.unflatten() raises FdtException on corrupt trees"""
+ with open('truncated_property.dtb', mode='rb') as f:
+ fdt_trunc = libfdt.FdtRo(f.read())
+ with self.assertRaises(libfdt.FdtException):
+ libpydt.DeviceTree.unflatten(fdt_trunc)
+
+ # Truncate size_dt_struct so the tree ends mid-subnode
+ corrupt = self.fdt.as_bytearray()
+ fdt_corrupt = libfdt.FdtRo(corrupt)
+ fdt_corrupt._fdt[36:40] = (32).to_bytes(4, 'big')
+ with self.assertRaises(libfdt.FdtException):
+ libpydt.DeviceTree.unflatten(fdt_corrupt)
+
+ def testNavigationAndProps(self):
+ """Test DeviceTree navigation and property setting"""
+ root = libpydt.DeviceTree()
+ self.assertIs(root, root.get_node(''))
+ self.assertIs(root, root.get_node('/'))
+ self.assertIsNone(root.get_node('/soc/i2c@4000'))
+
+ i2c = root.create_path('/soc/i2c@4000')
+ self.assertIs(i2c, root.get_node('/soc/i2c@4000'))
+ self.assertIs(i2c, root.get_node('soc/i2c@4000'))
+ self.assertEqual({'', '/soc', '/soc/i2c@4000'},
+ {path for path, _ in root.walk()})
+ self.assertEqual({'/', '/soc', '/soc/i2c@4000'},
+ {path for path, _ in root.walk('/')})
+
+ i2c.set_prop(Property.from_str('status', 'okay'))
+ self.assertEqual('okay', i2c.props['status'].as_str())
+ i2c.set_prop(Property.from_uint32('#address-cells', 1))
+ self.assertEqual(1, i2c.props['#address-cells'].as_uint32())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tests/meson.build b/tests/meson.build
index c25500606059..e37567864e04 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -119,7 +119,7 @@ if not pylibfdt_enabled
else
env += [
'PYTHON=' + py.full_path(),
- 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt',
+ 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt' + ':' + meson.project_source_root() / 'pylibfdt',
'NO_PYTHON=0',
]
endif
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index f750faa1f60a..3cacb68438df 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1149,17 +1149,16 @@ fdtoverlay_tests() {
run_wrap_test test "$bd" = "$pd"
}
-pylibfdt_tests () {
- run_dtc_test -I dts -O dtb -o test_props.dtb "$SRCDIR/test_props.dts"
+run_python_tests () {
TMP=/tmp/tests.stderr.$$
- $PYTHON "$SRCDIR/pylibfdt_tests.py" -v 2> $TMP
+ $PYTHON "$1" -v 2> $TMP
# Use the 'ok' message meaning the test passed, 'ERROR' meaning it failed
# and the summary line for total tests (e.g. 'Ran 17 tests in 0.002s').
# We could add pass + fail to get total tests, but this provides a useful
# sanity check.
pass_count=$(grep "ok$" $TMP | wc -l)
- fail_count=$(grep "^ERROR: " $TMP | wc -l)
+ fail_count=$(grep "^ERROR: \|^FAIL: " $TMP | wc -l)
total_tests=$(sed -n 's/^Ran \([0-9]*\) tests.*$/\1/p' $TMP)
cat $TMP
rm $TMP
@@ -1170,6 +1169,12 @@ pylibfdt_tests () {
tot_tests=$((tot_tests + $total_tests))
}
+pylibfdt_tests () {
+ run_dtc_test -I dts -O dtb -o test_props.dtb "$SRCDIR/test_props.dts"
+ run_python_tests "$SRCDIR/pylibfdt_tests.py"
+ run_python_tests "$SRCDIR/libpydt_tests.py"
+}
+
while getopts "vt:me" ARG ; do
case $ARG in
"v")
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr()
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (6 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`fdt_find_string_len_()` currently calls `memcmp()` at every byte offset
across the string table (`for (p = strtab; p <= last; p++)`). When
building or modifying trees with large string tables (such as DTBs and
overlays with hundreds of `/__symbols__` or `/__fixups__` labels via
`fdt_sw` or `fdt_rw`), scanning multi-kilobyte string tables byte by
byte dominates serialization time.
Advance `p` between candidate first-byte matches using `memchr()` (with
an explicit `slen == 0` check for empty property names and a `tabsize <
slen + 1` bounds check before computing `last`), preserving identical
suffix-sharing behavior.
On one stress test (running the fdtfactor tool across 438 arm64 Qualcomm
DTBs) this reduced `fdt_property()` time by ~5.6x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up fdt_find_string_len_() with memchr()
libfdt/fdt.c | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/libfdt/fdt.c b/libfdt/fdt.c
index d9ccb70345d0..c7cedfba7815 100644
--- a/libfdt/fdt.c
+++ b/libfdt/fdt.c
@@ -348,12 +348,20 @@ int fdt_next_subnode(const void *fdt, int offset)
const char *fdt_find_string_len_(const char *strtab, int tabsize, const char *s,
int slen)
{
- const char *last = strtab + tabsize - (slen + 1);
- const char *p;
+ const char *last;
+ const char *p = strtab;
- for (p = strtab; p <= last; p++)
+ if (tabsize < slen + 1)
+ return NULL;
+ if (slen == 0)
+ return memchr(strtab, '\0', tabsize);
+
+ last = strtab + tabsize - (slen + 1);
+ while (p <= last && (p = memchr(p, s[0], last - p + 1)) != NULL) {
if (memcmp(p, s, slen) == 0 && p[slen] == '\0')
return p;
+ p++;
+ }
return NULL;
}
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (7 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When a base device tree is compiled with `dtc -L` (`--local-fixups`), it
contains a root `/__local_fixups__` node recording the byte offsets of
all phandle references in its properties (for example, so an OS kernel
can identify phandle cells at runtime, so a composite `.dtb` built by
`fdtoverlay` such as Linux's `foo-dtbs := base.dtb bar.dtbo` retains
complete phandle metadata for tools like `fdtfactor`, or so a merged
tree matches what `dtc -L` produces).
`fdt_overlay_apply()` updates the base tree's `/__symbols__` via
`overlay_symbol_update()`, but leaves the base tree's
`/__local_fixups__` untouched. As a result, applying an overlay to a
base tree with `/__local_fixups__` leaves `/__local_fixups__` missing
the phandle references introduced by the overlay (both from the
overlay's `/__local_fixups__` and from its resolved `/__fixups__`), and
leaves stale offsets if the overlay overwrites a base property that
previously had phandle references.
Add `overlay_local_fixups_update()`, controlled by a 3-state
compile-time `FDT_OVERLAY_LOCAL_FIXUPS` setting in
`libfdt/libfdt_internal.h`:
- `0` (`FDT_OVERLAY_LOCAL_FIXUPS_IGNORE`): Leave `/__local_fixups__`
untouched. This is the `#ifndef` fallback in `libfdt_internal.h` so
that size-sensitive embedded environments (such as U-Boot SPL) that
vendor `libfdt` directly get zero code-size increase and no behavior
change unless they opt in.
- `1` (`FDT_OVERLAY_LOCAL_FIXUPS_STRIP`): Delete `/__local_fixups__`
from the base tree if present (~50 bytes of code), avoiding leaving a
stale/incomplete `/__local_fixups__` node in the merged tree.
- `2` (`FDT_OVERLAY_LOCAL_FIXUPS_UPDATE`): Fully update the base tree's
`/__local_fixups__` node when present (clearing entries for
overwritten properties, merging offsets from the overlay's
`/__local_fixups__` and `/__fixups__` in sorted ascending order, and
pruning any subnodes that become empty). This is enabled by default in
`dtc`'s Meson (`-Doverlay-local-fixups=2`) and Make
(`OVERLAY_LOCAL_FIXUPS=2`) builds for desktop tools and tests.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Split out internal helper fdt_add_empty_subnode_() so
overlay_fragment_local_fixups() avoids passing an in-blob pointer to
fdt_add_subnode_namelen() or casting fdt_get_name() (Simon Glass)
- Add test cases in tests/overlay.c where the overlay target has no
existing subnode under /__local_fixups__, with /__local_fixups__ both
before and after the target in the blob (Simon Glass)
- Pass OVERLAY_LOCAL_FIXUPS to tests/run_tests.sh, adjust
fdtoverlay_tests for all three modes, and use 4-space indentation
(Simon Glass, David Gibson)
Changes in v3:
- Move constifying fdto in overlay_apply_node() into a standalone
preparatory patch (David Gibson)
- Gate `/__local_fixups__` handling behind a 3-state compile-time
`FDT_OVERLAY_LOCAL_FIXUPS` flag (`0=ignore`, `1=strip`, `2=update`),
defaulting to `0` (`IGNORE`) in `libfdt_internal.h` for embedded
builds and `2` (`UPDATE`) in Meson/Make desktop builds (Simon Glass,
David Gibson)
- Rewrite commit message in the present tense with motivation up front,
and update `fdt_overlay_apply()` documentation in `libfdt/libfdt.h`
(Simon Glass)
- Require a strict relative path in `overlay_create_path()` and return
`-FDT_ERR_BADOVERLAY` on empty path components or trailing `'/'`
(David Gibson)
- Eliminate the 256-byte `namebuf` limit in
`overlay_fragment_local_fixups()` by re-reading the node name from the
shifted ancestor offset after `fdt_add_subnode_namelen()` (David
Gibson, Simon Glass)
- Return `-FDT_ERR_BADVALUE` instead of `-FDT_ERR_BADSTRUCTURE` in
`overlay_add_local_fixup_offset()` when an existing
`/__local_fixups__` property length is not a multiple of 4 (David
Gibson, Simon Glass)
- Drop redundant `overlay_validate_local_fixups()` pre-pass and inline
single-use helpers (Simon Glass)
Changes in v2:
- New patch updating base /__local_fixups__ in libfdt
fdt_overlay_apply() instead of stripping /__local_fixups__ by default
(David Gibson)
Makefile | 6 +-
libfdt/fdt_overlay.c | 525 ++++++++++++++++++++++++++++++++++++++-
libfdt/fdt_rw.c | 37 ++-
libfdt/libfdt.h | 7 +
libfdt/libfdt_internal.h | 23 ++
meson.build | 5 +
meson_options.txt | 2 +
tests/Makefile.tests | 6 +-
tests/meson.build | 1 +
tests/overlay.c | 154 ++++++++++++
tests/run_tests.sh | 32 +++
11 files changed, 780 insertions(+), 18 deletions(-)
diff --git a/Makefile b/Makefile
index 9110d17796b0..1560a5609fea 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,11 @@ CONFIG_LOCALVERSION =
# See libfdt_internal.h for details
ASSUME_MASK ?= 0
-CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK)
+# Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)
+# See libfdt_internal.h for details
+OVERLAY_LOCAL_FIXUPS ?= 2
+
+CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK) -DFDT_OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS)
WARNINGS = -Wall -Wpointer-arith -Wcast-qual -Wnested-externs -Wsign-compare \
-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow \
-Wwrite-strings
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 4b3e17bd087e..fb76fbafa661 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -1019,9 +1019,8 @@ static int overlay_split_fragment_path(const char *path, int path_len,
* overlay_symbol_update() updates the symbols of the base tree with the
* symbols of the applied overlay
*
- * This is the last step in the device tree overlay application
- * process, allowing the reference of overlay symbols by subsequent
- * overlay operations.
+ * This step in the device tree overlay application process allows
+ * the reference of overlay symbols by subsequent overlay operations.
*
* returns:
* 0 on success
@@ -1135,6 +1134,522 @@ static int overlay_symbol_update(void *fdt, void *fdto)
return 0;
}
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+/**
+ * overlay_create_path - Create a relative path under a parent node
+ * @fdt: Base Device Tree blob
+ * @parent: Node offset of the starting parent node
+ * @path: Relative path to traverse (must not start or end with '/' or contain "//")
+ * @path_len: Length of @path in bytes
+ *
+ * overlay_create_path() walks the slash-separated path components in @path
+ * starting at @parent, creating any missing intermediate subnodes along the
+ * way.
+ *
+ * returns:
+ * Node offset of the resolved or created subnode on success
+ * Negative error code on failure
+ */
+static int overlay_create_path(void *fdt, int parent, const char *path,
+ int path_len)
+{
+ int node = parent;
+
+ while (path_len > 0) {
+ const char *sep = memchr(path, '/', path_len);
+ int comp_len = sep ? (sep - path) : path_len;
+ int next;
+
+ if (!comp_len)
+ return -FDT_ERR_BADOVERLAY;
+
+ next = fdt_subnode_offset_namelen(fdt, node, path, comp_len);
+ if (next == -FDT_ERR_NOTFOUND)
+ next = fdt_add_subnode_namelen(fdt, node, path,
+ comp_len);
+ if (next < 0)
+ return next;
+ node = next;
+
+ if (!sep)
+ break;
+ path += comp_len + 1;
+ path_len -= comp_len + 1;
+ if (!path_len)
+ return -FDT_ERR_BADOVERLAY;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_fragment_local_fixups - Find or create a fragment's target under /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @create: Whether to create missing subnodes under /__local_fixups__
+ *
+ * overlay_fragment_local_fixups() resolves the target of @fragment in @fdt and
+ * locates (or creates, when @create is true) the corresponding node at the
+ * same path under @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * Node offset of the corresponding /__local_fixups__ node on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_local_fixups(void *fdt, const void *fdto,
+ int fragment, bool create)
+{
+ int root_lf, target, cur_fdt, node;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ target = fdt_overlay_target_offset(fdt, fdto, fragment, NULL);
+ if (target < 0)
+ return target;
+
+ cur_fdt = 0;
+ node = root_lf;
+ while (cur_fdt != target) {
+ int subnode, ancestor = -FDT_ERR_INTERNAL;
+ const char *name;
+ int namelen, next_lf;
+
+ fdt_for_each_subnode(subnode, fdt, cur_fdt) {
+ if (subnode <= target)
+ ancestor = subnode;
+ else
+ break;
+ }
+ if (ancestor < 0)
+ return ancestor;
+
+ name = fdt_get_name(fdt, ancestor, &namelen);
+ if (!name)
+ return namelen;
+
+ next_lf = fdt_subnode_offset_namelen(fdt, node, name, namelen);
+ if (next_lf == -FDT_ERR_NOTFOUND && create) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+ char *new_name;
+
+ next_lf = fdt_add_empty_subnode_(fdt, node, namelen,
+ &new_name);
+ if (next_lf < 0)
+ return next_lf;
+ if (next_lf <= ancestor) {
+ int delta = fdt_size_dt_struct(fdt) -
+ old_struct_size;
+
+ target += delta;
+ ancestor += delta;
+ name = fdt_get_name(fdt, ancestor, NULL);
+ }
+ memcpy(new_name, name, namelen);
+ } else if (next_lf < 0) {
+ return next_lf;
+ }
+
+ cur_fdt = ancestor;
+ node = next_lf;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_clear_overwritten_local_fixups - Remove local fixups for overwritten properties
+ * @fdt: Base Device Tree blob
+ * @target_lf: Node offset in @fdt's /__local_fixups__ subtree
+ * @fdto: Device tree overlay blob
+ * @overlay: Node offset in @fdto's __overlay__ subtree
+ *
+ * overlay_clear_overwritten_local_fixups() recursively deletes properties under
+ * @target_lf that correspond to properties overwritten by @overlay in @fdto.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_clear_overwritten_local_fixups(void *fdt, int target_lf,
+ const void *fdto, int overlay)
+{
+ int prop, subnode;
+
+ fdt_for_each_property_offset(prop, fdto, overlay) {
+ const char *name;
+ const void *val;
+ int len, ret;
+
+ val = fdt_getprop_by_offset(fdto, prop, &name, &len);
+ if (!val)
+ return len;
+
+ ret = fdt_delprop(fdt, target_lf, name);
+ if (ret && ret != -FDT_ERR_NOTFOUND)
+ return ret;
+ }
+
+ fdt_for_each_subnode(subnode, fdto, overlay) {
+ const char *name;
+ int namelen, sub_lf, ret;
+
+ name = fdt_get_name(fdto, subnode, &namelen);
+ if (!name)
+ return namelen;
+
+ sub_lf = fdt_subnode_offset_namelen(fdt, target_lf, name,
+ namelen);
+ if (sub_lf == -FDT_ERR_NOTFOUND)
+ continue;
+ if (sub_lf < 0)
+ return sub_lf;
+
+ ret = overlay_clear_overwritten_local_fixups(fdt, sub_lf,
+ fdto, subnode);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_add_local_fixup_offset - Insert a sorted u32 offset into a local fixup property
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ * @name: Property name to update
+ * @name_len: Length of @name in bytes
+ * @poffset: Byte offset of the phandle reference within the target property
+ *
+ * overlay_add_local_fixup_offset() inserts @poffset in ascending sorted order
+ * (ignoring duplicates) into the u32 array property @name on @lf_node.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_add_local_fixup_offset(void *fdt, int lf_node,
+ const char *name, int name_len,
+ uint32_t poffset)
+{
+ const fdt32_t *old_val;
+ fdt32_t *vals;
+ void *prop_data;
+ int old_len, count, i, idx, ret;
+
+ old_val = fdt_getprop_namelen(fdt, lf_node, name, name_len, &old_len);
+ if (!old_val && old_len != -FDT_ERR_NOTFOUND)
+ return old_len;
+
+ if (!old_val) {
+ old_len = 0;
+ count = 0;
+ idx = 0;
+ } else {
+ if (old_len % sizeof(uint32_t))
+ return -FDT_ERR_BADVALUE;
+ count = old_len / sizeof(uint32_t);
+ idx = count;
+ for (i = 0; i < count; i++) {
+ uint32_t cur = fdt32_ld(old_val + i);
+
+ if (cur == poffset)
+ return 0;
+ if (cur > poffset) {
+ idx = i;
+ break;
+ }
+ }
+ }
+
+ ret = fdt_setprop_placeholder_namelen(fdt, lf_node, name, name_len,
+ old_len + sizeof(uint32_t),
+ &prop_data);
+ if (ret)
+ return ret;
+
+ vals = prop_data;
+ if (idx < count)
+ memmove(vals + idx + 1, vals + idx,
+ (count - idx) * sizeof(uint32_t));
+ fdt32_st(vals + idx, poffset);
+
+ return 0;
+}
+
+/**
+ * overlay_merge_fixups_into_local_fixups - Record resolved /__fixups__ for a fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the current fragment in @fdto
+ * @target_frag_name: Name of @fragment
+ * @target_frag_len: Length of @target_frag_name in bytes
+ * @target_lfp: In/out cached node offset of @fragment under @fdt's /__local_fixups__
+ *
+ * overlay_merge_fixups_into_local_fixups() parses external phandle references
+ * belonging to @fragment in @fdto's /__fixups__ node and records their resolved
+ * property byte offsets in @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_merge_fixups_into_local_fixups(void *fdt, const void *fdto,
+ int fragment,
+ const char *target_frag_name,
+ int target_frag_len,
+ int *target_lfp)
+{
+ int fixups_off, prop;
+
+ fixups_off = fdt_subnode_offset(fdto, 0, "__fixups__");
+ if (fixups_off == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (fixups_off < 0)
+ return fixups_off;
+
+ fdt_for_each_property_offset(prop, fdto, fixups_off) {
+ const char *value, *label;
+ int len;
+
+ value = fdt_getprop_by_offset(fdto, prop, &label, &len);
+ if (!value)
+ return len;
+
+ do {
+ const char *path, *name, *frag_name, *rel_path;
+ uint32_t path_len, name_len, poffset;
+ int frag_name_len, rel_path_len, lf_node, ret;
+
+ ret = overlay_parse_fixup_entry(&value, &len, &path,
+ &path_len, &name,
+ &name_len, &poffset);
+ if (ret)
+ return ret;
+
+ ret = overlay_split_fragment_path(path, path_len,
+ &frag_name,
+ &frag_name_len,
+ &rel_path,
+ &rel_path_len);
+ if (ret < 0)
+ return ret;
+ if (!ret)
+ continue;
+
+ if (frag_name_len != target_frag_len ||
+ memcmp(frag_name, target_frag_name,
+ frag_name_len) != 0)
+ continue;
+
+ if (*target_lfp < 0) {
+ *target_lfp = overlay_fragment_local_fixups(
+ fdt, fdto, fragment, true);
+ if (*target_lfp < 0)
+ return *target_lfp;
+ }
+
+ lf_node = overlay_create_path(fdt, *target_lfp,
+ rel_path, rel_path_len);
+ if (lf_node < 0)
+ return lf_node;
+
+ ret = overlay_add_local_fixup_offset(fdt, lf_node,
+ name, name_len,
+ poffset);
+ if (ret)
+ return ret;
+ } while (len > 0);
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_prune_empty_local_fixups - Remove empty subnodes from /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ *
+ * overlay_prune_empty_local_fixups() recursively deletes any child subnodes
+ * under @lf_node that have neither properties nor subnodes remaining.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_prune_empty_local_fixups(void *fdt, int lf_node)
+{
+ int subnode = fdt_first_subnode(fdt, lf_node);
+
+ while (subnode >= 0) {
+ int next, ret;
+
+ ret = overlay_prune_empty_local_fixups(fdt, subnode);
+ if (ret < 0)
+ return ret;
+
+ next = fdt_next_subnode(fdt, subnode);
+ if (fdt_first_property_offset(fdt, subnode) == -FDT_ERR_NOTFOUND &&
+ fdt_first_subnode(fdt, subnode) == -FDT_ERR_NOTFOUND) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+
+ ret = fdt_del_node(fdt, subnode);
+ if (ret < 0)
+ return ret;
+ if (next >= 0)
+ next += fdt_size_dt_struct(fdt) -
+ old_struct_size;
+ }
+ subnode = next;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_fragment_update_local_fixups - Update base /__local_fixups__ for one fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @ov_lf_root: Node offset of /__local_fixups__ in @fdto (or -FDT_ERR_NOTFOUND)
+ *
+ * overlay_fragment_update_local_fixups() clears any base /__local_fixups__
+ * entries overwritten by @fragment, then merges phandle offsets for @fragment
+ * from @fdto's /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_update_local_fixups(void *fdt, const void *fdto,
+ int fragment, int ov_lf_root)
+{
+ const char *frag_name;
+ int frag_name_len, overlay, target_lf, lf_frag, lf_overlay, ret;
+
+ overlay = fdt_subnode_offset(fdto, fragment, "__overlay__");
+ if (overlay == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (overlay < 0)
+ return overlay;
+
+ frag_name = fdt_get_name(fdto, fragment, &frag_name_len);
+ if (!frag_name)
+ return frag_name_len;
+
+ target_lf = overlay_fragment_local_fixups(fdt, fdto, fragment, false);
+ if (target_lf < 0 && target_lf != -FDT_ERR_NOTFOUND)
+ return target_lf;
+
+ if (target_lf >= 0) {
+ ret = overlay_clear_overwritten_local_fixups(fdt, target_lf,
+ fdto, overlay);
+ if (ret)
+ return ret;
+ }
+
+ lf_overlay = -FDT_ERR_NOTFOUND;
+ if (ov_lf_root >= 0) {
+ lf_frag = fdt_subnode_offset_namelen(fdto, ov_lf_root,
+ frag_name, frag_name_len);
+ if (lf_frag >= 0)
+ lf_overlay = fdt_subnode_offset(fdto, lf_frag,
+ "__overlay__");
+ else if (lf_frag != -FDT_ERR_NOTFOUND)
+ return lf_frag;
+ }
+ if (lf_overlay < 0 && lf_overlay != -FDT_ERR_NOTFOUND)
+ return lf_overlay;
+
+ if (lf_overlay >= 0) {
+ if (target_lf < 0)
+ target_lf = overlay_fragment_local_fixups(fdt, fdto,
+ fragment,
+ true);
+ if (target_lf < 0)
+ return target_lf;
+
+ ret = overlay_apply_node(fdt, target_lf, fdto, lf_overlay);
+ if (ret)
+ return ret;
+ }
+
+ return overlay_merge_fixups_into_local_fixups(
+ fdt, fdto, fragment, frag_name, frag_name_len, &target_lf);
+}
+
+/**
+ * overlay_local_fixups_update - Update base /__local_fixups__ with overlay fixups
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ *
+ * overlay_local_fixups_update() updates the base device tree's
+ * /__local_fixups__ subtree (when present) by walking each overlay fragment in
+ * application order, clearing entries for properties overwritten by that
+ * fragment, and merging phandle reference offsets from @fdto's
+ * /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * This is part of the device tree overlay application process, when you want
+ * the merged device tree to retain complete local phandle fixup metadata.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf, ov_lf_root, fragment, ret;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ ov_lf_root = fdt_subnode_offset(fdto, 0, "__local_fixups__");
+ if (ov_lf_root < 0 && ov_lf_root != -FDT_ERR_NOTFOUND)
+ return ov_lf_root;
+
+ /*
+ * Process fragments in application order so that if a later fragment in
+ * fdto overwrites a property modified by an earlier fragment, the
+ * earlier fragment's local fixups for that property are cleared first.
+ */
+ fdt_for_each_subnode(fragment, fdto, 0) {
+ ret = overlay_fragment_update_local_fixups(fdt, fdto, fragment,
+ ov_lf_root);
+ if (ret)
+ return ret;
+ }
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ return overlay_prune_empty_local_fixups(fdt, root_lf);
+}
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ return fdt_del_node(fdt, root_lf);
+}
+#else
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ return 0;
+}
+#endif
+
int fdt_overlay_apply(void *fdt, void *fdto)
{
uint32_t delta;
@@ -1175,6 +1690,10 @@ int fdt_overlay_apply(void *fdt, void *fdto)
if (ret)
goto err;
+ ret = overlay_local_fixups_update(fdt, fdto);
+ if (ret)
+ goto err;
+
/*
* The overlay has been damaged, erase its magic.
*/
diff --git a/libfdt/fdt_rw.c b/libfdt/fdt_rw.c
index 850aafe44d27..99048f135cb5 100644
--- a/libfdt/fdt_rw.c
+++ b/libfdt/fdt_rw.c
@@ -365,8 +365,8 @@ int fdt_delprop(void *fdt, int nodeoffset, const char *name)
return fdt_splice_struct_(fdt, prop, proplen, 0);
}
-int fdt_add_subnode_namelen(void *fdt, int parentoffset,
- const char *name, int namelen)
+int fdt_add_empty_subnode_(void *fdt, int parentoffset, int namelen,
+ char **namep)
{
struct fdt_node_header *nh;
int offset, nextoffset;
@@ -375,14 +375,6 @@ int fdt_add_subnode_namelen(void *fdt, int parentoffset,
uint32_t tag;
fdt32_t *endtag;
- FDT_RW_PROBE(fdt);
-
- offset = fdt_subnode_offset_namelen(fdt, parentoffset, name, namelen);
- if (offset >= 0)
- return -FDT_ERR_EXISTS;
- else if (offset != -FDT_ERR_NOTFOUND)
- return offset;
-
/* Try to place the new node after the parent's properties */
tag = fdt_next_tag(fdt, parentoffset, &nextoffset);
/* the fdt_subnode_offset_namelen() should ensure this never hits */
@@ -402,9 +394,32 @@ int fdt_add_subnode_namelen(void *fdt, int parentoffset,
nh->tag = cpu_to_fdt32(FDT_BEGIN_NODE);
memset(nh->name, 0, FDT_TAGALIGN(namelen+1));
- memcpy(nh->name, name, namelen);
endtag = (fdt32_t *)((char *)nh + nodelen - FDT_TAGSIZE);
*endtag = cpu_to_fdt32(FDT_END_NODE);
+ *namep = nh->name;
+
+ return offset;
+}
+
+int fdt_add_subnode_namelen(void *fdt, int parentoffset,
+ const char *name, int namelen)
+{
+ char *namep;
+ int offset;
+
+ FDT_RW_PROBE(fdt);
+
+ offset = fdt_subnode_offset_namelen(fdt, parentoffset, name, namelen);
+ if (offset >= 0)
+ return -FDT_ERR_EXISTS;
+ else if (offset != -FDT_ERR_NOTFOUND)
+ return offset;
+
+ offset = fdt_add_empty_subnode_(fdt, parentoffset, namelen, &namep);
+ if (offset < 0)
+ return offset;
+
+ memcpy(namep, name, namelen);
return offset;
}
diff --git a/libfdt/libfdt.h b/libfdt/libfdt.h
index c69a18edf09f..c2f47a7d3e87 100644
--- a/libfdt/libfdt.h
+++ b/libfdt/libfdt.h
@@ -2472,6 +2472,13 @@ int fdt_del_node(void *fdt, int nodeoffset);
* fdt_overlay_apply() will apply the given device tree overlay on the
* given base device tree.
*
+ * If the base device tree contains a /__symbols__ node, it is updated with
+ * any symbols from the overlay. If the base device tree contains a
+ * /__local_fixups__ node, how it is handled depends on the build-time
+ * FDT_OVERLAY_LOCAL_FIXUPS setting (see libfdt_internal.h): left untouched
+ * (IGNORE), deleted (STRIP), or updated to reflect phandle references in the
+ * merged tree (UPDATE, the default when built via dtc's build system).
+ *
* Expect the base device tree to be modified, even if the function
* returns an error.
*
diff --git a/libfdt/libfdt_internal.h b/libfdt/libfdt_internal.h
index eefea2a3ccf3..315a7022fc32 100644
--- a/libfdt/libfdt_internal.h
+++ b/libfdt/libfdt_internal.h
@@ -32,6 +32,8 @@ static inline const char *fdt_find_string_(const char *strtab, int tabsize,
}
int fdt_node_end_offset_(void *fdt, int nodeoffset);
+int fdt_add_empty_subnode_(void *fdt, int parentoffset, int namelen,
+ char **namep);
static inline const void *fdt_offset_ptr_(const void *fdt, int offset)
{
@@ -99,6 +101,27 @@ static inline uint64_t fdt64_ld_(const fdt64_t *p)
#define FDT_PARENT_MAX_DEPTH 32
#endif
+/*
+ * Controls how fdt_overlay_apply() handles a /__local_fixups__ node in the
+ * base device tree:
+ * 0 (FDT_OVERLAY_LOCAL_FIXUPS_IGNORE): Leave /__local_fixups__ untouched
+ * (zero extra code size and preserves historical behavior; default when
+ * building libfdt standalone without dtc's build system).
+ * 1 (FDT_OVERLAY_LOCAL_FIXUPS_STRIP): Delete /__local_fixups__ from the
+ * base device tree so a stale/incomplete /__local_fixups__ node is not
+ * left behind (~50 bytes of code).
+ * 2 (FDT_OVERLAY_LOCAL_FIXUPS_UPDATE): Fully update /__local_fixups__ in
+ * the base device tree with the overlay's local and resolved external
+ * phandle fixups (default when building via dtc's Meson/Make build).
+ */
+#define FDT_OVERLAY_LOCAL_FIXUPS_IGNORE 0
+#define FDT_OVERLAY_LOCAL_FIXUPS_STRIP 1
+#define FDT_OVERLAY_LOCAL_FIXUPS_UPDATE 2
+
+#ifndef FDT_OVERLAY_LOCAL_FIXUPS
+#define FDT_OVERLAY_LOCAL_FIXUPS FDT_OVERLAY_LOCAL_FIXUPS_IGNORE
+#endif
+
/*
* Defines assumptions which can be enabled. Each of these can be enabled
* individually. For maximum safety, don't enable any assumptions!
diff --git a/meson.build b/meson.build
index 520cd35f9075..348d9b555e34 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,11 @@ add_project_arguments(
language: 'c'
)
+add_project_arguments(
+ '-DFDT_OVERLAY_LOCAL_FIXUPS=' + get_option('overlay-local-fixups').to_string(),
+ language: 'c'
+)
+
yamltree = 'yamltree.c'
yaml = dependency('yaml-0.1', version: '>=0.2.3', required: get_option('yaml'))
if not yaml.found()
diff --git a/meson_options.txt b/meson_options.txt
index a866b172e802..ac8ea4aa2a9f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -2,6 +2,8 @@ option('tools', type: 'boolean', value: true,
description: 'Build tools')
option('assume-mask', type: 'integer', value: 0,
description: 'Control the assumptions made (e.g. risking security issues) in the code.')
+option('overlay-local-fixups', type: 'integer', value: 2,
+ description: 'Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)')
option('yaml', type: 'feature', value: 'auto',
description: 'YAML support')
option('valgrind', type: 'feature', value: 'auto',
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index af5d309cf8a9..623d71786701 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -81,18 +81,18 @@ tests_clean:
rm -rf $(TESTS_CLEANDIRS)
check: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh
ifeq ($(NO_VALGRIND),1)
checkm:
@echo "make checkm requires valgrind, but NO_VALGRIND=1"
else
checkm: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh -m
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh -m
endif
checkv: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh -v
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh -v
ifneq ($(DEPTARGETS),)
-include $(TESTS_DEPFILES)
diff --git a/tests/meson.build b/tests/meson.build
index e37567864e04..9f0f51013530 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -128,6 +128,7 @@ if not yaml.found()
else
env += 'NO_YAML=0'
endif
+env += 'OVERLAY_LOCAL_FIXUPS=' + get_option('overlay-local-fixups').to_string()
run_test_types = [
'libfdt',
diff --git a/tests/overlay.c b/tests/overlay.c
index 2d27918a3369..d2cc0af0178c 100644
--- a/tests/overlay.c
+++ b/tests/overlay.c
@@ -9,6 +9,7 @@
#include <stdio.h>
#include <libfdt.h>
+#include "libfdt_internal.h"
#include "tests.h"
@@ -191,6 +192,156 @@ static void *open_dt(char *path)
return copy;
}
+static void fdt_overlay_local_fixups(char *base_path, char *overlay_path)
+{
+ void *fdt_base = open_dt(base_path);
+ void *fdt_overlay = open_dt(overlay_path);
+ int lf, tn, stn;
+
+ /*
+ * Add a /__local_fixups__ subtree to the base tree before applying
+ * the overlay:
+ * - /__local_fixups__/test-node: test-int-property = <0> (overwritten
+ * by the overlay with a non-phandle value)
+ * - /__local_fixups__/test-node/sub-test-node:
+ * new-sub-test-property = <0> (overwritten by the overlay, leaving
+ * sub-test-node empty so it should be pruned in UPDATE mode)
+ */
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_add_subnode(fdt_base, lf, "test-node");
+ CHECK(tn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, tn, "test-int-property", 0));
+ stn = fdt_add_subnode(fdt_base, tn, "sub-test-node");
+ CHECK(stn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, stn, "new-sub-test-property", 0));
+
+ CHECK(fdt_overlay_apply(fdt_base, fdt_overlay));
+
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val) !=
+ -FDT_ERR_NOTFOUND);
+ CHECK(fdt_path_offset(
+ fdt_base,
+ "/__local_fixups__/test-node/sub-test-node") !=
+ -FDT_ERR_NOTFOUND);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 1, &val));
+ CHECK(val != 4);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 1, &val));
+ CHECK(val != 4);
+ }
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") !=
+ -FDT_ERR_NOTFOUND);
+#else
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val) != -FDT_ERR_NOTFOUND);
+ }
+#endif
+
+ free(fdt_base);
+ free(fdt_overlay);
+}
+
+static void fdt_overlay_local_fixups_create(char *base_path, char *overlay_path,
+ bool lf_after_target)
+{
+ void *fdt_base = open_dt(base_path);
+ void *fdt_overlay = open_dt(overlay_path);
+ int lf, tn;
+
+ /*
+ * Create an empty /__local_fixups__ node (with no test-node child)
+ * either before or after /test-node in the blob, so that
+ * overlay_fragment_local_fixups() must create /__local_fixups__/test-node
+ * during fdt_overlay_apply().
+ */
+ if (lf_after_target) {
+ uint32_t ph;
+
+ tn = fdt_path_offset(fdt_base, "/test-node");
+ CHECK(tn < 0);
+ ph = fdt_get_phandle(fdt_base, tn);
+ CHECK(!ph);
+ CHECK(fdt_del_node(fdt_base, tn));
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_add_subnode(fdt_base, 0, "test-node");
+ CHECK(tn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, tn, "phandle", ph));
+ lf = fdt_path_offset(fdt_base, "/__local_fixups__");
+ CHECK(lf <= tn);
+ } else {
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_path_offset(fdt_base, "/test-node");
+ CHECK(tn < 0);
+ CHECK(lf >= tn);
+ }
+
+ CHECK(fdt_overlay_apply(fdt_base, fdt_overlay));
+
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 1, &val));
+ CHECK(val != 4);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 1, &val));
+ CHECK(val != 4);
+ }
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") !=
+ -FDT_ERR_NOTFOUND);
+#else
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") < 0);
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__/test-node") !=
+ -FDT_ERR_NOTFOUND);
+#endif
+
+ free(fdt_base);
+ free(fdt_overlay);
+}
+
int main(int argc, char *argv[])
{
void *fdt_base, *fdt_overlay;
@@ -219,6 +370,9 @@ int main(int argc, char *argv[])
if (fdt_path_offset(fdt_base, "/__symbols__") >= 0) {
fdt_overlay_local_phandle(fdt_base);
fdt_overlay_local_phandles(fdt_base);
+ fdt_overlay_local_fixups(argv[1], argv[2]);
+ fdt_overlay_local_fixups_create(argv[1], argv[2], false);
+ fdt_overlay_local_fixups_create(argv[1], argv[2], true);
}
PASS();
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 3cacb68438df..e16767d3f3a2 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -11,6 +11,10 @@ if [ -z "$PYTHON" ]; then
PYTHON=python3
fi
+if [ -z "$OVERLAY_LOCAL_FIXUPS" ]; then
+ OVERLAY_LOCAL_FIXUPS=2
+fi
+
if [ -n "$NO_PYTHON" ]; then
if [ "$NO_PYTHON" != "0" ]; then
no_python=true
@@ -1147,6 +1151,34 @@ fdtoverlay_tests() {
shorten_echo "check phandle to noda d wasn't overwritten: "
run_wrap_test test "$bd" = "$pd"
+
+ # verify how /__local_fixups__ in a -L base blob is handled when applying an overlay
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_phandle_lf.test.dtb "$SRCDIR/overlay_base_phandle.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_phandle_lf.test.dtb -o overlay_base_phandle_lfO.test.dtb overlay_overlay_phandle.test.dtb
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_lf.test.dtb "$SRCDIR/overlay_base.dts"
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb -cp /__local_fixups__/test-node/sub-test-node
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb /__local_fixups__/test-node/sub-test-node new-sub-test-property -tu 0
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay.fdtoverlay.test.dtb "$SRCDIR/overlay_overlay.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_lf.test.dtb -o overlay_base_lfO.test.dtb overlay_overlay.fdtoverlay.test.dtb
+ if [ "$OVERLAY_LOCAL_FIXUPS" -eq 2 ]; then
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b d
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/c b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d c
+
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-phandle
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-several-phandle
+ run_test check_path overlay_base_lfO.test.dtb not-exists "/__local_fixups__/test-node/sub-test-node"
+ elif [ "$OVERLAY_LOCAL_FIXUPS" -eq 1 ]; then
+ run_test check_path overlay_base_phandle_lfO.test.dtb not-exists "/__local_fixups__"
+ run_test check_path overlay_base_lfO.test.dtb not-exists "/__local_fixups__"
+ else
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b a
+ run_wrap_error_test $DTGET overlay_base_phandle_lfO.test.dtb /__local_fixups__/b d
+ run_fdtget_test 0 overlay_base_lfO.test.dtb /__local_fixups__/test-node/sub-test-node new-sub-test-property
+ fi
}
run_python_tests () {
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (8 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When shipping a single firmware or kernel image that supports many
closely related board variants or SKUs, bundling a full compiled .dtb
for every variant results in substantial duplication.
Add `fdtfactor`, a Python tool built on `pylibfdt` and `libpydt` that
takes a set of compiled device tree blobs (.dtb) compiled with `-@ -L`
(`/__symbols__` and `/__local_fixups__`) and splits them into a single
common `base.dtb` and per-board `.dtbo` overlays:
- Common nodes present across all input DTBs are placed in `base.dtb`,
with each common property assigned the value shared by at least two
boards that maximizes byte savings across overlays.
- Using `/__local_fixups__`, phandle references inside property values
are normalized to target node paths so common properties referencing
common nodes match even when numeric phandles differ across input
DTBs.
- Any common node that carries a phandle in any input DTB receives its
phandle in `base.dtb` so overlays only emit `phandle` properties on
newly added subnodes, matching the requirements of `libfdt`, Linux's
`drivers/of/overlay.c`, and coreboot's `dt_apply_overlay()`.
Add a `fdtfactor` test suite in `tests/run_tests.sh` verifying that
applying each generated `.dtbo` to `base.dtb` with `fdtoverlay`
reconstructs the original DTB (verified via `dtbs_equal_unordered`,
including across input DTBs with shifted phandle numbers).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Rename -o/--outdir to -o/--output so standalone DTB mode and FIT mode
share the same output option, and use 4-space indentation in
tests/run_tests.sh (David Gibson)
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Replace ParsedDtb.raw_bytes with uncompressed_size, use early continue
to reduce loop nesting, drop redundant set() before sorted() on local
fixup offsets, and clarify pre-order walk and phandle comments (David
Gibson)
- Allow dtc -@ -L inputs with no phandles (where dtc omits /__symbols__
and /__local_fixups__)
- Check _is_dtbo_overlay() on the unflattened DeviceTree, index
ParsedDtb.nodes and base_nodes by path, and skip unchanged base nodes
in _build_board_dtbo() to avoid repeated O(depth) traversals and
per-property comparisons
- Handle OVERLAY_LOCAL_FIXUPS != 2 in fdtfactor_tests (Simon Glass)
Changes in v3:
- Update fdtfactor to use devicetree.DeviceTree and a FactoredTree
subclass initializing self.phandle and self.fixups in __init__()
(David Gibson, Simon Glass)
- Rename -b/--base-dtb to --output-base-dtb to avoid confusion with an
input base DTB (David Gibson)
- Move the test rejecting input DTBs lacking /__local_fixups__ from the
-S patch into this base patch (Simon Glass)
Changes in v2:
- Preserve /__local_fixups__ on base.dtb by default (David Gibson)
- Store parsed trees directly in NodeTree with node.phandle and
node.fixups attributes instead of parallel dicts (David Gibson)
- Extract phandles in a separate pass after unflattening and reject
legacy linux,phandle properties (David Gibson)
- Exclude non-root common phandle nodes lacking a common symbol from
base.dtb instead of synthesizing __fdtfactor_ph_* symbols (David
Gibson)
- Drop -q/--quiet, rename -v/--verbose to -s/--stats, and move tests to
a dedicated fdtfactor test suite (David Gibson)
Makefile | 2 +-
README.md | 43 ++
fdtfactor | 881 +++++++++++++++++++++++++++++
meson.build | 1 +
tests/Makefile.tests | 4 +-
tests/fdtfactor_board1.dts | 72 +++
tests/fdtfactor_board2.dts | 70 +++
tests/fdtfactor_board3.dts | 68 +++
tests/fdtfactor_board3_shifted.dts | 68 +++
tests/meson.build | 2 +-
tests/run_tests.sh | 110 +++-
tests/testutils.sh | 1 +
12 files changed, 1315 insertions(+), 7 deletions(-)
create mode 100755 fdtfactor
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
create mode 100644 tests/fdtfactor_board3_shifted.dts
diff --git a/Makefile b/Makefile
index 1560a5609fea..fc06b0ae16e2 100644
--- a/Makefile
+++ b/Makefile
@@ -169,7 +169,7 @@ BIN += fdtget
BIN += fdtput
BIN += fdtoverlay
-SCRIPTS = dtdiff
+SCRIPTS = dtdiff fdtfactor
all: $(BIN) libfdt
diff --git a/README.md b/README.md
index 9d38c7e79fc6..9a8be74d47fe 100644
--- a/README.md
+++ b/README.md
@@ -201,6 +201,49 @@ merged blob to `-o`, `--output`:
$ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
```
+## fdtfactor
+
+`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
+compiled device tree blobs (`.dtb`) and splits them into a single common
+`base.dtb` and per-board `.dtbo` overlays so that applying a board's
+`.dtbo` to `base.dtb` recreates the original board device tree while
+reducing total storage size.
+
+Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
+and `/__local_fixups__` (used to identify phandle references inside
+property values) are present.
+
+Basic usage:
+
+```
+$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+```
+
+Options:
+
+* `--output-base-dtb`: Output filename for the common base `.dtb`
+ (default: `base.dtb`; placed in `--output` if specified, or the current
+ directory otherwise).
+* `-o`, `--output`: Output directory for generated `.dtbo` files and
+ `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
+ current directory for `--output-base-dtb`).
+* `-s`, `--stats`: Print size savings summary.
+
+### TODO
+
+* **Targeted overlay fragments**: Add an option to emit fine-grained
+ `/fragment@N` nodes targeting individual parent subnodes rather than
+ placing all overlay changes under a single `/fragment@0` targeting `/`.
+* **Multi-level overlays**: Support hierarchical overlay layers (for
+ example, SoC base -> board family intermediate overlay -> SKU leaf
+ overlay) when multiple boards share large sub-trees not present across
+ all boards. As an interim step, `fdtfactor` could accept an
+ already-generated `base.dtb` plus a subset of `.dtbo` overlays emitted
+ by an initial pass, leaving `base.dtb` untouched while factoring out
+ an intermediate `.dtbo` for the commonalities shared within that
+ subset.
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
diff --git a/fdtfactor b/fdtfactor
new file mode 100755
index 000000000000..dfbee0aea2d6
--- /dev/null
+++ b/fdtfactor
@@ -0,0 +1,881 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""fdtfactor - Split multiple compiled .dtb files into a base .dtb and .dtbo overlays.
+
+Given two or more input .dtb files compiled with `dtc -@ -L` (so they include
+`/__symbols__` and `/__local_fixups__`), `fdtfactor` extracts all nodes and
+properties shared across the inputs into a single common `base.dtb` and emits a
+small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
+`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
+original board DTB.
+"""
+
+import argparse
+from collections import Counter
+import os
+import struct
+import sys
+
+import libfdt
+import libpydt
+
+
+def _is_dtbo_overlay(root):
+ """Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
+
+ Args:
+ root: Root `libpydt.DeviceTree` instance to inspect.
+
+ Returns:
+ True if `root` contains `/__fixups__` or a top-level subnode with an
+ `__overlay__` child, False otherwise.
+ """
+ return (
+ '__fixups__' in root.children or
+ any('__overlay__' in child.children
+ for child in root.children.values())
+ )
+
+
+class FactoredTree(libpydt.DeviceTree):
+ """DeviceTree subclass with normalized phandle and fixup attributes.
+
+ Attributes:
+ phandle: Optional integer phandle popped from `props['phandle']`.
+ fixups: Dict mapping property name (str) to a tuple of
+ `(byte_offset, target_node_path)` phandle fixup pairs.
+ """
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+ self.phandle = None
+ self.fixups = {}
+
+
+class ParsedDtb:
+ """Complete input .dtb blob with phandle-normalized tree and FDT metadata.
+
+ While `libpydt.DeviceTree` is a generic live-tree representation of a
+ node hierarchy (with `libfdt.Property` instances and unprocessed
+ `/__symbols__`, `/__local_fixups__`, and `phandle` properties), `ParsedDtb`
+ wraps an entire input `.dtb` in a `FactoredTree` (`self.root`) and
+ normalizes it for cross-board comparison:
+
+ - Stores FDT header, memory reservation, and file-level metadata that live
+ outside the root node tree (`boot_cpuid_phys`, `mem_rsv`, `filename`,
+ `uncompressed_size`).
+ - Pops `/__symbols__` (parsed into `self.symbols`) and `/__local_fixups__`
+ out of `self.root`, and indexes the remaining tree in pre-order in
+ `self.nodes` (`path -> FactoredTree`).
+ - Moves each node's `phandle` property out of `node.props` into
+ `node.phandle`, and uses `/__local_fixups__` to zero out numeric phandle
+ cells in `node.props` while recording canonical `(offset, target_path)`
+ tuples in `node.fixups` so properties can be compared across DTBs
+ independent of numeric phandle numbering.
+ """
+
+ def __init__(self, filename):
+ """Parse an input FDT file into a normalized FactoredTree.
+
+ Args:
+ filename: File path of the `.dtb` file to read.
+ """
+ self.filename = filename
+ with open(filename, 'rb') as f:
+ raw_bytes = f.read()
+ self.uncompressed_size = len(raw_bytes)
+
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+
+ self.boot_cpuid_phys = fdt.boot_cpuid_phys()
+ self.mem_rsv = tuple(fdt.get_mem_rsv_list())
+ self.root = FactoredTree.unflatten(fdt)
+
+ if _is_dtbo_overlay(self.root):
+ raise ValueError(
+ f'{filename}: Input is already a DTB overlay (.dtbo)'
+ )
+
+ sym_node = self.root.children.pop('__symbols__', None)
+ lf_root = self.root.children.pop('__local_fixups__', None)
+ self.has_local_fixups = lf_root is not None
+
+ self.nodes = dict(self.root.walk())
+ phandle_to_path = self._extract_phandles()
+ if phandle_to_path and (sym_node is None or lf_root is None):
+ raise ValueError(
+ f'input DTB(s) lack /__symbols__ or /__local_fixups__: '
+ f'{filename} (compile input DTBs with dtc -@ -L)'
+ )
+ self.symbols = (
+ self._parse_symbols(sym_node) if sym_node is not None else {}
+ )
+ if lf_root is not None:
+ self._apply_local_fixups(lf_root, phandle_to_path)
+
+ def _extract_phandles(self):
+ """Move `phandle` properties out of `node.props`.
+
+ Iterates `self.nodes` in pre-order and, for each node:
+ - Pops `phandle` from `node.props` (so phandle properties are not
+ compared as ordinary byte properties across DTBs, since `dtc` may
+ assign different integer phandles to the same node in different
+ boards) and stores the decoded integer in `node.phandle`.
+ - Validates that `node.phandle` is not `0` or `0xffffffff`, and that no
+ two nodes in the DTB share the same phandle.
+
+ Returns:
+ Dict mapping each integer phandle in this DTB to its node path
+ string (`""` for root, `"/soc/..."` for subnodes), used by
+ `_apply_local_fixups()` to resolve numeric phandle cells to
+ canonical target node paths.
+ """
+ phandle_to_path = {}
+ for path, node in self.nodes.items():
+ if 'linux,phandle' in node.props:
+ raise ValueError(
+ f'{self.filename}: {path or "/"}: legacy linux,phandle '
+ f'property is not supported (compile with dtc -H epapr)'
+ )
+ ph_prop = node.props.pop('phandle', None)
+ if ph_prop is not None:
+ node.phandle = ph_prop.as_uint32()
+
+ if node.phandle is None:
+ continue
+ if node.phandle in (0, 0xffffffff):
+ raise ValueError(
+ f'{self.filename}: invalid phandle {node.phandle:#x} '
+ f'on {path or "/"}'
+ )
+ if node.phandle in phandle_to_path:
+ raise ValueError(
+ f'{self.filename}: duplicate phandle {node.phandle:#x} '
+ f'on {phandle_to_path[node.phandle] or "/"} and '
+ f'{path or "/"}'
+ )
+ phandle_to_path[node.phandle] = path
+ return phandle_to_path
+
+ def _parse_symbols(self, sym_node):
+ """Parse the popped `/__symbols__` node into a `label -> path` dict.
+
+ Each property on `/__symbols__` (ignoring any `phandle` property on the
+ `/__symbols__` node itself) maps a Devicetree label name to the full
+ path of the target node in the DTB. Root paths (`"/"`) are normalized to
+ `""` to match `DeviceTree` path conventions, and each target path is
+ verified to exist in `self.nodes`.
+
+ Args:
+ sym_node: The `FactoredTree` for `/__symbols__` popped from
+ `self.root`.
+
+ Returns:
+ Dict mapping symbol label name (str) to normalized target node path
+ (str, `""` for root or `"/..."` for subnodes).
+ """
+ symbols = {}
+ for label, prop in sym_node.props.items():
+ if label == 'phandle':
+ continue
+ raw_path = prop.as_str()
+ if not raw_path.startswith('/') or '//' in raw_path:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} has malformed '
+ f'path {raw_path!r}'
+ )
+ target_path = raw_path.rstrip('/')
+ if target_path not in self.nodes:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} references '
+ f'missing node {target_path or "/"!r}'
+ )
+ symbols[label] = target_path
+ return symbols
+
+ def _apply_local_fixups(self, lf_root, phandle_to_path):
+ """Normalize phandle references in `self.root` using `/__local_fixups__`.
+
+ The `/__local_fixups__` tree mirrors the structure of `self.root`: each
+ property in `/__local_fixups__` contains a list of 32-bit byte offsets
+ where the corresponding property in `self.root` holds a phandle cell.
+ For every such offset, this method:
+ - Reads the 32-bit integer phandle from `target_node.props[prop_name]`
+ and resolves it to a canonical target node path via `phandle_to_path`.
+ - Zeros out the 4-byte phandle cell in `target_node.props[prop_name]` so
+ property values can be compared across DTBs independent of numeric
+ phandle numbering.
+ - Records `((offset, target_path), ...)` in
+ `target_node.fixups[prop_name]`.
+
+ Args:
+ lf_root: The `FactoredTree` for `/__local_fixups__` popped from
+ `self.root`.
+ phandle_to_path: Dict mapping integer phandle values to normalized
+ node paths in `self.root`, as returned by `_extract_phandles()`.
+ """
+ for path, lf_node in lf_root.walk():
+ if not lf_node.props:
+ continue
+ target_node = self.nodes.get(path)
+ if target_node is None:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing node '
+ f'{path or "/"}'
+ )
+ for prop_name, lf_prop in lf_node.props.items():
+ raw_prop = target_node.props.get(prop_name)
+ if raw_prop is None:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing '
+ f'property {path or "/"}:{prop_name}'
+ )
+ offsets = sorted(lf_prop.as_uint32_list())
+ masked = bytearray(raw_prop)
+ targets = []
+ for off in offsets:
+ if off + 4 > len(raw_prop):
+ raise ValueError(
+ f'{self.filename}: fixup offset {off} out of range '
+ f'on {path or "/"}:{prop_name} (len {len(raw_prop)})'
+ )
+ if targets and off < targets[-1][0] + 4:
+ raise ValueError(
+ f'{self.filename}: overlapping fixup offset {off} '
+ f'on {path or "/"}:{prop_name}'
+ )
+ ph = struct.unpack_from('>I', raw_prop, off)[0]
+ target_path = phandle_to_path.get(ph)
+ if target_path is None:
+ raise ValueError(
+ f'{self.filename}: unknown phandle {ph:#x} at '
+ f'{path or "/"}:{prop_name}+{off}'
+ )
+ masked[off:off + 4] = b'\x00\x00\x00\x00'
+ targets.append((off, target_path))
+ target_node.set_prop(
+ libfdt.Property(prop_name, bytes(masked))
+ )
+ target_node.fixups[prop_name] = tuple(targets)
+
+
+def select_best_candidate(candidates, path, prop_name):
+ """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
+
+ Only candidates shared by at least two input DTBs (`count > 1`) are
+ eligible for the base DTB; a value that appears in only one DTB would add
+ one copy to `base.dtb` to save one overlay copy (zero net savings).
+
+ Byte cost model per overlay occurrence:
+ - Each FDT property carries a 12-byte structure header (`FDT_PROP` 4-byte
+ tag + 4-byte length + 4-byte `nameoff`) plus `len(masked_bytes)` value
+ bytes.
+ - Each phandle fixup in an overlay adds a string entry of the form
+ `"/fragment@0/__overlay__<path>:<prop_name>:<off>\\0"` to `/__fixups__`:
+ 22 bytes (`"/fragment@0/__overlay__"`) + `len(path)` + 1 (`":"`) +
+ `len(prop_name)` + 1 (`":"`) + ~1-2 digits for `<off>` + 1 (`"\\0"`) +
+ ~3 bytes amortized `/__fixups__` property overhead =
+ `len(path) + len(prop_name) + 28` bytes.
+
+ Args:
+ candidates: Sequence of `(masked_bytes, fixups)` tuples observed across
+ input DTBs for `prop_name` on `path`.
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+
+ Returns:
+ The `(masked_bytes, fixups)` tuple that yields the largest estimated
+ byte savings when placed in `base.dtb`, or `None` if no candidate
+ appears in at least two input DTBs.
+ """
+ fixup_entry_cost = len(path) + len(prop_name) + 28
+ counts = Counter(candidates)
+ best_val = None
+ best_key = None
+ for val, count in counts.items():
+ if count <= 1:
+ continue
+ masked_bytes, fixups = val
+ length = len(masked_bytes) + len(fixups) * fixup_entry_cost
+ # Placing `val` in base.dtb adds 1 copy and removes `count` overlay
+ # copies, saving `(count - 1) * (length + 12)` bytes across the set.
+ key = ((count - 1) * (length + 12), count, length)
+ if best_key is None or key > best_key:
+ best_key = key
+ best_val = val
+ return best_val
+
+
+def _validate_dtb_headers(dtbs):
+ """Verify that FDT header fields and `/memreserve/` entries match across DTBs.
+
+ `fdt_overlay_apply()` only merges the `.dtbo` structure block into the base
+ DTB and ignores the `.dtbo` header and `/memreserve/` block, so all DTBs
+ sharing a base DTB must have identical `boot_cpuid_phys` and `mem_rsv`.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances to validate.
+ """
+ for d in dtbs[1:]:
+ if d.boot_cpuid_phys != dtbs[0].boot_cpuid_phys:
+ raise ValueError(
+ f'boot_cpuid_phys mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].boot_cpuid_phys}) and {d.filename} '
+ f'({d.boot_cpuid_phys})'
+ )
+ if d.mem_rsv != dtbs[0].mem_rsv:
+ raise ValueError(
+ f'/memreserve/ mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].mem_rsv}) and {d.filename} ({d.mem_rsv})'
+ )
+
+
+def _find_common_nodes_and_phandles(dtbs):
+ """Build the common `FactoredTree` skeleton and assign base DTB phandles.
+
+ Includes nodes whose path is present in every input DTB. If a node has a
+ `phandle` in at least one input DTB:
+ - When it has a shared label in `/__symbols__` across all input DTBs (or is
+ the root node), it is kept in `base.dtb` and assigned a sequential
+ phandle. That satisfies references from common properties, `/__symbols__`,
+ and overlay `/__fixups__`.
+ - Otherwise (e.g. only some boards label the node, or boards use different
+ labels), the node and its subtree are excluded from `base.dtb` and emitted
+ per board in `.dtbo` overlays. That avoids adding a spurious `phandle` to
+ boards that lacked one and avoids overlays needing a `/__fixups__` target
+ missing from `base.dtb`'s `/__symbols__`.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ Dict mapping each common node path in pre-order (`""` for root,
+ `"/..."` for subnodes) to its `FactoredTree` node, with `node.phandle`
+ set to its assigned sequential integer phandle (or `None` if the node
+ has no phandle in any input DTB) and `node.fixups` initialized to `{}`.
+ """
+ common_paths = set(dtbs[0].nodes).intersection(
+ *(d.nodes for d in dtbs[1:])
+ )
+
+ common_sym_targets = {
+ path for label, path in dtbs[0].symbols.items()
+ if all(d.symbols.get(label) == path for d in dtbs[1:])
+ }
+
+ base_root = FactoredTree()
+ base_nodes = {}
+ next_ph = 1
+ # Iterate dtbs[0].nodes in pre-order (rather than iterating the unordered
+ # `common_paths` set) so parents are always visited before their children,
+ # subnode order is preserved, and `next_ph` assignment is deterministic.
+ for path in dtbs[0].nodes:
+ if path not in common_paths:
+ continue
+ if path and path.rpartition('/')[0] not in base_nodes:
+ continue
+ has_phandle = any(d.nodes[path].phandle is not None for d in dtbs)
+ if has_phandle and path and path not in common_sym_targets:
+ continue
+ base_node = base_root.create_path(path)
+ base_nodes[path] = base_node
+ if has_phandle:
+ base_node.phandle = next_ph
+ next_ph += 1
+ return base_nodes
+
+
+def _find_common_symbols(dtbs, base_nodes):
+ """Find symbols common to all input DTBs that point to the same base node.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+
+ Returns:
+ Tuple of `(base_dtb_symbols, path_to_base_dtb_symbol)` where:
+ * `base_dtb_symbols`: Dict mapping symbol label to target node path.
+ * `path_to_base_dtb_symbol`: Dict mapping target node path to its
+ primary symbol label in `base_dtb_symbols`.
+ """
+ base_dtb_symbols = {}
+ path_to_base_dtb_symbol = {}
+
+ for label, chosen_path in dtbs[0].symbols.items():
+ if not all(d.symbols.get(label) == chosen_path for d in dtbs[1:]):
+ continue
+ base_node = base_nodes.get(chosen_path)
+ if base_node is None:
+ continue
+ base_dtb_symbols[label] = chosen_path
+ if (base_node.phandle is not None and
+ chosen_path not in path_to_base_dtb_symbol):
+ path_to_base_dtb_symbol[chosen_path] = label
+
+ return base_dtb_symbols, path_to_base_dtb_symbol
+
+
+def _build_base_tree(dtbs):
+ """Construct the normalized `FactoredTree` nodes for the common base DTB.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ Pre-order `path -> FactoredTree` dict containing all common nodes,
+ their assigned `phandle` attributes, and their best shared
+ `(masked_bytes, fixups)` properties.
+ """
+ base_nodes = _find_common_nodes_and_phandles(dtbs)
+ for path, base_node in base_nodes.items():
+ dtb_nodes = [d.nodes[path] for d in dtbs]
+ common_prop_names = [
+ k for k in dtb_nodes[0].props
+ if all(k in n.props for n in dtb_nodes[1:])
+ ]
+ for prop_name in common_prop_names:
+ eligible = []
+ for n in dtb_nodes:
+ fixups = n.fixups.get(prop_name, ())
+ if not all(t_path in base_nodes for _, t_path in fixups):
+ continue
+ eligible.append((n.props[prop_name], fixups))
+
+ chosen = select_best_candidate(eligible, path, prop_name)
+ if chosen is None:
+ continue
+ prop, fixups = chosen
+ base_node.set_prop(prop)
+ base_node.fixups[prop_name] = fixups
+
+ return base_nodes
+
+
+def _serialize_base_dtb(base_nodes, base_dtb_symbols, rsvmap, boot_cpuid_phys,
+ emit_local_fixups=True):
+ """Materialize phandles, `/__symbols__`, and `/__local_fixups__` and flatten.
+
+ Args:
+ base_nodes: Pre-order `path -> FactoredTree` dict for the base DTB from
+ `_build_base_tree()`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ rsvmap: Sequence of `(address, size)` memory reservation tuples.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+ emit_local_fixups: If True, include `/__local_fixups__` on the
+ generated `base.dtb`.
+
+ Returns:
+ Serialized `.dtb` `bytes` for the common base DTB.
+ """
+ out_root = libpydt.DeviceTree()
+ lf_entries = []
+
+ for path, base_node in base_nodes.items():
+ out_node = out_root.create_path(path)
+ for prop_name, prop in base_node.props.items():
+ fixups = base_node.fixups.get(prop_name, ())
+ if not fixups:
+ out_node.set_prop(prop)
+ else:
+ val = bytearray(prop)
+ for off, t_path in fixups:
+ target_ph = base_nodes[t_path].phandle
+ struct.pack_into('>I', val, off, target_ph)
+ out_node.set_prop(libfdt.Property(prop_name, bytes(val)))
+ lf_entries.append((path, prop_name, [off for off, _ in fixups]))
+
+ if base_node.phandle is not None:
+ out_node.set_prop(
+ libfdt.Property.from_uint32('phandle', base_node.phandle)
+ )
+
+ any_phandle = any(n.phandle is not None for n in base_nodes.values())
+ if base_dtb_symbols or (emit_local_fixups and any_phandle):
+ sym_node = out_root.create_path('/__symbols__')
+ for label, target_path in base_dtb_symbols.items():
+ sym_node.set_prop(
+ libfdt.Property.from_str(label, target_path or '/')
+ )
+
+ if emit_local_fixups:
+ # Create `/__local_fixups__` on the base DTB so `fdt_overlay_apply()`
+ # updates it with the overlay's phandle fixups when merging.
+ lf_root = out_root.create_path('/__local_fixups__')
+ for path, prop_name, offs in lf_entries:
+ lf_root.create_path(path).set_prop(
+ libfdt.Property.from_uint32_list(prop_name, offs)
+ )
+
+ fdt = out_root.flatten(mem_rsv=rsvmap, boot_cpuid_phys=boot_cpuid_phys)
+ return bytes(fdt.as_bytearray())
+
+
+def _assign_local_phandles(dtb, base_nodes):
+ """Assign sequential overlay-local phandles (`1, 2, ...`) to board-only nodes.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+
+ Returns:
+ Dict mapping each board-only node path that has a phandle in `dtb` to
+ its sequential overlay-local integer phandle (`1, 2, ...`).
+ """
+ local_phandles = {}
+ next_local_ph = 1
+ for path, node in dtb.nodes.items():
+ if node.phandle is not None and path not in base_nodes:
+ local_phandles[path] = next_local_ph
+ next_local_ph += 1
+ return local_phandles
+
+
+def _encode_dtbo_property(dtb_filename, path, prop_name, masked_bytes, fixups,
+ base_nodes, path_to_base_dtb_symbol, local_phandles,
+ ext_fixups, ov_local_fixups):
+ """Encode a property value for `/fragment@0/__overlay__` and record fixups.
+
+ Args:
+ dtb_filename: Input DTB filename (for error reporting).
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+ masked_bytes: Normalized property bytes (with zeroed phandle cells).
+ fixups: Tuple of `(offset, target_path)` phandle fixups for this
+ property.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+ local_phandles: Dict mapping board-only node paths to overlay-local
+ phandles.
+ ext_fixups: Dict mapping target base symbol labels to lists of
+ `/__fixups__` entry strings (mutated in place).
+ ov_local_fixups: Dict mapping `(path, prop_name)` to lists of byte
+ offsets for `/__local_fixups__` (mutated in place).
+
+ Returns:
+ Encoded property value `bytes` with `0xffffffff` placeholders for
+ external base references and local phandle integers for overlay-local
+ references.
+ """
+ if not fixups:
+ return masked_bytes
+
+ val = bytearray(masked_bytes)
+ for off, t_path in fixups:
+ if t_path in base_nodes:
+ sym_label = path_to_base_dtb_symbol.get(t_path)
+ if sym_label is None:
+ raise ValueError(
+ f'{dtb_filename}: property {path or "/"}:{prop_name} '
+ f'references common base node {t_path or "/"} which lacks '
+ f'a common symbol in /__symbols__'
+ )
+ struct.pack_into('>I', val, off, 0xffffffff)
+ entry = f'/fragment@0/__overlay__{path}:{prop_name}:{off}'
+ ext_fixups.setdefault(sym_label, []).append(entry)
+ else:
+ struct.pack_into('>I', val, off, local_phandles[t_path])
+ ov_local_fixups.setdefault((path, prop_name), []).append(off)
+ return bytes(val)
+
+
+def _build_board_dtbo(dtb, base_nodes, base_dtb_symbols,
+ path_to_base_dtb_symbol):
+ """Construct and flatten the `.dtbo` overlay blob for a single board.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+
+ Returns:
+ Serialized `.dtbo` overlay `bytes` for `dtb`.
+ """
+ dtbo_root = libpydt.DeviceTree()
+ frag_node = dtbo_root.create_path('/fragment@0')
+ frag_node.set_prop(libfdt.Property.from_str('target-path', '/'))
+ ov_root = frag_node.create_path('__overlay__')
+
+ local_phandles = _assign_local_phandles(dtb, base_nodes)
+ ext_fixups = {}
+ ov_local_fixups = {}
+
+ for path, node in dtb.nodes.items():
+ base_node = base_nodes.get(path)
+ if (base_node is not None and
+ base_node.props == node.props and
+ base_node.fixups == node.fixups):
+ continue
+ diff_props = [
+ prop_name for prop_name, raw_val in node.props.items()
+ if (base_node is None or
+ base_node.props.get(prop_name) != raw_val or
+ base_node.fixups.get(prop_name, ()) !=
+ node.fixups.get(prop_name, ()))
+ ]
+ if base_node is not None and not diff_props:
+ continue
+ ov_node = ov_root.create_path(path)
+ for prop_name in diff_props:
+ ov_node.set_prop(
+ libfdt.Property(
+ prop_name,
+ _encode_dtbo_property(
+ dtb.filename, path, prop_name,
+ node.props[prop_name],
+ node.fixups.get(prop_name, ()), base_nodes,
+ path_to_base_dtb_symbol, local_phandles, ext_fixups,
+ ov_local_fixups,
+ ),
+ )
+ )
+ if path in local_phandles:
+ ov_node.set_prop(
+ libfdt.Property.from_uint32('phandle', local_phandles[path])
+ )
+
+ diff_syms = [
+ (label, t_path)
+ for label, t_path in dtb.symbols.items()
+ if base_dtb_symbols.get(label) != t_path
+ ]
+ if diff_syms:
+ sym_node = dtbo_root.create_path('/__symbols__')
+ for label, t_path in diff_syms:
+ ov_root.create_path(t_path)
+ sym_node.set_prop(
+ libfdt.Property.from_str(
+ label, f'/fragment@0/__overlay__{t_path}'
+ )
+ )
+
+ if ext_fixups:
+ fixups_node = dtbo_root.create_path('/__fixups__')
+ for label, entries in ext_fixups.items():
+ fixups_node.set_prop(
+ libfdt.Property.from_stringlist(label, entries)
+ )
+
+ if ov_local_fixups:
+ ov_lf_root = dtbo_root.create_path(
+ '/__local_fixups__/fragment@0/__overlay__'
+ )
+ for (path, prop_name), offs in ov_local_fixups.items():
+ ov_lf_root.create_path(path).set_prop(
+ libfdt.Property.from_uint32_list(prop_name, offs)
+ )
+
+ return bytes(dtbo_root.flatten().as_bytearray())
+
+
+def create_base_dtb_and_overlays(dtbs):
+ """Compute the base DTB and per-board DTBO blobs from parsed DTBs.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances (at least two).
+
+ Returns:
+ Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
+ serialized `base.dtb` `bytes` and `dtbo_blobs` is a list of serialized
+ `.dtbo` `bytes` corresponding 1:1 with `dtbs`.
+ """
+ _validate_dtb_headers(dtbs)
+ base_nodes = _build_base_tree(dtbs)
+ base_dtb_symbols, path_to_base_dtb_symbol = _find_common_symbols(
+ dtbs, base_nodes
+ )
+ base_dtb_blob = _serialize_base_dtb(
+ base_nodes, base_dtb_symbols,
+ dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
+ emit_local_fixups=any(d.has_local_fixups for d in dtbs),
+ )
+ dtbo_blobs = [
+ _build_board_dtbo(
+ d, base_nodes, base_dtb_symbols, path_to_base_dtb_symbol,
+ )
+ for d in dtbs
+ ]
+ return base_dtb_blob, dtbo_blobs
+
+
+def get_dtbo_output_path(dtb_path, outdir):
+ """Determine the output `.dtbo` file path for an input `.dtb` path.
+
+ Args:
+ dtb_path: Path to the input `.dtb` file.
+ outdir: Output directory path, or `None` to place the `.dtbo` alongside
+ `dtb_path`.
+
+ Returns:
+ Output `.dtbo` file path string.
+ """
+ dtb_dir, filename = os.path.split(dtb_path)
+ stem = filename[:-4] if filename.endswith('.dtb') else filename
+ target_dir = outdir if outdir is not None else dtb_dir
+ return os.path.join(target_dir, f'{stem}.dtbo')
+
+
+def parse_args(argv=None):
+ """Parse command-line arguments.
+
+ Args:
+ argv: Optional list of command-line argument strings (defaults to
+ `sys.argv[1:]`).
+
+ Returns:
+ Parsed `argparse.Namespace` instance.
+ """
+ parser = argparse.ArgumentParser(
+ description=(
+ 'Split a set of compiled device tree blobs (.dtb) into a single '
+ 'common base .dtb and per-board overlay blobs (.dtbo).'
+ )
+ )
+ parser.add_argument(
+ '--output-base-dtb',
+ default='base.dtb',
+ metavar='FILENAME',
+ help=(
+ 'Output common base .dtb filename (default: base.dtb; placed in '
+ '--output if specified, or current directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--output',
+ default=None,
+ metavar='DIR',
+ help=(
+ 'Output directory for generated .dtbo files and '
+ '--output-base-dtb (default: alongside each input .dtb for .dtbo, '
+ 'current directory for --output-base-dtb)'
+ ),
+ )
+ parser.add_argument(
+ '-s', '--stats',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+ parser.add_argument(
+ 'dtbs',
+ nargs='+',
+ metavar='DTB',
+ help='Input .dtb files (at least two required)',
+ )
+ args = parser.parse_args(argv)
+ if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a non-empty filename without directory '
+ 'components; use -o/--output to specify the output directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
+ return args
+
+
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
+ """Print uncompressed size savings summary.
+
+ Args:
+ dtbs: Sequence of input `ParsedDtb` instances.
+ base_dtb_blob: Uncompressed base DTB `bytes`.
+ dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ """
+ orig_uncomp = sum(d.uncompressed_size for d in dtbs)
+ base_dtb_uncomp = len(base_dtb_blob)
+ dtbo_uncomp = sum(len(b) for b in dtbo_blobs)
+ new_uncomp = base_dtb_uncomp + dtbo_uncomp
+ saved_uncomp = orig_uncomp - new_uncomp
+ pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
+ print(
+ f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
+ f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
+ )
+
+
+def factor_dtb_files(args):
+ """Factor a list of standalone .dtb files into a base .dtb and .dtbo files.
+
+ Args:
+ args: Parsed `argparse.Namespace` from `parse_args()`.
+ """
+ input_dtb_paths = args.dtbs
+ dtbs = [ParsedDtb(p) for p in input_dtb_paths]
+
+ base_dtb_path = (
+ os.path.join(args.output, args.output_base_dtb)
+ if args.output is not None
+ else args.output_base_dtb
+ )
+
+ input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
+ base_abs_path = os.path.abspath(base_dtb_path)
+ if base_abs_path in input_abs_paths:
+ raise ValueError(
+ f'base DTB output path {base_dtb_path!r} collides with '
+ f'an input DTB file'
+ )
+
+ dtbo_paths = []
+ seen_paths = {base_abs_path}
+ for dtb_path in input_dtb_paths:
+ out_path = get_dtbo_output_path(dtb_path, args.output)
+ abs_out = os.path.abspath(out_path)
+ if abs_out in input_abs_paths:
+ raise ValueError(
+ f'output path {out_path!r} collides with an input DTB file'
+ )
+ if abs_out in seen_paths:
+ raise ValueError(
+ f'output path collision on {out_path!r}; ensure input '
+ f'files have distinct names'
+ )
+ seen_paths.add(abs_out)
+ dtbo_paths.append(out_path)
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+
+ if args.output:
+ os.makedirs(args.output, exist_ok=True)
+ with open(base_dtb_path, 'wb') as f:
+ f.write(base_dtb_blob)
+
+ for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ out_dir = os.path.dirname(out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(out_path, 'wb') as f:
+ f.write(dtbo_blob)
+
+ if args.stats:
+ _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+
+
+def main(argv=None):
+ """Run the fdtfactor command-line tool.
+
+ Args:
+ argv: Optional list of command-line argument strings.
+
+ Returns:
+ Integer exit status (`0` on success).
+ """
+ args = parse_args(argv)
+ try:
+ factor_dtb_files(args)
+ except (OSError, ValueError, RuntimeError, struct.error,
+ libfdt.FdtException) as exc:
+ sys.exit(f'Error: {exc}')
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/meson.build b/meson.build
index 348d9b555e34..97dc66c24ab4 100644
--- a/meson.build
+++ b/meson.build
@@ -119,6 +119,7 @@ if get_option('tools') and not wheel_only
install_data(
'dtdiff',
+ 'fdtfactor',
install_dir: get_option('bindir'),
install_mode: 'rwxr-xr-x',
)
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 623d71786701..9e8d09d37d8b 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,10 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
-TESTS_CLEANDIRS_L = fs
+TESTS_CLEANDIRS_L = fs *.test.dir
TESTS_CLEANDIRS = $(TESTS_CLEANDIRS_L:%=$(TESTS_PREFIX)%)
.PHONY: tests
diff --git a/tests/fdtfactor_board1.dts b/tests/fdtfactor_board1.dts
new file mode 100644
index 000000000000..324a3f23a488
--- /dev/null
+++ b/tests/fdtfactor_board1.dts
@@ -0,0 +1,72 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board1";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ };
+
+ b1_node: board1-node {
+ prop-str = "board1";
+ ref-base = <&sn1>;
+ ref-local = <&b1_node &b1_child>;
+
+ b1_child: child {
+ prop-int = <1>;
+ };
+ };
+};
diff --git a/tests/fdtfactor_board2.dts b/tests/fdtfactor_board2.dts
new file mode 100644
index 000000000000..7e4a71299cd1
--- /dev/null
+++ b/tests/fdtfactor_board2.dts
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board2";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ board2-only-prop = "only-on-board2";
+ board2-ref = <&b2_node>;
+ };
+
+ b2_node: board2-node {
+ prop-str = "board2";
+ ref-base = <&ssn1>;
+ ref-self = <&b2_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3.dts b/tests/fdtfactor_board3.dts
new file mode 100644
index 000000000000..74087efbd493
--- /dev/null
+++ b/tests/fdtfactor_board3.dts
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-base = <&sn2>;
+ ref-self = <&b3_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3_shifted.dts b/tests/fdtfactor_board3_shifted.dts
new file mode 100644
index 000000000000..263503f851b8
--- /dev/null
+++ b/tests/fdtfactor_board3_shifted.dts
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-self = <&b3_node>;
+ ref-base = <&sn2>;
+ };
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+};
diff --git a/tests/meson.build b/tests/meson.build
index 9f0f51013530..5d1c3a798c96 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -144,7 +144,7 @@ run_test_deps = [
dtc_tools, treegen_dtb, tests_exe,
]
if pylibfdt_enabled
- run_test_types += 'pylibfdt'
+ run_test_types += ['pylibfdt', 'fdtfactor']
run_test_deps += pylibfdt
endif
foreach test_type : run_test_types
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index e16767d3f3a2..815233227df1 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1207,6 +1207,106 @@ pylibfdt_tests () {
run_python_tests "$SRCDIR/libpydt_tests.py"
}
+fdtfactor_tests () {
+ check_factored_dtb () {
+ exp="$1"
+ got="$2"
+ if [ "$OVERLAY_LOCAL_FIXUPS" -ne 2 ]; then
+ cp "$exp" "$got.exp.test.dtb"
+ $DTPUT -r "$got.exp.test.dtb" /__local_fixups__ 2>/dev/null || true
+ $DTPUT -r "$got" /__local_fixups__ 2>/dev/null || true
+ exp="$got.exp.test.dtb"
+ fi
+ run_test dtbs_equal_unordered "$exp" "$got"
+ }
+
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_board${i}.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+
+ # Default mode: factor 3 boards into base.dtb + .dtbo overlays and verify
+ # that majority properties land in base.dtb and fdtoverlay reconstructs
+ # each original DTB (including /__local_fixups__, /memreserve/, and
+ # boot_cpuid_phys).
+ rm -rf fdtfactor_out.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s --output-base-dtb base.test.dtb \
+ -o fdtfactor_out.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ run_fdtget_test "deadbeef" -tx fdtfactor_out.test.dir/base.test.dtb \
+ / prop-int
+ run_wrap_error_test $DTGET \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ /fragment@0/__overlay__ prop-int
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_out.test.dir/base.test.dtb \
+ -o fdtfactor_out.test.dir/merged${i}.test.dtb \
+ fdtfactor_out.test.dir/fdtfactor_board${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fdtfactor_out.test.dir/merged${i}.test.dtb
+ done
+
+ # Shifted phandles: fdtfactor_board3_shifted.dts defines board3-node before
+ # subnode@1 so all phandle numbers differ from board1 and board2; verify
+ # that fdtfactor normalizes them via /__local_fixups__.
+ run_dtc_test -b 2 -@ -L -I dts -O dtb \
+ -o fdtfactor_board3_shifted.test.dtb \
+ "$SRCDIR/fdtfactor_board3_shifted.dts"
+ rm -rf fdtfactor_shift.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -o fdtfactor_shift.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3_shifted.test.dtb
+ run_wrap_error_test $DTGET \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo \
+ /fragment@0/__overlay__/subnode@1 prop-ref
+ run_wrap_test $FDTOVERLAY -i fdtfactor_shift.test.dir/base.dtb \
+ -o fdtfactor_shift.test.dir/merged3.test.dtb \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo
+ check_factored_dtb fdtfactor_board3.test.dtb \
+ fdtfactor_shift.test.dir/merged3.test.dtb
+
+ # DTBs compiled with dtc -@ -L that contain no labels or phandle references
+ # (for which dtc omits /__symbols__ and /__local_fixups__) are accepted.
+ printf '/dts-v1/;\n/ { common = <1>; b = "a"; };\n' > fdtfactor_noph1.test.dts
+ printf '/dts-v1/;\n/ { common = <1>; b = "b"; };\n' > fdtfactor_noph2.test.dts
+ for i in 1 2; do
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_noph${i}.test.dtb \
+ fdtfactor_noph${i}.test.dts
+ run_test check_path fdtfactor_noph${i}.test.dtb \
+ not-exists "/__symbols__"
+ run_test check_path fdtfactor_noph${i}.test.dtb \
+ not-exists "/__local_fixups__"
+ done
+ rm -rf fdtfactor_noph.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -o fdtfactor_noph.test.dir \
+ fdtfactor_noph1.test.dtb fdtfactor_noph2.test.dtb
+ for i in 1 2; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_noph.test.dir/base.dtb \
+ -o fdtfactor_noph.test.dir/merged${i}.test.dtb \
+ fdtfactor_noph.test.dir/fdtfactor_noph${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_noph${i}.test.dtb \
+ fdtfactor_noph.test.dir/merged${i}.test.dtb
+ done
+
+ # Legacy linux,phandle properties (dtc -H both / -H legacy), overlay .dtbo
+ # inputs, and inputs lacking /__local_fixups__ are rejected.
+ run_dtc_test -H both -@ -L -I dts -O dtb \
+ -o fdtfactor_legacy_ph.test.dtb "$SRCDIR/fdtfactor_board1.dts"
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_legacy_ph.test.dtb fdtfactor_board2.test.dtb
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ fdtfactor_board2.test.dtb
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -I dts -O dtb \
+ -o fdtfactor_board${i}.nolf.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+}
+
while getopts "vt:me" ARG ; do
case $ARG in
"v")
@@ -1227,14 +1327,15 @@ done
if [ -z "$TESTSETS" ]; then
TESTSETS="libfdt utilfdt dtc dtbs_equal fdtget fdtput fdtdump fdtoverlay"
- # Test pylibfdt if the libfdt Python module is available.
+ # Test pylibfdt and fdtfactor if the libfdt Python module is available.
if ! $no_python; then
- TESTSETS="$TESTSETS pylibfdt"
+ TESTSETS="$TESTSETS pylibfdt fdtfactor"
fi
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -rf *.test.dir
for set in $TESTSETS; do
case $set in
@@ -1265,6 +1366,9 @@ for set in $TESTSETS; do
"fdtoverlay")
fdtoverlay_tests
;;
+ "fdtfactor")
+ fdtfactor_tests
+ ;;
esac
done
diff --git a/tests/testutils.sh b/tests/testutils.sh
index 6b2f0d14e3e5..7f35ba8e4158 100644
--- a/tests/testutils.sh
+++ b/tests/testutils.sh
@@ -27,6 +27,7 @@ DTGET=${TEST_BINDIR}/fdtget
DTPUT=${TEST_BINDIR}/fdtput
FDTDUMP=${TEST_BINDIR}/fdtdump
FDTOVERLAY=${TEST_BINDIR}/fdtoverlay
+FDTFACTOR=${SRCDIR}/../fdtfactor
verbose_run () {
if [ -z "$QUIET_TEST" ]; then
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (9 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
By default, `fdtfactor` preserves `/__local_fixups__` on `base.dtb` so
that when `fdt_overlay_apply()` merges a `.dtbo` overlay, the
reconstructed device tree retains full `/__local_fixups__` information
(for instance, if an OS kernel wants to know which property cells
contain phandles at runtime).
When `-L` was passed to `dtc` solely to help `fdtfactor` locate phandle
cells at build time and `/__local_fixups__` is not needed at runtime,
retaining `/__local_fixups__` in `base.dtb` wastes space.
Add `-S` / `--strip-local-fixups` to omit `/__local_fixups__` from
`base.dtb` (and therefore from the merged device trees produced by
`fdt_overlay_apply()`). On a set of 47 Qualcomm SC7180 Trogdor
Chromebook board DTBs, `fdtfactor` without `-S` reduces total
uncompressed size from 7.69 MB to 932 KB (88.2% savings), while
`fdtfactor -S` reduces it to 634 KB (91.8% savings, saving an additional
298 KB by omitting `/__local_fixups__` from `base.dtb`).
Add a test case in `tests/run_tests.sh` verifying that `fdtfactor -S`
followed by `fdtoverlay` reproduces DTBs compiled with `-@` (without
`-L`) via `dtbs_equal_unordered`.
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Use 4-space indentation instead of mixed tabs and spaces in
tests/run_tests.sh, and update context for -o/--output (David Gibson)
Changes in v3:
- Include both non--S (932 KB) and -S (634 KB) Trogdor size numbers in
the commit message to show the savings of -S itself (Simon Glass)
- Pass -S to the negative test verifying that inputs lacking
/__local_fixups__ are still rejected when -S is used, and move the
non--S rejection test to the base fdtfactor patch (Simon Glass)
Changes in v2:
- Split -S/--strip-local-fixups into a separate opt-in patch (David
Gibson)
README.md | 9 +++++++++
fdtfactor | 21 ++++++++++++++++++---
tests/run_tests.sh | 19 +++++++++++++++++++
3 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 9a8be74d47fe..198384cd8456 100644
--- a/README.md
+++ b/README.md
@@ -228,6 +228,15 @@ Options:
* `-o`, `--output`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ `base.dtb` and final reconstructed device trees. By default, `fdtfactor`
+ preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
+ merges a `.dtbo` overlay, the reconstructed device tree retains full
+ `/__local_fixups__` information (e.g., for an OS kernel that wants to
+ know which property cells contain phandles at runtime). When `-L` was
+ passed to `dtc` solely to help `fdtfactor` locate phandles at build time
+ and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
+ from `base.dtb` to save space.
* `-s`, `--stats`: Print size savings summary.
### TODO
diff --git a/fdtfactor b/fdtfactor
index dfbee0aea2d6..a083a8c380a0 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -675,11 +675,13 @@ def _build_board_dtbo(dtb, base_nodes, base_dtb_symbols,
return bytes(dtbo_root.flatten().as_bytearray())
-def create_base_dtb_and_overlays(dtbs):
+def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
"""Compute the base DTB and per-board DTBO blobs from parsed DTBs.
Args:
dtbs: Sequence of `ParsedDtb` instances (at least two).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated `base.dtb`.
Returns:
Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
@@ -694,7 +696,9 @@ def create_base_dtb_and_overlays(dtbs):
base_dtb_blob = _serialize_base_dtb(
base_nodes, base_dtb_symbols,
dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
- emit_local_fixups=any(d.has_local_fixups for d in dtbs),
+ emit_local_fixups=(
+ not strip_local_fixups and any(d.has_local_fixups for d in dtbs)
+ ),
)
dtbo_blobs = [
_build_board_dtbo(
@@ -757,6 +761,14 @@ def parse_args(argv=None):
'current directory for --output-base-dtb)'
),
)
+ parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from the generated base .dtb and '
+ 'reconstructed device trees'
+ ),
+ )
parser.add_argument(
'-s', '--stats',
action='store_true',
@@ -841,7 +853,10 @@ def factor_dtb_files(args):
seen_paths.add(abs_out)
dtbo_paths.append(out_path)
- base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ dtbs,
+ strip_local_fixups=args.strip_local_fixups,
+ )
if args.output:
os.makedirs(args.output, exist_ok=True)
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 815233227df1..4f5026e363fc 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1305,6 +1305,25 @@ fdtfactor_tests () {
done
run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+
+ # Strip local fixups (-S): verify that merged DTBs match DTBs compiled
+ # with -@ (without -L), and that inputs lacking /__local_fixups__ are
+ # still rejected even when -S is passed.
+ run_wrap_error_test $PYTHON $FDTFACTOR -S -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+ rm -rf fdtfactor_strip.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -S --output-base-dtb base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_strip.test.dir/base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir/merged${i}.test.dtb \
+ fdtfactor_strip.test.dir/fdtfactor_board${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
+ fdtfactor_strip.test.dir/merged${i}.test.dtb
+ done
}
while getopts "vt:me" ARG ; do
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (10 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Firmware and FIT build pipelines frequently store device tree blobs
compressed with LZ4, LZMA/XZ, or gzip (e.g., `.dtb.lz4`).
Add `-C` / `--compression` (`none`, `lz4`, `lz4_legacy`, `lzma`, `xz`,
`gzip`; default `none`) to `fdtfactor` to decompress compressed input
DTBs before factoring and compress the resulting `base.dtb` and `.dtbo`
outputs with the same algorithm:
- Support `lz4` and `lz4_legacy` (via the `lz4` CLI), `lzma` and `xz`
(via Python's `lzma` module, using `CHECK_CRC32` for XZ compatibility
with kernel/bootloader decompressors), and `gzip` (via Python's `gzip`
module).
- Default `--output-base-dtb` to `base.dtb.<ext>` matching `-C` /
`--compression` (e.g., `base.dtb.lz4` when `-C lz4` is passed), and
emit `.dtbo.<ext>` overlays.
- Report compressed size savings alongside uncompressed savings when
`-s` / `--stats` is specified with compression enabled.
On the 47 Qualcomm SC7180 Trogdor Chromebook `.dtb.lz4` files,
`fdtfactor -S -C lz4` reduces total LZ4-compressed size from 1.08 MB
(built without `-@ -L`) to 214 KB (80.2% savings).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Replace magic-byte/extension auto-detection and --base-compression
with a single explicit -C/--compression option controlling both input
and output compression (David Gibson)
- Drop unnecessary --stats documentation/help changes and use 4-space
indentation in tests/run_tests.sh (David Gibson)
- Use check_factored_dtb in fdtfactor_tests so tests pass when
OVERLAY_LOCAL_FIXUPS != 2 (Simon Glass)
Changes in v3:
- Update for -b/--base-dtb rename to --output-base-dtb (David Gibson)
Changes in v2:
- Add --base-compression to explicitly control base.dtb compression
while documenting auto-detection from --base-dtb's filename suffix
(David Gibson)
- Support legacy LZ4 frames (lz4 -l) and use CRC32 check for XZ outputs
(Simon Glass)
README.md | 11 ++-
fdtfactor | 208 ++++++++++++++++++++++++++++++++++++++-----
tests/Makefile.tests | 5 +-
tests/run_tests.sh | 51 ++++++++++-
4 files changed, 248 insertions(+), 27 deletions(-)
diff --git a/README.md b/README.md
index 198384cd8456..ac571384d421 100644
--- a/README.md
+++ b/README.md
@@ -204,7 +204,8 @@ $ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
## fdtfactor
`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`) and splits them into a single common
+compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
+`.dtb.lzma`, `.dtb.gz` with `-C`) and splits them into a single common
`base.dtb` and per-board `.dtbo` overlays so that applying a board's
`.dtbo` to `base.dtb` recreates the original board device tree while
reducing total storage size.
@@ -223,8 +224,12 @@ $ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
Options:
* `--output-base-dtb`: Output filename for the common base `.dtb`
- (default: `base.dtb`; placed in `--output` if specified, or the current
- directory otherwise).
+ (default: `base.dtb`, or `base.dtb.<ext>` when `-C` / `--compression` is
+ used; placed in `--output` if specified, or the current directory
+ otherwise).
+* `-C`, `--compression`: Compression format for both input `.dtb` files and
+ generated `base.dtb` / `.dtbo` outputs: `none`, `lz4`, `lz4_legacy`, `lzma`,
+ `xz`, or `gzip` (default: `none`).
* `-o`, `--output`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
diff --git a/fdtfactor b/fdtfactor
index a083a8c380a0..beeeff231ba6 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -12,14 +12,125 @@ original board DTB.
import argparse
from collections import Counter
+import gzip
+import lzma
import os
import struct
+import subprocess
import sys
import libfdt
import libpydt
+COMP_TO_EXT = {
+ 'none': '',
+ 'lz4': '.lz4',
+ 'lz4_legacy': '.lz4',
+ 'lzma': '.lzma',
+ 'xz': '.xz',
+ 'gzip': '.gz',
+}
+
+
+def _run_lz4(args, input_bytes, action):
+ """Run the `lz4` CLI with `args` on `input_bytes` and return stdout.
+
+ Args:
+ args: List of command-line arguments to pass to `lz4`.
+ input_bytes: Bytes to feed to `lz4` standard input.
+ action: Human-readable action description for error messages.
+
+ Returns:
+ Output `bytes` written to `lz4` standard output.
+ """
+ try:
+ res = subprocess.run(
+ ['lz4', *args],
+ input=input_bytes,
+ capture_output=True,
+ check=False,
+ )
+ except FileNotFoundError as exc:
+ raise RuntimeError(
+ 'lz4 command not found; install lz4 to process .lz4 files'
+ ) from exc
+ if res.returncode != 0:
+ err = res.stderr.decode('utf-8', errors='replace').strip()
+ raise RuntimeError(f'Failed to {action} with lz4: {err}')
+ return res.stdout
+
+
+def decompress_blob(data, comp, filename=''):
+ """Decompress a DTB blob according to `comp`.
+
+ Args:
+ data: Input file or property `bytes` (compressed or uncompressed).
+ comp: Compression format (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ filename: Optional filename or node description for error reporting.
+
+ Returns:
+ Uncompressed FDT `bytes`.
+ """
+ if comp in ('none', '', None):
+ return data
+ target = filename or 'blob'
+ if comp in ('lz4', 'lz4_legacy'):
+ return _run_lz4(['-d', '-q', '-c'], data, f'decompress {target}')
+ if comp == 'gzip':
+ try:
+ return gzip.decompress(data)
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {target} with gzip: {exc}'
+ ) from exc
+ if comp in ('lzma', 'xz'):
+ try:
+ return lzma.decompress(data)
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {target} with {comp}: {exc}'
+ ) from exc
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
+def compress_blob(raw_bytes, comp, filename=''):
+ """Compress an FDT blob using the specified compression algorithm.
+
+ Args:
+ raw_bytes: Uncompressed FDT `bytes`.
+ comp: Compression format (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ filename: Optional filename for error messages.
+
+ Returns:
+ Compressed (or uncompressed if `comp` is `'none'`) `bytes`.
+ """
+ if comp in ('none', '', None):
+ return raw_bytes
+ target = filename or 'blob'
+ if comp == 'lz4':
+ return _run_lz4(
+ ['-12', '-z', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'lz4_legacy':
+ return _run_lz4(
+ ['-l', '-12', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'gzip':
+ return gzip.compress(raw_bytes, mtime=0)
+ if comp == 'lzma':
+ return lzma.compress(raw_bytes, format=lzma.FORMAT_ALONE)
+ if comp == 'xz':
+ return lzma.compress(
+ raw_bytes, format=lzma.FORMAT_XZ, check=lzma.CHECK_CRC32
+ )
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
def _is_dtbo_overlay(root):
"""Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
@@ -64,7 +175,7 @@ class ParsedDtb:
- Stores FDT header, memory reservation, and file-level metadata that live
outside the root node tree (`boot_cpuid_phys`, `mem_rsv`, `filename`,
- `uncompressed_size`).
+ `compression`, `compressed_size`, `uncompressed_size`).
- Pops `/__symbols__` (parsed into `self.symbols`) and `/__local_fixups__`
out of `self.root`, and indexes the remaining tree in pre-order in
`self.nodes` (`path -> FactoredTree`).
@@ -75,15 +186,19 @@ class ParsedDtb:
independent of numeric phandle numbering.
"""
- def __init__(self, filename):
+ def __init__(self, filename, comp='none'):
"""Parse an input FDT file into a normalized FactoredTree.
Args:
- filename: File path of the `.dtb` file to read.
+ filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
+ comp: Compression format string (default: `'none'`).
"""
self.filename = filename
+ self.compression = comp
with open(filename, 'rb') as f:
- raw_bytes = f.read()
+ file_bytes = f.read()
+ self.compressed_size = len(file_bytes)
+ raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
self.uncompressed_size = len(raw_bytes)
try:
@@ -709,21 +824,26 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
-def get_dtbo_output_path(dtb_path, outdir):
- """Determine the output `.dtbo` file path for an input `.dtb` path.
+def get_dtbo_output_path(dtb_path, outdir, comp='none'):
+ """Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
Args:
- dtb_path: Path to the input `.dtb` file.
+ dtb_path: Path to the input `.dtb` (or `.dtb.*`) file.
outdir: Output directory path, or `None` to place the `.dtbo` alongside
`dtb_path`.
+ comp: Compression format string for the output `.dtbo` (default:
+ `'none'`).
Returns:
- Output `.dtbo` file path string.
+ Output `.dtbo` (or compressed `.dtbo.*`) file path string.
"""
dtb_dir, filename = os.path.split(dtb_path)
+ ext = COMP_TO_EXT[comp]
+ if ext and filename.endswith(ext):
+ filename = filename[:-len(ext)]
stem = filename[:-4] if filename.endswith('.dtb') else filename
target_dir = outdir if outdir is not None else dtb_dir
- return os.path.join(target_dir, f'{stem}.dtbo')
+ return os.path.join(target_dir, f'{stem}.dtbo{ext}')
def parse_args(argv=None):
@@ -738,17 +858,28 @@ def parse_args(argv=None):
"""
parser = argparse.ArgumentParser(
description=(
- 'Split a set of compiled device tree blobs (.dtb) into a single '
- 'common base .dtb and per-board overlay blobs (.dtbo).'
+ 'Split a set of compiled device tree blobs (.dtb, or compressed '
+ '.dtb.lz4/.xz/.lzma/.gz with -C) into a single common base .dtb '
+ 'and per-board overlay blobs (.dtbo).'
)
)
parser.add_argument(
'--output-base-dtb',
- default='base.dtb',
+ default=None,
metavar='FILENAME',
help=(
- 'Output common base .dtb filename (default: base.dtb; placed in '
- '--output if specified, or current directory otherwise)'
+ 'Output common base .dtb filename (default: base.dtb, or '
+ 'base.dtb.<ext> when -C/--compression is used; placed in --output '
+ 'if specified, or current directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '-C', '--compression',
+ choices=tuple(COMP_TO_EXT),
+ default='none',
+ help=(
+ 'Compression format for both input DTBs and output base.dtb/.dtbo '
+ 'files (default: none)'
),
)
parser.add_argument(
@@ -778,9 +909,14 @@ def parse_args(argv=None):
'dtbs',
nargs='+',
metavar='DTB',
- help='Input .dtb files (at least two required)',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
+ '(at least two required)'
+ ),
)
args = parser.parse_args(argv)
+ if args.output_base_dtb is None:
+ args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
parser.error(
'--output-base-dtb must be a non-empty filename without directory '
@@ -791,13 +927,17 @@ def parse_args(argv=None):
return args
-def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
- """Print uncompressed size savings summary.
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
+ dtbo_csizes):
+ """Print uncompressed and compressed size savings summary.
Args:
dtbs: Sequence of input `ParsedDtb` instances.
base_dtb_blob: Uncompressed base DTB `bytes`.
dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ comp: Compression format of the inputs and outputs.
+ base_dtb_csize: Byte size of the compressed base DTB output.
+ dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
"""
orig_uncomp = sum(d.uncompressed_size for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -811,6 +951,18 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
+ if comp != 'none':
+ orig_comp = sum(d.compressed_size for d in dtbs)
+ dtbo_csize = sum(dtbo_csizes)
+ new_comp = base_dtb_csize + dtbo_csize
+ saved_comp = orig_comp - new_comp
+ pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
+ print(
+ f'Compressed ({comp}): {len(dtbs)} files ({orig_comp:,} B) -> '
+ f'base {base_dtb_csize:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_csize:,} B (avg {dtbo_csize // len(dtbo_blobs):,} B) = '
+ f'{new_comp:,} B (saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ )
def factor_dtb_files(args):
@@ -819,8 +971,9 @@ def factor_dtb_files(args):
Args:
args: Parsed `argparse.Namespace` from `parse_args()`.
"""
+ comp = args.compression
input_dtb_paths = args.dtbs
- dtbs = [ParsedDtb(p) for p in input_dtb_paths]
+ dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
base_dtb_path = (
os.path.join(args.output, args.output_base_dtb)
@@ -839,7 +992,7 @@ def factor_dtb_files(args):
dtbo_paths = []
seen_paths = {base_abs_path}
for dtb_path in input_dtb_paths:
- out_path = get_dtbo_output_path(dtb_path, args.output)
+ out_path = get_dtbo_output_path(dtb_path, args.output, comp=comp)
abs_out = os.path.abspath(out_path)
if abs_out in input_abs_paths:
raise ValueError(
@@ -858,20 +1011,31 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
+ base_dtb_file_bytes = compress_blob(
+ base_dtb_blob, comp, filename=base_dtb_path
+ )
if args.output:
os.makedirs(args.output, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
- f.write(base_dtb_blob)
+ f.write(base_dtb_file_bytes)
+ dtbo_file_sizes = []
for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
+ out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
+ dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
- f.write(dtbo_blob)
+ f.write(out_bytes)
if args.stats:
- _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+ _print_dtb_stats(
+ dtbs, base_dtb_blob, dtbo_blobs,
+ comp=comp,
+ base_dtb_csize=len(base_dtb_file_bytes),
+ dtbo_csizes=dtbo_file_sizes,
+ )
def main(argv=None):
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 9e8d09d37d8b..1eed9ba6478a 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
+ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
+ tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
TESTS_CLEANDIRS_L = fs *.test.dir
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 4f5026e363fc..9b005396bc79 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1324,6 +1324,53 @@ fdtfactor_tests () {
run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
fdtfactor_strip.test.dir/merged${i}.test.dtb
done
+
+ # Compressed (.dtb.gz / .dtb.xz / .dtb.lzma / .dtb.lz4) modes via -C
+ for comp_spec in "gzip:gz:gzip -9 -k -f:gzip -d -f" \
+ "xz:xz:xz -k -f:xz -d -f" \
+ "lzma:lzma:xz --format=lzma -k -f:xz -d -f" \
+ "lz4:lz4:lz4 -12 -q -f -m:lz4 -d -q -f --rm" \
+ "lz4_legacy:lz4:lz4 -l -12 -q -f -m:lz4 -d -q -f --rm"; do
+ algo="${comp_spec%%:*}"
+ rest="${comp_spec#*:}"
+ ext="${rest%%:*}"
+ rest="${rest#*:}"
+ comp_cmd="${rest%%:*}"
+ decomp_cmd="${rest#*:}"
+ if ! command -v "${comp_cmd%% *}" >/dev/null 2>&1; then
+ continue
+ fi
+ run_wrap_test $comp_cmd \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ rm -rf fdtfactor_comp.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s -C "$algo" \
+ -o fdtfactor_comp.test.dir \
+ "fdtfactor_board1.test.dtb.$ext" \
+ "fdtfactor_board2.test.dtb.$ext" \
+ "fdtfactor_board3.test.dtb.$ext"
+ run_wrap_test $decomp_cmd "fdtfactor_comp.test.dir/base.dtb.$ext"
+ run_wrap_test $decomp_cmd \
+ "fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo.$ext"
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_comp.test.dir/base.dtb \
+ -o fdtfactor_comp.test.dir/merged1.test.dtb \
+ fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo
+ check_factored_dtb fdtfactor_board1.test.dtb \
+ fdtfactor_comp.test.dir/merged1.test.dtb
+ done
+
+ # Verify that mismatched compression (compressed inputs without -C, or
+ # uncompressed inputs with -C) fails rather than guessing.
+ if command -v gzip >/dev/null 2>&1; then
+ run_wrap_test gzip -9 -k -f \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_comp_err.test.dir \
+ fdtfactor_board1.test.dtb.gz fdtfactor_board2.test.dtb.gz
+ run_wrap_error_test $PYTHON $FDTFACTOR -C gzip \
+ -o fdtfactor_comp_err.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1353,7 +1400,9 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+ *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
+ *.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
for set in $TESTSETS; do
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (11 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Bootloaders such as depthcharge on Chromebooks and U-Boot use Flattened
Image Trees (FITs) to bundle a kernel image with multiple board FDT
blobs (`fdt-1`, `fdt-2`, ...) and per-board configuration nodes.
Add `-F`/`--fit`, `--min-overlap`, and `--single-base-dtb` options to
`fdtfactor` (writing the factored FIT to `-o`/`--output`, or updating
`--fit` in place by default):
- Parse non-overlay `type = "flat_dt"` nodes under `/images`,
decompressing their `data` blobs according to their `compression`
property (`none`, `lz4`, `lzma`, `xz`, `gzip`).
- Reject FITs containing `signature*` subnodes under `/images/*` or
`/configurations/*`, as modifying FDT blobs and configurations would
invalidate cryptographic signatures.
- Partition the `flat_dt` blobs into factor groups by compatible FDT
headers / FIT `arch` and `load` properties and shared normalized
property byte overlap (`>= --min-overlap`%, default `50`%, reusing
`select_best_candidate()`'s `_prop_byte_cost()` model) so a single FIT
containing multiple board/SoC families produces a separate
`fdt-base-1`, `fdt-base-2`, etc. for each family.
- Support `--single-base-dtb` to force all `flat_dt` blobs into a single
base DTB regardless of property overlap, failing only if their FDT
headers, `arch`, or `load` properties differ.
- Insert each `fdt-base-N` node into `/images` (picking the next unused
`fdt-base-N` name), replace each factored `fdt-*` node's `data` with
its `.dtbo` overlay (preserving each node's `compression` setting, and
stripping `/__local_fixups__` from any unfactored single-board
`flat_dt` nodes too when `-S` is passed), recompute any `hash*`
subnode digests, and prepend `fdt-base-N` to each matching
configuration's `fdt` string list while preserving all other
configuration properties (`description`, `compatible`, etc.,
populating `compatible` from the board DTB's root node when omitted)
and any pre-existing overlay entries.
Add uncompressed, LZ4-compressed multi-family, `--min-overlap`,
unfactored-singleton `-S`, pre-existing `fdt-base-*` / multi-pass,
signed-FIT rejection, and `--single-base-dtb` FIT test cases to the
`fdtfactor` test suite.
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Unify --outfile into -o/--output, and split standalone DTB vs. FIT
options into separate README.md subsections and --help groups (David
Gibson)
- Check _is_dtbo_overlay() directly in _extract_fit_fdt_entries()
instead of adding OverlayBlobError to ParsedDtb (David Gibson)
- Map standard FIT hash algorithm names explicitly via FIT_HASH_ALGOS
instead of passing raw names to hashlib.new() (David Gibson)
- Remove compression and entry from FIT grouping, document load, expose
--min-overlap, and compute FIT group overlap using normalized property
byte costs via _prop_byte_cost() instead of unweighted node paths
(David Gibson)
- Strip /__local_fixups__ from unfactored single-board FIT flat_dt
entries when -S/--strip-local-fixups is passed, and pick the next
unused fdt-base-N node name instead of rejecting FITs that already
contain fdt-base-* nodes
- Pass already-unflattened FactoredTree into ParsedDtb and intern
normalized properties to integer IDs in group_fit_dtbs() to speed up
multi-SoC FIT factoring
- Preserve whether input FIT FDT descriptions include a compression
extension instead of unconditionally appending one
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Use check_factored_dtb and 4-space indentation in tests/run_tests.sh
(Simon Glass, David Gibson)
Changes in v3:
- Use "FIT" instead of "FIT image" in documentation, comments, and
commit messages (Simon Glass)
- Update for DeviceTree and --output-base-dtb (Simon Glass, David
Gibson)
Changes in v2:
- Rename -f/--fit and -F/--fit-out to -F/--fit and --outfile and clarify
FIT mode in the --help description (David Gibson, Simon Glass)
- Reject signed FIT images with signature* subnodes (Simon Glass)
- Use '<base-node-name>.dtb' for FIT base FDT descriptions instead of
common-prefix heuristics (Simon Glass)
- Compare FIT group node overlap against the group's common node set
instead of only the first entry (Simon Glass)
- Document in README.md that U-Boot's boot_get_fdt_fit() currently
requires FIT overlays to be uncompressed (Simon Glass)
README.md | 89 +++-
fdtfactor | 850 +++++++++++++++++++++++++++++++++----
tests/Makefile.tests | 2 +-
tests/fdtfactor_lz4.its | 109 +++++
tests/fdtfactor_other1.dts | 67 +++
tests/fdtfactor_other2.dts | 67 +++
tests/fdtfactor_test.its | 109 +++++
tests/run_tests.sh | 225 +++++++++-
8 files changed, 1421 insertions(+), 97 deletions(-)
create mode 100644 tests/fdtfactor_lz4.its
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
diff --git a/README.md b/README.md
index ac571384d421..304611eb763b 100644
--- a/README.md
+++ b/README.md
@@ -203,14 +203,14 @@ $ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
## fdtfactor
-`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
-`.dtb.lzma`, `.dtb.gz` with `-C`) and splits them into a single common
-`base.dtb` and per-board `.dtbo` overlays so that applying a board's
-`.dtbo` to `base.dtb` recreates the original board device tree while
-reducing total storage size.
-
-Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`fdtfactor` is a Python tool built on `pylibfdt` that factors a set of
+compiled device tree blobs — either standalone `.dtb` files or
+`/images/fdt-*` nodes inside a compiled FIT — into a common base `.dtb`
+and per-board `.dtbo` overlays so that applying a board's `.dtbo` to the
+base `.dtb` recreates the original board device tree while reducing
+total storage size.
+
+Input `.dtb` blobs must be compiled with `dtc -@ -L` so that both
`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
and `/__local_fixups__` (used to identify phandle references inside
property values) are present.
@@ -219,30 +219,73 @@ Basic usage:
```
$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+$ fdtfactor -F image.fit -o image.factored.fit
```
-Options:
+Common options:
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ base `.dtb` and final reconstructed device trees. By default, `fdtfactor`
+ preserves `/__local_fixups__` on the base `.dtb` so that when
+ `fdt_overlay_apply()` merges a `.dtbo` overlay, the reconstructed device
+ tree retains full `/__local_fixups__` information (e.g., for an OS kernel
+ that wants to know which property cells contain phandles at runtime). When
+ `-L` was passed to `dtc` solely to help `fdtfactor` locate phandles at
+ build time and `/__local_fixups__` is not needed at runtime, passing `-S`
+ strips it from the base `.dtb` (and any unfactored FIT FDTs) to save space.
+* `-s`, `--stats`: Print size savings summary.
+
+### Standalone DTB Factoring
+
+When `-F` / `--fit` is not passed, `fdtfactor` takes two or more positional
+`.dtb` files (or compressed `.dtb.lz4`, `.dtb.xz`, `.dtb.lzma`, `.dtb.gz`
+files with `-C`) and writes a common base `.dtb` plus one `.dtbo` overlay per
+input file:
+
+* `-o`, `--output`: Output directory for generated `.dtbo` files and
+ `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo`
+ files, current directory for `--output-base-dtb`).
* `--output-base-dtb`: Output filename for the common base `.dtb`
(default: `base.dtb`, or `base.dtb.<ext>` when `-C` / `--compression` is
- used; placed in `--output` if specified, or the current directory
+ used; placed in `-o` / `--output` if specified, or the current directory
otherwise).
* `-C`, `--compression`: Compression format for both input `.dtb` files and
generated `base.dtb` / `.dtbo` outputs: `none`, `lz4`, `lz4_legacy`, `lzma`,
`xz`, or `gzip` (default: `none`).
-* `-o`, `--output`: Output directory for generated `.dtbo` files and
- `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
- current directory for `--output-base-dtb`).
-* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
- `base.dtb` and final reconstructed device trees. By default, `fdtfactor`
- preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
- merges a `.dtbo` overlay, the reconstructed device tree retains full
- `/__local_fixups__` information (e.g., for an OS kernel that wants to
- know which property cells contain phandles at runtime). When `-L` was
- passed to `dtc` solely to help `fdtfactor` locate phandles at build time
- and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
- from `base.dtb` to save space.
-* `-s`, `--stats`: Print size savings summary.
+
+### FIT Factoring
+
+When `-F` / `--fit` is passed, `fdtfactor` updates a compiled Flattened Image
+Tree:
+
+* `-F`, `--fit`: Input compiled FIT (`.fit` / `.itb`) to factor.
+* `-o`, `--output`: Output FIT path (default: overwrite `--fit` in place).
+* `--min-overlap`: Minimum percentage (`0`–`100`, default: `50`) of normalized
+ property bytes shared with a group's common properties required to place a
+ `flat_dt` node into that group.
+* `--single-base-dtb`: Force a single `/images/fdt-base-1` across all FDTs in
+ the FIT regardless of property overlap.
+
+FIT behavior and constraints:
+
+* Non-overlay `type = "flat_dt"` subnodes under `/images` are decompressed
+ according to their FIT `compression` property (`none`, `lz4`, `lzma`, `xz`,
+ `gzip`) and partitioned into factor groups that share compatible FDT headers,
+ `arch`, and `load` properties and at least `--min-overlap`% of normalized
+ property bytes (or forced into a single group when `--single-base-dtb` is
+ passed).
+* For each group, `fdtfactor` inserts `/images/fdt-base-N`, converts each
+ member `/images/fdt-*` node into a `.dtbo` overlay (preserving each node's
+ `compression` setting and stripping any `load` address), prepends
+ `fdt-base-N` to matching `/configurations/conf-*` `fdt` string lists
+ (populating `compatible` from the board DTB's root node when omitted), and
+ recomputes `/images/*/hash*` digests.
+* Note that while bootloaders such as depthcharge decompress compressed FIT
+ overlays, U-Boot's `boot_get_fdt_fit()` currently only decompresses the base
+ DTB and requires FIT overlays to be uncompressed.
+* Signed FITs containing `signature*` subnodes are rejected because modifying
+ FDT blobs and configurations would invalidate cryptographic signatures; sign
+ the FIT after factoring.
### TODO
diff --git a/fdtfactor b/fdtfactor
index beeeff231ba6..ebf40cdeaf92 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -8,16 +8,25 @@ properties shared across the inputs into a single common `base.dtb` and emits a
small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
original board DTB.
+
+Alternatively, `fdtfactor -F <image.fit>` can rewrite a compiled U-Boot /
+ChromeOS Flattened Image Tree (FIT) in place (or to `-o`/`--output`), factoring
+its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
+...) and overlays while updating `/configurations/conf-*` `fdt` and
+`compatible` properties and recalculating `/images/*/hash-*` digests.
"""
import argparse
from collections import Counter
+import copy
import gzip
+import hashlib
import lzma
import os
import struct
import subprocess
import sys
+import zlib
import libfdt
import libpydt
@@ -148,6 +157,24 @@ def _is_dtbo_overlay(root):
)
+def _open_fdt(raw_bytes, filename):
+ """Validate `raw_bytes` with `fdt_check_full()` and return an `FdtRo`.
+
+ Args:
+ raw_bytes: Uncompressed FDT blob `bytes`.
+ filename: File path or node label for error reporting.
+
+ Returns:
+ Validated `libfdt.FdtRo` instance.
+ """
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+ return fdt
+
+
class FactoredTree(libpydt.DeviceTree):
"""DeviceTree subclass with normalized phandle and fixup attributes.
@@ -186,30 +213,41 @@ class ParsedDtb:
independent of numeric phandle numbering.
"""
- def __init__(self, filename, comp='none'):
- """Parse an input FDT file into a normalized FactoredTree.
+ def __init__(self, filename, comp='none', fdt=None, root=None,
+ uncompressed_size=None, compressed_size=None):
+ """Parse an input FDT file or opened `FdtRo` into a `FactoredTree`.
Args:
- filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
+ filename: File path (or display name when `fdt` is provided).
comp: Compression format string (default: `'none'`).
+ fdt: Optional already-validated `libfdt.FdtRo` instance; if None,
+ read and decompressed from `filename`.
+ root: Optional already-unflattened `FactoredTree` for `fdt`.
+ uncompressed_size: Uncompressed byte length when `fdt` is provided.
+ compressed_size: Raw/compressed byte length when `fdt` is provided.
"""
self.filename = filename
self.compression = comp
- with open(filename, 'rb') as f:
- file_bytes = f.read()
- self.compressed_size = len(file_bytes)
- raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
- self.uncompressed_size = len(raw_bytes)
-
- try:
- fdt = libfdt.FdtRo(raw_bytes)
- libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
- except libfdt.FdtException as exc:
- raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+ if fdt is None:
+ with open(filename, 'rb') as f:
+ file_bytes = f.read()
+ self.compressed_size = len(file_bytes)
+ raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
+ self.uncompressed_size = len(raw_bytes)
+ fdt = _open_fdt(raw_bytes, filename)
+ else:
+ self.uncompressed_size = (
+ fdt.totalsize() if uncompressed_size is None
+ else uncompressed_size
+ )
+ self.compressed_size = (
+ self.uncompressed_size if compressed_size is None
+ else compressed_size
+ )
self.boot_cpuid_phys = fdt.boot_cpuid_phys()
self.mem_rsv = tuple(fdt.get_mem_rsv_list())
- self.root = FactoredTree.unflatten(fdt)
+ self.root = FactoredTree.unflatten(fdt) if root is None else root
if _is_dtbo_overlay(self.root):
raise ValueError(
@@ -379,16 +417,12 @@ class ParsedDtb:
target_node.fixups[prop_name] = tuple(targets)
-def select_best_candidate(candidates, path, prop_name):
- """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
-
- Only candidates shared by at least two input DTBs (`count > 1`) are
- eligible for the base DTB; a value that appears in only one DTB would add
- one copy to `base.dtb` to save one overlay copy (zero net savings).
+def _prop_byte_cost(path, prop_name, prop_bytes, fixups):
+ """Estimate the FDT structure + `/__fixups__` byte cost of a property.
- Byte cost model per overlay occurrence:
+ Byte cost model per occurrence:
- Each FDT property carries a 12-byte structure header (`FDT_PROP` 4-byte
- tag + 4-byte length + 4-byte `nameoff`) plus `len(masked_bytes)` value
+ tag + 4-byte length + 4-byte `nameoff`) plus `len(prop_bytes)` value
bytes.
- Each phandle fixup in an overlay adds a string entry of the form
`"/fragment@0/__overlay__<path>:<prop_name>:<off>\\0"` to `/__fixups__`:
@@ -397,6 +431,26 @@ def select_best_candidate(candidates, path, prop_name):
~3 bytes amortized `/__fixups__` property overhead =
`len(path) + len(prop_name) + 28` bytes.
+ Args:
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+ prop_bytes: Property value `bytes`.
+ fixups: Sequence of `(offset, target_path)` phandle fixups.
+
+ Returns:
+ Estimated byte cost (int).
+ """
+ fixup_cost = len(fixups) * (len(path) + len(prop_name) + 28)
+ return len(prop_bytes) + fixup_cost + 12
+
+
+def select_best_candidate(candidates, path, prop_name):
+ """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
+
+ Only candidates shared by at least two input DTBs (`count > 1`) are
+ eligible for the base DTB; a value that appears in only one DTB would add
+ one copy to `base.dtb` to save one overlay copy (zero net savings).
+
Args:
candidates: Sequence of `(masked_bytes, fixups)` tuples observed across
input DTBs for `prop_name` on `path`.
@@ -408,7 +462,6 @@ def select_best_candidate(candidates, path, prop_name):
byte savings when placed in `base.dtb`, or `None` if no candidate
appears in at least two input DTBs.
"""
- fixup_entry_cost = len(path) + len(prop_name) + 28
counts = Counter(candidates)
best_val = None
best_key = None
@@ -416,10 +469,10 @@ def select_best_candidate(candidates, path, prop_name):
if count <= 1:
continue
masked_bytes, fixups = val
- length = len(masked_bytes) + len(fixups) * fixup_entry_cost
+ cost = _prop_byte_cost(path, prop_name, masked_bytes, fixups)
# Placing `val` in base.dtb adds 1 copy and removes `count` overlay
- # copies, saving `(count - 1) * (length + 12)` bytes across the set.
- key = ((count - 1) * (length + 12), count, length)
+ # copies, saving `(count - 1) * cost` bytes across the set.
+ key = ((count - 1) * cost, count, cost)
if best_key is None or key > best_key:
best_key = key
best_val = val
@@ -824,6 +877,579 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
+# ---------------------------------------------------------------------------
+# Compiled FIT factoring (targeting depthcharge / coreboot / U-Boot)
+# ---------------------------------------------------------------------------
+
+FIT_COMPRESSIONS = ('none', 'lz4', 'lzma', 'xz', 'gzip')
+LZ4_LEGACY_MAGIC = b'\x02\x21\x4c\x18'
+
+FIT_HASH_ALGOS = {
+ 'crc32': lambda data: struct.pack('>I', zlib.crc32(data) & 0xffffffff),
+ 'md5': lambda data: hashlib.md5(data).digest(),
+ 'sha1': lambda data: hashlib.sha1(data).digest(),
+ 'sha256': lambda data: hashlib.sha256(data).digest(),
+ 'sha384': lambda data: hashlib.sha384(data).digest(),
+ 'sha512': lambda data: hashlib.sha512(data).digest(),
+}
+
+
+def _compute_fit_hash(algo_name, data_bytes):
+ """Compute a binary hash digest for a FIT `hash*` subnode.
+
+ Args:
+ algo_name: FIT hash algorithm name (`'crc32'`, `'md5'`, `'sha1'`,
+ `'sha256'`, `'sha384'`, or `'sha512'`).
+ data_bytes: Input `bytes` to hash.
+
+ Returns:
+ Binary hash digest `bytes`.
+ """
+ hash_fn = FIT_HASH_ALGOS.get(algo_name)
+ if hash_fn is None:
+ raise ValueError(f'Unsupported FIT hash algorithm: {algo_name!r}')
+ return hash_fn(data_bytes)
+
+
+def _update_fit_hashes(img_node):
+ """Recompute `value` properties on any FIT `hash*` subnodes of `img_node`.
+
+ Args:
+ img_node: `DeviceTree` for a `/images/*` subnode whose `hash*` child
+ nodes should be updated in place from `img_node.props['data']`.
+ """
+ data_prop = img_node.props.get('data')
+ if data_prop is None:
+ return
+ for sub_name, sub_node in img_node.children.items():
+ if not sub_name.startswith('hash'):
+ continue
+ if 'algo' not in sub_node.props:
+ raise ValueError(
+ f'FIT hash subnode {sub_name!r} lacks an algo property'
+ )
+ algo = sub_node.props['algo'].as_str()
+ sub_node.set_prop(
+ libfdt.Property('value', _compute_fit_hash(algo, data_prop))
+ )
+
+
+def _rewrite_fit_dtbo_desc(desc, comp):
+ """Rewrite a `.dtb` filename description to `.dtbo`, preserving suffix.
+
+ Args:
+ desc: Description string from a FIT `/images/fdt-*` node.
+ comp: Compression format string of the FIT `/images/fdt-*` node.
+
+ Returns:
+ Rewritten `.dtbo` (or `.dtbo<ext>`) description string if `desc` has no
+ spaces and ends with `.dtb` (or `.dtb<ext>`), or `None` to leave
+ free-form descriptions unchanged.
+ """
+ if not desc or ' ' in desc:
+ return None
+ if desc.endswith('.dtb'):
+ return f'{desc[:-4]}.dtbo'
+ ext = COMP_TO_EXT[comp]
+ if ext and desc.endswith(f'.dtb{ext}'):
+ return f'{desc[:-len(ext) - 4]}.dtbo{ext}'
+ return None
+
+
+def _intern_dtb_props(dtb, prop_ids, id_costs):
+ """Intern each normalized `(path, name, prop, fixups)` in `dtb` to an int ID.
+
+ Args:
+ dtb: `ParsedDtb` instance.
+ prop_ids: Mutable dict mapping `(path, prop_name, prop, fixups)` tuples
+ to sequential integer IDs across DTBs.
+ id_costs: Mutable list mapping each integer ID to `_prop_byte_cost()`.
+
+ Returns:
+ Tuple of `(entry_ids, entry_bytes)` where `entry_ids` is the `set` of
+ integer property IDs present in `dtb` and `entry_bytes` is their total
+ `_prop_byte_cost()`.
+ """
+ entry_ids = set()
+ entry_bytes = 0
+ for path, node in dtb.nodes.items():
+ fixups_map = node.fixups
+ for name, prop in node.props.items():
+ fixups = fixups_map.get(name, ()) if fixups_map else ()
+ key = (path, name, prop, fixups)
+ pid = prop_ids.get(key)
+ if pid is None:
+ pid = len(id_costs)
+ prop_ids[key] = pid
+ id_costs.append(_prop_byte_cost(path, name, prop, fixups))
+ entry_ids.add(pid)
+ entry_bytes += id_costs[pid]
+ return entry_ids, entry_bytes
+
+
+def group_fit_dtbs(fdt_entries, min_overlap=50.0):
+ """Partition FIT `flat_dt` entries into compatible factor groups.
+
+ Entries sharing a `fdt-base-N` node must match on `(arch, load,
+ boot_cpuid_phys, mem_rsv)` (`load` is the optional physical RAM address
+ where bootloaders such as U-Boot copy the base FDT before applying
+ overlays, so two FDTs requiring different RAM load addresses cannot share a
+ single `fdt-base-N` node) and share at least `min_overlap`% of normalized
+ property bytes (`shared_bytes / union_bytes * 100`) against the group's
+ common property set. When multiple groups qualify, the group with the
+ highest overlap percentage is chosen.
+
+ Args:
+ fdt_entries: Sequence of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+ min_overlap: Minimum byte-weighted Jaccard overlap percentage
+ (`0.0` to `100.0`, default `50.0`).
+
+ Returns:
+ List of entry-dict lists, one per factor group.
+ """
+ prop_ids = {}
+ id_costs = []
+ groups = []
+
+ for entry in fdt_entries:
+ dtb = entry['dtb']
+ header_key = (
+ entry['node'].props.get('arch'),
+ entry['node'].props.get('load'),
+ dtb.boot_cpuid_phys,
+ dtb.mem_rsv,
+ )
+ entry_ids, entry_bytes = _intern_dtb_props(dtb, prop_ids, id_costs)
+
+ best_grp = None
+ best_overlap = -1.0
+ best_shared_ids = None
+ best_shared_bytes = 0
+
+ for grp in groups:
+ if grp['key'] != header_key:
+ continue
+ max_shared = min(entry_bytes, grp['common_bytes'])
+ max_union = entry_bytes + grp['common_bytes'] - max_shared
+ if max_union and (max_shared / max_union * 100.0) < min_overlap:
+ continue
+ shared_ids = grp['common_ids'] & entry_ids
+ shared_bytes = sum(id_costs[pid] for pid in shared_ids)
+ union_bytes = entry_bytes + grp['common_bytes'] - shared_bytes
+ overlap = (
+ shared_bytes / union_bytes * 100.0 if union_bytes else 100.0
+ )
+ if overlap >= min_overlap and overlap > best_overlap:
+ best_overlap = overlap
+ best_grp = grp
+ best_shared_ids = shared_ids
+ best_shared_bytes = shared_bytes
+
+ if best_grp is not None:
+ best_grp['common_ids'] = best_shared_ids
+ best_grp['common_bytes'] = best_shared_bytes
+ best_grp['entries'].append(entry)
+ else:
+ groups.append({
+ 'key': header_key,
+ 'common_ids': entry_ids,
+ 'common_bytes': entry_bytes,
+ 'entries': [entry],
+ })
+
+ return [grp['entries'] for grp in groups]
+
+
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+ """Extract and parse non-overlay `flat_dt` entries from `/images`.
+
+ Args:
+ fit_in_path: Input FIT path (for error reporting).
+ images_node: `DeviceTree` for the FIT `/images` node.
+ configs_node: `DeviceTree` for the FIT `/configurations` node.
+
+ Returns:
+ List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
+ `flat_dt` image subnodes.
+ """
+ for conf_name, conf_node in configs_node.children.items():
+ for sub_name in conf_node.children:
+ if sub_name.startswith(('signature', 'hash')):
+ raise ValueError(
+ f'{fit_in_path}: /configurations/{conf_name}/{sub_name} '
+ f'is not supported; sign the FIT after factoring'
+ )
+
+ fdt_entries = []
+ for img_name, img_node in images_node.children.items():
+ if ('data-offset' in img_node.props or
+ 'data-position' in img_node.props):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} uses external data '
+ f'(data-offset/data-position), which is not supported'
+ )
+ for sub_name in img_node.children:
+ if sub_name.startswith('signature'):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name}/{sub_name} is not '
+ f'supported; sign the FIT after factoring'
+ )
+ type_prop = img_node.props.get('type')
+ if type_prop is None or type_prop.as_str() != 'flat_dt':
+ continue
+ raw_data = img_node.props.get('data')
+ if raw_data is None:
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} lacks a data property'
+ )
+ node_label = f'{fit_in_path}:/images/{img_name}'
+ comp_prop = img_node.props.get('compression')
+ comp = (comp_prop.as_str() if comp_prop is not None else '') or 'none'
+ if comp not in FIT_COMPRESSIONS:
+ raise ValueError(
+ f'{node_label}: Unsupported FIT compression type: {comp}'
+ )
+ if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
+ comp = 'lz4_legacy'
+ raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+ fdt = _open_fdt(raw_bytes, node_label)
+ root = FactoredTree.unflatten(fdt)
+ if _is_dtbo_overlay(root):
+ continue
+
+ parsed_dtb = ParsedDtb(
+ node_label,
+ comp=comp,
+ fdt=fdt,
+ root=root,
+ uncompressed_size=len(raw_bytes),
+ compressed_size=len(raw_data),
+ )
+ fdt_entries.append({
+ 'name': img_name,
+ 'node': img_node,
+ 'dtb': parsed_dtb,
+ 'fdt': fdt,
+ })
+
+ if len(fdt_entries) < 2:
+ raise ValueError(
+ 'FIT must contain at least two non-overlay flat_dt '
+ 'nodes to factor'
+ )
+
+ return fdt_entries
+
+
+def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
+ """Partition `fdt_entries` into factorable groups and unfactored singletons.
+
+ Args:
+ fdt_entries: List of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+ single_base_dtb: If True, require all `fdt_entries` to share a single
+ base DTB group regardless of property overlap.
+ min_overlap: Minimum normalized property byte overlap percentage (`0.0`
+ to `100.0`) used when `single_base_dtb` is False.
+
+ Returns:
+ Tuple of `(factorable_groups, unfactored_entries)` where
+ `factorable_groups` is a list of entry-dict lists (each containing at
+ least 2 entries) and `unfactored_entries` is a list of single-board
+ entry dicts left unfactored.
+ """
+ if single_base_dtb:
+ ref_entry = fdt_entries[0]
+ for entry in fdt_entries[1:]:
+ for prop in ('arch', 'load'):
+ if (entry['node'].props.get(prop) !=
+ ref_entry['node'].props.get(prop)):
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to '
+ f'use the same {prop} property: {ref_entry["name"]} vs '
+ f'{entry["name"]}'
+ )
+ for entry in fdt_entries:
+ entry.pop('fdt', None)
+ return [fdt_entries], []
+
+ groups = group_fit_dtbs(fdt_entries, min_overlap=min_overlap)
+ factorable_groups = [g for g in groups if len(g) >= 2]
+ if not factorable_groups:
+ raise ValueError(
+ 'FIT must contain at least two compatible '
+ 'flat_dt nodes to factor'
+ )
+ for g in factorable_groups:
+ for entry in g:
+ entry.pop('fdt', None)
+ unfactored_entries = [g[0] for g in groups if len(g) == 1]
+ for entry in unfactored_entries:
+ print(
+ f'Note: FIT flat_dt node {entry["name"]} has no '
+ f'compatible peers and will be left unfactored',
+ file=sys.stderr,
+ )
+ return factorable_groups, unfactored_entries
+
+
+def _strip_fit_unfactored_local_fixups(unfactored_entries):
+ """Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
+
+ Args:
+ unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
+ unfactored by `_partition_fit_fdt_entries()`.
+ """
+ for entry in unfactored_entries:
+ fdt = entry.pop('fdt')
+ tree = libpydt.DeviceTree.unflatten(fdt)
+ if tree.children.pop('__local_fixups__', None) is None:
+ continue
+ stripped_fdt = tree.flatten(
+ mem_rsv=entry['dtb'].mem_rsv,
+ boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
+ )
+ comp_bytes = compress_blob(
+ bytes(stripped_fdt.as_bytearray()),
+ entry['dtb'].compression,
+ filename=f'/images/{entry["name"]}',
+ )
+ img_node = entry['node']
+ img_node.set_prop(libfdt.Property('data', comp_bytes))
+ _update_fit_hashes(img_node)
+
+
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+ """Factor one group of FIT `flat_dt` entries into a base node and overlays.
+
+ Args:
+ entries: List of FIT `flat_dt` entry dicts in this group.
+ base_dtb_node_name: Name for the new `/images/<base_dtb_node_name>`
+ node (e.g., `'fdt-base-1'`).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated base DTB.
+
+ Returns:
+ Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
+ constructed `DeviceTree` for `/images/<base_dtb_node_name>` and
+ `stat_args` is an argument tuple for `_print_dtb_stats()`.
+ """
+ group_dtbs = [e['dtb'] for e in entries]
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ group_dtbs, strip_local_fixups=strip_local_fixups
+ )
+ base_comp = group_dtbs[0].compression
+
+ base_dtb_comp_bytes = compress_blob(
+ base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+ )
+ base_dtb_node = copy.deepcopy(entries[0]['node'])
+ for prop in ('phandle', 'compatible'):
+ base_dtb_node.props.pop(prop, None)
+ base_dtb_node.children = {
+ k: v
+ for k, v in base_dtb_node.children.items()
+ if k.startswith('hash')
+ }
+ for sub_node in base_dtb_node.children.values():
+ sub_node.props.pop('phandle', None)
+ if 'description' in base_dtb_node.props:
+ first_desc = base_dtb_node.props['description'].as_str()
+ ext = COMP_TO_EXT[base_comp]
+ base_suffix = ext if (ext and first_desc.endswith(ext)) else ''
+ base_dtb_node.set_prop(
+ libfdt.Property.from_str(
+ 'description', f'{base_dtb_node_name}.dtb{base_suffix}'
+ )
+ )
+ base_dtb_node.set_prop(libfdt.Property('data', base_dtb_comp_bytes))
+ if 'compression' in base_dtb_node.props or base_comp != 'none':
+ comp_prop = 'lz4' if base_comp == 'lz4_legacy' else base_comp
+ base_dtb_node.set_prop(
+ libfdt.Property.from_str('compression', comp_prop)
+ )
+ _update_fit_hashes(base_dtb_node)
+
+ dtbo_comp_sizes = []
+ for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ img_node = entry['node']
+ entry_comp = entry['dtb'].compression
+ dtbo_comp_bytes = compress_blob(
+ dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
+ )
+ dtbo_comp_sizes.append(len(dtbo_comp_bytes))
+
+ if 'description' in img_node.props:
+ new_desc = _rewrite_fit_dtbo_desc(
+ img_node.props['description'].as_str(), entry_comp
+ )
+ if new_desc is not None:
+ img_node.set_prop(
+ libfdt.Property.from_str('description', new_desc)
+ )
+
+ # Strip 'load' from overlay image nodes so bootloaders do not copy an
+ # overlay onto the base FDT's RAM load address.
+ img_node.props.pop('load', None)
+
+ img_node.set_prop(libfdt.Property('data', dtbo_comp_bytes))
+ _update_fit_hashes(img_node)
+
+ stat_args = (
+ group_dtbs,
+ base_dtb_blob,
+ dtbo_blobs,
+ base_comp,
+ len(base_dtb_comp_bytes),
+ dtbo_comp_sizes,
+ f'[{base_dtb_node_name}] ',
+ )
+ return base_dtb_node, stat_args
+
+
+def _update_fit_images_and_configs(images_node, configs_node,
+ insert_before_first, fdt_to_base_dtb):
+ """Insert `fdt-base-*` nodes into `/images` and update `/configurations`.
+
+ Args:
+ images_node: `DeviceTree` for `/images` (mutated in place).
+ configs_node: `DeviceTree` for `/configurations` (mutated in place).
+ insert_before_first: Dict mapping the first `fdt-*` subnode name of
+ each group to `(base_dtb_node_name, base_dtb_node)`.
+ fdt_to_base_dtb: Dict mapping each factored `fdt-*` subnode name to
+ `(base_dtb_node_name, orig_parsed_dtb)`.
+ """
+ new_images_children = {}
+ for img_name, img_node in images_node.children.items():
+ if img_name in insert_before_first:
+ b_name, b_node = insert_before_first[img_name]
+ new_images_children[b_name] = b_node
+ new_images_children[img_name] = img_node
+ images_node.children = new_images_children
+
+ for conf_name, conf_node in configs_node.children.items():
+ if 'fdt' not in conf_node.props:
+ continue
+ fdt_list = conf_node.props['fdt'].as_stringlist()
+ if not fdt_list:
+ continue
+ primary_fdt = fdt_list[0]
+ for extra_fdt in fdt_list[1:]:
+ if extra_fdt in fdt_to_base_dtb:
+ raise ValueError(
+ f'/configurations/{conf_name} lists factored flat_dt '
+ f'{extra_fdt!r} after the primary FDT'
+ )
+ if primary_fdt not in fdt_to_base_dtb:
+ continue
+ base_dtb_name, orig_dtb = fdt_to_base_dtb[primary_fdt]
+ conf_node.set_prop(
+ libfdt.Property.from_stringlist('fdt', [base_dtb_name] + fdt_list)
+ )
+ # If the configuration omitted 'compatible' (relying on the
+ # bootloader reading root 'compatible' from fdt_list[0]), copy the
+ # board's root 'compatible' property onto the configuration node
+ # now that fdt_list[0] is the shared base DTB.
+ if 'compatible' in conf_node.props:
+ continue
+ root_compat = orig_dtb.root.props.get('compatible')
+ if root_compat is not None:
+ conf_node.set_prop(root_compat)
+
+
+def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
+ single_base_dtb=False, min_overlap=50.0, stats=False):
+ """Factor all `flat_dt` images in a compiled FIT into base + overlays.
+
+ Args:
+ fit_in_path: Path to the input compiled FIT.
+ fit_out_path: Path where the factored FIT should be written.
+ strip_local_fixups: If True, omit `/__local_fixups__` from generated
+ base DTBs.
+ single_base_dtb: If True, force all `flat_dt` nodes into a single
+ `fdt-base-1` base DTB.
+ min_overlap: Minimum normalized property byte overlap percentage
+ (`0.0` to `100.0`, default `50.0`) when `single_base_dtb` is False.
+ stats: If True, print per-group and overall FIT size savings.
+ """
+ with open(fit_in_path, 'rb') as f:
+ orig_fit_bytes = f.read()
+
+ orig_fit_fdt = _open_fdt(orig_fit_bytes, fit_in_path)
+ fit_root = libpydt.DeviceTree.unflatten(orig_fit_fdt)
+ images_node = fit_root.children.get('images')
+ configs_node = fit_root.children.get('configurations')
+ if images_node is None or configs_node is None:
+ raise ValueError(
+ f'{fit_in_path}: FIT must contain /images and '
+ f'/configurations nodes'
+ )
+
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node
+ )
+ factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb, min_overlap=min_overlap
+ )
+ if strip_local_fixups and unfactored_entries:
+ _strip_fit_unfactored_local_fixups(unfactored_entries)
+ else:
+ for entry in unfactored_entries:
+ entry.pop('fdt', None)
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+ used_image_names = set(images_node.children)
+ next_base_idx = 1
+
+ for entries in factorable_groups:
+ while f'fdt-base-{next_base_idx}' in used_image_names:
+ next_base_idx += 1
+ base_dtb_node_name = f'fdt-base-{next_base_idx}'
+ used_image_names.add(base_dtb_node_name)
+ next_base_idx += 1
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
+ group_stats.append(stat_args)
+
+ _update_fit_images_and_configs(
+ images_node, configs_node, insert_before_first, fdt_to_base_dtb
+ )
+
+ new_fit_fdt = fit_root.flatten(
+ mem_rsv=orig_fit_fdt.get_mem_rsv_list(),
+ boot_cpuid_phys=orig_fit_fdt.boot_cpuid_phys(),
+ )
+ new_fit_bytes = new_fit_fdt.as_bytearray()
+
+ out_dir = os.path.dirname(fit_out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(fit_out_path, 'wb') as f:
+ f.write(new_fit_bytes)
+
+ if stats:
+ for stat_args in group_stats:
+ _print_dtb_stats(*stat_args)
+ fit_saved = len(orig_fit_bytes) - len(new_fit_bytes)
+ fit_pct = (
+ (fit_saved / len(orig_fit_bytes) * 100.0) if orig_fit_bytes else 0.0
+ )
+ print(
+ f'FIT total: {len(orig_fit_bytes):,} B -> '
+ f'{len(new_fit_bytes):,} B '
+ f'(saved {fit_saved:,} B / {fit_pct:.1f}%)'
+ )
+
+
def get_dtbo_output_path(dtb_path, outdir, comp='none'):
"""Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
@@ -857,23 +1483,69 @@ def parse_args(argv=None):
Parsed `argparse.Namespace` instance.
"""
parser = argparse.ArgumentParser(
+ prog='fdtfactor',
+ usage=(
+ '%(prog)s [-h] [-o DIR] [-S] [-s] [--output-base-dtb FILENAME]\n'
+ ' [-C {none,lz4,lz4_legacy,lzma,xz,gzip}] '
+ 'DTB DTB [DTB ...]\n'
+ ' %(prog)s -F FIT_FILE [-o FIT_OUT] [-S] [-s] '
+ '[--min-overlap PERCENT]\n'
+ ' [--single-base-dtb]'
+ ),
description=(
'Split a set of compiled device tree blobs (.dtb, or compressed '
- '.dtb.lz4/.xz/.lzma/.gz with -C) into a single common base .dtb '
- 'and per-board overlay blobs (.dtbo).'
- )
+ '.dtb.lz4/.xz/.lzma/.gz with -C) into a common base .dtb and '
+ 'per-board overlay blobs (.dtbo), or update a compiled FIT '
+ '(-F/--fit) in place (or to -o/--output) by factoring its '
+ '/images/fdt-* blobs into base DTBs and overlays.'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--output',
+ default=None,
+ metavar='PATH',
+ help=(
+ 'Output directory in standalone DTB mode (default: alongside each '
+ 'input .dtb for .dtbo, current directory for --output-base-dtb), '
+ 'or output FIT path in FIT mode (default: overwrite --fit in '
+ 'place)'
+ ),
)
parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from generated base DTB(s) and '
+ 'reconstructed device trees'
+ ),
+ )
+ parser.add_argument(
+ '-s', '--stats',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+
+ dtb_group = parser.add_argument_group('standalone DTB mode')
+ dtb_group.add_argument(
+ 'dtbs',
+ nargs='*',
+ metavar='DTB',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
+ '(at least two required)'
+ ),
+ )
+ dtb_group.add_argument(
'--output-base-dtb',
default=None,
metavar='FILENAME',
help=(
'Output common base .dtb filename (default: base.dtb, or '
- 'base.dtb.<ext> when -C/--compression is used; placed in --output '
- 'if specified, or current directory otherwise)'
+ 'base.dtb.<ext> when -C/--compression is used; placed in '
+ '-o/--output if specified, or current directory otherwise)'
),
)
- parser.add_argument(
+ dtb_group.add_argument(
'-C', '--compression',
choices=tuple(COMP_TO_EXT),
default='none',
@@ -882,62 +1554,82 @@ def parse_args(argv=None):
'files (default: none)'
),
)
- parser.add_argument(
- '-o', '--output',
+
+ fit_group = parser.add_argument_group('FIT mode')
+ fit_group.add_argument(
+ '-F', '--fit',
default=None,
- metavar='DIR',
+ metavar='FIT_FILE',
help=(
- 'Output directory for generated .dtbo files and '
- '--output-base-dtb (default: alongside each input .dtb for .dtbo, '
- 'current directory for --output-base-dtb)'
+ 'Input compiled FIT (.fit / .itb) to factor (updated in place '
+ 'unless -o/--output is specified)'
),
)
- parser.add_argument(
- '-S', '--strip-local-fixups',
- action='store_true',
+ fit_group.add_argument(
+ '--min-overlap',
+ type=float,
+ default=None,
+ metavar='PERCENT',
help=(
- 'Omit /__local_fixups__ from the generated base .dtb and '
- 'reconstructed device trees'
+ 'Minimum shared normalized property byte overlap percentage '
+ '(0-100) to group FIT flat_dt nodes into the same base DTB '
+ '(default: 50)'
),
)
- parser.add_argument(
- '-s', '--stats',
+ fit_group.add_argument(
+ '--single-base-dtb',
action='store_true',
- help='Print size statistics summary',
- )
- parser.add_argument(
- 'dtbs',
- nargs='+',
- metavar='DTB',
help=(
- 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
- '(at least two required)'
+ 'Force all flat_dt nodes in the FIT into a single fdt-base-1 base '
+ 'DTB, failing only if FDT headers, arch, or load differ'
),
)
args = parser.parse_args(argv)
- if args.output_base_dtb is None:
- args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
- if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ if args.single_base_dtb and not args.fit:
+ parser.error('--single-base-dtb requires -F/--fit')
+ if args.min_overlap is not None and not args.fit:
+ parser.error('--min-overlap requires -F/--fit')
+ if args.min_overlap is not None and args.single_base_dtb:
parser.error(
- '--output-base-dtb must be a non-empty filename without directory '
- 'components; use -o/--output to specify the output directory'
+ '--min-overlap cannot be combined with --single-base-dtb'
)
- if len(args.dtbs) < 2:
- parser.error('at least two input .dtb files are required')
+ if args.min_overlap is not None and not (0.0 <= args.min_overlap <= 100.0):
+ parser.error('--min-overlap must be between 0 and 100')
+ if args.min_overlap is None:
+ args.min_overlap = 50.0
+ if args.fit:
+ if (args.dtbs or args.output_base_dtb is not None or
+ args.compression != 'none'):
+ parser.error(
+ '-F/--fit cannot be combined with --output-base-dtb, '
+ '-C/--compression, or positional DTB arguments'
+ )
+ else:
+ if args.output_base_dtb is None:
+ args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
+ if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a non-empty filename without '
+ 'directory components; use -o/--output to specify the output '
+ 'directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
return args
-def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
- dtbo_csizes):
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp,
+ base_dtb_csize, dtbo_csizes, prefix=''):
"""Print uncompressed and compressed size savings summary.
Args:
dtbs: Sequence of input `ParsedDtb` instances.
base_dtb_blob: Uncompressed base DTB `bytes`.
dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
- comp: Compression format of the inputs and outputs.
+ base_dtb_comp: Compression format of the base DTB output.
base_dtb_csize: Byte size of the compressed base DTB output.
dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
+ prefix: Optional line prefix (e.g., `'[fdt-base-1] '`).
"""
orig_uncomp = sum(d.uncompressed_size for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -946,22 +1638,25 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
saved_uncomp = orig_uncomp - new_uncomp
pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
print(
- f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'{prefix}Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
- if comp != 'none':
+ comps = sorted({base_dtb_comp, *(d.compression for d in dtbs)})
+ if any(c != 'none' for c in comps):
+ comp_label = '/'.join(comps)
orig_comp = sum(d.compressed_size for d in dtbs)
dtbo_csize = sum(dtbo_csizes)
new_comp = base_dtb_csize + dtbo_csize
saved_comp = orig_comp - new_comp
pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
print(
- f'Compressed ({comp}): {len(dtbs)} files ({orig_comp:,} B) -> '
- f'base {base_dtb_csize:,} B + {len(dtbo_blobs)} overlays '
- f'{dtbo_csize:,} B (avg {dtbo_csize // len(dtbo_blobs):,} B) = '
- f'{new_comp:,} B (saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ f'{prefix}Compressed ({comp_label}): {len(dtbs)} files '
+ f'({orig_comp:,} B) -> base {base_dtb_csize:,} B + '
+ f'{len(dtbo_blobs)} overlays {dtbo_csize:,} B '
+ f'(avg {dtbo_csize // len(dtbo_blobs):,} B) = {new_comp:,} B '
+ f'(saved {saved_comp:,} B / {pct_comp:.1f}%)'
)
@@ -1032,7 +1727,7 @@ def factor_dtb_files(args):
if args.stats:
_print_dtb_stats(
dtbs, base_dtb_blob, dtbo_blobs,
- comp=comp,
+ base_dtb_comp=comp,
base_dtb_csize=len(base_dtb_file_bytes),
dtbo_csizes=dtbo_file_sizes,
)
@@ -1049,7 +1744,18 @@ def main(argv=None):
"""
args = parse_args(argv)
try:
- factor_dtb_files(args)
+ if args.fit:
+ fit_out = args.output if args.output is not None else args.fit
+ factor_fit(
+ args.fit,
+ fit_out,
+ strip_local_fixups=args.strip_local_fixups,
+ single_base_dtb=args.single_base_dtb,
+ min_overlap=args.min_overlap,
+ stats=args.stats,
+ )
+ else:
+ factor_dtb_files(args)
except (OSError, ValueError, RuntimeError, struct.error,
libfdt.FdtException) as exc:
sys.exit(f'Error: {exc}')
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 1eed9ba6478a..351d4ed199b5 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,7 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 *.test.fit \
*.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
*.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
tmp.* *.bak \
diff --git a/tests/fdtfactor_lz4.its b/tests/fdtfactor_lz4.its
new file mode 100644
index 000000000000..4939223e57da
--- /dev/null
+++ b/tests/fdtfactor_lz4.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test LZ4 FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "lz4";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board3.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb.lz4";
+ data = /incbin/("overlay_overlay_phandle.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/fdtfactor_other1.dts b/tests/fdtfactor_other1.dts
new file mode 100644
index 000000000000..1396815974e5
--- /dev/null
+++ b/tests/fdtfactor_other1.dts
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other1", "vendor,other";
+ model = "other1";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+ other-shared-data = <
+ 0xfefdfcfb 0xfaf9f8f7 0xf6f5f4f3 0xf2f1f0ef
+ 0xeeedeceb 0xeae9e8e7 0xe6e5e4e3 0xe2e1e0df
+ 0xdedddcdb 0xdad9d8d7 0xd6d5d4d3 0xd2d1d0cf
+ 0xcecdcccb 0xcac9c8c7 0xc6c5c4c3 0xc2c1c0bf
+ 0xbebdbcbb 0xbab9b8b7 0xb6b5b4b3 0xb2b1b0af
+ 0xaeadacab 0xaaa9a8a7 0xa6a5a4a3 0xa2a1a09f
+ 0x9e9d9c9b 0x9a999897 0x96959493 0x9291908f
+ 0x8e8d8c8b 0x8a898887 0x86858483 0x8281807f
+ 0x7e7d7c7b 0x7a797877 0x76757473 0x7271706f
+ 0x6e6d6c6b 0x6a696867 0x66656463 0x6261605f
+ 0x5e5d5c5b 0x5a595857 0x56555453 0x5251504f
+ 0x4e4d4c4b 0x4a494847 0x46454443 0x4241403f
+ 0x3e3d3c3b 0x3a393837 0x36353433 0x3231302f
+ 0x2e2d2c2b 0x2a292827 0x26252423 0x2221201f
+ 0x1e1d1c1b 0x1a191817 0x16151413 0x1211100f
+ 0x0e0d0c0b 0x0a090807 0x06050403 0x020100ff
+ 0xfdfbf9f7 0xf5f3f1ef 0xede9e7e5 0xe3e1dfdd
+ 0xdbdad8d6 0xd4d2d0ce 0xcccbc9c7 0xc5c3c1bf
+ 0xbdbbb9b7 0xb5b3b1af 0xadaba9a7 0xa5a3a19f
+ 0x9d9b9997 0x9593918f 0x8d8b8987 0x8583817f
+ 0x7d7b7977 0x7573716f 0x6d6b6967 0x6563615f
+ 0x5d5b5957 0x5553514f 0x4d4b4947 0x4543413f
+ 0x3d3b3937 0x3533312f 0x2d2b2927 0x2523211f
+ 0x1d1b1917 0x1513110f 0x0d0b0907 0x050301fe
+ 0xfcf8f4f0 0xece8e4e0 0xdcd8d4d0 0xccc8c4c0
+ 0xbcb8b4b0 0xaca8a4a0 0x9c989490 0x8c888480
+ 0x7c787470 0x6c686460 0x5c585450 0x4c484440
+ 0x3c383430 0x2c282420 0x1c181410 0x0c080400
+ 0xfbf7f3ef 0xebe7e3df 0xdbd7d3cf 0xcbc7c3bf
+ 0xbbb7b3af 0xaba7a39f 0x9b97938f 0x8b87837f
+ 0x7b77736f 0x6b67635f 0x5b57534f 0x4b47433f
+ 0x3b37332f 0x2b27231f 0x1b17130f 0x0b0703ff
+ >;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x1111>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o1_node: other1-node {
+ prop-str = "other1";
+ ref-base = <&osn1>;
+ ref-self = <&o1_node>;
+ };
+};
diff --git a/tests/fdtfactor_other2.dts b/tests/fdtfactor_other2.dts
new file mode 100644
index 000000000000..c5ad9b53c047
--- /dev/null
+++ b/tests/fdtfactor_other2.dts
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other2", "vendor,other";
+ model = "other2";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+ other-shared-data = <
+ 0xfefdfcfb 0xfaf9f8f7 0xf6f5f4f3 0xf2f1f0ef
+ 0xeeedeceb 0xeae9e8e7 0xe6e5e4e3 0xe2e1e0df
+ 0xdedddcdb 0xdad9d8d7 0xd6d5d4d3 0xd2d1d0cf
+ 0xcecdcccb 0xcac9c8c7 0xc6c5c4c3 0xc2c1c0bf
+ 0xbebdbcbb 0xbab9b8b7 0xb6b5b4b3 0xb2b1b0af
+ 0xaeadacab 0xaaa9a8a7 0xa6a5a4a3 0xa2a1a09f
+ 0x9e9d9c9b 0x9a999897 0x96959493 0x9291908f
+ 0x8e8d8c8b 0x8a898887 0x86858483 0x8281807f
+ 0x7e7d7c7b 0x7a797877 0x76757473 0x7271706f
+ 0x6e6d6c6b 0x6a696867 0x66656463 0x6261605f
+ 0x5e5d5c5b 0x5a595857 0x56555453 0x5251504f
+ 0x4e4d4c4b 0x4a494847 0x46454443 0x4241403f
+ 0x3e3d3c3b 0x3a393837 0x36353433 0x3231302f
+ 0x2e2d2c2b 0x2a292827 0x26252423 0x2221201f
+ 0x1e1d1c1b 0x1a191817 0x16151413 0x1211100f
+ 0x0e0d0c0b 0x0a090807 0x06050403 0x020100ff
+ 0xfdfbf9f7 0xf5f3f1ef 0xede9e7e5 0xe3e1dfdd
+ 0xdbdad8d6 0xd4d2d0ce 0xcccbc9c7 0xc5c3c1bf
+ 0xbdbbb9b7 0xb5b3b1af 0xadaba9a7 0xa5a3a19f
+ 0x9d9b9997 0x9593918f 0x8d8b8987 0x8583817f
+ 0x7d7b7977 0x7573716f 0x6d6b6967 0x6563615f
+ 0x5d5b5957 0x5553514f 0x4d4b4947 0x4543413f
+ 0x3d3b3937 0x3533312f 0x2d2b2927 0x2523211f
+ 0x1d1b1917 0x1513110f 0x0d0b0907 0x050301fe
+ 0xfcf8f4f0 0xece8e4e0 0xdcd8d4d0 0xccc8c4c0
+ 0xbcb8b4b0 0xaca8a4a0 0x9c989490 0x8c888480
+ 0x7c787470 0x6c686460 0x5c585450 0x4c484440
+ 0x3c383430 0x2c282420 0x1c181410 0x0c080400
+ 0xfbf7f3ef 0xebe7e3df 0xdbd7d3cf 0xcbc7c3bf
+ 0xbbb7b3af 0xaba7a39f 0x9b97938f 0x8b87837f
+ 0x7b77736f 0x6b67635f 0x5b57534f 0x4b47433f
+ 0x3b37332f 0x2b27231f 0x1b17130f 0x0b0703ff
+ >;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x2222>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o2_node: other2-node {
+ prop-str = "other2";
+ ref-base = <&ossn1>;
+ ref-self = <&o2_node>;
+ };
+};
diff --git a/tests/fdtfactor_test.its b/tests/fdtfactor_test.its
new file mode 100644
index 000000000000..dd515e17b66f
--- /dev/null
+++ b/tests/fdtfactor_test.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "none";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb";
+ data = /incbin/("fdtfactor_board3.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb";
+ data = /incbin/("overlay_overlay_phandle.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 9b005396bc79..237e399803bb 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1371,6 +1371,229 @@ fdtfactor_tests () {
-o fdtfactor_comp_err.test.dir \
fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
fi
+
+ # FIT factoring (uncompressed and compressed, multi-group by property
+ # byte overlap)
+ # fdtget -t r appends a trailing newline; strip that last byte.
+ fdtget_raw () {
+ $DTGET -t r "$1" "$2" "$3" > "$4.tmp"
+ head -c $(($($STATSZ "$4.tmp") - 1)) "$4.tmp" > "$4"
+ rm -f "$4.tmp"
+ }
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other1.test.dtb \
+ "$SRCDIR/fdtfactor_other1.dts"
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other2.test.dtb \
+ "$SRCDIR/fdtfactor_other2.dts"
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay_phandle.test.dtb \
+ "$SRCDIR/overlay_overlay_phandle.dts"
+
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor.test.fit \
+ "$SRCDIR/fdtfactor_test.its"
+ cp fdtfactor.test.fit fdtfactor_orig.test.fit
+ run_wrap_test $PYTHON $FDTFACTOR -s -F fdtfactor.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-1 description
+ run_fdtget_test "fdt-base-2.dtb" fdtfactor.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor.test.fit \
+ /configurations/conf-2 fdt
+ run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor.test.fit \
+ /configurations/conf-3 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor.test.fit \
+ /configurations/conf-4 fdt
+ run_fdtget_test "fdt-base-2 fdt-5" fdtfactor.test.fit \
+ /configurations/conf-5 fdt
+ # conf-1 already had an explicit compatible property; conf-2 did not, so
+ # fdtfactor should populate it from fdt-2's root compatible property.
+ run_fdtget_test "vendor,board1 vendor,common" fdtfactor.test.fit \
+ /configurations/conf-1 compatible
+ run_fdtget_test "test_fdtfactor" fdtfactor.test.fit \
+ /configurations/conf-2 compatible
+ run_fdtget_test "vendor,other1 vendor,other" fdtfactor.test.fit \
+ /configurations/conf-4 compatible
+ # Verify that hash-1/value was populated for fdt-base-1, fdt-base-2, and
+ # updated for fdt-1.
+ base1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-1/hash-1 value)
+ base2_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-2/hash-1 value)
+ fdt1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-1/hash-1 value)
+ if [ -z "$base1_hash" ] || [ -z "$base2_hash" ] || \
+ [ "$fdt1_hash" = "0 0 0 0 0 0 0 0" ]; then
+ FAIL "FIT hash-1 values were not updated"
+ fi
+ # Extract factored base DTBs and overlays from the FIT and verify all 5
+ # boards with fdtoverlay + dtbs_equal_unordered.
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-1 data fit_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor.test.fit /images/fdt-${i} data \
+ fit_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base1.test.dtb \
+ -o fit_merged${i}.test.dtb fit_fdt${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fit_merged${i}.test.dtb
+ done
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-2 data fit_base2.test.dtb
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor.test.fit /images/fdt-${idx} data \
+ fit_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base2.test.dtb \
+ -o fit_merged${idx}.test.dtb fit_fdt${idx}.test.dtbo
+ check_factored_dtb fdtfactor_other${i}.test.dtb \
+ fit_merged${idx}.test.dtb
+ done
+
+ # --min-overlap controls the shared property byte overlap threshold
+ run_wrap_test $PYTHON $FDTFACTOR --min-overlap 1 \
+ -F fdtfactor_orig.test.fit -o fdtfactor_minovl.test.fit
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_minovl.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-4" fdtfactor_minovl.test.fit \
+ /configurations/conf-4 fdt
+ run_wrap_error_test $PYTHON $FDTFACTOR --min-overlap 99 \
+ -F fdtfactor_orig.test.fit -o fdtfactor_minovl_hi.test.fit
+
+ # Unfactored singleton FIT entries retain /__local_fixups__ by default and
+ # strip it (updating hashes) when -S/--strip-local-fixups is passed.
+ cp fdtfactor_orig.test.fit fdtfactor_singleton_in.test.fit
+ run_wrap_test $DTPUT -r fdtfactor_singleton_in.test.fit \
+ /images/fdt-5 /configurations/conf-5
+ run_wrap_test $PYTHON $FDTFACTOR -F fdtfactor_singleton_in.test.fit \
+ -o fdtfactor_singleton_keep.test.fit
+ run_fdtget_test "fdt-4" fdtfactor_singleton_keep.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_singleton_keep.test.fit /images/fdt-4 data \
+ fit_singleton_keep.test.dtb
+ run_test check_path fit_singleton_keep.test.dtb exists "/__local_fixups__"
+ run_wrap_test $PYTHON $FDTFACTOR -S -F fdtfactor_singleton_in.test.fit \
+ -o fdtfactor_singleton_strip.test.fit
+ fdtget_raw fdtfactor_singleton_strip.test.fit /images/fdt-4 data \
+ fit_singleton_strip.test.dtb
+ run_test check_path fit_singleton_strip.test.dtb \
+ not-exists "/__local_fixups__"
+ run_test check_path fit_singleton_strip.test.dtb exists "/__symbols__"
+ fdt4_hash=$($DTGET -t bx fdtfactor_singleton_strip.test.fit \
+ /images/fdt-4/hash-1 value)
+ if [ -z "$fdt4_hash" ] || [ "$fdt4_hash" = "0 0 0 0 0 0 0 0" ]; then
+ FAIL "Singleton FIT hash-1 value was not updated under -S"
+ fi
+
+ # --single-base-dtb forces a single fdt-base-1 across all FDTs
+ run_wrap_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_orig.test.fit -o fdtfactor_single.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor_single.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_single.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-4" fdtfactor_single.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-base-1 data \
+ fit_single_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${i} data \
+ fit_single_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${i}.test.dtb fit_single_fdt${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fit_single_merged${i}.test.dtb
+ done
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${idx} data \
+ fit_single_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${idx}.test.dtb fit_single_fdt${idx}.test.dtbo
+ check_factored_dtb fdtfactor_other${i}.test.dtb \
+ fit_single_merged${idx}.test.dtb
+ done
+
+ # Verify that conflicting image properties (e.g. arch) in a group fail
+ cp fdtfactor_orig.test.fit fdtfactor_bad_arch.test.fit
+ run_wrap_test $DTPUT fdtfactor_bad_arch.test.fit /images/fdt-4 \
+ arch -ts "arm"
+ run_wrap_error_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_bad_arch.test.fit
+
+ # Verify that signed FITs (with signature* subnodes) and
+ # already-factored FITs (with no remaining compatible pairs) are rejected
+ cp fdtfactor_orig.test.fit fdtfactor_signed.test.fit
+ run_wrap_test $DTPUT fdtfactor_signed.test.fit -c \
+ /configurations/conf-1/signature-1
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor_signed.test.fit
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor.test.fit
+
+ # Pre-existing /images/fdt-base-* nodes do not collide: a second pass at
+ # --min-overlap 1 picks fdt-base-3 and produces a valid multi-stage chain.
+ run_wrap_test $PYTHON $FDTFACTOR -S --min-overlap 1 \
+ -F fdtfactor.test.fit -o fdtfactor_chained.test.fit
+ run_fdtget_test "fdt-base-3 fdt-base-1 fdt-1" fdtfactor_chained.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-3 fdt-base-2 fdt-4" fdtfactor_chained.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-base-3 data \
+ fit_chain_b3.test.dtb
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-base-1 data \
+ fit_chain_b1.test.dtbo
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-1 data \
+ fit_chain_f1.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_chain_b3.test.dtb \
+ -o fit_chain_merged1.test.dtb \
+ fit_chain_b1.test.dtbo fit_chain_f1.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.nolf.test.dtb \
+ fit_chain_merged1.test.dtb
+
+ if command -v lz4 >/dev/null 2>&1; then
+ # fdt-2 stays uncompressed in fdtfactor_lz4.its to verify mixed
+ # compression within a single FIT group.
+ run_wrap_test lz4 -12 -q -f -m \
+ fdtfactor_board1.test.dtb \
+ fdtfactor_board3.test.dtb fdtfactor_other1.test.dtb \
+ fdtfactor_other2.test.dtb overlay_overlay_phandle.test.dtb
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor_lz4.test.fit \
+ "$SRCDIR/fdtfactor_lz4.its"
+ run_wrap_test $PYTHON $FDTFACTOR -s \
+ -F fdtfactor_lz4.test.fit -o fdtfactor_lz4_out.test.fit
+ run_fdtget_test "fdt-base-1.dtb.lz4" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo.lz4" \
+ fdtfactor_lz4_out.test.fit /images/fdt-1 description
+ run_fdtget_test "fdtfactor_board2.test.dtbo" \
+ fdtfactor_lz4_out.test.fit /images/fdt-2 description
+ run_fdtget_test "fdt-base-2.dtb" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo" \
+ fdtfactor_lz4_out.test.fit /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-2 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-base-1 data \
+ fit_lz4_base1.test.dtb.lz4
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-1 data \
+ fit_lz4_fdt1.test.dtbo.lz4
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-2 data \
+ fit_lz4_fdt2.test.dtbo
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_base1.test.dtb.lz4
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_fdt1.test.dtbo.lz4
+ run_wrap_test $FDTOVERLAY -i fit_lz4_base1.test.dtb \
+ -o fit_lz4_merged1.test.dtb fit_lz4_fdt1.test.dtbo
+ check_factored_dtb fdtfactor_board1.test.dtb \
+ fit_lz4_merged1.test.dtb
+ run_wrap_test $FDTOVERLAY -i fit_lz4_base1.test.dtb \
+ -o fit_lz4_merged2.test.dtb fit_lz4_fdt2.test.dtbo
+ check_factored_dtb fdtfactor_board2.test.dtb \
+ fit_lz4_merged2.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1400,7 +1623,7 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.fit *.test.dtb.gz *.test.dtbo.gz \
*.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
*.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (12 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When factoring dozens or hundreds of compressed DTBs (either standalone
with `-C` or inside a compiled FIT with `-F`), running `gzip`, `lzma`,
`xz`, or external `lz4` processes sequentially spends several seconds
compressing and decompressing blobs one at a time.
Because Python's `zlib` and `_lzma` C extensions and `subprocess.run()`
for `lz4` all release the GIL, add a small `_map_blobs()` helper using
`concurrent.futures.ThreadPoolExecutor` to decompress input blobs and
compress output base/overlay blobs across worker threads in
deterministic input order whenever multiple compressed blobs are
processed.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch parallelizing blob compression and decompression across
worker threads
fdtfactor | 190 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 138 insertions(+), 52 deletions(-)
diff --git a/fdtfactor b/fdtfactor
index ebf40cdeaf92..ab59763d8863 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -18,6 +18,7 @@ its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
import argparse
from collections import Counter
+import concurrent.futures
import copy
import gzip
import hashlib
@@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+def _map_blobs(func, tasks, pool=None):
+ """Apply `func(blob, comp, filename)` across `tasks`, parallelizing if compressed.
+
+ Python's `zlib` and `_lzma` C extensions and external `lz4` subprocesses
+ release the GIL, so running multiple compressed tasks across a
+ `ThreadPoolExecutor` overlaps compression and decompression across CPU
+ cores while preserving deterministic input order.
+
+ Args:
+ func: `decompress_blob` or `compress_blob`.
+ tasks: Sequence of `(blob_bytes, comp, filename)` tuples.
+ pool: Optional shared `ThreadPoolExecutor` instance.
+
+ Returns:
+ List of processed `bytes` in the same order as `tasks`.
+ """
+ if len(tasks) < 2 or not any(
+ comp not in ('none', '', None) for _, comp, _ in tasks
+ ):
+ return [func(*t) for t in tasks]
+ if pool is not None:
+ return list(pool.map(lambda t: func(*t), tasks))
+ with concurrent.futures.ThreadPoolExecutor() as new_pool:
+ return list(new_pool.map(lambda t: func(*t), tasks))
+
+
def _is_dtbo_overlay(root):
"""Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
@@ -1061,13 +1088,14 @@ def group_fit_dtbs(fdt_entries, min_overlap=50.0):
return [grp['entries'] for grp in groups]
-def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node, pool=None):
"""Extract and parse non-overlay `flat_dt` entries from `/images`.
Args:
fit_in_path: Input FIT path (for error reporting).
images_node: `DeviceTree` for the FIT `/images` node.
configs_node: `DeviceTree` for the FIT `/configurations` node.
+ pool: Optional shared `ThreadPoolExecutor` for decompression.
Returns:
List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
@@ -1081,7 +1109,7 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
f'is not supported; sign the FIT after factoring'
)
- fdt_entries = []
+ candidates = []
for img_name, img_node in images_node.children.items():
if ('data-offset' in img_node.props or
'data-position' in img_node.props):
@@ -1112,7 +1140,19 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
)
if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
comp = 'lz4_legacy'
- raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+ candidates.append((img_name, img_node, node_label, comp, raw_data))
+
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(raw_data, comp, node_label)
+ for _, _, node_label, comp, raw_data in candidates],
+ pool=pool,
+ )
+
+ fdt_entries = []
+ for (img_name, img_node, node_label, comp, raw_data), raw_bytes in zip(
+ candidates, raw_blobs
+ ):
fdt = _open_fdt(raw_bytes, node_label)
root = FactoredTree.unflatten(fdt)
if _is_dtbo_overlay(root):
@@ -1194,13 +1234,15 @@ def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
return factorable_groups, unfactored_entries
-def _strip_fit_unfactored_local_fixups(unfactored_entries):
+def _strip_fit_unfactored_local_fixups(unfactored_entries, pool=None):
"""Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
Args:
unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
unfactored by `_partition_fit_fdt_entries()`.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
"""
+ to_recompress = []
for entry in unfactored_entries:
fdt = entry.pop('fdt')
tree = libpydt.DeviceTree.unflatten(fdt)
@@ -1210,17 +1252,26 @@ def _strip_fit_unfactored_local_fixups(unfactored_entries):
mem_rsv=entry['dtb'].mem_rsv,
boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
)
- comp_bytes = compress_blob(
- bytes(stripped_fdt.as_bytearray()),
+ stripped_bytes = bytes(stripped_fdt.as_bytearray())
+ to_recompress.append((
+ entry['node'],
+ stripped_bytes,
entry['dtb'].compression,
- filename=f'/images/{entry["name"]}',
- )
- img_node = entry['node']
+ f'/images/{entry["name"]}',
+ ))
+
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [(blob, comp, label) for _, blob, comp, label in to_recompress],
+ pool=pool,
+ )
+ for (img_node, _, _, _), comp_bytes in zip(to_recompress, comp_blobs):
img_node.set_prop(libfdt.Property('data', comp_bytes))
_update_fit_hashes(img_node)
-def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups,
+ pool=None):
"""Factor one group of FIT `flat_dt` entries into a base node and overlays.
Args:
@@ -1229,6 +1280,7 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
node (e.g., `'fdt-base-1'`).
strip_local_fixups: If True, omit `/__local_fixups__` from the
generated base DTB.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
Returns:
Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
@@ -1241,9 +1293,20 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
)
base_comp = group_dtbs[0].compression
- base_dtb_comp_bytes = compress_blob(
- base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, base_comp, f'/images/{base_dtb_node_name}'),
+ *(
+ (dtbo_blob, entry['dtb'].compression, f'/images/{entry["name"]}')
+ for entry, dtbo_blob in zip(entries, dtbo_blobs)
+ ),
+ ],
+ pool=pool,
)
+ base_dtb_comp_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
base_dtb_node = copy.deepcopy(entries[0]['node'])
for prop in ('phandle', 'compatible'):
base_dtb_node.props.pop(prop, None)
@@ -1272,12 +1335,9 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
_update_fit_hashes(base_dtb_node)
dtbo_comp_sizes = []
- for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ for entry, dtbo_comp_bytes in zip(entries, dtbo_comp_blobs):
img_node = entry['node']
entry_comp = entry['dtb'].compression
- dtbo_comp_bytes = compress_blob(
- dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
- )
dtbo_comp_sizes.append(len(dtbo_comp_bytes))
if 'description' in img_node.props:
@@ -1386,39 +1446,42 @@ def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
f'/configurations nodes'
)
- fdt_entries = _extract_fit_fdt_entries(
- fit_in_path, images_node, configs_node
- )
- factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
- fdt_entries, single_base_dtb, min_overlap=min_overlap
- )
- if strip_local_fixups and unfactored_entries:
- _strip_fit_unfactored_local_fixups(unfactored_entries)
- else:
- for entry in unfactored_entries:
- entry.pop('fdt', None)
-
- fdt_to_base_dtb = {}
- insert_before_first = {}
- group_stats = []
- used_image_names = set(images_node.children)
- next_base_idx = 1
-
- for entries in factorable_groups:
- while f'fdt-base-{next_base_idx}' in used_image_names:
- next_base_idx += 1
- base_dtb_node_name = f'fdt-base-{next_base_idx}'
- used_image_names.add(base_dtb_node_name)
- next_base_idx += 1
- base_dtb_node, stat_args = _factor_fit_group(
- entries, base_dtb_node_name, strip_local_fixups
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node, pool=pool
)
- insert_before_first[entries[0]['name']] = (
- base_dtb_node_name, base_dtb_node
+ factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb, min_overlap=min_overlap
)
- for entry in entries:
- fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
- group_stats.append(stat_args)
+ if strip_local_fixups and unfactored_entries:
+ _strip_fit_unfactored_local_fixups(unfactored_entries, pool=pool)
+ else:
+ for entry in unfactored_entries:
+ entry.pop('fdt', None)
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+ used_image_names = set(images_node.children)
+ next_base_idx = 1
+
+ for entries in factorable_groups:
+ while f'fdt-base-{next_base_idx}' in used_image_names:
+ next_base_idx += 1
+ base_dtb_node_name = f'fdt-base-{next_base_idx}'
+ used_image_names.add(base_dtb_node_name)
+ next_base_idx += 1
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups, pool=pool
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (
+ base_dtb_node_name, entry['dtb']
+ )
+ group_stats.append(stat_args)
_update_fit_images_and_configs(
images_node, configs_node, insert_before_first, fdt_to_base_dtb
@@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
"""
comp = args.compression
input_dtb_paths = args.dtbs
- dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
+ file_blobs = []
+ for p in input_dtb_paths:
+ with open(p, 'rb') as f:
+ file_blobs.append(f.read())
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(b, comp, p) for p, b in zip(input_dtb_paths, file_blobs)],
+ )
+ dtbs = [
+ ParsedDtb(
+ p,
+ comp=comp,
+ fdt=_open_fdt(raw, p),
+ uncompressed_size=len(raw),
+ compressed_size=len(fblob),
+ )
+ for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
+ ]
base_dtb_path = (
os.path.join(args.output, args.output_base_dtb)
@@ -1706,20 +1786,26 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
- base_dtb_file_bytes = compress_blob(
- base_dtb_blob, comp, filename=base_dtb_path
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, comp, base_dtb_path),
+ *((b, comp, p) for p, b in zip(dtbo_paths, dtbo_blobs)),
+ ],
)
+ base_dtb_file_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
if args.output:
os.makedirs(args.output, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
f.write(base_dtb_file_bytes)
dtbo_file_sizes = []
- for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ for out_path, out_bytes in zip(dtbo_paths, dtbo_comp_blobs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
- out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
f.write(out_bytes)
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread