* [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays
@ 2026-11-16 20:52 Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
` (13 more replies)
0 siblings, 14 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:52 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Devicetree overlays exist at both the source level (via `/include/` and
`.dtsi` files) and the binary level (where a compiled `.dtbo` overlay is
applied to a "base" `.dtb` at runtime to add or modify nodes).
Binary `.dtbo` overlays were created by transcribing source-level
overlay semantics to `.dtb` blobs to serve two main use cases:
1. Supporting add-on boards (like "hats" on a dev board), where a base
board's `.dtb` is combined with an add-on board's `.dtbo` at boot or
runtime (a use case that Hervé Codina's upcoming "addons" work aims
to model more cleanly).
2. Allowing constrained bootloaders or firmware images to support a
"family" of similar boards in a space-efficient way without shipping
a full `.dtb` for every variant. In upstream Linux, for example,
there are 47 distinct sc7180-trogdor boards supported. At the source
level these boards share common `.dtsi` files, but we still compile
(and ship) 47 separate full `.dtb` blobs. Binary devicetree overlays
can make this much more efficient by factoring the common bits into a
single base `.dtb` and representing each of the 47 boards as a small
`.dtbo` overlay.
Even though `.dtbo` overlay semantics are a natural fit for case #2
(since they derive from source-level `.dts` overlays, which were
designed for board/SoC families), upstream Linux has mostly reserved
compiled `.dtso` -> `.dtbo` overlays for case #1. Many Android phones
attempt to use `.dtso` overlays for case #2 by describing the SoC as the
base devicetree and describing boards in `.dtso` overlays, but forcing
the binary base/overlay split into the source files has drawbacks: there
is still lots of duplication between board overlays, and upstream
reception towards structuring board source files this way has been
lukewarm [1].
This patch series introduces a tool to decouple binary `.dtbo`
generation from `.dts` source structure. The idea (suggested by Konrad
Dybcio at LPC) is to continue to represent all the boards in a family as
normal standalone device trees in source code (using `.dtsi` includes as
makes sense for human readability). Then, after building the full `.dtb`
blobs, `fdtfactor` automatically factors them into a common base `.dtb`
and a set of per-board `.dtbo` overlays.
This tool has quite a few advantages over other solutions:
1. No change to the way device trees are stored / arranged upstream.
2. No need to deal with some of the awkwardness / limitations of
overlays.
3. Better savings than just a simple partitioning, like between SoC and
boards.
I've used `fdtfactor -S` to reduce sc7180-trogdor device tree files
(which are stored lz4 compressed) from 1.08 MB (built without `-@ -L`)
to 214 KB, an 80.2% savings.
To compare against the manual SoC-base + board-overlay split used by
downstream Android kernels, I also tested `fdtfactor -S` on the
downstream Pixel 10 family device trees (29 physical hardware board
variants: 27 phone variants across Pixel 10, Pixel 10 Pro, and Pixel 10
Pro XL plus 2 socketed dev board variants) by combining the downstream
base + overlays into full board DTBs and re-factoring them with
`fdtfactor -S`:
* Just B0 silicon (29 boards):
- Full unfactored DTBs (29 `.dtb`s): 26.30 MB uncompressed (5.28 MB
lz4)
- Downstream manual SoC base (`lga-b0.dtb`) + 29 board `.dtbo`s:
8.06 MB uncompressed (1.66 MB lz4)
- Auto-factored with `fdtfactor -S` (1 `base.dtb` + 29 `.dtbo`s):
3.27 MB uncompressed (668 KB lz4) -- 2.48x smaller than the manual
SoC split (and 3.68x smaller across the 27 phone boards alone, where
average overlay size drops from 252 KB to 44 KB uncompressed, or
52 KB to 9.1 KB lz4).
* Both A0 and B0 SoC steppings:
In practice, phone Proto boards (8 variants) only ever use A0 silicon,
phone EVT1.1+ boards (15 variants) only ever use B0 silicon, and only
phone EVT1 boards (4 variants) and the socketed dev boards (2
variants) can have either A0 or B0 silicon -- giving 35 realistic
(board, stepping) combinations (14 A0 + 21 B0), compared to 58 for a
naive 29 x 2 cross-product:
- Full unfactored DTBs (35 `.dtb`s): 31.71 MB uncompressed (6.37 MB
lz4)
- Downstream manual split (2 SoC bases `lga-a0.dtb` + `lga-b0.dtb` +
29 board `.dtbo`s): 8.78 MB uncompressed (1.79 MB lz4)
- Auto-factored with `fdtfactor -S` (1 shared `base.dtb` + 35
`.dtbo`s): 4.00 MB uncompressed (855 KB lz4). This could likely be
improved more with multiple levels of overlays, but we're already
2x smaller than what downstream is doing today.
NOTE: on Chromebooks like sc7180-trogdor, using overlays like this is
an optimization. On Android phones like Pixel 10, the bootloader logic
bakes in the need for overlays. Specifically, if we want to be able to
build a single image that supports multiple boards and have the
bootloader auto-construct the right devicetree, we need the boards to be
represented by overlays. This fdtfactor tool can make that a reality.
The fdtfactor tool can either work directly on a set of compiled
devicetree files or it can work on a FIT (flattened image tree) file.
The idea is that you'd create the FIT file containing all of the
duplicated devicetree files and then fdtfactor would "optimize" them.
To test multi-SoC FIT factoring, I also built an upstream Linux
`arch/arm64` `image.fit` containing all 438 Qualcomm device trees (367
standalone `.dtb`s plus 71 composite `.dtb`s built from 30 add-on
`.dtbo`s; gzip-compressed). Comparing the embedded FDT blob totals (with
non-factored cases built without `-@ -L`, and factored cases using
`fdtfactor -S`):
* Plain FIT (438 full `.dtb`s): 49.13 MB uncompressed (9.27 MB gzip)
* `FIT_DECOMPOSE_DTBS=1` alone (deduplicating the 71 composite `.dtb`s
into 367 `.dtb`s + 30 add-on `.dtbo`s): 36.23 MB uncompressed (6.94 MB
gzip)
* `fdtfactor` alone (41 `fdt-base-*` groups, 426 board `.dtbo`s, 12
single-board SoCs): 18.26 MB uncompressed (3.59 MB gzip -- 61.3%
compressed savings vs. plain)
* `FIT_DECOMPOSE_DTBS=1` + `fdtfactor` (41 `fdt-base-*` groups, 355
board `.dtbo`s, 30 add-on `.dtbo`s, 12 single-board SoCs): 15.28 MB
uncompressed (3.03 MB gzip -- 67.3% compressed savings vs. plain,
56.3% vs. `FIT_DECOMPOSE_DTBS=1` alone)
Before introducing `fdtfactor` itself, the series starts with a few
preparatory patches: documenting the existing tools in `README.md`,
adding a `DeviceTree` live-tree abstraction in `pylibfdt/libpydt.py`
(with supporting `Property`, `FdtRo`, and `FdtSw` enhancements),
speeding up `libfdt` string-table lookups with `memchr()`, and teaching
`libfdt`'s `fdt_overlay_apply()` to update `/__local_fixups__` on the
base tree so merged trees retain complete and accurate phandle location
metadata (for runtime consumers, composite `.dtb`s built by `fdtoverlay`
that are later fed into `fdtfactor`, and round-trip verification against
`dtc -L`).
I'll freely admit that the patches in this series were pretty heavily
vibe coded. I'll not pretend that I have a deep understanding of the
intricacies of how overlays are implemented. That being said:
* I've spent quite a bit of time reviewing the code from a high-level
perspective: confirming that functions are well documented, well
structured, and make sense.
* I believe that what the tool is doing and how the tool is used make
sense.
* I have had quite a number of internal "the AI reviews the AI" rounds
to the point of diminishing returns.
At this point, I'm not embarrassed to put my name behind this. While I
won't promise that there are zero cases of the AI doing something really
stupid, I'd be willing to bet that the end result is better (and cleaner
/ better structured) than what I would have coded up if I had spent a
few weeks doing this on my own.
[1] https://lore.kernel.org/all/20251114-elite-refined-yak-bf9e64@kuoka/
v1: https://lore.kernel.org/r/20261015182221.2931742-1-dianders@chromium.org
v2: https://lore.kernel.org/r/20261028215600.1713475-1-dianders@chromium.org
v3: https://lore.kernel.org/r/20261101220224.1436242-1-dianders@chromium.org
Changes in v4:
- New patch adding README.md blurbs for dtc and ancillary tools (David
Gibson)
- New patch making Property a subclass of bytes instead of bytearray
(David Gibson)
- New patch trimming the returned Fdt bytearray in FdtSw.as_fdt() so
DeviceTree.flatten() does not need to call fdt.pack() (David Gibson)
- New patch adding Property.from_*() creation classmethods so
DeviceTree does not need its own get_*/set_* wrapper methods (David
Gibson)
- New patch speeding up Property.as_list() with a single struct.unpack()
call
- New patch speeding up FdtRo.properties() with a C helper
- Start the commit message with the motivation for a live tree and
condense the summary (Simon Glass)
- Rename devicetree.py to libpydt.py (under pylibfdt/) to avoid
colliding with PyPI's python-devicetree package, and move its unit
tests into a separate tests/libpydt_tests.py file (Simon Glass, David
Gibson)
- Unflatten the tree in a single O(n) pass using fdt.next_node() instead
of recursive O(d * n) fdt.subnodes() traversal, raising FdtException
on corrupt/truncated trees, and add a unit test (David Gibson, Simon
Glass)
- Implement DeviceTree.walk() iteratively with an explicit stack instead
of recursive yield from, and populate node.props directly in
unflatten()
- Drop redundant fdt.pack() call from DeviceTree.flatten() (David
Gibson)
- Drop get_*/set_* typed wrapper methods from DeviceTree in favor of
Property.as_*() and Property.from_*() plus DeviceTree.set_prop()
(David Gibson)
- Drop unnecessary copy.deepcopy() and compare unordered node/child sets
instead of ordered lists in round-trip test (David Gibson)
- New patch speeding up fdt_find_string_len_() with memchr()
- Split out internal helper fdt_add_empty_subnode_() so
overlay_fragment_local_fixups() avoids passing an in-blob pointer to
fdt_add_subnode_namelen() or casting fdt_get_name() (Simon Glass)
- Add test cases in tests/overlay.c where the overlay target has no
existing subnode under /__local_fixups__, with /__local_fixups__ both
before and after the target in the blob (Simon Glass)
- Pass OVERLAY_LOCAL_FIXUPS to tests/run_tests.sh, adjust
fdtoverlay_tests for all three modes, and use 4-space indentation
(Simon Glass, David Gibson)
- Rename -o/--outdir to -o/--output so standalone DTB mode and FIT mode
share the same output option, and use 4-space indentation in
tests/run_tests.sh (David Gibson)
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Replace ParsedDtb.raw_bytes with uncompressed_size, use early continue
to reduce loop nesting, drop redundant set() before sorted() on local
fixup offsets, and clarify pre-order walk and phandle comments (David
Gibson)
- Allow dtc -@ -L inputs with no phandles (where dtc omits /__symbols__
and /__local_fixups__)
- Check _is_dtbo_overlay() on the unflattened DeviceTree, index
ParsedDtb.nodes and base_nodes by path, and skip unchanged base nodes
in _build_board_dtbo() to avoid repeated O(depth) traversals and
per-property comparisons
- Handle OVERLAY_LOCAL_FIXUPS != 2 in fdtfactor_tests (Simon Glass)
- Use 4-space indentation instead of mixed tabs and spaces in
tests/run_tests.sh, and update context for -o/--output (David Gibson)
- Replace magic-byte/extension auto-detection and --base-compression
with a single explicit -C/--compression option controlling both input
and output compression (David Gibson)
- Drop unnecessary --stats documentation/help changes and use 4-space
indentation in tests/run_tests.sh (David Gibson)
- Use check_factored_dtb in fdtfactor_tests so tests pass when
OVERLAY_LOCAL_FIXUPS != 2 (Simon Glass)
- Unify --outfile into -o/--output, and split standalone DTB vs. FIT
options into separate README.md subsections and --help groups (David
Gibson)
- Check _is_dtbo_overlay() directly in _extract_fit_fdt_entries()
instead of adding OverlayBlobError to ParsedDtb (David Gibson)
- Map standard FIT hash algorithm names explicitly via FIT_HASH_ALGOS
instead of passing raw names to hashlib.new() (David Gibson)
- Remove compression and entry from FIT grouping, document load, expose
--min-overlap, and compute FIT group overlap using normalized property
byte costs via _prop_byte_cost() instead of unweighted node paths
(David Gibson)
- Strip /__local_fixups__ from unfactored single-board FIT flat_dt
entries when -S/--strip-local-fixups is passed, and pick the next
unused fdt-base-N node name instead of rejecting FITs that already
contain fdt-base-* nodes
- Pass already-unflattened FactoredTree into ParsedDtb and intern
normalized properties to integer IDs in group_fit_dtbs() to speed up
multi-SoC FIT factoring
- Preserve whether input FIT FDT descriptions include a compression
extension instead of unconditionally appending one
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Use check_factored_dtb and 4-space indentation in tests/run_tests.sh
(Simon Glass, David Gibson)
- Drop patches 1-6 and 8-10 from v3 (applied upstream by David Gibson)
- Add preparatory README.md patch documenting dtc and ancillary tools
(David Gibson)
- Add pylibfdt preparatory patches making Property a subclass of bytes,
trimming FdtSw.as_fdt() to totalsize(), adding Property.from_*()
creation helpers (David Gibson), and speeding up Property.as_list()
and FdtRo.properties()
- Rename devicetree.py to pylibfdt/libpydt.py and address v3 review
feedback on DeviceTree (Simon Glass, David Gibson)
- Add preparatory libfdt patch speeding up fdt_find_string_len_() with
memchr()
- Update cover letter motivation for fdt_overlay_apply()
/__local_fixups__ handling (Simon Glass)
- Replace compression auto-detection with explicit -C/--compression,
unify -o/--output across DTB and FIT modes, and group FIT FDTs by
configurable --min-overlap shared normalized property byte cost
instead of a hardcoded 50% node-path heuristic (David Gibson)
- Speed up multi-SoC FIT factoring, parallelize blob
compression/decompression, and add Qualcomm FIT space-savings numbers
(and non-`-@ -L` sc7180-trogdor baseline) to cover letter
- New patch parallelizing blob compression and decompression across
worker threads
Changes in v3:
- Move NodeTree out of pylibfdt/libfdt.i into a new devicetree.py module
and rename it to DeviceTree (David Gibson)
- Store libfdt.Property objects in self.props instead of raw bytes
(David Gibson)
- Return a packed libfdt.Fdt object from flatten() instead of raw bytes
and use fdt.set_boot_cpuid_phys() (David Gibson, Simon Glass)
- Raise ValueError in unflatten() on duplicate property or sibling
subnode names (Simon Glass)
- Move constifying fdto in overlay_apply_node() into a standalone
preparatory patch (David Gibson)
- Gate `/__local_fixups__` handling behind a 3-state compile-time
`FDT_OVERLAY_LOCAL_FIXUPS` flag (`0=ignore`, `1=strip`, `2=update`),
defaulting to `0` (`IGNORE`) in `libfdt_internal.h` for embedded
builds and `2` (`UPDATE`) in Meson/Make desktop builds (Simon Glass,
David Gibson)
- Rewrite commit message in the present tense with motivation up front,
and update `fdt_overlay_apply()` documentation in `libfdt/libfdt.h`
(Simon Glass)
- Require a strict relative path in `overlay_create_path()` and return
`-FDT_ERR_BADOVERLAY` on empty path components or trailing `'/'`
(David Gibson)
- Eliminate the 256-byte `namebuf` limit in
`overlay_fragment_local_fixups()` by re-reading the node name from the
shifted ancestor offset after `fdt_add_subnode_namelen()` (David
Gibson, Simon Glass)
- Return `-FDT_ERR_BADVALUE` instead of `-FDT_ERR_BADSTRUCTURE` in
`overlay_add_local_fixup_offset()` when an existing
`/__local_fixups__` property length is not a multiple of 4 (David
Gibson, Simon Glass)
- Drop redundant `overlay_validate_local_fixups()` pre-pass and inline
single-use helpers (Simon Glass)
- Update fdtfactor to use devicetree.DeviceTree and a FactoredTree
subclass initializing self.phandle and self.fixups in __init__()
(David Gibson, Simon Glass)
- Rename -b/--base-dtb to --output-base-dtb to avoid confusion with an
input base DTB (David Gibson)
- Move the test rejecting input DTBs lacking /__local_fixups__ from the
-S patch into this base patch (Simon Glass)
- Include both non--S (932 KB) and -S (634 KB) Trogdor size numbers in
the commit message to show the savings of -S itself (Simon Glass)
- Pass -S to the negative test verifying that inputs lacking
/__local_fixups__ are still rejected when -S is used, and move the
non--S rejection test to the base fdtfactor patch (Simon Glass)
- Update for -b/--base-dtb rename to --output-base-dtb (David Gibson)
- Use "FIT" instead of "FIT image" in documentation, comments, and
commit messages (Simon Glass)
- Update for DeviceTree and --output-base-dtb (Simon Glass, David
Gibson)
- Drop "pylibfdt: Fix FdtSw.as_fdt() when buffer is nearly full" and
"pylibfdt: Add subnode/property iterators and get_mem_rsv_list()"
(applied upstream by David Gibson)
- Split fixup offset overflow fix, redundant memchr() check cleanup,
iterator docstring/test follow-up, Property.as_str()/as_stringlist()
empty-property handling, Fdt.set_boot_cpuid_phys(),
Property.__copy__()/__deepcopy__(), and overlay_apply_node() const
fdto into standalone patches (David Gibson, Simon Glass)
- Move NodeTree into a new devicetree.py module as DeviceTree (David
Gibson)
- Gate fdt_overlay_apply() /__local_fixups__ handling behind a 3-state
compile-time FDT_OVERLAY_LOCAL_FIXUPS option (Simon Glass, David
Gibson)
Changes in v2:
- Split live tree representation out of fdtfactor (David Gibson)
- New patch updating base /__local_fixups__ in libfdt
fdt_overlay_apply() instead of stripping /__local_fixups__ by default
(David Gibson)
- Preserve /__local_fixups__ on base.dtb by default (David Gibson)
- Store parsed trees directly in NodeTree with node.phandle and
node.fixups attributes instead of parallel dicts (David Gibson)
- Extract phandles in a separate pass after unflattening and reject
legacy linux,phandle properties (David Gibson)
- Exclude non-root common phandle nodes lacking a common symbol from
base.dtb instead of synthesizing __fdtfactor_ph_* symbols (David
Gibson)
- Drop -q/--quiet, rename -v/--verbose to -s/--stats, and move tests to
a dedicated fdtfactor test suite (David Gibson)
- Split -S/--strip-local-fixups into a separate opt-in patch (David
Gibson)
- Add --base-compression to explicitly control base.dtb compression
while documenting auto-detection from --base-dtb's filename suffix
(David Gibson)
- Support legacy LZ4 frames (lz4 -l) and use CRC32 check for XZ outputs
(Simon Glass)
- Rename -f/--fit and -F/--fit-out to -F/--fit and --outfile and clarify
FIT mode in the --help description (David Gibson, Simon Glass)
- Reject signed FIT images with signature* subnodes (Simon Glass)
- Use '<base-node-name>.dtb' for FIT base FDT descriptions instead of
common-prefix heuristics (Simon Glass)
- Compare FIT group node overlap against the group's common node set
instead of only the first entry (Simon Glass)
- Document in README.md that U-Boot's boot_get_fdt_fit() currently
requires FIT overlays to be uncompressed (Simon Glass)
- Drop "tests: Fix Python test environment setup in meson.build"
(applied upstream by David Gibson)
- Clarify source-level (.dts/.dtsi) vs. binary-level (.dtb/.dtbo)
overlay history in cover letter (David Gibson)
- Add Pixel 10 (Tensor G5) base + overlay comparison numbers to cover
letter (Simon Glass)
Douglas Anderson (14):
README: Add overview blurbs for dtc and ancillary tools
pylibfdt: Make Property a subclass of bytes instead of bytearray
pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt()
pylibfdt: Add Property.from_*() creation methods
pylibfdt: Unpack Property.as_list() in a single struct.unpack() call
pylibfdt: Speed up FdtRo.properties() with a C helper
libpydt: Add DeviceTree live tree representation
libfdt: Speed up fdt_find_string_len_() with memchr()
libfdt: overlay: Update base /__local_fixups__ on apply
fdtfactor: Add tool to split DTBs into a base DTB and overlays
fdtfactor: Add -S/--strip-local-fixups option
fdtfactor: Support compressed DTB inputs and outputs
fdtfactor: Add compiled FIT factoring support
fdtfactor: Parallelize blob compression and decompression
Makefile | 8 +-
README.md | 189 +++
fdtfactor | 1852 ++++++++++++++++++++++++++++
libfdt/fdt.c | 14 +-
libfdt/fdt_overlay.c | 525 +++++++-
libfdt/fdt_rw.c | 37 +-
libfdt/libfdt.h | 7 +
libfdt/libfdt_internal.h | 23 +
meson.build | 6 +
meson_options.txt | 2 +
pylibfdt/libfdt.i | 141 ++-
pylibfdt/libpydt.py | 184 +++
pylibfdt/meson.build | 2 +
tests/Makefile.tests | 13 +-
tests/fdtfactor_board1.dts | 72 ++
tests/fdtfactor_board2.dts | 70 ++
tests/fdtfactor_board3.dts | 68 +
tests/fdtfactor_board3_shifted.dts | 68 +
tests/fdtfactor_lz4.its | 109 ++
tests/fdtfactor_other1.dts | 67 +
tests/fdtfactor_other2.dts | 67 +
tests/fdtfactor_test.its | 109 ++
tests/libpydt_tests.py | 106 ++
tests/meson.build | 5 +-
tests/overlay.c | 154 +++
tests/pylibfdt_tests.py | 55 +
tests/run_tests.sh | 446 ++++++-
tests/testutils.sh | 1 +
28 files changed, 4354 insertions(+), 46 deletions(-)
create mode 100755 fdtfactor
create mode 100644 pylibfdt/libpydt.py
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
create mode 100644 tests/fdtfactor_board3_shifted.dts
create mode 100644 tests/fdtfactor_lz4.its
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
create mode 100644 tests/libpydt_tests.py
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
@ 2026-11-16 20:52 ` Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:52 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`README.md` currently jumps directly from building `dtc` and `pylibfdt`
to mailing list information without mentioning `dtc` itself or the
ancillary utilities built and installed alongside it.
Add brief overview sections with basic usage examples for `dtc`,
`dtdiff`, `fdtget`, `fdtput`, `fdtdump`, `convert-dtsv0`, and
`fdtoverlay` (along with a pointer to `Documentation/manual.txt` for
full command-line details).
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch adding README.md blurbs for dtc and ancillary tools (David
Gibson)
README.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 89 insertions(+)
diff --git a/README.md b/README.md
index fcfc2e55d804..9d38c7e79fc6 100644
--- a/README.md
+++ b/README.md
@@ -112,6 +112,95 @@ $ meson setup -Dpython=disabled build
More work remains to support all of libfdt, including access to numeric
values.
+## dtc
+
+`dtc` compiles device tree source (`.dts`) into flattened device tree
+blobs (`.dtb`), decompiles `.dtb` blobs or `/proc/device-tree`-style
+filesystem trees (`fs`) back to `.dts`, emits YAML (`.yaml`, when built
+with `libyaml`) or assembly (`.S`), and runs semantic checks on the
+tree:
+
+```
+$ dtc -I dts -O dtb -o board.dtb board.dts
+$ dtc -I dtb -O dts -o board.dts board.dtb
+```
+
+See `Documentation/manual.txt` for full command-line reference
+documentation for `dtc` and the C utility tools below.
+
+## dtdiff
+
+`dtdiff` normalizes and sorts two device trees via `dtc -s -O dts` and
+prints a unified diff between them. Each input may be a `.dts` source
+file, a `.dtb` / `.dtbo` blob, or a `/proc/device-tree`-style directory:
+
+```
+$ dtdiff before.dtb after.dtb
+$ dtdiff board.dts /proc/device-tree
+```
+
+## fdtget
+
+`fdtget` reads property values (`<node> <property>` pairs), lists
+property names (`-p`, `--properties`), or lists subnode names (`-l`,
+`--list`) from a `.dtb` blob. Use `-t`, `--type` (e.g. `-ts` for
+strings, `-ti` / `-tu` / `-tx` for signed, unsigned, or hex integers, or
+`-tr` for raw bytes) to control formatting, and `-d`, `--default` to
+provide a fallback when a node or property is missing:
+
+```
+$ fdtget board.dtb / model /cpus/cpu@0 reg
+$ fdtget -tx board.dtb /soc/serial@1000 reg
+$ fdtget -l board.dtb /soc
+```
+
+## fdtput
+
+`fdtput` modifies a `.dtb` blob in place: writing property values,
+creating nodes (`-c`, `--create`), deleting nodes and their subnodes
+(`-r`, `--remove`), or deleting properties (`-d`, `--delete`). Pass
+`-t`, `--type` (e.g. `-ts` for strings or `-tx` for hex integers;
+default is decimal integers) to control value encoding, and `-p`,
+`--auto-path` to create any missing nodes along the target path:
+
+```
+$ fdtput -p -ts board.dtb /soc/i2c@3000 compatible vendor,i2c
+$ fdtput -tx board.dtb /soc/serial@1000 reg 1000 100
+$ fdtput -d board.dtb /soc/i2c@3000 compatible
+```
+
+## fdtdump
+
+`fdtdump` is a low-level debugging tool that dumps the FDT header,
+`/memreserve/` entries, and raw structure-block contents of a `.dtb`
+blob (`-d`, `--debug` prints tag offsets; `-s`, `--scan` scans a larger
+file for an embedded FDT). For general decompilation of a `.dtb` to
+`.dts`, use `dtc -I dtb -O dts` instead:
+
+```
+$ fdtdump -d board.dtb
+```
+
+## convert-dtsv0
+
+`convert-dtsv0` converts obsolete version 0 `.dts` source files to
+`/dts-v1/;` syntax, writing each output to `<filename>v1`:
+
+```
+$ convert-dtsv0 old-board.dts
+```
+
+## fdtoverlay
+
+`fdtoverlay` applies one or more compiled `.dtbo` overlays (`-v`,
+`--verbose` for progress) onto a base `.dtb` blob (`-i`, `--input`,
+compiled with `dtc -@` when overlays reference symbols) and writes the
+merged blob to `-o`, `--output`:
+
+```
+$ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
+```
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
@ 2026-11-16 20:52 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:52 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`Property` represents a snapshot of a property's name and value read
from a flattened device tree. Making `Property` a subclass of the
immutable `bytes` type instead of mutable `bytearray` prevents callers
from accidentally mutating a `Property` instance in place when sharing
it across data structures (such as `DeviceTree`), makes `Property`
hashable, and allows `Property` instances to be passed directly to
`Fdt.setprop()` without converting to `bytes` first.
Also return `bytes` instead of `bytearray` from the
`fdt_get_property_by_offset()` SWIG typemap and drop the redundant
`bytearray()` conversion in `FdtRo.getprop()`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch making Property a subclass of bytes instead of bytearray
(David Gibson)
pylibfdt/libfdt.i | 17 +++++++++--------
tests/pylibfdt_tests.py | 1 +
2 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index fa5f499ed96d..8a7dfed85f15 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -492,7 +492,7 @@ class FdtRo(object):
Returns:
Value of property as a Property object (which can be used as a
- bytearray/string), or -ve error number. On failure, returns an
+ bytes/string), or -ve error number. On failure, returns an
integer error
Raises:
@@ -502,7 +502,7 @@ class FdtRo(object):
quiet)
if pdata[0] is None:
return pdata[1]
- return Property(prop_name, bytearray(pdata[0]))
+ return Property(prop_name, pdata[0])
def hasprop(self, nodeoffset, prop_name, quiet=()):
"""Check if a node has a property
@@ -893,7 +893,7 @@ class Fdt(FdtRo):
return check_err(fdt_del_node(self._fdt, nodeoffset), quiet)
-class Property(bytearray):
+class Property(bytes):
"""Holds a device tree property name and value.
This holds a copy of a property taken from the device tree. It does not
@@ -902,11 +902,12 @@ class Property(bytearray):
Properties:
name: Property name
- value: Property value as a bytearray
+ value: Property value as bytes
"""
- def __init__(self, name, value):
- bytearray.__init__(self, value)
- self.name = name
+ def __new__(cls, name, value):
+ obj = super().__new__(cls, value)
+ obj.name = name
+ return obj
def __copy__(self):
return type(self)(self.name, self)
@@ -1275,7 +1276,7 @@ typedef uint32_t fdt32_t;
if ($1) {
resultobj = PyUnicode_FromString(
fdt_string(fdt1, fdt32_to_cpu($1->nameoff)));
- buff = PyByteArray_FromStringAndSize(
+ buff = PyBytes_FromStringAndSize(
(const char *)($1 + 1), fdt32_to_cpu($1->len));
resultobj = SWIG_AppendOutput(resultobj, buff);
}
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index ae46b29a8f5c..78d8976c0cf3 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -191,6 +191,7 @@ class PyLibfdtBasicTests(unittest.TestCase):
root = 0
poffset = self.fdt.first_property_offset(root)
prop = self.fdt.get_property_by_offset(poffset)
+ self.assertIsInstance(prop, bytes)
self.assertEqual(prop.name, 'compatible')
self.assertEqual(prop, b'test_tree1\0')
for copied in (copy.copy(prop), copy.deepcopy(prop)):
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt()
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`fdt_finish()` relocates the string table directly after the structure
block and updates the FDT header `totalsize` to the packed size, but
`FdtSw.as_fdt()` left the backing Python `bytearray` at the
pre-allocated `FdtSw` buffer size (with trailing zero bytes beyond
`totalsize()`), forcing callers that serialize `fdt.as_bytearray()` to
call `fdt.pack()` just to truncate `_fdt`.
Truncate `fdtsw` to `fdt_totalsize(fdtsw)` in `FdtSw.as_fdt()` (matching
`Fdt.pack()`) so the returned `Fdt`'s `as_bytearray()` length matches
its `totalsize()` immediately.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch trimming the returned Fdt bytearray in FdtSw.as_fdt() so
DeviceTree.flatten() does not need to call fdt.pack() (David Gibson)
pylibfdt/libfdt.i | 1 +
tests/pylibfdt_tests.py | 2 ++
2 files changed, 3 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 8a7dfed85f15..68f22e316ece 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -1026,6 +1026,7 @@ class FdtSw(FdtRo):
fdtsw = bytearray(self._fdt)
while self.check_space(fdt_finish(fdtsw)):
fdtsw = bytearray(self._fdt)
+ del fdtsw[fdt_totalsize(fdtsw):]
return Fdt(fdtsw)
def check_space(self, val):
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 78d8976c0cf3..26915a015ccb 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -684,6 +684,7 @@ class PyLibfdtSwTests(unittest.TestCase):
# Make sure we did at least two resizes
self.assertTrue(len(fdt.as_bytearray()) > FdtSw.INC_SIZE * 2)
+ self.assertEqual(fdt.totalsize(), len(fdt.as_bytearray()))
# Check that as_fdt() resizes when fewer than 4 bytes remain for FDT_END
sw_exact = FdtSw(76)
@@ -692,6 +693,7 @@ class PyLibfdtSwTests(unittest.TestCase):
sw_exact.end_node()
fdt_exact = sw_exact.as_fdt()
self.assertEqual(80, fdt_exact.totalsize())
+ self.assertEqual(80, len(fdt_exact.as_bytearray()))
class PyLibfdtRoTests(unittest.TestCase):
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (2 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`libfdt.Property` provides `as_*()` methods (`as_cell`, `as_uint32`,
`as_int32`, `as_uint64`, `as_int64`, `as_list`, `as_uint32_list`,
`as_int32_list`, `as_uint64_list`, `as_int64_list`, `as_str`, and
`as_stringlist`) to decode typed values from a `Property`, but lacks the
corresponding creation methods to construct a `Property` from typed
values.
Add `from_*()` classmethods mirroring each `as_*()` method so callers
(such as `DeviceTree` users) can construct typed `Property` instances
directly without duplicating encoding helpers on `DeviceTree`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- New patch adding Property.from_*() creation classmethods so
DeviceTree does not need its own get_*/set_* wrapper methods (David
Gibson)
pylibfdt/libfdt.i | 56 +++++++++++++++++++++++++++++++++++++++++
tests/pylibfdt_tests.py | 49 ++++++++++++++++++++++++++++++++++++
2 files changed, 105 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 68f22e316ece..ff9210d123cb 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -962,6 +962,62 @@ class Property(bytes):
parts = self[:-1].split(b'\x00')
return list(map(lambda x: x.decode('utf-8'), parts))
+ @classmethod
+ def from_cell(cls, name, fmt, val):
+ return cls(name, struct.pack('>' + fmt, val))
+
+ @classmethod
+ def from_uint32(cls, name, val):
+ return cls.from_cell(name, 'L', val)
+
+ @classmethod
+ def from_int32(cls, name, val):
+ return cls.from_cell(name, 'l', val)
+
+ @classmethod
+ def from_uint64(cls, name, val):
+ return cls.from_cell(name, 'Q', val)
+
+ @classmethod
+ def from_int64(cls, name, val):
+ return cls.from_cell(name, 'q', val)
+
+ @classmethod
+ def from_list(cls, name, fmt, vals):
+ vals = list(vals)
+ return cls(name, struct.pack(f'>{len(vals)}{fmt}', *vals))
+
+ @classmethod
+ def from_uint32_list(cls, name, vals):
+ return cls.from_list(name, 'L', vals)
+
+ @classmethod
+ def from_int32_list(cls, name, vals):
+ return cls.from_list(name, 'l', vals)
+
+ @classmethod
+ def from_uint64_list(cls, name, vals):
+ return cls.from_list(name, 'Q', vals)
+
+ @classmethod
+ def from_int64_list(cls, name, vals):
+ return cls.from_list(name, 'q', vals)
+
+ @classmethod
+ def from_str(cls, name, val):
+ """Unicode is supported by encoding to UTF-8"""
+ if '\x00' in val:
+ raise ValueError('String value contains embedded nul characters')
+ return cls(name, val.encode('utf-8') + b'\x00')
+
+ @classmethod
+ def from_stringlist(cls, name, vals):
+ """Unicode is supported by encoding to UTF-8"""
+ vals = list(vals)
+ if any('\x00' in v for v in vals):
+ raise ValueError('String value contains embedded nul characters')
+ return cls(name, b''.join(v.encode('utf-8') + b'\x00' for v in vals))
+
class FdtSw(FdtRo):
"""Software interface to create a device tree from scratch
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 26915a015ccb..e2f9c6d5afe4 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -414,6 +414,16 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(9223372036854775807,
self.get_prop("prop-uint64").as_uint64())
self.assertEqual(-2, self.get_prop("prop-int64").as_int64())
+ self.assertEqual(self.get_prop("prop-hex32"),
+ libfdt.Property.from_uint32("prop-hex32", 0xdeadbeef))
+ self.assertEqual(self.get_prop("prop-int32"),
+ libfdt.Property.from_int32("prop-int32", -2))
+ self.assertEqual(
+ self.get_prop("prop-uint64"),
+ libfdt.Property.from_uint64("prop-uint64", 9223372036854775807),
+ )
+ self.assertEqual(self.get_prop("prop-int64"),
+ libfdt.Property.from_int64("prop-int64", -2))
def testGetIntListProperties(self):
"""Test that we can access properties as integer lists"""
@@ -425,6 +435,28 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.get_prop("prop-int64-array").as_int64_list())
self.assertEqual([0x100000000, 0x1],
self.get_prop("prop-uint64-array").as_uint64_list())
+ self.assertEqual(
+ self.get_prop("prop-int32-array"),
+ libfdt.Property.from_int32_list("prop-int32-array", [128, -16, -2]),
+ )
+ self.assertEqual(
+ self.get_prop("prop-uint32-array"),
+ libfdt.Property.from_uint32_list(
+ "prop-uint32-array", [0x1, 0x98765432, 0xdeadbeef]
+ ),
+ )
+ self.assertEqual(
+ self.get_prop("prop-int64-array"),
+ libfdt.Property.from_int64_list(
+ "prop-int64-array", [0x100000000, -2]
+ ),
+ )
+ self.assertEqual(
+ self.get_prop("prop-uint64-array"),
+ libfdt.Property.from_uint64_list(
+ "prop-uint64-array", [0x100000000, 0x1]
+ ),
+ )
def testGetStringlistProperties(self):
"""Test that we can access properties as string list"""
@@ -434,6 +466,17 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(["this is a placeholder string", "string2"],
self.fdt.getprop(node, "placeholder").as_stringlist())
self.assertEqual([], libfdt.Property('empty', b'').as_stringlist())
+ self.assertEqual(
+ self.fdt.getprop(node, "compatible"),
+ libfdt.Property.from_stringlist(
+ "compatible", ["subsubnode1", "subsubnode"]
+ ),
+ )
+ self.assertEqual(
+ b'', libfdt.Property.from_stringlist("empty", [])
+ )
+ with self.assertRaises(ValueError):
+ libfdt.Property.from_stringlist("bad", ["embedded\x00nul"])
with self.assertRaises(ValueError) as e:
libfdt.Property('bad', b'no-nul').as_stringlist()
self.assertIn('lacks nul termination', str(e.exception))
@@ -552,6 +595,12 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.fdt.setprop_str(node, prop, TEST_STRING_3)
self.assertEqual(TEST_STRING_3,
self.fdt.getprop(node, prop).as_str())
+ self.assertEqual(
+ self.fdt.getprop(node, prop),
+ libfdt.Property.from_str(prop, TEST_STRING_3),
+ )
+ with self.assertRaises(ValueError):
+ libfdt.Property.from_str(prop, 'embedded\x00nul')
def testHasProp(self):
"""Test that we can check if a node has a property"""
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (3 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`Property.as_list()` currently decodes multi-cell properties via
`list(map(lambda x: x[0], struct.iter_unpack('>' + fmt, self)))`, which
constructs an iterator and `map` wrapper and invokes a Python `lambda`
for every cell.
Compute the element size via `struct.calcsize('>' + fmt)` and unpack all
elements in a single C `struct.unpack()` call instead.
On one stress test (decoding ~301k `/__local_fixups__` cell lists across
438 arm64 Qualcomm DTBs) this reduced `Property.as_list()` time by
~2.1x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up Property.as_list() with a single struct.unpack()
call
pylibfdt/libfdt.i | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index ff9210d123cb..e4a9357d278a 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -931,7 +931,8 @@ class Property(bytes):
return self.as_cell('q')
def as_list(self, fmt):
- return list(map(lambda x: x[0], struct.iter_unpack('>' + fmt, self)))
+ size = struct.calcsize('>' + fmt)
+ return list(struct.unpack(f'>{len(self) // size}{fmt}', self))
def as_uint32_list(self):
return self.as_list('L')
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (4 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Iterating over a node's properties via `FdtRo.properties()` currently
crosses the Python/SWIG boundary `1 + 2 * num_props` times per node
(`first_property_offset()`, `get_property_by_offset()`, and
`next_property_offset()`, along with their Python `check_err()` and
`check_err_null()` wrappers). When walking every property across large
device trees from Python, those per-property SWIG round-trips dominate
runtime.
Add a C helper `fdt_node_properties_stub()` in `pylibfdt/libfdt.i`
backing `FdtRo.properties()` so all `Property` objects for a node are
collected in a single C call per node, interning property name strings
via `PyUnicode_InternFromString()`. Also add an invalid-offset exception
check to `testPropertiesIterator` in `tests/pylibfdt_tests.py`.
On one stress test (walking all ~573k nodes and ~2.06M properties across
438 arm64 Qualcomm DTBs) this reduced `FdtRo.properties()` time by
~1.8x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up FdtRo.properties() with a C helper
pylibfdt/libfdt.i | 64 ++++++++++++++++++++++++++++++++++++++---
tests/pylibfdt_tests.py | 3 ++
2 files changed, 63 insertions(+), 4 deletions(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index e4a9357d278a..333a25f27946 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -24,6 +24,61 @@ static int fdt_property_stub(void *fdt, const char *name, const void *val,
return fdt_property(fdt, name, val, len);
}
+static PyObject *fdt_node_properties_stub(const void *fdt, int nodeoffset,
+ PyObject *prop_cls)
+{
+ PyObject *list = PyList_New(0);
+ int poff;
+
+ if (!list)
+ return NULL;
+
+ poff = fdt_first_property_offset(fdt, nodeoffset);
+ while (poff >= 0) {
+ const char *pname;
+ int len;
+ const void *val = fdt_getprop_by_offset(fdt, poff, &pname, &len);
+ PyObject *name_obj, *val_obj, *prop_obj;
+ int rc;
+
+ if (!val) {
+ Py_DECREF(list);
+ return PyLong_FromLong(len);
+ }
+ name_obj = PyUnicode_InternFromString(pname);
+ if (!name_obj) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ val_obj = PyBytes_FromStringAndSize((const char *)val, len);
+ if (!val_obj) {
+ Py_DECREF(name_obj);
+ Py_DECREF(list);
+ return NULL;
+ }
+ prop_obj = PyObject_CallFunctionObjArgs(prop_cls, name_obj, val_obj,
+ NULL);
+ Py_DECREF(name_obj);
+ Py_DECREF(val_obj);
+ if (!prop_obj) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ rc = PyList_Append(list, prop_obj);
+ Py_DECREF(prop_obj);
+ if (rc < 0) {
+ Py_DECREF(list);
+ return NULL;
+ }
+ poff = fdt_next_property_offset(fdt, poff);
+ }
+ if (poff != -FDT_ERR_NOTFOUND) {
+ Py_DECREF(list);
+ return PyLong_FromLong(poff);
+ }
+ return list;
+}
+
%}
%pythoncode %{
@@ -477,10 +532,9 @@ class FdtRo(object):
Raises:
FdtException on error (e.g. invalid nodeoffset)
"""
- poff = self.first_property_offset(nodeoffset, quiet=QUIET_NOTFOUND)
- while poff >= 0:
- yield self.get_property_by_offset(poff)
- poff = self.next_property_offset(poff, quiet=QUIET_NOTFOUND)
+ return iter(check_err(
+ fdt_node_properties_stub(self._fdt, nodeoffset, Property)
+ ))
def getprop(self, nodeoffset, prop_name, quiet=()):
"""Get a property from a node
@@ -1420,5 +1474,7 @@ int fdt_property_cell(void *fdt, const char *name, uint32_t val);
* function and a struct, which confuses SWIG.
*/
int fdt_property_stub(void *fdt, const char *name, const void *val, int len);
+PyObject *fdt_node_properties_stub(const void *fdt, int nodeoffset,
+ PyObject *prop_cls);
%include <libfdt.h>
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index e2f9c6d5afe4..9c0104bfbabb 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -270,6 +270,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(props[0], b'subnode1\0')
leaf = self.fdt.path_offset('/subnode@1/ss1')
self.assertEqual(list(self.fdt.properties(leaf)), [])
+ with self.assertRaises(FdtException) as e:
+ list(self.fdt.properties(-1))
+ self.assertEqual(e.exception.err, -libfdt.BADOFFSET)
def testFirstNextSubnodeOffsetExceptions(self):
"""Check except handling for first/next subnode functions"""
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (5 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`pylibfdt`'s `Fdt` and `FdtRo` classes operate directly on flattened
device tree blobs, where inserting or deleting nodes and properties
shifts subsequent node offsets. That works well for in-place queries and
localized edits, but makes multi-tree comparison, structural diffing,
and bottom-up tree construction (as needed by the upcoming `fdtfactor`
tool) awkward and error-prone.
Add `pylibfdt/libpydt.py` providing `DeviceTree`, an in-memory live tree
representation where each node holds `props` (a dict mapping property
name to `libfdt.Property`) and `children` (a dict mapping child node
name to child `DeviceTree`), along with `unflatten()` and `flatten()` to
convert between `FdtRo`/`Fdt` blobs and live trees, and helpers for path
lookup, node creation, and pre-order traversal.
Add unit tests in `tests/libpydt_tests.py` covering `DeviceTree`
round-tripping, subtree unflattening, duplicate-name and corrupt-tree
rejection, and navigation helpers.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Start the commit message with the motivation for a live tree and
condense the summary (Simon Glass)
- Rename devicetree.py to libpydt.py (under pylibfdt/) to avoid
colliding with PyPI's python-devicetree package, and move its unit
tests into a separate tests/libpydt_tests.py file (Simon Glass, David
Gibson)
- Unflatten the tree in a single O(n) pass using fdt.next_node() instead
of recursive O(d * n) fdt.subnodes() traversal, raising FdtException
on corrupt/truncated trees, and add a unit test (David Gibson, Simon
Glass)
- Implement DeviceTree.walk() iteratively with an explicit stack instead
of recursive yield from, and populate node.props directly in
unflatten()
- Drop redundant fdt.pack() call from DeviceTree.flatten() (David
Gibson)
- Drop get_*/set_* typed wrapper methods from DeviceTree in favor of
Property.as_*() and Property.from_*() plus DeviceTree.set_prop()
(David Gibson)
- Drop unnecessary copy.deepcopy() and compare unordered node/child sets
instead of ordered lists in round-trip test (David Gibson)
Changes in v3:
- Move NodeTree out of pylibfdt/libfdt.i into a new devicetree.py module
and rename it to DeviceTree (David Gibson)
- Store libfdt.Property objects in self.props instead of raw bytes
(David Gibson)
- Return a packed libfdt.Fdt object from flatten() instead of raw bytes
and use fdt.set_boot_cpuid_phys() (David Gibson, Simon Glass)
- Raise ValueError in unflatten() on duplicate property or sibling
subnode names (Simon Glass)
Changes in v2:
- Split live tree representation out of fdtfactor (David Gibson)
pylibfdt/libpydt.py | 184 +++++++++++++++++++++++++++++++++++++++++
pylibfdt/meson.build | 2 +
tests/libpydt_tests.py | 106 ++++++++++++++++++++++++
tests/meson.build | 2 +-
tests/run_tests.sh | 13 ++-
5 files changed, 302 insertions(+), 5 deletions(-)
create mode 100644 pylibfdt/libpydt.py
create mode 100644 tests/libpydt_tests.py
diff --git a/pylibfdt/libpydt.py b/pylibfdt/libpydt.py
new file mode 100644
index 000000000000..6d8501f432ca
--- /dev/null
+++ b/pylibfdt/libpydt.py
@@ -0,0 +1,184 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+"""libpydt - Non-flattened in-memory device tree representation in Python."""
+
+import libfdt
+
+
+class DeviceTree:
+ """In-memory live tree representation of a devicetree node and its subtree.
+
+ `props` and `children` are public dictionaries intended to be accessed and
+ manipulated directly by callers (e.g. `node.props['status'].as_str()`,
+ `node.props.get('reg')`, `node.children.get('soc')`), with `set_prop()`
+ provided as a convenience for inserting a `libfdt.Property` under its own
+ `prop.name`.
+
+ Attributes:
+ props: Dict mapping property name (str) to `libfdt.Property`.
+ children: Dict mapping child node name (str) to child `DeviceTree`.
+ """
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+
+ @classmethod
+ def unflatten(cls, fdt, offset=0):
+ """Unflatten an FdtRo subtree starting at `offset` into a DeviceTree.
+
+ Args:
+ fdt: An `FdtRo` (or `Fdt`) instance.
+ offset: Starting node offset (default 0 for the root node).
+
+ Returns:
+ A `DeviceTree` representing the node at `offset` and its
+ descendants.
+
+ Raises:
+ ValueError: If a node contains duplicate property names or
+ duplicate sibling subnode names.
+ """
+ stack = []
+ depth = 0
+ while depth >= 0:
+ node = cls()
+ props = node.props
+ for prop in fdt.properties(offset):
+ name = prop.name
+ if name in props:
+ raise ValueError(f'Duplicate property {name!r}')
+ props[name] = prop
+ if depth == 0:
+ stack = [node]
+ else:
+ parent = stack[depth - 1]
+ subname = fdt.get_name(offset)
+ if subname in parent.children:
+ raise ValueError(f'Duplicate subnode {subname!r}')
+ parent.children[subname] = node
+ del stack[depth:]
+ stack.append(node)
+ offset, depth = fdt.next_node(offset, depth)
+ if offset < 0:
+ libfdt.check_err(offset)
+ return stack[0]
+
+ def _flatten_subtree(self, sw, name):
+ """Recursively write this DeviceTree and its children into an FdtSw.
+
+ Args:
+ sw: An `FdtSw` sequential-write instance.
+ name: Node name for this subtree (empty string `''` for root).
+ """
+ with sw.add_node(name):
+ for prop_name, val in self.props.items():
+ sw.property(prop_name, val)
+ for child_name, child_node in self.children.items():
+ child_node._flatten_subtree(sw, child_name)
+
+ def flatten(self, mem_rsv=(), boot_cpuid_phys=0):
+ """Serialize this root DeviceTree into a packed `libfdt.Fdt` object.
+
+ Args:
+ mem_rsv: Iterable of (address, size) tuples for the FDT memory
+ reservation block.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+
+ Returns:
+ Packed `libfdt.Fdt` instance.
+ """
+ sw = libfdt.FdtSw()
+ for addr, size in mem_rsv:
+ sw.add_reservemap_entry(addr, size)
+ sw.finish_reservemap()
+ self._flatten_subtree(sw, '')
+ fdt = sw.as_fdt()
+ fdt.set_boot_cpuid_phys(boot_cpuid_phys)
+ return fdt
+
+ def set_prop(self, prop):
+ """Insert or replace a `libfdt.Property` in `self.props` under `prop.name`.
+
+ Args:
+ prop: `libfdt.Property` instance to store at `self.props[prop.name]`.
+ """
+ self.props[prop.name] = prop
+
+ def get_node(self, path):
+ """Look up a descendant DeviceTree by path without modifying the tree.
+
+ Follows the slash-separated node names in `path` starting from `self`
+ (leading, trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically, and `''` or
+ `'/'` returns `self`). If any component along `path` is not present in
+ `children`, returns `None`.
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` at `path`, or `None` if not found.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ node = node.children.get(part)
+ if node is None:
+ return None
+ return node
+
+ def create_path(self, path):
+ """Return the descendant DeviceTree at `path`, creating missing nodes.
+
+ Looks up a child or deeper descendant `DeviceTree` by following the
+ slash-separated node names in `path` starting from `self` (leading,
+ trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically). If any
+ node along `path` does not exist yet in `children`, a new empty
+ `DeviceTree` is created for it automatically (similar to `mkdir -p`).
+ Passing `''` or `'/'` returns `self`.
+
+ Example:
+ root = DeviceTree()
+ # Automatically creates root.children['soc'] and
+ # root.children['soc'].children['i2c@4000'], and returns the
+ # 'i2c@4000' DeviceTree.
+ i2c = root.create_path('/soc/i2c@4000')
+ i2c.set_prop(libfdt.Property.from_str('status', 'okay'))
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` instance at `path`.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ if part not in node.children:
+ node.children[part] = type(self)()
+ node = node.children[part]
+ return node
+
+ def walk(self, path=''):
+ """Yield `(path, node)` pairs for this subtree in pre-order.
+
+ Args:
+ path: Path prefix for `self` (defaults to `''` for the root node,
+ producing paths like `''`, `'/soc'`, `'/soc/i2c@4000'`).
+
+ Yields:
+ Tuple of `(path, node)` for `self` followed in pre-order by each
+ descendant `DeviceTree`.
+ """
+ stack = [(path, self)]
+ while stack:
+ cur_path, node = stack.pop()
+ yield cur_path, node
+ prefix = cur_path.rstrip('/')
+ for name, child in reversed(node.children.items()):
+ stack.append((f'{prefix}/{name}', child))
diff --git a/pylibfdt/meson.build b/pylibfdt/meson.build
index b13d8021497f..ccf6c1bf78a6 100644
--- a/pylibfdt/meson.build
+++ b/pylibfdt/meson.build
@@ -19,3 +19,5 @@ pylibfdt = py.extension_module(
dependencies: [libfdt_dep, py.dependency()],
install: true,
)
+
+py.install_sources('libpydt.py', pure: false)
diff --git a/tests/libpydt_tests.py b/tests/libpydt_tests.py
new file mode 100644
index 000000000000..0f3d8d68945a
--- /dev/null
+++ b/tests/libpydt_tests.py
@@ -0,0 +1,106 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+# libpydt - Tests for in-memory DeviceTree representation in Python
+#
+
+import sys
+import unittest
+
+sys.path.insert(0, '../pylibfdt')
+import libfdt
+from libfdt import FdtSw, Property
+import libpydt
+
+
+class DeviceTreeTests(unittest.TestCase):
+ """Test class for libpydt.DeviceTree"""
+
+ def setUp(self):
+ with open('test_tree1.dtb', mode='rb') as f:
+ self.fdt = libfdt.FdtRo(f.read())
+
+ def testRoundTrip(self):
+ """Test unflattening and flattening a tree via DeviceTree"""
+ tree = libpydt.DeviceTree.unflatten(self.fdt)
+ self.assertIsInstance(tree.props['compatible'], Property)
+ rsvmap = self.fdt.get_mem_rsv_list()
+ cpuid = self.fdt.boot_cpuid_phys()
+ fdt2 = tree.flatten(rsvmap, cpuid)
+ self.assertIsInstance(fdt2, libfdt.Fdt)
+ self.assertEqual(rsvmap, fdt2.get_mem_rsv_list())
+ self.assertEqual(cpuid, fdt2.boot_cpuid_phys())
+
+ tree2 = libpydt.DeviceTree.unflatten(fdt2)
+ nodes1 = {
+ path: (n.props, set(n.children.keys()))
+ for path, n in tree.walk()
+ }
+ nodes2 = {
+ path: (n.props, set(n.children.keys()))
+ for path, n in tree2.walk()
+ }
+ self.assertEqual(nodes1, nodes2)
+
+ def testUnflattenSubtree(self):
+ """Test unflattening a non-root subtree offset"""
+ suboff = self.fdt.path_offset('/subnode@1')
+ sub = libpydt.DeviceTree.unflatten(self.fdt, suboff)
+ self.assertEqual('subnode1', sub.props['compatible'].as_str())
+ self.assertEqual({'subsubnode', 'ss1'}, set(sub.children.keys()))
+
+ def testDuplicates(self):
+ """Test that DeviceTree.unflatten() rejects duplicate names"""
+ sw_prop = FdtSw()
+ sw_prop.finish_reservemap()
+ with sw_prop.add_node(''):
+ sw_prop.property_u32('dup', 1)
+ sw_prop.property_u32('dup', 2)
+ with self.assertRaises(ValueError):
+ libpydt.DeviceTree.unflatten(sw_prop.as_fdt())
+
+ sw_node = FdtSw()
+ sw_node.finish_reservemap()
+ with sw_node.add_node(''):
+ with sw_node.add_node('dup'):
+ pass
+ with sw_node.add_node('dup'):
+ pass
+ with self.assertRaises(ValueError):
+ libpydt.DeviceTree.unflatten(sw_node.as_fdt())
+
+ def testCorruptTree(self):
+ """Test that DeviceTree.unflatten() raises FdtException on corrupt trees"""
+ with open('truncated_property.dtb', mode='rb') as f:
+ fdt_trunc = libfdt.FdtRo(f.read())
+ with self.assertRaises(libfdt.FdtException):
+ libpydt.DeviceTree.unflatten(fdt_trunc)
+
+ # Truncate size_dt_struct so the tree ends mid-subnode
+ corrupt = self.fdt.as_bytearray()
+ fdt_corrupt = libfdt.FdtRo(corrupt)
+ fdt_corrupt._fdt[36:40] = (32).to_bytes(4, 'big')
+ with self.assertRaises(libfdt.FdtException):
+ libpydt.DeviceTree.unflatten(fdt_corrupt)
+
+ def testNavigationAndProps(self):
+ """Test DeviceTree navigation and property setting"""
+ root = libpydt.DeviceTree()
+ self.assertIs(root, root.get_node(''))
+ self.assertIs(root, root.get_node('/'))
+ self.assertIsNone(root.get_node('/soc/i2c@4000'))
+
+ i2c = root.create_path('/soc/i2c@4000')
+ self.assertIs(i2c, root.get_node('/soc/i2c@4000'))
+ self.assertIs(i2c, root.get_node('soc/i2c@4000'))
+ self.assertEqual({'', '/soc', '/soc/i2c@4000'},
+ {path for path, _ in root.walk()})
+ self.assertEqual({'/', '/soc', '/soc/i2c@4000'},
+ {path for path, _ in root.walk('/')})
+
+ i2c.set_prop(Property.from_str('status', 'okay'))
+ self.assertEqual('okay', i2c.props['status'].as_str())
+ i2c.set_prop(Property.from_uint32('#address-cells', 1))
+ self.assertEqual(1, i2c.props['#address-cells'].as_uint32())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tests/meson.build b/tests/meson.build
index c25500606059..e37567864e04 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -119,7 +119,7 @@ if not pylibfdt_enabled
else
env += [
'PYTHON=' + py.full_path(),
- 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt',
+ 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt' + ':' + meson.project_source_root() / 'pylibfdt',
'NO_PYTHON=0',
]
endif
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index f750faa1f60a..3cacb68438df 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1149,17 +1149,16 @@ fdtoverlay_tests() {
run_wrap_test test "$bd" = "$pd"
}
-pylibfdt_tests () {
- run_dtc_test -I dts -O dtb -o test_props.dtb "$SRCDIR/test_props.dts"
+run_python_tests () {
TMP=/tmp/tests.stderr.$$
- $PYTHON "$SRCDIR/pylibfdt_tests.py" -v 2> $TMP
+ $PYTHON "$1" -v 2> $TMP
# Use the 'ok' message meaning the test passed, 'ERROR' meaning it failed
# and the summary line for total tests (e.g. 'Ran 17 tests in 0.002s').
# We could add pass + fail to get total tests, but this provides a useful
# sanity check.
pass_count=$(grep "ok$" $TMP | wc -l)
- fail_count=$(grep "^ERROR: " $TMP | wc -l)
+ fail_count=$(grep "^ERROR: \|^FAIL: " $TMP | wc -l)
total_tests=$(sed -n 's/^Ran \([0-9]*\) tests.*$/\1/p' $TMP)
cat $TMP
rm $TMP
@@ -1170,6 +1169,12 @@ pylibfdt_tests () {
tot_tests=$((tot_tests + $total_tests))
}
+pylibfdt_tests () {
+ run_dtc_test -I dts -O dtb -o test_props.dtb "$SRCDIR/test_props.dts"
+ run_python_tests "$SRCDIR/pylibfdt_tests.py"
+ run_python_tests "$SRCDIR/libpydt_tests.py"
+}
+
while getopts "vt:me" ARG ; do
case $ARG in
"v")
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr()
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (6 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
`fdt_find_string_len_()` currently calls `memcmp()` at every byte offset
across the string table (`for (p = strtab; p <= last; p++)`). When
building or modifying trees with large string tables (such as DTBs and
overlays with hundreds of `/__symbols__` or `/__fixups__` labels via
`fdt_sw` or `fdt_rw`), scanning multi-kilobyte string tables byte by
byte dominates serialization time.
Advance `p` between candidate first-byte matches using `memchr()` (with
an explicit `slen == 0` check for empty property names and a `tabsize <
slen + 1` bounds check before computing `last`), preserving identical
suffix-sharing behavior.
On one stress test (running the fdtfactor tool across 438 arm64 Qualcomm
DTBs) this reduced `fdt_property()` time by ~5.6x.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch speeding up fdt_find_string_len_() with memchr()
libfdt/fdt.c | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/libfdt/fdt.c b/libfdt/fdt.c
index d9ccb70345d0..c7cedfba7815 100644
--- a/libfdt/fdt.c
+++ b/libfdt/fdt.c
@@ -348,12 +348,20 @@ int fdt_next_subnode(const void *fdt, int offset)
const char *fdt_find_string_len_(const char *strtab, int tabsize, const char *s,
int slen)
{
- const char *last = strtab + tabsize - (slen + 1);
- const char *p;
+ const char *last;
+ const char *p = strtab;
- for (p = strtab; p <= last; p++)
+ if (tabsize < slen + 1)
+ return NULL;
+ if (slen == 0)
+ return memchr(strtab, '\0', tabsize);
+
+ last = strtab + tabsize - (slen + 1);
+ while (p <= last && (p = memchr(p, s[0], last - p + 1)) != NULL) {
if (memcmp(p, s, slen) == 0 && p[slen] == '\0')
return p;
+ p++;
+ }
return NULL;
}
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (7 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When a base device tree is compiled with `dtc -L` (`--local-fixups`), it
contains a root `/__local_fixups__` node recording the byte offsets of
all phandle references in its properties (for example, so an OS kernel
can identify phandle cells at runtime, so a composite `.dtb` built by
`fdtoverlay` such as Linux's `foo-dtbs := base.dtb bar.dtbo` retains
complete phandle metadata for tools like `fdtfactor`, or so a merged
tree matches what `dtc -L` produces).
`fdt_overlay_apply()` updates the base tree's `/__symbols__` via
`overlay_symbol_update()`, but leaves the base tree's
`/__local_fixups__` untouched. As a result, applying an overlay to a
base tree with `/__local_fixups__` leaves `/__local_fixups__` missing
the phandle references introduced by the overlay (both from the
overlay's `/__local_fixups__` and from its resolved `/__fixups__`), and
leaves stale offsets if the overlay overwrites a base property that
previously had phandle references.
Add `overlay_local_fixups_update()`, controlled by a 3-state
compile-time `FDT_OVERLAY_LOCAL_FIXUPS` setting in
`libfdt/libfdt_internal.h`:
- `0` (`FDT_OVERLAY_LOCAL_FIXUPS_IGNORE`): Leave `/__local_fixups__`
untouched. This is the `#ifndef` fallback in `libfdt_internal.h` so
that size-sensitive embedded environments (such as U-Boot SPL) that
vendor `libfdt` directly get zero code-size increase and no behavior
change unless they opt in.
- `1` (`FDT_OVERLAY_LOCAL_FIXUPS_STRIP`): Delete `/__local_fixups__`
from the base tree if present (~50 bytes of code), avoiding leaving a
stale/incomplete `/__local_fixups__` node in the merged tree.
- `2` (`FDT_OVERLAY_LOCAL_FIXUPS_UPDATE`): Fully update the base tree's
`/__local_fixups__` node when present (clearing entries for
overwritten properties, merging offsets from the overlay's
`/__local_fixups__` and `/__fixups__` in sorted ascending order, and
pruning any subnodes that become empty). This is enabled by default in
`dtc`'s Meson (`-Doverlay-local-fixups=2`) and Make
(`OVERLAY_LOCAL_FIXUPS=2`) builds for desktop tools and tests.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Split out internal helper fdt_add_empty_subnode_() so
overlay_fragment_local_fixups() avoids passing an in-blob pointer to
fdt_add_subnode_namelen() or casting fdt_get_name() (Simon Glass)
- Add test cases in tests/overlay.c where the overlay target has no
existing subnode under /__local_fixups__, with /__local_fixups__ both
before and after the target in the blob (Simon Glass)
- Pass OVERLAY_LOCAL_FIXUPS to tests/run_tests.sh, adjust
fdtoverlay_tests for all three modes, and use 4-space indentation
(Simon Glass, David Gibson)
Changes in v3:
- Move constifying fdto in overlay_apply_node() into a standalone
preparatory patch (David Gibson)
- Gate `/__local_fixups__` handling behind a 3-state compile-time
`FDT_OVERLAY_LOCAL_FIXUPS` flag (`0=ignore`, `1=strip`, `2=update`),
defaulting to `0` (`IGNORE`) in `libfdt_internal.h` for embedded
builds and `2` (`UPDATE`) in Meson/Make desktop builds (Simon Glass,
David Gibson)
- Rewrite commit message in the present tense with motivation up front,
and update `fdt_overlay_apply()` documentation in `libfdt/libfdt.h`
(Simon Glass)
- Require a strict relative path in `overlay_create_path()` and return
`-FDT_ERR_BADOVERLAY` on empty path components or trailing `'/'`
(David Gibson)
- Eliminate the 256-byte `namebuf` limit in
`overlay_fragment_local_fixups()` by re-reading the node name from the
shifted ancestor offset after `fdt_add_subnode_namelen()` (David
Gibson, Simon Glass)
- Return `-FDT_ERR_BADVALUE` instead of `-FDT_ERR_BADSTRUCTURE` in
`overlay_add_local_fixup_offset()` when an existing
`/__local_fixups__` property length is not a multiple of 4 (David
Gibson, Simon Glass)
- Drop redundant `overlay_validate_local_fixups()` pre-pass and inline
single-use helpers (Simon Glass)
Changes in v2:
- New patch updating base /__local_fixups__ in libfdt
fdt_overlay_apply() instead of stripping /__local_fixups__ by default
(David Gibson)
Makefile | 6 +-
libfdt/fdt_overlay.c | 525 ++++++++++++++++++++++++++++++++++++++-
libfdt/fdt_rw.c | 37 ++-
libfdt/libfdt.h | 7 +
libfdt/libfdt_internal.h | 23 ++
meson.build | 5 +
meson_options.txt | 2 +
tests/Makefile.tests | 6 +-
tests/meson.build | 1 +
tests/overlay.c | 154 ++++++++++++
tests/run_tests.sh | 32 +++
11 files changed, 780 insertions(+), 18 deletions(-)
diff --git a/Makefile b/Makefile
index 9110d17796b0..1560a5609fea 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,11 @@ CONFIG_LOCALVERSION =
# See libfdt_internal.h for details
ASSUME_MASK ?= 0
-CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK)
+# Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)
+# See libfdt_internal.h for details
+OVERLAY_LOCAL_FIXUPS ?= 2
+
+CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK) -DFDT_OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS)
WARNINGS = -Wall -Wpointer-arith -Wcast-qual -Wnested-externs -Wsign-compare \
-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow \
-Wwrite-strings
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 4b3e17bd087e..fb76fbafa661 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -1019,9 +1019,8 @@ static int overlay_split_fragment_path(const char *path, int path_len,
* overlay_symbol_update() updates the symbols of the base tree with the
* symbols of the applied overlay
*
- * This is the last step in the device tree overlay application
- * process, allowing the reference of overlay symbols by subsequent
- * overlay operations.
+ * This step in the device tree overlay application process allows
+ * the reference of overlay symbols by subsequent overlay operations.
*
* returns:
* 0 on success
@@ -1135,6 +1134,522 @@ static int overlay_symbol_update(void *fdt, void *fdto)
return 0;
}
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+/**
+ * overlay_create_path - Create a relative path under a parent node
+ * @fdt: Base Device Tree blob
+ * @parent: Node offset of the starting parent node
+ * @path: Relative path to traverse (must not start or end with '/' or contain "//")
+ * @path_len: Length of @path in bytes
+ *
+ * overlay_create_path() walks the slash-separated path components in @path
+ * starting at @parent, creating any missing intermediate subnodes along the
+ * way.
+ *
+ * returns:
+ * Node offset of the resolved or created subnode on success
+ * Negative error code on failure
+ */
+static int overlay_create_path(void *fdt, int parent, const char *path,
+ int path_len)
+{
+ int node = parent;
+
+ while (path_len > 0) {
+ const char *sep = memchr(path, '/', path_len);
+ int comp_len = sep ? (sep - path) : path_len;
+ int next;
+
+ if (!comp_len)
+ return -FDT_ERR_BADOVERLAY;
+
+ next = fdt_subnode_offset_namelen(fdt, node, path, comp_len);
+ if (next == -FDT_ERR_NOTFOUND)
+ next = fdt_add_subnode_namelen(fdt, node, path,
+ comp_len);
+ if (next < 0)
+ return next;
+ node = next;
+
+ if (!sep)
+ break;
+ path += comp_len + 1;
+ path_len -= comp_len + 1;
+ if (!path_len)
+ return -FDT_ERR_BADOVERLAY;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_fragment_local_fixups - Find or create a fragment's target under /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @create: Whether to create missing subnodes under /__local_fixups__
+ *
+ * overlay_fragment_local_fixups() resolves the target of @fragment in @fdt and
+ * locates (or creates, when @create is true) the corresponding node at the
+ * same path under @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * Node offset of the corresponding /__local_fixups__ node on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_local_fixups(void *fdt, const void *fdto,
+ int fragment, bool create)
+{
+ int root_lf, target, cur_fdt, node;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ target = fdt_overlay_target_offset(fdt, fdto, fragment, NULL);
+ if (target < 0)
+ return target;
+
+ cur_fdt = 0;
+ node = root_lf;
+ while (cur_fdt != target) {
+ int subnode, ancestor = -FDT_ERR_INTERNAL;
+ const char *name;
+ int namelen, next_lf;
+
+ fdt_for_each_subnode(subnode, fdt, cur_fdt) {
+ if (subnode <= target)
+ ancestor = subnode;
+ else
+ break;
+ }
+ if (ancestor < 0)
+ return ancestor;
+
+ name = fdt_get_name(fdt, ancestor, &namelen);
+ if (!name)
+ return namelen;
+
+ next_lf = fdt_subnode_offset_namelen(fdt, node, name, namelen);
+ if (next_lf == -FDT_ERR_NOTFOUND && create) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+ char *new_name;
+
+ next_lf = fdt_add_empty_subnode_(fdt, node, namelen,
+ &new_name);
+ if (next_lf < 0)
+ return next_lf;
+ if (next_lf <= ancestor) {
+ int delta = fdt_size_dt_struct(fdt) -
+ old_struct_size;
+
+ target += delta;
+ ancestor += delta;
+ name = fdt_get_name(fdt, ancestor, NULL);
+ }
+ memcpy(new_name, name, namelen);
+ } else if (next_lf < 0) {
+ return next_lf;
+ }
+
+ cur_fdt = ancestor;
+ node = next_lf;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_clear_overwritten_local_fixups - Remove local fixups for overwritten properties
+ * @fdt: Base Device Tree blob
+ * @target_lf: Node offset in @fdt's /__local_fixups__ subtree
+ * @fdto: Device tree overlay blob
+ * @overlay: Node offset in @fdto's __overlay__ subtree
+ *
+ * overlay_clear_overwritten_local_fixups() recursively deletes properties under
+ * @target_lf that correspond to properties overwritten by @overlay in @fdto.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_clear_overwritten_local_fixups(void *fdt, int target_lf,
+ const void *fdto, int overlay)
+{
+ int prop, subnode;
+
+ fdt_for_each_property_offset(prop, fdto, overlay) {
+ const char *name;
+ const void *val;
+ int len, ret;
+
+ val = fdt_getprop_by_offset(fdto, prop, &name, &len);
+ if (!val)
+ return len;
+
+ ret = fdt_delprop(fdt, target_lf, name);
+ if (ret && ret != -FDT_ERR_NOTFOUND)
+ return ret;
+ }
+
+ fdt_for_each_subnode(subnode, fdto, overlay) {
+ const char *name;
+ int namelen, sub_lf, ret;
+
+ name = fdt_get_name(fdto, subnode, &namelen);
+ if (!name)
+ return namelen;
+
+ sub_lf = fdt_subnode_offset_namelen(fdt, target_lf, name,
+ namelen);
+ if (sub_lf == -FDT_ERR_NOTFOUND)
+ continue;
+ if (sub_lf < 0)
+ return sub_lf;
+
+ ret = overlay_clear_overwritten_local_fixups(fdt, sub_lf,
+ fdto, subnode);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_add_local_fixup_offset - Insert a sorted u32 offset into a local fixup property
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ * @name: Property name to update
+ * @name_len: Length of @name in bytes
+ * @poffset: Byte offset of the phandle reference within the target property
+ *
+ * overlay_add_local_fixup_offset() inserts @poffset in ascending sorted order
+ * (ignoring duplicates) into the u32 array property @name on @lf_node.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_add_local_fixup_offset(void *fdt, int lf_node,
+ const char *name, int name_len,
+ uint32_t poffset)
+{
+ const fdt32_t *old_val;
+ fdt32_t *vals;
+ void *prop_data;
+ int old_len, count, i, idx, ret;
+
+ old_val = fdt_getprop_namelen(fdt, lf_node, name, name_len, &old_len);
+ if (!old_val && old_len != -FDT_ERR_NOTFOUND)
+ return old_len;
+
+ if (!old_val) {
+ old_len = 0;
+ count = 0;
+ idx = 0;
+ } else {
+ if (old_len % sizeof(uint32_t))
+ return -FDT_ERR_BADVALUE;
+ count = old_len / sizeof(uint32_t);
+ idx = count;
+ for (i = 0; i < count; i++) {
+ uint32_t cur = fdt32_ld(old_val + i);
+
+ if (cur == poffset)
+ return 0;
+ if (cur > poffset) {
+ idx = i;
+ break;
+ }
+ }
+ }
+
+ ret = fdt_setprop_placeholder_namelen(fdt, lf_node, name, name_len,
+ old_len + sizeof(uint32_t),
+ &prop_data);
+ if (ret)
+ return ret;
+
+ vals = prop_data;
+ if (idx < count)
+ memmove(vals + idx + 1, vals + idx,
+ (count - idx) * sizeof(uint32_t));
+ fdt32_st(vals + idx, poffset);
+
+ return 0;
+}
+
+/**
+ * overlay_merge_fixups_into_local_fixups - Record resolved /__fixups__ for a fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the current fragment in @fdto
+ * @target_frag_name: Name of @fragment
+ * @target_frag_len: Length of @target_frag_name in bytes
+ * @target_lfp: In/out cached node offset of @fragment under @fdt's /__local_fixups__
+ *
+ * overlay_merge_fixups_into_local_fixups() parses external phandle references
+ * belonging to @fragment in @fdto's /__fixups__ node and records their resolved
+ * property byte offsets in @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_merge_fixups_into_local_fixups(void *fdt, const void *fdto,
+ int fragment,
+ const char *target_frag_name,
+ int target_frag_len,
+ int *target_lfp)
+{
+ int fixups_off, prop;
+
+ fixups_off = fdt_subnode_offset(fdto, 0, "__fixups__");
+ if (fixups_off == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (fixups_off < 0)
+ return fixups_off;
+
+ fdt_for_each_property_offset(prop, fdto, fixups_off) {
+ const char *value, *label;
+ int len;
+
+ value = fdt_getprop_by_offset(fdto, prop, &label, &len);
+ if (!value)
+ return len;
+
+ do {
+ const char *path, *name, *frag_name, *rel_path;
+ uint32_t path_len, name_len, poffset;
+ int frag_name_len, rel_path_len, lf_node, ret;
+
+ ret = overlay_parse_fixup_entry(&value, &len, &path,
+ &path_len, &name,
+ &name_len, &poffset);
+ if (ret)
+ return ret;
+
+ ret = overlay_split_fragment_path(path, path_len,
+ &frag_name,
+ &frag_name_len,
+ &rel_path,
+ &rel_path_len);
+ if (ret < 0)
+ return ret;
+ if (!ret)
+ continue;
+
+ if (frag_name_len != target_frag_len ||
+ memcmp(frag_name, target_frag_name,
+ frag_name_len) != 0)
+ continue;
+
+ if (*target_lfp < 0) {
+ *target_lfp = overlay_fragment_local_fixups(
+ fdt, fdto, fragment, true);
+ if (*target_lfp < 0)
+ return *target_lfp;
+ }
+
+ lf_node = overlay_create_path(fdt, *target_lfp,
+ rel_path, rel_path_len);
+ if (lf_node < 0)
+ return lf_node;
+
+ ret = overlay_add_local_fixup_offset(fdt, lf_node,
+ name, name_len,
+ poffset);
+ if (ret)
+ return ret;
+ } while (len > 0);
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_prune_empty_local_fixups - Remove empty subnodes from /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ *
+ * overlay_prune_empty_local_fixups() recursively deletes any child subnodes
+ * under @lf_node that have neither properties nor subnodes remaining.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_prune_empty_local_fixups(void *fdt, int lf_node)
+{
+ int subnode = fdt_first_subnode(fdt, lf_node);
+
+ while (subnode >= 0) {
+ int next, ret;
+
+ ret = overlay_prune_empty_local_fixups(fdt, subnode);
+ if (ret < 0)
+ return ret;
+
+ next = fdt_next_subnode(fdt, subnode);
+ if (fdt_first_property_offset(fdt, subnode) == -FDT_ERR_NOTFOUND &&
+ fdt_first_subnode(fdt, subnode) == -FDT_ERR_NOTFOUND) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+
+ ret = fdt_del_node(fdt, subnode);
+ if (ret < 0)
+ return ret;
+ if (next >= 0)
+ next += fdt_size_dt_struct(fdt) -
+ old_struct_size;
+ }
+ subnode = next;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_fragment_update_local_fixups - Update base /__local_fixups__ for one fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @ov_lf_root: Node offset of /__local_fixups__ in @fdto (or -FDT_ERR_NOTFOUND)
+ *
+ * overlay_fragment_update_local_fixups() clears any base /__local_fixups__
+ * entries overwritten by @fragment, then merges phandle offsets for @fragment
+ * from @fdto's /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_update_local_fixups(void *fdt, const void *fdto,
+ int fragment, int ov_lf_root)
+{
+ const char *frag_name;
+ int frag_name_len, overlay, target_lf, lf_frag, lf_overlay, ret;
+
+ overlay = fdt_subnode_offset(fdto, fragment, "__overlay__");
+ if (overlay == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (overlay < 0)
+ return overlay;
+
+ frag_name = fdt_get_name(fdto, fragment, &frag_name_len);
+ if (!frag_name)
+ return frag_name_len;
+
+ target_lf = overlay_fragment_local_fixups(fdt, fdto, fragment, false);
+ if (target_lf < 0 && target_lf != -FDT_ERR_NOTFOUND)
+ return target_lf;
+
+ if (target_lf >= 0) {
+ ret = overlay_clear_overwritten_local_fixups(fdt, target_lf,
+ fdto, overlay);
+ if (ret)
+ return ret;
+ }
+
+ lf_overlay = -FDT_ERR_NOTFOUND;
+ if (ov_lf_root >= 0) {
+ lf_frag = fdt_subnode_offset_namelen(fdto, ov_lf_root,
+ frag_name, frag_name_len);
+ if (lf_frag >= 0)
+ lf_overlay = fdt_subnode_offset(fdto, lf_frag,
+ "__overlay__");
+ else if (lf_frag != -FDT_ERR_NOTFOUND)
+ return lf_frag;
+ }
+ if (lf_overlay < 0 && lf_overlay != -FDT_ERR_NOTFOUND)
+ return lf_overlay;
+
+ if (lf_overlay >= 0) {
+ if (target_lf < 0)
+ target_lf = overlay_fragment_local_fixups(fdt, fdto,
+ fragment,
+ true);
+ if (target_lf < 0)
+ return target_lf;
+
+ ret = overlay_apply_node(fdt, target_lf, fdto, lf_overlay);
+ if (ret)
+ return ret;
+ }
+
+ return overlay_merge_fixups_into_local_fixups(
+ fdt, fdto, fragment, frag_name, frag_name_len, &target_lf);
+}
+
+/**
+ * overlay_local_fixups_update - Update base /__local_fixups__ with overlay fixups
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ *
+ * overlay_local_fixups_update() updates the base device tree's
+ * /__local_fixups__ subtree (when present) by walking each overlay fragment in
+ * application order, clearing entries for properties overwritten by that
+ * fragment, and merging phandle reference offsets from @fdto's
+ * /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * This is part of the device tree overlay application process, when you want
+ * the merged device tree to retain complete local phandle fixup metadata.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf, ov_lf_root, fragment, ret;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ ov_lf_root = fdt_subnode_offset(fdto, 0, "__local_fixups__");
+ if (ov_lf_root < 0 && ov_lf_root != -FDT_ERR_NOTFOUND)
+ return ov_lf_root;
+
+ /*
+ * Process fragments in application order so that if a later fragment in
+ * fdto overwrites a property modified by an earlier fragment, the
+ * earlier fragment's local fixups for that property are cleared first.
+ */
+ fdt_for_each_subnode(fragment, fdto, 0) {
+ ret = overlay_fragment_update_local_fixups(fdt, fdto, fragment,
+ ov_lf_root);
+ if (ret)
+ return ret;
+ }
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ return overlay_prune_empty_local_fixups(fdt, root_lf);
+}
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ return fdt_del_node(fdt, root_lf);
+}
+#else
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ return 0;
+}
+#endif
+
int fdt_overlay_apply(void *fdt, void *fdto)
{
uint32_t delta;
@@ -1175,6 +1690,10 @@ int fdt_overlay_apply(void *fdt, void *fdto)
if (ret)
goto err;
+ ret = overlay_local_fixups_update(fdt, fdto);
+ if (ret)
+ goto err;
+
/*
* The overlay has been damaged, erase its magic.
*/
diff --git a/libfdt/fdt_rw.c b/libfdt/fdt_rw.c
index 850aafe44d27..99048f135cb5 100644
--- a/libfdt/fdt_rw.c
+++ b/libfdt/fdt_rw.c
@@ -365,8 +365,8 @@ int fdt_delprop(void *fdt, int nodeoffset, const char *name)
return fdt_splice_struct_(fdt, prop, proplen, 0);
}
-int fdt_add_subnode_namelen(void *fdt, int parentoffset,
- const char *name, int namelen)
+int fdt_add_empty_subnode_(void *fdt, int parentoffset, int namelen,
+ char **namep)
{
struct fdt_node_header *nh;
int offset, nextoffset;
@@ -375,14 +375,6 @@ int fdt_add_subnode_namelen(void *fdt, int parentoffset,
uint32_t tag;
fdt32_t *endtag;
- FDT_RW_PROBE(fdt);
-
- offset = fdt_subnode_offset_namelen(fdt, parentoffset, name, namelen);
- if (offset >= 0)
- return -FDT_ERR_EXISTS;
- else if (offset != -FDT_ERR_NOTFOUND)
- return offset;
-
/* Try to place the new node after the parent's properties */
tag = fdt_next_tag(fdt, parentoffset, &nextoffset);
/* the fdt_subnode_offset_namelen() should ensure this never hits */
@@ -402,9 +394,32 @@ int fdt_add_subnode_namelen(void *fdt, int parentoffset,
nh->tag = cpu_to_fdt32(FDT_BEGIN_NODE);
memset(nh->name, 0, FDT_TAGALIGN(namelen+1));
- memcpy(nh->name, name, namelen);
endtag = (fdt32_t *)((char *)nh + nodelen - FDT_TAGSIZE);
*endtag = cpu_to_fdt32(FDT_END_NODE);
+ *namep = nh->name;
+
+ return offset;
+}
+
+int fdt_add_subnode_namelen(void *fdt, int parentoffset,
+ const char *name, int namelen)
+{
+ char *namep;
+ int offset;
+
+ FDT_RW_PROBE(fdt);
+
+ offset = fdt_subnode_offset_namelen(fdt, parentoffset, name, namelen);
+ if (offset >= 0)
+ return -FDT_ERR_EXISTS;
+ else if (offset != -FDT_ERR_NOTFOUND)
+ return offset;
+
+ offset = fdt_add_empty_subnode_(fdt, parentoffset, namelen, &namep);
+ if (offset < 0)
+ return offset;
+
+ memcpy(namep, name, namelen);
return offset;
}
diff --git a/libfdt/libfdt.h b/libfdt/libfdt.h
index c69a18edf09f..c2f47a7d3e87 100644
--- a/libfdt/libfdt.h
+++ b/libfdt/libfdt.h
@@ -2472,6 +2472,13 @@ int fdt_del_node(void *fdt, int nodeoffset);
* fdt_overlay_apply() will apply the given device tree overlay on the
* given base device tree.
*
+ * If the base device tree contains a /__symbols__ node, it is updated with
+ * any symbols from the overlay. If the base device tree contains a
+ * /__local_fixups__ node, how it is handled depends on the build-time
+ * FDT_OVERLAY_LOCAL_FIXUPS setting (see libfdt_internal.h): left untouched
+ * (IGNORE), deleted (STRIP), or updated to reflect phandle references in the
+ * merged tree (UPDATE, the default when built via dtc's build system).
+ *
* Expect the base device tree to be modified, even if the function
* returns an error.
*
diff --git a/libfdt/libfdt_internal.h b/libfdt/libfdt_internal.h
index eefea2a3ccf3..315a7022fc32 100644
--- a/libfdt/libfdt_internal.h
+++ b/libfdt/libfdt_internal.h
@@ -32,6 +32,8 @@ static inline const char *fdt_find_string_(const char *strtab, int tabsize,
}
int fdt_node_end_offset_(void *fdt, int nodeoffset);
+int fdt_add_empty_subnode_(void *fdt, int parentoffset, int namelen,
+ char **namep);
static inline const void *fdt_offset_ptr_(const void *fdt, int offset)
{
@@ -99,6 +101,27 @@ static inline uint64_t fdt64_ld_(const fdt64_t *p)
#define FDT_PARENT_MAX_DEPTH 32
#endif
+/*
+ * Controls how fdt_overlay_apply() handles a /__local_fixups__ node in the
+ * base device tree:
+ * 0 (FDT_OVERLAY_LOCAL_FIXUPS_IGNORE): Leave /__local_fixups__ untouched
+ * (zero extra code size and preserves historical behavior; default when
+ * building libfdt standalone without dtc's build system).
+ * 1 (FDT_OVERLAY_LOCAL_FIXUPS_STRIP): Delete /__local_fixups__ from the
+ * base device tree so a stale/incomplete /__local_fixups__ node is not
+ * left behind (~50 bytes of code).
+ * 2 (FDT_OVERLAY_LOCAL_FIXUPS_UPDATE): Fully update /__local_fixups__ in
+ * the base device tree with the overlay's local and resolved external
+ * phandle fixups (default when building via dtc's Meson/Make build).
+ */
+#define FDT_OVERLAY_LOCAL_FIXUPS_IGNORE 0
+#define FDT_OVERLAY_LOCAL_FIXUPS_STRIP 1
+#define FDT_OVERLAY_LOCAL_FIXUPS_UPDATE 2
+
+#ifndef FDT_OVERLAY_LOCAL_FIXUPS
+#define FDT_OVERLAY_LOCAL_FIXUPS FDT_OVERLAY_LOCAL_FIXUPS_IGNORE
+#endif
+
/*
* Defines assumptions which can be enabled. Each of these can be enabled
* individually. For maximum safety, don't enable any assumptions!
diff --git a/meson.build b/meson.build
index 520cd35f9075..348d9b555e34 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,11 @@ add_project_arguments(
language: 'c'
)
+add_project_arguments(
+ '-DFDT_OVERLAY_LOCAL_FIXUPS=' + get_option('overlay-local-fixups').to_string(),
+ language: 'c'
+)
+
yamltree = 'yamltree.c'
yaml = dependency('yaml-0.1', version: '>=0.2.3', required: get_option('yaml'))
if not yaml.found()
diff --git a/meson_options.txt b/meson_options.txt
index a866b172e802..ac8ea4aa2a9f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -2,6 +2,8 @@ option('tools', type: 'boolean', value: true,
description: 'Build tools')
option('assume-mask', type: 'integer', value: 0,
description: 'Control the assumptions made (e.g. risking security issues) in the code.')
+option('overlay-local-fixups', type: 'integer', value: 2,
+ description: 'Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)')
option('yaml', type: 'feature', value: 'auto',
description: 'YAML support')
option('valgrind', type: 'feature', value: 'auto',
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index af5d309cf8a9..623d71786701 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -81,18 +81,18 @@ tests_clean:
rm -rf $(TESTS_CLEANDIRS)
check: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh
ifeq ($(NO_VALGRIND),1)
checkm:
@echo "make checkm requires valgrind, but NO_VALGRIND=1"
else
checkm: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh -m
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh -m
endif
checkv: tests ${TESTS_BIN} $(TESTS_PYLIBFDT)
- cd $(TESTS_PREFIX); ./run_tests.sh -v
+ cd $(TESTS_PREFIX); OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS) ./run_tests.sh -v
ifneq ($(DEPTARGETS),)
-include $(TESTS_DEPFILES)
diff --git a/tests/meson.build b/tests/meson.build
index e37567864e04..9f0f51013530 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -128,6 +128,7 @@ if not yaml.found()
else
env += 'NO_YAML=0'
endif
+env += 'OVERLAY_LOCAL_FIXUPS=' + get_option('overlay-local-fixups').to_string()
run_test_types = [
'libfdt',
diff --git a/tests/overlay.c b/tests/overlay.c
index 2d27918a3369..d2cc0af0178c 100644
--- a/tests/overlay.c
+++ b/tests/overlay.c
@@ -9,6 +9,7 @@
#include <stdio.h>
#include <libfdt.h>
+#include "libfdt_internal.h"
#include "tests.h"
@@ -191,6 +192,156 @@ static void *open_dt(char *path)
return copy;
}
+static void fdt_overlay_local_fixups(char *base_path, char *overlay_path)
+{
+ void *fdt_base = open_dt(base_path);
+ void *fdt_overlay = open_dt(overlay_path);
+ int lf, tn, stn;
+
+ /*
+ * Add a /__local_fixups__ subtree to the base tree before applying
+ * the overlay:
+ * - /__local_fixups__/test-node: test-int-property = <0> (overwritten
+ * by the overlay with a non-phandle value)
+ * - /__local_fixups__/test-node/sub-test-node:
+ * new-sub-test-property = <0> (overwritten by the overlay, leaving
+ * sub-test-node empty so it should be pruned in UPDATE mode)
+ */
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_add_subnode(fdt_base, lf, "test-node");
+ CHECK(tn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, tn, "test-int-property", 0));
+ stn = fdt_add_subnode(fdt_base, tn, "sub-test-node");
+ CHECK(stn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, stn, "new-sub-test-property", 0));
+
+ CHECK(fdt_overlay_apply(fdt_base, fdt_overlay));
+
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val) !=
+ -FDT_ERR_NOTFOUND);
+ CHECK(fdt_path_offset(
+ fdt_base,
+ "/__local_fixups__/test-node/sub-test-node") !=
+ -FDT_ERR_NOTFOUND);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 1, &val));
+ CHECK(val != 4);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 1, &val));
+ CHECK(val != 4);
+ }
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") !=
+ -FDT_ERR_NOTFOUND);
+#else
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val) != -FDT_ERR_NOTFOUND);
+ }
+#endif
+
+ free(fdt_base);
+ free(fdt_overlay);
+}
+
+static void fdt_overlay_local_fixups_create(char *base_path, char *overlay_path,
+ bool lf_after_target)
+{
+ void *fdt_base = open_dt(base_path);
+ void *fdt_overlay = open_dt(overlay_path);
+ int lf, tn;
+
+ /*
+ * Create an empty /__local_fixups__ node (with no test-node child)
+ * either before or after /test-node in the blob, so that
+ * overlay_fragment_local_fixups() must create /__local_fixups__/test-node
+ * during fdt_overlay_apply().
+ */
+ if (lf_after_target) {
+ uint32_t ph;
+
+ tn = fdt_path_offset(fdt_base, "/test-node");
+ CHECK(tn < 0);
+ ph = fdt_get_phandle(fdt_base, tn);
+ CHECK(!ph);
+ CHECK(fdt_del_node(fdt_base, tn));
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_add_subnode(fdt_base, 0, "test-node");
+ CHECK(tn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, tn, "phandle", ph));
+ lf = fdt_path_offset(fdt_base, "/__local_fixups__");
+ CHECK(lf <= tn);
+ } else {
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_path_offset(fdt_base, "/test-node");
+ CHECK(tn < 0);
+ CHECK(lf >= tn);
+ }
+
+ CHECK(fdt_overlay_apply(fdt_base, fdt_overlay));
+
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 1, &val));
+ CHECK(val != 4);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 1, &val));
+ CHECK(val != 4);
+ }
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") !=
+ -FDT_ERR_NOTFOUND);
+#else
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") < 0);
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__/test-node") !=
+ -FDT_ERR_NOTFOUND);
+#endif
+
+ free(fdt_base);
+ free(fdt_overlay);
+}
+
int main(int argc, char *argv[])
{
void *fdt_base, *fdt_overlay;
@@ -219,6 +370,9 @@ int main(int argc, char *argv[])
if (fdt_path_offset(fdt_base, "/__symbols__") >= 0) {
fdt_overlay_local_phandle(fdt_base);
fdt_overlay_local_phandles(fdt_base);
+ fdt_overlay_local_fixups(argv[1], argv[2]);
+ fdt_overlay_local_fixups_create(argv[1], argv[2], false);
+ fdt_overlay_local_fixups_create(argv[1], argv[2], true);
}
PASS();
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 3cacb68438df..e16767d3f3a2 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -11,6 +11,10 @@ if [ -z "$PYTHON" ]; then
PYTHON=python3
fi
+if [ -z "$OVERLAY_LOCAL_FIXUPS" ]; then
+ OVERLAY_LOCAL_FIXUPS=2
+fi
+
if [ -n "$NO_PYTHON" ]; then
if [ "$NO_PYTHON" != "0" ]; then
no_python=true
@@ -1147,6 +1151,34 @@ fdtoverlay_tests() {
shorten_echo "check phandle to noda d wasn't overwritten: "
run_wrap_test test "$bd" = "$pd"
+
+ # verify how /__local_fixups__ in a -L base blob is handled when applying an overlay
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_phandle_lf.test.dtb "$SRCDIR/overlay_base_phandle.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_phandle_lf.test.dtb -o overlay_base_phandle_lfO.test.dtb overlay_overlay_phandle.test.dtb
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_lf.test.dtb "$SRCDIR/overlay_base.dts"
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb -cp /__local_fixups__/test-node/sub-test-node
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb /__local_fixups__/test-node/sub-test-node new-sub-test-property -tu 0
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay.fdtoverlay.test.dtb "$SRCDIR/overlay_overlay.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_lf.test.dtb -o overlay_base_lfO.test.dtb overlay_overlay.fdtoverlay.test.dtb
+ if [ "$OVERLAY_LOCAL_FIXUPS" -eq 2 ]; then
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b d
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/c b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d c
+
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-phandle
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-several-phandle
+ run_test check_path overlay_base_lfO.test.dtb not-exists "/__local_fixups__/test-node/sub-test-node"
+ elif [ "$OVERLAY_LOCAL_FIXUPS" -eq 1 ]; then
+ run_test check_path overlay_base_phandle_lfO.test.dtb not-exists "/__local_fixups__"
+ run_test check_path overlay_base_lfO.test.dtb not-exists "/__local_fixups__"
+ else
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b a
+ run_wrap_error_test $DTGET overlay_base_phandle_lfO.test.dtb /__local_fixups__/b d
+ run_fdtget_test 0 overlay_base_lfO.test.dtb /__local_fixups__/test-node/sub-test-node new-sub-test-property
+ fi
}
run_python_tests () {
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (8 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When shipping a single firmware or kernel image that supports many
closely related board variants or SKUs, bundling a full compiled .dtb
for every variant results in substantial duplication.
Add `fdtfactor`, a Python tool built on `pylibfdt` and `libpydt` that
takes a set of compiled device tree blobs (.dtb) compiled with `-@ -L`
(`/__symbols__` and `/__local_fixups__`) and splits them into a single
common `base.dtb` and per-board `.dtbo` overlays:
- Common nodes present across all input DTBs are placed in `base.dtb`,
with each common property assigned the value shared by at least two
boards that maximizes byte savings across overlays.
- Using `/__local_fixups__`, phandle references inside property values
are normalized to target node paths so common properties referencing
common nodes match even when numeric phandles differ across input
DTBs.
- Any common node that carries a phandle in any input DTB receives its
phandle in `base.dtb` so overlays only emit `phandle` properties on
newly added subnodes, matching the requirements of `libfdt`, Linux's
`drivers/of/overlay.c`, and coreboot's `dt_apply_overlay()`.
Add a `fdtfactor` test suite in `tests/run_tests.sh` verifying that
applying each generated `.dtbo` to `base.dtb` with `fdtoverlay`
reconstructs the original DTB (verified via `dtbs_equal_unordered`,
including across input DTBs with shifted phandle numbers).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Rename -o/--outdir to -o/--output so standalone DTB mode and FIT mode
share the same output option, and use 4-space indentation in
tests/run_tests.sh (David Gibson)
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Replace ParsedDtb.raw_bytes with uncompressed_size, use early continue
to reduce loop nesting, drop redundant set() before sorted() on local
fixup offsets, and clarify pre-order walk and phandle comments (David
Gibson)
- Allow dtc -@ -L inputs with no phandles (where dtc omits /__symbols__
and /__local_fixups__)
- Check _is_dtbo_overlay() on the unflattened DeviceTree, index
ParsedDtb.nodes and base_nodes by path, and skip unchanged base nodes
in _build_board_dtbo() to avoid repeated O(depth) traversals and
per-property comparisons
- Handle OVERLAY_LOCAL_FIXUPS != 2 in fdtfactor_tests (Simon Glass)
Changes in v3:
- Update fdtfactor to use devicetree.DeviceTree and a FactoredTree
subclass initializing self.phandle and self.fixups in __init__()
(David Gibson, Simon Glass)
- Rename -b/--base-dtb to --output-base-dtb to avoid confusion with an
input base DTB (David Gibson)
- Move the test rejecting input DTBs lacking /__local_fixups__ from the
-S patch into this base patch (Simon Glass)
Changes in v2:
- Preserve /__local_fixups__ on base.dtb by default (David Gibson)
- Store parsed trees directly in NodeTree with node.phandle and
node.fixups attributes instead of parallel dicts (David Gibson)
- Extract phandles in a separate pass after unflattening and reject
legacy linux,phandle properties (David Gibson)
- Exclude non-root common phandle nodes lacking a common symbol from
base.dtb instead of synthesizing __fdtfactor_ph_* symbols (David
Gibson)
- Drop -q/--quiet, rename -v/--verbose to -s/--stats, and move tests to
a dedicated fdtfactor test suite (David Gibson)
Makefile | 2 +-
README.md | 43 ++
fdtfactor | 881 +++++++++++++++++++++++++++++
meson.build | 1 +
tests/Makefile.tests | 4 +-
tests/fdtfactor_board1.dts | 72 +++
tests/fdtfactor_board2.dts | 70 +++
tests/fdtfactor_board3.dts | 68 +++
tests/fdtfactor_board3_shifted.dts | 68 +++
tests/meson.build | 2 +-
tests/run_tests.sh | 110 +++-
tests/testutils.sh | 1 +
12 files changed, 1315 insertions(+), 7 deletions(-)
create mode 100755 fdtfactor
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
create mode 100644 tests/fdtfactor_board3_shifted.dts
diff --git a/Makefile b/Makefile
index 1560a5609fea..fc06b0ae16e2 100644
--- a/Makefile
+++ b/Makefile
@@ -169,7 +169,7 @@ BIN += fdtget
BIN += fdtput
BIN += fdtoverlay
-SCRIPTS = dtdiff
+SCRIPTS = dtdiff fdtfactor
all: $(BIN) libfdt
diff --git a/README.md b/README.md
index 9d38c7e79fc6..9a8be74d47fe 100644
--- a/README.md
+++ b/README.md
@@ -201,6 +201,49 @@ merged blob to `-o`, `--output`:
$ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
```
+## fdtfactor
+
+`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
+compiled device tree blobs (`.dtb`) and splits them into a single common
+`base.dtb` and per-board `.dtbo` overlays so that applying a board's
+`.dtbo` to `base.dtb` recreates the original board device tree while
+reducing total storage size.
+
+Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
+and `/__local_fixups__` (used to identify phandle references inside
+property values) are present.
+
+Basic usage:
+
+```
+$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+```
+
+Options:
+
+* `--output-base-dtb`: Output filename for the common base `.dtb`
+ (default: `base.dtb`; placed in `--output` if specified, or the current
+ directory otherwise).
+* `-o`, `--output`: Output directory for generated `.dtbo` files and
+ `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
+ current directory for `--output-base-dtb`).
+* `-s`, `--stats`: Print size savings summary.
+
+### TODO
+
+* **Targeted overlay fragments**: Add an option to emit fine-grained
+ `/fragment@N` nodes targeting individual parent subnodes rather than
+ placing all overlay changes under a single `/fragment@0` targeting `/`.
+* **Multi-level overlays**: Support hierarchical overlay layers (for
+ example, SoC base -> board family intermediate overlay -> SKU leaf
+ overlay) when multiple boards share large sub-trees not present across
+ all boards. As an interim step, `fdtfactor` could accept an
+ already-generated `base.dtb` plus a subset of `.dtbo` overlays emitted
+ by an initial pass, leaving `base.dtb` untouched while factoring out
+ an intermediate `.dtbo` for the commonalities shared within that
+ subset.
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
diff --git a/fdtfactor b/fdtfactor
new file mode 100755
index 000000000000..dfbee0aea2d6
--- /dev/null
+++ b/fdtfactor
@@ -0,0 +1,881 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""fdtfactor - Split multiple compiled .dtb files into a base .dtb and .dtbo overlays.
+
+Given two or more input .dtb files compiled with `dtc -@ -L` (so they include
+`/__symbols__` and `/__local_fixups__`), `fdtfactor` extracts all nodes and
+properties shared across the inputs into a single common `base.dtb` and emits a
+small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
+`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
+original board DTB.
+"""
+
+import argparse
+from collections import Counter
+import os
+import struct
+import sys
+
+import libfdt
+import libpydt
+
+
+def _is_dtbo_overlay(root):
+ """Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
+
+ Args:
+ root: Root `libpydt.DeviceTree` instance to inspect.
+
+ Returns:
+ True if `root` contains `/__fixups__` or a top-level subnode with an
+ `__overlay__` child, False otherwise.
+ """
+ return (
+ '__fixups__' in root.children or
+ any('__overlay__' in child.children
+ for child in root.children.values())
+ )
+
+
+class FactoredTree(libpydt.DeviceTree):
+ """DeviceTree subclass with normalized phandle and fixup attributes.
+
+ Attributes:
+ phandle: Optional integer phandle popped from `props['phandle']`.
+ fixups: Dict mapping property name (str) to a tuple of
+ `(byte_offset, target_node_path)` phandle fixup pairs.
+ """
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+ self.phandle = None
+ self.fixups = {}
+
+
+class ParsedDtb:
+ """Complete input .dtb blob with phandle-normalized tree and FDT metadata.
+
+ While `libpydt.DeviceTree` is a generic live-tree representation of a
+ node hierarchy (with `libfdt.Property` instances and unprocessed
+ `/__symbols__`, `/__local_fixups__`, and `phandle` properties), `ParsedDtb`
+ wraps an entire input `.dtb` in a `FactoredTree` (`self.root`) and
+ normalizes it for cross-board comparison:
+
+ - Stores FDT header, memory reservation, and file-level metadata that live
+ outside the root node tree (`boot_cpuid_phys`, `mem_rsv`, `filename`,
+ `uncompressed_size`).
+ - Pops `/__symbols__` (parsed into `self.symbols`) and `/__local_fixups__`
+ out of `self.root`, and indexes the remaining tree in pre-order in
+ `self.nodes` (`path -> FactoredTree`).
+ - Moves each node's `phandle` property out of `node.props` into
+ `node.phandle`, and uses `/__local_fixups__` to zero out numeric phandle
+ cells in `node.props` while recording canonical `(offset, target_path)`
+ tuples in `node.fixups` so properties can be compared across DTBs
+ independent of numeric phandle numbering.
+ """
+
+ def __init__(self, filename):
+ """Parse an input FDT file into a normalized FactoredTree.
+
+ Args:
+ filename: File path of the `.dtb` file to read.
+ """
+ self.filename = filename
+ with open(filename, 'rb') as f:
+ raw_bytes = f.read()
+ self.uncompressed_size = len(raw_bytes)
+
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+
+ self.boot_cpuid_phys = fdt.boot_cpuid_phys()
+ self.mem_rsv = tuple(fdt.get_mem_rsv_list())
+ self.root = FactoredTree.unflatten(fdt)
+
+ if _is_dtbo_overlay(self.root):
+ raise ValueError(
+ f'{filename}: Input is already a DTB overlay (.dtbo)'
+ )
+
+ sym_node = self.root.children.pop('__symbols__', None)
+ lf_root = self.root.children.pop('__local_fixups__', None)
+ self.has_local_fixups = lf_root is not None
+
+ self.nodes = dict(self.root.walk())
+ phandle_to_path = self._extract_phandles()
+ if phandle_to_path and (sym_node is None or lf_root is None):
+ raise ValueError(
+ f'input DTB(s) lack /__symbols__ or /__local_fixups__: '
+ f'{filename} (compile input DTBs with dtc -@ -L)'
+ )
+ self.symbols = (
+ self._parse_symbols(sym_node) if sym_node is not None else {}
+ )
+ if lf_root is not None:
+ self._apply_local_fixups(lf_root, phandle_to_path)
+
+ def _extract_phandles(self):
+ """Move `phandle` properties out of `node.props`.
+
+ Iterates `self.nodes` in pre-order and, for each node:
+ - Pops `phandle` from `node.props` (so phandle properties are not
+ compared as ordinary byte properties across DTBs, since `dtc` may
+ assign different integer phandles to the same node in different
+ boards) and stores the decoded integer in `node.phandle`.
+ - Validates that `node.phandle` is not `0` or `0xffffffff`, and that no
+ two nodes in the DTB share the same phandle.
+
+ Returns:
+ Dict mapping each integer phandle in this DTB to its node path
+ string (`""` for root, `"/soc/..."` for subnodes), used by
+ `_apply_local_fixups()` to resolve numeric phandle cells to
+ canonical target node paths.
+ """
+ phandle_to_path = {}
+ for path, node in self.nodes.items():
+ if 'linux,phandle' in node.props:
+ raise ValueError(
+ f'{self.filename}: {path or "/"}: legacy linux,phandle '
+ f'property is not supported (compile with dtc -H epapr)'
+ )
+ ph_prop = node.props.pop('phandle', None)
+ if ph_prop is not None:
+ node.phandle = ph_prop.as_uint32()
+
+ if node.phandle is None:
+ continue
+ if node.phandle in (0, 0xffffffff):
+ raise ValueError(
+ f'{self.filename}: invalid phandle {node.phandle:#x} '
+ f'on {path or "/"}'
+ )
+ if node.phandle in phandle_to_path:
+ raise ValueError(
+ f'{self.filename}: duplicate phandle {node.phandle:#x} '
+ f'on {phandle_to_path[node.phandle] or "/"} and '
+ f'{path or "/"}'
+ )
+ phandle_to_path[node.phandle] = path
+ return phandle_to_path
+
+ def _parse_symbols(self, sym_node):
+ """Parse the popped `/__symbols__` node into a `label -> path` dict.
+
+ Each property on `/__symbols__` (ignoring any `phandle` property on the
+ `/__symbols__` node itself) maps a Devicetree label name to the full
+ path of the target node in the DTB. Root paths (`"/"`) are normalized to
+ `""` to match `DeviceTree` path conventions, and each target path is
+ verified to exist in `self.nodes`.
+
+ Args:
+ sym_node: The `FactoredTree` for `/__symbols__` popped from
+ `self.root`.
+
+ Returns:
+ Dict mapping symbol label name (str) to normalized target node path
+ (str, `""` for root or `"/..."` for subnodes).
+ """
+ symbols = {}
+ for label, prop in sym_node.props.items():
+ if label == 'phandle':
+ continue
+ raw_path = prop.as_str()
+ if not raw_path.startswith('/') or '//' in raw_path:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} has malformed '
+ f'path {raw_path!r}'
+ )
+ target_path = raw_path.rstrip('/')
+ if target_path not in self.nodes:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} references '
+ f'missing node {target_path or "/"!r}'
+ )
+ symbols[label] = target_path
+ return symbols
+
+ def _apply_local_fixups(self, lf_root, phandle_to_path):
+ """Normalize phandle references in `self.root` using `/__local_fixups__`.
+
+ The `/__local_fixups__` tree mirrors the structure of `self.root`: each
+ property in `/__local_fixups__` contains a list of 32-bit byte offsets
+ where the corresponding property in `self.root` holds a phandle cell.
+ For every such offset, this method:
+ - Reads the 32-bit integer phandle from `target_node.props[prop_name]`
+ and resolves it to a canonical target node path via `phandle_to_path`.
+ - Zeros out the 4-byte phandle cell in `target_node.props[prop_name]` so
+ property values can be compared across DTBs independent of numeric
+ phandle numbering.
+ - Records `((offset, target_path), ...)` in
+ `target_node.fixups[prop_name]`.
+
+ Args:
+ lf_root: The `FactoredTree` for `/__local_fixups__` popped from
+ `self.root`.
+ phandle_to_path: Dict mapping integer phandle values to normalized
+ node paths in `self.root`, as returned by `_extract_phandles()`.
+ """
+ for path, lf_node in lf_root.walk():
+ if not lf_node.props:
+ continue
+ target_node = self.nodes.get(path)
+ if target_node is None:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing node '
+ f'{path or "/"}'
+ )
+ for prop_name, lf_prop in lf_node.props.items():
+ raw_prop = target_node.props.get(prop_name)
+ if raw_prop is None:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing '
+ f'property {path or "/"}:{prop_name}'
+ )
+ offsets = sorted(lf_prop.as_uint32_list())
+ masked = bytearray(raw_prop)
+ targets = []
+ for off in offsets:
+ if off + 4 > len(raw_prop):
+ raise ValueError(
+ f'{self.filename}: fixup offset {off} out of range '
+ f'on {path or "/"}:{prop_name} (len {len(raw_prop)})'
+ )
+ if targets and off < targets[-1][0] + 4:
+ raise ValueError(
+ f'{self.filename}: overlapping fixup offset {off} '
+ f'on {path or "/"}:{prop_name}'
+ )
+ ph = struct.unpack_from('>I', raw_prop, off)[0]
+ target_path = phandle_to_path.get(ph)
+ if target_path is None:
+ raise ValueError(
+ f'{self.filename}: unknown phandle {ph:#x} at '
+ f'{path or "/"}:{prop_name}+{off}'
+ )
+ masked[off:off + 4] = b'\x00\x00\x00\x00'
+ targets.append((off, target_path))
+ target_node.set_prop(
+ libfdt.Property(prop_name, bytes(masked))
+ )
+ target_node.fixups[prop_name] = tuple(targets)
+
+
+def select_best_candidate(candidates, path, prop_name):
+ """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
+
+ Only candidates shared by at least two input DTBs (`count > 1`) are
+ eligible for the base DTB; a value that appears in only one DTB would add
+ one copy to `base.dtb` to save one overlay copy (zero net savings).
+
+ Byte cost model per overlay occurrence:
+ - Each FDT property carries a 12-byte structure header (`FDT_PROP` 4-byte
+ tag + 4-byte length + 4-byte `nameoff`) plus `len(masked_bytes)` value
+ bytes.
+ - Each phandle fixup in an overlay adds a string entry of the form
+ `"/fragment@0/__overlay__<path>:<prop_name>:<off>\\0"` to `/__fixups__`:
+ 22 bytes (`"/fragment@0/__overlay__"`) + `len(path)` + 1 (`":"`) +
+ `len(prop_name)` + 1 (`":"`) + ~1-2 digits for `<off>` + 1 (`"\\0"`) +
+ ~3 bytes amortized `/__fixups__` property overhead =
+ `len(path) + len(prop_name) + 28` bytes.
+
+ Args:
+ candidates: Sequence of `(masked_bytes, fixups)` tuples observed across
+ input DTBs for `prop_name` on `path`.
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+
+ Returns:
+ The `(masked_bytes, fixups)` tuple that yields the largest estimated
+ byte savings when placed in `base.dtb`, or `None` if no candidate
+ appears in at least two input DTBs.
+ """
+ fixup_entry_cost = len(path) + len(prop_name) + 28
+ counts = Counter(candidates)
+ best_val = None
+ best_key = None
+ for val, count in counts.items():
+ if count <= 1:
+ continue
+ masked_bytes, fixups = val
+ length = len(masked_bytes) + len(fixups) * fixup_entry_cost
+ # Placing `val` in base.dtb adds 1 copy and removes `count` overlay
+ # copies, saving `(count - 1) * (length + 12)` bytes across the set.
+ key = ((count - 1) * (length + 12), count, length)
+ if best_key is None or key > best_key:
+ best_key = key
+ best_val = val
+ return best_val
+
+
+def _validate_dtb_headers(dtbs):
+ """Verify that FDT header fields and `/memreserve/` entries match across DTBs.
+
+ `fdt_overlay_apply()` only merges the `.dtbo` structure block into the base
+ DTB and ignores the `.dtbo` header and `/memreserve/` block, so all DTBs
+ sharing a base DTB must have identical `boot_cpuid_phys` and `mem_rsv`.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances to validate.
+ """
+ for d in dtbs[1:]:
+ if d.boot_cpuid_phys != dtbs[0].boot_cpuid_phys:
+ raise ValueError(
+ f'boot_cpuid_phys mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].boot_cpuid_phys}) and {d.filename} '
+ f'({d.boot_cpuid_phys})'
+ )
+ if d.mem_rsv != dtbs[0].mem_rsv:
+ raise ValueError(
+ f'/memreserve/ mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].mem_rsv}) and {d.filename} ({d.mem_rsv})'
+ )
+
+
+def _find_common_nodes_and_phandles(dtbs):
+ """Build the common `FactoredTree` skeleton and assign base DTB phandles.
+
+ Includes nodes whose path is present in every input DTB. If a node has a
+ `phandle` in at least one input DTB:
+ - When it has a shared label in `/__symbols__` across all input DTBs (or is
+ the root node), it is kept in `base.dtb` and assigned a sequential
+ phandle. That satisfies references from common properties, `/__symbols__`,
+ and overlay `/__fixups__`.
+ - Otherwise (e.g. only some boards label the node, or boards use different
+ labels), the node and its subtree are excluded from `base.dtb` and emitted
+ per board in `.dtbo` overlays. That avoids adding a spurious `phandle` to
+ boards that lacked one and avoids overlays needing a `/__fixups__` target
+ missing from `base.dtb`'s `/__symbols__`.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ Dict mapping each common node path in pre-order (`""` for root,
+ `"/..."` for subnodes) to its `FactoredTree` node, with `node.phandle`
+ set to its assigned sequential integer phandle (or `None` if the node
+ has no phandle in any input DTB) and `node.fixups` initialized to `{}`.
+ """
+ common_paths = set(dtbs[0].nodes).intersection(
+ *(d.nodes for d in dtbs[1:])
+ )
+
+ common_sym_targets = {
+ path for label, path in dtbs[0].symbols.items()
+ if all(d.symbols.get(label) == path for d in dtbs[1:])
+ }
+
+ base_root = FactoredTree()
+ base_nodes = {}
+ next_ph = 1
+ # Iterate dtbs[0].nodes in pre-order (rather than iterating the unordered
+ # `common_paths` set) so parents are always visited before their children,
+ # subnode order is preserved, and `next_ph` assignment is deterministic.
+ for path in dtbs[0].nodes:
+ if path not in common_paths:
+ continue
+ if path and path.rpartition('/')[0] not in base_nodes:
+ continue
+ has_phandle = any(d.nodes[path].phandle is not None for d in dtbs)
+ if has_phandle and path and path not in common_sym_targets:
+ continue
+ base_node = base_root.create_path(path)
+ base_nodes[path] = base_node
+ if has_phandle:
+ base_node.phandle = next_ph
+ next_ph += 1
+ return base_nodes
+
+
+def _find_common_symbols(dtbs, base_nodes):
+ """Find symbols common to all input DTBs that point to the same base node.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+
+ Returns:
+ Tuple of `(base_dtb_symbols, path_to_base_dtb_symbol)` where:
+ * `base_dtb_symbols`: Dict mapping symbol label to target node path.
+ * `path_to_base_dtb_symbol`: Dict mapping target node path to its
+ primary symbol label in `base_dtb_symbols`.
+ """
+ base_dtb_symbols = {}
+ path_to_base_dtb_symbol = {}
+
+ for label, chosen_path in dtbs[0].symbols.items():
+ if not all(d.symbols.get(label) == chosen_path for d in dtbs[1:]):
+ continue
+ base_node = base_nodes.get(chosen_path)
+ if base_node is None:
+ continue
+ base_dtb_symbols[label] = chosen_path
+ if (base_node.phandle is not None and
+ chosen_path not in path_to_base_dtb_symbol):
+ path_to_base_dtb_symbol[chosen_path] = label
+
+ return base_dtb_symbols, path_to_base_dtb_symbol
+
+
+def _build_base_tree(dtbs):
+ """Construct the normalized `FactoredTree` nodes for the common base DTB.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ Pre-order `path -> FactoredTree` dict containing all common nodes,
+ their assigned `phandle` attributes, and their best shared
+ `(masked_bytes, fixups)` properties.
+ """
+ base_nodes = _find_common_nodes_and_phandles(dtbs)
+ for path, base_node in base_nodes.items():
+ dtb_nodes = [d.nodes[path] for d in dtbs]
+ common_prop_names = [
+ k for k in dtb_nodes[0].props
+ if all(k in n.props for n in dtb_nodes[1:])
+ ]
+ for prop_name in common_prop_names:
+ eligible = []
+ for n in dtb_nodes:
+ fixups = n.fixups.get(prop_name, ())
+ if not all(t_path in base_nodes for _, t_path in fixups):
+ continue
+ eligible.append((n.props[prop_name], fixups))
+
+ chosen = select_best_candidate(eligible, path, prop_name)
+ if chosen is None:
+ continue
+ prop, fixups = chosen
+ base_node.set_prop(prop)
+ base_node.fixups[prop_name] = fixups
+
+ return base_nodes
+
+
+def _serialize_base_dtb(base_nodes, base_dtb_symbols, rsvmap, boot_cpuid_phys,
+ emit_local_fixups=True):
+ """Materialize phandles, `/__symbols__`, and `/__local_fixups__` and flatten.
+
+ Args:
+ base_nodes: Pre-order `path -> FactoredTree` dict for the base DTB from
+ `_build_base_tree()`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ rsvmap: Sequence of `(address, size)` memory reservation tuples.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+ emit_local_fixups: If True, include `/__local_fixups__` on the
+ generated `base.dtb`.
+
+ Returns:
+ Serialized `.dtb` `bytes` for the common base DTB.
+ """
+ out_root = libpydt.DeviceTree()
+ lf_entries = []
+
+ for path, base_node in base_nodes.items():
+ out_node = out_root.create_path(path)
+ for prop_name, prop in base_node.props.items():
+ fixups = base_node.fixups.get(prop_name, ())
+ if not fixups:
+ out_node.set_prop(prop)
+ else:
+ val = bytearray(prop)
+ for off, t_path in fixups:
+ target_ph = base_nodes[t_path].phandle
+ struct.pack_into('>I', val, off, target_ph)
+ out_node.set_prop(libfdt.Property(prop_name, bytes(val)))
+ lf_entries.append((path, prop_name, [off for off, _ in fixups]))
+
+ if base_node.phandle is not None:
+ out_node.set_prop(
+ libfdt.Property.from_uint32('phandle', base_node.phandle)
+ )
+
+ any_phandle = any(n.phandle is not None for n in base_nodes.values())
+ if base_dtb_symbols or (emit_local_fixups and any_phandle):
+ sym_node = out_root.create_path('/__symbols__')
+ for label, target_path in base_dtb_symbols.items():
+ sym_node.set_prop(
+ libfdt.Property.from_str(label, target_path or '/')
+ )
+
+ if emit_local_fixups:
+ # Create `/__local_fixups__` on the base DTB so `fdt_overlay_apply()`
+ # updates it with the overlay's phandle fixups when merging.
+ lf_root = out_root.create_path('/__local_fixups__')
+ for path, prop_name, offs in lf_entries:
+ lf_root.create_path(path).set_prop(
+ libfdt.Property.from_uint32_list(prop_name, offs)
+ )
+
+ fdt = out_root.flatten(mem_rsv=rsvmap, boot_cpuid_phys=boot_cpuid_phys)
+ return bytes(fdt.as_bytearray())
+
+
+def _assign_local_phandles(dtb, base_nodes):
+ """Assign sequential overlay-local phandles (`1, 2, ...`) to board-only nodes.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+
+ Returns:
+ Dict mapping each board-only node path that has a phandle in `dtb` to
+ its sequential overlay-local integer phandle (`1, 2, ...`).
+ """
+ local_phandles = {}
+ next_local_ph = 1
+ for path, node in dtb.nodes.items():
+ if node.phandle is not None and path not in base_nodes:
+ local_phandles[path] = next_local_ph
+ next_local_ph += 1
+ return local_phandles
+
+
+def _encode_dtbo_property(dtb_filename, path, prop_name, masked_bytes, fixups,
+ base_nodes, path_to_base_dtb_symbol, local_phandles,
+ ext_fixups, ov_local_fixups):
+ """Encode a property value for `/fragment@0/__overlay__` and record fixups.
+
+ Args:
+ dtb_filename: Input DTB filename (for error reporting).
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+ masked_bytes: Normalized property bytes (with zeroed phandle cells).
+ fixups: Tuple of `(offset, target_path)` phandle fixups for this
+ property.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+ local_phandles: Dict mapping board-only node paths to overlay-local
+ phandles.
+ ext_fixups: Dict mapping target base symbol labels to lists of
+ `/__fixups__` entry strings (mutated in place).
+ ov_local_fixups: Dict mapping `(path, prop_name)` to lists of byte
+ offsets for `/__local_fixups__` (mutated in place).
+
+ Returns:
+ Encoded property value `bytes` with `0xffffffff` placeholders for
+ external base references and local phandle integers for overlay-local
+ references.
+ """
+ if not fixups:
+ return masked_bytes
+
+ val = bytearray(masked_bytes)
+ for off, t_path in fixups:
+ if t_path in base_nodes:
+ sym_label = path_to_base_dtb_symbol.get(t_path)
+ if sym_label is None:
+ raise ValueError(
+ f'{dtb_filename}: property {path or "/"}:{prop_name} '
+ f'references common base node {t_path or "/"} which lacks '
+ f'a common symbol in /__symbols__'
+ )
+ struct.pack_into('>I', val, off, 0xffffffff)
+ entry = f'/fragment@0/__overlay__{path}:{prop_name}:{off}'
+ ext_fixups.setdefault(sym_label, []).append(entry)
+ else:
+ struct.pack_into('>I', val, off, local_phandles[t_path])
+ ov_local_fixups.setdefault((path, prop_name), []).append(off)
+ return bytes(val)
+
+
+def _build_board_dtbo(dtb, base_nodes, base_dtb_symbols,
+ path_to_base_dtb_symbol):
+ """Construct and flatten the `.dtbo` overlay blob for a single board.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_nodes: Pre-order `path -> FactoredTree` dict for `base.dtb`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+
+ Returns:
+ Serialized `.dtbo` overlay `bytes` for `dtb`.
+ """
+ dtbo_root = libpydt.DeviceTree()
+ frag_node = dtbo_root.create_path('/fragment@0')
+ frag_node.set_prop(libfdt.Property.from_str('target-path', '/'))
+ ov_root = frag_node.create_path('__overlay__')
+
+ local_phandles = _assign_local_phandles(dtb, base_nodes)
+ ext_fixups = {}
+ ov_local_fixups = {}
+
+ for path, node in dtb.nodes.items():
+ base_node = base_nodes.get(path)
+ if (base_node is not None and
+ base_node.props == node.props and
+ base_node.fixups == node.fixups):
+ continue
+ diff_props = [
+ prop_name for prop_name, raw_val in node.props.items()
+ if (base_node is None or
+ base_node.props.get(prop_name) != raw_val or
+ base_node.fixups.get(prop_name, ()) !=
+ node.fixups.get(prop_name, ()))
+ ]
+ if base_node is not None and not diff_props:
+ continue
+ ov_node = ov_root.create_path(path)
+ for prop_name in diff_props:
+ ov_node.set_prop(
+ libfdt.Property(
+ prop_name,
+ _encode_dtbo_property(
+ dtb.filename, path, prop_name,
+ node.props[prop_name],
+ node.fixups.get(prop_name, ()), base_nodes,
+ path_to_base_dtb_symbol, local_phandles, ext_fixups,
+ ov_local_fixups,
+ ),
+ )
+ )
+ if path in local_phandles:
+ ov_node.set_prop(
+ libfdt.Property.from_uint32('phandle', local_phandles[path])
+ )
+
+ diff_syms = [
+ (label, t_path)
+ for label, t_path in dtb.symbols.items()
+ if base_dtb_symbols.get(label) != t_path
+ ]
+ if diff_syms:
+ sym_node = dtbo_root.create_path('/__symbols__')
+ for label, t_path in diff_syms:
+ ov_root.create_path(t_path)
+ sym_node.set_prop(
+ libfdt.Property.from_str(
+ label, f'/fragment@0/__overlay__{t_path}'
+ )
+ )
+
+ if ext_fixups:
+ fixups_node = dtbo_root.create_path('/__fixups__')
+ for label, entries in ext_fixups.items():
+ fixups_node.set_prop(
+ libfdt.Property.from_stringlist(label, entries)
+ )
+
+ if ov_local_fixups:
+ ov_lf_root = dtbo_root.create_path(
+ '/__local_fixups__/fragment@0/__overlay__'
+ )
+ for (path, prop_name), offs in ov_local_fixups.items():
+ ov_lf_root.create_path(path).set_prop(
+ libfdt.Property.from_uint32_list(prop_name, offs)
+ )
+
+ return bytes(dtbo_root.flatten().as_bytearray())
+
+
+def create_base_dtb_and_overlays(dtbs):
+ """Compute the base DTB and per-board DTBO blobs from parsed DTBs.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances (at least two).
+
+ Returns:
+ Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
+ serialized `base.dtb` `bytes` and `dtbo_blobs` is a list of serialized
+ `.dtbo` `bytes` corresponding 1:1 with `dtbs`.
+ """
+ _validate_dtb_headers(dtbs)
+ base_nodes = _build_base_tree(dtbs)
+ base_dtb_symbols, path_to_base_dtb_symbol = _find_common_symbols(
+ dtbs, base_nodes
+ )
+ base_dtb_blob = _serialize_base_dtb(
+ base_nodes, base_dtb_symbols,
+ dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
+ emit_local_fixups=any(d.has_local_fixups for d in dtbs),
+ )
+ dtbo_blobs = [
+ _build_board_dtbo(
+ d, base_nodes, base_dtb_symbols, path_to_base_dtb_symbol,
+ )
+ for d in dtbs
+ ]
+ return base_dtb_blob, dtbo_blobs
+
+
+def get_dtbo_output_path(dtb_path, outdir):
+ """Determine the output `.dtbo` file path for an input `.dtb` path.
+
+ Args:
+ dtb_path: Path to the input `.dtb` file.
+ outdir: Output directory path, or `None` to place the `.dtbo` alongside
+ `dtb_path`.
+
+ Returns:
+ Output `.dtbo` file path string.
+ """
+ dtb_dir, filename = os.path.split(dtb_path)
+ stem = filename[:-4] if filename.endswith('.dtb') else filename
+ target_dir = outdir if outdir is not None else dtb_dir
+ return os.path.join(target_dir, f'{stem}.dtbo')
+
+
+def parse_args(argv=None):
+ """Parse command-line arguments.
+
+ Args:
+ argv: Optional list of command-line argument strings (defaults to
+ `sys.argv[1:]`).
+
+ Returns:
+ Parsed `argparse.Namespace` instance.
+ """
+ parser = argparse.ArgumentParser(
+ description=(
+ 'Split a set of compiled device tree blobs (.dtb) into a single '
+ 'common base .dtb and per-board overlay blobs (.dtbo).'
+ )
+ )
+ parser.add_argument(
+ '--output-base-dtb',
+ default='base.dtb',
+ metavar='FILENAME',
+ help=(
+ 'Output common base .dtb filename (default: base.dtb; placed in '
+ '--output if specified, or current directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--output',
+ default=None,
+ metavar='DIR',
+ help=(
+ 'Output directory for generated .dtbo files and '
+ '--output-base-dtb (default: alongside each input .dtb for .dtbo, '
+ 'current directory for --output-base-dtb)'
+ ),
+ )
+ parser.add_argument(
+ '-s', '--stats',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+ parser.add_argument(
+ 'dtbs',
+ nargs='+',
+ metavar='DTB',
+ help='Input .dtb files (at least two required)',
+ )
+ args = parser.parse_args(argv)
+ if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a non-empty filename without directory '
+ 'components; use -o/--output to specify the output directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
+ return args
+
+
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
+ """Print uncompressed size savings summary.
+
+ Args:
+ dtbs: Sequence of input `ParsedDtb` instances.
+ base_dtb_blob: Uncompressed base DTB `bytes`.
+ dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ """
+ orig_uncomp = sum(d.uncompressed_size for d in dtbs)
+ base_dtb_uncomp = len(base_dtb_blob)
+ dtbo_uncomp = sum(len(b) for b in dtbo_blobs)
+ new_uncomp = base_dtb_uncomp + dtbo_uncomp
+ saved_uncomp = orig_uncomp - new_uncomp
+ pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
+ print(
+ f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
+ f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
+ )
+
+
+def factor_dtb_files(args):
+ """Factor a list of standalone .dtb files into a base .dtb and .dtbo files.
+
+ Args:
+ args: Parsed `argparse.Namespace` from `parse_args()`.
+ """
+ input_dtb_paths = args.dtbs
+ dtbs = [ParsedDtb(p) for p in input_dtb_paths]
+
+ base_dtb_path = (
+ os.path.join(args.output, args.output_base_dtb)
+ if args.output is not None
+ else args.output_base_dtb
+ )
+
+ input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
+ base_abs_path = os.path.abspath(base_dtb_path)
+ if base_abs_path in input_abs_paths:
+ raise ValueError(
+ f'base DTB output path {base_dtb_path!r} collides with '
+ f'an input DTB file'
+ )
+
+ dtbo_paths = []
+ seen_paths = {base_abs_path}
+ for dtb_path in input_dtb_paths:
+ out_path = get_dtbo_output_path(dtb_path, args.output)
+ abs_out = os.path.abspath(out_path)
+ if abs_out in input_abs_paths:
+ raise ValueError(
+ f'output path {out_path!r} collides with an input DTB file'
+ )
+ if abs_out in seen_paths:
+ raise ValueError(
+ f'output path collision on {out_path!r}; ensure input '
+ f'files have distinct names'
+ )
+ seen_paths.add(abs_out)
+ dtbo_paths.append(out_path)
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+
+ if args.output:
+ os.makedirs(args.output, exist_ok=True)
+ with open(base_dtb_path, 'wb') as f:
+ f.write(base_dtb_blob)
+
+ for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ out_dir = os.path.dirname(out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(out_path, 'wb') as f:
+ f.write(dtbo_blob)
+
+ if args.stats:
+ _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+
+
+def main(argv=None):
+ """Run the fdtfactor command-line tool.
+
+ Args:
+ argv: Optional list of command-line argument strings.
+
+ Returns:
+ Integer exit status (`0` on success).
+ """
+ args = parse_args(argv)
+ try:
+ factor_dtb_files(args)
+ except (OSError, ValueError, RuntimeError, struct.error,
+ libfdt.FdtException) as exc:
+ sys.exit(f'Error: {exc}')
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/meson.build b/meson.build
index 348d9b555e34..97dc66c24ab4 100644
--- a/meson.build
+++ b/meson.build
@@ -119,6 +119,7 @@ if get_option('tools') and not wheel_only
install_data(
'dtdiff',
+ 'fdtfactor',
install_dir: get_option('bindir'),
install_mode: 'rwxr-xr-x',
)
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 623d71786701..9e8d09d37d8b 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,10 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
-TESTS_CLEANDIRS_L = fs
+TESTS_CLEANDIRS_L = fs *.test.dir
TESTS_CLEANDIRS = $(TESTS_CLEANDIRS_L:%=$(TESTS_PREFIX)%)
.PHONY: tests
diff --git a/tests/fdtfactor_board1.dts b/tests/fdtfactor_board1.dts
new file mode 100644
index 000000000000..324a3f23a488
--- /dev/null
+++ b/tests/fdtfactor_board1.dts
@@ -0,0 +1,72 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board1";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ };
+
+ b1_node: board1-node {
+ prop-str = "board1";
+ ref-base = <&sn1>;
+ ref-local = <&b1_node &b1_child>;
+
+ b1_child: child {
+ prop-int = <1>;
+ };
+ };
+};
diff --git a/tests/fdtfactor_board2.dts b/tests/fdtfactor_board2.dts
new file mode 100644
index 000000000000..7e4a71299cd1
--- /dev/null
+++ b/tests/fdtfactor_board2.dts
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board2";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ board2-only-prop = "only-on-board2";
+ board2-ref = <&b2_node>;
+ };
+
+ b2_node: board2-node {
+ prop-str = "board2";
+ ref-base = <&ssn1>;
+ ref-self = <&b2_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3.dts b/tests/fdtfactor_board3.dts
new file mode 100644
index 000000000000..74087efbd493
--- /dev/null
+++ b/tests/fdtfactor_board3.dts
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-base = <&sn2>;
+ ref-self = <&b3_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3_shifted.dts b/tests/fdtfactor_board3_shifted.dts
new file mode 100644
index 000000000000..263503f851b8
--- /dev/null
+++ b/tests/fdtfactor_board3_shifted.dts
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-self = <&b3_node>;
+ ref-base = <&sn2>;
+ };
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+ shared-data = <
+ 0x00010203 0x04050607 0x08090a0b 0x0c0d0e0f
+ 0x10111213 0x14151617 0x18191a1b 0x1c1d1e1f
+ 0x20212223 0x24252627 0x28292a2b 0x2c2d2e2f
+ 0x30313233 0x34353637 0x38393a3b 0x3c3d3e3f
+ 0x40414243 0x44454647 0x48494a4b 0x4c4d4e4f
+ 0x50515253 0x54555657 0x58595a5b 0x5c5d5e5f
+ 0x60616263 0x64656667 0x68696a6b 0x6c6d6e6f
+ 0x70717273 0x74757677 0x78797a7b 0x7c7d7e7f
+ 0x80818283 0x84858687 0x88898a8b 0x8c8d8e8f
+ 0x90919293 0x94959697 0x98999a9b 0x9c9d9e9f
+ 0xa0a1a2a3 0xa4a5a6a7 0xa8a9aaab 0xacadaeaf
+ 0xb0b1b2b3 0xb4b5b6b7 0xb8b9babb 0xbcbdbebf
+ 0xc0c1c2c3 0xc4c5c6c7 0xc8c9cacb 0xcccdcecf
+ 0xd0d1d2d3 0xd4d5d6d7 0xd8d9dadb 0xdcdddedf
+ 0xe0e1e2e3 0xe4e5e6e7 0xe8e9eaeb 0xecedeeef
+ 0xf0f1f2f3 0xf4f5f6f7 0xf8f9fafb 0xfcfdfeff
+ 0x01030507 0x090b0d0f 0x11131517 0x191b1d1f
+ 0x21232527 0x292b2d2f 0x31333537 0x393b3d3f
+ 0x41434547 0x494b4d4f 0x51535557 0x595b5d5f
+ 0x61636567 0x696b6d6f 0x71737577 0x797b7d7f
+ 0x81838587 0x898b8d8f 0x91939597 0x999b9d9f
+ 0xa1a3a5a7 0xa9abadaf 0xb1b3b5b7 0xb9bbbdbf
+ 0xc1c3c5c7 0xc9cbcdcf 0xd1d3d5d7 0xd9dbdddf
+ 0xe1e3e5e7 0xe9ebedef 0xf1f3f5f7 0xf9fbfdff
+ 0x02060a0e 0x12161a1e 0x22262a2e 0x32363a3e
+ 0x42464a4e 0x52565a5e 0x62666a6e 0x72767a7e
+ 0x82868a8e 0x92969a9e 0xa2a6aaae 0xb2b6babe
+ 0xc2c6cace 0xd2d6dade 0xe2e6eaee 0xf2f6fafe
+ 0x03070b0f 0x13171b1f 0x23272b2f 0x33373b3f
+ 0x43474b4f 0x53575b5f 0x63676b6f 0x73777b7f
+ 0x83878b8f 0x93979b9f 0xa3a7abaf 0xb3b7bbbf
+ 0xc3c7cbcf 0xd3d7dbdf 0xe3e7ebef 0xf3f7fbff
+ >;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+};
diff --git a/tests/meson.build b/tests/meson.build
index 9f0f51013530..5d1c3a798c96 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -144,7 +144,7 @@ run_test_deps = [
dtc_tools, treegen_dtb, tests_exe,
]
if pylibfdt_enabled
- run_test_types += 'pylibfdt'
+ run_test_types += ['pylibfdt', 'fdtfactor']
run_test_deps += pylibfdt
endif
foreach test_type : run_test_types
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index e16767d3f3a2..815233227df1 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1207,6 +1207,106 @@ pylibfdt_tests () {
run_python_tests "$SRCDIR/libpydt_tests.py"
}
+fdtfactor_tests () {
+ check_factored_dtb () {
+ exp="$1"
+ got="$2"
+ if [ "$OVERLAY_LOCAL_FIXUPS" -ne 2 ]; then
+ cp "$exp" "$got.exp.test.dtb"
+ $DTPUT -r "$got.exp.test.dtb" /__local_fixups__ 2>/dev/null || true
+ $DTPUT -r "$got" /__local_fixups__ 2>/dev/null || true
+ exp="$got.exp.test.dtb"
+ fi
+ run_test dtbs_equal_unordered "$exp" "$got"
+ }
+
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_board${i}.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+
+ # Default mode: factor 3 boards into base.dtb + .dtbo overlays and verify
+ # that majority properties land in base.dtb and fdtoverlay reconstructs
+ # each original DTB (including /__local_fixups__, /memreserve/, and
+ # boot_cpuid_phys).
+ rm -rf fdtfactor_out.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s --output-base-dtb base.test.dtb \
+ -o fdtfactor_out.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ run_fdtget_test "deadbeef" -tx fdtfactor_out.test.dir/base.test.dtb \
+ / prop-int
+ run_wrap_error_test $DTGET \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ /fragment@0/__overlay__ prop-int
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_out.test.dir/base.test.dtb \
+ -o fdtfactor_out.test.dir/merged${i}.test.dtb \
+ fdtfactor_out.test.dir/fdtfactor_board${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fdtfactor_out.test.dir/merged${i}.test.dtb
+ done
+
+ # Shifted phandles: fdtfactor_board3_shifted.dts defines board3-node before
+ # subnode@1 so all phandle numbers differ from board1 and board2; verify
+ # that fdtfactor normalizes them via /__local_fixups__.
+ run_dtc_test -b 2 -@ -L -I dts -O dtb \
+ -o fdtfactor_board3_shifted.test.dtb \
+ "$SRCDIR/fdtfactor_board3_shifted.dts"
+ rm -rf fdtfactor_shift.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -o fdtfactor_shift.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3_shifted.test.dtb
+ run_wrap_error_test $DTGET \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo \
+ /fragment@0/__overlay__/subnode@1 prop-ref
+ run_wrap_test $FDTOVERLAY -i fdtfactor_shift.test.dir/base.dtb \
+ -o fdtfactor_shift.test.dir/merged3.test.dtb \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo
+ check_factored_dtb fdtfactor_board3.test.dtb \
+ fdtfactor_shift.test.dir/merged3.test.dtb
+
+ # DTBs compiled with dtc -@ -L that contain no labels or phandle references
+ # (for which dtc omits /__symbols__ and /__local_fixups__) are accepted.
+ printf '/dts-v1/;\n/ { common = <1>; b = "a"; };\n' > fdtfactor_noph1.test.dts
+ printf '/dts-v1/;\n/ { common = <1>; b = "b"; };\n' > fdtfactor_noph2.test.dts
+ for i in 1 2; do
+ run_dtc_test -@ -L -I dts -O dtb -o fdtfactor_noph${i}.test.dtb \
+ fdtfactor_noph${i}.test.dts
+ run_test check_path fdtfactor_noph${i}.test.dtb \
+ not-exists "/__symbols__"
+ run_test check_path fdtfactor_noph${i}.test.dtb \
+ not-exists "/__local_fixups__"
+ done
+ rm -rf fdtfactor_noph.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -o fdtfactor_noph.test.dir \
+ fdtfactor_noph1.test.dtb fdtfactor_noph2.test.dtb
+ for i in 1 2; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_noph.test.dir/base.dtb \
+ -o fdtfactor_noph.test.dir/merged${i}.test.dtb \
+ fdtfactor_noph.test.dir/fdtfactor_noph${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_noph${i}.test.dtb \
+ fdtfactor_noph.test.dir/merged${i}.test.dtb
+ done
+
+ # Legacy linux,phandle properties (dtc -H both / -H legacy), overlay .dtbo
+ # inputs, and inputs lacking /__local_fixups__ are rejected.
+ run_dtc_test -H both -@ -L -I dts -O dtb \
+ -o fdtfactor_legacy_ph.test.dtb "$SRCDIR/fdtfactor_board1.dts"
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_legacy_ph.test.dtb fdtfactor_board2.test.dtb
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ fdtfactor_board2.test.dtb
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -I dts -O dtb \
+ -o fdtfactor_board${i}.nolf.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+}
+
while getopts "vt:me" ARG ; do
case $ARG in
"v")
@@ -1227,14 +1327,15 @@ done
if [ -z "$TESTSETS" ]; then
TESTSETS="libfdt utilfdt dtc dtbs_equal fdtget fdtput fdtdump fdtoverlay"
- # Test pylibfdt if the libfdt Python module is available.
+ # Test pylibfdt and fdtfactor if the libfdt Python module is available.
if ! $no_python; then
- TESTSETS="$TESTSETS pylibfdt"
+ TESTSETS="$TESTSETS pylibfdt fdtfactor"
fi
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -rf *.test.dir
for set in $TESTSETS; do
case $set in
@@ -1265,6 +1366,9 @@ for set in $TESTSETS; do
"fdtoverlay")
fdtoverlay_tests
;;
+ "fdtfactor")
+ fdtfactor_tests
+ ;;
esac
done
diff --git a/tests/testutils.sh b/tests/testutils.sh
index 6b2f0d14e3e5..7f35ba8e4158 100644
--- a/tests/testutils.sh
+++ b/tests/testutils.sh
@@ -27,6 +27,7 @@ DTGET=${TEST_BINDIR}/fdtget
DTPUT=${TEST_BINDIR}/fdtput
FDTDUMP=${TEST_BINDIR}/fdtdump
FDTOVERLAY=${TEST_BINDIR}/fdtoverlay
+FDTFACTOR=${SRCDIR}/../fdtfactor
verbose_run () {
if [ -z "$QUIET_TEST" ]; then
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (9 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
By default, `fdtfactor` preserves `/__local_fixups__` on `base.dtb` so
that when `fdt_overlay_apply()` merges a `.dtbo` overlay, the
reconstructed device tree retains full `/__local_fixups__` information
(for instance, if an OS kernel wants to know which property cells
contain phandles at runtime).
When `-L` was passed to `dtc` solely to help `fdtfactor` locate phandle
cells at build time and `/__local_fixups__` is not needed at runtime,
retaining `/__local_fixups__` in `base.dtb` wastes space.
Add `-S` / `--strip-local-fixups` to omit `/__local_fixups__` from
`base.dtb` (and therefore from the merged device trees produced by
`fdt_overlay_apply()`). On a set of 47 Qualcomm SC7180 Trogdor
Chromebook board DTBs, `fdtfactor` without `-S` reduces total
uncompressed size from 7.69 MB to 932 KB (88.2% savings), while
`fdtfactor -S` reduces it to 634 KB (91.8% savings, saving an additional
298 KB by omitting `/__local_fixups__` from `base.dtb`).
Add a test case in `tests/run_tests.sh` verifying that `fdtfactor -S`
followed by `fdtoverlay` reproduces DTBs compiled with `-@` (without
`-L`) via `dtbs_equal_unordered`.
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Use 4-space indentation instead of mixed tabs and spaces in
tests/run_tests.sh, and update context for -o/--output (David Gibson)
Changes in v3:
- Include both non--S (932 KB) and -S (634 KB) Trogdor size numbers in
the commit message to show the savings of -S itself (Simon Glass)
- Pass -S to the negative test verifying that inputs lacking
/__local_fixups__ are still rejected when -S is used, and move the
non--S rejection test to the base fdtfactor patch (Simon Glass)
Changes in v2:
- Split -S/--strip-local-fixups into a separate opt-in patch (David
Gibson)
README.md | 9 +++++++++
fdtfactor | 21 ++++++++++++++++++---
tests/run_tests.sh | 19 +++++++++++++++++++
3 files changed, 46 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index 9a8be74d47fe..198384cd8456 100644
--- a/README.md
+++ b/README.md
@@ -228,6 +228,15 @@ Options:
* `-o`, `--output`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ `base.dtb` and final reconstructed device trees. By default, `fdtfactor`
+ preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
+ merges a `.dtbo` overlay, the reconstructed device tree retains full
+ `/__local_fixups__` information (e.g., for an OS kernel that wants to
+ know which property cells contain phandles at runtime). When `-L` was
+ passed to `dtc` solely to help `fdtfactor` locate phandles at build time
+ and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
+ from `base.dtb` to save space.
* `-s`, `--stats`: Print size savings summary.
### TODO
diff --git a/fdtfactor b/fdtfactor
index dfbee0aea2d6..a083a8c380a0 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -675,11 +675,13 @@ def _build_board_dtbo(dtb, base_nodes, base_dtb_symbols,
return bytes(dtbo_root.flatten().as_bytearray())
-def create_base_dtb_and_overlays(dtbs):
+def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
"""Compute the base DTB and per-board DTBO blobs from parsed DTBs.
Args:
dtbs: Sequence of `ParsedDtb` instances (at least two).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated `base.dtb`.
Returns:
Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
@@ -694,7 +696,9 @@ def create_base_dtb_and_overlays(dtbs):
base_dtb_blob = _serialize_base_dtb(
base_nodes, base_dtb_symbols,
dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
- emit_local_fixups=any(d.has_local_fixups for d in dtbs),
+ emit_local_fixups=(
+ not strip_local_fixups and any(d.has_local_fixups for d in dtbs)
+ ),
)
dtbo_blobs = [
_build_board_dtbo(
@@ -757,6 +761,14 @@ def parse_args(argv=None):
'current directory for --output-base-dtb)'
),
)
+ parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from the generated base .dtb and '
+ 'reconstructed device trees'
+ ),
+ )
parser.add_argument(
'-s', '--stats',
action='store_true',
@@ -841,7 +853,10 @@ def factor_dtb_files(args):
seen_paths.add(abs_out)
dtbo_paths.append(out_path)
- base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ dtbs,
+ strip_local_fixups=args.strip_local_fixups,
+ )
if args.output:
os.makedirs(args.output, exist_ok=True)
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 815233227df1..4f5026e363fc 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1305,6 +1305,25 @@ fdtfactor_tests () {
done
run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+
+ # Strip local fixups (-S): verify that merged DTBs match DTBs compiled
+ # with -@ (without -L), and that inputs lacking /__local_fixups__ are
+ # still rejected even when -S is passed.
+ run_wrap_error_test $PYTHON $FDTFACTOR -S -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+ rm -rf fdtfactor_strip.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -S --output-base-dtb base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_strip.test.dir/base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir/merged${i}.test.dtb \
+ fdtfactor_strip.test.dir/fdtfactor_board${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
+ fdtfactor_strip.test.dir/merged${i}.test.dtb
+ done
}
while getopts "vt:me" ARG ; do
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (10 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Firmware and FIT build pipelines frequently store device tree blobs
compressed with LZ4, LZMA/XZ, or gzip (e.g., `.dtb.lz4`).
Add `-C` / `--compression` (`none`, `lz4`, `lz4_legacy`, `lzma`, `xz`,
`gzip`; default `none`) to `fdtfactor` to decompress compressed input
DTBs before factoring and compress the resulting `base.dtb` and `.dtbo`
outputs with the same algorithm:
- Support `lz4` and `lz4_legacy` (via the `lz4` CLI), `lzma` and `xz`
(via Python's `lzma` module, using `CHECK_CRC32` for XZ compatibility
with kernel/bootloader decompressors), and `gzip` (via Python's `gzip`
module).
- Default `--output-base-dtb` to `base.dtb.<ext>` matching `-C` /
`--compression` (e.g., `base.dtb.lz4` when `-C lz4` is passed), and
emit `.dtbo.<ext>` overlays.
- Report compressed size savings alongside uncompressed savings when
`-s` / `--stats` is specified with compression enabled.
On the 47 Qualcomm SC7180 Trogdor Chromebook `.dtb.lz4` files,
`fdtfactor -S -C lz4` reduces total LZ4-compressed size from 1.08 MB
(built without `-@ -L`) to 214 KB (80.2% savings).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Replace magic-byte/extension auto-detection and --base-compression
with a single explicit -C/--compression option controlling both input
and output compression (David Gibson)
- Drop unnecessary --stats documentation/help changes and use 4-space
indentation in tests/run_tests.sh (David Gibson)
- Use check_factored_dtb in fdtfactor_tests so tests pass when
OVERLAY_LOCAL_FIXUPS != 2 (Simon Glass)
Changes in v3:
- Update for -b/--base-dtb rename to --output-base-dtb (David Gibson)
Changes in v2:
- Add --base-compression to explicitly control base.dtb compression
while documenting auto-detection from --base-dtb's filename suffix
(David Gibson)
- Support legacy LZ4 frames (lz4 -l) and use CRC32 check for XZ outputs
(Simon Glass)
README.md | 11 ++-
fdtfactor | 208 ++++++++++++++++++++++++++++++++++++++-----
tests/Makefile.tests | 5 +-
tests/run_tests.sh | 51 ++++++++++-
4 files changed, 248 insertions(+), 27 deletions(-)
diff --git a/README.md b/README.md
index 198384cd8456..ac571384d421 100644
--- a/README.md
+++ b/README.md
@@ -204,7 +204,8 @@ $ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
## fdtfactor
`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`) and splits them into a single common
+compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
+`.dtb.lzma`, `.dtb.gz` with `-C`) and splits them into a single common
`base.dtb` and per-board `.dtbo` overlays so that applying a board's
`.dtbo` to `base.dtb` recreates the original board device tree while
reducing total storage size.
@@ -223,8 +224,12 @@ $ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
Options:
* `--output-base-dtb`: Output filename for the common base `.dtb`
- (default: `base.dtb`; placed in `--output` if specified, or the current
- directory otherwise).
+ (default: `base.dtb`, or `base.dtb.<ext>` when `-C` / `--compression` is
+ used; placed in `--output` if specified, or the current directory
+ otherwise).
+* `-C`, `--compression`: Compression format for both input `.dtb` files and
+ generated `base.dtb` / `.dtbo` outputs: `none`, `lz4`, `lz4_legacy`, `lzma`,
+ `xz`, or `gzip` (default: `none`).
* `-o`, `--output`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
diff --git a/fdtfactor b/fdtfactor
index a083a8c380a0..beeeff231ba6 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -12,14 +12,125 @@ original board DTB.
import argparse
from collections import Counter
+import gzip
+import lzma
import os
import struct
+import subprocess
import sys
import libfdt
import libpydt
+COMP_TO_EXT = {
+ 'none': '',
+ 'lz4': '.lz4',
+ 'lz4_legacy': '.lz4',
+ 'lzma': '.lzma',
+ 'xz': '.xz',
+ 'gzip': '.gz',
+}
+
+
+def _run_lz4(args, input_bytes, action):
+ """Run the `lz4` CLI with `args` on `input_bytes` and return stdout.
+
+ Args:
+ args: List of command-line arguments to pass to `lz4`.
+ input_bytes: Bytes to feed to `lz4` standard input.
+ action: Human-readable action description for error messages.
+
+ Returns:
+ Output `bytes` written to `lz4` standard output.
+ """
+ try:
+ res = subprocess.run(
+ ['lz4', *args],
+ input=input_bytes,
+ capture_output=True,
+ check=False,
+ )
+ except FileNotFoundError as exc:
+ raise RuntimeError(
+ 'lz4 command not found; install lz4 to process .lz4 files'
+ ) from exc
+ if res.returncode != 0:
+ err = res.stderr.decode('utf-8', errors='replace').strip()
+ raise RuntimeError(f'Failed to {action} with lz4: {err}')
+ return res.stdout
+
+
+def decompress_blob(data, comp, filename=''):
+ """Decompress a DTB blob according to `comp`.
+
+ Args:
+ data: Input file or property `bytes` (compressed or uncompressed).
+ comp: Compression format (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ filename: Optional filename or node description for error reporting.
+
+ Returns:
+ Uncompressed FDT `bytes`.
+ """
+ if comp in ('none', '', None):
+ return data
+ target = filename or 'blob'
+ if comp in ('lz4', 'lz4_legacy'):
+ return _run_lz4(['-d', '-q', '-c'], data, f'decompress {target}')
+ if comp == 'gzip':
+ try:
+ return gzip.decompress(data)
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {target} with gzip: {exc}'
+ ) from exc
+ if comp in ('lzma', 'xz'):
+ try:
+ return lzma.decompress(data)
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {target} with {comp}: {exc}'
+ ) from exc
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
+def compress_blob(raw_bytes, comp, filename=''):
+ """Compress an FDT blob using the specified compression algorithm.
+
+ Args:
+ raw_bytes: Uncompressed FDT `bytes`.
+ comp: Compression format (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ filename: Optional filename for error messages.
+
+ Returns:
+ Compressed (or uncompressed if `comp` is `'none'`) `bytes`.
+ """
+ if comp in ('none', '', None):
+ return raw_bytes
+ target = filename or 'blob'
+ if comp == 'lz4':
+ return _run_lz4(
+ ['-12', '-z', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'lz4_legacy':
+ return _run_lz4(
+ ['-l', '-12', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'gzip':
+ return gzip.compress(raw_bytes, mtime=0)
+ if comp == 'lzma':
+ return lzma.compress(raw_bytes, format=lzma.FORMAT_ALONE)
+ if comp == 'xz':
+ return lzma.compress(
+ raw_bytes, format=lzma.FORMAT_XZ, check=lzma.CHECK_CRC32
+ )
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
def _is_dtbo_overlay(root):
"""Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
@@ -64,7 +175,7 @@ class ParsedDtb:
- Stores FDT header, memory reservation, and file-level metadata that live
outside the root node tree (`boot_cpuid_phys`, `mem_rsv`, `filename`,
- `uncompressed_size`).
+ `compression`, `compressed_size`, `uncompressed_size`).
- Pops `/__symbols__` (parsed into `self.symbols`) and `/__local_fixups__`
out of `self.root`, and indexes the remaining tree in pre-order in
`self.nodes` (`path -> FactoredTree`).
@@ -75,15 +186,19 @@ class ParsedDtb:
independent of numeric phandle numbering.
"""
- def __init__(self, filename):
+ def __init__(self, filename, comp='none'):
"""Parse an input FDT file into a normalized FactoredTree.
Args:
- filename: File path of the `.dtb` file to read.
+ filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
+ comp: Compression format string (default: `'none'`).
"""
self.filename = filename
+ self.compression = comp
with open(filename, 'rb') as f:
- raw_bytes = f.read()
+ file_bytes = f.read()
+ self.compressed_size = len(file_bytes)
+ raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
self.uncompressed_size = len(raw_bytes)
try:
@@ -709,21 +824,26 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
-def get_dtbo_output_path(dtb_path, outdir):
- """Determine the output `.dtbo` file path for an input `.dtb` path.
+def get_dtbo_output_path(dtb_path, outdir, comp='none'):
+ """Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
Args:
- dtb_path: Path to the input `.dtb` file.
+ dtb_path: Path to the input `.dtb` (or `.dtb.*`) file.
outdir: Output directory path, or `None` to place the `.dtbo` alongside
`dtb_path`.
+ comp: Compression format string for the output `.dtbo` (default:
+ `'none'`).
Returns:
- Output `.dtbo` file path string.
+ Output `.dtbo` (or compressed `.dtbo.*`) file path string.
"""
dtb_dir, filename = os.path.split(dtb_path)
+ ext = COMP_TO_EXT[comp]
+ if ext and filename.endswith(ext):
+ filename = filename[:-len(ext)]
stem = filename[:-4] if filename.endswith('.dtb') else filename
target_dir = outdir if outdir is not None else dtb_dir
- return os.path.join(target_dir, f'{stem}.dtbo')
+ return os.path.join(target_dir, f'{stem}.dtbo{ext}')
def parse_args(argv=None):
@@ -738,17 +858,28 @@ def parse_args(argv=None):
"""
parser = argparse.ArgumentParser(
description=(
- 'Split a set of compiled device tree blobs (.dtb) into a single '
- 'common base .dtb and per-board overlay blobs (.dtbo).'
+ 'Split a set of compiled device tree blobs (.dtb, or compressed '
+ '.dtb.lz4/.xz/.lzma/.gz with -C) into a single common base .dtb '
+ 'and per-board overlay blobs (.dtbo).'
)
)
parser.add_argument(
'--output-base-dtb',
- default='base.dtb',
+ default=None,
metavar='FILENAME',
help=(
- 'Output common base .dtb filename (default: base.dtb; placed in '
- '--output if specified, or current directory otherwise)'
+ 'Output common base .dtb filename (default: base.dtb, or '
+ 'base.dtb.<ext> when -C/--compression is used; placed in --output '
+ 'if specified, or current directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '-C', '--compression',
+ choices=tuple(COMP_TO_EXT),
+ default='none',
+ help=(
+ 'Compression format for both input DTBs and output base.dtb/.dtbo '
+ 'files (default: none)'
),
)
parser.add_argument(
@@ -778,9 +909,14 @@ def parse_args(argv=None):
'dtbs',
nargs='+',
metavar='DTB',
- help='Input .dtb files (at least two required)',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
+ '(at least two required)'
+ ),
)
args = parser.parse_args(argv)
+ if args.output_base_dtb is None:
+ args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
parser.error(
'--output-base-dtb must be a non-empty filename without directory '
@@ -791,13 +927,17 @@ def parse_args(argv=None):
return args
-def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
- """Print uncompressed size savings summary.
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
+ dtbo_csizes):
+ """Print uncompressed and compressed size savings summary.
Args:
dtbs: Sequence of input `ParsedDtb` instances.
base_dtb_blob: Uncompressed base DTB `bytes`.
dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ comp: Compression format of the inputs and outputs.
+ base_dtb_csize: Byte size of the compressed base DTB output.
+ dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
"""
orig_uncomp = sum(d.uncompressed_size for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -811,6 +951,18 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
+ if comp != 'none':
+ orig_comp = sum(d.compressed_size for d in dtbs)
+ dtbo_csize = sum(dtbo_csizes)
+ new_comp = base_dtb_csize + dtbo_csize
+ saved_comp = orig_comp - new_comp
+ pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
+ print(
+ f'Compressed ({comp}): {len(dtbs)} files ({orig_comp:,} B) -> '
+ f'base {base_dtb_csize:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_csize:,} B (avg {dtbo_csize // len(dtbo_blobs):,} B) = '
+ f'{new_comp:,} B (saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ )
def factor_dtb_files(args):
@@ -819,8 +971,9 @@ def factor_dtb_files(args):
Args:
args: Parsed `argparse.Namespace` from `parse_args()`.
"""
+ comp = args.compression
input_dtb_paths = args.dtbs
- dtbs = [ParsedDtb(p) for p in input_dtb_paths]
+ dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
base_dtb_path = (
os.path.join(args.output, args.output_base_dtb)
@@ -839,7 +992,7 @@ def factor_dtb_files(args):
dtbo_paths = []
seen_paths = {base_abs_path}
for dtb_path in input_dtb_paths:
- out_path = get_dtbo_output_path(dtb_path, args.output)
+ out_path = get_dtbo_output_path(dtb_path, args.output, comp=comp)
abs_out = os.path.abspath(out_path)
if abs_out in input_abs_paths:
raise ValueError(
@@ -858,20 +1011,31 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
+ base_dtb_file_bytes = compress_blob(
+ base_dtb_blob, comp, filename=base_dtb_path
+ )
if args.output:
os.makedirs(args.output, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
- f.write(base_dtb_blob)
+ f.write(base_dtb_file_bytes)
+ dtbo_file_sizes = []
for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
+ out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
+ dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
- f.write(dtbo_blob)
+ f.write(out_bytes)
if args.stats:
- _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+ _print_dtb_stats(
+ dtbs, base_dtb_blob, dtbo_blobs,
+ comp=comp,
+ base_dtb_csize=len(base_dtb_file_bytes),
+ dtbo_csizes=dtbo_file_sizes,
+ )
def main(argv=None):
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 9e8d09d37d8b..1eed9ba6478a 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
+ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
+ tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
TESTS_CLEANDIRS_L = fs *.test.dir
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 4f5026e363fc..9b005396bc79 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1324,6 +1324,53 @@ fdtfactor_tests () {
run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
fdtfactor_strip.test.dir/merged${i}.test.dtb
done
+
+ # Compressed (.dtb.gz / .dtb.xz / .dtb.lzma / .dtb.lz4) modes via -C
+ for comp_spec in "gzip:gz:gzip -9 -k -f:gzip -d -f" \
+ "xz:xz:xz -k -f:xz -d -f" \
+ "lzma:lzma:xz --format=lzma -k -f:xz -d -f" \
+ "lz4:lz4:lz4 -12 -q -f -m:lz4 -d -q -f --rm" \
+ "lz4_legacy:lz4:lz4 -l -12 -q -f -m:lz4 -d -q -f --rm"; do
+ algo="${comp_spec%%:*}"
+ rest="${comp_spec#*:}"
+ ext="${rest%%:*}"
+ rest="${rest#*:}"
+ comp_cmd="${rest%%:*}"
+ decomp_cmd="${rest#*:}"
+ if ! command -v "${comp_cmd%% *}" >/dev/null 2>&1; then
+ continue
+ fi
+ run_wrap_test $comp_cmd \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ rm -rf fdtfactor_comp.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s -C "$algo" \
+ -o fdtfactor_comp.test.dir \
+ "fdtfactor_board1.test.dtb.$ext" \
+ "fdtfactor_board2.test.dtb.$ext" \
+ "fdtfactor_board3.test.dtb.$ext"
+ run_wrap_test $decomp_cmd "fdtfactor_comp.test.dir/base.dtb.$ext"
+ run_wrap_test $decomp_cmd \
+ "fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo.$ext"
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_comp.test.dir/base.dtb \
+ -o fdtfactor_comp.test.dir/merged1.test.dtb \
+ fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo
+ check_factored_dtb fdtfactor_board1.test.dtb \
+ fdtfactor_comp.test.dir/merged1.test.dtb
+ done
+
+ # Verify that mismatched compression (compressed inputs without -C, or
+ # uncompressed inputs with -C) fails rather than guessing.
+ if command -v gzip >/dev/null 2>&1; then
+ run_wrap_test gzip -9 -k -f \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_comp_err.test.dir \
+ fdtfactor_board1.test.dtb.gz fdtfactor_board2.test.dtb.gz
+ run_wrap_error_test $PYTHON $FDTFACTOR -C gzip \
+ -o fdtfactor_comp_err.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1353,7 +1400,9 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+ *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
+ *.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
for set in $TESTSETS; do
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (11 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
Bootloaders such as depthcharge on Chromebooks and U-Boot use Flattened
Image Trees (FITs) to bundle a kernel image with multiple board FDT
blobs (`fdt-1`, `fdt-2`, ...) and per-board configuration nodes.
Add `-F`/`--fit`, `--min-overlap`, and `--single-base-dtb` options to
`fdtfactor` (writing the factored FIT to `-o`/`--output`, or updating
`--fit` in place by default):
- Parse non-overlay `type = "flat_dt"` nodes under `/images`,
decompressing their `data` blobs according to their `compression`
property (`none`, `lz4`, `lzma`, `xz`, `gzip`).
- Reject FITs containing `signature*` subnodes under `/images/*` or
`/configurations/*`, as modifying FDT blobs and configurations would
invalidate cryptographic signatures.
- Partition the `flat_dt` blobs into factor groups by compatible FDT
headers / FIT `arch` and `load` properties and shared normalized
property byte overlap (`>= --min-overlap`%, default `50`%, reusing
`select_best_candidate()`'s `_prop_byte_cost()` model) so a single FIT
containing multiple board/SoC families produces a separate
`fdt-base-1`, `fdt-base-2`, etc. for each family.
- Support `--single-base-dtb` to force all `flat_dt` blobs into a single
base DTB regardless of property overlap, failing only if their FDT
headers, `arch`, or `load` properties differ.
- Insert each `fdt-base-N` node into `/images` (picking the next unused
`fdt-base-N` name), replace each factored `fdt-*` node's `data` with
its `.dtbo` overlay (preserving each node's `compression` setting, and
stripping `/__local_fixups__` from any unfactored single-board
`flat_dt` nodes too when `-S` is passed), recompute any `hash*`
subnode digests, and prepend `fdt-base-N` to each matching
configuration's `fdt` string list while preserving all other
configuration properties (`description`, `compatible`, etc.,
populating `compatible` from the board DTB's root node when omitted)
and any pre-existing overlay entries.
Add uncompressed, LZ4-compressed multi-family, `--min-overlap`,
unfactored-singleton `-S`, pre-existing `fdt-base-*` / multi-pass,
signed-FIT rejection, and `--single-base-dtb` FIT test cases to the
`fdtfactor` test suite.
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v4:
- Unify --outfile into -o/--output, and split standalone DTB vs. FIT
options into separate README.md subsections and --help groups (David
Gibson)
- Check _is_dtbo_overlay() directly in _extract_fit_fdt_entries()
instead of adding OverlayBlobError to ParsedDtb (David Gibson)
- Map standard FIT hash algorithm names explicitly via FIT_HASH_ALGOS
instead of passing raw names to hashlib.new() (David Gibson)
- Remove compression and entry from FIT grouping, document load, expose
--min-overlap, and compute FIT group overlap using normalized property
byte costs via _prop_byte_cost() instead of unweighted node paths
(David Gibson)
- Strip /__local_fixups__ from unfactored single-board FIT flat_dt
entries when -S/--strip-local-fixups is passed, and pick the next
unused fdt-base-N node name instead of rejecting FITs that already
contain fdt-base-* nodes
- Pass already-unflattened FactoredTree into ParsedDtb and intern
normalized properties to integer IDs in group_fit_dtbs() to speed up
multi-SoC FIT factoring
- Preserve whether input FIT FDT descriptions include a compression
extension instead of unconditionally appending one
- Update for libpydt rename, use Property.as_*(), Property.from_*(), and
DeviceTree.set_prop() instead of DeviceTree get_*/set_* wrapper
methods, and drop redundant bytes() copies on Property instances
(David Gibson, Simon Glass)
- Use check_factored_dtb and 4-space indentation in tests/run_tests.sh
(Simon Glass, David Gibson)
Changes in v3:
- Use "FIT" instead of "FIT image" in documentation, comments, and
commit messages (Simon Glass)
- Update for DeviceTree and --output-base-dtb (Simon Glass, David
Gibson)
Changes in v2:
- Rename -f/--fit and -F/--fit-out to -F/--fit and --outfile and clarify
FIT mode in the --help description (David Gibson, Simon Glass)
- Reject signed FIT images with signature* subnodes (Simon Glass)
- Use '<base-node-name>.dtb' for FIT base FDT descriptions instead of
common-prefix heuristics (Simon Glass)
- Compare FIT group node overlap against the group's common node set
instead of only the first entry (Simon Glass)
- Document in README.md that U-Boot's boot_get_fdt_fit() currently
requires FIT overlays to be uncompressed (Simon Glass)
README.md | 89 +++-
fdtfactor | 850 +++++++++++++++++++++++++++++++++----
tests/Makefile.tests | 2 +-
tests/fdtfactor_lz4.its | 109 +++++
tests/fdtfactor_other1.dts | 67 +++
tests/fdtfactor_other2.dts | 67 +++
tests/fdtfactor_test.its | 109 +++++
tests/run_tests.sh | 225 +++++++++-
8 files changed, 1421 insertions(+), 97 deletions(-)
create mode 100644 tests/fdtfactor_lz4.its
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
diff --git a/README.md b/README.md
index ac571384d421..304611eb763b 100644
--- a/README.md
+++ b/README.md
@@ -203,14 +203,14 @@ $ fdtoverlay -i base.dtb -o merged.dtb overlay1.dtbo overlay2.dtbo
## fdtfactor
-`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
-`.dtb.lzma`, `.dtb.gz` with `-C`) and splits them into a single common
-`base.dtb` and per-board `.dtbo` overlays so that applying a board's
-`.dtbo` to `base.dtb` recreates the original board device tree while
-reducing total storage size.
-
-Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`fdtfactor` is a Python tool built on `pylibfdt` that factors a set of
+compiled device tree blobs — either standalone `.dtb` files or
+`/images/fdt-*` nodes inside a compiled FIT — into a common base `.dtb`
+and per-board `.dtbo` overlays so that applying a board's `.dtbo` to the
+base `.dtb` recreates the original board device tree while reducing
+total storage size.
+
+Input `.dtb` blobs must be compiled with `dtc -@ -L` so that both
`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
and `/__local_fixups__` (used to identify phandle references inside
property values) are present.
@@ -219,30 +219,73 @@ Basic usage:
```
$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+$ fdtfactor -F image.fit -o image.factored.fit
```
-Options:
+Common options:
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ base `.dtb` and final reconstructed device trees. By default, `fdtfactor`
+ preserves `/__local_fixups__` on the base `.dtb` so that when
+ `fdt_overlay_apply()` merges a `.dtbo` overlay, the reconstructed device
+ tree retains full `/__local_fixups__` information (e.g., for an OS kernel
+ that wants to know which property cells contain phandles at runtime). When
+ `-L` was passed to `dtc` solely to help `fdtfactor` locate phandles at
+ build time and `/__local_fixups__` is not needed at runtime, passing `-S`
+ strips it from the base `.dtb` (and any unfactored FIT FDTs) to save space.
+* `-s`, `--stats`: Print size savings summary.
+
+### Standalone DTB Factoring
+
+When `-F` / `--fit` is not passed, `fdtfactor` takes two or more positional
+`.dtb` files (or compressed `.dtb.lz4`, `.dtb.xz`, `.dtb.lzma`, `.dtb.gz`
+files with `-C`) and writes a common base `.dtb` plus one `.dtbo` overlay per
+input file:
+
+* `-o`, `--output`: Output directory for generated `.dtbo` files and
+ `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo`
+ files, current directory for `--output-base-dtb`).
* `--output-base-dtb`: Output filename for the common base `.dtb`
(default: `base.dtb`, or `base.dtb.<ext>` when `-C` / `--compression` is
- used; placed in `--output` if specified, or the current directory
+ used; placed in `-o` / `--output` if specified, or the current directory
otherwise).
* `-C`, `--compression`: Compression format for both input `.dtb` files and
generated `base.dtb` / `.dtbo` outputs: `none`, `lz4`, `lz4_legacy`, `lzma`,
`xz`, or `gzip` (default: `none`).
-* `-o`, `--output`: Output directory for generated `.dtbo` files and
- `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
- current directory for `--output-base-dtb`).
-* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
- `base.dtb` and final reconstructed device trees. By default, `fdtfactor`
- preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
- merges a `.dtbo` overlay, the reconstructed device tree retains full
- `/__local_fixups__` information (e.g., for an OS kernel that wants to
- know which property cells contain phandles at runtime). When `-L` was
- passed to `dtc` solely to help `fdtfactor` locate phandles at build time
- and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
- from `base.dtb` to save space.
-* `-s`, `--stats`: Print size savings summary.
+
+### FIT Factoring
+
+When `-F` / `--fit` is passed, `fdtfactor` updates a compiled Flattened Image
+Tree:
+
+* `-F`, `--fit`: Input compiled FIT (`.fit` / `.itb`) to factor.
+* `-o`, `--output`: Output FIT path (default: overwrite `--fit` in place).
+* `--min-overlap`: Minimum percentage (`0`–`100`, default: `50`) of normalized
+ property bytes shared with a group's common properties required to place a
+ `flat_dt` node into that group.
+* `--single-base-dtb`: Force a single `/images/fdt-base-1` across all FDTs in
+ the FIT regardless of property overlap.
+
+FIT behavior and constraints:
+
+* Non-overlay `type = "flat_dt"` subnodes under `/images` are decompressed
+ according to their FIT `compression` property (`none`, `lz4`, `lzma`, `xz`,
+ `gzip`) and partitioned into factor groups that share compatible FDT headers,
+ `arch`, and `load` properties and at least `--min-overlap`% of normalized
+ property bytes (or forced into a single group when `--single-base-dtb` is
+ passed).
+* For each group, `fdtfactor` inserts `/images/fdt-base-N`, converts each
+ member `/images/fdt-*` node into a `.dtbo` overlay (preserving each node's
+ `compression` setting and stripping any `load` address), prepends
+ `fdt-base-N` to matching `/configurations/conf-*` `fdt` string lists
+ (populating `compatible` from the board DTB's root node when omitted), and
+ recomputes `/images/*/hash*` digests.
+* Note that while bootloaders such as depthcharge decompress compressed FIT
+ overlays, U-Boot's `boot_get_fdt_fit()` currently only decompresses the base
+ DTB and requires FIT overlays to be uncompressed.
+* Signed FITs containing `signature*` subnodes are rejected because modifying
+ FDT blobs and configurations would invalidate cryptographic signatures; sign
+ the FIT after factoring.
### TODO
diff --git a/fdtfactor b/fdtfactor
index beeeff231ba6..ebf40cdeaf92 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -8,16 +8,25 @@ properties shared across the inputs into a single common `base.dtb` and emits a
small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
original board DTB.
+
+Alternatively, `fdtfactor -F <image.fit>` can rewrite a compiled U-Boot /
+ChromeOS Flattened Image Tree (FIT) in place (or to `-o`/`--output`), factoring
+its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
+...) and overlays while updating `/configurations/conf-*` `fdt` and
+`compatible` properties and recalculating `/images/*/hash-*` digests.
"""
import argparse
from collections import Counter
+import copy
import gzip
+import hashlib
import lzma
import os
import struct
import subprocess
import sys
+import zlib
import libfdt
import libpydt
@@ -148,6 +157,24 @@ def _is_dtbo_overlay(root):
)
+def _open_fdt(raw_bytes, filename):
+ """Validate `raw_bytes` with `fdt_check_full()` and return an `FdtRo`.
+
+ Args:
+ raw_bytes: Uncompressed FDT blob `bytes`.
+ filename: File path or node label for error reporting.
+
+ Returns:
+ Validated `libfdt.FdtRo` instance.
+ """
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+ return fdt
+
+
class FactoredTree(libpydt.DeviceTree):
"""DeviceTree subclass with normalized phandle and fixup attributes.
@@ -186,30 +213,41 @@ class ParsedDtb:
independent of numeric phandle numbering.
"""
- def __init__(self, filename, comp='none'):
- """Parse an input FDT file into a normalized FactoredTree.
+ def __init__(self, filename, comp='none', fdt=None, root=None,
+ uncompressed_size=None, compressed_size=None):
+ """Parse an input FDT file or opened `FdtRo` into a `FactoredTree`.
Args:
- filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
+ filename: File path (or display name when `fdt` is provided).
comp: Compression format string (default: `'none'`).
+ fdt: Optional already-validated `libfdt.FdtRo` instance; if None,
+ read and decompressed from `filename`.
+ root: Optional already-unflattened `FactoredTree` for `fdt`.
+ uncompressed_size: Uncompressed byte length when `fdt` is provided.
+ compressed_size: Raw/compressed byte length when `fdt` is provided.
"""
self.filename = filename
self.compression = comp
- with open(filename, 'rb') as f:
- file_bytes = f.read()
- self.compressed_size = len(file_bytes)
- raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
- self.uncompressed_size = len(raw_bytes)
-
- try:
- fdt = libfdt.FdtRo(raw_bytes)
- libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
- except libfdt.FdtException as exc:
- raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+ if fdt is None:
+ with open(filename, 'rb') as f:
+ file_bytes = f.read()
+ self.compressed_size = len(file_bytes)
+ raw_bytes = decompress_blob(file_bytes, comp, filename=filename)
+ self.uncompressed_size = len(raw_bytes)
+ fdt = _open_fdt(raw_bytes, filename)
+ else:
+ self.uncompressed_size = (
+ fdt.totalsize() if uncompressed_size is None
+ else uncompressed_size
+ )
+ self.compressed_size = (
+ self.uncompressed_size if compressed_size is None
+ else compressed_size
+ )
self.boot_cpuid_phys = fdt.boot_cpuid_phys()
self.mem_rsv = tuple(fdt.get_mem_rsv_list())
- self.root = FactoredTree.unflatten(fdt)
+ self.root = FactoredTree.unflatten(fdt) if root is None else root
if _is_dtbo_overlay(self.root):
raise ValueError(
@@ -379,16 +417,12 @@ class ParsedDtb:
target_node.fixups[prop_name] = tuple(targets)
-def select_best_candidate(candidates, path, prop_name):
- """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
-
- Only candidates shared by at least two input DTBs (`count > 1`) are
- eligible for the base DTB; a value that appears in only one DTB would add
- one copy to `base.dtb` to save one overlay copy (zero net savings).
+def _prop_byte_cost(path, prop_name, prop_bytes, fixups):
+ """Estimate the FDT structure + `/__fixups__` byte cost of a property.
- Byte cost model per overlay occurrence:
+ Byte cost model per occurrence:
- Each FDT property carries a 12-byte structure header (`FDT_PROP` 4-byte
- tag + 4-byte length + 4-byte `nameoff`) plus `len(masked_bytes)` value
+ tag + 4-byte length + 4-byte `nameoff`) plus `len(prop_bytes)` value
bytes.
- Each phandle fixup in an overlay adds a string entry of the form
`"/fragment@0/__overlay__<path>:<prop_name>:<off>\\0"` to `/__fixups__`:
@@ -397,6 +431,26 @@ def select_best_candidate(candidates, path, prop_name):
~3 bytes amortized `/__fixups__` property overhead =
`len(path) + len(prop_name) + 28` bytes.
+ Args:
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+ prop_bytes: Property value `bytes`.
+ fixups: Sequence of `(offset, target_path)` phandle fixups.
+
+ Returns:
+ Estimated byte cost (int).
+ """
+ fixup_cost = len(fixups) * (len(path) + len(prop_name) + 28)
+ return len(prop_bytes) + fixup_cost + 12
+
+
+def select_best_candidate(candidates, path, prop_name):
+ """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
+
+ Only candidates shared by at least two input DTBs (`count > 1`) are
+ eligible for the base DTB; a value that appears in only one DTB would add
+ one copy to `base.dtb` to save one overlay copy (zero net savings).
+
Args:
candidates: Sequence of `(masked_bytes, fixups)` tuples observed across
input DTBs for `prop_name` on `path`.
@@ -408,7 +462,6 @@ def select_best_candidate(candidates, path, prop_name):
byte savings when placed in `base.dtb`, or `None` if no candidate
appears in at least two input DTBs.
"""
- fixup_entry_cost = len(path) + len(prop_name) + 28
counts = Counter(candidates)
best_val = None
best_key = None
@@ -416,10 +469,10 @@ def select_best_candidate(candidates, path, prop_name):
if count <= 1:
continue
masked_bytes, fixups = val
- length = len(masked_bytes) + len(fixups) * fixup_entry_cost
+ cost = _prop_byte_cost(path, prop_name, masked_bytes, fixups)
# Placing `val` in base.dtb adds 1 copy and removes `count` overlay
- # copies, saving `(count - 1) * (length + 12)` bytes across the set.
- key = ((count - 1) * (length + 12), count, length)
+ # copies, saving `(count - 1) * cost` bytes across the set.
+ key = ((count - 1) * cost, count, cost)
if best_key is None or key > best_key:
best_key = key
best_val = val
@@ -824,6 +877,579 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
+# ---------------------------------------------------------------------------
+# Compiled FIT factoring (targeting depthcharge / coreboot / U-Boot)
+# ---------------------------------------------------------------------------
+
+FIT_COMPRESSIONS = ('none', 'lz4', 'lzma', 'xz', 'gzip')
+LZ4_LEGACY_MAGIC = b'\x02\x21\x4c\x18'
+
+FIT_HASH_ALGOS = {
+ 'crc32': lambda data: struct.pack('>I', zlib.crc32(data) & 0xffffffff),
+ 'md5': lambda data: hashlib.md5(data).digest(),
+ 'sha1': lambda data: hashlib.sha1(data).digest(),
+ 'sha256': lambda data: hashlib.sha256(data).digest(),
+ 'sha384': lambda data: hashlib.sha384(data).digest(),
+ 'sha512': lambda data: hashlib.sha512(data).digest(),
+}
+
+
+def _compute_fit_hash(algo_name, data_bytes):
+ """Compute a binary hash digest for a FIT `hash*` subnode.
+
+ Args:
+ algo_name: FIT hash algorithm name (`'crc32'`, `'md5'`, `'sha1'`,
+ `'sha256'`, `'sha384'`, or `'sha512'`).
+ data_bytes: Input `bytes` to hash.
+
+ Returns:
+ Binary hash digest `bytes`.
+ """
+ hash_fn = FIT_HASH_ALGOS.get(algo_name)
+ if hash_fn is None:
+ raise ValueError(f'Unsupported FIT hash algorithm: {algo_name!r}')
+ return hash_fn(data_bytes)
+
+
+def _update_fit_hashes(img_node):
+ """Recompute `value` properties on any FIT `hash*` subnodes of `img_node`.
+
+ Args:
+ img_node: `DeviceTree` for a `/images/*` subnode whose `hash*` child
+ nodes should be updated in place from `img_node.props['data']`.
+ """
+ data_prop = img_node.props.get('data')
+ if data_prop is None:
+ return
+ for sub_name, sub_node in img_node.children.items():
+ if not sub_name.startswith('hash'):
+ continue
+ if 'algo' not in sub_node.props:
+ raise ValueError(
+ f'FIT hash subnode {sub_name!r} lacks an algo property'
+ )
+ algo = sub_node.props['algo'].as_str()
+ sub_node.set_prop(
+ libfdt.Property('value', _compute_fit_hash(algo, data_prop))
+ )
+
+
+def _rewrite_fit_dtbo_desc(desc, comp):
+ """Rewrite a `.dtb` filename description to `.dtbo`, preserving suffix.
+
+ Args:
+ desc: Description string from a FIT `/images/fdt-*` node.
+ comp: Compression format string of the FIT `/images/fdt-*` node.
+
+ Returns:
+ Rewritten `.dtbo` (or `.dtbo<ext>`) description string if `desc` has no
+ spaces and ends with `.dtb` (or `.dtb<ext>`), or `None` to leave
+ free-form descriptions unchanged.
+ """
+ if not desc or ' ' in desc:
+ return None
+ if desc.endswith('.dtb'):
+ return f'{desc[:-4]}.dtbo'
+ ext = COMP_TO_EXT[comp]
+ if ext and desc.endswith(f'.dtb{ext}'):
+ return f'{desc[:-len(ext) - 4]}.dtbo{ext}'
+ return None
+
+
+def _intern_dtb_props(dtb, prop_ids, id_costs):
+ """Intern each normalized `(path, name, prop, fixups)` in `dtb` to an int ID.
+
+ Args:
+ dtb: `ParsedDtb` instance.
+ prop_ids: Mutable dict mapping `(path, prop_name, prop, fixups)` tuples
+ to sequential integer IDs across DTBs.
+ id_costs: Mutable list mapping each integer ID to `_prop_byte_cost()`.
+
+ Returns:
+ Tuple of `(entry_ids, entry_bytes)` where `entry_ids` is the `set` of
+ integer property IDs present in `dtb` and `entry_bytes` is their total
+ `_prop_byte_cost()`.
+ """
+ entry_ids = set()
+ entry_bytes = 0
+ for path, node in dtb.nodes.items():
+ fixups_map = node.fixups
+ for name, prop in node.props.items():
+ fixups = fixups_map.get(name, ()) if fixups_map else ()
+ key = (path, name, prop, fixups)
+ pid = prop_ids.get(key)
+ if pid is None:
+ pid = len(id_costs)
+ prop_ids[key] = pid
+ id_costs.append(_prop_byte_cost(path, name, prop, fixups))
+ entry_ids.add(pid)
+ entry_bytes += id_costs[pid]
+ return entry_ids, entry_bytes
+
+
+def group_fit_dtbs(fdt_entries, min_overlap=50.0):
+ """Partition FIT `flat_dt` entries into compatible factor groups.
+
+ Entries sharing a `fdt-base-N` node must match on `(arch, load,
+ boot_cpuid_phys, mem_rsv)` (`load` is the optional physical RAM address
+ where bootloaders such as U-Boot copy the base FDT before applying
+ overlays, so two FDTs requiring different RAM load addresses cannot share a
+ single `fdt-base-N` node) and share at least `min_overlap`% of normalized
+ property bytes (`shared_bytes / union_bytes * 100`) against the group's
+ common property set. When multiple groups qualify, the group with the
+ highest overlap percentage is chosen.
+
+ Args:
+ fdt_entries: Sequence of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+ min_overlap: Minimum byte-weighted Jaccard overlap percentage
+ (`0.0` to `100.0`, default `50.0`).
+
+ Returns:
+ List of entry-dict lists, one per factor group.
+ """
+ prop_ids = {}
+ id_costs = []
+ groups = []
+
+ for entry in fdt_entries:
+ dtb = entry['dtb']
+ header_key = (
+ entry['node'].props.get('arch'),
+ entry['node'].props.get('load'),
+ dtb.boot_cpuid_phys,
+ dtb.mem_rsv,
+ )
+ entry_ids, entry_bytes = _intern_dtb_props(dtb, prop_ids, id_costs)
+
+ best_grp = None
+ best_overlap = -1.0
+ best_shared_ids = None
+ best_shared_bytes = 0
+
+ for grp in groups:
+ if grp['key'] != header_key:
+ continue
+ max_shared = min(entry_bytes, grp['common_bytes'])
+ max_union = entry_bytes + grp['common_bytes'] - max_shared
+ if max_union and (max_shared / max_union * 100.0) < min_overlap:
+ continue
+ shared_ids = grp['common_ids'] & entry_ids
+ shared_bytes = sum(id_costs[pid] for pid in shared_ids)
+ union_bytes = entry_bytes + grp['common_bytes'] - shared_bytes
+ overlap = (
+ shared_bytes / union_bytes * 100.0 if union_bytes else 100.0
+ )
+ if overlap >= min_overlap and overlap > best_overlap:
+ best_overlap = overlap
+ best_grp = grp
+ best_shared_ids = shared_ids
+ best_shared_bytes = shared_bytes
+
+ if best_grp is not None:
+ best_grp['common_ids'] = best_shared_ids
+ best_grp['common_bytes'] = best_shared_bytes
+ best_grp['entries'].append(entry)
+ else:
+ groups.append({
+ 'key': header_key,
+ 'common_ids': entry_ids,
+ 'common_bytes': entry_bytes,
+ 'entries': [entry],
+ })
+
+ return [grp['entries'] for grp in groups]
+
+
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+ """Extract and parse non-overlay `flat_dt` entries from `/images`.
+
+ Args:
+ fit_in_path: Input FIT path (for error reporting).
+ images_node: `DeviceTree` for the FIT `/images` node.
+ configs_node: `DeviceTree` for the FIT `/configurations` node.
+
+ Returns:
+ List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
+ `flat_dt` image subnodes.
+ """
+ for conf_name, conf_node in configs_node.children.items():
+ for sub_name in conf_node.children:
+ if sub_name.startswith(('signature', 'hash')):
+ raise ValueError(
+ f'{fit_in_path}: /configurations/{conf_name}/{sub_name} '
+ f'is not supported; sign the FIT after factoring'
+ )
+
+ fdt_entries = []
+ for img_name, img_node in images_node.children.items():
+ if ('data-offset' in img_node.props or
+ 'data-position' in img_node.props):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} uses external data '
+ f'(data-offset/data-position), which is not supported'
+ )
+ for sub_name in img_node.children:
+ if sub_name.startswith('signature'):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name}/{sub_name} is not '
+ f'supported; sign the FIT after factoring'
+ )
+ type_prop = img_node.props.get('type')
+ if type_prop is None or type_prop.as_str() != 'flat_dt':
+ continue
+ raw_data = img_node.props.get('data')
+ if raw_data is None:
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} lacks a data property'
+ )
+ node_label = f'{fit_in_path}:/images/{img_name}'
+ comp_prop = img_node.props.get('compression')
+ comp = (comp_prop.as_str() if comp_prop is not None else '') or 'none'
+ if comp not in FIT_COMPRESSIONS:
+ raise ValueError(
+ f'{node_label}: Unsupported FIT compression type: {comp}'
+ )
+ if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
+ comp = 'lz4_legacy'
+ raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+ fdt = _open_fdt(raw_bytes, node_label)
+ root = FactoredTree.unflatten(fdt)
+ if _is_dtbo_overlay(root):
+ continue
+
+ parsed_dtb = ParsedDtb(
+ node_label,
+ comp=comp,
+ fdt=fdt,
+ root=root,
+ uncompressed_size=len(raw_bytes),
+ compressed_size=len(raw_data),
+ )
+ fdt_entries.append({
+ 'name': img_name,
+ 'node': img_node,
+ 'dtb': parsed_dtb,
+ 'fdt': fdt,
+ })
+
+ if len(fdt_entries) < 2:
+ raise ValueError(
+ 'FIT must contain at least two non-overlay flat_dt '
+ 'nodes to factor'
+ )
+
+ return fdt_entries
+
+
+def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
+ """Partition `fdt_entries` into factorable groups and unfactored singletons.
+
+ Args:
+ fdt_entries: List of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+ single_base_dtb: If True, require all `fdt_entries` to share a single
+ base DTB group regardless of property overlap.
+ min_overlap: Minimum normalized property byte overlap percentage (`0.0`
+ to `100.0`) used when `single_base_dtb` is False.
+
+ Returns:
+ Tuple of `(factorable_groups, unfactored_entries)` where
+ `factorable_groups` is a list of entry-dict lists (each containing at
+ least 2 entries) and `unfactored_entries` is a list of single-board
+ entry dicts left unfactored.
+ """
+ if single_base_dtb:
+ ref_entry = fdt_entries[0]
+ for entry in fdt_entries[1:]:
+ for prop in ('arch', 'load'):
+ if (entry['node'].props.get(prop) !=
+ ref_entry['node'].props.get(prop)):
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to '
+ f'use the same {prop} property: {ref_entry["name"]} vs '
+ f'{entry["name"]}'
+ )
+ for entry in fdt_entries:
+ entry.pop('fdt', None)
+ return [fdt_entries], []
+
+ groups = group_fit_dtbs(fdt_entries, min_overlap=min_overlap)
+ factorable_groups = [g for g in groups if len(g) >= 2]
+ if not factorable_groups:
+ raise ValueError(
+ 'FIT must contain at least two compatible '
+ 'flat_dt nodes to factor'
+ )
+ for g in factorable_groups:
+ for entry in g:
+ entry.pop('fdt', None)
+ unfactored_entries = [g[0] for g in groups if len(g) == 1]
+ for entry in unfactored_entries:
+ print(
+ f'Note: FIT flat_dt node {entry["name"]} has no '
+ f'compatible peers and will be left unfactored',
+ file=sys.stderr,
+ )
+ return factorable_groups, unfactored_entries
+
+
+def _strip_fit_unfactored_local_fixups(unfactored_entries):
+ """Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
+
+ Args:
+ unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
+ unfactored by `_partition_fit_fdt_entries()`.
+ """
+ for entry in unfactored_entries:
+ fdt = entry.pop('fdt')
+ tree = libpydt.DeviceTree.unflatten(fdt)
+ if tree.children.pop('__local_fixups__', None) is None:
+ continue
+ stripped_fdt = tree.flatten(
+ mem_rsv=entry['dtb'].mem_rsv,
+ boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
+ )
+ comp_bytes = compress_blob(
+ bytes(stripped_fdt.as_bytearray()),
+ entry['dtb'].compression,
+ filename=f'/images/{entry["name"]}',
+ )
+ img_node = entry['node']
+ img_node.set_prop(libfdt.Property('data', comp_bytes))
+ _update_fit_hashes(img_node)
+
+
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+ """Factor one group of FIT `flat_dt` entries into a base node and overlays.
+
+ Args:
+ entries: List of FIT `flat_dt` entry dicts in this group.
+ base_dtb_node_name: Name for the new `/images/<base_dtb_node_name>`
+ node (e.g., `'fdt-base-1'`).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated base DTB.
+
+ Returns:
+ Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
+ constructed `DeviceTree` for `/images/<base_dtb_node_name>` and
+ `stat_args` is an argument tuple for `_print_dtb_stats()`.
+ """
+ group_dtbs = [e['dtb'] for e in entries]
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ group_dtbs, strip_local_fixups=strip_local_fixups
+ )
+ base_comp = group_dtbs[0].compression
+
+ base_dtb_comp_bytes = compress_blob(
+ base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+ )
+ base_dtb_node = copy.deepcopy(entries[0]['node'])
+ for prop in ('phandle', 'compatible'):
+ base_dtb_node.props.pop(prop, None)
+ base_dtb_node.children = {
+ k: v
+ for k, v in base_dtb_node.children.items()
+ if k.startswith('hash')
+ }
+ for sub_node in base_dtb_node.children.values():
+ sub_node.props.pop('phandle', None)
+ if 'description' in base_dtb_node.props:
+ first_desc = base_dtb_node.props['description'].as_str()
+ ext = COMP_TO_EXT[base_comp]
+ base_suffix = ext if (ext and first_desc.endswith(ext)) else ''
+ base_dtb_node.set_prop(
+ libfdt.Property.from_str(
+ 'description', f'{base_dtb_node_name}.dtb{base_suffix}'
+ )
+ )
+ base_dtb_node.set_prop(libfdt.Property('data', base_dtb_comp_bytes))
+ if 'compression' in base_dtb_node.props or base_comp != 'none':
+ comp_prop = 'lz4' if base_comp == 'lz4_legacy' else base_comp
+ base_dtb_node.set_prop(
+ libfdt.Property.from_str('compression', comp_prop)
+ )
+ _update_fit_hashes(base_dtb_node)
+
+ dtbo_comp_sizes = []
+ for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ img_node = entry['node']
+ entry_comp = entry['dtb'].compression
+ dtbo_comp_bytes = compress_blob(
+ dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
+ )
+ dtbo_comp_sizes.append(len(dtbo_comp_bytes))
+
+ if 'description' in img_node.props:
+ new_desc = _rewrite_fit_dtbo_desc(
+ img_node.props['description'].as_str(), entry_comp
+ )
+ if new_desc is not None:
+ img_node.set_prop(
+ libfdt.Property.from_str('description', new_desc)
+ )
+
+ # Strip 'load' from overlay image nodes so bootloaders do not copy an
+ # overlay onto the base FDT's RAM load address.
+ img_node.props.pop('load', None)
+
+ img_node.set_prop(libfdt.Property('data', dtbo_comp_bytes))
+ _update_fit_hashes(img_node)
+
+ stat_args = (
+ group_dtbs,
+ base_dtb_blob,
+ dtbo_blobs,
+ base_comp,
+ len(base_dtb_comp_bytes),
+ dtbo_comp_sizes,
+ f'[{base_dtb_node_name}] ',
+ )
+ return base_dtb_node, stat_args
+
+
+def _update_fit_images_and_configs(images_node, configs_node,
+ insert_before_first, fdt_to_base_dtb):
+ """Insert `fdt-base-*` nodes into `/images` and update `/configurations`.
+
+ Args:
+ images_node: `DeviceTree` for `/images` (mutated in place).
+ configs_node: `DeviceTree` for `/configurations` (mutated in place).
+ insert_before_first: Dict mapping the first `fdt-*` subnode name of
+ each group to `(base_dtb_node_name, base_dtb_node)`.
+ fdt_to_base_dtb: Dict mapping each factored `fdt-*` subnode name to
+ `(base_dtb_node_name, orig_parsed_dtb)`.
+ """
+ new_images_children = {}
+ for img_name, img_node in images_node.children.items():
+ if img_name in insert_before_first:
+ b_name, b_node = insert_before_first[img_name]
+ new_images_children[b_name] = b_node
+ new_images_children[img_name] = img_node
+ images_node.children = new_images_children
+
+ for conf_name, conf_node in configs_node.children.items():
+ if 'fdt' not in conf_node.props:
+ continue
+ fdt_list = conf_node.props['fdt'].as_stringlist()
+ if not fdt_list:
+ continue
+ primary_fdt = fdt_list[0]
+ for extra_fdt in fdt_list[1:]:
+ if extra_fdt in fdt_to_base_dtb:
+ raise ValueError(
+ f'/configurations/{conf_name} lists factored flat_dt '
+ f'{extra_fdt!r} after the primary FDT'
+ )
+ if primary_fdt not in fdt_to_base_dtb:
+ continue
+ base_dtb_name, orig_dtb = fdt_to_base_dtb[primary_fdt]
+ conf_node.set_prop(
+ libfdt.Property.from_stringlist('fdt', [base_dtb_name] + fdt_list)
+ )
+ # If the configuration omitted 'compatible' (relying on the
+ # bootloader reading root 'compatible' from fdt_list[0]), copy the
+ # board's root 'compatible' property onto the configuration node
+ # now that fdt_list[0] is the shared base DTB.
+ if 'compatible' in conf_node.props:
+ continue
+ root_compat = orig_dtb.root.props.get('compatible')
+ if root_compat is not None:
+ conf_node.set_prop(root_compat)
+
+
+def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
+ single_base_dtb=False, min_overlap=50.0, stats=False):
+ """Factor all `flat_dt` images in a compiled FIT into base + overlays.
+
+ Args:
+ fit_in_path: Path to the input compiled FIT.
+ fit_out_path: Path where the factored FIT should be written.
+ strip_local_fixups: If True, omit `/__local_fixups__` from generated
+ base DTBs.
+ single_base_dtb: If True, force all `flat_dt` nodes into a single
+ `fdt-base-1` base DTB.
+ min_overlap: Minimum normalized property byte overlap percentage
+ (`0.0` to `100.0`, default `50.0`) when `single_base_dtb` is False.
+ stats: If True, print per-group and overall FIT size savings.
+ """
+ with open(fit_in_path, 'rb') as f:
+ orig_fit_bytes = f.read()
+
+ orig_fit_fdt = _open_fdt(orig_fit_bytes, fit_in_path)
+ fit_root = libpydt.DeviceTree.unflatten(orig_fit_fdt)
+ images_node = fit_root.children.get('images')
+ configs_node = fit_root.children.get('configurations')
+ if images_node is None or configs_node is None:
+ raise ValueError(
+ f'{fit_in_path}: FIT must contain /images and '
+ f'/configurations nodes'
+ )
+
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node
+ )
+ factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb, min_overlap=min_overlap
+ )
+ if strip_local_fixups and unfactored_entries:
+ _strip_fit_unfactored_local_fixups(unfactored_entries)
+ else:
+ for entry in unfactored_entries:
+ entry.pop('fdt', None)
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+ used_image_names = set(images_node.children)
+ next_base_idx = 1
+
+ for entries in factorable_groups:
+ while f'fdt-base-{next_base_idx}' in used_image_names:
+ next_base_idx += 1
+ base_dtb_node_name = f'fdt-base-{next_base_idx}'
+ used_image_names.add(base_dtb_node_name)
+ next_base_idx += 1
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
+ group_stats.append(stat_args)
+
+ _update_fit_images_and_configs(
+ images_node, configs_node, insert_before_first, fdt_to_base_dtb
+ )
+
+ new_fit_fdt = fit_root.flatten(
+ mem_rsv=orig_fit_fdt.get_mem_rsv_list(),
+ boot_cpuid_phys=orig_fit_fdt.boot_cpuid_phys(),
+ )
+ new_fit_bytes = new_fit_fdt.as_bytearray()
+
+ out_dir = os.path.dirname(fit_out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(fit_out_path, 'wb') as f:
+ f.write(new_fit_bytes)
+
+ if stats:
+ for stat_args in group_stats:
+ _print_dtb_stats(*stat_args)
+ fit_saved = len(orig_fit_bytes) - len(new_fit_bytes)
+ fit_pct = (
+ (fit_saved / len(orig_fit_bytes) * 100.0) if orig_fit_bytes else 0.0
+ )
+ print(
+ f'FIT total: {len(orig_fit_bytes):,} B -> '
+ f'{len(new_fit_bytes):,} B '
+ f'(saved {fit_saved:,} B / {fit_pct:.1f}%)'
+ )
+
+
def get_dtbo_output_path(dtb_path, outdir, comp='none'):
"""Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
@@ -857,23 +1483,69 @@ def parse_args(argv=None):
Parsed `argparse.Namespace` instance.
"""
parser = argparse.ArgumentParser(
+ prog='fdtfactor',
+ usage=(
+ '%(prog)s [-h] [-o DIR] [-S] [-s] [--output-base-dtb FILENAME]\n'
+ ' [-C {none,lz4,lz4_legacy,lzma,xz,gzip}] '
+ 'DTB DTB [DTB ...]\n'
+ ' %(prog)s -F FIT_FILE [-o FIT_OUT] [-S] [-s] '
+ '[--min-overlap PERCENT]\n'
+ ' [--single-base-dtb]'
+ ),
description=(
'Split a set of compiled device tree blobs (.dtb, or compressed '
- '.dtb.lz4/.xz/.lzma/.gz with -C) into a single common base .dtb '
- 'and per-board overlay blobs (.dtbo).'
- )
+ '.dtb.lz4/.xz/.lzma/.gz with -C) into a common base .dtb and '
+ 'per-board overlay blobs (.dtbo), or update a compiled FIT '
+ '(-F/--fit) in place (or to -o/--output) by factoring its '
+ '/images/fdt-* blobs into base DTBs and overlays.'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--output',
+ default=None,
+ metavar='PATH',
+ help=(
+ 'Output directory in standalone DTB mode (default: alongside each '
+ 'input .dtb for .dtbo, current directory for --output-base-dtb), '
+ 'or output FIT path in FIT mode (default: overwrite --fit in '
+ 'place)'
+ ),
)
parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from generated base DTB(s) and '
+ 'reconstructed device trees'
+ ),
+ )
+ parser.add_argument(
+ '-s', '--stats',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+
+ dtb_group = parser.add_argument_group('standalone DTB mode')
+ dtb_group.add_argument(
+ 'dtbs',
+ nargs='*',
+ metavar='DTB',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
+ '(at least two required)'
+ ),
+ )
+ dtb_group.add_argument(
'--output-base-dtb',
default=None,
metavar='FILENAME',
help=(
'Output common base .dtb filename (default: base.dtb, or '
- 'base.dtb.<ext> when -C/--compression is used; placed in --output '
- 'if specified, or current directory otherwise)'
+ 'base.dtb.<ext> when -C/--compression is used; placed in '
+ '-o/--output if specified, or current directory otherwise)'
),
)
- parser.add_argument(
+ dtb_group.add_argument(
'-C', '--compression',
choices=tuple(COMP_TO_EXT),
default='none',
@@ -882,62 +1554,82 @@ def parse_args(argv=None):
'files (default: none)'
),
)
- parser.add_argument(
- '-o', '--output',
+
+ fit_group = parser.add_argument_group('FIT mode')
+ fit_group.add_argument(
+ '-F', '--fit',
default=None,
- metavar='DIR',
+ metavar='FIT_FILE',
help=(
- 'Output directory for generated .dtbo files and '
- '--output-base-dtb (default: alongside each input .dtb for .dtbo, '
- 'current directory for --output-base-dtb)'
+ 'Input compiled FIT (.fit / .itb) to factor (updated in place '
+ 'unless -o/--output is specified)'
),
)
- parser.add_argument(
- '-S', '--strip-local-fixups',
- action='store_true',
+ fit_group.add_argument(
+ '--min-overlap',
+ type=float,
+ default=None,
+ metavar='PERCENT',
help=(
- 'Omit /__local_fixups__ from the generated base .dtb and '
- 'reconstructed device trees'
+ 'Minimum shared normalized property byte overlap percentage '
+ '(0-100) to group FIT flat_dt nodes into the same base DTB '
+ '(default: 50)'
),
)
- parser.add_argument(
- '-s', '--stats',
+ fit_group.add_argument(
+ '--single-base-dtb',
action='store_true',
- help='Print size statistics summary',
- )
- parser.add_argument(
- 'dtbs',
- nargs='+',
- metavar='DTB',
help=(
- 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz with -C) files '
- '(at least two required)'
+ 'Force all flat_dt nodes in the FIT into a single fdt-base-1 base '
+ 'DTB, failing only if FDT headers, arch, or load differ'
),
)
args = parser.parse_args(argv)
- if args.output_base_dtb is None:
- args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
- if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ if args.single_base_dtb and not args.fit:
+ parser.error('--single-base-dtb requires -F/--fit')
+ if args.min_overlap is not None and not args.fit:
+ parser.error('--min-overlap requires -F/--fit')
+ if args.min_overlap is not None and args.single_base_dtb:
parser.error(
- '--output-base-dtb must be a non-empty filename without directory '
- 'components; use -o/--output to specify the output directory'
+ '--min-overlap cannot be combined with --single-base-dtb'
)
- if len(args.dtbs) < 2:
- parser.error('at least two input .dtb files are required')
+ if args.min_overlap is not None and not (0.0 <= args.min_overlap <= 100.0):
+ parser.error('--min-overlap must be between 0 and 100')
+ if args.min_overlap is None:
+ args.min_overlap = 50.0
+ if args.fit:
+ if (args.dtbs or args.output_base_dtb is not None or
+ args.compression != 'none'):
+ parser.error(
+ '-F/--fit cannot be combined with --output-base-dtb, '
+ '-C/--compression, or positional DTB arguments'
+ )
+ else:
+ if args.output_base_dtb is None:
+ args.output_base_dtb = f'base.dtb{COMP_TO_EXT[args.compression]}'
+ if not args.output_base_dtb or os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a non-empty filename without '
+ 'directory components; use -o/--output to specify the output '
+ 'directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
return args
-def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
- dtbo_csizes):
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp,
+ base_dtb_csize, dtbo_csizes, prefix=''):
"""Print uncompressed and compressed size savings summary.
Args:
dtbs: Sequence of input `ParsedDtb` instances.
base_dtb_blob: Uncompressed base DTB `bytes`.
dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
- comp: Compression format of the inputs and outputs.
+ base_dtb_comp: Compression format of the base DTB output.
base_dtb_csize: Byte size of the compressed base DTB output.
dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
+ prefix: Optional line prefix (e.g., `'[fdt-base-1] '`).
"""
orig_uncomp = sum(d.uncompressed_size for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -946,22 +1638,25 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, comp, base_dtb_csize,
saved_uncomp = orig_uncomp - new_uncomp
pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
print(
- f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'{prefix}Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
- if comp != 'none':
+ comps = sorted({base_dtb_comp, *(d.compression for d in dtbs)})
+ if any(c != 'none' for c in comps):
+ comp_label = '/'.join(comps)
orig_comp = sum(d.compressed_size for d in dtbs)
dtbo_csize = sum(dtbo_csizes)
new_comp = base_dtb_csize + dtbo_csize
saved_comp = orig_comp - new_comp
pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
print(
- f'Compressed ({comp}): {len(dtbs)} files ({orig_comp:,} B) -> '
- f'base {base_dtb_csize:,} B + {len(dtbo_blobs)} overlays '
- f'{dtbo_csize:,} B (avg {dtbo_csize // len(dtbo_blobs):,} B) = '
- f'{new_comp:,} B (saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ f'{prefix}Compressed ({comp_label}): {len(dtbs)} files '
+ f'({orig_comp:,} B) -> base {base_dtb_csize:,} B + '
+ f'{len(dtbo_blobs)} overlays {dtbo_csize:,} B '
+ f'(avg {dtbo_csize // len(dtbo_blobs):,} B) = {new_comp:,} B '
+ f'(saved {saved_comp:,} B / {pct_comp:.1f}%)'
)
@@ -1032,7 +1727,7 @@ def factor_dtb_files(args):
if args.stats:
_print_dtb_stats(
dtbs, base_dtb_blob, dtbo_blobs,
- comp=comp,
+ base_dtb_comp=comp,
base_dtb_csize=len(base_dtb_file_bytes),
dtbo_csizes=dtbo_file_sizes,
)
@@ -1049,7 +1744,18 @@ def main(argv=None):
"""
args = parse_args(argv)
try:
- factor_dtb_files(args)
+ if args.fit:
+ fit_out = args.output if args.output is not None else args.fit
+ factor_fit(
+ args.fit,
+ fit_out,
+ strip_local_fixups=args.strip_local_fixups,
+ single_base_dtb=args.single_base_dtb,
+ min_overlap=args.min_overlap,
+ stats=args.stats,
+ )
+ else:
+ factor_dtb_files(args)
except (OSError, ValueError, RuntimeError, struct.error,
libfdt.FdtException) as exc:
sys.exit(f'Error: {exc}')
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 1eed9ba6478a..351d4ed199b5 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,7 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 *.test.fit \
*.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
*.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
tmp.* *.bak \
diff --git a/tests/fdtfactor_lz4.its b/tests/fdtfactor_lz4.its
new file mode 100644
index 000000000000..4939223e57da
--- /dev/null
+++ b/tests/fdtfactor_lz4.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test LZ4 FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "lz4";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board3.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb.lz4";
+ data = /incbin/("overlay_overlay_phandle.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/fdtfactor_other1.dts b/tests/fdtfactor_other1.dts
new file mode 100644
index 000000000000..1396815974e5
--- /dev/null
+++ b/tests/fdtfactor_other1.dts
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other1", "vendor,other";
+ model = "other1";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+ other-shared-data = <
+ 0xfefdfcfb 0xfaf9f8f7 0xf6f5f4f3 0xf2f1f0ef
+ 0xeeedeceb 0xeae9e8e7 0xe6e5e4e3 0xe2e1e0df
+ 0xdedddcdb 0xdad9d8d7 0xd6d5d4d3 0xd2d1d0cf
+ 0xcecdcccb 0xcac9c8c7 0xc6c5c4c3 0xc2c1c0bf
+ 0xbebdbcbb 0xbab9b8b7 0xb6b5b4b3 0xb2b1b0af
+ 0xaeadacab 0xaaa9a8a7 0xa6a5a4a3 0xa2a1a09f
+ 0x9e9d9c9b 0x9a999897 0x96959493 0x9291908f
+ 0x8e8d8c8b 0x8a898887 0x86858483 0x8281807f
+ 0x7e7d7c7b 0x7a797877 0x76757473 0x7271706f
+ 0x6e6d6c6b 0x6a696867 0x66656463 0x6261605f
+ 0x5e5d5c5b 0x5a595857 0x56555453 0x5251504f
+ 0x4e4d4c4b 0x4a494847 0x46454443 0x4241403f
+ 0x3e3d3c3b 0x3a393837 0x36353433 0x3231302f
+ 0x2e2d2c2b 0x2a292827 0x26252423 0x2221201f
+ 0x1e1d1c1b 0x1a191817 0x16151413 0x1211100f
+ 0x0e0d0c0b 0x0a090807 0x06050403 0x020100ff
+ 0xfdfbf9f7 0xf5f3f1ef 0xede9e7e5 0xe3e1dfdd
+ 0xdbdad8d6 0xd4d2d0ce 0xcccbc9c7 0xc5c3c1bf
+ 0xbdbbb9b7 0xb5b3b1af 0xadaba9a7 0xa5a3a19f
+ 0x9d9b9997 0x9593918f 0x8d8b8987 0x8583817f
+ 0x7d7b7977 0x7573716f 0x6d6b6967 0x6563615f
+ 0x5d5b5957 0x5553514f 0x4d4b4947 0x4543413f
+ 0x3d3b3937 0x3533312f 0x2d2b2927 0x2523211f
+ 0x1d1b1917 0x1513110f 0x0d0b0907 0x050301fe
+ 0xfcf8f4f0 0xece8e4e0 0xdcd8d4d0 0xccc8c4c0
+ 0xbcb8b4b0 0xaca8a4a0 0x9c989490 0x8c888480
+ 0x7c787470 0x6c686460 0x5c585450 0x4c484440
+ 0x3c383430 0x2c282420 0x1c181410 0x0c080400
+ 0xfbf7f3ef 0xebe7e3df 0xdbd7d3cf 0xcbc7c3bf
+ 0xbbb7b3af 0xaba7a39f 0x9b97938f 0x8b87837f
+ 0x7b77736f 0x6b67635f 0x5b57534f 0x4b47433f
+ 0x3b37332f 0x2b27231f 0x1b17130f 0x0b0703ff
+ >;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x1111>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o1_node: other1-node {
+ prop-str = "other1";
+ ref-base = <&osn1>;
+ ref-self = <&o1_node>;
+ };
+};
diff --git a/tests/fdtfactor_other2.dts b/tests/fdtfactor_other2.dts
new file mode 100644
index 000000000000..c5ad9b53c047
--- /dev/null
+++ b/tests/fdtfactor_other2.dts
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other2", "vendor,other";
+ model = "other2";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+ other-shared-data = <
+ 0xfefdfcfb 0xfaf9f8f7 0xf6f5f4f3 0xf2f1f0ef
+ 0xeeedeceb 0xeae9e8e7 0xe6e5e4e3 0xe2e1e0df
+ 0xdedddcdb 0xdad9d8d7 0xd6d5d4d3 0xd2d1d0cf
+ 0xcecdcccb 0xcac9c8c7 0xc6c5c4c3 0xc2c1c0bf
+ 0xbebdbcbb 0xbab9b8b7 0xb6b5b4b3 0xb2b1b0af
+ 0xaeadacab 0xaaa9a8a7 0xa6a5a4a3 0xa2a1a09f
+ 0x9e9d9c9b 0x9a999897 0x96959493 0x9291908f
+ 0x8e8d8c8b 0x8a898887 0x86858483 0x8281807f
+ 0x7e7d7c7b 0x7a797877 0x76757473 0x7271706f
+ 0x6e6d6c6b 0x6a696867 0x66656463 0x6261605f
+ 0x5e5d5c5b 0x5a595857 0x56555453 0x5251504f
+ 0x4e4d4c4b 0x4a494847 0x46454443 0x4241403f
+ 0x3e3d3c3b 0x3a393837 0x36353433 0x3231302f
+ 0x2e2d2c2b 0x2a292827 0x26252423 0x2221201f
+ 0x1e1d1c1b 0x1a191817 0x16151413 0x1211100f
+ 0x0e0d0c0b 0x0a090807 0x06050403 0x020100ff
+ 0xfdfbf9f7 0xf5f3f1ef 0xede9e7e5 0xe3e1dfdd
+ 0xdbdad8d6 0xd4d2d0ce 0xcccbc9c7 0xc5c3c1bf
+ 0xbdbbb9b7 0xb5b3b1af 0xadaba9a7 0xa5a3a19f
+ 0x9d9b9997 0x9593918f 0x8d8b8987 0x8583817f
+ 0x7d7b7977 0x7573716f 0x6d6b6967 0x6563615f
+ 0x5d5b5957 0x5553514f 0x4d4b4947 0x4543413f
+ 0x3d3b3937 0x3533312f 0x2d2b2927 0x2523211f
+ 0x1d1b1917 0x1513110f 0x0d0b0907 0x050301fe
+ 0xfcf8f4f0 0xece8e4e0 0xdcd8d4d0 0xccc8c4c0
+ 0xbcb8b4b0 0xaca8a4a0 0x9c989490 0x8c888480
+ 0x7c787470 0x6c686460 0x5c585450 0x4c484440
+ 0x3c383430 0x2c282420 0x1c181410 0x0c080400
+ 0xfbf7f3ef 0xebe7e3df 0xdbd7d3cf 0xcbc7c3bf
+ 0xbbb7b3af 0xaba7a39f 0x9b97938f 0x8b87837f
+ 0x7b77736f 0x6b67635f 0x5b57534f 0x4b47433f
+ 0x3b37332f 0x2b27231f 0x1b17130f 0x0b0703ff
+ >;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x2222>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o2_node: other2-node {
+ prop-str = "other2";
+ ref-base = <&ossn1>;
+ ref-self = <&o2_node>;
+ };
+};
diff --git a/tests/fdtfactor_test.its b/tests/fdtfactor_test.its
new file mode 100644
index 000000000000..dd515e17b66f
--- /dev/null
+++ b/tests/fdtfactor_test.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "none";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb";
+ data = /incbin/("fdtfactor_board3.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb";
+ data = /incbin/("overlay_overlay_phandle.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 9b005396bc79..237e399803bb 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1371,6 +1371,229 @@ fdtfactor_tests () {
-o fdtfactor_comp_err.test.dir \
fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb
fi
+
+ # FIT factoring (uncompressed and compressed, multi-group by property
+ # byte overlap)
+ # fdtget -t r appends a trailing newline; strip that last byte.
+ fdtget_raw () {
+ $DTGET -t r "$1" "$2" "$3" > "$4.tmp"
+ head -c $(($($STATSZ "$4.tmp") - 1)) "$4.tmp" > "$4"
+ rm -f "$4.tmp"
+ }
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other1.test.dtb \
+ "$SRCDIR/fdtfactor_other1.dts"
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other2.test.dtb \
+ "$SRCDIR/fdtfactor_other2.dts"
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay_phandle.test.dtb \
+ "$SRCDIR/overlay_overlay_phandle.dts"
+
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor.test.fit \
+ "$SRCDIR/fdtfactor_test.its"
+ cp fdtfactor.test.fit fdtfactor_orig.test.fit
+ run_wrap_test $PYTHON $FDTFACTOR -s -F fdtfactor.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-1 description
+ run_fdtget_test "fdt-base-2.dtb" fdtfactor.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor.test.fit \
+ /configurations/conf-2 fdt
+ run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor.test.fit \
+ /configurations/conf-3 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor.test.fit \
+ /configurations/conf-4 fdt
+ run_fdtget_test "fdt-base-2 fdt-5" fdtfactor.test.fit \
+ /configurations/conf-5 fdt
+ # conf-1 already had an explicit compatible property; conf-2 did not, so
+ # fdtfactor should populate it from fdt-2's root compatible property.
+ run_fdtget_test "vendor,board1 vendor,common" fdtfactor.test.fit \
+ /configurations/conf-1 compatible
+ run_fdtget_test "test_fdtfactor" fdtfactor.test.fit \
+ /configurations/conf-2 compatible
+ run_fdtget_test "vendor,other1 vendor,other" fdtfactor.test.fit \
+ /configurations/conf-4 compatible
+ # Verify that hash-1/value was populated for fdt-base-1, fdt-base-2, and
+ # updated for fdt-1.
+ base1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-1/hash-1 value)
+ base2_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-2/hash-1 value)
+ fdt1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-1/hash-1 value)
+ if [ -z "$base1_hash" ] || [ -z "$base2_hash" ] || \
+ [ "$fdt1_hash" = "0 0 0 0 0 0 0 0" ]; then
+ FAIL "FIT hash-1 values were not updated"
+ fi
+ # Extract factored base DTBs and overlays from the FIT and verify all 5
+ # boards with fdtoverlay + dtbs_equal_unordered.
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-1 data fit_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor.test.fit /images/fdt-${i} data \
+ fit_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base1.test.dtb \
+ -o fit_merged${i}.test.dtb fit_fdt${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fit_merged${i}.test.dtb
+ done
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-2 data fit_base2.test.dtb
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor.test.fit /images/fdt-${idx} data \
+ fit_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base2.test.dtb \
+ -o fit_merged${idx}.test.dtb fit_fdt${idx}.test.dtbo
+ check_factored_dtb fdtfactor_other${i}.test.dtb \
+ fit_merged${idx}.test.dtb
+ done
+
+ # --min-overlap controls the shared property byte overlap threshold
+ run_wrap_test $PYTHON $FDTFACTOR --min-overlap 1 \
+ -F fdtfactor_orig.test.fit -o fdtfactor_minovl.test.fit
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_minovl.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-4" fdtfactor_minovl.test.fit \
+ /configurations/conf-4 fdt
+ run_wrap_error_test $PYTHON $FDTFACTOR --min-overlap 99 \
+ -F fdtfactor_orig.test.fit -o fdtfactor_minovl_hi.test.fit
+
+ # Unfactored singleton FIT entries retain /__local_fixups__ by default and
+ # strip it (updating hashes) when -S/--strip-local-fixups is passed.
+ cp fdtfactor_orig.test.fit fdtfactor_singleton_in.test.fit
+ run_wrap_test $DTPUT -r fdtfactor_singleton_in.test.fit \
+ /images/fdt-5 /configurations/conf-5
+ run_wrap_test $PYTHON $FDTFACTOR -F fdtfactor_singleton_in.test.fit \
+ -o fdtfactor_singleton_keep.test.fit
+ run_fdtget_test "fdt-4" fdtfactor_singleton_keep.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_singleton_keep.test.fit /images/fdt-4 data \
+ fit_singleton_keep.test.dtb
+ run_test check_path fit_singleton_keep.test.dtb exists "/__local_fixups__"
+ run_wrap_test $PYTHON $FDTFACTOR -S -F fdtfactor_singleton_in.test.fit \
+ -o fdtfactor_singleton_strip.test.fit
+ fdtget_raw fdtfactor_singleton_strip.test.fit /images/fdt-4 data \
+ fit_singleton_strip.test.dtb
+ run_test check_path fit_singleton_strip.test.dtb \
+ not-exists "/__local_fixups__"
+ run_test check_path fit_singleton_strip.test.dtb exists "/__symbols__"
+ fdt4_hash=$($DTGET -t bx fdtfactor_singleton_strip.test.fit \
+ /images/fdt-4/hash-1 value)
+ if [ -z "$fdt4_hash" ] || [ "$fdt4_hash" = "0 0 0 0 0 0 0 0" ]; then
+ FAIL "Singleton FIT hash-1 value was not updated under -S"
+ fi
+
+ # --single-base-dtb forces a single fdt-base-1 across all FDTs
+ run_wrap_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_orig.test.fit -o fdtfactor_single.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor_single.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_single.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-4" fdtfactor_single.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-base-1 data \
+ fit_single_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${i} data \
+ fit_single_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${i}.test.dtb fit_single_fdt${i}.test.dtbo
+ check_factored_dtb fdtfactor_board${i}.test.dtb \
+ fit_single_merged${i}.test.dtb
+ done
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${idx} data \
+ fit_single_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${idx}.test.dtb fit_single_fdt${idx}.test.dtbo
+ check_factored_dtb fdtfactor_other${i}.test.dtb \
+ fit_single_merged${idx}.test.dtb
+ done
+
+ # Verify that conflicting image properties (e.g. arch) in a group fail
+ cp fdtfactor_orig.test.fit fdtfactor_bad_arch.test.fit
+ run_wrap_test $DTPUT fdtfactor_bad_arch.test.fit /images/fdt-4 \
+ arch -ts "arm"
+ run_wrap_error_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_bad_arch.test.fit
+
+ # Verify that signed FITs (with signature* subnodes) and
+ # already-factored FITs (with no remaining compatible pairs) are rejected
+ cp fdtfactor_orig.test.fit fdtfactor_signed.test.fit
+ run_wrap_test $DTPUT fdtfactor_signed.test.fit -c \
+ /configurations/conf-1/signature-1
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor_signed.test.fit
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor.test.fit
+
+ # Pre-existing /images/fdt-base-* nodes do not collide: a second pass at
+ # --min-overlap 1 picks fdt-base-3 and produces a valid multi-stage chain.
+ run_wrap_test $PYTHON $FDTFACTOR -S --min-overlap 1 \
+ -F fdtfactor.test.fit -o fdtfactor_chained.test.fit
+ run_fdtget_test "fdt-base-3 fdt-base-1 fdt-1" fdtfactor_chained.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-3 fdt-base-2 fdt-4" fdtfactor_chained.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-base-3 data \
+ fit_chain_b3.test.dtb
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-base-1 data \
+ fit_chain_b1.test.dtbo
+ fdtget_raw fdtfactor_chained.test.fit /images/fdt-1 data \
+ fit_chain_f1.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_chain_b3.test.dtb \
+ -o fit_chain_merged1.test.dtb \
+ fit_chain_b1.test.dtbo fit_chain_f1.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.nolf.test.dtb \
+ fit_chain_merged1.test.dtb
+
+ if command -v lz4 >/dev/null 2>&1; then
+ # fdt-2 stays uncompressed in fdtfactor_lz4.its to verify mixed
+ # compression within a single FIT group.
+ run_wrap_test lz4 -12 -q -f -m \
+ fdtfactor_board1.test.dtb \
+ fdtfactor_board3.test.dtb fdtfactor_other1.test.dtb \
+ fdtfactor_other2.test.dtb overlay_overlay_phandle.test.dtb
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor_lz4.test.fit \
+ "$SRCDIR/fdtfactor_lz4.its"
+ run_wrap_test $PYTHON $FDTFACTOR -s \
+ -F fdtfactor_lz4.test.fit -o fdtfactor_lz4_out.test.fit
+ run_fdtget_test "fdt-base-1.dtb.lz4" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo.lz4" \
+ fdtfactor_lz4_out.test.fit /images/fdt-1 description
+ run_fdtget_test "fdtfactor_board2.test.dtbo" \
+ fdtfactor_lz4_out.test.fit /images/fdt-2 description
+ run_fdtget_test "fdt-base-2.dtb" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo" \
+ fdtfactor_lz4_out.test.fit /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-2 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-base-1 data \
+ fit_lz4_base1.test.dtb.lz4
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-1 data \
+ fit_lz4_fdt1.test.dtbo.lz4
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-2 data \
+ fit_lz4_fdt2.test.dtbo
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_base1.test.dtb.lz4
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_fdt1.test.dtbo.lz4
+ run_wrap_test $FDTOVERLAY -i fit_lz4_base1.test.dtb \
+ -o fit_lz4_merged1.test.dtb fit_lz4_fdt1.test.dtbo
+ check_factored_dtb fdtfactor_board1.test.dtb \
+ fit_lz4_merged1.test.dtb
+ run_wrap_test $FDTOVERLAY -i fit_lz4_base1.test.dtb \
+ -o fit_lz4_merged2.test.dtb fit_lz4_fdt2.test.dtbo
+ check_factored_dtb fdtfactor_board2.test.dtb \
+ fit_lz4_merged2.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1400,7 +1623,7 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.fit *.test.dtb.gz *.test.dtbo.gz \
*.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
*.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (12 preceding siblings ...)
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
@ 2026-11-16 20:53 ` Douglas Anderson
13 siblings, 0 replies; 15+ messages in thread
From: Douglas Anderson @ 2026-11-16 20:53 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Devicetree, Tudor Ambarus, Chen-Yu Tsai, Saravana Kannan,
David Gibson, André Draszik, Simon Glass, linux-kernel,
Brian Norris, Peter Griffin, Julius Werner, Douglas Anderson
When factoring dozens or hundreds of compressed DTBs (either standalone
with `-C` or inside a compiled FIT with `-F`), running `gzip`, `lzma`,
`xz`, or external `lz4` processes sequentially spends several seconds
compressing and decompressing blobs one at a time.
Because Python's `zlib` and `_lzma` C extensions and `subprocess.run()`
for `lz4` all release the GIL, add a small `_map_blobs()` helper using
`concurrent.futures.ThreadPoolExecutor` to decompress input blobs and
compress output base/overlay blobs across worker threads in
deterministic input order whenever multiple compressed blobs are
processed.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch parallelizing blob compression and decompression across
worker threads
fdtfactor | 190 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 138 insertions(+), 52 deletions(-)
diff --git a/fdtfactor b/fdtfactor
index ebf40cdeaf92..ab59763d8863 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -18,6 +18,7 @@ its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
import argparse
from collections import Counter
+import concurrent.futures
import copy
import gzip
import hashlib
@@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+def _map_blobs(func, tasks, pool=None):
+ """Apply `func(blob, comp, filename)` across `tasks`, parallelizing if compressed.
+
+ Python's `zlib` and `_lzma` C extensions and external `lz4` subprocesses
+ release the GIL, so running multiple compressed tasks across a
+ `ThreadPoolExecutor` overlaps compression and decompression across CPU
+ cores while preserving deterministic input order.
+
+ Args:
+ func: `decompress_blob` or `compress_blob`.
+ tasks: Sequence of `(blob_bytes, comp, filename)` tuples.
+ pool: Optional shared `ThreadPoolExecutor` instance.
+
+ Returns:
+ List of processed `bytes` in the same order as `tasks`.
+ """
+ if len(tasks) < 2 or not any(
+ comp not in ('none', '', None) for _, comp, _ in tasks
+ ):
+ return [func(*t) for t in tasks]
+ if pool is not None:
+ return list(pool.map(lambda t: func(*t), tasks))
+ with concurrent.futures.ThreadPoolExecutor() as new_pool:
+ return list(new_pool.map(lambda t: func(*t), tasks))
+
+
def _is_dtbo_overlay(root):
"""Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
@@ -1061,13 +1088,14 @@ def group_fit_dtbs(fdt_entries, min_overlap=50.0):
return [grp['entries'] for grp in groups]
-def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node, pool=None):
"""Extract and parse non-overlay `flat_dt` entries from `/images`.
Args:
fit_in_path: Input FIT path (for error reporting).
images_node: `DeviceTree` for the FIT `/images` node.
configs_node: `DeviceTree` for the FIT `/configurations` node.
+ pool: Optional shared `ThreadPoolExecutor` for decompression.
Returns:
List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
@@ -1081,7 +1109,7 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
f'is not supported; sign the FIT after factoring'
)
- fdt_entries = []
+ candidates = []
for img_name, img_node in images_node.children.items():
if ('data-offset' in img_node.props or
'data-position' in img_node.props):
@@ -1112,7 +1140,19 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
)
if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
comp = 'lz4_legacy'
- raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+ candidates.append((img_name, img_node, node_label, comp, raw_data))
+
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(raw_data, comp, node_label)
+ for _, _, node_label, comp, raw_data in candidates],
+ pool=pool,
+ )
+
+ fdt_entries = []
+ for (img_name, img_node, node_label, comp, raw_data), raw_bytes in zip(
+ candidates, raw_blobs
+ ):
fdt = _open_fdt(raw_bytes, node_label)
root = FactoredTree.unflatten(fdt)
if _is_dtbo_overlay(root):
@@ -1194,13 +1234,15 @@ def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
return factorable_groups, unfactored_entries
-def _strip_fit_unfactored_local_fixups(unfactored_entries):
+def _strip_fit_unfactored_local_fixups(unfactored_entries, pool=None):
"""Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
Args:
unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
unfactored by `_partition_fit_fdt_entries()`.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
"""
+ to_recompress = []
for entry in unfactored_entries:
fdt = entry.pop('fdt')
tree = libpydt.DeviceTree.unflatten(fdt)
@@ -1210,17 +1252,26 @@ def _strip_fit_unfactored_local_fixups(unfactored_entries):
mem_rsv=entry['dtb'].mem_rsv,
boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
)
- comp_bytes = compress_blob(
- bytes(stripped_fdt.as_bytearray()),
+ stripped_bytes = bytes(stripped_fdt.as_bytearray())
+ to_recompress.append((
+ entry['node'],
+ stripped_bytes,
entry['dtb'].compression,
- filename=f'/images/{entry["name"]}',
- )
- img_node = entry['node']
+ f'/images/{entry["name"]}',
+ ))
+
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [(blob, comp, label) for _, blob, comp, label in to_recompress],
+ pool=pool,
+ )
+ for (img_node, _, _, _), comp_bytes in zip(to_recompress, comp_blobs):
img_node.set_prop(libfdt.Property('data', comp_bytes))
_update_fit_hashes(img_node)
-def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups,
+ pool=None):
"""Factor one group of FIT `flat_dt` entries into a base node and overlays.
Args:
@@ -1229,6 +1280,7 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
node (e.g., `'fdt-base-1'`).
strip_local_fixups: If True, omit `/__local_fixups__` from the
generated base DTB.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
Returns:
Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
@@ -1241,9 +1293,20 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
)
base_comp = group_dtbs[0].compression
- base_dtb_comp_bytes = compress_blob(
- base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, base_comp, f'/images/{base_dtb_node_name}'),
+ *(
+ (dtbo_blob, entry['dtb'].compression, f'/images/{entry["name"]}')
+ for entry, dtbo_blob in zip(entries, dtbo_blobs)
+ ),
+ ],
+ pool=pool,
)
+ base_dtb_comp_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
base_dtb_node = copy.deepcopy(entries[0]['node'])
for prop in ('phandle', 'compatible'):
base_dtb_node.props.pop(prop, None)
@@ -1272,12 +1335,9 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
_update_fit_hashes(base_dtb_node)
dtbo_comp_sizes = []
- for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ for entry, dtbo_comp_bytes in zip(entries, dtbo_comp_blobs):
img_node = entry['node']
entry_comp = entry['dtb'].compression
- dtbo_comp_bytes = compress_blob(
- dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
- )
dtbo_comp_sizes.append(len(dtbo_comp_bytes))
if 'description' in img_node.props:
@@ -1386,39 +1446,42 @@ def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
f'/configurations nodes'
)
- fdt_entries = _extract_fit_fdt_entries(
- fit_in_path, images_node, configs_node
- )
- factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
- fdt_entries, single_base_dtb, min_overlap=min_overlap
- )
- if strip_local_fixups and unfactored_entries:
- _strip_fit_unfactored_local_fixups(unfactored_entries)
- else:
- for entry in unfactored_entries:
- entry.pop('fdt', None)
-
- fdt_to_base_dtb = {}
- insert_before_first = {}
- group_stats = []
- used_image_names = set(images_node.children)
- next_base_idx = 1
-
- for entries in factorable_groups:
- while f'fdt-base-{next_base_idx}' in used_image_names:
- next_base_idx += 1
- base_dtb_node_name = f'fdt-base-{next_base_idx}'
- used_image_names.add(base_dtb_node_name)
- next_base_idx += 1
- base_dtb_node, stat_args = _factor_fit_group(
- entries, base_dtb_node_name, strip_local_fixups
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node, pool=pool
)
- insert_before_first[entries[0]['name']] = (
- base_dtb_node_name, base_dtb_node
+ factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb, min_overlap=min_overlap
)
- for entry in entries:
- fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
- group_stats.append(stat_args)
+ if strip_local_fixups and unfactored_entries:
+ _strip_fit_unfactored_local_fixups(unfactored_entries, pool=pool)
+ else:
+ for entry in unfactored_entries:
+ entry.pop('fdt', None)
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+ used_image_names = set(images_node.children)
+ next_base_idx = 1
+
+ for entries in factorable_groups:
+ while f'fdt-base-{next_base_idx}' in used_image_names:
+ next_base_idx += 1
+ base_dtb_node_name = f'fdt-base-{next_base_idx}'
+ used_image_names.add(base_dtb_node_name)
+ next_base_idx += 1
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups, pool=pool
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (
+ base_dtb_node_name, entry['dtb']
+ )
+ group_stats.append(stat_args)
_update_fit_images_and_configs(
images_node, configs_node, insert_before_first, fdt_to_base_dtb
@@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
"""
comp = args.compression
input_dtb_paths = args.dtbs
- dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
+ file_blobs = []
+ for p in input_dtb_paths:
+ with open(p, 'rb') as f:
+ file_blobs.append(f.read())
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(b, comp, p) for p, b in zip(input_dtb_paths, file_blobs)],
+ )
+ dtbs = [
+ ParsedDtb(
+ p,
+ comp=comp,
+ fdt=_open_fdt(raw, p),
+ uncompressed_size=len(raw),
+ compressed_size=len(fblob),
+ )
+ for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
+ ]
base_dtb_path = (
os.path.join(args.output, args.output_base_dtb)
@@ -1706,20 +1786,26 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
- base_dtb_file_bytes = compress_blob(
- base_dtb_blob, comp, filename=base_dtb_path
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, comp, base_dtb_path),
+ *((b, comp, p) for p, b in zip(dtbo_paths, dtbo_blobs)),
+ ],
)
+ base_dtb_file_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
if args.output:
os.makedirs(args.output, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
f.write(base_dtb_file_bytes)
dtbo_file_sizes = []
- for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ for out_path, out_bytes in zip(dtbo_paths, dtbo_comp_blobs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
- out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
f.write(out_bytes)
--
2.56.0.679.gc527c65306-goog
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-11-16 20:54 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression Douglas Anderson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®