* [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:29 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr() Douglas Anderson
` (14 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
A malformed overlay with a negative or very large fixup offset in
/__fixups__ (such as "/fragment@0:target:-4" or
"/fragment@0:target:4294967292") can cause an out-of-bounds write during
overlay application:
1. overlay_fixup_phandle() parses the offset with strtoul(), which
accepts negative numbers and values larger than UINT32_MAX, and
truncates the result into a 32-bit integer.
2. fdt_setprop_inplace_namelen_partial() checks bounds with
`(unsigned)proplen < (len + idx)`, where `len + idx` wraps around to
a small number when `idx` is near UINT32_MAX, bypassing the check and
passing the huge offset to memcpy().
Tighten the fixup offset parsing in overlay_fixup_phandle(), fix the
bounds check in fdt_setprop_inplace_namelen_partial() to avoid unsigned
wrap-around, and add test cases covering both negative and overflowing
offsets.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Split the fixup offset overflow fix and fdt_wip.c bounds check out of
"libfdt: overlay: Prepare fixup and fragment path helpers for reuse"
into a standalone bugfix patch with dedicated tests (Simon Glass)
- Add a comment explaining why an initial ASCII digit check is needed
before strtoul() (David Gibson)
libfdt/fdt_overlay.c | 34 +++++++++++++++++-----
libfdt/fdt_wip.c | 7 ++++-
tests/overlay_bad_fixup_negative_index.dts | 9 ++++++
tests/overlay_bad_fixup_overflow_index.dts | 9 ++++++
tests/run_tests.sh | 2 ++
tests/setprop_inplace.c | 8 +++++
6 files changed, 60 insertions(+), 9 deletions(-)
create mode 100644 tests/overlay_bad_fixup_negative_index.dts
create mode 100644 tests/overlay_bad_fixup_overflow_index.dts
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 51a3859620a4..c6dc4774cff3 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -330,10 +330,10 @@ static int overlay_update_local_references(void *fdto, uint32_t delta)
static int overlay_fixup_one_phandle(void *fdto, int symbols_off,
const char *path, uint32_t path_len,
const char *name, uint32_t name_len,
- int poffset, uint32_t phandle)
+ uint32_t poffset, uint32_t phandle)
{
fdt32_t phandle_prop;
- int fixup_off;
+ int fixup_off, ret;
if (symbols_off < 0)
return symbols_off;
@@ -345,10 +345,20 @@ static int overlay_fixup_one_phandle(void *fdto, int symbols_off,
return fixup_off;
phandle_prop = cpu_to_fdt32(phandle);
- return fdt_setprop_inplace_namelen_partial(fdto, fixup_off,
- name, name_len, poffset,
- &phandle_prop,
- sizeof(phandle_prop));
+ ret = fdt_setprop_inplace_namelen_partial(fdto, fixup_off,
+ name, name_len, poffset,
+ &phandle_prop,
+ sizeof(phandle_prop));
+ /*
+ * fdt_setprop_inplace_namelen_partial() only returns -FDT_ERR_NOSPACE
+ * when poffset + sizeof(phandle_prop) exceeds the existing property's
+ * length in the overlay, which means the overlay is malformed (not
+ * that the base DTB buffer ran out of space).
+ */
+ if (ret == -FDT_ERR_NOSPACE)
+ return -FDT_ERR_BADOVERLAY;
+
+ return ret;
}
/**
@@ -409,7 +419,8 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
uint32_t fixup_len;
const char *sep;
char *endptr;
- int poffset, ret;
+ unsigned long poffset;
+ int ret;
fixup_end = memchr(value, '\0', len);
if (!fixup_end)
@@ -438,8 +449,15 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
if (!name_len)
return -FDT_ERR_BADOVERLAY;
+ /*
+ * Require an initial ASCII digit because strtoul() accepts
+ * leading whitespace and '+'/'-' signs (wrapping negative
+ * values to unsigned long).
+ */
+ if ((*(sep + 1) < '0') || (*(sep + 1) > '9'))
+ return -FDT_ERR_BADOVERLAY;
poffset = strtoul(sep + 1, &endptr, 10);
- if ((*endptr != '\0') || (endptr <= (sep + 1)))
+ if ((*endptr != '\0') || (poffset > UINT32_MAX))
return -FDT_ERR_BADOVERLAY;
ret = overlay_fixup_one_phandle(fdto, symbols_off,
diff --git a/libfdt/fdt_wip.c b/libfdt/fdt_wip.c
index c2d7566a67dc..dd163d91498c 100644
--- a/libfdt/fdt_wip.c
+++ b/libfdt/fdt_wip.c
@@ -23,7 +23,12 @@ int fdt_setprop_inplace_namelen_partial(void *fdt, int nodeoffset,
if (!propval)
return proplen;
- if ((unsigned)proplen < (len + idx))
+ /*
+ * Check that idx + len fits within proplen using subtraction rather
+ * than addition so that a large idx (or negative len) cannot wrap
+ * around 32-bit unsigned arithmetic.
+ */
+ if (idx > (unsigned)proplen || (unsigned)len > (unsigned)proplen - idx)
return -FDT_ERR_NOSPACE;
memcpy((char *)propval + idx, val, len);
diff --git a/tests/overlay_bad_fixup_negative_index.dts b/tests/overlay_bad_fixup_negative_index.dts
new file mode 100644
index 000000000000..02ccee905e85
--- /dev/null
+++ b/tests/overlay_bad_fixup_negative_index.dts
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+/include/ "overlay_bad_fixup_base.dtsi"
+
+/ {
+ __fixups__ {
+ test = "/fragment@0:target:-4";
+ };
+};
diff --git a/tests/overlay_bad_fixup_overflow_index.dts b/tests/overlay_bad_fixup_overflow_index.dts
new file mode 100644
index 000000000000..d7d05267d7b3
--- /dev/null
+++ b/tests/overlay_bad_fixup_overflow_index.dts
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+/include/ "overlay_bad_fixup_base.dtsi"
+
+/ {
+ __fixups__ {
+ test = "/fragment@0:target:4294967292";
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 5bc7817dd12a..0d815e307909 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -274,6 +274,8 @@ BAD_FIXUP_TREES="bad_index \
empty \
empty_index \
index_trailing \
+ negative_index \
+ overflow_index \
path_empty_prop \
path_only \
path_only_sep \
diff --git a/tests/setprop_inplace.c b/tests/setprop_inplace.c
index 61a0da16fce8..48182169fbf6 100644
--- a/tests/setprop_inplace.c
+++ b/tests/setprop_inplace.c
@@ -80,5 +80,13 @@ int main(int argc, char *argv[])
check_getprop(fdt, 0, "compatible", strlen(TEST_STRING_4_RESULT) + 1,
TEST_STRING_4_RESULT);
+ err = fdt_setprop_inplace_namelen_partial(fdt, 0, "compatible",
+ strlen("compatible"),
+ 0xfffffffcU,
+ TEST_STRING_4_PARTIAL, 4);
+ if (err != -FDT_ERR_NOSPACE)
+ FAIL("fdt_setprop_inplace_namelen_partial() with wrapping idx returned %d instead of -FDT_ERR_NOSPACE\n",
+ err);
+
PASS();
}
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset
2026-11-01 22:01 ` [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset Douglas Anderson
@ 2026-11-02 4:29 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:29 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 7604 bytes --]
On Sun, Nov 01, 2026 at 02:01:08PM -0800, Douglas Anderson wrote:
> A malformed overlay with a negative or very large fixup offset in
> /__fixups__ (such as "/fragment@0:target:-4" or
> "/fragment@0:target:4294967292") can cause an out-of-bounds write during
> overlay application:
>
> 1. overlay_fixup_phandle() parses the offset with strtoul(), which
> accepts negative numbers and values larger than UINT32_MAX, and
> truncates the result into a 32-bit integer.
> 2. fdt_setprop_inplace_namelen_partial() checks bounds with
> `(unsigned)proplen < (len + idx)`, where `len + idx` wraps around to
> a small number when `idx` is near UINT32_MAX, bypassing the check and
> passing the huge offset to memcpy().
>
> Tighten the fixup offset parsing in overlay_fixup_phandle(), fix the
> bounds check in fdt_setprop_inplace_namelen_partial() to avoid unsigned
> wrap-around, and add test cases covering both negative and overflowing
> offsets.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Thanks for this - this bug has been reported by a couple of people,
and I hadn't gotten around to making my own fix.
> ---
>
> Changes in v3:
> - Split the fixup offset overflow fix and fdt_wip.c bounds check out of
> "libfdt: overlay: Prepare fixup and fragment path helpers for reuse"
> into a standalone bugfix patch with dedicated tests (Simon Glass)
> - Add a comment explaining why an initial ASCII digit check is needed
> before strtoul() (David Gibson)
>
> libfdt/fdt_overlay.c | 34 +++++++++++++++++-----
> libfdt/fdt_wip.c | 7 ++++-
> tests/overlay_bad_fixup_negative_index.dts | 9 ++++++
> tests/overlay_bad_fixup_overflow_index.dts | 9 ++++++
> tests/run_tests.sh | 2 ++
> tests/setprop_inplace.c | 8 +++++
> 6 files changed, 60 insertions(+), 9 deletions(-)
> create mode 100644 tests/overlay_bad_fixup_negative_index.dts
> create mode 100644 tests/overlay_bad_fixup_overflow_index.dts
>
> diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
> index 51a3859620a4..c6dc4774cff3 100644
> --- a/libfdt/fdt_overlay.c
> +++ b/libfdt/fdt_overlay.c
> @@ -330,10 +330,10 @@ static int overlay_update_local_references(void *fdto, uint32_t delta)
> static int overlay_fixup_one_phandle(void *fdto, int symbols_off,
> const char *path, uint32_t path_len,
> const char *name, uint32_t name_len,
> - int poffset, uint32_t phandle)
> + uint32_t poffset, uint32_t phandle)
> {
> fdt32_t phandle_prop;
> - int fixup_off;
> + int fixup_off, ret;
>
> if (symbols_off < 0)
> return symbols_off;
> @@ -345,10 +345,20 @@ static int overlay_fixup_one_phandle(void *fdto, int symbols_off,
> return fixup_off;
>
> phandle_prop = cpu_to_fdt32(phandle);
> - return fdt_setprop_inplace_namelen_partial(fdto, fixup_off,
> - name, name_len, poffset,
> - &phandle_prop,
> - sizeof(phandle_prop));
> + ret = fdt_setprop_inplace_namelen_partial(fdto, fixup_off,
> + name, name_len, poffset,
> + &phandle_prop,
> + sizeof(phandle_prop));
> + /*
> + * fdt_setprop_inplace_namelen_partial() only returns -FDT_ERR_NOSPACE
> + * when poffset + sizeof(phandle_prop) exceeds the existing property's
> + * length in the overlay, which means the overlay is malformed (not
> + * that the base DTB buffer ran out of space).
> + */
> + if (ret == -FDT_ERR_NOSPACE)
> + return -FDT_ERR_BADOVERLAY;
> +
> + return ret;
> }
>
> /**
> @@ -409,7 +419,8 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
> uint32_t fixup_len;
> const char *sep;
> char *endptr;
> - int poffset, ret;
> + unsigned long poffset;
> + int ret;
>
> fixup_end = memchr(value, '\0', len);
> if (!fixup_end)
> @@ -438,8 +449,15 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
> if (!name_len)
> return -FDT_ERR_BADOVERLAY;
>
> + /*
> + * Require an initial ASCII digit because strtoul() accepts
> + * leading whitespace and '+'/'-' signs (wrapping negative
> + * values to unsigned long).
> + */
> + if ((*(sep + 1) < '0') || (*(sep + 1) > '9'))
> + return -FDT_ERR_BADOVERLAY;
> poffset = strtoul(sep + 1, &endptr, 10);
> - if ((*endptr != '\0') || (endptr <= (sep + 1)))
> + if ((*endptr != '\0') || (poffset > UINT32_MAX))
> return -FDT_ERR_BADOVERLAY;
>
> ret = overlay_fixup_one_phandle(fdto, symbols_off,
> diff --git a/libfdt/fdt_wip.c b/libfdt/fdt_wip.c
> index c2d7566a67dc..dd163d91498c 100644
> --- a/libfdt/fdt_wip.c
> +++ b/libfdt/fdt_wip.c
> @@ -23,7 +23,12 @@ int fdt_setprop_inplace_namelen_partial(void *fdt, int nodeoffset,
> if (!propval)
> return proplen;
>
> - if ((unsigned)proplen < (len + idx))
> + /*
> + * Check that idx + len fits within proplen using subtraction rather
> + * than addition so that a large idx (or negative len) cannot wrap
> + * around 32-bit unsigned arithmetic.
> + */
> + if (idx > (unsigned)proplen || (unsigned)len > (unsigned)proplen - idx)
> return -FDT_ERR_NOSPACE;
>
> memcpy((char *)propval + idx, val, len);
> diff --git a/tests/overlay_bad_fixup_negative_index.dts b/tests/overlay_bad_fixup_negative_index.dts
> new file mode 100644
> index 000000000000..02ccee905e85
> --- /dev/null
> +++ b/tests/overlay_bad_fixup_negative_index.dts
> @@ -0,0 +1,9 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +
> +/include/ "overlay_bad_fixup_base.dtsi"
> +
> +/ {
> + __fixups__ {
> + test = "/fragment@0:target:-4";
> + };
> +};
> diff --git a/tests/overlay_bad_fixup_overflow_index.dts b/tests/overlay_bad_fixup_overflow_index.dts
> new file mode 100644
> index 000000000000..d7d05267d7b3
> --- /dev/null
> +++ b/tests/overlay_bad_fixup_overflow_index.dts
> @@ -0,0 +1,9 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +
> +/include/ "overlay_bad_fixup_base.dtsi"
> +
> +/ {
> + __fixups__ {
> + test = "/fragment@0:target:4294967292";
> + };
> +};
> diff --git a/tests/run_tests.sh b/tests/run_tests.sh
> index 5bc7817dd12a..0d815e307909 100755
> --- a/tests/run_tests.sh
> +++ b/tests/run_tests.sh
> @@ -274,6 +274,8 @@ BAD_FIXUP_TREES="bad_index \
> empty \
> empty_index \
> index_trailing \
> + negative_index \
> + overflow_index \
> path_empty_prop \
> path_only \
> path_only_sep \
> diff --git a/tests/setprop_inplace.c b/tests/setprop_inplace.c
> index 61a0da16fce8..48182169fbf6 100644
> --- a/tests/setprop_inplace.c
> +++ b/tests/setprop_inplace.c
> @@ -80,5 +80,13 @@ int main(int argc, char *argv[])
> check_getprop(fdt, 0, "compatible", strlen(TEST_STRING_4_RESULT) + 1,
> TEST_STRING_4_RESULT);
>
> + err = fdt_setprop_inplace_namelen_partial(fdt, 0, "compatible",
> + strlen("compatible"),
> + 0xfffffffcU,
> + TEST_STRING_4_PARTIAL, 4);
> + if (err != -FDT_ERR_NOSPACE)
> + FAIL("fdt_setprop_inplace_namelen_partial() with wrapping idx returned %d instead of -FDT_ERR_NOSPACE\n",
> + err);
> +
> PASS();
> }
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr()
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:30 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets Douglas Anderson
` (13 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
In `overlay_fixup_phandle()`, `sep` is obtained via `memchr(..., ':',
...)`. When `sep` is non-NULL, `*sep` is guaranteed to be `':'`, so the
`|| *sep != ':'` checks are redundant. Drop them.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Split the removal of redundant *sep != ':' checks out of "libfdt:
overlay: Prepare fixup and fragment path helpers for reuse" into a
standalone patch
libfdt/fdt_overlay.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index c6dc4774cff3..8d96428af04f 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -432,7 +432,7 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
path = fixup_str;
sep = memchr(fixup_str, ':', fixup_len);
- if (!sep || *sep != ':')
+ if (!sep)
return -FDT_ERR_BADOVERLAY;
path_len = sep - path;
@@ -442,7 +442,7 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
fixup_len -= path_len + 1;
name = sep + 1;
sep = memchr(name, ':', fixup_len);
- if (!sep || *sep != ':')
+ if (!sep)
return -FDT_ERR_BADOVERLAY;
name_len = sep - name;
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr()
2026-11-01 22:01 ` [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr() Douglas Anderson
@ 2026-11-02 4:30 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:30 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 1729 bytes --]
On Sun, Nov 01, 2026 at 02:01:09PM -0800, Douglas Anderson wrote:
> In `overlay_fixup_phandle()`, `sep` is obtained via `memchr(..., ':',
> ...)`. When `sep` is non-NULL, `*sep` is guaranteed to be `':'`, so the
> `|| *sep != ':'` checks are redundant. Drop them.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes in v3:
> - Split the removal of redundant *sep != ':' checks out of "libfdt:
> overlay: Prepare fixup and fragment path helpers for reuse" into a
> standalone patch
>
> libfdt/fdt_overlay.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
> index c6dc4774cff3..8d96428af04f 100644
> --- a/libfdt/fdt_overlay.c
> +++ b/libfdt/fdt_overlay.c
> @@ -432,7 +432,7 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
>
> path = fixup_str;
> sep = memchr(fixup_str, ':', fixup_len);
> - if (!sep || *sep != ':')
> + if (!sep)
> return -FDT_ERR_BADOVERLAY;
>
> path_len = sep - path;
> @@ -442,7 +442,7 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
> fixup_len -= path_len + 1;
> name = sep + 1;
> sep = memchr(name, ':', fixup_len);
> - if (!sep || *sep != ':')
> + if (!sep)
> return -FDT_ERR_BADOVERLAY;
>
> name_len = sep - name;
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 01/15] libfdt: overlay: Fix out-of-bounds write on bad fixup offset Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 02/15] libfdt: overlay: Drop redundant *sep != ':' checks after memchr() Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:32 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist() Douglas Anderson
` (12 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Follow-up improvements to commit 6b0371afc532 ("pylibfdt: Add
subnode/property iterators and get_mem_rsv_list()"):
- Document in `FdtRo.subnodes()` and `FdtRo.properties()` that the
device tree must not be modified while iterating, as node or property
offsets may shift.
- Remove the inaccurate `Raises:` section from `FdtRo.subnodes()`, as
`fdt_first_subnode()` and `fdt_next_subnode()` map all negative
returns from `fdt_next_node()` to `-FDT_ERR_NOTFOUND` (which
`subnodes()` quiets).
- Add a test case in `tests/pylibfdt_tests.py` verifying that passing a
negative offset (`-1`) to `properties()` raises `FdtException` with
`-libfdt.BADOFFSET`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- New follow-up patch for already-applied commit 6b0371afc532
("pylibfdt: Add subnode/property iterators and get_mem_rsv_list()")
clarifying iterator docstrings and testing negative offsets (Simon
Glass, David Gibson)
pylibfdt/libfdt.i | 9 ++++++---
tests/pylibfdt_tests.py | 4 ++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 0db7110b4a38..5c8859b37df3 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -204,14 +204,14 @@ class FdtRo(object):
def subnodes(self, nodeoffset):
"""Iterate over the subnodes of a node
+ The device tree must not be modified while iterating, as node offsets
+ may shift.
+
Args:
nodeoffset: Offset of parent node to check
Yields:
Node offset of each direct subnode in order
-
- Raises:
- FdtException on error (e.g. invalid nodeoffset)
"""
soff = self.first_subnode(nodeoffset, quiet=QUIET_NOTFOUND)
while soff >= 0:
@@ -465,6 +465,9 @@ class FdtRo(object):
def properties(self, nodeoffset):
"""Iterate over the properties of a node
+ The device tree must not be modified while iterating, as property
+ offsets may shift.
+
Args:
nodeoffset: Offset of node to check
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 2a0509c7802e..93408745c575 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -171,6 +171,10 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.fdt.first_property_offset(node)
self.assertEqual(e.exception.err, -libfdt.NOTFOUND)
+ with self.assertRaises(FdtException) as e:
+ list(self.fdt.properties(-1))
+ self.assertEqual(e.exception.err, -libfdt.BADOFFSET)
+
def testGetName(self):
"""Check that we can get the name of a node"""
self.assertEqual(self.fdt.get_name(0), '')
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets
2026-11-01 22:01 ` [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets Douglas Anderson
@ 2026-11-02 4:32 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:32 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 3375 bytes --]
On Sun, Nov 01, 2026 at 02:01:10PM -0800, Douglas Anderson wrote:
> Follow-up improvements to commit 6b0371afc532 ("pylibfdt: Add
> subnode/property iterators and get_mem_rsv_list()"):
>
> - Document in `FdtRo.subnodes()` and `FdtRo.properties()` that the
> device tree must not be modified while iterating, as node or property
> offsets may shift.
> - Remove the inaccurate `Raises:` section from `FdtRo.subnodes()`, as
> `fdt_first_subnode()` and `fdt_next_subnode()` map all negative
> returns from `fdt_next_node()` to `-FDT_ERR_NOTFOUND` (which
> `subnodes()` quiets).
> - Add a test case in `tests/pylibfdt_tests.py` verifying that passing a
> negative offset (`-1`) to `properties()` raises `FdtException` with
> `-libfdt.BADOFFSET`.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes in v3:
> - New follow-up patch for already-applied commit 6b0371afc532
> ("pylibfdt: Add subnode/property iterators and get_mem_rsv_list()")
> clarifying iterator docstrings and testing negative offsets (Simon
> Glass, David Gibson)
>
> pylibfdt/libfdt.i | 9 ++++++---
> tests/pylibfdt_tests.py | 4 ++++
> 2 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> index 0db7110b4a38..5c8859b37df3 100644
> --- a/pylibfdt/libfdt.i
> +++ b/pylibfdt/libfdt.i
> @@ -204,14 +204,14 @@ class FdtRo(object):
> def subnodes(self, nodeoffset):
> """Iterate over the subnodes of a node
>
> + The device tree must not be modified while iterating, as node offsets
> + may shift.
> +
> Args:
> nodeoffset: Offset of parent node to check
>
> Yields:
> Node offset of each direct subnode in order
> -
> - Raises:
> - FdtException on error (e.g. invalid nodeoffset)
> """
> soff = self.first_subnode(nodeoffset, quiet=QUIET_NOTFOUND)
> while soff >= 0:
> @@ -465,6 +465,9 @@ class FdtRo(object):
> def properties(self, nodeoffset):
> """Iterate over the properties of a node
>
> + The device tree must not be modified while iterating, as property
> + offsets may shift.
> +
> Args:
> nodeoffset: Offset of node to check
>
> diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
> index 2a0509c7802e..93408745c575 100644
> --- a/tests/pylibfdt_tests.py
> +++ b/tests/pylibfdt_tests.py
> @@ -171,6 +171,10 @@ class PyLibfdtBasicTests(unittest.TestCase):
> self.fdt.first_property_offset(node)
> self.assertEqual(e.exception.err, -libfdt.NOTFOUND)
>
> + with self.assertRaises(FdtException) as e:
> + list(self.fdt.properties(-1))
> + self.assertEqual(e.exception.err, -libfdt.BADOFFSET)
> +
> def testGetName(self):
> """Check that we can get the name of a node"""
> self.assertEqual(self.fdt.get_name(0), '')
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist()
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (2 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 03/15] pylibfdt: Clarify iterator docstrings and test negative offsets Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:33 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys() Douglas Anderson
` (11 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Currently, calling `Property.as_str()` or `Property.as_stringlist()` on
an empty `Property` (`b''`) raises `IndexError: bytearray index out of
range` when indexing `self[-1]`.
Update `Property.as_str()` to raise `ValueError('Property lacks nul
termination')` on an empty property, and update
`Property.as_stringlist()` to return `[]` on an empty property. Add unit
tests in `tests/pylibfdt_tests.py`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Split Property.as_str() and Property.as_stringlist() empty-property
handling into a standalone patch and add direct Property unit tests
(Simon Glass, David Gibson)
pylibfdt/libfdt.i | 4 +++-
tests/pylibfdt_tests.py | 7 +++++++
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 5c8859b37df3..eb14312986ea 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -932,7 +932,7 @@ class Property(bytearray):
def as_str(self):
"""Unicode is supported by decoding from UTF-8"""
- if self[-1] != 0:
+ if not self or self[-1] != 0:
raise ValueError('Property lacks nul termination')
if 0 in self[:-1]:
raise ValueError('Property contains embedded nul characters')
@@ -940,6 +940,8 @@ class Property(bytearray):
def as_stringlist(self):
"""Unicode is supported by decoding from UTF-8"""
+ if not self:
+ return []
if self[-1] != 0:
raise ValueError('Property lacks nul termination')
parts = self[:-1].split(b'\x00')
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 93408745c575..20741d8f2df0 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -423,6 +423,10 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.fdt.getprop(node, "compatible").as_stringlist())
self.assertEqual(["this is a placeholder string", "string2"],
self.fdt.getprop(node, "placeholder").as_stringlist())
+ self.assertEqual([], libfdt.Property('empty', b'').as_stringlist())
+ with self.assertRaises(ValueError) as e:
+ libfdt.Property('bad', b'no-nul').as_stringlist()
+ self.assertIn('lacks nul termination', str(e.exception))
def testReserveMap(self):
"""Test that we can access the memory reserve map"""
@@ -523,6 +527,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
with self.assertRaises(ValueError) as e:
self.fdt.getprop(node, 'prop-int').as_str()
self.assertIn('lacks nul termination', str(e.exception))
+ with self.assertRaises(ValueError) as e:
+ libfdt.Property('empty', b'').as_str()
+ self.assertIn('lacks nul termination', str(e.exception))
node2 = self.fdt.path_offset('/subnode@1/subsubnode')
with self.assertRaises(ValueError) as e:
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist()
2026-11-01 22:01 ` [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist() Douglas Anderson
@ 2026-11-02 4:33 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:33 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 3545 bytes --]
On Sun, Nov 01, 2026 at 02:01:11PM -0800, Douglas Anderson wrote:
> Currently, calling `Property.as_str()` or `Property.as_stringlist()` on
> an empty `Property` (`b''`) raises `IndexError: bytearray index out of
> range` when indexing `self[-1]`.
>
> Update `Property.as_str()` to raise `ValueError('Property lacks nul
> termination')` on an empty property, and update
> `Property.as_stringlist()` to return `[]` on an empty property. Add unit
> tests in `tests/pylibfdt_tests.py`.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes in v3:
> - Split Property.as_str() and Property.as_stringlist() empty-property
> handling into a standalone patch and add direct Property unit tests
> (Simon Glass, David Gibson)
>
> pylibfdt/libfdt.i | 4 +++-
> tests/pylibfdt_tests.py | 7 +++++++
> 2 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> index 5c8859b37df3..eb14312986ea 100644
> --- a/pylibfdt/libfdt.i
> +++ b/pylibfdt/libfdt.i
> @@ -932,7 +932,7 @@ class Property(bytearray):
>
> def as_str(self):
> """Unicode is supported by decoding from UTF-8"""
> - if self[-1] != 0:
> + if not self or self[-1] != 0:
> raise ValueError('Property lacks nul termination')
> if 0 in self[:-1]:
> raise ValueError('Property contains embedded nul characters')
> @@ -940,6 +940,8 @@ class Property(bytearray):
>
> def as_stringlist(self):
> """Unicode is supported by decoding from UTF-8"""
> + if not self:
> + return []
> if self[-1] != 0:
> raise ValueError('Property lacks nul termination')
> parts = self[:-1].split(b'\x00')
> diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
> index 93408745c575..20741d8f2df0 100644
> --- a/tests/pylibfdt_tests.py
> +++ b/tests/pylibfdt_tests.py
> @@ -423,6 +423,10 @@ class PyLibfdtBasicTests(unittest.TestCase):
> self.fdt.getprop(node, "compatible").as_stringlist())
> self.assertEqual(["this is a placeholder string", "string2"],
> self.fdt.getprop(node, "placeholder").as_stringlist())
> + self.assertEqual([], libfdt.Property('empty', b'').as_stringlist())
> + with self.assertRaises(ValueError) as e:
> + libfdt.Property('bad', b'no-nul').as_stringlist()
> + self.assertIn('lacks nul termination', str(e.exception))
>
> def testReserveMap(self):
> """Test that we can access the memory reserve map"""
> @@ -523,6 +527,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
> with self.assertRaises(ValueError) as e:
> self.fdt.getprop(node, 'prop-int').as_str()
> self.assertIn('lacks nul termination', str(e.exception))
> + with self.assertRaises(ValueError) as e:
> + libfdt.Property('empty', b'').as_str()
> + self.assertIn('lacks nul termination', str(e.exception))
>
> node2 = self.fdt.path_offset('/subnode@1/subsubnode')
> with self.assertRaises(ValueError) as e:
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys()
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (3 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 04/15] pylibfdt: Handle empty properties in as_str() and as_stringlist() Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:34 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property Douglas Anderson
` (10 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
`FdtRo` exposes `boot_cpuid_phys()` to read the physical boot CPU ID
from the FDT header, and SWIG already wraps the underlying C function
`fdt_set_boot_cpuid_phys()`, but `class Fdt` lacked a corresponding
`set_boot_cpuid_phys()` method.
Add `Fdt.set_boot_cpuid_phys()` and test it in
`tests/pylibfdt_tests.py`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- New patch adding Fdt.set_boot_cpuid_phys() to pylibfdt (Simon Glass)
pylibfdt/libfdt.i | 8 ++++++++
tests/pylibfdt_tests.py | 3 +++
2 files changed, 11 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index eb14312986ea..51db9535bd39 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -684,6 +684,14 @@ class Fdt(FdtRo):
del self._fdt[self.totalsize():]
return err
+ def set_boot_cpuid_phys(self, boot_cpuid_phys):
+ """Set the physical boot CPU ID
+
+ Args:
+ boot_cpuid_phys: Physical boot CPU ID
+ """
+ fdt_set_boot_cpuid_phys(self._fdt, boot_cpuid_phys)
+
def add_mem_rsv(self, addr, size, quiet=()):
"""Add a memory reserve-map record
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 20741d8f2df0..18730965bd02 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -300,6 +300,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
self.assertEqual(self.fdt.version(), 17)
self.assertEqual(self.fdt.last_comp_version(), 16)
self.assertEqual(self.fdt.boot_cpuid_phys(), 0)
+ self.fdt.set_boot_cpuid_phys(0x42)
+ self.assertEqual(self.fdt.boot_cpuid_phys(), 0x42)
+ self.fdt.set_boot_cpuid_phys(0)
self.assertEqual(self.fdt.size_dt_strings(), 105)
self.assertEqual(self.fdt.size_dt_struct(), 564)
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys()
2026-11-01 22:01 ` [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys() Douglas Anderson
@ 2026-11-02 4:34 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:34 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]
On Sun, Nov 01, 2026 at 02:01:12PM -0800, Douglas Anderson wrote:
> `FdtRo` exposes `boot_cpuid_phys()` to read the physical boot CPU ID
> from the FDT header, and SWIG already wraps the underlying C function
> `fdt_set_boot_cpuid_phys()`, but `class Fdt` lacked a corresponding
> `set_boot_cpuid_phys()` method.
>
> Add `Fdt.set_boot_cpuid_phys()` and test it in
> `tests/pylibfdt_tests.py`.
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes in v3:
> - New patch adding Fdt.set_boot_cpuid_phys() to pylibfdt (Simon Glass)
>
> pylibfdt/libfdt.i | 8 ++++++++
> tests/pylibfdt_tests.py | 3 +++
> 2 files changed, 11 insertions(+)
>
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> index eb14312986ea..51db9535bd39 100644
> --- a/pylibfdt/libfdt.i
> +++ b/pylibfdt/libfdt.i
> @@ -684,6 +684,14 @@ class Fdt(FdtRo):
> del self._fdt[self.totalsize():]
> return err
>
> + def set_boot_cpuid_phys(self, boot_cpuid_phys):
> + """Set the physical boot CPU ID
> +
> + Args:
> + boot_cpuid_phys: Physical boot CPU ID
> + """
> + fdt_set_boot_cpuid_phys(self._fdt, boot_cpuid_phys)
> +
> def add_mem_rsv(self, addr, size, quiet=()):
> """Add a memory reserve-map record
>
> diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
> index 20741d8f2df0..18730965bd02 100644
> --- a/tests/pylibfdt_tests.py
> +++ b/tests/pylibfdt_tests.py
> @@ -300,6 +300,9 @@ class PyLibfdtBasicTests(unittest.TestCase):
> self.assertEqual(self.fdt.version(), 17)
> self.assertEqual(self.fdt.last_comp_version(), 16)
> self.assertEqual(self.fdt.boot_cpuid_phys(), 0)
> + self.fdt.set_boot_cpuid_phys(0x42)
> + self.assertEqual(self.fdt.boot_cpuid_phys(), 0x42)
> + self.fdt.set_boot_cpuid_phys(0)
> self.assertEqual(self.fdt.size_dt_strings(), 105)
> self.assertEqual(self.fdt.size_dt_struct(), 564)
>
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (4 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 05/15] pylibfdt: Add Fdt.set_boot_cpuid_phys() Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:35 ` David Gibson
2026-11-01 22:01 ` [PATCH v3 07/15] devicetree: Add DeviceTree live tree representation Douglas Anderson
` (9 subsequent siblings)
15 siblings, 1 reply; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
`libfdt.Property` subclasses `bytearray` and has a two-argument
`__init__(self, name, value)` constructor. Because `bytearray`'s default
copy/reduce protocol passes only the raw byte buffer to the constructor
(and does not copy instance attributes), calling `copy.copy()` or
`copy.deepcopy()` on a `Property` fails with `TypeError:
Property.__init__() missing 1 required positional argument: 'value'`.
Implement `__copy__()` and `__deepcopy__()` on `Property` so shallow and
deep copies preserve both `name` and the byte contents, and add a unit
test in `tests/pylibfdt_tests.py`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- New patch in v3 to support copying Property instances in DeviceTree
(Simon Glass)
pylibfdt/libfdt.i | 6 ++++++
tests/pylibfdt_tests.py | 6 ++++++
2 files changed, 12 insertions(+)
diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
index 51db9535bd39..fa5f499ed96d 100644
--- a/pylibfdt/libfdt.i
+++ b/pylibfdt/libfdt.i
@@ -908,6 +908,12 @@ class Property(bytearray):
bytearray.__init__(self, value)
self.name = name
+ def __copy__(self):
+ return type(self)(self.name, self)
+
+ def __deepcopy__(self, memo):
+ return type(self)(self.name, self)
+
def as_cell(self, fmt):
return struct.unpack('>' + fmt, self)[0]
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index 18730965bd02..ae46b29a8f5c 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -4,6 +4,7 @@
# Written by Simon Glass <sjg@chromium.org>
#
+import copy
import struct
import sys
import types
@@ -192,6 +193,11 @@ class PyLibfdtBasicTests(unittest.TestCase):
prop = self.fdt.get_property_by_offset(poffset)
self.assertEqual(prop.name, 'compatible')
self.assertEqual(prop, b'test_tree1\0')
+ for copied in (copy.copy(prop), copy.deepcopy(prop)):
+ self.assertIsInstance(copied, libfdt.Property)
+ self.assertIsNot(copied, prop)
+ self.assertEqual(copied.name, 'compatible')
+ self.assertEqual(copied, b'test_tree1\0')
with self.assertRaises(FdtException) as e:
self.fdt.get_property_by_offset(-2)
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property
2026-11-01 22:01 ` [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property Douglas Anderson
@ 2026-11-02 4:35 ` David Gibson
0 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:35 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 2856 bytes --]
On Sun, Nov 01, 2026 at 02:01:13PM -0800, Douglas Anderson wrote:
> `libfdt.Property` subclasses `bytearray` and has a two-argument
> `__init__(self, name, value)` constructor. Because `bytearray`'s default
> copy/reduce protocol passes only the raw byte buffer to the constructor
> (and does not copy instance attributes), calling `copy.copy()` or
> `copy.deepcopy()` on a `Property` fails with `TypeError:
> Property.__init__() missing 1 required positional argument: 'value'`.
>
> Implement `__copy__()` and `__deepcopy__()` on `Property` so shallow and
> deep copies preserve both `name` and the byte contents, and add a unit
> test in `tests/pylibfdt_tests.py`.
Oh, good catch.
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
>
> Assisted-by: Gemini:gemini-pro-prerelease
> Signed-off-by: Douglas Anderson <dianders@chromium.org>
> ---
>
> Changes in v3:
> - New patch in v3 to support copying Property instances in DeviceTree
> (Simon Glass)
>
> pylibfdt/libfdt.i | 6 ++++++
> tests/pylibfdt_tests.py | 6 ++++++
> 2 files changed, 12 insertions(+)
>
> diff --git a/pylibfdt/libfdt.i b/pylibfdt/libfdt.i
> index 51db9535bd39..fa5f499ed96d 100644
> --- a/pylibfdt/libfdt.i
> +++ b/pylibfdt/libfdt.i
> @@ -908,6 +908,12 @@ class Property(bytearray):
> bytearray.__init__(self, value)
> self.name = name
>
> + def __copy__(self):
> + return type(self)(self.name, self)
> +
> + def __deepcopy__(self, memo):
> + return type(self)(self.name, self)
> +
> def as_cell(self, fmt):
> return struct.unpack('>' + fmt, self)[0]
>
> diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
> index 18730965bd02..ae46b29a8f5c 100644
> --- a/tests/pylibfdt_tests.py
> +++ b/tests/pylibfdt_tests.py
> @@ -4,6 +4,7 @@
> # Written by Simon Glass <sjg@chromium.org>
> #
>
> +import copy
> import struct
> import sys
> import types
> @@ -192,6 +193,11 @@ class PyLibfdtBasicTests(unittest.TestCase):
> prop = self.fdt.get_property_by_offset(poffset)
> self.assertEqual(prop.name, 'compatible')
> self.assertEqual(prop, b'test_tree1\0')
> + for copied in (copy.copy(prop), copy.deepcopy(prop)):
> + self.assertIsInstance(copied, libfdt.Property)
> + self.assertIsNot(copied, prop)
> + self.assertEqual(copied.name, 'compatible')
> + self.assertEqual(copied, b'test_tree1\0')
>
> with self.assertRaises(FdtException) as e:
> self.fdt.get_property_by_offset(-2)
> --
> 2.56.0.317.g454f8302e1-goog
>
>
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v3 07/15] devicetree: Add DeviceTree live tree representation
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (5 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 06/15] pylibfdt: Support copy.copy() and copy.deepcopy() on Property Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 08/15] libfdt: overlay: Factor out fixup and fragment path parsing helpers Douglas Anderson
` (8 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Add `devicetree.py` providing `DeviceTree`, a non-flattened in-memory
representation of a devicetree node and its subtree:
- `DeviceTree.unflatten(fdt, offset=0)` builds a `DeviceTree` hierarchy
from an `FdtRo` instance, storing `libfdt.Property` instances in
`self.props` and raising `ValueError` if duplicate property or sibling
subnode names are encountered.
- `DeviceTree.flatten(mem_rsv=(), boot_cpuid_phys=0)` serializes a root
`DeviceTree` back into a packed `libfdt.Fdt` object via `FdtSw`.
- `get_node()`, `create_path()`, and `walk()` provide path lookup,
intermediate node creation, and pre-order traversal.
- `get_str()`, `set_str()`, `get_stringlist()`, `set_stringlist()`,
`get_u32()`, `set_u32()`, `get_u32_list()`, and `set_u32_list()`
provide typed property access and mutation.
Add unit tests in `tests/pylibfdt_tests.py` covering `DeviceTree`
unflatten/flatten round-tripping, duplicate name rejection, and helper
methods.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Move NodeTree out of pylibfdt/libfdt.i into a new devicetree.py module
and rename it to DeviceTree (David Gibson)
- Store libfdt.Property objects in self.props instead of raw bytes
(David Gibson)
- Return a packed libfdt.Fdt object from flatten() instead of raw bytes
and use fdt.set_boot_cpuid_phys() (David Gibson, Simon Glass)
- Raise ValueError in unflatten() on duplicate property or sibling
subnode names (Simon Glass)
Changes in v2:
- Split live tree representation out of fdtfactor (David Gibson)
devicetree.py | 276 ++++++++++++++++++++++++++++++++++++++++
meson.build | 1 +
tests/meson.build | 2 +-
tests/pylibfdt_tests.py | 77 +++++++++++
4 files changed, 355 insertions(+), 1 deletion(-)
create mode 100644 devicetree.py
diff --git a/devicetree.py b/devicetree.py
new file mode 100644
index 000000000000..198ab73eda88
--- /dev/null
+++ b/devicetree.py
@@ -0,0 +1,276 @@
+# SPDX-License-Identifier: (GPL-2.0-or-later OR BSD-2-Clause)
+"""devicetree - Non-flattened in-memory device tree representation in Python."""
+
+import struct
+
+import libfdt
+
+
+class DeviceTree:
+ """In-memory live tree representation of a devicetree node and its subtree.
+
+ Attributes:
+ props: Dict mapping property name (str) to `libfdt.Property`.
+ children: Dict mapping child node name (str) to child `DeviceTree`.
+ """
+
+ def __init__(self):
+ self.props = {}
+ self.children = {}
+
+ @classmethod
+ def unflatten(cls, fdt, offset=0):
+ """Unflatten an FdtRo subtree starting at `offset` into a DeviceTree.
+
+ Args:
+ fdt: An `FdtRo` (or `Fdt`) instance.
+ offset: Starting node offset (default 0 for the root node).
+
+ Returns:
+ A `DeviceTree` representing the node at `offset` and its
+ descendants.
+
+ Raises:
+ ValueError: If a node contains duplicate property names or
+ duplicate sibling subnode names.
+ """
+ node = cls()
+ for prop in fdt.properties(offset):
+ if prop.name in node.props:
+ raise ValueError(f'Duplicate property {prop.name!r}')
+ node.props[prop.name] = prop
+ for suboff in fdt.subnodes(offset):
+ subname = fdt.get_name(suboff)
+ if subname in node.children:
+ raise ValueError(f'Duplicate subnode {subname!r}')
+ node.children[subname] = cls.unflatten(fdt, suboff)
+ return node
+
+ def _flatten_subtree(self, sw, name):
+ """Recursively write this DeviceTree and its children into an FdtSw.
+
+ Args:
+ sw: An `FdtSw` sequential-write instance.
+ name: Node name for this subtree (empty string `''` for root).
+ """
+ with sw.add_node(name):
+ for prop_name, val in self.props.items():
+ sw.property(prop_name, bytes(val))
+ for child_name, child_node in self.children.items():
+ child_node._flatten_subtree(sw, child_name)
+
+ def flatten(self, mem_rsv=(), boot_cpuid_phys=0):
+ """Serialize this root DeviceTree into a packed `libfdt.Fdt` object.
+
+ Args:
+ mem_rsv: Iterable of (address, size) tuples for the FDT memory
+ reservation block.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+
+ Returns:
+ Packed `libfdt.Fdt` instance.
+ """
+ sw = libfdt.FdtSw()
+ for addr, size in mem_rsv:
+ sw.add_reservemap_entry(addr, size)
+ sw.finish_reservemap()
+ self._flatten_subtree(sw, '')
+ fdt = sw.as_fdt()
+ fdt.set_boot_cpuid_phys(boot_cpuid_phys)
+ fdt.pack()
+ return fdt
+
+ def get_node(self, path):
+ """Look up a descendant DeviceTree by path without modifying the tree.
+
+ Follows the slash-separated node names in `path` starting from `self`
+ (leading, trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically, and `''` or
+ `'/'` returns `self`). If any component along `path` is not present in
+ `children`, returns `None`.
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` at `path`, or `None` if not found.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ node = node.children.get(part)
+ if node is None:
+ return None
+ return node
+
+ def create_path(self, path):
+ """Return the descendant DeviceTree at `path`, creating missing nodes.
+
+ Looks up a child or deeper descendant `DeviceTree` by following the
+ slash-separated node names in `path` starting from `self` (leading,
+ trailing, and repeated `'/'` characters are ignored, so
+ `'/soc/i2c@4000'` and `'soc/i2c@4000'` behave identically). If any
+ node along `path` does not exist yet in `children`, a new empty
+ `DeviceTree` is created for it automatically (similar to `mkdir -p`).
+ Passing `''` or `'/'` returns `self`.
+
+ Example:
+ root = DeviceTree()
+ # Automatically creates root.children['soc'] and
+ # root.children['soc'].children['i2c@4000'], and returns the
+ # 'i2c@4000' DeviceTree.
+ i2c = root.create_path('/soc/i2c@4000')
+ i2c.set_str('status', 'okay')
+
+ Args:
+ path: Slash-separated node path starting from `self` (e.g.
+ `'/soc/i2c@4000'` or `'soc/i2c@4000'`).
+
+ Returns:
+ The `DeviceTree` instance at `path`.
+ """
+ node = self
+ for part in path.split('/'):
+ if not part:
+ continue
+ if part not in node.children:
+ node.children[part] = type(self)()
+ node = node.children[part]
+ return node
+
+ def walk(self, path=''):
+ """Yield `(path, node)` pairs for this subtree in pre-order.
+
+ Args:
+ path: Path prefix for `self` (defaults to `''` for the root node,
+ producing paths like `''`, `'/soc'`, `'/soc/i2c@4000'`).
+
+ Yields:
+ Tuple of `(path, node)` for `self` followed recursively by each
+ descendant `DeviceTree`.
+ """
+ prefix = path.rstrip('/')
+ yield path, self
+ for name, child in self.children.items():
+ yield from child.walk(f'{prefix}/{name}')
+
+ def get_str(self, name):
+ """Return property `name` decoded as a NUL-terminated UTF-8 string.
+
+ Args:
+ name: Property name to look up in `self.props`.
+
+ Returns:
+ Decoded string value (without the trailing NUL byte).
+
+ Raises:
+ KeyError: If `name` is not in `self.props`.
+ ValueError: If the property value is not a single NUL-terminated
+ UTF-8 string.
+ """
+ return self.props[name].as_str()
+
+ def set_str(self, name, val):
+ """Set property `name` to a NUL-terminated UTF-8 string.
+
+ Args:
+ name: Property name to set in `self.props`.
+ val: String value to encode as UTF-8 with a trailing NUL byte.
+
+ Raises:
+ ValueError: If `val` contains embedded NUL characters.
+ """
+ if '\x00' in val:
+ raise ValueError('String value contains embedded nul characters')
+ self.props[name] = libfdt.Property(name, val.encode('utf-8') + b'\x00')
+
+ def get_stringlist(self, name):
+ """Return property `name` decoded as a list of NUL-separated strings.
+
+ Args:
+ name: Property name to look up in `self.props`.
+
+ Returns:
+ List of decoded UTF-8 strings (`[]` if the property is empty).
+
+ Raises:
+ KeyError: If `name` is not in `self.props`.
+ ValueError: If a non-empty property lacks NUL termination.
+ """
+ return self.props[name].as_stringlist()
+
+ def set_stringlist(self, name, vals):
+ """Set property `name` to a NUL-separated list of UTF-8 strings.
+
+ Args:
+ name: Property name to set in `self.props`.
+ vals: Iterable of strings to encode as NUL-terminated UTF-8.
+
+ Raises:
+ ValueError: If any string in `vals` contains embedded NUL
+ characters.
+ """
+ vals = list(vals)
+ if any('\x00' in v for v in vals):
+ raise ValueError('String value contains embedded nul characters')
+ raw = b''.join(v.encode('utf-8') + b'\x00' for v in vals)
+ self.props[name] = libfdt.Property(name, raw)
+
+ def get_u32(self, name):
+ """Return property `name` decoded as a single big-endian uint32 cell.
+
+ Args:
+ name: Property name to look up in `self.props`.
+
+ Returns:
+ Integer value of the 32-bit cell.
+
+ Raises:
+ KeyError: If `name` is not in `self.props`.
+ ValueError: If the property length is not 4 bytes.
+ """
+ prop = self.props[name]
+ if len(prop) != 4:
+ raise ValueError(f'Property {name!r} length {len(prop)} != 4')
+ return prop.as_uint32()
+
+ def set_u32(self, name, val):
+ """Set property `name` to a single big-endian uint32 cell.
+
+ Args:
+ name: Property name to set in `self.props`.
+ val: Unsigned 32-bit integer value.
+ """
+ self.props[name] = libfdt.Property(name, struct.pack('>I', val))
+
+ def get_u32_list(self, name):
+ """Return property `name` decoded as a list of big-endian uint32 cells.
+
+ Args:
+ name: Property name to look up in `self.props`.
+
+ Returns:
+ List of 32-bit unsigned integers.
+
+ Raises:
+ KeyError: If `name` is not in `self.props`.
+ ValueError: If the property length is not a multiple of 4 bytes.
+ """
+ prop = self.props[name]
+ if len(prop) % 4 != 0:
+ raise ValueError(
+ f'Property {name!r} length {len(prop)} is not a multiple of 4'
+ )
+ return prop.as_uint32_list()
+
+ def set_u32_list(self, name, vals):
+ """Set property `name` to a sequence of big-endian uint32 cells.
+
+ Args:
+ name: Property name to set in `self.props`.
+ vals: Sequence of unsigned 32-bit integer values.
+ """
+ raw = struct.pack(f'>{len(vals)}I', *vals)
+ self.props[name] = libfdt.Property(name, raw)
diff --git a/meson.build b/meson.build
index 520cd35f9075..963d8f59b209 100644
--- a/meson.build
+++ b/meson.build
@@ -125,6 +125,7 @@ endforeach
if pylibfdt_enabled
subdir('pylibfdt')
+ py.install_sources('devicetree.py')
endif
if get_option('tests')
diff --git a/tests/meson.build b/tests/meson.build
index c25500606059..8dbfc3f9e8e2 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -119,7 +119,7 @@ if not pylibfdt_enabled
else
env += [
'PYTHON=' + py.full_path(),
- 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt',
+ 'PYTHONPATH=' + meson.project_build_root() / 'pylibfdt' + ':' + meson.project_source_root(),
'NO_PYTHON=0',
]
endif
diff --git a/tests/pylibfdt_tests.py b/tests/pylibfdt_tests.py
index ae46b29a8f5c..cfdb8f5749b6 100644
--- a/tests/pylibfdt_tests.py
+++ b/tests/pylibfdt_tests.py
@@ -11,6 +11,8 @@ import types
import unittest
sys.path.insert(0, '../pylibfdt')
+sys.path.insert(0, '..')
+import devicetree
import libfdt
from libfdt import Fdt, FdtSw, FdtException, QUIET_NOTFOUND, QUIET_ALL
@@ -717,6 +719,81 @@ class PyLibfdtRoTests(unittest.TestCase):
self.assertEqual(b'this is a placeholder string\0string2\0',
self.fdt.getprop(node, 'placeholder'))
+ def testDeviceTreeRoundTrip(self):
+ """Test unflattening and flattening a tree via DeviceTree"""
+ tree = copy.deepcopy(devicetree.DeviceTree.unflatten(self.fdt))
+ self.assertIsInstance(tree.props['compatible'], libfdt.Property)
+ rsvmap = self.fdt.get_mem_rsv_list()
+ cpuid = self.fdt.boot_cpuid_phys()
+ fdt2 = tree.flatten(rsvmap, cpuid)
+ self.assertIsInstance(fdt2, libfdt.Fdt)
+ self.assertEqual(rsvmap, fdt2.get_mem_rsv_list())
+ self.assertEqual(cpuid, fdt2.boot_cpuid_phys())
+
+ tree2 = devicetree.DeviceTree.unflatten(fdt2)
+ nodes1 = [(path, n.props, list(n.children.keys()))
+ for path, n in tree.walk()]
+ nodes2 = [(path, n.props, list(n.children.keys()))
+ for path, n in tree2.walk()]
+ self.assertEqual(nodes1, nodes2)
+
+ def testDeviceTreeDuplicates(self):
+ """Test that DeviceTree.unflatten() rejects duplicate names"""
+ sw_prop = FdtSw()
+ sw_prop.finish_reservemap()
+ with sw_prop.add_node(''):
+ sw_prop.property_u32('dup', 1)
+ sw_prop.property_u32('dup', 2)
+ with self.assertRaises(ValueError):
+ devicetree.DeviceTree.unflatten(sw_prop.as_fdt())
+
+ sw_node = FdtSw()
+ sw_node.finish_reservemap()
+ with sw_node.add_node(''):
+ with sw_node.add_node('dup'):
+ pass
+ with sw_node.add_node('dup'):
+ pass
+ with self.assertRaises(ValueError):
+ devicetree.DeviceTree.unflatten(sw_node.as_fdt())
+
+ def testDeviceTreeHelpers(self):
+ """Test DeviceTree navigation and typed property helpers"""
+ root = devicetree.DeviceTree()
+ self.assertIs(root, root.get_node(''))
+ self.assertIs(root, root.get_node('/'))
+ self.assertIsNone(root.get_node('/soc/i2c@4000'))
+
+ i2c = root.create_path('/soc/i2c@4000')
+ self.assertIs(i2c, root.get_node('/soc/i2c@4000'))
+ self.assertIs(i2c, root.get_node('soc/i2c@4000'))
+ self.assertEqual(['', '/soc', '/soc/i2c@4000'],
+ [path for path, _ in root.walk()])
+ self.assertEqual(['/', '/soc', '/soc/i2c@4000'],
+ [path for path, _ in root.walk('/')])
+
+ i2c.set_str('status', 'okay')
+ self.assertIsInstance(i2c.props['status'], libfdt.Property)
+ self.assertEqual('okay', i2c.get_str('status'))
+ with self.assertRaises(ValueError):
+ i2c.set_str('status', 'ok\x00ay')
+
+ i2c.set_stringlist('compatible', ['vendor,i2c', 'generic,i2c'])
+ self.assertEqual(['vendor,i2c', 'generic,i2c'],
+ i2c.get_stringlist('compatible'))
+ with self.assertRaises(ValueError):
+ i2c.set_stringlist('compatible', ['vendor\x00,i2c'])
+ i2c.set_stringlist('empty-list', [])
+ self.assertEqual([], i2c.get_stringlist('empty-list'))
+ with self.assertRaises(ValueError):
+ i2c.get_str('empty-list')
+
+ i2c.set_u32('#address-cells', 1)
+ self.assertEqual(1, i2c.get_u32('#address-cells'))
+
+ i2c.set_u32_list('reg', [0x4000, 0x100])
+ self.assertEqual([0x4000, 0x100], i2c.get_u32_list('reg'))
+
if __name__ == "__main__":
unittest.main()
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 08/15] libfdt: overlay: Factor out fixup and fragment path parsing helpers
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (6 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 07/15] devicetree: Add DeviceTree live tree representation Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 09/15] libfdt: overlay: Prepare fragment path helper for reuse Douglas Anderson
` (7 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
In preparation for updating the base device tree's `/__local_fixups__`
node during `fdt_overlay_apply()`, factor out two helper functions in
`libfdt/fdt_overlay.c`:
- Factor out `overlay_parse_fixup_entry()` from
`overlay_fixup_phandle()` to parse `"path:name:offset"` entries in
`/__fixups__` properties.
- Factor out `overlay_split_fragment_path()` from
`overlay_symbol_update()` to validate and split
`"/<fragment>/__overlay__[/<rel>]"` paths into the fragment name and
relative subnode path.
This commit is pure code movement with no functional change.
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Document in overlay_parse_fixup_entry()'s comment that @valuep and
@lenp are advanced past the consumed entry (David Gibson)
- Return 1 (under __overlay__), 0 (not under __overlay__), or
-FDT_ERR_BADVALUE from overlay_split_fragment_path() instead of taking
a bool *is_overlayp out-parameter (David Gibson)
Changes in v2:
- New patch factoring out overlay_parse_fixup_entry() and
overlay_split_fragment_path() helpers in libfdt/fdt_overlay.c
libfdt/fdt_overlay.c | 230 ++++++++++++++++++++++++++++---------------
1 file changed, 153 insertions(+), 77 deletions(-)
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 8d96428af04f..978c9c23bd56 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -361,6 +361,81 @@ static int overlay_fixup_one_phandle(void *fdto, int symbols_off,
return ret;
}
+/**
+ * overlay_parse_fixup_entry - Parse one "path:name:offset" entry from /__fixups__
+ * @valuep: In/out pointer to the current NUL-terminated fixup string;
+ * advanced past the terminating NUL byte to the next entry
+ * @lenp: In/out remaining length of the /__fixups__ property value;
+ * decremented by the consumed entry length (including its NUL byte)
+ * @pathp: Output pointer to the target node path
+ * @path_lenp: Output length of @pathp in bytes
+ * @namep: Output pointer to the target property name
+ * @name_lenp: Output length of @namep in bytes
+ * @poffsetp: Output byte offset within the target property
+ *
+ * returns:
+ * 0 on success
+ * -FDT_ERR_BADOVERLAY if the fixup entry is malformed
+ */
+static int overlay_parse_fixup_entry(const char **valuep, int *lenp,
+ const char **pathp, uint32_t *path_lenp,
+ const char **namep, uint32_t *name_lenp,
+ uint32_t *poffsetp)
+{
+ const char *path, *name, *fixup_end;
+ const char *fixup_str = *valuep;
+ uint32_t path_len, name_len;
+ uint32_t fixup_len;
+ const char *sep;
+ char *endptr;
+ unsigned long poffset;
+
+ fixup_end = memchr(*valuep, '\0', *lenp);
+ if (!fixup_end)
+ return -FDT_ERR_BADOVERLAY;
+ fixup_len = fixup_end - fixup_str;
+
+ *lenp -= fixup_len + 1;
+ *valuep += fixup_len + 1;
+
+ path = fixup_str;
+ sep = memchr(fixup_str, ':', fixup_len);
+ if (!sep)
+ return -FDT_ERR_BADOVERLAY;
+
+ path_len = sep - path;
+ if (path_len == (fixup_len - 1))
+ return -FDT_ERR_BADOVERLAY;
+
+ fixup_len -= path_len + 1;
+ name = sep + 1;
+ sep = memchr(name, ':', fixup_len);
+ if (!sep)
+ return -FDT_ERR_BADOVERLAY;
+
+ name_len = sep - name;
+ if (!name_len)
+ return -FDT_ERR_BADOVERLAY;
+
+ /*
+ * Require an initial ASCII digit because strtoul() accepts leading
+ * whitespace and '+'/'-' signs (wrapping negative values to unsigned
+ * long).
+ */
+ if ((*(sep + 1) < '0') || (*(sep + 1) > '9'))
+ return -FDT_ERR_BADOVERLAY;
+ poffset = strtoul(sep + 1, &endptr, 10);
+ if ((*endptr != '\0') || (poffset > UINT32_MAX))
+ return -FDT_ERR_BADOVERLAY;
+
+ *pathp = path;
+ *path_lenp = path_len;
+ *namep = name;
+ *name_lenp = name_len;
+ *poffsetp = poffset;
+ return 0;
+}
+
/**
* overlay_fixup_phandle - Set an overlay phandle to the base one
* @fdt: Base Device Tree blob
@@ -413,52 +488,14 @@ static int overlay_fixup_phandle(void *fdt, void *fdto, int symbols_off,
return -FDT_ERR_NOTFOUND;
do {
- const char *path, *name, *fixup_end;
- const char *fixup_str = value;
- uint32_t path_len, name_len;
- uint32_t fixup_len;
- const char *sep;
- char *endptr;
- unsigned long poffset;
+ const char *path, *name;
+ uint32_t path_len, name_len, poffset;
int ret;
- fixup_end = memchr(value, '\0', len);
- if (!fixup_end)
- return -FDT_ERR_BADOVERLAY;
- fixup_len = fixup_end - fixup_str;
-
- len -= fixup_len + 1;
- value += fixup_len + 1;
-
- path = fixup_str;
- sep = memchr(fixup_str, ':', fixup_len);
- if (!sep)
- return -FDT_ERR_BADOVERLAY;
-
- path_len = sep - path;
- if (path_len == (fixup_len - 1))
- return -FDT_ERR_BADOVERLAY;
-
- fixup_len -= path_len + 1;
- name = sep + 1;
- sep = memchr(name, ':', fixup_len);
- if (!sep)
- return -FDT_ERR_BADOVERLAY;
-
- name_len = sep - name;
- if (!name_len)
- return -FDT_ERR_BADOVERLAY;
-
- /*
- * Require an initial ASCII digit because strtoul() accepts
- * leading whitespace and '+'/'-' signs (wrapping negative
- * values to unsigned long).
- */
- if ((*(sep + 1) < '0') || (*(sep + 1) > '9'))
- return -FDT_ERR_BADOVERLAY;
- poffset = strtoul(sep + 1, &endptr, 10);
- if ((*endptr != '\0') || (poffset > UINT32_MAX))
- return -FDT_ERR_BADOVERLAY;
+ ret = overlay_parse_fixup_entry(&value, &len, &path, &path_len,
+ &name, &name_len, &poffset);
+ if (ret)
+ return ret;
ret = overlay_fixup_one_phandle(fdto, symbols_off,
path, path_len, name, name_len,
@@ -905,6 +942,72 @@ static int get_path_len(const void *fdt, int nodeoffset)
return len;
}
+/**
+ * overlay_split_fragment_path - Split "/<frag>/__overlay__[/<rel>]" into parts
+ * @path: NUL-terminated input path string
+ * @path_len: Length of @path in bytes (including terminating NUL)
+ * @frag_namep: Output pointer to the fragment name
+ * @frag_name_lenp: Output length of @frag_namep in bytes
+ * @rel_pathp: Output pointer to the relative subnode path after __overlay__
+ * @rel_path_lenp: Output length of @rel_pathp in bytes
+ *
+ * returns:
+ * 1 if @path is under an __overlay__ subnode
+ * 0 if @path is a valid path not under an __overlay__ subnode
+ * -FDT_ERR_BADVALUE if @path is malformed
+ */
+static int overlay_split_fragment_path(const char *path, int path_len,
+ const char **frag_namep,
+ int *frag_name_lenp,
+ const char **rel_pathp,
+ int *rel_path_lenp)
+{
+ int len, frag_name_len, rel_path_len;
+ const char *s, *e;
+ const char *frag_name;
+ const char *rel_path;
+
+ /* keep end marker to avoid strlen() */
+ e = path + path_len;
+
+ if (*path != '/')
+ return -FDT_ERR_BADVALUE;
+
+ /* get fragment name first */
+ s = strchr(path + 1, '/');
+ if (!s) {
+ /* Symbol refers to something that won't end
+ * up in the target tree */
+ return 0;
+ }
+
+ frag_name = path + 1;
+ frag_name_len = s - path - 1;
+
+ /* verify format; safe since "s" lies in \0 terminated prop */
+ len = sizeof("/__overlay__/") - 1;
+ if ((e - s) > len && (memcmp(s, "/__overlay__/", len) == 0)) {
+ /* /<fragment-name>/__overlay__/<relative-subnode-path> */
+ rel_path = s + len;
+ rel_path_len = e - rel_path - 1;
+ } else if ((e - s) == len
+ && (memcmp(s, "/__overlay__", len - 1) == 0)) {
+ /* /<fragment-name>/__overlay__ */
+ rel_path = "";
+ rel_path_len = 0;
+ } else {
+ /* Symbol refers to something that won't end
+ * up in the target tree */
+ return 0;
+ }
+
+ *frag_namep = frag_name;
+ *frag_name_lenp = frag_name_len;
+ *rel_pathp = rel_path;
+ *rel_path_lenp = rel_path_len;
+ return 1;
+}
+
/**
* overlay_symbol_update - Update the symbols of base tree after a merge
* @fdt: Base Device Tree blob
@@ -925,7 +1028,6 @@ static int overlay_symbol_update(void *fdt, void *fdto)
{
int root_sym, ov_sym, prop, path_len, fragment, target;
int len, frag_name_len, ret, rel_path_len;
- const char *s, *e;
const char *path;
const char *name;
const char *frag_name;
@@ -960,39 +1062,13 @@ static int overlay_symbol_update(void *fdt, void *fdto)
if (path_len < 1 || memchr(path, '\0', path_len) != &path[path_len - 1])
return -FDT_ERR_BADVALUE;
- /* keep end marker to avoid strlen() */
- e = path + path_len;
-
- if (*path != '/')
- return -FDT_ERR_BADVALUE;
-
- /* get fragment name first */
- s = strchr(path + 1, '/');
- if (!s) {
- /* Symbol refers to something that won't end
- * up in the target tree */
- continue;
- }
-
- frag_name = path + 1;
- frag_name_len = s - path - 1;
-
- /* verify format; safe since "s" lies in \0 terminated prop */
- len = sizeof("/__overlay__/") - 1;
- if ((e - s) > len && (memcmp(s, "/__overlay__/", len) == 0)) {
- /* /<fragment-name>/__overlay__/<relative-subnode-path> */
- rel_path = s + len;
- rel_path_len = e - rel_path - 1;
- } else if ((e - s) == len
- && (memcmp(s, "/__overlay__", len - 1) == 0)) {
- /* /<fragment-name>/__overlay__ */
- rel_path = "";
- rel_path_len = 0;
- } else {
- /* Symbol refers to something that won't end
- * up in the target tree */
+ ret = overlay_split_fragment_path(path, path_len,
+ &frag_name, &frag_name_len,
+ &rel_path, &rel_path_len);
+ if (ret < 0)
+ return ret;
+ if (!ret)
continue;
- }
/* find the fragment index in which the symbol lies */
ret = fdt_subnode_offset_namelen(fdto, 0, frag_name,
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 09/15] libfdt: overlay: Prepare fragment path helper for reuse
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (7 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 08/15] libfdt: overlay: Factor out fixup and fragment path parsing helpers Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 10/15] libfdt: overlay: Make fdto const in overlay_apply_node() Douglas Anderson
` (6 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Generalize `overlay_split_fragment_path()` in `libfdt/fdt_overlay.c` to
accept a length-bounded `path` slice that is not necessarily
NUL-terminated (using `memchr()` instead of `strchr()`, moving the
NUL-terminated property check back into `overlay_symbol_update()`, and
rejecting a trailing `'/'` after `"/__overlay__/"`) so it can also split
`:`-delimited paths from `/__fixups__` when updating
`/__local_fixups__`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Move the fixup offset overflow fix into its own standalone patch at
the start of the series (Simon Glass)
- Move the removal of the redundant *sep != ':' checks into its own
standalone patch
- Reject paths ending with a trailing slash ("/<frag>/__overlay__/") in
overlay_split_fragment_path() (David Gibson)
Changes in v2:
- New patch generalizing overlay_split_fragment_path() for
non-NUL-terminated paths in libfdt/fdt_overlay.c
libfdt/fdt_overlay.c | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 978c9c23bd56..36f2377cf7d8 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -944,8 +944,8 @@ static int get_path_len(const void *fdt, int nodeoffset)
/**
* overlay_split_fragment_path - Split "/<frag>/__overlay__[/<rel>]" into parts
- * @path: NUL-terminated input path string
- * @path_len: Length of @path in bytes (including terminating NUL)
+ * @path: Input path string (need not be NUL-terminated)
+ * @path_len: Length of @path in bytes (excluding any terminator)
* @frag_namep: Output pointer to the fragment name
* @frag_name_lenp: Output length of @frag_namep in bytes
* @rel_pathp: Output pointer to the relative subnode path after __overlay__
@@ -970,11 +970,11 @@ static int overlay_split_fragment_path(const char *path, int path_len,
/* keep end marker to avoid strlen() */
e = path + path_len;
- if (*path != '/')
+ if (path_len < 1 || *path != '/')
return -FDT_ERR_BADVALUE;
/* get fragment name first */
- s = strchr(path + 1, '/');
+ s = memchr(path + 1, '/', path_len - 1);
if (!s) {
/* Symbol refers to something that won't end
* up in the target tree */
@@ -984,13 +984,16 @@ static int overlay_split_fragment_path(const char *path, int path_len,
frag_name = path + 1;
frag_name_len = s - path - 1;
- /* verify format; safe since "s" lies in \0 terminated prop */
+ /* verify format */
len = sizeof("/__overlay__/") - 1;
if ((e - s) > len && (memcmp(s, "/__overlay__/", len) == 0)) {
/* /<fragment-name>/__overlay__/<relative-subnode-path> */
rel_path = s + len;
- rel_path_len = e - rel_path - 1;
- } else if ((e - s) == len
+ rel_path_len = e - rel_path;
+ } else if ((e - s) == len && (memcmp(s, "/__overlay__/", len) == 0)) {
+ /* Trailing slash after /__overlay__/ is invalid */
+ return -FDT_ERR_BADVALUE;
+ } else if ((e - s) == (len - 1)
&& (memcmp(s, "/__overlay__", len - 1) == 0)) {
/* /<fragment-name>/__overlay__ */
rel_path = "";
@@ -1062,7 +1065,7 @@ static int overlay_symbol_update(void *fdt, void *fdto)
if (path_len < 1 || memchr(path, '\0', path_len) != &path[path_len - 1])
return -FDT_ERR_BADVALUE;
- ret = overlay_split_fragment_path(path, path_len,
+ ret = overlay_split_fragment_path(path, path_len - 1,
&frag_name, &frag_name_len,
&rel_path, &rel_path_len);
if (ret < 0)
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 10/15] libfdt: overlay: Make fdto const in overlay_apply_node()
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (8 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 09/15] libfdt: overlay: Prepare fragment path helper for reuse Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 11/15] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
` (5 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
`overlay_apply_node()` only reads from the overlay blob `@fdto` while
merging properties and subnodes into `@fdt`. Mark `@fdto` as
`const void *` so callers holding a `const void *fdto` pointer (such as
the upcoming `/__local_fixups__` update logic) can call
`overlay_apply_node()` without casting away `const`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Split constifying fdto in overlay_apply_node() out of "libfdt:
overlay: Update base /__local_fixups__ on apply" into a standalone
patch (David Gibson)
libfdt/fdt_overlay.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 36f2377cf7d8..4b3e17bd087e 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -821,7 +821,7 @@ static int overlay_prevent_phandle_overwrite(void *fdt, void *fdto)
* Negative error code on failure
*/
static int overlay_apply_node(void *fdt, int target,
- void *fdto, int node)
+ const void *fdto, int node)
{
int property;
int subnode;
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 11/15] libfdt: overlay: Update base /__local_fixups__ on apply
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (9 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 10/15] libfdt: overlay: Make fdto const in overlay_apply_node() Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
` (4 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
When a base device tree is compiled with `dtc -L` (`--local-fixups`), it
contains a root `/__local_fixups__` node recording the byte offsets of
all phandle references in its properties (for example, so an OS kernel
can identify phandle cells at runtime, so a composite `.dtb` built by
`fdtoverlay` such as Linux's `foo-dtbs := base.dtb bar.dtbo` retains
complete phandle metadata for tools like `fdtfactor`, or so a merged
tree matches what `dtc -L` produces).
`fdt_overlay_apply()` updates the base tree's `/__symbols__` via
`overlay_symbol_update()`, but leaves the base tree's
`/__local_fixups__` untouched. As a result, applying an overlay to a
base tree with `/__local_fixups__` leaves `/__local_fixups__` missing
the phandle references introduced by the overlay (both from the
overlay's `/__local_fixups__` and from its resolved `/__fixups__`), and
leaves stale offsets if the overlay overwrites a base property that
previously had phandle references.
Add `overlay_local_fixups_update()`, controlled by a 3-state
compile-time `FDT_OVERLAY_LOCAL_FIXUPS` setting in
`libfdt/libfdt_internal.h`:
- `0` (`FDT_OVERLAY_LOCAL_FIXUPS_IGNORE`): Leave `/__local_fixups__`
untouched. This is the `#ifndef` fallback in `libfdt_internal.h` so
that size-sensitive embedded environments (such as U-Boot SPL) that
vendor `libfdt` directly get zero code-size increase and no behavior
change unless they opt in.
- `1` (`FDT_OVERLAY_LOCAL_FIXUPS_STRIP`): Delete `/__local_fixups__`
from the base tree if present (~50 bytes of code), avoiding leaving a
stale/incomplete `/__local_fixups__` node in the merged tree.
- `2` (`FDT_OVERLAY_LOCAL_FIXUPS_UPDATE`): Fully update the base tree's
`/__local_fixups__` node when present (clearing entries for
overwritten properties, merging offsets from the overlay's
`/__local_fixups__` and `/__fixups__` in sorted ascending order, and
pruning any subnodes that become empty). This is enabled by default in
`dtc`'s Meson (`-Doverlay-local-fixups=2`) and Make
(`OVERLAY_LOCAL_FIXUPS=2`) builds for desktop tools and tests.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Move constifying fdto in overlay_apply_node() into a standalone
preparatory patch (David Gibson)
- Gate `/__local_fixups__` handling behind a 3-state compile-time
`FDT_OVERLAY_LOCAL_FIXUPS` flag (`0=ignore`, `1=strip`, `2=update`),
defaulting to `0` (`IGNORE`) in `libfdt_internal.h` for embedded
builds and `2` (`UPDATE`) in Meson/Make desktop builds (Simon Glass,
David Gibson)
- Rewrite commit message in the present tense with motivation up front,
and update `fdt_overlay_apply()` documentation in `libfdt/libfdt.h`
(Simon Glass)
- Require a strict relative path in `overlay_create_path()` and return
`-FDT_ERR_BADOVERLAY` on empty path components or trailing `'/'`
(David Gibson)
- Eliminate the 256-byte `namebuf` limit in
`overlay_fragment_local_fixups()` by re-reading the node name from the
shifted ancestor offset after `fdt_add_subnode_namelen()` (David
Gibson, Simon Glass)
- Return `-FDT_ERR_BADVALUE` instead of `-FDT_ERR_BADSTRUCTURE` in
`overlay_add_local_fixup_offset()` when an existing
`/__local_fixups__` property length is not a multiple of 4 (David
Gibson, Simon Glass)
- Drop redundant `overlay_validate_local_fixups()` pre-pass and inline
single-use helpers (Simon Glass)
Changes in v2:
- New patch updating base /__local_fixups__ in libfdt
fdt_overlay_apply() instead of stripping /__local_fixups__ by default
(David Gibson)
Makefile | 6 +-
libfdt/fdt_overlay.c | 527 ++++++++++++++++++++++++++++++++++++++-
libfdt/libfdt.h | 7 +
libfdt/libfdt_internal.h | 21 ++
meson.build | 5 +
meson_options.txt | 2 +
tests/overlay.c | 80 ++++++
tests/run_tests.sh | 19 ++
8 files changed, 663 insertions(+), 4 deletions(-)
diff --git a/Makefile b/Makefile
index 9110d17796b0..1560a5609fea 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,11 @@ CONFIG_LOCALVERSION =
# See libfdt_internal.h for details
ASSUME_MASK ?= 0
-CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK)
+# Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)
+# See libfdt_internal.h for details
+OVERLAY_LOCAL_FIXUPS ?= 2
+
+CPPFLAGS = -I libfdt -I . -DFDT_ASSUME_MASK=$(ASSUME_MASK) -DFDT_OVERLAY_LOCAL_FIXUPS=$(OVERLAY_LOCAL_FIXUPS)
WARNINGS = -Wall -Wpointer-arith -Wcast-qual -Wnested-externs -Wsign-compare \
-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow \
-Wwrite-strings
diff --git a/libfdt/fdt_overlay.c b/libfdt/fdt_overlay.c
index 4b3e17bd087e..c22ac588fcb8 100644
--- a/libfdt/fdt_overlay.c
+++ b/libfdt/fdt_overlay.c
@@ -1019,9 +1019,8 @@ static int overlay_split_fragment_path(const char *path, int path_len,
* overlay_symbol_update() updates the symbols of the base tree with the
* symbols of the applied overlay
*
- * This is the last step in the device tree overlay application
- * process, allowing the reference of overlay symbols by subsequent
- * overlay operations.
+ * This step in the device tree overlay application process allows
+ * the reference of overlay symbols by subsequent overlay operations.
*
* returns:
* 0 on success
@@ -1135,6 +1134,524 @@ static int overlay_symbol_update(void *fdt, void *fdto)
return 0;
}
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+/**
+ * overlay_create_path - Create a relative path under a parent node
+ * @fdt: Base Device Tree blob
+ * @parent: Node offset of the starting parent node
+ * @path: Relative path to traverse (must not start or end with '/' or contain "//")
+ * @path_len: Length of @path in bytes
+ *
+ * overlay_create_path() walks the slash-separated path components in @path
+ * starting at @parent, creating any missing intermediate subnodes along the
+ * way.
+ *
+ * returns:
+ * Node offset of the resolved or created subnode on success
+ * Negative error code on failure
+ */
+static int overlay_create_path(void *fdt, int parent, const char *path,
+ int path_len)
+{
+ int node = parent;
+
+ while (path_len > 0) {
+ const char *sep = memchr(path, '/', path_len);
+ int comp_len = sep ? (sep - path) : path_len;
+ int next;
+
+ if (!comp_len)
+ return -FDT_ERR_BADOVERLAY;
+
+ next = fdt_subnode_offset_namelen(fdt, node, path, comp_len);
+ if (next == -FDT_ERR_NOTFOUND)
+ next = fdt_add_subnode_namelen(fdt, node, path,
+ comp_len);
+ if (next < 0)
+ return next;
+ node = next;
+
+ if (!sep)
+ break;
+ path += comp_len + 1;
+ path_len -= comp_len + 1;
+ if (!path_len)
+ return -FDT_ERR_BADOVERLAY;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_fragment_local_fixups - Find or create a fragment's target under /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @create: Whether to create missing subnodes under /__local_fixups__
+ *
+ * overlay_fragment_local_fixups() resolves the target of @fragment in @fdt and
+ * locates (or creates, when @create is true) the corresponding node at the
+ * same path under @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * Node offset of the corresponding /__local_fixups__ node on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_local_fixups(void *fdt, const void *fdto,
+ int fragment, bool create)
+{
+ int root_lf, target, cur_fdt, node;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ target = fdt_overlay_target_offset(fdt, fdto, fragment, NULL);
+ if (target < 0)
+ return target;
+
+ cur_fdt = 0;
+ node = root_lf;
+ while (cur_fdt != target) {
+ int subnode, ancestor = -FDT_ERR_INTERNAL;
+ const char *name;
+ int namelen, next_lf;
+
+ fdt_for_each_subnode(subnode, fdt, cur_fdt) {
+ if (subnode <= target)
+ ancestor = subnode;
+ else
+ break;
+ }
+ if (ancestor < 0)
+ return ancestor;
+
+ name = fdt_get_name(fdt, ancestor, &namelen);
+ if (!name)
+ return namelen;
+
+ next_lf = fdt_subnode_offset_namelen(fdt, node, name, namelen);
+ if (next_lf == -FDT_ERR_NOTFOUND && create) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+
+ next_lf = fdt_add_subnode_namelen(fdt, node, name,
+ namelen);
+ if (next_lf < 0)
+ return next_lf;
+ if (next_lf <= ancestor) {
+ int delta = fdt_size_dt_struct(fdt) -
+ old_struct_size;
+ char *new_name;
+
+ target += delta;
+ ancestor += delta;
+ name = fdt_get_name(fdt, ancestor, NULL);
+ new_name = (char *)(uintptr_t)fdt_get_name(
+ fdt, next_lf, NULL);
+ memcpy(new_name, name, namelen + 1);
+ }
+ } else if (next_lf < 0) {
+ return next_lf;
+ }
+
+ cur_fdt = ancestor;
+ node = next_lf;
+ }
+
+ return node;
+}
+
+/**
+ * overlay_clear_overwritten_local_fixups - Remove local fixups for overwritten properties
+ * @fdt: Base Device Tree blob
+ * @target_lf: Node offset in @fdt's /__local_fixups__ subtree
+ * @fdto: Device tree overlay blob
+ * @overlay: Node offset in @fdto's __overlay__ subtree
+ *
+ * overlay_clear_overwritten_local_fixups() recursively deletes properties under
+ * @target_lf that correspond to properties overwritten by @overlay in @fdto.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_clear_overwritten_local_fixups(void *fdt, int target_lf,
+ const void *fdto, int overlay)
+{
+ int prop, subnode;
+
+ fdt_for_each_property_offset(prop, fdto, overlay) {
+ const char *name;
+ const void *val;
+ int len, ret;
+
+ val = fdt_getprop_by_offset(fdto, prop, &name, &len);
+ if (!val)
+ return len;
+
+ ret = fdt_delprop(fdt, target_lf, name);
+ if (ret && ret != -FDT_ERR_NOTFOUND)
+ return ret;
+ }
+
+ fdt_for_each_subnode(subnode, fdto, overlay) {
+ const char *name;
+ int namelen, sub_lf, ret;
+
+ name = fdt_get_name(fdto, subnode, &namelen);
+ if (!name)
+ return namelen;
+
+ sub_lf = fdt_subnode_offset_namelen(fdt, target_lf, name,
+ namelen);
+ if (sub_lf == -FDT_ERR_NOTFOUND)
+ continue;
+ if (sub_lf < 0)
+ return sub_lf;
+
+ ret = overlay_clear_overwritten_local_fixups(fdt, sub_lf,
+ fdto, subnode);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_add_local_fixup_offset - Insert a sorted u32 offset into a local fixup property
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ * @name: Property name to update
+ * @name_len: Length of @name in bytes
+ * @poffset: Byte offset of the phandle reference within the target property
+ *
+ * overlay_add_local_fixup_offset() inserts @poffset in ascending sorted order
+ * (ignoring duplicates) into the u32 array property @name on @lf_node.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_add_local_fixup_offset(void *fdt, int lf_node,
+ const char *name, int name_len,
+ uint32_t poffset)
+{
+ const fdt32_t *old_val;
+ fdt32_t *vals;
+ void *prop_data;
+ int old_len, count, i, idx, ret;
+
+ old_val = fdt_getprop_namelen(fdt, lf_node, name, name_len, &old_len);
+ if (!old_val && old_len != -FDT_ERR_NOTFOUND)
+ return old_len;
+
+ if (!old_val) {
+ old_len = 0;
+ count = 0;
+ idx = 0;
+ } else {
+ if (old_len % sizeof(uint32_t))
+ return -FDT_ERR_BADVALUE;
+ count = old_len / sizeof(uint32_t);
+ idx = count;
+ for (i = 0; i < count; i++) {
+ uint32_t cur = fdt32_ld(old_val + i);
+
+ if (cur == poffset)
+ return 0;
+ if (cur > poffset) {
+ idx = i;
+ break;
+ }
+ }
+ }
+
+ ret = fdt_setprop_placeholder_namelen(fdt, lf_node, name, name_len,
+ old_len + sizeof(uint32_t),
+ &prop_data);
+ if (ret)
+ return ret;
+
+ vals = prop_data;
+ if (idx < count)
+ memmove(vals + idx + 1, vals + idx,
+ (count - idx) * sizeof(uint32_t));
+ fdt32_st(vals + idx, poffset);
+
+ return 0;
+}
+
+/**
+ * overlay_merge_fixups_into_local_fixups - Record resolved /__fixups__ for a fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the current fragment in @fdto
+ * @target_frag_name: Name of @fragment
+ * @target_frag_len: Length of @target_frag_name in bytes
+ * @target_lfp: In/out cached node offset of @fragment under @fdt's /__local_fixups__
+ *
+ * overlay_merge_fixups_into_local_fixups() parses external phandle references
+ * belonging to @fragment in @fdto's /__fixups__ node and records their resolved
+ * property byte offsets in @fdt's /__local_fixups__ subtree.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_merge_fixups_into_local_fixups(void *fdt, const void *fdto,
+ int fragment,
+ const char *target_frag_name,
+ int target_frag_len,
+ int *target_lfp)
+{
+ int fixups_off, prop;
+
+ fixups_off = fdt_subnode_offset(fdto, 0, "__fixups__");
+ if (fixups_off == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (fixups_off < 0)
+ return fixups_off;
+
+ fdt_for_each_property_offset(prop, fdto, fixups_off) {
+ const char *value, *label;
+ int len;
+
+ value = fdt_getprop_by_offset(fdto, prop, &label, &len);
+ if (!value)
+ return len;
+
+ do {
+ const char *path, *name, *frag_name, *rel_path;
+ uint32_t path_len, name_len, poffset;
+ int frag_name_len, rel_path_len, lf_node, ret;
+
+ ret = overlay_parse_fixup_entry(&value, &len, &path,
+ &path_len, &name,
+ &name_len, &poffset);
+ if (ret)
+ return ret;
+
+ ret = overlay_split_fragment_path(path, path_len,
+ &frag_name,
+ &frag_name_len,
+ &rel_path,
+ &rel_path_len);
+ if (ret < 0)
+ return ret;
+ if (!ret)
+ continue;
+
+ if (frag_name_len != target_frag_len ||
+ memcmp(frag_name, target_frag_name,
+ frag_name_len) != 0)
+ continue;
+
+ if (*target_lfp < 0) {
+ *target_lfp = overlay_fragment_local_fixups(
+ fdt, fdto, fragment, true);
+ if (*target_lfp < 0)
+ return *target_lfp;
+ }
+
+ lf_node = overlay_create_path(fdt, *target_lfp,
+ rel_path, rel_path_len);
+ if (lf_node < 0)
+ return lf_node;
+
+ ret = overlay_add_local_fixup_offset(fdt, lf_node,
+ name, name_len,
+ poffset);
+ if (ret)
+ return ret;
+ } while (len > 0);
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_prune_empty_local_fixups - Remove empty subnodes from /__local_fixups__
+ * @fdt: Base Device Tree blob
+ * @lf_node: Node offset in @fdt's /__local_fixups__ subtree
+ *
+ * overlay_prune_empty_local_fixups() recursively deletes any child subnodes
+ * under @lf_node that have neither properties nor subnodes remaining.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_prune_empty_local_fixups(void *fdt, int lf_node)
+{
+ int subnode = fdt_first_subnode(fdt, lf_node);
+
+ while (subnode >= 0) {
+ int next, ret;
+
+ ret = overlay_prune_empty_local_fixups(fdt, subnode);
+ if (ret < 0)
+ return ret;
+
+ next = fdt_next_subnode(fdt, subnode);
+ if (fdt_first_property_offset(fdt, subnode) == -FDT_ERR_NOTFOUND &&
+ fdt_first_subnode(fdt, subnode) == -FDT_ERR_NOTFOUND) {
+ int old_struct_size = fdt_size_dt_struct(fdt);
+
+ ret = fdt_del_node(fdt, subnode);
+ if (ret < 0)
+ return ret;
+ if (next >= 0)
+ next += fdt_size_dt_struct(fdt) -
+ old_struct_size;
+ }
+ subnode = next;
+ }
+
+ return 0;
+}
+
+/**
+ * overlay_fragment_update_local_fixups - Update base /__local_fixups__ for one fragment
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ * @fragment: Node offset of the fragment in @fdto
+ * @ov_lf_root: Node offset of /__local_fixups__ in @fdto (or -FDT_ERR_NOTFOUND)
+ *
+ * overlay_fragment_update_local_fixups() clears any base /__local_fixups__
+ * entries overwritten by @fragment, then merges phandle offsets for @fragment
+ * from @fdto's /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_fragment_update_local_fixups(void *fdt, const void *fdto,
+ int fragment, int ov_lf_root)
+{
+ const char *frag_name;
+ int frag_name_len, overlay, target_lf, lf_frag, lf_overlay, ret;
+
+ overlay = fdt_subnode_offset(fdto, fragment, "__overlay__");
+ if (overlay == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (overlay < 0)
+ return overlay;
+
+ frag_name = fdt_get_name(fdto, fragment, &frag_name_len);
+ if (!frag_name)
+ return frag_name_len;
+
+ target_lf = overlay_fragment_local_fixups(fdt, fdto, fragment, false);
+ if (target_lf < 0 && target_lf != -FDT_ERR_NOTFOUND)
+ return target_lf;
+
+ if (target_lf >= 0) {
+ ret = overlay_clear_overwritten_local_fixups(fdt, target_lf,
+ fdto, overlay);
+ if (ret)
+ return ret;
+ }
+
+ lf_overlay = -FDT_ERR_NOTFOUND;
+ if (ov_lf_root >= 0) {
+ lf_frag = fdt_subnode_offset_namelen(fdto, ov_lf_root,
+ frag_name, frag_name_len);
+ if (lf_frag >= 0)
+ lf_overlay = fdt_subnode_offset(fdto, lf_frag,
+ "__overlay__");
+ else if (lf_frag != -FDT_ERR_NOTFOUND)
+ return lf_frag;
+ }
+ if (lf_overlay < 0 && lf_overlay != -FDT_ERR_NOTFOUND)
+ return lf_overlay;
+
+ if (lf_overlay >= 0) {
+ if (target_lf < 0)
+ target_lf = overlay_fragment_local_fixups(fdt, fdto,
+ fragment,
+ true);
+ if (target_lf < 0)
+ return target_lf;
+
+ ret = overlay_apply_node(fdt, target_lf, fdto, lf_overlay);
+ if (ret)
+ return ret;
+ }
+
+ return overlay_merge_fixups_into_local_fixups(
+ fdt, fdto, fragment, frag_name, frag_name_len, &target_lf);
+}
+
+/**
+ * overlay_local_fixups_update - Update base /__local_fixups__ with overlay fixups
+ * @fdt: Base Device Tree blob
+ * @fdto: Device tree overlay blob
+ *
+ * overlay_local_fixups_update() updates the base device tree's
+ * /__local_fixups__ subtree (when present) by walking each overlay fragment in
+ * application order, clearing entries for properties overwritten by that
+ * fragment, and merging phandle reference offsets from @fdto's
+ * /__local_fixups__ and /__fixups__ subtrees.
+ *
+ * This is part of the device tree overlay application process, when you want
+ * the merged device tree to retain complete local phandle fixup metadata.
+ *
+ * returns:
+ * 0 on success
+ * Negative error code on failure
+ */
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf, ov_lf_root, fragment, ret;
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ ov_lf_root = fdt_subnode_offset(fdto, 0, "__local_fixups__");
+ if (ov_lf_root < 0 && ov_lf_root != -FDT_ERR_NOTFOUND)
+ return ov_lf_root;
+
+ /*
+ * Process fragments in application order so that if a later fragment in
+ * fdto overwrites a property modified by an earlier fragment, the
+ * earlier fragment's local fixups for that property are cleared first.
+ */
+ fdt_for_each_subnode(fragment, fdto, 0) {
+ ret = overlay_fragment_update_local_fixups(fdt, fdto, fragment,
+ ov_lf_root);
+ if (ret)
+ return ret;
+ }
+
+ root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+ if (root_lf < 0)
+ return root_lf;
+
+ return overlay_prune_empty_local_fixups(fdt, root_lf);
+}
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ int root_lf = fdt_subnode_offset(fdt, 0, "__local_fixups__");
+
+ if (root_lf == -FDT_ERR_NOTFOUND)
+ return 0;
+ if (root_lf < 0)
+ return root_lf;
+
+ return fdt_del_node(fdt, root_lf);
+}
+#else
+static int overlay_local_fixups_update(void *fdt, const void *fdto)
+{
+ return 0;
+}
+#endif
+
int fdt_overlay_apply(void *fdt, void *fdto)
{
uint32_t delta;
@@ -1175,6 +1692,10 @@ int fdt_overlay_apply(void *fdt, void *fdto)
if (ret)
goto err;
+ ret = overlay_local_fixups_update(fdt, fdto);
+ if (ret)
+ goto err;
+
/*
* The overlay has been damaged, erase its magic.
*/
diff --git a/libfdt/libfdt.h b/libfdt/libfdt.h
index c69a18edf09f..c2f47a7d3e87 100644
--- a/libfdt/libfdt.h
+++ b/libfdt/libfdt.h
@@ -2472,6 +2472,13 @@ int fdt_del_node(void *fdt, int nodeoffset);
* fdt_overlay_apply() will apply the given device tree overlay on the
* given base device tree.
*
+ * If the base device tree contains a /__symbols__ node, it is updated with
+ * any symbols from the overlay. If the base device tree contains a
+ * /__local_fixups__ node, how it is handled depends on the build-time
+ * FDT_OVERLAY_LOCAL_FIXUPS setting (see libfdt_internal.h): left untouched
+ * (IGNORE), deleted (STRIP), or updated to reflect phandle references in the
+ * merged tree (UPDATE, the default when built via dtc's build system).
+ *
* Expect the base device tree to be modified, even if the function
* returns an error.
*
diff --git a/libfdt/libfdt_internal.h b/libfdt/libfdt_internal.h
index eefea2a3ccf3..4e9cf2e1dc5d 100644
--- a/libfdt/libfdt_internal.h
+++ b/libfdt/libfdt_internal.h
@@ -99,6 +99,27 @@ static inline uint64_t fdt64_ld_(const fdt64_t *p)
#define FDT_PARENT_MAX_DEPTH 32
#endif
+/*
+ * Controls how fdt_overlay_apply() handles a /__local_fixups__ node in the
+ * base device tree:
+ * 0 (FDT_OVERLAY_LOCAL_FIXUPS_IGNORE): Leave /__local_fixups__ untouched
+ * (zero extra code size and preserves historical behavior; default when
+ * building libfdt standalone without dtc's build system).
+ * 1 (FDT_OVERLAY_LOCAL_FIXUPS_STRIP): Delete /__local_fixups__ from the
+ * base device tree so a stale/incomplete /__local_fixups__ node is not
+ * left behind (~50 bytes of code).
+ * 2 (FDT_OVERLAY_LOCAL_FIXUPS_UPDATE): Fully update /__local_fixups__ in
+ * the base device tree with the overlay's local and resolved external
+ * phandle fixups (default when building via dtc's Meson/Make build).
+ */
+#define FDT_OVERLAY_LOCAL_FIXUPS_IGNORE 0
+#define FDT_OVERLAY_LOCAL_FIXUPS_STRIP 1
+#define FDT_OVERLAY_LOCAL_FIXUPS_UPDATE 2
+
+#ifndef FDT_OVERLAY_LOCAL_FIXUPS
+#define FDT_OVERLAY_LOCAL_FIXUPS FDT_OVERLAY_LOCAL_FIXUPS_IGNORE
+#endif
+
/*
* Defines assumptions which can be enabled. Each of these can be enabled
* individually. For maximum safety, don't enable any assumptions!
diff --git a/meson.build b/meson.build
index 963d8f59b209..15a421e1296a 100644
--- a/meson.build
+++ b/meson.build
@@ -28,6 +28,11 @@ add_project_arguments(
language: 'c'
)
+add_project_arguments(
+ '-DFDT_OVERLAY_LOCAL_FIXUPS=' + get_option('overlay-local-fixups').to_string(),
+ language: 'c'
+)
+
yamltree = 'yamltree.c'
yaml = dependency('yaml-0.1', version: '>=0.2.3', required: get_option('yaml'))
if not yaml.found()
diff --git a/meson_options.txt b/meson_options.txt
index a866b172e802..ac8ea4aa2a9f 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -2,6 +2,8 @@ option('tools', type: 'boolean', value: true,
description: 'Build tools')
option('assume-mask', type: 'integer', value: 0,
description: 'Control the assumptions made (e.g. risking security issues) in the code.')
+option('overlay-local-fixups', type: 'integer', value: 2,
+ description: 'Control how fdt_overlay_apply() handles base /__local_fixups__ (0=ignore, 1=strip, 2=update)')
option('yaml', type: 'feature', value: 'auto',
description: 'YAML support')
option('valgrind', type: 'feature', value: 'auto',
diff --git a/tests/overlay.c b/tests/overlay.c
index 2d27918a3369..a7987680e587 100644
--- a/tests/overlay.c
+++ b/tests/overlay.c
@@ -9,6 +9,7 @@
#include <stdio.h>
#include <libfdt.h>
+#include "libfdt_internal.h"
#include "tests.h"
@@ -191,6 +192,84 @@ static void *open_dt(char *path)
return copy;
}
+static void fdt_overlay_local_fixups(char *base_path, char *overlay_path)
+{
+ void *fdt_base = open_dt(base_path);
+ void *fdt_overlay = open_dt(overlay_path);
+ int lf, tn, stn;
+
+ /*
+ * Add a /__local_fixups__ subtree to the base tree before applying
+ * the overlay:
+ * - /__local_fixups__/test-node: test-int-property = <0> (overwritten
+ * by the overlay with a non-phandle value)
+ * - /__local_fixups__/test-node/sub-test-node:
+ * new-sub-test-property = <0> (overwritten by the overlay, leaving
+ * sub-test-node empty so it should be pruned in UPDATE mode)
+ */
+ lf = fdt_add_subnode(fdt_base, 0, "__local_fixups__");
+ CHECK(lf < 0);
+ tn = fdt_add_subnode(fdt_base, lf, "test-node");
+ CHECK(tn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, tn, "test-int-property", 0));
+ stn = fdt_add_subnode(fdt_base, tn, "sub-test-node");
+ CHECK(stn < 0);
+ CHECK(fdt_setprop_u32(fdt_base, stn, "new-sub-test-property", 0));
+
+ CHECK(fdt_overlay_apply(fdt_base, fdt_overlay));
+
+#if FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_UPDATE
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val) !=
+ -FDT_ERR_NOTFOUND);
+ CHECK(fdt_path_offset(
+ fdt_base,
+ "/__local_fixups__/test-node/sub-test-node") !=
+ -FDT_ERR_NOTFOUND);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 1, &val));
+ CHECK(val != 4);
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-several-phandle", 1, &val));
+ CHECK(val != 4);
+ }
+#elif FDT_OVERLAY_LOCAL_FIXUPS == FDT_OVERLAY_LOCAL_FIXUPS_STRIP
+ CHECK(fdt_path_offset(fdt_base, "/__local_fixups__") !=
+ -FDT_ERR_NOTFOUND);
+#else
+ {
+ unsigned long val = 0;
+
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-int-property", 0, &val));
+ CHECK(val != 0);
+ CHECK(fdt_getprop_u32_by_poffset(
+ fdt_base, "/__local_fixups__/test-node",
+ "test-phandle", 0, &val) != -FDT_ERR_NOTFOUND);
+ }
+#endif
+
+ free(fdt_base);
+ free(fdt_overlay);
+}
+
int main(int argc, char *argv[])
{
void *fdt_base, *fdt_overlay;
@@ -219,6 +298,7 @@ int main(int argc, char *argv[])
if (fdt_path_offset(fdt_base, "/__symbols__") >= 0) {
fdt_overlay_local_phandle(fdt_base);
fdt_overlay_local_phandles(fdt_base);
+ fdt_overlay_local_fixups(argv[1], argv[2]);
}
PASS();
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 0d815e307909..d51a111c7819 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1141,6 +1141,25 @@ fdtoverlay_tests() {
shorten_echo "check phandle to noda d wasn't overwritten: "
run_wrap_test test "$bd" = "$pd"
+
+ # verify that /__local_fixups__ in a -L base blob is updated when applying an overlay
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_phandle_lf.test.dtb "$SRCDIR/overlay_base_phandle.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_phandle_lf.test.dtb -o overlay_base_phandle_lfO.test.dtb overlay_overlay_phandle.test.dtb
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/b d
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/c b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d a
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d b
+ run_fdtget_test 0 overlay_base_phandle_lfO.test.dtb /__local_fixups__/d c
+
+ run_dtc_test -@ -L -I dts -O dtb -o overlay_base_lf.test.dtb "$SRCDIR/overlay_base.dts"
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb -cp /__local_fixups__/test-node/sub-test-node
+ run_wrap_test $DTPUT overlay_base_lf.test.dtb /__local_fixups__/test-node/sub-test-node new-sub-test-property -tu 0
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay.fdtoverlay.test.dtb "$SRCDIR/overlay_overlay.dts"
+ run_wrap_test $FDTOVERLAY -i overlay_base_lf.test.dtb -o overlay_base_lfO.test.dtb overlay_overlay.fdtoverlay.test.dtb
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-phandle
+ run_fdtget_test "0 4" overlay_base_lfO.test.dtb /__local_fixups__/test-node test-several-phandle
+ run_test check_path overlay_base_lfO.test.dtb not-exists "/__local_fixups__/test-node/sub-test-node"
}
pylibfdt_tests () {
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (10 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 11/15] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 13/15] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
` (3 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
When shipping a single firmware or kernel image that supports many
closely related board variants or SKUs, bundling a full compiled .dtb
for every variant results in substantial duplication.
Add `fdtfactor`, a Python tool built on `pylibfdt` and `devicetree` that
takes a set of compiled device tree blobs (.dtb) compiled with `-@ -L`
(`/__symbols__` and `/__local_fixups__`) and splits them into a single
common `base.dtb` and per-board `.dtbo` overlays:
- Common nodes present across all input DTBs are placed in `base.dtb`,
with each common property assigned the value shared by at least two
boards that maximizes byte savings across overlays.
- Using `/__local_fixups__`, phandle references inside property values
are normalized to target node paths so common properties referencing
common nodes match even when numeric phandles differ across input
DTBs.
- Any common node that carries a phandle in any input DTB receives its
phandle in `base.dtb` so overlays only emit `phandle` properties on
newly added subnodes, matching the requirements of `libfdt`, Linux's
`drivers/of/overlay.c`, and coreboot's `dt_apply_overlay()`.
Add a `fdtfactor` test suite in `tests/run_tests.sh` verifying that
applying each generated `.dtbo` to `base.dtb` with `fdtoverlay`
reconstructs the original DTB (verified via `dtbs_equal_unordered`,
including across input DTBs with shifted phandle numbers).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Update fdtfactor to use devicetree.DeviceTree and a FactoredTree
subclass initializing self.phandle and self.fixups in __init__()
(David Gibson, Simon Glass)
- Rename -b/--base-dtb to --output-base-dtb to avoid confusion with an
input base DTB (David Gibson)
- Move the test rejecting input DTBs lacking /__local_fixups__ from the
-S patch into this base patch (Simon Glass)
Changes in v2:
- Preserve /__local_fixups__ on base.dtb by default (David Gibson)
- Store parsed trees directly in NodeTree with node.phandle and
node.fixups attributes instead of parallel dicts (David Gibson)
- Extract phandles in a separate pass after unflattening and reject
legacy linux,phandle properties (David Gibson)
- Exclude non-root common phandle nodes lacking a common symbol from
base.dtb instead of synthesizing __fdtfactor_ph_* symbols (David
Gibson)
- Drop -q/--quiet, rename -v/--verbose to -s/--stats, and move tests to
a dedicated fdtfactor test suite (David Gibson)
Makefile | 2 +-
README.md | 43 ++
fdtfactor | 842 +++++++++++++++++++++++++++++
meson.build | 1 +
tests/Makefile.tests | 4 +-
tests/fdtfactor_board1.dts | 38 ++
tests/fdtfactor_board2.dts | 36 ++
tests/fdtfactor_board3.dts | 34 ++
tests/fdtfactor_board3_shifted.dts | 34 ++
tests/meson.build | 2 +-
tests/run_tests.sh | 72 ++-
tests/testutils.sh | 1 +
12 files changed, 1102 insertions(+), 7 deletions(-)
create mode 100755 fdtfactor
create mode 100644 tests/fdtfactor_board1.dts
create mode 100644 tests/fdtfactor_board2.dts
create mode 100644 tests/fdtfactor_board3.dts
create mode 100644 tests/fdtfactor_board3_shifted.dts
diff --git a/Makefile b/Makefile
index 1560a5609fea..fc06b0ae16e2 100644
--- a/Makefile
+++ b/Makefile
@@ -169,7 +169,7 @@ BIN += fdtget
BIN += fdtput
BIN += fdtoverlay
-SCRIPTS = dtdiff
+SCRIPTS = dtdiff fdtfactor
all: $(BIN) libfdt
diff --git a/README.md b/README.md
index fcfc2e55d804..61fefc4424fe 100644
--- a/README.md
+++ b/README.md
@@ -112,6 +112,49 @@ $ meson setup -Dpython=disabled build
More work remains to support all of libfdt, including access to numeric
values.
+## fdtfactor
+
+`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
+compiled device tree blobs (`.dtb`) and splits them into a single common
+`base.dtb` and per-board `.dtbo` overlays so that applying a board's
+`.dtbo` to `base.dtb` recreates the original board device tree while
+reducing total storage size.
+
+Input `.dtb` files must be compiled with `dtc -@ -L` so that both
+`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
+and `/__local_fixups__` (used to identify phandle references inside
+property values) are present.
+
+Basic usage:
+
+```
+$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+```
+
+Options:
+
+* `--output-base-dtb`: Output filename for the common base `.dtb`
+ (default: `base.dtb`; placed in `--outdir` if specified, or the current
+ directory otherwise).
+* `-o`, `--outdir`: Output directory for generated `.dtbo` files and
+ `--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
+ current directory for `--output-base-dtb`).
+* `-s`, `--stats`: Print size savings summary.
+
+### TODO
+
+* **Targeted overlay fragments**: Add an option to emit fine-grained
+ `/fragment@N` nodes targeting individual parent subnodes rather than
+ placing all overlay changes under a single `/fragment@0` targeting `/`.
+* **Multi-level overlays**: Support hierarchical overlay layers (for
+ example, SoC base -> board family intermediate overlay -> SKU leaf
+ overlay) when multiple boards share large sub-trees not present across
+ all boards. As an interim step, `fdtfactor` could accept an
+ already-generated `base.dtb` plus a subset of `.dtbo` overlays emitted
+ by an initial pass, leaving `base.dtb` untouched while factoring out
+ an intermediate `.dtbo` for the commonalities shared within that
+ subset.
+
## Mailing lists
* The [devicetree-compiler](mailto:devicetree-compiler@vger.kernel.org)
diff --git a/fdtfactor b/fdtfactor
new file mode 100755
index 000000000000..f6944e36a0f2
--- /dev/null
+++ b/fdtfactor
@@ -0,0 +1,842 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""fdtfactor - Split multiple compiled .dtb files into a base .dtb and .dtbo overlays.
+
+Given two or more input .dtb files compiled with `dtc -@ -L` (so they include
+`/__symbols__` and `/__local_fixups__`), `fdtfactor` extracts all nodes and
+properties shared across the inputs into a single common `base.dtb` and emits a
+small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
+`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
+original board DTB.
+"""
+
+import argparse
+from collections import Counter
+import os
+import struct
+import sys
+
+import devicetree
+import libfdt
+
+
+def _is_dtbo_overlay(fdt):
+ """Return True if `fdt` is a `.dtbo` overlay rather than a base DTB.
+
+ Args:
+ fdt: A `libfdt.FdtRo` instance to inspect.
+
+ Returns:
+ True if `fdt` contains `/__fixups__` or a top-level subnode with an
+ `__overlay__` child, False otherwise.
+ """
+ if fdt.subnode_offset(0, '__fixups__', quiet=libfdt.QUIET_NOTFOUND) >= 0:
+ return True
+ return any(
+ fdt.subnode_offset(soff, '__overlay__', quiet=libfdt.QUIET_NOTFOUND) >= 0
+ for soff in fdt.subnodes(0)
+ )
+
+
+class FactoredTree(devicetree.DeviceTree):
+ """DeviceTree subclass with normalized phandle and fixup attributes.
+
+ Attributes:
+ phandle: Optional integer phandle popped from `props['phandle']`.
+ fixups: Dict mapping property name (str) to a tuple of
+ `(byte_offset, target_node_path)` phandle fixup pairs.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.phandle = None
+ self.fixups = {}
+
+
+class ParsedDtb:
+ """Complete input .dtb blob with phandle-normalized tree and FDT metadata.
+
+ While `devicetree.DeviceTree` is a generic live-tree representation of a
+ node hierarchy (with `libfdt.Property` instances and unprocessed
+ `/__symbols__`, `/__local_fixups__`, and `phandle` properties), `ParsedDtb`
+ wraps an entire input `.dtb` in a `FactoredTree` (`self.root`) and
+ normalizes it for cross-board comparison:
+
+ - Stores FDT header, memory reservation, and file-level metadata that live
+ outside the root node tree (`boot_cpuid_phys`, `mem_rsv`, `filename`,
+ `raw_bytes`).
+ - Pops `/__symbols__` (parsed into `self.symbols`) and `/__local_fixups__`
+ out of `self.root`.
+ - Moves each node's `phandle` property out of `node.props` into
+ `node.phandle`, and uses `/__local_fixups__` to zero out numeric phandle
+ cells in `node.props` while recording canonical `(offset, target_path)`
+ tuples in `node.fixups` so properties can be compared across DTBs
+ independent of numeric phandle numbering.
+ """
+
+ def __init__(self, filename):
+ """Parse an input FDT file into a normalized FactoredTree.
+
+ Args:
+ filename: File path of the `.dtb` file to read.
+ """
+ self.filename = filename
+ with open(filename, 'rb') as f:
+ raw_bytes = f.read()
+ self.raw_bytes = raw_bytes
+
+ try:
+ fdt = libfdt.FdtRo(raw_bytes)
+ libfdt.check_err(libfdt.fdt_check_full(fdt._fdt, len(raw_bytes)))
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
+
+ if _is_dtbo_overlay(fdt):
+ raise ValueError(
+ f'{filename}: Input is already a DTB overlay (.dtbo)'
+ )
+
+ self.boot_cpuid_phys = fdt.boot_cpuid_phys()
+ self.mem_rsv = tuple(fdt.get_mem_rsv_list())
+ self.root = FactoredTree.unflatten(fdt)
+
+ sym_node = self.root.children.pop('__symbols__', None)
+ lf_root = self.root.children.pop('__local_fixups__', None)
+ if sym_node is None or lf_root is None:
+ raise ValueError(
+ f'input DTB(s) lack /__symbols__ or /__local_fixups__: '
+ f'{filename} (compile input DTBs with dtc -@ -L)'
+ )
+
+ phandle_to_path = self._extract_phandles()
+ self.symbols = self._parse_symbols(sym_node)
+ self._apply_local_fixups(lf_root, phandle_to_path)
+
+ def _extract_phandles(self):
+ """Move `phandle` properties out of `node.props`.
+
+ Walks `self.root` in pre-order and, for each node:
+ - Pops `phandle` from `node.props` (so phandle properties are not
+ compared as ordinary byte properties across DTBs, since `dtc` may
+ assign different integer phandles to the same node in different
+ boards) and stores the decoded integer in `node.phandle`.
+ - Validates that `node.phandle` is not `0` or `0xffffffff`, and that no
+ two nodes in the DTB share the same phandle.
+
+ Returns:
+ Dict mapping each integer phandle in this DTB to its node path
+ string (`""` for root, `"/soc/..."` for subnodes), used by
+ `_apply_local_fixups()` to resolve numeric phandle cells to
+ canonical target node paths.
+ """
+ phandle_to_path = {}
+ for path, node in self.root.walk():
+ if 'linux,phandle' in node.props:
+ raise ValueError(
+ f'{self.filename}: {path or "/"}: legacy linux,phandle '
+ f'property is not supported (compile with dtc -H epapr)'
+ )
+ if 'phandle' in node.props:
+ node.phandle = node.get_u32('phandle')
+ del node.props['phandle']
+
+ if node.phandle is not None:
+ if node.phandle in (0, 0xffffffff):
+ raise ValueError(
+ f'{self.filename}: invalid phandle {node.phandle:#x} '
+ f'on {path or "/"}'
+ )
+ if node.phandle in phandle_to_path:
+ raise ValueError(
+ f'{self.filename}: duplicate phandle {node.phandle:#x} '
+ f'on {phandle_to_path[node.phandle] or "/"} and '
+ f'{path or "/"}'
+ )
+ phandle_to_path[node.phandle] = path
+ return phandle_to_path
+
+ def _parse_symbols(self, sym_node):
+ """Parse the popped `/__symbols__` node into a `label -> path` dict.
+
+ Each property on `/__symbols__` (ignoring any `phandle` property on the
+ `/__symbols__` node itself) maps a Devicetree label name to the full
+ path of the target node in the DTB. Root paths (`"/"`) are normalized to
+ `""` to match `DeviceTree` path conventions, and each target path is
+ verified to exist in `self.root`.
+
+ Args:
+ sym_node: The `FactoredTree` for `/__symbols__` popped from
+ `self.root`.
+
+ Returns:
+ Dict mapping symbol label name (str) to normalized target node path
+ (str, `""` for root or `"/..."` for subnodes).
+ """
+ symbols = {}
+ for label in sym_node.props:
+ if label == 'phandle':
+ continue
+ raw_path = sym_node.get_str(label)
+ if not raw_path.startswith('/') or '//' in raw_path:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} has malformed '
+ f'path {raw_path!r}'
+ )
+ target_path = raw_path.rstrip('/')
+ if self.root.get_node(target_path) is None:
+ raise ValueError(
+ f'{self.filename}: symbol {label!r} references '
+ f'missing node {target_path or "/"!r}'
+ )
+ symbols[label] = target_path
+ return symbols
+
+ def _apply_local_fixups(self, lf_root, phandle_to_path):
+ """Normalize phandle references in `self.root` using `/__local_fixups__`.
+
+ The `/__local_fixups__` tree mirrors the structure of `self.root`: each
+ property in `/__local_fixups__` contains a list of 32-bit byte offsets
+ where the corresponding property in `self.root` holds a phandle cell.
+ For every such offset, this method:
+ - Reads the 32-bit integer phandle from `target_node.props[prop_name]`
+ and resolves it to a canonical target node path via `phandle_to_path`.
+ - Zeros out the 4-byte phandle cell in `target_node.props[prop_name]` so
+ property values can be compared across DTBs independent of numeric
+ phandle numbering.
+ - Records `((offset, target_path), ...)` in
+ `target_node.fixups[prop_name]`.
+
+ Args:
+ lf_root: The `FactoredTree` for `/__local_fixups__` popped from
+ `self.root`.
+ phandle_to_path: Dict mapping integer phandle values to normalized
+ node paths in `self.root`, as returned by `_extract_phandles()`.
+ """
+ for path, lf_node in lf_root.walk():
+ if not lf_node.props:
+ continue
+ target_node = self.root.get_node(path)
+ if target_node is None:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing node '
+ f'{path or "/"}'
+ )
+ for prop_name in lf_node.props:
+ if prop_name not in target_node.props:
+ raise ValueError(
+ f'{self.filename}: local fixup references missing '
+ f'property {path or "/"}:{prop_name}'
+ )
+ offsets = sorted(set(lf_node.get_u32_list(prop_name)))
+ raw_prop = target_node.props[prop_name]
+ masked = bytearray(raw_prop)
+ targets = []
+ for off in offsets:
+ if off + 4 > len(raw_prop):
+ raise ValueError(
+ f'{self.filename}: fixup offset {off} out of range '
+ f'on {path or "/"}:{prop_name} (len {len(raw_prop)})'
+ )
+ if targets and off < targets[-1][0] + 4:
+ raise ValueError(
+ f'{self.filename}: overlapping fixup offset {off} '
+ f'on {path or "/"}:{prop_name}'
+ )
+ ph = struct.unpack_from('>I', raw_prop, off)[0]
+ target_path = phandle_to_path.get(ph)
+ if target_path is None:
+ raise ValueError(
+ f'{self.filename}: unknown phandle {ph:#x} at '
+ f'{path or "/"}:{prop_name}+{off}'
+ )
+ masked[off:off + 4] = b'\x00\x00\x00\x00'
+ targets.append((off, target_path))
+ target_node.props[prop_name] = libfdt.Property(
+ prop_name, bytes(masked)
+ )
+ target_node.fixups[prop_name] = tuple(targets)
+
+
+def select_best_candidate(candidates, path, prop_name):
+ """Choose the `(masked_bytes, fixups)` value that maximizes overlay savings.
+
+ Only candidates shared by at least two input DTBs (`count > 1`) are
+ eligible for the base DTB; a value that appears in only one DTB would add
+ one copy to `base.dtb` to save one overlay copy (zero net savings).
+
+ Byte cost model per overlay occurrence:
+ - Each FDT property carries a 12-byte structure header (`FDT_PROP` 4-byte
+ tag + 4-byte length + 4-byte `nameoff`) plus `len(masked_bytes)` value
+ bytes.
+ - Each phandle fixup in an overlay adds a string entry of the form
+ `"/fragment@0/__overlay__<path>:<prop_name>:<off>\\0"` to `/__fixups__`:
+ 22 bytes (`"/fragment@0/__overlay__"`) + `len(path)` + 1 (`":"`) +
+ `len(prop_name)` + 1 (`":"`) + ~1-2 digits for `<off>` + 1 (`"\\0"`) +
+ ~3 bytes amortized `/__fixups__` property overhead =
+ `len(path) + len(prop_name) + 28` bytes.
+
+ Args:
+ candidates: Sequence of `(masked_bytes, fixups)` tuples observed across
+ input DTBs for `prop_name` on `path`.
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+
+ Returns:
+ The `(masked_bytes, fixups)` tuple that yields the largest estimated
+ byte savings when placed in `base.dtb`, or `None` if no candidate
+ appears in at least two input DTBs.
+ """
+ fixup_entry_cost = len(path) + len(prop_name) + 28
+ counts = Counter(candidates)
+ best_val = None
+ best_key = None
+ for val, count in counts.items():
+ if count <= 1:
+ continue
+ masked_bytes, fixups = val
+ length = len(masked_bytes) + len(fixups) * fixup_entry_cost
+ # Placing `val` in base.dtb adds 1 copy and removes `count` overlay
+ # copies, saving `(count - 1) * (length + 12)` bytes across the set.
+ key = ((count - 1) * (length + 12), count, length)
+ if best_key is None or key > best_key:
+ best_key = key
+ best_val = val
+ return best_val
+
+
+def _validate_dtb_headers(dtbs):
+ """Verify that FDT header fields and `/memreserve/` entries match across DTBs.
+
+ `fdt_overlay_apply()` only merges the `.dtbo` structure block into the base
+ DTB and ignores the `.dtbo` header and `/memreserve/` block, so all DTBs
+ sharing a base DTB must have identical `boot_cpuid_phys` and `mem_rsv`.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances to validate.
+ """
+ for d in dtbs[1:]:
+ if d.boot_cpuid_phys != dtbs[0].boot_cpuid_phys:
+ raise ValueError(
+ f'boot_cpuid_phys mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].boot_cpuid_phys}) and {d.filename} '
+ f'({d.boot_cpuid_phys})'
+ )
+ if d.mem_rsv != dtbs[0].mem_rsv:
+ raise ValueError(
+ f'/memreserve/ mismatch between {dtbs[0].filename} '
+ f'({dtbs[0].mem_rsv}) and {d.filename} ({d.mem_rsv})'
+ )
+
+
+def _find_common_nodes_and_phandles(dtbs):
+ """Build the common `FactoredTree` skeleton and assign base DTB phandles.
+
+ Any common node that carries a phandle in any input DTB is assigned a
+ phandle in the base DTB so overlays never attempt to add a `phandle`
+ property to an already-existing base DTB node. Non-root nodes that carry a
+ phandle in some input DTB but lack a common symbol in `/__symbols__` across
+ all input DTBs (for example, when two boards define the same node path with
+ different label names) are excluded from `base.dtb` so overlays can create
+ them with board-local phandles and symbols.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ A `FactoredTree` containing all common nodes with `node.phandle` set to
+ their assigned sequential integer phandle (or `None` if the node has no
+ phandle in any input DTB) and `node.fixups` initialized to `{}`.
+ """
+ common_paths = {path for path, _ in dtbs[0].root.walk()}
+ for d in dtbs[1:]:
+ common_paths &= {path for path, _ in d.root.walk()}
+
+ common_sym_targets = {
+ path for label, path in dtbs[0].symbols.items()
+ if all(d.symbols.get(label) == path for d in dtbs[1:])
+ }
+
+ base_root = FactoredTree()
+ next_ph = 1
+ for path, _ in dtbs[0].root.walk():
+ if path not in common_paths:
+ continue
+ if path and base_root.get_node(path.rpartition('/')[0]) is None:
+ continue
+ has_phandle = any(
+ d.root.get_node(path).phandle is not None for d in dtbs
+ )
+ if has_phandle and path and path not in common_sym_targets:
+ continue
+ base_node = base_root.create_path(path)
+ if has_phandle:
+ base_node.phandle = next_ph
+ next_ph += 1
+ return base_root
+
+
+def _find_common_symbols(dtbs, base_root):
+ """Find symbols common to all input DTBs that point to the same base node.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+ base_root: `FactoredTree` containing the common nodes for `base.dtb`.
+
+ Returns:
+ Tuple of `(base_dtb_symbols, path_to_base_dtb_symbol)` where:
+ * `base_dtb_symbols`: Dict mapping symbol label to target node path.
+ * `path_to_base_dtb_symbol`: Dict mapping target node path to its
+ primary symbol label in `base_dtb_symbols`.
+ """
+ base_dtb_symbols = {}
+ path_to_base_dtb_symbol = {}
+
+ for label, chosen_path in dtbs[0].symbols.items():
+ if not all(d.symbols.get(label) == chosen_path for d in dtbs[1:]):
+ continue
+ base_node = base_root.get_node(chosen_path)
+ if base_node is None:
+ continue
+ base_dtb_symbols[label] = chosen_path
+ if (base_node.phandle is not None and
+ chosen_path not in path_to_base_dtb_symbol):
+ path_to_base_dtb_symbol[chosen_path] = label
+
+ return base_dtb_symbols, path_to_base_dtb_symbol
+
+
+def _build_base_tree(dtbs):
+ """Construct the normalized `FactoredTree` for the common base DTB.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances.
+
+ Returns:
+ A `FactoredTree` containing all common nodes, their assigned `phandle`
+ attributes, and their best shared `(masked_bytes, fixups)` properties.
+ """
+ base_root = _find_common_nodes_and_phandles(dtbs)
+ for path, base_node in base_root.walk():
+ dtb_nodes = [d.root.get_node(path) for d in dtbs]
+ common_prop_names = [
+ k for k in dtb_nodes[0].props
+ if all(k in n.props for n in dtb_nodes[1:])
+ ]
+ for prop_name in common_prop_names:
+ eligible = []
+ for n in dtb_nodes:
+ fixups = n.fixups.get(prop_name, ())
+ if not all(base_root.get_node(t_path) is not None
+ for _, t_path in fixups):
+ continue
+ eligible.append((bytes(n.props[prop_name]), fixups))
+
+ chosen = select_best_candidate(eligible, path, prop_name)
+ if chosen is not None:
+ masked_bytes, fixups = chosen
+ base_node.props[prop_name] = libfdt.Property(
+ prop_name, masked_bytes
+ )
+ base_node.fixups[prop_name] = fixups
+
+ return base_root
+
+
+def _serialize_base_dtb(base_root, base_dtb_symbols, rsvmap, boot_cpuid_phys):
+ """Materialize phandles, `/__symbols__`, and `/__local_fixups__` and flatten.
+
+ Args:
+ base_root: Normalized `FactoredTree` for the base DTB from
+ `_build_base_tree()`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ rsvmap: Sequence of `(address, size)` memory reservation tuples.
+ boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+
+ Returns:
+ Serialized `.dtb` `bytes` for the common base DTB.
+ """
+ out_root = devicetree.DeviceTree()
+ lf_entries = []
+
+ for path, base_node in base_root.walk():
+ out_node = out_root.create_path(path)
+ for prop_name, prop in base_node.props.items():
+ masked_bytes = bytes(prop)
+ fixups = base_node.fixups.get(prop_name, ())
+ if not fixups:
+ out_node.props[prop_name] = libfdt.Property(
+ prop_name, masked_bytes
+ )
+ else:
+ val = bytearray(masked_bytes)
+ for off, t_path in fixups:
+ target_ph = base_root.get_node(t_path).phandle
+ struct.pack_into('>I', val, off, target_ph)
+ out_node.props[prop_name] = libfdt.Property(
+ prop_name, bytes(val)
+ )
+ lf_entries.append((path, prop_name, [off for off, _ in fixups]))
+
+ if base_node.phandle is not None:
+ out_node.set_u32('phandle', base_node.phandle)
+
+ if base_dtb_symbols:
+ sym_node = out_root.create_path('/__symbols__')
+ for label, target_path in base_dtb_symbols.items():
+ sym_node.set_str(label, target_path or '/')
+
+ # Create `/__local_fixups__` on the base DTB so `fdt_overlay_apply()`
+ # updates it with the overlay's phandle fixups when merging.
+ lf_root = out_root.create_path('/__local_fixups__')
+ for path, prop_name, offs in lf_entries:
+ lf_root.create_path(path).set_u32_list(prop_name, offs)
+
+ fdt = out_root.flatten(mem_rsv=rsvmap, boot_cpuid_phys=boot_cpuid_phys)
+ return bytes(fdt.as_bytearray())
+
+
+def _assign_local_phandles(dtb, base_root):
+ """Assign sequential overlay-local phandles (`1, 2, ...`) to board-only nodes.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_root: `FactoredTree` containing the common nodes in `base.dtb`.
+
+ Returns:
+ Dict mapping each board-only node path that has a phandle in `dtb` to
+ its sequential overlay-local integer phandle (`1, 2, ...`).
+ """
+ local_phandles = {}
+ next_local_ph = 1
+ for path, node in dtb.root.walk():
+ if node.phandle is not None and base_root.get_node(path) is None:
+ local_phandles[path] = next_local_ph
+ next_local_ph += 1
+ return local_phandles
+
+
+def _encode_dtbo_property(dtb_filename, path, prop_name, masked_bytes, fixups,
+ base_root, path_to_base_dtb_symbol, local_phandles,
+ ext_fixups, ov_local_fixups):
+ """Encode a property value for `/fragment@0/__overlay__` and record fixups.
+
+ Args:
+ dtb_filename: Input DTB filename (for error reporting).
+ path: Normalized node path (`""` for root or `"/..."` for subnodes).
+ prop_name: Property name string.
+ masked_bytes: Normalized property bytes (with zeroed phandle cells).
+ fixups: Tuple of `(offset, target_path)` phandle fixups for this
+ property.
+ base_root: `FactoredTree` containing the common nodes in `base.dtb`.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+ local_phandles: Dict mapping board-only node paths to overlay-local
+ phandles.
+ ext_fixups: Dict mapping target base symbol labels to lists of
+ `/__fixups__` entry strings (mutated in place).
+ ov_local_fixups: Dict mapping `(path, prop_name)` to lists of byte
+ offsets for `/__local_fixups__` (mutated in place).
+
+ Returns:
+ Encoded property value `bytes` with `0xffffffff` placeholders for
+ external base references and local phandle integers for overlay-local
+ references.
+ """
+ if not fixups:
+ return masked_bytes
+
+ val = bytearray(masked_bytes)
+ for off, t_path in fixups:
+ if base_root.get_node(t_path) is not None:
+ sym_label = path_to_base_dtb_symbol.get(t_path)
+ if sym_label is None:
+ raise ValueError(
+ f'{dtb_filename}: property {path or "/"}:{prop_name} '
+ f'references common base node {t_path or "/"} which lacks '
+ f'a common symbol in /__symbols__'
+ )
+ struct.pack_into('>I', val, off, 0xffffffff)
+ entry = f'/fragment@0/__overlay__{path}:{prop_name}:{off}'
+ ext_fixups.setdefault(sym_label, []).append(entry)
+ else:
+ struct.pack_into('>I', val, off, local_phandles[t_path])
+ ov_local_fixups.setdefault((path, prop_name), []).append(off)
+ return bytes(val)
+
+
+def _build_board_dtbo(dtb, base_root, base_dtb_symbols,
+ path_to_base_dtb_symbol):
+ """Construct and flatten the `.dtbo` overlay blob for a single board.
+
+ Args:
+ dtb: `ParsedDtb` instance for the board.
+ base_root: Normalized `FactoredTree` for `base.dtb`.
+ base_dtb_symbols: Dict mapping common symbol labels to target paths.
+ path_to_base_dtb_symbol: Dict mapping common node paths to their symbol
+ labels in `base.dtb`.
+
+ Returns:
+ Serialized `.dtbo` overlay `bytes` for `dtb`.
+ """
+ dtbo_root = devicetree.DeviceTree()
+ frag_node = dtbo_root.create_path('/fragment@0')
+ frag_node.set_str('target-path', '/')
+ ov_root = frag_node.create_path('__overlay__')
+
+ local_phandles = _assign_local_phandles(dtb, base_root)
+ ext_fixups = {}
+ ov_local_fixups = {}
+
+ for path, node in dtb.root.walk():
+ base_node = base_root.get_node(path)
+ diff_props = [
+ prop_name for prop_name, raw_val in node.props.items()
+ if (base_node is None or
+ base_node.props.get(prop_name) != raw_val or
+ base_node.fixups.get(prop_name, ()) !=
+ node.fixups.get(prop_name, ()))
+ ]
+ if base_node is None or diff_props:
+ ov_node = ov_root.create_path(path)
+ for prop_name in diff_props:
+ ov_node.props[prop_name] = libfdt.Property(
+ prop_name,
+ _encode_dtbo_property(
+ dtb.filename, path, prop_name,
+ bytes(node.props[prop_name]),
+ node.fixups.get(prop_name, ()), base_root,
+ path_to_base_dtb_symbol, local_phandles, ext_fixups,
+ ov_local_fixups,
+ ),
+ )
+ if path in local_phandles:
+ ov_node.set_u32('phandle', local_phandles[path])
+
+ diff_syms = [
+ (label, t_path)
+ for label, t_path in dtb.symbols.items()
+ if base_dtb_symbols.get(label) != t_path
+ ]
+ if diff_syms:
+ sym_node = dtbo_root.create_path('/__symbols__')
+ for label, t_path in diff_syms:
+ ov_root.create_path(t_path)
+ sym_node.set_str(label, f'/fragment@0/__overlay__{t_path}')
+
+ if ext_fixups:
+ fixups_node = dtbo_root.create_path('/__fixups__')
+ for label, entries in ext_fixups.items():
+ fixups_node.set_stringlist(label, entries)
+
+ if ov_local_fixups:
+ ov_lf_root = dtbo_root.create_path(
+ '/__local_fixups__/fragment@0/__overlay__'
+ )
+ for (path, prop_name), offs in ov_local_fixups.items():
+ ov_lf_root.create_path(path).set_u32_list(prop_name, offs)
+
+ return bytes(dtbo_root.flatten().as_bytearray())
+
+
+def create_base_dtb_and_overlays(dtbs):
+ """Compute the base DTB and per-board DTBO blobs from parsed DTBs.
+
+ Args:
+ dtbs: Sequence of `ParsedDtb` instances (at least two).
+
+ Returns:
+ Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
+ serialized `base.dtb` `bytes` and `dtbo_blobs` is a list of serialized
+ `.dtbo` `bytes` corresponding 1:1 with `dtbs`.
+ """
+ _validate_dtb_headers(dtbs)
+ base_root = _build_base_tree(dtbs)
+ base_dtb_symbols, path_to_base_dtb_symbol = _find_common_symbols(
+ dtbs, base_root
+ )
+ base_dtb_blob = _serialize_base_dtb(
+ base_root, base_dtb_symbols,
+ dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
+ )
+ dtbo_blobs = [
+ _build_board_dtbo(
+ d, base_root, base_dtb_symbols, path_to_base_dtb_symbol,
+ )
+ for d in dtbs
+ ]
+ return base_dtb_blob, dtbo_blobs
+
+
+def get_dtbo_output_path(dtb_path, outdir):
+ """Determine the output `.dtbo` file path for an input `.dtb` path.
+
+ Args:
+ dtb_path: Path to the input `.dtb` file.
+ outdir: Output directory path, or `None` to place the `.dtbo` alongside
+ `dtb_path`.
+
+ Returns:
+ Output `.dtbo` file path string.
+ """
+ dtb_dir, filename = os.path.split(dtb_path)
+ stem = filename[:-4] if filename.endswith('.dtb') else filename
+ target_dir = outdir if outdir is not None else dtb_dir
+ return os.path.join(target_dir, f'{stem}.dtbo')
+
+
+def parse_args(argv=None):
+ """Parse command-line arguments.
+
+ Args:
+ argv: Optional list of command-line argument strings (defaults to
+ `sys.argv[1:]`).
+
+ Returns:
+ Parsed `argparse.Namespace` instance.
+ """
+ parser = argparse.ArgumentParser(
+ description=(
+ 'Split a set of compiled device tree blobs (.dtb) into a single '
+ 'common base .dtb and per-board overlay blobs (.dtbo).'
+ )
+ )
+ parser.add_argument(
+ '--output-base-dtb',
+ default='base.dtb',
+ metavar='FILENAME',
+ help=(
+ 'Output common base .dtb filename (default: base.dtb; placed in '
+ '--outdir if specified, or current directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '-o', '--outdir',
+ default=None,
+ metavar='DIR',
+ help=(
+ 'Output directory for generated .dtbo files and '
+ '--output-base-dtb (default: alongside each input .dtb for .dtbo, '
+ 'current directory for --output-base-dtb)'
+ ),
+ )
+ parser.add_argument(
+ '-s', '--stats',
+ action='store_true',
+ help='Print size statistics summary',
+ )
+ parser.add_argument(
+ 'dtbs',
+ nargs='+',
+ metavar='DTB',
+ help='Input .dtb files (at least two required)',
+ )
+ args = parser.parse_args(argv)
+ if os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a filename without directory '
+ 'components; use -o/--outdir to specify the output directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
+ return args
+
+
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
+ """Print uncompressed size savings summary.
+
+ Args:
+ dtbs: Sequence of input `ParsedDtb` instances.
+ base_dtb_blob: Uncompressed base DTB `bytes`.
+ dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ """
+ orig_uncomp = sum(len(d.raw_bytes) for d in dtbs)
+ base_dtb_uncomp = len(base_dtb_blob)
+ dtbo_uncomp = sum(len(b) for b in dtbo_blobs)
+ new_uncomp = base_dtb_uncomp + dtbo_uncomp
+ saved_uncomp = orig_uncomp - new_uncomp
+ pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
+ print(
+ f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
+ f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
+ )
+
+
+def factor_dtb_files(args):
+ """Factor a list of standalone .dtb files into a base .dtb and .dtbo files.
+
+ Args:
+ args: Parsed `argparse.Namespace` from `parse_args()`.
+ """
+ input_dtb_paths = args.dtbs
+ dtbs = [ParsedDtb(p) for p in input_dtb_paths]
+
+ base_dtb_path = (
+ os.path.join(args.outdir, args.output_base_dtb)
+ if args.outdir is not None
+ else args.output_base_dtb
+ )
+
+ input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
+ base_abs_path = os.path.abspath(base_dtb_path)
+ if base_abs_path in input_abs_paths:
+ raise ValueError(
+ f'base DTB output path {base_dtb_path!r} collides with '
+ f'an input DTB file'
+ )
+
+ dtbo_paths = []
+ seen_paths = {base_abs_path}
+ for dtb_path in input_dtb_paths:
+ out_path = get_dtbo_output_path(dtb_path, args.outdir)
+ abs_out = os.path.abspath(out_path)
+ if abs_out in input_abs_paths:
+ raise ValueError(
+ f'output path {out_path!r} collides with an input DTB file'
+ )
+ if abs_out in seen_paths:
+ raise ValueError(
+ f'output path collision on {out_path!r}; ensure input '
+ f'files have distinct names'
+ )
+ seen_paths.add(abs_out)
+ dtbo_paths.append(out_path)
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+
+ if args.outdir:
+ os.makedirs(args.outdir, exist_ok=True)
+ with open(base_dtb_path, 'wb') as f:
+ f.write(base_dtb_blob)
+
+ for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ out_dir = os.path.dirname(out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(out_path, 'wb') as f:
+ f.write(dtbo_blob)
+
+ if args.stats:
+ _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+
+
+def main(argv=None):
+ """Run the fdtfactor command-line tool.
+
+ Args:
+ argv: Optional list of command-line argument strings.
+
+ Returns:
+ Integer exit status (`0` on success).
+ """
+ args = parse_args(argv)
+ try:
+ factor_dtb_files(args)
+ except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc:
+ sys.exit(f'Error: {exc}')
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/meson.build b/meson.build
index 15a421e1296a..ed4cd269cec6 100644
--- a/meson.build
+++ b/meson.build
@@ -119,6 +119,7 @@ if get_option('tools') and not wheel_only
install_data(
'dtdiff',
+ 'fdtfactor',
install_dir: get_option('bindir'),
install_mode: 'rwxr-xr-x',
)
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index af5d309cf8a9..ea1d35cd8e4e 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,10 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
-TESTS_CLEANDIRS_L = fs
+TESTS_CLEANDIRS_L = fs *.test.dir
TESTS_CLEANDIRS = $(TESTS_CLEANDIRS_L:%=$(TESTS_PREFIX)%)
.PHONY: tests
diff --git a/tests/fdtfactor_board1.dts b/tests/fdtfactor_board1.dts
new file mode 100644
index 000000000000..c5f4e1672d98
--- /dev/null
+++ b/tests/fdtfactor_board1.dts
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board1";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ };
+
+ b1_node: board1-node {
+ prop-str = "board1";
+ ref-base = <&sn1>;
+ ref-local = <&b1_node &b1_child>;
+
+ b1_child: child {
+ reg = <1>;
+ };
+ };
+};
diff --git a/tests/fdtfactor_board2.dts b/tests/fdtfactor_board2.dts
new file mode 100644
index 000000000000..32172f755321
--- /dev/null
+++ b/tests/fdtfactor_board2.dts
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board2";
+ prop-int = <0xdeadbeef>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn1>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <&sn2>;
+ board2-only-prop = "only-on-board2";
+ board2-ref = <&b2_node>;
+ };
+
+ b2_node: board2-node {
+ prop-str = "board2";
+ ref-base = <&ssn1>;
+ ref-self = <&b2_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3.dts b/tests/fdtfactor_board3.dts
new file mode 100644
index 000000000000..b93dd2545a0f
--- /dev/null
+++ b/tests/fdtfactor_board3.dts
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-base = <&sn2>;
+ ref-self = <&b3_node>;
+ };
+};
diff --git a/tests/fdtfactor_board3_shifted.dts b/tests/fdtfactor_board3_shifted.dts
new file mode 100644
index 000000000000..3ada7bda4545
--- /dev/null
+++ b/tests/fdtfactor_board3_shifted.dts
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "test_fdtfactor";
+ model = "board3";
+ prop-int = <0x12345678>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ b3_node: board3-node {
+ prop-str = "board3";
+ ref-self = <&b3_node>;
+ ref-base = <&sn2>;
+ };
+
+ sn1: subnode@1 {
+ compatible = "subnode1";
+ reg = <1>;
+ prop-ref = <&sn1 0x10 &ssn1 0x20>;
+
+ ssn1: subsubnode {
+ compatible = "subsubnode1", "subsubnode";
+ prop-phandle = <&sn2>;
+ };
+ };
+
+ sn2: subnode@2 {
+ reg = <2>;
+ prop-maybe-phandle = <0>;
+ };
+};
diff --git a/tests/meson.build b/tests/meson.build
index 8dbfc3f9e8e2..44497da6cec1 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -143,7 +143,7 @@ run_test_deps = [
dtc_tools, treegen_dtb, tests_exe,
]
if pylibfdt_enabled
- run_test_types += 'pylibfdt'
+ run_test_types += ['pylibfdt', 'fdtfactor']
run_test_deps += pylibfdt
endif
foreach test_type : run_test_types
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index d51a111c7819..166a662708a5 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1183,6 +1183,68 @@ pylibfdt_tests () {
tot_tests=$((tot_tests + $total_tests))
}
+fdtfactor_tests () {
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_board${i}.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+
+ # Default mode: factor 3 boards into base.dtb + .dtbo overlays and verify
+ # that majority properties land in base.dtb and fdtoverlay reconstructs
+ # each original DTB (including /__local_fixups__, /memreserve/, and
+ # boot_cpuid_phys).
+ rm -rf fdtfactor_out.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s --output-base-dtb base.test.dtb \
+ -o fdtfactor_out.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb fdtfactor_board3.test.dtb
+ run_fdtget_test "deadbeef" -tx fdtfactor_out.test.dir/base.test.dtb \
+ / prop-int
+ run_wrap_error_test $DTGET \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ /fragment@0/__overlay__ prop-int
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_out.test.dir/base.test.dtb \
+ -o fdtfactor_out.test.dir/merged${i}.test.dtb \
+ fdtfactor_out.test.dir/fdtfactor_board${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.test.dtb \
+ fdtfactor_out.test.dir/merged${i}.test.dtb
+ done
+
+ # Shifted phandles: fdtfactor_board3_shifted.dts defines board3-node before
+ # subnode@1 so all phandle numbers differ from board1 and board2; verify
+ # that fdtfactor normalizes them via /__local_fixups__.
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_board3_shifted.test.dtb \
+ "$SRCDIR/fdtfactor_board3_shifted.dts"
+ rm -rf fdtfactor_shift.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -o fdtfactor_shift.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3_shifted.test.dtb
+ run_wrap_error_test $DTGET \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo \
+ /fragment@0/__overlay__/subnode@1 prop-ref
+ run_wrap_test $FDTOVERLAY -i fdtfactor_shift.test.dir/base.dtb \
+ -o fdtfactor_shift.test.dir/merged3.test.dtb \
+ fdtfactor_shift.test.dir/fdtfactor_board3_shifted.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board3.test.dtb \
+ fdtfactor_shift.test.dir/merged3.test.dtb
+
+ # Legacy linux,phandle properties (dtc -H both / -H legacy), overlay .dtbo
+ # inputs, and inputs lacking /__local_fixups__ are rejected.
+ run_dtc_test -H both -@ -L -I dts -O dtb \
+ -o fdtfactor_legacy_ph.test.dtb "$SRCDIR/fdtfactor_board1.dts"
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_legacy_ph.test.dtb fdtfactor_board2.test.dtb
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_out.test.dir/fdtfactor_board1.test.dtbo \
+ fdtfactor_board2.test.dtb
+ for i in 1 2 3; do
+ run_dtc_test -b 2 -@ -I dts -O dtb -o fdtfactor_board${i}.nolf.test.dtb \
+ "$SRCDIR/fdtfactor_board${i}.dts"
+ done
+ run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+}
+
while getopts "vt:me" ARG ; do
case $ARG in
"v")
@@ -1203,14 +1265,15 @@ done
if [ -z "$TESTSETS" ]; then
TESTSETS="libfdt utilfdt dtc dtbs_equal fdtget fdtput fdtdump fdtoverlay"
- # Test pylibfdt if the libfdt Python module is available.
+ # Test pylibfdt and fdtfactor if the libfdt Python module is available.
if ! $no_python; then
- TESTSETS="$TESTSETS pylibfdt"
+ TESTSETS="$TESTSETS pylibfdt fdtfactor"
fi
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -rf *.test.dir
for set in $TESTSETS; do
case $set in
@@ -1241,6 +1304,9 @@ for set in $TESTSETS; do
"fdtoverlay")
fdtoverlay_tests
;;
+ "fdtfactor")
+ fdtfactor_tests
+ ;;
esac
done
diff --git a/tests/testutils.sh b/tests/testutils.sh
index 6b2f0d14e3e5..7f35ba8e4158 100644
--- a/tests/testutils.sh
+++ b/tests/testutils.sh
@@ -27,6 +27,7 @@ DTGET=${TEST_BINDIR}/fdtget
DTPUT=${TEST_BINDIR}/fdtput
FDTDUMP=${TEST_BINDIR}/fdtdump
FDTOVERLAY=${TEST_BINDIR}/fdtoverlay
+FDTFACTOR=${SRCDIR}/../fdtfactor
verbose_run () {
if [ -z "$QUIET_TEST" ]; then
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 13/15] fdtfactor: Add -S/--strip-local-fixups option
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (11 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 12/15] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 14/15] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
` (2 subsequent siblings)
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
By default, `fdtfactor` preserves `/__local_fixups__` on `base.dtb` so
that when `fdt_overlay_apply()` merges a `.dtbo` overlay, the
reconstructed device tree retains full `/__local_fixups__` information
(for instance, if an OS kernel wants to know which property cells
contain phandles at runtime).
When `-L` was passed to `dtc` solely to help `fdtfactor` locate phandle
cells at build time and `/__local_fixups__` is not needed at runtime,
retaining `/__local_fixups__` in `base.dtb` wastes space.
Add `-S` / `--strip-local-fixups` to omit `/__local_fixups__` from
`base.dtb` (and therefore from the merged device trees produced by
`fdt_overlay_apply()`). On a set of 47 Qualcomm SC7180 Trogdor
Chromebook board DTBs, `fdtfactor` without `-S` reduces total
uncompressed size from 7.69 MB to 932 KB (88.2% savings), while
`fdtfactor -S` reduces it to 634 KB (91.8% savings, saving an additional
298 KB by omitting `/__local_fixups__` from `base.dtb`).
Add a test case in `tests/run_tests.sh` verifying that `fdtfactor -S`
followed by `fdtoverlay` reproduces DTBs compiled with `-@` (without
`-L`) via `dtbs_equal_unordered`.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Include both non--S (932 KB) and -S (634 KB) Trogdor size numbers in
the commit message to show the savings of -S itself (Simon Glass)
- Pass -S to the negative test verifying that inputs lacking
/__local_fixups__ are still rejected when -S is used, and move the
non--S rejection test to the base fdtfactor patch (Simon Glass)
Changes in v2:
- Split -S/--strip-local-fixups into a separate opt-in patch (David
Gibson)
README.md | 9 +++++++++
fdtfactor | 34 ++++++++++++++++++++++++++--------
tests/run_tests.sh | 18 ++++++++++++++++++
3 files changed, 53 insertions(+), 8 deletions(-)
diff --git a/README.md b/README.md
index 61fefc4424fe..bb6f041e1e78 100644
--- a/README.md
+++ b/README.md
@@ -139,6 +139,15 @@ Options:
* `-o`, `--outdir`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
+* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
+ `base.dtb` and final reconstructed device trees. By default, `fdtfactor`
+ preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
+ merges a `.dtbo` overlay, the reconstructed device tree retains full
+ `/__local_fixups__` information (e.g., for an OS kernel that wants to
+ know which property cells contain phandles at runtime). When `-L` was
+ passed to `dtc` solely to help `fdtfactor` locate phandles at build time
+ and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
+ from `base.dtb` to save space.
* `-s`, `--stats`: Print size savings summary.
### TODO
diff --git a/fdtfactor b/fdtfactor
index f6944e36a0f2..7442a2597399 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -442,7 +442,8 @@ def _build_base_tree(dtbs):
return base_root
-def _serialize_base_dtb(base_root, base_dtb_symbols, rsvmap, boot_cpuid_phys):
+def _serialize_base_dtb(base_root, base_dtb_symbols, rsvmap, boot_cpuid_phys,
+ strip_local_fixups=False):
"""Materialize phandles, `/__symbols__`, and `/__local_fixups__` and flatten.
Args:
@@ -451,6 +452,8 @@ def _serialize_base_dtb(base_root, base_dtb_symbols, rsvmap, boot_cpuid_phys):
base_dtb_symbols: Dict mapping common symbol labels to target paths.
rsvmap: Sequence of `(address, size)` memory reservation tuples.
boot_cpuid_phys: Physical boot CPU ID for the FDT header.
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated `base.dtb`.
Returns:
Serialized `.dtb` `bytes` for the common base DTB.
@@ -485,11 +488,12 @@ def _serialize_base_dtb(base_root, base_dtb_symbols, rsvmap, boot_cpuid_phys):
for label, target_path in base_dtb_symbols.items():
sym_node.set_str(label, target_path or '/')
- # Create `/__local_fixups__` on the base DTB so `fdt_overlay_apply()`
- # updates it with the overlay's phandle fixups when merging.
- lf_root = out_root.create_path('/__local_fixups__')
- for path, prop_name, offs in lf_entries:
- lf_root.create_path(path).set_u32_list(prop_name, offs)
+ if not strip_local_fixups:
+ # Create `/__local_fixups__` on the base DTB so `fdt_overlay_apply()`
+ # updates it with the overlay's phandle fixups when merging.
+ lf_root = out_root.create_path('/__local_fixups__')
+ for path, prop_name, offs in lf_entries:
+ lf_root.create_path(path).set_u32_list(prop_name, offs)
fdt = out_root.flatten(mem_rsv=rsvmap, boot_cpuid_phys=boot_cpuid_phys)
return bytes(fdt.as_bytearray())
@@ -638,11 +642,13 @@ def _build_board_dtbo(dtb, base_root, base_dtb_symbols,
return bytes(dtbo_root.flatten().as_bytearray())
-def create_base_dtb_and_overlays(dtbs):
+def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
"""Compute the base DTB and per-board DTBO blobs from parsed DTBs.
Args:
dtbs: Sequence of `ParsedDtb` instances (at least two).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated `base.dtb`.
Returns:
Tuple of `(base_dtb_blob, dtbo_blobs)` where `base_dtb_blob` is the
@@ -657,6 +663,7 @@ def create_base_dtb_and_overlays(dtbs):
base_dtb_blob = _serialize_base_dtb(
base_root, base_dtb_symbols,
dtbs[0].mem_rsv, dtbs[0].boot_cpuid_phys,
+ strip_local_fixups=strip_local_fixups,
)
dtbo_blobs = [
_build_board_dtbo(
@@ -719,6 +726,14 @@ def parse_args(argv=None):
'current directory for --output-base-dtb)'
),
)
+ parser.add_argument(
+ '-S', '--strip-local-fixups',
+ action='store_true',
+ help=(
+ 'Omit /__local_fixups__ from the generated base .dtb and '
+ 'reconstructed device trees'
+ ),
+ )
parser.add_argument(
'-s', '--stats',
action='store_true',
@@ -803,7 +818,10 @@ def factor_dtb_files(args):
seen_paths.add(abs_out)
dtbo_paths.append(out_path)
- base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(dtbs)
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ dtbs,
+ strip_local_fixups=args.strip_local_fixups,
+ )
if args.outdir:
os.makedirs(args.outdir, exist_ok=True)
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 166a662708a5..e5ac38dd2bab 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1243,6 +1243,24 @@ fdtfactor_tests () {
done
run_wrap_error_test $PYTHON $FDTFACTOR -o fdtfactor_legacy.test.dir \
fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+
+ # Strip local fixups (-S): verify that merged DTBs match DTBs compiled
+ # with -@ (without -L), and that inputs lacking /__local_fixups__ are
+ # still rejected even when -S is passed.
+ run_wrap_error_test $PYTHON $FDTFACTOR -S -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.nolf.test.dtb fdtfactor_board2.test.dtb
+ rm -rf fdtfactor_strip.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -S --output-base-dtb base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ for i in 1 2 3; do
+ run_wrap_test $FDTOVERLAY -i fdtfactor_strip.test.dir/base_strip.test.dtb \
+ -o fdtfactor_strip.test.dir/merged${i}.test.dtb \
+ fdtfactor_strip.test.dir/fdtfactor_board${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
+ fdtfactor_strip.test.dir/merged${i}.test.dtb
+ done
}
while getopts "vt:me" ARG ; do
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 14/15] fdtfactor: Support compressed DTB inputs and outputs
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (12 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 13/15] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-01 22:01 ` [PATCH v3 15/15] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-02 4:44 ` [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Firmware and FIT build pipelines frequently store device tree blobs
compressed with LZ4, LZMA/XZ, or gzip (e.g., `.dtb.lz4`).
Extend `fdtfactor` to detect compressed input blobs by magic bytes or
file extension, transparently decompress them before factoring, and
compress the resulting `.dtbo` and `base.dtb` outputs when appropriate:
- Support `.lz4` (including legacy `-l` frames via the `lz4` CLI),
`.lzma`/`.xz` (via Python's `lzma` module, using `CHECK_CRC32` for XZ
compatibility with kernel/bootloader decompressors), and `.gz` (via
Python's `gzip` module).
- Preserve the input compression format on generated `.dtbo` files and
compress `base.dtb` according to `--base-compression` (which defaults
to `auto`, auto-detecting from `--output-base-dtb`'s filename
extension such as `--output-base-dtb base.dtb.lz4`).
- Report compressed size savings alongside uncompressed savings when
`-s`/`--stats` is specified.
On the 47 Qualcomm SC7180 Trogdor Chromebook `.dtb.lz4` files,
`fdtfactor -S` reduces total LZ4-compressed size from 1.71 MB to 213 KB
(87.5% savings).
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Update for -b/--base-dtb rename to --output-base-dtb (David Gibson)
Changes in v2:
- Add --base-compression to explicitly control base.dtb compression
while documenting auto-detection from --base-dtb's filename suffix
(David Gibson)
- Support legacy LZ4 frames (lz4 -l) and use CRC32 check for XZ outputs
(Simon Glass)
README.md | 17 ++-
fdtfactor | 318 +++++++++++++++++++++++++++++++++++++++----
tests/Makefile.tests | 5 +-
tests/run_tests.sh | 82 ++++++++++-
4 files changed, 391 insertions(+), 31 deletions(-)
diff --git a/README.md b/README.md
index bb6f041e1e78..a3d1ad8c58f3 100644
--- a/README.md
+++ b/README.md
@@ -115,10 +115,10 @@ values.
## fdtfactor
`fdtfactor` is a Python tool built on `pylibfdt` that takes a set of
-compiled device tree blobs (`.dtb`) and splits them into a single common
-`base.dtb` and per-board `.dtbo` overlays so that applying a board's
-`.dtbo` to `base.dtb` recreates the original board device tree while
-reducing total storage size.
+compiled device tree blobs (`.dtb`, or compressed `.dtb.lz4`, `.dtb.xz`,
+`.dtb.lzma`, `.dtb.gz`) and splits them into a single common `base.dtb` and
+per-board `.dtbo` overlays so that applying a board's `.dtbo` to `base.dtb`
+recreates the original board device tree while reducing total storage size.
Input `.dtb` files must be compiled with `dtc -@ -L` so that both
`/__symbols__` (used to resolve overlay `/__fixups__` against `base.dtb`)
@@ -135,7 +135,12 @@ Options:
* `--output-base-dtb`: Output filename for the common base `.dtb`
(default: `base.dtb`; placed in `--outdir` if specified, or the current
- directory otherwise).
+ directory otherwise). By default (`--base-compression=auto`), `base.dtb`
+ is compressed if its filename carries a `.lz4`, `.xz`, `.lzma`, or `.gz`
+ suffix, whereas each `.dtbo` preserves its input `.dtb`'s compression.
+* `--base-compression`: Compression format for `--output-base-dtb`: `auto`,
+ `none`, `lz4`, `lz4_legacy`, `lzma`, `xz`, or `gzip` (default: `auto`,
+ which auto-detects from the `--output-base-dtb` filename suffix).
* `-o`, `--outdir`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
@@ -148,7 +153,7 @@ Options:
passed to `dtc` solely to help `fdtfactor` locate phandles at build time
and `/__local_fixups__` is not needed at runtime, passing `-S` strips it
from `base.dtb` to save space.
-* `-s`, `--stats`: Print size savings summary.
+* `-s`, `--stats`: Print uncompressed and compressed size savings.
### TODO
diff --git a/fdtfactor b/fdtfactor
index 7442a2597399..9bc8bfde2038 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -12,14 +12,223 @@ original board DTB.
import argparse
from collections import Counter
+import gzip
+import lzma
import os
import struct
+import subprocess
import sys
import devicetree
import libfdt
+FDT_MAGIC = b'\xd0\x0d\xfe\xed'
+LZ4_MAGIC = b'\x04\x22\x4d\x18'
+LZ4_LEGACY_MAGIC = b'\x02\x21\x4c\x18'
+GZIP_MAGIC = b'\x1f\x8b'
+XZ_MAGIC = b'\xfd7zXZ\x00'
+# Matches raw LZMA_ALONE headers with default lc=3, lp=0, pb=2 (0x5d) and
+# power-of-two dictionary sizes >= 64 KiB; .lzma extension handles any
+# non-default LZMA headers.
+LZMA_ALONE_MAGIC = b'\x5d\x00\x00'
+
+COMP_TO_EXT = {
+ 'none': '',
+ 'lz4': '.lz4',
+ 'lz4_legacy': '.lz4',
+ 'lzma': '.lzma',
+ 'xz': '.xz',
+ 'gzip': '.gz',
+}
+
+EXT_TO_COMP = {
+ '.lz4': 'lz4',
+ '.lzma': 'lzma',
+ '.xz': 'xz',
+ '.gz': 'gzip',
+}
+
+
+def detect_compression(raw_bytes, filename=''):
+ """Detect compression algorithm from magic bytes or filename extension.
+
+ Args:
+ raw_bytes: Raw file bytes to inspect for magic headers.
+ filename: Optional filename string used as a fallback if `raw_bytes`
+ does not match a known magic header.
+
+ Returns:
+ Compression format string (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ """
+ if raw_bytes.startswith(FDT_MAGIC):
+ return 'none'
+ if raw_bytes.startswith(LZ4_MAGIC):
+ return 'lz4'
+ if raw_bytes.startswith(LZ4_LEGACY_MAGIC):
+ return 'lz4_legacy'
+ if raw_bytes.startswith(GZIP_MAGIC):
+ return 'gzip'
+ if raw_bytes.startswith(XZ_MAGIC):
+ return 'xz'
+ if raw_bytes.startswith(LZMA_ALONE_MAGIC):
+ return 'lzma'
+ _, ext = os.path.splitext(filename)
+ if ext in EXT_TO_COMP:
+ return EXT_TO_COMP[ext]
+ return 'none'
+
+
+def _run_lz4(args, input_bytes, action):
+ """Run the `lz4` CLI with `args` on `input_bytes` and return stdout.
+
+ Args:
+ args: List of command-line arguments to pass to `lz4`.
+ input_bytes: Bytes to feed to `lz4` standard input.
+ action: Human-readable action description for error messages.
+
+ Returns:
+ Output `bytes` written to `lz4` standard output.
+ """
+ try:
+ res = subprocess.run(
+ ['lz4', *args],
+ input=input_bytes,
+ capture_output=True,
+ check=False,
+ )
+ except FileNotFoundError as exc:
+ raise RuntimeError(
+ 'lz4 command not found; install lz4 to process .lz4 files'
+ ) from exc
+ if res.returncode != 0:
+ err = res.stderr.decode('utf-8', errors='replace').strip()
+ raise RuntimeError(f'Failed to {action} with lz4: {err}')
+ return res.stdout
+
+
+def decompress_blob(raw_bytes, filename='', comp=None):
+ """Decompress a DTB blob if compressed, returning `(raw_bytes, comp)`.
+
+ Args:
+ raw_bytes: Input file or property `bytes` (compressed or uncompressed).
+ filename: Optional filename or node description for error reporting and
+ extension fallback.
+ comp: Optional explicit compression type (`'none'`, `'lz4'`,
+ `'lz4_legacy'`, `'lzma'`, `'xz'`, `'gzip'`), or `None` to
+ auto-detect via `detect_compression()`.
+
+ Returns:
+ Tuple of `(uncompressed_bytes, resolved_compression_type)`.
+ """
+ if not comp:
+ comp = detect_compression(raw_bytes, filename)
+ elif comp == 'lz4' and raw_bytes.startswith(LZ4_LEGACY_MAGIC):
+ comp = 'lz4_legacy'
+ if comp == 'none':
+ return raw_bytes, 'none'
+ if comp in ('lz4', 'lz4_legacy'):
+ return _run_lz4(['-d', '-q', '-c'], raw_bytes, f'decompress {filename}'), comp
+ if comp == 'gzip':
+ try:
+ return gzip.decompress(raw_bytes), 'gzip'
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {filename} with gzip: {exc}'
+ ) from exc
+ if comp in ('lzma', 'xz'):
+ try:
+ return lzma.decompress(raw_bytes), comp
+ except Exception as exc:
+ raise RuntimeError(
+ f'Failed to decompress {filename} with {comp}: {exc}'
+ ) from exc
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
+def compress_blob(raw_bytes, comp, filename=''):
+ """Compress an FDT blob using the specified compression algorithm.
+
+ Args:
+ raw_bytes: Uncompressed FDT `bytes`.
+ comp: Compression format (`'none'`, `'lz4'`, `'lz4_legacy'`, `'lzma'`,
+ `'xz'`, or `'gzip'`).
+ filename: Optional filename for error messages.
+
+ Returns:
+ Compressed (or uncompressed if `comp` is `'none'`) `bytes`.
+ """
+ if comp in ('none', '', None):
+ return raw_bytes
+ target = filename or 'blob'
+ if comp == 'lz4':
+ return _run_lz4(
+ ['-12', '-z', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'lz4_legacy':
+ return _run_lz4(
+ ['-l', '-12', '-q', '-c'], raw_bytes, f'compress {target}'
+ )
+ if comp == 'gzip':
+ return gzip.compress(raw_bytes, mtime=0)
+ if comp == 'lzma':
+ return lzma.compress(raw_bytes, format=lzma.FORMAT_ALONE)
+ if comp == 'xz':
+ return lzma.compress(
+ raw_bytes, format=lzma.FORMAT_XZ, check=lzma.CHECK_CRC32
+ )
+ prefix = f'{filename}: ' if filename else ''
+ raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+
+
+def split_dtb_filename(path):
+ """Split a DTB path into `(directory, stem, compression_type)`.
+
+ Strips a recognized compression suffix (`.lz4`, `.lzma`, `.xz`, `.gz`), if
+ present, followed by a `.dtb` or `.dtbo` suffix, if present. Note that for
+ input files, `ParsedDtb` detects the actual compression format from the
+ file's magic bytes via `decompress_blob()` rather than relying on the
+ filename extension; `split_dtb_filename()` is used to derive the output
+ `.dtbo` filename stem and (when `--base-compression=auto`) to infer the
+ default compression format for `--output-base-dtb` from its output
+ filename.
+
+ Examples:
+ >>> split_dtb_filename('/path/to/board.dtb')
+ ('/path/to', 'board', 'none')
+ >>> split_dtb_filename('board.dtbo')
+ ('', 'board', 'none')
+ >>> split_dtb_filename('/path/to/board.dtb.lz4')
+ ('/path/to', 'board', 'lz4')
+ >>> split_dtb_filename('base.dtb.xz')
+ ('', 'base', 'xz')
+ >>> split_dtb_filename('/path/to/board.rev2.dtb.gz')
+ ('/path/to', 'board.rev2', 'gzip')
+ >>> split_dtb_filename('/path/to/board.custom.lzma')
+ ('/path/to', 'board.custom', 'lzma')
+ >>> split_dtb_filename('my_blob')
+ ('', 'my_blob', 'none')
+
+ Args:
+ path: Input or output DTB file path string.
+
+ Returns:
+ Tuple of `(directory, stem, compression_type)`.
+ """
+ directory, filename = os.path.split(path)
+ comp = 'none'
+ prefix, sep, ext = filename.rpartition('.')
+ if prefix and sep and f'.{ext}' in EXT_TO_COMP:
+ comp = EXT_TO_COMP[f'.{ext}']
+ filename = prefix
+ stem, sep, ext = filename.rpartition('.')
+ if not stem or not sep or ext not in ('dtb', 'dtbo'):
+ stem = filename
+ return directory, stem, comp
+
+
def _is_dtbo_overlay(fdt):
"""Return True if `fdt` is a `.dtbo` overlay rather than a base DTB.
@@ -78,11 +287,13 @@ class ParsedDtb:
"""Parse an input FDT file into a normalized FactoredTree.
Args:
- filename: File path of the `.dtb` file to read.
+ filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
"""
self.filename = filename
with open(filename, 'rb') as f:
- raw_bytes = f.read()
+ file_bytes = f.read()
+ self.compressed_size = len(file_bytes)
+ raw_bytes, self.compression = decompress_blob(file_bytes, filename)
self.raw_bytes = raw_bytes
try:
@@ -674,21 +885,23 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
-def get_dtbo_output_path(dtb_path, outdir):
- """Determine the output `.dtbo` file path for an input `.dtb` path.
+def get_dtbo_output_path(dtb_path, outdir, comp='none'):
+ """Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
Args:
- dtb_path: Path to the input `.dtb` file.
+ dtb_path: Path to the input `.dtb` (or `.dtb.*`) file.
outdir: Output directory path, or `None` to place the `.dtbo` alongside
`dtb_path`.
+ comp: Compression format string for the output `.dtbo` (default:
+ `'none'`).
Returns:
- Output `.dtbo` file path string.
+ Output `.dtbo` (or compressed `.dtbo.*`) file path string.
"""
- dtb_dir, filename = os.path.split(dtb_path)
- stem = filename[:-4] if filename.endswith('.dtb') else filename
+ dtb_dir, stem, _ = split_dtb_filename(dtb_path)
+ dtbo_name = f'{stem}.dtbo{COMP_TO_EXT[comp]}'
target_dir = outdir if outdir is not None else dtb_dir
- return os.path.join(target_dir, f'{stem}.dtbo')
+ return os.path.join(target_dir, dtbo_name)
def parse_args(argv=None):
@@ -703,8 +916,9 @@ def parse_args(argv=None):
"""
parser = argparse.ArgumentParser(
description=(
- 'Split a set of compiled device tree blobs (.dtb) into a single '
- 'common base .dtb and per-board overlay blobs (.dtbo).'
+ 'Split a set of compiled device tree blobs (.dtb, or compressed '
+ '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and '
+ 'per-board overlay blobs (.dtbo).'
)
)
parser.add_argument(
@@ -712,8 +926,18 @@ def parse_args(argv=None):
default='base.dtb',
metavar='FILENAME',
help=(
- 'Output common base .dtb filename (default: base.dtb; placed in '
- '--outdir if specified, or current directory otherwise)'
+ 'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) filename '
+ '(default: base.dtb; placed in --outdir if specified, or current '
+ 'directory otherwise)'
+ ),
+ )
+ parser.add_argument(
+ '--base-compression',
+ choices=('auto', 'none', 'lz4', 'lz4_legacy', 'lzma', 'xz', 'gzip'),
+ default='auto',
+ help=(
+ 'Compression format for --output-base-dtb (default: auto, which '
+ 'auto-detects from the --output-base-dtb filename extension)'
),
)
parser.add_argument(
@@ -737,13 +961,16 @@ def parse_args(argv=None):
parser.add_argument(
'-s', '--stats',
action='store_true',
- help='Print size statistics summary',
+ help='Print size statistics and compression summary',
)
parser.add_argument(
'dtbs',
nargs='+',
metavar='DTB',
- help='Input .dtb files (at least two required)',
+ help=(
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files '
+ '(at least two required)'
+ ),
)
args = parser.parse_args(argv)
if os.path.dirname(args.output_base_dtb):
@@ -756,13 +983,17 @@ def parse_args(argv=None):
return args
-def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
- """Print uncompressed size savings summary.
+def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp='none',
+ base_dtb_csize=None, dtbo_csizes=None):
+ """Print uncompressed and compressed size savings summary.
Args:
dtbs: Sequence of input `ParsedDtb` instances.
base_dtb_blob: Uncompressed base DTB `bytes`.
dtbo_blobs: Sequence of uncompressed `.dtbo` overlay `bytes`.
+ base_dtb_comp: Compression format of the base DTB output.
+ base_dtb_csize: Byte size of the compressed base DTB output.
+ dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
"""
orig_uncomp = sum(len(d.raw_bytes) for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -776,6 +1007,25 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs):
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
)
+ all_comps = {base_dtb_comp, *(d.compression for d in dtbs)}
+ if any(c != 'none' for c in all_comps):
+ comp_label = '/'.join(sorted(all_comps))
+ hdr = (
+ f'Compressed ({comp_label}):'
+ if len(all_comps) == 1
+ else f'On-disk ({comp_label}): '
+ )
+ orig_comp = sum(d.compressed_size for d in dtbs)
+ dtbo_csize = sum(dtbo_csizes)
+ new_comp = base_dtb_csize + dtbo_csize
+ saved_comp = orig_comp - new_comp
+ pct_comp = (saved_comp / orig_comp * 100.0) if orig_comp else 0.0
+ print(
+ f'{hdr} {len(dtbs)} files ({orig_comp:,} B) -> '
+ f'base {base_dtb_csize:,} B + {len(dtbo_blobs)} overlays '
+ f'{dtbo_csize:,} B (avg {dtbo_csize // len(dtbo_blobs):,} B) = '
+ f'{new_comp:,} B (saved {saved_comp:,} B / {pct_comp:.1f}%)'
+ )
def factor_dtb_files(args):
@@ -792,6 +1042,13 @@ def factor_dtb_files(args):
if args.outdir is not None
else args.output_base_dtb
)
+ if args.base_compression == 'auto':
+ _, _, base_dtb_comp = split_dtb_filename(base_dtb_path)
+ if (base_dtb_comp == 'lz4' and
+ all(d.compression == 'lz4_legacy' for d in dtbs)):
+ base_dtb_comp = 'lz4_legacy'
+ else:
+ base_dtb_comp = args.base_compression
input_abs_paths = {os.path.abspath(p) for p in input_dtb_paths}
base_abs_path = os.path.abspath(base_dtb_path)
@@ -803,8 +1060,10 @@ def factor_dtb_files(args):
dtbo_paths = []
seen_paths = {base_abs_path}
- for dtb_path in input_dtb_paths:
- out_path = get_dtbo_output_path(dtb_path, args.outdir)
+ for dtb_path, d in zip(input_dtb_paths, dtbs):
+ out_path = get_dtbo_output_path(
+ dtb_path, args.outdir, comp=d.compression
+ )
abs_out = os.path.abspath(out_path)
if abs_out in input_abs_paths:
raise ValueError(
@@ -823,20 +1082,33 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
+ base_dtb_file_bytes = compress_blob(
+ base_dtb_blob, base_dtb_comp, filename=base_dtb_path
+ )
if args.outdir:
os.makedirs(args.outdir, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
- f.write(base_dtb_blob)
+ f.write(base_dtb_file_bytes)
- for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ dtbo_file_sizes = []
+ for out_path, dtbo_blob, d in zip(dtbo_paths, dtbo_blobs, dtbs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
+ out_bytes = compress_blob(
+ dtbo_blob, d.compression, filename=out_path
+ )
+ dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
- f.write(dtbo_blob)
+ f.write(out_bytes)
if args.stats:
- _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs)
+ _print_dtb_stats(
+ dtbs, base_dtb_blob, dtbo_blobs,
+ base_dtb_comp=base_dtb_comp,
+ base_dtb_csize=len(base_dtb_file_bytes),
+ dtbo_csizes=dtbo_file_sizes,
+ )
def main(argv=None):
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index ea1d35cd8e4e..300f244f41c5 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,10 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 tmp.* *.bak \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
+ *.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
+ tmp.* *.bak \
treegen
TESTS_CLEANFILES = $(TESTS) $(TESTS_CLEANFILES_L:%=$(TESTS_PREFIX)%)
TESTS_CLEANDIRS_L = fs *.test.dir
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index e5ac38dd2bab..c18d824c7bc7 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1261,6 +1261,84 @@ fdtfactor_tests () {
run_test dtbs_equal_unordered fdtfactor_board${i}.nolf.test.dtb \
fdtfactor_strip.test.dir/merged${i}.test.dtb
done
+
+ # Compressed (.dtb.gz / .dtb.xz / .dtb.lzma / .dtb.lz4) modes
+ for comp_spec in "gz:gzip -9 -k -f:gzip -d -f" \
+ "xz:xz -k -f:xz -d -f" \
+ "lzma:xz --format=lzma -k -f:xz -d -f" \
+ "lz4:lz4 -12 -q -f -m:lz4 -d -q -f --rm" \
+ "lz4:lz4 -l -12 -q -f -m:lz4 -d -q -f --rm"; do
+ ext="${comp_spec%%:*}"
+ rest="${comp_spec#*:}"
+ comp_cmd="${rest%%:*}"
+ decomp_cmd="${rest#*:}"
+ if ! command -v "${comp_cmd%% *}" >/dev/null 2>&1; then
+ continue
+ fi
+ run_wrap_test $comp_cmd \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ rm -rf fdtfactor_comp.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s \
+ --output-base-dtb "base.test.dtb.$ext" \
+ -o fdtfactor_comp.test.dir \
+ "fdtfactor_board1.test.dtb.$ext" \
+ "fdtfactor_board2.test.dtb.$ext" \
+ "fdtfactor_board3.test.dtb.$ext"
+ run_wrap_test $decomp_cmd "fdtfactor_comp.test.dir/base.test.dtb.$ext"
+ run_wrap_test $decomp_cmd \
+ "fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo.$ext"
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_comp.test.dir/base.test.dtb \
+ -o fdtfactor_comp.test.dir/merged1.test.dtb \
+ fdtfactor_comp.test.dir/fdtfactor_board1.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.test.dtb \
+ fdtfactor_comp.test.dir/merged1.test.dtb
+ done
+
+ # Test magic-byte detection on a compressed blob with a plain .dtb
+ # extension and an uncompressed --output-base-dtb alongside compressed
+ # inputs.
+ if command -v gzip >/dev/null 2>&1; then
+ run_wrap_test gzip -9 -k -f \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ cp fdtfactor_board1.test.dtb.gz fdtfactor_board1_magic.test.dtb
+ rm -rf fdtfactor_comp_mixed.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR -s \
+ --output-base-dtb base_uncomp.test.dtb \
+ -o fdtfactor_comp_mixed.test.dir \
+ fdtfactor_board1_magic.test.dtb \
+ fdtfactor_board2.test.dtb.gz \
+ fdtfactor_board3.test.dtb.gz
+ run_wrap_test gzip -d -f \
+ fdtfactor_comp_mixed.test.dir/fdtfactor_board1_magic.test.dtbo.gz
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_comp_mixed.test.dir/base_uncomp.test.dtb \
+ -o fdtfactor_comp_mixed.test.dir/merged1.test.dtb \
+ fdtfactor_comp_mixed.test.dir/fdtfactor_board1_magic.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.test.dtb \
+ fdtfactor_comp_mixed.test.dir/merged1.test.dtb
+
+ # Explicit --base-compression overriding the --output-base-dtb filename
+ # extension
+ rm -rf fdtfactor_base_comp.test.dir
+ run_wrap_test $PYTHON $FDTFACTOR --base-compression=gzip \
+ --output-base-dtb base_custom.test.dtb \
+ -o fdtfactor_base_comp.test.dir \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb
+ mv fdtfactor_base_comp.test.dir/base_custom.test.dtb \
+ fdtfactor_base_comp.test.dir/base_custom.test.dtb.gz
+ run_wrap_test gzip -d -f \
+ fdtfactor_base_comp.test.dir/base_custom.test.dtb.gz
+ run_wrap_test $FDTOVERLAY \
+ -i fdtfactor_base_comp.test.dir/base_custom.test.dtb \
+ -o fdtfactor_base_comp.test.dir/merged1.test.dtb \
+ fdtfactor_base_comp.test.dir/fdtfactor_board1.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.test.dtb \
+ fdtfactor_base_comp.test.dir/merged1.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1290,7 +1368,9 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+ *.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
+ *.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
for set in $TESTSETS; do
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* [PATCH v3 15/15] fdtfactor: Add compiled FIT factoring support
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (13 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 14/15] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
@ 2026-11-01 22:01 ` Douglas Anderson
2026-11-02 4:44 ` [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays David Gibson
15 siblings, 0 replies; 23+ messages in thread
From: Douglas Anderson @ 2026-11-01 22:01 UTC (permalink / raw)
To: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio
Cc: Peter Griffin, Saravana Kannan, Devicetree, Simon Glass,
Chen-Yu Tsai, Julius Werner, Brian Norris, linux-kernel,
Tudor Ambarus, André Draszik, David Gibson,
Douglas Anderson
Bootloaders such as depthcharge on Chromebooks and U-Boot use Flattened
Image Trees (FITs) to bundle a kernel image with multiple board FDT
blobs (`fdt-1`, `fdt-2`, ...) and per-board configuration nodes.
Add `-F`/`--fit`, `--outfile`, and `--single-base-dtb` options to
`fdtfactor` to factor a compiled FIT directly:
- Parse non-overlay `type = "flat_dt"` nodes under `/images`,
decompressing their `data` blobs according to their `compression`
property (`none`, `lz4`, `lzma`, `xz`, `gzip`).
- Reject FITs containing `signature*` subnodes under `/images/*` or
`/configurations/*`, as modifying FDT blobs and configurations would
invalidate cryptographic signatures.
- Partition the `flat_dt` blobs into compatible factor groups (by
compression, architecture, FDT header fields, and node-path overlap)
so a single FIT containing multiple board/SoC families produces a
separate `fdt-base-1`, `fdt-base-2`, etc. for each family.
- Support `--single-base-dtb` to force all `flat_dt` blobs into a single
base DTB regardless of node-path overlap, failing only if their FDT
headers, architecture, or compression differ.
- Insert each `fdt-base-N` node into `/images`, replace each factored
`fdt-*` node's `data` with its compressed `.dtbo` overlay, recompute
any `hash*` subnode digests, and prepend `fdt-base-N` to each matching
configuration's `fdt` string list while preserving all other
configuration properties (`description`, `compatible`, etc.,
populating `compatible` from the board DTB's root node when omitted)
and any pre-existing overlay entries.
Add uncompressed, LZ4-compressed multi-family, signed-FIT rejection, and
`--single-base-dtb` FIT test cases to the `fdtfactor` test suite.
Reviewed-by: Simon Glass <sjg@chromium.org>
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Changes in v3:
- Use "FIT" instead of "FIT image" in documentation, comments, and
commit messages (Simon Glass)
- Update for DeviceTree and --output-base-dtb (Simon Glass, David
Gibson)
Changes in v2:
- Rename -f/--fit and -F/--fit-out to -F/--fit and --outfile and clarify
FIT mode in the --help description (David Gibson, Simon Glass)
- Reject signed FIT images with signature* subnodes (Simon Glass)
- Use '<base-node-name>.dtb' for FIT base FDT descriptions instead of
common-prefix heuristics (Simon Glass)
- Compare FIT group node overlap against the group's common node set
instead of only the first entry (Simon Glass)
- Document in README.md that U-Boot's boot_get_fdt_fit() currently
requires FIT overlays to be uncompressed (Simon Glass)
README.md | 17 ++
fdtfactor | 577 +++++++++++++++++++++++++++++++++++--
tests/Makefile.tests | 2 +-
tests/fdtfactor_lz4.its | 109 +++++++
tests/fdtfactor_other1.dts | 33 +++
tests/fdtfactor_other2.dts | 33 +++
tests/fdtfactor_test.its | 109 +++++++
tests/run_tests.sh | 158 +++++++++-
8 files changed, 1011 insertions(+), 27 deletions(-)
create mode 100644 tests/fdtfactor_lz4.its
create mode 100644 tests/fdtfactor_other1.dts
create mode 100644 tests/fdtfactor_other2.dts
create mode 100644 tests/fdtfactor_test.its
diff --git a/README.md b/README.md
index a3d1ad8c58f3..7e2b99818730 100644
--- a/README.md
+++ b/README.md
@@ -129,6 +129,7 @@ Basic usage:
```
$ fdtfactor -o out_dir board1.dtb board2.dtb board3.dtb
+$ fdtfactor -F image.fit --outfile image.factored.fit
```
Options:
@@ -144,6 +145,22 @@ Options:
* `-o`, `--outdir`: Output directory for generated `.dtbo` files and
`--output-base-dtb` (default: alongside each input `.dtb` for `.dtbo` files,
current directory for `--output-base-dtb`).
+* `-F`, `--fit`: Factor `/images/fdt-*` blobs inside a compiled FIT
+ (creating `/images/fdt-base-1`, `/images/fdt-base-2`, ... per compatible
+ board/SoC family grouped by FDT headers and `>= 50%` node-path overlap,
+ converting each `/images/fdt-N` node into a `.dtbo` overlay, updating
+ `/configurations/conf-*` `fdt` and `compatible` properties, and
+ recalculating `/images/*/hash-*` digests). Each factored base DTB and
+ `.dtbo` overlay preserves the `compression` setting of its `/images/fdt-*`
+ node (note: while bootloaders such as depthcharge decompress compressed
+ FIT overlays, U-Boot's `boot_get_fdt_fit()` currently only decompresses
+ the base DTB and requires FIT overlays to be uncompressed). Signed FITs
+ with `signature*` subnodes are rejected because modifying FDT blobs
+ and configurations would invalidate cryptographic signatures.
+* `--outfile`: Output FIT path when `--fit` is used (default:
+ overwrite `--fit` in place).
+* `--single-base-dtb`: Force a single `/images/fdt-base-1` across all FDTs
+ in `--fit` mode rather than grouping by node-path overlap.
* `-S`, `--strip-local-fixups`: Omit `/__local_fixups__` from the generated
`base.dtb` and final reconstructed device trees. By default, `fdtfactor`
preserves `/__local_fixups__` on `base.dtb` so that when `fdt_overlay_apply()`
diff --git a/fdtfactor b/fdtfactor
index 9bc8bfde2038..379a27da139d 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -8,16 +8,25 @@ properties shared across the inputs into a single common `base.dtb` and emits a
small per-board `.dtbo` overlay for each input. Applying a board's `.dtbo` to
`base.dtb` (e.g., via `fdtoverlay` or `fdt_overlay_apply()`) reconstructs the
original board DTB.
+
+Alternatively, `fdtfactor -F <image.fit>` can rewrite a compiled U-Boot /
+ChromeOS Flattened Image Tree (FIT) in place (or to `--outfile`), factoring its
+embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`, ...)
+and overlays while updating `/configurations/conf-*` `fdt` and `compatible`
+properties and recalculating `/images/*/hash-*` digests.
"""
import argparse
from collections import Counter
+import copy
import gzip
+import hashlib
import lzma
import os
import struct
import subprocess
import sys
+import zlib
import devicetree
import libfdt
@@ -247,6 +256,10 @@ def _is_dtbo_overlay(fdt):
)
+class OverlayBlobError(ValueError):
+ """Raised when an input FDT blob is already a `.dtbo` overlay."""
+
+
class FactoredTree(devicetree.DeviceTree):
"""DeviceTree subclass with normalized phandle and fixup attributes.
@@ -283,17 +296,24 @@ class ParsedDtb:
independent of numeric phandle numbering.
"""
- def __init__(self, filename):
- """Parse an input FDT file into a normalized FactoredTree.
+ def __init__(self, filename, file_bytes=None, comp=None):
+ """Parse an input FDT file or blob into a normalized FactoredTree.
Args:
- filename: File path of the `.dtb` (or compressed `.dtb.*`) file.
+ filename: File path (or display name when `file_bytes` is given).
+ file_bytes: Optional raw (compressed or uncompressed) FDT bytes; if
+ None, read from `filename`.
+ comp: Optional explicit compression algorithm name, or None to
+ auto-detect via `decompress_blob()`.
"""
self.filename = filename
- with open(filename, 'rb') as f:
- file_bytes = f.read()
+ if file_bytes is None:
+ with open(filename, 'rb') as f:
+ file_bytes = f.read()
self.compressed_size = len(file_bytes)
- raw_bytes, self.compression = decompress_blob(file_bytes, filename)
+ raw_bytes, self.compression = decompress_blob(
+ file_bytes, filename, comp=comp
+ )
self.raw_bytes = raw_bytes
try:
@@ -303,7 +323,7 @@ class ParsedDtb:
raise ValueError(f'{filename}: Invalid FDT blob ({exc})') from exc
if _is_dtbo_overlay(fdt):
- raise ValueError(
+ raise OverlayBlobError(
f'{filename}: Input is already a DTB overlay (.dtbo)'
)
@@ -885,6 +905,460 @@ def create_base_dtb_and_overlays(dtbs, strip_local_fixups=False):
return base_dtb_blob, dtbo_blobs
+# ---------------------------------------------------------------------------
+# Compiled FIT factoring (targeting depthcharge / coreboot / U-Boot)
+# ---------------------------------------------------------------------------
+
+def _compute_fit_hash(algo_name, data_bytes):
+ """Compute a binary hash digest for a FIT `hash*` subnode.
+
+ Args:
+ algo_name: Hash algorithm name (e.g., `'crc32'`, `'sha1'`, `'sha256'`).
+ data_bytes: Input `bytes` to hash.
+
+ Returns:
+ Binary hash digest `bytes`.
+ """
+ if algo_name == 'crc32':
+ return struct.pack('>I', zlib.crc32(data_bytes) & 0xffffffff)
+ h = hashlib.new(algo_name)
+ h.update(data_bytes)
+ return h.digest()
+
+
+def _update_fit_hashes(img_node):
+ """Recompute `value` properties on any FIT `hash*` subnodes of `img_node`.
+
+ Args:
+ img_node: `DeviceTree` for a `/images/*` subnode whose `hash*` child
+ nodes should be updated in place from `img_node.props['data']`.
+ """
+ data_prop = img_node.props.get('data')
+ if data_prop is None:
+ return
+ data_bytes = bytes(data_prop)
+ for sub_name, sub_node in img_node.children.items():
+ if not sub_name.startswith('hash'):
+ continue
+ if 'algo' not in sub_node.props:
+ raise ValueError(
+ f'FIT hash subnode {sub_name!r} lacks an algo property'
+ )
+ algo = sub_node.get_str('algo')
+ sub_node.props['value'] = libfdt.Property(
+ 'value', _compute_fit_hash(algo, data_bytes)
+ )
+
+
+def _is_dtb_filename_desc(desc):
+ """Return True if `desc` looks like a `.dtb` filename.
+
+ Args:
+ desc: Description string from a FIT `/images/fdt-*` node.
+
+ Returns:
+ True if `desc` has no spaces and ends with `.dtb` (optionally followed
+ by a recognized compression suffix), False otherwise.
+ """
+ if not desc or ' ' in desc:
+ return False
+ _, stem, desc_comp = split_dtb_filename(desc)
+ return os.path.basename(desc) == f'{stem}.dtb{COMP_TO_EXT[desc_comp]}'
+
+
+def group_fit_dtbs(fdt_entries):
+ """Partition FIT `flat_dt` entries into compatible factor groups.
+
+ DTBs are grouped by `(compression, arch, load, entry, boot_cpuid_phys,
+ mem_rsv)` and node-path overlap (>= 50%) against the group's common node
+ set so that a single FIT containing multiple board/SoC families
+ produces one base DTB per family.
+
+ Args:
+ fdt_entries: Sequence of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+
+ Returns:
+ List of entry-dict lists, one per compatible group.
+ """
+ groups = []
+ for entry in fdt_entries:
+ dtb = entry['dtb']
+ header_key = (
+ dtb.compression,
+ *(entry['node'].props.get(p) for p in ('arch', 'load', 'entry')),
+ dtb.boot_cpuid_phys,
+ dtb.mem_rsv,
+ )
+ entry_nodes = {path for path, _ in dtb.root.walk()}
+
+ placed = False
+ for grp in groups:
+ if grp['key'] != header_key:
+ continue
+ common_nodes = grp['common_nodes']
+ union_len = len(entry_nodes | common_nodes)
+ overlap = (
+ len(entry_nodes & common_nodes) / union_len
+ if union_len
+ else 1.0
+ )
+ if overlap >= 0.5:
+ grp['common_nodes'] &= entry_nodes
+ grp['entries'].append(entry)
+ placed = True
+ break
+
+ if not placed:
+ groups.append({
+ 'key': header_key,
+ 'common_nodes': set(entry_nodes),
+ 'entries': [entry],
+ })
+
+ return [grp['entries'] for grp in groups]
+
+
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+ """Extract and parse non-overlay `flat_dt` entries from `/images`.
+
+ Args:
+ fit_in_path: Input FIT path (for error reporting).
+ images_node: `DeviceTree` for the FIT `/images` node.
+ configs_node: `DeviceTree` for the FIT `/configurations` node.
+
+ Returns:
+ List of dicts (`{'name', 'node', 'dtb'}`) for all non-overlay
+ `flat_dt` image subnodes.
+ """
+ if any(name.startswith('fdt-base-') for name in images_node.children):
+ raise ValueError(
+ f'{fit_in_path}: FIT already contains fdt-base-* node(s)'
+ )
+
+ for conf_name, conf_node in configs_node.children.items():
+ for sub_name in conf_node.children:
+ if sub_name.startswith(('signature', 'hash')):
+ raise ValueError(
+ f'{fit_in_path}: /configurations/{conf_name}/{sub_name} '
+ f'is not supported; sign the FIT after factoring'
+ )
+
+ fdt_entries = []
+ for img_name, img_node in images_node.children.items():
+ if ('data-offset' in img_node.props or
+ 'data-position' in img_node.props):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} uses external data '
+ f'(data-offset/data-position), which is not supported'
+ )
+ for sub_name in img_node.children:
+ if sub_name.startswith('signature'):
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name}/{sub_name} is not '
+ f'supported; sign the FIT after factoring'
+ )
+ img_type = img_node.get_str('type') if 'type' in img_node.props else ''
+ if img_type != 'flat_dt':
+ continue
+ raw_data = img_node.props.get('data')
+ if raw_data is None:
+ raise ValueError(
+ f'{fit_in_path}: /images/{img_name} lacks a data property'
+ )
+ node_label = f'{fit_in_path}:/images/{img_name}'
+ comp_prop = (
+ img_node.get_str('compression')
+ if 'compression' in img_node.props
+ else 'none'
+ ) or 'none'
+ try:
+ parsed_dtb = ParsedDtb(
+ node_label, file_bytes=bytes(raw_data), comp=comp_prop
+ )
+ except OverlayBlobError:
+ continue
+
+ fdt_entries.append({
+ 'name': img_name,
+ 'node': img_node,
+ 'dtb': parsed_dtb,
+ })
+
+ if len(fdt_entries) < 2:
+ raise ValueError(
+ 'FIT must contain at least two non-overlay flat_dt '
+ 'nodes to factor'
+ )
+
+ return fdt_entries
+
+
+def _partition_fit_fdt_entries(fdt_entries, single_base_dtb):
+ """Partition `fdt_entries` into factorable groups and validate constraints.
+
+ Args:
+ fdt_entries: List of FIT `flat_dt` entry dicts from
+ `_extract_fit_fdt_entries()`.
+ single_base_dtb: If True, require all `fdt_entries` to share a single
+ base DTB group rather than grouping by node-path overlap.
+
+ Returns:
+ List of factorable entry-dict groups (each containing at least 2
+ entries).
+ """
+ if single_base_dtb:
+ ref_entry = fdt_entries[0]
+ for entry in fdt_entries[1:]:
+ if entry['dtb'].compression != ref_entry['dtb'].compression:
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to use '
+ f'the same compression: {ref_entry["name"]} has '
+ f'{ref_entry["dtb"].compression!r} vs {entry["name"]} '
+ f'{entry["dtb"].compression!r}'
+ )
+ for prop in ('arch', 'load', 'entry'):
+ if (entry['node'].props.get(prop) !=
+ ref_entry['node'].props.get(prop)):
+ raise ValueError(
+ f'--single-base-dtb requires all FIT flat_dt nodes to '
+ f'use the same {prop} property: {ref_entry["name"]} vs '
+ f'{entry["name"]}'
+ )
+ return [fdt_entries]
+
+ groups = group_fit_dtbs(fdt_entries)
+ factorable_groups = [g for g in groups if len(g) >= 2]
+ if not factorable_groups:
+ raise ValueError(
+ 'FIT must contain at least two compatible '
+ 'flat_dt nodes to factor'
+ )
+ for g in groups:
+ if len(g) == 1:
+ print(
+ f'Note: FIT flat_dt node {g[0]["name"]} has no '
+ f'compatible peers and will be left unfactored',
+ file=sys.stderr,
+ )
+ return factorable_groups
+
+
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+ """Factor one compatible group of FIT `flat_dt` entries.
+
+ Args:
+ entries: List of FIT `flat_dt` entry dicts in this compatible group.
+ base_dtb_node_name: Name for the new `/images/<base_dtb_node_name>`
+ node (e.g., `'fdt-base-1'`).
+ strip_local_fixups: If True, omit `/__local_fixups__` from the
+ generated base DTB.
+
+ Returns:
+ Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
+ constructed `DeviceTree` for `/images/<base_dtb_node_name>` and
+ `stat_args` is an argument tuple for `_print_dtb_stats()`.
+ """
+ group_dtbs = [e['dtb'] for e in entries]
+ comp = group_dtbs[0].compression
+
+ base_dtb_blob, dtbo_blobs = create_base_dtb_and_overlays(
+ group_dtbs,
+ strip_local_fixups=strip_local_fixups,
+ )
+
+ base_dtb_comp_bytes = compress_blob(
+ base_dtb_blob, comp, filename=f'/images/{base_dtb_node_name}'
+ )
+ base_dtb_node = copy.deepcopy(entries[0]['node'])
+ for prop in ('phandle', 'compatible'):
+ base_dtb_node.props.pop(prop, None)
+ base_dtb_node.children = {
+ k: v
+ for k, v in base_dtb_node.children.items()
+ if k.startswith('hash')
+ }
+ for sub_node in base_dtb_node.children.values():
+ sub_node.props.pop('phandle', None)
+ if 'description' in base_dtb_node.props:
+ base_dtb_node.set_str(
+ 'description',
+ f'{base_dtb_node_name}.dtb{COMP_TO_EXT[comp]}',
+ )
+ base_dtb_node.props['data'] = libfdt.Property('data', base_dtb_comp_bytes)
+ if 'compression' in base_dtb_node.props or comp != 'none':
+ comp_prop = 'lz4' if comp == 'lz4_legacy' else comp
+ base_dtb_node.set_str('compression', comp_prop)
+ _update_fit_hashes(base_dtb_node)
+
+ dtbo_comp_sizes = []
+ for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ img_node = entry['node']
+ dtbo_comp_bytes = compress_blob(
+ dtbo_blob, comp, filename=f'/images/{entry["name"]}'
+ )
+ dtbo_comp_sizes.append(len(dtbo_comp_bytes))
+
+ if 'description' in img_node.props:
+ old_desc = img_node.get_str('description')
+ if _is_dtb_filename_desc(old_desc):
+ img_node.set_str(
+ 'description',
+ get_dtbo_output_path(old_desc, None, comp=comp),
+ )
+
+ # Strip 'load' and 'entry' from overlay image nodes so bootloaders do
+ # not load overlays at the base FDT address and overwrite the base FDT.
+ for prop in ('load', 'entry'):
+ img_node.props.pop(prop, None)
+
+ img_node.props['data'] = libfdt.Property('data', dtbo_comp_bytes)
+ _update_fit_hashes(img_node)
+
+ stat_args = (
+ group_dtbs,
+ base_dtb_blob,
+ dtbo_blobs,
+ comp,
+ len(base_dtb_comp_bytes),
+ dtbo_comp_sizes,
+ f'[{base_dtb_node_name}] ',
+ )
+ return base_dtb_node, stat_args
+
+
+def _update_fit_images_and_configs(images_node, configs_node,
+ insert_before_first, fdt_to_base_dtb):
+ """Insert `fdt-base-*` nodes into `/images` and update `/configurations`.
+
+ Args:
+ images_node: `DeviceTree` for `/images` (mutated in place).
+ configs_node: `DeviceTree` for `/configurations` (mutated in place).
+ insert_before_first: Dict mapping the first `fdt-*` subnode name of
+ each group to `(base_dtb_node_name, base_dtb_node)`.
+ fdt_to_base_dtb: Dict mapping each factored `fdt-*` subnode name to
+ `(base_dtb_node_name, orig_parsed_dtb)`.
+ """
+ new_images_children = {}
+ for img_name, img_node in images_node.children.items():
+ if img_name in insert_before_first:
+ b_name, b_node = insert_before_first[img_name]
+ new_images_children[b_name] = b_node
+ new_images_children[img_name] = img_node
+ images_node.children = new_images_children
+
+ for conf_name, conf_node in configs_node.children.items():
+ if 'fdt' not in conf_node.props:
+ continue
+ fdt_list = conf_node.get_stringlist('fdt')
+ if not fdt_list:
+ continue
+ primary_fdt = fdt_list[0]
+ for extra_fdt in fdt_list[1:]:
+ if extra_fdt in fdt_to_base_dtb:
+ raise ValueError(
+ f'/configurations/{conf_name} lists factored flat_dt '
+ f'{extra_fdt!r} after the primary FDT'
+ )
+ if primary_fdt in fdt_to_base_dtb:
+ base_dtb_name, orig_dtb = fdt_to_base_dtb[primary_fdt]
+ conf_node.set_stringlist('fdt', [base_dtb_name] + fdt_list)
+ # If the configuration omitted 'compatible' (relying on the
+ # bootloader reading root 'compatible' from fdt_list[0]), copy the
+ # board's root 'compatible' property onto the configuration node
+ # now that fdt_list[0] is the shared base DTB.
+ if 'compatible' not in conf_node.props:
+ root_compat = orig_dtb.root.props.get('compatible')
+ if root_compat is not None:
+ conf_node.props['compatible'] = copy.copy(root_compat)
+
+
+def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
+ single_base_dtb=False, stats=False):
+ """Factor all `flat_dt` images in a compiled FIT into base + overlays.
+
+ Args:
+ fit_in_path: Path to the input compiled FIT.
+ fit_out_path: Path where the factored FIT should be written.
+ strip_local_fixups: If True, omit `/__local_fixups__` from generated
+ base DTBs.
+ single_base_dtb: If True, force all `flat_dt` nodes into a single
+ `fdt-base-1` base DTB.
+ stats: If True, print per-group and overall FIT size savings.
+ """
+ with open(fit_in_path, 'rb') as f:
+ orig_fit_bytes = f.read()
+
+ try:
+ orig_fit_fdt = libfdt.FdtRo(orig_fit_bytes)
+ libfdt.check_err(
+ libfdt.fdt_check_full(orig_fit_fdt._fdt, len(orig_fit_bytes))
+ )
+ except libfdt.FdtException as exc:
+ raise ValueError(f'{fit_in_path}: Invalid FDT blob ({exc})') from exc
+
+ fit_root = devicetree.DeviceTree.unflatten(orig_fit_fdt)
+ images_node = fit_root.children.get('images')
+ configs_node = fit_root.children.get('configurations')
+ if images_node is None or configs_node is None:
+ raise ValueError(
+ f'{fit_in_path}: FIT must contain /images and '
+ f'/configurations nodes'
+ )
+
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node
+ )
+ factorable_groups = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb
+ )
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+
+ for idx, entries in enumerate(factorable_groups, 1):
+ base_dtb_node_name = f'fdt-base-{idx}'
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
+ group_stats.append(stat_args)
+
+ _update_fit_images_and_configs(
+ images_node, configs_node, insert_before_first, fdt_to_base_dtb
+ )
+
+ new_fit_fdt = fit_root.flatten(
+ mem_rsv=orig_fit_fdt.get_mem_rsv_list(),
+ boot_cpuid_phys=orig_fit_fdt.boot_cpuid_phys(),
+ )
+ new_fit_bytes = bytes(new_fit_fdt.as_bytearray())
+
+ out_dir = os.path.dirname(fit_out_path)
+ if out_dir:
+ os.makedirs(out_dir, exist_ok=True)
+ with open(fit_out_path, 'wb') as f:
+ f.write(new_fit_bytes)
+
+ if stats:
+ for stat_args in group_stats:
+ _print_dtb_stats(*stat_args)
+ fit_saved = len(orig_fit_bytes) - len(new_fit_bytes)
+ fit_pct = (
+ (fit_saved / len(orig_fit_bytes) * 100.0) if orig_fit_bytes else 0.0
+ )
+ print(
+ f'FIT total: {len(orig_fit_bytes):,} B -> '
+ f'{len(new_fit_bytes):,} B '
+ f'(saved {fit_saved:,} B / {fit_pct:.1f}%)'
+ )
+
+
def get_dtbo_output_path(dtb_path, outdir, comp='none'):
"""Determine the output `.dtbo[.comp]` file path for an input `.dtb` path.
@@ -917,13 +1391,15 @@ def parse_args(argv=None):
parser = argparse.ArgumentParser(
description=(
'Split a set of compiled device tree blobs (.dtb, or compressed '
- '.dtb.lz4/.xz/.lzma/.gz) into a single common base .dtb and '
- 'per-board overlay blobs (.dtbo).'
+ '.dtb.lz4/.xz/.lzma/.gz) into a common base .dtb and per-board '
+ 'overlay blobs (.dtbo), or update a compiled FIT '
+ '(-F/--fit) in-place (or to --outfile) by factoring its '
+ '/images/fdt-* blobs into base DTBs and overlays.'
)
)
parser.add_argument(
'--output-base-dtb',
- default='base.dtb',
+ default=None,
metavar='FILENAME',
help=(
'Output common base .dtb (or .dtb.lz4/.xz/.lzma/.gz) filename '
@@ -950,6 +1426,23 @@ def parse_args(argv=None):
'current directory for --output-base-dtb)'
),
)
+ parser.add_argument(
+ '-F', '--fit',
+ default=None,
+ metavar='FIT_FILE',
+ help=(
+ 'Input compiled FIT (.fit / .itb) to factor (updated '
+ 'in-place unless --outfile is specified)'
+ ),
+ )
+ parser.add_argument(
+ '--outfile',
+ default=None,
+ metavar='FIT_OUT',
+ help=(
+ 'Output path for the factored FIT (default: overwrite --fit)'
+ ),
+ )
parser.add_argument(
'-S', '--strip-local-fixups',
action='store_true',
@@ -958,6 +1451,15 @@ def parse_args(argv=None):
'reconstructed device trees'
),
)
+ parser.add_argument(
+ '--single-base-dtb',
+ action='store_true',
+ help=(
+ 'Force all flat_dt nodes in a FIT (-F/--fit) to be '
+ 'factored into a single fdt-base-1 base DTB, failing only if '
+ 'headers, arch, or compression differ'
+ ),
+ )
parser.add_argument(
'-s', '--stats',
action='store_true',
@@ -965,26 +1467,40 @@ def parse_args(argv=None):
)
parser.add_argument(
'dtbs',
- nargs='+',
+ nargs='*',
metavar='DTB',
help=(
- 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files '
- '(at least two required)'
+ 'Input .dtb (or compressed .dtb.lz4/.xz/.lzma/.gz) files (when '
+ '--fit is not used)'
),
)
args = parser.parse_args(argv)
- if os.path.dirname(args.output_base_dtb):
- parser.error(
- '--output-base-dtb must be a filename without directory '
- 'components; use -o/--outdir to specify the output directory'
- )
- if len(args.dtbs) < 2:
- parser.error('at least two input .dtb files are required')
+ if args.outfile and not args.fit:
+ parser.error('--outfile requires -F/--fit')
+ if args.single_base_dtb and not args.fit:
+ parser.error('--single-base-dtb requires -F/--fit')
+ if args.fit:
+ if (args.dtbs or args.output_base_dtb or args.outdir or
+ args.base_compression != 'auto'):
+ parser.error(
+ '-F/--fit cannot be combined with --output-base-dtb, '
+ '--base-compression, -o/--outdir, or positional DTB arguments'
+ )
+ else:
+ if args.output_base_dtb is None:
+ args.output_base_dtb = 'base.dtb'
+ if os.path.dirname(args.output_base_dtb):
+ parser.error(
+ '--output-base-dtb must be a filename without directory '
+ 'components; use -o/--outdir to specify the output directory'
+ )
+ if len(args.dtbs) < 2:
+ parser.error('at least two input .dtb files are required')
return args
def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp='none',
- base_dtb_csize=None, dtbo_csizes=None):
+ base_dtb_csize=None, dtbo_csizes=None, prefix=''):
"""Print uncompressed and compressed size savings summary.
Args:
@@ -994,6 +1510,7 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp='none',
base_dtb_comp: Compression format of the base DTB output.
base_dtb_csize: Byte size of the compressed base DTB output.
dtbo_csizes: Sequence of byte sizes of the compressed `.dtbo` outputs.
+ prefix: Optional line prefix (e.g., `'[fdt-base-1] '`).
"""
orig_uncomp = sum(len(d.raw_bytes) for d in dtbs)
base_dtb_uncomp = len(base_dtb_blob)
@@ -1002,7 +1519,7 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp='none',
saved_uncomp = orig_uncomp - new_uncomp
pct_uncomp = (saved_uncomp / orig_uncomp * 100.0) if orig_uncomp else 0.0
print(
- f'Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
+ f'{prefix}Uncompressed DTBs: {len(dtbs)} files ({orig_uncomp:,} B) -> '
f'base {base_dtb_uncomp:,} B + {len(dtbo_blobs)} overlays '
f'{dtbo_uncomp:,} B = {new_uncomp:,} B '
f'(saved {saved_uncomp:,} B / {pct_uncomp:.1f}%)'
@@ -1011,9 +1528,9 @@ def _print_dtb_stats(dtbs, base_dtb_blob, dtbo_blobs, base_dtb_comp='none',
if any(c != 'none' for c in all_comps):
comp_label = '/'.join(sorted(all_comps))
hdr = (
- f'Compressed ({comp_label}):'
+ f'{prefix}Compressed ({comp_label}):'
if len(all_comps) == 1
- else f'On-disk ({comp_label}): '
+ else f'{prefix}On-disk ({comp_label}): '
)
orig_comp = sum(d.compressed_size for d in dtbs)
dtbo_csize = sum(dtbo_csizes)
@@ -1122,7 +1639,17 @@ def main(argv=None):
"""
args = parse_args(argv)
try:
- factor_dtb_files(args)
+ if args.fit:
+ fit_out = args.outfile if args.outfile else args.fit
+ factor_fit(
+ args.fit,
+ fit_out,
+ strip_local_fixups=args.strip_local_fixups,
+ single_base_dtb=args.single_base_dtb,
+ stats=args.stats,
+ )
+ else:
+ factor_dtb_files(args)
except (OSError, ValueError, RuntimeError, libfdt.FdtException) as exc:
sys.exit(f'Error: {exc}')
return 0
diff --git a/tests/Makefile.tests b/tests/Makefile.tests
index 300f244f41c5..a89bcde1f038 100644
--- a/tests/Makefile.tests
+++ b/tests/Makefile.tests
@@ -53,7 +53,7 @@ TESTS_DEPFILES = $(TESTS:%=%.d) \
$(addprefix $(TESTS_PREFIX),testutils.d)
TESTS_CLEANFILES_L = $(STD_CLEANFILES) \
- *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 \
+ *.dtb *.test.dtbo *.test.dts *.test.dt.yaml *.dtsv1 *.test.fit \
*.test.dtb.gz *.test.dtbo.gz *.test.dtb.xz *.test.dtbo.xz \
*.test.dtb.lzma *.test.dtbo.lzma *.test.dtb.lz4 *.test.dtbo.lz4 \
tmp.* *.bak \
diff --git a/tests/fdtfactor_lz4.its b/tests/fdtfactor_lz4.its
new file mode 100644
index 000000000000..3f862b19c759
--- /dev/null
+++ b/tests/fdtfactor_lz4.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test LZ4 FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "lz4";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board2.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb.lz4";
+ data = /incbin/("fdtfactor_board3.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb.lz4";
+ data = /incbin/("fdtfactor_other1.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb.lz4";
+ data = /incbin/("fdtfactor_other2.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb.lz4";
+ data = /incbin/("overlay_overlay_phandle.test.dtb.lz4");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "lz4";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/fdtfactor_other1.dts b/tests/fdtfactor_other1.dts
new file mode 100644
index 000000000000..d6257d1fd29d
--- /dev/null
+++ b/tests/fdtfactor_other1.dts
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other1", "vendor,other";
+ model = "other1";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x1111>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o1_node: other1-node {
+ prop-str = "other1";
+ ref-base = <&osn1>;
+ ref-self = <&o1_node>;
+ };
+};
diff --git a/tests/fdtfactor_other2.dts b/tests/fdtfactor_other2.dts
new file mode 100644
index 000000000000..2d97f3595e3c
--- /dev/null
+++ b/tests/fdtfactor_other2.dts
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/memreserve/ 0xdeadbeef00000000 0x100000;
+
+/ {
+ compatible = "vendor,other2", "vendor,other";
+ model = "other2";
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ osn1: other-subnode@1 {
+ compatible = "other-subnode1";
+ reg = <1>;
+ prop-ref = <&osn1 0x10 &ossn1 0x20>;
+
+ ossn1: other-subsubnode {
+ compatible = "other-subsubnode";
+ prop-int = <0x2222>;
+ };
+ };
+
+ osn2: other-subnode@2 {
+ reg = <2>;
+ prop-phandle = <&osn2>;
+ };
+
+ o2_node: other2-node {
+ prop-str = "other2";
+ ref-base = <&ossn1>;
+ ref-self = <&o2_node>;
+ };
+};
diff --git a/tests/fdtfactor_test.its b/tests/fdtfactor_test.its
new file mode 100644
index 000000000000..dd515e17b66f
--- /dev/null
+++ b/tests/fdtfactor_test.its
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/dts-v1/;
+
+/ {
+ description = "Test FIT for fdtfactor";
+ #address-cells = <1>;
+
+ images {
+ kernel-1 {
+ description = "Unavailable";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "kernel_noload";
+ arch = "arm64";
+ os = "linux";
+ compression = "none";
+ load = <0>;
+ entry = <0>;
+ };
+ fdt-1 {
+ description = "fdtfactor_board1.test.dtb";
+ data = /incbin/("fdtfactor_board1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-2 {
+ description = "fdtfactor_board2.test.dtb";
+ data = /incbin/("fdtfactor_board2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-3 {
+ description = "fdtfactor_board3.test.dtb";
+ data = /incbin/("fdtfactor_board3.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-4 {
+ description = "fdtfactor_other1.test.dtb";
+ data = /incbin/("fdtfactor_other1.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-5 {
+ description = "fdtfactor_other2.test.dtb";
+ data = /incbin/("fdtfactor_other2.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ hash-1 {
+ algo = "sha1";
+ };
+ };
+ fdt-ovl-1 {
+ description = "overlay_overlay_phandle.test.dtb";
+ data = /incbin/("overlay_overlay_phandle.test.dtb");
+ type = "flat_dt";
+ arch = "arm64";
+ compression = "none";
+ };
+ };
+ configurations {
+ default = "conf-1";
+ conf-1 {
+ description = "board-meta-1";
+ kernel = "kernel-1";
+ fdt = "fdt-1";
+ compatible = "vendor,board1", "vendor,common";
+ };
+ conf-2 {
+ description = "board-meta-2";
+ kernel = "kernel-1";
+ fdt = "fdt-2";
+ };
+ conf-3 {
+ description = "board-meta-3";
+ kernel = "kernel-1";
+ fdt = "fdt-3", "fdt-ovl-1";
+ compatible = "vendor,board3", "vendor,common";
+ };
+ conf-4 {
+ description = "board-meta-4";
+ kernel = "kernel-1";
+ fdt = "fdt-4";
+ compatible = "vendor,other1", "vendor,other";
+ };
+ conf-5 {
+ description = "board-meta-5";
+ kernel = "kernel-1";
+ fdt = "fdt-5";
+ compatible = "vendor,other2", "vendor,other";
+ };
+ };
+};
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index c18d824c7bc7..812574203339 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -1339,6 +1339,162 @@ fdtfactor_tests () {
run_test dtbs_equal_unordered fdtfactor_board1.test.dtb \
fdtfactor_base_comp.test.dir/merged1.test.dtb
fi
+
+ # FIT factoring (uncompressed and compressed, multi-group by node
+ # overlap)
+ # fdtget -t r appends a trailing newline; strip that last byte.
+ fdtget_raw () {
+ $DTGET -t r "$1" "$2" "$3" > "$4.tmp"
+ head -c $(($($STATSZ "$4.tmp") - 1)) "$4.tmp" > "$4"
+ rm -f "$4.tmp"
+ }
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other1.test.dtb \
+ "$SRCDIR/fdtfactor_other1.dts"
+ run_dtc_test -b 2 -@ -L -I dts -O dtb -o fdtfactor_other2.test.dtb \
+ "$SRCDIR/fdtfactor_other2.dts"
+ run_dtc_test -@ -I dts -O dtb -o overlay_overlay_phandle.test.dtb \
+ "$SRCDIR/overlay_overlay_phandle.dts"
+
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor.test.fit \
+ "$SRCDIR/fdtfactor_test.its"
+ cp fdtfactor.test.fit fdtfactor_orig.test.fit
+ run_wrap_test $PYTHON $FDTFACTOR -s -F fdtfactor.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-1 description
+ run_fdtget_test "fdt-base-2.dtb" fdtfactor.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo" fdtfactor.test.fit \
+ /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-2" fdtfactor.test.fit \
+ /configurations/conf-2 fdt
+ run_fdtget_test "fdt-base-1 fdt-3 fdt-ovl-1" fdtfactor.test.fit \
+ /configurations/conf-3 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor.test.fit \
+ /configurations/conf-4 fdt
+ run_fdtget_test "fdt-base-2 fdt-5" fdtfactor.test.fit \
+ /configurations/conf-5 fdt
+ # conf-1 already had an explicit compatible property; conf-2 did not, so
+ # fdtfactor should populate it from fdt-2's root compatible property.
+ run_fdtget_test "vendor,board1 vendor,common" fdtfactor.test.fit \
+ /configurations/conf-1 compatible
+ run_fdtget_test "test_fdtfactor" fdtfactor.test.fit \
+ /configurations/conf-2 compatible
+ run_fdtget_test "vendor,other1 vendor,other" fdtfactor.test.fit \
+ /configurations/conf-4 compatible
+ # Verify that hash-1/value was populated for fdt-base-1, fdt-base-2, and
+ # updated for fdt-1.
+ base1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-1/hash-1 value)
+ base2_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-base-2/hash-1 value)
+ fdt1_hash=$($DTGET -t bx fdtfactor.test.fit \
+ /images/fdt-1/hash-1 value)
+ if [ -z "$base1_hash" ] || [ -z "$base2_hash" ] || \
+ [ "$fdt1_hash" = "0 0 0 0 0 0 0 0" ]; then
+ FAIL "FIT hash-1 values were not updated"
+ fi
+ # Extract factored base DTBs and overlays from the FIT and verify all 5
+ # boards with fdtoverlay + dtbs_equal_unordered.
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-1 data fit_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor.test.fit /images/fdt-${i} data \
+ fit_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base1.test.dtb \
+ -o fit_merged${i}.test.dtb fit_fdt${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.test.dtb \
+ fit_merged${i}.test.dtb
+ done
+ fdtget_raw fdtfactor.test.fit /images/fdt-base-2 data fit_base2.test.dtb
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor.test.fit /images/fdt-${idx} data \
+ fit_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_base2.test.dtb \
+ -o fit_merged${idx}.test.dtb fit_fdt${idx}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_other${i}.test.dtb \
+ fit_merged${idx}.test.dtb
+ done
+
+ # --single-base-dtb forces a single fdt-base-1 across all FDTs
+ run_wrap_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_orig.test.fit --outfile fdtfactor_single.test.fit
+ run_fdtget_test "fdt-base-1.dtb" fdtfactor_single.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_single.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-1 fdt-4" fdtfactor_single.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-base-1 data \
+ fit_single_base1.test.dtb
+ for i in 1 2 3; do
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${i} data \
+ fit_single_fdt${i}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${i}.test.dtb fit_single_fdt${i}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board${i}.test.dtb \
+ fit_single_merged${i}.test.dtb
+ done
+ for i in 1 2; do
+ idx=$((i + 3))
+ fdtget_raw fdtfactor_single.test.fit /images/fdt-${idx} data \
+ fit_single_fdt${idx}.test.dtbo
+ run_wrap_test $FDTOVERLAY -i fit_single_base1.test.dtb \
+ -o fit_single_merged${idx}.test.dtb fit_single_fdt${idx}.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_other${i}.test.dtb \
+ fit_single_merged${idx}.test.dtb
+ done
+
+ # Verify that conflicting image properties (e.g. arch) in a group fail
+ cp fdtfactor_orig.test.fit fdtfactor_bad_arch.test.fit
+ run_wrap_test $DTPUT fdtfactor_bad_arch.test.fit /images/fdt-4 \
+ arch -ts "arm"
+ run_wrap_error_test $PYTHON $FDTFACTOR --single-base-dtb \
+ -F fdtfactor_bad_arch.test.fit
+
+ # Verify that signed FITs (with signature* subnodes) and
+ # already-factored FITs (containing fdt-base-*) are rejected
+ cp fdtfactor_orig.test.fit fdtfactor_signed.test.fit
+ run_wrap_test $DTPUT fdtfactor_signed.test.fit -c \
+ /configurations/conf-1/signature-1
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor_signed.test.fit
+ run_wrap_error_test $PYTHON $FDTFACTOR -F fdtfactor.test.fit
+
+ if command -v lz4 >/dev/null 2>&1; then
+ run_wrap_test lz4 -12 -q -f -m \
+ fdtfactor_board1.test.dtb fdtfactor_board2.test.dtb \
+ fdtfactor_board3.test.dtb fdtfactor_other1.test.dtb \
+ fdtfactor_other2.test.dtb overlay_overlay_phandle.test.dtb
+ run_dtc_test -i . -I dts -O dtb -o fdtfactor_lz4.test.fit \
+ "$SRCDIR/fdtfactor_lz4.its"
+ run_wrap_test $PYTHON $FDTFACTOR -s \
+ -F fdtfactor_lz4.test.fit --outfile fdtfactor_lz4_out.test.fit
+ run_fdtget_test "fdt-base-1.dtb.lz4" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-1 description
+ run_fdtget_test "fdtfactor_board1.test.dtbo.lz4" \
+ fdtfactor_lz4_out.test.fit /images/fdt-1 description
+ run_fdtget_test "fdt-base-2.dtb.lz4" fdtfactor_lz4_out.test.fit \
+ /images/fdt-base-2 description
+ run_fdtget_test "fdtfactor_other1.test.dtbo.lz4" \
+ fdtfactor_lz4_out.test.fit /images/fdt-4 description
+ run_fdtget_test "fdt-base-1 fdt-1" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-1 fdt
+ run_fdtget_test "fdt-base-2 fdt-4" fdtfactor_lz4_out.test.fit \
+ /configurations/conf-4 fdt
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-base-1 data \
+ fit_lz4_base1.test.dtb.lz4
+ fdtget_raw fdtfactor_lz4_out.test.fit /images/fdt-1 data \
+ fit_lz4_fdt1.test.dtbo.lz4
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_base1.test.dtb.lz4
+ run_wrap_test lz4 -d -q -f --rm fit_lz4_fdt1.test.dtbo.lz4
+ run_wrap_test $FDTOVERLAY -i fit_lz4_base1.test.dtb \
+ -o fit_lz4_merged1.test.dtb fit_lz4_fdt1.test.dtbo
+ run_test dtbs_equal_unordered fdtfactor_board1.test.dtb \
+ fit_lz4_merged1.test.dtb
+ fi
}
while getopts "vt:me" ARG ; do
@@ -1368,7 +1524,7 @@ if [ -z "$TESTSETS" ]; then
fi
# Make sure we don't have stale blobs lying around
-rm -f *.test.dtb *.test.dtbo *.test.dts *.test.dtb.gz *.test.dtbo.gz \
+rm -f *.test.dtb *.test.dtbo *.test.dts *.test.fit *.test.dtb.gz *.test.dtbo.gz \
*.test.dtb.xz *.test.dtbo.xz *.test.dtb.lzma *.test.dtbo.lzma \
*.test.dtb.lz4 *.test.dtbo.lz4
rm -rf *.test.dir
--
2.56.0.317.g454f8302e1-goog
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays
2026-11-01 22:01 [PATCH v3 00/15] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
` (14 preceding siblings ...)
2026-11-01 22:01 ` [PATCH v3 15/15] fdtfactor: Add compiled FIT factoring support Douglas Anderson
@ 2026-11-02 4:44 ` David Gibson
15 siblings, 0 replies; 23+ messages in thread
From: David Gibson @ 2026-11-02 4:44 UTC (permalink / raw)
To: Douglas Anderson
Cc: Devicetree-compiler, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Konrad Dybcio, Peter Griffin, Saravana Kannan,
Devicetree, Simon Glass, Chen-Yu Tsai, Julius Werner,
Brian Norris, linux-kernel, Tudor Ambarus, André Draszik
[-- Attachment #1: Type: text/plain, Size: 544 bytes --]
On Sun, Nov 01, 2026 at 02:01:07PM -0800, Douglas Anderson wrote:
>
> Devicetree overlays exist at both the source level (via `/include/` and
> `.dtsi` files) and the binary level (where a compiled `.dtbo` overlay is
> applied to a "base" `.dtb` at runtime to add or modify nodes).
[...]
Patches 1..6 merged, still looking at the rest.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 23+ messages in thread