From: Douglas Anderson <dianders@chromium.org>
To: Devicetree-compiler@vger.kernel.org,
Rob Herring <robh@kernel.org>,
Krzysztof Kozlowski <krzk+dt@kernel.org>,
Conor Dooley <conor+dt@kernel.org>,
Konrad Dybcio <konradybcio@kernel.org>
Cc: Devicetree@vger.kernel.org,
"Tudor Ambarus" <tudor.ambarus@linaro.org>,
"Chen-Yu Tsai" <wenst@chromium.org>,
"Saravana Kannan" <saravanak@kernel.org>,
"David Gibson" <david@gibson.dropbear.id.au>,
"André Draszik" <andre.draszik@linaro.org>,
"Simon Glass" <sjg@chromium.org>,
linux-kernel@vger.kernel.org,
"Brian Norris" <briannorris@chromium.org>,
"Peter Griffin" <peter.griffin@linaro.org>,
"Julius Werner" <jwerner@chromium.org>,
"Douglas Anderson" <dianders@chromium.org>
Subject: [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression
Date: Mon, 16 Nov 2026 12:53:11 -0800 [thread overview]
Message-ID: <20261116205320.1621386-15-dianders@chromium.org> (raw)
In-Reply-To: <20261116205320.1621386-1-dianders@chromium.org>
When factoring dozens or hundreds of compressed DTBs (either standalone
with `-C` or inside a compiled FIT with `-F`), running `gzip`, `lzma`,
`xz`, or external `lz4` processes sequentially spends several seconds
compressing and decompressing blobs one at a time.
Because Python's `zlib` and `_lzma` C extensions and `subprocess.run()`
for `lz4` all release the GIL, add a small `_map_blobs()` helper using
`concurrent.futures.ThreadPoolExecutor` to decompress input blobs and
compress output base/overlay blobs across worker threads in
deterministic input order whenever multiple compressed blobs are
processed.
Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.
Changes in v4:
- New patch parallelizing blob compression and decompression across
worker threads
fdtfactor | 190 +++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 138 insertions(+), 52 deletions(-)
diff --git a/fdtfactor b/fdtfactor
index ebf40cdeaf92..ab59763d8863 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -18,6 +18,7 @@ its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
import argparse
from collections import Counter
+import concurrent.futures
import copy
import gzip
import hashlib
@@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
raise ValueError(f'{prefix}Unsupported compression type: {comp}')
+def _map_blobs(func, tasks, pool=None):
+ """Apply `func(blob, comp, filename)` across `tasks`, parallelizing if compressed.
+
+ Python's `zlib` and `_lzma` C extensions and external `lz4` subprocesses
+ release the GIL, so running multiple compressed tasks across a
+ `ThreadPoolExecutor` overlaps compression and decompression across CPU
+ cores while preserving deterministic input order.
+
+ Args:
+ func: `decompress_blob` or `compress_blob`.
+ tasks: Sequence of `(blob_bytes, comp, filename)` tuples.
+ pool: Optional shared `ThreadPoolExecutor` instance.
+
+ Returns:
+ List of processed `bytes` in the same order as `tasks`.
+ """
+ if len(tasks) < 2 or not any(
+ comp not in ('none', '', None) for _, comp, _ in tasks
+ ):
+ return [func(*t) for t in tasks]
+ if pool is not None:
+ return list(pool.map(lambda t: func(*t), tasks))
+ with concurrent.futures.ThreadPoolExecutor() as new_pool:
+ return list(new_pool.map(lambda t: func(*t), tasks))
+
+
def _is_dtbo_overlay(root):
"""Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
@@ -1061,13 +1088,14 @@ def group_fit_dtbs(fdt_entries, min_overlap=50.0):
return [grp['entries'] for grp in groups]
-def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node, pool=None):
"""Extract and parse non-overlay `flat_dt` entries from `/images`.
Args:
fit_in_path: Input FIT path (for error reporting).
images_node: `DeviceTree` for the FIT `/images` node.
configs_node: `DeviceTree` for the FIT `/configurations` node.
+ pool: Optional shared `ThreadPoolExecutor` for decompression.
Returns:
List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
@@ -1081,7 +1109,7 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
f'is not supported; sign the FIT after factoring'
)
- fdt_entries = []
+ candidates = []
for img_name, img_node in images_node.children.items():
if ('data-offset' in img_node.props or
'data-position' in img_node.props):
@@ -1112,7 +1140,19 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
)
if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
comp = 'lz4_legacy'
- raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+ candidates.append((img_name, img_node, node_label, comp, raw_data))
+
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(raw_data, comp, node_label)
+ for _, _, node_label, comp, raw_data in candidates],
+ pool=pool,
+ )
+
+ fdt_entries = []
+ for (img_name, img_node, node_label, comp, raw_data), raw_bytes in zip(
+ candidates, raw_blobs
+ ):
fdt = _open_fdt(raw_bytes, node_label)
root = FactoredTree.unflatten(fdt)
if _is_dtbo_overlay(root):
@@ -1194,13 +1234,15 @@ def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
return factorable_groups, unfactored_entries
-def _strip_fit_unfactored_local_fixups(unfactored_entries):
+def _strip_fit_unfactored_local_fixups(unfactored_entries, pool=None):
"""Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
Args:
unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
unfactored by `_partition_fit_fdt_entries()`.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
"""
+ to_recompress = []
for entry in unfactored_entries:
fdt = entry.pop('fdt')
tree = libpydt.DeviceTree.unflatten(fdt)
@@ -1210,17 +1252,26 @@ def _strip_fit_unfactored_local_fixups(unfactored_entries):
mem_rsv=entry['dtb'].mem_rsv,
boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
)
- comp_bytes = compress_blob(
- bytes(stripped_fdt.as_bytearray()),
+ stripped_bytes = bytes(stripped_fdt.as_bytearray())
+ to_recompress.append((
+ entry['node'],
+ stripped_bytes,
entry['dtb'].compression,
- filename=f'/images/{entry["name"]}',
- )
- img_node = entry['node']
+ f'/images/{entry["name"]}',
+ ))
+
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [(blob, comp, label) for _, blob, comp, label in to_recompress],
+ pool=pool,
+ )
+ for (img_node, _, _, _), comp_bytes in zip(to_recompress, comp_blobs):
img_node.set_prop(libfdt.Property('data', comp_bytes))
_update_fit_hashes(img_node)
-def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups,
+ pool=None):
"""Factor one group of FIT `flat_dt` entries into a base node and overlays.
Args:
@@ -1229,6 +1280,7 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
node (e.g., `'fdt-base-1'`).
strip_local_fixups: If True, omit `/__local_fixups__` from the
generated base DTB.
+ pool: Optional shared `ThreadPoolExecutor` for compression.
Returns:
Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
@@ -1241,9 +1293,20 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
)
base_comp = group_dtbs[0].compression
- base_dtb_comp_bytes = compress_blob(
- base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, base_comp, f'/images/{base_dtb_node_name}'),
+ *(
+ (dtbo_blob, entry['dtb'].compression, f'/images/{entry["name"]}')
+ for entry, dtbo_blob in zip(entries, dtbo_blobs)
+ ),
+ ],
+ pool=pool,
)
+ base_dtb_comp_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
base_dtb_node = copy.deepcopy(entries[0]['node'])
for prop in ('phandle', 'compatible'):
base_dtb_node.props.pop(prop, None)
@@ -1272,12 +1335,9 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
_update_fit_hashes(base_dtb_node)
dtbo_comp_sizes = []
- for entry, dtbo_blob in zip(entries, dtbo_blobs):
+ for entry, dtbo_comp_bytes in zip(entries, dtbo_comp_blobs):
img_node = entry['node']
entry_comp = entry['dtb'].compression
- dtbo_comp_bytes = compress_blob(
- dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
- )
dtbo_comp_sizes.append(len(dtbo_comp_bytes))
if 'description' in img_node.props:
@@ -1386,39 +1446,42 @@ def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
f'/configurations nodes'
)
- fdt_entries = _extract_fit_fdt_entries(
- fit_in_path, images_node, configs_node
- )
- factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
- fdt_entries, single_base_dtb, min_overlap=min_overlap
- )
- if strip_local_fixups and unfactored_entries:
- _strip_fit_unfactored_local_fixups(unfactored_entries)
- else:
- for entry in unfactored_entries:
- entry.pop('fdt', None)
-
- fdt_to_base_dtb = {}
- insert_before_first = {}
- group_stats = []
- used_image_names = set(images_node.children)
- next_base_idx = 1
-
- for entries in factorable_groups:
- while f'fdt-base-{next_base_idx}' in used_image_names:
- next_base_idx += 1
- base_dtb_node_name = f'fdt-base-{next_base_idx}'
- used_image_names.add(base_dtb_node_name)
- next_base_idx += 1
- base_dtb_node, stat_args = _factor_fit_group(
- entries, base_dtb_node_name, strip_local_fixups
+ with concurrent.futures.ThreadPoolExecutor() as pool:
+ fdt_entries = _extract_fit_fdt_entries(
+ fit_in_path, images_node, configs_node, pool=pool
)
- insert_before_first[entries[0]['name']] = (
- base_dtb_node_name, base_dtb_node
+ factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+ fdt_entries, single_base_dtb, min_overlap=min_overlap
)
- for entry in entries:
- fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
- group_stats.append(stat_args)
+ if strip_local_fixups and unfactored_entries:
+ _strip_fit_unfactored_local_fixups(unfactored_entries, pool=pool)
+ else:
+ for entry in unfactored_entries:
+ entry.pop('fdt', None)
+
+ fdt_to_base_dtb = {}
+ insert_before_first = {}
+ group_stats = []
+ used_image_names = set(images_node.children)
+ next_base_idx = 1
+
+ for entries in factorable_groups:
+ while f'fdt-base-{next_base_idx}' in used_image_names:
+ next_base_idx += 1
+ base_dtb_node_name = f'fdt-base-{next_base_idx}'
+ used_image_names.add(base_dtb_node_name)
+ next_base_idx += 1
+ base_dtb_node, stat_args = _factor_fit_group(
+ entries, base_dtb_node_name, strip_local_fixups, pool=pool
+ )
+ insert_before_first[entries[0]['name']] = (
+ base_dtb_node_name, base_dtb_node
+ )
+ for entry in entries:
+ fdt_to_base_dtb[entry['name']] = (
+ base_dtb_node_name, entry['dtb']
+ )
+ group_stats.append(stat_args)
_update_fit_images_and_configs(
images_node, configs_node, insert_before_first, fdt_to_base_dtb
@@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
"""
comp = args.compression
input_dtb_paths = args.dtbs
- dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
+ file_blobs = []
+ for p in input_dtb_paths:
+ with open(p, 'rb') as f:
+ file_blobs.append(f.read())
+ raw_blobs = _map_blobs(
+ decompress_blob,
+ [(b, comp, p) for p, b in zip(input_dtb_paths, file_blobs)],
+ )
+ dtbs = [
+ ParsedDtb(
+ p,
+ comp=comp,
+ fdt=_open_fdt(raw, p),
+ uncompressed_size=len(raw),
+ compressed_size=len(fblob),
+ )
+ for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
+ ]
base_dtb_path = (
os.path.join(args.output, args.output_base_dtb)
@@ -1706,20 +1786,26 @@ def factor_dtb_files(args):
strip_local_fixups=args.strip_local_fixups,
)
- base_dtb_file_bytes = compress_blob(
- base_dtb_blob, comp, filename=base_dtb_path
+ comp_blobs = _map_blobs(
+ compress_blob,
+ [
+ (base_dtb_blob, comp, base_dtb_path),
+ *((b, comp, p) for p, b in zip(dtbo_paths, dtbo_blobs)),
+ ],
)
+ base_dtb_file_bytes = comp_blobs[0]
+ dtbo_comp_blobs = comp_blobs[1:]
+
if args.output:
os.makedirs(args.output, exist_ok=True)
with open(base_dtb_path, 'wb') as f:
f.write(base_dtb_file_bytes)
dtbo_file_sizes = []
- for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+ for out_path, out_bytes in zip(dtbo_paths, dtbo_comp_blobs):
out_dir = os.path.dirname(out_path)
if out_dir:
os.makedirs(out_dir, exist_ok=True)
- out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
dtbo_file_sizes.append(len(out_bytes))
with open(out_path, 'wb') as f:
f.write(out_bytes)
--
2.56.0.679.gc527c65306-goog
prev parent reply other threads:[~2026-11-16 20:54 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-16 20:53 ` Douglas Anderson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116205320.1621386-15-dianders@chromium.org \
--to=dianders@chromium.org \
--cc=Devicetree-compiler@vger.kernel.org \
--cc=Devicetree@vger.kernel.org \
--cc=andre.draszik@linaro.org \
--cc=briannorris@chromium.org \
--cc=conor+dt@kernel.org \
--cc=david@gibson.dropbear.id.au \
--cc=jwerner@chromium.org \
--cc=konradybcio@kernel.org \
--cc=krzk+dt@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=peter.griffin@linaro.org \
--cc=robh@kernel.org \
--cc=saravanak@kernel.org \
--cc=sjg@chromium.org \
--cc=tudor.ambarus@linaro.org \
--cc=wenst@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®