mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Douglas Anderson <dianders@chromium.org>
To: Devicetree-compiler@vger.kernel.org,
	Rob Herring <robh@kernel.org>,
	Krzysztof Kozlowski <krzk+dt@kernel.org>,
	Conor Dooley <conor+dt@kernel.org>,
	Konrad Dybcio <konradybcio@kernel.org>
Cc: Devicetree@vger.kernel.org,
	"Tudor Ambarus" <tudor.ambarus@linaro.org>,
	"Chen-Yu Tsai" <wenst@chromium.org>,
	"Saravana Kannan" <saravanak@kernel.org>,
	"David Gibson" <david@gibson.dropbear.id.au>,
	"André Draszik" <andre.draszik@linaro.org>,
	"Simon Glass" <sjg@chromium.org>,
	linux-kernel@vger.kernel.org,
	"Brian Norris" <briannorris@chromium.org>,
	"Peter Griffin" <peter.griffin@linaro.org>,
	"Julius Werner" <jwerner@chromium.org>,
	"Douglas Anderson" <dianders@chromium.org>
Subject: [PATCH v4 14/14] fdtfactor: Parallelize blob compression and decompression
Date: Mon, 16 Nov 2026 12:53:11 -0800	[thread overview]
Message-ID: <20261116205320.1621386-15-dianders@chromium.org> (raw)
In-Reply-To: <20261116205320.1621386-1-dianders@chromium.org>

When factoring dozens or hundreds of compressed DTBs (either standalone
with `-C` or inside a compiled FIT with `-F`), running `gzip`, `lzma`,
`xz`, or external `lz4` processes sequentially spends several seconds
compressing and decompressing blobs one at a time.

Because Python's `zlib` and `_lzma` C extensions and `subprocess.run()`
for `lz4` all release the GIL, add a small `_map_blobs()` helper using
`concurrent.futures.ThreadPoolExecutor` to decompress input blobs and
compress output base/overlay blobs across worker threads in
deterministic input order whenever multiple compressed blobs are
processed.

Assisted-by: Gemini:gemini-pro-prerelease
Signed-off-by: Douglas Anderson <dianders@chromium.org>
---
Found this speedup while benchmarking fdtfactor. It's not critical to
functionality, but seems like a nice win.

Changes in v4:
- New patch parallelizing blob compression and decompression across
  worker threads

 fdtfactor | 190 +++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 138 insertions(+), 52 deletions(-)

diff --git a/fdtfactor b/fdtfactor
index ebf40cdeaf92..ab59763d8863 100755
--- a/fdtfactor
+++ b/fdtfactor
@@ -18,6 +18,7 @@ its embedded `/images/fdt-*` blobs into base DTBs (`fdt-base-1`, `fdt-base-2`,
 
 import argparse
 from collections import Counter
+import concurrent.futures
 import copy
 import gzip
 import hashlib
@@ -140,6 +141,32 @@ def compress_blob(raw_bytes, comp, filename=''):
     raise ValueError(f'{prefix}Unsupported compression type: {comp}')
 
 
+def _map_blobs(func, tasks, pool=None):
+    """Apply `func(blob, comp, filename)` across `tasks`, parallelizing if compressed.
+
+    Python's `zlib` and `_lzma` C extensions and external `lz4` subprocesses
+    release the GIL, so running multiple compressed tasks across a
+    `ThreadPoolExecutor` overlaps compression and decompression across CPU
+    cores while preserving deterministic input order.
+
+    Args:
+        func: `decompress_blob` or `compress_blob`.
+        tasks: Sequence of `(blob_bytes, comp, filename)` tuples.
+        pool: Optional shared `ThreadPoolExecutor` instance.
+
+    Returns:
+        List of processed `bytes` in the same order as `tasks`.
+    """
+    if len(tasks) < 2 or not any(
+        comp not in ('none', '', None) for _, comp, _ in tasks
+    ):
+        return [func(*t) for t in tasks]
+    if pool is not None:
+        return list(pool.map(lambda t: func(*t), tasks))
+    with concurrent.futures.ThreadPoolExecutor() as new_pool:
+        return list(new_pool.map(lambda t: func(*t), tasks))
+
+
 def _is_dtbo_overlay(root):
     """Return True if `root` is a `.dtbo` overlay tree rather than a base DTB.
 
@@ -1061,13 +1088,14 @@ def group_fit_dtbs(fdt_entries, min_overlap=50.0):
     return [grp['entries'] for grp in groups]
 
 
-def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
+def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node, pool=None):
     """Extract and parse non-overlay `flat_dt` entries from `/images`.
 
     Args:
         fit_in_path: Input FIT path (for error reporting).
         images_node: `DeviceTree` for the FIT `/images` node.
         configs_node: `DeviceTree` for the FIT `/configurations` node.
+        pool: Optional shared `ThreadPoolExecutor` for decompression.
 
     Returns:
         List of dicts (`{'name', 'node', 'dtb', 'fdt'}`) for all non-overlay
@@ -1081,7 +1109,7 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
                     f'is not supported; sign the FIT after factoring'
                 )
 
-    fdt_entries = []
+    candidates = []
     for img_name, img_node in images_node.children.items():
         if ('data-offset' in img_node.props or
                 'data-position' in img_node.props):
@@ -1112,7 +1140,19 @@ def _extract_fit_fdt_entries(fit_in_path, images_node, configs_node):
             )
         if comp == 'lz4' and raw_data.startswith(LZ4_LEGACY_MAGIC):
             comp = 'lz4_legacy'
-        raw_bytes = decompress_blob(raw_data, comp, filename=node_label)
+        candidates.append((img_name, img_node, node_label, comp, raw_data))
+
+    raw_blobs = _map_blobs(
+        decompress_blob,
+        [(raw_data, comp, node_label)
+         for _, _, node_label, comp, raw_data in candidates],
+        pool=pool,
+    )
+
+    fdt_entries = []
+    for (img_name, img_node, node_label, comp, raw_data), raw_bytes in zip(
+        candidates, raw_blobs
+    ):
         fdt = _open_fdt(raw_bytes, node_label)
         root = FactoredTree.unflatten(fdt)
         if _is_dtbo_overlay(root):
@@ -1194,13 +1234,15 @@ def _partition_fit_fdt_entries(fdt_entries, single_base_dtb, min_overlap=50.0):
     return factorable_groups, unfactored_entries
 
 
-def _strip_fit_unfactored_local_fixups(unfactored_entries):
+def _strip_fit_unfactored_local_fixups(unfactored_entries, pool=None):
     """Strip `/__local_fixups__` from unfactored FIT `flat_dt` entries.
 
     Args:
         unfactored_entries: List of single-board FIT `flat_dt` entry dicts left
             unfactored by `_partition_fit_fdt_entries()`.
+        pool: Optional shared `ThreadPoolExecutor` for compression.
     """
+    to_recompress = []
     for entry in unfactored_entries:
         fdt = entry.pop('fdt')
         tree = libpydt.DeviceTree.unflatten(fdt)
@@ -1210,17 +1252,26 @@ def _strip_fit_unfactored_local_fixups(unfactored_entries):
             mem_rsv=entry['dtb'].mem_rsv,
             boot_cpuid_phys=entry['dtb'].boot_cpuid_phys,
         )
-        comp_bytes = compress_blob(
-            bytes(stripped_fdt.as_bytearray()),
+        stripped_bytes = bytes(stripped_fdt.as_bytearray())
+        to_recompress.append((
+            entry['node'],
+            stripped_bytes,
             entry['dtb'].compression,
-            filename=f'/images/{entry["name"]}',
-        )
-        img_node = entry['node']
+            f'/images/{entry["name"]}',
+        ))
+
+    comp_blobs = _map_blobs(
+        compress_blob,
+        [(blob, comp, label) for _, blob, comp, label in to_recompress],
+        pool=pool,
+    )
+    for (img_node, _, _, _), comp_bytes in zip(to_recompress, comp_blobs):
         img_node.set_prop(libfdt.Property('data', comp_bytes))
         _update_fit_hashes(img_node)
 
 
-def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
+def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups,
+                      pool=None):
     """Factor one group of FIT `flat_dt` entries into a base node and overlays.
 
     Args:
@@ -1229,6 +1280,7 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
             node (e.g., `'fdt-base-1'`).
         strip_local_fixups: If True, omit `/__local_fixups__` from the
             generated base DTB.
+        pool: Optional shared `ThreadPoolExecutor` for compression.
 
     Returns:
         Tuple of `(base_dtb_node, stat_args)` where `base_dtb_node` is the
@@ -1241,9 +1293,20 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
     )
     base_comp = group_dtbs[0].compression
 
-    base_dtb_comp_bytes = compress_blob(
-        base_dtb_blob, base_comp, filename=f'/images/{base_dtb_node_name}'
+    comp_blobs = _map_blobs(
+        compress_blob,
+        [
+            (base_dtb_blob, base_comp, f'/images/{base_dtb_node_name}'),
+            *(
+                (dtbo_blob, entry['dtb'].compression, f'/images/{entry["name"]}')
+                for entry, dtbo_blob in zip(entries, dtbo_blobs)
+            ),
+        ],
+        pool=pool,
     )
+    base_dtb_comp_bytes = comp_blobs[0]
+    dtbo_comp_blobs = comp_blobs[1:]
+
     base_dtb_node = copy.deepcopy(entries[0]['node'])
     for prop in ('phandle', 'compatible'):
         base_dtb_node.props.pop(prop, None)
@@ -1272,12 +1335,9 @@ def _factor_fit_group(entries, base_dtb_node_name, strip_local_fixups):
     _update_fit_hashes(base_dtb_node)
 
     dtbo_comp_sizes = []
-    for entry, dtbo_blob in zip(entries, dtbo_blobs):
+    for entry, dtbo_comp_bytes in zip(entries, dtbo_comp_blobs):
         img_node = entry['node']
         entry_comp = entry['dtb'].compression
-        dtbo_comp_bytes = compress_blob(
-            dtbo_blob, entry_comp, filename=f'/images/{entry["name"]}'
-        )
         dtbo_comp_sizes.append(len(dtbo_comp_bytes))
 
         if 'description' in img_node.props:
@@ -1386,39 +1446,42 @@ def factor_fit(fit_in_path, fit_out_path, strip_local_fixups=False,
             f'/configurations nodes'
         )
 
-    fdt_entries = _extract_fit_fdt_entries(
-        fit_in_path, images_node, configs_node
-    )
-    factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
-        fdt_entries, single_base_dtb, min_overlap=min_overlap
-    )
-    if strip_local_fixups and unfactored_entries:
-        _strip_fit_unfactored_local_fixups(unfactored_entries)
-    else:
-        for entry in unfactored_entries:
-            entry.pop('fdt', None)
-
-    fdt_to_base_dtb = {}
-    insert_before_first = {}
-    group_stats = []
-    used_image_names = set(images_node.children)
-    next_base_idx = 1
-
-    for entries in factorable_groups:
-        while f'fdt-base-{next_base_idx}' in used_image_names:
-            next_base_idx += 1
-        base_dtb_node_name = f'fdt-base-{next_base_idx}'
-        used_image_names.add(base_dtb_node_name)
-        next_base_idx += 1
-        base_dtb_node, stat_args = _factor_fit_group(
-            entries, base_dtb_node_name, strip_local_fixups
+    with concurrent.futures.ThreadPoolExecutor() as pool:
+        fdt_entries = _extract_fit_fdt_entries(
+            fit_in_path, images_node, configs_node, pool=pool
         )
-        insert_before_first[entries[0]['name']] = (
-            base_dtb_node_name, base_dtb_node
+        factorable_groups, unfactored_entries = _partition_fit_fdt_entries(
+            fdt_entries, single_base_dtb, min_overlap=min_overlap
         )
-        for entry in entries:
-            fdt_to_base_dtb[entry['name']] = (base_dtb_node_name, entry['dtb'])
-        group_stats.append(stat_args)
+        if strip_local_fixups and unfactored_entries:
+            _strip_fit_unfactored_local_fixups(unfactored_entries, pool=pool)
+        else:
+            for entry in unfactored_entries:
+                entry.pop('fdt', None)
+
+        fdt_to_base_dtb = {}
+        insert_before_first = {}
+        group_stats = []
+        used_image_names = set(images_node.children)
+        next_base_idx = 1
+
+        for entries in factorable_groups:
+            while f'fdt-base-{next_base_idx}' in used_image_names:
+                next_base_idx += 1
+            base_dtb_node_name = f'fdt-base-{next_base_idx}'
+            used_image_names.add(base_dtb_node_name)
+            next_base_idx += 1
+            base_dtb_node, stat_args = _factor_fit_group(
+                entries, base_dtb_node_name, strip_local_fixups, pool=pool
+            )
+            insert_before_first[entries[0]['name']] = (
+                base_dtb_node_name, base_dtb_node
+            )
+            for entry in entries:
+                fdt_to_base_dtb[entry['name']] = (
+                    base_dtb_node_name, entry['dtb']
+                )
+            group_stats.append(stat_args)
 
     _update_fit_images_and_configs(
         images_node, configs_node, insert_before_first, fdt_to_base_dtb
@@ -1668,7 +1731,24 @@ def factor_dtb_files(args):
     """
     comp = args.compression
     input_dtb_paths = args.dtbs
-    dtbs = [ParsedDtb(p, comp=comp) for p in input_dtb_paths]
+    file_blobs = []
+    for p in input_dtb_paths:
+        with open(p, 'rb') as f:
+            file_blobs.append(f.read())
+    raw_blobs = _map_blobs(
+        decompress_blob,
+        [(b, comp, p) for p, b in zip(input_dtb_paths, file_blobs)],
+    )
+    dtbs = [
+        ParsedDtb(
+            p,
+            comp=comp,
+            fdt=_open_fdt(raw, p),
+            uncompressed_size=len(raw),
+            compressed_size=len(fblob),
+        )
+        for p, fblob, raw in zip(input_dtb_paths, file_blobs, raw_blobs)
+    ]
 
     base_dtb_path = (
         os.path.join(args.output, args.output_base_dtb)
@@ -1706,20 +1786,26 @@ def factor_dtb_files(args):
         strip_local_fixups=args.strip_local_fixups,
     )
 
-    base_dtb_file_bytes = compress_blob(
-        base_dtb_blob, comp, filename=base_dtb_path
+    comp_blobs = _map_blobs(
+        compress_blob,
+        [
+            (base_dtb_blob, comp, base_dtb_path),
+            *((b, comp, p) for p, b in zip(dtbo_paths, dtbo_blobs)),
+        ],
     )
+    base_dtb_file_bytes = comp_blobs[0]
+    dtbo_comp_blobs = comp_blobs[1:]
+
     if args.output:
         os.makedirs(args.output, exist_ok=True)
     with open(base_dtb_path, 'wb') as f:
         f.write(base_dtb_file_bytes)
 
     dtbo_file_sizes = []
-    for out_path, dtbo_blob in zip(dtbo_paths, dtbo_blobs):
+    for out_path, out_bytes in zip(dtbo_paths, dtbo_comp_blobs):
         out_dir = os.path.dirname(out_path)
         if out_dir:
             os.makedirs(out_dir, exist_ok=True)
-        out_bytes = compress_blob(dtbo_blob, comp, filename=out_path)
         dtbo_file_sizes.append(len(out_bytes))
         with open(out_path, 'wb') as f:
             f.write(out_bytes)
-- 
2.56.0.679.gc527c65306-goog


      parent reply	other threads:[~2026-11-16 20:54 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 20:52 [PATCH v4 00/14] fdtfactor: Introduce fdtfactor to auto-factor overlays Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 01/14] README: Add overview blurbs for dtc and ancillary tools Douglas Anderson
2026-11-16 20:52 ` [PATCH v4 02/14] pylibfdt: Make Property a subclass of bytes instead of bytearray Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 03/14] pylibfdt: Trim Fdt bytearray to totalsize() in FdtSw.as_fdt() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 04/14] pylibfdt: Add Property.from_*() creation methods Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 05/14] pylibfdt: Unpack Property.as_list() in a single struct.unpack() call Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 06/14] pylibfdt: Speed up FdtRo.properties() with a C helper Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 07/14] libpydt: Add DeviceTree live tree representation Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 08/14] libfdt: Speed up fdt_find_string_len_() with memchr() Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 09/14] libfdt: overlay: Update base /__local_fixups__ on apply Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 10/14] fdtfactor: Add tool to split DTBs into a base DTB and overlays Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 11/14] fdtfactor: Add -S/--strip-local-fixups option Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 12/14] fdtfactor: Support compressed DTB inputs and outputs Douglas Anderson
2026-11-16 20:53 ` [PATCH v4 13/14] fdtfactor: Add compiled FIT factoring support Douglas Anderson
2026-11-16 20:53 ` Douglas Anderson [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116205320.1621386-15-dianders@chromium.org \
    --to=dianders@chromium.org \
    --cc=Devicetree-compiler@vger.kernel.org \
    --cc=Devicetree@vger.kernel.org \
    --cc=andre.draszik@linaro.org \
    --cc=briannorris@chromium.org \
    --cc=conor+dt@kernel.org \
    --cc=david@gibson.dropbear.id.au \
    --cc=jwerner@chromium.org \
    --cc=konradybcio@kernel.org \
    --cc=krzk+dt@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peter.griffin@linaro.org \
    --cc=robh@kernel.org \
    --cc=saravanak@kernel.org \
    --cc=sjg@chromium.org \
    --cc=tudor.ambarus@linaro.org \
    --cc=wenst@chromium.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®