[PATCH] firmware: tegra: bpmp: populate debugfs in the background

[PATCH] firmware: tegra: bpmp: populate debugfs in the background

[Austin Schuh]

Populating the BPMP debugfs mirror takes a series of synchronous BPMP
requests, several per directory.  On an Orin Nano (Tegra234) the tree has
about 10,800 entries, and the BPMP probe doesn't return for over five
seconds.  Every device that gets its clocks, resets or power domains from
the BPMP waits for it, which on Tegra234 is nearly all of them, so PCIe,
NVMe and the root filesystem all come up five seconds late.

Commit c7fc690da6f1 ("firmware: tegra: bpmp: Allow BPMP debugfs population
to be skipped") lets a system that doesn't need the mirror turn it off with
tegra_bpmp.debugfs=0.  A system that does use it, for instance to set
clocks from userspace at boot, still pays the full cost.

Nothing in the kernel uses the mirror, and the probe already only logs an
error from it, so populate it from a workqueue instead.  It isn't an
async_schedule() job, since kernel_init() waits for all of those before it
starts init.  Userspace that reads the mirror right after boot may find it
still being filled in.

Measured with initcall_debug on an Orin Nano running 6.18.39 with
NVIDIA's patches: the BPMP probe returns after 1.2 seconds instead of
6.36 seconds.

Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
---

Notes:
    Based on tegra/for-7.4/firmware (1bbcf4a19f14).  Built for arm64, W=1,
    no new warnings.  The timing is from the same change on 6.18.39 with
    NVIDIA's patches (where the probe also has the VDK check), which is what
    our board runs; I haven't booted this branch on it.

 drivers/firmware/tegra/bpmp.c | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c
index d3508e98e338..535ebbd55a6f 100644
--- a/drivers/firmware/tegra/bpmp.c
+++ b/drivers/firmware/tegra/bpmp.c
@@ -16,6 +16,7 @@
 #include <linux/semaphore.h>
 #include <linux/sched/clock.h>
 #include <linux/slab.h>
+#include <linux/workqueue.h>
 
 #include <soc/tegra/bpmp.h>
 #include <soc/tegra/bpmp-abi.h>
@@ -1052,8 +1053,26 @@ int tegra_bpmp_init_dt_providers(struct tegra_bpmp *bpmp)
 	return 0;
 }
 
+struct tegra_bpmp_debugfs_work {
+	struct work_struct work;
+	struct tegra_bpmp *bpmp;
+};
+
+static void tegra_bpmp_debugfs_work_fn(struct work_struct *work)
+{
+	struct tegra_bpmp_debugfs_work *dw =
+		container_of(work, struct tegra_bpmp_debugfs_work, work);
+	int err;
+
+	err = tegra_bpmp_init_debugfs(dw->bpmp);
+	if (err < 0)
+		dev_err(dw->bpmp->dev, "debugfs initialization failed: %d\n",
+			err);
+}
+
 static int tegra_bpmp_probe(struct platform_device *pdev)
 {
+	struct tegra_bpmp_debugfs_work *debugfs_work;
 	struct tegra_bpmp *bpmp;
 	char tag[TAG_SZ];
 	int err;
@@ -1089,9 +1108,19 @@ static int tegra_bpmp_probe(struct platform_device *pdev)
 	if (err < 0)
 		dev_err(&pdev->dev, "failed to initialize sysfs: %d\n", err);
 
-	err = tegra_bpmp_init_debugfs(bpmp);
-	if (err < 0)
-		dev_err(&pdev->dev, "debugfs initialization failed: %d\n", err);
+	/*
+	 * Mirroring the BPMP's debugfs tree takes several requests per
+	 * directory, which on Tegra234 adds up to seconds.  Nothing else
+	 * waits for it, so do it in the background rather than hold up every
+	 * device that depends on the BPMP.
+	 */
+	debugfs_work = devm_kzalloc(&pdev->dev, sizeof(*debugfs_work),
+				    GFP_KERNEL);
+	if (debugfs_work) {
+		debugfs_work->bpmp = bpmp;
+		INIT_WORK(&debugfs_work->work, tegra_bpmp_debugfs_work_fn);
+		queue_work(system_dfl_long_wq, &debugfs_work->work);
+	}
 
 	return 0;
 

base-commit: 1bbcf4a19f1495f769b69e9569d8c1e81df074f5
-- 
2.47.3

Re: [PATCH] firmware: tegra: bpmp: populate debugfs in the background

[Mikko Perttunen]

On Saturday, October 17, 2026 3:50 PM Austin Schuh wrote:
> Populating the BPMP debugfs mirror takes a series of synchronous BPMP
> requests, several per directory.  On an Orin Nano (Tegra234) the tree has
> about 10,800 entries, and the BPMP probe doesn't return for over five
> seconds.  Every device that gets its clocks, resets or power domains from
> the BPMP waits for it, which on Tegra234 is nearly all of them, so PCIe,
> NVMe and the root filesystem all come up five seconds late.
> 
> Commit c7fc690da6f1 ("firmware: tegra: bpmp: Allow BPMP debugfs population
> to be skipped") lets a system that doesn't need the mirror turn it off with
> tegra_bpmp.debugfs=0.  A system that does use it, for instance to set
> clocks from userspace at boot, still pays the full cost.
> 
> Nothing in the kernel uses the mirror, and the probe already only logs an
> error from it, so populate it from a workqueue instead.  It isn't an
> async_schedule() job, since kernel_init() waits for all of those before it
> starts init.  Userspace that reads the mirror right after boot may find it
> still being filled in.
> 
> Measured with initcall_debug on an Orin Nano running 6.18.39 with
> NVIDIA's patches: the BPMP probe returns after 1.2 seconds instead of
> 6.36 seconds.
> 
> Assisted-by: LLM
> Signed-off-by: Austin Schuh <austin.linux@gmail.com>
> ---

Sashiko reports some hard to trigger but worth checking concurrency 
issues.

This does seem a good idea, but I am a bit concerned about the change to 
what userspace sees. Ideally we'd be able to have userspace stall if the 
debugfs population was still in progress. If that's not possible, at the 
very least userspace should have a way to voluntarily wait for the full 
debugfs to be available.

> 
> Notes:
>     Based on tegra/for-7.4/firmware (1bbcf4a19f14).  Built for arm64, W=1,
>     no new warnings.  The timing is from the same change on 6.18.39 with
>     NVIDIA's patches (where the probe also has the VDK check), which is what
>     our board runs; I haven't booted this branch on it.
> 
>  drivers/firmware/tegra/bpmp.c | 35 ++++++++++++++++++++++++++++++++---
>  1 file changed, 32 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/firmware/tegra/bpmp.c b/drivers/firmware/tegra/bpmp.c
> index d3508e98e338..535ebbd55a6f 100644
> --- a/drivers/firmware/tegra/bpmp.c
> +++ b/drivers/firmware/tegra/bpmp.c
> @@ -16,6 +16,7 @@
>  #include <linux/semaphore.h>
>  #include <linux/sched/clock.h>
>  #include <linux/slab.h>
> +#include <linux/workqueue.h>
>  
>  #include <soc/tegra/bpmp.h>
>  #include <soc/tegra/bpmp-abi.h>
> @@ -1052,8 +1053,26 @@ int tegra_bpmp_init_dt_providers(struct tegra_bpmp *bpmp)
>  	return 0;
>  }
>  
> +struct tegra_bpmp_debugfs_work {
> +	struct work_struct work;
> +	struct tegra_bpmp *bpmp;
> +};

Just inline the work_struct inside tegra_bpmp. In any case the work 
should be synchronously cancelled if the bpmp driver instance is going 
to go away (not that that will happen, but good practice).

Thanks
Mikko

> +
> +static void tegra_bpmp_debugfs_work_fn(struct work_struct *work)
> +{
> +	struct tegra_bpmp_debugfs_work *dw =
> +		container_of(work, struct tegra_bpmp_debugfs_work, work);
> +	int err;
> +
> +	err = tegra_bpmp_init_debugfs(dw->bpmp);
> +	if (err < 0)
> +		dev_err(dw->bpmp->dev, "debugfs initialization failed: %d\n",
> +			err);
> +}
> +
>  static int tegra_bpmp_probe(struct platform_device *pdev)
>  {
> +	struct tegra_bpmp_debugfs_work *debugfs_work;
>  	struct tegra_bpmp *bpmp;
>  	char tag[TAG_SZ];
>  	int err;
> @@ -1089,9 +1108,19 @@ static int tegra_bpmp_probe(struct platform_device *pdev)
>  	if (err < 0)
>  		dev_err(&pdev->dev, "failed to initialize sysfs: %d\n", err);
>  
> -	err = tegra_bpmp_init_debugfs(bpmp);
> -	if (err < 0)
> -		dev_err(&pdev->dev, "debugfs initialization failed: %d\n", err);
> +	/*
> +	 * Mirroring the BPMP's debugfs tree takes several requests per
> +	 * directory, which on Tegra234 adds up to seconds.  Nothing else
> +	 * waits for it, so do it in the background rather than hold up every
> +	 * device that depends on the BPMP.
> +	 */
> +	debugfs_work = devm_kzalloc(&pdev->dev, sizeof(*debugfs_work),
> +				    GFP_KERNEL);
> +	if (debugfs_work) {
> +		debugfs_work->bpmp = bpmp;
> +		INIT_WORK(&debugfs_work->work, tegra_bpmp_debugfs_work_fn);
> +		queue_work(system_dfl_long_wq, &debugfs_work->work);
> +	}
>  
>  	return 0;
>  
> 
> base-commit: 1bbcf4a19f1495f769b69e9569d8c1e81df074f5
> -- 
> 2.47.3
> 
> 

[PATCH v2] firmware: tegra: bpmp: populate debugfs in the background

[Austin Schuh]

Populating the BPMP debugfs mirror takes a series of synchronous BPMP
requests, several per directory.  On an Orin Nano (Tegra234) the tree has
about 10,800 entries, and the BPMP probe doesn't return for between two
and six seconds, depending on the L4T release.  Every device that gets its
clocks, resets or power domains from the BPMP waits for it, which on
Tegra234 is nearly all of them, so PCIe, NVMe and the root filesystem all
come up that much later.

Commit c7fc690da6f1 ("firmware: tegra: bpmp: Allow BPMP debugfs population
to be skipped") lets a system that doesn't need the mirror turn it off with
tegra_bpmp.debugfs=0.  A system that does use it, for instance to set
clocks from userspace at boot, still pays the full cost.

Nothing in the kernel uses the mirror, and the probe already only logs an
error from it, so populate it from a workqueue instead.  It isn't an
async_schedule() job, since kernel_init() waits for all of those before it
starts init.  The workqueue is freezable, so a system suspend waits for the
mirror instead of failing its requests, and the mirrors of a system with
several BPMPs are populated one at a time, as their probes did, so that one
failing can't remove the shared root while another is filling it in.

The mirror is filled in as .debug-populating and renamed to debug once it
is complete, so userspace never sees part of it.  Userspace that needs it
right after boot can wait for debug to appear: the rename sends inotify an
IN_MOVED_TO event on the directory above it.  With systemd, a path unit can
start the service that uses the mirror once it is there:

	# bpmp-debugfs.path
	[Path]
	PathExists=/sys/kernel/debug/bpmp/debug
	Unit=bpmp-clocks.service

	[Install]
	WantedBy=paths.target

A Type=oneshot bpmp-clocks.service needs RemainAfterExit=yes, or the path
unit starts it again each time it exits.

Measured with initcall_debug on an Orin Nano running 6.18.39 with
NVIDIA's patches: with an earlier L4T release, the probe returns after
1.2 seconds instead of 6.36.  With the BPMP firmware from L4T 39.2.1 it
returns after 0.51 seconds, and the mirror is complete and renamed 2.2
seconds later, about 1.3 seconds after init has started.

Assisted-by: LLM
Signed-off-by: Austin Schuh <austin.linux@gmail.com>
---

Notes:
    Changes in v2 (thanks, Mikko):
     - Embed the work_struct in struct tegra_bpmp instead of allocating it,
       and cancel it with cancel_work_sync() from a devm action.
     - Fill the mirror in as .debug-populating and rename it to debug once
       it is complete, so userspace never sees part of it, and can wait for
       it by waiting for debug to appear.
     - Queue the work on system_freezable_wq, so a system suspend waits for
       it instead of failing its BPMP requests with -EAGAIN and tearing the
       mirror down.
     - Hold bpmp_debugfs_root_lock across the whole population, so two BPMPs
       are populated one at a time, as their probes did, and one failing
       can't remove the shared root while the other is creating its
       directory in it.
     - Move the work into bpmp-debugfs.c; tegra_bpmp_probe() is unchanged.
    
    Based on tegra/for-7.4/firmware (1bbcf4a19f14).  Built for arm64, W=1,
    no new warnings.  Booted on an Orin Nano with the same change on 6.18.39
    with NVIDIA's patches: the mirror appears complete, with all 10,831
    entries, and reads correctly.  With a 20 second delay added before the
    rename, the path unit from the commit message started its service 3 ms
    after the rename, and the service saw all 10,831 entries.
    
    v1: https://lore.kernel.org/r/20261017065012.3063980-1-austin.linux@gmail.com

 drivers/firmware/tegra/bpmp-debugfs.c | 67 ++++++++++++++++++++++++---
 include/soc/tegra/bpmp.h              |  2 +
 2 files changed, 62 insertions(+), 7 deletions(-)

diff --git a/drivers/firmware/tegra/bpmp-debugfs.c b/drivers/firmware/tegra/bpmp-debugfs.c
index 2efc48b522ac..bfc13e990eb2 100644
--- a/drivers/firmware/tegra/bpmp-debugfs.c
+++ b/drivers/firmware/tegra/bpmp-debugfs.c
@@ -782,27 +782,30 @@ MODULE_PARM_DESC(debugfs,
 
 static struct dentry *bpmp_debugfs_get_root(void)
 {
-	guard(mutex)(&bpmp_debugfs_root_lock);
+	lockdep_assert_held(&bpmp_debugfs_root_lock);
 	if (!bpmp_debugfs_root)
 		bpmp_debugfs_root = debugfs_create_dir("bpmp", NULL);
 	return bpmp_debugfs_root;
 }
 
-int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp)
+static int tegra_bpmp_populate_debugfs(struct tegra_bpmp *bpmp)
 {
 	struct dentry *root, *d;
 	char name[32];
 	bool inband;
 	int err;
 
-	if (!bpmp_debugfs_enabled)
-		return 0;
-
 	inband = tegra_bpmp_mrq_is_supported(bpmp, MRQ_DEBUG);
 
 	if (!inband && !tegra_bpmp_mrq_is_supported(bpmp, MRQ_DEBUGFS))
 		return 0;
 
+	/*
+	 * Populate one BPMP at a time, so that one failing can't remove the
+	 * shared root while another is creating its directory in it.
+	 */
+	guard(mutex)(&bpmp_debugfs_root_lock);
+
 	root = bpmp_debugfs_get_root();
 	if (IS_ERR(root))
 		return PTR_ERR(root);
@@ -818,7 +821,11 @@ int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp)
 		}
 	}
 
-	bpmp->debugfs_mirror = debugfs_create_dir("debug", d);
+	/*
+	 * Fill the mirror in under another name and rename it to "debug"
+	 * once it is complete, so that userspace never sees part of it.
+	 */
+	bpmp->debugfs_mirror = debugfs_create_dir(".debug-populating", d);
 	if (IS_ERR(bpmp->debugfs_mirror)) {
 		err = PTR_ERR(bpmp->debugfs_mirror);
 		goto out;
@@ -830,12 +837,14 @@ int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp)
 	else
 		err = bpmp_populate_debugfs_shmem(bpmp);
 
+	if (!err)
+		err = debugfs_change_name(bpmp->debugfs_mirror, "debug");
+
 out:
 	if (err < 0) {
 		if (!IS_ERR(d))
 			debugfs_remove_recursive(d);
 
-		guard(mutex)(&bpmp_debugfs_root_lock);
 		if (root == d) {
 			bpmp_debugfs_root = NULL;
 		} else if (simple_empty(root)) {
@@ -846,3 +855,47 @@ int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp)
 
 	return err;
 }
+
+static void tegra_bpmp_debugfs_work(struct work_struct *work)
+{
+	struct tegra_bpmp *bpmp =
+		container_of(work, struct tegra_bpmp, debugfs_work);
+	int err;
+
+	err = tegra_bpmp_populate_debugfs(bpmp);
+	if (err < 0)
+		dev_err(bpmp->dev, "debugfs initialization failed: %d\n", err);
+}
+
+static void tegra_bpmp_cancel_debugfs(void *data)
+{
+	struct tegra_bpmp *bpmp = data;
+
+	cancel_work_sync(&bpmp->debugfs_work);
+}
+
+int tegra_bpmp_init_debugfs(struct tegra_bpmp *bpmp)
+{
+	int err;
+
+	if (!bpmp_debugfs_enabled)
+		return 0;
+
+	INIT_WORK(&bpmp->debugfs_work, tegra_bpmp_debugfs_work);
+
+	err = devm_add_action_or_reset(bpmp->dev, tegra_bpmp_cancel_debugfs,
+				       bpmp);
+	if (err < 0)
+		return err;
+
+	/*
+	 * Mirroring the BPMP's debugfs tree takes several requests per
+	 * directory, which on Tegra234 adds up to seconds.  Nothing else
+	 * waits for it, so do it in the background rather than hold up every
+	 * device that depends on the BPMP.  The workqueue is freezable so
+	 * that system suspend waits for it instead of failing its requests.
+	 */
+	queue_work(system_freezable_wq, &bpmp->debugfs_work);
+
+	return 0;
+}
diff --git a/include/soc/tegra/bpmp.h b/include/soc/tegra/bpmp.h
index f55db918ccb1..f961c265326c 100644
--- a/include/soc/tegra/bpmp.h
+++ b/include/soc/tegra/bpmp.h
@@ -12,6 +12,7 @@
 #include <linux/reset-controller.h>
 #include <linux/semaphore.h>
 #include <linux/types.h>
+#include <linux/workqueue.h>
 
 #include <soc/tegra/bpmp-abi.h>
 
@@ -103,6 +104,7 @@ struct tegra_bpmp {
 
 #ifdef CONFIG_DEBUG_FS
 	struct dentry *debugfs_mirror;
+	struct work_struct debugfs_work;
 #endif
 
 	bool suspended;

base-commit: 1bbcf4a19f1495f769b69e9569d8c1e81df074f5
-- 
2.47.3

Re: [PATCH] firmware: tegra: bpmp: populate debugfs in the background

[Austin Schuh]

On Fri, Oct 23, 2026 at 1:16 AM Mikko Perttunen <mperttunen@nvidia.com> wrote:
> Sashiko reports some hard to trigger but worth checking concurrency
> issues.

I couldn't find Sashiko's report, so I went looking myself and found
two, both fixed in v2:

 - The work could run across a system suspend.  Once suspend_noirq sets
   bpmp->suspended, its requests fail with -EAGAIN, and the error path
   removes the whole mirror.  v2 queues it on system_freezable_wq, so
   suspend waits for it.

 - With more than one BPMP, the populations used to run one after the
   other in their probes, and now run at once.  One failing could remove
   the shared root while the other was creating its directory in it.  v2
   holds bpmp_debugfs_root_lock across the whole population.

If you've got a link to Sashiko's report, that would be wonderful.

> This does seem a good idea, but I am a bit concerned about the change to
> what userspace sees. Ideally we'd be able to have userspace stall if the
> debugfs population was still in progress. If that's not possible, at the
> very least userspace should have a way to voluntarily wait for the full
> debugfs to be available.

debugfs has no way to stall a lookup, so v2 fills the mirror in as
.debug-populating and renames it to debug once it is complete.
Userspace either finds no debug directory or a complete one, and can
wait for it by waiting for debug to appear: polling, or inotify on the
parent, since the rename sends IN_MOVED_TO.  On the Orin Nano the rename
lands about 1.3 seconds after the initramfs starts, and before systemd
starts on the root filesystem.

If you'd rather have something explicit as well, I can add a file in
bpmp/ whose read blocks until population is done and returns its result.

> Just inline the work_struct inside tegra_bpmp. In any case the work
> should be synchronously cancelled if the bpmp driver instance is going
> to go away (not that that will happen, but good practice).

Done in v2: the work_struct is in struct tegra_bpmp, and a devm action
calls cancel_work_sync().

Thanks for the review!

Austin