[PATCH v4] scsi: libsas: Fix SMP IO deadlock during HA resume

[PATCH v4] scsi: libsas: Fix SMP IO deadlock during HA resume

[Xingui Yang]

smp_execute_task_sg() calls pm_runtime_get_sync() on the host before
issuing an SMP command. When that command is itself issued from the
HA resume path, the get_sync() deadlocks: it waits for the ongoing
resume (the device is RPM_RESUMING), while the resume is blocked in
sas_drain_work() waiting for that same SMP IO to complete.

The deadlock needs an expander-attached SATA disk. During
sas_resume_ha() -> sas_drain_work(), DISCE_RESUME ->
sas_resume_sata() -> ata_sas_port_resume() requests ATA_EH_RESET,
and the hard reset for such a disk is done via SMP PHY CONTROL
(sas_ata_hard_reset() -> sas_phy_reset() -> sas_smp_phy_control() ->
smp_execute_task_sg()). Direct-attached SATA resets through
lldd_control_phy() and SSP devices use TMFs, so neither hits this.

Replace the get_sync()/put_sync() pair with
pm_runtime_get_noresume()/pm_runtime_put(). smp_execute_task_sg()
only needs to hold off autosuspend while the SMP is in flight, and it
must not try to resume the host: a sync resume issued from the HA
resume path itself is what deadlocks, and by the time sas_resume_ha()
runs, hw_init has already reinitialized the hardware, so the device
is accessible without one.

The usage reference is still required. Discovery work normally runs
inside an event worker's PM reference, taken at
sas_notify_port_event() notify time and held until the handler has
flushed the disco queue. sas_rediscover_ex_phy() however requeues
DISCE_REVALIDATE_DOMAIN from within the revalidation worker itself,
and flush_workqueue() does not wait for work items queued during
execution, so that chained revalidation runs with no outer PM
reference - without the get_noresume(), its SMP could race
autosuspend.

For the BSG path, sas_smp_handler() is the only caller which may
find the host autosuspended: expander SMP requests do not go through
any SCSI device request queue, so nothing else in that path holds
the host awake. Resume it there with pm_runtime_resume_and_get()
and check the result.

Fixes: 3dbbbf656b850 ("scsi: libsas: Fix HA resume deadlock and hisi_sas disk-wake race")
Signed-off-by: Xingui Yang <yangxingui@huawei.com>
---
Changes since v3:
- Move the host resume to sas_smp_handler(), the only caller which may
  find the host autosuspended, as suggested by John Garry
- Replace get_sync()/put_sync() with get_noresume()/put() in
  smp_execute_task_sg(): a blocking resume issued from the HA resume
  path itself is what deadlocks
- Drop the racy SAS_HA_RESUMING check

Changes since v2:
- Drop the reference with pm_runtime_put() instead of
  pm_runtime_put_noidle().

Changes since v1:
- Use pm_runtime_get_noresume()/put_noidle() during HA resume instead
  of skipping the PM reference entirely, so an in-flight SMP IO always
  keeps autosuspend away.
- Convert pm_runtime_get_sync() to pm_runtime_resume_and_get() and
  check the result (pre-existing issue flagged by sashiko).

 drivers/scsi/libsas/sas_expander.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
index 811c9eb4fef1..26c2099c28b9 100644
--- a/drivers/scsi/libsas/sas_expander.c
+++ b/drivers/scsi/libsas/sas_expander.c
@@ -62,7 +62,11 @@ static int smp_execute_task_sg(struct domain_device *dev,
 		to_sas_internal(dev->port->ha->shost->transportt);
 	struct sas_ha_struct *ha = dev->port->ha;
 
-	pm_runtime_get_sync(ha->dev);
+	/*
+	 * Non-blocking: a sync resume here would deadlock against
+	 * sas_drain_work() during HA resume.
+	 */
+	pm_runtime_get_noresume(ha->dev);
 	mutex_lock(&dev->ex_dev.cmd_mutex);
 	for (retry = 0; retry < 3; retry++) {
 		if (test_bit(SAS_DEV_GONE, &dev->state)) {
@@ -135,7 +139,7 @@ static int smp_execute_task_sg(struct domain_device *dev,
 		}
 	}
 	mutex_unlock(&dev->ex_dev.cmd_mutex);
-	pm_runtime_put_sync(ha->dev);
+	pm_runtime_put(ha->dev);
 
 	BUG_ON(retry == 3 && task != NULL);
 	sas_free_task(task);
@@ -2222,6 +2226,11 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
 		goto out;
 	}
 
+	/* The host may have autosuspended, resume it here. */
+	ret = pm_runtime_resume_and_get(dev->port->ha->dev);
+	if (ret)
+		goto out;
+
 	ret = smp_execute_task_sg(dev, job->request_payload.sg_list,
 			job->reply_payload.sg_list);
 	if (ret >= 0) {
@@ -2229,6 +2238,7 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
 		rcvlen = job->reply_payload.payload_len - ret;
 		ret = 0;
 	}
+	pm_runtime_put(dev->port->ha->dev);
 
 out:
 	bsg_job_done(job, ret, rcvlen);
-- 
2.43.0

Re: [PATCH v4] scsi: libsas: Fix SMP IO deadlock during HA resume

[John Garry]

On 9/28/26 05:02, Xingui Yang wrote:
> smp_execute_task_sg() calls pm_runtime_get_sync() on the host before
> issuing an SMP command. When that command is itself issued from the
> HA resume path, the get_sync() deadlocks: it waits for the ongoing
> resume (the device is RPM_RESUMING), while the resume is blocked in
> sas_drain_work() waiting for that same SMP IO to complete.
> 
> The deadlock needs an expander-attached SATA disk. 

What do you mean by "deadlock needs an expander-attached SATA disk?

> During
> sas_resume_ha() -> sas_drain_work(), DISCE_RESUME ->
> sas_resume_sata() -> ata_sas_port_resume() requests ATA_EH_RESET,
> and the hard reset for such a disk is done via SMP PHY CONTROL
> (sas_ata_hard_reset() -> sas_phy_reset() -> sas_smp_phy_control() ->
> smp_execute_task_sg()). Direct-attached SATA resets through
> lldd_control_phy() and SSP devices use TMFs, so neither hits this.
> 
> Replace the get_sync()/put_sync() pair with
> pm_runtime_get_noresume()/pm_runtime_put(). smp_execute_task_sg()
> only needs to hold off autosuspend while the SMP is in flight, and it
> must not try to resume the host: a sync resume issued from the HA
> resume path itself is what deadlocks, and by the time sas_resume_ha()
> runs, hw_init has already reinitialized the hardware, so the device
> is accessible without one.
> 
> The usage reference is still required. 

Do you mean that usage reference from pm_runtime_get_noresume() is still 
required?

> Discovery work normally runs
> inside an event worker's PM reference, taken at
> sas_notify_port_event() notify time and held until the handler has
> flushed the disco queue. sas_rediscover_ex_phy() however requeues
> DISCE_REVALIDATE_DOMAIN from within the revalidation worker itself,
> and flush_workqueue() does not wait for work items queued during
> execution, so that chained revalidation runs with no outer PM
> reference - without the get_noresume(), its SMP could race
> autosuspend.
> 
> For the BSG path, sas_smp_handler() is the only caller which may
> find the host autosuspended: expander SMP requests do not go through
> any SCSI device request queue, so nothing else in that path holds
> the host awake. Resume it there with pm_runtime_resume_and_get()
> and check the result.
> 
> Fixes: 3dbbbf656b850 ("scsi: libsas: Fix HA resume deadlock and hisi_sas disk-wake race")
> Signed-off-by: Xingui Yang <yangxingui@huawei.com>

Question: do you have a (non-hisi_sas) SAS HBA card whose driver uses 
libsas? pm8001 would be such an example. It would be nice to verify that 
all these and other non-rpm libsas changes does cause regression there.

> ---
> Changes since v3:
> - Move the host resume to sas_smp_handler(), the only caller which may
>    find the host autosuspended, as suggested by John Garry
> - Replace get_sync()/put_sync() with get_noresume()/put() in
>    smp_execute_task_sg(): a blocking resume issued from the HA resume
>    path itself is what deadlocks
> - Drop the racy SAS_HA_RESUMING check
> 
> Changes since v2:
> - Drop the reference with pm_runtime_put() instead of
>    pm_runtime_put_noidle().
> 
> Changes since v1:
> - Use pm_runtime_get_noresume()/put_noidle() during HA resume instead
>    of skipping the PM reference entirely, so an in-flight SMP IO always
>    keeps autosuspend away.
> - Convert pm_runtime_get_sync() to pm_runtime_resume_and_get() and
>    check the result (pre-existing issue flagged by sashiko).
> 
>   drivers/scsi/libsas/sas_expander.c | 14 ++++++++++++--
>   1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index 811c9eb4fef1..26c2099c28b9 100644
> --- a/drivers/scsi/libsas/sas_expander.c
> +++ b/drivers/scsi/libsas/sas_expander.c
> @@ -62,7 +62,11 @@ static int smp_execute_task_sg(struct domain_device *dev,
>   		to_sas_internal(dev->port->ha->shost->transportt);
>   	struct sas_ha_struct *ha = dev->port->ha;
>   
> -	pm_runtime_get_sync(ha->dev);
> +	/*
> +	 * Non-blocking: a sync resume here would deadlock against
> +	 * sas_drain_work() during HA resume.
> +	 */
> +	pm_runtime_get_noresume(ha->dev);
>   	mutex_lock(&dev->ex_dev.cmd_mutex);
>   	for (retry = 0; retry < 3; retry++) {
>   		if (test_bit(SAS_DEV_GONE, &dev->state)) {
> @@ -135,7 +139,7 @@ static int smp_execute_task_sg(struct domain_device *dev,
>   		}
>   	}
>   	mutex_unlock(&dev->ex_dev.cmd_mutex);
> -	pm_runtime_put_sync(ha->dev);
> +	pm_runtime_put(ha->dev);
>   
>   	BUG_ON(retry == 3 && task != NULL);
>   	sas_free_task(task);
> @@ -2222,6 +2226,11 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
>   		goto out;
>   	}
>   
> +	/* The host may have autosuspended, resume it here. */
> +	ret = pm_runtime_resume_and_get(dev->port->ha->dev);
> +	if (ret)
> +		goto out;
> +
>   	ret = smp_execute_task_sg(dev, job->request_payload.sg_list,
>   			job->reply_payload.sg_list);
>   	if (ret >= 0) {
> @@ -2229,6 +2238,7 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
>   		rcvlen = job->reply_payload.payload_len - ret;
>   		ret = 0;
>   	}
> +	pm_runtime_put(dev->port->ha->dev);
>   
>   out:
>   	bsg_job_done(job, ret, rcvlen);

Re: [PATCH v4] scsi: libsas: Fix SMP IO deadlock during HA resume

[yangxingui]

Hi, John

On 2026/9/30 18:40, John Garry wrote:
> On 9/28/26 05:02, Xingui Yang wrote:
>> smp_execute_task_sg() calls pm_runtime_get_sync() on the host before
>> issuing an SMP command. When that command is itself issued from the
>> HA resume path, the get_sync() deadlocks: it waits for the ongoing
>> resume (the device is RPM_RESUMING), while the resume is blocked in
>> sas_drain_work() waiting for that same SMP IO to complete.
>>
>> The deadlock needs an expander-attached SATA disk. 
> 
> What do you mean by "deadlock needs an expander-attached SATA disk?
Yes, that is the only configuration whose resume-time hard reset is
implemented as an SMP command. The disk's phy belongs to the
expander, so the ATA EH hard reset is an SMP PHY CONTROL sent to the
expander (sas_ata_hard_reset() -> sas_phy_reset() ->
sas_smp_phy_control() -> smp_execute_task_sg()), a directly attached
SATA disk resets the HBA's own phy through lldd_control_phy(), and
SSP devices are reset through TMFs, so neither issues SMP on resume.
An SMP request from outside the resume path cannot construct the
cycle either: a BSG request arriving while a resume is in progress
blocks waiting for that resume (get_sync() before this patch,
resume_and_get() at the new call site) before it takes the expander
cmd_mutex, and the resume waits for nothing in the BSG context - it
can only be a victim of the deadlock, never a participant.

> 
>> During
>> sas_resume_ha() -> sas_drain_work(), DISCE_RESUME ->
>> sas_resume_sata() -> ata_sas_port_resume() requests ATA_EH_RESET,
>> and the hard reset for such a disk is done via SMP PHY CONTROL
>> (sas_ata_hard_reset() -> sas_phy_reset() -> sas_smp_phy_control() ->
>> smp_execute_task_sg()). Direct-attached SATA resets through
>> lldd_control_phy() and SSP devices use TMFs, so neither hits this.
>>
>> Replace the get_sync()/put_sync() pair with
>> pm_runtime_get_noresume()/pm_runtime_put(). smp_execute_task_sg()
>> only needs to hold off autosuspend while the SMP is in flight, and it
>> must not try to resume the host: a sync resume issued from the HA
>> resume path itself is what deadlocks, and by the time sas_resume_ha()
>> runs, hw_init has already reinitialized the hardware, so the device
>> is accessible without one.
>>
>> The usage reference is still required. 
> 
> Do you mean that usage reference from pm_runtime_get_noresume() is still 
> required?

Yes, the reference taken by the pm_runtime_get_noresume() which
replaces the get_sync(). It is not an artefact of the conversion:
the chained revalidation described in the next paragraph runs
outside the event workers' PM references, so without it its SMP
could race autosuspend.

>> Discovery work normally runs
>> inside an event worker's PM reference, taken at
>> sas_notify_port_event() notify time and held until the handler has
>> flushed the disco queue. sas_rediscover_ex_phy() however requeues
>> DISCE_REVALIDATE_DOMAIN from within the revalidation worker itself,
>> and flush_workqueue() does not wait for work items queued during
>> execution, so that chained revalidation runs with no outer PM
>> reference - without the get_noresume(), its SMP could race
>> autosuspend.
>>
>> For the BSG path, sas_smp_handler() is the only caller which may
>> find the host autosuspended: expander SMP requests do not go through
>> any SCSI device request queue, so nothing else in that path holds
>> the host awake. Resume it there with pm_runtime_resume_and_get()
>> and check the result.
>>
>> Fixes: 3dbbbf656b850 ("scsi: libsas: Fix HA resume deadlock and 
>> hisi_sas disk-wake race")
>> Signed-off-by: Xingui Yang <yangxingui@huawei.com>
> 
> Question: do you have a (non-hisi_sas) SAS HBA card whose driver uses 
> libsas? pm8001 would be such an example. It would be nice to verify that 
> all these and other non-rpm libsas changes does cause regression there.

We don't have a pm8001 card at hand. From analysis the patch is a
no-op for the LLDDs without runtime PM support: pm8001 and isci
only register system suspend/resume through SIMPLE_DEV_PM_OPS(), and
aic94xx and mvsas register no PM ops at all. For every PCI device
the PCI core forbids runtime PM by default and marks the device
runtime-active (pci_pm_init()), and enables runtime PM later
(pci_bus_add_device()), those devices are thus permanently RPM_ACTIVE
with the usage counter held, and can never runtime-suspend. In that
state get_noresume()/put() only moves the usage counter - exactly
what the get_sync()/put_sync() they replace did - and
pm_runtime_resume_and_get() in sas_smp_handler() succeeds (runtime
PM is enabled and the device is already active), so the new error
check never rejects a request on them.

The only observable difference would be userspace enabling runtime
PM (power/control = auto) on a driver which does not support it, and
that configuration is broken independently of this patch.

Conversely, should an LLDD gain runtime PM support later, it would
need this patch: the deadlock is in the shared resume path - any
runtime-resuming LLDD with an expander-attached SATA disk in the
domain hits it identically - and the reference/resume split
completes the generic protection model, while the LLDD-side
obligations (device links, hw reinit before sas_resume_ha()) are
unchanged.


Thanks,
Xingui