* [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
@ 2026-11-18 23:09 ` Jeff Layton
2026-11-18 23:09 ` [PATCH v3 02/11] scsi: mpt3sas: Hold event references while cleaning up the event queue Jeff Layton
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:09 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
When a topology change event finds a busy device,
_scsih_fw_event_requeue() retries the event. It queues the event's
delayed_work on firmware_event_thread, with a one-second delay.
_scsih_fw_event_cleanup_queue() cancels only the event's work, not its
delayed_work. Thus the timer stays armed. Then scsih_remove() destroys
the workqueue. After the workqueue is freed, the timer queues
_firmware_event_work_delayed() on it:
BUG: KASAN: slab-use-after-free in __queue_work+0xa12/0xb50
Read of size 4 at addr ffff88814fefd1c0 by task swapper/10/0
__queue_work
call_timer_fn
__run_timer_base
...
Allocated by task 11:
__alloc_workqueue
alloc_workqueue_noprof
_scsih_probe [mpt3sas]
Freed by task 0:
kfree
rcu_core
...
workqueue: cannot queue _firmware_event_work_delayed [mpt3sas] on wq fw_event_mpt3sas0
The cleanup does not wait for a running topology change event. If such
an event finds a busy device, it requeues itself. This can occur after
the cleanup passes the event.
Fix this:
- In the cleanup, cancel the delayed_work of a requeued event.
- Do not requeue an event while the cleanup runs or while the driver
removes the host. Do this check under fw_event_lock, which the
cleanup also takes.
- Instead of a requeue, drop the reference that the work held.
Previously, this reference leaked when firmware_event_thread was
NULL.
- Also drop the reference if the delayed_work cannot be queued.
Cleanups can overlap. Thus one cleanup can clear fw_events_cleanup
while another cleanup cancels the delayed_work. The cancel disables
the delayed_work.
- Initialize the delayed_work when the driver allocates the event, not
at the first requeue. Thus a cleanup never cancels the delayed_work
before it is initialized.
Fixes: ad5957193107 ("scsi: mpt3sas: Add firmware event requeue support for busy devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 49 ++++++++++++++++++++++++++++--------
1 file changed, 38 insertions(+), 11 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 04255286a2a6..545913123d01 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -320,6 +320,11 @@ static struct fw_event_work *alloc_fw_event_work(int len)
return NULL;
kref_init(&fw_event->refcount);
+ /*
+ * Initialize the delayed_work up front: a cleanup may cancel it
+ * while the event is being requeued.
+ */
+ INIT_DELAYED_WORK(&fw_event->delayed_work, _firmware_event_work_delayed);
return fw_event;
}
@@ -3696,6 +3701,9 @@ _scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
*
* Context: This function will acquire ioc->fw_event_lock.
*
+ * The reference held by the work that ran @fw_event passes to the delayed
+ * work. If the event can't be requeued, that reference is dropped.
+ *
* Return nothing.
*/
static void
@@ -3704,20 +3712,31 @@ _scsih_fw_event_requeue(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
{
unsigned long flags;
- if (ioc->firmware_event_thread == NULL)
- return;
-
spin_lock_irqsave(&ioc->fw_event_lock, flags);
+ /*
+ * Don't rearm the event while the queue is being flushed or the
+ * host is going away. _scsih_fw_event_cleanup_queue() may already
+ * be past it, and the timer could then fire after the workqueue
+ * has been destroyed.
+ */
+ if (!ioc->firmware_event_thread || ioc->remove_host ||
+ ioc->fw_events_cleanup)
+ goto drop;
+ fw_event->delayed_work_active = 1;
+ /*
+ * Cleanups can overlap, and one can clear fw_events_cleanup while
+ * another is cancelling, and so disabling, the delayed work.
+ */
+ if (!queue_delayed_work(ioc->firmware_event_thread,
+ &fw_event->delayed_work, msecs_to_jiffies(delay)))
+ goto drop;
fw_event_work_get(fw_event);
list_add_tail(&fw_event->list, &ioc->fw_event_list);
- if (!fw_event->delayed_work_active) {
- fw_event->delayed_work_active = 1;
- INIT_DELAYED_WORK(&fw_event->delayed_work,
- _firmware_event_work_delayed);
- }
- queue_delayed_work(ioc->firmware_event_thread, &fw_event->delayed_work,
- msecs_to_jiffies(delay));
spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+ return;
+drop:
+ spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+ fw_event_work_put(fw_event);
}
/**
@@ -3869,9 +3888,17 @@ _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
*
* If it did execute, we wait for it to finish, and the put will
* happen from _firmware_event_work()
+ *
+ * An event requeued by _scsih_fw_event_requeue() is pending
+ * on its delayed_work instead, and its timer must not fire
+ * after the workqueue is destroyed.
*/
- if (cancel_work_sync(&fw_event->work))
+ if (fw_event->delayed_work_active) {
+ if (cancel_delayed_work_sync(&fw_event->delayed_work))
+ fw_event_work_put(fw_event);
+ } else if (cancel_work_sync(&fw_event->work)) {
fw_event_work_put(fw_event);
+ }
}
ioc->fw_events_cleanup = 0;
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 02/11] scsi: mpt3sas: Hold event references while cleaning up the event queue
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
2026-11-18 23:09 ` [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown Jeff Layton
@ 2026-11-18 23:09 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 03/11] scsi: mpt3sas: Requeue topology events only for busy devices Jeff Layton
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:09 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
_scsih_fw_event_cleanup_queue() uses events without a reference to
them. But the work of an event frees the event when the work ends. This
causes these problems:
- dequeue_next_fw_event() drops the list's reference before the
cleanup cancels the event's work. If the work runs and ends between
these two steps, the cleanup calls cancel_work_sync() on a freed
event.
- The cleanup holds no reference to an event that it finds through
ioc->current_event. The worker drops its reference before it clears
current_event. Thus the cleanup can find a freed event there, and
set ->ignore on it or read its ->event. For a running
MPT3SAS_REMOVE_UNRESPONDING_DEVICES event, cancel_work_sync() waits
for the work. The work frees the event, and then cancel_work_sync()
enables the work again in freed memory.
- To identify a running event, the cleanup only compares the event
with current_event. It waits for the work of all other events that
it finds. But the cleanup also clears current_event. Also, cleanups
can overlap, because _base_fault_reset_work() calls a cleanup
without reset_in_progress_mutex. Thus a cleanup that holds
reset_in_progress_mutex can wait for a running event. That event is
about to take the same mutex for a hard reset.
Originally, commit 146b16c8071f ("mpt3sas: Refcount fw_events and fix
unsafe list usage") kept the list's reference until after the cancel.
Then commit 9e73ed2e4cf5 ("scsi: mpt3sas: Cancel the running work
during host reset") added the current_event path. Events on this path
had no reference to drop. To fix the resulting underflow, commit
991df3dd5144 ("scsi: mpt3sas: Fix use-after-free warning") dropped the
list's reference early.
Fix the reference handling:
- Let dequeue_next_fw_event() pass the list's reference to the cleanup
again. The cleanup drops the reference after the cancel.
- Set and clear current_event under fw_event_lock. Clear it before the
worker drops its reference. Then the cleanup can take a reference to
the current event under the lock.
- When the worker requeues the event, clear current_event in
_scsih_fw_event_requeue(). Do this under fw_event_lock, together
with the requeue or the drop of the event. _scsih_fw_event_requeue()
can now drop the reference.
- Also do the check for events to clean up under the lock.
Thus the cleanup finds the event either as the current event or on
fw_event_list. It cannot miss the event between the two.
When the worker starts an event, remove the event from fw_event_list
and make it the current event. Do both in one step under
fw_event_lock. If the event is no longer on the list, a cleanup removed
it, so drop it. Thus an event that the cleanup removes from the list
never runs. The cancel of its work then never waits for a running
event.
Do not change current_event in the cleanup. Let the worker manage it.
Wait for the running event only if it is
MPT3SAS_REMOVE_UNRESPONDING_DEVICES, as before.
Only the broadcast primitive handler clears broadcast_aen_busy. If the
cleanup drops a broadcast primitive event, broadcast_aen_busy stays
set. Later broadcast primitives then only count as pending. An event
that the cleanup cancels before it runs already causes this. With this
change, an event that the worker drops as it starts also causes it.
When the cleanup drops such an event, clear broadcast_aen_busy and the
stale pending count.
Fixes: 9e73ed2e4cf5 ("scsi: mpt3sas: Cancel the running work during host reset")
Fixes: 991df3dd5144 ("scsi: mpt3sas: Fix use-after-free warning")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 211 ++++++++++++++++++++++++-----------
1 file changed, 148 insertions(+), 63 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 545913123d01..b455b7e5d8f3 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -3672,25 +3672,37 @@ _scsih_fw_event_add(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
}
/**
- * _scsih_fw_event_del_from_list - delete fw_event from the list
+ * _scsih_fw_event_start - start running an event
* @ioc: per adapter object
* @fw_event: object describing the event
* Context: This function will acquire ioc->fw_event_lock.
*
- * If the fw_event is on the fw_event_list, remove it and do a put.
+ * An event is on fw_event_list from when it is queued until it starts.
+ * Take @fw_event off the list, dropping the list's reference, and make it
+ * the current event, in one step under fw_event_lock. If it is no longer
+ * on the list, _scsih_fw_event_cleanup_queue() took it off, and may be
+ * waiting for its work to end.
+ *
+ * Return: true if @fw_event should run, false if it should be dropped.
*/
-static void
-_scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
- *fw_event)
+static bool
+_scsih_fw_event_start(struct MPT3SAS_ADAPTER *ioc,
+ struct fw_event_work *fw_event)
{
unsigned long flags;
+ bool start = false;
spin_lock_irqsave(&ioc->fw_event_lock, flags);
if (!list_empty(&fw_event->list)) {
list_del_init(&fw_event->list);
+ /* the work still holds a reference */
fw_event_work_put(fw_event);
+ ioc->current_event = fw_event;
+ start = true;
}
spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+
+ return start;
}
/**
@@ -3704,6 +3716,12 @@ _scsih_fw_event_del_from_list(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
* The reference held by the work that ran @fw_event passes to the delayed
* work. If the event can't be requeued, that reference is dropped.
*
+ * The worker is done with @fw_event, so this also clears
+ * ioc->current_event. That happens under fw_event_lock together with
+ * requeueing or dropping the event, so that
+ * _scsih_fw_event_cleanup_queue() finds the event either as the current
+ * one or on fw_event_list.
+ *
* Return nothing.
*/
static void
@@ -3713,6 +3731,8 @@ _scsih_fw_event_requeue(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work
unsigned long flags;
spin_lock_irqsave(&ioc->fw_event_lock, flags);
+ if (ioc->current_event == fw_event)
+ ioc->current_event = NULL;
/*
* Don't rearm the event while the queue is being flushed or the
* host is going away. _scsih_fw_event_cleanup_queue() may already
@@ -3800,6 +3820,10 @@ mpt3sas_port_enable_complete(struct MPT3SAS_ADAPTER *ioc)
fw_event_work_put(fw_event);
}
+/*
+ * Remove the first event from fw_event_list and return it, with the
+ * reference the list held. The caller puts it.
+ */
static struct fw_event_work *dequeue_next_fw_event(struct MPT3SAS_ADAPTER *ioc)
{
unsigned long flags;
@@ -3810,13 +3834,47 @@ static struct fw_event_work *dequeue_next_fw_event(struct MPT3SAS_ADAPTER *ioc)
fw_event = list_first_entry(&ioc->fw_event_list,
struct fw_event_work, list);
list_del_init(&fw_event->list);
- fw_event_work_put(fw_event);
}
spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
return fw_event;
}
+/*
+ * ioc->current_event is set and cleared under fw_event_lock, and cleared
+ * before the worker drops its reference to the event. While it is set
+ * under the lock, the event is live.
+ */
+static void
+_scsih_set_current_event(struct MPT3SAS_ADAPTER *ioc,
+ struct fw_event_work *fw_event)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&ioc->fw_event_lock, flags);
+ ioc->current_event = fw_event;
+ spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+}
+
+/*
+ * Return the event the worker is running, with a reference that the
+ * caller puts, or NULL if there is none.
+ */
+static struct fw_event_work *
+_scsih_get_current_event(struct MPT3SAS_ADAPTER *ioc)
+{
+ struct fw_event_work *fw_event;
+ unsigned long flags;
+
+ spin_lock_irqsave(&ioc->fw_event_lock, flags);
+ fw_event = ioc->current_event;
+ if (fw_event)
+ fw_event_work_get(fw_event);
+ spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+
+ return fw_event;
+}
+
/**
* _scsih_fw_event_cleanup_queue - cleanup event queue
* @ioc: per adapter object
@@ -3830,10 +3888,13 @@ static void
_scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
{
struct fw_event_work *fw_event;
+ unsigned long flags;
+ bool idle;
- if ((list_empty(&ioc->fw_event_list) && !ioc->current_event) ||
- !ioc->firmware_event_thread)
+ if (!ioc->firmware_event_thread)
return;
+ spin_lock_irqsave(&ioc->fw_event_lock, flags);
+ idle = list_empty(&ioc->fw_event_list) && !ioc->current_event;
/*
* Set current running event as ignore, so that
* current running event will exit quickly.
@@ -3842,64 +3903,87 @@ _scsih_fw_event_cleanup_queue(struct MPT3SAS_ADAPTER *ioc)
*/
if (ioc->shost_recovery && ioc->current_event)
ioc->current_event->ignore = 1;
+ spin_unlock_irqrestore(&ioc->fw_event_lock, flags);
+ if (idle)
+ return;
+ /*
+ * Hold a reference to each event while it is handled here: its work
+ * can drop the last one as soon as it ends.
+ *
+ * Don't wait for the running event, other than
+ * MPT3SAS_REMOVE_UNRESPONDING_DEVICES, which stops waiting for the
+ * host reset when fw_events_cleanup is set. Another event may be
+ * issuing the hard reset this cleanup is part of, and cleanups can
+ * overlap, so whether an event is running doesn't depend on another
+ * cleanup. An event taken off fw_event_list here doesn't run, see
+ * _scsih_fw_event_start(), so cancelling it never waits for one.
+ *
+ * Orginal logic of cleaning the current_event is added
+ * for handling the back to back host reset issued by the user.
+ * i.e. during back to back host reset, driver use to process
+ * the two instances of MPT3SAS_REMOVE_UNRESPONDING_DEVICES
+ * event back to back and this made the drives to unregister
+ * the devices from SML.
+ */
ioc->fw_events_cleanup = 1;
- while ((fw_event = dequeue_next_fw_event(ioc)) ||
- (fw_event = ioc->current_event)) {
+ for (;;) {
+ while ((fw_event = dequeue_next_fw_event(ioc))) {
+ /*
+ * Driver has to clear ioc->start_scan flag when
+ * it is cleaning up MPT3SAS_PORT_ENABLE_COMPLETE,
+ * otherwise scsi_scan_host() API waits for the
+ * 5 minute timer to expire. If we exit from
+ * scsi_scan_host() early then we can issue the
+ * new port enable request as part of current diag
+ * reset.
+ */
+ if (fw_event->event == MPT3SAS_PORT_ENABLE_COMPLETE) {
+ ioc->port_enable_cmds.status |= MPT3_CMD_RESET;
+ ioc->start_scan = 0;
+ }
- /*
- * Don't call cancel_work_sync() for current_event
- * other than MPT3SAS_REMOVE_UNRESPONDING_DEVICES;
- * otherwise we may observe deadlock if current
- * hard reset issued as part of processing the current_event.
- *
- * Orginal logic of cleaning the current_event is added
- * for handling the back to back host reset issued by the user.
- * i.e. during back to back host reset, driver use to process
- * the two instances of MPT3SAS_REMOVE_UNRESPONDING_DEVICES
- * event back to back and this made the drives to unregister
- * the devices from SML.
- */
+ /*
+ * If this returns 1, the event never ran, and the
+ * reference its work held needs a put. Otherwise
+ * its work drops that reference.
+ *
+ * An event requeued by _scsih_fw_event_requeue() is
+ * pending on its delayed_work instead, and its timer
+ * must not fire after the workqueue is destroyed.
+ */
+ if (fw_event->delayed_work_active) {
+ if (cancel_delayed_work_sync(&fw_event->delayed_work))
+ fw_event_work_put(fw_event);
+ } else if (cancel_work_sync(&fw_event->work)) {
+ fw_event_work_put(fw_event);
+ }
- if (fw_event == ioc->current_event &&
- ioc->current_event->event !=
- MPT3SAS_REMOVE_UNRESPONDING_DEVICES) {
- ioc->current_event = NULL;
- continue;
- }
+ /*
+ * The broadcast callback queues one broadcast
+ * primitive event at a time, and only its handler,
+ * which won't run now, clears broadcast_aen_busy.
+ */
+ if (fw_event->event ==
+ MPI2_EVENT_SAS_BROADCAST_PRIMITIVE) {
+ ioc->broadcast_aen_busy = 0;
+ ioc->broadcast_aen_pending = 0;
+ }
- /*
- * Driver has to clear ioc->start_scan flag when
- * it is cleaning up MPT3SAS_PORT_ENABLE_COMPLETE,
- * otherwise scsi_scan_host() API waits for the
- * 5 minute timer to expire. If we exit from
- * scsi_scan_host() early then we can issue the
- * new port enable request as part of current diag reset.
- */
- if (fw_event->event == MPT3SAS_PORT_ENABLE_COMPLETE) {
- ioc->port_enable_cmds.status |= MPT3_CMD_RESET;
- ioc->start_scan = 0;
+ fw_event_work_put(fw_event);
}
- /*
- * Wait on the fw_event to complete. If this returns 1, then
- * the event was never executed, and we need a put for the
- * reference the work had on the fw_event.
- *
- * If it did execute, we wait for it to finish, and the put will
- * happen from _firmware_event_work()
- *
- * An event requeued by _scsih_fw_event_requeue() is pending
- * on its delayed_work instead, and its timer must not fire
- * after the workqueue is destroyed.
- */
- if (fw_event->delayed_work_active) {
- if (cancel_delayed_work_sync(&fw_event->delayed_work))
- fw_event_work_put(fw_event);
- } else if (cancel_work_sync(&fw_event->work)) {
+ fw_event = _scsih_get_current_event(ioc);
+ if (!fw_event)
+ break;
+ if (fw_event->event != MPT3SAS_REMOVE_UNRESPONDING_DEVICES) {
fw_event_work_put(fw_event);
+ break;
}
-
+ /* wait for it, then flush what was queued meanwhile */
+ if (cancel_work_sync(&fw_event->work))
+ fw_event_work_put(fw_event);
+ fw_event_work_put(fw_event);
}
ioc->fw_events_cleanup = 0;
}
@@ -11985,13 +12069,16 @@ mpt3sas_scsih_reset_done_handler(struct MPT3SAS_ADAPTER *ioc)
static void
_mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
{
- ioc->current_event = fw_event;
- _scsih_fw_event_del_from_list(ioc, fw_event);
+ if (!_scsih_fw_event_start(ioc, fw_event)) {
+ /* _scsih_fw_event_cleanup_queue() took it off the list */
+ fw_event_work_put(fw_event);
+ return;
+ }
/* the queue is being flushed so ignore this event */
if (ioc->remove_host || ioc->pci_error_recovery) {
+ _scsih_set_current_event(ioc, NULL);
fw_event_work_put(fw_event);
- ioc->current_event = NULL;
return;
}
@@ -12046,7 +12133,6 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
case MPI2_EVENT_SAS_TOPOLOGY_CHANGE_LIST:
if (_scsih_sas_topology_change_event(ioc, fw_event)) {
_scsih_fw_event_requeue(ioc, fw_event, 1000);
- ioc->current_event = NULL;
return;
}
break;
@@ -12090,14 +12176,13 @@ _mpt3sas_fw_work(struct MPT3SAS_ADAPTER *ioc, struct fw_event_work *fw_event)
case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
if (_scsih_pcie_topology_change_event(ioc, fw_event)) {
_scsih_fw_event_requeue(ioc, fw_event, 1000);
- ioc->current_event = NULL;
return;
}
break;
}
out:
+ _scsih_set_current_event(ioc, NULL);
fw_event_work_put(fw_event);
- ioc->current_event = NULL;
}
/**
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 03/11] scsi: mpt3sas: Requeue topology events only for busy devices
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
2026-11-18 23:09 ` [PATCH v3 01/11] scsi: mpt3sas: Don't leave requeued firmware events pending at teardown Jeff Layton
2026-11-18 23:09 ` [PATCH v3 02/11] scsi: mpt3sas: Hold event references while cleaning up the event queue Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 04/11] scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait() Jeff Layton
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
Since commit ad5957193107 ("scsi: mpt3sas: Add firmware event requeue
support for busy devices"), _scsih_sas_topology_change_event() requeues
the event whenever _scsih_add_device() returns non-zero. It retries the
event every second. But _scsih_add_device() returns 1 only for a busy
device. It returns -1 when it cannot add the device at all. This occurs
when:
- the driver cannot read the SAS Device Page 0 of the device
- the device is not an end device, or is not present
- discovery reported an access problem
- the device is already added
None of these conditions clear with time, and nothing limits the
number of retries. In a stress test, the driver retried the add of a
device that was not present 249 times.
A requeued event keeps the handle and phy number from its original
report. Meanwhile, the handle may go to another device, because a host
reset assigns all handles again. In one stress run, a requeued add
first ran after a host reset. At that time, its handle was not in use.
Thus the read of SAS Device Page 0 failed, and the driver requeued the
event. One second later, at the retry, a device that was returning on
another phy had that handle.
The retry then makes mpt3sas_transport_update_links() record the SAS
address of that device on the phy of the event. The add of the device
then makes _transport_sanity_check() delete the port of the drive on
that phy. Thus the driver removed the drive, although the drive never
disappeared.
When _scsih_add_device() cannot add the device, return 0, as
_scsih_pcie_add_device() does. Then the driver retries only a busy
device. Some failures of the page read do clear with time:
- -EAGAIN, when no internal command is free or after a reset
- an IOC status of BUSY or INSUFFICIENT_RESOURCES
Continue to retry these failures, up to command_retry_count times, as
for a busy device. This change also prevents a spin on such a device
in the loops that call _scsih_add_device() after a host reset.
Fixes: ad5957193107 ("scsi: mpt3sas: Add firmware event requeue support for busy devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index b455b7e5d8f3..2d98fcb8e135 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -8509,7 +8509,9 @@ _scsih_check_device(struct MPT3SAS_ADAPTER *ioc,
*
* Creating end device object, stored in ioc->sas_device_list.
*
- * Return: 0 for success, non-zero for failure.
+ * Return: 1 if adding the device should be retried: it is busy, or
+ * reading its SAS Device Page 0 failed in a way that can clear with time.
+ * 0 otherwise.
*/
static int
_scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
@@ -8525,12 +8527,15 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
u32 device_info;
u8 connector_name[5];
u8 port_id;
+ int r;
- if ((mpt3sas_config_get_sas_device_pg0(ioc, &mpi_reply, &sas_device_pg0,
- MPI2_SAS_DEVICE_PGAD_FORM_HANDLE, handle))) {
+ r = mpt3sas_config_get_sas_device_pg0(ioc, &mpi_reply, &sas_device_pg0,
+ MPI2_SAS_DEVICE_PGAD_FORM_HANDLE, handle);
+ if (r) {
ioc_err(ioc, "failure at %s:%d/%s()!\n",
__FILE__, __LINE__, __func__);
- return -1;
+ /* e.g. no free internal command, or a reset intervened */
+ return r == -EAGAIN && retry_count < command_retry_count;
}
ioc_status = le16_to_cpu(mpi_reply.IOCStatus) &
@@ -8538,13 +8543,15 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
ioc_err(ioc, "failure at %s:%d/%s()!\n",
__FILE__, __LINE__, __func__);
- return -1;
+ return (ioc_status == MPI2_IOCSTATUS_BUSY ||
+ ioc_status == MPI2_IOCSTATUS_INSUFFICIENT_RESOURCES) &&
+ retry_count < command_retry_count;
}
/* check if this is end device */
device_info = le32_to_cpu(sas_device_pg0.DeviceInfo);
if (!(_scsih_is_end_device(device_info)))
- return -1;
+ return 0;
set_bit(handle, ioc->pend_os_device_add);
sas_address = le64_to_cpu(sas_device_pg0.SASAddress);
@@ -8553,13 +8560,13 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
MPI2_SAS_DEVICE0_FLAGS_DEVICE_PRESENT)) {
ioc_err(ioc, "device is not present handle(0x04%x)!!!\n",
handle);
- return -1;
+ return 0;
}
/* check if there were any issues with discovery */
if (_scsih_check_access_status(ioc, sas_address, handle,
sas_device_pg0.AccessStatus))
- return -1;
+ return 0;
port_id = sas_device_pg0.PhysicalPort;
sas_device = mpt3sas_get_sdev_by_addr(ioc,
@@ -8567,7 +8574,7 @@ _scsih_add_device(struct MPT3SAS_ADAPTER *ioc, u16 handle, u8 retry_count,
if (sas_device) {
clear_bit(handle, ioc->pend_os_device_add);
sas_device_put(sas_device);
- return -1;
+ return 0;
}
if (sas_device_pg0.EnclosureHandle) {
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 04/11] scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (2 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 03/11] scsi: mpt3sas: Requeue topology events only for busy devices Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 05/11] scsi: mpt3sas: Restore unblock retry " Jeff Layton
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
_scsih_ublock_io_device_wait() gets the pcie_device on each pass of its
TEST UNIT READY retry loop. But it puts the pcie_device only at the
bottom of the loop. The DEVICE_RETRY/START_UNIT/STOP_UNIT/RETRY_UA
branch ends in a continue, which skips the put. Thus each retry of an
NVMe device leaks a reference, and the driver can never free the
_pcie_device.
The function needs the pcie_device only for the reset timeout and the
reset method. Put the pcie_device immediately after the function reads
these values.
Fixes: aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 2d98fcb8e135..45419f0ebedc 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4220,6 +4220,8 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
tr_timeout = pcie_device->reset_timeout;
tr_method = MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE;
}
+ if (pcie_device)
+ pcie_device_put(pcie_device);
rc = _scsih_wait_for_device_to_become_ready(ioc,
sas_target->handle, 0, (sas_target->flags &
MPT_TARGET_FLAGS_RAID_COMPONENT), sdev->lun, tr_timeout, tr_method);
@@ -4241,8 +4243,6 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
}
continue;
}
- if (pcie_device)
- pcie_device_put(pcie_device);
} while ((rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
&& count++ <= command_retry_count);
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 05/11] scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (3 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 04/11] scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait() Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled Jeff Layton
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
When the firmware reports that a device responds again, the event
worker unblocks the sdevs of the device. Previously, the worker used
_scsih_ublock_io_device() for this, which uses
_scsih_internal_device_unblock(). The unblock can fail with -EINVAL, if
the SD layer set the device to SDEV_RUNNING while our block still
stopped its queue. Then that helper blocks the device again and
unblocks it, to restart the queue.
_scsih_ublock_io_device_wait(), which does a TUR check, replaced
_scsih_ublock_io_device(). It open-codes the unblock without that
retry, so such a device can stay stalled. Use
_scsih_internal_device_unblock() in _scsih_ublock_io_device_wait() too.
Fixes: aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 45419f0ebedc..5d494ad12c54 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4247,10 +4247,9 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
&& count++ <= command_retry_count);
- sas_device_priv_data->block = 0;
if (rc != DEVICE_READY)
sas_device_priv_data->deleted = 1;
- scsi_internal_device_unblock_nowait(sdev, SDEV_RUNNING);
+ _scsih_internal_device_unblock(sdev, sas_device_priv_data);
if (rc != DEVICE_READY) {
sdev_printk(KERN_WARNING, sdev,
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (4 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 05/11] scsi: mpt3sas: Restore unblock retry " Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 07/11] scsi: mpt3sas: Reset the TUR retry count for each device Jeff Layton
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
After a hard reset, the MPT3SAS_REMOVE_UNRESPONDING_DEVICES event polls
each blocked device with TEST UNIT READY in
_scsih_ublock_io_all_device(). It retries a device for up to
command_retry_count seconds (144 by default).
If another hard reset starts during this time, it sets shost_recovery.
Then _scsih_fw_event_cleanup_queue() waits for the event in
cancel_work_sync(). While shost_recovery is set,
_scsih_wait_for_device_to_become_ready() returns DEVICE_RETRY and does
not test the device. Thus, before the reset can continue, the event
consumes all the retries of the device that it polls. The event then
marks that device deleted and offline, although it never tested the
device.
If the event is cancelled, or if the host is being removed, the event
already returns before it polls. Do the same check during polling, and
stop polling if the check is true. The remaining devices stay blocked.
The reset queues a new MPT3SAS_REMOVE_UNRESPONDING_DEVICES event, which
polls them after recovery ends. If the host is removed, the removal
deletes them.
Fixes: 72340fecd0c8 ("scsi: mpt3sas: Add configurable command retry limit for slow-to-respond devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 5d494ad12c54..5657bda241f7 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4057,6 +4057,22 @@ _scsih_internal_device_unblock(struct scsi_device *sdev,
}
}
+/**
+ * _scsih_unresponding_scan_cancelled - should polling devices stop
+ * @ioc: per adapter object
+ *
+ * While the MPT3SAS_REMOVE_UNRESPONDING_DEVICES event polls devices, a
+ * hard reset or the removal of the host waits for it to finish, and the
+ * readiness check doesn't test devices during a reset.
+ *
+ * Return: true if the event is being cancelled, false otherwise.
+ */
+static bool
+_scsih_unresponding_scan_cancelled(struct MPT3SAS_ADAPTER *ioc)
+{
+ return READ_ONCE(ioc->remove_host) || READ_ONCE(ioc->fw_events_cleanup);
+}
+
/**
* _scsih_ublock_io_all_device - unblock every device
* @ioc: per adapter object
@@ -4114,7 +4130,13 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
pcie_device_put(pcie_device);
} while ((rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
- && count++ < command_retry_count);
+ && count++ < command_retry_count &&
+ !_scsih_unresponding_scan_cancelled(ioc));
+ if (_scsih_unresponding_scan_cancelled(ioc)) {
+ /* a reset queues a new scan, which polls the rest */
+ scsi_device_put(sdev);
+ return;
+ }
sas_device_priv_data->block = 0;
if (rc != DEVICE_READY)
sas_device_priv_data->deleted = 1;
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 07/11] scsi: mpt3sas: Reset the TUR retry count for each device
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (5 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 06/11] scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 08/11] scsi: mpt3sas: Cache the missing delays the IOC actually uses Jeff Layton
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
_scsih_ublock_io_all_device() and _scsih_ublock_io_device_wait() poll
each blocked sdev with TEST UNIT READY, for up to command_retry_count
retries. These functions set the retry counter to 0 once per call, not
once per sdev. Thus retries on one device consume the retry budget of
all later devices. When the budget is empty, the driver marks a device
deleted after its first TUR and sets it SDEV_OFFLINE. This occurs even
for a device that needs only one retry.
Start the count from 0 for each sdev.
Fixes: 72340fecd0c8 ("scsi: mpt3sas: Add configurable command retry limit for slow-to-respond devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 5657bda241f7..1269446eec4e 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4089,7 +4089,7 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
enum device_responsive_state rc;
struct _sas_device *sas_device = NULL;
struct _pcie_device *pcie_device = NULL;
- int count = 0;
+ int count;
u8 tr_method = 0;
u8 tr_timeout = 30;
@@ -4113,6 +4113,7 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
continue;
}
+ count = 0;
do {
pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
if (pcie_device && (!ioc->tm_custom_handling) &&
@@ -4199,7 +4200,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
struct _pcie_device *pcie_device;
u8 tr_timeout = 30;
u8 tr_method = 0;
- int count = 0;
+ int count;
/* moving devices from SDEV_OFFLINE to SDEV_BLOCK */
shost_for_each_device(sdev, ioc->shost) {
@@ -4234,6 +4235,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
if (!sas_device_priv_data->block)
continue;
+ count = 0;
do {
host_reset_completion_count = 0;
pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 08/11] scsi: mpt3sas: Cache the missing delays the IOC actually uses
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (6 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 07/11] scsi: mpt3sas: Reset the TUR retry count for each device Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 09/11] scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown Jeff Layton
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
When the missing_delay module parameter is set,
mpt3sas_base_update_missing_delay() writes the new delays to SAS IO
Unit Page 1. If the write succeeds, the function caches the requested
values in ioc->device_missing_delay and ioc->io_missing_delay.
mpt3sas_config_set_sas_iounit_pg1() ignores the result of the
WRITE_CURRENT request. It returns the result of the subsequent
WRITE_NVRAM request. If the WRITE_CURRENT request fails and the
WRITE_NVRAM request succeeds, the cached delays do not match the delays
that the IOC uses. The device_delay sysfs attribute then reports the
wrong value.
After the write, read the page again and cache the delays that it
reports. If this read fails, cache the requested delays if the NVRAM
update succeeded, as before.
Fixes: f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_base.c | 46 +++++++++++++++++++++++++------------
1 file changed, 31 insertions(+), 15 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 5c957044eded..07e16acc8abc 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -4923,6 +4923,7 @@ mpt3sas_base_update_missing_delay(struct MPT3SAS_ADAPTER *ioc,
Mpi2ConfigReply_t mpi_reply;
u8 num_phys = 0;
u16 ioc_status;
+ int rc;
mpt3sas_config_get_number_hba_phys(ioc, &num_phys);
if (!num_phys)
@@ -4969,23 +4970,38 @@ mpt3sas_base_update_missing_delay(struct MPT3SAS_ADAPTER *ioc,
io_missing_delay_original = sas_iounit_pg1->IODeviceMissingDelay;
sas_iounit_pg1->IODeviceMissingDelay = io_missing_delay;
- if (!mpt3sas_config_set_sas_iounit_pg1(ioc, &mpi_reply, sas_iounit_pg1,
- sz)) {
- if (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_UNIT_16)
- dmd_new = (dmd &
- MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK) * 16;
- else
- dmd_new =
- dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK;
- ioc_info(ioc, "device_missing_delay: old(%d), new(%d)\n",
- dmd_orignal, dmd_new);
- ioc_info(ioc, "ioc_missing_delay: old(%d), new(%d)\n",
- io_missing_delay_original,
- io_missing_delay);
- ioc->device_missing_delay = dmd_new;
- ioc->io_missing_delay = io_missing_delay;
+ /*
+ * This returns the result of the NVRAM update, not of the update
+ * to the current page, so read back the delays the IOC now uses.
+ * If that fails, assume the requested delays are used if the NVRAM
+ * update succeeded.
+ */
+ rc = mpt3sas_config_set_sas_iounit_pg1(ioc, &mpi_reply,
+ sas_iounit_pg1, sz);
+ if (!mpt3sas_config_get_sas_iounit_pg1(ioc, &mpi_reply,
+ sas_iounit_pg1, sz) &&
+ (le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK) ==
+ MPI2_IOCSTATUS_SUCCESS) {
+ dmd = sas_iounit_pg1->ReportDeviceMissingDelay;
+ io_missing_delay = sas_iounit_pg1->IODeviceMissingDelay;
+ } else {
+ ioc_warn(ioc, "failed to read back the missing delays\n");
+ if (rc)
+ goto out;
}
+ if (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_UNIT_16)
+ dmd_new = (dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK) * 16;
+ else
+ dmd_new = dmd & MPI2_SASIOUNIT1_REPORT_MISSING_TIMEOUT_MASK;
+ ioc_info(ioc, "device_missing_delay: old(%d), new(%d)\n",
+ dmd_orignal, dmd_new);
+ ioc_info(ioc, "ioc_missing_delay: old(%d), new(%d)\n",
+ io_missing_delay_original,
+ io_missing_delay);
+ ioc->device_missing_delay = dmd_new;
+ ioc->io_missing_delay = io_missing_delay;
+
out:
kfree(sas_iounit_pg1);
}
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 09/11] scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (7 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 08/11] scsi: mpt3sas: Cache the missing delays the IOC actually uses Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices Jeff Layton
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
On DELAY_NOT_RESPONDING, the event callback runs
_scsih_block_io_device() at interrupt time. This function walks the
sdevs with shost_for_each_device(). It uses sdev->hostdata and its
sas_target, and holds no lock.
The sdev reference pins only the scsi_device. __scsi_remove_device()
can still call ->sdev_destroy() at the same time, for example when a
scan fails to probe a new LUN. scsih_sdev_destroy() frees hostdata,
which causes a use-after-free. scsi_target_reap() frees the hostdata of
the target, after sdev_destroy.
Add ioc->block_lock. Take it:
- in _scsih_block_io_device(), for each sdev
- around the code that sets and clears sdev->hostdata in
scsih_sdev_init() and scsih_sdev_destroy()
The sdev is on shost->__devices before sdev_init. Because the driver
publishes hostdata under the lock, the lock also orders the sas_target
store. While the lock is held and hostdata is non-NULL, hostdata and
its target are live.
Fixes: f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_base.h | 2 ++
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 30 ++++++++++++++++++++++--------
2 files changed, 24 insertions(+), 8 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 47255bf9cdda..419fdf164fc6 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1249,6 +1249,7 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
* @pcie_device_list: pcie device object list
* @pcie_device_init_list: pcie device object list (used only at init time)
* @pcie_device_lock:
+ * @block_lock: serializes sdev blocking against sdev hostdata teardown
* @io_missing_delay: time for IO completed by fw when PDR enabled
* @device_missing_delay: time for device missing by fw when PDR enabled
* @sas_id : used for setting volume target IDs
@@ -1482,6 +1483,7 @@ struct MPT3SAS_ADAPTER {
struct list_head pcie_device_list;
struct list_head pcie_device_init_list;
spinlock_t pcie_device_lock;
+ spinlock_t block_lock;
struct list_head raid_device_list;
spinlock_t raid_device_lock;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 1269446eec4e..09dfa3124ebf 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -2132,12 +2132,17 @@ scsih_sdev_init(struct scsi_device *sdev)
sas_target_priv_data = starget->hostdata;
sas_target_priv_data->num_luns++;
sas_device_priv_data->sas_target = sas_target_priv_data;
+
+ shost = dev_to_shost(&starget->dev);
+ ioc = shost_priv(shost);
+
+ /* _scsih_block_io_device() may already see this sdev */
+ spin_lock_irqsave(&ioc->block_lock, flags);
sdev->hostdata = sas_device_priv_data;
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
if ((sas_target_priv_data->flags & MPT_TARGET_FLAGS_RAID_COMPONENT))
sdev->no_uld_attach = 1;
- shost = dev_to_shost(&starget->dev);
- ioc = shost_priv(shost);
if (starget->channel == RAID_CHANNEL) {
spin_lock_irqsave(&ioc->raid_device_lock, flags);
raid_device = _scsih_raid_device_find_by_id(ioc,
@@ -2189,6 +2194,7 @@ scsih_sdev_init(struct scsi_device *sdev)
static void
scsih_sdev_destroy(struct scsi_device *sdev)
{
+ struct MPT3SAS_DEVICE *sas_device_priv_data;
struct MPT3SAS_TARGET *sas_target_priv_data;
struct scsi_target *starget;
struct Scsi_Host *shost;
@@ -2231,8 +2237,11 @@ scsih_sdev_destroy(struct scsi_device *sdev)
spin_unlock_irqrestore(&ioc->sas_device_lock, flags);
}
- kfree(sdev->hostdata);
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ sas_device_priv_data = sdev->hostdata;
sdev->hostdata = NULL;
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ kfree(sas_device_priv_data);
}
/**
@@ -4384,26 +4393,30 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
struct MPT3SAS_DEVICE *sas_device_priv_data;
struct scsi_device *sdev;
struct _sas_device *sas_device;
+ unsigned long flags;
sas_device = mpt3sas_get_sdev_by_handle(ioc, handle);
shost_for_each_device(sdev, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data)
- continue;
+ goto next;
if (sas_device_priv_data->sas_target->handle != handle)
- continue;
+ goto next;
if (sas_device_priv_data->block)
- continue;
+ goto next;
if (sas_device && sas_device->pend_sas_rphy_add)
- continue;
+ goto next;
if (sas_device_priv_data->ignore_delay_remove) {
sdev_printk(KERN_INFO, sdev,
"%s skip device_block for SES handle(0x%04x)\n",
__func__, sas_device_priv_data->sas_target->handle);
- continue;
+ goto next;
}
_scsih_internal_device_block(sdev, sas_device_priv_data);
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
}
if (sas_device)
@@ -13697,6 +13710,7 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
spin_lock_init(&ioc->raid_device_lock);
spin_lock_init(&ioc->pcie_device_lock);
spin_lock_init(&ioc->diag_trigger_lock);
+ spin_lock_init(&ioc->block_lock);
INIT_LIST_HEAD(&ioc->sas_device_list);
INIT_LIST_HEAD(&ioc->sas_device_init_list);
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (8 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 09/11] scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:10 ` [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks Jeff Layton
2026-11-18 23:13 ` [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
_scsih_ublock_io_all_device() and _scsih_ublock_io_device_wait() send
TURs to each blocked sdev until it is ready, for up to
command_retry_count rounds. They sleep between retries. They read
sdev->hostdata and its sas_target once. Then they use both during
polling and after it.
The sdev reference pins only the scsi_device. __scsi_remove_device()
can call ->sdev_destroy() during this time. For example, this occurs
when a scan fails to probe a new LUN, or when the device is deleted
through sysfs. ->sdev_destroy() frees the hostdata. If the sdev was the
last LUN, scsi_target_reap() then frees the hostdata of the target.
Thus the worker reads and writes freed memory.
Before each TUR, read the handle and the target flags under block_lock.
If the hostdata is gone, stop polling. Do the final unblock under the
lock, and only if the hostdata is unchanged. Copy the target fields
that the offline logging needs. Take block_lock in the other
worker-side walkers that block or unblock sdevs too.
Fixes: aee682fad6cd ("scsi: mpt3sas: Improve device discovery and readiness handling for slow devices")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_base.h | 3 +-
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 210 ++++++++++++++++++++++++++++-------
2 files changed, 173 insertions(+), 40 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index 419fdf164fc6..f5e1edf1a280 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1249,7 +1249,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
* @pcie_device_list: pcie device object list
* @pcie_device_init_list: pcie device object list (used only at init time)
* @pcie_device_lock:
- * @block_lock: serializes sdev blocking against sdev hostdata teardown
+ * @block_lock: serializes sdev blocking and unblocking against sdev
+ * hostdata teardown
* @io_missing_delay: time for IO completed by fw when PDR enabled
* @device_missing_delay: time for device missing by fw when PDR enabled
* @sas_id : used for setting volume target IDs
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 09dfa3124ebf..814c18289cf1 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4066,6 +4066,99 @@ _scsih_internal_device_unblock(struct scsi_device *sdev,
}
}
+/*
+ * Target fields the firmware event worker uses while it polls a device
+ * without holding block_lock.
+ */
+struct _scsih_target_ids {
+ u64 sas_address;
+ struct hba_port *port;
+ u16 handle;
+ u8 raid_component;
+};
+
+/*
+ * Copy the target fields of @sdev into @ids, if @sas_device_priv_data is
+ * still its hostdata. The caller holds block_lock.
+ */
+static bool
+__scsih_get_target_ids(struct scsi_device *sdev,
+ struct MPT3SAS_DEVICE *sas_device_priv_data,
+ struct _scsih_target_ids *ids)
+{
+ struct MPT3SAS_TARGET *sas_target;
+
+ if (sdev->hostdata != sas_device_priv_data)
+ return false;
+ sas_target = sas_device_priv_data->sas_target;
+ ids->sas_address = sas_target->sas_address;
+ ids->port = sas_target->port;
+ ids->handle = sas_target->handle;
+ ids->raid_component =
+ !!(sas_target->flags & MPT_TARGET_FLAGS_RAID_COMPONENT);
+ return true;
+}
+
+/**
+ * _scsih_get_target_ids - read the target fields of a polled device
+ * @ioc: per adapter object
+ * @sdev: scsi device struct
+ * @sas_device_priv_data: hostdata of @sdev when polling began
+ * @ids: filled in with the target fields
+ *
+ * A host reset can change the handle while the device is polled, so
+ * the handle is read again before each command.
+ *
+ * Return: false if @sdev lost its hostdata, true otherwise.
+ */
+static bool
+_scsih_get_target_ids(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
+ struct MPT3SAS_DEVICE *sas_device_priv_data,
+ struct _scsih_target_ids *ids)
+{
+ unsigned long flags;
+ bool live;
+
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ live = __scsih_get_target_ids(sdev, sas_device_priv_data, ids);
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ return live;
+}
+
+/**
+ * _scsih_end_poll - unblock a device after polling it
+ * @ioc: per adapter object
+ * @sdev: scsi device struct
+ * @sas_device_priv_data: hostdata of @sdev when polling began
+ * @ready: the device passed the readiness check
+ * @ids: filled in with the target fields, if true is returned
+ *
+ * Unblock @sdev, and mark it deleted if it is not @ready. Nothing is
+ * done if @sdev lost its hostdata while it was polled.
+ *
+ * Return: true if the caller should act on @ready, false otherwise.
+ */
+static bool
+_scsih_end_poll(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
+ struct MPT3SAS_DEVICE *sas_device_priv_data,
+ bool ready, struct _scsih_target_ids *ids)
+{
+ unsigned long flags;
+ bool ret = false;
+
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ if (!__scsih_get_target_ids(sdev, sas_device_priv_data, ids))
+ goto out;
+ if (!ready)
+ sas_device_priv_data->deleted = 1;
+ if (sas_device_priv_data->block)
+ _scsih_internal_device_unblock(sdev, sas_device_priv_data);
+ ret = true;
+out:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ return ret;
+}
+
/**
* _scsih_unresponding_scan_cancelled - should polling devices stop
* @ioc: per adapter object
@@ -4098,10 +4191,11 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
enum device_responsive_state rc;
struct _sas_device *sas_device = NULL;
struct _pcie_device *pcie_device = NULL;
+ struct _scsih_target_ids ids;
int count;
u8 tr_method = 0;
u8 tr_timeout = 30;
-
+ unsigned long flags;
shost_for_each_device(sdev, ioc->shost) {
sas_device_priv_data = sdev->hostdata;
@@ -4116,23 +4210,40 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
continue;
if ((no_turs) || (!issue_scsi_cmd_to_bringup_drive)) {
- sdev_printk(KERN_WARNING, sdev, "device_unblocked handle(0x%04x)\n",
- sas_device_priv_data->sas_target->handle);
- _scsih_internal_device_unblock(sdev, sas_device_priv_data);
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ if (sdev->hostdata != sas_device_priv_data)
+ goto next;
+ if (sas_device_priv_data->block) {
+ sdev_printk(KERN_WARNING, sdev,
+ "device_unblocked handle(0x%04x)\n",
+ sas_target->handle);
+ _scsih_internal_device_unblock(sdev,
+ sas_device_priv_data);
+ }
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
continue;
}
+ /*
+ * Polling can take minutes, and sdev_destroy() and
+ * scsi_target_reap() can free the hostdata meanwhile.
+ */
+ rc = DEVICE_ERROR;
count = 0;
do {
- pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
+ if (!_scsih_get_target_ids(ioc, sdev,
+ sas_device_priv_data, &ids))
+ break;
+ pcie_device = mpt3sas_get_pdev_by_handle(ioc, ids.handle);
if (pcie_device && (!ioc->tm_custom_handling) &&
(!(mpt3sas_scsih_is_pcie_scsi_device(pcie_device->device_info)))) {
tr_timeout = pcie_device->reset_timeout;
tr_method = MPI26_SCSITASKMGMT_MSGFLAGS_PROTOCOL_LVL_RST_PCIE;
}
rc = _scsih_wait_for_device_to_become_ready(ioc,
- sas_target->handle, 0, (sas_target->flags &
- MPT_TARGET_FLAGS_RAID_COMPONENT), sdev->lun, tr_timeout, tr_method);
+ ids.handle, 0, ids.raid_component, sdev->lun,
+ tr_timeout, tr_method);
if (rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
ssleep(1);
@@ -4147,26 +4258,23 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
scsi_device_put(sdev);
return;
}
- sas_device_priv_data->block = 0;
- if (rc != DEVICE_READY)
- sas_device_priv_data->deleted = 1;
-
- _scsih_internal_device_unblock(sdev, sas_device_priv_data);
+ if (!_scsih_end_poll(ioc, sdev, sas_device_priv_data,
+ rc == DEVICE_READY, &ids))
+ continue;
if (rc != DEVICE_READY) {
sdev_printk(KERN_WARNING, sdev, "%s: device_offlined,\n"
"handle(0x%04x)\n",
- __func__, sas_device_priv_data->sas_target->handle);
+ __func__, ids.handle);
scsi_device_set_state(sdev, SDEV_OFFLINE);
sas_device = mpt3sas_get_sdev_by_addr(ioc,
- sas_device_priv_data->sas_target->sas_address,
- sas_device_priv_data->sas_target->port);
+ ids.sas_address, ids.port);
if (sas_device) {
_scsih_display_enclosure_chassis_info(NULL, sas_device, sdev, NULL);
sas_device_put(sas_device);
} else {
pcie_device = mpt3sas_get_pdev_by_wwid(ioc,
- sas_device_priv_data->sas_target->sas_address);
+ ids.sas_address);
if (pcie_device) {
if (pcie_device->enclosure_handle != 0)
sdev_printk(KERN_INFO, sdev, "enclosure logical id\n"
@@ -4184,7 +4292,7 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
} else
sdev_printk(KERN_WARNING, sdev, "device_unblocked,\n"
"handle(0x%04x)\n",
- sas_device_priv_data->sas_target->handle);
+ ids.handle);
}
}
@@ -4207,27 +4315,32 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
int host_reset_completion_count;
struct _sas_device *sas_device;
struct _pcie_device *pcie_device;
+ struct _scsih_target_ids ids;
u8 tr_timeout = 30;
u8 tr_method = 0;
int count;
+ unsigned long flags;
/* moving devices from SDEV_OFFLINE to SDEV_BLOCK */
shost_for_each_device(sdev, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data)
- continue;
+ goto next;
sas_target = sas_device_priv_data->sas_target;
if (!sas_target)
- continue;
+ goto next;
if (sas_target->sas_address != sas_address ||
sas_target->port != port)
- continue;
+ goto next;
if (sdev->sdev_state == SDEV_OFFLINE) {
sas_device_priv_data->block = 1;
sas_device_priv_data->deleted = 0;
scsi_device_set_state(sdev, SDEV_RUNNING);
scsi_internal_device_block_nowait(sdev);
}
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
}
/* moving devices from SDEV_BLOCK to SDEV_RUNNING state */
@@ -4244,10 +4357,18 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
if (!sas_device_priv_data->block)
continue;
+ /*
+ * Polling can take minutes, and sdev_destroy() and
+ * scsi_target_reap() can free the hostdata meanwhile.
+ */
+ rc = DEVICE_ERROR;
count = 0;
do {
host_reset_completion_count = 0;
- pcie_device = mpt3sas_get_pdev_by_handle(ioc, sas_target->handle);
+ if (!_scsih_get_target_ids(ioc, sdev,
+ sas_device_priv_data, &ids))
+ break;
+ pcie_device = mpt3sas_get_pdev_by_handle(ioc, ids.handle);
if (pcie_device && (!ioc->tm_custom_handling) &&
(!(mpt3sas_scsih_is_pcie_scsi_device(pcie_device->device_info)))) {
tr_timeout = pcie_device->reset_timeout;
@@ -4256,8 +4377,8 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
if (pcie_device)
pcie_device_put(pcie_device);
rc = _scsih_wait_for_device_to_become_ready(ioc,
- sas_target->handle, 0, (sas_target->flags &
- MPT_TARGET_FLAGS_RAID_COMPONENT), sdev->lun, tr_timeout, tr_method);
+ ids.handle, 0, ids.raid_component, sdev->lun,
+ tr_timeout, tr_method);
if (rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA) {
do {
@@ -4266,10 +4387,13 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
} while (rc == DEVICE_RETRY &&
ioc->shost_recovery);
if (host_reset_completion_count > 1) {
+ /* the reset may have changed the handle */
+ if (!_scsih_get_target_ids(ioc, sdev,
+ sas_device_priv_data, &ids))
+ break;
rc = _scsih_wait_for_device_to_become_ready(ioc,
- sas_target->handle, 0, (sas_target->flags &
- MPT_TARGET_FLAGS_RAID_COMPONENT), sdev->lun,
- tr_timeout, tr_method);
+ ids.handle, 0, ids.raid_component,
+ sdev->lun, tr_timeout, tr_method);
if (rc == DEVICE_RETRY || rc == DEVICE_START_UNIT ||
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
msleep(500);
@@ -4280,23 +4404,23 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
rc == DEVICE_STOP_UNIT || rc == DEVICE_RETRY_UA)
&& count++ <= command_retry_count);
- if (rc != DEVICE_READY)
- sas_device_priv_data->deleted = 1;
- _scsih_internal_device_unblock(sdev, sas_device_priv_data);
+ if (!_scsih_end_poll(ioc, sdev, sas_device_priv_data,
+ rc == DEVICE_READY, &ids))
+ continue;
if (rc != DEVICE_READY) {
sdev_printk(KERN_WARNING, sdev,
"%s: device_offlined, handle(0x%04x)\n",
- __func__, sas_device_priv_data->sas_target->handle);
+ __func__, ids.handle);
sas_device = mpt3sas_get_sdev_by_handle(ioc,
- sas_device_priv_data->sas_target->handle);
+ ids.handle);
if (sas_device) {
_scsih_display_enclosure_chassis_info(NULL, sas_device, sdev, NULL);
sas_device_put(sas_device);
} else {
pcie_device = mpt3sas_get_pdev_by_handle(ioc,
- sas_device_priv_data->sas_target->handle);
+ ids.handle);
if (pcie_device) {
if (pcie_device->enclosure_handle != 0)
sdev_printk(KERN_INFO, sdev,
@@ -4317,7 +4441,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
} else {
sdev_printk(KERN_WARNING, sdev,
"device_unblocked, handle(0x%04x)\n",
- sas_device_priv_data->sas_target->handle);
+ ids.handle);
}
}
}
@@ -4336,19 +4460,23 @@ _scsih_ublock_io_device(struct MPT3SAS_ADAPTER *ioc,
{
struct MPT3SAS_DEVICE *sas_device_priv_data;
struct scsi_device *sdev;
+ unsigned long flags;
shost_for_each_device(sdev, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data || !sas_device_priv_data->sas_target)
- continue;
+ goto next;
if (sas_device_priv_data->sas_target->sas_address
!= sas_address)
- continue;
+ goto next;
if (sas_device_priv_data->sas_target->port != port)
- continue;
+ goto next;
if (sas_device_priv_data->block)
_scsih_internal_device_unblock(sdev,
sas_device_priv_data);
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
}
}
@@ -4363,20 +4491,24 @@ _scsih_block_io_all_device(struct MPT3SAS_ADAPTER *ioc)
{
struct MPT3SAS_DEVICE *sas_device_priv_data;
struct scsi_device *sdev;
+ unsigned long flags;
shost_for_each_device(sdev, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data)
- continue;
+ goto next;
if (sas_device_priv_data->block)
- continue;
+ goto next;
if (sas_device_priv_data->ignore_delay_remove) {
sdev_printk(KERN_INFO, sdev,
"%s skip device_block for SES handle(0x%04x)\n",
__func__, sas_device_priv_data->sas_target->handle);
- continue;
+ goto next;
}
_scsih_internal_device_block(sdev, sas_device_priv_data);
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
}
}
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (9 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 10/11] scsi: mpt3sas: Don't use freed hostdata while polling devices Jeff Layton
@ 2026-11-18 23:10 ` Jeff Layton
2026-11-18 23:13 ` [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:10 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel, Jeff Layton
We have an agent that writes "- - -" to the sysfs scan attribute at
regular intervals. Several of these scans deadlocked:
1. A drive in reset failed its initial scan. Thus the worker started
to remove the drive.
2. A sysfs scan took sas_host->lock, allocated an sdev for the drive
and sent an INQUIRY.
3. A DELAY_NOT_RESPONDING event blocked the new sdev.
4. The worker waited for sas_host->lock in sas_rphy_free():
__mutex_lock
sas_port_delete+0x75 [scsi_transport_sas]
mpt3sas_transport_port_remove
_scsih_add_device
After that, all later scan attempts hung while they waited for
sas_host->lock.
On DELAY_NOT_RESPONDING, the event callback blocks every sdev on the
handle. Only these two paths release the block:
- the callback, if the firmware later reports TARG_NOT_RESPONDING
- the firmware event worker, when it processes a later event
Commit e4bc7f5c21a1 ("mpt3sas: Don't block the drive when drive addition
under the control of SML") partially fixed a similar issue. That commit
stops the driver from blocking sdevs that a scan still probes. This is
not sufficient. When LUN 0 is visible, the scan also sends REPORT LUNS
to it. LUN 0 may be blocked before or during the scan.
Within the report device missing delay, one of two things occurs:
- The firmware reports the device as not responding, and the callback
unblocks it.
- The device returns.
After that delay, the block has no further use.
Schedule delayed_work on system_dfl_wq for the missing delay plus 30s.
When the work runs, unblock all expired sdevs, except during a host
reset. Do not wait for SCSI EH to end. scsi_send_eh_cmnd() waits for
SDEV_BLOCK to end only up to the EH command timeout. Thus EH would
fail its TURs to a device that is back, and set it offline.
Meanwhile, the unblock by the event worker waits for EH to end.
The driver tracked only the SAS missing delay. Also read the PCIe
missing delay from PCIe IO Unit Page 1. The driver already reads this
page for the NVMe queue depth. Use the delay that applies to the device
type. Until the first read of the page, assume the longest PCIe delay
that the firmware supports. Because the timeouts differ, always queue
the work for the earliest deadline.
Before the worker polls a blocked sdev, clear the deadline of the sdev
under ioc->block_lock. Then dnr_unblock_work leaves the sdev to the
worker. Do not put a timed block on offline sdevs. Expiry would set
them running again with no TUR check. This includes sdevs that failed
the TUR check of the worker. They are marked deleted under block_lock,
but set offline only after the lock is released.
If a hard reset cancels the polling, give new deadlines to the sdevs
that are still blocked. A failed reset queues no new scan. Also, the
next scan removes missing devices before it polls the others. That
removal can deadlock again.
If a host reset starts during a broadcast primitive pass, the pass
skips its unblock. For the same reason as above, give deadlines to the
sdevs that the pass blocks. Let sdevs that are already blocked keep
their deadlines. While the pass queries and aborts the commands of
these sdevs, prevent dnr_unblock_work from unblocking them.
Exclude offline sdevs from the pass. They take no I/O. The unblock by
the pass, or an expiry, would set them running again without a TUR
check.
A repeat DNR on a blocked sdev restarts its deadline, but at most
MPT3SAS_DNR_BLOCK_MAX_EXTENDS times. Thus a drive that continues to
flap cannot make a scan wait on it for as long as the flapping lasts.
An expired block may end before the device returns. Thus, at its next
unblock, the worker still sends TURs to those sdevs. It sets them
offline if they are not ready. If the driver makes a new delayed
not-responding block during polling, keep that block until its own
expiry or the next event.
Fixes: f92363d12359 ("[SCSI] mpt3sas: add new driver supporting 12GB SAS")
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
drivers/scsi/mpt3sas/mpt3sas_base.c | 16 ++
drivers/scsi/mpt3sas/mpt3sas_base.h | 26 ++-
drivers/scsi/mpt3sas/mpt3sas_scsih.c | 339 +++++++++++++++++++++++++++++++++--
3 files changed, 369 insertions(+), 12 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 07e16acc8abc..b7169253ca6e 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -5421,6 +5421,7 @@ _base_update_diag_trigger_pages(struct MPT3SAS_ADAPTER *ioc)
/**
* _base_assign_fw_reported_qd - Get FW reported QD for SAS/SATA devices.
* - On failure set default QD values.
+ * - Also get the PCIe device missing delay.
* @ioc : per adapter object
*
* Returns 0 for success, non-zero for failure.
@@ -5432,6 +5433,7 @@ static int _base_assign_fw_reported_qd(struct MPT3SAS_ADAPTER *ioc)
Mpi2SasIOUnitPage1_t sas_iounit_pg1;
Mpi26PCIeIOUnitPage1_t pcie_iounit_pg1;
u16 depth;
+ u8 dmd;
int rc = 0;
ioc->max_wideport_qd = MPT3SAS_SAS_QUEUE_DEPTH;
@@ -5469,6 +5471,20 @@ static int _base_assign_fw_reported_qd(struct MPT3SAS_ADAPTER *ioc)
ioc->max_nvme_qd = (le16_to_cpu(pcie_iounit_pg1.NVMeMaxQueueDepth)) ?
(le16_to_cpu(pcie_iounit_pg1.NVMeMaxQueueDepth)) :
MPT3SAS_NVME_QUEUE_DEPTH;
+
+ if ((le16_to_cpu(mpi_reply.IOCStatus) & MPI2_IOCSTATUS_MASK) !=
+ MPI2_IOCSTATUS_SUCCESS) {
+ ioc_warn(ioc, "PCIe device missing delay unknown, using %us\n",
+ ioc->pcie_device_missing_delay);
+ goto out;
+ }
+ dmd = pcie_iounit_pg1.DMDReportPCIe;
+ if (dmd & MPI26_PCIEIOUNIT1_DMDRPT_UNIT_16_SEC)
+ ioc->pcie_device_missing_delay =
+ (dmd & MPI26_PCIEIOUNIT1_DMDRPT_DELAY_TIME_MASK) * 16;
+ else
+ ioc->pcie_device_missing_delay =
+ dmd & MPI26_PCIEIOUNIT1_DMDRPT_DELAY_TIME_MASK;
out:
dinitprintk(ioc, pr_err(
"MaxWidePortQD: 0x%x MaxNarrowPortQD: 0x%x MaxSataQD: 0x%x MaxNvmeQD: 0x%x\n",
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index f5e1edf1a280..6d3ded18c3a2 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -469,6 +469,14 @@ struct MPT3SAS_TARGET {
* @flags: MPT_DEVICE_XXX flags
* @configured_lun: lun is configured
* @block: device is in SDEV_BLOCK state
+ * @block_deadline: jiffies after which a delayed not-responding block is
+ * released without waiting for the firmware event worker, 0 if none
+ * @dnr_expired: a delayed not-responding block expired before the firmware
+ * event worker checked the device; it still needs a readiness check
+ * @block_extends: times a repeat delayed not-responding event has pushed
+ * out @block_deadline
+ * @block_seq: incremented on each new block, so the firmware event worker
+ * can tell whether the device was blocked again while it polled it
* @tlr_snoop_check: flag used in determining whether to disable TLR
* @eedp_enable: eedp support enable bit
* @eedp_type: 0(type_1), 1(type_2), 2(type_3)
@@ -486,6 +494,10 @@ struct MPT3SAS_DEVICE {
u8 ignore_delay_remove;
/* Iopriority Command Handling */
u8 ncq_prio_enable;
+ u8 dnr_expired;
+ u8 block_extends;
+ unsigned int block_seq;
+ unsigned long block_deadline;
/*
* Bug workaround for SATL handling: the mpt2/3sas firmware
* doesn't return BUSY or TASK_SET_FULL for subsequent
@@ -1165,6 +1177,9 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
* pci resource handling
* @fault_reset_work_q: fw fault workqueue
* @fault_reset_work: fw fault work
+ * @dnr_unblock_work: releases expired delayed not-responding blocks
+ * @dnr_unblock_deadline: jiffies dnr_unblock_work is set to run at, 0 if
+ * it isn't pending. Protected by block_lock.
* @firmware_event_thread: fw event work queue
* @fw_event_lock:
* @fw_event_list: list of fw events
@@ -1172,6 +1187,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
* @fw_event_cleanup: set to one while cleaning up the fw events
* @aen_event_read_flag: event log was read
* @broadcast_aen_busy: broadcast aen waiting to be serviced
+ * @broadcast_tm_active: broadcast primitive pass running, with every device
+ * blocked; dnr_unblock_work waits for it
* @shost_recovery: host reset in progress
* @ioc_reset_in_progress_lock:
* @ioc_link_reset_in_progress: phy/hard reset in progress
@@ -1249,10 +1266,11 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
* @pcie_device_list: pcie device object list
* @pcie_device_init_list: pcie device object list (used only at init time)
* @pcie_device_lock:
- * @block_lock: serializes sdev blocking and unblocking against sdev
- * hostdata teardown
+ * @block_lock: serializes sdev blocking and unblocking against
+ * dnr_unblock_work and sdev hostdata teardown
* @io_missing_delay: time for IO completed by fw when PDR enabled
* @device_missing_delay: time for device missing by fw when PDR enabled
+ * @pcie_device_missing_delay: time for PCIe device missing by fw
* @sas_id : used for setting volume target IDs
* @pcie_target_id: used for setting pcie target IDs
* @blocking_handles: bitmask used to identify which devices need blocking
@@ -1352,6 +1370,8 @@ struct MPT3SAS_ADAPTER {
/* fw fault handler */
struct workqueue_struct *fault_reset_work_q;
struct delayed_work fault_reset_work;
+ struct delayed_work dnr_unblock_work;
+ unsigned long dnr_unblock_deadline;
/* fw event handler */
struct workqueue_struct *firmware_event_thread;
@@ -1363,6 +1383,7 @@ struct MPT3SAS_ADAPTER {
/* misc flags */
int aen_event_read_flag;
u8 broadcast_aen_busy;
+ u8 broadcast_tm_active;
u16 broadcast_aen_pending;
u8 shost_recovery;
u8 got_task_abort_from_ioctl;
@@ -1490,6 +1511,7 @@ struct MPT3SAS_ADAPTER {
spinlock_t raid_device_lock;
u8 io_missing_delay;
u16 device_missing_delay;
+ u16 pcie_device_missing_delay;
int sas_id;
int pcie_target_id;
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 814c18289cf1..6eb92bd4f122 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -4014,6 +4014,8 @@ _scsih_internal_device_block(struct scsi_device *sdev,
sdev_printk(KERN_INFO, sdev, "device_block, handle(0x%04x)\n",
sas_device_priv_data->sas_target->handle);
sas_device_priv_data->block = 1;
+ sas_device_priv_data->block_deadline = 0;
+ sas_device_priv_data->block_seq++;
r = scsi_internal_device_block_nowait(sdev);
if (r == -EINVAL)
@@ -4039,6 +4041,7 @@ _scsih_internal_device_unblock(struct scsi_device *sdev,
sdev_printk(KERN_WARNING, sdev, "device_unblock and setting to running, "
"handle(0x%04x)\n", sas_device_priv_data->sas_target->handle);
sas_device_priv_data->block = 0;
+ sas_device_priv_data->block_deadline = 0;
r = scsi_internal_device_unblock_nowait(sdev, SDEV_RUNNING);
if (r == -EINVAL) {
/* The device has been set to SDEV_RUNNING by SD layer during
@@ -4125,23 +4128,106 @@ _scsih_get_target_ids(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
return live;
}
+/*
+ * Margin on top of the firmware's report device missing delay before
+ * _scsih_dnr_unblock_work() releases a delayed not-responding block.
+ */
+#define MPT3SAS_DNR_BLOCK_SLACK 30
+
+/*
+ * A repeat DNR on a blocked device extends its block, at most this many
+ * times, so a drive that keeps flapping can't hold it forever.
+ */
+#define MPT3SAS_DNR_BLOCK_MAX_EXTENDS 3
+
+static unsigned long
+_scsih_dnr_block_timeout(struct MPT3SAS_ADAPTER *ioc,
+ struct MPT3SAS_TARGET *sas_target)
+{
+ u16 delay = (sas_target->flags & MPT_TARGET_FLAGS_PCIE_DEVICE) ?
+ ioc->pcie_device_missing_delay : ioc->device_missing_delay;
+
+ return (delay + MPT3SAS_DNR_BLOCK_SLACK) * HZ;
+}
+
+/*
+ * Make dnr_unblock_work run no later than @deadline. The caller holds
+ * block_lock. Timeouts differ between SAS and PCIe devices, so a new
+ * block can expire before the one the work is pending for.
+ */
+static void
+__scsih_dnr_unblock_work_by(struct MPT3SAS_ADAPTER *ioc,
+ unsigned long deadline)
+{
+ if (ioc->dnr_unblock_deadline &&
+ !time_before(deadline, ioc->dnr_unblock_deadline))
+ return;
+ ioc->dnr_unblock_deadline = deadline;
+ mod_delayed_work(system_dfl_wq, &ioc->dnr_unblock_work,
+ max_t(long, deadline - jiffies, 1));
+}
+
+/**
+ * _scsih_claim_block - keep dnr_unblock_work off a blocked sdev
+ * @ioc: per adapter object
+ * @sdev: scsi device struct
+ * @sas_device_priv_data: hostdata of @sdev, as read by the caller
+ * @block_seq: set to the block_seq of @sdev, for _scsih_end_poll()
+ *
+ * Called by the firmware event worker before it polls a device with TURs.
+ * The worker unblocks the device when polling ends, and must not have
+ * dnr_unblock_work send I/O to it in the meantime.
+ *
+ * A device whose block expired is already running, but still gets polled,
+ * and offlined if it is not ready.
+ *
+ * Return: true if @sdev needs polling: it is blocked, or its block expired
+ * since the last poll. False otherwise.
+ */
+static bool
+_scsih_claim_block(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
+ struct MPT3SAS_DEVICE *sas_device_priv_data,
+ unsigned int *block_seq)
+{
+ unsigned long flags;
+ bool claimed = false;
+
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ if (sdev->hostdata == sas_device_priv_data &&
+ (sas_device_priv_data->block || sas_device_priv_data->dnr_expired)) {
+ sas_device_priv_data->block_deadline = 0;
+ sas_device_priv_data->dnr_expired = 0;
+ *block_seq = sas_device_priv_data->block_seq;
+ claimed = true;
+ }
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ return claimed;
+}
+
/**
* _scsih_end_poll - unblock a device after polling it
* @ioc: per adapter object
* @sdev: scsi device struct
* @sas_device_priv_data: hostdata of @sdev when polling began
+ * @block_seq: block_seq of @sdev when _scsih_claim_block() claimed it
* @ready: the device passed the readiness check
* @ids: filled in with the target fields, if true is returned
*
* Unblock @sdev, and mark it deleted if it is not @ready. Nothing is
* done if @sdev lost its hostdata while it was polled.
*
+ * Nothing is done either if a new delayed not-responding block was made
+ * on @sdev. Polling then began before the firmware last reported the
+ * device missing, and the expiry of that block or the firmware's next
+ * event decides instead.
+ *
* Return: true if the caller should act on @ready, false otherwise.
*/
static bool
_scsih_end_poll(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
struct MPT3SAS_DEVICE *sas_device_priv_data,
- bool ready, struct _scsih_target_ids *ids)
+ unsigned int block_seq, bool ready,
+ struct _scsih_target_ids *ids)
{
unsigned long flags;
bool ret = false;
@@ -4149,8 +4235,15 @@ _scsih_end_poll(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
spin_lock_irqsave(&ioc->block_lock, flags);
if (!__scsih_get_target_ids(sdev, sas_device_priv_data, ids))
goto out;
+ if (sas_device_priv_data->block_seq != block_seq) {
+ sdev_printk(KERN_INFO, sdev,
+ "blocked again while polled, handle(0x%04x)\n",
+ ids->handle);
+ goto out;
+ }
if (!ready)
sas_device_priv_data->deleted = 1;
+ /* not blocked if dnr_unblock_work released it before polling */
if (sas_device_priv_data->block)
_scsih_internal_device_unblock(sdev, sas_device_priv_data);
ret = true;
@@ -4175,6 +4268,50 @@ _scsih_unresponding_scan_cancelled(struct MPT3SAS_ADAPTER *ioc)
return READ_ONCE(ioc->remove_host) || READ_ONCE(ioc->fw_events_cleanup);
}
+/**
+ * _scsih_cancel_poll - hand devices back to dnr_unblock_work
+ * @ioc: per adapter object
+ * @sdev: device being polled when the scan was cancelled
+ * @sas_device_priv_data: hostdata of @sdev when it was claimed
+ * @block_seq: block_seq of @sdev when it was claimed
+ *
+ * _scsih_ublock_io_all_device() cleared the deadlines of the devices it
+ * was going to poll. Give those still blocked a new one: a failed reset
+ * queues no new scan, and the scan removes missing devices before it
+ * polls the others. If @sdev's block expired, it still needs a TUR check.
+ */
+static void
+_scsih_cancel_poll(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev,
+ struct MPT3SAS_DEVICE *sas_device_priv_data,
+ unsigned int block_seq)
+{
+ struct MPT3SAS_DEVICE *priv;
+ struct scsi_device *iter;
+ unsigned long flags, deadline;
+
+ /* dnr_unblock_work is disabled, and the devices are going away */
+ if (READ_ONCE(ioc->remove_host))
+ return;
+
+ shost_for_each_device(iter, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ priv = iter->hostdata;
+ if (!priv || !priv->sas_target || priv->sas_target->deleted)
+ goto next;
+ if (priv->block && !priv->block_deadline) {
+ deadline = (jiffies +
+ _scsih_dnr_block_timeout(ioc, priv->sas_target)) ?: 1;
+ priv->block_deadline = deadline;
+ __scsih_dnr_unblock_work_by(ioc, deadline);
+ } else if (iter == sdev && priv == sas_device_priv_data &&
+ priv->block_seq == block_seq && !priv->block) {
+ priv->dnr_expired = 1;
+ }
+next:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ }
+}
+
/**
* _scsih_ublock_io_all_device - unblock every device
* @ioc: per adapter object
@@ -4195,8 +4332,22 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
int count;
u8 tr_method = 0;
u8 tr_timeout = 30;
+ unsigned int block_seq;
unsigned long flags;
+ /* polling one device can outlast another's deadline */
+ if (!no_turs && issue_scsi_cmd_to_bringup_drive) {
+ shost_for_each_device(sdev, ioc->shost) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ sas_device_priv_data = sdev->hostdata;
+ if (sas_device_priv_data &&
+ sas_device_priv_data->sas_target &&
+ !sas_device_priv_data->sas_target->deleted)
+ sas_device_priv_data->block_deadline = 0;
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ }
+ }
+
shost_for_each_device(sdev, ioc->shost) {
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data)
@@ -4206,13 +4357,14 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
if (!sas_target || sas_target->deleted)
continue;
- if (!sas_device_priv_data->block)
- continue;
-
if ((no_turs) || (!issue_scsi_cmd_to_bringup_drive)) {
spin_lock_irqsave(&ioc->block_lock, flags);
if (sdev->hostdata != sas_device_priv_data)
goto next;
+ /*
+ * dnr_unblock_work may have released it. Leave
+ * dnr_expired for the next scan that sends TURs.
+ */
if (sas_device_priv_data->block) {
sdev_printk(KERN_WARNING, sdev,
"device_unblocked handle(0x%04x)\n",
@@ -4225,6 +4377,9 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
continue;
}
+ if (!_scsih_claim_block(ioc, sdev, sas_device_priv_data,
+ &block_seq))
+ continue;
/*
* Polling can take minutes, and sdev_destroy() and
* scsi_target_reap() can free the hostdata meanwhile.
@@ -4255,11 +4410,13 @@ _scsih_ublock_io_all_device(struct MPT3SAS_ADAPTER *ioc, u8 no_turs)
!_scsih_unresponding_scan_cancelled(ioc));
if (_scsih_unresponding_scan_cancelled(ioc)) {
/* a reset queues a new scan, which polls the rest */
+ _scsih_cancel_poll(ioc, sdev, sas_device_priv_data,
+ block_seq);
scsi_device_put(sdev);
return;
}
if (!_scsih_end_poll(ioc, sdev, sas_device_priv_data,
- rc == DEVICE_READY, &ids))
+ block_seq, rc == DEVICE_READY, &ids))
continue;
if (rc != DEVICE_READY) {
@@ -4318,6 +4475,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
struct _scsih_target_ids ids;
u8 tr_timeout = 30;
u8 tr_method = 0;
+ unsigned int block_seq;
int count;
unsigned long flags;
@@ -4333,6 +4491,8 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
if (sas_target->sas_address != sas_address ||
sas_target->port != port)
goto next;
+ /* polling one device can outlast another's deadline */
+ sas_device_priv_data->block_deadline = 0;
if (sdev->sdev_state == SDEV_OFFLINE) {
sas_device_priv_data->block = 1;
sas_device_priv_data->deleted = 0;
@@ -4354,7 +4514,8 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
if (sas_target->sas_address != sas_address ||
sas_target->port != port)
continue;
- if (!sas_device_priv_data->block)
+ if (!_scsih_claim_block(ioc, sdev, sas_device_priv_data,
+ &block_seq))
continue;
/*
@@ -4405,7 +4566,7 @@ _scsih_ublock_io_device_wait(struct MPT3SAS_ADAPTER *ioc, u64 sas_address,
&& count++ <= command_retry_count);
if (!_scsih_end_poll(ioc, sdev, sas_device_priv_data,
- rc == DEVICE_READY, &ids))
+ block_seq, rc == DEVICE_READY, &ids))
continue;
if (rc != DEVICE_READY) {
@@ -4472,6 +4633,8 @@ _scsih_ublock_io_device(struct MPT3SAS_ADAPTER *ioc,
goto next;
if (sas_device_priv_data->sas_target->port != port)
goto next;
+ /* the caller has decided this device won't be polled */
+ sas_device_priv_data->dnr_expired = 0;
if (sas_device_priv_data->block)
_scsih_internal_device_unblock(sdev,
sas_device_priv_data);
@@ -4485,19 +4648,26 @@ _scsih_ublock_io_device(struct MPT3SAS_ADAPTER *ioc,
* @ioc: per adapter object
*
* During device pull we need to appropriately set the sdev state.
+ *
+ * The caller unblocks the devices when its broadcast primitive pass ends,
+ * and dnr_unblock_work leaves them alone until then. The blocks still
+ * get a deadline: if a host reset skips that unblock, the next unblock is
+ * after the firmware event worker removes unresponding devices, which
+ * needs the locks a scan waiting on one of these devices holds.
*/
static void
_scsih_block_io_all_device(struct MPT3SAS_ADAPTER *ioc)
{
struct MPT3SAS_DEVICE *sas_device_priv_data;
struct scsi_device *sdev;
- unsigned long flags;
+ unsigned long flags, deadline;
shost_for_each_device(sdev, ioc->shost) {
spin_lock_irqsave(&ioc->block_lock, flags);
sas_device_priv_data = sdev->hostdata;
if (!sas_device_priv_data)
goto next;
+ /* it keeps its deadline, if it has one */
if (sas_device_priv_data->block)
goto next;
if (sas_device_priv_data->ignore_delay_remove) {
@@ -4506,12 +4676,124 @@ _scsih_block_io_all_device(struct MPT3SAS_ADAPTER *ioc)
__func__, sas_device_priv_data->sas_target->handle);
goto next;
}
+ /*
+ * Leave offline devices alone: they take no I/O, and the
+ * caller's unblock, or an expiry, would bring them back
+ * running without a TUR check. That includes devices that
+ * failed their TUR check, which are marked deleted under
+ * block_lock before they are set offline.
+ */
+ if (sdev->sdev_state == SDEV_OFFLINE ||
+ sas_device_priv_data->deleted)
+ goto next;
_scsih_internal_device_block(sdev, sas_device_priv_data);
+ deadline = (jiffies + _scsih_dnr_block_timeout(ioc,
+ sas_device_priv_data->sas_target)) ?: 1;
+ sas_device_priv_data->block_deadline = deadline;
+ sas_device_priv_data->block_extends = 0;
+ __scsih_dnr_unblock_work_by(ioc, deadline);
next:
spin_unlock_irqrestore(&ioc->block_lock, flags);
}
}
+/**
+ * _scsih_dnr_unblock_sdev - release an expired delayed not-responding block
+ * @ioc: per adapter object
+ * @sdev: scsi device struct
+ *
+ * Return: the deadline of an unexpired block on @sdev, or 0 if none.
+ */
+static unsigned long
+_scsih_dnr_unblock_sdev(struct MPT3SAS_ADAPTER *ioc, struct scsi_device *sdev)
+{
+ struct MPT3SAS_DEVICE *sas_device_priv_data;
+ unsigned long flags, deadline = 0;
+
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ sas_device_priv_data = sdev->hostdata;
+ if (!sas_device_priv_data || !sas_device_priv_data->block ||
+ !sas_device_priv_data->block_deadline)
+ goto out;
+ if (time_before(jiffies, sas_device_priv_data->block_deadline)) {
+ deadline = sas_device_priv_data->block_deadline;
+ goto out;
+ }
+ /*
+ * Wait out a host reset, which can begin while
+ * _scsih_dnr_unblock_work() walks the devices. Don't wait for SCSI
+ * EH, though: scsi_send_eh_cmnd() waits out SDEV_BLOCK only up to
+ * the EH command timeout, so EH would fail its TURs to a device
+ * that has come back and offline it, while the event worker's
+ * unblock waits for EH to end. Past the deadline, the device is
+ * back, or the firmware has reported it missing and its handle is
+ * no longer valid.
+ *
+ * A broadcast primitive pass keeps every device blocked while it
+ * queries and aborts their commands, and unblocks them when it ends.
+ */
+ if (ioc->shost_recovery || ioc->pci_error_recovery ||
+ READ_ONCE(ioc->broadcast_tm_active)) {
+ deadline = (jiffies + HZ) ?: 1;
+ goto out;
+ }
+ sdev_printk(KERN_WARNING, sdev,
+ "delayed not-responding block expired, handle(0x%04x)\n",
+ sas_device_priv_data->sas_target->handle);
+ _scsih_internal_device_unblock(sdev, sas_device_priv_data);
+ /* if the device came back, the firmware event worker still polls it */
+ if (issue_scsi_cmd_to_bringup_drive)
+ sas_device_priv_data->dnr_expired = 1;
+out:
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ return deadline;
+}
+
+/**
+ * _scsih_dnr_unblock_work - release expired delayed not-responding blocks
+ * @work: the dnr_unblock_work of the adapter
+ *
+ * Within the device missing delay, the firmware either reports the device
+ * as not responding, and the event callback unblocks it, or the device
+ * comes back, and only the firmware event worker unblocks it. The worker
+ * may first need shost->scan_mutex or the SAS transport's host lock, and
+ * a scan holding those can have a command queued on the blocked sdev.
+ * Past the delay the block has no further use, so release it here.
+ */
+static void
+_scsih_dnr_unblock_work(struct work_struct *work)
+{
+ struct MPT3SAS_ADAPTER *ioc = container_of(to_delayed_work(work),
+ struct MPT3SAS_ADAPTER, dnr_unblock_work);
+ struct scsi_device *sdev;
+ unsigned long flags, deadline, next = 0;
+
+ /* from here on, a new block requeues the work */
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ ioc->dnr_unblock_deadline = 0;
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+
+ if (ioc->remove_host)
+ return;
+
+ shost_for_each_device(sdev, ioc->shost) {
+ deadline = _scsih_dnr_unblock_sdev(ioc, sdev);
+ if (deadline && (!next || time_before(deadline, next)))
+ next = deadline;
+ }
+
+ /*
+ * Requeue for the earliest deadline seen here; the next run
+ * requeues for the rest. A block made meanwhile may already
+ * have requeued the work for an earlier deadline; keep that.
+ */
+ if (next) {
+ spin_lock_irqsave(&ioc->block_lock, flags);
+ __scsih_dnr_unblock_work_by(ioc, next);
+ spin_unlock_irqrestore(&ioc->block_lock, flags);
+ }
+}
+
/**
* _scsih_block_io_device - set the device state to SDEV_BLOCK
* @ioc: per adapter object
@@ -4525,7 +4807,7 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
struct MPT3SAS_DEVICE *sas_device_priv_data;
struct scsi_device *sdev;
struct _sas_device *sas_device;
- unsigned long flags;
+ unsigned long flags, deadline;
sas_device = mpt3sas_get_sdev_by_handle(ioc, handle);
@@ -4536,10 +4818,30 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
goto next;
if (sas_device_priv_data->sas_target->handle != handle)
goto next;
- if (sas_device_priv_data->block)
+ deadline = (jiffies + _scsih_dnr_block_timeout(ioc,
+ sas_device_priv_data->sas_target)) ?: 1;
+ if (sas_device_priv_data->block) {
+ /* a repeat DNR restarts the firmware's missing delay */
+ if (sas_device_priv_data->block_deadline &&
+ sas_device_priv_data->block_extends <
+ MPT3SAS_DNR_BLOCK_MAX_EXTENDS) {
+ sas_device_priv_data->block_extends++;
+ sas_device_priv_data->block_deadline = deadline;
+ __scsih_dnr_unblock_work_by(ioc, deadline);
+ }
goto next;
+ }
if (sas_device && sas_device->pend_sas_rphy_add)
goto next;
+ /*
+ * Expiry would bring it back running without a TUR check.
+ * _scsih_ublock_io_device_wait() polls offline devices anyway.
+ * A device that failed its TUR check is marked deleted under
+ * block_lock, before it is set offline.
+ */
+ if (sdev->sdev_state == SDEV_OFFLINE ||
+ sas_device_priv_data->deleted)
+ goto next;
if (sas_device_priv_data->ignore_delay_remove) {
sdev_printk(KERN_INFO, sdev,
"%s skip device_block for SES handle(0x%04x)\n",
@@ -4547,6 +4849,9 @@ _scsih_block_io_device(struct MPT3SAS_ADAPTER *ioc, u16 handle)
goto next;
}
_scsih_internal_device_block(sdev, sas_device_priv_data);
+ sas_device_priv_data->block_deadline = deadline;
+ sas_device_priv_data->block_extends = 0;
+ __scsih_dnr_unblock_work_by(ioc, deadline);
next:
spin_unlock_irqrestore(&ioc->block_lock, flags);
}
@@ -10215,6 +10520,7 @@ _scsih_sas_broadcast_primitive_event(struct MPT3SAS_ADAPTER *ioc,
ioc_info(ioc, "%s: enter: phy number(%d), width(%d)\n",
__func__, event_data->PhyNum, event_data->PortWidth);
+ WRITE_ONCE(ioc->broadcast_tm_active, 1);
_scsih_block_io_all_device(ioc);
spin_lock_irqsave(&ioc->scsi_lookup_lock, flags);
@@ -10345,6 +10651,7 @@ _scsih_sas_broadcast_primitive_event(struct MPT3SAS_ADAPTER *ioc,
ioc->broadcast_aen_busy = 0;
if (!ioc->shost_recovery)
_scsih_ublock_io_all_device(ioc, 1);
+ WRITE_ONCE(ioc->broadcast_tm_active, 0);
mutex_unlock(&ioc->tm_cmds.mutex);
}
@@ -12846,6 +13153,7 @@ static void scsih_remove(struct pci_dev *pdev)
return;
ioc->remove_host = 1;
+ disable_delayed_work_sync(&ioc->dnr_unblock_work);
if (!pci_device_is_present(pdev)) {
mpt3sas_base_pause_mq_polling(ioc);
@@ -12946,6 +13254,7 @@ scsih_shutdown(struct pci_dev *pdev)
return;
ioc->remove_host = 1;
+ disable_delayed_work_sync(&ioc->dnr_unblock_work);
if (!pci_device_is_present(pdev)) {
mpt3sas_base_pause_mq_polling(ioc);
@@ -13843,6 +14152,14 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
spin_lock_init(&ioc->pcie_device_lock);
spin_lock_init(&ioc->diag_trigger_lock);
spin_lock_init(&ioc->block_lock);
+ /*
+ * Until PCIe IO Unit Page 1 is read, assume the longest delay the
+ * firmware can be set to, so blocked PCIe devices aren't released
+ * before the firmware has reported them missing. A reset keeps
+ * the delay read before if it can't read the page.
+ */
+ ioc->pcie_device_missing_delay =
+ MPI26_PCIEIOUNIT1_DMDRPT_DELAY_TIME_MASK * 16;
INIT_LIST_HEAD(&ioc->sas_device_list);
INIT_LIST_HEAD(&ioc->sas_device_init_list);
@@ -13859,6 +14176,7 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
INIT_LIST_HEAD(&ioc->delayed_tr_volume_list);
INIT_LIST_HEAD(&ioc->reply_queue_list);
INIT_LIST_HEAD(&ioc->port_table_list);
+ INIT_DELAYED_WORK(&ioc->dnr_unblock_work, _scsih_dnr_unblock_work);
sprintf(ioc->name, "%s_cm%d", ioc->driver_name, ioc->id);
@@ -13969,6 +14287,7 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
mpt3sas_setup_debugfs(ioc);
return 0;
out_add_shost_fail:
+ disable_delayed_work_sync(&ioc->dnr_unblock_work);
mpt3sas_base_detach(ioc);
out_attach_fail:
destroy_workqueue(ioc->firmware_event_thread);
--
2.56.1
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs
2026-11-18 23:09 [PATCH v3 00/11] scsi: mpt3sas: Fix delayed not-responding block deadlock and related bugs Jeff Layton
` (10 preceding siblings ...)
2026-11-18 23:10 ` [PATCH v3 11/11] scsi: mpt3sas: Time out delayed not-responding device blocks Jeff Layton
@ 2026-11-18 23:13 ` Jeff Layton
11 siblings, 0 replies; 13+ messages in thread
From: Jeff Layton @ 2026-11-18 23:13 UTC (permalink / raw)
To: Sathya Prakash, Sreekanth Reddy, Suganath Prabu Subramani,
Ranjan Kumar, James E.J. Bottomley, Martin K. Petersen,
Nagalakshmi Nandigama, James Bottomley
Cc: Keith Busch, Sreekanth Reddy, MPT-FusionLinux.pdl, linux-scsi,
linux-kernel
On Wed, 2026-11-18 at 15:09 -0800, Jeff Layton wrote:
> I posted v2 some time ago. Since then, I did a lot of adversarial review
> with AI agents. The review found a large number of bugs. Many of these
> bugs were preexisting.
>
> This series fixes a deadlock that occurs regularly in Meta's fleet. The
> fix unblocks sdevs after a timeout, and this breaks the deadlock.
>
> I don't do much SCSI driver work, but this problem came to me. I
> relied heavily on AI agents for this series.
>
> If this series is acceptable, I would like to see it in linux-next
> soon.
>
> Thanks,
> Jeff
>
> Signed-off-by: Jeff Layton <jlayton@kernel.org>
> ---
> Changes in v3:
> - New patches 1-8 and 10 fix pre-existing bugs in the code that this
> series changes:
> - 1-2: fix UAFs and refcounting in the cleanup of the fw event queue.
> Do not leave requeued events pending at teardown.
> - 3: requeue topology events only for busy devices. Do not requeue
> them for devices that the driver cannot add.
> - 4-7: fix these TUR polling bugs:
> - a pcie_device leak
> - a lost unblock retry
> - polling that continues after the scan is cancelled
> - a retry count that is not per device
> - 8: cache the missing delays that the IOC actually uses
> - 10: do not use freed hostdata while the driver polls devices
> - Patch 9: also publish sdev->hostdata under ioc->block_lock in
> scsih_sdev_init()
> - Patch 11:
> - Use the PCIe missing delay (from PCIe IO Unit Page 1) for NVMe
> devices.
> - Arm dnr_unblock_work for the earliest deadline.
> - Skip the timed unblock during host reset, but not during SCSI EH.
> - Clear the deadline of an sdev under ioc->block_lock before the fw
> event worker polls the sdev.
> - Do not put a timed block on offline sdevs.
> - When a hard reset cancels polling, give new deadlines to the sdevs
> that stay blocked.
> - Give deadlines to the sdevs that a broadcast primitive pass blocks.
> Prevent dnr_unblock_work from unblocking them while the pass runs.
> - A repeat DNR restarts the deadline, at most
> MPT3SAS_DNR_BLOCK_MAX_EXTENDS times.
> - At the next worker unblock, do a TUR check on expired sdevs.
> - Keep the blocks that the driver makes during polling.
> - Use system_dfl_wq instead of system_wq.
> - Link to v2: https://lore.kernel.org/r/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org
>
> Changes in v2:
> - New patch 1: use a new ioc->block_lock to serialize the
> interrupt-time sdev block with the hostdata teardown in
> scsih_sdev_destroy(). This fixes a pre-existing UAF.
> - Do the timed unblock under ioc->block_lock. This prevents a race with
> an interrupt-time block.
> - Clear block_deadline before the fw event worker polls a returning
> device with TURs. Then dnr_unblock_work cannot unblock the device
> during the poll.
> - Link to v1: https://lore.kernel.org/r/20261030-sas-abba-v1-1-b8cdd714a37c@kernel.org
>
> ---
> Jeff Layton (11):
> scsi: mpt3sas: Don't leave requeued firmware events pending at teardown
> scsi: mpt3sas: Hold event references while cleaning up the event queue
> scsi: mpt3sas: Requeue topology events only for busy devices
> scsi: mpt3sas: Fix pcie_device leak in _scsih_ublock_io_device_wait()
> scsi: mpt3sas: Restore unblock retry in _scsih_ublock_io_device_wait()
> scsi: mpt3sas: Stop polling devices when the unresponding scan is cancelled
> scsi: mpt3sas: Reset the TUR retry count for each device
> scsi: mpt3sas: Cache the missing delays the IOC actually uses
> scsi: mpt3sas: Serialize interrupt-time sdev block with hostdata teardown
> scsi: mpt3sas: Don't use freed hostdata while polling devices
> scsi: mpt3sas: Time out delayed not-responding device blocks
>
> drivers/scsi/mpt3sas/mpt3sas_base.c | 62 ++-
> drivers/scsi/mpt3sas/mpt3sas_base.h | 25 +
> drivers/scsi/mpt3sas/mpt3sas_scsih.c | 869 +++++++++++++++++++++++++++++------
> 3 files changed, 810 insertions(+), 146 deletions(-)
> ---
> base-commit: e8017d29067fd1310a65c3494ce1a1b492adbac6
> change-id: 20261030-sas-abba-eacfcc45fe36
>
> Best regards,
My apologies, this should have been sent as v4 of the series. I had
forgotten that I had sent a v3 a couple of weeks ago. Please treat this
series as a v4. The v3 series is here:
https://lore.kernel.org/linux-scsi/20261102-sas-abba-v2-0-b3b9ba874b5a@kernel.org/
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 13+ messages in thread