[PATCH v3] scsi: lpfc: Fix race conditions in ELS retry handling

[PATCH v3] scsi: lpfc: Fix race conditions in ELS retry handling

[Kyle Mahlkuch]

This patch addresses a race condition in the lpfc driver's ELS retry
event handling that can lead to a use-after-free in the
lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared
before acquiring the lock to check if the retry event is queued. This
creates a window where the timer can be rescheduled and fire, causing
both the cancel path and the worker thread to release the same reference,
resulting in a double-put and use-after-free.

Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing
LPFC_EVT_ELS_RETRY in lpfc_work_list_done().

Signed-off-by: Kyle Mahlkuch <kmahlkuc@linux.ibm.com>
---
Changes since v2:
- Capture ndlp and clear evt_arg1 under hbalock while dequeuing
   LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by
   lpfc_els_retry_delay() could cause a NULL dereference in the worker.
   (AI review)
- Whitespace and line-wrapping cleanup.

Link: https://lore.kernel.org/lkml/29073ebc-b616-475b-bc05-384da09fbb54@linux.ibm.com/T/#u

Changes since v1:
- Leave test_and_clear_bit op and timer_delete_sync at the beginning of
   lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)

Link: https://lore.kernel.org/lkml/1c8a764c-fce1-4ce1-b797-47ac328cf3f2@linux.ibm.com/T/#u
---
  drivers/scsi/lpfc/lpfc_els.c     | 29 +++++++++++++++++++++++------
  drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++-
  2 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
index 6f6394a0047c..98d1824aefe1 100644
--- a/drivers/scsi/lpfc/lpfc_els.c
+++ b/drivers/scsi/lpfc/lpfc_els.c
@@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry)
  void
  lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct lpfc_nodelist *nlp)
  {
-	struct lpfc_work_evt *evtp;
+	struct lpfc_hba *phba = vport->phba;
+	struct lpfc_work_evt *evtp = &nlp->els_retry_evt;
+	struct lpfc_nodelist *arg_ndlp = NULL;
+	unsigned long flags;
  
  	if (!test_and_clear_bit(NLP_DELAY_TMO, &nlp->nlp_flag))
  		return;
  	timer_delete_sync(&nlp->nlp_delayfunc);
  	nlp->nlp_last_elscmd = 0;
-	if (!list_empty(&nlp->els_retry_evt.evt_listp)) {
-		list_del_init(&nlp->els_retry_evt.evt_listp);
-		/* Decrement nlp reference count held for the delayed retry */
-		evtp = &nlp->els_retry_evt;
-		lpfc_nlp_put((struct lpfc_nodelist *)evtp->evt_arg1);
+
+	/*
+	 * If retry event is queued, remove it under lock and consume
+	 * its payload to prevent double-put. The event holds a reference
+	 * to the nodelist that must be released exactly once.
+	 */
+	spin_lock_irqsave(&phba->hbalock, flags);
+	if (!list_empty(&evtp->evt_listp)) {
+		list_del_init(&evtp->evt_listp);
+		/* Take local copy and clear to prevent reuse */
+		arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1;
+		evtp->evt_arg1 = NULL;
+		smp_wmb(); /* Ensure NULL visible before unlock */
  	}
+	spin_unlock_irqrestore(&phba->hbalock, flags);
+
+	/* Drop the event-held reference outside the lock */
+	if (arg_ndlp)
+		lpfc_nlp_put(arg_ndlp);
+
  	if (test_and_clear_bit(NLP_NPR_2B_DISC, &nlp->nlp_flag)) {
  		if (vport->num_disc_nodes) {
  			if (vport->port_state < LPFC_VPORT_READY) {
diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 4c673dffa671..75457f19d4ea 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)
  	while (!list_empty(&phba->work_list)) {
  		list_remove_head((&phba->work_list), evtp, typeof(*evtp),
  				 evt_listp);
+		/*
+		 * The ELS retry event can be requeued by lpfc_els_retry_delay()
+		 * as soon as hbalock is dropped. Consume the payload within the
+		 * hbalock to prevent reuse or double-put.
+		 */
+		if (evtp->evt == LPFC_EVT_ELS_RETRY) {
+			ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1);
+			evtp->evt_arg1 = NULL;
+		}
  		spin_unlock_irq(&phba->hbalock);
  		hba_pci_err = test_bit(HBA_PCI_ERR, &phba->bit_flags);
  		free_evt = 1;
  		switch (evtp->evt) {
  		case LPFC_EVT_ELS_RETRY:
-			ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1);
+			/* ndlp was already captured under hbalock */
  			if (!hba_pci_err) {
  				lpfc_els_retry_delay_handler(ndlp);
  				free_evt = 0; /* evt is part of ndlp */
-- 
2.55.0

Re: [PATCH v3] scsi: lpfc: Fix race conditions in ELS retry handling

[Hannes Reinecke]

On 11/10/26 9:16 PM, Kyle Mahlkuch wrote:
> This patch addresses a race condition in the lpfc driver's ELS retry
> event handling that can lead to a use-after-free in the
> lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared
> before acquiring the lock to check if the retry event is queued. This
> creates a window where the timer can be rescheduled and fire, causing
> both the cancel path and the worker thread to release the same reference,
> resulting in a double-put and use-after-free.
> 
> Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing
> LPFC_EVT_ELS_RETRY in lpfc_work_list_done().
> 
> Signed-off-by: Kyle Mahlkuch <kmahlkuc@linux.ibm.com>
> ---
> Changes since v2:
> - Capture ndlp and clear evt_arg1 under hbalock while dequeuing
>    LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by
>    lpfc_els_retry_delay() could cause a NULL dereference in the worker.
>    (AI review)
> - Whitespace and line-wrapping cleanup.
> 
> Link: https://lore.kernel.org/lkml/29073ebc-b616-475b- 
> bc05-384da09fbb54@linux.ibm.com/T/#u
> 
> Changes since v1:
> - Leave test_and_clear_bit op and timer_delete_sync at the beginning of
>    lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)
> 
> Link: https://lore.kernel.org/lkml/1c8a764c-fce1-4ce1- 
> b797-47ac328cf3f2@linux.ibm.com/T/#u
> ---
>   drivers/scsi/lpfc/lpfc_els.c     | 29 +++++++++++++++++++++++------
>   drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++-
>   2 files changed, 33 insertions(+), 7 deletions(-)
> 
> diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
> index 6f6394a0047c..98d1824aefe1 100644
> --- a/drivers/scsi/lpfc/lpfc_els.c
> +++ b/drivers/scsi/lpfc/lpfc_els.c
> @@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, 
> uint8_t retry)
>   void
>   lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct 
> lpfc_nodelist *nlp)
>   {
> -    struct lpfc_work_evt *evtp;
> +    struct lpfc_hba *phba = vport->phba;
> +    struct lpfc_work_evt *evtp = &nlp->els_retry_evt;
> +    struct lpfc_nodelist *arg_ndlp = NULL;
> +    unsigned long flags;
> 
>       if (!test_and_clear_bit(NLP_DELAY_TMO, &nlp->nlp_flag))
>           return;
>       timer_delete_sync(&nlp->nlp_delayfunc);
>       nlp->nlp_last_elscmd = 0;
> -    if (!list_empty(&nlp->els_retry_evt.evt_listp)) {
> -        list_del_init(&nlp->els_retry_evt.evt_listp);
> -        /* Decrement nlp reference count held for the delayed retry */
> -        evtp = &nlp->els_retry_evt;
> -        lpfc_nlp_put((struct lpfc_nodelist *)evtp->evt_arg1);
> +
> +    /*
> +     * If retry event is queued, remove it under lock and consume
> +     * its payload to prevent double-put. The event holds a reference
> +     * to the nodelist that must be released exactly once.
> +     */
> +    spin_lock_irqsave(&phba->hbalock, flags);
> +    if (!list_empty(&evtp->evt_listp)) {
> +        list_del_init(&evtp->evt_listp);
> +        /* Take local copy and clear to prevent reuse */
> +        arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1;
> +        evtp->evt_arg1 = NULL;
> +        smp_wmb(); /* Ensure NULL visible before unlock */
>       }
> +    spin_unlock_irqrestore(&phba->hbalock, flags);
> +
> +    /* Drop the event-held reference outside the lock */
> +    if (arg_ndlp)
> +        lpfc_nlp_put(arg_ndlp);
> +
>       if (test_and_clear_bit(NLP_NPR_2B_DISC, &nlp->nlp_flag)) {
>           if (vport->num_disc_nodes) {
>               if (vport->port_state < LPFC_VPORT_READY) {
> diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/ 
> lpfc_hbadisc.c
> index 4c673dffa671..75457f19d4ea 100644
> --- a/drivers/scsi/lpfc/lpfc_hbadisc.c
> +++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
> @@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)
>       while (!list_empty(&phba->work_list)) {
>           list_remove_head((&phba->work_list), evtp, typeof(*evtp),
>                    evt_listp);
> +        /*
> +         * The ELS retry event can be requeued by lpfc_els_retry_delay()
> +         * as soon as hbalock is dropped. Consume the payload within the
> +         * hbalock to prevent reuse or double-put.
> +         */
> +        if (evtp->evt == LPFC_EVT_ELS_RETRY) {
> +            ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1);
> +            evtp->evt_arg1 = NULL;
> +        }
>           spin_unlock_irq(&phba->hbalock);
>           hba_pci_err = test_bit(HBA_PCI_ERR, &phba->bit_flags);
>           free_evt = 1;
>           switch (evtp->evt) {
>           case LPFC_EVT_ELS_RETRY:
> -            ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1);
> +            /* ndlp was already captured under hbalock */
>               if (!hba_pci_err) {
>                   lpfc_els_retry_delay_handler(ndlp);
>                   free_evt = 0; /* evt is part of ndlp */

Good catch.

Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich