* [PATCH v3 1/4] scsi: target: iscsi: Complete an aborted WRITE stuck before the first R2T
2026-10-26 12:43 [PATCH v3 0/4] scsi: target: iscsi: fix stalled and doubly completed aborted WRITEs Laurence Oberman
@ 2026-10-26 12:43 ` Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 2/4] scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent Laurence Oberman
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Laurence Oberman @ 2026-10-26 12:43 UTC (permalink / raw)
To: mkp
Cc: linux-scsi, target-devel, mlombard, michael.christie, bvanassche,
linux-kernel, stable
A WRITE that arrives with immediate data, and whose data length exceeds
FirstBurstLength, is left in TRANSPORT_WRITE_PENDING by
iscsit_execute_cmd() until the first R2T is built. If a LUN_RESET or
ABORT_TASK from another session sets CMD_T_ABORTED in that window,
iscsit_execute_cmd() returns without sending the R2T.
Nothing then completes the command. No R2T has been sent, so the
initiator cannot legally send more data; no DataOut timer was armed, so
the ERL=0 timeout cannot fire either. The command stays in
TRANSPORT_WRITE_PENDING with CMD_T_ABORTED set, and
core_tmr_drain_state_list() blocks in target_put_cmd_and_wait() on it for
as long as the connection stays up:
INFO: task kworker/5:2 blocked for more than 46 seconds.
target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
core_tmr_drain_state_list+0x2dc/0x380 [target_core_mod]
core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]
Commit aaa00cc93c1d ("scsi: target/core: Fix TAS handling for
aborted commands") is what left it that way. Before it, this site
called transport_check_aborted_status(), which sent the delayed
TASK_ABORTED status for the command; that commit replaced the call
with a bare CMD_T_ABORTED test and a return, on the basis that the
target core would send TASK_ABORTED from the ABORT TMF context
instead. That holds only for commands the core can finish on its
own. A WRITE still in TRANSPORT_WRITE_PENDING has never been passed
to the backend, so the core's abort path is itself waiting in
target_put_cmd_and_wait() for the fabric to complete it, and since
that commit the fabric no longer does.
Complete it in iscsit_execute_cmd() instead. No DataOut path can race
with this: without an R2T the initiator cannot send further data for the
command, and the branch is only reached when no unsolicited DataOut is
expected either.
That introduces a second site which can complete a command that
iscsit_release_commands_from_conn() would also complete during connection
teardown, because t_state is still TRANSPORT_WRITE_PENDING there.
target_complete_cmd() on a command that already has CMD_T_ABORTED set
goes straight into target_cmd_interrupted(), which queues
target_abort_work and returns without setting CMD_T_COMPLETE and without
changing t_state, so nothing in target core records that a completion has
happened. A second target_complete_cmd() re-initialises and re-queues
se_cmd->work while it is still queued and drops a command reference that
was never taken. Add a one-shot claim, taken under se_cmd->t_state_lock,
so that only one of the two sites completes the command.
This was reported against an Oracle RAC cluster on two-session iSCSI LIO
targets, where one node's LUN_RESET aborts WRITEs belonging to the other
node's session. It was not reproducible in a lab, so the diagnosis came
from instrumented kernels in the reporter's environment. Across two
hosts, every stalled command had the same signature: DMA_TO_DEVICE,
TRANSPORT_WRITE_PENDING, CMD_T_ABORTED | CMD_T_TAS, outstanding_r2ts == 0,
write_data_done == first_burst_len, and an empty cmd_r2t_list. 217 of 222
LUN_RESET drains completed in under a second; the rest took 46 to 54
seconds, which is where the hung-task panics came from. One 53.7 second
stall recovered on its own and the host ran for a further 19 hours. The
stall only ever ended when the initiator gave up and dropped the
connection, which let iscsit_release_commands_from_conn() complete the
command.
Fixes: aaa00cc93c1d ("scsi: target/core: Fix TAS handling for aborted commands")
Cc: stable@vger.kernel.org
Signed-off-by: Laurence Oberman <loberman@redhat.com>
---
drivers/target/iscsi/iscsi_target.c | 12 +++++--
drivers/target/iscsi/iscsi_target_erl1.c | 10 +++++-
drivers/target/iscsi/iscsi_target_util.c | 46 ++++++++++++++++++++++++
drivers/target/iscsi/iscsi_target_util.h | 2 ++
include/target/iscsi/iscsi_target_core.h | 6 ++++
5 files changed, 73 insertions(+), 3 deletions(-)
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 124ff26..aab9237 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -4198,10 +4198,18 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn)
if (cmd->se_cmd.t_state == TRANSPORT_WRITE_PENDING) {
/*
* We never submitted the cmd to LIO core, so we have
- * to tell LIO to perform the completion process.
+ * to tell LIO to perform the completion process --
+ * unless iscsit_execute_cmd() already did. t_state
+ * stays TRANSPORT_WRITE_PENDING across that
+ * completion, so the claim is the only thing that can
+ * tell us.
*/
+ bool claimed = iscsit_claim_completion_locked(cmd);
+
spin_unlock_irq(&se_cmd->t_state_lock);
- target_complete_cmd(&cmd->se_cmd, SAM_STAT_TASK_ABORTED);
+ if (claimed)
+ target_complete_cmd(&cmd->se_cmd,
+ SAM_STAT_TASK_ABORTED);
continue;
}
spin_unlock_irq(&se_cmd->t_state_lock);
diff --git a/drivers/target/iscsi/iscsi_target_erl1.c b/drivers/target/iscsi/iscsi_target_erl1.c
index d8ca06e..cf72c79 100644
--- a/drivers/target/iscsi/iscsi_target_erl1.c
+++ b/drivers/target/iscsi/iscsi_target_erl1.c
@@ -927,8 +927,16 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
if (!(cmd->cmd_flags &
ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
- if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
+ /*
+ * No R2T has been sent for this command yet,
+ * so returning here leaves the initiator with
+ * nothing to send and no DataOut timer armed.
+ * Complete it, or nothing ever will.
+ */
+ if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+ iscsit_complete_aborted_cmd(cmd);
return 0;
+ }
iscsit_set_dataout_sequence_values(cmd);
conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index c1888c4..31248cb 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -14,6 +14,7 @@
#include <scsi/scsi_tcq.h>
#include <scsi/iscsi_proto.h>
#include <target/target_core_base.h>
+#include <target/target_core_backend.h>
#include <target/target_core_fabric.h>
#include <target/iscsi/iscsi_transport.h>
@@ -186,6 +187,51 @@ struct iscsit_cmd *iscsit_allocate_cmd(struct iscsit_conn *conn, int state)
}
EXPORT_SYMBOL(iscsit_allocate_cmd);
+/*
+ * A WRITE command still in TRANSPORT_WRITE_PENDING has never been passed to
+ * the backend, so when a TMR aborts it the fabric has to complete it. More
+ * than one site can reach that conclusion for the same command, and
+ * target_complete_cmd() on a command that already has CMD_T_ABORTED set
+ * neither sets CMD_T_COMPLETE nor changes t_state, so target core keeps no
+ * state that would stop a second completion. Serialise them here: whichever
+ * site claims the completion performs it and every other site leaves the
+ * command alone.
+ *
+ * Caller must hold cmd->se_cmd.t_state_lock.
+ */
+bool iscsit_claim_completion_locked(struct iscsit_cmd *cmd)
+{
+ lockdep_assert_held(&cmd->se_cmd.t_state_lock);
+
+ if (cmd->completion_claimed)
+ return false;
+
+ cmd->completion_claimed = true;
+ return true;
+}
+
+/* Complete an aborted command at most once. Must not hold t_state_lock. */
+void iscsit_complete_aborted_cmd(struct iscsit_cmd *cmd)
+{
+ struct se_cmd *se_cmd = &cmd->se_cmd;
+ unsigned long flags;
+ bool claimed;
+
+ spin_lock_irqsave(&se_cmd->t_state_lock, flags);
+ claimed = iscsit_claim_completion_locked(cmd);
+ spin_unlock_irqrestore(&se_cmd->t_state_lock, flags);
+
+ if (!claimed)
+ return;
+
+ /*
+ * No-op unless a DataOut timer was armed for this command; callers
+ * that complete before the first R2T have none.
+ */
+ iscsit_stop_dataout_timer(cmd);
+ target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
+}
+
struct iscsi_seq *iscsit_get_seq_holder_for_datain(
struct iscsit_cmd *cmd,
u32 seq_send_order)
diff --git a/drivers/target/iscsi/iscsi_target_util.h b/drivers/target/iscsi/iscsi_target_util.h
index 7ae48a8..8f3fe74 100644
--- a/drivers/target/iscsi/iscsi_target_util.h
+++ b/drivers/target/iscsi/iscsi_target_util.h
@@ -18,6 +18,8 @@ extern struct iscsi_r2t *iscsit_get_r2t_from_list(struct iscsit_cmd *);
extern void iscsit_free_r2t(struct iscsi_r2t *, struct iscsit_cmd *);
extern void iscsit_free_r2ts_from_list(struct iscsit_cmd *);
extern struct iscsit_cmd *iscsit_allocate_cmd(struct iscsit_conn *, int);
+extern bool iscsit_claim_completion_locked(struct iscsit_cmd *cmd);
+extern void iscsit_complete_aborted_cmd(struct iscsit_cmd *cmd);
extern struct iscsi_seq *iscsit_get_seq_holder_for_datain(struct iscsit_cmd *, u32);
extern struct iscsi_seq *iscsit_get_seq_holder_for_r2t(struct iscsit_cmd *);
extern struct iscsi_r2t *iscsit_get_holder_for_r2tsn(struct iscsit_cmd *, u32);
diff --git a/include/target/iscsi/iscsi_target_core.h b/include/target/iscsi/iscsi_target_core.h
index 51ca80a..5eb3cf2 100644
--- a/include/target/iscsi/iscsi_target_core.h
+++ b/include/target/iscsi/iscsi_target_core.h
@@ -496,6 +496,12 @@ struct iscsit_cmd {
u32 first_data_sg_off;
u32 kmapped_nents;
sense_reason_t sense_reason;
+ /*
+ * Set once one site has claimed the completion of this command.
+ * The claim is taken before target_complete_cmd() is called.
+ * Protected by se_cmd.t_state_lock.
+ */
+ bool completion_claimed;
} ____cacheline_aligned;
struct iscsi_tmr_req {
--
2.43.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v3 2/4] scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent
2026-10-26 12:43 [PATCH v3 0/4] scsi: target: iscsi: fix stalled and doubly completed aborted WRITEs Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 1/4] scsi: target: iscsi: Complete an aborted WRITE stuck before the first R2T Laurence Oberman
@ 2026-10-26 12:43 ` Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 3/4] scsi: target: iscsi: Claim the completion in __iscsit_check_dataout_hdr() Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 4/4] scsi: target: iscsi: Claim the completion on the DataOut submit path Laurence Oberman
3 siblings, 0 replies; 5+ messages in thread
From: Laurence Oberman @ 2026-10-26 12:43 UTC (permalink / raw)
To: mkp
Cc: linux-scsi, target-devel, mlombard, michael.christie, bvanassche,
linux-kernel
iscsit_check_dataout_payload() hands a command to iscsit_get_dataout() to
request the next burst whenever iscsit_check_post_dataout() returns
DATAOUT_SEND_R2T, without checking CMD_T_ABORTED first.
If a TMR aborted the command while the burst that just finished was in
flight, no further DataOut will arrive: the initiator is about to be told
the command was aborted. The CMD_T_ABORTED handling in
__iscsit_check_dataout_hdr() does not cover this, because it only runs
for DataOut PDUs belonging to R2Ts that were already issued, and
outstanding_r2ts has already been decremented to zero for the burst that
completed. The command is left in TRANSPORT_WRITE_PENDING with nothing to
complete it.
Complete it here instead, through iscsit_complete_aborted_cmd() so that
the completion is claimed. target_complete_cmd() on a command that
already has CMD_T_ABORTED set goes to target_cmd_interrupted(), which
queues target_abort_work and returns without setting CMD_T_COMPLETE or
changing t_state. The command therefore stays in TRANSPORT_WRITE_PENDING
and on conn_cmd_list until target_abort_work reaches
iscsit_aborted_task(), and a connection drop in that window would let
iscsit_release_commands_from_conn() complete it a second time. Guard on
outstanding_r2ts so that with MaxOutstandingR2T > 1 the completion is
still left to the last R2T to drain in __iscsit_check_dataout_hdr().
Signed-off-by: Laurence Oberman <loberman@redhat.com>
---
Notes:
This window was never observed in the field: instrumented kernels on two
hosts recorded no completions at this site across 300 reset cycles, and
the reproducer that triggered patch 1 never reached it. It is closed by
code inspection, for symmetry with the other paths that can leave an
aborted WRITE_PENDING command stranded.
drivers/target/iscsi/iscsi_target.c | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index aab9237..3dad7f6 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1669,6 +1669,22 @@ iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr,
if ((rc == DATAOUT_NORMAL) || (rc == DATAOUT_WITHIN_COMMAND_RECOVERY))
return 0;
else if (rc == DATAOUT_SEND_R2T) {
+ /*
+ * A new R2T would never be answered for an aborted command.
+ * The CMD_T_ABORTED handling in __iscsit_check_dataout_hdr()
+ * does not cover this: it only runs for DataOut PDUs
+ * belonging to R2Ts that were already issued, and
+ * iscsit_dataout_update_r2t() has already decremented
+ * outstanding_r2ts for the burst that just finished. Guard
+ * on outstanding_r2ts so that with MaxOutstandingR2T > 1 the
+ * completion is still left to the last R2T to drain.
+ */
+ if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+ if (cmd->outstanding_r2ts < 1)
+ iscsit_complete_aborted_cmd(cmd);
+ return 0;
+ }
+
iscsit_set_dataout_sequence_values(cmd);
conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
} else if (rc == DATAOUT_SEND_TO_TRANSPORT) {
--
2.43.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v3 3/4] scsi: target: iscsi: Claim the completion in __iscsit_check_dataout_hdr()
2026-10-26 12:43 [PATCH v3 0/4] scsi: target: iscsi: fix stalled and doubly completed aborted WRITEs Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 1/4] scsi: target: iscsi: Complete an aborted WRITE stuck before the first R2T Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 2/4] scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent Laurence Oberman
@ 2026-10-26 12:43 ` Laurence Oberman
2026-10-26 12:43 ` [PATCH v3 4/4] scsi: target: iscsi: Claim the completion on the DataOut submit path Laurence Oberman
3 siblings, 0 replies; 5+ messages in thread
From: Laurence Oberman @ 2026-10-26 12:43 UTC (permalink / raw)
To: mkp
Cc: linux-scsi, target-devel, mlombard, michael.christie, bvanassche,
linux-kernel, stable
Commit d5869dae5080 ("scsi: target: iscsi: Fix hang for aborted
WRITE_PENDING commands") made __iscsit_check_dataout_hdr() complete an
aborted WRITE once the final DataOut PDU for the last outstanding R2T has
been received. That call records nothing to say the command has been
completed.
target_complete_cmd() on a command that already has CMD_T_ABORTED set goes
to target_cmd_interrupted(), which queues target_abort_work and returns
without setting CMD_T_COMPLETE and without changing t_state. The command
therefore stays in TRANSPORT_WRITE_PENDING and on conn_cmd_list until
target_abort_work reaches iscsit_aborted_task(). If the connection drops in
that window, iscsit_release_commands_from_conn() finds the command still in
TRANSPORT_WRITE_PENDING and completes it a second time, which
re-initialises and re-queues an se_cmd->work that is already queued.
Route this site through iscsit_complete_aborted_cmd() so that the
completion is claimed and connection teardown leaves the command alone. The
helper makes the same iscsit_stop_dataout_timer() and target_complete_cmd()
calls when it takes the claim, so the completing path is unchanged.
Fixes: d5869dae5080 ("scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands")
Cc: stable@vger.kernel.org
Signed-off-by: Laurence Oberman <loberman@redhat.com>
---
Notes:
This depends on patch 1 of this series, which adds
iscsit_complete_aborted_cmd(). d5869dae5080 has been backported to 5.10.y,
5.15.y, 6.1.y, 6.6.y, 6.12.y, 6.18.y and 7.2.y, none of which carry the
helper, so a backport to those branches needs patch 1 as well, or the claim
open-coded. Happy to send a standalone version for stable if you would
rather not carry patch 1 there.
drivers/target/iscsi/iscsi_target.c | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 3dad7f6..83184de 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1533,10 +1533,8 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
*/
if (se_cmd->transport_state & CMD_T_ABORTED) {
if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
- --cmd->outstanding_r2ts < 1) {
- iscsit_stop_dataout_timer(cmd);
- target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
- }
+ --cmd->outstanding_r2ts < 1)
+ iscsit_complete_aborted_cmd(cmd);
return iscsit_dump_data_payload(conn, payload_length, 1);
}
--
2.43.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH v3 4/4] scsi: target: iscsi: Claim the completion on the DataOut submit path
2026-10-26 12:43 [PATCH v3 0/4] scsi: target: iscsi: fix stalled and doubly completed aborted WRITEs Laurence Oberman
` (2 preceding siblings ...)
2026-10-26 12:43 ` [PATCH v3 3/4] scsi: target: iscsi: Claim the completion in __iscsit_check_dataout_hdr() Laurence Oberman
@ 2026-10-26 12:43 ` Laurence Oberman
3 siblings, 0 replies; 5+ messages in thread
From: Laurence Oberman @ 2026-10-26 12:43 UTC (permalink / raw)
To: mkp
Cc: linux-scsi, target-devel, mlombard, michael.christie, bvanassche,
linux-kernel
When the final DataOut PDU for a WRITE has been received,
iscsit_check_dataout_payload() submits the command with
target_execute_cmd(). For a command that has already been aborted,
target_execute_cmd() calls target_cmd_interrupted() before it touches
t_state, so it queues target_abort_work and returns with the command still
in TRANSPORT_WRITE_PENDING and still on conn_cmd_list, recording nothing to
say that a completion has been started.
A connection drop before target_abort_work reaches iscsit_aborted_task()
then lets iscsit_release_commands_from_conn() complete the command a second
time.
Complete aborted commands through iscsit_complete_aborted_cmd() rather than
handing them to target_execute_cmd(). Both reach target_cmd_interrupted()
and queue target_abort_work; the work target_execute_cmd() does after that
test is not reached for an aborted command either way. The dataout timer
has already been stopped a few lines above, so the helper's call to
iscsit_stop_dataout_timer() is a no-op here.
Fixes: ea87981a0ee8 ("scsi: target: iscsi: Handle abort for WRITE_PENDING cmds")
Signed-off-by: Laurence Oberman <loberman@redhat.com>
---
Notes:
Not observed in the field; closed by inspection after the review of v2
pointed out that this site has the same shape as the one in
__iscsit_check_dataout_hdr(). I have deliberately left Cc: stable off,
since the window predates d5869dae5080 and has never been seen -- I would
rather it sat in mainline for a cycle first. Happy to drop this patch if
you would prefer the series to stop at patch 3.
drivers/target/iscsi/iscsi_target.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 83184de..e6113a5 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1699,6 +1699,16 @@ iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr,
iscsit_stop_dataout_timer(cmd);
if (ooo_cmdsn)
return 0;
+ /*
+ * target_execute_cmd() tests CMD_T_ABORTED before it touches
+ * t_state, so for an aborted command it would queue
+ * target_abort_work and return with the command still in
+ * TRANSPORT_WRITE_PENDING and the completion unclaimed.
+ */
+ if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+ iscsit_complete_aborted_cmd(cmd);
+ return 0;
+ }
target_execute_cmd(&cmd->se_cmd);
return 0;
} else /* DATAOUT_CANNOT_RECOVER */
--
2.43.0
^ permalink raw reply [flat|nested] 5+ messages in thread