mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Ming Lei <tom.leiming@gmail.com>
To: Qiliang Yuan <qiliang.yuan@linux.dev>
Cc: "Martin K. Petersen" <mkp@kernel.org>,
	Shuah Khan <shuah@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>,
	Randy Dunlap <rdunlap@infradead.org>,
	linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
	target-devel@vger.kernel.org, linux-kselftest@vger.kernel.org,
	linux-doc@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
	io-uring@vger.kernel.org,
	Mike Christie <michael.christie@oracle.com>,
	Bodo Stroesser <bostroesser@gmail.com>
Subject: Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore
Date: Sun, 15 Nov 2026 20:32:02 -0600	[thread overview]
Message-ID: <avprIhtn1TuYLX1b@fedora-laptop> (raw)
In-Reply-To: <20261115-feat-target-core-uring-v1-0-28cdc137eac6@linux.dev>

On Sun, Nov 15, 2026 at 07:17:53PM -0500, Qiliang Yuan wrote:
> TCM/USER is how a userspace process serves a LIO device, for instance
> to export distributed storage over iSCSI or vhost-scsi. All commands of
> a device go through one command ring under one mutex, the data of each
> command is copied into and out of a shared data area, and the server
> has to parse every SCSI CDB itself.
> 
> This series adds target_core_uring, a second user backstore next to
> TCM/USER, built the way ublk is:
> 
> - The kernel emulates the SCSI commands. The server only sees reads,
>   writes, flushes, UNMAP and WRITE SAME as an operation, an offset and a
>   length.
> - A device has several queues. Each queue is served through io_uring
>   commands on its own ring, with no lock shared between queues.
> - Data is copied once, with pread()/pwrite() on the character device, or
>   not at all when the server registers the pages of a command as an
>   io_uring fixed buffer.

This is ublk's uapi re-implemented inside LIO: FETCH/COMMIT_AND_FETCH
per tag, pread()/pwrite() with (qid, tag) in the offset, and buffer
registration via io_buffer_register_bvec(). It starts without what
ublk added afterwards (queue affinity, recovery, auto buffer
registration, batching), and the sashiko reviews already found the
same abort and lifetime races ublk fixed years ago.

Follows two alternatives, especially with zero kernel change:

1) If LIO is required (vhost-scsi, FC, SRP, iSER), serve the device
with ublk and export it via target_core_iblock. Same per-queue
io_uring, same user copy and zero copy path, no new uapi. Please
benchmark that configuration, with UBLK_F_USER_COPY and with
UBLK_F_AUTO_BUF_REG, same 32 queues and threads.

2) If the goal is iSCSI for userspace storage, keep the kernel out of
the data path entirely. A write here still goes NIC -> iscsit -> LIO
-> uring_cmd -> pread() -> storage client, and a distributed storage
client needs the data in its own memory anyway, so the zero copy path
only helps file and block backends, which iblock and fileio already
cover.

A userspace iSCSI target on io_uring receives straight into the server's
buffer, with one copy fewer and no new kernel ABI. ioutgt

		https://github.com/io-target/ioutgt

is that design: one ring per queue thread for socket and backend I/O, lock-free
data path, NVMe/TCP and NVMe/RDMA today, iSCSI on the roadmap. It already beats
the kernel target by a wide margin: about 2x nvmet on 4k NVMe/TCP random read
and write, and 2x on 64k over both TCP and RDMA, on 100GbE. Adding iSCSI
there is a bounded userspace project.
  
> Patches 1-4 create the device, its queues and the character device.
> Patches 5-8 pass reads and writes to the server. Patches 9-11 handle a
> server that exits or hangs, and aborts. Patches 12-14 add flush, UNMAP
> and WRITE SAME, patch 15 adds fixed buffers. Patches 16-18 add a small
> server, a selftest and the documentation.
> 
> Not covered yet: protection information, and a way for existing
> tcmu-runner handlers to move over without changes, as they implement
> SCSI rather than block commands.
> 
> Exporting a 2 GiB device through tcm_loop to fio (io_uring, direct I/O,
> iodepth 32 for 4k and 8 for 1M, 6 seconds per job) on a 32 CPU VM, with
> TCM/USER and a single-threaded handler, then with this series, 32 queues
> and one server thread per queue. Both servers keep the data in memory.

32 threads against one, over tcm_loop with data in memory, measures
only the hand-off. Please run a real fabric and explain the -13%
single job result.



Thanks,
Ming

      parent reply	other threads:[~2026-11-16  2:32 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  0:17 Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 01/18] scsi: target: uring: Introduce " Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 02/18] scsi: target: uring: Allocate per-queue command descriptors Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 03/18] scsi: target: uring: Create a character device per backstore device Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 04/18] scsi: target: uring: Map the command descriptors into the server Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 05/18] scsi: target: uring: Queue commands until a server fetches them Qiliang Yuan
2026-11-16  0:17 ` [PATCH RFC 06/18] scsi: target: uring: Hand commands out through TCMUR_IO_FETCH Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 07/18] scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 08/18] scsi: target: uring: Copy command data with pread() and pwrite() Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 09/18] scsi: target: uring: Fail in-flight commands when the server exits Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 10/18] scsi: target: uring: Time out commands the server does not complete Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 11/18] scsi: target: uring: Drop aborted commands that are still queued Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 12/18] scsi: target: uring: Support SYNCHRONIZE CACHE Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 13/18] scsi: target: uring: Implement UNMAP Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 14/18] scsi: target: uring: Handle WRITE SAME Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 15/18] scsi: target: uring: Register command data as io_uring fixed buffers Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 16/18] selftests: target_core_uring: Add a minimal server Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 17/18] selftests: target_core_uring: Test a device exported through tcm_loop Qiliang Yuan
2026-11-16  0:18 ` [PATCH RFC 18/18] Documentation: target: Describe the uring backstore Qiliang Yuan
2026-11-16  2:32 ` Ming Lei [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=avprIhtn1TuYLX1b@fedora-laptop \
    --to=tom.leiming@gmail.com \
    --cc=axboe@kernel.dk \
    --cc=bostroesser@gmail.com \
    --cc=corbet@lwn.net \
    --cc=io-uring@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=michael.christie@oracle.com \
    --cc=mkp@kernel.org \
    --cc=qiliang.yuan@linux.dev \
    --cc=rdunlap@infradead.org \
    --cc=shuah@kernel.org \
    --cc=skhan@linuxfoundation.org \
    --cc=target-devel@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®