From: Qiliang Yuan <qiliang.yuan@linux.dev>
To: Ming Lei <tom.leiming@gmail.com>
Cc: Qiliang Yuan <qiliang.yuan@linux.dev>,
"Martin K. Petersen" <mkp@kernel.org>,
Shuah Khan <shuah@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
Randy Dunlap <rdunlap@infradead.org>,
linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
target-devel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-doc@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
io-uring@vger.kernel.org,
Mike Christie <michael.christie@oracle.com>,
Bodo Stroesser <bostroesser@gmail.com>
Subject: Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore
Date: Wed, 18 Nov 2026 20:02:33 -0500 [thread overview]
Message-ID: <20261119010233.2917434-1-qiliang.yuan@linux.dev> (raw)
In-Reply-To: <av0CAptnbIZP-53U@fedora-laptop>
On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote:
> Can you share your setup? It is supposed that you run fio from VM,
> otherwise you can just compare ublk with scsi backed with tcmu.
>
> But if you run fio from VM, I believe you can setup virtio-scsi backed by
> ublk block device directly, which should cover your requirement, or maybe
> I miss something?
Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16
sessions over 127.0.0.1), fio runs in that guest on the initiator
disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the
vhost-scsi disk, with the target and the backstore server in the outer
guest. In the ublk runs, kublk serves the file as /dev/ublkbN and
target_core_iblock exports that through the same fabric.
You are not missing anything. This backstore has block semantics: the
kernel emulates the SCSI commands, PR and ALUA included, exactly as for
iblock, so ublk behind target_core_iblock offers the same function on
every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of
the ublk device covers it as well. What the series adds over that is
only performance, and outside tcm_loop the numbers I sent do not show a
consistent gain.
[...]
> My question is actually why there is regression wrt. 4k randread:
>
> workload TCM/USER (MB/s) uring (MB/s) change
> 4k randread, 1 job 426 (409-463) 369 (366-384) -13%
>
> Maybe you can focus on single job perf only, which is easy to compare and
> investigate.
It comes from the scheduler moving fio between CPUs. The pwrite() I
pointed to in my last mail is not it: doing that copy inside
io_uring_enter() instead makes no difference.
tcm_loop queues the LIO submission and completion work on the CPU that
submitted the command, so with one job, fio's CPU always has bound
kworkers and ksoftirqd runnable next to fio. The load balancer cannot
move those and moves fio instead. This backstore gives it more chances:
the completion work on fio's CPU handles 14 commands per run here
against 24 with TCM/USER, so per 1000 I/Os the completion kworker and
ksoftirqd run about 57 times against 35, fio is preempted as often, and
it is moved two to three times as often. With balancing on, fio's own
CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us
with TCM/USER, spread over the whole submission path rather than in one
function.
Turning load balancing off for fio's CPUs and nothing else reverses the
result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run,
five alternating runs, median (min-max). fio runs in a cpuset of CPUs
4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either
a normal member (balanced) or an isolated partition (not balanced):
fio CPUs 4-31, backstore bandwidth (MB/s) change
balanced, TCM/USER 439 (398-462)
balanced, uring 379 (371-388) -13.7%
not balanced, TCM/USER 662 (544-710)
not balanced, uring 760 (575-798) +14.8%
Booting the guest with idle=poll removes the gap as well: with fio free
to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s
here.
That points at what each switch costs in a guest. An idle vCPU executes
HLT, which exits to the host and puts the vCPU thread to sleep; waking
it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host
wakeup and a VM entry, and the host may run the thread on another
physical CPU, so the guest code then runs with cold caches. With
idle=poll the vCPUs spin instead, stay on their physical CPUs and take
the IPI without an exit. This backstore makes more of those switches
per I/O, so it pays more of that cost, and fio's slowdown spread over
the whole submission path looks like cold caches.
What is measured is that idle=poll removes the gap and that the gap
follows the switches and migrations. Which part of the cost is the exit
and wakeup latency and which part is the vCPU thread landing on another
physical CPU is inferred: I cannot trace the host side here. Waking an
idle CPU is much cheaper on bare metal, so I expect the gap to be
smaller there, but I have not measured it. idle=poll is only a
diagnostic, it burns a host CPU per idle vCPU.
[...]
> When userspace target becomes more efficient, I'd suggest you to think
> about why these applications can't be improved or refactored for the
> potential (bigger)perf requirement.
Agreed. Since this backstore only takes block commands, an application
that can move to it can just as well move to ublk, and one that needs
SCSI semantics in userspace is not served by it either.
Thanks,
Qiliang
prev parent reply other threads:[~2026-11-19 1:02 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 0:17 Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 01/18] scsi: target: uring: Introduce " Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 02/18] scsi: target: uring: Allocate per-queue command descriptors Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 03/18] scsi: target: uring: Create a character device per backstore device Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 04/18] scsi: target: uring: Map the command descriptors into the server Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 05/18] scsi: target: uring: Queue commands until a server fetches them Qiliang Yuan
2026-11-16 0:17 ` [PATCH RFC 06/18] scsi: target: uring: Hand commands out through TCMUR_IO_FETCH Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 07/18] scsi: target: uring: Complete commands with TCMUR_IO_COMMIT_AND_FETCH Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 08/18] scsi: target: uring: Copy command data with pread() and pwrite() Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 09/18] scsi: target: uring: Fail in-flight commands when the server exits Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 10/18] scsi: target: uring: Time out commands the server does not complete Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 11/18] scsi: target: uring: Drop aborted commands that are still queued Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 12/18] scsi: target: uring: Support SYNCHRONIZE CACHE Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 13/18] scsi: target: uring: Implement UNMAP Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 14/18] scsi: target: uring: Handle WRITE SAME Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 15/18] scsi: target: uring: Register command data as io_uring fixed buffers Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 16/18] selftests: target_core_uring: Add a minimal server Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 17/18] selftests: target_core_uring: Test a device exported through tcm_loop Qiliang Yuan
2026-11-16 0:18 ` [PATCH RFC 18/18] Documentation: target: Describe the uring backstore Qiliang Yuan
2026-11-16 2:32 ` [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Ming Lei
2026-11-17 2:46 ` Qiliang Yuan
2026-11-18 1:40 ` Ming Lei
2026-11-19 1:02 ` Qiliang Yuan [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119010233.2917434-1-qiliang.yuan@linux.dev \
--to=qiliang.yuan@linux.dev \
--cc=axboe@kernel.dk \
--cc=bostroesser@gmail.com \
--cc=corbet@lwn.net \
--cc=io-uring@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=michael.christie@oracle.com \
--cc=mkp@kernel.org \
--cc=rdunlap@infradead.org \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=target-devel@vger.kernel.org \
--cc=tom.leiming@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®