From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=91.218.175.237 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795050165; b=gWRjsMuc6RWyEDGVaxUkk/H7vr1kdVjX4g/AReJx0arVPvXHDakhe2ndadgLWzHggv4A kPGAoFt7FSEVndFm8GQxHPTOOF4P434rVdDbny8CDjPlZ/vP7vNedZy8egTVlc5F7Ixc0 Ofi94n6I0xwXgDtDyh5LnlsXyQwp7So2/xNFgjxrKAt8jKlobyjteAZhZErMJbm6Lzw6Y 6kX5vfp+NmDVkb0wF/3GUI7IYt3SCHf+ArQk0OSEjFwnXIuWdSPxzaqGKVMSa0a5t2qt5 JHkXItI8uZYLOD5VKzFoLGgx8CUQHDJqI2VAI31gHoAfaHmlbRBC9T9iC1bSrL67Ttg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795050165; h=DMARC-Filter:X-Envelope-To:DKIM-Signature:X-Envelope-To:Received: X-Mizu-Trace-ID:X-Migadu-Flow:From:To:Cc:Subject:Date:Message-ID: X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=BlL4W75h1MApHqdiv3CTVmX4B2zbrj1hbYs9giA+q5g=; b=d1Yj1TzphU8BdN+p+tUUzUWHkCAr/i29p7Xg4jCX0L3uCSSUabogLJZTqDH/tsFexV0A b02XB0Apd9l+XIgXkDyyeKbe23vJ9L8xu4qZ3aF/dbU7v4k04PLfXL9EzcVrIVuKy18wa 274T2l4ojOMzKaeaAbVaizHap+ObpoTmxtAWR2PIv82Qq0gYEjX7yLg0G6Aeg/KlN/o3k uN8+Qvlv+iLF3/Vwy81xDy/DntOxpxqHWlxo57KrkEGwuUsmpUlhlTHHz+paHHjcNpmgo bx6E7VpAnr+HOq3EJoOSBJB+qftuobwSX5wWwvXwojJOWX73VIEsemVFEqJHIKaFqDQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=fpFl4L4R; arc=none smtp.remote-ip=91.218.175.237 Received: from mta0.migadu.com (out-237.mta0.migadu.com [91.218.175.237]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEC0D606A32F for ; Thu, 19 Nov 2026 01:02:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org BEC0D606A32F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux.dev header.i=@linux.dev header.a=rsa-sha256 header.s=key1 header.b=fpFl4L4R X-Envelope-To: linux-kernel@vger.kernel.org DKIM-Signature: a=rsa-sha256; bh=ux0kfY5j7nM7oh8a7BfUtbSzQISJcSgoiHsY5y3C7fo=; c=simple/simple; d=linux.dev; h=from:to:subject:date:message-id:mime-version:content-type; s=key1; t=1795050162; v=1; x=1795654962; b=fpFl4L4RdOYRvsiHM+GOdTXfY0EaUgQcWvxjZu5E3Iv9+BQZa/n8b9XA4f9Z5qDlBLG+tUXI lzbf8E45R6pbwSlF/bLPvWn8nZdmKdusdtPneU0j6QT68TFJKw7pUq/iGap/n11nC8WuHFZhFq1 iOkoa/h2KlJBvxgUm+WIY/0U= X-Envelope-To: linux-kernel@vger.kernel.org Received: by smtp.migadu.com with ESMTPS id 5966f3fd4a58c3d7; Thu, 19 Nov 2026 01:02:42 +0000 X-Mizu-Trace-ID: 5966f3fd4a58c3d7 X-Migadu-Flow: FLOW_OUT From: Qiliang Yuan To: Ming Lei Cc: Qiliang Yuan , "Martin K. Petersen" , Shuah Khan , Jonathan Corbet , Shuah Khan , Randy Dunlap , linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, target-devel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, Jens Axboe , io-uring@vger.kernel.org, Mike Christie , Bodo Stroesser Subject: Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore Date: Wed, 18 Nov 2026 20:02:33 -0500 Message-ID: <20261119010233.2917434-1-qiliang.yuan@linux.dev> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: <20261115-feat-target-core-uring-v1-0-28cdc137eac6@linux.dev> <20261117024640.3184554-1-qiliang.yuan@linux.dev> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote: > Can you share your setup? It is supposed that you run fio from VM, > otherwise you can just compare ublk with scsi backed with tcmu. > > But if you run fio from VM, I believe you can setup virtio-scsi backed by > ublk block device directly, which should cover your requirement, or maybe > I miss something? Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16 sessions over 127.0.0.1), fio runs in that guest on the initiator disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the vhost-scsi disk, with the target and the backstore server in the outer guest. In the ublk runs, kublk serves the file as /dev/ublkbN and target_core_iblock exports that through the same fabric. You are not missing anything. This backstore has block semantics: the kernel emulates the SCSI commands, PR and ALUA included, exactly as for iblock, so ublk behind target_core_iblock offers the same function on every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of the ublk device covers it as well. What the series adds over that is only performance, and outside tcm_loop the numbers I sent do not show a consistent gain. [...] > My question is actually why there is regression wrt. 4k randread: > > workload TCM/USER (MB/s) uring (MB/s) change > 4k randread, 1 job 426 (409-463) 369 (366-384) -13% > > Maybe you can focus on single job perf only, which is easy to compare and > investigate. It comes from the scheduler moving fio between CPUs. The pwrite() I pointed to in my last mail is not it: doing that copy inside io_uring_enter() instead makes no difference. tcm_loop queues the LIO submission and completion work on the CPU that submitted the command, so with one job, fio's CPU always has bound kworkers and ksoftirqd runnable next to fio. The load balancer cannot move those and moves fio instead. This backstore gives it more chances: the completion work on fio's CPU handles 14 commands per run here against 24 with TCM/USER, so per 1000 I/Os the completion kworker and ksoftirqd run about 57 times against 35, fio is preempted as often, and it is moved two to three times as often. With balancing on, fio's own CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us with TCM/USER, spread over the whole submission path rather than in one function. Turning load balancing off for fio's CPUs and nothing else reverses the result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run, five alternating runs, median (min-max). fio runs in a cpuset of CPUs 4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either a normal member (balanced) or an isolated partition (not balanced): fio CPUs 4-31, backstore bandwidth (MB/s) change balanced, TCM/USER 439 (398-462) balanced, uring 379 (371-388) -13.7% not balanced, TCM/USER 662 (544-710) not balanced, uring 760 (575-798) +14.8% Booting the guest with idle=poll removes the gap as well: with fio free to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s here. That points at what each switch costs in a guest. An idle vCPU executes HLT, which exits to the host and puts the vCPU thread to sleep; waking it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host wakeup and a VM entry, and the host may run the thread on another physical CPU, so the guest code then runs with cold caches. With idle=poll the vCPUs spin instead, stay on their physical CPUs and take the IPI without an exit. This backstore makes more of those switches per I/O, so it pays more of that cost, and fio's slowdown spread over the whole submission path looks like cold caches. What is measured is that idle=poll removes the gap and that the gap follows the switches and migrations. Which part of the cost is the exit and wakeup latency and which part is the vCPU thread landing on another physical CPU is inferred: I cannot trace the host side here. Waking an idle CPU is much cheaper on bare metal, so I expect the gap to be smaller there, but I have not measured it. idle=poll is only a diagnostic, it burns a host CPU per idle vCPU. [...] > When userspace target becomes more efficient, I'd suggest you to think > about why these applications can't be improved or refactored for the > potential (bigger)perf requirement. Agreed. Since this backstore only takes block commands, an application that can move to it can just as well move to ublk, and one that needs SCSI semantics in userspace is not served by it either. Thanks, Qiliang