From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16D9E3E0C7A for ; Tue, 13 Oct 2026 12:28:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791894538; cv=none; b=dG1Q4xktmugJ4nz24z2KI2GZTkXjHRQYEHvvrL8DcD7NryYnqiGe32392VBI+6ltRnpEDjwmr+HUyWEuSTFZtxjrlBS5Rlk5gNASRtFCDK9PWAlVY4/TqgPC9Z4WP7XnRNieNju+cm24W6ILw0gABLhAB+Omf489Fs+ag37RVms= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791894538; c=relaxed/simple; bh=YveNTi8ah6bA5pnf4dMIhjxq/suH8nT/i2ueA0M6OQA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=UZ2Ddf7fVof58Fb6UX1Uz7dGIudjRc6+0Orsj9YStuUECP2rz+tQxn5FxYNaY1+aGkApyMUBxP1Y9XjyJpc2s5lRDvGhgwFn7M8eEo6YrS8h2hIsq6rzvHb89+HvYySL12yzBnFAlgYvaB6yE1P539DyGvzgPaHoRtO/GlGf6oQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=qm0BIgro; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qm0BIgro" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-c2e5e801487so426280766b.3 for ; Tue, 13 Oct 2026 05:28:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791894535; x=1792499335; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=MSrW2QsU5sD6aU9GldcitIknJmBTZQB6L4UNDgWjpRU=; b=qm0BIgroWB4dNBPgaUZBIptiUeHsQhlj0vMDwR/abHdMbwQWVqoimkiQysDhLEnlr8 mhLJTGKV/wnmyJUZhpcjL0r2xudr1DwQ11BLDQVV3Ph8azHg+nYNY3hlpZfP5qjFqUab W09v72fEJjPVkC6iLlpVto0EQUQ8HDUlkABiKC6lBUfGJADltaAW2pTKcnb4k9Fa2RSd ZfXjy/1f3EcNqS8X5p97aFo1ty4bIslQGAKwsS0dH0zH3RZQwhYukCRfYR9AIPw5FN5J /h74FEDGUZ7S5v5NuCpWQis+53+x5x4wmnkatBhmkzEKXDEBPB7wKbHFz+OMy/a1+y3q jkkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791894535; x=1792499335; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=MSrW2QsU5sD6aU9GldcitIknJmBTZQB6L4UNDgWjpRU=; b=2FRHjGtwkSE02ZKl9GcnEezTIBOAPV5maMvtNznqjPjHLLeRDruaLQgmFUMq7iZ72j tQ5l6NUupSIZgmif3jSMskdAaFTP4yX0EmsiqV3RWj1A2FJH9GfQmD7RmfMNdhSdwOeL f8c8Vi4/vRqYwpppnUsSrH4j1O14++Fa5U/LwDvpSVoUDaBXJ07+hifs+mL2N4w4s8do jOhVXhOWNUmHviLwY0OSrlKQTuC7E/OR6fy4zDctNTZ8OXaLi8Dg4s6FvWGGBg0oivYs xiIavG4vUdbUlRK1yOVYtLcGa0TzaP2E/jJWsUJKDTQsOvvhBEiYfyup4GcJJmkC2y87 i+WA== X-Forwarded-Encrypted: i=1; AKwUvBwcx3fJK5xdzfgWwYn86A1nDS+YisWvnrhwYyJ2kiSRnBvpRK++P9fUnKahrqwe6VxFxpqHUkaWzjjGhU8=@vger.kernel.org X-Gm-Message-State: AFq9FYIC1TEv95ypi5R2Bf7raICgwhTaETveAHDN08lCUo374+f4wIUe a6bajvtuuxA8olxXFvQ1Epj+aPw7dI4A7kwFuGChHZvL1dswqfiynHsC X-Gm-Gg: AYBFou0IEYNc0uQz7vZPb4s+UF3h1c+y/s3gbxhtyuDvbRKIP4pZdFpDsXCW6FK227D ofeVdO0rXGS7aUb+U1+E6aB7xcVJtPWo2I72u7/HH2rm3rJYe/ha2QE01R2CD7GBWV5felh9woX S6b0q53Y40v/9diJOXDm+rnvsox6pIP5Ks5vkPcG0VNNYuCx4t4y1wLy12O2O5P6hBfZVV4tHEI /a00FKTPvWuetd5qILaaF5L/1f+Z1TNCA5tgbkQi2h2x+EMwV0IIZWZgqfjH+KnhykVFraC33vy PETWt4NAd1fyEbbeY/nlm0y3i63rzwoMa6WNa0W0ERcqgvyZ4MwAqjHeHdoib58qIxn+U8Uv65U uKh8eWLYDJEau97Egm1On1LUCOb9P8ZbAmZqngQxEGM63fnEPQWt53QB9XCNJ61cSd7NKA2LUYX Rh7GQOoxd/wGLhCrs/OJN63J/RP3VNSxizJmA56A1j0sy55TEHHvwqwvyQ4+IWIdsrt307s1T+t 2sygWIhlchN/KydYuf/rRufqU44V9NHy8XqltIsXCzeZDiK3pBlSxnUGz+V49W1nBDrI9H4rZAb d5E= X-Received: by 2002:a17:907:7f90:b0:c26:1648:a061 with SMTP id a640c23a62f3a-c31aa04aabamr1238002866b.28.1791894535091; Tue, 13 Oct 2026 05:28:55 -0700 (PDT) Received: from fedora-laptop ([172.245.82.59]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c31a9766743sm577597466b.4.2026.10.13.05.28.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Oct 2026 05:28:54 -0700 (PDT) Date: Tue, 13 Oct 2026 07:28:45 -0500 From: Ming Lei To: Aaron Tomlin Cc: axboe@kernel.dk, hare@suse.de, sean@ashe.io, tj@kernel.org, neelx@suse.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Subject: Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty Message-ID: References: <20261012012905.267920-1-atomlin@atomlin.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Mon, Oct 12, 2026 at 12:06:25PM -0400, Aaron Tomlin wrote: > On Sun, Oct 11, 2026 at 09:51:25PM -0500, Ming Lei wrote: > > > Since kblockd_workqueue is WQ_PERCPU, the responsibility rests with blk-mq > > > to provide a non-isolated CPU when dispatching &hctx->run_work, no? > > > > Does every WORK_CPU_UNBOUND user have to handle it? > > [Adding Tejun Heo] > > No, because almost all other subsystems passing WORK_CPU_UNBOUND do so on > unbound workqueues (e.g. system_unbound_wq). > > In kernel/workqueue.c, when WORK_CPU_UNBOUND is queued on a WQ_UNBOUND > workqueue, workqueue core already handles CPU isolation automatically via > wq_select_unbound_cpu(), which respects wq_unbound_cpumask and steers the > work item to a housekeeping CPU. > > kblockd_workqueue is unique in that it is allocated as WQ_PERCPU to > preserve per-CPU locality for multi-queue devices. For WQ_PERCPU > workqueues, workqueue core cannot silently migrate WORK_CPU_UNBOUND to a > remote CPU because callers of bound per-CPU workqueues expect local per-CPU > execution. Thus, blk-mq is in the unusual position of passing > WORK_CPU_UNBOUND to a bound per-CPU workqueue with the expectation that it > avoids isolated cores. No. For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq), a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs on that CPU. blk-mq isn't special here; this is the general tradeoff of submitting IO from an isolated CPU. > > > > I suspect, steering the asynchronous dispatch to a housekeeping CPU is both > > > architecturally necessary and fully aligned with the rest of the kernel, > > > no? When a queue is plugged, throttled, or returns BLK_STS_RESOURCE, blk-mq > > > defers execution to a background workqueue. That work runs asynchronously > > > at some indeterminate time in the future, long after the thread has > > > returned to its time-critical user loop. Having a kworker suddenly preempt > > > the thread later is likely not desired. > > > > We are talking about `if a task executing on an isolated CPU submits I/O `. > > > > If the task has been submitting IO on isolated CPU, why can't we keep > > scheduling WQ on isolated CPU for running blk-mq queue? I am not sure your change > > makes any difference from user viewpoint. > > When a task submits I/O, synchronous submission occurs directly in syscall > context. Note, in the following example, IOCB_DIRECT is applied since the > file was opened with file->f_flags & O_DIRECT: > > __x64_sys_write > ksys_write > vfs_write > else if (file->f_op->write_iter) > new_sync_write(file, buf, count, pos) > // filp->f_op->write_iter(&kiocb, &iter) > ext4_file_write_iter > if (iocb->ki_flags & IOCB_DIRECT) > ext4_dio_write_iter(iocb, from) > iomap_dio_rw(iocb, from, &ext4_iomap_ops, > &ext4_dio_write_ops, dio_flags, NULL, 0) > __iomap_dio_rw > ... > iomap_dio_submit_bio > blk_crypto_submit_bio(bio) > if (!bio_has_crypt_ctx(bio) ...) > submit_bio(bio) > ... > __submit_bio > blk_mq_submit_bio(bio) > // blk_mq_run_dispatch_ops(q, blk_mq_try_issue_directly(hctx, rq)) > __blk_mq_run_dispatch_ops(q, true blk_mq_try_issue_directly(hctx, rq)) > blk_mq_try_issue_directly(hctx, rq) > __blk_mq_issue_directly(hctx, rq, true) > // q->mq_ops->queue_rq(hctx, &bd) > > > However, if dispatch must be deferred (e.g. BLK_STS_RESOURCE), > blk-mq queues &hctx->run_work. Now, if &hctx->run_work is queued on the > isolated CPU, it must be executed by the local kworker. Because that > kworker runs under SCHED_NORMAL, the user's SCHED_FIFO task completely > starves the kworker. As a result, blk_mq_run_work_fn() never executes, > hardware dispatch stalls, and the I/O pipeline hangs indefinitely. > Albeit, I believe an enabled Fair server, under the EEVDF scheduler, will > prevent this stall. > > Anyhow, offloading deferred dispatch to an online housekeeping CPU allows > the kworker to run immediately and complete dispatch without priority > inversion. As noted in commit a46c27026da1: > "it isn't expected to let block IO disturb these CPUs. blk-mq kworker > shouldn't be scheduled on isolated CPUs." You already disturb isolated CPUs by submitting IO via syscall in this case, so why does it matter here to dispatch on same isolated CPU one more time? Thanks, Ming