From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from va-2-35.ptr.blmpb.com (va-2-35.ptr.blmpb.com [209.127.231.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 278543B38B4 for ; Thu, 9 Apr 2026 12:34:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.127.231.35 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775738059; cv=none; b=jmS2hNx5kcRIK817SER2saejkErbLVvD50pvmmWOGFtmVy4TmInciidfvAVw14q4+xShhzi/uSzcR4sHCYsj9nY2WuEAlhgkA1TJkwxH0CxXg5lGCaDSogTgV3kP6KZe46g1TckhkIWSEG/XdmtCY/zGyRggSDFbEmAmazzP100= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775738059; c=relaxed/simple; bh=+NpDxzfw8kRgkIJFjXGPNLsP7BtxYqTDJLG8wzv7378=; h=Mime-Version:References:Cc:Date:Message-Id:Subject:In-Reply-To: Content-Type:To:From; b=i4ym6II51Tr292Bxou1E4kjUGXu00P0k0ihCgfMjMGLGdKNjtNQ4zqVo9tH2L0XpcOvParYMNzipaV4Cy3MLqT+vCBo8yqQoSSVpLBya9pMnBSYUS5ojxxdeQvimRUWvohlqqpfCYSk996nflXbPtU7MOLk9YLHr3qalg1apUE4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fnnas.com; spf=pass smtp.mailfrom=fnnas.com; dkim=pass (2048-bit key) header.d=fnnas-com.20200927.dkim.feishu.cn header.i=@fnnas-com.20200927.dkim.feishu.cn header.b=nReRy8Tl; arc=none smtp.client-ip=209.127.231.35 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=fnnas.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fnnas.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=fnnas-com.20200927.dkim.feishu.cn header.i=@fnnas-com.20200927.dkim.feishu.cn header.b="nReRy8Tl" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=s1; d=fnnas-com.20200927.dkim.feishu.cn; t=1775738045; h=from:subject:mime-version:from:date:message-id:subject:to:cc: reply-to:content-type:mime-version:in-reply-to:message-id; bh=6FAgFXvip6RdwiSAnKn65CFtEQZSAiYkFyfTIRcXxEc=; b=nReRy8TlBf2ZjgtfBOk8GrTUN/EIPhkfgpyGOm+o1r72T+o6Eym/A24/68kv+w2f1+guN/ VfTwEV0AR2XCO2+tn9gSgTBUeLMquokbTR5DDaAIp0bmIiQTegdc96SgdvMJjoszGZsmZT slqggZMGKxJXW5V+5DezUZ1IVfYBr7/MZdE2zTsOgc2IZxbo8uZhFkk4KUqW8CIMshKTqo RP2q0f3YeoJgWHc0LiPXAdOFHanMdlzC5YUA5X52GZCc9E+/wSFusjGQHeOzQ6kHmArT64 0SAD5bSovXBIllnbRHYiQdYTyuRgZ3AEqQZuu5LE07cFKyaWFxsjmH0mmIrdww== Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <005201dcc7ae$54a33010$fde99030$@telus.net> X-Lms-Return-Path: Cc: "'Nilay Shroff'" , "'Bart Van Assche'" , "'Jens Axboe'" , , Date: Thu, 9 Apr 2026 20:34:00 +0800 Message-Id: <9c1424f1-5d81-4308-bc93-0bdcde788501@fnnas.com> Content-Language: en-US Content-Transfer-Encoding: quoted-printable Subject: Re: REGRESSION BISECTED: mq-deadline: covert to use request_queue->async_depth User-Agent: Mozilla Thunderbird In-Reply-To: <005201dcc7ae$54a33010$fde99030$@telus.net> X-Original-From: Yu Kuai Reply-To: yukuai@fnnas.com Content-Type: text/plain; charset=UTF-8 Received: from [192.168.1.104] ([39.182.0.184]) by smtp.feishu.cn with ESMTPS; Thu, 09 Apr 2026 20:34:02 +0800 To: "Doug Smythies" From: "Yu Kuai" Hi, =E5=9C=A8 2026/4/9 7:20, Doug Smythies =E5=86=99=E9=81=93: > Hi, > > I observed a couple of regressions in a workflow (described further below= ). > I have not started to investigate the first regression yet. > The first regression was of less magnitude and was introduced somewhere b= etween kernels 6.18 and 6.19-rc1. > For the second regression, I bisected the kernel, and then double checked= the result: > >> commit 988bb1b9ededab9aed83df8c1f5be0232b71ded3 >> Author: Yu Kuai >> Date: Tue Feb 3 16:19:47 2026 +0800 >> mq-deadline: covert to use request_queue->async_depth >> >> In downstream kernel, we test with mq-deadline with many fio workloa= ds, and >> we found a performance regression after commit 39823b47bbd4 >> ... Do you ever use the async_depth for mq-deadline? For this patch, the old mq-deadline sysfs api is now removed, if you ever s= et this, then you'll need to set the new api under request_queue. Otherwise, if you're using the default value before, I don't see how this p= atch can make a difference for IO. > Workflow: > My version of "critical-jobs", an attempt to do similar to the non-free S= PECjbb critical-jOPS. > > The data base file is 500 Gigabytes. > Each requested job does 2 record lookups and a tiny bit of work for each = lookup. > 340 jobs per second are requested, but the system can only handle an aver= age of > about 231 jobs per second. Jobs keep being launched until the maximum num= ber > of processes of 1000 is hit, at which time the jobs rate is throttled bac= k. > The requested test execution time is extended such that all jobs are comp= leted. > > The test measures latency between the launch of the job and completion. > The issue is that some jobs seem to get forgotten about for a long long t= ime. > > Processor: Intel(R) Core(TM) i5-10600K CPU @ 4.10GHz, 6 cores 12 CPUs. > HWP: Enabled. Test Disk: HDD Seagate Ironwolf Pro16TB NAS > OS: Ubuntu 24.04.4 LTS (server, no GUI). Other than the test, the system = is very idle. > > Kernels used for this email: bad: > As of commit 988bb1b9eded called "kuai" > With 988bb1b9eded reverted, called "kuai-revert" > I.E.: > > 2d7cf26b0d4b (HEAD -> kuai) Revert "mq-deadline: covert to use request_qu= eue->async_depth" > 988bb1b9eded mq-deadline: covert to use request_queue->async_depth > 8cbe62f4d8c3 kyber: covert to use request_queue->async_depth > f98afe4f31bb blk-mq: add a new queue sysfs attribute async_depth > > Results: > > kuai: > 340.0 Requested jobs per second. > 230.9 Actual jobs per second. > 110.4 Execution time (Seconds) > 13.8 job latency, 90th percentile (Seconds), 85% regression > 21.9 job latency, 95th percentile, 144% regression > 40.7 job latency, 99th percentile, 235% regression > 99.2 maximum job latency, 313% regression > > kuai-revert (reference): > 340.0 Requested jobs per second. > 231.8 Actual jobs per second. > 110.0 Execution time (Seconds) > 7.47 job latency, 90th percentile (Seconds) > 8.96 job latency, 95th percentile > 12.14 job latency, 99th percentile > 24.01 maximum job latency > > I have yet to investigate the first regression, > but just for readers potential interest, > results for a kernel before 6.19-rc1: > 340.0 Requested jobs per second. > 233.2 Actual jobs per second. > 109.4 Execution time (Seconds). > 5.7 job latency, 90th percentile (Seconds) > 6.32 job latency, 95th percentile > 7.37 job latency, 99th percentile > 12.60 maximum job latency > > --=20 Thansk, Kuai