[RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[孙魁 (Kui Sun)]

Dear Kernel Maintainers,

During our upgrade from Linux kernel 5.15 to Linux kernel 6.18, we observed a significant performance regression in random I/O workloads―with a maximum degradation of 27.7%.
This issue is particularly pronounced in single-threaded, small-block I/O scenarios。

To illustrate the impact, we conducted benchmark tests using AnTuTu on Unisoc T615 devices.
The results are summarized below:

Table  1：Random Read/Write Speed Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            20906   20508   21362   20925.33
T615    6.18            20164   21107   21077   20782.67

Table  2：Multi-threaded Mixed Random Read/Write Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            4670    4701    4457    4609.33
T615    6.18            4311    4697    4471    4493.00

Table   3：Mixed Random Read/Write Speed Scores（Single-threaded）
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            18604   18314   17732   18216.67
T615    6.18            13372   13081   13081   13178.00（↓27.66%）

Notably, only the single-threaded test (Table 3) shows severe degradation, while multi-threaded tests exhibit minimal change (<3%).
This strongly suggests the regression is tied to increased per-request scheduling or interrupt overhead in low-concurrency, small-block (e.g., 4KB) I/O paths.

Root Cause Identification

Through investigation, we identified that upstream commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is likely responsible for this performance drop.
After locally reverting this commit on kernel 6.18, performance fully recovered:

Table 4：Mixed Random Read/Write Speed Scores（After Revert）
Device  Kernel Version                  Test1   Test2   Test3   Average
T615    5.15                                    18604   18314   17732   18216.67
T615    6.18                            13372   13081   13081   13178.00（↓27.66%）
T615    6.18（reverted 3c7ac40)  18314   18604   18604   18507.33

Technical Analysis

We believe the change introduced additional interrupt or scheduling latency.
In multi-threaded workloads, a single interrupt can process multiple 4KB requests (e.g., 8 requests), amortizing the scheduling cost to M/8 per request (where M is the total overhead).
In contrast, single-threaded I/O handles only one request per interrupt, incurring the full cost M per operation.
Consequently, single-threaded small I/O is highly sensitive to such latency increases, explaining the disproportionate impact observed in Table 3.

Request and Recommendations

Given the tangible impact on mobile user experience, we kindly request the community to:
1.      Consider reverting commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, or
2.      Re-evaluate the proposed change in light of its effect on low-concurrency I/O paths, as discussed here:
https://lore.kernel.org/lkml/88d31a258feb36425ad73d0323077972f85f8341.camel@linaro.org/
3,      Could we add a flag to allow our UFS driver to choose between using an interrupt or an interrupt thread? [1]

Furthermore, we recommend that future evaluations of similar changes include:
*       Small-block (e.g., 4KB or 8KB) random read/write benchmarks, and
*       Single-threaded workloads,
as these are critical for mobile and embedded systems.

We appreciate your attention and are happy to provide additional data or assist in validating potential fixes.

[1]: https://lore.kernel.org/linux-scsi/20260710065948.467514-1-kui.sun@unisoc.com/T/#u

Re: [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[André Draszik]

Hi,

On Fri, 2026-07-10 at 07:17 +0000, 孙魁 (Kui Sun) wrote:
> Dear Kernel Maintainers,
> 
> During our upgrade from Linux kernel 5.15 to Linux kernel 6.18, we observed a significant performance regression in random I/O
> workloads—with a maximum degradation of 27.7%.

[...]

> This issue is particularly pronounced in single-threaded, small-block I/O scenarios。
> 
> To illustrate the impact, we conducted benchmark tests using AnTuTu on Unisoc T615 devices.
> The results are summarized below:

[...]


> Root Cause Identification
> 
> Through investigation, we identified that upstream commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is likely
> responsible for this performance drop.
> After locally reverting this commit on kernel 6.18, performance fully recovered:
> 
> Table 4：Mixed Random Read/Write Speed Scores（After Revert）
> Device  Kernel Version                  Test1   Test2   Test3   Average
> T615    5.15                                    18604   18314   17732   18216.67
> T615    6.18                            13372   13081   13081   13178.00（↓27.66%）
> T615    6.18（reverted 3c7ac40)  18314   18604   18604   18507.33
> 

Thank you for your above analysis. Your numbers match up well with my own
observations at the time in
https://lore.kernel.org/all/88d31a258feb36425ad73d0323077972f85f8341.camel@linaro.org/

[...]

> Request and Recommendations
> 
> Given the tangible impact on mobile user experience, we kindly request the community to:
> 1.      Consider reverting commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, or
> 2.      Re-evaluate the proposed change in light of its effect on low-concurrency I/O paths, as discussed here:
> https://lore.kernel.org/lkml/88d31a258feb36425ad73d0323077972f85f8341.camel@linaro.org/

While originally I was trying to find a solution that doesn't regress existing
platforms and still works for the newly added platform, I can only second the request
for revert, given how important UFS is for mobile.

Mani was also in favour of reverting:
https://lore.kernel.org/all/4enen7mopxtx4ijl5qyrd2gnxvv3kygtlnhxpr64egckpvkja4@hjli25ndhxwc/


Cheers,
Andre'

Re: [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[Bart Van Assche]

On 7/10/26 12:17 AM, 孙魁 (Kui Sun) wrote:
> Through investigation, we identified that upstream commit
> 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is
> likely responsible for this performance drop.
Two fixes for that commit are present in the upstream kernel. Are these
fixes present in your kernel tree?

commit eabcac808ca3ee9878223d4b49b750979029016b
Author: Bart Van Assche <bvanassche@acm.org>
Date:   Fri Aug 15 08:58:23 2025 -0700

     scsi: ufs: core: Fix IRQ lock inversion for the SCSI host lock

commit 034d319c8899e8c5c0a35c6692c7fc7e8c12c374
Author: Nitin Rawat <quic_nitirawa@quicinc.com>
Date:   Tue Jul 29 04:27:11 2025 +0530

     scsi: ufs: core: Fix interrupt handling for MCQ Mode

Thanks,

Bart.

答复: [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322

[孙魁 (Kui Sun)]

We have already included these two fixes, which are related to stability. The issue we're encountering is performance-related

Thank You.

-----邮件原件-----
发件人: Bart Van Assche <bvanassche@acm.org> 
发送时间: 2026年7月10日 22:56
收件人: 孙魁 (Kui Sun) <kui.sun@unisoc.com>; Neil Armstrong <neil.armstrong@linaro.org>; Alim Akhtar <alim.akhtar@samsung.com>; Avri Altman <avri.altman@wdc.com>; James E.J. Bottomley <James.Bottomley@HansenPartnership.com>; Martin K. Petersen <martin.petersen@oracle.com>; andre.draszik@linaro.org
抄送: Peter Griffin <peter.griffin@linaro.org>; Tudor Ambarus <tudor.ambarus@linaro.org>; Will McVicker <willmcvicker@google.com>; Manivannan Sadhasivam <mani@kernel.org>; kernel-team@android.com; linux-samsung-soc@vger.kernel.org; linux-scsi@vger.kernel.org; linux-kernel@vger.kernel.org; stable@vger.kernel.org; linux-arm-msm@vger.kernel.org; 张如泉 (Rain Zhang) <Rain.Zhang@unisoc.com>; cixi.geng@linux.dev; 唐月林 (Yuelin Tang) <yuelin.tang@unisoc.com>; 陈文超 (Wenchao Chen) <Wenchao.Chen@unisoc.com>; André Draszik <andre.draszik@linaro.org>
主题: Re: [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322


注意: 这封邮件来自于外部。除非你确定邮件内容安全，否则不要点击任何链接和附件。
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



On 7/10/26 12:17 AM, 孙魁 (Kui Sun) wrote:
> Through investigation, we identified that upstream commit 
> 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is 
> likely responsible for this performance drop.
Two fixes for that commit are present in the upstream kernel. Are these fixes present in your kernel tree?

commit eabcac808ca3ee9878223d4b49b750979029016b
Author: Bart Van Assche <bvanassche@acm.org>
Date:   Fri Aug 15 08:58:23 2025 -0700

     scsi: ufs: core: Fix IRQ lock inversion for the SCSI host lock

commit 034d319c8899e8c5c0a35c6692c7fc7e8c12c374
Author: Nitin Rawat <quic_nitirawa@quicinc.com>
Date:   Tue Jul 29 04:27:11 2025 +0530

     scsi: ufs: core: Fix interrupt handling for MCQ Mode

Thanks,

Bart.