mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322
@ 2026-07-10  7:17 孙魁 (Kui Sun)
  2026-07-10  9:54 ` André Draszik
  2026-07-10 14:56 ` Bart Van Assche
  0 siblings, 2 replies; 4+ messages in thread
From: 孙魁 (Kui Sun) @ 2026-07-10  7:17 UTC (permalink / raw)
  To: Neil Armstrong, Bart Van Assche, Alim Akhtar, Avri Altman,
	James E.J. Bottomley, "Martin K. Petersen",
	andre.draszik
  Cc: Peter Griffin, Tudor Ambarus, Will McVicker,
	Manivannan Sadhasivam, kernel-team, linux-samsung-soc,
	linux-scsi, linux-kernel, stable, linux-arm-msm,
	张如泉 (Rain Zhang),
	cixi.geng, 唐月林 (Yuelin Tang),
	陈文超 (Wenchao Chen)

Dear Kernel Maintainers,

During our upgrade from Linux kernel 5.15 to Linux kernel 6.18, we observed a significant performance regression in random I/O workloads―with a maximum degradation of 27.7%.
This issue is particularly pronounced in single-threaded, small-block I/O scenarios。

To illustrate the impact, we conducted benchmark tests using AnTuTu on Unisoc T615 devices.
The results are summarized below:

Table  1：Random Read/Write Speed Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            20906   20508   21362   20925.33
T615    6.18            20164   21107   21077   20782.67

Table  2：Multi-threaded Mixed Random Read/Write Scores
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            4670    4701    4457    4609.33
T615    6.18            4311    4697    4471    4493.00

Table   3：Mixed Random Read/Write Speed Scores（Single-threaded）
Device  Kernel Version  Test 1  Test 2  Test 3  Average
T615    5.15            18604   18314   17732   18216.67
T615    6.18            13372   13081   13081   13178.00（↓27.66%）

Notably, only the single-threaded test (Table 3) shows severe degradation, while multi-threaded tests exhibit minimal change (<3%).
This strongly suggests the regression is tied to increased per-request scheduling or interrupt overhead in low-concurrency, small-block (e.g., 4KB) I/O paths.

Root Cause Identification

Through investigation, we identified that upstream commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, merged in April 2025, is likely responsible for this performance drop.
After locally reverting this commit on kernel 6.18, performance fully recovered:

Table 4：Mixed Random Read/Write Speed Scores（After Revert）
Device  Kernel Version                  Test1   Test2   Test3   Average
T615    5.15                                    18604   18314   17732   18216.67
T615    6.18                            13372   13081   13081   13178.00（↓27.66%）
T615    6.18（reverted 3c7ac40)  18314   18604   18604   18507.33

Technical Analysis

We believe the change introduced additional interrupt or scheduling latency.
In multi-threaded workloads, a single interrupt can process multiple 4KB requests (e.g., 8 requests), amortizing the scheduling cost to M/8 per request (where M is the total overhead).
In contrast, single-threaded I/O handles only one request per interrupt, incurring the full cost M per operation.
Consequently, single-threaded small I/O is highly sensitive to such latency increases, explaining the disproportionate impact observed in Table 3.

Request and Recommendations

Given the tangible impact on mobile user experience, we kindly request the community to:
1.      Consider reverting commit 3c7ac40d732232fec0ba31d0a5e3cc9c112fc2e7, or
2.      Re-evaluate the proposed change in light of its effect on low-concurrency I/O paths, as discussed here:
https://lore.kernel.org/lkml/88d31a258feb36425ad73d0323077972f85f8341.camel@linaro.org/
3,      Could we add a flag to allow our UFS driver to choose between using an interrupt or an interrupt thread? [1]

Furthermore, we recommend that future evaluations of similar changes include:
*       Small-block (e.g., 4KB or 8KB) random read/write benchmarks, and
*       Single-threaded workloads,
as these are critical for mobile and embedded systems.

We appreciate your attention and are happy to provide additional data or assist in validating potential fixes.

[1]: https://lore.kernel.org/linux-scsi/20260710065948.467514-1-kui.sun@unisoc.com/T/#u

^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-07-11 11:22 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-07-10  7:17 [RFC] Significant Random I/O Performance Regression in Linux Kernel 6.18 (Up to 27.7%) Likely Caused by Commit 3c7ac40d7322 孙魁 (Kui Sun)
2026-07-10  9:54 ` André Draszik
2026-07-10 14:56 ` Bart Van Assche
2026-07-11 11:21   ` 答复: " 孙魁 (Kui Sun)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox