mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Naman Jain <namjain@linux.microsoft.com>
To: Fengnan <changfengnan@bytedance.com>,
	Michael Kelley <mhklinux@outlook.com>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Cc: Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
	Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Luigi Rizzo <lrizzo@google.com>
Subject: Re: [RFC PATCH 0/3] nvme-pci: yield completions under scheduler pressure
Date: Mon, 12 Oct 2026 14:21:59 +0530	[thread overview]
Message-ID: <41d9bb59-7490-498c-8ec9-770e08b2ca38@linux.microsoft.com> (raw)
In-Reply-To: <758e4c6b-1b1a-4191-baeb-3f9799783ee6@bytedance.com>



On 10/12/2026 12:30 PM, Fengnan wrote:
> 在 2026/10/12 14:48, Naman Jain 写道:
>>
>>
>> On 10/9/2026 5:06 PM, Fengnan wrote:
>>> Hi Naman:
>>>
>>> This is an issue I hadn’t noticed before.
>>> I'll take a closer look at this issue later.
>>>
>>> Regarding testing, there are a few points to note:
>>> 1. My patch has been updated to V2, https://lore.kernel.org/linux-nvme/20260826061546.56006-1-changfengnan@bytedance.com/
>>> 2. In my experience, when do performance tests, it’s best to do precondition
>>> first and then execute an ABBA test; this ensures reliable results.
>>> 3. IIRC, disabling interrupts in the QEMU kernel does not actually disable hardware
>>> interrupts; QEMU still handles them, which has a significant impact on performance testing.
>>> Perhaps you could test this on a physical machine.
>>>
>>> Also, I’d like to ask: does this softlockup issue only occur on QEMU+ARM?
>>> or does it also occur on x86 physical machines?
>>> Have you tried enabling the “Posted Interrupt” feature in VM? Maybe this will solve
>>> this problem in VM.
>>>
>>> Thanks.
>>
>> Hi Fengnan,
>> Thanks for the suggestion.
>>
>> I was using QEMU just for sanity, all the test results are based on experiments in an Azure ARM64 Hyper-V VM. I could not create enough load on my local QEMU VM to cause a lockup, so I am relying only on the Azure VM I have. I would not rule out x86 for this issue, as we have seen spurious lockups in the fleet, which are more common on certain SKUs, but my current experiments are limited to arm64. The proposed solution should solve it for both.
> 
> I set up a test environment on my own this morning.
> Although I wasn't able to reproduce the soft lockup issue, I did observe
> I/O latency of several hundred milliseconds.

Acked.

> 
>>
>> The guest exposes a GICv3 CPU interface and Hyper-V vPCI maps the NVMe MSI-X vectors as guest GIC SPIs. I do not see an ITS, LPIs/vLPIs, GICv4 VPEs, or another direct virtual interrupt injection interface exposed to the guest. Also, posted/direct interrupts would reduce host-to-guest interrupt injection overhead, but the lockup here occurs after delivery, while nvme_irq() repeatedly drains completion queues in guest hardirq context. So I would not expect posted interrupts alone to provide the missing scheduling boundary.
> 
> I'm thinking that adding a scheduling check to my patch should also resolve
> this issue.
> That would be simpler and more acceptable, we can work together to refine it.
> 
> Thanks.
> 

Sure, happy to collaborate. Please include me in any new discussions you 
have in your patch. And let's hear what Christoph and other maintainers 
have to say, and then we can converge on that approach going forward.


Regards,
Naman

      reply	other threads:[~2026-10-12  8:52 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-09  5:05 Naman Jain
2026-10-09  5:05 ` [RFC PATCH 1/3] nvme-pci: select IRQ_POLL Naman Jain
2026-10-09  5:05 ` [RFC PATCH 2/3] nvme-pci: make completion queue polling softirq-safe Naman Jain
2026-10-09  5:05 ` [RFC PATCH 3/3] nvme-pci: defer completions when rescheduling is needed Naman Jain
2026-10-09 15:30   ` Keith Busch
2026-10-12  6:47     ` Naman Jain
2026-10-09  5:54 ` [RFC PATCH 0/3] nvme-pci: yield completions under scheduler pressure Michael Kelley
2026-10-09  6:51   ` Naman Jain
2026-10-09 10:16     ` Naman Jain
2026-10-09 11:27       ` Luigi Rizzo
2026-10-09 14:58         ` Luigi Rizzo
2026-10-12  8:19           ` Naman Jain
2026-10-12  9:17             ` Luigi Rizzo
2026-10-09 11:36       ` Fengnan
2026-10-12  6:48         ` Naman Jain
2026-10-12  7:00           ` Fengnan
2026-10-12  8:51             ` Naman Jain [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=41d9bb59-7490-498c-8ec9-770e08b2ca38@linux.microsoft.com \
    --to=namjain@linux.microsoft.com \
    --cc=axboe@kernel.dk \
    --cc=changfengnan@bytedance.com \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=lrizzo@google.com \
    --cc=mhklinux@outlook.com \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®