mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Jiwei Sun <jiweisun126@126.com>
To: Keith Busch <kbusch@kernel.org>
Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org,
	axboe@fb.com, hch@lst.de, sagi@grimberg.me, ahuang12@lenovo.com,
	sunjw10@lenovo.com
Subject: Re: [PATCH] nvme: add cond_resched() to nvme_complete_batch()
Date: Tue, 16 May 2023 21:58:57 +0800	[thread overview]
Message-ID: <1d32a220-d941-6e67-58db-3e949d599812@126.com> (raw)
In-Reply-To: <ZGKY1A1q5Px64v9h@kbusch-mbp>

Hi Keith,

On 2023/5/16 04:40, Keith Busch wrote:
> On Tue, May 02, 2023 at 08:54:12PM +0800, jiweisun126@126.com wrote:
>> From: Jiwei Sun <sunjw10@lenovo.com>
>>
>> A soft lockup issue will be triggered when do fio test on a 448-core
>> server, such as the following warning:
> ...
>
>> According to the above two logs, we can know the nvme_irq() cost too much
>> time, in the above case, about 4.8 second. And we can also know that the
>> main bottlenecks is in the competition for the spin lock pool->lock.
> The most recent 6.4-rc has included a significant changeset to the pool
> allocator that may show a considerable difference in pool->lock timing.
> It would be interesting to hear if it changes your observation with your
> 448-core setup. Would you be able to re-run your experiements that
> produced the soft lockup with this kernel on that machine?
We have done some testes with the latest kernel, the issue can not be 
reproduced,
and we have analyzed the ftrace log of nvme_irq, we did NOT find any 
competition for
the spin lock pool->lock, and all the dma_pool_free function completed 
within 2us.

  287)               |        dma_pool_free() {
  287)   0.150 us    |          _raw_spin_lock_irqsave();
  287)   0.421 us    |          _raw_spin_unlock_irqrestore();
  287)   1.472 us    |        }
+-- 63 lines: 287)               |        mempool_free() {-----------
  435)               |        dma_pool_free() {
  435)   0.170 us    |          _raw_spin_lock_irqsave();
  435)   0.210 us    |          _raw_spin_unlock_irqrestore();
  435)   1.172 us    |        }
+--145 lines: 435)               |        mempool_free() {---------
  317)               |        dma_pool_free() {
  317)   0.160 us    |          _raw_spin_lock_irqsave();
  317)   0.401 us    |          _raw_spin_unlock_irqrestore();
  317)   1.252 us    |        }

Based on the test results and analysis of the code principles, your 
patch has fixed this performance issue.

By the way, another task hung issue was triggered in the test. We are 
analyzing it, but this is another story,
we can discuss it in other thread.

Thanks,
Regards,
Jiwei


      reply	other threads:[~2023-05-16 14:18 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-02 12:54 jiweisun126
2023-05-03 16:07 ` Christoph Hellwig
2023-05-03 16:28 ` Keith Busch
2023-05-03 23:42   ` Keith Busch
2023-05-04  9:39     ` Niklas Cassel
2023-05-04 10:12       ` Niklas Cassel
2023-05-04 15:06         ` Keith Busch
2023-05-04 15:18           ` Niklas Cassel
2023-05-15 20:40 ` Keith Busch
2023-05-16 13:58   ` Jiwei Sun [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1d32a220-d941-6e67-58db-3e949d599812@126.com \
    --to=jiweisun126@126.com \
    --cc=ahuang12@lenovo.com \
    --cc=axboe@fb.com \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    --cc=sunjw10@lenovo.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®