mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Marek Szyprowski <m.szyprowski@samsung.com>
To: Michael Kelley <mhklinux@outlook.com>,
	Frank Chen <frankchen158@126.com>, "hch@lst.de" <hch@lst.de>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"fan.du@intel.com" <fan.du@intel.com>,
	"jun.miao@intel.com" <jun.miao@intel.com>,
	"chenhgs@chinatelecom.cn" <chenhgs@chinatelecom.cn>
Subject: Re: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in
Date: Tue, 4 Aug 2026 12:55:39 +0200	[thread overview]
Message-ID: <8e51a0f6-b73e-40a9-a157-aa31b5106320@samsung.com> (raw)
In-Reply-To: <BN7PR02MB41489054FA36992B3DB2800FD4C32@BN7PR02MB4148.namprd02.prod.outlook.com>

On 20.07.2026 20:30, Michael Kelley wrote:
> From: Frank Chen <frankchen158@126.com> Sent: Monday, July 13, 2026 8:25 PM
>> From: chenhuguanshen <chenhgs@chinatelecom.cn>
>>
>> Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
>> performs an atomic_long_add_return() plus a CAS loop on the global
>> used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
>> All CPUs contend on the same cacheline, causing measurable throughput
>> degradation at scale.
>>
>> Historically these counters were only compiled in under CONFIG_DEBUG_FS,
>> which means production kernels with debugfs paid the atomic overhead
>> unconditionally once the symbols were present. Make the tracking
>> boot-time opt-in instead so that it is disabled by default with near-zero
>> overhead via static_call, and can be enabled via "swiotlb=track_hiwater"
>> parameter on demand for debugging.
>>
>> Changes:
>>
>> - The previous inc_used_and_hiwater() maintained a separate atomic
>>   total_used counter for accurate hiwater updates. This is replaced with
>>   mem_used() which sums across areas. While less precise (no cross-area
>>   locking), the result is adequate for high watermark statistics and
>>   avoids the overhead of maintaining total_used on every alloc/free.
> I'm not sold on giving up an accurate hiwater calculation. If it's
> optional, let's have the option be to maintain total_used on every
> alloc/free so the hiwater calculation is accurate.
>
> Maintaining hiwater based on mem_used() is already pretty low
> overhead in that it only does a global atomic operation if the hiwater is
> raised. After a startup transient that establishes a typical steady-state
> hiwater, most ongoing swiotlb alloc/free operations won't raise
> the hiwater and so won't do any atomic operations to maintain it.
>
> To recap, there are three levels of hiwater maintenance:
>
> 1) None
> 2) Based on the less precise mem_used()
> 3) Based on an atomically update global total_used value
>
> The perf difference between #1 and #2 seems minimal outside of
> the initial ramp to a steady-state hiwater value. The perf impact
> of #3 can indeed be more significant, especially at high CPU counts.
>
> Having the kernel boot line offer all three is arguably overkill.
> I'd suggest offering #1 and #3 (with #1 as the default), or #2
> and #3 (with #2 as the default and !CONFIG_DEBUG_FS providing
> #1).
>
> There's another aspect here as well. Robin Murphy had 
> suggested a sysctl option, so that the accurate hiwater maintenance
> could be turned on/off at runtime, not just a boot time. My earlier
> comment about boot time was probably unclear. I was thinking in
> terms of a runtime option that could *also* be set at boot time, which
> sysctl does. But I'll admit that enabling accurate hiwater calculation
> at runtime is tricky because some swiotlb buffers may be allocated
> at the time of the enablement, and total_used could go negative
> when those are freed. For runtime sysctl enablement, would
> probably have to do stop_machine() to freeze everything, and set
> total_used to the value of mem_used() so that everything is
> correct. I haven't worked through all the details.
>
> All that said, I don't want to make too big of an issue out of this,
> and end up over-engineering with complexity that exceeds the
> value. I could be persuaded to have boot time enablement but
> not runtime enablement. I'm less happy with losing entirely
> the accurate calculation (my #3 above) that's in the current
> upstream code.

Indeed keeping the accurate measurements is important to track potential

spikes of hiwater level. However imho it would be enough to use boot time

enablement.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland


  reply	other threads:[~2026-08-04 10:55 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20260622122114.2563254-1-jun.miao@intel.com>
2026-06-23  2:35 ` [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking Michael Kelley
2026-06-23 12:22   ` Robin Murphy
2026-06-25  7:30   ` Du, Fan
2026-06-25 15:53     ` Michael Kelley
2026-06-26  3:12       ` Du, Fan
2026-06-26 16:00         ` Michael Kelley
2026-06-27 23:20           ` Du, Fan
2026-06-28  1:30             ` Michael Kelley
2026-07-01  2:53               ` Du, Fan
2026-07-02 16:00                 ` Michael Kelley
2026-07-13  1:18                   ` Du, Fan
2026-07-14  3:25 ` [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in Frank Chen
2026-07-20 18:30   ` Michael Kelley
2026-08-04 10:55     ` Marek Szyprowski [this message]
2026-08-05  2:29 ` [PATCH v3] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS Frank Chen
2026-08-09 15:50   ` Michael Kelley
2026-08-12  7:04 ` [PATCH v4] " Frank Chen
2026-08-12 15:01   ` Michael Kelley
2026-08-13  6:24     ` Marek Szyprowski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8e51a0f6-b73e-40a9-a157-aa31b5106320@samsung.com \
    --to=m.szyprowski@samsung.com \
    --cc=chenhgs@chinatelecom.cn \
    --cc=fan.du@intel.com \
    --cc=frankchen158@126.com \
    --cc=hch@lst.de \
    --cc=iommu@lists.linux.dev \
    --cc=jun.miao@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mhklinux@outlook.com \
    --cc=robin.murphy@arm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®