From: Marek Szyprowski <m.szyprowski@samsung.com>
To: Michael Kelley <mhklinux@outlook.com>,
Frank Chen <frankchen158@126.com>, "hch@lst.de" <hch@lst.de>,
"robin.murphy@arm.com" <robin.murphy@arm.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
"fan.du@intel.com" <fan.du@intel.com>,
"jun.miao@intel.com" <jun.miao@intel.com>,
"chenhgs@chinatelecom.cn" <chenhgs@chinatelecom.cn>
Subject: Re: [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in
Date: Tue, 4 Aug 2026 12:55:39 +0200 [thread overview]
Message-ID: <8e51a0f6-b73e-40a9-a157-aa31b5106320@samsung.com> (raw)
In-Reply-To: <BN7PR02MB41489054FA36992B3DB2800FD4C32@BN7PR02MB4148.namprd02.prod.outlook.com>
On 20.07.2026 20:30, Michael Kelley wrote:
> From: Frank Chen <frankchen158@126.com> Sent: Monday, July 13, 2026 8:25 PM
>> From: chenhuguanshen <chenhgs@chinatelecom.cn>
>>
>> Under heavy concurrent DMA traffic on CoCo VMs, inc_used_and_hiwater()
>> performs an atomic_long_add_return() plus a CAS loop on the global
>> used_hiwater, and dec_used() performs an atomic_long_sub() on total_used.
>> All CPUs contend on the same cacheline, causing measurable throughput
>> degradation at scale.
>>
>> Historically these counters were only compiled in under CONFIG_DEBUG_FS,
>> which means production kernels with debugfs paid the atomic overhead
>> unconditionally once the symbols were present. Make the tracking
>> boot-time opt-in instead so that it is disabled by default with near-zero
>> overhead via static_call, and can be enabled via "swiotlb=track_hiwater"
>> parameter on demand for debugging.
>>
>> Changes:
>>
>> - The previous inc_used_and_hiwater() maintained a separate atomic
>> total_used counter for accurate hiwater updates. This is replaced with
>> mem_used() which sums across areas. While less precise (no cross-area
>> locking), the result is adequate for high watermark statistics and
>> avoids the overhead of maintaining total_used on every alloc/free.
> I'm not sold on giving up an accurate hiwater calculation. If it's
> optional, let's have the option be to maintain total_used on every
> alloc/free so the hiwater calculation is accurate.
>
> Maintaining hiwater based on mem_used() is already pretty low
> overhead in that it only does a global atomic operation if the hiwater is
> raised. After a startup transient that establishes a typical steady-state
> hiwater, most ongoing swiotlb alloc/free operations won't raise
> the hiwater and so won't do any atomic operations to maintain it.
>
> To recap, there are three levels of hiwater maintenance:
>
> 1) None
> 2) Based on the less precise mem_used()
> 3) Based on an atomically update global total_used value
>
> The perf difference between #1 and #2 seems minimal outside of
> the initial ramp to a steady-state hiwater value. The perf impact
> of #3 can indeed be more significant, especially at high CPU counts.
>
> Having the kernel boot line offer all three is arguably overkill.
> I'd suggest offering #1 and #3 (with #1 as the default), or #2
> and #3 (with #2 as the default and !CONFIG_DEBUG_FS providing
> #1).
>
> There's another aspect here as well. Robin Murphy had
> suggested a sysctl option, so that the accurate hiwater maintenance
> could be turned on/off at runtime, not just a boot time. My earlier
> comment about boot time was probably unclear. I was thinking in
> terms of a runtime option that could *also* be set at boot time, which
> sysctl does. But I'll admit that enabling accurate hiwater calculation
> at runtime is tricky because some swiotlb buffers may be allocated
> at the time of the enablement, and total_used could go negative
> when those are freed. For runtime sysctl enablement, would
> probably have to do stop_machine() to freeze everything, and set
> total_used to the value of mem_used() so that everything is
> correct. I haven't worked through all the details.
>
> All that said, I don't want to make too big of an issue out of this,
> and end up over-engineering with complexity that exceeds the
> value. I could be persuaded to have boot time enablement but
> not runtime enablement. I'm less happy with losing entirely
> the accurate calculation (my #3 above) that's in the current
> upstream code.
Indeed keeping the accurate measurements is important to track potential
spikes of hiwater level. However imho it would be enough to use boot time
enablement.
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
next prev parent reply other threads:[~2026-08-04 10:55 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20260622122114.2563254-1-jun.miao@intel.com>
2026-06-23 2:35 ` [PATCH] swiotlb: eliminate per-map atomic contention on used/hiwater tracking Michael Kelley
2026-06-23 12:22 ` Robin Murphy
2026-06-25 7:30 ` Du, Fan
2026-06-25 15:53 ` Michael Kelley
2026-06-26 3:12 ` Du, Fan
2026-06-26 16:00 ` Michael Kelley
2026-06-27 23:20 ` Du, Fan
2026-06-28 1:30 ` Michael Kelley
2026-07-01 2:53 ` Du, Fan
2026-07-02 16:00 ` Michael Kelley
2026-07-13 1:18 ` Du, Fan
2026-07-14 3:25 ` [PATCH] dma/swiotlb: make high watermark tracking boot-time opt-in Frank Chen
2026-07-20 18:30 ` Michael Kelley
2026-08-04 10:55 ` Marek Szyprowski [this message]
2026-08-05 2:29 ` [PATCH v3] dma/swiotlb: decouple high watermark tracking from CONFIG_DEBUG_FS Frank Chen
2026-08-09 15:50 ` Michael Kelley
2026-08-12 7:04 ` [PATCH v4] " Frank Chen
2026-08-12 15:01 ` Michael Kelley
2026-08-13 6:24 ` Marek Szyprowski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8e51a0f6-b73e-40a9-a157-aa31b5106320@samsung.com \
--to=m.szyprowski@samsung.com \
--cc=chenhgs@chinatelecom.cn \
--cc=fan.du@intel.com \
--cc=frankchen158@126.com \
--cc=hch@lst.de \
--cc=iommu@lists.linux.dev \
--cc=jun.miao@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mhklinux@outlook.com \
--cc=robin.murphy@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®