From: Bharata B Rao <bharata@amd.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
mgorman@suse.de, peterz@infradead.org, mingo@redhat.com,
bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org,
akpm@linux-foundation.org, luto@kernel.org, tglx@linutronix.de,
yue.li@memverge.com, Ravikumar.Bangoria@amd.com
Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing
Date: Fri, 24 Feb 2023 08:58:36 +0530 [thread overview]
Message-ID: <d4c0e8c0-877e-8278-acb1-2fcd43ed9325@amd.com> (raw)
In-Reply-To: <87lekz8nrl.fsf@yhuang6-desk2.ccr.corp.intel.com>
On 15-Feb-23 11:37 AM, Huang, Ying wrote:
> Bharata B Rao <bharata@amd.com> writes:
>
>> On 13-Feb-23 12:00 PM, Huang, Ying wrote:
>>>> I have a microbenchmark where two sets of threads bound to two
>>>> NUMA nodes access the two different halves of memory which is
>>>> initially allocated on the 1st node.
>>>>
>>>> On a two node Zen4 system, with 64 threads in each set accessing
>>>> 8G of memory each from the initial allocation of 16G, I see that
>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time
>>>> to complete a fixed number of memory accesses. This could well
>>>> be the best case and real workloads/benchmarks may not get this much
>>>> uplift, but it does show the potential gain to be had.
>>>
>>> Can you find a way to show the overhead of the original implementation
>>> and your method? Then we can compare between them? Because you think
>>> the improvement comes from the reduced overhead.
>>
>> Sure, will measure the overhead.
>>
>>>
>>> I also have interest in the pages migration throughput per second during
>>> the test, because I suspect your method can migrate pages faster.
>>
>> I have some data on pages migrated over time for the benchmark I mentioned
>> above.
>>
>>
>> Pages migrated vs Time(s)
>> 2500000 +---------------------------------------------------------------+
>> | + + + + + + + |
>> | Default ******* |
>> | IBS ####### |
>> | |
>> | ****************************|
>> | * |
>> 2000000 |-+ * +-|
>> | * |
>> | ** |
>> P | * ## |
>> a | *### |
>> g | **# |
>> e 1500000 |-+ *## +-|
>> s | ## |
>> | # |
>> m | # |
>> i | *# |
>> g | *# |
>> r | ## |
>> a 1000000 |-+ # +-|
>> t | # |
>> e | #* |
>> d | #* |
>> | # * |
>> | # * |
>> 500000 |-+ # * +-|
>> | # * |
>> | # * |
>> | # * |
>> | ## * |
>> | # * |
>> | # + * + + + + + + |
>> 0 +---------------------------------------------------------------+
>> 0 20 40 60 80 100 120 140 160
>> Time (s)
>>
>> So acting upon the relevant accesses early enough seem to result in
>> pages migrating faster in the beginning.
>
> One way to prove this is to output the benchmark performance
> periodically. So we can find how the benchmark score change over time.
Here is the data from a different run that captures the benchmark scores
periodically. The benchmark touches a fixed amount of memory a fixed number
of times iteratively. I am capturing the iteration number for one of the
threads that starts touching memory which is completely remote at the
beginning. The higher iteration number suggests that the thread is making
progress quickly which eventually reflects as the benchmark score.
\f
Access iterations vs Time
500 +-------------------------------------------------------------------+
| + + + + + + + + * |
| Default ******* |
450 |-+ # IBS #######-|
| # * |
| # * |
| # * |
400 |-+ # * +-|
| # * |
A | ****#********************************************* |
c 350 |-+ * # +-|
c | * # |
e | * # |
s 300 |-+ * # +-|
s | * # |
| * # |
i 250 |-+ * # +-|
t | * # |
e | * # |
r | * # |
a 200 |-+ * # +-|
t | *# |
i | * # |
o 150 |-+ *# +-|
n | *# |
s | *# |
100 |-+ *# +-|
| # |
| # |
| # |
50 |-# +-|
|# |
|# + + + + + + + + |
0 +-------------------------------------------------------------------+
0 20 40 60 80 100 120 140 160 180
Time (s)
The way the number of migrated pages varies for the above runs is shown in
the below graph:
\f
Pages migrated vs Time
2500000 +---------------------------------------------------------------+
| + + + + + + + + + |
| Default ******* |
| IBS ####### |
| |
| ******** |
| * |
2000000 |-+ ** +-|
| *** |
| ** |
p | * |
a | ** |
g | ** |
e 1500000 |-+ * +-|
s | *** |
| ** |
m | ** |
i | * |
g | ** |
r | * |
a 1000000 |-+ * +-|
t | * |
e | * |
d | * |
| * |
| ##* |
500000 |-+ # * +-|
| ## * |
| ## * |
| ### * |
| # * |
| #### * |
| # + * + + + + + + + |
0 +---------------------------------------------------------------+
0 20 40 60 80 100 120 140 160 180 200
Time (s)
The final benchmark scores for the above runs compare like this:
Default IBS
Time (us) 174459192.0 54710778.0
Regards,
Bharata.
next prev parent reply other threads:[~2023-02-24 3:28 UTC|newest] Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-02-08 7:35 Bharata B Rao 2023-02-08 7:35 ` [RFC PATCH 1/5] x86/ibs: In-kernel IBS driver for page access profiling Bharata B Rao 2023-02-08 7:35 ` [RFC PATCH 2/5] x86/ibs: Drive NUMA balancing via IBS access data Bharata B Rao 2023-02-08 7:35 ` [RFC PATCH 3/5] x86/ibs: Enable per-process IBS from sched switch path Bharata B Rao 2023-02-08 7:35 ` [RFC PATCH 4/5] x86/ibs: Adjust access faults sampling period Bharata B Rao 2023-02-08 7:35 ` [RFC PATCH 5/5] x86/ibs: Delay the collection of HW-provided access info Bharata B Rao 2023-02-08 18:03 ` [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing Peter Zijlstra 2023-02-08 18:12 ` Dave Hansen 2023-02-09 6:04 ` Bharata B Rao 2023-02-09 14:28 ` Dave Hansen 2023-02-10 4:28 ` Bharata B Rao 2023-02-10 4:40 ` Dave Hansen 2023-02-10 15:10 ` Bharata B Rao 2023-02-09 5:57 ` Bharata B Rao 2023-02-13 2:56 ` Huang, Ying 2023-02-13 3:23 ` Bharata B Rao 2023-02-13 3:34 ` Huang, Ying 2023-02-13 3:26 ` Huang, Ying 2023-02-13 5:52 ` Bharata B Rao 2023-02-13 6:30 ` Huang, Ying 2023-02-14 4:55 ` Bharata B Rao 2023-02-15 6:07 ` Huang, Ying 2023-02-24 3:28 ` Bharata B Rao [this message] 2023-02-16 8:41 ` Bharata B Rao 2023-02-17 6:03 ` Huang, Ying 2023-02-24 3:36 ` Bharata B Rao 2023-02-27 7:54 ` Huang, Ying 2023-03-01 11:21 ` Bharata B Rao 2023-03-02 8:10 ` Huang, Ying 2023-03-03 5:25 ` Bharata B Rao 2023-03-03 5:53 ` Huang, Ying 2023-03-06 15:30 ` Bharata B Rao 2023-03-07 2:33 ` Huang, Ying
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d4c0e8c0-877e-8278-acb1-2fcd43ed9325@amd.com \
--to=bharata@amd.com \
--cc=Ravikumar.Bangoria@amd.com \
--cc=akpm@linux-foundation.org \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
--cc=ying.huang@intel.com \
--cc=yue.li@memverge.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®