From: Marcelo Tosatti <mtosatti@redhat.com>
To: Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>
Cc: Takuya Yoshikawa <takuya.yoshikawa@gmail.com>,
avi@redhat.com, kvm@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 3/4 changelog-v2] KVM: Switch to srcu-less get_dirty_log()
Date: Wed, 7 Mar 2012 20:25:20 -0300 [thread overview]
Message-ID: <20120307232520.GB29570@amt.cnet> (raw)
In-Reply-To: <20120307170745.3147f776.yoshikawa.takuya@oss.ntt.co.jp>
On Wed, Mar 07, 2012 at 05:07:45PM +0900, Takuya Yoshikawa wrote:
> Marcelo Tosatti <mtosatti@redhat.com> wrote:
>
> > > Partly yes: my method mainly depends on the number of dirty pages,
> > > not slot size.
> > >
> > > But it is not a new problem: traversing all shadow pages for that
> > > also takes linearly increasing time.
> >
> > It was not necessary to read the bitmap under mmu_lock previously.
>
> Let's check actual data!
>
> Below I pasted a simple test result to show that reading bitmap is not
> a problem at all compared to traversing shadow pages.
>
> ** During doing the same live migration test as:
>
> For real workloads, both VGA and live migration, we have observed pure
> improvements: when the guest was reading a file during live migration,
> we originally saw a few ms of latency, but with the new method the
> latency was less than 200us.
>
> I measured how long the current method takes to just write protect sptes
> with the mmu_lock held - kvm_mmu_slot_remove_write_access() time.
>
>
> You can see many ms order of protection times from this result: for me this
> is more problematic than downtime problem many people like to improve.
>
> In contrast my method only took 200us in the worst case: actually what I
> measured for that was the entire kvm_vm_ioctl_get_dirty_log() time which
> contained more extra tasks, e.g. copy_to_user().
>
> FYI: changing the guest memory size from 4GB to 8GB did not show any
> siginificant change to my method, but, as you can guess, traversing
> shadow pages will need more time for increased shadow pages.
>
>
> If we have 4K shadow pages in the slot, kvm_mmu_slot_remove_write_access()
> have to traverse all of them, checking all 512 entries in them.
>
> Compared to that, the bitmap size of 4GB memory slot is 1M bits = 128KB.
> Reading this 8 pages is negligible.
Right, thanks for checking it.
> My unit-test experiment has also showed that xchg overheads is not so much,
> compared to others:
>
> 493900.4 15911.9 60.2 125% 5% 8K
> 760268.2 5929.6 46.4 63% 199% 16K
> 1238709.6 7123.5 37.8 23% 173% 32K
> 2359523.6 3121.7 36.0 -9% 87% 64K
> 4540780.6 10155.6 34.6 -27% 30% 128K
> 8747274.0 10973.3 33.3 -31% -3% 256K
>
> Note that these cases need to xchg the entire dirty bitmap because at least
> one bit is set for each unsigned-long-word.
>
> The big difference came from the number of sptes to protect alone.
>
> Takuya
What is worrying are large memory cases: think of the 50GB slot case.
100ms hold time is pretty bad (and reacquiring the lock is relatively
simple).
>
> ===
> funcgraph_entry: + 25.123 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 35.746 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 922.886 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 20.153 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 20.424 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 17.595 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 20.240 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 9783.060 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1992.718 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1312.128 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2028.900 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1455.889 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1382.795 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2030.321 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1407.248 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2189.321 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1444.344 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2291.976 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1801.848 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1993.104 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1531.858 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2394.283 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1613.203 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1699.472 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2416.467 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1566.451 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1772.670 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1700.544 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1590.114 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2311.419 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1923.888 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2534.780 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2083.623 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1664.170 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2867.553 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2684.615 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1706.371 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2655.976 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1720.777 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2993.758 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1924.842 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3091.190 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1776.427 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2808.984 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2669.008 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2359.525 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2703.617 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2623.198 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1942.833 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1906.551 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2981.093 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2168.301 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1949.932 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2992.925 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3360.511 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1993.321 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3187.857 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 1989.417 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2001.865 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2047.220 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3107.808 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2039.732 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2057.575 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2417.748 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2076.445 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2308.323 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3216.713 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2148.263 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2269.673 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 2133.566 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3757.388 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3372.302 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3679.316 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3516.200 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 630.067 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 3191.830 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: ! 658.717 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 66.683 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: + 31.027 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.274 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.568 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.460 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.358 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.197 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.306 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.259 us | kvm_mmu_slot_remove_write_access();
> funcgraph_entry: 0.181 us | kvm_mmu_slot_remove_write_access();
next prev parent reply other threads:[~2012-03-07 23:28 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-01 10:30 [PATCH 0/4] KVM: srcu-less dirty logging -v2 Takuya Yoshikawa
2012-03-01 10:31 ` [PATCH 1/4] KVM: MMU: Split the main body of rmap_write_protect() off from others Takuya Yoshikawa
2012-03-12 7:39 ` Takuya Yoshikawa
2012-03-12 7:52 ` Takuya Yoshikawa
2012-03-01 10:32 ` [PATCH 2/4] KVM: Avoid checking huge page mappings in get_dirty_log() Takuya Yoshikawa
2012-03-02 2:56 ` Takuya Yoshikawa
2012-03-02 5:11 ` Takuya Yoshikawa
2012-03-12 18:04 ` Avi Kivity
2012-03-13 9:20 ` Takuya Yoshikawa
2012-03-13 10:12 ` Avi Kivity
2012-03-13 23:04 ` Marcelo Tosatti
2012-03-14 1:04 ` Takuya Yoshikawa
2012-03-14 5:34 ` Takuya Yoshikawa
2012-03-14 10:58 ` Marcelo Tosatti
2012-03-01 10:33 ` [PATCH 3/4] KVM: Switch to srcu-less get_dirty_log() Takuya Yoshikawa
2012-03-03 5:21 ` [PATCH 3/4 changelog-v2] " Takuya Yoshikawa
2012-03-06 11:15 ` Marcelo Tosatti
2012-03-06 14:43 ` Takuya Yoshikawa
2012-03-06 15:01 ` Marcelo Tosatti
2012-03-06 15:23 ` Takuya Yoshikawa
2012-03-06 15:28 ` Marcelo Tosatti
2012-03-07 8:07 ` Takuya Yoshikawa
2012-03-07 23:25 ` Marcelo Tosatti [this message]
2012-03-08 1:35 ` Takuya Yoshikawa
2012-03-09 0:08 ` Marcelo Tosatti
2012-03-12 12:05 ` Avi Kivity
2012-03-07 8:18 ` Takuya Yoshikawa
2012-03-07 23:20 ` Marcelo Tosatti
2012-03-16 5:03 ` [PATCH 3/4] " Xiao Guangrong
2012-03-16 6:55 ` Takuya Yoshikawa
2012-03-16 7:30 ` Xiao Guangrong
2012-03-16 7:55 ` Takuya Yoshikawa
2012-03-16 8:28 ` Xiao Guangrong
2012-03-16 9:44 ` Takuya Yoshikawa
2012-03-19 9:34 ` Xiao Guangrong
2012-03-19 10:15 ` Takuya Yoshikawa
2012-03-01 10:34 ` [PATCH 4/4] KVM: Remove unused dirty_bitmap_head and nr_dirty_pages Takuya Yoshikawa
2012-03-03 5:12 ` [PATCH 0/4] KVM: srcu-less dirty logging -v2 Takuya Yoshikawa
2012-03-20 14:43 ` Avi Kivity
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120307232520.GB29570@amt.cnet \
--to=mtosatti@redhat.com \
--cc=avi@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=takuya.yoshikawa@gmail.com \
--cc=yoshikawa.takuya@oss.ntt.co.jp \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®