[PART1 RFC v4 00/11] KVM: x86: Introduce SVM AVIC support

[Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>]

CHANGES FROM RFCv3:
==================
(https://lkml.org/lkml/2016/3/18/24)
    * Added reviewed by in patch 3, 4, and 5.
    * Make minor changes to some variable name (per Paolo).
    * Merge AVIC vmexit handler and trace points patches (per Paolo)
    * Delay the x86_set_memory_region() call from vm_init phrase to
      when we create the first vcpu.
    * Clean up x2APIC disabling code when enabling AVIC.
    * Clean up CR8 interception code when enabling AVIC.
    * Clean up avic_cpu_load() and avic_set_running() (per Radim).
    * Note that nested-vm is not currently supported with AVIC enabled.
      However, it should still work with AVIC disabled. This is still work
      in progress.

CHANGES FROM RFCv2:
==================
(https://lkml.org/lkml/2016/3/4/746)
    * Do not use bit-field in VMCB.
    * Get rid of struct svm_vm_data, and consolidate the struct members
      into struct kvm_arch.
    * Do not abbreviate AVIC vmexit function and structure names.
    * Adding VM init/uninit function hooks and use it to initialize /
      uninitialize per-VM data structures.  
    * Simplify AVIC backing page allocation to use the emulated lapic
      register page.
    * Introduce kvm_vcpu_wake_up() (from Radim).
    * Clean up AVIC vmexit handling functions.
    * Replace pr_debug with new AVIC tracepoints.
    * Use per-vcpu AVIC enable check (besides the global avic flag).
    * Add smb_mb__after_atomic() in svm_deliver_avic_intr.
    * Get rid of iommu related code for now.

CHANGES FROM RFCv1:
==================
(https://lkml.org/lkml/2016/2/12/225)
    * Rebased from tip.git
    * Use the vAPIC backing page as the emulated LAPIC register page.
    * Clarified with HW engineer that Avic sets the IRR of all the cores
      targeted whether they are running or not. It sends doorbells to the
      ones that are running to get them to evaluate their new state.
      If one or more weren't running it does the VMEXIT to indicate that
      the host needs to go run the not running guests. It gives it the
      ID of one of the guests that it observed not running which should
      be enough of a hint for the hypervisor to track them all down.
    * Rewrite the logic for handling AVIC_INCOMPLETE_IPI #vmexit for
      IPI target not running case based on information above.
    * Rewrite the logic in avic_vcpu_load() and avic_set_running().
    * Rewrite the interrupt injection to remove the avic_pending_cnt
      variable.
    * Adding vcpu blocking/unblocking function hooks

GITHUB
======
Latest git tree can be found at:
    http://github.com/ssuthiku/linux.git    avic_part1_rfc_v4

OVERVIEW
========
This patch set is the first of the two-part patch series to introduce 
the new AMD Advance Virtual Interrupt Controller (AVIC) support.

Basically, SVM AVIC hardware virtualizes local APIC registers of each
vCPU via the virtual APIC (vAPIC) backing page. This allows guest access
to certain APIC registers without the need to emulate the hardware behavior
in the hypervisor. More information about AVIC can be found in the
AMD64 Architecture Programmer’s Manual Volume 2 - System Programming.

  http://support.amd.com/TechDocs/24593.pdf

For SVM AVIC, we extend the existing kvm_amd driver to:
  * Check CPUID to detect AVIC support in the processor
  * Program new fields in VMCB to enable AVIC
  * Introduce new AVIC data structures and add code to manage them
  * Handle two new AVIC #VMEXITs
  * Add new interrupt intjection code using vAPIC backing page
    instead of the existing V_IRQ, V_INTR_PRIO, V_INTR_VECTOR,
    and V_IGN_TPR fields

Currently, this patch series does not enable AVIC by default.
Users can enable SVM AVIC by specifying avic=1 during insmod kvm-amd.

Later, in part 2, we will introduce the IOMMU AVIC support, which
provides speed up for PCI device pass-through use case by allowing
the IOMMU hardware to inject interrupt directly into the guest via
the vAPIC backing page.

PERFORMANCE RESULTS
===================
Currently, AVIC is supported in the AMD family 15h models 6Xh
(Carrizo) processors. Therefore, it is used to collect the 
perforamance data shown below.

Generaly, SVM AVIC alone (w/o IOMMU AVIC) should provide speedup for
IPI interrupt since hypervisor does not require VMEXIT to inject
these interrupts. Also, it should speed up the case when hypervisor
wants to inject an interrupt into a running guest by setting the
corresponded IRR bit in the vAPIC backing page and trigger
AVIC_DOORBELL MSR.

IPI PERFORMANCE
===============

* BENCHMARK 1: HACKBENCH
For IPI, I have collected some performance number on 2 and 4 CPU running
hackbech with the following detail:

  hackbench -p -l 100000
  Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
  Each sender will pass 100000 messages of 100 bytes

                |    2 vcpus     |    4 vcpus 
 ------------------------------------------------
         Vanila |  273.76        |  190.21	
  AVIC disabled |  260.51 (~5%)  |  184.40 (~5%)
           AVIC |  248.53 (~10%) |  155.01 (~20%)

OVERALL PERFORMANCE
===================
Enabling AVIC should helps speeding up workloads, which generate
large amount of interrupts. However, it requires additional logics to
maintain AVIC-specific data structures during vCPU load/unload
due to vcpu scheduling.

The goal is to minimize the overhead of AVIC in most cases, so that
we can achieve equivalent or improvement in overall performance when
enabling AVIC.

* BENCHMARK 1: TAR DECOMPRESSION
This test measures the average running time (of 10 runs) of the following
tar decompression command with 1, 2, and 4 vcpus.

   tar xf linux-4.3.3.tar.xz

                |      4 vcpus 
 ---------------------------------
         Vanila |  10.26  
  AVIC disabled |  10.10 (~1.5%)
           AVIC |  10.07 (~1.8%)

Note: The unit of result below is in seconds (lower is better). 

* BENCHMARK 2: NETPERF w/ virtual network
This test creates a virtual network by setting up bridge and tap device
on the host and pass it into the VM as virtio-net-pci device w/ vhost. 
Then it sets up netserver in the host machine, and run netperf
in the VM with following option:

  netperf -H <netserver ip> -l 60 -t TCP_RR -D 2

                |    1 vcpu      
 ------------------------------------
         Vanila |  21623.887
  AVIC disabled |  21538.09 (~-.4%)
           AVIC |  21712.68 (~0.4%)

Note: The unit of result below is trans/sec (higher is better).

Preliminary result of both benchmarks show AVIC performance are slightly
better than the other two cases.

CURRENT UNTESTED USE-CASES
===========================
    - VM Migration (work in progress)
    - Nested VM

Any feedback and comments are very much appreciated.

Thank you,
Suravee

Radim Krčmář (1):
  KVM: split kvm_vcpu_wake_up from kvm_vcpu_kick

Suravee Suthikulpanit (10):
  KVM: x86: Misc LAPIC changes to expose helper functions
  KVM: x86: Introducing kvm_x86_ops VM init/uninit hooks
  KVM: x86: Introducing kvm_x86_ops VCPU blocking/unblocking hooks
  svm: Introduce new AVIC VMCB registers
  KVM: x86: Detect and Initialize AVIC support
  svm: Add interrupt injection via AVIC
  svm: Add VMEXIT handlers for AVIC
  svm: Do not expose x2APIC when enable AVIC
  svm: Do not intercept CR8 when enable AVIC
  svm: Manage vcpu load/unload when enable AVIC

 arch/x86/include/asm/kvm_host.h |  25 +-
 arch/x86/include/asm/svm.h      |  12 +-
 arch/x86/include/uapi/asm/svm.h |   9 +-
 arch/x86/kvm/lapic.c            | 127 ++++----
 arch/x86/kvm/lapic.h            |  32 ++
 arch/x86/kvm/svm.c              | 637 +++++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/trace.h            |  57 ++++
 arch/x86/kvm/x86.c              |   8 +
 include/linux/kvm_host.h        |   1 +
 virt/kvm/kvm_main.c             |  19 +-
 10 files changed, 833 insertions(+), 94 deletions(-)

-- 
1.9.1