From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1427374AbcBSL6K (ORCPT ); Fri, 19 Feb 2016 06:58:10 -0500 Received: from mail-bl2on0060.outbound.protection.outlook.com ([65.55.169.60]:35059 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1425958AbcBSL6F (ORCPT ); Fri, 19 Feb 2016 06:58:05 -0500 Authentication-Results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=amd.com; Subject: Re: [PART1 RFC 6/9] svm: Add interrupt injection via AVIC To: Paolo Bonzini , , , References: <1455285574-27892-1-git-send-email-suravee.suthikulpanit@amd.com> <1455285574-27892-7-git-send-email-suravee.suthikulpanit@amd.com> <56BE005D.1040905@redhat.com> <56BE0678.7060103@amd.com> <56BE2223.5060506@redhat.com> CC: , , , , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= From: Suravee Suthikulpanit Message-ID: <56C70332.40603@amd.com> Date: Fri, 19 Feb 2016 18:57:38 +0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56BE2223.5060506@redhat.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [27.55.173.158] X-ClientProxiedBy: SIXPR04CA0013.apcprd04.prod.outlook.com (10.141.119.13) To SN1PR12MB0446.namprd12.prod.outlook.com (25.162.105.14) X-MS-Office365-Filtering-Correlation-Id: c16c27d3-7671-4b86-db07-08d33923ea4b X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0446;2:o6phLFvRaqD6zP1AQygz5Ky0OTbZ+RwFuB3IAB5mzIQgcZEhRwvNdBCmnef1FP5y/rsDoOec5rhzdrpMpOMh/M0he/I+jU1vRsIMDNslBtEKXpMkNq0BuURKiSkI47kaISjZDG6PoA1wz86+0Ce/heBQt2cs036j6dBBeJgo/CpaOt+lclQQqreBbr+UuLuh;3:Ovar5mh1CuN+sxsgFW2h7ppOAjzzse00c5HufdaX9aMsx48HNkB3DbXcbsxjt8Uj5DFqlGs7hhB1+g9OLjXAVr8+1xIutw8bA/V2vk+lApnpUzUneeFMfBspAzvLsUVt;25:9RatdeRI3+N+Vp343Ej1KNXUb6izZl4R8d2xF/aIqxzkZnx453SWtLvPGcaVCVjVoleiEi/NcBLe2+UiC9T/D/nO4DXhPAGEZLOvZi8wCZwqdWC7Ay6mrCfOksq04O1QdZ9s66UEbH6zOY1HUY/bkI0TXtXGXYpha7uSisS7vFqiMX7WyEntWDIBnnI5z/WCqjrU+If19/5yH+MhVQVAIc+W96ZQitAJMnMuooME07IkvsxcSukOPL6f1rx9AzQKXC7xPrlS3TcevcuU0i3Ui/1gy2kb2SlrOFm1b5xFXLxgtDNBbMg0LlhEIqtFwQF3jeRcvC1n0saL3dHuE0YKq3Xf7CEQH73e4wofARRwDvE= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR12MB0446; X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0446;20:mrsByCRjewkfWVA4HrvKwqKbCYMMaHHIJt0+QX2iP9JZj6MGCcYtIBw7SCNSQW+L+c6s4ZNIn+140DnnSmWLQQeLuVecGPeZJ76RNLbQQn5d7ydU/fU5DRRsEi4VoquaIa9qTkNZ5p4h6rk2/ePdq8d0hj02+cAaLkVuBDDTlmYpQM+aHb+GaOzZI4KG0KT4LXrxdd+i2TVDVU0TbKkSwuH49oJPv2gBKyERsVdYYYJ1vAWU2Y6av+W4QfT4Do0H3nycHFH3oYPzwHUS2XXqj0R1+lbVwj+8t/wrQsDyDIes0MKkBMK1TggkolJdmT+2N9jO4krtVX/r233J74v0D9L51oq0ghoqR3adlPhgNIzWBqmlZum466dVr0uBoMXiUk/rK6VwkHQjOJqz6QBWrxqLt3iaQu3ONHubFCUG5W5hhB+WHifuymtFT2vctafh3yOiEMoCqslVOYog+FptHJHcbt/3xRr3Gb1p9Gzuz0GLUnR/1grE6hkzxJRVPk/o;4:SeOyrF4zvKgBh5ut8e/s2xKxMA8OApT3IB/27w9JymZKds09gQyoD+7p7l3BUZxIZECmlrZz/vo8jaiV1Ua32bCPwuY6IvaiZrB36ChAfpyvGFC2peNtsyXuPqdEFG8Ovm++xU2C11J+V0UVs3Sk8FBPc95LVDYEKKf1hw2qoaBdkzjhekI3EakzrFlfAAM4f+PWRvEno21ZvJGToeLMKEg/Y771F/UZw2mxla2YyCuYiMjyOX8xtJE2nMEKGBUTV/wC85h9j8fRNDsn5uB8i4HCAEgbJlAywkIgGQpdK5Y+aLGADCsne6GDIOO8ZGVL+eCLYUoOi3pED5Ejk7WPwbIAx2kflXU4oAWlgxU5tnJHHUTAldQpYQqwyjuHnOHV X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:SN1PR12MB0446;BCL:0;PCL:0;RULEID:;SRVR:SN1PR12MB0446; X-Forefront-PRVS: 08572BD77F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(377454003)(479174004)(24454002)(164054003)(77096005)(2950100001)(33656002)(50986999)(2201001)(40100003)(5008740100001)(54356999)(65816999)(5004730100002)(87976001)(42186005)(76176999)(19580395003)(86362001)(93886004)(122386002)(64126003)(47776003)(5001770100001)(4326007)(117156001)(5001960100002)(189998001)(92566002)(3846002)(2906002)(230700001)(36756003)(1096002)(23746002)(66066001)(586003)(65956001)(6116002)(50466002);DIR:OUT;SFP:1101;SCL:1;SRVR:SN1PR12MB0446;H:[192.168.43.18];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;SN1PR12MB0446;23:rnBPgcQnuv6QWvFyAFC/fFXmIxHoPIzewLqO3?= =?Windows-1252?Q?6hYB1DTxMWzSJd7/cyBMO8ZiN0UnrQKiol7+IKD+rujQ6yRNt/fsyKVk?= =?Windows-1252?Q?MEy5odaZ2Ao/SgUHAh3Rc1bG/S7PvPSwudKzeKyBL2P99Cmq5dWYbnEG?= =?Windows-1252?Q?Z03VGlb8QCoGAtHS9cKDIGVHWaJLQIH/SudSudNXoVHpnv4G/SwUTpv+?= =?Windows-1252?Q?6yQfCJZfb9+1b6jzHTw8kp9raVm01W3rldmjJyfeeodVWo6TCsp+5ExI?= =?Windows-1252?Q?8OdMN5Iz1sK9q6TrGzARI0J067jl2Vf7+XXo57HMEko20VnkT0FL7LOF?= =?Windows-1252?Q?y1TBo3Nd7yLlCX9fRb7K+bPbjuyebwOwvixTWg3qyPZ5GCZiIZGOYpbT?= =?Windows-1252?Q?8mg0INHwqzEWAKK+UPAzsgzLPv9YZdECGdAh98gWHIVC8UqMJ9+JCPiK?= =?Windows-1252?Q?kIxTQH2xLstnjBzLhfLyBRTPYy0QbUegLfA8DLSu0lP7EiofFpk6rCHx?= =?Windows-1252?Q?28WD6yrwX7PtIeqrEjJq9ukzZny0etYv9acIokZCrpu5VCrhZJbB2Qgl?= =?Windows-1252?Q?qkfBDaQ1U2RBXT6XOmjzSnMGseZ8bRP9OUdhQhCopGkj8Qy3x8K7Ahjq?= =?Windows-1252?Q?8CzSIc/dYy2bKMBjqfB/9Fu/tEVKvOST1rTkpurpfyQ54rfIfquOxg34?= =?Windows-1252?Q?cFz7bVCH0KV26l1pvWmrDsVaCQvw77wdnuoh+ZrCjx71JH0qefewaWSN?= =?Windows-1252?Q?FehqaacaUs4BBylgaaGfYEGSvuWjlFMqQGOJFkbtgTK6iYv0bDusNmSX?= =?Windows-1252?Q?+vY4hGwfx0Vo1VT82JU6S61oeLeo3zCP8I9q++LakeI73uiBZJ5Lt2dg?= =?Windows-1252?Q?jqlmIMaVhDpdnCZFcodBORCNtYDl8iMDnGGGZyBzM6h2UNEsStcB5oGK?= =?Windows-1252?Q?f7xRdvbaDSLmNJ/VNY1LpgniyOG09idzqZyJMsXpCO5uODWoEK4TkfYE?= =?Windows-1252?Q?b8Y401ktVWRpEwCY9pR2gjeXLwwA/1Pl5RGvQZbl0BmSiSAiEARl8ucB?= =?Windows-1252?Q?qoTrDHaCTSJEj3D4u0N4LIbB8Rcgzga/pFlh/uOtJ5PeVaVDbrdKZkfN?= =?Windows-1252?Q?o0BswcRDRpdaW2OL+slenBuJjRXqHnEwyhJf3SeYm83RhKWvudzV0zr2?= =?Windows-1252?Q?neZxwCY3tOcaG5UZOwCFDtGS5+xPGGfN8/OdXdU9QtJDTtfZIOW?= X-Microsoft-Exchange-Diagnostics: 1;SN1PR12MB0446;5:4/ih8i+ua7P4rnUQIw6vIyR3UyW9yR/YxO9G3E5UdqZHUOismathO2FPzc3n4zAZA11WYT0s7D9RnJ6X4V+6X1mdsp0NRMMo+rruUzMS/w24JE4AzfxS8f+QF9Kf6uUQhHE3rb/Z1YfbDaJ2WPk5QQ==;24:0xujytMjhwT12rfezFweE7zLRsqj8RgEVNEDW23DFfeJDF8UyFvgsrVkwAUy4/18DVnLKQJrFe5RFpN++ve0MwZXKRFbgxam5KXkzV0YLqY=;20:zWUoHJ+6Oyqrx4QMvvRXCMY5sEuQiwqx11P7HTlNxiLaM41vCEG9morRwRtbqr+NyVShtoXrx6uCPfWHA5B9b6XS1jIUmv3YtCKEXMaQYA6P2F3ITUKXeJqPr8oQUVBYxdAy6iqTpJpav27tfpOxt/n7l2llm//Fszz2DKPZ2vJeZXxnhOEoii6aHWas2rHPt/n37ggXDDPsrQem4N3briDByAufnOmIW89OhmlH4mHS5Wv6Nb5loRM0+eMFkdie X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2016 11:57:56.3254 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB0446 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Paolo, On 2/13/16 01:19, Paolo Bonzini wrote: > > > On 12/02/2016 17:21, Suravee Suthikulpanit wrote: >> Hi Paolo, >> >> On 02/12/2016 10:55 PM, Paolo Bonzini wrote: >>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c >>>>> index 4244c2b..2def290 100644 >>>>> --- a/arch/x86/kvm/x86.c >>>>> +++ b/arch/x86/kvm/x86.c >>>>> @@ -8087,7 +8087,9 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) >>>>> if (is_guest_mode(vcpu) && kvm_x86_ops->check_nested_events) >>>>> kvm_x86_ops->check_nested_events(vcpu, false); >>>>> >>>>> - return kvm_vcpu_running(vcpu) || kvm_vcpu_has_events(vcpu); >>>>> + return (kvm_vcpu_running(vcpu) || kvm_vcpu_has_events(vcpu) || >>>>> + (kvm_x86_ops->apicv_intr_pending && >>>>> + kvm_x86_ops->apicv_intr_pending(vcpu))); >>>>> } >>> I think this is not necessary. What you need is to make kvm_lapic's >>> regs field point to the backing page. Then when the processor writes to >>> IRR, kvm_apic_has_interrupt (called through kvm_vcpu_has_events) will >>> see it. >>> >>> avic_pending_cnt shouldn't be necessary either. >>> >>> Paolo Actually, I also found out during another benchmark (running tar xf linux.tar.gz) on the VM w/ multiple cpus, that the performance is quite bad due to large amount of AVIC_INCOMP_IPI vmexit for to target not running. The same issue does not happen with 1 vcpu, or taskset the tar process to one vcpu, or if I put in the logic above in kvm_arch_vcpu_runnable() to delay the halting. >> >> So, the other thing I am using the avic_pending_cnt for is for the part >> 2 of the series (to enable AVIC support in IOMMU) that I am planning to >> send out later. However, it might be good to discuss this at this point. > > It's better to discuss it later. For now, I would prefer the AVIC > patches to be as clean as possible, and not know about the IOMMU at all. > Also, there are a lot of assumptions about how to use kvm_lapic's regs > field for APIC virtualization---dating back to when Intel only > virtualized the TPR field. Deviating for that would be a recipe for > trouble. :) > > Regarding the IOMMU, I'm actually very happy with the way the Intel VT-d > posted interrupts patches worked out, so I would be even more happy if > everything you do fits in the same scheme and reuses the same hooks! :D > >> When the IOMMU cannot inject interrupts into the guest vcpu due to it is >> not running (therefore, it cannot doorbell the vcpu directly), it logs >> the interrupt in the GA log buffer. >> >> Then it generates interrupt to >> notify the IOMMU driver that it needs to handle the log entry. Here, the >> IOMMU driver will end up notifying the SVM to scheduling the VCPU in to >> process interrupt. >> >> Here, I have run into issue where the vcpu often goes into idle (i.e. >> scheduled out), and ended up causing IOMMU to generate a lot of the >> entries in the GA log. This really hurts device pass-through performance >> (e.g. for XGBE NIC). >> >> So, what I ended up experimenting with is to set the avic_pending_cnt to >> a larger value (i.e. avic_ga_log_threshold) whenever we processing the >> GA log entry. The intention is to delay the vcpu schedule out in >> expecting that there might be more interrupts coming in soon. I also >> make this threshold value tunable as a module_param. >> >> This actually works well in my experiment, where I can actually get >> about 5% speed up in my netperf test on XGBE NIC pass-through test. >> However, I am not sure if this is an acceptable approach. Actually, I >> think it's similar to the halt_poll_ns, but specifically for IOMMU GA >> log in this case. > > Have you retested now that the halt_poll_ns mechanism is dynamic and > enabled by default? If I read patch 9 right, halt_poll_ns would delay > vcpu_put and IsRunning=0. Hopefully this is enough to avoid this kind > of notification and make the issue moot. > > Paolo > I assume that the halt_poll_ns mechanism would have been already enabled by default as off commit 93c9247cfd1e608e262274616a28632681abb2d3. So, I have tried playing with halt_poll_ns, halt_poll_ns_[grow|shrink], but it doesn't seem to help much. Thanks, Suravee