From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 526311BA26F for ; Mon, 9 Sep 2024 13:24:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725888289; cv=none; b=feAucXLofHyM6FMyfVCNMzhTSKfCVOW9EdD8I0tLW95B+JoarqvQGgtiSeCAc84UwSHWmoIerG+ajDs/SIJW93dv911J9gqMIK5lQjPtO0Yhu9cA2PVPtWJeR5TDCHXp5ICDGJvbsfr+SzbHUOYT83Lf8eKiAh6n8eZFNEmjyFM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725888289; c=relaxed/simple; bh=aycUlN9RGtut06nhV6g6SMP5Z8jpjITcj8790qdzFvY=; h=Message-ID:Date:MIME-Version:Subject:To:References:From: In-Reply-To:Content-Type; b=MtW7oVzWg2zT3Q5YJQom4MzoofKCg/F2P+eM1gjJznmSEko/2KwX0gaEVa9zVHiizuyhP4fMrp4v4reNuiLJ0BSlAmAEKDQVHvsYOGT/QtxQWbbs9ZA1jFHNAG2airFDji+70a0yUvClq6iHdZ0Aq+S3TkxwAdrI3YTFhTH5vog= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=RBry5JQv; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RBry5JQv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725888286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=PJp6zde+o4+GuZZoS3lJrZtass9VDmatcifcWI992vs=; b=RBry5JQvGqJztIFf4rFRN1TTQlYfSpHbnv4QvaIIyU0mKWE8IZ9WBnwrVmzeoObcid2DO+ cHVL7Ek2aSAWE/kETt9UXXJnZSDUnFbYlWm5Ej9VFM58cXytaI9VIR9bfD/EYT4y3FhbwG nYoDa00dnk5yDPjRX6KQqpldAsvNFgs= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-227-XwxE41HROz63veOjYmqlcw-1; Mon, 09 Sep 2024 09:24:44 -0400 X-MC-Unique: XwxE41HROz63veOjYmqlcw-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-42cb471a230so9921895e9.3 for ; Mon, 09 Sep 2024 06:24:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725888283; x=1726493083; h=content-transfer-encoding:in-reply-to:autocrypt:content-language :from:references:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PJp6zde+o4+GuZZoS3lJrZtass9VDmatcifcWI992vs=; b=l9PiOPX4UddEaf0+16GBLiovtmWIKthKYwZliAsbY63UgHSBZDCx8afVha/bQwD9I5 wx9ZdEygHcf7OHWjWaqqN+LkXw3N9MwI9qTTKOOrJeTczD2PKQcgrvHXwYYUqoid2iiS tHj+ksHvWbcOggV3x/Z/XWG6/DnJgACwWZyRcgq8gSoklJ2g7tBHszBpH1lO4n7nFWif uONssroTH5oqXtFWg/RINPA/0Cqfqbju5ROW/rwhGUp/dP8x0is53NC75P9ZxRIJeD0r KfU/t4owbrW0Ezo2gwiwoqcDRGwnm3geZJbN7cywFQNZFug482XqSG91oFp5wgzrW4C8 pgeA== X-Forwarded-Encrypted: i=1; AJvYcCUxjOMKv3LZ73ExftfKqOMpcQZcMi882ytMZuUVOlzypwwvZc9slDBpb+b1RX+b8Iay+8MmcxXUpRsnuLI=@vger.kernel.org X-Gm-Message-State: AOJu0YwX889reVqOS+BdPQ3/SKd6+oF86b8ajweHTQFnmp/ozDYIonxP ThDrqwzpozOss05uVzcbULdtHFczZjsgGBrGVHzirriycACysxbwiVAxYy07NidkuUONyOx07pr TDWC+YX4nSt7L1LIXJNLh9rz8vk6IkrNFRxwGp3c0ASvjMXgtA8oJZsMGW8RajQ== X-Received: by 2002:a05:600c:470c:b0:42c:b52b:4335 with SMTP id 5b1f17b1804b1-42cb52b4671mr36819705e9.10.1725888283466; Mon, 09 Sep 2024 06:24:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF7wPSiSQaKkIioo/IjBCYves7XYbxdDqrPjlqHWw+mhwaLjI1mnzfbJTf9KmISlVSpPIMo/A== X-Received: by 2002:a05:600c:470c:b0:42c:b52b:4335 with SMTP id 5b1f17b1804b1-42cb52b4671mr36819415e9.10.1725888282948; Mon, 09 Sep 2024 06:24:42 -0700 (PDT) Received: from [192.168.10.81] ([151.95.101.29]) by smtp.googlemail.com with ESMTPSA id ffacd0b85a97d-378956653f9sm6041288f8f.31.2024.09.09.06.24.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 09 Sep 2024 06:24:42 -0700 (PDT) Message-ID: Date: Mon, 9 Sep 2024 15:24:40 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 5/5] KVM: VMX: Always honor guest PAT on CPUs that support self-snoop To: Yan Zhao , Sean Christopherson , Vitaly Kuznetsov , Gerd Hoffmann , kvm@vger.kernel.org, rcu@vger.kernel.org, linux-kernel@vger.kernel.org, Kevin Tian , Yiwei Zhang , Lai Jiangshan , "Paul E. McKenney" , Josh Triplett References: <871q26unq8.fsf@redhat.com> <87jzfutmfc.fsf@redhat.com> <87frqgu2t0.fsf@redhat.com> From: Paolo Bonzini Content-Language: en-US Autocrypt: addr=pbonzini@redhat.com; keydata= xsEhBFRCcBIBDqDGsz4K0zZun3jh+U6Z9wNGLKQ0kSFyjN38gMqU1SfP+TUNQepFHb/Gc0E2 CxXPkIBTvYY+ZPkoTh5xF9oS1jqI8iRLzouzF8yXs3QjQIZ2SfuCxSVwlV65jotcjD2FTN04 hVopm9llFijNZpVIOGUTqzM4U55sdsCcZUluWM6x4HSOdw5F5Utxfp1wOjD/v92Lrax0hjiX DResHSt48q+8FrZzY+AUbkUS+Jm34qjswdrgsC5uxeVcLkBgWLmov2kMaMROT0YmFY6A3m1S P/kXmHDXxhe23gKb3dgwxUTpENDBGcfEzrzilWueOeUWiOcWuFOed/C3SyijBx3Av/lbCsHU Vx6pMycNTdzU1BuAroB+Y3mNEuW56Yd44jlInzG2UOwt9XjjdKkJZ1g0P9dwptwLEgTEd3Fo UdhAQyRXGYO8oROiuh+RZ1lXp6AQ4ZjoyH8WLfTLf5g1EKCTc4C1sy1vQSdzIRu3rBIjAvnC tGZADei1IExLqB3uzXKzZ1BZ+Z8hnt2og9hb7H0y8diYfEk2w3R7wEr+Ehk5NQsT2MPI2QBd wEv1/Aj1DgUHZAHzG1QN9S8wNWQ6K9DqHZTBnI1hUlkp22zCSHK/6FwUCuYp1zcAEQEAAc0j UGFvbG8gQm9uemluaSA8cGJvbnppbmlAcmVkaGF0LmNvbT7CwU0EEwECACMFAlRCcBICGwMH CwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRB+FRAMzTZpsbceDp9IIN6BIA0Ol7MoB15E 11kRz/ewzryFY54tQlMnd4xxfH8MTQ/mm9I482YoSwPMdcWFAKnUX6Yo30tbLiNB8hzaHeRj jx12K+ptqYbg+cevgOtbLAlL9kNgLLcsGqC2829jBCUTVeMSZDrzS97ole/YEez2qFpPnTV0 VrRWClWVfYh+JfzpXmgyhbkuwUxNFk421s4Ajp3d8nPPFUGgBG5HOxzkAm7xb1cjAuJ+oi/K CHfkuN+fLZl/u3E/fw7vvOESApLU5o0icVXeakfSz0LsygEnekDbxPnE5af/9FEkXJD5EoYG SEahaEtgNrR4qsyxyAGYgZlS70vkSSYJ+iT2rrwEiDlo31MzRo6Ba2FfHBSJ7lcYdPT7bbk9 AO3hlNMhNdUhoQv7M5HsnqZ6unvSHOKmReNaS9egAGdRN0/GPDWr9wroyJ65ZNQsHl9nXBqE AukZNr5oJO5vxrYiAuuTSd6UI/xFkjtkzltG3mw5ao2bBpk/V/YuePrJsnPFHG7NhizrxttB nTuOSCMo45pfHQ+XYd5K1+Cv/NzZFNWscm5htJ0HznY+oOsZvHTyGz3v91pn51dkRYN0otqr bQ4tlFFuVjArBZcapSIe6NV8C4cEiSTOwE0EVEJx7gEIAMeHcVzuv2bp9HlWDp6+RkZe+vtl KwAHplb/WH59j2wyG8V6i33+6MlSSJMOFnYUCCL77bucx9uImI5nX24PIlqT+zasVEEVGSRF m8dgkcJDB7Tps0IkNrUi4yof3B3shR+vMY3i3Ip0e41zKx0CvlAhMOo6otaHmcxr35sWq1Jk tLkbn3wG+fPQCVudJJECvVQ//UAthSSEklA50QtD2sBkmQ14ZryEyTHQ+E42K3j2IUmOLriF dNr9NvE1QGmGyIcbw2NIVEBOK/GWxkS5+dmxM2iD4Jdaf2nSn3jlHjEXoPwpMs0KZsgdU0pP JQzMUMwmB1wM8JxovFlPYrhNT9MAEQEAAcLBMwQYAQIACQUCVEJx7gIbDAAKCRB+FRAMzTZp sadRDqCctLmYICZu4GSnie4lKXl+HqlLanpVMOoFNnWs9oRP47MbE2wv8OaYh5pNR9VVgyhD OG0AU7oidG36OeUlrFDTfnPYYSF/mPCxHttosyt8O5kabxnIPv2URuAxDByz+iVbL+RjKaGM GDph56ZTswlx75nZVtIukqzLAQ5fa8OALSGum0cFi4ptZUOhDNz1onz61klD6z3MODi0sBZN Aj6guB2L/+2ZwElZEeRBERRd/uommlYuToAXfNRdUwrwl9gRMiA0WSyTb190zneRRDfpSK5d usXnM/O+kr3Dm+Ui+UioPf6wgbn3T0o6I5BhVhs4h4hWmIW7iNhPjX1iybXfmb1gAFfjtHfL xRUr64svXpyfJMScIQtBAm0ihWPltXkyITA92ngCmPdHa6M1hMh4RDX+Jf1fiWubzp1voAg0 JBrdmNZSQDz0iKmSrx8xkoXYfA3bgtFN8WJH2xgFL28XnqY4M6dLhJwV3z08tPSRqYFm4NMP dRsn0/7oymhneL8RthIvjDDQ5ktUjMe8LtHr70OZE/TT88qvEdhiIVUogHdo4qBrk41+gGQh b906Dudw5YhTJFU3nC6bbF2nrLlB4C/XSiH76ZvqzV0Z/cAMBo5NF/w= In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 9/9/24 07:30, Yan Zhao wrote: > On Thu, Sep 05, 2024 at 05:43:17PM +0800, Yan Zhao wrote: >> On Wed, Sep 04, 2024 at 05:41:06PM -0700, Sean Christopherson wrote: >>> On Wed, Sep 04, 2024, Yan Zhao wrote: >>>> On Wed, Sep 04, 2024 at 10:28:02AM +0800, Yan Zhao wrote: >>>>> On Tue, Sep 03, 2024 at 06:20:27PM +0200, Vitaly Kuznetsov wrote: >>>>>> Sean Christopherson writes: >>>>>> >>>>>>> On Mon, Sep 02, 2024, Vitaly Kuznetsov wrote: >>>>>>>> FWIW, I use QEMU-9.0 from the same C10S (qemu-kvm-9.0.0-7.el10.x86_64) >>>>>>>> but I don't think it matters in this case. My CPU is "Intel(R) Xeon(R) >>>>>>>> Silver 4410Y". >>>>>>> >>>>>>> Has this been reproduced on any other hardware besides SPR? I.e. did we stumble >>>>>>> on another hardware issue? >>>>>> >>>>>> Very possible, as according to Yan Zhao this doesn't reproduce on at >>>>>> least "Coffee Lake-S". Let me try to grab some random hardware around >>>>>> and I'll be back with my observations. >>>>> >>>>> Update some new findings from my side: >>>>> >>>>> BAR 0 of bochs VGA (fb_map) is used for frame buffer, covering phys range >>>>> from 0xfd000000 to 0xfe000000. >>>>> >>>>> On "Sapphire Rapids XCC": >>>>> >>>>> 1. If KVM forces this fb_map range to be WC+IPAT, installer/gdm can launch >>>>> correctly. >>>>> i.e. >>>>> if (gfn >= 0xfd000 && gfn < 0xfe000) { >>>>> return (MTRR_TYPE_WRCOMB << VMX_EPT_MT_EPTE_SHIFT) | VMX_EPT_IPAT_BIT; >>>>> } >>>>> return MTRR_TYPE_WRBACK << VMX_EPT_MT_EPTE_SHIFT; >>>>> >>>>> 2. If KVM forces this fb_map range to be UC+IPAT, installer failes to show / gdm >>>>> restarts endlessly. (though on Coffee Lake-S, installer/gdm can launch >>>>> correctly in this case). >>>>> >>>>> 3. On starting GDM, ttm_kmap_iter_linear_io_init() in guest is called to set >>>>> this fb_map range as WC, with >>>>> iosys_map_set_vaddr_iomem(&iter_io->dmap, ioremap_wc(mem->bus.offset, mem->size)); >>>>> >>>>> However, during bochs_pci_probe()-->bochs_load()-->bochs_hw_init(), pfns for >>>>> this fb_map has been reserved as uc- by ioremap(). >>>>> Then, the ioremap_wc() during starting GDM will only map guest PAT with UC-. >>>>> >>>>> So, with KVM setting WB (no IPAT) to this fb_map range, the effective >>>>> memory type is UC- and installer/gdm restarts endlessly. >>>>> >>>>> 4. If KVM sets WB (no IPAT) to this fb_map range, and changes guest bochs driver >>>>> to call ioremap_wc() instead in bochs_hw_init(), gdm can launch correctly. >>>>> (didn't verify the installer's case as I can't update the driver in that case). >>>>> >>>>> The reason is that the ioremap_wc() called during starting GDM will no longer >>>>> meet conflict and can map guest PAT as WC. >>> >>> Huh. The upside of this is that it sounds like there's nothing broken with WC >>> or self-snoop. >> Considering a different perspective, the fb_map range is used as frame buffer >> (vram), with the guest writing to this range and the host reading from it. >> If the issue were related to self-snooping, we would expect the VNC window to >> display distorted data. However, the observed behavior is that the GDM window >> shows up correctly for a sec and restarts over and over. >> >> So, do you think we can simply fix this issue by calling ioremap_wc() for the >> frame buffer/vram range in bochs driver, as is commonly done in other gpu >> drivers? >> >> --- a/drivers/gpu/drm/tiny/bochs.c >> +++ b/drivers/gpu/drm/tiny/bochs.c >> @@ -261,7 +261,9 @@ static int bochs_hw_init(struct drm_device *dev) >> if (pci_request_region(pdev, 0, "bochs-drm") != 0) >> DRM_WARN("Cannot request framebuffer, boot fb still active?\n"); >> >> - bochs->fb_map = ioremap(addr, size); >> + bochs->fb_map = ioremap_wc(addr, size); >> if (bochs->fb_map == NULL) { >> DRM_ERROR("Cannot map framebuffer\n"); >> return -ENOMEM; While this is a fix for future kernels, it doesn't change the result for VMs already in existence. I don't think there's an alternative to putting this behind a quirk. Paolo