From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AFDF1F8901 for ; Tue, 21 Jan 2025 22:57:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737500225; cv=none; b=pAhuehiTrRfv+DUdiLOSFPQw0T2s3cXQ0iTPotmXDLU7raoVNmh91ijQ0USMjNOljNq8P/zxEkX6Fet3cm8533EdpYx0Q1nInY++1LCNkj8KwJY7j70/1FgtKdetAgeY/cjLwqEcUOKyp47raalLNuV5y7YLcsr67eN01a7wUy8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737500225; c=relaxed/simple; bh=8at++4Fkk1xiHDOzqEEnDbMsjBVs6dfC+bWTGD+uqMg=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=C6FNFuYPWaphXlgj+2B23rOV8IYFHqtdUWoFvNGgF76FZE62Kw7FWJ8LKjDAGqSfH1jr8UGFPt+GM28BzKC3Fx/W95yLwE0VWP1q2YxyuqS8nQDS6/78pxYAGmTf8sChYl0F/CwmFDsBSzypptjHNmj1dy/spzzNck7fvR0MHFE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=hSAchyN7; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hSAchyN7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737500222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b3D9YlXhxHkD2mD5Dh2VNJI1gCYrObVyeG/ox6a1e/Q=; b=hSAchyN7lCygALVGJPHUURce/jdYj7/eDrwpRfByW7D0QIQCkS/VAx2RY46dGXviNI3gjW pbnEnV9GX7YnX7ERIHPjUCXpvfW0ob6co7qEU5IYNCUpsUfS9f8qqmU/ivq9WijNIA3UpN /yUyinaTJaoMAvgdGD8mamKRVWEgtgg= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-620-5_OZ47T7MK2C6BI4grIjMQ-1; Tue, 21 Jan 2025 17:57:00 -0500 X-MC-Unique: 5_OZ47T7MK2C6BI4grIjMQ-1 X-Mimecast-MFC-AGG-ID: 5_OZ47T7MK2C6BI4grIjMQ Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6dfa69e6983so95950096d6.1 for ; Tue, 21 Jan 2025 14:57:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737500220; x=1738105020; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=b3D9YlXhxHkD2mD5Dh2VNJI1gCYrObVyeG/ox6a1e/Q=; b=RUzrHBrcRiDpZL4GLxS34tLUO4XatxkqUtJ46SoY1B/H59xZJuFigiQ9p0byr7HAlm IdEEhbIt15dsL9zaaigA9kkg7PWw0PyAEkJfwpFjOENia00WNbpR0iKE90ghext2vW/T tVnJV5vtrtsgTS5ngSH5ViPgbar4slBcAFM/73ELmLrrsCbpBya9pMlL+ZxSNZyNkE68 /Nhl88PR6CjsdmKnkK9Cx18Ch+XaEuTjXIpL4AAziiaN1ulVCgAHxtVa6oRVeOTFLEVx gGkxoh95IlOa0zydOvGkC/NsgngPhK5TwDnWMrkrBFAoeL1DSEV+RDp01mc/fnj/JmHG DWkQ== X-Forwarded-Encrypted: i=1; AJvYcCWN07Gk0llR+NHwM0GQElx8wknZybY/CTRfElJwJm4eBZn4UvJ1fEJjZvLjtwXDu/yo9rZRiEASBmMo3ME=@vger.kernel.org X-Gm-Message-State: AOJu0YzQTZx82K2Bvj1/oApOzMX/gNeWgd/hTCvF+3GalXM4hSokP6me n3uF1EvPBi9Iy62l/l5BgL+/qDuyd3jmnwZDmlQk/oA/BPQXUz/2NAgzbhgJLAi5949H5EMM3ad cGR8kW1dbJKP4XeS6OETIE6z0eHI+c+WGJIkimB2VhdNDIOJilJSYcPghNHOm0A== X-Gm-Gg: ASbGncsmW244ouu5yjP+OOXwfVzloDWcvks6FHyAib8CXtI54BgbluJPEsZ08LcUKvb MGOA4R+hJ/FXMEDLTUxM6MUMrwVXtCjt6E8x3m0ccFudh/K9KdZbJ1o1sLi3157MABlsYyDXT/z 9fRV2gWJtEWjxVfjkeD44kncs+DmyWs4fPI/fwOISMySvI7UuNkPnbQdO6cHhg0H3BzJ2Ol2HwE zyh+Oe/reoo4mC5AExUmt36RLX6hNZDMPKZDjakVI84K4w1bJZAT/CaQZhUevU8v9KCsA== X-Received: by 2002:a05:6214:2f81:b0:6da:dc79:a3cd with SMTP id 6a1803df08f44-6e1b19d8728mr352367916d6.0.1737500220075; Tue, 21 Jan 2025 14:57:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IFsj5TQXnSlY74PgtyV/qFWWKlRmrnSql4FvfOIgnBEwg0jRLE2q7PFa3M24dlyznMqCF6JQg== X-Received: by 2002:a05:6214:2f81:b0:6da:dc79:a3cd with SMTP id 6a1803df08f44-6e1b19d8728mr352367626d6.0.1737500219752; Tue, 21 Jan 2025 14:56:59 -0800 (PST) Received: from starship ([2607:fea8:fc01:8d8d:6adb:55ff:feaa:b156]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e1e2efcecesm11285206d6.17.2025.01.21.14.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jan 2025 14:56:59 -0800 (PST) Message-ID: Subject: Re: vmx_pmu_caps_test fails on Skylake based CPUS due to read only LBRs From: Maxim Levitsky To: Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Date: Tue, 21 Jan 2025 17:56:58 -0500 In-Reply-To: <1e4dc1788b846215126c5bcbce8d33cae6abd408.camel@redhat.com> References: <948408887cbe83cbcf05452a53d33fb5aaf79524.camel@redhat.com> <1e4dc1788b846215126c5bcbce8d33cae6abd408.camel@redhat.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit On Fri, 2024-12-13 at 19:20 -0500, Maxim Levitsky wrote: > On Thu, 2024-11-21 at 22:35 -0500, Maxim Levitsky wrote: > > On Sun, 2024-11-03 at 18:32 -0500, Maxim Levitsky wrote: > > > On Mon, 2024-10-28 at 08:55 -0700, Sean Christopherson wrote: > > > > On Fri, Oct 18, 2024, Maxim Levitsky wrote: > > > > > Hi, > > > > > > > > > > Our CI found another issue, this time with vmx_pmu_caps_test. > > > > > > > > > > On 'Intel(R) Xeon(R) Gold 6328HL CPU' I see that all LBR msrs (from/to and > > > > > TOS), are always read only - even when LBR is disabled - once I disable the > > > > > feature in DEBUG_CTL, all LBR msrs reset to 0, and you can't change their > > > > > value manually. Freeze LBRS on PMI seems not to affect this behavior. > > > > > > > > > > I don't know if this is how the hardware is supposed to work (Intel's manual > > > > > doesn't mention anything about this), or if it is something platform > > > > > specific, because this system also was found to have LBRs enabled > > > > > (IA32_DEBUGCTL.LBR == 1) after a fresh boot, as if BIOS left them enabled - I > > > > > don't have an idea on why. > > > > > > > > > > The problem is that vmx_pmu_caps_test writes 0 to LBR_TOS via KVM_SET_MSRS, > > > > > and KVM actually passes this write to actual hardware msr (this is somewhat > > > > > wierd), > > > > > > > > When the "virtual" LBR event is active in host perf, the LBR MSRs are passed > > > > through to the guest, and so KVM needs to propagate the guest values into hardware. > > > > > > Yes, but usually KVM_SET_MSRS doesn't touch hardware directly, even for registers/msrs > > > that are passed through, but rather the relevant values are loaded when the guest vCPU > > > is loaded and/or when the guest is entered. > > > I don't know the details though. > > > > > > > > > > > and since the MSR is not writable and silently drops writes instead, > > > > > once the test tries to read it, it gets some random value instead. > > > > > > > > This just showed up in our testing too (delayed backport on our end). I haven't > > > > (yet) tried debugging our setup, but is there any chance Intel PT is interfering? > > > > > > > > 33.3.1.2 Model Specific Capability Restrictions > > > > Some processor generations impose restrictions that prevent use of > > > > LBRs/BTS/BTM/LERs when software has enabled tracing with Intel Processor Trace. > > > > On these processors, when TraceEn is set, updates of LBR, BTS, BTM, LERs are > > > > suspended but the states of the corresponding IA32_DEBUGCTL control fields > > > > remained unchanged as if it were still enabled. When TraceEn is cleared, the > > > > LBR array is reset, and LBR/BTS/BTM/LERs updates will resume. > > > > Further, reads of these registers will return 0, and writes will be dropped. > > > > > > > > The list of MSRs whose updates/accesses are restricted follows. > > > > > > > > • MSR_LASTBRANCH_x_TO_IP, MSR_LASTBRANCH_x_FROM_IP, MSR_LBR_INFO_x, MSR_LASTBRANCH_TOS > > > > • MSR_LER_FROM_LIP, MSR_LER_TO_LIP > > > > • MSR_LBR_SELECT > > > > > > > > For processors with CPUID DisplayFamily_DisplayModel signatures of 06_3DH, > > > > 06_47H, 06_4EH, 06_4FH, 06_56H, and 06_5EH, the use of Intel PT and LBRs are > > > > mutually exclusive. > > > > > > > > If Intel PT is NOT responsible, i.e. the behavior really is due to DEBUG_CTL.LBR=0, > > > > then I don't see how KVM can sanely virtualize LBRs. > > > > > > > > > > Hi! > > > > > > > > > I will check PT influence soon, but to me it looks like the hardware implementation has changed. > > > It is just too consistent: > > > > > > When DEBUG_CTL.LBR=1, the LBRs do work, I see all the registers update, although > > > TOS does seem to be stuck at one value, but it does change sometimes, and it's non zero. > > > > > > The FROM/TO do show healthy amount of updates > > > > > > Note that I read all msrs using 'rdmsr' userspace tool. > > > > > > However as soon as I disable DEBUG_CTL.LBR, all these MSRs reset to 0, and can't be changed. > > > > Hi, > > I tested this on another skylake based machine (Intel(R) Xeon(R) Silver 4214) and I see the same behavior: > > LBR_TOS is readonly: > > > > It's 0 when LBRS disabled in DEBUG_CTL, and running (changes all the time as expected) > > when LBRS are enabled in the DEBUG_CTL. > > > > IA32_RTIT_CTL.TraceEn is disabled (msr 0x570 is 0). > > > > Also on this machine BIOS didn't left LBRs running. > > > > I guess we need to at least disable the check in the unit test or at least > > speak with someone from Intel to clarify on what is going on. > > Any update on this? Hi, I hate to sound like a broken record, but any update on this? Best regards, Maxim Levitsky > > > > > What do you think? > > > > Best regards, > > Maxim Levitsky > > > > > I'll check this on another Skylake based machine and see if I see the same thing. > > > > > > Best regards, > > > Maxim Levitsky > > > > > > > > >