From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AD5E3B2A0 for ; Fri, 7 Nov 2025 03:48:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762487297; cv=none; b=KNty10Pr80k/q+2ngVRqxVUS7FVfs/qUpApljpnuM5Ea2bS1KM3Qtv3Z19lpq4kwfxsGmc1xTeNsvinskZ0X8yO20a5Ks5Xmjmj+Vy408F94f+/r3R0AsaK2VxXkan9spF5izxBjbJOl4QedFuhoyjE7Cdo7L/Lyl9Kq/SDjW84= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762487297; c=relaxed/simple; bh=442eqadcHzio0qf/stWiPJed66HJJL13Xmo3zWX5t9w=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type; b=qJkKzKy+T9pXDoHKRSR+nw7Sht6miNIDlVKMSeW8+Xg84USyEjQ3pvaNb9T7iflyXcQj7xx8Zugl8qTTj7ecDvSb39wwGB9thlNyesjdLtgWSewgOB0dSUadFDncI+3DpMIdfGjYXXQrt3QgW1oWjFSw4Gm0XFWLIBBHKAuYZFw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aUiJEjyg; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aUiJEjyg" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-34188ba567eso236443a91.1 for ; Thu, 06 Nov 2025 19:48:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762487295; x=1763092095; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=MxGmPbv+36kk7i4AWF1GAropKIoZf8pJ2g2kJsx2bXg=; b=aUiJEjygcQmIL34LHkQ5bwi3jduGxN+kzqtHBKRcJUTIwuJEhNhBZYrCxIRqBy/LAO 6pMMHHJlDgouznCsoyesHvD6uPglk2+2mx5oh0QDurDuTJe/4hn/9HLVhEyK3px4PqIH wVfbs4oztbFzvZI8l0wGqxMUCJ6REajeREOiZ8Ru4pE0UuID5NkcmKz9Q8b9KCMvay96 J7chA6wVeS08B/OpNFQQvAu76RVn+uwBf5dIp6+yJqgIP+9ABtgV++1dtJItj+TH+WB/ C7rcymD0r6xG91j9Vx3dYUGP590KgzjzY0iNVYNAm9Yq6PyGRRhGr3Bd2mYP2iGLLXKj NUVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762487295; x=1763092095; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MxGmPbv+36kk7i4AWF1GAropKIoZf8pJ2g2kJsx2bXg=; b=kL7U9I9HdXt77tVG+15Hp2H+UWe4B3XGo6zihkBmdUl7zPNmQ5tcYIIfzMFzAf81hx f+3QgszQGLur7n6S18Q0/590TGYlW39bwpJauVfrJSB03wUU+J1y3wiIvKcwitRZBzXe psuFSxbcYP+AjQGeAfKoULE+RS+X1GOKwVUU1dRWyDlfRYfUMlZ1x9PO7HTtMaej7oNo wHZHHyQoNGrS6+c3YaceJBb9w1X4YMsqXUHY6deHYLOjEBumEs4tSz3tOAkpG/trg/1i 26oGUy7+37wBs0ty3WNzlUuPgM8se9KR6dgPzdO6KF2PObxY+708PU0RUqLxJZWwJklm +YNw== X-Forwarded-Encrypted: i=1; AJvYcCUAFjKo+dwq0613D62uJGD6W9hYP83CzHG0OphCj98LfkVoJFBxvzONZjVjG2i45s1cvYekO7L1GhS9/PY=@vger.kernel.org X-Gm-Message-State: AOJu0YyppXv42EhTZDqeqx1zQw1nYsrYnvCh8tmwrYZ8pI3XnUEU+AFV QHbnXOMoFraSGchaWFBlZqBrxBFAxx3NIAjnPoMGOK0CRa4jg9UE1RG4 X-Gm-Gg: ASbGncuQb32Y9UiBCNQXWtc2phWZhQ/m1aT2wfwNWnnbBICLg0awR5tY8X4+JqGhn/B iVyN5e5gzpa69R3P2ph2YoFsT2Qw0GHais8HbCvXoCAvAwzhfyXeupATglnUnGuqxC2zP2wvwIg 3d4QEOwU9JEF93sKbyLfn19IB0KkZNrTBRJ/OgABzYsaZufdO/45lPioclc2kLb2eWrEgrresjx OdcXyRFAp8ndaatz+msODmem7hqumlJMgCIcmk31mu2oQalSjWvEJ2/sDesZK8IN6NkhkPEWUUr 3TPcF3BnzIyt9Y58wuqJFrf4G4OlaevwwW/Kd4Arx8StGF/S0JT4Sq3XR0vs0syw9RA/iirOpaB gjOmkUAPuiWcotThKnFUn5BQQXluFkidtolpFmqBOM5vuZyCXVzFkf8jkH61NeCTKXVonulCy/+ BBHdpBSsu9l0b100ZWbw== X-Google-Smtp-Source: AGHT+IEsn7E1xrWwBVAA3UJlPFxnkFIozsp7ECdOPBKH4YIoxMww2ThRv51KM04jqJiCgXo74xxoJg== X-Received: by 2002:a17:90b:5250:b0:33e:2d0f:479c with SMTP id 98e67ed59e1d1-3434c5633cbmr1561991a91.22.1762487294718; Thu, 06 Nov 2025 19:48:14 -0800 (PST) Received: from localhost.localdomain ([129.227.63.233]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3434c356300sm989552a91.18.2025.11.06.19.48.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 06 Nov 2025 19:48:14 -0800 (PST) From: fuqiang wang To: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Marcelo Tosatti , "H . Peter Anvin" , Maxim Levitsky , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: fuqiang wang , yu chen <33988979@163.com>, dongxu zhang Subject: [PATCH v5 0/1] KVM: x86: fix some kvm period timer BUG Date: Fri, 7 Nov 2025 11:47:59 +0800 Message-ID: <20251107034802.39763-1-fuqiang.wng@gmail.com> X-Mailer: git-send-email 2.47.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=y Content-Transfer-Encoding: 8bit This patch fixes two issues with the period timer: ==================================================================== issue 1: avoid hv timer fallback to sw timer if delay exceeds period ==================================================================== When the guest uses the APIC periodic timer, if the next period has already expired, e.g. due to the period being smaller than the delay in processing the timer, the delta will be negative. nsec_to_cycles() may then convert this delta into an absolute value larger than guest_l1_tsc, resulting in a negative tscdeadline. Since the hv timer supports a maximum bit width of cpu_preemption_timer_multi + 32, this causes the hv timer setup to fail and switch to the sw timer. Moreover, due to the commit 98c25ead5eda ("KVM: VMX: Move preemption timer <=> hrtimer dance to common x86"), if the guest is using the sw timer before blocking, it will continue to use the sw timer after being woken up, and will not switch back to the hv timer until the relevant APIC timer register is reprogrammed. Since the periodic timer does not require frequent APIC timer register programming, the guest may continue to use the software timer for an extended period. Link [1] reproduces this issue by injecting a kernel module. This module creates a periodic hrtimer and adds a certain delay in its callback, making the delay longer than the KVM periodic timer period. ====================================================================== issue 2: VM hard lockup after prolonged suspend with periodic HV timer ====================================================================== Resuming a virtual machine after it has been suspended for a long time may trigger a hard lockup. The main reason is that the KVM periodic HV timer only advances during the VM-exit “VMX-preemption timer expired” event and when the vCPU is suspended or returns to user space for other reasons, the KVM timer stops advancing. Since the periodic timer expiration callback advances the timer by one period per invocation, this results in the callback being executed many times to catch up the expiration to the current timer value. Due to issue 1, the KVM periodic HV timer will switch to the software timer, and these catch-up will be executed within a single clock interrupt. If this process lasts long enough, it can easily lead to a hard lockup. One of our Windows virtual machines in the production environment triggered this case: NMI watchdog: Watchdog detected hard LOCKUP on cpu 45 ... RIP: 0010:advance_periodic_target_expiration+0x4d/0x80 [kvm] ... RSP: 0018:ff4f88f5d98d8ef0 EFLAGS: 00000046 RAX: fff0103f91be678e RBX: fff0103f91be678e RCX: 00843a7d9e127bcc RDX: 0000000000000002 RSI: 0052ca4003697505 RDI: ff440d5bfbdbd500 RBP: ff440d5956f99200 R08: ff2ff2a42deb6a84 R09: 000000000002a6c0 R10: 0122d794016332b3 R11: 0000000000000000 R12: ff440db1af39cfc0 R13: ff440db1af39cfc0 R14: ffffffffc0d4a560 R15: ff440db1af39d0f8 FS: 00007f04a6ffd700(0000) GS:ff440db1af380000(0000) knlGS:000000e38a3b8000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000d5651feff8 CR3: 000000684e038002 CR4: 0000000000773ee0 PKRU: 55555554 Call Trace: apic_timer_fn+0x31/0x50 [kvm] __hrtimer_run_queues+0x100/0x280 hrtimer_interrupt+0x100/0x210 ? ttwu_do_wakeup+0x19/0x160 smp_apic_timer_interrupt+0x6a/0x130 apic_timer_interrupt+0xf/0x20 And in link [2], Marcelo also reported this issue. But I don't think it can reproduce the issue. Because of commit [3], as long as the KVM timer is running, target_expiration will keep catching up to now (unless every single delay from timer virtualization is longer than the period, which is a pretty extreme case). Also, this patch is based on the patch of link [2], but with some differences: In link [2], target_expiration is updated to "now - period"(I'm not sure why it doesn't just catch up to now -- maybe I'm missing something?). In this patch, I set target_expiration to catch up to now just like how update_target_expiration handles the remaining. Link [4] provides details of the hard lockup details and as well as how to reproduce the KVM timer stop by pausing the virtual machine. ================================= Fix both issues in a single patch ================================= In versions v2 and v3, I split these two issues into two separate patches for fixing. However, this caused patch 2 to revert some of the changes made by patch 1. In patch 4, I attempted to merge the two patches into one and tried to describe both issues in the commit message, but I did not do it well. In this version, I have included more details in the commit message and the cover letter. Changes in v5: - Add more details in commit messages and letters. - link to v4: https://lore.kernel.org/all/20251105135340.33335-1-fuqiang.wng@gmail.com/ Changes in v4: - merge two patch into one - link to v3: https://lore.kernel.org/all/20251022150055.2531-1-fuqiang.wng@gmail.com/ Changes in v3: - Fix: advanced SW timer (hrtimer) expiration does not catch up to current time. - optimize the commit message of patch 2 - link to v2: https://lore.kernel.org/all/20251021154052.17132-1-fuqiang.wng@gmail.com/ Changes in v2: - Added a bugfix for hardlockup in v2 - link to v1: https://lore.kernel.org/all/20251013125117.87739-1-fuqiang.wng@gmail.com/ [1]: https://github.com/cai-fuqiang/kernel_test/tree/master/period_timer_test [2]: https://lore.kernel.org/kvm/YgahsSubOgFtyorl@fuller.cnet/ [3]: commit d8f2f498d9ed ("x86/kvm: fix LAPIC timer drift when guest uses periodic mode") [4]: https://github.com/cai-fuqiang/md/tree/master/case/intel_kvm_period_timer fuqiang wang (1): KVM: x86: Fix VM hard lockup after prolonged suspend with periodic HV timer arch/x86/kvm/lapic.c | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) -- 2.47.0