[PATCH] KVM: VMX: Fix rflags cache during vCPU reset

[PATCH] KVM: VMX: Fix rflags cache during vCPU reset

[Wanpeng Li]

From: Wanpeng Li <wanpeng.li@hotmail.com>

Reported by syzkaller:

   *** Guest State ***
   CR0: actual=0x0000000080010031, shadow=0x0000000060000010, gh_mask=fffffffffffffff7
   CR4: actual=0x0000000000002061, shadow=0x0000000000000000, gh_mask=ffffffffffffe8f1
   CR3 = 0x000000002081e000
   RSP = 0x000000000000fffa  RIP = 0x0000000000000000
   RFLAGS=0x00023000         DR7 = 0x0000000000000002
   ..........................
   ------------[ cut here ]------------
   WARNING: CPU: 6 PID: 24431 at /home/kernel/linux/arch/x86/kvm//x86.c:7302 kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
   CPU: 6 PID: 24431 Comm: reprotest Tainted: G        W  OE   4.14.0+ #26
   RIP: 0010:kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
   RSP: 0018:ffff880291d179e0 EFLAGS: 00010202
   Call Trace:
    ? synchronize_rcu.part.59+0x9b/0xf0
    ? synchronize_rcu_expedited+0xa0/0xa0
    ? kfree_call_rcu+0x20/0x20
    ? trace_raw_output_rcu_utilization+0x70/0x70
    ? kvm_arch_vcpu_runnable+0x220/0x220 [kvm]
    ? wait_for_completion+0x222/0x2a0
    ? __wait_rcu_gp+0x1d3/0x230
    ? kvm_vcpu_ioctl+0x479/0x880 [kvm]
    kvm_vcpu_ioctl+0x479/0x880 [kvm]
    ? kvm_dev_ioctl+0xba0/0xba0 [kvm]
    ? pick_next_task_fair+0x629/0xce0
    ? sched_clock+0x1b/0x20
    ? check_chain_key+0x131/0x1e0
    ? lock_acquire+0x2c0/0x2c0
    ? check_flags.part.44+0x62/0x250
    ? check_chain_key+0x131/0x1e0
    ? __lock_acquire+0xd2e/0x1cb0
    do_vfs_ioctl+0x142/0x9a0
    ? ioctl_preallocate+0x150/0x150
    ? lock_acquire+0x2c0/0x2c0
    ? lock_acquire+0x118/0x2c0
    ? __lock_is_held+0x4b/0xe0
    ? __fget+0x16a/0x250
    SyS_ioctl+0x74/0x80
    entry_SYSCALL_64_fastpath+0x23/0x9a

The syzkaller testcase tries to setup the processor specific debug registers 
and configure vCPU for handling guest debug events through KVM_SET_GUEST_DEBUG.
The KVM_SET_GUEST_DEBUG ioctl will get and set rflags in order to set TF bit 
if single step is needed. All regs' caches are reset to avail and GUEST_RFLAGS 
vmcs field is reset to 0x2 during vCPU reset. However, the cache of rflags is 
not reset during vCPU reset. The function vmx_get_rflags() returns an unreset 
rflags cache value since the cache is marked avail, it is 0 after boot. Vmentry 
fails if the rflags reserved bit 1 is 0.

This patch fixes it by resetting both the GUEST_RFLAGS vmcs field and its cache 
to 0x2 during vCPU reset.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: Nadav Amit <nadav.amit@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
 arch/x86/kvm/vmx.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index b348920..131fa1c 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -5590,6 +5590,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 		vmcs_write64(GUEST_IA32_DEBUGCTL, 0);
 	}
 
+	vmx->rflags = 0x02;
 	vmcs_writel(GUEST_RFLAGS, 0x02);
 	kvm_rip_write(vcpu, 0xfff0);
 
-- 
2.7.4

Re: [PATCH] KVM: VMX: Fix rflags cache during vCPU reset

[David Hildenbrand]

On 20.11.2017 10:17, Wanpeng Li wrote:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
> 
> Reported by syzkaller:
> 
>    *** Guest State ***
>    CR0: actual=0x0000000080010031, shadow=0x0000000060000010, gh_mask=fffffffffffffff7
>    CR4: actual=0x0000000000002061, shadow=0x0000000000000000, gh_mask=ffffffffffffe8f1
>    CR3 = 0x000000002081e000
>    RSP = 0x000000000000fffa  RIP = 0x0000000000000000
>    RFLAGS=0x00023000         DR7 = 0x0000000000000002
>    ..........................
>    ------------[ cut here ]------------
>    WARNING: CPU: 6 PID: 24431 at /home/kernel/linux/arch/x86/kvm//x86.c:7302 kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
>    CPU: 6 PID: 24431 Comm: reprotest Tainted: G        W  OE   4.14.0+ #26
>    RIP: 0010:kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
>    RSP: 0018:ffff880291d179e0 EFLAGS: 00010202
>    Call Trace:
>     ? synchronize_rcu.part.59+0x9b/0xf0
>     ? synchronize_rcu_expedited+0xa0/0xa0
>     ? kfree_call_rcu+0x20/0x20
>     ? trace_raw_output_rcu_utilization+0x70/0x70
>     ? kvm_arch_vcpu_runnable+0x220/0x220 [kvm]
>     ? wait_for_completion+0x222/0x2a0
>     ? __wait_rcu_gp+0x1d3/0x230
>     ? kvm_vcpu_ioctl+0x479/0x880 [kvm]
>     kvm_vcpu_ioctl+0x479/0x880 [kvm]
>     ? kvm_dev_ioctl+0xba0/0xba0 [kvm]
>     ? pick_next_task_fair+0x629/0xce0
>     ? sched_clock+0x1b/0x20
>     ? check_chain_key+0x131/0x1e0
>     ? lock_acquire+0x2c0/0x2c0
>     ? check_flags.part.44+0x62/0x250
>     ? check_chain_key+0x131/0x1e0
>     ? __lock_acquire+0xd2e/0x1cb0
>     do_vfs_ioctl+0x142/0x9a0
>     ? ioctl_preallocate+0x150/0x150
>     ? lock_acquire+0x2c0/0x2c0
>     ? lock_acquire+0x118/0x2c0
>     ? __lock_is_held+0x4b/0xe0
>     ? __fget+0x16a/0x250
>     SyS_ioctl+0x74/0x80
>     entry_SYSCALL_64_fastpath+0x23/0x9a
> 
> The syzkaller testcase tries to setup the processor specific debug registers 
> and configure vCPU for handling guest debug events through KVM_SET_GUEST_DEBUG.
> The KVM_SET_GUEST_DEBUG ioctl will get and set rflags in order to set TF bit 
> if single step is needed. All regs' caches are reset to avail and GUEST_RFLAGS 
> vmcs field is reset to 0x2 during vCPU reset. However, the cache of rflags is 
> not reset during vCPU reset. The function vmx_get_rflags() returns an unreset 
> rflags cache value since the cache is marked avail, it is 0 after boot. Vmentry 
> fails if the rflags reserved bit 1 is 0.
> 
> This patch fixes it by resetting both the GUEST_RFLAGS vmcs field and its cache 
> to 0x2 during vCPU reset.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Radim Krčmář <rkrcmar@redhat.com>
> Cc: Nadav Amit <nadav.amit@gmail.com>
> Cc: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> ---
>  arch/x86/kvm/vmx.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index b348920..131fa1c 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5590,6 +5590,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
>  		vmcs_write64(GUEST_IA32_DEBUGCTL, 0);
>  	}

Indeed, looks like the right think to do when looking at vmx_set_rflags().

Reviewed-by: David Hildenbrand <david@redhat.com>

>  
> +	vmx->rflags = 0x02;
>  	vmcs_writel(GUEST_RFLAGS, 0x02);
>  	kvm_rip_write(vcpu, 0xfff0);
>  
> 


-- 

Thanks,

David / dhildenb

Re: [PATCH] KVM: VMX: Fix rflags cache during vCPU reset

[Paolo Bonzini]

On 20/11/2017 10:17, Wanpeng Li wrote:
>    ------------[ cut here ]------------
>    WARNING: CPU: 6 PID: 24431 at /home/kernel/linux/arch/x86/kvm//x86.c:7302 kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
>    CPU: 6 PID: 24431 Comm: reprotest Tainted: G        W  OE   4.14.0+ #26
>    RIP: 0010:kvm_arch_vcpu_ioctl_run+0x651/0x2ea0 [kvm]
>    RSP: 0018:ffff880291d179e0 EFLAGS: 00010202
>    Call Trace:
>     ? synchronize_rcu.part.59+0x9b/0xf0
>     ? synchronize_rcu_expedited+0xa0/0xa0
>     ? kfree_call_rcu+0x20/0x20
>     ? trace_raw_output_rcu_utilization+0x70/0x70
>     ? kvm_arch_vcpu_runnable+0x220/0x220 [kvm]
>     ? wait_for_completion+0x222/0x2a0
>     ? __wait_rcu_gp+0x1d3/0x230
>     ? kvm_vcpu_ioctl+0x479/0x880 [kvm]
>     kvm_vcpu_ioctl+0x479/0x880 [kvm]
>     ? kvm_dev_ioctl+0xba0/0xba0 [kvm]
>     ? pick_next_task_fair+0x629/0xce0
>     ? sched_clock+0x1b/0x20
>     ? check_chain_key+0x131/0x1e0
>     ? lock_acquire+0x2c0/0x2c0
>     ? check_flags.part.44+0x62/0x250
>     ? check_chain_key+0x131/0x1e0
>     ? __lock_acquire+0xd2e/0x1cb0
>     do_vfs_ioctl+0x142/0x9a0
>     ? ioctl_preallocate+0x150/0x150
>     ? lock_acquire+0x2c0/0x2c0
>     ? lock_acquire+0x118/0x2c0
>     ? __lock_is_held+0x4b/0xe0
>     ? __fget+0x16a/0x250
>     SyS_ioctl+0x74/0x80
>     entry_SYSCALL_64_fastpath+0x23/0x9a

Please leave out the "?" lines of the backtrace, and if possible include
the C code for the testcase.

> The syzkaller testcase tries to setup the processor specific debug registers 
> and configure vCPU for handling guest debug events through KVM_SET_GUEST_DEBUG.
> The KVM_SET_GUEST_DEBUG ioctl will get and set rflags in order to set TF bit 
> if single step is needed. All regs' caches are reset to avail and GUEST_RFLAGS 
> vmcs field is reset to 0x2 during vCPU reset. However, the cache of rflags is 
> not reset during vCPU reset. The function vmx_get_rflags() returns an unreset 
> rflags cache value since the cache is marked avail, it is 0 after boot. Vmentry 
> fails if the rflags reserved bit 1 is 0.
> 
> This patch fixes it by resetting both the GUEST_RFLAGS vmcs field and its cache 
> to 0x2 during vCPU reset.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Radim Krčmář <rkrcmar@redhat.com>
> Cc: Nadav Amit <nadav.amit@gmail.com>
> Cc: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> ---
>  arch/x86/kvm/vmx.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> index b348920..131fa1c 100644
> --- a/arch/x86/kvm/vmx.c
> +++ b/arch/x86/kvm/vmx.c
> @@ -5590,6 +5590,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
>  		vmcs_write64(GUEST_IA32_DEBUGCTL, 0);
>  	}
>  
> +	vmx->rflags = 0x02;
>  	vmcs_writel(GUEST_RFLAGS, 0x02);
>  	kvm_rip_write(vcpu, 0xfff0);
>  
> 

Maybe this should be just kvm_set_rflags(vcpu, 2); just like in svm.c's
init_vmcb?

Thanks,

Paolo