From: Mukesh R <mrathor@linux.microsoft.com>
To: Michael Kelley <mhklinux@outlook.com>,
"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: "haiyangz@microsoft.com" <haiyangz@microsoft.com>,
"wei.liu@kernel.org" <wei.liu@kernel.org>,
"decui@microsoft.com" <decui@microsoft.com>,
"kameroncarr@linux.microsoft.com"
<kameroncarr@linux.microsoft.com>
Subject: Re: [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall
Date: Thu, 19 Nov 2026 17:31:26 -0800 [thread overview]
Message-ID: <e6bffec5-064c-aa37-6e24-11d90124805d@linux.microsoft.com> (raw)
In-Reply-To: <MN2PR02MB68639EE1AC00187FDA3D5A42D47B2@MN2PR02MB6863.namprd02.prod.outlook.com>
On 11/17/26 09:41, Michael Kelley wrote:
> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Monday, November 16, 2026 6:29 PM
>>
>> On 11/15/26 15:48, Michael Kelley wrote:
>>> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Friday, November 13, 2026 6:53 PM
>>>>
>>>> There are few issues/bugs around repeat hypercall:
>>>> o If the hypercall fails, it could still have processed some entries.
>>>> Fix places where it assumes none of the entries were processed and will
>>>> try to undo entries already processed or print incorrect completions.
>>>>
>>>> o Hypercall returns back with timeouts only for HW confidential VMs. In
>>>> that case it should be retried instead of returning back. Returning back
>>>> results in error conditions and take down of VMs in most cases.
>>>>
>>>> o For non confidential VMs, the hypercall could be streamlined as it will
>>>> never return back with timeouts.
>>>>
>>>> o Rename HV_STATUS_TIME_OUT to HV_STATUS_TIMEOUT to match hypervisor
>>>> public header file. There is no reason for it to be different.
>>>>
>>>> o Minor formatting fix for checkpatch. Error printing missing in couple
>>>> places when rep hypercall fails.
>
> In my view, the changes to the call sites of hv_do_rep_hypercall() should
> be in a separate patch. These are just bugs in how the undo is handled
> in case of an error, and the fixes presumably want to backported.
>
> Handling a TIMEOUT return should also be a separate patch. This is
> also a bug and the fix should be backported. The TIMEOUT is
> generated under a specific set of circumstances -- it's not a generic
> error. The patch should describe how and why those circumstances arise
> in the overall context of hypercall continuation. There's not any such
> documentation elsewhere, so the info should be captured in the
> patch commit message or code comments for future reference.
>
> Finally, if the looping behavior in a normal VM is to be removed,
> that would be a third patch. Given that this changes long-standing
> behavior, some explanation of "why" is needed in the commit
> message. I don't think this change should be backported.
Well, turns out the timeouts need to be handled for all hypercalls,
not just repeat hypercalls.... so i think it is best to just split
the series. I'll send separate patches now. JFYI.
Thanks,
-Mukesh
>>>>
>>>> Signed-off-by: Mukesh R <mrathor@linux.microsoft.com>
>>>
>>> [snip]
>>>
>>>> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
>>>> index 396bcb3db0bc..87c784b9daa6 100644
>>>> --- a/include/asm-generic/mshyperv.h
>>>> +++ b/include/asm-generic/mshyperv.h
>>>> @@ -28,6 +28,9 @@
>>>>
>>>> #define VTPM_BASE_ADDRESS 0xfed40000
>>>>
>>>> +#define HV_REP_COUNT_MAX \
>>>> + (HV_HYPERCALL_REP_COMP_MASK >> HV_HYPERCALL_REP_COMP_OFFSET)
>>>
>>> Sashiko noted that this new #define is never used anywhere.
>>>
>>>> +
>>>> enum hv_partition_type {
>>>> HV_PARTITION_TYPE_GUEST,
>>>> HV_PARTITION_TYPE_ROOT,
>>>> @@ -128,6 +131,11 @@ static inline bool hv_result_success(u64 status)
>>>> return hv_result(status) == HV_STATUS_SUCCESS;
>>>> }
>>>>
>>>> +static inline bool hv_result_timeout(u64 status)
>>>> +{
>>>> + return hv_result(status) == HV_STATUS_TIMEOUT;
>>>> +}
>>>> +
>>>> static inline unsigned int hv_repcomp(u64 status)
>>>> {
>>>> /* Bits [43:32] of status have 'Reps completed' data. */
>>>> @@ -135,45 +143,63 @@ static inline unsigned int hv_repcomp(u64 status)
>>>> HV_HYPERCALL_REP_COMP_OFFSET;
>>>> }
>>>>
>>>> -/*
>>>> - * Rep hypercalls. Callers of this functions are supposed to ensure that
>>>> - * rep_count, varhead_size, and rep_start comply with Hyper-V hypercall
>>>> - * definition.
>>>> - */
>>>> -static inline u64 hv_do_rep_hypercall_ex(u16 code, u16 rep_count,
>>>> - u16 varhead_size, u16 rep_start,
>>>> - void *input, void *output)
>>>> +static inline u64 hv_do_rep_hypercall_reg(u16 code, u16 rep_count,
>>>> + u16 varhead_size, u16 rep_start,
>>>> + void *input, void *output)
>>>> {
>>>> u64 control = code;
>>>> - u64 status;
>>>> - u16 rep_comp;
>>>>
>>>> control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>>>> control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>>>> control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>>>>
>>>> - do {
>>>
>>> Dropped the looping behavior here may be much more significant
>>> than just "streamlining" as described in the commit message. The looping
>>> behavior was included when rep hypercalls were originally added in
>>> commit 806c89273bab in kernel 4.14, long before any SNP/TDX/CCA
>>> functionality was added. So it was present for "normal" VMs doing
>>> hypercalls on x86/x64 the normal way. The looping ensured that
>>> rep hypercalls could not return with no errors but with the completed
>>> rep count less than the requested rep count.
>>>
>>> However, in my mind it *is* debatable whether that scenario could
>>> ever occur. The Hyper-V TLFS describes "Hypercall Continuation" [1]
>>> as essentially transparent to the caller. If a rep hypercall stops
>>> processing reps because of hitting the 50 microsecond time limit,
>>> it should restart and continue with the next set of reps without
>>> the caller having to loop. Commit 806c89273bab mentions
>>> hypercall continuation, but does not describe why the explicit
>>> looping was added or how that aligns with the TLFS statements.
>>> Vitaly -- it was a long time ago, but do you have any recollection?
>>>
>>> Have you gotten an explicit assurance from the Hyper-V team
>>> that the transparent hypercall continuation really does work,
>>> and the explicit looping has all along been unnecessary? This
>>
>> Correct, hence the patch. Waiting on confirmation that it is
>> same behavior all the way back to server 2016.
>
> Sounds good. FWIW, old TLFS version 2.0A, which is specifically
> titled to cover Windows Server 2008 R2, has the paragraph
> about hypercall continuation. So that functionality was evidently
> present very early on. It is still curious that Vitaly found it
> necessary to include the looping behavior in the Linux guest
> code.
>
>>
>> Basically, the hypervisor will just adjust the RIP back to the
>> vmcall upon internal timeouts, and reschedule the vcpu. But for
>> hw confidential vms, ie, tdx/snp/cca, it cannot adjust the RIP,
>> hence return back to the guest with TIMEOUT, so cvm has to do in
>> a loop.
>>
>> Now, for paravisor support, I was just told by a paravisor team
>> dev, that it will never return with TIMEOUTs. My view on that set
>> aside, I guess for paravisor we'll just call the _reg function
>> then. Thanks for bringing attention to that...
>
> Yes, that would be what I expected.
>
>>
>>> assurance would be needed back to and including Windows
>>> Server 2016, which is now the oldest Hyper-V version supported
>>> by Linux. Also note that the rep hypercall code is architecture
>>> independent, so the transparent hypercall continuation would
>>> also need to work on arm64 when the hypercall is invoked via
>>> the standard path of using the HVC instruction (the TLFS doesn't
>>> cover arm64, so this is unclear).
>>
>> Both x86 and arm are same for this issue.
>
> OK, good.
>
>>
>>> If you have such assurance, then I'm good with the removal of
>>> the looping. But please make the removal a separate patch, with
>>> sufficient discussion in the commit message to justify why it's OK
>>> to do the removal (including that evidently the original commit
>>> misinterpreted how continuation worked). And instead of looping,
>>> perhaps do a sanity check that the completed rep count matches
>>> the requested rep count.
>>
>> Not sure if it's worth the overhead of sanity check, looks like
>> callers are checking completed and loop anyways... But I would
>> be OK with BUG(), since it would be a bug for that to happen. FWIW,
>> it's been running internally with couple months, and haven't seen
>> any issues.
>>
>>
>>> Without such assurance, I think the looping needs to stay. Callers
>>> are not prepared for a rep hypercall to return with no errors, but
>>> with the completed rep count less than the requested rep count.
>>>
>>> [1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-
>> windows/tlfs/hypercall-interface#hypercall-continuation
>>>
>>>> + return hv_do_hypercall(control, input, output);
>>>> +}
>>>> +
>>>> +static inline u64 hv_do_rep_hypercall_cvm(u16 code, u16 rep_count,
>>>> + u16 varhead_size, u16 rep_start,
>>>> + void *input, void *output)
>>>> +{
>>>> + u64 status, control = code;
>>>> + u16 rep_comp = 0;
>>>> +
>>>> + control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>>>> + control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>>>> + control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>>>> +
>>>> + for (;;) {
>>>> status = hv_do_hypercall(control, input, output);
>>>> - if (!hv_result_success(status))
>>>> +
>>>> + if (!hv_result_timeout(status))
>>>> return status;
>>>>
>>>> rep_comp = hv_repcomp(status);
>>>> -
>>>> control &= ~HV_HYPERCALL_REP_START_MASK;
>>>> control |= (u64)rep_comp << HV_HYPERCALL_REP_START_OFFSET;
>>>>
>>>> touch_nmi_watchdog();
>>>> - } while (rep_comp < rep_count);
>>>> + }
>>>>
>>>> return status;
>>>
>>> And Sashiko noted that this "return status" is dead code. The only
>>> way to exit this function is via the "return status" up in the loop.
>>>
>>>> }
>>>>
>>>> -/* For the typical case where rep_start is 0 */
>>>> +/*
>>>> + * Rep hypercalls for HW confidential VMs return back when timeouts occur,
>>>> + * whereas for regular VMs they return only on success or failure. In case
>>>> + * of successful completion, entire rep_count is done. But a hypercall could
>>>> + * fail with partial completions, hence callers must always check that
>>>> + * in case of failure. Lastly, a rep_count value of 0 is allowed for some
>>>> + * hypercalls.
>>>> + */
>>>> static inline u64 hv_do_rep_hypercall(u16 code, u16 rep_count, u16 varhead_size,
>>>> void *input, void *output)
>>>> {
>>>> - return hv_do_rep_hypercall_ex(code, rep_count, varhead_size, 0,
>>>> - input, output);
>>>> + if (hv_isolation_type_hw())
>>>
>>> I don't think this is the right test, so let me explain. You don't describe the
>>> context for when HV_STATUS_TIMEOUT may be returned, and there's no
>>> TLFS documentation of this status, but I'm assuming the context is hypercall
>>> continuation. This isn't the typical "something isn't responding" timeout. I
>>> suspect it means the hypervisor hit the 50 us limit, and returned to the guest
>>> VM with some reps still to be done. In a normal VM when a hypercall is
>>> initiated by the hypercall page on x86/x64 or the HVC instruction on arm64,
>>> the transparent hypercall continuation mechanism makes a TIMEOUT status
>>> unnecessary. But with SNP, TDX, and CCA, hypercall invocation may be
>>> piggybacked on other transports, and transparent continuation isn't available.
>>> So Hyper-V returns HV_STATUS_TIMEOUT with the expectation that the
>>> invoking code will loop to do the remaining reps.
>>>
>>> The key factor is whether the hypercall is piggybacked on other transports.
>>> The tests in hv_isolation_type_hw() don't capture that. For example, if
>>> a paravisor is present, hv_isolation_type_snp() will return "true" even
>>> though hypercalls are invoked via the standard x86/x64 hypercall page.
>>> The hypercalls trap to the paravisor. If the paravisor is functioning properly,
>>> it should do transparent hypercall continuation just like the hypervisor would.
>>> But this is something to be confirmed with the hypervisor and paravisor teams.
>>>
>>> Piggybacking is done only when no paravisor is present, so the
>>> hv_do_rep_hypercall_cvm() path should be taken only in that circumstance.
>>> Your current code works, but it goes down the more expensive
>>> hv_do_rep_hypercall_cvm() path when it doesn't need to. (I'm not completely
>>> clear on how arm64 works with and without a paravisor, so my statement
>>> applies only to SNP and TDX. Need to sort out the CCA case with Kameron.)
>>>
>>> Given the non-trivial complexity in sorting all this out, I wonder if
>>> distinguishing the _cvm and _reg cases is really worth it. The current
>>> code for the _reg case is admittedly inefficient because it does work for
>>> the looping case before likely deciding that no looping is needed. But
>>> that could easily be fixed. When no looping is necessary, which is the
>>> vast majority of cases, the looping code could end up being only a
>>> couple of instructions slower than a separately coded non-looping
>>> version (I did some builds and looked at the generated code). Since
>>> there's overhead in deciding which to use, there may not be any
>>> performance win from having both versions.
>>
>> Well, to me it's correctness issue also. If the ABI says that it will
>> not return with TIMEOUTs for non-cvms, then adding loop just masks it
>> instead of opening a bug report against the hypervisor.
>
> I see your point, but would not call it a "correctness issue". A "correctness
> issue" is a case where the result is wrong, and there's no wrong result
> here. I've definitely fixed edge-case issues that might never occur in
> practice just in the interest of overall "correctness", but that's not this case.
>
> Your point opens a different can-of-worms. The threat model for Linux
> guests on Hyper-V has historically been that the guest trusts the Hyper-V
> host to do the right thing. A buggy or malicious host can arbitrarily read
> or write guest memory, so there's really not much choice but for the
> guest to trust the host (of course, in a CoCo VM the threat model is
> different). As such, Linux code has never tried to fully validate what it
> gets back from Hyper-V. Various ad hoc validations have been added
> when it seemed helpful to do so, but it's not a priority or a directive.
> Doing such a validation here is not objectionable, but I wouldn't add
> a significant complexity to do so.
>
> FWIW, here's pseudo-code for an alternate approach that doesn't
> require splitting into separate _reg and _cvm versions:
>
> static inline u64 hv_do_rep_hypercall_ex(u16 code, u16 rep_count,
> u16 varhead_size, u16 rep_start,
> void *input, void *output)
> {
> u64 status, control = code;
> u16 rep_comp = 0;
>
> control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
> control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
> control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>
> for (;;) {
> status = hv_do_hypercall(control, input, output);
>
> if (!hv_result_timeout(status))
> /* If result is success, could validate
> completed reps == rep_count - rep_start */
> return status;
>
> if (! <hypercall is over alt transport> ) {
> /* Should not get TIMEOUT with normal calling seq */
> WARN();
> return HV_STATUS_INVALID_PARAMETER; /* or some such */
> }
>
> rep_comp = hv_repcomp(status);
> control &= ~HV_HYPERCALL_REP_START_MASK;
> control |= (u64)rep_comp << HV_HYPERCALL_REP_START_OFFSET;
> }
> }
>
> In almost all cases, the result is not TIMEOUT, so the code returns
> immediately with no extra overhead except to check for TIMEOUT.
> If it *is* TIMEOUT, then go to the trouble to figure out whether
> TIMEOUT is valid. If the VM is using the normal hypercall invocation
> mechanism, flag that as a hypervisor bug and don't loop.
>
>>
>> Hope that makes sense.
>>
>> On a side note, to solicit your thoughts, i am not sure i like the
>> touch_nmi_watchdog()... it appears the the WD is triggered in seconds,
>> and if a hypercall is that long, schedule() really should be done in
>> between. IOW, remove the touch, instead any very long running hypercalls
>> should call schedule(). Of course, that would be a different patch.
>
> I agree. I would be fine with removing the touch_nmi_watchdog().
>
> Michael
prev parent reply other threads:[~2026-11-20 1:31 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 2:53 [PATCH V1 0/2] Hyper-V: " Mukesh R
2026-11-14 2:53 ` [PATCH V1 1/2] Drivers: hv: Add new function hv_isolation_type_hw() Mukesh R
2026-11-14 2:53 ` [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall Mukesh R
2026-11-15 23:48 ` Michael Kelley
2026-11-16 16:53 ` Vitaly Kuznetsov
2026-11-17 2:29 ` Mukesh R
2026-11-17 17:41 ` Michael Kelley
2026-11-17 23:09 ` Mukesh R
2026-11-18 14:01 ` Vitaly Kuznetsov
2026-11-18 16:57 ` Michael Kelley
2026-11-19 10:28 ` Vitaly Kuznetsov
2026-11-20 1:31 ` Mukesh R [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e6bffec5-064c-aa37-6e24-11d90124805d@linux.microsoft.com \
--to=mrathor@linux.microsoft.com \
--cc=decui@microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=kameroncarr@linux.microsoft.com \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mhklinux@outlook.com \
--cc=vkuznets@redhat.com \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®