[PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Sahil Chandna]

A guest can wait indefinitely in wait_for_response() for the host to
send either a rescind message or a packet completion. If the
host does not send either, the guest can remain blocked with no
diagnostic indicating a reason.
This was observed during a guest kernel upgrade in which the
host-side application handling the PCI channel faulted, causing the
guest to never receive the completion request.
Add a periodic warning in wait_for_response() when the wait exceeds
a timeout so that such a hang is visible in the guest's kernel log
and can be correlated with host-side state.

Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
---
This was sent earlier upstream [1]
[1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
---
 drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index cfc8fa403dad..c4fba0039164 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)

 /*
  * There is no good way to get notified from vmbus_onoffer_rescind(),
- * so let's use polling here, since this is not a hot path.
+ * so let's use polling here, since this is not a hot path. If
+ * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
+ * without either a rescind or completion, add a periodic warning.
  */
+#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
+
 static int wait_for_response(struct hv_device *hdev,
 			     struct completion *comp)
 {
+	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
+	u64 timeout = get_jiffies_64() + delay;
+
 	while (true) {
 		if (hdev->channel->rescind) {
 			dev_warn_once(&hdev->device, "The device is gone.\n");
@@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,

 		if (wait_for_completion_timeout(comp, HZ / 10))
 			break;
+
+		if (time_after64(get_jiffies_64(), timeout)) {
+			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
+			timeout = get_jiffies_64() + delay;
+		}
 	}

 	return 0;
--
2.53.0

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Naman Jain]

On 8/25/2026 10:47 AM, Sahil Chandna wrote:
> A guest can wait indefinitely in wait_for_response() for the host to
> send either a rescind message or a packet completion. If the
> host does not send either, the guest can remain blocked with no
> diagnostic indicating a reason.
> This was observed during a guest kernel upgrade in which the
> host-side application handling the PCI channel faulted, causing the
> guest to never receive the completion request.
> Add a periodic warning in wait_for_response() when the wait exceeds
> a timeout so that such a hang is visible in the guest's kernel log
> and can be correlated with host-side state.
> 
> Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
> ---
> This was sent earlier upstream [1]
> [1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
> ---
>   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
>   1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
> index cfc8fa403dad..c4fba0039164 100644
> --- a/drivers/pci/controller/pci-hyperv.c
> +++ b/drivers/pci/controller/pci-hyperv.c
> @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
> 
>   /*
>    * There is no good way to get notified from vmbus_onoffer_rescind(),
> - * so let's use polling here, since this is not a hot path.
> + * so let's use polling here, since this is not a hot path. If
> + * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
> + * without either a rescind or completion, add a periodic warning.
>    */
> +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
> +
>   static int wait_for_response(struct hv_device *hdev,
>   			     struct completion *comp)
>   {
> +	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
> +	u64 timeout = get_jiffies_64() + delay;
> +
>   	while (true) {
>   		if (hdev->channel->rescind) {
>   			dev_warn_once(&hdev->device, "The device is gone.\n");
> @@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
> 
>   		if (wait_for_completion_timeout(comp, HZ / 10))
>   			break;
> +
> +		if (time_after64(get_jiffies_64(), timeout)) {
> +			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
> +			timeout = get_jiffies_64() + delay;
> +		}
>   	}

There can be some enhancements in above patch to address these problems:
1. Logging forever every 5 minutes in case of no completion or rescind.
2. If we now print warning once, not knowing if completion ever arrived.
3. On solving pt. 1 and 2 by adding a print for completion, one should 
avoid adding a print by default for regular timely completions.

Basically something like this:

#define PCI_RESPONSE_WARN_TIMEOUT_SEC 300

static int wait_for_response(struct hv_device *hdev,
                  struct completion *comp)
{
     unsigned long warn_at =
         jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
     bool warned = false;

     while (true) {
         if (hdev->channel->rescind) {
             dev_warn_once(&hdev->device, "The device is gone.\n");
             return -ENODEV;
         }

         if (wait_for_completion_timeout(comp, HZ / 10)) {
             if (warned || time_after_eq(jiffies, warn_at))
                 dev_warn(&hdev->device,
                      "PCI response received after prolonged wait.\n");
             return 0;
         }

         if (!warned && time_after_eq(jiffies, warn_at)) {
             dev_warn(&hdev->device,
                  "PCI still waiting for response.\n");
             warned = true;
         }
     }
}

Regards,
Naman

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Sahil Chandna]

On 25-08-2026 12:16, Naman Jain wrote:
> 
> 
> On 8/25/2026 10:47 AM, Sahil Chandna wrote:
>> A guest can wait indefinitely in wait_for_response() for the host to
>> send either a rescind message or a packet completion. If the
>> host does not send either, the guest can remain blocked with no
>> diagnostic indicating a reason.
>> This was observed during a guest kernel upgrade in which the
>> host-side application handling the PCI channel faulted, causing the
>> guest to never receive the completion request.
>> Add a periodic warning in wait_for_response() when the wait exceeds
>> a timeout so that such a hang is visible in the guest's kernel log
>> and can be correlated with host-side state.
>>
>> Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
>> Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
>> ---
>> This was sent earlier upstream [1]
>> [1] https://lore.kernel.org/linux-hyperv/20260612174010.2598695-1-hamzamahfooz@linux.microsoft.com/
>> ---
>>   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
>>   1 file changed, 13 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
>> index cfc8fa403dad..c4fba0039164 100644
>> --- a/drivers/pci/controller/pci-hyperv.c
>> +++ b/drivers/pci/controller/pci-hyperv.c
>> @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev *hpdev)
>>
>>   /*
>>    * There is no good way to get notified from vmbus_onoffer_rescind(),
>> - * so let's use polling here, since this is not a hot path.
>> + * so let's use polling here, since this is not a hot path. If
>> + * wait_for_response() has been polling for PCI_RESPONSE_HANG_TIMEOUT_SEC
>> + * without either a rescind or completion, add a periodic warning.
>>    */
>> +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
>> +
>>   static int wait_for_response(struct hv_device *hdev,
>>   			     struct completion *comp)
>>   {
>> +	unsigned long delay = secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
>> +	u64 timeout = get_jiffies_64() + delay;
>> +
>>   	while (true) {
>>   		if (hdev->channel->rescind) {
>>   			dev_warn_once(&hdev->device, "The device is gone.\n");
>> @@ -1053,6 +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
>>
>>   		if (wait_for_completion_timeout(comp, HZ / 10))
>>   			break;
>> +
>> +		if (time_after64(get_jiffies_64(), timeout)) {
>> +			dev_warn(&hdev->device, "PCI stuck waiting for response.\n");
>> +			timeout = get_jiffies_64() + delay;
>> +		}
>>   	}
> 
> There can be some enhancements in above patch to address these problems:
> 1. Logging forever every 5 minutes in case of no completion or rescind.
> 2. If we now print warning once, not knowing if completion ever arrived.
> 3. On solving pt. 1 and 2 by adding a print for completion, one should 
> avoid adding a print by default for regular timely completions.
> 
Ack.
> Basically something like this:
> 
> #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> 
> static int wait_for_response(struct hv_device *hdev,
>                   struct completion *comp)
> {
>      unsigned long warn_at =
>          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
>      bool warned = false;
> 
>      while (true) {
>          if (hdev->channel->rescind) {
>              dev_warn_once(&hdev->device, "The device is gone.\n");
>              return -ENODEV;
>          }
> 
>          if (wait_for_completion_timeout(comp, HZ / 10)) {
>              if (warned || time_after_eq(jiffies, warn_at))
>                  dev_warn(&hdev->device,
>                       "PCI response received after prolonged wait.\n");
>              return 0;
>          }
> 
>          if (!warned && time_after_eq(jiffies, warn_at)) {
>              dev_warn(&hdev->device,
>                   "PCI still waiting for response.\n");
>              warned = true;
>          }
>      }
> }
> 
Thanks for review, I align on repeated warning every 5 minutes for
stalled guest would add to dmesg noise.I will wait for other review
comments as well and share v2 addressing this.
> Regards,
> Naman

RE: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Long Li]

> 
> 
> On 8/25/2026 10:47 AM, Sahil Chandna wrote:
> > A guest can wait indefinitely in wait_for_response() for the host to
> > send either a rescind message or a packet completion. If the host does
> > not send either, the guest can remain blocked with no diagnostic
> > indicating a reason.
> > This was observed during a guest kernel upgrade in which the host-side
> > application handling the PCI channel faulted, causing the guest to
> > never receive the completion request.
> > Add a periodic warning in wait_for_response() when the wait exceeds a
> > timeout so that such a hang is visible in the guest's kernel log and
> > can be correlated with host-side state.
> >
> > Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
> > Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
> > ---
> > This was sent earlier upstream [1]
> > [1]
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
> > .kernel.org%2Flinux-hyperv%2F20260612174010.2598695-1-
> hamzamahfooz%40l
> >
> inux.microsoft.com%2F&data=05%7C02%7Clongli%40microsoft.com%7C22b60
> f06
> >
> 81624fda801f08df02748dce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7
> C0%7C
> >
> 639232371774209409%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOn
> RydWUsIlY
> >
> iOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7
> C0%
> >
> 7C%7C%7C&sdata=1uUKHOonR9CtiJfdofsRbWLbuKYHlWgBXdwByQWdk2A%3
> D&reserved
> > =0
> > ---
> >   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
> >   1 file changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/controller/pci-hyperv.c
> > b/drivers/pci/controller/pci-hyperv.c
> > index cfc8fa403dad..c4fba0039164 100644
> > --- a/drivers/pci/controller/pci-hyperv.c
> > +++ b/drivers/pci/controller/pci-hyperv.c
> > @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev
> > *hpdev)
> >
> >   /*
> >    * There is no good way to get notified from
> > vmbus_onoffer_rescind(),
> > - * so let's use polling here, since this is not a hot path.
> > + * so let's use polling here, since this is not a hot path. If
> > + * wait_for_response() has been polling for
> > + PCI_RESPONSE_HANG_TIMEOUT_SEC
> > + * without either a rescind or completion, add a periodic warning.
> >    */
> > +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
> > +
> >   static int wait_for_response(struct hv_device *hdev,
> >   			     struct completion *comp)
> >   {
> > +	unsigned long delay =
> secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
> > +	u64 timeout = get_jiffies_64() + delay;
> > +
> >   	while (true) {
> >   		if (hdev->channel->rescind) {
> >   			dev_warn_once(&hdev->device, "The device is
> gone.\n"); @@ -1053,6
> > +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
> >
> >   		if (wait_for_completion_timeout(comp, HZ / 10))
> >   			break;
> > +
> > +		if (time_after64(get_jiffies_64(), timeout)) {
> > +			dev_warn(&hdev->device, "PCI stuck waiting for
> response.\n");
> > +			timeout = get_jiffies_64() + delay;
> > +		}
> >   	}
> 
> There can be some enhancements in above patch to address these problems:
> 1. Logging forever every 5 minutes in case of no completion or rescind.
> 2. If we now print warning once, not knowing if completion ever arrived.
> 3. On solving pt. 1 and 2 by adding a print for completion, one should avoid
> adding a print by default for regular timely completions.
> 
> Basically something like this:
> 
> #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> 
> static int wait_for_response(struct hv_device *hdev,
>                   struct completion *comp) {
>      unsigned long warn_at =
>          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
>      bool warned = false;
> 
>      while (true) {
>          if (hdev->channel->rescind) {
>              dev_warn_once(&hdev->device, "The device is gone.\n");
>              return -ENODEV;
>          }
> 
>          if (wait_for_completion_timeout(comp, HZ / 10)) {
>              if (warned || time_after_eq(jiffies, warn_at))
>                  dev_warn(&hdev->device,
>                       "PCI response received after prolonged wait.\n");
>              return 0;
>          }
> 
>          if (!warned && time_after_eq(jiffies, warn_at)) {
>              dev_warn(&hdev->device,
>                   "PCI still waiting for response.\n");
>              warned = true;
>          }
>      }
> }
> 
> Regards,
> Naman

This looks better.

Thanks,
Long

RE: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Michael Kelley]

From: Long Li <longli@microsoft.com> Sent: Tuesday, August 25, 2026 10:01 AM

[snip]

> >
> > Basically something like this:
> >
> > #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> >
> > static int wait_for_response(struct hv_device *hdev,
> >                   struct completion *comp) {
> >      unsigned long warn_at =
> >          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
> >      bool warned = false;
> >
> >      while (true) {
> >          if (hdev->channel->rescind) {
> >              dev_warn_once(&hdev->device, "The device is gone.\n");
> >              return -ENODEV;
> >          }
> >
> >          if (wait_for_completion_timeout(comp, HZ / 10)) {
> >              if (warned || time_after_eq(jiffies, warn_at))
> >                  dev_warn(&hdev->device,
> >                       "PCI response received after prolonged wait.\n");
> >              return 0;
> >          }
> >
> >          if (!warned && time_after_eq(jiffies, warn_at)) {
> >              dev_warn(&hdev->device,
> >                   "PCI still waiting for response.\n");
> >              warned = true;
> >          }
> >      }
> > }
> >
> > Regards,
> > Naman
> 
> This looks better.
> 

I like getting the "response received" message if the response
eventually does come in. It's a judgment call, but I would be OK
with outputting the "still waiting" message after each wait interval
rather than doing it only once. And I would make the interval smaller
than 300 seconds. Five minutes is a long time to wait and wonder
what's going on when things are hung. 60 or 120 seconds would
be OK -- a line in dmesg every 1 or 2 minutes doesn't seem like
spamming to me when something is fundamentally broken.
And probably don't expect a VM in this broken state to keep running
for hours -- the sysadmin or automatic monitoring software will
reboot it to get it working again.

Just my $.02. Outputting the "still waiting" message only once is
also OK. Your call.

Michael

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Saurabh Singh Sengar]

On Thu, Aug 27, 2026 at 04:07:06PM +0000, Michael Kelley wrote:
> From: Long Li <longli@microsoft.com> Sent: Tuesday, August 25, 2026 10:01 AM
> 
> [snip]
> 
> > >
> > > Basically something like this:
> > >
> > > #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> > >
> > > static int wait_for_response(struct hv_device *hdev,
> > >                   struct completion *comp) {
> > >      unsigned long warn_at =
> > >          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
> > >      bool warned = false;
> > >
> > >      while (true) {
> > >          if (hdev->channel->rescind) {
> > >              dev_warn_once(&hdev->device, "The device is gone.\n");
> > >              return -ENODEV;
> > >          }
> > >
> > >          if (wait_for_completion_timeout(comp, HZ / 10)) {
> > >              if (warned || time_after_eq(jiffies, warn_at))
> > >                  dev_warn(&hdev->device,
> > >                       "PCI response received after prolonged wait.\n");
> > >              return 0;
> > >          }
> > >
> > >          if (!warned && time_after_eq(jiffies, warn_at)) {
> > >              dev_warn(&hdev->device,
> > >                   "PCI still waiting for response.\n");
> > >              warned = true;
> > >          }
> > >      }
> > > }
> > >
> > > Regards,
> > > Naman
> > 
> > This looks better.
> > 
> 
> I like getting the "response received" message if the response
> eventually does come in. It's a judgment call, but I would be OK
> with outputting the "still waiting" message after each wait interval
> rather than doing it only once. And I would make the interval smaller
> than 300 seconds. Five minutes is a long time to wait and wonder
> what's going on when things are hung. 60 or 120 seconds would
> be OK -- a line in dmesg every 1 or 2 minutes doesn't seem like
> spamming to me when something is fundamentally broken.
> And probably don't expect a VM in this broken state to keep running
> for hours -- the sysadmin or automatic monitoring software will
> reboot it to get it working again.

I will also vote for repeated message as long as interval between two
messages is greater than 60 seconds.

- Saurabh

> 
> Just my $.02. Outputting the "still waiting" message only once is
> also OK. Your call.
> 
> Michael

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Naman Jain]

On 8/27/2026 11:04 PM, Saurabh Singh Sengar wrote:
> On Thu, Aug 27, 2026 at 04:07:06PM +0000, Michael Kelley wrote:
>> From: Long Li <longli@microsoft.com> Sent: Tuesday, August 25, 2026 10:01 AM
>>
>> [snip]
>>
>>>>
>>>> Basically something like this:
>>>>
>>>> #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
>>>>
>>>> static int wait_for_response(struct hv_device *hdev,
>>>>                    struct completion *comp) {
>>>>       unsigned long warn_at =
>>>>           jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
>>>>       bool warned = false;
>>>>
>>>>       while (true) {
>>>>           if (hdev->channel->rescind) {
>>>>               dev_warn_once(&hdev->device, "The device is gone.\n");
>>>>               return -ENODEV;
>>>>           }
>>>>
>>>>           if (wait_for_completion_timeout(comp, HZ / 10)) {
>>>>               if (warned || time_after_eq(jiffies, warn_at))
>>>>                   dev_warn(&hdev->device,
>>>>                        "PCI response received after prolonged wait.\n");
>>>>               return 0;
>>>>           }
>>>>
>>>>           if (!warned && time_after_eq(jiffies, warn_at)) {
>>>>               dev_warn(&hdev->device,
>>>>                    "PCI still waiting for response.\n");
>>>>               warned = true;
>>>>           }
>>>>       }
>>>> }
>>>>
>>>> Regards,
>>>> Naman
>>>
>>> This looks better.
>>>
>>
>> I like getting the "response received" message if the response
>> eventually does come in. It's a judgment call, but I would be OK
>> with outputting the "still waiting" message after each wait interval
>> rather than doing it only once. And I would make the interval smaller
>> than 300 seconds. Five minutes is a long time to wait and wonder
>> what's going on when things are hung. 60 or 120 seconds would
>> be OK -- a line in dmesg every 1 or 2 minutes doesn't seem like
>> spamming to me when something is fundamentally broken.
>> And probably don't expect a VM in this broken state to keep running
>> for hours -- the sysadmin or automatic monitoring software will
>> reboot it to get it working again.

I think this is a main point of this discussion. FWIK, this is not 
automatically recovered as of now, ever. With these changes, we could 
get dmesg logs completely filled with this same log, which does not add 
any value. On the other hand, I was suggesting Sahil if this problem is 
not recoverable by Azure fabric layer or other monitoring services, and 
is extremely rare and the VM is unusable, perhaps we should consider 
adding a bug/timeout in this path instead of just logging about it.

Regards,
Naman

> 
> I will also vote for repeated message as long as interval between two
> messages is greater than 60 seconds.
> 
> - Saurabh
> 
>>
>> Just my $.02. Outputting the "still waiting" message only once is
>> also OK. Your call.
>>
>> Michael

Re: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely

[Hamza Mahfooz]

On Fri, Aug 28, 2026 at 11:10:22AM +0530, Naman Jain wrote:
> 
> 
> On 8/27/2026 11:04 PM, Saurabh Singh Sengar wrote:
> > On Thu, Aug 27, 2026 at 04:07:06PM +0000, Michael Kelley wrote:
> > > From: Long Li <longli@microsoft.com> Sent: Tuesday, August 25, 2026 10:01 AM
> > > 
> > > [snip]
> > > 
> > > > > 
> > > > > Basically something like this:
> > > > > 
> > > > > #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> > > > > 
> > > > > static int wait_for_response(struct hv_device *hdev,
> > > > >                    struct completion *comp) {
> > > > >       unsigned long warn_at =
> > > > >           jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
> > > > >       bool warned = false;
> > > > > 
> > > > >       while (true) {
> > > > >           if (hdev->channel->rescind) {
> > > > >               dev_warn_once(&hdev->device, "The device is gone.\n");
> > > > >               return -ENODEV;
> > > > >           }
> > > > > 
> > > > >           if (wait_for_completion_timeout(comp, HZ / 10)) {
> > > > >               if (warned || time_after_eq(jiffies, warn_at))
> > > > >                   dev_warn(&hdev->device,
> > > > >                        "PCI response received after prolonged wait.\n");
> > > > >               return 0;
> > > > >           }
> > > > > 
> > > > >           if (!warned && time_after_eq(jiffies, warn_at)) {
> > > > >               dev_warn(&hdev->device,
> > > > >                    "PCI still waiting for response.\n");
> > > > >               warned = true;
> > > > >           }
> > > > >       }
> > > > > }
> > > > > 
> > > > > Regards,
> > > > > Naman
> > > > 
> > > > This looks better.
> > > > 
> > > 
> > > I like getting the "response received" message if the response
> > > eventually does come in. It's a judgment call, but I would be OK
> > > with outputting the "still waiting" message after each wait interval
> > > rather than doing it only once. And I would make the interval smaller
> > > than 300 seconds. Five minutes is a long time to wait and wonder
> > > what's going on when things are hung. 60 or 120 seconds would
> > > be OK -- a line in dmesg every 1 or 2 minutes doesn't seem like
> > > spamming to me when something is fundamentally broken.
> > > And probably don't expect a VM in this broken state to keep running
> > > for hours -- the sysadmin or automatic monitoring software will
> > > reboot it to get it working again.
> 
> I think this is a main point of this discussion. FWIK, this is not
> automatically recovered as of now, ever. With these changes, we could get
> dmesg logs completely filled with this same log, which does not add any
> value. On the other hand, I was suggesting Sahil if this problem is not
> recoverable by Azure fabric layer or other monitoring services, and is
> extremely rare and the VM is unusable, perhaps we should consider adding a
> bug/timeout in this path instead of just logging about it.
> 
> Regards,
> Naman

Along with what everyone else has suggested I would suggest printing out the
state of the device's VMBus channel (i.e. the child_relid and inbound and
outbound status in particular). Since, that information is useful to the hv
guys and should help disambiguate between "host never picked it up" vs. 
"host picked it up but never replied."

Hamza

> 
> > 
> > I will also vote for repeated message as long as interval between two
> > messages is greater than 60 seconds.
> > 
> > - Saurabh
> > 
> > > 
> > > Just my $.02. Outputting the "still waiting" message only once is
> > > also OK. Your call.
> > > 
> > > Michael
> 
>