From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D91A386C39 for ; Tue, 21 Apr 2026 09:19:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=205.220.168.131 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776763165; cv=none; b=If2eZIJ4jUSWzNI+kgS8Fw6JPWzg9Rnj7YZy2Vf251/JsGsEAN0Bgf11+PrXX1zJLYo9/VK0DNsZoY5pBtoZ69PLPMU4C4LQ7AB8f2VvSw1sj98asoPxXmyyqaHvrMW/c0svW7VZ2wYXaGn5aIYUyat2yOD2xSIabrYJUMFtwaU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776763165; c=relaxed/simple; bh=vA8dG9GlY96VNHOZhGCF9p2/cBCk0zZRcFSc89U9vNo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=C0JSaKJbRfLg7B6l4BgupAJ8KGdCZME5W5Rb7wgvVGV+tU6p3ao5l+grQ373XryVHN5E6cWzX45iQOrWtfAcCPsO4URxstks184F0l+k9c5oyH0rzb3q4PYJWFau/3J6sMczlmrUKDwhK0N/m6juCFYqEXXS+Kn/uoCPXmH/m3A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com; spf=pass smtp.mailfrom=oss.qualcomm.com; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b=J+zdq84W; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b=TpcDUlNG; arc=none smtp.client-ip=205.220.168.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oss.qualcomm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=qualcomm.com header.i=@qualcomm.com header.b="J+zdq84W"; dkim=pass (2048-bit key) header.d=oss.qualcomm.com header.i=@oss.qualcomm.com header.b="TpcDUlNG" Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 63L74LYD1538545 for ; Tue, 21 Apr 2026 09:19:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qualcomm.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= HcbE2WqfgUl698UND0roUyAgaaJ3j0TBz8C3IFVjikA=; b=J+zdq84WcK4QhWLj hjfKtDhLZ0v41QRPzMExRW01Y2zQfug8MnfnBTUXdkfvCSGn4mXoj92tPhkCZZ1j WRsGH3Vw3NECXgkyQfHMWyVOkzLfCmDbw+iwo0EglZTUn6iVsqNmwxZOElgUeiw/ hhIeQCAPwHhyU4h51KyIKq/agVW0Ed8NnDV49o8ISmAytSt0TExCF3HYcL5haiKX 0KHILJCrlZANdWwB2m3xOsrRkSudKXn/CJ/OjBUh86jLK+gbUwtvGq2TrMD+/vRO +GYzM0EOkc9XnJ3O0RnGKVKvDLt1ALPbxU2TOVk22K7EJAmeY5BRADVr96T6Ix2M CMpjdw== Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 4dp4haggu5-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Tue, 21 Apr 2026 09:19:23 +0000 (GMT) Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-35fc22424d9so8039098a91.2 for ; Tue, 21 Apr 2026 02:19:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oss.qualcomm.com; s=google; t=1776763163; x=1777367963; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=HcbE2WqfgUl698UND0roUyAgaaJ3j0TBz8C3IFVjikA=; b=TpcDUlNGIYL/YrTqSco6xRKT3FnBuykNrWrnbdcg6Q+rLUmftUfDWYu9ww2y0N8mdd NtX739MZAd9JUSAe8gKkI8GnYVY5L9gznFtETHjVTe/nQHiI2SJHOnuFSH+inb5Bulx0 0GWLj6lUFt7C4Pm/kC+6jta8sQ/tMhYOMtRG3JLKlxy3OoYviSjLeKAr1Efytr+dIS0b uB0vmR+k82B9ildHm448xA0QvU0B5ds9sCB1TcUg42nDB2viTYTofXcz/QbTisFZw3Ld sor3ham9fZ3+pqeqXHBAzrZcjKRyybW6Fa94UoLq0WkmZjIUGEClkhgnYj5mhNg1v6JU selw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776763163; x=1777367963; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HcbE2WqfgUl698UND0roUyAgaaJ3j0TBz8C3IFVjikA=; b=jKhpEFGMEWSDEL75/gJi/xJ5wWyWv8Tz2GwxgG2HRnPBMxKNJreViUK3oT/u+WYgqd 5P6llvXiZtqVJ9lpYWLoWAmgbnmzHGLlqOcg+uJiDHUNTaB0XZjjZQODy1C40muCLWQe idJ3lWeUrFxg2kTBnty3r/RzTaS+dVwiM1Gksax4ig3LrqOuDEqnGvD5tv8FLdRbE9nZ u3nTlLpgws6TAz9gEcAlO04kNT3ZutZ7SrSFS9swufILMK/IR3DrOw3HNq9Oj8o11eun 8LTeK6ftH/3BA3nqg9u+K8G+gg+Xsg1prjRAi/YK87Ga3QaeQZz5iAPmKALjE9nAQajg yc5w== X-Forwarded-Encrypted: i=1; AFNElJ9Skvv/8lWSW+r/bgnN9ISdH4GLVi0MBCG6wDUP3zPzCK1PyzRPO3S8tFU+If9tK85NDo7MYl/rwidluxg=@vger.kernel.org X-Gm-Message-State: AOJu0Yxim6NwBYNq4X1n6XjKrL6vvshT3Hc2RhE8t+bYMw7b/34d7se2 UrN8z+jgCYQibch/MKBfHJR1/ZSHXNWI+xr/1FthO3jhprZiVY4nB+Zva7uuP9nv5t1Rll3YUIt KKtHSMXBhqOdiT/gFp2dHMJdF9fiMsTMK2CYZy7hAnw0p4baTdjQPzUgY1MK+cklRUXf/Pi18vx XZsg== X-Gm-Gg: AeBDies6kky/mNrNtdWR/5uVsOmZ6nzR7RGiegGKU7mt1esOHFN+9o6539/qE3R+ZMw gnuA7XiaT5vPuZoA7+0RQFFNjb1nYE3aW7tppSSClM2iHrb2sDDRdhfUEpzQo7SLcv63hJPcTWB eH4ynBrreYKBDsMyj46WTMxqO3ISSCIBLPzzGA1LgTeFSDdFaja6C7vaNjXtBMAe2hgmxsYckDM LcAt9oRf9tTsLz/gBgVrdUgi7jh6SYZdRePX4208hC/qqs5UQQVv5YateOtATW/fUMS/MBmuz1I uINhRlcv4FpK8V32XrOh3cQrtctz4Ds15SK9E/dofTaq+BtnUZCDtol+MBX6HsW6kJWLKozi9WL 6lVHg9lWToO0z/nPgv5WUl+l14/ny29rJdVdlRHhdFbpKM6Et11IRaRAy X-Received: by 2002:a17:90b:5203:b0:35f:b9ea:8f9f with SMTP id 98e67ed59e1d1-36140464842mr15574564a91.16.1776763162659; Tue, 21 Apr 2026 02:19:22 -0700 (PDT) X-Received: by 2002:a17:90b:5203:b0:35f:b9ea:8f9f with SMTP id 98e67ed59e1d1-36140464842mr15574527a91.16.1776763162107; Tue, 21 Apr 2026 02:19:22 -0700 (PDT) Received: from [10.253.14.183] ([114.94.8.21]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-36140ff148dsm12809307a91.7.2026.04.21.02.19.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Apr 2026 02:19:21 -0700 (PDT) Message-ID: Date: Tue, 21 Apr 2026 17:18:59 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] virtio_console: add timeout to __send_to_port() spin loop Content-Language: en-GB To: Arnd Bergmann , Amit Shah , Greg Kroah-Hartman Cc: kernel@quicinc.com, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, kernel@oss.qualcomm.com References: <20260420-add_timeout_to___send_to_port-v1-1-6c32d33bf4f2@oss.qualcomm.com> <37b513d6-770b-4442-b3b3-93b2801f3c71@app.fastmail.com> From: Peng Yang In-Reply-To: <37b513d6-770b-4442-b3b3-93b2801f3c71@app.fastmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Authority-Analysis: v=2.4 cv=bOQm5v+Z c=1 sm=1 tr=0 ts=69e7411b cx=c_pps a=0uOsjrqzRL749jD1oC5vDA==:117 a=Uz3yg00KUFJ2y2WijEJ4bw==:17 a=IkcTkHD0fZMA:10 a=A5OVakUREuEA:10 a=s4-Qcg_JpJYA:10 a=VkNPw1HP01LnGYTKEx00:22 a=u7WPNUs3qKkmUXheDGA7:22 a=Um2Pa8k9VHT-vaBCBUpS:22 a=TpNw2uGOYBV3wa7vfS4A:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=mQ_c8vxmzFEMiUWkPHU9:22 X-Proofpoint-GUID: QYYjc7PntCI_fTYNDhYr9qH80VIL86CD X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNDIxMDA5MCBTYWx0ZWRfX25fPY+8901lw 3j8FbpHnh5VvHPQ1WBl6Q7DQq1cB8NdrInsHvm2PY7j1Z7Vfm52bnDMAbDuenSvk9fi+kAhdu1J s5OTwxWx2ujw2k8ugjVXva45mFnkxYgHAcqOvg5AL5so41orzdqe7QVtP2XtLTwH9tTqaA8j1z1 RpIaeytjeanBwG4GtkTyYWNrlvt+Iu0tedBgVwWL5VFHQ0Mk38YtZ6x6ne+OSUV3n6v4C7FKjzc VmcQUktnXwlu/HVA+huPgkR7wSYUSxmC6TrwJXlRWNKlCLIU3aNRI2ibUMSBH7NDYiorVzh3lo3 Xxkw9LL05E9pY9B3UBIEah294iYt2zpvkUepxjDRD3wR0qH2EK89Q3ouduWEu7/63AfXA0RUJ+J xd7+86YDlf/A7GqScGiFpbNijeQ5pS8u5oYSKw64gx+cwLEJwRc+AKzH9HeDPl98+A46LDKpOSI zlGpipx6KWSUWJ+xjWg== X-Proofpoint-ORIG-GUID: QYYjc7PntCI_fTYNDhYr9qH80VIL86CD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-04-21_01,2026-04-20_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 malwarescore=0 bulkscore=0 priorityscore=1501 clxscore=1015 impostorscore=0 lowpriorityscore=0 adultscore=0 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2604070000 definitions=main-2604210090 On 4/21/2026 3:38 PM, Arnd Bergmann wrote: > On Tue, Apr 21, 2026, at 08:57, Peng Yang wrote: >> __send_to_port() busy-waits in virtqueue_get_buf() while holding >> outvq_lock with IRQs disabled. If the host stops draining the TX >> virtqueue, this loop never terminates. >> >> This was observed during secondary VM boot: virtio_mem plugged memory >> in multiple iterations, each emitting dev_info() messages through the >> hvc console. A writev() on the hvc TTY entered __send_to_port() and >> stalled in the spin loop. When the watchdog bark ISR fired on another >> CPU, it attempted printk(), which tried to acquire outvq_lock through >> the same path and spun indefinitely. With all CPUs stuck, the watchdog >> could not be serviced and triggered a bite. >> >> Add a 200 ms deadline using ktime_get_mono_fast_ns() to bound the spin >> loop. ktime_get_mono_fast_ns() reads the hardware counter directly and >> is safe to call with IRQs disabled and spinlocks held. >> >> The 200 ms value is chosen to be far above normal host response latency >> (microseconds) to avoid spurious exits, yet well below the watchdog >> bark-to-bite window (typically 3 s) so that CPUs can escape the loop >> and complete the bark handler before a bite occurs. > > Which host implementation do you use? The way the virtio_console > driver works really assumes that virtqueue_kick() consumes the > buffer synchronously. Even though that is not how virtio is > specified, this does tend to work. ;-) > We are using crosvm as the host VMM with its virtio-console backend, running on Android. The trigger is Android host reboot/shutdown: when Android initiates a reboot, the crosvm process exits and tears down the virtio-console backend. At that point, the TX virtqueue is no longer being drained by the host and will never be consumed again. The crash dump from the actual failure confirms the exact deadlock scenario: Core 3 holds outvq_lock and spins forever in virtqueue_get_buf waiting for the host to consume the buffer: virtqueue_get_buf __send_to_port put_chars hvc_push hvc_write n_tty_write <- writev() syscall Core 0 has a watchdog bark ISR fire and attempts printk, holds the console lock, but spins on _raw_spin_lock_irqsave waiting to acquire outvq_lock: queued_spin_lock_slowpath _raw_spin_lock_irqsave __send_to_port put_chars hvc_console_print console_flush_all console_unlock vprintk_emit <- printk (watchdog bark handler) Core 1 has a virtio_mem worker calling _dev_info, spinning inside vprintk_emit waiting to acquire the console lock which is held by Core 0: vprintk_emit <- spinning for console lock dev_vprintk_emit dev_printk_emit __dev_printk _dev_info <- virtio_mem worker Core 2 (khvcd kernel thread) is also blocked in __hvc_poll trying to acquire outvq_lock: queued_spin_lock_slowpath _raw_spin_lock_irqsave __hvc_poll khvcd With all CPUs stuck, the watchdog cannot be serviced and a bite occurs before the graceful shutdown can complete. The 200 ms timeout is intended as a bounded escape from this "backend already gone" scenario. It is well above normal crosvm response latency (microseconds) to avoid false exits under normal operation, and well below the watchdog bark-to-bite window so that CPUs can escape the loop and allow the graceful shutdown to proceed. >> @@ -632,10 +634,18 @@ static ssize_t __send_to_port(struct port *port, struct scatterlist *sg, >> * buffer and relax the spinning requirement. The downside is >> * we need to kmalloc a GFP_ATOMIC buffer each time the >> * console driver writes something out. >> + * >> + * To avoid spinning forever if the host stops processing the >> + * TX virtqueue (e.g. during VM shutdown), a 200ms deadline is >> + * used to break out of the loop as a fallback. > */ > > Did you by any chance mean to use microseconds instead of milliseconds? > Waiting this long with interrupts disabled likely breaks a lot > of assumptions, e.g. in the scheduler. If you have to deal with > a hypervisor that does not handle the console output synchronously, > the alternative suggested in the existing comment would likely > be more appropriate. > > Arnd The unit is intentionally milliseconds, not microseconds. The timeout only triggers in the abnormal case where crosvm has already exited — under normal operation crosvm drains the TX virtqueue in microseconds and the loop exits immediately. The 200 ms value is chosen to be well below the watchdog bark-to-bite window (typically a few seconds) so that CPUs can escape the spin loop and allow the watchdog bark handler to complete before a bite occurs. You are right that holding IRQs disabled for up to 200 ms is far from ideal and breaks scheduler assumptions. We considered the alternative suggested in the existing comment — copying data to a GFP_ATOMIC buffer to avoid the spin entirely — but that approach has a fundamental limitation in our case: the spin loop is not just in __send_to_port called from the hvc TTY path, but also in send_control_msg, and more critically, the deadlock occurs because printk itself ends up calling put_chars → __send_to_port while holding the console lock with IRQs disabled. Refactoring to eliminate the spin loop entirely would require a more invasive rework of the console write path, which is beyond the scope of this fix. The 200 ms timeout is intended as a minimal, targeted workaround to prevent the watchdog bite in our specific scenario. We are open to suggestions on a better long-term approach.