From: Jarkko Sakkinen <jarkko@kernel.org>
To: Breno Leitao <leitao@debian.org>
Cc: Peter Huewe <peterhuewe@gmx.de>, Jason Gunthorpe <jgg@ziepe.ca>,
linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org,
surenkj@amazon.com, noodles@meta.com,
David Laight <david.laight.linux@gmail.com>,
Stefan Mahnke-Hartmann <stefan.mahnke-hartmann@infineon.com>,
kernel-team@meta.com
Subject: Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
Date: Fri, 20 Nov 2026 23:29:52 +0200 [thread overview]
Message-ID: <awC70MnZ4eXVQrdS@kernel.org> (raw)
In-Reply-To: <20261120-tpm_unresponsive-v4-0-b9043735ec1b@debian.org>
On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> On Meta's datacenter, given the size, TPMs that stop working write about
> 54 million lines a day to the kernel log, because nothing in the kernel
> gives up on them.
>
> Two cases make up nearly all of it.
>
> Some TPMs stop responding. Every command then fails only after the
> transport timeout, while the hwrng thread and userspace keep sending
> more.
>
> Other TPMs get reset under the running OS, and answer every command with
> TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> every 10 seconds:
>
> tpm tpm0: A TPM error (256) occurred attempting get random
>
> The goal is to retire TPMs that are not properly working, similarly to
> what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> retire the worker if a wedged call ever returns")
>
> Patch 1 disables the chip once two commands in a row have timed out,
> instead of keeping it in this desolate state. Patch 2, from Jarkko,
> disables it once it answers TPM_RC_INITIALIZE to anything but
> TPM2_Startup.
>
> Disabling the chip makes the failure explicit: every later command
> fails at once instead of timing out.
>
> A health-check daemon, such as machine checker, can then detect
> the broken hardware and send the machine to repair.
>
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
> Changes in v4:
> - Replace the count of timeouts and the tpm.max_missed_responses
> parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> tolerated, and a second one in a row disables the TPM (Jarkko)
> - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> claim that some of them recover
> - Fix sashiko findings:
> - A timeout while a signal is pending no longer counts, since some
> drivers return -ETIME when a signal interrupts the wait
> - tpm1_do_selftest() clears the flag on the timeouts it expects from
> TPM 1.2 chips that block during the self test
> - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@debian.org
>
> Changes in v3:
> - Rename tpm.max_unanswered to tpm.max_missed_responses and
> nr_unanswered to nr_missed_responses, and drop a redundant comment
> (Jarkko)
> - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> - Check the limit with a modulo (Jarkko), guarded so that 0 still
> means never
> - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org
>
> Changes in v2:
> - Disable the TPM after 60 commands in a row time out, instead of after
> ten minutes of timeouts, and make the limit the tpm.max_unanswered
> boot parameter (Jarkko)
> - Open-code the check in tpm_transmit() and drop the "timeout" field
> names (Jarkko)
> - Replace the hwrng failure counter with Jarkko's patch, which disables
> the TPM on TPM_RC_INITIALIZE
> - In that patch, move the check ahead of the RETRY/TESTING break, where
> it never ran, and skip TPM2_Startup, which a TPM that is already
> started fails with TPM_RC_INITIALIZE
> - Fix sashiko findings:
> - Two timeouts ten minutes apart no longer disable an idle TPM
> - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org
>
> ---
> Breno Leitao (1):
> tpm: Disable a TPM that stops responding
>
> Jarkko Sakkinen (1):
> tpm: Disable TPM on TPM_RC_INITIALIZE
>
> drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> drivers/char/tpm/tpm1-cmd.c | 1 +
> drivers/char/tpm/tpm2-cmd.c | 2 ++
> include/linux/tpm.h | 1 +
> 4 files changed, 24 insertions(+)
> ---
> base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> change-id: 20261110-tpm_unresponsive-a5f71c1db026
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Br, Jarkko
next prev parent reply other threads:[~2026-11-20 21:29 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 20:47 Breno Leitao
2026-11-20 20:47 ` [PATCH v4 1/2] tpm: Disable a TPM that stops responding Breno Leitao
2026-11-20 20:47 ` [PATCH v4 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE Breno Leitao
2026-11-20 21:29 ` Jarkko Sakkinen [this message]
2026-11-20 21:30 ` [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=awC70MnZ4eXVQrdS@kernel.org \
--to=jarkko@kernel.org \
--cc=david.laight.linux@gmail.com \
--cc=jgg@ziepe.ca \
--cc=kernel-team@meta.com \
--cc=leitao@debian.org \
--cc=linux-integrity@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=noodles@meta.com \
--cc=peterhuewe@gmx.de \
--cc=stefan.mahnke-hartmann@infineon.com \
--cc=surenkj@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®