[PATCH v3 0/2] tpm: Stop retrying TPMs that have stopped working

[PATCH v3 0/2] tpm: Stop retrying TPMs that have stopped working

[Breno Leitao]

On Meta's datacenter, given the size, TPMs that stop working write about
54 million lines a day to the kernel log, because nothing in the kernel
gives up on them.

Two cases make up nearly all of it.

Some TPMs stop responding. Every command then fails only after the
transport timeout, while the hwrng thread and userspace keep sending
more.

Other TPMs get reset under the running OS, and answer every command with
TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
every 10 seconds:

  tpm tpm0: A TPM error (256) occurred attempting get random

The goal is to retire TPMs that are not properly working, similarly to
what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
retire the worker if a wedged call ever returns")

Patch 1 disables the chip once 60 commands in a row have timed out,
instead of keeping it in this desolate state. Patch 2, from Jarkko,
disables it once it answers TPM_RC_INITIALIZE to anything but
TPM2_Startup.

Disabling the chip makes the failure explicit: the log gets one last
line about it, and every later command fails at once instead of timing
out.

A health-check daemon, such as machine checker, can then detect
the broken hardware and send the machine to repair.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v3:
- Rename tpm.max_unanswered to tpm.max_missed_responses and
  nr_unanswered to nr_missed_responses, and drop a redundant comment
  (Jarkko)
- Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
- Check the limit with a modulo (Jarkko), guarded so that 0 still
  means never
- Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@debian.org

Changes in v2:
- Disable the TPM after 60 commands in a row time out, instead of after
  ten minutes of timeouts, and make the limit the tpm.max_unanswered
  boot parameter (Jarkko)
- Open-code the check in tpm_transmit() and drop the "timeout" field
  names (Jarkko)
- Replace the hwrng failure counter with Jarkko's patch, which disables
  the TPM on TPM_RC_INITIALIZE
- In that patch, move the check ahead of the RETRY/TESTING break, where
  it never ran, and skip TPM2_Startup, which a TPM that is already
  started fails with TPM_RC_INITIALIZE
- Fix sashiko findings:
  - Two timeouts ten minutes apart no longer disable an idle TPM
- Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@debian.org

---
Breno Leitao (1):
      tpm: Disable a TPM that stops responding

Jarkko Sakkinen (1):
      tpm: Disable TPM on TPM_RC_INITIALIZE

 Documentation/admin-guide/kernel-parameters.txt |  8 ++++++++
 drivers/char/tpm/tpm-interface.c                | 19 +++++++++++++++++++
 drivers/char/tpm/tpm2-cmd.c                     |  2 ++
 include/linux/tpm.h                             |  1 +
 4 files changed, 30 insertions(+)
---
base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
change-id: 20261110-tpm_unresponsive-a5f71c1db026

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH v3 1/2] tpm: Disable a TPM that stops responding

[Breno Leitao]

A TPM that stops responding fails each command only after the transport
timeout, and its callers keep sending more: the hwrng thread every 10
seconds forever, and userspace through /dev/tpmrm0. Each attempt holds
tpm_mutex for the whole timeout and logs an error:

  tpm tpm0: invalid TPM_STS.x 0xff, dumping stack for forensics
  tpm tpm0: tpm_try_transmit: send(): error -62
  tpm tpm0: tpm_try_transmit: send(): error -62
  ...

The core already stops using a TPM it cannot trust: tpm_try_get_ops()
fails with -EIO once TPM_CHIP_FLAG_DISABLE is set, which
tpm2_load_null() does when it cannot restore the null key. Set the same
flag once 60 commands in a row have timed out.

A completed command, whatever its TPM return code, resets the count.
Some TPMs stall for half a minute, time out a handful of commands, and
then recover, so a short streak must not disable them.

With only the hwrng retrying every 10 seconds, 60 timeouts take more
than ten minutes. A userspace poller gets there sooner, but each of its
commands still waits out the transport timeout.

The limit is the tpm.max_missed_responses boot parameter, so that slow
pre-production parts, such as FPGA prototypes, can raise it, or set it
to 0 to never disable the TPM. A disabled TPM stays disabled until the
driver is rebound or the machine reboots.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 Documentation/admin-guide/kernel-parameters.txt |  8 ++++++++
 drivers/char/tpm/tpm-interface.c                | 14 ++++++++++++++
 include/linux/tpm.h                             |  1 +
 3 files changed, 23 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1e8317483844d9..cedea8bd468aba 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7682,6 +7682,14 @@ Kernel parameters
 			causing a major performance hit, and the space where
 			machines are deployed is by other means guarded.
 
+	tpm.max_missed_responses= [HW,TPM]
+			Number of commands in a row that may time out before
+			the TPM is disabled. It stays disabled until the
+			driver is rebound or the machine reboots. Set to 0
+			to never disable the TPM.
+			Format: <unsigned int>
+			Default: 60
+
 	tpm_crb_ffa.busy_timeout_ms= [ARM64,TPM]
 			Maximum time in milliseconds to retry sending a message
 			to the TPM service before giving up. This parameter controls
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index e0385b16616b46..977e205807033d 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -40,6 +40,11 @@ module_param_named(suspend_pcr, tpm_suspend_pcr, uint, 0644);
 MODULE_PARM_DESC(suspend_pcr,
 		 "PCR to use for dummy writes to facilitate flush on suspend.");
 
+static unsigned int tpm_max_missed_responses = 60;
+module_param_named(max_missed_responses, tpm_max_missed_responses, uint, 0444);
+MODULE_PARM_DESC(max_missed_responses,
+		 "Disable the TPM after this many commands in a row time out (0 = never).");
+
 /**
  * tpm_calc_ordinal_duration() - calculate the maximum command duration
  * @chip:    TPM chip to use.
@@ -268,6 +273,15 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
 		delay_msec *= 2;
 		memcpy(buf, save, save_size);
 	}
+
+	if (ret >= 0) {
+		chip->nr_missed_responses = 0;
+	} else if (ret == -ETIME && tpm_max_missed_responses &&
+		   !(++chip->nr_missed_responses % tpm_max_missed_responses)) {
+		dev_err(&chip->dev, "%u commands in a row timed out, disabling the TPM\n",
+			chip->nr_missed_responses);
+		chip->flags |= TPM_CHIP_FLAG_DISABLE;
+	}
 	return ret;
 }
 
diff --git a/include/linux/tpm.h b/include/linux/tpm.h
index 0db277af45c393..6ad1e254831f5f 100644
--- a/include/linux/tpm.h
+++ b/include/linux/tpm.h
@@ -131,6 +131,7 @@ struct tpm_chip {
 	struct hwrng hwrng;
 
 	struct mutex tpm_mutex;	/* tpm is processing */
+	unsigned int nr_missed_responses; /* commands in a row that timed out */
 
 	unsigned long timeout_a; /* jiffies */
 	unsigned long timeout_b; /* jiffies */

-- 
2.53.1-Meta

[PATCH v3 2/2] tpm: Disable TPM on TPM_RC_INITIALIZE

[Breno Leitao]

From: Jarkko Sakkinen <jarkko@kernel.org>

If the TPM has been spuriously disabled, the game is over. Let's reflect
this truth with TPM_CHIP_FLAG_DISABLE.

Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
[leitao@debian.org: check rc before the loop breaks out on anything but
 RETRY/TESTING, skip TPM2_Startup, which a TPM that is already started
 fails with TPM2_RC_INITIALIZE, and fix the TPM2_RC_INITIALIZE and
 chip->flags typos]
Reviewed-by: Jarkko Sakkinen <jarkko@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
---
 drivers/char/tpm/tpm-interface.c | 5 +++++
 drivers/char/tpm/tpm2-cmd.c      | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index 977e205807033d..dd6bb518ab7d9c 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -252,6 +252,11 @@ ssize_t tpm_transmit(struct tpm_chip *chip, u8 *buf, size_t bufsiz)
 		if (ret < 0)
 			break;
 		rc = be32_to_cpu(header->return_code);
+		/* A TPM that is already started fails TPM2_Startup this way. */
+		if (rc == TPM2_RC_INITIALIZE && cc != TPM2_CC_STARTUP) {
+			chip->flags |= TPM_CHIP_FLAG_DISABLE;
+			break;
+		}
 		if (rc != TPM2_RC_RETRY && rc != TPM2_RC_TESTING)
 			break;
 		/*
diff --git a/drivers/char/tpm/tpm2-cmd.c b/drivers/char/tpm/tpm2-cmd.c
index 4b2bb2ddc321a9..65928ae85b496c 100644
--- a/drivers/char/tpm/tpm2-cmd.c
+++ b/drivers/char/tpm/tpm2-cmd.c
@@ -724,6 +724,8 @@ int tpm2_auto_startup(struct tpm_chip *chip)
 		goto out;
 
 	if (rc == TPM2_RC_INITIALIZE) {
+		chip->flags &= ~TPM_CHIP_FLAG_DISABLE;
+
 		rc = tpm2_startup(chip);
 		if (rc)
 			goto out;

-- 
2.53.1-Meta