[PATCH v6 3/3] serial: core: reject baud_base values that overflow port->uartclk in uart_set_info()

[Hui Peng <benquike@gmail.com>]

In uart_set_info(), new_info->baud_base is multiplied by 16 and stored in
uport->uartclk (an unsigned int):

  uport->uartclk = new_info->baud_base * 16;

While uart_set_info() checks if (uartclk == 0), when new_info->baud_base
exceeds UINT_MAX / 16 with low bits set (for example, 0x10000001),
multiplying by 16 wraps around in 32-bit unsigned arithmetic to a small
non-zero value (16), bypassing uartclk == 0 and setting uport->uartclk = 16.

Use check_mul_overflow(new_info->baud_base, 16, &uartclk) in
uart_set_info() to reject baud_base values that overflow 32-bit unsigned
arithmetic.

Tested in QEMU against Linux 7.3.0-rc3 by calling ioctl(fd, TIOCSSERIAL,
&ss) with ss.baud_base = 0x10000001 on /dev/ttyS1: on the unfixed kernel
TIOCSSERIAL succeeds (ret = 0) and wraps uport->uartclk to 16
(TIOCGSERIAL reports baud_base = 1), whereas with the fix applied
TIOCSSERIAL returns -EINVAL and preserves the existing uport->uartclk.

Fixes: 6eabce6608d6 ("serial: core: check uartclk for zero to avoid divide by zero")
Cc: stable@vger.kernel.org
Reviewed-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@gmail.com>
---
Changes in v6:
- Use check_mul_overflow() as suggested by Jiri Slaby.

Changes in v5:
- Collect Reviewed-by from Ilpo Järvinen.

 drivers/tty/serial/serial_core.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index 95774b0f1484..75bb1eaa27e7 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -935,11 +935,6 @@ static int uart_set_info(struct tty_struct *tty, struct tty_port *port,
 		unsigned int uartclk;

 		/* check needs to be done here before other settings made */
-		if (new_info->baud_base > UINT_MAX / 16)
-			return -EINVAL;
-
-		uartclk = new_info->baud_base * 16;
-		if (uartclk == 0)
+		if (check_mul_overflow(new_info->baud_base, 16, &uartclk) || !uartclk)
 			return -EINVAL;
 	}

-- 
2.47.3