From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.210.49 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1794196284; b=zkv0Ns1ZowsnSNyfwTNSOsnw/+lydoqZKWC7i0d+RS2cSJTSbJ9bQKleAw8NzB2BBfNf MnF1jWFwolTK9MTxhpBXOWFeqG8llRJiT6Q//TpLuCJzwE4F77YkZN+HH4FX7paPv86LA j/Rwp3sy9fmSAPdRICcl7kLXiJ6Xee1lp/heE9ejIb8flxyC7eVl+f0jPC2dKrVvUSfke mKsaNCKRWv/QY1En6z/NBNNGOVZsG5RE2Cn2DFI1UsQABEth8HI1wDbCtdHKL40w5tzt/ gMcHeu5SQHyttQKEdaIAgNfM07rOhAEaTF7NEbLBm4f+R7TQPVbdU/GTJw7XjNxIRiQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1794196284; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Gm-Gg:X-Received:Received:From:Date:Subject: MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id: References:In-Reply-To:To:Cc:X-Mailer; bh=bx/GLf5acqAm+qHz8cVXpIgh0vyBh0TaavgV6EQqWjE=; b=PD5NyX6d+daugFOoHshp/HmCo0flCwbDp2My1G/+I9pZJtLgmzHnQzshZmAf6GcG3Kcr P/EFJte+zcHFsWDFnwcI9Zs3rEikLXkGgwVqufjuS2F8YTh2Nlrx6P8HNk+plngLhD+fT G2BM0iL09qpu8Q+I4B8FIqX8d/OseQG1zuo2GyabiUZdPcTF8tg+zBZcP08r8uEKtZad2 ey6+yFPk5ELfm7Lk7mf/eYfiIYw8CwMJyNT+OE8Rba45J0I3CCk+8YsTDNiow6ONOH7lC x2oAri+Df8CiCqwvmcp3DI2GLugoGBL2l6IJcTVxGRTjAJ9Y6r5kEwT2cbOvrr8FrRg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=Qbcnm0yC; arc=none smtp.remote-ip=209.85.210.49 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 622E4606A321 for ; Mon, 9 Nov 2026 03:51:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 622E4606A321 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20260707 header.b=Qbcnm0yC Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-8490a7d6696so210715a34.3 for ; Sun, 08 Nov 2026 19:51:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1794196282; x=1794801082; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bx/GLf5acqAm+qHz8cVXpIgh0vyBh0TaavgV6EQqWjE=; b=Qbcnm0yC2aoA7aXu8fczndo+jLREP61J/gYETBf4chie5Tm2nfgxH7lJiGZbtpJ0oP fIJ9cv0I8ndKLvdm7XAIbBHQ7szXlKmikz+wnrgy7uM3pbzePQzckHuepTsie9y4/OdQ nwtsCeMqAS9Kt9IWYn13tG4Ib9uX87i+2ij5q+4lKNd2PqbkTg+Hh3geyOueYIDp0b6T x6zxomueyjQPYTmGBEmGqNtgZe1XzHFiPQ64lYRV9KNX1hf957UMEDneb8+LhojOKPiO X5vLEdD53OfEkiV3kJ/zeLYYHrVyIHfjsC2ts8MhhszMTJGoeao/mSsRZjI6k9eJkUHc S4Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794196282; x=1794801082; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bx/GLf5acqAm+qHz8cVXpIgh0vyBh0TaavgV6EQqWjE=; b=aOdfgYNS2Qbu2HfuqN+AIn3nJABTks0zIbNOVPR7YJEHMeglMnUG7KLnMjr7LnFoI3 KXmf7K87FOGy5dR+0IA+kZpr5e567X/6x6RMedmjCJIHMpk+orrsPCRpV1Nd4EmX38H3 XH7trvhUL1e5tM/N8xv4T9iIs/8zFcaOlqvPI0VmHYsQcs+ak8RskMsou/MAIKKGv8Kr WKwSxjRpRiXAT336o5sb4qu6oNraPoDhwkOR4d2Fj26A5SqlAdYqSE7FgG2mToDJc0/1 0mQUh6z7alroktnuQKejxBslnYd2Ja+tifOTPd6dvk4FTgxnGXA8JDNdnGZzPuvr9Pme ARew== X-Gm-Message-State: APuU4Z8zgpRN+yGamU5CycoK4ypVSD4/EVw2xoVnn8OAOoTRRgYSY5zq 9t6KOxE4wIYyBijHv0YxkQV7IqZh68nVtqvsxYkhqGq4misO6du0vqXV X-Gm-Gg: AYBFou1xaXrE9uS5RDSAFR1L5XbVx2r5gqKH9gfpBp0ZHw1SUkqJs8dHOogklUYBzyU UWAsFuxIO3jxsfS9x10DGjGW8G/pYzAGm/CTpVOvh+GCmQnN0dbeJfQMElVqspchSaylv+iUB24 wyQY+AufpB3VacUoaCIUvSIzbr8VTvW7Bxviu8mY7SyDImttPQWmFYYMwdWIfH1B8PCN1Q94aYv T+iRrJPdyssNBKxhwbfjSrCoY7Irg56B9aWqC8uF6ZYWf805EIQAvx1aF0UoW4yGMHTPZJJLdtk yTBD+Tsq8fufUeOJvHZdHMYPTip55vUCi2ESDWiRKp1Hc0bcn+c9lRuZTiNBY0Lf1znvgLC4psp BLDAmBWlkN/ETnjNOAamo+q1Bk5WXSA/nD8JROoUE9gmZMzYO1xD97Qd3lpGF07zHg4wvYj1CmD /hw/X4wQ0X6D2DcMbRCdPXRmz6exlsqW9R4ubR/Rbk6qx1pc7P05bDlUUvdT/agWk79/0x2QB8p 8HxfEqrArOBhiPyTq6qtQduS5//DW72SaMgYO10hpsF2y8lDfhnJVHoCc2l2RRKA2YXS73FQMj6 J3pV3i/0fveKJq7vfrD/WLbvSjWJHALShBio+NE1dQjd6lzhPCMJP2ArGRsyKnI0hGZYs66S5ZF k4GSChYzp40sjH7GsKOZxkbIOAuV3lz6tgStATaoT5wTTNi/ZidYoN1SfDgGZ6vpOwxVj4ijXF0 Dn/dYDbSho6AR2/mBj7COeAGq6k9jcseuwBkCGBDgiDyTRhV0mcRZpjgxcv+dtJ+MITQnmxKdNa w== X-Received: by 2002:a05:6830:a183:10b0:85f:49ec:c974 with SMTP id 46e09a7af769-85f506e18bdmr1252749a34.14.1794196282153; Sun, 08 Nov 2026 19:51:22 -0800 (PST) Received: from [127.0.1.1] (174-29-1-49.hlrn.qwest.net. [174.29.1.49]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-85f4fad32cdsm2800428a34.6.2026.11.08.19.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Nov 2026 19:51:21 -0800 (PST) From: James Hilliard Date: Sun, 08 Nov 2026 20:50:16 -0700 Subject: [PATCH v10 7/8] serial: 8250: retire canceled RX DMA before reusing the receive buffer X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261108-submit-serial-8250-dma-cancel-v1-v10-7-e6227ddb95b1@gmail.com> References: <20261108-submit-serial-8250-dma-cancel-v1-v10-0-e6227ddb95b1@gmail.com> In-Reply-To: <20261108-submit-serial-8250-dma-cancel-v1-v10-0-e6227ddb95b1@gmail.com> To: Greg Kroah-Hartman , Jiri Slaby , Raul E Rangel , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Andy Shevchenko , Heikki Krogerus , Tony Lindgren , Sebastian Andrzej Siewior , Richard Genoud , Nicolas Ferre , Alexandre Belloni , Claudiu Beznea , Jacky Huang , Shan-Chun Hung , =?utf-8?q?Pali_Roh=C3=A1r?= , Frank Li , Sascha Hauer , Pengutronix Kernel Team , Fabio Estevam , "Maciej W. Rozycki" , Timur Tabi , Peter Ujfalusi , Vignesh Raghavendra , Patrice Chotard Cc: linux-kernel@vger.kernel.org, linux-serial@vger.kernel.org, linux-arm-kernel@lists.infradead.org, imx@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, James Hilliard X-Mailer: b4 0.15.2 The generic RX timeout path pauses a transfer, copies its data and terminates it asynchronously. Clearing rx_running at that point lets a new receive reuse the buffer and cookie while the old callback has not yet run. Inspecting the new cookie cannot distinguish that callback from the completion of a new transfer. Reserve the receive buffer until a worker has drained the canceled callbacks. Keep FIFO access separate from callback retirement: after a successful pause and termination, clear rx_running while rx_terminating prevents another descriptor from reusing the buffer. Deliver the paused data before allowing the interrupt handler to drain the FIFO through PIO. Normal retirement only releases ownership and leaves subsequent receive handling to the UART interrupt. Preserve the normal completion-path LSR read and gate DMA resubmission on the receive interrupt enable, so stop_rx() prevents another transfer while the status read retains its existing side effects. Reject new flush work and PIO receive handling during shutdown, including late Intel MID DMA timeouts. Stop and drain retirement work before freeing the channels and buffer. Once RX has stopped, allow the idle flush used by DesignWare LCR recovery during shutdown. Clear the shutdown guard before startup configures the UART and tests its interrupts. Check pause failures before accessing the buffer. A successful pause is the quiescence guarantee, even when a provider still reports IN_PROGRESS. Also accept DMA_COMPLETE after a failed pause: completion can stop the channel before its callback acquires the port lock, leaving nothing for the provider to pause. Retire that completed transfer normally. A failed pause with an incomplete cookie is ambiguous: hardware may have finished while its completion interrupt is still pending. Retain the transfer and retry pause and status in the worker, with receive interrupts masked to prevent a timeout interrupt storm. The worker copies only after a successful pause or reported completion, then terminates and drains the old callback before reuse. The pending-pause guard keeps callbacks from copying or resubmitting the transfer concurrently. A bounded retry window could instead enter synchronous recovery. Successful dmaengine_terminate_sync() would stop the transfer and drain its callbacks, making buffer reuse and PIO access safe under the DMAengine contract. The concern is data preservation, not memory safety after successful termination: this recovery path discards undelivered receive data and disables RX DMA until reopen. Retrying ambiguous pause failures preserves the chance to deliver that data and continue using DMA when the provider recovers. Reserve the existing recovery path for DMA_ERROR or termination failure rather than selecting it solely on elapsed time. RX DMA admission already requires cmd_pause. A provider whose hardware cannot pause must report that capability correctly so the unsupported channel is rejected before receiving data. Start pause retries at 1 ms and double the delay up to one second to limit CPU use during persistent failure. Sleep outside the port lock on a timed waitqueue, with completion and shutdown publishing a wakeup under the same lock as the worker's retry check. This also catches notifications arriving just before the worker sleeps, avoiding a full backoff delay after completion or during shutdown. Reset the delay for each new flush. Report a stall lasting at least one second once per flush. Keep the warning local to that flush so simultaneous stalls on different ports and later stalls on the same port are reported independently. Retries and diagnostics do not guarantee progress or lossless reception. If a provider never pauses or completes, a partial transfer can remain pending indefinitely while a request/response peer waits for a reply. Receive interrupts stay masked, the worker holds its runtime PM reference, and DesignWare line-setting updates cannot proceed through the idle sequence. Further incoming data can overrun the FIFO. This retains DMA ownership until an error or shutdown; the backoff bounds retry cost, not the duration of the stall. On DMA_ERROR or termination failure, let the worker stop DMA before handing the FIFO to PIO. Keep the requested enables in the existing IER shadow and filter hardware writes during pause retries and recovery, including termios, RS485 and overrun-backoff changes. Apply the current enables after retirement so stop_rx() and shutdown remain effective. Hold a runtime PM reference around worker register accesses and keep RX on PIO until reopen after a confirmed error. Abort the DesignWare idle sequence when RX cannot be stopped. Allow the empty-FIFO timeout workaround once DMA has released the FIFO, including while callbacks retire. Exclude FIFO resets during RX DMA recovery. If the provider still cannot terminate the channel in process context, retain its buffer and leave receive interrupts masked. Fixes: 9ee4b83e51f7 ("serial: 8250: Add support for dmaengine") Signed-off-by: James Hilliard --- drivers/tty/serial/8250/8250.h | 46 +++++++- drivers/tty/serial/8250/8250_core.c | 2 +- drivers/tty/serial/8250/8250_dma.c | 226 +++++++++++++++++++++++++++++------- drivers/tty/serial/8250/8250_dw.c | 28 +++-- drivers/tty/serial/8250/8250_port.c | 36 ++++-- 5 files changed, 276 insertions(+), 62 deletions(-) diff --git a/drivers/tty/serial/8250/8250.h b/drivers/tty/serial/8250/8250.h index 1e739aceff39..25ca59aa701d 100644 --- a/drivers/tty/serial/8250/8250.h +++ b/drivers/tty/serial/8250/8250.h @@ -11,6 +11,7 @@ #include #include #include +#include #include #include "../serial_mctrl_gpio.h" @@ -23,6 +24,10 @@ struct uart_8250_dma { /* Completion processing with the port lock held. */ void (*tx_complete)(struct uart_8250_port *p); struct work_struct tx_flush_work; + struct work_struct rx_flush_work; + wait_queue_head_t rx_pause_wait; + /* Written under the port lock; sampled locklessly by the wait condition. */ + bool rx_pause_wakeup; struct uart_8250_port *port; /* Filter function */ @@ -55,11 +60,16 @@ struct uart_8250_dma { unsigned char tx_running; unsigned char tx_err; + /* RX DMA may access the FIFO; independent of callback retirement. */ unsigned char rx_running; /* Protected by the UART port lock. */ bool shutting_down:1; bool tx_terminating:1; bool tx_flushed:1; + bool rx_terminating:1; + /* Retry pause without discarding a completion awaiting its DMA IRQ. */ + bool rx_pause_pending:1; + bool rx_err:1; }; struct old_serial_port { @@ -148,6 +158,29 @@ static inline void serial_out(struct uart_8250_port *up, int offset, int value) up->port.serial_out(&up->port, offset, value); } +static inline bool serial8250_rx_dma_running(struct uart_8250_port *up) +{ + return up->dma && up->dma->rx_running; +} + +static inline bool serial8250_rx_dma_recovering(struct uart_8250_port *up) +{ + return up->dma && + (up->dma->rx_pause_pending || (up->dma->rx_err && up->dma->rx_terminating)); +} + +/* Keep the desired enables in up->ier while RX DMA recovery masks hardware. */ +static inline void serial8250_out_IER(struct uart_8250_port *up, u8 ier) +{ + lockdep_assert_held_once(&up->port.lock); + + if (up->dma && up->dma->shutting_down) + ier = 0; + else if (serial8250_rx_dma_recovering(up)) + ier &= ~(UART_IER_RDI | UART_IER_RLSI | UART_IER_RTOIE); + serial_out(up, UART_IER, ier); +} + /** * serial_lsr_in - Read LSR register and preserve flags across reads * @up: uart 8250 port @@ -221,7 +254,7 @@ static inline bool serial8250_set_THRI(struct uart_8250_port *up) if (up->ier & UART_IER_THRI) return false; up->ier |= UART_IER_THRI; - serial_out(up, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); return true; } @@ -233,7 +266,7 @@ static inline bool serial8250_clear_THRI(struct uart_8250_port *up) if (!(up->ier & UART_IER_THRI)) return false; up->ier &= ~UART_IER_THRI; - serial_out(up, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); return true; } @@ -417,7 +450,7 @@ extern void serial8250_tx_dma_flush(struct uart_8250_port *); void serial8250_tx_dma_sync_buffer(struct uart_8250_port *p, struct scatterlist *sgl, unsigned int nents); extern int serial8250_rx_dma(struct uart_8250_port *); -extern void serial8250_rx_dma_flush(struct uart_8250_port *); +int serial8250_rx_dma_flush(struct uart_8250_port *p); extern int serial8250_request_dma(struct uart_8250_port *); extern void serial8250_release_dma(struct uart_8250_port *); @@ -473,7 +506,12 @@ static inline int serial8250_rx_dma(struct uart_8250_port *p) { return -1; } -static inline void serial8250_rx_dma_flush(struct uart_8250_port *p) { } + +static inline int serial8250_rx_dma_flush(struct uart_8250_port *p) +{ + return 0; +} + static inline int serial8250_request_dma(struct uart_8250_port *p) { return -1; diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c index 42319843cb5b..ca06288038ed 100644 --- a/drivers/tty/serial/8250/8250_core.c +++ b/drivers/tty/serial/8250/8250_core.c @@ -738,7 +738,7 @@ static void serial_8250_overrun_backoff_work(struct work_struct *work) guard(uart_port_lock_irqsave)(&up->port); up->ier |= UART_IER_RLSI | UART_IER_RDI; - serial_out(up, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } static int __serial8250_register_8250_port(struct uart_8250_port *uart, diff --git a/drivers/tty/serial/8250/8250_dma.c b/drivers/tty/serial/8250/8250_dma.c index 739a8238e4de..1ea22954150c 100644 --- a/drivers/tty/serial/8250/8250_dma.c +++ b/drivers/tty/serial/8250/8250_dma.c @@ -4,6 +4,9 @@ * * Copyright (C) 2013 Intel Corporation */ +#include +#include +#include #include #include #include @@ -84,53 +87,174 @@ void serial8250_tx_dma_sync_buffer(struct uart_8250_port *p, } EXPORT_SYMBOL_GPL(serial8250_tx_dma_sync_buffer); -static void __dma_rx_complete(struct uart_8250_port *p) +static void serial8250_rx_dma_copy(struct uart_8250_port *p, + const struct dma_tx_state *state) { - struct uart_8250_dma *dma = p->dma; - struct tty_port *tty_port = &p->port.state->port; - struct dma_tx_state state; - enum dma_status dma_status; - int count; - - /* - * New DMA Rx can be started during the completion handler before it - * could acquire port's lock and it might still be ongoing. Don't to - * anything in such case. - */ - dma_status = dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state); - if (dma_status == DMA_IN_PROGRESS) - return; - - count = dma->rx_size - state.residue; + struct uart_8250_dma *dma = p->dma; + struct tty_port *tty_port = &p->port.state->port; + size_t count = dma->rx_size - state->residue; if (tty_insert_flip_string(tty_port, dma->rx_buf, count) != count) p->port.icount.buf_overrun++; p->port.icount.rx += count; - dma->rx_running = 0; tty_flip_buffer_push(tty_port); } +static void serial8250_rx_dma_recover(struct uart_8250_port *p) +{ + struct uart_8250_dma *dma = p->dma; + + /* Keep PIO and new descriptors out until the worker stops the channel. */ + dma->rx_err = true; + dma->rx_terminating = true; + serial8250_out_IER(p, p->ier); +} + +/* Try to obtain a stable residue before retiring the current descriptor. */ +static int serial8250_rx_dma_try_flush(struct uart_8250_port *p) +{ + struct uart_8250_dma *dma = p->dma; + struct dma_tx_state state; + enum dma_status status; + int ret; + + lockdep_assert_held_once(&p->port.lock); + + ret = dmaengine_pause(dma->rxchan); + status = dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state); + if (status == DMA_ERROR) { + ret = -EIO; + goto recover; + } + if (ret && status != DMA_COMPLETE) { + /* + * Hardware may be idle before the provider publishes completion. + * Leave the descriptor alive and retry outside interrupt context; + * neither copying its buffer nor canceling it is safe yet. + */ + dma->rx_pause_pending = true; + serial8250_out_IER(p, p->ier); + return ret; + } + /* Some providers still report IN_PROGRESS after a successful pause. */ + serial8250_rx_dma_copy(p, &state); + dma->rx_terminating = true; + ret = dmaengine_terminate_async(dma->rxchan); + if (ret) + goto recover; + + /* DMA released the FIFO; retain the buffer until callbacks retire. */ + dma->rx_running = 0; + return 0; + +recover: + serial8250_rx_dma_recover(p); + return ret; +} + static void dma_rx_complete(void *param) { struct uart_8250_port *p = param; struct uart_8250_dma *dma = p->dma; - unsigned long flags; + struct dma_tx_state state; + enum dma_status status; - uart_port_lock_irqsave(&p->port, &flags); - if (dma->shutting_down) - goto out; - if (dma->rx_running) - __dma_rx_complete(p); + guard(uart_port_lock_irqsave)(&p->port); - /* - * Cannot be combined with the previous check because __dma_rx_complete() - * changes dma->rx_running. - */ - if (!dma->rx_running && (serial_lsr_in(p) & UART_LSR_DR)) - p->dma->rx_dma(p); -out: - uart_port_unlock_irqrestore(&p->port, flags); + if (dma->shutting_down || dma->rx_terminating) + return; + if (dma->rx_pause_pending) { + /* Let the worker observe completion without waiting for its timeout. */ + WRITE_ONCE(dma->rx_pause_wakeup, true); + wake_up(&dma->rx_pause_wait); + return; + } + if (dma->rx_running) { + status = dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state); + if (status == DMA_ERROR) { + serial8250_rx_dma_recover(p); + schedule_work(&dma->rx_flush_work); + return; + } + if (status != DMA_COMPLETE) + return; + serial8250_rx_dma_copy(p, &state); + dma->rx_running = 0; + } + + /* Preserve LSR read side effects when stop_rx() has disabled resubmission. */ + if ((serial_lsr_in(p) & UART_LSR_DR) && (p->ier & UART_IER_RDI)) + dma->rx_dma(p); +} + +static void serial8250_rx_dma_flush_work(struct work_struct *work) +{ + struct uart_8250_dma *dma = container_of(work, struct uart_8250_dma, rx_flush_work); + struct uart_8250_port *p = dma->port; + unsigned int delay_us = 1 * USEC_PER_MSEC; + ktime_t start = ktime_get(); + bool recover, retry, restore_rx; + bool warned = false; + int ret; + + guard(serial8250_rpm)(p); + + for (;;) { + scoped_guard(uart_port_lock_irqsave, &p->port) { + /* Shutdown cancels this work and drains both DMA channels. */ + if (dma->shutting_down) + return; + /* Serialize reset with completion and shutdown notifications. */ + WRITE_ONCE(dma->rx_pause_wakeup, false); + retry = dma->rx_pause_pending && !dma->rx_terminating; + if (retry) { + ret = serial8250_rx_dma_try_flush(p); + retry = dma->rx_pause_pending && !dma->rx_terminating; + } + recover = dma->rx_err; + } + if (!retry) + break; + if (!warned && ktime_ms_delta(ktime_get(), start) >= 1 * MSEC_PER_SEC) { + dev_warn(p->port.dev, + "RX DMA pause stalled: %d; retaining transfer and retrying\n", + ret); + warned = true; + } + /* Notifications also cover completion or shutdown before sleeping. */ + wait_event_hrtimeout(dma->rx_pause_wait, READ_ONCE(dma->rx_pause_wakeup), + ns_to_ktime(delay_us * NSEC_PER_USEC)); + delay_us = min(delay_us * 2, 1 * USEC_PER_SEC); + } + + if (recover) { + ret = dmaengine_terminate_sync(dma->rxchan); + if (ret) { + /* Retain ownership and keep RX masked if DMA cannot be stopped. */ + dev_err(p->port.dev, "failed to stop RX DMA: %d\n", ret); + return; + } + } else { + dmaengine_synchronize(dma->rxchan); + } + + scoped_guard(uart_port_lock_irqsave, &p->port) { + restore_rx = recover || dma->rx_pause_pending; + dma->rx_pause_pending = false; + dma->rx_terminating = false; + dma->rx_running = 0; + if (dma->shutting_down) + return; + + if (!restore_rx) + return; + + /* Apply the current enables, including any intervening stop_rx(). */ + serial8250_out_IER(p, p->ier); + } + if (recover) + dev_warn(p->port.dev, "RX DMA failed, using PIO\n"); } int serial8250_tx_dma(struct uart_8250_port *p) @@ -240,6 +364,9 @@ int serial8250_rx_dma(struct uart_8250_port *p) if (dma->shutting_down || dma->rx_running) return 0; + /* A retiring transfer has released the FIFO, but not its buffer. */ + if (dma->rx_err || dma->rx_terminating) + return -EIO; serial8250_do_prepare_rx_dma(p); @@ -260,16 +387,26 @@ int serial8250_rx_dma(struct uart_8250_port *p) return 0; } -void serial8250_rx_dma_flush(struct uart_8250_port *p) +int serial8250_rx_dma_flush(struct uart_8250_port *p) { struct uart_8250_dma *dma = p->dma; + int ret; - if (!dma || !dma->rxchan || !dma->rx_running) - return; + lockdep_assert_held_once(&p->port.lock); - dmaengine_pause(dma->rxchan); - __dma_rx_complete(p); - dmaengine_terminate_async(dma->rxchan); + if (!serial8250_rx_dma_running(p)) + return 0; + /* A late interrupt must not hand a still-active receiver to PIO. */ + if (dma->shutting_down) + return -ESHUTDOWN; + if (dma->rx_terminating) + return -EIO; + if (dma->rx_pause_pending) + return -EAGAIN; + + ret = serial8250_rx_dma_try_flush(p); + schedule_work(&dma->rx_flush_work); + return ret; } EXPORT_SYMBOL_GPL(serial8250_rx_dma_flush); @@ -366,10 +503,16 @@ int serial8250_request_dma(struct uart_8250_port *p) dma->port = p; INIT_WORK(&dma->tx_flush_work, serial8250_tx_dma_flush_work); + INIT_WORK(&dma->rx_flush_work, serial8250_rx_dma_flush_work); + init_waitqueue_head(&dma->rx_pause_wait); scoped_guard(uart_port_lock_irqsave, &p->port) { dma->tx_terminating = false; dma->tx_flushed = false; + dma->rx_terminating = false; + dma->rx_pause_pending = false; + WRITE_ONCE(dma->rx_pause_wakeup, false); + dma->rx_err = false; dma->shutting_down = false; } dev_dbg_ratelimited(p->port.dev, "got both dma channels\n"); @@ -393,10 +536,15 @@ void serial8250_release_dma(struct uart_8250_port *p) return; cancel_work_sync(&dma->tx_flush_work); + cancel_work_sync(&dma->rx_flush_work); /* Release RX resources */ dmaengine_terminate_sync(dma->rxchan); - dma->rx_running = 0; + scoped_guard(uart_port_lock_irqsave, &p->port) { + dma->rx_running = 0; + dma->rx_terminating = false; + dma->rx_pause_pending = false; + } dma_free_coherent(dmaengine_get_dma_device(dma->rxchan), dma->rx_size, dma->rx_buf, dma->rx_addr); dma_release_channel(dma->rxchan); diff --git a/drivers/tty/serial/8250/8250_dw.c b/drivers/tty/serial/8250/8250_dw.c index 9cbe076463e0..2f6681910351 100644 --- a/drivers/tty/serial/8250/8250_dw.c +++ b/drivers/tty/serial/8250/8250_dw.c @@ -112,7 +112,7 @@ static void dw8250_idle_exit(struct uart_port *p) if (up->capabilities & UART_CAP_FIFO) serial_port_out(p, UART_FCR, up->fcr); serial_port_out(p, UART_MCR, up->mcr); - serial_port_out(p, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); /* DMA Rx is restarted by IRQ handler as needed. */ if (up->dma) @@ -134,6 +134,7 @@ static int dw8250_idle_enter(struct uart_port *p) unsigned int usr_reg = d->pdata ? d->pdata->usr_reg : DW_UART_USR; struct uart_8250_port *up = up_to_u8250p(p); int retries; + int ret; u32 lsr; lockdep_assert_held_once(&p->lock); @@ -147,7 +148,12 @@ static int dw8250_idle_enter(struct uart_port *p) serial_port_out(p, UART_IER, 0); if (up->dma) { - serial8250_rx_dma_flush(up); + ret = serial8250_rx_dma_flush(up); + if (ret) { + serial8250_out_IER(up, up->ier); + d->in_idle = 0; + return ret; + } if (serial8250_tx_dma_running(up)) serial8250_tx_dma_pause(up); } @@ -390,9 +396,9 @@ static void dw8250_quirk_ier_kick(struct uart_port *p) if (!(lsr & UART_LSR_THRE)) return; - serial_port_out(p, UART_IER, up->ier | UART_IER_THRI); + serial8250_out_IER(up, up->ier | UART_IER_THRI); serial_port_in(p, UART_LCR); /* safe, no side-effects */ - serial_port_out(p, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } static int dw8250_handle_irq(struct uart_port *p) @@ -429,6 +435,9 @@ static int dw8250_handle_irq(struct uart_port *p) d->no_int_count = 0; + if (up->dma && up->dma->shutting_down) + return 1; + /* * There are ways to get Designware-based UARTs into a state where * they are asserting UART_IIR_RX_TIMEOUT but there is no actual @@ -436,10 +445,11 @@ static int dw8250_handle_irq(struct uart_port *p) * read. If we don't do this then the "RX TIMEOUT" interrupt will * fire forever. * - * This problem has only been observed so far when not in DMA mode - * so we limit the workaround only to non-DMA mode. + * Only read the FIFO when DMA no longer owns it, including PIO fallback + * after a DMA error. */ - if (!(quirks & DW_UART_QUIRK_SKIP_EMPTY_FIFO_READ) && !up->dma && rx_timeout) { + if (!(quirks & DW_UART_QUIRK_SKIP_EMPTY_FIFO_READ) && + !serial8250_rx_dma_running(up) && rx_timeout) { status = serial_lsr_in(up); /* @@ -454,7 +464,9 @@ static int dw8250_handle_irq(struct uart_port *p) } /* Manually stop the Rx DMA transfer when acting as flow controller */ - if (quirks & DW_UART_QUIRK_IS_DMA_FC && up->dma && up->dma->rx_running && rx_timeout) { + if ((quirks & DW_UART_QUIRK_IS_DMA_FC) && rx_timeout && + serial8250_rx_dma_running(up) && + !up->dma->rx_terminating && !up->dma->rx_pause_pending) { status = serial_lsr_in(up); if (status & (UART_LSR_DR | UART_LSR_BI)) { diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c index cfbd154bf5f9..49d6a0b19fb4 100644 --- a/drivers/tty/serial/8250/8250_port.c +++ b/drivers/tty/serial/8250/8250_port.c @@ -1306,7 +1306,7 @@ static void serial8250_stop_rx(struct uart_port *port) guard(serial8250_rpm)(up); up->ier &= ~(UART_IER_RLSI | UART_IER_RDI); - serial_port_out(port, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } /** @@ -1332,7 +1332,8 @@ void serial8250_em485_stop_tx(struct uart_8250_port *p, bool toggle_ier) * Enable previously disabled RX interrupts. */ if (!(p->port.rs485.flags & SER_RS485_RX_DURING_TX)) { - serial8250_clear_and_reinit_fifos(p); + if (!serial8250_rx_dma_recovering(p)) + serial8250_clear_and_reinit_fifos(p); if (toggle_ier) { /* @@ -1346,7 +1347,7 @@ void serial8250_em485_stop_tx(struct uart_8250_port *p, bool toggle_ier) lockdep_assert_held_once(&p->port.lock); p->ier |= UART_IER_RLSI | UART_IER_RDI; - serial_port_out(&p->port, UART_IER, p->ier); + serial8250_out_IER(p, p->ier); } } } @@ -1600,7 +1601,7 @@ static void serial8250_disable_ms(struct uart_port *port) mctrl_gpio_disable_ms_no_sync(up->gpios); up->ier &= ~UART_IER_MSI; - serial_port_out(port, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } static void serial8250_enable_ms(struct uart_port *port) @@ -1619,7 +1620,7 @@ static void serial8250_enable_ms(struct uart_port *port) up->ier |= UART_IER_MSI; guard(serial8250_rpm)(up); - serial_port_out(port, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } void serial8250_read_char(struct uart_8250_port *up, u16 lsr) @@ -1816,8 +1817,7 @@ static bool handle_rx_dma(struct uart_8250_port *up, unsigned int iir) fallthrough; case UART_IIR_RLSI: case UART_IIR_RX_TIMEOUT: - serial8250_rx_dma_flush(up); - return true; + return !serial8250_rx_dma_flush(up); } return up->dma->rx_dma(up); } @@ -1837,15 +1837,22 @@ void serial8250_handle_irq_locked(struct uart_port *port, unsigned int iir) lockdep_assert_held_once(&port->lock); + if (up->dma && up->dma->shutting_down) + return; + status = serial_lsr_in(up); /* * Recover from no-data-ready and FIFO error condition to avoid getting * stuck in the ISR. */ - if (!(status & UART_LSR_DR) && (status & UART_LSR_FIFOE)) + if (!(status & UART_LSR_DR) && (status & UART_LSR_FIFOE) && + !serial8250_rx_dma_running(up)) serial8250_clear_and_reinit_fifos(up); + if (serial8250_rx_dma_recovering(up)) + skip_rx = true; + /* * If port is stopped and there are no error conditions in the * FIFO, then don't drain the FIFO, as this may lead to TTY buffer @@ -2345,6 +2352,12 @@ int serial8250_do_startup(struct uart_port *port) guard(serial8250_rpm)(up); + /* The previous shutdown has drained DMA and its callbacks. */ + scoped_guard(uart_port_lock_irqsave, port) { + if (up->dma) + up->dma->shutting_down = false; + } + serial8250_startup_special(port); /* @@ -2447,8 +2460,11 @@ void serial8250_do_shutdown(struct uart_port *port) * Synchronize UART_IER access against the console. */ scoped_guard(uart_port_lock_irqsave, port) { - if (up->dma) + if (up->dma) { up->dma->shutting_down = true; + WRITE_ONCE(up->dma->rx_pause_wakeup, true); + wake_up(&up->dma->rx_pause_wait); + } up->ier = 0; serial_port_out(port, UART_IER, 0); } @@ -2786,7 +2802,7 @@ static void serial8250_set_ier(struct uart_port *port, struct ktermios *termios) if (up->capabilities & UART_CAP_RTOIE) up->ier |= UART_IER_RTOIE; - serial_port_out(port, UART_IER, up->ier); + serial8250_out_IER(up, up->ier); } static void serial8250_set_efr(struct uart_port *port, struct ktermios *termios) -- 2.53.0