* [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver
@ 2026-11-05 7:56 Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core Jack Wu via B4 Relay
` (5 more replies)
0 siblings, 6 replies; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
T9XX is the PCIe host device driver for MediaTek's
t900 modem. The driver uses the WWAN framework
infrastructure to create the following control ports
and network interfaces for data transactions.
* /dev/wwan0at0 - Interface that supports AT commands.
* /dev/wwan0mbim0 - Interface conforming to the MBIM
protocol.
* wwan0-X - Primary network interface for IP traffic.
The main blocks in the T9XX driver are:
* HW layer - Abstracts the hardware bus operations for
the device, and provides generic interfaces for the
transaction layer to get the device's information and
control the device's behavior. It includes:
* PCIe - Implements probe, removal and interrupt
handling.
* MHCCIF (Modem Host Cross-Core Interface) - Provides
interrupt channels for bidirectional event
notification such as handshake and port enumeration.
* Transaction layer - Implements data transactions for
the control plane and the data plane. It includes:
* DPMAIF (Data Plane Modem AP Interface) - Controls
the hardware that provides uplink and downlink
queues for the data path. The data exchange takes
place using circular buffers to share data buffer
addresses and metadata to describe the packets.
* CLDMA (Cross Layer DMA) - Manages the hardware
used by the port layer to send control messages to
the device using MediaTek's CCCI (Cross-Core
Communication Interface) protocol.
* TX Services - Dispatch packets from the port layer
to the device.
* RX Services - Dispatch packets to the port layer
when receiving packets from the device.
* Port layer - Provides control plane and data plane
interfaces to userspace. It includes:
* Control Plane - Provides device node interfaces
for controlling data transactions.
* Data Plane - Provides network link interfaces
wwanX (0, 1, 2...) for IP data transactions.
* Device lifecycle management - Contains the logic that
keeps the device working. It includes:
* FSM (Finite State Machine) - Drives the T9xx
device lifecycle (boot handshake, error recovery,
removal), and notifies each module when the state
changes.
The compilation of the T9XX driver is enabled by the
CONFIG_MTK_T9XX config option, which depends on
CONFIG_WWAN.
This submission covers the control plane only
(patches 1-6). The data plane will follow in a
separate series once the control plane is accepted.
Statistics are intentionally left out of this series.
---
Changes in v11:
- Patch 1 (Add PCIe core):
- Trimmed mtk_dev.h to <linux/bits.h>, <linux/types.h> and a forward
declaration of struct device; seven unused includes dropped
(Loic Poulain)
- Replaced the SET_HW_BITS() statement-expression macro with a static
inline helper, used for the reverse event mapping too (Loic Poulain)
- Removed mtk_pci_ack_dev_state(), which has no caller in the series
(Sashiko)
- mtk_pci_probe() saves and stores the PCI state before
pci_set_master(), so the restore in remove() no longer re-enables bus
mastering; remove()'s pci_clear_master() is gone (Sashiko)
- Added .shutdown, which runs the remove() teardown and reset; without
it a kexec reboot left the firmware running with its one-shot boot
notification already consumed (Sashiko)
- Comments: irq_cb_lock covers irq_cb_list[] and the irq_cb_data[]
store; a device misreporting its BAR sizes is out of scope rather
than impossible (Sashiko)
- Still no pci_resource_len() checks: the kernel assumes a device
behaves to its specification, and the BAR sizes of the two bound
products are fixed in hardware (Sashiko)
- The probe error path disables the ATR tables it enabled; removed a
leftover probe-time pci_restore_state(), an unused include and a
duplicate mdev->dev store; file-local helpers are static
- remove(): warn if the ACPI reset fails with the link already down,
sleep only after a doorbell that was actually sent, and poll up to 1 s
for the device before restoring its config space
- Clear MHCCIF level 1 before reading level 2, so an event raised while
the work runs is not lost
- pci_driver.name is KBUILD_MODNAME; .class and .class_mask are named in
the ID table; parameter and registration errors return
-EINVAL/-EBUSY/-ENOENT instead of -EFAULT
- Commit message and kernel-doc: the doorbell fallback also covers a
failing MRST._RST, every shutdown resets the modem, and the MHCCIF
callback registration states its context rules
- Patch 2 (Add control plane transaction layer framework):
- Removed the unused mtk_ctrl_trans.ctrl_blk back-pointer
- Patch 3 (Add control DMA interface):
- mtk_cldma_rx_done_work(): the buffer-reload and DMA-mapping-error
exits go through the need_restart handling, so a queue waiting for a
restart is not left stalled (Sashiko)
- Removed the BD (multi-descriptor) CLDMA path, its DMA pool and the
frag_size fields: every queue has MTU == frag_size, so it never ran
- Removed dead code: 30 unread register-table fields, a duplicate
queue-count macro, an MTU self-compare, a single-pass do/while, a stub
dev_start(), the rx_done() -EAGAIN retry loop and its force_recv
- rearm_lock, introduced here, serialises the instance reset and re-arm
against RX resume/restart and TX kicks; a re-armed TX queue starts at
the first descriptor the device still owns
- A re-arm that leaves a started TX queue stopped is no longer a failed
recovery; a failed TX start with a dead link fails its requests at
once; a DISABLE flush no longer fails an ENABLE queued behind it
- The RX worker re-queues itself when a refill fails; the RX cursor high
word is re-read and an all-ones half treated as link loss
- const register and queue tables; trans->available is a bool under
submit_lock; queue numbers print with %u; static_assert() that struct
trb still fits in skb->cb
- Patch 4 (Add control port):
- mtk_port_status_update(): reject a port-enumeration message shorter
than its header; the only length check came after three header reads
(Sashiko)
- Stamp the TX sequence number when the trb thread hands the packet to
CLDMA and advance it only on success, so a packet that fails after it
was queued takes no number instead of leaving a gap (Sashiko)
- mtk_port_check_rx_seq(): skip the check only while rx_seq holds a
reset sentinel, warn rate-limited and always deliver; a stored 0
disabled the check, and a gap dropped a valid packet (Sashiko)
- Port state transitions use test_and_set_bit()/test_and_clear_bit()
(Sashiko)
- Commit message: the port-state, RX dispatch and TRB kref paths only
become live with the FSM patch, together with the teardown order they
rely on (Sashiko)
- Comments: tx_seq has a single writer, at hardware hand-off; a receive
callback and its argument must stay valid until the receive path is
drained, because close alone does not wait (Sashiko)
- The raw port pointers in trb->priv and rxq->arg are safe, and a failed
channel disable is only logged: the queues and trb services are torn
down before the ports are freed (Sashiko)
- Removed dead code: the cdev RX buffering fields, trb->hdr_added, the
port counter, a duplicated completion callback and two macros that hid
a continue
- The blocking write waits with wait_event_interruptible() and its
result follows the TRB outcome; the PORT_S_FLUSH exit that reported a
full write is gone
- RX dispatch drops a packet whose channel peer does not match the
receiving CLDMA instance; mtk_port_strip_header() uses pskb_may_pull()
and does the pull; RX warnings are rate-limited
- The internal receive callback is published with release/acquire
ordering; control skbs are sized for their payload, not the queue MTU
- Comments: a port must outlive the trbs and RX queues holding its
pointer; PORT_S_FLUSH has one setter; status bits are state, not locks
- Patch 5 (Add FSM thread):
- mtk_cldma_dev_exit(): call dma_pool_destroy() unconditionally; a pool
with blocks outstanding keeps its pages anyway, so the special case
only leaked the struct dma_pool (Sashiko)
- mtk_cldma_rearm_queues(): never restart a queue that is being
stopped, and do nothing once the instance is quiesced; the quiesce
uses drain_workqueue() (Sashiko)
- mtk_fsm_startup_act(): on a handshake error unmask only that
handshake's channel, and re-check the event gate after unmasking so a
callback racing removal cannot undo its masking (Sashiko)
- A MUST_SUPPORT feature carries no host data, so HS3 still echoes it
back with data_len 0; a comment there says why no per-feature handler
table is needed (Sashiko)
- Removed the per-transition KOBJ_CHANGE uevent (non KEY=VALUE payload,
no consumer) and mtk_utility.h with it
- Removed dead code: the pre-transition notifier list and its register
parameter, the event mdev and len fields, an impossible hs_id check,
an unreachable DEV_ADD state and mtk_fsm_start()
- mtk_fsm_evt_submit() returns 0 or a negative errno instead of private
codes, always allocates with GFP_ATOMIC, and probe passes its error on
- QUEUE_ERROR is no longer re-unmasked from the ISR; txq/rxq_free()
clear the latched error and flush err_work after the final mask; a TX
stop failing with -ENODEV counts as stopped
- One HS2 handler for both peers, its duplicate guard claimed by
cmpxchg(); mtk_fsm_exit() reuses mtk_fsm_stop(); the MHCCIF callbacks
unmask through a gate-aware helper
- pcie/mtk_trans_ctrl.h no longer includes mtk_port.h
- Patch 6 (Add AT & MBIM WWAN ports):
- mtk_port_common_write(): in blocking mode wait on the port's trb wait
queue for HIF queue space instead of bypassing the depth check; v10
had left the blocking path with no bound (Sashiko)
- mtk_port_enable_by_type() walks the port table in place and clears
port->enable before enabling; the blocking TX wait returns
-ERESTARTSYS; the tx_mtu guard rejects a header-only MTU
- Link to v10: https://patch.msgid.link/20261014-t9xx_driver_v1-v10-0-4d0dde3613f5@compal.com
Changes in v10:
- Patch 1 (Add PCIe core):
- Kconfig: depends on PCI && ACPI -> PCI_MSI && ACPI; the driver only
ever requests MSI-X, so a CONFIG_PCI_MSI=n build linked but could
never probe
- t9xx_pci_table[] uses PCI_VENDOR_ID_MEDIATEK in place of the
private MTK_PCI_VENDOR_ID define, matching t7xx
- Dropped CEI_PCI_VENDOR_ID, which named the wrong company; the
second entry is a bare 0x03f0 with an /* HP RMM-G4 */ comment and
the commit message now names both bound products
- ATR inbound WIN0 source address: pci_resource_start() ->
pci_bus_address(); the endpoint matches a PCI bus address, not a
CPU physical address
- mtk_pci_bar_init() comment: the MSI-X window reaches 0x3103, not
0x3080, and the fixed offsets are guaranteed by the only two
devices this driver binds, whose BAR sizes are fixed by hardware
- mtk_pci_unregister_irq() publishes the NULL callback under
irq_cb_lock before synchronize_irq() and clears irq_cb_data only
afterwards, so a live callback cannot be paired with NULL data
- mtk_pci_free_irq() sets priv->irq_cnt = 0 after
pci_free_irq_vectors(), so mtk_pci_get_virq_id() stops handing out
virqs for vectors that no longer exist
- mtk_pci_remove() frees the IRQ vectors before mtk_mhccif_exit(),
the order the probe error path already used, so the unmask at the
tail of mtk_mhccif_isr_work() cannot re-open the source
- mtk_pci_remove() calls pci_restore_state() and pci_clear_master()
after pci_load_and_free_saved_state(): the reset leaves the device
at power-on defaults and the state was saved after pci_set_master()
- mtk_pci_remove() sleeps after the MHCCIF reset doorbell, which
unlike MRST._RST returns before the device has reset, so the config
restore needs it to settle first
- Commit message: MRST._RST is a platform-level reset that does not
depend on firmware state, and the MHCCIF doorbell is a best-effort
fallback that cannot recover stopped firmware
- The three ACPI capability checks in mtk_pci_pldr_setup() are
dev_dbg() rather than dev_err(): probe does not require MRST._RST,
and without it only removal is degraded
- Removed mtk_pci_clear_ext_evt() and its prototype; Patch 5 removed
both callers, leaving no user in this series
- Declined a read-back after the MHCCIF mask write: the posted write
is followed by synchronize_irq(), which orders it
- Deferred .shutdown to the follow-up series, where it lands together
with dev_pm_ops
- Patch 2 (Add control plane transaction layer framework):
- Subject gains "framework", and the commit message says
mtk_ctrl_init() and mtk_ctrl_exit() get their first callers in
Patch 3; the patch is unreachable at its own commit by construction
- mtk_ctrl_init() kerneldoc: ctrl_blk is devres managed, so it is
released at unbind and not by mtk_ctrl_exit()
- mtk_ctrl_blk::ctrl_hw_priv is void * from the start, instead of a
PCIe-specific type in a generic header
- <linux/kref.h> is no longer added here; Patch 4 adds it together
with struct trb::kref, its first user, so Patch 3 no longer has to
delete it again
- Patch 3 (Add control DMA interface):
- BD-mode RX clamps each BD's received length with the host-side
req->frag_size as well as the device-written data_allow_len, so an
oversized device value cannot overrun the buffer
- mtk_cldma_rx_done() retries mtk_cldma_reload_rx_skb() on the next
pass and re-arms the slot, instead of leaving a slot whose second
reload failed marked complete
- mtk_cldma_reload_rx_skb() frees the skb and clears both req->skb
and req->data_dma_addr on dma_map failure, so the retry does not
re-map a stale address
- mtk_cldma_rxq_free() stops the RX queue again after the flush
unless the stop returned -ETIMEDOUT; rx_done_work can resume or
restart the engine after it read need_exit
- mtk_cldma_rearm_queues() clears tx_started instead of starting TX
from req_pool[free_idx], which can already be complete, and queues
tx_done_work so a lost XFER_DONE still advances free_idx
- mtk_cldma_start_xfer() stops the queue and clears tx_started on
failure, so mtk_cldma_txq_flush() does not run against a queue the
hardware still owns
- mtk_cldma_drv_reset() writes only this CLDMA's bit to RST0_SET and
RST0_CLR instead of read-modify-writing the whole bank, and reads
each register back to flush the posted write
- That reverses the answer already sent for the reset sequencing
question: the read-modify-write on RST0_CLR releases every other
block held in reset in the same bank
- mtk_ctrl_srv_exit() calls mtk_ctrl_chs_flush(srv) right after
kthread_stop() instead of at the thread's tail, so queued TRBs are
flushed even when the thread never ran
- DISABLE insertion tracks the last ENABLE and __skb_queue_after()
it; with no ENABLE present it falls back to __skb_queue_head()
rather than the tail
- trans->queue_tbl is an xarray instead of a radix_tree, which
removes the radix_tree_delete() inside a radix_tree_for_each_slot()
walk and the missing rcu_read_lock()
- struct trb::kref moved out of mtk_ctrl_plane.h into Patch 4, where
mtk_port_trb_init() first takes the reference, rather than taking
the reference earlier here
- mtk_ctrl_trb_handler() publishes trb->status with
smp_store_release(), pairing with the port layer's
smp_load_acquire() so it cannot be seen before the geometry stores
- mtk_cldma_open() keeps the usr_cnt increment when the open
completes -EBUSY, with a comment at the decrement: -EBUSY leaves
the channel open, so its eventual DISABLE releases the count
- Patch 4 (Add control port):
- mtk_ctrl_init() drops the err_free_mem label, which freed nothing,
and clears mdev->ctrl_blk on failure so probe does not leave a
dangling block behind
- The cfg pointers are const through mtk_port_layer_cfg and the port
table helpers, which also removes the (struct mtk_port_cfg *) cast
on .port_cfg
- Driver prefix on three globals: mtk_port_mngr_grp_mtx, struct
mtk_port_ops, and a static mtk_port_ops_tbl[] behind
mtk_port_get_ops() in place of the exported ports_ops[] table
- Removed the MTK_TRB_HEADER_ADDED (0xADDED) in-flight sentinel,
which made a blocking write return 712173 as a byte count; struct
trb carries explicit bools and that write returns -EBADF
- mtk_port_trb_done() reads smp_load_acquire(&trb->done), set by
smp_store_release() in all three completions, instead of inferring
completion from a status written before the callback ran
- The queue geometry copy moved from mtk_port_open_trb_complete() to
mtk_port_ch_enable() after the wait, and runs for -EBUSY too, so
such an ENABLE publishes tx_mtu like any other
- Deleted the empty mtk_port_io_init()/mtk_port_io_exit(), so this
patch no longer rewrites pcie/mtk_pci.c at all; the commit message
documents the trb kref instead
- mtk_port_get_by_name() does the lookup and the kref_get() in one
critical section; mtk_port_search_by_name() is static and gains
lockdep_assert_held()
- mtk_port_add_header() calls skb_cow_head() before skb_push() and
returns the error instead of dropping it
- The tx_seq rollback samples seq_taken into a local first, with a
comment that the kref_put()s below it free the skb that trb lives
in
- port->tx_seq = 0 / rx_seq = -1 moved from mtk_port_struct_init() to
the top of mtk_port_ch_enable(), so CCCI sequence numbering
restarts with each modem session
- Patch 5 (Add FSM thread):
- mtk_fsm_idle_evt_handler() returns void, so the handshake channels
are unmasked whether or not the STARTUP submit succeeded; the old
early return left them masked forever
- mtk_fsm_hs1_handler() masks the channel before the submit and
unmasks it again on failure, so it cannot overwrite a re-arm the
FSM thread just did on its error path
- Removed both mtk_pci_clear_ext_evt() calls; the ISR already
acknowledges the MHCCIF event, so the handlers were acking twice
- mtk_cldma_dev_exit() leaves the CLDMA vector masked after the reset
so the ISR tail cannot re-open it, and moves the WRITE_ONCE()
unpublish to the top, where the comment claims it is
- mtk_cldma_txq_free()/mtk_cldma_rxq_free() mask QUEUE_XFER_DONE and
QUEUE_ERROR first, clear the latched error bit, flush err_work then
the worker, and mask XFER_DONE once more after that final flush
- mtk_cldma_err_work() is split into mtk_cldma_err_tx_one() and
mtk_cldma_err_rx_one(), so an early exit on the TX side cannot skip
the RX error carrying the same index
- mtk_cldma_rxq_restart() does the stop itself and leaves the queue
stopped on failure, so the restart is the only context that
programs an RX queue
- TX submit returns early on -EPIPE instead of falling into the
flush, which would unmap buffers err_work kept for the device, and
re-checks is_stopping under ring_lock before publishing a new HWO
- Added mtk_cldma_drv_quiesce() and exported mtk_cldma_quiesce();
mtk_pcie_hif_exit() calls it before joining the trb services, which
the IRQ handler and the workqueue read as well
- mtk_cldma_exit() comment corrected: not all of its callers hold
submit_lock
- The dma_pool_destroy() comment now says it releases the backing
pages even when blocks are outstanding, so the rings are leaked
deliberately, bounded by the device instance and reported
- mtk_fsm_feature_set_match() gained comments on three arms: a
mandatory feature is fatal only on an outright refusal, and the
backward-compat arm decides on the device's declaration alone
- Patch 6 (Add AT & MBIM WWAN ports):
- mtk_port_common_write() caps one write() at MTK_PORT_MAX_TX_PKT
packets and rejects more with -EMSGSIZE; packets after the first
carry force_send, which bypasses the HIF depth check
- force_send starts out as the blocking flag, so a blocking fd no
longer gets -EAGAIN on its first packet and the blocking worst case
is one queue depth rather than depth - 1
- mtk_port_common_write() reports the prefix already on the wire with
dev_err_ratelimited() when a submit fails mid-message, and samples
skb->len before the submit, which may free the skb
- Its comment states why that write is not made atomic: the WWAN
core already turns a partial write into a whole-write failure, and
the failed submit cannot be retried once the skb is freed
- mtk_port_wwan_tx_pause() drops w_lock around
mtk_pcie_hif_cmd_func(), which takes submit_lock; teardown holds
submit_lock while calling back into this port under w_lock
- mtk_port_wwan_open() publishes w_priv.w_port in the same critical
section that sets PORT_S_OPEN, so mtk_port_wwan_recv() cannot see
an open port with a NULL w_port and drop a reply
- mtk_port_get_locked() returns void and only takes the reference;
the WWAN core owns the port pointer across its .start/.stop pair
- Link to v9: https://patch.msgid.link/20260930-t9xx_driver_v1-v9-0-3248b9256956@compal.com
Changes in v9:
- Patch 1 (Add PCIe core):
- Dropped the priv->irq_type term from the mask/unmask/clear IRQ
guards; the field was write-only and made the probe-failure path
log a false "input irq_id=28" error
- mtk_pci_pldr() no longer requests an ACPI_ALLOCATE_BUFFER output
buffer from acpi_evaluate_object(): nothing read it and the
failure branch leaked it
- mtk_mhccif_exit() masks the MHCCIF source again after
cancel_work_sync(), so a work item already past the unregister
cannot leave remove() with an unmasked source and no callback
- pcim_enable_device() -> pci_enable_device(), with explicit
pci_clear_master()/pci_disable_device() in remove(): the devres
action touched config space after the ACPI reset
- The AER handler prints pci_channel_state_t with %u
- Deleted struct mtk_dev_ops and the mtk_dev_* inline wrappers;
every caller uses the mtk_pci_* implementation directly, and
mtk_dev_get_dev_cfg()/mtk_pci_get_dev_cfg() are gone (no callers)
- Single module: mtk_dev.c, pcie/Makefile and CONFIG_MTK_T9XX_PCI
are gone, driver init/exit live in pcie/mtk_pci.c, and every
EXPORT_SYMBOL_GPL is removed with the module boundary
- Removed the PCI dispatch-by-hw_ver table; there is one
implementation and it is called directly
- Declined a BAR length sanity check: the driver binds two fixed PCI
IDs and pcim_iomap_region() already fails probe for the
generically detectable cases
- Declined reordering the probe unwind to match remove(): probe
unwinds in reverse acquisition order, remove() deviates
deliberately and the reason is commented at the call site
- No delay added after the reset: MRST._RST is an ACPI method, so
acpi_evaluate_object() returns when the AML has finished, and it
is the last device access in remove()
- Removed the PCIe PM registers and the MHCCIF PM/DPMAIF/CLDMA
channel bits; nothing in this series reads them
- Removed the unused IRQ sources, MTK_USER_MIN/DATA, DEV_EVT_*_MAX,
ATR_DST_AXIM_1-3 and the <linux/debugfs.h> include
- Patch 2 (Add control plane transaction layer):
- Deleted struct mtk_ctrl_hif_ops and ctrl_blk->ops with the rest of
the single-implementation dispatch layers; mtk_ctrl_blk now holds
only mdev and trans
- Patch 3 (Add control DMA interface):
- Deleted struct cldma_drv_ops, drv_info->drv_ops,
cldma_drv_info_tbl[] and mtk_cldma_drv_m9xx.c/.h; the register
table stays as data and is assigned directly in dev_init()
- Deleted pcie/mtk_ctrl_cfg_m9xx.c: the queue, service and port
tables are file-scope statics in pcie/mtk_trans_ctrl.c and the
hw_ver lookup is gone
- REG_CLDMA_INT_WF_MASK is no longer introduced here; it had no
reader and Patch 5 used to remove it again
- Removed CLDMA_GPD_FLAG_BPS, enum mtk_ip_busy_src, DIR_MAX, unused
mtk_intr_type members, six reg_* fields and the *_done_cnt counters
- mtk_cldma_txq_alloc()/rxq_alloc() take the queue_info the caller
already resolved, dropping a repeated lookup that cannot fail
- Patch 4 (Add control port):
- Deleted the stale-list and device-id-IDA machinery instead of
hardening it: nothing here can reach the stale list and dev_id was
only handed back to ida_free(), so both reported call sites are gone
- mtk_port_tbl_destroy() uses the file's own radix-tree accessor
(rcu_dereference_raw()); radix_tree_deref_slot() splats under
CONFIG_PROVE_RCU in this lock-free teardown
- Added port_mngr->port_tbl_mtx around the radix insert/delete and
the port_cnt update, and struct mtk_port gained a rcu_head so
mtk_port_release() frees with kfree_rcu()
- Removed PORT_F_ALLOW_DROP (set, never read) and PORT_F_FORCE_SEND
(read, never set, so the term was provably zero)
- mtk_port_internal_disable() logs a failed mtk_port_ch_disable()
with dev_warn(); teardown still continues and PORT_S_ENABLE is
still cleared, or the port could never re-enable
- clear_bit(PORT_S_WR) in mtk_port_internal_disable() is followed by
wake_up_all(): the blocking writer loops back on timeout and
otherwise only noticed one trb timeout later
- mtk_ctrl_init() takes a struct mtk_port_layer_cfg * directly, the
struct mtk_ctrl_cfg wrapper is gone, and mtk_port.c calls
mtk_pcie_hif_submit_skb() instead of dispatching through ops
- Removed the Q_MTU_*/Q_FRAG_* sizes no queue uses, MTK_PEER_ID_SAP
and _MD, and PORT_S_DFLT/PORT_S_RD/PORT_S_STOP
- Patch 5 (Add FSM thread):
- mtk_pcie_hif_exit() stops the TRB service kthreads before
mtk_cldma_exit() frees the rings they walk; the two wake_up()
sites skip a NULL trb_srv so the reorder cannot oops instead
- One owner per direction on error: err_work sets txq->is_stopping
under ring_lock and mtk_cldma_start_xfer() rechecks it there, and
the RX side hands the queue to rx_done_work through need_restart
- mtk_cldma_rx_done_work() consumes need_restart before honouring
-ENXIO, so a queue left with no start address after a reset is
restarted instead of skipped
- The RX length is clamped against req->mtu (host state) instead of
the GPD's data_allow_len, and an oversized frame is dropped with
-EPROTO
- fsm->hif_err became a per-producer array and the recorder stores
unconditionally, so a successful handshake retry clears its own
slot instead of refusing FSM_STATE_READY forever
- The invalid dev_stage path of mtk_fsm_early_bootup_handler() uses
goto exit, re-arming the boot-flow channel that was masked on
entry
- Both CLDMA dma_pools are created with align 16; with align 4 the
24-byte descriptors were only 8-byte aligned every other block
- cldma_drv_info[] is published with smp_store_release() and read
with smp_load_acquire(); the removal sites and the txq/rxq slots
use WRITE_ONCE(), completing the pairing already used elsewhere
- struct runtime_feature_entry is __packed: offset advances by a
device-supplied length, so a later entry's data_len can land
unaligned. The layout is unchanged
- Removed MTK_UEVENT_MINIDUMP and MTK_UEVENT_LOWPOWER, declared with
no emitter anywhere in the series
- Commit message rewritten to cover the CLDMA scope: the ISR, the
QUEUE_ERROR worker and its -EPIPE completion policy, the
ring_leaked rule, the teardown changes and the FSM uevent payload
- Kept the uevent rather than reporting state through the wwan
framework: wwan exposes ports and netdevs, neither carries a
driver state machine, and adding one is a core change
- Declined splitting the CLDMA work into its own patch: the ISR is
registered by dev_init(), which only the FSM listener added here
calls, so either split order leaves a non-functional commit
- mtk_fsm.c calls mtk_pci_* directly and its six EXPORT_SYMBOL_GPL
are removed
- Removed FEATURE_TYPE_MUST/OPTIONAL, FSM_PRIO_0, MTK_UEVENT_UNDEF
and _MAX, DEV_CFG_NORMAL, QUERY_RTFT_ID_MAX and the TAG define
- Patch 6 (Add AT & MBIM WWAN ports):
- Dropped .tx_poll: it poll_wait()ed on a waitqueue inside struct
mtk_port, which wwan_remove_port() releases while the fd stays
open, and wake_up_pollfree() is not available to modules
- TX back-pressure now reaches poll() through wwan_port_txon() and
wwan_port_txoff() on the core's waitqueue, whose lifetime the open
file pins; this also removes the errno/queue-full poll confusion
- mtk_port_common_write() builds every CCCI packet onto a local
queue before submitting any, so an allocation or copy failure can
no longer truncate a message whose first packets are on the wire
- caps.frag_len is tx_mtu minus the CCCI header, so one core
fragment is exactly one CCCI packet instead of a full packet plus
a 16-byte runt
- w_lock now covers PORT_S_OPEN on the wwan ops: a receive that saw
the port open completes wwan_port_rx() before stop() returns and
the core purges its rx queue
- clear_bit(PORT_S_WR) in mtk_port_wwan_disable() is followed by
wake_up_all(), and a failed mtk_port_ch_disable() is logged with
dev_warn() as in Patch 4
- Commit message corrected: init() only prepares the port object,
wwan_create_port() is in enable(), which FSM_STATE_READY reaches
through the new mtk_port_enable_by_type(PORT_TBL_MD) hook
- w_port is still published after PORT_S_ENABLE/PORT_S_WR: it is
wwan_create_port()'s return value and the node becomes openable
inside that call, so a comment now records the -ENXIO window
- The AT/MBIM port and queue rows are added directly to the
file-scope tables in pcie/mtk_trans_ctrl.c, and mtk_port_io.c
calls mtk_pcie_hif_cmd_func() directly
- Link to v8: https://patch.msgid.link/20260914-t9xx_driver_v1-v8-0-5206c2e6bea0@compal.com
Changes in v8:
- Patch 1 (Add PCIe core):
- Commit message: reworded the MHCCIF bullet (the driver masks all
channels at init and does not clear pending status) and added a
paragraph on the ACPI reset performed on removal
- Added irq_cb_lock around IRQ callback register/unregister;
unregister now masks the vector and calls synchronize_irq() before
clearing the slot, so no callback runs after it returns
- mtk_pci_pldr() evaluates MRST._RST on the device's own ACPI node
instead of PXP._OFF/_ON on the upstream bridge: the bridge power
resource is shared and its 500 ms off/on induced a link down/up
that a hotplug-capable port reports as a remove/add
- Removed mtk_pci_fldr(), enum mtk_reset_type, mtk_pci_dev_reset()
and mtk_pci_reset(): dead here, they belong with the devlink
firmware-update series that will use them
- MHCCIF interrupts are now acked before dispatch, not after;
mtk_mhccif_init() masks every channel but deliberately does not
ack, as the status is one-shot latched by the device
- Removed the write-only bar[] array and MTK_PCI_BAR_NUM
- Switched to pcim_iomap_region(), replacing the deprecated
pcim_iomap_regions()/pcim_iomap_table() pair
- Dropped the BIT(30) exemption from the all-ones MMIO check; a
stray bit is now logged with dev_err_ratelimited() and cleared
- probe() masks all MHCCIF channels unconditionally; removed the
dead configuration plumbing that used to select the mask
- Check pci_save_state() and return -ENOMEM instead of -EFAULT
- Reordered remove(): dev_exit, mask all 32 channels, mhccif_exit,
free_irq, clear bus master, PLDR reset last
- Removed the unused enum mtk_atr_type and the .type field
- Patch 2 (Add control plane transaction layer):
- Converted the remaining EXPORT_SYMBOL() to EXPORT_SYMBOL_GPL()
(mtk_ctrl_init/exit here, mtk_port_trb_free in Patch 4,
mtk_fsm_start/evt_submit/init/exit in Patch 5) and deleted the
unused mtk_fsm_notifier_register/unregister exports
- Patch 3 (Add control DMA interface):
- Clamp the RX length reported by the GPD against data_allow_len and
drop the packet with -EPROTO instead of trusting the device
- Publish txq[]/rxq[] with smp_store_release(); the readers added in
Patch 5 pair with smp_load_acquire()
- mtk_cldma_stop_queue() returns -ENODEV on an all-ones read,
escalates to cldma_drv_reset() on timeout and only leaks the ring
if the queue still refuses to stop
- Added NULL checks to all six radix_tree_lookup() call sites
- mtk_cldma_rxq_free() splits ownership three ways (host-owned,
device-owned, in-flight) instead of unmapping everything
- New mtk_cldma_hw_recovery(); mtk_cldma_start_xfer() rewritten to
run entirely under ring_lock so a resume cannot race a submit
- Populate the control-queue entries of the queue info table here
instead of in Patch 4, keeping each commit self-consistent
- srv_cfg is const int (*)[HW_QUE_NUM] instead of int **, dropping
the type-punning cast
- Use skb_queue_len_lockless() where the list lock is not held
- Hold the skb_list lock across every list read; the one place it
must be dropped (sleeping submit) is documented
- Check mtk_cldma_trb_process() return and add
mtk_cldma_txq_flush() to complete pending requests on error
- Initialise submit_lock and trans->available in
mtk_trans_ctrl_init(), with a NULL srv guard
- mtk_pcie_hif_exit() order: cldma_exit, then trb_srv_exit, then
remove the radix tree
- Evaluate CHECK_TX_FULL under submit_lock
- Removed dead code found while auditing the above
- Reject a non-linear skb with -EINVAL instead of WARN_ON_ONCE()
- Commit message: probe only registers the transport plane; the
CLDMA instances and the TRB service thread are started by the
later "Add FSM thread" patch
- A stop timeout in mtk_cldma_txq_free()/rxq_free() resets the whole
IP, so the new mtk_cldma_rearm_queues() re-arms every queue still
published on that instance, not just the one being freed
- The RX re-arm is handed to mtk_cldma_rx_done_work() through a
need_restart flag; it owns free_idx, which the re-arm used to read
without synchronisation
- mtk_cldma_submit_tx() rejects a NULL dev, closing the window
between mtk_cldma_exit() clearing trans->dev and the TRB service
thread being stopped
- Patch 5 (Add FSM thread):
- New mtk_fsm_hif_err_record(): a transport init failure is sticky,
so the STARTUP handler refuses the BOOTUP->READY promotion instead
of reaching READY on a dead transport; DEV_ADD clears it
- A feature the device reports as NOT_EXIST/NOT_SUPPORT now fails
the handshake with -EPROTO when the host marks it MUST_SUPPORT
- Both ctrl-msg handlers consume the skb on every path and return 0;
the caller no longer frees behind the handler's back
- The duplicate-HS2 and invalid-id paths no longer touch rt_data, so
an epilogue cannot free data owned by a queued STARTUP event
- mtk_fsm_idle_evt_handler() returns an error on a failed submit and
unmasks its handshake channels only on success;
mtk_fsm_early_bootup_handler() latches last_dev_state only after
the handler chain succeeded, so a failure is retried
- Re-arm BOOT_FLOW_SYNC in mtk_fsm_early_bootup_handler(): the
channel was masked on entry and never unmasked, so a device still
booting when the driver binds never delivered its DEV_STAGE_IDLE
notification and the FSM stayed at FSM_STATE_ON with no ports
- The in-flight runtime-data skb is owned by the STARTUP event it
was submitted with and freed when that event completes; rt_data is
demoted to a duplicate-HS2 guard and rt_data_len removed
- mtk_fsm_init() unwinds through explicit labels and no longer leaks
the kthread on any error path
- mtk_fsm_exit() calls mtk_fsm_ctrl_ch_stop(), releasing the control
ports while the port table they reference is still alive
- Added the 0x0900 entry to cldma_drv_info_tbl[]; the primary
MediaTek PCI ID no longer fails CLDMA dev_init with -EIO
- QUEUE_ERROR is now cleared, re-unmasked and handled by a new
err_work that stops the errored queues and completes their pending
requests with -EPIPE, instead of being logged against a masked
interrupt that is never re-armed
- mtk_cldma_dev_exit() masks the L1 interrupt and calls
synchronize_irq() before unregistering the callback
- mtk_cldma_dev_exit() quiesces the IP (interrupt output disabled,
cldma_drv_reset()) before releasing descriptor memory, and leaks
the DMA pools as well if a ring was already leaked
- New mtk_fsm_stop(), called by mtk_pci_dev_exit() after a possibly
failed DEV_RM and before mtk_trans_ctrl_exit(), so the FSM thread
cannot run on a dismantled transport plane; mtk_pcie_hif_exit() is
idempotent, mtk_cldma_exit() latches trans->dev, and the FSM
kthread is held with get_task_struct() so a thread that died
abnormally cannot be stopped through a freed task_struct
- mtk_cldma_txq_free()/rxq_free() flush err_work as well: it may
already hold the queue they are about to free
- err_work no longer completes and unmaps the requests of a queue
that failed to stop; the device may still be reading them
- The rt_data duplicate-HS2 guard is accessed with READ_ONCE() and
WRITE_ONCE(), the rx path and the FSM kthread being concurrent
- Patch 6 (Add AT & MBIM WWAN ports):
- Commit message: note the copy path now uses skb_copy_bits()
- Removed union user_buf and mtk_port_copy_data_from(); the
kernel-buffer branch had no user
- Removed mtk_port_common_write_frag_skb() and the dead
scatter-gather gate it was reached through
- Fixed a heap overread: the TX path copied from a paged skb as if
it were linear; use skb_copy_bits()
- mtk_port_send_data() takes explicit blocking/force_send arguments,
no longer mutates the caller's flags, and is all-or-nothing
- The AT and MBIM TX callbacks are thin wrappers over a shared
mtk_port_wwan_tx()
- A port DISABLE is inserted ahead of pending ENABLEs (Patch 3) and
mtk_port_internal_enable() unwinds on failure (Patch 4), so a
close cannot be starved by a queued enable
- Declined the reordering suggested in review, with a comment
explaining the ordering constraint; w_port is read inside w_lock
- wwan_port_rx() is called only when PORT_S_OPEN is set; otherwise
the packet is dropped with -ENXIO and dev_dbg_ratelimited()
- Link to v7: https://patch.msgid.link/20260828-t9xx_driver_v1-v7-0-bf8f6074d88a@compal.com
Changes in v7:
- Cover letter: renamed "Core logic" to "Device lifecycle management".
The FSM is T9xx-specific (same model as the in-tree t7xx driver) and
stays in the driver rather than the WWAN framework.
- Patch 1 (Add PCIe core):
- Removed RGU wording from commit message, no RGU code in the series
- Removed `select NET_DEVLINK` from Kconfig, no devlink consumer
- Reordered ATR setup: program TRSL_ADDR/TRSL_PARAM before enabling
ATR in SRC_ADDR_LSB, with a readback, matching t7xx
- Adopted the t7xx IRQ model: require exactly MTK_IRQ_CNT_MAX MSI-X
vectors with one handler per vector; removed the merged-IRQ mode
and scoped dispatch to each vector's own status bits
- Fixed IRQ callback publish ordering: WRITE_ONCE + smp_wmb on
register, READ_ONCE + smp_rmb in the interrupt handler
- Added hw_bits == 0 guard in mtk_pci_send_ext_evt() to reject
unmapped event channels
- MHCCIF callbacks stay under spin_lock_bh by design; added a comment
documenting the must-not-sleep callback contract
- Added a drain loop in mtk_mhccif_exit() to free any remaining
callback nodes
- Moved cancel_work_sync(&priv->mhccif_work) before mtk_pci_pldr()
in remove, preventing worker MMIO during the power cycle
- Squashed the MAINTAINERS entry into this patch (was Patch 7); the
series is now 6 patches
- Patch 2 (Add control plane transaction layer):
- Commit message and Kconfig help text now name both modules
(mtk_t9xx and mtk_t9xx_pcie)
- Reworded mtk_ctrl_exit() kernel-doc: the allocation itself is
devres-managed and freed on driver detach
- Patch 3 (Add control DMA interface):
- RX re-map failure now keeps the buffer in its slot instead of
freeing it and advancing free_idx, preventing a permanent ring
stall from a stale GPD DMA address
- mtk_cldma_stop_queue() returns -ETIMEDOUT on poll timeout; alloc
paths abort on failure, free paths warn and proceed
- Added READ_ONCE() for the free_idx and tx_started reads in
mtk_cldma_start_xfer()
- mtk_cldma_get_tx_start_addr() reads ADDRL+ADDRH as u64, so a >4GB
DMA address is not mistaken for "queue not started"
- mtk_cldma_open() decrements usr_cnt on failure. Beyond the emailed
reply: mtk_pcie_hif_init() re-zeroes usr_cnt and ENABLE/DISABLE no
longer gate on TX budget, so a channel reopens after devlink reload
- mtk_cldma_close() completes the TRB with -EPIPE when drv_info is
already gone, instead of stalling the submitter until timeout
- mtk_cldma_txbuf_set() returns -ENOMEM instead of -EAGAIN on
dma_map failure (no infinite retry); dev_err_ratelimited
- Added the 0x0900 (MTK reference) entry to mtk_ctrl_info_tbl[];
both PCI IDs are now present in all dispatch tables
- Documented the single-consumer assumption of
mtk_ctrl_trb_handler() in a comment
- Added a submit_lock mutex closing the submit vs. hif_exit race on
trans->available; hif_cmd_func() now checks available
- Kept the empty queue table: queue_info entries are populated by
the next patch "Add control port"
- Commit message notes TX/RX paths are wired up by later patches
- hif_exit has no caller at this commit, but ops->init is not called
either, so nothing is allocated to leak; patch 5 wires up both
- The trb_handler default case completes the TRB with -EINVAL
instead of silently leaking the SKB
- Documented the non-BD skb_headlen() == skb->len invariant
- trb_open_priv overlay size is safe by construction
- Removed the unused CLDMA4 support here at the source, instead of
introducing it and deleting it again in Patch 5
- Fixed an skb leak in the mtk_cldma_rxq_free() BD path: the parent
skb was never freed after detaching frag_list
- Replaced the free_idx/wr_idx boundary check added in v5 with a
per-queue ring_lock following drivers/net/wwan/t7xx: the check
raced with submit_tx and could stall TX forever; the lock also
closes the submit/reclaim race the check was originally added for
- Removed the write-only queues_cnt and tx_req.data_vm_addr fields
- Patch 5 (Add FSM thread):
- Commit message: five FSM states (not seven); now describes the
HS1/HS2/HS3 handshake and CLDMA bring-up/teardown in detail
- Check ops->init() return in FSM_STATE_ON; NULL guard for srv in
trb_srv_exit(); check mtk_cldma_dev_init() return in the listener
- Kept the HS2 feature parser as is: the protocol requires dense
ascending order and the ft_id check catches violations
- Reject data_len == 0 before calling the query_rtft handler
- Drop duplicate HS2 messages when hs_info->rt_data is already set
- Added a notifier_lock mutex protecting FSM notifier list walks
- Reordered HS2 handling: parse HS2, send HS3, and only then switch
state, so FSM_STATE_READY implies HS3 was delivered
- Unmask the HS channels on handshake error so the modem can retry
- Moved wake_up_process() under evtq_lock with a fsm_handler NULL
check; exit sets GATECLOSED and clears fsm_handler under the lock
- mtk_fsm_hs_info_init() propagates event-registration failures and
unwinds on error
- The CLDMA ISR logs, clears and re-unmasks QUEUE_ERROR bits
- CLDMA dev_exit unregisters the IRQ callback before mask +
synchronize_irq, preventing ISR re-arm after teardown
- Reordered FSM_STATE_OFF teardown: stop the TRB threads before
freeing CLDMA, with the per-hif teardown moved into
mtk_cldma_exit(); fixes a silently skipped teardown that broke the
next reset cycle and leaked the CLDMA resources
- mtk_trans_ctrl_exit() falls back to hif_exit when the
FSM_STATE_OFF teardown never ran (guarded by trans->available)
- mtk_pci_dev_start() propagates evt_submit/fsm_start errors
- Restored TASK_INTERRUPTIBLE for the FSM event kthread: the v2 flip
to TASK_UNINTERRUPTIBLE was not requested by review, and an idle
D-state kthread blocks suspend and trips the hung-task detector
- Patch 6 (Add AT & MBIM WWAN ports):
- Check and log mtk_port_enable_by_type() failure in the FSM state
handler
- A short WWAN write now returns -EIO; consume_skb() on success
- The wwan_enable failure path calls mtk_port_ch_disable() to undo
usr_cnt and partial resources
- Publish the wwan_port pointer only after IS_ERR validation, so the
RX path never sees an ERR_PTR
- Pass wwan_port_caps (frag_len = negotiated tx_mtu, CCCI header
headroom) to wwan_create_port(); a tx_mtu == 0 guard prevents
userspace write() from spinning forever on frag_len == 0
- Set PORT_S_WR/PORT_S_ENABLE before wwan_create_port() and clear
both bits on failure
- Link to v6: https://patch.msgid.link/20260811-t9xx_driver_v1-v6-0-2c969fad57c6@compal.com
Changes in v6:
- Patch 3 (Add control DMA interface):
- Move extern declarations for mtk_cldma_regs_m9xx and
cldma_drv_ops_m9xx from Patch 5 to Patch 3 where the
symbols are first defined, fixing sparse W=1 C=1 warnings
- Fix RX done recycle path: skip buffer recycle in BD mode
(nr_bds > 0) to prevent BDP flag mismatch causing HW to
misinterpret data buffer address as BD chain pointer
[sashiko]
- Fix RX done recycle path: on dma_map_single failure,
free skb and advance free_idx instead of goto out, to
prevent ring stall [sashiko]
- Patch 4 (Add control port):
- Replace wake_up_interruptible_all() with wake_up_all()
for trb_wq and rx_wq in open/close/tx completion
callbacks and common_close, fixing mismatch with
wait_event_timeout (TASK_UNINTERRUPTIBLE) waiters in
ch_enable/ch_disable that caused spurious timeouts
[sashiko]
- Link to v5: https://patch.msgid.link/20260723-t9xx_driver_v1-v5-0-b27afb99ccbb@compal.com
Changes in v5:
- Patch 1 (Add PCIe core):
- Remove LE32_TO_U32(cpu_to_le32()) no-op endianness
roundtrip, return hw_bits directly
- Use "%s" format string in pci_request_irq() to prevent
format string injection
- Return PCI_ERS_RESULT_DISCONNECT from AER error_detected,
driver does not support AER recovery
- Patch 2 (Add control plane transaction layer):
- Rewrite commit message to accurately describe skeleton
content
- Patch 3 (Add control DMA interface):
- Replace rmb() with dma_rmb() after HWO check in TX/RX
done paths for correct DMA descriptor ordering
- Add ring boundary check in TX done to prevent crossing
into submit territory
- Fix RX stall on reload failure: recycle old buffer with
skb_trim() + dma_map_single() instead of losing descriptor
- Convert devm_kcalloc to kcalloc for req_pool and
bd_dsc_pool, matching existing kfree in teardown
- Set bd_dsc->skb = NULL after dev_kfree_skb_any() in
reload error path to prevent use-after-free
- Use skb_headlen(skb) instead of skb->len for linear
segment size in txbuf_set non-BD path
- Use WRITE_ONCE() for wr_idx update in submit_tx
- Patch 4 (Add control port):
- Fix direct mtk_port_release() calls to use kref_put()
in mtk_port_free_or_backup and stale_list_grp_cleanup
- Add ida_free() before kfree(s_list) in stale list
cleanup to prevent IDA leak
- Remove double-free of skb in mtk_port_internal_recv()
drop_data path, caller owns the skb
- Add port_cnt bounds check against data_len in
mtk_port_status_update() to prevent OOB read
- Replace wait_event_interruptible_timeout with
wait_event_timeout in ch_enable/ch_disable to prevent
busy-spin on pending signals
- Patch 5 (Add FSM thread):
- Validate rtft data_len before passing to query_rtft
callback to prevent OOB access
- Zero-initialize HS3 skb data with memset() to prevent
heap infoleak
- Set hs_info->rt_data = NULL after dev_kfree_skb() in
SAP/MD ctrl msg handlers to prevent use-after-free
- Move wake_up_process() inside spinlock in evt_submit,
clear fsm_handler under lock in exit to fix race
- Replace kcalloc + radix_tree_gang_lookup with
radix_tree_for_each_slot() in mtk_port_disable()
- Add rt_data cleanup loop in FSM teardown path
- Patch 7 (Add maintainers entry):
- Add Robert Yu and Jeff Chang as maintainers
- Link to v4: https://patch.msgid.link/20260709-t9xx_driver_v1-v4-0-a8c009d509c5@compal.com
Changes in v4:
- Patch 1 (Add PCIe core):
- Add ACPI dependency to Kconfig (depends on PCI && ACPI)
and remove #ifdef CONFIG_ACPI / #else guards from
mtk_pci_fldr() and mtk_pci_pldr()
- Remove unnecessary parentheses in (mdev)->dev across
all dev_err/dev_warn calls
- Remove devm_kfree() calls from mtk_pci_probe() error
path and mtk_pci_remove(), devres handles cleanup
- Remove mtk_dev_alloc()/mtk_dev_free() wrappers, use
devm_kzalloc() directly in mtk_pci_probe()
- Convert ext_evt callbacks from devm_kzalloc/devm_kfree
to kzalloc/kfree for runtime-managed resources
- Patch 2 (Add control plane transaction layer):
- Remove mtk_dev_alloc()/mtk_dev_free() declarations and
implementations from mtk_dev.h and mtk_dev.c
- Remove empty module_init/module_exit stubs, defer to
Patch 4 where they have actual content
- Remove devm_kfree() and unused ctrl_blk variable in
mtk_ctrl_exit(), devres handles cleanup
- Patch 3 (Add control DMA interface):
- Remove inline keyword from mtk_cldma_clr_bd_dsc() in
.c file, let compiler decide inlining
- Replace dev_warn() with dev_warn_ratelimited() for SKB
alloc/map failure messages in data path
- Replace open-coded do/while polling loop with
read_poll_timeout() from <linux/iopoll.h> in
mtk_cldma_stop_queue()
- Remove ctrl_port_chl_mtu module_param, MODULE_PARM_DESC,
and mtk_ctrl_queue_info_update() with associated macros
- Remove unnecessary parentheses in (mdev)->dev across
mtk_cldma.c and mtk_trans_ctrl.c
- Convert CLDMA txq/rxq/cd from devm_kzalloc/devm_kfree
to kzalloc/kfree for runtime-managed resources
- Convert trb_srv/srv_que from devm_kzalloc/devm_kfree
to kzalloc/kfree for runtime-managed resources
- Remove devm_kfree() and unused variables from
mtk_cldma_exit() and mtk_trans_ctrl_exit(), devres
handles cleanup for device-lifetime resources
- Remove unnecessary NULL checks before kfree() for
bd_dsc_pool in txq_free/rxq_free
- Patch 4 (Add control port):
- Add module_init/module_exit with mtk_port_io_init/exit
content, moved from Patch 2
- Remove unnecessary parentheses in (mdev)->dev in
mtk_port.c
- Remove devm_kfree() from mtk_port_mngr_init() error
path and mtk_port_mngr_exit(), devres handles cleanup
- Simplify mtk_trans_ctrl_init() error path: replace goto
err_free_trans with direct return -ENOMEM
- Patch 5 (Add FSM thread):
- Remove unnecessary parentheses in (mdev)->dev across
mtk_fsm.c and mtk_ctrl_plane.c
- Convert FSM notifiers from devm_kzalloc/devm_kfree to
kzalloc/kfree for runtime-managed resources
- Convert CLDMA drv_info from devm_kzalloc/devm_kfree to
kzalloc/kfree for runtime-managed resources
- Remove devm_kfree() from mtk_fsm_init()/mtk_fsm_exit()
and mtk_ctrl_init()/mtk_ctrl_exit(), devres handles
cleanup for device-lifetime resources
- Link to v3: https://patch.msgid.link/20260624-t9xx_driver_v1-v3-0-73ff03f60c48@compal.com
Changes in v3:
- Address sashiko AI code review comments and fix sparse warnings
- Patch 1 (Add PCIe core):
- Move extern declaration of mtk_dev_cfg_0900 from mtk_pci.c to mtk_pci.h to fix sparse warning
- Remove mtk_pci_bar_exit(): pcim_iounmap_region() was called with bitmask instead of BAR index, and pcim_iomap_regions() is devres-managed so manual unmap is redundant [sashiko]
- Remove pci_disable_device() from probe error path and remove path: pcim_enable_device() registers devres cleanup, manual disable causes enable_cnt underflow [sashiko]
- Patch 3 (Add control DMA interface):
- Add #include "mtk_cldma.h" in mtk_cldma_drv_m9xx.c to fix sparse undeclared symbol warnings for mtk_cldma_regs_m9xx and cldma_drv_ops_m9xx
- Move extern declaration of mtk_ctrl_info_m9xx from mtk_trans_ctrl.c to mtk_trans_ctrl.h to fix sparse undeclared symbol warning
- Replace kcalloc + radix_tree_gang_lookup with radix_tree_for_each_slot() in mtk_ctrl_remove_radix_tree() to eliminate allocation in teardown path [sashiko]
- Add missing mtk_pci_dev_exit() call in mtk_pci_remove() to properly clean up FSM and trans_ctrl resources before device removal [sashiko]
- Patch 4 (Add control port):
- Replace kcalloc + radix_tree_gang_lookup with radix_tree_for_each_slot() and single-entry gang_lookup in mtk_port_tbl_destroy() to eliminate allocation failure in teardown path [sashiko]
- Add port = NULL after mtk_port_put_locked() in mtk_port_internal_open() error path to prevent returning un-refcounted pointer [sashiko]
- Restore skb_unlink + trb_complete for non-EAGAIN errors in mtk_ctrl_trb_handler() TX path to prevent infinite retry loop [sashiko]
- Patch 5 (Add FSM thread):
- Check mtk_pci_register_irq() return value in mtk_cldma_dev_init() and add err_destroy_wq error label to fix unreachable dead code [sashiko]
- Propagate specific error codes (ENOMEM/EIO/EINVAL) from mtk_cldma_dev_init() error paths instead of generic -EIO [sashiko]
- Free head SKB after detaching frag_list in mtk_cldma_rxq_free() scatter-gather mode to fix memory leak [sashiko]
- Patch 6 (Add AT & MBIM WWAN ports):
- Remove unused write_lock mutex from struct mtk_port and its mutex_init call [sashiko]
- Link to v2: https://patch.msgid.link/20260610-t9xx_driver_v1-v2-0-c65addf23b3f@compal.com
Changes in v2:
- Split series into control plane (this v2) and data plane (follow-up)
- Patch 1 (Add PCIe core):
- Rename BAR_NUM to MTK_PCI_BAR_NUM for driver prefix consistency
- Replace magic numbers in mtk_pci_setup_atr() with named defines
- Remove redundant ATR register comments, use blank line separators
- Add kernel-doc comments to all non-static functions
- Convert 4 MMIO wrapper functions to static inline in header [sashiko]
- Remove unnecessary unlikely() from IRQ validation paths
- Add irq_cnt == 0 and irq_id < 0 guards in mtk_pci_get_virq_id() [sashiko]
- Initialize hw_bits at declaration for consistency
- Merge same-type variable declarations into single lines
- Add #else/#endif comments for CONFIG_ACPI blocks
- Add newlines in mtk_pci_pldr() for readability
- Move return into default case in mtk_pci_dev_reset()
- Simplify mtk_mhccif_init() error path to use direct returns
- Change -EFAULT to -ENOLINK for PCIe link check failure
- Rename goto label "out" to "log_err" in mtk_pci_probe()
- Wrap long lines to stay within 80 columns
- Fix IRQ vector leak: add pci_free_irq_vectors() on error path [sashiko]
- Fix mtk_pci_remove() ordering: free IRQ before cancel_work_sync [sashiko]
- Fix mtk_pci_pldr() ACPI buffer leak: free first result before second call [sashiko]
- Replace msleep(500) with MTK_PLDR_POWER_OFF_DELAY_MS define
- Remove unused EXT_EVT_H2D_DRM_DISABLE_AP and related register define [sashiko]
- Increase MTK_IRQ_NAME_LEN from 20 to 32 to fix W=1 format-truncation warning [sashiko]
- Patch 2 (Add control plane transaction layer):
- Add kernel-doc comments to mtk_ctrl_init() and mtk_ctrl_exit()
- Change mtk_ctrl_exit() return type from int to void
- Set mdev->ctrl_blk to NULL after freeing in mtk_ctrl_exit() [sashiko]
- Change ctrl_blk from void* to typed struct mtk_ctrl_blk* [sashiko]
- Remove redundant "depends on MTK_T9XX" from MTK_T9XX_PCI Kconfig [sashiko]
- Use mtk_dev_free() instead of devm_kfree() in mtk_pci_probe() error path [sashiko]
- Patch 3 (Add control DMA interface):
- Add @ops kernel-doc parameter for mtk_ctrl_init()
- Rename 'err' to 'ret' consistently throughout the patch
- Reorder variable declarations to follow reverse Christmas tree style
- Change mtk_cldma_txq_free() return type from int to void
- Change mtk_cldma_rxq_free() return type from int to void
- Change mtk_cldma_exit() return type from int to void
- Remove unnecessary zero-initialization of ret in mtk_cldma_start_xfer()
- Remove unnecessary zero-initialization of ret in mtk_cldma_tx()
- Use direct return instead of goto out in mtk_cldma_submit_tx() error paths
- Move software state before HWO flag in mtk_cldma_submit_tx()
- Squash variable declarations in mtk_cldma_check_intr_status()
- Remove unlikely() from validation paths in mtk_cldma_check_ch_cfg()
- Clamp data_recv_len with min_t to prevent skb_over_panic in mtk_cldma_rx_skb_adjust() [sashiko]
- Use READ_ONCE() for HWO flag polling in mtk_cldma_check_rx_req() [sashiko]
- Fix mtk_cldma_rx_done_work() to always unmask interrupt on error path [sashiko]
- Add DMA address guard in mtk_cldma_txq_free() teardown loop [sashiko]
- Add IS_ERR() check for kthread_run() in mtk_ctrl_trb_srv_init() [sashiko]
- Fix queue_info memory leak on validation failure in mtk_pcie_hif_init() [sashiko]
- Handle non-EAGAIN errors in mtk_ctrl_trb_handler() TX path [sashiko]
- Fix 'err' typo to 'ret' in mtk_cldma_txbuf_set() error message
- Remove unused variable mdev in mtk_cldma_rx_check_again() [sashiko]
- Remove unused variables trans and ctrl_blk in mtk_cldma_txq_free() and mtk_cldma_rxq_free() [sashiko]
- Patch 4 (Add control port):
- Add @cfg kernel-doc parameter for mtk_ctrl_init()
- Update mtk_ctrl_init() return description to cover additional error codes
- Fix double list_del in mtk_port_stale_list_grp_cleanup() [sashiko]
- Fix direct mtk_port_trb_free() call to use kref_put() in mtk_port_ch_enable() error path [sashiko]
- Fix direct mtk_port_trb_free() call to use kref_put() in mtk_port_ch_disable() error path [sashiko]
- Add mtk_port_tbl_destroy() in mtk_port_mngr_init() error path to prevent port memory leak [sashiko]
- Change port_ops exit/reset/enable/disable callbacks from int to void
- Move -EIO dispatch comment to where the code was introduced
- Patch 5 (Add FSM thread):
- Add bounds check for rtft_entry in mtk_fsm_parse_hs2_msg() [sashiko]
- Add skb length validation before accessing ctrl_msg_header in mtk_fsm_sap_ctrl_msg_handler() [sashiko]
- Fix skb leak on CTRL_MSG_HS2 mismatch return in mtk_fsm_sap_ctrl_msg_handler() [sashiko]
- Add skb length validation before accessing ctrl_msg_header in mtk_fsm_md_ctrl_msg_handler() [sashiko]
- Replace devm_kzalloc/devm_kfree with kzalloc/kfree for FSM events [sashiko]
- Fix mtk_fsm_evt_submit() to return -ETIMEDOUT on blocking event timeout [sashiko]
- Change FSM kthread from TASK_INTERRUPTIBLE to TASK_UNINTERRUPTIBLE [sashiko]
- Remove unused variable hw_id in mtk_cldma_dev_exit() [sashiko]
- Patch 6 (Add AT & MBIM WWAN ports):
- Use imperative mode in commit message
- Remove unnecessary zero-initialization of ret in mtk_port_copy_data_from()
- Change copy_from_user() error code from -EFAULT to -EINVAL in mtk_port_copy_data_from()
- Return -EINVAL for zero-length write in mtk_port_common_write()
- Change mtk_port_wwan_exit/enable/disable() return type from int to void
- Fix packet_size to account for CCCI header reservation in mtk_port_common_write() [sashiko]
- Fix WWAN tx callbacks to consume skb and return 0 per wwan_port_ops contract [sashiko]
- Fix wwan_create_port() error path: clear ERR_PTR to NULL and call mtk_port_ch_disable() [sashiko]
- Patch 7 (Add maintainers entry): new patch
- Link to v1: https://patch.msgid.link/20260529-t9xx_driver_v1-v1-0-bdbfe2c01e57@compal.com
To: Loic Poulain <loic.poulain@oss.qualcomm.com>
To: Sergey Ryazanov <ryazanov.s.a@gmail.com>
To: Johannes Berg <johannes@sipsolutions.net>
To: Andrew Lunn <andrew+netdev@lunn.ch>
To: "David S. Miller" <davem@davemloft.net>
To: Eric Dumazet <edumazet@google.com>
To: Jakub Kicinski <kuba@kernel.org>
To: Paolo Abeni <pabeni@redhat.com>
To: Jack Wu <jackbb_wu@compal.com>
To: Robert Yu <robert_yu@compal.com>
To: Jeff Chang <Jeff_Chang@compal.com>
To: Wen-Zhi Huang <wen-zhi.huang@mediatek.com>
To: Shi-Wei Yeh <shi-wei.yeh@mediatek.com>
To: Minano Tseng <Minano.tseng@mediatek.com>
To: Matthias Brugger <matthias.bgg@gmail.com>
To: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
Cc: linux-kernel@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-mediatek@lists.infradead.org
---
Jack Wu (6):
net: wwan: t9xx: Add PCIe core
net: wwan: t9xx: Add control plane transaction layer framework
net: wwan: t9xx: Add control DMA interface
net: wwan: t9xx: Add control port
net: wwan: t9xx: Add FSM thread
net: wwan: t9xx: Add AT & MBIM WWAN ports
MAINTAINERS | 11 +
drivers/net/wwan/Kconfig | 11 +
drivers/net/wwan/Makefile | 1 +
drivers/net/wwan/t9xx/Makefile | 16 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 111 ++
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 82 ++
drivers/net/wwan/t9xx/mtk_dev.h | 43 +
drivers/net/wwan/t9xx/mtk_fsm.c | 1085 +++++++++++++++++
drivers/net/wwan/t9xx/mtk_fsm.h | 153 +++
drivers/net/wwan/t9xx/mtk_port.c | 762 ++++++++++++
drivers/net/wwan/t9xx/mtk_port.h | 149 +++
drivers/net/wwan/t9xx/mtk_port_io.c | 642 ++++++++++
drivers/net/wwan/t9xx/mtk_port_io.h | 33 +
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 1689 +++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 128 ++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c | 505 ++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 118 ++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1106 ++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 139 +++
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 35 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 673 +++++++++++
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 88 ++
22 files changed, 7580 insertions(+)
---
base-commit: eb3f4b7426cfd2b79d65b7d37155480b32259a11
change-id: 20260529-t9xx_driver_v1-1744f8af7739
Best regards,
--
Jack Wu <jackbb_wu@compal.com>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 2/6] net: wwan: t9xx: Add control plane transaction layer framework Jack Wu via B4 Relay
` (4 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
Registers the T900 device driver with the kernel. Set up all
the fundamental configurations for the device: PCIe layer,
Modem Host Cross Core Interface (MHCCIF), modem common control
operations, build infrastructure and MAINTAINERS entry.
Besides the T900 itself (14c3:0900) the driver binds its
HP-branded variant, the HP RMM-G4 (03f0:01ca); both IDs are
exported via MODULE_DEVICE_TABLE for module autoloading.
* PCIe layer code implements driver probe and removal, MSI-X
interrupt initialization and de-initialization, and the way
of resetting the device.
* MHCCIF provides the interrupt channels used for the boot
handshake and for the host-to-device reset doorbell.
* Modem common control operations provide the basic read/write
functions of the device's hardware registers,
mask/unmask/get/clear functions of the device's interrupt
registers and inquiry functions of the device's status.
* Add MAINTAINERS entry for the MediaTek T9XX 5G WWAN modem
device driver.
Removal resets the modem, because the next driver load has to
redo the device handshake and status sync and cannot do that
against firmware left running by the previous load. Shutdown
runs the same teardown and reset, because a kexec reboot gets no
platform reset and the next kernel would otherwise find that
firmware still running; a driver cannot tell kexec from any other
reboot or power-off, so every shutdown takes this path and waits
up to 1 s for the link after MRST._RST, or 2 s after the doorbell
below.
mtk_pci_remove() evaluates MRST._RST on the device's own ACPI
node: that is a platform-level reset and does not depend on the
firmware's state. Where the platform has no MRST._RST, or the
method fails, removal falls back to the MHCCIF
DEV_EVT_H2D_DEVICE_RESET doorbell if the link is still up. The
doorbell is a request to the firmware, so it is best effort and
cannot recover firmware that has stopped responding - the case
the reset exists for. Probe therefore does not require
MRST._RST; without it the driver still binds and only removal is
degraded.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
MAINTAINERS | 11 +
drivers/net/wwan/Kconfig | 11 +
drivers/net/wwan/Makefile | 1 +
drivers/net/wwan/t9xx/Makefile | 9 +
drivers/net/wwan/t9xx/mtk_dev.h | 39 ++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1044 ++++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_pci.h | 139 ++++
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 34 +
8 files changed, 1288 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 461a3eed6129..8f13ef7440ce 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16494,6 +16494,17 @@ L: netdev@vger.kernel.org
S: Supported
F: drivers/net/wwan/t7xx/
+MEDIATEK T9XX 5G WWAN MODEM DRIVER
+M: Jack Wu <jackbb_wu@compal.com>
+M: Robert Yu <robert_yu@compal.com>
+M: Jeff Chang <Jeff_Chang@compal.com>
+R: Wen-Zhi Huang <wen-zhi.huang@mediatek.com>
+R: Shi-Wei Yeh <shi-wei.yeh@mediatek.com>
+R: Minano Tseng <Minano.tseng@mediatek.com>
+L: netdev@vger.kernel.org
+S: Supported
+F: drivers/net/wwan/t9xx/
+
MEDIATEK USB3 DRD IP DRIVER
M: Chunfeng Yun <chunfeng.yun@mediatek.com>
L: linux-usb@vger.kernel.org
diff --git a/drivers/net/wwan/Kconfig b/drivers/net/wwan/Kconfig
index 88df55d78d90..f510cc8ab4e4 100644
--- a/drivers/net/wwan/Kconfig
+++ b/drivers/net/wwan/Kconfig
@@ -121,6 +121,17 @@ config MTK_T7XX
If unsure, say N.
+config MTK_T9XX
+ tristate "MediaTek PCIe 5G WWAN modem T9xx device"
+ depends on PCI_MSI && ACPI
+ help
+ Enables MediaTek PCIe based 5G WWAN modem (T9xx series) device.
+
+ To compile this driver as a module, choose M here: the module will be
+ called mtk_t9xx.
+
+ If unsure, say N.
+
endif # WWAN
endmenu
diff --git a/drivers/net/wwan/Makefile b/drivers/net/wwan/Makefile
index 3960c0ae2445..7361eef4c472 100644
--- a/drivers/net/wwan/Makefile
+++ b/drivers/net/wwan/Makefile
@@ -14,3 +14,4 @@ obj-$(CONFIG_QCOM_BAM_DMUX) += qcom_bam_dmux.o
obj-$(CONFIG_RPMSG_WWAN_CTRL) += rpmsg_wwan_ctrl.o
obj-$(CONFIG_IOSM) += iosm/
obj-$(CONFIG_MTK_T7XX) += t7xx/
+obj-$(CONFIG_MTK_T9XX) += t9xx/
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
new file mode 100644
index 000000000000..bd6063c85c2b
--- /dev/null
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+ccflags-y += -I$(src)/pcie
+ccflags-y += -I$(src)
+
+obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
+
+mtk_t9xx-y := \
+ pcie/mtk_pci.o
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
new file mode 100644
index 000000000000..fc25418bb957
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_DEV_H__
+#define __MTK_DEV_H__
+
+#include <linux/bits.h>
+#include <linux/types.h>
+
+struct device;
+
+#define MTK_DEV_STR_LEN 16
+
+enum mtk_user_id {
+ MTK_USER_CTRL,
+ MTK_USER_MAX
+};
+
+enum mtk_dev_evt_h2d {
+ DEV_EVT_H2D_DEVICE_RESET = BIT(2),
+};
+
+enum mtk_dev_evt_d2h {
+ DEV_EVT_D2H_BOOT_FLOW_SYNC = BIT(4),
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP = BIT(5),
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD = BIT(6),
+};
+
+/* mtk_md_dev defines the structure of MTK modem device */
+struct mtk_md_dev {
+ struct device *dev;
+ void *hw_priv;
+ u32 hw_ver;
+ char dev_str[MTK_DEV_STR_LEN];
+};
+
+#endif /* __MTK_DEV_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
new file mode 100644
index 000000000000..902bcbe1512e
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -0,0 +1,1044 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/acpi.h>
+#include <linux/bitfield.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+#define MTK_MHCCIF_RC_BASE_ADDR 0x1000A000
+
+#define MTK_MHCCIF_RESET_DELAY_MS 2000
+#define MTK_PLDR_READY_POLL_MS 20
+#define MTK_PLDR_READY_POLLS 50
+
+static const int mtk_pci_irq_tbl[MTK_IRQ_SRC_MAX] = {
+ [MTK_IRQ_SRC_MHCCIF] = 28,
+ [MTK_IRQ_SRC_CLDMA0] = 27,
+ [MTK_IRQ_SRC_CLDMA1] = 26,
+};
+
+#define MTK_PCI_TRANSPARENT_ATR_SIZE (0x3F)
+#define MTK_PCI_MINIMUM_ATR_SIZE (0x1000)
+#define ATR_SIZE_LO32_MASK GENMASK_ULL(31, 0)
+#define ATR_SIZE_HI32_MASK GENMASK_ULL(63, 32)
+#define ATR_SIZE_BIAS_FROM_LO32 2
+#define ATR_ADDR_ALIGN_MASK 0xFFFFF000
+#define ATR_EN BIT(0)
+#define ATR_PARAM_OFFSET 16
+
+struct mtk_mhccif_cb {
+ struct list_head entry;
+ int (*evt_cb)(u32 status, void *data);
+ void *data;
+ u32 chs;
+};
+
+/**
+ * mtk_pci_setup_atr() - Configure a PCIe address translation rule
+ * @mdev: MTK MD device
+ * @cfg: ATR configuration parameters
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int mtk_pci_setup_atr(struct mtk_md_dev *mdev, struct mtk_atr_cfg *cfg)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ u32 addr, val, size_h, size_l;
+ int atr_size, pos, offset;
+
+ if (cfg->transparent) {
+ /* No address conversion is performed */
+ atr_size = MTK_PCI_TRANSPARENT_ATR_SIZE;
+ } else {
+ if (cfg->size < MTK_PCI_MINIMUM_ATR_SIZE)
+ cfg->size = MTK_PCI_MINIMUM_ATR_SIZE;
+
+ if (cfg->src_addr & (cfg->size - 1)) {
+ dev_err(mdev->dev, "Invalid atr src addr is not aligned to size\n");
+ return -EINVAL;
+ }
+
+ if (cfg->trsl_addr & (cfg->size - 1)) {
+ dev_err(mdev->dev,
+ "Invalid atr trsl addr is not aligned to size, %llx, %llx\n",
+ cfg->trsl_addr, cfg->size - 1);
+ return -EINVAL;
+ }
+
+ size_l = FIELD_GET(ATR_SIZE_LO32_MASK, cfg->size);
+ size_h = FIELD_GET(ATR_SIZE_HI32_MASK, cfg->size);
+ pos = ffs(size_l);
+ if (pos) {
+ atr_size = pos - ATR_SIZE_BIAS_FROM_LO32;
+ } else {
+ pos = ffs(size_h);
+ atr_size = pos + 32 - ATR_SIZE_BIAS_FROM_LO32;
+ }
+ }
+
+ /* Calculate table offset */
+ offset = ATR_PORT_OFFSET * cfg->port + ATR_TABLE_OFFSET * cfg->table;
+
+ addr = REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB + offset;
+ val = (u32)(cfg->src_addr >> 32);
+ mtk_pci_mac_write32(priv, addr, val);
+
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB + offset;
+ val = (u32)(cfg->trsl_addr >> 32);
+ mtk_pci_mac_write32(priv, addr, val);
+
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB + offset;
+ val = (u32)(cfg->trsl_addr & ATR_ADDR_ALIGN_MASK);
+ mtk_pci_mac_write32(priv, addr, val);
+
+ /* TRSL_PARAM */
+ addr = REG_ATR_PCIE_WIN0_T0_TRSL_PARAM + offset;
+ val = (cfg->trsl_param << ATR_PARAM_OFFSET) | cfg->trsl_id;
+ mtk_pci_mac_write32(priv, addr, val);
+
+ /* Enable ATR last, after translation target is fully programmed */
+ addr = REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset;
+ val = (u32)(cfg->src_addr & ATR_ADDR_ALIGN_MASK) | (atr_size << 1) | ATR_EN;
+ mtk_pci_mac_write32(priv, addr, val);
+
+ /* Ensure ATR is set */
+ mtk_pci_mac_read32(priv, addr);
+
+ return 0;
+}
+
+/**
+ * mtk_pci_atr_disable() - Disable all PCIe address translation rules
+ * @priv: MTK PCI private data
+ */
+static void mtk_pci_atr_disable(struct mtk_pci_priv *priv)
+{
+ int port, tbl, offset;
+ u32 val;
+
+ /* Disable all ATR table for all ports */
+ for (port = ATR_SRC_PCI_WIN0; port <= ATR_SRC_AXIS_3; port++)
+ for (tbl = 0; tbl < ATR_TABLE_NUM_PER_ATR; tbl++) {
+ /* Calculate table offset */
+ offset = ATR_PORT_OFFSET * port + ATR_TABLE_OFFSET * tbl;
+ val = mtk_pci_mac_read32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset);
+ val = val & (~BIT(0));
+ /* Disable table by SRC_ADDR_L */
+ mtk_pci_mac_write32(priv, REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB + offset, val);
+ }
+}
+
+/**
+ * mtk_pci_get_dev_state() - Read the device state from the modem
+ * @mdev: MTK MD device
+ *
+ * Return: Device state value.
+ */
+u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev)
+{
+ return mtk_pci_mac_read32(mdev->hw_priv, REG_PCIE_DEBUG_DUMMY_7);
+}
+
+/**
+ * mtk_pci_get_irq_id() - Map an IRQ source to its hardware IRQ ID
+ * @mdev: MTK MD device
+ * @irq_src: IRQ source enum
+ *
+ * Return: IRQ ID on success, -EINVAL on failure.
+ */
+int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src)
+{
+ int irq_id = -EINVAL;
+
+ if (irq_src > MTK_IRQ_SRC_MIN && irq_src < MTK_IRQ_SRC_MAX) {
+ irq_id = mtk_pci_irq_tbl[irq_src];
+ if (irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX)
+ irq_id = -EINVAL;
+ }
+
+ return irq_id;
+}
+
+/**
+ * mtk_pci_get_virq_id() - Get the Linux virtual IRQ for a hardware IRQ ID
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ *
+ * Return: Virtual IRQ number on success, negative error code on failure.
+ */
+int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (irq_id < 0 || irq_id >= priv->irq_cnt)
+ return -EINVAL;
+
+ return pci_irq_vector(pdev, irq_id);
+}
+
+/**
+ * mtk_pci_register_irq() - Register a callback for a hardware IRQ
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ * @irq_cb: Callback function
+ * @data: Private data passed to callback
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id,
+ int (*irq_cb)(int irq_id, void *data), void *data)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if ((irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) || !irq_cb)
+ return -EINVAL;
+
+ spin_lock(&priv->irq_cb_lock);
+ if (priv->irq_cb_list[irq_id]) {
+ spin_unlock(&priv->irq_cb_lock);
+ dev_err(mdev->dev,
+ "Unable to register irq, irq_id=%d, it's already been register by %ps.\n",
+ irq_id, priv->irq_cb_list[irq_id]);
+ return -EBUSY;
+ }
+ priv->irq_cb_data[irq_id] = data;
+ smp_wmb(); /* Ensure data is visible before callback */
+ WRITE_ONCE(priv->irq_cb_list[irq_id], irq_cb);
+ spin_unlock(&priv->irq_cb_lock);
+
+ return 0;
+}
+
+/**
+ * mtk_pci_unregister_irq() - Unregister a hardware IRQ callback
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int virq_id;
+
+ if (irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX)
+ return -EINVAL;
+
+ if (!READ_ONCE(priv->irq_cb_list[irq_id])) {
+ dev_err(mdev->dev, "irq_id=%d has not been registered\n", irq_id);
+ return -ENOENT;
+ }
+
+ /* mtk_pci_irq_handler() loads the callback and its data separately, so
+ * the data may only be cleared once no handler can still hold a live
+ * callback pointer.
+ */
+ mtk_pci_mask_irq(mdev, irq_id);
+
+ spin_lock(&priv->irq_cb_lock);
+ WRITE_ONCE(priv->irq_cb_list[irq_id], NULL);
+ spin_unlock(&priv->irq_cb_lock);
+
+ virq_id = mtk_pci_get_virq_id(mdev, irq_id);
+ if (virq_id >= 0)
+ synchronize_irq(virq_id);
+
+ priv->irq_cb_data[irq_id] = NULL;
+
+ return 0;
+}
+
+/**
+ * mtk_pci_mask_irq() - Mask (disable) a hardware IRQ
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) {
+ dev_err(mdev->dev, "Failed to mask irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+/**
+ * mtk_pci_unmask_irq() - Unmask (enable) a hardware IRQ
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) {
+ dev_err(mdev->dev, "Failed to unmask irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_SET_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+/**
+ * mtk_pci_clear_irq() - Clear (acknowledge) a hardware IRQ
+ * @mdev: MTK MD device
+ * @irq_id: Hardware IRQ ID
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ if (irq_id < 0 || irq_id >= MTK_IRQ_CNT_MAX) {
+ dev_err(mdev->dev, "Failed to clear irq: input irq_id=%d\n", irq_id);
+ return -EINVAL;
+ }
+
+ mtk_pci_mac_write32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0, BIT(irq_id));
+
+ return 0;
+}
+
+static inline u32 mtk_pci_map_evt_bit(u32 val, u32 from, u32 to)
+{
+ return (val & from) ? to : 0;
+}
+
+static u32 mtk_pci_ext_d2h_evt_hw_bits(u32 chs)
+{
+ return mtk_pci_map_evt_bit(chs, DEV_EVT_D2H_BOOT_FLOW_SYNC,
+ MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC) |
+ mtk_pci_map_evt_bit(chs, DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP,
+ MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP) |
+ mtk_pci_map_evt_bit(chs, DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD,
+ MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD);
+}
+
+static u32 mtk_pci_ext_d2h_evt_chs(u32 hw_bits)
+{
+ return mtk_pci_map_evt_bit(hw_bits, MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC,
+ DEV_EVT_D2H_BOOT_FLOW_SYNC) |
+ mtk_pci_map_evt_bit(hw_bits, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP,
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP) |
+ mtk_pci_map_evt_bit(hw_bits, MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD,
+ DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD);
+}
+
+/**
+ * mtk_pci_register_ext_evt() - Register a callback for MHCCIF device events
+ * @mdev: MTK MD device
+ * @chs: Bitmask of event channels to register; must not overlap the
+ * channels of a callback already registered
+ * @evt_cb: Callback function, called from a work item with mhccif_lock
+ * held: it must not sleep or (un)register MHCCIF callbacks
+ * @data: Private data passed to callback
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs,
+ int (*evt_cb)(u32 status, void *data), void *data)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_mhccif_cb *cb;
+ int ret = 0;
+
+ if (!chs || !evt_cb)
+ return -EINVAL;
+
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry(cb, &priv->mhccif_cb_list, entry) {
+ if (cb->chs & chs) {
+ ret = -EBUSY;
+ dev_err(mdev->dev,
+ "Unable to register evt, intersection: chs=0x%08x&0x%08x cb=%ps\n",
+ chs, cb->chs, cb->evt_cb);
+ goto err_spin_unlock;
+ }
+ }
+ cb = kzalloc(sizeof(*cb), GFP_ATOMIC);
+ if (!cb) {
+ ret = -ENOMEM;
+ goto err_spin_unlock;
+ }
+ cb->evt_cb = evt_cb;
+ cb->data = data;
+ cb->chs = chs;
+ list_add_tail(&cb->entry, &priv->mhccif_cb_list);
+err_spin_unlock:
+ spin_unlock_bh(&priv->mhccif_lock);
+
+ return ret;
+}
+
+/**
+ * mtk_pci_unregister_ext_evt() - Unregister an MHCCIF device event callback
+ * @mdev: MTK MD device
+ * @chs: Bitmask of event channels to unregister, exactly as registered
+ */
+void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_mhccif_cb *cb, *next;
+
+ if (!chs)
+ return;
+
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry_safe(cb, next, &priv->mhccif_cb_list, entry) {
+ if (cb->chs == chs) {
+ list_del(&cb->entry);
+ kfree(cb);
+ goto out;
+ }
+ }
+ dev_warn(mdev->dev,
+ "Unable to unregister evt, no chs=0x%08x has been registered.\n", chs);
+out:
+ spin_unlock_bh(&priv->mhccif_lock);
+}
+
+/**
+ * mtk_pci_mask_ext_evt() - Mask (disable) MHCCIF device events
+ * @mdev: MTK MD device
+ * @chs: Bitmask of event channels to mask
+ */
+void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ u32 hw_bits = mtk_pci_ext_d2h_evt_hw_bits(chs);
+
+ mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_EAP_MASK_SET, hw_bits);
+}
+
+/**
+ * mtk_pci_unmask_ext_evt() - Unmask (enable) MHCCIF device events
+ * @mdev: MTK MD device
+ * @chs: Bitmask of event channels to unmask
+ */
+void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs)
+{
+ u32 hw_bits = mtk_pci_ext_d2h_evt_hw_bits(chs);
+
+ mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR, hw_bits);
+}
+
+static u32 mtk_pci_ext_h2d_evt_hw_bits(u32 chs)
+{
+ return mtk_pci_map_evt_bit(chs, DEV_EVT_H2D_DEVICE_RESET,
+ MHCCIF_RC2EP_EVT_DEVICE_RESET);
+}
+
+/**
+ * mtk_pci_send_ext_evt() - Send an MHCCIF event to the modem
+ * @mdev: MTK MD device
+ * @ch: Event channel to trigger (must be a single bit)
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+static int mtk_pci_send_ext_evt(struct mtk_md_dev *mdev, u32 ch)
+{
+ u32 rc_base = MTK_MHCCIF_RC_BASE_ADDR;
+ u32 hw_bits;
+
+ /* Only allow one ch to be triggered at a time */
+ if (!is_power_of_2(ch)) {
+ dev_err(mdev->dev, "Unsupported ext evt ch=0x%08x\n", ch);
+ return -EINVAL;
+ }
+
+ hw_bits = mtk_pci_ext_h2d_evt_hw_bits(ch);
+ if (!hw_bits) {
+ dev_err(mdev->dev, "Unmapped ext evt ch=0x%08x\n", ch);
+ return -EINVAL;
+ }
+
+ mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_BSY, hw_bits);
+ mtk_pci_write32(mdev, rc_base + MHCCIF_RC2EP_SW_TCHNUM, ffs(hw_bits) - 1);
+ return 0;
+}
+
+static u32 mtk_pci_get_ext_evt_hw_status(struct mtk_md_dev *mdev)
+{
+ return mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_STS);
+}
+
+static void mtk_pci_ack_ext_evt_hw(struct mtk_md_dev *mdev, u32 hw_bits)
+{
+ mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_ACK, hw_bits);
+ /* Flush the ack before the callbacks run: one landing late would
+ * erase an event they make the device raise.
+ */
+ mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_STS);
+}
+
+/**
+ * mtk_pci_pldr() - Reset the modem via its ACPI MRST._RST method
+ * @mdev: MTK MD device
+ *
+ * Return:
+ * * 0 - Success, device was reset.
+ * * -ENODEV - No ACPI, no handle or no MRST._RST method. Device
+ * untouched.
+ * * -EIO - MRST._RST failed. Device untouched, still running;
+ * a different reset may be attempted.
+ */
+static int mtk_pci_pldr(struct mtk_md_dev *mdev)
+{
+ acpi_status acpi_ret;
+ acpi_handle handle;
+
+ if (acpi_disabled) {
+ dev_dbg(mdev->dev, "Unsupported, acpi function isn't enable\n");
+ return -ENODEV;
+ }
+
+ handle = ACPI_HANDLE(mdev->dev);
+ if (!handle) {
+ dev_dbg(mdev->dev, "Unsupported, acpi handle isn't found\n");
+ return -ENODEV;
+ }
+ if (!acpi_has_method(handle, "MRST._RST")) {
+ dev_dbg(mdev->dev, "Unsupported, pldr method isn't supported\n");
+ return -ENODEV;
+ }
+ acpi_ret = acpi_evaluate_object(handle, "MRST._RST", NULL, NULL);
+ if (ACPI_FAILURE(acpi_ret)) {
+ dev_err(mdev->dev, "Failed to execute MRST._RST method: %s\n",
+ acpi_format_exception(acpi_ret));
+ return -EIO;
+ }
+
+ return 0;
+}
+
+/**
+ * mtk_pci_link_check() - Check if the PCIe link to the modem is active
+ * @mdev: MTK MD device
+ *
+ * Return: true if the device is present, false otherwise.
+ */
+static bool mtk_pci_link_check(struct mtk_md_dev *mdev)
+{
+ return pci_device_is_present(to_pci_dev(mdev->dev));
+}
+
+static void mtk_mhccif_isr_work(struct work_struct *work)
+{
+ struct mtk_pci_priv *priv =
+ container_of(work, struct mtk_pci_priv, mhccif_work);
+ struct mtk_md_dev *mdev = priv->mdev;
+ struct mtk_mhccif_cb *cb;
+ u32 stat, mask, chs;
+
+ /* Clear level 1 before reading level 2: an event raised after the
+ * clear latches level 1 again and fires on the unmask below, one
+ * raised before it is in the read.
+ */
+ mtk_pci_clear_irq(mdev, priv->mhccif_irq_id);
+ stat = mtk_pci_get_ext_evt_hw_status(mdev);
+ mask = mtk_pci_read32(mdev, MTK_MHCCIF_RC_BASE_ADDR
+ + MHCCIF_EP2RC_SW_INT_EAP_MASK);
+ if (unlikely(stat == U32_MAX && !(mtk_pci_link_check(mdev)))) {
+ /* When link failed, we don't need to unmask. */
+ dev_err(mdev->dev, "Failed to check link in MHCCIF handler.\n");
+ return;
+ }
+
+ stat &= ~mask;
+ /* Acknowledge level 2 before dispatch: an event that re-asserts
+ * while a callback runs must survive as a new status bit.
+ */
+ if (stat)
+ mtk_pci_ack_ext_evt_hw(mdev, stat);
+
+ chs = mtk_pci_ext_d2h_evt_chs(stat);
+ /* Callbacks must not sleep or modify mhccif_cb_list */
+ spin_lock_bh(&priv->mhccif_lock);
+ list_for_each_entry(cb, &priv->mhccif_cb_list, entry) {
+ if (cb->chs & chs)
+ cb->evt_cb(cb->chs & chs, cb->data);
+ }
+ spin_unlock_bh(&priv->mhccif_lock);
+
+ mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+}
+
+static const struct pci_device_id t9xx_pci_table[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_MEDIATEK, 0x0900),
+ .class = MTK_PCI_CLASS, .class_mask = ~0 },
+ { PCI_DEVICE(0x03f0, 0x01ca), /* HP RMM-G4 */
+ .class = MTK_PCI_CLASS, .class_mask = ~0 },
+ {/* end: all zeroes */}
+};
+
+MODULE_DEVICE_TABLE(pci, t9xx_pci_table);
+
+static int mtk_pci_atr_init(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_atr_cfg cfg;
+ int port, ret;
+
+ mtk_pci_atr_disable(priv);
+
+ /* Config ATR for RC to access device's register */
+ cfg.src_addr = pci_bus_address(pdev, MTK_BAR_2_3_IDX);
+ cfg.size = ATR_PCIE_REG_SIZE;
+ cfg.trsl_addr = ATR_PCIE_REG_TRSL_ADDR;
+ cfg.port = ATR_PCIE_REG_PORT;
+ cfg.table = ATR_PCIE_REG_TABLE_NUM;
+ cfg.trsl_id = ATR_PCIE_REG_TRSL_PORT;
+ cfg.trsl_param = 0x0;
+ cfg.transparent = 0x0;
+ ret = mtk_pci_setup_atr(mdev, &cfg);
+ if (ret)
+ return ret;
+
+ /* Config ATR for EP to access RC's memory */
+ for (port = ATR_SRC_AXIS_0; port <= ATR_SRC_AXIS_3; port++) {
+ cfg.src_addr = ATR_PCIE_DEV_DMA_SRC_ADDR;
+ cfg.size = ATR_PCIE_DEV_DMA_SIZE;
+ cfg.trsl_addr = ATR_PCIE_DEV_DMA_TRSL_ADDR;
+ cfg.port = port;
+ cfg.table = ATR_PCIE_DEV_DMA_TABLE_NUM;
+ cfg.trsl_id = ATR_DST_PCI_TRX;
+ cfg.trsl_param = 0x0;
+ /* Enable transparent translation */
+ cfg.transparent = ATR_PCIE_DEV_DMA_TRANSPARENT;
+ ret = mtk_pci_setup_atr(mdev, &cfg);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int mtk_pci_bar_init(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+
+ /* Fixed offsets used on these mappings (MSI-X registers up to
+ * 0x3103 on BAR0/1, the ATR-biased MHCCIF window on BAR2/3) are
+ * guaranteed by the only two devices this driver binds (0x0900,
+ * 0x01CA), whose BAR sizes are fixed by hardware. A device that
+ * misreports them is out of scope rather than impossible, so no
+ * pci_resource_len() checks.
+ */
+ priv->mac_reg_base = pcim_iomap_region(pdev, MTK_BAR_0_1_IDX,
+ mdev->dev_str);
+ if (IS_ERR(priv->mac_reg_base)) {
+ dev_err(mdev->dev, "Failed to map BAR0/1\n");
+ return PTR_ERR(priv->mac_reg_base);
+ }
+
+ priv->bar23_addr = pcim_iomap_region(pdev, MTK_BAR_2_3_IDX,
+ mdev->dev_str);
+ if (IS_ERR(priv->bar23_addr)) {
+ dev_err(mdev->dev, "Failed to map BAR2/3\n");
+ return PTR_ERR(priv->bar23_addr);
+ }
+
+ /* We use MD view base address "0" to observe registers */
+ priv->ext_reg_base = priv->bar23_addr - ATR_PCIE_REG_TRSL_ADDR;
+
+ return 0;
+}
+
+static int mtk_mhccif_irq_cb(int irq_id, void *data)
+{
+ struct mtk_md_dev *mdev = data;
+ struct mtk_pci_priv *priv;
+
+ priv = mdev->hw_priv;
+ queue_work(system_highpri_wq, &priv->mhccif_work);
+
+ return 0;
+}
+
+static int mtk_mhccif_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int ret;
+
+ INIT_LIST_HEAD(&priv->mhccif_cb_list);
+ spin_lock_init(&priv->mhccif_lock);
+ INIT_WORK(&priv->mhccif_work, mtk_mhccif_isr_work);
+
+ /* The write below goes through the BAR 2/3 ATR window, so this
+ * function must run after mtk_pci_atr_init().
+ * Mask every channel; consumers unmask the channels they need.
+ * Do NOT ack here: EP2RC status bits are latched one-shot
+ * notifications (e.g. BOOT_FLOW_SYNC raised before the driver
+ * loads) that the device never re-sends. Masking blocks
+ * delivery but preserves the bit for delivery on unmask; an
+ * ack would erase it and the boot flow would hang silently.
+ */
+ mtk_pci_write32(mdev, MTK_MHCCIF_RC_BASE_ADDR +
+ MHCCIF_EP2RC_SW_INT_EAP_MASK_SET, U32_MAX);
+
+ ret = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_MHCCIF);
+ if (ret < 0) {
+ dev_err(mdev->dev, "Failed to get mhccif_irq_id. ret=%d\n", ret);
+ return ret;
+ }
+ priv->mhccif_irq_id = ret;
+
+ ret = mtk_pci_register_irq(mdev, priv->mhccif_irq_id, mtk_mhccif_irq_cb, mdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to register mhccif_irq callback\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static void mtk_mhccif_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_mhccif_cb *cb, *tmp;
+
+ mtk_pci_unregister_irq(mdev, priv->mhccif_irq_id);
+ cancel_work_sync(&priv->mhccif_work);
+ /* The work unmasks the source on its way out, so leave it masked. */
+ mtk_pci_mask_irq(mdev, priv->mhccif_irq_id);
+
+ list_for_each_entry_safe(cb, tmp, &priv->mhccif_cb_list, entry) {
+ list_del(&cb->entry);
+ kfree(cb);
+ }
+}
+
+static irqreturn_t mtk_pci_irq_handler(struct mtk_md_dev *mdev, u32 irq_state)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int irq_id;
+
+ /* Check whether each set bit has a callback, if has, call it */
+ do {
+ int (*cb)(int irq_id, void *data);
+
+ irq_id = fls(irq_state) - 1;
+ irq_state &= ~BIT(irq_id);
+ cb = READ_ONCE(priv->irq_cb_list[irq_id]);
+ if (likely(cb)) {
+ smp_rmb(); /* Ensure data is read after callback */
+ cb(irq_id, priv->irq_cb_data[irq_id]);
+ } else {
+ dev_err_ratelimited(mdev->dev,
+ "Unhandled irq_id=%d, no callback for it.\n",
+ irq_id);
+ mtk_pci_clear_irq(mdev, irq_id);
+ }
+ } while (irq_state);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t mtk_pci_irq_msix(int irq, void *data)
+{
+ struct mtk_pci_irq_desc *irq_desc = data;
+ struct mtk_md_dev *mdev = irq_desc->mdev;
+ struct mtk_pci_priv *priv;
+ u32 irq_state, irq_enable;
+
+ priv = mdev->hw_priv;
+ irq_state = mtk_pci_mac_read32(priv, REG_MSIX_ISTATUS_HOST_GRP0_0);
+ irq_enable = mtk_pci_mac_read32(priv, REG_IMASK_HOST_MSIX_GRP0_0);
+ irq_state &= irq_enable;
+
+ if (unlikely(irq_state == U32_MAX && irq_enable == U32_MAX))
+ return IRQ_NONE; /* device gone */
+
+ irq_state &= irq_desc->msix_bits; /* scope to this vector */
+ if (unlikely(!irq_state))
+ return IRQ_NONE;
+
+ /* Mask the bit; the consumer unmasks it when it is done */
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, irq_state);
+
+ return mtk_pci_irq_handler(mdev, irq_state);
+}
+
+static int mtk_pci_request_irq_msix(struct mtk_md_dev *mdev)
+{
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct mtk_pci_irq_desc *irq_desc;
+ struct pci_dev *pdev;
+ int ret, i;
+
+ pdev = to_pci_dev(mdev->dev);
+ irq_desc = priv->irq_desc;
+
+ priv->irq_cnt = MTK_IRQ_CNT_MAX;
+
+ for (i = 0; i < MTK_IRQ_CNT_MAX; i++) {
+ irq_desc[i].mdev = mdev;
+ irq_desc[i].msix_bits = BIT(i);
+ snprintf(irq_desc[i].name, MTK_IRQ_NAME_LEN, "msix%d-%s", i, mdev->dev_str);
+ ret = pci_request_irq(pdev, i, mtk_pci_irq_msix, NULL,
+ &irq_desc[i], "%s", irq_desc[i].name);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to request %s: ret=%d\n",
+ irq_desc[i].name, ret);
+ for (i--; i >= 0; i--)
+ pci_free_irq(pdev, i, &irq_desc[i]);
+ priv->irq_cnt = 0;
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int mtk_pci_request_irq(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ int ret;
+
+ ret = pci_alloc_irq_vectors(pdev, MTK_IRQ_CNT_MAX,
+ MTK_IRQ_CNT_MAX, PCI_IRQ_MSIX);
+ if (ret < 0) {
+ dev_err(mdev->dev,
+ "Unable to alloc %d MSI-X vectors: ret=%d\n",
+ MTK_IRQ_CNT_MAX, ret);
+ return ret;
+ }
+
+ ret = mtk_pci_request_irq_msix(mdev);
+ if (ret)
+ pci_free_irq_vectors(pdev);
+
+ return ret;
+}
+
+static void mtk_pci_free_irq(struct mtk_md_dev *mdev)
+{
+ struct pci_dev *pdev = to_pci_dev(mdev->dev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ int i;
+
+ for (i = 0; i < priv->irq_cnt; i++)
+ pci_free_irq(pdev, i, &priv->irq_desc[i]);
+
+ pci_free_irq_vectors(pdev);
+ priv->irq_cnt = 0;
+}
+
+static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct device *dev = &pdev->dev;
+ struct mtk_pci_priv *priv;
+ struct mtk_md_dev *mdev;
+ int ret;
+
+ mdev = devm_kzalloc(dev, sizeof(*mdev), GFP_KERNEL);
+ if (!mdev) {
+ ret = -ENOMEM;
+ goto log_err;
+ }
+ mdev->dev = dev;
+
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv) {
+ ret = -ENOMEM;
+ goto log_err;
+ }
+
+ pci_set_drvdata(pdev, mdev);
+ priv->mdev = mdev;
+ mdev->hw_ver = pdev->device;
+ mdev->hw_priv = priv;
+ strscpy(mdev->dev_str, pci_name(pdev), MTK_DEV_STR_LEN);
+
+ ret = pci_enable_device(pdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to enable pci device.\n");
+ goto log_err;
+ }
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ dev_err(mdev->dev, "Failed to set DMA Mask and Coherent. (ret=%d)\n", ret);
+ goto disable_dev;
+ }
+
+ ret = mtk_pci_bar_init(mdev);
+ if (ret)
+ goto disable_dev;
+
+ ret = mtk_pci_atr_init(mdev);
+ if (ret)
+ goto disable_atr;
+
+ spin_lock_init(&priv->irq_cb_lock);
+
+ ret = mtk_mhccif_init(mdev);
+ if (ret)
+ goto disable_atr;
+
+ /* Mask every source before the handlers go in, so a source
+ * left enabled by firmware or a previous bind cannot fire
+ * into a half-initialised driver.
+ */
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
+
+ ret = mtk_pci_request_irq(mdev);
+ if (ret)
+ goto free_mhccif;
+
+ if (!mtk_pci_link_check(mdev)) {
+ ret = -ENOLINK;
+ goto free_irq;
+ }
+
+ ret = pci_save_state(pdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to save PCI state: %d\n", ret);
+ goto free_irq;
+ }
+
+ priv->saved_state = pci_store_saved_state(pdev);
+ if (!priv->saved_state) {
+ ret = -ENOMEM;
+ goto free_irq;
+ }
+
+ pci_set_master(pdev);
+ mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+
+ return 0;
+
+free_irq:
+ mtk_pci_free_irq(mdev);
+free_mhccif:
+ mtk_mhccif_exit(mdev);
+disable_atr:
+ mtk_pci_atr_disable(priv);
+disable_dev:
+ pci_disable_device(pdev);
+log_err:
+ dev_err(dev, "Failed to probe device, ret=%d\n", ret);
+
+ return ret;
+}
+
+static void mtk_pci_remove(struct pci_dev *pdev)
+{
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+ struct mtk_pci_priv *priv = mdev->hw_priv;
+ struct device *dev = &pdev->dev;
+ int i, ret;
+
+ /* Silence every source before tearing anything down. */
+ mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
+
+ /* Free the vectors first: once no handler can run, the unmask at the
+ * tail of mtk_mhccif_isr_work() cannot re-open the source behind
+ * mtk_mhccif_exit()'s back.
+ */
+ mtk_pci_free_irq(mdev);
+ mtk_mhccif_exit(mdev);
+ pci_clear_master(pdev);
+ pci_disable_device(pdev);
+
+ /* Reset last: the endpoint comes back at power-on defaults, so no MMIO
+ * and no MSI-X teardown may follow it.
+ */
+ ret = mtk_pci_pldr(mdev);
+ if (!ret) {
+ /* MRST._RST may return before the link has retrained, and a
+ * config write the device does not take is simply lost.
+ */
+ for (i = 0; i < MTK_PLDR_READY_POLLS && !mtk_pci_link_check(mdev); i++)
+ msleep(MTK_PLDR_READY_POLL_MS);
+ if (!mtk_pci_link_check(mdev))
+ dev_warn(dev, "device not back %d ms after reset, its config space may not be restored\n",
+ MTK_PLDR_READY_POLLS * MTK_PLDR_READY_POLL_MS);
+ } else if (!mtk_pci_link_check(mdev)) {
+ dev_warn(dev, "PLDR failed (%d) and link is down, modem not reset\n", ret);
+ } else {
+ dev_warn(dev, "PLDR failed (%d), trying MHCCIF reset\n", ret);
+ if (mtk_pci_send_ext_evt(mdev, DEV_EVT_H2D_DEVICE_RESET)) {
+ dev_err(dev, "MHCCIF reset failed\n");
+ } else {
+ /* Unlike MRST._RST the doorbell returns before the device
+ * has reset, so let it settle before the restore below.
+ */
+ msleep(MTK_MHCCIF_RESET_DELAY_MS);
+ }
+ }
+
+ /* Without this a runtime-PM suspend after unbind would snapshot the
+ * reset state over our copy, and whoever binds next finds BARs the
+ * device does not decode.
+ */
+ pci_load_and_free_saved_state(pdev, &priv->saved_state);
+ pci_restore_state(pdev);
+}
+
+/* A kexec reboot neither resets nor powers down the slot, and the boot
+ * notification the next kernel waits for is one-shot, so reset the modem
+ * on the way down the same way remove() does.
+ */
+static void mtk_pci_shutdown(struct pci_dev *pdev)
+{
+ mtk_pci_remove(pdev);
+}
+
+static pci_ers_result_t mtk_pci_error_detected(struct pci_dev *pdev,
+ pci_channel_state_t state)
+{
+ struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
+
+ dev_err(mdev->dev, "AER detected: pci_channel_state_t=%u\n", state);
+
+ /* AER recovery not supported, disconnect the device */
+ return PCI_ERS_RESULT_DISCONNECT;
+}
+
+static const struct pci_error_handlers mtk_pci_err_handler = {
+ .error_detected = mtk_pci_error_detected,
+};
+
+static struct pci_driver mtk_pci_drv = {
+ .name = KBUILD_MODNAME,
+ .id_table = t9xx_pci_table,
+ .probe = mtk_pci_probe,
+ .remove = mtk_pci_remove,
+ .shutdown = mtk_pci_shutdown,
+ .err_handler = &mtk_pci_err_handler
+};
+
+module_pci_driver(mtk_pci_drv);
+
+MODULE_DESCRIPTION("MediaTek T9xx PCIe WWAN driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.h b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
new file mode 100644
index 000000000000..ffb5f62a6729
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.h
@@ -0,0 +1,139 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PCI_H__
+#define __MTK_PCI_H__
+
+#include <linux/pci.h>
+
+#include "mtk_dev.h"
+
+enum mtk_irq_src {
+ MTK_IRQ_SRC_MIN,
+ MTK_IRQ_SRC_MHCCIF,
+ MTK_IRQ_SRC_CLDMA0,
+ MTK_IRQ_SRC_CLDMA1,
+ MTK_IRQ_SRC_MAX
+};
+
+enum mtk_atr_src_port {
+ ATR_SRC_PCI_WIN0 = 0,
+ ATR_SRC_PCI_WIN1,
+ ATR_SRC_AXIS_0,
+ ATR_SRC_AXIS_1,
+ ATR_SRC_AXIS_2,
+ ATR_SRC_AXIS_3,
+};
+
+enum mtk_atr_dst_port {
+ ATR_DST_PCI_TRX = 0,
+ ATR_DST_AXIM_0 = 4,
+};
+
+#define MTK_PCI_CLASS 0x0D4000
+
+#define MTK_BAR_0_1_IDX 0
+#define MTK_BAR_2_3_IDX 2
+
+#define MTK_IRQ_CNT_MAX 32
+#define MTK_IRQ_NAME_LEN 32
+
+#define ATR_PORT_OFFSET 0x100
+#define ATR_TABLE_OFFSET 0x20
+#define ATR_TABLE_NUM_PER_ATR 8
+#define ATR_PCIE_REG_TRSL_ADDR 0x10000000
+#define ATR_PCIE_REG_SIZE 0x00400000
+#define ATR_PCIE_REG_PORT ATR_SRC_PCI_WIN0
+#define ATR_PCIE_REG_TABLE_NUM 1
+#define ATR_PCIE_REG_TRSL_PORT ATR_DST_AXIM_0
+#define ATR_PCIE_DEV_DMA_SRC_ADDR 0x00000000
+#define ATR_PCIE_DEV_DMA_TRANSPARENT 1
+#define ATR_PCIE_DEV_DMA_SIZE 0
+#define ATR_PCIE_DEV_DMA_TABLE_NUM 0
+#define ATR_PCIE_DEV_DMA_TRSL_ADDR 0x00000000
+
+struct mtk_pci_irq_desc {
+ struct mtk_md_dev *mdev;
+ u32 msix_bits;
+ char name[MTK_IRQ_NAME_LEN];
+};
+
+struct mtk_pci_priv {
+ struct mtk_md_dev *mdev;
+ void __iomem *bar23_addr;
+ void __iomem *mac_reg_base;
+ void __iomem *ext_reg_base;
+ int irq_cnt;
+ /* irq_cb_lock: protects irq_cb_list[] and, on the register side,
+ * irq_cb_data[]. mtk_pci_unregister_irq() clears irq_cb_data[]
+ * after synchronize_irq() and outside the lock, because a handler
+ * past its READ_ONCE() of irq_cb_list[] still needs the data.
+ * Callers must not register and unregister one irq_id concurrently.
+ */
+ spinlock_t irq_cb_lock;
+ void *irq_cb_data[MTK_IRQ_CNT_MAX];
+
+ int (*irq_cb_list[MTK_IRQ_CNT_MAX])(int irq_id, void *data);
+ struct mtk_pci_irq_desc irq_desc[MTK_IRQ_CNT_MAX];
+ struct list_head mhccif_cb_list;
+ /* mhccif_lock: lock to protect mhccif_cb_list */
+ spinlock_t mhccif_lock;
+ struct work_struct mhccif_work;
+ int mhccif_irq_id;
+ struct pci_saved_state *saved_state;
+};
+
+struct mtk_atr_cfg {
+ u64 src_addr;
+ u64 trsl_addr;
+ u64 size;
+ u32 port; /* Port number */
+ u32 table; /* Table number (8 tables for each port) */
+ u32 trsl_id;
+ u32 trsl_param;
+ u32 transparent;
+};
+
+/* BAR 0/1 MMIO access */
+static inline u32 mtk_pci_mac_read32(struct mtk_pci_priv *priv, u64 addr)
+{
+ return ioread32(priv->mac_reg_base + addr);
+}
+
+static inline void mtk_pci_mac_write32(struct mtk_pci_priv *priv, u64 addr, u32 val)
+{
+ iowrite32(val, priv->mac_reg_base + addr);
+}
+
+/* BAR 2/3 MMIO access */
+static inline u32 mtk_pci_read32(struct mtk_md_dev *mdev, u64 addr)
+{
+ return ioread32(((struct mtk_pci_priv *)mdev->hw_priv)->ext_reg_base + addr);
+}
+
+static inline void mtk_pci_write32(struct mtk_md_dev *mdev, u64 addr, u32 val)
+{
+ iowrite32(val, ((struct mtk_pci_priv *)mdev->hw_priv)->ext_reg_base + addr);
+}
+
+/* Device operations */
+u32 mtk_pci_get_dev_state(struct mtk_md_dev *mdev);
+/* IRQ Related operations */
+int mtk_pci_get_irq_id(struct mtk_md_dev *mdev, enum mtk_irq_src irq_src);
+int mtk_pci_get_virq_id(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_register_irq(struct mtk_md_dev *mdev, int irq_id,
+ int (*irq_cb)(int irq_id, void *data), void *data);
+int mtk_pci_unregister_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_mask_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_unmask_irq(struct mtk_md_dev *mdev, int irq_id);
+int mtk_pci_clear_irq(struct mtk_md_dev *mdev, int irq_id);
+/* External event related */
+int mtk_pci_register_ext_evt(struct mtk_md_dev *mdev, u32 chs,
+ int (*evt_cb)(u32 status, void *data), void *data);
+void mtk_pci_unregister_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+void mtk_pci_mask_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+void mtk_pci_unmask_ext_evt(struct mtk_md_dev *mdev, u32 chs);
+
+#endif /* __MTK_PCI_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
new file mode 100644
index 000000000000..a97ad6630dcf
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PCI_REG_H__
+#define __MTK_PCI_REG_H__
+
+#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_LSB 0x0600
+#define REG_ATR_PCIE_WIN0_T0_SRC_ADDR_MSB 0x0604
+#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_LSB 0x0608
+#define REG_ATR_PCIE_WIN0_T0_TRSL_ADDR_MSB 0x060C
+#define REG_ATR_PCIE_WIN0_T0_TRSL_PARAM 0x0610
+#define REG_PCIE_DEBUG_DUMMY_7 0x0D1C
+#define REG_MSIX_ISTATUS_HOST_GRP0_0 0x0F00
+#define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000
+#define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080
+#define REG_IMASK_HOST_MSIX_GRP0_0 0x3100
+
+/* mhccif registers */
+#define MHCCIF_RC2EP_SW_BSY 0x4
+#define MHCCIF_RC2EP_SW_TCHNUM 0xC
+#define MHCCIF_RC2EP_EVT_DEVICE_RESET BIT(13)
+
+#define MHCCIF_EP2RC_SW_INT_STS 0x10
+#define MHCCIF_EP2RC_SW_INT_ACK 0x14
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK 0x20
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK_SET 0x30
+#define MHCCIF_EP2RC_SW_INT_EAP_MASK_CLR 0x40
+#define MHCCIF_EP2RC_EVT_BOOT_FLOW_SYNC BIT(5)
+#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_SAP BIT(15)
+#define MHCCIF_EP2RC_EVT_ASYNC_HS_NOTIFY_MD BIT(16)
+
+#endif /* __MTK_PCI_REG_H__ */
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 2/6] net: wwan: t9xx: Add control plane transaction layer framework
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface Jack Wu via B4 Relay
` (3 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
Introduce the control plane and transaction layer framework
for the T9XX WWAN driver. This patch adds the core data
structures (mtk_ctrl_blk, mtk_ctrl_trans) and initialization
entry points that subsequent patches build upon.
The control plane block (ctrl_blk) serves as the central
context for managing the transaction layer, port layer, and
FSM interactions. The actual DMA engine and TX/RX service
implementations are added in subsequent patches, and
mtk_ctrl_init() and mtk_ctrl_exit() get their first callers
there, in pcie/mtk_trans_ctrl.c.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 1 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 46 +++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 21 +++++++++++++
drivers/net/wwan/t9xx/mtk_dev.h | 3 ++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 1 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 19 ++++++++++++
6 files changed, 91 insertions(+)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index bd6063c85c2b..0a07507f1ddf 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -6,4 +6,5 @@ ccflags-y += -I$(src)
obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
mtk_t9xx-y := \
+ mtk_ctrl_plane.o \
pcie/mtk_pci.o
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
new file mode 100644
index 000000000000..81f6b3fbc0bd
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -0,0 +1,46 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ * Copyright (c) 2022-2023, Intel Corporation.
+ */
+
+#include <linux/device.h>
+
+#include "mtk_ctrl_plane.h"
+
+/**
+ * mtk_ctrl_init() - Initialize the control plane block.
+ * @mdev: Pointer to the MTK modem device.
+ *
+ * Allocates and initializes the control plane block
+ * associated with @mdev. Runs once per bind; the allocation is
+ * devres managed and is released at unbind, not by
+ * mtk_ctrl_exit().
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+int mtk_ctrl_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk;
+
+ ctrl_blk = devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL);
+ if (!ctrl_blk)
+ return -ENOMEM;
+
+ ctrl_blk->mdev = mdev;
+ mdev->ctrl_blk = ctrl_blk;
+
+ return 0;
+}
+
+/**
+ * mtk_ctrl_exit() - Clean up the control plane block.
+ * @mdev: Pointer to the MTK modem device.
+ *
+ * Clears the control plane block pointer. The allocation
+ * itself is managed by devres and freed on driver detach.
+ */
+void mtk_ctrl_exit(struct mtk_md_dev *mdev)
+{
+ mdev->ctrl_blk = NULL;
+}
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
new file mode 100644
index 000000000000..46f1ab9f0551
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_CTRL_PLANE_H__
+#define __MTK_CTRL_PLANE_H__
+
+#include <linux/skbuff.h>
+
+#include "mtk_dev.h"
+
+struct mtk_ctrl_blk {
+ struct mtk_md_dev *mdev;
+ void *ctrl_hw_priv;
+};
+
+int mtk_ctrl_init(struct mtk_md_dev *mdev);
+void mtk_ctrl_exit(struct mtk_md_dev *mdev);
+
+#endif /* __MTK_CTRL_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
index fc25418bb957..ff3cbaa4329e 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.h
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -28,12 +28,15 @@ enum mtk_dev_evt_d2h {
DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD = BIT(6),
};
+struct mtk_ctrl_blk;
+
/* mtk_md_dev defines the structure of MTK modem device */
struct mtk_md_dev {
struct device *dev;
void *hw_priv;
u32 hw_ver;
char dev_str[MTK_DEV_STR_LEN];
+ struct mtk_ctrl_blk *ctrl_blk;
};
#endif /* __MTK_DEV_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index 902bcbe1512e..27785092b2f7 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -12,6 +12,7 @@
#include <linux/module.h>
#include "mtk_dev.h"
+#include "mtk_trans_ctrl.h"
#include "mtk_pci.h"
#include "mtk_pci_reg.h"
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
new file mode 100644
index 000000000000..9b42f33bd008
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_TRANS_CTRL_H__
+#define __MTK_TRANS_CTRL_H__
+
+#include <linux/list.h>
+#include <linux/skbuff.h>
+#include <linux/types.h>
+
+#include "mtk_dev.h"
+
+struct mtk_ctrl_trans {
+ struct mtk_md_dev *mdev;
+};
+
+#endif
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 2/6] net: wwan: t9xx: Add control plane transaction layer framework Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 4/6] net: wwan: t9xx: Add control port Jack Wu via B4 Relay
` (2 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
Cross Layer Direct Memory Access(CLDMA) is the hardware
interface used by the control plane and designated to
translate data between the host and the device. It supports
8 hardware queues for the device AP and modem respectively.
CLDMA driver uses General Purpose Descriptor (GPD) to
describe transaction information that can be recognized by
CLDMA hardware. Once CLDMA hardware transaction is started,
it would fetch and parse GPD to transfer data correctly.
To facilitate the CLDMA transaction, a GPD ring for each
queue is used. Once the transaction is started, CLDMA
hardware will traverse the GPD ring to transfer data between
the host and the device until no GPD is available.
CLDMA TX flow:
Once a TX service receives the TX data from the port layer,
it uses APIs exported by the CLDMA driver to configure GPD
with the DMA address of TX data. After that, the service
triggers CLDMA to fetch the first available GPD to transfer
data.
CLDMA RX flow:
When there is RX data from the MD, CLDMA hardware asserts an
interrupt to notify the host to fetch data and dispatch it
to FSM (for handshake messages) or the port layer.
After CLDMA opening is finished, All RX GPDs are fulfilled
and ready to receive data from the device.
With this commit, probe registers the transport plane through
new mtk_pci_dev_init()/mtk_pci_dev_exit() hooks: the
transaction context, which points at the static queue info
table for the AP and modem control channels, is attached to
the control plane. Nothing is brought up yet - the FSM
listener that calls mtk_pcie_hif_init(), which builds the
per-channel queue table, the CLDMA instances and the TRB
service thread, arrives with the later
"Add FSM thread" patch; the userspace-visible port
interfaces arrive with "Add control port".
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 5 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 46 +
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 1207 +++++++++++++++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 121 +++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c | 505 +++++++++++
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 109 +++
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 34 +-
drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h | 1 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 618 ++++++++++++++
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 66 ++
10 files changed, 2708 insertions(+), 4 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index 0a07507f1ddf..74bef150e3a1 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -7,4 +7,7 @@ obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
mtk_t9xx-y := \
mtk_ctrl_plane.o \
- pcie/mtk_pci.o
+ pcie/mtk_pci.o \
+ pcie/mtk_trans_ctrl.o \
+ pcie/mtk_cldma.o \
+ pcie/mtk_cldma_drv.o
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 46f1ab9f0551..cf5d6d95f3ee 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -10,6 +10,52 @@
#include "mtk_dev.h"
+#define Q_MTU_3_5K (0xE00)
+
+enum mtk_ccci_ch {
+ /* to sAP */
+ CCCI_SAP_CONTROL_RX = 0x1000,
+ CCCI_SAP_CONTROL_TX = 0x1001,
+ /* to MD */
+ CCCI_CONTROL_RX = 0x2000,
+ CCCI_CONTROL_TX = 0x2001,
+};
+
+enum mtk_trb_cmd_type {
+ TRB_CMD_MIN,
+ TRB_CMD_ENABLE,
+ TRB_CMD_TX,
+ TRB_CMD_DISABLE,
+ TRB_CMD_STOP,
+ TRB_CMD_RECOVER,
+ TRB_CMD_MAX,
+};
+
+enum mtk_hif_dev_ctrl_cmd {
+ HIF_CTRL_CMD_CHECK_TX_FULL,
+};
+
+struct trb_open_priv {
+ u8 log_rg_offset;
+ u32 tx_mtu;
+ /* consumes the skb; the return value is informational */
+ int (*rx_done)(struct sk_buff *skb, void *priv);
+};
+
+struct trb {
+ u32 channel_id;
+ enum mtk_trb_cmd_type cmd;
+ int status;
+ void *priv;
+ int (*trb_complete)(struct sk_buff *skb);
+};
+
+static_assert(sizeof(struct trb) <= sizeof_field(struct sk_buff, cb));
+
+union ctrl_hif_cmd_data {
+ u32 rx_ch;
+};
+
struct mtk_ctrl_blk {
struct mtk_md_dev *mdev;
void *ctrl_hw_priv;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
new file mode 100644
index 000000000000..d8763cdd67da
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
@@ -0,0 +1,1207 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kdev_t.h>
+#include <linux/kernel.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+#include "mtk_pci.h"
+#include "mtk_cldma.h"
+#include "mtk_cldma_drv.h"
+#include "mtk_dev.h"
+
+#define WAIT_HWO_ROUND (10)
+#define WAIT_HWO_TIME (5)
+#define NO_BUDGET (0)
+
+static const int mtk_cldma_hw_id_tbl[NR_CLDMA] = {
+ [CLDMA0] = CLDMA0_HW_ID,
+ [CLDMA1] = CLDMA1_HW_ID,
+};
+
+static void mtk_cldma_tx_done_work(struct work_struct *work)
+{
+ struct txq *txq = container_of(work, struct txq, tx_done_work);
+ struct cldma_drv_info *drv_info;
+ struct mtk_ctrl_trans *trans;
+ struct mtk_md_dev *mdev;
+ struct sk_buff *skb;
+ struct trb_srv *srv;
+ struct tx_req *req;
+ unsigned int state;
+ bool was_starved;
+ struct trb *trb;
+ int i, hif_id;
+ u32 txqno;
+
+ drv_info = txq->drv_info;
+ hif_id = drv_info->hif_id;
+ txqno = txq->txqno;
+ mdev = drv_info->mdev;
+ trans = drv_info->cd->trans;
+
+again:
+ for (i = 0; i < txq->nr_gpds; i++) {
+ spin_lock(&txq->ring_lock);
+
+ req = txq->req_pool + txq->free_idx;
+
+ /* Ownership is decided by the slot alone: an empty slot has no
+ * skb, and one the hardware still owns has HWO set. The
+ * producer publishes both under ring_lock, so a slot can never
+ * be observed half-filled and there is no need to bound the
+ * walk by wr_idx.
+ */
+ if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
+ spin_unlock(&txq->ring_lock);
+ break;
+ }
+
+ dma_rmb(); /* read descriptor fields after HWO check */
+
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->data_len, DMA_TO_DEVICE);
+
+ skb = req->skb;
+ req->data_dma_addr = 0;
+ req->data_len = 0;
+ req->skb = NULL;
+
+ txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
+ was_starved = atomic_fetch_inc(&txq->req_budget) == NO_BUDGET;
+
+ spin_unlock(&txq->ring_lock);
+
+ trb = (struct trb *)skb->cb;
+ trb->status = 0;
+ trb->trb_complete(skb);
+
+ /* The service may already have been torn down; completing the
+ * TRB is still correct, there is just nobody left to wake.
+ */
+ srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
+ if (was_starved && srv)
+ wake_up(&srv->trb_waitq);
+ }
+
+ state = mtk_cldma_check_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+ if (state) {
+ if (unlikely(state == LINK_ERROR_VAL))
+ goto out;
+
+ mtk_cldma_clr_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+
+ cond_resched();
+
+ goto again;
+ }
+
+out:
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+}
+
+/* data_recv_len is read back from a buffer the device writes, so it is
+ * clamped against req->mtu before it reaches skb_put().
+ *
+ * Returns -EPROTO when the device over-reports so the caller drops the packet
+ * instead of delivering a malformed skb.
+ */
+static int mtk_cldma_rx_skb_adjust(struct mtk_md_dev *mdev, struct rx_req *req)
+{
+ u32 recv_len;
+ int ret = 0;
+
+ if (req->data_dma_addr) {
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ req->data_dma_addr = 0;
+ }
+ recv_len = le16_to_cpu(req->gpd->rx_gpd.data_recv_len);
+ if (recv_len > req->mtu) {
+ ret = -EPROTO;
+ recv_len = req->mtu;
+ }
+ req->skb->len = 0;
+ skb_reset_tail_pointer(req->skb);
+ skb_put(req->skb, recv_len);
+
+ req->gpd->rx_gpd.data_recv_len = 0;
+
+ return ret;
+}
+
+static int mtk_cldma_reload_rx_skb(struct mtk_md_dev *mdev, struct rx_req *req)
+{
+ int ret;
+
+ req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
+ if (!req->skb)
+ return -ENOMEM;
+
+ req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
+ req->mtu, DMA_FROM_DEVICE);
+ ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(ret)) {
+ dev_warn_ratelimited(mdev->dev, "Failed to map SKB data\n");
+ req->data_dma_addr = 0;
+ dev_kfree_skb_any(req->skb);
+ req->skb = NULL;
+ return -EFAULT;
+ }
+ req->gpd->rx_gpd.data_buff_ptr_h = cpu_to_le32((u64)req->data_dma_addr >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+
+ return 0;
+}
+
+static int mtk_cldma_check_rx_req(struct cldma_drv_info *drv_info, struct rxq *rxq)
+{
+ struct rx_req *req = rxq->req_pool + rxq->free_idx;
+ u64 curr_addr;
+ int i;
+
+ curr_addr = mtk_cldma_get_rx_curr_addr(drv_info, rxq->rxqno);
+ if (unlikely(!curr_addr))
+ return -ENXIO;
+
+ if (req->gpd_dma_addr == curr_addr)
+ return -EAGAIN;
+ for (i = 0; i < WAIT_HWO_ROUND; i++) {
+ udelay(WAIT_HWO_TIME);
+ if (!(READ_ONCE(req->gpd->rx_gpd.gpd_flags) & CLDMA_GPD_FLAG_HWO))
+ break;
+ }
+ if (i == WAIT_HWO_ROUND) {
+ dev_err((drv_info->mdev)->dev, "Failed to check HWO=0\n");
+ return -EAGAIN;
+ }
+
+ return 0;
+}
+
+static bool mtk_cldma_rx_check_again(struct rxq *rxq)
+{
+ struct cldma_drv_info *drv_info;
+ u32 state;
+ int rxqno;
+
+ drv_info = rxq->drv_info;
+ rxqno = rxq->rxqno;
+
+ state = mtk_cldma_check_intr_status(drv_info, DIR_RX,
+ rxqno, QUEUE_XFER_DONE);
+ if (!state || unlikely(state == LINK_ERROR_VAL))
+ return false;
+
+ mtk_cldma_clr_intr_status(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
+ cond_resched();
+ return true;
+}
+
+/* Point the device at the GPD it should fill next and start the queue.
+ * free_idx is owned by mtk_cldma_rx_done_work(), so this may only run
+ * from that worker, at a point where every consumed GPD is re-armed.
+ */
+static void mtk_cldma_rxq_restart(struct cldma_drv_info *drv_info, struct rxq *rxq)
+{
+ mtk_cldma_setup_start_addr(drv_info, DIR_RX, rxq->rxqno,
+ rxq->req_pool[rxq->free_idx].gpd_dma_addr);
+ mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
+}
+
+static void mtk_cldma_rx_done_work(struct work_struct *work)
+{
+ struct rx_req *req = NULL, *pre_req = NULL;
+ struct rxq *rxq = container_of(work, struct rxq, rx_done_work);
+ struct cldma_drv_info *drv_info;
+ struct mtk_md_dev *mdev;
+ struct sk_buff *rx_skb;
+ int i, ret, idx, len_err;
+
+ drv_info = rxq->drv_info;
+ mdev = drv_info->mdev;
+
+again:
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + rxq->free_idx;
+ if (!req->skb) {
+ /* An empty slot means an earlier pass could not refill
+ * it and re-queued this work to retry it.
+ */
+ if (mtk_cldma_reload_rx_skb(mdev, req))
+ goto out_retry;
+ goto arm_slot;
+ }
+
+ if (req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)
+ break;
+
+ dma_rmb(); /* read descriptor fields after HWO check */
+
+ len_err = mtk_cldma_rx_skb_adjust(mdev, req);
+ rx_skb = req->skb;
+ req->skb = NULL;
+
+ ret = mtk_cldma_reload_rx_skb(mdev, req);
+ if (ret) {
+ /* Alloc failed — recycle old buffer, drop packet. */
+ skb_trim(rx_skb, 0);
+ req->skb = rx_skb;
+ req->data_dma_addr = dma_map_single(mdev->dev,
+ rx_skb->data,
+ req->mtu,
+ DMA_FROM_DEVICE);
+ if (dma_mapping_error(mdev->dev, req->data_dma_addr)) {
+ /* An unmapped skb left in the slot would be
+ * read back as a completed descriptor.
+ */
+ dev_kfree_skb_any(rx_skb);
+ req->skb = NULL;
+ req->data_dma_addr = 0;
+ goto out_retry;
+ } else {
+ req->gpd->rx_gpd.data_buff_ptr_h =
+ cpu_to_le32((u64)req->data_dma_addr >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l =
+ cpu_to_le32(req->data_dma_addr);
+ }
+ } else if (len_err) {
+ dev_err_ratelimited(mdev->dev,
+ "Drop oversized packet on cldma%d rxq%u\n",
+ drv_info->hw_id, rxq->rxqno);
+ dev_kfree_skb_any(rx_skb);
+ } else {
+ rxq->rx_done(rx_skb, rxq->arg);
+ }
+
+arm_slot:
+ wmb(); /* ensure addr set done before HWO setup done */
+
+ idx = rxq->free_idx == 0 ? rxq->nr_gpds - 1 : rxq->free_idx - 1;
+ pre_req = rxq->req_pool + idx;
+ pre_req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+ rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
+ }
+
+ ret = mtk_cldma_check_rx_req(drv_info, rxq);
+ if (!ret)
+ goto again;
+
+ /* -ENXIO means the queue has no start address programmed, which is
+ * exactly what a restart installs, so consume need_restart first.
+ * Only the resume below is skipped in that case.
+ */
+ mutex_lock(&drv_info->rearm_lock);
+ if (!atomic_read(&rxq->need_exit)) {
+ if (atomic_xchg(&rxq->need_restart, 0))
+ mtk_cldma_rxq_restart(drv_info, rxq);
+ else if (ret != -ENXIO)
+ mtk_cldma_resume_queue(drv_info, DIR_RX, rxq->rxqno);
+ }
+ mutex_unlock(&drv_info->rearm_lock);
+
+ if (ret == -ENXIO)
+ goto out;
+
+ if (mtk_cldma_rx_check_again(rxq))
+ goto again;
+ goto out;
+
+out_retry:
+ /* The empty slot keeps the GPD before it unarmed: once the device
+ * has used up the armed ones, no XFER_DONE comes to run this work
+ * again, so retry from here. The sleep paces a failure that does
+ * not wait, such as a DMA mapping one.
+ */
+ usleep_range(1000, 2000);
+ if (!atomic_read(&rxq->need_exit))
+ queue_work(drv_info->wq, &rxq->rx_done_work);
+out:
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
+ mtk_cldma_clear_ip_busy(drv_info);
+}
+
+static struct txq *mtk_cldma_txq_alloc(struct cldma_drv_info *drv_info, struct queue_info *que)
+{
+ struct mtk_md_dev *mdev;
+ struct tx_req *next;
+ struct tx_req *req;
+ struct txq *txq;
+ int i, ret;
+
+ mdev = drv_info->mdev;
+
+ txq = kzalloc_obj(*txq);
+ if (!txq)
+ return NULL;
+
+ txq->que = que;
+ txq->drv_info = drv_info;
+ txq->txqno = txq->que->txqno;
+ txq->nr_gpds = txq->que->tx_nr_gpds;
+ atomic_set(&txq->req_budget, txq->que->tx_nr_gpds);
+ spin_lock_init(&txq->ring_lock);
+
+ txq->req_pool = kcalloc(txq->nr_gpds, sizeof(*req), GFP_KERNEL);
+ if (!txq->req_pool)
+ goto err_free_txq;
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ req->mtu = txq->que->tx_mtu;
+ req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
+ if (!req->gpd)
+ goto err_free_req;
+ }
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ next = txq->req_pool + ((i + 1) % txq->nr_gpds);
+ req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
+ req->gpd->tx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
+ req->gpd->tx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
+ }
+
+ INIT_WORK(&txq->tx_done_work, mtk_cldma_tx_done_work);
+
+ ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txq->txqno);
+ if (ret) {
+ dev_warn(mdev->dev, "Failed to stop TX queue %u before alloc\n", txq->txqno);
+ goto err_free_req;
+ }
+ /* Publish the queue before unmasking its interrupts: an interrupt
+ * that fires in between must find a valid txq, or the sources it
+ * masks are never unmasked again. Not before the stop: the error
+ * path above frees the queue, so it must never have been visible.
+ * The release pairs with the acquire load in the ISR.
+ */
+ smp_store_release(&drv_info->txq[txq->txqno], txq);
+ txq->tx_started = false;
+ mtk_cldma_setup_start_addr(drv_info, DIR_TX, txq->txqno,
+ txq->req_pool[0].gpd_dma_addr);
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_ERROR);
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_XFER_DONE);
+
+ return txq;
+
+err_free_req:
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + i;
+ if (!req->gpd)
+ break;
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ }
+ kfree(txq->req_pool);
+err_free_txq:
+ kfree(txq);
+ return NULL;
+}
+
+/* mtk_cldma_drv_init() rewrites IP-global configuration (including the interrupt
+ * mask) and mtk_cldma_drv_reset() wipes every queue of the instance, so all
+ * queues still published in drv_info have to be re-armed, not only the one
+ * that triggered the reset. A queue already unpublished (NULL slot) is being
+ * torn down and is deliberately left stopped.
+ */
+static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
+{
+ struct tx_req *req;
+ struct txq *txq;
+ struct rxq *rxq;
+ int i;
+
+ /* The caller holds rearm_lock from the reset to the end of the re-arm:
+ * no queue is kicked or resumed while the instance has no addresses.
+ */
+ lockdep_assert_held(&drv_info->rearm_lock);
+
+ mtk_cldma_drv_init(drv_info);
+
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ txq = drv_info->txq[i];
+ if (txq) {
+ spin_lock(&txq->ring_lock);
+ /* The device halts on an HWO=0 descriptor and RESUME
+ * never moves the cursor past it, so do not restart
+ * from free_idx - which recovery usually finds behind
+ * the hardware. Leave the queue stopped and let the
+ * next start_xfer() issue a fresh START.
+ */
+ req = txq->req_pool + txq->free_idx;
+ if (READ_ONCE(txq->tx_started) &&
+ !(req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
+ WRITE_ONCE(txq->tx_started, false);
+ /* Its XFER_DONE may have been lost in the
+ * reset; without this free_idx never advances.
+ */
+ queue_work(drv_info->wq, &txq->tx_done_work);
+ } else {
+ mtk_cldma_setup_start_addr(drv_info, DIR_TX, i,
+ req->gpd_dma_addr);
+ }
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_ERROR);
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_XFER_DONE);
+ if (READ_ONCE(txq->tx_started))
+ mtk_cldma_start_queue(drv_info, DIR_TX, i);
+ spin_unlock(&txq->ring_lock);
+ }
+
+ rxq = drv_info->rxq[i];
+ if (rxq) {
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_ERROR);
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_XFER_DONE);
+ /* free_idx belongs to rx_done_work, which may be halfway
+ * through a packet: let it program the start address
+ * once it reaches a consistent point.
+ */
+ atomic_set(&rxq->need_restart, 1);
+ queue_work(drv_info->wq, &rxq->rx_done_work);
+ }
+ }
+}
+
+static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
+{
+ struct mtk_md_dev *mdev;
+ struct tx_req *req;
+ struct txq *txq;
+ struct trb *trb;
+ int irq_id;
+ int i, ret;
+
+ mdev = drv_info->mdev;
+
+ txq = drv_info->txq[txqno];
+ /* Unpublish before teardown; pairs with the acquire loads of this
+ * slot. No release is needed, there is no prior store to expose.
+ */
+ WRITE_ONCE(drv_info->txq[txqno], NULL);
+ /* stop HW tx transaction; -ENODEV means the link is dead, so the
+ * device cannot be walking the ring and the free may proceed.
+ */
+ ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
+ if (ret == -ETIMEDOUT) {
+ dev_err(mdev->dev, "TX queue %u stop timed out, resetting CLDMA%d\n",
+ txqno, drv_info->hw_id);
+ /* One rearm_lock section from the reset to the re-arm: no queue
+ * may be resumed or started in between, while it has no address.
+ */
+ mutex_lock(&drv_info->rearm_lock);
+ mtk_cldma_drv_reset(drv_info);
+ ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
+ /* the reset took the whole instance down with this queue */
+ mtk_cldma_rearm_queues(drv_info);
+ mutex_unlock(&drv_info->rearm_lock);
+ }
+ txq->tx_started = false;
+
+ irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(irq_id);
+ /* flush on-going work */
+ flush_work(&txq->tx_done_work);
+ mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+ mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR);
+
+ if (ret == -ETIMEDOUT) {
+ /* Still a live DMA target: leak the ring rather than hand it
+ * back to the allocator.
+ */
+ dev_err(mdev->dev, "TX queue %u cannot be stopped, leaking its ring\n",
+ txqno);
+ return;
+ }
+
+ /* Free tx req resource. No ring_lock is taken here: txq was already
+ * unpublished from drv_info->txq[] above, so no new producer can enter,
+ * and synchronize_irq() plus flush_work() have retired the consumer.
+ */
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + txq->free_idx;
+ if (req->skb) {
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->data_len, DMA_TO_DEVICE);
+ trb = (struct trb *)req->skb->cb;
+ trb->status = -EPIPE;
+ trb->trb_complete(req->skb);
+ }
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
+ }
+
+ kfree(txq->req_pool);
+ kfree(txq);
+}
+
+static void mtk_cldma_rxq_alloc_cancel(struct cldma_drv_info *drv_info, struct rx_req *req)
+{
+ struct mtk_md_dev *mdev = drv_info->mdev;
+
+ if (req->skb) {
+ if (!dma_mapping_error(mdev->dev, req->data_dma_addr))
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ req->data_dma_addr = 0;
+ dev_kfree_skb_any(req->skb);
+ }
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+}
+
+static struct rxq *mtk_cldma_rxq_alloc(struct cldma_drv_info *drv_info, struct queue_info *que,
+ struct sk_buff *skb)
+{
+ struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_md_dev *mdev;
+ struct rx_req *next;
+ struct rx_req *req;
+ struct rxq *rxq;
+ int ret;
+ int i;
+
+ mdev = drv_info->mdev;
+
+ rxq = kzalloc_obj(*rxq);
+ if (!rxq)
+ return NULL;
+
+ rxq->que = que;
+ rxq->drv_info = drv_info;
+ rxq->rxqno = rxq->que->rxqno;
+ if (rxq->que->rx_nr_gpds < MIN_GPD_NUM) {
+ dev_err(mdev->dev,
+ "Failed to alloc cldma%d rxq%u due to gpd number < 2\n",
+ drv_info->hw_id, rxq->rxqno);
+ goto err_free_rxq;
+ }
+ rxq->nr_gpds = rxq->que->rx_nr_gpds;
+ rxq->arg = trb->priv;
+ rxq->rx_done = trb_open_priv->rx_done;
+ atomic_set(&rxq->need_exit, 0);
+ atomic_set(&rxq->need_restart, 0);
+
+ rxq->req_pool = kcalloc(rxq->nr_gpds, sizeof(*req), GFP_KERNEL);
+ if (!rxq->req_pool)
+ goto err_free_rxq;
+
+ /* setup rx request */
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ req->mtu = rxq->que->rx_mtu;
+ req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
+ if (!req->gpd)
+ goto err_free_req;
+ req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
+ if (!req->skb)
+ goto err_free_req;
+ req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
+ req->mtu, DMA_FROM_DEVICE);
+ ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(ret))
+ goto err_free_req;
+ }
+
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ next = rxq->req_pool + ((i + 1) % rxq->nr_gpds);
+ req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
+ req->gpd->rx_gpd.data_allow_len = cpu_to_le16(req->mtu);
+ req->gpd->rx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
+ req->gpd->rx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
+ req->gpd->rx_gpd.data_buff_ptr_h = cpu_to_le32((u64)(req->data_dma_addr) >> 32);
+ req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+ if (i != rxq->nr_gpds - 1)
+ req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+ }
+
+ INIT_WORK(&rxq->rx_done_work, mtk_cldma_rx_done_work);
+
+ ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxq->rxqno);
+ if (ret) {
+ dev_warn(mdev->dev, "Failed to stop RX queue %u before alloc\n", rxq->rxqno);
+ goto err_free_req;
+ }
+ /* Pairs with the acquire load in the ISR, as for txq above. */
+ smp_store_release(&drv_info->rxq[rxq->rxqno], rxq);
+ mtk_cldma_setup_start_addr(drv_info, DIR_RX,
+ rxq->rxqno, rxq->req_pool[0].gpd_dma_addr);
+ mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_ERROR);
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
+
+ return rxq;
+
+err_free_req:
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + i;
+ if (!req->gpd)
+ break;
+ mtk_cldma_rxq_alloc_cancel(drv_info, req);
+ }
+
+ kfree(rxq->req_pool);
+err_free_rxq:
+ kfree(rxq);
+ return NULL;
+}
+
+static void mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
+{
+ struct mtk_md_dev *mdev;
+ struct rx_req *req;
+ struct rxq *rxq;
+ int irq_id;
+ int i, ret;
+
+ mdev = drv_info->mdev;
+
+ rxq = drv_info->rxq[rxqno];
+ /* Unpublish before teardown; pairs with the acquire loads of this
+ * slot. No release is needed, there is no prior store to expose.
+ */
+ WRITE_ONCE(drv_info->rxq[rxqno], NULL);
+
+ /* stop HW rx transaction; -ENODEV means the link is dead, so the
+ * device cannot be walking the ring and the free may proceed.
+ */
+ atomic_set(&rxq->need_exit, 1);
+ ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
+ if (ret == -ETIMEDOUT) {
+ dev_err(mdev->dev, "RX queue %u stop timed out, resetting CLDMA%d\n",
+ rxqno, drv_info->hw_id);
+ /* One rearm_lock section from the reset to the re-arm: no queue
+ * may be resumed or started in between, while it has no address.
+ */
+ mutex_lock(&drv_info->rearm_lock);
+ mtk_cldma_drv_reset(drv_info);
+ ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
+ /* the reset took the whole instance down with this queue */
+ mtk_cldma_rearm_queues(drv_info);
+ mutex_unlock(&drv_info->rearm_lock);
+ }
+
+ irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(irq_id);
+ /* not flush_work(): the worker re-queues itself on a refill failure */
+ cancel_work_sync(&rxq->rx_done_work);
+ /* a cancelled pass would have cleared it on its way out */
+ mtk_cldma_clear_ip_busy(drv_info);
+ /* only now: a worker retired by the cancel unmasks XFER_DONE on its
+ * way out
+ */
+ mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
+ mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR);
+
+ /* The worker can resume or restart the queue between its need_exit
+ * check and the MMIO write that follows it, so cancel_work_sync() does
+ * not prove the engine was left stopped. Stopping again is idempotent.
+ */
+ if (ret != -ETIMEDOUT)
+ ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
+
+ if (ret == -ETIMEDOUT) {
+ /* Still a live DMA target: leak the ring rather than hand it
+ * back to the allocator.
+ */
+ dev_err(mdev->dev, "RX queue %u cannot be stopped, leaking its ring\n",
+ rxqno);
+ return;
+ }
+
+ /* free rx req resource */
+ for (i = 0; i < rxq->nr_gpds; i++) {
+ req = rxq->req_pool + rxq->free_idx;
+ if (!(req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO) &&
+ le16_to_cpu(req->gpd->rx_gpd.data_recv_len)) {
+ if (!mtk_cldma_rx_skb_adjust(mdev, req)) {
+ rxq->rx_done(req->skb, rxq->arg);
+ req->skb = NULL;
+ }
+ }
+ if (req->skb) {
+ if (req->data_dma_addr)
+ dma_unmap_single(mdev->dev, req->data_dma_addr,
+ req->mtu, DMA_FROM_DEVICE);
+ dev_kfree_skb_any(req->skb);
+ req->skb = NULL;
+ }
+ dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
+ rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
+ }
+
+ kfree(rxq->req_pool);
+ kfree(rxq);
+}
+
+/* The device reported a zero TX start address: it lost its queue state. */
+static int mtk_cldma_hw_recovery(struct cldma_drv_info *drv_info, struct txq *txq, u32 qno)
+{
+ u64 val;
+
+ dev_err(drv_info->mdev->dev,
+ "CLDMA%d lost its queue state, re-initializing\n",
+ drv_info->hw_id);
+
+ mtk_cldma_rearm_queues(drv_info);
+
+ /* A started queue the re-arm left stopped has no address yet, on
+ * purpose: the caller programs one. Only a dead link, or a 0 on a
+ * queue still marked started, means the re-init failed.
+ */
+ val = mtk_cldma_get_tx_start_addr(drv_info, qno);
+ if (val == U64_MAX || (!val && READ_ONCE(txq->tx_started)))
+ return -EIO;
+
+ return 0;
+}
+
+/* After a re-arm the slot at free_idx may hold a descriptor the device
+ * completed but tx_done_work has not reaped yet; a START there parks the
+ * engine on HWO=0. Start at the first slot the device still owns.
+ */
+static u32 mtk_cldma_tx_start_idx(struct txq *txq)
+{
+ u32 idx = txq->free_idx;
+ struct tx_req *req;
+ int i;
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ req = txq->req_pool + idx;
+ if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO))
+ return idx;
+ idx = (idx + 1) % txq->nr_gpds;
+ }
+
+ return txq->free_idx;
+}
+
+static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno)
+{
+ struct txq *txq;
+ int ret = 0;
+ u64 val;
+
+ txq = drv_info->txq[qno];
+
+ /* a reset in another queue's teardown must not land between the
+ * address check and the kick
+ */
+ mutex_lock(&drv_info->rearm_lock);
+ val = mtk_cldma_get_tx_start_addr(drv_info, qno);
+ if (unlikely(val == U64_MAX)) {
+ ret = -EIO;
+ goto out;
+ }
+
+ if (unlikely(!val)) {
+ ret = mtk_cldma_hw_recovery(drv_info, txq, qno);
+ if (ret)
+ goto out;
+ }
+
+ /* Hold ring_lock across programming and kicking the queue so the
+ * start address given to the device is the one free_idx still
+ * names; tx_done_work advances free_idx under the same lock.
+ */
+ spin_lock(&txq->ring_lock);
+ if (unlikely(!READ_ONCE(txq->tx_started))) {
+ mtk_cldma_setup_start_addr(drv_info, DIR_TX, qno,
+ txq->req_pool[mtk_cldma_tx_start_idx(txq)].gpd_dma_addr);
+ mtk_cldma_start_queue(drv_info, DIR_TX, qno);
+ WRITE_ONCE(txq->tx_started, true);
+ } else {
+ mtk_cldma_resume_queue(drv_info, DIR_TX, qno);
+ }
+ spin_unlock(&txq->ring_lock);
+out:
+ mutex_unlock(&drv_info->rearm_lock);
+ return ret;
+}
+
+int mtk_cldma_init(struct mtk_ctrl_trans *trans)
+{
+ struct cldma_dev *cd;
+
+ cd = kzalloc_obj(*cd);
+ if (!cd)
+ return -ENOMEM;
+
+ cd->trans = trans;
+ trans->dev = cd;
+
+ return 0;
+}
+
+void mtk_cldma_exit(struct mtk_ctrl_trans *trans)
+{
+ if (!trans->dev)
+ return;
+
+ kfree(trans->dev);
+ trans->dev = NULL;
+}
+
+static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct queue_info *que;
+ struct txq *txq;
+ struct rxq *rxq;
+ int ret = 0;
+
+ que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ if (unlikely(!que)) {
+ trb->status = -EINVAL;
+ trb->trb_complete(skb);
+ return -EINVAL;
+ }
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (!drv_info) {
+ ret = -EIO;
+ goto out;
+ }
+
+ if (que->tx_mtu == 0 || que->rx_mtu == 0) {
+ dev_err((cd->trans->mdev)->dev,
+ "Failed to enable cldma%d txq%u rxq%u due to wrong mtu\n",
+ drv_info->hw_id, que->txqno, que->rxqno);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ trb_open_priv->tx_mtu = que->tx_mtu;
+
+ if (drv_info->txq[que->txqno] || drv_info->rxq[que->rxqno]) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ txq = mtk_cldma_txq_alloc(drv_info, que);
+ if (!txq) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ rxq = mtk_cldma_rxq_alloc(drv_info, que, skb);
+ if (!rxq) {
+ ret = -ENOMEM;
+ mtk_cldma_txq_free(drv_info, txq->txqno);
+ goto out;
+ }
+
+out:
+ /* -EBUSY leaves the channel open to the submitter, so the count this
+ * ENABLE added is released by its eventual DISABLE instead.
+ */
+ if (ret && ret != -EBUSY)
+ cd->trans->usr_cnt[que->hif_id][que->txqno]--;
+
+ /* Pairs with the acquire load of status on the submitting side: the
+ * payload written to the skb above must be visible with it.
+ */
+ smp_store_release(&trb->status, ret);
+ trb->trb_complete(skb);
+
+ return ret;
+}
+
+/* Complete every submitted-but-unkicked request with an error so no TRB
+ * is stranded in the ring when the doorbell cannot be rung.
+ */
+static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info,
+ struct txq *txq, int err)
+{
+ struct mtk_ctrl_trans *trans = drv_info->cd->trans;
+ int hif_id = drv_info->hif_id;
+ u32 txqno = txq->txqno;
+ struct sk_buff *skb;
+ struct trb_srv *srv;
+ struct tx_req *req;
+ bool was_starved;
+ struct trb *trb;
+ int i;
+
+ for (i = 0; i < txq->nr_gpds; i++) {
+ spin_lock(&txq->ring_lock);
+
+ req = txq->req_pool + txq->free_idx;
+ if (!req->skb) {
+ spin_unlock(&txq->ring_lock);
+ break;
+ }
+
+ req->gpd->tx_gpd.gpd_flags &= ~CLDMA_GPD_FLAG_HWO;
+
+ dma_unmap_single(drv_info->mdev->dev, req->data_dma_addr,
+ req->data_len, DMA_TO_DEVICE);
+
+ skb = req->skb;
+ req->data_dma_addr = 0;
+ req->data_len = 0;
+ req->skb = NULL;
+
+ txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
+ was_starved = atomic_fetch_inc(&txq->req_budget) == NO_BUDGET;
+
+ spin_unlock(&txq->ring_lock);
+
+ trb = (struct trb *)skb->cb;
+ trb->status = err;
+ trb->trb_complete(skb);
+
+ /* The service may already have been torn down; completing the
+ * TRB is still correct, there is just nobody left to wake.
+ */
+ srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
+ if (was_starved && srv)
+ wake_up(&srv->trb_waitq);
+ }
+}
+
+static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct mtk_md_dev *mdev;
+ struct queue_info *que;
+ struct txq *txq;
+ int ret, err;
+
+ que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ if (unlikely(!que))
+ return -EPIPE;
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info))
+ return -EPIPE;
+ txq = drv_info->txq[que->txqno];
+ if (unlikely(!txq))
+ return -EPIPE;
+
+ mdev = drv_info->mdev;
+
+ ret = mtk_cldma_start_xfer(drv_info, que->txqno);
+ if (unlikely(ret)) {
+ dev_err(mdev->dev, "Failed to trigger cldma tx\n");
+ /* A failed start does not mean the engine is idle - recovery
+ * may already have issued UL_START - and the flush below
+ * unmaps the buffers. If the queue cannot be stopped, leak the
+ * requests rather than unmap memory the device still owns.
+ */
+ err = mtk_cldma_stop_queue(drv_info, DIR_TX, que->txqno);
+ /* -ENODEV: the link is dead, nothing is walking the ring */
+ if (err && err != -ENODEV) {
+ dev_err(mdev->dev,
+ "TX queue %u stop failed, keeping its requests\n",
+ que->txqno);
+ return ret;
+ }
+
+ spin_lock(&txq->ring_lock);
+ WRITE_ONCE(txq->tx_started, false);
+ spin_unlock(&txq->ring_lock);
+ mtk_cldma_txq_flush(drv_info, txq, ret);
+ }
+
+ return ret;
+}
+
+static int mtk_cldma_close(struct cldma_dev *cd, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct queue_info *que;
+
+ que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ if (unlikely(!que)) {
+ trb->status = -EPIPE;
+ trb->trb_complete(skb);
+ return -EPIPE;
+ }
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info)) {
+ trb->status = -EPIPE;
+ trb->trb_complete(skb);
+ return -EPIPE;
+ }
+
+ if (drv_info->txq[que->txqno])
+ mtk_cldma_txq_free(drv_info, que->txqno);
+ if (drv_info->rxq[que->rxqno])
+ mtk_cldma_rxq_free(drv_info, que->rxqno);
+
+ trb->status = 0;
+ trb->trb_complete(skb);
+
+ return 0;
+}
+
+static int mtk_cldma_txbuf_set(struct cldma_drv_info *drv_info, struct sk_buff *skb,
+ struct tx_req *req)
+{
+ struct mtk_md_dev *mdev = drv_info->mdev;
+ int ret;
+
+ /* Only the linear area is mapped; a nonlinear SKB here would be
+ * silently truncated to skb_headlen().
+ */
+ if (WARN_ON_ONCE(skb_is_nonlinear(skb)))
+ return -EINVAL;
+
+ req->data_dma_addr = dma_map_single(mdev->dev, skb->data,
+ skb_headlen(skb), DMA_TO_DEVICE);
+ ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
+ if (unlikely(ret)) {
+ req->data_dma_addr = 0;
+ dev_err_ratelimited(mdev->dev, "Failed to map dma! error:%d\n", ret);
+ return -ENOMEM;
+ }
+
+ req->gpd->tx_gpd.data_buff_ptr_h = cpu_to_le32((u64)(req->data_dma_addr) >> 32);
+ req->gpd->tx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
+
+ return 0;
+}
+
+int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct queue_info *que;
+ struct tx_req *req;
+ struct txq *txq;
+ int ret;
+
+ if (unlikely(!cd))
+ return -EINVAL;
+
+ que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
+ if (unlikely(!que))
+ return -EINVAL;
+ drv_info = cd->cldma_drv_info[que->hif_id];
+ if (unlikely(!drv_info))
+ return -EINVAL;
+
+ txq = drv_info->txq[que->txqno];
+ if (unlikely(!txq))
+ return -EINVAL;
+
+ spin_lock(&txq->ring_lock);
+
+ if (!atomic_read(&txq->req_budget)) {
+ spin_unlock(&txq->ring_lock);
+ return -EAGAIN;
+ }
+
+ req = txq->req_pool + txq->wr_idx;
+ req->gpd->tx_gpd.debug_id = 0x01;
+ ret = mtk_cldma_txbuf_set(drv_info, skb, req);
+ if (ret) {
+ spin_unlock(&txq->ring_lock);
+ return ret;
+ }
+
+ req->gpd->tx_gpd.data_buff_len = cpu_to_le16(skb->len);
+
+ req->data_len = skb->len;
+ req->skb = skb;
+
+ dma_wmb(); /* ensure req and data msg set done before HWO setup */
+
+ req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
+
+ txq->wr_idx = (txq->wr_idx + 1) % txq->nr_gpds;
+ atomic_dec(&txq->req_budget);
+
+ spin_unlock(&txq->ring_lock);
+
+ return 0;
+}
+
+int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct txq *txq;
+
+ if (unlikely(hif_id >= NR_CLDMA || qno >= HW_QUE_NUM || !cd))
+ return -EINVAL;
+
+ drv_info = cd->cldma_drv_info[hif_id];
+ if (!drv_info)
+ return -EINVAL;
+ txq = drv_info->txq[qno];
+ if (!txq)
+ return -EINVAL;
+ return atomic_read(&txq->req_budget);
+}
+
+static int (*trb_act_tbl[TRB_CMD_MAX])(struct cldma_dev *cd, struct sk_buff *skb) = {
+ [TRB_CMD_ENABLE] = mtk_cldma_open,
+ [TRB_CMD_TX] = mtk_cldma_tx,
+ [TRB_CMD_DISABLE] = mtk_cldma_close,
+};
+
+int mtk_cldma_trb_process(void *dev, struct sk_buff *skb)
+{
+ struct cldma_dev *cd;
+ struct trb *trb;
+
+ if (!dev || !skb)
+ return -EINVAL;
+
+ cd = (struct cldma_dev *)dev;
+ trb = (struct trb *)skb->cb;
+
+ if (!(trb->cmd > TRB_CMD_MIN && trb->cmd < TRB_CMD_STOP))
+ return -EINVAL;
+
+ return trb_act_tbl[trb->cmd](cd, skb);
+}
+
+int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = dev;
+ struct mtk_md_dev *mdev;
+ struct txq *txq;
+ struct rxq *rxq;
+
+ mdev = cd->trans->mdev;
+ drv_info = cd->cldma_drv_info[que->hif_id];
+
+ if (!drv_info) {
+ dev_err(mdev->dev, "CLDMA%d has not been initialized\n",
+ mtk_cldma_hw_id_tbl[que->hif_id]);
+ return -EINVAL;
+ }
+
+ txq = drv_info->txq[que->txqno];
+ rxq = drv_info->rxq[que->rxqno];
+ if (!txq || !rxq) {
+ dev_err(mdev->dev,
+ "CLDMA%d txq%u rxq%u has not been enabled\n",
+ mtk_cldma_hw_id_tbl[que->hif_id], que->txqno, que->rxqno);
+ return -EINVAL;
+ }
+
+ return 0;
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
new file mode 100644
index 000000000000..8abde07ebd55
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
@@ -0,0 +1,121 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_CLDMA_H__
+#define __MTK_CLDMA_H__
+
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/interrupt.h>
+#include <linux/list.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#include "mtk_ctrl_plane.h"
+#include "mtk_trans_ctrl.h"
+
+struct mtk_fsm_param;
+
+#define TXQ(N) (N)
+#define RXQ(N) (N)
+
+#define CLDMA_GPD_FLAG_HWO BIT(0)
+#define CLDMA_GPD_FLAG_IOC BIT(7)
+
+union gpd {
+ struct {
+ u8 gpd_flags;
+ u8 non_used1;
+ __le16 data_allow_len;
+ __le32 next_gpd_ptr_h;
+ __le32 next_gpd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_recv_len;
+ u8 non_used2;
+ u8 debug_id;
+ } rx_gpd;
+
+ struct {
+ u8 gpd_flags;
+ u8 non_used1;
+ u8 non_used2;
+ u8 debug_id;
+ __le32 next_gpd_ptr_h;
+ __le32 next_gpd_ptr_l;
+ __le32 data_buff_ptr_h;
+ __le32 data_buff_ptr_l;
+ __le16 data_buff_len;
+ __le16 non_used3;
+ } tx_gpd;
+} __packed;
+
+struct rx_req {
+ union gpd *gpd;
+ u32 mtu;
+ struct sk_buff *skb;
+ dma_addr_t gpd_dma_addr;
+ dma_addr_t data_dma_addr;
+};
+
+struct rxq {
+ struct cldma_drv_info *drv_info;
+ u32 rxqno;
+ struct queue_info *que;
+ struct work_struct rx_done_work;
+ struct rx_req *req_pool;
+ u32 nr_gpds;
+ u32 free_idx;
+ void *arg;
+ /* consumes the skb; the return value is informational */
+ int (*rx_done)(struct sk_buff *skb, void *priv);
+ atomic_t need_exit;
+ /* set when the queue must be programmed and started again after an IP
+ * reset; consumed by mtk_cldma_rx_done_work(), the owner of free_idx
+ */
+ atomic_t need_restart;
+};
+
+struct tx_req {
+ union gpd *gpd;
+ u32 mtu;
+ size_t data_len;
+ dma_addr_t data_dma_addr;
+ dma_addr_t gpd_dma_addr;
+ struct sk_buff *skb;
+};
+
+struct txq {
+ struct cldma_drv_info *drv_info;
+ u32 txqno;
+ struct queue_info *que;
+ struct work_struct tx_done_work;
+ struct tx_req *req_pool;
+ u32 nr_gpds;
+ atomic_t req_budget;
+ /* ring_lock: serializes the producer (TRB service thread) against the
+ * consumer (tx_done_work) over req_budget, wr_idx, free_idx and the
+ * tx_req/GPD slots they index.
+ */
+ spinlock_t ring_lock;
+ u32 wr_idx;
+ u32 free_idx;
+ bool tx_started;
+};
+
+struct cldma_dev {
+ struct cldma_drv_info *cldma_drv_info[NR_CLDMA];
+ struct mtk_ctrl_trans *trans;
+};
+
+int mtk_cldma_init(struct mtk_ctrl_trans *trans);
+void mtk_cldma_exit(struct mtk_ctrl_trans *trans);
+int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb);
+int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno);
+int mtk_cldma_trb_process(void *dev, struct sk_buff *skb);
+void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_ctrl_trans *trans);
+int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que);
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
new file mode 100644
index 000000000000..65e78421f9cf
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
@@ -0,0 +1,505 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/iopoll.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/kdev_t.h>
+#include <linux/kernel.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/sched.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <linux/timer.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+
+#include "mtk_cldma_drv.h"
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_pci_reg.h"
+
+#define WAIT_QUEUE_STOP (70)
+/* per-queue stride of the L/H start and current address registers */
+#define ADDR_STRIDE (8)
+
+#define CLDMA0_BASE_ADDR (0x1021C000)
+#define CLDMA1_BASE_ADDR (0x1021E000)
+
+/* CLDMA IN(Tx) */
+#define REG_CLDMA_UL_START_ADDRL_0 (0x0004)
+#define REG_CLDMA_UL_START_ADDRH_0 (0x0008)
+#define REG_CLDMA_UL_STATUS (0x0084)
+#define REG_CLDMA_UL_START_CMD (0x0088)
+#define REG_CLDMA_UL_RESUME_CMD (0x008C)
+#define REG_CLDMA_UL_STOP_CMD (0x0090)
+#define REG_CLDMA_UL_CFG (0x0098)
+#define REG_CLDMA_UL_DUMMY_0 (0x009C)
+
+/* CLDMA OUT(Rx) */
+#define REG_CLDMA_SO_START_CMD (0x0400 + 0x01BC)
+#define REG_CLDMA_SO_RESUME_CMD (0x0400 + 0x01C0)
+#define REG_CLDMA_SO_STOP_CMD (0x0400 + 0x01C4)
+#define REG_CLDMA_SO_DUMMY_0 (0x0400 + 0x0108)
+#define REG_CLDMA_SO_CFG (0x0400 + 0x0004)
+#define REG_CLDMA_SO_START_ADDRL_0 (0x0400 + 0x0078)
+#define REG_CLDMA_SO_START_ADDRH_0 (0x0400 + 0x007C)
+#define REG_CLDMA_SO_CUR_ADDRL_0 (0x0400 + 0x00B8)
+#define REG_CLDMA_SO_CUR_ADDRH_0 (0x0400 + 0x00BC)
+#define REG_CLDMA_SO_STATUS (0x0400 + 0x00F8)
+
+/* CLDMA MISC */
+#define REG_CLDMA_L2TISAR0 (0x0800 + 0x0010)
+#define REG_CLDMA_L2TIMR0 (0x0800 + 0x0018)
+#define REG_CLDMA_L2TIMCR0 (0x0800 + 0x0020)
+#define REG_CLDMA_L2TIMSR0 (0x0800 + 0x0028)
+#define REG_CLDMA_L3TISAR0 (0x0800 + 0x0030)
+#define REG_CLDMA_L3TISAR1 (0x0800 + 0x0034)
+#define REG_CLDMA_L2RISAR0 (0x0800 + 0x0050)
+#define REG_CLDMA_L3RISAR0 (0x0800 + 0x0070)
+#define REG_CLDMA_L3RISAR1 (0x0800 + 0x0074)
+#define REG_CLDMA_IP_BUSY (0x0800 + 0x00B4)
+#define REG_CLDMA_L3TISAR2 (0x0800 + 0x00C0)
+
+#define REG_CLDMA_L2RIMR0 (0x0800 + 0x00E8)
+#define REG_CLDMA_L2RIMCR0 (0x0800 + 0x00F0)
+#define REG_CLDMA_L2RIMSR0 (0x0800 + 0x00F8)
+
+#define REG_CLDMA_INT_EAP_USIP_MASK (0x0800 + 0x011C)
+
+#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET (0x0800 + 0x0198)
+#define REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR (0x0800 + 0x019C)
+
+#define REG_CLDMA_RX_WORK_TO_REG_MASK_SET (0x0800 + 0x021C)
+
+/* CLDMA RESET */
+#define REG_INFRA_RST0_SET (0x120)
+#define REG_INFRA_RST0_CLR (0x124)
+#define REG_CLDMA0_RST_SET_BIT (8)
+#define REG_CLDMA0_RST_CLR_BIT (8)
+
+const struct cldma_hw_regs mtk_cldma_regs_m9xx = {
+ .cldma0_base_addr = CLDMA0_BASE_ADDR,
+ .cldma1_base_addr = CLDMA1_BASE_ADDR,
+ .reg_cldma_ul_start_addrl_0 = REG_CLDMA_UL_START_ADDRL_0,
+ .reg_cldma_ul_start_addrh_0 = REG_CLDMA_UL_START_ADDRH_0,
+ .reg_cldma_ul_status = REG_CLDMA_UL_STATUS,
+ .reg_cldma_ul_start_cmd = REG_CLDMA_UL_START_CMD,
+ .reg_cldma_ul_resume_cmd = REG_CLDMA_UL_RESUME_CMD,
+ .reg_cldma_ul_stop_cmd = REG_CLDMA_UL_STOP_CMD,
+ .reg_cldma_ul_cfg = REG_CLDMA_UL_CFG,
+ .reg_cldma_ul_dummy_0 = REG_CLDMA_UL_DUMMY_0,
+ .reg_cldma_so_start_cmd = REG_CLDMA_SO_START_CMD,
+ .reg_cldma_so_resume_cmd = REG_CLDMA_SO_RESUME_CMD,
+ .reg_cldma_so_stop_cmd = REG_CLDMA_SO_STOP_CMD,
+ .reg_cldma_so_dummy_0 = REG_CLDMA_SO_DUMMY_0,
+ .reg_cldma_so_cfg = REG_CLDMA_SO_CFG,
+ .reg_cldma_so_start_addrl_0 = REG_CLDMA_SO_START_ADDRL_0,
+ .reg_cldma_so_start_addrh_0 = REG_CLDMA_SO_START_ADDRH_0,
+ .reg_cldma_so_current_addrl_0 = REG_CLDMA_SO_CUR_ADDRL_0,
+ .reg_cldma_so_current_addrh_0 = REG_CLDMA_SO_CUR_ADDRH_0,
+ .reg_cldma_so_status = REG_CLDMA_SO_STATUS,
+ .reg_cldma_l2tisar0 = REG_CLDMA_L2TISAR0,
+ .reg_cldma_l2timr0 = REG_CLDMA_L2TIMR0,
+ .reg_cldma_l2timcr0 = REG_CLDMA_L2TIMCR0,
+ .reg_cldma_l2timsr0 = REG_CLDMA_L2TIMSR0,
+ .reg_cldma_l3tisar0 = REG_CLDMA_L3TISAR0,
+ .reg_cldma_l3tisar1 = REG_CLDMA_L3TISAR1,
+ .reg_cldma_l3tisar2 = REG_CLDMA_L3TISAR2,
+ .reg_cldma_l2risar0 = REG_CLDMA_L2RISAR0,
+ .reg_cldma_l2rimr0 = REG_CLDMA_L2RIMR0,
+ .reg_cldma_l2rimcr0 = REG_CLDMA_L2RIMCR0,
+ .reg_cldma_l2rimsr0 = REG_CLDMA_L2RIMSR0,
+ .reg_cldma_l3risar0 = REG_CLDMA_L3RISAR0,
+ .reg_cldma_l3risar1 = REG_CLDMA_L3RISAR1,
+ .reg_cldma_ip_busy = REG_CLDMA_IP_BUSY,
+ .reg_cldma_int_mask = REG_CLDMA_INT_EAP_USIP_MASK,
+ .reg_cldma_ip_busy_to_pcie_mask_set = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_SET,
+ .reg_cldma_ip_busy_to_pcie_mask_clr = REG_CLDMA_IP_BUSY_TO_PCIE_MASK_CLR,
+ .reg_cldma_rx_work_to_reg_mask_set = REG_CLDMA_RX_WORK_TO_REG_MASK_SET,
+ .reg_infra_rst0_set = REG_INFRA_RST0_SET,
+ .reg_infra_rst0_clr = REG_INFRA_RST0_CLR,
+};
+
+void mtk_cldma_drv_reset(struct cldma_drv_info *drv_info)
+{
+ const struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ u32 val;
+
+ mdev = drv_info->mdev;
+ hw_regs = drv_info->hw_regs;
+
+ val = 1 << (REG_CLDMA0_RST_SET_BIT + drv_info->hw_id);
+ mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_set, val);
+
+ /* Read back to flush the posted write, so the reset is in effect at
+ * the device before the pulse is timed and before it is released.
+ */
+ mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_set);
+ udelay(1);
+
+ val = 1 << (REG_CLDMA0_RST_CLR_BIT + drv_info->hw_id);
+ mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr, val);
+ mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr);
+}
+
+void mtk_cldma_drv_init(struct cldma_drv_info *drv_info)
+{
+ const struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ int base;
+ u32 val;
+
+ mdev = drv_info->mdev;
+ base = drv_info->base_addr;
+ hw_regs = drv_info->hw_regs;
+
+ /* set CLDMA to 64 bit mode GPD */
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_ul_cfg);
+ val = (val & (~(0x7 << 5))) | ((0x4) << 5);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_cfg, val);
+
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_so_cfg);
+ val = (val & (~(0x7 << 10))) | ((0x4) << 10) | (1 << 2);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_cfg, val);
+
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_rx_work_to_reg_mask_set, ALLQ);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_set,
+ ALLQ << 16);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ip_busy_to_pcie_mask_clr,
+ ALLQ << 24);
+
+ /* enable interrupt to PCIe */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
+
+ /* disable illegal memory check */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_dummy_0, 1);
+}
+
+void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, dma_addr_t addr)
+{
+ const struct cldma_hw_regs *hw_regs;
+ unsigned int addr_l;
+ unsigned int addr_h;
+ int base;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX) {
+ addr_l = base + hw_regs->reg_cldma_ul_start_addrl_0 + qno * ADDR_STRIDE;
+ addr_h = base + hw_regs->reg_cldma_ul_start_addrh_0 + qno * ADDR_STRIDE;
+ } else {
+ addr_l = base + hw_regs->reg_cldma_so_start_addrl_0 + qno * ADDR_STRIDE;
+ addr_h = base + hw_regs->reg_cldma_so_start_addrh_0 + qno * ADDR_STRIDE;
+ }
+
+ mtk_pci_write32(drv_info->mdev, addr_l, (u32)addr);
+ mtk_pci_write32(drv_info->mdev, addr_h, (u32)((u64)addr >> 32));
+}
+
+void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ const struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2timsr0;
+ else
+ addr = base + hw_regs->reg_cldma_l2rimsr0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ const struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2timcr0;
+ else
+ addr = base + hw_regs->reg_cldma_l2rimcr0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ const struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+ mdev = drv_info->mdev;
+
+ if (type == QUEUE_ERROR) {
+ if (dir == DIR_TX) {
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar0);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar0, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar1, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3tisar2);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3tisar2, val);
+ } else {
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar0);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar0, val);
+ val = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l3risar1);
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l3risar1, val);
+ }
+ }
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2tisar0;
+ else
+ addr = base + hw_regs->reg_cldma_l2risar0;
+
+ if (qno == ALLQ)
+ val = qno << type;
+ else
+ val = BIT(qno) << type;
+
+ mtk_pci_write32(mdev, addr, val);
+ val = mtk_pci_read32(mdev, addr);
+}
+
+u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type)
+{
+ const struct cldma_hw_regs *hw_regs;
+ u32 addr, val, sta;
+ int base;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_l2tisar0;
+ else
+ addr = base + hw_regs->reg_cldma_l2risar0;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+ if (val == LINK_ERROR_VAL)
+ sta = val;
+ else if (qno == ALLQ)
+ sta = (val >> type) & 0xFF;
+ else
+ sta = (val >> type) & BIT(qno);
+
+ return sta;
+}
+
+void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ const struct cldma_hw_regs *hw_regs;
+ u32 val = BIT(qno);
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_start_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_start_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ const struct cldma_hw_regs *hw_regs;
+ u32 val = BIT(qno);
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_resume_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_resume_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+}
+
+static u32 mtk_cldma_queue_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ const struct cldma_hw_regs *hw_regs;
+ int base;
+ u32 addr;
+ u32 val;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_status;
+ else
+ addr = base + hw_regs->reg_cldma_so_status;
+
+ val = mtk_pci_read32(drv_info->mdev, addr);
+
+ if (qno == ALLQ || val == LINK_ERROR_VAL)
+ return val;
+
+ return val & BIT(qno);
+}
+
+int mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno)
+{
+ u32 val = (qno == ALLQ) ? qno : BIT(qno);
+ const struct cldma_hw_regs *hw_regs;
+ unsigned int active;
+ int base;
+ u32 addr;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+
+ if (dir == DIR_TX)
+ addr = base + hw_regs->reg_cldma_ul_stop_cmd;
+ else
+ addr = base + hw_regs->reg_cldma_so_stop_cmd;
+
+ mtk_pci_write32(drv_info->mdev, addr, val);
+
+ if (read_poll_timeout(mtk_cldma_queue_status, active,
+ active == LINK_ERROR_VAL || !active,
+ WAIT_QUEUE_STOP, WAIT_QUEUE_STOP * 10, false,
+ drv_info, dir, qno))
+ return -ETIMEDOUT;
+
+ /* An all-ones read means the link is dead, not that the queue
+ * stopped; let the caller tell the two apart.
+ */
+ if (active == LINK_ERROR_VAL)
+ return -ENODEV;
+
+ return 0;
+}
+
+void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info)
+{
+ mtk_pci_write32(drv_info->mdev, drv_info->base_addr +
+ drv_info->hw_regs->reg_cldma_ip_busy, 0x01);
+}
+
+void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_sta, u32 *rx_sta)
+{
+ const struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ u32 tx_mask, rx_mask;
+ int base;
+
+ mdev = drv_info->mdev;
+ base = drv_info->base_addr;
+ hw_regs = drv_info->hw_regs;
+
+ *tx_sta = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2tisar0);
+ tx_mask = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2timr0);
+ *rx_sta = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2risar0);
+ rx_mask = mtk_pci_read32(mdev, base + hw_regs->reg_cldma_l2rimr0);
+
+ *tx_sta = (*tx_sta) & (~tx_mask);
+ *rx_sta = (*rx_sta) & (~rx_mask);
+
+ if (*tx_sta) {
+ /* TX XFER_DONE and QUEUE_ERROR mask */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2timsr0, *tx_sta);
+ /* TX XFER_DONE clear */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2tisar0,
+ (*tx_sta) & (0xFF << QUEUE_XFER_DONE));
+ }
+
+ if (*rx_sta) {
+ /* RX XFER_DONE and QUEUE_ERROR mask */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2rimsr0, *rx_sta);
+ /* RX XFER_DONE clear */
+ mtk_pci_write32(mdev, base + hw_regs->reg_cldma_l2risar0,
+ (*rx_sta) & (0xFF << QUEUE_XFER_DONE));
+ }
+}
+
+u64 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno)
+{
+ const struct cldma_hw_regs *hw_regs = drv_info->hw_regs;
+ struct mtk_md_dev *mdev = drv_info->mdev;
+ int base = drv_info->base_addr;
+ u32 addr_h, addr_l;
+
+ addr_l = mtk_pci_read32(mdev,
+ base + hw_regs->reg_cldma_ul_start_addrl_0 + qno * ADDR_STRIDE);
+ addr_h = mtk_pci_read32(mdev,
+ base + hw_regs->reg_cldma_ul_start_addrh_0 + qno * ADDR_STRIDE);
+
+ return ((u64)addr_h << 32) | addr_l;
+}
+
+u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno)
+{
+ const struct cldma_hw_regs *hw_regs;
+ u32 curr_addr_h, curr_addr_l;
+ struct mtk_md_dev *mdev;
+ u64 addr_h, addr_l;
+ int base;
+
+ hw_regs = drv_info->hw_regs;
+ base = drv_info->base_addr;
+ mdev = drv_info->mdev;
+
+ addr_h = base + hw_regs->reg_cldma_so_current_addrh_0 + (u64)qno * ADDR_STRIDE;
+ addr_l = base + hw_regs->reg_cldma_so_current_addrl_0 + (u64)qno * ADDR_STRIDE;
+ curr_addr_h = mtk_pci_read32(mdev, addr_h);
+ curr_addr_l = mtk_pci_read32(mdev, addr_l);
+ /* the cursor may have moved to a GPD with another high word in between */
+ if (mtk_pci_read32(mdev, addr_h) != curr_addr_h) {
+ curr_addr_h = mtk_pci_read32(mdev, addr_h);
+ curr_addr_l = mtk_pci_read32(mdev, addr_l);
+ }
+ /* no GPD address has an all-ones half, so either one means link loss */
+ if (curr_addr_h == LINK_ERROR_VAL || curr_addr_l == LINK_ERROR_VAL)
+ return 0;
+ return ((u64)curr_addr_h << 32) | curr_addr_l;
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
new file mode 100644
index 000000000000..f7b5018d3a73
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
@@ -0,0 +1,109 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2023, MediaTek Inc.
+ */
+
+#ifndef __MTK_CLDMA_DRV_H__
+#define __MTK_CLDMA_DRV_H__
+
+#include "mtk_trans_ctrl.h"
+
+#define ALLQ (0xFF)
+#define LINK_ERROR_VAL (0xFFFFFFFF)
+#define CLDMA0_HW_ID (0)
+#define CLDMA1_HW_ID (1)
+
+struct cldma_hw_regs {
+ u16 reg_cldma_so_cfg;
+ u16 reg_cldma_so_start_addrl_0;
+ u16 reg_cldma_so_start_addrh_0;
+ u16 reg_cldma_so_current_addrl_0;
+ u16 reg_cldma_so_current_addrh_0;
+ u16 reg_cldma_so_status;
+ u16 reg_cldma_l2rimr0;
+ u16 reg_cldma_l2rimcr0;
+ u16 reg_cldma_l2rimsr0;
+ u16 reg_cldma_int_mask;
+ u16 reg_cldma_ip_busy_to_pcie_mask_set;
+ u16 reg_cldma_ip_busy_to_pcie_mask_clr;
+ u16 reg_cldma_rx_work_to_reg_mask_set;
+ u16 reg_infra_rst0_set;
+ u16 reg_infra_rst0_clr;
+ u32 cldma0_base_addr;
+ u32 cldma1_base_addr;
+ u32 reg_cldma_ul_start_addrl_0;
+ u32 reg_cldma_ul_start_addrh_0;
+ u32 reg_cldma_ul_status;
+ u32 reg_cldma_ul_start_cmd;
+ u32 reg_cldma_ul_resume_cmd;
+ u32 reg_cldma_ul_stop_cmd;
+ u32 reg_cldma_ul_cfg;
+ u32 reg_cldma_ul_dummy_0;
+ u32 reg_cldma_so_start_cmd;
+ u32 reg_cldma_so_resume_cmd;
+ u32 reg_cldma_so_stop_cmd;
+ u32 reg_cldma_so_dummy_0;
+ u32 reg_cldma_l2tisar0;
+ u32 reg_cldma_l2timr0;
+ u32 reg_cldma_l2timcr0;
+ u32 reg_cldma_l2timsr0;
+ u32 reg_cldma_l2risar0;
+ u32 reg_cldma_l3tisar0;
+ u32 reg_cldma_l3tisar1;
+ u32 reg_cldma_l3tisar2;
+ u32 reg_cldma_l3risar0;
+ u32 reg_cldma_l3risar1;
+ u32 reg_cldma_ip_busy;
+};
+
+enum mtk_intr_type {
+ QUEUE_XFER_DONE = 0,
+ QUEUE_ERROR = 16,
+};
+
+enum mtk_tx_rx {
+ DIR_TX,
+ DIR_RX,
+};
+
+struct cldma_drv_info {
+ int hif_id;
+ int hw_id;
+ int base_addr;
+ int pci_ext_irq_id;
+ struct mtk_md_dev *mdev;
+ struct cldma_dev *cd;
+ struct txq *txq[HW_QUE_NUM];
+ struct rxq *rxq[HW_QUE_NUM];
+ struct dma_pool *gpd_dma_pool;
+ struct workqueue_struct *wq;
+ const struct cldma_hw_regs *hw_regs;
+ /* serialises the instance reset and re-arm against queue kicks and
+ * RX resume/restart: no queue is kicked while it has no address
+ */
+ struct mutex rearm_lock;
+};
+
+extern const struct cldma_hw_regs mtk_cldma_regs_m9xx;
+
+void mtk_cldma_drv_init(struct cldma_drv_info *drv_info);
+void mtk_cldma_drv_reset(struct cldma_drv_info *drv_info);
+void mtk_cldma_setup_start_addr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, dma_addr_t addr);
+void mtk_cldma_mask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_unmask_intr(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_clr_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+u32 mtk_cldma_check_intr_status(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir,
+ u32 qno, enum mtk_intr_type type);
+void mtk_cldma_start_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+void mtk_cldma_resume_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+int mtk_cldma_stop_queue(struct cldma_drv_info *drv_info, enum mtk_tx_rx dir, u32 qno);
+void mtk_cldma_clear_ip_busy(struct cldma_drv_info *drv_info);
+void mtk_cldma_get_intr_status(struct cldma_drv_info *drv_info, u32 *tx_sta, u32 *rx_sta);
+u64 mtk_cldma_get_tx_start_addr(struct cldma_drv_info *drv_info, u32 qno);
+u64 mtk_cldma_get_rx_curr_addr(struct cldma_drv_info *drv_info, u32 qno);
+
+#endif
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index 27785092b2f7..68e39ea4f69d 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -854,6 +854,24 @@ static void mtk_pci_free_irq(struct mtk_md_dev *mdev)
priv->irq_cnt = 0;
}
+static int mtk_pci_dev_init(struct mtk_md_dev *mdev)
+{
+ int ret;
+
+ ret = mtk_trans_ctrl_init(mdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void mtk_pci_dev_exit(struct mtk_md_dev *mdev)
+{
+ mtk_trans_ctrl_exit(mdev);
+}
+
static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
struct device *dev = &pdev->dev;
@@ -916,21 +934,27 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
if (ret)
goto free_mhccif;
+ ret = mtk_pci_dev_init(mdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to init dev.\n");
+ goto free_irq;
+ }
+
if (!mtk_pci_link_check(mdev)) {
ret = -ENOLINK;
- goto free_irq;
+ goto dev_exit;
}
ret = pci_save_state(pdev);
if (ret) {
dev_err(mdev->dev, "Failed to save PCI state: %d\n", ret);
- goto free_irq;
+ goto dev_exit;
}
priv->saved_state = pci_store_saved_state(pdev);
if (!priv->saved_state) {
ret = -ENOMEM;
- goto free_irq;
+ goto dev_exit;
}
pci_set_master(pdev);
@@ -938,6 +962,8 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
return 0;
+dev_exit:
+ mtk_pci_dev_exit(mdev);
free_irq:
mtk_pci_free_irq(mdev);
free_mhccif:
@@ -959,6 +985,8 @@ static void mtk_pci_remove(struct pci_dev *pdev)
struct device *dev = &pdev->dev;
int i, ret;
+ mtk_pci_dev_exit(mdev);
+
/* Silence every source before tearing anything down. */
mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
index a97ad6630dcf..f65a8065d945 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci_reg.h
@@ -16,6 +16,7 @@
#define REG_IMASK_HOST_MSIX_SET_GRP0_0 0x3000
#define REG_IMASK_HOST_MSIX_CLR_GRP0_0 0x3080
#define REG_IMASK_HOST_MSIX_GRP0_0 0x3100
+#define REG_DEV_INFRA_BASE 0x10001000
/* mhccif registers */
#define MHCCIF_RC2EP_SW_BSY 0x4
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
new file mode 100644
index 000000000000..826e548b2d4d
--- /dev/null
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -0,0 +1,618 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/device.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/sched.h>
+#include <linux/wait.h>
+
+#include "mtk_cldma.h"
+#include "mtk_ctrl_plane.h"
+#include "mtk_dev.h"
+#include "mtk_pci.h"
+#include "mtk_trans_ctrl.h"
+
+#define QUEUE_CHL_MASK 0xFFFF
+#define TRB_SRV_NUM (1)
+
+static const int mtk_srv_cfg[NR_CLDMA][HW_QUE_NUM] = {
+ {0},
+ {0},
+};
+
+/* the number of RX GPDs should be at least two */
+static const struct queue_info mtk_queue_info[] = {
+ {CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
+ {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
+};
+
+static bool mtk_queue_list_is_full(struct mtk_ctrl_trans *trans, struct queue_info *que)
+{
+ return skb_queue_len_lockless(&trans->trans_list[que->hif_id].skb_list[que->txqno]) >=
+ SKB_LIST_MAX_LEN;
+}
+
+static bool mtk_ctrl_chs_is_busy_or_empty(struct trb_srv *srv)
+{
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list) {
+ struct sk_buff *skb;
+ struct trb *trb;
+
+ skb = skb_peek(&srv->trans->trans_list[i].skb_list[srv_que->qno]);
+ if (!skb)
+ continue;
+
+ /* ENABLE and DISABLE use no TX descriptor; gating them on
+ * TX budget would make a queue that cannot drain
+ * impossible to close.
+ */
+ trb = (struct trb *)skb->cb;
+ if (trb->cmd != TRB_CMD_TX ||
+ mtk_cldma_get_tx_budget(srv->trans->dev, i, srv_que->qno))
+ return false;
+ }
+ }
+
+ return true;
+}
+
+/* With @to_enable, stop at an ENABLE: it and what follows belong to the
+ * next user of the queue.
+ */
+static void mtk_ctrl_ch_flush(struct sk_buff_head *skb_list, bool to_enable)
+{
+ struct sk_buff *skb;
+ struct trb *trb;
+
+ while ((skb = skb_dequeue(skb_list))) {
+ trb = (struct trb *)skb->cb;
+ if (to_enable && trb->cmd == TRB_CMD_ENABLE) {
+ skb_queue_head(skb_list, skb);
+ break;
+ }
+ trb->status = -EIO;
+ trb->trb_complete(skb);
+ }
+}
+
+static void mtk_ctrl_chs_flush(struct trb_srv *srv)
+{
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list)
+ mtk_ctrl_ch_flush(&srv->trans->trans_list[i].skb_list[srv_que->qno],
+ false);
+}
+
+static int mtk_ch_status_check(struct mtk_ctrl_trans *trans, struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct trb_open_priv *trb_open_priv;
+ struct queue_info *que;
+ int ret = 0;
+
+ que = xa_load(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_MASK);
+
+ switch (trb->cmd) {
+ case TRB_CMD_ENABLE:
+ trb_open_priv = (struct trb_open_priv *)skb->data;
+ trb_open_priv->log_rg_offset = que->log_rg_offset;
+ /* Not unwound here. A rejection below is balanced by the
+ * DISABLE the user issues for every ENABLE failure except
+ * -EBUSY; a failure inside mtk_cldma_open() is unwound there,
+ * and that DISABLE then finds the count at zero and is refused.
+ */
+ trans->usr_cnt[que->hif_id][que->txqno]++;
+ if (trans->usr_cnt[que->hif_id][que->txqno] == 1)
+ break;
+ trb_open_priv->tx_mtu = que->tx_mtu;
+ if (mtk_cldma_check_ch_cfg(trans->dev, que)) {
+ trb->status = -EINVAL;
+ ret = -EINVAL;
+ } else {
+ trb->status = -EBUSY;
+ ret = -EBUSY;
+ }
+ trb->trb_complete(skb);
+ break;
+ case TRB_CMD_DISABLE:
+ if (trans->usr_cnt[que->hif_id][que->txqno] > 0) {
+ trans->usr_cnt[que->hif_id][que->txqno]--;
+ if (!trans->usr_cnt[que->hif_id][que->txqno])
+ break;
+ }
+ trb->status = -EBUSY;
+ trb->trb_complete(skb);
+ ret = -EBUSY;
+ break;
+ default:
+ dev_err((trans->mdev)->dev, "Invalid trb command(%d)\n", trb->cmd);
+ ret = -EINVAL;
+ break;
+ }
+ return ret;
+}
+
+/* Single consumer per srv_que — only this kthread dequeues from skb_list.
+ * This function holds the list lock around every list read (peek, is_last,
+ * peek_next, unlink) so a producer inserting a DISABLE at the head cannot race the
+ * traversal, but it is dropped across submit and dispatch: those paths
+ * may allocate with GFP_KERNEL and thus sleep. Dropping the lock there
+ * is safe because no other consumer can steal the peeked skb.
+ */
+static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_list, u32 qno)
+{
+ struct sk_buff_head *skb_list = &trans_list->skb_list[qno];
+ struct mtk_ctrl_trans *trans = srv->trans;
+ struct sk_buff *skb, *skb_next;
+ struct trb *trb, *trb_next;
+ unsigned long flags;
+ bool kick = false;
+ int loop = 0;
+ int err;
+
+ do {
+ spin_lock_irqsave(&skb_list->lock, flags);
+ skb = skb_peek(skb_list);
+ if (!skb) {
+ spin_unlock_irqrestore(&skb_list->lock, flags);
+ break;
+ }
+ trb = (struct trb *)skb->cb;
+
+ switch (trb->cmd) {
+ case TRB_CMD_ENABLE:
+ case TRB_CMD_DISABLE:
+ __skb_unlink(skb, skb_list);
+ spin_unlock_irqrestore(&skb_list->lock, flags);
+ err = mtk_ch_status_check(trans, skb);
+ if (!err) {
+ kick = true;
+ if (trb->cmd == TRB_CMD_DISABLE)
+ mtk_ctrl_ch_flush(skb_list, true);
+ }
+ break;
+ case TRB_CMD_TX:
+ spin_unlock_irqrestore(&skb_list->lock, flags);
+ err = mtk_cldma_submit_tx(trans->dev, skb);
+ if (err) {
+ if (trans_list->tx_burst_cnt[qno]) {
+ kick = true;
+ break;
+ }
+ if (err == -EAGAIN)
+ return;
+
+ skb_unlink(skb, skb_list);
+ trb->status = err;
+ trb->trb_complete(skb);
+ break;
+ }
+
+ trans_list->tx_burst_cnt[qno]++;
+ spin_lock_irqsave(&skb_list->lock, flags);
+ if (trans_list->tx_burst_cnt[qno] >= TX_BURST_MAX_CNT ||
+ skb_queue_is_last(skb_list, skb)) {
+ kick = true;
+ } else {
+ skb_next = skb_peek_next(skb, skb_list);
+ trb_next = (struct trb *)skb_next->cb;
+ if (trb_next->cmd != TRB_CMD_TX)
+ kick = true;
+ }
+
+ __skb_unlink(skb, skb_list);
+ spin_unlock_irqrestore(&skb_list->lock, flags);
+ break;
+ default:
+ __skb_unlink(skb, skb_list);
+ spin_unlock_irqrestore(&skb_list->lock, flags);
+ trb->status = -EINVAL;
+ trb->trb_complete(skb);
+ break;
+ }
+
+ if (kick) {
+ err = mtk_cldma_trb_process(trans->dev, skb);
+ if (err)
+ dev_err_ratelimited((trans->mdev)->dev,
+ "Failed to process trb on queue %u: %d\n",
+ qno, err);
+ trans_list->tx_burst_cnt[qno] = 0;
+ kick = false;
+ }
+
+ loop++;
+ } while (loop < TRB_NUM_PER_ROUND);
+}
+
+static void mtk_ctrl_trb_process(struct trb_srv *srv)
+{
+ struct mtk_ctrl_trans *trans = srv->trans;
+ struct srv_que *srv_que;
+ int i;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ list_for_each_entry(srv_que, &srv->srv_q_list[i], list)
+ mtk_ctrl_trb_handler(srv, &trans->trans_list[i], srv_que->qno);
+}
+
+static int mtk_ctrl_trb_thread(void *args)
+{
+ struct trb_srv *srv = args;
+
+ for (;;) {
+ wait_event_interruptible(srv->trb_waitq,
+ !mtk_ctrl_chs_is_busy_or_empty(srv) ||
+ kthread_should_stop() || kthread_should_park());
+ if (kthread_should_stop())
+ break;
+
+ if (kthread_should_park())
+ kthread_parkme();
+
+ do {
+ mtk_ctrl_trb_process(srv);
+ cond_resched();
+ } while (!mtk_ctrl_chs_is_busy_or_empty(srv) && !kthread_should_stop() &&
+ !kthread_should_park());
+ }
+ return 0;
+}
+
+static int mtk_ctrl_trb_srv_init(struct mtk_ctrl_trans *trans)
+{
+ struct srv_que *srv_que;
+ struct trb_srv *srv;
+ int i, j;
+ int ret;
+
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ srv = kzalloc_obj(*srv);
+ if (!srv) {
+ ret = -ENOMEM;
+ goto err_free_srv;
+ }
+
+ srv->trans = trans;
+ trans->trb_srv[i] = srv;
+
+ init_waitqueue_head(&srv->trb_waitq);
+ for (j = 0; j < NR_CLDMA; j++)
+ INIT_LIST_HEAD(&srv->srv_q_list[j]);
+ }
+
+ for (i = 0; i < NR_CLDMA; i++)
+ for (j = 0; j < HW_QUE_NUM; j++) {
+ if (trans->srv_cfg[i][j] < 0 ||
+ trans->srv_cfg[i][j] >= trans->trb_srv_num)
+ trans->srv_cfg[i][j] = 0;
+ srv_que = kzalloc_obj(*srv_que);
+ if (!srv_que) {
+ ret = -ENOMEM;
+ goto err_free_srv_que;
+ }
+ srv_que->hif_id = i;
+ srv_que->qno = j;
+ list_add_tail(&srv_que->list,
+ &trans->trb_srv[trans->srv_cfg[i][j]]->srv_q_list[i]);
+ }
+
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ trans->trb_srv[i]->trb_thread = kthread_run(mtk_ctrl_trb_thread, trans->trb_srv[i],
+ "mtk_trb_srv%d_%s", i,
+ trans->mdev->dev_str);
+ if (IS_ERR(trans->trb_srv[i]->trb_thread)) {
+ ret = PTR_ERR(trans->trb_srv[i]->trb_thread);
+ trans->trb_srv[i]->trb_thread = NULL;
+ goto err_stop_kthread;
+ }
+ }
+
+ return 0;
+err_stop_kthread:
+ while (--i >= 0)
+ kthread_stop(trans->trb_srv[i]->trb_thread);
+err_free_srv_que:
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ for (j = 0; j < NR_CLDMA; j++) {
+ struct srv_que *next_srv_que;
+
+ list_for_each_entry_safe(srv_que, next_srv_que,
+ &trans->trb_srv[i]->srv_q_list[j], list) {
+ list_del(&srv_que->list);
+ kfree(srv_que);
+ }
+ }
+ }
+err_free_srv:
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ if (!trans->trb_srv[i])
+ break;
+ kfree(trans->trb_srv[i]);
+ trans->trb_srv[i] = NULL;
+ }
+
+ return ret;
+}
+
+static void mtk_ctrl_trb_srv_exit(struct mtk_ctrl_trans *trans)
+{
+ struct srv_que *srv_que, *next_srv_que;
+ struct trb_srv *srv;
+ int i, j;
+
+ for (i = 0; i < trans->trb_srv_num; i++) {
+ srv = trans->trb_srv[i];
+ if (!srv)
+ continue;
+ kthread_stop(srv->trb_thread);
+ mtk_ctrl_chs_flush(srv);
+ for (j = 0; j < NR_CLDMA; j++) {
+ list_for_each_entry_safe(srv_que, next_srv_que,
+ &trans->trb_srv[i]->srv_q_list[j], list) {
+ list_del(&srv_que->list);
+ kfree(srv_que);
+ }
+ }
+ trans->trb_srv[i] = NULL;
+ kfree(srv);
+ }
+}
+
+static void mtk_ctrl_remove_queue_tbl(struct mtk_ctrl_trans *trans)
+{
+ struct queue_info *queue;
+ unsigned long index;
+
+ xa_for_each(&trans->queue_tbl, index, queue) {
+ xa_erase(&trans->queue_tbl, index);
+ kfree(queue);
+ }
+ xa_destroy(&trans->queue_tbl);
+}
+
+int mtk_pcie_hif_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ const struct queue_info *queue_info;
+ struct queue_info *queue;
+ struct mtk_ctrl_trans *trans;
+ int i, j;
+ int ret;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ queue_info = trans->queue_info;
+
+ xa_init(&trans->queue_tbl);
+ for (i = 0; i < trans->queue_info_num; i++) {
+ queue = kmemdup(queue_info + i, sizeof(*queue), GFP_KERNEL);
+ if (!queue) {
+ ret = -ENOMEM;
+ goto err_free_queue_tbl;
+ }
+ if (queue->txqno >= HW_QUE_NUM || queue->rxqno >= HW_QUE_NUM ||
+ queue->hif_id >= NR_CLDMA) {
+ dev_err(mdev->dev, "Failed to get correct queue info %x\n",
+ queue->rx_chl);
+ kfree(queue);
+ ret = -EINVAL;
+ goto err_free_queue_tbl;
+ }
+ ret = xa_insert(&trans->queue_tbl, queue->rx_chl & QUEUE_CHL_MASK, queue,
+ GFP_KERNEL);
+ if (ret) {
+ dev_err(mdev->dev, "Insert %x fail, ret: %d", queue->rx_chl, ret);
+ kfree(queue);
+ goto err_free_queue_tbl;
+ }
+ }
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ for (j = 0; j < HW_QUE_NUM; j++) {
+ skb_queue_head_init(&trans->trans_list[i].skb_list[j]);
+ trans->trans_list[i].tx_burst_cnt[j] = 0;
+ /* usr_cnt tracks the queues rebuilt by mtk_cldma_init()
+ * below, so it must be reset with them. Otherwise a
+ * count left over from a torn-down cycle makes the
+ * channel permanently unopenable.
+ */
+ trans->usr_cnt[i][j] = 0;
+ }
+ }
+ ret = mtk_cldma_init(trans);
+ if (ret)
+ goto err_free_queue_tbl;
+
+ ret = mtk_ctrl_trb_srv_init(trans);
+ if (ret)
+ goto err_cldma_exit;
+
+ mutex_lock(&trans->submit_lock);
+ trans->available = true;
+ mutex_unlock(&trans->submit_lock);
+
+ return 0;
+
+err_cldma_exit:
+ mtk_cldma_exit(trans);
+err_free_queue_tbl:
+ mtk_ctrl_remove_queue_tbl(trans);
+
+ return ret;
+}
+
+int mtk_pcie_hif_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+
+ mutex_lock(&trans->submit_lock);
+ trans->available = false;
+ mutex_unlock(&trans->submit_lock);
+
+ /* Join the service threads before freeing what they dereference: they
+ * read trans->dev without a lock, so clearing it cannot stop a
+ * consumer that has already loaded the pointer.
+ */
+ mtk_ctrl_trb_srv_exit(trans);
+ mtk_cldma_exit(trans);
+
+ /* Late submitters may still hold the lock and walk the tree. */
+ mutex_lock(&trans->submit_lock);
+ mtk_ctrl_remove_queue_tbl(trans);
+ mutex_unlock(&trans->submit_lock);
+
+ return 0;
+}
+
+int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct queue_info *que;
+ struct trb *trb;
+ int ret;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ trb = (struct trb *)skb->cb;
+
+ if (trb->cmd == TRB_CMD_STOP || trb->cmd == TRB_CMD_RECOVER) {
+ trb->trb_complete(skb);
+ return 0;
+ }
+
+ mutex_lock(&trans->submit_lock);
+
+ if (!trans->available) {
+ ret = -EIO;
+ goto unlock;
+ }
+
+ que = xa_load(&trans->queue_tbl, trb->channel_id & QUEUE_CHL_MASK);
+ if (!que) {
+ dev_warn(mdev->dev, "lookup que fail, ch_id: %x\n",
+ trb->channel_id);
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ if (mtk_queue_list_is_full(trans, que) && !force_send) {
+ ret = -EAGAIN;
+ goto unlock;
+ }
+
+ if (trb->cmd == TRB_CMD_DISABLE) {
+ struct sk_buff *last_enable = NULL;
+ struct sk_buff_head *list;
+ struct sk_buff *iter;
+ unsigned long flags;
+
+ /* A disable may overtake queued data, so teardown does not
+ * wait for a TX backlog, but it must never overtake a pending
+ * ENABLE for the same queue: a DISABLE consumed before its
+ * ENABLE closes nothing while the ENABLE then arms rings
+ * nobody owns.
+ */
+ list = &trans->trans_list[que->hif_id].skb_list[que->txqno];
+ spin_lock_irqsave(&list->lock, flags);
+ skb_queue_walk(list, iter) {
+ if (((struct trb *)iter->cb)->cmd == TRB_CMD_ENABLE)
+ last_enable = iter;
+ }
+ if (last_enable)
+ __skb_queue_after(list, last_enable, skb);
+ else
+ __skb_queue_head(list, skb);
+ spin_unlock_irqrestore(&list->lock, flags);
+ } else {
+ skb_queue_tail(&trans->trans_list[que->hif_id].skb_list[que->txqno], skb);
+ }
+
+ wake_up(&trans->trb_srv[trans->srv_cfg[que->hif_id][que->txqno]]->trb_waitq);
+ ret = 0;
+
+unlock:
+ mutex_unlock(&trans->submit_lock);
+ return ret;
+}
+
+int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+ struct queue_info *que;
+ int ret;
+
+ switch (cmd) {
+ case HIF_CTRL_CMD_CHECK_TX_FULL:
+ trans = ctrl_blk->ctrl_hw_priv;
+ mutex_lock(&trans->submit_lock);
+ if (!trans->available) {
+ ret = -EIO;
+ break;
+ }
+ que = xa_load(&trans->queue_tbl,
+ ((union ctrl_hif_cmd_data *)data)->rx_ch & QUEUE_CHL_MASK);
+ if (!que) {
+ dev_warn(mdev->dev, "Failed to find que to check tx full\n");
+ ret = -EINVAL;
+ break;
+ }
+ ret = mtk_queue_list_is_full(trans, que);
+ break;
+ default:
+ return -EINVAL;
+ }
+ mutex_unlock(&trans->submit_lock);
+
+ return ret;
+}
+
+int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_ctrl_trans *trans;
+ struct mtk_ctrl_blk *ctrl_blk;
+ int err;
+
+ trans = devm_kzalloc(mdev->dev, sizeof(*trans), GFP_KERNEL);
+ if (!trans)
+ return -ENOMEM;
+ trans->mdev = mdev;
+ mutex_init(&trans->submit_lock);
+ trans->available = false;
+
+ memcpy(trans->srv_cfg, mtk_srv_cfg, sizeof(mtk_srv_cfg));
+ trans->queue_info = mtk_queue_info;
+ trans->queue_info_num = ARRAY_SIZE(mtk_queue_info);
+ trans->trb_srv_num = TRB_SRV_NUM;
+
+ err = mtk_ctrl_init(mdev);
+ if (err)
+ return err;
+
+ ctrl_blk = mdev->ctrl_blk;
+ ctrl_blk->ctrl_hw_priv = trans;
+
+ return 0;
+}
+
+int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev)
+{
+ mtk_ctrl_exit(mdev);
+
+ return 0;
+}
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
index 9b42f33bd008..ee036f01a875 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -7,13 +7,79 @@
#define __MTK_TRANS_CTRL_H__
#include <linux/list.h>
+#include <linux/mutex.h>
#include <linux/skbuff.h>
#include <linux/types.h>
+#include <linux/xarray.h>
#include "mtk_dev.h"
+#define TRB_SRV_MAX_NUM (1)
+#define HW_QUE_NUM (8)
+#define TX_GPD_NUM (16)
+#define RX_GPD_NUM (TX_GPD_NUM)
+#define MIN_GPD_NUM (2)
+#define SKB_LIST_MAX_LEN (16)
+#define TRB_NUM_PER_ROUND (TX_GPD_NUM)
+#define TX_BURST_MAX_CNT (TX_GPD_NUM / 4 + 1)
+
+enum mtk_hif_id {
+ CLDMA0,
+ CLDMA1,
+ NR_CLDMA
+};
+
+struct queue_info {
+ u32 tx_chl;
+ u32 rx_chl;
+ enum mtk_hif_id hif_id;
+ u32 txqno;
+ u32 rxqno;
+ u32 tx_mtu;
+ u32 rx_mtu;
+ u32 tx_nr_gpds;
+ u32 rx_nr_gpds;
+ u8 log_rg_offset;
+};
+
+struct trans_list {
+ struct sk_buff_head skb_list[HW_QUE_NUM];
+ u8 tx_burst_cnt[HW_QUE_NUM];
+};
+
struct mtk_ctrl_trans {
+ struct trb_srv *trb_srv[TRB_SRV_MAX_NUM];
+ struct trans_list trans_list[NR_CLDMA];
+ void *dev;
+ struct xarray queue_tbl;
struct mtk_md_dev *mdev;
+ int usr_cnt[NR_CLDMA][HW_QUE_NUM];
+ struct mutex submit_lock; /* protects available flag and submit/exit paths */
+ bool available;
+ int srv_cfg[NR_CLDMA][HW_QUE_NUM];
+ const struct queue_info *queue_info;
+ int queue_info_num;
+ int trb_srv_num;
+};
+
+struct srv_que {
+ u32 hif_id;
+ u32 qno;
+ struct list_head list;
};
+struct trb_srv {
+ struct list_head srv_q_list[NR_CLDMA];
+ struct mtk_ctrl_trans *trans;
+ wait_queue_head_t trb_waitq;
+ struct task_struct *trb_thread;
+};
+
+int mtk_pcie_hif_init(struct mtk_md_dev *mdev);
+int mtk_pcie_hif_exit(struct mtk_md_dev *mdev);
+int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send);
+int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data);
+int mtk_trans_ctrl_init(struct mtk_md_dev *mdev);
+int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev);
+
#endif
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 4/6] net: wwan: t9xx: Add control port
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
` (2 preceding siblings ...)
2026-11-05 7:56 ` [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports Jack Wu via B4 Relay
5 siblings, 1 reply; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
The control port consists of port I/O and port manager.
Port I/O provides a common operation as defined by
"struct mtk_port_ops", and the operation is managed by the
"port manager". It provides interfaces to internal users, the
implemented internal interfaces are open, close, write and
recv_register.
The port manager defines and implements port management interfaces and
structures. It is responsible for port creation, destroying, and managing
port states. It sends data from port I/O to CLDMA via TRB ( Transaction
Request Block ), and dispatches received data from CLDMA to port I/O.
A TRB lives in the control buffer of the skb it describes, so it is
freed together with that skb. This patch therefore gives struct trb a
kref: the port layer initialises it in mtk_port_trb_init() and owns the
release function mtk_port_trb_free(). mtk_ctrl_trb_handler() holds a
reference for the duration of a dispatch, including on the -EAGAIN
early return, so a completion running concurrently cannot free the skb
underneath the handler.
The TX sequence number is written into the header when the trb thread
hands the packet to CLDMA, and only advanced once CLDMA has put it on
the ring, so a packet refused before that takes no number. A packet
already on the ring keeps its number even if the queue then fails to
start or is flushed: it may have reached the modem by then, so the
number is not reused.
At this commit probe and remove only build and tear down the port
table: ->init() runs for each port, and ->disable() and ->exit() find
no port enabled. Nothing calls ->enable(), ->recv() or the internal
port interface yet, and no TRB is processed: the port-state, RX
dispatch and TRB kref paths become live with "Add FSM thread". That
patch also provides the teardown order these paths rely on - the CLDMA
queues are quiesced and the trb services stopped before the port
manager frees the ports - so neither a TRB completion nor an armed RX
queue outlives the port it points to.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 2 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 24 +-
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 12 +-
drivers/net/wwan/t9xx/mtk_port.c | 667 ++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_port.h | 133 ++++++
drivers/net/wwan/t9xx/mtk_port_io.c | 229 ++++++++++
drivers/net/wwan/t9xx/mtk_port_io.h | 32 ++
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 27 +-
8 files changed, 1119 insertions(+), 7 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index 74bef150e3a1..d44266009362 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -7,6 +7,8 @@ obj-$(CONFIG_MTK_T9XX) += mtk_t9xx.o
mtk_t9xx-y := \
mtk_ctrl_plane.o \
+ mtk_port.o \
+ mtk_port_io.o \
pcie/mtk_pci.o \
pcie/mtk_trans_ctrl.o \
pcie/mtk_cldma.o \
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
index 81f6b3fbc0bd..af3064753b15 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -7,21 +7,24 @@
#include <linux/device.h>
#include "mtk_ctrl_plane.h"
+#include "mtk_port.h"
/**
* mtk_ctrl_init() - Initialize the control plane block.
* @mdev: Pointer to the MTK modem device.
+ * @port_layer_cfg: Port layer configuration.
*
* Allocates and initializes the control plane block
* associated with @mdev. Runs once per bind; the allocation is
* devres managed and is released at unbind, not by
* mtk_ctrl_exit().
*
- * Return: 0 on success, -ENOMEM on allocation failure.
+ * Return: 0 on success, negative error code on failure.
*/
-int mtk_ctrl_init(struct mtk_md_dev *mdev)
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_port_layer_cfg *port_layer_cfg)
{
struct mtk_ctrl_blk *ctrl_blk;
+ int err;
ctrl_blk = devm_kzalloc(mdev->dev, sizeof(*ctrl_blk), GFP_KERNEL);
if (!ctrl_blk)
@@ -30,6 +33,13 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev)
ctrl_blk->mdev = mdev;
mdev->ctrl_blk = ctrl_blk;
+ err = mtk_port_mngr_init(ctrl_blk, port_layer_cfg->port_cfg,
+ port_layer_cfg->port_cnt);
+ if (err) {
+ mdev->ctrl_blk = NULL;
+ return err;
+ }
+
return 0;
}
@@ -37,10 +47,16 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev)
* mtk_ctrl_exit() - Clean up the control plane block.
* @mdev: Pointer to the MTK modem device.
*
- * Clears the control plane block pointer. The allocation
- * itself is managed by devres and freed on driver detach.
+ * Tears down what mtk_ctrl_init() set up and clears the control plane
+ * block pointer. The allocation itself is managed by devres and freed
+ * on driver detach.
+ *
+ * Context: Process context; may sleep.
*/
void mtk_ctrl_exit(struct mtk_md_dev *mdev)
{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+
+ mtk_port_mngr_exit(ctrl_blk);
mdev->ctrl_blk = NULL;
}
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index cf5d6d95f3ee..3ba66d9a4b60 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -6,6 +6,7 @@
#ifndef __MTK_CTRL_PLANE_H__
#define __MTK_CTRL_PLANE_H__
+#include <linux/kref.h>
#include <linux/skbuff.h>
#include "mtk_dev.h"
@@ -46,6 +47,11 @@ struct trb {
u32 channel_id;
enum mtk_trb_cmd_type cmd;
int status;
+ /* Store-release by the completion callback, load-acquire by the
+ * waiter: seeing it set means seeing everything the callback wrote.
+ */
+ bool done;
+ struct kref kref;
void *priv;
int (*trb_complete)(struct sk_buff *skb);
};
@@ -56,12 +62,16 @@ union ctrl_hif_cmd_data {
u32 rx_ch;
};
+struct mtk_port_layer_cfg;
+struct mtk_port_mngr;
+
struct mtk_ctrl_blk {
struct mtk_md_dev *mdev;
+ struct mtk_port_mngr *port_mngr;
void *ctrl_hw_priv;
};
-int mtk_ctrl_init(struct mtk_md_dev *mdev);
+int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_port_layer_cfg *port_layer_cfg);
void mtk_ctrl_exit(struct mtk_md_dev *mdev);
#endif /* __MTK_CTRL_PLANE_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
new file mode 100644
index 000000000000..7fec696bd6db
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -0,0 +1,667 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/kernel.h>
+#include <linux/netdevice.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+
+#include "mtk_port.h"
+#include "mtk_port_io.h"
+#include "mtk_trans_ctrl.h"
+
+#define MTK_DFLT_TRB_TIMEOUT (5 * HZ)
+#define MTK_DFLT_TRB_STATUS (0x1)
+#define MTK_CHECK_RX_SEQ_MASK (0x7fff)
+#define MTK_INVALID_RX_SEQ (0xffff)
+
+#define MTK_PORT_ENUM_VER (0)
+#define MTK_PORT_ENUM_HEAD_PATTERN (0x5a5a5a5a)
+#define MTK_PORT_ENUM_TAIL_PATTERN (0xa5a5a5a5)
+
+struct mtk_port_info {
+ __le16 channel;
+ __le16 reserved;
+} __packed;
+
+struct mtk_port_enum_msg {
+ __le32 head_pattern;
+ __le16 port_cnt;
+ __le16 version;
+ __le32 tail_pattern;
+ u8 data[];
+} __packed;
+
+/* Returns NULL for an empty slot, or after restarting the walk at an
+ * internal node; the caller then skips to the next slot.
+ */
+static struct mtk_port *mtk_port_tbl_slot(void __rcu ***slot, struct radix_tree_iter *iter)
+{
+ struct mtk_port *port = rcu_dereference_raw(**slot);
+
+ if (radix_tree_is_internal_node(port)) {
+ *slot = radix_tree_iter_retry(iter);
+ return NULL;
+ }
+
+ return port;
+}
+
+/* Serialises a lookup and its kref_get() against the removal of the port
+ * from the table and the kref_put() that may free it.
+ */
+DEFINE_MUTEX(mtk_port_mngr_grp_mtx);
+
+/* Called with mtk_port_mngr_grp_mtx held. */
+void mtk_port_release(struct kref *port_kref)
+{
+ struct mtk_port *port;
+
+ port = container_of(port_kref, struct mtk_port, kref);
+ mtk_port_get_ops(port->info.type)->exit(port);
+ kfree_rcu(port, rcu);
+}
+
+static int mtk_port_tbl_add(struct mtk_port_mngr *port_mngr, struct mtk_port *port)
+{
+ int ret;
+
+ mutex_lock(&port_mngr->port_tbl_mtx);
+ ret = radix_tree_insert(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->info.rx_ch)],
+ MTK_CH_ID(port->info.rx_ch), port);
+ mutex_unlock(&port_mngr->port_tbl_mtx);
+
+ if (ret)
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "port(%s) add to port_tbl failed, return %d\n",
+ port->info.name, ret);
+
+ return ret;
+}
+
+static void mtk_port_tbl_del(struct mtk_port_mngr *port_mngr, struct mtk_port *port)
+{
+ mutex_lock(&port_mngr->port_tbl_mtx);
+ radix_tree_delete(&port_mngr->port_tbl[MTK_PORT_TBL_TYPE(port->info.rx_ch)],
+ MTK_CH_ID(port->info.rx_ch));
+ mutex_unlock(&port_mngr->port_tbl_mtx);
+}
+
+static struct mtk_port *mtk_port_alloc_and_add(struct mtk_port_mngr *port_mngr,
+ const struct mtk_port_cfg *dflt_info)
+{
+ struct mtk_port *port;
+ int ret;
+
+ port = kzalloc_obj(*port, GFP_KERNEL);
+ if (!port) {
+ ret = -ENOMEM;
+ goto err_alloc_port;
+ }
+ memcpy(&port->info, dflt_info, sizeof(*dflt_info));
+
+ ret = mtk_port_tbl_add(port_mngr, port);
+ if (ret < 0) {
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to add port(%s) to port tbl\n", dflt_info->name);
+ goto err_free_port;
+ }
+
+ port->port_mngr = port_mngr;
+ ret = mtk_port_get_ops(port->info.type)->init(port);
+ if (ret < 0) {
+ mtk_port_tbl_del(port_mngr, port);
+ goto err_free_port;
+ }
+
+ return port;
+
+err_free_port:
+ kfree(port);
+err_alloc_port:
+ return ERR_PTR(ret);
+}
+
+static void mtk_port_free(struct mtk_port_mngr *port_mngr, struct mtk_port *port)
+{
+ mutex_lock(&mtk_port_mngr_grp_mtx);
+ mtk_port_tbl_del(port_mngr, port);
+ kref_put(&port->kref, mtk_port_release);
+ mutex_unlock(&mtk_port_mngr_grp_mtx);
+}
+
+/* No lock: every caller runs in a context that is stopped before the control
+ * plane destroys the port table. A caller without that guarantee must use
+ * mtk_port_get_by_name() instead.
+ */
+static struct mtk_port *mtk_port_search_by_id(struct mtk_port_mngr *port_mngr, int rx_ch)
+{
+ int tbl_type = MTK_PORT_TBL_TYPE(rx_ch);
+
+ if (tbl_type < PORT_TBL_SAP || tbl_type >= PORT_TBL_MAX)
+ return NULL;
+
+ return radix_tree_lookup(&port_mngr->port_tbl[tbl_type], MTK_CH_ID(rx_ch));
+}
+
+static struct mtk_port *mtk_port_search_by_name(struct mtk_port_mngr *port_mngr,
+ const char *name)
+{
+ int tbl_type = PORT_TBL_SAP;
+ struct radix_tree_iter iter;
+ struct mtk_port *port;
+ void __rcu **slot;
+
+ lockdep_assert_held(&mtk_port_mngr_grp_mtx);
+
+ do {
+ radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], &iter, 0) {
+ port = mtk_port_tbl_slot(&slot, &iter);
+ if (!port)
+ continue;
+ if (!strncmp(port->info.name, name, MTK_DFLT_PORT_NAME_LEN))
+ return port;
+ }
+ tbl_type++;
+ } while (tbl_type < PORT_TBL_MAX);
+
+ return NULL;
+}
+
+struct mtk_port *mtk_port_get_by_name(struct mtk_port_mngr *port_mngr, const char *name,
+ enum mtk_port_type type)
+{
+ struct mtk_port *port;
+
+ mutex_lock(&mtk_port_mngr_grp_mtx);
+ port = mtk_port_search_by_name(port_mngr, name);
+ if (port && port->info.type == type)
+ kref_get(&port->kref);
+ else
+ port = NULL;
+ mutex_unlock(&mtk_port_mngr_grp_mtx);
+
+ return port;
+}
+
+static int mtk_port_tbl_create(struct mtk_port_mngr *port_mngr, const struct mtk_port_cfg *cfg,
+ const int port_cnt)
+{
+ const struct mtk_port_cfg *dflt_port;
+ struct mtk_port *port;
+ int i;
+
+ INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_SAP], GFP_KERNEL);
+ INIT_RADIX_TREE(&port_mngr->port_tbl[PORT_TBL_MD], GFP_KERNEL);
+
+ /* copy ports from static port cfg table */
+ for (i = 0; i < port_cnt; i++) {
+ dflt_port = cfg + i;
+ if (!mtk_port_search_by_id(port_mngr, dflt_port->rx_ch)) {
+ port = mtk_port_alloc_and_add(port_mngr, dflt_port);
+ if (IS_ERR(port))
+ return PTR_ERR(port);
+ }
+ }
+
+ return 0;
+}
+
+static void mtk_port_tbl_destroy(struct mtk_port_mngr *port_mngr)
+{
+ struct radix_tree_iter iter;
+ struct mtk_port *port;
+ void __rcu **slot;
+ int tbl_type;
+
+ tbl_type = PORT_TBL_SAP;
+ do {
+ radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type], &iter, 0) {
+ port = mtk_port_tbl_slot(&slot, &iter);
+ if (!port)
+ continue;
+ mtk_port_get_ops(port->info.type)->disable(port);
+ }
+
+ while (radix_tree_gang_lookup(&port_mngr->port_tbl[tbl_type],
+ (void **)&port, 0, 1))
+ mtk_port_free(port_mngr, port);
+ } while (++tbl_type < PORT_TBL_MAX);
+}
+
+void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_trb_cmd_type cmd,
+ int (*trb_complete)(struct sk_buff *skb))
+{
+ kref_init(&trb->kref);
+ trb->channel_id = port->info.rx_ch;
+ trb->status = MTK_DFLT_TRB_STATUS;
+ trb->done = false;
+ /* No reference is taken: the port must outlive every trb that names
+ * it and every RX queue armed with this pointer as rxq->arg.
+ */
+ trb->priv = port;
+ trb->cmd = cmd;
+ trb->trb_complete = trb_complete;
+}
+
+static bool mtk_port_trb_done(struct trb *trb)
+{
+ /* Pairs with the smp_store_release() in the completion callbacks. */
+ return smp_load_acquire(&trb->done);
+}
+
+void mtk_port_trb_free(struct kref *trb_kref)
+{
+ struct trb *trb = container_of(trb_kref, struct trb, kref);
+
+ dev_kfree_skb_any(container_of((char *)trb, struct sk_buff, cb[0]));
+}
+
+static int mtk_port_ch_trb_complete(struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_port *port = trb->priv;
+
+ /* Pairs with the smp_load_acquire() of trb->done in the waiters. */
+ smp_store_release(&trb->done, true);
+ wake_up_all(&port->trb_wq);
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return 0;
+}
+
+static int mtk_port_tx_complete(struct sk_buff *skb)
+{
+ struct trb *trb = (struct trb *)skb->cb;
+ struct mtk_port *port = trb->priv;
+
+ if (trb->status < 0)
+ dev_warn_ratelimited(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to send data: status:%d, port:%s\n",
+ trb->status, port->info.name);
+
+ /* Pairs with the smp_load_acquire() of trb->done in the waiters. */
+ smp_store_release(&trb->done, true);
+ wake_up_all(&port->trb_wq);
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return 0;
+}
+
+int mtk_port_status_check(struct mtk_port *port)
+{
+ if (!test_bit(PORT_S_ENABLE, &port->status))
+ return -ENODEV;
+
+ if (!test_bit(PORT_S_OPEN, &port->status) || test_bit(PORT_S_FLUSH, &port->status) ||
+ !test_bit(PORT_S_WR, &port->status))
+ return -EBADF;
+
+ return 0;
+}
+
+int mtk_port_send_data(struct mtk_port *port, void *data, bool blocking, bool force_send)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct sk_buff *skb = data;
+ struct trb *trb;
+ int ret, len;
+
+ port_mngr = port->port_mngr;
+
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_TX, mtk_port_tx_complete);
+ len = skb->len;
+ kref_get(&trb->kref); /* kref count 1->2 */
+
+ /* add ccci header */
+ ret = mtk_port_add_header(skb);
+ if (!ret)
+ ret = mtk_port_status_check(port);
+ if (!ret)
+ ret = mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb,
+ force_send);
+
+ if (ret < 0) {
+ kref_put(&trb->kref, mtk_port_trb_free); /* kref count 2->1 */
+ kref_put(&trb->kref, mtk_port_trb_free); /* kref count 1->0 */
+ goto out;
+ }
+
+ if (!blocking) {
+ kref_put(&trb->kref, mtk_port_trb_free);
+ ret = len;
+ goto out;
+ }
+
+ /* no timeout in tx blocking mode */
+ ret = wait_event_interruptible(port->trb_wq,
+ mtk_port_trb_done(trb) ||
+ test_bit(PORT_S_FLUSH, &port->status) ||
+ !test_bit(PORT_S_WR, &port->status));
+ /* Only a completed trb has a meaningful status; any other wake is
+ * a write failure.
+ */
+ if (ret)
+ ret = -EINTR;
+ else if (mtk_port_trb_done(trb))
+ ret = trb->status ? trb->status : len;
+ else
+ ret = -EBADF;
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+out:
+ return ret;
+}
+
+/* A gap only means an earlier packet was lost; this one is still delivered. */
+static void mtk_port_check_rx_seq(struct mtk_port *port, struct mtk_ccci_header *ccci_h)
+{
+ u16 seq_num, assert_bit, channel;
+ struct mtk_md_dev *mdev;
+
+ seq_num = FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status));
+ assert_bit = FIELD_GET(MTK_HDR_FLD_AST, le32_to_cpu(ccci_h->status));
+ if (assert_bit && port->rx_seq != MTK_INVALID_RX_SEQ &&
+ ((seq_num - port->rx_seq) & MTK_CHECK_RX_SEQ_MASK) != 1) {
+ mdev = port->port_mngr->ctrl_blk->mdev;
+ channel = FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status));
+ dev_warn_ratelimited(mdev->dev,
+ "<ch: %04x> seq num out-of-order %d->%d, len(%u)\n",
+ channel, port->rx_seq, seq_num,
+ le32_to_cpu(ccci_h->packet_len));
+ }
+}
+
+static int mtk_port_rx_dispatch(struct sk_buff *skb, void *priv)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ccci_header *ccci_h;
+ struct mtk_port *port = priv;
+ int ret = -EPROTO;
+ u16 channel;
+
+ if (!skb || !priv) {
+ pr_err("Invalid input value in rx dispatch\n");
+ return -EINVAL;
+ }
+
+ port_mngr = port->port_mngr;
+
+ ccci_h = mtk_port_strip_header(skb);
+ if (unlikely(!ccci_h)) {
+ dev_warn_ratelimited(port_mngr->ctrl_blk->mdev->dev,
+ "Unsupported: skb length(%u) is less than ccci header\n",
+ skb->len);
+ goto drop_data;
+ }
+
+ channel = FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status));
+ /* A CLDMA instance carries the channels of one peer only (sAP or MD),
+ * the peer of the port that armed this queue.
+ */
+ if (MTK_PEER_ID(channel) == MTK_PEER_ID(port->info.rx_ch))
+ port = mtk_port_search_by_id(port_mngr, channel);
+ else
+ port = NULL;
+ if (unlikely(!port)) {
+ dev_warn_ratelimited(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to find port by channel:%d\n", channel);
+ goto drop_data;
+ }
+
+ mtk_port_check_rx_seq(port, ccci_h);
+ port->rx_seq = FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status));
+
+ ret = mtk_port_get_ops(port->info.type)->recv(port, skb);
+ if (ret < 0)
+ goto drop_data;
+
+ return ret;
+
+drop_data:
+ dev_kfree_skb_any(skb);
+ return ret;
+}
+
+int mtk_port_add_header(struct sk_buff *skb)
+{
+ struct mtk_ccci_header *ccci_h;
+ struct mtk_port *port;
+ struct trb *trb;
+ int ret;
+
+ trb = (struct trb *)skb->cb;
+ port = trb->priv;
+ if (!port)
+ return -EINVAL;
+
+ ret = skb_cow_head(skb, sizeof(*ccci_h));
+ if (ret)
+ return ret;
+
+ ccci_h = skb_push(skb, sizeof(*ccci_h));
+
+ ccci_h->packet_header = cpu_to_le32(0);
+ ccci_h->packet_len = cpu_to_le32(skb->len);
+ ccci_h->ex_msg = cpu_to_le32(0);
+ /* SEQ is stamped at hand-off, see mtk_port_tx_seq_stamp() */
+ ccci_h->status = cpu_to_le32(FIELD_PREP(MTK_HDR_FLD_CHN, port->info.tx_ch) |
+ FIELD_PREP(MTK_HDR_FLD_AST, 1));
+
+ return 0;
+}
+
+/* Called around the hardware hand-off of a TX TRB, so a packet that
+ * fails before it reaches the ring does not consume a number.
+ */
+void mtk_port_tx_seq_stamp(struct sk_buff *skb)
+{
+ struct mtk_ccci_header *ccci_h = (struct mtk_ccci_header *)skb->data;
+ struct mtk_port *port = ((struct trb *)skb->cb)->priv;
+ u32 status = le32_to_cpu(ccci_h->status) & ~MTK_HDR_FLD_SEQ;
+
+ ccci_h->status = cpu_to_le32(status |
+ FIELD_PREP(MTK_HDR_FLD_SEQ, port->tx_seq));
+}
+
+void mtk_port_tx_seq_advance(struct sk_buff *skb)
+{
+ struct mtk_port *port = ((struct trb *)skb->cb)->priv;
+
+ port->tx_seq++;
+}
+
+struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb)
+{
+ struct mtk_ccci_header *ccci_h;
+
+ if (!pskb_may_pull(skb, sizeof(*ccci_h)))
+ return NULL;
+
+ ccci_h = (struct mtk_ccci_header *)skb->data;
+ skb_pull(skb, sizeof(*ccci_h));
+
+ return ccci_h;
+}
+
+int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_len)
+{
+ struct mtk_port_enum_msg *msg = data;
+ struct mtk_port_info *port_info;
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_port *port;
+ int port_id;
+ u16 ch_id;
+
+ if (unlikely(!mdev || !msg))
+ return -EINVAL;
+
+ if (data_len < sizeof(*msg))
+ return -EPROTO;
+
+ ctrl_blk = mdev->ctrl_blk;
+ port_mngr = ctrl_blk->port_mngr;
+ if (le16_to_cpu(msg->version) != MTK_PORT_ENUM_VER ||
+ le32_to_cpu(msg->head_pattern) != MTK_PORT_ENUM_HEAD_PATTERN ||
+ le32_to_cpu(msg->tail_pattern) != MTK_PORT_ENUM_TAIL_PATTERN)
+ return -EPROTO;
+
+ if (data_len < sizeof(*msg) +
+ le16_to_cpu(msg->port_cnt) * sizeof(*port_info))
+ return -EPROTO;
+
+ for (port_id = 0; port_id < le16_to_cpu(msg->port_cnt); port_id++) {
+ port_info = (struct mtk_port_info *)(msg->data +
+ (sizeof(*port_info) * port_id));
+ ch_id = FIELD_GET(MTK_INFO_FLD_CHID, le16_to_cpu(port_info->channel));
+ port = mtk_port_search_by_id(port_mngr, ch_id);
+ if (!port)
+ continue;
+ port->enable = FIELD_GET(MTK_INFO_FLD_EN, le16_to_cpu(port_info->channel));
+ }
+
+ return 0;
+}
+
+int mtk_port_ch_enable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr = port->port_mngr;
+ struct trb_open_priv *trb_open_priv;
+ struct sk_buff *skb;
+ struct trb *trb;
+ int ret;
+
+ /* The modem restarts its sequence numbering with the channel, and
+ * the first packet after it is not checked.
+ */
+ port->tx_seq = 0;
+ port->rx_seq = MTK_INVALID_RX_SEQ;
+
+ skb = __dev_alloc_skb(sizeof(*trb_open_priv), GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ trb_open_priv = (struct trb_open_priv *)skb->data;
+ trb_open_priv->rx_done = mtk_port_rx_dispatch;
+
+ skb_put(skb, sizeof(struct trb_open_priv));
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_ENABLE, mtk_port_ch_trb_complete);
+ kref_get(&trb->kref);
+
+ ret = mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb, true);
+ if (ret) {
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to submit trb for port(%s), ret=%d\n",
+ port->info.name, ret);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ return ret;
+ }
+
+ ret = wait_event_timeout(port->trb_wq, mtk_port_trb_done(trb),
+ MTK_DFLT_TRB_TIMEOUT);
+ if (!ret) {
+ ret = -ETIMEDOUT;
+ } else {
+ /* The trb->done acquire in mtk_port_trb_done() orders the
+ * geometry read below, on the -EBUSY completion too. Pairs
+ * with the smp_store_release() of trb->status in mtk_cldma_open().
+ */
+ ret = smp_load_acquire(&trb->status);
+
+ /* -EBUSY means another user already owns the queue, and the
+ * geometry it reports is that queue's - also valid here.
+ */
+ if (!ret || ret == -EBUSY)
+ port->tx_mtu = trb_open_priv->tx_mtu - MTK_CCCI_H_ELEN;
+ }
+
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return ret;
+}
+
+int mtk_port_ch_disable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr = port->port_mngr;
+ struct sk_buff *skb;
+ struct trb *trb;
+ int ret;
+
+ skb = __dev_alloc_skb(0, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ trb = (struct trb *)skb->cb;
+ mtk_port_trb_init(port, trb, TRB_CMD_DISABLE, mtk_port_ch_trb_complete);
+ kref_get(&trb->kref);
+
+ ret = mtk_pcie_hif_submit_skb(port_mngr->ctrl_blk->mdev, skb, true);
+ if (ret) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to submit trb for port(%s), ret=%d\n",
+ port->info.name, ret);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ kref_put(&trb->kref, mtk_port_trb_free);
+ return ret;
+ }
+
+ ret = wait_event_timeout(port->trb_wq, mtk_port_trb_done(trb),
+ MTK_DFLT_TRB_TIMEOUT);
+ if (!ret)
+ ret = -ETIMEDOUT;
+ else
+ ret = trb->status;
+
+ kref_put(&trb->kref, mtk_port_trb_free);
+
+ return ret;
+}
+
+int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, const struct mtk_port_cfg *port_cfg,
+ int port_cnt)
+{
+ struct mtk_port_mngr *port_mngr;
+ int ret = -ENOMEM;
+
+ port_mngr = devm_kzalloc(ctrl_blk->mdev->dev, sizeof(*port_mngr), GFP_KERNEL);
+ if (unlikely(!port_mngr)) {
+ dev_err((ctrl_blk->mdev)->dev, "Failed to alloc memory for port_mngr\n");
+ goto err_out;
+ }
+
+ port_mngr->ctrl_blk = ctrl_blk;
+ mutex_init(&port_mngr->port_tbl_mtx);
+
+ ret = mtk_port_tbl_create(port_mngr, port_cfg, port_cnt);
+ if (unlikely(ret)) {
+ dev_err((ctrl_blk->mdev)->dev, "Failed to create port_tbl\n");
+ goto err_destroy_tbl;
+ }
+
+ ctrl_blk->port_mngr = port_mngr;
+
+ return ret;
+
+err_destroy_tbl:
+ mtk_port_tbl_destroy(port_mngr);
+err_out:
+ return ret;
+}
+
+void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk)
+{
+ struct mtk_port_mngr *port_mngr = ctrl_blk->port_mngr;
+
+ mtk_port_tbl_destroy(port_mngr);
+
+ ctrl_blk->port_mngr = NULL;
+}
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
new file mode 100644
index 000000000000..157eda4b4220
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -0,0 +1,133 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PORT_H__
+#define __MTK_PORT_H__
+
+#include <linux/bits.h>
+#include <linux/device.h>
+#include <linux/radix-tree.h>
+#include <linux/skbuff.h>
+#include <linux/types.h>
+
+#include "mtk_ctrl_plane.h"
+#include "mtk_dev.h"
+
+#define MTK_PEER_ID_MASK (0xF000)
+#define MTK_PEER_ID_SHIFT (12)
+#define MTK_PEER_ID(ch) (((ch) & MTK_PEER_ID_MASK) >> MTK_PEER_ID_SHIFT)
+#define MTK_CH_ID_MASK (0x0FFF)
+#define MTK_CH_ID(ch) ((ch) & MTK_CH_ID_MASK)
+#define MTK_DFLT_PORT_NAME_LEN (20)
+
+/* Mapping MTK_PEER_ID and mtk_port_tbl index */
+#define MTK_PORT_TBL_TYPE(ch) (MTK_PEER_ID(ch) - 1)
+
+/* ccci header length + reserved space that is used in exception flow */
+#define MTK_CCCI_H_ELEN (128)
+
+#define MTK_HDR_FLD_AST ((u32)BIT(31))
+#define MTK_HDR_FLD_SEQ GENMASK(30, 16)
+#define MTK_HDR_FLD_CHN GENMASK(15, 0)
+
+#define MTK_INFO_FLD_EN ((u16)BIT(15))
+#define MTK_INFO_FLD_CHID GENMASK(14, 0)
+
+/* State bits, not locks: callers serialise ->enable(), ->disable() and
+ * open of a port. The atomic bitops only make a repeated call a no-op.
+ */
+enum mtk_port_status {
+ PORT_S_ENABLE,
+ PORT_S_OPEN,
+ PORT_S_WR,
+ PORT_S_FLUSH,
+};
+
+enum mtk_port_flag {
+ PORT_F_DFLT = 0,
+ PORT_F_BLOCKING = BIT(1),
+};
+
+enum mtk_port_tbl {
+ PORT_TBL_SAP,
+ PORT_TBL_MD,
+ PORT_TBL_MAX
+};
+
+enum mtk_port_type {
+ PORT_TYPE_INTERNAL,
+ PORT_TYPE_MAX
+};
+
+struct mtk_internal_port {
+ void *arg;
+ int (*recv_cb)(void *arg, struct sk_buff *skb);
+};
+
+struct mtk_port_cfg {
+ enum mtk_ccci_ch tx_ch;
+ enum mtk_ccci_ch rx_ch;
+ enum mtk_port_type type;
+ char name[MTK_DFLT_PORT_NAME_LEN];
+ unsigned char flags;
+};
+
+struct mtk_port {
+ struct mtk_port_cfg info;
+ struct kref kref;
+ struct rcu_head rcu;
+ bool enable;
+ unsigned long status;
+ /* tx_seq is stamped and advanced only at hardware hand-off, by the one
+ * thread serving this port's TX queue; rx_seq only from the rx_done
+ * callback of the one queue serving this channel. Both are reset
+ * while the channel is disabled, so neither needs a lock.
+ */
+ unsigned short tx_seq;
+ unsigned short rx_seq;
+ unsigned int tx_mtu;
+ wait_queue_head_t trb_wq;
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_internal_port i_priv;
+};
+
+struct mtk_port_mngr {
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct radix_tree_root port_tbl[PORT_TBL_MAX];
+ /* protects port_tbl[] against concurrent mutators */
+ struct mutex port_tbl_mtx;
+};
+
+struct mtk_ccci_header {
+ __le32 packet_header;
+ __le32 packet_len;
+ __le32 status;
+ __le32 ex_msg;
+};
+
+struct mtk_port_layer_cfg {
+ const struct mtk_port_cfg *port_cfg;
+ int port_cnt;
+};
+
+void mtk_port_release(struct kref *port_kref);
+void mtk_port_trb_free(struct kref *trb_kref);
+struct mtk_port *mtk_port_get_by_name(struct mtk_port_mngr *port_mngr, const char *name,
+ enum mtk_port_type type);
+int mtk_port_add_header(struct sk_buff *skb);
+void mtk_port_tx_seq_stamp(struct sk_buff *skb);
+void mtk_port_tx_seq_advance(struct sk_buff *skb);
+struct mtk_ccci_header *mtk_port_strip_header(struct sk_buff *skb);
+int mtk_port_status_check(struct mtk_port *port);
+int mtk_port_send_data(struct mtk_port *port, void *data, bool blocking, bool force_send);
+int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_len);
+int mtk_port_ch_enable(struct mtk_port *port);
+int mtk_port_ch_disable(struct mtk_port *port);
+int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, const struct mtk_port_cfg *port_cfg,
+ int port_cnt);
+void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk);
+void mtk_port_trb_init(struct mtk_port *port, struct trb *trb, enum mtk_trb_cmd_type cmd,
+ int (*trb_complete)(struct sk_buff *skb));
+#endif /* __MTK_PORT_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mtk_port_io.c
new file mode 100644
index 000000000000..e92ce1cb7c00
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port_io.c
@@ -0,0 +1,229 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+#include <linux/netdevice.h>
+
+#include "mtk_port_io.h"
+
+static void mtk_port_put_locked(struct mtk_port *port)
+{
+ mutex_lock(&mtk_port_mngr_grp_mtx);
+ kref_put(&port->kref, mtk_port_release);
+ mutex_unlock(&mtk_port_mngr_grp_mtx);
+}
+
+static void mtk_port_struct_init(struct mtk_port *port)
+{
+ clear_bit(PORT_S_ENABLE, &port->status);
+ kref_init(&port->kref);
+ init_waitqueue_head(&port->trb_wq);
+}
+
+static int mtk_port_internal_init(struct mtk_port *port)
+{
+ mtk_port_struct_init(port);
+ port->enable = false;
+
+ return 0;
+}
+
+static void mtk_port_internal_exit(struct mtk_port *port)
+{
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ mtk_port_get_ops(port->info.type)->disable(port);
+}
+
+static void mtk_port_internal_enable(struct mtk_port *port)
+{
+ int ret;
+
+ if (test_and_set_bit(PORT_S_ENABLE, &port->status))
+ return;
+
+ ret = mtk_port_ch_enable(port);
+ if (ret && ret != -EBUSY) {
+ /* On -ETIMEDOUT the ENABLE trb may still be queued; queue a
+ * DISABLE behind it so the channel does not end up armed with
+ * no software owner.
+ */
+ mtk_port_ch_disable(port);
+ clear_bit(PORT_S_ENABLE, &port->status);
+ return;
+ }
+
+ set_bit(PORT_S_WR, &port->status);
+}
+
+static void mtk_port_internal_disable(struct mtk_port *port)
+{
+ int ret;
+
+ if (!test_and_clear_bit(PORT_S_ENABLE, &port->status))
+ return;
+
+ /* PORT_S_WR is in the blocking TX wait condition: wake the waiters. */
+ clear_bit(PORT_S_WR, &port->status);
+ wake_up_all(&port->trb_wq);
+
+ /* Failure must not abort teardown or keep PORT_S_ENABLE set:
+ * usr_cnt and the queues are rebuilt together by the next
+ * mtk_pcie_hif_init(), and a port left with the bit set could never
+ * be re-enabled (mtk_port_internal_enable() early-returns).
+ */
+ ret = mtk_port_ch_disable(port);
+ if (ret)
+ dev_warn(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to disable channel for port(%s): %d\n",
+ port->info.name, ret);
+}
+
+static int mtk_port_internal_recv(struct mtk_port *port, struct sk_buff *skb)
+{
+ int (*recv_cb)(void *priv, struct sk_buff *skb);
+ struct mtk_internal_port *priv;
+ int ret = -ENXIO;
+
+ if (!test_bit(PORT_S_OPEN, &port->status))
+ goto drop_data;
+
+ priv = &port->i_priv;
+ /* Pairs with the smp_store_release() in mtk_port_internal_recv_register(). */
+ recv_cb = smp_load_acquire(&priv->recv_cb);
+ if (!recv_cb || !priv->arg)
+ goto drop_data;
+
+ ret = recv_cb(priv->arg, skb);
+ return ret;
+
+drop_data:
+ return ret;
+}
+
+static int mtk_port_common_open(struct mtk_port *port)
+{
+ int ret = 0;
+
+ if (!test_bit(PORT_S_ENABLE, &port->status))
+ return -ENODEV;
+
+ if (test_and_set_bit(PORT_S_OPEN, &port->status))
+ return -EBUSY;
+
+ clear_bit(PORT_S_FLUSH, &port->status);
+
+ return ret;
+}
+
+static void mtk_port_common_close(struct mtk_port *port)
+{
+ clear_bit(PORT_S_OPEN, &port->status);
+
+ set_bit(PORT_S_FLUSH, &port->status);
+ wake_up_all(&port->trb_wq);
+}
+
+void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_port *port;
+ int ret;
+
+ ctrl_blk = mdev->ctrl_blk;
+ port_mngr = ctrl_blk->port_mngr;
+
+ port = mtk_port_get_by_name(port_mngr, name, PORT_TYPE_INTERNAL);
+ if (!port)
+ goto out;
+
+ ret = mtk_port_common_open(port);
+ if (ret) {
+ mtk_port_put_locked(port);
+ port = NULL;
+ goto out;
+ }
+
+ if (flag & O_NONBLOCK)
+ port->info.flags &= ~PORT_F_BLOCKING;
+ else
+ port->info.flags |= PORT_F_BLOCKING;
+out:
+ return port;
+}
+
+int mtk_port_internal_close(void *i_port)
+{
+ struct mtk_port *port = i_port;
+ int ret = 0;
+
+ if (!port) {
+ ret = -EINVAL;
+ goto end;
+ }
+
+ if (!test_and_clear_bit(PORT_S_OPEN, &port->status)) {
+ pr_err("Port(%s) has been closed\n", port->info.name);
+ ret = -EBADF;
+ goto end;
+ }
+
+ mtk_port_common_close(port);
+ mtk_port_put_locked(port);
+end:
+ return ret;
+}
+
+/* Consumes @skb, on failure too. */
+int mtk_port_internal_write(void *i_port, struct sk_buff *skb)
+{
+ struct mtk_port *port = i_port;
+ bool blocking;
+
+ if (!port || !skb) {
+ if (skb)
+ dev_kfree_skb_any(skb);
+ pr_err_ratelimited("Internal write: invalid input\n");
+ return -EINVAL;
+ }
+
+ blocking = !!(port->info.flags & PORT_F_BLOCKING);
+
+ return mtk_port_send_data(port, skb, blocking, blocking);
+}
+
+/* @cb and @arg must stay valid until the receive path has been drained:
+ * closing the port does not unregister them or wait for a callback that is
+ * already running, so close alone does not make freeing @arg safe.
+ * @cb owns the skb when it returns 0 or more; on a negative return the
+ * caller frees it, so @cb must not.
+ */
+void mtk_port_internal_recv_register(void *i_port,
+ int (*cb)(void *priv, struct sk_buff *skb),
+ void *arg)
+{
+ struct mtk_port *port = i_port;
+ struct mtk_internal_port *priv;
+
+ priv = &port->i_priv;
+ priv->arg = arg;
+ /* Pairs with the smp_load_acquire() in mtk_port_internal_recv(). */
+ smp_store_release(&priv->recv_cb, cb);
+}
+
+static const struct mtk_port_ops port_internal_ops = {
+ .init = mtk_port_internal_init,
+ .exit = mtk_port_internal_exit,
+ .enable = mtk_port_internal_enable,
+ .disable = mtk_port_internal_disable,
+ .recv = mtk_port_internal_recv,
+};
+
+static const struct mtk_port_ops *mtk_port_ops_tbl[PORT_TYPE_MAX] = {
+ &port_internal_ops,
+};
+
+const struct mtk_port_ops *mtk_port_get_ops(enum mtk_port_type type)
+{
+ return mtk_port_ops_tbl[type];
+}
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mtk_port_io.h
new file mode 100644
index 000000000000..db1d9ae52d9a
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_port_io.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_PORT_IO_H__
+#define __MTK_PORT_IO_H__
+
+#include <linux/skbuff.h>
+
+#include "mtk_port.h"
+
+extern struct mutex mtk_port_mngr_grp_mtx;
+
+struct mtk_port_ops {
+ int (*init)(struct mtk_port *port);
+ void (*exit)(struct mtk_port *port);
+ void (*enable)(struct mtk_port *port);
+ void (*disable)(struct mtk_port *port);
+ int (*recv)(struct mtk_port *port, struct sk_buff *skb);
+};
+
+const struct mtk_port_ops *mtk_port_get_ops(enum mtk_port_type type);
+
+void *mtk_port_internal_open(struct mtk_md_dev *mdev, char *name, int flag);
+int mtk_port_internal_close(void *i_port);
+int mtk_port_internal_write(void *i_port, struct sk_buff *skb);
+void mtk_port_internal_recv_register(void *i_port,
+ int (*cb)(void *priv, struct sk_buff *skb),
+ void *arg);
+
+#endif /* __MTK_PORT_IO_H__ */
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
index 826e548b2d4d..8b61c3eac035 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -13,6 +13,7 @@
#include "mtk_ctrl_plane.h"
#include "mtk_dev.h"
#include "mtk_pci.h"
+#include "mtk_port.h"
#include "mtk_trans_ctrl.h"
#define QUEUE_CHL_MASK 0xFFFF
@@ -31,6 +32,18 @@ static const struct queue_info mtk_queue_info[] = {
Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
};
+static const struct mtk_port_cfg mtk_port_cfg_tbl[] = {
+ {CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL",
+ PORT_F_DFLT},
+ {CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL",
+ PORT_F_DFLT},
+};
+
+static struct mtk_port_layer_cfg mtk_port_layer_cfg = {
+ .port_cfg = mtk_port_cfg_tbl,
+ .port_cnt = ARRAY_SIZE(mtk_port_cfg_tbl),
+};
+
static bool mtk_queue_list_is_full(struct mtk_ctrl_trans *trans, struct queue_info *que)
{
return skb_queue_len_lockless(&trans->trans_list[que->hif_id].skb_list[que->txqno]) >=
@@ -170,6 +183,7 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
break;
}
trb = (struct trb *)skb->cb;
+ kref_get(&trb->kref);
switch (trb->cmd) {
case TRB_CMD_ENABLE:
@@ -185,14 +199,17 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
break;
case TRB_CMD_TX:
spin_unlock_irqrestore(&skb_list->lock, flags);
+ mtk_port_tx_seq_stamp(skb);
err = mtk_cldma_submit_tx(trans->dev, skb);
if (err) {
if (trans_list->tx_burst_cnt[qno]) {
kick = true;
break;
}
- if (err == -EAGAIN)
+ if (err == -EAGAIN) {
+ kref_put(&trb->kref, mtk_port_trb_free);
return;
+ }
skb_unlink(skb, skb_list);
trb->status = err;
@@ -200,6 +217,7 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
break;
}
+ mtk_port_tx_seq_advance(skb);
trans_list->tx_burst_cnt[qno]++;
spin_lock_irqsave(&skb_list->lock, flags);
if (trans_list->tx_burst_cnt[qno] >= TX_BURST_MAX_CNT ||
@@ -233,6 +251,8 @@ static void mtk_ctrl_trb_handler(struct trb_srv *srv, struct trans_list *trans_l
kick = false;
}
+ kref_put(&trb->kref, mtk_port_trb_free);
+
loop++;
} while (loop < TRB_NUM_PER_ROUND);
}
@@ -479,6 +499,9 @@ int mtk_pcie_hif_exit(struct mtk_md_dev *mdev)
return 0;
}
+/* skb->cb holds a trb set up by mtk_port_trb_init(); a TX skb starts with
+ * its CCCI header and has the sending port as trb->priv.
+ */
int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
@@ -600,7 +623,7 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
trans->queue_info_num = ARRAY_SIZE(mtk_queue_info);
trans->trb_srv_num = TRB_SRV_NUM;
- err = mtk_ctrl_init(mdev);
+ err = mtk_ctrl_init(mdev, &mtk_port_layer_cfg);
if (err)
return err;
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
` (3 preceding siblings ...)
2026-11-05 7:56 ` [PATCH v11 4/6] net: wwan: t9xx: Add control port Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports Jack Wu via B4 Relay
5 siblings, 1 reply; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
The FSM (Finite-state Machine) thread is responsible for
synchronizing the actions of different modules. The
asynchronous events from the device or the OS will trigger
a state transition.
The FSM thread will append it to the event queue when an
event arrives. It handles the events sequentially. After
processing the event, the FSM thread notifies other modules
of the state transition.
Five FSM states are defined. They can transition from one
state to another, and self-transition in some states.
Also wire the HS1/HS2/HS3 runtime feature handshake into
the FSM: the host opens a control port on device readiness,
sends HS1 with the supported feature set, parses the HS2
reply, and confirms with HS3. MD and SAP each run this
sequence independently; the FSM advances to READY only
after both complete.
CLDMA hardware bring-up and teardown are driven by FSM
state transitions: mtk_cldma_dev_init() on FSM_STATE_BOOTUP
allocates DMA pools, creates a workqueue and registers the
MSI-X IRQ, and mtk_cldma_dev_exit() tears them down on
FSM_STATE_OFF.
Registering that IRQ is what first makes the completion
path built by the earlier patches run: mtk_cldma_isr()
dispatches tx_done_work and rx_done_work, and queues
err_work on a QUEUE_ERROR interrupt. err_work owns the
recovery policy: on a TX error it stops the offending
queue and completes every outstanding request with -EPIPE
rather than retrying it; a queue whose stop times out
keeps its requests and refuses new ones. An RX error is
handed to rx_done_work instead, which is the only context
that programs an RX queue, so a stop/restart pair cannot
race the worker's own resume.
A queue that will not stop within the timeout is still a
live DMA target, so its ring is deliberately leaked instead
of being unmapped and freed. The DMA pools are destroyed
regardless: dma_pool_destroy() does not free the pages of a
pool that still has blocks outstanding, so the leaked
descriptors stay allocated and only the pool bookkeeping is
released. The leak is bounded by the device instance and is
reported with dev_err().
Teardown is ordered to match. mtk_pcie_hif_exit() first
calls mtk_cldma_quiesce(), which silences each CLDMA
instance's interrupt output, masks the vector, waits out an
in-flight handler, drops the handler and drains the
workqueue. A quiesced instance is never re-armed, and the
trb service threads cannot queue CLDMA work through a
recovery once it is, so nothing is queued again before the
threads are joined and mtk_cldma_exit() frees what they
dereference.
Joining the kthreads is not sufficient on its own, because
the IRQ handler and the workqueue read trans->trb_srv[]
too.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/Makefile | 1 +
drivers/net/wwan/t9xx/mtk_ctrl_plane.c | 49 ++
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 1 +
drivers/net/wwan/t9xx/mtk_dev.h | 1 +
drivers/net/wwan/t9xx/mtk_fsm.c | 1085 +++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_fsm.h | 153 ++++
drivers/net/wwan/t9xx/mtk_port.c | 64 ++
drivers/net/wwan/t9xx/mtk_port.h | 2 +
drivers/net/wwan/t9xx/pcie/mtk_cldma.c | 534 ++++++++++++-
drivers/net/wwan/t9xx/pcie/mtk_cldma.h | 7 +
drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h | 11 +-
drivers/net/wwan/t9xx/pcie/mtk_pci.c | 37 +-
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 34 +-
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h | 3 +
14 files changed, 1948 insertions(+), 34 deletions(-)
diff --git a/drivers/net/wwan/t9xx/Makefile b/drivers/net/wwan/t9xx/Makefile
index d44266009362..fcfddf2e5147 100644
--- a/drivers/net/wwan/t9xx/Makefile
+++ b/drivers/net/wwan/t9xx/Makefile
@@ -9,6 +9,7 @@ mtk_t9xx-y := \
mtk_ctrl_plane.o \
mtk_port.o \
mtk_port_io.o \
+ mtk_fsm.o \
pcie/mtk_pci.o \
pcie/mtk_trans_ctrl.o \
pcie/mtk_cldma.o \
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
index af3064753b15..95c97efaf5b9 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.c
@@ -8,6 +8,41 @@
#include "mtk_ctrl_plane.h"
#include "mtk_port.h"
+#include "mtk_trans_ctrl.h"
+
+static void mtk_ctrl_trans_fsm_state_handler(struct mtk_fsm_param *param,
+ struct mtk_ctrl_blk *ctrl_blk)
+{
+ struct mtk_md_dev *mdev = ctrl_blk->mdev;
+ int ret;
+
+ switch (param->to) {
+ case FSM_STATE_OFF:
+ mtk_pcie_hif_exit(mdev);
+ mtk_pcie_hif_fsm_indication(mdev, param);
+ break;
+ case FSM_STATE_ON:
+ ret = mtk_pcie_hif_init(mdev);
+ mtk_fsm_hif_err_record(mdev, MTK_FSM_HIF_USER_CTRL, ret);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to init HIF: %d\n", ret);
+ break;
+ }
+ fallthrough;
+ default:
+ mtk_pcie_hif_fsm_indication(mdev, param);
+ break;
+ }
+}
+
+static void mtk_ctrl_fsm_state_listener(struct mtk_fsm_param *param, void *data)
+{
+ struct mtk_ctrl_blk *ctrl_blk = data;
+
+ mtk_port_mngr_fsm_state_handler(param, ctrl_blk->port_mngr);
+ mtk_ctrl_trans_fsm_state_handler(param, ctrl_blk);
+ mtk_port_mngr_fsm_state_handler_late(param, ctrl_blk->port_mngr);
+}
/**
* mtk_ctrl_init() - Initialize the control plane block.
@@ -40,7 +75,20 @@ int mtk_ctrl_init(struct mtk_md_dev *mdev, struct mtk_port_layer_cfg *port_layer
return err;
}
+ err = mtk_fsm_notifier_register(mdev, MTK_USER_CTRL, mtk_ctrl_fsm_state_listener,
+ ctrl_blk, FSM_PRIO_1);
+ if (err) {
+ dev_err(mdev->dev, "Fail to register fsm notification(ret = %d)\n", err);
+ goto err_port_exit;
+ }
+
return 0;
+
+err_port_exit:
+ mtk_port_mngr_exit(ctrl_blk);
+ mdev->ctrl_blk = NULL;
+
+ return err;
}
/**
@@ -57,6 +105,7 @@ void mtk_ctrl_exit(struct mtk_md_dev *mdev)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ mtk_fsm_notifier_unregister(mdev, MTK_USER_CTRL);
mtk_port_mngr_exit(ctrl_blk);
mdev->ctrl_blk = NULL;
}
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 3ba66d9a4b60..617c022a24c3 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -10,6 +10,7 @@
#include <linux/skbuff.h>
#include "mtk_dev.h"
+#include "mtk_fsm.h"
#define Q_MTU_3_5K (0xE00)
diff --git a/drivers/net/wwan/t9xx/mtk_dev.h b/drivers/net/wwan/t9xx/mtk_dev.h
index ff3cbaa4329e..fe876a8edbe1 100644
--- a/drivers/net/wwan/t9xx/mtk_dev.h
+++ b/drivers/net/wwan/t9xx/mtk_dev.h
@@ -37,6 +37,7 @@ struct mtk_md_dev {
u32 hw_ver;
char dev_str[MTK_DEV_STR_LEN];
struct mtk_ctrl_blk *ctrl_blk;
+ struct mtk_md_fsm *fsm;
};
#endif /* __MTK_DEV_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_fsm.c b/drivers/net/wwan/t9xx/mtk_fsm.c
new file mode 100644
index 000000000000..3350f6df5064
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_fsm.c
@@ -0,0 +1,1085 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#include <linux/bitfield.h>
+#include <linux/device.h>
+#include <linux/kref.h>
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/pci.h>
+#include <linux/sched/signal.h>
+#include <linux/sched/task.h>
+#include <linux/skbuff.h>
+#include <linux/wait.h>
+
+#include "mtk_fsm.h"
+#include "mtk_pci.h"
+#include "mtk_port.h"
+#include "mtk_port_io.h"
+
+#define EVT_TF_GATECLOSED (1)
+
+#define FSM_HS_START_MASK (FSM_F_SAP_HS_START | FSM_F_MD_HS_START)
+#define FSM_HS2_DONE_MASK (FSM_F_SAP_HS2_DONE | FSM_F_MD_HS2_DONE)
+
+#define RTFT_DATA_SIZE (3 * 1024)
+#define EVT_HANDLER_TIMEOUT (HZ * 30)
+#define BLOCKING_EVT_TIMEOUT (2 * EVT_HANDLER_TIMEOUT)
+
+#define REGION_BITMASK 0xF
+#define DEVICE_CFG_SHIFT 24
+#define DEVICE_CFG_REGION_MASK 0x3
+
+enum device_stage {
+ DEV_STAGE_IDLE = 4,
+ DEV_STAGE_MAX
+};
+
+enum device_cfg {
+ DEV_CFG_MD_ONLY = 1,
+};
+
+enum runtime_feature_support_type {
+ RTFT_TYPE_NOT_EXIST = 0,
+ RTFT_TYPE_NOT_SUPPORT = 1,
+ RTFT_TYPE_MUST_SUPPORT = 2,
+ RTFT_TYPE_OPTIONAL_SUPPORT = 3,
+ RTFT_TYPE_SUPPORT_BACKWARD_COMPAT = 4,
+};
+
+#define FEATURE_TYPE_NOT FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT)
+
+enum query_runtime_feature_id {
+ QUERY_RTFT_ID_MD_PORT_ENUM = 0,
+ QUERY_RTFT_ID_SAP_PORT_ENUM = 1,
+ QUERY_RTFT_ID_MD_PORT_CFG = 2,
+};
+
+enum ctrl_msg_id {
+ CTRL_MSG_HS1 = 0,
+ CTRL_MSG_HS2 = 1,
+ CTRL_MSG_HS3 = 2,
+};
+
+struct ctrl_msg_header {
+ __le32 id;
+ __le32 ex_msg;
+ __le32 data_len;
+ u8 reserved[];
+} __packed;
+
+struct runtime_feature_entry {
+ u8 feature_id;
+ struct runtime_feature_info support_info;
+ u8 reserved[2];
+ __le32 data_len;
+ u8 data[];
+} __packed;
+
+struct feature_query {
+ __le32 head_pattern;
+ struct runtime_feature_info ft_set[FEATURE_CNT];
+ __le32 tail_pattern;
+};
+
+static int mtk_fsm_send_hs1_msg(struct fsm_hs_info *hs_info)
+{
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct feature_query *ft_query;
+ struct sk_buff *skb;
+ int ret, msg_size;
+
+ msg_size = sizeof(*ctrl_msg_h) + sizeof(*ft_query);
+ skb = __dev_alloc_skb(msg_size, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+
+ skb_put(skb, msg_size);
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ ctrl_msg_h->id = cpu_to_le32(CTRL_MSG_HS1);
+ ctrl_msg_h->ex_msg = 0;
+ ctrl_msg_h->data_len = cpu_to_le32(sizeof(*ft_query));
+
+ ft_query = (struct feature_query *)(skb->data + sizeof(*ctrl_msg_h));
+ ft_query->head_pattern = cpu_to_le32(FEATURE_QUERY_PATTERN);
+ memcpy(ft_query->ft_set, hs_info->query_ft_set, sizeof(hs_info->query_ft_set));
+ ft_query->tail_pattern = cpu_to_le32(FEATURE_QUERY_PATTERN);
+
+ /* send handshake1 message to device */
+ ret = mtk_port_internal_write(hs_info->ctrl_port, skb);
+ if (ret <= 0)
+ return ret;
+
+ return 0;
+}
+
+static int mtk_fsm_feature_set_match(enum runtime_feature_support_type *cur_ft_spt,
+ struct runtime_feature_info rtft_info_st,
+ struct runtime_feature_info rtft_info_cfg)
+{
+ int ret = 0;
+
+ switch (FIELD_GET(FEATURE_TYPE, rtft_info_st.feature)) {
+ case RTFT_TYPE_NOT_EXIST:
+ fallthrough;
+ case RTFT_TYPE_NOT_SUPPORT:
+ /* Only a mandatory feature refused outright is fatal, here
+ * and in the MUST_SUPPORT case: no action a feature can carry
+ * is required to reach READY, so the other cases downgrade
+ * instead of failing.
+ */
+ if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_MUST_SUPPORT)
+ ret = -EPROTO;
+ else
+ *cur_ft_spt = RTFT_TYPE_NOT_EXIST;
+ break;
+ case RTFT_TYPE_MUST_SUPPORT:
+ if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_EXIST ||
+ FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_SUPPORT)
+ ret = -EPROTO;
+ else
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ break;
+ case RTFT_TYPE_OPTIONAL_SUPPORT:
+ if (FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_EXIST ||
+ FIELD_GET(FEATURE_TYPE, rtft_info_cfg.feature) == RTFT_TYPE_NOT_SUPPORT) {
+ *cur_ft_spt = RTFT_TYPE_NOT_SUPPORT;
+ } else {
+ if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) ==
+ FIELD_GET(FEATURE_VER, rtft_info_cfg.feature))
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ else
+ *cur_ft_spt = RTFT_TYPE_NOT_SUPPORT;
+ }
+ break;
+ case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT:
+ /* Decided on the device's declaration alone: the host's
+ * configured type can be the older of the two, and gating on
+ * it would keep off a feature the device just called
+ * compatible.
+ */
+ if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) >=
+ FIELD_GET(FEATURE_VER, rtft_info_cfg.feature))
+ *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
+ else
+ *cur_ft_spt = RTFT_TYPE_NOT_EXIST;
+ break;
+ default:
+ ret = -EPROTO;
+ }
+
+ return ret;
+}
+
+static int (*query_rtft_action[FEATURE_CNT])(struct mtk_md_dev *mdev,
+ void *rt_data, u32 data_len) = {
+ [QUERY_RTFT_ID_MD_PORT_ENUM] = mtk_port_status_update,
+ [QUERY_RTFT_ID_SAP_PORT_ENUM] = mtk_port_status_update,
+};
+
+/* The runtime data skb is owned by the caller (the FSM kthread, which got
+ * it attached to the STARTUP event); its pointer and length are read once
+ * from the skb itself, never from a shared field the rx path could rewrite.
+ */
+static int mtk_fsm_parse_hs2_msg(struct mtk_md_fsm *fsm, struct fsm_hs_info *hs_info,
+ struct sk_buff *skb)
+{
+ enum runtime_feature_support_type cur_ft_spt;
+ struct runtime_feature_entry *rtft_entry;
+ unsigned int ft_id, offset, data_len;
+ unsigned int rt_data_len;
+ char *rt_data;
+ int ret = 0;
+
+ if (!skb)
+ return -EINVAL;
+
+ rt_data = skb->data;
+ rt_data_len = skb->len;
+
+ offset = sizeof(struct feature_query);
+ for (ft_id = 0; ft_id < FEATURE_CNT; ft_id++) {
+ if (offset + sizeof(*rtft_entry) > rt_data_len)
+ break;
+
+ rtft_entry = (struct runtime_feature_entry *)(rt_data + offset);
+ ret = mtk_fsm_feature_set_match(&cur_ft_spt,
+ rtft_entry->support_info,
+ hs_info->query_ft_set[ft_id]);
+ if (ret < 0)
+ break;
+
+ data_len = le32_to_cpu(rtft_entry->data_len);
+ if (data_len > rt_data_len - offset - sizeof(*rtft_entry))
+ break;
+
+ if (cur_ft_spt == RTFT_TYPE_MUST_SUPPORT && query_rtft_action[ft_id]) {
+ if (!data_len) {
+ dev_err(fsm->mdev->dev,
+ "RTFT feature %u: zero data_len\n", ft_id);
+ ret = -EPROTO;
+ break;
+ }
+ ret = query_rtft_action[ft_id](fsm->mdev,
+ rtft_entry->data,
+ data_len);
+ }
+ if (ret < 0)
+ break;
+
+ offset += sizeof(*rtft_entry) + data_len;
+ }
+
+ if (ft_id != FEATURE_CNT) {
+ dev_err((fsm->mdev)->dev, "Unable to handle mistake hs2 msg, ft_id=%u\n", ft_id);
+ ret = -EPROTO;
+ }
+
+ return ret;
+}
+
+static int mtk_fsm_append_rtft_entries(void *feature_data, unsigned int *len,
+ struct sk_buff *skb)
+{
+ struct runtime_feature_entry *rtft_entry;
+ int ft_id, ret = 0, rtdata_len = 0;
+ struct feature_query *ft_query;
+
+ if (!skb || skb->len < sizeof(*ft_query)) {
+ ret = -EPROTO;
+ goto hs_err;
+ }
+
+ ft_query = (struct feature_query *)skb->data;
+ if (le32_to_cpu(ft_query->head_pattern) != FEATURE_QUERY_PATTERN ||
+ le32_to_cpu(ft_query->tail_pattern) != FEATURE_QUERY_PATTERN) {
+ ret = -EPROTO;
+ goto hs_err;
+ }
+
+ /* parse runtime feature query and fill runtime feature entry */
+ rtft_entry = feature_data;
+ for (ft_id = 0; ft_id < FEATURE_CNT && rtdata_len < RTFT_DATA_SIZE; ft_id++) {
+ rtft_entry->feature_id = ft_id;
+ rtft_entry->data_len = 0;
+
+ switch (FIELD_GET(FEATURE_TYPE, ft_query->ft_set[ft_id].feature)) {
+ case RTFT_TYPE_NOT_EXIST:
+ fallthrough;
+ case RTFT_TYPE_NOT_SUPPORT:
+ fallthrough;
+ case RTFT_TYPE_MUST_SUPPORT:
+ /* A MUST_SUPPORT feature is mandatory on the device side
+ * and takes no runtime data from the host, so it is
+ * echoed back unchanged with data_len 0.
+ */
+ rtft_entry->support_info = ft_query->ft_set[ft_id];
+ break;
+ case RTFT_TYPE_OPTIONAL_SUPPORT:
+ fallthrough;
+ case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT:
+ rtft_entry->support_info.feature = FEATURE_TYPE_NOT;
+ rtft_entry->support_info.feature |= FEATURE_VER_0;
+ break;
+ }
+
+ rtdata_len += sizeof(*rtft_entry) + le32_to_cpu(rtft_entry->data_len);
+ rtft_entry = (struct runtime_feature_entry *)(feature_data + rtdata_len);
+ }
+ *len = rtdata_len;
+ return 0;
+
+hs_err:
+ *len = 0;
+ return ret;
+}
+
+static int mtk_fsm_send_hs3_msg(struct fsm_hs_info *hs_info, struct sk_buff *rt_skb)
+{
+ unsigned int data_len, msg_size = 0;
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct sk_buff *skb;
+ int ret;
+
+ skb = __dev_alloc_skb(RTFT_DATA_SIZE, GFP_KERNEL);
+ if (!skb)
+ return -ENOMEM;
+ memset(skb->data, 0, RTFT_DATA_SIZE);
+
+ msg_size += sizeof(*ctrl_msg_h);
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ ctrl_msg_h->id = cpu_to_le32(CTRL_MSG_HS3);
+ ctrl_msg_h->ex_msg = 0;
+ ret = mtk_fsm_append_rtft_entries(skb->data + sizeof(*ctrl_msg_h),
+ &data_len, rt_skb);
+ if (ret) {
+ dev_kfree_skb(skb);
+ return ret;
+ }
+
+ ctrl_msg_h->data_len = cpu_to_le32(data_len);
+ msg_size += data_len;
+ skb_put(skb, msg_size);
+ ret = mtk_port_internal_write(hs_info->ctrl_port, skb);
+ if (ret <= 0)
+ return ret;
+
+ return 0;
+}
+
+/* The handler owns the skb on every path and always returns 0: the caller
+ * (mtk_port_rx_dispatch) frees the skb itself on a negative return, so
+ * returning an error after dev_kfree_skb() would free it twice.
+ *
+ * On success the skb is attached to the submitted event (event->data) and
+ * from then on belongs to the FSM kthread; hs_info->rt_data is kept only
+ * as the duplicate-HS2 guard, claimed with cmpxchg() and cleared by the
+ * kthread once the event has been consumed. An event discarded instead
+ * leaves its skb on the guard, and mtk_fsm_exit() frees it there. The
+ * kthread's store and the accesses below are the only concurrent users of
+ * the field, so they are marked: a stale non-NULL read at worst drops one
+ * retransmitted HS2, which the device repeats.
+ */
+static int mtk_fsm_ctrl_msg_handler(void *__hs_info, struct sk_buff *skb)
+{
+ struct fsm_hs_info *hs_info = __hs_info;
+ struct ctrl_msg_header *ctrl_msg_h;
+ struct mtk_md_fsm *fsm;
+ int ret;
+
+ fsm = container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]);
+ if (skb->len < sizeof(*ctrl_msg_h)) {
+ dev_kfree_skb(skb);
+ return 0;
+ }
+
+ ctrl_msg_h = (struct ctrl_msg_header *)skb->data;
+ if (le32_to_cpu(ctrl_msg_h->id) != CTRL_MSG_HS2) {
+ dev_err(fsm->mdev->dev, "Invalid ctrl msg id (hs%d)\n", hs_info->id);
+ dev_kfree_skb(skb);
+ return 0;
+ }
+
+ skb_pull(skb, sizeof(*ctrl_msg_h));
+ if (cmpxchg(&hs_info->rt_data, NULL, skb)) {
+ /* The guarded skb belongs to a still-queued event; only the
+ * new skb may be freed here.
+ */
+ dev_warn(fsm->mdev->dev, "Duplicate HS2 (hs%d), dropping\n", hs_info->id);
+ dev_kfree_skb(skb);
+ return 0;
+ }
+
+ ret = mtk_fsm_evt_submit(fsm->mdev, FSM_EVT_STARTUP,
+ hs_info->fsm_flag_hs2, skb, 0);
+ if (ret) {
+ WRITE_ONCE(hs_info->rt_data, NULL);
+ dev_kfree_skb(skb);
+ }
+
+ return 0;
+}
+
+/* For the MHCCIF callbacks, which can race DEV_RM masking the channels from
+ * the FSM thread. DEV_RM closes the event gate before it masks, so an
+ * unmask that lands after its mask sees the gate closed and is undone.
+ */
+static void mtk_fsm_unmask_ext_evt(struct mtk_md_fsm *fsm, u32 chs)
+{
+ mtk_pci_unmask_ext_evt(fsm->mdev, chs);
+ /* order the unmask before the gate test; pairs with the barrier in
+ * mtk_fsm_dev_rm_act()
+ */
+ mb();
+ if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag))
+ mtk_pci_mask_ext_evt(fsm->mdev, chs);
+}
+
+static void mtk_fsm_idle_evt_handler(struct mtk_md_dev *mdev,
+ u32 dev_state, struct mtk_md_fsm *fsm)
+{
+ u32 dev_cfg = dev_state >> DEVICE_CFG_SHIFT & DEVICE_CFG_REGION_MASK;
+ int hs_id;
+
+ if (dev_cfg == DEV_CFG_MD_ONLY)
+ WRITE_ONCE(fsm->hs_done_flag, FSM_F_MD_HS_START | FSM_F_MD_HS2_DONE);
+ else
+ WRITE_ONCE(fsm->hs_done_flag, FSM_HS_START_MASK | FSM_HS2_DONE_MASK);
+
+ if (mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP, FSM_F_DFLT,
+ NULL, 0))
+ dev_err(mdev->dev, "Failed to submit STARTUP evt\n");
+
+ /* The notification carrying this state is one-shot, so there is no
+ * retry to wait for: open the handshake channels whether or not the
+ * STARTUP above was queued. A bare HS1 drives ON->BOOTUP as well.
+ */
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++)
+ mtk_fsm_unmask_ext_evt(fsm, fsm->hs_info[hs_id].mhccif_ch);
+}
+
+static int mtk_fsm_early_bootup_handler(u32 status, void *__fsm)
+{
+ struct mtk_md_fsm *fsm = __fsm;
+ struct mtk_md_dev *mdev;
+ u32 dev_state, dev_stage;
+
+ mdev = fsm->mdev;
+ mtk_pci_mask_ext_evt(mdev, status);
+
+ dev_state = mtk_pci_get_dev_state(mdev);
+ dev_stage = dev_state & REGION_BITMASK;
+ if (dev_stage >= DEV_STAGE_MAX) {
+ dev_err(mdev->dev, "Invalid dev state 0x%x\n", dev_state);
+ goto exit;
+ }
+
+ if (dev_state == fsm->last_dev_state)
+ goto exit;
+
+ if (dev_stage == DEV_STAGE_IDLE)
+ mtk_fsm_idle_evt_handler(mdev, dev_state, fsm);
+
+ fsm->last_dev_state = dev_state;
+
+exit:
+ /* Re-arm the channel masked on entry: the device notifies once per
+ * boot stage, so leaving it masked drops every later stage.
+ */
+ mtk_fsm_unmask_ext_evt(fsm, status);
+ return 0;
+}
+
+static int mtk_fsm_ctrl_ch_start(struct mtk_md_fsm *fsm, struct fsm_hs_info *hs_info, int flag)
+{
+ if (!hs_info->ctrl_port) {
+ hs_info->ctrl_port = mtk_port_internal_open(fsm->mdev, hs_info->port_name, flag);
+ if (!hs_info->ctrl_port) {
+ dev_err(fsm->mdev->dev, "Failed to open ctrl port(%s)\n",
+ hs_info->port_name);
+ return -ENODEV;
+ }
+
+ mtk_port_internal_recv_register(hs_info->ctrl_port,
+ mtk_fsm_ctrl_msg_handler, hs_info);
+ }
+
+ return 0;
+}
+
+static void mtk_fsm_ctrl_ch_stop(struct mtk_md_fsm *fsm)
+{
+ struct fsm_hs_info *hs_info;
+ int hs_id;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ hs_info = &fsm->hs_info[hs_id];
+ if (hs_info->ctrl_port) {
+ mtk_port_internal_close(hs_info->ctrl_port);
+ hs_info->ctrl_port = NULL;
+ }
+ }
+}
+
+static void mtk_fsm_switch_state(struct mtk_md_fsm *fsm,
+ enum mtk_fsm_state to_state, struct mtk_fsm_evt *event)
+{
+ struct mtk_fsm_notifier *nt;
+ struct mtk_fsm_param param;
+
+ param.from = fsm->state;
+ param.to = to_state;
+ param.evt_id = event ? event->id : FSM_EVT_MAX;
+ param.fsm_flag = event ? event->fsm_flag : FSM_F_DFLT;
+
+ mutex_lock(&fsm->notifier_lock);
+ fsm->state = to_state;
+ fsm->fsm_flag |= event ? event->fsm_flag : FSM_F_DFLT;
+
+ list_for_each_entry(nt, &fsm->post_notifiers, entry)
+ nt->cb(¶m, nt->data);
+ mutex_unlock(&fsm->notifier_lock);
+}
+
+static int mtk_fsm_startup_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ enum mtk_fsm_state to_state = FSM_STATE_BOOTUP;
+ struct mtk_md_dev *mdev = fsm->mdev;
+ struct fsm_hs_info *hs_info;
+ struct sk_buff *skb = NULL;
+ u32 hs_done_flag;
+ int ret = 0;
+
+ if (event->fsm_flag & FSM_HS2_DONE_MASK) {
+ /* The HS2 event owns its runtime-data skb. */
+ skb = event->data;
+ hs_info = &fsm->hs_info[(event->fsm_flag & FSM_F_MD_HS2_DONE) ?
+ HS_ID_MD : HS_ID_SAP];
+ } else {
+ /* HS1 events carry the hs_info; the bare STARTUP carries NULL. */
+ hs_info = event->data;
+ }
+
+ if (fsm->state != FSM_STATE_ON && fsm->state != FSM_STATE_BOOTUP) {
+ ret = -EPROTO;
+ goto free_skb;
+ }
+
+ if (!(event->fsm_flag & (FSM_HS_START_MASK | FSM_HS2_DONE_MASK))) {
+ /* Bare STARTUP from the boot-flow notification: only the
+ * state switch.
+ */
+ if (fsm->state != FSM_STATE_BOOTUP)
+ mtk_fsm_switch_state(fsm, to_state, event);
+ return 0;
+ }
+
+ if (event->fsm_flag & FSM_HS_START_MASK) {
+ /* An HS1 that raced ahead of the bare STARTUP must both
+ * switch ON to BOOTUP and run the handshake below; returning
+ * after the switch would consume the only trigger.
+ */
+ mtk_fsm_switch_state(fsm, to_state, event);
+
+ ret = mtk_fsm_ctrl_ch_start(fsm, hs_info, O_NONBLOCK);
+ if (!ret)
+ ret = mtk_fsm_send_hs1_msg(hs_info);
+ if (ret)
+ goto hs_err;
+ } else if (event->fsm_flag & FSM_HS2_DONE_MASK) {
+ ret = mtk_fsm_parse_hs2_msg(fsm, hs_info, skb);
+ if (!ret)
+ ret = mtk_fsm_send_hs3_msg(hs_info, skb);
+ dev_kfree_skb(skb);
+ skb = NULL;
+ /* re-open the duplicate guard */
+ WRITE_ONCE(hs_info->rt_data, NULL);
+ if (ret)
+ goto hs_err;
+ mtk_fsm_switch_state(fsm, to_state, event);
+ }
+
+ hs_done_flag = READ_ONCE(fsm->hs_done_flag);
+ if (((fsm->fsm_flag | event->fsm_flag) & hs_done_flag) == hs_done_flag) {
+ int user;
+
+ for (user = 0; user < MTK_FSM_HIF_USER_MAX; user++) {
+ if (fsm->hif_err[user]) {
+ dev_err(mdev->dev,
+ "Refusing READY: HIF user %d init failed with %d\n",
+ user, fsm->hif_err[user]);
+ return fsm->hif_err[user];
+ }
+ }
+
+ to_state = FSM_STATE_READY;
+ mtk_fsm_switch_state(fsm, to_state, NULL);
+ }
+
+ return 0;
+
+free_skb:
+ if (skb) {
+ dev_kfree_skb(skb);
+ WRITE_ONCE(hs_info->rt_data, NULL);
+ }
+hs_err:
+ /* Re-arm only this handshake's channel, which its HS1 masked; the
+ * other one may still have its own HS1 queued.
+ */
+ if (hs_info)
+ mtk_pci_unmask_ext_evt(mdev, hs_info->mhccif_ch);
+ dev_err(mdev->dev, "Failed to hs with device %d:0x%x, ret=%d",
+ fsm->state, fsm->fsm_flag, ret);
+ return ret;
+}
+
+static void mtk_fsm_evt_release(struct kref *kref)
+{
+ struct mtk_fsm_evt *event = container_of(kref, struct mtk_fsm_evt, kref);
+
+ kfree(event);
+}
+
+static void mtk_fsm_evt_put(struct mtk_fsm_evt *event)
+{
+ kref_put(&event->kref, mtk_fsm_evt_release);
+}
+
+static void mtk_fsm_evt_finish(struct mtk_md_fsm *fsm,
+ struct mtk_fsm_evt *event, int retval)
+{
+ if (event->mode & EVT_MODE_BLOCKING) {
+ event->status = retval;
+ wake_up(&fsm->evt_waitq);
+ }
+ mtk_fsm_evt_put(event);
+}
+
+static void mtk_fsm_evt_cleanup(struct mtk_md_fsm *fsm, struct list_head *evtq)
+{
+ struct mtk_fsm_evt *event, *tmp;
+
+ list_for_each_entry_safe(event, tmp, evtq, entry) {
+ list_del(&event->entry);
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL);
+ }
+}
+
+static int mtk_fsm_enter_off_state(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ int hs_id;
+
+ if (fsm->state == FSM_STATE_OFF || fsm->state == FSM_STATE_INVALID)
+ return -EPROTO;
+
+ mtk_pci_mask_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++)
+ mtk_pci_mask_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch);
+
+ mtk_fsm_ctrl_ch_stop(fsm);
+ mtk_fsm_switch_state(fsm, FSM_STATE_OFF, event);
+
+ return 0;
+}
+
+static int mtk_fsm_dev_rm_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ set_bit(EVT_TF_GATECLOSED, &fsm->t_flag);
+ mtk_fsm_evt_cleanup(fsm, &fsm->evtq);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+ /* gate closed before the masks; see mtk_fsm_unmask_ext_evt() */
+ mb();
+
+ return mtk_fsm_enter_off_state(fsm, event);
+}
+
+static int mtk_fsm_hs1_handler(u32 status, void *__hs_info)
+{
+ struct fsm_hs_info *hs_info = __hs_info;
+ struct mtk_md_dev *mdev;
+ struct mtk_md_fsm *fsm;
+
+ fsm = container_of(hs_info, struct mtk_md_fsm, hs_info[hs_info->id]);
+ mdev = fsm->mdev;
+ /* Mask before the submit so exactly one side owns the channel: this
+ * handler until the event is queued, the FSM thread afterwards,
+ * which may already have re-armed it for a retry.
+ */
+ mtk_pci_mask_ext_evt(mdev, hs_info->mhccif_ch);
+ if (mtk_fsm_evt_submit(mdev, FSM_EVT_STARTUP, hs_info->fsm_flag_hs1,
+ hs_info, 0)) {
+ dev_err(mdev->dev, "Failed to submit HS1 evt(hs%d)\n", hs_info->id);
+ /* Nothing was handed over, so the channel is still ours */
+ mtk_fsm_unmask_ext_evt(fsm, hs_info->mhccif_ch);
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void mtk_fsm_hs_info_init_by_hsid(struct mtk_md_fsm *fsm, int hs_id)
+{
+ struct fsm_hs_info *hs_info;
+
+ hs_info = &fsm->hs_info[hs_id];
+ hs_info->id = hs_id;
+ hs_info->ctrl_port = NULL;
+ hs_info->rt_data = NULL;
+ switch (hs_id) {
+ case HS_ID_MD:
+ snprintf(hs_info->port_name, PORT_NAME_LEN, "MDCTRL");
+ hs_info->mhccif_ch = DEV_EVT_D2H_ASYNC_HS_NOTIFY_MD;
+ hs_info->fsm_flag_hs1 = FSM_F_MD_HS_START;
+ hs_info->fsm_flag_hs2 = FSM_F_MD_HS2_DONE;
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT);
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM].feature |=
+ FIELD_PREP(FEATURE_VER, 0);
+ hs_info->query_ft_set[QUERY_RTFT_ID_MD_PORT_CFG].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_NOT_SUPPORT);
+ break;
+ case HS_ID_SAP:
+ snprintf(hs_info->port_name, PORT_NAME_LEN, "SAPCTRL");
+ hs_info->mhccif_ch = DEV_EVT_D2H_ASYNC_HS_NOTIFY_SAP;
+ hs_info->fsm_flag_hs1 = FSM_F_SAP_HS_START;
+ hs_info->fsm_flag_hs2 = FSM_F_SAP_HS2_DONE;
+ hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature =
+ FIELD_PREP(FEATURE_TYPE, RTFT_TYPE_MUST_SUPPORT);
+ hs_info->query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM].feature |=
+ FIELD_PREP(FEATURE_VER, 0);
+ break;
+ }
+}
+
+static int mtk_fsm_hs_info_init(struct mtk_md_fsm *fsm)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ struct fsm_hs_info *hs_info;
+ int hs_id, ret;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ mtk_fsm_hs_info_init_by_hsid(fsm, hs_id);
+ hs_info = &fsm->hs_info[hs_id];
+ ret = mtk_pci_register_ext_evt(mdev, hs_info->mhccif_ch,
+ mtk_fsm_hs1_handler, hs_info);
+ if (ret)
+ goto err_unregister;
+ }
+
+ return 0;
+
+err_unregister:
+ /* All or nothing: the caller must not have to know how far we got. */
+ while (--hs_id >= 0)
+ mtk_pci_unregister_ext_evt(mdev, fsm->hs_info[hs_id].mhccif_ch);
+
+ return ret;
+}
+
+static void mtk_fsm_hs_info_exit(struct mtk_md_fsm *fsm)
+{
+ struct mtk_md_dev *mdev = fsm->mdev;
+ struct fsm_hs_info *hs_info;
+ int hs_id;
+
+ for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++) {
+ hs_info = &fsm->hs_info[hs_id];
+ mtk_pci_unregister_ext_evt(mdev, hs_info->mhccif_ch);
+ }
+}
+
+static int mtk_fsm_dev_add_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
+{
+ if (fsm->state != FSM_STATE_INVALID)
+ return -EPROTO;
+
+ mtk_fsm_switch_state(fsm, FSM_STATE_ON, event);
+ mtk_pci_unmask_ext_evt(fsm->mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+
+ return 0;
+}
+
+static int (*evts_act_tbl[FSM_EVT_MAX])(struct mtk_md_fsm *__fsm, struct mtk_fsm_evt *event) = {
+ [FSM_EVT_STARTUP] = mtk_fsm_startup_act,
+ [FSM_EVT_DEV_RM] = mtk_fsm_dev_rm_act,
+ [FSM_EVT_DEV_ADD] = mtk_fsm_dev_add_act,
+};
+
+/* Record a transport-plane init failure so the STARTUP path refuses the
+ * BOOTUP-to-READY promotion: the notifier callbacks are void and cannot
+ * reject a transition, so without this the FSM would advertise readiness
+ * on top of a data path that was never created. Each producer owns one
+ * slot and stores its latest result there, so a retried init that
+ * succeeds clears the failure it recorded before.
+ */
+void mtk_fsm_hif_err_record(struct mtk_md_dev *mdev, enum mtk_fsm_hif_user user, int err)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+
+ if (fsm)
+ fsm->hif_err[user] = err;
+}
+
+static void mtk_fsm_notifier_cleanup(struct mtk_md_dev *mdev, struct list_head *ntq)
+{
+ struct mtk_fsm_notifier *nt, *tmp;
+
+ list_for_each_entry_safe(nt, tmp, ntq, entry) {
+ list_del(&nt->entry);
+ dev_warn(mdev->dev, "Having to free notifier(%d) by FSM!\n", nt->id);
+ kfree(nt);
+ }
+}
+
+static void mtk_fsm_notifier_insert(struct mtk_fsm_notifier *notifier, struct list_head *head)
+{
+ struct mtk_fsm_notifier *nt;
+
+ list_for_each_entry(nt, head, entry) {
+ if (notifier->prio > nt->prio) {
+ list_add(¬ifier->entry, nt->entry.prev);
+ return;
+ }
+ }
+ list_add_tail(¬ifier->entry, head);
+}
+
+int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id,
+ void (*cb)(struct mtk_fsm_param *, void *data),
+ void *data, enum mtk_fsm_prio prio)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_notifier *notifier;
+
+ if (!fsm)
+ return -EINVAL;
+
+ if (id >= MTK_USER_MAX || !cb || prio >= FSM_PRIO_MAX)
+ return -EINVAL;
+
+ notifier = kzalloc_obj(*notifier);
+ if (!notifier)
+ return -ENOMEM;
+
+ INIT_LIST_HEAD(¬ifier->entry);
+ notifier->id = id;
+ notifier->cb = cb;
+ notifier->data = data;
+ notifier->prio = prio;
+
+ mutex_lock(&fsm->notifier_lock);
+ mtk_fsm_notifier_insert(notifier, &fsm->post_notifiers);
+ mutex_unlock(&fsm->notifier_lock);
+
+ return 0;
+}
+
+int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id id)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_notifier *nt, *tmp;
+
+ if (!fsm)
+ return -EINVAL;
+
+ mutex_lock(&fsm->notifier_lock);
+ list_for_each_entry_safe(nt, tmp, &fsm->post_notifiers, entry) {
+ if (nt->id == id) {
+ list_del(&nt->entry);
+ kfree(nt);
+ break;
+ }
+ }
+ mutex_unlock(&fsm->notifier_lock);
+ return 0;
+}
+
+int mtk_fsm_evt_submit(struct mtk_md_dev *mdev,
+ enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag,
+ void *data, unsigned char mode)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct mtk_fsm_evt *event;
+ unsigned long flags;
+ int ret = 0;
+
+ if (!fsm || id >= FSM_EVT_MAX) {
+ dev_err(mdev->dev, "Invalid param!\n");
+ return -EINVAL;
+ }
+
+ if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) {
+ dev_err(mdev->dev, "Failed to submit evt, fsm has been removed!\n");
+ return -ENODEV;
+ }
+
+ /* callable from atomic context unless EVT_MODE_BLOCKING */
+ event = kzalloc(sizeof(*event), GFP_ATOMIC);
+ if (!event)
+ return -ENOMEM;
+
+ kref_init(&event->kref);
+ event->id = id;
+ event->fsm_flag = flag;
+ event->status = FSM_EVT_RET_ONGOING;
+ event->data = data;
+ event->mode = mode;
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ if (test_bit(EVT_TF_GATECLOSED, &fsm->t_flag)) {
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+ mtk_fsm_evt_put(event);
+ dev_err(mdev->dev, "Failed to add event, fsm dev has been removed!\n");
+ return -ENODEV;
+ }
+
+ kref_get(&event->kref);
+ if (mode & EVT_MODE_TOHEAD)
+ list_add(&event->entry, &fsm->evtq);
+ else
+ list_add_tail(&event->entry, &fsm->evtq);
+ if (fsm->fsm_handler)
+ wake_up_process(fsm->fsm_handler);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ if (mode & EVT_MODE_BLOCKING) {
+ ret = wait_event_timeout(fsm->evt_waitq,
+ (event->status != 0), BLOCKING_EVT_TIMEOUT);
+ if (event->status == FSM_EVT_RET_DONE) {
+ ret = 0;
+ } else if (!ret) {
+ dev_err(mdev->dev, "Handling fsm blocking event timeout!\n");
+ ret = -ETIMEDOUT;
+ } else {
+ ret = -EIO;
+ }
+ }
+ mtk_fsm_evt_put(event);
+
+ return ret;
+}
+
+static int mtk_fsm_evt_handler(void *__fsm)
+{
+ struct mtk_md_fsm *fsm = __fsm;
+ struct mtk_fsm_evt *event;
+ unsigned long flags;
+ int ret;
+
+wake_up:
+ set_current_state(TASK_INTERRUPTIBLE);
+ while (!kthread_should_stop() && !list_empty(&fsm->evtq)) {
+ set_current_state(TASK_RUNNING);
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ event = list_first_entry(&fsm->evtq, struct mtk_fsm_evt, entry);
+ list_del(&event->entry);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ if (event->id < FSM_EVT_MAX) {
+ ret = evts_act_tbl[event->id](fsm, event);
+ if (ret) {
+ dev_err((fsm->mdev)->dev,
+ "Failed to handle evt, fsm state = %d, ret = %d\n",
+ fsm->state, ret);
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_FAIL);
+ } else {
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE);
+ }
+ } else {
+ mtk_fsm_evt_finish(fsm, event, FSM_EVT_RET_DONE);
+ }
+ }
+
+ if (kthread_should_stop()) {
+ set_current_state(TASK_RUNNING);
+ return 0;
+ }
+
+ schedule();
+ goto wake_up;
+}
+
+int mtk_fsm_init(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm;
+ int ret;
+
+ fsm = devm_kzalloc(mdev->dev, sizeof(*fsm), GFP_KERNEL);
+ if (!fsm)
+ return -ENOMEM;
+
+ fsm->fsm_handler = kthread_create(mtk_fsm_evt_handler, fsm, "fsm_evt_thread%u_%s",
+ mdev->hw_ver, mdev->dev_str);
+ if (IS_ERR(fsm->fsm_handler))
+ return PTR_ERR(fsm->fsm_handler);
+
+ /* Keep our own reference so that kthread_stop() in the teardown
+ * paths stays safe even if the thread has already exited (e.g.
+ * it was killed by an oops in a listener it called): without it
+ * the task_struct may be freed and kthread_stop() would hit a
+ * use-after-free through its internal get_task_struct().
+ */
+ get_task_struct(fsm->fsm_handler);
+
+ fsm->mdev = mdev;
+ fsm->state = FSM_STATE_INVALID;
+ fsm->fsm_flag = FSM_F_DFLT;
+
+ INIT_LIST_HEAD(&fsm->evtq);
+ spin_lock_init(&fsm->evtq_lock);
+ init_waitqueue_head(&fsm->evt_waitq);
+
+ INIT_LIST_HEAD(&fsm->post_notifiers);
+ mutex_init(&fsm->notifier_lock);
+
+ ret = mtk_pci_register_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC,
+ mtk_fsm_early_bootup_handler, fsm);
+ if (ret)
+ goto err_stop_thread;
+
+ ret = mtk_fsm_hs_info_init(fsm);
+ if (ret)
+ goto err_unregister_evt;
+
+ mdev->fsm = fsm;
+ return 0;
+
+err_unregister_evt:
+ mtk_pci_unregister_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+err_stop_thread:
+ /* mdev->fsm is never published on failure, so mtk_fsm_exit() cannot
+ * stop the parked kthread for us; it must be stopped here.
+ */
+ kthread_stop(fsm->fsm_handler);
+ put_task_struct(fsm->fsm_handler);
+ return ret;
+}
+
+/* Close the event gate and join the handler thread, then release the
+ * control ports while the port table they reference is still alive.
+ * Called from the removal path before the transport plane is dismantled;
+ * every step is idempotent, so mtk_fsm_exit() may repeat them safely.
+ */
+int mtk_fsm_stop(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+ struct task_struct *handler;
+ unsigned long flags;
+
+ if (!fsm)
+ return -EINVAL;
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ set_bit(EVT_TF_GATECLOSED, &fsm->t_flag);
+ handler = fsm->fsm_handler;
+ fsm->fsm_handler = NULL;
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ if (handler) {
+ kthread_stop(handler);
+ put_task_struct(handler);
+ }
+
+ spin_lock_irqsave(&fsm->evtq_lock, flags);
+ mtk_fsm_evt_cleanup(fsm, &fsm->evtq);
+ spin_unlock_irqrestore(&fsm->evtq_lock, flags);
+
+ mtk_fsm_ctrl_ch_stop(fsm);
+
+ return 0;
+}
+
+int mtk_fsm_exit(struct mtk_md_dev *mdev)
+{
+ struct mtk_md_fsm *fsm = mdev->fsm;
+
+ if (!fsm)
+ return -EINVAL;
+
+ /* also releases the control ports when DEV_RM never ran */
+ mtk_fsm_stop(mdev);
+
+ for (int i = 0; i < HS_ID_MAX; i++) {
+ if (fsm->hs_info[i].rt_data) {
+ dev_kfree_skb(fsm->hs_info[i].rt_data);
+ fsm->hs_info[i].rt_data = NULL;
+ }
+ }
+
+ mutex_lock(&fsm->notifier_lock);
+ mtk_fsm_notifier_cleanup(mdev, &fsm->post_notifiers);
+ mutex_unlock(&fsm->notifier_lock);
+
+ mtk_pci_unregister_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
+ mtk_fsm_hs_info_exit(fsm);
+ mdev->fsm = NULL;
+
+ return 0;
+}
diff --git a/drivers/net/wwan/t9xx/mtk_fsm.h b/drivers/net/wwan/t9xx/mtk_fsm.h
new file mode 100644
index 000000000000..f809579dc57d
--- /dev/null
+++ b/drivers/net/wwan/t9xx/mtk_fsm.h
@@ -0,0 +1,153 @@
+/* SPDX-License-Identifier: GPL-2.0-only
+ *
+ * Copyright (c) 2022, MediaTek Inc.
+ */
+
+#ifndef __MTK_FSM_H__
+#define __MTK_FSM_H__
+
+#include <linux/mutex.h>
+
+#include "mtk_dev.h"
+
+#define FEATURE_CNT (64)
+#define FEATURE_QUERY_PATTERN (0x49434343)
+
+#define FEATURE_TYPE GENMASK(3, 0)
+#define FEATURE_VER GENMASK(7, 4)
+
+#define FEATURE_VER_0 FIELD_PREP(FEATURE_VER, 0)
+
+#define EVT_MODE_BLOCKING (0x01)
+#define EVT_MODE_TOHEAD (0x02)
+
+#define FSM_EVT_RET_FAIL (-1)
+#define FSM_EVT_RET_ONGOING (0)
+#define FSM_EVT_RET_DONE (1)
+
+enum mtk_fsm_flag {
+ FSM_F_DFLT = 0,
+ FSM_F_SAP_HS_START = BIT(0),
+ FSM_F_SAP_HS2_DONE = BIT(1),
+ FSM_F_MD_HS_START = BIT(2),
+ FSM_F_MD_HS2_DONE = BIT(3),
+};
+
+enum mtk_fsm_state {
+ FSM_STATE_INVALID = 0,
+ FSM_STATE_OFF,
+ FSM_STATE_ON,
+ FSM_STATE_BOOTUP,
+ FSM_STATE_READY,
+};
+
+enum mtk_fsm_evt_id {
+ FSM_EVT_STARTUP = 0,
+ FSM_EVT_DEV_RM,
+ FSM_EVT_DEV_ADD,
+ FSM_EVT_MAX
+};
+
+enum mtk_fsm_prio {
+ FSM_PRIO_1 = 1,
+ FSM_PRIO_MAX
+};
+
+/* One slot per transport-plane producer, so that a retried init clears
+ * only its own record.
+ */
+enum mtk_fsm_hif_user {
+ MTK_FSM_HIF_USER_CTRL = 0,
+ MTK_FSM_HIF_USER_CLDMA0,
+ MTK_FSM_HIF_USER_CLDMA1,
+ MTK_FSM_HIF_USER_MAX
+};
+
+struct mtk_fsm_param {
+ enum mtk_fsm_state from;
+ enum mtk_fsm_state to;
+ enum mtk_fsm_evt_id evt_id;
+ enum mtk_fsm_flag fsm_flag;
+};
+
+#define PORT_NAME_LEN 20
+
+enum handshake_info_id {
+ HS_ID_MD = 0,
+ HS_ID_SAP,
+ HS_ID_MAX
+};
+
+struct runtime_feature_info {
+ u8 feature;
+};
+
+struct fsm_hs_info {
+ unsigned char id;
+ void *ctrl_port;
+ char port_name[PORT_NAME_LEN];
+ unsigned int mhccif_ch;
+ unsigned int fsm_flag_hs1;
+ unsigned int fsm_flag_hs2;
+ /* the feature that the device should support */
+ struct runtime_feature_info query_ft_set[FEATURE_CNT];
+ /* Duplicate-HS2 guard only: the in-flight runtime-data skb itself
+ * is owned by the STARTUP event it was submitted with.
+ */
+ void *rt_data;
+};
+
+struct mtk_md_fsm {
+ struct mtk_md_dev *mdev;
+ struct task_struct *fsm_handler;
+ struct fsm_hs_info hs_info[HS_ID_MAX];
+ unsigned int hs_done_flag;
+ unsigned long t_flag;
+ u32 last_dev_state;
+ /* last transport-plane init result per producer; any non-zero entry
+ * refuses READY
+ */
+ int hif_err[MTK_FSM_HIF_USER_MAX];
+ enum mtk_fsm_state state;
+ unsigned int fsm_flag;
+ struct list_head evtq;
+ /* protect evtq */
+ spinlock_t evtq_lock;
+ /* waitq for fsm blocking submit */
+ wait_queue_head_t evt_waitq;
+ struct list_head post_notifiers;
+ /* protects post_notifiers */
+ struct mutex notifier_lock;
+};
+
+struct mtk_fsm_evt {
+ struct list_head entry;
+ struct kref kref;
+ enum mtk_fsm_evt_id id;
+ unsigned int fsm_flag;
+ int status;
+ unsigned char mode;
+ void *data;
+};
+
+struct mtk_fsm_notifier {
+ struct list_head entry;
+ enum mtk_user_id id;
+ void (*cb)(struct mtk_fsm_param *param, void *data);
+ void *data;
+ enum mtk_fsm_prio prio;
+};
+
+int mtk_fsm_init(struct mtk_md_dev *mdev);
+int mtk_fsm_exit(struct mtk_md_dev *mdev);
+int mtk_fsm_stop(struct mtk_md_dev *mdev);
+void mtk_fsm_hif_err_record(struct mtk_md_dev *mdev, enum mtk_fsm_hif_user user, int err);
+int mtk_fsm_notifier_register(struct mtk_md_dev *mdev, enum mtk_user_id id,
+ void (*cb)(struct mtk_fsm_param *, void *data),
+ void *data, enum mtk_fsm_prio prio);
+int mtk_fsm_notifier_unregister(struct mtk_md_dev *mdev, enum mtk_user_id id);
+int mtk_fsm_evt_submit(struct mtk_md_dev *mdev,
+ enum mtk_fsm_evt_id id, enum mtk_fsm_flag flag,
+ void *data, unsigned char mode);
+
+#endif /* __MTK_FSM_H__ */
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
index 7fec696bd6db..101ec14fc993 100644
--- a/drivers/net/wwan/t9xx/mtk_port.c
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -626,6 +626,70 @@ int mtk_port_ch_disable(struct mtk_port *port)
return ret;
}
+static void mtk_port_disable(struct mtk_port_mngr *port_mngr)
+{
+ struct radix_tree_iter iter;
+ struct mtk_port *port;
+ void __rcu **slot;
+ int tbl_type;
+
+ tbl_type = PORT_TBL_SAP;
+ do {
+ radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type],
+ &iter, 0) {
+ port = mtk_port_tbl_slot(&slot, &iter);
+ if (!port)
+ continue;
+ mtk_port_get_ops(port->info.type)->disable(port);
+ }
+ } while (++tbl_type < PORT_TBL_MAX);
+}
+
+void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg)
+{
+ struct mtk_port_mngr *port_mngr;
+
+ if (!fsm_param || !arg)
+ return;
+
+ port_mngr = arg;
+
+ switch (fsm_param->to) {
+ case FSM_STATE_OFF:
+ mtk_port_disable(port_mngr);
+ break;
+ default:
+ break;
+ }
+}
+
+void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param, void *arg)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct mtk_port *port;
+
+ if (!fsm_param || !arg)
+ return;
+
+ port_mngr = arg;
+
+ switch (fsm_param->to) {
+ case FSM_STATE_BOOTUP:
+ if (fsm_param->fsm_flag & FSM_F_MD_HS_START) {
+ port = mtk_port_search_by_id(port_mngr, CCCI_CONTROL_RX);
+ if (port)
+ mtk_port_get_ops(port->info.type)->enable(port);
+ } else if (fsm_param->fsm_flag & FSM_F_SAP_HS_START) {
+ port = mtk_port_search_by_id(port_mngr, CCCI_SAP_CONTROL_RX);
+ if (port)
+ mtk_port_get_ops(port->info.type)->enable(port);
+ }
+ break;
+ default:
+ break;
+ }
+}
+
int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, const struct mtk_port_cfg *port_cfg,
int port_cnt)
{
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
index 157eda4b4220..d160f73c1834 100644
--- a/drivers/net/wwan/t9xx/mtk_port.h
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -125,6 +125,8 @@ int mtk_port_send_data(struct mtk_port *port, void *data, bool blocking, bool fo
int mtk_port_status_update(struct mtk_md_dev *mdev, void *data, u32 data_len);
int mtk_port_ch_enable(struct mtk_port *port);
int mtk_port_ch_disable(struct mtk_port *port);
+void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg);
+void mtk_port_mngr_fsm_state_handler_late(struct mtk_fsm_param *fsm_param, void *arg);
int mtk_port_mngr_init(struct mtk_ctrl_blk *ctrl_blk, const struct mtk_port_cfg *port_cfg,
int port_cnt);
void mtk_port_mngr_exit(struct mtk_ctrl_blk *ctrl_blk);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
index d8763cdd67da..4893edfebf79 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
@@ -27,15 +27,176 @@
#include "mtk_cldma_drv.h"
#include "mtk_dev.h"
+#define DMA_POOL_NAME_LEN (64)
#define WAIT_HWO_ROUND (10)
#define WAIT_HWO_TIME (5)
#define NO_BUDGET (0)
+static void mtk_cldma_err_work(struct work_struct *work);
+
+static int mtk_cldma_isr(int irq_id, void *param)
+{
+ struct cldma_drv_info *drv_info = param;
+ u32 tx_err, rx_err;
+ struct mtk_md_dev *mdev;
+ u32 tx_done, rx_done;
+ u32 tx_sta, rx_sta;
+ struct txq *txq;
+ struct rxq *rxq;
+ int i;
+
+ mdev = drv_info->mdev;
+ mtk_cldma_get_intr_status(drv_info, &tx_sta, &rx_sta);
+ tx_done = (tx_sta >> QUEUE_XFER_DONE) & 0xFF;
+ rx_done = (rx_sta >> QUEUE_XFER_DONE) & 0xFF;
+ tx_err = (tx_sta >> QUEUE_ERROR) & 0xFF;
+ rx_err = (rx_sta >> QUEUE_ERROR) & 0xFF;
+
+ if (tx_err || rx_err) {
+ dev_err_ratelimited(mdev->dev, "CLDMA%d queue error: TX 0x%x RX 0x%x\n",
+ drv_info->hif_id, tx_err, rx_err);
+ /* Left masked by mtk_cldma_get_intr_status() until err_work
+ * has handled the queue: a queue still in error would raise
+ * it again as soon as it is unmasked.
+ */
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ if (tx_err & BIT(i))
+ mtk_cldma_clr_intr_status(drv_info, DIR_TX,
+ i, QUEUE_ERROR);
+ if (rx_err & BIT(i))
+ mtk_cldma_clr_intr_status(drv_info, DIR_RX,
+ i, QUEUE_ERROR);
+ }
+ atomic_or(tx_err, &drv_info->tx_err_qs);
+ atomic_or(rx_err, &drv_info->rx_err_qs);
+ queue_work(drv_info->wq, &drv_info->err_work);
+ }
+
+ if (tx_done) {
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ /* pairs with smp_store_release() in txq_alloc */
+ txq = smp_load_acquire(&drv_info->txq[i]);
+ if (!(tx_done & BIT(i)) || !txq)
+ continue;
+ queue_work(drv_info->wq, &txq->tx_done_work);
+ }
+ }
+ if (rx_done) {
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ /* pairs with smp_store_release() in rxq_alloc */
+ rxq = smp_load_acquire(&drv_info->rxq[i]);
+ if (!(rx_done & BIT(i)) || !rxq)
+ continue;
+ queue_work(drv_info->wq, &rxq->rx_done_work);
+ }
+ }
+
+ mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id);
+ mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id);
+
+ return IRQ_HANDLED;
+}
+
static const int mtk_cldma_hw_id_tbl[NR_CLDMA] = {
[CLDMA0] = CLDMA0_HW_ID,
[CLDMA1] = CLDMA1_HW_ID,
};
+static int mtk_cldma_dev_init(struct cldma_dev *cd, int hif_id)
+{
+ char gpd_pool_name[DMA_POOL_NAME_LEN];
+ struct cldma_drv_info *drv_info;
+ const struct cldma_hw_regs *hw_regs;
+ struct mtk_md_dev *mdev;
+ unsigned int flag;
+ int hw_id, ret;
+
+ if (!cd || hif_id >= NR_CLDMA)
+ return -EINVAL;
+
+ if (cd->cldma_drv_info[hif_id])
+ return 0;
+
+ hw_id = mtk_cldma_hw_id_tbl[hif_id];
+ mdev = cd->trans->mdev;
+ drv_info = kzalloc_obj(*drv_info);
+ if (!drv_info)
+ return -ENOMEM;
+
+ drv_info->cd = cd;
+ drv_info->mdev = mdev;
+ drv_info->hif_id = hif_id;
+ drv_info->hw_id = hw_id;
+ drv_info->hw_regs = &mtk_cldma_regs_m9xx;
+
+ hw_regs = drv_info->hw_regs;
+ snprintf(gpd_pool_name, DMA_POOL_NAME_LEN, "cldma%d_gpd_pool_%s",
+ hw_id, mdev->dev_str);
+ drv_info->gpd_dma_pool = dma_pool_create(gpd_pool_name, mdev->dev,
+ sizeof(union gpd), 16, 0);
+ if (!drv_info->gpd_dma_pool) {
+ dev_err(mdev->dev, "Failed to alloc gpd dma pool for cldma%d\n", hw_id);
+ ret = -ENOMEM;
+ goto err_free_drv_info;
+ }
+
+ switch (hif_id) {
+ case CLDMA0:
+ drv_info->pci_ext_irq_id = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA0);
+ drv_info->base_addr = hw_regs->cldma0_base_addr;
+ break;
+ case CLDMA1:
+ drv_info->pci_ext_irq_id = mtk_pci_get_irq_id(mdev, MTK_IRQ_SRC_CLDMA1);
+ drv_info->base_addr = hw_regs->cldma1_base_addr;
+ break;
+ default:
+ ret = -EINVAL;
+ goto err_destroy_gpd_pool;
+ }
+
+ flag = WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_HIGHPRI;
+ drv_info->wq = alloc_workqueue("cldma%d_workq_%s", flag, 0, hw_id, mdev->dev_str);
+ if (!drv_info->wq) {
+ dev_err(mdev->dev, "Failed to alloc work queue for cldma%d\n", hw_id);
+ ret = -ENOMEM;
+ goto err_destroy_gpd_pool;
+ }
+
+ INIT_WORK(&drv_info->err_work, mtk_cldma_err_work);
+ mutex_init(&drv_info->rearm_lock);
+ atomic_set(&drv_info->tx_err_qs, 0);
+ atomic_set(&drv_info->rx_err_qs, 0);
+
+ mtk_cldma_drv_init(drv_info);
+
+ /* mask/clear PCI CLDMA L1 interrupt */
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+ mtk_pci_clear_irq(mdev, drv_info->pci_ext_irq_id);
+
+ /* register CLDMA interrupt handler */
+ ret = mtk_pci_register_irq(mdev, drv_info->pci_ext_irq_id, mtk_cldma_isr, drv_info);
+ if (ret)
+ goto err_destroy_wq;
+
+ /* unmask PCI CLDMA L1 interrupt */
+ mtk_pci_unmask_irq(mdev, drv_info->pci_ext_irq_id);
+
+ /* Publish only once drv_info is fully built; pairs with the acquire
+ * loads in the transfer paths below.
+ */
+ smp_store_release(&cd->cldma_drv_info[hif_id], drv_info);
+ return 0;
+
+err_destroy_wq:
+ destroy_workqueue(drv_info->wq);
+err_destroy_gpd_pool:
+ dma_pool_destroy(drv_info->gpd_dma_pool);
+err_free_drv_info:
+ kfree(drv_info);
+
+ return ret;
+}
+
static void mtk_cldma_tx_done_work(struct work_struct *work)
{
struct txq *txq = container_of(work, struct txq, tx_done_work);
@@ -56,6 +217,10 @@ static void mtk_cldma_tx_done_work(struct work_struct *work)
txqno = txq->txqno;
mdev = drv_info->mdev;
trans = drv_info->cd->trans;
+ /* One load for the whole walk: the slot is written only by service
+ * init and teardown, and teardown cannot run while this worker does.
+ */
+ srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
again:
for (i = 0; i < txq->nr_gpds; i++) {
@@ -93,10 +258,9 @@ static void mtk_cldma_tx_done_work(struct work_struct *work)
trb->status = 0;
trb->trb_complete(skb);
- /* The service may already have been torn down; completing the
- * TRB is still correct, there is just nobody left to wake.
+ /* The service may never have been started; completing the TRB
+ * is still correct, there is just nobody left to wake.
*/
- srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
if (was_starved && srv)
wake_up(&srv->trb_waitq);
}
@@ -215,15 +379,32 @@ static bool mtk_cldma_rx_check_again(struct rxq *rxq)
return true;
}
-/* Point the device at the GPD it should fill next and start the queue.
- * free_idx is owned by mtk_cldma_rx_done_work(), so this may only run
- * from that worker, at a point where every consumed GPD is re-armed.
+/* May only run from mtk_cldma_rx_done_work(), which owns free_idx and is the
+ * only context allowed to program an RX queue. A work item never runs
+ * concurrently with itself, so keeping the stop here too - rather than in
+ * whoever asked for the restart - serialises it without a lock.
*/
static void mtk_cldma_rxq_restart(struct cldma_drv_info *drv_info, struct rxq *rxq)
{
+ int ret;
+
+ ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxq->rxqno);
+ if (ret) {
+ /* Leave it stopped, with QUEUE_ERROR masked: TRB_CMD_DISABLE
+ * tears the queue down and has the leak-the-ring handling for
+ * a stop that timed out.
+ */
+ dev_err(drv_info->mdev->dev,
+ "RX queue %u stop failed (%d), leaving it stopped\n",
+ rxq->rxqno, ret);
+ return;
+ }
+
mtk_cldma_setup_start_addr(drv_info, DIR_RX, rxq->rxqno,
rxq->req_pool[rxq->free_idx].gpd_dma_addr);
mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
+ /* the ISR left it masked if an error is what asked for the restart */
+ mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_ERROR);
}
static void mtk_cldma_rx_done_work(struct work_struct *work)
@@ -331,7 +512,7 @@ static void mtk_cldma_rx_done_work(struct work_struct *work)
* not wait, such as a DMA mapping one.
*/
usleep_range(1000, 2000);
- if (!atomic_read(&rxq->need_exit))
+ if (!atomic_read(&rxq->need_exit) && !READ_ONCE(drv_info->quiesced))
queue_work(drv_info->wq, &rxq->rx_done_work);
out:
mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
@@ -358,6 +539,7 @@ static struct txq *mtk_cldma_txq_alloc(struct cldma_drv_info *drv_info, struct q
txq->nr_gpds = txq->que->tx_nr_gpds;
atomic_set(&txq->req_budget, txq->que->tx_nr_gpds);
spin_lock_init(&txq->ring_lock);
+ txq->is_stopping = false;
txq->req_pool = kcalloc(txq->nr_gpds, sizeof(*req), GFP_KERNEL);
if (!txq->req_pool)
@@ -427,10 +609,13 @@ static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
struct rxq *rxq;
int i;
- /* The caller holds rearm_lock from the reset to the end of the re-arm:
- * no queue is kicked or resumed while the instance has no addresses.
+ /* The caller holds rearm_lock across the whole re-arm, so the quiesce
+ * either drains what this queued or makes it return here: no work is
+ * queued after a quiesce.
*/
lockdep_assert_held(&drv_info->rearm_lock);
+ if (drv_info->quiesced)
+ return;
mtk_cldma_drv_init(drv_info);
@@ -445,8 +630,12 @@ static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
* next start_xfer() issue a fresh START.
*/
req = txq->req_pool + txq->free_idx;
- if (READ_ONCE(txq->tx_started) &&
- !(req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
+ if (txq->is_stopping) {
+ /* Being stopped: restarting it would let the
+ * device walk buffers the stopper is unmapping.
+ */
+ } else if (READ_ONCE(txq->tx_started) &&
+ !(req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
WRITE_ONCE(txq->tx_started, false);
/* Its XFER_DONE may have been lost in the
* reset; without this free_idx never advances.
@@ -458,7 +647,7 @@ static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
}
mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_ERROR);
mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_XFER_DONE);
- if (READ_ONCE(txq->tx_started))
+ if (READ_ONCE(txq->tx_started) && !txq->is_stopping)
mtk_cldma_start_queue(drv_info, DIR_TX, i);
spin_unlock(&txq->ring_lock);
}
@@ -512,12 +701,32 @@ static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
}
txq->tx_started = false;
+ /* Mask before waiting on anything: a QUEUE_ERROR latched after the
+ * flushes below would outlive this queue and be acted on for the
+ * next occupant of the same queue number.
+ */
+ mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
+ mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR);
irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
synchronize_irq(irq_id);
- /* flush on-going work */
+ /* flush on-going work, err_work first: it may have loaded this txq
+ * before it was unpublished above and still be stopping and
+ * flushing it.
+ */
+ flush_work(&drv_info->err_work);
flush_work(&txq->tx_done_work);
+ /* workers retired by the flushes unmask XFER_DONE on their way out,
+ * and err_work re-arms QUEUE_ERROR for a queue it handled
+ */
mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR);
+ /* Drop an error latched before that mask, and the err_work it
+ * queued: the slot is unpublished, so either could only act on the
+ * next occupant of this number.
+ */
+ synchronize_irq(irq_id);
+ atomic_andnot(BIT(txqno), &drv_info->tx_err_qs);
+ flush_work(&drv_info->err_work);
if (ret == -ETIMEDOUT) {
/* Still a live DMA target: leak the ring rather than hand it
@@ -525,6 +734,7 @@ static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
*/
dev_err(mdev->dev, "TX queue %u cannot be stopped, leaking its ring\n",
txqno);
+ drv_info->ring_leaked = true;
return;
}
@@ -696,17 +906,35 @@ static void mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
mutex_unlock(&drv_info->rearm_lock);
}
+ /* Mask before waiting on anything: a QUEUE_ERROR latched after the
+ * flushes below would outlive this queue and restart the next
+ * occupant of the same queue number.
+ */
+ mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
+ mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR);
irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
synchronize_irq(irq_id);
+ /* flush on-going work, err_work first: it ends by queueing
+ * rx_done_work, so flushing the per-queue worker first would leave
+ * it free to queue it again afterwards.
+ */
+ flush_work(&drv_info->err_work);
/* not flush_work(): the worker re-queues itself on a refill failure */
cancel_work_sync(&rxq->rx_done_work);
/* a cancelled pass would have cleared it on its way out */
mtk_cldma_clear_ip_busy(drv_info);
- /* only now: a worker retired by the cancel unmasks XFER_DONE on its
- * way out
+ /* a worker retired by the cancel unmasks XFER_DONE on its way out,
+ * and a restart re-arms QUEUE_ERROR
*/
mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR);
+ /* Drop an error latched before that mask, and the err_work it
+ * queued: the slot is unpublished, so either could only act on the
+ * next occupant of this number.
+ */
+ synchronize_irq(irq_id);
+ atomic_andnot(BIT(rxqno), &drv_info->rx_err_qs);
+ flush_work(&drv_info->err_work);
/* The worker can resume or restart the queue between its need_exit
* check and the MMIO write that follows it, so cancel_work_sync() does
@@ -721,6 +949,7 @@ static void mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
*/
dev_err(mdev->dev, "RX queue %u cannot be stopped, leaking its ring\n",
rxqno);
+ drv_info->ring_leaked = true;
return;
}
@@ -771,6 +1000,105 @@ static int mtk_cldma_hw_recovery(struct cldma_drv_info *drv_info, struct txq *tx
return 0;
}
+/* Silence the IP's interrupt output as well as the vector: a handler already
+ * past the mask unmasks it again on its way out. Repeated calls are no-ops,
+ * so the teardown path can run it again without unregistering twice.
+ */
+static void mtk_cldma_drv_quiesce(struct cldma_drv_info *drv_info)
+{
+ struct mtk_md_dev *mdev = drv_info->mdev;
+
+ mutex_lock(&drv_info->rearm_lock);
+ if (drv_info->quiesced) {
+ mutex_unlock(&drv_info->rearm_lock);
+ return;
+ }
+ /* also read by the RX worker: no refill retry outlives the drain */
+ WRITE_ONCE(drv_info->quiesced, true);
+ mutex_unlock(&drv_info->rearm_lock);
+
+ mtk_pci_write32(mdev, drv_info->base_addr + drv_info->hw_regs->reg_cldma_int_mask,
+ LINK_ERROR_VAL);
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+ synchronize_irq(mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id));
+ mtk_pci_unregister_irq(mdev, drv_info->pci_ext_irq_id);
+ /* not flush: err_work queues rx_done_work */
+ drain_workqueue(drv_info->wq);
+}
+
+/* Must be called before freeing state the workers dereference: joining the
+ * service kthreads leaves the interrupt and the workqueue running.
+ */
+void mtk_cldma_quiesce(struct mtk_ctrl_trans *trans)
+{
+ struct cldma_drv_info *drv_info;
+ struct cldma_dev *cd = trans->dev;
+ int i;
+
+ if (!cd)
+ return;
+
+ for (i = 0; i < NR_CLDMA; i++) {
+ drv_info = READ_ONCE(cd->cldma_drv_info[i]);
+ if (drv_info)
+ mtk_cldma_drv_quiesce(drv_info);
+ }
+}
+
+static int mtk_cldma_dev_exit(struct cldma_dev *cd, int hif_id)
+{
+ struct cldma_drv_info *drv_info;
+ struct mtk_md_dev *mdev;
+ int i;
+
+ if (!cd || hif_id >= NR_CLDMA)
+ return -EINVAL;
+
+ if (!cd->cldma_drv_info[hif_id])
+ return 0;
+
+ /* free cldma descriptor */
+ drv_info = cd->cldma_drv_info[hif_id];
+ mdev = cd->trans->mdev;
+
+ /* Unpublish before teardown; pairs with the acquire loads of this
+ * slot. No release is needed, there is no prior store to expose.
+ */
+ WRITE_ONCE(cd->cldma_drv_info[hif_id], NULL);
+
+ mtk_cldma_drv_quiesce(drv_info);
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ if (drv_info->txq[i])
+ mtk_cldma_txq_free(drv_info, drv_info->txq[i]->txqno);
+ if (drv_info->rxq[i])
+ mtk_cldma_rxq_free(drv_info, drv_info->rxq[i]->rxqno);
+ }
+
+ flush_workqueue(drv_info->wq);
+ destroy_workqueue(drv_info->wq);
+
+ /* Best-effort: reset the IP so it drops its queue state before the
+ * descriptor memory goes away. Not guaranteed to stop the engine,
+ * hence ring_leaked.
+ */
+ mtk_cldma_drv_reset(drv_info);
+ /* A handler retired by the quiesce may have unmasked the vector on
+ * its way out, and nothing handles it any more: leave it masked.
+ */
+ mtk_pci_mask_irq(mdev, drv_info->pci_ext_irq_id);
+
+ if (drv_info->ring_leaked)
+ dev_err(mdev->dev, "CLDMA%d rings leaked\n", drv_info->hw_id);
+ /* A pool with blocks outstanding keeps its pages, so a leaked ring
+ * stays allocated; only the pool bookkeeping is freed.
+ */
+ dma_pool_destroy(drv_info->gpd_dma_pool);
+
+ kfree(drv_info);
+
+ return 0;
+}
+
/* After a re-arm the slot at free_idx may hold a descriptor the device
* completed but tx_done_work has not reaped yet; a START there parks the
* engine on HWO=0. Start at the first slot the device still owns.
@@ -820,6 +1148,12 @@ static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno)
* names; tx_done_work advances free_idx under the same lock.
*/
spin_lock(&txq->ring_lock);
+ if (unlikely(txq->is_stopping)) {
+ spin_unlock(&txq->ring_lock);
+ ret = -EPIPE;
+ goto out;
+ }
+
if (unlikely(!READ_ONCE(txq->tx_started))) {
mtk_cldma_setup_start_addr(drv_info, DIR_TX, qno,
txq->req_pool[mtk_cldma_tx_start_idx(txq)].gpd_dma_addr);
@@ -850,11 +1184,22 @@ int mtk_cldma_init(struct mtk_ctrl_trans *trans)
void mtk_cldma_exit(struct mtk_ctrl_trans *trans)
{
- if (!trans->dev)
+ struct cldma_dev *cd = trans->dev;
+ int i;
+
+ if (!cd)
return;
- kfree(trans->dev);
+ /* Clear up front so any later submit bails out instead of walking
+ * freed queues. Callers hold trans->submit_lock, except the init
+ * unwind, where trans->available is still false.
+ */
trans->dev = NULL;
+
+ for (i = 0; i < NR_CLDMA; i++)
+ mtk_cldma_dev_exit(cd, i);
+
+ kfree(cd);
}
static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
@@ -873,7 +1218,8 @@ static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
trb->trb_complete(skb);
return -EINVAL;
}
- drv_info = cd->cldma_drv_info[que->hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[que->hif_id]);
if (!drv_info) {
ret = -EIO;
goto out;
@@ -939,6 +1285,11 @@ static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info,
struct trb *trb;
int i;
+ /* One load for the whole walk: the slot is written only by service
+ * init and teardown, and teardown cannot run while this flush does.
+ */
+ srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
+
for (i = 0; i < txq->nr_gpds; i++) {
spin_lock(&txq->ring_lock);
@@ -967,15 +1318,94 @@ static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info,
trb->status = err;
trb->trb_complete(skb);
- /* The service may already have been torn down; completing the
- * TRB is still correct, there is just nobody left to wake.
+ /* The service may never have been started; completing the TRB
+ * is still correct, there is just nobody left to wake.
*/
- srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
if (was_starved && srv)
wake_up(&srv->trb_waitq);
}
}
+static void mtk_cldma_err_tx_one(struct cldma_drv_info *drv_info, u32 qno)
+{
+ struct txq *txq;
+ int ret;
+
+ /* pairs with smp_store_release() in txq_alloc */
+ txq = smp_load_acquire(&drv_info->txq[qno]);
+ if (!txq)
+ return;
+
+ /* Keep the producer out so start_xfer cannot ring the doorbell
+ * between the stop and the flush.
+ */
+ spin_lock(&txq->ring_lock);
+ txq->is_stopping = true;
+ spin_unlock(&txq->ring_lock);
+
+ ret = mtk_cldma_stop_queue(drv_info, DIR_TX, qno);
+ /* -ENODEV: the link is dead, nothing is walking the ring */
+ if (ret && ret != -ENODEV) {
+ /* the device may still be walking the ring: unmapping its
+ * buffers here would leave it writing into unmapped memory.
+ * is_stopping is left set so nothing submits to it again,
+ * and QUEUE_ERROR masked until an instance re-arm.
+ */
+ dev_err(drv_info->mdev->dev,
+ "TX queue %u stop failed (%d), keeping its requests\n",
+ qno, ret);
+ return;
+ }
+
+ spin_lock(&txq->ring_lock);
+ WRITE_ONCE(txq->tx_started, false);
+ spin_unlock(&txq->ring_lock);
+ mtk_cldma_txq_flush(drv_info, txq, -EPIPE);
+
+ spin_lock(&txq->ring_lock);
+ txq->is_stopping = false;
+ spin_unlock(&txq->ring_lock);
+ /* handled: re-arm what the ISR left masked */
+ mtk_cldma_unmask_intr(drv_info, DIR_TX, qno, QUEUE_ERROR);
+}
+
+/* Handed to the worker because it owns free_idx and is the only context
+ * allowed to program an RX queue; a stopped queue raises no further
+ * interrupt, so nothing else would re-arm it.
+ */
+static void mtk_cldma_err_rx_one(struct cldma_drv_info *drv_info, u32 qno)
+{
+ struct rxq *rxq;
+
+ /* pairs with smp_store_release() in rxq_alloc */
+ rxq = smp_load_acquire(&drv_info->rxq[qno]);
+ if (!rxq)
+ return;
+
+ atomic_set(&rxq->need_restart, 1);
+ queue_work(drv_info->wq, &rxq->rx_done_work);
+}
+
+/* The latched bits are consumed up front, so an early exit on one direction
+ * must not swallow the other's error.
+ */
+static void mtk_cldma_err_work(struct work_struct *work)
+{
+ struct cldma_drv_info *drv_info = container_of(work, struct cldma_drv_info, err_work);
+ u32 tx_err, rx_err;
+ int i;
+
+ tx_err = atomic_xchg(&drv_info->tx_err_qs, 0);
+ rx_err = atomic_xchg(&drv_info->rx_err_qs, 0);
+
+ for (i = 0; i < HW_QUE_NUM; i++) {
+ if (tx_err & BIT(i))
+ mtk_cldma_err_tx_one(drv_info, i);
+ if (rx_err & BIT(i))
+ mtk_cldma_err_rx_one(drv_info, i);
+ }
+}
+
static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
{
struct trb *trb = (struct trb *)skb->cb;
@@ -988,7 +1418,8 @@ static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
if (unlikely(!que))
return -EPIPE;
- drv_info = cd->cldma_drv_info[que->hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[que->hif_id]);
if (unlikely(!drv_info))
return -EPIPE;
txq = drv_info->txq[que->txqno];
@@ -998,6 +1429,13 @@ static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
mdev = drv_info->mdev;
ret = mtk_cldma_start_xfer(drv_info, que->txqno);
+ /* The queue is being stopped and nothing was kicked: the requests in
+ * the ring are the ones recovery is keeping, so this must not fall
+ * into the flush below and unmap them.
+ */
+ if (unlikely(ret == -EPIPE))
+ return ret;
+
if (unlikely(ret)) {
dev_err(mdev->dev, "Failed to trigger cldma tx\n");
/* A failed start does not mean the engine is idle - recovery
@@ -1035,7 +1473,8 @@ static int mtk_cldma_close(struct cldma_dev *cd, struct sk_buff *skb)
trb->trb_complete(skb);
return -EPIPE;
}
- drv_info = cd->cldma_drv_info[que->hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[que->hif_id]);
if (unlikely(!drv_info)) {
trb->status = -EPIPE;
trb->trb_complete(skb);
@@ -1096,7 +1535,8 @@ int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb)
que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
if (unlikely(!que))
return -EINVAL;
- drv_info = cd->cldma_drv_info[que->hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[que->hif_id]);
if (unlikely(!drv_info))
return -EINVAL;
@@ -1106,6 +1546,14 @@ int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb)
spin_lock(&txq->ring_lock);
+ /* Publishing another HWO descriptor into a ring the device may still
+ * be walking is exactly what the stop is trying to prevent.
+ */
+ if (unlikely(txq->is_stopping)) {
+ spin_unlock(&txq->ring_lock);
+ return -EPIPE;
+ }
+
if (!atomic_read(&txq->req_budget)) {
spin_unlock(&txq->ring_lock);
return -EAGAIN;
@@ -1145,7 +1593,8 @@ int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno)
if (unlikely(hif_id >= NR_CLDMA || qno >= HW_QUE_NUM || !cd))
return -EINVAL;
- drv_info = cd->cldma_drv_info[hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[hif_id]);
if (!drv_info)
return -EINVAL;
txq = drv_info->txq[qno];
@@ -1177,6 +1626,38 @@ int mtk_cldma_trb_process(void *dev, struct sk_buff *skb)
return trb_act_tbl[trb->cmd](cd, skb);
}
+void mtk_cldma_fsm_state_listener(struct mtk_fsm_param *param, struct mtk_ctrl_trans *trans)
+{
+ struct cldma_dev *cd = trans->dev;
+ enum mtk_fsm_hif_user user;
+ enum mtk_hif_id hif_id;
+ int ret;
+
+ switch (param->to) {
+ case FSM_STATE_BOOTUP:
+ if (param->fsm_flag & FSM_F_SAP_HS_START) {
+ hif_id = CLDMA0;
+ user = MTK_FSM_HIF_USER_CLDMA0;
+ } else if (param->fsm_flag & FSM_F_MD_HS_START) {
+ hif_id = CLDMA1;
+ user = MTK_FSM_HIF_USER_CLDMA1;
+ } else {
+ break;
+ }
+
+ ret = mtk_cldma_dev_init(cd, hif_id);
+ if (ret)
+ dev_err(trans->mdev->dev, "Failed to init CLDMA%d: %d\n", hif_id, ret);
+ /* Record the success too: HS1 is retried, so a later good init
+ * has to clear what the failed attempt left behind.
+ */
+ mtk_fsm_hif_err_record(trans->mdev, user, ret);
+ break;
+ default:
+ break;
+ }
+}
+
int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
{
struct cldma_drv_info *drv_info;
@@ -1186,7 +1667,8 @@ int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
struct rxq *rxq;
mdev = cd->trans->mdev;
- drv_info = cd->cldma_drv_info[que->hif_id];
+ /* pairs with smp_store_release() in mtk_cldma_dev_init */
+ drv_info = smp_load_acquire(&cd->cldma_drv_info[que->hif_id]);
if (!drv_info) {
dev_err(mdev->dev, "CLDMA%d has not been initialized\n",
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
index 8abde07ebd55..f6d26b80589f 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.h
@@ -103,6 +103,12 @@ struct txq {
u32 wr_idx;
u32 free_idx;
bool tx_started;
+ /* Set by err_work while it stops this queue, and left set if the
+ * stop failed: the device may still be walking the ring, so new
+ * requests are refused with -EPIPE and the ones already queued are
+ * kept. Read under ring_lock by the producer.
+ */
+ bool is_stopping;
};
struct cldma_dev {
@@ -111,6 +117,7 @@ struct cldma_dev {
};
int mtk_cldma_init(struct mtk_ctrl_trans *trans);
+void mtk_cldma_quiesce(struct mtk_ctrl_trans *trans);
void mtk_cldma_exit(struct mtk_ctrl_trans *trans);
int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb);
int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
index f7b5018d3a73..b32301b0240a 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.h
@@ -78,8 +78,17 @@ struct cldma_drv_info {
struct dma_pool *gpd_dma_pool;
struct workqueue_struct *wq;
const struct cldma_hw_regs *hw_regs;
+ struct work_struct err_work;
+ atomic_t tx_err_qs;
+ atomic_t rx_err_qs;
+ /* a ring could not be stopped and was leaked on free */
+ bool ring_leaked;
+ /* this instance's interrupt and workers have already been retired;
+ * makes the quiesce safe to repeat on the teardown path
+ */
+ bool quiesced;
/* serialises the instance reset and re-arm against queue kicks and
- * RX resume/restart: no queue is kicked while it has no address
+ * RX resume/restart, and re-arming against setting quiesced
*/
struct mutex rearm_lock;
};
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
index 68e39ea4f69d..8a5b4a143ed2 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_pci.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
@@ -12,6 +12,7 @@
#include <linux/module.h>
#include "mtk_dev.h"
+#include "mtk_fsm.h"
#include "mtk_trans_ctrl.h"
#include "mtk_pci.h"
#include "mtk_pci_reg.h"
@@ -858,18 +859,41 @@ static int mtk_pci_dev_init(struct mtk_md_dev *mdev)
{
int ret;
- ret = mtk_trans_ctrl_init(mdev);
+ ret = mtk_fsm_init(mdev);
if (ret) {
- dev_err(mdev->dev, "Failed to initialize control plane: %d\n", ret);
+ dev_err(mdev->dev, "Failed to initialize FSM: %d\n", ret);
return ret;
}
+ ret = mtk_trans_ctrl_init(mdev);
+ if (ret)
+ goto free_fsm;
+
return 0;
+free_fsm:
+ mtk_fsm_exit(mdev);
+ return ret;
}
static void mtk_pci_dev_exit(struct mtk_md_dev *mdev)
{
+ int ret;
+
+ ret = mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_RM, 0, NULL,
+ EVT_MODE_BLOCKING | EVT_MODE_TOHEAD);
+ if (ret < 0)
+ dev_err(mdev->dev, "FSM DEV_RM failed: %d, forcing cleanup\n", ret);
+ /* Close the event gate and park the FSM thread before the transport
+ * plane it drives is torn down.
+ */
+ mtk_fsm_stop(mdev);
mtk_trans_ctrl_exit(mdev);
+ mtk_fsm_exit(mdev);
+}
+
+static int mtk_pci_dev_start(struct mtk_md_dev *mdev)
+{
+ return mtk_fsm_evt_submit(mdev, FSM_EVT_DEV_ADD, 0, NULL, 0);
}
static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
@@ -960,8 +984,17 @@ static int mtk_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
pci_set_master(pdev);
mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
+ ret = mtk_pci_dev_start(mdev);
+ if (ret) {
+ dev_err(mdev->dev, "Failed to start dev.\n");
+ goto clear_master;
+ }
+
return 0;
+clear_master:
+ pci_clear_master(pdev);
+ pci_load_and_free_saved_state(pdev, &priv->saved_state);
dev_exit:
mtk_pci_dev_exit(mdev);
free_irq:
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
index 8b61c3eac035..d87cb61d5326 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -480,19 +480,27 @@ int mtk_pcie_hif_exit(struct mtk_md_dev *mdev)
trans = ctrl_blk->ctrl_hw_priv;
+ /* Hold submit_lock across the whole teardown: late submitters either
+ * see available and finish before the teardown starts, or block here
+ * and then bail out on !available. Also makes this exit idempotent,
+ * so both the FSM listener and device removal may call it.
+ */
mutex_lock(&trans->submit_lock);
+ if (!trans->available) {
+ mutex_unlock(&trans->submit_lock);
+ return 0;
+ }
trans->available = false;
- mutex_unlock(&trans->submit_lock);
-
+ /* The CLDMA interrupt and its workers read trans->trb_srv[] too, so
+ * joining the service threads below does not cover them.
+ */
+ mtk_cldma_quiesce(trans);
/* Join the service threads before freeing what they dereference: they
* read trans->dev without a lock, so clearing it cannot stop a
* consumer that has already loaded the pointer.
*/
mtk_ctrl_trb_srv_exit(trans);
mtk_cldma_exit(trans);
-
- /* Late submitters may still hold the lock and walk the tree. */
- mutex_lock(&trans->submit_lock);
mtk_ctrl_remove_queue_tbl(trans);
mutex_unlock(&trans->submit_lock);
@@ -573,6 +581,15 @@ int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool f
return ret;
}
+void mtk_pcie_hif_fsm_indication(struct mtk_md_dev *mdev, struct mtk_fsm_param *param)
+{
+ struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
+ struct mtk_ctrl_trans *trans;
+
+ trans = ctrl_blk->ctrl_hw_priv;
+ mtk_cldma_fsm_state_listener(param, trans);
+}
+
int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data)
{
struct mtk_ctrl_blk *ctrl_blk = mdev->ctrl_blk;
@@ -635,6 +652,13 @@ int mtk_trans_ctrl_init(struct mtk_md_dev *mdev)
int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev)
{
+ /* FSM_STATE_OFF normally tears the HIF down. If that never ran, the
+ * trb kthreads and the CLDMA irq callback would outlive the devm
+ * allocations they point at, so do it here. mtk_pcie_hif_exit() is
+ * idempotent under trans->submit_lock, so calling it again is safe.
+ */
+ mtk_pcie_hif_exit(mdev);
+
mtk_ctrl_exit(mdev);
return 0;
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
index ee036f01a875..960ccc9fc706 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.h
@@ -75,9 +75,12 @@ struct trb_srv {
struct task_struct *trb_thread;
};
+struct mtk_fsm_param;
+
int mtk_pcie_hif_init(struct mtk_md_dev *mdev);
int mtk_pcie_hif_exit(struct mtk_md_dev *mdev);
int mtk_pcie_hif_submit_skb(struct mtk_md_dev *mdev, struct sk_buff *skb, bool force_send);
+void mtk_pcie_hif_fsm_indication(struct mtk_md_dev *mdev, struct mtk_fsm_param *param);
int mtk_pcie_hif_cmd_func(struct mtk_md_dev *mdev, int cmd, void *data);
int mtk_trans_ctrl_init(struct mtk_md_dev *mdev);
int mtk_trans_ctrl_exit(struct mtk_md_dev *mdev);
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
` (4 preceding siblings ...)
2026-11-05 7:56 ` [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread Jack Wu via B4 Relay
@ 2026-11-05 7:56 ` Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
5 siblings, 1 reply; 12+ messages in thread
From: Jack Wu via B4 Relay @ 2026-11-05 7:56 UTC (permalink / raw)
To: Loic Poulain, Sergey Ryazanov, Johannes Berg, Andrew Lunn,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Jack Wu, Wen-Zhi Huang, Shi-Wei Yeh, Minano Tseng,
Matthias Brugger, AngeloGioacchino Del Regno, Simon Horman,
Jonathan Corbet, Shuah Khan, Robert Yu, Jeff Chang
Cc: linux-kernel, netdev, linux-arm-kernel, linux-mediatek, linux-doc
From: Jack Wu <jackbb_wu@compal.com>
Add AT & MBIM ports to the port infrastructure.
mtk_port_wwan_init() only prepares the port object; wwan_create_port()
is called from mtk_port_wwan_enable(). enable() runs on FSM_STATE_READY
through the new mtk_port_enable_by_type(PORT_TBL_MD) hook, which enables
every MD-table port the device marked enabled, internal ports included.
CCCI_UART2 and CCCI_MBIM get CLDMA1 TXQ(5)/RXQ(5) and TXQ(2)/RXQ(2) in
mtk_queue_info[].
The implemented WWAN port operations are start, stop, tx and
tx_blocking. The tx path copies with skb_copy_bits(), handling the
non-linear skb the WWAN core supplies for writes larger than one
fragment. One write() is capped at MTK_PORT_MAX_TX_PKT CCCI packets
(SKB_LIST_MAX_LEN, the HIF queue depth) and anything larger is rejected
with -EMSGSIZE: every packet after the first is submitted with
force_send, which bypasses the depth check, so without a cap a single
write could queue an unbounded number of skbs. The same cap bounds the
second copy of the message the tx path holds and the number of times
skb_copy_bits() re-walks the source fragment chain.
The first packet of every write() is depth checked. A blocking write
waits on the port's trb wait queue for the HIF queue to drop below its
depth before it submits anything, rather than bypassing the check, so
blocking writers whose waits are cut short by signals cannot grow the
queue either.
start() publishes the wwan_port under w_priv.w_lock together with
PORT_S_OPEN, so the receive path can never see an open port with a NULL
wwan_port and drop the reply to a command that open made possible.
TX back-pressure is reported through wwan_port_txon()/txoff() rather
than a .tx_poll operation: a queue-full submit failure pauses the port
and a per-port tx_complete hook resumes it from TRB completion, so
poll() sleeps on the WWAN core's own waitqueue - whose lifetime is
pinned by the open file - instead of a driver waitqueue that
wwan_remove_port() can outlive. The pause path drops w_lock before
querying the HIF for queue occupancy, because HIF teardown holds
trans->submit_lock while the TRB completions it flushes call back into
the port under w_lock.
Signed-off-by: Jack Wu <jackbb_wu@compal.com>
---
drivers/net/wwan/t9xx/mtk_ctrl_plane.h | 4 +
drivers/net/wwan/t9xx/mtk_port.c | 31 +++
drivers/net/wwan/t9xx/mtk_port.h | 14 +
drivers/net/wwan/t9xx/mtk_port_io.c | 413 ++++++++++++++++++++++++++++
drivers/net/wwan/t9xx/mtk_port_io.h | 1 +
drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c | 8 +
6 files changed, 471 insertions(+)
diff --git a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
index 617c022a24c3..295c3c532c93 100644
--- a/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
+++ b/drivers/net/wwan/t9xx/mtk_ctrl_plane.h
@@ -21,6 +21,10 @@ enum mtk_ccci_ch {
/* to MD */
CCCI_CONTROL_RX = 0x2000,
CCCI_CONTROL_TX = 0x2001,
+ CCCI_UART2_RX = 0x200A,
+ CCCI_UART2_TX = 0x200C,
+ CCCI_MBIM_RX = 0x20D0,
+ CCCI_MBIM_TX = 0x20D1,
};
enum mtk_trb_cmd_type {
diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
index 101ec14fc993..a0defeec9abf 100644
--- a/drivers/net/wwan/t9xx/mtk_port.c
+++ b/drivers/net/wwan/t9xx/mtk_port.c
@@ -285,6 +285,13 @@ static int mtk_port_tx_complete(struct sk_buff *skb)
"Failed to send data: status:%d, port:%s\n",
trb->status, port->info.name);
+ /* The hook may sleep, but it also runs from HIF teardown with
+ * trans->submit_lock held, so it must not take a lock that is ever
+ * held across a call into the HIF.
+ */
+ if (mtk_port_get_ops(port->info.type)->tx_complete)
+ mtk_port_get_ops(port->info.type)->tx_complete(port);
+
/* Pairs with the smp_load_acquire() of trb->done in the waiters. */
smp_store_release(&trb->done, true);
wake_up_all(&port->trb_wq);
@@ -626,6 +633,27 @@ int mtk_port_ch_disable(struct mtk_port *port)
return ret;
}
+static void mtk_port_enable_by_type(struct mtk_port_mngr *port_mngr, int tbl_type)
+{
+ struct radix_tree_iter iter;
+ struct mtk_port *port;
+ void __rcu **slot;
+
+ radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type],
+ &iter, 0) {
+ port = mtk_port_tbl_slot(&slot, &iter);
+ if (!port)
+ continue;
+ /* Consumed, so a later boot acts only on what its own port
+ * enumeration advertised.
+ */
+ if (port->enable) {
+ port->enable = false;
+ mtk_port_get_ops(port->info.type)->enable(port);
+ }
+ }
+}
+
static void mtk_port_disable(struct mtk_port_mngr *port_mngr)
{
struct radix_tree_iter iter;
@@ -658,6 +686,9 @@ void mtk_port_mngr_fsm_state_handler(struct mtk_fsm_param *fsm_param, void *arg)
case FSM_STATE_OFF:
mtk_port_disable(port_mngr);
break;
+ case FSM_STATE_READY:
+ mtk_port_enable_by_type(port_mngr, PORT_TBL_MD);
+ break;
default:
break;
}
diff --git a/drivers/net/wwan/t9xx/mtk_port.h b/drivers/net/wwan/t9xx/mtk_port.h
index d160f73c1834..0951c38359e1 100644
--- a/drivers/net/wwan/t9xx/mtk_port.h
+++ b/drivers/net/wwan/t9xx/mtk_port.h
@@ -11,6 +11,7 @@
#include <linux/radix-tree.h>
#include <linux/skbuff.h>
#include <linux/types.h>
+#include <linux/wwan.h>
#include "mtk_ctrl_plane.h"
#include "mtk_dev.h"
@@ -58,6 +59,7 @@ enum mtk_port_tbl {
enum mtk_port_type {
PORT_TYPE_INTERNAL,
+ PORT_TYPE_WWAN,
PORT_TYPE_MAX
};
@@ -66,6 +68,17 @@ struct mtk_internal_port {
int (*recv_cb)(void *arg, struct sk_buff *skb);
};
+struct mtk_wwan_port {
+ /* w_lock serialises w_port and PORT_S_OPEN against the receive path,
+ * and txoff/txon against TX completion. It is a leaf: TX completion
+ * can take it with trans->submit_lock held, so it is never held
+ * across a call into the HIF.
+ */
+ struct mutex w_lock;
+ enum wwan_port_type w_type;
+ struct wwan_port *w_port;
+};
+
struct mtk_port_cfg {
enum mtk_ccci_ch tx_ch;
enum mtk_ccci_ch rx_ch;
@@ -91,6 +104,7 @@ struct mtk_port {
wait_queue_head_t trb_wq;
struct mtk_port_mngr *port_mngr;
struct mtk_internal_port i_priv;
+ struct mtk_wwan_port w_priv;
};
struct mtk_port_mngr {
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mtk_port_io.c
index e92ce1cb7c00..d5232cb150bf 100644
--- a/drivers/net/wwan/t9xx/mtk_port_io.c
+++ b/drivers/net/wwan/t9xx/mtk_port_io.c
@@ -3,8 +3,13 @@
* Copyright (c) 2022, MediaTek Inc.
*/
#include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+#include <linux/wwan.h>
#include "mtk_port_io.h"
+#include "mtk_trans_ctrl.h"
static void mtk_port_put_locked(struct mtk_port *port)
{
@@ -20,6 +25,122 @@ static void mtk_port_struct_init(struct mtk_port *port)
init_waitqueue_head(&port->trb_wq);
}
+/* Packets after the first carry force_send, which bypasses the HIF depth
+ * check, so one write() must not exceed one queue depth.
+ */
+#define MTK_PORT_MAX_TX_PKT SKB_LIST_MAX_LEN
+
+/* A blocking write waits here rather than have its first packet refused by
+ * the depth check: a refused packet is freed with its trb and cannot be
+ * submitted again. TX completions and port state changes wake trb_wq.
+ */
+static int mtk_port_wait_tx_space(struct mtk_port *port)
+{
+ union ctrl_hif_cmd_data hif_cmd = { .rx_ch = port->info.rx_ch };
+ struct mtk_md_dev *mdev = port->port_mngr->ctrl_blk->mdev;
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
+ int ret = 0;
+
+ add_wait_queue(&port->trb_wq, &wait);
+ /* The check takes submit_lock, so it cannot be a wait_event()
+ * condition. An error from it is left to the submit to report.
+ */
+ while (mtk_pcie_hif_cmd_func(mdev, HIF_CTRL_CMD_CHECK_TX_FULL, &hif_cmd) > 0 &&
+ !test_bit(PORT_S_FLUSH, &port->status) &&
+ test_bit(PORT_S_WR, &port->status)) {
+ if (signal_pending(current)) {
+ ret = -ERESTARTSYS;
+ break;
+ }
+ wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
+ }
+ remove_wait_queue(&port->trb_wq, &wait);
+
+ return ret;
+}
+
+/* Every packet is built before any is submitted: the WWAN core turns a
+ * partial write into a whole-write failure and userspace re-sends the whole
+ * message, so a submit that fails midway leaves a prefix on the wire. It
+ * cannot be retried here - the skb is freed with its trb - so it is logged.
+ *
+ * -EINTR means the skb was already submitted, so that packet counts as
+ * accepted and the remaining ones are still sent.
+ */
+static int mtk_port_common_write(struct mtk_port *port, struct sk_buff *src, bool blocking)
+{
+ u32 packet_size, left_cnt = src->len, cur_pos;
+ u32 payload_mtu, sent_len = 0;
+ struct sk_buff_head list;
+ bool force_send = false;
+ struct sk_buff *skb;
+ int ret;
+
+ ret = mtk_port_status_check(port);
+ if (ret)
+ return ret;
+
+ payload_mtu = port->tx_mtu - sizeof(struct mtk_ccci_header);
+ if (src->len > MTK_PORT_MAX_TX_PKT * payload_mtu)
+ return -EMSGSIZE;
+
+ if (blocking) {
+ ret = mtk_port_wait_tx_space(port);
+ if (ret)
+ return ret;
+ }
+
+ __skb_queue_head_init(&list);
+
+ while (left_cnt) {
+ skb = __dev_alloc_skb(port->tx_mtu, GFP_KERNEL);
+ if (!skb) {
+ ret = -ENOMEM;
+ goto err_purge;
+ }
+
+ skb_reserve(skb, sizeof(struct mtk_ccci_header));
+
+ packet_size = min_t(u32, left_cnt, payload_mtu);
+ cur_pos = src->len - left_cnt;
+ ret = skb_copy_bits(src, cur_pos, skb_put(skb, packet_size), packet_size);
+ if (ret) {
+ dev_err(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to copy data for port(%s)\n", port->info.name);
+ dev_kfree_skb_any(skb);
+ goto err_purge;
+ }
+
+ __skb_queue_tail(&list, skb);
+ left_cnt -= packet_size;
+ }
+
+ while ((skb = __skb_dequeue(&list))) {
+ /* mtk_port_send_data() may free the skb, so sample its length
+ * while it is still valid.
+ */
+ packet_size = skb->len;
+
+ ret = mtk_port_send_data(port, skb, blocking, force_send);
+ if (ret < 0 && ret != -EINTR)
+ goto err_submit;
+
+ sent_len += packet_size;
+ force_send = true;
+ }
+
+ return 0;
+
+err_submit:
+ if (sent_len)
+ dev_err_ratelimited(port->port_mngr->ctrl_blk->mdev->dev,
+ "Partial write on port(%s): %u of %u bytes sent, error:%d\n",
+ port->info.name, sent_len, src->len, ret);
+err_purge:
+ __skb_queue_purge(&list);
+ return ret;
+}
+
static int mtk_port_internal_init(struct mtk_port *port)
{
mtk_port_struct_init(port);
@@ -219,8 +340,300 @@ static const struct mtk_port_ops port_internal_ops = {
.recv = mtk_port_internal_recv,
};
+static void mtk_port_get_locked(struct mtk_port *port)
+{
+ mutex_lock(&mtk_port_mngr_grp_mtx);
+ kref_get(&port->kref);
+ mutex_unlock(&mtk_port_mngr_grp_mtx);
+}
+
+static int mtk_port_wwan_open(struct wwan_port *w_port)
+{
+ struct mtk_port *port;
+ int ret;
+
+ port = wwan_port_get_drvdata(w_port);
+ mtk_port_get_locked(port);
+
+ /* mtk_port_common_open() is what sets PORT_S_OPEN, and the RX path
+ * tests it together with w_port, so both are published under w_lock.
+ */
+ mutex_lock(&port->w_priv.w_lock);
+ ret = mtk_port_common_open(port);
+ if (!ret)
+ port->w_priv.w_port = w_port;
+ mutex_unlock(&port->w_priv.w_lock);
+ if (ret) {
+ mtk_port_put_locked(port);
+ return ret;
+ }
+
+ /* WWAN_PORT_TX_OFF persists on the wwan_port across close/open
+ * cycles; start every session writable.
+ */
+ wwan_port_txon(w_port);
+
+ return 0;
+}
+
+static void mtk_port_wwan_close(struct wwan_port *w_port)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+
+ /* Clear PORT_S_OPEN under the same lock mtk_port_wwan_recv() holds, so
+ * a receive that saw the port open has finished its wwan_port_rx()
+ * before stop() returns and the core purges its rx queue.
+ */
+ mutex_lock(&port->w_priv.w_lock);
+ mtk_port_common_close(port);
+ mutex_unlock(&port->w_priv.w_lock);
+
+ mtk_port_put_locked(port);
+}
+
+/* w_lock closes the txoff-after-drain window described in
+ * mtk_port_wwan_tx_pause().
+ */
+static void mtk_port_wwan_tx_complete(struct mtk_port *port)
+{
+ mutex_lock(&port->w_priv.w_lock);
+ if (port->w_priv.w_port)
+ wwan_port_txon(port->w_priv.w_port);
+ mutex_unlock(&port->w_priv.w_lock);
+}
+
+/* Pause TX after a queue-full submit failure. The queue may have drained
+ * between that failure and the txoff below, and that tx_complete is lost, so
+ * recheck and undo the txoff if the queue is no longer full.
+ *
+ * w_lock is dropped for the recheck to keep it a leaf with respect to
+ * trans->submit_lock. A tx_complete landing in the gap can only add a txon,
+ * so TX is never left off with nothing to wake it.
+ */
+static void mtk_port_wwan_tx_pause(struct mtk_port *port)
+{
+ union ctrl_hif_cmd_data hif_cmd;
+ struct mtk_ctrl_blk *ctrl_blk;
+ int ret;
+
+ ctrl_blk = port->port_mngr->ctrl_blk;
+
+ mutex_lock(&port->w_priv.w_lock);
+ if (!port->w_priv.w_port) {
+ mutex_unlock(&port->w_priv.w_lock);
+ return;
+ }
+
+ wwan_port_txoff(port->w_priv.w_port);
+ mutex_unlock(&port->w_priv.w_lock);
+
+ hif_cmd.rx_ch = port->info.rx_ch;
+ ret = mtk_pcie_hif_cmd_func(ctrl_blk->mdev, HIF_CTRL_CMD_CHECK_TX_FULL,
+ &hif_cmd);
+ if (ret <= 0)
+ mtk_port_wwan_tx_complete(port);
+}
+
+static int mtk_port_wwan_tx(struct wwan_port *w_port, struct sk_buff *skb, bool blocking)
+{
+ struct mtk_port *port = wwan_port_get_drvdata(w_port);
+ int ret;
+
+ if (unlikely(!skb->len)) {
+ consume_skb(skb);
+ return 0;
+ }
+
+ ret = mtk_port_common_write(port, skb, blocking);
+ if (ret < 0) {
+ if (ret == -EAGAIN)
+ mtk_port_wwan_tx_pause(port);
+ return ret;
+ }
+
+ consume_skb(skb);
+ return 0;
+}
+
+static int mtk_port_wwan_write(struct wwan_port *w_port, struct sk_buff *skb)
+{
+ return mtk_port_wwan_tx(w_port, skb, false);
+}
+
+static int mtk_port_wwan_write_blocking(struct wwan_port *w_port, struct sk_buff *skb)
+{
+ return mtk_port_wwan_tx(w_port, skb, true);
+}
+
+/* No .tx_poll: it would poll_wait() on port->trb_wq, which lives in a
+ * mtk_port that wwan_remove_port() lets go of while the file stays open,
+ * and wake_up_pollfree() is not available to modules. TX back-pressure
+ * reaches poll() through wwan_port_txon()/txoff() on the core's own
+ * waitqueue instead, whose lifetime is pinned by the open file.
+ */
+static const struct wwan_port_ops wwan_ops = {
+ .start = mtk_port_wwan_open,
+ .stop = mtk_port_wwan_close,
+ .tx = mtk_port_wwan_write,
+ .tx_blocking = mtk_port_wwan_write_blocking,
+};
+
+static int mtk_port_wwan_init(struct mtk_port *port)
+{
+ mtk_port_struct_init(port);
+ port->enable = false;
+
+ mutex_init(&port->w_priv.w_lock);
+
+ switch (port->info.rx_ch) {
+ case CCCI_MBIM_RX:
+ port->w_priv.w_type = WWAN_PORT_MBIM;
+ break;
+ case CCCI_UART2_RX:
+ port->w_priv.w_type = WWAN_PORT_AT;
+ break;
+ default:
+ port->w_priv.w_type = WWAN_PORT_UNKNOWN;
+ break;
+ }
+
+ return 0;
+}
+
+static void mtk_port_wwan_exit(struct mtk_port *port)
+{
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ mtk_port_get_ops(port->info.type)->disable(port);
+ mutex_destroy(&port->w_priv.w_lock);
+}
+
+static void mtk_port_wwan_enable(struct mtk_port *port)
+{
+ struct mtk_port_mngr *port_mngr;
+ struct wwan_port_caps caps;
+ struct wwan_port *wp;
+ int ret;
+
+ port_mngr = port->port_mngr;
+
+ if (test_bit(PORT_S_ENABLE, &port->status))
+ return;
+
+ ret = mtk_port_ch_enable(port);
+ if (ret && ret != -EBUSY) {
+ /* On -ETIMEDOUT the enable's outcome is not yet known: the
+ * ENABLE trb may still be queued. The DISABLE is queued
+ * behind it, so the channel cannot stay armed unowned.
+ */
+ mtk_port_ch_disable(port);
+ return;
+ }
+
+ /* A zero frag_len would make wwan_port_fops_write() loop forever. */
+ if (port->tx_mtu <= sizeof(struct mtk_ccci_header)) {
+ dev_err(port_mngr->ctrl_blk->mdev->dev,
+ "Invalid tx_mtu for port(%s)\n", port->info.name);
+ mtk_port_ch_disable(port);
+ return;
+ }
+
+ /* The core allocates frag_len + headroom_len and skb_put()s frag_len,
+ * so frag_len is the payload budget: subtract the CCCI header to make
+ * one core fragment exactly one CCCI packet.
+ */
+ caps.frag_len = port->tx_mtu - sizeof(struct mtk_ccci_header);
+ caps.headroom_len = sizeof(struct mtk_ccci_header);
+
+ /* These bits must be set before wwan_create_port(): the device node
+ * becomes openable inside it and mtk_port_common_open() rejects a
+ * port without PORT_S_ENABLE.
+ */
+ set_bit(PORT_S_WR, &port->status);
+ set_bit(PORT_S_ENABLE, &port->status);
+
+ wp = wwan_create_port(port_mngr->ctrl_blk->mdev->dev,
+ port->w_priv.w_type,
+ &wwan_ops, &caps, port);
+ if (IS_ERR(wp)) {
+ dev_warn(port_mngr->ctrl_blk->mdev->dev,
+ "Failed to create wwan port for (%s)\n", port->info.name);
+ clear_bit(PORT_S_ENABLE, &port->status);
+ clear_bit(PORT_S_WR, &port->status);
+ mtk_port_ch_disable(port);
+ return;
+ }
+
+ /* .start publishes w_port for an opened node; this covers the node
+ * nobody opened, which still needs w_port to be removed.
+ */
+ mutex_lock(&port->w_priv.w_lock);
+ port->w_priv.w_port = wp;
+ mutex_unlock(&port->w_priv.w_lock);
+}
+
+static void mtk_port_wwan_disable(struct mtk_port *port)
+{
+ struct wwan_port *w_port;
+ int ret;
+
+ if (!test_and_clear_bit(PORT_S_ENABLE, &port->status))
+ return;
+
+ /* PORT_S_WR is in the blocking TX wait condition: wake the waiters. */
+ clear_bit(PORT_S_WR, &port->status);
+ wake_up_all(&port->trb_wq);
+
+ /* w_lock must be dropped before wwan_remove_port(): that takes the
+ * core's ops_lock and forces stop(), which is close() taking w_lock.
+ */
+ mutex_lock(&port->w_priv.w_lock);
+ w_port = port->w_priv.w_port;
+ port->w_priv.w_port = NULL;
+ mutex_unlock(&port->w_priv.w_lock);
+
+ ret = mtk_port_ch_disable(port);
+ if (ret)
+ dev_warn(port->port_mngr->ctrl_blk->mdev->dev,
+ "Failed to disable channel for port(%s): %d\n",
+ port->info.name, ret);
+
+ wwan_remove_port(w_port);
+}
+
+static int mtk_port_wwan_recv(struct mtk_port *port, struct sk_buff *skb)
+{
+ /* Drop frames when nobody has the device open: wwan_port_rx() queues
+ * without bound and only a reader drains the queue, so accepting
+ * unsolicited traffic here would grow the rxq indefinitely. Both
+ * conditions are read under w_lock, which mtk_port_wwan_close() also
+ * takes, so a frame accepted here cannot land on a queue the core is
+ * about to purge.
+ */
+ mutex_lock(&port->w_priv.w_lock);
+ if (!test_bit(PORT_S_OPEN, &port->status) || !port->w_priv.w_port) {
+ mutex_unlock(&port->w_priv.w_lock);
+ dev_dbg_ratelimited(port->port_mngr->ctrl_blk->mdev->dev,
+ "Drop RX for unopened port(%s)\n", port->info.name);
+ return -ENXIO;
+ }
+
+ wwan_port_rx(port->w_priv.w_port, skb);
+ mutex_unlock(&port->w_priv.w_lock);
+ return 0;
+}
+
+static const struct mtk_port_ops port_wwan_ops = {
+ .init = mtk_port_wwan_init,
+ .exit = mtk_port_wwan_exit,
+ .enable = mtk_port_wwan_enable,
+ .disable = mtk_port_wwan_disable,
+ .recv = mtk_port_wwan_recv,
+ .tx_complete = mtk_port_wwan_tx_complete,
+};
+
static const struct mtk_port_ops *mtk_port_ops_tbl[PORT_TYPE_MAX] = {
&port_internal_ops,
+ &port_wwan_ops,
};
const struct mtk_port_ops *mtk_port_get_ops(enum mtk_port_type type)
diff --git a/drivers/net/wwan/t9xx/mtk_port_io.h b/drivers/net/wwan/t9xx/mtk_port_io.h
index db1d9ae52d9a..e1e695100ab1 100644
--- a/drivers/net/wwan/t9xx/mtk_port_io.h
+++ b/drivers/net/wwan/t9xx/mtk_port_io.h
@@ -18,6 +18,7 @@ struct mtk_port_ops {
void (*enable)(struct mtk_port *port);
void (*disable)(struct mtk_port *port);
int (*recv)(struct mtk_port *port, struct sk_buff *skb);
+ void (*tx_complete)(struct mtk_port *port);
};
const struct mtk_port_ops *mtk_port_get_ops(enum mtk_port_type type);
diff --git a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
index d87cb61d5326..7492a797b682 100644
--- a/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
+++ b/drivers/net/wwan/t9xx/pcie/mtk_trans_ctrl.c
@@ -26,6 +26,10 @@ static const int mtk_srv_cfg[NR_CLDMA][HW_QUE_NUM] = {
/* the number of RX GPDs should be at least two */
static const struct queue_info mtk_queue_info[] = {
+ {CCCI_UART2_TX, CCCI_UART2_RX, CLDMA1, TXQ(5), RXQ(5),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
+ {CCCI_MBIM_TX, CCCI_MBIM_RX, CLDMA1, TXQ(2), RXQ(2),
+ Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
{CCCI_CONTROL_TX, CCCI_CONTROL_RX, CLDMA1, TXQ(0), RXQ(0),
Q_MTU_3_5K, Q_MTU_3_5K, TX_GPD_NUM, RX_GPD_NUM, 0},
{CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, CLDMA0, TXQ(0), RXQ(0),
@@ -33,6 +37,10 @@ static const struct queue_info mtk_queue_info[] = {
};
static const struct mtk_port_cfg mtk_port_cfg_tbl[] = {
+ {CCCI_UART2_TX, CCCI_UART2_RX, PORT_TYPE_WWAN, "AT",
+ PORT_F_DFLT},
+ {CCCI_MBIM_TX, CCCI_MBIM_RX, PORT_TYPE_WWAN, "MBIM",
+ PORT_F_DFLT},
{CCCI_CONTROL_TX, CCCI_CONTROL_RX, PORT_TYPE_INTERNAL, "MDCTRL",
PORT_F_DFLT},
{CCCI_SAP_CONTROL_TX, CCCI_SAP_CONTROL_RX, PORT_TYPE_INTERNAL, "SAPCTRL",
--
2.34.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core
2026-11-05 7:56 ` [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core Jack Wu via B4 Relay
@ 2026-11-10 11:03 ` netdev-bot+sashiko
0 siblings, 0 replies; 12+ messages in thread
From: netdev-bot+sashiko @ 2026-11-10 11:03 UTC (permalink / raw)
To: jackbb_wu
Cc: loic.poulain, ryazanov.s.a, johannes, andrew+netdev, davem,
edumazet, kuba, pabeni, wen-zhi.huang, shi-wei.yeh, Minano.tseng,
matthias.bgg, angelogioacchino.delregno, horms, corbet, skhan,
robert_yu, Jeff_Chang, linux-kernel, netdev, linux-arm-kernel,
linux-mediatek, linux-doc
Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider.
Critical: 0 · High: 0 · Medium: 4 · Low: 2
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [Medium] The modem is reset only on the way down, in mtk_pci_remove()
and in mtk_pci_shutdown() (which calls mtk_pci_remove()).
- [Medium] In mtk_pci_remove(), which every shutdown also reaches through
mtk_pci_shutdown(), the MHCCIF doorbell fallback runs when…
- [Medium] mtk_pci_bar_init() maps BAR0/1 and BAR2/3 with
pcim_iomap_region(), which maps exactly the BAR length the device…
- [Medium] mtk_pci_request_irq() allocates exactly 32 MSI-X vectors.
- [Low] mtk_pci_bar_init() passes mdev->dev_str as the @name argument to
both pcim_iomap_region() calls.
- [Low] mtk_pci_remove() writes U32_MAX to
REG_IMASK_HOST_MSIX_CLR_GRP0_0, then calls mtk_pci_free_irq(), and…
--
net: wwan: t9xx: Add PCIe core
This adds the PCIe core of the MediaTek T9xx 5G WWAN modem driver. It
covers probe and remove, MSI-X setup, the MHCCIF event interface and the
register access helpers. Remove and shutdown reset the modem through the
ACPI MRST._RST method, with an MHCCIF reset doorbell as the fallback.
> diff --git a/drivers/net/wwan/t9xx/pcie/mtk_pci.c b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
> new file mode 100644
> index 0000000000000..902bcbe1512e9
> --- /dev/null
> +++ b/drivers/net/wwan/t9xx/pcie/mtk_pci.c
> @@ -0,0 +1,1044 @@
[ ... ]
> +static int mtk_pci_bar_init(struct mtk_md_dev *mdev)
> +{
> + struct pci_dev *pdev = to_pci_dev(mdev->dev);
> + struct mtk_pci_priv *priv = mdev->hw_priv;
> +
> + /* Fixed offsets used on these mappings (MSI-X registers up to
> + * 0x3103 on BAR0/1, the ATR-biased MHCCIF window on BAR2/3) are
> + * guaranteed by the only two devices this driver binds (0x0900,
> + * 0x01CA), whose BAR sizes are fixed by hardware. A device that
> + * misreports them is out of scope rather than impossible, so no
> + * pci_resource_len() checks.
> + */
> + priv->mac_reg_base = pcim_iomap_region(pdev, MTK_BAR_0_1_IDX,
> + mdev->dev_str);
[Severity: Medium]
pcim_iomap_region() maps exactly the BAR length the device reports. Binding
only needs a vendor/device/class match. So what happens with a device that
spoofs 14c3:0900 or 03f0:01ca and reports small BARs, for example a
hot-plugged Thunderbolt/USB4 device? It would get these fixed-offset
accesses:
mtk_pci_probe() writes U32_MAX at BAR0 + 0x3080
mtk_pci_irq_msix() reads BAR0 + 0x0F00 and BAR0 + 0x3100
mtk_mhccif_init() writes U32_MAX at bar23_addr + 0xA030 (via ext_reg_base)
The comment treats such a device as out of scope. Still, can these writes
land past the ioremap area and its guard page, in neighbouring ioremap or
vmalloc mappings?
Two pci_resource_len() checks (BAR0 >= 0x3104, BAR2 >= 0xA044) look like
enough to rule this out.
> + if (IS_ERR(priv->mac_reg_base)) {
> + dev_err(mdev->dev, "Failed to map BAR0/1\n");
> + return PTR_ERR(priv->mac_reg_base);
> + }
> +
> + priv->bar23_addr = pcim_iomap_region(pdev, MTK_BAR_2_3_IDX,
> + mdev->dev_str);
[Severity: Low]
Is mdev->dev_str the intended name argument for both pcim_iomap_region()
calls? The kernel-doc in drivers/pci/devres.c describes @name as "Name of
the driver requesting the resource". It becomes the resource name in
/proc/iomem and in resource conflict messages.
dev_str is a copy of pci_name(pdev), so the BARs get labelled with the PCI
address (e.g. 0000:01:00.0) instead of mtk_t9xx. t7xx passes DRIVER_NAME
to the same API. Would KBUILD_MODNAME work here?
[ ... ]
> +static int mtk_pci_request_irq(struct mtk_md_dev *mdev)
> +{
> + struct pci_dev *pdev = to_pci_dev(mdev->dev);
> + int ret;
> +
> + ret = pci_alloc_irq_vectors(pdev, MTK_IRQ_CNT_MAX,
> + MTK_IRQ_CNT_MAX, PCI_IRQ_MSIX);
> + if (ret < 0) {
> + dev_err(mdev->dev,
> + "Unable to alloc %d MSI-X vectors: ret=%d\n",
> + MTK_IRQ_CNT_MAX, ret);
> + return ret;
> + }
> +
> + ret = mtk_pci_request_irq_msix(mdev);
> + if (ret)
> + pci_free_irq_vectors(pdev);
> +
> + return ret;
> +}
[Severity: Medium]
Should the device's MSI-X vector configuration be programmed here? The
handler only takes status bit i on vector i:
mtk_pci_irq_msix() {
...
irq_state &= irq_desc->msix_bits; /* scope to this vector */
if (unlikely(!irq_state))
return IRQ_NONE;
...
}
Nothing writes the MAC register that tells the device how many host
vectors exist.
The register map matches t7xx: ISTATUS at 0x0F00, IMASK SET/CLR at
0x3000/0x3080, and the status dummy register at 0x0D1C. t7xx writes
T7XX_PCIE_CFG_MSIX (0x03ec) with ffs(irq_count) * 2 - 1 after allocating
its vectors:
t7xx_setup_msix()
t7xx_pcie_set_mac_msix_cfg(t7xx_dev, EXT_INT_NUM)
It writes it again in t7xx_pcie_interrupt_reinit().
Here the 1:1 layout depends on the register's reset or leftover value. If
that value merges sources onto fewer vectors, would the MHCCIF source
(bit 28) land on a vector whose handler drops it and returns IRQ_NONE?
That would leave the boot handshake unserviced. The spurious IRQ detector
could also end up disabling the line.
[ ... ]
> +static void mtk_pci_remove(struct pci_dev *pdev)
> +{
> + struct mtk_md_dev *mdev = pci_get_drvdata(pdev);
> + struct mtk_pci_priv *priv = mdev->hw_priv;
> + struct device *dev = &pdev->dev;
> + int i, ret;
> +
> + /* Silence every source before tearing anything down. */
> + mtk_pci_mac_write32(priv, REG_IMASK_HOST_MSIX_CLR_GRP0_0, U32_MAX);
> +
> + /* Free the vectors first: once no handler can run, the unmask at the
> + * tail of mtk_mhccif_isr_work() cannot re-open the source behind
> + * mtk_mhccif_exit()'s back.
> + */
> + mtk_pci_free_irq(mdev);
> + mtk_mhccif_exit(mdev);
[Severity: Low]
Does this ordering give the guarantee the comment describes?
cancel_work_sync() only runs later, inside mtk_mhccif_exit(). So a
mhccif_work queued before the mask-all write can still run after
mtk_pci_free_irq(), and it ends with:
mtk_mhccif_isr_work() {
...
mtk_pci_unmask_irq(mdev, priv->mhccif_irq_id);
}
This writes BIT(28) to REG_IMASK_HOST_MSIX_SET_GRP0_0. By then
pci_free_irq_vectors()->pci_msix_shutdown() has cleared
PCI_MSIX_FLAGS_ENABLE and called pci_intx_for_msi(dev, 1). The source is
unmasked with INTx enabled until mtk_mhccif_exit() masks it again.
There is a second window between the mask-all write and
mtk_pci_free_irq(). A pending work can unmask the source there, take a new
MSI-X interrupt and requeue itself.
The end state is masked. Would calling mtk_mhccif_exit() before
mtk_pci_free_irq() close both windows? While the vectors still exist,
mtk_pci_unregister_irq() gets a valid virq for synchronize_irq(), and
cancel_work_sync() plus the final mask finish before the vectors are
freed.
> + pci_clear_master(pdev);
> + pci_disable_device(pdev);
> +
> + /* Reset last: the endpoint comes back at power-on defaults, so no MMIO
> + * and no MSI-X teardown may follow it.
> + */
> + ret = mtk_pci_pldr(mdev);
> + if (!ret) {
> + /* MRST._RST may return before the link has retrained, and a
> + * config write the device does not take is simply lost.
> + */
> + for (i = 0; i < MTK_PLDR_READY_POLLS && !mtk_pci_link_check(mdev); i++)
> + msleep(MTK_PLDR_READY_POLL_MS);
> + if (!mtk_pci_link_check(mdev))
> + dev_warn(dev, "device not back %d ms after reset, its config space may not be restored\n",
> + MTK_PLDR_READY_POLLS * MTK_PLDR_READY_POLL_MS);
> + } else if (!mtk_pci_link_check(mdev)) {
> + dev_warn(dev, "PLDR failed (%d) and link is down, modem not reset\n", ret);
> + } else {
> + dev_warn(dev, "PLDR failed (%d), trying MHCCIF reset\n", ret);
> + if (mtk_pci_send_ext_evt(mdev, DEV_EVT_H2D_DEVICE_RESET)) {
> + dev_err(dev, "MHCCIF reset failed\n");
> + } else {
> + /* Unlike MRST._RST the doorbell returns before the device
> + * has reset, so let it settle before the restore below.
> + */
> + msleep(MTK_MHCCIF_RESET_DELAY_MS);
> + }
> + }
[Severity: Medium]
The PLDR branch polls mtk_pci_link_check() because "a config write the
device does not take is simply lost", and it warns if the device has not
come back. The doorbell branch only sleeps a fixed 2000 ms and then goes
on to the restore.
If the doorbell reset takes the endpoint off the link and it has not
retrained within 2 s, are the pci_restore_state() writes below silently
dropped?
pci_load_and_free_saved_state() has already freed priv->saved_state and
set it to NULL, and pci_restore_state() clears dev->state_saved. Nothing
is left to retry with. Would the next bind then see BARs the device does
not decode, with nothing in the log?
Probe allows platforms without MRST._RST, and there every remove and
shutdown takes this branch. Could it use the same link poll and warning
as the PLDR branch?
> +
> + /* Without this a runtime-PM suspend after unbind would snapshot the
> + * reset state over our copy, and whoever binds next finds BARs the
> + * device does not decode.
> + */
> + pci_load_and_free_saved_state(pdev, &priv->saved_state);
> + pci_restore_state(pdev);
> +}
> +
> +/* A kexec reboot neither resets nor powers down the slot, and the boot
> + * notification the next kernel waits for is one-shot, so reset the modem
> + * on the way down the same way remove() does.
> + */
> +static void mtk_pci_shutdown(struct pci_dev *pdev)
> +{
> + mtk_pci_remove(pdev);
> +}
[Severity: Medium]
The commit message says the shutdown reset is needed because "a kexec
reboot gets no platform reset and the next kernel would otherwise find
that firmware still running". Is crash kexec (kdump, kexec -p) covered?
That path is:
__crash_kexec()
machine_crash_shutdown()
machine_kexec()
It never calls device_shutdown(), so mtk_pci_shutdown() does not run. The
capture kernel then probes a modem whose firmware is still running.
By then the crashed kernel's mtk_mhccif_isr_work() has already acked
BOOT_FLOW_SYNC, and the comment in mtk_mhccif_init() says:
* Do NOT ack here: EP2RC status bits are latched one-shot
* notifications (e.g. BOOT_FLOW_SYNC raised before the driver
* loads) that the device never re-sends.
mtk_pci_probe() neither resets the modem nor checks whether firmware is
already running. Would the boot flow wait for a BOOT_FLOW_SYNC that never
comes? Later in the series, mtk_fsm_dev_add_act() moves to FSM_STATE_ON,
unmasks BOOT_FLOW_SYNC and waits.
The same state also seems reachable after mtk_pci_remove() if the reset
does not take effect. That covers a down link, or no MRST._RST combined
with firmware that ignores the doorbell. The commit message says the
doorbell "cannot recover firmware that has stopped responding - the case
the reset exists for".
Resetting the modem, or detecting running firmware, in probe would cover
these cases. It would also remove the 1-2 s wait that every reboot and
power-off now has, which still does not help kdump.
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261105-t9xx_driver_v1-v11-0-2084aa512864%40compal.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface
2026-11-05 7:56 ` [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface Jack Wu via B4 Relay
@ 2026-11-10 11:03 ` netdev-bot+sashiko
0 siblings, 0 replies; 12+ messages in thread
From: netdev-bot+sashiko @ 2026-11-10 11:03 UTC (permalink / raw)
To: jackbb_wu
Cc: loic.poulain, ryazanov.s.a, johannes, andrew+netdev, davem,
edumazet, kuba, pabeni, wen-zhi.huang, shi-wei.yeh, Minano.tseng,
matthias.bgg, angelogioacchino.delregno, horms, corbet, skhan,
robert_yu, Jeff_Chang, linux-kernel, netdev, linux-arm-kernel,
linux-mediatek, linux-doc
Thank you for your contribution! Sashiko AI review found 5 potential
issue(s) to consider.
Critical: 0 · High: 1 · Medium: 3 · Low: 1
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] net: wwan: t9xx: RX queue stall on reset race in
mtk_cldma_rx_done_work()
- [Medium] mtk_cldma_rearm_queues() can leave a started TX queue stopped
with descriptors still pending.
- [Medium] mtk_cldma_hw_recovery() is called from mtk_cldma_start_xfer()
when one TX queue's start-address register reads back 0.
- [Medium] When a single queue fails to stop while its channel is closing
(mtk_cldma_stop_queue() returns -ETIMEDOUT), mtk_cldma_txq_free() and…
- [Low] mtk_cldma_drv_init() writes 1 to REG_CLDMA_UL_DUMMY_0 and
REG_CLDMA_SO_DUMMY_0 under the comment "disable illegal memory check".
--
net: wwan: t9xx: Add control DMA interface
This adds the CLDMA (Cross Layer DMA) layer for the t9xx WWAN control
plane. It covers GPD ring handling for the TX and RX queues, the TRB
service thread, and register access for the two CLDMA instances. Probe
now attaches the transaction context to the control plane through
mtk_pci_dev_init() and mtk_pci_dev_exit().
> diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
> new file mode 100644
> index 0000000000000..d8763cdd67dac
> --- /dev/null
> +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
> @@ -0,0 +1,1207 @@
[ ... ]
> +static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
> +{
[ ... ]
> + mtk_cldma_drv_init(drv_info);
> +
> + for (i = 0; i < HW_QUE_NUM; i++) {
> + txq = drv_info->txq[i];
> + if (txq) {
> + spin_lock(&txq->ring_lock);
> + /* The device halts on an HWO=0 descriptor and RESUME
> + * never moves the cursor past it, so do not restart
> + * from free_idx - which recovery usually finds behind
> + * the hardware. Leave the queue stopped and let the
> + * next start_xfer() issue a fresh START.
> + */
> + req = txq->req_pool + txq->free_idx;
> + if (READ_ONCE(txq->tx_started) &&
> + !(req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
> + WRITE_ONCE(txq->tx_started, false);
> + /* Its XFER_DONE may have been lost in the
> + * reset; without this free_idx never advances.
> + */
> + queue_work(drv_info->wq, &txq->tx_done_work);
> + } else {
> + mtk_cldma_setup_start_addr(drv_info, DIR_TX, i,
> + req->gpd_dma_addr);
> + }
[Severity: Medium]
Can this branch leave a started TX queue stopped while it still has
descriptors pending?
Suppose the instance reset comes from another queue's
mtk_cldma_txq_free() or mtk_cldma_rxq_free() -ETIMEDOUT path, or from
mtk_cldma_hw_recovery(). Suppose also that this queue's slot at free_idx
is completed but not yet reaped (HWO=0, req->skb set), and later slots
still have HWO=1.
In that case this branch sets tx_started to false and only queues
tx_done_work. It programs no start address and issues no UL START.
mtk_cldma_tx_done_work() then reaps the completed slot and stops at the
first HWO=1 slot:
if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
spin_unlock(&txq->ring_lock);
break;
}
Nothing restarts the engine after that. The remaining descriptors and
their TRBs seem to stay stuck until another TX is submitted on this
queue. Later in the series, a blocking writer in mtk_port_send_data()
waits for these TRBs with no timeout.
When the next TX does arrive, mtk_cldma_start_xfer() reads a start
address of 0, which this branch left unprogrammed on purpose. It treats
that as lost device state:
if (unlikely(!val)) {
ret = mtk_cldma_hw_recovery(drv_info, txq, qno);
That logs "lost its queue state", runs mtk_cldma_drv_init() again and
sets need_restart on every RX queue of the instance.
mtk_cldma_tx_start_idx() already finds the first slot the device still
owns. Could it be used here, under the ring_lock already held, to
program that slot and keep the queue running?
This branch is unchanged at the end of the series. The "Add FSM thread"
patch only adds an is_stopping check.
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_ERROR);
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_XFER_DONE);
> + if (READ_ONCE(txq->tx_started))
> + mtk_cldma_start_queue(drv_info, DIR_TX, i);
> + spin_unlock(&txq->ring_lock);
> + }
[ ... ]
> +static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
> +{
[ ... ]
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
> + if (ret == -ETIMEDOUT) {
> + dev_err(mdev->dev, "TX queue %u stop timed out, resetting CLDMA%d\n",
> + txqno, drv_info->hw_id);
> + /* One rearm_lock section from the reset to the re-arm: no queue
> + * may be resumed or started in between, while it has no address.
> + */
> + mutex_lock(&drv_info->rearm_lock);
> + mtk_cldma_drv_reset(drv_info);
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
> + /* the reset took the whole instance down with this queue */
> + mtk_cldma_rearm_queues(drv_info);
> + mutex_unlock(&drv_info->rearm_lock);
> + }
[Severity: Medium]
If one queue fails to stop while its channel is closing, this pulses the
infra reset for the whole CLDMA instance. Is that safe while other
channels on the same instance may have DMA in flight?
The other queues are not stopped before mtk_cldma_drv_reset(). At the
end of the series CLDMA1 carries three channels (TXQ 0, 2 and 5 in
mtk_queue_info[]).
A TX GPD that another queue was sending keeps HWO=1, and
mtk_cldma_rearm_queues() restarts it from free_idx. Could the modem then
receive a partial copy followed by a full resend? An RX GPD that was
being filled would be left half-written.
mtk_cldma_rxq_free() reaches the same reset.
Also, mtk_cldma_drv_reset() returns right after releasing the reset:
mtk_pci_write32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr, val);
mtk_pci_read32(mdev, REG_DEV_INFRA_BASE + hw_regs->reg_infra_rst0_clr);
The next accesses are the mtk_cldma_stop_queue() poll and the
read-modify-write of UL_CFG and SO_CFG in mtk_cldma_drv_init(), which
selects 64-bit GPD mode.
Does the IP need a settle delay or a readiness check after RST0_CLR
before those accesses?
[ ... ]
> +/* The device reported a zero TX start address: it lost its queue state. */
> +static int mtk_cldma_hw_recovery(struct cldma_drv_info *drv_info, struct txq *txq, u32 qno)
> +{
> + u64 val;
> +
> + dev_err(drv_info->mdev->dev,
> + "CLDMA%d lost its queue state, re-initializing\n",
> + drv_info->hw_id);
> +
> + mtk_cldma_rearm_queues(drv_info);
[Severity: Medium]
mtk_cldma_start_xfer() calls this when one TX queue's start address
reads back 0. It then calls mtk_cldma_rearm_queues() without resetting
or stopping the instance first. Is the re-arm safe on an IP that may
still be running?
mtk_cldma_rearm_queues() looks written for the state right after
mtk_cldma_drv_reset(). On this path it read-modify-writes UL_CFG and
SO_CFG and rewrites the interrupt and busy masks on a live IP.
For every other published TX queue with tx_started set and HWO=1 at
free_idx, it also does:
mtk_cldma_setup_start_addr(drv_info, DIR_TX, i,
req->gpd_dma_addr);
...
if (READ_ONCE(txq->tx_started))
mtk_cldma_start_queue(drv_info, DIR_TX, i);
Nothing shows those engines are idle. Could a running queue be
reprogrammed and restarted mid-transfer, so a descriptor is processed
twice or a transfer is cut short?
The RX side has the same pattern in this patch. need_restart makes
mtk_cldma_rxq_restart() program SO_START_ADDR and issue START without
stopping the queue.
The later "Add FSM thread" patch adds mtk_cldma_stop_queue() to
mtk_cldma_rxq_restart(), but the TX side is unchanged at the end of the
series.
[ ... ]
> diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
> new file mode 100644
> index 0000000000000..65e78421f9cf1
> --- /dev/null
> +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma_drv.c
> @@ -0,0 +1,505 @@
[ ... ]
> +void mtk_cldma_drv_init(struct cldma_drv_info *drv_info)
> +{
[ ... ]
> + /* enable interrupt to PCIe */
> + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_int_mask, 0);
> +
> + /* disable illegal memory check */
> + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_ul_dummy_0, 1);
> + mtk_pci_write32(mdev, base + hw_regs->reg_cldma_so_dummy_0, 1);
> +}
[Severity: Low]
This may not be a bug, but could the reason for disabling the illegal
memory check be documented? Neither the code nor the commit message
explains it.
The write runs on every instance init and on every
mtk_cldma_rearm_queues() recovery.
Some paths can leave a stale buffer address in a GPD. One is the re-arm
after a reset. Another is an RX slot in mtk_cldma_rx_done_work() whose
skb was freed after a mapping failure while data_buff_ptr still holds
the old address.
With the check turned off, the device would not catch such an address.
Without an IOMMU, it could turn into a stray DMA to host memory.
The t7xx CLDMA driver does not appear to do this. Does the hardware
require this write?
[ ... ]
> @@ -0,0 +1,1207 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Copyright (c) 2022, MediaTek Inc.
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/dma-mapping.h>
> +#include <linux/dmapool.h>
> +#include <linux/err.h>
> +#include <linux/interrupt.h>
> +#include <linux/kdev_t.h>
> +#include <linux/kernel.h>
> +#include <linux/kthread.h>
> +#include <linux/list.h>
> +#include <linux/module.h>
> +#include <linux/mutex.h>
> +#include <linux/netdevice.h>
> +#include <linux/sched.h>
> +#include <linux/skbuff.h>
> +#include <linux/slab.h>
> +#include <linux/timer.h>
> +#include <linux/wait.h>
> +#include <linux/workqueue.h>
> +#include "mtk_pci.h"
> +#include "mtk_cldma.h"
> +#include "mtk_cldma_drv.h"
> +#include "mtk_dev.h"
> +
> +#define WAIT_HWO_ROUND (10)
> +#define WAIT_HWO_TIME (5)
> +#define NO_BUDGET (0)
> +
> +static const int mtk_cldma_hw_id_tbl[NR_CLDMA] = {
> + [CLDMA0] = CLDMA0_HW_ID,
> + [CLDMA1] = CLDMA1_HW_ID,
> +};
> +
> +static void mtk_cldma_tx_done_work(struct work_struct *work)
> +{
> + struct txq *txq = container_of(work, struct txq, tx_done_work);
> + struct cldma_drv_info *drv_info;
> + struct mtk_ctrl_trans *trans;
> + struct mtk_md_dev *mdev;
> + struct sk_buff *skb;
> + struct trb_srv *srv;
> + struct tx_req *req;
> + unsigned int state;
> + bool was_starved;
> + struct trb *trb;
> + int i, hif_id;
> + u32 txqno;
> +
> + drv_info = txq->drv_info;
> + hif_id = drv_info->hif_id;
> + txqno = txq->txqno;
> + mdev = drv_info->mdev;
> + trans = drv_info->cd->trans;
> +
> +again:
> + for (i = 0; i < txq->nr_gpds; i++) {
> + spin_lock(&txq->ring_lock);
> +
> + req = txq->req_pool + txq->free_idx;
> +
> + /* Ownership is decided by the slot alone: an empty slot has no
> + * skb, and one the hardware still owns has HWO set. The
> + * producer publishes both under ring_lock, so a slot can never
> + * be observed half-filled and there is no need to bound the
> + * walk by wr_idx.
> + */
> + if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
> + spin_unlock(&txq->ring_lock);
> + break;
> + }
> +
> + dma_rmb(); /* read descriptor fields after HWO check */
> +
> + dma_unmap_single(mdev->dev, req->data_dma_addr,
> + req->data_len, DMA_TO_DEVICE);
> +
> + skb = req->skb;
> + req->data_dma_addr = 0;
> + req->data_len = 0;
> + req->skb = NULL;
> +
> + txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
> + was_starved = atomic_fetch_inc(&txq->req_budget) == NO_BUDGET;
> +
> + spin_unlock(&txq->ring_lock);
> +
> + trb = (struct trb *)skb->cb;
> + trb->status = 0;
> + trb->trb_complete(skb);
> +
> + /* The service may already have been torn down; completing the
> + * TRB is still correct, there is just nobody left to wake.
> + */
> + srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
> + if (was_starved && srv)
> + wake_up(&srv->trb_waitq);
> + }
> +
> + state = mtk_cldma_check_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
> + if (state) {
> + if (unlikely(state == LINK_ERROR_VAL))
> + goto out;
> +
> + mtk_cldma_clr_intr_status(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
> +
> + cond_resched();
> +
> + goto again;
> + }
> +
> +out:
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
> +}
> +
> +/* data_recv_len is read back from a buffer the device writes, so it is
> + * clamped against req->mtu before it reaches skb_put().
> + *
> + * Returns -EPROTO when the device over-reports so the caller drops the packet
> + * instead of delivering a malformed skb.
> + */
> +static int mtk_cldma_rx_skb_adjust(struct mtk_md_dev *mdev, struct rx_req *req)
> +{
> + u32 recv_len;
> + int ret = 0;
> +
> + if (req->data_dma_addr) {
> + dma_unmap_single(mdev->dev, req->data_dma_addr,
> + req->mtu, DMA_FROM_DEVICE);
> + req->data_dma_addr = 0;
> + }
> + recv_len = le16_to_cpu(req->gpd->rx_gpd.data_recv_len);
> + if (recv_len > req->mtu) {
> + ret = -EPROTO;
> + recv_len = req->mtu;
> + }
> + req->skb->len = 0;
> + skb_reset_tail_pointer(req->skb);
> + skb_put(req->skb, recv_len);
> +
> + req->gpd->rx_gpd.data_recv_len = 0;
> +
> + return ret;
> +}
> +
> +static int mtk_cldma_reload_rx_skb(struct mtk_md_dev *mdev, struct rx_req *req)
> +{
> + int ret;
> +
> + req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
> + if (!req->skb)
> + return -ENOMEM;
> +
> + req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
> + req->mtu, DMA_FROM_DEVICE);
> + ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
> + if (unlikely(ret)) {
> + dev_warn_ratelimited(mdev->dev, "Failed to map SKB data\n");
> + req->data_dma_addr = 0;
> + dev_kfree_skb_any(req->skb);
> + req->skb = NULL;
> + return -EFAULT;
> + }
> + req->gpd->rx_gpd.data_buff_ptr_h = cpu_to_le32((u64)req->data_dma_addr >> 32);
> + req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
> +
> + return 0;
> +}
> +
> +static int mtk_cldma_check_rx_req(struct cldma_drv_info *drv_info, struct rxq *rxq)
> +{
> + struct rx_req *req = rxq->req_pool + rxq->free_idx;
> + u64 curr_addr;
> + int i;
> +
> + curr_addr = mtk_cldma_get_rx_curr_addr(drv_info, rxq->rxqno);
> + if (unlikely(!curr_addr))
> + return -ENXIO;
> +
> + if (req->gpd_dma_addr == curr_addr)
> + return -EAGAIN;
> + for (i = 0; i < WAIT_HWO_ROUND; i++) {
> + udelay(WAIT_HWO_TIME);
> + if (!(READ_ONCE(req->gpd->rx_gpd.gpd_flags) & CLDMA_GPD_FLAG_HWO))
> + break;
> + }
> + if (i == WAIT_HWO_ROUND) {
> + dev_err((drv_info->mdev)->dev, "Failed to check HWO=0\n");
> + return -EAGAIN;
> + }
> +
> + return 0;
> +}
> +
> +static bool mtk_cldma_rx_check_again(struct rxq *rxq)
> +{
> + struct cldma_drv_info *drv_info;
> + u32 state;
> + int rxqno;
> +
> + drv_info = rxq->drv_info;
> + rxqno = rxq->rxqno;
> +
> + state = mtk_cldma_check_intr_status(drv_info, DIR_RX,
> + rxqno, QUEUE_XFER_DONE);
> + if (!state || unlikely(state == LINK_ERROR_VAL))
> + return false;
> +
> + mtk_cldma_clr_intr_status(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
> + cond_resched();
> + return true;
> +}
> +
> +/* Point the device at the GPD it should fill next and start the queue.
> + * free_idx is owned by mtk_cldma_rx_done_work(), so this may only run
> + * from that worker, at a point where every consumed GPD is re-armed.
> + */
> +static void mtk_cldma_rxq_restart(struct cldma_drv_info *drv_info, struct rxq *rxq)
> +{
> + mtk_cldma_setup_start_addr(drv_info, DIR_RX, rxq->rxqno,
> + rxq->req_pool[rxq->free_idx].gpd_dma_addr);
> + mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
> +}
> +
> +static void mtk_cldma_rx_done_work(struct work_struct *work)
> +{
> + struct rx_req *req = NULL, *pre_req = NULL;
> + struct rxq *rxq = container_of(work, struct rxq, rx_done_work);
> + struct cldma_drv_info *drv_info;
> + struct mtk_md_dev *mdev;
> + struct sk_buff *rx_skb;
> + int i, ret, idx, len_err;
> +
> + drv_info = rxq->drv_info;
> + mdev = drv_info->mdev;
> +
> +again:
> + for (i = 0; i < rxq->nr_gpds; i++) {
> + req = rxq->req_pool + rxq->free_idx;
> + if (!req->skb) {
> + /* An empty slot means an earlier pass could not refill
> + * it and re-queued this work to retry it.
> + */
> + if (mtk_cldma_reload_rx_skb(mdev, req))
> + goto out_retry;
> + goto arm_slot;
> + }
> +
> + if (req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)
> + break;
> +
> + dma_rmb(); /* read descriptor fields after HWO check */
> +
> + len_err = mtk_cldma_rx_skb_adjust(mdev, req);
> + rx_skb = req->skb;
> + req->skb = NULL;
> +
> + ret = mtk_cldma_reload_rx_skb(mdev, req);
> + if (ret) {
> + /* Alloc failed — recycle old buffer, drop packet. */
> + skb_trim(rx_skb, 0);
> + req->skb = rx_skb;
> + req->data_dma_addr = dma_map_single(mdev->dev,
> + rx_skb->data,
> + req->mtu,
> + DMA_FROM_DEVICE);
> + if (dma_mapping_error(mdev->dev, req->data_dma_addr)) {
> + /* An unmapped skb left in the slot would be
> + * read back as a completed descriptor.
> + */
> + dev_kfree_skb_any(rx_skb);
> + req->skb = NULL;
> + req->data_dma_addr = 0;
> + goto out_retry;
> + } else {
> + req->gpd->rx_gpd.data_buff_ptr_h =
> + cpu_to_le32((u64)req->data_dma_addr >> 32);
> + req->gpd->rx_gpd.data_buff_ptr_l =
> + cpu_to_le32(req->data_dma_addr);
> + }
> + } else if (len_err) {
> + dev_err_ratelimited(mdev->dev,
> + "Drop oversized packet on cldma%d rxq%u\n",
> + drv_info->hw_id, rxq->rxqno);
> + dev_kfree_skb_any(rx_skb);
> + } else {
> + rxq->rx_done(rx_skb, rxq->arg);
> + }
> +
> +arm_slot:
> + wmb(); /* ensure addr set done before HWO setup done */
> +
> + idx = rxq->free_idx == 0 ? rxq->nr_gpds - 1 : rxq->free_idx - 1;
> + pre_req = rxq->req_pool + idx;
> + pre_req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
> + rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
> + }
> +
> + ret = mtk_cldma_check_rx_req(drv_info, rxq);
> + if (!ret)
> + goto again;
> +
> + /* -ENXIO means the queue has no start address programmed, which is
> + * exactly what a restart installs, so consume need_restart first.
> + * Only the resume below is skipped in that case.
> + */
> + mutex_lock(&drv_info->rearm_lock);
> + if (!atomic_read(&rxq->need_exit)) {
> + if (atomic_xchg(&rxq->need_restart, 0))
> + mtk_cldma_rxq_restart(drv_info, rxq);
> + else if (ret != -ENXIO)
> + mtk_cldma_resume_queue(drv_info, DIR_RX, rxq->rxqno);
> + }
> + mutex_unlock(&drv_info->rearm_lock);
> +
> + if (ret == -ENXIO)
> + goto out;
> +
> + if (mtk_cldma_rx_check_again(rxq))
> + goto again;
> + goto out;
> +
> +out_retry:
> + /* The empty slot keeps the GPD before it unarmed: once the device
> + * has used up the armed ones, no XFER_DONE comes to run this work
> + * again, so retry from here. The sleep paces a failure that does
> + * not wait, such as a DMA mapping one.
> + */
> + usleep_range(1000, 2000);
> + if (!atomic_read(&rxq->need_exit))
> + queue_work(drv_info->wq, &rxq->rx_done_work);
> +out:
> + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
> + mtk_cldma_clear_ip_busy(drv_info);
> +}
> +
> +static struct txq *mtk_cldma_txq_alloc(struct cldma_drv_info *drv_info, struct queue_info *que)
> +{
> + struct mtk_md_dev *mdev;
> + struct tx_req *next;
> + struct tx_req *req;
> + struct txq *txq;
> + int i, ret;
> +
> + mdev = drv_info->mdev;
> +
> + txq = kzalloc_obj(*txq);
> + if (!txq)
> + return NULL;
> +
> + txq->que = que;
> + txq->drv_info = drv_info;
> + txq->txqno = txq->que->txqno;
> + txq->nr_gpds = txq->que->tx_nr_gpds;
> + atomic_set(&txq->req_budget, txq->que->tx_nr_gpds);
> + spin_lock_init(&txq->ring_lock);
> +
> + txq->req_pool = kcalloc(txq->nr_gpds, sizeof(*req), GFP_KERNEL);
> + if (!txq->req_pool)
> + goto err_free_txq;
> +
> + for (i = 0; i < txq->nr_gpds; i++) {
> + req = txq->req_pool + i;
> + req->mtu = txq->que->tx_mtu;
> + req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
> + if (!req->gpd)
> + goto err_free_req;
> + }
> +
> + for (i = 0; i < txq->nr_gpds; i++) {
> + req = txq->req_pool + i;
> + next = txq->req_pool + ((i + 1) % txq->nr_gpds);
> + req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
> + req->gpd->tx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
> + req->gpd->tx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
> + }
> +
> + INIT_WORK(&txq->tx_done_work, mtk_cldma_tx_done_work);
> +
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txq->txqno);
> + if (ret) {
> + dev_warn(mdev->dev, "Failed to stop TX queue %u before alloc\n", txq->txqno);
> + goto err_free_req;
> + }
> + /* Publish the queue before unmasking its interrupts: an interrupt
> + * that fires in between must find a valid txq, or the sources it
> + * masks are never unmasked again. Not before the stop: the error
> + * path above frees the queue, so it must never have been visible.
> + * The release pairs with the acquire load in the ISR.
> + */
> + smp_store_release(&drv_info->txq[txq->txqno], txq);
> + txq->tx_started = false;
> + mtk_cldma_setup_start_addr(drv_info, DIR_TX, txq->txqno,
> + txq->req_pool[0].gpd_dma_addr);
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_ERROR);
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, txq->txqno, QUEUE_XFER_DONE);
> +
> + return txq;
> +
> +err_free_req:
> + for (i = 0; i < txq->nr_gpds; i++) {
> + req = txq->req_pool + i;
> + if (!req->gpd)
> + break;
> + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
> + }
> + kfree(txq->req_pool);
> +err_free_txq:
> + kfree(txq);
> + return NULL;
> +}
> +
> +/* mtk_cldma_drv_init() rewrites IP-global configuration (including the interrupt
> + * mask) and mtk_cldma_drv_reset() wipes every queue of the instance, so all
> + * queues still published in drv_info have to be re-armed, not only the one
> + * that triggered the reset. A queue already unpublished (NULL slot) is being
> + * torn down and is deliberately left stopped.
> + */
> +static void mtk_cldma_rearm_queues(struct cldma_drv_info *drv_info)
> +{
> + struct tx_req *req;
> + struct txq *txq;
> + struct rxq *rxq;
> + int i;
> +
> + /* The caller holds rearm_lock from the reset to the end of the re-arm:
> + * no queue is kicked or resumed while the instance has no addresses.
> + */
> + lockdep_assert_held(&drv_info->rearm_lock);
> +
> + mtk_cldma_drv_init(drv_info);
> +
> + for (i = 0; i < HW_QUE_NUM; i++) {
> + txq = drv_info->txq[i];
> + if (txq) {
> + spin_lock(&txq->ring_lock);
> + /* The device halts on an HWO=0 descriptor and RESUME
> + * never moves the cursor past it, so do not restart
> + * from free_idx - which recovery usually finds behind
> + * the hardware. Leave the queue stopped and let the
> + * next start_xfer() issue a fresh START.
> + */
> + req = txq->req_pool + txq->free_idx;
> + if (READ_ONCE(txq->tx_started) &&
> + !(req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO)) {
> + WRITE_ONCE(txq->tx_started, false);
> + /* Its XFER_DONE may have been lost in the
> + * reset; without this free_idx never advances.
> + */
> + queue_work(drv_info->wq, &txq->tx_done_work);
> + } else {
> + mtk_cldma_setup_start_addr(drv_info, DIR_TX, i,
> + req->gpd_dma_addr);
> + }
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_ERROR);
> + mtk_cldma_unmask_intr(drv_info, DIR_TX, i, QUEUE_XFER_DONE);
> + if (READ_ONCE(txq->tx_started))
> + mtk_cldma_start_queue(drv_info, DIR_TX, i);
> + spin_unlock(&txq->ring_lock);
> + }
> +
> + rxq = drv_info->rxq[i];
> + if (rxq) {
> + mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_ERROR);
> + mtk_cldma_unmask_intr(drv_info, DIR_RX, i, QUEUE_XFER_DONE);
> + /* free_idx belongs to rx_done_work, which may be halfway
> + * through a packet: let it program the start address
> + * once it reaches a consistent point.
> + */
> + atomic_set(&rxq->need_restart, 1);
> + queue_work(drv_info->wq, &rxq->rx_done_work);
> + }
> + }
> +}
> +
> +static void mtk_cldma_txq_free(struct cldma_drv_info *drv_info, u32 txqno)
> +{
> + struct mtk_md_dev *mdev;
> + struct tx_req *req;
> + struct txq *txq;
> + struct trb *trb;
> + int irq_id;
> + int i, ret;
> +
> + mdev = drv_info->mdev;
> +
> + txq = drv_info->txq[txqno];
> + /* Unpublish before teardown; pairs with the acquire loads of this
> + * slot. No release is needed, there is no prior store to expose.
> + */
> + WRITE_ONCE(drv_info->txq[txqno], NULL);
> + /* stop HW tx transaction; -ENODEV means the link is dead, so the
> + * device cannot be walking the ring and the free may proceed.
> + */
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
> + if (ret == -ETIMEDOUT) {
> + dev_err(mdev->dev, "TX queue %u stop timed out, resetting CLDMA%d\n",
> + txqno, drv_info->hw_id);
> + /* One rearm_lock section from the reset to the re-arm: no queue
> + * may be resumed or started in between, while it has no address.
> + */
> + mutex_lock(&drv_info->rearm_lock);
> + mtk_cldma_drv_reset(drv_info);
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, txqno);
> + /* the reset took the whole instance down with this queue */
> + mtk_cldma_rearm_queues(drv_info);
> + mutex_unlock(&drv_info->rearm_lock);
> + }
> + txq->tx_started = false;
> +
> + irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
> + synchronize_irq(irq_id);
> + /* flush on-going work */
> + flush_work(&txq->tx_done_work);
> + mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_XFER_DONE);
> + mtk_cldma_mask_intr(drv_info, DIR_TX, txqno, QUEUE_ERROR);
> +
> + if (ret == -ETIMEDOUT) {
> + /* Still a live DMA target: leak the ring rather than hand it
> + * back to the allocator.
> + */
> + dev_err(mdev->dev, "TX queue %u cannot be stopped, leaking its ring\n",
> + txqno);
> + return;
> + }
> +
> + /* Free tx req resource. No ring_lock is taken here: txq was already
> + * unpublished from drv_info->txq[] above, so no new producer can enter,
> + * and synchronize_irq() plus flush_work() have retired the consumer.
> + */
> + for (i = 0; i < txq->nr_gpds; i++) {
> + req = txq->req_pool + txq->free_idx;
> + if (req->skb) {
> + dma_unmap_single(mdev->dev, req->data_dma_addr,
> + req->data_len, DMA_TO_DEVICE);
> + trb = (struct trb *)req->skb->cb;
> + trb->status = -EPIPE;
> + trb->trb_complete(req->skb);
> + }
> + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
> + txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
> + }
> +
> + kfree(txq->req_pool);
> + kfree(txq);
> +}
> +
> +static void mtk_cldma_rxq_alloc_cancel(struct cldma_drv_info *drv_info, struct rx_req *req)
> +{
> + struct mtk_md_dev *mdev = drv_info->mdev;
> +
> + if (req->skb) {
> + if (!dma_mapping_error(mdev->dev, req->data_dma_addr))
> + dma_unmap_single(mdev->dev, req->data_dma_addr,
> + req->mtu, DMA_FROM_DEVICE);
> + req->data_dma_addr = 0;
> + dev_kfree_skb_any(req->skb);
> + }
> + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
> +}
> +
> +static struct rxq *mtk_cldma_rxq_alloc(struct cldma_drv_info *drv_info, struct queue_info *que,
> + struct sk_buff *skb)
> +{
> + struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
> + struct trb *trb = (struct trb *)skb->cb;
> + struct mtk_md_dev *mdev;
> + struct rx_req *next;
> + struct rx_req *req;
> + struct rxq *rxq;
> + int ret;
> + int i;
> +
> + mdev = drv_info->mdev;
> +
> + rxq = kzalloc_obj(*rxq);
> + if (!rxq)
> + return NULL;
> +
> + rxq->que = que;
> + rxq->drv_info = drv_info;
> + rxq->rxqno = rxq->que->rxqno;
> + if (rxq->que->rx_nr_gpds < MIN_GPD_NUM) {
> + dev_err(mdev->dev,
> + "Failed to alloc cldma%d rxq%u due to gpd number < 2\n",
> + drv_info->hw_id, rxq->rxqno);
> + goto err_free_rxq;
> + }
> + rxq->nr_gpds = rxq->que->rx_nr_gpds;
> + rxq->arg = trb->priv;
> + rxq->rx_done = trb_open_priv->rx_done;
> + atomic_set(&rxq->need_exit, 0);
> + atomic_set(&rxq->need_restart, 0);
> +
> + rxq->req_pool = kcalloc(rxq->nr_gpds, sizeof(*req), GFP_KERNEL);
> + if (!rxq->req_pool)
> + goto err_free_rxq;
> +
> + /* setup rx request */
> + for (i = 0; i < rxq->nr_gpds; i++) {
> + req = rxq->req_pool + i;
> + req->mtu = rxq->que->rx_mtu;
> + req->gpd = dma_pool_zalloc(drv_info->gpd_dma_pool, GFP_KERNEL, &req->gpd_dma_addr);
> + if (!req->gpd)
> + goto err_free_req;
> + req->skb = __dev_alloc_skb(req->mtu, GFP_KERNEL);
> + if (!req->skb)
> + goto err_free_req;
> + req->data_dma_addr = dma_map_single(mdev->dev, req->skb->data,
> + req->mtu, DMA_FROM_DEVICE);
> + ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
> + if (unlikely(ret))
> + goto err_free_req;
> + }
> +
> + for (i = 0; i < rxq->nr_gpds; i++) {
> + req = rxq->req_pool + i;
> + next = rxq->req_pool + ((i + 1) % rxq->nr_gpds);
> + req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_IOC;
> + req->gpd->rx_gpd.data_allow_len = cpu_to_le16(req->mtu);
> + req->gpd->rx_gpd.next_gpd_ptr_h = cpu_to_le32((u64)(next->gpd_dma_addr) >> 32);
> + req->gpd->rx_gpd.next_gpd_ptr_l = cpu_to_le32(next->gpd_dma_addr);
> + req->gpd->rx_gpd.data_buff_ptr_h = cpu_to_le32((u64)(req->data_dma_addr) >> 32);
> + req->gpd->rx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
> + if (i != rxq->nr_gpds - 1)
> + req->gpd->rx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
> + }
> +
> + INIT_WORK(&rxq->rx_done_work, mtk_cldma_rx_done_work);
> +
> + ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxq->rxqno);
> + if (ret) {
> + dev_warn(mdev->dev, "Failed to stop RX queue %u before alloc\n", rxq->rxqno);
> + goto err_free_req;
> + }
> + /* Pairs with the acquire load in the ISR, as for txq above. */
> + smp_store_release(&drv_info->rxq[rxq->rxqno], rxq);
> + mtk_cldma_setup_start_addr(drv_info, DIR_RX,
> + rxq->rxqno, rxq->req_pool[0].gpd_dma_addr);
> + mtk_cldma_start_queue(drv_info, DIR_RX, rxq->rxqno);
> + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_ERROR);
> + mtk_cldma_unmask_intr(drv_info, DIR_RX, rxq->rxqno, QUEUE_XFER_DONE);
> +
> + return rxq;
> +
> +err_free_req:
> + for (i = 0; i < rxq->nr_gpds; i++) {
> + req = rxq->req_pool + i;
> + if (!req->gpd)
> + break;
> + mtk_cldma_rxq_alloc_cancel(drv_info, req);
> + }
> +
> + kfree(rxq->req_pool);
> +err_free_rxq:
> + kfree(rxq);
> + return NULL;
> +}
> +
> +static void mtk_cldma_rxq_free(struct cldma_drv_info *drv_info, u32 rxqno)
> +{
> + struct mtk_md_dev *mdev;
> + struct rx_req *req;
> + struct rxq *rxq;
> + int irq_id;
> + int i, ret;
> +
> + mdev = drv_info->mdev;
> +
> + rxq = drv_info->rxq[rxqno];
> + /* Unpublish before teardown; pairs with the acquire loads of this
> + * slot. No release is needed, there is no prior store to expose.
> + */
> + WRITE_ONCE(drv_info->rxq[rxqno], NULL);
> +
> + /* stop HW rx transaction; -ENODEV means the link is dead, so the
> + * device cannot be walking the ring and the free may proceed.
> + */
> + atomic_set(&rxq->need_exit, 1);
> + ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
> + if (ret == -ETIMEDOUT) {
> + dev_err(mdev->dev, "RX queue %u stop timed out, resetting CLDMA%d\n",
> + rxqno, drv_info->hw_id);
> + /* One rearm_lock section from the reset to the re-arm: no queue
> + * may be resumed or started in between, while it has no address.
> + */
> + mutex_lock(&drv_info->rearm_lock);
> + mtk_cldma_drv_reset(drv_info);
> + ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
> + /* the reset took the whole instance down with this queue */
> + mtk_cldma_rearm_queues(drv_info);
> + mutex_unlock(&drv_info->rearm_lock);
> + }
> +
> + irq_id = mtk_pci_get_virq_id(mdev, drv_info->pci_ext_irq_id);
> + synchronize_irq(irq_id);
> + /* not flush_work(): the worker re-queues itself on a refill failure */
> + cancel_work_sync(&rxq->rx_done_work);
> + /* a cancelled pass would have cleared it on its way out */
> + mtk_cldma_clear_ip_busy(drv_info);
> + /* only now: a worker retired by the cancel unmasks XFER_DONE on its
> + * way out
> + */
> + mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_XFER_DONE);
> + mtk_cldma_mask_intr(drv_info, DIR_RX, rxqno, QUEUE_ERROR);
> +
> + /* The worker can resume or restart the queue between its need_exit
> + * check and the MMIO write that follows it, so cancel_work_sync() does
> + * not prove the engine was left stopped. Stopping again is idempotent.
> + */
> + if (ret != -ETIMEDOUT)
> + ret = mtk_cldma_stop_queue(drv_info, DIR_RX, rxqno);
> +
> + if (ret == -ETIMEDOUT) {
> + /* Still a live DMA target: leak the ring rather than hand it
> + * back to the allocator.
> + */
> + dev_err(mdev->dev, "RX queue %u cannot be stopped, leaking its ring\n",
> + rxqno);
> + return;
> + }
> +
> + /* free rx req resource */
> + for (i = 0; i < rxq->nr_gpds; i++) {
> + req = rxq->req_pool + rxq->free_idx;
> + if (!(req->gpd->rx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO) &&
> + le16_to_cpu(req->gpd->rx_gpd.data_recv_len)) {
> + if (!mtk_cldma_rx_skb_adjust(mdev, req)) {
> + rxq->rx_done(req->skb, rxq->arg);
> + req->skb = NULL;
> + }
> + }
> + if (req->skb) {
> + if (req->data_dma_addr)
> + dma_unmap_single(mdev->dev, req->data_dma_addr,
> + req->mtu, DMA_FROM_DEVICE);
> + dev_kfree_skb_any(req->skb);
> + req->skb = NULL;
> + }
> + dma_pool_free(drv_info->gpd_dma_pool, req->gpd, req->gpd_dma_addr);
> + rxq->free_idx = (rxq->free_idx + 1) % rxq->nr_gpds;
> + }
> +
> + kfree(rxq->req_pool);
> + kfree(rxq);
> +}
> +
> +/* The device reported a zero TX start address: it lost its queue state. */
> +static int mtk_cldma_hw_recovery(struct cldma_drv_info *drv_info, struct txq *txq, u32 qno)
> +{
> + u64 val;
> +
> + dev_err(drv_info->mdev->dev,
> + "CLDMA%d lost its queue state, re-initializing\n",
> + drv_info->hw_id);
> +
> + mtk_cldma_rearm_queues(drv_info);
> +
> + /* A started queue the re-arm left stopped has no address yet, on
> + * purpose: the caller programs one. Only a dead link, or a 0 on a
> + * queue still marked started, means the re-init failed.
> + */
> + val = mtk_cldma_get_tx_start_addr(drv_info, qno);
> + if (val == U64_MAX || (!val && READ_ONCE(txq->tx_started)))
> + return -EIO;
> +
> + return 0;
> +}
> +
> +/* After a re-arm the slot at free_idx may hold a descriptor the device
> + * completed but tx_done_work has not reaped yet; a START there parks the
> + * engine on HWO=0. Start at the first slot the device still owns.
> + */
> +static u32 mtk_cldma_tx_start_idx(struct txq *txq)
> +{
> + u32 idx = txq->free_idx;
> + struct tx_req *req;
> + int i;
> +
> + for (i = 0; i < txq->nr_gpds; i++) {
> + req = txq->req_pool + idx;
> + if (!req->skb || (req->gpd->tx_gpd.gpd_flags & CLDMA_GPD_FLAG_HWO))
> + return idx;
> + idx = (idx + 1) % txq->nr_gpds;
> + }
> +
> + return txq->free_idx;
> +}
> +
> +static int mtk_cldma_start_xfer(struct cldma_drv_info *drv_info, u32 qno)
> +{
> + struct txq *txq;
> + int ret = 0;
> + u64 val;
> +
> + txq = drv_info->txq[qno];
> +
> + /* a reset in another queue's teardown must not land between the
> + * address check and the kick
> + */
> + mutex_lock(&drv_info->rearm_lock);
> + val = mtk_cldma_get_tx_start_addr(drv_info, qno);
> + if (unlikely(val == U64_MAX)) {
> + ret = -EIO;
> + goto out;
> + }
> +
> + if (unlikely(!val)) {
> + ret = mtk_cldma_hw_recovery(drv_info, txq, qno);
> + if (ret)
> + goto out;
> + }
> +
> + /* Hold ring_lock across programming and kicking the queue so the
> + * start address given to the device is the one free_idx still
> + * names; tx_done_work advances free_idx under the same lock.
> + */
> + spin_lock(&txq->ring_lock);
> + if (unlikely(!READ_ONCE(txq->tx_started))) {
> + mtk_cldma_setup_start_addr(drv_info, DIR_TX, qno,
> + txq->req_pool[mtk_cldma_tx_start_idx(txq)].gpd_dma_addr);
> + mtk_cldma_start_queue(drv_info, DIR_TX, qno);
> + WRITE_ONCE(txq->tx_started, true);
> + } else {
> + mtk_cldma_resume_queue(drv_info, DIR_TX, qno);
> + }
> + spin_unlock(&txq->ring_lock);
> +out:
> + mutex_unlock(&drv_info->rearm_lock);
> + return ret;
> +}
> +
> +int mtk_cldma_init(struct mtk_ctrl_trans *trans)
> +{
> + struct cldma_dev *cd;
> +
> + cd = kzalloc_obj(*cd);
> + if (!cd)
> + return -ENOMEM;
> +
> + cd->trans = trans;
> + trans->dev = cd;
> +
> + return 0;
> +}
> +
> +void mtk_cldma_exit(struct mtk_ctrl_trans *trans)
> +{
> + if (!trans->dev)
> + return;
> +
> + kfree(trans->dev);
> + trans->dev = NULL;
> +}
> +
> +static int mtk_cldma_open(struct cldma_dev *cd, struct sk_buff *skb)
> +{
> + struct trb_open_priv *trb_open_priv = (struct trb_open_priv *)skb->data;
> + struct trb *trb = (struct trb *)skb->cb;
> + struct cldma_drv_info *drv_info;
> + struct queue_info *que;
> + struct txq *txq;
> + struct rxq *rxq;
> + int ret = 0;
> +
> + que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
> + if (unlikely(!que)) {
> + trb->status = -EINVAL;
> + trb->trb_complete(skb);
> + return -EINVAL;
> + }
> + drv_info = cd->cldma_drv_info[que->hif_id];
> + if (!drv_info) {
> + ret = -EIO;
> + goto out;
> + }
> +
> + if (que->tx_mtu == 0 || que->rx_mtu == 0) {
> + dev_err((cd->trans->mdev)->dev,
> + "Failed to enable cldma%d txq%u rxq%u due to wrong mtu\n",
> + drv_info->hw_id, que->txqno, que->rxqno);
> + ret = -EINVAL;
> + goto out;
> + }
> +
> + trb_open_priv->tx_mtu = que->tx_mtu;
> +
> + if (drv_info->txq[que->txqno] || drv_info->rxq[que->rxqno]) {
> + ret = -EBUSY;
> + goto out;
> + }
> +
> + txq = mtk_cldma_txq_alloc(drv_info, que);
> + if (!txq) {
> + ret = -ENOMEM;
> + goto out;
> + }
> +
> + rxq = mtk_cldma_rxq_alloc(drv_info, que, skb);
> + if (!rxq) {
> + ret = -ENOMEM;
> + mtk_cldma_txq_free(drv_info, txq->txqno);
> + goto out;
> + }
> +
> +out:
> + /* -EBUSY leaves the channel open to the submitter, so the count this
> + * ENABLE added is released by its eventual DISABLE instead.
> + */
> + if (ret && ret != -EBUSY)
> + cd->trans->usr_cnt[que->hif_id][que->txqno]--;
> +
> + /* Pairs with the acquire load of status on the submitting side: the
> + * payload written to the skb above must be visible with it.
> + */
> + smp_store_release(&trb->status, ret);
> + trb->trb_complete(skb);
> +
> + return ret;
> +}
> +
> +/* Complete every submitted-but-unkicked request with an error so no TRB
> + * is stranded in the ring when the doorbell cannot be rung.
> + */
> +static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info,
> + struct txq *txq, int err)
> +{
> + struct mtk_ctrl_trans *trans = drv_info->cd->trans;
> + int hif_id = drv_info->hif_id;
> + u32 txqno = txq->txqno;
> + struct sk_buff *skb;
> + struct trb_srv *srv;
> + struct tx_req *req;
> + bool was_starved;
> + struct trb *trb;
> + int i;
> +
> + for (i = 0; i < txq->nr_gpds; i++) {
> + spin_lock(&txq->ring_lock);
> +
> + req = txq->req_pool + txq->free_idx;
> + if (!req->skb) {
> + spin_unlock(&txq->ring_lock);
> + break;
> + }
> +
> + req->gpd->tx_gpd.gpd_flags &= ~CLDMA_GPD_FLAG_HWO;
> +
> + dma_unmap_single(drv_info->mdev->dev, req->data_dma_addr,
> + req->data_len, DMA_TO_DEVICE);
> +
> + skb = req->skb;
> + req->data_dma_addr = 0;
> + req->data_len = 0;
> + req->skb = NULL;
> +
> + txq->free_idx = (txq->free_idx + 1) % txq->nr_gpds;
> + was_starved = atomic_fetch_inc(&txq->req_budget) == NO_BUDGET;
> +
> + spin_unlock(&txq->ring_lock);
> +
> + trb = (struct trb *)skb->cb;
> + trb->status = err;
> + trb->trb_complete(skb);
> +
> + /* The service may already have been torn down; completing the
> + * TRB is still correct, there is just nobody left to wake.
> + */
> + srv = trans->trb_srv[trans->srv_cfg[hif_id][txqno]];
> + if (was_starved && srv)
> + wake_up(&srv->trb_waitq);
> + }
> +}
> +
> +static int mtk_cldma_tx(struct cldma_dev *cd, struct sk_buff *skb)
> +{
> + struct trb *trb = (struct trb *)skb->cb;
> + struct cldma_drv_info *drv_info;
> + struct mtk_md_dev *mdev;
> + struct queue_info *que;
> + struct txq *txq;
> + int ret, err;
> +
> + que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
> + if (unlikely(!que))
> + return -EPIPE;
> + drv_info = cd->cldma_drv_info[que->hif_id];
> + if (unlikely(!drv_info))
> + return -EPIPE;
> + txq = drv_info->txq[que->txqno];
> + if (unlikely(!txq))
> + return -EPIPE;
> +
> + mdev = drv_info->mdev;
> +
> + ret = mtk_cldma_start_xfer(drv_info, que->txqno);
> + if (unlikely(ret)) {
> + dev_err(mdev->dev, "Failed to trigger cldma tx\n");
> + /* A failed start does not mean the engine is idle - recovery
> + * may already have issued UL_START - and the flush below
> + * unmaps the buffers. If the queue cannot be stopped, leak the
> + * requests rather than unmap memory the device still owns.
> + */
> + err = mtk_cldma_stop_queue(drv_info, DIR_TX, que->txqno);
> + /* -ENODEV: the link is dead, nothing is walking the ring */
> + if (err && err != -ENODEV) {
> + dev_err(mdev->dev,
> + "TX queue %u stop failed, keeping its requests\n",
> + que->txqno);
> + return ret;
> + }
> +
> + spin_lock(&txq->ring_lock);
> + WRITE_ONCE(txq->tx_started, false);
> + spin_unlock(&txq->ring_lock);
> + mtk_cldma_txq_flush(drv_info, txq, ret);
> + }
> +
> + return ret;
> +}
> +
> +static int mtk_cldma_close(struct cldma_dev *cd, struct sk_buff *skb)
> +{
> + struct trb *trb = (struct trb *)skb->cb;
> + struct cldma_drv_info *drv_info;
> + struct queue_info *que;
> +
> + que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
> + if (unlikely(!que)) {
> + trb->status = -EPIPE;
> + trb->trb_complete(skb);
> + return -EPIPE;
> + }
> + drv_info = cd->cldma_drv_info[que->hif_id];
> + if (unlikely(!drv_info)) {
> + trb->status = -EPIPE;
> + trb->trb_complete(skb);
> + return -EPIPE;
> + }
> +
> + if (drv_info->txq[que->txqno])
> + mtk_cldma_txq_free(drv_info, que->txqno);
> + if (drv_info->rxq[que->rxqno])
> + mtk_cldma_rxq_free(drv_info, que->rxqno);
> +
> + trb->status = 0;
> + trb->trb_complete(skb);
> +
> + return 0;
> +}
> +
> +static int mtk_cldma_txbuf_set(struct cldma_drv_info *drv_info, struct sk_buff *skb,
> + struct tx_req *req)
> +{
> + struct mtk_md_dev *mdev = drv_info->mdev;
> + int ret;
> +
> + /* Only the linear area is mapped; a nonlinear SKB here would be
> + * silently truncated to skb_headlen().
> + */
> + if (WARN_ON_ONCE(skb_is_nonlinear(skb)))
> + return -EINVAL;
> +
> + req->data_dma_addr = dma_map_single(mdev->dev, skb->data,
> + skb_headlen(skb), DMA_TO_DEVICE);
> + ret = dma_mapping_error(mdev->dev, req->data_dma_addr);
> + if (unlikely(ret)) {
> + req->data_dma_addr = 0;
> + dev_err_ratelimited(mdev->dev, "Failed to map dma! error:%d\n", ret);
> + return -ENOMEM;
> + }
> +
> + req->gpd->tx_gpd.data_buff_ptr_h = cpu_to_le32((u64)(req->data_dma_addr) >> 32);
> + req->gpd->tx_gpd.data_buff_ptr_l = cpu_to_le32(req->data_dma_addr);
> +
> + return 0;
> +}
> +
> +int mtk_cldma_submit_tx(void *dev, struct sk_buff *skb)
> +{
> + struct trb *trb = (struct trb *)skb->cb;
> + struct cldma_drv_info *drv_info;
> + struct cldma_dev *cd = dev;
> + struct queue_info *que;
> + struct tx_req *req;
> + struct txq *txq;
> + int ret;
> +
> + if (unlikely(!cd))
> + return -EINVAL;
> +
> + que = xa_load(&cd->trans->queue_tbl, trb->channel_id & 0xFFFF);
> + if (unlikely(!que))
> + return -EINVAL;
> + drv_info = cd->cldma_drv_info[que->hif_id];
> + if (unlikely(!drv_info))
> + return -EINVAL;
> +
> + txq = drv_info->txq[que->txqno];
> + if (unlikely(!txq))
> + return -EINVAL;
> +
> + spin_lock(&txq->ring_lock);
> +
> + if (!atomic_read(&txq->req_budget)) {
> + spin_unlock(&txq->ring_lock);
> + return -EAGAIN;
> + }
> +
> + req = txq->req_pool + txq->wr_idx;
> + req->gpd->tx_gpd.debug_id = 0x01;
> + ret = mtk_cldma_txbuf_set(drv_info, skb, req);
> + if (ret) {
> + spin_unlock(&txq->ring_lock);
> + return ret;
> + }
> +
> + req->gpd->tx_gpd.data_buff_len = cpu_to_le16(skb->len);
> +
> + req->data_len = skb->len;
> + req->skb = skb;
> +
> + dma_wmb(); /* ensure req and data msg set done before HWO setup */
> +
> + req->gpd->tx_gpd.gpd_flags |= CLDMA_GPD_FLAG_HWO;
> +
> + txq->wr_idx = (txq->wr_idx + 1) % txq->nr_gpds;
> + atomic_dec(&txq->req_budget);
> +
> + spin_unlock(&txq->ring_lock);
> +
> + return 0;
> +}
> +
> +int mtk_cldma_get_tx_budget(void *dev, enum mtk_hif_id hif_id, u32 qno)
> +{
> + struct cldma_drv_info *drv_info;
> + struct cldma_dev *cd = dev;
> + struct txq *txq;
> +
> + if (unlikely(hif_id >= NR_CLDMA || qno >= HW_QUE_NUM || !cd))
> + return -EINVAL;
> +
> + drv_info = cd->cldma_drv_info[hif_id];
> + if (!drv_info)
> + return -EINVAL;
> + txq = drv_info->txq[qno];
> + if (!txq)
> + return -EINVAL;
> + return atomic_read(&txq->req_budget);
> +}
> +
> +static int (*trb_act_tbl[TRB_CMD_MAX])(struct cldma_dev *cd, struct sk_buff *skb) = {
> + [TRB_CMD_ENABLE] = mtk_cldma_open,
> + [TRB_CMD_TX] = mtk_cldma_tx,
> + [TRB_CMD_DISABLE] = mtk_cldma_close,
> +};
> +
> +int mtk_cldma_trb_process(void *dev, struct sk_buff *skb)
> +{
> + struct cldma_dev *cd;
> + struct trb *trb;
> +
> + if (!dev || !skb)
> + return -EINVAL;
> +
> + cd = (struct cldma_dev *)dev;
> + trb = (struct trb *)skb->cb;
> +
> + if (!(trb->cmd > TRB_CMD_MIN && trb->cmd < TRB_CMD_STOP))
> + return -EINVAL;
> +
> + return trb_act_tbl[trb->cmd](cd, skb);
> +}
> +
> +int mtk_cldma_check_ch_cfg(void *dev, struct queue_info *que)
> +{
> + struct cldma_drv_info *drv_info;
> + struct cldma_dev *cd = dev;
> + struct mtk_md_dev *mdev;
> + struct txq *txq;
> + struct rxq *rxq;
> +
> + mdev = cd->trans->mdev;
> + drv_info = cd->cldma_drv_info[que->hif_id];
> +
> + if (!drv_info) {
> + dev_err(mdev->dev, "CLDMA%d has not been initialized\n",
> + mtk_cldma_hw_id_tbl[que->hif_id]);
> + return -EINVAL;
> + }
> +
> + txq = drv_info->txq[que->txqno];
> + rxq = drv_info->rxq[que->rxqno];
> + if (!txq || !rxq) {
> + dev_err(mdev->dev,
> + "CLDMA%d txq%u rxq%u has not been enabled\n",
> + mtk_cldma_hw_id_tbl[que->hif_id], que->txqno, que->rxqno);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
[Severity: High]
Can this restart the RX engine on a descriptor the device has already
completed?
mtk_cldma_rx_done_work() leaves the descriptor loop when the slot at
rxq->free_idx still has CLDMA_GPD_FLAG_HWO set. In that case
mtk_cldma_check_rx_req() returns -EAGAIN because SO_CUR_ADDR points at
that slot.
Between that check and mutex_lock(&drv_info->rearm_lock), the device
can finish the packet and clear HWO. Another context can then take
rearm_lock to reset the instance. Possible callers are the -ETIMEDOUT
paths in mtk_cldma_txq_free() and mtk_cldma_rxq_free() for another
queue on the same instance, or mtk_cldma_hw_recovery(). The worker may
also just be blocked on rearm_lock while the reset runs. Either way,
mtk_cldma_rearm_queues() sets need_restart and queues rx_done_work
again.
When the worker gets the lock, it consumes need_restart. Then
mtk_cldma_rxq_restart() programs SO_START_ADDR with
req_pool[free_idx].gpd_dma_addr and issues SO START. Nothing checks
HWO on that slot again, so the engine stops at once on an HWO=0
descriptor.
The re-queued pass then reaps that slot. Its arm_slot step sets HWO on
the previous slot and leaves the reaped one as the new HWO=0 tail. That
is where the device cursor is parked.
The next slot never completes, so mtk_cldma_check_rx_req() times out
with "Failed to check HWO=0" and returns -EAGAIN. need_restart is
already 0, so only mtk_cldma_resume_queue() is issued. As the comment
in mtk_cldma_rearm_queues() notes, RESUME never moves the cursor past
an HWO=0 descriptor. No further XFER_DONE comes to run the worker.
That seems to leave the RX queue wedged until the channel is closed.
On the MD and sAP control channels, that would stop all handshake
messages from the device.
Should the worker re-check the slot at free_idx after taking
rearm_lock? If the device has completed it, the worker could set
need_restart again and loop to reap it before restarting. After a
reset, the RX engine should stay idle until mtk_cldma_rxq_restart()
issues START, so that check would be stable under the lock.
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261105-t9xx_driver_v1-v11-0-2084aa512864%40compal.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v11 4/6] net: wwan: t9xx: Add control port
2026-11-05 7:56 ` [PATCH v11 4/6] net: wwan: t9xx: Add control port Jack Wu via B4 Relay
@ 2026-11-10 11:03 ` netdev-bot+sashiko
0 siblings, 0 replies; 12+ messages in thread
From: netdev-bot+sashiko @ 2026-11-10 11:03 UTC (permalink / raw)
To: jackbb_wu
Cc: loic.poulain, ryazanov.s.a, johannes, andrew+netdev, davem,
edumazet, kuba, pabeni, wen-zhi.huang, shi-wei.yeh, Minano.tseng,
matthias.bgg, angelogioacchino.delregno, horms, corbet, skhan,
robert_yu, Jeff_Chang, linux-kernel, netdev, linux-arm-kernel,
linux-mediatek, linux-doc
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] The struct mtk_port comment in drivers/net/wwan/t9xx/mtk_port.h
(lines 83-86) says tx_seq and rx_seq "are reset while the channel is…
- [Low] mtk_port_rx_dispatch() (drivers/net/wwan/t9xx/mtk_port.c:408-409)
picks the destination port from the CCCI header channel, which the…
- [Low] Concern: mtk_port_status_update() and the port->enable field it
writes are dead code at this commit and are not mentioned in the…
- [Low] Concern: mtk_port_internal_enable() sends a compensating DISABLE
for every ENABLE failure other than -EBUSY, including failures before…
- [Low] Concern: mtk_port_send_data() never checks skb->len against
port->tx_mtu, and mtk_cldma_submit_tx() writes cpu_to_le16(skb->len)…
- [Low] Concern: mtk_port_internal_close() runs
test_and_clear_bit(PORT_S_OPEN) and then mtk_port_common_close() runs…
- [Low] Concern: the radix-tree port table has no consistent lock or RCU
scheme.
--
net: wwan: t9xx: Add control port
This patch adds the t9xx control port layer. A port manager builds a
radix-tree port table and passes TRBs between port I/O and CLDMA, and an
internal port type provides open/close/write/recv_register. struct trb
gets a kref so mtk_ctrl_trb_handler() can hold a reference across a
dispatch, and TX sequence numbers are stamped at hardware hand-off.
> diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
> new file mode 100644
> index 0000000000000..7fec696bd6db1
> --- /dev/null
> +++ b/drivers/net/wwan/t9xx/mtk_port.c
> @@ -0,0 +1,667 @@
[ ... ]
> +static int mtk_port_rx_dispatch(struct sk_buff *skb, void *priv)
> +{
[ ... ]
> + channel = FIELD_GET(MTK_HDR_FLD_CHN, le32_to_cpu(ccci_h->status));
> + /* A CLDMA instance carries the channels of one peer only (sAP or MD),
> + * the peer of the port that armed this queue.
> + */
> + if (MTK_PEER_ID(channel) == MTK_PEER_ID(port->info.rx_ch))
> + port = mtk_port_search_by_id(port_mngr, channel);
> + else
> + port = NULL;
> + if (unlikely(!port)) {
> + dev_warn_ratelimited(port_mngr->ctrl_blk->mdev->dev,
> + "Failed to find port by channel:%d\n", channel);
> + goto drop_data;
> + }
> +
> + mtk_port_check_rx_seq(port, ccci_h);
> + port->rx_seq = FIELD_GET(MTK_HDR_FLD_SEQ, le32_to_cpu(ccci_h->status));
[Severity: Low]
Should this also check that the channel belongs to the queue the frame
arrived on?
The destination port is taken from the channel field of the CCCI header,
and the device writes that field. The only check is that its peer ID
matches the peer of the port that armed this RX queue.
Later in the series the MD peer has three ports, each on its own queue:
MDCTRL on CLDMA1 RXQ0, MBIM on CLDMA1 RXQ2 and AT on CLDMA1 RXQ5. If a
frame arrives on the MBIM or AT queue carrying channel CCCI_CONTROL_RX,
it goes to the MDCTRL internal port's ->recv().
Each queue has its own CLDMA RX work item on a WQ_UNBOUND workqueue. Two
queues' workers can therefore run mtk_port_check_rx_seq(), write
port->rx_seq and call ->recv() for the same port at the same time. That
breaks the rule in struct mtk_port that rx_seq is written "only from the
rx_done callback of the one queue serving this channel". That rule is
the reason rx_seq has no lock.
Could the frame be dropped unless MTK_CH_ID(channel) matches the arming
port's channel, or unless the channel's queue matches the receiving
queue? t7xx does this by matching rxq_index.
[ ... ]
> +int mtk_port_ch_enable(struct mtk_port *port)
> +{
> + struct mtk_port_mngr *port_mngr = port->port_mngr;
> + struct trb_open_priv *trb_open_priv;
> + struct sk_buff *skb;
> + struct trb *trb;
> + int ret;
> +
> + /* The modem restarts its sequence numbering with the channel, and
> + * the first packet after it is not checked.
> + */
> + port->tx_seq = 0;
> + port->rx_seq = MTK_INVALID_RX_SEQ;
[Severity: Low]
Is the channel always closed when these counters are reset? The comment
in struct mtk_port in mtk_port.h depends on it:
/* tx_seq is stamped and advanced only at hardware hand-off, by the one
* thread serving this port's TX queue; rx_seq only from the rx_done
* callback of the one queue serving this channel. Both are reset
* while the channel is disabled, so neither needs a lock.
*/
These are plain stores, made before the ENABLE is submitted. Nothing
checks that the CLDMA queue is closed. I can see two ways the queue can
still be live at this point.
The first is a retry after a timeout. On -ETIMEDOUT,
mtk_port_internal_enable() does this:
ret = mtk_port_ch_enable(port);
if (ret && ret != -EBUSY) {
...
mtk_port_ch_disable(port);
clear_bit(PORT_S_ENABLE, &port->status);
return;
}
The ENABLE trb is still queued, with the DISABLE behind it. With "Add
FSM thread" later in the series, the handshake then fails in
mtk_fsm_ctrl_ch_start()->mtk_port_internal_open() with -ENODEV. The
device sends HS1 again, and mtk_port_mngr_fsm_state_handler_late() calls
->enable() a second time.
Suppose the trb thread has already run the first ENABLE. The RX queue is
then armed with rxq->arg = port, and the DISABLE has not closed it yet.
mtk_port_rx_dispatch() can then write port->rx_seq while this function
resets it.
The second is -EBUSY, which mtk_port_ch_enable() treats as success.
Suppose the queue was left open, for example because an earlier DISABLE
submit failed with -ENOMEM. Both counters are then reset while the queue
is live. The trb thread may still be running mtk_port_tx_seq_stamp() and
mtk_port_tx_seq_advance() on queued TX packets.
rx_seq only feeds the rate-limited warning in mtk_port_check_rx_seq(),
so the visible effect is a spurious or missed out-of-order warning.
KCSAN would still report the race.
Could the guarantee in the comment be enforced, or could the comment be
reworded to match what the code actually ensures?
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261105-t9xx_driver_v1-v11-0-2084aa512864%40compal.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread
2026-11-05 7:56 ` [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread Jack Wu via B4 Relay
@ 2026-11-10 11:03 ` netdev-bot+sashiko
0 siblings, 0 replies; 12+ messages in thread
From: netdev-bot+sashiko @ 2026-11-10 11:03 UTC (permalink / raw)
To: jackbb_wu
Cc: loic.poulain, ryazanov.s.a, johannes, andrew+netdev, davem,
edumazet, kuba, pabeni, wen-zhi.huang, shi-wei.yeh, Minano.tseng,
matthias.bgg, angelogioacchino.delregno, horms, corbet, skhan,
robert_yu, Jeff_Chang, linux-kernel, netdev, linux-arm-kernel,
linux-mediatek, linux-doc
Thank you for your contribution! Sashiko AI review found 6 potential
issue(s) to consider.
Critical: 0 · High: 1 · Medium: 2 · Low: 3
If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).
- [High] net: wwan: t9xx: late is_stopping check in
mtk_cldma_start_xfer()
- [Medium] When READY is refused, mtk_fsm_startup_act() has no way to
recover.
- [Medium] In mtk_fsm_feature_set_match(), the
RTFT_TYPE_SUPPORT_BACKWARD_COMPAT case (line 163) compares only…
- [Low] mtk_fsm_parse_hs2_msg() runs the runtime-feature actions before
the HS2 framing has been checked.
- [Low] Two comments say the FSM kthread is "parked", but nothing in this
code ever parks it.
- [Low] When mtk_cldma_err_tx_one() cannot stop a TX queue
(mtk_cldma_stop_queue returns something other than 0 or -ENODEV), it…
--
net: wwan: t9xx: Add FSM thread
This commit adds an FSM kthread to the t9xx WWAN driver and wires the
HS1/HS2/HS3 runtime feature handshake into it. CLDMA bring-up and teardown
are driven from FSM state transitions, including registering the IRQ and
adding err_work for queue error recovery.
> diff --git a/drivers/net/wwan/t9xx/mtk_fsm.c b/drivers/net/wwan/t9xx/mtk_fsm.c
> new file mode 100644
> index 0000000000000..3350f6df5064a
> --- /dev/null
> +++ b/drivers/net/wwan/t9xx/mtk_fsm.c
> @@ -0,0 +1,1085 @@
[ ... ]
> +static int mtk_fsm_feature_set_match(enum runtime_feature_support_type *cur_ft_spt,
> + struct runtime_feature_info rtft_info_st,
> + struct runtime_feature_info rtft_info_cfg)
> +{
[ ... ]
> + case RTFT_TYPE_SUPPORT_BACKWARD_COMPAT:
> + /* Decided on the device's declaration alone: the host's
> + * configured type can be the older of the two, and gating on
> + * it would keep off a feature the device just called
> + * compatible.
> + */
> + if (FIELD_GET(FEATURE_VER, rtft_info_st.feature) >=
> + FIELD_GET(FEATURE_VER, rtft_info_cfg.feature))
> + *cur_ft_spt = RTFT_TYPE_MUST_SUPPORT;
> + else
> + *cur_ft_spt = RTFT_TYPE_NOT_EXIST;
> + break;
[Severity: Medium]
Should this case also check the host's configured type?
Every host query entry is version 0. The entries the host never configured
are zero, because the fsm comes from devm_kzalloc(). So the version test is
always true. Any feature the device marks BACKWARD_COMPAT gets promoted to
RTFT_TYPE_MUST_SUPPORT, even when the host set it to NOT_EXIST or
NOT_SUPPORT. The MUST_SUPPORT case returns -EPROTO for that host config,
and the OPTIONAL case downgrades it.
In mtk_fsm_hs_info_init_by_hsid(), the MD handshake leaves
query_ft_set[QUERY_RTFT_ID_SAP_PORT_ENUM] as NOT_EXIST. The SAP handshake
leaves query_ft_set[QUERY_RTFT_ID_MD_PORT_ENUM] as NOT_EXIST. Both slots
have a handler in query_rtft_action[]. So mtk_fsm_parse_hs2_msg() would
run mtk_port_status_update() on port data the host never asked for:
if (cur_ft_spt == RTFT_TYPE_MUST_SUPPORT && query_rtft_action[ft_id]) {
if (!data_len) {
...
ret = -EPROTO;
break;
}
If that entry has data_len == 0, the whole HS2 fails with -EPROTO. No HS3
is sent and the FSM never reaches READY.
The comment says the host's type is ignored because it "can be the older
of the two". That is about versions. Does it really cover a host that
never queried the feature, or one that explicitly refused it?
> +static int (*query_rtft_action[FEATURE_CNT])(struct mtk_md_dev *mdev,
> + void *rt_data, u32 data_len) = {
> + [QUERY_RTFT_ID_MD_PORT_ENUM] = mtk_port_status_update,
> + [QUERY_RTFT_ID_SAP_PORT_ENUM] = mtk_port_status_update,
> +};
[ ... ]
> +static int mtk_fsm_parse_hs2_msg(struct mtk_md_fsm *fsm, struct fsm_hs_info *hs_info,
> + struct sk_buff *skb)
> +{
[ ... ]
> + rt_data = skb->data;
> + rt_data_len = skb->len;
> +
> + offset = sizeof(struct feature_query);
> + for (ft_id = 0; ft_id < FEATURE_CNT; ft_id++) {
[Severity: Low]
This skips the feature_query header without checking head_pattern or
tail_pattern. It then runs query_rtft_action[] on the entries, and
mtk_port_status_update() writes port->enable. The FEATURE_QUERY_PATTERN
check on the same skb only runs later:
mtk_fsm_startup_act()
mtk_fsm_parse_hs2_msg() <- actions applied
mtk_fsm_send_hs3_msg()
mtk_fsm_append_rtft_entries() <- -EPROTO on bad patterns
So an HS2 with a bad header but valid entries updates the port table
first, and only then is rejected. mtk_fsm_startup_act() goes to hs_err
and re-arms the channel. The port enable state from the rejected message
stays in place, and any port the retried HS2 does not list keeps that
value.
Could the pattern check move into mtk_fsm_parse_hs2_msg(), before any
action runs?
[ ... ]
> +static int mtk_fsm_startup_act(struct mtk_md_fsm *fsm, struct mtk_fsm_evt *event)
> +{
[ ... ]
> + hs_done_flag = READ_ONCE(fsm->hs_done_flag);
> + if (((fsm->fsm_flag | event->fsm_flag) & hs_done_flag) == hs_done_flag) {
> + int user;
> +
> + for (user = 0; user < MTK_FSM_HIF_USER_MAX; user++) {
> + if (fsm->hif_err[user]) {
> + dev_err(mdev->dev,
> + "Refusing READY: HIF user %d init failed with %d\n",
> + user, fsm->hif_err[user]);
> + return fsm->hif_err[user];
> + }
> + }
[Severity: Medium]
Once READY is refused here, how does the FSM recover?
This returns directly and does not go through hs_err, so no handshake
channel is re-armed. Both channels were masked by their HS1 handlers. No
further STARTUP can arrive, and the FSM stays in BOOTUP until the driver
is rebound.
One way to get here is a DEV_CFG_MD_ONLY device. mtk_fsm_idle_evt_handler()
sets hs_done_flag to the MD bits only, but it still unmasks both channels:
for (hs_id = 0; hs_id < HS_ID_MAX; hs_id++)
mtk_fsm_unmask_ext_evt(fsm, fsm->hs_info[hs_id].mhccif_ch);
If a SAP HS1 arrives and mtk_cldma_dev_init(CLDMA0) fails,
hif_err[MTK_FSM_HIF_USER_CLDMA0] stays set. When the MD handshake
completes, READY is refused for good, even though SAP is not required in
that configuration. Can firmware send a SAP HS1 in MD-only mode?
Separately, the comment above mtk_fsm_hif_err_record() says:
so a retried init that succeeds clears the failure it recorded before.
The MTK_FSM_HIF_USER_CTRL slot is only written by
mtk_ctrl_trans_fsm_state_handler(), on the transition to FSM_STATE_ON.
mtk_fsm_dev_add_act() allows that transition only once, from
FSM_STATE_INVALID. Is there any retry path for that slot?
The commit message says "the FSM advances to READY only after both
complete". It does not mention the MD-only case or the hif_err gate.
[ ... ]
> +err_unregister_evt:
> + mtk_pci_unregister_ext_evt(mdev, DEV_EVT_D2H_BOOT_FLOW_SYNC);
> +err_stop_thread:
> + /* mdev->fsm is never published on failure, so mtk_fsm_exit() cannot
> + * stop the parked kthread for us; it must be stopped here.
> + */
> + kthread_stop(fsm->fsm_handler);
> + put_task_struct(fsm->fsm_handler);
> + return ret;
> +}
[Severity: Low]
This isn't a bug, but this thread is not parked. It was only created by
kthread_create() and never woken, and nothing in mtk_fsm.c calls
kthread_park().
The comment in mtk_pci_dev_exit() has the same problem:
/* Close the event gate and park the FSM thread before the transport
* plane it drives is torn down.
*/
mtk_fsm_stop(mdev);
mtk_fsm_stop() clears fsm->fsm_handler and calls kthread_stop() and
put_task_struct(). The thread is terminated, not parked.
[ ... ]
> diff --git a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
> index d8763cdd67dac..4893edfebf79e 100644
> --- a/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
> +++ b/drivers/net/wwan/t9xx/pcie/mtk_cldma.c
[ ... ]
> @@ -967,15 +1318,94 @@ static void mtk_cldma_txq_flush(struct cldma_drv_info *drv_info,
[ ... ]
> +static void mtk_cldma_err_tx_one(struct cldma_drv_info *drv_info, u32 qno)
> +{
> + struct txq *txq;
> + int ret;
> +
[ ... ]
> + spin_lock(&txq->ring_lock);
> + txq->is_stopping = true;
> + spin_unlock(&txq->ring_lock);
> +
> + ret = mtk_cldma_stop_queue(drv_info, DIR_TX, qno);
> + /* -ENODEV: the link is dead, nothing is walking the ring */
> + if (ret && ret != -ENODEV) {
> + /* the device may still be walking the ring: unmapping its
> + * buffers here would leave it writing into unmapped memory.
> + * is_stopping is left set so nothing submits to it again,
> + * and QUEUE_ERROR masked until an instance re-arm.
> + */
> + dev_err(drv_info->mdev->dev,
> + "TX queue %u stop failed (%d), keeping its requests\n",
> + qno, ret);
> + return;
> + }
[Severity: High]
The is_stopping test only runs after the start address has been read.
If that address was zero, mtk_cldma_hw_recovery() has also already
re-armed the whole instance by this point. Shouldn't a queue that
err_work owns be turned away before any of that?
Take a queue whose stop timed out in mtk_cldma_err_tx_one(). It keeps
is_stopping set and tx_started true. Later, a stop timeout in another
queue's mtk_cldma_txq_free() or mtk_cldma_rxq_free() resets the
instance. mtk_cldma_rearm_queues() then skips this queue on purpose and
leaves its start address at 0. The next mtk_cldma_tx() on it goes:
mtk_cldma_tx()
mtk_cldma_start_xfer()
mtk_cldma_get_tx_start_addr() <- reads 0
mtk_cldma_hw_recovery()
mtk_cldma_rearm_queues() <- instance re-armed again
returns -EIO <- tx_started is still true
mtk_cldma_stop_queue()
mtk_cldma_txq_flush(..., -EIO)
A U64_MAX read on a dead link takes the same early -EIO exit. The same
thing can also happen while mtk_cldma_err_tx_one() is still polling its
stop, if the device lost its queue state along with the error.
The error is -EIO, not -EPIPE, so the new -EPIPE check in
mtk_cldma_tx() does not catch it. mtk_cldma_tx() then stops the queue,
clears tx_started and flushes the ring itself. Meanwhile
mtk_cldma_err_tx_one() is either doing the same thing or has decided
to keep those requests. The flushed TRBs complete with -EIO, not the
-EPIPE that recovery promises. Nothing clears is_stopping afterwards,
so the queue keeps refusing new submissions.
There is also a gap in the other direction. If the stop inside
mtk_cldma_tx() times out, it keeps the requests just as
mtk_cldma_err_tx_one() does, but it never sets is_stopping. So
mtk_cldma_submit_tx() keeps publishing HWO descriptors into a ring the
device may still be walking. That is exactly what is_stopping was
added to prevent.
Would it work to test is_stopping under ring_lock before the start
address read and the recovery? The stop-timeout path in mtk_cldma_tx()
could then set is_stopping the same way err_work does.
[Severity: Low]
What clears is_stopping after this return? The only place that clears it
is the end of a later mtk_cldma_err_tx_one() pass whose stop succeeds, and
that pass needs a new QUEUE_ERROR.
Another queue's stop can later time out in mtk_cldma_txq_free() or
mtk_cldma_rxq_free(). The instance is then reset with
mtk_cldma_drv_reset(), so the engine is idle. But mtk_cldma_rearm_queues()
skips this queue:
if (txq->is_stopping) {
/* Being stopped: restarting it would let the
* device walk buffers the stopper is unmapping.
*/
} else if (...
It programs no start address and issues no START. It does not flush the
kept requests, and it does not clear the flag. QUEUE_ERROR is unmasked,
but a reset queue that is not started will not raise one, so err_work
never runs for it again.
After that, mtk_cldma_submit_tx() keeps returning -EPIPE. The kept TRBs
stay incomplete until the channel is closed via mtk_cldma_txq_free(),
even though the reset made it safe to flush them. Is that intended? The
comment's "until an instance re-arm" suggests the re-arm should recover
this queue.
> +
> + spin_lock(&txq->ring_lock);
> + WRITE_ONCE(txq->tx_started, false);
> + spin_unlock(&txq->ring_lock);
> + mtk_cldma_txq_flush(drv_info, txq, -EPIPE);
> +
> + spin_lock(&txq->ring_lock);
> + txq->is_stopping = false;
> + spin_unlock(&txq->ring_lock);
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261105-t9xx_driver_v1-v11-0-2084aa512864%40compal.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports
2026-11-05 7:56 ` [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports Jack Wu via B4 Relay
@ 2026-11-10 11:03 ` netdev-bot+sashiko
0 siblings, 0 replies; 12+ messages in thread
From: netdev-bot+sashiko @ 2026-11-10 11:03 UTC (permalink / raw)
To: jackbb_wu
Cc: loic.poulain, ryazanov.s.a, johannes, andrew+netdev, davem,
edumazet, kuba, pabeni, wen-zhi.huang, shi-wei.yeh, Minano.tseng,
matthias.bgg, angelogioacchino.delregno, horms, corbet, skhan,
robert_yu, Jeff_Chang, linux-kernel, netdev, linux-arm-kernel,
linux-mediatek, linux-doc
Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.
Low priority / nit picks for your consideration:
- [Low] This patch makes the header comment on enum mtk_port_status
(drivers/net/wwan/t9xx/mtk_port.h:39-41) wrong.
- [Low] The new WWAN TX path disagrees with the existing driver about
what port->tx_mtu means.
- [Low] In mtk_port_enable_by_type() (mtk_port.c:636-655), port->enable
is consumed and cleared only when FSM_STATE_READY is reached.
--
net: wwan: t9xx: Add AT & MBIM WWAN ports
This commit adds AT and MBIM WWAN port types to the t9xx port
infrastructure. The WWAN ports are created from the FSM READY handler,
and the commit adds start, stop, tx and tx_blocking operations. The tx
path splits each write into CCCI packets, capped at the HIF queue depth,
and reports TX back-pressure through wwan_port_txon()/txoff().
> diff --git a/drivers/net/wwan/t9xx/mtk_port.c b/drivers/net/wwan/t9xx/mtk_port.c
> index 101ec14fc9934..a0defeec9abf8 100644
> --- a/drivers/net/wwan/t9xx/mtk_port.c
> +++ b/drivers/net/wwan/t9xx/mtk_port.c
[ ... ]
> @@ -626,6 +633,27 @@ int mtk_port_ch_disable(struct mtk_port *port)
> return ret;
> }
>
> +static void mtk_port_enable_by_type(struct mtk_port_mngr *port_mngr, int tbl_type)
> +{
> + struct radix_tree_iter iter;
> + struct mtk_port *port;
> + void __rcu **slot;
> +
> + radix_tree_for_each_slot(slot, &port_mngr->port_tbl[tbl_type],
> + &iter, 0) {
> + port = mtk_port_tbl_slot(&slot, &iter);
> + if (!port)
> + continue;
> + /* Consumed, so a later boot acts only on what its own port
> + * enumeration advertised.
> + */
> + if (port->enable) {
> + port->enable = false;
> + mtk_port_get_ops(port->info.type)->enable(port);
> + }
[Severity: Low]
Can a stale port->enable reach READY here? mtk_port_status_update() only
writes port->enable for channels that appear in the current enumeration:
port->enable = FIELD_GET(MTK_INFO_FLD_EN, le16_to_cpu(port_info->channel));
Nothing resets the ports that are not listed.
In mtk_fsm_startup_act(), mtk_fsm_parse_hs2_msg() applies the MD port
enumeration first. If a later RTFT entry is malformed, or
mtk_fsm_send_hs3_msg() fails, the code takes hs_err and re-arms the
channel so the modem can retry the handshake:
ret = mtk_fsm_parse_hs2_msg(fsm, hs_info, skb);
if (!ret)
ret = mtk_fsm_send_hs3_msg(hs_info, skb);
...
hs_err:
if (hs_info)
mtk_pci_unmask_ext_evt(mdev, hs_info->mhccif_ch);
Suppose the retried enumeration leaves out a port that the first one
marked enabled. That port keeps port->enable == true. On READY,
mtk_port_wwan_enable() would then arm its CLDMA1 queue and call
wwan_create_port() for a channel the final enumeration did not advertise.
That goes against the comment above.
This depends on the modem firmware sending a different port list across
handshake retries, so it may not happen in practice. FSM_STATE_OFF is
terminal in this driver, so the handshake retry seems to be the only way
to get here, not a later boot.
Would clearing all port->enable flags before applying a new enumeration
close this gap?
> + }
> +}
[ ... ]
> diff --git a/drivers/net/wwan/t9xx/mtk_port_io.c b/drivers/net/wwan/t9xx/mtk_port_io.c
> index e92ce1cb7c00f..d5232cb150bff 100644
> --- a/drivers/net/wwan/t9xx/mtk_port_io.c
> +++ b/drivers/net/wwan/t9xx/mtk_port_io.c
[ ... ]
> +static int mtk_port_common_write(struct mtk_port *port, struct sk_buff *src, bool blocking)
> +{
> + u32 packet_size, left_cnt = src->len, cur_pos;
> + u32 payload_mtu, sent_len = 0;
> + struct sk_buff_head list;
> + bool force_send = false;
> + struct sk_buff *skb;
> + int ret;
> +
> + ret = mtk_port_status_check(port);
> + if (ret)
> + return ret;
> +
> + payload_mtu = port->tx_mtu - sizeof(struct mtk_ccci_header);
[Severity: Low]
Is the CCCI header subtracted twice here? mtk_port_ch_enable() already
removes it from port->tx_mtu:
port->tx_mtu = trb_open_priv->tx_mtu - MTK_CCCI_H_ELEN;
mtk_port.h documents MTK_CCCI_H_ELEN as:
/* ccci header length + reserved space that is used in exception flow */
#define MTK_CCCI_H_ELEN (128)
So port->tx_mtu already looks like the payload budget. This code and
caps.frag_len in mtk_port_wwan_enable() both subtract
sizeof(struct mtk_ccci_header) a second time. Each packet then carries
16 bytes less payload: with Q_MTU_3_5K, frag_len is 3440 instead of 3456.
The error is in the safe direction, so nothing overflows. Still, this
math, the "payload budget" comment in mtk_port_wwan_enable(), and the
MTK_CCCI_H_ELEN definition can't all be right. Which meaning of
port->tx_mtu is intended?
> + if (src->len > MTK_PORT_MAX_TX_PKT * payload_mtu)
> + return -EMSGSIZE;
[ ... ]
> +static void mtk_port_wwan_enable(struct mtk_port *port)
> +{
[ ... ]
> + /* These bits must be set before wwan_create_port(): the device node
> + * becomes openable inside it and mtk_port_common_open() rejects a
> + * port without PORT_S_ENABLE.
> + */
> + set_bit(PORT_S_WR, &port->status);
> + set_bit(PORT_S_ENABLE, &port->status);
[Severity: Low]
This isn't a functional bug, but does this patch make the comment on enum
mtk_port_status in mtk_port.h inaccurate?
/* State bits, not locks: callers serialise ->enable(), ->disable() and
* open of a port. The atomic bitops only make a repeated call a no-op.
*/
For the WWAN ports, open comes from userspace through the WWAN core:
mtk_port_wwan_open() runs under the core's ops_lock. ->enable() and
->disable() come from the FSM through mtk_port_mngr_fsm_state_handler().
Nothing serialises the two paths, and the comment above set_bit()
here expects open to run at the same time as enable.
The open vs. enable and open vs. disable interleavings look safe because
of the w_lock ordering, but the header comment no longer describes how
this works. Could it be updated so later code doesn't count on caller
serialisation that doesn't exist?
> +
> + wp = wwan_create_port(port_mngr->ctrl_blk->mdev->dev,
> + port->w_priv.w_type,
> + &wwan_ops, &caps, port);
[ ... ]
--
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261105-t9xx_driver_v1-v11-0-2084aa512864%40compal.com
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-11-10 11:04 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-05 7:56 [PATCH v11 0/6] net: wwan: t9xx: Add MediaTek T9XX WWAN driver Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 1/6] net: wwan: t9xx: Add PCIe core Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 2/6] net: wwan: t9xx: Add control plane transaction layer framework Jack Wu via B4 Relay
2026-11-05 7:56 ` [PATCH v11 3/6] net: wwan: t9xx: Add control DMA interface Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 4/6] net: wwan: t9xx: Add control port Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 5/6] net: wwan: t9xx: Add FSM thread Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
2026-11-05 7:56 ` [PATCH v11 6/6] net: wwan: t9xx: Add AT & MBIM WWAN ports Jack Wu via B4 Relay
2026-11-10 11:03 ` netdev-bot+sashiko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®