Re: [RFC PATCH net-next 01/28] docs: networking: ti: add driver doc for CPSW Proxy Client

[Siddharth Vadapalli <s-vadapalli@ti.com>]

On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote:

Andrew,

I have spent time going through your feedback, trying to understand your
suggestions. This email is the complete reply corresponding to my earlier
reply at:
https://lore.kernel.org/r/0b0c1b07-756e-439e-bfc5-53824fd2a61c@ti.com/
which was simply meant to serve as an acknowledgement that I have seen
your email.

> On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote:
> > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on
> > a remote core to enable Ethernet functionality for applications running
> > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW
> > Ethernet Switch on the SoC and acts as the Server, offering services to
> > Clients running on various cores.
> 
> I'm not sure we as a community what this architecture. We want Linux
> to be driving the hardware, not firmware. So expect linux to be
> running the server.

Due to the use-case requirements, Linux cannot be the server. Some of
the requirements are:
1. Fast startup and configuration of CPSW independent of Linux and Other
OS running on any of the cores on the SoC. The configuration of CPSW has
to be performed in parallel while the Bootloader starts Linux.
2. CPSW has to be functional and configurable even when Linux has been
suspended. One of the non-Linux Clients happens to be the AUTOSAR Client
which continues exchanging network data via CPSW even when Linux has
been suspended. So the server has to be functional even then, in order
to cater to the AUTOSAR Client's requests to configure CPSW. CPSW's
configuration is not static in the sense that it gets programmed and
will no longer be modified. Therefore the server has to be functional at
all times to update CPSW's configuration based on the demands of any of
the Clients.

For more details about the Ethernet Switch Firmware (EthFw) and the set
of Clients running on remote cores, please refer:
https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_02_00_05/exports/docs/ethfw/docs/user_guide/ethfw_c_ug_top.html#ethfw_remote_clients

> 
> > +The "am65-cpsw-nuss.c" driver in Linux at:
> > +drivers/net/ethernet/ti/am65-cpsw-nuss.c
> > +provides Ethernet functionality for applications on Linux.
> > +It also handles both the control-path and data-path, namely:
> > +Control => Configuration of the CPSW Peripheral
> > +Data => Configuration of the DMA Channels to transmit/receive data
> 
> So nuss is capable of controlling the hardware. nuss has an upper
> interface which is switchdev, and a lower interface which somehow acts
> on the hardware, maybe invoking RPCs into the firmware?

There are no RPCs used by the "am65-cpsw-nuss.c" driver. It assumes that
it is the only user of CPSW Ethernet Switch. It doesn't interface with
any firmware. Based on the switchdev framework, it receives commands
from userspace which it then uses to directly write to CPSW's registers.

> 
> So it is not too big a step to put the server functionality in Linux,
> on top of the nuss driver.

Maybe it isn't a big step but it doesn't help with the use-case that I
have described above. For that reason, while it might be a "good to have"
feature, it is not solving the problem.

> 
> Now take a step backwards. This concept of a switch with different
> CPUs attached to it is nothing special.
> 
> Some Marvell switches have a Z80 connected to a dedicated port of the
> switch. You can run applications on that Z80. Those applications might
> be as simple as spanning tree, so you can have a white box 8 port
> ethernet switch without needing an external CPU. But there is an SDK,
> you could run any sort of application on the Z80.
> 
> The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe
> interface. Linux can control the switch via the PCIe interface, but
> there could also be applications running on the Cortex.
> 
> Look at the Broadcom BCM89586M:
> https://docs.broadcom.com/doc/89586M-PB
> 
> It is similar to the microchip device, an M7 CPU, and a PCIe
> interface. It seems like a Linux host could be controlling the switch
> via PCIe, and applications running on the M7.
> 
> I expect there are more examples, but you get the idea.

I have gone through the examples above. All of them are referring to the
Hardware Capabilities of the Ethernet Switch, which aren't applicable to
the CPSW Ethernet Switch. I am listing why each of them isn't applicable:
1. Marvel Z80 Switch:
I assume that you are referring to:
https://wiki.espressobin.net/tiki-index.php?page=Topaz+Switch
with the "Integrated 200MHz Z80 microprocessor". CPSW doesn't have an
embedded microprocessor dedicated to programming it. The closest it
could get to the Z80 is the external R5 Core running EthFw as far as
configuring the Switch is concerned. But how does it handle the use-case
where there are applications running simultaneously on different cores,
all of which require Ethernet Functionality with the same Ethernet
Switch, in a dynamic manner?
2. Microchip LAN996x:
CPSW doesn't have a PCIe interface.
3. Broadcom BCM89586M:
Again, CPSW doesn't have a PCIe interface.

An important point to note is that all applications you have mentioned
are running on a single core. The current framework being proposed to
solve the problem is for the use-case where there are applications
running across various cores with different criticality (not all
applications may be running all the time, Linux for example will be
suspended as well).

> 
> A completely different angle to look at is VF/PF and eswitches. A
> server style CPU running virtual machines, a VM getting a VF passed
> through to it. This is not something i know much about, so we might
> need to pull in some data centre specialists. But we have a different
> CPU, a virtual CPU, making use of part of the switch. Its the same
> concept really.

CPSW doesn't support SR-IOV. However, if you are referring to modelling
CPSW as an SR-IOV capable Ethernet Switch by having EthFw pose as the
Driver for the "Virtual" Physical Function of CPSW, with each Client
Driver mapping to one of the modelled "Virtual" Virtual Functions
exposed by EthFw, then yes, I will spend time looking at how that could
be implemented. The term "Virtual" has been added in the previous
sentence to clarify that CPSW isn't truly SR-IOV capable and we are
simply making it look that way via EthFw. Even in SR-IOV, the
communicatoin between PF and VF drivers happens via Hardware Mailbox
which means RPMsg is coming back into the picture. The current
implementation also is using RPMsg to exchange control information
between EthFw and all the Clients.

> 
> My main point is, please start with an abstract view of the problem. A
> lot of the solution should be generic, it can be applied to all these
> devices. And then there probably needs to be a small part which is
> specific to TI devices. It could be parts of the solutions already
> exist, e.g. VF/PF have port represents, which might be a useful
> concept here as well. Switchdev exists and provides a generic
> interface for configuring switches...

The summary of the problem statement is:
We require a framework in which the Ethernet Switch (CPSW) has to be
shared across the applications running on different cores of the SoC.
Since CPSW doesn't have an Integrated Processor, some core on the SoC
has to act as the Central Entity which is responsible for arbitrating
configuration requests from various cores and performing the appropriate
configuration of CPSW. Additionally, apart from performing configuration
of CPSW, the Central Entity is also responsible for allocating resources
to different cores, including DMA Channels/Flows (There are 8 TX DMA
Channels which have to be split across different cores to allow each
core to send traffic to CPSW for example). CPSW should be functional
even when some of the Clients (including Linux) might be suspended for
Low Power use-case. The forwarding path of CPSW should be functional
within 100s of milliseconds after the Bootloader stage.

Kindly share your feedback on possible implementations to address the
problem summarized above. Thank you for sharing your valuable feedback
so far on this series.

Regards,
Siddharth.