RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> Sent: Wednesday, February 10, 2021 12:54 PM
> To: netdev@vger.kernel.org
> Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>; pierre-
> louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> devices/dlb.rst
> new file mode 100644
> index 000000000000..aa79be07ee49
> --- /dev/null
> +++ b/Documentation/misc-devices/dlb.rst
> @@ -0,0 +1,259 @@
> +.. SPDX-License-Identifier: GPL-2.0-only
> +
> +===========================================
> +Intel(R) Dynamic Load Balancer Overview
> +===========================================
> +
> +:Authors: Gage Eads and Mike Ximing Chen
> +
> +Contents
> +========
> +
> +- Introduction
> +- Scheduling
> +- Queue Entry
> +- Port
> +- Queue
> +- Credits
> +- Scheduling Domain
> +- Interrupts
> +- Power Management
> +- User Interface
> +- Reset
> +
> +Introduction
> +============
> +
> +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> +provides load-balanced, prioritized scheduling of core-to-core communication.
> +
> +Intel DLB is an accelerator for the event-driven programming model of
> +DPDK's Event Device Library[2]. The library is used in packet processing
> +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> +variety of packet distribution and synchronization schemes.
> +
> +Intel DLB device consists of queues and arbiters that connect producer
> +cores and consumer cores. The device implements load-balanced queueing
> features
> +including:
> +- Lock-free multi-producer/multi-consumer operation.
> +- Multiple priority levels for varying traffic types.
> +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> +- Simple unordered load-balanced distribution.
> +- Atomic lock free load balancing across multiple consumers.
> +- Queue element reordering feature allowing ordered load-balanced distribution.
> +

Hi Jakub/Dave,
This is a device driver for a HW core-to-core communication accelerator. It is submitted 
to "linux-kernel" for a module under device/misc. Greg suggested (see below) that we
also sent it to you for any potential feedback in case there is any interaction with
networking initiatives. The device is used to handle the load balancing among CPU cores
after the packets are received and forwarded to CPU. We don't think it interferes
with networking operations, but would appreciate very much your review/comment on this.
 
Thanks for your help.
Mike

> As this is a networking related thing, I would like you to get the
>proper reviews/acks from the networking maintainers before I can take
>this.
>
>Or, if they think it has nothing to do with networking, that's fine too,
>but please do not try to route around them.
>
>thanks,
>
>greg k-h

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[gregkh]

On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> 
> 
> > -----Original Message-----
> > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > Sent: Wednesday, February 10, 2021 12:54 PM
> > To: netdev@vger.kernel.org
> > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>; pierre-
> > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > 
> > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > devices/dlb.rst
> > new file mode 100644
> > index 000000000000..aa79be07ee49
> > --- /dev/null
> > +++ b/Documentation/misc-devices/dlb.rst
> > @@ -0,0 +1,259 @@
> > +.. SPDX-License-Identifier: GPL-2.0-only
> > +
> > +===========================================
> > +Intel(R) Dynamic Load Balancer Overview
> > +===========================================
> > +
> > +:Authors: Gage Eads and Mike Ximing Chen
> > +
> > +Contents
> > +========
> > +
> > +- Introduction
> > +- Scheduling
> > +- Queue Entry
> > +- Port
> > +- Queue
> > +- Credits
> > +- Scheduling Domain
> > +- Interrupts
> > +- Power Management
> > +- User Interface
> > +- Reset
> > +
> > +Introduction
> > +============
> > +
> > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > +
> > +Intel DLB is an accelerator for the event-driven programming model of
> > +DPDK's Event Device Library[2]. The library is used in packet processing
> > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > +variety of packet distribution and synchronization schemes.
> > +
> > +Intel DLB device consists of queues and arbiters that connect producer
> > +cores and consumer cores. The device implements load-balanced queueing
> > features
> > +including:
> > +- Lock-free multi-producer/multi-consumer operation.
> > +- Multiple priority levels for varying traffic types.
> > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > +- Simple unordered load-balanced distribution.
> > +- Atomic lock free load balancing across multiple consumers.
> > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > +
> 
> Hi Jakub/Dave,
> This is a device driver for a HW core-to-core communication accelerator. It is submitted 
> to "linux-kernel" for a module under device/misc. Greg suggested (see below) that we
> also sent it to you for any potential feedback in case there is any interaction with
> networking initiatives. The device is used to handle the load balancing among CPU cores
> after the packets are received and forwarded to CPU. We don't think it interferes
> with networking operations, but would appreciate very much your review/comment on this.

It's the middle of the merge window, getting maintainers to review new
stuff until after 5.12-rc1 is out is going to be a very difficult thing
to do.

In the meantime, why don't you all help out and review submitted patches
to the mailing lists for the subsystems you all are trying to get this
patch into.  I know maintainers would appreciate the help, right?

thanks,

greg k-h

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
> Sent: Thursday, February 18, 2021 2:53 AM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> Williams, Dan J <dan.j.williams@intel.com>; pierre-louis.bossart@linux.intel.com
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Sent: Wednesday, February 10, 2021 12:54 PM
> > > To: netdev@vger.kernel.org
> > > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>;
> pierre-
> > > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > > devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > > +variety of packet distribution and synchronization schemes.
> > > +
> > > +Intel DLB device consists of queues and arbiters that connect producer
> > > +cores and consumer cores. The device implements load-balanced queueing
> > > features
> > > +including:
> > > +- Lock-free multi-producer/multi-consumer operation.
> > > +- Multiple priority levels for varying traffic types.
> > > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > > +- Simple unordered load-balanced distribution.
> > > +- Atomic lock free load balancing across multiple consumers.
> > > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > > +
> >
> > Hi Jakub/Dave,
> > This is a device driver for a HW core-to-core communication accelerator. It is
> submitted
> > to "linux-kernel" for a module under device/misc. Greg suggested (see below) that
> we
> > also sent it to you for any potential feedback in case there is any interaction with
> > networking initiatives. The device is used to handle the load balancing among CPU
> cores
> > after the packets are received and forwarded to CPU. We don't think it interferes
> > with networking operations, but would appreciate very much your
> review/comment on this.
> 
> It's the middle of the merge window, getting maintainers to review new
> stuff until after 5.12-rc1 is out is going to be a very difficult thing
> to do.
> 
> In the meantime, why don't you all help out and review submitted patches
> to the mailing lists for the subsystems you all are trying to get this
> patch into.  I know maintainers would appreciate the help, right?
> 
> thanks,
> 
> greg k-h

Sure. I am a little new to the community and process, but will try to help.
Thanks
Mike

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: gregkh@linuxfoundation.org <gregkh@linuxfoundation.org>
> Sent: Thursday, February 18, 2021 2:53 AM
> To: Chen, Mike Ximing <mike.ximing.chen@intel.com>
> Cc: netdev@vger.kernel.org; Linux Kernel Mailing List <linux-
> kernel@vger.kernel.org>; davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> Williams, Dan J <dan.j.williams@intel.com>; pierre-louis.bossart@linux.intel.com
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Thu, Feb 18, 2021 at 07:34:31AM +0000, Chen, Mike Ximing wrote:
> >
> >
> > > -----Original Message-----
> > > From: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Sent: Wednesday, February 10, 2021 12:54 PM
> > > To: netdev@vger.kernel.org
> > > Cc: davem@davemloft.net; kuba@kernel.org; arnd@arndb.de;
> > > gregkh@linuxfoundation.org; Williams, Dan J <dan.j.williams@intel.com>;
> pierre-
> > > louis.bossart@linux.intel.com; Gage Eads <gage.eads@intel.com>
> > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > > devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > > +variety of packet distribution and synchronization schemes.
> > > +
> > > +Intel DLB device consists of queues and arbiters that connect producer
> > > +cores and consumer cores. The device implements load-balanced queueing
> > > features
> > > +including:
> > > +- Lock-free multi-producer/multi-consumer operation.
> > > +- Multiple priority levels for varying traffic types.
> > > +- 'Direct' traffic (i.e. multi-producer/single-consumer)
> > > +- Simple unordered load-balanced distribution.
> > > +- Atomic lock free load balancing across multiple consumers.
> > > +- Queue element reordering feature allowing ordered load-balanced distribution.
> > > +
> >
> > Hi Jakub/Dave,
> > This is a device driver for a HW core-to-core communication accelerator. It is
> submitted
> > to "linux-kernel" for a module under device/misc. Greg suggested (see below) that
> we
> > also sent it to you for any potential feedback in case there is any interaction with
> > networking initiatives. The device is used to handle the load balancing among CPU
> cores
> > after the packets are received and forwarded to CPU. We don't think it interferes
> > with networking operations, but would appreciate very much your
> review/comment on this.
> 
> It's the middle of the merge window, getting maintainers to review new
> stuff until after 5.12-rc1 is out is going to be a very difficult thing
> to do.
> 

Hi Jakub/Dave,
Just wonder if you had a chance to take a look at our patch. With the close of 5.12
merge window, we would like to get the process moving again.

> In the meantime, why don't you all help out and review submitted patches
> to the mailing lists for the subsystems you all are trying to get this
> patch into.  I know maintainers would appreciate the help, right?
> 
> thanks,
> 
> greg k-h

Did a few reviews last weekend, and will continue to help.

Thanks
Mike

RE: [PATCH v10 00/20] dlb: introduce DLB device driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Friday, May 14, 2021 10:33 AM
> To: Williams, Dan J <dan.j.williams@intel.com>
> > Hi Greg,
> >
> > So, for the last few weeks Mike and company have patiently waded
> > through my questions and now I think we are at a point to work through
> > the upstream driver architecture options and tradeoffs. You were not
> > alone in struggling to understand what this device does because it is
> > unlike any other accelerator Linux has ever considered. It shards /
> > load balances a data stream for processing by CPU threads. This is
> > typically a network appliance function / protocol, but could also be
> > any other generic thread pool like the kernel's padata. It saves the
> > CPU cycles spent load balancing work items and marshaling them through
> > a thread pool pipeline. For example, in DPDK applications, DLB2 frees
> > up entire cores that would otherwise be consumed with scheduling and
> > work distribution. A separate proof-of-concept, using DLB2 to
> > accelerate the kernel's "padata" thread pool for a crypto workload,
> > demonstrated ~150% higher throughput with hardware employed to manage
> > work distribution and result ordering. Yes, you need a sufficiently
> > high touch / high throughput protocol before the software load
> > balancing overhead coordinating CPU threads starts to dominate the
> > performance, but there are some specific workloads willing to switch
> > to this regime.
> >
> > The primary consumer to date has been as a backend for the event
> > handling in the userspace networking stack, DPDK. DLB2 has an existing
> > polled-mode-userspace driver for that use case. So I said, "great,
> > just add more features to that userspace driver and you're done". In
> > fact there was DLB1 hardware that also had a polled-mode-userspace
> > driver. So, the next question is "what's changed in DLB2 where a
> > userspace driver is no longer suitable?". The new use case for DLB2 is
> > new hardware support for a host driver to carve up device resources
> > into smaller sets (vfio-mdevs) that can be assigned to guests (Intel
> > calls this new hardware capability SIOV: Scalable IO Virtualization).
> >
> > Hardware resource management is difficult to handle in userspace
> > especially when bare-metal hardware events need to coordinate with
> > guest-VM device instances. This includes a mailbox interface for the
> > guest VM to negotiate resources with the host driver. Another more
> > practical roadblock for a "DLB2 in userspace" proposal is the fact
> > that it implements what are in-effect software-defined-interrupts to
> > go beyond the scalability limits of PCI MSI-x (Intel calls this
> > Interrupt Message Store: IMS). So even if hardware resource management
> > was awkwardly plumbed into a userspace daemon there would still need
> > to be kernel enabling for device-specific extensions to
> > drivers/vfio/pci/vfio_pci_intrs.c for it to understand the IMS
> > interrupts of DLB2 in addition to PCI MSI-x.
> >
> > While that still might be solvable in userspace if you squint at it, I
> > don't think Linux end users are served by pushing all of hardware
> > resource management to userspace. VFIO is mostly built to pass entire
> > PCI devices to guests, or in coordination with a kernel driver to
> > describe a subset of the hardware to a virtual-device (vfio-mdev)
> > interface. The rub here is that to date kernel drivers using VFIO to
> > provision mdevs have some existing responsibilities to the core kernel
> > like a network driver or DMA offload driver. The DLB2 driver offers no
> > such service to the kernel for its primary role of accelerating a
> > userspace data-plane. I am assuming here that  the padata
> > proof-of-concept is interesting, but not a compelling reason to ship a
> > driver compared to giving end users competent kernel-driven
> > hardware-resource assignment for deploying DLB2 virtual instances into
> > guest VMs.
> >
> > My "just continue in userspace" suggestion has no answer for the IMS
> > interrupt and reliable hardware resource management support
> > requirements. If you're with me so far we can go deeper into the
> > details, but in answer to your previous questions most of the TLAs
> > were from the land of "SIOV" where the VFIO community should be
> > brought in to review. The driver is mostly a configuration plane where
> > the fast path data-plane is entirely in userspace. That configuration
> > plane needs to manage hardware events and resourcing on behalf of
> > guest VMs running on a partitioned subset of the device. There are
> > worthwhile questions about whether some of the uapi can be refactored
> > to common modules like uacce, but I think we need to get to a first
> > order understanding on what DLB2 is and why the kernel has a role
> > before diving into the uapi discussion.
> >
> > Any clearer?
> 
> A bit, yes, thanks.
> 
> > So, in summary drivers/misc/ appears to be the first stop in the
> > review since a host driver needs to be established to start the VFIO
> > enabling campaign. With my community hat on, I think requiring
> > standalone host drivers is healthier for Linux than broaching the
> > subject of VFIO-only drivers. Even if, as in this case, the initial
> > host driver is mostly implementing a capability that could be achieved
> > with a userspace driver.
> 
> Ok, then how about a much "smaller" kernel driver for all of this, and a whole lot of documentation to
> describe what is going on and what all of the TLAs are.
> 
> thanks,
> 
> greg k-h

Hi Greg,

tl;dr: We have been looking into various options to reduce the kernel driver size and ABI surface, such as moving more responsibility to user space, reusing existing kernel modules (uacce, for example), and converting functionality from ioctl to sysfs. End result 10 ioctls will be replaced by sysfs, the rest of them (20 ioctls) will be replaced by configfs. Some concepts are moved to device-special files rather than ioctls that produce file descriptors.

Details:
We investigated the possibility of using uacce (https://www.kernel.org/doc/html/latest/misc-devices/uacce.html) in our kernel driver.  The uacce interface fits well with accelerators that process user data with known source and destination addresses. For a DLB (Dynamic Load Balancer), however,  the destination port depends on the system load and is unknown to the application. While uacce exposes  "queues" to user, the dlb driver has to handle much complicated resource managements, such as credits, ports, queues and domains. We would have to add a lot of more concepts and code, which are not useful for other accelerators,  in uacce to make it working for DLB. This may also lead to a bigger code size over all.

We also took a another look at moving resource management functionality from kernel space to user space. Much of kernel driver supports both PF (Physical Function) on host and VFs (Virtual Functions) on VMs. Since only the PF on the host has permissions to setup resource and configure the DLB HW, all the requests on VFs are forwarded to PF via the VF-PF mail boxes, which are handled by the kernel driver. The driver also maintains various virtual id to physical id translations (for VFs, ports, queues, etc), and provides virtual-to-physical id mapping info DLB HW so that an application in VM can access the resources with virtual IDs only. Because of the VF/VDEV support, we have to keep the resource management, which is more than one half of the code size, in the driver.

To simplify the user interface, we explored the ways to reduce/eliminate ioctl interface, and found that we can utilize configfs for many of the DLB functionalities. Our current plan is to replace all the ioctls in the driver with sysfs and configfs. We will use configfs for most of setup and configuration for both physical function and virtual functions. This may not reduce the overall driver size greatly, but it will lessen much of ABI maintenance burden (with the elimination of ioctls).  I hope this is something that is in line with what you like to see for the driver.

Thanks
Mike

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Dan Williams]

On Mon, Mar 8, 2021 at 8:53 PM Chen, Mike Ximing
<mike.ximing.chen@intel.com> wrote:
>
>
> > -----Original Message-----
> > From: Greg KH <gregkh@linuxfoundation.org>
> > On Mon, Mar 08, 2021 at 08:00:00PM +0000, Chen, Mike Ximing wrote:
> > >
> > > Hi Greg,
> > >
> > > While waiting for the feedback from the networking maintainers, I am
> > > wondering if you have any other comments/suggestions that I  should address
> > > in parallel.
> >
> > It's in my "to-review" queue, which is huge at the moment.  But the
> > networking developers review will determine how this should go forward,
> > so I'll just wait for them to get to it.
> >
>
> I see the status of the submission (to netdev)  is now marked as "Not Applicable"
> at netdev's patchwork site
> https://patchwork.kernel.org/project/netdevbpf/list/?series=&submitter=197673&state=*&q=&archive=both&delegate=
>
> Looks like that the patch set is considered as not being networking related. (?)

That's just to clean up their queue of things that need to be pulled
into the net tree. This driver is being requested to just be acked-by
netdev before Greg considers it for drivers/misc/.

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> On Mon, Mar 08, 2021 at 08:00:00PM +0000, Chen, Mike Ximing wrote:
> >
> > Hi Greg,
> >
> > While waiting for the feedback from the networking maintainers, I am
> > wondering if you have any other comments/suggestions that I  should address
> > in parallel.
> 
> It's in my "to-review" queue, which is huge at the moment.  But the
> networking developers review will determine how this should go forward,
> so I'll just wait for them to get to it.
> 

I see the status of the submission (to netdev)  is now marked as "Not Applicable" 
at netdev's patchwork site
https://patchwork.kernel.org/project/netdevbpf/list/?series=&submitter=197673&state=*&q=&archive=both&delegate=

Looks like that the patch set is considered as not being networking related. (?)

Thanks
Mike 

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Monday, March 8, 2021 5:06 PM
> To: Greg KH <gregkh@linuxfoundation.org>
> Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Linux Kernel Mailing List
> <linux-kernel@vger.kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-Louis
> Bossart <pierre-louis.bossart@linux.intel.com>; Brandeburg, Jesse
> <jesse.brandeburg@intel.com>; kuba@kernel.org; davem@davemloft.net;
> Karlsson, Magnus <magnus.karlsson@intel.com>; Laatz, Kevin
> <kevin.laatz@intel.com>; maximmi@mellanox.com
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> [ add Kevin and Maxim ]
> 
> On Mon, Mar 8, 2021 at 12:13 PM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Mon, Mar 08, 2021 at 08:00:00PM +0000, Chen, Mike Ximing wrote:
> [..]
> > > Hi Greg,
> > >
> > > While waiting for the feedback from the networking maintainers, I am
> > > wondering if you have any other comments/suggestions that I  should
> address
> > > in parallel.
> >
> > It's in my "to-review" queue, which is huge at the moment.  But the
> > networking developers review will determine how this should go forward,
> > so I'll just wait for them to get to it.
> 
> 
> Mike,
> 
> Perhaps it would help to solicit feedback from other networking
> developers that have delivered kernel features to be consumed by DPDK.
> A coarse look potentially flagsKevin and Maxim working on AF_XDP
> updates that DPDK consumes, Perhaps they would be willing to take a
> look?

Thanks, Dan.

Kevin/Maxim, 
Please feel free to let me know if you have any question about the submission.

Thanks
Mike

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Dan Williams]

[ add Kevin and Maxim ]

On Mon, Mar 8, 2021 at 12:13 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Mon, Mar 08, 2021 at 08:00:00PM +0000, Chen, Mike Ximing wrote:
[..]
> > Hi Greg,
> >
> > While waiting for the feedback from the networking maintainers, I am
> > wondering if you have any other comments/suggestions that I  should address
> > in parallel.
>
> It's in my "to-review" queue, which is huge at the moment.  But the
> networking developers review will determine how this should go forward,
> so I'll just wait for them to get to it.


Mike,

Perhaps it would help to solicit feedback from other networking
developers that have delivered kernel features to be consumed by DPDK.
A coarse look potentially flagsKevin and Maxim working on AF_XDP
updates that DPDK consumes, Perhaps they would be willing to take a
look?

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Greg KH]

On Mon, Mar 08, 2021 at 08:00:00PM +0000, Chen, Mike Ximing wrote:
> 
> 
> > -----Original Message-----
> > From: Dan Williams <dan.j.williams@intel.com>
> > Sent: Tuesday, February 9, 2021 11:30 AM
> > To: Greg KH <gregkh@linuxfoundation.org>
> > Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Linux Kernel Mailing List
> > <linux-kernel@vger.kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-Louis
> > Bossart <pierre-louis.bossart@linux.intel.com>; Gage Eads <gage.eads@intel.com>
> > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> > 
> > On Tue, Feb 9, 2021 at 5:36 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote:
> > > > Add basic driver functionality (load, unload, probe, and remove callbacks)
> > > > for the DLB driver.
> > > >
> > > > Add documentation which describes in detail the hardware, the user
> > > > interface, device interrupts, and the driver's power-management strategy.
> > > > For more details about the driver see the documentation in the patch.
> > > >
> > > > Add a DLB entry to the MAINTAINERS file.
> > > >
> > > > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > > > Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > > Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
> > > > Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> > > > ---
> > > >  Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
> > > >  Documentation/misc-devices/index.rst |   1 +
> > > >  MAINTAINERS                          |   8 +
> > > >  drivers/misc/Kconfig                 |   1 +
> > > >  drivers/misc/Makefile                |   1 +
> > > >  drivers/misc/dlb/Kconfig             |  18 ++
> > > >  drivers/misc/dlb/Makefile            |   9 +
> > > >  drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
> > > >  drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
> > > >  drivers/misc/dlb/dlb_main.h          |  37 ++++
> > > >  10 files changed, 522 insertions(+)
> > > >  create mode 100644 Documentation/misc-devices/dlb.rst
> > > >  create mode 100644 drivers/misc/dlb/Kconfig
> > > >  create mode 100644 drivers/misc/dlb/Makefile
> > > >  create mode 100644 drivers/misc/dlb/dlb_hw_types.h
> > > >  create mode 100644 drivers/misc/dlb/dlb_main.c
> > > >  create mode 100644 drivers/misc/dlb/dlb_main.h
> > > >
> > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> > devices/dlb.rst
> > > > new file mode 100644
> > > > index 000000000000..aa79be07ee49
> > > > --- /dev/null
> > > > +++ b/Documentation/misc-devices/dlb.rst
> > > > @@ -0,0 +1,259 @@
> > > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > > +
> > > > +===========================================
> > > > +Intel(R) Dynamic Load Balancer Overview
> > > > +===========================================
> > > > +
> > > > +:Authors: Gage Eads and Mike Ximing Chen
> > > > +
> > > > +Contents
> > > > +========
> > > > +
> > > > +- Introduction
> > > > +- Scheduling
> > > > +- Queue Entry
> > > > +- Port
> > > > +- Queue
> > > > +- Credits
> > > > +- Scheduling Domain
> > > > +- Interrupts
> > > > +- Power Management
> > > > +- User Interface
> > > > +- Reset
> > > > +
> > > > +Introduction
> > > > +============
> > > > +
> > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > > +provides load-balanced, prioritized scheduling of core-to-core
> > communication.
> > > > +
> > > > +Intel DLB is an accelerator for the event-driven programming model of
> > > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > > +pipelines that arrange for multi-core scalability, dynamic load-balancing,
> > and
> > > > +variety of packet distribution and synchronization schemes.
> > >
> > > As this is a networking related thing, I would like you to get the
> > > proper reviews/acks from the networking maintainers before I can take
> > > this.
> > >
> > > Or, if they think it has nothing to do with networking, that's fine too,
> > > but please do not try to route around them.
> > 
> > To be clear, I did not sense any attempt to route around networking
> > review as it appeared generically centered around hardware accelerated
> > IPC. At the same time I don't know what I don't know about how this
> > might interact with networking initiatives so the review trip seems
> > reasonable to me.
> 
> Hi Greg,
> 
> While waiting for the feedback from the networking maintainers, I am
> wondering if you have any other comments/suggestions that I  should address
> in parallel.

It's in my "to-review" queue, which is huge at the moment.  But the
networking developers review will determine how this should go forward,
so I'll just wait for them to get to it.

thanks,

greg k-h

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Tuesday, February 9, 2021 11:30 AM
> To: Greg KH <gregkh@linuxfoundation.org>
> Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Linux Kernel Mailing List
> <linux-kernel@vger.kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-Louis
> Bossart <pierre-louis.bossart@linux.intel.com>; Gage Eads <gage.eads@intel.com>
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Tue, Feb 9, 2021 at 5:36 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote:
> > > Add basic driver functionality (load, unload, probe, and remove callbacks)
> > > for the DLB driver.
> > >
> > > Add documentation which describes in detail the hardware, the user
> > > interface, device interrupts, and the driver's power-management strategy.
> > > For more details about the driver see the documentation in the patch.
> > >
> > > Add a DLB entry to the MAINTAINERS file.
> > >
> > > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > > Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
> > > Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> > > ---
> > >  Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
> > >  Documentation/misc-devices/index.rst |   1 +
> > >  MAINTAINERS                          |   8 +
> > >  drivers/misc/Kconfig                 |   1 +
> > >  drivers/misc/Makefile                |   1 +
> > >  drivers/misc/dlb/Kconfig             |  18 ++
> > >  drivers/misc/dlb/Makefile            |   9 +
> > >  drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
> > >  drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
> > >  drivers/misc/dlb/dlb_main.h          |  37 ++++
> > >  10 files changed, 522 insertions(+)
> > >  create mode 100644 Documentation/misc-devices/dlb.rst
> > >  create mode 100644 drivers/misc/dlb/Kconfig
> > >  create mode 100644 drivers/misc/dlb/Makefile
> > >  create mode 100644 drivers/misc/dlb/dlb_hw_types.h
> > >  create mode 100644 drivers/misc/dlb/dlb_main.c
> > >  create mode 100644 drivers/misc/dlb/dlb_main.h
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core
> communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing,
> and
> > > +variety of packet distribution and synchronization schemes.
> >
> > As this is a networking related thing, I would like you to get the
> > proper reviews/acks from the networking maintainers before I can take
> > this.
> >
> > Or, if they think it has nothing to do with networking, that's fine too,
> > but please do not try to route around them.
> 
> To be clear, I did not sense any attempt to route around networking
> review as it appeared generically centered around hardware accelerated
> IPC. At the same time I don't know what I don't know about how this
> might interact with networking initiatives so the review trip seems
> reasonable to me.

Hi Greg,

While waiting for the feedback from the networking maintainers, I am
wondering if you have any other comments/suggestions that I  should address
in parallel.

Thanks
Mike

RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Chen, Mike Ximing]

> -----Original Message-----
> From: Dan Williams <dan.j.williams@intel.com>
> Sent: Tuesday, February 9, 2021 11:30 AM
> To: Greg KH <gregkh@linuxfoundation.org>
> Cc: Chen, Mike Ximing <mike.ximing.chen@intel.com>; Linux Kernel Mailing List
> <linux-kernel@vger.kernel.org>; Arnd Bergmann <arnd@arndb.de>; Pierre-Louis
> Bossart <pierre-louis.bossart@linux.intel.com>; Gage Eads <gage.eads@intel.com>
> Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> 
> On Tue, Feb 9, 2021 at 5:36 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote:
> > > Add basic driver functionality (load, unload, probe, and remove callbacks)
> > > for the DLB driver.
> > >
> > > Add documentation which describes in detail the hardware, the user
> > > interface, device interrupts, and the driver's power-management strategy.
> > > For more details about the driver see the documentation in the patch.
> > >
> > > Add a DLB entry to the MAINTAINERS file.
> > >
> > > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > > Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > > Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
> > > Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> > > ---
> > >  Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
> > >  Documentation/misc-devices/index.rst |   1 +
> > >  MAINTAINERS                          |   8 +
> > >  drivers/misc/Kconfig                 |   1 +
> > >  drivers/misc/Makefile                |   1 +
> > >  drivers/misc/dlb/Kconfig             |  18 ++
> > >  drivers/misc/dlb/Makefile            |   9 +
> > >  drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
> > >  drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
> > >  drivers/misc/dlb/dlb_main.h          |  37 ++++
> > >  10 files changed, 522 insertions(+)
> > >  create mode 100644 Documentation/misc-devices/dlb.rst
> > >  create mode 100644 drivers/misc/dlb/Kconfig
> > >  create mode 100644 drivers/misc/dlb/Makefile
> > >  create mode 100644 drivers/misc/dlb/dlb_hw_types.h
> > >  create mode 100644 drivers/misc/dlb/dlb_main.c
> > >  create mode 100644 drivers/misc/dlb/dlb_main.h
> > >
> > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-
> devices/dlb.rst
> > > new file mode 100644
> > > index 000000000000..aa79be07ee49
> > > --- /dev/null
> > > +++ b/Documentation/misc-devices/dlb.rst
> > > @@ -0,0 +1,259 @@
> > > +.. SPDX-License-Identifier: GPL-2.0-only
> > > +
> > > +===========================================
> > > +Intel(R) Dynamic Load Balancer Overview
> > > +===========================================
> > > +
> > > +:Authors: Gage Eads and Mike Ximing Chen
> > > +
> > > +Contents
> > > +========
> > > +
> > > +- Introduction
> > > +- Scheduling
> > > +- Queue Entry
> > > +- Port
> > > +- Queue
> > > +- Credits
> > > +- Scheduling Domain
> > > +- Interrupts
> > > +- Power Management
> > > +- User Interface
> > > +- Reset
> > > +
> > > +Introduction
> > > +============
> > > +
> > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > > +
> > > +Intel DLB is an accelerator for the event-driven programming model of
> > > +DPDK's Event Device Library[2]. The library is used in packet processing
> > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > > +variety of packet distribution and synchronization schemes.
> >
> > As this is a networking related thing, I would like you to get the
> > proper reviews/acks from the networking maintainers before I can take
> > this.
> >
> > Or, if they think it has nothing to do with networking, that's fine too,
> > but please do not try to route around them.
> >
> >thanks,
> >
> >greg k-
> >
> To be clear, I did not sense any attempt to route around networking
> review as it appeared generically centered around hardware accelerated
> IPC. At the same time I don't know what I don't know about how this
> might interact with networking initiatives so the review trip seems
> reasonable to me.

OK. I have forwarded the patch set to netdev.

Thanks!
Mike

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Dan Williams]

On Tue, Feb 9, 2021 at 5:36 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote:
> > Add basic driver functionality (load, unload, probe, and remove callbacks)
> > for the DLB driver.
> >
> > Add documentation which describes in detail the hardware, the user
> > interface, device interrupts, and the driver's power-management strategy.
> > For more details about the driver see the documentation in the patch.
> >
> > Add a DLB entry to the MAINTAINERS file.
> >
> > Signed-off-by: Gage Eads <gage.eads@intel.com>
> > Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> > Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
> > Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> > ---
> >  Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
> >  Documentation/misc-devices/index.rst |   1 +
> >  MAINTAINERS                          |   8 +
> >  drivers/misc/Kconfig                 |   1 +
> >  drivers/misc/Makefile                |   1 +
> >  drivers/misc/dlb/Kconfig             |  18 ++
> >  drivers/misc/dlb/Makefile            |   9 +
> >  drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
> >  drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
> >  drivers/misc/dlb/dlb_main.h          |  37 ++++
> >  10 files changed, 522 insertions(+)
> >  create mode 100644 Documentation/misc-devices/dlb.rst
> >  create mode 100644 drivers/misc/dlb/Kconfig
> >  create mode 100644 drivers/misc/dlb/Makefile
> >  create mode 100644 drivers/misc/dlb/dlb_hw_types.h
> >  create mode 100644 drivers/misc/dlb/dlb_main.c
> >  create mode 100644 drivers/misc/dlb/dlb_main.h
> >
> > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst
> > new file mode 100644
> > index 000000000000..aa79be07ee49
> > --- /dev/null
> > +++ b/Documentation/misc-devices/dlb.rst
> > @@ -0,0 +1,259 @@
> > +.. SPDX-License-Identifier: GPL-2.0-only
> > +
> > +===========================================
> > +Intel(R) Dynamic Load Balancer Overview
> > +===========================================
> > +
> > +:Authors: Gage Eads and Mike Ximing Chen
> > +
> > +Contents
> > +========
> > +
> > +- Introduction
> > +- Scheduling
> > +- Queue Entry
> > +- Port
> > +- Queue
> > +- Credits
> > +- Scheduling Domain
> > +- Interrupts
> > +- Power Management
> > +- User Interface
> > +- Reset
> > +
> > +Introduction
> > +============
> > +
> > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> > +provides load-balanced, prioritized scheduling of core-to-core communication.
> > +
> > +Intel DLB is an accelerator for the event-driven programming model of
> > +DPDK's Event Device Library[2]. The library is used in packet processing
> > +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> > +variety of packet distribution and synchronization schemes.
>
> As this is a networking related thing, I would like you to get the
> proper reviews/acks from the networking maintainers before I can take
> this.
>
> Or, if they think it has nothing to do with networking, that's fine too,
> but please do not try to route around them.

To be clear, I did not sense any attempt to route around networking
review as it appeared generically centered around hardware accelerated
IPC. At the same time I don't know what I don't know about how this
might interact with networking initiatives so the review trip seems
reasonable to me.

Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver

[Greg KH]

On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote:
> Add basic driver functionality (load, unload, probe, and remove callbacks)
> for the DLB driver.
> 
> Add documentation which describes in detail the hardware, the user
> interface, device interrupts, and the driver's power-management strategy.
> For more details about the driver see the documentation in the patch.
> 
> Add a DLB entry to the MAINTAINERS file.
> 
> Signed-off-by: Gage Eads <gage.eads@intel.com>
> Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
> Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
> Reviewed-by: Dan Williams <dan.j.williams@intel.com>
> ---
>  Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
>  Documentation/misc-devices/index.rst |   1 +
>  MAINTAINERS                          |   8 +
>  drivers/misc/Kconfig                 |   1 +
>  drivers/misc/Makefile                |   1 +
>  drivers/misc/dlb/Kconfig             |  18 ++
>  drivers/misc/dlb/Makefile            |   9 +
>  drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
>  drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
>  drivers/misc/dlb/dlb_main.h          |  37 ++++
>  10 files changed, 522 insertions(+)
>  create mode 100644 Documentation/misc-devices/dlb.rst
>  create mode 100644 drivers/misc/dlb/Kconfig
>  create mode 100644 drivers/misc/dlb/Makefile
>  create mode 100644 drivers/misc/dlb/dlb_hw_types.h
>  create mode 100644 drivers/misc/dlb/dlb_main.c
>  create mode 100644 drivers/misc/dlb/dlb_main.h
> 
> diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst
> new file mode 100644
> index 000000000000..aa79be07ee49
> --- /dev/null
> +++ b/Documentation/misc-devices/dlb.rst
> @@ -0,0 +1,259 @@
> +.. SPDX-License-Identifier: GPL-2.0-only
> +
> +===========================================
> +Intel(R) Dynamic Load Balancer Overview
> +===========================================
> +
> +:Authors: Gage Eads and Mike Ximing Chen
> +
> +Contents
> +========
> +
> +- Introduction
> +- Scheduling
> +- Queue Entry
> +- Port
> +- Queue
> +- Credits
> +- Scheduling Domain
> +- Interrupts
> +- Power Management
> +- User Interface
> +- Reset
> +
> +Introduction
> +============
> +
> +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
> +provides load-balanced, prioritized scheduling of core-to-core communication.
> +
> +Intel DLB is an accelerator for the event-driven programming model of
> +DPDK's Event Device Library[2]. The library is used in packet processing
> +pipelines that arrange for multi-core scalability, dynamic load-balancing, and
> +variety of packet distribution and synchronization schemes.

As this is a networking related thing, I would like you to get the
proper reviews/acks from the networking maintainers before I can take
this.

Or, if they think it has nothing to do with networking, that's fine too,
but please do not try to route around them.

thanks,

greg k-h

[PATCH v10 01/20] dlb: add skeleton for DLB driver

[Mike Ximing Chen]

Add basic driver functionality (load, unload, probe, and remove callbacks)
for the DLB driver.

Add documentation which describes in detail the hardware, the user
interface, device interrupts, and the driver's power-management strategy.
For more details about the driver see the documentation in the patch.

Add a DLB entry to the MAINTAINERS file.

Signed-off-by: Gage Eads <gage.eads@intel.com>
Signed-off-by: Mike Ximing Chen <mike.ximing.chen@intel.com>
Reviewed-by: Magnus Karlsson <magnus.karlsson@intel.com>
Reviewed-by: Dan Williams <dan.j.williams@intel.com>
---
 Documentation/misc-devices/dlb.rst   | 259 +++++++++++++++++++++++++++
 Documentation/misc-devices/index.rst |   1 +
 MAINTAINERS                          |   8 +
 drivers/misc/Kconfig                 |   1 +
 drivers/misc/Makefile                |   1 +
 drivers/misc/dlb/Kconfig             |  18 ++
 drivers/misc/dlb/Makefile            |   9 +
 drivers/misc/dlb/dlb_hw_types.h      |  32 ++++
 drivers/misc/dlb/dlb_main.c          | 156 ++++++++++++++++
 drivers/misc/dlb/dlb_main.h          |  37 ++++
 10 files changed, 522 insertions(+)
 create mode 100644 Documentation/misc-devices/dlb.rst
 create mode 100644 drivers/misc/dlb/Kconfig
 create mode 100644 drivers/misc/dlb/Makefile
 create mode 100644 drivers/misc/dlb/dlb_hw_types.h
 create mode 100644 drivers/misc/dlb/dlb_main.c
 create mode 100644 drivers/misc/dlb/dlb_main.h

diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst
new file mode 100644
index 000000000000..aa79be07ee49
--- /dev/null
+++ b/Documentation/misc-devices/dlb.rst
@@ -0,0 +1,259 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+===========================================
+Intel(R) Dynamic Load Balancer Overview
+===========================================
+
+:Authors: Gage Eads and Mike Ximing Chen
+
+Contents
+========
+
+- Introduction
+- Scheduling
+- Queue Entry
+- Port
+- Queue
+- Credits
+- Scheduling Domain
+- Interrupts
+- Power Management
+- User Interface
+- Reset
+
+Introduction
+============
+
+The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that
+provides load-balanced, prioritized scheduling of core-to-core communication.
+
+Intel DLB is an accelerator for the event-driven programming model of
+DPDK's Event Device Library[2]. The library is used in packet processing
+pipelines that arrange for multi-core scalability, dynamic load-balancing, and
+variety of packet distribution and synchronization schemes.
+
+Intel DLB device consists of queues and arbiters that connect producer
+cores and consumer cores. The device implements load-balanced queueing features
+including:
+- Lock-free multi-producer/multi-consumer operation.
+- Multiple priority levels for varying traffic types.
+- 'Direct' traffic (i.e. multi-producer/single-consumer)
+- Simple unordered load-balanced distribution.
+- Atomic lock free load balancing across multiple consumers.
+- Queue element reordering feature allowing ordered load-balanced distribution.
+
+Note: this document uses 'DLB' when discussing the device hardware and 'dlb' when
+discussing the driver implementation.
+
+Following diagram illustrates the functional blocks of an Intel DLB device.
+
+                                       +----+
+                                       |    |
+                        +----------+   |    |   +-------+
+                       /|   IQ     |---|----|--/|       |
+                      / +----------+   |    | / |  CP   |
+                     /                 |    |/  +-------+
+        +--------+  /                  |    |
+        |        | /    +----------+   |   /|   +-------+
+        |  PP    |------|   IQ     |---|----|---|       |
+        +--------+ \    +----------+   | /  |   |  CP   |
+                    \                  |/   |   +-------+
+           ...       \     ...         |    |
+        +--------+    \               /|    |   +-------+
+        |        |     \+----------+ / |    |   |       |
+        |  PP    |------|   IQ     |/--|----|---|  CP   |
+        +--------+      +----------+   |    |   +-------+
+                                       |    |
+                                       +----+     ...
+PP: Producer Port                        |
+CP: Consumer Port                        |
+IQ: Internal Queue                   DLB Scheduler
+
+
+Scheduling Types
+================
+
+Intel DLB supports four types of scheduling of 'events' (using DPDK
+terminology), where an event can represent any type of data (e.g. a network
+packet). The first, ``directed``, is multi-producer/single-consumer style
+core-to-core communication. The remaining three are
+multi-producer/multi-consumer, and support load-balancing across the consumers.
+
+- ``Directed``: events are scheduled to a single consumer.
+
+- ``Unordered``: events are load-balanced across consumers without any ordering
+                 guarantees.
+
+- ``Ordered``: events are load-balanced across consumers, and the consumer can
+               re-enqueue its events so the device re-orders them into the
+               original order. This scheduling type allows software to
+               parallelize ordered event processing without the synchronization
+               cost of re-ordering packets.
+
+- ``Atomic``: events are load-balanced across consumers, with the guarantee that
+              events from a particular 'flow' are only scheduled to a single
+              consumer at a time (but can migrate over time). This allows, for
+              example, packet processing applications to parallelize while
+              avoiding locks on per-flow data and maintaining ordering within a
+              flow.
+
+Intel DLB provides hierarchical priority scheduling, with eight priority
+levels within each. Each consumer selects up to eight queues to receive events
+from, and assigns a priority to each of these 'connected' queues. To schedule
+an event to a consumer, the device selects the highest priority non-empty queue
+of the (up to) eight connected queues. Within that queue, the device selects
+the highest priority event available (selecting a lower priority event for
+starvation avoidance 1% of the time, by default).
+
+The device also supports four load-balanced scheduler classes of service. Each
+class of service receives a (user-configurable) guaranteed percentage of the
+scheduler bandwidth, and any unreserved bandwidth is divided evenly among the
+four classes.
+
+Queue Entry
+===========
+
+Each event is contained in a queue entry (QE), the fundamental unit of
+communication through the device, which consists of 8B of data and 8B of
+metadata, as depicted below.
+
+QE structure format
+::
+    data     :64
+    opaque   :16
+    qid      :8
+    sched    :2
+    priority :3
+    msg_type :3
+    lock_id  :16
+    rsvd     :8
+    cmd      :8
+
+The ``data`` field can be any type that fits within 8B (pointer, integer,
+etc.); DLB merely copies this field from producer to consumer. The
+``opaque`` and ``msg_type`` fields behave the same way.
+
+``qid`` is set by the producer to specify to which DLB 2.0 queue it wishes to
+enqueue this QE. The ID spaces for load-balanced and directed queues are both
+zero-based.
+
+``sched`` controls the scheduling type: atomic, unordered, ordered, or
+directed. The first three scheduling types are only valid for load-balanced
+queues, and the directed scheduling type is only valid for directed queues.
+This field distinguishes whether ``qid`` is load-balanced or directed, since
+their ID spaces overlap.
+
+``priority`` is the priority with which this QE should be scheduled.
+
+``lock_id``, used for atomic scheduling and ignored for ordered and unordered
+scheduling, identifies the atomic flow to which the QE belongs. When sending a
+directed event, ``lock_id`` is simply copied like the ``data``, ``opaque``, and
+``msg_type`` fields.
+
+``cmd`` specifies the operation, such as:
+- Enqueue a new QE
+- Forward a QE that was dequeued
+- Complete/terminate a QE that was dequeued
+- Return one or more consumer queue tokens.
+- Arm the port's consumer queue interrupt.
+
+Port
+====
+
+A core's interface to the DLB is called a "port," and consists of an MMIO
+region (producer port) through which the core enqueues a queue entry, and an
+in-memory queue (the "consumer queue" or cosumer port) to which the device
+schedules QEs. A core enqueues a QE to a device queue, then the device
+schedules the event to a port. Software specifies the connection of queues
+and ports; i.e. for each queue, to which ports the device is allowed to
+schedule its events. The device uses a credit scheme to prevent overflow of
+the on-device queue storage.
+
+Applications interface directly with the device by mapping the port's memory
+and MMIO regions into the application's address space for enqueue and dequeue
+operations, but call into the kernel driver for configuration operations. An
+application can be polling- or interrupt-driven; DLB supports both modes
+of operation.
+
+Internal Queue
+==============
+
+A DLB device supports an implementation specific and runtime discoverable
+number of load-balanced (i.e. capable of atomic, ordered, and unordered
+scheduling) and directed queues. Each internal queue supports a set of
+priority levels.
+
+A load-balanced queue is capable of scheduling its events to any combination of
+load-balanced ports, whereas each directed queue has one-to-one mapping with a
+directed port. There is no restriction on port or queue types when a port
+enqueues an event to a queue; that is, a load-balanced port can enqueue to a
+directed queue and vice versa.
+
+Credits
+=======
+
+The Intel DLB uses a credit scheme to prevent overflow of the on-device
+queue storage, with separate credits for load-balanced and directed queues. A
+port spends one credit when it enqueues a QE, and one credit is replenished
+when a QE is scheduled to a consumer queue. Each scheduling domain has one pool
+of load-balanced credits and one pool of directed credits; software is
+responsible for managing the allocation and replenishment of these credits among
+the scheduling domain's ports.
+
+Scheduling Domain
+=================
+
+Device resources -- including ports, queues, and credits -- are contained
+within a scheduling domain. Scheduling domains are isolated from one another; a
+port can only enqueue to and dequeue from queues within its scheduling domain.
+
+The scheduling domain creation ioctl returns a domain file descriptor, through
+which the domain's resources are configured. For a multi-process scenario, the
+owner of this descriptor must share it with the other processes (e.g. inherited
+through fork() or shared over a unix domain socket).
+
+Consumer Queue Interrupts
+=========================
+
+Each port has its own interrupt which fires, if armed, when the consumer queue
+depth becomes non-zero. Software arms an interrupt by enqueueing a special
+'interrupt arm' command to the device through the port's MMIO window.
+
+Power Management
+================
+
+The kernel driver keeps the device in D3Hot when not in use. The driver
+transitions the device to D0 when the first device file is opened, and keeps it
+there until there are no open device files or memory mappings.
+
+User Interface
+==============
+
+The dlb driver uses ioctls as its primary interface. It provides two types of
+files: the dlb device file and the scheduling domain file.
+
+The two types support different ioctl interfaces; the dlb device file is used
+for device-wide operations (including scheduling domain creation), and the
+scheduling domain device file supports operations on the scheduling domain's
+resources such as port and queue configuration.
+
+The dlb device file is created during driver probe and is located at
+/dev/dlb<N>, where N is the zero-based device ID. The scheduling domain fd is
+an anonymous inode created by a dlb device ioctl.
+
+The driver also exports an mmap interface through port files, which are
+acquired through scheduling domain ioctls. This mmap interface is used to map
+a port's memory and MMIO window into the process's address space. Once the
+ports are mapped, applications may use 64-byte direct-store instructions such
+movdir64b or enqcmd to enqueue the events for better performance.
+
+Reset
+=====
+
+The dlb driver currently supports scheduling domain reset.
+
+Scheduling domain reset occurs when an application stops using its domain.
+Specifically, when no more file references or memory mappings exist. At this
+time, the driver resets all the domain's resources (flushes its queues and
+ports) and puts them in their respective available-resource lists for later
+use.
diff --git a/Documentation/misc-devices/index.rst b/Documentation/misc-devices/index.rst
index 64420b3314fe..c9954e8b72a3 100644
--- a/Documentation/misc-devices/index.rst
+++ b/Documentation/misc-devices/index.rst
@@ -17,6 +17,7 @@ fit into other categories.
    ad525x_dpot
    apds990x
    bh1770glc
+   dlb
    eeprom
    c2port
    ibmvmc
diff --git a/MAINTAINERS b/MAINTAINERS
index 6eff4f720c72..b408c1ec8fdf 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8896,6 +8896,14 @@ L:	linux-kernel@vger.kernel.org
 S:	Supported
 F:	arch/x86/include/asm/intel-family.h
 
+INTEL DYNAMIC LOAD BALANCER DRIVER
+M:	Mike Ximing Chen <mike.ximing.chen@intel.com>
+M:	Gage Eads <gage.eads@intel.com>
+S:	Maintained
+F:	Documentation/ABI/testing/sysfs-driver-dlb
+F:	drivers/misc/dlb/
+F:	include/uapi/linux/dlb_user.h
+
 INTEL DRM DRIVERS (excluding Poulsbo, Moorestown and derivative chipsets)
 M:	Jani Nikula <jani.nikula@linux.intel.com>
 M:	Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index fafa8b0d8099..fef26819eb1e 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -481,4 +481,5 @@ source "drivers/misc/ocxl/Kconfig"
 source "drivers/misc/cardreader/Kconfig"
 source "drivers/misc/habanalabs/Kconfig"
 source "drivers/misc/uacce/Kconfig"
+source "drivers/misc/dlb/Kconfig"
 endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index d23231e73330..a0bafe336277 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -57,3 +57,4 @@ obj-$(CONFIG_HABANA_AI)		+= habanalabs/
 obj-$(CONFIG_UACCE)		+= uacce/
 obj-$(CONFIG_XILINX_SDFEC)	+= xilinx_sdfec.o
 obj-$(CONFIG_HISI_HIKEY_USB)	+= hisi_hikey_usb.o
+obj-$(CONFIG_INTEL_DLB)		+= dlb/
diff --git a/drivers/misc/dlb/Kconfig b/drivers/misc/dlb/Kconfig
new file mode 100644
index 000000000000..cfa978c705bd
--- /dev/null
+++ b/drivers/misc/dlb/Kconfig
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config INTEL_DLB
+	tristate "Intel Dynamic Load Balancer Driver"
+	depends on 64BIT && PCI && X86
+	help
+	  This driver supports the Intel Dynamic Load Balancer (DLB), a
+	  PCIe device (PCI ID 8086:27xx) that provides load-balanced,
+	  prioritized scheduling of core-to-core communication and improves
+	  DPDK Event Device library performance.
+
+	  The user-space interface is described in
+	  include/uapi/linux/dlb_user.h
+
+	  To compile this driver as a module, choose M here. The module
+	  will be called dlb.
+
+	  If unsure, select N.
diff --git a/drivers/misc/dlb/Makefile b/drivers/misc/dlb/Makefile
new file mode 100644
index 000000000000..8911375effd2
--- /dev/null
+++ b/drivers/misc/dlb/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+#
+# Makefile for the Intel(R) Dynamic Load Balancer (dlb.ko) driver
+#
+
+obj-$(CONFIG_INTEL_DLB) := dlb.o
+
+dlb-objs := dlb_main.o
diff --git a/drivers/misc/dlb/dlb_hw_types.h b/drivers/misc/dlb/dlb_hw_types.h
new file mode 100644
index 000000000000..778ec8665ea0
--- /dev/null
+++ b/drivers/misc/dlb/dlb_hw_types.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_HW_TYPES_H
+#define __DLB_HW_TYPES_H
+
+#define DLB_MAX_NUM_VDEVS			16
+#define DLB_MAX_NUM_DOMAINS			32
+#define DLB_MAX_NUM_LDB_QUEUES			32 /* LDB == load-balanced */
+#define DLB_MAX_NUM_DIR_QUEUES			64 /* DIR == directed */
+#define DLB_MAX_NUM_LDB_PORTS			64
+#define DLB_MAX_NUM_DIR_PORTS			DLB_MAX_NUM_DIR_QUEUES
+#define DLB_MAX_NUM_LDB_CREDITS			8192
+#define DLB_MAX_NUM_DIR_CREDITS			2048
+#define DLB_MAX_NUM_HIST_LIST_ENTRIES		2048
+#define DLB_MAX_NUM_AQED_ENTRIES		2048
+#define DLB_MAX_NUM_QIDS_PER_LDB_CQ		8
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_GROUPS	2
+#define DLB_MAX_NUM_SEQUENCE_NUMBER_MODES	5
+#define DLB_QID_PRIORITIES			8
+#define DLB_NUM_ARB_WEIGHTS			8
+#define DLB_MAX_WEIGHT				255
+#define DLB_NUM_COS_DOMAINS			4
+#define DLB_MAX_CQ_COMP_CHECK_LOOPS		409600
+#define DLB_MAX_QID_EMPTY_CHECK_LOOPS		(32 * 64 * 1024 * (800 / 30))
+#define DLB_HZ					800000000
+#define DLB_FUNC_BAR				0
+#define DLB_CSR_BAR				2
+
+#define PCI_DEVICE_ID_INTEL_DLB_PF		0x2710
+
+#endif /* __DLB_HW_TYPES_H */
diff --git a/drivers/misc/dlb/dlb_main.c b/drivers/misc/dlb/dlb_main.c
new file mode 100644
index 000000000000..fbd77203b398
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.c
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#include <linux/aer.h>
+#include <linux/cdev.h>
+#include <linux/delay.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+
+#include "dlb_main.h"
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Intel(R) Dynamic Load Balancer (DLB) Driver");
+
+static struct class *dlb_class;
+static dev_t dlb_devt;
+static DEFINE_IDR(dlb_ids);
+static DEFINE_SPINLOCK(dlb_ids_lock);
+
+/**********************************/
+/****** PCI driver callbacks ******/
+/**********************************/
+
+static int dlb_probe(struct pci_dev *pdev, const struct pci_device_id *pdev_id)
+{
+	struct dlb *dlb;
+	int ret;
+
+	dlb = devm_kzalloc(&pdev->dev, sizeof(*dlb), GFP_KERNEL);
+	if (!dlb)
+		return -ENOMEM;
+
+	pci_set_drvdata(pdev, dlb);
+
+	dlb->pdev = pdev;
+
+	spin_lock(&dlb_ids_lock);
+	dlb->id = idr_alloc(&dlb_ids, (void *)dlb, 0, DLB_MAX_NUM_DEVICES - 1,
+			    GFP_KERNEL);
+	spin_unlock(&dlb_ids_lock);
+
+	if (dlb->id < 0) {
+		dev_err(&pdev->dev, "probe: device ID allocation failed\n");
+
+		ret = dlb->id;
+		goto alloc_id_fail;
+	}
+
+	ret = pcim_enable_device(pdev);
+	if (ret != 0) {
+		dev_err(&pdev->dev, "pcim_enable_device() returned %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	ret = pcim_iomap_regions(pdev,
+				 (1U << DLB_CSR_BAR) | (1U << DLB_FUNC_BAR),
+				 "dlb");
+	if (ret != 0) {
+		dev_err(&pdev->dev, "pcim_iomap_regions(): returned %d\n", ret);
+
+		goto pci_enable_device_fail;
+	}
+
+	pci_set_master(pdev);
+
+	ret = pci_enable_pcie_error_reporting(pdev);
+	if (ret != 0)
+		dev_info(&pdev->dev, "Failed to enable AER %d\n", ret);
+
+	return 0;
+
+pci_enable_device_fail:
+	spin_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	spin_unlock(&dlb_ids_lock);
+alloc_id_fail:
+	return ret;
+}
+
+static void dlb_remove(struct pci_dev *pdev)
+{
+	struct dlb *dlb = pci_get_drvdata(pdev);
+
+	pci_disable_pcie_error_reporting(pdev);
+
+	spin_lock(&dlb_ids_lock);
+	idr_remove(&dlb_ids, dlb->id);
+	spin_unlock(&dlb_ids_lock);
+}
+
+static struct pci_device_id dlb_id_table[] = {
+	{ PCI_DEVICE_DATA(INTEL, DLB_PF, DLB_PF) },
+	{ 0 }
+};
+MODULE_DEVICE_TABLE(pci, dlb_id_table);
+
+static struct pci_driver dlb_pci_driver = {
+	.name		 = "dlb",
+	.id_table	 = dlb_id_table,
+	.probe		 = dlb_probe,
+	.remove		 = dlb_remove,
+};
+
+static int __init dlb_init_module(void)
+{
+	int err;
+
+	dlb_class = class_create(THIS_MODULE, "dlb");
+
+	if (IS_ERR(dlb_class)) {
+		pr_err("dlb: class_create() returned %ld\n",
+		       PTR_ERR(dlb_class));
+
+		return PTR_ERR(dlb_class);
+	}
+
+	err = alloc_chrdev_region(&dlb_devt, 0, DLB_MAX_NUM_DEVICES, "dlb");
+
+	if (err < 0) {
+		pr_err("dlb: alloc_chrdev_region() returned %d\n", err);
+
+		goto alloc_chrdev_fail;
+	}
+
+	err = pci_register_driver(&dlb_pci_driver);
+	if (err < 0) {
+		pr_err("dlb: pci_register_driver() returned %d\n", err);
+
+		goto pci_register_fail;
+	}
+
+	return 0;
+
+pci_register_fail:
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+alloc_chrdev_fail:
+	class_destroy(dlb_class);
+
+	return err;
+}
+
+static void __exit dlb_exit_module(void)
+{
+	pci_unregister_driver(&dlb_pci_driver);
+
+	unregister_chrdev_region(dlb_devt, DLB_MAX_NUM_DEVICES);
+
+	class_destroy(dlb_class);
+}
+
+module_init(dlb_init_module);
+module_exit(dlb_exit_module);
diff --git a/drivers/misc/dlb/dlb_main.h b/drivers/misc/dlb/dlb_main.h
new file mode 100644
index 000000000000..ab2a2014bafd
--- /dev/null
+++ b/drivers/misc/dlb/dlb_main.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(C) 2016-2020 Intel Corporation. All rights reserved. */
+
+#ifndef __DLB_MAIN_H
+#define __DLB_MAIN_H
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+
+#include "dlb_hw_types.h"
+
+/*
+ * The dlb driver uses a different minor number for each device file, of which
+ * there are:
+ * - 33 per device (PF or VF/VDEV): 1 for the device, 32 for scheduling domains
+ * - Up to 17 devices per PF: 1 PF and up to 16 VFs/VDEVs
+ * - Up to 16 PFs per system
+ */
+#define DLB_MAX_NUM_PFS	  16
+#define DLB_NUM_FUNCS_PER_DEVICE (1 + DLB_MAX_NUM_VDEVS)
+#define DLB_MAX_NUM_DEVICES	 (DLB_MAX_NUM_PFS * DLB_NUM_FUNCS_PER_DEVICE)
+
+enum dlb_device_type {
+	DLB_PF,
+};
+
+struct dlb {
+	struct pci_dev *pdev;
+	int id;
+};
+
+#endif /* __DLB_MAIN_H */
-- 
2.17.1