[OT] scatter-gather ops within host memory on a PC

[OT] scatter-gather ops within host memory on a PC

[Mark Frazer]

I'd like to aggregate a bunch of ethernet packet payloads into a single
large buffer after examining the packet headers to determine the order of
the packets.  The data volume makes offloading this to a DMA controller
desirable.

Do PC chipsets have DMA controllers on them to do these scatter-gather
operations?  I know a lot of embedded SoC processors do, but am not
to familiar with the current PC chipsets.

I've glanced at the NFS code but haven't studied it a lot.
rxrpc/transport.c seems to copy some data around, so I assume that linux
uses the host processor to gather the individual ethernet packets into
pages that go into the page cache.

Are there any good books anyone would recommend on PC architecture?

thanks
-mark
-- 
The laws of science be a harsh mistress. - Bender

Re: [OT] scatter-gather ops within host memory on a PC

[Richard B. Johnson]

On Tue, 25 May 2004, Mark Frazer wrote:

> I'd like to aggregate a bunch of ethernet packet payloads into a single
> large buffer after examining the packet headers to determine the order of
> the packets.  The data volume makes offloading this to a DMA controller
> desirable.
>
> Do PC chipsets have DMA controllers on them to do these scatter-gather
> operations?  I know a lot of embedded SoC processors do, but am not
> to familiar with the current PC chipsets.
>
> I've glanced at the NFS code but haven't studied it a lot.
> rxrpc/transport.c seems to copy some data around, so I assume that linux
> uses the host processor to gather the individual ethernet packets into
> pages that go into the page cache.
>
> Are there any good books anyone would recommend on PC architecture?
>
> thanks
> -mark
> --

Scatter-lists with DMA operations are quite common. Many Ethernet
controllers that perform bus-master DMA use a chip that is either
a (PLX) PCI 9080 or pretty-much a clone.

The problem is that DMA does data-transfer to physical locations.
If you only have one physical location (a page in DMA-able memory)
its virtual to physical translation needs to be done only once.
But, if you expect to write to/from user-mode buffers, the translation
needs to be done every time the scatter-list is built. This is not
some simple operation since user's pages are assembled from anywhere
into what looks like contiguous space to the user, but physically
can be anywhere. So somebody has got to scan a bunch of page-tables
looking for the user's pages every time you build the scatter-list,
i.e., every DMA transfer. This will surely take much more time than
the current scheme.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: [OT] scatter-gather ops within host memory on a PC

[Mark Frazer]

Richard B. Johnson <root@chaos.analogic.com> [04/05/25 13:52]:
> Scatter-lists with DMA operations are quite common. Many Ethernet
> controllers that perform bus-master DMA use a chip that is either
> a (PLX) PCI 9080 or pretty-much a clone.

Yup, you're talking about PCI adapter to host scatter gather.  I was
staying within the host memory.  I was wanting to use the standard
ethernet drivers to do the transfer from the cards to the host.  Then
peek at the headers of the rx'd skbufs to set up another scatter-gather
operation to unmix the possibly out-of-order and certainly discontiguous
in host memory packets.

> The problem is that DMA does data-transfer to physical locations.
> If you only have one physical location (a page in DMA-able memory)
> its virtual to physical translation needs to be done only once.
> But, if you expect to write to/from user-mode buffers, the translation
> needs to be done every time the scatter-list is built. This is not
> some simple operation since user's pages are assembled from anywhere
> into what looks like contiguous space to the user, but physically
> can be anywhere. So somebody has got to scan a bunch of page-tables
> looking for the user's pages every time you build the scatter-list,
> i.e., every DMA transfer. This will surely take much more time than
> the current scheme.

You're getting at DIO: get_user_pages, pci_map_sg, sg_dma_address,
sg_dma_len and friends.  I can't quite do that for this application as I
want to reorder the data in the driver and present the application with
the in-order reassembled image.  The destination buffer for the
un-mixing SG operation will be the result of get_user_pages and friends
though.

cheers
-mark
-- 
They're great! They're like sex except I'm having them. - Fry