Networking question

Networking question

[Bloch, Jack]

I have an application which uses a device driver which I wrote to receive
UDP/IP messages. This driver does not use interrupts but polls to see if
messages are available. Once a message is detected I call netif_rx to pass
it up the stack. The application running from user space knows that a
message was received and does a recvfrom on my socket. On a 2.2 Kernel, this
works every time. i.e. I see a message and pass it up the stack and the
recvfrom does indeed get the message from the socket. In a 2.4 environment I
see that netif_rx is using softirq to handle the message as opposed to a BH.
There seems to be a latency introduced because of this. The ksoftirqd runs
at a low priority and my application runs at a high priority (nice value of
-10), Now it seems that the message is not waiting for me when I do a
recvfrom. I do not want to yield my program for too long since the
application is real-time intensive (i.e it must process 30 000msgs/second
which it has been able to do on a 2.2 Kernel). Is there any way to increasy
the priority of the softirq daemon or ensure that it is always awoken when a
netif_rx is called? Please CC me directly on any responses.

Jack Bloch
Siemens Carrier Networks
e-mail    : jack.bloch@icn.siemens.com
phone     : (561) 923-6550

Re: Networking question

[kuznet]

Hello!

> the priority of the softirq daemon or ensure that it is always awoken when a
> netif_rx is called?

You should suppound it with local_bh_disable()/enable(), when using
from process context.

Alexey

Re: Networking question

[Maksim (Max) Krasnyanskiy]

> > the priority of the softirq daemon or ensure that it is always awoken 
> when a
> > netif_rx is called?
>
>You should suppound it with local_bh_disable()/enable(), when using
>from process context.
Actually he should call netif_rx_ni() instead of netif_rx().
_ni stands for non-interrupt context.

Max

RE: Networking question

[Bloch, Jack]

I have tried both methods and they work. I would of course like to use the
most real-time efficient and "Kernel standard" methods. Looking into the
code code of netif_rx_ni seems to be logically the most correct way since it
will trigger do_softirq directly after the call to netif_rx.

Jack Bloch
Siemens Carrier Networks
e-mail    : jack.bloch@icn.siemens.com
phone     : (561) 923-6550


-----Original Message-----
From: Maksim (Max) Krasnyanskiy [mailto:maxk@qualcomm.com]
Sent: Wednesday, July 17, 2002 6:32 PM
To: kuznet@ms2.inr.ac.ru; Bloch, Jack
Cc: linux-kernel@vger.kernel.org
Subject: Re: Networking question



> > the priority of the softirq daemon or ensure that it is always awoken 
> when a
> > netif_rx is called?
>
>You should suppound it with local_bh_disable()/enable(), when using
>from process context.
Actually he should call netif_rx_ni() instead of netif_rx().
_ni stands for non-interrupt context.

Max

Re: Networking question

[kuznet]

Hello!

> code code of netif_rx_ni seems to be logically the most correct way since it
> will trigger do_softirq directly after the call to netif_rx.

Yes, of course. I simply forgot that _ni was evetually added.

Alexey

Networking Question

[Ryan P. Nicholl]

I've been unable to find any Linux API for asynchronously waiting for the TCP send buffer to drain.

The problem I have, in a nutshell, is noted in this part of the documentation:

       If fildes refers to a socket, close() shall cause the socket to
       be destroyed. If the socket is in connection-mode, and the
       SO_LINGER option is set for the socket with non-zero linger time,
       and the socket has untransmitted data, then close() shall block
       for up to the current linger interval until all data is
       transmitted.

Ok, so not good for asynchronous programming, so I could disable the SO_LINGER option, but that leaves me with another problem, namely that I *want* the socket to linger.

The behavior I want is something like, calling "close", getting EAGAIN instead of triggering TCP RST, and something like EPOLLWRITEFLUSHED to wait for the TCP send buffer to be drained. I know neither of these are possible.

Right now the only solution I can think of is to enable SO_LINGER and spawn a thread to run close in, but this might spawn a lot of threads, and doesn't support cancellation well.

Alternatively, I could call getsockopt with TCP_INFO in a loop, but this triggers a lot of wake-ups and might result in sockets hanging around a lot longer than they need to.

I want to allow linger indefinitely on close until some event happens like running out of ram or other resources, basically to intelligently do something like: "OK, we're running low on RAM/resources, time to send RST and drop the send buffer for the 5k worst behaving connections". So unfortunately even with the timeout provided by SO_LINGER, even assuming close would somehow complete in the background, this could be an issue.

Is there any way to do this properly on Linux? If not, any possibility that something like adding EPOLLWRITEFLUSHED would be a welcome addition?

Please CC me on responses.

--
Ryan P. Nicholl

Re: Networking Question

[Bernd Petrovitsch]

Hi all!

On 21/06/2022 02:29, Ryan P. Nicholl wrote:
> I've been unable to find any Linux API for asynchronously waiting for the TCP send buffer to drain.
> 
> The problem I have, in a nutshell, is noted in this part of the documentation:
> 
>         If fildes refers to a socket, close() shall cause the socket to
>         be destroyed. If the socket is in connection-mode, and the

That's not really a Linux kernel question as such (because that should 
work that way on all TCP connections anywhere) but the shutdown() 
syscall is probably what you need:
- your side shuts down the sending part of the socket.
- the other side reads data and gets eventually EOF
- the other side call shutdown() for it's sending side when it's done.
- your side gets EOF.
And then your side knows that no data is in flight.
- finally, you clean up with close(). You can shutdown() the receiving
   side too but doesn't change anything.

[ deleted SO_LINGER stuff - that's for something completelly different ... ]

Kind regards,
	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
      There is NO CLOUD, just other people's computers. - FSFE
                      LUGA : http://www.luga.at

Re: Networking Question

[Ryan P. Nicholl]

Thanks for taking time to respond to this question, but unfortunately Linux's shutdown(2) cannot do what I need. This isn't a general question since POSIX has no standard async API other than AIO which isn't implemented efficiently in GNU/Linux, and I don't think the kernel supports any aio calls directly, instead exposing clone and epoll_* facilities to solve the concurrency issues. However; none of them do what I need efficiently.

In theory, shutdown(2) could be used, the problem is that it's just not efficiently scalable to 100s of thousands or millions of connections. I'm not speaking on behalf of my employeer, but I work for a financial company that processes a lot of network traffic.

What I want essentially is similar to TIME_WAIT but instead of sending an RST packet I want to be notified when the tcp connection is actually drained.

If you call shutdown, my understanding is that you get one of two behaviors:
1. You send RST and immediately discard the send buffer.
2. The call blocks for the SO_LINGER timeout.

Option 1 isn't acceptable because it gives the wrong behavior. Option 2 can be made to work but the downside is that Linux only exposes a synchronous API for it which requires me to have at least 1 thread per shutdown operation. So it could be quite bad if the network is very congested. Unfortunately threads are many times more expensive than sockets/tcp connections. So ideally having few threads and many sockets gives the best use of resources.



--
Ryan P. Nicholl


------- Original Message -------
On Tuesday, June 21st, 2022 at 3:02 AM, Bernd Petrovitsch <bernd@petrovitsch.priv.at> wrote:


> Hi all!
>
> On 21/06/2022 02:29, Ryan P. Nicholl wrote:
>
> > I've been unable to find any Linux API for asynchronously waiting for the TCP send buffer to drain.
> >
> > The problem I have, in a nutshell, is noted in this part of the documentation:
> >
> > If fildes refers to a socket, close() shall cause the socket to
> > be destroyed. If the socket is in connection-mode, and the
>
>
> That's not really a Linux kernel question as such (because that should
> work that way on all TCP connections anywhere) but the shutdown()
> syscall is probably what you need:
> - your side shuts down the sending part of the socket.
> - the other side reads data and gets eventually EOF
> - the other side call shutdown() for it's sending side when it's done.
> - your side gets EOF.
> And then your side knows that no data is in flight.
> - finally, you clean up with close(). You can shutdown() the receiving
> side too but doesn't change anything.
>
> [ deleted SO_LINGER stuff - that's for something completelly different ... ]
>
> Kind regards,
> Bernd
> --
> Bernd Petrovitsch Email : bernd@petrovitsch.priv.at
> There is NO CLOUD, just other people's computers. - FSFE
> LUGA : http://www.luga.at