Monitoring file descriptors without switching context

Monitoring file descriptors without switching context

[Erik Rigtorp]

Hi!

I'm working on a low latency soft real time system. We currently run
on Solaris because it has a lower latency TCP stack and less IPC
overhead. I did some experiments with IPC using pipes on Linux. If i
use non-blocking pipes and do a busy reading loop I can achieve
latencies around 1µs. Using blocking pipes or select()/poll()
latencies increase to about 10µs. In general when I use some blocking
system call latencies increase because my process will be put on a
wait queue and the scheduler will do a context switch. However I have
more cores than I have processes/threads  so I don't mind blocking a
whole core to wait for IO, that way I don't incur a full context
switch when the process waits for IO.

What I really need is some way to have the hardware block and monitor
IO for me. Would it be feasible to implement a special low latency
poll() syscall that uses MONITOR/MWAIT instructions to monitor file
descriptors?

I think in order to utilize new multicore CPUs like the Tileras we
need a better signaling mechanism with lower latency than what we
currently have using the scheduler.

Erik Rigtorp

Re: Monitoring file descriptors without switching context

[Alan Cox]

> What I really need is some way to have the hardware block and monitor
> IO for me. Would it be feasible to implement a special low latency
> poll() syscall that uses MONITOR/MWAIT instructions to monitor file
> descriptors?

The futex locks are probably the lowest level interface you've got for
interlocking between tasks and are very fast. You can also pin tasks to
groups of cpus so you can create an I/O only core for example.

Alan

Re: Monitoring file descriptors without switching context

[Erik Rigtorp]

On Mon, Jan 11, 2010 at 11:59, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> What I really need is some way to have the hardware block and monitor
>> IO for me. Would it be feasible to implement a special low latency
>> poll() syscall that uses MONITOR/MWAIT instructions to monitor file
>> descriptors?
>
> The futex locks are probably the lowest level interface you've got for
> interlocking between tasks and are very fast. You can also pin tasks to
> groups of cpus so you can create an I/O only core for example.
>

For core to core signaling the fastest way is to use a memory polling
loop. Cache control instructions allows me to make sure I use L3 for
intra-cpu communication. The problem is that I also need to monitor
file descriptors and if I can make them fast enough that would be
preferable because I'm also signaled when they close.

Erik