Monitoring file descriptors without switching context

[Erik Rigtorp <erik@rigtorp.com>]

Hi!

I'm working on a low latency soft real time system. We currently run
on Solaris because it has a lower latency TCP stack and less IPC
overhead. I did some experiments with IPC using pipes on Linux. If i
use non-blocking pipes and do a busy reading loop I can achieve
latencies around 1µs. Using blocking pipes or select()/poll()
latencies increase to about 10µs. In general when I use some blocking
system call latencies increase because my process will be put on a
wait queue and the scheduler will do a context switch. However I have
more cores than I have processes/threads  so I don't mind blocking a
whole core to wait for IO, that way I don't incur a full context
switch when the process waits for IO.

What I really need is some way to have the hardware block and monitor
IO for me. Would it be feasible to implement a special low latency
poll() syscall that uses MONITOR/MWAIT instructions to monitor file
descriptors?

I think in order to utilize new multicore CPUs like the Tileras we
need a better signaling mechanism with lower latency than what we
currently have using the scheduler.

Erik Rigtorp