latency measurements on 2.4.21 (EL 3.0) on SMP (4-cpu)

[Mohit Jaggi <jaggi_mohit@yahoo.com>]

Folks,
I am testing a PCI card that accelerates and offloads
certain functions. I have a multi-threaded real-time
priority (SCHED_FIFO) process to send data to the card
and receive the results. Sending is done by a an
ingress thread(priority=25) and receiving is handled
by an egress thread(priority=35). Using affinity calls
I have made sure that they run concurrently on two
different CPUs. My goal is get the lowest possible
latency from the card. I am measuring it using the
delta of the values returned by gettimeofday() from
ingress thread just before I send the data to the card
and from egress thread just after I receive it. 
While trying various scenarios I noticed that if the
ingress thread send a few messages and then sleeps for
time 't' using usleep(t), then depending on 't' I see
that the latency measured as described above is
different. For example, for t=20k I observe latency of
around 20ms. If I use t=50k I observe 60ms. For 40k  I
see 40ms. Since my delta is not counting this sleep I
find this behaviour quite surprising. I would
appreciate if anyone can venture an explanation.

I have been trying to find out how gettimeofday()
works on SMP linux. Any pointers?

Is there a catch to using real-time priority processes
that I should be aware of? Because the latency
measurements I am doing are of the order of
microseconds I believe I have to have real time
scheduling otherwise time-slicing which is of the
order of milliseconds will mess it up.

Thanks,
Mohit.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com