mq_open() fails with ENOMEM for 'large' message sizes?

[Clem Taylor <clem.taylor@gmail.com>]

I've been having problems with mq_open() returning ENOMEM when trying
to allocate a large message queue (I want to open a queue with 3
1mbyte messages).

I increased the size of the sysctl limits:
fs.mqueue.msgsize_max = 1052672 (1M+4K)
fs.mqueue.msg_max = 256
fs.mqueue.queues_max = 256

On a 2.6.12 (i686) system with 2G of memory (with large swap), I can't
seem to create a single message queue with a message larger then 799K
and a 3 message queue with a message larger then 266K.

On a 2.6.13 (mips32) system with 64M of memory (no swap), I can't seem
to create a single message queue with a message larger then about
606K.

Any ideas where this limit is coming from? The maximum message size
doesn't seem to be proportional to the amount of physical memory. Is
there some additional tunable I need to adjust that would allow me to
create a message queue with larger messages?

                        Thanks,
                        Clem Taylor