Re: Max number of posix queues in vanilla kernel (/proc/sys/fs/mqueue/queues_max)

[m@silodev.com]

Yes, limiting queue count to the memory sounds correct!

Thanks,
Madars

> On 2/7/2014 3:11 PM, Davidlohr Bueso wrote:
>> On Thu, 2014-02-06 at 12:21 +0200, m@silodev.com wrote:
>>> Hi Folks,
>>>
>>> I have recently ported my multi-process application (like a classical
>>> open
>>> system) which uses POSIX Queues as IPC to one of the latest Linux
>>> kernels,
>>> and I have faced issue that number of maximum queues are dramatically
>>> limited down to 1024 (see include/linux/ipc_namespace.h, #define
>>> HARD_QUEUESMAX 1024).
>>>
>>> Previously the max number of queues was INT_MAX (on 64bit system was:
>>> 2147483647).
>>
>> Hmm yes, 1024 is quite unrealistic for some workloads and breaks
>> userspace - I don't see any reasons for _this_ specific value in the
>> changelog or related changes in the patchset that introduced commits
>> 93e6f119 and 02967ea0.
>
> There wasn't a specific selection of that number other than a general
> attempt to make the max more reasonable (INT_MAX isn't really reasonable
> given the overhead of each individual queue, even if the queue number
> and max msg size are small).
>
>> And the fact that this limit is per namespace
>> makes no difference really. Hell, if nothing else, the mq_overview(7)
>> manpage description is evidence enough. For privileged users:
>>
>> The default value for queues_max is 256; it can be changed to any value
>> in the range 0 to INT_MAX.
>
> That was obviously never updated to match the change.
>
> In hindsight, I'm not sure we really even care though.  Since the limit
> on queues is per namespace, and we can make as many namespaces as we
> want, the limit is more or less meaningless and only serves as a
> nuisance to people.  Since we have accounting on a per user basis that
> spans across namespaces and across queues, maybe that should be
> sufficient and the limit on queues should simply be removed and we
> should instead just rely on memory limits.  When the user has exhausted
> their allowed memory usage, whether by large queue sizes, large message
> sizes, or large queue counts, then they are done.  When they haven't,
> they can keep allocating.  Would make things considerably easier and
> would avoid the breakage we are talking about here.
>
>>>
>>> This update imposes bad limits on our multi-process application. As our
>>> app uses approaches that each process opens its own set of queues
>>> (usually
>>> something about 3-5 queues per process). In some scenarios we might run
>>> up
>>> to 3000 processes or more (which of-course for linux is not a problem).
>>> Thus we might need up to 9000 queues or more. All processes run under
>>> one
>>> user.
>>>
>>> But now we have this limit, which limits our software down and we are
>>> getting in trouble. We could patch the kernel manually, but not all
>>> customers are capable of this and willing to do the patching.
>>>
>>> Thus I *kindly* ask you guys to increase this limit to something like
>>> 1M
>>> queues or more (or to technical limit i.e. leave the same INT_MAX).
>
> Technically, INT_MAX isn't (and never was) a valid limit.  Because the
> queue overhead memory size is accounted against the user when creating a
> queue, they can never effectively get to INT_MAX whether it's allowed or
> not.
>
>>> If
>>> user can screw up the system by setting or using maximums, let it leave
>>> to
>>> the user. As it is doing system tuning and he is responsible for kernel
>>> parameters.
>>>
>>> The kernel limit was introduced by:
>>> -
>>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=93e6f119c0ce8a1bba6e81dc8dd97d67be360844
>>>
>>> Also I see other people are claiming issues with this, see:
>>> - https://bugs.launchpad.net/ubuntu/+source/manpages/+bug/1155695 - for
>>> them some database software is not working after the kernel upgrade...
>>
>> Surprised we didn't hear about this earlier by Michael Kerrisk. At least
>> the upstream manpages haven't been updated to reflect this new behavior,
>> it would have been the wrong way to go.
>>
>>>
>>> Also I think that when people will upgrade from RHEL 5 or RHEL 6 to
>>> next
>>> versions where this hard limit will be defined, I suspect that many
>>> will
>>> claim problem about it...
>>
>> Agreed, RHEL 7 will ship with some baseline version of the 3.10 kernel
>> and users will be exposed to this. Of course, the same goes for just
>> about any distro, and Ubuntu users are already complaining about it.
>>
>> I believe that instead of bumping up this HARD limit of 1024, we should
>> go back to the original behavior. If we just increase it, instead, then
>> how high is high enough?
>
> I think it can be removed entirely myself.  The memory limit is really
> all we need worry about unless Viro comes back and says 100s of
> thousands of queues in a single namespace will kill queue lookup or
> something like that.
>
>
>
>