Should __FD_SETSIZE still be set to 1024 ?

Should __FD_SETSIZE still be set to 1024 ?

[Dag Wieers]

Hi,

Is there a reason for __FD_SETSIZE to be 1024 in
linux/posix_types.h and gnu/types.h ?
Why can't we increase this number by default ?

Shouldn't it be set to the real limit of the kernel ? (And let
applications define their own limit if there is a need for one ?)

PS The LKML faq remarks that increasing this can break select.
Is this still an issue ? Under what conditions ?

Thanks in advance,

--   dag wieers,  dag@wieers.com,  http://dag.wieers.com/   --
«Onder voorbehoud van hetgeen niet uitdrukkelijk wordt erkend»

Re: Should __FD_SETSIZE still be set to 1024 ?

[Chris Wedgwood]

On Fri, Jun 22, 2001 at 04:59:36PM +0200, Dag Wieers wrote:

    Is there a reason for __FD_SETSIZE to be 1024 in
    linux/posix_types.h and gnu/types.h ?
    Why can't we increase this number by default ?

It might break stuff, like things that link with code that assumes it
is only 1024.

    Shouldn't it be set to the real limit of the kernel ?

Nah... the kernel limit is 1024^2 --- you don't want to use select
anywhere near that.

    (And let applications define their own limit if there is a need
    for one ?)

Well, squid and friends do this anyhow.

Not only that, using a greatly increased value should be a run-time
decision, lest you want your code to break on early 2.2.x kernels and
before.




  --cw

Re: Should __FD_SETSIZE still be set to 1024 ?

[Dag Wieers]

On Sat, 23 Jun 2001, Chris Wedgwood wrote:

> On Fri, Jun 22, 2001 at 04:59:36PM +0200, Dag Wieers wrote:
>
>     Is there a reason for __FD_SETSIZE to be 1024 in
>     linux/posix_types.h and gnu/types.h ?
>     Why can't we increase this number by default ?
>
> It might break stuff, like things that link with code that assumes it
> is only 1024.

So if someone wants to increase it for an application he needs to be sure
that everything that it is linked with is compiled with a similar
__FD_SETSIZE ?

Why can you safely increase the value in Squid then ?


>     Shouldn't it be set to the real limit of the kernel ?
>
> Nah... the kernel limit is 1024^2 --- you don't want to use select
> anywhere near that.

Yes, but still, why 1024 ?


>     (And let applications define their own limit if there is a need
>     for one ?)
>
> Well, squid and friends do this anyhow.

No, squid takes the lowest of both (FD_SETSIZE and SQUID_MAXFD) in main.c.
And the Squid configure gets the FD_SETSIZE value from linux/posix_types.h
;(


> Not only that, using a greatly increased value should be a run-time
> decision, lest you want your code to break on early 2.2.x kernels and
> before.

I'm still not convinced that something might break, since everybody
advices to increase __FD_SETSIZE before compiling Squid.

And if linux/posix_types.h defines the limit of open file descriptors of
the system, 1024 is (IMO) a wrong number. But then again, nobody bothered
to change it...

Thanks for your respons.

--   dag wieers,  dag@wieers.com,  http://dag.wieers.com/   --
«Onder voorbehoud van hetgeen niet uitdrukkelijk wordt erkend»

Re: Should __FD_SETSIZE still be set to 1024 ?

[Chris Wedgwood]

On Fri, Jun 22, 2001 at 06:25:41PM +0200, Dag Wieers wrote:

    So if someone wants to increase it for an application he needs to
    be sure that everything that it is linked with is compiled with a
    similar __FD_SETSIZE ?

Well... only things that care about __FD_SETSIZE :)
    
    Why can you safely increase the value in Squid then ?

Because it has been tested and works. I assume libc doesn't care is
this value is increased, not sure about other libraries.

Using libc5 and linux 2.1.9x + scts large-fd patch (which got merged
in later kernels), I manged to work with 100K FDs without too many
problems.

The defintion of 'fd_set' changes with __FD_SETSIZE, so anything
makes these kind of assumptions might break.

For example, if you have some code which assumes this is 1024 and you
try to copy a value into a preallocated array of structure it
defined, then it will break.

So will precompiled code such as:

struct cookie {
	int	type;
	fd_set	chocolate_feeds;
	int	cc_count;
}monster;

sort of thing; because fd_set's size will be wrong.

    Yes, but still, why 1024 ?

Because thats what is has been for a very long time, a safe default.

If your application knows better, it can change it.

    No, squid takes the lowest of both (FD_SETSIZE and SQUID_MAXFD)
    in main.c.  And the Squid configure gets the FD_SETSIZE value
    from linux/posix_types.h ;(

Really? I though for the last couple of years squid would use poll
over select anyhow?
    
    I'm still not convinced that something might break, since everybody
    advices to increase __FD_SETSIZE before compiling Squid.

So do it then.
    
    And if linux/posix_types.h defines the limit of open file
    descriptors of the system, 1024 is (IMO) a wrong number. But then
    again, nobody bothered to change it...

Because 1024 works everywhere, a large or smaller value might not.



  --cw