Re: Pipe buffers' limit of 16 * 4K

["Fausto Richetti Blanco" <fausto.blanco@gmail.com>]

Yes, I need an unspecified amount (it is the data content of a HTTP POST).

The tee system call appeared in the 2.6.17 (I'm using 2.6.9). That's
because I did my implementation using a buffer and copying it using
write. Unfortunately, both solutions suffer from the kernel buffer
limitation.

As I said, there's a lot of other solutions to my specific problem.
I'll problably move my solution inside the lib =/ But I decided to
write to this list to ask for dynamically adjustable sizes for the
pipes' buffers. Is there any good reason for this not to be pushed to
the kernel head ?

On Wed, May 28, 2008 at 7:05 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>
> On Wednesday 2008-05-28 23:22, Fausto Richetti Blanco wrote:
>>>
>>>    It works very well, except when the input has more than 4K (or 16
>>> * 4K in more recent kernels) because the restore_input() blocks at
>>> this limit.
>>>
>>>    I kwow there are other solutions to my problem (e.g: using a
>>> thread, moving our decisions to the lib, etc...) by I'm wondering if
>>> making the pipe buffers' limit adjustable is not a good idea. Maybe it
>>> should be helpful for another things too (like Jan Engelhardt said in
>>> his email).
>>>
>>>    In fact, I didn't find any way of restoring the input (with the
>>> input being the 'read side' of a pipe) other than using pipes. That's
>>> because I've decided to ask this in the linux-kernel list. Is there a
>>> reason for this limit not to be an adjustable parameter ?
>
> You could have a look at the tee(2) system call and see whether it helps
> you a bit. Something along the lines of:
>
>        int pfd[2];
>        pipe(pfd); /* tee() wants an fd... */
>        tee(STDIN_FILENO, pfd[1], len, SPLICE_F_NONBLOCK);
>        read(pfd[0], ..., also in nonblock-mode)
>
> Of course this also has a certain drawback, namely that the pipe will
> only give you as much bytes as it carries, and no more than that,
> because the write side of the pipe at STDIN_FILENO is currently
> blocking exactly because the pipe is full.
> In other words, at most "4K" to be read with tee().
>
> Alternatively, if you need to consume an unspecified amount, it is
> probably best to go the thread way.