Re: openbkweb-0.0

[Willy Tarreau <willy@w.ods.org>]

On Wed, Feb 05, 2003 at 08:37:37PM -0800, Larry McVoy wrote:
> > You may want to enable mod_deflate, and then scripts can easily make use
> > of gzip compressed data. May not be an end-all, but something to
> > consider.
> 
> Gzip will give 4:1 what these scripts are doing is more like 1000:1.  
> So gzipping the data gets you down to 250:1.  That's still way more
> bandwidth, way too much to be acceptable.

Larry, would it be acceptable/possible to regularly push some data/metadata
to sites like kernel.org that people already consult for kernel development ?
This way, Andrea's tool would only have to check kernel.org, and not bkbits.net.

Another solution is to fetch from a reverse proxy-cache on a high-bandwidth
site, provided that we know what to cache, of course. This could even reduce
your current HTTP usage since nearly everything should be cacheable for a very
long period.

You know, sometimes I fetch changesets from bkbits.net with my browser, and
I later convert them from html to text with a very trivial sed script. And it
happens that I remember you saying that the bandwidth costs you very much, then
I feel a bit guilty (although about once a week may not be too much) and I
wonder what would happen if everyone did the same regularly.

You may also try to cap the bandwidth from the web server, to dissuade people
from using it, but still not closing it. This could also help you not to pay
for the extra bytes.

Just some suggestions, of course. I wouldn't like your http service to be
closed since I sometimes use it.

Cheers,
Willy