Bug in nonlocal-bind (transparent proxy)?

Bug in nonlocal-bind (transparent proxy)?

[Nadav Har'El]

I am writing a transparent-proxy-like application, that needs to be able to
bind a TCP socket with a non-local address (i.e., the proxy contacts the
origin-server, in the local network, pretending to be the original client.
The reply will get back to the proxy because it acts as the default
gateway, and the kernel needs to pass that reply to the socket).

Bind()ing a non-local address worked fine in the 2.2 line of kernels if a
certain compile-time option was enabled (TRANSPARENT_PROXY, or something
like that). But it no longer seems to be working in the 2.4 kernels (I
tried this on 2.4.2 coming from the Redhat 7.1 distribution).

First, bind() simply refused to work when given a non-local address (returning
EADDRNOTAVAIL). Reading the kernel's source I discovered that an undocumented
"ip_nonlocal_bind" sysctl makes the kernel agree to do such a bind (this
should really be in the bind() documentation...). Enabling this option
allowed bind to work (it can even catch the case of two sockets trying to
bind the same address), but the later connect() fails!
I tryed reading the kernel sources to figure out what's wrong with the
connect(), but failed to understand why it returns a EINVAL. I think this
is a bug, and include below a short program to reproduce it:

If you run the program below, connect() will fail with EINVAL (it will do
so before even trying to output a packet). To see that nothing's actually
wrong with the connect, change the #if 1 to #if 0, eliminating the bind(),
and see that the connect works (or at least fails with a connection refused,
as it should because of the random IP address).
Note that you must run the program as root, and do
	echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind 
to get the bind() to work at all. But once you do that, and bind() works,
how come connect() doesn't work?

Thanks in advance for any insights or fixes!

Here's the program to reproduce the bug:

/* try non-local bind on 2.4 kernel...

   echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
   doesn't seem to help :(
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <errno.h>
#include <sys/socket.h>
#include <arpa/inet.h>

main(){
    int s,c;
    struct sockaddr_in addr;

    s=socket(PF_INET,SOCK_STREAM,0);
    if(s<0)
        perror("socket");

#if 1
    printf("binding %d\n",s);
    addr.sin_family=AF_INET;
    addr.sin_port=htons(5678);
    inet_aton("2.3.4.5",&addr.sin_addr);
    /* this requires echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind */
    if(bind(s, (struct sockaddr *)&addr, sizeof(addr))<0)
        perror("bind");
#endif

    printf("connecting %d\n",s);
    addr.sin_family=AF_INET;
    addr.sin_port=htons(22);
    inet_aton("1.2.3.4",&addr.sin_addr);
    c=connect(s,(struct sockaddr *)&addr, sizeof(addr));
    if(c<0)
        perror("connect");

    printf("end.\n");
}


-- 
Nadav Har'El                        |     Thursday, Jun  7 2001, 16 Sivan 5761
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |(On the back of a VW Beetle) Don't honk,
http://nadav.harel.org.il           |I'm peddling as fast as I can.

Re: Bug in nonlocal-bind (transparent proxy)?

[Andrey Savochkin]

Hi,

It's not a bug, it's willful.

On Thu, Jun 07, 2001 at 05:08:25PM +0300, Nadav Har'El wrote:
> I am writing a transparent-proxy-like application, that needs to be able to
> bind a TCP socket with a non-local address (i.e., the proxy contacts the
> origin-server, in the local network, pretending to be the original client.
> The reply will get back to the proxy because it acts as the default
> gateway, and the kernel needs to pass that reply to the socket).
> 
> Bind()ing a non-local address worked fine in the 2.2 line of kernels if a
> certain compile-time option was enabled (TRANSPARENT_PROXY, or something
> like that). But it no longer seems to be working in the 2.4 kernels (I
> tried this on 2.4.2 coming from the Redhat 7.1 distribution).
> 
> First, bind() simply refused to work when given a non-local address (returning
> EADDRNOTAVAIL). Reading the kernel's source I discovered that an undocumented
> "ip_nonlocal_bind" sysctl makes the kernel agree to do such a bind (this
> should really be in the bind() documentation...). Enabling this option
> allowed bind to work (it can even catch the case of two sockets trying to
> bind the same address), but the later connect() fails!
> I tryed reading the kernel sources to figure out what's wrong with the
> connect(), but failed to understand why it returns a EINVAL. I think this
> is a bug, and include below a short program to reproduce it:
> 
> If you run the program below, connect() will fail with EINVAL (it will do
> so before even trying to output a packet). To see that nothing's actually
> wrong with the connect, change the #if 1 to #if 0, eliminating the bind(),
> and see that the connect works (or at least fails with a connection refused,
> as it should because of the random IP address).
> Note that you must run the program as root, and do
> 	echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind 
> to get the bind() to work at all. But once you do that, and bind() works,
> how come connect() doesn't work?
> 
> Thanks in advance for any insights or fixes!

To make a custom kernel where you can use non-local addresses more freely,
find source address checks in ip_route_output_slow() and get rid of all of
them except considering
	MULTICAST(saddr) || BADCLASS(saddr) || ZERONET(saddr) ||
		saddr == htonl(INADDR_BROADCAST)
as invalid.

	Andrey

Re: Bug in nonlocal-bind (transparent proxy)?

[Nadav Har'El]

On Fri, Jun 08, 2001, Andrey Savochkin wrote about "Re: Bug in nonlocal-bind (transparent proxy)?":
> On Thu, Jun 07, 2001 at 05:08:25PM +0300, Nadav Har'El wrote:
> Hi,
> > Bind()ing a non-local address worked fine in the 2.2 line of kernels if a
> > certain compile-time option was enabled (TRANSPARENT_PROXY, or something
> > like that). But it no longer seems to be working in the 2.4 kernels (I
> > tried this on 2.4.2 coming from the Redhat 7.1 distribution).
> 
> It's not a bug, it's willful.

I don't understand what is willful: why does the ip_nonlocal_bind sysctl
exist if it doesn't help? Getting bind() to work (which is what
ip_nonlocal_bind does) but later not being able to connect() this socket
isn't very useful...

> To make a custom kernel where you can use non-local addresses more freely,
> find source address checks in ip_route_output_slow() and get rid of all of
> them except considering
> 	MULTICAST(saddr) || BADCLASS(saddr) || ZERONET(saddr) ||
> 		saddr == htonl(INADDR_BROADCAST)
> as invalid.

Thanks for the idea - I'll try doing that, and see if it works.

Is there any special reason why, instead of that ip_nonlocal_bind that doesn't
work, we don't have instead a compile-time variable (obviously a run-time
sysctl or socket-specific option would be even better) which enables both
ip_nonlocal_bind and the hacks needed to get connect() to work?

-- 
Nadav Har'El                        |       Friday, Jun  8 2001, 17 Sivan 5761
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |A conscience does not prevent sin. It
http://nadav.harel.org.il           |only prevents you from enjoying it.

Re: Bug in nonlocal-bind (transparent proxy)?

[David S. Miller]

Nadav Har'El writes:
 > I don't understand what is willful: why does the ip_nonlocal_bind sysctl
 > exist if it doesn't help? Getting bind() to work (which is what
 > ip_nonlocal_bind does) but later not being able to connect() this socket
 > isn't very useful...

ip_nonlocal_bind is meant for another purpose all together.
It allows you to bind to IP addresses which aren't configured
up at the moment.

It's meant for dial-on-demand type situations, nothing more.

Later,
David S. Miller
davem@redhat.com

Transparent proxy support in 2.4 - revisited

[Nadav Har'El]

A few months ago, I asked on this list why the transparent proxy feature
(CONFIG_IP_TRANSPARENT_PROXY) that was supported in Linux 2.2 is no longer
supported in Linux 2.4:

> > I am writing a transparent-proxy-like application, that needs to be able to
> > bind a TCP socket with a non-local address (i.e., the proxy contacts the
> > origin-server, in the local network, pretending to be the original client.
> > The reply will get back to the proxy because it acts as the default
> > gateway, and the kernel needs to pass that reply to the socket).

I included an example code, where bind() on a foreign address would work (if
I set ip_nonlocal_bind sysctl to 1), but a later connect() on that socket
would not.

Andrey Savochkin wrote a useful reply, on how to make the connect work():

> To make a custom kernel where you can use non-local addresses more freely,
> find source address checks in ip_route_output_slow() and get rid of all of
> them except considering
> 	MULTICAST(saddr) || BADCLASS(saddr) || ZERONET(saddr) ||
> 		saddr == htonl(INADDR_BROADCAST)
> as invalid.

I did that, and indeed now connect() works, and sends out (when considering
TCP) the appropriate SYN packet.

Unfortunately, that's not enough. When the return SYN-ACK packet carrying a
non-local destination address is received (in practice, the transparent
proxy machine is acting as a default gateway to the other machine), this
packet is either ignored or forwarded out (depending on ip_forward), but is
never accepted as a local packet and transfered to the appropriate socket
as it should.
To do that, some cache of open sockets will need to be consulted for
incoming packets that are not destined to local addresses (whether or not
ip_forward is enabled). This was done in kernel 2.2, where I found
CONFIG_IP_TRANSPARENT_PROXY-related changes in several files (af_inet.c,
icmp.c, ip_forward.c, ip_fw.c, ip_input.c, raw.c, route.c, syncookies.c,
tcp_ipv4.c and udp.c).

As I'm not an experienced Linux kernel hacker, and I'm barely familar with
Linux's networking code, so I hesitate to attempt such a feat (resurrecting
the transparent proxy code) myself, so I was wondering if anyone else had any
plans of doing that?

If not, does anyone know of a reason why this feature was abandoned on the
change to Linux 2.4: Was it a deliberate decision (e.g., because of performance
reasons, because it blurred the distinction between layer 2 and 3 when routing,
etc.) or simply a matter of priorities (e.g., adding transparent proxy support
would take up valuable kernel programmer time and was not deemed important
enough)?

Thanks,
	Nadav.

-- 
Nadav Har'El                        |        Monday, Sep  3 2001, 15 Elul 5761
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |It's fortunate I have back luck - without
http://nadav.harel.org.il           |it I would have no luck at all!

Re: Transparent proxy support in 2.4 - revisited

[Andrey Savochkin]

Hi,

On Mon, Sep 03, 2001 at 01:12:40PM +0300, Nadav Har'El wrote:
> A few months ago, I asked on this list why the transparent proxy feature
> (CONFIG_IP_TRANSPARENT_PROXY) that was supported in Linux 2.2 is no longer
> supported in Linux 2.4:
> 
[snip]
> Andrey Savochkin wrote a useful reply, on how to make the connect work():
> 
> > To make a custom kernel where you can use non-local addresses more freely,
> > find source address checks in ip_route_output_slow() and get rid of all of
> > them except considering
> > 	MULTICAST(saddr) || BADCLASS(saddr) || ZERONET(saddr) ||
> > 		saddr == htonl(INADDR_BROADCAST)
> > as invalid.
> 
> I did that, and indeed now connect() works, and sends out (when considering
> TCP) the appropriate SYN packet.
> 
> Unfortunately, that's not enough. When the return SYN-ACK packet carrying a
> non-local destination address is received (in practice, the transparent
> proxy machine is acting as a default gateway to the other machine), this
> packet is either ignored or forwarded out (depending on ip_forward), but is
> never accepted as a local packet and transfered to the appropriate socket
> as it should.

Right.
In 2.2 kernel you also needed to configure which incoming packets you want to
handle locally.

If you want to handle locally all packets destined to a specific IP address,
just add local route.
If you want some complex matching rules, check iptables, there was something
about "redirects" there.  Alternatively, you may set up policy routing, but
it'll be considerably more difficult.

You seemed to start to solve your problems from the wrong end.
First of all, decide how to handle incoming packets.
Then consider outgoing.

	Andrey

Re: Transparent proxy support in 2.4 - revisited

[Nadav Har'El]

On Mon, Sep 03, 2001, Andrey Savochkin wrote about "Re: Transparent proxy support in 2.4 - revisited":
> > Unfortunately, that's not enough. When the return SYN-ACK packet carrying a
> > non-local destination address is received (in practice, the transparent
> > proxy machine is acting as a default gateway to the other machine), this
> > packet is either ignored or forwarded out (depending on ip_forward), but is
> > never accepted as a local packet and transfered to the appropriate socket
> > as it should.
> 
> Right.
> In 2.2 kernel you also needed to configure which incoming packets you want to
> handle locally.

If I remember correctly, no such configuration was needed if the
CONFIG_IP_TRANSPARENT_PROXY compilation flag was enabled. As I understand from
a cursory examination of the 2.2 source, Whenever packets were received and
were not meant for the local address, a check would be made if they meant for
one of the transparent-proxied connections (i.e., connections whose local
endpoint isn't local).

This is the kind of thing I need in Linux 2.4 too.
I'm still puzzled by the fact that this support simply disappeared between
2.2 and 2.4, and nobody seems to know why (or people who know why don't
reply).

> If you want to handle locally all packets destined to a specific IP address,
> just add local route.
> If you want some complex matching rules, check iptables, there was something
> about "redirects" there.

Remember that the reverse proxy machine, the one faking connections with an
adjacent server (as if they are coming from the actual clients) also serves
as the the gateway for that server and needs to forward packets for it.
So I can't redirect *all* packets to local sockets, yet I also can't pick
specific IP addresses to redirect (unless I write some sort of hack to
modify the iptables tables dynamically as new non-local bind()s happen).
Not to mention that the standard redirect, which also rewrites the destination
address on the packet (if I remember correctly), isn't quite what I need when
I already have a socket bound to a non-local address.


> You seemed to start to solve your problems from the wrong end.
> First of all, decide how to handle incoming packets.
> Then consider outgoing.

You're right. But I started at the easier end :)



-- 
Nadav Har'El                        |        Monday, Sep  3 2001, 15 Elul 5761
nyh@math.technion.ac.il             |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |I'm a peripheral visionary: I see into
http://nadav.harel.org.il           |the future, but mostly off to the sides.

Re: Transparent proxy support in 2.4 - revisited

[Andrey Savochkin]

Hi,

On Mon, Sep 03, 2001 at 04:16:21PM +0300, Nadav Har'El wrote:
> On Mon, Sep 03, 2001, Andrey Savochkin wrote about "Re: Transparent proxy support in 2.4 - revisited":
> > In 2.2 kernel you also needed to configure which incoming packets you want to
> > handle locally.
> 
> If I remember correctly, no such configuration was needed if the
> CONFIG_IP_TRANSPARENT_PROXY compilation flag was enabled. As I understand from
> a cursory examination of the 2.2 source, Whenever packets were received and
> were not meant for the local address, a check would be made if they meant for
> one of the transparent-proxied connections (i.e., connections whose local
> endpoint isn't local).

I don't remember for sure, but packet processing layering suggests that you
still needed to do something.  Otherwise the packet will be forwarded on the
routing level and will never reach socket layer.
So, ipchains rule was still necessary.
The really difference of 2.2 was amasingly complex socket lookup procedure
with CONFIG_IP_TRANSPARENT_PROXY turned on.

> This is the kind of thing I need in Linux 2.4 too.
> I'm still puzzled by the fact that this support simply disappeared between
> 2.2 and 2.4, and nobody seems to know why (or people who know why don't
> reply).

It had been broken in 2.2 for months and nobody repaired it => nobody needed
it.  I don't know, whether it works now or not.
The implementation was crooked, difficult to understand and maintain...

> > If you want to handle locally all packets destined to a specific IP address,
> > just add local route.
> > If you want some complex matching rules, check iptables, there was something
> > about "redirects" there.
> 
> Remember that the reverse proxy machine, the one faking connections with an
> adjacent server (as if they are coming from the actual clients) also serves
> as the the gateway for that server and needs to forward packets for it.

How are you going to determine whether the packet is destined to you or two
the real server?
That's the main question.

> So I can't redirect *all* packets to local sockets, yet I also can't pick
> specific IP addresses to redirect (unless I write some sort of hack to
> modify the iptables tables dynamically as new non-local bind()s happen).
> Not to mention that the standard redirect, which also rewrites the destination
> address on the packet (if I remember correctly), isn't quite what I need when

If it's true, the redirecting module is the place to change the policy to what
you need.
If the module always rewrites the destination IP address, and it can't be
turned off, it's certainly a misfeature.
Make it conditional, or just make a quick hack for yourself, or copy the
existing redirect module, fix it and use the new module.

> I already have a socket bound to a non-local address.

Best regards
		Andrey

Re: Transparent proxy support in 2.4 - revisited

[Julio Sanchez Fernandez]

Andrey Savochkin <saw@saw.sw.com.sg> writes:

> It had been broken in 2.2 for months and nobody repaired it => nobody needed
> it.  I don't know, whether it works now or not.

Broken?  It worked for me on all kernels I tried from RH, I currently
run it on 2.2.19.

> How are you going to determine whether the packet is destined to you or two
> the real server?
> That's the main question.

Maybe follow the method used by NAT now?  I mean, I presume no one
wants to go back to the old implementation...

> If the module always rewrites the destination IP address, and it can't be
> turned off, it's certainly a misfeature.
> Make it conditional, or just make a quick hack for yourself, or copy the
> existing redirect module, fix it and use the new module.

Well, that's another one to get used to, the old code made the output
from netstat or lsof extremely informative, something that is pretty
much lost now in 2.4.x.  Right?

Julio

Re: Transparent proxy support in 2.4 - revisited

[Nerijus Baliunas]

NH> This is the kind of thing I need in Linux 2.4 too.
NH> I'm still puzzled by the fact that this support simply disappeared between
NH> 2.2 and 2.4, and nobody seems to know why (or people who know why don't
NH> reply).

Please look at the archives here: http://lists.samba.org/mailman/listinfo/netfilter-devel

Regards,
Nerijus

Re: Transparent proxy support in 2.4 - revisited

[Alexey Kuznetsov]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 11006 bytes --]

Hello!

> As I'm not an experienced Linux kernel hacker, and I'm barely familar with
> Linux's networking code, so I hesitate to attempt such a feat (resurrecting
> the transparent proxy code) myself, so I was wondering if anyone else had any
> plans of doing that?

Well, the patch which I used for this purpose, is enclosed.

The patch does not contain line óONFIG_IP_NONLOCAL_CONNECT in Config.in,
so that you have to add it yourself. Also it requires to call
setsockopt(..., SOL_IP, IP_NONLOCAL, 1, ) before binding to foreign
address.


> If not, does anyone know of a reason why this feature was abandoned 

Not __this__ feature was abandoned. The feature to __listen__ foreign
addresses and redirecting them to another port inside TCP i.e.
"transparent proxÕ" was moved to netfilter (not abandoned!).
Its implementation inside TCP even in 2.2 was full of bugs and
in 2.4 it is just impossible.

What's about active connect(), using nonlocal address, it was not abandoned.
Actually, I even was not aware of this feature and consider its presence
in 2.2 as very hard bug. It was not supposed to work and I have no idea
who opened this ugly hole. Even the person was me, it was misprint. :-)

But, actually, if someone reminded about this in time it could be repaired
in 2.2 in way proposed in the enclosed patch. I also would prefer to have it
in core, it is useful.

[ Dave, this forward is purely informational, of course. It is not a patch
 to apply :-) ]

Alexey


diff -ur ../vger3-010728/linux/include/linux/in.h linux/include/linux/in.h
--- ../vger3-010728/linux/include/linux/in.h	Thu Oct 26 18:06:21 2000
+++ linux/include/linux/in.h	Sun Aug  5 22:37:54 2001
@@ -67,6 +67,7 @@
 #define	IP_RECVTOS	13
 #define IP_MTU		14
 #define IP_FREEBIND	15
+#define IP_NONLOCAL	16
 
 /* BSD compatibility */
 #define IP_RECVRETOPTS	IP_RETOPTS
diff -ur ../vger3-010728/linux/include/net/route.h linux/include/net/route.h
--- ../vger3-010728/linux/include/net/route.h	Fri Jul 20 22:12:10 2001
+++ linux/include/net/route.h	Sun Aug  5 22:34:40 2001
@@ -36,10 +36,17 @@
 #endif
 
 #define RTO_ONLINK	0x01
-#define RTO_TPROXY	0x80000000
+#define RTO_TPROXY	0x8000
 
 #define RTO_CONN	0
 
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+#define RTO_SCONN(sk)	((sk)->reuse > 1 ? RTO_TPROXY : 0)
+#else
+#define RTO_SCONN(sk)	0
+#endif
+#define RT_CONN_FLAGS(sk)   (RT_TOS(sk->protinfo.af_inet.tos) | RTO_SCONN(sk) | sk->localroute)
+
 struct rt_key
 {
 	__u32			dst;
@@ -49,7 +56,7 @@
 #ifdef CONFIG_IP_ROUTE_FWMARK
 	__u32			fwmark;
 #endif
-	__u8			tos;
+	__u16			tos;
 	__u8			scope;
 };
 
diff -ur ../vger3-010728/linux/net/ipv4/af_inet.c linux/net/ipv4/af_inet.c
--- ../vger3-010728/linux/net/ipv4/af_inet.c	Wed Jun 13 21:14:05 2001
+++ linux/net/ipv4/af_inet.c	Sun Aug  5 23:24:15 2001
@@ -485,6 +485,24 @@
 
 	chk_addr_ret = inet_addr_type(addr->sin_addr.s_addr);
 
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+	if (sk->reuse > 1) {
+		if (chk_addr_ret != RTN_UNICAST) {
+			/* Not a foreign address really, hence port is ours. */ 
+			sk->reuse = 1;
+		} else {
+			if (!addr->sin_port)
+				return -EINVAL;
+			printk(KERN_DEBUG "bind to nonlocal\n");
+			sk->protinfo.af_inet.freebind = 1;
+		}
+	} else {
+		/* Not priviledged freebind option is not compatible with
+		 * nonlocal connect. */
+		sk->protinfo.af_inet.freebind = 0;
+	}
+#endif
+
 	/* Not specified by any standard per-se, however it breaks too
 	 * many applications when removed.  It is unfortunate since
 	 * allowing applications to make a non-local bind solves
diff -ur ../vger3-010728/linux/net/ipv4/ip_forward.c linux/net/ipv4/ip_forward.c
--- ../vger3-010728/linux/net/ipv4/ip_forward.c	Wed Dec 13 22:23:13 2000
+++ linux/net/ipv4/ip_forward.c	Sun Aug  5 22:01:02 2001
@@ -84,6 +84,15 @@
 	if (skb->pkt_type != PACKET_HOST)
 		goto drop;
 
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+	/* Could be done with a netfilter hook. Not clear how exactly. */
+	if (skb->nh.iph->protocol == IPPROTO_TCP) {
+		if ((skb = tcp_v4_nonlocal_deliver(skb)) == NULL)
+			return NET_RX_SUCCESS;
+		opt = &(IPCB(skb)->opt);
+	}
+#endif
+
 	skb->ip_summed = CHECKSUM_NONE;
 	
 	/*
diff -ur ../vger3-010728/linux/net/ipv4/ip_output.c linux/net/ipv4/ip_output.c
--- ../vger3-010728/linux/net/ipv4/ip_output.c	Fri Jul 20 22:12:11 2001
+++ linux/net/ipv4/ip_output.c	Sun Aug  5 22:01:02 2001
@@ -357,7 +357,7 @@
 		 * out.
 		 */
 		if (ip_route_output(&rt, daddr, sk->saddr,
-				    RT_TOS(sk->protinfo.af_inet.tos) | RTO_CONN | sk->localroute,
+				    RT_CONN_FLAGS(sk),
 				    sk->bound_dev_if))
 			goto no_route;
 		__sk_dst_set(sk, &rt->u.dst);
diff -ur ../vger3-010728/linux/net/ipv4/ip_sockglue.c linux/net/ipv4/ip_sockglue.c
--- ../vger3-010728/linux/net/ipv4/ip_sockglue.c	Tue Feb 20 22:13:53 2001
+++ linux/net/ipv4/ip_sockglue.c	Sun Aug  5 23:24:57 2001
@@ -392,7 +392,7 @@
 			    (1<<IP_RETOPTS) | (1<<IP_TOS) | 
 			    (1<<IP_TTL) | (1<<IP_HDRINCL) | 
 			    (1<<IP_MTU_DISCOVER) | (1<<IP_RECVERR) | 
-			    (1<<IP_ROUTER_ALERT) | (1<<IP_FREEBIND))) || 
+			    (1<<IP_ROUTER_ALERT) | (1<<IP_FREEBIND) | (1<<IP_NONLOCAL))) || 
 				optname == IP_MULTICAST_TTL || 
 				optname == IP_MULTICAST_LOOP) { 
 		if (optlen >= sizeof(int)) {
@@ -624,6 +624,17 @@
 			sk->protinfo.af_inet.freebind = !!val; 
 	                break;			
  
+		case IP_NONLOCAL:
+			if (optlen<1)
+				goto e_inval;
+			if (!capable(CAP_NET_ADMIN))
+				goto e_inval;
+			if (val)
+				sk->reuse = 2;
+			else if (sk->reuse == 2)
+				sk->reuse = 1;
+	                break;			
+
 		default:
 #ifdef CONFIG_NETFILTER
 			err = nf_setsockopt(sk, PF_INET, optname, optval, 
diff -ur ../vger3-010728/linux/net/ipv4/route.c linux/net/ipv4/route.c
--- ../vger3-010728/linux/net/ipv4/route.c	Fri Jul 20 22:12:11 2001
+++ linux/net/ipv4/route.c	Sun Aug  5 23:24:15 2001
@@ -1689,6 +1689,16 @@
 
 /*
  * Major route resolver routine.
+ *
+ * NOTE: about CONFIG_IP_NONLOCAL_CONNECT. Here it concides to
+ * CONFIG_IP_TRANSPARENT_PROXY used in linux-2.2. However, 
+ * [ Also, I constructed new song: "Yesterday, all my troubles..."
+ *   Whaat?! Well, words are the same, music may be the same too. Why not?
+ *   But its sense is absolutely different!
+ * ]
+ * jokes apart, it is _not_ used to steal connections, only
+ * connections opened by us are nonlocal, hence we have
+ * no problems with TCP port shifting etc.
  */
 
 int ip_route_output_slow(struct rtable **rp, const struct rt_key *oldkey)
@@ -1728,8 +1738,19 @@
 
 		/* It is equivalent to inet_addr_type(saddr) == RTN_LOCAL */
 		dev_out = ip_dev_find(oldkey->src);
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+		/* If address is not local, test for nonlocal flag;
+		 * if address is local --- clear the flag.
+		 */
+		if (dev_out == NULL) {
+			if (!(oldkey->tos & RTO_TPROXY) || inet_addr_type(oldkey->src) != RTN_UNICAST)
+				goto out;
+			flags |= RTCF_TPROXY;
+		}
+#else
 		if (dev_out == NULL)
 			goto out;
+#endif
 
 		/* I removed check for oif == dev_out->oif here.
 		   It was wrong by three reasons:
@@ -1740,6 +1761,9 @@
 		 */
 
 		if (oldkey->oif == 0
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+		    && dev_out
+#endif
 		    && (MULTICAST(oldkey->dst) || oldkey->dst == 0xFFFFFFFF)) {
 			/* Special hack: user can direct multicasts
 			   and limited broadcast via necessary interface
diff -ur ../vger3-010728/linux/net/ipv4/syncookies.c linux/net/ipv4/syncookies.c
--- ../vger3-010728/linux/net/ipv4/syncookies.c	Fri May 11 21:40:10 2001
+++ linux/net/ipv4/syncookies.c	Sun Aug  5 22:01:12 2001
@@ -178,7 +178,7 @@
 			    opt && 
 			    opt->srr ? opt->faddr : req->af.v4_req.rmt_addr,
 			    req->af.v4_req.loc_addr,
-			    sk->protinfo.af_inet.tos | RTO_CONN,
+			    RT_TOS(sk->protinfo.af_inet.tos) | RTO_CONN,
 			    0)) { 
 		tcp_openreq_free(req);
 		goto out; 
diff -ur ../vger3-010728/linux/net/ipv4/tcp_ipv4.c linux/net/ipv4/tcp_ipv4.c
--- ../vger3-010728/linux/net/ipv4/tcp_ipv4.c	Wed Apr 25 21:02:18 2001
+++ linux/net/ipv4/tcp_ipv4.c	Sun Aug  5 23:24:15 2001
@@ -213,6 +213,8 @@
 				break;
 	}
 	if (tb != NULL && tb->owners != NULL) {
+		if (sk->reuse > 1)
+			goto success;
 		if (tb->fastreuse != 0 && sk->reuse != 0 && sk->state != TCP_LISTEN) {
 			goto success;
 		} else {
@@ -221,6 +223,7 @@
 
 			for( ; sk2 != NULL; sk2 = sk2->bind_next) {
 				if (sk != sk2 &&
+				    sk2->reuse <= 1 &&
 				    sk->bound_dev_if == sk2->bound_dev_if) {
 					if (!sk_reuse	||
 					    !sk2->reuse	||
@@ -660,7 +663,8 @@
 	}
 
 	tmp = ip_route_connect(&rt, nexthop, sk->saddr,
-			       RT_TOS(sk->protinfo.af_inet.tos)|RTO_CONN|sk->localroute, sk->bound_dev_if);
+			       RT_CONN_FLAGS(sk),
+			       sk->bound_dev_if);
 	if (tmp < 0)
 		return tmp;
 
@@ -676,7 +680,7 @@
 		daddr = rt->rt_dst;
 
 	err = -ENOBUFS;
-	buff = alloc_skb(MAX_TCP_HEADER + 15, GFP_KERNEL);
+	buff = alloc_skb(MAX_TCP_HEADER + 15, sk->allocation);
 
 	if (buff == NULL)
 		goto failure;
@@ -1704,6 +1708,44 @@
 	goto discard_it;
 }
 
+#ifdef CONFIG_IP_NONLOCAL_CONNECT
+/* Could be done with netfilter hook. Not clear how to hook this in right place. */
+
+struct sk_buff *tcp_v4_nonlocal_deliver(struct sk_buff *skb)
+{
+	struct sock *sk;
+	struct tcphdr *th;
+	int ihl;
+
+	if (skb->nh.iph->frag_off & htons(IP_MF|IP_OFFSET)) {
+		skb = ip_defrag(skb);
+		if (!skb)
+			return NULL;
+	}
+
+	ihl = skb->nh.iph->ihl*4;
+
+	if (!pskb_may_pull(skb, ihl+8))
+		goto out;
+
+	th = (struct tcphdr*)(skb->nh.raw + ihl);
+
+	sk = __tcp_v4_lookup_established(skb->nh.iph->saddr, th->source,
+					 skb->nh.iph->daddr, ntohs(th->dest),
+					 0);
+	if (sk) {
+		sock_put(sk);
+		ip_local_deliver(skb);
+		return NULL;
+	}
+
+out:
+	return skb;
+}
+#endif
+
+
+
 /* With per-bucket locks this operation is not-atomic, so that
  * this version is not worse.
  */
@@ -1776,7 +1818,7 @@
 		daddr = sk->protinfo.af_inet.opt->faddr;
 
 	err = ip_route_output(&rt, daddr, sk->saddr,
-			      RT_TOS(sk->protinfo.af_inet.tos) | RTO_CONN | sk->localroute,
+			      RT_CONN_FLAGS(sk),
 			      sk->bound_dev_if);
 	if (!err) {
 		__sk_dst_set(sk, &rt->u.dst);
diff -ur ../vger3-010728/linux/net/ipv4/udp.c linux/net/ipv4/udp.c
--- ../vger3-010728/linux/net/ipv4/udp.c	Wed Mar  7 22:17:47 2001
+++ linux/net/ipv4/udp.c	Sun Aug  5 22:01:15 2001
@@ -724,7 +724,7 @@
 	sk_dst_reset(sk);
 
 	err = ip_route_connect(&rt, usin->sin_addr.s_addr, sk->saddr,
-			       sk->protinfo.af_inet.tos|sk->localroute, sk->bound_dev_if);
+			       RT_TOS(sk->protinfo.af_inet.tos), sk->bound_dev_if);
 	if (err)
 		return err;
 	if ((rt->rt_flags&RTCF_BROADCAST) && !sk->broadcast) {
diff -ur ../vger3-010728/linux/net/ipv6/tcp_ipv6.c linux/net/ipv6/tcp_ipv6.c
--- ../vger3-010728/linux/net/ipv6/tcp_ipv6.c	Wed Jun 13 21:14:05 2001
+++ linux/net/ipv6/tcp_ipv6.c	Sun Aug  5 22:01:18 2001
@@ -656,7 +656,7 @@
 	tp->mss_clamp = IPV6_MIN_MTU - sizeof(struct tcphdr) - sizeof(struct ipv6hdr);
 
 	err = -ENOBUFS;
-	buff = alloc_skb(MAX_TCP_HEADER + 15, GFP_KERNEL);
+	buff = alloc_skb(MAX_TCP_HEADER + 15, sk->allocation);
 
 	if (buff == NULL)
 		goto failure;