RFC: ethernet links should remember routes the same as addresses

RFC: ethernet links should remember routes the same as addresses

[Christopher Friesen]

I just wanted to get some opinions on this for possible inclusion in 2.5.  
Alexey, if you have any comments...

The scenario is as follows:

Suppose I have a fancy routing setup, dynamically configured by different
binaries, scripts, etc, complete with multiple addresses per link, additional
routing rules and tables specified using iproute2, etc.

An ethernet driver hangs.  Could be a software bug, an intermittent hardware
issue, whatever.  It can be fixed up by setting the link down and up.

Currently, if I run "ip link set dev ethX down", all routes associated with that
IP address in the additional routing tables are lost.  This is somewhat
understandable, as the addresses are not actually available anymore.  However,
the addresses are still visible associated with the link.  Then I run "ip link
set dev ethX up".  The route in the main routing table comes back, but none of
the other routes do.  Somehow, all of those additional routes must be re-added.

Wouldn't it be nice if we could keep track of these additional routes?  Then you
could simply 'down' and 'up' the link and everything would be back the way it
was before.

Does this sound like a good idea?  How hard would this be to implement (not
knowing what the current code looks like, I don't know how this would be done)?

Chris Friesen


-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: RFC: ethernet links should remember routes the same as addresses

[Colonel]

In clouddancer.list.kernel, you wrote:
>
>
>I just wanted to get some opinions on this for possible inclusion in 2.5.  
>Alexey, if you have any comments...
>
>The scenario is as follows:
>
>Suppose I have a fancy routing setup, dynamically configured by different
>binaries, scripts, etc, complete with multiple addresses per link, additional
>routing rules and tables specified using iproute2, etc.
....
>Does this sound like a good idea?  How hard would this be to implement (not
>knowing what the current code looks like, I don't know how this would be done)?


Most routing daemons handle your problem just fine.  Perhaps you
should look into those daemons first, try 'bird' for instance.


-- 
Windows 2001: "I'm sorry Dave ...  I'm afraid I can't do that."

Re: RFC: ethernet links should remember routes the same as addresses

[Andi Kleen]

"Christopher Friesen" <cfriesen@nortelnetworks.com> writes:

> Suppose I have a fancy routing setup, dynamically configured by different
> binaries, scripts, etc, complete with multiple addresses per link, additional
> routing rules and tables specified using iproute2, etc.
> 
> An ethernet driver hangs.  Could be a software bug, an intermittent hardware
> issue, whatever.  It can be fixed up by setting the link down and up.
> 
> Currently, if I run "ip link set dev ethX down", all routes associated with that
> IP address in the additional routing tables are lost.  This is somewhat
> understandable, as the addresses are not actually available anymore.  However,
> the addresses are still visible associated with the link.  Then I run "ip link
> set dev ethX up".  The route in the main routing table comes back, but none of
> the other routes do.  Somehow, all of those additional routes must be re-added.

ip route list dev device > BACKUP

...

while read i ; do ip route add $i ; done < BACKUP


> 
> Wouldn't it be nice if we could keep track of these additional routes?  Then you
> could simply 'down' and 'up' the link and everything would be back the way it
> was before.
> 
> Does this sound like a good idea?  How hard would this be to implement (not
> knowing what the current code looks like, I don't know how this would be done)?

In kernel very easy. The IP addresses and the driver init/cleanup are 
completely separated and can be easily done independent. You may need
some way to prevent packets getting submitted to the driver (e.g. a
netif_carrier_off but make sure to not confuse it with a real 
netif_carrier_off done by the driver, so you'll likely need a new flag) 

I'm not sure it it worth it though given how easily it is to simulate
in user space. If you really wanted it I guess best would be to add new 
ioctls for it. Coding should be easy.

-Andi

Re: RFC: ethernet links should remember routes the same as addresses

[Christopher Friesen]

Andi Kleen wrote:
> 
> "Christopher Friesen" <cfriesen@nortelnetworks.com> writes:
> > Currently, if I run "ip link set dev ethX down", all routes associated with that
> > IP address in the additional routing tables are lost.  This is somewhat
> > understandable, as the addresses are not actually available anymore.  However,
> > the addresses are still visible associated with the link.  Then I run "ip link
> > set dev ethX up".  The route in the main routing table comes back, but none of
> > the other routes do.  Somehow, all of those additional routes must be re-added.
> 
> ip route list dev device > BACKUP
> 
> ...
> 
> while read i ; do ip route add $i ; done < BACKUP

Unfortunately, this seems to only list the routes in the main routing table, and
these routes are recreated automatically when I bring the link back up.

The problem is that I have routes in multiple other routing tables, and they
don't show up in this command.  I assume I'd have to run a similar command for
each routing table, which is kind of a pain.

If the driver re-init is totally separate from the routing code, is there any
real reason why shutting down the driver *should* remove all routes to that
device?  Maybe the simplest solution would be a new ioctl that would be a link
*reset*...just down/up the link without affecting anything else....


-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: RFC: ethernet links should remember routes the same as addresses

[Ben Greear]

Christopher Friesen wrote:


> If the driver re-init is totally separate from the routing code, is there any
> real reason why shutting down the driver *should* remove all routes to that
> device?  Maybe the simplest solution would be a new ioctl that would be a link
> *reset*...just down/up the link without affecting anything else....


What would want want the down/up to do?  The reason I ask is that there is
an MII-DIAG option to reset the transceiver, if that's all you want to do.

If you want to remove/re-install the driver, then you have to clean up
all the links pointing to it (ie the routes associated with the
device), or you will have all kinds of nasty dangling pointers (logical
or otherwise) to clean up...

Ben

> 
> 
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: RFC: ethernet links should remember routes the same as addresses

[Christopher Friesen]

Ben Greear wrote:
> 
> Christopher Friesen wrote:
> 
> > If the driver re-init is totally separate from the routing code, is there any
> > real reason why shutting down the driver *should* remove all routes to that
> > device?  Maybe the simplest solution would be a new ioctl that would be a link
> > *reset*...just down/up the link without affecting anything else....
> 
> What would want want the down/up to do?  The reason I ask is that there is
> an MII-DIAG option to reset the transceiver, if that's all you want to do.
> 
> If you want to remove/re-install the driver, then you have to clean up
> all the links pointing to it (ie the routes associated with the
> device), or you will have all kinds of nasty dangling pointers (logical
> or otherwise) to clean up...

We've got what looks like a bug in the tulip driver (or maybe buggy hardware)
that under certain circumstances can block with what looks like a problem in the
ring buffer.  Someone is looking at the driver, but in the meantime the only way
to fix it is to do "ip link set dev ethX down" followed by "ip link set dev ethX
up".  Whatever code path this causes in the driver is enough to fix the issue. 
I assume its due to the re-initialization of the buffer descriptors.

On some consideration, maybe it would make more sense in this case to add
something to the driver itself to re-initialize itself rather than involving the
rest of the kernel.  However, I do think that taking down a link and bringing it
back up should be inverse procedures.  If there were routes associated with a
link before taking it down, then they should be associated with it after
bringing it back up, whether or not they stick around when the link is actually
down.  This is how addresses behave, so I think routes should be similar.

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: RFC: ethernet links should remember routes the same as addresses

[Andi Kleen]

On Thu, Nov 29, 2001 at 03:59:52PM -0500, Christopher Friesen wrote:
> Unfortunately, this seems to only list the routes in the main routing table, and
> these routes are recreated automatically when I bring the link back up.

Well, it is not that hard to extend it. 

Just iterate over ip rule ls output too.

(left as an exercise for the reader) 

> If the driver re-init is totally separate from the routing code, is there any
> real reason why shutting down the driver *should* remove all routes to that
> device?  Maybe the simplest solution would be a new ioctl that would be a link
> *reset*...just down/up the link without affecting anything else....

If you just want to do a link reset you can use ethertool today with most
drivers. What is different is a full driver reset to work around software/
hardware bugs.  That would need to be a new ioctl.


-Andi

Re: RFC: ethernet links should remember routes the same as addresses

[Julian Anastasov]

	Hello,

Christopher Friesen wrote:

> the addresses are still visible associated with the link.  Then I run "ip link
> set dev ethX up".  The route in the main routing table comes back, but none of
> the other routes do.  Somehow, all of those additional routes must be re-added.

	One solution is already implemented: static routes

http://www.linuxvirtualserver.org/~julian/
Static, Alternative Routes, Dead Gateway Detection, NAT

Read about the motivation here:
http://www.linuxvirtualserver.org/~julian/dgd-usage.txt

More specific pathes:
00_static_routes-2.2.19-4.diff
00_static_routes-2.4.12-5.diff

Regards

--
Julian Anastasov <ja@ssi.bg>