Re: [PATCH net] bonding: fix missing down path on speed/duplex failure after FAIL->UP

[Jay Vosburgh <jv@jvosburgh.net>]

Timur Stukan <timur.stukan.linux@gmail.com> wrote:

>In bond_miimon_commit(), when a slave is proposed as BOND_LINK_UP but
>bond_update_speed_duplex() fails in bonding mode that needs speed/duplex
>(802.3ad, tlb, alb), the slave is set to BOND_LINK_DOWN directly. This
>bypasses bond_set_slave_link_state(), bond_miimon_link_change() and
>failover.
>
>This is fine for slave that coming from BOND_LINK_BACK, which was never
>brought up and is not in use. But since commit 1899bb325149 ("bonding:
>fix state transition issue in link monitoring") the commit phase also
>handles BOND_LINK_FAIL -> BOND_LINK_UP which means a slave that is still
>in use and may be curr_active_slave.
>
>If the link recovers within downdelay and goes down again before the
>commit phase reads speed/duplex, the slave is silently marked down while
>remaining the active slave in worst cases. In alb/tlb the bond MAC stays
>on the dead port, no failover happens and TLB/RLB state is not cleaned,
>so the bond stops passing traffic until that link comes back even though
>other slaves are up. A non-active slave is affected too, in alb, RLB
>clients assigned to it are not reassigned and become unreachable.
>
>Reproduced in tlb and alb mode with miimon=100 and downdelay=10000 by
>flapping a slave within downdelay and dropping it again before commit
>(a longer delay was added between inspect and commit to make window
>wider and easier to test):
>  bond0: (slave enp0s3): link status down for interface, disabling it in 10000 ms
>  bond0: (slave enp0s3): link status up again after 3400 ms
>  bond0: (slave enp0s3): failed to get link speed/duplex
>
>procfs then shows the active slave with "MII Status: down" and "Link
>Failure Count: 0", and all traffic is lost.
>
>Fix this by recording the link state before the miimon commit and when
>the speed/duplex update fails for a slave coming from BOND_LINK_FAIL,
>taking it through the regular down path, including failover. The
>BOND_LINK_BACK case keeps the current behaviour. The down path is moved
>into bond_miimon_slave_down() so both callers share it.
>
>After the fix:
>  bond0: (slave enp0s3): failed to get link speed/duplex
>  bond0: (slave enp0s3): link status definitely down, disabling slave
>  bond0: (slave enp0s4): making interface the new active one
>
>Fixes: 1899bb325149 ("bonding: fix state transition issue in link monitoring")
>Signed-off-by: Timur Stukan <timur.stukan.linux@gmail.com>
>---

	Reading the description and patch, I'm concerned that this patch
is introducing the same sort of issue that 1899bb325149 was solving: two
state variables that end up becoming unsynchronized.

	For example, here you set link_prev_state in one place, inspect
it in another, but never clear it.  Thus, once set, it will remain set
forever, raising the possibility of false matches in the future.  Not
necessarily from this patch, which appears to always set then check in
that order, but from subsequent changes down the road that inadvertently
break this guarantee.

	Yes, that can be resolved by clearing it after use, but I'd
first ask whether there is an effective solution that does not require
keeping additional state?

	-J

>
>Notes:
>    sizeof(struct slave) grows from 608 to 616 bytes due to padding.
>    I kept link_prev_state next to link/link_new_state for readability;
>    it can be moved into the hole after 'delay' if preferred.
>
> drivers/net/bonding/bond_main.c | 53 ++++++++++++++++++++++-----------
> include/net/bonding.h           |  1 +
> 2 files changed, 36 insertions(+), 18 deletions(-)
>
>diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
>index 09aa45d71fab..602ad697a9c2 100644
>--- a/drivers/net/bonding/bond_main.c
>+++ b/drivers/net/bonding/bond_main.c
>@@ -2779,6 +2779,27 @@ static void bond_miimon_link_change(struct bonding *bond,
> 	}
> }
> 
>+static bool bond_miimon_slave_down(struct bonding *bond,
>+				   struct slave *slave)
>+{
>+	if (slave->link_failure_count < UINT_MAX)
>+		slave->link_failure_count++;
>+
>+	bond_set_slave_link_state(slave, BOND_LINK_DOWN,
>+				  BOND_SLAVE_NOTIFY_NOW);
>+
>+	if (BOND_MODE(bond) == BOND_MODE_ACTIVEBACKUP ||
>+	    BOND_MODE(bond) == BOND_MODE_8023AD)
>+		bond_set_slave_inactive_flags(slave,
>+					      BOND_SLAVE_NOTIFY_NOW);
>+
>+	slave_info(bond->dev, slave->dev, "link status definitely down, disabling slave\n");
>+
>+	bond_miimon_link_change(bond, slave, BOND_LINK_DOWN);
>+
>+	return slave == rcu_access_pointer(bond->curr_active_slave);
>+}
>+
> static void bond_miimon_commit(struct bonding *bond)
> {
> 	struct slave *slave, *primary, *active;
>@@ -2807,10 +2828,21 @@ static void bond_miimon_commit(struct bonding *bond)
> 			err = bond_update_speed_duplex(slave);
> 			netdev_unlock_ops(slave->dev);
> 			if (err && bond_needs_speed_duplex(bond)) {
>-				slave->link = BOND_LINK_DOWN;
> 				if (net_ratelimit())
> 					slave_warn(bond->dev, slave->dev,
> 						   "failed to get link speed/duplex\n");
>+				/* Coming from FAIL, the slave is still in
>+				 * use and may be curr_active_slave, so take
>+				 * it through the full down path. Coming from
>+				 * DOWN/BACK, it was never brought up, so just
>+				 * revert it to DOWN.
>+				 */
>+				if (slave->link_prev_state == BOND_LINK_FAIL) {
>+					if (bond_miimon_slave_down(bond, slave))
>+						do_failover = true;
>+					continue;
>+				}
>+				slave->link = BOND_LINK_DOWN;
> 				continue;
> 			}
> 			bond_set_slave_link_state(slave, BOND_LINK_UP,
>@@ -2839,24 +2871,8 @@ static void bond_miimon_commit(struct bonding *bond)
> 			continue;
> 
> 		case BOND_LINK_DOWN:
>-			if (slave->link_failure_count < UINT_MAX)
>-				slave->link_failure_count++;
>-
>-			bond_set_slave_link_state(slave, BOND_LINK_DOWN,
>-						  BOND_SLAVE_NOTIFY_NOW);
>-
>-			if (BOND_MODE(bond) == BOND_MODE_ACTIVEBACKUP ||
>-			    BOND_MODE(bond) == BOND_MODE_8023AD)
>-				bond_set_slave_inactive_flags(slave,
>-							      BOND_SLAVE_NOTIFY_NOW);
>-
>-			slave_info(bond->dev, slave->dev, "link status definitely down, disabling slave\n");
>-
>-			bond_miimon_link_change(bond, slave, BOND_LINK_DOWN);
>-
>-			if (slave == rcu_access_pointer(bond->curr_active_slave))
>+			if (bond_miimon_slave_down(bond, slave))
> 				do_failover = true;
>-
> 			continue;
> 
> 		case BOND_LINK_FAIL:
>@@ -2919,6 +2935,7 @@ static void bond_mii_monitor(struct work_struct *work)
> 
> 		if (commit) {
> 			bond_for_each_slave(bond, slave, iter) {
>+				slave->link_prev_state = slave->link;
> 				bond_commit_link_state(slave,
> 						       BOND_SLAVE_NOTIFY_LATER);
> 			}
>diff --git a/include/net/bonding.h b/include/net/bonding.h
>index 598d56b1bc97..b2b102946fe1 100644
>--- a/include/net/bonding.h
>+++ b/include/net/bonding.h
>@@ -165,6 +165,7 @@ struct slave {
> 	unsigned long target_last_arp_rx[BOND_MAX_ARP_TARGETS];
> 	s8     link;		/* one of BOND_LINK_XXXX */
> 	s8     link_new_state;	/* one of BOND_LINK_XXXX */
>+	s8     link_prev_state; /* one of BOND_LINK_XXXX */
> 	u8     backup:1,   /* indicates backup slave. Value corresponds with
> 			      BOND_STATE_ACTIVE and BOND_STATE_BACKUP */
> 	       inactive:1, /* indicates inactive slave */
>-- 
>2.43.0
>

---
	-Jay Vosburgh, jv@jvosburgh.net