Most people working with IRF will be aware of the MAD (Multiple Active (Master) Detection) process.
In this article we will review the operation of MAD LACP on Comware5 and the changes in the implementation for Comware7 devices.
MAD is required to fix a split-stack situation, where a failure of the stacking network links would result in 2 or more masters on the network, which all claim to have the same MAC and IP address (which results in unpredictable topologies for LACP, xSTP, OSPF, etc.).
MAD will try to detect multiple masters and keep only 1 master online, the others will shutdown their interfaces (effectively removing them from the network).
In Comware5, there were several MAD detection processes:
- MAD LACP : Based on proprietary LACP extension. Can use existing LinkAggregation links (in-band), but peer must be Comware device (recent Provision software also has MAD LACP support !). This requirement only applies to the split-brain detection Bridge Aggregation link of course, so all other peer devices can be Cisco, Avaya, ESX host etc.
- MAD BFD : Based on BFD ip protocol. Requires a dedicated link between the devices.
- MAD ARP : Based on IPv4 ARP. (I have not used this in an implementation yet).
- MAD ND : Based on IPv6 ND. Same principle as ARP method.
Basic principle of MAD
When there is a split stack (the stacking network links have failed), there will be 2 masters. These 2 masters should be able to reach each other via either:
- Peer devices link aggregation (BAGG) to process MAD LACP
- Directly connected dedicated links to process MAD BFD (direct links should follow different physical path from the stack links for obvious reasons). An intermediate L2 switch can be used as well to save ports on the devices, just make sure the ports are forwarding (no xSTP or other possible protocols which could block the link).
When the masters can reach each other through LACP or BFD, they will be able to exchange their Member ID (unit ID), and the lowest Member ID will win, the other one will shutdown all the local interfaces (except the MAD-excluded interfaces).
This will result in a stable network, since only 1 master remains online.
MAD LACP on Comware5
MAD LACP on Comware5 will include the domain ID and Master Unit ID into each LACP packet. When there is a split stack, there will be 2 IRF sections, each with their own master.
Assume this start situation, with an IRF system of 4 switches and a neighbor device to support the MAD LACP (this could be an IRF system as well, it is just simplified in this diagram):
Next a double link failure occurs (link from unit1 to unit2 and link from unit1 to unit4).
This will immediately result in 2 IRF systems, each with their own master. Master election on the 2,3,4 side will be done based on configured priority (highest wins), uptime and unit MAC (this order). Assume here that unit2 had the best priority configured:
Now, each master will send the LACP packet with his own Unit ID, which the peer device will relay to the other member ports of the Link Aggregation group. These packets are send over each member port, so the right-side IRF would transmit 3x the LACP packet, while the left-side IRF would transmit 1x the LACP packet:
The neighbor switch does not know where the split-stack has occurred, so whenever it receives an LACP packet on a BAGG member port, it will replicate the information to all other member ports. This means the original single (1) LACP packet from Unit1 would arrive 3 times (through unit2, unit3 and unit4, which forward the LACP packet inside the IRF stack to the master unit2):
When the other master receives the LACP packet with the remote master Unit ID, it will compare it to his own Unit ID. The lowest Unit ID will win, so:
- If the remote Unit ID is higher : write a message in the log file about MAD master conflict, but do nothing else (this device will remain online)
- If the remote Unit ID is lower : write a message in the log file about MAD master conflict, then shutdown all local interfaces, except the configured MAD excluded interfaces (this device will be offline – removed from the network)
Resulting shutdown ports situation:
Although this is a very predictable mechanism, the used diagrams also show that the side with the lowest master unit ID will win, so in this example, only 1 switch will remain online, shutting down the other 3 devices. The same could happen with an IRF with 8 systems of course.
MAD LACP on Comware7 : Side with most online members wins !
In Comware7 switches, the LACP proprietary TLVs have been extended with an additional field to exchange the current online members in the IRF system.
This online member count information will be included in the selection process of which Master will win/loose the MAD process.
The new selection process would be:
- Side with most online members will win
- If equal members, use classic method : lowest master unit ID wins
So, we start from the same setup as the Comware5 example, but this time we have an IRF system with 4 Comware7 devices. The same IRF links fail, and the MAD LACP packets are exchanged. However, this time, the online member count is included:
So when this LACP packet if relayed by the neighbor switch to the remote IRF system:
The new selection process will result in the Unit1 which will shutdown its local ports:
And the resulting final topology has only unit1 down:
So thanks to the added step in Comware7, the side with most units online will remain online. This result in units 2,3,4 remaining online, while unit 1 would shutdown its interfaces.
Note1: When the reported online members is equal for both sides, the lowest master unit ID will win again.
Note2: In the diagrams, the LACP exchange is shown after the split stack failure, but this was actually already running before the split-stack. So under normal conditions, the Master will send out the extended LACP information over all the BAGG member ports, the neighbor switch will relay each packet back over the member ports, and the Master will receive its own packet information back. Since the received LACP information contains his own Master ID, nothing needs to be done.
Be careful when combining MAD methods
In Comware5, it did not matter if you selected LACP, BFD, etc method or any combination of these (you could have MAD LACP and BFD running at the same time), since the outcome would always be the same.
As of Comware7, you should realize that MAD BFD, ARP and ND are still following the classic (Comware5) rules, only the MAD LACP selection process was changed.
So do not mix MAD LACP with MAD BFD methods on Comware7 devices, since this would lead to unpredictable results, depending on which method detects the split brain first.