Comware has 2 types of link-aggregation:

• static link-aggregation: no control protocol is used, based on the configuration, the link-aggregation member interfaces will be actively participating (Selected) in the link-aggregation at the moment the interfaces come up.
• dynamic link-aggregation: LACP control protocol is used. Interfaces are added to the link-aggregation based on the config. The operational state (Selected/Unselected) depends on the result of the LACP negotiation with the peer

In some situations, a mix of both is required. This post shows how this can be configured.

The problem

When an LACP link-aggregation (bond/team/channel/trunk/etc) is created between a Server and network switch, both ends of the link-aggregation will exchange LACP packets to validate the links are properly connected:

• Validate all links connect to the same peer
• Validate all links connect to the same link-aggregation interface on the peer side
• Validate all links are allowed to be used for forwarding (in case there would be a configuration mismatch between link-aggregation member and parent port)

This works fine under normal conditions.

However, when the server performs the initial boot or a reboot, the server teaming or bond configuration is not yet active. The same applies when the initial server installation is being performed.

LACP link-aggregation safe mode

The Comware switch would not see any LACP activity at this point and will revert to safe mode for the link-aggregation. This means only 1 member port will be set to the Selected state and the other member ports will be set to the Unselected state.

Ports in Unselected state block all incoming and outgoing communication.

When the server is initially booted or being installed, while the network switch is already configured with an LACP link-aggregation, it means that only the first NIC of the server has network access. The second NIC would be connected to the Unselected state port on the switch, so no PXE boot or initial DHCP/IP/iSCSI configuration is possible on this port, since all communication is dropped by the Unselected port.

The solution: LACP Edge-port

In the current Comware software, it is possible to configure a Bridge-Aggregation as lacp  edge-port. This means the member ports of the link-aggregation will initially be placed in the Individual state for as long as no LACP traffic is received, or when the LACP peer times out on the ports (3x LACP hello of 30sec, which means after 90 seconds)

Ports in the individual state are forwarding inbound and outbound traffic.

At the moment any LACP traffic is received on a member port, all the link-aggregation member ports transition to normal operation and classic LACP rules will apply to decide on the Selected/Unselected state.

Configuration example

In this example, 2 switches are used, 1 will assume the role of the server.

Default behavior

First we will look at the default behavior of a switch with an LACP link-aggregation.

On the SERVER simulating switch, a static (no LACP) bagg is configured (device is an IRF system, but that has no impact on the example)

[SERVER] interface Bridge-Aggregation 1
[SERVER-Bridge-Aggregation1] quit

[SERVER] interface range ten 1/0/1 ten 2/0/1
[SERVER-if-range] port link-aggregation group 1
[SERVER-if-range] quit

[SERVER] display link-aggregation verbose
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Static
Loadsharing Type: Shar
  Port             Status  Priority Oper-Key
--------------------------------------------------------------------------------
  XGE1/0/1         S       32768    1
  XGE2/0/1         S       32768    1
[SERVER]

On the SWITCH side, configure a link-aggregation with LACP. The ports to the SERVER NICs are configured with LACP.

[SWITCH] interface Bridge-Aggregation 1
[SWITCH-Bridge-Aggregation1] link-aggregation mode dynamic
[SWITCH-Bridge-Aggregation1] quit

[SWITCH] interface range ten 1/0/51 ten 1/0/52
[SWITCH-if-range] port link-aggregation group 1
[SWITCH-if-range] quit
[SWITCH]

The status of the link-aggregation shows the LACP safe mode of the switch with 1 Selected port and the other ports Unselected. No Remote SystemID is shown, since there is no LACP traffic.

[SWITCH] display link-aggregation verbose Bridge-Aggregation 1
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x8000, 4431-926b-6874
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        S       32768    1         {ACDEFG}
  XGE1/0/52        U       32768    1         {ACG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  XGE1/0/51        0       32768    0         0x8000, 0000-0000-0000 {DEF}
  XGE1/0/52        0       32768    0         0x8000, 0000-0000-0000 {DEF}
[SWITCH]

This means that any traffic sent by the SERVER side over the link connecting to port Ten /0/52 (Unselected) on the switch side would be dropped.

LACP Edge-port configuration – Server side without LACP

Now the SWITCH side link-aggregation is configured with the LACP edge-port:

[SWITCH]interface bridge 1
[SWITCH-Bridge-Aggregation1]lacp edge-port
[SWITCH-Bridge-Aggregation1]quit

And as a result, the member ports are now placed in the Individual state (since no LACP traffic was received at all)

[SWITCH] display link-aggregation verbose Bridge-Aggregation 1
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x8000, 4431-926b-6874
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        I       32768    1         {AG}
  XGE1/0/52        I       32768    1         {AG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  XGE1/0/51        0       32768    0         0x8000, 0000-0000-0000 {DEF}
  XGE1/0/52        0       32768    0         0x8000, 0000-0000-0000 {DEF}
[SWITCH]

Since the physical member ports are now in individual state, the BAGG is actually down, the member ports are operating in standalone mode:

[SWITCH]dis int brief | i BAGG|51|52
BAGG1                DOWN auto    A      A    1
XGE1/0/51            UP   10G(a)  F(a)   A    1
XGE1/0/52            UP   10G(a)  F(a)   A    1
[SWITCH]

LACP Edge-port configuration – Server side with LACP

Now we enable LACP on the SERVER side switch to see what happens on the SWITCH side.

On the SERVER side:

[SERVER] int bridge 1
[SERVER-Bridge-Aggregation1] link-aggregation mode dynamic
[SERVER-Bridge-Aggregation1]

Side note: I do not know when this change was implemented, but it is now possible to change the static to dynamic mode on the fly (no need to remove the physical interfaces anymore), which is a welcome enhancement as well !

On the SWITCH side, the log output shows the physical ports are now member of the BAGG again:

%Mar 26 16:35:48:419 2011 SWITCH IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/52 is down.
%Mar 26 16:35:48:509 2011 SWITCH IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/51 is down.
%Mar 26 16:35:48:533 2011 SWITCH LAGG/6/LAGG_ACTIVE: Member port XGE1/0/52 of aggregation group BAGG1 changed to the active state.

%Mar 26 16:35:48:540 2011 SWITCH IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/52 is up.
%Mar 26 16:35:48:541 2011 SWITCH IFNET/3/PHY_UPDOWN: Bridge-Aggregation1 link status is up.
%Mar 26 16:35:48:542 2011 SWITCH IFNET/5/LINK_UPDOWN: Line protocol on the interface Bridge-Aggregation1 is up.
%Mar 26 16:35:49:756 2011 SWITCH LAGG/6/LAGG_ACTIVE: Member port XGE1/0/51 of aggregation group BAGG1 changed to the active state.

%Mar 26 16:35:49:758 2011 SWITCH IFNET/5/LINK_UPDOWN: Line protocol on the interface Ten-GigabitEthernet1/0/51 is up.

And the status of the link-aggregation has transitioned to a classic LACP link-aggregation

[SWITCH]display link-aggregation verbose Bridge-Aggregation 1
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x8000, 4431-926b-6874
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        S       32768    1         {ACDEF}
  XGE1/0/52        S       32768    1         {ACDEF}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  XGE1/0/51        1       32768    1         0x8000, 4431-9231-39a4 {ACDEF}
  XGE1/0/52        130     32768    1         0x8000, 4431-9231-39a4 {ACDEF}
[SWITCH]

LACP Edge-port configuration – Server reboot

To simulate a server reboot, we updated the SERVER side device:

• shutdown the links, disable LACP and enable the links again

[SERVER]int bridge 1
[SERVER-Bridge-Aggregation1]shutdown
[SERVER-Bridge-Aggregation1]undo link-aggregation mode
[SERVER-Bridge-Aggregation1]undo shutdown

Directly after the links are UP, the status on the SWITCH side shows both ports in the Individual state, so forwarding:

[SWITCH]display link-aggregation verbose Bridge-Aggregation 1
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, I -- Individual
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x8000, 4431-926b-6874
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        I       32768    1         {AG}
  XGE1/0/52        I       32768    1         {AG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  XGE1/0/51        0       32768    0         0x8000, 0000-0000-0000 {DEF}
  XGE1/0/52        0       32768    0         0x8000, 0000-0000-0000 {DEF}
[SWITCH]

This concludes the configuration example. The test was configured with Release 2311P01.