This is a feature that seems to exist already for a while in Comware 7 (it works even in Simware, which is pretty old by now), but I just discovered it recently: IP unnumbered interfaces on Ethernet with OSPF point-to-point links.The key advantage is that in the IPv4 routed backbone, no more /30 subnets need to be configured (so no more separate addressing plan either :)) on all those ethernet routed connections between the routers. You just borrow the Loopback IP and go!
LLDP – ARP Rule
The challenge with ethernet is of course that it is a multi-access network, so how can we discover the IP/MAC relation of the peer (ARP)?
This is solved with LLDP on the link. As part of an outgoing LLDP frame, the switch will include the Management Address TLV. This can be set to the Loopback address of the device. The peer device will now get an LLDP frame with the IP address announcement of its neighbor, and it will use the source MAC address of the LLDP frame to make a static ARP record for this MAC/IP of the peer.
This ARP entry allows already for a direct peer ping from loopback to loopback. One problem solved.
Next you need to configure the routed link as an OSPF point-to-point link, so there is no OSPF DR election. When the OSPF mcast is sent out, the source IP will be included, so with the ARP entry, the peer now knows how to reach the peer and establish the OSPF adjacency. Easy and done!
Define a Loopback on sw1
<sw1>dis cur int loop 0 # interface LoopBack0 ip address 188.8.131.52 255.255.255.255 # return
Define a Loopback on sw2
<sw2>dis cur int loop 0 # interface LoopBack0 ip address 184.108.40.206 255.255.255.255 # return
Configure OSPF on both devices, this example just makes a single area 0.
<sw1>dis cur conf ospf # ospf 1 area 0.0.0.0 network 0.0.0.0 255.255.255.255 # return
<sw2>dis cur conf ospf # ospf 1 area 0.0.0.0 network 0.0.0.0 255.255.255.255 # return
Configure the link between the switches:
- as a routed link
- set the IP to the Loopback IP
- configure the Loopback 0 IP as the LLDP management address (outbound)
- enable the programming of static ARP rule for the LLDP management IP/MAC (inbound)
- set as OSPF P2P
<sw1>dis cur int g1/0/1 # interface GigabitEthernet1/0/1 port link-mode route ip address unnumbered interface LoopBack0 ospf network-type p2p lldp management-address arp-learning lldp tlv-enable basic-tlv management-address-tlv interface LoopBack0 # return <sw1>
Repeat this on the peer device and all other routed links in the backbone
<sw2>dis cur int g1/0/1 # interface GigabitEthernet1/0/1 port link-mode route ip address unnumbered interface LoopBack0 ospf network-type p2p lldp management-address arp-learning lldp tlv-enable basic-tlv management-address-tlv interface LoopBack0 # return
Once the links come up, LLDP will do its work, so a new LLDP NEIGHBOR is created. This results in a static (rule based) ARP entry that will be generated for the neighbor. Next OSPF P2P link will establish the adjacency and transition to FULL.
Check out the timing (I know this is just a single link, but still nice to have OSPF FULL fairly quickly after the link is UP)
%Oct 6 13:02:12:939 2016 sw1 IFNET/3/PHY_UPDOWN: Physical state on the interface GigabitEthernet1/0/1 changed to up. %Oct 6 13:02:12:939 2016 sw1 IFNET/5/LINK_UPDOWN: Line protocol state on the interface GigabitEthernet1/0/1 changed to up. %Oct 6 13:02:13:233 2016 sw1 LLDP/6/LLDP_CREATE_NEIGHBOR: Nearest bridge agent neighbor created on port GigabitEthernet1/0/1 (IfIndex 2), neighbor's chassis ID is 24a5-5c86-0200, port ID is GigabitEthernet1/0/1. %Oct 6 13:02:13:242 2016 sw1 OSPF/5/OSPF_NBR_CHG: OSPF 1 Neighbor 220.127.116.11(GigabitEthernet1/0/1) changed from LOADING to FULL.
Some review output screens, this shows the LLDP neighbor details (as you can see, this was done with Simware). Note the Management IPv4 that was announced by the peer:
<sw1>dis ll ne list Chassis ID : * -- -- Nearest nontpmr bridge neighbor # -- -- Nearest customer bridge neighbor Default -- -- Nearest bridge neighbor System Name Local Interface Chassis ID Port ID sw2 GE1/0/1 24a5-5c86-0200 GigabitEthernet1/0/1 <sw1>dis ll ne int g1/0/1 ver <sw1>dis ll ne int g1/0/1 verbose LLDP neighbor-information of port 2[GigabitEthernet1/0/1]: LLDP agent nearest-bridge: LLDP neighbor index : 1 Update time : 0 days, 2 hours, 40 minutes, 30 seconds Chassis type : MAC address Chassis ID : 24a5-5c86-0200 Port ID type : Interface name Port ID : GigabitEthernet1/0/1 Time to live : 121 Port description : GigabitEthernet1/0/1 Interface System name : sw2 System description : H3C Comware Platform Software, Software Version 7.1.059, Alpha 7159 H3C S5820V2-54QS-GE Copyright (c) 2004-2014 Hangzhou H3C Tech. Co., Ltd. All rights reserved. System capabilities supported : Bridge, Router, Customer Bridge, Service Bridge System capabilities enabled : Bridge, Router, Customer Bridge Management address type : IPv4 Management address : 18.104.22.168 Management address interface type : IfIndex Management address interface ID : 1413 Management address OID : 0 Link aggregation supported : Yes Link aggregation enabled : No Aggregation port ID : 0 Auto-negotiation supported : No Auto-negotiation enabled : No OperMau : Speed(0)/Duplex(Unknown) Maximum frame size : 1500 <sw1>
Next the device will you the source MAC of the LLDP frame. Since the L3 interface MAC may be (will be) different from the base system MAC (as announced in the LLDP frame Chassis ID), it would not make sense to use the LLDP Chassis ID MAC. Also keep in mind that the routed port interface MAC may be changed manually to some other MAC, so that scenario also needs to be handled.
This is why the ARP rule will use the source MAC of the LLDP frame. This can be seen with a ‘display interface’ command, but to really verify it, just activate some LLDP and ethernet packet debug on the routed port:
<sw1>debug lldp packet <sw1>debug ethernet packet int g1/0/1 *Oct 6 13:13:16:114 2016 sw1 ETH/7/Eth_rcv: Received an ethernet packet, interface: GigabitEthernet1/0/1, format: 0, src_addr: 24a5-5c86-0206, dst_addr: 0180-c200-000e, payload: 88 cc *Oct 6 13:13:16:114 2016 sw1 LLDP/7/Packet received: Interface GigabitEthernet1/0/1 nearest-bridge; Length is 299. Chassis type : MAC address Chassis ID : 24a5-5c86-0200 Port ID type : Interface name Port ID : GigabitEthernet1/0/1 Time to live : 121
Next review the ARP entry that was created by LLDP. Note the type ‘R’ = Rule. This refers to LLDP that has created this entry for you.
<sw1>dis arp Type: S-Static D-Dynamic O-Openflow R-Rule M-Multiport I-Invalid IP address MAC address VLAN Interface Aging Type 22.214.171.124 24a5-5c86-0206 N/A GE1/0/1 N/A R <sw1>
Then you can verify the OSPF peers:
[sw1]dis ospf peer OSPF Process 1 with Router ID 126.96.36.199 Neighbor Brief Information Area: 0.0.0.0 Router ID Address Pri Dead-Time State Interface 188.8.131.52 184.108.40.206 1 33 Full/ - GE1/0/1 [sw1]
And check out the IP routing table:
[sw1]dis ip routing Destinations : 10 Routes : 10 Destination/Mask Proto Pre Cost NextHop Interface 0.0.0.0/32 Direct 0 0 127.0.0.1 InLoop0 220.127.116.11/32 Direct 0 0 127.0.0.1 InLoop0 18.104.22.168/32 O_INTRA 10 1 22.214.171.124 GE1/0/1 127.0.0.0/8 Direct 0 0 127.0.0.1 InLoop0 127.0.0.0/32 Direct 0 0 127.0.0.1 InLoop0 127.0.0.1/32 Direct 0 0 127.0.0.1 InLoop0 127.255.255.255/32 Direct 0 0 127.0.0.1 InLoop0 126.96.36.199/4 Direct 0 0 0.0.0.0 NULL0 188.8.131.52/24 Direct 0 0 0.0.0.0 NULL0 255.255.255.255/32 Direct 0 0 127.0.0.1 InLoop0 [sw1]
Note on BGP
A similar feature has been added to support BGP next hop handling with this IP Unnumbered system, so the ARP record will be added as a local, direct route. This ensures that the basic BGP next-hop route recursion lookup rule (is next-hop available through regular routing table?) will now be success and BGP routing will also work on the unnumbered link.
You still need to manually define the BGP peer in your BGP configuration (at least on 1 side), but that is just a minor step. This is done with the ‘arp direct-route advertise’ command. (this feature is not available in Simware)
interface GigabitEthernet1/0/1 port link-mode route ip address unnumbered interface LoopBack0 lldp management-address arp-learning lldp tlv-enable basic-tlv management-address-tlv interface LoopBack0 arp route-direct advertise
Thank you for that review
I need a little more informations (not really in the scope of your post but very border and interesting in my opinion)
For the context I plan to deploy 10x leafs and 4 spines (all are 5940 series) for delivering cloud services between 3 DCs
I m evaluating OSPF underlay with ip unnumbered (one loopback per device) in VSR envirronement
A point of design keep my attention
On each router mono unnumbered link results in one arp entry per link
so after routing processing neighbor lookup (arp) is coherent with equal multipath routing process (ECMP)
multi unnumbered parrallel links (multi adjacencies) results in multiple ARP entries contains same IP and MAC address but different interfaces
So it seem to be multiport ARP
I anderstand that ECMP and ARP selection are independant and unaware mecanisms
But in reality it depend on the HPE implementation
but it is not documented …
Do you know if multi unnumbered parrallel links are supported with ECMP ?
If true, do you know how ECMP impose the outbound interface in the multiport arp entries selection ?
thx for all your posts