Sorting out System MAC addresses with VPC and VSS – Part 1

A number of multichassis aggregation technologies are deployed in the data center today, for example, Cisco’s Multichassis EtherChannel (MEC) on catalyst 6500 VSS, and Virtual Port Channel (vPC) on Nexus platforms. Inter-chassis aggregation greatly increases link utilization, while simplifying design by eliminating topology dependence on spanning tree protocol. STP becomes passive as most links are forwarding, and most failure scenarios no longer require STP re-convergence, thus minimizing disruptions. Furthermore, a more elegant data center design can be achieved, with lower operational complexity, and higher return on investment.  

System MAC address exists on individual devices, often used for device level negotiation, for example, bridge ID field in STP BPDU, or as part of LACP LAGID.

When multiple chassis operate in unison, software simulates the behavior of a common logical system, with the use of common virtual identifiers. Differentiating and sorting out the use of virtual system identifier and various MAC addresses is helpful for understanding, designing and deploying such systems.

It can be illustrated with a simple topology such as the one shown in the diagram, in which a pair of Nexus (in VPC domain 100) is connected to another pair (in VPC domain 101) on back to back VPCs.

BGP RIB-failure and effect on route advertisement

When examining routes advertised to BGP neighbor, notice some routes are tagged with “r”:

rtr1#sh ip bgp neighbor 10.11.19.21 advertised

BGP table version is 1735468, local router ID is 10.115.254.254

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter, a additional-path

Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path

r>i10.115.254.0/30 10.115.254.9            0    100      0 i

*> 10.115.254.0/23 10.115.254.9           21         32768 i

r>i10.115.254.4/30 10.115.254.9            0    100      0 i

r>i10.115.254.8/30 10.115.254.9            0    100      0 i

Note this is BGP “RIB-failure”, which indicates BGP fails to install the route in routing table. According to this link, the likely cause is the route is already installed by IGP which has a lower AD.

Beware of unpredictable OSPF to OSPF Mutual Redistribution

Just as it may be necessary to run multiple OSPF processes, It may also be necessary to redistribute routes between them. What happens when two OSPF processes redistribute routes to each other? The results can be quite surprising.

As shown in illustration, R1 runs two OSPF processes. R1 learns a network 10.0.0.0/24 from both routing processes. From OSPF 1 (left) R1 learns it as an inter-area route. From OSPF 2 (right) R1 learns it as an External route. Which direction would R1 prefer?

A simple test demonstrates show results can be unpredictable. By shutting down the interface towards OSPF 1 and turn it back on, R1 prefers E1 route to reach 10.0.0.0/24 via OSPF 2. Subsequently, by resetting the interface towards OSPF 2, R1 prefers inter-area route to reach 10.0.0.0/24 via OSPF 1.

OSPF’s preference of intra-area over inter-area over external applies to routes learned via the same process only; it does not apply to routes learned from multiple OSPF processes.

So what determines preference between routing processes? It’s Admin Distance. Since the default AD for OSPF processes are the same, thus the unpredictable results. Therefore the results can be “swung” by resetting interfaces.

When Administrative Distances are equal, the process that first installs the route in the routing table wins, regardless of metric and type.

How to make it deterministic? The key is obviously around AD. But to apply AD properly as a solution, the desired behavior must be clearly defined. First, identify potentially overlapping networks, that is, networks that can be advertised by both processes. Next, how should the network behave for those networks?

If all overlapping networks should prefer one process, for example, OSPF 1 inter-area should be preferred over OSPF 2 E1, then AD of OSPF 2 can be increased:

router ospf 2

distance ospf external 120

The diagram illustrates the result of the fix, routes from OSPF 1 will now be preferred due to deterministic AD.

If the desired behavior is specific to networks, then AD must be selectively adjusted using filter list. And AD may need to be adjusted on both OSPF processes to arrive at the specific preference for specific networks.

The end results should always be deterministic and predictable, verified using tests in normal and failure scenarios.

As a side note, Before Cisco bug ID CSCdw10987 (integrated in Cisco IOS Software Releases 12.2(07.04)S, 12.2(07.04)T, and later), the last process to make an shortest path first algorithm (SPF) would have won, and the two processes overwrite other routes in the routing table. Now, if a route is installed via one process, it is not overwritten by another OSPF process with the same administrative domain (AD), unless the route is first deleted from the routing table by the process that initially installed the route in the routing table.

Understand Nexus 7000 Module Forwarding Capacity

Nexus 7000’s line card forwarding (FIB) capacity may be a limiting factor. This is particularly true in a virtualized data center, where multiple VDCs and VRFs are used. This note explains the importance of understanding and managing module capacity.

Nexus 7000 has forwarding engine on line cards. The larger the size of the routing table, the bigger the FIB required for hardware forwarding. When designing with multiple VDCs and VRFs, multiple routing processes and routing tables consume more resources on the module.

Below shows a module using 64k of 65k allocated capacity:

#show hardware capacity forwarding

…

Module 1 usage:

Route Type              Used      %Used      Free      %Free      Total    

                     (Log/Phys)            (Log/Phys)           (Log/Phys)   

-------------------------------------------------------------------------

IPv4 Unicast:       64161/64161     97     1375/1375       2    65536/65536 

L2VPN Peer:             0/0          0        0/0          0        0/0     

MPLS:                   0/0          0        0/0          0        0/0     

IPv4 Multicast:         4/8          0    14332/28664     99    14336/28672 

L2VPN IPv4 Mcast:       0/0          0        0/0          0        0/0     

IPv6 Unicast:         195/390        1    14141/28282     98    14336/28672 

L2VPN IPv6 Mcast:       0/0          0        0/0          0        0/0     

IPv6 Multicast:         5/20         0     2043/8172      99     2048/8192  

Building a private cloud using MPLS and selective tag switching

To support virtualization requirements, it may be necessary to create a private cloud within the enterprise. MPLS is a proven technology to extend virtualization across WAN networks. Most existing network equipment can be enabled for MPLS, thus allowing an enterprise to gain significant new capabilities without the expense of equipment and facility cost.

To transition an existing production network into one that is MPLS enabled, a detailed design is required to select and place key infrastructure elements such as P, PE, and CE routers. Given that existing production traffic may already be routed by these devices, it is often desirable to leave that untouched.

Selective tag switching allows certain VPN traffic to be label switched by the MPLS network, while the other traffic continues to be routed.

 IP address planning is a critical prerequisite to enable selective tag switching. It is highly desirable to place MPLS infrastructure and the new private cloud extensions in a new and distinctive IP address block.

On the P router, private cloud traffic can be identified by PE router loopback addresses, here using 10.88.0.0/24 as an example

access-list 88 permit 10.88.0.0 0.0.0.255

By default, label switching is applied to all traffic. Therefore it is important to reverse that behavior, and turn on label switching for the target traffic identified by access list.

no mpls ldp advertise-labels

mpls ldp advertise-labels for 88

As a result, only MPLS VPN traffic will be tag switched, while existing traffic shows “untagged”

P-router#sh mpls forward

Local  Outgoing    Prefix            Bytes tag  Outgoing   Next Hop

tag    tag or VC   or Tunnel Id      switched   interface

16     Untagged    10.99.0.120/30   0          Fa0/0      10.99.32.206

17     Untagged    10.99.32.232/30  0          Fa0/0      10.99.32.206

18     Untagged    10.99.248.13/32  0          Fa0/0      10.99.32.206

19     Untagged    10.99.32.220/30  0          Fa1/0      10.99.32.214

20     Untagged    10.99.248.7/32   0          Fa1/0      10.99.32.214

21     Pop tag     10.88.0.160/30   0          Fa0/0      10.99.32.206

22     Pop tag     10.88.0.192/30   0          Fa1/0      10.99.32.214

23     Pop tag     10.88.0.1/32     544932     Et3/0      10.88.0.129

24     21          10.88.0.3/32     399694     Fa0/0      10.99.32.206

25     21          10.88.0.5/32     416019     Fa1/0      10.99.32.214

26     Pop tag     10.88.0.136/30   0          Et3/0      10.88.0.129

27     Pop tag     10.88.0.144/30   0          Et3/0      10.88.0.129

28     22          10.88.0.168/30   0          Fa0/0      10.99.32.206

29     Pop tag     10.88.0.172/30   0          Fa0/0      10.99.32.206

...

 “traceroute” also provides an excellent tool to demonstrate if a packet is routed or tag switched throughout a network.

A potential problem with Juniper’s implementation of OSPF Router ID

First of all, this non-compliant behavior is observed only on some Juniper devices, not all.

The potential effect of the observed behavior is such that certain OSPF routes fail to propagate as expected.

Here is the scenario: SRX originally uses a loopback 10.0.0.11 for its Router ID.  When that loopback was deleted, it changed its Router ID to a different address 10.0.17.4. This is expected behavior so far.

srx-node0> show interfaces lo0.1

error: interface lo0.1 not found

srx-node0> show ospf overview instance VR   

Instance: VR

  Router ID: 10.0.17.4

…

However, a closer look at SRX OSPF database reveals that it still has LSAs with 10.0.0.11 (the old Router ID) as its Advertising Router ID.

srx-node0> show ospf database summary instance VR

    OSPF database, Area 0.0.0.0

 Type       ID          Adv Rtr           Seq      Age  Opt  Cksum  Len

Summary  10.0.17.0    10.0.0.11      0x80000008  1423  0x22 0xc857  28

Summary *10.0.17.0    10.0.17.4      0x80000002   792  0x22 0x8794  28

    OSPF database, Area 0.0.0.69

 Type       ID          Adv Rtr           Seq      Age  Opt  Cksum  Len

Summary  10.0.16.3    10.0.0.11      0x8000001c  2280  0x22 0xfbfc  28

Summary  10.0.16.7    10.0.0.11      0x8000001c  2137  0x22 0xd321  28

Summary  10.0.16.8    10.0.0.11      0x8000001c  1994  0x22 0xbf35  28

Summary  10.0.17.16   10.0.0.11      0x8000001d  3423  0x22 0xfdfc  28

Summary  10.0.17.32   10.0.0.11      0x8000001d  1851  0x22 0xaf2e  28

According to RFC2328:

If a router's OSPF Router ID is changed, the router's OSPF software should be restarted before the new Router ID takes effect.  In this case the router should flush its self-originated LSAs from the routing domain before restarting

Note there are two desired behaviors when Router ID changes: 1) OSPF restarts; 2) originated LSA flushes. When that did not happen with JUNOS, the resulting behavior is that the old Router ID is still the “Advertising Router ID” in the LSA, an address that is no longer valid.

Why is that a problem? Because these LSAs will be flooded to neighbors (assuming the router here is an ABR). The neighbor would have noticed the change of Router ID, and thus it will check the validity of Advertising Router ID.

Again, definition of Advertising router according to RFC2328: 

This field indicates the Router ID of the router advertising the summary-LSA or AS-external-LSA that led to this path.

Since the neighbor sees the Advertising Router ID (the old Router ID) no longer matches the new Router ID, it will discard the LSA.

When troubleshooting OSPF routing involving Juniper devices, check OSPF databases for invalid entries.

To prevent such pitfalls, always set Router ID in OSPF. And more importantly, set Router ID using loopbacks, and make sure they are not accidentally deleted. 

ERSPAN with Nexus 1000v in a Virtualized Data Center

Encapsulated remote SPAN, or ERSPAN can be used to monitor traffic remotely. In a Nexus 1000v environment, it is not feasible to attach probe directly to the virtual switch. Therefore it is particularly valuable to monitor host traffic using ERSPAN, by routing monitored traffic through IP network to designated network analyzer.

A functioning ERSPAN system consists of these components working together:

·         Nexus1000v with specific port profile and SPAN session

·         Host configured to support monitoring interface

·         Destination switch to forward monitoring traffic to probe

A sample reference model is provided here, using Nexus 7000 attached probe as a common example.

ERSPAN - Cisco Networks

Nexus 1000v

First, choose a routed VLAN (2000) to carrying ERSPAN traffic. Chose a subnet size that will accommodate growth of hosts (each host uses an IP address). To illustrate, 10.1.0.0/24 is used for VLAN 2000.

Create a port profile for this VLAN on Nexus1000v, note this VLAN must be a system VLAN.

port-profile type vethernet ERSPAN_2000

  capability l3control

  vmware port-group

  vmware max-ports 64

  switchport mode access

  switchport access vlan 2000

  no shutdown

  system vlan 2000

  state enabled

Next, create a test ERSPAN session, for example, monitor VM on Veth88, send monitored traffic to destination 10.2.0.88. See Nexus 7000 section for destination configuration.

monitor session 1 type erspan-source

 source interface Vethernet88 both

  destination ip 10.2.0.88

  erspan-id 51

  ip ttl 64

  ip prec 0

  ip dscp 0

  mtu 1500

  header-type 2

  no shut

Add a VMKNIC for each host

Must be done from vCenter, for each host. An IP address in VLAN 2000 10.1.0.0/24 is required for each host.

Reference Vmware configuration guide for details.

Nexus 7000

The destination probe is connected to Nexus 7000. We’d want monitored traffic originating from Nexus 1000v, to be forwarded to the probe.

The destination 10.2.0.1 specified by ERSPAN session (on N1kv) has an ARP entry in vlan 3000. There is also a corresponding static MAC address entry pointing to the port which the probe is connected. As a result, the ERSPAN traffic destined for 10.2.0.1 will be forwarded to the probe.

interface Vlan2000

  …

  ip address 10.1.0.2/24

  hsrp 2000

ip 10.1.0.1

interface Vlan3001

  …

  ip address 10.2.0.1/24

  ip arp 10.2.0.88 00AA.BBCC.DD66

interface Ethernet2/2

  switchport

  switchport access vlan 3000

  no shutdown

mac address-table static 00AA.BBCC.DD66 vlan 3000 interface Ethernet2/2

Data Center ISP Load Sharing Part 4 – Tuning

Does full Internet routing table work the best for load sharing with multi-homed ISP connections? We have shown it is often not the case.

Part 1 of the posting shows  the challenges of dual ISP design, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs.

Part 2 of the posting shows the advantage of simple default route based Internet load sharing design.

Part 3 of the posting introduces a design that combines the simplicity of default based load sharing to dual ISP, and flexibility of selectively filtering subsets of Internet routes for optimal path selection.

In this final part we look at why and how the results should be fine-tuned.

At the initial design stage, you could estimate the number of routes filtered in from your respective ISPs, using the BGP regular expression you designed. Route count estimate provides the basis for your filtering design.  For example, you can count that approximately 50000 routes will be allowed in by a filter specifying only adjacent networks to a tier one ISP. You also count that approximately 40000 routes will be allowed in by another filter specifying adjacent networks as well as those one hop away from a tier two ISP.

After implementation, you will notice the actual number of specific networks allowed in will be less than the combined total of 50000 plus 40000. The total number (for example 80000) is less than the total due to duplicates. In other words, you learn the same 10000 routes from both ISPs because those networks are adjacent to both ISPs. This is common and to be expected.

You might have expected the duplicate routes to be split more or less evenly across the two ISPs, which is often not the case. Therefore, the effect of duplicate routes on load sharing requires some careful observation. ISPs may operate in different tiers of the internet hierarchy, thus affecting the routes they advertise to have shorter or longer AS path length.  AS path length is a primary criterion in BGP path selection, therefore you will likely see almost all of the duplicate routes favoring one ISP. This may affect load sharing, thus require further adjusting the filters.

A second example is the influence of ISP metrics. Some ISP may advertise routes with a metric, while others advertise all routes with zero metric. Zero metric will be preferred if other more priority criteria is equal.

The diagram shows the original design may have expected a load sharing design of 5/4. However, the result shows load sharing between ISP1 and ISP2 turns out to be 5/3, due to all duplicate routes favoring ISP1. Depending on your specific requirements, fine-tuning of the filter may be necessary.

Design for redundant Internet architecture is unique to every organization’s requirements, its national or global data center architecture, the ISPs selected, and the nature of its Internet traffic. The scenarios described hopefully have provide simple templates as references to adjust for your particular data center.