Data Center ISP Load Sharing Part 3 – Route Selection

Part 2 of the posting shows the advantage of simple default route based Internet load sharing design. This part further optimize the design.

Using the entire Internet routing table for outbound load sharing proves to be resource intensive, and ineffective for load balancing. Default route only provides simplicity and better load balancing. To further optimize, a subset of Internet routes, when selected according to the unique environment, can complement the default route design very well.

Route Selection

Route selection refers to filtering and allow a subset of the Internet routing table to be introduced into the data center. The desired effect is to take the shorter path to content that is directly attached to specific ISPs, while the rest of the traffic load share equally to both ISPs.

The effectiveness of the design is largely based on route selection techniques applicable to the specific data center environment. In the example shown below, BGP regular expression is used to select a subset of Internet destinations adjacent to each ISP.

ISP1 is a tier one, therefore has more directly attached networks. BGP expression is used to select those directly attached networks, with the objective that traffic destined for those networks will exit on this ISP for optimal path.

ISP2 is a tier two, with less number of directly attached networks. BGP expression is used to select those directly attached networks as well as those one additional hop away, with the objective that roughly equal number of specific target networks will prefer ISP2 as the exit point, thus achieving load sharing with both ISPs.

Implementation

On the respective internet router connected to each ISP, AS path filtering is applied on a route map, which is then applied to BGP inbound route filtering. As a result, the default route, as well as a subset of Internet routes is received from each ISP, in order to optimize outbound traffic to take the more direct path to destinations.

ip as-path access-list 1 permit ^3549_[0-9]*$

route-map ISP1in permit 10 

 match ip address prefix default 

route-map ISP1in permit 20 

 match as-path 1 

router bgp

 neighbor … route-map ISP1in in

Verification and Tuning

At the planning stage, counting number of routes using BGP regular expression filter may serve to arrive at the initial route selection design. By filtering in similar amount of specific routes from each ISP, the desired load sharing can usually be achieved. 

However, equivalent number of routes does not always result in equivalent amount of traffic. Over time, actual load on the respective ISP connections will provide more accurate information about traffic in the particular data center. ISP specific characteristics may also factor in. Part 4 will show why fine-tuning may be necessary.

Data Center ISP Load Sharing Part 2 – Default Method

Part 1 of the posting shows  the challenges of dual ISP design, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs. And when routes received from ISPs have different characteristics such as AS Path and metric, the result is often undesirable. To achieve better outcome by design, in part 2 we will start with a simple alternative.

Replacing the entire Internet routing table with just the default route is an extremely simple method that offers a number of advantages.

Load balancing

Instead of getting the entire Internet routing table, only default route is received and installed in the routing table. As a result, IGP can load balance to two equal cost default routes. For outbound traffic, the simple design achieve near 50/50 load balancing, as well as resiliency.

Simplicity, Stability and Resource Efficiency

The design is extremely simple to implement and support. Resource usage on devices can be greatly reduced, from holding tens and thousands of Internet routes, to just default route. Route flapping and any disruptive convergence due to instability in any parts of the Internet is virtually eliminated.

The simplicity advantage is well suited for a large number of enterprise data centers.

Disadvantage

The design essentially “splits” the Internet in half, by two equal cost default routes to dual ISPs. Therefore, the exit point may not be optimal, especially for networks directly attached to an ISP, which may require the longer path to get to.

For vast majority of applications, the selection of exit ISP is not noticeable. However, lower latency access of large amount of media content may be highly desirable when a direct path is available. An optimized solution is presented in part 3.

Data Center ISP Load Sharing Part 1 - The challenge

Redundant ISP connections are the standard in most data centers. There is surprisingly little update-to-date information about data center Internet architecture out there. In fact, traffic flow on the internet connections while increasingly business critical, is probably among the least well understood area in enterprise network architecture.  Consequently, the potential for cost saving and performance improvement is substantial.

This is the first part of a series that explore this interesting and important topic, which starts by examining a classic solution and its associated challenge.

For a large data center, it is often believed that obtaining the entire Internet routing table from each ISP provides the most comprehensive and complete routing information to direct traffic towards the Internet. However, here are a few potential issues to consider:

Heavy resource utilization

The design may propagate the entire internet routing table in the internal BGP environment within the data center, so that gateways can determine which ISP to exit to for a given Internet destination. Most Internet routers are designed to handle a large number of routes. However gateway devices particularly older generation switches may be running near their CPU and memory maximum. You might want to check CPU utilization on all your internet facing devices - high CPU utilization is often indicative that some routes are software switched. Performance will degrade as a result.

This diagram from CIDR shows continuous growth of BGP prefixes over the years. Expensive upgrades are often required just to keep up with the growth, in order to maintain the performance of hardware forwarding.

Imbalance of ISP bandwidth

The second issue varies with the particular ISP you chose. When you select your ISPs, did you consider how the selection may affect your design? It is very common to have a tier one ISP and tier two ISP in one data center. With hierarchical nature of the Internet, the routes received from a tier one ISP may have generally shorter AS path than the same routes from a tier two ISP. With BGP ultimately selecting one route, a tier one ISP may be heavily favored. As a result, you will likely see your outbound traffic mostly going to one of the ISPs. Having one circuit running over capacity while another is under-utilized is a poor scenario to justify for additional bandwidth upgrade expenses.

Asymmetrical routing

Most data centers advertise routes out to the Internet equally, to achieve resiliency and inbound load balancing. Extreme imbalance of outbound use combined with balanced inbound traffic, results in significant amount of asymmetrical routing. While asymmetrical routing is common on the Internet, excessive amount may lead to operational headaches such as difficulty in troubleshooting. You have probably been on a phone call that one party can hear much better than the other. As voice, video and other delay sensitive traffic continue to increase in terms of volume and significance, controlling quality and user experience requires minimizing asymmetrical routing whenever possible.

In conclusion, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs. And more often than not, the traffic is not well balanced. To achieve better outcome by design, in part 2 we will start with a simple alternative.

Cisco Flow Control with NetApp NAS

Update Nov 2012: Thanks for the comment, Paul. I have added an update with another look at flow control.

When NAS is used with virtualization, performance and throughput are potential concerns. Enabling jumbo has proven to be the most effective method. Flow control, on the other hand, is not always easily agreeable by all parties.

NetApp has a one paragraph “best practice” on the subject. The recommendation is to set flow control to “receive on” on the switch port. In other words, allow the switch to receive “pause” from NAS.

As it appears, NetApp can send a lot of “pause”. This is easily shown on the port channel or physical interfaces connected to NAS:

Nexus5k# show interface e2/7

Ethernet2/7 is up

…

  30 seconds input rate 116486568 bits/sec, 2079 packets/sec

  30 seconds output rate 33970464 bits/sec, 792 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 57.99 Mbps, 1.58 Kpps; output rate 32.18 Mbps, 693 pps

  RX

    20604669084 unicast packets  176831374 multicast packets  0 broadcast packet

s

    20781500458 input packets  63305126825983 bytes

    8255225288 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    176409426 Rx pause

  TX

    12091810302 unicast packets  52908470 multicast packets  2650659 broadcast p

ackets

    12147369431 output packets  48540927928599 bytes

    7298956759 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  0 interface resets

So what does it mean? Flow control is only meaningful if the receiving party acts on it. Therefore the expectation here is for switch to “slow down” the transmission, since it is hearing NAS saying “slow down, I can’t keep up”.

According to Cisco, once Pause is enabled and received on switch egress port, it will back pressure the ingress port, eventually packets will be buffered on the ingress port. If pause is enabled on the ingress port, it can further send the pause to the upstream switch. In the ideal scenario, the pause eventually reaches the source, which is the ESXi host, thus slowing down the origination of the transmission.

However, taking a second look at the NetApp diagram here, the recommendation is to configure the end points, ESX servers & NetApp arrays with flow control set to ‘send on’ and ‘receive off’. In other words, ESX is not expected to receive flow control, which leaves only the network to absorb Pause.

Now let’s revisit the flow control scenario again, NAS sends Pause, switches receives it, there is no use applying back pressure all the way to the host since host won’t receive it. The best a switch can do is to buffer it, or apply back pressure to upstream, and have the upstream switch buffer somewhere.

How well that works really depends on the switch and linecard models, each have different capabilities and buffer size. In many cases, it is highly questionable how far back flow control is propagating to have any positive effect. In any case, you want to check:

• Switch interface to NAS, to see the amount of Pause received
• All interfaces where NAS traffic flows, to see if there are drops

 More clarifications on this topic are probably required from vendors. Device behavior will likely evolve with technology advancements. For now, it’s best to turn flow control on switch side, but monitor network behavior closely.

Where to place VSM and vCenter

With Nexus 1000v, VSM and vCenter can run as VM under VEM, but that doesn’t mean they always should.

VSM is the “supervisor” for VEMs (virtual line cards). It also communicates with vCenter which is the central management and provisioning center for Vmware virtual switching.

As a network designer, we will need to work with host team to determine VSM’s form factor:

• As a VM running under VEM (taking a veth port)
• As a VM running under a vSwitch
• As a separate physical machine
• As an appliance (Nexus 1010 VSA)

As you can see, options range from complete integration in the virtualized environment, to complete separation, at increasing cost. Arguably, in a large and complex virtualization environment, the advantage of having separate control points will become more apparent. Here we briefly touch on two practical considerations.

Failure Scenarios

When everything works, there is really no disadvantage having VSM and vCenter plugged into a VEM. In theory, VSM can communicate even before VEMs are boot up, through control and packet VLANs which should be system VLANs. However, it could become a lot more complex to troubleshoot, when something is not right. For example, misconfiguration on vCenter leading to communication failure, software bug on the Nexus 1000v leading to partial VLAN failures, having a faulty line card with packet drops.

The point is, if there is a failure, we want to know quickly if it is in the control plane or the data plane. We often rely on the control plane to analyze what is going on in the data plane. Mixing VSM with VEM increases the risk of having control plane and data plane failure at the same time, making root cause isolation more difficult. However unlikely we may think, failure scenarios could happen. When it does, having access to VSM and vCenter is essential to troubleshooting and problem isolation. We know VEM does not rely on the availability of VSM to pass packets; however having VSM under VEM essentially places it under the same DVS that it manages, therefore subject to DVS port corruption error as an example. When a VEM fails, imagine losing access to VSM and vCenter as well because they are running under it.

Administrative Boundary

VSM and vCenter, due to their critical nature, needs to be protected. To prevent administrators from mistakenly change vCenter and VSM while making changes to other VMs, there should be as much administrative boundary established as the infrastructure supports.

Having VSM and vCenter in a separate control cluster with dedicated hosts creates clear administrative boundary. The use of a Vmware virtual switches (vDS) instead of VEM for vCenter and VSM will further decouple dependency. The vDS should be clearly named; its special purpose will be understood by all administrators, therefore minimizing the chance for mistakes.

The diagram shows a sample of placing VSM and vCenter as VMs on a separate control cluster separate from the applications VM they manage.

Spanning tree in a Nexus virtualized data center

Nexus VPC reduces the reliance on Spanning Tree in a data center design, and improves link utilization and load sharing. However, spanning tree is still a necessary component of the design, often seemingly more complex, due to various options available, and connection to third party devices.

There was an earlier post about Bridge Assurance. This note provides a more complete reference model (R-PVST), to capture recommended best practices in a typical design, followed by notes explaining why some of the choices made.
Click to see larger image.

Nexus 7000 VPC peer link and non-VPC link

Enable BA with “spanning-tree port-type network” on both ends of trunk (port channel)

Why Bridge Assurance: new Cisco STP feature works in conjunction with Rapid-PVST BPDUs to protect from bridging loops (also supported by MST). BA must be supported by and configured on both switches on a point to point link, otherwise blocking will occur. BA uses bidirectional hello to prevent looping conditions caused by unidirectional links or a malfunctioning switch.  If a BA port stops receiving BPDUs, the port is moved into the blocking state.

Nexus 7000 connection to third party Load Balancer or Firewall devices

Enable port fast with “spanning-tree port-type edge trunk”

Why Port Fast: This is for third party devices which can be treated like hosts connected to access (they do not send BPDUs). Setting port-type to edge enables Port Fast which allows access port to enter the forwarding state immediately, instead of waiting for STP to converge. These ports should not receive bridge protocol data units (BPDUs), otherwise they will immediately transition to the blocking state. The trunk keyword enables edge behavior on a trunk port.

Why BPDU Guard: BPDU Guard works together with Port Fast on edge ports. In a valid design, edge ports should not receive BPDUs. Reception of a BPDU indicates an error in configuration, such as connection of an unauthorized device.  By shutting down a port that receives a BPDU, BPDU Guard protects the network, since only an administrator can put the edge port back in service.

Nexus 7000 downlink to Nexus 5000 access switches (back to back VPC)

Enable root guard on “spanning-tree port-type normal”, note this is applied on VPC

Why Root Guard: Note Nexus 7000s are SPT root. Root Guard is a feature placed on aggregation port facing access switches, preventing it from becoming a root port. In this case it prevents Nexus 5000 to become a SPT root switch, to ensure that a configuration error on an access layer switch does not cause STP disruption and instability.

SPT on VPC is a topic in itself. Cisco has a good document here with more details. Note bridge assurance is usually not needed, since VPC consistency check ensures the integrity of configurations.

Nexus 5000 uplink to Nexus 7000 (back to back VPC)

Enable loop guard on “spanning-tree port-type normal”, note this is applied on VPC

Why Loop Guard: For additional protection against loops, Loop Guard is enabled on root and alternate ports (facing root bridge). When Loop Guard on Nexus 5000 detects that BPDUs are no longer being received, the port is moved into a loop-inconsistent state instead of transitioning through SPT convergence. This will break a Layer 2 loop immediately due to misconfiguration or unidirectional link.

Nexus 5000 connection to hosts

Enable port fast with “spanning-tree port-type edge trunk”

Why Port Fast: see above, same as Nexus 7000 connection to third party.

Why BPDU Guard: see above, same as Nexus 7000 connection to third party.

A final note, Cisco Nexus 1000V does not run SPT, it does not generate BPDUs, nor does it respond to them. Nexus 1000V examines source and destination MAC address to prevent loops. This is the reason Nexus 5000 access port connected to Nexus 1000v virtual switch should be set to type edge, just like an access port connected to a physical host.

vmware ESX/ESXi host network load sharing options - Simplified

Why load sharing

With virtualization, network and server domains converge; there are usually multiple vendors for server, NIC, storage, and networks. A typical example is using Nexus 1000v which is a Cisco product, embedded in ESX which is VMware, utilizing an HP NIC, and interacting with NAS which is yet another vendor.

Why load sharing? It is high desirable to have redundant uplinks from an ESX host for high availability. In addition, load sharing over redundant uplinks improves performance and utilization. So what are the load sharing options from the host?

A fundamental design is how traffic flows to and from VM to the rest of the network. In this example, VMs reside in ESX, but the concept is the same for any virtualization host interacting with the network.

There have been numerous vendor documents, often covering a certain aspect in detail, occasionally conflicting and confusing as technologies have been evolving. I posted earlier about load sharing mainly on Nexus switch side. Scott Lowe has a series of excellent articles on the topic. Why the summary here? I found it necessary to organize multiple concepts around host load sharing under a simple framework to make it easier to understand and apply.

Options

The following table summarizes common load sharing options, from the least desirable to the most.

click to see full size image of the summary.

Additional Considerations - LACP

The last two options require some clarification. Without Nexus 1000v, VMware does not support dynamic DHCP.  Therefore VMware documentation usually specifies “mode on” as the recommended configuration. Cisco definitely supports dynamic LACP, although earlier Nexus 1000v releases may have had some LACP specific bugs. The latest Nexus 1000v Release 4.2(1) SV1(4) contains many fixes, but is to be proven in a production system, it also has the additional benefit of LACP offload from VSM to VEM. If you know of more recent developments or road map, please kindly let me know.

Jumbo MTU on Nexus 1000v – Why and How

I posted earlier about the use of “system mtu” on Nexus 1000v.  The internetworking of virtual switch in a physical topology can still be confusing, and the existence of likely display bugs probably didn’t help with the situation.

So here is my understanding of why and how, hopefully in more simple terms than any documentation I have seen.

First, why enable jumbo MTU on Nexus 1000v? Jumbo support is desirable for applications such as storage, for example your ESX host in a NAS environment. Enabling jumbo on Nexus 1000v is part of the “end-to-end” jumbo configuration, starting from the host. In a typical topology, you will need to configure Nexus 7000, Nexus 5000, and Nexus 1000v since it is the virtual switch that partly resides on the host.  You can refer to more configuration details here.

A diagram is helpful to illustrate the relationship between Nexus 5000 and Nexus 1000v, particularly the confusing port channel relationship.  Note on 5000 VPCs are typically configured (Po101-103). On Nexus 1000v, only system uplink profile is predefined. Port channel are created only when hosts/VEMs are added. Adding host on vCenter triggers the creation of matching port channels (Po5-7) in VSM.

Next, how to enable jumbo MTU on Nexus 1000v?

1.       Set system jumbomtu (set by default)

            You really don’t need to do anything here, because by default it has already been set to maximum

\N1kv# sh run all | inc jumbo

system jumbomtu 9000

2.       Set system mtu

This is really the only configuration required on Nexus1000v. By setting system mtu, we can preserve the physical NIC setting through ESX reboot

port-profile type ethernet systemuplink_portchannel

  system mtu 9000

Lastly, how to verify that jumbo is working?

This is the most confusing part. Depending on your port type (LCAP etc), you may get different display results. In my experience, both “show int port-channel” and “show interface Ethernet” displays MTU 1500 (which is incorrect). This display error is similar to one on Nexus 5000 (CSCsl21529). But on Nexus 1000v there is no jumbo packet count, thus making verification even more difficult.

Nexus-1000v-VSM# sh int po1

port-channel1 is up

  Hardware: Port-Channel, address: 0050.5655.cbe0 (bia 0050.5655.cbe0)

  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

Nexus-1000v-VSM # sh int e3/5

Ethernet3/5 is up

  Hardware: Ethernet, address: 0050.5655.dbc8 (bia 0050.5655.dbc8)

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 0/255, txload 0/255, rxload 0/255

To alternative to verify that jumbo is really working on Nexus 1000v, is simply to check upstream switch. For example, on Nexus 5000, check either the corresponding port channel or physical interface. Since Nexus 5000 is receiving (RX) jumbo packets from the host, it is a clear indication of the function of Nexus 1000v uplink profile.

Nexus-5010-a# sh int po101

port-channel101 is up

 vPC Status: Up, vPC number: 101

  Hardware: Port-Channel, address: 0005.9b76.308f (bia 0005.9b76.308f)

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  EtherType is 0x8100

  Members in this channel: Eth1/8

  Last clearing of "show interface" counters never

  30 seconds input rate 4344 bits/sec, 543 bytes/sec, 1 packets/sec

  30 seconds output rate 44352 bits/sec, 5544 bytes/sec, 48 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 4.50 Kbps, 1 pps; output rate 47.82 Kbps, 47 pps

  RX

    11483591 unicast packets  205769 multicast packets  30036 broadcast packets

    11719396 input packets  2923241828 bytes

    274665 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    134696185 unicast packets  163700959 multicast packets  12680857 broadcast p

ackets

    311078001 output packets  240793383992 bytes

    47357469 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  2 interface resets


Nexus 1000v is only a component in the larger picture. Scott Lowe provides a great example here to show how jumbo support is enabled on ESX server and NAS. In particular, testing jumbo from ESX using vmkping command:

vmkping -s 9000