AWS hybrid cloud – shared customer gateway to multiple VPCs

VPN connection is the most commonly used method to connect enterprise data centers to AWS Virtual Private Cloud. More often than not, enterprises use multiple VPCs, sometimes across multiple regions. For a large enterprise doing business internationally, its global data center footprint can be linked to AWS data centers across multiple continents, thus forming a global hybrid cloud.

Clearly, it is desirable to have a redundant pair of Customer Gateways to terminate multiple VPN connections to multiple VPCs. Having dedicated gateway to each VPC is neither scalable nor cost efficient. Automation will be impossible if new hardware is required each time a new VPC is created.

However, AWS’s current VPN connection creation logic is designed to ensure unique tunnel IP addresses for each connection within a single VPC, but not necessarily across multiple VPCs. In other words, each time a new VPC establishes tunnels to existing customer gateway, there is a real risk that address conflict will bring down existing production traffic.

In this AWS guide, a couple of workarounds are suggested to support connecting a single customer router to multiple VPCs. However, neither workaround is elegant. In the first case, implementing VRF just for the sake of this deficiency creates unnecessary complexity on the customer’s part, not to mention some devices may not support VRF. In the second case, which is essentially trial and error, automated VPN creation through scripting and API becomes extremely cumbersome.

AWS Feature Request

Because each customer gateway is uniquely identifiable (at least within the region), it seems logical (and desirable) for AWS to implement a feature to ensure uniqueness of tunnel address generated for shared customer gateways.

•    Phase 1 may support uniqueness within the region, as illustrated.
•     Phase 2 may support uniqueness across multiple regions. In a global hybrid cloud, enterprise data centers may connect to multiple AWS regions in multiple continents.

Sorting out System MAC addresses with VPC and VSS – Part 2

Following Part 1 which starts with VPC on Nexus platform, here VSS on Catalyst is compared side by side.

A simple and interesting topology can be used to illustrate. In this case, Nexus and Catalyst use different multichassis technology (VPC and VSS respectively), forming back to back virtual port channel. The effective logical topology becomes greatly simplified (shown on the right side), with benefits including utilization of full bisectional bandwidth, stable all forwarding STP, high resiliency, and ease of adding/removing physical members etc.

VSS Domain ID is very much similar to VPC Domain ID. It is a unique identifier in the topology, which represents logical virtual switch formed by two physical chassis. Only one VSS pair is associated with a particular domain.

Sorting out System MAC addresses with VPC and VSS – Part 1

A number of multichassis aggregation technologies are deployed in the data center today, for example, Cisco’s Multichassis EtherChannel (MEC) on catalyst 6500 VSS, and Virtual Port Channel (vPC) on Nexus platforms. Inter-chassis aggregation greatly increases link utilization, while simplifying design by eliminating topology dependence on spanning tree protocol. STP becomes passive as most links are forwarding, and most failure scenarios no longer require STP re-convergence, thus minimizing disruptions. Furthermore, a more elegant data center design can be achieved, with lower operational complexity, and higher return on investment.  

System MAC address exists on individual devices, often used for device level negotiation, for example, bridge ID field in STP BPDU, or as part of LACP LAGID.

When multiple chassis operate in unison, software simulates the behavior of a common logical system, with the use of common virtual identifiers. Differentiating and sorting out the use of virtual system identifier and various MAC addresses is helpful for understanding, designing and deploying such systems.

It can be illustrated with a simple topology such as the one shown in the diagram, in which a pair of Nexus (in VPC domain 100) is connected to another pair (in VPC domain 101) on back to back VPCs.

vmware ESX/ESXi host network load sharing options - Simplified

Why load sharing

With virtualization, network and server domains converge; there are usually multiple vendors for server, NIC, storage, and networks. A typical example is using Nexus 1000v which is a Cisco product, embedded in ESX which is VMware, utilizing an HP NIC, and interacting with NAS which is yet another vendor.

Why load sharing? It is high desirable to have redundant uplinks from an ESX host for high availability. In addition, load sharing over redundant uplinks improves performance and utilization. So what are the load sharing options from the host?

A fundamental design is how traffic flows to and from VM to the rest of the network. In this example, VMs reside in ESX, but the concept is the same for any virtualization host interacting with the network.

There have been numerous vendor documents, often covering a certain aspect in detail, occasionally conflicting and confusing as technologies have been evolving. I posted earlier about load sharing mainly on Nexus switch side. Scott Lowe has a series of excellent articles on the topic. Why the summary here? I found it necessary to organize multiple concepts around host load sharing under a simple framework to make it easier to understand and apply.

Options

The following table summarizes common load sharing options, from the least desirable to the most.

click to see full size image of the summary.

Additional Considerations - LACP

The last two options require some clarification. Without Nexus 1000v, VMware does not support dynamic DHCP.  Therefore VMware documentation usually specifies “mode on” as the recommended configuration. Cisco definitely supports dynamic LACP, although earlier Nexus 1000v releases may have had some LACP specific bugs. The latest Nexus 1000v Release 4.2(1) SV1(4) contains many fixes, but is to be proven in a production system, it also has the additional benefit of LACP offload from VSM to VEM. If you know of more recent developments or road map, please kindly let me know.

Nexus7000/5000/1000v – A closer look at VPC and port channel load balancing

Port channel is a great redundacy and load sharing feature in data centers. Cisco Nexus takes it one step further with Virtual Port Channel (VPC). There are numerous good documentations about VPC, including the downloadable design guide.
Sometimes you need to look under the cover, and see exactly how each physical port is utilized by port channels. This note highlights a few useful commands.

Strictly speaking, port channel does load sharing, not load balancing. So there should not be an expectation for 50/50 balance. How well load sharing works largely depends on the traffic and the hashing method selected. For example, Nexus 1000v supports 17 hashing algorithms to load-share traffic across physical interfaces in a PortChannel, including source-based hashing and flow-based hashing. The default is source-mac.

Another important concept: hashing is uni-directional, determined by the sending party. Therefore there is no guarantee that load sharing will be symmetrical. For illustration, Nexus 7k and 5k are connected in what is known as “back to back” VPCs (port channel 75). A Nexus 7k has multiple physical connections southbound on the same logical channel. It determines the physical port to send traffic based on local hashing, as illustrated by the green arrow.

Here is the first command which shows the hashing algorithem used:
show port-channel load-balance
N-7010-1# sh port-c load
Port Channel Load-Balancing Configuration:
System: source-dest-ip-vlan
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-dest-mac
IP: source-dest-ip-vlan

The second command shows how well load sharing is working on your port channels. Note statistics are accumulative, and is reset by clearing corresponding interface counters.
Show port-channel traffic

N-7010-1# show port-c traffic
ChanId Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
75 Eth1/5 53.16% 42.58% 49.91% 53.70% 44.03% 44.86%
75 Eth1/6 46.83% 57.41% 50.08% 46.29% 55.96% 55.13%

As in the diagram, orange arrow indicates Nexus 5k northbound load sharing based on its hashing algorithem. The blue arrow indicates Nexus 1kv northbound load sharing based on its hashing.

Since Netflow is not yet supported on Nexus 5k, how do we tell which physical interface will a certain flow take? Here is the third command:

show port-channel load-balance forwarding-path interface ...

The following example shows how different IP address pairs yields different port utilization.
N-5010-1# show port-channel load-balance forwarding-path interface port-channel 75 vlan 25 src-ip 10.17.19.15 dst-ip 10.17.21.15
Missing params will be substituted by 0's.
Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 90 Outgoing port id: Ethernet1/3
Param(s) used to calculate load-balance:
dst-ip: 10.174.21.15
src-ip: 10.174.19.15
dst-mac: 0000.0000.0000
src-mac: 0000.0000.0000

N-5010-sw1# show port-channel load-balance forwarding-path interface port-channel 75 vlan 25 src-ip 10.17.19.10 dst-ip 10.17.34.15
Missing params will be substituted by 0's.
Load-balance Algorithm on switch: source-dest-ip
crc8_hash: 179 Outgoing port id: Ethernet1/4
Param(s) used to calculate load-balance:
dst-ip: 10.174.34.15
src-ip: 10.174.19.10
dst-mac: 0000.0000.0000
src-mac: 0000.0000.0000

If you are running a test, and notices that the traffic is not well balanced, now you have a method to check the physical port allocation for your particular end points. You also have the option to experimenting with different hashing algorithem to suit your needs.