AWS VPC VGW Multipath Routing - difference between Direct Connect and VPN

VPC VGW multi-path scenario

To connect a VPC to enterprise networks or other VPCs, we use Direct Connect or VPN. It is common to have multiple connection paths from a VPC. Routing outbound from a VPC is controlled by VGW. The question is, how does VGW which is an AWS internal logical router handle multi-path routing?

Multi-path is a requirement for high availability. Load sharing on multi-path is often desirable. How VGW handles multi-path routing is actually different based on connection type. Specifically, Direct Connect supports ECMP. VPN does not (after Oct 2015).

Direct Connect

Direct Connect supports the configuration option of redundant paths with Active/Active (BGP multipath), VGW routes traffic over multiple equal cost paths. As a result, we can leverage all bandwidth resources provisioned for DX.

https://docs.aws.amazon.com/directconnect/latest/UserGuide/getstarted_sub1g_provider.html#redundant_connections_sub1g_provider

VPN

With VPN, VGW currently does not support BGP multipath. VPN chooses one BGP path only.

https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html#MultipleVPNConnections

What if we use static route instead of BGP, can static be used to load share traffic across multiple paths?

In the scenario shown in the diagram, there are dual VPN connections going to two remote CGWs, each with redundant tunnels. If static routes are defined equally, does VGW route ECMP out multiple paths?

• VGW created prior to Oct 28 2015 supports static multipath.
• VGW created after Oct 28 2015 selects one active path out of multiple paths defined

The scenario is tested with a new VGW in one VPC, and a pair of customer VPN appliances in aonther VPC. With 4 tunnels/paths, it seems all traffic goes to one tunnel only. AWS support confirmed the behavior that VGW only selects one path only.

Why AWS should support VPN multipath

With VPN, it may be desirable to spread load across multiple customer gateways, because those customer gateways may be Cisco or Palo Alto appliances that has licensed throughput capacity. It is more optimal to spread load across multiple destinations rather than sending all traffic to one while other paths sit idle.

Hopefully AWS will bring consistent multipath routing to VPN, with BGP multipath and static ECMP.

Enable BGP load sharing with Multipath-relax

If your network is like most large enterprise’s, chances are you have Internet connection from dual ISPs to your data center.

It used to be a popular tactic to have a primary provider, and a backup provider which you pay lesser bandwidth for. In recent years, usage based circuits has given way to flat rate, often larger bandwidth circuits from both ISPs. After all, users demand more bandwidth, even in failure scenarios.

Why would you pay for expensive Internet access and only letting it sitting idle most of the time? Therefore, load sharing on redundant ISP connections has become highly desirable for data centers.

By default, BGP only installs one route into the routing table. To accomplish load sharing, BGP multipath needs to be enabled. To further complicate the matter, BGP requires all paths to have the following equal characteristics in order to become equal cost path:
• Weight
• Local preference
• AS-PATH length
• Origin
• MED
• AS-PATH

Since neighboring AS or AS-PATH must be equal, that means BGP can only load balance to a single AS or single ISP. Since different ISP will have different AS and AS-PATH, BGP will only pick one best route.

So how does Cisco support load balancing to two ISPs? There is a hidden command bgp bestpath as-path multipath-relax

I don’t know why it is hidden command, since it appears to be a very useful feature, and it is supported. But do use with caution, as the number of routing entries will increase, often overwhelming the limited memory you have on the Internet router.

In this year’s Networker, an alternative method was also shown, essentially separating out the Internet routing table by even and odd digits, for outbound traffic.