AWS VPC VGW Multipath Routing - difference between Direct Connect and VPN

VPC VGW multi-path scenario

To connect a VPC to enterprise networks or other VPCs, we use Direct Connect or VPN. It is common to have multiple connection paths from a VPC. Routing outbound from a VPC is controlled by VGW. The question is, how does VGW which is an AWS internal logical router handle multi-path routing?

Multi-path is a requirement for high availability. Load sharing on multi-path is often desirable. How VGW handles multi-path routing is actually different based on connection type. Specifically, Direct Connect supports ECMP. VPN does not (after Oct 2015).

Direct Connect

Direct Connect supports the configuration option of redundant paths with Active/Active (BGP multipath), VGW routes traffic over multiple equal cost paths. As a result, we can leverage all bandwidth resources provisioned for DX.

https://docs.aws.amazon.com/directconnect/latest/UserGuide/getstarted_sub1g_provider.html#redundant_connections_sub1g_provider

VPN

With VPN, VGW currently does not support BGP multipath. VPN chooses one BGP path only.

https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html#MultipleVPNConnections

What if we use static route instead of BGP, can static be used to load share traffic across multiple paths?

In the scenario shown in the diagram, there are dual VPN connections going to two remote CGWs, each with redundant tunnels. If static routes are defined equally, does VGW route ECMP out multiple paths?

• VGW created prior to Oct 28 2015 supports static multipath.
• VGW created after Oct 28 2015 selects one active path out of multiple paths defined

The scenario is tested with a new VGW in one VPC, and a pair of customer VPN appliances in aonther VPC. With 4 tunnels/paths, it seems all traffic goes to one tunnel only. AWS support confirmed the behavior that VGW only selects one path only.

Why AWS should support VPN multipath

With VPN, it may be desirable to spread load across multiple customer gateways, because those customer gateways may be Cisco or Palo Alto appliances that has licensed throughput capacity. It is more optimal to spread load across multiple destinations rather than sending all traffic to one while other paths sit idle.

Hopefully AWS will bring consistent multipath routing to VPN, with BGP multipath and static ECMP.

BGP RIB-failure and effect on route advertisement

When examining routes advertised to BGP neighbor, notice some routes are tagged with “r”:

rtr1#sh ip bgp neighbor 10.11.19.21 advertised

BGP table version is 1735468, local router ID is 10.115.254.254

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter, a additional-path

Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path

r>i10.115.254.0/30 10.115.254.9            0    100      0 i

*> 10.115.254.0/23 10.115.254.9           21         32768 i

r>i10.115.254.4/30 10.115.254.9            0    100      0 i

r>i10.115.254.8/30 10.115.254.9            0    100      0 i

Note this is BGP “RIB-failure”, which indicates BGP fails to install the route in routing table. According to this link, the likely cause is the route is already installed by IGP which has a lower AD.

Data Center ISP Load Sharing Part 4 – Tuning

Does full Internet routing table work the best for load sharing with multi-homed ISP connections? We have shown it is often not the case.

Part 1 of the posting shows  the challenges of dual ISP design, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs.

Part 2 of the posting shows the advantage of simple default route based Internet load sharing design.

Part 3 of the posting introduces a design that combines the simplicity of default based load sharing to dual ISP, and flexibility of selectively filtering subsets of Internet routes for optimal path selection.

In this final part we look at why and how the results should be fine-tuned.

At the initial design stage, you could estimate the number of routes filtered in from your respective ISPs, using the BGP regular expression you designed. Route count estimate provides the basis for your filtering design.  For example, you can count that approximately 50000 routes will be allowed in by a filter specifying only adjacent networks to a tier one ISP. You also count that approximately 40000 routes will be allowed in by another filter specifying adjacent networks as well as those one hop away from a tier two ISP.

After implementation, you will notice the actual number of specific networks allowed in will be less than the combined total of 50000 plus 40000. The total number (for example 80000) is less than the total due to duplicates. In other words, you learn the same 10000 routes from both ISPs because those networks are adjacent to both ISPs. This is common and to be expected.

You might have expected the duplicate routes to be split more or less evenly across the two ISPs, which is often not the case. Therefore, the effect of duplicate routes on load sharing requires some careful observation. ISPs may operate in different tiers of the internet hierarchy, thus affecting the routes they advertise to have shorter or longer AS path length.  AS path length is a primary criterion in BGP path selection, therefore you will likely see almost all of the duplicate routes favoring one ISP. This may affect load sharing, thus require further adjusting the filters.

A second example is the influence of ISP metrics. Some ISP may advertise routes with a metric, while others advertise all routes with zero metric. Zero metric will be preferred if other more priority criteria is equal.

The diagram shows the original design may have expected a load sharing design of 5/4. However, the result shows load sharing between ISP1 and ISP2 turns out to be 5/3, due to all duplicate routes favoring ISP1. Depending on your specific requirements, fine-tuning of the filter may be necessary.

Design for redundant Internet architecture is unique to every organization’s requirements, its national or global data center architecture, the ISPs selected, and the nature of its Internet traffic. The scenarios described hopefully have provide simple templates as references to adjust for your particular data center.

Data Center ISP Load Sharing Part 3 – Route Selection

Part 2 of the posting shows the advantage of simple default route based Internet load sharing design. This part further optimize the design.

Using the entire Internet routing table for outbound load sharing proves to be resource intensive, and ineffective for load balancing. Default route only provides simplicity and better load balancing. To further optimize, a subset of Internet routes, when selected according to the unique environment, can complement the default route design very well.

Route Selection

Route selection refers to filtering and allow a subset of the Internet routing table to be introduced into the data center. The desired effect is to take the shorter path to content that is directly attached to specific ISPs, while the rest of the traffic load share equally to both ISPs.

The effectiveness of the design is largely based on route selection techniques applicable to the specific data center environment. In the example shown below, BGP regular expression is used to select a subset of Internet destinations adjacent to each ISP.

ISP1 is a tier one, therefore has more directly attached networks. BGP expression is used to select those directly attached networks, with the objective that traffic destined for those networks will exit on this ISP for optimal path.

ISP2 is a tier two, with less number of directly attached networks. BGP expression is used to select those directly attached networks as well as those one additional hop away, with the objective that roughly equal number of specific target networks will prefer ISP2 as the exit point, thus achieving load sharing with both ISPs.

Implementation

On the respective internet router connected to each ISP, AS path filtering is applied on a route map, which is then applied to BGP inbound route filtering. As a result, the default route, as well as a subset of Internet routes is received from each ISP, in order to optimize outbound traffic to take the more direct path to destinations.

ip as-path access-list 1 permit ^3549_[0-9]*$

route-map ISP1in permit 10 

 match ip address prefix default 

route-map ISP1in permit 20 

 match as-path 1 

router bgp

 neighbor … route-map ISP1in in

Verification and Tuning

At the planning stage, counting number of routes using BGP regular expression filter may serve to arrive at the initial route selection design. By filtering in similar amount of specific routes from each ISP, the desired load sharing can usually be achieved. 

However, equivalent number of routes does not always result in equivalent amount of traffic. Over time, actual load on the respective ISP connections will provide more accurate information about traffic in the particular data center. ISP specific characteristics may also factor in. Part 4 will show why fine-tuning may be necessary.

Data Center ISP Load Sharing Part 2 – Default Method

Part 1 of the posting shows  the challenges of dual ISP design, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs. And when routes received from ISPs have different characteristics such as AS Path and metric, the result is often undesirable. To achieve better outcome by design, in part 2 we will start with a simple alternative.

Replacing the entire Internet routing table with just the default route is an extremely simple method that offers a number of advantages.

Load balancing

Instead of getting the entire Internet routing table, only default route is received and installed in the routing table. As a result, IGP can load balance to two equal cost default routes. For outbound traffic, the simple design achieve near 50/50 load balancing, as well as resiliency.

Simplicity, Stability and Resource Efficiency

The design is extremely simple to implement and support. Resource usage on devices can be greatly reduced, from holding tens and thousands of Internet routes, to just default route. Route flapping and any disruptive convergence due to instability in any parts of the Internet is virtually eliminated.

The simplicity advantage is well suited for a large number of enterprise data centers.

Disadvantage

The design essentially “splits” the Internet in half, by two equal cost default routes to dual ISPs. Therefore, the exit point may not be optimal, especially for networks directly attached to an ISP, which may require the longer path to get to.

For vast majority of applications, the selection of exit ISP is not noticeable. However, lower latency access of large amount of media content may be highly desirable when a direct path is available. An optimized solution is presented in part 3.

Data Center ISP Load Sharing Part 1 - The challenge

Redundant ISP connections are the standard in most data centers. There is surprisingly little update-to-date information about data center Internet architecture out there. In fact, traffic flow on the internet connections while increasingly business critical, is probably among the least well understood area in enterprise network architecture.  Consequently, the potential for cost saving and performance improvement is substantial.

This is the first part of a series that explore this interesting and important topic, which starts by examining a classic solution and its associated challenge.

For a large data center, it is often believed that obtaining the entire Internet routing table from each ISP provides the most comprehensive and complete routing information to direct traffic towards the Internet. However, here are a few potential issues to consider:

Heavy resource utilization

The design may propagate the entire internet routing table in the internal BGP environment within the data center, so that gateways can determine which ISP to exit to for a given Internet destination. Most Internet routers are designed to handle a large number of routes. However gateway devices particularly older generation switches may be running near their CPU and memory maximum. You might want to check CPU utilization on all your internet facing devices - high CPU utilization is often indicative that some routes are software switched. Performance will degrade as a result.

This diagram from CIDR shows continuous growth of BGP prefixes over the years. Expensive upgrades are often required just to keep up with the growth, in order to maintain the performance of hardware forwarding.

Imbalance of ISP bandwidth

The second issue varies with the particular ISP you chose. When you select your ISPs, did you consider how the selection may affect your design? It is very common to have a tier one ISP and tier two ISP in one data center. With hierarchical nature of the Internet, the routes received from a tier one ISP may have generally shorter AS path than the same routes from a tier two ISP. With BGP ultimately selecting one route, a tier one ISP may be heavily favored. As a result, you will likely see your outbound traffic mostly going to one of the ISPs. Having one circuit running over capacity while another is under-utilized is a poor scenario to justify for additional bandwidth upgrade expenses.

Asymmetrical routing

Most data centers advertise routes out to the Internet equally, to achieve resiliency and inbound load balancing. Extreme imbalance of outbound use combined with balanced inbound traffic, results in significant amount of asymmetrical routing. While asymmetrical routing is common on the Internet, excessive amount may lead to operational headaches such as difficulty in troubleshooting. You have probably been on a phone call that one party can hear much better than the other. As voice, video and other delay sensitive traffic continue to increase in terms of volume and significance, controlling quality and user experience requires minimizing asymmetrical routing whenever possible.

In conclusion, the traditional approach of outbound load sharing based on entire internet routing table will largely depend on the particular ISPs. And more often than not, the traffic is not well balanced. To achieve better outcome by design, in part 2 we will start with a simple alternative.

OSPF EIGRP BGP dual point mutual redistribution - Part 3

In part 1 and part 2 of the post, we mainly focused on OSPF and EIGRP mutual redistribution. We use administrative distance to control preference, and use tags to prevent loops.

In a typical enterprise WAN environment, carrier MPLS can also be used to carry traffic between sites and data centers. In a resilient architecture, there are multiple paths to the same destination. The business requirement may be such that certain traffic should take one type of link as its primary path, while still having a backup path in case of failure.

In the illustration, MPLS provides the WAN backup path for direct facilities (OSPF-EIGRP). BGP is used as the dynamic routing protocol through the MPLS cloud.

Recall from part 2, tag 25 is used to indicate routes originated in OSPF, and prevented from feed back from EIGRP back to OSPF. Why is there a third issue with BGP? Because the same route is advertised out of OSPF to MPLS via BGP. The data center running EIGRP will also learn the same route from MPLS cloud as a BGP route, in this case not tagged. On EIGRP to OSPF redistribution point, the tag filter does not stop feedback from a route learned via BGP. As long as east coast has a feasible successor (one with metric lower than current best FD), then this route will be advertised to west coast, with EIGRP distance of 100, thus preventing redistribution.

This is an example of network with the feedback issue. Note update tagged “1979” is sent due to better next hop FD. The end result is a network originated from west coast advertised out MPLS, became advertised back from east coast back on EIGRP, and preventing desired redistribution from OSPF into EIGRP.

East-RTR1#sh ip eigrp top 172.31.44.0 255.255.254.0
EIGRP-IPv4 (AS 100): Topology default(0) entry for 172.31.44.0/23
State is Passive, Query origin flag is 1, 2 Successor(s), FD is 30464
Routing Descriptor Blocks:
10.48.137.101 (GigabitEthernet0/0/1), from 10.48.137.101, Send flag is 0x0
Composite metric is (30464/30208), Route is External
Vector metric:
Minimum bandwidth is 625000 Kbit
Total delay is 1030 microseconds
Reliability is 255/255
Load is 31/255
Minimum MTU is 1500
Hop count is 3
External data:
Originating router is 168.147.152.166
AS number of route is 1
External protocol is OSPF, external metric is 20
Administrator tag is 250 (0x000000FA)
10.48.138.101 (GigabitEthernet0/0/2), from 10.48.138.101, Send flag is 0x0
Composite metric is (30464/30208), Route is External
Vector metric:
Minimum bandwidth is 625000 Kbit
Total delay is 1030 microseconds
Reliability is 255/255
Load is 2/255
Minimum MTU is 1500
Hop count is 3
External data:
Originating router is 168.147.152.166
AS number of route is 1
External protocol is OSPF, external metric is 20
Administrator tag is 250 (0x000000FA)
10.250.32.205 (GigabitEthernet0/2/0), from 10.250.32.205, Send flag is 0x0
Composite metric is (32768/7168), Route is External
Vector metric:
Minimum bandwidth is 625000 Kbit
Total delay is 1120 microseconds
Reliability is 255/255
Load is 2/255
Minimum MTU is 1500
Hop count is 3
External data:
Originating router is 10.250.248.1
AS number of route is 64601
External protocol is BGP, external metric is 0
Administrator tag is 1979 (0x0000369B)

The fix is by also setting tag (25) from routes coming from the west coast data center (identified by originating AS number) on the redistribution point from BGP to EIGRP. These tagged routes can be prevented from “feedback” on the EIGRP link with a route map. The route map can be applied on EIGRP interface distribute-list.

Optimize Google and Netflix with ISP

This is the first part of a series to explore, regarding optimizing enterprise Internet applications.

I estimated that on average, I use Google at least a dozen times on an average work day. I don’t think I will be far off to say that it is the top Internet application in almost every enterprise. While writing this article, I searched a couple of key words and arrived at a few web sites to find the specific information I was looking for. I did not have any books or references with me. More often, business users expect information on demand. It is often faster to get the information on the Internet than finding on the hard drive, and sometimes more accurate than asking your buddies.

Why this matters? Google is just an example here. For an Internet application that triggers millions of transactions almost every day in your enterprise, it is worthwhile to look “into the cloud”, and understand how your enterprise is interconnected to the rest of the world.

Google’s primary AS is 15169. It originates more than a hundred IPv4 prefixes for common apps, such as www.google.com. So how are you connected to AS15169? The average AS path length from Google is almost 3. But if you share one of the ISPs that Google peer directly with, then your path will be shorter.

The following lists the top IPv4 peers for AS 15169:

ASN Name

AS3356 Level 3 Communications

AS3549 Global Crossing

AS1299 TeliaNet Global Network

AS7018 AT&T Services, Inc.

AS174 Cogent Communications

AS6762 Telecom Italia Sparkle

AS3320 Deutsche Telekom AG

AS286 KPN Internet Backbone

AS6939 Hurricane Electric, Inc.

AS3257 Tinet SpA

As an example, check the internet routing table to see how you reach IP address for Google.com. The second shorter path (through a tier one ISP) will always be preferred over the first one.

Internet_router#show ip bgp 173.194.33.104
BGP routing table entry for 173.194.33.0/24, version 2010015
Paths: (2 available, best #2, table default)
Not advertised to any peer
13789 2828 15169
…
3356 15169
…

The significance goes beyond the length of the path or how many ISPs to transit through. Larger ISPs are becoming content providers in order to further optimize performance. Just a few days ago, Level 3 signed a deal with Netflix to “support storage for the entire Netflix library of content”. Level 3 has become the primary content network for Netflix, while Akamai and Limelight remain as partners. In this case, if your users are in the entertainment business, you will be at significant advantage by peering directly with Level 3.

There are more considerations, inbound vs outbound, streaming, CDN, geographical, international, resilience and cost… more to follow.

BGP timer and Cisco/Juniper negotiation

Juniper SRX integrates firewall features with full routing capabilities. In a network architecture where there are increasing demand for security segmentation and stateful firewall inspection within the network, multi-vender routing interaction may become necessary.

Risks and complexity exist with a mixed vendor environment, even when both sides support standard protocols. Here is such an example.

To establish BGP neighbor relationship, both sides need to agree on timer values. RFC4271 defines the characteristics of the timer, but does not specify value for keepalive and hold time.

BGP uses keepalive timers to monitor established connections. If for a period of time that exceeds hold time, BGP considers the neighbor connection down. Note the hold time is always three times the keepalive time.

By default, vendors set timers differently:
vendor Keepalive (second) Hold time (second)
Cisco Nexus (4.2) 60 180
Juniper SRX (10.0) 30 90

The timers should match between vendors. This is based on configuration, or negotiated behavior. Note with Juniper SRX, BGP on the local routing device uses the smaller of either the local hold-time value or the peer’s hold-time value received in the open message as the hold time for the BGP connection between the two peers. Therefore, by setting timer on Cisco Nexus to a smaller value (10/30), it is used by SRX.

To reduce convergence, the test sets Nexus keepalive to 10 seconds. This shows after successful negotiation, SRX (with a local hold time of 90) uses Nexus’s keepalive of 10 seconds, and active hold time of 30 seconds.

On Juniper SRX:
Peer: 10.88.15.14+179 AS 65000 Local: 10.88.14.4+54104 AS 65000
Type: Internal State: Established Flags:
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: None
Export: [ bgp_outbound_policy ]
Options:
Local Address: 10.88.14.4 Holdtime: 90 Preference: 170
Number of flaps: 0
Peer ID: 10.88.120.130 Local ID: 10.88.14.4 Active Holdtime: 30
Keepalive Interval: 10 Peer index: 0
…

On Cisco Nexus:
Nexus7k# sh ip bgp nei vrf test
BGP neighbor is 10.88.14.4, remote AS 65000, ibgp link, Peer index 1
BGP version 4, remote router ID 10.88.14.4
BGP state = Established, up for 00:53:15
Last read 00:00:05, hold time = 30, keepalive interval is 10 seconds
Last written 0.956168, keepalive timer expiry due 00:00:09
Received 20433 messages, 1 notifications, 0 bytes in queue
Sent 18392 messages, 0 notifications, 0 bytes in queue
Connections established 2, dropped 1
Last reset by peer 02:05:25, due to peer deconfigured
Last reset by us never, due to process restart

Neighbor capabilities:
Dynamic capability: advertised (mp, refresh, gr)
Dynamic capability (old): advertised
Route refresh capability (new): advertised received
Route refresh capability (old): advertised received
4-Byte AS capability: advertised received
Address family IPv4 Unicast: advertised received
Graceful Restart capability: advertised received
…

Enable BGP load sharing with Multipath-relax

If your network is like most large enterprise’s, chances are you have Internet connection from dual ISPs to your data center.

It used to be a popular tactic to have a primary provider, and a backup provider which you pay lesser bandwidth for. In recent years, usage based circuits has given way to flat rate, often larger bandwidth circuits from both ISPs. After all, users demand more bandwidth, even in failure scenarios.

Why would you pay for expensive Internet access and only letting it sitting idle most of the time? Therefore, load sharing on redundant ISP connections has become highly desirable for data centers.

By default, BGP only installs one route into the routing table. To accomplish load sharing, BGP multipath needs to be enabled. To further complicate the matter, BGP requires all paths to have the following equal characteristics in order to become equal cost path:
• Weight
• Local preference
• AS-PATH length
• Origin
• MED
• AS-PATH

Since neighboring AS or AS-PATH must be equal, that means BGP can only load balance to a single AS or single ISP. Since different ISP will have different AS and AS-PATH, BGP will only pick one best route.

So how does Cisco support load balancing to two ISPs? There is a hidden command bgp bestpath as-path multipath-relax

I don’t know why it is hidden command, since it appears to be a very useful feature, and it is supported. But do use with caution, as the number of routing entries will increase, often overwhelming the limited memory you have on the Internet router.

In this year’s Networker, an alternative method was also shown, essentially separating out the Internet routing table by even and odd digits, for outbound traffic.