Understand Nexus 7000 Module Forwarding Capacity

Nexus 7000’s line card forwarding (FIB) capacity may be a limiting factor. This is particularly true in a virtualized data center, where multiple VDCs and VRFs are used. This note explains the importance of understanding and managing module capacity.

Nexus 7000 has forwarding engine on line cards. The larger the size of the routing table, the bigger the FIB required for hardware forwarding. When designing with multiple VDCs and VRFs, multiple routing processes and routing tables consume more resources on the module.

Below shows a module using 64k of 65k allocated capacity:

#show hardware capacity forwarding

…

Module 1 usage:

Route Type              Used      %Used      Free      %Free      Total    

                     (Log/Phys)            (Log/Phys)           (Log/Phys)   

-------------------------------------------------------------------------

IPv4 Unicast:       64161/64161     97     1375/1375       2    65536/65536 

L2VPN Peer:             0/0          0        0/0          0        0/0     

MPLS:                   0/0          0        0/0          0        0/0     

IPv4 Multicast:         4/8          0    14332/28664     99    14336/28672 

L2VPN IPv4 Mcast:       0/0          0        0/0          0        0/0     

IPv6 Unicast:         195/390        1    14141/28282     98    14336/28672 

L2VPN IPv6 Mcast:       0/0          0        0/0          0        0/0     

IPv6 Multicast:         5/20         0     2043/8172      99     2048/8192  

Not all modules are equal

Below shows three types of cards on a N7k. XL capable cards can support up to 1M addresses, with license.

Module	FIB capacity	Data Sheet
N7K-M132XP-12	128k	link
N7K-M108X2-12L	128k – 1M (XL mode)	link
N7K-M148GT-11	128k	link

Default, Maximum and Dynamic Allocation

Module allocates its FIB TCAM to IPv4 and IPv6 (both unicast and multicast) automatically. IPv4 unicast is discussed here. For non-XL module with 128k total, the default allocation is 56k. Since Cisco currently only has a limited number of XL modules, chances are, you will be running in “mixed” mode, thus 112k is the maximum allocated to IPv4 unicast entries

Allocation is not configurable. Dynamic allocation is done by module software, and provided incrementally “as needed” in 8k increments.

VDC and dynamic allocation

What happens when multiple VDCs are created? It depends on if a module is shared by more than one VDC. If a module is dedicated to a single VDC, then system can allocate the maximum physical resources to that VDC.

If there are multiple VDCs using the same module (by assigning some interfaces to one VDC and others to another VDC from the same line card), then the system must allocate FIB TCAM to both VDCs.

Again, allocation is not configurable, despite of what various documentation may suggest.

Conclusion

• Monitor resource utilization with “show hardware capacity”, keep track of total capacity as well as FIB allocated to VDC
• reduce unnecessary large routing table especially in a multi-VRF design
• Plan FIB capacity in multi-VDC design, perhaps by dedicating modules to a VDC
• Plan FIB capacity chassis wide, i.e., desirable to build all XL chassis, or non-XL chassis if no requirements for large routing table.