Network based TCP MSS adjustment

Maximum Segment Size (MSS) is set by end points during initial TCP handshake. In special circumstances, router can step in to alter MSS.

Let’s look at such a scenario when two hosts communicate through an SSL tunnel. End points sees a path MTU of 1500 byte, and set MSS to be 1500. However, SSL adds extra overhead. Therefore, when a 1500 byte packet arrives at tunnel end points, it becomes a little larger. Furthermore, SSL often sets DF (Do not Fragment). Since the packet is now larger than 1500 byte, with DF set, the router drops it. This results in communication failure between hosts (while ping and traceroute appears to be working). An extended ping with varying packet size will verify this exact behavior.

Jumbo MTU on Nexus 1000v – Why and How

I posted earlier about the use of “system mtu” on Nexus 1000v.  The internetworking of virtual switch in a physical topology can still be confusing, and the existence of likely display bugs probably didn’t help with the situation.

So here is my understanding of why and how, hopefully in more simple terms than any documentation I have seen.

First, why enable jumbo MTU on Nexus 1000v? Jumbo support is desirable for applications such as storage, for example your ESX host in a NAS environment. Enabling jumbo on Nexus 1000v is part of the “end-to-end” jumbo configuration, starting from the host. In a typical topology, you will need to configure Nexus 7000, Nexus 5000, and Nexus 1000v since it is the virtual switch that partly resides on the host.  You can refer to more configuration details here.

A diagram is helpful to illustrate the relationship between Nexus 5000 and Nexus 1000v, particularly the confusing port channel relationship.  Note on 5000 VPCs are typically configured (Po101-103). On Nexus 1000v, only system uplink profile is predefined. Port channel are created only when hosts/VEMs are added. Adding host on vCenter triggers the creation of matching port channels (Po5-7) in VSM.

Next, how to enable jumbo MTU on Nexus 1000v?

1.       Set system jumbomtu (set by default)

            You really don’t need to do anything here, because by default it has already been set to maximum

\N1kv# sh run all | inc jumbo

system jumbomtu 9000

2.       Set system mtu

This is really the only configuration required on Nexus1000v. By setting system mtu, we can preserve the physical NIC setting through ESX reboot

port-profile type ethernet systemuplink_portchannel

  system mtu 9000

Lastly, how to verify that jumbo is working?

This is the most confusing part. Depending on your port type (LCAP etc), you may get different display results. In my experience, both “show int port-channel” and “show interface Ethernet” displays MTU 1500 (which is incorrect). This display error is similar to one on Nexus 5000 (CSCsl21529). But on Nexus 1000v there is no jumbo packet count, thus making verification even more difficult.

Nexus-1000v-VSM# sh int po1

port-channel1 is up

  Hardware: Port-Channel, address: 0050.5655.cbe0 (bia 0050.5655.cbe0)

  MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

Nexus-1000v-VSM # sh int e3/5

Ethernet3/5 is up

  Hardware: Ethernet, address: 0050.5655.dbc8 (bia 0050.5655.dbc8)

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 0/255, txload 0/255, rxload 0/255

To alternative to verify that jumbo is really working on Nexus 1000v, is simply to check upstream switch. For example, on Nexus 5000, check either the corresponding port channel or physical interface. Since Nexus 5000 is receiving (RX) jumbo packets from the host, it is a clear indication of the function of Nexus 1000v uplink profile.

Nexus-5010-a# sh int po101

port-channel101 is up

 vPC Status: Up, vPC number: 101

  Hardware: Port-Channel, address: 0005.9b76.308f (bia 0005.9b76.308f)

  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 10 Gb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  EtherType is 0x8100

  Members in this channel: Eth1/8

  Last clearing of "show interface" counters never

  30 seconds input rate 4344 bits/sec, 543 bytes/sec, 1 packets/sec

  30 seconds output rate 44352 bits/sec, 5544 bytes/sec, 48 packets/sec

  Load-Interval #2: 5 minute (300 seconds)

    input rate 4.50 Kbps, 1 pps; output rate 47.82 Kbps, 47 pps

  RX

    11483591 unicast packets  205769 multicast packets  30036 broadcast packets

    11719396 input packets  2923241828 bytes

    274665 jumbo packets  0 storm suppression packets

    0 runts  0 giants  0 CRC  0 no buffer

    0 input error  0 short frame  0 overrun   0 underrun  0 ignored

    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop

    0 input with dribble  0 input discard

    0 Rx pause

  TX

    134696185 unicast packets  163700959 multicast packets  12680857 broadcast p

ackets

    311078001 output packets  240793383992 bytes

    47357469 jumbo packets

    0 output errors  0 collision  0 deferred  0 late collision

    0 lost carrier  0 no carrier  0 babble

    0 Tx pause

  2 interface resets


Nexus 1000v is only a component in the larger picture. Scott Lowe provides a great example here to show how jumbo support is enabled on ESX server and NAS. In particular, testing jumbo from ESX using vmkping command:

vmkping -s 9000

Nexus1000v – when to use “system mtu”

I can’t be the first one confused about jumbo, mtu, and system mtu on Nexus 1000v. After reading some excellent posts, all signs were indicating that “system mtu” was designed to solve the “chicken and egg” problem of running VSM on IP storage.

Like "system vlan", “system mtu” applies to system uplink profile only. So if VSM is not even running under VEM (it runs on vSwitch), there is no need to set “system mtu”, right?

Well, not quite. It turns out system mtu is still needed to preserve the connection to VEM. Assuming jumbo is used (for storage as an example), reboot of ESX will revert the physical NIC to default MTU (1500), which results in mismatched MTU between physical NIC and virtual NIC, and loss of connectivity. “system mtu” preserves the setting on physical NIC, and thus preventing VEM from disappearing.

To further clarify, here is an example of configuring jumbo of 9000 on Nexus 1000v
1. “system jumbomtu 9000” (global)
2. “system mtu 9000” (uplink port profile)

That is all. Note once set, “system mtu” overwrites “mtu”, therefore there is no need to set interface mtu explicitly.

A couple of things potentially confusing:
-The show commands on Nexus 1000v is not exactly accurate for MTU, fix is supposed to be coming
-There is an error in Cisco Nexus command reference, which states “The value that is configured for system mtu command must be less then value configured in the system jumbomtu command”. It should be “less or equal to”; There are no reason to set system mtu to 8998 unless hardware dictates so.

I hope that clear up some confusions. If you notice any behavior inconsistent with this understanding, please kindly let me know.

enabling jumbo MTU on Nexus

In our data center, enabling jumbo MTU has proven to result in significant throughput improvements. It should have been straightforward, but unfortunately not with Nexus. This note is meant to fill in a gap where existing documentation may be lacking.

In a typical Nexus 7000/5000/1000v architecture, use these steps to enable jumbo MTU end to end, and then verify.

1. System jumbomtu
It defines the maximum MTU size for the switch, which must be configured on ALL devices.
Note the max value depend on hardware, for example it’s 9000 on Nexus 1000v, and 9216 elsewhere.

2. On Nexus 5000, must configure with system QoS policy
Note it can not be configured on the interface! This is quite different from any other device I am aware of. To make it even more confusing, the command varies between releases.

For example, with 4.0.1:
policy-map jumbo
 class class-default
  mtu 9216
system qos 
 service-policy jumbo

With 4.1, 4.2 it becomes:
policy-map type network-qos jumbo
 class type network-qos class-default
  mtu 9216
system qos
 service-policy type network-qos jumbo

3. On Nexus 1000v, configure on port channel
You will see some documentation showing configuration on physical interface, and disappointed when you actually try to do it. No, it won’t take the command. You must configure on the port channel! And then, magic, it will appear on the physical interface.

More detailed explanation on Nexus1000v is here.

4. On Nexus 7000, configure on the interface
Don’t forget “system jumbomtu” on ALL devices

Verification
So how to verify? Assuming you have jumbo traffic, then you will see the difference before and after with “show interface … counter detail”

Nexus5000# sh int e1/1 count detail
Ethernet1/1
Rx Packets: 95396987709
Rx Unicast Packets: 95396962128
Rx Multicast Packets: 21481
Rx Broadcast Packets: 4100
Rx Jumbo Packets: 89336029550
Rx Bytes: 136394308168545
Rx Packets from 0 to 64 bytes: 6028626455
Rx Packets from 65 to 127 bytes: 611554
Rx Packets from 128 to 255 bytes: 221439
Rx Packets from 256 to 511 bytes: 188679
Rx Packets from 512 to 1023 bytes: 579970
Rx Packets from 1024 to 1518 bytes: 30730062
Rx Trunk Packets: 95396966228
Tx Packets: 2613558
Tx Unicast Packets: 257688
Tx Multicast Packets: 2355702
Tx Broadcast Packets: 168
Tx Bytes: 202281759
Tx Packets from 0 to 64 bytes: 128120
Tx Packets from 65 to 127 bytes: 2472242
Tx Packets from 128 to 255 bytes: 8732
Tx Packets from 256 to 511 bytes: 4391
Tx Packets from 512 to 1023 bytes: 63
Tx Packets from 1024 to 1518 bytes: 10
Tx Trunk Packets: 2307778

Known Issues

Last but not the least, be aware of some known issues (depending on version):

CSCsl21529, Symptom: An incorrect MTU value is displayed in the show interface command output. The Cisco Nexus 5000 Series switch only supports class-based MTU. Per-interface level MTU configuration is not supported. The switch supports jumbo frames by default. However, the show interface command output currently displays an incorrect MTU value of 1500 bytes.
Also, VEM may not retain jumbo MTU setting after reboot. Just something to watch and monitor.