Manual:Maximum Transmission Unit on RouterBoards

Background

It is sole responsibility of administrator to configure MTUs such that intended services and applications can be successfully implemented in network. In other words - administrator must make sure that MTUs are configured in a way that packet sizes does not exceed the capabilities of network equipment.

There are several different kins of MTUs out there:

• Full frame MTU - 1518 bytes - for standard (IEEE) Ethernet frames, this include Layer-2 header (14 bytes) and FCS (Frame Check Sum) 4bytes - this is MTU of the frames that travel thought the wires
• Physical/Layer-2 MTU - 1514 bytes - for standard (IEEE) Ethernet frames, this include Layer-2 header (14 bytes), but without FCS - This is MTU of the frame that software work with
• IP/Layer-3 MTU - 1500 bytes - for standard (IEEE) Layer-3 (IP) packets, this excludes both Layer-2 header and FCS - This is value that can be configured in MikroTik RouterOS

So basically all those values are the exactly the same thing only from different point of view (from physical, Layer-2 and Layer-3 point of view)

Originally MTU was introduced because of the high error rates and low speed of communications. Fragmentation of the data stream gives ability to correct corruption errors only by resending corrupted fragment, not the whole stream. Also on low speed connections such as modems it can take too much time to send a big fragment, so in this case communication is possible only with smaller fragments.

But in present days we have much lower error rates and higher speed of communication, this opens a possibility to increase the value of MTU. By increasing value of MTU we will result in less protocol overhead and reduce CPU utilization mostly due to interrupt reduction.

This way some non-standard frames started to emerge:

• Giant or Jumbo frames - frames that are bigger than standard (IEEE) Ethernet MTU
• Baby Giant or Baby Jumbo frames - frames that are just slightly bigger that standard (IEEE) Ethernet MTU

It is common now for Ethernet interfaces to support physical MTU above standard, but this can not be taken for granted. Abilities of other network equipment must be taken into account as well - for example, if 2 routers with Ethernet interfaces supporting physical MTU 1526 are connected through Ethernet switch, in order to successfully implement some application that will produce this big Ethernet frames, switch must also support forwarding such frames.

MTU on RouterBoards

This table shows how many additional bytes is supported by Mikrotik RouterBoards outside of standard (IEEE) Ethernet MTU:

RouterBoard	ether1	ether2	ether3	ether4	ether5	ether6	ether7	ether8	ether9
RB493, RB493AH	+22	+18	+18	+18	+18	+18	+18	+18	+18
RB450	+22	+18	+18	+18	+18
RB433, RB433AH	+22	+18	+18
RB411, RB411A, RB411AH	+22
CrossRoads	+100
RB1000	+8000	+8000	+8000	+8000
RB600, RB600A	+8000	+8000	+7500

So if table indicate "+22" - interface has Layer 2 MTU 14+22+1500=1534 bytes, and can handle 22 bytes (over standard Ethernet MTU of 1500) that can be utilized for VLAN tags, MPLS tags, Increase of layer-3 MTU

All wireless interfaces in RouterOS (including Nstreme2) Layer-2 1600 byte Physical MTU, so you have spare +86 bytes for VLAN tags, MPLS tags, Increase of layer-3 MTU

MTU in RouterOS

IP/Layer-3 MTU

Configured as interface mtu setting (/interface <type> <name> set mtu=X). Specifies how big IP packets router is allowed to send out the particular interface.

If router receives IP packet of size 1500, but MTU for outgoing interface is set to 1400, router will either fragment the packet (if "Don't Fragment" bit is not set in IP header) or drop the packet and send ICMP "Need Fragmentation" error back to originator (this is essential for Path MTU Discovery to work).

Sometimes it can be bad idea to change IP MTU from its default 1500 bytes on router interfaces if complete path end-to-end is not in administrators control. Although IP fragmentation and end-to-end Path MTU Discovery is intended to handle this situation, if ICMP Need Fragmentation errors are filtered somewhere along the path, Path MTU Discovery will not work.

There are several features in MikroTik RouterOS that is can benefit from possibility to exceed standard MTU

MPLS MTU

Configured in "/mpls interface" menu, specifies how big packets, including MPLS labels, router is allowed to send out the particular interface (default is 1508).

Note that when actually sending packet out the interface, link layer header is going to get attached, so it is only safe to configure MPLS MTU up to the value of physical MTU minus size of any link layer header(s) that is going to get applied (e.g. 14 in regular ethernet case, or 18 in case of one VLAN tag).

MPLS MTU affects packets depending on what action MPLS router is performing. It is strongly recommended that MPLS MTU is configured to the same value on all routers forming MPLS cloud because of effects MPLS MTU has on MPLS switched packets. This requirement means that all interfaces participating in MPLS cloud must be configured to the smallest MPLS MTU values among participating interfaces, therefore care must be taken to properly select hardware to be used.

MPLS Switching

If packet with labels included is bigger than MPLS MTU, MPLS tries to guess protocol that is carried inside MPLS frame.

If this is IP packet, MPLS produces ICMP Need Fragment error. This behaviour mimics IP protocol behaviour. Note that this ICMP error is not routed back to originator of packet but is switched towards end of LSP, so that egress router can route it back.

If this is not IP packet, MPLS simply drops it, because it does not know how to interpret the contents of packet. This feature is very important in situations where MPLS applications such as VPLS are used (where frames that are MPLS tagged are not IP packets, but e.g. encapsulated ethernet frames as in case of VPLS) - if somewhere along the LSP MPLS MTU will be less than packet size prepared by ingress router, frames will simply get dropped.

IP ingress

When router first introduces label (or labels) on IP packet, and resulting packet size including MPLS labels exceeds MPLS MTU, router behaves as if interface MTU was exceeded - either fragments packet in fragments that does not exceed MPLS MTU when labels are attached (if IP Dont Fragment is not set), or generates ICMP Need Fragmentation error that is sent back to originator.

VPLS ingress

When router encapsulates ethernet frame for forwarding over VPLS pseudowire, it checks if packet size with VPLS Control Word (4 bytes) and any necessary labels (usually 2 labels - 8 bytes), exceeds MPLS MTU of outgoing interface. If it does, VPLS fragments packet so that it honours MPLS MTU of outgoing interface. Packet is defragmented at egress point of VPLS pseudowire.

MTU examples for common technologies

Examples on how different technologies and their combinations relate to MTUs are given below. All examples are given assuming common standard 1500 byte IP packet size and that outgoing interface is ethernet. The total size of packet after all encapsulations must not exceed the physical MTU of outgoing interface.

Simple Routing

The image shows the packet MTU size for simple routing,

VLAN

Each VLAN tag is 4 bytes long.

Packet example, result of encapsulating in Q-in-Q:

<ethernet header: 14><outer VLAN tag: 4><inner VLAN tag: 4><IP packet: 1500> total: 1522 bytes

This means that physical MTU of outgoing interface must be at least 1522 bytes. If physical MTU of outgoing ethernet interface is 1518 bytes, interface is not capable of passing such frame (with 1500 byte data payload), it can only pass frame with single VLAN tag.

In order to get it to work - you will have to go into interface menu and reduce Layer-3/IP MTU to 1496 bytes.

Simple MPLS

When MPLS is used as plain replacement for IP routing, only one label is attached to every packet, therefore packet size increases by 4 bytes. In order to be able to forward standard size (1500 bytes) IP packet without fragmentation, MPLS MTU must be set to at least 1504.

Packet example:

<ethernet header: 14><MPLS label: 4><IP packet: 1500>total: 1518 bytes

Simple MPLS over VLAN

If simple MPLS packets (with 1 label) are forwarded across ethernet additionally encapsulated in Vlan:

<ethernet header: 14><VLAN tag: 4><MPLS label: 4><IP packet: 1500> total: 1522 bytes

Note that MPLS MTU of VLAN interface matters in this case and it should be at least 1504.

MPLS IP VPNs

MPLS IP VPNs add 2 MPLS labels to IP packets - inner for VPNv4 route identification and outer for transporting to remote VPNv4 router:

<ethernet header: 14><transport MPLS label: 4><VPNv4 MPLS label: 4><IP packet: 1500> total: 1522 bytes

VPLS

VPLS adds 4 bytes VPLS Control word and 2 MPLS labels - inner for VPLS identification and outer for transporting to remote VPLS endpoint:

<ethernet header: 14><transport MPLS label: 4><VPLS label: 4><VPLS CW: 4> <<ethernet header:14><IP payload: 1500>> total: 1540 bytes

In order to avoid VPLS encapsulated frame fragmentation, MPLS MTU on outgoing interface (the one over which MPLS frame will be sent) must be set to at least 1540 - 14 = 1526.

VLAN tagged frames over VPLS

When forwarding VLAN tagged frames over VPLS, size of packets as encapsulated by VPLS increase by space taken by VLAN tags, e.g. when one VLAN tag is attached:

<ethernet header: 14><transport MPLS label: 4><VPLS label: 4><VPLS CW: 4> <<ethernet header:14><VLAN tag: 4><IP payload: 1500>> total: 1544 bytes

In order to avoid VPLS encapsulated frame fragmentation, MPLS MTU on outgoing interface (the one over which MPLS frame will be sent) must be set to at least 1544 - 14 = 1530.