Manual:Layer2 misconfiguration

Applies to RouterOS: v6.41 +

Introduction

There are certain configuration that are known to have major flaws by design and should be avoided by all means possible. Misconfigured Layer2 can sometimes cause hard to detect network errors, random performance drops, certain segments of a network to be unreachable, certain networking services to be malfunctioning or a complete network failure. This page will contain some common and not so very common configurations that will cause issues in your network.

Bridges on a single switch chip

Consider the following scenario, you have a device with a built-in switch chip and you need to isolate certain ports from each other, for this reason you have created multiple bridges and enabled hardware offloading on them. Since each bridge is located on a different Layer2 domain, then Layer2 frames will not be forwarded between these bridges, as a result ports in each bridge are isolated from other ports in a different bridge.

Configuration

/interface bridge
add name=bridge1
add name=bridge2
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge2 interface=ether3
add bridge=bridge2 interface=ether4

Problem

After a simple performance test you might notice that one bridge is capable of forwarding traffic at wire-speed while the second, third, ... bridge is not able to forward as much data as the first bridge. Another symptom might be that there exists a huge latency for packets that need to be routed. After a quick inspection you might notice that the CPU is always at full load, this is because hardware offloading is not available on all bridges, but is available only on one bridge. By checking the hardware offloading status you will notice that only one bridge has it active:

[admin@MikroTik] > /interface bridge port print
Flags: X - disabled, I - inactive, D - dynamic, H - hw-offload 
 #     INTERFACE                                 BRIDGE                                 HW
 0   H ether1                                    bridge1                                yes
 1   H ether2                                    bridge1                                yes
 2     ether3                                    bridge2                                yes
 3     ether4                                    bridge2                                yes

The reason why only one bridge has the hardware offloading flag available is because the device does not support port isolation. If port isolation is not supported, then only one bridge will be able to offload the traffic to the switch chip.

Solution

Not all device devices support port isolation, currently only CRS1xx/CRS2xx series devices support it, other devices will have to use the CPU to forward the packets on other bridges. This is usually a hardware limitation and a different device might be required. Bridge split horizon parameter is a software feature that disables hardware offloading and when using bridge filter rules you need to enable forward all packets to the CPU, which requires the hardware offloading to be disabled. You can control which bridge will be hardware offloaded with the hw=yes flag and by setting hw=no to other bridges, for example:

/interface bridge port set [find where bridge=bridge1] hw=no
/interface bridge port set [find where bridge=bridge2] hw=yes

Sometimes it is possible to restructure a network topology to use VLANs, which is the proper way to isolate Layer2 networks.

VLAN interface on a slave interface

Consider the following scenario, you have created a bridge and you want a DHCP Server to give out IP addresses only to a certain tagged VLAN traffic, for this reason you have created a VLAN interface, specified a VLAN ID and created a DHCP Server on it, but for some reasons it is not working properly.

Configuration

/interface bridge
add name=bridge
/interface bridge port
add interface=ether1 bridge=bridge
add interface=ether2 bridge=bridge
/interface vlan
add name=VLAN99 interface=ether1 vlan-id=99
/ip pool
add name=VLAN99_POOL range=192.168.99.100-192.168.99.200
/ip address add address=192.168.99.1/24 interface=VLAN99
/ip dhcp-server
add interface=VLAN99 address-pool=VLAN99_POOL disabled=no
/ip dhcp-server network
add address=192.168.99.0/24 gateway=192.168.99.1 dns-server=192.168.99.1

Problem

When you add an interface to a bridge, the bridge becomes the master interface and all bridge ports become slave ports, this means that all traffic that is received on a bridge port is captured by the bridge interface and all traffic is forwarded to the CPU using the bridge interface instead of the physical interface. As a result VLAN interface that is created on a slave interface will never capture any traffic at all since it is immediately forwarded to the master interface before any packet processing is being done.

Solution

Change the interface on which the VLAN interface will be listening for traffic, change it to the master interface:

/interface vlan set VLAN99 interface=bridge

VLAN on a bridge in a bridge

Consider the following scenario, you have a set of interfaces (don't have to be physical interfaces) and you want all of them to be in the same Layer2 segment, the solution is to add them to a single bridge, but you require that traffic from one port tags all traffic into a certain VLAN. This can be done by creating a VLAN interface on top of the bridge interface and by creating a separate bridge that contains this newly created VLAN interface and the interface, which will send out tagged traffic. Network diagram can be found below:

VLAN on bridge in bridge topology

Configuration

/interface bridge
add name=bridge1
add name=bridge2
/interface vlan
add interface=bridge1 name=VLAN vlan-id=99
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge2 interface=VLAN
add bridge=bridge2 interface=ether3

Problem

Packets coming from ether3 will be sent out tagged and traffic won't be flooded through ether1 and ether2, but if another port is added to bridge2, then traffic will be flooded. Similar issue arises when traffic needs to be sent from ether1 to ether3 since MAC learning is only possible between bridge ports and not interfaces that are created on top of the bridge interface. As a result unicast traffic will be flooded to ether2 and ether3. If a device behind ether3 is using (R)STP, then ether1 and ether2 will send out tagged BPDUs. Because of the broken MAC learning functionality and broken (R)STP this setup and configuration must be avoided.

Solution

Use bridge VLAN filtering. The proper way to tag traffic is to assign a VLAN ID whenever traffic enters a bridge, this behaviour can easily be achieved by specifying PVID value for a bridge port and specifying which ports are tagged (trunk) ports and which are untagged (access) ports. Below is an example how such setup should have been configured:

/interface bridge
add name=bridge vlan-filtering=yes
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3 pvid=99
/interface bridge vlan
add bridge=bridge tagged=ether1 untagged=ether3 vlan-ids=99

Warning: By enabling vlan-filtering you will be filtering out traffic destined to the CPU, before enabling VLAN filtering you should make sure that you set up a Management port

VLAN in bridge with a physical interface

Very similar case to VLAN on a bridge in a bridge, there are multiple possible scenarios where this could could have been used, most popular use case is when you want to send out tagged traffic through a physical interface, in such a setup you want traffic from one interface to receive only certain tagged traffic and send out this tagged traffic as tagged through a physical interface (simplified trunk/access port setup) by just using VLAN interfaces and a bridge.

Configuration

/interface vlan
add interface=ether1 name=VLAN99 vlan-id=99
/interface bridge
add name=bridge
/interface bridge port
add interface=ether2 bridge=bridge
add interface=VLAN99 bridge=bridge

Problem

This setup and configuration will work on most cases, but it violates the IEEE 802.1W standard when (R)STP is used. If this is the only device in your Layer2 domain, then this should not cause problems, but problems can arise when there are other vendor switches. The reason for this is that (R)STP on a bridge interface is enabled by default and BPDUs coming from ether1 will be sent out tagged since everything sent into ether1 will be sent out through ether2 as tagged traffic, not all switches can understand tagged BPDUs. Precautions should be made with this configuration in a more complex network where there are multiple network topologies for certain (group of) VLANs, this is relevant to MSTP and PVSTP(+) with mixed vendor devices. In a ring-like topology with multiple network topologies for certain VLANs, one port from the switch will be blocked, but in MSTP and PVSTP(+) a path can be opened for a certain VLAN, in such a situation it is possible that devices that don't support PVSTP(+) will untag the BPDUs and forward the BPDU, as a result the switch will receive its own packet, trigger a loop detection and block a port, this can happen to other protocols as well, but (R)STP is the most common case.

Solution

To avoid compatibility issues you should use bridge VLAN filtering. Below you can find an example how the same traffic tagging effect can be achieved with a bridge VLAN filtering configuration:

/interface bridge
add name=bridge vlan-filtering=yes
/interface bridge port
add bridge=bridge interface=ether1 pvid=99
add bridge=bridge interface=ether2
/interface bridge vlan
add bridge=bridge tagged=ether2 untagged=ether1 vlan-ids=99

Warning: By enabling vlan-filtering you will be filtering out traffic destined to the CPU, before enabling VLAN filtering you should make sure that you set up a Management port

Bridged VLAN on physical interfaces

Very similar case to VLAN on a bridge in a bridge, consider the following scenario, you have a couple of switches in your network and you are using VLANs to isolate certain Layer2 domains and connect these switches are connected to a router that assigns addresses and routes the traffic to the world. For redundancy you connect switches all switches directly to the router, but to be able to setup DHCP Server you decide that you can create a VLAN interface for each VLAN on each physical interface that is connected to a switch and add these VLAN interfaces in a bridge. Network diagram can be found bellow:

Bridges VLANs topology

Configuration

Only the router part is relevant to this case, switch configuration doesn't really matter as long as ports are switched. Router configuration can be found bellow:

/interface bridge
add name=bridge10
add name=bridge20
/interface vlan
add interface=ether1 name=ether1_v10 vlan-id=10
add interface=ether1 name=ether1_v20 vlan-id=20
add interface=ether2 name=ether2_v10 vlan-id=10
add interface=ether2 name=ether2_v20 vlan-id=20
/interface bridge port
add bridge=bridge10 interface=ether1_v10
add bridge=bridge10 interface=ether2_v10
add bridge=bridge20 interface=ether1_v20
add bridge=bridge20 interface=ether2_v20

Problem

You might notice that the network is having some weird delays or even the network is unresponsive, you might notice that there is a loop detected (packet received with own MAC address) and some traffic is being generated out of nowhere. The problem occurs because a broadcast packet that is coming from either one of the VLAN interface created on the Router will be sent out the physical interface, packet will be forwarded through the physical interface, through a switch and will be received back on a different physical interface, in this case broadcast packets sent out ether1_v10 will be received on ether2, packet will be captured by ether2_v10, which is bridged with ether1_v10 and will get forwarded again the same path (loop). (R)STP might not always detect this loop since (R)STP is not aware of any VLANs, a loop does not exist with untagged traffic, but exists with tagged traffic. In this scenario it is quite obvious to spot the loop, but in more complex setups it is not always easy to detect the network design flaw. Sometimes this network design flaw might get unnoticed for a very long time if your network does not use broadcast traffic, usually Nieghbor Discovery Protocol is broadcasting packets from the VLAN interface. Sometimes it is useful to capture the packet that triggered a loop detection, this can be done in multiple ways:

/tool sniffer
set filter-mac-address=CC:2D:E0:15:2D:E1/FF:FF:FF:FF:FF:FF \
filter-interface=ether1 filter-direction=rx file-name=loop_packet.pcap

Or a simplified way:

/interface bridge filter
add action=log chain=forward src-mac-address=CC:2D:E0:15:2D:E1/FF:FF:FF:FF:FF:FF
add action=log chain=input src-mac-address=CC:2D:E0:15:2D:E1/FF:FF:FF:FF:FF:FF