Manual:CRS3xx VLANs with Bonds: Difference between revisions

From MikroTik Wiki
Jump to navigation Jump to search
← Older edit
No edit summary
 
(27 intermediate revisions by one other user not shown)
Line 1: Line 1:
{{Versions| v6.32 +}}
{{Versions| v6.42 +}}


{{Warning|This manual is moved to https://help.mikrotik.com/docs/display/ROS/CRS3xx%2C+CRS5xx%2C+CCR2116%2C+CCR2216+VLANs+with+Bonds}}
__TOC__
__TOC__




===Summary===
=Summary=


This page will show how to configure multiple switches to use bonding interfaces and port based VLANs, it will also show a working example with a DHCP-Server, interVLAN routing, management IP and invalid VLAN filtering  configuration.
This page will show how to configure multiple switches to use bonding interfaces and port based VLANs, it will also show a working example with a DHCP-Server, interVLAN routing, management IP and invalid VLAN filtering  configuration.


{{Warning | This article applies to CRS3xx series devices and not CRS1xx/CRS2xx.}}
{{Warning | This article applies to CRS3xx series devices and not CRS1xx/CRS2xx. For a similar setup for CRS1xx/CRS2xx series switches you can check [[Manual:CRS1xx/2xx_VLANs_with_Trunks | CRS1xx/2xx VLANs with Trunks]] guide.}}


[[File:Crs1xx crs2xx trunk vlan.png|700px|thumb|center|alt=Alt text|CRS1xx/CRS2xx port trunking with port based VLANs]]
[[File:Crs3xx vlans with bonds.png|700px|thumb|center|alt=Alt text|CRS3xx bonds and port based VLANs]]


{{Note | For this network topology we will be using two CRS326-24G-2S+, one CRS317-1G-16S+ and one CCR1072-1G-8S+, but same principles can be applied to any CRS3xx series devices and a router.}}
{{Note | For this network topology we will be using two CRS326-24G-2S+, one CRS317-1G-16S+ and one CCR1072-1G-8S+, but same principles can be applied to any CRS3xx series devices and a router.}}
Line 16: Line 17:
In this setup SwitchA and SwitchC will tag all traffic from ports ether1-ether8 to VLAN ID 10, ether9-ether16 to VLAN ID 20, ether17-ether24 to VLAN ID 30. Management will only be possible if user is connecting with tagged traffic with VLAN ID 99 from ether1 on SwitchA or SwitchB, connecting to all devices will also be possible from the router using tagged traffic with VLAN ID 99. SFP+ ports in this setup are going to be used as VLAN trunk ports while being in a bond to create a LAG interface.
In this setup SwitchA and SwitchC will tag all traffic from ports ether1-ether8 to VLAN ID 10, ether9-ether16 to VLAN ID 20, ether17-ether24 to VLAN ID 30. Management will only be possible if user is connecting with tagged traffic with VLAN ID 99 from ether1 on SwitchA or SwitchB, connecting to all devices will also be possible from the router using tagged traffic with VLAN ID 99. SFP+ ports in this setup are going to be used as VLAN trunk ports while being in a bond to create a LAG interface.


===Port switching===
=Port switching=


All switches in this setup require that all used ports are switched together. Use these commands on '''SwitchA''', '''SwitchB''', '''SwitchC''':
All switches in this setup require that all used ports are switched together (except for ports that are going to be part of a bonding interface). Use these commands on '''SwitchA''' and '''SwitchC''':
<pre>
/interface ethernet
set [ find default-name=ether2 ] master-port=ether1
set [ find default-name=ether3 ] master-port=ether1
set [ find default-name=ether4 ] master-port=ether1
set [ find default-name=ether5 ] master-port=ether1
set [ find default-name=ether6 ] master-port=ether1
set [ find default-name=ether7 ] master-port=ether1
set [ find default-name=ether8 ] master-port=ether1
set [ find default-name=ether9 ] master-port=ether1
set [ find default-name=ether10 ] master-port=ether1
set [ find default-name=ether11 ] master-port=ether1
set [ find default-name=ether12 ] master-port=ether1
set [ find default-name=ether13 ] master-port=ether1
set [ find default-name=ether14 ] master-port=ether1
set [ find default-name=ether15 ] master-port=ether1
set [ find default-name=ether16 ] master-port=ether1
set [ find default-name=ether17 ] master-port=ether1
set [ find default-name=ether18 ] master-port=ether1
set [ find default-name=ether19 ] master-port=ether1
set [ find default-name=ether20 ] master-port=ether1
set [ find default-name=ether21 ] master-port=ether1
set [ find default-name=ether22 ] master-port=ether1
set [ find default-name=ether23 ] master-port=ether1
set [ find default-name=ether24 ] master-port=ether1
</pre>
 
Disable SFP interface for security reasons (in case it is not being used):
<pre>
/interface ethernet set [find where name~"sfp"] disabled=yes
</pre>
 
* In case using RouterOS 6.41+, a bridge must be created instead with disabled RSTP and IGMP Snooping and no VLAN filtering:
<pre>
<pre>
/interface bridge
/interface bridge
add name=bridge protocol-mode=none igmp-snooping=no vlan-filtering=no
add name=bridge vlan-filtering=no
/interface bridge port
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=ether1 pvid=10 hw=yes
add bridge=bridge interface=ether2
add bridge=bridge interface=ether2 pvid=10 hw=yes
add bridge=bridge interface=ether3
add bridge=bridge interface=ether3 pvid=10 hw=yes
add bridge=bridge interface=ether4
add bridge=bridge interface=ether4 pvid=10 hw=yes
add bridge=bridge interface=ether5
add bridge=bridge interface=ether5 pvid=10 hw=yes
add bridge=bridge interface=ether6
add bridge=bridge interface=ether6 pvid=10 hw=yes
add bridge=bridge interface=ether7
add bridge=bridge interface=ether7 pvid=10 hw=yes
add bridge=bridge interface=ether8
add bridge=bridge interface=ether8 pvid=10 hw=yes
add bridge=bridge interface=ether9
add bridge=bridge interface=ether9 pvid=20 hw=yes
add bridge=bridge interface=ether10
add bridge=bridge interface=ether10 pvid=20 hw=yes
add bridge=bridge interface=ether11
add bridge=bridge interface=ether11 pvid=20 hw=yes
add bridge=bridge interface=ether12
add bridge=bridge interface=ether12 pvid=20 hw=yes
add bridge=bridge interface=ether13
add bridge=bridge interface=ether13 pvid=20 hw=yes
add bridge=bridge interface=ether14
add bridge=bridge interface=ether14 pvid=20 hw=yes
add bridge=bridge interface=ether15
add bridge=bridge interface=ether15 pvid=20 hw=yes
add bridge=bridge interface=ether16
add bridge=bridge interface=ether16 pvid=20 hw=yes
add bridge=bridge interface=ether17
add bridge=bridge interface=ether17 pvid=30 hw=yes
add bridge=bridge interface=ether18
add bridge=bridge interface=ether18 pvid=30 hw=yes
add bridge=bridge interface=ether19
add bridge=bridge interface=ether19 pvid=30 hw=yes
add bridge=bridge interface=ether20
add bridge=bridge interface=ether20 pvid=30 hw=yes
add bridge=bridge interface=ether21
add bridge=bridge interface=ether21 pvid=30 hw=yes
add bridge=bridge interface=ether22
add bridge=bridge interface=ether22 pvid=30 hw=yes
add bridge=bridge interface=ether23
add bridge=bridge interface=ether23 pvid=30 hw=yes
add bridge=bridge interface=ether24
add bridge=bridge interface=ether24 pvid=30 hw=yes
add bridge=bridge interface=sfp1
</pre>
</pre>


{{ Note | If required, it is possible to use STP/RSTP/MSTP and IGMP Snooping with hardware offloading, make sure your device supports it. }}
{{ Note | If required, it is possible to use STP/RSTP/MSTP and IGMP Snooping with hardware offloading, make sure your device supports it. }}


===Port trunking===
{{ Warning | In this setup <code>vlan-filtering</code> is required, but it should be disabled while you are setting up the device. If you create a bridge with <code>vlan-filtering</code> enabled at the beginning, then you might loose access to the switch while you are configuring it. It is recommended to enable <code>vlan-filtering</code> only when management port and bridge VLAN table is configured. }}
 
=Bonding=
 
Bonding interfaces are used when a larger amount of bandwidth is required, this is done by creating a link aggregation group, which also provides hardware automatic failover and load balancing for CRS3xx series switches. By adding two 10Gbps interfaces to a bonding, you can increase the theoretical bandwidth limit to 20Gbps. Make sure that all bonded interfaces are linked to the same speed rates.
 
{{Note | CRS3xx series switches aggregate traffic using the built-in Switch Chip without using CPU resources, to route the traffic a router with a powerful CPU is required to handle the aggregated traffic.}}


Port trunking is used when a larger amount of bandwidth is required, this is done by creating a static link aggregation group, which also provides hardware automatic failover and load balancing for CRS1xx/CRS2xx series switches. By adding two 1Gbps interfaces to a trunk, you can increase the theoretical bandwidth limit to 2Gbps. Make sure that all trunked interfaces are linked to the same speed rates.
To create a 20Gbps bonding interface from sfp-sfpplus1 and sfp-sfpplus2 between SwitchA to SwitchB and between SwitchC to SwitchB, use these commands on '''SwitchA''' and '''SwitchC''':
<pre>
/interface bonding
add mode=802.3ad name=bond_1-2 slaves=sfp-sfpplus1,sfp-sfpplus2
</pre>


{{Note | CRS1xx/CRS2xx series switches aggregate traffic using the built-in Switch Chip without using CPU resources, to route the traffic a router with a powerful CPU is required to handle the aggregated traffic.}}
To create a 40Gbps bonding interface between SwitchB and the Router and 20Gbps bonding interfaces between SwitchA and SwitchC, use these commands on '''SwitchB''':
<pre>
/interface bonding
add mode=802.3ad name=bond_1-2 slaves=sfp-sfpplus1,sfp-sfpplus2
add mode=802.3ad name=bond_3-4 slaves=sfp-sfpplus3,sfp-sfpplus4
add mode=802.3ad name=bond_5-6-7-8 slaves=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8
</pre>


To create a 2Gbps port trunk from ether1 and ether2 between SwitchA, SwitchB and SwitchC, use these commands on '''SwitchA''' and '''SwitchC''':
When all the bonding interfaces are create, they must be added as a bridge port. Use these commands on '''SwitchA''' and '''SwitchB''':
<pre>
<pre>
/interface ethernet switch trunk
/interface bridge port
add member-ports=ether1,ether2 name=trunk-1-2
add bridge=bridge interface=bond_1-2 hw=yes
</pre>
</pre>


To create a 4Gbps port trunk from ether1,ether2,ether3,ether4 between SWitchB and the Router, use these commands on '''SwitchB''':
Add all bonding interfaces to a single bridge on SwitchB by using these commands on '''SwitchB''':
<pre>
<pre>
/interface ethernet switch trunk
/interface bridge port
add member-ports=ether1,ether2,ether3,ether4 name=trunk-1-2-3-4
add bridge=bridge interface=bond_1-2 hw=yes
add bridge=bridge interface=bond_3-4 hw=yes
add bridge=bridge interface=bond_5-6-7-8 hw=yes
</pre>
</pre>


On SwitchB ether5 and ether6 will be used to connect with SwitchA at 2Gbps and ether7,ether8 will be used to connect with SwitchC at 2Gbps. Use these command on '''SwitchB''':
In our case the Router needs a software based bonding interface, use these commands on '''Router''':
<pre>
<pre>
/interface ethernet switch trunk
/interface bonding
add member-ports=ether5,ether6 name=trunk-5-6
add mode=802.3ad name=bond_1-2-3-4 slaves=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4
add member-ports=ether7,ether8 name=trunk-7-8
</pre>
</pre>


===Management IP===
{{ Note | Interface bonding does not create a interface with a larger link speed. Interface bonding creates a virtual interface that can load balance traffic over multiple interfaces. More details can be found in the [[Manual:Layer2_misconfiguration#LAG_interfaces_and_load_balancing | LAG interfaces and load balancing]] page. }}
 
=Management IP=


It is very useful to create a management interface and assign an IP address to it in order to preserve access to the switch. This is also very useful when updating your switches since such traffic to the switch will be blocked when enabling invalid VLAN filtering.
It is very useful to create a management interface and assign an IP address to it in order to preserve access to the switch. This is also very useful when updating your switches since such traffic to the switch will be blocked when enabling invalid VLAN filtering.


Create a VLAN interface on '''SwitchA''', '''SwitchB''', '''SwitchC''':
Create a VLAN interface on '''SwitchA''', '''SwitchB''' and '''SwitchC''':
<pre>
/interface vlan
add interface=bridge name=MGMT vlan-id=99
</pre>
 
The Router needs the VLAN interface to be created on the bonding interface, use these commands to create a VLAN interface on ''Router''':
<pre>
<pre>
/interface vlan
/interface vlan
add interface=ether1 name=Vlan99 vlan-id=99
add interface=bond_1-2-3-4 name=MGMT vlan-id=99
</pre>
</pre>


{{Note | VLAN interface must be created on the master-port interface since it is the only interface that will be able to communicate the CPU.}}
{{Note | VLAN interface must be created on the bridge interface since it is the only interface that will be able to communicate the CPU.}}


For this guide we are going to use these addresses for each device:
For this guide we are going to use these addresses for each device:
Line 143: Line 135:
<pre>
<pre>
/ip address
/ip address
add address=192.168.99.X/24 interface=Vlan99
add address=192.168.99.X/24 interface=MGMT
</pre>
</pre>


Line 157: Line 149:
<pre>
<pre>
/ip address
/ip address
add address=192.168.99.1/24 interface=Vlan99
add address=192.168.99.1/24 interface=MGMT
</pre>
</pre>


===Bonding===
=Invalid VLAN filtering=
 
Unlike CRS1xx/CRS2xx series switches that use the built-in Switch Chip to create a aggregated link group, a router will use the CPU to create the aggregated link group.


To create a bonding interface for ether1,ether2,ether3,ether4, use these commands on the '''Router''':
Enable ingress traffic filtering for more security, use these commands on '''SwitchA''', '''SwitchB''' and '''SwitchC''':
<pre>
<pre>
/interface bonding
/interface bridge port
add mode=balance-xor name=bond1 slaves=ether1,ether2,ether3,ether4\
set [f] ingress-filtering=yes
transmit-hash-policy=layer-2-and-3
</pre>
</pre>


{{Warning | Don't use bonding interfaces on CRS series devices, bonding interface does NOT use the built-in Switch Chip to create aggregated link group and will overload the CPU instantly. For CRS series device use only port trunking.}}
Since most ports on SwitchA and SwitchC are going to be access ports, you can set all ports to accept only certain types of packets, in this case we will want SwitchA and SwitchC to only accept untagged packets, use these commands on '''SwitchA''' and '''SwitchC''':
 
Now add a VLAN interface can be created on the newly created bonding interface for management and assign an IP address to it, use these commands on the '''Router''':
<pre>
<pre>
/interface vlan
/interface bridge port
add interface=bond1 name=Vlan99 vlan-id=99
set [f] frame-types=admit-only-untagged-and-priority-tagged
/ip address
add address=192.168.99.1/24 interface=Vlan99
</pre>
</pre>


===Port based VLAN===
There is an exception for frame types on SwitchA and SwitchB, in this setup access to management port is required from ether1, bonding interfaces require that only tagged traffic can be forwarded. Use these commands on '''SwitchA''' and '''SwitchC''':
 
When using port trunks, the main difference is that access ports are now trunk ports and they should be used in the Egress VLAN tag table, Ingress VLAN translation table and VLAN table instead of physical Ethernet interfaces.
 
To create each trunk port as access port, use these commands on '''SwitchA''' and '''SwitchC''':
<pre>
<pre>
/interface ethernet switch egress-vlan-tag
/interface bridge port
add tagged-ports=trunk-1-2 vlan-id=10
set [find where interface=ether1] frame-types=admit-all
add tagged-ports=trunk-1-2 vlan-id=20
set [find where interface=bond_1-2] frame-types=admit-only-vlan-tagged
add tagged-ports=trunk-1-2 vlan-id=30
add tagged-ports=trunk-1-2 vlan-id=40
</pre>
</pre>


Similarly add entries to the Egress VLAN tag table for SwitchB, use these commands on '''SwitchB''':
On SwitchB only tagged packets should be forwarded, use these commands on '''SwitchB''':
<pre>
<pre>
/interface ethernet switch egress-vlan-tag
/interface bridge port
add tagged-ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8 vlan-id=10
set [f] frame-types=admit-only-vlan-tagged
add tagged-ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8 vlan-id=20
add tagged-ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8 vlan-id=30
add tagged-ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8 vlan-id=40
</pre>
 
{{Note | Management VLAN ID is not addedd to Egress VLAN tag table since a VLAN interface has been already created that will only send out tagged traffic either way.}}
 
Specify for each Ethernet interface a VLAN ID that will be assigned for a device that uses the port, use these commands for '''SwitchA''' and '''SwitchC''':
<pre>
/interface ethernet switch ingress-vlan-translation
add new-customer-vid=10 ports=ether3,ether4,ether5,ether6
add new-customer-vid=20 ports=ether7,ether8,ether9,ether10,ether11,ether12
add new-customer-vid=30 ports=ether13,ether14,ether15,ether16,ether17,ether18
add new-customer-vid=40 ports=ether19,ether20,ether21,ether22,ether23,ether24
</pre>
</pre>


Similarly specify a VLAN ID for each Ethernet interface on SwitchB, use these commands on '''SwitchB''':
It is required to setup bridge VLAN table. In this network setup we need to allow VLAN 10 on ether1-ether8, VLAN 20 on ether9-ether16, VLAN 30 on ether17-ether24, VLAN 10,20,30,99 on bond_1-2 and a special case for ether1 to allow to forward VLAN 99 on SwitchA and SwitchC. Use these commands on '''SwitchA''' and '''SwitchC''':
<pre>
<pre>
/interface ethernet switch ingress-vlan-translation
/interface bridge vlan
add new-customer-vid=10 ports=ether9,ether10,ether11,ether12
add bridge=bridge tagged=bond_1-2 untagged=ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-ids=10
add new-customer-vid=20 ports=ether13,ether14,ether15,ether16
add bridge=bridge tagged=bond_1-2 untagged=ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16 vlan-ids=20
add new-customer-vid=30 ports=ether17,ether18,ether19,ether20
add bridge=bridge tagged=bond_1-2 untagged=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 vlan-ids=30
add new-customer-vid=40 ports=ether21,ether22,ether23,ether24
add bridge=bridge tagged=bridge,bond_1-2,ether1 vlan-ids=99
</pre>
</pre>


It is required add allowed VLAN IDs to the VLAN table in order for VLAN filtering to work properly. Specify each VLAN ID and each port that is allowed to forward a certain VLAN ID. Use trunk ports instead of physical Ethernet interfaces. Use these commands on '''SwitchA''' and '''SwitchC''':
Similarly it is required to setup bridge VLAN table for SwitchB. Use these commands on '''SwitchB''':
<pre>
<pre>
/interface ethernet switch vlan
/interface bridge vlan
add ports=trunk-1-2,ether3,ether4,ether5,ether6 vlan-id=10
add bridge=bridge tagged=bond_1-2,bond_3-4,bond_5-6-7-8 vlan-ids=10,20,30
add ports=trunk-1-2,ether7,ether8,ether9,ether10,ether11,ether12 vlan-id=20
add bridge=bridge tagged=bond_1-2,bond_3-4,bond_5-6-7-8,bridge vlan-ids=9
add ports=trunk-1-2,ether13,ether14,ether15,ether16,ether17,ether18 vlan-id=30
add ports=trunk-1-2,ether19,ether20,ether21,ether22,ether23,ether24 vlan-id=40
add ports=trunk-1-2,switch1-cpu vlan-id=99
</pre>
</pre>


Similarly add entries to the VLAN table for SwitchB, use the commands '''on SwitchB''':
When everything is configured, VLAN filtering can be enabled. Use these commands on '''SwitchA''', '''SwitchB''' and '''SwitchC''':
<pre>
<pre>
/interface ethernet switch vlan
/interface bridge
add ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8,ether9,ether10,ether11,ether12 vlan-id=10
set bridge vlan-filtering=yes
add ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8,ether13,ether14,ether15,ether16 vlan-id=20
add ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8,ether17,ether18,ether19,ether20 vlan-id=30
add ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8,ether21,ether22,ether23,ether24 vlan-id=40
add ports=trunk-1-2-3-4,trunk-5-6,trunk-7-8,switch1-cpu vlan-id=99
</pre>
</pre>


{{Note | It is required to specify <code>switch1-cpu</code> port in the VLAN table in order to allow access the the previously created VLAN interface for management purposes.}}
{{ Warning | Double check if port based VLANs are set up properly. If a mistake was made, you might loose access to the switch and it can only be regained by resetting the configuration or by using the serial console. }}
 
===Invalid VLAN filtering===
 
If ingress VLAN translation table, egress VLAN tag table and VLAN table is properly set, invalid VLAN filtering can be enabled, which will drop any other packet that does not a suitable entry in the VLAN table.
 
{{Warning | Double check if port based VLAN is set up properly. If a mistake was made, you might loose access to the switch and can only be regained by resetting configuration or using the serial console.}}
 
To enable invalid VLAN filtering, use these commands on '''SwitchA''', '''SwitchB''', '''SwitchC''':
<pre>
/interface ethernet switch
set drop-if-invalid-or-src-port-not-member-of-vlan-on-ports="ether1,ether2,ether3,ether4\
,ether5,ether6,ether7,ether8,ether9,ether10,ether11,ether12,ether13\
,ether14,ether15,ether16,ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24"
</pre>


===InterVLAN routing===
=InterVLAN routing=


To create InterVLAN routing, VLAN interface for each customer VLAN ID must be created on the router and must have an IP address assigned to it. The VLAN interface must be created on the bonding interface created previously.
To create InterVLAN routing, VLAN interface for each customer VLAN ID must be created on the router and must have an IP address assigned to it. The VLAN interface must be created on the bonding interface created previously.
Line 266: Line 210:
<pre>
<pre>
/interface vlan
/interface vlan
add interface=bond1 name=Vlan10 vlan-id=10
add interface=bond_1-2-3-4 name=VLAN10 vlan-id=10
add interface=bond1 name=Vlan20 vlan-id=20
add interface=bond_1-2-3-4 name=VLAN20 vlan-id=20
add interface=bond1 name=Vlan30 vlan-id=30
add interface=bond_1-2-3-4 name=VLAN30 vlan-id=30
add interface=bond1 name=Vlan40 vlan-id=40
/ip address
/ip address
add address=192.168.10.1/24 interface=Vlan10
add address=192.168.10.1/24 interface=VLAN10
add address=192.168.20.1/24 interface=Vlan20
add address=192.168.20.1/24 interface=VLAN20
add address=192.168.30.1/24 interface=Vlan30
add address=192.168.30.1/24 interface=VLAN30
add address=192.168.40.1/24 interface=Vlan40
</pre>
</pre>


{{Note | These commands are required for DHCP-Server. In case interVLAN routing is not desired but a DHCP-Server on a single router is required, then use [[Manual:IP/Firewall/Filter| Firewall Filter]] to block access between different subnets.}}
{{Note | These commands are required for DHCP-Server. In case interVLAN routing is not desired but a DHCP-Server on a single router is required, then use [[Manual:IP/Firewall/Filter| Firewall Filter]] to block access between different subnets.}}


===DHCP-Server===
=DHCP-Server=


To get the DHCP-Server working for each VLAN ID, the server must be set up on the previously created VLAN interfaces (one server for each VLAN ID). Preferably each VLAN ID should have its own subnet and its own IP pool. DNS Server could be specified as the router's IP address for particular VLAN ID or a global DNS Server could be used, but this address must be reachable.
To get the DHCP-Server working for each VLAN ID, the server must be set up on the previously created VLAN interfaces (one server for each VLAN ID). Preferably each VLAN ID should have its own subnet and its own IP pool. DNS Server could be specified as the router's IP address for particular VLAN ID or a global DNS Server could be used, but this address must be reachable.
Line 286: Line 228:
<pre>
<pre>
/ip pool
/ip pool
add name=Vlan10_pool ranges=192.168.10.100-192.168.10.200
add name=VLAN10_POOL ranges=192.168.10.100-192.168.10.200
add name=Vlan20_pool ranges=192.168.20.100-192.168.20.200
add name=VLAN20_POOL ranges=192.168.20.100-192.168.20.200
add name=Vlan30_pool ranges=192.168.30.100-192.168.30.200
add name=VLAN30_POOL ranges=192.168.30.100-192.168.30.200
add name=Vlan40_pool ranges=192.168.40.100-192.168.40.200
/ip dhcp-server
/ip dhcp-server
add address-pool=Vlan10_pool disabled=no interface=Vlan10 name=Vlan10_DHCP
add address-pool=VLAN10_POOL disabled=no interface=VLAN10 name=VLAN10_DHCP
add address-pool=Vlan20_pool disabled=no interface=Vlan20 name=Vlan20_DHCP
add address-pool=VLAN20_POOL disabled=no interface=VLAN20 name=VLAN20_DHCP
add address-pool=Vlan30_pool disabled=no interface=Vlan30 name=Vlan30_DHCP
add address-pool=VLAN30_POOL disabled=no interface=VLAN30 name=VLAN30_DHCP
add address-pool=Vlan40_pool disabled=no interface=Vlan40 name=Vlan40_DHCP
/ip dhcp-server network
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=192.168.10.1 gateway=192.168.10.1
add address=192.168.10.0/24 dns-server=192.168.10.1 gateway=192.168.10.1
add address=192.168.20.0/24 dns-server=192.168.20.1 gateway=192.168.20.1
add address=192.168.20.0/24 dns-server=192.168.20.1 gateway=192.168.20.1
add address=192.168.30.0/24 dns-server=192.168.30.1 gateway=192.168.30.1
add address=192.168.30.0/24 dns-server=192.168.30.1 gateway=192.168.30.1
add address=192.168.40.0/24 dns-server=192.168.40.1 gateway=192.168.40.1
</pre>
</pre>


Line 310: Line 249:
{{Warning | Make sure to secure your local DNS Server with Firewall from the outside when using <code>allow-remote-requests</code> set to <code>yes</code> since your DNS Server can be used for DDoS attacks if it is accessible from the Internet by anyone.}}
{{Warning | Make sure to secure your local DNS Server with Firewall from the outside when using <code>allow-remote-requests</code> set to <code>yes</code> since your DNS Server can be used for DDoS attacks if it is accessible from the Internet by anyone.}}


Don't forget to create NAT, assuming that sfp-sfpplus1 is used as WAN port, use these commands on the '''Router''':
Don't forget to create NAT, assuming that sfp-sfpplus8 is used as WAN port, use these commands on the '''Router''':
<pre>
<pre>
/ip firewall nat
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp-sfpplus1
add action=masquerade chain=srcnat out-interface=sfp-sfpplus8
</pre>
</pre>


===Jumbo frames===
=Jumbo frames=


One can increase the total throughput in such a setup by enabling jumbo frames. This reduces the packet overhead by increasing the Maximum Transmission Unit (MTU). If a device in your network does not support jumbo frames, then it will not benefit from a larger MTU. Usually the whole network does not support jumbo frames, but you can still benefit when sending data between devices that support jumbo frames, including all switches in the path.
One can increase the total throughput in such a setup by enabling jumbo frames. This reduces the packet overhead by increasing the Maximum Transmission Unit (MTU). If a device in your network does not support jumbo frames, then it will not benefit from a larger MTU. Usually the whole network does not support jumbo frames, but you can still benefit when sending data between devices that support jumbo frames, including all switches in the path.
Line 338: Line 277:
{{Note | Don't forget to change the MTU on your client devices too, otherwise above mentioned settings will not have any effect.}}
{{Note | Don't forget to change the MTU on your client devices too, otherwise above mentioned settings will not have any effect.}}


== See also ==
=See also=


* [[M:Interface/Bonding | Bonding]]
* [[M:Interface/Bonding | Bonding]]
* [[M:CRS_examples | CRS examples]]
* [[M:CRS3xx_series_switches | CRS3xx manual]]
* [[M:CRS_features | CRS features]]
* [[M:Switch_Chip_Features | Switch Chip Features]]
* [[M:Switch_Chip_Features | Switch Chip Features]]
* [[M:IP/DNS | IP/DNS]]
* [[M:IP/DNS | IP/DNS]]
Line 349: Line 287:
* [[M:Interface/VLAN | VLAN]]
* [[M:Interface/VLAN | VLAN]]
* [[M:Maximum_Transmission_Unit_on_RouterBoards | MTU on RouterBOARD]]
* [[M:Maximum_Transmission_Unit_on_RouterBoards | MTU on RouterBOARD]]
{{cont}}
[[Category:Bridging and switching]]
[[Category:Examples]]

Latest revision as of 07:52, 28 July 2022

Applies to RouterOS: v6.42 +



Summary

This page will show how to configure multiple switches to use bonding interfaces and port based VLANs, it will also show a working example with a DHCP-Server, interVLAN routing, management IP and invalid VLAN filtering configuration.

Warning: This article applies to CRS3xx series devices and not CRS1xx/CRS2xx. For a similar setup for CRS1xx/CRS2xx series switches you can check CRS1xx/2xx VLANs with Trunks guide.


Alt text
CRS3xx bonds and port based VLANs

Note: For this network topology we will be using two CRS326-24G-2S+, one CRS317-1G-16S+ and one CCR1072-1G-8S+, but same principles can be applied to any CRS3xx series devices and a router.


In this setup SwitchA and SwitchC will tag all traffic from ports ether1-ether8 to VLAN ID 10, ether9-ether16 to VLAN ID 20, ether17-ether24 to VLAN ID 30. Management will only be possible if user is connecting with tagged traffic with VLAN ID 99 from ether1 on SwitchA or SwitchB, connecting to all devices will also be possible from the router using tagged traffic with VLAN ID 99. SFP+ ports in this setup are going to be used as VLAN trunk ports while being in a bond to create a LAG interface.

Port switching

All switches in this setup require that all used ports are switched together (except for ports that are going to be part of a bonding interface). Use these commands on SwitchA and SwitchC: 

/interface bridge
add name=bridge vlan-filtering=no
/interface bridge port
add bridge=bridge interface=ether1 pvid=10 hw=yes
add bridge=bridge interface=ether2 pvid=10 hw=yes
add bridge=bridge interface=ether3 pvid=10 hw=yes
add bridge=bridge interface=ether4 pvid=10 hw=yes
add bridge=bridge interface=ether5 pvid=10 hw=yes
add bridge=bridge interface=ether6 pvid=10 hw=yes
add bridge=bridge interface=ether7 pvid=10 hw=yes
add bridge=bridge interface=ether8 pvid=10 hw=yes
add bridge=bridge interface=ether9 pvid=20 hw=yes
add bridge=bridge interface=ether10 pvid=20 hw=yes
add bridge=bridge interface=ether11 pvid=20 hw=yes
add bridge=bridge interface=ether12 pvid=20 hw=yes
add bridge=bridge interface=ether13 pvid=20 hw=yes
add bridge=bridge interface=ether14 pvid=20 hw=yes
add bridge=bridge interface=ether15 pvid=20 hw=yes
add bridge=bridge interface=ether16 pvid=20 hw=yes
add bridge=bridge interface=ether17 pvid=30 hw=yes
add bridge=bridge interface=ether18 pvid=30 hw=yes
add bridge=bridge interface=ether19 pvid=30 hw=yes
add bridge=bridge interface=ether20 pvid=30 hw=yes
add bridge=bridge interface=ether21 pvid=30 hw=yes
add bridge=bridge interface=ether22 pvid=30 hw=yes
add bridge=bridge interface=ether23 pvid=30 hw=yes
add bridge=bridge interface=ether24 pvid=30 hw=yes

Note: If required, it is possible to use STP/RSTP/MSTP and IGMP Snooping with hardware offloading, make sure your device supports it.


Warning: In this setup vlan-filtering is required, but it should be disabled while you are setting up the device. If you create a bridge with vlan-filtering enabled at the beginning, then you might loose access to the switch while you are configuring it. It is recommended to enable vlan-filtering only when management port and bridge VLAN table is configured.


Bonding

Bonding interfaces are used when a larger amount of bandwidth is required, this is done by creating a link aggregation group, which also provides hardware automatic failover and load balancing for CRS3xx series switches. By adding two 10Gbps interfaces to a bonding, you can increase the theoretical bandwidth limit to 20Gbps. Make sure that all bonded interfaces are linked to the same speed rates.

Note: CRS3xx series switches aggregate traffic using the built-in Switch Chip without using CPU resources, to route the traffic a router with a powerful CPU is required to handle the aggregated traffic.


To create a 20Gbps bonding interface from sfp-sfpplus1 and sfp-sfpplus2 between SwitchA to SwitchB and between SwitchC to SwitchB, use these commands on SwitchA and SwitchC: 

/interface bonding
add mode=802.3ad name=bond_1-2 slaves=sfp-sfpplus1,sfp-sfpplus2

To create a 40Gbps bonding interface between SwitchB and the Router and 20Gbps bonding interfaces between SwitchA and SwitchC, use these commands on SwitchB: 

/interface bonding
add mode=802.3ad name=bond_1-2 slaves=sfp-sfpplus1,sfp-sfpplus2
add mode=802.3ad name=bond_3-4 slaves=sfp-sfpplus3,sfp-sfpplus4
add mode=802.3ad name=bond_5-6-7-8 slaves=sfp-sfpplus5,sfp-sfpplus6,sfp-sfpplus7,sfp-sfpplus8

When all the bonding interfaces are create, they must be added as a bridge port. Use these commands on SwitchA and SwitchB: 

/interface bridge port
add bridge=bridge interface=bond_1-2 hw=yes

Add all bonding interfaces to a single bridge on SwitchB by using these commands on SwitchB: 

/interface bridge port
add bridge=bridge interface=bond_1-2 hw=yes
add bridge=bridge interface=bond_3-4 hw=yes
add bridge=bridge interface=bond_5-6-7-8 hw=yes

In our case the Router needs a software based bonding interface, use these commands on Router: 

/interface bonding
add mode=802.3ad name=bond_1-2-3-4 slaves=sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4

Note: Interface bonding does not create a interface with a larger link speed. Interface bonding creates a virtual interface that can load balance traffic over multiple interfaces. More details can be found in the LAG interfaces and load balancing page.


Management IP

It is very useful to create a management interface and assign an IP address to it in order to preserve access to the switch. This is also very useful when updating your switches since such traffic to the switch will be blocked when enabling invalid VLAN filtering.

Create a VLAN interface on SwitchA, SwitchB and SwitchC: 

/interface vlan
add interface=bridge name=MGMT vlan-id=99

The Router needs the VLAN interface to be created on the bonding interface, use these commands to create a VLAN interface on Router': 

/interface vlan
add interface=bond_1-2-3-4 name=MGMT vlan-id=99

Note: VLAN interface must be created on the bridge interface since it is the only interface that will be able to communicate the CPU.


For this guide we are going to use these addresses for each device:

Address Device
192.168.99.1 Router
192.168.99.2 SwitchA
192.168.99.3 SwitchB
192.168.99.4 SwitchC

Add an IP address for each device on the VLAN interface (change X to appropriate number): 

/ip address
add address=192.168.99.X/24 interface=MGMT

Don't forget to add the default gateway and specify a DNS server: 

/ip route
add gateway=192.168.99.1
/ip dns
set servers=192.168.99.1

Add the IP address on the Router: 

/ip address
add address=192.168.99.1/24 interface=MGMT

Invalid VLAN filtering

Enable ingress traffic filtering for more security, use these commands on SwitchA, SwitchB and SwitchC: 

/interface bridge port
set [f] ingress-filtering=yes

Since most ports on SwitchA and SwitchC are going to be access ports, you can set all ports to accept only certain types of packets, in this case we will want SwitchA and SwitchC to only accept untagged packets, use these commands on SwitchA and SwitchC: 

/interface bridge port
set [f] frame-types=admit-only-untagged-and-priority-tagged

There is an exception for frame types on SwitchA and SwitchB, in this setup access to management port is required from ether1, bonding interfaces require that only tagged traffic can be forwarded. Use these commands on SwitchA and SwitchC: 

/interface bridge port
set [find where interface=ether1] frame-types=admit-all
set [find where interface=bond_1-2] frame-types=admit-only-vlan-tagged

On SwitchB only tagged packets should be forwarded, use these commands on SwitchB: 

/interface bridge port
set [f] frame-types=admit-only-vlan-tagged

It is required to setup bridge VLAN table. In this network setup we need to allow VLAN 10 on ether1-ether8, VLAN 20 on ether9-ether16, VLAN 30 on ether17-ether24, VLAN 10,20,30,99 on bond_1-2 and a special case for ether1 to allow to forward VLAN 99 on SwitchA and SwitchC. Use these commands on SwitchA and SwitchC: 

/interface bridge vlan
add bridge=bridge tagged=bond_1-2 untagged=ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-ids=10
add bridge=bridge tagged=bond_1-2 untagged=ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16 vlan-ids=20
add bridge=bridge tagged=bond_1-2 untagged=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 vlan-ids=30
add bridge=bridge tagged=bridge,bond_1-2,ether1 vlan-ids=99

Similarly it is required to setup bridge VLAN table for SwitchB. Use these commands on SwitchB: 

/interface bridge vlan
add bridge=bridge tagged=bond_1-2,bond_3-4,bond_5-6-7-8 vlan-ids=10,20,30
add bridge=bridge tagged=bond_1-2,bond_3-4,bond_5-6-7-8,bridge vlan-ids=9

When everything is configured, VLAN filtering can be enabled. Use these commands on SwitchA, SwitchB and SwitchC: 

/interface bridge
set bridge vlan-filtering=yes

Warning: Double check if port based VLANs are set up properly. If a mistake was made, you might loose access to the switch and it can only be regained by resetting the configuration or by using the serial console.


InterVLAN routing

To create InterVLAN routing, VLAN interface for each customer VLAN ID must be created on the router and must have an IP address assigned to it. The VLAN interface must be created on the bonding interface created previously.

Use these commands on the Router: 

/interface vlan
add interface=bond_1-2-3-4 name=VLAN10 vlan-id=10
add interface=bond_1-2-3-4 name=VLAN20 vlan-id=20
add interface=bond_1-2-3-4 name=VLAN30 vlan-id=30
/ip address
add address=192.168.10.1/24 interface=VLAN10
add address=192.168.20.1/24 interface=VLAN20
add address=192.168.30.1/24 interface=VLAN30

Note: These commands are required for DHCP-Server. In case interVLAN routing is not desired but a DHCP-Server on a single router is required, then use Firewall Filter to block access between different subnets.


DHCP-Server

To get the DHCP-Server working for each VLAN ID, the server must be set up on the previously created VLAN interfaces (one server for each VLAN ID). Preferably each VLAN ID should have its own subnet and its own IP pool. DNS Server could be specified as the router's IP address for particular VLAN ID or a global DNS Server could be used, but this address must be reachable.

To set up the DHCP-Server, use these commands on the Router: 

/ip pool
add name=VLAN10_POOL ranges=192.168.10.100-192.168.10.200
add name=VLAN20_POOL ranges=192.168.20.100-192.168.20.200
add name=VLAN30_POOL ranges=192.168.30.100-192.168.30.200
/ip dhcp-server
add address-pool=VLAN10_POOL disabled=no interface=VLAN10 name=VLAN10_DHCP
add address-pool=VLAN20_POOL disabled=no interface=VLAN20 name=VLAN20_DHCP
add address-pool=VLAN30_POOL disabled=no interface=VLAN30 name=VLAN30_DHCP
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=192.168.10.1 gateway=192.168.10.1
add address=192.168.20.0/24 dns-server=192.168.20.1 gateway=192.168.20.1
add address=192.168.30.0/24 dns-server=192.168.30.1 gateway=192.168.30.1

In case the router's DNS Server is being used, don't forget to allow remote requests and make sure DNS Servers are configured on the router. Use these commands on the Router: 

/ip dns
set allow-remote-requests=yes servers=8.8.8.8

Warning: Make sure to secure your local DNS Server with Firewall from the outside when using allow-remote-requests set to yes since your DNS Server can be used for DDoS attacks if it is accessible from the Internet by anyone.


Don't forget to create NAT, assuming that sfp-sfpplus8 is used as WAN port, use these commands on the Router: 

/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp-sfpplus8

Jumbo frames

One can increase the total throughput in such a setup by enabling jumbo frames. This reduces the packet overhead by increasing the Maximum Transmission Unit (MTU). If a device in your network does not support jumbo frames, then it will not benefit from a larger MTU. Usually the whole network does not support jumbo frames, but you can still benefit when sending data between devices that support jumbo frames, including all switches in the path.

In this case, if clients behind SwitchA and client behind SwitchC supports jumbo frames, then enabling jumbo frames will be beneficial. Before enabling jumbo frames, determine the MAX-L2MTU by using this command: 

[admin@MikroTik] > /interface> print 
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU      
 0  R  ether1                              ether            1500  1580       4064

Note: More information can be found in MTU manual page.


When MAX-L2MTU is determined, choose the MTU size depending on the traffic on your network, use this command on SwitchA, SwitchB and SwitchC: 

/interface ethernet
set [ find ] l2mtu=4064 mtu=4040

Note: Don't forget to change the MTU on your client devices too, otherwise above mentioned settings will not have any effect.


See also

[ Top | Back to Content ]

Retrieved from "https://wiki.mikrotik.com/index.php?title=Manual:CRS3xx_VLANs_with_Bonds&oldid=34554"