Manual:Interface/Bonding: Difference between revisions
No edit summary |
m (→Notes) |
||
(44 intermediate revisions by 8 users not shown) | |||
Line 1: | Line 1: | ||
{{Versions|v3, v4}} | |||
{{Warning|This manual is moved to https://help.mikrotik.com/docs/display/ROS/Bonding}} | |||
== Summary == | == Summary == | ||
Bonding is a technology that allows | Bonding is a technology that allows aggregation of multiple ethernet-like interfaces into a single virtual link, thus getting higher data rates and providing failover. | ||
{{ Note | Interface bonding does not create a interface with a larger link speed. Interface bonding creates a virtual interface that can load balance traffic over multiple interfaces. More details can be found in the [[Manual:Layer2_misconfiguration#LAG_interfaces_and_load_balancing | LAG interfaces and load balancing]] page. }} | |||
== Specifications == | == Specifications == | ||
<ul class="bullets"> | |||
<li>'''Packages required''': system <br/> | |||
<li>'''License required''': Level1 <br/> | |||
<li>'''Submenu level''': <code>/interface bonding</code> <br/> | |||
<li>'''Standards and Technologies''': None <br/> | |||
<li>'''Hardware usage''': Not significant <br/> | |||
</ul> | |||
== Quick Setup Guide == | == Quick Setup Guide == | ||
Line 16: | Line 23: | ||
Let us assume that we have 2 NICs in each router (Router1 and Router2) and want to get maximum data rate between 2 routers. To make this possible, follow these steps: | Let us assume that we have 2 NICs in each router (Router1 and Router2) and want to get maximum data rate between 2 routers. To make this possible, follow these steps: | ||
<ul class="bullets"> | |||
<li>Make sure that you do not have IP addresses on interfaces which will be enslaved for bonding interface! | |||
<li>Add bonding interface on Router1: | |||
</ul> | |||
[admin@Router1] interface bonding> add slaves=ether1,ether2 | [admin@Router1] interface bonding> add slaves=ether1,ether2 | ||
Line 32: | Line 39: | ||
Test the link from Router1: | Test the link from Router1: | ||
<pre> | |||
[admin@Router1] interface bonding> /pi 172.16.0.2 | |||
172.16.0.2 ping timeout | |||
172.16.0.2 ping timeout | |||
172.16.0.2 ping timeout | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
</pre> | |||
{{Note | bonding interface needs a couple of seconds to get connectivity with its peer. }} | |||
== Link monitoring == | == Link monitoring == | ||
It is critical that one of available link monitoring options | It is critical that one of the available link monitoring options is enabled. In the above example, if one of the bonded links were to fail, the bonding driver will still continue to send packets over the failed link which will lead to network degradation. | ||
Bonding in RouterOS currently supports two schemes for monitoring a link state of slave devices: MII and ARP monitoring. It is not possible to use both methods at the same time due to restrictions in the bonding driver. | |||
=== ARP Monitoring === | === ARP Monitoring === | ||
Line 53: | Line 60: | ||
If balance-rr and balance-xor modes are set, then the switch should be configured to evenly distribute packets across all links. Otherwise all replies from the ARP targets will be received on the same link which could cause other links to fail. | If balance-rr and balance-xor modes are set, then the switch should be configured to evenly distribute packets across all links. Otherwise all replies from the ARP targets will be received on the same link which could cause other links to fail. | ||
ARP monitoring is enabled by setting three properties <var>link-monitoring</var>, <var>arp-ip-targets</var> and <var>arp-interval</var>. Meaning of each option is described later in this article. | ARP monitoring is enabled by setting three properties <var>link-monitoring</var>, <var>arp-ip-targets</var> and <var>arp-interval</var>. Meaning of each option is described later in this article. | ||
It is possible to specify multiple ARP targets that can be useful in | It is possible to specify multiple ARP targets that can be useful in High Availability setups. If only one target is set, the target itself may go down. Having additional targets increases the reliability of the ARP monitoring. | ||
Enable ARP monitoring | Enable ARP monitoring | ||
Line 59: | Line 66: | ||
[admin@Router2] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.1 | [admin@Router2] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.1 | ||
We will not change arp-interval value in our example, RouterOS sets arp-interval to 100ms by default. | We will not change <var>arp-interval</var> value in our example, RouterOS sets <var>arp-interval</var> to 100ms by default. | ||
Unplug one of the cables to test if link monitoring works correctly, you will notice some ping timeouts until arp monitoring detects link failure. | Unplug one of the cables to test if the link monitoring works correctly, you will notice some ping timeouts until arp monitoring detects link failure. | ||
<pre> | |||
[admin@Router1] interface bonding> /pi 172.16.0.2 | |||
172.16.0.2 ping timeout | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
172.16.0.2 ping timeout | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
172.16.0.2 ping timeout | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
172.16.0.2 64 byte ping: ttl=64 time=2 ms | |||
</pre> | |||
{{ Note | For ARP monitoring to work properly it is not required to have any IP address on the device, ARP monitoring will work regardless of the IP address that is set on any interface. }} | |||
{{ Warning | When ARP monitoring is used, bonding slaves will send out ARP requests without a VLAN tag, even if an IP address is set on a VLAN interface in the same subnet as the <var>arp-ip-targets</var> }} | |||
=== MII monitoring === | === MII monitoring === | ||
MII monitoring monitors only the state of the local interface. | MII monitoring monitors only the state of the local interface. ''MII Type 1'' - device driver determines whether link is up or down. If device driver does not support this option then link will appear as always up. Main disadvantage is that MII monitoring can't tell if the link can actually pass packets or not, even if the link is detected as being up. | ||
MII monitoring is configured by setting the variables <var>link-monitoring</var> mode and <var>mii-interval</var>. | |||
Enable MII Type1 monitoring: | |||
<pre> | |||
[admin@Router1] interface bonding> set 0 link-monitoring=mii | |||
[admin@Router2] interface bonding> set 0 link-monitoring=mii | |||
</pre> | |||
We will leave <var>mii-interval</var> to it's default value (100ms) | |||
When unplugging one of the cables, the failure will be detected almost instantly compared to ARP link monitoring. | |||
== Bonding modes == | |||
=== 802.3ad === | |||
802.3ad mode is an IEEE standard also called LACP (Link Aggregation Control Protocol). It includes automatic configuration of the aggregates, so minimal configuration of the switch is needed. This standard also mandates that frames will be delivered in order and connections should not see mis-ordering of packets. The standard also mandates that all devices in the aggregate must operate at the same speed and duplex mode. | |||
LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address and if available, the VLAN tag, and the IPv4/IPv6 source and destination address. How this is calculated depends on <var>transmit-hash-policy</var> parameter. The ARP link monitoring is not recommended, because the ARP replies might arrive only on one slave port due to transmit hash policy on the LACP peer device. This can result in unbalanced transmitted traffic, so MII link monitoring is the recommended option. | |||
{{ Note | <var>layer-3-and-4</var> transmit hash mode is not fully compatible with LACP. More details can be found in https://www.kernel.org/doc/Documentation/networking/bonding.txt}} | |||
'''Configuration example''' | |||
[[File:bonding-lacp-example.png]] | |||
Example connects two ethernet interfaces on a router to the Edimax switch as a single, load balanced and fault tolerant link. More interfaces can be added to increase throughput and fault tolerance. Since frame ordering is mandatory on Ethernet links then any traffic between two devices always flows over the same physical link limiting the maximum speed to that of one interface. The transmit algorithm attempts to use as much information as it can to distinguish different traffic flows and balance across the available interfaces. | |||
Router R1 configuration: | |||
<pre> | |||
/inteface bonding add slaves=ether1,ether2 mode=802.3ad lacp-rate=30secs link-monitoring=mii \ | |||
transmit-hash-policy=layer-2-and-3 | |||
</pre> | |||
== | Configuration on a switch: | ||
<pre> | |||
Intelligent Switch : Trunk Configuration | |||
================== | |||
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 M1 M2 | |||
1 - v - v - - - - - - - - - - - - - - - - - - - - - - | |||
2 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
3 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
4 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
5 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
6 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
7 - - - - - - - - - - - - - - - - - - - - - - - - - - | |||
TRK1 LACP | |||
TRK2 Disable | |||
TRK3 Disable | |||
TRK4 Disable | |||
TRK5 Disable | |||
TRK6 Disable | |||
TRK7 Disable | |||
</pre> | |||
Notice that LACP is enabled on first trunk group (TRK1) and switch ports on first trunk group are bound with 'v' flag. In our case port 2 and port4 will run LACP. | |||
Verify if LACP is working: | |||
On the switch we should first verify if LACP protocol is enabled and running: | |||
<pre> | |||
Intelligent Switch : LACP Port State Active Configuration | |||
================== | |||
Port State Activity Port State Activity | |||
--------------------------- --------------------------- | |||
2 Active | |||
4 Active | |||
</pre> | |||
After that we can ensure that LACP negotiated with our router. If you don't see both ports on the list then something is wrong and LACP is not going to work. | |||
<pre> | |||
Intelligent Switch : LACP Group Status | |||
================== | |||
Group | |||
[Actor] [Partner] | |||
Priority: 1 65535 | |||
MAC : 000E2E2206A9 000C42409426 | |||
Port_No Key Priority Active Port_No Key Priority | |||
2 513 1 selected 1 9 255 | |||
4 513 1 selected 2 9 255 | |||
</pre> | |||
After we verified that switch successfully negotiated LACP with our router, we can start traffic from Client1 and Client2 to the Server and check how traffic is evenly forwarded through both bonding slaves: | |||
<pre> | |||
[admin@test-host] /interface> monitor-traffic ether1,ether2,bonding1 | |||
rx-packets-per-second: 8158 8120 16278 | |||
rx-drops-per-second: 0 0 0 | |||
rx-errors-per-second: 0 0 0 | |||
rx-bits-per-second: 98.8Mbps 98.2Mbps 197.0Mbps | |||
tx-packets-per-second: 4833 4560 9394 | |||
tx-drops-per-second: 0 0 0 | |||
tx-errors-per-second: 0 0 0 | |||
tx-bits-per-second: 2.7Mbps 3.0Mbps 5.8Mbps | |||
</pre> | |||
{{ Note | On some switches you need to set correct link aggregation protocol, to make balancing work in both directions }} | |||
=== balance-rr === | === balance-rr === | ||
Line 101: | Line 204: | ||
Balance-rr is the only mode that will send packets across multiple interfaces that belong to the same TCP/IP connection. | Balance-rr is the only mode that will send packets across multiple interfaces that belong to the same TCP/IP connection. | ||
<br /> | <br /> | ||
When utilizing multiple sending and multiple receiving links, packets often | When utilizing multiple sending and multiple receiving links, packets are often received out of order, which result in segment retransmission, for other protocols such as UDP it is not a problem if client software can tolerate out-of-order packets.<br /> | ||
If switch is used to aggregate links together, then appropriate switch port configuration is required, however many switches do not support balance-rr.<br /> | If switch is used to aggregate links together, then appropriate switch port configuration is required, however many switches do not support balance-rr.<br /> | ||
[[#Quick_Setup_Guide | Quick setup guide]] demonstrates use of the balance-rr bonding mode. As you can see, it is quite simple to set up. Balance-rr is also useful for bonding several wireless links, however it requires equal bandwidth for all bonded links. If bandwidth of one bonded link drops, then total bandwidth of bond will be equal to bandwidth of the slowest bonded link. | [[#Quick_Setup_Guide | Quick setup guide]] demonstrates use of the balance-rr bonding mode. As you can see, it is quite simple to set up. Balance-rr is also useful for bonding several wireless links, however it requires equal bandwidth for all bonded links. If bandwidth of one bonded link drops, then total bandwidth of bond will be equal to the bandwidth of the slowest bonded link. | ||
=== active-backup === | === active-backup === | ||
This mode uses only one active slave to transmit packets. | This mode uses only one active slave to transmit packets. The additional slave only becomes active if the primary slave fails. The MAC address of the bonding interface is presented onto the active port to avoid confusing the switch. | ||
Active-backup is best choice in high availability setups with multiple switches that are interconnected. | Active-backup is the best choice in high availability setups with multiple switches that are interconnected. | ||
{{Note | ARP monitoring in this mode will not work correctly if both routers are directly connected. In such setups <var>mii</var> monitoring must be used or a switch should be put between routers. }} | |||
=== balance-xor === | === balance-xor === | ||
This mode balances outgoing traffic across the active ports based on the hashed protocol header information and accepts incoming traffic from any active port. Mode is very similar to [[#802.3ad | LACP]] except that it is not standardized and works with '''layer-3-and-4''' hash policy. | |||
=== broadcast === | === broadcast === | ||
When ports are configured with broadcast mode, all slave ports transmit the same packets to the destination to provide fault tolerance. This mode does not provide load balancing. | |||
=== balance-tlb === | === balance-tlb === | ||
This mode balances outgoing traffic by peer. | This mode balances outgoing traffic by peer. | ||
Each link can be a different speed and duplex and no specific switch configuration is required as | Each link can be a different speed and duplex mode and no specific switch configuration is required as for the other modes. | ||
Downside of this mode is that only MII link monitoring is supported and incoming traffic is not balanced. Incoming traffic will use the link that is configured as "primary". | Downside of this mode is that only MII link monitoring is supported (ARP link monitoring is ignored when configured) and incoming traffic is not balanced. Incoming traffic will use the link that is configured as "primary". | ||
<b> Configuration example</b><br /> | <b> Configuration example</b><br /> | ||
Line 126: | Line 234: | ||
/interface bonding add mode=balance-tlb slaves=ether1,ether2 primary=ether1 | /interface bonding add mode=balance-tlb slaves=ether1,ether2 primary=ether1 | ||
</pre> | </pre> | ||
No additional configuration is required for the switch.<br /> | |||
[[Image:bon-tlb.png|500px|balance-tlb]] | [[Image:bon-tlb.png|500px|balance-tlb]] | ||
<br /> | <br /> | ||
Image above illustrates how balance-tlb mode works. As you can see router can communicate to all the clients connected to switch with total bandwidth of both links (15Mbps). But as you already know, balance-tlb is not balancing incoming traffic. In our example clients can communicate to router with total bandwidth of primary link which is 10Mbps in our configuration. | Image above illustrates how <var>balance-tlb</var> mode works. As you can see router can communicate to all the clients connected to the switch with a total bandwidth of both links (15Mbps). But as you already know, balance-tlb is not balancing incoming traffic. In our example clients can communicate to router with total bandwidth of primary link which is 10Mbps in our configuration. | ||
=== balance-alb === | === balance-alb === | ||
Mode is basically the same as < | Mode is basically the same as <var>balance-tlb</var> but incoming IPv4 traffic is also balanced. Only MII link monitoring is supported (ARP link monitoring is ignored when configured), additional downside of this mode is that it requires device driver capability to change MAC address. Most of the cheap cards do not support this mode.<br /> | ||
[[Image:bon-alb.png|500px|balance-alb]] | [[Image:bon-alb.png|500px|balance-alb]] | ||
<br /> | <br /> | ||
Image above illustrates how balance-alb mode works. Compared to balance-tlb traffic from clients also | Image above illustrates how <var>balance-alb</var> mode works. Compared to <var>balance-tlb</var> mode, traffic from clients can also use the secondary link to communicate with the router. | ||
== Property Description == | == Property Description == | ||
Line 146: | Line 255: | ||
<td><var><b>arp</b></var> (<em>disabled | enabled | proxy-arp | reply-only</em>; Default: <b>enabled</b>)</td> | <td><var><b>arp</b></var> (<em>disabled | enabled | proxy-arp | reply-only</em>; Default: <b>enabled</b>)</td> | ||
<td> Address Resolution Protocol for the interface. | <td> Address Resolution Protocol for the interface. | ||
<ul class="bullets"> | |||
<li> <var>disabled</var> - the interface will not use ARP | |||
<li> <var>enabled</var> - the interface will use ARP | |||
<li> <var>proxy-arp</var> - the interface will use the ARP proxy feature | |||
<li> <var>reply-only</var> - the interface will only reply to requests originated from matching IP address/MAC address combinations which are entered as static entries in the "/ip arp" table. No dynamic entries will be automatically stored in the "/ip arp" table. Therefore for communications to be successful, a valid static entry must already exist. | |||
</ul> | |||
</td> | |||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>arp-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td> | <td><var><b>arp-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td> | ||
<td> | <td> Time in milliseconds which defines how often to monitor ARP requests </td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>arp-ip-targets</b></var> (<em>IP | <td><var><b>arp-ip-targets</b></var> (<em>IP address</em>; Default: <b></b>)</td> | ||
<td> IP target address which will be monitored if <var>link-monitoring</var> is set to arp. You can specify multiple IP addresses, separated by comma </td> | <td> IP target address which will be monitored if <var>link-monitoring</var> is set to arp. You can specify multiple IP addresses, separated by comma </td> | ||
</tr> | |||
<tr> | |||
<td><var><b>comment</b></var> (<em>string</em>; Default: )</td> | |||
<td> Short description of the interface </td> | |||
</tr> | |||
<tr> | |||
<td><var><b>disabled</b></var> (<em>yes | no</em>; Default: <b>no</b>)</td> | |||
<td> Changes whether the bonding interface is disabled </td> | |||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>down-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td> | <td><var><b>down-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td> | ||
<td> | <td> If a link failure has been detected, bonding interface is disabled for <var>down-delay</var> time. Value should be a multiple of <var>mii-interval</var>, otherwise it will be rounded down to the nearest value.</td> | ||
</tr> | |||
<tr> | |||
<td><var><b>forced-mac-address</b></var> (<em>MAC address</em>; Default: <b>none</b>)</td> | |||
<td> By default, bonding interface will use MAC address of the first selected slave interface. This property allows to configure static MAC address for the bond interface (all zeros, broadcast or multicast addresses will not apply). RouterOS will automatically change the MAC address for slave interfaces and it will be visible in <code>/interface ethernet</code> configuration export</td> | |||
</tr> | </tr> | ||
<tr> | <tr> | ||
Line 168: | Line 292: | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>link-monitoring</b></var> (<em>arp | mii | <td><var><b>link-monitoring</b></var> (<em>arp | mii | none</em>; Default: <b>mii</b>)</td> | ||
<td> | <td> Method to use for monitoring the link (whether it is up or down) | ||
<ul class="bullets"> | |||
<li> <var>arp</var> - uses Address Resolution Protocol to determine whether the remote interface is reachable | |||
<li> <var>mii</var> - uses Media Independent Interface to determine link status. Link status determination relies on the device driver. | |||
<li> <var>none</var> - no method for link monitoring is used. | |||
</ul> | |||
<b>Note:</b> some bonding modes require specific link monitoring to work properly.</td> | <b>Note:</b> some bonding modes require specific link monitoring to work properly.</td> | ||
</tr> | |||
<tr> | |||
<td><var><b>min-links</b></var> (<em>integer: 0..4294967295</em>; Default: <b>0</b>)</td> | |||
<td> How many active slave links needed for bonding to become active</td> | |||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>mii-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td> | <td><var><b>mii-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td> | ||
<td> | <td> How often to monitor the link for failures (parameter used only if <var>link-monitoring</var> is mii)</td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>mode</b></var> (<em>802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast</em>; Default: <b>balance-rr</b>)</td> | <td><var><b>mode</b></var> (<em>802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast</em>; Default: <b>balance-rr</b>)</td> | ||
<td> Specifies one of the bonding policies | <td> Specifies one of the bonding policies | ||
<ul> | <ul class="bullets"> | ||
<li><var>802.3ad</var> - IEEE 802.3ad dynamic link aggregation. In this mode, the interfaces are aggregated in a group where each slave shares the same speed. Provides fault tolerance and load balancing. Slave selection for outgoing traffic is done according to the < | <li><var>802.3ad</var> - IEEE 802.3ad dynamic link aggregation. In this mode, the interfaces are aggregated in a group where each slave shares the same speed. Provides fault tolerance and load balancing. Slave selection for outgoing traffic is done according to the <var>transmit-hash-policy</var> [[#802.3ad|more>]] | ||
<li><var>active-backup</var> - provides link backup. Only one slave can be active at a time. Another slave becomes active | <li><var>active-backup</var> - provides link backup. Only one slave can be active at a time. Another slave only becomes active, if first one fails. [[#active-backup|more>]] | ||
<li><var>balance-alb</var> - adaptive load balancing. The same as < | <li><var>balance-alb</var> - adaptive load balancing. The same as <var>balance-tlb</var> but received traffic is also balanced. Device driver should have support for changing it's MAC address. [[#balance-alb|more>]] | ||
<li><var>balance-rr</var> - round-robin load balancing. Slaves in bonding interface will transmit and receive data in sequential order. Provides load balancing and fault tolerance. [[#balance-rr|more>]] | <li><var>balance-rr</var> - round-robin load balancing. Slaves in bonding interface will transmit and receive data in sequential order. Provides load balancing and fault tolerance. [[#balance-rr|more>]] | ||
<li><var>balance-tlb</var> - Outgoing traffic is distributed according to the current load on each slave. Incoming traffic is not balanced and is received by the current slave. If receiving slave fails, then another slave takes the MAC address of the failed slave. [[#balance-tlb|more>]] | <li><var>balance-tlb</var> - Outgoing traffic is distributed according to the current load on each slave. Incoming traffic is not balanced and is received by the current slave. If receiving slave fails, then another slave takes the MAC address of the failed slave. [[#balance-tlb|more>]] | ||
<li><var>balance-xor</var> - Transmit based on the selected < | <li><var>balance-xor</var> - Transmit based on the selected <var>transmit-hash-policy</var>. This mode provides load balancing and fault tolerance. [[#balance-xor|more>]] | ||
<li><var>broadcast</var> - Broadcasts the same data on all interfaces at once. This provides fault tolerance but slows down traffic throughput on some slow machines. [[#broadcast|more>]] | <li><var>broadcast</var> - Broadcasts the same data on all interfaces at once. This provides fault tolerance but slows down traffic throughput on some slow machines. [[#broadcast|more>]] | ||
</ul> | </ul> | ||
</td> | </td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>mtu</b></var> (<em>integer</em>; Default: <b>1500</b>)</td> | <td><var><b>mtu</b></var> (<em>integer</em>; Default: <b>1500</b>)</td> | ||
<td> Maximum Transmit Unit in bytes </td> | <td> Maximum Transmit Unit in bytes. Must be smaller or equal to the smallest L2MTU value of a bonding slave.</td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>name</b></var> (<em>string</em>; Default: <b></b>)</td> | <td><var><b>name</b></var> (<em>string</em>; Default: <b></b>)</td> | ||
<td> | <td> Name of the bonding interface </td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>primary</b></var> (<em>string</em>; Default: <b></b>)</td> | <td><var><b>primary</b></var> (<em>string</em>; Default: <b></b>)</td> | ||
<td> | <td> Controls the primary interface between active slave ports, works only for <var>active-backup</var>, <var>balance-tlb</var> and <var>balance-alb</var> modes. For <var>active-backup</var> mode, it controls which running interface is supposed to send and receive the traffic. For <var>balance-tlb</var> mode, it controls which running interface is supposed to receive all the traffic, but for <var>balance-alb</var> mode, it controls which interface is supposed to receive the unbalanced traffic (the non-IPv4 traffic). When none of the interfaces are selected as primary, device will automatically select the interface that is configured as the first one.</td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>slaves</b></var> (<em>string</em>; Default: <b>none</b>)</td> | <td><var><b>slaves</b></var> (<em>string</em>; Default: <b>none</b>)</td> | ||
<td> | <td> At least two ethernet-like interfaces separated by a comma, which will be used for bonding</td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>up-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td> | <td><var><b>up-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td> | ||
<td> | <td> If a link has been brought up, bonding interface is disabled for <var>up-delay</var> time and after this time it is enabled. Value should be a multiple of <var>mii-interval</var>, otherwise it will be rounded down to the nearest value.</td> | ||
</tr> | </tr> | ||
<tr> | <tr> | ||
<td><var><b>transmit-hash-policy</b></var> (<em>layer-2 | layer-2-and-3 | layer-3-and-4</em>; Default: <b>layer-2</b>)</td> | <td><var><b>transmit-hash-policy</b></var> (<em>layer-2 | layer-2-and-3 | layer-3-and-4</em>; Default: <b>layer-2</b>)</td> | ||
<td> Selects the transmit hash policy to use for slave selection in balance-xor and 802.3ad modes | <td> Selects the transmit hash policy to use for slave selection in balance-xor and 802.3ad modes | ||
<ul class="bullets"> | |||
<li> <var>layer-2</var> - Uses XOR of hardware MAC addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. This algorithm is 802.3ad compliant. | |||
<li> <var>layer-2-and-3</var> - This policy uses a combination of layer2 and layer3 protocol information to generate the hash. Uses XOR of hardware MAC addresses and IP addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations. This algorithm is 802.3ad compliant. | |||
<li> <var>layer-3-and-4</var> - This policy uses upper layer protocol information, when available, to generate the hash. This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves. For fragmented TCP or UDP packets and all other IP protocol traffic, the source and destination port information is omitted. For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This algorithm is not fully 802.3ad compliant. | |||
</ul> | |||
</td> | |||
</tr> | </tr> | ||
</table> | </table> | ||
== Notes == | |||
The rate at which link failure detection and failover takes place can depend on the network adapters used. On Intel cards, for example, failover takes place in less than a second after link loss, while on some other cards, it may require up to 20 seconds. Also, active load balancing (<code>mode=balance-alb</code>) does not work on some cheap cards. | |||
L2 MTU of a bonding interface is determined by the lowest L2MTU value among its slave interfaces. | |||
== See also == | == See also == | ||
Line 233: | Line 367: | ||
* [[Bonding Examples]] | * [[Bonding Examples]] | ||
[[Category: Manual]] | {{Cont}} | ||
[[Category: QoS]] | |||
[[Category: | [[Category: Manual|B]] | ||
[[Category: QoS|B]] | |||
[[Category:Interface|B]] |
Latest revision as of 07:36, 9 July 2020
Warning: This manual is moved to https://help.mikrotik.com/docs/display/ROS/Bonding
Summary
Bonding is a technology that allows aggregation of multiple ethernet-like interfaces into a single virtual link, thus getting higher data rates and providing failover.
Note: Interface bonding does not create a interface with a larger link speed. Interface bonding creates a virtual interface that can load balance traffic over multiple interfaces. More details can be found in the LAG interfaces and load balancing page.
Specifications
- Packages required: system
- License required: Level1
- Submenu level:
/interface bonding
- Standards and Technologies: None
- Hardware usage: Not significant
Quick Setup Guide
Let us assume that we have 2 NICs in each router (Router1 and Router2) and want to get maximum data rate between 2 routers. To make this possible, follow these steps:
- Make sure that you do not have IP addresses on interfaces which will be enslaved for bonding interface!
- Add bonding interface on Router1:
[admin@Router1] interface bonding> add slaves=ether1,ether2
And on Router2:
[admin@Router2] interface bonding> add slaves=ether1,ether2
Add addresses to bonding interfaces:
[admin@Router1] ip address> add address=172.16.0.1/24 interface=bonding1 [admin@Router2] ip address> add address=172.16.0.2/24 interface=bonding1
Test the link from Router1:
[admin@Router1] interface bonding> /pi 172.16.0.2 172.16.0.2 ping timeout 172.16.0.2 ping timeout 172.16.0.2 ping timeout 172.16.0.2 64 byte ping: ttl=64 time=2 ms 172.16.0.2 64 byte ping: ttl=64 time=2 ms
Link monitoring
It is critical that one of the available link monitoring options is enabled. In the above example, if one of the bonded links were to fail, the bonding driver will still continue to send packets over the failed link which will lead to network degradation. Bonding in RouterOS currently supports two schemes for monitoring a link state of slave devices: MII and ARP monitoring. It is not possible to use both methods at the same time due to restrictions in the bonding driver.
ARP Monitoring
ARP monitoring sends ARP queries and uses the response as an indication that the link is operational. This also gives assurance that traffic is actually flowing over the links. If balance-rr and balance-xor modes are set, then the switch should be configured to evenly distribute packets across all links. Otherwise all replies from the ARP targets will be received on the same link which could cause other links to fail. ARP monitoring is enabled by setting three properties link-monitoring, arp-ip-targets and arp-interval. Meaning of each option is described later in this article. It is possible to specify multiple ARP targets that can be useful in High Availability setups. If only one target is set, the target itself may go down. Having additional targets increases the reliability of the ARP monitoring.
Enable ARP monitoring
[admin@Router1] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.2 [admin@Router2] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.1
We will not change arp-interval value in our example, RouterOS sets arp-interval to 100ms by default.
Unplug one of the cables to test if the link monitoring works correctly, you will notice some ping timeouts until arp monitoring detects link failure.
[admin@Router1] interface bonding> /pi 172.16.0.2 172.16.0.2 ping timeout 172.16.0.2 64 byte ping: ttl=64 time=2 ms 172.16.0.2 ping timeout 172.16.0.2 64 byte ping: ttl=64 time=2 ms 172.16.0.2 ping timeout 172.16.0.2 64 byte ping: ttl=64 time=2 ms 172.16.0.2 64 byte ping: ttl=64 time=2 ms 172.16.0.2 64 byte ping: ttl=64 time=2 ms
Note: For ARP monitoring to work properly it is not required to have any IP address on the device, ARP monitoring will work regardless of the IP address that is set on any interface.
Warning: When ARP monitoring is used, bonding slaves will send out ARP requests without a VLAN tag, even if an IP address is set on a VLAN interface in the same subnet as the arp-ip-targets
MII monitoring
MII monitoring monitors only the state of the local interface. MII Type 1 - device driver determines whether link is up or down. If device driver does not support this option then link will appear as always up. Main disadvantage is that MII monitoring can't tell if the link can actually pass packets or not, even if the link is detected as being up.
MII monitoring is configured by setting the variables link-monitoring mode and mii-interval.
Enable MII Type1 monitoring:
[admin@Router1] interface bonding> set 0 link-monitoring=mii [admin@Router2] interface bonding> set 0 link-monitoring=mii
We will leave mii-interval to it's default value (100ms)
When unplugging one of the cables, the failure will be detected almost instantly compared to ARP link monitoring.
Bonding modes
802.3ad
802.3ad mode is an IEEE standard also called LACP (Link Aggregation Control Protocol). It includes automatic configuration of the aggregates, so minimal configuration of the switch is needed. This standard also mandates that frames will be delivered in order and connections should not see mis-ordering of packets. The standard also mandates that all devices in the aggregate must operate at the same speed and duplex mode.
LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address and if available, the VLAN tag, and the IPv4/IPv6 source and destination address. How this is calculated depends on transmit-hash-policy parameter. The ARP link monitoring is not recommended, because the ARP replies might arrive only on one slave port due to transmit hash policy on the LACP peer device. This can result in unbalanced transmitted traffic, so MII link monitoring is the recommended option.
Note: layer-3-and-4 transmit hash mode is not fully compatible with LACP. More details can be found in https://www.kernel.org/doc/Documentation/networking/bonding.txt
Configuration example
Example connects two ethernet interfaces on a router to the Edimax switch as a single, load balanced and fault tolerant link. More interfaces can be added to increase throughput and fault tolerance. Since frame ordering is mandatory on Ethernet links then any traffic between two devices always flows over the same physical link limiting the maximum speed to that of one interface. The transmit algorithm attempts to use as much information as it can to distinguish different traffic flows and balance across the available interfaces.
Router R1 configuration:
/inteface bonding add slaves=ether1,ether2 mode=802.3ad lacp-rate=30secs link-monitoring=mii \ transmit-hash-policy=layer-2-and-3
Configuration on a switch:
Intelligent Switch : Trunk Configuration ================== 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 M1 M2 1 - v - v - - - - - - - - - - - - - - - - - - - - - - 2 - - - - - - - - - - - - - - - - - - - - - - - - - - 3 - - - - - - - - - - - - - - - - - - - - - - - - - - 4 - - - - - - - - - - - - - - - - - - - - - - - - - - 5 - - - - - - - - - - - - - - - - - - - - - - - - - - 6 - - - - - - - - - - - - - - - - - - - - - - - - - - 7 - - - - - - - - - - - - - - - - - - - - - - - - - - TRK1 LACP TRK2 Disable TRK3 Disable TRK4 Disable TRK5 Disable TRK6 Disable TRK7 Disable
Notice that LACP is enabled on first trunk group (TRK1) and switch ports on first trunk group are bound with 'v' flag. In our case port 2 and port4 will run LACP.
Verify if LACP is working:
On the switch we should first verify if LACP protocol is enabled and running:
Intelligent Switch : LACP Port State Active Configuration ================== Port State Activity Port State Activity --------------------------- --------------------------- 2 Active 4 Active
After that we can ensure that LACP negotiated with our router. If you don't see both ports on the list then something is wrong and LACP is not going to work.
Intelligent Switch : LACP Group Status ================== Group [Actor] [Partner] Priority: 1 65535 MAC : 000E2E2206A9 000C42409426 Port_No Key Priority Active Port_No Key Priority 2 513 1 selected 1 9 255 4 513 1 selected 2 9 255
After we verified that switch successfully negotiated LACP with our router, we can start traffic from Client1 and Client2 to the Server and check how traffic is evenly forwarded through both bonding slaves:
[admin@test-host] /interface> monitor-traffic ether1,ether2,bonding1 rx-packets-per-second: 8158 8120 16278 rx-drops-per-second: 0 0 0 rx-errors-per-second: 0 0 0 rx-bits-per-second: 98.8Mbps 98.2Mbps 197.0Mbps tx-packets-per-second: 4833 4560 9394 tx-drops-per-second: 0 0 0 tx-errors-per-second: 0 0 0 tx-bits-per-second: 2.7Mbps 3.0Mbps 5.8Mbps
Note: On some switches you need to set correct link aggregation protocol, to make balancing work in both directions
balance-rr
If this mode is set, packets are transmitted in sequential order from the first available slave to the last.
Balance-rr is the only mode that will send packets across multiple interfaces that belong to the same TCP/IP connection.
When utilizing multiple sending and multiple receiving links, packets are often received out of order, which result in segment retransmission, for other protocols such as UDP it is not a problem if client software can tolerate out-of-order packets.
If switch is used to aggregate links together, then appropriate switch port configuration is required, however many switches do not support balance-rr.
Quick setup guide demonstrates use of the balance-rr bonding mode. As you can see, it is quite simple to set up. Balance-rr is also useful for bonding several wireless links, however it requires equal bandwidth for all bonded links. If bandwidth of one bonded link drops, then total bandwidth of bond will be equal to the bandwidth of the slowest bonded link.
active-backup
This mode uses only one active slave to transmit packets. The additional slave only becomes active if the primary slave fails. The MAC address of the bonding interface is presented onto the active port to avoid confusing the switch. Active-backup is the best choice in high availability setups with multiple switches that are interconnected.
Note: ARP monitoring in this mode will not work correctly if both routers are directly connected. In such setups mii monitoring must be used or a switch should be put between routers.
balance-xor
This mode balances outgoing traffic across the active ports based on the hashed protocol header information and accepts incoming traffic from any active port. Mode is very similar to LACP except that it is not standardized and works with layer-3-and-4 hash policy.
broadcast
When ports are configured with broadcast mode, all slave ports transmit the same packets to the destination to provide fault tolerance. This mode does not provide load balancing.
balance-tlb
This mode balances outgoing traffic by peer. Each link can be a different speed and duplex mode and no specific switch configuration is required as for the other modes. Downside of this mode is that only MII link monitoring is supported (ARP link monitoring is ignored when configured) and incoming traffic is not balanced. Incoming traffic will use the link that is configured as "primary".
Configuration example
Lets assume than router has two links - ether1 max bandwidth is 10Mbps and ether2 max bandwidth is 5Mbps.
First link has more bandwidth so we set it as primary link
/interface bonding add mode=balance-tlb slaves=ether1,ether2 primary=ether1
No additional configuration is required for the switch.
Image above illustrates how balance-tlb mode works. As you can see router can communicate to all the clients connected to the switch with a total bandwidth of both links (15Mbps). But as you already know, balance-tlb is not balancing incoming traffic. In our example clients can communicate to router with total bandwidth of primary link which is 10Mbps in our configuration.
balance-alb
Mode is basically the same as balance-tlb but incoming IPv4 traffic is also balanced. Only MII link monitoring is supported (ARP link monitoring is ignored when configured), additional downside of this mode is that it requires device driver capability to change MAC address. Most of the cheap cards do not support this mode.
Image above illustrates how balance-alb mode works. Compared to balance-tlb mode, traffic from clients can also use the secondary link to communicate with the router.
Property Description
Property | Description |
---|---|
arp (disabled | enabled | proxy-arp | reply-only; Default: enabled) | Address Resolution Protocol for the interface.
|
arp-interval (time; Default: 00:00:00.100) | Time in milliseconds which defines how often to monitor ARP requests |
arp-ip-targets (IP address; Default: ) | IP target address which will be monitored if link-monitoring is set to arp. You can specify multiple IP addresses, separated by comma |
comment (string; Default: ) | Short description of the interface |
disabled (yes | no; Default: no) | Changes whether the bonding interface is disabled |
down-delay (time; Default: 00:00:00) | If a link failure has been detected, bonding interface is disabled for down-delay time. Value should be a multiple of mii-interval, otherwise it will be rounded down to the nearest value. |
forced-mac-address (MAC address; Default: none) | By default, bonding interface will use MAC address of the first selected slave interface. This property allows to configure static MAC address for the bond interface (all zeros, broadcast or multicast addresses will not apply). RouterOS will automatically change the MAC address for slave interfaces and it will be visible in /interface ethernet configuration export |
lacp-rate (1sec | 30secs; Default: 30secs) | Link Aggregation Control Protocol rate specifies how often to exchange with LACPDUs between bonding peer. Used to determine whether link is up or other changes have occurred in the network. LACP tries to adapt to these changes providing failover. |
link-monitoring (arp | mii | none; Default: mii) | Method to use for monitoring the link (whether it is up or down)
|
min-links (integer: 0..4294967295; Default: 0) | How many active slave links needed for bonding to become active |
mii-interval (time; Default: 00:00:00.100) | How often to monitor the link for failures (parameter used only if link-monitoring is mii) |
mode (802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast; Default: balance-rr) | Specifies one of the bonding policies
|
mtu (integer; Default: 1500) | Maximum Transmit Unit in bytes. Must be smaller or equal to the smallest L2MTU value of a bonding slave. |
name (string; Default: ) | Name of the bonding interface |
primary (string; Default: ) | Controls the primary interface between active slave ports, works only for active-backup, balance-tlb and balance-alb modes. For active-backup mode, it controls which running interface is supposed to send and receive the traffic. For balance-tlb mode, it controls which running interface is supposed to receive all the traffic, but for balance-alb mode, it controls which interface is supposed to receive the unbalanced traffic (the non-IPv4 traffic). When none of the interfaces are selected as primary, device will automatically select the interface that is configured as the first one. |
slaves (string; Default: none) | At least two ethernet-like interfaces separated by a comma, which will be used for bonding |
up-delay (time; Default: 00:00:00) | If a link has been brought up, bonding interface is disabled for up-delay time and after this time it is enabled. Value should be a multiple of mii-interval, otherwise it will be rounded down to the nearest value. |
transmit-hash-policy (layer-2 | layer-2-and-3 | layer-3-and-4; Default: layer-2) | Selects the transmit hash policy to use for slave selection in balance-xor and 802.3ad modes
|
Notes
The rate at which link failure detection and failover takes place can depend on the network adapters used. On Intel cards, for example, failover takes place in less than a second after link loss, while on some other cards, it may require up to 20 seconds. Also, active load balancing (mode=balance-alb
) does not work on some cheap cards.
L2 MTU of a bonding interface is determined by the lowest L2MTU value among its slave interfaces.
See also
[ Top | Back to Content ]