Difference between revisions of "Manual:Interface/Bonding"

From MikroTik Wiki
Jump to: navigation, search
(See also)
Line 5: Line 5:
 
== Specifications ==
 
== Specifications ==
  
'''Packages required''': system <br/>
+
:'''Packages required''': system <br/>
'''License required''': Level1 <br/>
+
:'''License required''': Level1 <br/>
'''Submenu level''': /interface bonding <br/>
+
:'''Submenu level''': <code>/interface bonding</code> <br/>
'''Standards and Technologies''': None <br/>
+
:'''Standards and Technologies''': None <br/>
'''Hardware usage''': Not significant <br/>
+
:'''Hardware usage''': Not significant <br/>
 +
 
  
 
== Quick Setup Guide ==
 
== Quick Setup Guide ==
Line 90: Line 91:
 
When unplugging one of the cables, notice that failure was detected almost instantly compared to ARP link monitoring.
 
When unplugging one of the cables, notice that failure was detected almost instantly compared to ARP link monitoring.
  
== Property Description ==
 
 
* '''arp''' (disabled | enabled | proxy-arp | reply-only; default: enabled) - Address Resolution Protocol for the interface
 
**disabled - the interface will not use ARP
 
**enabled - the interface will use ARP
 
**proxy-arp - the interface will use the ARP proxy feature
 
**reply-only - the interface will only reply to the requests originated to its own IP addresses. Neighbour MAC addresses will be resolved using /ip arp statically set table only
 
  
*'''arp-interval''' (time; default: 00:00:00.100) - time in milliseconds which defines how often to monitor ARP requests
+
== Bonding modes ==
  
*'''arp-ip-targets''' (IP address; default: "") - IP target address which will be monitored if link-monitoring is set to arp. You can specify multiple IP addresses, separated by comma
 
  
*'''down-delay''' (time; default: 00:00:00) - if a link failure has been detected, bonding interface is disabled for down-delay time. Value should be a multiple of mii-interval
+
== Property Description ==
 
 
*'''lacp-rate''' (1sec | 30secs; default: 30secs) - Link Aggregation Control Protocol rate specifies how often to exchange with LACPDUs between bonding peer. Used to determine whether link is up or other changes have occured in the network. LACP tries to adapt to these changes providing failover.
 
 
 
*'''link-monitoring''' (arp | mii-type1 | mii-type2 | none; default: none) - method to use for monitoring the link (whether it is up or down)
 
**arp - uses Address Resolution Protocol to determine whether the remote interface is reachable
 
**mii-type1 - uses Media Independent Interface type1 to determine link status. Link status determenation relies on the device driver. If bonding shows that the link status is up, when it should not be, then it means that this card don't support this possibility.
 
**mii-type2 - uses MII type2 to determine link status (used if mii-type1 is not supported by the NIC)
 
none - no method for link monitoring is used. If a link fails, it is not considered as down (but no traffic passes through it, thus).
 
 
 
*'''mac-address''' (read-only: MAC address) - MAC address of the bonding interface
 
 
 
*'''mii-interval''' (time; default: 00:00:00.100) - how often to monitor the link for failures (parameter used only if link-monitoring is mii-type1 or mii-type2)
 
 
 
*'''mode''' (802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast; default: balance-rr) - interface bonding mode. Can be one of:
 
**802.3ad - IEEE 802.3ad dynamic link aggregation. In this mode, the interfaces are aggregated in a group where each slave shares the same speed. If you use a switch between 2 bonding routers, be sure that this switch supports IEEE 802.3ad standard. Provides fault tolerance and load balancing.
 
**active-backup - provides link backup. Only one slave can be active at a time. Another slave becomes active only, if first one fails.
 
**balance-alb - adaptive load balancing. It includes balance-tlb and received traffic is also balanced. Device driver should support for setting the mac address, then it is active. Otherwise balance-alb doesn't work. No special switch is required.
 
**balance-rr - round-robin load balancing. Slaves in bonding interface will transmit and receive data in sequential order. Provides load balancing and fault tolerance.
 
**balance-tlb - Outgoing traffic is distributed according to the current load on each slave. Incoming traffic is received by the current slave. If receiving slave fails, then another slave takes the MAC address of the failed slave. Doesn't require any special switch support.
 
**balance-xor - Use XOR policy for transmit. Provides only failover (in very good quality), but not load balancing, yet.
 
**broadcast - Broadcasts the same data on all interfaces at once. This provides fault tolerance but slows down traffic throughput on some slow machines.
 
 
 
*'''mtu''' (integer: 68..1500; default: 1500) - Maximum Transmit Unit in bytes
 
 
 
*'''name''' (name) - descriptive name of bonding interface
 
 
 
*'''primary''' (name; default: none) - Interface is used as primary output media. If primary interface fails, only then others slaves will be used. This value works only with mode=active-backup
 
 
 
*'''slaves''' (name) - at least two ethernet-like interfaces separated by a comma, which will be used for bonding
 
  
*'''up-delay''' (time; default: 00:00:00) - if a link has been brought up, bonding interface is disabled for up-delay time and after this time it is enabled. Value should be a multiple of mii-interval
+
<table class="styled_table">
 +
<tr>
 +
  <th width="350">Property</th>
 +
  <th >Description</th>
 +
</tr>
 +
<tr>
 +
    <td><var><b>arp</b></var> (<em>disabled | enabled | proxy-arp | reply-only</em>; Default: <b>enabled</b>)</td>
 +
    <td> Address Resolution Protocol for the interface.
 +
:<var>disabled</var> - the interface will not use ARP
 +
:<var>enabled</var> - the interface will use ARP
 +
:<var>proxy-arp</var> - the interface will use the ARP proxy feature
 +
:<var>reply-only</var> - the interface will only reply to the requests originated to its own IP addresses. Neighbour MAC addresses will be resolved using /ip arp statically set table only </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>arp-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td>
 +
    <td> time in milliseconds which defines how often to monitor ARP requests </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>arp-ip-targets</b></var> (<em>IP addres</em>; Default: <b></b>)</td>
 +
    <td> IP target address which will be monitored if <var>link-monitoring</var> is set to arp. You can specify multiple IP addresses, separated by comma </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>down-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td>
 +
    <td> if a link failure has been detected, bonding interface is disabled for down-delay time. Value should be a multiple of <var>mii-interval</var> </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>lacp-rate</b></var> (<em>1sec | 30secs</em>; Default: <b>30secs</b>)</td>
 +
    <td> Link Aggregation Control Protocol rate specifies how often to exchange with LACPDUs between bonding peer. Used to determine whether link is up or other changes have occurred in the network. LACP tries to adapt to these changes providing failover.</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>link-monitoring</b></var> (<em>arp | mii-type1 | mii-type2 | none</em>; Default: <b>none</b>)</td>
 +
    <td> method to use for monitoring the link (whether it is up or down)
 +
:<var>arp</var> - uses Address Resolution Protocol to determine whether the remote interface is reachable
 +
:<var>mii-type1</var> - uses Media Independent Interface type1 to determine link status. Link status determenation relies on the device driver
 +
:<var>mii-type2</var> - similar as mii-type1, but status determination does not rely on the device driver
 +
:<var>none</var> - no method for link monitoring is used.
 +
<b>Note:</b> some bonding modes require specific link monitoring to work properly.</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>mii-interval</b></var> (<em>time</em>; Default: <b>00:00:00.100</b>)</td>
 +
    <td> how often to monitor the link for failures (parameter used only if <var>link-monitoring</var> is mii-type1 or mii-type2)</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>mode</b></var> (<em>802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast</em>; Default: <b>balance-rr</b>)</td>
 +
    <td> Specifies one of the bonding policies
 +
:<var>802.3ad</var> - IEEE 802.3ad dynamic link aggregation. In this mode, the interfaces are aggregated in a group where each slave shares the same speed. Provides fault tolerance and load balancing. Slave selection for outgoing traffic is done according to the <var>transmit-hash-policy</var>
 +
:<var>active-backup</var> - provides link backup. Only one slave can be active at a time. Another slave becomes active only, if first one fails.
 +
:<var>balance-alb</var> - adaptive load balancing. It includes <var>balance-tlb</var> and received traffic is also balanced. Device driver should support for setting the mac address
 +
:<var>balance-rr</var> - round-robin load balancing. Slaves in bonding interface will transmit and receive data in sequential order. Provides load balancing and fault tolerance.
 +
:<var>balance-tlb</var> - Outgoing traffic is distributed according to the current load on each slave. Incoming traffic is received by the current slave. If receiving slave fails, then another slave takes the MAC address of the failed slave.
 +
:<var>balance-xor</var> - Transmit based on the selected <var>transmit-hash-policy</var>. This mode provides load balancing and fault tolerance.
 +
:<var>broadcast</var> - Broadcasts the same data on all interfaces at once. This provides fault tolerance but slows down traffic throughput on some slow machines.</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>mtu</b></var> (<em>integer</em>; Default: <b>1500</b>)</td>
 +
    <td> Maximum Transmit Unit in bytes </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>name</b></var> (<em>string</em>; Default: <b></b>)</td>
 +
    <td> descriptive name of bonding interface </td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>primary</b></var> (<em>string</em>; Default: <b></b>)</td>
 +
    <td> Interface is used as primary output interface. If primary interface fails, only then others slaves will be used. This value works only with active-backup mode</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>slaves</b></var> (<em>string</em>; Default: <b>none</b>)</td>
 +
    <td> at least two ethernet-like interfaces separated by a comma, which will be used for bonding</td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>up-delay</b></var> (<em>time</em>; Default: <b>00:00:00</b>)</td>
 +
    <td> if a link has been brought up, bonding interface is disabled for up-delay time and after this time it is enabled. Value should be a multiple of <var>mii-interval</var></td>
 +
</tr>
 +
<tr>
 +
    <td><var><b>transmit-hash-policy</b></var> (<em>layer-2 | layer-2-and-3 | layer-3-and-4</em>; Default: <b>layer-2</b>)</td>
 +
    <td> Selects the transmit hash policy to use for slave selection in balance-xor and 802.3ad modes
 +
:<var>layer-2</var> - Uses XOR of hardware MAC addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. This algorithm is 802.3ad compliant.
 +
:<var>layer-2-and-3</var> - This policy uses a combination of layer2 and layer3 protocol information to generate the hash. Uses XOR of hardware MAC addresses and IP addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave.  For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations. This algorithm is 802.3ad compliant.
 +
:<var>layer-3-and-4</var> - This policy uses upper layer protocol information, when available, to generate the hash.  This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves. For fragmented TCP or UDP packets and all other IP protocol traffic, the source and destination port information is omitted.  For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This algorithm is not fully 802.3ad compliant.</td>
 +
</tr>
 +
</table>
  
*'''transmit-hash-policy''' ( layer-2 | layer-2-and-3 | layer-3-and-4 ) - used in modes ''balance-xor'' and ''802.3ad'':
 
**layer-2 - Uses XOR of hardware MAC addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. This algorithm is 802.3ad compliant.
 
**layer-2-and-3 - This policy uses a combination of layer2 and layer3 protocol information to generate the hash. Uses XOR of hardware MAC addresses and IP addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave.  For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations. This algorithm is 802.3ad compliant.
 
**layer-3-and-4  - This policy uses upper layer protocol information, when available, to generate the hash.  This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves. For fragmented TCP or UDP packets and all other IP protocol traffic, the source and destination port information is omitted.  For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to mimic the behavior of certain switches, notably Cisco switches with PFC2 as well as some Foundry and IBM products. This algorithm is not fully 802.3ad compliant.  A single TCP or UDP conversation containing both fragmented and unfragmented packets will see packets striped across two interfaces.  This may result in out of order delivery.  Most traffic types will not meet this criteria, as TCP rarely  segments traffic, and most UDP traffic is not involved in extendedconversations.  Other implementations of 802.3ad mayor may not tolerate this noncompliance.
 
  
 
== Notes ==
 
== Notes ==

Revision as of 11:54, 30 April 2009

Summary

Bonding is a technology that allows to aggregate multiple ethernet-like interfaces into a single virtual link, thus getting higher data rates and providing failover.

Specifications

Packages required: system
License required: Level1
Submenu level: /interface bonding
Standards and Technologies: None
Hardware usage: Not significant


Quick Setup Guide

Let us assume that we have 2 NICs in each router (Router1 and Router2) and want to get maximum data rate between 2 routers. To make this possible, follow these steps:

  • Make sure that you do not have IP addresses on interfaces which will be enslaved for bonding interface!
  • Add bonding interface on Router1:
[admin@Router1] interface bonding> add slaves=ether1,ether2

And on Router2:

[admin@Router2] interface bonding> add slaves=ether1,ether2

Add addresses to bonding interfaces:

[admin@Router1] ip address> add address=172.16.0.1/24 interface=bonding1
[admin@Router2] ip address> add address=172.16.0.2/24 interface=bonding1

Test the link from Router1:

     [admin@Router1] interface bonding> /pi 172.16.0.2
     172.16.0.2 ping timeout
     172.16.0.2 ping timeout
     172.16.0.2 ping timeout
     172.16.0.2 64 byte ping: ttl=64 time=2 ms
     172.16.0.2 64 byte ping: ttl=64 time=2 ms

Note that bonding interface needs a couple of seconds to get connectivity with its peer.


Link monitoring

It is critical that one of available link monitoring options are enabled. In example above if one of the bonded links fail, bonding driver will still continue to send packets over failed link which will lead to network degradation. Currently bonding in RouterOS supports two schemes for monitoring a link state of slave devices: MII and ARP monitoring. It is not possible to use both methods at a time due to restrictions in the bonding driver.

ARP Monitoring

ARP monitoring sends ARP queries and uses the response as an indication that the link is operational. This also gives assurance that traffic is actually flowing over the links. If balance-rr and balance-xor modes are set, then the switch should be configured to evenly distribute packets across all links. Otherwise all replies from the ARP targets will be received on the same link which could cause other links to fail. ARP monitoring is enabled by setting three properties link-monitoring, arp-ip-targets and arp-interval. Meaning of each option is described later in this article. It is possible to specify multiple ARP targets that can be useful in a High Availability setups. If only one target is set, the target itself may go down. Having an additional targets increases the reliability of the ARP monitoring.

Enable ARP monitoring

[admin@Router1] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.2
[admin@Router2] interface bonding> set 0 link-monitoring=arp arp-ip-targets=172.16.0.1

We will not change arp-interval value in our example, RouterOS sets arp-interval to 100ms by default.

Unplug one of the cables to test if link monitoring works correctly, you will notice some ping timeouts until arp monitoring detects link failure.

     [admin@Router1] interface bonding> /pi 172.16.0.2
     172.16.0.2 ping timeout
     172.16.0.2 64 byte ping: ttl=64 time=2 ms
     172.16.0.2 ping timeout
     172.16.0.2 64 byte ping: ttl=64 time=2 ms
     172.16.0.2 ping timeout
     172.16.0.2 64 byte ping: ttl=64 time=2 ms
     172.16.0.2 64 byte ping: ttl=64 time=2 ms
     172.16.0.2 64 byte ping: ttl=64 time=2 ms

MII monitoring

MII monitoring monitors only the state of the local interface. In RouterOS it is possible to configure MII monitoring in two ways:

  • MII Type 1 - device driver determines whether link is up or down. If device driver does not support this option then link will appear as always up.
  • MII Type 2 - deprecated calling sequences within the kernel are used to determine if link is up. This method is less efficient but can be used on all devices. This mode should be set only if MII type 1 is not supported.

Main disadvantage is that MII monitoring can't tell if the link actually can pass the packets or not even if the link is detected as up.

MII monitoring is configured setting desired link-monitoring mode and mii-interval.

Enable MII Type2 monitoring:

[admin@Router1] interface bonding> set 0 link-monitoring=mii-type-2
[admin@Router2] interface bonding> set 0 link-monitoring=mii-type-2

We will leave mii-interval to it's default value (100ms)

When unplugging one of the cables, notice that failure was detected almost instantly compared to ARP link monitoring.


Bonding modes

Property Description

Property Description
arp (disabled | enabled | proxy-arp | reply-only; Default: enabled) Address Resolution Protocol for the interface.
disabled - the interface will not use ARP
enabled - the interface will use ARP
proxy-arp - the interface will use the ARP proxy feature
reply-only - the interface will only reply to the requests originated to its own IP addresses. Neighbour MAC addresses will be resolved using /ip arp statically set table only
arp-interval (time; Default: 00:00:00.100) time in milliseconds which defines how often to monitor ARP requests
arp-ip-targets (IP addres; Default: ) IP target address which will be monitored if link-monitoring is set to arp. You can specify multiple IP addresses, separated by comma
down-delay (time; Default: 00:00:00) if a link failure has been detected, bonding interface is disabled for down-delay time. Value should be a multiple of mii-interval
lacp-rate (1sec | 30secs; Default: 30secs) Link Aggregation Control Protocol rate specifies how often to exchange with LACPDUs between bonding peer. Used to determine whether link is up or other changes have occurred in the network. LACP tries to adapt to these changes providing failover.
link-monitoring (arp | mii-type1 | mii-type2 | none; Default: none) method to use for monitoring the link (whether it is up or down)
arp - uses Address Resolution Protocol to determine whether the remote interface is reachable
mii-type1 - uses Media Independent Interface type1 to determine link status. Link status determenation relies on the device driver
mii-type2 - similar as mii-type1, but status determination does not rely on the device driver
none - no method for link monitoring is used.
Note: some bonding modes require specific link monitoring to work properly.
mii-interval (time; Default: 00:00:00.100) how often to monitor the link for failures (parameter used only if link-monitoring is mii-type1 or mii-type2)
mode (802.3ad | active-backup | balance-alb | balance-rr | balance-tlb | balance-xor | broadcast; Default: balance-rr) Specifies one of the bonding policies
802.3ad - IEEE 802.3ad dynamic link aggregation. In this mode, the interfaces are aggregated in a group where each slave shares the same speed. Provides fault tolerance and load balancing. Slave selection for outgoing traffic is done according to the transmit-hash-policy
active-backup - provides link backup. Only one slave can be active at a time. Another slave becomes active only, if first one fails.
balance-alb - adaptive load balancing. It includes balance-tlb and received traffic is also balanced. Device driver should support for setting the mac address
balance-rr - round-robin load balancing. Slaves in bonding interface will transmit and receive data in sequential order. Provides load balancing and fault tolerance.
balance-tlb - Outgoing traffic is distributed according to the current load on each slave. Incoming traffic is received by the current slave. If receiving slave fails, then another slave takes the MAC address of the failed slave.
balance-xor - Transmit based on the selected transmit-hash-policy. This mode provides load balancing and fault tolerance.
broadcast - Broadcasts the same data on all interfaces at once. This provides fault tolerance but slows down traffic throughput on some slow machines.
mtu (integer; Default: 1500) Maximum Transmit Unit in bytes
name (string; Default: ) descriptive name of bonding interface
primary (string; Default: ) Interface is used as primary output interface. If primary interface fails, only then others slaves will be used. This value works only with active-backup mode
slaves (string; Default: none) at least two ethernet-like interfaces separated by a comma, which will be used for bonding
up-delay (time; Default: 00:00:00) if a link has been brought up, bonding interface is disabled for up-delay time and after this time it is enabled. Value should be a multiple of mii-interval
transmit-hash-policy (layer-2 | layer-2-and-3 | layer-3-and-4; Default: layer-2) Selects the transmit hash policy to use for slave selection in balance-xor and 802.3ad modes
layer-2 - Uses XOR of hardware MAC addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. This algorithm is 802.3ad compliant.
layer-2-and-3 - This policy uses a combination of layer2 and layer3 protocol information to generate the hash. Uses XOR of hardware MAC addresses and IP addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave. For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This policy is intended to provide a more balanced distribution of traffic than layer2 alone, especially in environments where a layer3 gateway device is required to reach most destinations. This algorithm is 802.3ad compliant.
layer-3-and-4 - This policy uses upper layer protocol information, when available, to generate the hash. This allows for traffic to a particular network peer to span multiple slaves, although a single connection will not span multiple slaves. For fragmented TCP or UDP packets and all other IP protocol traffic, the source and destination port information is omitted. For non-IP traffic, the formula is the same as for the layer2 transmit hash policy. This algorithm is not fully 802.3ad compliant.


Notes

Link failure detection and failover is working significantly better with expensive network cards, for example, made by Intel, then with more cheap ones. For example, on Intel cards failover is taking place in less than a second after link loss, while on some other cards, it may require up to 20 seconds. Also, the Active load balancing (mode=balance-alb) does not work on some cheap cards.

See also