User/Mutual internet backup between two small ISP

From MikroTik Wiki
Jump to: navigation, search

Motivation

Mutual inet backup.png

Assume two small ISP which operate in surrounding area and have own physical link between them. Both network use dynamic routing OSPF with own backbone and have their own upstream internet connection with NATing. As they operate in different area they don't compete to each other and so they have settle on mutual supplying backup internet connection for free for case of failure of their main link or part of their network.

Realization

Starting conditions

Both networks have their own independent OSPF routing table and one default gateway. Both network are addressable form each other thanks to static routing. Internetwork neutral subnet is addressable too as connected subnets are imported to OSFP.

ABR1:

/ip address
add address=172.16.0.1/30 broadcast=172.16.0.3 comment="" disabled=no interface=wlan1 network=172.16.0.0
add address=1.0.0.1/8 broadcast=1.255.255.255 comment="" disabled=no interface=ether1 network=1.0.0.0

/ip route
(imported form OSPF) add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=1.0.0.2 scope=30 target-scope=10
add comment=ISP2 disabled=no distance=1 dst-address=2.0.0.0/8 gateway=172.16.0.2 scope=30 target-scope=10

/routing ospf instance
set default comment="" disabled=no distribute-default=no in-filter=ospf-in metric-bgp=20 \
    metric-connected=20 metric-default=1 metric-other-ospf=auto metric-rip=20 metric-static=20 name=default \
    out-filter=ospf-out redistribute-bgp=no redistribute-connected=as-type-1 redistribute-other-ospf=no redistribute-rip=\
    no redistribute-static=as-type-1 router-id=0.0.0.0

/routing ospf area
set backbone area-id=0.0.0.0 comment="" disabled=no instance=default name=backbone type=default

/routing ospf network
add area=backbone comment="" disabled=no network=1.0.0.0/8

ABR2:

/ip address
add address=172.16.0.2/30 broadcast=172.16.0.3 comment="" disabled=no interface=wlan1 network=172.16.0.0
add address=2.0.0.1/8 broadcast=2.255.255.255 comment="" disabled=no interface=ether1 network=2.0.0.0

/ip route
(imported form OSPF) add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=2.0.0.2 scope=30 target-scope=10
add comment=ISP1 disabled=no distance=1 dst-address=1.0.0.0/8 gateway=172.16.0.1 scope=30 target-scope=10

/routing ospf instance
set default comment="" disabled=no distribute-default=no in-filter=ospf-in metric-bgp=20 \
    metric-connected=20 metric-default=1 metric-other-ospf=auto metric-rip=20 metric-static=20 name=default \
    out-filter=ospf-out redistribute-bgp=no redistribute-connected=as-type-1 redistribute-other-ospf=no redistribute-rip=\
    no redistribute-static=as-type-1 router-id=0.0.0.0

/routing ospf area
set backbone area-id=0.0.0.0 comment="" disabled=no instance=default name=backbone type=default

/routing ospf network
add area=backbone comment="" disabled=no network=2.0.0.0/8


Setup secondary default gateway

As both network use OSPF routing protocol they can get advantage of automatic propagation of multiple default gateways with defined priority. So they have to inject each other primary default gateway as secondary default gateway to own routing table.


  • We need to add to both routers new default gateway with destination to each other. But as static default gateway will take over dynamically imported primary default gateway we need to insert default gateway to different routing table.

ABR1:

/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=172.16.0.2 routing-mark=backup scope=30 target-scope=10

ABR2:

/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=172.16.0.1 routing-mark=backup scope=30 target-scope=10


  • Then we need also to mark right packets using firewall mangle to our new routing table. All packets originating from other network with destination different than local network should be directed to backup routing table.

ABR1:

/ip firewall mangle
add action=mark-routing chain=prerouting comment="" disabled=no new-routing-mark=backup passthrough=no \
   src-address=2.0.0.0/8 dst-address=!1.0.0.0/8

ABR2:

/ip route
add action=mark-routing chain=prerouting comment="" disabled=no new-routing-mark=backup passthrough=no \
   src-address=1.0.0.0/8 dst-address=!2.0.0.0/8


  • Default gateway is not yet imported to OSPF so we need to set OSPF to import default gateway and set higher metric value for example 5000. All default gateways should have to same metric type. If secondary default gateway have type1 and primary type2 than secondary will be elected even if primary would have lower metric.

Both ABR1 and ABR2:

/routing ospf instance
set default comment="" disabled=no distribute-default=if-installed-as-type-2 in-filter=ospf-in metric-bgp=20 \
    metric-connected=20 metric-default=5000 metric-other-ospf=auto metric-rip=20 metric-static=20 name=default \
    out-filter=ospf-out redistribute-bgp=no redistribute-connected=as-type-1 redistribute-other-ospf=no redistribute-rip=\
    no redistribute-static=as-type-1 router-id=0.0.0.0


  • You can check presence of default gateways in OSPF table /routing ospf lsa. You can use traceroute for connection route checking.

Checking if default gateway is working

We have two default gateways now but how we assure to be static default gateways entry properly disable on both routers in case of link failure? RouterOS allow to set check-gateway parameter to automatically check reachability of gateway using ping or arp. Then if gateway is not reachable entry is disabled and OSPF discard entry distributed routing table as parameter redistribute-default-gateway is setted to if-installed. So this can handle two cases if interface is down and if gateway is not responding to ping.

But this could be not enough for real life situation. We need to check some external high available address for reachability and on both routers use only proper gateway.

There more things to take into account. Because we need to disable/enable gateway we make sure that ping test will be able to reach selected host in internet even if default gateway is disabled. So we need setup another routing table with our static default gateway for purpose of ping checking.

Example is presented for single router. In fact all routers which propagate default route should have ping check of internet connection. In our example test should run on ABR1, ABR2, GW1 and GW2.


  • Change Comment of static default route in main routing table to 'Default Route'. This route will be enabled/disabled.


  • Add another routing table with static default route for ping check in case of main default route is redirected to backup route. Ping check should goes always to checked route thus needs static default route.
/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=x.x.x.x \
    routing-mark=StaticDefaultGateway scope=30 target-scope=10

ABR1: gateway=172.16.0.2

ABR2: gateway=172.16.0.1

GW1: gateway=192.168.0.2

GW2: gateway=192.168.1.2


  • Add some high available host address for ping testing. It is better to test multiple internet hosts for reachability. If checked host itself goes down this event could cause false action. Three hosts should be enough for reliable testing. Repeat count of ping can be change too but will extend reaction time.

Checking script in clear form:

:local Repeat 1
:local Count1 [/ping 195.47.235.3  routing-table=StaticDefaultGateway count=$Repeat size=1500];
:local Count2 [/ping 74.125.87.99  routing-table=StaticDefaultGateway count=$Repeat size=1500];
:local Count3 [/ping 77.75.76.3  routing-table=StaticDefaultGateway count=$Repeat size=1500];

# Check if not reachable
:if (([/ip route get [find comment="Default Route"] disabled]=false) \
&& (($Count1 = 0) && ($Count2 = 0) && ($Count3 = 0))) do={
  /ip route set [find comment="Default Route"] disabled=yes
  :log info "Default route down"
}

# Check if reachable
:if (([/ip route get [find comment="Default Route"] disabled]=true) \
&& (($Count1 = $Repeat) || ($Count2 = $Repeat) || ($Count3 = $Repeat))) do={
  /ip route set [find comment="Default Route"] disabled=no
  :log info "Default route up"
}

Same script in exported form:

/system script
add name=DefaultGatewayDown policy=ftp,read,write,policy,test,winbox source=":if (([/ip route get [find commen\
    t=\"Default Route\"] disabled]=false) \\\r\
    \n&& ([/tool netwatch get [find comment=\"Test1\"] status]=\"down\") \\\r\
    \n&& ([/tool netwatch get [find comment=\"Test2\"] status]=\"down\") \\\r\
    \n&& ([/tool netwatch get [find comment=\"Test3\"] status]=\"down\"))  do={\r\
    \n/ip route set [find comment=\"Default Route\"] disabled=yes\r\
    \n:log info \"backup gw down\"\r\
    \n}\r\
    \n\r\
    \n\r\
    \n"
add name=DefaultGatewayUp policy=ftp,read,write,policy,test,winbox source=":if (([/ip route get [find comment=\
    \"Default Route\"] disabled]=true) \\\r\
    \n&& (([/tool netwatch get [find comment=\"Test1\"] status]=\"up\") \\\r\
    \n|| ([/tool netwatch get [find comment=\"Test2\"] status]=\"up\") \\\r\
    \n|| ([/tool netwatch get [find comment=\"Test3\"] status]=\"up\"))) do={\r\
    \n/ip route set [find comment=\"Default Route\"] disabled=no\r\
    \n:log info \"backup gw up\"\
    \n}"
add name=PingCheck policy=read,write,test,sniff source=":local Repeat 1\r\
    \n:local Count1 [/ping 195.47.235.3  routing-table=StaticDefaultGateway count=\$Repeat size=1500];\r\
    \n:local Count2 [/ping 74.125.87.99  routing-table=StaticDefaultGateway count=\$Repeat size=1500];\r\
    \n:local Count3 [/ping 77.75.76.3  routing-table=StaticDefaultGateway count=\$Repeat size=1500];\r\
    \n\r\
    \n# Check if not reachable\r\
    \n:if (([/ip route get [find comment=\"Default Route\"] disabled]=false) \\\r\
    \n&& ((\$Count1 = 0) && (\$Count2 = 0) && (\$Count3 = 0))) do={\r\
    \n  /ip route set [find comment=\"Default Route\"] disabled=yes\r\
    \n  :log info \"Default route down\"\r\
    \n}\r\
    \n\r\
    \n# Check if reachable\r\
    \n:if (([/ip route get [find comment=\"Default Route\"] disabled]=true) \\\r\
    \n&& ((\$Count1 = \$Repeat) || (\$Count2 = \$Repeat) || (\$Count3 = \$Repeat))) do={\r\
    \n  /ip route set [find comment=\"Default Route\"] disabled=no\r\
    \n  :log info \"Default route up\"\r\
    \n}\r\
    \n"


  • Set script for execution by scheduler. Set interval according count of your checked hosts, ping counts per host and timeout.
/system scheduler
add comment="" disabled=no interval=5s name=schedule_default_gw_check on-event="/system script run PingCheck" \
    policy=read,write,test start-date=jan/01/1970 start-time=00:00:00

See also