Difference between revisions of "Manual:Virtual Routing and Forwarding"

From MikroTik Wiki
Jump to: navigation, search
(New page: ''Preliminary version - will be expanded!'' == Description == New '''routing-test''' package in version 3.x allows to create multiple Virtual Routing and Forwarding instances on a single...)
 
m (add note about dhcp-server)
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
''Preliminary version - will be expanded!''
+
{{Versions|3, v4+}}
  
== Description ==
+
''Packages required: '''routing-test''', '''mpls-test''' for RouterOS v3;  '''routing''', '''mpls''' for  RouterOS v4+''
  
New '''routing-test''' package in version 3.x allows to create multiple Virtual Routing and Forwarding instances on a single router. This is useful for BGP based MPLS VPNs. Unlike MPLS BGP VPLS, which is OSI Layer 2 technology, BGP VPNs works in Layer 3 and as such exchanges IP prefixes between routers. VRFs solves the problem of overlapping IP prefixes, and provides the required privacy (via separated routing for different VPNs).
+
= Description =
  
To create a VRF, configure it under '''/ip route vrf'''. You can now add routes to that VRF - simply specify '''routing-mark''' attribute. Connected routes from interfaces belonging to a VRF will be installed in right routing table automatically.
+
RouterOS 3.x allows to create multiple Virtual Routing and Forwarding instances on a single router. This is useful for BGP based MPLS VPNs. Unlike BGP VPLS, which is OSI Layer 2 technology, BGP VRF VPNs work in Layer 3 and as such exchange IP prefixes between routers. VRFs solve the problem of overlapping IP prefixes, and provide the required privacy (via separated routing for different VPNs).
  
Technically VRFs are based on policy routing. There is exactly one policy routing table for each active VRF. Note that existing policy routing support will not be changed but you will not be able to have policy routing ''within'''a VRF. The main difference between VRF tables and simple policy routing is that routes in VRF tables resolve nexthops in their own routing table by default, while policy routes always use main routing table. Read-only route attribute '''gateway-table''' displays information about which table is used
+
To create a VRF, configure it under '''/ip route vrf'''. You can now add routes to that VRF - simply specify '''routing-mark''' attribute. Connected routes from interfaces belonging to a VRF will be installed in the right routing table automatically.
for a particular route (default is '''main''').
 
  
The real fun begins when you start to configure BGP. You can use multiprotocol BGP to distribute those routes - not only to other routers, but also to different routing tables in the router itself. Route installation in VRF tables is controlled by BGP extended communities attributes. Configure import and export lists under '''/ip route vrf''', '''import-route-target''' and '''export-route-target'''. Export route target list for a VRF should contained at least the route distinguisher for that VRF.
+
Technically VRFs are based on policy routing. There is exactly one policy route table for each active VRF. The existing policy routing support in MT RouterOS is not changed; but on the other hand, it is not possible to have policy routing ''within'' a VRF. The main differences between VRF tables and simple policy routing are:
 +
* Routes in VRF tables resolve next-hops in their own route table by default, while policy routes always use the main route table. Read-only route attribute '''gateway-table''' displays information about which table is used for a particular route (default is '''main''').
 +
* Route lookup is different. For policy routing: after route lookup has been done in policy-route table, and no route was found, route lookup proceeds to the main route table. For VRFs: if lookup is done, and no route is found in VRF route table, the lookup fails with "network unreachable" error. (You can still override this behavior with custom route lookup rules, as they have precedence.)
  
Active multiprotocol BGP routes are installed in a separate routing table, which can be observed at '''/routing bgp vpnv4-route'''. These so called VPNv4 routes has prefix that consists of '''route-distinguisher''' and an IPv4 route prefix. This way you can have overlapping IPv4 prefixes distributed in BGP. '''route-distinguisher''' can be configured under '''/ip route vrf'''. There should be one-to-one correspondence between route-distinguishers and VRFs.
+
{{ Note | ''When a DHCP-Relay server is attached to an interface in a vrf, the communications from that DHCP-Relay to the remote DHCP-Server will not be routed via the vrf!'' }}
  
Please not that a VPNv4 route will be distributed only if it has a valid MPLS label. You need to install '''mpls''' package and configure valid label range (default is OK) for this to work.
 
  
== An example with Cisco ==
+
You can use multi-protocol BGP with VPNv4 address family to distribute routes from VRF route tables - not only to other routers, but also to different routing tables in the router itself. First configure the route distinguisher for a VRF. It can be done under '''/ip route vrf'''. Usually there will be one-to-one correspondence between route distinguishers and VRFs, but that's not a mandatory requirement. Route installation in VRF tables is controlled by BGP extended communities attribute. Configure import and export lists under '''/ip route vrf''', '''import-route-targets''' and '''export-route-targets'''. Export route target list for a VRF should contained at least the route distinguisher for that VRF. Then configure a list of VRFs for each BGP instance that will participate in VRF routing.
  
In this example we create two VPNs for cust-one and cust-two and exchange all routes between them.
+
Once list of VRFs for BGP instance, route distinguisher and export route targets has been configured, some active VPNv4 address family routes may be created, depending on BGP redistribution settings. They are installed in a separate route table and, if present, visible under '''/routing bgp vpnv4-route'''. These so called VPNv4 routes have prefix that consists of a route distinguisher and an IPv4 network prefix. This way you can have overlapping IPv4 prefixes distributed in BGP.
  
[[Image:VRF.png]]
+
Please note that a VPNv4 route will be distributed only if it has a valid MPLS label. You need to install '''mpls-test''' package and configure valid label range for this to work. (Default configuration has valid label range.)
  
=== Configuration  ===
+
= Examples =
  
'''Mikrotik''': (note that VRF names are not important, they are the same only by convention)
+
== The simplest MPLS VPN setup ==
  
Addresses are like this:
+
[[Image:l3vpn-simple.png|740px]]
[admin@MikroTik] > /ip address p
 
Flags: X - disabled, I - invalid, D - dynamic
 
  #  ADDRESS            NETWORK        BROADCAST      INTERFACE
 
  0  10.0.0.131/24      10.0.0.0        10.0.0.255      ether1
 
  1  1.1.1.1/24        1.1.1.0        1.1.1.255      ether2
 
  
Add default route
+
In this example rudimentary MPLS backbone (consisting of two Provider Edge (PE) routers PE1 and PE2) is created and configured to forward traffic between Customer Edge (CE) routers CE1 and CE2
/ip route add gateway=10.0.0.1
+
routers that belong to ''cust-one'' VPN.
  
Configure VRFs:
+
=== CE1 Router ===  
/ip route vrf add interfaces=ether2 route-distinguisher=1.1.1.1:111 export-route-targets=1.1.1.1:111 \
 
    import-route-targets=1.1.1.1:111,2.2.2.2:222 routing-mark=cust-one
 
/ip route vrf add interfaces=ether3 route-distinguisher=2.2.2.2:222 export-route-targets=2.2.2.2:222 \
 
    import-route-targets=1.1.1.1:111,2.2.2.2:222 routing-mark=cust-two
 
  
Configure VPNv4 redistribution and multiprotocol BGP:
+
  /ip address add address=10.1.1.1/24 interface=ether1
  /routing bgp instance set default as=64550 redistribute-connected=yes vrf=cust-one,cust-two
+
# use static routing
  /routing bgp peer add remote-address=10.0.11.202 remote-as=64550 instance=default address-families=vpnv4
+
  /ip route add dst-address=10.3.3.0/24 gateway=10.1.1.2
  
'''Cisco''':
+
=== CE2 Router ===
interface FastEthernet0/0
 
  ip address 10.0.11.202 255.255.255.0
 
ip vrf cust-one
 
  rd 1.1.1.1:111
 
  route-target export 1.1.1.1:111
 
  route-target import 1.1.1.1:111
 
  route-target import 2.2.2.2:222
 
  exit
 
ip vrf cust-two
 
  rd 2.2.2.2:222
 
  route-target export 2.2.2.2:222
 
  route-target import 1.1.1.1:111
 
  route-target import 2.2.2.2:222
 
  exit
 
interface Tunnel1
 
  ip vrf forwarding cust-one
 
  ip address 2.2.2.2 255.255.255.0
 
  tunnel source 10.0.11.202
 
  tunnel destination X.X.X.X
 
interface Tunnel2
 
  ip vrf forwarding cust-two
 
  ip address 3.3.3.3 255.255.255.0
 
  tunnel source 10.0.11.202
 
  tunnel destination X.X.X.X
 
router bgp 64550
 
  neighbor 10.0.0.131 remote-as 64550
 
  address-family vpnv4
 
  neighbor 10.0.0.131 activate
 
  neighbor 10.0.0.131 send-community both
 
  exit-address-family
 
  address-family ipv4 vrf cust-one
 
  redistribute connected
 
  exit-address-family
 
  address-family ipv4 vrf cust-two
 
  redistribute connected
 
  exit-address-family
 
  
 +
/ip address add address=10.3.3.4/24 interface=ether1
 +
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3
 +
 +
=== PE1 Router ===
 +
 +
/interface bridge add name=lobridge
 +
/ip address add address=10.1.1.2/24 interface=ether1
 +
/ip address add address=10.2.2.2/24 interface=ether2
 +
/ip address add address=10.5.5.2/32 interface=lobridge
 +
/ip route vrf add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
 +
    export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111 interfaces=ether1
 +
/mpls ldp set enabled=yes transport-address=10.5.5.2
 +
/mpls ldp interface add interface=ether2
 +
/routing bgp instance set default as=65000
 +
/routing bgp instance vrf add instance=default routing-mark=cust-one redistribute-connected=yes
 +
/routing bgp peer add remote-address=10.5.5.3 remote-as=65000 address-families=vpnv4 \
 +
    update-source=lobridge
 +
  # add route to the remote BGP peer's loopback address
 +
/ip route add dst-address=10.5.5.3/32 gateway=10.2.2.3
 +
 +
=== PE2 Router (Cisco) ===
 +
<pre>
 +
ip vrf cust-one
 +
rd 1.1.1.1:111
 +
route-target export 1.1.1.1:111
 +
route-target import 1.1.1.1:111
 +
exit
 +
 +
interface Loopback0
 +
ip address 10.5.5.3 255.255.255.255
 +
 +
mpls ldp router-id Loopback0 force
 +
mpls label protocol ldp
 +
 +
interface FastEthernet0/0
 +
ip address 10.2.2.3 255.255.255.0
 +
mpls ip
 +
 +
interface FastEthernet1/0
 +
ip vrf forwarding cust-one
 +
ip address 10.3.3.3 255.255.255.0
 +
 +
router bgp 65000
 +
neighbor 10.5.5.2 remote-as 65000
 +
neighbor 10.5.5.2 update-source Loopback0
 +
address-family vpnv4
 +
  neighbor 10.5.5.2 activate
 +
  neighbor 10.5.5.2 send-community both
 +
  exit-address-family
 +
address-family ipv4 vrf cust-one
 +
  redistribute connected
 +
  exit-address-family
 +
 +
ip route 10.5.5.2 255.255.255.255 10.2.2.2
 +
</pre>
 
=== Results ===
 
=== Results ===
  
'''Cisco''':
+
Check that VPNv4 route redistribution is working:
C7200#show ip bgp vpnv4 all
+
<pre>
BGP table version is 20, local router ID is 10.0.11.202
+
[admin@PE1] > /routing bgp vpnv4-route print detail
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
+
Flags: L - label present
              r RIB-failure, S Stale
+
  0 L route-distinguisher=1.1.1.1:111 dst-address=10.3.3.0/24 gateway=10.5.5.3
  Origin codes: i - IGP, e - EGP, ? - incomplete
+
    interface=ether2 in-label=17 out-label=17 bgp-local-pref=100 bgp-med=0
    Network          Next Hop            Metric LocPrf Weight Path
+
    bgp-origin=incomplete bgp-ext-communities="RT:1.1.1.1:111"
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one)
+
 
*>i1.1.1.0/24       10.0.0.131                    100     0 ?
+
  1 L route-distinguisher=1.1.1.1:111 dst-address=10.1.1.0/24 interface=ether1
*> 2.2.2.0/24      0.0.0.0                  0        32768 ?
+
    in-label=16 bgp-ext-communities="RT:1.1.1.1:111"
*> 3.3.3.0/24      0.0.0.0                  0        32768 ?
+
</pre>
  Route Distinguisher: 2.2.2.2:222 (default for vrf cust-two)
+
Check that the 10.3.3.0 is installed in IP routes, in cust-one route table:
*>i1.1.1.0/24       10.0.0.131                    100      0 ?
 
*> 2.2.2.0/24      0.0.0.0                  0        32768 ?
 
*> 3.3.3.0/24      0.0.0.0                  0        32768 ?
 
  
  C7200#show ip route vrf cust-one
+
  [admin@PE1] > /ip route print
Routing Table: cust-one
+
Flags: X - disabled, A - active, D - dynamic,
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
+
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
 +
B - blackhole, U - unreachable, P - prohibit
 +
  #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 +
  0 ADC  10.1.1.0/24        10.1.1.2        ether1            0
 +
  1 ADb  10.3.3.0/24                        10.5.5.3 recursi... 20
 +
  2 ADC  10.2.2.0/24        10.2.2.2        ether2            0
 +
  3 ADC  10.5.5.2/32        10.5.5.2        lobridge          0
 +
  4 A S  10.5.5.3/32                        10.2.2.3 reachab... 1
 +
 
 +
Let's take closer look at IP routes in cust-one VRF.
 +
The 10.1.1.0/24 IP prefix is a connected route that belongs to an interface that was configured to belong to cust-one VRF.
 +
The 10.3.3.0/24 IP prefix was advertised via BGP as VPNv4 route from PE2 and is imported in this VRF routing table, because our configured '''import-route-targets''' matched the BGP extended communities attribute it was advertised with.
 +
<pre>
 +
[admin@PE1] /ip route> print detail where routing-mark=cust-one
 +
Flags: X - disabled, A - active, D - dynamic,
 +
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
 +
B - blackhole, U - unreachable, P - prohibit
 +
0 ADC  dst-address=10.1.1.0/24 pref-src=10.1.1.2 gateway=ether1 distance=0 scope=10
 +
        routing-mark=cust-one
 +
 
 +
1 ADb  dst-address=10.3.3.0/24 gateway=10.5.5.3 recursive via 10.2.2.3 ether2
 +
        distance=20 scope=40 target-scope=30 routing-mark=cust-one
 +
        bgp-local-pref=100 bgp-origin=incomplete
 +
        bgp-ext-communities="RT:1.1.1.1:111"
 +
</pre>
 +
The same for Cisco:
 +
<pre>
 +
PE2#show ip bgp vpnv4 all
 +
BGP table version is 5, local router ID is 10.5.5.3
 +
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
 +
              r RIB-failure, S Stale
 +
Origin codes: i - IGP, e - EGP, ? - incomplete
 +
 
 +
  Network          Next Hop            Metric LocPrf Weight Path
 +
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one)
 +
*>i10.1.1.0/24      10.5.5.2                      100      0 ?
 +
*> 10.3.3.0/24      0.0.0.0                  0        32768 ?
 +
 
 +
PE2#show ip route vrf cust-one
 +
 
 +
Routing Table: cust-one
 +
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
 
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
Line 111: Line 159:
 
       ia - IS-IS inter area, * - candidate default, U - per-user static route
 
       ia - IS-IS inter area, * - candidate default, U - per-user static route
 
       o - ODR, P - periodic downloaded static route
 
       o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
 
    1.0.0.0/24 is subnetted, 1 subnets
 
B      1.1.1.0 [200/0] via 10.0.0.131, 00:02:51
 
    2.0.0.0/24 is subnetted, 1 subnets
 
C      2.2.2.0 is directly connected, Tunnel1
 
    3.0.0.0/24 is subnetted, 1 subnets
 
B      3.3.3.0 is directly connected, 00:01:20, Tunnel2
 
  
'''Mikrotik''':
+
Gateway of last resort is not set
[admin@MikroTik] /routing bgp> vpnv4-route print detail
+
 
Flags: N - no label
+
    10.0.0.0/24 is subnetted, 1 subnets
  0   route-distinguisher=1.1.1.1:111 dst-address=2.2.2.0/24 gateway=10.0.11.202
+
B      10.1.1.0 [200/0] via 10.5.5.2, 00:05:33
      interface=ether1 in-label=17 out-label=16 bgp-local-pref=100 bgp-med=0
+
    10.0.0.0/24 is subnetted, 1 subnets
      bgp-origin=incomplete bgp-ext-communities="RT:1.1.1.1:111"
+
C      10.3.3.0 is directly connected, FastEthernet1/0
  1  route-distinguisher=2.2.2.2:222 dst-address=3.3.3.0/24 gateway=10.0.11.202
+
</pre>
      interface=ether1 in-label=18 out-label=17 bgp-local-pref=100 bgp-med=0
+
You should be able to ping from CE1 to CE2 and vice versa.
      bgp-origin=incomplete bgp-ext-communities="RT:2.2.2.2:222"
+
 
  2  route-distinguisher=1.1.1.1:111 dst-address=1.1.1.0/24 interface=ether2
+
[admin@CE1] > /ping 10.3.3.4
      in-label=16 bgp-ext-communities="RT:1.1.1.1:111"
+
10.3.3.4 64 byte ping: ttl=62 time=18 ms
 +
10.3.3.4 64 byte ping: ttl=62 time=13 ms
 +
10.3.3.4 64 byte ping: ttl=62 time=13 ms
 +
10.3.3.4 64 byte ping: ttl=62 time=14 ms
 +
4 packets transmitted, 4 packets received, 0% packet loss
 +
round-trip min/avg/max = 13/14.5/18 ms
 +
 
 +
== A more complicated setup (changes only) ==
 +
 
 +
[[Image:l3vpn-two-customers.png|740px]]
 +
 
 +
As opposed to the simplest setup, in this example we have two customers: cust-one and cust-two.
  
[admin@MikroTik] /routing bgp> /ip route p detail
+
We configure two VPNs for then, cust-one and cust-two respectively, and exchange all routes between them. (This is also called "route leaking").
Flags: X - disabled, A - active, D - dynamic,
+
 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
+
Note that this could be not the most typical setup, because routes are usually not exchanged between different customers. In contrast, by default it should not be possible to gain access from one VRF site to a different VRF site in another VPN. (This is the "Private" aspect of VPNs.) Separate routing is a way to provide privacy; and it is also required to solve the problem of overlapping IP network prefixes. Route exchange is in direct conflict with these two requirement but may sometimes be needed (e.g. temp. solution when two customers are migrating to single network infrastructure).
B - blackhole, U - unreachable, P - prohibit
+
 
  0 ADC dst-address=1.1.1.0/24 pref-src=1.1.1.1 interface=ether2 distance=0 scope=10
+
=== CE1 Router, ''cust-one'' ===
        routing-mark=cust-one
+
  /ip route add dst-address=10.4.4.0/24 gateway=10.1.1.2
  1 ADb dst-address=2.2.2.0/24 gateway=10.0.11.202 interface=ether1 gateway-table=main
+
 
        gateway-state=recursive distance=20 scope=40 target-scope=30
+
=== CE2 Router, ''cust-one'' ===
        routing-mark=cust-one bgp-local-pref=100 bgp-med=0 bgp-origin=incomplete
+
/ip route add dst-address=10.4.4.0/24 gateway=10.3.3.3
        bgp-ext-communities="RT:1.1.1.1:111"
+
 
  2 ADb dst-address=3.3.3.0/24 gateway=10.0.11.202 interface=ether1 gateway-table=main
+
=== CE1 Router, ''cust-two''  ===
        gateway-state=recursive distance=20 scope=40 target-scope=30
+
/ip address add address=10.4.4.5 interface=ether1
        routing-mark=cust-one bgp-local-pref=100 bgp-med=0 bgp-origin=incomplete
+
  /ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3
        bgp-ext-communities="RT:2.2.2.2:222"
+
/ip route add dst-address=10.3.3.0/24 gateway=10.3.3.3
  3 ADb dst-address=1.1.1.0/24 distance=20 routing-mark=cust-two
+
 
        bgp-ext-communities="RT:1.1.1.1:111"
+
=== PE1 Router ===  
  4 ADb  dst-address=2.2.2.0/24 gateway=10.0.11.202 interface=ether1 gateway-table=main
+
 
        gateway-state=recursive distance=20 scope=40 target-scope=30
+
# replace the old VRF with this:
        routing-mark=cust-two bgp-local-pref=100 bgp-med=0 bgp-origin=incomplete
+
/ip route vrf add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
        bgp-ext-communities="RT:1.1.1.1:111"
+
    export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111,2.2.2.2:222 interfaces=ether1
  5 ADb dst-address=3.3.3.0/24 gateway=10.0.11.202 interface=ether1 gateway-table=main
+
 
        gateway-state=recursive distance=20 scope=40 target-scope=30
+
=== PE2 Router (Cisco) ===
        routing-mark=cust-two bgp-local-pref=100 bgp-med=0 bgp-origin=incomplete
+
<pre>
        bgp-ext-communities="RT:2.2.2.2:222"
+
ip vrf cust-one
  6 A S dst-address=0.0.0.0/0 gateway=10.0.0.1 interface=ether1 gateway-table=main
+
rd 1.1.1.1:111
        gateway-state=reachable distance=1 scope=30 target-scope=10
+
route-target export 1.1.1.1:111
  7 ADC dst-address=10.0.0.0/24 pref-src=10.0.0.131 interface=ether1 distance=0
+
route-target import 1.1.1.1:111
        scope=10
+
route-target import 2.2.2.2:222
 +
exit
 +
 
 +
ip vrf cust-two
 +
rd 2.2.2.2:222
 +
route-target export 2.2.2.2:222
 +
route-target import 1.1.1.1:111
 +
route-target import 2.2.2.2:222
 +
exit
 +
 
 +
interface FastEthernet2/0
 +
ip vrf forwarding cust-two
 +
ip address 10.4.4.3 255.255.255.0
 +
 
 +
router bgp 65000
 +
  address-family ipv4 vrf cust-two
 +
  redistribute connected
 +
  exit-address-family
 +
</pre>
 +
== Variation: replace the Cisco with another MT ==
 +
 
 +
=== PE2 Mikrotik config ===
 +
/interface bridge add name=lobridge
 +
/ip address
 +
  add address=10.2.2.3/24 interface=ether1
 +
  add address=10.3.3.3/24 interface=ether2
 +
  add address=10.4.4.3/24 interface=ether3
 +
  add address=10.5.5.3/32 interface=lobridge
 +
/ip route vrf
 +
  add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
 +
      export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111,2.2.2.2:222 \
 +
      interfaces=ether2
 +
  add disabled=no routing-mark=cust-two route-distinguisher=2.2.2.2:222 \
 +
    export-route-targets=2.2.2.2:222 import-route-targets=1.1.1.1:111,2.2.2.2:222 \
 +
    interfaces=ether3
 +
  /mpls ldp set enabled=yes transport-address=10.5.5.3
 +
/mpls ldp interface add interface=ether1
 +
/routing bgp instance set default as=65000
 +
/routing bgp instance vrf add instance=default routing-mark=cust-one redistribute-connected=yes
 +
/routing bgp instance vrf add instance=default routing-mark=cust-two redistribute-connected=yes
 +
/routing bgp peer add remote-address=10.5.5.2 remote-as=65000 address-families=vpnv4 \
 +
    update-source=lobridge
 +
  # add route to the remote BGP peer's loopback address
 +
/ip route add dst-address=10.5.5.2/32 gateway=10.2.2.2
 +
 
 +
=== Results ===  
 +
The output of '''/ip route print''' now is interesting enough to deserve detailed observation.
 +
<pre>
 +
[admin@PE2] /ip route> print
 +
Flags: X - disabled, A - active, D - dynamic,
 +
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
 +
B - blackhole, U - unreachable, P - prohibit
 +
#      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 +
0 ADb  10.1.1.0/24                        10.5.5.2 recurs... 20
 +
1 ADC  10.3.3.0/24        10.3.3.3        ether2            0
 +
2 ADb  10.4.4.0/24                                          20
 +
3 ADb  10.1.1.0/24                        10.5.5.2 recurs... 20
 +
4 ADb  10.3.3.0/24                                          20
 +
5 ADC  10.4.4.0/24        10.4.4.3        ether3            0
 +
6 ADC  10.2.2.0/24        10.2.2.3        ether1            0
 +
7 A S  10.5.5.2/32                        10.2.2.2 reacha... 1
 +
8 ADC  10.5.5.3/32        10.5.5.3        lobridge          0
 +
</pre>
 +
The route 10.1.1.0/24 was received from remote BGP peer and is installed in both VRF routing tables.
 +
 
 +
The routes 10.3.3.0/24 and 10.4.4.0/24 are also installed in both VRF routing tables. Each is as connected route in one table and as BGP route in another table. This has nothing to do with their being advertised via BGP. They are simply being "advertised" to local VPNv4 route table and locally reimported after that. Import and export '''route-targets''' determine in which tables they will end up.
 +
 
 +
This can be deduced from its attributes - they don't have the usual BGP properties. (Route 10.4.4.0/24.)
 +
<pre>
 +
[admin@PE2] /ip route> print detail where routing-mark=cust-one
 +
Flags: X - disabled, A - active, D - dynamic,
 +
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
 +
B - blackhole, U - unreachable, P - prohibit
 +
0 ADb  dst-address=10.1.1.0/24 gateway=10.5.5.2 recursive via 10.2.2.2 ether1
 +
        distance=20 scope=40 target-scope=30 routing-mark=cust-one
 +
        bgp-local-pref=100 bgp-origin=incomplete
 +
        bgp-ext-communities="RT:1.1.1.1:111"
 +
 
 +
1 ADC dst-address=10.3.3.0/24 pref-src=10.3.3.3 gateway=ether2 distance=0 scope=10
 +
        routing-mark=cust-one
 +
 
 +
2 ADb  dst-address=10.4.4.0/24 distance=20 scope=40 target-scope=10
 +
        routing-mark=cust-one bgp-ext-communities="RT:2.2.2.2:222"
 +
</pre>
 +
 
 +
== Static inter-VRF routes  ==
 +
 
 +
In general it is recommended that all routes between VRF should be exchanged using BGP local import and export functionality. If that is not enough, static routes can be used to achieve this so-called route leaking.
 +
 
 +
There are two ways to install a route that has gateway in different routing table than the route itself.
 +
 
 +
The first way is to explicitly specify routing table in gateway field when adding route. This is only possible for the "main" routing table.
 +
Example:
 +
# add route to 5.5.5.0/24 in 'vrf1' routing table with gateway in the main routing table
 +
  add dst-address=5.5.5.0/24 gateway=10.3.0.1@main routing-mark=vrf1
 +
 
 +
The second way is to explicitly specify interface in gateway field. The interface specified can belong to a VRF instance.
 +
Example:
 +
# add route to 5.5.5.0/24 in the main routing table with gateway at 'ether2' VRF interface
 +
add dst-address=5.5.5.0/24 gateway=10.3.0.1%ether2 routing-mark=main
 +
 +
# add route to 5.5.5.0/24 in the main routing table with 'ptp-link-1' VRF interface as gateway
 +
  add dst-address=5.5.5.0/24 gateway=ptp-link-1 routing-mark=main
 +
 
 +
As can be observed, there are two variations possible - to specify gateway as ''ip_address%interface'' or to simply specify ''interface''. The first should be used for broadcast interfaces in most cases. The second should be used for point-to-point interfaces, and also for broadcast interfaces, if the route is a connected route in some VRF. For example, if you have address <code>1.2.3.4/24</code> on interface ''ether2'' that is put in a VRF, there will be connected route to <code>1.2.3.0/24</code> in that VRF's routing table. It is acceptable to add static route  <code>1.2.3.0/24</code> in a different routing table with interface-only gateway, even though ''ether2'' is a broadcast interface:
 +
add dst-address=1.2.3.0/24 gateway=ether2 routing-mark=main
  
== References ==
+
= References =
 
[http://www.ietf.org/rfc/rfc4364.txt RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)]
 
[http://www.ietf.org/rfc/rfc4364.txt RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)]
  
 
MPLS Fundamentals, chapter 7, ''Luc De Ghein'', Cisco Press 2006
 
MPLS Fundamentals, chapter 7, ''Luc De Ghein'', Cisco Press 2006
 +
 +
[[Category:Case Studies|Virtual routing]]
 +
[[Category:Routing|Virtual routing]]
 +
[[Category:Internetworking|Virtual routing]]
 +
[[Category:Manual|Virtual routing]]

Latest revision as of 16:22, 7 January 2014

Version.png

Applies to RouterOS: 3, v4+

Packages required: routing-test, mpls-test for RouterOS v3; routing, mpls for RouterOS v4+

Description

RouterOS 3.x allows to create multiple Virtual Routing and Forwarding instances on a single router. This is useful for BGP based MPLS VPNs. Unlike BGP VPLS, which is OSI Layer 2 technology, BGP VRF VPNs work in Layer 3 and as such exchange IP prefixes between routers. VRFs solve the problem of overlapping IP prefixes, and provide the required privacy (via separated routing for different VPNs).

To create a VRF, configure it under /ip route vrf. You can now add routes to that VRF - simply specify routing-mark attribute. Connected routes from interfaces belonging to a VRF will be installed in the right routing table automatically.

Technically VRFs are based on policy routing. There is exactly one policy route table for each active VRF. The existing policy routing support in MT RouterOS is not changed; but on the other hand, it is not possible to have policy routing within a VRF. The main differences between VRF tables and simple policy routing are:

  • Routes in VRF tables resolve next-hops in their own route table by default, while policy routes always use the main route table. Read-only route attribute gateway-table displays information about which table is used for a particular route (default is main).
  • Route lookup is different. For policy routing: after route lookup has been done in policy-route table, and no route was found, route lookup proceeds to the main route table. For VRFs: if lookup is done, and no route is found in VRF route table, the lookup fails with "network unreachable" error. (You can still override this behavior with custom route lookup rules, as they have precedence.)
Icon-note.png

Note: When a DHCP-Relay server is attached to an interface in a vrf, the communications from that DHCP-Relay to the remote DHCP-Server will not be routed via the vrf!



You can use multi-protocol BGP with VPNv4 address family to distribute routes from VRF route tables - not only to other routers, but also to different routing tables in the router itself. First configure the route distinguisher for a VRF. It can be done under /ip route vrf. Usually there will be one-to-one correspondence between route distinguishers and VRFs, but that's not a mandatory requirement. Route installation in VRF tables is controlled by BGP extended communities attribute. Configure import and export lists under /ip route vrf, import-route-targets and export-route-targets. Export route target list for a VRF should contained at least the route distinguisher for that VRF. Then configure a list of VRFs for each BGP instance that will participate in VRF routing.

Once list of VRFs for BGP instance, route distinguisher and export route targets has been configured, some active VPNv4 address family routes may be created, depending on BGP redistribution settings. They are installed in a separate route table and, if present, visible under /routing bgp vpnv4-route. These so called VPNv4 routes have prefix that consists of a route distinguisher and an IPv4 network prefix. This way you can have overlapping IPv4 prefixes distributed in BGP.

Please note that a VPNv4 route will be distributed only if it has a valid MPLS label. You need to install mpls-test package and configure valid label range for this to work. (Default configuration has valid label range.)

Examples

The simplest MPLS VPN setup

L3vpn-simple.png

In this example rudimentary MPLS backbone (consisting of two Provider Edge (PE) routers PE1 and PE2) is created and configured to forward traffic between Customer Edge (CE) routers CE1 and CE2 routers that belong to cust-one VPN.

CE1 Router

/ip address add address=10.1.1.1/24 interface=ether1
# use static routing
/ip route add dst-address=10.3.3.0/24 gateway=10.1.1.2

CE2 Router

/ip address add address=10.3.3.4/24 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3

PE1 Router

/interface bridge add name=lobridge
/ip address add address=10.1.1.2/24 interface=ether1
/ip address add address=10.2.2.2/24 interface=ether2
/ip address add address=10.5.5.2/32 interface=lobridge
/ip route vrf add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
    export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111 interfaces=ether1
/mpls ldp set enabled=yes transport-address=10.5.5.2
/mpls ldp interface add interface=ether2
/routing bgp instance set default as=65000
/routing bgp instance vrf add instance=default routing-mark=cust-one redistribute-connected=yes
/routing bgp peer add remote-address=10.5.5.3 remote-as=65000 address-families=vpnv4 \
    update-source=lobridge
 # add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.3/32 gateway=10.2.2.3

PE2 Router (Cisco)

ip vrf cust-one
 rd 1.1.1.1:111
 route-target export 1.1.1.1:111
 route-target import 1.1.1.1:111
 exit

interface Loopback0
 ip address 10.5.5.3 255.255.255.255

mpls ldp router-id Loopback0 force
mpls label protocol ldp

interface FastEthernet0/0
 ip address 10.2.2.3 255.255.255.0
 mpls ip

interface FastEthernet1/0
 ip vrf forwarding cust-one
 ip address 10.3.3.3 255.255.255.0

router bgp 65000
 neighbor 10.5.5.2 remote-as 65000
 neighbor 10.5.5.2 update-source Loopback0
 address-family vpnv4
  neighbor 10.5.5.2 activate
  neighbor 10.5.5.2 send-community both
  exit-address-family
 address-family ipv4 vrf cust-one
  redistribute connected
  exit-address-family

ip route 10.5.5.2 255.255.255.255 10.2.2.2

Results

Check that VPNv4 route redistribution is working:

[admin@PE1] > /routing bgp vpnv4-route print detail
Flags: L - label present
 0 L route-distinguisher=1.1.1.1:111 dst-address=10.3.3.0/24 gateway=10.5.5.3
     interface=ether2 in-label=17 out-label=17 bgp-local-pref=100 bgp-med=0
     bgp-origin=incomplete bgp-ext-communities="RT:1.1.1.1:111"

 1 L route-distinguisher=1.1.1.1:111 dst-address=10.1.1.0/24 interface=ether1
     in-label=16 bgp-ext-communities="RT:1.1.1.1:111"

Check that the 10.3.3.0 is installed in IP routes, in cust-one route table:

[admin@PE1] > /ip route print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADC  10.1.1.0/24         10.1.1.2         ether1             0
 1 ADb  10.3.3.0/24                         10.5.5.3 recursi... 20
 2 ADC  10.2.2.0/24         10.2.2.2         ether2             0
 3 ADC  10.5.5.2/32         10.5.5.2         lobridge           0
 4 A S  10.5.5.3/32                         10.2.2.3 reachab... 1

Let's take closer look at IP routes in cust-one VRF. The 10.1.1.0/24 IP prefix is a connected route that belongs to an interface that was configured to belong to cust-one VRF. The 10.3.3.0/24 IP prefix was advertised via BGP as VPNv4 route from PE2 and is imported in this VRF routing table, because our configured import-route-targets matched the BGP extended communities attribute it was advertised with.

[admin@PE1] /ip route> print detail where routing-mark=cust-one
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 0 ADC  dst-address=10.1.1.0/24 pref-src=10.1.1.2 gateway=ether1 distance=0 scope=10
        routing-mark=cust-one

 1 ADb  dst-address=10.3.3.0/24 gateway=10.5.5.3 recursive via 10.2.2.3 ether2
        distance=20 scope=40 target-scope=30 routing-mark=cust-one
        bgp-local-pref=100 bgp-origin=incomplete
        bgp-ext-communities="RT:1.1.1.1:111"

The same for Cisco:

PE2#show ip bgp vpnv4 all
BGP table version is 5, local router ID is 10.5.5.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 1.1.1.1:111 (default for vrf cust-one)
*>i10.1.1.0/24      10.5.5.2                      100      0 ?
*> 10.3.3.0/24      0.0.0.0                  0         32768 ?

PE2#show ip route vrf cust-one

Routing Table: cust-one
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/24 is subnetted, 1 subnets
B       10.1.1.0 [200/0] via 10.5.5.2, 00:05:33
     10.0.0.0/24 is subnetted, 1 subnets
C       10.3.3.0 is directly connected, FastEthernet1/0

You should be able to ping from CE1 to CE2 and vice versa.

[admin@CE1] > /ping 10.3.3.4
10.3.3.4 64 byte ping: ttl=62 time=18 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=13 ms
10.3.3.4 64 byte ping: ttl=62 time=14 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 13/14.5/18 ms

A more complicated setup (changes only)

L3vpn-two-customers.png

As opposed to the simplest setup, in this example we have two customers: cust-one and cust-two.

We configure two VPNs for then, cust-one and cust-two respectively, and exchange all routes between them. (This is also called "route leaking").

Note that this could be not the most typical setup, because routes are usually not exchanged between different customers. In contrast, by default it should not be possible to gain access from one VRF site to a different VRF site in another VPN. (This is the "Private" aspect of VPNs.) Separate routing is a way to provide privacy; and it is also required to solve the problem of overlapping IP network prefixes. Route exchange is in direct conflict with these two requirement but may sometimes be needed (e.g. temp. solution when two customers are migrating to single network infrastructure).

CE1 Router, cust-one

/ip route add dst-address=10.4.4.0/24 gateway=10.1.1.2

CE2 Router, cust-one

/ip route add dst-address=10.4.4.0/24 gateway=10.3.3.3

CE1 Router, cust-two

/ip address add address=10.4.4.5 interface=ether1
/ip route add dst-address=10.1.1.0/24 gateway=10.3.3.3
/ip route add dst-address=10.3.3.0/24 gateway=10.3.3.3

PE1 Router

# replace the old VRF with this:
/ip route vrf add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
    export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111,2.2.2.2:222 interfaces=ether1

PE2 Router (Cisco)

ip vrf cust-one
 rd 1.1.1.1:111
 route-target export 1.1.1.1:111
 route-target import 1.1.1.1:111
 route-target import 2.2.2.2:222
 exit

ip vrf cust-two
 rd 2.2.2.2:222
 route-target export 2.2.2.2:222
 route-target import 1.1.1.1:111
 route-target import 2.2.2.2:222
 exit

interface FastEthernet2/0
 ip vrf forwarding cust-two
 ip address 10.4.4.3 255.255.255.0

router bgp 65000
 address-family ipv4 vrf cust-two
  redistribute connected
  exit-address-family

Variation: replace the Cisco with another MT

PE2 Mikrotik config

/interface bridge add name=lobridge
/ip address
 add address=10.2.2.3/24 interface=ether1
 add address=10.3.3.3/24 interface=ether2
 add address=10.4.4.3/24 interface=ether3
 add address=10.5.5.3/32 interface=lobridge
/ip route vrf
 add disabled=no routing-mark=cust-one route-distinguisher=1.1.1.1:111 \
     export-route-targets=1.1.1.1:111 import-route-targets=1.1.1.1:111,2.2.2.2:222 \
     interfaces=ether2
 add disabled=no routing-mark=cust-two route-distinguisher=2.2.2.2:222 \
    export-route-targets=2.2.2.2:222 import-route-targets=1.1.1.1:111,2.2.2.2:222 \
    interfaces=ether3
/mpls ldp set enabled=yes transport-address=10.5.5.3
/mpls ldp interface add interface=ether1
/routing bgp instance set default as=65000
/routing bgp instance vrf add instance=default routing-mark=cust-one redistribute-connected=yes
/routing bgp instance vrf add instance=default routing-mark=cust-two redistribute-connected=yes
/routing bgp peer add remote-address=10.5.5.2 remote-as=65000 address-families=vpnv4 \
    update-source=lobridge
 # add route to the remote BGP peer's loopback address
/ip route add dst-address=10.5.5.2/32 gateway=10.2.2.2

Results

The output of /ip route print now is interesting enough to deserve detailed observation.

[admin@PE2] /ip route> print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  10.1.1.0/24                        10.5.5.2 recurs... 20
 1 ADC  10.3.3.0/24        10.3.3.3        ether2             0
 2 ADb  10.4.4.0/24                                           20
 3 ADb  10.1.1.0/24                        10.5.5.2 recurs... 20
 4 ADb  10.3.3.0/24                                           20
 5 ADC  10.4.4.0/24        10.4.4.3        ether3             0
 6 ADC  10.2.2.0/24        10.2.2.3        ether1             0
 7 A S  10.5.5.2/32                        10.2.2.2 reacha... 1
 8 ADC  10.5.5.3/32        10.5.5.3        lobridge           0

The route 10.1.1.0/24 was received from remote BGP peer and is installed in both VRF routing tables.

The routes 10.3.3.0/24 and 10.4.4.0/24 are also installed in both VRF routing tables. Each is as connected route in one table and as BGP route in another table. This has nothing to do with their being advertised via BGP. They are simply being "advertised" to local VPNv4 route table and locally reimported after that. Import and export route-targets determine in which tables they will end up.

This can be deduced from its attributes - they don't have the usual BGP properties. (Route 10.4.4.0/24.)

[admin@PE2] /ip route> print detail where routing-mark=cust-one
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 0 ADb  dst-address=10.1.1.0/24 gateway=10.5.5.2 recursive via 10.2.2.2 ether1
        distance=20 scope=40 target-scope=30 routing-mark=cust-one
        bgp-local-pref=100 bgp-origin=incomplete
        bgp-ext-communities="RT:1.1.1.1:111"

 1 ADC  dst-address=10.3.3.0/24 pref-src=10.3.3.3 gateway=ether2 distance=0 scope=10
        routing-mark=cust-one

 2 ADb  dst-address=10.4.4.0/24 distance=20 scope=40 target-scope=10
        routing-mark=cust-one bgp-ext-communities="RT:2.2.2.2:222"

Static inter-VRF routes

In general it is recommended that all routes between VRF should be exchanged using BGP local import and export functionality. If that is not enough, static routes can be used to achieve this so-called route leaking.

There are two ways to install a route that has gateway in different routing table than the route itself.

The first way is to explicitly specify routing table in gateway field when adding route. This is only possible for the "main" routing table. Example:

# add route to 5.5.5.0/24 in 'vrf1' routing table with gateway in the main routing table
add dst-address=5.5.5.0/24 gateway=10.3.0.1@main routing-mark=vrf1

The second way is to explicitly specify interface in gateway field. The interface specified can belong to a VRF instance. Example:

# add route to 5.5.5.0/24 in the main routing table with gateway at 'ether2' VRF interface
add dst-address=5.5.5.0/24 gateway=10.3.0.1%ether2 routing-mark=main

# add route to 5.5.5.0/24 in the main routing table with 'ptp-link-1' VRF interface as gateway
add dst-address=5.5.5.0/24 gateway=ptp-link-1 routing-mark=main

As can be observed, there are two variations possible - to specify gateway as ip_address%interface or to simply specify interface. The first should be used for broadcast interfaces in most cases. The second should be used for point-to-point interfaces, and also for broadcast interfaces, if the route is a connected route in some VRF. For example, if you have address 1.2.3.4/24 on interface ether2 that is put in a VRF, there will be connected route to 1.2.3.0/24 in that VRF's routing table. It is acceptable to add static route 1.2.3.0/24 in a different routing table with interface-only gateway, even though ether2 is a broadcast interface:

add dst-address=1.2.3.0/24 gateway=ether2 routing-mark=main

References

RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)

MPLS Fundamentals, chapter 7, Luc De Ghein, Cisco Press 2006