Manual:Layer-3 MPLS VPN example

This is a kind of "putting it all together" setup.

Technologies used:

• LDP for MPLS label distribution
• BGP for VPNv4 route distribution
• OSPF as CE - PE routing protocol

Software:

• PE and P routers have RouterOS 3.17 with routing-test and mpls-test packages.
• CE routers have RouterOS 3.17 with routing-test package. (routing package and older versions can be used here as well.)

IP addressing & routing

Provider's network

On Router B:

/ip address add address=10.1.1.2/24 interface=ether2
/ip address add address=10.2.2.2/24 interface=ether3
 
# put PE-CE interface in a VRF
/ip route vrf add routing-mark=vrf1 interfaces=ether2 \
 route-distinguisher=10.1.1.1:111 import-route-targets=10.1.1.1:111 export-route-targets=10.1.1.1:111
 
# loopback interface
/interface bridge add name=lobridge
/ip address add address=10.9.9.2/32 interface=lobridge
 
# add routes to loopback addresses
# (static routing is used for destinations inside providers network)
/ip route add dst-address=10.9.9.3/32 gateway=10.2.2.3
/ip route add dst-address=10.9.9.4/32 gateway=10.2.2.3

On Router C:

/ip address add address=10.2.2.3/24 interface=ether3
/ip address add address=10.3.3.3/24 interface=ether2
 
# loopback interface
/interface bridge add name=lobridge
/ip address add address=10.9.9.3/32 interface=lobridge
 
# add routes to loopback addresses
/ip route add dst-address=10.9.9.2/32 gateway=10.2.2.2
/ip route add dst-address=10.9.9.4/32 gateway=10.3.3.4

On Router D:

/ip address add address=10.3.3.4/24 interface=ether2
/ip address add address=10.4.4.4/24 interface=ether3
 
# put PE-CE interface in a VRF
/ip route vrf add routing-mark=vrf1 interfaces=ether3 \
 route-distinguisher=10.1.1.1:111 import-route-targets=10.1.1.1:111 export-route-targets=10.1.1.1:111
 
# loopback interface
/interface bridge add name=lobridge
/ip address add address=10.9.9.4/32 interface=lobridge
 
# add routes to loopback addresses
/ip route add dst-address=10.9.9.2/32 gateway=10.3.3.3
/ip route add dst-address=10.9.9.3/32 gateway=10.3.3.3

Client's sites

On Router A:

/ip address add address=10.1.1.1/24 interface=<ToRouterB>

On Router E:

/ip address add address=10.4.4.5/24 interface=<ToRouterD>
/ip address add address=10.7.7.5/24 interface=<ToLocalNetwork>

LDP

On Router B:

/mpls ldp set enabled=yes transport-address=10.9.9.2
/mpls ldp interface add interface=ether3

On Router C:

/mpls ldp set enabled=yes transport-address=10.9.9.3
/mpls ldp interface add interface=ether2
/mpls ldp interface add interface=ether3

On Router D:

/mpls ldp set enabled=yes transport-address=10.9.9.4
/mpls ldp interface add interface=ether2

Setting transport address for LDP is not required, but very recommended. If the address is not set, the router will pick any address at random, which may be an address belonging to VRF, and as such not connectible from internal P routers.

Results

[admin@C] > /mpls ldp neighbor print
Flags: X - disabled, D - dynamic, O - operational, T - sending-targeted-hello, V - vpls
 #      TRANSPORT       LOCAL-TRANSPORT PEER                       SEN ADDRESSES
 0  O   10.9.9.2        10.9.9.3        10.1.1.2:0                 no  10.1.1.2
                                                                       10.2.2.2
                                                                       10.9.9.2
 1      10.3.3.4                                                   no
 2  O   10.9.9.4        10.9.9.3        10.3.3.4:0                 no  10.3.3.4
                                                                       10.4.4.4
                                                                       10.9.9.4

BGP

On Router B:

/routing bgp instance vrf add instance=default routing-mark=vrf1 redistribute-connected=yes \
 redistribute-ospf=yes
/routing bgp peer add remote-address=10.9.9.3 remote-as=65530 address-families=vpnv4 \
 update-source=lobridge

On Router C:

/routing bgp peer add remote-address=10.9.9.2 remote-as=65530 route-reflect=yes \
  address-families=vpnv4 update-source=lobridge
/routing bgp peer add remote-address=10.9.9.4 remote-as=65530 route-reflect=yes \
  address-families=vpnv4 update-source=lobridge
# client-to-client-reflection is on by default
#/routing bgp instance set default client-to-client-reflection=yes

On Router D:

/routing bgp instance vrf add instance=default routing-mark=vrf1 redistribute-connected=yes \
  redistribute-ospf=yes
/routing bgp peer add remote-address=10.9.9.3 remote-as=65530 address-families=vpnv4 \
  update-source=lobridge

Note that route reflection here is used for the sake of an example. A simpler configuration would work as well - one where there is a BGP session between B and D and C is not running BGP at all.

Results

Check for routes on PE routers:

/routing bgp vpn vpnv4-route print

and

/ip route print where bgp

OSPF

On Router A:

/routing ospf network add network=10.1.1.0/24 area=backbone

On Router B:

/routing ospf instance set default routing-table=vrf1 redistribute-bgp=as-type-1
/routing ospf network add network=10.1.1.0/24 area=backbone

On Router D:

/routing ospf instance set default routing-table=vrf1 redistribute-bgp=as-type-1
/routing ospf network add network=10.4.4.0/24 area=backbone

On Router E:

/routing ospf network add network=10.4.4.0/24 area=backbone
/routing ospf network add network=10.7.7.0/24 area=backbone

Results

Routing table on CE router A:

[admin@A] > /ip route pr
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADC  10.1.1.0/24        10.1.1.1        ether2             0
 1 ADo  10.4.4.0/24                        10.1.1.2 reachab... 110
 2 ADo  10.7.7.0/24                        10.1.1.2 reachab... 110

Routing table on CE router E:

[admin@E] > /ip route pr
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADo  10.1.1.0/24                        10.4.4.4 reachab... 110
 1 ADC  10.4.4.0/24        10.4.4.5        ether2             0
 2 ADC  10.7.7.0/24        10.7.7.5        ether3             0

Test

On Router A:

Ping from CE1 -> to PE1:

[admin@A] > /ping 10.1.1.2
10.1.1.2 64 byte ping: ttl=64 time=8 ms
10.1.1.2 64 byte ping: ttl=64 time=4 ms
10.1.1.2 64 byte ping: ttl=64 time=5 ms
10.1.1.2 64 byte ping: ttl=64 time=5 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 4/5.5/8 ms

Ping from CE1 -> to CE2:

[admin@A] > /ping 10.4.4.5
10.4.4.5 64 byte ping: ttl=61 time=12 ms
10.4.4.5 64 byte ping: ttl=61 time=5 ms
10.4.4.5 64 byte ping: ttl=61 time=6 ms
10.4.4.5 64 byte ping: ttl=61 time=8 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 5/7.7/12 ms

[admin@A] > /ping 10.7.7.5

10.7.7.5 64 byte ping: ttl=61 time=14 ms
10.7.7.5 64 byte ping: ttl=61 time=4 ms
10.7.7.5 64 byte ping: ttl=61 time=8 ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 4/8.6/14 ms

[admin@A] > /tool traceroute 10.7.7.5
     ADDRESS                                    STATUS
   1         10.1.1.2 3ms 6ms 2ms
   2         0.0.0.0 timeout timeout timeout
   3         10.3.3.4 4ms 3ms 3ms
   4         10.7.7.5 3ms 3ms 3ms

The second hop failure is normal.

To see whole MPLS cloud as one IP hop, configure propagate-ttl=no. This setting should be the same on all provider's routers.

On Routers B,C,D:

/mpls set propagate-ttl=no

[admin@A] > /tool traceroute 10.7.7.5
     ADDRESS                                    STATUS
   1         10.1.1.2 6ms 3ms 5ms
   2         10.3.3.4 5ms 3ms 6ms
   3         10.7.7.5 9ms 9ms 6ms

No failures here.

Connecting from PE to CE

In this case routing-table must be specified manually.

Ping from PE1 -> to CE1:

[admin@B] > ping 10.1.1.1 routing-table=vrf1
10.1.1.1 64 byte ping: ttl=64 time=9 ms
10.1.1.1 64 byte ping: ttl=64 time=6 ms
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 6/7.5/9 ms