MPLS VPN’s over mGRE

This blog post outlines what “MPLS VPNs over mGRE” is all about as well as provide an example of such a configuration.

So what is “MPLS VPNs over mGRE”? – Well, basically its taking regular MPLS VPN’s and using it over an IP only core network. Since VPN’s over MPLS is one of the primary drivers for implementing an MPLS network in the first place, using the same functionality over an IP-only core might be very compelling for some not willing/able to run MPLS label switching in the core.

Instead of using labels to switch the traffic from one PE to another, mGRE (Multipoint GRE) is used as the encapsulation technology instead.

Be advised that 1 label is still being used however. This is the VPN label that’s used to identify which VRF interface to switch the traffic to when its received by a PE. This label is, just as in regular MPLS VPN’s, assigned by the PE through MP-BGP.

So how is this actually performed? – Well, lets take a look at an example.

The topology I will be using is as follows:

Topology for MPLS VPN's over mGRE

Topology for MPLS VPN’s over mGRE

** Note: I ran into an issue with VIRL, causing my CSR-3 to R3 to fail when establishing EIGRP adjacency. So i will not be using this in the examples to come. I noted this behavior on the VIRL community forums in case you are interested.

In this topology we have a core network, consisting of CSR-1 to CSR-5. They are all running OSPF in area 0. No MPLS is configured, so its pure IP routing end-to-end.

Lets take a look at CSR-5’s RIB:

CSR-5#sh ip route | beg Gateway
Gateway of last resort is not set

      1.0.0.0/32 is subnetted, 1 subnets
O        1.1.1.1 [110/2] via 192.168.15.1, 00:39:00, GigabitEthernet2
      2.0.0.0/32 is subnetted, 1 subnets
O        2.2.2.2 [110/2] via 192.168.25.2, 00:38:50, GigabitEthernet3
      3.0.0.0/32 is subnetted, 1 subnets
O        3.3.3.3 [110/2] via 192.168.35.3, 00:38:50, GigabitEthernet4
      4.0.0.0/32 is subnetted, 1 subnets
O        4.4.4.4 [110/2] via 192.168.45.4, 00:39:10, GigabitEthernet5
      5.0.0.0/32 is subnetted, 1 subnets
C        5.5.5.5 is directly connected, Loopback0
      192.168.15.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.15.0/24 is directly connected, GigabitEthernet2
L        192.168.15.5/32 is directly connected, GigabitEthernet2
      192.168.25.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.25.0/24 is directly connected, GigabitEthernet3
L        192.168.25.5/32 is directly connected, GigabitEthernet3
      192.168.35.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.35.0/24 is directly connected, GigabitEthernet4
L        192.168.35.5/32 is directly connected, GigabitEthernet4
      192.168.45.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.45.0/24 is directly connected, GigabitEthernet5
L        192.168.45.5/32 is directly connected, GigabitEthernet5

And to verify that we are not running any MPLS switching:

CSR-5#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface

So we have our connected interfaces along with the loopbacks of all the routers in the core network.

Lets take a look at CSR-1’s configuration, with regards to its VRF configuration in particular:

vrf definition CUST-A
 rd 100:100
 !
 address-family ipv4
  route-target export 100:100
  route-target import 100:100
 exit-address-family
!
!
interface GigabitEthernet2
 vrf forwarding CUST-A
 ip address 10.0.1.1 255.255.255.0
 negotiation auto
!
router eigrp 1
!
address-family ipv4 vrf CUST-A autonomous-system 100
  redistribute bgp 1 metric 1 1 1 1 1
  network 0.0.0.0
 exit-address-family

We have our VRF CUST-A configured, with a RD of 100:100 along with 100:100 as both import and export Route-Targets. Just as we would configure for a regular MPLS L3 VPN.

We use our GigabithEthernet2 interface as our attachment circuit to our CUST-A. In addition we have EIGRP 100 running as the VRF aware IGP towards R1. And finally we are redistributing BGP into the VRF.

Lets make sure we are receiving routes from R1 into the VRF RIB:

CSR-1#sh ip route vrf CUST-A eigrp | beg Gateway
Gateway of last resort is not set

      100.0.0.0/32 is subnetted, 3 subnets
D        100.100.100.1 [90/130816] via 10.0.1.100, 00:45:37, GigabitEthernet2

Looks good, we are receiving the loopback prefix from R1. This is as we would expect.

A similar configuration exists on CSR-2, CSR-3 and CSR-4. Nothing different from a regular MPLS L3 VPN service.

Now for the core configuration utilizing MP-BGP.
We are using CSR-5 as a VPN-v4 route-reflector in order to avoid having a full mesh of iBGP sessions.

So the configuration on R5 looks like this:

CSR-5#sh run | sec router bgp
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 remote-as 1
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 2.2.2.2 remote-as 1
 neighbor 2.2.2.2 update-source Loopback0
 neighbor 3.3.3.3 remote-as 1
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 4.4.4.4 remote-as 1
 neighbor 4.4.4.4 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 send-community extended
  neighbor 1.1.1.1 route-reflector-client
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 send-community extended
  neighbor 2.2.2.2 route-reflector-client
  neighbor 3.3.3.3 activate
  neighbor 3.3.3.3 send-community extended
  neighbor 3.3.3.3 route-reflector-client
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 send-community extended
  neighbor 4.4.4.4 route-reflector-client
 exit-address-family

Pretty straightforward really.

Then on CSR-1:

 CSR-1#sh run | sec router bgp
 router bgp 1
  bgp log-neighbor-changes
  no bgp default ipv4-unicast
  neighbor 5.5.5.5 remote-as 1
  neighbor 5.5.5.5 update-source Loopback0
  !
  address-family ipv4
  exit-address-family
  !
  address-family vpnv4
   neighbor 5.5.5.5 activate
   neighbor 5.5.5.5 send-community extended
   neighbor 5.5.5.5 route-map MODIFY-INBOUND in
  exit-address-family
  !
  address-family ipv4 vrf CUST-A
   redistribute eigrp 100
  exit-address-family

Here we have a single neighbor configured (R5 being the RR) using our loopback address. We are also redistributing routes from the VRF into BGP for VPNv4 announcements to the other PE’s. Whats really important (and differs from regular MPLS L3 VPN’s) is the route-map we apply inbound (MODIFY-INBOUND). Lets take a closer look at that:

CSR-1#sh route-map
route-map MODIFY-INBOUND, permit, sequence 10
  Match clauses:
  Set clauses:
    ip next-hop encapsulate l3vpn L3VPN-PROFILE
  Policy routing matches: 0 packets, 0 bytes

So all this does is set the next-hop according to a l3vpn profile called L3VPN-PROFILE. Now this is really the heart of the technology. Lets look at the profile in more detail:

CSR-1#sh run | beg L3VPN
l3vpn encapsulation ip L3VPN-PROFILE
 !

Well, that wasnt very informative. It simply defines a standard profile (which means mGRE) with our desired name.
You can get more detail by using the show commands:

CSR-1#sh l3vpn encapsulation ip 

 Profile: L3VPN-PROFILE
  transport ipv4 source Auto: Loopback0
  protocol gre
  payload mpls
   mtu default
  Tunnel Tunnel0 Created [OK]
  Tunnel Linestate [OK]
  Tunnel Transport Source (Auto) Loopback0 [OK]

So this tells us, that by default Loopback0 was chosen as the source of the tunnel and that Tunnel0 was created automatically. So lets take a look at the Tunnel0 in more detail:

 CSR-1#sh interface Tunnel0
 Tunnel0 is up, line protocol is up
   Hardware is Tunnel
   Interface is unnumbered. Using address of Loopback0 (1.1.1.1)
   MTU 9976 bytes, BW 10000 Kbit/sec, DLY 50000 usec,
      reliability 255/255, txload 1/255, rxload 1/255
   Encapsulation TUNNEL, loopback not set
   Keepalive not set
   Tunnel linestate evaluation up
   Tunnel source 1.1.1.1 (Loopback0)
    Tunnel Subblocks:
       src-track:
          Tunnel0 source tracking subblock associated with Loopback0
           Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>
   Tunnel protocol/transport multi-GRE/IP
     Key disabled, sequencing disabled
     Checksumming of packets disabled
   Tunnel TTL 255, Fast tunneling enabled
   Tunnel transport MTU 1476 bytes
   Tunnel transmit bandwidth 8000 (kbps)
   Tunnel receive bandwidth 8000 (kbps)
   Last input never, output never, output hang never
   Last clearing of "show interface" counters 00:54:16
   Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 3
   Queueing strategy: fifo
   Output queue: 0/0 (size/max)
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec
      0 packets input, 0 bytes, 0 no buffer
      Received 0 broadcasts (0 IP multicasts)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      0 packets output, 0 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 unknown protocol drops
      0 output buffer failures, 0 output buffers swapped out

Whats important here is that the Tunnel protocol/transport is multi-GRE/IP, which is the whole point of it all.

So to recap, when we receive prefixes reflected by our RR (this is besides the point, it could just as well be a full mesh), we set our IP Next-Hop to the other PE’s loopback address and tell the router to do the mGRE encapsulation when traffic is to be routed to these prefixes.

Lets take a look at our BGP table on CSR-1:

CSR-1#sh bgp vpnv4 uni vrf CUST-A | beg Network
     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:100 (default for vrf CUST-A)
 *>  10.0.1.0/24      0.0.0.0                  0         32768 ?
 *>i 10.0.2.0/24      2.2.2.2                  0    100      0 ?
 *>i 10.0.3.0/24      3.3.3.3                  0    100      0 ?
 *>i 10.0.4.0/24      4.4.4.4                  0    100      0 ?
 *>  100.100.100.1/32 10.0.1.100          130816         32768 ?
 *>i 100.100.100.2/32 2.2.2.2             130816    100      0 ?
 *>i 100.100.100.4/32 4.4.4.4             130816    100      0 ?

(**Note: Remember CSR-3 is broken because of VIRL)

Lets take a look at what information is present for 100.100.100.2/32:

 CSR-1#sh bgp vpnv4 uni vrf CUST-A 100.100.100.2/32
 BGP routing table entry for 100:100:100.100.100.2/32, version 19
 Paths: (1 available, best #1, table CUST-A)
   Not advertised to any peer
   Refresh Epoch 1
   Local
     2.2.2.2 (metric 3) (via default) (via Tunnel0) from 5.5.5.5 (5.5.5.5)
       Origin incomplete, metric 130816, localpref 100, valid, internal, best
       Extended Community: RT:100:100 Cost:pre-bestpath:128:130816
         0x8800:32768:0 0x8801:100:128256 0x8802:65281:2560 0x8803:65281:1500
         0x8806:0:1684300802
       Originator: 2.2.2.2, Cluster list: 5.5.5.5
       mpls labels in/out nolabel/17
       rx pathid: 0, tx pathid: 0x0

Important to note here is that we are being told to use label nr. 17 as the VPN label for this prefix when sending it to 2.2.2.2 (CSR-2).

And finally lets take a look at what CEF thinks about it all:

CSR-1#sh ip cef vrf CUST-A 100.100.100.2 detail
100.100.100.2/32, epoch 0, flags [rib defined all labels]
  nexthop 2.2.2.2 Tunnel0 label 17

So CEF will assign label 17 to the packet and then use Tunnel0 to reach CSR-2. Just as we would expect.

As a final verification ive done an Embedded Packet Capture on CSR-5 while doing a ping from R1’s loopback to R2’s loopback and this is what you can see here:

 6  142   29.028990   1.1.1.1          ->  2.2.2.2          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800000 0000FF2F B5490101 01010202   ......./.I......
  0020:  02020000 88470001 11FE4500 00640000   .....G....E..d..
  0030:  0000FE01 2BCD6464 64016464 64020800   ....+.ddd.ddd...

   7  142   29.106989   1.1.1.1          ->  2.2.2.2          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800001 0000FF2F B5480101 01010202   ......./.H......
  0020:  02020000 88470001 11FE4500 00640001   .....G....E..d..
  0030:  0000FE01 2BCC6464 64016464 64020800   ....+.ddd.ddd...

   8  142   29.184988   1.1.1.1          ->  2.2.2.2          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800002 0000FF2F B5470101 01010202   ......./.G......
  0020:  02020000 88470001 11FE4500 00640002   .....G....E..d..
  0030:  0000FE01 2BCB6464 64016464 64020800   ....+.ddd.ddd...

   9  142   29.241037   1.1.1.1          ->  2.2.2.2          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800003 0000FF2F B5460101 01010202   ......./.F......
  0020:  02020000 88470001 11FE4500 00640003   .....G....E..d..
  0030:  0000FE01 2BCA6464 64016464 64020800   ....+.ddd.ddd...

  10  142   29.287024   1.1.1.1          ->  2.2.2.2          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800004 0000FF2F B5450101 01010202   ......./.E......
  0020:  02020000 88470001 11FE4500 00640004   .....G....E..d..
  0030:  0000FE01 2BC96464 64016464 64020800   ....+.ddd.ddd...

As you can see, the encapsulation is GRE, just as expected.

So thats all there is to this technology. Very useful if you have an IP-only core network.

I hope its been useful and i will soon attach all the configurations for the routers in case you want to take a closer look.

Thanks for reading!

Update: Link to configs here

Comments are closed.