GETVPN Example

A couple of weeks ago I had the good fortune of attending Jeremy Filliben’s CCDE Bootcamp.
It was a great experience, which I will elaborate on in another post. But one of the technology areas I had a bit of difficult with, was GETVPN.

So in this post a I am going to setup a scenario in which a customer has 3 sites, 2 “normal” sites and a Datacenter site. The customer wants to encrypt traffic from Site 1 to Site 2.

Currently the customer has a regular L3VPN service from a provider (which is beyond the scope of this post). There is full connectivity between the 3 sites through this service.

The topology is as follows:


GETVPN consists of a few components, namely the Key Server (KS) and Group Members (GM’s), which is where it derives its name: Group Encrypted Transport. A single SA (Security Association) is used for the encryption. The Key Server distributes the information to the Group Members through a secure transport, where the Group Members then use this information (basically an ACL) to encrypt/decrypt the data packets.

The routing for the topology is fairly simple. (See Routing Diagram) Each client as well as the KeyServer uses a default route to reach the rest of the topology. Each CE router runs eBGP with the provider, where it redistributes the conntected interfaces into BGP for full reachability between the sites.


At this point, lets verify that we have full connectivty through the L3VPN SP.

On CE-1:

CE1#sh ip bgp
BGP table version is 7, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>                  0         32768 ?
 *>                              0 100 100 ?
 *>                              0 100 100 ?
 *>                  0         32768 ?
 *>                              0 100 100 ?
 *>                              0 100 100 ?

We are learning the routes to the other sites.

And connectivity from Client-1:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/10/25 ms

The interesting part takes place on the KeyServer along wi th CE1 and CE3.

If we take a look at the configuration on the KeyServer.

First off, we have a regular extended ACL that defines what traffic we want to encrypt. This ACL is the one that gets “downloaded” to CE1 and CE3:

ip access-list extended CRYPTO_ACL
 permit ip
 permit ip



Next up we have an ISAKMP policy which is used during the information communication with the KeyServer. This policy is present on all the Group Members (GM’s) and the KeyServer:

crypto isakmp policy 10
 encr aes 256
 hash sha256
 authentication pre-share
 group 2
crypto isakmp key SUPERSECRET address        

In this example we use a simple Pre Shared Key with the Any address form. This can (and probably should) be either certificate based. However, this complicates matters, so i skipped that.

Next is the transform set for IPsec which will be used. Notice that we use tunnel mode.

crypto ipsec transform-set GET-VPN-TRANSFORM-SET esp-aes esp-sha256-hmac 
 mode tunnel

This transform set is being referenced in a IPsec profile configuration:

crypto ipsec profile GETVPN-PROFILE
 set transform-set GET-VPN-TRANSFORM-SET 

This is nesecary in order for the next configuration, which is the entire GDOI aspect:

crypto gdoi group GDOI-GROUP
 identity number 100
 server local
  rekey authentication mypubkey rsa GETVPN-KEY
  rekey transport unicast
  sa ipsec 1
   match address ipv4 CRYPTO_ACL
   replay counter window-size 64
   no tag
  address ipv4

Here we are creating a GDOI configuration, where we have a unique identifier for this group configuration (100). We are telling the router that its the server. Next is the public key we have created with an name this time (“crypto key generate rsa label “). This is used for rekeying purposes. And notice that we are using unicasting for the key material. This could just as well have been multicast, but again, that requires you have your infrastructure multicast capable and ready.

We then reference our previous IPsec profile and specify our crypt “ACL”. Lastly we specify which “update source” should be used for this server (which the other GM’s will use to communicate to/from).

If we then match this to what is configured on CE1 and CE3:

crypto isakmp policy 10
 encr aes 256
 hash sha256
 authentication pre-share
 group 2
crypto isakmp key SUPERSECRET address        
crypto gdoi group GDOI-GROUP
 identity number 100
 server address ipv4
crypto map MYMAP 10 gdoi 
 set group GDOI-GROUP
 crypto map MYMAP

And on the interface towards the SP we apply the crypto map:

CE1#sh run int g1.10
Building configuration...

Current configuration : 115 bytes
interface GigabitEthernet1.10
 encapsulation dot1Q 10
 ip address
 crypto map MYMAP


Crypto Map Topology

We can see that we have the ISAKMP configuration which I mentioned thats being used for a secure communication channel. Next we simply have the location of our KeyServer and the Identifier and thats pretty much all. Everything else is being learned from the Key Server.

After everything has been configured, you can see the log showing the registration process:

*May 15 10:37:53.245: %CRYPTO-5-GM_REGSTER: Start registration to KS for group GDOI-GROUP using address fvrf default ivrf default
*May 15 10:38:23.356: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
*May 15 10:38:23.395: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated 0x5DB57E80F97A9A1DC16B9DBBCF7CB169
*May 15 10:38:23.395: %GDOI-5-GM_REGS_COMPL: Registration to KS complete for group GDOI-GROUP using address fvrf default ivrf default
*May 15 10:38:23.668: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS for group GDOI-GROUP & gm identity fvrf default ivrf default

Another form of verification is the “show crypto gdoi” command structure, which gives you alot of information on the process:

CE1#sh crypto gdoi 

    Group Name               : GDOI-GROUP
    Group Identity           : 100
    Group Type               : GDOI (ISAKMP)
    Crypto Path              : ipv4
    Key Management Path      : ipv4
    Rekeys received          : 0
    IPSec SA Direction       : Both

     Group Server list       :
Group Member Information For Group GDOI-GROUP:
    IPSec SA Direction       : Both
    ACL Received From KS     : gdoi_group_GDOI-GROUP_temp_acl

    Group member             :       vrf: None
       Local addr/port       :
       Remote addr/port      :
       fvrf/ivrf             : None/None
       Version               : 1.0.16
       Registration status   : Registered
       Registered with       :
       Re-registers in       : 1580 sec
       Succeeded registration: 1
       Attempted registration: 3
       Last rekey from       :
       Last rekey seq num    : 0
       Unicast rekey received: 0
       Rekey ACKs sent       : 0
       Rekey Received        : never
       DP Error Monitoring   : OFF
       IPSEC init reg executed    : 0
       IPSEC init reg postponed   : 0
       Active TEK Number     : 1
       SA Track (OID/status) : disabled

       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 0
       After latest register : 0
       Rekey Acks sents      : 0

 ACL Downloaded From KS
   access-list   permit ip
   access-list   permit ip

    Rekey Transport Type     : Unicast
    Lifetime (secs)          : 84613
    Encrypt Algorithm        : 3DES
    Key Size                 : 192     
    Sig Hash Algorithm       : HMAC_AUTH_SHA
    Sig Key Length (bits)    : 2352    

TEK POLICY for the current KS-Policy ACEs Downloaded:
    IPsec SA:
        spi: 0xA3D6592E(2748733742)
        KGS: Disabled
        transform: esp-aes esp-sha256-hmac 
        sa timing:remaining key lifetime (sec): (1815)
        Anti-Replay(Counter Based) : 64
        tag method : disabled
        alg key size: 16 (bytes)
        sig key size: 32 (bytes)
        encaps: ENCAPS_TUNNEL

Among the most interesting is the KEK policy and the ACL thats in place.

If we then verify from Client-1, we can see that we have a couple of seconds timeout while the encryption is being setup, and from there we have connectivity:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 60 percent (3/5), round-trip min/avg/max = 2/2/2 ms

So from something thats in theory very complex, this is very efficient from a both a configuration as well as a control-plane point of view. I know it certainly helped me understand the steps involved in setting GETVPN to create this lab, so I hope its been relevant for you as well!

February – A busy month indeed!

Wow, what a busy month this has been!

So I started my new job on February 1st and thus far, everything has been really great.
My new coworkers are very friendly and helpful.

I’ve spent the better part of february, trying to get to grips with the SP network I will be focusing on from now on. Im still not where I want to be yet, but im getting there. One of the guys I will be working very closely with, started cleaning up the network when he was hired 9 months ago and he’s done a really great job with what he’s had to work with.

There are still some work to be done however, which is the very reason they have hired me and another very good friend of mine. A well run network is a dynamic beast which needs to be tamed. On top of that, the company growth has been around 30% a year, so alot of structure and processes needs to come with that growth, which is where I can really make a difference.

I’ve also had the good fortune of being selected as a 2016 Cisco Champion, which was a very nice surprise. I tried to squeeze in a few good technical posts last year, which I hope was useful to someone on the net. I’ve attended a few briefings so far, but they have mainly been about topics which I dont know enough about to offer any commentary on (UCS for example). Im hoping they are working on something in the routing realm as well πŸ™‚

So for 2016, my primary objective is as previously stated, the CCDE certification.
Next month is Jeremy Filliben’s CCDE bootcamp, which I will be attending. I hope this will kick my butt into gear (I knew the change of jobs would hit my CCDE preparation). Im still aiming for a shot at the practical in late August.

Our Slack study group (Which Daniel Dib and I started) has grown quite a bit and includes a fair number of experts in different areas. If you are serious about CCDE or network design in general, dont be shy to mail me for an invite.

There are however, technologies which I also want to be familiar with to the level of blogging about them.

These include:

– Practical Segment Routing.
– Cisco’s iWAN solution.
– A deep knowledge of the ASR9K platform.
– Programmability (Python, API’s, etc.).
Now, back to work I go πŸ™‚


Doing right in the VAR role!

This post is my follow-up on a recent discussion on twitter.

Working for a VAR (Value Added Reseller) is not always the glamours life some make it out to be.

Working as a consultant, what you are really doing, is being the CEO of your own service company.
What you are selling, is basically your own services. The fact that your paycheck is being signed by someone else doesnt/shouldnt really matter.

The customer is building a relationship with you, as much as the company you are working for.
On top of that, you are continually building rapor in the networking world, so in my opinion, I would rather leave the customer with a good solution, rather than having to stick with the insane budgets that sales people end up shaving a project down to, just to get the contract.

So what can you do to create the outcome that is beneficial for all parties concerned (The customer, Your employer and yourself)?

Well, what I have tried in the past, is try and emphasize the importance of leaving the customer with the right solution based on his/her requirements and constraints. This discussion should involve both the technical side of things, as well as any account manager(s)/sales people involved. Try and focus on the long term results, such as customer satisfaction and reoccurring sales because of it.

Toward the customer, do your best to explain why solution X is better than Y, because of the requirements that are in place. Most people are sensible enough that, if you just take your time to explain the solution and have your reasoning in place, they will understand. Both of these (explaining and reasoning), is important for you to build the before mentioned rapor with the customer.

In the end, you should end up with a customer that will ask you for advice when in need, and trust your judgement when you recommend a solution. By doing this you effectively put the “fluff” from the account manager(s) aside and focus on the important work.

As engineers, we tend to focus on the technical side of a solution, but to be successful in our role(s), we also need to pay attention to the human/social aspect. Personally, this is an ongoing exercise, which I try to be very cognitive about when engaging with the customer.

So to summarize:

– Be a teamplayer, but know you are the one who has to face the customer regularly.
– Do your best to understand the customer and his/her requirements.
– Take your time to explain your solution to the customer.
– Never take the customer for granted.
– Pay attention to the social/human aspect when engaging with both the customer and your coworkers.

Now go and have a sit-down with that customer of yours!


Passed the CCDE written. Now what?

I was fortunate enough to finally pass the CCDE written exam yesterday morning.

That begs the question of “Now What?”

Well, I will spend a couple of days putting together a study strategy, based on where I am now compared to where I need to be in order to pass the exam. As it looks now, I am probably going for a fall 2016 exam date. That gives me enough time to settle into a new job with everything that entails.

It also means that I will need to spend 2-3 hours of study per day (some weekends more than that), with a combination of watching Cisco Live 365 videos and reading CVD’s/Books.

On top of that, my good friend Daniel Dib and I, along with hopefully a few others will have some design discussions using Webex. We have been told its really important to iron out different design ideas with other people. Especially if we can get a group together with people from different areas of expertise (Datacenter, Service Provider, Enterprise etc.).

Alas, an update to this story will come shortly! πŸ™‚

Take care!

Time for change

Its time for a change!

It was a tough decision, but i’ve decided that I need some new challenges in my professional life. To that effect, i’ve quit my old job and joined a different VAR/SP where I will be working in a skilled team of network engineers.

My duties will include maintaining and expanding a growing MPLS network, with all the services one can build on top of such a beast. Along with that, I will be attached to large enterprise customers, helping with design and implementation.

The new job is very supportive of my effort to go after the elusive CCDE certification, which was a big part of my decision as well, so expect more updates in that direction!

I’ve had some great years with awesome coworkers, but I have great confidence in the coming years as well!

Finally, a big thanks to my family and friends for supporting me through this decision process!


Stay tuned for important news in December!!

I’ve got some important news which I will reveal in the beginning of December!

So stay tuned πŸ™‚


Why Cisco?

Why do i keep focusing so much on Cisco, when there are clearly alot of different vendors out there with similar products and technologies?

There are several reasons for this.

1) I began the professional part of my networking career with Cisco.
2) Cisco has a proven track record when it comes to education and learning.
3) Even though not always the best match for all use-cases, Cisco is a big player in almost all areas of networking.
4) The networking opportunities provided by Cisco is by far the best of what ive seen. Take for example the Cisco Learning Network.
5) Cisco Press is really awesome in my opinion. They have alot of really high quality books out there.
6) Great opportunities to interact with the company. By this i mean to participate in programs like Cisco Champions and different SME (Subject Matter Expert) related activities.
7) Cisco documentation is not perfect, but its hands down the best I’ve seen across multiple vendors.
8) And ofcourse Cisco Live! πŸ™‚

That being said, recently i have begun to take a more neutral look at technologies. The reason being, that in this day and age, proprietary technologies become less prefered than open and standardised ones. This means that more interopability is possible, and to understand the big picture one cannot rule out other players.

All in all, i see alot of value in leaning towards Cisco. At least thats my outlook on things at the moment.

The case for lifelong learning.

People often ask me why i keep studying and when i will be “done”.
To me, this type of question seems odd, because i am committed to lifelong learning.

I am of the opinion that going through life without learning something all the time would be a life wasted. I think this goes back to the early explorers. Discovering new things, whether it be a new continent or simply a piece of knowledge really excites a certain type of people.

I am by no means comparing myself to these great explorers, but i understand what drove these legendary people to do the things they did, whether it be Columbus or more recently modern day astronauts.

My studies, whether they be in the field of networking or more personal related, will continue until the day i leave this crazy world.

There so much information and knowledge thats readily available in our day and age, that i would find it hard to simply ignore it and just lean back and say: “thats it, im done!”.

As I write this post, its about 6am in the morning. Part of my morning ritual is getting to the office early and spending some time studying before i start work. It helps me get into the mental rythm.

So I have a job for your! Tonight, instead of spending an hour or more watching TV, try and pick a book on a topic thats of interest to you, and read for a bit. You’ll be amazed at how it makes you feel. Afterwards you will have picked up a bit of knowledge you didnt have before! – Its THAT easy!

Thanks for taking the time to read this post!

My first Cisco Live!

Even thoughΒ im still in San Diego, Cisco Live! US 2015 is but a memory.

But what a memory it is! It being my first time attending a Cisco Live conference, I didn’t really know what to expect.

What I was met with, was a conference full of really sharp and nice people. The conference staff was very helpful and polite and really made an impression on me, from the time I first stepped onto the pavement outside San Diego convention center.

We (I brought my better half to the US) arrived very late on saturday, so after a good nights sleep I took the bus to the convention center to register and pick up the first piece of swag, the famous Cisco Live bag.

One of the great benefits of attending the conference was meeting with my good friend Daniel Dib (from I hadn’t seen him since January, so it was really cool to meet up with him during the week.

On Monday Daniel and I attended a session together, but most other sessions I went to alone. For the record, I paid for this trip out of my own pocket, so I didn’t have any co-workers or anything to tag along.

Tuesday was also spent in sessions, but in the evening there was the famous CCIE party, where my +1 was a friend of Daniel. Rihkka, a network engineer from Finland. She’s also in the VIP program along with Daniel. It was very nice meeting her. Also met with people from back home for morning coffee (which was actually Iced Coffee since it was fairly hot outside).

Wednesday I gave the CCDE written a shot, but unfortunately didn’t pass. Its a really weird exam if you ask me. Its supposed to be a technology design exam, but, I dont know. It just irks me somehow. Also, I had the good fortune of having a talk with Jeremy Filliben (, a renowned CCDE trainer. He gave some good advice along with information about his upcoming CCDE training programs. Very nice guy indeed. Since I lost track of Daniel for lunch, I was fortunate enough to run into Rihkka again. So we had lunch together where she could explain a bit more about Finland to me πŸ™‚

Thursday, things slowed down a bit. I decided to cancel some sessions to get time to meet up with folks. During this day I also had a chance to meet with some folks from the Cisco Champion program, which was very good. Unfortunately a private QoS session for Cisco Champions was cancelled, which I was pretty bummed about. It would have been pretty awesome to meet Tim Szigeti. The QoS guy πŸ™‚

Finally I said my goodbye’s to Daniel and headed back to the hotel. My first Cisco Live conference under my belt!

Would I recommend this as a good value for a network engineer? Absolutely!! the inspiration from the breakout sessions alone would be enough to justify it, but the social aspect of it all is what really makes the point!

I will do my very best to come back next year!

MPLS VPN’s over mGRE

This blog post outlines what “MPLS VPNs over mGRE” is all about as well as provide an example of such a configuration.

So what is “MPLS VPNs over mGRE”? – Well, basically its taking regular MPLS VPN’s and using it over an IP only core network. Since VPN’s over MPLS is one of the primary drivers for implementing an MPLS network in the first place, using the same functionality over an IP-only core might be very compelling for some not willing/able to run MPLS label switching in the core.

Instead of using labels to switch the traffic from one PE to another, mGRE (Multipoint GRE) is used as the encapsulation technology instead.

Be advised that 1 label is still being used however. This is the VPN label that’s used to identify which VRF interface to switch the traffic to when its received by a PE. This label is, just as in regular MPLS VPN’s, assigned by the PE through MP-BGP.

So how is this actually performed? – Well, lets take a look at an example.

The topology I will be using is as follows:

Topology for MPLS VPN's over mGRE

Topology for MPLS VPN’s over mGRE

** Note: I ran into an issue with VIRL, causing my CSR-3 to R3 to fail when establishing EIGRP adjacency. So i will not be using this in the examples to come. I noted this behavior on the VIRL community forums in case you are interested.

In this topology we have a core network, consisting of CSR-1 to CSR-5. They are all running OSPF in area 0. No MPLS is configured, so its pure IP routing end-to-end.

Lets take a look at CSR-5’s RIB:

CSR-5#sh ip route | beg Gateway
Gateway of last resort is not set is subnetted, 1 subnets
O [110/2] via, 00:39:00, GigabitEthernet2 is subnetted, 1 subnets
O [110/2] via, 00:38:50, GigabitEthernet3 is subnetted, 1 subnets
O [110/2] via, 00:38:50, GigabitEthernet4 is subnetted, 1 subnets
O [110/2] via, 00:39:10, GigabitEthernet5 is subnetted, 1 subnets
C is directly connected, Loopback0 is variably subnetted, 2 subnets, 2 masks
C is directly connected, GigabitEthernet2
L is directly connected, GigabitEthernet2 is variably subnetted, 2 subnets, 2 masks
C is directly connected, GigabitEthernet3
L is directly connected, GigabitEthernet3 is variably subnetted, 2 subnets, 2 masks
C is directly connected, GigabitEthernet4
L is directly connected, GigabitEthernet4 is variably subnetted, 2 subnets, 2 masks
C is directly connected, GigabitEthernet5
L is directly connected, GigabitEthernet5

And to verify that we are not running any MPLS switching:

CSR-5#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface

So we have our connected interfaces along with the loopbacks of all the routers in the core network.

Lets take a look at CSR-1’s configuration, with regards to its VRF configuration in particular:

vrf definition CUST-A
 rd 100:100
 address-family ipv4
  route-target export 100:100
  route-target import 100:100
interface GigabitEthernet2
 vrf forwarding CUST-A
 ip address
 negotiation auto
router eigrp 1
address-family ipv4 vrf CUST-A autonomous-system 100
  redistribute bgp 1 metric 1 1 1 1 1

We have our VRF CUST-A configured, with a RD of 100:100 along with 100:100 as both import and export Route-Targets. Just as we would configure for a regular MPLS L3 VPN.

We use our GigabithEthernet2 interface as our attachment circuit to our CUST-A. In addition we have EIGRP 100 running as the VRF aware IGP towards R1. And finally we are redistributing BGP into the VRF.

Lets make sure we are receiving routes from R1 into the VRF RIB:

CSR-1#sh ip route vrf CUST-A eigrp | beg Gateway
Gateway of last resort is not set is subnetted, 3 subnets
D [90/130816] via, 00:45:37, GigabitEthernet2

Looks good, we are receiving the loopback prefix from R1. This is as we would expect.

A similar configuration exists on CSR-2, CSR-3 and CSR-4. Nothing different from a regular MPLS L3 VPN service.

Now for the core configuration utilizing MP-BGP.
We are using CSR-5 as a VPN-v4 route-reflector in order to avoid having a full mesh of iBGP sessions.

So the configuration on R5 looks like this:

CSR-5#sh run | sec router bgp
router bgp 1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor remote-as 1
 neighbor update-source Loopback0
 neighbor remote-as 1
 neighbor update-source Loopback0
 neighbor remote-as 1
 neighbor update-source Loopback0
 neighbor remote-as 1
 neighbor update-source Loopback0
 address-family ipv4
 address-family vpnv4
  neighbor activate
  neighbor send-community extended
  neighbor route-reflector-client
  neighbor activate
  neighbor send-community extended
  neighbor route-reflector-client
  neighbor activate
  neighbor send-community extended
  neighbor route-reflector-client
  neighbor activate
  neighbor send-community extended
  neighbor route-reflector-client

Pretty straightforward really.

Then on CSR-1:

 CSR-1#sh run | sec router bgp
 router bgp 1
  bgp log-neighbor-changes
  no bgp default ipv4-unicast
  neighbor remote-as 1
  neighbor update-source Loopback0
  address-family ipv4
  address-family vpnv4
   neighbor activate
   neighbor send-community extended
   neighbor route-map MODIFY-INBOUND in
  address-family ipv4 vrf CUST-A
   redistribute eigrp 100

Here we have a single neighbor configured (R5 being the RR) using our loopback address. We are also redistributing routes from the VRF into BGP for VPNv4 announcements to the other PE’s. Whats really important (and differs from regular MPLS L3 VPN’s) is the route-map we apply inbound (MODIFY-INBOUND). Lets take a closer look at that:

CSR-1#sh route-map
route-map MODIFY-INBOUND, permit, sequence 10
  Match clauses:
  Set clauses:
    ip next-hop encapsulate l3vpn L3VPN-PROFILE
  Policy routing matches: 0 packets, 0 bytes

So all this does is set the next-hop according to a l3vpn profile called L3VPN-PROFILE. Now this is really the heart of the technology. Lets look at the profile in more detail:

CSR-1#sh run | beg L3VPN
l3vpn encapsulation ip L3VPN-PROFILE

Well, that wasnt very informative. It simply defines a standard profile (which means mGRE) with our desired name.
You can get more detail by using the show commands:

CSR-1#sh l3vpn encapsulation ip 

  transport ipv4 source Auto: Loopback0
  protocol gre
  payload mpls
   mtu default
  Tunnel Tunnel0 Created [OK]
  Tunnel Linestate [OK]
  Tunnel Transport Source (Auto) Loopback0 [OK]

So this tells us, that by default Loopback0 was chosen as the source of the tunnel and that Tunnel0 was created automatically. So lets take a look at the Tunnel0 in more detail:

 CSR-1#sh interface Tunnel0
 Tunnel0 is up, line protocol is up
   Hardware is Tunnel
   Interface is unnumbered. Using address of Loopback0 (
   MTU 9976 bytes, BW 10000 Kbit/sec, DLY 50000 usec,
      reliability 255/255, txload 1/255, rxload 1/255
   Encapsulation TUNNEL, loopback not set
   Keepalive not set
   Tunnel linestate evaluation up
   Tunnel source (Loopback0)
    Tunnel Subblocks:
          Tunnel0 source tracking subblock associated with Loopback0
           Set of tunnels with source Loopback0, 1 member (includes iterators), on interface <OK>
   Tunnel protocol/transport multi-GRE/IP
     Key disabled, sequencing disabled
     Checksumming of packets disabled
   Tunnel TTL 255, Fast tunneling enabled
   Tunnel transport MTU 1476 bytes
   Tunnel transmit bandwidth 8000 (kbps)
   Tunnel receive bandwidth 8000 (kbps)
   Last input never, output never, output hang never
   Last clearing of "show interface" counters 00:54:16
   Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 3
   Queueing strategy: fifo
   Output queue: 0/0 (size/max)
   5 minute input rate 0 bits/sec, 0 packets/sec
   5 minute output rate 0 bits/sec, 0 packets/sec
      0 packets input, 0 bytes, 0 no buffer
      Received 0 broadcasts (0 IP multicasts)
      0 runts, 0 giants, 0 throttles
      0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
      0 packets output, 0 bytes, 0 underruns
      0 output errors, 0 collisions, 0 interface resets
      0 unknown protocol drops
      0 output buffer failures, 0 output buffers swapped out

Whats important here is that the Tunnel protocol/transport is multi-GRE/IP, which is the whole point of it all.

So to recap, when we receive prefixes reflected by our RR (this is besides the point, it could just as well be a full mesh), we set our IP Next-Hop to the other PE’s loopback address and tell the router to do the mGRE encapsulation when traffic is to be routed to these prefixes.

Lets take a look at our BGP table on CSR-1:

CSR-1#sh bgp vpnv4 uni vrf CUST-A | beg Network
     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 100:100 (default for vrf CUST-A)
 *>                  0         32768 ?
 *>i                  0    100      0 ?
 *>i                  0    100      0 ?
 *>i                  0    100      0 ?
 *>          130816         32768 ?
 *>i             130816    100      0 ?
 *>i             130816    100      0 ?

(**Note: Remember CSR-3 is broken because of VIRL)

Lets take a look at what information is present for

 CSR-1#sh bgp vpnv4 uni vrf CUST-A
 BGP routing table entry for 100:100:, version 19
 Paths: (1 available, best #1, table CUST-A)
   Not advertised to any peer
   Refresh Epoch 1
   Local (metric 3) (via default) (via Tunnel0) from (
       Origin incomplete, metric 130816, localpref 100, valid, internal, best
       Extended Community: RT:100:100 Cost:pre-bestpath:128:130816
         0x8800:32768:0 0x8801:100:128256 0x8802:65281:2560 0x8803:65281:1500
       Originator:, Cluster list:
       mpls labels in/out nolabel/17
       rx pathid: 0, tx pathid: 0x0

Important to note here is that we are being told to use label nr. 17 as the VPN label for this prefix when sending it to (CSR-2).

And finally lets take a look at what CEF thinks about it all:

CSR-1#sh ip cef vrf CUST-A detail, epoch 0, flags [rib defined all labels]
  nexthop Tunnel0 label 17

So CEF will assign label 17 to the packet and then use Tunnel0 to reach CSR-2. Just as we would expect.

As a final verification ive done an Embedded Packet Capture on CSR-5 while doing a ping from R1’s loopback to R2’s loopback and this is what you can see here:

 6  142   29.028990          ->          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800000 0000FF2F B5490101 01010202   ......./.I......
  0020:  02020000 88470001 11FE4500 00640000   .....G....E..d..
  0030:  0000FE01 2BCD6464 64016464 64020800   ....+.ddd.ddd...

   7  142   29.106989          ->          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800001 0000FF2F B5480101 01010202   ......./.H......
  0020:  02020000 88470001 11FE4500 00640001   .....G....E..d..
  0030:  0000FE01 2BCC6464 64016464 64020800   ....+.ddd.ddd...

   8  142   29.184988          ->          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800002 0000FF2F B5470101 01010202   ......./.G......
  0020:  02020000 88470001 11FE4500 00640002   .....G....E..d..
  0030:  0000FE01 2BCB6464 64016464 64020800   ....+.ddd.ddd...

   9  142   29.241037          ->          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800003 0000FF2F B5460101 01010202   ......./.F......
  0020:  02020000 88470001 11FE4500 00640003   .....G....E..d..
  0030:  0000FE01 2BCA6464 64016464 64020800   ....+.ddd.ddd...

  10  142   29.287024          ->          GRE
  0000:  FA163E39 EC39FA16 3ECEC705 08004500   ..>9.9..>.....E.
  0010:  00800004 0000FF2F B5450101 01010202   ......./.E......
  0020:  02020000 88470001 11FE4500 00640004   .....G....E..d..
  0030:  0000FE01 2BC96464 64016464 64020800   ....+.ddd.ddd...

As you can see, the encapsulation is GRE, just as expected.

So thats all there is to this technology. Very useful if you have an IP-only core network.

I hope its been useful and i will soon attach all the configurations for the routers in case you want to take a closer look.

Thanks for reading!

Update: Link to configs here