Practical OTV


Practical OTV

————-

This post is all about OTV (Overlay Transport Virtualization) on the CSR1000v.

I wanted to create the post because there are alot of acronyms and terminology involved.

A secondary objective was to have a “real” multicast network in the middle, as the examples I have seen around the web, have used a direct P2P network for the DCI.

Instead, I wanted to have full multicast running in the SP core in order to gain a full understanding of the packet forwarding and encapsulation.

First off, lets talk about the topology I will be using:

Datacenters:

————

We have 2 Datacenters, one represented by Site 1 and the other by Site 2.

In the middle, we have what is in all respects a SP provider network. In your environment, this may or may not be your own transport network.

In site 1, CSR-1 is our “server”, basically all thats configured on it is an IP address (192.168.100.1/24) on its G1 interface.

SW-9 is our L2 switch, which is configured with 2 VLAN’s (Vlan 100 (SERVER-VLAN) and Vlan 900 (SITE-VLAN)). The port (e0/0) going to CSR-1 is configured as an access-port in Vlan 100.

The ports going to CSR-2 and CSR-3 (e0/1 and e0/2) are trunk ports.

In site 1, the CSR-2 and CSR-3 routers are our OTV Edge devices, which is basically just a naming convention to designate your OTV encapsulation/decapsulation devices. In site 1 we are running two of these in order to show how the redundancy is performed.

Site 2 is very similar, although here I have selected to only have 1 OTV edge device (CSR-7).

In all sites, our OTV edge devices use their G2 interface as whats called their “Join Interface”. All that really means is that this is the L3 interface going towards the DCI “cloud”.

Also on all OTV edge devices, the G1 port is a L2 only interface, connecting to the internals of the respective site.

Transport Network:

——————

We have OSPF running as our IGP between all devices, providing full unicast reachability between our routers.

Inside the transport network, we are running Any Source Multicast (ASM), with the Rendevouz Point (RP) being the loopback0 interface of CSR-5 (5.5.5.5/32). All other routers (CSR-4 and CSR-6) having statically this RP configured.

Its important to note that no PIM adjacency exists between CSR-2 and CSR-4, and the same between CSR-3 and CSR-4. And the other way around no PIM adjacency exists between CSR-6 and CSR-7 either.

The only thing thats required is that we enable PIM on the link on CSR-4 and CSR-6. The reason behind that is that in IOS configuration, this is what also enables IGMP which is what we are really after in this solution. So you can think of CSR-2, CSR-3 and CSR-7 as “clients” of the multicast network sending IGMP joins (actually these are reports) to the transport network, which then handles the real multicast forwarding.

Terminology:

————

We have already covered quite a bit of terminology in the previous introduction, but let me iterate a few here:

– Join interface = Simply the L3 interface on the edge device, which face the transport network.

– Edge Device = Just an OTV router. Sits at the boundary between the L2 network you want transported and the L3 transport network.

– AED Device = Authoritive Edge Device. This is the “active” router, doing the transport of a certain VLAN. Only 1 AED for each VLAN on the site.

– Site VLAN = The Edge devices need to elect an AED for each VLAN that needs transporting. This is the function of the Site VLAN.

– Internal Interface = A L2 interface going towards the internal datacenter site. This is where we receive the frames we need to extend across OTV.

– Overlay Interface = The logical representation within Cisco IOS that ties all the pieces together.

Verification:

————-

Enough theory, lets see this beast in action on the command-line.

First off is our “servers”, which is CSR-1 and CSR-8:

CSR-1#sh run int g1
Building configuration...
Current configuration : 122 bytes
!
interface GigabitEthernet1
 ip address 192.168.100.1 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
end

and CSR-8:

CSR-8#sh run int g1
Building configuration...
Current configuration : 122 bytes
!
interface GigabitEthernet1
 ip address 192.168.100.8 255.255.255.0
 negotiation auto
 no mop enabled
 no mop sysid
end

Very simple. Nothing else of interest is configured on these devices, they simply serve as our validation platform.

What about the configuration on SW-9?

SW-9#sh vlan b
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Et0/3
100  SERVER-VLAN                      active    Et0/0
900  SITE-VLAN                        active
1002 fddi-default                     act/unsup
1003 token-ring-default               act/unsup
1004 fddinet-default                  act/unsup
1005 trnet-default                    act/unsup
SW-9#sh run int e0/0
Building configuration...
Current configuration : 81 bytes
!
interface Ethernet0/0
 switchport access vlan 100
 switchport mode access
end
SW-9#sh run int e0/1
Building configuration...
Current configuration : 90 bytes
!
interface Ethernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
end
SW-9#sh run int e0/2
Building configuration...
Current configuration : 90 bytes
!
interface Ethernet0/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

And we can see the spanning-tree result for this configuration:

SW-9#sh span vl 900
VLAN0900
  Spanning tree enabled protocol rstp
  Root ID    Priority    33668
             Address     aabb.cc00.0900
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
  Bridge ID  Priority    33668  (priority 32768 sys-id-ext 900)
             Address     aabb.cc00.0900
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Et0/1               Desg FWD 100       128.2    Shr
Et0/2               Desg FWD 100       128.3    Shr
SW-9#sh span vl 100
VLAN0100
  Spanning tree enabled protocol rstp
  Root ID    Priority    32868
             Address     aabb.cc00.0900
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
  Bridge ID  Priority    32868  (priority 32768 sys-id-ext 100)
             Address     aabb.cc00.0900
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec
Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Et0/0               Desg FWD 100       128.1    Shr
Et0/1               Desg FWD 100       128.2    Shr
Et0/2               Desg FWD 100       128.3    Shr

Everything looks good so far!

Now for the interesting part, which is CSR-2, CSR-3 and CSR-7:

CSR-2:

otv site bridge-domain 900
!
otv site-identifier 0001.0001.0001
!
interface Overlay1
 no ip address
 otv control-group 239.1.1.1
 otv data-group 232.1.1.0/24
 otv join-interface GigabitEthernet2
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
!
interface GigabitEthernet1
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 900 ethernet
  encapsulation dot1q 900
  bridge-domain 900
 !
!
interface GigabitEthernet2
 ip address 172.2.4.2 255.255.255.0
 ip pim passive
 ip igmp version 3
 negotiation auto
 no mop enabled
 no mop sysid

First off, we set our Site Vlan to be 900, which is again what is used for AED election, locally to the site. This Vlan should never be extended over the OTV tunnel.

Then i set the identifier for our site. This is what is used in the loop prevention, so its very important that this is unique per site!

In the overlay interface configuration, I define a few things. The first of which is the multicast configuration, where we use group address 239.1.1.1 for our control traffic and 232.1.1.0/24 for data traffic. Next I specify that our Join interface is GigabitEthernet2. Finally I configure that we want to extend vlan 100 through the use of a service instance configuration snippet.

Toward the L2 site, we have GigabitEthernet1, where I have configured a L2 configuration using two Vlan’s. We want to have the router “listening” to both our Site Vlan (900) and our Data or Server Vlan (100) which is the one we want to extend across our DCI.

Last, but not least, we have GigabitEthernet2, which is our Join interface. This is a standard L3 interface configuration, with two important statements. First is “ip pim passive” which makes the interface run multicast, but not establish any pim adjacency and the other is “ip igmp version 3” which in effect makes the interface able to utilize SSM.

On CSR-3, the exact same configuration is present, with the exception of a different Join interface IP address:

otv site bridge-domain 900
!
otv site-identifier 0001.0001.0001
!
interface Overlay1
 no ip address
 otv control-group 239.1.1.1
 otv data-group 232.1.1.0/24
 otv join-interface GigabitEthernet2
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
!
interface GigabitEthernet1
 no ip address
 negotiation auto
 no mop enabled
 no mop sysid
 service instance 100 ethernet
  encapsulation dot1q 100
  bridge-domain 100
 !
 service instance 900 ethernet
  encapsulation dot1q 900
  bridge-domain 900
 !
!
interface GigabitEthernet2
 ip address 172.3.4.3 255.255.255.0
 ip pim passive
 ip igmp version 3
 negotiation auto
 no mop enabled
 no mop sysid

Lets use some verification commands to see if everything is as we expect:

The first of which is “show otv adjacency”:

CSR-2#show otv adja
Overlay Adjacency Database for overlay 1
Hostname                       System-ID      Dest Addr       Site-ID        Up Time   State
CSR-7                          001e.e6de.0500 172.6.7.7       0002.0002.0002 00:28:01  UP
CSR-3                          001e.7aa9.0100 172.3.4.3       0001.0001.0001 00:28:01  UP

This command verifies that we have ISIS adjacencies up and running toward both CSR-3 (in site: 0001.0001.0001) and CSR-7 in site 0002.0002.0002, which is a very good start.

Next command i want, is to check if our Server vlan (100) is active on CSR-2 and CSR-3 (remember that only one of the routers should be active for this purpose):

CSR-2#show otv vlan
Key:  SI - Service Instance, NA - Non AED, NFC - Not Forward Capable.
Overlay 1 VLAN Configuration Information
 Inst VLAN BD   Auth ED              State                Site If(s)
 0    100  100   CSR-3               inactive(NA)        Gi1:SI100
 Total VLAN(s): 1

This command states that for Vlan 100, the AED is CSR-3 and that CSR-2 (this router) is inactive for this Vlan.

The reverse should be true when looking at CSR-3:

CSR-3#show otv vlan
Key:  SI - Service Instance, NA - Non AED, NFC - Not Forward Capable.
Overlay 1 VLAN Configuration Information
 Inst VLAN BD   Auth ED              State                Site If(s)
 0    100  100  *CSR-3               active              Gi1:SI100
 Total VLAN(s): 1

Thankfully this output verifies this behavior!

For site 2, we only have one edge device, so this router (CSR-7) should be the AED for site 2:

CSR-7#show otv vlan
Key:  SI - Service Instance, NA - Non AED, NFC - Not Forward Capable.
Overlay 1 VLAN Configuration Information
 Inst VLAN BD   Auth ED              State                Site If(s)
 0    100  100  *CSR-7               active              Gi1:SI100
 Total VLAN(s): 1

Which it indeed is.

For the next verification, I need to figure out the Mac addresses of our “servers”, namely CSR-1 and CSR-8:

CSR-1#sh int g1 | incl address
  Hardware is CSR vNIC, address is 5000.0001.0000 (bia 5000.0001.0000)
CSR-8#sh int g1 | incl address
  Hardware is CSR vNIC, address is 5000.0008.0000 (bia 5000.0008.0000)

So lets check out whether or not we have those Mac address announced through the OTV control plane:

On CSR-3 in Site-1:

CSR-3#show otv route
Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route
OTV Unicast MAC Routing Table for Overlay1
 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    100  100    001e.bdf9.e2bc 40    BD Eng Gi1:SI100
 0    100  100    5000.0001.0000 40    BD Eng Gi1:SI100
 0    100  100    5000.0008.0000 50    ISIS   CSR-7
3 unicast routes displayed in Overlay1
----------------------------------------------------------
3 Total Unicast Routes Displayed

Great! – We see both addresses, one on our G1 interface on the service 100 instance, which is the L2 interface going to our local server and another address known from CSR-7 from Site-2!

The opposite should be true from CSR-7’s point of view:

CSR-7#show otv route
Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route
OTV Unicast MAC Routing Table for Overlay1
 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    100  100    001e.bdf9.e2bc 50    ISIS   CSR-3
 0    100  100    5000.0001.0000 50    ISIS   CSR-3
 0    100  100    5000.0008.0000 40    BD Eng Gi1:SI100
3 unicast routes displayed in Overlay1
----------------------------------------------------------
3 Total Unicast Routes Displayed

To complete the picture of the solution, I would like to show the state of the transport network from a multicast perspective.

CSR-2 and CSR-3 should send an IGMP join/report to select traffic from 239.1.1.1 group which we previously configured, so lets check out if CSR-4 receives this:

CSR-4#sh ip igmp groups
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
239.1.1.1        GigabitEthernet4         01:38:12  00:02:53  172.3.4.3
239.1.1.1        GigabitEthernet3         01:38:17  00:02:42  172.2.4.2
224.0.1.40       Loopback0                01:38:31  00:02:29  4.4.4.4

Excellent, CSR-4 has received information on both the G3 and G4 interface, going to CSR-2 and CSR-3 respectively.

When we in turn uses this IGMP, our multicast network, built by PIM should have some state information created:

CSR-4#sh ip mroute 239.1.1.1 |  beg Outgoing
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.1.1.1), 01:40:59/stopped, RP 5.5.5.5, flags: SJCF
  Incoming interface: GigabitEthernet1, RPF nbr 10.4.5.5
  Outgoing interface list:
    GigabitEthernet4, Forward/Sparse, 01:40:55/00:02:11
    GigabitEthernet3, Forward/Sparse, 01:40:59/00:02:56
(172.6.7.7, 239.1.1.1), 01:40:16/00:01:29, flags: JT
  Incoming interface: GigabitEthernet2, RPF nbr 10.4.6.6
  Outgoing interface list:
    GigabitEthernet3, Forward/Sparse, 01:40:16/00:02:56
    GigabitEthernet4, Forward/Sparse, 01:40:16/00:02:11
(172.3.4.3, 239.1.1.1), 01:40:54/00:03:16, flags: FT
  Incoming interface: GigabitEthernet4, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet2, Forward/Sparse, 01:40:14/00:03:27
    GigabitEthernet3, Forward/Sparse, 01:40:54/00:02:56
(172.2.4.2, 239.1.1.1), 01:40:57/00:03:15, flags: FT
  Incoming interface: GigabitEthernet3, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet2, Forward/Sparse, 01:40:14/00:02:36
    GigabitEthernet4, Forward/Sparse, 01:40:55/00:02:11

This rather large output confirms that we have a (*,G) entry for 239.1.1.1 going to our RP which is CSR-5 and we have individual (S,G) from each of our OTV edge devices!

Somewhat the same information should be present on CSR-6:

CSR-6#sh ip mroute 239.1.1.1 |  beg Outgoing
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.1.1.1), 01:43:05/stopped, RP 5.5.5.5, flags: SJCF
  Incoming interface: GigabitEthernet1, RPF nbr 10.5.6.5
  Outgoing interface list:
    GigabitEthernet3, Forward/Sparse, 01:43:05/00:02:52
(172.2.4.2, 239.1.1.1), 01:42:26/00:00:55, flags: JT
  Incoming interface: GigabitEthernet2, RPF nbr 10.4.6.4
  Outgoing interface list:
    GigabitEthernet3, Forward/Sparse, 01:42:26/00:02:52
(172.3.4.3, 239.1.1.1), 01:42:26/00:01:00, flags: JT
  Incoming interface: GigabitEthernet2, RPF nbr 10.4.6.4
  Outgoing interface list:
    GigabitEthernet3, Forward/Sparse, 01:42:26/00:02:52
(172.6.7.7, 239.1.1.1), 01:43:03/00:03:14, flags: FT
  Incoming interface: GigabitEthernet3, RPF nbr 0.0.0.0
  Outgoing interface list:
    GigabitEthernet2, Forward/Sparse, 01:42:19/00:03:29

Which indeed is the case.

So now that we have established that all the control-plane technology is working, lets try out the data-plane from CSR-1 to CSR-8:

CSR-1#ping 192.168.100.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.100.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/10/24 ms
CSR-1#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.100.1           -   5000.0001.0000  ARPA   GigabitEthernet1
Internet  192.168.100.8           9   5000.0008.0000  ARPA   GigabitEthernet1

And thats it!

To summarize, what I have gone through in this post, is how to use the CSR1K platform to provide for DCI (Data Center Interconnect) using OTV. We have gone through the configuration of the individual devices, as well as having provided a “real” multicast transport network. We then verified the control-plane information and lastly we tested our dataplane connectivity.

I hope you have enjoyed this post!