View source for Routing with NSX using multiple sites

This article will be about routing using different protocols and routing domain setups.
As you know with routing we advertise network prefixes so that the networks are known on other places on the network. In this process, we can influence the traffic flow when multiple routing/network paths are available.
This can be done egress (from inside to outside – typically south to north) and ingress (from outside to inside – typically from north to south).

To influence the path that is taken there are different methods per protocol possible.

In this article, I will only discuss OSPF cost and BGP weight because these are the only mechanisms that are supported by the Edges of NSX. 
I will also only talk about egress routing as ingress routing is usually influenced on other devices with different attributes / mechanisms.

The general rule for OSPF cost is that the lower the cost is the more preferred the route is.
For BGP this is the higher the weight the more preferred the route is.

== NSX Multisite deployments ==

When we are using vSphere together with NSX Multisite deployments are supported.
The following options are possible:

# Multisite with multiple vCenters
## With active/passive site egress (Routing Metric or Local Egress Utilized)
## With active/active site egress (Local Egress Utilized)
# Multisite with single vCenter (stretched storage required)
## With active/passive site egress (Routing Metric or Local Egress Utilized)
## With active/active site egress (Local Egress Utilized)

I will only discuss 1a in this article with Routing Metric route manipulation.

== Diagram ==

The following diagram will be used for our setup.

>> {{Download|file=LAB100_-_NSX_Routing_based_on_eBGP_v2.pdf|LAB100_-_NSX_Routing_based_on_eBGP_v2.pdf|LAB100_-_NSX_Routing_based_on_eBGP_v2.pdf}}

[[File:nsx-routing-blog-1.png|600px]]

== Components ==

I have used the following components:

=== CORE ===

* CS01 = Cisco 3550 L3 Core Switch

=== DC1 ===

* External PSC (6.5)
* vCenter Server Appliance (6.5)
* NSX Manager – Primary (6.3.1)
* RT-A-01 = CSR1000V
* RT-A-02 = CSR1000V
* ESG-A = NSX ESG (standalone, no ECMP, no HA)
* UDLR-01 (standalone, no HA)
* UDLR-02 (local egress enabled)
* 3-TIER APP 

=== DC2 ===

* External PSC (6.5)
* vCenter Server Appliance (6.5)
* NSX Manager – Secondary (6.3.1)
* RT-B-01 = CSR1000V
* RT-B-02 = CSR1000V
* ESG-B = NSX ESG (standalone, no ECMP, no HA)
* UDLR-02 (local egress enabled)
* 3-TIER APP 

== Use&ndash;Cases ==

I have thought of the following use-cases below.

As it is not possible to test active/passive and active/active with one UDLR I am using two UDLR instances here.

For testing purpose, I will use three tenants with each having their own Web, App and DB tiers.

Tenant one and two will be using the first UDLR (UDLR-01) with active/passive site egress.
Tenant three will be using the second UDLR (UDLR-02) with active/active site egress.

=== Tenant one ===
Tenant one will have workloads in DC1 and in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site.
When the primary site is down the traffic should exit from the secondary site.

=== Tenant two ===
Tenant two will have workloads in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site.
When the primary site is down the traffic should exit from the secondary site.

This will be demonstrated in this article by using tenant one because tenant one has workloads in both sites.

=== Tenant three ===
Tenant three will have workloads in DC1 and in DC2 and because it is using local egress the traffic will exit from the Cisco CSR1000V routers local to that site.
When the primary site is down the traffic should exit from the secondary site and when the secondary is down traffic should exit from the primary.

The following implementation options are available and I am going to deploy and test all of them.

This will not be demonstrated in this article, but in another article.

== Routing protocol options ==

# Option 1
## eBGP peering between the UDLR and the ESG’s
## eBGP peering between the ESG’s and the external routers
## eBGP peering between the external routers and the CORE
# Option 2
## iBGP peering between the UDLR and the ESG’s
## eBGP peering between the ESG’s and the external routers
## eBGP peering between the external routers and the CORE
# Option 3
## iBGP peering between the UDLR and the ESG’s
## iBGP peering between the ESG’s and the external routers
## eBGP peering between the external routers and the CORE
# Option 4
## OSPF peering between the UDLR and the ESG’s (where the ESG is an Area Border Router (ABR))
## OSPF peering between the ESG’s and the external routers (where the ESG is an Area Border Router (ABR))
## eBGP peering between the external routers and the CORE
# Option 5
## OSPF peering between the UDLR and the ESG’s (in the same area)
## OSPF peering between the ESG’s and the external routers (in the same area)
## eBGP peering between the external routers and the CORE

== Option 1 ==

Because of the many options available in the setup I want to start with the following:
Tenant 1 workloads with the use of UDLR-01 and option 1 routing.
The other options will be outlined in other articles.

=== UDLR configuration ===

==== DC1 ====

Manage --> Firewall 

# Make sure the firewall is disabled

[[File:nsx-routing-blog-2.png|600px]]

Manage --> Routing --> Global Configuration

# Enable ECMP
# Make sure that the default gateway is not configured
# Configure a router ID 

[[File:nsx-routing-blog-3.png|600px]]

Manage --> Routing --> BGP

# Enable BGP
# Configure the Local AS
# Disable Graceful Restart
# Configure the BGP peers towards ESG-A and ESG-B

[[File:nsx-routing-blog-4.png|600px]]

Manage --> Routing --> Route Redistribution

# Enable route redistribution for BGP
# Specify what you allow to redistribute

[[File:nsx-routing-blog-5.png|600px]]

==== DC2 ====

The UDLR control VM does not exist in DC2.

=== ESG configuration ===

==== DC1 ====

Manage --> Firewall 

# Make sure the firewall is disabled

[[File:nsx-routing-blog-6.png|600px]]

Manage --> Routing --> Global Configuration

# Enable ECMP
# Make sure that the default gateway is not configured
# Configure a router ID 

[[File:nsx-routing-blog-7.png|600px]]

Manage --> Routing --> BGP

# Enable BGP
# Configure the Local AS
# Disable Graceful Restart
# Configure the BGP peers towards the external routers and the UDLR.

[[File:nsx-routing-blog-8.png|600px]]

Manage --> Routing --> Route Redistribution

# Enable route redistribution for BGP
# Specify what you allow to redistribute

[[File:nsx-routing-blog-9.png|600px]]

==== DC2 ====

Manage --> Firewall 

# Make sure the firewall is disabled

[[File:nsx-routing-blog-10.png|600px]]

Manage --> Routing --> Global Configuration

# Enable ECMP
# Make sure that the default gateway is not configured
# Configure a router ID 

[[File:nsx-routing-blog-11.png|600px]]

Manage --> Routing --> BGP

# Enable BGP
# Configure the Local AS
# Disable Graceful Restart
# Configure the BGP peers towards the external routers and the UDLR

[[File:nsx-routing-blog-12.png|600px]]

Manage --> Routing --> Route Redistribution

# Enable route redistribution for BGP
# Specify what you allow to redistribute

[[File:nsx-routing-blog-13.png|600px]]

=== Cisco1000V configuration ===

==== DC1 ====

{{console|body=
!
##bl##hostname rt-a-01
!
router bgp 65511
 bgp router-id 10.11.11.31
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.19.2 remote-as 65521
 neighbor 10.100.19.2 description ESG-A
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.100.19.2 activate
 exit-address-family
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!
}}

{{console|body=
!
##bl##hostname rt-a-02
!
router bgp 65511
 bgp router-id 10.11.11.32
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.100.21.2 remote-as 65521
 neighbor 10.100.21.2 description ESG-A
 !
<...> missing?
!
ip route 10.200.19.0 255.255.255.0 10.11.11.253
ip route 10.200.21.0 255.255.255.0 10.11.11.253
!
}}

==== DC2 ====

{{console|body=
!
##bl##hostname rt-b-01
!
router bgp 65512
 bgp router-id 10.11.11.33
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.19.2 remote-as 65522
 neighbor 10.200.19.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.19.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!
}}

{{console|body=
!
##bl##hostname rt-b-02
!
router bgp 65512
 bgp router-id 10.11.11.34
 bgp log-neighbor-changes
 neighbor 10.11.11.253 remote-as 65510
 neighbor 10.11.11.253 description CS01
 neighbor 10.200.21.2 remote-as 65522
 neighbor 10.200.21.2 description ESG-B
 !
 address-family ipv4
  neighbor 10.11.11.253 activate
  neighbor 10.200.21.2 activate
 exit-address-family
!
ip route 10.100.19.0 255.255.255.0 10.11.11.253
ip route 10.100.21.0 255.255.255.0 10.11.11.253
!
}}

=== Core configuration ===

{{console|body=
!
##bl##hostname cs-01
!
router bgp 65510
 bgp router-id 10.11.11.253
 bgp log-neighbor-changes
 neighbor 10.11.11.31 remote-as 65511
 neighbor 10.11.11.31 description RT-A-01
 neighbor 10.11.11.32 remote-as 65511
 neighbor 10.11.11.32 description RT-A-02
 neighbor 10.11.11.33 remote-as 65512
 neighbor 10.11.11.33 description RT-B-01
 neighbor 10.11.11.34 remote-as 65512
 neighbor 10.11.11.34 description RT-B-02
 !
 address-family ipv4
  network 10.11.11.0 mask 255.255.255.0
  neighbor 10.11.11.31 activate
  neighbor 10.11.11.31 next-hop-self
  neighbor 10.11.11.32 activate
  neighbor 10.11.11.32 next-hop-self
  neighbor 10.11.11.33 activate
  neighbor 10.11.11.33 next-hop-self
  neighbor 10.11.11.34 activate
  neighbor 10.11.11.34 next-hop-self
  no auto-summary
 exit-address-family
!
}}

=== UDLR route peering verification ===

==== DC1 ====

{{console|body=
##bl##REGX-UDLR01-0> show ip bgp neighbors

##y##BGP neighbor is 172.39.39.1,   remote AS 65521,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2886 messages, Sent 2903 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x576e9e8c
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 4 sent 12 advertised 12
Connections established 2, dropped 3
Local host: 172.39.39.13, Local port: 179
Remote host: 172.39.39.1, Remote port: 37933


##y##BGP neighbor is 172.39.39.2,   remote AS 65522,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2903 messages, Sent 2905 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x576e9e8c
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 6 sent 10 advertised 10
Connections established 1, dropped 1
Local host: 172.39.39.13, Local port: 55933
Remote host: 172.39.39.2, Remote port: 179

REGX-UDLR01-0>
}}

==== DC2 ====

The UDLR control VM does not exist in DC2.

=== ESG route peering verification ===

==== DC1 ====

{{console|body=
##bl##REGA-ESG01-0> show ip bgp neighbors

##y##BGP neighbor is 10.100.19.1,   remote AS 65511,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2758 messages, Sent 2869 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x9c4708ec
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 2 sent 15 advertised 15
Connections established 1, dropped 1
Local host: 10.100.19.2, Local port: 25030
Remote host: 10.100.19.1, Remote port: 179


##y##BGP neighbor is 10.100.21.1,   remote AS 65511,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2754 messages, Sent 2873 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x9c4708ec
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 2 sent 15 advertised 15
Connections established 1, dropped 1
Local host: 10.100.21.2, Local port: 31420
Remote host: 10.100.21.1, Remote port: 179


##y##BGP neighbor is 172.39.39.13,   remote AS 65530,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2877 messages, Sent 2859 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 3 Identifier 0x9c4708ec
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 12 sent 4 advertised 4
Connections established 1, dropped 1
Local host: 172.39.39.1, Local port: 37933
Remote host: 172.39.39.13, Remote port: 179

REGA-ESG01-0>
}}

==== DC2 ====

{{console|body=
##bl##REGB-ESG01-0> show ip bgp neighbors

##y##BGP neighbor is 10.200.19.1,   remote AS 65512,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2804 messages, Sent 2936 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 1 Identifier 0x359d82ac
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 2 sent 13 advertised 13
Connections established 1, dropped 1
Local host: 10.200.19.2, Local port: 43134
Remote host: 10.200.19.1, Remote port: 179


##y##BGP neighbor is 10.200.21.1,   remote AS 65512,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2801 messages, Sent 2937 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 2 Identifier 0x359d82ac
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 2 sent 13 advertised 13
Connections established 1, dropped 1
Local host: 10.200.21.2, Local port: 19055
Remote host: 10.200.21.1, Remote port: 179


##y##BGP neighbor is 172.39.39.13,   remote AS 65530,
##y##BGP state = Established, up
Hold time is 180, Keep alive interval is 60 seconds
Neighbor capabilities:
         Route refresh: advertised and received
         Address family IPv4 Unicast:advertised and received
         Graceful restart Capability:none
                 Restart remain time: 0
Received 2926 messages, Sent 2937 messages
Default minimum time between advertisement runs is 30 seconds
For Address family IPv4 Unicast:advertised and received
         Index 3 Identifier 0x359d82ac
         Route refresh request:received 0 sent 0
         ##y##Prefixes received 10 sent 6 advertised 6
Connections established 2, dropped 1
Local host: 172.39.39.2, Local port: 179
Remote host: 172.39.39.13, Remote port: 55933

REGB-ESG01-0>
}}

=== Cisco1000V route peering verification ===

==== DC1 ====

{{console|body=
##bl##rt-a-01#show ip bgp summary
BGP router identifier 10.11.11.31, local AS number 65511
BGP table version is 145, main routing table version 145
15 network entries using 3720 bytes of memory
27 path entries using 3240 bytes of memory
4/2 BGP path/bestpath attribute entries using 960 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 8008 total bytes of memory
BGP activity 24/9 prefixes, 105/78 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
##y##10.11.11.253    4        65510   12586   12590      145    0    0 1w0d           12
##y##10.100.19.2     4        65521    2880    2769      145    0    0 1d17h          15
rt-a-01#
}}

{{console|body=
##bl##rt-a-02#show ip bgp summary
BGP router identifier 10.11.11.32, local AS number 65511
BGP table version is 145, main routing table version 145
15 network entries using 3720 bytes of memory
27 path entries using 3240 bytes of memory
4/2 BGP path/bestpath attribute entries using 960 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 8008 total bytes of memory
BGP activity 31/16 prefixes, 105/78 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
##y##10.11.11.253    4        65510   12588   12583      145    0    0 1w0d           12
##y##10.100.21.2     4        65521    2884    2766      145    0    0 1d17h          15
rt-a-02#
}}

==== DC2 ====

{{console|body=
##bl##rt-b-01#show ip bgp summary
BGP router identifier 10.11.11.33, local AS number 65512
BGP table version is 152, main routing table version 152
15 network entries using 3720 bytes of memory
18 path entries using 2160 bytes of memory
3/2 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 6688 total bytes of memory
BGP activity 42/27 prefixes, 104/86 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
##y##10.11.11.253    4        65510   12575   12587      152    0    0 1w0d            5
##y##10.200.19.2     4        65522    2946    2813      152    0    0 1d18h          13
rt-b-01#
}}

{{console|body=
##bl##rt-b-02#show ip bgp summary
BGP router identifier 10.11.11.34, local AS number 65512
BGP table version is 152, main routing table version 152
15 network entries using 3720 bytes of memory
18 path entries using 2160 bytes of memory
3/2 BGP path/bestpath attribute entries using 720 bytes of memory
3 BGP AS-PATH entries using 88 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 6688 total bytes of memory
BGP activity 44/29 prefixes, 104/86 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
##y##10.11.11.253    4        65510   12587   12587      152    0    0 1w0d            5
##y##10.200.21.2     4        65522    2947    2812      152    0    0 1d18h          13
rt-b-02#
}}

=== Core route peering verification ===

{{console|body=
##bl##cs-01#show ip bgp summary 
BGP router identifier 10.11.11.253, local AS number 65510
BGP table version is 153, main routing table version 153
15 network entries using 2040 bytes of memory
54 path entries using 2808 bytes of memory
3/3 BGP path/bestpath attribute entries using 372 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5268 total bytes of memory
BGP activity 32/17 prefixes, 256/202 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
##y##10.11.11.31     4        65511   12594   12590      153    0    0 1w0d           13
##y##10.11.11.32     4        65511   12586   12591      153    0    0 1w0d           13
##y##10.11.11.33     4        65512   12589   12577      153    0    0 1w0d           13
##y##10.11.11.34     4        65512   12588   12587      153    0    0 1w0d           13
cs-01#
}}

=== UDLR routing tables ===

==== DC1 ====

{{console|body=
##bl##REGX-UDLR01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 15

##y##B       10.11.11.0/24        [20/0]        via 172.39.39.2
B       10.22.22.0/24        [20/0]        via 172.39.39.2
B       10.100.19.0/24       [20/0]        via 172.39.39.1
B       10.100.21.0/24       [20/0]        via 172.39.39.1
B       10.200.19.0/24       [20/0]        via 172.39.39.2
B       10.200.21.0/24       [20/0]        via 172.39.39.2
##y##C       172.20.1.0/24        [0/0]         via 172.20.1.254
##y##C       172.20.2.0/24        [0/0]         via 172.20.2.254
##y##C       172.20.3.0/24        [0/0]         via 172.20.3.254
C       172.20.8.0/24        [0/0]         via 172.20.8.254
C       172.20.9.0/24        [0/0]         via 172.20.9.254
C       172.20.10.0/24       [0/0]         via 172.20.10.254
C       172.39.39.0/28       [0/0]         via 172.39.39.13
B       172.39.39.16/28      [20/0]        via 172.39.39.1
B       172.39.39.32/28      [20/0]        via 172.39.39.2
REGX-UDLR01-0>
}}

==== DC2 ====

The UDLR control VM does not exist in DC2.

=== ESG routing tables ===

==== DC1 ====

{{console|body=
##bl##REGA-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 15

##y##B       10.11.11.0/24        [20/0]        via 172.39.39.14
B       10.22.22.0/24        [20/0]        via 172.39.39.14
C       10.100.19.0/24       [0/0]         via 10.100.19.2
C       10.100.21.0/24       [0/0]         via 10.100.21.2
B       10.200.19.0/24       [20/0]        via 172.39.39.14
B       10.200.21.0/24       [20/0]        via 172.39.39.14
##y##B       172.20.1.0/24        [20/0]        via 172.39.39.14
##y##B       172.20.2.0/24        [20/0]        via 172.39.39.14
##y##B       172.20.3.0/24        [20/0]        via 172.39.39.14
B       172.20.8.0/24        [20/0]        via 172.39.39.14
B       172.20.9.0/24        [20/0]        via 172.39.39.14
B       172.20.10.0/24       [20/0]        via 172.39.39.14
C       172.39.39.0/28       [0/0]         via 172.39.39.1
C       172.39.39.16/28      [0/0]         via 172.39.39.17
B       172.39.39.32/28      [20/0]        via 172.39.39.14
REGA-ESG01-0>
}}

==== DC2 ====

{{console|body=
##bl##REGB-ESG01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,
C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 15

##y##B       10.11.11.0/24        [20/0]        via 10.200.19.1
##y##B       10.11.11.0/24        [20/0]        via 10.200.21.1
B       10.22.22.0/24        [20/0]        via 10.200.19.1
B       10.22.22.0/24        [20/0]        via 10.200.21.1
B       10.100.19.0/24       [20/0]        via 172.39.39.14
B       10.100.21.0/24       [20/0]        via 172.39.39.14
C       10.200.19.0/24       [0/0]         via 10.200.19.2
C       10.200.21.0/24       [0/0]         via 10.200.21.2
##y##B       172.20.1.0/24        [20/0]        via 172.39.39.14
##y##B       172.20.2.0/24        [20/0]        via 172.39.39.14
##y##B       172.20.3.0/24        [20/0]        via 172.39.39.14
B       172.20.8.0/24        [20/0]        via 172.39.39.14
B       172.20.9.0/24        [20/0]        via 172.39.39.14
B       172.20.10.0/24       [20/0]        via 172.39.39.14
C       172.39.39.0/28       [0/0]         via 172.39.39.2
B       172.39.39.16/28      [20/0]        via 172.39.39.14
C       172.39.39.32/28      [0/0]         via 172.39.39.33
REGB-ESG01-0>
}}

=== Cisco1000V routing tables ===

==== DC1 ====

{{console|body=
##bl##rt-a-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.31/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.101/32 is directly connected, GigabitEthernet1
C        10.100.19.0/24 is directly connected, GigabitEthernet3
L        10.100.19.1/32 is directly connected, GigabitEthernet3
B        10.100.21.0/24 [20/0] via 10.100.19.2, 1d17h
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
##y##B        172.20.1.0 [20/0] via 10.100.19.2, 1d17h
##y##B        172.20.2.0 [20/0] via 10.100.19.2, 1d17h
##y##B        172.20.3.0 [20/0] via 10.100.19.2, 1d17h
B        172.20.8.0 [20/0] via 10.100.19.2, 1d17h
B        172.20.9.0 [20/0] via 10.100.19.2, 1d17h
B        172.20.10.0 [20/0] via 10.100.19.2, 1d17h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.19.2, 1d17h
B        172.39.39.16 [20/0] via 10.100.19.2, 1d17h
B        172.39.39.32 [20/0] via 10.100.19.2, 1d17h
rt-a-01#
}}

{{console|body=
##bl##rt-a-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.32/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h
C        10.100.1.0/24 is directly connected, GigabitEthernet1
L        10.100.1.102/32 is directly connected, GigabitEthernet1
B        10.100.19.0/24 [20/0] via 10.100.21.2, 1d17h
C        10.100.21.0/24 is directly connected, GigabitEthernet3
L        10.100.21.1/32 is directly connected, GigabitEthernet3
S        10.200.19.0/24 [1/0] via 10.11.11.253
S        10.200.21.0/24 [1/0] via 10.11.11.253
      172.20.0.0/24 is subnetted, 6 subnets
##y##B        172.20.1.0 [20/0] via 10.100.21.2, 1d17h
##y##B        172.20.2.0 [20/0] via 10.100.21.2, 1d17h
##y##B        172.20.3.0 [20/0] via 10.100.21.2, 1d17h
B        172.20.8.0 [20/0] via 10.100.21.2, 1d17h
B        172.20.9.0 [20/0] via 10.100.21.2, 1d17h
B        172.20.10.0 [20/0] via 10.100.21.2, 1d17h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.100.21.2, 1d17h
B        172.39.39.16 [20/0] via 10.100.21.2, 1d17h
B        172.39.39.32 [20/0] via 10.100.21.2, 1d17h
rt-a-02#
}}

==== DC2 ====

{{console|body=
##bl##rt-b-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.33/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.101/32 is directly connected, GigabitEthernet1
C        10.200.19.0/24 is directly connected, GigabitEthernet3
L        10.200.19.1/32 is directly connected, GigabitEthernet3
B        10.200.21.0/24 [20/0] via 10.200.19.2, 1d18h
      172.20.0.0/24 is subnetted, 6 subnets
##y##B        172.20.1.0 [20/0] via 10.200.19.2, 1d18h
##y##B        172.20.2.0 [20/0] via 10.200.19.2, 1d18h
##y##B        172.20.3.0 [20/0] via 10.200.19.2, 1d18h
B        172.20.8.0 [20/0] via 10.200.19.2, 1d18h
B        172.20.9.0 [20/0] via 10.200.19.2, 1d18h
B        172.20.10.0 [20/0] via 10.200.19.2, 1d18h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.19.2, 1d18h
B        172.39.39.16 [20/0] via 10.200.19.2, 1d17h
B        172.39.39.32 [20/0] via 10.200.19.2, 1d18h
rt-b-01#
}}

{{console|body=
##bl##rt-b-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.11.11.0/24 is directly connected, GigabitEthernet2
L        10.11.11.34/32 is directly connected, GigabitEthernet2
B        10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h
S        10.100.19.0/24 [1/0] via 10.11.11.253
S        10.100.21.0/24 [1/0] via 10.11.11.253
C        10.200.1.0/24 is directly connected, GigabitEthernet1
L        10.200.1.102/32 is directly connected, GigabitEthernet1
B        10.200.19.0/24 [20/0] via 10.200.21.2, 1d18h
C        10.200.21.0/24 is directly connected, GigabitEthernet3
L        10.200.21.1/32 is directly connected, GigabitEthernet3
      172.20.0.0/24 is subnetted, 6 subnets
##y##B        172.20.1.0 [20/0] via 10.200.21.2, 1d18h
##y##B        172.20.2.0 [20/0] via 10.200.21.2, 1d18h
##y##B        172.20.3.0 [20/0] via 10.200.21.2, 1d18h
B        172.20.8.0 [20/0] via 10.200.21.2, 1d18h
B        172.20.9.0 [20/0] via 10.200.21.2, 1d18h
B        172.20.10.0 [20/0] via 10.200.21.2, 1d18h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.200.21.2, 1d18h
B        172.39.39.16 [20/0] via 10.200.21.2, 1d17h
B        172.39.39.32 [20/0] via 10.200.21.2, 1d18h
rt-b-02#
}}

=== Core routing table ===

{{console|body=
##bl##cs-01#show ip route bgp 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 10.11.11.254 to network 0.0.0.0

      172.20.0.0/24 is subnetted, 6 subnets
##y##B        172.20.1.0 [20/0] via 10.11.11.31, 2d02h
##y##B        172.20.2.0 [20/0] via 10.11.11.31, 2d02h
##y##B        172.20.3.0 [20/0] via 10.11.11.31, 2d02h
B        172.20.8.0 [20/0] via 10.11.11.31, 2d02h
B        172.20.9.0 [20/0] via 10.11.11.31, 2d02h
B        172.20.10.0 [20/0] via 10.11.11.31, 2d02h
      172.39.0.0/28 is subnetted, 3 subnets
B        172.39.39.0 [20/0] via 10.11.11.31, 2d02h
B        172.39.39.16 [20/0] via 10.11.11.31, 2d02h
B        172.39.39.32 [20/0] via 10.11.11.34, 2d02h
cs-01#
}}

=== UDLR BGP tables ===

==== DC1 ====

[[File:nsx-routing-blog-14.png|600px]]

==== DC2 ====

The UDLR control VM does not exist in DC2.

=== ESG BGP tables ===

==== DC1 ====

{{console|body=
##bl##REGA-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
##y##    10.11.11.0/24      10.100.19.1       0     100      60   65511 65510 i
##y##    10.11.11.0/24      10.100.21.1       0     100      60   65511 65510 i
##y##  > 10.11.11.0/24      172.39.39.14      0     100      60   65530 i
    10.22.22.0/24      10.100.19.1       0     100      60   65511 65510 i
    10.22.22.0/24      10.100.21.1       0     100      60   65511 65510 i
  > 10.22.22.0/24      172.39.39.14      0     100      60   65530 i
  > 10.100.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.100.21.0/24     0.0.0.0           0     100   32768   ?
  > 10.200.19.0/24     172.39.39.14      0     100      60   65530 ?
  > 10.200.21.0/24     172.39.39.14      0     100      60   65530 ?
  ##y##> 172.20.1.0/24      172.39.39.14      0     100      60   65530 ?
  ##y##> 172.20.2.0/24      172.39.39.14      0     100      60   65530 ?
  ##y##> 172.20.3.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.8.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.9.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.10.0/24     172.39.39.14      0     100      60   65530 ?
    172.39.39.0/28     172.39.39.14      0     100      60   65530 ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.16/28    0.0.0.0           0     100   32768   ?
  > 172.39.39.32/28    172.39.39.14      0     100      60   65530 ?
REGA-ESG01-0> 
}}

==== DC2 ====

{{console|body=
##bl##REGB-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
##y##  > 10.11.11.0/24      10.200.19.1       0     100      60   65512 65510 i
##y##    10.11.11.0/24      10.200.21.1       0     100      60   65512 65510 i
  > 10.22.22.0/24      10.200.19.1       0     100      60   65512 65510 i
    10.22.22.0/24      10.200.21.1       0     100      60   65512 65510 i
  > 10.100.19.0/24     172.39.39.14      0     100      60   65530 ?
  > 10.100.21.0/24     172.39.39.14      0     100      60   65530 ?
  > 10.200.19.0/24     0.0.0.0           0     100   32768   ?
  > 10.200.21.0/24     0.0.0.0           0     100   32768   ?
##y##  > 172.20.1.0/24      172.39.39.14      0     100      60   65530 ?
##y##  > 172.20.2.0/24      172.39.39.14      0     100      60   65530 ?
##y##  > 172.20.3.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.8.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.9.0/24      172.39.39.14      0     100      60   65530 ?
  > 172.20.10.0/24     172.39.39.14      0     100      60   65530 ?
    172.39.39.0/28     172.39.39.14      0     100      60   65530 ?
  > 172.39.39.0/28     0.0.0.0           0     100   32768   ?
  > 172.39.39.16/28    172.39.39.14      0     100      60   65530 ?
  > 172.39.39.32/28    0.0.0.0           0     100   32768   ?
REGB-ESG01-0>
}}

=== Cisco1000V BGP tables ===

==== DC1 ====

{{console|body=
##bl##rt-a-01#show ip bgp
BGP table version is 145, local router ID is 10.11.11.31
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
##y## r   10.11.11.0/24    10.100.19.2                            0 65521 i
##y## r>                   10.11.11.253             0             0 65510 i
 *   10.22.22.0/24    10.100.19.2                            0 65521 i
 *>                   10.11.11.253             0             0 65510 i
 r>  10.100.19.0/24   10.100.19.2                            0 65521 ?
 *>  10.100.21.0/24   10.100.19.2                            0 65521 ?
 r>  10.200.19.0/24   10.100.19.2                            0 65521 ?
 r                    10.11.11.253                           0 65510 65512 65522 ?
 r>  10.200.21.0/24   10.100.19.2                            0 65521 ?
 r                    10.11.11.253                           0 65510 65512 65522 ?
##y## *>  172.20.1.0/24    10.100.19.2                            0 65521 ?
##y## *                    10.11.11.253                           0 65510 65512 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
##y## *>  172.20.2.0/24    10.100.19.2                            0 65521 ?
##y## *                    10.11.11.253                           0 65510 65512 65522 ?
##y## *>  172.20.3.0/24    10.100.19.2                            0 65521 ?
##y## *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.8.0/24    10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.9.0/24    10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.10.0/24   10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.0/28   10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.16/28  10.100.19.2                            0 65521 ?
 *>  172.39.39.32/28  10.100.19.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
rt-a-01#
}}

{{console|body=
##bl##rt-a-02#show ip bgp
BGP table version is 145, local router ID is 10.11.11.32
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 ##y##r   10.11.11.0/24    10.100.21.2                            0 65521 i
 ##y##r>                   10.11.11.253             0             0 65510 i
 *  10.22.22.0/24    10.100.21.2                            0 65521 i
 *>                  10.11.11.253             0             0 65510 i
 *>  10.100.19.0/24   10.100.21.2                            0 65521 ?
 r>  10.100.21.0/24   10.100.21.2                            0 65521 ?
 r>  10.200.19.0/24   10.100.21.2                            0 65521 ?
 r                    10.11.11.253                           0 65510 65512 65522 ?
 r>  10.200.21.0/24   10.100.21.2                            0 65521 ?
 r                    10.11.11.253                           0 65510 65512 65522 ?
##y## *>   172.20.1.0/24    10.100.21.2                            0 65521 ?
##y## *                     10.11.11.253                           0 65510 65512 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
##y## *>   172.20.2.0/24    10.100.21.2                            0 65521 ?
##y## *                    10.11.11.253                           0 65510 65512 65522 ?
##y## *>  172.20.3.0/24    10.100.21.2                            0 65521 ?
##y## *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.8.0/24    10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.9.0/24    10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.20.10.0/24   10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.0/28   10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
 *>  172.39.39.16/28  10.100.21.2                            0 65521 ?
 *>  172.39.39.32/28  10.100.21.2                            0 65521 ?
 *                    10.11.11.253                           0 65510 65512 65522 ?
rt-a-02#
}}

==== DC2 ====

{{console|body=
##bl##rt-b-01#show ip bgp
BGP table version is 152, local router ID is 10.11.11.33
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
##y## r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 r   10.100.19.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>                   10.200.19.2                            0 65522 ?
 r   10.100.21.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>                   10.200.19.2                            0 65522 ?
 r>  10.200.19.0/24   10.200.19.2                            0 65522 ?
 *>  10.200.21.0/24   10.200.19.2                            0 65522 ?
##y## *>  172.20.1.0/24    10.200.19.2                            0 65522 ?
##y## *>  172.20.2.0/24    10.200.19.2                            0 65522 ?
##y## *>  172.20.3.0/24    10.200.19.2                            0 65522 ?
 *>  172.20.8.0/24    10.200.19.2                            0 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.9.0/24    10.200.19.2                            0 65522 ?
 *>  172.20.10.0/24   10.200.19.2                            0 65522 ?
 *>  172.39.39.0/28   10.200.19.2                            0 65522 ?
 *   172.39.39.16/28  10.11.11.253                           0 65510 65511 65521 ?
 *>                   10.200.19.2                            0 65522 ?
 *>  172.39.39.32/28  10.200.19.2                            0 65522 ?
rt-b-01#
}}

{{console|body=
##bl##rt-b-02#show ip bgp
BGP table version is 152, local router ID is 10.11.11.34
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
##y## r>  10.11.11.0/24    10.11.11.253             0             0 65510 i
 *>  10.22.22.0/24    10.11.11.253             0             0 65510 i
 r   10.100.19.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>                   10.200.21.2                            0 65522 ?
 r   10.100.21.0/24   10.11.11.253                           0 65510 65511 65521 ?
 r>                   10.200.21.2                            0 65522 ?
 *>  10.200.19.0/24   10.200.21.2                            0 65522 ?
 r>  10.200.21.0/24   10.200.21.2                            0 65522 ?
##y## *>  172.20.1.0/24    10.200.21.2                            0 65522 ?
##y## *>  172.20.2.0/24    10.200.21.2                            0 65522 ?
##y## *>  172.20.3.0/24    10.200.21.2                            0 65522 ?
 *>  172.20.8.0/24    10.200.21.2                            0 65522 ?
     Network          Next Hop            Metric LocPrf Weight Path
 *>  172.20.9.0/24    10.200.21.2                            0 65522 ?
 *>  172.20.10.0/24   10.200.21.2                            0 65522 ?
 *>  172.39.39.0/28   10.200.21.2                            0 65522 ?
 *   172.39.39.16/28  10.11.11.253                           0 65510 65511 65521 ?
 *>                   10.200.21.2                            0 65522 ?
 *>  172.39.39.32/28  10.200.21.2                            0 65522 ?
rt-b-02#
}}

=== Core BGP tables ===

{{console|body=
##bl##cs-01#show ip bgp
BGP table version is 153, local router ID is 10.11.11.253
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
##y##*> 10.11.11.0/24    0.0.0.0                  0         32768 i
##y##*> 10.22.22.0/24    0.0.0.0                  0         32768 i
r  10.100.19.0/24   10.11.11.32                            0 65511 65521 ?
r                   10.11.11.33                            0 65512 65522 ?
r                   10.11.11.34                            0 65512 65522 ?
r>                  10.11.11.31                            0 65511 65521 ?
r  10.100.21.0/24   10.11.11.32                            0 65511 65521 ?
r                   10.11.11.33                            0 65512 65522 ?
r                   10.11.11.34                            0 65512 65522 ?
r>                  10.11.11.31                            0 65511 65521 ?
r  10.200.19.0/24   10.11.11.31                            0 65511 65521 ?
r                   10.11.11.32                            0 65511 65521 ?
r                   10.11.11.33                            0 65512 65522 ?
r>                  10.11.11.34                            0 65512 65522 ?
r  10.200.21.0/24   10.11.11.31                            0 65511 65521 ?
r                   10.11.11.32                            0 65511 65521 ?
r                   10.11.11.33                            0 65512 65522 ?
r>                  10.11.11.34                            0 65512 65522 ?
##y##*  172.20.1.0/24    10.11.11.31                            0 65511 65521 ?
##y##*                   10.11.11.32                            0 65511 65521 ?
##y##*>                  10.11.11.33                            0 65512 65522 ?
##y##*                   10.11.11.34                            0 65512 65522 ?
##y##*  172.20.2.0/24    10.11.11.31                            0 65511 65521 ?
##y##*                   10.11.11.32                            0 65511 65521 ?
##y##*>                  10.11.11.33                            0 65512 65522 ?
##y##*                   10.11.11.34                            0 65512 65522 ?
##y##*  172.20.3.0/24    10.11.11.31                            0 65511 65521 ?
##y##*                   10.11.11.32                            0 65511 65521 ?
##y##*>                  10.11.11.33                            0 65512 65522 ?
##y##*                   10.11.11.34                            0 65512 65522 ?
*  172.20.8.0/24    10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*  172.20.9.0/24    10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*  172.20.10.0/24   10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*>                  10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*  172.39.39.0/28   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.31                            0 65511 65521 ?
*>                  10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*  172.39.39.16/28  10.11.11.32                            0 65511 65521 ?
*                   10.11.11.33                            0 65512 65522 ?
*                   10.11.11.34                            0 65512 65522 ?
*>                  10.11.11.31                            0 65511 65521 ?
*  172.39.39.32/28  10.11.11.31                            0 65511 65521 ?
*                   10.11.11.32                            0 65511 65521 ?
*                   10.11.11.33                            0 65512 65522 ?
*>                  10.11.11.34                            0 65512 65522 ?
cs-01#
}}

=== Routing path verifications ===

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1 and T1&ndash;WEB&ndash;2 ====

The expectation is that the traffic will route through DC1.

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1     1 ms     1 ms     2 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms     1 ms     1 ms  172.20.1.1

Trace complete
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1 and T1&ndash;WEB&ndash;2 to the external client VM ====

{{console|body=
##bl##root@Web01:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.713 ms  0.649 ms  0.613 ms
 2  172.39.39.2 (172.39.39.2)  0.562 ms  0.599 ms  0.554 ms
 ##y##3  10.200.19.1 (10.200.19.1)  0.741 ms  0.893 ms  1.023 ms
 4  10.11.11.50 (10.11.11.50)  1.231 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.113 ms  0.080 ms  0.065 ms
 2  172.39.39.2 (172.39.39.2)  0.488 ms  0.526 ms  0.481 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.861 ms  0.792 ms  0.729 ms
 4  10.11.11.50 (10.11.11.50)  1.233 ms * *
root@Web02:~#
}}

We are seeing that the ingress AND egress paths that are taken is the path trough DC2.
I want this to be DC1 so I will have to make the weight higher on the UDLR towards EPG-A.

[[File:nsx-routing-blog-15.png|600px]]

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1 and T1&ndash;WEB&ndash;2 ====

The expectation is that the traffic will route through DC1.

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     5 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1 and T1&ndash;WEB&ndash;2 to the external client VM ====

The expectation is that the traffic will route through DC1.

{{console|body=
##bl##root@Web01:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.169 ms  0.083 ms  0.096 ms
 2  172.39.39.1 (172.39.39.1)  0.195 ms  0.265 ms  0.273 ms
##y## 3  10.100.21.1 (10.100.21.1)  1.075 ms  0.964 ms  0.921 ms
 4  10.11.11.50 (10.11.11.50)  0.958 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.190 ms  0.145 ms  0.160 ms
 2  172.39.39.1 (172.39.39.1)  0.445 ms  0.324 ms  0.389 ms
##y## 3  10.100.19.1 (10.100.19.1)  0.679 ms  0.821 ms  0.744 ms
 4  10.11.11.50 (10.11.11.50)  0.938 ms * *
root@Web02:~#
}}

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 ====

The expectation is that the traffic will route through DC1.

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5   202 ms     1 ms     1 ms  172.20.1.3

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5   779 ms     1 ms    <1 ms  172.20.1.4

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 to the external client VM ====

The expectation is that the traffic will route through DC1.

{{console|body=
##bl##root@Web03:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.255 ms  0.228 ms  0.190 ms
 2  172.39.39.1 (172.39.39.1)  0.608 ms  0.568 ms  0.605 ms
##y## 3  10.100.21.1 (10.100.21.1)  1.086 ms  1.197 ms  1.148 ms
 4  10.11.11.50 (10.11.11.50)  1.603 ms * *
root@Web03:~#
}}

{{console|body=
##bl##root@Web04:~# traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.127 ms  0.079 ms  0.058 ms
 2  172.39.39.1 (172.39.39.1)  0.682 ms  0.751 ms  0.702 ms
##y## 3  10.100.21.1 (10.100.21.1)  1.278 ms  1.348 ms  1.300 ms
 4  10.11.11.50 (10.11.11.50)  1.341 ms * *
root@Web04:~#
}}

The routing components in DC1 may fail or in case of a disaster the full site may go down.
In that case, the routing should flow through DC2. 

Let’s test this…

Turn RT-A-01 off.
Verify paths again.
The expectation is that the traffic will route through RT-A-02.

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1, T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 ====

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1     1 ms     2 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     5 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms     1 ms     1 ms  172.20.1.3

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.32
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms     1 ms  172.20.1.4

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1, T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 to the external client VM ====

{{console|body=
##bl##root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.141 ms  0.066 ms  0.118 ms
 2  172.39.39.1 (172.39.39.1)  0.213 ms  0.246 ms  0.196 ms
##y## 3  10.100.21.1 (10.100.21.1)  0.734 ms  0.662 ms  0.612 ms
 4  10.11.11.50 (10.11.11.50)  0.831 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  1.098 ms  1.051 ms  1.055 ms
 2  172.39.39.1 (172.39.39.1)  0.815 ms  0.799 ms  0.851 ms
##y## 3  10.100.21.1 (10.100.21.1)  0.778 ms  0.765 ms  0.980 ms
 4  10.11.11.50 (10.11.11.50)  0.846 ms * *
root@Web02:~#
}}

{{console|body=
##bl##root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.524 ms  0.480 ms  0.435 ms
 2  172.39.39.1 (172.39.39.1)  0.879 ms  0.945 ms  0.920 ms
##y## 3  10.100.21.1 (10.100.21.1)  1.224 ms  1.401 ms  1.400 ms
 4  10.11.11.50 (10.11.11.50)  2.391 ms * *
root@Web03:~#
}}

{{console|body=
##bl##root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.197 ms  0.174 ms  0.194 ms
 2  172.39.39.1 (172.39.39.1)  0.726 ms  0.699 ms  0.649 ms
##y## 3  10.100.21.1 (10.100.21.1)  1.137 ms  1.258 ms  1.255 ms
 4  10.11.11.50 (10.11.11.50)  1.297 ms * *
root@Web04:~#
}}

Turn RT-A-02 off.
Verify paths again.
The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down.

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1 , T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 ====

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms     1 ms    <1 ms  172.20.1.1

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     7 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1     2 ms     1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms     1 ms     1 ms  172.20.1.3

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1     8 ms     3 ms     2 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms     4 ms    <1 ms  172.39.39.14
  5     4 ms    <1 ms    <1 ms  172.20.1.4

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1, T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 to the external client VM ====

{{console|body=
##bl##root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.118 ms  0.079 ms  0.102 ms
 2  172.39.39.2 (172.39.39.2)  0.767 ms  0.753 ms  0.691 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.855 ms  0.869 ms  0.972 ms
 4  10.11.11.50 (10.11.11.50)  1.105 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.135 ms  0.085 ms  0.076 ms
 2  172.39.39.2 (172.39.39.2)  0.919 ms  0.872 ms  0.830 ms
##y## 3  10.200.19.1 (10.200.19.1)  1.138 ms  1.075 ms  1.034 ms
 4  10.11.11.50 (10.11.11.50)  1.795 ms * *
root@Web02:~#
}}

{{console|body=
##bl##root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.199 ms  0.155 ms  0.095 ms
 2  172.39.39.2 (172.39.39.2)  0.353 ms  0.400 ms  0.639 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.722 ms  0.803 ms  0.818 ms
 4  10.11.11.50 (10.11.11.50)  0.961 ms * *
root@Web03:~#
}}

{{console|body=
##bl##root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.184 ms  0.181 ms  0.120 ms
 2  172.39.39.2 (172.39.39.2)  0.435 ms  0.477 ms  0.587 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.867 ms  0.887 ms  0.819 ms
 4  10.11.11.50 (10.11.11.50)  0.913 ms * *
root@Web04:~#
}}

Turn RT-A-01 and RT-A-02 back on and turn ESG-A off.
Verify paths again.
The expectation is that the traffic will route through DC2 the only ESG in DC1 is down.

Before I turned off ESG-A I have verified if the traffic was flowing through DC1 again as an extra verification step. This is not shown below.
For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1.

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1 , T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 ====

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms     1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     2 ms     1 ms    <1 ms  172.20.1.1

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     3 ms     1 ms     1 ms  172.20.1.2

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.3

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.33
##y##  3    <1 ms    <1 ms    <1 ms  10.200.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.4

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1, T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 to the external client VM ====

{{console|body=
##bl##root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.142 ms  0.082 ms  0.094 ms
 2  172.39.39.2 (172.39.39.2)  2.189 ms  2.164 ms  2.132 ms
##y## 3  10.200.19.1 (10.200.19.1)  2.081 ms  5.269 ms  5.250 ms
 4  10.11.11.50 (10.11.11.50)  5.254 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.186 ms  0.094 ms  0.110 ms
 2  172.39.39.2 (172.39.39.2)  0.623 ms  0.637 ms  0.547 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.843 ms  0.773 ms  0.740 ms
 4  10.11.11.50 (10.11.11.50)  1.089 ms * *
root@Web02:~#
}}

{{console|body=
##bl##root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.091 ms  0.108 ms  0.068 ms
 2  172.39.39.2 (172.39.39.2)  0.227 ms  0.274 ms  0.363 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.669 ms  0.645 ms  0.525 ms
 4  10.11.11.50 (10.11.11.50)  0.636 ms * *
root@Web03:~#
}}

{{console|body=
##bl##root@Web04:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.149 ms  0.178 ms  0.127 ms
 2  172.39.39.2 (172.39.39.2)  0.331 ms  0.383 ms  0.300 ms
##y## 3  10.200.19.1 (10.200.19.1)  0.513 ms  0.576 ms  0.536 ms
 4  10.11.11.50 (10.11.11.50)  0.794 ms * *
root@Web04:~#
}}

Turn RT-A-01, RT-A-02 and turn ESG-A back on.
Verify paths again.
The expectation is that everything is back to normal now.

For some reason after turning on the ESG on DC1 the routes kept flowing through DC2.
The same “issue” we saw before.
So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2.

==== Routing path verification from the external client VM to T1&ndash;WEB&ndash;1 , T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 ====

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.1

Tracing route to 172.20.1.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.21.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     1 ms    <1 ms    <1 ms  172.20.1.1

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.2

Tracing route to 172.20.1.2 over a maximum of 30 hops

  1     1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     3 ms    <1 ms    <1 ms  172.20.1.2

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.3

Tracing route to 172.20.1.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms    <1 ms  10.11.11.31
##y##  3     1 ms    <1 ms    <1 ms  10.100.19.2
  4     1 ms    <1 ms    <1 ms  172.39.39.14
  5     5 ms     1 ms     2 ms  172.20.1.3

Trace complete.
}}

{{console|body=
##bl##C:\Users\Administrator>tracert 172.20.1.4

Tracing route to 172.20.1.4 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  cs-01.home.local [10.11.11.253]
  2    <1 ms    <1 ms     1 ms  10.11.11.31
##y##  3    <1 ms    <1 ms    <1 ms  10.100.19.2
  4    <1 ms    <1 ms    <1 ms  172.39.39.14
  5     5 ms     1 ms    <1 ms  172.20.1.4

Trace complete.
}}

==== Routing path verification from the T1&ndash;WEB&ndash;1, T1&ndash;WEB&ndash;2, T1&ndash;WEB&ndash;3 and T1&ndash;WEB&ndash;4 to the external client VM ====

{{console|body=
##bl##root@Web01:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.153 ms  0.118 ms  0.186 ms
 2  172.39.39.1 (172.39.39.1)  0.239 ms  0.272 ms  0.292 ms
##y## 3  10.100.21.1 (10.100.21.1)  0.643 ms  0.611 ms  0.581 ms
 4  10.11.11.50 (10.11.11.50)  0.959 ms * *
root@Web01:~#
}}

{{console|body=
##bl##root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.137 ms  0.083 ms  0.069 ms
 2  172.39.39.1 (172.39.39.1)  0.285 ms  0.386 ms  0.330 ms
##y## 3  10.100.19.1 (10.100.19.1)  0.639 ms  0.542 ms  0.519 ms
 4  10.11.11.50 (10.11.11.50)  0.785 ms * *
root@Web02:~# 
}}

{{console|body=
##bl##root@Web03:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.127 ms  0.072 ms  0.069 ms
 2  172.39.39.1 (172.39.39.1)  0.579 ms  0.704 ms  0.671 ms
##y## 3  10.100.19.1 (10.100.19.1)  0.631 ms  0.896 ms  1.064 ms
 4  10.11.11.50 (10.11.11.50)  1.063 ms * *
root@Web03:~#
}}

{{console|body=
##bl##root@Web02:~#  traceroute 10.11.11.50
traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets
 1  172.20.1.254 (172.20.1.254)  0.142 ms  0.199 ms  0.163 ms
 2  172.39.39.1 (172.39.39.1)  0.555 ms  0.450 ms  0.395 ms
##y## 3  10.100.19.1 (10.100.19.1)  0.884 ms  0.866 ms  0.934 ms
 4  10.11.11.50 (10.11.11.50)  1.103 ms * *
root@Web02:~#
}}

=== Additional investigation on routing table problem ===

I did some investigation “why” DC1 would not become the primary routing path when the DC1 full path is available again.

In the BGP tables in the beginning (show ip bgp) the UDLR only showed ONE path to the 10.11.11.0/24 network.
The UDLR selects 172.39.39.2 (the interface of the ESG in DC2) as the next hop.

{{console|body=
##bl##REGX-UDLR01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
##y##  > 10.11.11.0/24      172.39.39.2      0     100      30   65522 i
REGA-ESG01-0>
}}

The ESG of DC1 showed three paths to the 10.11.11.0/24 network. (The output below is truncated to make it more readable)
One is retrieved from RT-A-01 the other from RT-A-02 and the other one is retrieved from the UDLR on interface 172.39.39.14.

{{console|body=
##bl##REGA-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
##y##    10.11.11.0/24      10.100.19.1       0     100      60   65511 65510 i
##y##    10.11.11.0/24      10.100.21.1       0     100      60   65511 65510 i
##y##  > 10.11.11.0/24      172.39.39.14      0     100      60   65530 i
REGA-ESG01-0>
}}

If we look at the ESG in DC2 we see only two paths to the 10.11.11.0/24 network.
One is retrieved from RT-B-01 the other from RT-B-02. (The output below is truncated to make it more readable)

{{console|body=
##bl##REGB-ESG01-0> show ip bgp

Status codes: s - suppressed, d - damped, > - best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete

    Network            Next Hop       Metric  LocPrf  Weight AS Path
##y##  > 10.11.11.0/24      10.200.19.1       0     100      60   65512 65510 i
##y##    10.11.11.0/24      10.200.21.1       0     100      60   65512 65510 i
REGB-ESG01-0>
}}

I did not find a real answer, other than the fact that the UDLR should display two routes towards the 10.11.11.0/24 network and it should show these routes in the BGP table.
For some reason this is not happening and I have no explanation why.
But in terms of operations this does not matter because the goal is to have two paths to the north (10.11.11.0.24) and if one goes down the other one should take over and this happens as we could see above.

<br />

[[Category: NSX]]
[[Category:Networking]]
[[Category:VMware]]