Routing with NSX using multiple sites: Difference between revisions
From Iwan
m (→Diagram: Fix download links) |
m (Applying replacements) |
||
| Line 64: | Line 64: | ||
* 3-TIER APP | * 3-TIER APP | ||
== Use | == Use–Cases == | ||
I have thought of the following use-cases below. | I have thought of the following use-cases below. | ||
| Line 1,222: | Line 1,222: | ||
=== Routing path verifications === | === Routing path verifications === | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1 and T1–WEB–2 ==== | ||
The expectation is that the traffic will route through DC1. | The expectation is that the traffic will route through DC1. | ||
| Line 1,254: | Line 1,254: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1 and T1–WEB–2 to the external client VM ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,281: | Line 1,281: | ||
[[File:nsx-routing-blog-15.png|600px]] | [[File:nsx-routing-blog-15.png|600px]] | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1 and T1–WEB–2 ==== | ||
The expectation is that the traffic will route through DC1. | The expectation is that the traffic will route through DC1. | ||
| Line 1,313: | Line 1,313: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1 and T1–WEB–2 to the external client VM ==== | ||
The expectation is that the traffic will route through DC1. | The expectation is that the traffic will route through DC1. | ||
| Line 1,337: | Line 1,337: | ||
}} | }} | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–3 and T1–WEB–4 ==== | ||
The expectation is that the traffic will route through DC1. | The expectation is that the traffic will route through DC1. | ||
| Line 1,369: | Line 1,369: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–3 and T1–WEB–4 to the external client VM ==== | ||
The expectation is that the traffic will route through DC1. | The expectation is that the traffic will route through DC1. | ||
| Line 1,402: | Line 1,402: | ||
The expectation is that the traffic will route through RT-A-02. | The expectation is that the traffic will route through RT-A-02. | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,460: | Line 1,460: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,506: | Line 1,506: | ||
The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down. | The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down. | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4 ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,564: | Line 1,564: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,613: | Line 1,613: | ||
For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1. | For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1. | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4 ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,671: | Line 1,671: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,721: | Line 1,721: | ||
So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2. | So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2. | ||
==== Routing path verification from the external client VM to T1 | ==== Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4 ==== | ||
{{console|body= | {{console|body= | ||
| Line 1,779: | Line 1,779: | ||
}} | }} | ||
==== Routing path verification from the T1 | ==== Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM ==== | ||
{{console|body= | {{console|body= | ||
Latest revision as of 20:28, 15 March 2024
This article will be about routing using different protocols and routing domain setups. As you know with routing we advertise network prefixes so that the networks are known on other places on the network. In this process, we can influence the traffic flow when multiple routing/network paths are available. This can be done egress (from inside to outside – typically south to north) and ingress (from outside to inside – typically from north to south).
To influence the path that is taken there are different methods per protocol possible.
In this article, I will only discuss OSPF cost and BGP weight because these are the only mechanisms that are supported by the Edges of NSX. I will also only talk about egress routing as ingress routing is usually influenced on other devices with different attributes / mechanisms.
The general rule for OSPF cost is that the lower the cost is the more preferred the route is. For BGP this is the higher the weight the more preferred the route is.
NSX Multisite deployments
When we are using vSphere together with NSX Multisite deployments are supported. The following options are possible:
- Multisite with multiple vCenters
- With active/passive site egress (Routing Metric or Local Egress Utilized)
- With active/active site egress (Local Egress Utilized)
- Multisite with single vCenter (stretched storage required)
- With active/passive site egress (Routing Metric or Local Egress Utilized)
- With active/active site egress (Local Egress Utilized)
I will only discuss 1a in this article with Routing Metric route manipulation.
Diagram
The following diagram will be used for our setup.
>> LAB100_-_NSX_Routing_based_on_eBGP_v2.pdf
Components
I have used the following components:
CORE
- CS01 = Cisco 3550 L3 Core Switch
DC1
- External PSC (6.5)
- vCenter Server Appliance (6.5)
- NSX Manager – Primary (6.3.1)
- RT-A-01 = CSR1000V
- RT-A-02 = CSR1000V
- ESG-A = NSX ESG (standalone, no ECMP, no HA)
- UDLR-01 (standalone, no HA)
- UDLR-02 (local egress enabled)
- 3-TIER APP
DC2
- External PSC (6.5)
- vCenter Server Appliance (6.5)
- NSX Manager – Secondary (6.3.1)
- RT-B-01 = CSR1000V
- RT-B-02 = CSR1000V
- ESG-B = NSX ESG (standalone, no ECMP, no HA)
- UDLR-02 (local egress enabled)
- 3-TIER APP
Use–Cases
I have thought of the following use-cases below.
As it is not possible to test active/passive and active/active with one UDLR I am using two UDLR instances here.
For testing purpose, I will use three tenants with each having their own Web, App and DB tiers.
Tenant one and two will be using the first UDLR (UDLR-01) with active/passive site egress. Tenant three will be using the second UDLR (UDLR-02) with active/active site egress.
Tenant one
Tenant one will have workloads in DC1 and in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site. When the primary site is down the traffic should exit from the secondary site.
Tenant two
Tenant two will have workloads in DC2 and because it is the routing protocol to determine the egress path the traffic will exit (based on the configuration) from the primary site. When the primary site is down the traffic should exit from the secondary site.
This will be demonstrated in this article by using tenant one because tenant one has workloads in both sites.
Tenant three
Tenant three will have workloads in DC1 and in DC2 and because it is using local egress the traffic will exit from the Cisco CSR1000V routers local to that site. When the primary site is down the traffic should exit from the secondary site and when the secondary is down traffic should exit from the primary.
The following implementation options are available and I am going to deploy and test all of them.
This will not be demonstrated in this article, but in another article.
Routing protocol options
- Option 1
- eBGP peering between the UDLR and the ESG’s
- eBGP peering between the ESG’s and the external routers
- eBGP peering between the external routers and the CORE
- Option 2
- iBGP peering between the UDLR and the ESG’s
- eBGP peering between the ESG’s and the external routers
- eBGP peering between the external routers and the CORE
- Option 3
- iBGP peering between the UDLR and the ESG’s
- iBGP peering between the ESG’s and the external routers
- eBGP peering between the external routers and the CORE
- Option 4
- OSPF peering between the UDLR and the ESG’s (where the ESG is an Area Border Router (ABR))
- OSPF peering between the ESG’s and the external routers (where the ESG is an Area Border Router (ABR))
- eBGP peering between the external routers and the CORE
- Option 5
- OSPF peering between the UDLR and the ESG’s (in the same area)
- OSPF peering between the ESG’s and the external routers (in the same area)
- eBGP peering between the external routers and the CORE
Option 1
Because of the many options available in the setup I want to start with the following: Tenant 1 workloads with the use of UDLR-01 and option 1 routing. The other options will be outlined in other articles.
UDLR configuration
DC1
Manage --> Firewall
- Make sure the firewall is disabled
Manage --> Routing --> Global Configuration
- Enable ECMP
- Make sure that the default gateway is not configured
- Configure a router ID
Manage --> Routing --> BGP
- Enable BGP
- Configure the Local AS
- Disable Graceful Restart
- Configure the BGP peers towards ESG-A and ESG-B
Manage --> Routing --> Route Redistribution
- Enable route redistribution for BGP
- Specify what you allow to redistribute
DC2
The UDLR control VM does not exist in DC2.
ESG configuration
DC1
Manage --> Firewall
- Make sure the firewall is disabled
Manage --> Routing --> Global Configuration
- Enable ECMP
- Make sure that the default gateway is not configured
- Configure a router ID
Manage --> Routing --> BGP
- Enable BGP
- Configure the Local AS
- Disable Graceful Restart
- Configure the BGP peers towards the external routers and the UDLR.
Manage --> Routing --> Route Redistribution
- Enable route redistribution for BGP
- Specify what you allow to redistribute
DC2
Manage --> Firewall
- Make sure the firewall is disabled
Manage --> Routing --> Global Configuration
- Enable ECMP
- Make sure that the default gateway is not configured
- Configure a router ID
Manage --> Routing --> BGP
- Enable BGP
- Configure the Local AS
- Disable Graceful Restart
- Configure the BGP peers towards the external routers and the UDLR
Manage --> Routing --> Route Redistribution
- Enable route redistribution for BGP
- Specify what you allow to redistribute
Cisco1000V configuration
DC1
! root ##bl##hostname rt-a-01 ! router bgp 65511 bgp router-id 10.11.11.31 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.100.19.2 remote-as 65521 neighbor 10.100.19.2 description ESG-A ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.100.19.2 activate exit-address-family ! ip route 10.200.19.0 255.255.255.0 10.11.11.253 ip route 10.200.21.0 255.255.255.0 10.11.11.253 !
! root ##bl##hostname rt-a-02 ! router bgp 65511 bgp router-id 10.11.11.32 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.100.21.2 remote-as 65521 neighbor 10.100.21.2 description ESG-A ! <...> missing? ! ip route 10.200.19.0 255.255.255.0 10.11.11.253 ip route 10.200.21.0 255.255.255.0 10.11.11.253 !
DC2
! root ##bl##hostname rt-b-01 ! router bgp 65512 bgp router-id 10.11.11.33 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.200.19.2 remote-as 65522 neighbor 10.200.19.2 description ESG-B ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.200.19.2 activate exit-address-family ! ip route 10.100.19.0 255.255.255.0 10.11.11.253 ip route 10.100.21.0 255.255.255.0 10.11.11.253 !
! root ##bl##hostname rt-b-02 ! router bgp 65512 bgp router-id 10.11.11.34 bgp log-neighbor-changes neighbor 10.11.11.253 remote-as 65510 neighbor 10.11.11.253 description CS01 neighbor 10.200.21.2 remote-as 65522 neighbor 10.200.21.2 description ESG-B ! address-family ipv4 neighbor 10.11.11.253 activate neighbor 10.200.21.2 activate exit-address-family ! ip route 10.100.19.0 255.255.255.0 10.11.11.253 ip route 10.100.21.0 255.255.255.0 10.11.11.253 !
Core configuration
! root ##bl##hostname cs-01 ! router bgp 65510 bgp router-id 10.11.11.253 bgp log-neighbor-changes neighbor 10.11.11.31 remote-as 65511 neighbor 10.11.11.31 description RT-A-01 neighbor 10.11.11.32 remote-as 65511 neighbor 10.11.11.32 description RT-A-02 neighbor 10.11.11.33 remote-as 65512 neighbor 10.11.11.33 description RT-B-01 neighbor 10.11.11.34 remote-as 65512 neighbor 10.11.11.34 description RT-B-02 ! address-family ipv4 network 10.11.11.0 mask 255.255.255.0 neighbor 10.11.11.31 activate neighbor 10.11.11.31 next-hop-self neighbor 10.11.11.32 activate neighbor 10.11.11.32 next-hop-self neighbor 10.11.11.33 activate neighbor 10.11.11.33 next-hop-self neighbor 10.11.11.34 activate neighbor 10.11.11.34 next-hop-self no auto-summary exit-address-family !
UDLR route peering verification
DC1
root ##bl##REGX-UDLR01-0> show ip bgp neighbors root ##y##BGP neighbor is 172.39.39.1, remote AS 65521, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2886 messages, Sent 2903 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 1 Identifier 0x576e9e8c Route refresh request:received 0 sent 0 Prefixes received 4 sent 12 advertised 12 Connections established 2, dropped 3 Local host: 172.39.39.13, Local port: 179 Remote host: 172.39.39.1, Remote port: 37933 root ##y##BGP neighbor is 172.39.39.2, remote AS 65522, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2903 messages, Sent 2905 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 2 Identifier 0x576e9e8c Route refresh request:received 0 sent 0 Prefixes received 6 sent 10 advertised 10 Connections established 1, dropped 1 Local host: 172.39.39.13, Local port: 55933 Remote host: 172.39.39.2, Remote port: 179 REGX-UDLR01-0>
DC2
The UDLR control VM does not exist in DC2.
ESG route peering verification
DC1
root ##bl##REGA-ESG01-0> show ip bgp neighbors root ##y##BGP neighbor is 10.100.19.1, remote AS 65511, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2758 messages, Sent 2869 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 1 Identifier 0x9c4708ec Route refresh request:received 0 sent 0 Prefixes received 2 sent 15 advertised 15 Connections established 1, dropped 1 Local host: 10.100.19.2, Local port: 25030 Remote host: 10.100.19.1, Remote port: 179 root ##y##BGP neighbor is 10.100.21.1, remote AS 65511, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2754 messages, Sent 2873 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 2 Identifier 0x9c4708ec Route refresh request:received 0 sent 0 Prefixes received 2 sent 15 advertised 15 Connections established 1, dropped 1 Local host: 10.100.21.2, Local port: 31420 Remote host: 10.100.21.1, Remote port: 179 root ##y##BGP neighbor is 172.39.39.13, remote AS 65530, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2877 messages, Sent 2859 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 3 Identifier 0x9c4708ec Route refresh request:received 0 sent 0 Prefixes received 12 sent 4 advertised 4 Connections established 1, dropped 1 Local host: 172.39.39.1, Local port: 37933 Remote host: 172.39.39.13, Remote port: 179 REGA-ESG01-0>
DC2
root ##bl##REGB-ESG01-0> show ip bgp neighbors root ##y##BGP neighbor is 10.200.19.1, remote AS 65512, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2804 messages, Sent 2936 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 1 Identifier 0x359d82ac Route refresh request:received 0 sent 0 Prefixes received 2 sent 13 advertised 13 Connections established 1, dropped 1 Local host: 10.200.19.2, Local port: 43134 Remote host: 10.200.19.1, Remote port: 179 root ##y##BGP neighbor is 10.200.21.1, remote AS 65512, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2801 messages, Sent 2937 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 2 Identifier 0x359d82ac Route refresh request:received 0 sent 0 Prefixes received 2 sent 13 advertised 13 Connections established 1, dropped 1 Local host: 10.200.21.2, Local port: 19055 Remote host: 10.200.21.1, Remote port: 179 root ##y##BGP neighbor is 172.39.39.13, remote AS 65530, root ##y##BGP state = Established, up Hold time is 180, Keep alive interval is 60 seconds Neighbor capabilities: Route refresh: advertised and received Address family IPv4 Unicast:advertised and received Graceful restart Capability:none Restart remain time: 0 Received 2926 messages, Sent 2937 messages Default minimum time between advertisement runs is 30 seconds For Address family IPv4 Unicast:advertised and received Index 3 Identifier 0x359d82ac Route refresh request:received 0 sent 0 Prefixes received 10 sent 6 advertised 6 Connections established 2, dropped 1 Local host: 172.39.39.2, Local port: 179 Remote host: 172.39.39.13, Remote port: 55933 REGB-ESG01-0>
Cisco1000V route peering verification
DC1
root ##bl##rt-a-01#show ip bgp summary BGP router identifier 10.11.11.31, local AS number 65511 BGP table version is 145, main routing table version 145 15 network entries using 3720 bytes of memory 27 path entries using 3240 bytes of memory 4/2 BGP path/bestpath attribute entries using 960 bytes of memory 3 BGP AS-PATH entries using 88 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 8008 total bytes of memory BGP activity 24/9 prefixes, 105/78 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd root ##y##10.11.11.253 4 65510 12586 12590 145 0 0 1w0d 12 root ##y##10.100.19.2 4 65521 2880 2769 145 0 0 1d17h 15 rt-a-01#
root ##bl##rt-a-02#show ip bgp summary BGP router identifier 10.11.11.32, local AS number 65511 BGP table version is 145, main routing table version 145 15 network entries using 3720 bytes of memory 27 path entries using 3240 bytes of memory 4/2 BGP path/bestpath attribute entries using 960 bytes of memory 3 BGP AS-PATH entries using 88 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 8008 total bytes of memory BGP activity 31/16 prefixes, 105/78 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd root ##y##10.11.11.253 4 65510 12588 12583 145 0 0 1w0d 12 root ##y##10.100.21.2 4 65521 2884 2766 145 0 0 1d17h 15 rt-a-02#
DC2
root ##bl##rt-b-01#show ip bgp summary BGP router identifier 10.11.11.33, local AS number 65512 BGP table version is 152, main routing table version 152 15 network entries using 3720 bytes of memory 18 path entries using 2160 bytes of memory 3/2 BGP path/bestpath attribute entries using 720 bytes of memory 3 BGP AS-PATH entries using 88 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 6688 total bytes of memory BGP activity 42/27 prefixes, 104/86 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd root ##y##10.11.11.253 4 65510 12575 12587 152 0 0 1w0d 5 root ##y##10.200.19.2 4 65522 2946 2813 152 0 0 1d18h 13 rt-b-01#
root ##bl##rt-b-02#show ip bgp summary BGP router identifier 10.11.11.34, local AS number 65512 BGP table version is 152, main routing table version 152 15 network entries using 3720 bytes of memory 18 path entries using 2160 bytes of memory 3/2 BGP path/bestpath attribute entries using 720 bytes of memory 3 BGP AS-PATH entries using 88 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 6688 total bytes of memory BGP activity 44/29 prefixes, 104/86 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd root ##y##10.11.11.253 4 65510 12587 12587 152 0 0 1w0d 5 root ##y##10.200.21.2 4 65522 2947 2812 152 0 0 1d18h 13 rt-b-02#
Core route peering verification
root ##bl##cs-01#show ip bgp summary BGP router identifier 10.11.11.253, local AS number 65510 BGP table version is 153, main routing table version 153 15 network entries using 2040 bytes of memory 54 path entries using 2808 bytes of memory 3/3 BGP path/bestpath attribute entries using 372 bytes of memory 2 BGP AS-PATH entries using 48 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 5268 total bytes of memory BGP activity 32/17 prefixes, 256/202 paths, scan interval 60 secs Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd root ##y##10.11.11.31 4 65511 12594 12590 153 0 0 1w0d 13 root ##y##10.11.11.32 4 65511 12586 12591 153 0 0 1w0d 13 root ##y##10.11.11.33 4 65512 12589 12577 153 0 0 1w0d 13 root ##y##10.11.11.34 4 65512 12588 12587 153 0 0 1w0d 13 cs-01#
UDLR routing tables
DC1
root ##bl##REGX-UDLR01-0> show ip route Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived, C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2, IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2, N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 Total number of routes: 15 root ##y##B 10.11.11.0/24 [20/0] via 172.39.39.2 B 10.22.22.0/24 [20/0] via 172.39.39.2 B 10.100.19.0/24 [20/0] via 172.39.39.1 B 10.100.21.0/24 [20/0] via 172.39.39.1 B 10.200.19.0/24 [20/0] via 172.39.39.2 B 10.200.21.0/24 [20/0] via 172.39.39.2 root ##y##C 172.20.1.0/24 [0/0] via 172.20.1.254 root ##y##C 172.20.2.0/24 [0/0] via 172.20.2.254 root ##y##C 172.20.3.0/24 [0/0] via 172.20.3.254 C 172.20.8.0/24 [0/0] via 172.20.8.254 C 172.20.9.0/24 [0/0] via 172.20.9.254 C 172.20.10.0/24 [0/0] via 172.20.10.254 C 172.39.39.0/28 [0/0] via 172.39.39.13 B 172.39.39.16/28 [20/0] via 172.39.39.1 B 172.39.39.32/28 [20/0] via 172.39.39.2 REGX-UDLR01-0>
DC2
The UDLR control VM does not exist in DC2.
ESG routing tables
DC1
root ##bl##REGA-ESG01-0> show ip route Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived, C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2, IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2, N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 Total number of routes: 15 root ##y##B 10.11.11.0/24 [20/0] via 172.39.39.14 B 10.22.22.0/24 [20/0] via 172.39.39.14 C 10.100.19.0/24 [0/0] via 10.100.19.2 C 10.100.21.0/24 [0/0] via 10.100.21.2 B 10.200.19.0/24 [20/0] via 172.39.39.14 B 10.200.21.0/24 [20/0] via 172.39.39.14 root ##y##B 172.20.1.0/24 [20/0] via 172.39.39.14 root ##y##B 172.20.2.0/24 [20/0] via 172.39.39.14 root ##y##B 172.20.3.0/24 [20/0] via 172.39.39.14 B 172.20.8.0/24 [20/0] via 172.39.39.14 B 172.20.9.0/24 [20/0] via 172.39.39.14 B 172.20.10.0/24 [20/0] via 172.39.39.14 C 172.39.39.0/28 [0/0] via 172.39.39.1 C 172.39.39.16/28 [0/0] via 172.39.39.17 B 172.39.39.32/28 [20/0] via 172.39.39.14 REGA-ESG01-0>
DC2
root ##bl##REGB-ESG01-0> show ip route Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived, C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2, IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2, N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 Total number of routes: 15 root ##y##B 10.11.11.0/24 [20/0] via 10.200.19.1 root ##y##B 10.11.11.0/24 [20/0] via 10.200.21.1 B 10.22.22.0/24 [20/0] via 10.200.19.1 B 10.22.22.0/24 [20/0] via 10.200.21.1 B 10.100.19.0/24 [20/0] via 172.39.39.14 B 10.100.21.0/24 [20/0] via 172.39.39.14 C 10.200.19.0/24 [0/0] via 10.200.19.2 C 10.200.21.0/24 [0/0] via 10.200.21.2 root ##y##B 172.20.1.0/24 [20/0] via 172.39.39.14 root ##y##B 172.20.2.0/24 [20/0] via 172.39.39.14 root ##y##B 172.20.3.0/24 [20/0] via 172.39.39.14 B 172.20.8.0/24 [20/0] via 172.39.39.14 B 172.20.9.0/24 [20/0] via 172.39.39.14 B 172.20.10.0/24 [20/0] via 172.39.39.14 C 172.39.39.0/28 [0/0] via 172.39.39.2 B 172.39.39.16/28 [20/0] via 172.39.39.14 C 172.39.39.32/28 [0/0] via 172.39.39.33 REGB-ESG01-0>
Cisco1000V routing tables
DC1
root ##bl##rt-a-01#show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks C 10.11.11.0/24 is directly connected, GigabitEthernet2 L 10.11.11.31/32 is directly connected, GigabitEthernet2 B 10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h C 10.100.1.0/24 is directly connected, GigabitEthernet1 L 10.100.1.101/32 is directly connected, GigabitEthernet1 C 10.100.19.0/24 is directly connected, GigabitEthernet3 L 10.100.19.1/32 is directly connected, GigabitEthernet3 B 10.100.21.0/24 [20/0] via 10.100.19.2, 1d17h S 10.200.19.0/24 [1/0] via 10.11.11.253 S 10.200.21.0/24 [1/0] via 10.11.11.253 172.20.0.0/24 is subnetted, 6 subnets root ##y##B 172.20.1.0 [20/0] via 10.100.19.2, 1d17h root ##y##B 172.20.2.0 [20/0] via 10.100.19.2, 1d17h root ##y##B 172.20.3.0 [20/0] via 10.100.19.2, 1d17h B 172.20.8.0 [20/0] via 10.100.19.2, 1d17h B 172.20.9.0 [20/0] via 10.100.19.2, 1d17h B 172.20.10.0 [20/0] via 10.100.19.2, 1d17h 172.39.0.0/28 is subnetted, 3 subnets B 172.39.39.0 [20/0] via 10.100.19.2, 1d17h B 172.39.39.16 [20/0] via 10.100.19.2, 1d17h B 172.39.39.32 [20/0] via 10.100.19.2, 1d17h rt-a-01#
root ##bl##rt-a-02#show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks C 10.11.11.0/24 is directly connected, GigabitEthernet2 L 10.11.11.32/32 is directly connected, GigabitEthernet2 B 10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h C 10.100.1.0/24 is directly connected, GigabitEthernet1 L 10.100.1.102/32 is directly connected, GigabitEthernet1 B 10.100.19.0/24 [20/0] via 10.100.21.2, 1d17h C 10.100.21.0/24 is directly connected, GigabitEthernet3 L 10.100.21.1/32 is directly connected, GigabitEthernet3 S 10.200.19.0/24 [1/0] via 10.11.11.253 S 10.200.21.0/24 [1/0] via 10.11.11.253 172.20.0.0/24 is subnetted, 6 subnets root ##y##B 172.20.1.0 [20/0] via 10.100.21.2, 1d17h root ##y##B 172.20.2.0 [20/0] via 10.100.21.2, 1d17h root ##y##B 172.20.3.0 [20/0] via 10.100.21.2, 1d17h B 172.20.8.0 [20/0] via 10.100.21.2, 1d17h B 172.20.9.0 [20/0] via 10.100.21.2, 1d17h B 172.20.10.0 [20/0] via 10.100.21.2, 1d17h 172.39.0.0/28 is subnetted, 3 subnets B 172.39.39.0 [20/0] via 10.100.21.2, 1d17h B 172.39.39.16 [20/0] via 10.100.21.2, 1d17h B 172.39.39.32 [20/0] via 10.100.21.2, 1d17h rt-a-02#
DC2
root ##bl##rt-b-01#show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks C 10.11.11.0/24 is directly connected, GigabitEthernet2 L 10.11.11.33/32 is directly connected, GigabitEthernet2 B 10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h S 10.100.19.0/24 [1/0] via 10.11.11.253 S 10.100.21.0/24 [1/0] via 10.11.11.253 C 10.200.1.0/24 is directly connected, GigabitEthernet1 L 10.200.1.101/32 is directly connected, GigabitEthernet1 C 10.200.19.0/24 is directly connected, GigabitEthernet3 L 10.200.19.1/32 is directly connected, GigabitEthernet3 B 10.200.21.0/24 [20/0] via 10.200.19.2, 1d18h 172.20.0.0/24 is subnetted, 6 subnets root ##y##B 172.20.1.0 [20/0] via 10.200.19.2, 1d18h root ##y##B 172.20.2.0 [20/0] via 10.200.19.2, 1d18h root ##y##B 172.20.3.0 [20/0] via 10.200.19.2, 1d18h B 172.20.8.0 [20/0] via 10.200.19.2, 1d18h B 172.20.9.0 [20/0] via 10.200.19.2, 1d18h B 172.20.10.0 [20/0] via 10.200.19.2, 1d18h 172.39.0.0/28 is subnetted, 3 subnets B 172.39.39.0 [20/0] via 10.200.19.2, 1d18h B 172.39.39.16 [20/0] via 10.200.19.2, 1d17h B 172.39.39.32 [20/0] via 10.200.19.2, 1d18h rt-b-01#
root ##bl##rt-b-02#show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks C 10.11.11.0/24 is directly connected, GigabitEthernet2 L 10.11.11.34/32 is directly connected, GigabitEthernet2 B 10.22.22.0/24 [20/0] via 10.11.11.253, 1d18h S 10.100.19.0/24 [1/0] via 10.11.11.253 S 10.100.21.0/24 [1/0] via 10.11.11.253 C 10.200.1.0/24 is directly connected, GigabitEthernet1 L 10.200.1.102/32 is directly connected, GigabitEthernet1 B 10.200.19.0/24 [20/0] via 10.200.21.2, 1d18h C 10.200.21.0/24 is directly connected, GigabitEthernet3 L 10.200.21.1/32 is directly connected, GigabitEthernet3 172.20.0.0/24 is subnetted, 6 subnets root ##y##B 172.20.1.0 [20/0] via 10.200.21.2, 1d18h root ##y##B 172.20.2.0 [20/0] via 10.200.21.2, 1d18h root ##y##B 172.20.3.0 [20/0] via 10.200.21.2, 1d18h B 172.20.8.0 [20/0] via 10.200.21.2, 1d18h B 172.20.9.0 [20/0] via 10.200.21.2, 1d18h B 172.20.10.0 [20/0] via 10.200.21.2, 1d18h 172.39.0.0/28 is subnetted, 3 subnets B 172.39.39.0 [20/0] via 10.200.21.2, 1d18h B 172.39.39.16 [20/0] via 10.200.21.2, 1d17h B 172.39.39.32 [20/0] via 10.200.21.2, 1d18h rt-b-02#
Core routing table
root ##bl##cs-01#show ip route bgp Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP + - replicated route, % - next hop override Gateway of last resort is 10.11.11.254 to network 0.0.0.0 172.20.0.0/24 is subnetted, 6 subnets root ##y##B 172.20.1.0 [20/0] via 10.11.11.31, 2d02h root ##y##B 172.20.2.0 [20/0] via 10.11.11.31, 2d02h root ##y##B 172.20.3.0 [20/0] via 10.11.11.31, 2d02h B 172.20.8.0 [20/0] via 10.11.11.31, 2d02h B 172.20.9.0 [20/0] via 10.11.11.31, 2d02h B 172.20.10.0 [20/0] via 10.11.11.31, 2d02h 172.39.0.0/28 is subnetted, 3 subnets B 172.39.39.0 [20/0] via 10.11.11.31, 2d02h B 172.39.39.16 [20/0] via 10.11.11.31, 2d02h B 172.39.39.32 [20/0] via 10.11.11.34, 2d02h cs-01#
UDLR BGP tables
DC1
DC2
The UDLR control VM does not exist in DC2.
ESG BGP tables
DC1
root ##bl##REGA-ESG01-0> show ip bgp Status codes: s - suppressed, d - damped, > - best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight AS Path root ##y## 10.11.11.0/24 10.100.19.1 0 100 60 65511 65510 i root ##y## 10.11.11.0/24 10.100.21.1 0 100 60 65511 65510 i root ##y## > 10.11.11.0/24 172.39.39.14 0 100 60 65530 i 10.22.22.0/24 10.100.19.1 0 100 60 65511 65510 i 10.22.22.0/24 10.100.21.1 0 100 60 65511 65510 i > 10.22.22.0/24 172.39.39.14 0 100 60 65530 i > 10.100.19.0/24 0.0.0.0 0 100 32768 ? > 10.100.21.0/24 0.0.0.0 0 100 32768 ? > 10.200.19.0/24 172.39.39.14 0 100 60 65530 ? > 10.200.21.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.1.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.2.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.3.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.8.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.9.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.10.0/24 172.39.39.14 0 100 60 65530 ? 172.39.39.0/28 172.39.39.14 0 100 60 65530 ? > 172.39.39.0/28 0.0.0.0 0 100 32768 ? > 172.39.39.16/28 0.0.0.0 0 100 32768 ? > 172.39.39.32/28 172.39.39.14 0 100 60 65530 ? REGA-ESG01-0>
DC2
root ##bl##REGB-ESG01-0> show ip bgp Status codes: s - suppressed, d - damped, > - best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight AS Path root ##y## > 10.11.11.0/24 10.200.19.1 0 100 60 65512 65510 i root ##y## 10.11.11.0/24 10.200.21.1 0 100 60 65512 65510 i > 10.22.22.0/24 10.200.19.1 0 100 60 65512 65510 i 10.22.22.0/24 10.200.21.1 0 100 60 65512 65510 i > 10.100.19.0/24 172.39.39.14 0 100 60 65530 ? > 10.100.21.0/24 172.39.39.14 0 100 60 65530 ? > 10.200.19.0/24 0.0.0.0 0 100 32768 ? > 10.200.21.0/24 0.0.0.0 0 100 32768 ? root ##y## > 172.20.1.0/24 172.39.39.14 0 100 60 65530 ? root ##y## > 172.20.2.0/24 172.39.39.14 0 100 60 65530 ? root ##y## > 172.20.3.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.8.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.9.0/24 172.39.39.14 0 100 60 65530 ? > 172.20.10.0/24 172.39.39.14 0 100 60 65530 ? 172.39.39.0/28 172.39.39.14 0 100 60 65530 ? > 172.39.39.0/28 0.0.0.0 0 100 32768 ? > 172.39.39.16/28 172.39.39.14 0 100 60 65530 ? > 172.39.39.32/28 0.0.0.0 0 100 32768 ? REGB-ESG01-0>
Cisco1000V BGP tables
DC1
root ##bl##rt-a-01#show ip bgp BGP table version is 145, local router ID is 10.11.11.31 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path root ##y## r 10.11.11.0/24 10.100.19.2 0 65521 i root ##y## r> 10.11.11.253 0 0 65510 i * 10.22.22.0/24 10.100.19.2 0 65521 i *> 10.11.11.253 0 0 65510 i r> 10.100.19.0/24 10.100.19.2 0 65521 ? *> 10.100.21.0/24 10.100.19.2 0 65521 ? r> 10.200.19.0/24 10.100.19.2 0 65521 ? r 10.11.11.253 0 65510 65512 65522 ? r> 10.200.21.0/24 10.100.19.2 0 65521 ? r 10.11.11.253 0 65510 65512 65522 ? root ##y## *> 172.20.1.0/24 10.100.19.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? Network Next Hop Metric LocPrf Weight Path root ##y## *> 172.20.2.0/24 10.100.19.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? root ##y## *> 172.20.3.0/24 10.100.19.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.8.0/24 10.100.19.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.9.0/24 10.100.19.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.10.0/24 10.100.19.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.39.39.0/28 10.100.19.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.39.39.16/28 10.100.19.2 0 65521 ? *> 172.39.39.32/28 10.100.19.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? rt-a-01#
root ##bl##rt-a-02#show ip bgp BGP table version is 145, local router ID is 10.11.11.32 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path r 10.11.11.0/24 10.100.21.2 0 65521 i r> 10.11.11.253 0 0 65510 i * 10.22.22.0/24 10.100.21.2 0 65521 i *> 10.11.11.253 0 0 65510 i *> 10.100.19.0/24 10.100.21.2 0 65521 ? r> 10.100.21.0/24 10.100.21.2 0 65521 ? r> 10.200.19.0/24 10.100.21.2 0 65521 ? r 10.11.11.253 0 65510 65512 65522 ? r> 10.200.21.0/24 10.100.21.2 0 65521 ? r 10.11.11.253 0 65510 65512 65522 ? root ##y## *> 172.20.1.0/24 10.100.21.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? Network Next Hop Metric LocPrf Weight Path root ##y## *> 172.20.2.0/24 10.100.21.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? root ##y## *> 172.20.3.0/24 10.100.21.2 0 65521 ? root ##y## * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.8.0/24 10.100.21.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.9.0/24 10.100.21.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.20.10.0/24 10.100.21.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.39.39.0/28 10.100.21.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? *> 172.39.39.16/28 10.100.21.2 0 65521 ? *> 172.39.39.32/28 10.100.21.2 0 65521 ? * 10.11.11.253 0 65510 65512 65522 ? rt-a-02#
DC2
root ##bl##rt-b-01#show ip bgp BGP table version is 152, local router ID is 10.11.11.33 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path root ##y## r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i r 10.100.19.0/24 10.11.11.253 0 65510 65511 65521 ? r> 10.200.19.2 0 65522 ? r 10.100.21.0/24 10.11.11.253 0 65510 65511 65521 ? r> 10.200.19.2 0 65522 ? r> 10.200.19.0/24 10.200.19.2 0 65522 ? *> 10.200.21.0/24 10.200.19.2 0 65522 ? root ##y## *> 172.20.1.0/24 10.200.19.2 0 65522 ? root ##y## *> 172.20.2.0/24 10.200.19.2 0 65522 ? root ##y## *> 172.20.3.0/24 10.200.19.2 0 65522 ? *> 172.20.8.0/24 10.200.19.2 0 65522 ? Network Next Hop Metric LocPrf Weight Path *> 172.20.9.0/24 10.200.19.2 0 65522 ? *> 172.20.10.0/24 10.200.19.2 0 65522 ? *> 172.39.39.0/28 10.200.19.2 0 65522 ? * 172.39.39.16/28 10.11.11.253 0 65510 65511 65521 ? *> 10.200.19.2 0 65522 ? *> 172.39.39.32/28 10.200.19.2 0 65522 ? rt-b-01#
root ##bl##rt-b-02#show ip bgp BGP table version is 152, local router ID is 10.11.11.34 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path root ##y## r> 10.11.11.0/24 10.11.11.253 0 0 65510 i *> 10.22.22.0/24 10.11.11.253 0 0 65510 i r 10.100.19.0/24 10.11.11.253 0 65510 65511 65521 ? r> 10.200.21.2 0 65522 ? r 10.100.21.0/24 10.11.11.253 0 65510 65511 65521 ? r> 10.200.21.2 0 65522 ? *> 10.200.19.0/24 10.200.21.2 0 65522 ? r> 10.200.21.0/24 10.200.21.2 0 65522 ? root ##y## *> 172.20.1.0/24 10.200.21.2 0 65522 ? root ##y## *> 172.20.2.0/24 10.200.21.2 0 65522 ? root ##y## *> 172.20.3.0/24 10.200.21.2 0 65522 ? *> 172.20.8.0/24 10.200.21.2 0 65522 ? Network Next Hop Metric LocPrf Weight Path *> 172.20.9.0/24 10.200.21.2 0 65522 ? *> 172.20.10.0/24 10.200.21.2 0 65522 ? *> 172.39.39.0/28 10.200.21.2 0 65522 ? * 172.39.39.16/28 10.11.11.253 0 65510 65511 65521 ? *> 10.200.21.2 0 65522 ? *> 172.39.39.32/28 10.200.21.2 0 65522 ? rt-b-02#
Core BGP tables
root ##bl##cs-01#show ip bgp BGP table version is 153, local router ID is 10.11.11.253 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, x best-external Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path root ##y##*> 10.11.11.0/24 0.0.0.0 0 32768 i root ##y##*> 10.22.22.0/24 0.0.0.0 0 32768 i r 10.100.19.0/24 10.11.11.32 0 65511 65521 ? r 10.11.11.33 0 65512 65522 ? r 10.11.11.34 0 65512 65522 ? r> 10.11.11.31 0 65511 65521 ? r 10.100.21.0/24 10.11.11.32 0 65511 65521 ? r 10.11.11.33 0 65512 65522 ? r 10.11.11.34 0 65512 65522 ? r> 10.11.11.31 0 65511 65521 ? r 10.200.19.0/24 10.11.11.31 0 65511 65521 ? r 10.11.11.32 0 65511 65521 ? r 10.11.11.33 0 65512 65522 ? r> 10.11.11.34 0 65512 65522 ? r 10.200.21.0/24 10.11.11.31 0 65511 65521 ? r 10.11.11.32 0 65511 65521 ? r 10.11.11.33 0 65512 65522 ? r> 10.11.11.34 0 65512 65522 ? root ##y##* 172.20.1.0/24 10.11.11.31 0 65511 65521 ? root ##y##* 10.11.11.32 0 65511 65521 ? root ##y##*> 10.11.11.33 0 65512 65522 ? root ##y##* 10.11.11.34 0 65512 65522 ? root ##y##* 172.20.2.0/24 10.11.11.31 0 65511 65521 ? root ##y##* 10.11.11.32 0 65511 65521 ? root ##y##*> 10.11.11.33 0 65512 65522 ? root ##y##* 10.11.11.34 0 65512 65522 ? root ##y##* 172.20.3.0/24 10.11.11.31 0 65511 65521 ? root ##y##* 10.11.11.32 0 65511 65521 ? root ##y##*> 10.11.11.33 0 65512 65522 ? root ##y##* 10.11.11.34 0 65512 65522 ? * 172.20.8.0/24 10.11.11.31 0 65511 65521 ? * 10.11.11.32 0 65511 65521 ? *> 10.11.11.33 0 65512 65522 ? * 10.11.11.34 0 65512 65522 ? * 172.20.9.0/24 10.11.11.31 0 65511 65521 ? * 10.11.11.32 0 65511 65521 ? *> 10.11.11.33 0 65512 65522 ? * 10.11.11.34 0 65512 65522 ? * 172.20.10.0/24 10.11.11.31 0 65511 65521 ? * 10.11.11.32 0 65511 65521 ? *> 10.11.11.33 0 65512 65522 ? * 10.11.11.34 0 65512 65522 ? * 172.39.39.0/28 10.11.11.32 0 65511 65521 ? * 10.11.11.31 0 65511 65521 ? *> 10.11.11.33 0 65512 65522 ? * 10.11.11.34 0 65512 65522 ? * 172.39.39.16/28 10.11.11.32 0 65511 65521 ? * 10.11.11.33 0 65512 65522 ? * 10.11.11.34 0 65512 65522 ? *> 10.11.11.31 0 65511 65521 ? * 172.39.39.32/28 10.11.11.31 0 65511 65521 ? * 10.11.11.32 0 65511 65521 ? * 10.11.11.33 0 65512 65522 ? *> 10.11.11.34 0 65512 65522 ? cs-01#
Routing path verifications
Routing path verification from the external client VM to T1–WEB–1 and T1–WEB–2
The expectation is that the traffic will route through DC1.
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 1 ms 1 ms 2 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms 1 ms 1 ms 172.20.1.1 Trace complete
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 <1 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.2 Trace complete.
Routing path verification from the T1–WEB–1 and T1–WEB–2 to the external client VM
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.713 ms 0.649 ms 0.613 ms 2 172.39.39.2 (172.39.39.2) 0.562 ms 0.599 ms 0.554 ms 3 10.200.19.1 (10.200.19.1) 0.741 ms 0.893 ms 1.023 ms 4 10.11.11.50 (10.11.11.50) 1.231 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.113 ms 0.080 ms 0.065 ms 2 172.39.39.2 (172.39.39.2) 0.488 ms 0.526 ms 0.481 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.861 ms 0.792 ms 0.729 ms 4 10.11.11.50 (10.11.11.50) 1.233 ms * * root@Web02:~#
We are seeing that the ingress AND egress paths that are taken is the path trough DC2. I want this to be DC1 so I will have to make the weight higher on the UDLR towards EPG-A.
Routing path verification from the external client VM to T1–WEB–1 and T1–WEB–2
The expectation is that the traffic will route through DC1.
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.1 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 5 ms <1 ms <1 ms 172.20.1.2 Trace complete.
Routing path verification from the T1–WEB–1 and T1–WEB–2 to the external client VM
The expectation is that the traffic will route through DC1.
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.169 ms 0.083 ms 0.096 ms 2 172.39.39.1 (172.39.39.1) 0.195 ms 0.265 ms 0.273 ms root ##y## 3 10.100.21.1 (10.100.21.1) 1.075 ms 0.964 ms 0.921 ms 4 10.11.11.50 (10.11.11.50) 0.958 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.190 ms 0.145 ms 0.160 ms 2 172.39.39.1 (172.39.39.1) 0.445 ms 0.324 ms 0.389 ms root ##y## 3 10.100.19.1 (10.100.19.1) 0.679 ms 0.821 ms 0.744 ms 4 10.11.11.50 (10.11.11.50) 0.938 ms * * root@Web02:~#
Routing path verification from the external client VM to T1–WEB–3 and T1–WEB–4
The expectation is that the traffic will route through DC1.
root ##bl##C:\Users\Administrator>tracert 172.20.1.3 Tracing route to 172.20.1.3 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 202 ms 1 ms 1 ms 172.20.1.3 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.4 Tracing route to 172.20.1.4 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 779 ms 1 ms <1 ms 172.20.1.4 Trace complete.
Routing path verification from the T1–WEB–3 and T1–WEB–4 to the external client VM
The expectation is that the traffic will route through DC1.
root ##bl##root@Web03:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.255 ms 0.228 ms 0.190 ms 2 172.39.39.1 (172.39.39.1) 0.608 ms 0.568 ms 0.605 ms root ##y## 3 10.100.21.1 (10.100.21.1) 1.086 ms 1.197 ms 1.148 ms 4 10.11.11.50 (10.11.11.50) 1.603 ms * * root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.127 ms 0.079 ms 0.058 ms 2 172.39.39.1 (172.39.39.1) 0.682 ms 0.751 ms 0.702 ms root ##y## 3 10.100.21.1 (10.100.21.1) 1.278 ms 1.348 ms 1.300 ms 4 10.11.11.50 (10.11.11.50) 1.341 ms * * root@Web04:~#
The routing components in DC1 may fail or in case of a disaster the full site may go down. In that case, the routing should flow through DC2.
Let’s test this…
Turn RT-A-01 off. Verify paths again. The expectation is that the traffic will route through RT-A-02.
Routing path verification from the external client VM to T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 1 ms 2 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.1 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 <1 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 5 ms <1 ms <1 ms 172.20.1.2 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.3 Tracing route to 172.20.1.3 over a maximum of 30 hops 1 2 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 2 ms 1 ms 1 ms 172.20.1.3 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.4 Tracing route to 172.20.1.4 over a maximum of 30 hops 1 2 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.32 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms 1 ms 172.20.1.4 Trace complete.
Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.141 ms 0.066 ms 0.118 ms 2 172.39.39.1 (172.39.39.1) 0.213 ms 0.246 ms 0.196 ms root ##y## 3 10.100.21.1 (10.100.21.1) 0.734 ms 0.662 ms 0.612 ms 4 10.11.11.50 (10.11.11.50) 0.831 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 1.098 ms 1.051 ms 1.055 ms 2 172.39.39.1 (172.39.39.1) 0.815 ms 0.799 ms 0.851 ms root ##y## 3 10.100.21.1 (10.100.21.1) 0.778 ms 0.765 ms 0.980 ms 4 10.11.11.50 (10.11.11.50) 0.846 ms * * root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.524 ms 0.480 ms 0.435 ms 2 172.39.39.1 (172.39.39.1) 0.879 ms 0.945 ms 0.920 ms root ##y## 3 10.100.21.1 (10.100.21.1) 1.224 ms 1.401 ms 1.400 ms 4 10.11.11.50 (10.11.11.50) 2.391 ms * * root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.197 ms 0.174 ms 0.194 ms 2 172.39.39.1 (172.39.39.1) 0.726 ms 0.699 ms 0.649 ms root ##y## 3 10.100.21.1 (10.100.21.1) 1.137 ms 1.258 ms 1.255 ms 4 10.11.11.50 (10.11.11.50) 1.297 ms * * root@Web04:~#
Turn RT-A-02 off. Verify paths again. The expectation is that the traffic will route through DC2 as both upstream routers in DC1 are down.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 2 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms 1 ms <1 ms 172.20.1.1 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 <1 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 7 ms <1 ms <1 ms 172.20.1.2 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.3 Tracing route to 172.20.1.3 over a maximum of 30 hops 1 2 ms 1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 1 ms 1 ms 1 ms 172.20.1.3 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.4 Tracing route to 172.20.1.4 over a maximum of 30 hops 1 8 ms 3 ms 2 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms 4 ms <1 ms 172.39.39.14 5 4 ms <1 ms <1 ms 172.20.1.4 Trace complete.
Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.118 ms 0.079 ms 0.102 ms 2 172.39.39.2 (172.39.39.2) 0.767 ms 0.753 ms 0.691 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.855 ms 0.869 ms 0.972 ms 4 10.11.11.50 (10.11.11.50) 1.105 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.135 ms 0.085 ms 0.076 ms 2 172.39.39.2 (172.39.39.2) 0.919 ms 0.872 ms 0.830 ms root ##y## 3 10.200.19.1 (10.200.19.1) 1.138 ms 1.075 ms 1.034 ms 4 10.11.11.50 (10.11.11.50) 1.795 ms * * root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.199 ms 0.155 ms 0.095 ms 2 172.39.39.2 (172.39.39.2) 0.353 ms 0.400 ms 0.639 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.722 ms 0.803 ms 0.818 ms 4 10.11.11.50 (10.11.11.50) 0.961 ms * * root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.184 ms 0.181 ms 0.120 ms 2 172.39.39.2 (172.39.39.2) 0.435 ms 0.477 ms 0.587 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.867 ms 0.887 ms 0.819 ms 4 10.11.11.50 (10.11.11.50) 0.913 ms * * root@Web04:~#
Turn RT-A-01 and RT-A-02 back on and turn ESG-A off. Verify paths again. The expectation is that the traffic will route through DC2 the only ESG in DC1 is down.
Before I turned off ESG-A I have verified if the traffic was flowing through DC1 again as an extra verification step. This is not shown below. For some reason the traffic kept flowing through DC1 so I had to turn BGP off/on again on the ESG in DC2 and the this caused the traffic to flow back trough DC1.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 <1 ms <1 ms 1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 2 ms 1 ms <1 ms 172.20.1.1 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 3 ms 1 ms 1 ms 172.20.1.2 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.3 Tracing route to 172.20.1.3 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.3 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.4 Tracing route to 172.20.1.4 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.33 root ##y## 3 <1 ms <1 ms <1 ms 10.200.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.4 Trace complete.
Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.142 ms 0.082 ms 0.094 ms 2 172.39.39.2 (172.39.39.2) 2.189 ms 2.164 ms 2.132 ms root ##y## 3 10.200.19.1 (10.200.19.1) 2.081 ms 5.269 ms 5.250 ms 4 10.11.11.50 (10.11.11.50) 5.254 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.186 ms 0.094 ms 0.110 ms 2 172.39.39.2 (172.39.39.2) 0.623 ms 0.637 ms 0.547 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.843 ms 0.773 ms 0.740 ms 4 10.11.11.50 (10.11.11.50) 1.089 ms * * root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.091 ms 0.108 ms 0.068 ms 2 172.39.39.2 (172.39.39.2) 0.227 ms 0.274 ms 0.363 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.669 ms 0.645 ms 0.525 ms 4 10.11.11.50 (10.11.11.50) 0.636 ms * * root@Web03:~#
root ##bl##root@Web04:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.149 ms 0.178 ms 0.127 ms 2 172.39.39.2 (172.39.39.2) 0.331 ms 0.383 ms 0.300 ms root ##y## 3 10.200.19.1 (10.200.19.1) 0.513 ms 0.576 ms 0.536 ms 4 10.11.11.50 (10.11.11.50) 0.794 ms * * root@Web04:~#
Turn RT-A-01, RT-A-02 and turn ESG-A back on. Verify paths again. The expectation is that everything is back to normal now.
For some reason after turning on the ESG on DC1 the routes kept flowing through DC2. The same “issue” we saw before. So, I turned BGP off/on again on the ESG in DC2. This is not shown below. Below you will see the outputs after the BGP “reset” on DC2.
Routing path verification from the external client VM to T1–WEB–1 , T1–WEB–2, T1–WEB–3 and T1–WEB–4
root ##bl##C:\Users\Administrator>tracert 172.20.1.1 Tracing route to 172.20.1.1 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.21.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 1 ms <1 ms <1 ms 172.20.1.1 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.2 Tracing route to 172.20.1.2 over a maximum of 30 hops 1 1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 3 ms <1 ms <1 ms 172.20.1.2 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.3 Tracing route to 172.20.1.3 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms <1 ms 10.11.11.31 root ##y## 3 1 ms <1 ms <1 ms 10.100.19.2 4 1 ms <1 ms <1 ms 172.39.39.14 5 5 ms 1 ms 2 ms 172.20.1.3 Trace complete.
root ##bl##C:\Users\Administrator>tracert 172.20.1.4 Tracing route to 172.20.1.4 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms cs-01.home.local [10.11.11.253] 2 <1 ms <1 ms 1 ms 10.11.11.31 root ##y## 3 <1 ms <1 ms <1 ms 10.100.19.2 4 <1 ms <1 ms <1 ms 172.39.39.14 5 5 ms 1 ms <1 ms 172.20.1.4 Trace complete.
Routing path verification from the T1–WEB–1, T1–WEB–2, T1–WEB–3 and T1–WEB–4 to the external client VM
root ##bl##root@Web01:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.153 ms 0.118 ms 0.186 ms 2 172.39.39.1 (172.39.39.1) 0.239 ms 0.272 ms 0.292 ms root ##y## 3 10.100.21.1 (10.100.21.1) 0.643 ms 0.611 ms 0.581 ms 4 10.11.11.50 (10.11.11.50) 0.959 ms * * root@Web01:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.137 ms 0.083 ms 0.069 ms 2 172.39.39.1 (172.39.39.1) 0.285 ms 0.386 ms 0.330 ms root ##y## 3 10.100.19.1 (10.100.19.1) 0.639 ms 0.542 ms 0.519 ms 4 10.11.11.50 (10.11.11.50) 0.785 ms * * root@Web02:~#
root ##bl##root@Web03:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.127 ms 0.072 ms 0.069 ms 2 172.39.39.1 (172.39.39.1) 0.579 ms 0.704 ms 0.671 ms root ##y## 3 10.100.19.1 (10.100.19.1) 0.631 ms 0.896 ms 1.064 ms 4 10.11.11.50 (10.11.11.50) 1.063 ms * * root@Web03:~#
root ##bl##root@Web02:~# traceroute 10.11.11.50 traceroute to 10.11.11.50 (10.11.11.50), 30 hops max, 60 byte packets 1 172.20.1.254 (172.20.1.254) 0.142 ms 0.199 ms 0.163 ms 2 172.39.39.1 (172.39.39.1) 0.555 ms 0.450 ms 0.395 ms root ##y## 3 10.100.19.1 (10.100.19.1) 0.884 ms 0.866 ms 0.934 ms 4 10.11.11.50 (10.11.11.50) 1.103 ms * * root@Web02:~#
Additional investigation on routing table problem
I did some investigation “why” DC1 would not become the primary routing path when the DC1 full path is available again.
In the BGP tables in the beginning (show ip bgp) the UDLR only showed ONE path to the 10.11.11.0/24 network. The UDLR selects 172.39.39.2 (the interface of the ESG in DC2) as the next hop.
root ##bl##REGX-UDLR01-0> show ip bgp Status codes: s - suppressed, d - damped, > - best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight AS Path root ##y## > 10.11.11.0/24 172.39.39.2 0 100 30 65522 i REGA-ESG01-0>
The ESG of DC1 showed three paths to the 10.11.11.0/24 network. (The output below is truncated to make it more readable) One is retrieved from RT-A-01 the other from RT-A-02 and the other one is retrieved from the UDLR on interface 172.39.39.14.
root ##bl##REGA-ESG01-0> show ip bgp Status codes: s - suppressed, d - damped, > - best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight AS Path root ##y## 10.11.11.0/24 10.100.19.1 0 100 60 65511 65510 i root ##y## 10.11.11.0/24 10.100.21.1 0 100 60 65511 65510 i root ##y## > 10.11.11.0/24 172.39.39.14 0 100 60 65530 i REGA-ESG01-0>
If we look at the ESG in DC2 we see only two paths to the 10.11.11.0/24 network. One is retrieved from RT-B-01 the other from RT-B-02. (The output below is truncated to make it more readable)
root ##bl##REGB-ESG01-0> show ip bgp Status codes: s - suppressed, d - damped, > - best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight AS Path root ##y## > 10.11.11.0/24 10.200.19.1 0 100 60 65512 65510 i root ##y## 10.11.11.0/24 10.200.21.1 0 100 60 65512 65510 i REGB-ESG01-0>
I did not find a real answer, other than the fact that the UDLR should display two routes towards the 10.11.11.0/24 network and it should show these routes in the BGP table. For some reason this is not happening and I have no explanation why. But in terms of operations this does not matter because the goal is to have two paths to the north (10.11.11.0.24) and if one goes down the other one should take over and this happens as we could see above.