iwan.hoogendoorn
Oracle Cloud · · 28 min read

NAT on DRG Part 5: Overlapping CIDRs between OCI and On-Premises with BGP

Part 5 of 5 · NAT on DRG

Originally published as an Oracle tutorial on Oracle Docs. Part 5 of 5 in the NAT on DRG series.

Introduction

In NAT on DRG Part 3: Overlapping CIDRs between OCI and On-Premises an Oracle Cloud Infrastructure (OCI) VCN and an on-premises network with the same IP range were connected over a Site-to-Site VPN, with NAT on DRG doing the translation on the IPSec tunnel attachments. The tunnels in that part were policy-based: the encryption domain decided which traffic went into the tunnel, and it was also the place where the Dynamic Routing Gateway (DRG) learned its route to on-premises.

This part builds the same use case with BGP dynamic routing. That is how most production VPNs are built, and it is also how a FastConnect virtual circuit exchanges routes, so the routing model in this tutorial is the one to expect there as well. The topology, the addresses and the NAT rule are identical to Part 3:

Side Real subnet CIDR Shadow CIDR seen by the other side
OCI VCN uc3-vcn-oci 10.0.1.0/24 192.168.10.0/24
On-premises LAN 10.0.1.0/24 192.168.20.0/24

What changes is how each side learns where the other side is. With BGP there is one question that decides everything: does the DRG translate the routes that are exchanged over BGP? The answer, tested for this tutorial, is no:

  • Routes that the DRG advertises are not translated. The DRG advertises the contents of the DRG route table of the tunnel attachment as they are. It does not replace the real CIDR of the VCN with its shadow CIDR.
  • Routes that the DRG receives are not translated. If the on-premises router advertises its real CIDR 10.0.1.0/24, the DRG does not turn it into 192.168.20.0/24. Because that prefix overlaps with the attached VCN, the DRG ignores it completely.

NAT on DRG translates packets, not routes. With BGP you therefore have to put the shadow CIDRs into the route exchange yourself, on both sides:

Direction What has to be advertised How
On-premises to Oracle 192.168.20.0/24 (the shadow CIDR of on-premises) The on-premises router advertises it with BGP.
Oracle to on-premises 192.168.10.0/24 (the shadow CIDR of OCI) A static route for it in the DRG route table of the tunnel attachments, which the DRG then advertises.

The on-premises network is simulated with a VCN in another region, with one Oracle Linux instance that runs Libreswan for the IPSec tunnels and FRRouting (FRR) for BGP. The lab is the one from Part 3 with the VPN converted to BGP, which is why the resources in the screenshots are named uc3-....

Other Use Cases

This tutorial is part of a series. Every part solves a different overlap scenario with the same building blocks.

Diagram

Overlapping IPs between VCN subnet CIDR and on-premises or remote network with stateless 1:1 NAT on DRG, with BGP route exchange over the IPSec tunnels

The diagram shows every object you configure in this tutorial:

  • The OCI side in the Canada Southeast (Montreal) region: the VCN uc3-vcn-oci (10.0.0.0/16) with the subnet 10.0.1.0/24 and the test instance uc3-inst-oci (10.0.1.92).
  • The simulated on-premises side in the Canada Southeast (Toronto) region: a VCN with a WAN subnet 10.0.99.0/24 and a LAN subnet 10.0.1.0/24. The router instance uc3-inst-op has a leg in both subnets (10.0.99.146 and 10.0.1.164), and uc3-inst-op-lan (10.0.1.130) is a host on the LAN.
  • One DRG with a VCN attachment and a Site-to-Site VPN with two IPSec tunnels, each with a BGP session between Oracle (AS 31898) and the on-premises router (AS 65000).
  • Two DRG route tables: uc3-drgrt-vcn-oci (orange text) for the VCN attachment, which learns 192.168.20.0/24 from the tunnels, and uc3-drgrt-ipsec (green text) for the IPSec tunnel attachments, with two static routes to the VCN. The second one, for 192.168.10.0/24, is there to be advertised to on-premises.
  • One DRG NAT policy (the ellipse) with a single rule, associated with both IPSec tunnel attachments.

Prerequisites

  • An OCI tenancy that is subscribed to two regions, and a compartment in which you are allowed to manage networking resources (VCNs, DRGs, Site-to-Site VPN, DRG route tables and DRG NAT policies) and compute instances.
  • The networks on both sides already exist. This tutorial does not create them. In the lab they are the VCN uc3-vcn-oci and the VCN uc3-vcn-onprem that simulates the on-premises network, each with its subnets, an internet gateway and route tables that send 0.0.0.0/0 to the internet gateway. Steps 01 and 02 walk through this starting point.
  • A DRG that supports NAT. NAT on DRG is not available on legacy (V1) DRGs, and not on DRGs that have disintermediation, high-throughput mode or route unification enabled. A newly created DRG works.
  • An on-premises router that supports route-based IPSec and BGP, or the simulated one from this tutorial: an Oracle Linux 8 instance with Libreswan and FRR, a public IP address and a second VNIC in the LAN subnet.
  • A private BGP AS number for the on-premises router (65000 in this tutorial) and two small subnets for the inside of the tunnels (10.255.255.0/30 and 10.255.255.4/30 in this tutorial).
  • Compute instances to test with. In this tutorial they are called uc3-inst-oci (10.0.1.92) on the OCI side and uc3-inst-op-lan (10.0.1.130) on the on-premises side.
  • An SSH key pair for the instances, and the public IP address you connect from (to allow it in the security lists).
  • Basic knowledge of Site-to-Site VPN, BGP, VCN route tables and DRG route tables. If NAT on DRG is new to you, start with NAT on DRG Part 1: Overlapping CIDRs between two local VCNs.

Steps

  • STEP 01: Review the VCN on the OCI side
  • STEP 02: Review the simulated on-premises network
  • STEP 03: Configure the security lists
  • STEP 04: Configure the routing on the on-premises side
  • STEP 05: Create the DRG and attach the VCN
  • STEP 06: Create the Site-to-Site VPN with BGP dynamic routing
  • STEP 07: Configure Libreswan on the on-premises side
  • STEP 08: Configure BGP on the on-premises side
  • STEP 09: Configure the DRG route tables
  • STEP 10: Configure the VCN route table on the OCI side
  • STEP 11: Create the DRG NAT policy and rule
  • STEP 12: Associate the NAT policy with the IPSec tunnel attachments
  • STEP 13: Verify the BGP route exchange
  • STEP 14: Test the connectivity

Step 01: Review the VCN on the OCI side

We start on the OCI side with an ordinary VCN, one public subnet and one test instance.

  • Select the Canada Southeast (Montreal) region.
  • Open the navigation menu, click on Networking and then on Virtual cloud networks.
  • Notice the VCN uc3-vcn-oci with the IPv4 CIDR Block 10.0.0.0/16.

  • Click on uc3-vcn-oci.
  • Notice the IPv4 CIDR Block 10.0.0.0/16 on the Details tab.

  • Click on the Subnets tab.
  • Notice the subnet uc3-subnet-oci with the IPv4 CIDR Block 10.0.1.0/24.

The internet gateway is only here so that you can reach the test instance over SSH.

  • Click on the Gateways tab.
  • Notice the internet gateway uc3-igw-oci.

  • Open the navigation menu, click on Compute and then on Instances.
  • Notice the instance uc3-inst-oci with the private IP address 10.0.1.92.

Step 02: Review the simulated on-premises network

The on-premises network is simulated with a VCN in another region. It uses the same address space as the OCI side, which is the whole point of this tutorial. If you have a real on-premises network and router, you can read this step for reference and continue with Step 05.

  • Select the Canada Southeast (Toronto) region.
  • Open the navigation menu, click on Networking and then on Virtual cloud networks.
  • Notice the VCN uc3-vcn-onprem with the same IPv4 CIDR Block 10.0.0.0/16.

  • Click on uc3-vcn-onprem.
  • Notice the IPv4 CIDR Block 10.0.0.0/16 on the Details tab.

The VCN has two subnets. Together they give the Libreswan instance the two sides that a real router has.

  • Click on the Subnets tab.
  • Notice the two subnets:
    • uc3-subnet-wan (10.0.99.0/24), the outside of the on-premises router.
    • uc3-subnet-op (10.0.1.0/24), the on-premises LAN. This is the range that overlaps with the OCI side.

  • Click on the Gateways tab.
  • Notice the internet gateway uc3-igw-op. In this simulation it plays the role of the on-premises internet connection.

  • Open the navigation menu, click on Compute and then on Instances.
  • Notice the two instances:
    • uc3-inst-op, the Libreswan router, with the public IP address 40.233.76.145 and the private IP address 10.0.99.146 in the WAN subnet. It has a second VNIC in the LAN subnet with the IP address 10.0.1.164.
    • uc3-inst-op-lan, a host on the LAN, with the private IP address 10.0.1.130.

Step 03: Configure the security lists

The security lists must allow two kinds of traffic: the IPSec tunnel itself and the ping that we use as a test.

  • IPSec: UDP port 500 (IKE), UDP port 4500 (NAT traversal) and the ESP protocol.
  • ICMP: from the private ranges. Because of the translation, a ping from on-premises arrives in OCI with a source address in 192.168.20.0/24, and a ping from OCI arrives on-premises with a source address in 192.168.10.0/24.

The lab allows ICMP from all three private ranges. That is broader than this tutorial needs. On the OCI side the only range that matters is the on-premises shadow CIDR 192.168.20.0/24. On the simulated on-premises side you need the OCI shadow CIDR 192.168.10.0/24 and also the local 10.0.1.0/24, because traffic from the LAN hosts passes through the LAN VNIC of the Libreswan router, and the security list is checked there as well.

  • In uc3-vcn-oci, click on the Security tab and click on the default security list.
  • Click on the Security rules tab.
  • Make sure the security list has the following ingress rules, and add the ones that are missing:
    • TCP port 22 from your own public IP address, for SSH.
    • ICMP from 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16.
    • UDP port 500, UDP port 4500 and ESP from 0.0.0.0/0.

  • Open the default security list of uc3-vcn-onprem in the same way.
  • Make sure it has the same ingress rules.

Step 04: Configure the routing on the on-premises side

On-premises hosts must know how to reach OCI. They use the shadow CIDR of the OCI side, 192.168.10.0/24, and the route for it points to the on-premises router. The router itself learns that prefix with BGP in Step 08.

  • In uc3-vcn-onprem, click on the Routing tab and click on the route table uc3-rt-wan. This is the route table of the WAN subnet.
  • Notice the single route rule 0.0.0.0/0 with the target uc3-igw-op. The router uses it to reach the Oracle VPN endpoints over the internet.

  • Go back and click on the route table uc3-rt-op. This is the route table of the LAN subnet.
  • Add a route rule for the shadow CIDR of the OCI side, so that the route table has these two route rules:
    • 0.0.0.0/0 with the target uc3-igw-op, for SSH access from the internet.
    • 192.168.10.0/24 (the shadow CIDR of the OCI side) with the target type Private IP and the target 10.0.1.164, the LAN address of the Libreswan router.

Step 05: Create the DRG and attach the VCN

Back on the OCI side we create the DRG. It gets a VCN attachment now, and two IPSec tunnel attachments are added automatically in the next step.

  • Select the Canada Southeast (Montreal) region.
  • Open the navigation menu, click on Networking, then on Customer connectivity and then on Dynamic routing gateway.
  • Click on the Create dynamic routing gateway button and create a DRG with the name uc3-drg-oci.
  • Notice that the DRG is Available.

  • Click on uc3-drg-oci and click on the Attachments tab.
  • Click on the Create virtual cloud network attachment button and create the attachment uc3-att-vcn-oci for the VCN uc3-vcn-oci.
  • Notice that the attachment uses the DRG route table uc3-drgrt-vcn-oci and that it has no DRG NAT policy. The DRG route tables are configured in Step 09.

  • Click on the attachment uc3-att-vcn-oci.
  • Notice the DRG route table uc3-drgrt-vcn-oci and the empty DRG NAT policy field.

Step 06: Create the Site-to-Site VPN with BGP dynamic routing

The Site-to-Site VPN connects the DRG to the on-premises router. It needs a customer-premises equipment (CPE) object that represents the public IP address of the router, and an IPSec connection that creates two tunnels for redundancy. The difference with Part 3 is the routing type of the tunnels. With BGP dynamic routing each tunnel gets:

  • An inside tunnel interface on both ends, a small subnet that only exists inside the tunnel. The BGP session runs between these two addresses.
  • A BGP AS number for the on-premises router. Oracle always uses AS 31898.
  • No encryption domain to configure. A route-based tunnel carries any traffic (0.0.0.0/0 on both sides), and the routing tables decide what goes into it.
Tunnel Oracle VPN IP address Inside IP, Oracle Inside IP, on-premises On-premises AS
1 168.138.66.213 10.255.255.1/30 10.255.255.2/30 65000
2 168.138.65.198 10.255.255.5/30 10.255.255.6/30 65000
  • Open the navigation menu, click on Networking, then on Customer connectivity and then on Customer-premises equipment.
  • Click on the Create CPE button and create a CPE with the name uc3-cpe-op and the IP address 40.233.76.145, the public IP address of the on-premises router.
  • Notice the CPE uc3-cpe-op with the IP address 40.233.76.145.

  • Click on Site-to-Site VPN in the menu on the left.
  • Click on the Create IPSec connection button and create a connection with the following values:
    • Name: uc3-ipsec
    • Customer-premises equipment: uc3-cpe-op
    • Dynamic routing gateway: uc3-drg-oci
    • Routing type for both tunnels: BGP dynamic routing
    • Your BGP ASN for both tunnels: 65000
    • IPv4 inside tunnel interface - CPE and IPv4 inside tunnel interface - Oracle: the addresses from the table above
    • IKE version for both tunnels: IKEv1
  • Notice the IPSec connection uc3-ipsec with the CPE uc3-cpe-op and the DRG uc3-drg-oci.

  • Click on uc3-ipsec and click on the Tunnels tab.
  • Notice the two tunnels, each with its own Oracle VPN IP address and the Routing type BGP dynamic routing. The IPSec status and the IPv4 BGP status become Up after Step 08, which is the state in the screenshot. The IPv6 BGP status stays Down, because this tutorial only uses IPv4.

  • Click on the first tunnel.
  • Notice the BGP information of the tunnel: the Oracle BGP ASN 31898, the Customer BGP ASN 65000, and the inside tunnel interfaces 10.255.255.2/30 (CPE) and 10.255.255.1/30 (Oracle).

  • Go back and click on the second tunnel.
  • Notice the same AS numbers, and the inside tunnel interfaces 10.255.255.6/30 (CPE) and 10.255.255.5/30 (Oracle).

  • Click on each tunnel and write down its Oracle VPN IP address and its Shared secret. You need both for the configuration of the on-premises router.

If you start from the policy-based VPN of Part 3, you do not have to create a new IPSec connection. You can edit each tunnel and change its routing type. With the OCI CLI that looks like this for the first tunnel:

Terminal window
cat > tunnel-1-bgp.json <<'EOF'
{
"routing": "BGP",
"bgpSessionConfig": {
"oracleInterfaceIp": "10.255.255.1/30",
"customerInterfaceIp": "10.255.255.2/30",
"customerBgpAsn": "65000"
}
}
EOF
oci raw-request --http-method PUT \
--target-uri "https://iaas.ca-montreal-1.oraclecloud.com/20160918/ipsecConnections/<ipsec_connection_ocid>/tunnels/<tunnel_ocid>" \
--request-body file://tunnel-1-bgp.json

Step 07: Configure Libreswan on the on-premises side

The on-premises router needs a route-based IPSec configuration: one tunnel interface per tunnel, and traffic selectors that accept everything. If you use a real router, configure it with IKEv1, a pre-shared key, a tunnel interface per tunnel with the inside addresses from Step 06, and 0.0.0.0/0 as local and remote traffic selector.

With Libreswan a tunnel interface is a VTI interface. Both connections have the same selectors (0.0.0.0/0 on both sides), so each one gets its own mark. The mark is what lets the Linux kernel keep the two tunnels apart, and it is also what keeps the 0.0.0.0/0 selectors from capturing all other traffic of the router: only traffic that is routed into a VTI interface gets the mark and is encrypted.

  • Connect to uc3-inst-op with SSH.
  • Create the file /etc/ipsec.d/uc3-t2.conf for the first tunnel (168.138.66.213):
conn uc3-t2
auto=start
authby=secret
keyexchange=ike
ikev2=no
type=tunnel
left=%defaultroute
leftid=40.233.76.145
leftsubnet=0.0.0.0/0
right=168.138.66.213
rightid=168.138.66.213
rightsubnet=0.0.0.0/0
ike=aes256-sha256;modp1536
phase2alg=aes256-sha256
pfs=yes
encapsulation=yes
ikelifetime=28800s
salifetime=3600s
mark=1/0xffffffff
vti-interface=vti1
vti-routing=no
vti-shared=no
leftvti=10.255.255.2/30
  • Create the file /etc/ipsec.d/uc3.conf for the second tunnel (168.138.65.198). It is the same, except for the name, the Oracle VPN IP address, the mark, the VTI interface and the inside address:
conn uc3
auto=start
authby=secret
keyexchange=ike
ikev2=no
type=tunnel
left=%defaultroute
leftid=40.233.76.145
leftsubnet=0.0.0.0/0
right=168.138.65.198
rightid=168.138.65.198
rightsubnet=0.0.0.0/0
ike=aes256-sha256;modp1536
phase2alg=aes256-sha256
pfs=yes
encapsulation=yes
ikelifetime=28800s
salifetime=3600s
mark=2/0xffffffff
vti-interface=vti2
vti-routing=no
vti-shared=no
leftvti=10.255.255.6/30
  • Create a secrets file for each tunnel with the shared secret from Step 06, and restrict its permissions with chmod 600:
/etc/ipsec.d/uc3-t2.secrets
40.233.76.145 168.138.66.213 : PSK "<shared secret of tunnel 1>"
# /etc/ipsec.d/uc3.secrets
40.233.76.145 168.138.65.198 : PSK "<shared secret of tunnel 2>"
  • Start Libreswan:
Terminal window
sudo systemctl enable --now ipsec
sudo systemctl restart ipsec

There are two settings to point out:

  • vti-routing=no stops Libreswan from adding routes itself. With selectors of 0.0.0.0/0 it would otherwise route everything into the tunnel. The routes come from BGP in the next step.

  • leftvti puts the inside address of the tunnel on the VTI interface. Libreswan only does this when it creates the interface. If vti1 and vti2 already exist from an earlier configuration, delete them first with sudo ip link del vti1 and sudo ip link del vti2, and then restart Libreswan.

  • Run sudo ipsec status | grep -E "erouted|Total IPsec".

  • Notice that both connections use the traffic selectors 0.0.0.0/0 on both sides, and notice the line Total IPsec connections: loaded 2, active 2.

  • Run ip -br addr | grep vti.
  • Notice the two VTI interfaces with their inside addresses 10.255.255.2/30 and 10.255.255.6/30.

  • Run ping -c 2 10.255.255.1; ping -c 2 10.255.255.5 to ping the Oracle side of both tunnels.
  • Notice that both addresses answer. The tunnels carry traffic, and the BGP sessions can be built on top of them.

Step 08: Configure BGP on the on-premises side

The tunnels are up, but nothing is routed through them yet. That is the job of BGP. On the simulated on-premises router BGP is provided by FRRouting (FRR).

This is the step where the overlap has to be dealt with on the on-premises side. The router must not advertise its real LAN 10.0.1.0/24: the DRG ignores that prefix, because it overlaps with the attached VCN, and NAT on DRG does not translate it into the shadow CIDR. The router advertises the shadow CIDR 192.168.20.0/24 instead. No interface on the router has an address in that range, so a static route to Null0 puts the prefix in the routing table, which is what BGP needs to be able to advertise it. That route never handles a packet: traffic for on-premises arrives with its destination already translated back to 10.0.1.x.

The two prefix lists make the exchange explicit. The router advertises only its own shadow CIDR, and it accepts only the shadow CIDR of OCI. The second filter matters, because Oracle also advertises the real 10.0.1.0/24 of the VCN (you see why in Step 09), and that prefix is the on-premises LAN itself.

  • Install FRR and enable the BGP daemon:
Terminal window
sudo dnf -y install frr
sudo sed -i 's/^bgpd=no/bgpd=yes/' /etc/frr/daemons
  • Create the file /etc/frr/frr.conf:
frr defaults traditional
hostname uc3-inst-op
log syslog informational
service integrated-vtysh-config
!
! The on-premises shadow CIDR. It only exists so that BGP has something to advertise.
ip route 192.168.20.0/24 Null0
!
ip prefix-list TO-OCI seq 10 permit 192.168.20.0/24
ip prefix-list FROM-OCI seq 10 permit 192.168.10.0/24
!
router bgp 65000
bgp router-id 10.0.99.146
no bgp ebgp-requires-policy
neighbor 10.255.255.1 remote-as 31898
neighbor 10.255.255.1 description OCI-tunnel-1
neighbor 10.255.255.1 timers 5 15
neighbor 10.255.255.5 remote-as 31898
neighbor 10.255.255.5 description OCI-tunnel-2
neighbor 10.255.255.5 timers 5 15
!
address-family ipv4 unicast
network 192.168.20.0/24
neighbor 10.255.255.1 soft-reconfiguration inbound
neighbor 10.255.255.1 prefix-list FROM-OCI in
neighbor 10.255.255.1 prefix-list TO-OCI out
neighbor 10.255.255.5 soft-reconfiguration inbound
neighbor 10.255.255.5 prefix-list FROM-OCI in
neighbor 10.255.255.5 prefix-list TO-OCI out
exit-address-family
!
line vty
  • Start FRR:
Terminal window
sudo chown frr:frr /etc/frr/frr.conf
sudo systemctl enable --now frr
  • Run sudo vtysh -c "show ip bgp summary".
  • Notice that both neighbors are established: the last column shows a number of received prefixes and not a state such as Active or Connect.

Step 09: Configure the DRG route tables

The DRG has two kinds of attachments, and each kind gets its own DRG route table. With BGP these tables do two jobs at once: they forward packets, and they decide what is exchanged over BGP.

The route table of the IPSec tunnel attachments gets two static routes, both to the VCN attachment, and each one is there for a different reason:

Destination Why it is there
10.0.1.0/24 Forwarding. A packet from on-premises is translated by the NAT policy first, and the DRG then looks up the translated destination, which is the real CIDR of the VCN.
192.168.10.0/24 Advertising. The DRG advertises the routes of this table to the on-premises router. This route makes it advertise the shadow CIDR of OCI. No packet ever uses it, because a packet for 192.168.10.x has been translated to 10.0.1.x before the route lookup.

This is the Oracle half of the answer to the overlap: the DRG does not translate the routes it advertises, so the shadow CIDR has to be put into the table by hand.

  • In uc3-drg-oci, click on the Routing tab and create a DRG route table with the name uc3-drgrt-ipsec.
  • Click on the Static route rules tab and click on the Add static route rules button.
  • Add two rules, both with the Next hop attachment uc3-att-vcn-oci:
    • Destination CIDR block 10.0.1.0/24
    • Destination CIDR block 192.168.10.0/24
  • Notice that both static routes are Active.

  • Go back to the Attachments tab of uc3-drg-oci, scroll down to IPSec tunnel attachments and edit both tunnel attachments to select uc3-drgrt-ipsec as their DRG route table.

The route table of the VCN attachment handles traffic that comes from the VCN and is addressed to the on-premises shadow CIDR 192.168.20.0/24. This traffic has to go to a tunnel. A static route rule in a DRG route table cannot point to an IPSec tunnel, so the route is imported from the tunnels with an import route distribution. With BGP the imported route is the one that the on-premises router advertises.

  • On the Routing tab of uc3-drg-oci, create an import route distribution with the name IMPORT-IPSEC and add two statements:
    • Priority 10, Match type Attachment, the attachment of the first IPSec tunnel, Action Accept.
    • Priority 20, Match type Attachment, the attachment of the second IPSec tunnel, Action Accept.
  • Create a DRG route table with the name uc3-drgrt-vcn-oci and select IMPORT-IPSEC as its Import route distribution.
  • Edit the attachment uc3-att-vcn-oci and select uc3-drgrt-vcn-oci as its DRG route table.
  • Click on the import route distribution IMPORT-IPSEC and click on the Statements tab.
  • Notice the two statements, one for each IPSec tunnel attachment, both with the action ACCEPT.

  • Click on the DRG route table uc3-drgrt-vcn-oci and click on the Static route rules tab.
  • Notice that the table has no static route rules. This is expected.

  • Go back to the Details tab and click on the Get all route rules button.
  • Notice the two routes of the type DYNAMIC for 192.168.20.0/24, each with an IPSec tunnel attachment as next hop. This is the prefix that the on-premises router advertises with BGP in Step 08.
  • Notice the Route status. The DRG uses the route with the status Active. The route through the other tunnel has the status Conflict, which here only means that it is the standby for the same destination.

Step 10: Configure the VCN route table on the OCI side

The instances in the VCN must send traffic for the on-premises shadow CIDR to the DRG.

  • In uc3-vcn-oci, click on the Routing tab and click on the route table uc3-rt-oci. This is the route table that is assigned to uc3-subnet-oci.
  • Click on the Route Rules tab.
  • Add a route rule for the on-premises shadow CIDR, so that the route table has these two route rules:
    • 0.0.0.0/0 with the target uc3-igw-oci, for SSH access from the internet.
    • 192.168.20.0/24 (the shadow CIDR of on-premises) with the target uc3-drg-oci.

Step 11: Create the DRG NAT policy and rule

This is the step the whole tutorial is about. One DRG NAT policy with one rule does all the translation, in both directions.

The policy is associated with the IPSec tunnel attachments, so the rule is written for traffic that arrives from on-premises. Such a packet has an on-premises host as source (10.0.1.x) and the shadow address of an OCI instance as destination (192.168.10.x):

Original Translated Meaning
Source 10.0.1.0/24 192.168.20.0/24 The real addresses of on-premises are presented in OCI as 192.168.20.0/24.
Destination 192.168.10.0/24 10.0.1.0/24 The shadow CIDR of OCI is replaced with the real CIDR of the VCN.
  • Open the navigation menu, click on Networking, then on Customer connectivity and then on DRG NAT policy.
  • Click on the Create policy button and create a policy with the name uc3-policy-oci.
  • Notice that the policy is Active.

  • Click on uc3-policy-oci and click on the Rules tab.
  • Click on the Add rule button and add a rule with the following values:
    • Priority: 10
    • Original source CIDR: 10.0.1.0/24
    • Translated source CIDR: 192.168.20.0/24
    • Original destination CIDR: 192.168.10.0/24
    • Translated destination CIDR: 10.0.1.0/24
  • Notice that the rule is listed on the Rules tab.

Step 12: Associate the NAT policy with the IPSec tunnel attachments

The policy is associated with both IPSec tunnel attachments, so that the translation is the same no matter which tunnel carries the traffic.

  • In uc3-policy-oci, click on the Attachments tab.
  • Click on the Associate attachments button, select the two DRG Attachment for IPSec Tunnel attachments and confirm.
  • Notice that both attachments are listed with the Attachment type IPSec tunnel, together with their DRG route table uc3-drgrt-ipsec.

Step 13: Verify the BGP route exchange

Before sending traffic it is worth looking at what is exchanged over BGP, because this is where this tutorial differs from every other part of the series. The check is done on the on-premises router.

  • Connect to uc3-inst-op with SSH.
  • Run sudo vtysh -c "show ip bgp neighbors 10.255.255.1 advertised-routes".
  • Notice that the router advertises a single prefix: its shadow CIDR 192.168.20.0/24.

  • Run sudo vtysh -c "show ip bgp neighbors 10.255.255.1 received-routes".
  • Notice that Oracle advertises two prefixes: the real CIDR of the VCN 10.0.1.0/24, and the shadow CIDR 192.168.10.0/24. These are the two static routes of the DRG route table uc3-drgrt-ipsec, advertised as they are.
  • Notice the last line: Total number of prefixes 2 (1 filtered). The filtered prefix is 10.0.1.0/24, which the prefix list from Step 08 does not accept.

  • Run sudo vtysh -c "show ip bgp".
  • Notice that only 192.168.10.0/24 is accepted from Oracle, through both tunnels, next to the router’s own 192.168.20.0/24.

  • Run ip route show 192.168.10.0/24.
  • Notice that the route to the shadow CIDR of OCI is installed with both tunnel interfaces as next hop.

The OCI Console shows the same exchange from the Oracle side.

  • Go back to the OCI Console, open the IPSec connection uc3-ipsec and click on the first tunnel.
  • Click on the BGP routes received tab.
  • Notice the single prefix 192.168.20.0/24, received from the on-premises router (AS 65000).

  • Click on the BGP routes advertised tab.
  • Notice the two prefixes that Oracle advertises: 10.0.1.0/24 and 192.168.10.0/24.

  • Go back, click on the second tunnel and click on the BGP routes received tab.
  • Notice the same prefix 192.168.20.0/24.

  • Click on the BGP routes advertised tab.
  • Notice the same two prefixes 10.0.1.0/24 and 192.168.10.0/24.

The received routes are the proof of the statement in the introduction. A NAT policy is associated with these tunnel attachments, and the DRG still advertises 10.0.1.0/24 untranslated. The shadow CIDR 192.168.10.0/24 is only there because of the static route that was added in Step 09.

Step 14: Test the connectivity

Everything is in place, so it is time to send a packet. The last octet of an address is preserved by the 1:1 translation, which tells you which address to ping:

Instance Real IP address Address to use from the other side
uc3-inst-oci 10.0.1.92 192.168.10.92
uc3-inst-op-lan 10.0.1.130 192.168.20.130

This is what happens to a ping from uc3-inst-oci to the on-premises host:

Where Source Destination What happens
Leaves uc3-inst-oci 10.0.1.92 192.168.20.130 The route table uc3-rt-oci sends 192.168.20.0/24 to the DRG.
Route lookup in uc3-drgrt-vcn-oci 10.0.1.92 192.168.20.130 The route for 192.168.20.0/24, learned with BGP, points to an IPSec tunnel.
Leaves the DRG on the tunnel attachment 192.168.10.92 10.0.1.130 The DRG applies the reverse of the rule in uc3-policy-oci.
On-premises router 192.168.10.92 10.0.1.130 The packet comes out of the tunnel and is forwarded to the LAN.

And this is the reply:

Where Source Destination What happens
Leaves uc3-inst-op-lan 10.0.1.130 192.168.10.92 The route table uc3-rt-op sends 192.168.10.0/24 to the router 10.0.1.164.
On-premises router 10.0.1.130 192.168.10.92 The route for 192.168.10.0/24, learned with BGP, points to a tunnel interface.
Enters the DRG on the tunnel attachment 192.168.20.130 10.0.1.92 The rule in uc3-policy-oci translates source and destination.
Route lookup in uc3-drgrt-ipsec 192.168.20.130 10.0.1.92 10.0.1.0/24 points to uc3-att-vcn-oci.
  • Connect to uc3-inst-oci with SSH.
  • Run ip a and notice that the instance has the IP address 10.0.1.92.
  • Run ping -c 5 -W 2 192.168.20.130 to ping the shadow address of the on-premises host.
  • Notice that all five packets are answered.

  • Connect to uc3-inst-op-lan with SSH.
  • Run ip a and notice that the host has the IP address 10.0.1.130, in the same subnet range as the OCI instance.
  • Run ping -c 5 -W 2 192.168.10.92 to ping the shadow address of uc3-inst-oci.
  • Notice that all five packets are answered.

A packet capture on the on-premises host shows the translation at work. The host never sees the real address 10.0.1.92 of the OCI instance, only its shadow address:

[opc@uc3-inst-op-lan ~]$ sudo tcpdump -ni any icmp
IP 10.0.1.130 > 192.168.10.92: ICMP echo request, id 18, seq 1, length 64
IP 192.168.10.92 > 10.0.1.130: ICMP echo reply, id 18, seq 1, length 64

Conclusion

An OCI VCN and an on-premises network with exactly the same subnet CIDR communicate over a Site-to-Site VPN with BGP dynamic routing. The NAT rule is the same single rule as in Part 3, on the same IPSec tunnel attachments. What is different is the way the routes get where they need to be.

The main finding of this part is that NAT on DRG and BGP are independent of each other. The DRG translates packets, and it does not translate the routes that it advertises or receives. A real CIDR that overlaps with an attached VCN is of no use in the exchange: Oracle advertises it untranslated, and the DRG ignores it when it is received. Both sides therefore advertise their shadow CIDR. On the on-premises side that is a prefix the router originates itself, and on the Oracle side it is a static route in the DRG route table of the tunnel attachments whose only purpose is to be advertised.

Compared to the policy-based VPN of NAT on DRG Part 3: Overlapping CIDRs between OCI and On-Premises there is no encryption domain to keep in line with the translation, and failover between the tunnels is handled by BGP. The same routing model, an exchange of shadow prefixes over BGP, is what to expect when the NAT policy is associated with a FastConnect virtual circuit attachment, although that has not been tested for this series.

Was this article useful? Be the first to rate it.