View source for Site to Site VPN with Static Routing between two OCI Regions using Libreswan as a CPE on one side and the DRG on the other side


When you have your applications, databases, or services spread into different OCI Regions (or even Tenants) and you need to allow network communication between the two OCI Regions the preferred method is the use of RPC Peering. If RPC peering is not the way you wait to use another option is to use a Site-to-Site IPSEC VPN connection. OCI does not support sending up a Site-to-Site IPSEC VPN connection using the DRG but you can set up a Site-to-Site IPSEC VPN connection using a custom VPN endpoint (like Libreswan) on one side and use the DRG on the other side.

[[File:c73202c389689fb700bc7a79b2743acb.png|800px]]

This tutorial will explain how to set up a Site-to-Site VPN with Static Routing between two OCI Regions using Libreswan as a CPE on one side and the DRG on the other side.

[[File:e87b879f04deda95528a9b8eb7e5edd9.png|800px]]

You can also use this method if you need to set up a site-to-site VPN between ON-PREM and OCI and do not want to use the DRG as the VPN endpoint but your own custom VPN endpoint.

= Prerequisites =

Before you start you need to have two OCI Regions available:

'''OCI Region 1 (destination)''':
* VCN
* (Private) Subnet
* DRG
* VCN Attachments
* Instances

In this region the DRG will be the VPN endpoint and all traffic will be routed using the Internet connection of the DRG.

'''OCI Region 2 (source)''':
* VCN
* (Public) Subnet
* Internet Gateway
* Instances

In this region the Libreswan CPE (Instance inside OCI) will be the VPN endpoint and all traffic will be routed using the Internet connection of the Internet Gateway.

Below is a visual representation of what we have discussed so far.
Note that the Libreswan CPE is NOT on this picture yet.

[[File:0e47532d3e658049248f26b42ff3c48a.png|800px]]

= The Steps =

* [ ] STEP 01: Review the Destination OCI Region (right) VCN, Subnet, DRG, and Instances
* [ ] STEP 02: Review the Source OCI Region (left) VCN, Public Subnet, Internet Gateway, and Instances
* [ ] STEP 03: Collect the public IP address of the CPE Instance in the Source OCI Region (left)
* [ ] STEP 04: Create a new CPE in the Destination OCI Region (right)
* [ ] STEP 05: Configure the Site-to-Site VPN in the Destination OCI Region (right)
* [ ] STEP 06: Collect the public IP address of the IPsec Tunnels in the Destination OCI Region (right) and download the CPE configuration
* [ ] STEP 07: Configure the CPE Instance in the Source OCI Region (left) and install and configure Libreswan
* [ ] STEP 08: Open the firewall on the CPE Instance in the Source OCI Region and configure the VCN/Subnet Security Lists to allow the ingress ports required for the IPSec connection
* [ ] STEP 09: Activate and verify if the IPSec tunnel is up on both sides
* [ ] STEP 10: Configure Static Routing
* [ ] STEP 11: Do one final ping initiated from the source and destination Instances
* [ ] STEP 12: Look at the OCI dashboards to verify the Site-to-site VPN status
* [ ] STEP 13: Enable ECMP Routing

= STEP 01 - Review the Destination OCI Region -right- VCN, Subnet, DRG, VCN Attachments and Instances =

Creating the prerequisites is out of the scope of this tutorial; however, I will show you what we have in place to get started. 

== VCN ==

# Review the '''destination''' OCI Region (in my case I used the Germany Central (Frankfurt) OCI region).
# Review the VCN, where my subnets and Instances will be.

[[File:60cb70f9ee62138c900af758460df4e2.png|800px]]

== Subnets ==

* Review the private Subnet to where my Instances will be attached to.

[[File:436800508da4499d973621dfc6fc3474.png|800px]]

== DRG ==

* Review the DRG that will be used as a VPN endpoint to terminate the VPN.

[[File:7db16c16aa12dfab73cc577414b747ae.png|800px]]

== VCN Attachment ==

* Review the VCN Attachment, to make sure the VCN is attached to the DRG so that the DRG can route VPN traffic to the correct VCN, Subnet, and Instance.

[[File:82c3c3c3b0eb7fb801c5f308bb6c1575.png|800px]]

== Instance ==

* Review the Instance that we will use as a network endpoint to perform our network tests.

[[File:b68e9e40b759699662ed9cabcfe7fa14.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:4b439ca2a5f02dca721a2bbc999ae98d.png|800px]]

= STEP 02 - Review the Source OCI Region -left- VCN, Public Subnet, Internet Gateway, and Instances =

== VCN ==

# Review the '''source''' OCI Region (in my case I used the Netherlands Northwest (Amsterdam) OCI region).
# Review the VCN, where my subnets and Instances will be in.

[[File:415182507001d7e3e88c40a5c1514cde.png|800px]]

== Public Subnet ==

# Review the public Subnet to which my Instances and the Libreswan VPN endpoint will be attached.
# Review the default route table for the VCN.

[[File:b9b2e9b804c631257d588318f2604ea5.png|800px]]

* We need to have a public subnet here, as we need to make sure that the Libreswan VPN endpoint can communicate to the internet to set up the VPN connection with the other side.

== Internet Gateway ==

# Review the Internet Gateway (to allow internet connectivity)
# Review the default route table for the VCN.

[[File:b63c51563748d606362eba4b32b54b96.png|800px]]

== Route Table ==

* Review the VCN route table, and make sure all traffic is routed to the internet gateway.

[[File:e561c91903dd23c44e16f9c551cace9d.png|800px]]

== Instances ==

* Review the Instance that we will use as a network endpoint to perform our network tests.
* For both CLIENT and CPE I have used Oracle Linux 8 as the main OS.

[[File:f4d7dbc21f71e7bfe163fa76cb1828d2.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:0365a8c6e4096ea70adb4944aa68ee21.png|800px]]

= STEP 03 - Collect the public IP address of the CPE Instance in the Source OCI Region -left- =

# In the source OCI Region we deploy an Instance that will be responsible for the VPN termination (VPN endpoint)
# Collect the public IP address that is configured on this VPN endpoint. In my case, this public IP address starts with 143.


[[File:15f3b1e063ab7f50519eb8f8840005de.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:ddc37a81ac0aabc5901441c8d48cdfdf.png|800px]]

= STEP 04 - Create a new CPE in the Destination OCI Region -right- =

* Browse to the DRG on the OCI Console by going through '''Networking > Customer Connectivity > Dynamic routing gateway'''.
* Click on the DRG.

[[File:4d433571d6fe81b06047774bba89f75e.png|800px]]

# Click on '''Customer premises equipment'''.
# Click on the '''Create CPE''' button.

[[File:c74930aa53cab65ed1959f46abed6b38.png|800px]]

# Specify a '''name''' for the CPE.
# Specify the '''public IP address''' that you have collected in the step before (of the CPE).
# Select the CPE Vendor to be '''Libreswan'''.
# Select the CPE Platform version.
# Click on the '''Create CPE''' button.

[[File:897832df2cb724b75a7d650ef15b5ea1.png|800px]]

# Notice that the CPE is now created.
# Notice the Public IP address of the CPE.

[[File:3c1a54d2ce5810c723b2880bc668ccb0.png|800px]]

= STEP 05 - Configure the Site-to-Site VPN in the Destination OCI Region -right- =

# Click on '''Site-to-Site VPN'''.
# Click on the '''Create IPSec connection''' button.

[[File:a74eccd8024dea1f0f472fd1fdc5626b.png|800px]]

# Specify a '''name''' for the IPSec connection.
# Select the '''CPE''' you just created.
# Select the '''DRG'''.
# Specify the '''REMOTE network''' (left side) that you want to route through the IPSec connection. In my case, this is the 10.222.10.0/24 network.
# Scroll down.

[[File:e36ee178884b560c545e406435ec9072.png|800px]]

# Specify the '''name''' for the first tunnel.
# Specify the IKE version to be '''IKEv1'''.
# Select '''Static Routing'''.
# Scroll down.

[[File:57fb59e5c7bbcf471ea869f14c2e8ba2.png|800px]]

# Specify the '''name''' for the second tunnel.
# Specify the IKE version to be '''IKEv1'''.
# Select '''Static Routing'''.
# Click on the '''Create IPSec connection''' button.

[[File:8df26e658b71a5c0ae3374b376d038bf.png|800px]]

= STEP 06 - Collect the public IP address of the IPsec Tunnels in the Destination OCI Region -right- and download the CPE configuration =

* Now that the IPSec connection is created the public IP addresses for both IPSec tunnels will be available and we will need to configure the other side of the VPN.
* Click on the Site-to-Site VPN we just created.

[[File:95c361d527cf9433f6fa79e2e88c2205.png|800px]]

# Notice '''tunnel 1''' of the IPSec connection.
# Note down the public IP address for tunnel # In my case, this IP address starts with '''193'''.
# Notice '''tunnel 2''' of the IPSec connection.
# Note down the public IP address for tunnel # In my case, this IP address starts with '''130'''.

[[File:4314c5cd54dcb01a7690113cce4082a0.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:f64acb5a37f62f9c196bf92bac96b2d0.png|800px]]

= STEP 07 -  Configure the CPE Instance in the Source OCI Region -left- and install and configure Libreswan =

# Set up an SSH session for the CPE located in the source OCI Region / VCN.
# Make sure you are logged in.

[[File:3c13cd0f756fc7071903e943134889b6.png|800px]]

* Issue the following command to upgrade the software.

{{console|body=
[opc@cpe ~]$ sudo dnf upgrade -y
}}

[[File:a845d02132919bc1045a8d013991df29.png|800px]]

* Make sure the upgrade is successfully upgraded.

[[File:499fd07d3d2c3552da1805261da57090.png|800px]]

* Issue the following command so you can execute the command where higher privileges are required.

{{console|body=
[opc@cpe ~] sudo su
}}

[[File:a529bf59f1cbb06e80b451ce403e9357.png|800px]]

# Issue the following command to install the Libreswan software.
# Notice that the software is successfully installed.

{{console|body=
[root@cpe opc]# sudo yum install libreswan -y
}}

[[File:d7d634835e7ef7c375545d0dd45f39e1.png|800px]]

* Edit the following file to enable IP Forwarding.

{{console|body=
[root@cpe etc]# nano /etc/sysctl.conf
}}

[[File:23b5d820b589ab02a600667e5277d113.png|800px]]

* Make sure the `/etc/sysctl.conf` file contains the following content.

{{console|body=
kernel.unknown_nmi_panic = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens3.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.ens3.accept_redirects = 0
}}

[[File:4fa5c1004c465eae9bb89d3320f8fbf2.png|800px]]

* Make sure you save the file with `CNTRL + X`  and type `“Y”`.

# Issue the following command to verify the file's content.
# Notice the output will be the same as the content of the file.

{{console|body=
[root@cpe opc]# more /etc/sysctl.conf
}}

[[File:18f43de296a484673dd2cd6fe7385eb5.png|800px]]

* Issue the following command to reload/apply the file (to enable forwarding).

{{console|body=
[root@cpe etc]# sudo sysctl -p
}}

[[File:750f65262d073dd71983ccbbb5aca012.png|800px]]

# Issue the following command to verify if IP forwarding is enabled
# Notice that the returned value id `"1"` and this means that IP forwarding is enabled. A `"0"` means that IP forwarding is disabled.

{{console|body=
cat /proc/sys/net/ipv4/ip_forward
}}

[[File:8af3bf00d220a130a380f364fd1dc21d.png|800px]]

#  Issue the following command to change the directory where you need to configure the IPSEC tunnels.
# Create/Edit the following file to configure the first IPSEC tunnel (tunnel1).

{{console|body=
[root@cpe opc]# cd /etc/ipsec.d/
}}

{{console|body=
[root@cpe ipsec.d]# nano /etc/ipsec.d/tunnel1.conf
}}

[[File:b9a93b875d744703c589689833f1a8d8.png|800px]]

* Make sure the `/etc/ipsec.d/tunnel1.conf` file contains the following content.

{{console|body=
conn tunnel1
    keyexchange=ike
    pfs=yes
    ikev2=no
    ike=aes256-sha2_256;modp1536
    phase2alg=aes256-sha1;modp1536
    left=10.222.10.70
    leftid=143.xxx.xxx.xxx
    right=193.xxx.xxx.xxx
    rightid= 193.xxx.xxx.xxx
    authby=secret
    leftsubnet=0.0.0.0/0
    rightsubnet=0.0.0.0/0
    auto=start
    mark=5/0xffffffff
    vti-interface=vti1
    vti-routing=no
    encapsulation=auto
    ikelifetime=28800s
}}

[[File:ee5906d1cfd00b395057537a40280a52.png|800px]]

* Make sure you save the file with `CNTRL + X`  and type `“Y”`.

* Create/Edit the following file to configure the second IPSEC tunnel (tunnel2).

{{console|body=
[root@cpe ipsec.d]# nano /etc/ipsec.d/tunnel2.conf
}}

[[File:a780c8449e480b26f1ea05290beefdaf.png|800px]]

* Make sure the `/etc/ipsec.d/tunnel2.conf` file contains the following content.

{{console|body=
conn tunnel2
    keyexchange=ike
    pfs=yes
    ikev2=no
    ike=aes256-sha2_256;modp1536
    phase2alg=aes256-sha1;modp1536
    left=10.222.10.70
    leftid=143.xxx.xxx.xxx
    right=130.xxx.xxx.xxx
    rightid=130.xxx.xxx.xxx
    authby=secret
    leftsubnet=0.0.0.0/0
    rightsubnet=0.0.0.0/0
    auto=start
    mark=6/0xffffffff
    vti-interface=vti2
    vti-routing=no
    encapsulation=auto
    ikelifetime=28800s
}}

[[File:6ad95b8dabfbd47a4a38eb5ad9c16761.png|800px]]

* Make sure you save the file with `CNTRL + X`  and type `“Y”`.

# Issue the following command to verify the file's content for tunnel1.
# Notice the output will be the same as the content of the file.
# Issue the following command to verify the file's content for tunnel2.
# Notice the output will be the same as the content of the file.

{{console|body=
[root@cpe ipsec.d]# more /etc/ipsec.d/tunnel1.conf
}}

{{console|body=
[root@cpe ipsec.d]# more /etc/ipsec.d/tunnel2.conf
}}

[[File:01dd89eeefe96431ce7186bd6d6dbfbe.png|800px]]

Now that we have configured the IPSEC tunnels we also need to configure the shared secrets as we are using a secret (key) for authentication.

When we created the Site-to-Site VPN in Step 5, the tunnels were created and in this process, OCI also generated the shared secret keys per tunnel. To get them and configure them in Libreswan to match the same shared secret, we need to go back to the OCI console.

# Browse to the Site-to-Site settings on the OCI Console by going through '''Networking > Customer Connectivity > Site-to-Site VPN > The configured VPN.
# Click on the first tunnel configuration.

[[File:8978fe54bc11b529c062ecdc1d273d7e.png|800px]]

* For the '''shared secret''' click on '''show'''.

[[File:9bf0832905546c9a9de59b9dcd68fb98.png|800px]]

# Copy the shared secret for tunnel 1 and save it on a notepad so you can access it quickly later.
# Click on the '''Close''' button.

[[File:6f60815756fbe572da2cd7e97593f6ec.png|800px]]

# Browse to the Site-to-Site settings on the OCI Console by going through '''Networking > Customer Connectivity > Site-to-Site VPN > The configured VPN.
# Click on the second tunnel configuration.

[[File:53f249e8ee8ca8bb12d04b70ff6bb0ba.png|800px]]

* For the '''shared secret''' click on '''show'''.

[[File:0f4135e985c0a8c7aba3c054ad439dfc.png|800px]]

# Copy the shared secret for tunnel 2 and save it on a notepad so you can access it quickly later.
# Click on the '''Close''' button.

[[File:f40ebc808cebcda0de49a1964de660cf.png|800px]]

* Now that you have collected the shared secrets you need to configure them on Libreswan.

* Create/Edit the following file to configure the shared secrets for both tunnels.

{{console|body=
[root@cpe ipsec.d]# nano /etc/ipsec.d/shared.secrets
}}

[[File:40c59eea84db671958114903fab4b652.png|800px]]

* Make sure the `/etc/ipsec.d/shared.secrets` file contains the following content.

{{console|body=
143.xxx.xxx.xxx 193.xxx.xxx.xxx : PSK "1blwzMdgQ5XXXoiQwF96tqc7c7"
143.xxx.xxx.xxx 130.xxx.xxx.xxx : PSK "npLt23Ym6E1XXXhr5egvYSuzKC"
}}

[[File:e14f410588ca513bc38316b3ce6f3c1b.png|800px]]

# Issue the following command to verify the file's content.
# Notice the output will be the same as the content of the file.

{{console|body=
[root@cpe ipsec.d]# more /etc/ipsec.d/shared.secrets
}}

[[File:e1b36b35c0f2e88cebf12d4430b8856a.png|800px]]

* Issue the following command to start the IPSEC Service on the Libreswan.
* This will NOT establish the tunnels yet.

{{console|body=
[root@cpe ipsec.d]# ipsec start
}}

[[File:713c9c79e71061547e166581c7885bb9.png|800px]]

# Issue this command to verify the status of the IPsec connections.
# Notice that the verification looks ok, without any strange errors.

{{console|body=
[root@cpe ipsec.d]# ipsec verify
}}

[[File:e0783e37e735dff43ce266c872db1e6a.png|800px]]

= STEP 08 -  Open the firewall on the CPE Instance in the Source OCI Region -left- and configure the VCN and Subnet Security Lists to allow the ingress ports required for the IPSec connection =

To allow the tunnels to be established correctly, you need to make sure the network security on both sides is allowing the ports that are required.

First, we start on the source OCI Region that hosts the Libreswan CPE.

# Browse to the Security List settings on the OCI Console by going through '''Networking > Virtual cloud networks > Select the VCN > Security Lists'''.
# Make sure you select the '''Default Security List'''.
# Scroll down.


[[File:357c20b6e0ab9dbcdea96b8f196352b2.png|800px]]

* Make sure you have added the following Ingress Security Rules:

{| class="wikitable"
! Source !! IP Protocol !! Source Port !! Destination Port
|-
| 0.0.0.0/0 || UDP || All || 500
|-
| 0.0.0.0/0 || UDP || All || 4500
|}

[[File:56090a821664b74d461ef27a3abe97b8.png|800px]]

* On the Libreswan CPE itself, you also need to open the firewall ports.

# Issue this command to review the existing configured firewall rules.
# Notice that there are no rules configured related to the IPSEC ports.
# Issue this command to allow UDP port 500 on the CPE.
# Issue this command to allow UDP port 4500 on the CPE.
# Issue this command to make the firewall rules permanent (so they will remain after a reboot)
# Issue this command to review the existing configured firewall rules
# Notice that there are no rules configured related to the IPSEC ports.

{{console|body=
[root@cpe ipsec.d]# sudo firewall-cmd --list-all
}}

{{console|body=
[root@cpe ipsec.d]# sudo firewall-cmd --add-port=500/udp
}}

{{console|body=
[root@cpe ipsec.d]# sudo firewall-cmd --add-port=4500/udp
}}

{{console|body=
[root@cpe ipsec.d]# sudo firewall-cmd --runtime-to-permanent
}}

{{console|body=
[root@cpe ipsec.d]# sudo firewall-cmd --list-all
}}

[[File:6a72e508a9b4052b08542454503ccfcd.png|800px]]

# Issue this command to make sure the firewall service of the Libreswan is running.
# Notice that the firewall service is active and running. 

{{console|body=
[root@cpe ipsec.d]# systemctl status firewalld
}}

[[File:10c2437c9187b614d482281ed5ce399c.png|800px]]

* Issue this command to restart the IPSEC service.

{{console|body=
[root@cpe ipsec.d]# service ipsec restart
}}

[[File:5b410fd1b743a9dc57369dc1a1a305ce.png|800px]]

# STEP 09: Activate and verify if the IPSec tunnel is up on both sides

* Issue the following command to verify the IPSEC status for both tunnels.

{{console|body=
[root@cpe ipsec.d]# ipsec status
}}

[[File:2cf157d03025a1c507bd04400416e408.png|800px]]

# Notice the configuration and status for the first IPSEC tunnel.
# Notice the configuration and status for the second IPSEC tunnel.
# Notice that the number of IPSEC loaded is '''2''', and the active number is '''0'''.

[[File:c689a962cab7f374b457097496cbe3de.png|800px]]

* Issue the following command to add tunnel1.

{{console|body=
[root@cpe ipsec.d]# ipsec auto --add tunnel1
}}

* Notice the output given by the terminal when tunnel1 is added.

{{console|body=
002 "tunnel1": terminating SAs using this connection
002 "tunnel1" #3: deleting state (STATE_QUICK_I2) aged 3.504567s and sending notification
005 "tunnel1" #3: ESP traffic information: in=0B out=0B
002 "tunnel1" #1: deleting state (STATE_MAIN_I4) aged 3.541172s and sending notification
002 "tunnel1": added IKEv1 connection
}}

* Issue the following command to bring tunnel1 up.

{{console|body=
[root@cpe ipsec.d]# ipsec auto --up tunnel1
}}

* Notice the output given by the terminal when tunnel1 is brought up.

{{console|body=
002 "tunnel1" #5: initiating IKEv1 Main Mode connection
102 "tunnel1" #5: sent Main Mode request
104 "tunnel1" #5: sent Main Mode I2
106 "tunnel1" #5: sent Main Mode I3
002 "tunnel1" #5: Peer ID is ID_IPV4_ADDR: '193.122.0.91'
004 "tunnel1" #5: IKE SA established {auth=PRESHARED_KEY cipher=AES_CBC_256 integ=HMAC_SHA2_256 group=MODP1536}
002 "tunnel1" #6: initiating Quick Mode IKEv1+PSK+ENCRYPT+TUNNEL+PFS+UP+IKE_FRAG_ALLOW+ESN_NO+ESN_YES {using isakmp#5 msgid:b6364de1 proposal=AES_CBC_256-HMAC_SHA1_96-MODP1536 pfsgroup=MODP1536}
115 "tunnel1" #6: sent Quick Mode request
002 "tunnel1" #6: up-client output: vti interface "vti1" already exists with conflicting setting (perhaps need vti-sharing=yes ?
002 "tunnel1" #6: prepare-client output: vti interface "vti1" already exists with conflicting setting (perhaps need vti-sharing=yes ?
004 "tunnel1" #6: IPsec SA established tunnel mode {ESPinUDP=>0x5036cdcc <0x33c964f9 xfrm=AES_CBC_256-HMAC_SHA1_96 NATD=193.122.0.91:4500 DPD=passive}
}}

* Issue the following command to add tunnel2.

{{console|body=
[root@cpe ipsec.d]# ipsec auto --add tunnel2
}}

* Notice the output given by the terminal when tunnel2 is added.

{{console|body=
002 "tunnel2": terminating SAs using this connection
002 "tunnel2" #4: deleting state (STATE_QUICK_I2) aged 25.694856s and sending notification
005 "tunnel2" #4: ESP traffic information: in=0B out=0B
002 "tunnel2" #2: deleting state (STATE_MAIN_I4) aged 25.731704s and sending notification
002 "tunnel2": added IKEv1 connection
}}

* Issue the following command to bring tunnel2 up.

{{console|body=
[root@cpe ipsec.d]# ipsec auto --up tunnel2
}}

* Notice the output given by the terminal when tunnel2 is brought up.

{{console|body=
002 "tunnel2" #7: initiating IKEv1 Main Mode connection
102 "tunnel2" #7: sent Main Mode request
104 "tunnel2" #7: sent Main Mode I2
106 "tunnel2" #7: sent Main Mode I3
002 "tunnel2" #7: Peer ID is ID_IPV4_ADDR: '130.61.66.255'
004 "tunnel2" #7: IKE SA established {auth=PRESHARED_KEY cipher=AES_CBC_256 integ=HMAC_SHA2_256 group=MODP1536}
002 "tunnel2" #8: initiating Quick Mode IKEv1+PSK+ENCRYPT+TUNNEL+PFS+UP+IKE_FRAG_ALLOW+ESN_NO+ESN_YES {using isakmp#7 msgid:aeb4eb18 proposal=AES_CBC_256-HMAC_SHA1_96-MODP1536 pfsgroup=MODP1536}
115 "tunnel2" #8: sent Quick Mode request
002 "tunnel2" #8: up-client output: vti interface "vti2" already exists with conflicting setting (perhaps need vti-sharing=yes ?
002 "tunnel2" #8: prepare-client output: vti interface "vti2" already exists with conflicting setting (perhaps need vti-sharing=yes ?
004 "tunnel2" #8: IPsec SA established tunnel mode {ESPinUDP=>0x8bef7076 <0xe27d84a0 xfrm=AES_CBC_256-HMAC_SHA1_96 NATD=130.61.66.255:4500 DPD=passive}
[root@cpe ipsec.d]#
}}

* Issue the following command again to verify the IPSEC status for both tunnels.

{{console|body=
[root@cpe ipsec.d]# ipsec status
}}

* Notice that the active tunnels have gone from 0 to '''2'''.

[[File:37694130d4a568617b48e836aed39777.png|800px]]

* Let's move back to the destination side on the OCI console.

# Browse to the Site-to-Site settings on the OCI Console by going through '''Networking > Customer Connectivity > Site-to-Site VPN.
# Click on the VPN connection.

[[File:fa15819d2e8cfdf862567f6f26523682.png|800px]]

* Notice that Tunnel 1 and Tunnel 2 are both Available and Up. 

[[File:1902918b85c68c9f7808346de69d727b.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:11e394c6353d9926c66c1966522be76a.png|800px]]

= STEP 10 - Configure Static Routing =

* Now that the tunnels are up we need to make sure the required traffic is routed through the tunnel.

# Issue the following command to review the configured (tunnel) interfaces.
# Notice '''vti1''' that is responsible for tunnel1.
# Notice '''vti2''' that is responsible for tunnel2.

{{console|body=
[root@cpe ipsec.d]# ifconfig
}}

[[File:077460e8051370feec6d1d0561518455.png|800px]]

* Issue the following command to route the `172.16.0.0/16` network through the `vti1` and `vti2` interfaces.

{{console|body=
[root@cpe ipsec.d]# ip route add 172.16.0.0/16 nexthop dev vti1 weight 1 nexthop dev vti2 weight 1
}}

[[File:44a312f35bba3f5a30273aa0076a0ad1.png|800px]]

# Issue the following command to review if the route is added.
# Notice that the routes are added.

{{console|body=
[root@cpe ipsec.d]# ip route
}}

[[File:98241e66871832be05fb590451111bb7.png|800px]]

= STEP 11 - Do one final ping initiated from the source and destination Instances =

# Issue the following command from the SOURCE (CPE) to verify if the ping is working from the Source OCI Region to the Destination OCI Region.
# Notice that we have 0% packet loss.

{{console|body=
[root@cpe ipsec.d]# ping 172.16.1.93 -c 4
}}

[[File:134cc8e7272c896c92f4275cb5c18f46.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:6ffce5f4a75798b069ef7c53921a81ae.png|800px]]

# Issue the following command from the DESTINATION to verify if the ping is working from the Destination OCI Region to the Source OCI Region (CPE).
# Notice that we have 0% packet loss.

{{console|body=
[opc@ih-instance-vcn-a ~]$ ping 10.222.10.70 -c 4
}}

[[File:163eba1eee88fed24ebb29f24aa475ad.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:56b06d935b3394b5045879694682a18f.png|800px]]

If you want to route all the traffic from the other instances (in the same subnet) on the source OCI Region through the tunnels, you need to add a static route on the VCN Route table. This route routes all traffic destined for`172.16.0.0/24` to the CPE (`10.222.10.70`). 

Below is a visual representation of what we have discussed so far (with the full route table).
[[File:1a877ebe97f3e50109c7cfebcac29e4e.png|800px]]

Before you can add a route towards a Private IP address (Libreswan CPE) you first need to enable the Skip Source/Destination check on the Instance (Libreswan CPE) VNIC.

# Browse to '''Compute > Instances'''.
# Select the '''CPE'''.
# Scroll down.

[[File:0bb1ec14dfaa65a53644a0b8c37f1867.png|800px]]

# Click on '''Attached VNICs'''.
# Click on the '''three dots'''.
# Click on '''Edit VNIC'''.

[[File:50932d0a6dffb1e888ac82c763750785.png|800px]]

# Check the box '''Skip source/destination check'''.
# Click on the '''Save changes''' button.

[[File:e3b53e8bc6629ce44aa916589c4e1011.png|800px]]

Now let's add the route.

# Browse to '''Networking > Virtual cloud networks > Select the VCN > Route Tables'''.
# Select the '''Default Route Table'''.
# Add a '''Route Rule''' for the destination `172.16.0.0/16` with the target type to be a '''Private IP''' and the Target to be `10.222.10.70` (The Libreswan CPE)

[[File:42fb199c41d3c7fd183ef393fbc03606.png|800px]]

# Issue the following command to verify the IP address of the CLIENT.
# Notice that the CLIENT's IP address is `10.222.10.19`.
# Issue the following command from the SOURCE (CLIENT) to verify if the ping is working from the Source OCI Region to the Destination OCI Region.
# Notice that we have 0% packet loss.

{{console|body=
[opc@client ~]$ ip a
}}

{{console|body=
[opc@client ~]$ ping 172.16.1.93 -c 4
}}

[[File:b445395d5bf1febe5a7077be0580dbd8.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:01e547522f1c8968657d5449ad439a58.png|800px]]

# Issue the following command from the DESTINATION to verify if the ping is working from the Destination OCI Region to the Source OCI Region (CLIENT).
# Notice that we have 0% packet loss.

{{console|body=
[opc@ih-instance-vcn-a ~]$ ping 10.222.10.19 -c 4
}}

[[File:040020fdc574258e26cea8de01906fd8.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:32bf2e44dc98f45d5c5e2f0bd6beafd9.png|800px]]

= STEP 12 - Look at the OCI dashboards to verify the Site-to-site VPN status =

Now that the VPN Tunnels are up and traffic is flowing through then we can look at the dashboards in OCI on a per-tunnel basis.

# Browse to the Site-to-Site settings on the OCI Console by going through '''Networking > Customer Connectivity > Site-to-Site VPN.
# Click on the VPN connection.

[[File:a44d613abe5057e99c0721c7af0ab455.png|800px]]

* Click on the first tunnel configuration.

[[File:877c755cc61f0246a59bc42e8a346046.png|800px]]

* Scroll down.

[[File:cf2b1f93a63d3e566ebb5f3470382a53.png|800px]]

# Notice the IPSEC Tunnel State graph is constantly on '''1''', indicating the tunnel is up.
# Scroll down.

[[File:f5c64da26a0df6306da4ef3be4a5f3fb.png|800px]]

# Notice the Packets Received on this tunnel is '''4''', which corresponds with the number of ping packets I sent from the source.
# Notice the Packets Sent on this tunnel is '''4''', which corresponds with the number of ping packets I sent from the destination.
# Notice the number of Bytes Received from the source.
# Notice the number of Bytes Sent from the destination.

[[File:0d321a6e4802c7086f6d75f8dc9fd194.png|800px]]

= STEP 13 - Enable ECMP Routing =

* By default, the traffic is always sent using one tunnel interface (vti). If we want to use both tunnels for traffic forwarding we need to enable the Equal Cost Multi-Path (ECMP) routing on both sides.

On the Libreswan CPE issue this command:

{{console|body=
[root@cpe ipsec.d]# sysctl -w net.ipv4.fib_multipath_hash_policy=1
}}

Below is a visual representation of what we have discussed so far.

[[File:677e0b0b875f1af5777c917045cfd93b.png|800px]]

On the destination OCI Region:

* Browse to the DRG on the OCI Console by going through '''Networking > Customer Connectivity > Dynamic routing gateway > Select the DRG > DRG route table > Select the DRG route table that is responsible for the routing''' 
* Click on the '''Get all route rules''' button.

[[File:9eb521a0267e8f68eb34f65571cf1e07.png|800px]]

# Notice that there are two routes available for the 1`0.222.10.0/24` network.
# Notice that one route rule for the `10.222.10.0/24` is marked as '''Conflict''' and the other as '''Active'''.
# Click on the '''Close''' button.

[[File:be4ace953af34eed6055025c60f967c2.png|800px]]

* Click on the '''Edit''' button.

[[File:d1ec4077213839c7ced258a0a607fa53.png|800px]]

# Check the box to '''Enable ECMP'''.
# Click on the '''Save Changes''' button.

[[File:6170c7129e0e3ec0cfa583fec2468508.png|800px]]

* Click on the '''Get all route rules''' button.

[[File:9eb521a0267e8f68eb34f65571cf1e07.png|800px]]

# Notice that both route rules for the `10.222.10.0/24` are now marked as '''Active'''.

[[File:f64c8666cfd429182e6994f285090fac.png|800px]]

Below is a visual representation of what we have discussed so far.

[[File:d91d13209a698b989bac516cc6981d35.png|800px]]

= Conclusion =
In this tutorial, we have connected two different OCI regions using an IPSEC VPN with two tunnels and with ECMP enabled. We used a CPE with Libreswan software in one OCI Region and the Site-to-Site VPN configured on the DRG in the other Region.

[[Category:Oracle Cloud]]