Use iPerf to test the throughput inside an OCI Hub and Spoke VCN Routing architecture

From Iwan
Jump to: navigation, search

In today's rapidly evolving cloud environments, ensuring optimal network performance is crucial for seamless operations. Oracle Cloud Infrastructure (OCI) provides robust networking capabilities, including the Hub and Spoke Virtual Cloud Network (VCN) architecture, to facilitate efficient communication and resource management. One essential aspect of maintaining this architecture is regularly testing the network throughput to identify potential bottlenecks and optimize performance.

In this tutorial, we will use iPerf, a powerful network testing tool, to measure and analyze the throughput within an OCI Hub and Spoke VCN Routing architecture. By the end of this guide, you'll be equipped with the knowledge to effectively assess and enhance your OCI network's performance, ensuring your applications and services run smoothly.

Iperf-testing-in-oci-intro.png

Disclaimer

The test results obtained using iPerf depend highly on various factors, including network conditions, hardware configurations, and software settings specific to your environment. As such, these results may differ significantly from those in other environments. Please do not use these results to make any definitive conclusions about the expected performance of your network or equipment. They should be considered as indicative rather than absolute measures of performance.

The Steps

  • [ ] STEP 01: Review the OCI Hub and Spoke VCN Routing architecture
  • [ ] STEP 02: Install iPerf on the Hub Instances
  • [ ] STEP 03: Install iPerf on the Spoke Instances
  • [ ] STEP 04: Install iPerf on the ONPREM Instances
  • [ ] STEP 05: Install iPerf2
  • [ ] STEP 06: Define the iPerf Tests and prepare the iPerf commands
  • [ ] STEP 07: Perform iPerf tests within the same VCN in the same subnet
  • [ ] STEP 08: Perform iPerf tests within the same VCN across different subnets
  • [ ] STEP 09: Perform iPerf tests between two different VCNs
  • [ ] STEP 10: Perform iPerf tests between different VCNs (bypassing the pfSense Firewall)
  • [ ] STEP 11: Perform iPerf tests between ONPREM and OCI Hub VCN
  • [ ] STEP 12: Perform iPerf tests between ONPREM and OCI Spoke VCN
  • [ ] STEP 13: Perform iPerf tests between ONPREM and OCI Spoke VCN (bypassing the pfSense Firewall)
  • [ ] STEP 14: Perform iPerf tests between the INTERNET and the OCI Hub VCN
  • [ ] STEP 15: Perform iPerf tests within the same subnet ONPREM

iPerf versions

iPerf, iPerf2, and iPerf3 are tools used to measure network bandwidth, performance, and throughput between two endpoints. However, they have some key differences in terms of features, performance, and development status. Here's a breakdown:

iPerf (original)

  • Release: Initially released around 2003.
  • Development: The original iPerf has largely been replaced by its successors (iPerf2 and iPerf3).
  • Features: Basic functionality for testing network bandwidth using TCP and UDP.
  • Limitations: Over time, it became outdated due to a lack of support for modern networking features.

iPerf2

  • Release: Forked from the original iPerf and maintained independently.
  • Development: Actively maintained, especially by ESnet (Energy Sciences Network).
  • Features:
    • Supports both TCP and UDP tests.
    • Multithreading: iPerf2 supports multithreaded testing, which can be useful when testing high-throughput environments.
    • UDP multicast and bidirectional tests.
    • Protocol Flexibility: Better handling of IPv6, multicast, and other advanced networking protocols.
  • Performance: Performs better than the original iPerf for higher throughputs due to multithreading support.
  • Use Case: Best for situations where legacy features, such as IPv6 and multicast, are necessary, or if you require multithreading in testing.

iPerf3

  • Release: Rewritten and released by the same team (ESnet) that maintains iPerf# The rewrite focused on cleaning up the codebase and modernizing the tool.
  • Development: Actively maintained with frequent updates.
  • Features:
    • Supports both TCP and UDP tests.
    • Single-threaded: iPerf3 does not support multithreading, which can be a limitation for high throughput in certain environments.
    • Supports reverse mode for testing in both directions, bidirectional tests, and multiple streams for TCP tests.
    • JSON output for easier integration with other tools.
    • Improved error reporting and network statistics.
    • Optimized for modern network interfaces and features like QoS and congestion control.
  • Performance: iPerf3 is optimized for modern networks but lacks multithreaded capabilities, which can sometimes limit its performance on high-bandwidth or multi-core systems.
  • Use Case: Best for most modern networking environments where simpler performance tests are required without the need for multithreading.

Key Differences

Feature iPerf iPerf2 iPerf3
Development Discontinued Actively Maintained Actively Maintained
TCP and UDP Tests Yes Yes Yes
Multithreading Support No Yes No
UDP Multicast No Yes No
IPv6 Support No Yes Yes
JSON Output No No Yes
Reverse Mode No Yes Yes
Note

We will use iPerf2 where this is possible throughout this tutorial.

Best for High Throughputs?

For high-throughput environments, iPerf2 is often the best choice due to its multithreading capabilities, which can take full advantage of multiple CPU cores. This is especially important if you're working with network interfaces capable of handling multiple gigabits per second (Gbps) of traffic.

If multithreading isn't crucial, iPerf3 is a good choice for simpler setups or modern networks with features like QoS and congestion control. However, in very high-throughput environments, its single-threaded nature might become a bottleneck.

Why is MSS Clamping used?

Note

When traffic is flowing through an IPSEC tunnel through the pfSense Firewall MSS is something to pay attention to.

MSS Clamping refers to "Maximum Segment Size Clamping," which is a technique used in network communications, particularly in TCP/IP networks, to adjust the maximum segment size (MSS) of a TCP packet during the connection setup process. The MSS defines the largest amount of data that a device can handle in a single TCP segment, and it's typically negotiated between the communicating devices during the TCP handshake.

MSS Clamping is often employed by network devices such as routers, firewalls, or VPNs to avoid issues related to packet fragmentation. Here's how it works:

  1. Packet Fragmentation Issues: If the MSS is too large, packets may exceed the Maximum Transmission Unit (MTU) of the network path, leading to fragmentation. This can cause inefficiency, increased overhead, or in some cases, packet loss if the network doesn't handle fragmentation well.
  2. Reducing the MSS: MSS Clamping allows the network device to adjust (or "clamp") the MSS value downward during the TCP handshake, making sure that the packet sizes are small enough to traverse the network path without needing fragmentation.
  3. Use in VPNs: MSS Clamping is commonly used in VPN scenarios where the MTU size is reduced due to encryption overhead. Without MSS Clamping, packets might get fragmented, reducing performance.

Example of MSS clamping

If a client device sends an MSS value of 1460 bytes during the TCP handshake but the network's MTU is limited to 1400 bytes due to VPN encapsulation, the network device can clamp the MSS to 1360 bytes (allowing for the extra overhead) to avoid fragmentation issues.

Important Information (before you start)

Ports Used

The default ports used by iPerf2 and iPerf3 for TCP and UDP are:

TCP Port UDP Port
iPerf2 5001 5001
iPerf3 5201 5201

Both versions allow you to specify a different port using the -p flag if necessary.

For testing purposes, I recommend opening ALL the ports between the SOURCE and DESTINATION IP addresses of the iPerf endpoints.

MTU Sizes

iPerf will send data between

When running an iPerf test, understanding the MTU (Maximum Transmission Unit) size is crucial because it directly impacts network performance, packet fragmentation, and test accuracy. Here's what you should consider regarding MTU sizes during an iPerf test:

Default MTU Size

  • The default MTU size for Ethernet is 1500 bytes, but this can vary based on the network configuration.
  • Larger or smaller MTU sizes can affect the maximum size of packets sent during the iPerf test. Smaller MTU sizes will require more packets for the same amount of data, while larger MTU sizes can reduce the overhead.

Packet Fragmentation

  • If the MTU size is set too small, or if the iPerf packet size is larger than the network's MTU, packets may be fragmented. Fragmented packets can lead to higher latency and reduced performance in your test.
  • iPerf can generate packets up to a specific size, and if they exceed the MTU, they’ll need to be split, introducing extra overhead and making the results less reflective of real-world performance.

Jumbo Frames

  • Some networks support jumbo frames, where the MTU is larger than the standard 1500 bytes, sometimes reaching 9000 bytes. When testing in environments with jumbo frames enabled, configuring iPerf to match this larger MTU can maximize throughput by reducing overhead from headers and fragmentation.

MTU Discovery and Path MTU

  • Path MTU discovery helps ensure that packets do not exceed the MTU of any intermediate network. If iPerf sends packets larger than the path MTU and fragmentation is not allowed, the packets might get dropped.
  • It’s important to ensure that ICMP "Fragmentation Needed" messages are not blocked by firewalls, as these help with path MTU discovery. Without it, larger packets may not be successfully delivered, resulting in performance issues.

TCP vs UDP Testing

  • In TCP mode, iPerf automatically handles packet size and adjusts according to the path MTU.
  • In UDP mode, the packet size is controlled by the user (using the -l flag), and this size must be less than or equal to the MTU to avoid fragmentation.

Adjusting MTU in iPerf

  • Use the -l option in iPerf to manually set the length of UDP datagrams.
  • For testing with specific MTU sizes, it's useful to ensure that your network and interfaces are configured to match the desired MTU value to avoid mismatches.

Consistency Across Network Segments

  • Ensure the MTU size is consistent across all network devices between the two endpoints. Mismatched MTU settings can cause inefficiency due to fragmentation or dropped packets, leading to inaccurate test results.

VPN Related

When using a VPN (Virtual Private Network), MTU size and network performance become even more significant due to the additional layers of encapsulation and encryption. VPNs introduce extra overhead, which can affect the performance of tools like iPerf.

Here’s a deeper look at VPN connections and their impact on network testing:

Key Concepts of VPN and MTU

  1. Encapsulation Overhead
  • VPN protocols, such as IPsec, OpenVPN, WireGuard, PPTP, or L2TP, add extra headers to the original data packet for encryption and tunneling purposes.
  • This extra overhead reduces the effective MTU size because the VPN must accommodate both the original packet and the added VPN headers. For example:
    • IPsec adds around 56 to 73 bytes of overhead.
    • OpenVPN adds about 40-60 bytes, depending on the configuration (e.g., UDP vs. TCP).
    • WireGuard adds around 60 bytes.
  • If you don’t adjust the MTU, packets larger than the adjusted MTU may get fragmented or dropped.
  1. MTU and Path MTU Discovery in VPNs
  • VPNs often create tunnels that span multiple networks, and the path MTU between the two ends of the tunnel can be smaller than what would be used on a direct connection. Path MTU discovery helps VPNs avoid fragmentation, but some networks block ICMP messages, which are essential for this discovery.
  • If ICMP messages like "Fragmentation Needed" are blocked, the VPN tunnel may send packets that are too large for an intermediate network, causing packet loss or retransmissions.
  1. Fragmentation Issues
  • When an MTU mismatch occurs, the VPN will either fragment the packets at the network level or, if fragmentation is not allowed (DF, or "Don’t Fragment" bit is set), drop the packets. Fragmentation introduces additional latency, lowers throughput, and can cause packet loss.
  • VPNs often have a lower effective MTU (e.g., 1400 bytes instead of 1500), which accounts for the added headers and prevents fragmentation.
  1. Adjusting MTU for VPN Connections
  • Most VPN clients or routers allow the user to adjust the MTU size to avoid fragmentation. For example, reducing the MTU size on a VPN tunnel to 1400 or 1350 bytes is common to account for VPN overhead.

Instance Network Speeds

Within OCI the speed of the Network Adapter (vNIC) or your instance is bound to the Instance Shape and the amount of CPUs you have assigned to that shape. In this tutorial, I am using E4.Flex shapes with an Oracle Linux 8 Image with 1 OCPU. This means I will get a (maximum) network bandwidth of 1 Gbps (for all my iPerf test results).

Below I have provided an example of one of my instances.

  1. Notice that the shape is E4.Flex.
  2. Notice that the OCPU count is 1.
  3. Notice that the network bandwidth is 1 Gbps.

07d27b1228653fb8635517663c802d89.png

Note

It is possible to increase the network bandwidth by choosing another shape and increasing the amount of OCPUs.

STEP 01: Review the OCI Hub and Spoke VCN Routing architecture

We will use the following architecture below for all the iPerf throughput tests throughout this tutorial.

876642643d8d8cbc5e1af9085ffafe59.png

Notice that this is a full hub and spoke routing architecture with ON-PREM connected with an IPSec VPN tunnel. If you want to recreate this routing topology please read the following tutorials:

STEP 02: Install iPerf3 on the Hub Instances

Before we can use iPerf we need to make sure iPerf is installed. We will assume that iPerf is not installed.

Note

In this step we will install iPerf3, and we will install iPerf2 in the next step.

Hub Stepstone

The Hub Stepstone is a Windows Server Instance. There are different iPerf distributions available for [windows] and I have downloaded this one [here].

Download the zip file and unpack the file on the Hub Stepstone.

Cb63ed1efd0cfc206a84f3defbee3ee9.png

  1. Browse to the directory where you have unpacked the iPerf zip file.
  2. Verify if the unpacked folder is available.
  3. Notice another iPerf folder is there.
  4. Change the directory to go one level deeper in the folder.
  5. Verify what files are inside the iPerf folder.
  6. Notice the iPerf.exe file that we need to perform the actual tests.

57b9ed9d7d3b3356f98afd4d0572223f.png

  • Execute the iPerf.exe command just to see if it works.

E31d3aa90c4cd7539990dc1bd4e9c7a2.png

pfSense Firewall

To install iPerf on the pfSense we need to install a package through the Package Manager.

  1. Browse to System Menu.
  2. Select Package Manager.

49e3ca05bd0faf194b423747342185e5.png

  • Click on Available Packages.

Cd38106da10721d92a000f66b5f5da8f.png

  1. Type in the keyword "iPerf".
  2. Click on the Search button.
  3. Notice that there will be one result and this is the iPerf package version 3.0.3 (at the time of writing).
  4. Click on the +Install button.

Db00e070c8c7ecb4e98b2f40557bbb21.png

  • Click on the Confirm button.

77ed1180f26af0cffc024560f0df864e.png

  • Notice that the number of packages installed is 2.

Fcb49398414224c3713f5062e62d6d8e.png

  1. Browse to Diagnostics Menu.
  2. Select iPerf.

29098033fd050f4cc121cad1dc0f133f.png

  • Click on the Client tab.

25cdd799e1b4481fc948406ff89087e2.png

  • Click on the Server tab.

Bffb4acb1902e2f8e41feb0c841281e1.png

Note

The pfSense firewall does not have the option (by default) to install the iPerf version 2 packages.

STEP 03: Install iPerf3 on the Spoke Instances

Now we are going to install iPerf3 on the Linux Instances (inside OCI) we have in our architecture.

Spoke Instance A1 and A2

The instance A1 already has iPerf3 installed.

  1. Connect to the Instance A1.
  2. Issue the following command: sudo dnf install iPerf3
  3. Notice that iPerf3 is already installed.

3f0f7d7b2b266e66be93525821a6c010.png

  • Issue the command iPerf3 -v to verify the iPerf version that is installed.

2517a27f9d3040da59a36a0a823b0479.png

The instance A2 does not have iPerf3 installed.

  1. Connect to the Instance A2.
  2. Issue the following command: sudo dnf install iPerf3
  3. Type in "Y".

577fee3ff3e46bee483dc6d4b68295d4.png

  • iPerf3 will install and notice that the installation has been completed.

0c8d6c9cf7c85fb4550fe407873ac078.png

Spoke Instance B

  1. Connect to the B Instance.
  2. Issue the command the install iPerf 3 (provided in the previous section) and if required, complete the installation and if iPerf3 is already available, you will get a message that iPerf is already installed.

Cd84293b2d89ca941fb2fd631f519cff.png

Spoke Instance C

  1. Connect to the C Instance.
  2. Issue the command the install iPerf 3 (provided in the previous section) and if required, complete the installation and if iPerf3 is already available, you will get a message that iPerf is already installed.

50bf85cc6248aae48da45502f0082e18.png

Instance D

  1. Connect to the D Instance.
  2. Issue the command the install iPerf 3 (provided in the previous section) and if required, complete the installation and if iPerf3 is already available, you will get a message that iPerf is already installed.

396bfbbed8324e37fa5f5e37fc690230.png

STEP 04: Install iPerf on the ONPREM Instances

Now we are going to install iPerf3 on the Linux Instances (ONPREM) we have in our architecture.

Oracle Linux Client

  1. Connect to the ON-PREM Linux Client Instance.
  2. Issue the command the install iPerf 3 (provided in the previous section) and if required, complete the installation and if iPerf3 is already available, you will get a message that iPerf is already installed.

C7bc9cb999e3aea5f7a92a4415306815.png

Oracle Linux Client CPE

  1. Connect to the ON-PREM Linux CPE Instance.
  2. Issue the command the install iPerf 3 (provided in the previous section) and if required, complete the installation and if iPerf3 is already available, you will get a message that iPerf is already installed.

044a3a10c81320106338025d7e3f2b55.png

STEP 05: Install iPerf2

Now that we have installed iPerf3, we are going to install iPerf2 on ALL the Linux Instances throughout the architecture.

We are using Oracle Linux 8 so we will need the following iPerf 2 package: [Oracle Linux 8 (x86_64) EPEL]

If you are using Oracle Linux 9, use this package: [Oracle Linux 9 (x86_64) EPEL]

When you use another OS or Linux distribution use a package that is compiled for your OS.

  • Use the command below to install iPerf 2 on all Oracle Linux 8 Instances.
sudo dnf install https://yum.oracle.com/repo/OracleLinux/OL8/developer/EPEL/x86_64/getPackage/iPerf-2.1.6-2.el8.x86_64.rpm
  • Confirm the installation with Y.

268586268bbd763be6f2af6e546596e4.png

  • iPerf2 will install and notice that the installation has been completed.

E5be365187db4aa5f93a8e9b16078e6f.png

  1. Issue the command iPerf -v to verify the iPerf version that is installed.
  2. Notice that iPerf v 2.1.6 is installed.

C81c0fb78c72d6892605f9da873b9e63.png

Note

Make sure you install iPerf2 on all other Instances as well.

For the Windows-based Hub Stepstone, we can download a standalone [iPerf-2.2.n-win64] executable.

  1. Execute the iPerf.exe command just to see if it works.
  2. Issue the command iPerf -v to verify the iPerf version that is installed.
  3. Notice that iPerf v 2.2.n is installed.

Acaaef3624a54eec61cb80d1b3633640.png

STEP 06: Define the iPerf Tests and prepare the iPerf commands

Below I will provide some iPerf commands with the additional flags and explain what they mean. Some more information on the commands can be found here: [Oracle Network Performance documentation].

Basic iPerf commands for testing with TCP:

On the iPerf server side:

iPerf3 -s

On the iPerf client side:

iPerf3 -c <server_instance_private_ip_address>

iPerf commands that we will use for testing with TCP:

  • Bi-directional bandwidth measurement: (-r argument)
  • TCP Window size: (-w argument)

On the iPerf server side:

iPerf3 -s -w 4000

On the iPerf client side:

iPerf3 -c <server_instance_private_ip_address> -r -w 2000

iPerf3 -c <server_instance_private_ip_address> -r -w 4000

iPerf commands that we will use for testing with UDP:

  • UDP tests: (-u), bandwidth settings (-b)

On the iPerf server side:

iPerf -s -u -i 1

On the iPerf client side:

iPerf -c <server_instance_private_ip_address> -u -b 10m

iPerf -c <server_instance_private_ip_address> -u -b 100m

iPerf -c <server_instance_private_ip_address> -u -b 1000m

iPerf -c <server_instance_private_ip_address> -u -b 10000m

iPerf -c <server_instance_private_ip_address> -u -b 100000m

iPerf commands that we will use for testing with TCP (with MSS):

  • Maximum Segment Size (-m argument) display:

On the iPerf server side:

iPerf -s

On the iPerf client side:

iPerf -c <server_instance_private_ip_address> -m

iPerf commands that we will use for testing with TCP (parallel ):

On the iPerf server side:

iPerf -s

On the iPerf client side:

iPerf -c <server_instance_private_ip_address> -P 2
Note

For all of the tests we will perform in this tutorial we will use the commands below.

iPerf FINAL command for testing

  • Bandwidth settings (-b)
  • Parallel tests (-P argument):

To test the throughput for a 100Gb connection with 100Gbps we set the throughput to 9Gbps with 11 parallel streams.

On the iPerf server side:

iPerf -s

On the iPerf client side:

iPerf -c <server_instance_private_ip_address> -b 9G -P 11

STEP 07: Preform iPerf tests within the same VCN in the same subnet

During this step we are going to perform an iPerf2 throughput test within the same VCN and the same subnet. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

9364f71c82af62e43904be4306c91c53.png

From Instance-A1 to Instance-A2

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.1.50
IP of the iPerf client 172.16.1.93
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.1.50 -b 9G -P 5
Tested Bandwidth (SUM) 1.05 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

0399ec0de8d03b5f942f5aa77a3cfa94.png

De1d9effd89b97df4c2ba700a09414fb.png

Ed4175e9bec81f6955edfb901e22f74a.png

From Instance-A2 to Instance-A1

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.1.93
IP of the iPerf client 172.16.1.50
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.1.93 -b 9G -P 5
Tested Bandwidth (SUM) 1.05 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

70b572c988a22b89f4c458154c74dcb4.png

D45919d9777c6b271e92ecd86c416feb.png

Db1b0b702e0519a81d4797952aa2e9bb.png

STEP 08: Preform iPerf tests within the same VCN across different subnets

During this step, we are going to perform an iPerf3 throughput test within the same VCN but two different subnets. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

00c8c66744a27302a2aebed2e65a2fb1.png

From pfSense Firewall to hub Stepstone

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.0.252
IP of the iPerf client 172.16.0.20
iPerf command on the server iPerf3 -s
iPerf command on the client iPerf3 -c 172.16.0.252
Tested Bandwidth (SUM) 958 Mbytes/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

Bdb4607243db2cee24995640bbfb1a41.png

D4d594ba06743eee2617ae50f384c863.png

Ca3dd2230576b40b435be0078b706fbd.png

B820beee6921e4539c57d1f4d4ab474a.png

From hub Stepstone to pfSense Firewall

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.0.20
IP of the iPerf client 172.16.0.252
iPerf command on the server iPerf3 -s
iPerf command on the client iPerf3 -c 172.16.0.20
Tested Bandwidth (SUM) 1.01 Gbit/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

98ce96a5789f08bc0a324655fd680521.png

8fe3ea765ab8f9aab59c347e5d4a04e7.png

F24637bf44721f9b68508bbc6e3f50d7.png

936a798eaf7bb76384997202e43e79ea.png

STEP 09: Preform iPerf tests between two different VCNs

During this step we are going to perform an iPerf2 throughput test between two different VCNs and two different subnets. Note that the test will go through a firewall that is located in the Hub VCN. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

6715ed814141735654ccc2fb86e871a7.png

From Instance-A1 to Instance-B

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.2.88
IP of the iPerf client 172.16.1.93
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.2.88 -b 9G -P 5
Tested Bandwidth (SUM) 1.02 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

57511b32617b43e8f69ad80e5d6c2923.png

612556c962db7afd42630fccb03752be.png

496e3114c5345c2c4f5ac7eb9c810e3c.png

From Instance B to Instance A1

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.1.93
IP of the iPerf client 172.16.2.99
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.1.93 -b 9G -P 5
Tested Bandwidth (SUM) 1.02 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

D437ee9e6c986204db44af96b598ca49.png

632b0aefff9c5d9f6110e6e46df42e7a.png

F37a942ea09385252d52dceb57aefc86.png

STEP 10: Preform iPerf tests between different VCNs (bypassing the pfSense Firewall)

During this step we are going to perform an iPerf2 throughput test between two different VCNs and two different subnets. Note that the test will bypass the firewall that is located in the Hub VCN. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.


Ac5aa396c67bc49e4a09f56ef75df1a8.png

From Instance-C to Instance-D

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.4.14
IP of the iPerf client 172.16.3.63
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.4.14 -b 9G -P 5
Tested Bandwidth (SUM) 1.04 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

F5f20ce48a2aba14a59ebc28989d558e.png

F2e970429950c059276a90e8d39ff17d.png

851ac3f3f4ff3f0d66675516016b9230.png

From Instance-D to Instance-C

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.3.63
IP of the iPerf client 172.16.4.14
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.3.63 -b 9G -P 5
Tested Bandwidth (SUM) 1.05 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

5c0a09e12131939f7571e07db970bd57.png

5cd8034fccae2a222fa209d636ac6974.png

0448b63d2bc88ed1dd95b1068d7166fb.png

STEP 11: Preform iPerf tests between ONPREM and OCI Hub VCN

During this step we are going to perform an iPerf2 throughput test between ONPREM and OCI using a Site-to-Site IPSec VPN tunnel. Note that the test will go through the firewall that is located in the Hub VCN. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

E29c4c972933f8cceae0a7b3e27672f8.png

Note

When you are performing throughput tests (with or without iPerf) using a VPN IPSec tunnel and a pfSense (firewall) MTU and MSS an important factor to take into account, when this is done wrong, the throughput results will be invalid and not as expected.

With iPerf you can tweak the packet stream so that the packets are sent with a specific MSS, you can use this if you are not able to change the MSS settings on the devices in the path between your source or destination.

Maximum Segment Size Clamping

In my case the ONPREM side had an MTU of 9000 sending a packet with the MSS value of 1500 + IPSec overhead.

The pfSense Interface MTU is 1500 ... causing fragmentation issues.

By setting the interface MSS to 1300 it changes the size "on the fly" and this technique is called "Maximum Segment Size Clamping". More information about this is provided at the beginning of this tutorial.

MSS Change on the pfSense

19dc05200d4bfed9a5d353d3df3049d3.png


A79076e276ee1d222ad8f0b0658c0c2d.png

Eb04833cc406cabf2bb54e2f1473efe1.png

Be6da11150a0fde571780329adc55021.png

60e4a39f2c18a1f9fc4ef23924d08060.png

From VPN Client Instance (ONPREM) to Hub Stepstone

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.0.252
IP of the iPerf client 10.222.10.19
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.0.252 -b 9G -P 5
Tested Bandwidth (SUM) 581 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

8a869f19c6549e550d0d49fe08b6c139.png

Efbc1f8cb087660a36e31e7c24c2ed2c.png

F8e1e979d5ed8f31cc8cfc338e05e4e4.png

From Hub Stepstone to VPN Client Instance (ONPREM)

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.19
IP of the iPerf client 172.16.0.252
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.19 -b 9G -P 5
Tested Bandwidth (SUM) 732 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

Ed4e334019f8c728ce0c4c4a912f9b1d.png

37dc15ffdc0e13087a976219073cbef7.png

C0ce5968793ddc1354742e17477e7ae2.png

STEP 12: Preform iPerf tests between ONPREM and OCI Spoke VCN

During this step we are going to perform an iPerf2 throughput test between ONPREM and OCI using a Site-to-Site IPSec VPN tunnel. Note that the test will go through the firewall that is located in the Hub VCN. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

939d9d323f749f8e953ced17d6c4bd56.png

From VPN Client Instance (ONPREM) to Instance-A1

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.1.93
IP of the iPerf client 10.222.10.19
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.1.93 -b 9G -P 5
Tested Bandwidth (SUM) 501 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

E0bd6711d51208a91fa7a2956361c1a2.png

349b4a0fd43653b2e794684ea6c278a1.png

D8b0b11c43ef13f09f9ff43daff8bd90.png

NEW TESTS WITH MMS IN iPerf COMMAND

Note

With iPerf you can tweak the packet stream so that the packets are sent with a specific MSS, you can use the following commands if you are not able to change the MSS settings on the devices in the path between your source or destination.

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.1.93
IP of the iPerf client 10.222.10.19
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.1.93 -b 9G -P 5 -M 1200
Tested Bandwidth (SUM) 580 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

Ba71d4b9b233ca08fd8733bf623d20e9.png

12fa9a66ebcde1afd31b9cd1197e6284.png

489bb3c50be5a428e8607cdc92ee3b73.png

From Instance-A1 to VPN Client Instance (ONPREM)

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.19
IP of the iPerf client 172.16.1.93
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.19 -b 9G -P 5
Tested Bandwidth (SUM) 620 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

E78b6f281e0bfd29a4c0b0a0fd12be31.png

F46132c0a4a0b74192bcd9052d825236.png

780d204f7bd456ff982473e8e04a56a8.png

NEW TESTS WITH MMS IN iPerf COMMAND

Note

With iPerf you can tweak the packet stream so that the packets are sent with a specific MSS, you can use the following commands if you are not able to change the MSS settings on the devices in the path between your source or destination.

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.19
IP of the iPerf client 172.16.1.93
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.19 -b 9G -P 5 -M 1200
Tested Bandwidth (SUM) 805 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

57dccae6ab554ace2e500526902e2ceb.png

D82e5d153e49c1badc2f9a71736ef04b.png

Bd4006d2ed2cfd5da6254bfc34e1c715.png

STEP 13: Preform iPerf tests between ONPREM and OCI Spoke VCN (bypassing the pfSense Firewall)

During this step we are going to perform an iPerf2 throughput test between ONPREM and OCI using a Site-to-Site IPSec VPN tunnel. Note that the test will bypass the firewall that is located in the Hub VCN. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

C487532b28eb2675dd4020991b91b3a0.png

From VPN Client Instance (ONPREM) to Instance-D

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 172.16.4.14
IP of the iPerf client 10.222.10.19
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 172.16.4.14 -b 9G -P 5
Tested Bandwidth (SUM) 580 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

F58bf895b8e4505a14715c72a1ad0b97.png

91f1bf236bea07dcd39cd58a75b9f75d.png

60a416f44c3c20b47f92b88a4116b651.png

From Instance-D to VPN Client Instance (ONPREM)

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.19
IP of the iPerf client 172.16.4.14
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.19 -b 9G -P 5
Tested Bandwidth (SUM) 891 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

6f8fe3a35b4d8e4660b79e2bbaafe988.png

A09389398e96ad4e9589cfd610bbd5e4.png

D5c5b2c32d7b8f74ac2d312c2a23bc49.png

STEP 14: Preform iPerf tests between the INTERNET and the OCI Hub VCN

During this step we are going to perform an iPerf2 throughput test between a client on the internet and OCI using the internet. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

Cf633d0f8805c1a9a788e15fc8becba8.png

From Internet to Hub Stepstone

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server xxx.xxx.xxx.178
IP of the iPerf client xxx.xxx.xxx.152
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c xxx.xxx.xxx.178 -b 9G -P 5
Tested Bandwidth (SUM) 251 Mbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

Be8a8035a65d22189715e4d51c5bf7d5.png

0785a8a6e7cca760c0f42c2c364bb5f1.png

3e8a17c8e6b767c73ca130160e6083c0.png

STEP 15: Preform iPerf tests within the same subnet ONPREM

During this step we are going to perform an iPerf2 throughput test between two ONPREM instances. The image below shows the paths with the arrows between what two endpoints where we are going to perform the throughput tests.

0909c896b4c6774110da9e1854d5fe39.png

From VPN Client Instance (ONPREM) to StrongSwan CPE Instance (ONPREM)

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.70
IP of the iPerf client 10.222.10.19
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.70 -b 9G -P 5
Tested Bandwidth (SUM) 1.05 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

Ec8f560397de48bd34c82254d21db7ed.png

15f4ed7acd45576fcbfa4e56e416379d.png

E157ceb5bf6eb97853731fbb9006797e.png

From StrongSwan CPE Instance (ONPREM) to VPN Client Instance (ONPREM)

In the table below you will find the IP address of the client and the server (used in this test), and the commands used to perform the iPerf test with the test results.

IP of the iPerf server 10.222.10.19
IP of the iPerf client 10.222.10.70
iPerf command on the server iPerf -s
iPerf command on the client iPerf -c 10.222.10.19 -b 9G -P 5
Tested Bandwidth (SUM) 1.05 Gbits/sec

In the next screenshots, you will also find the full testing outputs of the iPerf tests.

A11ae9e31066634209e70ee36f74527b.png

1b43592a235473a62a52a75be979adb0.png

8081dfb82a7fca2df4a8b13b9341e3fe.png

Conclusion

In this tutorial, we have performed different types of throughput tests using iPerf2 and iPerf# The tests were performed on various different sources and destinations in the full network architecture with different paths.

In the table below you can see a summary of of the test results that we collected.

Test Type Bandwidth Result
From Instance-A1 to Instance-A2 1.05 Gbits/sec OCI internal
From Instance-A2 to Instance-A1 1.05 Gbits/sec OCI internal
From pfSense Firewall to hub Stepstone 958 Mbytes/sec OCI internal
From hub Stepstone to pfSense Firewall 1.01 Gbit/sec OCI internal
From Instance-A1 to Instance-B 1.02 Gbits/sec OCI internal
From Instance B to Instance A1 1.02 Gbits/sec OCI internal
From Instance-C to Instance-D 1.04 Gbits/sec OCI internal
From Instance-D to Instance-C 1.05 Gbits/sec OCI internal
From VPN Client Instance (ONPREM) to Hub Stepstone 581 Mbits/sec ONPREM to OCI trough firewall
From Hub Stepstone to VPN Client Instance (ONPREM) 732 Mbits/sec ONPREM to OCI trough firewall
From VPN Client Instance (ONPREM) to Instance-A1 501Mbits/sec ONPREM to OCI trough firewall
From Instance-A1 to VPN Client Instance (ONPREM) 620 Mbits/sec ONPREM to OCI trough firewall
From VPN Client Instance (ONPREM) to Instance-D 580 Mbits/sec ONPREM to OCI firewall bypass
From Instance-D to VPN Client Instance (ONPREM) 891 Mbits/sec ONPREM to OCI firewall bypass
From Internet to Hub Stepstone 251 Mbits/sec INTERNET to OCI
From VPN Client Instance (ONPREM) to StrongSwan CPE Instance (ONPREM) 1.05 Gbits/sec ONPREM to ONPREM
From StrongSwan CPE Instance (ONPREM) to VPN Client Instance (ONPREM) 1.05 Gbits/sec ONPREM to ONPREM