Instances, Databases, and OCI Network Services are the typical building blocks for your applications inside OCI. When you build an application on the network it is good practice to know if the traffic you are sending from a source is reaching the destination. OCI offers the Network Path Analyser to verify the path, but sometimes you want to have a bit more information on what is happening on the endpoints itself on a hop-by-hop basis.

This tutorial will provide a few techniques that you can use to follow your packets in a Hub and Spoke Network Architecture inside OCI.

The different routing flow scenarios

Multiple routing flow scenarios are applicable in a Hub and Spoke routing architecture that are all listed below.

SCENARIO 01: SPOKE INSTANCE > SPOKE INSTANCE
SCENARIO 02: SPOKE INSTANCE > HUB INSTANCE
SCENARIO 03: HUB INSTANCE > SPOKE INSTANCE
SCENARIO 04: SPOKE INSTANCE > INTERNET
SCENARIO 05: SPOKE INSTANCE > OCI SERVICE NETWORK
SCENARIO 06: HUB > INTERNET
SCENARIO 07: INTERNET > HUB
SCENARIO 08: ON-PREM > SPOKE INSTANCE
SCENARIO 09: SPOKE INSTANCE > ON-PREM
SCENARIO 10: EXTERNAL > LB > SPOKE INSTANCE

In this tutorial, we are only going to fully explain how you can follow the packet with SCENARIO 0# For the other SCENARIOS, you can use the same method and this will be a lot of repetition explaining the same thing over and over again.

SCENARIO 01: SPOKE INSTANCE > SPOKE INSTANCE

In this scenario we are going to follow the packet from one Spoke VCN Instance to an Instance in another Spoke VCN.

The diagram is shown below, with the hops specified.

The A - F markers indicate the places where you can enable some form of logging or packet capturing. We will explore all these places one by one so we can follow the packet.

The diagram

The Steps

Before we start we need to determine our source and destination and the port that we are going to test with. This will ensure we have a targeted way of analyzing the packet without too much distraction.

In this example, we are using the following:

Source	Destination	Destination Port
172.16.1.93	172.16.2.88	TCP/80

To properly start this journey, the order of operations is important to be as effective as possible with setting up the logging and capturing and collecting the information properly.

Below are the steps that I followed to collect all the data that was required for my defined source and destination.

[ ] STEP 01: Note down the time.
[ ] STEP 02: Open the first SSH session on Instance A.
[ ] STEP 03: Open a second SSH session on Instance A.
[ ] STEP 04: Open an SSH session on Instance B.
[ ] STEP 05: Open a WEB session on the pfSense Firewall (navigate to the packet capture page).
[ ] STEP 06: Enable logging (ALL LOGS) on the Spoke A Private Subnet.
[ ] STEP 07: Enable logging (ALL LOGS) on the Spoke B Private Subnet.
[ ] STEP 08: Enable logging (ALL LOGS) on the Hub Private Subnet.
[ ] STEP 09: On the second SSH session on Instance A start the TCP dump session.
[ ] STEP 10: On the SSH session on Instance B start the TCP dump session.
[ ] STEP 11: On the pfSense Firewall start the packet capture.
[ ] STEP 12: On the first SSH session on Instance A do a CURL to the Instance B web server.
[ ] STEP 13: Look at all the logging and packet captures (and follow the path).

We will go over each step one by one and look at the results.

STEP 01: Note down the time

I am using a central Computer to collect all the data.
Note down the time when you start the analytics (In my case this is now 8.44 AM).

STEP 02: Open the first SSH session on Instance A

We will use this session to initiate the HTTP connection to the destination.

Connect with SSH to the terminal of Instance A located in VCN A.
Try to use a tabbed connection. This enables quick switching between the sessions.
Issue the command `timedatectl` to get the current time of the Instance.
Make sure the time set is ok. (In my case this is now 8.44 AM).

STEP 03: Open a second SSH session on Instance A

Open a SECOND connection to the Instance A located in VCN A.
We will use this session to enable the TCPdump commands so we can start the packet following using TCPdump.

STEP 04: Open an SSH session on Instance B

We will use this session to receive the HTTP connection from the source.

Connect with SSH to the terminal of Instance B located in VCN B.
Try to use a tabbed connection. This enables quick switching between the sessions.
Issue the command `timedatectl` to get the current time of the Instance.
Make sure the time set is ok. (In my case this is now 8.45 AM).

STEP 05: Open a WEB session on the pfSense Firewall (navigate to the packet capture page)

Because we are using a Hub and Spoke routing architecture the traffic will transit the pfSense firewall if traffic is flowing from one Spoke to another Spoke. The pfSense firewall will either permit or deny the traffic. We want to see this in the pfSense firewall using the packet captures.

Open the management console of the pfSense Firewall.
Click on Diagnostics > Packet Capture.

Make sure you have the pfSense Firewall diagnostics/packet capture screen open.
DO NOT START the packet capture yet, we just want to have it open.

STEP 06: Enable logging (ALL LOGS) on the Spoke A Private Subnet

Because our source is attached to the Private Subnet in the Spoke A VCN we will enable logging on the Subnet level in the VCN.

Click on the hamburger menu in the OCI console.
Click on Networking.
Click on Virtual cloud networks.

Click on the Spoke A VCN.

Click on the Private Subnet inside VCN A.

Click on Logs.
Toggle the Enable Log switch for all logs and set it to enabled.

Leave all log settings default and click on the Enable log button.

Notice that the Status will change to Creating.

Notice that the Status will change from Creating to Active.
Notice that the Enable log switch for all is now Enabled.

STEP 07: Enable logging (ALL LOGS) on the Spoke B Private Subnet

Follow the same steps as you did (in STEP 06) to enable the logs for the Private Subnet in the Spoke VCN A, but now do it for the Spoke VCN B Private Subnet.

Follow the path: Networking > Virtual Cloud Networking > Spoke B VCN > Subnet Details > Logs.
Make sure that the Status is set to Active.
Make sure the Enable log switch for all is Enabled.

STEP 08: Enable logging (ALL LOGS) on the Hub Private Subnet

Follow the same steps as you did (in STEP 06) to enable the logs for the Private Subnet in the Spoke VCN A, but now do it for the Spoke VCN B Private Subnet.

Follow the path: Networking > Virtual Cloud Networking > Hub VCN > Subnet Details > Logs.
Make sure that the Status is set to Active.
Make sure the Enable log switch for all is Enabled.

STEP 09: On the second SSH session on Instance A start the TCP dump session

Switch back to the second SSH Session of Instance A.
Issue the following command to enable TCPdump: `sudo tcpdump -i ens3 dst 172.16.2.88 and src 172.16.1.93 and dst port 80`
Notice that TCPdump will be enabled.

When using the TCPdump command make sure you use proper filters, so you ONLY see information on the traffic you are looking for. Here I specified the source, destination, and port.

STEP 10: On the SSH session on Instance B start the TCP dump session

Switch back to the SSH Session of Instance B.
Issue the following command to enable TCPdump: `sudo tcpdump -i ens3 src 172.16.0.20 and port 80`
Notice that TCPdump will be enabled.

`

STEP 11: On the pfSense Firewall start the packet capture.

Switch back to the pfSense Firewall console.
Scroll down to the Custom Filter Options section.
In the Untagged Filter section:
- Select: include any of.
- For the destination: Select all of and specify 172.16.2.88.
- For the port: Select any of and specify 80.
Click on the Start button.

Scroll down.

Notice that there are already packets captured.
This will be ALL traffic that is on port 80 and that is for our destination IP address (172.16.2.88).
There is already traffic visible because I have a load balancer active that is doing health checks on port 80 as well.
We will leave the capture on for a few minutes while we do our HTTP test from our source so that we can use the Wireshark application later to filter out the correct information that we are looking for.

STEP 12: On the first SSH session on Instance A do a CURL to the Instance B web server

After preparing ALL the TCPdumps, packet captures, and logging it is finally time to initiate traffic from our source to our destination.

Switch back to the first SSH Session of Instance A.
Issue the command `timedatectl` again to get the current time of the Instance.
Make a note of the time. (In my case this is now 8.56 AM).

Issue the following command to send HTTP traffic: `curl http://172.16.2.88`
Notice that the web server on Instance B is replying with the HTML content.

Repeat the process by issuing the following command three or four times to send HTTP traffic multiple times: `curl http://172.16.2.88`

STEP 13: Look at all the logging and packet captures (and follow the path)

Now that we have all the information we need we can start collecting the information so we can follow the packet.

Stop all logging, packet captures, and TCP dumps

Before we start the collection let's first STOP all the logging, packet captures and TCP dumps just to make sure we do not get too much information.

Switch back to the second SSH Session of Instance A and press CNTRL + C to stop TCP DUMP.
Switch back to the SSH Session of Instance B and press CNTRL + C to stop TCP DUMP.

Switch back to the pfSense Firewall management console and stop the packet capture.

On the Subnets Disable Logging.

Follow the path: Networking > Virtual Cloud Networking > Hub VCN > Subnet Details > Logs.
Disable the logging for all by toggling the switch.

The status will change to updating.

The status will change from updating to inactive.

Follow the path: Networking > Virtual Cloud Networking > Spoke VCN A > Subnet Details > Logs.
Make sure that the Status is set to Inactive.
Make sure the Enable log switch for all is Disabled.

Follow the path: Networking > Virtual Cloud Networking > Spoke VCN B > Subnet Details > Logs.
Make sure that the Status is set to Inactive.
Make sure the Enable log switch for all is Disabled.

Now that we have disabled all the logging, packet captures and TCP dumps it is time to finally look at the data.

The diagram (with logging points)

On the diagram, you will find the points of logging and these are all marked with A, B, C, D, E, and F. This makes it easier to follow along.

Logging Point A (on diagram) - (TCP Dump on Instance A)

Switch back to the second SSH Session of Instance A.
Notice the TCP dump output that was generated.

I have copied/pasted the full raw output below so you can have a look.

[opc@ih-instance-vcn-a ~]$ sudo tcpdump -i ens3 dst 172.16.2.88 and src 172.16.1.93 and dst port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
08:56:50.489159 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [S], seq 1817516973, win 62720, options [mss 8960,sackOK,TS val 3736091466 ecr 0,nop,wscale 7], length
0
08:56:50.491649 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [.], ack 372277210, win 490, options [nop,nop,TS val 3736091468 ecr 671349263], length 0
08:56:50.491702 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736091468 ecr 671349263], length 75: HTTP: GET / HTTP/1.1
08:56:50.492114 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.492210 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.492982 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.493488 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.57422 > 172.16.2.88.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736091470 ecr 671349267], length 0
08:57:09.284210 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [S], seq 3049648180, win 62720, options [mss 8960,sackOK,TS val 3736110261 ecr 0,nop,wscale 7], length
0
08:57:09.284913 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [.], ack 728296369, win 490, options [nop,nop,TS val 3736110261 ecr 671368058], length 0
08:57:09.284973 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736110261 ecr 671368058], length 75: HTTP: GET / HTTP/1.1
08:57:09.285422 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.285522 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.285590 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.286060 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54210 > 172.16.2.88.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736110263 ecr 671368059], length 0
08:57:10.963760 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [S], seq 1863177286, win 62720, options [mss 8960,sackOK,TS val 3736111940 ecr 0,nop,wscale 7], length
0
08:57:10.964504 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [.], ack 464215720, win 490, options [nop,nop,TS val 3736111941 ecr 671369737], length 0
08:57:10.964551 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736111941 ecr 671369737], length 75: HTTP: GET / HTTP/1.1
08:57:10.965048 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.965135 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.965351 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.965865 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.54214 > 172.16.2.88.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736111942 ecr 671369739], length 0
08:58:23.663598 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [S], seq 407465876, win 62720, options [mss 8960,sackOK,TS val 3736184640 ecr 0,nop,wscale 7], length 0
08:58:23.664381 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [.], ack 3047424129, win 490, options [nop,nop,TS val 3736184641 ecr 671442437], length 0
08:58:23.664430 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736184641 ecr 671442437], length 75: HTTP: GET / HTTP/1.1
08:58:23.665005 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736184641 ecr 671442438], length 0
08:58:23.665129 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736184642 ecr 671442438], length 0
08:58:23.665297 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.53200 > 172.16.2.88.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736184642 ecr 6714
08:58:23.666655 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 874, win 485, options [nop,nop,TS val 3736184643 ecr 671442440], l
08:58:24.529502 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [S], seq 3610633804, win 62720, options [mss 8960,sackOK,TS val 3736185506
0
08:58:24.530337 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 575035162, win 490, options [nop,nop,TS val 3736185507 ecr 6714433
08:58:24.530386 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736185507 ecr 6714
 / HTTP/1.1
08:58:24.530886 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 239, win 489, options [nop,nop,TS val 3736185507 ecr 671443304], l
08:58:24.530966 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 873, win 485, options [nop,nop,TS val 3736185507 ecr 671443304], l
08:58:24.531347 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736185508 ecr 6714
08:58:24.531891 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 874, win 485, options [nop,nop,TS val 3736185508 ecr 671443305], l
08:58:25.562327 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [S], seq 3415948315, win 62720, options [mss 8960,sackOK,TS val 3736186539
0
08:58:25.563186 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 2043572435, win 490, options [nop,nop,TS val 3736186540 ecr 671444
08:58:25.563244 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736186540 ecr 6714
 / HTTP/1.1
08:58:25.563718 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 239, win 489, options [nop,nop,TS val 3736186540 ecr 671444337], l
08:58:25.563845 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 873, win 485, options [nop,nop,TS val 3736186540 ecr 671444337], l
08:58:25.564627 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736186541 ecr 6714
08:58:25.565113 IP ih-instance-vcn-a.ihprivatesubnet.spokevcna.oraclevcn.com.5
gs [.], ack 874, win 485, options [nop,nop,TS val 3736186542 ecr 671444338], l

When we filter out the lines (and analyze what is happening) we get something like this:

The first output section of the curl request that was made.
The second output section of the curl request that was made.
The source DNS/IP.
The destination DNS/IP.
The port.
The TCP Flags.

This output indicates that HTTP traffic was sent OUT of Instance A.

Logging Point B (on diagram) - (Logging on the Spoke VCN A Private Subnet)

Now it's time to also look at the subnet level logging to see IF the packets are also visible on the VCN's subnet.

Navigate to Networking > Virtual cloud networks > Spoke VCN A > (Private) Subnet Details > Logs.
Click on the log name for the all category.

Click on the Actions dropdown menu.
Click on Explore with Log Search.

In the Custom filters specify the following filters:

- `data.destinationAddress='172.16.2.88'`
- `data.sourceAddress='172.16.1.93'`

The filter for the destination is now added.

The filter for the source is now added.

Change the filter by time to Past hour.
Note that the bar chart indicates that there is log data available for the source and destination we use in the filter.
Scroll down.

Notice the source IP.
Notice the destination IP.
Notice the port.

Scroll up.
Click on Logs.

Logging Point C (on diagram) - (Logging on the Hub VCN Private Subnet)

Click on the Log name for the Hub VCN.

Because the traffic is not going from the firewall to the Instance B VCN we need to change the source and destination filters here.

In the Custom filters specify the following filters:
1. `data.destinationAddress='172.16.0.20'
2. `data.sourceAddress='172.16.1.93'`
3. `data.destinationPort='80'`
Note that the bar chart indicates that there is log data available for the source and destination we use in the filter.
Scroll down.

Notice the detailed logging section.
Notice the source IP.
Notice the destination IP.
Notice the port.

Logging Point D (on diagram) - (Packet capture on the pfSense Firewall)

Switch back to the pfSense Firewall management console and click on the download button.
Make sure the capture is downloaded.

Open the packet capture in the Wireshark application.

Specify the following Wireshark filter:
1. `(ip.dst == 172.16.2.88 && ip.src == 172.16.1.93) || (ip.dst == 172.16.2.88 && ip.src == 172.16.1.93) && http`
Notice the source IP.
Notice the destination IP.
Notice the port/Protocol.

Logging Point E (on diagram) - (Logging on the Spoke VCN B Private Subnet)

Navigate to Logging > Logs.
Click on the Log name for the Spoke VCN B.

Click on the Actions dropdown menu.
Click on Explore with Log Search.

In the Custom filters specify the following filters:
1. `data.destinationAddress='172.16.2.88'`
2. `data.sourceAddress='172.16.0.20'`
Note that the bar chart indicates that there is log data available for the source and destination we use in the filter.
Scroll down.

Notice the detailed logging section.
Notice the source IP.
Notice the destination IP.
Notice the port.

Logging Point F (on diagram) * (TCP Dump on Instance B)

Switch back to the SSH Session of Instance B.
Notice the TCP dump output that was generated.

I have copied/pasted the full raw output below so you can have a look.

[opc@ih-instance-vcn-b ~]$ sudo tcpdump -i ens3 src 172.16.0.20 and port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
08:56:50.489551 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 1817516973, win 62720, options [mss 8960,sackOK,TS val 3736091466 ecr 0,nop,wscale 7], length
0
08:56:50.491813 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 372277210, win 490, options [nop,nop,TS val 3736091468 ecr 671349263], length 0
08:56:50.491849 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736091468 ecr 671349263], length 75: HTTP: GET / HTTP/1.1
08:56:50.492311 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.492327 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.493247 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736091469 ecr 671349265], length 0
08:56:50.493667 IP 172.16.0.20.23408 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736091470 ecr 671349267], length 0
08:57:09.284597 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 3049648180, win 62720, options [mss 8960,sackOK,TS val 3736110261 ecr 0,nop,wscale 7], length
0
08:57:09.285040 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 728296369, win 490, options [nop,nop,TS val 3736110261 ecr 671368058], length 0
08:57:09.285102 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736110261 ecr 671368058], length 75: HTTP: GET / HTTP/1.1
08:57:09.285534 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.285641 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.285855 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736110262 ecr 671368059], length 0
08:57:09.286211 IP 172.16.0.20.22060 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736110263 ecr 671368059], length 0
08:57:10.964151 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 1863177286, win 62720, options [mss 8960,sackOK,TS val 3736111940 ecr 0,nop,wscale 7], length
0
08:57:10.964645 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 464215720, win 490, options [nop,nop,TS val 3736111941 ecr 671369737], length 0
08:57:10.964694 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736111941 ecr 671369737], length 75: HTTP: GET / HTTP/1.1
08:57:10.965157 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.965243 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.965612 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736111942 ecr 671369738], length 0
08:57:10.966035 IP 172.16.0.20.47275 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736111942 ecr 671369739], length 0
08:58:23.664041 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 407465876, win 62720, options [mss 8960,sackOK,TS val 3736184640 ecr 0,nop,wscale 7], length 0
08:58:23.664613 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 3047424129, win 490, options [nop,nop,TS val 3736184641 ecr 671442437], length 0
08:58:23.664620 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736184641 ecr 671442437], length 75: HTTP: GET / HTTP/1.1
08:58:23.665349 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736184641 ecr 671442438], length 0
08:58:23.666335 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736184642 ecr 671442438], length 0
08:58:23.666344 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736184642 ecr 671442438], length 0
08:58:23.666816 IP 172.16.0.20.16336 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736184643 ecr 671442440], length 0
08:58:24.529970 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 3610633804, win 62720, options [mss 8960,sackOK,TS val 3736185506 ecr 0,nop,wscale 7], length
0
08:58:24.530465 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 575035162, win 490, options [nop,nop,TS val 3736185507 ecr 671443303], length 0
08:58:24.530545 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736185507 ecr 671443303], length 75: HTTP: GET / HTTP/1.1
08:58:24.531020 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736185507 ecr 671443304], length 0
08:58:24.531079 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736185507 ecr 671443304], length 0
08:58:24.531619 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736185508 ecr 671443304], length 0
08:58:24.532042 IP 172.16.0.20.27720 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736185508 ecr 671443305], length 0
08:58:25.562872 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [S], seq 3415948315, win 62720, options [mss 8960,sackOK,TS val 3736186539 ecr 0,nop,wscale 7], length
0
08:58:25.563301 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 2043572435, win 490, options [nop,nop,TS val 3736186540 ecr 671444336], length 0
08:58:25.563424 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [P.], seq 0:75, ack 1, win 490, options [nop,nop,TS val 3736186540 ecr 671444336], length 75: HTTP: GET / HTTP/1.1
08:58:25.563812 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 239, win 489, options [nop,nop,TS val 3736186540 ecr 671444337], length 0
08:58:25.563960 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 873, win 485, options [nop,nop,TS val 3736186540 ecr 671444337], length 0
08:58:25.564877 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [F.], seq 75, ack 873, win 485, options [nop,nop,TS val 3736186541 ecr 671444337], length 0
08:58:25.565259 IP 172.16.0.20.33101 > ih-instance-vcn-b.ihprivatesubnet.spokevcnb.oraclevcn.com.http: Flags [.], ack 874, win 485, options [nop,nop,TS val 3736186542 ecr 671444338], length 0

The first output section of the curl request that was made.
The second output section of the curl request that was made.
The third output section of the curl request that was made.
The source DNS/IP.
The destination DNS/IP.
The port.
The TCP Flags.

This output indicates that HTTP traffic was received from the Hub FW IP.

Other scenario drawings

Below you will find the diagrams for the other scenarios. The way of collecting the logs, packet captures and TCPdumps will be the same.

SCENARIO 02: SPOKE INSTANCE > HUB INSTANCE

The diagram

SCENARIO 03: HUB INSTANCE > SPOKE INSTANCE

The diagram

SCENARIO 04: SPOKE INSTANCE > INTERNET

The diagram

SCENARIO 05: SPOKE INSTANCE > OCI SERVICE NETWORK

The diagram

SCENARIO 06: HUB > INTERNET

The diagram

SCENARIO 07: INTERNET > HUB

The diagram

SCENARIO 08: ON-PREM > SPOKE INSTANCE

The diagram

SCENARIO 09: SPOKE INSTANCE > ON-PREM

The diagram

SCENARIO 10: EXTERNAL > LB > SPOKE INSTANCE

The diagram

Overall Conclusion

In this tutorial, we looked at a method of how to follow a network packet inside an OCI Tenant environment with a Hub and Spoke Network Routing Architecture. We used a combination of Packet captures, TCPdumps, and subnet-level logging inside the VCNs. After collecting all the log information we performed some analytics on the data and this way we were able to follow the packet and the full path. This method is useful to verify if there is an issue with your end-to-end connection. This way you can verify on what hop a possible issue occurs.

Follow the packet inside an OCI Hub and Spoke VCN Routing architecture

SCENARIO 01: SPOKE INSTANCE > SPOKE INSTANCE

The diagram

The Steps

STEP 01: Note down the time

STEP 02: Open the first SSH session on Instance A

STEP 03: Open a second SSH session on Instance A

STEP 04: Open an SSH session on Instance B

STEP 05: Open a WEB session on the pfSense Firewall (navigate to the packet capture page)

STEP 06: Enable logging (ALL LOGS) on the Spoke A Private Subnet

STEP 07: Enable logging (ALL LOGS) on the Spoke B Private Subnet

STEP 08: Enable logging (ALL LOGS) on the Hub Private Subnet

STEP 09: On the second SSH session on Instance A start the TCP dump session

STEP 10: On the SSH session on Instance B start the TCP dump session

STEP 11: On the pfSense Firewall start the packet capture.

STEP 12: On the first SSH session on Instance A do a CURL to the Instance B web server

STEP 13: Look at all the logging and packet captures (and follow the path)

Stop all logging, packet captures, and TCP dumps

The diagram (with logging points)

Logging Point A (on diagram) - (TCP Dump on Instance A)

Logging Point B (on diagram) - (Logging on the Spoke VCN A Private Subnet)

Logging Point C (on diagram) - (Logging on the Hub VCN Private Subnet)

Logging Point D (on diagram) - (Packet capture on the pfSense Firewall)

Logging Point E (on diagram) - (Logging on the Spoke VCN B Private Subnet)

Logging Point F (on diagram) * (TCP Dump on Instance B)

Other scenario drawings

SCENARIO 02: SPOKE INSTANCE > HUB INSTANCE

The diagram

SCENARIO 03: HUB INSTANCE > SPOKE INSTANCE

The diagram

SCENARIO 04: SPOKE INSTANCE > INTERNET

The diagram

SCENARIO 05: SPOKE INSTANCE > OCI SERVICE NETWORK

The diagram

SCENARIO 06: HUB > INTERNET

The diagram

SCENARIO 07: INTERNET > HUB

The diagram

SCENARIO 08: ON-PREM > SPOKE INSTANCE

The diagram

SCENARIO 09: SPOKE INSTANCE > ON-PREM

The diagram

SCENARIO 10: EXTERNAL > LB > SPOKE INSTANCE

The diagram

Overall Conclusion