View source for VMware Cloud Director Basics with NSX-T

This article will explain how you can integrate NSX-T into VMware Cloud Director to provide Network and Security capabilities across different tenants.
To integrate NSX-T with VMware Cloud Director, you also need to know a bit about VMware Cloud Director and how it is installed, configured, and typical use-cases.

=Software Versions used in this article=

{| class="wikitable"
|-
! Software !! Version
|-
| vCenter Server  || 7.0.1 (17005016)
|-
| ESXi || 7.0.1 (16850804)
|-
| NSX-T || 3.1
|-
|Cloud Director || 10.2
|}

=Configuration Steps=

* STEP 1) NSX-T configuration
* STEP 2) Add vCenter Server Instances to VMware Cloud Director
* STEP 3) Add NSX-T Manager to VMware Cloud Director
* STEP 4) Create Network Pools
* STEP 5) Create External Networks (per tenant)
* STEP 6) Create Provider VDCs
* STEP 7) Create Organization and Organization VDCs (per tenant)
* STEP 8) Create Edge Gateways (per tenant)
* STEP 9) Create an Organization Virtual Data Center Network
* STEP 10) Share Edge Gateways between Organization VDCs (of the same tenant)
* STEP 11) Verify the Data Center Group Networks
* STEP 12) Create Provider VDCs VM Placement Policies
* STEP 13) Create several VMs that will simulate an application
* STEP 14) Configure Distributed Firewall Rules on the applications

=vSphere Deployment specifications=

In this article, I will be working with a stretched vSphere (METRO) cluster, where I will have two Data Center Sites where all the vSphere Clusters are stretched across these sites.
One vCenter Server will "manage" the management hosts (in the management vSphere Cluster) of the first and second Data Center.
Another vCenter Server will "manage" the compute hosts (in the compute vSphere Cluster) of the first and second Data Center.

Because NSX-T will also be installed on the Management Cluster, we don't want this to interfere or impact the compute cluster in case of errors/mistakes.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-vSphere-Design.png|600px]]</span>

=NSX-T Deployment specifications=

Two NSX-T Manager Clusters will be deployed.
One NSX-T Manager will provide network and security services to the management vSphere Cluster, and the other one will provide network and security services to the compute vSphere Cluster.

In this article, we will only work with the NSX-T environment that is working together with the compute cluster.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-NSX Manager-Design.png|600px]]</span>

=VMware Cloud Director deployment=

The deployment of the VMware Cloud Director appliance is out of scope for this article.
There are plenty of good articles that will show you how to deploy the VMware Cloud Director appliance like [https://captainvops.com/2020/04/09/deploying-vcloud-director-v10-appliance/ this one] or [https://www.vxpert.in/vmware-vcloud-director-standalone-installation/ this one]. 

=STEP 1» NSX-T configuration=

The deployment of NSX-T is also out of this article's scope, but it is important to know how to configure the VRF enabled Tier-0 Gateways.
There are plenty of good articles that will show you how to deploy NSX-T like [https://vmarena.com/how-to-install-nsx-t-3-0-data-center-appliance-step-by-step/ this one].

==Configure a Tier&ndash;0 Gateway with two VRF enabled Gateways==

I have written an article on how to deploy VRF enabled Tier-0 Gateways [[Configure NSX-T VRF Lite|here]].

You cn see my configured VRF enabled Tier-0 Gateways here:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:T0-VRF-BGP-01.png|800px]]</span>

My dedicated BGP neighbour configuration for Tenant-A:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:T0-VRF-BGP-02.png|800px]]</span>

My dedicated BGP neighbor configuration for Tenant-B:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:T0-VRF-BGP-03.png|800px]]</span>

On the "physical routing" side, the BGP neighbors are also active.
The yellow marked output below is dedicated to Tenant-A, and the green is dedicated to Tenant-B.

{{console|body=
vyos@Pod-100-Router:~$ show ip bgp summary

IPv4 Unicast Summary:
BGP router identifier 10.203.100.1, local AS number 65000 vrf-id 0
BGP table version 155
RIB entries 24, using 4416 bytes of memory
Peers 12, using 245 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
##y##10.203.11.2     4      65001    8027    8042        0    0    0 02:07:12            5
##y##10.203.11.3     4      65001    8028    8042        0    0    0 02:07:22            5
##y##10.203.12.2     4      65001    8027    8041        0    0    0 02:07:12            5
##y##10.203.12.3     4      65001    8028    8042        0    0    0 02:07:21            5
##g##10.203.13.2     4      65001    8013    8035        0    0    0 3d01h04m            0
##g##10.203.13.3     4      65001    8012    8034        0    0    0 3d01h38m            0
##g##10.203.14.2     4      65001    8012    8032        0    0    0 3d01h04m            0
##g##10.203.14.3     4      65001    8012    8031        0    0    0 3d01h38m            0
10.203.106.2    4      65001   27854   27860        0    0    0 3d01h04m            8
10.203.106.3    4      65001   27841   27857        0    0    0 3d01h38m            8
10.203.107.2    4      65001   27861   27870        0    0    0 3d01h04m            8
10.203.107.3    4      65001   27848   27866        0    0    0 3d01h38m            8

Total number of neighbors 12
vyos@Pod-100-Router:~$
}}

=VMware Cloud Director configuration=

Before we can use the NSX-T network and security services, we need to configure some basic things that will be explained below.

Log in to VMware Cloud Director:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:LOG-IN-VCD-1.png|300px]]</span>

This is the screen when you log in to a blank VMware Cloud Director:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:LOG-IN-VCD-2.png|800px]]</span>

=STEP 2{{fqm}} Add vCenter Server Instances to VMware Cloud Director=

Click on "Infrastructure Resources >> vCenter Server Instances " to add a vCenter Server:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-5.png|800px]]</span>

Add the vCenter Server details:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-1.png|800px]]</span>

As I don't have NSX-v I will disable this as I don't want to configure any NSX-v settings:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-2.png|800px]]</span>

I an not enabling "Tenant access" or want to "generate proxies":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-3.png|800px]]</span>

Review the summary screen:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-4.png|800px]]</span>

The vCenter Server is successfully added to Cloud Director:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-VC-6.png|800px]]</span>

=STEP 3{{fqm}} Add NSX&ndash;T Manager to VMware Cloud Director=

Click on "Infrastructure Resources >> NSX-T Managers" to add an NSX-T Manager:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NSXT-1.png|800px]]</span>

Add the NSX-T Manager details:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NSXT-2.png|400px]]</span>

The NSX-T Manager is successfully added to Cloud Director:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NSXT-3.png|800px]]</span>

=STEP 4{{fqm}} Create Network Pools=

To make sure that Cloud Director can make use of the overlay networks that are offered by NSX-T, you need to configure a Network Pool:

Go to "Resources >> Network Pools" to add a new Network Pool: 

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-1.png|800px]]</span>

Specify a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-2.png|800px]]</span>

Select "Geneve backed":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-3.png|800px]]</span>

Select the NSX-T Manager:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-4.png|800px]]</span>

Select the "Transport Zone":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-5.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-6.png|800px]]</span>

The Network Pool is successfully added to Cloud Director:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-NETWORK-POOLS-7.png|800px]]</span>

=STEP 5{{fqm}} Create External Networks 〈per tenant〉=
It would be best if you created an external network per tenant.
This network (pool) specifies the network (IP address Pool) that is going to be used between your Tier-0 (in my case, the VRF enabled Tier-0) and the (Tier-1) Tenant Edge (Gateways) that are going to be created. The "transit" network between the Tier-0 and Tier-1 Gateways.

Go to "Cloud Resources >> External Networks" to create an external network:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-1.png|800px]]</span>

Select "NSX-T Resources":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-2.png|800px]]</span>

As this is an external network for my Tenant-A give it a name related to Tenant-A:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-3.png|800px]]</span>

Select the Tenant-A VRF enabled Tier-0 Gateway:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-4.png|800px]]</span>

Specify a "gateway" IP address that will be configured on the downlink interface of the VRF enabled Tier-0 Gateway for Tenant-A:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-5.png|800px]]</span>

Specify a network pool, for the amount of the (Tier-1) Tenant Edge (Gateways)you are planning to add:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-6.png|400px]]</span>

Verify if the pool is configured correctly:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-7.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-8.png|800px]]</span>

The External Network for Tenant-A is successfully created:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-9.png|800px]]</span>

Also, create an External Network for Tenant-B with like I did below:

Select "NSX-T Resources":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-10.png|800px]]</span>

As this is an external network for my Tenant-B give it a name related to Tenant-B:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-11.png|800px]]</span>

Select the Tenant-B VRF enabled Tier-0 Gateway:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-12.png|800px]]</span>

Specify a "gateway" IP address that will be configured on the downlink interface of the VRF enabled Tier-0 Gateway for Tenant-B:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-13.png|800px]]</span>

Specify a network pool, for the amount of the (Tier-1) Tenant Edge (Gateways)you are planning to add:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-14.png|400px]]</span>

Verify if the pool is configured correctly:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-15.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-16.png|800px]]</span>

The External Network for Tenant-B is successfully created:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-EXTERNAL-NETWORKS-17.png|800px]]</span>

=STEP 6{{fqm}} Create Provider VDCs=
It would be best if you created a Provider VCD.

Go to "Cloud Resources >> Provider VDCs" to add a new Provider VCD:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-1.png|800px]]</span>

Specify a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-2.png|800px]]</span>

Select a vCenter Server:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-3.png|800px]]</span>

Select a Resource Pool (vSphere Cluster in my case):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-4.png|800px]]</span>

Select the Storage Policy:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-5.png|800px]]</span>

Select the NSX-T Manager and Network Pool:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-6.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-7.png|800px]]</span>

The Provider VDC is now successfully added:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-PVDC-8.png|800px]]</span>

=STEP 7{{fqm}} Create Organization and Organization VDCs 〈per tenant〉=
You now need to Organization and Organization VDCs for each Tenant.
I will create two Organizations one for each Tenant.
Inside the Organization I will create multiple Organization VDCs per tenant:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-VCD-overview.png|600px]]</span>

Go to "Cloud Resources >> Organizations" to add a new Organization:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-1.png|800px]]</span>

Specify the Organization name for Tenant-A:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-2.png|800px]]</span>

Review the Organization creation:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-3.png|800px]]</span>

Specify the Organization name for Tenant-B:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-4.png|800px]]</span>

Review the Organization creation:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-5.png|800px]]</span>

Go to "Cloud Resources >> Organization VDCs" to add a new Organization VDC:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-6.png|800px]]</span>

Specify a name for the first Organization VDC for Tenant-A.
This Organization VDC will be bound to a vSphere STRETCHED cluster.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-7.png|800px]]</span>

Select "Tenant-A" as the Organization:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-8.png|800px]]</span>

Select the Provider VDC:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-9.png|800px]]</span>

Select an allocation model (I am using Pay-as-you-go):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-10.png|800px]]</span>

Specify the Pay-as-you-go details:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-11.png|800px]]</span>

Select the Storage Policy and I enabled thin provisioning (as this is a nested lab):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-12.png|800px]]</span>

Select a network pool:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-13.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-14.png|800px]]</span>

The Organization VDC is now created:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-15.png|800px]]</span>

Specify a name for the second Organization VDC for Tenant-B. This Organization VDC will be bound to a vSphere STRETCHED cluster.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-16.png|800px]]</span>

Select "Tenant-B" as the Organization:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-17.png|800px]]</span>

Select the Provider VDC:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-18.png|800px]]</span>

Select an allocation model (I am using Pay-as-you-go):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-19.png|800px]]</span>

Specify the Pay-as-you-go details:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-20.png|800px]]</span>

Select the Storage Policy and I enabled thin provisioning (as this is a nested lab):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-21.png|800px]]</span>

Select a network pool:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-22.png|800px]]</span>

Review the summary (screenshot is not provided).

The Organization VDC is now created:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-23.png|800px]]</span>

OPTIONAL: Create "the other "local/non-stretched" Organization VDCs the same way I did above:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-OVDC-24.png|800px]]</span>

=STEP 8{{fqm}} Create Edge Gateways 〈per tenant〉=
With Cloud Director, you can also create the Tier-1 Gateways that is bound to an Organization VDC.
These are called "Edge Gateways" inside Cloud Director. 

These Edge Gateways will be connected to the VRF enabled Tier-0 Gateways like I have illustrated on the figure below:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-VCD-overview-NSX-T.png|600px]]</span>

Go to "Cloud Resources >> Edge Gateways" to add a new Edge Gateway:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-1.png|800px]]</span>

Select the Organization VDC where the Edge Gateway (Tier-1 Gateway) will be dedicated to:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-2.png|800px]]</span>

Specify a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-3.png|800px]]</span>

Select the (external)network pool that was created that is going to use as a "transit" subnet between the VRF enabled Tier-0 Gateways and the new Edge Gateway (Tier-1 Gateway):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-4.png|800px]]</span>

When you have a dedicated Edge Cluster for the Tier-1 Gateways, you can choose the second option; in my case, I am using the same Edge Cluster that I specified in the External Network.

This means that my Tier-0 Gateway and its corresponding VRF Enabled Gateways will all be configured on one Edge Transport Node Cluster:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-5.png|800px]]</span>

Leave the allocated IP addresses default:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-6.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-7.png|800px]]</span>

The (first) Edge Gateway for Tenant-A is now created successfully:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-8.png|800px]]</span>

Create another Edge Gateway for Tenant-B,
Select the Organization VDC where the Edge Gateway (Tier-1 Gateway) will be dedicated to (This time dedicated to Tenant-B):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-9.png|800px]]</span>

Specify a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-10.png|800px]]</span>

Select the (external)network pool that was created that is going to use as a "transit" subnet between the VRF enabled Tier-0 Gateways and the new Edge Gateway (Tier-1 Gateway):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-11.png|800px]]</span>

Select the NSX-T Edge (Transport Node) Cluster:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-12.png|800px]]</span>

Leave the allocated IP addresses default:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-13.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-14.png|800px]]</span>

The (second) Edge Gateway for Tenant-B is now created successfully:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-15.png|800px]]</span>

When I take a sneak peek at the NSX-T Manager GUI, I can see that the Edge Gateways are created in the form of a Tier-1 Gateway on the NSX-T Manager:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-EDGE-GW-16.png|800px]]</span>

=STEP 9{{fqm}} Create an Organization Virtual Data Center Network=
Now I will create the Guest Networks that I want to use that will be connected to the Edge Gateways (Tier-1 Gateways).

Go to "Cloud Resources >> Organization VDCs."
Click on the arrow/link of the first Organization VDC to "impersonate" the Organisation Administrator for this specific tenant (ORGANIZATION-VDC-TENANT-A).

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-1.png|800px]]</span>

Now you are logged in as an Organisation Administrator for this specific tenant (ORGANIZATION-VDC-TENANT-A):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-2.png|800px]]</span>

Go to "Data Centers >> Networks >> Networks" to add new Guest VM Networks:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-3.png|800px]]</span>

Select the "Scope."
In my case, I am only creating a network (for now) that is bound to a specific VDC, and not shared across other VDCs (yet):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-4.png|800px]]</span>

Select "Routed" as we are attaching this network to the created Edge Gateway (Tier-1 Gateway) for Tenant-A that we created in the previous steps:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-5.png|800px]]</span>

Select the Edge Gateway (Tier-1 Gateway) for Tenant-A that we created in the previous steps:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-6.png|800px]]</span>

Specify the gateway address that will be configured on the Edge Gateways (Tier-1 Gateway) downlink.
This will be the gateway IP address for your Guest VMs that will be connected to this network:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-7.png|800px]]</span>

Specify a network pool for this network.
This is Cloud Directors' internal IPAM tool to assign IP addresses to the Guest Virtual Machines when they are created on the network you are creating now:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-8.png|800px]]</span>

Specify DNS details, if you have any:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-9.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-10.png|800px]]</span>

Repeat the previous steps to also create the APP and DB networks for Tenant-A (ORGANIZATION-VDC-TENANT-A):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-11.png|800px]]</span>

Repeat the previous steps to also create the WEB, APP and DB networks for Tenant-B (ORGANIZATION-VDC-TENANT-B):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-12.png|800px]]</span>

When I look at the NSX-T GUI, I can now see that the "linked segments" on the Tier-1 Gateways have been increased to 3.
this means that the networks we just created using Cloud Director have been added to NSX-T (as a Segment) and are attached to the Tier-1 Gateway:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-15.png|800px]]</span>

While being in "Organization Tenant mode, go to "Data Centers >> Virtual Datacenter" and click on another Organization VDC (ORGANIZATION-VDC-TENANT-A-DCA or ORGANIZATION-VDC-TENANT-A-DCB).

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-13.png|800px]]</span>

Go to "Data Centers >> Networking >> Networks," and review that this network section is empty as we have configured our networks in another Organization VDC (ORGANIZATION-VDC-TENANT-A-DCA or ORGANIZATION-VDC-TENANT-A-DCB).

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-TENANT-NET-14.png|800px]]</span>

=STEP 10{{fqm}} Share Edge Gateways between Organization VDCs 〈of the same tenant〉=
To "share" the networks that I just created with another Organization VDC (the one that was just empty), I need to create a Data Center Group.

The figure below illustrates how this will look like:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-Data-Center-Groups.png|600px]]</span>

==Create Data Center Groups==

===Configure Compute Provider Scope ===
Before a Data Center Group can be configured, I need to specify a "Compute Provider Scope."

Go (back) to the Cloud Director administration panel and go to "Infrastructure Resources >> vCenter Server Instances " to edit the vCenter Server:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-1.png|800px]]</span>

Configure the Compute Provider Scope:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-2.png|500px]]</span>

Review the Compute Provider Scope:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-3.png|800px]]</span>

===Create Data Center Groups===

Go back to the "Organization VDC administration" screen and go to "Networking >> Data Center Groups" to add a Data Center Group:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-4.png|800px]]</span>

Select the "primary" Organization VDC:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-5.png|800px]]</span>

Give the Data Center Group a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-6.png|800px]]</span>

Select the remaining Organization VDCs that you want to form a group with:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-7.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-8.png|800px]]</span>

The new Data Center Group is successfully created:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-9.png|800px]]</span>

==Assign an Edge Gateway to a Data Center Group==

Click on the Data Center group you have just created, and go to Edge Gateway.
Select "Add Edge":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-10.png|800px]]</span>

Select the Edge Gateway (Tier-1 Gateway) that is created before:

{{important|When you add the Edge Gateway to a Data Center Group, this means all the networks that were created on this Edge Gateway  in another Organization VDC are now automatically added to the other Organization VDCs that are part of this Data Center Group}}

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-11.png|800px]]</span>

The Edge Gateaway is added to the Data Center Group successfully (for Tenant-A):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-12.png|800px]]</span>

For Tenant-B you need to follow the same steps as above, but then you need to use the Tenant-B Organization VCDs and Edge Gateways with the below figure as a result:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CREATE-DC-GROUP-13.png|800px]]</span>

=STEP 11{{fqm}} Verify the Data Center Group Networks=
Now its time to see if the networks that were created in the "ORGANIZATION-VDC-TENANT-A" VDC are also added to the "ORGANIZATION-VDC-TENANT-A-DCA" and "ORGANIZATION-VDC-TENANT-A-DCB" VCD:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CHECK-DC-GROUP-NET-1.png|800px]]</span>

Review the "ORGANIZATION-VDC-TENANT-A" networks:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CHECK-DC-GROUP-NET-2.png|800px]]</span>

Review the "ORGANIZATION-VDC-TENANT-A-DCA" networks:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CHECK-DC-GROUP-NET-3.png|800px]]</span>

Review the "ORGANIZATION-VDC-TENANT-A-DCB" networks:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:CHECK-DC-GROUP-NET-4.png|800px]]</span>

=STEP 12{{fqm}} Create Provider VDCs VM Placement Policies=
When you look specifically at the "ORGANIZATION-VDC-TENANT-A" VDC, this is a VDC that is tied to a stretched vSphere Cluster.

When you want to place Virtual Machines on DC-A or DC-B, you can do two things:
# Use the "ORGANIZATION-VDC-TENANT-A-DCA" VDC (local to DC-A)
# Use the "ORGANIZATION-VDC-TENANT-A-DCB" VDC (local to DC-A)
# Use the "ORGANIZATION-VDC-TENANT-A" VDC (stretched between DC-A and DC-B) with the use of VM Placement Policies

How to achieve method #3, I will explain this in the section below.

This is also illustrated in the figure below:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:Drawings-Placement-Policies.png|600px]]</span>

First, log in to the vCenter Server and go to the VM/Host Groups vSphere Cluster settings:

==Create a Host Group 〈in vCenter Server〉==
Create 2 host groups, one for DC-A and the other for DC-B and place the hosts corresponding to the correct site in the correct group:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-1.png|800px]]</span>

==Create a VM Group 〈in vCenter Server〉==

Create 2 VM groups, one for DC-A and the other for DC-B and place the VMs corresponding to the correct site in the correct group:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-2.png|800px]]</span>

==Create a VM to Host Rule 〈in vCenter Server〉==

Create a VM/Host rule, one for DC-A and the other for DC-B, and make sure that you specify that the VMs in the VM Group for DC-A must run on the Hosts in Host Group for DC-A.
Do the same for DC-B.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-3.png|800px]]</span>

When you have done this, go to "Infrastructure Resources >> vCenter Server Instances " and click on "Refresh" to make sure Cloud Director detects the Groups and Rules.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-4.png|800px]]</span>

==Create VM Placement Policies 〈in the Provider VDC〉==

Go to "Cloud Resources >> Provider VDCs" to edit the Provider VCD:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-5.png|800px]]</span>

Go to "Cloud Resources >> Provider VDC >> PROVIDER-VDC >> Policies >> VM Placement" to add a new Placement Policy:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-6.png|800px]]</span>

Leave this default and click next:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-7.png|800px]]</span>

Specify a name:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-8.png|800px]]</span>

Select the "rule" you just created in vCenter Server corresponding to DC-A:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-9.png|800px]]</span>

Review the summary:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-10.png|800px]]</span>

The Placement Policy for DC-A is created successfully:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-11.png|800px]]</span>

Follow the same steps to create another Placement Policy for DC-B:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ADD-PLACEMENT-GROUP-NET-12.png|800px]]</span>

=STEP 13{{fqm}} Create several VMs that will simulate an application=
To test the networks you just added, you need to create Guest Virtual Machines.
I have done this in the next section:

==Create several VMs that will simulate an application 〈Web App and DB〉==

I have pre-created a 3-Tier vAPP that was based on an OVF.
This is out of scope for this article.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-1.png|800px]]</span>

When you click on the vApp, you will see the networks that this vAPP is using.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-2.png|800px]]</span>

Review the Network Diagram:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-3.png|800px]]</span>

Review the Virtual Machines that are in the vAPP and select the first VM (Web01):

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-4.png|800px]]</span>

Verify the available networks:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-5.png|800px]]</span>

Verify the NIC settings and select the created WEB network for the WEB01 Guest VM that is part of this vAPP.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-6.png|800px]]</span>

Verify the NIC settings:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-7.png|800px]]</span>

Make sure that VM Guest Customization is enabled.
This is needed to make sure the IP address is assigned from the custom Cloud Director Pool.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-8.png|800px]]</span>

When the VM is stopped, make sure you Power on the Guest VM using the "Force Recustomization" option.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:VAPP-NET-9.png|800px]]</span>

=STEP 14{{fqm}} Configure Distributed Firewall Rules on the applications=

You can configure the "Distributed Firewall" within this Data Center Group.

Go to the "Organization VDC administration" screen (ORGANIZATION-VDC-TENANT-A) and go to "Networking >> Data Center Groups >> General."

You will see that by default, the "Distributed Firewall" is not active.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ENABLE-SEGMENTATION-1.png|800px]]</span>

When you click on active, you activate this.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ENABLE-SEGMENTATION-2.png|800px]]</span>

Once you have activated the Distributed Firewall, you will see in the NSX-T Manager GUI that a new Security Policy with a new Distributed Firewall Rule is created dedicated to the Data Center Group.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:ENABLE-SEGMENTATION-3.png|800px]]</span>

=Lessons learned=

Wenn playing around with Cloud Director and NSX-T, a few things are worth mentioning here.

==Configure Route advertisement on the Edge Gateway 〈Tier&ndash;1 Gateway〉 through VMware Cloud Director==

There is a setting that you can enable "Dedicate External Network" on the Edge Gateway (Tier-1 Gateway) you create.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-ROUTE-CONFIG-1.png|400px]]</span>

When you enable this setting, you will configure "Routing Settings" on the Edge on the Edge Gateway (Tier-1 Gateway.

{{important|You can only enable this setting on ONE single Edge Gateway (Tier-1 Gateway. From a theoretical point of view, when network advertisement could be done on multiple Edge Gateways (Tier-1 Gateways), the same networks could be advertised, causing a routing conflict.}}

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-ROUTE-CONFIG-2.png|400px]]</span>

Once you enable "Dedicate External Network" you will see the additional routing sections:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-ROUTE-CONFIG-3.png|800px]]</span>

I have configured three networks (172.16.10.0/24, 172.16.20.0/24, and 172.16.30.0/24) to be advertised.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-ROUTE-CONFIG-4.png|800px]]</span>

We can verify this setting in the NSX-T Manager by looking at the corresponding Tier-1 Gateway routing settings.

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-ROUTE-CONFIG-5.png|800px]]</span>

An output of the routing table of the "external router" is provided below:

{{console|body=
B>* 172.16.10.0/24 [20/0] via 10.203.11.2, eth1.11, 00:01:49
  *                       via 10.203.11.3, eth1.11, 00:01:49
  *                       via 10.203.12.2, eth1.12, 00:01:49
  *                       via 10.203.12.3, eth1.12, 00:01:49
B>* 172.16.20.0/24 [20/0] via 10.203.11.2, eth1.11, 00:00:46
  *                       via 10.203.11.3, eth1.11, 00:00:46
  *                       via 10.203.12.2, eth1.12, 00:00:46
  *                       via 10.203.12.3, eth1.12, 00:00:46
B>* 172.16.30.0/24 [20/0] via 10.203.11.2, eth1.11, 00:00:46
  *                       via 10.203.11.3, eth1.11, 00:00:46
  *                       via 10.203.12.2, eth1.12, 00:00:46
  *                       via 10.203.12.3, eth1.12, 00:00:46
}}

==Configure Edge Gateway Firewall 〈Tier&ndash;1 Gateway〉 through VMware Cloud Director==

When you configure an Edge Gateway (Tier-1 Gateway) from Cloud Director, the default Gateway firewall setting is set to "DENY."

In the figure below I have changed this to "ALLOW":

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-FW-CONFIG-2.png|800px]]</span>

Verify if the firewall rule is now set to allow:

<span style="border:3px solid #f4c613;display: inline-block;">[[File:EDGE-FW-CONFIG-1.png|800px]]</span>

[[Category:NSX]]
[[Category:Networking]]
[[Category:VMware]]