Terraform: NSX-T infra segments

Now that we prepared the environment to start our Terraform coding it is time to actually start some Terraform coding. For our automated nested lab we need some network segments that the infra that we are building will consume.

The Segments we need to create with the subnets are listed below:

• LX-APP-MGMT-11 | 192.168.11.0/24
  • Subnet for Management applications, like vCenter Server, NSX-T Manager, etc.
• LX-ESXI-MGMT-UT | 192.168.12.0/24
  • Subnet for all vmk0 interfaces for ESXi management traffic.
• LX-VMOTION-13 | 192.168.13.0/24
  • Subnet for all vmk1 interfaces for ESXi vMotion traffic.
• LX-VSAN-14 | 192.168.14.0/24
  • Subnet for all vmk2 interfaces for ESXi VSAN traffic.
• LX-REPLICATION-15 | 192.168.15.0/24
  • Subnet for all vmk3 interfaces for ESXi Replication traffic.
• LX-OVERLAY-HOST-16 | 192.168.16.0/24
  • Subnet for all Host (ESXi) Transport Node TEP interfaces for GENEVE traffic.
• LX-OVERLAY-EDGE-17 | 192.168.17.0/24
  • Subnet for all Edge (VM) Transport Node TEP interfaces for GENEVE traffic.
• LX-BGP-UPLINK-01-18 | 192.168.18.0/24
  • Subnet for BGP uplink 01 for N/S Connectivity.
• LX-BGP-UPLINK-02-19 | 192.168.19.0/24
  • Subnet for BGP uplink 02 for N/S Connectivity.
• LX-TRANSIT-LEAF-SPINE-20 | 192.168.20.0/24
  • Subnet to connect the virtual ToR switches/leafs (VX01 and VX02) with the core/spine (VX00).

Note

Where the “X” is used we need to replace this with a number that will reflect our lab number.

The terraform script in order to deploy these segments on my NSX-T infrastructure NSX-T Manager can be found below:

❯ tree
├── main.tf

main.tf

CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (main.tf code) ===> :

root # use Terraform 0.12 to execute script

root # NSX-T Manager Credentials
provider "nsxt" {
    host                     = “<your NSX-T manager IP/Hostname>”
    username                 = “admin”
    password                 = “<your NSX-T manager password>”
    allow_unverified_ssl     = true
    max_retries              = 10
    retry_min_delay          = 500
    retry_max_delay          = 5000
    retry_on_status_codes    = [429]
}

data “nsxt_policy_transport_zone” “overlay_tz” {
    display_name = “TZ-OVERLAY”
}

resource “nsxt_policy_segment” “tf_app_mgmt” {
    display_name = “LX-APP-MGMT-11”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_esxi_mgmt” {
    display_name = “LX-ESXI-MGMT-UT”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_vmotion” {
    display_name = “LX-VMOTION-13”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_vsan” {
    display_name = “LX-VSAN-14”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource "nsxt_policy_segment” “tf_replication” {
    display_name = “LX-REPLICATION-15”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_overlay_host” {
    display_name = “LX-OVERLAY-HOST-16”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_overlay_edge” {
    display_name = “LX-OVERLAY-EDGE-17”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_uplink_01” {
    display_name = “LX-BGP-UPLINK-01-18”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_uplink_02” {
    display_name = “LX-BGP-UPLINK-02-19”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

resource “nsxt_policy_segment” “tf_transit_leaf_spine” {
    display_name = “LX-TRANSIT-LEAF-SPINE-20”
    description = “Segment created by Terraform”
    transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path
}

Validate your code:

ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24
[INFO] Switching to v0.12.24
[INFO] Switching completed
ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate

Plan your code:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan

Execute your code to implement the Segments:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply

When the segments need to be removed again you can revert the implementation:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy

Before the above script can be executed (as the time of this writing: March 2020) we need to compile the latest NSX-T Terraform Provider from source. The NSX-T Terraform Provider that is downloaded automatically is a version that does not contain all features and the policy management features are not fully available and in beta at the moment.

In order to create the segments with terraform in NSX-T in the Simple UI the default nsx-t terraform provider is not supported. With the default you can only create segments in the advanced UI (that is not visible in the Simple UI). In order to get the latest terraform NSX-T provider you need to compile this yourself.

Using the latest version of the NSX–T provider

By default when using the NSX-T provider in your Terraform script the “Default” provider will be downloaded and used. This “default” version is the latest and stable version of the provider, but not the one with the latest features. There are a lot more features and commands under development and these can only be used with the latest version of the NSX-T provider. In order to get this new NSX-T provider with the latest and greatest features we need to clone the GitHub repo of the provider and compile it from source using Go Lang.

So the first step is to install Go Lang on my Mac. We do this using brew. How brew works can be found here.

ihoogendoor-a01:~ iwanhoogendoorn$ # Go development

ihoogendoor-a01:~ iwanhoogendoorn$ export GOPATH=“${HOME}/.go”

ihoogendoor-a01:~ iwanhoogendoorn$ export GOROOT=“$(brew —prefix golang)/libexec”

ihoogendoor-a01:~ iwanhoogendoorn$ export PATH=“$PATH:${GOPATH}/bin:${GOROOT}/bin”

ihoogendoor-a01:~ iwanhoogendoorn$ test -d “${GOPATH}”

ihoogendoor-a01:~ iwanhoogendoorn$ brew install go
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
azcopy                      container-structure-test    dhall-yaml                  mtoc                        publish                     swift-format
==> Updated Formulae
appium                      dvc                         gsoap                       libvirt                     run                         tiny-fugue
arp-sk                      ettercap                    hugo                        minio                       selenium-server-standalone  tomee-plume
arping                      exploitdb                   inspircd                    minio-mc                    skaffold                    tomee-plus
arpoison                    fop                         legit                       miniserve                   spades                      tomee-webprofile
balena-cli                  frege                       libarchive                  nanopb-generator            sslsplit                    travis
bash-completion@2           gitlab-gem                  libcbor                     nativefier                  suricata                    vice
calicoctl                   gitversion                  libfido2                    poppler                     tccutil                     wpscan
docbook                     glooctl                     libnet                      ripgrep                     tcptraceroute               wtfutil
dpkg                        golang-migrate              libnids                     root                        tflint                      zeek
==> Updated Casks
1password-cli                     fontforge                         lazarus                           qownnotes                         unity-android-support-for-editor
angry-ip-scanner                  fontgoggles                       lrtimelapse                       rectangle                         unity-download-assistant
anydo                             frhelper                          maccy                             refined-github-safari             unity-ios-support-for-editor
beaker-browser                    gifox                             macgamestore                      ripcord                           unity-lumin-support-for-editor
chromium                          glyphs                            metaz                             supercollider                     unity-webgl-support-for-editor
clashx                            hbuilderx                         meteorologist                     texworks                          unity-windows-support-for-editor
electron-fiddle                   icq                               mochi                             thonny                            webcatalog
engine-prime                      jami                              multitouch                        tinderbox
espresso                          jprofiler                         odrive                            trilium-notes
exodus                            kapitainsky-rclone-browser        pd-l2ork                          unity
==> Deleted Casks
google-nik-collection

==> Downloading https»//homebrew.bintray.com/bottles/go-1.14.catalina.bottle.tar.gz 
==> Downloading from https»//akamai.bintray.com/a4/a4e81b1025b77af039a34888545ce0f4a3a332b1ed6358ca8107e878d25e46aa?__gda__=exp=1584369109~hmac=e7bb7a5cce083ba4b6c323ef9 
root ######################################################################## 100.0%
==> Pouring go-1.14.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/go/1.14: 9,431 files, 424MB
ihoogendoor-a01:~ iwanhoogendoorn$

Now that Go Lang is installed we can clone the new NSX-Tprovider code:

ihoogendoor-a01:~ iwanhoogendoorn$ cd Coding/nestedlabs/Terraform/

ihoogendoor-a01:Terraform iwanhoogendoorn$ mkdir terratemp

ihoogendoor-a01:Terraform iwanhoogendoorn$ cd terratemp/

ihoogendoor-a01:terratemp iwanhoogendoorn$ pwd
/Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/terratemp

ihoogendoor-a01:terratemp iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git 
Cloning into ‘terraform-provider-nsxt’…
remote: Enumerating objects: 86, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 15107 (delta 46), reused 51 (delta 34), pack-reused 15021
Receiving objects: 100% (15107/15107), 16.23 MiB

Now that the code is downloaded we need to compile it from source:

CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (compile provider from source output) ===> :

ihoogendoor-a01:terratemp iwanhoogendoorn$ ls -l
total 0
drwxr-xr-x  20 iwanhoogendoorn  staff  640 Mar 16 15:04 terraform-provider-nsxt
ihoogendoor-a01:terratemp iwanhoogendoorn$ cd terraform-provider-nsxt/
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls
CHANGELOG.md    GNUmakefile    NOTICE.txt    go.mod        main.go        scripts        vendor
CONTRIBUTING.md    LICENSE.txt    README.md    go.sum        nsxt        tools        website

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go build
go: downloading github.com/hashicorp/terraform-plugin-sdk v1.1.0 
go: downloading github.com/google/uuid v1.1.1 
go: downloading github.com/vmware/go-vmware-nsxt v0.0.0-20191219213550-f4221331f638 
go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.2.0 
go: downloading github.com/vmware/vsphere-automation-sdk-go/runtime v0.1.1 
go: downloading github.com/vmware/vsphere-automation-sdk-go/lib v0.1.1 
go: downloading github.com/hashicorp/go-multierror v1.0.0 
go: downloading github.com/zclconf/go-cty v1.1.0 
go: downloading github.com/beevik/etree v1.1.0 
go: downloading github.com/hashicorp/go-hclog v0.9.2 
go: downloading github.com/gorilla/mux v1.7.3 
go: downloading golang.org/x/text v0.3.2 
go: downloading github.com/mitchellh/mapstructure v1.1.2 
go: downloading github.com/agext/levenshtein v1.2.2 
go: downloading google.golang.org/grpc v1.23.0 
go: downloading github.com/vmihailenco/msgpack v3.3.3+incompatible 
go: downloading github.com/mitchellh/reflectwalk v1.0.1 
go: downloading github.com/hashicorp/go-version v1.2.0 
go: downloading github.com/spf13/afero v1.2.2 
go: downloading github.com/hashicorp/hcl2 v0.0.0-20190821123243-0c888d1241f6 
go: downloading github.com/hashicorp/logutils v1.0.0 
go: downloading github.com/mitchellh/copystructure v1.0.0 
go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 
go: downloading github.com/hashicorp/go-getter v1.4.0 
go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db 
go: downloading golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586 
go: downloading github.com/davecgh/go-spew v1.1.1 
go: downloading github.com/apparentlymart/go-textseg v1.0.0 
go: downloading github.com/hashicorp/go-plugin v1.0.1 
go: downloading google.golang.org/api v0.9.0 
go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d 
go: downloading github.com/aws/aws-sdk-go v1.19.39 
go: downloading golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa 
go: downloading github.com/mitchellh/go-wordwrap v1.0.0 
go: downloading cloud.google.com/go v0.45.1 
go: downloading github.com/google/go-cmp v0.3.0 
go: downloading github.com/hashicorp/hil v0.0.0-20190212112733-ab17b08d6590 
go: downloading github.com/hashicorp/errwrap v1.0.0 
go: downloading github.com/mitchellh/cli v1.0.0 
go: downloading github.com/gibson042/canonicaljson-go v1.0.3 
go: downloading github.com/oklog/run v1.0.0 
go: downloading github.com/armon/go-radix v1.0.0 
go: downloading github.com/mitchellh/go-homedir v1.1.0 
go: downloading github.com/ulikunitz/xz v0.5.5 
go: downloading github.com/hashicorp/go-cleanhttp v0.5.1 
go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 
go: downloading github.com/hashicorp/go-uuid v1.0.1 
go: downloading github.com/golang/protobuf v1.3.2 
go: downloading github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d 
go: downloading github.com/hashicorp/terraform-config-inspect v0.0.0-20190821133035-82a99dc22ef4 
go: downloading github.com/posener/complete v1.2.1 
go: downloading github.com/hashicorp/go-safetemp v1.0.0 
go: downloading github.com/bgentry/speakeasy v0.1.0 
go: downloading github.com/hashicorp/hcl v0.0.0-20170504190234-a4b07c25de5f 
go: downloading github.com/mattn/go-isatty v0.0.4 
go: downloading github.com/googleapis/gax-go/v2 v2.0.5 
go: downloading github.com/zclconf/go-cty-yaml v1.0.1 
go: downloading github.com/fatih/color v1.7.0 
go: downloading github.com/mitchellh/go-testing-interface v1.0.0 
go: downloading github.com/mattn/go-colorable v0.0.9 
go: downloading go.opencensus.io v0.22.0
go: downloading github.com/apparentlymart/go-cidr v1.0.1 
go: downloading github.com/hashicorp/golang-lru v0.5.1 
go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af

When we look in the directory we see the following file “terraform-provider-nsxt":

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls -l
total 88824
-rw-r--r--    1 iwanhoogendoorn  staff      8857 Mar 16 15:04 CHANGELOG.md
-rw-r—r—    1 iwanhoogendoorn  staff      2422 Mar 16 15:04 CONTRIBUTING.md
-rw-r—r—    1 iwanhoogendoorn  staff      1839 Mar 16 15:04 GNUmakefile
-rw-r—r—    1 iwanhoogendoorn  staff     15248 Mar 16 15:04 LICENSE.txt
-rw-r—r—    1 iwanhoogendoorn  staff       485 Mar 16 15:04 NOTICE.txt
-rw-r—r—    1 iwanhoogendoorn  staff      7322 Mar 16 15:04 README.md
-rw-r—r—    1 iwanhoogendoorn  staff       456 Mar 16 15:23 go.mod
-rw-r—r—    1 iwanhoogendoorn  staff     28565 Mar 16 15:04 go.sum
-rw-r—r—    1 iwanhoogendoorn  staff       427 Mar 16 15:04 main.go
drwxr-xr-x  248 iwanhoogendoorn  staff      7936 Mar 16 15:04 nsxt
drwxr-xr-x    6 iwanhoogendoorn  staff       192 Mar 16 15:04 scripts
-rwxr-xr-x    1 iwanhoogendoorn  staff  45391620 Mar 16 15:23 terraform-provider-nsxt <-- new file
drwxr-xr-x   10 iwanhoogendoorn  staff       320 Mar 16 15:04 tools
drwxr-xr-x    8 iwanhoogendoorn  staff       256 Mar 16 15:04 vendor
drwxr-xr-x    4 iwanhoogendoorn  staff       128 Mar 16 15:04 website
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$

We need to copy this file in our working directory where our Terraform script is in that we want to execute for deployment.

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test/terraform-provider-nsxt

ihoogendoor-a01:Test iwanhoogendoorn$ cd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test

I needed to remove the cached provided, because I executed the "terraform init" before where it used the downloaded “default” NSX-T provider. I want to use the”local” and “just compiled” provider so I need to remove the .terraform directory.

ihoogendoor-a01:Test iwanhoogendoorn$ ls -lah
total 88664
drwxr-xr-x   5 iwanhoogendoorn  staff   160B Mar 16 15:36 .
drwxr-xr-x  15 iwanhoogendoorn  staff   480B Mar 16 15:43 ..
drwxr-xr-x   3 iwanhoogendoorn  staff    96B Mar 14 22:59 .terraform
-rw-r—r—   1 iwanhoogendoorn  staff   838B Mar 16 15:28 networks.tf
-rwxr-xr-x   1 iwanhoogendoorn  staff    43M Mar 16 15:26 terraform-provider-nsxt
ihoogendoor-a01:Test iwanhoogendoorn$ rm -R .terraform/

Now lets run the “init” command:

ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24
[INFO] Switching to v0.12.24
[INFO] Switching completed
ihoogendoor-a01:Test iwanhoogendoorn$ terraform init

Initializing the backend...

Initializing provider plugins…

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialise your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

And when we check the version we see the “unversioned” remark, and this tells us we are using our NSX-T provider we have just compiled.

ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24
[INFO] Switching to v0.12.24
[INFO] Switching completed
ihoogendoor-a01:Test iwanhoogendoorn$ terraform version
Terraform v0.12.24
+ provider.nsxt (unversioned)

So we are ready to use the “newest" version of the NSX-T provider that is required to deploy our NSX-T segments.

Validate your code:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate

Plan your code:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan

Execute your code to implement the Segments:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply

When the segments need to be removed again you can revert the implementation:

ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy

The terraform provider is currently under development. If you download and compile the latest version (that is still under development) it can be that old functionality that was working first, is now broken. To go back to a specific version with specific commits you can do this by following the below procedure (Thank you Rutger Blom for this):

Browse to a new fresh directory:

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cd terraform-provider-nsxt-v0/

The directory is really empty:

ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls

Clone the full repository:

ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git
Cloning into 'terraform-provider-nsxt'...
remote: Enumerating objects: 142, done.
remote: Counting objects: 100% (142/142), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 15163 (delta 84), reused 78 (delta 54), pack-reused 15021
Receiving objects: 100% (15163/15163), 16.25 MiB  8.33 MiB/s, done.
Resolving deltas: 100% (8351/8351), done.
Checking out files: 100% (4088/4088), done.

Now lets browse into the directory:

ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls
terraform-provider-nsxt
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ cd terraform-provider-nsxt
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls
CHANGELOG.md	GNUmakefile	NOTICE.txt	go.mod		main.go		scripts		vendor
CONTRIBUTING.md	LICENSE.txt	README.md	go.sum		nsxt		tools		website

Fetch a specific version + commits:

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git fetch origin c01d221ceb98e114aadd84126707633d25fcb43b
From https://github.com/terraform-providers/terraform-provider-nsxt
 * branch            c01d221ceb98e114aadd84126707633d25fcb43b -> 
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git reset --hard 
 is now at c01d221 Policy block subnet - wait for delete realization

Compile the specific version :

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go build
go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.1.1
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$

'Copy the new provider to the repository directory of your NSX-T Terraform code:

ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/#create-T1

Sources

• Source 1
• Source 2
• Source 3
• Source 4
• Source 5
• Source 6