Terraform: NSX-T infra segments
Now that we prepared the environment to start our Terraform coding it is time to actually start some Terraform coding. For our automated nested lab we need some network segments that the infra that we are building will consume.
The Segments we need to create with the subnets are listed below:
- LX-APP-MGMT-11 | 192.168.11.0/24
- Subnet for Management applications, like vCenter Server, NSX-T Manager, etc.
- LX-ESXI-MGMT-UT | 192.168.12.0/24
- Subnet for all vmk0 interfaces for ESXi management traffic.
- LX-VMOTION-13 | 192.168.13.0/24
- Subnet for all vmk1 interfaces for ESXi vMotion traffic.
- LX-VSAN-14 | 192.168.14.0/24
- Subnet for all vmk2 interfaces for ESXi VSAN traffic.
- LX-REPLICATION-15 | 192.168.15.0/24
- Subnet for all vmk3 interfaces for ESXi Replication traffic.
- LX-OVERLAY-HOST-16 | 192.168.16.0/24
- Subnet for all Host (ESXi) Transport Node TEP interfaces for GENEVE traffic.
- LX-OVERLAY-EDGE-17 | 192.168.17.0/24
- Subnet for all Edge (VM) Transport Node TEP interfaces for GENEVE traffic.
- LX-BGP-UPLINK-01-18 | 192.168.18.0/24
- Subnet for BGP uplink 01 for N/S Connectivity.
- LX-BGP-UPLINK-02-19 | 192.168.19.0/24
- Subnet for BGP uplink 02 for N/S Connectivity.
- LX-TRANSIT-LEAF-SPINE-20 | 192.168.20.0/24
- Subnet to connect the virtual ToR switches/leafs (VX01 and VX02) with the core/spine (VX00).
Where the “X” is used we need to replace this with a number that will reflect our lab number.
The terraform script in order to deploy these segments on my NSX-T infrastructure NSX-T Manager can be found below:
❯ tree ├── main.tf
main.tf
CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (main.tf code) ===> :
Validate your code:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate
Plan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan
Execute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply
When the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy
Before the above script can be executed (as the time of this writing: March 2020) we need to compile the latest NSX-T Terraform Provider from source. The NSX-T Terraform Provider that is downloaded automatically is a version that does not contain all features and the policy management features are not fully available and in beta at the moment.
In order to create the segments with terraform in NSX-T in the Simple UI the default nsx-t terraform provider is not supported. With the default you can only create segments in the advanced UI (that is not visible in the Simple UI). In order to get the latest terraform NSX-T provider you need to compile this yourself.
Using the latest version of the NSX–T provider
By default when using the NSX-T provider in your Terraform script the “Default” provider will be downloaded and used. This “default” version is the latest and stable version of the provider, but not the one with the latest features. There are a lot more features and commands under development and these can only be used with the latest version of the NSX-T provider. In order to get this new NSX-T provider with the latest and greatest features we need to clone the GitHub repo of the provider and compile it from source using Go Lang.
So the first step is to install Go Lang on my Mac. We do this using brew. How brew works can be found here.
ihoogendoor-a01:~ iwanhoogendoorn$ # Go development
ihoogendoor-a01:~ iwanhoogendoorn$ export GOPATH=“${HOME}/.go”
ihoogendoor-a01:~ iwanhoogendoorn$ export GOROOT=“$(brew —prefix golang)/libexec”
ihoogendoor-a01:~ iwanhoogendoorn$ export PATH=“$PATH:${GOPATH}/bin:${GOROOT}/bin”
ihoogendoor-a01:~ iwanhoogendoorn$ test -d “${GOPATH}”
ihoogendoor-a01:~ iwanhoogendoorn$ brew install go Updating Homebrew... ==> Auto-updated Homebrew! Updated 2 taps (homebrew/core and homebrew/cask). ==> New Formulae azcopy container-structure-test dhall-yaml mtoc publish swift-format ==> Updated Formulae appium dvc gsoap libvirt run tiny-fugue arp-sk ettercap hugo minio selenium-server-standalone tomee-plume arping exploitdb inspircd minio-mc skaffold tomee-plus arpoison fop legit miniserve spades tomee-webprofile balena-cli frege libarchive nanopb-generator sslsplit travis bash-completion@2 gitlab-gem libcbor nativefier suricata vice calicoctl gitversion libfido2 poppler tccutil wpscan docbook glooctl libnet ripgrep tcptraceroute wtfutil dpkg golang-migrate libnids root tflint zeek ==> Updated Casks 1password-cli fontforge lazarus qownnotes unity-android-support-for-editor angry-ip-scanner fontgoggles lrtimelapse rectangle unity-download-assistant anydo frhelper maccy refined-github-safari unity-ios-support-for-editor beaker-browser gifox macgamestore ripcord unity-lumin-support-for-editor chromium glyphs metaz supercollider unity-webgl-support-for-editor clashx hbuilderx meteorologist texworks unity-windows-support-for-editor electron-fiddle icq mochi thonny webcatalog engine-prime jami multitouch tinderbox espresso jprofiler odrive trilium-notes exodus kapitainsky-rclone-browser pd-l2ork unity ==> Deleted Casks google-nik-collection ==> Downloading https»//homebrew.bintray.com/bottles/go-1.14.catalina.bottle.tar.gz ==> Downloading from https»//akamai.bintray.com/a4/a4e81b1025b77af039a34888545ce0f4a3a332b1ed6358ca8107e878d25e46aa?__gda__=exp=1584369109~hmac=e7bb7a5cce083ba4b6c323ef9 root ######################################################################## 100.0% ==> Pouring go-1.14.catalina.bottle.tar.gz 🍺 /usr/local/Cellar/go/1.14: 9,431 files, 424MB ihoogendoor-a01:~ iwanhoogendoorn$
Now that Go Lang is installed we can clone the new NSX-Tprovider code:
ihoogendoor-a01:~ iwanhoogendoorn$ cd Coding/nestedlabs/Terraform/ ihoogendoor-a01:Terraform iwanhoogendoorn$ mkdir terratemp ihoogendoor-a01:Terraform iwanhoogendoorn$ cd terratemp/ ihoogendoor-a01:terratemp iwanhoogendoorn$ pwd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/terratemp ihoogendoor-a01:terratemp iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git Cloning into ‘terraform-provider-nsxt’… remote: Enumerating objects: 86, done. remote: Counting objects: 100% (86/86), done. remote: Compressing objects: 100% (51/51), done. remote: Total 15107 (delta 46), reused 51 (delta 34), pack-reused 15021 Receiving objects: 100% (15107/15107), 16.23 MiB
Now that the code is downloaded we need to compile it from source:
CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (compile provider from source output) ===> :
When we look in the directory we see the following file “terraform-provider-nsxt":
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls -l total 88824 -rw-r--r-- 1 iwanhoogendoorn staff 8857 Mar 16 15:04 CHANGELOG.md -rw-r—r— 1 iwanhoogendoorn staff 2422 Mar 16 15:04 CONTRIBUTING.md -rw-r—r— 1 iwanhoogendoorn staff 1839 Mar 16 15:04 GNUmakefile -rw-r—r— 1 iwanhoogendoorn staff 15248 Mar 16 15:04 LICENSE.txt -rw-r—r— 1 iwanhoogendoorn staff 485 Mar 16 15:04 NOTICE.txt -rw-r—r— 1 iwanhoogendoorn staff 7322 Mar 16 15:04 README.md -rw-r—r— 1 iwanhoogendoorn staff 456 Mar 16 15:23 go.mod -rw-r—r— 1 iwanhoogendoorn staff 28565 Mar 16 15:04 go.sum -rw-r—r— 1 iwanhoogendoorn staff 427 Mar 16 15:04 main.go drwxr-xr-x 248 iwanhoogendoorn staff 7936 Mar 16 15:04 nsxt drwxr-xr-x 6 iwanhoogendoorn staff 192 Mar 16 15:04 scripts -rwxr-xr-x 1 iwanhoogendoorn staff 45391620 Mar 16 15:23 terraform-provider-nsxt <-- new file drwxr-xr-x 10 iwanhoogendoorn staff 320 Mar 16 15:04 tools drwxr-xr-x 8 iwanhoogendoorn staff 256 Mar 16 15:04 vendor drwxr-xr-x 4 iwanhoogendoorn staff 128 Mar 16 15:04 website ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$
We need to copy this file in our working directory where our Terraform script is in that we want to execute for deployment.
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test/terraform-provider-nsxt ihoogendoor-a01:Test iwanhoogendoorn$ cd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test
I needed to remove the cached provided, because I executed the "terraform init" before where it used the downloaded “default” NSX-T provider. I want to use the”local” and “just compiled” provider so I need to remove the .terraform directory.
ihoogendoor-a01:Test iwanhoogendoorn$ ls -lah total 88664 drwxr-xr-x 5 iwanhoogendoorn staff 160B Mar 16 15:36 . drwxr-xr-x 15 iwanhoogendoorn staff 480B Mar 16 15:43 .. drwxr-xr-x 3 iwanhoogendoorn staff 96B Mar 14 22:59 .terraform -rw-r—r— 1 iwanhoogendoorn staff 838B Mar 16 15:28 networks.tf -rwxr-xr-x 1 iwanhoogendoorn staff 43M Mar 16 15:26 terraform-provider-nsxt ihoogendoor-a01:Test iwanhoogendoorn$ rm -R .terraform/
Now lets run the “init” command:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform init Initializing the backend... Initializing provider plugins… Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.
If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialise your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
And when we check the version we see the “unversioned” remark, and this tells us we are using our NSX-T provider we have just compiled.
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform version Terraform v0.12.24 + provider.nsxt (unversioned)
So we are ready to use the “newest" version of the NSX-T provider that is required to deploy our NSX-T segments.
Validate your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate
Plan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan
Execute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply
When the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy
The terraform provider is currently under development. If you download and compile the latest version (that is still under development) it can be that old functionality that was working first, is now broken. To go back to a specific version with specific commits you can do this by following the below procedure (Thank you Rutger Blom for this):
Browse to a new fresh directory:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cd terraform-provider-nsxt-v0/
The directory is really empty:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls
Clone the full repository:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git Cloning into 'terraform-provider-nsxt'... remote: Enumerating objects: 142, done. remote: Counting objects: 100% (142/142), done. remote: Compressing objects: 100% (87/87), done. remote: Total 15163 (delta 84), reused 78 (delta 54), pack-reused 15021 Receiving objects: 100% (15163/15163), 16.25 MiB 8.33 MiB/s, done. Resolving deltas: 100% (8351/8351), done. Checking out files: 100% (4088/4088), done.
Now lets browse into the directory:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls terraform-provider-nsxt ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ cd terraform-provider-nsxt ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls CHANGELOG.md GNUmakefile NOTICE.txt go.mod main.go scripts vendor CONTRIBUTING.md LICENSE.txt README.md go.sum nsxt tools website
Fetch a specific version + commits:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git fetch origin c01d221ceb98e114aadd84126707633d25fcb43b From https://github.com/terraform-providers/terraform-provider-nsxt * branch c01d221ceb98e114aadd84126707633d25fcb43b -> ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git reset --hard is now at c01d221 Policy block subnet - wait for delete realization
Compile the specific version :
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go build go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.1.1 ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$
'Copy the new provider to the repository directory of your NSX-T Terraform code:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/#create-T1