Terraform: NSX-T infra segments
Now that we prepared the environment to start our Terraform coding it is time to actually start some Terraform coding. For our automated nested lab we need some network segments that the infra that we are building will consume.
The Segments we need to create with the subnets are listed below:
- LX-APP-MGMT-11 | 192.168.11.0/24
- Subnet for Management applications, like vCenter Server, NSX-T Manager, etc.
- LX-ESXI-MGMT-UT | 192.168.12.0/24
- Subnet for all vmk0 interfaces for ESXi management traffic.
- LX-VMOTION-13 | 192.168.13.0/24
- Subnet for all vmk1 interfaces for ESXi vMotion traffic.
- LX-VSAN-14 | 192.168.14.0/24
- Subnet for all vmk2 interfaces for ESXi VSAN traffic.
- LX-REPLICATION-15 | 192.168.15.0/24
- Subnet for all vmk3 interfaces for ESXi Replication traffic.
- LX-OVERLAY-HOST-16 | 192.168.16.0/24
- Subnet for all Host (ESXi) Transport Node TEP interfaces for GENEVE traffic.
- LX-OVERLAY-EDGE-17 | 192.168.17.0/24
- Subnet for all Edge (VM) Transport Node TEP interfaces for GENEVE traffic.
- LX-BGP-UPLINK-01-18 | 192.168.18.0/24
- Subnet for BGP uplink 01 for N/S Connectivity.
- LX-BGP-UPLINK-02-19 | 192.168.19.0/24
- Subnet for BGP uplink 02 for N/S Connectivity.
- LX-TRANSIT-LEAF-SPINE-20 | 192.168.20.0/24
- Subnet to connect the virtual ToR switches/leafs (VX01 and VX02) with the core/spine (VX00).
Where the “X” is used we need to replace this with a number that will reflect our lab number.
The terraform script in order to deploy these segments on my NSX-T infrastructure NSX-T Manager can be found below:
❯ tree ├── main.tf
main.tf
CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (main.tf code) ===> :
root # use Terraform 0.12 to execute script root # NSX-T Manager Credentials provider "nsxt" { host = “<your NSX-T manager IP/Hostname>” username = “admin” password = “<your NSX-T manager password>” allow_unverified_ssl = true max_retries = 10 retry_min_delay = 500 retry_max_delay = 5000 retry_on_status_codes = [429] } data “nsxt_policy_transport_zone” “overlay_tz” { display_name = “TZ-OVERLAY” } resource “nsxt_policy_segment” “tf_app_mgmt” { display_name = “LX-APP-MGMT-11” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_esxi_mgmt” { display_name = “LX-ESXI-MGMT-UT” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_vmotion” { display_name = “LX-VMOTION-13” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_vsan” { display_name = “LX-VSAN-14” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource "nsxt_policy_segment” “tf_replication” { display_name = “LX-REPLICATION-15” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_overlay_host” { display_name = “LX-OVERLAY-HOST-16” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_overlay_edge” { display_name = “LX-OVERLAY-EDGE-17” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_uplink_01” { display_name = “LX-BGP-UPLINK-01-18” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_uplink_02” { display_name = “LX-BGP-UPLINK-02-19” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path } resource “nsxt_policy_segment” “tf_transit_leaf_spine” { display_name = “LX-TRANSIT-LEAF-SPINE-20” description = “Segment created by Terraform” transport_zone_path = data.nsxt_policy_transport_zone.overlay_tz.path }
Validate your code:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate
Plan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan
Execute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply
When the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy
Before the above script can be executed (as the time of this writing: March 2020) we need to compile the latest NSX-T Terraform Provider from source. The NSX-T Terraform Provider that is downloaded automatically is a version that does not contain all features and the policy management features are not fully available and in beta at the moment.
In order to create the segments with terraform in NSX-T in the Simple UI the default nsx-t terraform provider is not supported. With the default you can only create segments in the advanced UI (that is not visible in the Simple UI). In order to get the latest terraform NSX-T provider you need to compile this yourself.
Using the latest version of the NSX–T provider
By default when using the NSX-T provider in your Terraform script the “Default” provider will be downloaded and used. This “default” version is the latest and stable version of the provider, but not the one with the latest features. There are a lot more features and commands under development and these can only be used with the latest version of the NSX-T provider. In order to get this new NSX-T provider with the latest and greatest features we need to clone the GitHub repo of the provider and compile it from source using Go Lang.
So the first step is to install Go Lang on my Mac. We do this using brew. How brew works can be found here.
ihoogendoor-a01:~ iwanhoogendoorn$ # Go development
ihoogendoor-a01:~ iwanhoogendoorn$ export GOPATH=“${HOME}/.go”
ihoogendoor-a01:~ iwanhoogendoorn$ export GOROOT=“$(brew —prefix golang)/libexec”
ihoogendoor-a01:~ iwanhoogendoorn$ export PATH=“$PATH:${GOPATH}/bin:${GOROOT}/bin”
ihoogendoor-a01:~ iwanhoogendoorn$ test -d “${GOPATH}”
ihoogendoor-a01:~ iwanhoogendoorn$ brew install go Updating Homebrew... ==> Auto-updated Homebrew! Updated 2 taps (homebrew/core and homebrew/cask). ==> New Formulae azcopy container-structure-test dhall-yaml mtoc publish swift-format ==> Updated Formulae appium dvc gsoap libvirt run tiny-fugue arp-sk ettercap hugo minio selenium-server-standalone tomee-plume arping exploitdb inspircd minio-mc skaffold tomee-plus arpoison fop legit miniserve spades tomee-webprofile balena-cli frege libarchive nanopb-generator sslsplit travis bash-completion@2 gitlab-gem libcbor nativefier suricata vice calicoctl gitversion libfido2 poppler tccutil wpscan docbook glooctl libnet ripgrep tcptraceroute wtfutil dpkg golang-migrate libnids root tflint zeek ==> Updated Casks 1password-cli fontforge lazarus qownnotes unity-android-support-for-editor angry-ip-scanner fontgoggles lrtimelapse rectangle unity-download-assistant anydo frhelper maccy refined-github-safari unity-ios-support-for-editor beaker-browser gifox macgamestore ripcord unity-lumin-support-for-editor chromium glyphs metaz supercollider unity-webgl-support-for-editor clashx hbuilderx meteorologist texworks unity-windows-support-for-editor electron-fiddle icq mochi thonny webcatalog engine-prime jami multitouch tinderbox espresso jprofiler odrive trilium-notes exodus kapitainsky-rclone-browser pd-l2ork unity ==> Deleted Casks google-nik-collection ==> Downloading https»//homebrew.bintray.com/bottles/go-1.14.catalina.bottle.tar.gz ==> Downloading from https»//akamai.bintray.com/a4/a4e81b1025b77af039a34888545ce0f4a3a332b1ed6358ca8107e878d25e46aa?__gda__=exp=1584369109~hmac=e7bb7a5cce083ba4b6c323ef9 root ######################################################################## 100.0% ==> Pouring go-1.14.catalina.bottle.tar.gz 🍺 /usr/local/Cellar/go/1.14: 9,431 files, 424MB ihoogendoor-a01:~ iwanhoogendoorn$
Now that Go Lang is installed we can clone the new NSX-Tprovider code:
ihoogendoor-a01:~ iwanhoogendoorn$ cd Coding/nestedlabs/Terraform/ ihoogendoor-a01:Terraform iwanhoogendoorn$ mkdir terratemp ihoogendoor-a01:Terraform iwanhoogendoorn$ cd terratemp/ ihoogendoor-a01:terratemp iwanhoogendoorn$ pwd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/terratemp ihoogendoor-a01:terratemp iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git Cloning into ‘terraform-provider-nsxt’… remote: Enumerating objects: 86, done. remote: Counting objects: 100% (86/86), done. remote: Compressing objects: 100% (51/51), done. remote: Total 15107 (delta 46), reused 51 (delta 34), pack-reused 15021 Receiving objects: 100% (15107/15107), 16.23 MiB
Now that the code is downloaded we need to compile it from source:
CLICK ON EXPAND ===> ON THE RIGHT ===> TO SEE THE OUTPUT (compile provider from source output) ===> :
ihoogendoor-a01:terratemp iwanhoogendoorn$ ls -l total 0 drwxr-xr-x 20 iwanhoogendoorn staff 640 Mar 16 15:04 terraform-provider-nsxt ihoogendoor-a01:terratemp iwanhoogendoorn$ cd terraform-provider-nsxt/ ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls CHANGELOG.md GNUmakefile NOTICE.txt go.mod main.go scripts vendor CONTRIBUTING.md LICENSE.txt README.md go.sum nsxt tools website ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go build go: downloading github.com/hashicorp/terraform-plugin-sdk v1.1.0 go: downloading github.com/google/uuid v1.1.1 go: downloading github.com/vmware/go-vmware-nsxt v0.0.0-20191219213550-f4221331f638 go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.2.0 go: downloading github.com/vmware/vsphere-automation-sdk-go/runtime v0.1.1 go: downloading github.com/vmware/vsphere-automation-sdk-go/lib v0.1.1 go: downloading github.com/hashicorp/go-multierror v1.0.0 go: downloading github.com/zclconf/go-cty v1.1.0 go: downloading github.com/beevik/etree v1.1.0 go: downloading github.com/hashicorp/go-hclog v0.9.2 go: downloading github.com/gorilla/mux v1.7.3 go: downloading golang.org/x/text v0.3.2 go: downloading github.com/mitchellh/mapstructure v1.1.2 go: downloading github.com/agext/levenshtein v1.2.2 go: downloading google.golang.org/grpc v1.23.0 go: downloading github.com/vmihailenco/msgpack v3.3.3+incompatible go: downloading github.com/mitchellh/reflectwalk v1.0.1 go: downloading github.com/hashicorp/go-version v1.2.0 go: downloading github.com/spf13/afero v1.2.2 go: downloading github.com/hashicorp/hcl2 v0.0.0-20190821123243-0c888d1241f6 go: downloading github.com/hashicorp/logutils v1.0.0 go: downloading github.com/mitchellh/copystructure v1.0.0 go: downloading golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 go: downloading github.com/hashicorp/go-getter v1.4.0 go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db go: downloading golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/apparentlymart/go-textseg v1.0.0 go: downloading github.com/hashicorp/go-plugin v1.0.1 go: downloading google.golang.org/api v0.9.0 go: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d go: downloading github.com/aws/aws-sdk-go v1.19.39 go: downloading golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa go: downloading github.com/mitchellh/go-wordwrap v1.0.0 go: downloading cloud.google.com/go v0.45.1 go: downloading github.com/google/go-cmp v0.3.0 go: downloading github.com/hashicorp/hil v0.0.0-20190212112733-ab17b08d6590 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/mitchellh/cli v1.0.0 go: downloading github.com/gibson042/canonicaljson-go v1.0.3 go: downloading github.com/oklog/run v1.0.0 go: downloading github.com/armon/go-radix v1.0.0 go: downloading github.com/mitchellh/go-homedir v1.1.0 go: downloading github.com/ulikunitz/xz v0.5.5 go: downloading github.com/hashicorp/go-cleanhttp v0.5.1 go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 go: downloading github.com/hashicorp/go-uuid v1.0.1 go: downloading github.com/golang/protobuf v1.3.2 go: downloading github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d go: downloading github.com/hashicorp/terraform-config-inspect v0.0.0-20190821133035-82a99dc22ef4 go: downloading github.com/posener/complete v1.2.1 go: downloading github.com/hashicorp/go-safetemp v1.0.0 go: downloading github.com/bgentry/speakeasy v0.1.0 go: downloading github.com/hashicorp/hcl v0.0.0-20170504190234-a4b07c25de5f go: downloading github.com/mattn/go-isatty v0.0.4 go: downloading github.com/googleapis/gax-go/v2 v2.0.5 go: downloading github.com/zclconf/go-cty-yaml v1.0.1 go: downloading github.com/fatih/color v1.7.0 go: downloading github.com/mitchellh/go-testing-interface v1.0.0 go: downloading github.com/mattn/go-colorable v0.0.9 go: downloading go.opencensus.io v0.22.0 go: downloading github.com/apparentlymart/go-cidr v1.0.1 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
When we look in the directory we see the following file “terraform-provider-nsxt":
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls -l total 88824 -rw-r--r-- 1 iwanhoogendoorn staff 8857 Mar 16 15:04 CHANGELOG.md -rw-r—r— 1 iwanhoogendoorn staff 2422 Mar 16 15:04 CONTRIBUTING.md -rw-r—r— 1 iwanhoogendoorn staff 1839 Mar 16 15:04 GNUmakefile -rw-r—r— 1 iwanhoogendoorn staff 15248 Mar 16 15:04 LICENSE.txt -rw-r—r— 1 iwanhoogendoorn staff 485 Mar 16 15:04 NOTICE.txt -rw-r—r— 1 iwanhoogendoorn staff 7322 Mar 16 15:04 README.md -rw-r—r— 1 iwanhoogendoorn staff 456 Mar 16 15:23 go.mod -rw-r—r— 1 iwanhoogendoorn staff 28565 Mar 16 15:04 go.sum -rw-r—r— 1 iwanhoogendoorn staff 427 Mar 16 15:04 main.go drwxr-xr-x 248 iwanhoogendoorn staff 7936 Mar 16 15:04 nsxt drwxr-xr-x 6 iwanhoogendoorn staff 192 Mar 16 15:04 scripts -rwxr-xr-x 1 iwanhoogendoorn staff 45391620 Mar 16 15:23 terraform-provider-nsxt <-- new file drwxr-xr-x 10 iwanhoogendoorn staff 320 Mar 16 15:04 tools drwxr-xr-x 8 iwanhoogendoorn staff 256 Mar 16 15:04 vendor drwxr-xr-x 4 iwanhoogendoorn staff 128 Mar 16 15:04 website ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$
We need to copy this file in our working directory where our Terraform script is in that we want to execute for deployment.
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test/terraform-provider-nsxt ihoogendoor-a01:Test iwanhoogendoorn$ cd /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/Test
I needed to remove the cached provided, because I executed the "terraform init" before where it used the downloaded “default” NSX-T provider. I want to use the”local” and “just compiled” provider so I need to remove the .terraform directory.
ihoogendoor-a01:Test iwanhoogendoorn$ ls -lah total 88664 drwxr-xr-x 5 iwanhoogendoorn staff 160B Mar 16 15:36 . drwxr-xr-x 15 iwanhoogendoorn staff 480B Mar 16 15:43 .. drwxr-xr-x 3 iwanhoogendoorn staff 96B Mar 14 22:59 .terraform -rw-r—r— 1 iwanhoogendoorn staff 838B Mar 16 15:28 networks.tf -rwxr-xr-x 1 iwanhoogendoorn staff 43M Mar 16 15:26 terraform-provider-nsxt ihoogendoor-a01:Test iwanhoogendoorn$ rm -R .terraform/
Now lets run the “init” command:
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform init Initializing the backend... Initializing provider plugins… Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.
If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialise your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
And when we check the version we see the “unversioned” remark, and this tells us we are using our NSX-T provider we have just compiled.
ihoogendoor-a01:#Test iwanhoogendoorn$ tfenv use 0.12.24 [INFO] Switching to v0.12.24 [INFO] Switching completed ihoogendoor-a01:Test iwanhoogendoorn$ terraform version Terraform v0.12.24 + provider.nsxt (unversioned)
So we are ready to use the “newest" version of the NSX-T provider that is required to deploy our NSX-T segments.
Validate your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform validate
Plan your code:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform plan
Execute your code to implement the Segments:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform apply
When the segments need to be removed again you can revert the implementation:
ihoogendoor-a01:Test iwanhoogendoorn$ terraform destroy
The terraform provider is currently under development. If you download and compile the latest version (that is still under development) it can be that old functionality that was working first, is now broken. To go back to a specific version with specific commits you can do this by following the below procedure (Thank you Rutger Blom for this):
Browse to a new fresh directory:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cd terraform-provider-nsxt-v0/
The directory is really empty:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls
Clone the full repository:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ git clone https://github.com/terraform-providers/terraform-provider-nsxt.git Cloning into 'terraform-provider-nsxt'... remote: Enumerating objects: 142, done. remote: Counting objects: 100% (142/142), done. remote: Compressing objects: 100% (87/87), done. remote: Total 15163 (delta 84), reused 78 (delta 54), pack-reused 15021 Receiving objects: 100% (15163/15163), 16.25 MiB 8.33 MiB/s, done. Resolving deltas: 100% (8351/8351), done. Checking out files: 100% (4088/4088), done.
Now lets browse into the directory:
ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ ls terraform-provider-nsxt ihoogendoor-a01:terraform-provider-nsxt-v0 iwanhoogendoorn$ cd terraform-provider-nsxt ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ ls CHANGELOG.md GNUmakefile NOTICE.txt go.mod main.go scripts vendor CONTRIBUTING.md LICENSE.txt README.md go.sum nsxt tools website
Fetch a specific version + commits:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git fetch origin c01d221ceb98e114aadd84126707633d25fcb43b From https://github.com/terraform-providers/terraform-provider-nsxt * branch c01d221ceb98e114aadd84126707633d25fcb43b -> ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ git reset --hard is now at c01d221 Policy block subnet - wait for delete realization
Compile the specific version :
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ go build go: downloading github.com/vmware/vsphere-automation-sdk-go/services/nsxt v0.1.1 ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$
'Copy the new provider to the repository directory of your NSX-T Terraform code:
ihoogendoor-a01:terraform-provider-nsxt iwanhoogendoorn$ cp terraform-provider-nsxt /Users/iwanhoogendoorn/Coding/nestedlabs/Terraform/#create-T1