Create Universal Security Policies with Cross-VC NSX active/active site deployments
I am currently involved in a project with several Cross-VC NSX deployments. Developing a security policy around this with an active/active Cross-VC NSX deployment can be confusing when you don't know how it should be designed or what the "rules" are around the design.
After reading this blog post of Humair Ahmed It became clear that there are some constraints around the use of
- Universal Security Tags
- Universal Security Groups
- Universal IP Sets
- Universal Distributed Firewall rules
I decided to test this out to have a clear understanding of what works and what not.
Just to stress this once more this is an active / active Cross-VC NSX setup.
This is the diagram that I have used for the tests.
The following test is a ping test to verify if the communication is working between DC-A and DC-B.
Test if PING is working from WEB-A to WEB-B.
root@Web01:~# ping -c 4 172.20.1.3 PING 172.20.1.3 (172.20.1.3) 56(84) bytes of data. 64 bytes from 172.20.1.3: icmp_req=1 ttl=64 time=0.666 ms 64 bytes from 172.20.1.3: icmp_req=2 ttl=64 time=0.592 ms 64 bytes from 172.20.1.3: icmp_req=3 ttl=64 time=0.588 ms 64 bytes from 172.20.1.3: icmp_req=4 ttl=64 time=0.585 ms --- 172.20.1.3 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3000ms rtt min/avg/max/mdev = 0.585/0.607/0.666/0.045 ms root@Web01:~#
Test if PING is working from WEB-A to APP-B.
root@Web01:~# ping -c 4 172.20.2.3 PING 172.20.2.3 (172.20.2.3) 56(84) bytes of data. 64 bytes from 172.20.2.3: icmp_req=1 ttl=63 time=948 ms 64 bytes from 172.20.2.3: icmp_req=2 ttl=63 time=0.625 ms 64 bytes from 172.20.2.3: icmp_req=3 ttl=63 time=0.661 ms 64 bytes from 172.20.2.3: icmp_req=4 ttl=63 time=0.649 ms --- 172.20.2.3 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 2999ms rtt min/avg/max/mdev = 0.625/237.547/948.256/410.328 ms root@Web01:~#
Test 1
- Create Universal IP Sets with specific IP addresses
- Create Universal Security Groups and statically select the corresponding Universal IP Sets
- Create Universal Distributed Firewall rules with the Universal Security Groups as the source and destination
- Do ping test from DC-A to DC-B and verify if communication is blocked.
Create the following Universal IP Sets with the following IP addresses:
| Universal IP Set Name | IP Information | Corresponding VM's |
|---|---|---|
| IPSET-WEB-A | 172.20.1.1/24
172.20.1.2/24 |
T1-WEB-1
T1-WEB-2 |
| IPSET-WEB-B | 172.20.1.3/24
172.20.1.4/24 |
T1-WEB-3
T1-WEB-4 |
| IPSET-APP-A | 172.20.2.1/24
172.20.2.1/24 |
T1-APP-1
T1-APP-2 |
| IPSET-APP-B | 172.20.2.3/24
172.20.2.4/24 |
T1-APP-3
T1-APP-4 |
| IPSET-DB-A | 172.20.3.1/24 | T1-DB-1 |
| IPSET-DB-B | 172.20.3.2/24 | T1-DB-2 |
Create the following Universal Security Groups and make the Universal IP Set member of this group.
| Universal Security Group | IP Set Member |
|---|---|
| SG-IPSET-WEB-A | IPSET-WEB-A |
| SG-IPSET-WEB-B | IPSET-WEB-B |
| SG-IPSET-APP-A | IPSET-APP-A |
| SG-IPSET-APP-B | IPSET-APP-B |
| SG-IPSET-DB-A | IPSET-DB-A |
| SG-IPSET-DB-B | IPSET-DB-B |
Create the following Universal Distributed Firewall rule:
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-IPSET-WEB-A | SG-IPSET-WEB-B | BLOCK | ANY |
Start the ping test:
root@Web01:~# ping -c 4 172.20.1.3 PING 172.20.1.3 (172.20.1.3) 56(84) bytes of data. --- 172.20.1.3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms root@Web01:~#
Conclusion: This works as expected
Create the following Universal Distributed Firewall rule:
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-IPSET-WEB-A | SG-IPSET-APP-B | BLOCK | ANY |
Start the ping test:
root@Web01:~# ping -c 4 172.20.2.3 PING 172.20.2.3 (172.20.2.3) 56(84) bytes of data. --- 172.20.2.3 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 3000ms root@Web01:~#
Conclusion: This works as expected
Create the following Universal Distributed Firewall rule:
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-IPSET-WEB-A | SG-IPSET-APP-A | BLOCK | ANY |
Start the ping test:
root@Web01:~# ping -c 4 172.20.2.1 PING 172.20.2.1 (172.20.2.1) 56(84) bytes of data. --- 172.20.2.1 ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms root@Web01:~#
Conclusion: This works as expected
Test 2
- Create Universal Security Tag
- Create Universal Security Groups and statically select the corresponding Universal Security Tag
- Assign the corresponding VM's to the Universal Security Tags
- Create Universal Distributed Firewall rules with the Universal Security Groups as the source and destination
- Do ping test from DC-A to DC-B and verify if communication is blocked.
Create the following Universal Security Tags:
| Universal Security Tags |
|---|
| TAG-WEB-A |
| TAG-WEB-B |
| TAG-APP-A |
| TAG-APP-B |
| TAG-DB-A |
| TAG-DB-B |
Create the following Universal Security Groups and make the Universal Security Tags member of this group.
| Universal Security Group | Universal Security Tag |
|---|---|
| SG-TAG-WEB-A | TAG-WEB-A |
| SG-TAG-WEB-B | TAG-WEB-B |
| SG-TAG-APP-A | TAG-APP-A |
| SG-TAG-APP-B | TAG-APP-B |
| SG-TAG-DB-A | TAG-DB-A |
| SG-TAG-DB-B | TAG-DB-B |
After the creation of the first Universal Security Group I discovered that it is NOT possible to add a Universal Security Tag to a Universal Security Group. To continue I have chosen to configure "Global" Security Groups and include the "Global" Universal Security Tags.
Tags on the Primary NSX Manager:
Tags on the Secondary NSX Manager:
It is not possible to assign "global" tags to a Universal Security Group
So assigning "Global" Security Tags to "Global" Security Groups
The next step is to assign the Universal Security Tag to the corresponding VM's When assigning an Universal Security Tag to a VM I am only allowed to select the "local" VM objects.
This is understandable because the the Universal Security Tag is only copied or synced to the secondary NSX manager. We can only select from VM objects "local" to this vCenter Server and (primary) NSX Manager.
When I click "OK" I get an error:
The "Assign Security Tag" operation failed for the entity with the following error message. Unique ID selection criteria for universal security tag and VM mapping is not set.
After some research on the internet I stumbled up on this link
It clearly states: "The unique ID selection criteria is used when assigning tags to Virtual Machines on active standby deployments."
As our goal is to apply the security policies in active/active mode and not in active/standby mode the way we are trying to implement this stops here.
The creation of the following Universal Distributed Firewall rules did not make any sense anymore because the above reasons.
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-TAG-WEB-A | SG-TAG-WEB-A=B | BLOCK | ANY |
| SG-TAG-WEB-A | SG-TAG-APP-B | BLOCK | ANY |
| SG-TAG-WEB-A | SG-TAG-APP-A | BLOCK | ANY |
The first two rules: There are no test results available because the configuration how I had this in mind was not possible. The last rule: This would be possible when we use a "global" Distributed Firewall rule and keep the (global) groups / (global) tags / (global) rules local to the primary vCenter Server + NSX Manager pair. But this was not the goal of this test.
Test 3
- Create Universal Security Groups and statically select the corresponding VM's
- Create Universal Distributed Firewall rules with the Universal Security Groups as the source and destination
- Do ping test from DC-A to DC-B and verify if communication is blocked.
Create the following Universal Security Groups and make the VM's member of this group.
| Universal Security Group | VM's |
|---|---|
| SG-STATIC-WEB-A | T1-WEB-1
T1-WEB-2 |
| SG-STATIC-WEB-B | T1-WEB-3
T1-WEB-4 |
| SG-STATIC-APP-A | T1-APP-1
T1-APP-2 |
| SG-STATIC-APP-B | T1-APP-3
T1-APP-4 |
| SG-STATIC-DB-A | T1-DB-1 |
| SG-STATIC-DB-B | T1-DB-2 |
After the creation of the first Universal Security Group and trying to add a VM to this group I discovered that it is not possible. It is not possible to select "VM" objects within Universal Security Groups. In this case I (have to) choose to create a (global) Security Group and assign the VM's statically to the group.
When doing this the Primary vCenter Server + NSX Manager pair can not see the inventory of the Secondary vCenter Server + NSX Manager pair and vica versa. So we need to create the DC-B groups on the Secondary NSX Manager.
Assigning VM's statically to a "Global" Security Group on the Primary NSX Manager
Assigning VM's statically to a "Global" Security Group on the Secondary NSX Manager
The "Global" Security Group to VM overview on the Primary NSX Manager
The "Global" Security Group to VM overview on the Secondary NSX Manager
Create the following Universal Distributed Firewall rule:
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-STATIC-WEB-A | SG-STATIC-WEB-B | BLOCK | ANY |
As expected I am not able to create an Universal Distributed Firewall rule with "global" Security Groups. Again the test stops here.
Creating the following rules were also not possible:
| SOURCE | DESTINATION | ACTION | TRAFFIC |
|---|---|---|---|
| SG-STATIC-WEB-A | SG-STATIC-APP-B | BLOCK | ANY |
| SG-STATIC-WEB-A | SG-STATIC-APP-A | BLOCK | ANY |
The first two rules: There are no test results available because the configuration how I had this in mind was not possible. The last rule: This would be possible when we use a "global" Distributed Firewall rule and keep the (global) groups / (global) tags / (global) rules local to the primary vCenter Server + NSX Manager pair. But this was not the goal of this test.
Conclusion
Only test 1 works in combination with Universal Security Groups, Universal IP Set's and Universal Distributed Firewall Rules across two Data Centers in active / active mode. In the case of test 2 and 3 the entire application most reside either in DC-A or in DC-B like Humair states in his blog post.