Recently i got the question if in one Cloud Director Tenant (Organization) Granular Role Based Access Control and separation of rights can be configured between multiple teams within that Organization.
In our Test case Team-A is responsible for Org VDC A and can only manage and view the Edge GW resources (networks, Edge Gateways) within that VDC-Group. Team-B responsible for Org VDC B and can manage and view the all resources in all VDC-Groups, Except the Edge GW in Org VDC A. This Edge GW can only be managed by the Team-A.
Also because of some Tenant requirements the T0 (VRF) is also split between Internet and Customer specific. You can read more about this setup in an this post.
Separation of Rights between Org VDCs
Shared networks between Org VDCs
Team-A Can only manage the Edge GW in ORG VDC A
Team-B can manage all resources in both ORG VDCs except the Edge GW in ORG VDC A
One Provider VDC (vCenter)
One Organization in Cloud Director (Tenant1)
Two Org VDC connected to the same Provider VDC (ORG VDC A & ORG VDC B)
Two Data Center Groups (VDC Group A & VDC Group B)
Two Edge GW (Edge A connected to VDC Groupp A & Edge Bconnected to VDC Group B)
Tenant Acces Role Team-A
Tenant Acces Role Team-B
From version 10.2, VMware Cloud Director supports Data Center Group networking backed by NSX-T Data Center.
A Data Center Group acts as a Cross-VDC router that provides centralized networking administration, egress point configuration, and east-west traffic between all networks within the group.
Using Data Center Groups, we can share organization networks across various ORG VDCs. To do so we first group the virtual data centers, then create a VDC network that is scoped to the Data Center Group. A data center group can contain between one and 16 virtual data centers that are configured to share multiple egress points.
We need to created 2 Data Center Groups and connect them to the participating VDC & Edges
VDC Group A -> ORG VDC A (24-2 in picture below) & Edge A VDC Group B -> ORG VDC B (24 in picture below) & Edge B
By default, organization VDCs are shared with all users and groups that have a role which includes the Allow Access to All Organization VDCs right.
As an Organization Administrator, you can limit the access to each of the organization VDCs in your organization to specific users and groups.
Our organization has multiple organization VDCs and we want to have them managed separately, so create a custom role that would function as an organization VDC administrator and assign it to specific users or groups within your organization, providing them with access only to a specific VDC’s compute and networking resources.
For Team-B we can use the predefined role Organization Administrator. In this role the following right is allowed: Allow Access to All Organization VDCs
This permission is exactly what is tells you it does, giving you permission to ALL Organizations VDC in the Organization. So with this role we are able to manage VM’s, networks, etc in all the Organization VDC’s.
Exactly what we need for Team-B.
For Team-A we need to make a new role with more granular permissions. Create a new role and exclude the Allow Access to All Organization VDCs right. Set the rest of the permissions to view and manage the Edges and Networks.
Publish both Roles to the Tenant and create 2 users in the Tenant.
Limit Access to ORG-VDC
Now we need to limit the access to the Org VDC, on the Virtual Data Center dashboard screen, click the card of the virtual data center that you want to limit access to.
Under Settings, click Sharing, The list of users and groups within the organization that have access to the VDC appears. To change the access settings to the organization VDC, click Edit.
Select Specific Users and Groups, From the Users list, select the users that you want to provide with access to the VDC, same procedure if you are using groups.
So for ORG VDC A select Team-A, Team-B already has access to all ORG VDC because of the Allow Access to All Organization VDCs right.
To share the VDC with the selected users and groups, click Share. At this moment Team-A can only view and manage Edges and Networks in ORG VDC A, and Team B can view and manage all resources in both ORG VDCs, also the Edge in ORG VDC A. if we need to get that sorted out we need to created roles and groups for every thinkable Resource set (Edge Admin, VM Admin, DFW Admin, etc) in every ORG VDC.
But another requirement in this Test Case is the Shared Network between ORG VDCs. For this requirement it is needed to add the other VDC to the participating VDCs in the Data Center Group.
As soon as you configure this Team-A (which can only see the Edge-A in ORG VDC A), immediately can see the Edge-B under ORG VDC B. A no go in our case. The rights are distributed very horizontal. So as soon as you have multiple participating VDC in your Data Center group the team that was restricted to viewing and managing the resources in ORG VDC A, can now also view and manage the resources it is permitted to in ORG VDC B.
For this Test Case the outcome was negative as we needed shared networks between ORG VDCs. If the sharing of networks is not needed, you set a very Granular RBAC model, but keep in mind when you set the Allow Access to All Organization VDCs right in a role, the users/groups that have this role are allowed to show all resources they are eligible to in All Organization VDCs
Half way december i switched to another team at my current employer, and got my hands dirty with Cloud director, NSX-T and AVI. As this was my first real hands-on with VMware Cloud Director. I was given the task to investigate some scenarios in which a tenant is given a second Edge Gateway, for the separation of Internet and Customer Networks traffic.
Before describing the scenarios, I’ll assume you have basic NSX-T, routing and VCD knowledge.
Current Tenant Setup
In the current setup a tenant is given:
Cloud Director (10.3): Organization (example: Tenant1)
Cloud Director: Org VDC
NSX-T (3.2): VRF based tier-0 gateway for both Internet VRF and Customer VRF connected to Parent T0
Cloud Director NSX-T: 1 Edge Gateway (T1)
Clloud Director NSX-T: 1 VDC Group
If Load balancing is used: dedicated AVI Service Engine Group
Because there is a 1:1 relation between the T0 and the Edge Gateway (dedicated T0), route advertisement of connected tenant networks is available.
The traffic for both the customer networks and Internet is flowing through the same Tenant Edge gateway and VRF based tier-0 gateway.
The first scenario I tested is based on a Shared T0 (in VCD) and 2 Edge Gateways (1 for Internet and 1 for Customer Networks).
Cloud Director: Organization (example: Tenant1)
Cloud Director: Org VDC
NSX-T: VRF based tier-0 gateway for both Internet VRF and Customer VRF connected to Parent T0 (parent T0 is shared between 5 Tenants)
Cloud Director NSX-T: Set T0 to shared
Cloud Director NSX-T: two Edge Gateway (T1) connected to the same T0 (shared)
Cloud Director NSX-T: two VDC Groups (Data Center Groups)
If Load balancing is used: Shared AVI Service Engine Group (dedicated as described in current setup can also be used).
The downside of a Shared T0 is that route advertising of Tenant network connected to the Edge Gateway (T1) isn’t available to the T0.
Tenant VMs can connect to the internet or customer networks by using NAT and firewall rules. SNAT rules need to be created for outbound traffic and DNAT rules inbound traffic. Only use private IP spaces can be used for tenant networks connected to the Edge Gateway (T1).
Also because we use two Edge Gateways in an Org VDC we need to create 2 VDC Groups (Data Center Groups), because there is a restriction of 1 Edge Gateway per VDC Group (Data Center Group), two Datacenter Groups also mean 2 Distributed Firewalls to manage.
If you need connection between a VM in VDC Group 1 and a VM in VDC Group 2, you can create a network that will be spanning across both VDC groups.
In this scenario we still have the BGPs for both Internet and Customer configured on 1 VRF based tier-0 gateway, so this is not completely dedicated. And it is giving a lot of extra effort configuring the SNAT and DNAT.
We also worked out a scenario if the tenant is using AVI. In the normal setup the tenat is given a dedicated Service Engine Group. We also discovered the option to share the Service Engine Group between the Internet and Customer Networks. This option is a valid solution as the AVI per Default configures separates the traffic based on VRF in the Service Engines.
Scenario 2 – 2 Edge GW with dedicated T0
For the 2nd scenario we decoupled also the VRF Based T0 and creating again a 1:1 relation (dedicated T0) between the Edge Gateway and the VRF Based T0’s. Because of this 1:1 relationship route advertisements are available to the T0.
Cloud Director: Organization (example: Tenant1)
Cloud Director: Org VDC
NSX-T: VRF based tier-0 gateway for Internet VRF connected to Parent T0
NSX-T: VRF based tier-0 gateway for Customer VRF connected to Parent T0
Cloud Director NSX-T: Set T0 to be dedicated
Cloud Director NSX-T: two Edge Gateway (T1) connected to the dedicated T0
Cloud Director NSX-T: two VDC Groups (Data Center Groups)
If Load balancing is used: Dedicated AVI Service Engine Group (because no design Change is needed for current model)
If you need connection between a VM in VDC Group 1 and a VM in VDC Group 2, you can created a network that will be spanning across both VDC groups.
The downside of the Scenario is that there are a lot of extra resources needed, and these resources will be billed to the tenant.
We still wanted a Design that met the customer requirements and also is easy to implement in the current setup. Which already in use by several tenants and was the default tenant setup when the platform was initially designed.
In Scenario 1 the Shared setup of the shared VRF based T0 doesn’t met the Customer requirements of separating Customer Networks and Internet traffic. Also the lack of the route advertisements was an issue for the Tenant.
In Scenario 2 where we separated all parts of the setup, all the Tenant requirements were met. The downside of this Scenario is that because all parts of the setup are decoupled, the resources a tenant needs using this setup are doubled. This means:
Recently we saw some warnings about expiring certificates in the NSX-T Global Manager and Local Manager.
When we clicked one of the alerts we got a small description and some API calls we can fire to apply new certificates.
In the Certificates overview (System > Certificates > Certificates), we could see that the certificates Issued to the Local Manager and Global Manager were expiring. The certification id’s were also corresponding to the ones in the alert (not the ones in my screenshots).
The API calls that were mentioned in the Alert description are for the renewal of certificate to the HTTP service (UI), not the Local/Global Manager certificates. The VMware Docs don’t explain in good detail how to change these certificates, i couln’t find it.
The only give away i could find was in step 6: (NSX Federation and the service type).
So before we can replace the certificates, we need to create new Self Signed Certificates for the Local Manager, and the Global Manager.
Create CSR on GM/LM:
to create a CSR (Certificate Signing Request) on the Global or Local Manager go to: System > Certificates > CSRs and click on “Generate CSR“.
For the Global Manager do this via the Global Manager Appliance and for the Local Manager use the Local Manager Appliance, or use the drop down on the top of the screen to choose between your Global Manager or Local Managers.
Fill in all the fields and hit the GENERATE button, example below is for the Global Manager. For The local Manager just change the word global to local:
Now we can see a new CSR in the list, the next step is to self-sign the Global Manager CSR, select the CSR and under actions choose “Self Sign Certificate for CSR”
Choose your number of days:
Now we have a new Self-Signed certificate for the Global Manager in the certificates list, with this certificate we can replace the Principal Identity certificate for the Global Manager.
For the local manager certificates, follow the steps mentioned above on the local manager appliance.
Apply Self-Signed Certificate on the Global Manager
Before we can apply the SS certificate to the Global Manager we need to copy the certificate id, click on the ID, and then select the whole id in the pop-up and copy/paste it for later use:
Now we can Fire up Postman to apply the certificate by API:
Change the ACTION drop down to POST
Paste the following url to your Global Manager: Step 1 to 4
Set Authorization the same as the previous API Calls
Select Body and set it to none
If the Certificate is used by a node look for the used_by part, when there is a node_id, the certificate is still in use and can’t be deleted. If it is empty, you can delete the Certificate in the UI, you can do this check on the new certificate to see if it is used by the same node.
During a failover test with the Bare Metal Edges we ran into an issue when pulling the plug on 1 of the TOR switches. (TOR-LEFT). During that test all BGPs on both Bare Metal edges went down. So no North-South routing anymore 🙁
So why this behaviour? And what happens when we pull the plug on the other TOR switch (TOR-RIGHT). After performing the test with the TOR-RIGHT, the BGPs connected to TOR Left stayed established. So it has something to do with switch TOR-LEFT?
After checking the configuration on the TOR-LEFT switch we didn’t identified something that could cause this issue. But what could it be? Edges were configured by VMware guidelines and were identical configuration wise.
So going through the logs was the next step in the process, and i stumbled upon this part in the log file:
2022-10-17T10:37:08.578Z Update device fp-eth0 state to DOWN
2022-10-17T10:37:08.578Z Self Node 00363d34-fcdd-11ea-8e07-e4434ba66042 status changed from Up to Down (RTEP device down)
Can it have something to do with the federated setup (RTEP), is the RTEP only connecting over fp-eth0?
Again i went through the setup but now i also checked the fp-eth0 connections to the switches. On both BareMetal Edges the fp-eth0 was connected to the TOR-LEFT. So when we pulled the plug on that Switch it triggered the RTEP going down, which led to all BGP session going down.
This is expected behavior according to VMware!
The solution to this issue was pretty simple after we identified the cause. We switched the connection on the second Bare Metal Edge, so the pnics connected to TOR LEFT will be on TOR-RIGHT and vice versa. The opposite of the first Bare Metal Edge.
A while ago we ran into an issue after we did the upgrade from NSX-T version 126.96.36.199 to 188.8.131.52. In the alarms section at one Site. Still wanted to do a post about the issue and the solution/workaround:
Time to check the connection! Login to the Edges and grap the VRF id of the RTEP TUNNEL.
Check the BGP and ping between the RTEP ip addresses on both sites.
As you can see all BGPs are established and the ping commands give a reply. Let’s do another check from Postman:
Open Postman and fire a GET api call to the nsx-manager to grab the edge id we need in the next api call: API GET call:
Just select Basic Auth under the Authorization tab and fill in the Admin credentials.
Hit Send, when getting a reply in the Body, search for the edge name and the corresponding id.
Now we use this id to get the RTEP status:
Check the output and the Return Status for issues, as you can see in the example above the BGP to one of the peers is establised.
So it seems like the issue is known in the 184.108.40.206 version in a 3 manager nodes setup.
The node which has generated the alarm, only that node can clear alarm from in-memory when it will receive remove alarm from the edge node. The Alarm was resolved on 1 of the manager nodes, but it was showing on other nodes and it was keeping the alarm as active.
The following workaround will remove the alarm: Restart the proton service on ALL manager nodes.
– SSH with the admin user to the NSX-T manager nodes: – execute the following commands:
Stop service proton Start service proton UPDATE: The issue is fixed in version 3.2.1
Last week i was at the VMware Tech Summit in Cork, Ireland. I attended a session about NSX-T troubleshooting. During this session a lot of issues came to the stage which i dealt with in the last year or so.
One of the is issues was about the IP Bindings of a VM to a segment. In our case a tenant manually edited the Ipv4 address on the network Interface of the VM, and after this the connection to this VM dropped.
DFW checked, routing checked al was ok. After some digging we found out that the VM had 2 IP addresses in the realized bindings section on the logical switch. This view can be found in the manager UI of the segment port.
How can you find these Realized bindings and Fix it?
To get to the right port you can do the following, Go to segments and look for the segment to which the VM is connected. Once you found it click on the number you see beneath the ports.
This opens a window where you can find all connected port to the segment, copy the Segment Port Name.
Now search in the Search Bar for this Segment Port Name, and click the one with Resource Type Logical Ports.
This takes you to the Manager UI of this Logical port. You can always go through Networking -> then Switch to Manager UI in the upper right corner -> Select logical switches -> Search through the list for the right port.
Select Address Bindings, here you can see the Auto Discovered Bindings, both with the current IP from the VM. One learned from VMware Tools and the other by ARP Snooping. But if you take a close look at the Realized Bindings you can see a different IP learned by ARP Snooping. This was the original IP the Vm had when it connected.
This can cause connection problems! In our case the whole routing was messed up and the traffic went out via the wrong Uplink.
We can fix this quickly with moving the entry with the old IP address to the Ignore Bindings:
It will take a few seconds to updated the realized Bindings with the new lP address learned by ARP Snooping.
After this the connection came up and the tenant was happy! But this was nothing more than a quickfix, what if all tenants gone mad and they are manually changing their IP addresses in the OS…….
So why is the IP address staying in the Realized Bindings section and keeps bringing carnage?
By default, the discovery methods ARP snooping and ND snooping operate in a mode called trust on first use (TOFU). In TOFU mode, when an address is discovered and added to the realized bindings list, that binding remains in the realized list forever.
Can we modify that mode? Yes we can!
In NSX-T we use several profiles, one of those is the IP Discovery profile. This profile can be found in the Policy UI under Segments -> Segment Profiles
Create a new Ip Discovery Profile and disable the TOFU setting, When you do this, TOFU changes to Trust On Every Use (TOEU). In this TOEU mode, the discovered IP addresses are placed in the binding list and deleted when they expire. DHCP snooping and VMware tools always operate in TOEU mode.
Now we need to adjust the segment to use the new IP Discovery Profile, go to the segment and click edit. Under Segment profiles select the new TOFU Profile, click Save and the Close Editing.
Now when a tenant changes the IP of the Network Interface Manually the old IP learned the first time by ARP Snooping is not present anymore in the Realize Bindings section.
During an upgrade of NSX-T from 220.127.116.11 to 18.104.22.168 i came across an issue in the UI. When i clicked the update button, the screen was blank and was not showing any data, sometimes after a wait of half an hour or more the screen came through and i could proceed with the upgrade.
This is ofcourse not the way it should so i wanted to get rid of the issue.
Check Manager Cluster Nodes
First i wanted to check if all the cluster nodes were stable and the services were running AOK, so i ran the following command on all 3 cluster nodes:
get cluster status
All servers seem to be running fine, and didn’t show any anomalies, next i checked if there was maybe an old update plan stuck or something like that:
get node upgrade status % Post reboot node upgrade is not in progress
But no luck with that either, now i tested what if we start the Upgrade from another manager node.
For that to be possible i needed to execute the following command on the manager node we wnat to become the orchestrator node:
But after testing all nodes, no luck at all. The UI still gave me a blank screen on the Upgrade page.
Time to get support (Cause):
We raised an SR at VMware and within a few hours we got feedback. This issue was probably caused by an inconsistent Corfu DB, that was possibly triggered by an action we did in the past an re-deployement a Manager Node after a failure.
You can identify a possible inconsistent Corfu DB by an high EPOCH number that is increasing in the /var/log/corfu/corfu-compactor-audit.log
2022-05-27T10:53:35.446Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for fc2ada82-3ef8-335a-9fdb-c35991d3960c, entries(0), cpSize(1) bytes at snapshot Token(epoch=2888, sequence=1738972197) in 65 ms
2022-05-27T11:05:21.346Z INFO main CheckpointWriter – appendCheckpoint: completed checkpoint for fc2ada82-3ef8-335a-9fdb-c35991d3960c, entries(0), cpSize(1) bytes at snapshot Token(epoch=2921, sequence=173893455) in 34 ms
redeploy the manager nodes one-by-one…..
so here we go:
First we need to retrieve the UUID of the node we want to detach from the cluster.
get cluster status
Next run the command to detach the failed node from the cluster, from another cluster node.
detach node failed_node_uuid
The detach process might take some time, when the detaching process finishes, get the status of the cluster and check if there is indeed only 2 nodes present in the cluster.
Get cluster status
The Manager node is now detached, but the VM is still present in the vSphere Inventory, Power it down and Delete the VM. You can keep it ofcourse. But we are going to deploy a new Node with the exact same parameters, fqdn and ip. So best to disconnect the network interfaces in that case.
Now we can deploy a new Manager Node, we can do this in 2 ways.
1. From the UI
We can use the way if there is a compute manager configured where the Manager Node can be deployed.
Navigate to System > Configuration> Appliances and click Add NSX Appliance
Fill in the Hostname, IP/DNS and NTP settings and choose the Deployment size of the appliance. In our Case this is Large and click Next.
Next fill in the configuration details for the new appliance and hit next
Followed by the credentials and the enablement of SSH and Root Access, after that hit install appliance.
Now be patient until the appliance will be deployed on the environment.
When the new appliance deployed successfully wait till all services become stable and all lights are green, check the cluster status on the CLI of the managers with:
get cluster status
If all services are stable and running on every node, you can detach the next one in line and start over until all appliances are redeployed.
2. Deploy with OVA
When you can’t deploy the new appliance from the UI, you can build it with the use of the OVA file. Download the OVA file from the VMware website:
When the join operation is successful, wait for all services to restart.
You can check the cluster status on the UI select System > Appliances. and check if all services are up.
Check the cluster config on the manager nodes by running:
Get cluster config
When you have a inconsistent corfu DB, in some cases the redeployment of all manager nodes can be the solution. Be aware that you only detach 1 node and then redeploy the new one and so on. always keep 2 ore more nodes in the cluster to keep it healthy.
Recently I needed to resize the Edge VMs because we had a memory issue and the upgrade to NSX-T version 22.214.171.124 was not solving the issue. The already deployed Edge VMs were on Medium size which was not enough for the current load. The Edge VMs in the cluster are only used for T1 and have no stateful services and they are only being used for DR.
The current NSX version has 4 Form Factors, from Small to Extra Large. In this post we are going from a medium sized Edge Node VM to a Large Sized Edge Node VM.
The resize can also be done from edge Node VM to Bare metal or vice versa, although it is not advised to mix them in the same edge cluster.
Create a New Edge Node VM
Navigate to System > Fabric > Nodes > Edge Transport Nodes > Add Edge Node
Fill in the new name and FQDN, and choose for the Large Form Factor:
Click Next and fill in all credential details:
Again Click on Next and fill in the Deployment Details:
In the Next screen configure the Node Settings like IP, Gateway, DNS, NTP and Select the correct Management Interface. Please check if the new Edge VM is added to the DNS before deploying.
In the following screen we will configure the NSX Switches (nvds), keep the NVDS names identical to the other EdgeVMs. We encountered a BUG when selecting the interfaces during the configuration.
We have distributed Virtual Port groups configured as a Trunk for the uplinks of the Edge Node VMs. But for some reason these port groups were not showing up in the list. So as a workaround we selected a VLAN Backed Segment as a temporary fix, later on we will reset the Uplinks to the right interfaces.
Fill in the rest of the fields, key is to keep the configuration identical to the Edge Node VM we are going to replace.
Hit the Finish button and wait until the Node Status shows Up and the Configuration State shows Succes. The Edge Node VM is deployed Successfully.
Now we have to reconfigure the Uplinks of the NVDS to the correct distributed virtual port groups.
Select the newly deployed Edge Node Vm and hit Edit
As you can see the Uplinks are not configured to an interface, Select Interfaces for both Uplinks. Uplink-1 > EdgeVM-Uplink-1-Trunk01 Uplink-2 > EdgeVM-Uplink-1-Trunk02
But when we hit SAVE, we are getting an error:
Again we have a workaround to get around this issue, log in on another node in the NSX Manager cluster and try again to configure the right distributed virtual port groups to the uplinks of the Edge Node VM.
Hit SAVE , this time the config will be saved correctly. Before proceeding, compare the new Edge Node VM with the one you will replace.
Maintenance Mode Current Edge Node VM
Navigate to System > Fabric > Nodes > Edge Transport Nodes
Select de Edge Node VM you want to replace with the new Large Form Factor Edge Node VM.
Under Actions choose “Enter NSX Maintenance Mode”, wait until the Edge Node VM is placed in NSX Maintenance Mode. This will shutdown the Dataplane services on the Edge Node and trigger a failover for any Active SR on the Edge Node VM. This can cause a small disruption for the T1’s. Please check the connections after the failover.
Replace Edge Cluster Member
Now we are going to replace the Edge VM Node we put in NSX Maintenance Mode with the fresh deployed large Edge Node VM.
Navigate to System > Fabric > Nodes > Edge Clusters and select the Edge Cluster, click Actions and select “Replace Edge Cluster Member”
Choose the Edge Node VMs:
Hit SAVE and within a few seconds all configuration is migrated and The new large sized Edge Node VM should now replace the medium sized Edge Node VM and become a part of the Edge Cluster.
Check if all the DR & SR Constructs of T0 and T1 Gateways. SSH into the new Edge Node VM and run “get logical routers”
Check if the Edge Node is available for the T1’s. Navigate to Networking > Tier-1 Gateways and Select a T1 that is running on the New Edge Node VM. Select Auto-Allocated
Now you can see if the New Edge Node VM is running in active or standby for the selected T1. If the new Edge Node VM is standby, you can trigger a failover by putting the active Node in Maintenance Mode.
Traceflow can be used to check if the connection flows are indeed running over the ne Edge Node VM:
Check if all the tunnels are up and the Edge Node VM has no active alarms, the replacement was succesful!