SSP 5.2 – DFW 1-2-3-4 Hands-On

Uncategorized, , , , , , ,

In part 2 of my SSP series I explained the four stages of DFW 1-2-3-4 at a high level. In this post I go through every step in my demo environment, including the CSV files I used, with Security considerations in view. You can follow along if you have SSP 5.1 or newer.

Before you start

A few things to arrange first:

  • SSP is deployed and onboarded to NSX, and Security Intelligence is available. Log in with an admin account on the SSP FQDN. Logging in on the IP address doesn’t work.
  • Check your NSX version. CSV import for segments, Tier-0 and Tier-1 gateways needs NSX 4.2.1 or newer. Static IP-based applications need 4.2.2 or newer. Application member resolution for DVPGs doesn’t work on a few versions, among them 4.2.0 to 4.2.3 and VCF 9.X
  • Set your naming conventions first. Go to Monitor & Plan > Settings > General Settings and click Edit. Here you can set the asset delimiter (underscore by default, or a period or hyphen) and the prefixes for assets, policies and rules. These only apply to unpublished assets, so do this before your first import.

How to start the journey

There are two ways to proceed: click Begin With Security Segmentation Journey on the SSP home page, or use Journey in the top navigation. The journey page shows the four stages. When you finish all tasks of a stage, you click the round status indicator to mark it as complete.

Stage 1: Security Segmentation Assessment & Report

  1. Go to System – Platform & Features and activate Security Intelligence. If it’s already active, skip this.
  1. Wait at least an hour. The report needs flow data before it can calculate anything.
  2. Click Monitor & Plan – Security Segmentation Report to generate the report.

The score is based on the traffic flows of the last 30 days, which is the maximum. You can run the report again, for example after major infrastructure changes. I’d run it at the start and after each stage, so you can show the progress.

My score before:

The CSV import: how it works (used in stages 2, 3 and 4)

Stages 2, 3 and 4 all use the same import workflow, so I explain it once for Security clarity. It’s called Segmentation Planning and it has five steps:

Inventory in NSX before importing

Before the import of the CSV file in my lab environment there are no groups (tip: filter on generic, to create a better view)

Also in the Virtual machines Inventory in NSX there are no tags assigned to the virtual machines.

And under Tags, no tags created yet.

Preparing the CSV

Use the sample CSV from the UI as reference. What matters:

  • An asset identifier. This can be a name, IP address or UUID. You can use up to three identifier columns. They are evaluated in order, and the lookup stops at the first match. Each row needs at least one non-empty identifier. The default priority is asset ID, then IP, then name.
  • The hierarchy. You can use 1 to 5 columns for region, zone, environment, application and tier. If you only provide region, zone and environment, SSP doesn’t propose applications or tiers.
  • Optional columns: an asset type (VMs default if the column is missing) and infrastructure servers.
  • Keep it clean. No hidden characters and no byte order mark. The file can be up to 500 MB. Don’t give an asset multiple tags for the same scope.

One tip from the documentation I really agree with: test with a small CSV first. The scope hierarchy can’t be changed during an analysis, so plan it carefully. Below is a sample CSV we use to import in our lab environment. You can see applications, tiers, infrastructure services regions and zones added to virtual machines on the platform.

Importing the CSV

  1. Go to Monitor Plan – Policy Planning and click Select a File (or drag and drop the file).
  2. SSP parses the CSV and validates the format and data.
  3. Check the column mappings. On your first import, columns with headers that differ from the sample CSV stay blank and you have to map them yourself. On later imports, SSP remembers the mapping if the headers match the previous import.
  4. Click Preview.
  5. f there are errors, SSP gives you a link to an annotated CSV with the errors highlighted. Fix it and upload again. If you continue anyway, the incorrect data is left out.
  6. Click Start Analysis. The duration depends on the size of your file. When it’s done, open the Summary tab.

It is possible to add new columns to the CSV file and map them in the screen above to the right scope.

As you see in the screenshots above it uses all the information in the provided CSV and map those to the right scope.

Publishing

My advice, and the documentation’s as well: publish the inventory assets and tags individually first, check them, and only then use Publish all. Before using Publish all, make sure there are no tagging conflicts, otherwise the publication can fail.

In the summary you can find a quick overview of everything that is found, mapped and ready for publishing to NSX.

In the Publish Inventory Assets you can go through the individual groups like Infrastructure groups, Regions & Zones, Environments and Applications. Per tab you can review and publish them.

After publishing all Inventory assets the status should reflect Published with a green check mark

If you now go back to NSX and review the inventory , you will see that there are in my case 46 groups created

Great! Next we are going to check if Security Intelligence detected some Servers which correspond with the Infrastructure Services running.

Stage 2: Secure Infrastructure Services

The goal here is to find the legitimate infrastructure servers and block the rogue ones. SSP supports 14 infrastructure services, including DHCP, DNS, LDAP, NTP, SNMP and Syslog. It finds them in two ways: from your CSV and by looking at the flows.

In the previous step we already added the csv which had the infrastructure services in, so this step we can skip. Security Intelligence has ML built in to check all the flows that it gathers and can figure out which of the flows/vms belong to one of the infrastructure services we can map.

CSV Infra sample:

The column names and the way you indicate the service must match the sample CSV in your UI. Map the columns if they differ.

Under the “Publish detected Servers to Infrastructure Groups” tab we can see the system detected some servers corresponding with the LDAP service, which we not had added to the CSV file.

Because there is no existing infrastructure group for this service. Upon publishing, a global LDAPS infrastructure group will be created for the selected servers. and you can add the selected server to this group. Or just suppress this server if you don’t want to do anything with this system and ignore it. You can find suppressed servers under the Suppressed Servers tab later on.

Publish Distributed Firewall Policies for the Infrastructure Services

Under the Publish Distributed Firewall Policies and the Environment Rules you can review and publish the Firewall Policies that are suggested for the Environments, Zones zones and Regions. So up front there will be no policies for these in NSX under the Environment tab in DFW.

After publishing the policies are created:

Check the Policy Monitoring Tab in SSP and select Infrastructure Services. Here you get an overview of what is currently in place, how many services are detected, if there are incomplete rulesets or even no rules attached to a security policy. You can go through these Infrastructure services one-by-one or you can review and add all of them in once.

For DNS-UDP we have an Incomplete ruleset detected, lets generate some DFW rules to complete this ruleset.

Review the rules and hit Publish.

After this action, we can see that all rules are now in place for this Infrastructure Service

For the rest we are going to do them all in once, hit Review and Add Rules on the main screen of Policy Monitoring

In this screen you can review all rules and add Lockdown rules for all services if you want this level of security

In NSX the policies and rules are now updated under the DFW Infrastructure TAB.

Now we re-calculate the scores to see what this stage will give us:

Stage 3: Environments (zones)

Now we split the datacenter into broader zones, like Production, Development and Test. When you import the CSV, SSP pushes tags to the VMs so they become members of the right environment. It then creates groups and DFW rules for the traffic between the environments.

The CSV can come from a CMDB like ServiceNow, from vCenter, or from a spreadsheet you make yourself. For this stage I used three hierarchy levels:

In the previous step we already added the csv which had this in formation in it in, so this step we can skip. The Hierarchical levels are created with tags.

Publish Distributed Firewall Policies for the Environment

Under the Publish Distributed Firewall Policies and the Environment Rules you can review and publish the Firewall Policies that are suggested for the Environments, Regions and Zones. Check if this reflects with your CSV and environments. In this case all the logic will be done in the application layer of the DFW. because of the Jump to Application action.

So up front there will be no policies for these in NSX under the Environment tab in DFW.

After publishing:

Check the Policy Monitoring Tab in SSP and select Environments. Here you get an overview of what is currently in place, how many environments and if there are inter-environment rules, if there are no protection rules in place and if there is traffic detected inter-environments.

As you can see we have no inter-Environment policies yet configured between the environments, we have 2 views here, Grid view and the List view. The list view is familiar and looks like the other screens presented earlier. From here we can run traffic based recommendations (select under Action menu) or add a Policy directly to a environment pair.

In the Grid view you can hover over a tile to see the configuration and click on it to see the actions, the same as in list view but i find the grid view a great first view to check the iterator environment policies.

If you select Add policy you get thepolicy and a protection rule to monitor the traffic between the environments.

When you select Run recommendation, you can run a recommendation based on the traffic between the 2 environments, and review and proceed with these recommendations in the policy recommendations tab

In this way you can easily protect your inter-environment traffic with policies and rules. And constantly monitor the traffic between those 2.

After putting the inter-environment rules in place the matrix view looks like this:

All Inter-Environment is now segmented, the flows between prod <-> prod and dev <-> dev will be handled in the application layer because Jump–to–application is selected. Inter-environment traffic between other environment like dev <–> prod, pairs will be blocked.

When you want to set the default Environment Category rule action you can change this easily.

Again we re-calculate the scores to see what this stage will give us:

Stage 4: Application micro-segmentation

This is the final stage, and it’s done in two phases: first ring-fencing to define the controls between applications, then the controls between the tiers inside each application (web, app, database).

For this I use the full hierarchy, with the application and tier columns added:

My test file:

Steps:

  1. Import the CSV, check the hierarchy in the preview and start the analysis. SSP creates application groups, which are also used for the environment exception policies from stage 3 (already done)
  2. Publish the inventory assets (already done)
  3. Monitor the applications on the application monitoring page. SSP follows the flows of each application, before and after you publish rules.
  4. Run the policy recommendations for the applications. Start with the ring-fencing rules, then the tier rules.
  5. Review the recommended rules, publish them, and keep monitoring. Fine-tune where needed.

Ass you can see in the screenshot below on the application monitoring page. SSP follows the flows of each application, before and after you publish rules. At this moment we have 8 applications, no application is marked critical, no rule hits detected and no policies applied. Let’s get to work.

Let’s pick the first application, we can see that we have around 131 unique flows, those are all east-west and is split between Incoming and outgoing. Also a few Intra application flows are detected. You can also see that SSP already have 3 Unpublished recommendations available for this application. And in the chart you can see that most of the flows are unproitected at this point. The ones that are already protected are the Infra and Inter-Environment flows.

Let’s see what recommendations we have, these recommendations where run during the environment step as you can see on the Category mentioned.

When you click on the icon on the right, you can switch to canvas view to view the flows in a canvas. Here you can filter between applications(as shown below) or Groups, or compute (ip addresses, etc). You can adjust the time range of which flows you want to use. With the buttons on the right bottom corner you can tweak your view. Zoom in/out, drag & drop, etc.

Back to the recommendations, we need to start a new recommendation for the Applications

We need to select the entity we are runnin g this recommendation against and the Time range for traffic where the recommendation will be based on.

Lets select our application, hit SAVE and NEXT to go to select the Segmentation strategy (optional).

When you run the policy recommendations for an application, SSP asks you to choose a strategy to auto-plumb rules.
Two things on that screen are worth knowing before you click anything. The auto-plumbed rules are added at the bottom of the new policy. And once you publish them, the protected traffic is excluded from future traffic-based recommendations. So your choice decides which traffic you close with a default action, and which traffic SSP keeps watching to generate granular rule recommendations.

There are three strategies:

Intra-application trafficIncoming and outgoing trafficDefault rule
RingfencingAllowed by defaultMonitored, for granular recommendationsAllow (you can change the action)
MicrosegmentationMonitoredMonitoredAllow (you can change the action)
CustomYou chooseYou chooseYou choose

Ringfencing

SSP allows the traffic inside the application by default, and starts monitoring the incoming and outgoing traffic to generate granular rule recommendations for it. Inside the application nothing changes, so the tiers keep talking to each other as before. Only the boundary of the application gets attention. This is the ring-fencing phase of the journey, where you define the controls between applications.

Pick this one when:

  • It’s your first pass at an application, especially in a brownfield environment where you don’t know exactly how the tiers talk to each other.
  • You want to control what goes in and out of the application, without risking the traffic inside it.
  • You want a low-risk start. In my demo this was the strategy marked as Strategy Currently In Place.

Microsegmentation

SSP monitors all the traffic: incoming, intra-application and outgoing. It then generates granular rule recommendations for all of it, including the traffic between the tiers (web, app, database). This is the second phase of the journey, where you establish the controls across the tiers inside the application.

Pick this one when:

  • The ring-fencing is in place and you want to go further, to tier-to-tier rules.
  • The application is sensitive (think card data or personal data) and “everything inside is allowed” isn’t good enough.
  • You have enough flow data for the application. More monitored traffic means more recommendations to review, so plan some time for it.

Custom

With Custom you decide per traffic type. The options are incoming traffic, outgoing traffic, intra-application traffic and the default rule. For the types you select, you choose to allow or block them by default. The remaining flows are monitored to generate granular rule recommendations.

Pick this one when neither of the two fits. Some examples:

  • An application with many outgoing connections you don’t want to model rule by rule (for example updates or external services). You allow outgoing by default, and keep monitoring the incoming and internal traffic.
  • A tier that should never receive traffic from outside the application. You block incoming by default and keep monitoring the rest.

My advice

Start with Ringfencing, and move to Microsegmentation per application when you’re comfortable with the flows. Use Custom for the exceptions. In all cases, look at what the default rule does before you publish. With the default rule on Allow, traffic that no specific rule covers still goes through, which keeps the application working while you’re learning. Switching a default to block is the moment things can break, so do that after you validated the recommendations.

In the next screen, there’s one more choice when you start a recommendation: the Recommendation Service Type. It decides what kind of rules SSP generates. L4 Services is the default.

L4 Services

The recommended rules are based on ports and protocols, for example TCP 443. SSP aggregates the flows by the service (the port or protocol) they used, so you get one rule per service for the sources and destinations that use it. It’s simple to read, easy to review, and it’s what most DFW rules look like.

L7 Context Profiles

Here the recommended rules can also contain a context profile. A context profile holds one or more Layer 7 App IDs, so the rule doesn’t only say “TCP 443”, it says which application is allowed on that port. NSX has built-in App IDs for common infrastructure and enterprise applications, like SSL/TLS and CIFS/SMB.

How SSP decides, as described in the documentation:

  • For flows that have an App ID, SSP creates a rule with the matching context profile, as long as it can identify that profile from the App ID. SSP does not create new context profiles. If the profile doesn’t exist, you get an L4 rule for that flow instead.
  • For flows without an App ID, SSP creates an L4 rule.
  • When SSP modifies an existing rule that already contains context profiles, it keeps those profiles. It also matches existing rules on the same port, protocol and App ID.
  • If no L7 context data is available for the analysis, you get L4 rules, even though you selected L7.

So expect a mix of L7 and L4 rules in the result, not only L7 rules.

Which one to pick?

L4 ServicesL7 Context Profiles
Rule is based onPort and protocolPort, protocol and App ID
NeedsNothing extraFlows with App ID data, and context profiles that exist in NSX
Best forStarting out, ring-fencing, well-known portsPorts that can carry more than one application, and stricter control

My advice:

start with L4 Services. It’s the simplest and it gives you a working rule base fast. Use L7 Context Profiles where the port alone isn’t specific enough, for example when you only want a certain application on port 443 and not anything that happens to use TLS. Keep in mind that L7 has a cost. In NSX, a packet that matches a rule with an L7 context profile is sent to the DPI engine to determine the App ID, which means more processing than a plain L4 rule. The NSX design guidelines also recommend combining a context profile that has a single App ID with the matching ports in the L4 service field. So use it where it adds value, not everywhere.

In the same screen you have a few other otions like excluding protocal and ports and the option to separate the rules by the service. When enabled, each service gets its own separate rule even if they share the same src and dst. We leave them disabled for now and hit Start Discovery. The recommendation will be started and can be found under Policy Recommendations.

Let’s click the 3 ellipses and Review and Publish the recommendation!

You can click the Show graph to see an graphical overview of the recommendation, and in the list below you can review Rules and Groups that are proposed. Review this and if you are alright with the recommendation you can hit Proceed. if you want to change an action for a specific Service or rule you can simply adjust the Action. This will also immediatly reflect in the graphical overview.

In the next screen you can rearrange the sequence of the rules, Recommended Rules are placed at the top of existing Firewall rules by default.

After publishing the policies we can see in the recommendation overview that it is Published:

In NSX, we can see that the policy and rules are published and part of the Application layer of the DFW. You can see the policy name is the name you had set in the recommendation. this name is easily adjustable in the UI.

Just click on the name and change it, in our case to peoplesoft and publish the firewall change.

Now when we check the Policy Monitoring view in SSP again, and check on the Application we can see in the graph we went from a few protected flow to all flows protected for this Application.

Now for another application we will run a recommendtation and select L7 services instead of L4.

The change is when we review and Publish this recommendtaion is that we now have Profiules added to the policy based on the flow data. In this case MYSQL and SSL traffic.

We will select different strategy throughout the applications and when all recommendations and strategies are publshed we will run the report again to see how we did.

Run the report again

Generate the Security Segmentation Report again, like in stage 1, and compare the score.As you can see we are not quite there yet. The last step for this is to Lockdown the applications, like we did earlier at Infrastructure level.

So once ready, update the Segmentation Strategy and switch the default rule action to ‘Drop’ to block all unauthorized traffic. As you can see below for both Infrastructure services and applications the lockdown rules are now in place.

When we re-calculate the score again we have a big hit

Now we can see that we can win extra point with enabling the Malicious IP rules on the DFW level.

The Malicious IP rules are the two default rules that NSX/vDefend adds to the Distributed Firewall. They block traffic to or from IP addresses that are known to be bad.

How they work

Malicious IP Filtering blocks network traffic to or from known malicious IP addresses, such as those used by hackers, botnets or malware servers. The system downloads the IPs from the vDefend Threat Intelligence Service (VTIS) cloud service and puts them in a malicious IP group. The default policy, called Default Malicious IP Block Rules, contains two rules: one that matches the malicious group at the source and one at the destination. They sit in the Infrastructure category. Malicious IP Filtering – Broadcom TechDocs +3

A few details that matter in practice:

  • The feed is downloaded every 12 hours once auto-download is on. You can also trigger a download by hand.
  • If auto-update is off, rules with malicious IP groups can be enforced with outdated data. broadcom
  • The feed is IPv4 only. broadcom
  • You can create custom malicious IP groups for exceptions that must not be blocked, and you can build your own rules with malicious IP groups. broadcom

Why use them

  • Low effort, real value. It’s a baseline you get without modeling any flows. Every workload gets protection against reaching known botnet and malware infrastructure, enforced on the host rather than on an appliance. In my view, this makes it a good first layer for outbound command and control traffic.
  • Automatic updates. You don’t maintain the list yourself.
  • Visibility. There’s a Malicious IPs Filtering and Analysis dashboard where you can see which IPs were blocked. broadcom

Why be careful

Limited coverage. It only knows IPs that are already flagged, and only IPv4. It’s not a replacement for segmentation or for IDPS and NDR. Also, the Manager has to be able to fetch the feed from the cloud service, so in isolated environments check how the updates work.

Licensing surprise. The feature isn’t entitled under NSX Enterprise Plus. If you add a vDefend Firewall or vDefend Firewall with ATP license to an environment that started on Enterprise Plus, the feature is enabled automatically and starts blocking traffic to flagged IPs. Customers get caught by this. broadcom

False positives you can’t override from below. Reputation feeds sometimes flag a destination you need. Broadcom has a KB about traffic being dropped despite user-created allow rules, caused by the destination being in the malicious IP database. Because the rules sit at the top of the Infrastructure category, your own allow rules further down the table don’t help. The fix is to add the IP to the exception list, or to put an allow rule for the known-good destinations above the Default Malicious IP Block Rules. broadcom

Troubleshooting confusion. If a team reports “the firewall drops my traffic” and all your own rules look fine, the default malicious IP rules are an easy thing to miss. Packet drops under Default Malicious IP Block Rules are the sign that the feature is active. broadcom

Final Score Calculation

Lessons learned

My tips, based on the documentation and my demo:

  • Start with a small CSV, and with VMs you know.
  • Fix your naming conventions before the first import.
  • Keep your hierarchy consistent. Don’t give an asset more than one tag for the same scope.
  • Publish assets and tags step by step. Use Publish all only when there are no tagging conflicts

Thanks all for reading if you made it to the end of this very lenghty blog post…..and watchout for more SSP goodness coming up.

READ MORE

SSP 5.1: DFW 1-2-3-4, a guided path to microsegmentation

vDefend, , , , , ,

This is part 2 of my SSP series. In the first post I explained what SSP is. SSP 5.1 arrived together with VCF 9.0, and one of its big additions is DFW 1-2-3-4. It’s a feature of Security Intelligence, available through SSP 5.1.

What is DFW 1-2-3-4?

It’s a guided workflow that takes you through segmentation in stages, following the lateral traffic patterns in your environment. Under the hood an analytics engine discovers communication patterns, identifies unprotected traffic and recommends firewall rules.

The idea is to prevent you from jumping straight to application-level microsegmentation without the visibility you need. Instead you start with the quick wins and work towards the application level.

The four stages

1. Security Segmentation Assessment & Report

You generate a Security Segmentation Report that shows your current posture. The segmentation score re-calibrates automatically when your environment changes, so you can track progress over time and show it to management or auditors.

2. Infrastructure (shared) services

Protect DNS, NTP, Syslog, SNMP, DHCP and LDAP first. The workflow discovers these services for you, or you can feed your known endpoints through a CSV file. Locking down DNS alone already removes some of the most common command and control and exfiltration paths.

3. Environment (zone) segmentation

Define zones like Dev and Production. You can import the metadata from a CSV, for example exported from a CMDB like ServiceNow or from vCenter. The workflow assigns the security tags, creates default zone-level rules and monitors for traffic leaking between zones.

4. Application microsegmentation

This stage has three steps:

  • 4a. Map workloads to applications, again via CSV. The workflow auto-tags them and creates the application groups.
  • 4b. Define ring-fencing for each application. The system recommends rules that only allow communication between the permitted entities.
  • 4c. Continuously monitor the applications, before and after publishing rules, and fine-tune the rules between the application tiers.

My take

VMware says a typical rollout can be done in as little as a few weeks. I’d treat that as a best case, because the CSV input (zones, workload-to-application mapping) depends on how good your CMDB data is. But the structure alone is useful, especially in brownfield environments where nobody knows exactly what talks to what.

READ MORE

What is the Security Services Platform (SSP) in VMware vDefend?

vDefend, , , , , , , , , , ,

When I started working with vDefend, I quickly noticed that SSP is often treated as “that thing you also have to deploy”. In reality it’s the foundation for a big part of the vDefend feature set. This is the first post in a small series about SSP. Here I explain what SSP is and what runs on it. In the next posts I’ll go through what’s new in SSP 5.1 and 5.2.

What is SSP?

SSP is a high-performance, scalable platform that runs vDefend features like the Security Segmentation Report and Score, Security Intelligence, Metrics, Network Detection and Response, Malware Prevention and Network Traffic Analysis. Think of it as a scale-out data lake that ingests network flow records and telemetry. On top of that data it gives you flow visibility, security assessment scores, policy recommendations and guided workflows.

Under the hood it’s Kubernetes. You deploy it with the SSP Installer (SSPi) appliance, and once it’s running you can follow what’s happening with plain kubectl commands. I used that in a previous post to follow an NSX onboarding.

What doesn’t run on SSP?

This one confuses people. The enforcement itself stays in NSX. Distributed Firewall, Gateway Firewall and the malware policy and service VM management are still done through the NSX API. So your firewall keeps working without SSP. What you miss is the analysis, the guided workflows and the advanced threat prevention on top of it.

Why a separate platform?

Processing flow data from a whole private cloud needs its own scalable layer, separate from the NSX management plane. SSP is a self-contained, scale-out platform with a simplified network design and streamlined lifecycle management. That makes deploying Security Intelligence, NDR and Malware Prevention a lot easier than it used to be.

Related

Resolve SSP 5.1.1 Errors During NSX Onboarding

READ MORE

Resolve SSP 5.1.1 Errors During NSX Onboarding

NSX-T, Security, vDefend, , , , , , , ,

During the deployment of SSP (Security Services Platform) 5.1.1, I ran into an issue while onboarding the NSX instance.

This error means that there is still an old SSP installation onboarded in NSX. In my lab environment this was the case and the old SSP installation got corrupted due to storage issues and could not get correctly ofboarded and decommisioned. Luckily there is a kb that can help you clean up NSX and SSP.

https://knowledge.broadcom.com/external/article/382295/offboarding-a-site-during-force-deletion.html

This kb explains that a scripts needs to be ran at SSP version 5.0 or 5.1 to get the onboarded NSX sites properly offboarded.

  1. Download the right script for the version of SSP
  2. Upload the script to the SSPi, Installer appliance of SSP.
  3. Login via ssh to the SSPi appliance as sysadmin user (for SSP 5.1) and make the script executable (chmod 777)

Next you can run the script:

Example usage:
./site-offboarding-cleanup-nsx-ssp5.1.sh [-h|--host <mgr_host_ip>] [-u|--username <username>] [-p|--password <password>]
site-offboarding-cleanup-nsx-ssp5.1.sh

./site-offboarding-cleanup-nsx-ssp5.1.sh -h 10.10.10.0 -u admin -p p@ssw0rd

After running the script the NSX site cleanup is complete, and you can go try to onboard NSX again to SSP. During the onboarding you can follow the onboarding process:

k get site -A

k describe site 098ad7b9-cdea-46b8-82a7-1f2e3d1fd366  -n nsxi-platform

When you see “OnboardingInProgress”, you just have to hold on a little bit until it says “OnboardingComplete”.

In the SSP you now shall see that NSX is correctly onboarded!

READ MORE

Aria Ops for Networks deployment fails from Aria Suite LifeCycle Manager.

NSX-T, VCF, , , , , , , ,

In my lab I tried to deploy Aria Operations for Networks 6.12.1 (AON/vRNI) from Aria Suite LifeCycle 8.16 (ASLC/vRLCM). Before the deployment of AON I successfully deployed other products:

vIDM 3.3.7
Aria Operations for Logs 8.16 (vRLI)
Aria Operations 8.17.1 (vRops)

Attempted to deploy Aria Ops for Networks 6.12.1 but it failed with LCMVSPHERECONFIG1000016 error.  

-----------------------------------------------------------------------------------------------------------
java.io.IOException: com.vmware.vim.binding.vmodl.fault.SystemError
	at com.vmware.vrealize.lcm.drivers.vsphere65.vlsi.utils.ExceptionMappingUtils.mapAndThrowImportVAppExceptions(ExceptionMappingUtils.java:78)
	at com.vmware.vrealize.lcm.drivers.vsphere65.deploy.impl.BaseOvfDeploy.importOvf(BaseOvfDeploy.java:713)
	at com.vmware.vrealize.lcm.plugin.core.vsphere.tasks.DeployOvfTask.execute(DeployOvfTask.java:251)
	at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
-----------------------------------------------------------------------------------------------------------

Error Code: LCMVSPHERECONFIG1000016
IO Exception occurred while performing the operation. Check the logs for more information.
Unexpected ioexception occurred.

——————————————————————————————————–

After some scavenging around, I ran into the known issues in the Release Notes from version 8.14

So let’s check which user is being used between the ASLC and vCenter:

Now login the vCenter and check if the privileges assigned to this account match the 2 from the release notes:

As you can see above the privileges are not configured on this role, let’s configure the 2 privileges:

Now we need to resync the account between vCenter and ASLC.

After this I Hit the retry button on the request in ASLC and the ovf is being deployed in vCenter.

READ MORE

Compute Manager not visible in NSX after Upgrade of the 3 Manager Appliances

NSX-T, , ,

After a successful upgrade of NSX, after the last step the upgrade of the management plane the compute manager disappeared, let’s see how we can fix that!

When i try to add the vCenter it says it is already registered, let’s check with the API.

First do a API GET in Postman to get the compute manager id:

Output:

Now we have the compute manager id, we can check if it is registered and up:

Output:


As you can see the compute manager is registered and up, why is it not showing up in the UI?

Solution:

Login with the admin user by SSH, and run the following command.

start search resync inventory

Wait a few seconds and refresh the UI, now the Compute Manager is back!


READ MORE

Test Case – Cloud Director NSX-T Role Based Access

Cloud, Cloud Director, NSX-T, , , , , ,

Recently i got the question if in one Cloud Director Tenant (Organization) Granular Role Based Access Control and separation of rights can be configured between multiple teams within that Organization.

Details:

In our Test case Team-A is responsible for Org VDC A and can only manage and view the Edge GW resources (networks, Edge Gateways) within that VDC-Group. Team-B responsible for Org VDC B and can manage and view the all resources in all VDC-Groups, Except the Edge GW in Org VDC A. This Edge GW can only be managed by the Team-A.

Also because of some Tenant requirements the T0 (VRF) is also split between Internet and Customer specific. You can read more about this setup in an this post.

Requirements:

  • Separation of Rights between Org VDCs
  • Shared networks between Org VDCs
  • Team-A Can only manage the Edge GW in ORG VDC A
  • Team-B can manage all resources in both ORG VDCs except the Edge GW in ORG VDC A

Setup:

  • One Provider VDC (vCenter)
  • One Organization in Cloud Director (Tenant1)
  • Two Org VDC connected to the same Provider VDC (ORG VDC A & ORG VDC B)
  • Two Data Center Groups (VDC Group A & VDC Group B)
  • Two Edge GW (Edge A connected to VDC Groupp A & Edge B connected to VDC Group B)
  • Tenant Acces Role Team-A
  • Tenant Acces Role Team-B

Datacenter Groups

From version 10.2, VMware Cloud Director supports Data Center Group networking backed by NSX-T Data Center.

A Data Center Group acts as a Cross-VDC router that provides centralized networking administration, egress point configuration, and east-west traffic between all networks within the group. 

Using Data Center Groups, we can share organization networks across various ORG VDCs. To do so we first group the virtual data centers, then create a VDC network that is scoped to the Data Center Group.
A data center group can contain between one and 16 virtual data centers that are configured to share multiple egress points. 

We need to created 2 Data Center Groups and connect them to the participating VDC & Edges

VDC Group A -> ORG VDC A (24-2 in picture below) & Edge A
VDC Group B -> ORG VDC B (24 in picture below) & Edge B

Roles

By default, organization VDCs are shared with all users and groups that have a role which includes the Allow Access to All Organization VDCs right.

As an Organization Administrator, you can limit the access to each of the organization VDCs in your organization to specific users and groups.

Our organization has multiple organization VDCs and we want to have them managed separately, so create a custom role that would function as an organization VDC administrator and assign it to specific users or groups within your organization, providing them with access only to a specific VDC’s compute and networking resources.

For Team-B we can use the predefined role Organization Administrator. In this role the following right is allowed: Allow Access to All Organization VDCs

This permission is exactly what is tells you it does, giving you permission to ALL Organizations VDC in the Organization. So with this role we are able to manage VM’s, networks, etc in all the Organization VDC’s.

Exactly what we need for Team-B.

For Team-A we need to make a new role with more granular permissions. Create a new role and exclude the Allow Access to All Organization VDCs right. Set the rest of the permissions to view and manage the Edges and Networks.

Publish both Roles to the Tenant and create 2 users in the Tenant.

Limit Access to ORG-VDC

Now we need to limit the access to the Org VDC, on the Virtual Data Center dashboard screen, click the card of the virtual data center that you want to limit access to.

Under Settings, click Sharing, The list of users and groups within the organization that have access to the VDC appears. To change the access settings to the organization VDC, click Edit.

Select Specific Users and Groups, From the Users list, select the users that you want to provide with access to the VDC, same procedure if you are using groups.

So for ORG VDC A select Team-A, Team-B already has access to all ORG VDC because of the Allow Access to All Organization VDCs right.

To share the VDC with the selected users and groups, click Share.
At this moment Team-A can only view and manage Edges and Networks in ORG VDC A, and Team B can view and manage all resources in both ORG VDCs, also the Edge in ORG VDC A. if we need to get that sorted out we need to created roles and groups for every thinkable Resource set (Edge Admin, VM Admin, DFW Admin, etc) in every ORG VDC.

But another requirement in this Test Case is the Shared Network between ORG VDCs.
For this requirement it is needed to add the other VDC to the participating VDCs in the Data Center Group.

As soon as you configure this Team-A (which can only see the Edge-A in ORG VDC A), immediately can see the Edge-B under ORG VDC B. A no go in our case. The rights are distributed very horizontal. So as soon as you have multiple participating VDC in your Data Center group the team that was restricted to viewing and managing the resources in ORG VDC A, can now also view and manage the resources it is permitted to in ORG VDC B.

Conclusion

For this Test Case the outcome was negative as we needed shared networks between ORG VDCs.
If the sharing of networks is not needed, you set a very Granular RBAC model, but keep in mind when you set the Allow Access to All Organization VDCs right in a role, the users/groups that have this role are allowed to show all resources they are eligible to in All Organization VDCs

READ MORE

Cloud Director NSX-T – Tenant Design Scenarios with 2 Edge Gateways

1 CommentCloud, Cloud Director, NSX-T, , , , , ,

Half way december i switched to another team at my current employer, and got my hands dirty with Cloud director, NSX-T and AVI. As this was my first real hands-on with VMware Cloud Director.
I was given the task to investigate some scenarios in which a tenant is given a second Edge Gateway, for the separation of Internet and Customer Networks traffic.

Before describing the scenarios, I’ll assume you have basic NSX-T, routing and VCD knowledge.

Current Tenant Setup

In the current setup a tenant is given:

  • Cloud Director (10.3): Organization (example: Tenant1)
  • Cloud Director: Org VDC
  • NSX-T (3.2): VRF based tier-0 gateway for both Internet VRF and Customer VRF connected to Parent T0
  • Cloud Director NSX-T: 1 Edge Gateway (T1)
  • Clloud Director NSX-T: 1 VDC Group
  • If Load balancing is used: dedicated AVI Service Engine Group

Because there is a 1:1 relation between the T0 and the Edge Gateway (dedicated T0), route advertisement of connected tenant networks is available.

The traffic for both the customer networks and Internet is flowing through the same Tenant Edge gateway and VRF based tier-0 gateway.

Scenario 1 – 2 Edge Gateways Connected to 1 Shared T0.

The first scenario I tested is based on a Shared T0 (in VCD) and 2 Edge Gateways (1 for Internet and 1 for Customer Networks).

Setup:

  • Cloud Director: Organization (example: Tenant1)
  • Cloud Director: Org VDC
  • NSX-T: VRF based tier-0 gateway for both Internet VRF and Customer VRF connected to Parent T0 (parent T0 is shared between 5 Tenants)
  • Cloud Director NSX-T: Set T0 to shared
  • Cloud Director NSX-T: two Edge Gateway (T1) connected to the same T0 (shared)
  • Cloud Director NSX-T: two VDC Groups (Data Center Groups)
  • If Load balancing is used: Shared AVI Service Engine Group (dedicated as described in current setup can also be used).

The downside of a Shared T0 is that route advertising of Tenant network connected to the Edge Gateway (T1) isn’t available to the T0.

Tenant VMs can connect to the internet or customer networks by using NAT and firewall rules.
SNAT rules need to be created for outbound traffic and DNAT rules inbound traffic.
Only use private IP spaces can be used for tenant networks connected to the Edge Gateway (T1).

Also because we use two Edge Gateways in an Org VDC we need to create 2 VDC Groups (Data Center Groups), because there is a restriction of 1 Edge Gateway per VDC Group (Data Center Group), two Datacenter Groups also mean 2 Distributed Firewalls to manage.

If you need connection between a VM in VDC Group 1 and a VM in VDC Group 2, you can create a network that will be spanning across both VDC groups.

In this scenario we still have the BGPs for both Internet and Customer configured on 1 VRF based tier-0 gateway, so this is not completely dedicated. And it is giving a lot of extra effort configuring the SNAT and DNAT.

We also worked out a scenario if the tenant is using AVI. In the normal setup the tenat is given a dedicated Service Engine Group. We also discovered the option to share the Service Engine Group between the Internet and Customer Networks. This option is a valid solution as the AVI per Default configures separates the traffic based on VRF in the Service Engines.

Scenario 2 – 2 Edge GW with dedicated T0

For the 2nd scenario we decoupled also the VRF Based T0 and creating again a 1:1 relation (dedicated T0) between the Edge Gateway and the VRF Based T0’s. Because of this 1:1 relationship route advertisements are available to the T0.

Setup:

  • Cloud Director: Organization (example: Tenant1)
  • Cloud Director: Org VDC
  • NSX-T: VRF based tier-0 gateway for Internet VRF connected to Parent T0
  • NSX-T: VRF based tier-0 gateway for Customer VRF connected to Parent T0
  • Cloud Director NSX-T: Set T0 to be dedicated
  • Cloud Director NSX-T: two Edge Gateway (T1) connected to the dedicated T0
  • Cloud Director NSX-T: two VDC Groups (Data Center Groups)
  • If Load balancing is used: Dedicated AVI Service Engine Group (because no design Change is needed for current model)

If you need connection between a VM in VDC Group 1 and a VM in VDC Group 2, you can created a network that will be spanning across both VDC groups.

The downside of the Scenario is that there are a lot of extra resources needed, and these resources will be billed to the tenant.

Conclusion

We still wanted a Design that met the customer requirements and also is easy to implement in the current setup. Which already in use by several tenants and was the default tenant setup when the platform was initially designed.

In Scenario 1 the Shared setup of the shared VRF based T0 doesn’t met the Customer requirements of separating Customer Networks and Internet traffic. Also the lack of the route advertisements was an issue for the Tenant.

In Scenario 2 where we separated all parts of the setup, all the Tenant requirements were met. The downside of this Scenario is that because all parts of the setup are decoupled, the resources a tenant needs using this setup are doubled. This means:

  1. Double Billing
  2. Double Resources, check the Config Maxs
  3. Increases operational effort

Scenario 2 is basically a copy of the current scenario, so the initial setup will cost less time, where Scenario 1 will have some design changes (shared T0, no route advertisments, SNAT/DNAT).

This design was based upon some requirements of a tenant, i tried to take in account that we also van use this for other tenants with the same requirements.

I loved investigating these scenarios, and discovering Cloud Director! Watch out for the next post about a Test Case with RBAC on Cloud Director!

Thank you for reading!

I also would like to mention the following blog article that helped me creating this post, Thanks Daniël!

https://blog.zuthof.nl/2020/06/30/cloud-director-nsx-t-inter-tenant-routing/


READ MORE

Replace NSX-T expiring localmanager and globalmanager self-signed Certificates.

NSX-T, , , , , , ,

Recently we saw some warnings about expiring certificates in the NSX-T Global Manager and Local Manager.

When we clicked one of the alerts we got a small description and some API calls we can fire to apply new certificates.

In the Certificates overview (System > Certificates > Certificates), we could see that the certificates Issued to the Local Manager and Global Manager were expiring. The certification id’s were also corresponding to the ones in the alert (not the ones in my screenshots).

The API calls that were mentioned in the Alert description are for the renewal of certificate to the HTTP service (UI), not the Local/Global Manager certificates.
The VMware Docs don’t explain in good detail how to change these certificates, i couln’t find it.

Try it yourself:
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

The only give away i could find was in step 6: (NSX Federation and the service type).

So before we can replace the certificates, we need to create new Self Signed Certificates for the Local Manager, and the Global Manager.

Create CSR on GM/LM:

to create a CSR (Certificate Signing Request) on the Global or Local Manager go to: System > Certificates > CSRs and click on “Generate CSR“.

For the Global Manager do this via the Global Manager Appliance and for the Local Manager use the Local Manager Appliance, or use the drop down on the top of the screen to choose between your Global Manager or Local Managers.

Fill in all the fields and hit the GENERATE button, example below is for the Global Manager. For The local Manager just change the word global to local:

Now we can see a new CSR in the list, the next step is to self-sign the Global Manager CSR, select the CSR and under actions choose “Self Sign Certificate for CSR”

Choose your number of days:

Now we have a new Self-Signed certificate for the Global Manager in the certificates list, with this certificate we can replace the Principal Identity certificate for the Global Manager.

For the local manager certificates, follow the steps mentioned above on the local manager appliance.

Apply Self-Signed Certificate on the Global Manager

Before we can apply the SS certificate to the Global Manager we need to copy the certificate id, click on the ID, and then select the whole id in the pop-up and copy/paste it for later use:

Now we can Fire up Postman to apply the certificate by API:

  1. Change the ACTION drop down to POST
  2. Paste the following url to your Global Manager:

    Step 1 to 4

    5. Change to Body, select raw and from the drop-down choose JSON
    6. put the following JSON in the Body:

    {
    “cert_id”: “<new cert id>”,
    “service_type”: “GLOBAL_MANAGER”
    }

    And hit the SEND button, if you get back Status 200 OK, the API call was succesful:

    Apply Self-Signed Certificate on the Local Manager

    This procedure is almost the same as on the Global Manager. Go to the Local Manager Appliance and copy the new certificate id.

    Go to Postman:

    1. Change the ACTION drop down to POST
    2. Paste the following url to your Global Manager:

      Check the old expiring certificate

      Now since we are on version 3.1, we need to check by API if the old certificate is released so we know it isn’t used anymore before we delete it.
      Note: In Newer version you have a “Where Used” field in the certificate overview.

      Go to Postman:

      1. Change the action to GET
      2. Paste the following url to your Global Manager: (use the id of the expiring certificate)
        https://<global-manager>/api/v1/trust-management/certificates/dee8d78b-5e04-4deb-8d36-6b86f79f058b
        Set Authorization the same as the previous API Calls
      3. Select Body and set it to none
      4. Hit Send

      If the Certificate is used by a node look for the used_by part, when there is a node_id, the certificate is still in use and can’t be deleted. If it is empty, you can delete the Certificate in the UI, you can do this check on the new certificate to see if it is used by the same node.

          "used_by": [
        {
            "node_id": "515f0642-b1eb-ef39-9acc-82f8760ab0b9",
            "service_types": [
                "GLOBAL_MANAGER"
            ]
        }
    ],

      Sometimes the Certificates won’t release itself, so let’s release the damn thing:

      Release a Certificate

      Please keep in mind that you only release the certificate from the node_id if you are absolutly sure, if not please raise a ticket to VMware Support.

      1. login with the admin user to the manager with ssh
      2. then type st e, enter the root password and you are now at the shell
      3. Use the certificate id and the node_id from the previous step:
      4. now use the following API call to release the Certificate of the node_id:
        curl -k -X POST -H “Content-Type: application/json” -H ‘X-NSX-Username:admin’ -H ‘X-NSX-Groups:superuser’ -d ‘{“service_type”:”API”,”node_id”:”<node_id>“}’  “http://localhost:7440/nsxapi/api/v1/trust-management/certificates/<certificate-id>?action=release“
      5. This should do it, you can check the certificate again with the previous step and look for the used_by, this should be emtpy now.
      curl -k -X POST -H "Content-Type: application/json" -H 'X-NSX-Username:admin' -H 'X-NSX-Groups:superuser' -d '{"service_type":"GLOBAL_MANAGER","node_id":""647defab-13c5-5g62-93e4-da4a345d666"}'  "http://localhost:7440/nsxapi/api/v1/trust-management/certificates/ea5771fb-161e-49dd-97d7-c1483d2790666691?action=release"

      If the used_by is empty, you can now safely remove the certificate.

      All the steps are almost identical for the Global and Local Manager, just change the service types for GLOBAL_MANAGER to LOCAL_MANAGER, etc.

      And the Alerts are now Resolved!

      READ MORE

Design Consideration – For Bare Metal Edge with 4 PNIC in a NSX-T Federated Environment

NSX-T, , , , , ,

During a failover test with the Bare Metal Edges we ran into an issue when pulling the plug on 1 of the TOR switches. (TOR-LEFT). During that test all BGPs on both Bare Metal edges went down. So no North-South routing anymore 🙁

Edge Setup:

The Bare Metal Edges were configured following the design guidelines from VMware:
https://nsx.techzone.vmware.com/resource/nsx-t-reference-design-guide-3-0#_Edge_Node_and_1

Indentifying the problem:

So why this behaviour? And what happens when we pull the plug on the other TOR switch (TOR-RIGHT).
After performing the test with the TOR-RIGHT, the BGPs connected to TOR Left stayed established. So it has something to do with switch TOR-LEFT?

After checking the configuration on the TOR-LEFT switch we didn’t identified something that could cause this issue. But what could it be? Edges were configured by VMware guidelines and were identical configuration wise.

So going through the logs was the next step in the process, and i stumbled upon this part in the log file:

2022-10-17T10:37:08.578Z Update device fp-eth0 state to DOWN
2022-10-17T10:37:08.578Z Self Node 00363d34-fcdd-11ea-8e07-e4434ba66042 status changed from Up to Down (RTEP device down)

Can it have something to do with the federated setup (RTEP), is the RTEP only connecting over fp-eth0?

Cause:

Again i went through the setup but now i also checked the fp-eth0 connections to the switches. On both BareMetal Edges the fp-eth0 was connected to the TOR-LEFT. So when we pulled the plug on that Switch it triggered the RTEP going down, which led to all BGP session going down.

This is expected behavior according to VMware!

Solution:

The solution to this issue was pretty simple after we identified the cause. We switched the connection on the second Bare Metal Edge, so the pnics connected to TOR LEFT will be on TOR-RIGHT and vice versa. The opposite of the first Bare Metal Edge.

READ MORE