SSP 5.2 – DFW 1-2-3-4 Hands-On

Uncategorized, , , , , , ,

In part 2 of my SSP series I explained the four stages of DFW 1-2-3-4 at a high level. In this post I go through every step in my demo environment, including the CSV files I used, with Security considerations in view. You can follow along if you have SSP 5.1 or newer.

Before you start

A few things to arrange first:

  • SSP is deployed and onboarded to NSX, and Security Intelligence is available. Log in with an admin account on the SSP FQDN. Logging in on the IP address doesn’t work.
  • Check your NSX version. CSV import for segments, Tier-0 and Tier-1 gateways needs NSX 4.2.1 or newer. Static IP-based applications need 4.2.2 or newer. Application member resolution for DVPGs doesn’t work on a few versions, among them 4.2.0 to 4.2.3 and VCF 9.X
  • Set your naming conventions first. Go to Monitor & Plan > Settings > General Settings and click Edit. Here you can set the asset delimiter (underscore by default, or a period or hyphen) and the prefixes for assets, policies and rules. These only apply to unpublished assets, so do this before your first import.

How to start the journey

There are two ways to proceed: click Begin With Security Segmentation Journey on the SSP home page, or use Journey in the top navigation. The journey page shows the four stages. When you finish all tasks of a stage, you click the round status indicator to mark it as complete.

Stage 1: Security Segmentation Assessment & Report

  1. Go to System – Platform & Features and activate Security Intelligence. If it’s already active, skip this.
  1. Wait at least an hour. The report needs flow data before it can calculate anything.
  2. Click Monitor & Plan – Security Segmentation Report to generate the report.

The score is based on the traffic flows of the last 30 days, which is the maximum. You can run the report again, for example after major infrastructure changes. I’d run it at the start and after each stage, so you can show the progress.

My score before:

The CSV import: how it works (used in stages 2, 3 and 4)

Stages 2, 3 and 4 all use the same import workflow, so I explain it once for Security clarity. It’s called Segmentation Planning and it has five steps:

Inventory in NSX before importing

Before the import of the CSV file in my lab environment there are no groups (tip: filter on generic, to create a better view)

Also in the Virtual machines Inventory in NSX there are no tags assigned to the virtual machines.

And under Tags, no tags created yet.

Preparing the CSV

Use the sample CSV from the UI as reference. What matters:

  • An asset identifier. This can be a name, IP address or UUID. You can use up to three identifier columns. They are evaluated in order, and the lookup stops at the first match. Each row needs at least one non-empty identifier. The default priority is asset ID, then IP, then name.
  • The hierarchy. You can use 1 to 5 columns for region, zone, environment, application and tier. If you only provide region, zone and environment, SSP doesn’t propose applications or tiers.
  • Optional columns: an asset type (VMs default if the column is missing) and infrastructure servers.
  • Keep it clean. No hidden characters and no byte order mark. The file can be up to 500 MB. Don’t give an asset multiple tags for the same scope.

One tip from the documentation I really agree with: test with a small CSV first. The scope hierarchy can’t be changed during an analysis, so plan it carefully. Below is a sample CSV we use to import in our lab environment. You can see applications, tiers, infrastructure services regions and zones added to virtual machines on the platform.

Importing the CSV

  1. Go to Monitor Plan – Policy Planning and click Select a File (or drag and drop the file).
  2. SSP parses the CSV and validates the format and data.
  3. Check the column mappings. On your first import, columns with headers that differ from the sample CSV stay blank and you have to map them yourself. On later imports, SSP remembers the mapping if the headers match the previous import.
  4. Click Preview.
  5. f there are errors, SSP gives you a link to an annotated CSV with the errors highlighted. Fix it and upload again. If you continue anyway, the incorrect data is left out.
  6. Click Start Analysis. The duration depends on the size of your file. When it’s done, open the Summary tab.

It is possible to add new columns to the CSV file and map them in the screen above to the right scope.

As you see in the screenshots above it uses all the information in the provided CSV and map those to the right scope.

Publishing

My advice, and the documentation’s as well: publish the inventory assets and tags individually first, check them, and only then use Publish all. Before using Publish all, make sure there are no tagging conflicts, otherwise the publication can fail.

In the summary you can find a quick overview of everything that is found, mapped and ready for publishing to NSX.

In the Publish Inventory Assets you can go through the individual groups like Infrastructure groups, Regions & Zones, Environments and Applications. Per tab you can review and publish them.

After publishing all Inventory assets the status should reflect Published with a green check mark

If you now go back to NSX and review the inventory , you will see that there are in my case 46 groups created

Great! Next we are going to check if Security Intelligence detected some Servers which correspond with the Infrastructure Services running.

Stage 2: Secure Infrastructure Services

The goal here is to find the legitimate infrastructure servers and block the rogue ones. SSP supports 14 infrastructure services, including DHCP, DNS, LDAP, NTP, SNMP and Syslog. It finds them in two ways: from your CSV and by looking at the flows.

In the previous step we already added the csv which had the infrastructure services in, so this step we can skip. Security Intelligence has ML built in to check all the flows that it gathers and can figure out which of the flows/vms belong to one of the infrastructure services we can map.

CSV Infra sample:

The column names and the way you indicate the service must match the sample CSV in your UI. Map the columns if they differ.

Under the “Publish detected Servers to Infrastructure Groups” tab we can see the system detected some servers corresponding with the LDAP service, which we not had added to the CSV file.

Because there is no existing infrastructure group for this service. Upon publishing, a global LDAPS infrastructure group will be created for the selected servers. and you can add the selected server to this group. Or just suppress this server if you don’t want to do anything with this system and ignore it. You can find suppressed servers under the Suppressed Servers tab later on.

Publish Distributed Firewall Policies for the Infrastructure Services

Under the Publish Distributed Firewall Policies and the Environment Rules you can review and publish the Firewall Policies that are suggested for the Environments, Zones zones and Regions. So up front there will be no policies for these in NSX under the Environment tab in DFW.

After publishing the policies are created:

Check the Policy Monitoring Tab in SSP and select Infrastructure Services. Here you get an overview of what is currently in place, how many services are detected, if there are incomplete rulesets or even no rules attached to a security policy. You can go through these Infrastructure services one-by-one or you can review and add all of them in once.

For DNS-UDP we have an Incomplete ruleset detected, lets generate some DFW rules to complete this ruleset.

Review the rules and hit Publish.

After this action, we can see that all rules are now in place for this Infrastructure Service

For the rest we are going to do them all in once, hit Review and Add Rules on the main screen of Policy Monitoring

In this screen you can review all rules and add Lockdown rules for all services if you want this level of security

In NSX the policies and rules are now updated under the DFW Infrastructure TAB.

Now we re-calculate the scores to see what this stage will give us:

Stage 3: Environments (zones)

Now we split the datacenter into broader zones, like Production, Development and Test. When you import the CSV, SSP pushes tags to the VMs so they become members of the right environment. It then creates groups and DFW rules for the traffic between the environments.

The CSV can come from a CMDB like ServiceNow, from vCenter, or from a spreadsheet you make yourself. For this stage I used three hierarchy levels:

In the previous step we already added the csv which had this in formation in it in, so this step we can skip. The Hierarchical levels are created with tags.

Publish Distributed Firewall Policies for the Environment

Under the Publish Distributed Firewall Policies and the Environment Rules you can review and publish the Firewall Policies that are suggested for the Environments, Regions and Zones. Check if this reflects with your CSV and environments. In this case all the logic will be done in the application layer of the DFW. because of the Jump to Application action.

So up front there will be no policies for these in NSX under the Environment tab in DFW.

After publishing:

Check the Policy Monitoring Tab in SSP and select Environments. Here you get an overview of what is currently in place, how many environments and if there are inter-environment rules, if there are no protection rules in place and if there is traffic detected inter-environments.

As you can see we have no inter-Environment policies yet configured between the environments, we have 2 views here, Grid view and the List view. The list view is familiar and looks like the other screens presented earlier. From here we can run traffic based recommendations (select under Action menu) or add a Policy directly to a environment pair.

In the Grid view you can hover over a tile to see the configuration and click on it to see the actions, the same as in list view but i find the grid view a great first view to check the iterator environment policies.

If you select Add policy you get thepolicy and a protection rule to monitor the traffic between the environments.

When you select Run recommendation, you can run a recommendation based on the traffic between the 2 environments, and review and proceed with these recommendations in the policy recommendations tab

In this way you can easily protect your inter-environment traffic with policies and rules. And constantly monitor the traffic between those 2.

After putting the inter-environment rules in place the matrix view looks like this:

All Inter-Environment is now segmented, the flows between prod <-> prod and dev <-> dev will be handled in the application layer because Jump–to–application is selected. Inter-environment traffic between other environment like dev <–> prod, pairs will be blocked.

When you want to set the default Environment Category rule action you can change this easily.

Again we re-calculate the scores to see what this stage will give us:

Stage 4: Application micro-segmentation

This is the final stage, and it’s done in two phases: first ring-fencing to define the controls between applications, then the controls between the tiers inside each application (web, app, database).

For this I use the full hierarchy, with the application and tier columns added:

My test file:

Steps:

  1. Import the CSV, check the hierarchy in the preview and start the analysis. SSP creates application groups, which are also used for the environment exception policies from stage 3 (already done)
  2. Publish the inventory assets (already done)
  3. Monitor the applications on the application monitoring page. SSP follows the flows of each application, before and after you publish rules.
  4. Run the policy recommendations for the applications. Start with the ring-fencing rules, then the tier rules.
  5. Review the recommended rules, publish them, and keep monitoring. Fine-tune where needed.

Ass you can see in the screenshot below on the application monitoring page. SSP follows the flows of each application, before and after you publish rules. At this moment we have 8 applications, no application is marked critical, no rule hits detected and no policies applied. Let’s get to work.

Let’s pick the first application, we can see that we have around 131 unique flows, those are all east-west and is split between Incoming and outgoing. Also a few Intra application flows are detected. You can also see that SSP already have 3 Unpublished recommendations available for this application. And in the chart you can see that most of the flows are unproitected at this point. The ones that are already protected are the Infra and Inter-Environment flows.

Let’s see what recommendations we have, these recommendations where run during the environment step as you can see on the Category mentioned.

When you click on the icon on the right, you can switch to canvas view to view the flows in a canvas. Here you can filter between applications(as shown below) or Groups, or compute (ip addresses, etc). You can adjust the time range of which flows you want to use. With the buttons on the right bottom corner you can tweak your view. Zoom in/out, drag & drop, etc.

Back to the recommendations, we need to start a new recommendation for the Applications

We need to select the entity we are runnin g this recommendation against and the Time range for traffic where the recommendation will be based on.

Lets select our application, hit SAVE and NEXT to go to select the Segmentation strategy (optional).

When you run the policy recommendations for an application, SSP asks you to choose a strategy to auto-plumb rules.
Two things on that screen are worth knowing before you click anything. The auto-plumbed rules are added at the bottom of the new policy. And once you publish them, the protected traffic is excluded from future traffic-based recommendations. So your choice decides which traffic you close with a default action, and which traffic SSP keeps watching to generate granular rule recommendations.

There are three strategies:

Intra-application trafficIncoming and outgoing trafficDefault rule
RingfencingAllowed by defaultMonitored, for granular recommendationsAllow (you can change the action)
MicrosegmentationMonitoredMonitoredAllow (you can change the action)
CustomYou chooseYou chooseYou choose

Ringfencing

SSP allows the traffic inside the application by default, and starts monitoring the incoming and outgoing traffic to generate granular rule recommendations for it. Inside the application nothing changes, so the tiers keep talking to each other as before. Only the boundary of the application gets attention. This is the ring-fencing phase of the journey, where you define the controls between applications.

Pick this one when:

  • It’s your first pass at an application, especially in a brownfield environment where you don’t know exactly how the tiers talk to each other.
  • You want to control what goes in and out of the application, without risking the traffic inside it.
  • You want a low-risk start. In my demo this was the strategy marked as Strategy Currently In Place.

Microsegmentation

SSP monitors all the traffic: incoming, intra-application and outgoing. It then generates granular rule recommendations for all of it, including the traffic between the tiers (web, app, database). This is the second phase of the journey, where you establish the controls across the tiers inside the application.

Pick this one when:

  • The ring-fencing is in place and you want to go further, to tier-to-tier rules.
  • The application is sensitive (think card data or personal data) and “everything inside is allowed” isn’t good enough.
  • You have enough flow data for the application. More monitored traffic means more recommendations to review, so plan some time for it.

Custom

With Custom you decide per traffic type. The options are incoming traffic, outgoing traffic, intra-application traffic and the default rule. For the types you select, you choose to allow or block them by default. The remaining flows are monitored to generate granular rule recommendations.

Pick this one when neither of the two fits. Some examples:

  • An application with many outgoing connections you don’t want to model rule by rule (for example updates or external services). You allow outgoing by default, and keep monitoring the incoming and internal traffic.
  • A tier that should never receive traffic from outside the application. You block incoming by default and keep monitoring the rest.

My advice

Start with Ringfencing, and move to Microsegmentation per application when you’re comfortable with the flows. Use Custom for the exceptions. In all cases, look at what the default rule does before you publish. With the default rule on Allow, traffic that no specific rule covers still goes through, which keeps the application working while you’re learning. Switching a default to block is the moment things can break, so do that after you validated the recommendations.

In the next screen, there’s one more choice when you start a recommendation: the Recommendation Service Type. It decides what kind of rules SSP generates. L4 Services is the default.

L4 Services

The recommended rules are based on ports and protocols, for example TCP 443. SSP aggregates the flows by the service (the port or protocol) they used, so you get one rule per service for the sources and destinations that use it. It’s simple to read, easy to review, and it’s what most DFW rules look like.

L7 Context Profiles

Here the recommended rules can also contain a context profile. A context profile holds one or more Layer 7 App IDs, so the rule doesn’t only say “TCP 443”, it says which application is allowed on that port. NSX has built-in App IDs for common infrastructure and enterprise applications, like SSL/TLS and CIFS/SMB.

How SSP decides, as described in the documentation:

  • For flows that have an App ID, SSP creates a rule with the matching context profile, as long as it can identify that profile from the App ID. SSP does not create new context profiles. If the profile doesn’t exist, you get an L4 rule for that flow instead.
  • For flows without an App ID, SSP creates an L4 rule.
  • When SSP modifies an existing rule that already contains context profiles, it keeps those profiles. It also matches existing rules on the same port, protocol and App ID.
  • If no L7 context data is available for the analysis, you get L4 rules, even though you selected L7.

So expect a mix of L7 and L4 rules in the result, not only L7 rules.

Which one to pick?

L4 ServicesL7 Context Profiles
Rule is based onPort and protocolPort, protocol and App ID
NeedsNothing extraFlows with App ID data, and context profiles that exist in NSX
Best forStarting out, ring-fencing, well-known portsPorts that can carry more than one application, and stricter control

My advice:

start with L4 Services. It’s the simplest and it gives you a working rule base fast. Use L7 Context Profiles where the port alone isn’t specific enough, for example when you only want a certain application on port 443 and not anything that happens to use TLS. Keep in mind that L7 has a cost. In NSX, a packet that matches a rule with an L7 context profile is sent to the DPI engine to determine the App ID, which means more processing than a plain L4 rule. The NSX design guidelines also recommend combining a context profile that has a single App ID with the matching ports in the L4 service field. So use it where it adds value, not everywhere.

In the same screen you have a few other otions like excluding protocal and ports and the option to separate the rules by the service. When enabled, each service gets its own separate rule even if they share the same src and dst. We leave them disabled for now and hit Start Discovery. The recommendation will be started and can be found under Policy Recommendations.

Let’s click the 3 ellipses and Review and Publish the recommendation!

You can click the Show graph to see an graphical overview of the recommendation, and in the list below you can review Rules and Groups that are proposed. Review this and if you are alright with the recommendation you can hit Proceed. if you want to change an action for a specific Service or rule you can simply adjust the Action. This will also immediatly reflect in the graphical overview.

In the next screen you can rearrange the sequence of the rules, Recommended Rules are placed at the top of existing Firewall rules by default.

After publishing the policies we can see in the recommendation overview that it is Published:

In NSX, we can see that the policy and rules are published and part of the Application layer of the DFW. You can see the policy name is the name you had set in the recommendation. this name is easily adjustable in the UI.

Just click on the name and change it, in our case to peoplesoft and publish the firewall change.

Now when we check the Policy Monitoring view in SSP again, and check on the Application we can see in the graph we went from a few protected flow to all flows protected for this Application.

Now for another application we will run a recommendtation and select L7 services instead of L4.

The change is when we review and Publish this recommendtaion is that we now have Profiules added to the policy based on the flow data. In this case MYSQL and SSL traffic.

We will select different strategy throughout the applications and when all recommendations and strategies are publshed we will run the report again to see how we did.

Run the report again

Generate the Security Segmentation Report again, like in stage 1, and compare the score.As you can see we are not quite there yet. The last step for this is to Lockdown the applications, like we did earlier at Infrastructure level.

So once ready, update the Segmentation Strategy and switch the default rule action to ‘Drop’ to block all unauthorized traffic. As you can see below for both Infrastructure services and applications the lockdown rules are now in place.

When we re-calculate the score again we have a big hit

Now we can see that we can win extra point with enabling the Malicious IP rules on the DFW level.

The Malicious IP rules are the two default rules that NSX/vDefend adds to the Distributed Firewall. They block traffic to or from IP addresses that are known to be bad.

How they work

Malicious IP Filtering blocks network traffic to or from known malicious IP addresses, such as those used by hackers, botnets or malware servers. The system downloads the IPs from the vDefend Threat Intelligence Service (VTIS) cloud service and puts them in a malicious IP group. The default policy, called Default Malicious IP Block Rules, contains two rules: one that matches the malicious group at the source and one at the destination. They sit in the Infrastructure category. Malicious IP Filtering – Broadcom TechDocs +3

A few details that matter in practice:

  • The feed is downloaded every 12 hours once auto-download is on. You can also trigger a download by hand.
  • If auto-update is off, rules with malicious IP groups can be enforced with outdated data. broadcom
  • The feed is IPv4 only. broadcom
  • You can create custom malicious IP groups for exceptions that must not be blocked, and you can build your own rules with malicious IP groups. broadcom

Why use them

  • Low effort, real value. It’s a baseline you get without modeling any flows. Every workload gets protection against reaching known botnet and malware infrastructure, enforced on the host rather than on an appliance. In my view, this makes it a good first layer for outbound command and control traffic.
  • Automatic updates. You don’t maintain the list yourself.
  • Visibility. There’s a Malicious IPs Filtering and Analysis dashboard where you can see which IPs were blocked. broadcom

Why be careful

Limited coverage. It only knows IPs that are already flagged, and only IPv4. It’s not a replacement for segmentation or for IDPS and NDR. Also, the Manager has to be able to fetch the feed from the cloud service, so in isolated environments check how the updates work.

Licensing surprise. The feature isn’t entitled under NSX Enterprise Plus. If you add a vDefend Firewall or vDefend Firewall with ATP license to an environment that started on Enterprise Plus, the feature is enabled automatically and starts blocking traffic to flagged IPs. Customers get caught by this. broadcom

False positives you can’t override from below. Reputation feeds sometimes flag a destination you need. Broadcom has a KB about traffic being dropped despite user-created allow rules, caused by the destination being in the malicious IP database. Because the rules sit at the top of the Infrastructure category, your own allow rules further down the table don’t help. The fix is to add the IP to the exception list, or to put an allow rule for the known-good destinations above the Default Malicious IP Block Rules. broadcom

Troubleshooting confusion. If a team reports “the firewall drops my traffic” and all your own rules look fine, the default malicious IP rules are an easy thing to miss. Packet drops under Default Malicious IP Block Rules are the sign that the feature is active. broadcom

Final Score Calculation

Lessons learned

My tips, based on the documentation and my demo:

  • Start with a small CSV, and with VMs you know.
  • Fix your naming conventions before the first import.
  • Keep your hierarchy consistent. Don’t give an asset more than one tag for the same scope.
  • Publish assets and tags step by step. Use Publish all only when there are no tagging conflicts

Thanks all for reading if you made it to the end of this very lenghty blog post…..and watchout for more SSP goodness coming up.

READ MORE

SSP 5.1: DFW 1-2-3-4, a guided path to microsegmentation

vDefend, , , , , ,

This is part 2 of my SSP series. In the first post I explained what SSP is. SSP 5.1 arrived together with VCF 9.0, and one of its big additions is DFW 1-2-3-4. It’s a feature of Security Intelligence, available through SSP 5.1.

What is DFW 1-2-3-4?

It’s a guided workflow that takes you through segmentation in stages, following the lateral traffic patterns in your environment. Under the hood an analytics engine discovers communication patterns, identifies unprotected traffic and recommends firewall rules.

The idea is to prevent you from jumping straight to application-level microsegmentation without the visibility you need. Instead you start with the quick wins and work towards the application level.

The four stages

1. Security Segmentation Assessment & Report

You generate a Security Segmentation Report that shows your current posture. The segmentation score re-calibrates automatically when your environment changes, so you can track progress over time and show it to management or auditors.

2. Infrastructure (shared) services

Protect DNS, NTP, Syslog, SNMP, DHCP and LDAP first. The workflow discovers these services for you, or you can feed your known endpoints through a CSV file. Locking down DNS alone already removes some of the most common command and control and exfiltration paths.

3. Environment (zone) segmentation

Define zones like Dev and Production. You can import the metadata from a CSV, for example exported from a CMDB like ServiceNow or from vCenter. The workflow assigns the security tags, creates default zone-level rules and monitors for traffic leaking between zones.

4. Application microsegmentation

This stage has three steps:

  • 4a. Map workloads to applications, again via CSV. The workflow auto-tags them and creates the application groups.
  • 4b. Define ring-fencing for each application. The system recommends rules that only allow communication between the permitted entities.
  • 4c. Continuously monitor the applications, before and after publishing rules, and fine-tune the rules between the application tiers.

My take

VMware says a typical rollout can be done in as little as a few weeks. I’d treat that as a best case, because the CSV input (zones, workload-to-application mapping) depends on how good your CMDB data is. But the structure alone is useful, especially in brownfield environments where nobody knows exactly what talks to what.

READ MORE

What is the Security Services Platform (SSP) in VMware vDefend?

vDefend, , , , , , , , , , ,

When I started working with vDefend, I quickly noticed that SSP is often treated as “that thing you also have to deploy”. In reality it’s the foundation for a big part of the vDefend feature set. This is the first post in a small series about SSP. Here I explain what SSP is and what runs on it. In the next posts I’ll go through what’s new in SSP 5.1 and 5.2.

What is SSP?

SSP is a high-performance, scalable platform that runs vDefend features like the Security Segmentation Report and Score, Security Intelligence, Metrics, Network Detection and Response, Malware Prevention and Network Traffic Analysis. Think of it as a scale-out data lake that ingests network flow records and telemetry. On top of that data it gives you flow visibility, security assessment scores, policy recommendations and guided workflows.

Under the hood it’s Kubernetes. You deploy it with the SSP Installer (SSPi) appliance, and once it’s running you can follow what’s happening with plain kubectl commands. I used that in a previous post to follow an NSX onboarding.

What doesn’t run on SSP?

This one confuses people. The enforcement itself stays in NSX. Distributed Firewall, Gateway Firewall and the malware policy and service VM management are still done through the NSX API. So your firewall keeps working without SSP. What you miss is the analysis, the guided workflows and the advanced threat prevention on top of it.

Why a separate platform?

Processing flow data from a whole private cloud needs its own scalable layer, separate from the NSX management plane. SSP is a self-contained, scale-out platform with a simplified network design and streamlined lifecycle management. That makes deploying Security Intelligence, NDR and Malware Prevention a lot easier than it used to be.

Related

Resolve SSP 5.1.1 Errors During NSX Onboarding

READ MORE

Resolve SSP 5.1.1 Errors During NSX Onboarding

NSX-T, Security, vDefend, , , , , , , ,

During the deployment of SSP (Security Services Platform) 5.1.1, I ran into an issue while onboarding the NSX instance.

This error means that there is still an old SSP installation onboarded in NSX. In my lab environment this was the case and the old SSP installation got corrupted due to storage issues and could not get correctly ofboarded and decommisioned. Luckily there is a kb that can help you clean up NSX and SSP.

https://knowledge.broadcom.com/external/article/382295/offboarding-a-site-during-force-deletion.html

This kb explains that a scripts needs to be ran at SSP version 5.0 or 5.1 to get the onboarded NSX sites properly offboarded.

  1. Download the right script for the version of SSP
  2. Upload the script to the SSPi, Installer appliance of SSP.
  3. Login via ssh to the SSPi appliance as sysadmin user (for SSP 5.1) and make the script executable (chmod 777)

Next you can run the script:

Example usage:
./site-offboarding-cleanup-nsx-ssp5.1.sh [-h|--host <mgr_host_ip>] [-u|--username <username>] [-p|--password <password>]
site-offboarding-cleanup-nsx-ssp5.1.sh

./site-offboarding-cleanup-nsx-ssp5.1.sh -h 10.10.10.0 -u admin -p p@ssw0rd

After running the script the NSX site cleanup is complete, and you can go try to onboard NSX again to SSP. During the onboarding you can follow the onboarding process:

k get site -A

k describe site 098ad7b9-cdea-46b8-82a7-1f2e3d1fd366  -n nsxi-platform

When you see “OnboardingInProgress”, you just have to hold on a little bit until it says “OnboardingComplete”.

In the SSP you now shall see that NSX is correctly onboarded!

READ MORE

vSAN HCL DB Out of Date – Offline Update

1 CommentVCF, vSAN, , ,

During the upgrade pre-check test of the Workload domain I got the following error: vSAN HCL DB Out of Date.

Because my SDDC manager has no direct internet connection I needed to get the file on my jump host. This can be done by browsing to https://partnerweb.vmware.com/service/vsan/all.json and copy the entire content and create a new file with extension”.json”

This can also be done via the bundle-transfer util, and you also should be able to transfer the file to the SDDC manager with this util. How to get the tool and install it can be found on https://www.aaronrombaut.com/real-world-use-of-vmware-bundle-transfer-utility/

If you have the bundle-transfer util installed we can get the vSAN HCL file with the following command:

lcm-bundle-transfer-util.bat --vsanHclDownload --outputDirectory F:\

After I got the all.json file on my jump-host, I need to update it to the SDDC manager:

lcm-bundle-transfer-util.bat --vsanHclUpload --inputDirectory "F:\vsan\hcl\all.json" --sddcMgrFqdn sddc-l-01a.corp.local --sddcMgrUser vcf

But as you can see below this command gave me an error:

Fails due to not generating token, let’s see is we can upload the file by API, let’s use Postman for this.


Get Bearer token for the authentication with the SDDC manager:

  1. Select POST as action
  2. Fill in the api url: https://sddc-l-01a.corp.local/v1/tokens
  3. Under the Authorization tab select No Auth
  1. Under Headers create a new Key Content-Type with Value application/json
  1. Under Body select raw and JSON, in the field, put the sso admin and password for the SDDC Manager

{
"username" : "administrator@vsphere.local",
"password" : "XXXXXXX"
}

  1. Now hit the send button and wait for the output:

The bearer token is needed in the command to update the HCL DB on the SDDC manager:

Upload all.json to SDDC Manager via Postman:

  1. Select PUT as Action
  2. Use the following API URL: https://<sddc-fqdn>/v1/vsan-hcl/content
  3. Under Authorization select Bearer Token and Paste the Token from the previous step in the field.
  1. Under Headers add a new Key Content-Type with Value text/plain
  1. Under Body choose raw and JSON
  2. In the field copy the content of the all.json file
  3. Hit Send.

There is no output only a Status 200 OK

Let’s check if this particular pre-check error now is solved.

VICTORY!!

READ MORE

Quick Fix – Found zero ssd devices for SSD cache tier

VCF, vSAN, , , , ,

When i added a host to an existing vSAN cluster via the SDDC manager the task failed with the following error: “Found zero ssd devices for SSD cache tier”

To quickly fix this we need to set the cache disk on the ESXi host to SSD, you can check the current value with the vdq -q command. As you can see in the picture below the disk I want to use for the cache is marked with a value of “0”, so it is not recognized as SSD drive.

In the past you had to set the the disk to SSD with SATP claim rules, but from version 7.x and 8.x there is a new and simpler command to do this. Run the following ESXCLI command and use the storage device ID and the -M option with value of true (or false to revert the change) to mark the device as an SSD.

esxcli storage hpp device set -d naa.6000c299027de72c68de829e23455e88 -M true

READ MORE

Aria Ops for Networks deployment fails from Aria Suite LifeCycle Manager.

NSX-T, VCF, , , , , , , ,

In my lab I tried to deploy Aria Operations for Networks 6.12.1 (AON/vRNI) from Aria Suite LifeCycle 8.16 (ASLC/vRLCM). Before the deployment of AON I successfully deployed other products:

vIDM 3.3.7
Aria Operations for Logs 8.16 (vRLI)
Aria Operations 8.17.1 (vRops)

Attempted to deploy Aria Ops for Networks 6.12.1 but it failed with LCMVSPHERECONFIG1000016 error.  

-----------------------------------------------------------------------------------------------------------
java.io.IOException: com.vmware.vim.binding.vmodl.fault.SystemError
	at com.vmware.vrealize.lcm.drivers.vsphere65.vlsi.utils.ExceptionMappingUtils.mapAndThrowImportVAppExceptions(ExceptionMappingUtils.java:78)
	at com.vmware.vrealize.lcm.drivers.vsphere65.deploy.impl.BaseOvfDeploy.importOvf(BaseOvfDeploy.java:713)
	at com.vmware.vrealize.lcm.plugin.core.vsphere.tasks.DeployOvfTask.execute(DeployOvfTask.java:251)
	at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:62)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
-----------------------------------------------------------------------------------------------------------

Error Code: LCMVSPHERECONFIG1000016
IO Exception occurred while performing the operation. Check the logs for more information.
Unexpected ioexception occurred.

——————————————————————————————————–

After some scavenging around, I ran into the known issues in the Release Notes from version 8.14

So let’s check which user is being used between the ASLC and vCenter:

Now login the vCenter and check if the privileges assigned to this account match the 2 from the release notes:

As you can see above the privileges are not configured on this role, let’s configure the 2 privileges:

Now we need to resync the account between vCenter and ASLC.

After this I Hit the retry button on the request in ASLC and the ovf is being deployed in vCenter.

READ MORE

VCF 5.0 – Remove vSAN datastore after VCF deployment failed

VCF, , , , ,

After a failed deployment of VCF 5.0, i was left with a vSAN Datastore on the first host in the cluster, and this was blocking a retry of the deployment.

In this state the vsanDatastore is unable to be deleted. If I try to delete it, the option is greyed out.

To delete the datastore and the partitions of the disks, we first need to SSH into the host and get the cluster.

We need the Sub-Cluster Master UUID, copy it to the clipboard. To leave the cluster the command is:

esxcli vsan cluster leave -u <Sub-Cluster Master UUID>

The ESXi UI now shows that the datastore is gone.

Let’s check the cli, if there still is a vSAN cluster or storage:

esxcli vsan cluster list
esxcli vsan storage list

No vSAN clusters or storage is configured on the host, after this the retry was successfully and the VCF management cluster was deployed.

READ MORE

VCF 5.0 – Error connecting to ESXi Host SSL certificate common name doesn’t match ESXi FQDN.

VCF, , ,

During a new lab deployment of VCF 5.0 i ran in to an small issue running the validation.

I deployed the hosts up front and made them available and unique before the validation. ran the following command to regenerate the certs and restart the services:

/sbin/generate-certificates
/etc/init.d/hostd restart && /etc/init.d/vpxa restart

Next i wanted to see what Common Name (CN) was on the certificate:

As you can see the Common Name (CN) contains only the hostname.

Next i changed the hostname on the ESXi hosts to have the complete FQDN.

Checked the hostname on the cli, and regenerated the certificates again:

After the regeneration of the certificate on the Host, you have to restart the services:

/etc/init.d/hostd restart && /etc/init.d/vpxa restart

Check the Certificate in the browser and now the Common Name had the FQDN in it, and the validation finished successfully.


READ MORE

VCF 5.0 – Retry Bring up fails on Generate input for SSH Keys

VCF, , , ,

After a Failed VCF bring-up, I wanted to retry the bring-up. Luckily the error I encountered before was resolved but again I ran in to an issue during the retry.

Now the issue was with the import of the SSH Keys

Going through some internal resources i stumbled upon the solution, since this is a nested lab environment on top of VCD you have to reset the MAC address of the ESXi Host.

READ MORE