VMware Hands-on Labs - HOL-1892-01-CHG


Lab Overview - HOL-1892-01-CHG - VMware NSX - Challenge Lab

Lab Guidance


Note: It will take more than 90 minutes to complete this lab. You should expect to only finish 2-3 of the modules during your time.  The modules are independent of each other so you can start at the beginning of any module and proceed from there. You can use the Table of Contents to access any module of your choosing.

The Table of Contents can be accessed in the upper right-hand corner of the Lab Manual.

It is your first day on the new job and you need to spend some time learning what applications are running in the data center and how your predecessor configured the networking and security.

This time is used to understand the environment.  This is a small enclave supporting the most critical application in the business.  The previous staff, who are no longer there, purchased VMware NSX and implemented it last year.  The CIO is curious to understand what is being done to protect the application as well as to keep them up and running so that he can meet his commitments to the business.

Lab Module List:

 Lab Captain:

This lab may be available in other languages.  To set your language preference and have a localized manual deployed with your lab, you may utilize this document to help guide you through the process:

http://docs.hol.vmware.com/announcements/nee-default-language.pdf


 

VMware Technology Network (VMTN)

For additional hints and to discuss the challenges presented in the lab further, be sure to visit the VMware Technology Netowork (VMTN) Community Pages:

https://communities.vmware.com/community/vmtn/challenge-lab/vrealize-operations

 

 

Location of the Main Console

 

  1. The area in the RED box contains the Main Console.  The Lab Manual is on the tab to the Right of the Main Console.
  2. A particular lab may have additional consoles found on separate tabs in the upper left. You will be directed to open another specific console if needed.
  3. Your lab starts with 90 minutes on the timer.  The lab can not be saved.  All your work must be done during the lab session.  But you can click the EXTEND to increase your time.  If you are at a VMware event, you can extend your lab time twice, for up to 30 minutes.  Each click gives you an additional 15 minutes.  Outside of VMware events, you can extend your lab time up to 9 hours and 30 minutes. Each click gives you an additional hour.

 

 

Alternate Methods of Keyboard Data Entry

During this module, you will input text into the Main Console. Besides directly typing it in, there are two very helpful methods of entering data which make it easier to enter complex data.

 

 

Click and Drag Lab Manual Content Into Console Active Window

You can also click and drag text and Command Line Interface (CLI) commands directly from the Lab Manual into the active window in the Main Console.  

 

 

Accessing the Online International Keyboard

 

You can also use the Online International Keyboard found in the Main Console.

  1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar.

 

 

Activation Prompt or Watermark

 

When you first start your lab, you may notice a watermark on the desktop indicating that Windows is not activated.  

One of the major benefits of virtualization is that virtual machines can be moved and run on any platform.  The Hands-on Labs utilizes this benefit and we are able to run the labs out of multiple datacenters.  However, these datacenters may not have identical processors, which triggers a Microsoft activation check through the Internet.

Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoft licensing requirements.  The lab that you are using is a self-contained pod and does not have full access to the Internet, which is required for Windows to verify the activation.  Without full access to the Internet, this automated process fails and you see this watermark.

This cosmetic issue has no effect on your lab.  

 

 

Look at the lower right portion of the screen

 

Please check to see that your lab is finished all the startup routines and is ready for you to start. If you see anything other than "Ready", please wait a few minutes.  If after 5 minutes you lab has not changed to "Ready", please ask for assistance.

 

Overview - VMWare NSX


VMware NSX® is the network virtualization and security platform for the Software-Defined Data Center (SDDC), delivering the operational model of a virtual machine for entire networks. With NSX, network functions including switching, routing, and firewalling are embedded in the hypervisor and distributed across the environment. This effectively creates a “network hypervisor” that acts as a platform for virtual networking and security services. Similar to the operational model of virtual machines, virtual networks are programmatically provisioned and managed independently of underlying hardware. NSX reproduces the entire network model in software, enabling any network topology—from simple to complex multitier networks—to be created and provisioned in seconds.  Users can create multiple virtual networks with diverse requirements, leveraging a combination of the services offered via NSX to build inherently more secure environments.

KEY BENEFITS

• Micro-segmentation and granular security delivered to the individual workload

• Reduced network provisioning time from days to seconds and improved operational efficiency through automation

• Workload mobility independent of physical network topology within and across data centers

• Enhanced security and advanced networking services through an ecosystem of leading third-party vendors

USE CASES FOR NSX

Security

NSX embeds security functions right into the hypervisor. It delivers micro-segmentation and granular security to the individual workload, enabling a fundamentally more secure data center. Security policies travel with the workloads, independent of where workloads are in the network topology.

Application Continuity

NSX abstracts networking from the underlying hardware and attaches networking and security policies to their associated workloads. Applications and data can reside and be accessible anywhere. Move workloads from one data center to another, or deploy them into a hybrid cloud environment.

Automation

NSX lets you treat your physical network as a pool of transport capacity, with network and security services attached to workloads using a policy-driven approach. This automates networking operations and eliminates bottlenecks associated with hardware-based networks.

Compliance

NSX enables micro-segmentation and granular security of workloads in virtualized networks, isolating sensitive systems and reducing both risk and scope of compliance. Use NSX to help ensure and demonstrate compliant operations with many regulations such as PCI DSS, HIPAA, FedRAMP, SOC, CJIS, DISA STIG, and more.

 

 

In this lab, you will have the ability to explore some of these functions in a controlled and safe environment.  You will be given specific tasks to complete along both the security and network aspects of the platform.  You wil also have the opportunity  to conduct attacks, validate functionality and connectivity to see if you have successfully configured the platform correctly based on given requirements.

 

 


 

Scenario - First day on the job

This time is used to understand the environment.  This is a small enclave supporting the most critical applications in the business.  The previous administrator, who is no longer there, purchased VMWare NSX and implemented it last year.  The CIO is curious to understand what is being done to protect the applications as well as to keep them up and running so that he can meet his commitments to the business.

 

It is your first day on the new job and you need to spend some time learning about the environment that you will be taking over.  You were given a high level overview of the environment during the interview but now you should take some time and discover what applications are running in the data center and how your predecessor configured the networking and security.

 

Module 1 - Introduction to the Environment (15 Minutes)

Introduction to the Challenge Lab


In this module, you will become familiar with your new environment.  The previous admin has completed some work but you are unsure as to what exactly has been completed.  You will review the configuration of the vSphere environment, the NSX Network and Security Platform as well as the Applications that your organization relies upon.

 

Your CIO has requested that you look into adding additional security to the network and you need to start working on a plan to achieve this.  Network stability has been good since VMware's NSX product has been installed but there appears to be more that can be obtained from the product.

 

The old admin has left to start with a new company is not available for any knowledge transfer.  Your supervisior has given you the credentials and some time to get familiar with the environment.

 


vSphere and VMs


 

Open a browser by double clicking on the Google Chrome icon on the desktop.

 

Double Click on Google Chrome

If for any reason the vCenter console does not automatically load, select the bookmark folder labeled RegionA.  You will need to use the RegionA Client for this lab.  

 

You will see a screen like this 

 

You will now have the option to log into vCenter.  Do not use the option to login with Windows session authentication as it will not allow you to view any of the network and security options that exist.

 

Your access to the vSphere GUI is: 

USERNAME: administrator@vsphere.local

PASSWORD: VMware1!

 

Enter these credentials to get started.  Once logged on, you should see vSphere Web Console.

 


 

Core Application - Customer Database

The Core Application that you will be look after is called the Customer Database.  This is the most critical application as it allows everyone to access key details on the firm's customers.  It must be up and running at all times as well as secure from hackers.  Open a new browser tab in Chrome and access the application

 

 

 

Customer Database

 

Clicking on the blank space immediate to the right of the vSphere Web Client Tab will open a new tab in the browser

 

Choose the bookmark under Region A Called Site A WebServer

 

 

 

Clicking this link will launch the application

 

 

VMs that power the Customer Application

 

Now that you have loaded the application, let's explore the components that make up the application.  As you are unsure as to the how the application is put together, lets take some to to discover how this is app is configured.

 

The Three VM's that make the application are called

You will find these VM's in the Hosts and Clusters view in vCenter.  Switch back to the browser tab that has vCenter loaded and choose the hosts and clusters view

 

 

 

Let's examine the properties of the Web-01a Virtual Machine.  Clicking on the web-01a vm in the Navigator window will allow you to view the properties of the VM

Take notice of the Network Adapter that the Web-01a Virtual Machine is connected to.  It should say 'VM-RegionA10-vDS-COMP'  Is this a normal VLAN Backed Port Group or did the previous Administrator promote this Virtual Machine to a new Virtual Network.  To find out, let's examine the networks that this vCenter knows about.

 

Click on the Network Icon at the top of the Navigator Screen

This will take you to the network configuration screen in vCenter

 

 

Networking Configuration

 

You will have to expand the RegionA01-vDS-COMP Virtual Distributed Switch.  Use the small triangle beside it to expand the Port Groups.

 

Click on the Configure Tab as well as the VMs tab to get more detailed information on the Network Connectivity of the Application.

 

Clicking on the Port Group VM-RegionA01-vDS-COMP and then looking at the associated VMs on this port group will reveal something interesting.  

 

All three of the Virtual Machines are in this same port group!  Given the way in which networking works, is this a good design for the most critical application in the Enterprise?

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37010

There are multiple ways to approach this issue.  The next sections will examine some of the more interesting options available to you.

 

NSX Configuration


At this point, we can not be 100% certain that this port group is a VLAN backed or Virtual Network backed port group.  In order to confirm, we can move into the network and security area of vCenter in order to see what state NSX is in at this point.

 

To do this, Locate and click on the HOME icon at the top of the screen and select Networking and Security

 

You should see this screen

 

This is where all of the main NSX Networking and Security configurations are completed.  Take some time and review the configuration of your environment.  Examine the Installation tab as well as the Logical Switches

 

 

 

NSX does appear to be installed but we still are not 100% certain that the Logical Networks are being used.  NSX will create a port group that references the name and Segment ID of the logical switch.  If we review the port groups in the Network section, we will see the following Port Groups.

 

Navigate to the Networking section in the vSphere Web Client

 

 

We can make the following conclusion based on this information:

Port Group Corresponding Virtual Network
vxw-vds-40-virtualwire-2-sid-5000-Web-Tier-LS
Web-Tier-LS
vxw-vds-40-virtualwire-2-sid-5001-App-Tier-LS
App-Tier-LS
vxw-vds-40-virtualwire-2-sid-5002-DB-Tier-LS
DB-Tier-LS

If you now look at the Virtual Machines attached to a Port Group, you will discover that the Web Server, Application Server and DataBase Server are all connected to the same VLAN Backed Port Group called VM-RegaionA01-vDS-COMP

 

Based on this, it appears that the Virtual Machines that make up our Customer DataBase Application are some of the NSX Networking features like logical switching or Networking.


Module 1 -Conclusion


You have reviewed the vSphere and NSX Environment and learnt how the existing application is currently working.  


 

You've finished Module 1

 

Congratulations on completing Module 1.

If you are looking for additional information on NSX, try one of these:

Please continue with Module 2 or proceed to any module below which interests you most.

 

 

Module 2 - Protecting the Application (45 Minutes)

Fingerprinting the Application


Given that all of the Virtual Machines that make up our application are not currently using the Virtual Networking or any type of security, it is our task to start to develop a plan to increase the security and availability of the application.  In this section, you will use of the most common tools that bad actors will use to develop a picture of a potential target.  You will also examine the potential mitigation steps that can be taken to increase security and offer better availability.

 

 

 

 

  1. Click on the blank space to open a new tab
  2. Click on Region A in the BookMarks
  3. Choose Site A Web Server

 

 

The application should open as above.  What is the first thing that you notice?  Was there any authentication required to read the data?  No, the developers choose to allow access without a username and password.  So, this means as long as anyone has IP level access to the application, they can at least access your most sensitive data!


 

Reconnaissance

How will a bad actor (hacker) view this application?  It appears that as long as you have access to the network, the application can be accessed.  Let's run some of the more common tools to understand what can be seen by just having IP level access to the servers.  There is an instance of one of the more common attack tools, called Kali Linux Running in the environment.  Logging into this via SSH will offer you a vast set of tools that are most commonly used to perform the reconnaissance that is needed to launch an attack on a site.

 

Please note that this is a tightly controlled environment and there is no outside access with these tools.  This is a unique opportunity to try certain types of attacks that you might not want to run in your environment.  There is no risk of causing real issues outside this Lab.  These types of programs should never be run against systems where you do not have express consent to do so.  Let's get started

Click on the putty shortcut on the desktop

 

 

  1. Click on Kali Linux to select the profile
  2. Click on Load the Kali Linux Profile
  3. Click Open to start the SSH Session

 

This will open a terminal session to the Kali Linux System.

 

You will be asked to enter a password for the admin account

 

The password VMware1!

 

 

 

 

 

 

 

Kali Linux - Ethical Hacking Distribution

Take a few minutes to look around at some of the tools available to you.  There are several categories of tools that can be used.  The high level categories are:

In this lab, we will focus on Information Gathering only but you are free to poke around and look some of the more common Vulnerability Analysis tools as well as Web Application and Exploit tools.

The most common tools will be Nmap, Metasploit, Maltego, SET (Social Engineer Toolkit) and BEEF.  There are wonderful tutorials all over the internet on how to use these tools.

 

 

 

NMAP

Let's start with a simple scan of the web site itself.  To do this, let try out Nmap.  

Nmap ("Network Mapper") is a free and open source utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. It was designed to rapidly scan large networks, but works fine against single hosts.

Enter the following command into the Kali Linux Terminal Session that you just opened in PUTTY

 

nmap -v web-01a

 

While this is not completely unexpected, the output does show that there are two web ports; Port 80 (Unencrypted HTTP) as well as Port 443 (Encrypted HTTP) as well as port 22, commonly used for SSH traffic.  This is just the start of what can be learnt from this tool

Next, try this command

nmap -v -A  web-01a

This command will take some time to complete.

This might be a bit of a surprise.  The SSH server is running OpenSSH Version 7.1 and the webserver is running NGINX version 1.10.0

 

 

 

 

Exploitable?

 

This might be a bit of a surprise to you.  Your scan was able to determine the versions of the services running on those open ports.  The SSH server is running OpenSSH Version 7.1 and the webserver is running NGINX version 1.10.0

Some quick google searches will show that these are mostly update to date applications without any serious vulnerabilities.  Please note that there is no internet access from within the lab itself.

 

 

 

 

Results

These are the exact steps that an attacker will undertake to map out your environment and look for any vulnerabilities to exploit.  This reflects part of why Security is so hard; The good guys have to be right all the time!  The bad guys just have to be right once!  The odds are really stacked against the good guys in this game.

The good news is that currently, there are no significant exploitable vulnerabilities in the application itself.  This is not to say that something might be found that could led to an exploit being developed down the road. And while patching remains the most important task in the security arsenal, there are several times when patches simply can't be applied in a timely fashion.  So what are we do to?

One of the most basic steps that we can take is to narrow down the attack surface.  In this example, given that we are using traditional VLAN backed port groups, we are still able to construct a security policy that can limit who was access to what.  Some examples of things to consider might be:

 

Spend the next 30 or minutes building a Micro-Segmentation Policy to implement against the application.  Your access from the Main Console reflects the access that an average user will have to your application.  Ensure that the application still responses to requests and consider limiting access to everything else.  The lab will not show any steps for you to follow but if you need a refresher on how to profile an application, consider review in the content of HOL-1803-02-NET( VMware NSX - Distributed Firewall and Micro-Segmentation.

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37011

 

Implement Micro-Segmentation using the NSX Distributed Firewall


The NSX Distributed firewall (DFW) is a hypervisor kernel-embedded firewall that provides visibility and control for virtualized workloads and networks. You can create access control policies based on VMware vCenter objects like datacenters and clusters and virtual machine names; network constructs like IP or IPSets, VLAN (DVS port-groups), VXLAN (logical switches), security groups, as well as user group identity from Active Directory. Firewall rules are enforced at the vNIC level of each virtual machine to provide consistent access control even when the virtual machine gets vMotioned. The hypervisor-embedded nature of the firewall delivers close to line rate throughput to enable higher workload consolidation on physical servers. The distributed nature of the firewall provides a scale-out architecture that automatically extends firewall capacity when additional hosts are added to a datacenter.

Micro-segmentation is powered by the Distributed Firewall (DFW) component of NSX. DFW operates at the ESXi hypervisor kernel layer and processes packets at near line-rate speed. Each VM has its own firewall rules and context. Workload mobility (vMotion) is fully supported with DFW, and active connections remain intact during the move. This advance security capability makes the data center network more secure by isolating each related group of virtual machines onto a distinct logical network segment, allowing the administrator to firewall traffic traveling from one segment of the data center to another (east-west traffic). This limits attackers ability to move laterally in the data center.

At a high level, your tasks should include:

There are several methods that can be used to increase security.  At this point, all Virtual Machines are in the same VLAN backed port group.  The CIO will not allow you to take an outage to the application as this is the busiest time of the year.  Consider what options exist to improve security meeting the goals as above and proceed with this plan.  You will want to ensure that the uptime of the application is maintained at all times when doing these changes.

 

 

 

 

If you examine the default posture of the NSX DFW, you will notice that the default rule is set to allow.  This is by design so as not to cause issues when first installing the VMWare NSX Product.  It can be a great way to start learning about the application, all you would have to do is enable logging and watch what the application does.

There are however a number of other tools included with NSX that can also assist you with the development of the less privilege ruleset.  Please review HOL-1803-02-NET - VMware NSX - Distributed Firewall and Micro-Segmentation if more details are required to profile the application and create a Firewall Policy that is based on Least Privilege.

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37011


Reviewing the FingerPrints of the Application


Now that you have successfully implemented VLAN backed Micro-Segmentation of the application, let's go back and review the external finger print of the application.

 

Log back into the Kali Linux Machine via PUTTY and check the foot print of the Web Server.  Were you able to limit the amount of information that an outsider can obtain?  Does Port 22 on the web server still respond?  As a normal end user, should you even be able to see any open ports on the Application or DataBase server?  

 

Have you considered outbound filtering?  What does the application need to access to perform it's tasks?  Are there external sources of information that are being polled for?  Did the tools that you used for Application Profiling show any requirement for external access?

 

 

This is an example security policy that might be used in protecting the application at your company.


 

Dynamic Security Policy

This is but one example of a firewall policy that will offer the least amount of 'surface area' from an attacker's perspective.  This might be considered a good first step but there are some other options to consider.  In this lab, there is only one server in each tier.  This potentially does not make for a very highly available application.  Most applications will use multiple servers in each tier, often front ended with some type of load balancer.  Creating Firewall rules based on machine name or IP address does not take full advantage of all of the capabilities of NSX.  Consider using something like this:

 

 

 

This will create a Dynamic Object that auto includes any servers that are classified as web servers.  This means that as you scale up or scale down on the number of web servers, the firewall policy will automatically be updated.  Remember that Attackers have to be right only once and the Good Guys have to be right all the time, these types of rules will allow us to focus on all of the other tasks in security that we don't ever get to because of all of the adds, moves and changes that managing a data center demand.

 

There are several other bits of MetaData that we can key in on using Security Groups in NSX.  These include:

Therefore you could add a firewall rule that looks for an Active Directory Security Group and if the user is a member of the Web Server Administrator Group, then SSH access could be granted only to those users.  That would ensure that the footprint available to the general users would not include port 22 from even showing up in the scan.

 

Module 2 -Conclusion


You have reviewed the Mission Critical application that your new Employeer is using and have implemented a Least Privilege approach to securing the application.


 

You've finished Module 2

 

Congratulations on completing Module 2.

If you are looking for additional information on NSX, try one of these:

Please continue with Module 3 or proceed to any module below which interests you most.

 

 

Module 3 - Configuring the Virtual Network (30 Minutes)

Overview of the NSX Virtual Networks


Up until this point, the application has been using a more traditional VLAN backed Port Group.  This really excludes most of NSX's features with the exception of being able to Micro Segment based on VLANs.  You were able to protect the application without having to modify the application in any way.  This is a huge benefit to your organization.  However, to be able to take advantage of the Virtual Switching and Routing capabilities, you need to be able to 'Promote' the application to use a Virtual Network.

 

 


 

Logical Switches

As a reminder, there are several Logical Switches that have been created by the previous administrator.  These will correspond to Port Groups in vSphere that you can attach Virtual Machines to.

 

If your application owners are like most places, their technical dept owing on the application is huge.  Re addressing some applications may not even be possible in that all of the dependancies within the application are unknown.  So what options are even available to make this transition?  There are a few strategies that are available to you.

 

 

Click on the Green + sign and you will be able to create a new Logical Switch called Consolidated-LS

 

You will then see a Port Group like this

 

 

You can move your three VMs to this new port group and avoid having to readdress any of the Virtual Machines that make up your application.  This can be a great way to access some of the more advanced Network and Security features with in NSX.

 

 

However, if you try moving the VMs to the new port group, you will discover that the VMs and the Application will become unresponsive.  This is because there is no connectivity between the new logical Switch and the outside world.  In the typical design, a Distributed Logical Router (DLR) and Edge Services Gateway are used to provide the connectivity to the outside world

 

 

In this case, you could consider moving the entire 192.168.120.0 subnet to the newly created Logical Network and use routing to access from the External Network to the application.  There are also other design options available like Network Address Translation (NAT) that can be employed if you're unable to completely move the entire subnet.  The next module will examine some of these options.

 

 

 

Using the Virtual Network


The previous admin must have been planning to use the Virtual Networking features of NSX.  There are several Logical switches, a Distributed Logical Router and an Edge Services gateway all configured up waiting for you.  Have a look, navigate to the Networking and Security Tab from the vCenter console.  You can always access this by clicking on the HOME icon on the Main Page and Selecting Networking and Security 

 

 

Take a minute to review the tabs.  Try clicking on the Logical Switches tab as well as the NSX Edges Tab to see what the previous admin has done.

 

 


 

Using the Virtual Network

There also appears to be a complete copy of the Virtual Machines that make up the application that are ready go be tested.  Click on the home button and select Hosts and Clusters to bring you to this screen:

 

 

  1. Right Click on Web-01b and choose Power, then click Power On
  2. Right Click on App-01b and choose Power, then click Power On
  3. Right Click on DB-01b and choose Power, then click Power On

This will launch the new copy of the Application.  You can test this out by accessing the URL in the Region A Bookmark called NewSite

 

 

Your predecessor left a diagram on the desk that show what is configured so far

 

 

It turns out that this is a complete copy of the main application that the company uses in the day to day operations.  There are some issues to consider here:

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37012

 

Connectivity to Outside world


Now that the new copies of the application VM's are powered up and running, check to see if the application is working.

 

Open a new browser tab and choose the shortcut labeled NewSite

 

 

If you see a similar screen to this one, the second copy of the application is up and running

 

 


 

Exploring the Test Application

It does appear that someone was able to successfully modify the application to run on a new set of IP addresses.  There will be times that it is simply not possible to change an applications IP address.  Is cases like these, Network Address Translation can be used as a potential strategy to overcome the old addresses.

You will find a new logical switch in the environment called Consolidated Logical Switch.  It is possible to migrate the orignal VM's to this new consolidated virtual switch and use NAT to have the users access it using a new IP Address.  The orignal IP Networking is persevered within the application and thus no changes are required.  You also should have noticed that all traffic for the application is totally self contained and no external access was required.  

 

Now, take some time to examine the path to this new copy of the application:

Start a command prompt - look for the Icon in the task tray in the bottom Left hand corner:

 

 

tracert -d web-01b

 

You will see that there are several Hops before the web server is actual contacted.

 

What are the IP addresses 192.168.100.3 and 192.168.5.2?

 

To answer this, let's examine the two NSX Edge's we saw earlier.  Navigate to the Networking and Security tab, and choose edge-1

 

 

 

 

Exploring the Virtual Network

 

Double Clicking on edge-1 will reveal

 

Choose Interfaces and you should see one of the addresses that are in use

 

We are starting to build up the picture of how this is all put together.  Spend a little more time on edge-1 and examine the routing configuration as well as the firewall configuration.  When done, click on the back arrow to go back to the NSX Edge

 

You may have to click back a few times to get back to this screen.

 

Let's click on edge-2 and see what is configured

 

Clicking on Interfaces should complete the diagram for you

 

 

This brings us to the end of the module.  In the next module, you will have some real world challenges to overcome all in an effort to make your application more available and stable.

 

Module 3 -Conclusion


You have completed the implement of Logical networking in NSX.  The test copy of the application is running on NSX Logical Networks and is ready for additional services and bandwidth to be applied.


 

You've finished Module 3

 

Congratulations on completing Module 3.

If you are looking for additional information on NSX, try one of these:

Please continue with Module 4 or proceed to any module below which interests you most.

 

 

Module 4 - Network Tasks (30 Minutes)

Adding Resiliency to the Environment


In the last module, you were able to develop a good understanding of how the current application is configured as well as how the new version of the application might look when you are ready to get it into production

 

As a review, here is what the work looks like up to now;

 

 

After reviewing this diagram with your manager, it was noted that there is no redundancy in the uplink to the physical router.  As the NSX Edge GW is simply a Virtual Machine, it is easy enough to add a second instance of the edge.  Given that your manager would also like to offer additional bandwidth, the choice to add a backup edge seems less effective.  To review the options for High Availability and Additional Bandwidth

 


 

Options for Additional Redundancy

 

Edge High Availability ensures that the services provided by NSX Edge appliances are available even when a hardware or software failure renders a single appliance unavailable. Please keep in mind that NSX Edge HA is not a fault tolerant solution, but it helps to minimize failover downtime.

The high availability provided is stateful, meaning that NSX Edge HA synchronizes the connection tracker of the stateful firewall or the stateful information held by the load balancer.

Primary and secondary NSX Edge appliances are respectively inactive and standby states, and all services run on the active appliance. The primary appliance maintains a heartbeat with the standby appliance and sends service updates through an internal interface.

If a heartbeat is not received from the primary appliance within the specified time (default value is 15seconds), the primary appliance is declared dead. The standby appliance moves to the active state, takesover the interface configuration of the primary appliance, and starts the NSX Edge services that were running on the primary appliance.

 

It is possible to add the High Availability option to an existing and running Edge Services Gateway.  To do this, Navigate to the Networking and Security Section from the Home Icon and Click on the Edge-01

 

Double Click edge-1

 

Choose the Change Icon in the HA Configuration Screen

 

 

 

Choose Enable and Select the vNIC Transit

 

You will have to select a timer value of at least 3.  This will ensure that NSX will not declare an Edge being down prematurely

 

Select OK and wait for the changes to take effect.  This will take a bit of time as a new edge is being deployed and for the configuration on both edges to take effect

ping -t web-01b

start a continuous PING to the new web server and once the HA status goes to UP

 

with the PING going, go back to the hosts and clusters view and shutdown the VM edge

 

Once the Primary edge is shutdown, the backup edge will take over and the ping will resume.  It is important to note that this failure is in fact stateful meaning that any firewall or load balancer states are moved to the backup upon the primary edge failing.  The Active / Passive nature of HA does not however increase bandwidth to the application.

 

Once you are done with this testing, close the continuous PING and restart the vm edge-01-0

 

Additional Edge Services with the Edge Services Gateway


NSX Edge provides network edge security and gateway services to isolate a virtualized network. The NSX Edge gateway connects isolated, stub networks to shared (uplink) networks by providing common gateway services such as DHCP, VPN, NAT, dynamic routing, and Load Balancing. Common deployments of NSX Edge include in the DMZ, VPN Extranets, and multi-tenant Cloud environments where the NSX Edge creates virtual boundaries for each tenant.

The Edge Services Gateway has some other features that you may wish to consider implementing in your efforts to add stability and resiliency for the application.  The features of the Edge Services Gateway include:

This module will focus on using the NSX Load Balancer to allow future additional Virtual Machines to be adding offering better overall capacity and availability.  

 Navigate back to the Networking and Security Section in the vSphere Web Client

 

Click on edge-1 to enter the configuration for this Edge.  You will notice that even though we have an HA pair of Edge's, configuration only needs to be applied once.

 

Choose NSX Edges an Click on edge-1

 

Choose Manage an Select Load Balancing

 


 

Configuring the Load Balancer

Click on Load Balancer and let's review the options for creation of the Load Balancer to front end our public facing web server.  Just a quick review, the NSX Load Balancer offers:

Our application can benefit from Layer 7 Load Balancing, using HTTPS as well as some of the advanced Processing rules that can be applied to the inbound traffic.  Go ahead and setup a Layer 7 or Socket Based Load Balancer

 

 

As a note, the next few dialog boxes don't fit in a 1024x768 screen.  You will want to use the Browser Zoom Feature to give you a little more screen realestate in order to complete the Load Balancer configuration.

 

1 Click on the Browser Option Tab in the Far Upper Right Hand Corner

2 Click the "Minus" Icon to reduce down to 80%

 

Load Balancing, NAT and Firewalling are all tightly linked in terms of features,  You will therefore need to turn on Firewalling

 

Please note that simply clicking enable does not Apply the change to the edge.  You must click Publish Changes to allow the change to take effect

 

 

The Firewall is now enabled but will have a permit any any policy at this point.  NAT and Load Balancing require the firewall to be enabled

Next Turn on Load Balancing globally

 

Enabling Load Balancing and Acceleration will allow for any configuration to be used.  It is a good practise to enable Logging here as well to give the required metrics for any of the features that Load Balancing can perform.

Next, configure an Application Profile

Remember to use the zoom feature on the browser to allow you to see the entire dialog box.

 

 

Give your Application Profile an name and choose HTTPS.  Don't forget to Enable SSL PassThrought as we won't be doing any SSL intercept in this module.

 

Next, take notice of the Service Monitoring Profiles.  We will leave everything here as default

 

Now, create a Pool.  This Pool will reference the web servers that we are going to load balance.  As of now, there is only one copy of the web server (web-01b) but there seems to be plans to add at least two more web servers in the near future.  Add a pool that will automatically add or remove any web servers in the environment.

 

 

This will add the pool to the load balancer

Lastly, we need to enable a Virtual Server.  In order to do this, we will need an IP address that exists on the outside of the NSX Edge.  Click on Settings and Then interfaces and add the IP Address 192.168.100.234 as a secondary IP Address

 

 

You will immediately see that this IP will start to respond to PING requests.  

Now, let's complete the Virtual Server  Click on Load Balancer, Virtual Servers and add a new Server

 

  1. Enable the Virtual Server
  2. Enable acceleration
  3. Reference the Application Profile  (MyApp)
  4. Give the Virtual Server a name (Application)
  5. Click Select IP Address to click on the IP address of 192.168.100.234 that you just added to the Edge
  6. Choose HTTPS for the protocol
  7. Reference the Pool that you created (web)

 

In order to test this configuration, try going to the following URL

 

https://192.168.100.234/cgi-bin/app.py

Accept the certificate warning and this should open the usual Application

 

This is the most basic form of load balancing.  It is sometimes called transparent mode.  One or more real servers are used to answer inbound requests.  The Edge will use a DNAT translation to send the traffic to a real server.  The return traffic will have the original VIP substituted back in order to maintain the state.  With NSX, you can also have a firewall running on the Edge as well as at the workloads such as those web-01b.   This Firewall policy will prevent anyone but the load balancer from accessing the servers directly.

Load Balancing can be done on VLAN and VXLAN backed networks

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37013

 

Adding Capacity and Resiliency


Up until now, the changes that you have made have only really added higher availability.  No additional bandwidth has been added.  There is one option that will add both availability as well as increased bandwidth.  This would be do deploy two or more NSX Edge Services gateways and run a routing protocol with ECMP enabled.  This would give us something like this:

 

 


 

Adding Capacity with ECMP

As you will need Edge-01 in an upcoming module, you can safely leave it unattached as you add the two new edges.  Once the new edges are added, you will reconfigure the transit network connections to attach the DLR to the two new edges.  The best way to accomplish this will be to use the disconnect interface feature on the edge-01

 

 

 

 

 

Create two new ESGs

Add two new NSX Edges, configured as Edge Services Gateways.  Please name than as ecmp1 and ecmp2

 

 

 

 

 

 

 

 

Attach the newly created ESGs

Once this new Edge is installed and configured, ensure that the internal Interfaces are configured to attach to the Distributed Logical Router

 

ecmp2 will use an Uplink to the VM-RegionA01-vDS-MGMT Distributed Port Group and its IP address will be 192.168.100.5/24

The internal uplink connected to the same Logical Switch (Uplink-A-LS) but will use a primary address of 192.168.5.5/24

 

The OSPF configuration will mirror the first edge with the following configuration:

 

 

 

 

repeat the previous configuration steps for the second ESG ecmp2

 

 

Double the bandwidth

 

Now, the goal is to have more than one path available for traffic to be able to flow over.  If our uplinks from the management cluster to the Physical Router were 10 Gig, you just doubled the bandwidth to 20 gig.  NSX supports up to 8 ECMP Edges providing up to 80 Gigabits of bandwidth!

 

In order to ensure that your configuration is correct, Open an SSH session to the NSX Manager and use the central CLI to examine the routing tables

 

 

 

Choose

nsxmgr-01a.corp.local and choose load and then open

 

The password is VMware1!

 

show edge all

 

You can see the all of the currently running edges in the environment.  Let's examine the routing table of the DLR or edge-2

show edge edge-2 ip route

 

If your configuration is working, you should see two routing table entries for each IP address.  As above, the default routes out of the Virtual Environment are via the two edges 192.168.100.1 (ecmp1) and 192.168.100.5 (ecmp2)

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37013

 

 

 

Module 4 -Conclusion


You have added additional capacity and bandwidth for your application to utilize.


 

You've finished Module 4

 

Congratulations on completing Module 4.

If you are looking for additional information on NSX, try one of these:

Please continue with Module 5.

 

 

Module 5 - Troubleshooting (30 minutes)

Review of the Tools available


There are a number of tools that are included with NSX that will help with the Day 2 Operations of the Platform.  These include:

These tools are covered in HOL1803-03.  There is also a supplementary tool called vRealize Network Insight (vRNI) that exists as a standalone tool.  Please visit HOL-1829 for more details on this tool.  Please take some time to review these tools in your environment and once ready, proceed to the next section and troubleshoot some issues with the application.

 

 

 


 

Application Rule Manager

 

VMware NSX 6.3 introduced a new toolset within NSX that can be leveraged for quick micro-segmentation planning. The Application Rule Manager (ARM) within NSX, provides a new way to help create security rulesets quickly for new or existing applications on a bigger scale than Log Insight, but smaller scale than vRNI. With that in mind, you can use ARM in NSX to create our rulesets using the same basic methodologies for an application.

The Application Rule Manager in VMware NSX leverages real-time flow information to discover the communications both in and out, and between an application workload so a security model can be built around the application. ARM can monitor up to 30 VMs in one session and have 5 sessions running at a time. The beauty of ARM is that it can correlate the information that you would typically have to either have or look up in Log Insight to create your rulesets and significantly reduces time to value. ARM can also show you blocked flows and the rules that are doing the blocking.

 

 

Traceflow

 

Traceflow adds functionality to NSX that provides assistance with the Operations of the NSX Network Virtualisation platform. Traceflow allows the injection of varying types of packetsinto application topologies. As the name suggests traces the flow through the path. It collects observation of actions, hosts, relevant components, and their names. This is used to help administrators visualise a topology path.  As this uses Synthetic traffic, it is non distribute and allows an administrator to test the network and security of an application from one VM all the way to another VM.

 

Routing and Firewall Troubleshooting


In order to save some time, please run the following PowerShell Script

  1. Open Explorer
  2. Navigate to C:\HOL
  3. Locate script mod5.p1
  4. Right Click on mod5.ps1 and choose Run With PowerShell
  5. Wait for the script to finish

 

 

 


 

Troubleshoot OSPF

  1. Open the console for the newly created ESG "TroubleshootingESG-0"
  2. Login to the ESG with the username admin and password VMware1!VMware1!
  3. Use the built in ESG commands to troubleshoot the connection to the TroubleshootingDLR-0
  4. Continue to troubleshoot until the ESG and DLR show as neighbours in the OSPF neighbour table

To view the solution click here

 

 

Troubleshoot DFW

The Distributed Firewall has been configured to allow access to the VM web-01a only on HTTPS.  When testing the environment, ICMP from the Main Console is still allowed.  Verify this and correct the issue

  1. Open the command prompt on the Main Console
  2. Ping the IP for web-01a from the Main Console
ping web-01a

 

  1. Ping web-01a.corp.local
ping web-01a.corp.local
  1. Note that the ping is sucessfull
  2. Correct the Distributed Firewall Configuration to allow only HTTPS inbound to web-01a

To view the solution click here

You can also view hints on the VMware Technology Network Communities here:

https://communities.vmware.com/docs/DOC-37014

 

 

Prevent Lateral Movement

Configure and test the environment

 

  1. Open Windows explorer
  2. Navigate to c:\hol\
  3. Right click on "Check Web-DB.ps1"
  4. Select Run With Powershell

 

  1. Observe the results
  2. Configure the DFW to dissallow communication between the application VMs (web-01a, app-01a, and db-01a) while still allowing the application to function.  Use the built in tools to configure the segmentation.  For the lab to check correctly do not disallow SSH from the Main Console to web-01a.

To view an example of one solution to the problem click here

 

 

Correct OSPF Settings

  1. Observe the output of the debug commands
debug ip ospf
  1. Note the mask and timer mismatch
  2. Correct the settings on the ESG
  3. Stop the debuging of OSPD on the ESG
no debug ip ospf
  1. Show the OSPF neighbours
show ip ospf neighbours
  1. Observe the neighbour

 

 

Correct DWF Setting

  1. Navigate to the Distributed Firewall Exclusion list
  2. Remove web-01a from the exclusion list
  3. Ensure the DFW rules are configured as below

 

 

 

 

 

Correct Segmentation Settings

Here is a reminder of the appropriate Micro-Segmentation based rules for the Application:

 

 

 

 

Module 5 -Conclusion


You have looked at some of the common troubleshooting tasks that an NSX Admin is likely to encounter.


 

You've finished Module 5

 

Congratulations on completing Module 5.

If you are looking for additional information on NSX, try one of these:

 

 

Conclusion

Thank you for participating in the VMware Hands-on Labs. Be sure to visit http://hol.vmware.com/ to continue your lab experience online.

Lab SKU: HOL-1892-01-CHG

Version: 20180202-123744