HOL-1842-01-NET - VMware AppDefense - Secure Datacenter Endpoints
Note: There are only two modules in this lab. The expected time of completion is 45-60 minutes.
The Table of Contents can be accessed in the upper right-hand corner of the Lab Manual.
VMware AppDefense is a datacenter security tool that enables Application Control, Detection, and Response. It is meant to provide foundational elements of Cloud Workload Protection, such as System Integrity, App Control, and Memory Monitoring.
Lab Module List:
Lab Captains & Support:
This lab would not have been possible without the dedication of the AppDefense engineering team. Their support and assistance to make this something suitable for the VMworld HOL environment was instrumental. We would like to thank the following:
This lab manual can be downloaded from the Hands-on Labs Document site found here:
This lab may be available in other languages. To set your language preference and have a localized manual deployed with your lab, you may utilize this document to help guide you through the process:
During this module, you will input text into the Main Console. Besides directly typing it in, there are two very helpful methods of entering data which make it easier to enter complex data.
You can also click and drag text and Command Line Interface (CLI) commands directly from the Lab Manual into the active window in the Main Console.
You can also use the Online International Keyboard found in the Main Console.
Please check to see that your lab is finished all the startup routines and is ready for you to start. If you see anything other than "Ready", please wait a few minutes. If after 5 minutes your lab has not changed to "Ready", please ask for assistance.
When you first start your lab, you may notice a watermark on the desktop indicating that Windows is not activated.
One of the major benefits of virtualization is that virtual machines can be moved and run on any platform. The Hands-on Labs utilizes this benefit and we are able to run the labs out of multiple datacenters. However, these datacenters may not have identical processors, which triggers a Microsoft activation check through the Internet.
Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoft licensing requirements. The lab that you are using is a self-contained pod and does not have full access to the Internet, which is required for Windows to verify the activation. Without full access to the Internet, this automated process fails and you see this watermark.
This cosmetic issue has no effect on your lab.
Module 1- Overview of VMware AppDefense (15 Minutes)
In this section, you will read about VMware's new AppDefense security platform.
AppDefense is a data center endpoint security product that embeds threat detection and response into the virtualization layer on which applications and data live. Leveraging VMware vSphere®, AppDefense delivers three key advantages over existing endpoint security solutions:
Authoritative knowledge of application intended state When you know what’s good, you can detect what’s bad.
From inside the vSphere hypervisor, AppDefense has an authoritative understanding of how data center endpoints are meant to behave and is the first to know when changes are made. This contextual intelligence removes the guesswork involved in determining which changes are legitimate and which are real threats. AppDefense does not look at a guest workload in isolation. Instead, it manages workloads as part of broader “Security Scopes”. These scopes allow AppDefense to have a deeper understanding of complex interactive behaviour patterns in the data center as opposed to simply individual machine behaviour.
Automated, precise threat response The right response at the right time. When a threat is detected, AppDefense can trigger vSphere and VMware NSX® to orchestrate the correct response to the threat, without the need for manual intervention. For example, AppDefense can automatically:
• Block process communication
• Snapshot an endpoint for forensic analysis
• Suspend an endpoint
• Shut down an endpoint LICATIONS IN VIRTUALIZEDAND CLOUD ENVIRONMENTS WITH VMWARE APPDEFENSE
Isolation from the attack surface The platform allows us to protect the protector. The first thing that most malware variants do when they reach an endpoint is disable antivirus and other agent-based endpoint security solutions. The hypervisor provides a protected location from which AppDefense can operate, ensuring that even if an endpoint is compromised, AppDefense itself is protected.
The primary components of the VMware AppDefense platform are:
The AppDefense Manager console is a multi-tenant cloud service provided instance to define the intended behaviour and protection rules of your applications in one place. You can monitor the enforcement of configuration, security events and alarms from here.
The AppDefense Appliance is an on-premises based control point for ingress and egress of data from and to the Manager. It brokers connections to the VMware management components (e.g. vCenter) and makes outbound connections to the AppDefense Manager.
The AppDefense Guest module deployed in the customer VM, along with supporting AppDefense Host Modules (in the form of VMware Installable Bundles) deployed on the ESX host. These two components work in concert to monitor and enforce the intended state of the guest behaviour as well as ensure that the protection controls are isolated in the hypervisor away from the guest "attack surface".
vCenter is used to gather inventory data on the customer’s site. This inventory data is used for security scope assignment, guest readiness (based on OS information) and guest to host assignment. AppDefense can also use vCenter to perform remediation actions in response to security events, such as suspending a guest.
NSX (Optional) is used as an additional, optional remediation channel for AppDefense. Specifically, NSX can be used to automatically or manually quarantine the machine(s) if any of the protection rules are violated.
vRealize Automation/vRealize Orchestrator (Optional) can be optionally used to capture application context at provisioning time from the Application blueprint.
The AppDefense platform provides:
AppDefense's operation from within the hypervisor also provides protection and abstraction not available with traditional end-point protection platforms creating the most effective least-privilege model for the application layer.
Protection of the Protector in a Separate "Trust Zone"
Kernel Level Monitoring
Memory and Process Monitoring
Unlike other endpoint security products, AppDefense is isolated from the attack surface without sacrificing the context necessary to provide accurate security alerts. Furthermore, AppDefense works with NSX and other infrastructure control points to automate the response to detected threats, minimizing the potential for data exfiltration and the impact to the business.
Congratulations on completing Module 1 -- An overview of the VMware App Defense Platform.
Proceed to the next module:
Lab Module List:
If you would like to end the lab now, you can simply click the "End" button in the upper part of your screen. Otherwise, please proceed to Module 2.
Module 2 - Exploring & Utilizing the AppDefense Platform (45 Minutes)
We will log in to both vSphere and the AppDefense Manager in preparation for the lab.
Launching Chrome should take you automatically to the vSphere Web Client login page. Use the following credentials to access the web client.
2. CLICK Login
NOTE: If after a few minutes, your browser still does not take you to the login page, simply refresh the browser session.
This action will result in a new tab launched in Chrome to the "AppDefense Manager" VM.
3. CLICK On the new tab. (not shown in the visual)
Note: Depending on how long the Lab Environment has been waiting prior to beginning the lab, you may be taken immediately to the main AppDefense dashboard, if that is the case, you can skip to this step. If you are presented with a main logon page, then proceed to the next step.
2. CLICK Sign In
In this section, we will walk through the AppDefense Manager and the various sections of the User Interface.
As was mentioned in the overview module, the cloud manager is a multi-tenant service that defines the intended behaviour and protection rules of your applications in one place. In the current version, this manger runs in the public cloud as a service. This is where customers will monitor the enforcement of configuration, security events, and alarms.
After VMware establishes your account and tenant in the AppDefense Cloud Manager, you will receive an email to join the service. Once you have joined the service, you can log in and connect appliances, as well as perform other platform configurations.
For the purpose of the VMware Hands-on-Labs environment, however, this manager is running in a local instance. The user "firstname.lastname@example.org" has been created under its own tenant. You are currently logged in under this user's account.
In this default view, you can see Protection Coverage, Security Scopes, Alarms and Events. You will explore these in more detail as we proceed through the lab.
Note: The Provisioning Events shown are artifacts from the construction of this lab. You can disregard them.
On the Manager in the bottom left corner, you will see a settings icon next to the name "email@example.com".
The AppDefense appliance is a single, on-premises control point for ingress and egress of data to and from the AppDefense manager. It brokers connections to the VMware management components (like vCenter) and makes outbound connections to the AppDefense Cloud Manager.
The AppDefense appliance is deployed at the customer site through a standard .ovf deployment workflow. Once the appliance is deployed, the user will log into the AppDefense Cloud Manager and connect the appliance to the their tenant. In addition, the appliance will be connected to various sources in the datacenter (e.g. vCenter or NSX).
Return to the settings menu in the lower left hand corner. You will see a little gear icon next to the name "firstname.lastname@example.org".
In this view, you can toggle between the ESXi hosts and the VMs that are available within the inventory and determine whether or not their respective modules are installed, OS versions, etc. This is accomplished by simply clicking on the "Hosts" or "VMs" tab at the top of the menu.
Remember that AppDefense is watching at the guest and host level for activity, corruption or any other anomalous behaviour.
Return to the settings menu in the lower left hand corner. You will see a little gear icon next to the name "email@example.com".
This view shows the virtual machines in the inventory that have not been currently assigned to any security scopes. It will also show the current operational status of the host and guest modules for those particular virtual machines.
The orange and red areas represent VMs that are either in discovery mode or under protection.
The On-Prem build of the AppDefense Manager used in this lab does not support automatic downloads, so the image on this step is from the actual production cloud based AppDefense Manager. You can see that all documentation, ova files, VIBs and guest modules are available in the management portal itself.
A Security Scope in AppDefense is the foundational component that establishes what the intended state and specific allowed behaviors of an application should be. In this section, you will walk through the workflow of creating a security scope against a small test application.
A Security Scope defines the relevant configuration elements to protect an application and its constituent workloads. These configuration elements constitute a "blueprint" or "birth certificate" for the application. It contains a description, member workloads, rules and behaviors.
This is fundamental to the AppDefense philosophy. By focusing on applications as opposed to just indvidual endpoints, AppDefense derives a greater contextual knowledge of the intended state of the application.
You will now create a security scope to protect the sample mortgage app running in the HOL environment.
In the AppDefense Manager tab, locate the "Scopes" section in the left hand menu.
After you click create, the UI will default to that scope highlighted in the left hand menu.
In the Service Description, you can specify other information of your choosing. This is not mandatory, but can be useful in operational environments to denote additional relevant information on the service.
AppDefense can tie into provisioning systems such as vRealize Automation or Puppet to define appropriate and allowed behaviors. It can also function in learning mode. This mode of operation allows the system to populate allowed behaviour based on a runtime view of the application over a period of time.
In this mode, the AppDefense platform will dynamically learn behaviors (e,g.ports, processes, etc.) of the targeted application. There may be some cases where you will want to specifically define allowed behaviours. For the purposes of this lab, we will not specify any behaviors in this window.
You will now execute the same workflow to create the DB service for use in the application's security scope.
This section continues to walk through the basic workflow of using AppDefense by modifying the security scope to change the alerting actions based on specific threat vectors.
As we have discussed, AppDefense creates a list of allowed behaviors (e.g. ports, processes, etc.) to build a "blueprint" or "whitelist" of the intended state of the application. AppDefense can create this blueprint with assistance from provisioning interfaces such as vRealize Automation, vRealize Orchestrator, Puppet, or similar engines.
However, when the application is already deployed, AppDefense can also "learn" these behaviors. After a Security Scope is created and applied to an application, it defaults to "Learning Mode". During this time, all relevant activity is recorded as the application is functioning.
Once reviewed, this master list of intended activity can be validated and/or modified by a security operations team or application owner. After the final intended state is determined, the security scope is placed into "protected mode".
Once the scope is moved into this mode, AppDefense will use the allowed behavior list to enforce the correct security context and posture for the workload against any deviation from that list.
DO NOT CLICK VERIFY AND PROTECT AT THIS TIME.
The types of information displayed:
Path: This is the location where the process was launched from within the OS file structure.
Hash: This is a hash value on the process. This is an extra protection in case a rogue process using a trusted name were to be launched.
Trust Score and Threat Score: These two values are provided by a backend integration with a Carbon Black reputation service for Windows based services. This integration provides insight into Behaviors that are learned which may not be known by the security team. This provides extra protection against the question "What happens if my system is already compromised during the learning process?" For Linux based systems, these scores are derived by integration from various package deployment sites. In this environment, the scores here have been pre-populated for illustration. The isolated nature of the HOL environment prevents live integration based results.
Package: Metadata would be populated based on integration with orchestration platforms such as Puppet.
Outbound and Inbound Connections: These sections provide information on what ports and addresses are being listened to and communicated across.
Note: As was stated in the previous page, when operating in learning mode, the typical time recommended is between 7-14 days depending on the application. We do not have that long in this HOL environment, so we have automated some of the learning as well as wild carded some of the process information.
In the next step, you will generate some typical traffic to observe the learning process.
Leave the Mortgage App tab open, and return to the AppDefense Manager tab.
There will be some delay in the environment between the time the Mortgage App is accessed and the additional Behaviors show up in the AppDefense manager console. This should only be a few minutes. The important element here is to see the number of learned Behaviors increment from the original 60 that were created by our automation that was described when we created the security scope.
As you can see, the act of logging into the Mortgage application generated new activity. On the DB_Tier, one of the new services discovered was the sqlservr.exe process. The details including inbound connection information can be viewed here.
To insure that a later capability is demonstrated correctly, we need to verify that the python service on the Web_Tier has been learned.
Note: If the python.exe does not show up in the Web Tier list of Behaviors, open up new tabs and generate traffic to the Mortgage App.
Putting the Scope into Protect mode changes the Behavior of the AppDefense platform. The platform will now evaluate runtime behaviors against the list or "manifest" of Behaviors that was discovered during the learning process.
In the next section, you will examine the various response routines that AppDefense can use when protecting applications.
In this section, we will explore the various response and remediation routines that are available in AppDefense as well as the integration with VMware's NSX platform.
Once the Security Scope is in "Protect Mode", you can still review and edit services associated with the scope. By default, the only actions for the Remediation Rules are set to "Alert" and the enforcement is "Automatically".
As you can see, there are 4 vectors that we can remediate against:
The default setting for these enforcement policies is set to Alert. In the next step, you will change one of those behaviours and we will examine the AppDefense integration with NSX.
The screen will launch with the view above. However, you may need to scroll down some to see all four of the remediation vectors and their options.
This will change the default behavior for Guest OS integrity issues to provide a manual quarantine option for the VM using an NSX policy. The other remediation actions (e.g. Suspend, Power Off, & Snapshot) are done directly at the vSphere/vCenter level.
It will take two to three minutes for the change to be completely updated. Now, you will look at the NSX integration with security tags while the update is occurring.
Note: If you have been logged out, you can log back into the vSphere Web Client with the following username and password. If not, you can proceed to Step 2 and 3.
This tag is automatically applied to the list when the AppDefense Appliance is connected to the NSX Manager during installation.
NOTE: Depending on resolution, you may need to re-size the Name column width.
2. NOTICE that there are no VMs in the "AppDefense Quarantine" Policy--This policy was also automatically built when the AppDefense Appliance was integrated with the NSX manager during installation.
In the resulting window, the default location of the Firewall Rules are right of the Guest Introspection Table. You may need to resize the window or navigate over to see the entire set of Firewall Rules. Once there, you should see two rules that block all services from and to the Policy Security Group VMs.
Note: Depending on your resolution, you may need to zoom out on the Chrome browser window to be able to scroll over.
The Security Group membership is based on the presence of the "AppDefense" tag that we saw earlier.
Next, you will perform some scripted attacks against the application.
Now that you have defined what the intended state of your application should be, you will use tools provided to attack the application and observe the results.
At this point, you have reviewed the installation of AppDefense and its integration with NSX and vCenter. You have created a security scope and added the web and db services to it.
In addition, we have modified rules of the Guest Integrity section of the DB service so that it would require a manual interaction prior to being quarantined by an NSX policy that was automatically built by AppDefense.
In this final section of the lab, we will generate an outbound network connection attempt which will generate an alarm against our security scope. Then we will be using some provided scripts to simulate attacks on the kernel and host level AppDefense modules . Once the attacks are executed, we will validate the alarms in AppDefense Manager and perform a quarantine of the VM. Finally, we will test to ensure the DB VM is isolated.
Minimize the browser window you have open to return to the Main Console desktop.
This will automatically log you into the DB VM of the Mortgage Test application
After a minute or so, these two windows will pop-up. You close them and proceed to the next step.
You may at some point see a pop-up window appear regarding Windows Update Service like the visual above. Simply dismiss by clicking on the "Close" button. and proceed.
Wait until you see "Server not found."
GIRogue is a process that can be used to simulate different attacks at both the Guest and Host level. The details and specifics of this tool are beyond the scope of this lab. We have scripted two different attacks against the Host and Guest level. However, before we can execute the attacks, we need to ensure that the GIRogue process is running.
Once you have clicked the shortcut, the resulting window will appear.
You can feel free to execute this 3-5 times. This will generate more alarms in the system.
You can feel free to execute this 3-5 times. This will generate more alarms in the system.
Within the HOL environment, there may be varying degrees of delay between the VMs and the Manager. It may take 3-5 minutes for the alarms to show up. Refreshing the AppDefense browser window is perfectly acceptable until you see the alarms depicted. Depending on other actions, you may see more alarms in the manager. You may also see alarms in a different order.
There are several ways to get to the alarms page for a service, or complete scope.
3. CLICK on the red number in the DB_Tier Service (your number may be different depending on timing and other actions).
You will get a confirmation window (not pictured here). CLICK on "Go Ahead" to proceed to the DB_Tier service alarms.
The number and order of your alarms may vary. You should see at a minimum both kernel and AppDefense Module Tampering alarms. These are the result of the GIRogue process attacks we executed previously.
Within the HOL environment, there may be varying degrees of delay between the VMs and the Manager. It may take 3-5 minutes for the alarms to show up. Refreshing the AppDefense browser window is perfectly acceptable until you see the alarms depicted. Depending on other actions, you may see more alarms in the manager. The key alarms you are looking for are listed above.
In the AppDefense Alarms window, find the Kernel Integrity Compromised alarm that we acted upon. You will see in the "Remediation Action" a status of the action. It will start in the Queued: Quarantine phase .
After a few minutes, the column should change to Action taken: Quarantine.
(Note: you can also refresh the AppDefense manager tab in the browser--it still may take a minute or two to show up)
Once that action has been taken, we will verify that the DB virtual machine is indeed isolated based on the NSX security policy we examined earlier.
VERIFY the "db" is present on the Virtual Machines tab in the resulting window
Given that only the db virtual machine is isolated, you should be able to still access the web tier of the application.
Since the DB virtual machine is isolated, the web virtual machine will not receive any response.
After a few minutes the web login should time out proving that AppDefense has invoked an NSX Quarantine policy.
NOTE: The application could take as long as 3-4 minutes prior to timing out and displaying the screen you see.
Return to the vSphere Client Tab. You should be on the Service Composer Menu from the previous step when we verified the security tag was added to the "db" VM through the AppDefense workflow.
You will remove the Security Tag from the VM and thus remove the quarantine policy.
In the resulting window:
If the application returns the account detail screen, then you have successfully removed the quarantine policy from the NSX platform. In the next section, we will examine what happens when application components are upgraded when in a protected Security Scope.
In this section, we will examine how upgrades of application components are identified and processed with the AppDefense platform.
First, let's verify that we see the existing Python service in the Web Tier's list of allowed behaviors.
NOTE: You may see new and other alarms in the scope. This is perfectly normal with some of the lsass windows processes and/or RDP sessions that were conducted earlier in the lab. While we attempted to wildcard most of the noisier processes in the application, there are still others that will spawn given the short time frame we are dealing with in the HOL environments. As was mentioned earlier in the lab, VMware recommends 14-21 days to learn all processes and their CLI variants.
Minimize the browser and locate the RDP short-cut to the Web VM on the desktop of the Main Console VM.
NOTE: When you log in to the VM, you may see a perl and STAF window launch. Just close or minimize these windows as they are not needed for the lab.
Locate the shortcut to the CMD prompt on the web VM taskbar.
Verify that the version is 3.5.2. This is the current running version on the web server.
You will now upgrade the python code n the web vm using a script provided.
You can now see that the version of python.exe is 3.6.4.
Now you will generate new behaviour using the upgraded python process by visiting our Mortgage App website.
You may want to perform these three steps a couple of times, but it is not required.
NOTE: Depending on some factors, it could take as long as 3-5 minutes before this alarm shows. If you find that to be the case, you can refresh the AppDefense console page and/or generate more instances of the web traffic.
In this page, you can see some of the details associated with what AppDefense believes to be an upgrade scenario. The arrow in the graphic highlights the process hash that was observed. Compare the last 4 digits of this hash to the one you noted at the beginning of this section and you will that they are different. This is one of the key indicators that we have a new process albeit similar and performing similar functionality.
When AppDefense is determining whether a new process is an upgrade scenario, we look at several factors. Some of these factors are:
This allows us to reduce potential false positives of an "intended state" security model through the lifecycle of the application.
1. CLICK Allow Behavior in the upper right-hand corner
Return to the Web_Tier of the HOL-App Scope and see the new python behaviour added to the Security Scope.
We do not automatically remove the old process list. We keep it for history and because there could be other VMs in the scope that still are using the older version with the older hash. The scope can be manually edited if needed.
NOTE: Your alarm numbers may vary than what is shown on the screen shot.
In this module, you went through the basic workflow of creating security scopes, service definitions and remediation policies. You then simulated an attack on a test application and observed VMware AppDefense quarantine the virtual machine using VMware NSX security policies. Finally, you upgraded a application component to see how AppDefense deals with upgrade scenarios in an intended state model.
If you would like more information on VMware's AppDefense, please check out our products page at: www.vmware.com/appdefense
You can proceed to any module in the lab below.
Module 1 - Overview of VMware App Defense (15 minutes) - Basic - This module will walk you through the structure of the platform.
Module 2- Exploring and Using the AppDefense Platform (45 minutes) - Basic - This module will walk you through the creation of a security scope. Secondly, you will monitor the application after various attacks have been made. Finally, you will perform remediation, quarantine, and upgrade actions.
For more information on vSphere Security features, please check out the HOL-1811-01-SDC - vSphere v6.5 - What's New?
If you want to download the manual for this or any other Hands on Lab, please visit http://docs.hol.vmware.com
Thank you for participating in the VMware Hands-on Labs. Be sure to visit http://hol.vmware.com/ to continue your lab experience online.
Lab SKU: HOL-1842-01-NET