VMware Hands-on Labs - HOL-2033-01-CNA


Lab Overview - HOL-2033-01-CNA - Managing and Extending Kubernetes - Getting Started

Lab Guidance


Note: It will take more than 90 minutes to complete this lab. You should expect to only finish 2-3 of the modules during your time.  The modules are independent of each other so you can start at the beginning of any module and proceed from there. You can use the Table of Contents to access any module of your choosing.

The Table of Contents can be accessed in the upper right-hand corner of the Lab Manual.

Discover  VMware's OSS Kubeadm solution and three other VMware open source projects that extend Kubernetes for use in production environments. Explore Kubernetes cluster lifecycle management, Contour as an ingress controller for Kubernetes, Sonobuoy to understand the state of a Kubernetes cluster, and Velero to backup, and restore your Kubernetes cluster resources and persistent volumes.

Lab Module List:

 Lab Captains: 

This lab manual can be downloaded from the Hands-On Labs Document site found here:

http://docs.hol.vmware.com

This lab may be available in other languages.  To set your language preference and have a localized manual deployed with your lab, you may utilize this document to help guide you through the process:

http://docs.hol.vmware.com/announcements/nee-default-language.pdf


 

Location of the Main Console

 

  1. The area in the RED box contains the Main Console.  The Lab Manual is on the tab to the right of the Main Console.
  2. A particular lab may have additional consoles found on separate tabs in the upper left. You will be directed to open another specific console if needed.
  3. Your lab starts with 90 minutes on the timer.  The lab can not be saved.  All your work must be done during the lab session.  But you can click EXTEND to increase your time.  If you are at a VMware event, you can extend your lab time twice, for up to 30 minutes.  Each click gives you an additional 15 minutes.  Outside of VMware events, you can extend your lab time up to 9 hours and 30 minutes. Each click gives you an additional hour.

 

 

Alternate Methods of Keyboard Data Entry

During this module, you will input text into the Main Console. Besides directly typing it in, there are two very helpful methods of entering data which make it easier to enter complex data.

 

 

Click and Drag Lab Manual Content Into Console Active Window

You can also click and drag text and Command Line Interface (CLI) commands directly from the Lab Manual into the active window in the Main Console.  

 

 

Accessing the Online International Keyboard

 

You can also use the Online International Keyboard found in the Main Console.

  1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar.

 

 

Click once in the active console window

 

In this example, you will use the Online Keyboard to enter the "@" sign used in email addresses. The "@" sign is Shift-2 on US keyboard layouts.

  1. Click once in the active console window.
  2. Click on the Shift key.

 

 

Click on the @ key

 

  1. Click on the "@ key".

Notice the @ sign entered in the active console window.

 

 

Activation Prompt or Watermark

 

When you first start your lab, you may notice a watermark on the desktop indicating that Windows is not activated.  

One of the major benefits of virtualization is that virtual machines can be moved and run on any platform.  The Hands-on Labs utilizes this benefit and we are able to run the labs out of multiple datacenters.  However, these datacenters may not have identical processors, which triggers a Microsoft activation check through the Internet.

Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoft licensing requirements.  The lab that you are using is a self-contained pod and does not have full access to the Internet, which is required for Windows to verify the activation.  Without full access to the Internet, this automated process fails and you see this watermark.

This cosmetic issue has no effect on your lab.  

 

 

Look at the lower right portion of the screen

 

Please check to see that your lab is finished all the startup routines and is ready for you to start. If you see anything other than "Ready", please wait a few minutes.  If after 5 minutes your lab has not changed to "Ready", please ask for assistance.

 

Module 1 - Kubernetes Cluster Lifecycle Management with Kubeadm (30 mins)

Introduction



 

Lab Environment

 

Our lab consists of two Kubernetes clusters, each with one master node and two worker nodes, although you will join k8s-worker-02a to Cluster A in this module.  All nodes are running CentOS 7.6.  Calico is used in the pod to provide networking.  Harbor is used as the local image registry we will use to deploy applications.  

The Kubernetes pod was deployed using upstream Kubernetes version 1.15.1.

 

 

What Is kubeadm?

 

kubeadm is a tool included in every Kubernetes release that helps users set up a best-practice Kubernetes cluster. kubeadm provides support along the entire lifecycle of a Kubernetes cluster including. creation, upgrade, and teardown.

Due to the variety of ways in which machines can be provisioned, the scope of kubeadm is intentionally limited to bootstrapping rather than provisioning—it is intended to be a building block, and higher-level tools take advantage of kubeadm.  kubeadm allows these higher-level tools to ensure clusters are conformant and look as much alike as possible. Setting up add-ons such as CNI, the Kubernetes dashboard, and monitoring tools is outside the scope of kubeadm.

 

 

Why Does VMware Recommend kubeadm?

One of the reasons VMware recommends bootstrapping your Kubernetes cluster with kubeadm is because it provides production-grade defaults for many flags of the control plane components. In addition to these defaults, VMware recommends some minor changes to the kubeadm configuration to properly set up your cluster’s control plane.  As modifications are made, VMware recommends that you track any changes to Kubernetes configuration in source control.

kubeadm is the primary deployment tool VMware recommends for bootstrapping a Kubernetes cluster, based on some key advantages:

So where does kubeadm fit into a complete deployment solution for Kubernetes customers? kubeadm is not a one-click-install solution. As stated above, kubeadm is intended to be a building block and part of a larger solution. VMware is investing significantly in this area to bring a declarative, API-driven model to cluster creation and operations, where clusters are treated as immutable (i.e. upgrades equate to a new deployment versus an in-place upgrade). VMware plans to leverage and contribute to the upstream work on the Cluster API to make this real.  More will be shared in the future.

 

Use kubeadm to Expand Your Cluster


You will use kubeadm to add an additional worker node to your cluster.


 

Accessing the VM-CLI Machine

In our lab, we have a Linux server configured with all the needed tools to get through the lab lessons. We access this box via PuTTY, which already has a saved session for this machine.

 

 

  1. Click the PuTTY icon in the Windows toolbar;
  2. Scroll all the way down and select the "vm-cli.corp.local" machine;
  3. Click Open in order to open the VM-CLI console.

The user configured for auto-login in the PuTTY VM-CLI session is holuser. We are going to use this user for the whole module while connected to the VM-CLI machine.

 

 

Check the Status of the Cluster

 

Use kubectl to check the status of the cluster:

  1. At the command line, type kubectl get nodes
  2. Note the 1 node with the role of master, and 1 node that has a role of <none>, which is a worker node

 

 

Join an Additional Worker Node to the Cluster

 

Open a new SSH session to a master node

  1. In the PuTTy window, click the icon in the upper left-hand corner
  2. Click Saved Sessions
  3. Click k8s-master-01a.corp.local

 

 

Get Join Token

 

In this step, we will generate a bootstrap token that will be used to establish a bidirectional trust between a node joining the cluster and our control plane node.  

  1. At the command line for k8s-master-01a, type kubeadm token create --print-join-command and press <Enter>

 

  1. Note the output.  Highlight the output with the mouse to copy it to the clipboard.  The output should be in this format:  'kubeadm join 192.168.120.11:6443 --token cil09b.qiu6je65v6e15orl     --discovery-token-ca-cert-hash sha256:9c97f9dd6fad76265d82d049fabc5d56987596aff601bc7814bc97b1ab7838ba', but your token will differ.  

We will paste that output into the command line of our new worker node to join it to the cluster.

 

 

SSH to the New Worker Node

 

  1. In the PuTTy window, click the icon in the upper left-hand corner
  2. Click Saved Sessions
  3. Click k8s-worker-02a.corp.local

 

 

Join a New Worker Node to the Cluster

 

The VM "k8s-worker-02a" has already had Kubernetes installed and configured, so we will use the command and token we generated in the previous step to join it to our cluster.

  1. At the command line, right-click to paste the join command and token, and press <Enter>
  2. Wait for the command to complete.  You should see "This node has joined the cluster:" in the output

Let's return to the the vm-cli session to check the status of the cluster

 

 

Check the Status of the Cluster

 

  1. On the taskbar, click your vm-cli PuTTY session
  2. Type kubectl get nodes and press <Enter>
  3. Note that k8s-worker-02a is now a worker node (ROLES of <none>).  It should have a STATUS of Ready

Repeat Step 2 until k8s-worker-02a has a STATUS of Ready before proceeding.  

 

Safely Drain a Node for Maintenance


You will use the "cordon" and "drain" commands to keep new pods from being scheduled on a node, then safely remove any running pods.  


 

Cordon and Drain One of Your Worker Nodes

In the normal management of your Kubernetes clusters, you would want to prepare a node for maintenance to ensure that any running pods are safely removed from your node.  Note that we don't currently have any pods running in our cluster, but the commands will be the same.

 

 

Cordon Your Worker Node

Cordoning a node marks it as unschedulable and prevents new pods from being scheduled to that node, but does not affect any existing pods on the node. This is useful as a preparatory step before a node reboot, etc.

 

  1. In the your VM-CLI PuTTY window, type kubectl get nodes and press <Enter>.  Note the current status of your Master and Worker Nodes
  2. Type kubectl cordon k8s-worker-02a.corp.local and press <Enter>.  Note the response that the node has been cordoned
  3. Type kubectl get nodes and press <Enter>.  Note the current status of our worker node.  Although it is still Ready, scheduling any new pods has been disabled.

 

Use kubeadm to Shrink Your Cluster


In this step, you will remove the worker node you added to the cluster.


 

Remove New Worker Node from the Cluster

 

  1. In your  VM-CLI PuTTY session type kubectl delete node k8s-worker-02a.corp.local and press <Enter>
  2. You should confirmation that the node was deleted.

 

 

Reset Changes on Worker Node

 

We will use kubeadm on the worker node to reset any changes to k8s-worker-02a made during the join command

  1. Click on the session for root@k8s-worker-02... on the taskbar.
  2. Type kubeadm reset -f and press <Enter>
  3. Note the output while the worker node is reset.

You have successfully removed the worker node k8s-worker-02a from Cluster A.

 

Conclusion


This concludes "Module 1 - Kubernetes Lifecycle Management with Kubeadm"

You should now have an understanding of why and how to use kubeadm to help manage your Kubernetes deployment.  


 

You've finished Module 1

 

Congratulations on completing Module 1!

If you are looking for additional information on kubeadm, try one of these:

Proceed to any module below which interests you most regarding managing and extending Kubernetes.

 

 

Test Your Skills!

 

Now that you’ve completed this lab, try testing your skills with VMware Odyssey, our newest Hands-on Labs gamification program. We have taken Hands-on Labs to the next level by adding gamification elements to the labs you know and love. Experience the fully automated VMware Odyssey as you race against the clock to complete tasks and reach the highest ranking on the leaderboard. Try the Kubernetes Odyssey lab

 

 

How to End Lab

 

To end your lab click on the END button.  

 

Module 2 - Contour - Control and Manage Ingress for Kubernetes (30 minutes)

Introduction


This module contains the following lessons:


 

Introduction to Contour

One of the most critical needs for in running workloads at scale with Kubernetes is efficient and smooth traffic ingress management at the Layer 7 level. Getting an application up and running is not always the entire story; it may still need a way for users to access it. Filling that operational gap is what Contour was designed to do by providing a way to allow users to access applications within a Kubernetes cluster. Contour is an Ingress controller for Kubernetes that works by deploying the Envoy proxy as a reverse proxy and load balancer. Contour supports dynamic configuration updates out of the box while maintaining a lightweight profile.

Contour offers the following benefits for users: * A simple installation mechanism to quickly deploy and integrate Envoy * A safe way to support ingress in multiteam Kubernetes clusters * Clean integration with the Kubernetes object model * Dynamic updates to ingress configuration without dropped connections

 

 

What Is Ingress?

Kubernetes Ingress is a set of configurations that define how external traffic can be routed to an application inside a Kubernetes cluster. A controller (Contour) watches for changes to objects in the cluster, then wires together the configurations to create a data path for the request to be resolved, implementing the configurations defined. It makes decisions based on the request received (e.g., example.com/blog), provides TLS termination, and performs other functions.

Ingress is an important component of a cloud native system because it allows for a clean separation between the application and how it’s accessed. A cluster administrator deals with providing access to the controller, and the application engineer deals with deploying the application. Ingress is the glue that ties the two together.

 

 

Contour in Detail

 

Since it was added in Kubernetes 1.1, Ingress hasn't gotten much attention but is still very popular in the community. Many controllers rely on annotations on the Ingress object to clarify, restrict, or augment the structure imposed by it, which is no different from how Contour supports Ingress.

At the same time a number of web application deployment patterns, such as blue/green deployments, explicit load-balancing strategies, and presenting more than one Kubernetes service behind a single route are difficult to achieve with Ingress as it stands today. Contour has introduced a new Custom Resource Definition (CRD) that allows for a new data model called IngressRoute and enhances what Ingress can do by enabling new features that were not previously possible.

IngressRoute is designed to provide a sensible home for configuration parameters as well as to share an ingress controller across multiple namespaces and teams in the same Kubernetes cluster. It does this by using a process we call delegation. This delegation concept patterns off of the way a subdomain is delegated from one domain name server to another, and allows teams to define and self-manage IngressRoute resources safely.

 

Deploy Contour to Your Cluster


In this step, you will deploy Contour from a local registry.


 

Connecting to the VM-CLI Machine

In our lab, we have a Linux server configured with all the tools needed to get through the lab lessons. We access this box via PuTTY, which already has a saved session for this machine.  

 

 

  1. Click the "PuTTY" icon in the Windows toolbar
  2. Scroll all the way down and select the "vm-cli.corp.local" machine
  3. Click Open in order to open the VM-CLI console.

The user configured for auto-login in the PuTTY VM-CLI session is holuser. We are going to use this user for the whole module while connected to VM-CLI machine.

 

 

Deploy Contour from a Local Registry

 

 

Deploy Contour Using yaml File

 

  1. Type kubectl apply -f contour-files/contour-deployment-rbac.yaml and press <Enter>

Note: If you receive the error  "Unexpected error when reading response body: ..." after Step 1, press <Ctrl> + <C> and re-run the command.

  1. Once the deployment completes, type kubectl get deployments -n heptio-contour and press <Enter>
  2. Type kubectl get pods -n heptio-contour and click <Enter>
  3. Note the status of the replicas
  4. Type kubectl get events -n heptio-contour --sort-by='{.lastTimestamp}'  and press <Enter>
  5. Review the events

Wait until both replicas are showing READY of 2/2 and STATUS Running (Step 4) before moving on to the next step.  Re-run the command from Step 3 to refresh the status if necessary.

 

Deploy A Test Application


In this step, you will deploy a test application to show Contour working for ingress control.


 

Deploy kuard Test Application

 

kuard is a demo application for Kubernetes, and stands for "Kubernetes up and Running"

  1. At the command line for VM-CLI, type cat contour-files/contour-kuard-example.yaml and press <Enter>
  2. Review the yaml configuration file for the deployment
  3. Type kubectl apply -f contour-files/contour-kuard-example.yaml and press <Enter>

Note: If you receive the error  "Unexpected error when reading response body: ..." after Step 1, press <Ctrl> + <C> and re-run the command.

 

 

Check Status of kuard

 

  1. Type kubectl get pods and press <Enter>
  2. Wait for all replicas to have the STATUS of "Running" before proceeding
  3. Type kubectl get -n heptio-contour service contour -o wide and press <Enter>
  4. Note the ingress ports (numbers after 80: and 443:). We will use the port number that translates to port 80 to connect to the kuard application.  In this example it is 31810.  Your number will be different.
  5. Note the TYPE is NodePort, and is not load balanced, so we will connect directly to one of the worker nodes where the application is running

 

 

Test kuard

 

  1. Click the Chrome icon to open a browser
  2. In the address bar, type http://k8s-worker-01a.corp.local:<your_port_number>, where <your_port_number> will be from Step 4 previously,  and press <Enter>
  3. Feel free to explore the "Kubernetes Up and Running" application by clicking on the tabs on the left-hand side

 

Remove Test Application and Contour


In this step, you will remove kuard and Contour from the cluster.


 

Remove kuard

 

  1. Return to the vm-cli PuTTY session by clicking the "holuser@vm-cli:" icon on the taskbar.
  2. At the command line, type kubectl delete -f contour-files/contour-kuard-example.yaml and press <Enter>
  3. Type kubectl get pods and press <Enter>
  4. Note that you may see a pod terminating.  Re-run the command from Step 3 until you see "No resources found."  

 

 

Remove Contour

 

  1. Type kubectl delete -f contour-files/contour-deployment-rbac.yaml and press <Enter>
  2. Type kubectl get pods -n contour-heptio and press <Enter>  
  3. You should see "No resources found."  

 

Conclusion


This concludes "Module 2 - Contour - Control and Manage Ingress for Kubernetes"

You should now have an understanding how to deploy Contour and use it in an application you deploy.


 

You've finished Module 2.

 

Congratulations on completing  Module 2!

If you are looking for additional information on Contour and ingress control, try one of these:

Or use your smart device to scan the QR Code.

Proceed to any module below that interests you most regarding Managing and Extending Kubernetes.

 

 

 

How to End Lab

 

To end your lab click on the END button.  

 

Module 3 - Velero - Back Up and Migrate Kubernetes Resources and Persistent Volumes (30 minutes)

Introduction


 

This module contains the following lessons:


 

Why Velero?

Velero was designed specifically to back up and restore Kubernetes cluster resources. This gives Velero some advantages over etcd backups.

Velero uses the Kubernetes API discovery capabilities to collect backup data. This means that Velero can back up new APIs without updating Velero itself. Velero does not need to take backups of etcd.

A discovery approach allows Velero to back up clusters that include aggregated API servers, which otherwise requires creating an etcd backup of each server. Velero can also perform backups in scenarios where there is no direct access to etcd, such as a cluster running on GKE.

Velero lets you select specific resources to back up, because it does not create an atomic snapshot by backing up etcd. This approach means that Velero lets you restore a subset of a backup.

Velero also associates snapshots of persistent volumes with each backup. These snapshots allow Velero to restore both what was running in the cluster and the data associated with the cluster.

 

 

Backup

Velero supports on-demand backups as well as scheduled periodic backups. You can configure Velero to back up your entire cluster or only specified resources based on namespaces, resources, and labels you include or exclude.

Every cluster has different backup needs. You should carefully consider your own needs when creating a backup plan. Consider the following scenarios as you plan for your environment.

 

 

Full Cluster

A full cluster backup is the default behavior if you run velero create backup without any additional flags. This approach is the simplest way to get started with Velero, but might not be adequate for all purposes.

 

 

Per-Namespace

A per-namespace backup lets you restore a namespace with just the base velero create restore command.

This approach is useful if you have multi tenant clusters where each tenant has its own namespace in the cluster. You can restore a single namespace without disrupting the other tenants.

Depending on your needs, you might also want full cluster backups together with your namespace backups.

 

 

Strategy

We recommend that you identify critical functions in your system and create backups that include only the resources needed to restore those critical functions. This makes restoring simpler because you restore the entire backup instead of having to filter it. This is especially important to help reduce errors in case of disaster recovery.

 

 

Scheduling Backups

Velero supports scheduled backups with a cron syntax. You can also set a time to live (TTL). These two features allow you to create backup configurations for different recovery point objectives (RPO).

A common configuration is the hourly backup. Here is an example of an hourly backup configuration that includes hourly, daily, weekly, and monthly backup schedules all with TTLs that remove backups once they are part of the larger chronological archive.

 

 

Backup Hooks

Hooks in Velero let you run a command inside a container before and after a backup. Velero provides both pre-backup and post-backup hooks. Hooks are configured using pod annotations. The Velero documentation provides an example of hooks to call fsfreeze on a file system before and after Velero performs a backup, plus the full details of hook annotations.

 

 

Restore

Velero supports restoring backups that are created manually or according to a schedule. It lets you perform full or partial restores of backups. Velero performs a full restore of a backup by default.

 

 

Lifecycle of Velero Objects

Understanding the lifecycle of a Velero object can help you understand the details of your Velero jobs.

  1. The New phase shows that the requested backup, restore, or schedule object is created by the API, but the object has not been processed by its respective controller.
  2. The next phase is validation. The Newobject is validated by the processing controller. If the controller cannot validate the object, it moves to the FailedValidation phase and no further processing is attempted.
  3. After successful validation, if you are creating a schedule, the object moves to the Enabled phase. When a schedule is Enabled it triggers backups in accordance with its schedule spec.
  4. If you are creating a backup or restore, the object moves to the InProgress phase. During the InProgress phase, Velero attempts to perform all the operations codified in the backup or restore object. Relevant errors and warnings are counted during this process and captured in the status. The type of object being created determines the next phase.
  5. Both backup and restore objects have a Completed phase. This phase shows that the requested backup or restore object and all its operations have been performed. The Completed phase does not automatically mean there were no errors or warnings.

Two other phases are possible for backups:

 

 

Monitoring

Velero provides metrics for backups and restores. You can use these metrics to add recording and alerting rules to your Prometheus configuration.

 

 

Migrations

Moving resources from one cluster to another is a common scenario. Velero can help with migration. The Velero documentation provides further details on how to use Velero to perform a cluster migration.

 

 

Disaster Recovery

VMware recommends that you create a periodic backup schedule as part of your disaster recovery plan. Velero can then help with the recovery of your cluster and its resources in case of a disaster.

During a recovery, Velero can be configured to restoreOnly mode. restoreOnly mode ensures that Velero does not take any backups during the restore process so that you do not have worry about cleaning up backups that might contain only a partially restored cluster.

The Velero documentation provides more details on how to use Velero to recover cluster resources.

 

 

Testing Your Backups

Testing your backups is just as important as creating a periodic backup schedule. Testing gives you the opportunity to verify that the backups restore all of the expected state. Testing and verifying your backups is a critical part of any disaster recovery plan.

 

 

Security

By default, Velero runs using a service account with cluster-admin permissions, and is not scoped by any role-based access controls (RBAC). These permissions allow Velero to back up and restore all Kubernetes resources. However, this means users who have permissions to back up and restore with Velero effectively have cluster-admin permissions.

This means that only trusted administrators should have access to create backups and restores.

You can configure Velero to run with reduced permissions, but this means that only the resources the related service account can access can be backed up and restored.

Velero does not currently support multi tenancy in a single instance. A scenario with multiple, tightly scoped instances is untested.

 

Velero Overview


Velero gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. Velero lets you:

Velero consists of:

You can run Velero in clusters on a cloud provider or on-premise. For detailed information, see Compatible Storage Providers.

Because our lab environment has no Internet access, we are going to install and use Velero in an on-premise fashion.


 

How Velero Works

Each Velero operation - on-demand backup, scheduled backup, restore - is a custom resource, defined with a Kubernetes Custom Resource Definition (CRD) and stored in etcd. Velero also includes controllers that process the custom resources to perform backups, restores, and all related operations.

You can back up or restore all objects in your cluster, or you can filter objects by type, namespace and/or label.

Velero is ideal for the disaster recovery use case, as well as for snapshotting your application state, prior to performing system operations on your cluster (e.g., upgrades).

 

 

On-Demand Backups

The backup operation:

  1. Uploads a tarball of copied Kubernetes objects into cloud object storage.
  2. Calls the cloud provider API to make disk snapshots of persistent volumes, if specified.

You can optionally specify hooks to be executed during the backup. For example, you might need to tell a database to flush its in-memory buffers to disk before taking a snapshot. More about hooks.

Note that cluster backups are not strictly atomic. If Kubernetes objects are being created or edited at the time of backup, they might not be included in the backup. The odds of capturing inconsistent information are low, but it is possible.

 

 

Scheduled Backups

The schedule operation allows you to back up your data at recurring intervals. The first backup is performed when the schedule is first created, and subsequent backups happen at the schedule's specified interval. These intervals are specified by a cron expression.

Scheduled backups are saved with the name <SCHEDULE NAME>-<TIMESTAMP>, where <TIMESTAMP> is formatted as YYYYMMDDhhmmss.

 

 

Restores

The restore operation allows you to restore all of the objects and persistent volumes from a previously created backup. You can also restore only a filtered subset of objects and persistent volumes. Velero supports multiple namespace remapping-  for example, in a single restore, objects in namespace "abc" can be recreated under namespace "def", and the objects in namespace "123" under "456".

The default name of a restore is <BACKUP NAME>-<TIMESTAMP>, where <TIMESTAMP> is formatted as YYYYMMDDhhmmss. You can also specify a custom name. A restored object also includes a label with key velero.io/restore-name and value <RESTORE NAME>.

You can also run the Velero server in restore-only mode, which disables backup, schedule, and garbage collection functionality during disaster recovery.

 

 

Backup Workflow

 

When you run velero backup create test-backup:

  1. The Velero client makes a call to the Kubernetes API server to create a Backup object.
  2. The BackupController notices the new Backup object and performs validation.
  3. The BackupController begins the backup process. It collects the data to back up by querying the API server for resources.
  4. The BackupController makes a call to the object storage service - for example, AWS S3 - to upload the backup file.

By default, velero backup create makes disk snapshots of any persistent volumes. You can adjust the snapshots by specifying additional flags. Run velero backup create --help to see available flags. Snapshots can be disabled with the option --snapshot-volumes=false.

 

 

Backed-Up API Versions

Velero backs up resources using the Kubernetes API server's preferred version for each group/resource. When restoring a resource, this same API group/version must exist in the target cluster in order for the restore to be successful.

For example, if the cluster being backed up has a gizmos resource in the things API group, with group/versions things/v1alpha1, things/v1beta1, and things/v1, and the server's preferred group/version is things/v1, then all gizmos will be backed up from the things/v1 API endpoint. When backups from this cluster are restored, the target cluster must have the things/v1 endpoint in order for gizmos to be restored. Note that things/v1 does not need to be the preferred version in the target cluster; it just needs to exist.

 

 

Set a Backup to Expire

When you create a backup, you can specify a TTL by adding the flag --ttl <DURATION>. If Velero sees that an existing backup resource is expired, it removes:

 

 

Object Storage Sync

Velero treats object storage as the source of truth. It continuously checks to see that the correct backup resources are always present. If there is a properly formatted backup file in the storage bucket, but no corresponding backup resource in the Kubernetes API, Velero synchronizes the information from object storage to Kubernetes.

This allows restore functionality to work in a cluster migration scenario, where the original backup objects do not exist in the new cluster.

Likewise, if a backup object exists in Kubernetes but not in object storage, it will be deleted from Kubernetes since the backup tarball no longer exists.

 

Velero Install and Configure


Prerequisites


 

About MinIO

MinIO is a cloud native application designed to scale in a sustainable manner in multi tenant environments. It is an object storage server released under Apache License v2.0. Orchestration platforms provide a perfect launchpad for MinIO to scale. MinIO is supported on multiple orchestration platforms:

Why Is MinIO Cloud Native?

The term cloud native revolves around the idea of applications deployed as microservices that scale well. It is not about just retrofitting monolithic applications onto a modern, container based compute environment. A cloud native application is portable and resilient by design, and can scale horizontally by simply replicating. Modern orchestration platforms like Swarm, Kubernetes, and DC/OS make replicating and managing containers in huge clusters easier than ever.

While containers provide an isolated application execution environment, orchestration platforms allow seamless scaling by helping replicate and manage containers. MinIO extends this by adding an isolated storage environment for each tenant.

MinIO is built from the ground up on the cloud native premise. With features like erasure-coding, distributed and shared setup, it focuses only on storage and does it very well. Plus it can be scaled by just replicating MinIO instances per tenant via an orchestration platform.

If you wish to know more about MinIO, take a look at the documentation.

We are going to use MinIO in our lab. Since we do not have access to the Internet, we can't access a Kubernetes cluster running on Amazon AWS.  We are going to use MinIO in an on-premises fashion, which means we are going to use MinIO with our local Kubernetes cluster, leveraging its features to provide our cluster with S3-compatible storage for our Velero operations.

We are going to run MinIO in a docker container using vm-cli as our docker host.

In our lab we are going to explore basic Velero functionality. Configuring MinIO for production is out of scope.

 

 

Connecting to the VM-CLI Machine

In our lab, we have a Linux server configured with all the needed tools to get through the lab lessons. We access this box via PuTTY, which already has a saved session for this machine.

 

 

  1. Click the PuTTY icon in the Windows toolbar;
  2. Scroll all the way down and select the vm-cli.corp.local machine;
  3. Click <Open> in order to open the vm-cli console.

The user configured for auto-login in the puTTY VM-CLI session is holuser. We are going to use this user for the whole module while connected to the VM-CLI machine.

 

 

Installing Velero

In our lab we are going to set up the Velero server and client, then back up and restore a sample application.

For simplicity, we are going to use MinIO, an S3-compatible storage service that runs locally on your cluster. For additional functionality with this setup, see the docs on how to expose Minio outside your cluster.

Velero is an open source project backed by VMware that is publicly available from our git repository at https://github.com/heptio/velero/releases.

Since the velero command-line client is just a binary file, the only thing we need to do is download the archive for the right platform and extract it to a place that make sense. In our lab, the archive was already downloaded, so we only need to extract it to a reasonable place like the local binary directory.

 

 

 

Extracting Velero Binary

 

  1. Type pwd and hit <ENTER> to make sure you are in the right directory. It should display "/home/holuser";
  2. Type ls -l and hit <ENTER>. You should see a directory called "/velero-files";
  3. Type cd velero-files and hit <ENTER>;
  4. Type ls -l and hit <ENTER>. Find the Velero archive and its version - "velero-v1.0.0-linux-amd64.tar.gz";
  5. Type sudo tar -zxvf velero-v1.0.0-linux-amd64.tar.gz -C /usr/local/bin/ --strip-components=1 velero-v1.0.0-linux-amd64/velero and hit <ENTER> to extract the Velero binary to the local binary directory (the command must be typed in a single line);
  6. Type which velero and hit <ENTER> to check if the binary was extracted to the right location. It should display "/usr/local/bin/velero".

 

 

Checking Our Kubernetes Cluster

Before we start running Velero, let's take a look at our Kubernetes cluster, its nodes and its configuration.

 

  1. Type kubectl get nodes and hit <ENTER> to check the nodes and their roles in the Kubernetes cluster. You should see two nodes, one master/etcd, and one worker.
  2. Type kubectl get namespaces and hit <ENTER> to check the existing namespaces in our cluster. Notice that we have only the default ones: "default", "kube-node-lease", "kube-public", and "kube-system".

 

 

Setting Up Velero Server

Now that we have the Velero command-line client configured, let's set up the Velero server.

 

 

Running the MinIO Storage Service Container

In order to have a S3-like storage service ,we are going to run MinIO in a container using our VM-CLI machine as a docker host. We are going to expose the MinIO service on port 9000 and use a local directory to map to the MinIO data directory. We are also going to create the MinIO server credentials in order to pass to the Velero server later on. The image for the MinIO container is stored in our local Harbor registry.

 

  1. Type sudo mkdir /mnt/minio-data and press <ENTER> in order to create the local directory to be mapped to the MinIO data directory.
  2. Type sudo docker run --name minio -d -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=minio123" -v /mnt/minio-data/:/data harbor-01a.corp.local:80/minio/minio server /data and hit <ENTER> to run the MinIO server in a container.
  3. Type sudo docker ps to make sure the MinIO container is running.

 

 

Accessing the MinIO UI and Creating the S3 Bucket

 

  1. Click the <Chrome> icon in the Windows toolbar to open the Chrome browser.
  2. Type vm-cli.corp.local:9000 in the Chrome address bar and hit <ENTER>. You should get the MinIO UI login page.
  3. Type the MinIO credentials (defined in the arguments passed to the container) where the Access Key is "minio" and the Secret Key is "minio123" and hit <ENTER>.
  4. Click the PLUS sign in the bottom left and then in the yellow icon to create a bucket (not shown). Type velero as the name of the bucket and hit <ENTER>.
  5. Now you can see we have one S3-compatible bucket called "velero". If you look at the right side you will see that this bucket is empty. When we create our backups they will be stored on this bucket.

 

 

Starting the Velero Server

Since we don't have Internet access we need to specify the container images needed by Velero in the velero install command. Our images are stored on a local Harbor registry running on the harbor-01a.corp.local server.

 

  1. Type pwd and hit <ENTER> to make sure we are in the "/velero-files" directory.
  2. Type cat credentials-velero to check the MinIO credentials to be passed to Velero install.
  3. Type velero install --provider aws --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --image harbor-01a.corp.local:80/heptio-images/velero:latest in a single line and hit <ENTER> to start the Velero server. Notice the message "Velero is Installed!".
  4. Type kubectl get ns to check the new namespace created called velero.
  5. Type kubectl get deployments -n velero and hit <ENTER>. Notice that we also have a deployment called "velero".
  6. Type kubectl get pod -n velero and hit <ENTER>. Check the new Velero pod called "velero-<id>".

 

Velero Backup and Restore


Now that we have our Velero server up and running with our MinIO storage service we are ready to back up some Kubernetes resources.

In our lab we are going to deploy a sample NGINX application to test the backup and restore features.

If you are not logged into the VM-CLI console, please do it now by following the "Accessing the VM-CLI console" section.


 

Deploying a Sample Application

 

  1. Type pwd and hit <ENTER> to make sure you are on the "/velero-files" directory.
  2. Type ls and hit <ENTER>. Look for a directory called "/nginx-app".
  3. Type cd nginx-app and hit <ENTER> to change to this directory.
  4. Type ls and hit <ENTER>. Look for a file called "base.yaml". This file contains all the specifications to our nginx deployment.
  5. Type vi base.yaml and hit <ENTER> to edit the deployment.
  6. Change the deployment image. Look for the line "- image: harbor-01a.corp.local:80/library/nginx:1.7.9" and change the tag from 1.7.9 to 1.15-alpine.
  7. Type <ESC> and hit <ENTER> . Type :wq and hit <ENTER> to save and quit vi editor.
  8. Type kubectl apply -f base.yaml and hit <ENTER> to create the nginx sample application deployment.
  9. Type kubectl get ns and hit <ENTER> and notice that we now have a namespace called "nginx-example".
  10. Type kubectl get deployment -n nginx-example and hit <ENTER> to list the nginx deployment in the "nginx-example" namespace. You should see a deployment called "nginx-deployment" with 2/2 annotation in the "READY" column. That means we have two nginx pods in this deployment.
  11. Type kubectl get pod -n nginx-example and hit <ENTER> to list the pods in the nginx deployment.
  12. Type kubectl get svc -n nginx-example and hit <ENTER> to find out what type of service is configured for our nginx deployment and also on which port it is running. You should see a service called "my-nginx" of type "NodePort" with the translation from the default web port 80 to port 31355 (this port may be different for your deployment).
  13. Since we only have one worker node, those nginx pods are certainly running on it. Type curl http://k8s-worker-01a.corp.local:31355 and hit <ENTER>. You should see a nginx welcome page in HTML code, of course! You can test it in a browser too if desired. The sample application is working. Remember that the port may be different for your environment.

 

 

Backing Up the Sample Application

 

  1. Type cat base.yaml | grep -A 1 selector and hit <ENTER> to list the "selectors" our sample app has. Notice that it has at least one selector called "app:nginx". We are going to use this selector to create Velero backups.
  2. Type velero backup create nginx-backup --selector app=nginx and hit <ENTER> to create a backup from our sample app called "nginx-backup".
  3. Type velero backup get and hit <ENTER> to list all the backups we have. Notice that right now we only have the backup we've just created, "nginx-backup" with the status "Completed".
  4. Type velero schedule create nginx-daily --schedule="0 1 * * *" --selector app=nginx and hit <ENTER> to create a daily schedule backup for our sample app called "nginx-daily". Notice that the flag --schedule uses a pattern based in the cron utility.
  5. Type velero schedule get and hit <ENTER> to list all the existant schedules. Right now we only have the schedule we've just created, "nginx-daily" with the status "Enabled".
  6. Type velero backup get and hit <ENTER> again and notice that now we have another backup called "nginx-daily-<TIMESTAMP>" with the status "Completed". This backup was created by the backup schedule created in the previous steps.
  7. Now let's delete our sample app to simulate a disaster. Type kubectl delete namespace nginx-example and hit <ENTER>.
  8. Type kubectl get ns and hit <ENTER> and notice that we no longer have the namespace "nginx-example".
  9. Type kubectl get svc -n nginx-example and hit <ENTER> to check if we still have the service; you should see "No resources found.". That's because every time we delete a Kubernetes namespace all the resources associated with it are also deleted.
  10. Type curl http://k8s-worker-01a.corp.local:31355 and hit <ENTER>. You should get a "Failed to connect..." message because our sample app was completely removed. Remember that the port in your deployment may be different.

 

 

Checking the Backup from MinIO UI

 

 

  1. Click the <Chrome> icon in the Windows toolbar to open the Chrome browser.
  2. Type vm-cli.corp.local:9000 and hit <ENTER> to open the MinIO UI. You should be already logged in; if not, go back to the step where you did log in and repeat it. Remember that the port used may be different from this one.
  3. Click <velero> and notice that now we have a folder called "/backups". Feel free to explore it.

 

 

Restoring the Sample Application

 

  1. Type velero backup get and hit <ENTER>. Confirm we have a backup of our sample app called "nginx-backup".
  2. Type velero restore create --from-backup nginx-backup and hit <ENTER> to restore our sample app from the "nginx-backup".
  3. Type velero restore get and hit <ENTER>. Confirm that we have a restore job called "nginx-backup-<TIMESTAMP>" with the status "Completed" and without any warning or errors.
  4. Type kubectl get ns and hit <ENTER>. Confirm that the namespace "nginx-example" is listed again.
  5. Type kubectl get svc -n nginx-example and hit <ENTER>. Notice that the service "my-nginx" is back in place but this time in a different port: 32300. Remember that the port may be different for your deployment.
  6. Type curl http://k8s-worker-01a.corp.local:32300 and hit <ENTER> and you should get the welcome page for our sample app again! Our restore worked like a charm!

 

 

Checking the Restore from MinIO UI

 

 

  1. Click the <Chrome> icon in the Windows toolbar to open the Chrome Browser.
  2. Type vm-cli.corp.local:9000 and hit <ENTER> to open the MinIO UI. You should be already logged in; if not, get back to the step where you did log in and repeat it. Remember that the port used may be different from this one. Refresh the browser if the session was already opened.
  3. Click <velero> and notice that besides the "/backups" folder we now also have a "/restores" folder. Feel free to explore it.

 

Velero Disaster Recovery


Using Schedules and Restore-Only Mode

If you periodically back up your cluster's resources, you are able to return to a previous state in case of some unexpected mishap, such as a service outage.

The process is very similar to a regular backup, the difference being we take it for all the resources in the cluster, which gives us a backup for the entire cluster.

In this lesson we are going to create a scheduled backup, simulate a disaster, and restore the cluster resources.

If you are not connected to the VM-CLI Linux box, follow the instructions from the "Velero Install and Configure" lesson to do so.


 

Creating a Kubernetes Cluster Scheduled Backup

 

  1. Type pwd and hit <ENTER> to make sure you are in the "velero-files" directory; if not, change to it before proceeding.
  2. Type kubectl get ns and hit <ENTER>. Notice that we still have the namespaces "velero" and "nginx-example".
  3. Type velero create schedule k8s-cluster-a --schedule "0 7 * * *" and hit <ENTER> to create a weekly scheduled backup for the entire cluster. Notice that we did not use the component selector as we did before and the schedule name is in a "domain form". That means we are creating a scheduled backup for the entire cluster, not only for specific resources.
  4. Type velero get schedule and hit <ENTER>. We now also have a scheduled backup for the entire cluster with the name "k8s-cluster-a".
  5. Type velero get backup and hit <ENTER> to see that our first backup from the created schedule routine was already taken under the name "k8s-cluster-a-<TIMESTAMP>" and has the status "Completed".

 

 

Simulating a Disaster

To simulate a disaster we are going to delete Velero and the nginx-example deployment. In fact, we are going to delete the entire namespace for both of them. Since we have our MinIO S3 bucket with all the needed backups running as a container, we will only need to connect the new Velero install to it.

 

  1. Type kubectl get ns and notice we have both namespaces nginx-example and velero.
  2. Type kubectl delete ns nginx-example and hit <ENTER> to delete the namespace "nginx-example".
  3. Type kubectl get deployments -n velero and hit <ENTER> to see the the existing deployments in the "velero" namespace.
  4. Type kubectl delete namespace/velero clusterrolebinding/velero and hit <ENTER>. This will delete the entire Velero namespace as well as the Cluster Role Bindings created during the Velero install.
  5. Type kubectl delete crds -l component=velero and hit <ENTER>. This will delete all the Custom Resources created during Velero install.
  6. Type kubectl get ns and hit <ENTER>. Notice that we no longer have the velero and nginx-example namespaces.

The disaster was simulated. Now, let's recover from it.

 

 

Running Velero in Restore-Only Mode

In order to recover our Kubernetes cluster from a disaster, we need to re-install the Velero server pointing to the same storage server where our backups were stored. The difference now is that we are going to use the flag --restore-only true. This is because it will prevent backup objects from being created or deleted during your restore process.

 

  1. Type pwd and hit <ENTER> to make sure you are in the "velero-files" directory; if not, change to it before proceeding.
  2. Type velero install --provider aws --bucket velero --secret-file ./credentials-velero restore-only --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --image harbor-01a.corp.local:80/heptio-images/velero:latest in a single line and hit <ENTER> to start the Velero server in "Restore-Only" mode.
  3. Type kubectl get deployment -n velero and hit <ENTER>. You should now see the Velero server up and running again.

 

 

Recovering the Kubernetes Cluster

 

  1. Type velero get backup and hit <ENTER>. You should see the last cluster backup we took in the previous lesson with the name "k8s.corp.local-<TIMESTAMP>".
  2. Type velero restore create --from-backup k8s-cluster-a-<TIMESTAMP> and hit <ENTER>. The restore process begins.
  3. Type velero get restore and hit <ENTER>. You'll see our restore job with the status "InProgress". Wait a couple of minutes and try the command again. You should now see that the restore has a "Completed" status.
  4. Type kubectl get ns and hit <ENTER>. You should see the namespace "nginx-example" again and our cluster fully restored.
  5. Type kubectl get pod -n nginx-example and hit <ENTER> to see the "nginx" pods come back to life.
  6. Type kubectl get svc -n nginx-example and hit <ENTER> to see that the "my-nginx" service is also back.

Our lab environment is fairly simple, and we just saw one resource to be recovered because it was all we had at the time of backup. Keep in mind that this recovery process could work for an infinite number of resources in a Kubernetes cluster.

 

Velero Cluster Migration


Using Backups and Restores

Velero can help you port your resources from one cluster to another as long as you point each Velero instance to the same cloud object storage location. In this scenario, we are also assuming that your clusters are hosted by the same cloud provider.

The migration process is very similar to the restore process we saw in the last chapter, the only difference is that the restore takes place in another Kubernetes cluster.

Note: Velero does not support the migration of persistent volumes across cloud providers.

In our lab, we have two Kubernetes clusters so we can test the cluster migration functionality. The process is very similar to disaster recovery, but the difference is the recover process occurs in another Kubernetes cluster and not in the same one.


 

Overall Migration Process

Follow the overall migration process:

  1. (Cluster A) Assuming you haven't already been checkpointing your data with the Velero schedule operation, you need to first back up your entire cluster (replacing <BACKUP-NAME> as desired) with the command velero backup create <BACKUP-NAME>. The default TTL is 30 days (720 hours); you can use the --ttl flag to change this as necessary.
  2. (Cluster B) Add the --restore-only flag to the server spec in the Velero deployment YAML.
  3. (Cluster B) Make sure that the BackupStorageLocation and VolumeSnapshotLocation CRDs match the ones from Cluster 1, so that your new Velero server instance points to the same bucket.
  4. (Cluster B) Make sure that the Velero backup object is created with the command velero backup describe <BACKUP-NAME> . Velero resources are synchronized with the backup files in cloud storage. Note: The default sync interval is 1 minute, so make sure to wait before checking. You can configure this interval with the --backup-sync-period flag to the Velero server.
  5. (Cluster B) Once you have confirmed that the correct backup (<BACKUP-NAME>) is now present, you can restore everything with velero restore create --from-backup <BACKUP-NAME>
  6. (Cluster B) Run the command velero restore get and velero restore describe <RESTORE-NAME-FROM-GET-COMMAND> to check that the second cluster is behaving as expected.
  7. If you encounter any issues make sure that Velero is running in the same namespace in both clusters.

 

 

Checking Kubernetes Clusters

In our lab environment we have two Kubernetes clusters: Cluster A and Cluster B. The way we access those different clusters within the same kubectl environment is by making use of the kubeconfig files. We already have both kubeconfig files saved in our VM-CLI machine and also two variables configured for each one of them.

Notice that there are other ways to access multiple Kubernetes clusters. To learn more, take a look at the article "Configure Access to Multiple Clusters".

 

  1. Type kubectl get nodes --kubeconfig=$KUBECONFIG_A and press <ENTER>. This is the cluster that we are now calling Cluster A which we've been working with during the lab. As you already know, this cluster has two nodes, one master/etcd, and one worker.
  2. Type kubectl get nodes --kubeconfig=$KUBECONFIG_B and press <ENTER>. This is the new cluster, Cluster B, which we are going to use to migrate the resources to in this chapter. As you can see, this cluster has three nodes, one master/etcd and two workers.
  3. Type kubectl get ns --kubeconfig=$KUBECONFIG_A and press <ENTER>. As expected, we have both namespaces we created from previous modules, nginx-example and Velero.
  4. Type kubectl get ns --kubeconfig=$KUBECONFIG_B and press <ENTER>. Notice that we only have the default Kubernetes namespaces since we did not create any kind of resource in this cluster yet.

 

 

Migrating Resources to Cluster B - Simulating a Disaster

In order to simulate a disaster, we are going to completely shut down Cluster A. Since our backup files are stored in our MinIO server running in a docker container in the VM-CLI machine, we can safely shut down Cluster A without losing access to the backup store.

 

  1. Click the PuTTY icon on the upper left corner on the current windows session.
  2. Go to Saved Sessions and click k8s-worker-01a.corp.local.
  3. Once the session to the worker node of Cluster A is opened, type shutdown -h now and press <ENTER>. !!!MAKE SURE YOU ARE SHUTTING DOWN THE RIGHT HOST BY CHECKING THE NAME IN THE COMMAND PROMPT!!!
  4. Click OK and close the inactive window session.
  5. (NOT SHOWN) Repeat the same above steps to shut down the k8s-master-01a.corp.local. !!!MAKE SURE YOU ARE SHUTTING DOWN THE RIGHT HOST BY CHECKING THE NAME IN THE COMMAND PROMPT!!!
  6. Go back to the VM-CLI machine console to make sure Cluster A is out of service. Type kubectl get nodes --kubeconfig=$KUBECONFIG_A. You should receive an error "Unable to connect to the server..."; if not, wait a couple more minutes until the Cluster A nodes are powered off.

 

 

Installing Velero on Cluster B

 

Before we install Velero on Cluster B, let's move the Cluster B kubeconfig file to the default location so we don't have to pass it as a parameter every time we run a command.

  1. Type cp $KUBECONFIG_B /home/holuser/.kube/config and press <ENTER> to copy the Cluster B config file to the default config file location.
  2. Type kubectl get nodes and press <ENTER> to confirm that we are now accessing the Kubernetes Cluster B.
  3. Type velero install --provider aws --bucket velero --secret-file ./credentials-velero restore-only --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://vm-cli.corp.local:9000 --image harbor-01a.corp.local:80/heptio-images/velero:latest in a single line and press <ENTER>. Notice that this command is the exact same command we ran in the previous lesson to install Velero after the simulated disaster. The only difference is that we are now installing Velero in a different cluster.
  4. Type kubectl get ns and press <ENTER>. Notice that we now have the Velero namespace.
  5. Type kubectl get deployment -n velero and press <ENTER> to check the Velero deployment.
  6. Type kubectl get pods -n velero and press <ENTER> to check that the Velero pod is running.

 

 

Checking Existing Backups

In the previous chapter, we created a backup and a backup schedule for the entire cluster configuration. It is good practice to keep a backup schedule for your Kubernetes clusters in order to recover in case of a failure or migration to another cluster, depending on the use case. We are going to use that previously created backup to migrate resources from Cluster A to Cluster B.

 

  1. Type velero backup get and press <ENTER> to get the backup name. The name should be k8s-cluster-a-<TIMESTAMP>.
  2. Type velero backup describe k8s-cluster-a-<TIMESTAMP> and press <ENTER> to get a detailed description of the backup. Notice that every namespace and every resource in the cluster was backed up. We ensured that by not specifying a Label selector in the backup command.

 

 

Migrating (Restoring) Resources

 

  1. Type velero restore create --from-backup k8s-cluster-a-<TIMESTAMP> and press <ENTER> to restore the backup we just got from previous steps.
  2. Type velero restore get and press <ENTER>. You'll see our restore job with a status "InProgress". Wait a couple of minutes and try the command again. You should now see that the restore has a "Completed" status and about 10 warnings.
  3. Type velero restore describe k8s-cluster-a-<TIMESTAMP>-<TIMESTAMP> and press <ENTER>. Notice that we now have two TIMESTAMPS in the restore job's name; the first is related to the time the backup was taken and the second is related to the time the restore took place.
  4. Here you can see what the 10 warnings were about. Since we backed up everything in Cluster A, as expected, some of the default resources/namespaces were not restored because they were already there. Those are some of the default resources common to all Kubernetes deployments. In the next steps, we are going to see what was indeed restored.

 

 

Checking the New Cluster B Resources

 

  1. Type kubectl get ns and press <ENTER> and notice that we now have a namespace called "nginx-example".  This is the namespace created with the nginx appliction from Cluster A. This application was there before the cluster shutdown.
  2. Type kubectl get deployment -n nginx-example and press <ENTER> to list the nginx deployment in the "nginx-example" namespace. You should see a deployment called "nginx-deployment" with 2/2 annotation in the "READY" column. That means we have two nginx pods in this deployment, as expected from the former cluster.
  3. Type kubectl get pod -n nginx-example -o wide and press <ENTER> to list the pods in the nginx deployment as well as on which worker they are running. Since we have two worker nodes, Kubernetes intelligently runs each pod on each host for the sake of load balancing.
  4. Type kubectl get svc -n nginx-example and press <ENTER> to find out what type of service is configured for our nginx deployment as well as on which port it is running. You should see a service called "my-nginx" of type "NodePort" with the translation from the default web port 80 to port 32415 (this port may be different for your deployment).
  5. You could use either worker 1 or 2 since we have a pod running on each. The results should be the same. Type curl http://k8s-worker-01b.corp.local:32415 and press <ENTER>.  You should see a nginx welcome page in HTML code, of course! You can test it in a browser, too, if desired. Remember that the port may be different for your environment.

 

Conclusion


This concludes "Module 3 - Velero - Back Up and Migrate Kubernetes Resources and Persistent Volumes". 

In this module we saw how to use Velero to back up, restore, and recover from a disaster as well as how to migrate resources in a Kubernetes deployment.


 

You've finished Module 3!

Congratulations on completing  Module 3!

If you are looking for additional information on Velero, try one of these:

Proceed to any module below that interests you most:

 

 

 

How to End Lab

 

To end your lab click the END button.  

 

Module 4 - Sonobuoy: Validate Your Kubernetes Deployment - iSIM Module (15 minutes)

Introduction


 

Welcome to Module 4 - Sonobuoy: Test Your Kubernetes Deployment

This Module contains the following lessons:


 

Sonobuoy Overview

Sonobuoy is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of plugins (including Kubernetes conformance tests) in an accessible and non-destructive manner. It is a customizable, extendable, and cluster-agnostic way to generate clear, informative reports about your cluster.

Its selective data dumps of Kubernetes resource objects and cluster nodes allow for the following use cases:

Sonobuoy supports three Kubernetes minor versions: the current release and two minor versions before it. Sonobuoy is currently versioned to track the Kubernetes minor version to clarify the support matrix. For example, Sonobuoy v0.14.x would support Kubernetes 1.14.x, 1.13.x, and 1.12.x.

You can skip this version enforcement by running Sonobuoy with the --skip-preflight flag.

 

Sonobuoy Installation and Conformance Tests


This part of the lab is presented as a Hands-on Labs Interactive Simulation. This will allow you to experience steps which are too time-consuming or resource intensive to do live in the lab environment. In this simulation, you can use the software interface as if you are interacting with a live environment.

  1. Click here to open the interactive simulation. It will open in a new browser window or tab.
  2. When finished, click the “Return to the lab” link to continue with this lab.

The lab continues to run in the background. If the lab goes into standby mode, you can resume it after completing the module.


Conclusion


This concludes "Module 4 - Test Your Kubernetes Deployment".

In this module we saw how to use Sonobuoy to run a conformance test in accordance with "CNCF - Cloud Native Computing Foundation" and how to analyze the test results.


 

You've finished Module 4.

Congratulations on completing  Module 4!

If you are looking for additional information on Sonobuoy, try one of these:

Proceed to any module below which interests you most:

 

 

 

Test Your Skills!

 

Now that you’ve completed this lab, try testing your skills with VMware Odyssey, our newest Hands-on Labs gamification program. We have taken Hands-on Labs to the next level by adding gamification elements to the labs you know and love. Experience the fully automated VMware Odyssey as you race against the clock to complete tasks and reach the highest ranking on the leaderboard. Try the Kubernetes Odyssey lab

 

 

How to End Lab

 

To end your lab click on the END button.  

 

Conclusion

Thank you for participating in the VMware Hands-on Labs. Be sure to visit http://hol.vmware.com/ to continue your lab experience online.

Lab SKU: HOL-2033-01-CNA

Version: 20200731-001733