Blog

Automated Data Cloning for Cloud-Based Testing of Software Applications

Current software development best practices emphasize frequent integration testing in order to catch problems as early as possible. This ensures that the source control branch used for the next release is always in a working state, and will not need be overhauled right before going to production. DevOps workflows implement CI (Continuous Integration) pipelines to perform end-to-end testing of applications and services on demand, which is typically triggered by a source code check-in. This testing requires a fully-functional environment in which to run, complete with compute, networking, and storage. Supplementing the environment with a test copy of production data, however, can be a tricky problem to solve.

In this article, we will demonstrate how Cloud Volumes ONTAP dramatically simplifies the provisioning of data storage clones for cloud-based testing, and how this can be automated using the most common tools and platforms currently used for DevOps.

DevOps Testing Environments with Cloud Volumes ONTAP


Automating the setup of software testing environments is one of the primary concerns of DevOps engineers. Not only are these environments important for CI and CD (Continuous Delivery & Continuous Deployment), which goes a step further than CI and ensures that applications are readily deployable to production, but also for software developers to perform independent testing of the application changes they are working on. An essential part of setting up a test environment is having access to test copies of production data, such as databases, as this will provide the most realistic conditions under which to test a new software release.

DevOps teams use a variety of tools for automating the setup of new environments, which allow them to treat Infrastructure as Code. Tools such as Ansible, Puppet, and Chef enable server and infrastructure deployments to be defined within a set of files, which can be used to recreate entire environments consistently, repeatably, and without human intervention. These files can also be managed and versioned under source control, just like application code. Ideally, during the setup of an environment for AWS automated testing, or a test environment in Azure, we would be able to easily create a cloud testing copy of production data alongside other infrastructure, and clean it up automatically at the end of a test run.

Integration with Cloud Volumes ONTAP


Cloud Volumes ONTAP is NetApp’s solution for managing enterprise data in AWS and Azure, and provides an extensive feature set that covers storage efficiency, scalability, and data protection, to name a few. By catering for NFS, SMB, and iSCSI block-level devices, Cloud Volumes ONTAP gives you unified control over data storage, in hybrid and multicloud environments.

Cloud Manager is the central platform for deploying and managing instances of Cloud Volumes ONTAP across an entire organization. Setting up a new instance of Cloud Volumes ONTAP can be completed in minutes using an innovative, web-based user interface. Discovering existing deployments, such as on-premises ONTAP systems, is trivial, and this allows for NetApp SnapMirror® replication to the cloud to be set up with a simple drag-and-drop operation.

The Cloud Manager working environments dashboard
The Cloud Manager working environments dashboard for a specific tenant.

Performing these operations through the UI makes it fast and easy to get started with Cloud Volumes ONTAP, however, advanced users need the additional ability to automate these operations as part of a wider orchestration plan. To facilitate this, Cloud Manager provides a REST API interface with full coverage of all available functionality. This gives you an integration point for external tools and processes over the ubiquitous HTTP protocol.

Data Cloning for Cloud Testing with Ansible and Cloud Volumes ONTAP


The feature of Cloud Volumes ONTAP that DevOps teams find most appealing is the ability to instantly create a writable, zero-capacity cost clone of any existing on-prem, AWS, or Azure storage volume, regardless of its size. This is achieved through NetApp FlexClone®, and can be used to create cloud testing copies of production databases, file shares, or any other type of storage residing in Cloud Volumes ONTAP. The clone only requires storage space for the data changes that are made to it, so initially requires no storage at all. After completing a test run, you can clean up the clone by simply deleting it. This saves a huge amount of developer time and capacity that can be put to better use. Saving time and saving money are the real benefits that come with using FlexClone.

Using the Cloud Volumes ONTAP REST API, we can integrate data cloning with the rest of the infrastructure setup for a test environment. Since REST has become the standard for client/server communication, this is possible from nearly every platform in the DevOps tool suite. For example, take a look at the following Ansible playbook that connects to Cloud Manager and automates cloning operations for cloud-based testing:

- hosts: 127.0.0.1
 tasks:
 - name: Login to Cloud Manager
   uri:
     url: http://<cloud manager server>/occm/api/auth/login
     method: POST
     body: '{ "email": "john@example.com","password": "secret" }'
     body_format: json
     status_code: 204
   register: login

 - name: Create a volume clone
   Uri:
     # production_data is the name of the volume to clone
     url: http://<cloud manager server>/occm/api/vsa/volumes/<working environment>/<svm>/production_data/clone
     method: POST
     body: '{ "newVolumeName": "production_data_clone" }'
     body_format: json
     status_code: 204
     headers:
       Cookie: ""

# This would typically be done in a different playbook after all testing is complete
 - name: Delete the volume clone
   uri:
     url: http://<cloud manager server>/occm/api/vsa/volumes/<working environment>/<svm>/production_data_clone
     method: DELETE
     body_format: json
     status_code: 202
     headers:
       Cookie: ""
       
 - name: Logout of Cloud Manager
   uri:
     url: http://<cloud manager server>/occm/api/auth/logout
     method: POST
     body_format: json
     status_code: 204
     headers:
       Cookie: ""
       

These operations can be integrated with any other Ansible tasks, such as creating servers and deploying applications. This makes provisioning up-to-date test copies of production data a natural part of CI/CD pipelines and test environment management.

Conclusion


Cloud Volumes ONTAP provides an elegant solution for integrating the setup of test data with DevOps processes for cloud-based testing. Using NetApp FlexClone, multiple clones of the same storage volume can be made available for concurrent use, and with Cloud Manager’s REST API these clones can be created from tools such as Ansible and Chef. In fact, the deployment of Cloud Volumes ONTAP itself can be automated and integrated with Infrastructure as Code platforms, such as Terraform.

Find out more about cloud automation & testing software with data cloning and Cloud Volumes ONTAP, or start a free trial today in AWS or Azure.
-